Comparing Forecasting Models for 2011 Women's NCAA Tourney

Happy New Year! I wanted to close out discussion of the 2011 NCAA women's volleyball tournament by examining the effectiveness of my newly developed Conference-Adjusted Combined Offensive/Defensive (CACOD) ranking system at predicting the outcome of tournament matches.

Details of the formula and the full set of CACOD rankings are available here. In short, however, for each team in the NCAA field, the CACOD took the "ratio of its own overall [regular] season hitting percentage (offense) divided by the overall hitting percentage it has allowed the opposition (defense)." This ratio was then multiplied by an adjustment factor based on a team's conference (the stronger the conference, the more the adjustment factor raised the team's ranking).

For each of the 63 matches in the tournament, I simply looked at whether the team with the higher CACOD rating won or lost. The CACOD's record is shown below, along with those from other leading rating systems (shown in a screen capture from a VolleyTalk discussion thread). You may click on the graphics below to enlarge them.

The CACOD successfully predicted the winner of 45 tournament matches, which means it generally did as well as the more established ranking systems did. (The reason some of the above records include only 62 matches is that the captured image was from before the final match.* I suspect that, in cases where other systems' records don't add up to 62, it's because some matches featured teams that were tied in the rankings.) What's unique about the CACOD is that teams' win-loss records during the regular season play no role in formulating the rankings, just offensive and defensive hitting-percentage statistics.

During the Sweet Sixteen round and beyond, the CACOD seemed to outperform the other systems, as the CACOD didn't do so well regarding the 48 matches of the first two rounds (32 in the first round, 16 in the second). The results of the two initial rounds are shown in the next graphic.

Indeed, the CACOD trailed the top performing system (Pablo) by five matches after the first two rounds. However, the CACOD went 10-5 the rest of the way to catch up. The results of the last 15 matches are listed below, with teams' CACOD rankings at the close of the regular season shown in parentheses. Successful predictions appear in black, unsuccessful ones in red.

Sweet Sixteen

Texas (7) d. Kentucky (39)
UCLA (11) d. Penn State (12)
Florida St. (28) d. Purdue (2)
Iowa St. (9) d. Minnesota (31)
Illinois (14) d. Ohio State (25)
Florida (10) d. Michigan (33)
USC (5) d. Hawai'i (6)
Pepperdine (29) d. Kansas St. (40)

Elite Eight

UCLA (11) d. Texas (7)
Florida St. (28) d. Iowa St. (9)
Illinois (14) d. Florida (10)
USC (5) d. Pepperdine (29)

Final Four

UCLA (11) d. Florida St. (28)
Illinois (14) d. USC (5)

Championship

UCLA (11) d. Illinois (14)

For next year, I may tweak the formula a little to, for example, place greater weight on hitting-percentage statistics from later in the season than earlier. Seeing how the CACOD did in the end, however, any revisions will likely be more minor than I had expected would be the case after the first two rounds!

---
*I overlooked this point in my original posting, but have now added it.