After all the dust had settled from the State meet I wanted to go back and look at how the projections for regionals, sectionals and state turned out overall.
Regional Projections vs Actual
For the regionals I wanted to focus on how close the projected qualifying teams were with the projections.
| Regional Team Qualifier Projections vs Actual Team Qualifiers | ||||||
| 1A Boys | 1A Girls | 2A Boys | 2A Girls | 3A Boys | 3A Girls | Overall |
| 90/102 | 60/67 | 70/72 | 66/69 | 64/72 | 65/72 | 415/454 |
| 88.20% | 89.50% | 97.20% | 95.60% | 88.90% | 90.20% | 91.40% |
Â
Overall, the projections were reasonably accurate with 91% of the qualifying teams picked correctly. 2A had the highest accuracy with 97.2% and 95.6%, respectively. 1A and 3A Boy had the lowest accuracy at 88%. Looking at genders, the girls projections were more accurate overall than the boys.
Sectional Projections vs Actual
For sectional projections I wanted to look at accuracy of number of qualifying teams, the place differential and the number of teams that accurately predicted their sectional placing
| Sectional Projections vs Actual | |||||||
| Â | 1A Boys | 1A Girls | 2A Boys | 2A Girls | 3A Boys | 3A Girls | Overall |
| Team Qualifers | 29/30 | 26/30 | 25/28 | 25/28 | 24/28 | 26/28 | 155/172 |
| Percentage Qualifiers | 96.7% | 86.7% | 89.3% | 89.3% | 85.7% | 92.9% | 90.1% |
| Place Differential | 1.17 | 0.94 | 0.7305 | 1.39 | 1.27 | 2.33 | 1.305 |
| Percentage Places Right | 24.80% | 39% | 51% | 28% | 34% | 48% | 37.47% |
For Sectionals, again the projections were reasonably accurate with 90.1% of the projected qualifying teams qualifying. 1A Boys was the most accurate followed by 3A girls. In terms of projecting actual places in the sectionals for the various teams the projections on average had an average absolute differential of 1.305 places. For reference I would like to have the team place differential under three. For the accuracy of actual finishing place in the sectional, the projections were not very accurate only achieving a highest percentage of 51% for 2A Boys, but overall, only projecting 37.47% of the places correctly.
State Simulation Analysis
State Simulation Explained
For the state simulation I take each runner’s seasonal rating along with all the performances they have accumulated during the year to come up with the standard deviation for each competitor. The standard deviation accounts for the variation of performance within the simulation. I use the average standard deviation for the whole field in each race.
Once I have the standard deviation for the field, I can randomly generate performances for each runner using their seasonal ratings. For the simulated results I do this 1000 times to get an average finish for each runner and the average team scores.
What did the simulation show?
From the projections that were posted you could see the average finishing total for each team, their finishing range, and percentage chance of winning, earning a trophy and finishing top 10. To dive into it a little deeper, the projection overall for all classes shows that the simulation would correctly project 3.4 team champions, 12.4 of the trophy teams and 49.9 of the teams that finished in the top 10 correctly. To compare with the actual team finish, 4 team champions were picked correctly, 11 teams projected to finish in a trophy position did and 50 of the teams projected to finish in the top 10 did.
Projected vs Actual
Below is a summary of the projected versus actual for each class along with projections accuracy metrics to show average absolute place difference and the number of teams within their finishing range. For all classes, the place differential was 2.55 and the number of teams that fell into their projected finishing range was 97.07%. One thing I found is that the simulation isn’t an accurate projector of team points but that’s expected with human variation. Of the 6 races I would say that projections were most accurate for 1A and 2A Boys. 3A girls was the least accurate of the 6 races.
1A Boys
Metric | Value |
|---|---|
Perfect Place Prediction Rate | 40.00% |
Average Absolute Place Difference | 1.73 places |
Within Range Rate | 100.00% |
Mean Absolute Score Error (MAE) | 33.97 points |
1A Girls
Metric | Value |
|---|---|
Perfect Place Prediction Rate | 6.67% |
Average Absolute Place Difference | 2.73 places |
Within Range Rate | 96.67% |
Mean Absolute Score Error (MAE) | 42.67 points |
2A Boys
Metric | Value |
|---|---|
Perfect Place Prediction Rate | 35.7% |
Average Absolute Place Difference | 1.64 places |
Within Range Rate | 100% |
Mean Absolute Score Error (MAE) | 31.86 points |
2A Girls
Metric | Value |
|---|---|
Perfect Place Prediction Rate | 25% |
Average Absolute Place Difference | 3.40 places |
Within Range Rate | 96.4% |
Mean Absolute Score Error (MAE) | 24.33 points |
3A Boys
Metric | Value |
|---|---|
Perfect Place Prediction Rate | 25.00% |
Average Absolute Place Difference | 2.21 places |
Within Range Rate | 96.4% |
Mean Absolute Score Error (MAE) | 24.33 points |
3A Girls
Metric | Value |
|---|---|
Perfect Place Prediction Rate | 11% |
Average Absolute Place Difference | 3.64 places |
Within Range Rate | 89.29% |
Mean Absolute Score Error (MAE) | 64.57 points |
Wondering what a similar analysis of individual runners would show. Presumably team accuracy is higher than runner accuracy because of offsetting differences within a team’s runners (runner accuracy measurement also depends upon the defined margin of acceptable error for an individual).
Also curious about how accuracy changed throughout the races, using split times and scores, though that may not be materially more insightful than the actual split team scores and changes in team position. (It might not be possible to do when some splits are missing.)
I’m fairly certain that team score/rank is more variant the earlier the split score is taken in the race. As you progress through the race, I think it’s reasonable to conclude that the scores should converge toward a rank of the teams abilities over three miles, which is what these ratings measure. Also, teams can artificially place themselves higher or lower in the pack with more control early in the race, so you’re more likely to have a team run outside of their projected range 1st mile simply because their tactic calls for it (Plainfield north boys as an example, lake Zurich girls at Richard spring as another).