Most Deserving or Best?
25,000 College Football Fans Have Spoken
Background
College football is unique. There are 134 teams in FBS division 1 playing for one trophy, and a very limited number of games to determine who gets it.
To make things easier, we’ve loosely organized the teams into 10 conferences.
Five of the ten (the doomed Group of 5) are considered bottom dwellers and are basically out of the running before the season starts and the Pac12 basically ✌️’d out last year (RIP).
That leaves the “Power 4”. Those four conferences represent 67 of the 134 FBS teams and are generally considered the only ones to really be in the running for the top prize. Depending on who you ask, there’s even one more level to this caste, but we’ll skip that part for now. Oh, and of course there are the holdouts that won’t join a conference.
So how do we take all of those groups and decide a single winner?
Well, since 1998 there have been 3 different approaches used. The BCS attempted to put the top two teams against each other, as determined by a combination of human and computer rankings.
The 4 team College Football Playoff came along in 2013 to address the shortcomings of that system, most specifically that people were not always thrilled with the teams selected to compete for the title. Along with the widened field came a new poll used to select the teams in the tournament - the committee. That group would avoid the silliness of preseason rankings, starting their process in week 10. They were professionals who would watch every single game and have no ulterior motives. From the outset everyone knew they were going to fix all the wrongs of the BCS era.
Ok maybe that's not true. Almost immediately, people were complaining. For one thing, there were 5 "power" conferences at the time, so every year a winner was going to be left out. Fans wanted an even more expanded playoff. Plenty of proposals floated around the CFB crowd, usually for an 8 or 16 team field. For 11 years we rolled with the 4 team playoff, though, until last season brought us the 12 team bracket. The committee was kind enough to stick around to help us figure out "who's in".
The thing that has persisted through all of that post-season chaos is the importance of rankings, and it's no mystery why. With 12 regular season games and conference champions to determine, teams are not able to play many OOC games. Further, they are typically not very incentivized to play top tier teams in those games. Therefore you end up with sets of teams that you can cleanly rank, but very little data to help you rank the sets against each other.
Despite the indisputable genius behind "the committee", they often made controversial decisions. In 2016, a conference champ was left out in favor of another team from that conference that didn't even play in the championship game. In 2023, an undefeated conference champion from the coveted "Power 4" was left out.
Adding more teams to the mix seemed like it might ease some of the pressure on the committee getting things exactly right, but after year one it looks like more "bubble" teams might just mean more strife. In year one of our 12 team playoff, multiple 2 and 3-loss teams that didn't even play for their conference titles felt slighted when they weren't included. Also, the exact format of the playoff came under scrutiny as having a higher seed meant a bye week or home game to start off the tourny. In fact, none of the teams that missed out on those honors made it past the first game.
Understandably, coaches, fans, and players alike suddenly feel the need to defend their team’s ‘resume’ with anecdotes like “our best win was better than theirs” and “they didn’t play anybody”. Fans don’t just root for their teams, but their whole conference and all of their opponents, so that they can have the best possible argument to make the postseason.
The issue, though, is that without an objective criteria set to determine the final 12 teams, we’re left with subjectivity and bias. The arguments that someone might use to defend their alma mater against the team ranked immediately behind them could very well hurt that team if applied across all teams in the full set.
With that in mind, I made a quiz to help people determine where they really think teams should land in the rankings. The methodology is simple - probably too much so, but the answers to the questions asked are interesting and give a glimpse into the psyche of the 25,000 college football fans that took the time to give them.
The first set of results
Question 1
This was the most controversial question. It had the second lowest number of '3' responses, and about 50% were leaning Green while 37% leaned Blue. We can't seem to agree on who should be ranked higher, but we do think that one of them should be higher.
This question deals mostly with a respondent's opinion on whether a ranking system should award the more deserving team or the team that's currently better. It's also incredibly similar to the situation that FSU and Alabama found themselves in in 2023, which likely caused some bias in the results. Unfortunately, I didn't ask for favorite teams along with quiz results at the time of the original viral post on r/cfb. It might have been interesting here.
The highest scoring answer (at about 41%) is that Green should be ranked a bit higher than Blue. That suggests that most respondents thought that the more deserving team, based purely on record in this scenario, should be given the nod.
Question 2
If question 1 was mostly trying to work out someone's preference on 'deserving' or 'best' team, question 2 was like a full back running straight at that question head on. Most of us (62%) seem to think that the underdog should be ranked higher here - the more deserving team.
Question 3
What about strength of schedule? Question 3 uses bowl eligibility as a simple read of the strength of a team’s opponents. Green has two more losses than Blue, but the vast majority of their teams couldn’t get 6 wins. Respondents mostly thought that winning an easier schedule was more important than losing a hard one.
Question 4
What if you take the overall record out of it? A very common argument I’ve seen on message board for one team over another lays out the losses and best win of the two teams discussed. This question puts a team with two strong losses and no strong wins against a team with a bad loss and a great win. There was more agreement here than on any other question that the bad loss isn’t as important as having an impressive win. Only 16% disagreed with the majority opinion and said that Green should be ranked higher.
Question 5
This question was surprisingly topical last season, mostly because conferences are growing and (similar to the problem of determining an FBS champion) it’s becoming harder to decide on who should even get a shot at their titles. At its core, this question asks if you should consider the championship game to be another conference game or something more. Put differently, it asks if the committee should ‘punish’ CCG losers. The general feeling here is that those losers should not be punished.
Question 6
How about streaks? The next question picks apart a respondent’s appetite for playing the hot hand, and is incredibly even in its responses. Blue gets the nod slightly with their better record. 40% of respondents ranked them higher compared to 38% picking green.
Question 7
This question has more than one factor to consider, with overall record, strength of opponents, and win margin against a common opponent playing in. People lean towards the team that has the worse record, but against stronger competition and a better showing against Team C. In fact, roughly the same number of respondents think that Blue should be well ahead of Green as those that think Green should be ahead at all. What's interesting about this answer is that respondents broke from picking the team with a better record, suggesting that with enough evidence to support it, SOS can trump a simple comparison of wins and losses.
Question 8
This question features two 9-3 teams. One dominated a weaker conference, though, while the other had three losses in a slightly stronger one. Unsurprisingly, many people couldn’t rank these teams. Of those that could, it was more important to beat harder competition than it was to take care of business against conference foes.
Question 9
This question again challenges you on record vs. strength of opponents. It’s more granular than question 3 though in that it gives an overall opponent win %. Blue has just one loss to Green’s 3, but has played much easier competition. These questions were not based on real scenarios, but it’s worth noting that Green’s opponents’ records are stronger than any team’s in 2024 while Blue’s would have put them right towards the bottom of the league. Respondents didn’t care, though, and rewarded Blue for beating teams they were supposed to beat.
Taking this with question 7, you can see see how important overall record really is. In (7) we overlooked a 1-win difference because of a stronger set of opponents. Here, the SOS gap is even wider, but 2 losses were too much to overcome.
Question 10
The final question pins an on-the-field measure (win margin) against talent composite, and roughly 60% of respondents thought that results on the gridiron matter more.
Summary
These results show a very diverse set of opinions across the college football crowd. Generally, record is the most important indicator for people to assess a team, with evidence of stronger schedules following closely behind. We seem to think that the ‘deserving’ team should go over the ‘best’ team, but disagree a bit on what constitutes ‘deserving’. We also care more about big wins than shocking losses, and off-the-field things like talent ratings just don’t matter all that much.
At least that’s what this data suggests. When toe meets leather we’ll see how things shift. Since the first pass, I’ve updated the quiz to ask for team preference at the end. It’ll be interesting to see how team preference impacts ‘unbiased’ opinions.
Come back and visit in the coming weeks as I take a look at the last several seasons and how the final rankings stack up to our unbiased ones.