Special(ly Long) Report:  Ranking The Polls (posted 8/28/08)

Ever wonder how voters are chosen and what they’re instructed to do?  Here is your chance to find out.  We contacted officials at the Associated Press (AP), USA Today, American Football Coaches Association (AFCA) and Harris Interactive to find out how they conduct their polls and compare them to each other.  In typical Pollspeak fashion, we even rank their answers in each category.  Let’s see how they like it! 

Conference or Geographical Quotas:

AP  – Votes are based on how many FBS (Football Bowl Subdivision) schools are in a state -- 1 - 3 schools equals one vote, 4 - 6 equals two, etc.

Coaches – Slots are apportioned by conference, with a slight edge in representation given to those conferences that were most successful during the previous season. Independents are also in the mix.  How is success determined?  Mel Pulliam at the AFCA uses a formula that generally reflects how conferences were represented in the final poll. It's not a major factor but it usually means there may be one more voter in the Big Ten than in the WAC, for example.

Harris -- Panelist slots are filled by conference, each represented equally and all randomly selected to be invited to participate from among its list of nominees.  Eleven conferences each have 10 panelists and the independent institutions comprise the remaining 4, for a total of 114.

Pollspeak’s Take – The AP is straying from its own formula.  Texas has 10 FBS schools and has 4 votes.  California has 7 FBS schools and has 4 votes  (so we can assume 7 – 10 schools = 4 votes).  Florida has 7 FBS schools but only has 3 votes.   Ohio has 8 FBS schools (6 in the MAC alone) and only has 3 votes (one of which is Kirk Herbstreit, who should be listed under National.).  Illinois has 3 FBS schools but still gets 2 votes.  There are more examples, but the AP needs to rework or stick to its own formula. 

With Harris every conference has the same representation (although we’ll need to verify this after they announce the voters).  The Coaches’ Poll has a more complicated formula, and they haven’t shared the details.  The AFCA has not responded for comment (although the USA today has been very helpful).  So we can’t verify if they’re following their own formula.  However, we do know that the conferences are currently unbalanced.   See our article on the Coaches’ Poll below.  We also know that the intent is to give an extra reward to the conferences that do best in the polls.  This means that if votes are unfairly balanced in the first place, this system will reward those already at an advantage.  Basically, the rich are getting richer.  While the AP may have made mistakes, at least the overall concept is fair.

Choosing the Voters:

AP – The bureau chief, many in conjunction with the sports writer, select the voter(s) from their state/region. Some rotate the vote every year to give different members a chance, some don't.  About half of the writers/broadcasters let the local sports writer or bureau manager know he/she is interested in voting; we also seek out writers/broadcasters.   Usually we contact the individual directly.

Coaches – All 119 Division I-A coaches are eligible, though some decline to participate because of the time commitment necessary for voters to fulfill their weekly obligations. We contact the prospective voters (coaches, not universities) and ask if they are willing to take part and meet the requirements that have been outlined.  Representatives of the USA Today and AFCA meet to review the proposed list.      

Harris -- Panelists are randomly selected from among more than 300 nominations submitted by the conference offices and the independent institutions. On behalf of the Bowl Championship Series (BCS), Harris Interactive constructs a panel of former players, coaches, administrators and current and former media who are committed to ranking the college teams each week during the college football season.

Pollspeak’s Take – The Coaches’ and Harris Interactive polls have a more structured approach to choosing voters at the national level whereas the AP delegates choices to the state level.  This means different states can handle things differently in the AP.  Some states might be constantly changing their voters while others never change.  This inconsistency drops the AP to third.  Harris edges out the Coaches’ simply because it has a MUCH larger pool of candidates to draw from.

Voter Qualifications:

AP – They have to be involved in college football coverage locally, regionally or nationally. 

Coaches – Most coaches have a lot of experience in the game, so we look for reliability and adherence to the basic guidelines.

Harris -- Panelists must be former players, coaches, administrators or current/former media who are committed to ranking the college teams each week during the college football season.

Pollspeak’s Take – All of these are pretty much equal responses – beyond the need to belong to the proper organization, they only need to have a pulse and know how to work a phone.  We would have given the edge to anyone who mentioned a certain level of experience, a requirement to watch a voter training video or watch a certain number of games each week.  We’ll call this a tie for third, since a tie for first would imply they’re all doing great.

Turning Down Voters:

AP – We turned down a voter who was also involved in Harris Interactive or whatever it was called before.

Coaches – When reviewing the list of possible voters, over the years we have steered away from one or two who might have been shown a pattern for being late with their ballot. No more than one or two have not been invited back because of questionable selections.

Harris -- Harris Interactive communicates a set of principles with panelists which they are expected to adhere to.  One principle is that they receive permission from their employer, if such permission is required.  Many media outlets preclude their employees from participating in the Harris Interactive College Football Poll.

Pollspeak’s Take – OK, once again we’ve got a tie for third.  These are all logistical reasons (or don't pertain to new voters).  We were hoping for answers like, “yes, we turned down somebody who wrote in his bio that one of his all time favorite sports moments was ‘Every time Tennessee and Auburn lose a football game.’”  or “We avoided a coach who let his Director of Player Development vote his own team #1 instead of Texas as they had planned.”  etc.

Voter Guidelines:

These are the instructions each poll sends to the voters:

AP – 

·         Base your vote on performance, not reputation or preseason speculation.

·         Avoid regional bias, for or against. Your local team does not deserve any special handling when it comes to your ballot.

·         Pay attention to head-to-head results and

·         Don’t hesitate to make significant changes in your ballot from week to week. There’s no rule against jumping the 16th-ranked team over the eighth-ranked team, if No. 16 is coming off a big victory and No. 8 just lost 52-6 to a so-so team.

·         Teams on NCAA probation ARE eligible for the AP poll.

Some ethics guidelines:

·         Voters should have no professional or booster connection to the schools they cover. This could pose a conflict of interest.

·         Voters cannot write for team media guides or even independent fan magazines associated with certain schools. This has the potential of compromising a reporter’s objectivity.

·         Adhere to APME and APSE standards of conduct: We do not accept free tickets, deals, discounts or gifts, etc.

·         Please be advised: "Homerism" will be challenged and could lead to dismissal from the poll board.

Also noted, “We vote for schools on probation.”

Coaches –

Before the preseason poll is conducted, all voters receive a letter from AFCA executive director Grant Teaff reminding them of what he and the newspaper expect each week. Basically, it's a ballot that's cast without fear or favor and delivered on time. It must be personally drawn up by the coach and preferably phoned in by him as well, though it is acceptable to have a staff member call if necessary. Failure to adhere to these guidelines can result in removal from the board of voters. The AFCA also reviews the guidelines in a letter sent out before the final regular-season poll:

·         The coaches are told they can vote for any eligible Division I-A team. They are told that any team on major NCAA or conference probation is not eligible (if a team's status should change during the season, we do notify them).

·         They are asked to keep their votes thoughtful, reasoned, consistent and fair throughout the season.

·         They are reminded that no abstentions are permitted. If you commit to the poll, you commit to it for every week of the season.

·         They also are reminded that, for the post-bowls poll, that the AFCA recognizes the winner of the BCS title game as the No. 1 team.

Harris –

·         Panelists receive permission from their employer, if such permission is required.

·         Each panelist agrees that ranking teams is a time-consuming process each week.

·         Panelists agree that rankings must be based on their personal observations and analysis of team performance.

Pollspeak’s Take – Harris is basically saying that a well-prepared voter is one who knows how to use a remote control for long periods of time without complaining about carpal tunnel.  The AP wins out over the Coaches’ poll because they give specific and useful advice and spell out the penalty for bias.   The Coaches’ Poll is at a disadvantage here since everyone who votes in their poll is a school employee with an inherent bias.

Monitoring and Replacing Voters During the Season:

AP – We monitor ballots on a weekly basis.   They can call, fax or e-mail, and there's a human on the phone.  Other than not following the guidelines, a voter may be replaced for “repeatedly failing to call in a vote promptly,” or for a “conflict of interest,” or “refusing to vote for a team on probation.”   (Also) we evaluate all voters every year.

Coaches – - Votes are called in by phone and recorded. One of our college desk staffers then enters them into a computer. He follows up on any ballots that might be questionable. (This individual has been doing it for more than 10 years.)

We closely examine each ballot every Sunday. If we see something that seems the least bit out of whack, we contact the coach, with whom we can usually resolve the issue. Most of the time, it's over an inadvertent omission. If it cannot be resolved there, we will bring an AFCA representative into the discussion. 

The only voters who have been replaced during the season have been a few who were dismissed from their coaching position and opted not to continue to participate in the voting. Our preference is that they stay on through the bowl games, and most have agreed to do so. Those who have declined have been succeeded by the coach who replaced them. 

Harris -- Each week, Harris Interactive has rigorous processes in place to ensure that each panelist’s rankings reflect their intent.  We utilize three methods for voting, including phone (live interviewer), fax, and online. (Also)  if those members of the media who are panelists change employers either before or during the season, they may become ineligible for the HICFP.

Pollspeak’s Take – The AP earns top spot for taking action during the season for the way a person has voted.  Coaches’ come second for at least replacing missing voters.  Harris comes in third since it seems that clerical issues are the main concern.  See last year’s Pollspeak Report to Harris Interactive - College Football 2007 which shows that Harris Interactive actually allows missing ballots during the season, seemingly without penalty to the voter during that season.

Accessibility:

AP – The AP was completely open and very responsive.   The people directly in charge of the poll were kind enough to answer quickly even from Beijing.

Coaches – The USA Today ranks right up there with the AP.  Again, the right people answered quickly even while in Beijing.  However, the AFCA was completely unresponsive.  Like a 1-star recruit, they dragged down the Coaches’ Poll class ranking.

Harris – A very nice PR person acted as a liaison but no direct access to those in charge of the poll.  Also, response time wasn’t as good as the AP or USA Today…and they lose a spot for not being in Beijing.

Pollspeak’s Take – We would like to thank all of the organizations (except the AFCA) for taking the time to answer our questions. 

Secret Ballots:

AP – Has no secret ballots.  Every ballot is posted for every voter every week.  This allows football fans to BROWSE them or analyze them in-depth using POLLSTALKER.

Coaches – “We are satisfied the current process provides the soundest football rankings.”

Harris – “The BCS has set the guideline that the final rankings will be made public.” 

Pollspeak’s Take – Secret Ballots are like the bowl game of our questions.  Both the Coaches’ Poll and Harris Interactive are satisfied with keeping ballots secret because they aren’t forced to do otherwise (other than the last poll of the regular season).  We asked them if there was some advantage to this method, but never received a proper response.   The AP is the only major poll to have every ballot public and is therefore the only poll to receive points in this category.  Why aren’t the other two tied for second?  Because we have declared them ineligible for post-season play and forbid our voters from ranking them.  How do you like them apples?  (Pollspeak sponsors a PETITION to make all ballots public.)

The Final Tally:

Score is determined in the standard way.  Every first place vote is worth 3 points, second place is 2 points, and third place is 1 point.  So the final scores are:

Congratulations to the AP for being ranked pre-season #1!  The poll isn’t perfect, but it is the best we have.   Keep in mind that these are just pre-season rankings.  If the polls keep fighting each week to improve and provide us with timely and accurate information, they may jump in our future rankings.