SIVACRACY.NET: Remaining Questions About the US Election

One of my favorite professors at Cornell taught statistics, and recently he started blogging about something important:

Explaining the exit polls?
Posted by Paul Velleman at Left2Right on January 31, 2005

"In the 1970’s and 80’s I worked on election nights for a major network as part of a team of statisticians making “calls” in statewide races (President, Senator, and Governor). Eventually, the team was disbanded because exit polls were so accurate that our expertise was no longer needed.

"But in the past election, the exit polls differed from the recorded vote by an unprecedented amount. Nationwide, exit polls predicted that Kerry had won by 3%, but the final tally showed Bush ahead by 2.5%. Errors in some key states were even larger. As a statistician, I have been concerned that the errors were unexplained. Last week, Edison Media Research and Mitofsky International, who conducted the exit polls on contract for the major networks and news services, released a report on the errors in their polls. Now an organization called US Count Votes has released an analysis of the E/M data that raises serious questions about E/M’s proposed explanations for the exit poll biases.

"All who have considered the problem agree that there are three plausible explanations:

1. Chance error,
2. Bias in the exit polls, and
3. Inaccurate election tallies, or (say it softly) election fraud.

"Edison/Mitofsky and all who have examined the data agree that (1) is not plausible. The errors were so extraordinarily beyond what could occur simply by chance that we can safely exclude this possibility. Anyone who has taken a freshman Statistics course can do the calculations.

"The E/M analysis arbitrarily ignores (3) and considers only possible biases. In one sense, this is understandable. Their job was to predict the final vote totals and they failed miserably. Some commentators were made to sound foolish when they started talking as if Kerry had won by about 7pm on election night. Clearly E/M’s clients didn’t get their money’s worth.

"But as citizens, our job is somewhat different. If the bias in the exit polls can be explained by errors in E/M's methods or by other factors, it would reduce the concern for the vote itself. This analysis of the E/M data makes it clear that the E/M report fails to provide such an explanation. And, of course, if the exit polls were not themselves flawed, that would raise questions about the honesty of the vote itself."

I have looked at the E/M data and participated in the discussion as the US Count Votes document was refined. I have signed their report as a member of the team that reviewed the work. I am posting this commentary here to direct readers of the blog to that report. I think their points warrant serious consideration and that they clarify the need for further analyses of the voting and polling data.

The underlying scientific consideration is that any theory or model that claims to account for patterns in data must make sense no matter how we view the data and must be consistent with other external information. Specifically, if there are biases inherent in the E/M exit polls, we should expect them to follow their own consistent patterns. For example, it isn’t plausible to posit different biases in different places without offering any account of the differences. Such ad hoc explanations are not scientifically or statistically supportable.

Nor is it plausible that a bias present in some locations should inexplicably be absent in others. Conversely, if there is a pattern to the biases that makes sense, the very existence of such a pattern makes the proposed explanation more plausible. For example, one early explanation of the exit poll errors was that (more Democratic) women tend to vote early, while (more Republican) men vote later. Such an explanation could be supported by finding trends in the biases during election day (E/M reported poll results in three waves). But no such pattern is found, and E/M do not propose that explanation. They analyze data from the end of the polling day, and it is those data that show the biases they are trying to explain.

So what do they propose? E/M’s explanation is simply that Bush voters were substantially more likely than Kerry voters to decline to be interviewed. (Specifically that 56% of Bush voters but only 50% of Kerry voters declined.) E/M offer no evidence of this other than the obvious fact that the polls don’t match the recorded vote and the unstated fact that they can find no other explanation. But there are no underlying patterns to support their explanation. For example, E/M look for patterns in refusals and find none. In fact, as the US Count Votes report points out, the response rate of voters willing to be polled is actually a bit larger (although probably not significantly so) in states that voted more strongly for Bush—the opposite of the pattern that would support E/M's hypothesis.

Does the E/M hypothesis account for other patterns in the data? They propose no such explanations and admit to some patterns that their hypothesis fails to explain. For example, states that voted with paper ballots showed only small random errors between exit polls and votes, well within statistical error. States that used automated systems showed large errors fairly consistently biased toward Kerry. It doesn’t seem plausible that voting method would influence a voter’s willingness to talk with pollsters (nor do E/M claim that this happened).

Let me be very clear. I do not assert there was extensive fraud. I would prefer not to think that, and I had hoped the E/M report would reveal a systematic flaw in their methods that accounted for the errors. But it hasn’t, and the issue is still open. The E/M report does not account for the biases in a manner that would support explanation (2). The exit polls may well have been flawed, but we have yet to see a plausible account of how or why.

The data released thus far beg for a more thorough analysis. E/M have not released precinct-level data, which would be necessary to determine whether voting technology is a factor. I hope that they will do so soon. I also hope that the news media report this story so that the public can be widely informed about it. I recognize that if significant problems with the reported vote are found, Republicans will feel that the effort was somehow directed against them. But honest voting is a value that should be supported equally by both Left and Right. We cite discrepancies between exit polls and votes in elections in other countries as evidence of problems. Especially when we have been called to spread liberty and democracy throughout the world, it behooves us to make our own democracy as open and honest as we can.