I see the fine folks at the Washington Examiner who were so badly burned before the election by shoddy Republican polling feel they’ve found a smoking gun. “PPP’s polls were rigged all along!” trumpets the link in the paper’s Morning Examiner roundup, and over at a blog called The Unlikely Voter, we find an interview cribbed from New York Magazine in which Tom Jensen, the head of the Democratic polling firm, says that “We just projected that African-American, Hispanic, and young voter turnout would be as high in 2012 as it was in 2008, and we weighted our polls accordingly. When you look at polls that succeeded and those that failed that was the difference.” The accompanying commentary from New York:
But Jensen conceded that the secret to PPP’s success was what boiled down to a well informed but still not entirely empirical hunch. Given the methodological challenges currently confronting pollsters, those hunches are only going to prove more important. “The art part of polling, as opposed to the science part,” Jensen said, “is becoming a bigger and bigger part of the equation in having accurate polls.”
One suspects that it’s less Jensen’s methodology that the Examiner faults than his assertion that “These supposed polling experts on the conservative side are morons” and that Jay Cost, whose pseudo-scientific interpretation of polls at the Weekly Standard produced some of the worst pre-election projections, “is an idiot.” Harsh. (But honestly, polls aside, Cost really is kind of an idiot.)
So was PPP really “cooking the books all along,” as The Unlikely Voter’s Neil Stevens so gleefully asserts? Sigh. No, no, they weren’t. Let’s review what the debate about “skewed” polls was about. As the National Journal’s Reid Wilson writes in an evaluation of the GOP’s “embarrassing polling failures”:
The underlying causes of the errant numbers are the assumptions that the pollsters made about the nature of the electorate. Most pollsters believed the electorate would look something like the voters who turned out in 2008, just with slightly lower numbers of African-Americans, younger people, and Hispanics heading to the polls.
But exit polls actually showed a much more diverse electorate than the one forecast. Black turnout stayed consistent with 2008, Hispanic turnout was up, and younger voters made up a higher percentage of the electorate than they had four years ago.
People like Nate Silver indeed criticized Republicans for assuming an electorate that would look more like 2004 than 2008, but that wasn’t what made pollsters like Rasmussen and “unskewers” like Dean Chambers laughingstocks. The argument over the makeup of the electorate is a legitimate one, as it determines how a survey company weights each class of respondent: Do you go for a universe with more blacks and fewer whites, one with more high-income earners or one with more lower-class voters? Silver and company argued that the Republican weighting was faulty, but not that such weighting was an inexcusable exercise. Every pollster does it; it’s impossible not to, as calling random numbers will never generate a pool of respondents that exactly mirrors the racial and gender breakdown of the nation.
What led independent pollsters and mainstream analysts to conclude that outfits like Rasmussen were “fixing” their polls was the practice of weighting by party ID. Neil Stevens doesn’t seem to understand this difference. He hears that PPP made an educated guess about the electorate – as every pollster does – and cries foul. But what PPP doesn’t do is weight by party ID, the real sin that everyone who laughed at Dean Chambers were pointing out. While weighting surveys by immutable characteristics like race and income level is standard, weighting a survey by shifting preferences like party ID defeats the purpose of giving a survey at all. It’s one thing to weight a survey for eight percent African Americans on the assumption that eight percent will turn out and you want to know how they’ll vote. Whether that should be eight or ten percent is a legitimate disagreement, and Silver wasn’t saying it was illegitimate, just that the Republican assumptions were incorrect. What’s illegitimate is weighting a survey by thinking eight percent will vote Republican. That doesn’t tell you anything you didn’t go in already knowing. If you know in advance how many people are going to vote Republican – say, 39 percent, an estimate bandied about by conservatives that turned out to dramatically overstate the GOP’s share of the vote – then why bother asking “for whom will you vote?” at all?
After the election, some Republican pollsters conceded that their metrics were flawed, admitting that they “missed Obama’s growing share of the Latino and Asian vote.” On its blog, Public Opinion Strategies — the firm of Romney pollster Neil Newhouse — posted the following:
To me, the biggest surprise, and concern, is that Democrats were 38% of the electorate, while Republicans were just 32%. The spread in 2008 was 39% Dem/32% GOP – so this electorate was really no different.
Also:
I had made the case that, as a party, we were overpolling older voters in Presidential years. (Heck, we are in non-Presidential years too). Now, one big assumption I made was, between enthusiasm drops and the economy, younger voters would fall to a 40-42% share of the electorate. That turned out not to be true.
Props to POS for at least admitting its error, especially given some of its bordering-on-ridiculous results — on November 3, it labeled Minnesota (!) a “toss-up” due to (wait for it) “the low minority population of the state.” Still, as Politico notes, all mea culpas aside, “the question that Republican pollsters haven’t really answered yet, is why nobody seemed to revisit their core assumptions about the makeup of the electorate and the value of GOP enthusiasm in the face of so much public and Democratic polling showing contrary results.”
In fact, GOP pollsters even went a step further, ignoring sensible metrics — age, for one, as well as race and income — in pursuit of a universe perfectly weighted by party ID. They instead used criteria that prejudiced the opinions, not the identities, of their respondents. Rasmussen looked for conservative respondents, not – as other pollsters did – simply Latino respondents, or Southern respondents. A Republican poll in Michigan with a breakdown of eight percent African Americans was deemed acceptable, despite the fact that blacks made up 13 percent of the electorate even in 2004, as long as it contained a certain (favorable) number of Republicans. Rasmussen was more concerned with weighting by party than with the voters it missed by not calling cell phones – and therefore it missed the voters most likely to vote for President Obama. and instead used measures that prejudiced the opinions, not the identities, of their respondents. “As a part of the Republican polling community,” writes POS in its what-went-wrong blog post, “our prescription includes doing at least one-third of the interviews with cell phone respondents going forward.”
Other Republican polling outfits may not be as willing to reevaluate their methods, however. In discussing the aforementioned Michigan poll, Nate Silver speaks to the pollster, who provides a good example of the conservative reliance on unscientific assumptions and convenient predictions. Steven Mitchell, who “simply doesn’t believe the exit polls” about African American turnout, says that “I do not believe blacks represented 12% of the vote in 2008 and I don’t believe they will in 2012.” Silver is incredulous:
What is the evidence for Mr. Mitchell’s claim? He didn’t present any of it in the memo . . . . It is precisely because of this problem [of response rates among minorities] that essentially all polling firms do weight by race, usually calibrating their numbers based on Census Bureau or exit poll data. Mr. Mitchell, however, seems to have his own ideas about what Michigan’s electorate will look like. There is a certain amount of art in political polling, but I’ve never heard of a pollster treating the demographic makeup of a state as essentially a matter of opinion.
That is what was wrong with Republican polls — not their weighting by race (which, yes, PPP does as well) but their decision to base such weighting on faulty, factually untenable models. Combined with the tendency to “skew” polls by factoring in party ID, the GOP was naturally headed for disaster. As Rob Jesmer, head of the National Republican Senatorial Committee, laments to the National Journal, “Everyone thought the election was going to be close. How did [Republicans] not know we were going to get our ass kicked?”
. . . So Well Read. . . 