Numbers Don’t Lie, But Do They Fib?

Nate Silver’s election forecast (NYT)

Few things in politics are as simultaneously fascinating and frustrating than the horse-race polls. Part of me is interested enough in how the country sees the candidates to scroll through the cross-tabs, parsing which income levels throw their support to Romney and which age groups break for the president, but part of me wants to roll its eyes and wonder, What’s all the fuss? Between the polls themselves and the endless analyses that they produce, a lot of energy is wasted on a question that could best be answered simply by waiting until November 6. This time of year, the polls come fast and furious; National Journal counts 82 in the last 12 days alone — and that doesn’t even include state-level surveys for down-ballot races for the Senate and House. Unsurprisingly, the media wastes more ink on spinning the numbers, or at least reporting on the spin, than on the actual results themselves. There are, after all, only so many lead articles one can write about President Obama’s three-point lead, or about the swing vote in Wisconsin and Ohio. Discussion of those points and percentages, however, could fill books.

Thus we are treated to the spectacle of Fox News contributor and Wall Street Journal columnist Karl Rove declaring that Obama is “in desperate shape in territory he carried with ease in 2008” while his employers provide evidence directly to the contrary: “Headwinds for Romney in Latest Poll Results,” the Journal writes, offering a laundry list of states in which the Democrat is topping the Republican, including a surprisingly large eight-point margin in Iowa. “Obama has edge over Romney in three battleground states,” Fox reports, and even admits that the president’s lead is more than a paltry percent or two: “Obama tops Romney by seven percentage points among likely voters in both Ohio (49-42 percent) and Virginia (50-43 percent). In Florida, the president holds a five-point edge (49-44 percent),” roughly the same margins found by the WSJ. (Don’t give Fox too much credit. The next line: “The good news for Romney is that among voters who are ‘extremely’ interested in this year’s election, the races are much tighter.”

Though Romney likes to accuse President Obama of making excuses for a poor economy, his backers come up with an astounding number of their own excuses for Romney’s lagging numbers. And the longer the Republican trails in the polls, the more strained and unbelievable the excuses become. Success is defined down; now Romney is winning simply by virtue of running anywhere close to the incumbent — an incumbent, you’ll remember, that in the Republican worldview has done such a horrible, incompetent job that he should just resign tomorrow. His game plan “doesn’t need a turnaround,” Romney said in a CBS interview. “We’ve got a campaign which is tied with an incumbent president to the United States.” Republican pollster Whit Ayres makes the case that Obama’s steady lead is too narrow to say anything about Romney. “Ayres said Obama’s average lead in the public polls was 2.3 percent in June, 2.5 percent in July, 2.4 percent in August, and has been 2.6 percent in September,” National Journal reports, quoting Ayres as saying (a tad defensively, I’d imagine): “It is a flat line…. That doesn’t strike me as a done deal, a race that is over.”

So why aren’t the American people seeing through the Obama facade? A sampling of the grasping-for-straws logic: blame it on the convention bounce, the Bill Clinton “sugar high,” the food stamps that the president is ostensibly doling out to buy the votes of the poor. “This year’s DNC was the latest incumbent convention in American history!” exclaims Jay Cost at The Weekly Standard. “That absolutely has to be taken into account when examining the president’s standing in the polls, and it means we would be wise to discount his margin by a little bit.” Admirably reaching beyond the worn-out (and ultimately inaccurate, as Reagan actually led in the polls after the convention) Reagan-Carter comparison of a late-breaking challenger, Cost argues that “historically speaking, this president is in weaker shape than any postwar incumbent who went on to victory, with the possible exception of Harry Truman.” Sean Trende, a conservative analyst whose association with Real Clear Politics and its reputable daily poll averages gives him a veneer of neutrality, makes the case that the polls just aren’t registering the dire state of the economy. “So if the election were held today, President Obama would probably win comfortably,” he writes. “But the election isn’t today. In the next seven weeks, the economy, the president’s tepid job approval ratings, and Romney’s spending campaign will continue to exert gravitational forces on Obama’s re-election efforts.” For Romney to overcome the president’s current lead is “not a particularly tall order.” The Washington Examiner, which runs at least one article per day discounting Obama’s edge in the polls as fabricated by the liberal media, tries to put a positive spin on Romney’s status, opining that a poll of 12 swing states “shows Obama up by just two points over Romney 48 percent to 46 percent” and immediately adds a caveat: “That same poll showed 22% of voters could change their mind before election day.” Considering that conservatives spend an inordinate amount of time pointing out “fishy” percentages in mainstream polls, it’s curious that no one at the Examiner found it notable that every other survey puts undecided voters around a mere six percent of the electorate. Of course, this is the same outfit that describes a poll conducted for the flagship Murdoch broadsheet thus: “A new NBC/WSJ poll purports to show that Obama is beating Romney 50 percent to 45 percent.” Well, a poll either shows something or not. You can argue over its accuracy, but a poll can’t purport to show a result any more than the SAT can purport to give Junior a score of 1425.

For as closely as both sides watch the polls for any hint of an uptick for their candidates, partisans actually spend the majority of the time trying to discredit numbers they don’t like.  Conservatives are almost uniformly more hostile to polling (and more hostile, I’d note, to anything that smacks of science – evolution, climate change – or academia in general). Not coincidentally, such hostility spikes when the data is unfriendly, as it has been for Romney over the past week or two. Don’t like the results? Quibble with the methodology. Before most surveys shifted to a likely voter screen in the past month, Republicans complained that the universe of registered voters favored Democrats who wouldn’t actually turn out to vote. Now that the likely screen is in place and the results are equally as dismal — Romney indeed does better among likely voters, but not well enough to erase the president’s edge of one to eight points — conservatives have fallen silent about this particular argument. Predictions that Romney would swamp Obama once polls switched to a more favorable pool didn’t pan out (somewhat improbably, the Fox News and UPI polls actually have Obama doing better among likely voters), so they’ve gone back to their standard whine about polls in general: Like the rest of the media, pollsters are unavoidably biased toward liberals and are doing their part to boost the president’s reelection chances.

The most common “evidence” for such bias is that pollsters – everyone except for the vaunted Rasmussen, who self-identifies as conservative and pens opinion pieces about the dangers of giving government bureaucrats control over health care – routinely oversample democrats. “It is hard to imagine more pro-Obama turn-out being devised by either Marist or Quinnipiac,” wails radio talker Hugh Hewitt in the Washington Examiner. (He ironically faults the methodology of polls “paid for by the MSM.” Apparently Fox News, which gives Obama a larger lead than nearly any other poll, is now mainstream media.) The argument contains a grain of truth, though the impact of such alleged oversampling is almost certainly exaggerated. There are legitimate reasons that more respondents identify as Democrats, beginning with the simple fact that more people simply identify as Democrats. The Pew Research Center’s national numbers put Dems at 32 percent of the electorate, Republicans at 24 percent, and Independents at 38 percent. Elections are close-run things, dividing the country almost exactly down the middle, but it is a fallacy to assume that party identification precisely tracks voting patterns. (If you believe 50 percent of the country must be Democrats and 50 percent Republicans, what happens when one party registers hundreds of thousands more new voters, as Dems did in 2008?) Sean Trende writes that “the same thing has occurred in every election. The losing side objects to the partisan composition of polling. The polls then proceed to get the final result roughly correct.” In 2004, it was Democrats doing the complaining; this year, it has become “a consistent theme among Republicans this cycle: looking at the party ID numbers and discounting polls that show substantial Democratic advantages.”

Polls do indeed include more Democrats, but this is due less to conspiracy than the edge the party has held for decades in voter identification and turnout. Andrew Breitbart’s Big Government site, which sees conspiracies behind more trees than the John Birch Society, flags a poll from Marquette University that shows an unusual uptick in respondents identifying as Democrats — and immediately concludes that “the University inflates their sample and lifts Obama into a 14-point lead.” When “reporter” Mike Flynn wonders, “Why did the school only radically change its sample after Romney began erasing Obama’s lead in the polls?” he showcases his ignorance. Most reputable pollsters can’t “change” their samples because they don’t choose their samples to begin with, at least not on the basis of party ID. With the exception of Rasmussen, polling outfits treat party ID as a response — something they are asking people, the point of conducting a survey in the first place — rather than as an intrinsic characteristic, like race or gender, for which the sample should be weighted. The Pew Research Center, a frequent target of conservative attacks, supplies this explanation: “Party identification is one of the aspects of public opinion that our surveys are trying to measure, not something that we know ahead of time like the share of adults who are African American, female, or who live in the South.”

There are far more mundane explanations for a seeming excess of Democrats than Marquette’s liberal academic cabal. Even if 2010 (when Tea Party fervor prompted a surge in Republican turnout that gave the parties parity at the polls) is used as a reference point instead of 2008 (a year of unprecedented liberal enthusiasm), the party ID numbers, with their steady increases in Independents at the expense of Republicans, still favor Democrats. In other words, even in a year in which Republicans voted in above-average numbers, they barely equaled the depressed number of Democrats. 2012 is not 2010 — most obviously, it is a presidential election, not a low-turnout midterm — and the partisan breakdown at the exit polls should land somewhere between 2008 and 2010. Andrew Hacker, writing in the New York Review of Books, observes that Republican confidence in a 2010-esque electorate may be “misplaced.” Forty-five million invigorated Republican voters turned out for the Congressional election, “well below the some 67 million they will need this year for a presidential majority.” In addition, “The 2010 electorate was also older, conspicuously white, and invigorated by its Tea Party allies . . . But 2012 will bring a much larger and more varied number of voters to the polls.” The question, of course, is how many “more varied” voters will show up. Will anti-Obama grievances produce an electorate that more closely resembles 2010, or will the president’s 2008 coalition of minority-educated-female voters reassert itself?

Regardless of the exact numbers, when conservatives complain that a poll has Democrats at +8. they ignore the fact that even the most tentative turnout estimates have Dems at +4 to begin with. A look at the responses to Pew’s long-standing question of “what would you call yourself?” shows that the country’s percentage of Democrats has held steady over the past two decades (from 33 percent in 1990 to 32 percent in 2012), while Independents have made major strides (29 percent to 38 percent) almost entirely at the expense of Republicans, who have lost eight percent of voters (31 percent to 24 percent). Conservative critics ignore the surge in people identifying as Independents, who are obviously voting for one candidate or the other – and to keep the near fifty-fifty split, it must often be the Republican. In an ideal conservative poll, party identification would be weighted in the same manner as other demographic characteristics like age or income. Indeed, Rasmussen employs just such weighting, which is one reason its results diverge from practically every other survey, giving Romney a one- or two-point edge that is seen nowhere else. (Rasmussen also has other issues with its methodology — see below.) Rasmussen’s tactics are a solution in search of a problem, as Sean Trende points out: “The problem is that party identification is not an immutable characteristic, such as race, age, or gender. It fluctuates.”

Another weakness in the conservative oversampling argument is its focus on party identification rather than ideology, which Trende notes tends to be “more consistent over time and . . . less susceptible to question wording and ordering.” A sample that is heavily Democratic is not necessarily heavily liberal. A Washington Post poll that talked to 25 percent Republicans, 34 percent Democrats and 34 percent Independents found that a whopping 39 percent actually identified as conservative, while only 29 percent described themselves as “liberal” and 20 percent as “moderate.” This is hardly a new trend, as the electorate has contained far fewer self-described liberals for years. Presumably, as most presidential elections are nearly 50-50 affairs, some of those “conservatives” vote for the Democrat, just as some of the people registered as Democrats in the surveys criticized by the right obviously vote for the Republican candidate. Such discrepancies are particularly evident in the South, where many older people in bright-red states are still registered as Democrats, a relic of the years in which the party still had a lock on white voters in the region. Even National Review, in its more sober moments, admits that Democrats have an advantage. Katrina Trinko offers “another possibility” to the usual accusation of bias:

At this particular point in the race, a higher percentage of voters may be identifying themselves as Democrats. That doesn’t mean they will still see themselves as Democrats on Election Day, but for whatever reason (such as excitement over the convention) they do now.

Interestingly enough, even center-right analysts like Sean Trende note that party registration is not the same thing as party loyalty; in a CBS/NYT poll of swing states much criticized by conservatives, the Florida breakdown was 27% Republican, 36% Democrat, and 32% Independent — ratios slightly more favorable to Democrats than the 2008 exit polls (34 R/37 D/29 I) showed. But when the survey asked about party registration, the Republican numbers actually started looking closer to the parity conservatives insist should exist: fully 42 percent were registered as Democrats, with 36 percent as Republicans and 20 percent as Independents. How a person is registered is clearly not the same as how that person votes. Trende dismisses the idea that Romney’s poor poll numbers can be attributed to an oversampling of Democrats. Because the ideological sample tracks more closely with the 2010 exit polls that conservatives prefer over the 2008 polls that reflect an energized Democratic Party, “this suggests that the ideological orientation of the surveys isn’t particularly skewed, but that Romney is doing unusually poorly among self-identified moderates.” Conveniently, Romney backers never mention that, while the electorate may officially skew Democratic, it also skews heavily conservative.

By far the most bizarre and deliberately mendacious conservative objection to standard polling methodology comes via Republican pollster John McLaughlin, who tells National Review that “the Democrats want to convince [these anti-Obama voters] falsely that Romney will lose to discourage them from voting.” It’s worth unpacking his “argument” point by point to see just how off-base it is, and just how damaging such an analysis should be to McLaughlin’s business prospects. Would you hire an expert who can’t even get the facts straight on his own field?

So they lobby the pollsters to weight their surveys to emulate the 2008 Democrat-heavy models. They are lobbying them now to affect early voting. IVR [Interactive Voice Response] polls are heavily weighted.

IVR polls (which use robo-calls — press one if you like Romney! —  instead of live interviewers) are indeed heavily weighted. But the only major robo-poller who weights its samples for party identification is . . . Rasmussen, which routinely puts Romney ahead of the president and hardly uses “Democrat-heavy models.” Nate Silver reminds us that “automated polls, like those from the Rasmussen Reports, have had lukewarm results for Mr. Obama. A Rasmussen Reports poll released on Thursday, for instance, put Mr. Obama three points behind in Iowa.” Despite reservations about robo-poll methodology, Rasmussen’s dubious weighting technique, and the exclusion of cell-phone owners, “Mr. Romney would much prefer the robopolls, warts and all.” So much for that lobbying.

You can weight to whatever result you want. Some polls have included sizable segments of voters who say they are ‘not enthusiastic’ to vote or non-voters to dilute Republicans. Major pollsters have samples with Republican affiliation in the 20 to 30 percent range, at such low levels not seen since the 1960s in states like Virginia, Florida, North Carolina and which then place Obama ahead.

You could weight to whatever results you want. But legitimate pollsters don’t. Only Rasmussen does. As for Republican affiliation in the 20s and 30s being “low” . . . . in 2012, 24% of Americans identified as Republicans.

The intended effect is to suppress Republican turnout through media polling bias. We’ll see a lot more of this. Then there’s the debate between calling off a random-digit dial of phone exchanges vs. a known sample of actual registered voters. Most polls favoring Obama are random and not off the actual voter list. That’s too expensive for some pollsters.

This is perhaps the weirdest criticism of all. Polls that call random digits still only speak to registered voters. In fact, the very first question on the survey — and I gave out lots of them as an interviewer at a push-poll outfit in 2004 — is a screen: “Are you registered to vote at this address?” The polls favoring Obama have never talked to “random” people. Earlier in the year, they spoke with all registered voters; as the election years, they have narrowed their sample even further to “likely voters,” a change which helps Romney put which still leaves him trailing the president by one to two points. And how interesting that McLaughlin should prefer the “actual voter list,” considering the efforts his party has made to kick Democrats off voter lists across the country . . . only to find out, as in Colorado, that the vast majority (around 99 percent) of those suspect names were legitimate Americans, not illegal immigrants itching to get deported by showing up at the voting booth.

Of course, liberals have their own critiques of poll methodology, many of which I – naturally – find more convincing than the fallacies trotted out by the right. They are certainly arguments made by a higher number of experts, as opposed to pundits whose only knowledge of statistics comes from twisting them to fit a thesis. While not every argument is as tinged with partisanship as the Huffington Post’s contention that Gallup undersamples minority voters, Nate Silver’s questions about the validity of polls (like Rasmussen) that rely on robo-calls (which aren’t allowed to dial cell phones) instead of live interviews or that fail to include cell phones in their samples are reasonable concerns at a time in which approximately one-third of Americans don’t have or regularly use landlines. Such Americans, of course, are disproportionately minority and low-income, which suggests that pollsters who don’t include them are employing the same logic behind the Republican argument that voter ID requirements are “fair” because they only disenfranchise a small slice of the electorate . . . . a slice that happens to vote nearly 100% Democratic. (For what it’s worth, Rebecca Rosen makes an interesting, though neither particularly persuasive nor fully fleshed-out, counter-argument at The Atlantic that the cell phone gap is meaningless because most well-off people have landlines, and the well-off are much more likely to vote.) Indeed, the surveys that exclude cell phones routinely lowball Obama’s lead in comparison to the live-interview consensus. Given that robo-poller Rasmussen also weights its surveys by party identification, which liberals would argue creates a sample with a higher percentage of Republicans actually exist, Nate Cohn observes at The New Republic that “it’s not hard to envision how [robo-polls] could systemically underestimate Obama’s standing. In a certain respect, it’s surprising that Obama ever took a lead in Rasmussen’s tracker after the DNC, since it required Obama to hold an unrealistically large advantage among independent voters.”

Some of the complaints are flimsier, like the Obama campaign’s objection to a New York Times “panel-back” poll that re-surveyed people previously contacted for an earlier poll. “The usual complaint is that the respondents who agree to be reinterviewed are different from those who don’t,” political scientist John Sides writes at The Monkey Cage. Re-interviewees are obviously more willing to talk to pollsters, with all the free time and affability such willingness implies, and may have been more motivated to seek out information about politics in the interim. Why this is a disadvantage for Obama is not entirely clear; why, for example, would the opinions of such a sample, even if affected by the initial survey experience, necessarily be cooler toward the president?

The left also engages in its own version of data cherry-picking, delving into the crosstabs of polls to find hopeful signs buried below more neutral or negative top-line results. Professional spin doctors like top Chicago strategist Jim Messina downplay positive numbers to avoid a sense of complacency in campaign volunteers. “I think you will see a tightening in the national polls going forward,” he predicted on a conference call with supporters. “Ignore the polls . . . . None of that matters. What matters is your voter contacts in your state.” Liberal blogger Greg Sargent cites a National Journal survey that gives the president good top-line numbers (50-43) but focuses on details he finds particularly encouraging:

Crucially, though, Obama holds a commanding 57 percent to 34 percent advantage among those who say their finances are unchanged. One reason for that critical tilt in his direction: Voters who say their finances are unchanged also say, by a resounding 53 percent to 33 percent margin, that they believe the country has been better off over these past four years because Obama, rather than another candidate, won in 2008.

It wouldn’t, after all, be as convenient to quote from the National Journal’s own analysis of the poll, in which Ronald Brownstein offers the caveat that “the saving grace for Republicans is that even as these surveys show Obama opening a consistent advantage, the president has not been able to push his support much past the critical 50 percent level, even after several difficult weeks for Romney that began with a poorly reviewed GOP convention.”

Often, the same numbers produce opposite interpretations from the two parties. While Republicans offer rough parity in the daily tracking polls as a sign of success — Romney pollster Neil Newhouse’s famous “sugar high” memo touted “a margin-of-error race with an incumbent President” — liberals see the glass as half-empty for Romney. “Obama has pulled into a tie with Romney on the economy in the last eight national polls,” Sargent crows in another post. “That’s after Romney led on the issue in many polls for nearly a year.” The sheer number of polls, especially unreliable push-polls conducted on a state level by a candidate’s own campaign or by a partisan lobbying outfit (e.g. the Chamber of Commerce), also enables contrasting versions of reality. CNN reports “good news” for swing-state incumbents, writing that “Sen. Sherrod Brown of Ohio holds a seven-point lead over Ohio state Treasurer Josh Mandell, the GOP candidate, in two surveys over the past two weeks.” Meanwhile, over at National Review, we’re told that “Mandel is down four points, but it remains a competitive race.” Interestingly enough, while CNN only reports on numbers from reputable outfits like Quinnipac and the WSJ/NBC poll and often cites its sources, the National Review only offers the nebulous “down four points” formulation. Four points in what, the internal polls ordered by Mandel’s strategists and used (as all candidates use internal polling) to bolster one’s case among supporters? The best way to deal with such “data” is to brush it off, as the Post’s “Fix” blog does with a similar claim by liberals: “Democrats insist they have data that shows Arizona to be a low single-digit race, but we remain unconvinced Obama can win there after losing it in 2008.”

Because there are only so many ways to spin a single set of “for whom would you vote” numbers, and because the candidates are running roughly even in the daily tracking polls, both sides turn to the more specific (and often more touchy-feely: which candidate would you rather invite to dinner?) questions to “prove” their frontman’s superiority. Andrew Sullivan highlights the liberal focus on Romney’s unfavorables: “Has a major-party Presidential candidate ever had to focus so much energy on getting his own party to be willing to vote for him?” Indeed, Romney’s favorability ratings actually slipped five points in September, despite the unflagging attempts by everyone from Ann Romney to former Bain co-workers at the Republican National Convention to “humanize” him. “Romney ranks as the first challenger in memory to have higher unfavorable than favorable numbers this late in the race,” reports National Journal, providing gleeful Chicago strategists with an irresistible hook for the next morning’s e-mail blast. Strangely, the Republican rebuttal amounted to this: “The fact is that while Obama was driving Romney’s negatives up, Obama’s own negatives were going up.” It’s not clear which polls supposedly showed a dip in Obama’s approval ratings; over the last month, his numbers across multiple surveys, as collected at Pollster.com, have either increased or held steady in the low fifties.

While Republicans talk up Romney’s advantage on the economy and the deficit, Obama’s supporters point to the president’s massive lead in favorability and likability ratings, as well as the edge voters hand him on helping the middle class and understanding the problems of average people. Romney’s edge on the economy and job creation is in fact evaporating; the president outpolled him on this question for the first time in the latest NYT/CBS survey, leaving “handling the federal budget deficit” the only issue on which the Republican led. Foreign policy, Medicare, the middle class, even honesty and trustworthiness: on none of these questions did Romney garner the better ratings. Nevertheless, Boston insiders continue to insist that high unemployment and low consumer confidence will ultimately sink the incumbent, bragging that “voters already believe Romney has a better chance of fixing the economy. We have to tell them just how it will be better for them.”

Cataloging the various partisan attempts to pick apart or promote friendly poll results is a never-ending exercise. Analyzing the minutiae of the polls themselves could (and has) filled books. To come up with his 538 election forecast, Nate Silver runs 25,001 simulations each day. Most of this math is, needless to say, over my head — and over Romney’s and Obama’s as well. But it all boils down to who’s ahead. The New Yorker’s John Cassidy extracts perhaps the most relevant — and, for Romney, perhaps the most damning — nugget from all the noise:

According to the Real Clear Politics poll-of-polls, which averages out all the most recent surveys, Obama is leading by more than three points. Since the start of the year, Romney hasn’t led the poll-of-polls. He has only drawn level once—immediately after the Republican convention.

Of course, Romney could stage a comeback. (And if you believe Rasmussen and Gallup, he’s doing well enough to not even need a comeback.) There are still 40-odd days until the election, which is why there is such a gap between Silver’s November 6 forecast, which puts the likelihood of an Obama victory at 77.6 percent, and his “Now-cast” (if the election were held today), which favors Obama a whopping 95.6 percent of the time. None of the poll results or favorability ratings will matter one whit in November, when it comes down to a single figure: 270, the number of electoral votes required to win the White House.