A Snowy Graveyard For Pols And Polls

By Richard Morin and Claudia Deane
Washington Post Staff Writers
Monday, January 26, 2004; Page C01

Now I know why they call this the Granite State. It's so hard to crack.

-- Bob Dole

America's beleaguered pollsters know exactly how Dole felt in 1996 after losing a second New Hampshire primary. This state has proven to be as chilly and inhospitable to survey researchers as it has been to a legion of presidential wannabes.

Bad methodology, bad timing and simple bad luck have conspired to produce some of the most memorable miscues in the error-filled annals of media polling. And with no fewer than seven separate survey organizations blanketing the state with numbers, it's a good bet that tomorrow's primary will add to New Hampshire's reputation as the graveyard of pollsters. Consider these recent flubs:

In 2000, the Chicago Sun-Times headline on an Associated Press day-before-the-primary story was "Nearing the N.H. finish line: Polls declare GOP dead heat." John McCain went on to beat George W. Bush by 18 percentage points.

The New Hampshire-based American Research Group's tracking poll ended up buried deepest in the snowbank: They had Bush winning by two points the day before the primary -- merely 20 off the mark. On the Democratic side, the pollster furthest from the mark at least got the winner right: The Quinnipiac University poll predicted Al Gore would win by 17 percentage points, but he actually won by four.

It was the second debacle for American Research Group (ARG) in as many New Hampshire Republican primaries. The day before the 1996 contest, ARG's Dick Bennett told the Manchester Union Leader, "It looks like Dole's going to win," based on the Kansas senator's seven-point advantage in his organization's tracking poll. He lost to Pat Buchanan by a single point.

Exit pollsters aren't immune. In 1992, Voter Research and Surveys' exit poll showed George H.W. Bush beating Buchanan by a relatively narrow six points, only to have Bush finish 16 points ahead on election night. "I've never done anything that bad," lamented Warren Mitofsky, then head of VRS and co-director of the current news media exit poll consortium.

In 1988, it was the Gallup poll that fell victim to the New Hampshire curse. Gallup's final preelection survey had Dole up by eight points. He ended up losing to Bush senior by nine points.

Gallup "went on to call the rest of the primaries that season without a hitch. . . . Haven't had many questions about how we succeeded in so many of those subsequent races, however," former Gallup head Andrew Kohut, now with the Pew Research Center, reminisced the other day. "Moral of the story: On my tombstone it will say, 'Here lies Andy Kohut -- got NH wrong in '88.' " (The Washington Post had nothing to boast about here either, with its final product suggesting Dole up by three.)

So is New Hampshire cursed?

Not necessarily. Tracking polls such as the ones currently being done in New Hampshire are exceedingly difficult to do. Few people bother to turn out in primaries, making it hard for pollsters to reach people who truly will vote on primary day. The rules in the Granite State are particularly quirky -- voters can register and vote on the same day. They also may list their partisan affiliation as "undeclared," further complicating the task of identifying Democratic or Republican primary voters.

But that's only part of the story -- the part pollsters will tell you. What they don't share is this dirty little secret: Too many of the most widely reported preelection polls cut corners, take big risks and use methods that are less than gold standard.

Perhaps the best-known of the bunch, Zogby International, does all kinds of controversial things to produce its headline-grabbing tracking polls. John Zogby calls only people with listed telephone numbers, missing those who are unlisted. About 30 percent of the people in his samples were called during the day -- a good time to reach retirees and housewives but a bad time to reach most working people.

"He is more a salesman and a self-promoter than a pollster," Mitofsky said. "He has made lots of mistakes on election outcomes -- five in 2002. . . . I have heard of volatile campaigns, but he has volatile polls."

Zogby acknowledged on his Web site last February that "this past election cycle was not my finest hour." He also noted that he got the winner right in 12 of the 17 races in which he did publicly released polls in 2002, and tweaked his methodology as the result of a post-election internal review.

He also recalled with a chuckle during an interview how he was vilified in 1996 for his practice of adjusting the percentage of Republicans, Democrats and independents in his samples to match the partisan makeup of the electorate, as reported in previous exit polls -- a practice since adopted by many media polling organizations, including The Washington Post and ABC News in some of their election polls.

"I know I do some things different than others," Zogby said. "I know the so-called 'Poll-ice' would deny it, but there's art as well as science involved in this."

Before the Iowa caucuses, a brief item slamming Zogby appeared in the Note, the daily online briefing prepared by the ABC political unit. "The svengalis at ABC News and some major papers don't like Zogby's tracking," it said, going on to quote an anonymous "ABC guru" who called Zogby's tracking polls "crack for the weak."

"I saw that," said Zogby, laughing, adding he was not surprised. ABC's polling department has reviewed his methodology and rated his polls as "not airworthy," he said. "But that's all right. We're doing okay without ABC," noting his deals with NBC, MSNBC and Reuters, among others. "I'm a humble guy from Utica just plying my trade. I'm just not a member of the club."

Other widely reported New Hampshire polls have problems of their own. Surveys by Franklin Pierce College, which released a widely reported Democratic preference poll on Friday, uses samples based on lists of registered voters that have proved to be incomplete, outdated or both. Suffolk University, which is polling for a Boston television station, asks a curiously convoluted candidate-preference question that ends: "toward whom would you vote or lean?" Franklin Pierce, Suffolk and the University of New Hampshire all use student interviewers, who tend to be inexperienced and need strong supervision.

Polling directors at the schools insist that the kids are all right: "Their quality is tremendous," said Richard Killion, who oversees the Franklin Pierce polls, later adding: "It really improved when I started paying them."

Despite a plethora of problematic methodologies, the track record of polls in New Hampshire may be no worse than elsewhere -- slim comfort to pollsters who see their standing with the public sink with each bad poll.

"I think you probably would find this kind of thing in other primary states, but New Hampshire is much more visible," said University of New Hampshire pollster Andrew Smith. "But New Hampshire does have some quirks that, if a pollster doesn't pay attention, they're going to be wrong."