Interview: The Power of Polling
The Hundred #20: July 19, 2022
Thank you for reading The Hundred, a newsletter in which experts provide analysis on questions that matter. Subscribe below to support the project.
G. Elliott Morris is a data-driven journalist at The Economist. He just wrote a book about polls. It’s called Strength in Numbers. Our questions are in bold, his answers in quotation marks.
At its most basic, how do opinion polls work?
“Pollsters like to use a metaphor about soup to talk about sampling, so I will too. It goes like this: When you’re tasting a pot of soup, you don’t have to eat the whole pot to know if it’s cooked through, seasoned right, or fully incorporated. You simply dip a spoon in — you sample the soup. If it is mixed properly, then the sample is representative of the larger pot. Polls work on a similar principle, backed by statistics, that if you talk to enough people in the right groups then you can roughly represent the views of the population as a whole. Now, that is a large “if” (as recent elections have shown), and pollsters often talk to samples that end up being unrepresentative of the population, but that’s the general idea.”
When did polling first start and what did it look like?
“Researchers have traced the first unscientific polls back to the 1824 presidential election. Back then, they were called “straw polls.” I say they were unscientific because they did not follow any proper methodological guidelines about sampling, interviewing, question-wording or what-have-you. Instead, a random person would report unofficial tallies from military roll-calls, July 4th parades, and large parties back to newspaper reporters and editors. Those tallies would get reported out in broader coverage about the race — not usually as predictions, for example, but as components to broader pieces endorsing candidate X or talking about the geographic support of candidate Y.
Straw polls got more sophisticated and comprehensive over the years. But that did not make them scientific or accurate. In the book, for example, I tell the story of how a straw poll conducted by the Literary Digest in the 1936 election received responses from over 2 million Americans — and it still underestimated Franklin Roosevelt’s margin of victory over Alf Landon, the Republican nominee for that election, by 38 percentage points.
That was the same year that modern, scientific polls were developed. A 35-year-old Iowa farm boy-turned-market researcher by the name of George Gallup had started up the first polling company so he could sell the insights to newspaper syndicates in a weekly column called “America Speaks!”. And he bet big against the Digest poll, saying he could come up with a better prediction of the election with fewer respondents. More than that, he said he could predict exactly what the Digest poll would say! And he was right. On election day, Gallup’s poll showed Roosevelt winning by 12 points. He won by 24. That’s still a large error, especially by modern standards, but not the catastrophic error that the Digest made.”
What differentiates a good poll from a bad poll?
“One thing that I want people to take away from the book is that there’s no secret formula for a good poll. We really have to be discerning of differences in quality along multiple metrics. But there are a few good heuristics for deciding whether to trust a number or take a deeper look at the poll’s methods. First, does the poll release its methodology? Preferably in a PDF, stating how respondents were reached (phone v online v something else), how many were interviewed, whether the sample was weighted, and how (including partisan variables, such as party affiliation or past vote, may be necessary for today’s non-response environment), and including the precise wording questions that were asked. If those questions are biased, maybe don’t trust the poll.
At the end of the day, this is not easy, though. This really is a prescription for learning how polls work, and how to do a good one. In a way, I want everyone reporting on polls to engage the number as a science reporter would. Ask: what process generated these data? Do I agree with that process, or are there steps along the way that I think were conducted suboptimally?”
What are some of the main reasons why even good polls get things wrong?
“Let’s talk about soup again. Even good polls can get an unrepresentative sample of voters just by random chance. But I also like to emphasize that people are not like tomato soup; they are not homogenous. Pollsters today have to get representative samples not just of Americans on average but among each important demographic group: whites and non-whites; non-college-educated and college-educated; rural and urban; etc. In that way, the soup metaphor works better for minestrone: if the sample spoon has too many beans or tomatoes in it, it is not going to taste like the pot as a whole. So in, say, 2016 and 2020, when too few Republicans are answering the phone, but pollsters aren’t weighting on that — aren’t making sure their samples are balanced by demographics as well as by party — they end up with biased samples.”
How does polling influence elected leaders?
“The stories I tell in the book are really fascinating on this. I don’t want to give everything away (buy the book!) but I will give you a fun anecdote and a summary.
In one example, political scientists fielded a survey on what voters wanted the New Mexico legislature to do with a budget surplus in 2008: spend it for economic relief caused by the recession or put it in their rainy-day fund. The researchers sent a randomly selected half of legislators the polling results and didn’t mail the other half anything. They found that the legislators receiving their district-specific survey results were much more likely to vote in line with constituent opinion than those who did not.
Most examples, however, show that polls play a less direct role in the process. Broadly, what happens is that leaders’ political advisors use polls to show them the areas of their policy agendas that are most popular. Those leaders typically respond by emphasizing their most popular stances in speeches and memos — and they underplay their least popular ones. So you end up with stories of how, for example, John F Kennedy switched to talking more about Medicare after his pollsters found that Medicare was getting more popular; and how Franklin Roosevelt’s election forecaster convinced the president to tone down what was an all-out rhetorical assault on big business and the Supreme Court in 1935 and worked with Congress to save some aspects of his New Deal.”
In what ways can polling have a negative impact on democracies?
“Well, bad actors are going to use any tool at their disposal for bad things. That is certainly true for polls. You can find plenty of examples of corporate interests using polls in press releases, for example, that may or may not influence how legislators vote and leaders talk. So my advice here is for reporters to be careful with the polls they are presented with. Ask those rigorous methodological questions and think through the data-generating process for each number you’re shown. And, honestly, if you get sent a surprising poll from a very partisan advocacy organization or a corporate interest group, be skeptical about it.”
How did the internet and accompanying technological advances change polling?
“The internet caused what I call in the book a real revolution in polling. Polls fielded online could be done pretty cheaply, fielded quickly, and reach a seemingly unlimited number of people. But they also came with real methodological challenges. How do you make sure the people you talk to online are representative? Well, pollsters figured out you could match up lists of respondents to data from the census and select people to interview who were representative of the broader population along multiple demographic attributes. That way, the people they selected from a seemingly unrepresentative panel could be made representative.
And that had two effects. First, it increased the number of polls that get conducted. That helps people, like me, who average polls together to tell stories about people and predict elections. And it increased the number of subjects that pollsters ask about. But it also increased the methodological rigor of all polls. The first good internet polls were developed by statisticians and political methodologists, meaning they were very smart in how they processed data. That has spilled over to the ways other polls are weighted and corrected for certain biases, etc.”
What do people often get wrong about polls?
“One big thing I want people to learn is that polls are uncertain — or “noisy” — measurements in isolation, produced by a scientific process that can go wrong at many steps and in many unpredictable ways. That means you have to think about surveys not as measures of the ground truth, but as estimates — with margins of error and all the assorted caveats about whether they are “right” or “wrong.”
A related idea is that aggregates of polls are also prone to more uniform bias today than they used to be. That’s because there’s no concrete solution to partisan nonresponse, which can affect all polls simultaneously, like there is for demographic nonresponse. And it’s this combination of individual noise and systematic bias that makes the future accuracy of polls hard to predict. So people should think about polls like a statistician, not a political pundit.”
How is polling likely to change in the future?
“I think pollsters are going to face increasing pressures to address partisan nonresponse. That will manifest itself in two ways. First, pollsters are going to spend a lot of time and money trying to develop new methods for surveying that have much higher response rates. So you have the Pew Research Center, for example, sending out surveys by mail to get response rates closer to 25-30%, instead of the 1-5% they get over the phone. And this turns out to reach the lower-response-rate voters, namely very conservative Republicans and religious Christians, at much higher rates than the other modes of surveying. At the same time, pollsters are developing tools to field surveys over text message, which is better at reaching younger people and maybe minorities. And some pollsters are using very sophisticated methods, adopted from the 2012 Obama campaign, to reach quotas of Americans that are representative of previous election results. So there is a lot of R&D going into how you iterate on recent ways surveys were designed — the decisions you make about the survey before you field it.
But methodologies are also shifting for how pollsters process their data after the fact. They are developing new ways of weighting polls that are less subject to noise. Some outlets are collaborating on high-response-rate surveys to provide partisan weighting benchmarks for the entire industry. And others are experimenting with very complex statistical models to help identify likely voters. These advances are thanks in large part to advances in computing over the last decade or so.
All this means that the future of surveys is likely to get a lot more complex. That makes it all the more important to think about how each individual survey is generated. Some surveys are going to use methods that are better than others, and you want to put more weight on those better surveys.”
What is a question you wish you were asked and what is your answer to it?
“Nobody has asked me for an example of a fun poll! And I have so many answers to this question. In 1951, Gallup asked adults “In summertime, do you think men should be required to wear coats in hotels and restaurants if women are present?” Sixty-eight percent of adults said yes and 27% said no.
Now, there’s nothing really important about that poll except in saying that there are tens of thousands of topics that pollsters have asked people about over the years. One reason I wrote the book was to get people thinking about how they could use the polls for good. One way to do that is to think about how to ask questions that are important to the average person. Polls are, after all, just another way of talking to the people.”
That’s it for The Hundred #20. Please share this post with friends and colleagues if you found it interesting. Subscribe below to support the project.