New Yorker Cartoon by Dabin Han
As has become routine practice in every election cycle, polling is once again a main topic of conversation in political coverage. In the wake of some high-profile public polling ahead of various Democratic primaries—much of which “missed”—pundits, journalists, and strategists are questioning the validity of polling and musing about how its purpose has mutated in response to new trends, demands, and financial incentives.
We are two of the principal researchers at Change Research. Stephen Clermont is the company’s Head of Polling and has over three decades of experience as a pollster and campaign strategist. Lauren Goldstein is a political psychologist and pollster and leads the company’s advocacy polling practice. We set out to co-write this piece because we’d like to pull back the curtain on some of the intricacies of how electoral research is conducted, as we answer for whether the criticisms of our industry are fair or off-target. In particular, we address some of the claims and arguments made in a recent NYT piece that was published last weekend.
Myth #1: Polls Are Supposed To Measure, Not Move
“Polls are traditionally used to measure public opinion, not mold it.”
“Every candidate in 2028 will have their own cooked poll factory,” said Lis Smith, a Democratic strategist who was a senior aide on Pete Buttigieg’s 2020 campaign for president. “Polls now are not just a way of measuring the horse race. It’s a way for candidates and their allies to influence the horse race.”
We characterize both of these assertions as off-base. Here’s why.
If your only purpose in doing a poll is to find out who is “winning” and “losing,” it’s not worth the effort or expense of running one. Polls have many purposes, and one of those has always been to mold public opinion. One way pollsters help clients meet this goal is through narrative research, where we typically conduct qualitative research along with rigorous message testing to determine the most effective communication strategy for a campaign to use in order to move the public in their direction.
The other is by publicly releasing data that can help drive momentum for a campaign. Sometimes that’s by showing a viable path to victory to drive donations and fundraising efforts, other times it’s by persuading other candidates to drop out of the race and coalesce behind the leading candidate. This is neither new nor nefarious, when the research is methodologically sound and the pollster is transparent about how they sampled voters and what questions were asked. This is a mainstay of electoral campaigns and has been for decades.
Some consumers of polls and political news will take issue with the idea of a poll serving any purpose other than a straight, down-the-middle measurement of public opinion. There are two problems with this all-or-nothing view of the utility of polling.
The candidate horserace is the least important question in any poll. While it is necessary to benchmark where a race currently stands, other questions such as name identification and favorability, which candidates voters would consider, which candidates people reject outright, strength of vote choice, and what would happen once voters gain equal information about all the candidates tell far more about who will ultimately prevail. That is the purpose of campaign polling, and all of that information is closely held by the campaign and is not figured in any of the polling aggregators or the broader polling dialogue.
There’s no such thing as “true” public opinion. Firstly, most voters aren’t ideologically coherent, they’re malleable. Trying to extract a “true” baseline is not a fruitful pursuit. We can assess what we call an “uninformed” baseline before we provide information, framing, or messaging. But that is one read on public opinion, it’s not more legitimate than a read with framing, context, and cues built in. Framing effects pervade all forms of political communication and polls are no exception, so it’s better to acknowledge that reality and explicitly test it than to treat polling as a purity test. More on this from Mind the Gap later this week.
Myth #2: It’s Become Too Expensive To Do Good Research
“It has become even more expensive and difficult to conduct high-quality public opinion surveys.”
“Ms. Stevens sent The Times a memo about her experience dealing with polling that did not reflect the reality of the race…In the interview, she attributed the polling miss to an industry that has largely moved away from expensive “live caller” phone interviews with voters. Instead, most contemporary polling is done online or through texting, mediums likely to include more young people even though the voting electorate skews older. (Live-caller polls can be seriously flawed, too, and it is generally considered best to analyze surveys from a range of pollsters using a mix of methods.)”
There are a few points to tackle here. We break them into three parts: the known upsides and downsides associated with different polling approaches, the importance of methodological transparency, and the extent to which successful, representative sampling boils down to something simpler than the commentary suggests.
The North Star in any survey research is giving the broadest possible audience, randomly selected, an equal opportunity to participate. Limiting a likely voter universe to people who have voted in X of the last comparable elections not only excludes people aging into the electorate, but it also fails to account for how turnout might look different than prior cycles depending on the current environment (for example, a historically unpopular president that may motivate infrequent voters to participate, a motivation advantage for Democrats, and indicators that Trump’s gains with Black and brown voters in 2024 were a transient phenomenon that won’t hold this year). We have seen record Democratic participation in primaries this year, which makes limiting the polling universe in this way even more fraught.
There are plenty of wrong ways to poll, but there is no one “right” method. One way to assess poll quality is to look at how much of their methodology is disclosed publicly. Live calling, text recruitment, online recruitment, online panels, and other methods have advantages and disadvantages in reaching targeted voters. It is on the researcher to detail the choices they made in recruitment, the processes they used to reach voters, and the tradeoffs made between using one research method over the other. Without that context, the results should be viewed even more skeptically.
The fact that certain methods are more or less successful at reaching certain groups is not an excuse for inaccurate or incomplete sampling. The fact that online and text-to-web recruitment methods historically skew towards younger poll participants1 is not an excuse for accepting imbalances in the final dataset. We are often asked by clients, “how do you ensure a proportional representation of underrepresented groups in your polling?” We give methodological answers about Change’s success with multi-modal recruitment methods and our data team’s constant iterating on how to improve our response rates. But an equally important answer to this question is simply this: we aren’t lazy about it. We don’t just upweight underrepresented groups to make up for the fact that they’re harder to reach. We make every effort to interview the correct proportions, including when it’s hard. We conduct tests on efficacy of different outreach messages, we allocate higher budgets to harder-to-reach demographics, we lengthen fielding time, we adjust our ratio of recruitment methods on a poll-by-poll basis depending on what’s proving most effective and with whom. When it comes to sampling methodology, one thing that distinguishes a good pollster from a shoddy pollster is actually attempting to poll a representative sample, not just taking whatever data comes in, applying large weights to move the proportions around, and manufacturing a representative dataset.
Myth #3: The Modern Polling Industrial Complex (!) Is Lawless
“...this year, several Democratic campaigns have seized on unreliable polls to push the perception that their candidates are surging to try to solidify support, attract donors and scare off rivals. The rise of prediction markets has also created financial incentives for untrustworthy or even fake political polls.”
“The modern polling industrial complex is a Wild West in which numbers often circulate online with little regard to whether they come from reliable sources.”
We’d characterize most of this as true, but there are two different dynamics at play to explain why. There are longstanding features of primary polling that pose challenges for even the best pollsters. At the same time, there is a new problem of bad actors. We take each separately.
Primary polling is challenging to do well, and correctly interpreting these polls requires a level of polling literacy that is often absent from simplistic horserace coverage.
First, every state has different rules on who can participate in a primary. States with no party registration, where every registered voter is eligible to vote in a Democratic primary, requires a broader outreach than states with closed primaries. It also makes sampling more challenging because it’s harder to determine what the demographic proportions of a likely electorate should look like. In closed primaries, the demographic composition of a primary electorate is less unpredictable. Notably, Michigan and Wisconsin are both states with open primaries.
Second, restricting your sampling to only include people with a past vote history in midterms means your poll will exclude all potential new voters in a high turnout environment. This was perhaps most evident in the 2018 Florida gubernatorial primary, when most pollsters polled from a list of previous primary voters and had candidates other than Andrew Gillum, the ultimate winner, ahead. Change Research took a broader approach to the race and found Gillum ahead in multiple surveys. Picking a new primary voter out of the electorate can be like picking a needle out of a haystack. The only way to account for that is to reach more people and ask them more questions to determine their likelihood to vote. There are few shortcuts.
Third, the economic principle of the “indifference curve” applies most strongly to primaries. An indifference curve shows all the combinations of two things that make you equally happy– you’re “indifferent” between any of them. Say you’re choosing between pizza slices and beers. Maybe 4 slices + 1 beer feels just as good to you as 2 slices + 3 beers, and also just as good as 1 slice + 6 beers. Plot those combinations on a graph and connect them: that’s one indifference curve. When choosing a primary candidate, voters balance between different things such as perceived “electability,” alignment with candidates on issues, candidates’ personal traits, and other factors. When nobody is strongly disliked, all the candidates land in a narrow band. This contributes to results varying so much from poll to poll, and gaps between polling results and electoral outcomes.
Fourth, intervening events between polls and elections can and do have an impact on results. Think of the 2008 New Hampshire Democratic primary where most polls had Barack Obama leading after winning Iowa 5 days earlier. Many voters liked both him and Hillary Clinton. After a shaky debate (“you are likeable enough, Hillary”) many voters moved to Hillary in the final hours. She won by 2 points. This is less about “polling error” or “polling misses” and more about voters adapting to new, sometimes inconsistent, information flows.
Finally, polling literacy plays a role in how people evaluate poll accuracy. When it comes to assessing polling accuracy, most coverage focuses on the margin (X candidate leads Y candidate by 15 points), but whether or not either candidate clears 50% pre-messaging is the actual litmus test by which polls should be evaluated. If a candidate is not polling over 50%, the campaign should not feel secure, and the campaign pollster is calculating all the ways the candidate could lose and making recommendations to inoculate against those weaknesses. That never gets seen or heard publicly.
Political campaigns and the polling world are also confronting a new problem: bad actors conducting shoddy polls in the name of “social experiments.”
New groups deliberately stoking disruption by introducing fake poll data into the political ecosystem is absolutely a problem. But this phenomenon is also an outlier in the world of polling that’s generating excess attention, and we should take pains not to conflate this with general polling methodological challenges. Media outlets, poll aggregators, and political news consumers can be part of the solution by building their readers’ poll literacy and demanding high transparency standards from pollsters.
Every pollster should disclose the following: who paid for the poll, how many people were interviewed, when the poll was conducted, where the sample for the poll came from and how voters were selected, what methods were used to interview voters, what limitations each method has, what questions were asked in what order, what weighting variables were used, what assumptions went into adjusting the poll to reflect the likely voter universe, and how much impact that had on the survey’s margin of error. Change Research is part of the Transparency Initiative of the American Association of Public Opinion Researchers (AAPOR). We urge every serious researcher in this field to join. It will make all of our work stronger and marginalize the bad actors.
Takeaways
Trying to use polls to perfectly predict results will only lead to disappointment. We understand the temptation to treat polls as harbingers of outcomes, but doing so only amplifies the misperception that polling is broken and flawed. Quality polls are more than a collection of numbers on a spreadsheet. They provide solutions to problems, such as, “how can I get people to pay attention to me so they will vote for me?” A reliable and worthwhile poll provides context around a race with methodological rigor and transparency. While horserace numbers provide excitement that people can take to the prediction markets, the meatiest information is usually elsewhere in the dataset.
This piece shares our personal views and lens on public opinion data and does not reflect any stances or positions of Change Research.
This is becoming less and less true over time. Older Americans are just as likely to answer online surveys as they are live caller surveys while reaching younger Americans on these methods has become harder and more expensive.
Very well said as always. Statistical tools are valuable in the hands of people who can honestly interpret their meaning...and dangerous in the hands of those who don't understand them.