As results of the 2020 election began to pour in on election night and in the following days, a familiar cry went up across the country: Just as they had in 2016, the pollsters got it wrong!

But even as pollsters took heat for missing the mark in the presidential race nationally and in key battleground states, they came remarkably close to the final results in Colorado’s two top-ticket, statewide races, accurately pegging Joe Biden’s margin over Donald Trump and John Hickenlooper’s win over Cory Gardner in the state’s U.S. Senate race.

Colorado Politics turned to leading Colorado pollsters and national public opinion experts to determine how the national polls went so far astray while pollsters in Colorado got it mostly right.

The short answer, they said, is that after all the votes were counted, most national and battleground states polls weren’t that far off, but a confluence of unusual factors could have conspired to throw the more errant surveys out of whack.

Those factors include a growing split between voters who trust mainstream political institutions – like polling – and those who don’t, with the latter far less likely to take surveys while at the same time leaning almost exclusively toward Trump. It’s a recent development that threatens to undermine an assumption behind the whole notion of public opinion surveys, some warn.

In addition, possibly widening the gulf between Americans who talk to pollsters and those who don’t, Democrats and others opposed to Trump approached the election with sky-high enthusiasm, according to just about every measure.

Throw in some curveballs unique to 2020 – including so many states changing their voting procedures due to the pandemic and late-breaking news like Justice Ruth Bader Ginsberg’s death and Trump’s brief hospitalization with the coronavirus – and you’ve got a recipe for volatile public opinion that doesn’t always land the way the analysts who construct the modeling behind polls expect.

“I think there was a lot of the same stuff as happened in 2016,” said Ryan Winger, the director of Data Analysis and Campaign Strategy for Magellan Strategies, the Louisville-based Republican polling firm.

“What we saw was a lot of non-response bias,” he said. “You had Republicans who simply weren’t taking these surveys because, in the age of Trump, they’ve come to associate surveys with the mainstream media and people who are trying to make Republicans look bad. You have that critical mass of Trump supporters, and if you’re not picking them up in the polls, you’re not going to measure Trump support.”

It’s a relatively recent phenomenon, Winger and other pollsters said, tracing the emergence of that cohesive hidden group of voters to Trump’s arrival on the scene.

“There’s always someone you’re going to miss,” he said. “But what used to be the case was the people you were missing weren’t significantly different from the electorate as a whole, or not different enough to make a difference in your polls. Now they are.”

Who did (and didn’t) respond

Nationally, Biden beat Trump handily, winning both the popular vote and the Electoral College by comfortable margins. Biden won by nearly 7 million total votes, with his 306-232 electoral vote win matching Trump’s performance four years ago, when he declared he’d won an electoral “landslide” – but it was a lot closer than most polls had predicted.

While votes are still being counted in a few states that accept late-arriving mail ballots, through Dec. 2 Biden had just over 81 million votes to Trump’s roughly 74 million, for a lead of 4.4 percentage points, according to the Cook Political Report.

On Election Day, the rolling national average for presidential polls maintained by political data gurus at FiveThirtyEight pegged Biden with an 8.4 percentage point lead – nearly twice what the Democrat wound up getting.

What’s more, the polls were off by even wider margins down-ballot in most of the Senate battleground states, where Republicans held on to seats that had looked like probable Democratic pick-ups. In the most extreme case, there was a double-digit swing in Maine, where Republican Susan Collins kept her seat by 9 percentage points even though polls going into Election Day showed her Democratic challenger winning by a couple points.

Winger’s explanation is similar to a theory advanced by David Shor, a data analyst who worked for Barack Obama’s presidential campaigns and used to run Civis Analytics. In an interview with Vox, he explained how polling misses a certain brand of Trump supporter, no matter how pollsters adjust their screening and weight their samples.

“It used to be that once you control for age and race and gender and education, that people who trusted their neighbors basically voted the same as people who didn’t trust their neighbors,” Shor said. “But then, starting in 2016, suddenly that shifted.”

What he terms “low-trust people” still vote, but where they used to be fairly evenly spread across the political spectrum, they have gravitated to Trump, and while they still vote as much as the population at large, they don’t answer surveys.

“This gets to something that’s really scary about polling, which is that polling is fundamentally built on this assumption that people who answer surveys are the same as people who don’t, once you condition on enough things,” he said.

Winger said that measurable differences in response rates weren’t just due to some Republicans and conservative voters disdaining pollsters.

“On the flip side, Democrats and younger unaffiliated, Democrat-leaning voters were just itching to take these polls and talk about how much they hate Trump,” he said.

“That compounded the problem further because you couldn’t get these people off the phones. It’s a mix of those two things – real excitement on the Democratic side and not even hearing from the Trump people. It’s not that these Trump supporters weren’t going to vote – they were going to vote, they just weren’t excited to take surveys.”

A shifting electorate

He noted that some polling firms might have been led astray by the enthusiasm among Democrats.

“At Magellan, how we view it is we’re not going to tweak our turnout model or how we weigh it because one group is super excited and another group isn’t,” he said. “We’re making a call here in Colorado that unaffiliateds are going to punch above their weight. But it’s kind of baked in how we’re weighting these surveys based on party affiliation and age – we’re going to weight a survey based on historical data, not adjusting based on the response rates to a survey.”

Andrew Baumann, a Denver-based senior vice president at national polling firm Global Strategy Group, said it’s a constant challenge to keep up with an evolving electorate.

“If we just try to fix the mistakes of last time there’s going to be new problems,” he said. “In 2016, it was pretty clear the big problem was a lot of us weren’t weighting for education,” which meant surveys missed a lot of white voters who don’t have college degrees.

That demographic turned out to be a solid share of Trump’s vote, different from earlier election cycles when support for one party or the other’s presidential candidate hadn’t broken so cleanly along educational lines.

Baumann pointed out that polling in the 2018 midterms and in off-year elections in 2017 and 2019 went back to being typically accurate, with some misses but plenty of hits.

“Then in 2020 Trump was back on the ballot and things went kablooey,” he said.

That was the case nationally and in some states GSG polled, he said, though the firm’s September polling for the Colorado Senate race precisely pegged the final result.

“I think it’s a combination of likely a couple of things,” Baumann said. “One, that we corrected for the problem of not having enough white, non-college folks in 2016. You’re always making an assumption about what the electorate is going to look like – but in some states we still didn’t have enough white, non-college voters.”

Addressing another popular theory that’s floated around this year, Baumann said: “It’s not the ‘shy’ Trump voter – people weren’t lying to pollsters – it’s that they weren’t talking to us. Take two white, non-college independents outside Pittsburgh and everything about them is the same except one is a Biden voter and one is a Trump voter, you just get one who will take part in your survey, but you don’t know until after the election which one it was.”

How Colorado was different

In Colorado, polls that had consistently shown Democrats Biden and Hickenlooper with double-digit leads turned out to be nearly spot-on, down to Biden finishing a few more points ahead of Trump than Hickenlooper prevailed over Gardner.

Colorado’s voters gave Biden 55.4% of the vote to Trump’s 41.9%, a margin of 13.5 percentage points, according to the final, official results posted by the Colorado Secretary of State’s Office, just a point different than the 12.5-percentage-point lead Biden had on Election Day in FiveThirtyEight’s rolling average of Colorado polls.

Hickenlooper beat Gardner 53.5% to 44.2%, with a 9.3-percentage-point margin – within a point or two of nearly every publicly released poll conducted in the six weeks before the election, according to FiveThirtyEight.

It wasn’t just a handful that got it right in the Colorado Senate race, either. Pollsters that came close to hitting the mark during the run-up to the election include Global Strategy Group, Survey USA, Keating Research, YouGov, RMG Research, Morning Consult and Data for Progress.

As far as states where the polls missed the mark outside the margin of error – Maine and some of the Midwestern states that came down to the wire, though polls had shown Biden with solid leads – Baumann said, “Places that were further off tended to have higher levels of non-college white voters, and they tended to be a little more Trumpy. Places like Colorado have been moving more toward Democrats, but it’s more of an anti-Trump state than it is an anti-Republican state.”

He said GSG encountered the same thing the other pollsters noted.

“We saw our response rate go way up starting this spring, likely among Biden voters,” he said. “They were more likely to stay home, more likely to take COVID seriously and more likely to answer the phone.”

Chris Keating, the Democratic pollster behind Colorado-based Keating Research, shrugged off some of the simpler explanations for inaccurate polling this cycle.

“Our polling – not only our public polling that we released but also the polling I did for various clients on ballot questions and candidates – was extremely accurate in helping them and predicting what was going to happen in Colorado,” he said, adding, “There were a number of companies in Colorado that were pretty accurate.”

Know your voter

Keating conducted two Colorado polls in October in conjunction with OnSight Public Affairs and political strategist Mike Melanson – dubbed the KOM Colorado Poll – and both came within a point or two of the final results in the state’s presidential vote and the Senate outcome.

Said Keating: “I don’t think we have all the answers but one thing I’m doing in Colorado is when I try to estimate the electorate and the distribution of the electorate by party registration, it’s pretty much spot on, and that’s because I try to do the due diligence to get that part right. In some states it may be harder to do that if they don’t have party registration in the voter file, but we were able to be spot on with the results accurately representing who the voters were going to be here in Colorado.”

He said it came down to modeling Colorado’s turnout accurately.

“It was the combination of the 2016 voter with the higher turnout we were seeing in 2018,” he said.

Unlike in many states and the nation at large, however, Colorado’s polling wasn’t as subject to many of the influences that bedeviled national polling and polling in other states, pollsters said, with fewer of the hidden Trump voters and plenty of experience voting by mail.

The Hickenlooper-Gardner race, featuring a challenger as well-known to voters as the incumbent – unlike contests in nearly every other Senate battleground states – appears to have been mostly baked in all year, reducing the chances that news would sway the outcome.

On top of that, Colorado’s extremely high turnout rate – consistently ranking near the top in the nation – means it’s easier for pollsters to determine which voters are likely to vote, since pretty much all of them are, removing another variable that can throw polls elsewhere.

Curtis Hubbard, a Democratic strategist at OnSight, said he wasn’t surprised the KOM polls turned out to be accurate.

“Colorado is familiar territory and our election process was not changed in a significant way by the pandemic. For us, it’s obviously easier than pollsters in states where there’s suddenly an entirely different method of voting,” he said, pointing to states that had voted mostly in-person before this year.

Winger noted that it’s getting “harder and more expensive” to conduct polls, requiring pollsters to innovate methods of contacting potential respondents.

The why of polling

Pollsters say it’s important to understand what political polls do and how they’re used.

The head-to-head, so-called horserace polls that most people see are only a fraction of what most pollsters produce for their clients.

Moreover, while the media and the public appear to have an endless hunger for those kinds of polls, most of those surveys aren’t meant to predict an outcome but to provide a picture at the point the poll is conducted.

“The horserace is the snapshot – it’s the classic, we always say, ‘If the election were held today,’ but it never is held today,” Winger said with a chuckle. “it’s just a snapshot, and we fully expect things will change.”

Most of the polling public opinion firms do ask very different questions, he said, even in surveys conducted for candidates in the heat of a campaign.

“If they’re behind, the most critical part of a survey for us is trying to figure out the messages that resonate with voters and the issues that voters want addressed,” he said.

Hubbard, who ran several successful campaigns this year involving ballot measures, made a similar point.

“You can get great actionable information and understand what messages work well with what audiences and what don’t,” he said.

“Polling is not just to decide who’s going to win and who’s going to lose,” he added. “We asked voters if they intended to vote early, and we saw a huge spike in Democrats and Democrat-leaning unaffiliateds that they intended to vote earlier than in other elections, so we built that into our campaigns.”

Baumann agreed.

“The vast majority of the polling we do for clients is to understand the why of things and how you can change things and educate people and move voters on a race or an issue,” he said. “Having an accurate measure of a race or support for your policy is certainly important, but it’s certainly well less than half of what we are hired to do.”

As for the difficulties pollsters have encountered measuring Trump voters, Baumann said that might be something pollsters need to deal with for the foreseeable future.

“Theoretically, when Trump isn’t on the ballot, things might be better, but we can’t hang our hat on that. Even if he doesn’t run in 2024, there will be a Trump candidate – whether it’s Trump Jr. or Josh Hawley or Tom Cotton, you’ll have someone trying to capture that Trump mantle. And the disillusionment with institutions and willingness to take polls, that’s not going to change.”

What lies ahead

Winger said he hopes his firm’s clients keep in mind that the kind of polls saying which candidates are ahead and by how much are just one aspect of what the industry produces.

“Is there an over-abundance of horse-race polling? We might not need as much of that,” he said. “You hear a lot about the death of the polling industry. To me, what’s the alternative? Would you rather have campaigns and candidates out there going with their gut and not picking up on these movements and groundswells happening? Flying blind without knowing where you stand or what issues voters care about – you’re setting yourself up to fail.”

He also pushed back against the narrative spun by Trump and some of his most aggressive supporters, who suggested before the election that the polls were somehow rigged to favor Biden, supposedly to suppress turnout among Republicans.

“What these Trump supporters don’t understand, if you’re a pollster who’s out there trying to paint a rosier picture than exists, you’re not going to be around long because you’re not going to be right hardly ever,” Winger said.

Keating said he believes the public can judge pollsters by their track record.

“What I do with Curtis and Mike Melanson is, we try to release a poll every year to see if we’re accurate,” he said. “I want to put it out there, lay it on the line.”

Averaging the two statewide Colorado polls Keating conducted in October comes up with a Biden lead of 13.5 percentage points – precisely the final result, Hubbard pointed out after the election.

The KOM poll, he noted, has nailed the statewide results in Colorado for three presidential elections running, also correctly predicting Hillary Clinton’s 5-point win in 2016 and Obama’s 4-point win in 2012.

“In doing post-election analysis, it’s important to single out those pollsters who consistently get it right, and once again that list includes Chris Keating in Colorado,” Hubbard said in a statement. “That accuracy is owed to his ability to reliably model the electorate and our decades of experience with elections in Colorado.”

Keating said this year’s fuss could be misplaced.

“I don’t think when people say polling is wrong, I don’t think it’s polling,” he said. “I think it’s certain pollsters, or certain pollsters that are using certain methodologies that are doing it wrong; they’re not doing their due diligence.

“In 2016 some of the polling companies got it wrong. They said it’s because we had too many voters who were educated. Then everybody adjusted their model, and a lot of people still got it wrong. Now they’re saying there are these Trump voters who aren’t answering polls. To me, there’s always a combination of things.”

Still, Keating said, “It’s not just about trying to figure out who’s going to win a certain race. How do people feel on issues, what should we be doing on the environment or education? All those things – we still need to understand what people want. And we’ve got a big statewide election coming up in 2022.”