Why The Polls Are Too Few And Too Small

Aside from Morning Consult no pollster is consistently polling the race with sample sizes high enough to allow us to visualize the volatility of the primaries. The the case of the primaries where we have very few historical data points and limited polling among an extremely inconsistent electorate requires sample sizes in the 800-1000 range to handle margin of error.

We see this in a lot of the 538 B- to A+ rated pollsters. Monmouth and Quinnipiac are running sample sizes in the 300-500 range and the sub sample volatility is very high. In a field with a huge number of candidates this is a problem. Bernie went from 13% white and 7% non-white to 13% white and 21% non-white in only a month, moving his vote share from 11% to 16%. As few as 10 voters out of 100 could shift the numbers like that. It’s just not very reliable. And of course I haven’t even talked about the sample weights.

Several pollsters are massively over-weighting older people in the voting sample relative to the 2016 exit polls and which polls did well. This is especially true when you consider pollsters that are not universities or who have polls not commissioned by major media outlets. Now media commissioned polls are a bit too old also but its not comparable to groups like Tel Opinion Research or Optimous. These polls are heavily boosting Biden because of the age weighting. Conversely online polls and polls with younger samples like Emerson and Change Research are much less confident about Biden’s numbers.

Another important factor is that the polls are super inconsistent in timing and frequency. Poll averages will totally ignore this. Running daily or weekly averages makes no sense when you might only get one non-Morning Consult poll a week. We also see how the media will often spin. Reporting lately has Bernie falling in the polls but actually he is gaining. They are focusing on results from polls that did not poll right before or right after Biden launching. So they are playing catch up. Fox and Monmouth are good examples. But when we look at Quinnipiac or Morning Consult or even Harris we can see what’s happening.

How The Mainstream Media Misreports The Polls

Media content courtesy of: https://twitter.com/greg06897
Videos and images of events discussed in this section can be found here with corresponding IDs: https://electingberniesanders.net/media-inaccuracies-benefitting-biden/

The mainstream media consistently fails to report relevant data from polls besides the top lines. They will hunt around and search for the sub sample data that makes Bernie look bad and they’ll report different data from different polls to maintain the illusion. Many people believe the media does this on purpose but it is also possible they are simply compelled by their biases.

There are several documented cases of MSNBC reporting incorrect poll numbers on their graphics. Typically something like a + or – switched in the trend line or accidentally swapping a sub sample data point for the wrong one which happens to move Biden ahead of Bernie.

Examples:

  • The Washington Post published a graphic of a poll they commission with unprompted support for candidates. [54% No Preference|13% Biden|9% Sanders|5% Buttigieg|4% Harris|4% Warren|3% Beto] Their graphic had Biden’s bar at 31% and no one noticed.
  • MSNBC had a graphic on screen showing support from non-white voters for Bernie and Biden. The real result was 27% Bernie for the poll and 25% Biden. They somehow used Biden’s 28% from whites with no degree on the graphic and talked about him being ahead.
  • MSNBC, on Meet The Press, had a discussion of Bernie, Biden, and Warren support in 3 recent polls. They described Biden as gaining in 2 out of 3 and Bernie falling in all 3 while Warren grew. However they had switched a +5 in the Quinnipiac for Bernie to a -5.

In the 3rd example they also engaged in a bit of a shady practice. The only poll where Biden dropped, the same poll they reversed Bernie’s trend on, was the only poll that had had results both before and after Biden announcing. The other two polls had only had a poll prior to the announcement. This is indicative of the ignorant, or malicious if you want to go there, way in which the press discuss polls. There were several major polls which had polled before and after Biden announcing and had a third, recent, poll right before the show. These would have accurately reflected Biden’s current trend.

How The Media Cherry Picks Sub Sample Results To Hurt Bernie

Another major issue with media coverage, which you can explain by unconscious bias or conscious maliciousness is the way in which the media addresses sub-sample results. Both on Twitter and on TV the media will select the pro-Biden data to make Bernie look bad.

You can see this a lot from Nate Silver, Harry Enten, and others on Twitter. Let’s look at Quinnipiac since we know they screwed up on that one before. Bernie went from 15/8 Men/Women to 16/16. On White/Non-white he went from 13/7 to 13/21. He gained +5 in the poll. In the first poll these numbers got a lot of attention. What did they talk about in the second poll? How there was a question about how much attention you pay to the campaign and Biden was doing well among people who paid a lot and Bernie among those who paid little.

The pre-announcement poll in March was 29/19, post announcement 38/11, most recent 35/16. Warren went from 4 to 12 to 13. Beto went 12 to 5 to 2. Harris 8 to 8 to 8. Buttgieg went 4 to 10 to 5. They only talked about Warren. In fact this is another error they made. They said it was Warren’s first poll in double digits and she gained on Bernie. In fact she gained one but Bernie gained 5. She had led him 12 to 11 in the previous poll. Jennifer Rubin made the same error. Additionally Warren had broken double digits in the Emerson poll, which was actually Bernie’s best poll. 33/25/10. Harris also got 10. Warren broke 10 in Monmouth as well released a day later.

Bernie has trended upwards since Biden announced, when the last major round of polls was taken. However he was portrayed as losing ground to both Biden and Warren. On average Bernie actually gains in tandem with Warren, she doesn’t appear to damage his vote share.

Both on TV and on Twitter you will only tend to see sub sample results from whatever poll is the worst for Bernie.

The Polls Have Been Remarkably Stable Since March

Month Bernie Biden Margin
March 25.0%(+0.0%) 31.4%(+0.0%) -6.4%(+0.0%)
March 24.0%(-1.0%) 29.6%(-1.8%) -5.6%(+0.8%)
April 19.4%(-4.6%) 29.8%(+0.2%) -10.4%(-4.8%)
April 22.6%(+3.2%) 30.6%(+0.8%) -8.0%(+2.4%)
April 21.6%(-1.0%) 28.0%(-2.8%) -6.4%(+1.6%)

Bernie Has Been Climbing In The Polls Since The Week After Biden Announced

Month Bernie Biden Margin
Announcement 14.6%(-7.0%) 41.4%(+13.4%) -26.8%(-20.4%)
May 16.0%(+1.4%) 36.2%(-5.2%) -20.2%(+6.6%)
May 19.0%(+3.0%) 35.6%(-0.6%) -16.6%(+3.6%)
May 17.2%(-1.8%) 34.2%(-1.4%) -17.0%(-0.4%)

Bernie has actually been rising in the polls for the last 3 weeks. Biden peaked May 9th and after that began a fast slide. Biden is 6.2% above his lowest point and still on a downward trend. We’ll have a new Morning Consult in 4 days. CNN is due to come out soon. Harris puts out a constant stream of polls for different groups. They have like 4 streams of polls.

Bernie has gained 2.6% since the polls that came out right after Biden announced. Biden has fallen 7.2% since his peak. The margin has therefore closed 9.8%. Bernie is still 4.4% off his peak while Biden is 6.2% on his low. He has not lost support but consistently gained every week. Don’t expect to see this data in the mainstream media. It is relevant to note that the last two sets of 5 polls are from the same week and Emerson was in the first set.

Interpreting The Polls And Understanding Polling Trends And Sub-Samples

One significant thing I haven’t covered yet is that a lot of the ways the media, even traditional polling experts, cover the polls is a big mistake and a misunderstanding of things. There are several key things to know. Polls typically have some sort of house bias. You should be looking at the trends between polls by the same pollsters and not the average trend. Looking at sub samples is fun and sometimes useful but its tricky with small sample sizes. Let’s do details.

Most polls have some sort of built in sample bias. Basically the pollster is weighting for certain groups they believe they have some reason to consider more likely to vote. Even with my particular way of grabbing a poll average over time you have to deal with the fact that outlier polls can cause the average to swing quite a bit. Emerson Polling, Change Research, and Morning Consult are more favorable to Bernie, in that order. Quinnpiac, CNN, and Monmouth tend to be less favorable, in that order. McLaughlin, Harris, and Ipsos tend to give lower results for Biden and Sanders both. One specific stream of the Harris poll is very friendly to Biden. Morning Consult also tends to have higher numbers for both Sanders and Biden.

These house biases in the polls can cause what seem to be wild swings and unsophisticated media pundits often get caught up. An Emerson poll followed by a Quinnipiac poll would have a huge spread. This was especially the case around Biden announcing because Emerson came right before that and Quinnipiac right after. Bernie went from 29 to 11 if you looked at those two polls. Biden went from 24 to 38. That’s a 32 point swing. If you used the very Biden favoring Harris polling stream the swing would have been 40%, as Harris polled Biden at 46. However if you looked at Emerson before and Emerson after, while acknowledging Emerson was a week after the best Biden polls, Bernie dropped 4 and Biden went up 9. Similarly, Morning Consult had +10 and -5, as did CNN. Quinnipiac had +9 and -8, while Harris had a similar larger change. But none of the polls swing 32-40%.

Sub samples are often small and quite unreliable. This can even alter overall results. Bernie went from 13 and 7 to 13 and 21, a 5% top line swing, in Quinnipiac. Seems very unlikely his white support stayed the same and his non-white support went so wild. In the table below of gender and race sub samples from polls that provided them you can see that his drop was from 13 and 25 before that. That white support is implausibly consistent especially during Biden’s announcement when every other poll had significant differences. In contrast Emerson had it at 27 and 30 to 22 and 28. Their March poll was 25 and 29. His white support rose as he rose, then dropped as Biden announced. I expect it will go back up when Emerson polls again in a week or two.

Conclusion

Bernie was closing the gap right until Biden announced and he took a big hit after that happened. Biden is quickly losing that bounce and Bernie is regaining ground. The polls actually haven’t moved that much if you compare them to the results at the tail of 2018 except that Buttigieg is up and Beto has fallen. The truth is that aside from announcements not a lot has happened and its still early. Debates are a month out. Realistically things are just about to start getting interesting.

Now that we’ve established that Bernie is actually in decent shape I’m going to be writing a long series on the groups he needs to win big with and what he can do to get real movement among them. Those groups include liberals, progressives, leftists, young voters, women, working class whites, African American voters, Hispanic voters, and climate voters. Of course many of these groups overlap.

Extra Tables

Bernie Does The Best With Non-White Voters And He Is Almost Even By Gender

Pollster Date White Non-White Black Hispanic Asian Native Other
Harris C+ May 23-25 13% 18% 14% 21% 11% -% 28%
Emerson B+ May 10-13 22% 28% 35% 22% 19% -% -%
Harris C+ May 8-13 15% 28% 19% 5% 21% -% -%
Morning Consult B- April 21 – May 5 19% 24% 20% 27% 23%  30% 28%
Emerson B+ April 11-14 25% 33% 29% 38% 34% -% 28%
Emerson B+ March 17-18 25% 29% 35% 16% 45% -% 27%

National Race And Gender Vs Total March To May

Pollster Date Total Men Women White Non-White
Harris C+ May 23-25 15% 13% 16% 13% 18%
Change C+ May 16-20 22% 21% 14% 24% 19%
Monmouth A+ May 16-20 15% 20% 12% 15% 16%
Quinnipiac A- May 16-20 16% 16% 16% 13% 21%
Fox A May 11-14 17% 19% 15% 18% 14%
Emerson B+ May 10-13 25% 24% 25% 22% 28%
Harris C+ May 8-13 20% 20% 20% 15% 28%
McLaughlin C- May 7-11 19% 21% 17% 16% 24%
Quinnipiac A- April 26-29 11% 15% 8% 13% 7%
Morning Consult B- April 21 – May 5 21% -% -% 19% 24%
Monmouth A+ April 11-15 20% 22% 19% 15% 27%
Emerson B+ April 11-14 29% 25% 34% 27% 30%
Quinnipiac A- March 21-25 19% 22% 17% 13% 25%
Emerson B+ March 17-18 26% 23% 30% 25% 29%
Monmouth A+ March 1-4 25% 25% 23% 30% 21%

4 thoughts on “The Problem With Polling: What Polling Can And Can’t Tell You 8 Months Before The Election, Why Polls Are Too Few And Too Small, How Mainstream Media Abuses Them, And What It All Means For Bernie(He Is Actually Doing Pretty, Pretty, Pretty Good)

  1. A lot of good analysis. Thanks. I still say #pollsschmolls. I don’t want any more politicians who are guided by polls. It’s a losing strategy. They are misleading at best and manipulative at worst. It’s one of the numerous reason bernie has my vote. His positions aren’t driven by polls but rather personal convictions that have remained unchanged for 40 yrs whether popular or not.

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s