Many Americans were shocked at the results of the 2016 presidential election. This included President-elect Donald Trump. But perhaps the most befuddled Americans were the pollsters and data collectors who used their results to repeatedly say that Hilary Clinton was all but ensured to become the first female American president.

Americans are fascinated, and increasingly reliant, on the stories that poll results purport to tell. Unfortunately, the stories they tell can often be misleading, at best – or just fiction, at worst.

The first known poll dates back to the 1824 presidential election. That particular poll was conducted in “taverns, militia offices and at public meetings” and then published in the Harrisburg Pennsylvanian on 24 July 1824. The results showed Andrew Jackson leading John Quincy Adams in the race for president. With Jackson going on to win the election, this new methodology of taking polls became popular. 

By the 20th century, little had changed. That is until 1936 when George Gallup created modern polling. During that year’s presidential campaign, Gallup’s main competitor, industry leader The Literary Digest used a 2.3 million postcard campaign to predict the 1936 presidential contest between Republican Alf Landon and Democrat Franklin D. Roosevelt. Ignoring the Republican bias of this postcard polling, The Literary Digest predicted that Alf Landon was destined to be elected president in 1936.

Gallup selected a smaller but more demographically representative sample of citizens. As a result of this broader selection of people Gallup’s poll showed a different result. By 1937, both Alf Landon and The Literary Digest would become just a footnote in history. George Gallup’s methodology of surveying public opinion through broader and smaller samples became the industry norm as well as the industry leader for decades.

The ensuing years have seen an increasing reliance on polling results. Journalists, television news, businesses, politicians and special interest groups are just some that now rely on polling to shape their practices. Polls have shaped product marketing, news stories, movie and television narratives, elections from high schools to presidents and guide the legalisation of long illegal recreational drugs like marijuana.

The modern leader in polling today is considered to be the non-profit Pew Research Center. In 2012, Pew made themselves subject to a poll. What they found was that “fewer than one in 10 Americans contacted for a Pew survey bothered to respond.” 

A good question to ask is are polls therefore reliable gauges of the American consensus?

There are two primary types of polling, “scientific” and “unscientific.” Scientific polls can accurately reflect and describe public opinion (Gallup, Pew Research) whereas unscientific polls simply report what people reply (Buzzfeed, Facebook). This is not to say that polls on the internet can’t be scientific. However, it’s important to note that, according to Pew Research, only about 10% of Americans use the internet.

For example, if you consider current polling in the race for the Democratic presidential nomination. Polling aggregator Real Clear Politics provides a lengthy and robust look at various pollsters and their results (scientific). That site shows Joe Biden having a sizable if decreasing, lead. Conversely, if you look at a recent poll by news aggregator Daily Kos, it shows Elizabeth Warren with a commanding lead over all candidates (unscientific).

Currently, Biden’s lead sits at +/- 9% over the second-place Warren. That percentage difference accounts for the 10% internet usage difference. As time becomes more of a cherished resource, Americans have become more reliant on the information that polls provide. This could then have someone wondering which polls to believe. Without doing a cursory look at the data, and methodology inherent in the poll, they can be egregiously misleading.

But if you take a brief pause and look into the poll results, there are a series of questions worth considering. Just thinking about five basic questions can help you have a clearer understanding of the poll you are looking at:  

Who did the poll?

The easiest and most basic question. Was it an academic institution, a polling firm, political campaign or some other entity or person? Any poll worth examining will notate who did the poll. If that basic information isn’t transparent or is actively withheld, it’s safe to say it’s not a reliable poll. 

Who paid for the poll? 

Polls are conducted for very specific reasons. With that in mind, they’re also not free to execute. Knowing who paid for it can help tell you what these pollsters and polls are considering to be important. Businesses’ may take a poll to test a new marketing campaign, a special interest group may want to take the temperature of the public on a topic or a politician may want to better understand where they stand in the run-up to election day.

However, with both politicians or special interest polls, knowing who paid for the poll will help determine what the objective of the poll was. A political candidate or special interest group may frame questions in such a way as to bias the answers.

For example, a poll asking Americans to cut foreign aid spending from the budget may first result in a hair raising negative reaction because Americans believe foreign aid to be over 26% of the budget. A closer look reveals something different. According to Oxfam, foreign aid accounts for less than 1% of the American budget. In this hypothetical poll, if more accurate information was involved in the polling preamble, it may yield a different result. 

How many people were interviewed?

As George Gallup proved in 1936, more is not necessarily better with polling. A poll’s purpose is to provide estimated outcomes and if all things are equal, the more people interviewed the smaller the margin of error. Although, in America, and elsewhere in the world, things are never equal. Considering that, it’s safe to say that a smaller more diverse representation will furnish more accurate, or representative, data. 

How were the people chosen?

Using a random, or probability sample is the foundation of scientific polling. This foundation states that when provided the opportunity to select people in a target population it would then ensure the results would more properly reflect that population if randomly gathered.

In other words, selecting people at random will result in more representative information (presuming there is no bias in the phrasing of the questions). This is the reason that 1,600 American adults can reflect the opinions of  327 million Americans with only a limited margin of error.

What region and what group does it represent?

This is critical in political polls. In the lead up to big elections, where roughly 60% of Americans vote. In the less substantial primaries or pre-election polls, only about  25% show up to vote. Since it’s these less substantial polls that can influence a vote in the larger, more impactful, elections it’s imperative to know from what sample the poll is drawn from.

That is to say that polling results derived from only veterinarians in Texas will not, in any way, reflect the view of all Americans because not all Americans are veterinarians. To lessen any confusion, misrepresentation or fabrication, especially in political polls, look for wording like “registered voters”, “likely voters” or “among veterinarians” (or applicable group). Such distinctions are important to understanding all polling but are particularly critical for political polls.

For better or worse, over the past 195 years, polling has embedded itself into American life. As time becomes more valuable and attention spans become shorter, Americans are relying on polls to help them in their decision making. Pausing to consider basic questions can help better understand what exactly the information is showing.

Polling is no longer just informative, it has proven, culminating in the election of Donald Trump in 2016, that it can have profound sociological and political consequences. Bestowing one person, group or entity with such influence and power should give pause.

In and of themselves, polls are informative tools and at their very best, they are transparent with their data and would reflective of the population or constituency they are serving.

However, at their very worst, polls will be misunderstood and misinterpreted and their inherent margin of error could lead to catastrophic influence in many aspects of American life.