Study reveals how language impacts survey results

When it comes to conducting surveys, clear questions and response options are important.

But a new study in the Wall Street Journal reveals that it’s not that simple.

The study compared four recent COVID-19 vaccine surveys conducted in the US. Their results provide some troubling conclusions for pharma companies, not to mention anyone who considers survey results a reliable source of information.

Here’s what’s so worrying.

For survey conductors

The study found that the more answer options respondents have, the more their responses will vary. This makes sense, but it also makes for a few problems.

A Gallup poll included in the study only allowed respondents to answer “Yes” or “No” to the question “If an FDA-approved vaccine to prevent coronavirus/COVID19 was available right now and at no cost, would you agree to be vaccinated?” 63% of participants answered “Yes”, while 37% answered “No.”

But the three other surveys offered up to five different answer options, making results less straightforward and reducing the percentage of purely positive responses. These results provide more in-depth information than a “yes” or “no” survey, but the variations also make it difficult to compare answers across surveys or to give a quick, general idea of public opinion.

For those who rely on surveys for information

For many of us, surveys seem like an easy way to find a truthful answer in the midst of sensationalized stories and clickbait headlines. But this study makes it clear how dramatically statistics can change when respondents are given more survey answer options or when questions are slightly altered.

Due to their different wording and answer options, affirmative responses about getting the COVID-19 vaccine “ranged from 20 to 63%” across the different surveys.

Another troubling thought that will undoubtably arise after reading the article is that surveys can be manipulated. As Civis Analytics’ lead data scientist Franklin Marsh observes, “The number and type of responses has a strong effect on results….Giving people a ‘not sure’ option decreases the amount that say yes.”

This may make skeptics wonder if data they trusted was carefully constructed for particular results.

In addition to these concerning revelations, there’s another dimension to the study: the impact of language on respondents’ answer choices.

For instance, one survey was able to change 7% of responses simply by adding or omitting a particular detail -- which leads to another bit of surprising information. Civis Analytics discovered that if respondents were told that the COVID-19 vaccine is FDA-approved, 7% less of them said they would be willing to get vaccinated.

Crystal Son, Civis Analytics’ director of healthcare analytics, sees this as a reflection of a lack of trust in government institutions felt by many Americans.

Son observed another surprising phenomenon. Most respondents said that concerns about health risks and side effects were the biggest factors that kept them from wanting to get vaccinated, but survey questions that pushed the safety of the vaccine seemed to cause an increased negative response.

What did seem to make more people receptive to getting vaccinated, however, was sharing a story. When Civis Analytics’ survey conductors told the story of a healthy young American who died of COVID-19, the number of respondents who said they would likely get the vaccine increased by 5%.

In recent years, storytelling has been touted as an effective medical marketing technique. The strategy helps consumers relate to information in a more personal way, which is especially important when a treatment or product might be complex or difficult to understand or appreciate. Civis Analytics’ findings only seem to reinforce this idea.

There are several important takeaways from this COVID-19 vaccination survey study, from the outcome of offering more than two answer options, to what evokes positive and negative responses to getting vaccinated in 2021.

But the biggest takeaway may be that, as in so many things in life, language matters. Whether you’re an organization who’s planning to conduct a survey, or a reader who sees statistics and survey results as a reliable source of insight and information, the study reminds us that we have to consider what’s being asked and how respondents are allowed to answer.

Image source

Contact Our Writer – Alysa Salzberg