Why surveys on vaccination coverage can be off the mark

Experimental method study by Chemnitz University of Technology and University of Konstanz suggests overestimation of vaccination coverage against Covid-19 in population surveys – “social desirability” biases survey results

With the colder weather, Corona cases in Germany are on the rise again in the Fall of 2021. Compared to the previous year, however, there is one important difference: a large proportion of the population has now received at least one dose of a Covid-19 vaccine. But how high the vaccination coverage actually is in Germany has been hotly debated for some time. Several population surveys on the vaccination status of the respective respondents have shown a significantly higher proportion of people vaccinated (at least once) than recorded by the official digital vaccination coverage monitoring of the Robert Koch Institute (RKI). In the public debate, this discrepancy was partly attributed to a significant under-reporting of actual vaccination figures by general practitioners and vaccination centers. However, there could also be another explanation: A recent, as yet unpublished study by researchers from Chemnitz University of Technology and University of Konstanz shows that a survey-based estimate of vaccination coverage is subject to bias. The study is part of a project funded by the German Research Foundation (DFG) at Johannes Gutenberg University Mainz and Chemnitz University of Technology.

“Social desirability” as an incentive to lie in surveys

“We can show that social desirability leads respondents to report vaccination against Covid-19 in survey studies even when it has not occurred at all”, says Prof. Dr. Jochen Mayerl of the Professorship of Sociology with Specialization in Empirical Social Research at Chemnitz University of Technology. “If we assume that vaccination against Covid-19 is considered normatively desirable, and that most adults in Germany perceive this norm – regardless of whether they personally consider the norm to be right or wrong – then there is an incentive for the unvaccinated to deviate from the truth in order to avoid disapproval by whomever”, adds Dr. Felix Wolter, research associate at the Professorship of Microsociology and member of the cluster of excellence “The Politics of Inequality” at the University of Konstanz.

Comparison of survey methods

That such response biases occur in surveys is a well-established finding in empirical social research. In the current study by sociologists at Chemnitz University of Technology and the University of Konstance, it has now been experimentally demonstrated that this also applies to the question of vaccination status against Covid-19. From September 10 to 20, 2021, the researchers surveyed a total of 7,530 people living in Germany between the ages of 18 and 70 who had Internet access. A portion of the sample was asked the direct question about having been vaccinated against coronavirus for the first time (“I have been vaccinated against corona virus at least once: Yes/No”). Another group of respondents was asked the same question using an special questioning technique designed to provide an anonymous setting in which to respond called the item count technique (ICT). Here, respondents do not answer the question on vaccination status directly, but together with other “more innocuous” questions. The analysis is thus conducted using aggregated data, and it is impossible to know whether a given respondent is vaccinated, or not. An estimate of the vaccination coverage in this sample was then determined using statistical methods.

Direct question about vaccination status leads to overestimation of vaccination coverage

For the age group between 18 and 70 years, a vaccination coverage of just under 85 percent was determined for the survey period in mid-September 2021 using the direct question (at least one dose of the vaccine). “Although this value is not comparable with the official data of the RKI – children, adolescents and the elderly were not part of the sample – it indicates a significant overestimation of the actual vaccination coverage”, says Mayerl. Because if you give respondents the opportunity to answer completely anonymously, the picture becomes much different: “Here, at 75 percent the estimated vaccination coverage is about ten percentage points lower compared to the direct question. The difference is statistically significant; the probability that the differences between the direct question and ICT arose purely by chance is extremely low”, adds Wolter. “Thus, our study underscores that conventional direct questions about vaccination status against Covid-19 overestimate the actual proportion of the population that is vaccinated”, Mayerl says. Moreover, the sociologists assume that the overestimation would probably have been even higher in a face-to-face or telephone survey where respondents interact directly with interviewers.

Further information is available from Prof. Dr. Jochen Mayerl, email jochen.mayerl@soziologie.tu-chemnitz.de, and Dr. Felix Wolter, e-mail felix.wolter@uni-konstanz.de. A detailed presentation of the results can be found under this link: https://osf.io/8kbnd/.

(Author: Mario Steinebach / Translation: Sai Teja Paila)

Matthias Fejes
18.11.2021

All "University News" articles