Response Rate — An Indicator of the Research Quality or a Methodological Problem? How We Estimated the Response Rate in the Mixed Survey

Abstract

Detailed and accurate information on the refusal rate in sociological surveys, as well as the response rate, are considered the key to scientific research and are of particular importance for collecting high-quality data. The article presents the results of an experiment to calculate the response rate based on the materials of the current collaborative survey «Open Opinion — St. Petersburg. The Image of the Ideal Governor», conducted in May—July 2019. The article focuses on the methodological and technical difficulties of collecting, coding and interpreting paradata in a mixed survey (with separate subsamples of telephone (CATI) and street (face-to -face) surveys). The standard methodology of the American Association for Public Opinion Research (AAPOR) was used to assess the effectiveness of the surveys, as well as to analyze their failures. The author describes the main problems that arise during the assessment of the response rate and paradata quality in a mixed survey with disposition codes recommended by the AAPOR. The study shows that paradata registered during the survey contain many omissions and inaccuracies, as well as errors in coding refusals, which lead to bias in estimations of the response rate and effectiveness of the survey. The main problem is the paradata quality. The information presented in the article is primarily intended to draw the attention of researchers to this problem and its consequences. The author proposes possible solutions, which range from improving the qualifications of interviewers, developing the process of collecting the data accompanying the survey, and introducing an automatic system for recording paradata, to developing a standardized, well-documented and reproducible procedure for collecting paradata, adapted to different methods of data collection and Russian context.

Acknowledgements. The author expresses gratitude to the Open Opinion Project for the open data sets and materials of the «Open Opinion — St. Petersburg. The Image of the Ideal Governor» study, which allowed to examine the «objectivity» and present the results of the study, focusing on the difficulties that the researchers had to face together with unresolved methodological problems. Also, many thanks to my colleagues Maria Matskevich, Igor Zadorin, Vladimir Zvonovsky, Roman Kuznetsov, Yulia Baskakova, and Nadezhda Korytnikova for the advice and constructive criticism they provided in the process of working on research materials.