Assuring the quality of survey data: Incentives, detection and documentation of deviant behavior
Abstract
Research data are fragile and subject to classical measurement error as well as to the risk of manipulation. This also applies to survey data which might be affected by deviant behavior at different stages of the data collection process. Assuring data quality requires focusing on the incentives to which all actors in the process are exposed. Relevant actors and some specific incentives are presented. The role of data based methods for detection of deviant behavior is highlighted as well as limitations when actors are aware of them. Conclusions are drawn on how settings can be improved to provide positive incentives. Furthermore, it is stressed that a proper documentation of data quality issues in survey data is required both in order to increase trust in the data eventually used for analysis and to provide input for the development of new methods for detection of deviant behavior.
References
[1] | Akerlof G.A., The market for ``lemons'': Quality uncertainty and the market mechanism, The Quarterly Journal of Economics 84: ((1970) ), 488-500. |
[2] | Benford F., The law of anomalous numbers, Proceedings of the American Philosophical Society 78: (1) ((1938) ), 551-572. |
[3] | Birnbaum B., , Borriello G., and Flaxman A.D., , DeRenzi Brian, and Karlin A.R., Using behavioral data to identify interviewer fabrication in surveys, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '13. ACM. New York, NY, USA, (2013) , pp. 2911-2920. |
[4] | Blasius J., and Thiessen V., Detecting poorly conducted interviews. In: Interviewers' Deviations in Surveys - Impact, Reasons, Detection and Prevention, P. Winker, N. Menold and R. Porst, eds, (2013) , pp. 67-88. Peter Lang. Frankfurt. |
[5] | Bredl S., , Storfinger N., and Menold N., A literature review of methods to detect fabricated survey data. In: Interviewers' Deviations in Surveys - Impact, Reasons, Detection and Prevention, P. Winker, N. Menold and R. Porst, eds, (2013) , pp. 3-24. Peter Lang. Frankfurt a. M. |
[6] | Bredl S., , Winker P., and Kötschau K., A statistical approach to detect interviewer falsification of survey data, Survey Methodology 38: (1) ((2012) ), 1-10. |
[7] | Cantwell P.J., , Bushery J.M., and Biemer P.P., Toward a quality improvement system for field interviews: Putting content reinterview into perspective. In: Proceedings of the American Statistical Association (Survey Research Methods Section), (1992) , pp. 74-83. |
[8] | Crespi L.P., The cheater problem in polling, The Public Opinion Quarterly 9: (4) ((1945) ), 431-445. |
[9] | De Haas S., and Winker P., Identification of partial falsifications in survey data, Statistical Journal of the IAOS 30: ((2014) ), 271-281. |
[10] | De Haas S., and Winker P., Detecting fraudulent interviewers by improved clustering methods - the case of falsifications of answers to parts of a questionnaire, Journal of Official Statistics p. forthcoming, (2016) . |
[11] | Durtschi C., , Hillison W., and Pacini C., The effective use of Benford's law to assist in detecting fraud in accounting data, Journal of Forensic Accounting 5: ((2004) ), 17-34. |
[12] | Duvendack M., , Palmer-Jones R.W., and Reed W.R., Replications in economics: A progress report, Econ Journal Watch 12: ((2015) ), 164-191. |
[13] | European Commission ((2010) ). Report on Greek government deficit and debt statistics. Technical report. European Commission. Brussels. |
[14] | Fanelli D., How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data, PLoS ONE 4: ((2009) ), e5738. |
[15] | Freedman S.M., and Zhe Jin G., Learning by doing with asymmetric information: Evidence from prosper.com. Working Paper 16855. National Bureau of Economic Research, (2011) . |
[16] | Gwartney P.A., Mischief versus mistakes: Motivating interviewers to not deviate, in: Interviewers' Deviations in Surveys - Impact, Reasons, Detection and Prevention, P. Winker, N. Menold and R. Porst, eds, (2013) , pp. 195-215. Peter Lang. Frankfurt. |
[17] | Hauck M., Is survey postcard verification effective? Public Opinion Quarterly 33: (1) ((1969) ), 117-120. |
[18] | Hood C.C., and Bushery M., Getting more bang from the reinterviewer buck: Identifying `at risk' interviewers. In: Proceedings of the American Statistical Association (Survey Research Methods Section), (1997) , pp. 820-824. |
[19] | Hornbeck S., , Peng D., , Studner C., and Jodice D., Ensuring data quality in conflict zones. Technical report. D3 Systems, (2010) . |
[20] | Judge G., and Schechter L., Detecting problems in survey data using Benford's law, Journal of Human Resources 44: (1) ((2009) ), 1-24. |
[21] | Kennickell A., Curbstoning and culture, Statistical Journal of the IAOS 31: ((2015) ), 237-240. |
[22] | Koczela S., Discussion of Identification of partial falsification in survey data, Statistical Journal of the IAOS 30: ((2014) ), 283-284. |
[23] | Kuriakose N., and Robbins M., Don't get duped: Fraud through duplication in public opinion surveys, Statistical Journal of the IAOS p. forthcoming, ((2016) ). |
[24] | Lawrence C., and Love E., Characteristics of falsified interviews. In: Proceedings of the American Statistical Association (Survey Research Methods Section), (2010) , pp. 4771-4780. |
[25] | Massing N., , Ackermann D., , Martin S., , Zabal A., and Rammstedt B., Controling interviewers' work in PIAAC - the programme for the international assessment of adult competencies, in: Interviewers' Deviations in Surveys - Impact, Reasons, Detection and Prevention, P. Winker, N. Menold and R. Porst, eds, (2013) , pp. 117-130. Peter Lang. Frankfurt. |
[26] | McNutt M., Editorial retraction. Science, (2015) . |
[27] | Menold N., , Winker P., , Storfinger N., and Kemper C.J., A method for ex-post identification of falsifications in survey data, in: Interviewers' Deviations in Surveys - Impact, Reasons, Detection and Prevention, P. Winker, N. Menold and R. Porst, eds, (2013) , pp. 25-47. Peter Lang. Frankfurt. |
[28] | Michalski T., and Stoltz G., Do countries falsify economic data strategically? Some Evidence That They Might 95: ((2013) ), 591-616. |
[29] | Nigrini M., Benford's Law: Applications for Forensic Accounting, Auditing, and Fraud Detection. Wiley. Hoboken, NJ, (2012) . |
[30] | Schräpler J.-P., and Wagner G.G., Characteristics and impact of faked interviews in surveys - an analysis of genuine fakes in the raw data of SOEP, Allgemeines Statistisches Archive 89: (1) ((2005) ), 7-20. |
[31] | Simmons K., , Mercer A., , Schwarzer S., and Kennedy C., Evaluating a new proposal for detecting data falsification in surveys. Technical report. PewResearchCenter. Washington, DC. last accessed: 4/28/2016, (2016) . |
[32] | Storfinger N., and Winker P., Assessing the performance of clustering methods in falsification using bootstrap, in: Interviewers' Deviations in Surveys - Impact, Reasons, Detection and Prevention, P. Winker, N. Menold and R. Porst, eds, (2013) , pp. 49-65. Peter Lang. Frankfurt a. M. |
[33] | Winker P., , Kruse K.-W., , Menold N., and Landrock U., Interviewer effects in real and falsified interviews - results from a large scale experiment, Statistical Journal of the IAOS 31: ((2015) ), 423-434. |
[34] | Yan J., , Yu W., and Zhao J.L., How signaling and search costs affect information asymmetry in p2p lending: The economics of big data. Financial Innovation 1: (1) ((2015) ), 1-11. |
