ARTICLE
ABSTRACT
In surveys, non-sampling errors, due to their complex nature, are more challenging to quantify compared to sampling errors. Avoiding the release of these errors, however, results in biased survey estimates. In our previous paper, we devised the best interviewer allocation technique by using a nested experimental design to study response error estimation. In this study, in order to illustrate the effectiveness of this methodology in a different context, we apply it in interview-reinterview surveys relating to the time use and life satisfaction of academicians at Middle East Technical University, Turkey. An analysis of the pilot survey data showed that only half of the data was reliable, while the other half revealed interviewer effects. Prior to the main survey, interviewers underwent training in the course of which particular emphasis was put on the above-mentioned questions. In effect, the previously observed response variances which accounted for the total variance and data unreliability, were reduced considerably, increasing the quality of the main survey.
KEYWORDS
correlated response error, interviewer allocation assignments, quality of survey research, reinterview procedure, sample survey design
REFERENCES
Ayhan, H. Ö., (2003). Models of response error components in supervised interview– reinterview surveys. Journal of Applied Statistics, 30(9), pp. 1047–1054.
Ayhan, H. Ö., (2012). Response reliability and response error estimation in personal interview surveys. AISC: International Conference on Advances in Interdisciplinary Statistics and Combinatorics, Greensboro, NC, October 5–7, 2012.
Ayhan, H. Ö., Işiksal, S., (2004). Memory recall errors in retrospective surveys: A reverse record check stidy. Quality and Quantity, 38, pp. 475–493.
Bailar, B. A., Dalenius, T., (1969). Estimating the response variance components of the U.S. Bureau of the Census’ survey model. Sankhya: The Indian Journal of Statistics, Series B, Vol. 31(3/4), pp. 341–360.
Bassi, F., Fabbris, L., (1997). Estimators of nonsampling errors in interview – reinterview supervised surveys with interpenetrated assignments. In Survey Measurement and Process Quality, L. Lyberg, P. Biemer, M. Collins, E. DeLeeuw, C. Dippo, N. Schwarz, and D. Trewin, eds., John Wiley and Sons, New York, pp. 733–751.
Batmaz, I., Fahmi, F. M., (2015). Response variance estimation in personal interview surveys with several allocation schemes, Proceedings of the 60th World Statistics Congress of the International Statistical Institute, Rio de Janeiro, Brazil, pp. 1–6.
Biemer, P. P., (2012). Latent Class Analysis of Survey Error. John Wiley and Sons, New Jersey.
Biemer, P. P., Lyberg, L. E., (2003). Introduction to Survey Quality. Hoboken: Wiley Series in Survey Methodology
Biemer, P. P., Stokes, S. L., (1985). Optimal design of interviewer variance experiments in complex surveys. Journal of the American Statistical Association, 80, pp. 158–166.
Biemer, P. P., Stokes, S. L., (2004). Approaches to the modeling of measurement error. In Measurement Errors in Surveys Edited by P. Biemer, R.M. Groves, L. Lyberg, N.A. Mathiowetz, and S. Sudman. Wiley Series in Survey Methodology, New Jersey, pp. 487–516.
Biemer, P. P., Groves, R.M., Lyberg, L. E., Mathiowetz, N. A., Sudman, S., (2004). Measurement Errors in Surveys. Wiley Series in Survey Methodology, New Jersey.
Box, G. E. P., Hunter, J. S., Hunter, W. G., (2005). Statistics for Experimenters: Design, Innovation, and Discovery. John Wiley and Sons, New Jersey.
Cicchetti, D. V., (1972). A new measure of agreement between rank-ordered variables. Proceedings of the 80th Annual Conference of the American Psychological Association 1972, pp. 17–18.
Cohen, J., (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, pp. 37–46.
Cohen, J., (1968). Weighted Kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, pp. 213–220.
Dahlhamer, J., Zablotsky, B., Zelaya, C., Maitland, A., (2020). Modeling interviewer effects in the National Health Interview Survey. Chapter 21 in Olson, K., Smyth, J.D., Dykema, J., Holbrook, A.L., Kreuter, F., West, B.T., Eds. (2020 a). Interviewer Effects from a Total Survey Error Perspective. New York: Chapman and Hall/CRC., pp. 295–310.
Dijkstra, W., (1983). How interviewer variance can bias the results of research on interviewer effects. Quality and Quantity, 17, 179–187.
Edwards, B., Sun, H., Hubbard, R., (2020). Behavior change techniques for reducing interviewer contributions to total survey error. Chapter 6 in Olson, K., Smyth, J.D., Dykema, J., Holbrook, A.L., Kreuter, F., West, B.T., Eds. (2020 a). Interviewer Effects from a Total Survey Error Perspective. New York: Chapman and Hall/CRC., pp. 77–90.
Fahmi, F. M., (2013). Estimation of Response Errors in Complex Sample Surveys. Ph.D. thesis, Graduate School of Natural and Applied Sciences, Middle East Technical University, Ankara, Turkey.
Fellegi, I. P., (1964). Response variance and its estimation. Journal of the American Statistical Association, 69, pp. 19–54.
Forsman, G., Schreiner, I., (2004). The design and analysis of reinterview: An overview. Chapter 15 in Measurement Errors in Surveys Edited by P. Biemer, R.M. Groves, L. Lyberg, N.A. Mathiowetz, and S. Sudman. Wiley Series in Survey Methodology, New Jersey, pp. 279–301.
Fuller, W. A., (1995). Estimation in the presence of measurement error. International Statistical Review, 63(2), pp, 121–147.
Groves, R. M., Magilavy, L. J., (1986). Measuring and explaining interviewer effects in centralized telephone facilities. Public Opinion Quarterly, 50, pp. 251–266.
Moser, C. A., Kalton, G., (1979). Survey Methods in Social Investigation. Heinemann Educational Books, London.
Hansen, M. H., Hurwitz, W. N., Bershad, M. A., (1961). Measurement errors in censuses and surveys. Bulletin of the International Statistical Institute, 38(2), pp. 359–374.
Hansen, M. H., Hurwitz, W. N., Marks, E. S., Mauldin, W. P., (1951). Response errors in surveys. Journal of the American Statistical Association, 46(254), pp. 147–190.
Hansen, M. H., Hurwitz, W. N., Pritzker, L., (1964). The estimation and interpretation of gross differences and the simple response variance. In C.R. RAO (Editor): Contributions to Statistics, Presented to Professor P.C. Mahalanobis on the Occasion of his 70th Birthday. Calcutta, India: Pergamon Press, Oxford and Statistical Publishing Society.
Hagenaars, J. A., Heinen, T. G., (1982). Effects of role-independent interviewer characteristics on responses. In W.Dijkstra and J.van der Zouwen (eds.), Response Behaviour in the Survey Interview. Academic Press, London, pp. 91–130.
Hox, J. J., (1994). Hierarchical regression models for interviewer and respondent effects. Sociological Methods and Research, 22, pp. 300–318.
Hox, J. J., De Leeuw, E. D., Kreft, I. G. G., (2004). The effect of interviewer and respondent characteristics on the quality of survey data: A multilevel model. Chapter 22 in Biemer et al. (2004) Measurement Errors in Surveys. Wiley Series in Survey Methodology, New Jersey, pp.439–461.
Kalton, G., (1983). Introduction to survey sampling. Quantitative Applications in the Social Sciences, A SAGE University Papers Series, No. 35, Newbury Park.
Kish, L., (1962). Studies of interviewer variance for attitudinal variables. Journal of the American Statistical Association, 57, pp. 91–115.
Kish, L., (1965). Survey Sampling. John Wiley and Sons, New York. Landis, J. R., Koch, G., (1977). A review of statistical methods in the analysis of data rising from observer reliability studies. Statistica Neerlandica, 29, pp. 101–123 and pp. 151–161.
Lyberg, L., Kasprzyk, D., (2004). Data collection methods and measurement error: An overview. Chapter 13 in Biemer et al. (2004) Measurement Errors in Surveys. Wiley Series in Survey Methodology, New Jersey, pp. 237–257.
Mahalanobis, P. C., (1946). Recent Experiments in Statistical Sampling in the Indian Statistical Institute. Journal of the Royal Statistical Society, Vol. 109, pp. 325–378.
McCarthy, P. J., (1966). Replication: An approach to the analysis of data from complex surveys. National Center for Health Statistics, Vital and Health Statistics Series, No. 2(14).
McNabb, D. E., (2014). Nonsampling Error in Social Surveys. Los Angeles: SAGE Publications, Inc.
Montgomery, D. C., (2012). Design and Analysis of Experiments. New York: John Wiley and Sons.
Olson, K., Smyth, J. D., Dykema, J., Holbrook, A. L., Kreuter, F., West, B. T., Eds., (2020a). Interviewer Effects from a Total Survey Error Perspective. New York: Chapman and Hall/CRC.
Olson, K., Smyth, J. D., Dykema, J., Holbrook, A. L., Kreuter, F., West, B. T., (2020 b). The past, present, and future of research on interviewer effects. Chapter 1 in Olson, K., Smyth, J.D., Dykema, J., Holbrook, A.L., Kreuter, F., West, B.T., Eds. (2020 a). Interviewer Effects from a Total Survey Error Perspective. New York: Chapman and Hall/CRC., pp. 3–16.
O’Muircheartaigh, C. A., (1976). Response errors in attitudinal sample surveys. Quality and Quantity, 10, pp. 97–115.
O’Muircheartaigh, C. A., (1977). Response errors. In Analysis of Survey Data, Vol. 2, John Wiley and Sons, London, pp. 193–239.
O’Muircheartaigh, C. A., (1982). Methodology of the Response Errors Project. Fertility Survey, Scientific Reports, No. 28, London, UK. 32 pp.
O’Muircheartaigh, C. A., (1984). The pattern of response variance in the Peru fertility survey. WFS Scientific Report, No. 45.
O’Muircheartaigh, C. A., (1986). Correlates of reinterview response inconsistency in the Current Population Survey. Proceedings of the Second Annual Research Conference, U.S. Bureau of the Census, pp. 208–234.
O’Muircheartaigh, C. A., (1991). Simple response variance: estimation and determinants. In Measurement Errors in Surveys Edited by P. Biemer, R.M. Groves, L. Lyberg, N.A. Mathiowetz, and S. Sudman. Wiley Series in Survey Methodology, New Jersey, pp. 551–574.
O’Muircheartaigh, C. A., Campanelli, P., (1998). The relative impact of interviewer effects and sample design effects on survey precision. Journal of the Royal Statistical Society, A 161, pp. 63–77.
O’Muircheartaigh, C. A., Marckwardt, A. M., (1980). An assessment of the reliability of WFS data. World Fertility Survey Conference 1980 London, Record of Proceedings Volume 3, pp. 313–379.
Salant, P., Dillman, D. A., (1994). How to Conduct your own Survey? John Wiley and Sons, New York.
Verma, V., (1980). Sampling for national fertility surveys. World Fertility Survey Conference 1980 London, Record of Proceedings, Vol. 3, pp. 389–448.
West, B. T., (2020). Designing studies for comparing interviewer variance in two groups of survey interviewers. Chapter 23 in Olson, K., Smyth, J.D., Dykema, J., Holbrook, A.L., Kreuter, F., West, B.T., Eds. (2020 a). Interviewer Effects from a Total Survey Error Perspective. New York: Chapman and Hall/CRC., pp. 323–334.
WFS, (1977). Guidelines for Country Report No.1. WFS Basic Documentation No.8, London, U.K.: World Fertility Survey.
Yu, C. H., Ohlund, B., Digangi, S., Jannasch-Pennell, A., (2000). Estimating the reliability of self-reporting data for web-based instruction. Instruction and Research Support, Arizona State University, US.
