arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Social Sciences · Sociology and Political Science

A COMPARISON OF TWO BEHAVIOR CODING SYSTEMS FOR PRETESTING QUESTIONNAIRES

W. Sherman Edwards, Vasudha Narayanan, Stéphanie Fry, Joseph A. Catania, Lance M. Pollack

See on the map

Abstract

Introduction Coding of interviewer and respondent behavior is well established as a tool for evaluating survey questionnaires (as noted by Schaeffer, 1991, e.g.), particularly during pretests. Perhaps the most widely used behavior coding system for questionnaire evaluation was developed at the University of Michigan. As described in Oksenberg et al (1991), coders listen to interviews and assign codes to the interviewer’s reading of a question (read exactly as worded, read with slight change, read with major change) and to the respondent’s subsequent behavior (interruption with answer, request for clarification, adequate answer, qualified answer, inadequate answer, don’t know, refusal to answer). For a given question in a given interview, a coder assigns one interviewer behavior code and one or more respondent code, and the codes are tallied for each question across coded interviews. Each code other than “read exactly as worded” or “adequate answer” may indicate a problem with the question. Oksenberg et al used behavioral coding, special probes designed into the interview, and coder and interviewer debriefing to evaluate the pretest questionnaire in their experimental study. They conclude that behavior coding is reliable and efficient, that it can be implemented either live or with tape recorded interviews, and that a simpler coding scheme than the one described in the 1991 paper would be quite effective for pretesting questions. Interpretation of behavioral coding results is both a quantitative and qualitative exercise. Oksenberg et al note that tallying codes does not necessarily identify the reason for apparent problems. Schaeffer also points out that applying statistical tests to tallies of behavior codes may not be appropriate because of the clustering of behaviors by interviewer and respondent. Westat has sporadically used behavior coding to evaluate pretest questionnaires, and has not yet centralized the process, so coders are typically inexperienced. Westat researchers have reported (personal communications) that the system described by Oksenberg et al is too complex to institute live, particularly when the coders are telephone center team leaders with other responsibilities. In one recent pretest of a telephone survey, Westat reduced the behavior coding scheme to a single code – whether or not the respondent gave a response to the question that the interviewer could record without probing or offering clarification. Essentially, this simplified system combines all of the Oksenberg et al respondent problem codes except “interrupts with answer” and, perhaps, “qualified answer.” All behavioral coding was done live. Results of this coding were combined with qualitative monitoring and interviewer debriefing to inform final revisions to a telephone survey questionnaire. After this experience, we wondered how much more information would have been provided by a more complete set of codes, how comparable the results would have been in assigning codes, and whether live coding differed from coding of tape recorded interviews. This paper describes such a comparison made on a pretest of the Urban Men’s Health Study 3. The UMHS-3 is the third in a series of randomdigit-dial surveys of men who have sex with men (MSM) in San Francisco, conducted by researchers at the University of California at San Francisco (UCSF). The UMHS-3 will be used to compare the extent of risky sexual behavior of MSM with and without a history of childhood sexual abuse. Westat is collecting the survey data. Part of Westat’s responsibility was to conduct a pretest, using essentially the evaluation techniques described by Oksenberg et al.

More like this