Wellbeing Measurement
Interview Method
May
30, 2008
Abstract
The
influence of interview method on life-satisfaction judgments was examined in the
German Socio-Economic Panel data (N = 44,386 respondents, 335,984 responses).
The main finding was that reported life-satisfaction was higher in oral
interviews than in mailed questionnaires. This finding indicates that wellbeing
levels based on oral interviews are inflated by a tendency to respond socially
desirable. A number of limitations are discussed.
Introduction
A
common concern in survey research is that possibility of social desirable
responding. Studies of responses on anonymous paper-pencil questionnaires show
little evidence of social desirable responding
(WellbeingMeasurementSocialDesirability). However, it is possible that social
desirability has a pronounced effect on reports of wellbeing when respondents
have to report their wellbeing to an interviewer or when the report is made in
the presence of a third person. Few studies have systematically examined this
question. The data from the German Socio-Economic Panel (SOEP) provide a unique
opportunity to examine this question in a large, representative sample.
Moreover, the SOEP data have been used in numerous research reports. Thus, even
if the results cannot be generalized to other surveys, the results are
relevant. [Although I am reporting my own analyses, the data are freely
available to researchers who sign a contract with the DIW. Feel free to run
your own analyses and contact me if your results do not replicate the findings
reported on this website]
Method
and Results
Respondents
were asked to report their life-satisfaction on an 11-point scale ranging from
0 = totally dissatisfied to 10 = totally satisfied (SOEP Info Documentation). A
total of 335,984 responses from 44,386 respondents (average 7.60 responses per
respondent). For most responses, the interview method was also recorded. Table
1 presents the main interview methods, number of respondents, and number of
responses for the most common interview methods. These methods cover 86% of all
responses.
Table 1
|
Method |
N Respondents |
N Responses |
Mean |
SD |
|
Oral Interview |
25,348 |
117,405 |
7.12 |
1.63 |
|
Written Q. with Interviewer |
17,253 |
77,506 |
6.89 |
1.65 |
|
Written Q without Interviewer |
10,207 |
18686 |
7.10 |
1.88 |
|
CAPI (since 1998) |
13,906 |
42,972 |
7.25 |
1.52 |
|
Mailed-questionnaire |
6,974 |
33,396 |
6.78 |
1.71 |
The
results in Table 1 simply average across all responses. To control for
potential selection effects, it is possible to compare responses by the same
individuals who used different methods. These results are reported in Table 2.
Table 2
|
Method 1 |
Method 2 |
N Respondents |
Mean 1 |
Mean 2 |
Correlation . |
|
Written Q with Interviewer |
8,448 |
7.19a |
6.78b |
.50 |
|
|
Oral Interview |
Written Q. without Interviewer |
5,828 |
7.08a |
7.02c |
.46 |
|
Oral Interview |
CAPI |
6,296 |
7.17a |
7.07d |
.59 |
|
Oral Interview |
Mailed questionnaire |
3,408 |
7.22a |
6.64e |
.42 |
|
Written Q. with Interviewer |
Written Q. without Interviewer |
5,869 |
6.83b |
7.03c |
.57 |
|
Written Q. with Interviewer |
CAPI |
3,444 |
6.88b |
7.21d |
.50 |
|
Written Q. with Interviewer |
Mailed questionnaire |
3,031 |
6.89b |
6.72e |
.50 |
|
Written Q. without Interviewer |
CAPI |
1,878 |
7.14c |
7.09d |
.39 |
|
Written Q. without Interviewer |
Mailed questionnaire |
1,803 |
7.12c |
6.68e |
.41 |
|
CAPI |
Mailed-questionnaire |
772 |
7.46d |
6.66e |
.48
. |
Table
2 replicates most of the mean differences reported in Table 1. This finding
suggests that the results are not due to selection effects, but rather effects
of the assessment method. In addition, Table 2 shows the correlations between
the different methods. Differences in correlations may reveal method effects on
the rank order of respondents, which would occur if a subgroup of respondents
changes their responses in response to a method. For example, only some
individuals respond socially desirable in an oral interview. However, many
other confounding factors can influence these correlations. These preliminary
analyses show correlations in a fairly narrow range from .39 to .57. The
correlation between oral interview and mailed questionnaire is at the low end
of this range, suggesting that method effects attenuate estimates of stability.
The
main pattern of results is that oral interview, CAPI, and written Q without
interview produce similar means. In comparison, mailed questionnaires and
written Q without interviewer produce lower means.
Conclusion
Consistent
with a social desirability effect, mailed questionnaires produced lower means
than most interview methods. However, the comparison of the means for
respondents who completed written questionnaires with or without the
interviewer does not show a social desirability effect. The largest difference
exists between CAPI and mailed back questionnaires (mean difference = .47).
Although this effect is small by standard criteria, a half-point difference in
means can produce misleading conclusions about wellbeing. As a result, caution
is warranted when results from surveys with different assessment methods are
compared.
Limitations
1.
The results are based on a single data set. It is unclear whether these results
generalize to wellbeing reports in general.
2. The analyses are limited to a question about life-satisfaction. Other
questions could produce different results.
3. The results are based on repeated responses in a panel study and may differ
from results when respondents answer these questions only once.
4. The results are correlational. It is possible that assessment methods were
influenced by wellbeing levels.