|
Measurement in Practice: Hints for Writing Effective Survey Items
Please note that this section
is an archive and is no longer being updated.
Prepared by Kimberly Raiford Wildes, MA
1. Place sensitive items near the end of the survey.
Sensitive questions may provoke embarrassment or
resentment and may result in nonresponse. As such,
a survey should begin with emotionally neutral questions
that the respondent is willing to answer. By the time
respondents reach the sensitive items, they will be
"warmed up"; however, if they are not, then
the researcher only risks a few missing items rather
than an entire questionnaire. (Ref: 1-3)
2. Be careful when using negatively worded items.
Example: "I am satisfied with my physician"
(positive) versus "I am not satisfied with my physician"
(negative).
Measurement experts have suggested using negatively
worded items, as well as positively worded items,
to reduce response bias or responding to every item
in a similar way (Nunnaly, 1967). However, it seems
that negatively worded items have the propensity to
load on a single, separate factor (Roberts, Lewinsohn,
& Seeley, 1993). One explanation concerning why
negative items behave differently has been given by
Marsh (1986): it is a cognitive-developmental phenomenon.
Those who are younger or who have poorer reading skills
may not accurately comprehend negative items. Other
studies have found that this bias is not limited to
children, however (Schrieshiem & Hill, 1981).
It is suggested that negative items be avoided especially
when "strongly disagree to strongly agree"
response categories are used. In general, researchers
wishing to use negatively worded items should carefully
interpret the data and factor loadings of the scale.
(Ref: 4-7)
3. Do not use double-barreled items.
Questions that contain more than one imbedded concept
are difficult for a respondent to answer, and thus
impossible to interpret. An item should only address
one issue. If a researcher wishes to assess two separate
ideas, then two separate items should be used. The
use of "and" or "or" often denotes
a double-barreled item. For example, the single item
"I believe that my physician is trustworthy and
competent," should be two items: "I believe
my physician is trustworthy," and "I believe
my physician is competent." In the double-barreled
item, it is impossible to distinguish if the respondent
is answering to the physician's "trustworthiness,"
"competence," both, or neither. (Ref: 3, 8)
4. Give consideration to item layout.
The appearance of a survey may influence response.
Aday suggests that questions and their response categories
should never be broken up between pages. There should
also be enough space in between items for responses.
Dillman's "Total Design Method" suggests
using white space as a divider between items, rather
than dividing lines. Dillman offers a number of other
suggestions for maximizing survey design: do not print
items on the front or back cover pages, provide space
for additional comments at the end of the survey,
include a cover page with a catchy title and illustration,
use larger font as necessary (e.g. elderly samples),
and make self-administered questionnaires no longer
than twelve pages. (Ref: 9-10)
5. Use simple language.
Use the fewest words and simplest grammatical structure
possible. Try to avoid compound sentences, which may
result in a respondent's struggle to synthesize several
pieces of information. Items must be phrased in ways
that all respondents can understand. Responses should
not be so complex that they cause respondent burden.
Pilot-testing is effective at revealing items that
may have double meanings or other problems. (Ref: 1, 8)
6. Avoid technical jargon.
Most respondents will not understand very technical,
industry-laden terms. To ensure maximum response,
use words common to the general population. For example,
do not use "palpitations" or "anhedonia"
when it can be explained in simpler terms. Focus groups
and pilot-testing can be useful in identifying difficult
jargon. (Ref: 3)
7. Do not use leading questions.
Avoid leading the respondent to answer in a favorable
way. Leading questions will bias results, even if
unintended. In other words, questions should be fair
to the respondent and not one-sided. In this way,
leading items "beg the question" by suggesting
the truth of something that is not known. Asking,
"What are the three main things that you do not
like about your physician?" is clearly leading
the respondent to believe that there are things they
should not like. These questions should only be used
in attitude assessment when it is deemed a necessary
way to elicit a response. (Ref: 1, 8)
8. Ensure that questions have face validity.
Only questions that are necessary should be asked.
Avoid including questions that might yield interesting
and informative information. Make sure that the questions
accurately relate to the concept you wish to measure.
Face validity may be determined by obtaining subject/participant
and expert input, for example, by organizing focus
groups. Adequate literature review will also help
ensure face validity. (Ref: 1, 8)
9. Use mutually exclusive response categories.
If response options are not mutually exclusive, the
respondent will have more than one available answer
choice. For example, if option categories for age
are "18 to 19," "19 to 20," and
"Older than 20," the respondent can answer
in more than one category, causing confusion and ultimately
skewing results. This issue is particularly relevant
to reporting of ethnicity, where such problems commonly
arise. (Ref: 8)
10. If using skip patterns, provide
clear instructions.
Items that require skip patterns can become confusing
for respondents. The interviewer and respondent should
be clear on when and how to utilize skips. Instructions
may be provided in parenthesis next to the relevant
response choice, or as Dillman recommends, arrows
may be used in self-administered surveys to indicate
the response flow. The patterns should be checked
and tested several times to ensure that the items
perform correctly, before using the survey in the
field. (Ref: 9-10)
11. Ensure that question and answer options match.
Question and answer responses are often written in
confusing and grammatically incorrect ways. For example,
"yes/no" questions are paired with frequency
responses such as "never to always" and
frequency questions are given "agree to disagree"
options. The question "Do you visit your physician?"
should have response options of "yes/no,"
but may in fact have the options of "never/seldom/often/always."
(Ref: 3)
12. Institute item checks.
As aforementioned, pilot-testing
is a beneficial way to eliminate complex or technical
questions and ensure face validity, as well as to ensure
that item format, length, and placement is appropriate.
Besides pilot-testing, there are other ways to ensure
that items work as intended: have respondents rephrase
items as they understand them, question respondents
about their interpretations of items, and have them
think aloud as they derive answers. (Ref: 1, 11)
13. When assessing attitudes, place
general questions before specific questions.
Some researchers suggest that the
answer to a general question may be influenced by specific
questions that come prior. For this reason, general
questions such as "Are you generally happy?"
should be placed before "Are you happy in your
marriage?" Other researchers suggest random
ordering of items in order to evaluate the existence of
order effects; the placement of items should be based
on whether or not order effects are found. (Ref: 1)
14. Always say "thank you" at the end
of the survey.
1. Abramson, J.H., & Abramson, Z.H. (1999). Constructing
a questionnaire. Survey methods in community medicine
(5th ed.) (pp. 221-232). Edinburgh: Churchill Livingstone.
2. Frary, R.B. (1996). Hints for designing effective questionnaires.
Practical Assessment, Research, and Evaluation [On-line],
5(3). Available:
http://pareonline.net/getvn.cfm?v=5&n=3.
3. Hulley, S.B., Cummings, S.R.,
Browner, W.S., Grady, D., Hearst, N., & Newman, T.B.
(2001). Designing questionnaires and data collection
instruments. Designing clinical research (2nd ed.) (pp.
321-245). Philadelphia, PA: Lippincott Williams &
Wilkins.
4. Marsh, H.W. (1986). Negative item bias
in ratings scales for preadolecscent children: A
cognitive-developmental phenomenon. Developmental Psychology,
22(1), 37-49.
5. Nunnaly, J.C. (1967). Psychometric theory. New York:
McGraw Hill.
6. Roberts, R.E., Lewinsohn, P.M., & Seeley, J.R.
(1993). A brief measure of loneliness suitable for
use with adolescents. Psychological Reports, 71, 275-78.
7. Schrieshiem, C.A., & Hill, K.D. (1981). Controlling
acquiescence response bias by item reversals: The
effect on questionnaire validity. Educational and
Psychological Measurement, 41, 1101-1114.
8. Narins, P. (date unknown). Write more
effective survey questions.
9. Aday, L.A. (1996). Guidelines for
formatting the questionnaire. Designing and conducting
health surveys (2nd ed.) (pp. 261-280). San Francisco,
CA: Jossey Bass.
10. Dillman, D.A. (1978). Mail and telephone surveys:
The total design method. New York: Wiley.
11. Foddy, W. (1993). Checks to ensure that questions
work as intended. Constructing questions for interviews
and questionnaires: Theory and practice in social
research (pp. 181-188). Cambridge, UK: Cambridge University
Press.
|
