55

Chapter 5 - Questionnaire Design

5.0 Introduction
A questionnaire (or form) is a group or sequence of questions designed to obtain information on a
subject from a respondent. Questionnaires include forms used for sample surveys and censuses as well as
administrative forms. Questionnaires play a central role in the data collection process. They have a major
impact on data quality since the questionnaire is the process whereby data are collected. They also
influence the image that the statistical agency projects to the public.

The questions posed must conform to the survey’s Statement of Objectives and provide information that
is useful for data analysis. All information requirements must be met, but each question should have an
explicit rationale for being on the questionnaire. It must be known why each question is being asked and
how the information is to be used. The question wording must be clear. The questions must be sequenced
logically for the respondent. The questions must be designed so that they are easily understood and can be
accurately answered by respondents. Finally, the questionnaire should be tested before implementation
using cognitive testing, focus groups, informal testing and other methods described in this chapter.

A well-designed questionnaire should:

- collect data efficiently with a minimum number of errors and inconsistencies;
- be respondent friendly and interviewer friendly (if interviewer-assisted);
- lead to an overall reduction in the cost and time associated with data collection.

The purpose of this chapter is to provide an overview of the steps of questionnaire design, including the
development and testing of questionnaires. The main types of questions – open versus closed – are
described as well as their advantages and disadvantages. Guidelines for developing questions are also
covered. Finally, response error and processing issues related to questionnaire design are addressed.

5.1 The Questionnaire Design Process

The questionnaire design process starts with the formulation of survey objectives and information
requirements (Chapter 2 - Formulation of the Statement of Objectives) and continues with the
following steps:
- consult with data users and respondents;
- review previous questionnaires;
- draft the questionnaire;
- review and revise questionnaire;
- test and revise questionnaire;
- finalise questionnaire.

5.1.1 Consult with Data Users and Respondents

The process of consulting with data users, which began during with the formulation of the survey’s
objectives during the planning phase, continues during the design and development of the questionnaire.
This extended consultation is particularly important for major, if not all surveys conducted by a statistical
agency. By clearly understanding how the data are to be used, the statistical agency should be able to
develop a well-designed questionnaire that meets the users’ needs.

STATISTICS CANADA
56 SURVEY METHODS AND PRACTICES

Not only data users, but respondents, experts in the field of study and those who have conducted similar
surveys in the past should be consulted before the questionnaire is drafted. They should be able to provide
insight into the type of information that respondents can provide and help clarify the concepts to be
studied. Meeting with respondents can help identify issues and concerns that are important to them and
may affect decisions regarding the content of the questionnaire. It can also be helpful in identifying the
wording and language that respondents use to describe the survey concepts – this can provide useful
insights into how questions and response categories should be worded. These discussions can take the
form of in-depth consultations or focus groups (see Section 5.1.5.3).

5.1.2 Review Previous Questionnaires

Other surveys are a good source of information when developing a survey. Examining questions that were
used by other surveys on the same or a similar topic can be a useful starting point when considering how
to draft a question (namely, the question wording). In some situations, the same questions should be used,
for instance when trying to compare results from different surveys. Also, documentation on the data
quality from such surveys should be examined to assess the effectiveness of the questionnaire (e.g.,
problems with question wording, response burden, refusal rates, etc.).

5.1.3 Draft the Questionnaire

The next step is to develop a draft of the whole questionnaire. Because the questionnaire is affected by the
whole survey design, along with the survey’s objectives the following factors must be considered:

i. Method of data collection

The length of the questionnaire and the manner in which questions are posed depends on the data
collection methodology. For self-enumeration surveys, questionnaires should be less complex and shorter
than for interviewer-assisted methods and preferably be ‘stand-alone’ meaning all relevant information
(e.g., instructions, contact information, examples) is included on the questionnaire. With interviewer-
assisted methods, the wording of a question is often different than for self-enumeration questionnaires.
That is, the question should sound natural when asked out loud. With self-enumeration and personal
interviews more response categories are possible than with telephone interviews which should be kept
short.

The appearance, organisation and data recording structure will also be very different depending on
whether the questionnaire is for self-enumeration, for telephone or personal interview, paper and pencil,
or computer-assisted.

ii. Characteristics of respondents

The characteristics of respondents should be considered when drafting the questions. They may have an
impact on the terminology or complexity of language used for the questions. Questions intended for a
survey of the general public should be easily understandable by all respondents, while a survey aimed at
professionals may use technical or professional language related to the work of the respondents.

STATISTICS CANADA
 QUESTIONNAIRE DESIGN 57

iii. Response burden

Every effort should be made to consider the burden of responding to the questionnaire, the time and effort
required to provide answers and whether other individuals or records have to be consulted. The number
of questions should be kept to a minimum. Each question must have a reason for being on the
questionnaire. (The purpose of some questions may be to ensure that respondents understand a subsequent
question, or for evaluation purposes.)

iv. Complexity of the data to be collected

Careful wording of the questions is necessary when collecting complex data. Instructions should be
included in questions covering complex topics. This will help the interviewer to explain questions and the
respondent to accurately answer them.

v. Confidentiality and sensitivity of the information

These issues may directly influence how the questions are drafted. The questionnaire instrument should
include introductory statements regarding how the confidentiality of respondent data is to be protected. It
should also explain how the data are to be used, who has access to the data, how long the data will be
kept, etc. When asking questions about sensitive issues (a question that respondents may not feel
comfortable answering), it may be necessary to use techniques that soften the impact of asking the
question. This increases the likelihood of a response (for more, see section 5.3.8).

vi. Translation

The questionnaire should be translated into all languages that are commonly spoken by the target
population. Care must be taken when translating questions from another language to take account of not
only the language, but also different customs and culture.

vii. Comparability of results with other surveys

When survey results are to be compared with other surveys, the questions must be drafted in the same
manner. Each version of the question must cover the issue in the same way and have the same meaning in
the context of the question. To ensure the comparability of the results with other surveys, the same
question wording should be used after the quality of the earlier results is confirmed. Also, some questions
may be sensitive to questions immediately preceding them.

viii. Consistency

The wording of the question must have the same meaning for all respondents as that intended by the
statistical agency. If the questionnaire is to be translated into different languages it is particularly
important to test all language versions.

ix. Other factors

Other factors to consider when drafting questions include:

- the availability of the data being sought;
- the respondent’s willingness to provide the data;
- the likelihood of nonresponse;
- administrative requirements;

STATISTICS CANADA
58 SURVEY METHODS AND PRACTICES

- the type of questions;

- the wording of each question;
- the layout of the questionnaire ;
- sources of measurement or response error;
- processing of the questionnaire.

Administrative requirements of the survey organisation include such things as data sharing agreements, a
statement informing respondents of the confidentiality of their responses, bilingual versions of the
questionnaire, etc.

Questions can be either open or closed (closed questions provide response categories). Types of questions
are discussed in detail in section 5.2. Guidelines for wording survey questions are covered in section 5.3.
Sources of response error are discussed in section 5.4. The layout of the questionnaire and processing
considerations are presented in sections 5.5 and 5.6.

5.1.4 Review and Revise the Questionnaire

It is essential that the questionnaire be reviewed internally before it is tested. This review should identify
any obvious problems with the questionnaire such as mistakes in spelling or grammar or awkward
wording. It is also useful at this stage to have the questionnaire reviewed by people who are not directly
involved with the project. Reviewers could include experts in the field of study, people who have
experience designing questionnaires, interviewers or members of the population to be surveyed. They can
often provide helpful comments and suggestions leading to revisions in the questions and response
categories.

5.1.5 Test and Revise the Questionnaire

It is important to test all versions (i.e., all language versions) of the questionnaire on ‘typical’ respondents
long before data collection begins (i.e., typical for the target population, which may mean respondents of
a certain age, sex and level of education). It may also be important to test the questionnaire on specific
subpopulations that may have problems with particular questions.

Responding to a question is a complex process. Respondents must first understand the question. They
must then search their memories or records to retrieve the requested information. After retrieving the
information, they must think about the correct answer to the question and how much of that answer they
are willing to reveal. Only then do they communicate an answer to the question. Each of these processes
may be a source of error.

Testing methods are intended to identify difficulties and errors with the questions. Testing can also
identify whether the question order affects the interpretation of questions, whether instructions are clear
and how respondents feel about the appearance of the questionnaire. The benefit of questionnaire testing
includes the production of a respondent and interviewer friendly questionnaire which permits the
collection of accurate data, with a format that facilitates data capture and coding thereby minimising
errors and reducing the cost and time of data collection, capture and processing.

Methods used to test questionnaires (their content, layout, etc.) are typically based on small, subjective
non-probability samples of respondents from the target population. The methods described in the
following sections are:

STATISTICS CANADA
 QUESTIONNAIRE DESIGN 59

- informal testing (pretesting);

- cognitive methods;
- focus groups;
- interviewer debriefings;
- behaviour coding of interviewer/respondent interactions;
- split sample tests;
- pilot tests.

5.1.5.1 Informal Testing (Pretesting)

Informal testing (sometimes referred to as pretesting) is easy and inexpensive and is a fundamental step in
the development of a questionnaire. If no other testing of the questionnaire is done, as a minimum,
informal testing should be performed. The size of the informal test sample can range from 20 to 100 or
more respondents. If the main purpose is to discover wording or sequencing problems, very few
interviews are needed. More interviews (50 to 100) are necessary to determine response categories for
closed questions based on the responses to open questions in the informal test. The questionnaire should
be administered in the same manner as planned for the main survey (e.g., interviewer-assisted in person or
by telephone; paper or computer-assisted). However, self-enumeration questionnaires should be tested
using an interviewer.

With informal testing, respondents are not debriefed; the respondent simply completes the questionnaire
or interview, mirroring what would happen during the actual data collection. Informal testing only
indicates where there is a problem. Without further investigation, it does not identify why or how it can
be corrected. Also, informal testing may not identify all problems with the questionnaire.

Informal testing of questionnaires is used to:

- discover poor question wording or ordering;
- identify errors in questionnaire layout or instructions;
- identify problems with the computer software application, if computer-assisted;
- determine problems caused by the respondent's inability or unwillingness to answer the questions;
- suggest additional response categories that can be pre-coded on the questionnaire;
- provide a preliminary indication of the interview length and response rate (including item
nonresponse).

5.1.5.2 Cognitive Methods

Cognitive methods are especially useful in testing questionnaires because they are designed to investigate
the steps in the response process. Cognitive methods provide the means to examine a respondent’s
thought processes as he or she answers the survey questions. In this way, cognitive methods help assess
the validity of questions and identify potential sources of response and nonresponse error.

Cognitive interviews provide the opportunity to evaluate the questionnaire from the respondent’s point of
view. They focus on issues such as comprehension and reactions to the form. This brings the respondent’s
perspective directly into the questionnaire design process, and leads to the design of respondent-friendly
questionnaires that can be easily understood and accurately completed.

Cognitive interviews often take place in a ‘laboratory’ setting or room that has a one-way mirror for
observation. Sample sizes are relatively small. As few as 12 to 15 to as many as 100 or more cognitive

STATISTICS CANADA
60 SURVEY METHODS AND PRACTICES

interviews are used to test a preliminary version of a questionnaire. Because sample sizes are relatively
small, an iterative approach is sometimes adopted whereby changes are made to the questionnaire
following a few cognitive interviews before further testing takes place.

Cognitive testing methods include:

i. The observation of respondents

The respondent is observed while completing the questionnaire. Observation provides information on
respondent behaviour such as:
- areas of the questionnaire that are read;
- the sequence in which questions are answered;
- whether reference is made to instructions;
- the types of records consulted;
- whether another person is consulted;
- the time required to complete each section;
- corrections or changes made to responses.

ii. Think aloud interviews

The respondent is asked to ‘think aloud’ while answering the questions, commenting on each question,
and explaining how the final response was chosen. This type of think-aloud interview is called a
concurrent think-aloud interview. If instead, the respondent explains his/her thought processes afterwards
during a follow-up discussion, then the interview is called a retrospective think-aloud interview. Both of
these methods are very useful in testing questionnaires and in identifying potential sources of error and
improvements that can be made.

Specific techniques such as probing questions, paraphrasing and confidence ratings are used during
cognitive interviews.

a. Probing questions

Probing questions are used to focus on particular aspects of the response process (i.e., understanding,
retrieving, thinking or responding). For example, the interviewer may ask how and why a respondent
chose an answer, or how concepts, words and time reference periods were interpreted.

b. Paraphrasing

The respondent is asked to repeat the instructions or the question in his or her own words, or to explain
the meaning of terms and concepts. Paraphrasing helps determine whether a respondent has read and
understood the instructions and questions as intended.

c. Confidence ratings

The respondent rates the degree of confidence in the accuracy of his or her own answers. This technique
indicates to what extent the respondent had difficulty in formulating an answer to a question and whether
or not guessing was involved.

STATISTICS CANADA
 QUESTIONNAIRE DESIGN 61

5.1.5.3 Focus Groups

A focus group is a discussion of a selected topic by participants who are chosen from the population of
interest. Focus groups provide the opportunity to consult with members of the target population, data users, or
interviewers so that their points of view can be brought into the questionnaire design process. In the early
stages of questionnaire development, focus groups can be used to clarify survey objectives and data
requirements, to identify salient research issues, definitions, and concepts. Focus groups are also useful in
testing questionnaires. They are used to evaluate a respondent's understanding of the language and wording
used in questions and instructions, and to evaluate alternative question wordings and formats.

A focus group is led by a moderator who is knowledgeable about group interviewing techniques and the
purpose of the discussion. The size of each group is usually between 6 and 12 persons, with the optimum size
being 7 to 9 persons. A focus group usually lasts for about two hours. Focus groups are audio-recorded (and
sometimes video-recorded) and can be viewed by observers in an adjoining room behind a one-way mirror.
The observation of focus groups by the questionnaire developers is recommended. Although observers do not
take part in the focus group discussion, their observations can be helpful when communicated to the
moderator at the end of the focus group session.

If a self-enumeration questionnaire is being tested, then the questionnaire can be completed immediately prior
to the focus group discussion (time permitting) or completed in advance and brought to the focus group
session by the respondent. If the questionnaire is interviewer-assisted, the interviewers can administer the
questionnaire a few days before the focus group takes place.

A useful technique is to begin the focus group discussion by asking participants to provide their overall
reaction to the questionnaire. Specific questions and issues concerning the questionnaire are then discussed.
The focus group moderator reviews the whole questionnaire question-by-question or focuses on specific
questions of interest. The moderator should be skilled at probing since some focus group participants may be
reluctant to make negative comments even if they are pertinent. The moderator should also give each member
the opportunity to speak, to avoid situations where one or two persons dominate the discussion. The focus
group can be concluded by asking participants to recommend the most important improvement that they feel
should be made to the questionnaire.

5.1.5.4 Interviewer Debriefings

Interviewer debriefings often take place after a focus group session or during informal tests. Interviewers
discuss their experiences interviewing respondents and provide an understanding of how the
questionnaire is performing. Their perspective can identify questionnaire improvements. Interviewer
debriefings usually take place in a group setting that is very similar to a focus group. Alternatively, rating
forms or debriefing questionnaires can be used to collect information about problems encountered with
the questionnaire and suggestions for improvements.

5.1.5.5 Behaviour Coding of Interviewer/Respondent Interactions

While the questionnaire is being administered by an interviewer, behaviour coding can be conducted.
This type of testing involves the systematic coding by a third party of the interaction between the
interviewer and the respondent. The focus is on how the interviewer asks the questions and how the
respondent reacts. The test interview is often audio-recorded and the interviewer-respondent relation is
later analysed. Behaviour coding helps identify problems such as the interviewer failing to read the

STATISTICS CANADA
62 SURVEY METHODS AND PRACTICES

question as worded or a respondent asking for clarifications. If behaviour coding indicates difficulties
with a certain question, then corrective action may be warranted. Behaviour coding does not, however,
usually provide information about the reasons for the problems nor solutions to them. A large sample size
is often required to analyse the results of behaviour coding, especially if the questionnaire has many skip
patterns that take respondents through various paths of the questionnaire.

5.1.5.6 Split Sample Tests

Split sample tests are conducted to determine the ‘best’ of two or more alternative versions of a
questionnaire or question. Split sample testing is sometimes called ‘split ballot’ or ‘split panel’
experiments. It involves an experimental sample design that is incorporated into the data collection
process. In a simple split sample design, half of the sample receives one version of the questionnaire
while the other half receives the other version.

Besides comparing variations in questions, split sample tests can also compare different methods of data
collection in order to determine which method is better. A probability sample design and adequate sample
sizes are required to analyse differences between the samples.

5.1.5.7 Pilot Test

A pilot test is conducted in order to observe all stages of the survey process, including administration of
the questionnaire. A pilot study is a ‘dress rehearsal’; it duplicates the final survey design on a small scale
from beginning to end, including data processing and analysis. It allows the statistical agency to see how
well the questionnaire performs during all the steps in a survey (data collection, editing, imputation,
processing, data analysis, etc.). Normally, the questionnaire is thoroughly tested using the methods
mentioned previously before a pilot test takes place.

5.1.6 Finalise the Questionnaire

Questionnaire design is an iterative process: throughout questionnaire development and testing, changes
are continually being made. Objectives and information requirements are formulated and re-evaluated,
data users and respondents are consulted, proposed questions are drafted and tested, questions are
reviewed and revised until a final questionnaire is developed. The final stage of the process comes when
the decision is made that no further changes will be made to the questionnaire. At this point, the
questionnaire is finalised and printed or programmed, depending on the data capture method being used.

5.2 Types of Questions: Open versus Closed

There are two types of questions: open and closed. Open questions are those where response categories
are not provided to the respondent. The respondent provides an exact numerical figure or answers the
question in his or her own words which is either written down, in the case of self-enumeration, or
recorded verbatim by the interviewer. Open questions should be followed by sufficient space to record the
response.

An example of an open question is:

STATISTICS CANADA
 QUESTIONNAIRE DESIGN 63

What is the most important problem facing Canada?

Closed questions are those where response categories are listed with the question. Closed questions are
answered by checking a box or circling the proper response from those provided. The possible responses
listed for a question are called response categories.

An example of a closed question is:

Which of the following is the most important problem facing Canada? (Please check one only)
P Unemployment
P Economy/Recession
P Federal Deficit
P Taxes
P National Unity
P Crime/Violence
P Environment
P Other

Open questions allow the respondent to express an answer without being influenced by the response
categories that would be provided by a closed question. However, this allows the question to be
interpreted different ways. An open question therefore generally expands the scope of the question while
the closed version gives the respondent clues about how the question is to be interpreted. Closed
questions also restrict the respondent to a predetermined set of responses.

There are several applications of open questions. One advantage of open questions is that they allow the
respondent the opportunity for self-expression or elaboration. This is important when examining an issue
that is not well understood or is very broad. Consequently, open questions are often used during
questionnaire development and testing. For example, they are used in focus groups to obtain comments
and opinions on the question presented and to generate discussion. Open questions also allow the
statistical agency to obtain the respondent’s own ‘natural’ wording. This is important when examining
question wording and response categories for a closed question.

Another advantage of open questions is that they can be used to obtain exact numerical data, for example
a respondent’s exact age. Business surveys often ask for exact amounts for reported revenues and
expenses. Exact numerical data are necessary for certain types of data analysis (e.g., to calculate an
average or median).

Another use of open questions is to provide follow-up to closed questions. For example, a closed question
may ask:

Do you have any suggestions to improve our service to our customers?

P No
P Yes

If the respondent answers ‘Yes,’ then a follow-up open question might be:

STATISTICS CANADA
64 SURVEY METHODS AND PRACTICES

If yes, what are your suggestions?

Open questions such as ‘Do you have any additional comments?’ are often used at the end of sections of
questions or at the end of the questionnaire. This ensures that the respondent has the opportunity to
include everything that he or she feels is relevant to the issues being addressed by the questionnaire.
Some respondents may want to include additional information to clarify an answer. It is important to plan
for this type of information on the questionnaire.

There are disadvantages to open questions with write-in (as opposed to numeric) responses. They are
burdensome since the respondent must determine the intent of the question and formulate an answer
without the aid of response categories. And for self-enumeration surveys, recording the answer is time-
consuming. For interviewer-assisted surveys, data collection, capture and coding is burdensome. It is
often difficult for the interviewer to capture the respondent’s answer verbatim and after collection, in
order to facilitate processing and data analysis, all of the different responses are usually categorised and
assigned a numerical code. Open questions, therefore, usually lead to more costly, time-consuming and
error-prone processing than closed questions.

In the case of numerical data, open questions are more difficult for respondents to answer than closed
questions and more difficult and prone to error during data capture.

Examples of open questions requiring write-in responses are:

What products or services does your business provide?

What do you think you will be doing 5 years from now?

Please comment further on any questions or issues covered in this questionnaire

Examples of open questions requiring numeric responses are:

What is your best estimate of the total income before taxes and deductions of all household
members from all sources in the past 12 months?

How many years has the current owner operated this establishment?

In a typical month, how often do members of your household use the Internet at home?

There are numerous advantages to closed questions, the most important ones being that they are less
burdensome to respondents and data collection and capture is cheaper and easier. They are quicker and
easier to answer since respondents simply choose the appropriate response categories rather than
formulate a response and state it in their own words. The respondent is more likely to answer the question
as intended because the response categories indicate the focus of the questions. It is easier to analyse data
that has been collected using closed questions, since the answers are more consistent and already grouped.
Also, if a question is being used for several surveys, using the same response categories facilitates
comparisons across surveys.

STATISTICS CANADA
 QUESTIONNAIRE DESIGN 65

There are several disadvantages to closed questions. When drafting the questions, effort is sometimes
required to develop the response categories (i.e., coding is done before collection, but it can still be
difficult). As with all coding, the response categories must be mutually exclusive and exhaustive. If the
response categories are not clearly formulated, the respondent could have more problems than if an open
question were used. Another concern with closed questions is that because response categories are listed,
the respondent may feel obliged to choose one of the response categories whether or not her or she has
formulated an answer or even has the knowledge required to answer the question. For questions where an
opinion is asked, this can force the respondent to choose a category that does not correspond to his or her
opinion, or to express an opinion when in fact he or she has none. (To avoid this situation, a ‘Do not
know’ and/or ‘Not applicable’ category is sometimes included. For interviewer-assisted questionnaires, it
is common practice to include a ‘Refused’ response category.) Another potential problem is that response
categories can oversimplify an issue by restricting a respondent’s possible answer.

There are several kinds of closed questions: the most frequently used are two choice, multiple choice and
checklist, ranking and rating questions. They are described below.

For more information on coding open and closed questions, see Chapter 10 - Processing.

5.2.1 Two-Choice Questions

The two-choice question is the simplest version of a closed question. Often it is a yes/no question and is
used to split the respondents into two distinct groups. Two-choice questions are also used as screening
questions to prevent respondents from being asked a series of questions that do not apply to them. The
direction ‘Go to question X’ then appears directly after one of the response categories so that the
respondents can skip some questions. This results in what is referred to as skip patterns in a questionnaire.

For example:

Did you smoke cigarettes yesterday?

P Yes
P No -------------Go to Question 14

5.2.2 Multiple Choice and Checklist Questions

The multiple choice question asks the respondent to select one response from a list of choices, whereas
the checklist question asks the respondent to pick one or more responses from the list. It may not be
evident to the respondent whether the question is multiple choice or checklist. Therefore, instructions
should be included. Note that an ‘Other (specify)’ category is normally provided to ensure that the list is
exhaustive.

Example of a multiple choice question:

What type of dwelling is this? (Check one only)

STATISTICS CANADA
66 SURVEY METHODS AND PRACTICES

P Single house
P Double (side-by-side)
P Garden home, town house or row house
P Duplex (one-above-another)
P Low-rise apartment (less than 5 stories)
P High-rise apartment (5 stories or more)
P Other (please specify)______________

Example of a checklist question:

During your vacation, in what types of accommodation did you stay? (Check as many as apply)
P Hotel (including tourist home)
P Motel
P Camping or trailer park
P Home of friends or relatives
P Commercial cottage or cabin
P Other (hostels, universities, etc.)

The response categories for multiple choice and checklist questions require careful drafting. The list of
response categories should be mutually exclusive and exhaustive. Consider the following example:

How old are you in years?

P 20-30
P 30-40
P 40-50
P 50 or more

The categories in this example are not mutually exclusive; they overlap. A respondent who is 30, 40 or 50
years old has a choice of two response categories. Data analysis would be difficult since it is impossible
to know which category such a respondent would choose. Another problem with the response categories
is that the list is not exhaustive. If persons under 20 are part of the target population, how would they
answer this question? A better choice of response categories would be:

How old are you in years?

P under 20
P 20-29
P 30-39
P 40-49
P 50 or more

5.2.3 Ranking Questions

Another type of closed question is the ranking question, where the respondent is asked to order the
response categories. For example:

Here is a list of some of the ways that people go about finding jobs. Please rank them in order of
effectiveness by placing the number ‘1’ beside the method you think would be most helpful, a ‘2’
beside the method you think would be second most helpful, and so on.

STATISTICS CANADA
 QUESTIONNAIRE DESIGN 67

____ Mailing out résumés

____ Newspaper or magazine ads
____ Government Employment Centres
____ Checking with friends
____ A private placement service
____ Direct contact with employers
____ Other (please specify) ________________________________

Respondents often find it burdensome to rank categories, especially if the items to be ranked are very
different from each other, or if the interview is conducted over the telephone. Another problem with
ranking questions is that the sizes of the rank intervals are unknown and unlikely to be equal: the interval
between 1 and 2 cannot be interpreted to be the same as that between 2 and 3. This complicates data
analysis. For example, if three items are listed to be ranked, a respondent will rank them 1, 2 and 3, but it
may be that the respondent considers the first two to be very close and the last item to be a distant third.
Such information cannot be gleaned from the simple ranking. Another problem is that a respondent may
assign two or more items the same ranking. In the above example, a respondent might rank both
‘Newspaper or magazine ads’ and ‘Government Employment Centres’ as 1. Another problem with
ranking questions is that respondents may not be able to rank all the choices on the list. It may be
reasonable to expect respondents to only rank a few (e.g., five or less).

The following is an example of a ranking question where the respondent is asked to select the most
important factors and then rank only the ones that are important.

Please rank the 5 most important factors that influence your company's choice of a
transportation carrier. This information will help us to focus our attention and resources on
areas that are critical to meeting your service needs.

Please rank their importance by placing the number ‘1’ beside the factor that you think is the
most important, a ‘2’ beside the factor that you think is the second most important, and so on.
___ Damage-free transportation
___ Price
___ Marketing and sales representatives
___ Customer service representatives
___ Prompt resolution of service problems
___ Freight claims handling
___ Service consistency
___ Service frequency
___ Transit time
___ Timely notification of service delays
___ Invoicing accuracy
___ Other (please specify)

5.2.4 Rating Questions

With rating questions, respondents are asked to rate their answer. For example:

STATISTICS CANADA
68 SURVEY METHODS AND PRACTICES

How satisfied are you with our customer service?

P Very satisfied
P Satisfied
P Dissatisfied
P Very dissatisfied

There are several considerations when formulating a rating question. First, how many categories should
the rating scale have? It could have just two – agree, disagree – or as many as 10, ranging from 1(not at
all important) to 10 (extremely important).

Second, there is the issue of whether or not the rating scale should have a neutral alternative – for
example, neither satisfied nor dissatisfied. In the absence of a neutral alternative, the respondent is forced
to make a choice. On the other hand, when a neutral response category is offered, respondents tend to
choose it. One option is to include the neutral alternative in an interviewer-assisted questionnaire but not
offer it to the respondent. This way it is selected only if the respondent spontaneously expresses it.

The above example does not offer a neutral alternative; the one below does.

How satisfied are you with our customer service?

P Very satisfied
P Satisfied
P Neither satisfied nor dissatisfied
P Dissatisfied
P Very dissatisfied

The third consideration when formulating a rating question is whether to include a category such as ‘Do
not know/no opinion’ or ‘Not applicable’. This depends on the question being asked. For example, when
asking about a particular service that the respondent may have never used, it is necessary to include a ‘not
applicable’ category.

For each of these issues (the number of categories on rating scale, the use of a neutral alternative, the use
of ‘not applicable’), the solution depends on the survey objectives, the item to be rated, the method of
data collection and the statistical agency's own preferences.

The ranking question presented earlier in section 5.2.3 about the choice of transportation carriers can be
reformatted as a rating question as follows:

Listed below are a number of factors that influence a company's choice of a transportation
carrier. Some may be more or less important to your company than others. According to your
company's priorities, please rate the importance of each factor from 1 to 10 where 1 means ‘Not
at all important’ and 10 means ‘Extremely important’.
___ Damage-free transportation
___ Price
___ Marketing and sales representatives
___ Customer service representatives
___ Prompt resolution of service problems
___ Service consistency
___ Service frequency
___ Transit time
___ Timely notification of service delays
___ Invoicing accuracy

STATISTICS CANADA
 QUESTIONNAIRE DESIGN 69

While the rating question version would take longer to ask in an interview, it is much easier for the
respondent to rate each response category than to rank the response categories. This is particularly true for
telephone interviews.

5.3 Guidelines for Wording Survey Questions

Questions should be formed so that they are clear and meaningful to survey respondents. Survey data will
be of higher quality if respondents can easily understand the meaning of the words used. Respondents will
also be more willing and able to provide information if they clearly understand what is being asked of
them. Furthermore, it is essential that respondents understand a survey question in the way intended by
the questionnaire designer.

The wording of a question can distort survey results and provide inaccurate data if respondents:
- do not understand what the words in a question mean;
- interpret the words differently than intended;
- are unfamiliar with the concept(s) conveyed by the wording of a question.

To avoid these problems, the following sections describe some general guidelines to keep in mind.

5.3.1 Keep it Simple

The best way to communicate clearly with respondents is to use simple, everyday words and to make sure
that all terms are appropriate for the population being surveyed. The following question does not use
simple, everyday language.

Are you aware of the impending amalgamation of surrounding constituencies into the New Metro
area?

Many survey respondents may not be aware of, or understand, the meaning of the terms: impending
amalgamation, constituencies, or New Metro area.

Always consider the language skills of respondents when developing questions. Try to choose words that
are easy for everybody to understand. When surveying more educated populations such as lawyers,
teachers or other professionals, it is possible to use more complex language. The important thing is that
the language used should correspond to what is commonly understood by the target population.

In general, avoid using technical words or specialised jargon with which respondents are unfamiliar.
However, if such terms are needed, respondents should be given additional clarification or definitions.
New or complex concepts must be defined so that all respondents have the same understanding of the
question. Definitions can be included in the question, in a set of instructions for respondents that appears
elsewhere on the questionnaire or in a separate instruction booklet (although a separate booklet is less
likely to be read).

The following question does not use simple, common words.

Have you ever received a pneumococcus vaccination?

STATISTICS CANADA