Tip Sheet:

5 Steps for Planning Surveys

A survey is a data collection method that allows you to systematically gather information
from a defined population. This tip sheet walks through five steps to take when planning
surveys and provides additional resources that can help you in the process.

Before you start thinking about what data you need

to collect, whether through surveys or other
Case Study
methods, it’s critical to first identify what you want
to learn and why. Take adequate time to define Caroline manages a refugee extended case
purposeful monitoring and evaluation (M&E) management (ECM) program. She wants to
questions, then identify what data you need to collect data to help make her services more
client-responsive and identifies three M&E
answer them, whether qualitative or quantitative.
questions to help improve her program:
For a detailed overview of this process, see META’s
• Do clients find our ECM services to be
Choosing Data Collection Methods e-learning
useful?
module and downloadable tools. • Do clients find our ECM services to be
culturally appropriate?
Step 1: Assess whether surveys are an • Do clients feel that significant changes
appropriate data collection method for occurred in their lives as a result of
participating in our ECM program?
answering your M&E questions.
Caroline thinks these questions will allow
Once you have defined your M&E questions and the
her to collect all of the data she needs and
data needed to answer them, decide whether a no unnecessary data. To answer them, she
survey is the best method for gathering this data. needs to know:
In making this decision, bear in mind that surveys • The percentage of current and closed-
out clients who report that the ECM
are best-suited for capturing well-defined
services have helped them achieve
information and easily observable behaviors. When
their service plan goals.
they contain closed-ended questions with defined • The percentage of current and closed-
answer choices, surveys produce quantitative data, out clients who report the ECM
which is generally easier to aggregate and analyze. services are culturally appropriate.
But note that for this data to be useful, the person • Qualitative perspectives about the
creating the survey requires good knowledge of the most significant changes that have
subject area as well as specialized skills in survey occurred in clients’ lives.
questionnaire development.
Surveys are also among relatively few data
collection methods with the potential to produce Pros Cons
generalizable data. When data is generalizable,

Self-Administered
No interviewer bias
findings from a sample of a population can, with Respondents need to
Lower cost
statistical probability, inform conclusions about the be literate
No need to arrange
full population (see below, Step 2, Sampling). If survey is online, need
interviews
internet access and
For a more detailed comparison of the pros and Can reach larger computer literacy
geographical area
cons of different data collection methods, including
surveys, focus group discussions, interviews and Higher rate of Possibility of
observation, see META’s Overview of Common Data survey completion enumerator bias

Enumerated
Collection Methods quick reference table. Enumerator can Higher cost
explore
Need to arrange, hold
respondents’
Step 2: Develop a data collection plan. answers and assist
interviews
with unfamiliar May be difficult to reach
Before writing a questionnaire, create an appropriate words/questions certain populations
plan that reflects the gender, culture, and language
considerations relevant for the respondent Type of instrument. Whether they are self-
population. A survey data collection plan can be administered or enumerated, surveys can be
developed on its own or as part of an evaluation or completed on paper, online, or on a mobile device
research plan. It should cover the following topics, such as a tablet or a phone.
including relevant roles and responsibilities for each:
Frequency of data collection. Clearly describe the
Sampling. The generalizability of survey data timeline on which data will be collected.
depends on the sample size and composition. A
sample is a subset of a whole population selected Pre-testing and piloting. See Step 5, below.
to study and inform conclusions about the
population as a whole. Unless you plan to collect Data management and quality. Plan your system for
data from all members of a target population, managing survey data, such as a well-structured MS
sampling will be involved, and this should be Excel database that includes controlled lists
carefully planned.1 (dropdown menus) for each survey answer choice.
Also plan how you will maintain data quality—
Ethics, including processes for obtaining informed ensuring data is complete, correct and consistent.
consent. 2 For more information, see META’s 5
Questions for Obtaining Meaningful Informed Data analysis, interpretation and use: Plan for analysis
Consent and sample Informed Consent Form. methods and tools that are appropriate for what you
aim to learn. Excel is well-suited to generating
Mode of administration. Surveys may be self- descriptive statistics, while products like SPSS or
administered, completed by respondents STATA enable more rigorous statistical analysis.
themselves, or enumerated, administered to Maintain the focus on purpose when planning for
respondents through interviews conducted by a
interpreting, disseminating and using your results.
person—called an enumerator—who is trained to
Plan how you will ensure findings are presented and
record responses. The table at the top right lists
shared in the right way, with the right audience, and
some of the pros and cons of these two modes.
within the right timeframe to inform decision-making.

1 For a detailed introduction to sampling strategy, see Practical Advice for Selecting Sample Sizes, Donor Committee for
Enterprise Development, May 2015. You can also find sample size calculators online to help you determine sample size.
2 For more information on ethical considerations in data collection, see Christina Clark-Kazak, ”Ethical Considerations:

Research with People in Situations of Forced Migration”, Refuge: Canada’s Journal on Refugees, Vol. 33, No. 2, 2017.
 For our case study, this mapping might start like this:
Case Study: Steps 1 - 2
M&E Do clients find our ECM services to
Caroline decides that surveys are an Question be useful?
appropriate data collection method to
answer her first two M&E questions. She can % of clients who agree that the ECM
services have helped them achieve
gather the data she needs using closed- Data
their service plan goals
ended questions, helping to facilitate Needed (disaggregated by gender and case
analysis. She can also identify an appropriate status: open or closed)
sample of current and past clients to whom
How true would you rate the
enumerators can administer the survey. following statement? “Working with
However, Caroline realizes that a survey is
Survey Caseworker [Name] helped me to
not the most appropriate method for her Question achieve my goals.”
Answer choices: “Very true,”
third M&E question: What are the most “Somewhat true,” “Not at all true.”
significant changes that have occurred in
clients’ lives as a result of participating in Question
Pick one answer choice
our ECM program? She will work to answer Type
this through individual in-depth interviews,
with more flexibility to ask open-ended Throughout this process, keep the focus on
questions and explore topics in depth. purpose. Though it can be tempting to include
“wouldn’t it be nice to know” questions in your
In developing her data collection plan,
survey, only include questions that will help you
Caroline notes that many target respondents
learn what you want to learn.
speak limited English. Therefore, she decides
to develop versions of the survey in multiple
Throughout this process, don’t write new questions
languages and work with survey enumerators
who speak clients’ native languages to unless you have to! If a survey questionnaire already
administer it. Enumerators will use mobile exists that would enable you to collect the data you
tablets to collect responses which will later need, consider using it—especially if it has been
be translated into English for analysis. validated, or tested and determined to consistently
measure what it intends to measure. However, if you
Caroline also recognizes the importance of
need to modify it, keep in mind that using questions
gender-sensitive data collection, and will
from a validated tool doesn’t mean that your tool is
ensure that her enumerator training includes
then validated.
guidance on speaking to women alone in a
comfortable, private location, among other For the questions you do develop, use clear, neutral
considerations. language. Avoid acronyms, jargon, and complicated
sentence structure. Your survey will not yield useful
results if respondents misinterpret or fail to
Step 3: Develop your survey understand what they are being asked. Language
questionnaire. 3 choice is important not only in question wording but
also in any predefined answer choices or scales.
Ensure that each question in your survey
questionnaire directly aligns with your learning The table on the following page lists some common
priorities. Map out your M&E questions, the data
issues found with survey questions, which may
needed to answer them, and the survey questions
mislead or confuse respondents and/or introduce
that will enable you to collect that data, including bias into your results.
the type or format each question will use.

3For more on writing survey questions, see Writing Good Survey Questions: Tips and Advice, University of California – San
Diego, 2013 and Piroska Bisits Bullen, “How to Write Awesome Survey Questions, Parts 1 and 2” Tools4Dev.org, May 2013.
 Issues to Avoid When Writing Survey Questions
Issue Example Revision
How useful did you find the transportation
How useful did you find the
Double-barreled orientation?
transportation orientation and the
question meetings with your caseworker? How useful did you find meetings with your
caseworker?
Rate whether you agree or disagree with Rate whether you agree or disagree with the
the following statement: following statement:
Double negative I am not inconvenienced by the time of The weekly ESL class is held at a convenient
the weekly ESL class. time.
Would you be available to attend a
Would you be available to attend a weekend
Leading question weekend job fair if it increased your
job fair?
chances of getting a job?
How would you rate your overall How would you rate your overall experience
experience interacting with Caseworker interacting with Caseworker Amina?
Unbalanced Amina? - Poor
answer scale - Good - Fair
- Very good - Good
- Excellent - Excellent

Be sure to carefully consider your respondent Step 4: Develop an adequate budget.

population(s) at this stage. A best practice is to
closely involve members of the population of Survey costs can be high relative to other data
interest in developing survey questions. 4 When collection methods. They can depend on several
possible, work with native speakers to write factors including sample size, mode of
administration (enumerated surveys are generally
questions in respondents’ language first. If this is
more expensive), choice of instrument, and
not possible, or if the survey will be delivered in
linguistic needs such as translation or
many languages, use back translation to check for
interpretation. A typical survey budget will include:
any inconsistencies in translated versions. If you
are not able to translate surveys in advance, Staff time for personnel with the necessary
engage a qualified sight translator and/or skills in survey design, sampling and data
analysis
interpreter when enumerating the survey.
Staff benefits based on total staff time needed
Also, carefully consider the sensitivity of the Travel costs if needed for enumerator
training, site visits, and dissemination
questions you include. If not strictly needed,
Costs associated with administering surveys
remove any questions that respondents may be
such as mobile devices and data plans
unwilling to answer. Where sensitive questions are Costs associated with managing and
unavoidable, it is extremely important to prioritize analyzing data, such as software licenses
respondent safety, ensure confidentiality and Data collection and project management
protect data collected, including personally supplies (pens, paper, etc.)
identifying information (PII). Finally, only include
questions if respondents can immediately access If you are considering conducting a survey as part of
research or evaluation, META’s Research/ Evaluation
the information needed to respond. Don’t expect
Budget Checklist can help guide your overall budget
people to remember too much—they may not recall
development process. Also see META’s Research/
as much as you assume.
Evaluation Time & Effort Calculator.

4 For a comprehensive guide of best practices when developing and administering a survey that involves different languages,

countries, and cultures, see the Cross-Cultural Survey Guidelines published by the University of Michigan.
Step 5: Pre-test and pilot your survey.
Case Study: Step 5
Before administering your survey at scale, dedicate
Pre-testing: Caroline selects eight people
adequate time to pre-testing and piloting. These
from her target group. She then asks each
critical steps can help you identify confusing
person to take the survey while thinking out
questions, questions that don’t yield expected loud, and she makes sure to record what
results, linguistic concerns that did not emerge they say. Caroline observes how
during translation, additional needs for enumerator enumerators ask questions and how
training, unanticipated budgetary concerns, etc. respondents complete the survey, and she
makes a note of any questions that lead to
First, pre-test your survey: Identify a group of 5-10
hesitation or misunderstanding.
people who share characteristics with the target
population. Arrange for them to complete the After going over her notes, Caroline makes
survey in your presence, one at a time. Ask them to improvements to her questionnaire
read each question out loud (or, if not literate, to wording, in collaboration with native
listen to the question aloud and discuss it with you). speakers of the target languages.
Encourage them to ask questions and provide Piloting: Caroline’s survey has a planned
feedback, taking notes throughout. sample size of 200 clients, and she plans to
work with eight enumerators. For her pilot,
Second, pilot your survey. An effective pilot she decides to start by training three
implements the entire survey data collection enumerators and administering the survey
process, but on a smaller scale than is ultimately to 50 respondents. She collects data, has
planned. This includes recruiting and training the responses translated, then enters the
enumerators (if applicable), administering the completed survey data into an MS Excel
survey to members of the target population using spreadsheet, and analyzes the data.
the chosen instrument and draft questionnaire, and
The piloting process brings several issues
managing, quality checking, and analyzing the data
to light. First, many survey respondents
collected. This enables you to collect initial data were unclear about who would see their
while identifying issues that might be costly and data, even after signing the consent form.
difficult to resolve at scale. A robust pilot process is Second, there was variation among
especially useful for more complex surveys. 5 enumerators in the way follow-up
questions were asked. Third, there was a
If you have feedback or questions on this tip higher response rate among men, even
though the sample included equal numbers
sheet, or have advice about how your
of men and women.
organization uses surveys in data collection
projects, contact [email protected]. Before administering the survey at scale,
Caroline decides to revise the explanation
of data sharing protocols in the consent
The International Rescue Committee, Inc. received $225,000 form. She also realizes that enumerator
through competitive funding through the U.S. Department of
training must be more thorough to improve
Health and Human Services, Administration for Children and
Families, Grant #90RB0051-03. The META project is financed consistency in how follow-up questions are
100% through Federal funds. The materials presented here asked. Finally, she decides that she will
are solely the responsibility of the authors and do not match enumerators and respondents by
necessarily represent the official views of the U.S. gender to help ensure female clients are
Department of Health and Human Services, Administration more comfortable participating.
for Children and Families.

5For more guidance on pre-testing and piloting surveys, see Piroska Bisits Bullen, “How to Pretest and Pilot a Survey
Questionnaire,” Tools4Dev.org, May 2013.