696 RESEARCH METHODS

INSTRUMENTATION: QUESTIONNAIRES

Step One: Create the Conceptual Model
Step Two: Produce the Questionnaire
Write the questions
Closed-ended responses
Open-ended responses
Question Wording and Response Wording
Question order
Step Three: Pre-test and Pilot test
 

Introduction

    The questionnaire translates the research objective into specific questions. The answers to those questions provide the data for testing the research hypothesis. Questions must also interest the respondents enough that they will provide the information.

    The first step in designing a questionnaire is to create a conceptual model. This includes specifying the research problem, the purpose of the research, the research design, the variables and hypothesis, and operational definitions and valid and reliable measures of the variables, as well as the intended population, and the plans for data analysis.

    The second step is to produce the questionnaire. This includes writing the introduction, the statement of informed consent, and the questions and responses, as well as designing the overall format ("look and feel") of the questionnaire.

    The third step is to pre-test the questionnaire, to revise, and to conduct a pilot test of how the questionnaire will be used.
 
 

I. Step One: Create the Conceptual Model

Research Problem:
Some supervisors are not making good decisions on the job
Research Purpose:
Determine which factors lead to better decision-making on the job
Research Design:
Cross-sectional survey of supervisors
Population:
Currently employed supervisors
Data Analysis:
Indicate the level of measurement needed for each question
Hypotheses:
The more training in statistics, the better the decision-making
The more recent the training in statistics, the better the decision-making
The quality of decision making will vary across departments
Variables:
Decision-making quality (ratio level)
Amount of training in statistics (ratio level)
Recency of training in statistics (ratio level)
Department of employment (nominal level)

 
Variable Name: Operational definition:  Measurement indicator
Decision-making quality The extent to which supervisors' personnel decisions are questioned by superiors or subordinates (percentage) Q1. How many personnel decisions did you make between January 1 and June 30? ___ 

Q2. How many of the personnel decisions you made between January 1 and June 30 were questioned by either superiors or subordinates? ___
Amount of Statistics 

training

 The amount of training in statistics Q3. How many statistics courses did you complete in college? ___
Recency of training in statistics The recency of training in statistics Q4. In what month and year did you complete your last college statistics course? 

month___ year___
Employment  Department in the organization in which the supervisor is employed Q5. In which department are you currently employed? 

_____Accounting 

_____Sales 

_____Engineering
 
 

II. Step Two: Produce the Questionnaire

A. Write the introduction
The introduction:
explains the nature and purpose of the survey
names the person to contact in case of questions
says whether the questionnaire is anonymous
says whether the information will be kept confidential
enumerates the respondent's rights
stresses the importance of the study
thanks the respondent for participating

 

B. Write the questions

1. Bring together key actors in the question and response writing session, to make sure that the type of data that is needed will be produced by the questionnaire.

2. There must be at least one questions for every variable of importance in the study; and there may be more than one question for each variable. For example, to measure a worker's job satisfaction, there may be three questions: satisfaction with working conditions, satisfaction with pay, and satisfaction with promotional opportunities.

3. Questions must be designed to elicit data measured at the desired level--nominal, ordinal, interval or ratio--so that the appropriate statistics may be applied and the questions posed by the research may be adequately answered.
 

4. Questions can elicit also different types of information: facts, values, and knowledge.

 
Factual questions: age, sex, marital status, annual income, job title, education;
Value questions: attitudes, opinions, expectations;
Knowledge questions: tests, etc.

5. Questions can have two general types of response options: open-ended or closed-ended.

a. Closed-ended response options provide respondents with a limited set of response choices, such as Yes or No; low, medium or high; high school, bachelor, masters; etc.
 

Example 1: "Why did you leave your last job?" (Check the one most important reason)
_____Better pay
_____Better opportunities
_____Fired
_____Personal reasons.

 
Example 2: "What is your marital status?" (Check the box that applies)
Single, never married
Married
Divorced
Separated
Widowed
Other:_____

 

Example 3: "How satisfied are you with your job?" (Circle the number that represents your response)

Very dissatisfied    Dissatisfied    Neutral    Satisfied    Very satisfied
            1                        2                3                4                    5
 

    Matrix questions are merely a series of questions that share the same set of closed-ended response options. For example:
 
Question Very Dissatisfied Dissatisfied Satisfied Very Satisfied
1. How satisfied are you with your working conditions? 1 2 3 4
2. How satisfied are you with your pay? 1 2 3 4
3. How satisfied are you with your supervisor? 1 2 3 4
 
 

    Closed ended response options must contain every possible response, including some response category for those who don't want any of the other choices. Such response categories may be "other," "don't know," "not sure," "undecided," or "not applicable."

    Closed-ended response categories are useful for gathering quantitative, easily coded and analyzed data. They are most suitable for mass surveys, especially for mailed or telephone surveys. The reliability and validity of closed-ended response categories depends on the completeness of the response categories provided.

    Other types of closed-ended response categories include rank ordering or assigning percentages. Rank order responses ask the respondent to put a list of responses into an order of preference. For example, rank the following issues in order of their importance for your community (1 being most important and 7 being least important): crime; pollution; schools; employment; recreation; immigration; taxes.

    Assigning percentage responses ask the respondent to divide 100% of something into a list of categories. For example, "What percentage of your working day do you spend on each of the following activities: planning; organizing; supervising; reporting."

b. Open-ended response options provide respondents with the opportunity to respond to the question in whatever way they see fit.
 

For example, "Why did you leave your last job?"
(followed by blank space for the respondent to write in the response)

 
 
Advantages of open-ended response options:
allow the respondents to suggest a range of possibilities unknown to the researchers;
avoid the bias of a list of response possibilities;
generate rich details, key words and phrases;
are sometimes easier (e.g., write in your state rather than pick from a list of all states);
are often used in exploratory research or in early stages of a project;
can generate ideas for future closed-ended response options;
can gather data to explain responses to closed-ended response options (if you said yes, why?);
allows the respondent to have input to the researcher or the sponsor of the research;
can promote a positive feeling for the respondent, that the input is valuable;

    Open-ended response categories are useful for gathering qualitative data. They are most suitable for elite or expert surveys, especially for mailed or in-person surveys. The reliability and validity of open-ended response categories depends on whether all the respondents interpret and respond to the question in the same way as the researcher(s) intended
 

Problems with open-ended response categories:
need careful pre-testing;
data may be more difficult to code and analyze;
may be left unanswered more often than closed-ended response categories;
requires motivation on the part of the respondent;
requires well-trained interviewers for in-person interviews;
takes more time for analysis for large samples.

Open or closed?
 
Closed-ended Open-ended
Researcher's objectives Confirm or deny the researcher's hypothesis learn about how respondents come to possess a particular point of view
Level of Information needed Never know how the answer was arrived at Obvious shortcomings in the respondent's knowledge become apparent
Influence on responses Response choices may influence respondents who were previously unaware No fixed response alternatives can jog respondent's memory; may produce less information but attitude formation is not affected by the response options
Amount of effort required Easy to reply with few communication problems More frequent refusals to answers or "don't know" responses are possible? Also a chance to educate the respondent.
 
 

6. Two other types of questions are filter and contingency questions.

    Filter questions have closed-ended response options that divide the sample into subgroups. They are also called screening questions. For example, "Do you own or rent your home?"

    Contingency questions are questions that are only to be answered by some subgroup(s) of respondents. For example, "If you own your home, how long have you owned it?" Contingency questions may have either closed-ended or open-ended response options.

    Filter and contingency question combinations are best used either on telephone surveys. They are more difficult to implement on mailed surveys because they may be easily misinterpreted by respondents. And they require skilled interviewers to implement for in-person interviews.

    Contingency questions are a trade-off between becoming too confusing because of skipping around, and having all respondents answer all questions, even though some may be irrelevant to some of them. Answering all questions increases data processing costs and makes data analysis more complex since some answers will have to be ignored.
 

C. Question Wording and Response Wording

1. Questions need to be clear, specific, and short
2. Responses must fit the question
3. Questions can only ask one thing; divide compound questions into two
4. Responses need to elicit data measured on the appropriate scale
5. Endeavor to lessen discomfort, harm, or embarrassment to the respondent
6. Respondents tend to choose socially accepted answers
7. Include equal numbers of positive and negative options on rating scales
8. Avoided "loaded" or "hot button" words
9. Avoid acronyms, abbreviations, jargon, etc.
10. Avoid questions with conjunctions (and, but, or)
 

D. Order the questions

1. Include clear directions on how to answer questions; whenever the type of response category changes, be sure to include relevant directions.

2. Questions that establish eligibility

    Put first any questions that establish whether the respondent is eligible for the study. For example, if only employees who are supervisors are intended to be in the sample, the first question should elicit that information. These are a type of filter or screening question. If the respondent is not eligible, indicate what should be done with the questionnaire.

3. Put relevant and easy to answer questions first, to build interest and rapport with the respondent

4. Group major questions into sets by subject or issue. Provide transition statements to orient the respondent when the issue or subject changes.

5. "Response Set": avoid putting many questions together which the respondent is likely to answer all the same way. For example, ten questions on the environment that are all worded positively toward conservation. Respondents would be likely to either answer all the questions as "agree" or "disagree." Phrase some questions positively, and some questions negatively, in order to combat the effects of "response set."

6. Put demographic or personal information questions at the end, when the respondent is most likely to answer them. This is called a funnel sequence, with broader and more general questions first, narrowing to more sensitive or open-ended questions at the end.

7. Put a brief "thanks" at the end of the questions

8. Include instructions on when and where to return the questionnaire if applicable.
 

III. Step Three: Pre-test and Pilot test

A. Pre-test the questionnaire

1. Show it to colleagues and key decision-makers. Are the questions and responses valid and reliable? Are they appropriate? Are they both necessary and sufficient?

2. Administer the questionnaire to a small number of people who resemble or are drawn from the population of interest, but it does not have to be a random sample. Measure how much time it takes to complete each questionnaire. Debrief the respondents (and the interviewers if applicable) after they complete the questionnaire. Analyze the information provided to clarify directions, question wording, or response categories where necessary. Revise as needed.
 

B. Pilot Study

A pilot study is a dress referral of the full project, including the questionnaire, the interviewers, and all other aspects. Often a sample of 30-50 responses are obtained, coded, and analyzed. Questions that are not providing useful data are discarded, and the final revisions of the questionnaire are made.