Phone Survey
Content
It All Adds Up to Cleaner Air Evaluation Questionnaire Annotated Format
Community partners in the It All Adds Up to Cleaner Air initiative are encouraged to use the following questionnaire as the starting point for preparing pre-campaign and post-campaign evaluation surveys. --The questions are aligned with the It All Adds Up to Cleaner Air objectives of increasing awareness and knowledge of transportation choices that contribute to cleaner air and a reduction in traffic congestion. --Community partners will be encouraged to share the results of their evaluation research with one another. Comparisons will be facilitated where the same questions have been used. --Using the standardized questions saves the time and expense of question development. Questions may be added or deleted to match the specific objectives of your program, but whenever possible, the question order, question wording, and answer categories should be preserved to permit meaningful comparisons of results across It All Adds Up to Cleaner Air programs. This annotated version of the questionnaire provides a brief rationale for each question on the survey. Words in ALL CAPS are instructions to interviewers and are not read aloud to the respondent. Record respondent answers by circling the number next to the answer category. In this format, interviewers circle the number of the response. Other formats (i.e., self-administered questionnaire and Web based surveys) will use check boxes.
2
Start time: ____________ INTRODUCTION: Hello. This is ___________ calling from ____________________. We are conducting an opinion survey on quality-of-life issues. The survey will only take ten minutes to complete. The name of the interviewing service is typically identified in the introduction, not the sponsor of the survey. Identifying the sponsor of the survey tips off the respondent to the subject matter and may influence responses. Some agencies may require the introduction to provide the respondent with more detailed information about the survey such as: --The survey is sponsored by XXX agency. --The findings will be used to improve communications on public policy issues. --Participation in the survey is voluntary. --The survey takes about ten minutes to complete. Even if not required, interviewers should be provided with this information and be allowed to share it, as necessary, to gain the cooperation of respondents. May I please speak with the youngest (ROTATE: man/woman) in the household who is a licensed driver and is 18 years of age or older? (IF NOT AVAILABLE, ARRANGE FOR CALLBACK.) The recommendation is to focus the evaluation survey on the adult population, age 18 and older. While many young people start driving at 16 years of age, their life experiences and responsibilities are quite different from adults. Furthermore, parental permission may be necessary for surveys that include respondents under the age of 18 years. If a communications program is targeted at new drivers, then different approaches need to be considered. Younger people are more difficult to reach, so the introduction asks for the youngest adult licensed driver to increase the opportunities of interviewing younger adults. By rotating whether a man or woman is asked for first, a mix of men and women are interviewed.
3
If the youngest adult licensed driver in the household is not at home at the time of the call, the interviewer is instructed to interview the youngest adult licensed driver at home. The respondent selection procedures can be more rigorous or less rigorous than the approach suggested in this introduction. The tradeoff is time and the cost of reaching a representative sample of your audience of interest. The Evaluation section in the It All Adds Up to Cleaner Air Resource Toolkit provides additional guidance on respondent selection and callback procedures. (IF NECESSARY: This is an opinion survey. We have nothing to sell.) GEOGRAPHIC SCREENING QUESTIONS Counties, zip codes, and towns may be used to define the geographic area of interest for the program. Proper geographic alignment of the audience exposed to your program messages and your survey sample is key to the evaluation process. In developing your program strategies, clearly define the geographic boundaries of your target audience. Focus your sampling on these areas. Most purchased samples of telephone numbers are selected based on desired geographic characteristics, so this screening question typically excludes less than 5% of households. This screening question should not be necessary for statewide telephone surveys because, in most cases, area codes do not cross state lines. 1. What is the five-digit zip code for your home address? 1 2 3 98 99 XXXXX XXXXX XXXXX OTHER ZIP CODE NOT LISTED—TERMINATE DON’T KNOW—TERMINATE
4 A. In what city, town, or municipality do you live?
1 2 98 99
XXXX XXXX OTHER CITY OR TOWN NOT LISTED—TERMINATE DON’T KNOW—TERMINATE
B. In which county do you live? Add pre-codes for geographic areas of interest. Zip code ranges may be used in some cases. It may be easier for the interviewer to directly enter the zip code rather than checking a pre-code. Zip code can be a key variable for weighting or classification purposes. The 2000 Census provides its data by zip code area, permitting comparisons of survey results to recent population statistics. Provide a TERMINATE instruction for out-of-area respondents.
1 2 98 99
2.
XXXX County XXXX County OTHER COUNTY NOT LISTED—TERMINATE DON’T KNOW—TERMINATE
Are you a licensed driver 18 years of age or older?
1 YES (CONTINUE) 2 NO (REPEAT INTRODUCTION TO IDENTIFY AN ADULT LICENSED DRIVER) 3 DON’T KNOW/REFUSED (REPEAT INTRODUCTION TO IDENTIFY AN ADULT LICENSED DRIVER)
This question double-checks the qualification in the introduction for a licensed driver. Screening for licensed drivers is recommended because It All Adds Up to Cleaner Air messages are typically targeted at drivers. People who do not have a driver’s license are not in a position to change their driving habits and are not a typical target for the It All Adds Up to Cleaner Air initiative. In 2000, there were 191 million licensed drivers in the United States. This is 88% of the driving age population. (2000 Our Nation’s Highways).
5 3. In a typical week, about how many times do you drive a private vehicle? Count each time you get behind the wheel and drive someplace. Do you typically drive a private vehicle zero to ten times, 11 to 20 times, 21 to 30 times, or more than 30 times a week? (IF NECESSARY: By private vehicle, I mean a vehicle owned, rented, or leased by an individual rather than by an employer or company.)
1 2 3 4 5
0 to 10 TIMES PER WEEK (TERMINATE) 11 to 20 TIMES PER WEEK (CONTINUE) 21 TO 30 TIMES PER WEEK (CONTINUE) MORE THAN 30 TIMES PER WEEK (CONTINUE) DON’T KNOW/REFUSED (TERMINATE)
This question serves as both a screening question and collects classification information. Typically, It All Adds Up to Cleaner Air messages are targeted at persons who drive more than 10 times per week. The messages will be less relevant to non-drivers and infrequent drivers. (Most licensed drivers drive more than 10 trips per week.) The decision to use “how many times do you drive” is consistent with the definition of “person trip” used in transportation studies. The National Personal Transportation Survey (1995) showed an average of 4.3 vehicular person trips per day or 30 trips per week. (People over the age of 65 average 3.0 trips per day.) This question is not designed to be a sensitive measure of driving activity or miles driven. However, the answers can be used to classify drivers into light, medium, and heavy driver categories.
The first three screening questions taken together will help focus the survey on the target audience (active licensed adult drivers in your program area). The impact of screening will depend on the source of the list used for the sample. A sample of licensed drivers age 18 and older with listed numbers in the region of interest will be more efficient than a random-digit-dialing approach designed to include unlisted numbers. Refer to the Evaluation section of the Resource Toolkit for more information on sampling issues. In some cases, the audience may need to be further focused. For example, if a communications program limits itself to newspapers (stories, editorials, or ads), consider interviewing only newspaper readers.
6 4. Before this interview, have you ever seen or heard this phrase: It All Adds Up to Cleaner Air?
1 YES—ASK Q 5 2 NO—SKIP TO Q 6 3 DON’T KNOW/REFUSED—SKIP TO Q 6
If your program is using its own tag line combined with It All Adds Up to Cleaner Air, this question should be edited to refer to the combined tag line.
This is the basic aided awareness question for the It All Adds Up to Cleaner Air tag line. While awareness of this tag line does not demonstrate whether or not the messages have been communicated, recognition of the tag line is a key indicator of the visibility of the communications efforts and how well the program has been branded. On the pre-program baseline research, a good number of respondents will say they have heard of It All Adds Up to Cleaner Air. They may have actually seen or heard of it in national media or in other markets, or they may give a false positive response (say they heard of it when they did not). These respondents are not necessarily lying; the phrase simply may sound familiar to them. The percent of “yes” responses on the pre-program survey may range from 15% to 55% or higher. The test is whether or not your communications program leads to a significant increase in awareness of the tag line from the time of the baseline survey to the tracking survey. The “ROTATE ORDER” instruction means that the interviewer changes the order in which the answer categories are read from one interview to the next. This is done because the order in which the items are read can make a difference in the response. Rotating the items controls for this. On paper and pencil questionnaires, a supervisor marks the starting position on each question. Computerassisted telephone interviewing systems can do this automatically.
7
IF “YES” ON Q. 4, ASK: 5. Do you recall or not recall seeing or hearing It All Adds Up to Cleaner Air: YES, RECALL 1 1 1 1 1 1 NO, DON’T RECALL 2 2 2 2 2 2 DON’T KNOW 3 3 3 3 3 3
On television? On the radio? In a newspaper or magazine? On a billboard or poster? In something you received in the mail? In a presentation or exhibit?
This question provides an additional measure of the visibility of the campaign and indicates which media were most effective. If your campaign uses some other communications media (door-to-door visits, neighborhood watch meetings, etc.), they can be added to this list. This question is asked only of people who say they have heard of It All Adds Up to Cleaner Air on question 4. Others skip to question 6. The recommended format for this question is aided yes/no. Another approach is to ask an unaided, open-end question with answer categories provided (Where do you recall seeing or hearing It All Adds Up to Cleaner Air?). The aided approach provides a more focused review of relevant media and requires less interviewing time than an open-end approach. Do not rotate the order of these items. Keep the more popular media at the top of the list. To keep the question simple and easy to answer, the suggested answer categories do not distinguish between advertising and editorial placements. In some cases, such a distinction may be important and the question wording would need to be revised.
8 6. Do you think It All Adds Up to Cleaner Air is about: (READ ANSWER CATEGORIES. ROTATE ORDER OF FIRST 3 ANSWER CATEGORIES)
1 How to calculate the air quality index, 2 Things people can do to improve air quality and reduce traffic congestion, 3 How to change filters on heating and air conditioning systems, 4 Or do you have no idea?
All respondents are asked this question, even if they have not heard of the tag line. This is an “aided proven recall” question. Naturally, people can guess at the answer and have a good chance of getting the right answer. However, the “have no idea” answer category attracts many people who are unsure of the tag line’s meaning. An effective communications program will significantly increase the percentage of people who select that It All Adds Up to Cleaner Air is about “things people can do to improve air quality and reduce traffic congestion.” In the analysis, you can look at how well the people who haven’t heard the message do at “guessing” the correct answer. This provides an indication of how well the tag line describes the program and makes sense to the target audience. 7. How important to you personally is each of the following issues? In your opinion, is INSERT ITEM a very important, somewhat important, not very important, or not at all important issue?
(ROTATE ORDER) Air quality Traffic congestion Drinking water quality Recycling
Very 1 1 1 1
Important SomeNot what very 2 3 2 3 2 3 2 3
Not at All 4 4 4 4
DON’T KNOW 5 5 5 5
This question provides a general assessment of the importance of clean air and traffic congestion issues to the adult driver. Each of these items can be used as a stand-alone measure of a respondent’s mindset. They can also be used in the analysis to compare and contrast people who answer “very important” to people who see each issue as less important. To provide additional context, they are asked along with items about drinking water and recycling. Do not expect a lot of people to say these items are “not very” or “not at all” important.” The challenge is to increase
9 the percent of “very important” responses even if air quality improves and traffic congestion becomes less of a problem. 8. In the past year, do you recall, or not, seeing, hearing, or reading about good reasons to do each of the following?
(ROTATE ORDER) Do multiple errands on one trip Fuel your vehicle during cooler parts of the day Make sure your vehicle’s tires are properly inflated Properly maintain your vehicle Not top off your car’s fuel tank Carpool at least once or twice a week Telecommute instead of traveling to work Take the bus or train when available Walk, in-line skate, or ride a bike instead of driving your vehicle Get traffic updates before going someplace
YES, RECALL 1 1 1 1 1 1 1 1 1 1
NO, DON’T RECALL 2 2 2 2 2 2 2 2 2 2
DON’T KNOW/ REFUSED 3 3 3 3 3 3 3 3 3 3
This question is key to evaluating which messages would benefit from further promotion. The list should be modified to reflect the messages featured in your program. The objective of the It All Adds Up to Cleaner Air program is to increase awareness of reasons for selected behaviors that will reduce traffic congestion and improve air quality as well as benefit drivers in other ways (save time, save money, reduce stress, improve health, etc.). This battery of questions focuses on awareness of the messages being promoted. This is an aided recall question. Clearly, unaided recall questions would provide better proof of exposure to these key It All Adds Up messages. However, realistically, that process would take too much time and require too many resources for coding. The key here is to compare the results of
10 tracking surveys with baseline results, looking for changes in self-reported recall. The timeframe given at the beginning of the question (“In the past year”) should correspond to when your program was conducted. Annual evaluations are typical, but in some cases six months or even the past month may be appropriate. The order in which the items are asked should be rotated to control for possible order bias. 9. A. I am going to read five benefits of reducing traffic congestion and improving the quality of our air. Which one of these benefits do you find most appealing? (READ BENEFITS. ROTATE ORDER) B. Of the remaining four benefits, which one is most appealing? (REREAD REMAINING FOUR BENEFITS IF NECESSARY). C. Of the remaining three benefits, which one is least appealing? (REREAD REMAINING THREE BENEFITS AS NECESSARY)
SELECT ONE NUMBER IN EACH COLUMN
(ROTATE ORDER) Saving money on what you spend on transportation Reducing the amount of stress in your life Reducing the amount of time you spend in traffic congestion Reducing health risks through improved air quality Reducing our nation’s dependence on foreign oil DON’T KNOW/NO OPINION
A. Most Appealing
B. Second Most Appealing
C. Least Appealing
1 2 3 4 5 6
1 2 3 4 5 6
1 2 3 4 5 6
Saving money, reducing stress, saving time, and improved health are rationales for many of the It All Adds Up to Cleaner Air messages. All of these benefits are appealing to customers. This question checks to see the relative appeal of these elements of the cleaner air story, helping you decide what messages will be most effective with your audience.
11 10. From where you live, is access to public transportation, such as a bus, train, or subway, unavailable, available but inconvenient, or available and convenient?
1 2 3 4
UNAVAILABLE AVAILABLE, BUT INCONVENIENT AVAILABLE AND CONVENIENT REFUSED/DON’T KNOW
The availability and convenience of public transportation is key to understanding personal transportation attitudes and behavior. Awareness, attitudes, and actions may vary based on access, or perceived access.
12 11. For each of the following activities, tell me if you are unwilling to do it, willing to do it, or if you already do it.
(ROTATE ORDER) Take public transportation, such as the bus, train, or subway, at least once a week Avoid topping off your vehicle’s gas tank Carpool or vanpool, at least once a week Get traffic updates before going someplace Maintain your vehicle according to the owner’s manual Walk or ride a bike to destinations rather than drive your vehicle, at least once a week Check and adjust your vehicles’ tires once a month to make sure they are properly inflated Telecommute to work at least once a month Combine your errands into one trip, instead of taking multiple trips, at least once a week Fuel your vehicle during the cooler parts of the day
Unwilling 1 1 1 1 1
Willing 2 2 2 2 2
Already Do It 3 3 3 3 3
DON’T KNOW 4 4 4 4 4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
The objective of the It All Adds Up to Cleaner Air communications program is to create an environment that supports behavior change. This question provides a basic test regarding attitude, or willingness to change, as well as a simple measure of behavior. Attitudes are indicators of the political correctness of each activity. Changes in attitude are precursors to behavior change, although not necessarily predictors of behavior change. While behavior change is the ultimate objective, avoid making behavior change the focus of your communications program evaluation, because long-term
13 behavior change is a result of much more than an effective communications program (e.g., economy and employment). 12. In general, do you favor, oppose, or have no position on:
(ROTATE ORDER) Increased public investment in mass transit? Reduced speed limits on highways? Special lanes for carpoolers? Tougher emission standards for new cars and trucks or tougher fuel standards? Mandatory state emissions inspections of vehicles? Restrictions on the number of parking spaces at office buildings?
Favor Oppose 1 1 1 1 1 1 2 2 2 2 2 2
No Position 3 3 3 3 3 3
DON’T KNOW 4 4 4 4 4 4
This question does not focus on the It All Adds Up to Cleaner Air messages. However, your local program may have specific agenda items for which the core messages will help build support. This battery of attitude questions is a convenient place to test public support for such objectives. The items in the question are only examples. Customize this question to match your local situation. Drop this question if your program objectives are covered by the other survey questions. If you keep them, keep the wording as it is, so comparisons can be made. The last few questions help in the analysis of the survey results. 13. What kind of vehicle do you drive most often: a car, a van, an S.U.V., a truck, or something else?
1 2 3 4 5 6
CAR VAN S.U.V. TRUCK SOMETHING ELSE; SPECIFY: DON’T KNOW
These are the categories now being used on national transportation studies.
14
14.
How many children age 17 or younger live in your household?
1 2 3 4 5 6 7
NONE OR ZERO ONE TWO THREE FOUR FIVE OR MORE REFUSED
The presence of children in a household increases the need for trips. This can be a useful variable for analyzing acceptance of trip chaining. However, if you need to drop questions, this is a good candidate for deletion. 15. In what year were you born? ENTER YEAR: _______________ IF REFUSED ASK: Could you tell me if you were born: (READ CATEGORIES)
1 2 3 4 5 6
Before 1924 From 1925 to 1942 From 1943 to 1960 From 1961 to 1981, or From 1982 to now? REFUSED
Age is an important criterion in evaluating how well your sample represents your target population. Age (or generation) is also typically correlated with travel behavior, willingness to change, and media usage. Asking the respondent for year of birth is the fastest way to get this information. In case of a refusal, read the categories listed. Using these classification categories supports looking at attitudes and behaviors of your target audience by generations that have meaning in advertising, marketing, and communications. -- Born 1924 and before (G.I. Generation) -- Born 1925 to 1942 (Silent Generation) -- Born 1943 to 1960 (Boomers) -- Born 1961 to 1981 (Gen Xers) -- Born 1982 and later (Millennials)
15 (See Generations, by Neil Howe and William Strauss for a discussion of these generational categories.) You may also consider using the age categories commonly used by the Census to describe the population (Under 25 years, 25-34, 35-44, 45-54, 55-64, 65-74, 75 and older). See: American FactFinder--Demos for US (http://factfinder.census.gov/bf/_lang=en_vt_name=DEC_2000_SF1_U_D P1_geo_id=01000US.html) 16. What is the last grade of school you completed? (READ CATEGORIES, IF NECESSARY)
1 HIGH SCHOOL GRADUATE/GED OR LESS 2 SOME COLLEGE/ COMMUNITY COLLEGE GRADUATE/AA: ASSOCIATE OF ARTS/ TECHNICAL SCHOOL/PROFESSIONAL BUSINESS SCHOOL 3 COLLEGE GRADUATE (BA OR BS: BACHELOR OF ARTS OR SCIENCE DEGREE)/ POST-GRADUATE DEGREE (MASTERS, PH.D., LAWYER, MEDICAL DOCTOR) 4 REFUSED
Education is an important contributor to socio-economic status and differences are often observed in attitudes and exposure to message across education categories. 17. Are you employed full-time, employed part-time, retired, a student, or are you currently unemployed? (MULTIPLE RESPONSES ACCEPTED)
1 2 3 4 5 6
FULL-TIME PART-TIME RETIRED STUDENT UNEMPLOYED DON’T KNOW/REFUSED
Multiple responses are accepted because a very busy person could have a full-time job, a part-time job, and be a student.
16 These last background questions are optional. 18. What was your total household income in 2001: (READ ANSWER CATEGORIES)
1 2 3 4 5
Under $20,000 $20,000 to $39,000 $40,000 to $69,000 Or $70,000 or more DON’T KNOW/REFUSED
In 2000, U.S. households were evenly divided across these four income categories. Results will differ by area of the country. Differences in attitudes are often observed across categories of household income. However, household income varies with age, number of people employed, education, and region of the country. Furthermore, refusals to the income question are typically above 15%, thus reducing its usefulness as an analytical question. Household income is not an essential question for this evaluation. Include the household income question if you feel the answer will be important to understanding differences in opinions across your target audience. You may want to modify the answer categories to match your market. Please update the year specified in the question to the previous year. 19. Are you of Hispanic origin? (THIS INCLUDES BEING OF SPANISH ORIGIN)
1 YES 2 NO 3 DON’T KNOW/REFUSED
Being of Hispanic or Spanish origin is a cultural classification and is selfdefined by the respondent. Note that Hispanic is not a racial classification.
17 20. What is your race? (CODE ALL THAT APPLY)
1 WHITE, CAUCASIAN 2 BLACK OR AFRICAN-AMERICAN, 3 AMERICAN INDIAN (NATIVE AMERICAN) OR ALASKA NATIVE 4 ASIAN (E.G., ASIAN INDIAN, CHINESE, FILIPINO, JAPANESE, KOREAN, VIETNAMESE) 5 NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER (E.G., SAMOAN, GUAMANIAN, OR CHAMORRO) 6 OTHER—SPECIFY:______________________ 7 DON’T KNOW, REFUSED
The Census now accepts multiple responses to the race question, recognizing the increasing number of people with complex ancestries. People who respond “mixed” should be classified as other if they do not specify the mix. RECORD WITHOUT ASKING: 21. GENDER
1 MALE 2 FEMALE
LENGTH OF INTERVIEW: ___________ MINUTES The interviewer should check the START TIME at the beginning of the interview and record how long the interview required. This information is useful in budgeting and evaluating respondent burden. RESPONDENT’S AREA CODE AND PHONE NUMBER: ____ ____ _______ The telephone number information is important to record in case there is a need to re-contact the respondent to clarify an answer or resolve a problem. However, the proper steps must be taken to preserve anonymity of response. The identity of respondents (by way of the telephone number) must not become a part of the survey database. The list of phone numbers of survey participants should be deleted as soon as the database is finalized. Any completed questionnaires should be destroyed once the data has been processed. INTERVIEWER’S NAME: _____________________________________
18
Of course, many other questions could be used to benchmark and monitor the effectiveness of your It All Adds Up communications program. These questions are designed to: --Keep the focus on your target audience, --Measure changes in awareness of the It All Adds Up to Cleaner Air theme, --Monitor exposure to your key messages, --Check on attitudes towards your messages, and --Provide basic respondent background information to help in the analysis. You may see a need for additional questions to respond to the demands of your program. Add them as necessary. Some guidelines to consider in developing your questionnaire: --Avoid redundant questions. Don’t waste respondents’ time with subtle differences in questions. --Resist the temptation to add questions about travel behavior to this survey. While such information is important, it dilutes the focus of this research on the effectiveness of the communications effort to increase awareness and inform people of ways to reduce traffic congestion and improve air quality. --Keep the survey as brief as possible. This shows respect for the time of the respondents, controls costs, and forces you to prioritize what information is most important to you. Based on pretest results, this questionnaire takes an average of ten minutes to administer over the telephone. --Test the questionnaire with ten to twenty people before interviewing a large sample. This will help you fine-tune the questionnaire instructions and do a better job of fielding the survey. --Identify who will handle the data processing and analysis before you start interviewing. Have that person review the questionnaire format before you begin. --Think through the analysis plan and report outline before interviewing begins to make certain you have the questions needed to tell your story. A short version of this survey that is appropriate for use as a selfadministered questionnaire can be found at www.italladdsup.gov/. This can be used for intercept surveys (such as in shopping malls or business districts) or with groups of people.
19
And please plan to share your survey findings with your partners in the It All Adds Up to Cleaner Air initiative. You are encouraged to submit your reports through the It All Adds Up to Cleaner Air Exchange “Demonstrating the Benefits of Your Program” group on the It All Adds Up Web site, www.italladdsup.gov/exchange.
Community partners in the It All Adds Up to Cleaner Air initiative are encouraged to use the following questionnaire as the starting point for preparing pre-campaign and post-campaign evaluation surveys. --The questions are aligned with the It All Adds Up to Cleaner Air objectives of increasing awareness and knowledge of transportation choices that contribute to cleaner air and a reduction in traffic congestion. --Community partners will be encouraged to share the results of their evaluation research with one another. Comparisons will be facilitated where the same questions have been used. --Using the standardized questions saves the time and expense of question development. Questions may be added or deleted to match the specific objectives of your program, but whenever possible, the question order, question wording, and answer categories should be preserved to permit meaningful comparisons of results across It All Adds Up to Cleaner Air programs. This annotated version of the questionnaire provides a brief rationale for each question on the survey. Words in ALL CAPS are instructions to interviewers and are not read aloud to the respondent. Record respondent answers by circling the number next to the answer category. In this format, interviewers circle the number of the response. Other formats (i.e., self-administered questionnaire and Web based surveys) will use check boxes.
2
Start time: ____________ INTRODUCTION: Hello. This is ___________ calling from ____________________. We are conducting an opinion survey on quality-of-life issues. The survey will only take ten minutes to complete. The name of the interviewing service is typically identified in the introduction, not the sponsor of the survey. Identifying the sponsor of the survey tips off the respondent to the subject matter and may influence responses. Some agencies may require the introduction to provide the respondent with more detailed information about the survey such as: --The survey is sponsored by XXX agency. --The findings will be used to improve communications on public policy issues. --Participation in the survey is voluntary. --The survey takes about ten minutes to complete. Even if not required, interviewers should be provided with this information and be allowed to share it, as necessary, to gain the cooperation of respondents. May I please speak with the youngest (ROTATE: man/woman) in the household who is a licensed driver and is 18 years of age or older? (IF NOT AVAILABLE, ARRANGE FOR CALLBACK.) The recommendation is to focus the evaluation survey on the adult population, age 18 and older. While many young people start driving at 16 years of age, their life experiences and responsibilities are quite different from adults. Furthermore, parental permission may be necessary for surveys that include respondents under the age of 18 years. If a communications program is targeted at new drivers, then different approaches need to be considered. Younger people are more difficult to reach, so the introduction asks for the youngest adult licensed driver to increase the opportunities of interviewing younger adults. By rotating whether a man or woman is asked for first, a mix of men and women are interviewed.
3
If the youngest adult licensed driver in the household is not at home at the time of the call, the interviewer is instructed to interview the youngest adult licensed driver at home. The respondent selection procedures can be more rigorous or less rigorous than the approach suggested in this introduction. The tradeoff is time and the cost of reaching a representative sample of your audience of interest. The Evaluation section in the It All Adds Up to Cleaner Air Resource Toolkit provides additional guidance on respondent selection and callback procedures. (IF NECESSARY: This is an opinion survey. We have nothing to sell.) GEOGRAPHIC SCREENING QUESTIONS Counties, zip codes, and towns may be used to define the geographic area of interest for the program. Proper geographic alignment of the audience exposed to your program messages and your survey sample is key to the evaluation process. In developing your program strategies, clearly define the geographic boundaries of your target audience. Focus your sampling on these areas. Most purchased samples of telephone numbers are selected based on desired geographic characteristics, so this screening question typically excludes less than 5% of households. This screening question should not be necessary for statewide telephone surveys because, in most cases, area codes do not cross state lines. 1. What is the five-digit zip code for your home address? 1 2 3 98 99 XXXXX XXXXX XXXXX OTHER ZIP CODE NOT LISTED—TERMINATE DON’T KNOW—TERMINATE
4 A. In what city, town, or municipality do you live?
1 2 98 99
XXXX XXXX OTHER CITY OR TOWN NOT LISTED—TERMINATE DON’T KNOW—TERMINATE
B. In which county do you live? Add pre-codes for geographic areas of interest. Zip code ranges may be used in some cases. It may be easier for the interviewer to directly enter the zip code rather than checking a pre-code. Zip code can be a key variable for weighting or classification purposes. The 2000 Census provides its data by zip code area, permitting comparisons of survey results to recent population statistics. Provide a TERMINATE instruction for out-of-area respondents.
1 2 98 99
2.
XXXX County XXXX County OTHER COUNTY NOT LISTED—TERMINATE DON’T KNOW—TERMINATE
Are you a licensed driver 18 years of age or older?
1 YES (CONTINUE) 2 NO (REPEAT INTRODUCTION TO IDENTIFY AN ADULT LICENSED DRIVER) 3 DON’T KNOW/REFUSED (REPEAT INTRODUCTION TO IDENTIFY AN ADULT LICENSED DRIVER)
This question double-checks the qualification in the introduction for a licensed driver. Screening for licensed drivers is recommended because It All Adds Up to Cleaner Air messages are typically targeted at drivers. People who do not have a driver’s license are not in a position to change their driving habits and are not a typical target for the It All Adds Up to Cleaner Air initiative. In 2000, there were 191 million licensed drivers in the United States. This is 88% of the driving age population. (2000 Our Nation’s Highways).
5 3. In a typical week, about how many times do you drive a private vehicle? Count each time you get behind the wheel and drive someplace. Do you typically drive a private vehicle zero to ten times, 11 to 20 times, 21 to 30 times, or more than 30 times a week? (IF NECESSARY: By private vehicle, I mean a vehicle owned, rented, or leased by an individual rather than by an employer or company.)
1 2 3 4 5
0 to 10 TIMES PER WEEK (TERMINATE) 11 to 20 TIMES PER WEEK (CONTINUE) 21 TO 30 TIMES PER WEEK (CONTINUE) MORE THAN 30 TIMES PER WEEK (CONTINUE) DON’T KNOW/REFUSED (TERMINATE)
This question serves as both a screening question and collects classification information. Typically, It All Adds Up to Cleaner Air messages are targeted at persons who drive more than 10 times per week. The messages will be less relevant to non-drivers and infrequent drivers. (Most licensed drivers drive more than 10 trips per week.) The decision to use “how many times do you drive” is consistent with the definition of “person trip” used in transportation studies. The National Personal Transportation Survey (1995) showed an average of 4.3 vehicular person trips per day or 30 trips per week. (People over the age of 65 average 3.0 trips per day.) This question is not designed to be a sensitive measure of driving activity or miles driven. However, the answers can be used to classify drivers into light, medium, and heavy driver categories.
The first three screening questions taken together will help focus the survey on the target audience (active licensed adult drivers in your program area). The impact of screening will depend on the source of the list used for the sample. A sample of licensed drivers age 18 and older with listed numbers in the region of interest will be more efficient than a random-digit-dialing approach designed to include unlisted numbers. Refer to the Evaluation section of the Resource Toolkit for more information on sampling issues. In some cases, the audience may need to be further focused. For example, if a communications program limits itself to newspapers (stories, editorials, or ads), consider interviewing only newspaper readers.
6 4. Before this interview, have you ever seen or heard this phrase: It All Adds Up to Cleaner Air?
1 YES—ASK Q 5 2 NO—SKIP TO Q 6 3 DON’T KNOW/REFUSED—SKIP TO Q 6
If your program is using its own tag line combined with It All Adds Up to Cleaner Air, this question should be edited to refer to the combined tag line.
This is the basic aided awareness question for the It All Adds Up to Cleaner Air tag line. While awareness of this tag line does not demonstrate whether or not the messages have been communicated, recognition of the tag line is a key indicator of the visibility of the communications efforts and how well the program has been branded. On the pre-program baseline research, a good number of respondents will say they have heard of It All Adds Up to Cleaner Air. They may have actually seen or heard of it in national media or in other markets, or they may give a false positive response (say they heard of it when they did not). These respondents are not necessarily lying; the phrase simply may sound familiar to them. The percent of “yes” responses on the pre-program survey may range from 15% to 55% or higher. The test is whether or not your communications program leads to a significant increase in awareness of the tag line from the time of the baseline survey to the tracking survey. The “ROTATE ORDER” instruction means that the interviewer changes the order in which the answer categories are read from one interview to the next. This is done because the order in which the items are read can make a difference in the response. Rotating the items controls for this. On paper and pencil questionnaires, a supervisor marks the starting position on each question. Computerassisted telephone interviewing systems can do this automatically.
7
IF “YES” ON Q. 4, ASK: 5. Do you recall or not recall seeing or hearing It All Adds Up to Cleaner Air: YES, RECALL 1 1 1 1 1 1 NO, DON’T RECALL 2 2 2 2 2 2 DON’T KNOW 3 3 3 3 3 3
On television? On the radio? In a newspaper or magazine? On a billboard or poster? In something you received in the mail? In a presentation or exhibit?
This question provides an additional measure of the visibility of the campaign and indicates which media were most effective. If your campaign uses some other communications media (door-to-door visits, neighborhood watch meetings, etc.), they can be added to this list. This question is asked only of people who say they have heard of It All Adds Up to Cleaner Air on question 4. Others skip to question 6. The recommended format for this question is aided yes/no. Another approach is to ask an unaided, open-end question with answer categories provided (Where do you recall seeing or hearing It All Adds Up to Cleaner Air?). The aided approach provides a more focused review of relevant media and requires less interviewing time than an open-end approach. Do not rotate the order of these items. Keep the more popular media at the top of the list. To keep the question simple and easy to answer, the suggested answer categories do not distinguish between advertising and editorial placements. In some cases, such a distinction may be important and the question wording would need to be revised.
8 6. Do you think It All Adds Up to Cleaner Air is about: (READ ANSWER CATEGORIES. ROTATE ORDER OF FIRST 3 ANSWER CATEGORIES)
1 How to calculate the air quality index, 2 Things people can do to improve air quality and reduce traffic congestion, 3 How to change filters on heating and air conditioning systems, 4 Or do you have no idea?
All respondents are asked this question, even if they have not heard of the tag line. This is an “aided proven recall” question. Naturally, people can guess at the answer and have a good chance of getting the right answer. However, the “have no idea” answer category attracts many people who are unsure of the tag line’s meaning. An effective communications program will significantly increase the percentage of people who select that It All Adds Up to Cleaner Air is about “things people can do to improve air quality and reduce traffic congestion.” In the analysis, you can look at how well the people who haven’t heard the message do at “guessing” the correct answer. This provides an indication of how well the tag line describes the program and makes sense to the target audience. 7. How important to you personally is each of the following issues? In your opinion, is INSERT ITEM a very important, somewhat important, not very important, or not at all important issue?
(ROTATE ORDER) Air quality Traffic congestion Drinking water quality Recycling
Very 1 1 1 1
Important SomeNot what very 2 3 2 3 2 3 2 3
Not at All 4 4 4 4
DON’T KNOW 5 5 5 5
This question provides a general assessment of the importance of clean air and traffic congestion issues to the adult driver. Each of these items can be used as a stand-alone measure of a respondent’s mindset. They can also be used in the analysis to compare and contrast people who answer “very important” to people who see each issue as less important. To provide additional context, they are asked along with items about drinking water and recycling. Do not expect a lot of people to say these items are “not very” or “not at all” important.” The challenge is to increase
9 the percent of “very important” responses even if air quality improves and traffic congestion becomes less of a problem. 8. In the past year, do you recall, or not, seeing, hearing, or reading about good reasons to do each of the following?
(ROTATE ORDER) Do multiple errands on one trip Fuel your vehicle during cooler parts of the day Make sure your vehicle’s tires are properly inflated Properly maintain your vehicle Not top off your car’s fuel tank Carpool at least once or twice a week Telecommute instead of traveling to work Take the bus or train when available Walk, in-line skate, or ride a bike instead of driving your vehicle Get traffic updates before going someplace
YES, RECALL 1 1 1 1 1 1 1 1 1 1
NO, DON’T RECALL 2 2 2 2 2 2 2 2 2 2
DON’T KNOW/ REFUSED 3 3 3 3 3 3 3 3 3 3
This question is key to evaluating which messages would benefit from further promotion. The list should be modified to reflect the messages featured in your program. The objective of the It All Adds Up to Cleaner Air program is to increase awareness of reasons for selected behaviors that will reduce traffic congestion and improve air quality as well as benefit drivers in other ways (save time, save money, reduce stress, improve health, etc.). This battery of questions focuses on awareness of the messages being promoted. This is an aided recall question. Clearly, unaided recall questions would provide better proof of exposure to these key It All Adds Up messages. However, realistically, that process would take too much time and require too many resources for coding. The key here is to compare the results of
10 tracking surveys with baseline results, looking for changes in self-reported recall. The timeframe given at the beginning of the question (“In the past year”) should correspond to when your program was conducted. Annual evaluations are typical, but in some cases six months or even the past month may be appropriate. The order in which the items are asked should be rotated to control for possible order bias. 9. A. I am going to read five benefits of reducing traffic congestion and improving the quality of our air. Which one of these benefits do you find most appealing? (READ BENEFITS. ROTATE ORDER) B. Of the remaining four benefits, which one is most appealing? (REREAD REMAINING FOUR BENEFITS IF NECESSARY). C. Of the remaining three benefits, which one is least appealing? (REREAD REMAINING THREE BENEFITS AS NECESSARY)
SELECT ONE NUMBER IN EACH COLUMN
(ROTATE ORDER) Saving money on what you spend on transportation Reducing the amount of stress in your life Reducing the amount of time you spend in traffic congestion Reducing health risks through improved air quality Reducing our nation’s dependence on foreign oil DON’T KNOW/NO OPINION
A. Most Appealing
B. Second Most Appealing
C. Least Appealing
1 2 3 4 5 6
1 2 3 4 5 6
1 2 3 4 5 6
Saving money, reducing stress, saving time, and improved health are rationales for many of the It All Adds Up to Cleaner Air messages. All of these benefits are appealing to customers. This question checks to see the relative appeal of these elements of the cleaner air story, helping you decide what messages will be most effective with your audience.
11 10. From where you live, is access to public transportation, such as a bus, train, or subway, unavailable, available but inconvenient, or available and convenient?
1 2 3 4
UNAVAILABLE AVAILABLE, BUT INCONVENIENT AVAILABLE AND CONVENIENT REFUSED/DON’T KNOW
The availability and convenience of public transportation is key to understanding personal transportation attitudes and behavior. Awareness, attitudes, and actions may vary based on access, or perceived access.
12 11. For each of the following activities, tell me if you are unwilling to do it, willing to do it, or if you already do it.
(ROTATE ORDER) Take public transportation, such as the bus, train, or subway, at least once a week Avoid topping off your vehicle’s gas tank Carpool or vanpool, at least once a week Get traffic updates before going someplace Maintain your vehicle according to the owner’s manual Walk or ride a bike to destinations rather than drive your vehicle, at least once a week Check and adjust your vehicles’ tires once a month to make sure they are properly inflated Telecommute to work at least once a month Combine your errands into one trip, instead of taking multiple trips, at least once a week Fuel your vehicle during the cooler parts of the day
Unwilling 1 1 1 1 1
Willing 2 2 2 2 2
Already Do It 3 3 3 3 3
DON’T KNOW 4 4 4 4 4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
The objective of the It All Adds Up to Cleaner Air communications program is to create an environment that supports behavior change. This question provides a basic test regarding attitude, or willingness to change, as well as a simple measure of behavior. Attitudes are indicators of the political correctness of each activity. Changes in attitude are precursors to behavior change, although not necessarily predictors of behavior change. While behavior change is the ultimate objective, avoid making behavior change the focus of your communications program evaluation, because long-term
13 behavior change is a result of much more than an effective communications program (e.g., economy and employment). 12. In general, do you favor, oppose, or have no position on:
(ROTATE ORDER) Increased public investment in mass transit? Reduced speed limits on highways? Special lanes for carpoolers? Tougher emission standards for new cars and trucks or tougher fuel standards? Mandatory state emissions inspections of vehicles? Restrictions on the number of parking spaces at office buildings?
Favor Oppose 1 1 1 1 1 1 2 2 2 2 2 2
No Position 3 3 3 3 3 3
DON’T KNOW 4 4 4 4 4 4
This question does not focus on the It All Adds Up to Cleaner Air messages. However, your local program may have specific agenda items for which the core messages will help build support. This battery of attitude questions is a convenient place to test public support for such objectives. The items in the question are only examples. Customize this question to match your local situation. Drop this question if your program objectives are covered by the other survey questions. If you keep them, keep the wording as it is, so comparisons can be made. The last few questions help in the analysis of the survey results. 13. What kind of vehicle do you drive most often: a car, a van, an S.U.V., a truck, or something else?
1 2 3 4 5 6
CAR VAN S.U.V. TRUCK SOMETHING ELSE; SPECIFY: DON’T KNOW
These are the categories now being used on national transportation studies.
14
14.
How many children age 17 or younger live in your household?
1 2 3 4 5 6 7
NONE OR ZERO ONE TWO THREE FOUR FIVE OR MORE REFUSED
The presence of children in a household increases the need for trips. This can be a useful variable for analyzing acceptance of trip chaining. However, if you need to drop questions, this is a good candidate for deletion. 15. In what year were you born? ENTER YEAR: _______________ IF REFUSED ASK: Could you tell me if you were born: (READ CATEGORIES)
1 2 3 4 5 6
Before 1924 From 1925 to 1942 From 1943 to 1960 From 1961 to 1981, or From 1982 to now? REFUSED
Age is an important criterion in evaluating how well your sample represents your target population. Age (or generation) is also typically correlated with travel behavior, willingness to change, and media usage. Asking the respondent for year of birth is the fastest way to get this information. In case of a refusal, read the categories listed. Using these classification categories supports looking at attitudes and behaviors of your target audience by generations that have meaning in advertising, marketing, and communications. -- Born 1924 and before (G.I. Generation) -- Born 1925 to 1942 (Silent Generation) -- Born 1943 to 1960 (Boomers) -- Born 1961 to 1981 (Gen Xers) -- Born 1982 and later (Millennials)
15 (See Generations, by Neil Howe and William Strauss for a discussion of these generational categories.) You may also consider using the age categories commonly used by the Census to describe the population (Under 25 years, 25-34, 35-44, 45-54, 55-64, 65-74, 75 and older). See: American FactFinder--Demos for US (http://factfinder.census.gov/bf/_lang=en_vt_name=DEC_2000_SF1_U_D P1_geo_id=01000US.html) 16. What is the last grade of school you completed? (READ CATEGORIES, IF NECESSARY)
1 HIGH SCHOOL GRADUATE/GED OR LESS 2 SOME COLLEGE/ COMMUNITY COLLEGE GRADUATE/AA: ASSOCIATE OF ARTS/ TECHNICAL SCHOOL/PROFESSIONAL BUSINESS SCHOOL 3 COLLEGE GRADUATE (BA OR BS: BACHELOR OF ARTS OR SCIENCE DEGREE)/ POST-GRADUATE DEGREE (MASTERS, PH.D., LAWYER, MEDICAL DOCTOR) 4 REFUSED
Education is an important contributor to socio-economic status and differences are often observed in attitudes and exposure to message across education categories. 17. Are you employed full-time, employed part-time, retired, a student, or are you currently unemployed? (MULTIPLE RESPONSES ACCEPTED)
1 2 3 4 5 6
FULL-TIME PART-TIME RETIRED STUDENT UNEMPLOYED DON’T KNOW/REFUSED
Multiple responses are accepted because a very busy person could have a full-time job, a part-time job, and be a student.
16 These last background questions are optional. 18. What was your total household income in 2001: (READ ANSWER CATEGORIES)
1 2 3 4 5
Under $20,000 $20,000 to $39,000 $40,000 to $69,000 Or $70,000 or more DON’T KNOW/REFUSED
In 2000, U.S. households were evenly divided across these four income categories. Results will differ by area of the country. Differences in attitudes are often observed across categories of household income. However, household income varies with age, number of people employed, education, and region of the country. Furthermore, refusals to the income question are typically above 15%, thus reducing its usefulness as an analytical question. Household income is not an essential question for this evaluation. Include the household income question if you feel the answer will be important to understanding differences in opinions across your target audience. You may want to modify the answer categories to match your market. Please update the year specified in the question to the previous year. 19. Are you of Hispanic origin? (THIS INCLUDES BEING OF SPANISH ORIGIN)
1 YES 2 NO 3 DON’T KNOW/REFUSED
Being of Hispanic or Spanish origin is a cultural classification and is selfdefined by the respondent. Note that Hispanic is not a racial classification.
17 20. What is your race? (CODE ALL THAT APPLY)
1 WHITE, CAUCASIAN 2 BLACK OR AFRICAN-AMERICAN, 3 AMERICAN INDIAN (NATIVE AMERICAN) OR ALASKA NATIVE 4 ASIAN (E.G., ASIAN INDIAN, CHINESE, FILIPINO, JAPANESE, KOREAN, VIETNAMESE) 5 NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER (E.G., SAMOAN, GUAMANIAN, OR CHAMORRO) 6 OTHER—SPECIFY:______________________ 7 DON’T KNOW, REFUSED
The Census now accepts multiple responses to the race question, recognizing the increasing number of people with complex ancestries. People who respond “mixed” should be classified as other if they do not specify the mix. RECORD WITHOUT ASKING: 21. GENDER
1 MALE 2 FEMALE
LENGTH OF INTERVIEW: ___________ MINUTES The interviewer should check the START TIME at the beginning of the interview and record how long the interview required. This information is useful in budgeting and evaluating respondent burden. RESPONDENT’S AREA CODE AND PHONE NUMBER: ____ ____ _______ The telephone number information is important to record in case there is a need to re-contact the respondent to clarify an answer or resolve a problem. However, the proper steps must be taken to preserve anonymity of response. The identity of respondents (by way of the telephone number) must not become a part of the survey database. The list of phone numbers of survey participants should be deleted as soon as the database is finalized. Any completed questionnaires should be destroyed once the data has been processed. INTERVIEWER’S NAME: _____________________________________
18
Of course, many other questions could be used to benchmark and monitor the effectiveness of your It All Adds Up communications program. These questions are designed to: --Keep the focus on your target audience, --Measure changes in awareness of the It All Adds Up to Cleaner Air theme, --Monitor exposure to your key messages, --Check on attitudes towards your messages, and --Provide basic respondent background information to help in the analysis. You may see a need for additional questions to respond to the demands of your program. Add them as necessary. Some guidelines to consider in developing your questionnaire: --Avoid redundant questions. Don’t waste respondents’ time with subtle differences in questions. --Resist the temptation to add questions about travel behavior to this survey. While such information is important, it dilutes the focus of this research on the effectiveness of the communications effort to increase awareness and inform people of ways to reduce traffic congestion and improve air quality. --Keep the survey as brief as possible. This shows respect for the time of the respondents, controls costs, and forces you to prioritize what information is most important to you. Based on pretest results, this questionnaire takes an average of ten minutes to administer over the telephone. --Test the questionnaire with ten to twenty people before interviewing a large sample. This will help you fine-tune the questionnaire instructions and do a better job of fielding the survey. --Identify who will handle the data processing and analysis before you start interviewing. Have that person review the questionnaire format before you begin. --Think through the analysis plan and report outline before interviewing begins to make certain you have the questions needed to tell your story. A short version of this survey that is appropriate for use as a selfadministered questionnaire can be found at www.italladdsup.gov/. This can be used for intercept surveys (such as in shopping malls or business districts) or with groups of people.
19
And please plan to share your survey findings with your partners in the It All Adds Up to Cleaner Air initiative. You are encouraged to submit your reports through the It All Adds Up to Cleaner Air Exchange “Demonstrating the Benefits of Your Program” group on the It All Adds Up Web site, www.italladdsup.gov/exchange.
