670 Cross-cutting techniques

PPA 670 POLICY ANALYSIS

CROSS-CUTTING METHODS

Selecting Techniques
Cross-Cutting Methods
Identifying and gathering data
Library search methods
Interviewing for policy data
Quick surveys
Assessing information quality
Basic data analysis
Communicating the analysis

SELECTING TECHNIQUES

Selecting the appropriate techniques to use in policy analysis depends on a variety of factors:: what the client wants to know; the time available; knowledge of the decision criteria; complexity of the issue; available data

Some techniques commonly used in various stages of policy analysis include:
1. Verifying, Defining, and Detailing the Problem: Back-of-the-envelope calculations; Quick decision analysis; Creation of valid operational definitions; Political analysis; Issue paper/first cut analysis

2. Establishing Evaluation Criteria: Technical feasibility studies; Economic and financial feasibility studies; Political viability studies; Administrative operability studies

3. Identifying Alternatives: Researched analysis; No-action analysis; Quick surveys; Literature reviews; Comparison of real-world experiences; Passive collection and classification; Development of typologies; Analogy, metaphor, and synectics; Brainstorming; Comparison with an ideal; Feasible manipulations; Modifying existing solutions

4. Assessing Alternative Policies: Extrapolation; Theoretical forecasting; Intuitive forecasting; Discounting; Cost/Benefit analysis; Sensitivity analysis; Allocation formulas; Quick decision analysis; Political feasibility analysis; Implementation analysis; Scenario writing

5. Displaying Alternatives and Distinguishing Among Them: Paired comparisons; Satisficing; Lexicographic ordering; Non-dominated alternatives method; Equivalent alternatives method; Standard-alternative method; Matrix display systems; Scenario writing

6. Implementing, Monitoring, and Evaluating Policies: Before-and-after comparisons; With-and-without comparisons; Actual-versus-planned performance; Experimental models; Quasi-experimental models; Cost-oriented approaches

7. Cross-Cutting Methods: Identifying and gathering data; Library search methods; Interviewing for policy data; Quick surveys; Basic data analysis; Assessing information quality; Communicating the analysis

CROSS-CUTTING METHODS

Cross-cutting methods are techniques of policy analysis that can be used at nearly any stage in the analysis. They are useful tools for the policy analyst to know how to use. They include:: Identifying and gathering data; Library search methods; Interviewing for policy data; Quick surveys; Assessing information quality; Basic data analysis; Communicating the analysis

IDENTIFYING AND GATHERING DATA

Policy analysts need to know how to search for existing information, such as: academic journal articles; archives; census records; hearings; legislative history; news media reports; past policy analyses; public agency reports; public records

People are also good sources of information, including: advocacy groups; experts; issue networks; personal contacts; professional colleagues

Even personal observation can be a source of data. Personal observation can furnish data on usage patterns, compliance patterns, insights into the problem, anecdotes, and innovative suggestions. However, observation is time consuming and may suffer from problems with accuracy, bias, limited samples, and difficult to quantify data. Observational methods include "sidewalk surveys," mechanical counting devices, measures of erosion, satellite images, etc.

Other sources of information include:: federal agencies; libraries; local agencies; non-profit agencies; private organizations; research institutes; state agencies; think tanks; universities

Policy analysts should seek information from multiple sources ("triangulation"), especially on controversial data. Problems with sources of data include:: outdated statistics; irrelevant data; misleading data; poor quality data; biased data

Looking for documents that may be helpful in doing the policy analysis is important. But three questions that must be asked are:: 1) do such documents exist?; 2) can they be obtained in a reasonable time?; 3) when is additional searching no longer worthwhile?

LIBRARY SEARCH METHODS

Libraries are excellent sources of policy-related information. To make the most of library resources, follow these strategies:

1) look up basic policy-related terms and definitions in encyclopedias, dictionaries, or a subject-related thesaurus; each policy issue area has its own terms and jargon

2) develop a list of search terms for searching computerized bibliographic data bases, electronic guides to library holdings, and Internet access;

3) identify key journals in the field and skim their table of contents for the past 1-2 years;

4) check guides to current periodicals, newspapers, news magazines, trade journals, and guides to the literature

5) check annual reviews in the policy subject area; conference proceedings on the subject; government hearings on the subject, etc.

The federal government offers a wide variety of sources:: Congressional Directory; Government Yearbooks; Guide to Federal Statistics; International Statistics; Population Reports; Statistical Abstract of the U.S.; U.S. Census; U.S. Government Printing Office catalogues; U.S. Government Manual

Many sources of legal information have bearing on policy issues:: Adjudication and case law; Agency regulations; Code of Federal Regulations; Federal and State statutes; Federal Register; Legal Periodicals; Municipal ordinances; Nexus-Lexis (on-line system); Supreme Court decisions

INTERVIEWING FOR POLICY DATA

Interviewing is typically conducted with either mass, elite, intensive, or focus group methodologies. Interviewing is typically used:: to gather historical background, context, and evolution of the policy; to gather basic facts about the problem; to assess political attitudes and resources of major players; to gather ideas about the future, trends, and forecasts; to generate additional contacts and materials (snowball technique)

Elite (specialist) interviewing is most typically used when:: it is a short-term policy project; it is on a new topic; there is a lack of existing literature; informants are reluctant to put information into writing; no quantitative data are available; it is not feasible to use hired interviewers

To set up interviews, the policy analyst usually:: arranges appointments in advance; makes formal or informal requests (letterhead, telephone); sends a reminder letter and follows up with a phone call; gives the name of a mutual friend or influential person as a reference; collects background information prior to the interview; will conduct a telephone interview if a face-to-face interview is not possible

When conducting the actual interview, it is usually accepted behavior to:: ask before using any recording device; promise anonymity and/or confidentiality of information; take notes during the interview; keep to the allotted time; thank the person for the interview; send a follow-up letter

QUICK SURVEYS

Surveys can be conducted by mail, in person, or by telephone. Survey methodology is described in many standard research texts. Cross-sectional interviews are conducted at one point in time across a wide sample of the population. Longitudinal interviews are conducted repeatedly over many time intervals (months, years, decades) with the same individuals. A comparison of the advantages and disadvantages of the most typical surveys is displayed below.

Type of Survey	Response Rate	Sample Concerns	Staffing	Other concerns
Mail	15%	May not be representative	Least staff time required	Response rate improves with gifts
Telephone	50%	Limited to those with telephones	Moderate staff time required	Short and simple questions
In-Person	75%	May be needed for less-educated	Most intensive; most supervisors	Can cover complex issues

ASSESSING INFORMATION QUALITY

When collecting information and data for policy analysis, the analyst must assess the quality of the information and data collected.

Document Analysis:: When was the document generated?; What was the original purpose of the document?; Is there an obvious bias in the document?; What is the pattern of word usage?; Does the document omit important information?; Are there errors in the document?

Assessing Interviews:: Was the information plausible?; Was the information consistent?; Does the information diverge from accepted facts?; Did the respondent report direct experience?; Did the respondent have ulterior motives?; Did the respondent operate under some constraints?; Was the respondent candid?; Did the respondent acknowledge areas of ignorance?; Was the respondent self-critical?

Data quality:: Are multiple sources of information consistent?; Were data collected independently, from separate sources?; Is the data original or re-organized?; Do the data pertain to a particular geographic locale?; Were the data collection methods systematic?; For what purpose were the data originally collected?; How old are the data? Were they affected by timing?; Was there bias or special motivation in the collection of the data?

BASIC DATA ANALYSIS

Data are not generally useful in their raw form. Instead, they must be analyzed. Data are most often analyzed using descriptive and/or inferential statistics. Descriptive statistics search for patterns in the data and look for relationships to gain insight into the problem. Inferential statistics attempt to estimate a characteristic of a population from data gathered from a sample.

Descriptive univariate statistics look for patterns in the data. They are best presented in graphical form, using frequency distributions, cumulative distributions, bar charts, histograms, pie charts, and frequency polygons. Statistics include the mean, median, and mode, as well as the range, stnadard deviation, and variance.

Descriptive bivariate statistics look for relationships in the data. They are best presented in tables, plots, scattergrams, and time series graphs. Measures of association include Lambda, Gamma, and Pearson's r.

Inferential statistics make probabilistic statements (or inferences) about a whole population based on the results obtained from a partial sample. Measures of statistical significance are used to estimate whether two groups differ from one another, or whether there is a chance that a relationship observed in a sample also exists in a population. These measures include Chi Square, Z-scores, t-tests, and F-tests.

COMMUNICATING THE ANALYSIS

Written Communication:: Work from an outline--keep separate folders for each section of the analysis; Work from goals and deadlines--generate a complete draft and then fill in the holes; Get help--with editing of rough drafts; revise for clarity; incorporate new ideas; Include a Table of Contents--sections include Executive Summary; Problem Definition; Decision Criteria; Alternatives; Comparison of Alternatives; Conclusions and Recommendations;; Use graphics--charts, graphs, flow charts, tables, maps, pictures, diagrams, drawings, etc.; Use geographic information systems (GIS)--to generate maps of data distributions; Simplicity--use the active voice for verbs; Accuracy--verify facts; triangulate; check all calculations; Documentation--note all formulas used and assumptions made; Fairness--use references and give credit to your sources of data; Neatness--use good grammar, spelling, punctuation, syntax, etc.
Oral Presentations: Know your audience; Keep it short and simple; Use visual aids and handouts; Allow time for questions, comments and criticisms