California State University Long Beach

Graduate Center for Public Policy and Administration

Summer 2002, Third Session

PPA 590 WOMEN & PUBLIC POLICY

EVALUATION RESEARCH

Stages in Evaluation Research

            Evaluation research is research which is characterized by 1) a focus on a plan or program of deliberate intervention, not just a natural or accidental event; 2) an objective or goal which is considered desirable or which has some positive value; 3) a method for determining the degree to which the planned program achieves the desired objectives.

Stages in Evaluation Research

Stage I. From Efficiency to the New Deal.

            As early as the late 19^th century, researchers were doing evaluations: Frederick Taylor evaluated the effectiveness of particular approaches to work and management; Elton Mayo evaluated the effects of production policies at the Hawthorne plant on the informal organization of small work groups. However not until the 20^th century was it an accepted research technique.

            During the New Deal, government agencies and programs were created which provided social scientists excellent opportunities to use experimental evaluation research methods. Studies were made of the effects of work relief compared to direct relief, public housing projects, and the treatment of juvenile delinquents.

            The application of experimental and quasi-experimental methods yielded important methodological discoveries (e.g., the Hawthorne effect).

Stage II. Field Research 

            The experience during the previous paradigm was mostly in controlled environments. During the second paradigm, field research was born as a way of designing, planning and managing investigations in real-life settings, such as schools, work camps, street corners, homes and the battle-field. The researchers were primarily sociologists and psychologists, who evaluated programs which were meant to affect attitudes, such as improvement of morale. Field research combined both qualitative and quantitative aspects of research, and acknowledged the role of the social scientist as a participant in the social world.

Stage III. Social Experimentation.

            During the Johnson administration, initiation of Great Society programs provided unique opportunities for the application of research methods to government programs. These programs marked the first incursion of the Federal Government into the arena of social welfare, which was formerly the province of local and State authorities, if not private agencies. Almost immediately concerns arose over the integrity of these programs, their administration and effect. At first the programs were evaluated by the agencies themselves, but by 1970 Congress mandated or authorized evaluations of a wide variety of programs and gave the GAO increased responsibility for conducting program impact studies.

            For the first time evaluators realized the difficulties of conducting studies in a "turbulent environment", marked by 1) the tendency of a program to shift as it was being evaluated; 2) the fact that the program was embedded in the organizational structure and that the structure itself has an effect on program outcomes; and 3) conflicting role expectations between program administrators and program evaluators, including high turnover among program evaluators, use of part-time evaluators, the use of poorly qualified evaluators, and a general carelessness in the conduct of evaluations.

            Major social experiments were conducted, including negative income taxes, housing vouchers, and compensatory educational programs, which were seen as allowing for better planning and greater efficiency.

Stage IV. Public Policymaking.

            By the mid 1970s, evaluation research had emerged as a central component in the intergovernmental power structure and in the policymaking process in general. Congress mandated evaluation of state and local programs which received federal funds, thereby augmenting its power through constitutionally mandated powers of oversight. Federal funding for evaluation grew from nearly nothing in 1960 to $63.6 million in 1977. Additionally, this greatly increased the prevalence of evaluation research at the State and local level. Reagan's New Federalism lead to an even greater reliance on evaluation research in the policy decision process at this level, in deciding how to distribute limited funds among competing programs.

              While policymaking is still a political process, evaluation research was seen as infusing rationality into public policymaking, and as meeting the demands of decision makers at all levels for control of social programs. As part of their need for control, decisionmakers turned to social scientists for help in understanding the causes and solutions to social problems. The growth of the field of evaluation research was part of a larger trend of reliance on scientific experts, in order to base decisions on objective measures of program effectiveness. This was not the end of ideology, but the adoption of an ideology which reflected current trends in social policy making.

Stage V. The Qualitative vs. Quantitative Debate.

            Evaluation Research has changed from an activity conducted in a few situations to multiple methods applied at every level of government. It is now clearly an integral component of public administration. Past paradigms have not vanished, rather each has been incorporated into succeeding paradigms. Lately there has been some loss of confidence in the quantitative methods of evaluation research (e.g., cost benefit, cost effectiveness, and cost utility analysis), and a call for more direct involvement of the evaluator in the policy process. However, it is unlikely that either approach will eventually dominate the field completely.

WHAT IS EVALUATION?

            Evaluation is a process of judging; evaluation research uses the scientific method to control subjectivity. Both examine the relationship between variables, and they both use the same research designs and the same methods of data collection and analysis. However, the differences between evaluation research and pure research are:

What is Program Evaluation?

            Evaluation research, or program evaluation, includes activities designed to support the ongoing decision-making process. It differs from basic research in that it is applied, and its purpose is to evaluate a particular activity. Its purpose is to allow decision-makes to confront the problem of resource allocation among competing social objectives and alternative ways of meeting those goals, and to clarify whether programs should be expanded, changed, or abolished.

Steps in the Process of Evaluation

1. Identify the decision maker(s), their perceptions, how the information will be used, and deadlines.

2. Define the nature and scope of the problem as the decision-makers see it; review the history of the program and problem(s), and any past evaluations.

3. Determine what the program’s objectives are or should be, including intended benefits, how they are to be obtained, intended recipients of services, intended and unintended consequences, and whether multiple objectives conflict.

4. Specify comprehensive measures to quantify how a program’s goal is being met, including tangible measures, side-effects, and distribution.

5. Prepare a detailed plan to cover problems; assumptions; constraints; methods; resources; schedules and deadlines.

6. Assemble the evaluation team, establish lines of communication, and determine the methods of evaluation to be used, considering the level of validity, reliability, significance, efficiency, and timeliness desired.

7. Specify procedures for using the results, including the nature of the reports, who they are sent to, implementation, and follow-up.

Designs for Evaluation Research

            Evaluation research can use many types of research design. Social, political, ethical or other human considerations may impede the application of controlled experiments, and so pre-experimental or quasi-experimental designs may be the most appropriate in many instances.

Pre-Experimental. There is little control of extraneous factors.

            1) The One-Shot Case study

            2) The Pretest-Posttest design

            3) Posttest only comparison group design

Quasi-Experimental. There is statistical control of extraneous factors.

            4) Contrasted Group Design

            5) Planned Variations Design

            6) Time Series Design

            7) Control Series Design

            8) Correlational Design

            9) Panel Designs

            10) Combined Designs

Experimental Designs. Only experimental designs can control for covariation, time order, nonspuriousness, and rationale. Validity is greater since this design controls for the effects of History, maturation, experimental mortality, instrumentation, testing and statistical regression, as well as interaction effects. Experimental designs ensure greater generalizability of findings.

            11) Controlled Experiment

            12) Solomon 4-group Design

            13) Posttest only Control Group Design

            14) Time-Delayed Effect Design

            15) Factorial Designs

Why are there demands for evaluation research?

-Changes in the nature of social problems, from individual to institutional

-Changes in the structure and function of public agencies

-Increasing number of professionals trained in research

-Changes in the needs and expectation of the public for government services

Who does evaluation/uses evaluation/benefits from evaluation research?

            Program Managers                  Administrators                        Agency personnel

             Practitioners                           Recipients/Clients                   Other agencies

            Policy (law)-makers                Academics                              Consultants

            Mass media                             Interest Groups                       Outside groups

            Elected Officials                      Taxpayers                               Congress

What are the purposes of evaluation research?
OVERT:
To initiate or terminate a program
To add or drop parts of a program
To extend a program to new areas
To allocate resources among programs
To test a proposed program


COVERT:
Postpone
Duck responsibility
Public Relations
Fulfill grant requirements
Stir up debate

When is evaluation research used?

Policy research/evaluation is not utilized only at the end of the policy making process (i.e., as feedback) but in every stage:

            1. Value formation

            2. Goal setting

            3. Setting attainment criteria

            4. Identifying goal-producing activities (Policy evaluation)

            5. Program Implementation & operation

            6. Assessment of goal attainment (Program Evaluation)

Problems with evaluation research are:

1.There is generally uncertainty over policy goals, which may be diffuse, unclear, and even conflicting. How can you tell if the program works?

2.There are generally many effects: intended and unintended effects; intended and unintended groups are affected; there are intended and unintended costs; and there are intended and unintended benefits.

3.There are methodological problems involved in attempting to establish causality.

4.How can data be obtained that is unbiased? Was the record-keeping changed during the program/during the evaluation?

5.Friction with program administrators and program workers may engender resistance or even sabotage evaluation efforts.

Other Problems:

            Turbulent setting of ongoing programs           Personality differences

            Role differences (admin. vs. evaluator)           Conflicting goals, values, ideology

            Lack of clear definition of authority                Institutional histories

            Data collection                                                Changes in record keeping

            Selection of participants                                  Controls

            Feedback into program operations                  Feasibility, acceptability

THE ROLE OF EVALUATION IN PUBLIC POLICY

            There may be problems with the day-to-day operation of the evaluation, which depend greatly on the skills of the evaluator. Tensions may develop between evaluators and administrators, which require the evaluator to use many strategies. Evaluation should have a high priority in the organization and ideally be conducted by a consultant or third-party organization. Insiders have more knowledge about the organization, but may be less objective, less able to question the status quo, less able to mediate, and less able to devote time to the research. The fragmented multi-centered policy process is used by political actors to help them carry out their own goals (not the evaluator's). Evaluation information has to compete with other non-scientific factors which policymakers consider relevant in policymaking. Neither are crystal clear in pointing the way to a perfect decision. It is doubtful that evaluation impacts policy in other than an incremental fashion. It may not be used in short-term decision making; the evaluator is only an advisor and has no formal power; disagreements between evaluators and administrators may jeopardize the study; staff and resources to carry out recommendations may be lacking; political decisions may be preferred over technical decisions.

            Ethical problems may arise with respect to the privacy of the individual, and whether there is a right to know anything about the individual; maintaining the confidentiality of the information; obtaining informed consent and being sure the individual has the capacity to agree to participate in the evaluation; assuring that the data are protected.

What types of failure can evaluation research identify?

Why policies don't have intended effects:

-Not enough resources were committed

            -No administrative mechanism was set up

            -Problems are multifaceted and policies are single-minded]

            -People respond or adjust to negate the impact of policies

            -Policies have incompatible goals that conflict

            -Solutions may cost more than the problem

            -Some problems don't have solutions

            -The 'problem' changes over time

            -New problems emerge which impact the area under evaluation

Program evaluation may be a form of social criticism. It can slowly change conventional beliefs of policy-makers over time. It can open up new options in the long run and on a broader scale. Evaluation research is a way to demonstrate the legitimacy of government, since there is no profit margin as in the public sector. Evaluation research can demonstrate the efficiency, effectiveness, and social worth of government. It can use the prestige of the research method to demonstrate the value of government programs.