Research on the Stages of Implementation
It appears that most of what is known about implementation of evidence-based practices and programs is known at the exploration (e.g., Rogers, 1995) and initial implementation stages (e.g., Leschied & Cunningham, 2002; Washington State Institute for Public Policy, 2002). A test of evidence-based practice or program effectiveness at implementation sites should occur only after they are fully operational, that is, at the point where the interventions and the systems supporting those interventions within an agency are well integrated and have a chance to be fully implemented. After analyzing the apparent failure of a program, Gilliam, Ripple, Zigler, & Leiter (2000) concluded that, “Outcome evaluations should not be attempted until well after quality and participation have been maximized and documented in a process evaluation. Although outcome data can determine the effectiveness of a program, process data determine whether a program exists in the first place.” (p. 56). While we did not systematically assess the timing variable, our impression was that most evaluations of attempted program implementations occur during the initial implementation stage, not the full operation stage. Thus, evaluations of newly implemented programs may result in poor results, not because the program at an implementation site is ineffective, but because the results at the implementation site were assessed before the program was completely implemented and fully operational.
Contribution of Implementation Factors Across Stages
Research on the stages on implementation is rare, especially research that evaluates the relative contributions of implementation factors across stages. In one well-designed study, McCormick, Steckler, & McLeroy (1995) randomly assigned school districts to experimental or control conditions. All districts were provided with a choice of middle school tobacco prevention curricula. Health education teachers and administrators in the experimental school districts also received in-depth training on the curriculum. In their analysis of the results, the authors found that smaller school districts (smaller numbers of teachers, less bureaucratic administrations) were more likely to decide to adopt a curriculum at the conclusion of the exploration stage. However, during the initial implementation stage, larger school districts (more resources, greater flexibility) were more likely to implement more of the curriculum. A positive organizational climate (job satisfaction, perceived risk taking, managing conflict, involvement in decision making) was associated with both the adoption decision and with the extent of implementation. However, they found no carry-over effects. That is, events measured during the exploration stage did not affect outcomes during the implementation stage.
Implementation Variables:
Complex Interactive Factors
The complexity of implementation variables was captured by Panzano et al., (in press). These researchers conducted an evaluation of 91 behavioral healthcare organizations that adopted or considered adopting one or more evidence-based practices and programs: Cluster-Based Planning (a consumer classification scheme; N=23); Multisystemic Therapy (home-based treatment; N=16); Ohio Medication Algorithms Project (medication management for persons with serious mental illness; N=15); and Integrated Dual Disorder Treatment (for consumers with mental illness and substance abuse problems; N=37). A range of interview, survey, and implementation outcome data were included in a longitudinal design that allowed the authors to relate the data from earlier stages to later stages. The authors tested the usefulness of four conceptual models: Model 1: the adoption decision; Model 2: multi-level model of implementation success; Model 3: cross-phase effects on implementation outcomes; Model 4: effects of implementation variables on outcomes over time.
The results indicated that out of the original 91 organizations, 50 decided to adopt an evidence-based practice or program. During the exploration stage (Model 1), perceived risk discriminated the adopters from the non-adopters (also see Anderson, & Narasimhan, 1979, for measures of risk assessment). Risk was seen as lower when the organization staff members felt they could manage the risks involved in implementation, when management support was high, and resources were dedicated specifically to implementation. Thus, during the exploration stage, perceived risk discriminated the adopters from the non-adopters.
During the initial implementation stage (Model 2), positive consumer outcomes were positively related to fidelity (conversely, “reinvention” was associated with poorer outcomes) and positively related to assimilation into the agency (making the new program a permanent part of ongoing operations). At the next level of analysis, assimilation was related to quality of the communication between the purveyor and the organization, the extent to which the organization was seen as having a learning culture and a centralized decision making structure, the availability of dedicated resources, and the extent to which implementation was seen as relatively easy and as compatible with the organization’s treatment philosophy. Overall implementation effectiveness was positively related to having a system in place for monitoring implementation progress, access to technical assistance, the perceived ability of the organization to manage risks, and belief in the scientific evidence in support of the program. Overall implementation effectiveness was negatively related to the extent to which the program had been modified from its prescribed form. Thus, during the initial implementation stage, implementation success was associated with a range of contextual, organizational, and purveyor variables and with fidelity to the evidence-based practice or program as described.
In the next two analyses, Panzano et al., (in press) examined the relationships among variables across stages. They found that later assimilation and positive implementation outcomes were higher when, during the exploration stage, the advantages of the program were seen as outweighing the disadvantages, staff had high expectations of the benefits of the program for consumers, the organization staff felt they had a good relationship with the purveyor, and the outcomes of implementation were demonstrable. In addition, objective decision making strategies that involved staff members, good information about the intervention, and organizational leadership support during the exploration stage were positively related to assimilation during the implementation stage. Thus, the methods used to consider adoption appear to have an impact on the later success of implementation (Model 3).
Another interesting analysis indicated that proximal factors exerted greater influence on current outcomes (Model 4), a conclusion similar to the one reached by McCormick et al., (1995). Top management support and access to dedicated resources during the exploration stage were important to the adoption decision but were not related to later implementation outcomes. However, top management support and access to dedicated resources during the initial implementation stage were directly related to implementation outcomes. Similarly, access to technical assistance during the exploration stage was related to 3 of the 7 later implementation outcomes while access to technical assistance during the initial implementation stage was related to all 7 outcomes. Thus, implementation seems to require a sustained effort in order to produce desired outcomes.
The studies by McCormick et al., (1995) and Panzano et al., (in press) offer insight into the complex interactive factors that seem to be important within the early stages of implementation and how those factors may interact across stages and time. Panzano and her colleagues also provide a model for how to do longitudinal, integrative research across the stages of implementation.
