Evaluation
The evaluation component of the initiative will provide essential feedback on the design, implementation and results of the initiative. The PSS evaluation will be implemented in a two-phase process that will generate information and feedback at multiple levels. Central to both phases of the evaluation is the 'theory of change' which gounds the initiative. The theory of change proposes that:
- Barriers to learning can be mitigated by high-quality OST programs and other wrap-around services, and
- Services that are matched to students' unique characteristics can lead to improved developmental and academic outcomes.
Setting the stage for evaluation
Boston Beyond staff and partners engaged in a 9-month design and planning process before beginning to implement PSS in the fall of 2006. Elizabeth Reisner from Policy Studies Associates (PSA) advised the design team on evaluation issues and the creation of an evaluation plan. The plan evolved through discussions with experienced OST evaluators, conversations with key stakeholders and funders, and PSS team meetings.Key questions
Plans for data collection, information systems, analysis, and reporting will grow out of the following research questions:- How do schools and OST providers implement the PSS initiative?
- To what extent do targeted students engage in PSS-sponsored experiences that are well matched to their needs?
- How do the partners contribute to high-quality implementation?
- To what extent does the initiative achieve intended system outcomes?
- How, if at all, are targeted students affected by PSS?
- What services, activities, staffing, and other practices and resources are associated with positive student outcomes? What are the PSS-related conditions for student success?
Phase One: Implementation Analysis
During the early stages of the initiative, an implementation analysis will be conducted using both quantitative and qualitative methods over a two year period. The first stage will consist of detailed case studies of the first five sites. These case studies will describe how PSS was implemented at each site and identify which sites have implemented the initiative with sufficient adherence to the theory of change. This approach will accommodate the unique characteristics and anticipated variation across sites and allow us to learn from the sites' diversity. Data collection methods will include parent and student surveys and focus groups, interviews with school and OST program staff, attendance data, and school records. PSA began implementing the evaluation in September 2006.
Phase Two: Outcomes Evaluation
A second stage of the implementation evaluation will be conducted with the second five PSS schools in 2007 resulting in a second set of more modest set of case studies. Through these implementation studies, PSS hopes to identify the processes and actions that lead to high-quality implementation of each component of PSS.
In fall 2008, the evaluator will begin the outcome evaluation with the sites identified as having effectively implemented the initiative. Based on the theory of change, PSS predicts that sites that do not implement the initiative well cannot expect to achieve long-term outcomes. The outcome study will utilize a quasi-experimental design, which will compare outcomes for students in PSS schools with youth in similar schools that are not implementing PSS. The outcomes measured will be informed by the implementation analysis and may include school and OST program engagement, positive interactions with peers and adults, and academic measures (e.g., grade promotion, test scores). PSS anticipates having preliminary results in the summer of 2007, with final results in the fall of 2009.
