How will we evaluate whether or not the program is meeting the needs of the children and of the community?
Evaluating the efficacy of afterschool programs is a complex issue, as complex as evaluating schools themselves. Researchers and policy-makers are currently at work developing and refining new tools with which program administrators, funders and other stakeholders can set goals and assess outcomes for afterschool programs. The issue has many facets, and the field is constantly changing. Below, we offer you a framework for evaluating your program, and a list of resources to further your understanding of the evaluation process.
Begin with clear goals.
As we discussed above, clear goals and continuous self-evaluation is an essential part of any good afterschool program. If goals are not clearly defined at the outset, it will be hard to assess what progress has been made, or whether the program is meeting stated needs. With a clear set of criteria for program success, you'll be able to tell how well you are meeting your objectives and the needs of the community. With this information you can:
- improve the quality of care and learning activities your program provides
- garner more support and funding
- make strategic plans for the future of the program
Jennifer Davis, formerly of Boston's 2-6 Afterschool Initiative, talks about the importance of evaluations.
Because each program is different, there is no template for evaluation that will suit each one. Depending on the focus of your program, you'll be looking at different variables and numbers to determine your success. For example, programs designed as safe havens for older children will want to look at statistics that monitor safety: crime in afterschool hours, drug use, victimization. On the other hand, a program that is focused on academics will look at student progress through test scores and other more traditional methods of assessment. However, all programs should consider these types of assessments as just the starting point.
Consider the whole ecology of the program in your evaluation.
Traditionally, evaluations of afterschool programs have focused on numbers, like improving test scores or lowering crime statistics. While these are critical, it's not so easy to measure the other essential parts of a good program: developing relationships between youth and adults, fostering the moral and ethical development of children, even allowing them time to play. While each program must develop its own set of specific evaluation criteria, all programs should consider the broader range of desired outcomes, especially the social and emotional development of each child. These affective components should be in the foreground as you set your goals, structure your program, and finally assess your success. All evaluations, whether conducted by program staff or outside evaluators, must incorporate multiple measures of success.
Solicit feedback, gather data, ask questions.
Feedback from the community is one important tool in measuring your program's efficacy. Solicit feedback through surveys and meetings, collecting quantitative data that will help you determine the program's strengths and the areas where change is needed. Have conversations with the children and youth in your program to make sure they feel safe, engaged, excited, and happy where they are.
More formally, you should look at the components of your program and consider whether the quality of each activity meets your stated goals. To evaluate specific activities, you might begin with an outline like the one below, suggested by Karen Walker, director of community studies at Public/Private Ventures (P/PV), which focuses on work in youth development and community initiatives:
Examine how well the activities are structured and managed by staff:
- Do staff show up on time?
- How do they handle disputes among youth?
- Do the activity's day-to-day tasks address the activity's stated goals?
Examine relationships between youth and staff:
- How responsive are the youth to the staff's directions?
- How effective are staff in providing support to youth in accomplishing their tasks?
- What is the emotional tenor of the relationship?
Examine the activity's level of challenge:
Do youth report that they are challenged?
Do staff monitor youth's level of frustration and provide clear direction when frustration mounts?
According to Karen Walker, it's important to make assessments by observing an activity and asking youth about their experience of it through short surveys or focus groups. As your program evolves, training in methods of evaluation should become part of staff development. As your staff receives more training, resources, and guidance in higher-level evaluation skills, and uses these to refine the program itself, the quality of your program will continue to improve.
For a full interview with Walker, see "The Evaluation Exchange," the newsletter of the Harvard Family Research Project. The newsletter also offers a list of programs and agencies nationwide that are engaged in important evaluations of various issues critical to afterschool programs. You can view this list, called "Evaluations to Watch" on their Web site. Also check out Concept to Classroom's "Making Family and Community Connections" workshop, in which some of the Harvard Family Research Project personnel served as workshop experts.
Ellen Gannett responds to the following question:
Q: How can staff assess whether or not they are doing a good job?
Adriana de Kanter La Perla responds to the following question:
Q: What is the relationship between program evaluation and funding?