Evaluating Challenges and Opportunities: A Tale of Two Programs

Every evaluation presents challenges and opportunities, offering lessons that extend beyond the scope of a single project. Such is the case with two evaluations we recently conducted for obesity-reduction programs.

Obesity has received intensive media attention as a national epidemic. As a result, a dizzying array of programs and initiatives has emerged to help communities address the problem. It’s difficult to tell what’s working, let alone which programs have the potential to create lasting change.

In our work on these projects, we addressed issues that are unique to obesity reduction programs: the particular challenges of working with adult populations around sensitive topics, and the need to demonstrate short-term results for programs that advocate a change in lifestyle and/or behavior. More importantly, these projects serve as illustrative examples for multi-site evaluations in general.

Program 1: The Healing Zone
The Healing Zone is a three-year, faith-based community health leadership program of the National Caucus and Center on Black Aged (NCBA). The project works with African American seniors to increase community awareness about the negative impact of unhealthy lifestyle practices, and to reduce obesity and risk factors leading to chronic illness or diseases such as heart disease, hypertension, stroke, and diabetes. The project began in early 2004 in four locations, adding two sites later that year.

Challenge: For a variety of complex reasons, program sites were not able to collect quantitative data to measure changes in weight, diet, and other indicators to gauge the program’s effectiveness. Thus the evaluation team relied on qualitative data to help measure effectiveness. We conducted site visits to four participating sites. During our visits, we facilitated two focus groups with program participants and one with the Senior Health Advocates, the women who led the Healing Zone program at each site. We also conducted several key informant interviews and an online survey, and obtained all program available program documents for review.

Opportunity: The evaluation revealed a lack of buy-in from the participating sites and stakeholders regarding data collection. The results pointed to opportunities to improve program management, from an evaluation perspective as well as for the program’s implementation overall. Evaluation results also helped NCBA to target program materials to the faith-based, African American communities that were the focus of the program. As a result, in future iterations of the Healing Zone, program operations will be more coordinated and relevant to the sites and intended audience, and data collection will be more streamlined.

Program 2: Shape Up…Live Well
In September 2004, CareFirst BlueCross BlueShield awarded $500,000 in grants through its “Shape Up…Live Well” Initiative. Each of the fourteen innovative and creative programs funded through the Initiative targets a particular age group: children and youth, adults, or senior citizens. The goals of grantees encompass nutrition education, health education, and strategies to increase participant levels of daily activity. While each program uses different strategies, they are all working towards the same long-term objective: to lower the incidence of obesity and obesity-related health risks among their target population.

Challenge: Although there were only fourteen grantees, no two programs were the same. The evaluation team was challenged to develop an evaluation process for each grantee that would be responsive to different grantee approaches, outcomes, and indicators. At the outset of the program, we worked with CareFirst and the grantees to align their outcomes. First we constructed a theory of change for CareFirst. Next, each grantee developed a logic model for its program. The grantee logic models were then mapped against CareFirst’s theory of change so that everyone could see how all of the grantees work together to achieve the funder’s overarching goal.

Opportunity: Once the outcomes of the grantees were aligned with CareFirst’s, the grantees were able to develop systematic data collection processes. Preliminary results show that programs targeting children have had better outcomes due to higher participation rates. Since the youth oriented programs are tied to school programs, they have had more success ensuring youth participation compared to those programs focused on serving an adult population.