Key Ingredients in the Program Recipe: Insights from Baking for Program Evaluation
October 29, 2014 09:00 AM
By Laura R. Peck, Principal Scientist at Abt Associates Inc., Social & Economic Policy Division
I occasionally get to bake and decorate cupcakes. Though that is usually for a non-work purpose, the measuring, sifting, stirring, smelling, and tasting inevitably makes me think about my work life, too. If it were as easy to know that a half teaspoon of baking powder to cake batter was akin to a week of job club’s contributions to a job search assistance (JSA) program, then we would know that the JSA program could not rise without its component part.
As the art and science of policy experimentation evolve, government, and foundation funders increasingly demand more for their evaluation dollars. Not only are they interested in learning about the coarse causal effects of policy interventions (the recipe), but they also want to know what specifically it is about the intervention (the baking powder) that is responsible for observed effects. These impact “drivers” can be considered the ingredients that comprise an intervention’s recipe. Social welfare programs tend to be multifaceted, and we conventionally evaluate them as such: randomizing to the program recipe as a whole instead of to selected ingredients. This evaluation approach prevents us from easily detecting what it is about the recipe that makes it successful. However, those who design and administer such programs find it useful to know what about their programs are the essential ingredients.
Using program evaluation tools to uncover “what works” is an active and exciting area of research. Both government and foundations have held workshops within the last year to explore the methods we use for understanding how to detect those impact drivers. Likewise, we at Abt Associates partnered with APPAM to host a forum on experimental evaluation research in practice, a segment of which focused on innovations that enhance researchers’ ability to identify the drivers of policy impacts.
My passion for evaluating social welfare programs is driven by a commitment to providing the most useful information possible to the field about what works, what doesn’t work, and what might work better. This involves both the up/down vote on whether we should invest in a given intervention and also the more interesting questions about essential program ingredients. In the years to come, the analytic tools and approaches that are coming from the field’s most innovative researchers are poised to help us understand the relationship between essential ingredients and the recipe as a whole. Bon Appetit!
Additional ideas and information:
What Works for Addressing the 'What Works' Question, an Abt Thought Leadership paper
Dr. Laura Peck is a member of the APPAM Policy Council and is a leading research authority on social welfare and employment policies and programs. Previously, she was an Associate Professor at Arizona State University (ASU) School of Public Affairs. She was also the Founding Associate Dean of Barrett, the Honors College at ASU's Phoenix campus.