APPAM Institutional Member Forum - Leverage Points for Stronger, More Informative Impact Evaluations of Social Programs

August 18, 2020

Virtually, through Zoom

Registration required. 

Researchers conducting rigorous evaluations of social programs continue to push the methodological frontier—by trying to get “inside the black box” to discover what drives impacts, by working to generalize results from sample to nation, and by seeking to validate quasi-experimental approaches using experimental benchmarks. As these investigations proceed, some fundamentals of strong social experiments are under-attended.This forum hosted by APPAM and Westat will address three of these, presenting ideas of universal relevance to random assignment evaluations illustrated through employment assistance studies.

One fundamental move forward is to test only policies/programs with strong potential for favorable impacts. The forum’s opening session presents three strategies for following this path—strategies for obtaining qualitative and quantitative assessments of an employment intervention’s potential for generating earnings gains before implementing a large-scale experimental test. Each of these strategies—which can be applied beyond workforce policy—is underutilized by evaluators and evaluation planners.

Another fundamental need in social experiments is to observe the right policy contrast between the intervention scenario and the counterfactual world (the treatment and control group contrast in a randomized experiment). The right contrast depends on a study’s motivating purpose: What policy or funding decision will be made using the impact estimate? What long-term conclusion on an employment strategy’s overall effectiveness will be drawn from the evidence? The forum’s second session frames these issues and presents new methods for analyzing and interpreting data from workforce assistance experiments that get the right answers to the right intervention/counterfactual contrasts.

None of these strategies will work unless well-chosen and well-constructed social experiments are put into the field effectively. The final forum session invites all attendees to share the latest best practices for assuring site readiness and intervention fidelity when conducting employment-focused random assignment evaluations. It is experience from the field and practitioners’ synergies that will push progress in these aspects of social experimentation.

Finally, the forum concludes with reflections on the messages of the day, addressing the following questions: Has the forum provided good advice for guiding future impact evaluations—in employment policy and in general? What are the next topics that should be addressed to strengthen implementation and scientific tools for more informative social policy impact evaluations?

Schedule at a Glance

The full playlist is here.