by Alex Osei-Kojo, PhD Student, School of Public Affairs University of Colorado Denver
Following the theme for this year’s Fall Research Conference: Evidence for Action: Encouraging Innovation and Improvement, this session examined the role of systematic reviews in evidence-based policymaking. The moderator, Douglas Besharov, from the University of Maryland, introduced the session with the question: “Is it time for a more sophisticated approach to systematic reviews?” There were four speakers who provided separate responses.
The first speaker was Julia Littel, from Bryn Mawr College. Her presentation was titled “Systematic Reviews, Why Short Cuts Don’t Cut It.” Littel argued that despite the availability of a science of research synthesis, demonstrating how to minimize bias and error, these standards were ignored in some systematic reviews. Such poorly conducted systematic reviews provide weak evidence to support policy-making. To solve this problem, she proposed the following: investing in high quality primary research, conducting systematic reviews in a comprehensive manner, and reporting studies with mixed results.
The second speaker, Jacob Alex Klerman, presented on “Evidence-Based Policy: Filling in the theory for Action.” Building on the premise that most policy interventions fail to achieve their desired outcomes, Klerman explained that the focus of evidenced-based policymaking should be “doing something that helps the most.” To achieve this goal, he proposed three steps: identify, adopt, improve. The first step focuses on finding effective program models. The second step means applying programs that are found to be effective. The third step means constantly seeking better outcomes for such effective programs.
The third speaker was Rebecca Maynard, from the University of Pennsylvania. Her presentation was titled, “The Role of Systematic Reviews in Policy Development: A Vision for the Future.” She explained that current policymaking demands evidence that “counts.” This means thinking about the broad implications of the quality of evidence used for policymaking and focusing on its relevance for a specific policy context. In conclusion, she recommended building on current review protocols and standards and scrutinizing evidence before adoption.
The last speaker was Jon Baron, from the Laura and John Arnold Foundation. Jon’s presentation focused on the “overall goal of evidenced-based policy and how systematic reviews fit it in.” He argued that the purpose of evidence-based policy “is to improve the human condition.” This means that policies are implemented, at least from a normative standpoint, to improve a situation that is perceived to be undesirable. He further added that the major challenge with systematic reviews is the “problem of having to trust the researcher.” Jon explained that this is a challenge because some systematic reviews were of a low quality.
The key policy implications from these presentations were that: First, systematic reviews have an important role in evidence-based policymaking. Second, there is the need for public managers to distinguish useful evidence from poor evidence in policymaking. Third, improving the conditions of people should be at the center of evidence-based policymaking.
Pictured: Panelists for Systematic Reviews: The Next Step in Evidence-Based Policy Analysis at #2018APPAM