Wednesday, March 18, 2020

APPAM Launches COVID-19 Member Resources, Data and Research Collaboration Hub

As many professional organizations around the globe explore ways that they can contribute to addressing the current public health crisis, APPAM would like to provide members with a platform for interacting and sharing ideas focused around the COVID-19 topic. To that end, we are launching a Resources, Data, and Research Collaboration Hub.


Social Experiments in Practice: The Why, When, Where, and How of Experimental Design & Analysis

March 7, 2014 01:00 PM

Social Experiments in Practice: The Why, When, Where, and How of Experimental Design & Analysis

Videos for most of the sessions can be seen below or directly via APPAM's YouTube channel.

The federal government increasingly is looking for strategies to identify promising social programs for broad-scale rollout. Abt Associates and APPAM held a day-long Institutional Member Forum to explore social experiments in practice and how recent methodological advances can address real-world realities in the United States and abroad. The forum  considered the type of policy questions that evidence from experimental evaluations can provide to program administrators and policymakers, and addressed questions on the timing of such experiments.

Featured speakers included: Naomi Goldstein, Director of the Office of Planning, Research and Evaluation in the Administration for Children and Families at the Department of Health and Human Services; Tom Cook of Northwestern University; Elizabeth Stuart of Johns Hopkins University; Rebecca Maynard of the University of Pennsylvania; Larry Mead of New York University; Larry Orr of Johns Hopkins University; and Stephen Bell, Jacob Klerman, Rob Olsen, Laura Peck, and Howard Rolston of Abt Associates.

Additional reading and citations for all sessions

Event Agenda

9:15 - 10:45 am

On the Frontier of the “Why” and “When” of Social Experiments: Doing the Right Thing at the Right Time

Download the slides.

In the larger field of program and policy evaluation, there is an active debate on the “best” designs for assessing programs’ causal impacts. In this debate, some have criticized experiments on ethical, scientific, feasibility and cost grounds. The introduction to this forum explores a range of concerns in these areas to consider which should be—and which should not be— grounds for not engaging in social experimentation when researchers and evaluation sponsors seek reliable evidence of policy effectiveness. In addition, presenters consider when in the development of social program approaches and information on their potential success the right time to conduct a rigorous, randomized impact evaluation arrives.

Moderator: Naomi Goldstein, Director of the Office of Planning, Research and Evaluation in the Administration for Children and Families, Department of Health and Human Services

The “Why” of Experimental Evaluation: Are Objections to Randomized Experiments Ever Insurmountable Given the Importance of Reliable Guidance for Social Policy Choices?
Howard Rolston, Abt Associates

When Is a Program Ready to be Evaluated?
Jacob Klerman, Abt Associates

Reactor: Tom Cook, Northwestern University

Audience Q&A

10:45 - 11:00 am

Networking break

11:00 am - 12:30 pm

On the Frontier of the “Where” of Social Experiments: Enhancing External Validity in the Relevant Policy Space

Download the slides for Rob Olsen.

Download the slides for Elizabeth Stuart.

Most social experiments are conducted in a non-randomly selected set of sites (e.g., schools, counties, or local program offices who agree to be evaluated or that are convenient for researchers to include). This raises concerns that the results from social experiments may not strictly apply in other locations. This part of the forum addresses the limitations of social experiments conducted in non-randomly selected sites; it also discusses possible approaches to addressing this challenge.

Moderator: Naomi Goldstein, Director of the Office of Planning, Research and Evaluation in the Administration for Children and Families, Department of Health and Human Services

Design Innovations for Improved External Validity
Rob Olsen, Abt Associates

Analysis Methods for Improved External Validity
Elizabeth Stuart, Johns Hopkins University

Reactor: Demetra Smith Nightingale, Chief Evaluation Officer, U.S. Department of Labor

Audience Q&A

12:30 - 12:45 pm

Networking break

12:45 - 1:45 pm

Lunch and Plenary
Social Experiments in the International Sphere
Iqbal Dhaliwal, Deputy Director of the Abdul Latif Jameel Poverty Action Lab (J-PAL)

1:45 - 2:00 pm

Networking break

2:00 - 3:30 pm

On the Frontier of the “How” of Social Experiments: Getting Inside the Black Box

Download the slides.

View the archived webinar of this session.  P lease note an email address is required to access the webcast. 

Researchers and policymakers are increasingly dissatisfied with experimental findings that address only the “average treatment effect a social program—as a whole and 'as is.'" This part of the forum will explore cutting-edge research approaches for identifying facets of social programs that are “impact drivers”—features of the intervention that, when present, cause favorable impacts to occur or to be greater in magnitude than they would be otherwise. Presenters consider both impact design and impact analysis strategies for this purpose, as well as contributions of cutting-edge implementation evaluation methods to delve into the “black box” to learn more about “what works” in social policy.

Moderator: Naomi Goldstein, Director of the Office of Planning, Research and Evaluation in the Administration for Children and Families, Department of Health and Human Services

Opportunities for Implementation Research
Larry Mead, New York University

Innovations in Experimental Analysis
Laura Peck, Abt Associates

Innovations in Experimental Design
Steve Bell, Abt Associates

Reactor: Naomi Goldstein

Audience Q&A

3:30 - 3:45 pm

Networking break

3:45 - 4:30 pm

Closing: Where To From Here?

Download the slides.

Highlights and Perspectives
Rebecca Maynard, University of Pennsylvania

Highlights and Perspectives
Larry Orr, Johns Hopkins University

Audience Q&A

Speaker Bios

Stephen Bell

Stephen_BellStephen Bell, Ph.D., an expert in random-assignment evaluation methods, first joined Abt Associates in 1983, then rejoined the company in 2005 as an Abt Fellow and a Principal Associate/Scientist in the Social and Economic Policy Division. Bell specializes in econometric impact evaluation of programs to assist disadvantaged workers and families. Among his projects has been the National Head Start Impact Study, a landmark evaluation of early childhood development assistance as an anti-poverty strategy, and on studies of employment strategies for people receiving disability benefits in the United States and in Britain. During the 1980s and early 1990s, Bell played a lead role in the National Job Training Partnership Act Study, the nation's seminal random assignment employment and training program evaluation. The results of this research helped reshape federal employment training policy and set a standard for rigorous evidence on program effectiveness in the federal government.

Thomas D. Cook

Thomas_D_CookThomas D. Cook is a Professor of Sociology, Psychology, Education and Social Policy at Northwestern University and a faculty fellow at the Institute for Policy Research. Cook’s primary field of research is the methodology of program evaluation; in particular, field experimentation and quasi-experimentation. He coauthored some of the most important books in the field of program evaluation: Quasi-Experimentation: Design & Analysis Issues for Field Settings (1979, with Donald Campbell); Foundations of Program Evaluation: Theories of Practice (1991, with William Shadish and Laura Leviton); and Experimental and Quasi-Experimental Designs for Generalized Causal Inference (2002, with Campbell and Shadish). These books are required reading for graduate students across the social sciences.

Iqbal Dhaliwal

Iqbal_headshotIqbal Dhaliwal is the Deputy Director of the Abdul Latif Jameel Poverty Action Lab (J-PAL) at MIT's economics department, a network of 100+ professors from 42 universities worldwide whose mission is to promote evidence informed policy by evaluating development programs. Iqbal works with policy makers worldwide to disseminate the implications of J-PAL's 500+ research projects, facilitate new program evaluations, and implement the scale-up of successful programs. As Scientific Director of J-PAL South Asia based in India, he helps oversee research, policy outreach, and capacity building across dozens of local projects. 

Naomi Goldstein

GoldsteinNaomi Goldstein is Director of the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF) at the Department of Health and Human Services. She is responsible for advising the Assistant Secretary for Children and Families on increasing the effectiveness and efficiency of ACF programs.

Prior to her appointment as Director of OPRE in November, 2004, Ms. Goldstein served as Director of the Division of Child and Family Development in OPRE. Previously she directed the United States Postal Service Commission on A Safe and Secure Workplace, an independent commission that examined workplace violence affecting the Postal Service and the nation. She served as Project Manager for the Urban Institute’s Assessing the New Federalism project, and as Executive Officer in the Office of the Assistant Secretary for Planning and Evaluation at HHS. Earlier in her career, she served in the Massachusetts state government and developed infant mortality prevention programs at Brigham and Women’s Hospital in Boston.

Jacob Alex Klerman

Jacob_Klerman_bioJacob A. Klerman is a widely respected economist with more than 25 years of experience in social policy research. An expert in both experimental (i.e., random assignment) and quasi-experimental evaluation of social programs, Klerman is Co-Director of the Abt Associates Center for Evaluation Methods, the founder and Co-Director of Abt’s internal seminar (the Journal Author Support Group), and one of Abt’s eight Senior Fellows. His research has focused on social welfare policy (including AFDC/TANF, SNAP, Medicaid, housing assistance, and Unemployment Insurance), labor markets (including job training, maternity leave, and worker safety), nutrition policy (including SNAP and child nutrition programs), military manpower, data quality issues, and applied econometrics.

Rebecca Maynard

Dr. Maynard recently returned to the University of Pennsylvania after a two-year leave to serve as Commissioner of the National Center for Education Evaluation and Regional Assistance (NCEE) at the Institute of Educational Sciences.  She is a leading expert in the design and conduct of randomized controlled trials in the areas of education and social policy, has conducted influential methodology research, and recently published open-ware tools to support the efficient design of rigorous impact evaluations.  Dr. Maynard has been a leader in the development and application of methods for conducting systematic reviews of evidence on program effectiveness, including serving on the technical review team during the design and development of the What Works Clearinghouse and being active in the workgroup that laid the groundwork for the Cambell Collaboration.  Prior to joining the faculty at the University of Pennsylvania in 1993, she was Senior Vice President at Mathematica Policy Research, Inc.

She is a Fellow of the American Education Research Association (AERA); past president of the Association for Public Policy Analysis and Management (APPAM); recipient of the Peter H. Rossi Award for Contributions to the Theory and Practice of Program Evaluation (2009); co-recipient of the Society of Prevention Research (SPR) Public Service Award (2008); and recipient of the Best Book Award, Society for Research on Adolescents (SRA) (1998).  

Larry Mead

Mead_Lawrence_ResizedLarry Mead is Professor of Politics and Public Policy at New York University's Robert F. Wagner Graduate School of Public Service, where he teaches public policy and American government. He is also a Visiting Scholar at the American Enterprise Institute in Washington. Mead is an expert on the problems of poverty and welfare in the United States. Among academics, he was the principal exponent of work requirements in welfare, the approach that now dominates national policy. He has consulted with federal, state, and local governments in this country and with several countries abroad. He testifies regularly to Congress on poverty, welfare, and social policy, and he often comments on these subjects in the media.

Demetra Nightingale

nightingaleDemetra Smith Nightingale is Chief Evaluation Officer for the U.S. Department of Labor. As the Chief Evaluation Officer, she is responsible for coordinating the Department’s evaluation agenda and working with all agencies in the Department to design and implement evaluations. She has extensively studied employment policy, workforce development, labor markets, social policies and programs, and conducted many evaluations of federal, state, and local programs aimed at increasing employment, skills, and income for workers and families.

Nightingale is the author or co-author of five books and dozens of articles. Her books include Repairing the U.S. Social Safety Net (with Martha Burt) and Reshaping the American Workforce in a Changing Economy (with Harry Holzer). Before joining the Obama Administration she was a Senior Fellow at the Urban Institute, and, previously, on the faculty of Graduate Program in Public Policy at Johns Hopkins University.

Robert Olsen

Robert Olsen, Ph.D., is a Principal Scientist in the Social and Economic Policy Division at Abt Associates, Inc.  Dr. Olsen has over 15 years of experience conducting policy research in education, training, and welfare.  Over the past decade, he has played leadership roles on several large-scale random assignment evaluations, including two evaluations of the Upward Bound program, an evaluation of charter schools, and the design of an evaluation to test interventions to turn around chronically low-performing schools.

Dr. Olsen currently plays leading roles on several major evaluations of interventions to improve outcomes for disadvantaged youth.  For the Investing in Innovation (i3) program - a signature initiative of the Obama administration - he is leading a systematic review of the evidence side effects of more than 90 educational interventions that were funded by the i3 program.  He is also the Principal Investigator of two major randomized control trials, one to estimate the effects of an enhanced college advising program for disadvantaged high school students participating in the Upward Bound program, and the other to estimate the effects of the Transitional Living Program for homeless youth.

Larry Orr

Larry Orr, Ph.D., teaches the Program Evaluation course at the Institute for Health and social Policy at Johns Hopkins University and works as an independent consultant on the design and analysis of evaluation of public programs.  He currently serves as Evaluation Specialist for an evaluation of results-based aid to education in Ethiopia, as a reviewer for the Coalition for Evidence-Based Policy Top Tier Initiative, and as Co-Investigator for an NSF grant studying the external validity of randomized trials with purposively chosen sites.  He has served as co-investigator for external reviews of the IES and Department of Labor research and evaluation programs.

Dr. Orr has authored, co-authored, or edited six books and served as a member of the editorial board of Evaluation and Program Planning and on the National Academy of Sciences Committee on Postsecondary Education and Training.  From 1982-2007, Dr. Orr served as Project Director and Chief Economist at Abt Associates.

Laura Peck

Laura_Peck_2012Laura R. Peck, Ph.D., is a Principal Scientist at Abt Associates and has over 18 years of experience evaluating social welfare and employment policies and programs, both in research and academic settings. A policy analyst by training, Peck specializes in innovative ways to estimate program impacts in experimental and quasi-experimental evaluations, and she applies this to many social safety net programs. At Abt Associates, Peck is the PI, Co-PI and Director of Analysis for several major national evaluations for the U.S. Departments of Health and Human Services, Labor, Agriculture, and Housing and Urban Development.

Howard Rolston

HowardRolstonWebRolston has been involved in funding, promoting, designing, and implementing social policy experiments for more than 30 years. During that time, he served for more than two decades as the Director of Research and Evaluation at the Administration for Children and Families in the United States Department of Health and Human Services. Initially his responsibilities involved welfare policy and evaluation, but they later expanded into other areas, including early childhood education, child welfare and child support enforcement. In 2006, he joined Abt Associates as a Principal Associate and is currently principal investigator for two large, multi-site social policy experiments.

Elizabeth Stuart

STUART_elizabeth_1_headshotElizabeth Stuart is an Associate Professor of Mental Health and Biostatistics at Johns Hopkins Bloomberg School of Public Health (JHSPH). Stuart has extensive experience with causal inference in randomized trials and non-experimental studies, with particular interests in the trade-offs of different study designs. Her research focuses on two areas: 1) propensity score methods for estimating causal effects in non-experimental studies, and 2) the generalizability of randomized trials to target populations. She has received funding for these projects from NIH and NSF and serves on a number of federal advisory panels, including the inaugural PCORI advisory panel on clinical trials.


« Back

Association for Public Policy Analysis and Management (APPAM)
NEW ADDRESS! 1100 Vermont Avenue, NW, Suite 650 Washington, DC 20005
Phone: 202.496.0130 | Fax: 202.496.0134
Twitter Facebook LinkedIn Subscribe to me on YouTube

Home|About APPAM|Membership|Public Policy News|Conference & Events|Publications| Awards|Careers & Education|Members Only

Web site design and web site development by

© 2020 Association for Public Policy Analysis & Management. All Rights Reserved.
Site Map | Privacy Policy | Terms of Use | Events | Add Your Event