Monday, November 21, 2016

APPAM Leadership and Policy Council Election Update

An update on the 2017 APPAM Elections

PRINT PAGE

Lecture Recap: Beyond Internal Validity

November 10, 2014 12:20 PM

By Sarah Momilani Marshall, University of Hawaii-Manoa

Dr. Larry Orr of John Hopkins University was selected to receive the Peter H. Rossi Award at this year’s APPAM Fall Research Conference. Orr’s contribution to the field of evaluation over the past forty years continues to impact his peers and colleagues regarding how to design and implement research in the field. Among those who discussed his contributions to the field were Rebecca Maynard, University of Pennsylvania, Jacob Kleman, Abt Associates, Inc., and Judith Gueron, MDRC.

Orr briefly highlighted the battle over internal validity by declaring it a victory with the universal acknowledgment that randomized trials have become the Gold Standard in research design. He went on to focus, however, on what he felt was the next significant battle for research design: the external validity of the Standard Model of impact evaluation. He discussed the shortcoming of current impact evaluation research in its frequent failure to adequately define the population of interest regarding policy impact, and its failure to secure representative samples from the population of policy impact interest. He acknowledged that there were no easy answers to this shortcoming, as random nationally representative samples are difficult to secure. Orr urged the research community to continue seeking creative and innovative approaches that may need to go against the status quo.

Among his recommendations for increasing the reliability of research efforts was to conduct experiments in two stages. The first stage would rely heavily on searching the literature more rigorously for potentially successful proposals and evaluating for the impact of primary interest using administrative data. Then, the second stage would focus on more intensive data collection similar to the Standard Model. One of the risks involved with this approach was the risk of false positives, or Type 1 errors, but the best prevention could be found in rigorous analysis, randomized trials, and replicated results.

You can view Orr's entire lecture on APPAM's YouTube channel.

 

« Back

 
 
 
Association for Public Policy Analysis and Management (APPAM)
NEW ADDRESS! 1100 Vermont Avenue, NW, Suite 650 Washington, DC 20005
Phone: 202.496.0130 | Fax: 202.496.0134
Twitter Facebook LinkedIn Subscribe to me on YouTube

Home|About APPAM|Membership|Public Policy News|Conference & Events|Publications| Awards|Careers & Education|Members Only

Web site design and web site development by Americaneagle.com

© 2016 Association for Public Policy Analysis & Management. All Rights Reserved.
Site Map | Privacy Policy | Terms of Use | Events | Add Your Event