Lecture Recap: Beyond Internal Validity
November 10, 2014 12:20 PM
By Sarah Momilani Marshall, University of Hawaii-Manoa
Dr. Larry Orr of John Hopkins University was selected to receive the Peter H. Rossi Award at this year’s APPAM Fall Research Conference. Orr’s contribution to the field of evaluation over the past forty years continues to impact his peers and colleagues regarding how to design and implement research in the field. Among those who discussed his contributions to the field were Rebecca Maynard, University of Pennsylvania, Jacob Kleman, Abt Associates, Inc., and Judith Gueron, MDRC.
Orr briefly highlighted the battle over internal validity by declaring it a victory with the universal acknowledgment that randomized trials have become the Gold Standard in research design. He went on to focus, however, on what he felt was the next significant battle for research design: the external validity of the Standard Model of impact evaluation. He discussed the shortcoming of current impact evaluation research in its frequent failure to adequately define the population of interest regarding policy impact, and its failure to secure representative samples from the population of policy impact interest. He acknowledged that there were no easy answers to this shortcoming, as random nationally representative samples are difficult to secure. Orr urged the research community to continue seeking creative and innovative approaches that may need to go against the status quo.
Among his recommendations for increasing the reliability of research efforts was to conduct experiments in two stages. The first stage would rely heavily on searching the literature more rigorously for potentially successful proposals and evaluating for the impact of primary interest using administrative data. Then, the second stage would focus on more intensive data collection similar to the Standard Model. One of the risks involved with this approach was the risk of false positives, or Type 1 errors, but the best prevention could be found in rigorous analysis, randomized trials, and replicated results.
You can view Orr's entire lecture on APPAM's YouTube channel.