Dr. Larry Orr, Johns Hopkins University
Rossi Award Winner Larry Orr Talks Career, Successes, and Inspiration in Evaluation
October 31, 2014 09:00 AM
Earlier this month, APPAM announced Dr. Larry L. Orr of Johns Hopkins University as the latest recipient of the Peter H. Rossi Award. For the last 40 years, Orr has made numerous notable contributions to the field of evaluation. He is also known for helping other researchers conceptualize and conduct their own high quality work. Orr will formally receive the Rossi award next week at APPAM’s Fall Research Conference in Albuequerque, New Mexico and will facilitate a discussion on Thursday, November 6, entitled Beyond Internal Validity. He will discuss the challenges facing the standard model of evaluation now that the great methodological battle of the last 40 years--how to ensure internal validity--is essentially resolved. Orr will be joined by Jacob Klerman, Abt Associates, Inc; Judith M. Gueron, MDRC; and Rebecca Maynard, University of Pennsylvania.
Orr took a moment to talk about his work, the Rossi Award, those who have impacted his work, and his career successes with APPAM.
What is your current focus/research about?
I’m working on several research projects that I find quite exciting and stimulating. I consult with a British firm on an evaluation of aid to education in Ethiopia. Having spent nearly my entire career working on U.S. domestic programs, this is a fascinating opportunity to see first-hand some of the social problems and policy issues in developing countries.
I’ve also had the opportunity to work on a methodological problem that has concerned me for some time – the external validity of the evaluations we conduct. Liz Stuart, Rob Olsen, Steve Bell, and I have a National Science Foundation (NSF) grant to find ways to quantify external validity bias and develop methods to reduce it. In addition to the intrinsic interest of the problem, this is a fantastic group of people to work with.
I serve on various review and advisory panels, which I find is a really good way to keep up with what is happening in the profession. Particularly rewarding has been my work for the Coalition for Evidence-Based Policy, reviewing randomized tests of social programs to determine whether they meet the Coalition’s “top tier” evidence standard.
And, of course, I really enjoy working with the students in my Program Evaluation course at Hopkins. They keep me on my toes!
What does winning the Rossi award mean to you?
The Rossi award is a tremendous honor. There is no professional recognition as meaningful as the recognition of one’s peers. And I have always considered APPAM to be the professional association of my colleagues and peers.
It is also a great honor to be included in the company of Rossi awardees. The prior awardees are an elite group of researchers for whom I have enormous respect, both personally and professionally.
What direction do you see your career taking in the next five/ten years?
I plan to continue doing what I’ve been doing for 40-odd years—seizing every opportunity I can to work with smart people on interesting problems. And teaching! I really find teaching enjoyable and rewarding.
Who do you consider a mentor in your field? Why?
I’m not sure I would call any of them mentors, but I have had the good fortune to work with, and learn from, some really great colleagues. At the risk of committing a sin of omission, I might name a few that have been particularly important to me.
Rob Hollister is one of my longest-standing professional colleagues. When I was a kid just out of grad school, drowning in my first professional job as an Assistant Professor, Rob took me under his wing, helped me direct my first research project (a summer project, with a budget of maybe $100k), and then, as if that weren’t enough, got me a job for which I was much better suited and that I really loved! Over the years, Rob has always been available for sage advice or just to lend an ear.
Another early—and formative—professional relationship was with Joe Newhouse, with whom I worked closely for seven years on the design and implementation of the Health Insurance Experiment. Joe, and his team at RAND, taught me what experimental design was all about.
In the 1980s and 90s, I spent eight of the most important – and rewarding – years of my life working with Judy Gueron and Howard Bloom on the National JTPA Study. Wow! What I learned from that pair! And not just about research methodology (though it was a real education in that), but about how one thinks about evaluation and designs an evaluation and, very importantly, implements an evaluation in the field. We didn’t always agree—especially about the latter—but it was certainly a learning experience.
One of my most valued current colleagues is Steve Bell, with whom I have collaborated on various projects, including the JTPA Study, over the past 30 years. Steve showed me how to solve design problems from first principles and put structure on even the messiest research problem, while still maintaining high standards of rigor.
Last, but not least, I should acknowledge my professional debt to my wife, Kathleen Flanagan, who worked with me on evaluation projects for 4 or 5 years—and still married me! Kathleen taught me how to work as part of a research team, and how to make a team more than the sum of its parts.
Any advice for graduates or new people stepping into the policy research field?
Wow! That’s hard. I know what worked for me, but that may not be everybody’s cup of tea. I guess the best advice I can give is to look for opportunities to work with, and learn from, really good people. But don’t be afraid to challenge them; everybody is at their best when they are pushed and forced to think.
What do you consider is your greatest success in evaluation over your distinguished career?
Another hard one... I’ve worked on a lot of great projects. But if I have to choose one, I would have to say the Health Insurance Experiment. I conceived the experiment in response to a very important policy question: what would happen to the demand for medical care under a major health insurance reform? This was not just an academic exercise – at the time, I had just finished a stint working for the Deputy Undersecretary for Policy at the Department of Health, Education, and Welfare, designing a legislative proposal for national health insurance.h Hejh I pushed the experiment through a very resistant bureaucracy and assembled an incredibly talented team at Rand and MPR to conduct it. Thirty years later, it is still the best evidence we have on this question. And there is good evidence that the experiment’s results had a major effect on the way private insurers structure health insurance policies. I am very proud of that project.