When Is a Result “Significant”
November 19, 2014 11:00 AM
By Becky Kelleman, Rutgers University
As a Master’s student, I was very happy to see the new Data Analytics Track at the APPAM Fall Research Conference this year. In my program at the Bloustein School for Planning and Public Policy, students have the option to concentrate their studies in a number of different areas. One of the least favorite for many of the students is the Methods track, the track that focus strongly on quantitative and qualitative analysis.
To be fair, many of us have various backgrounds in liberal arts, so although we all are now familiar with statistical analysis and understand its importance, most do not study additional courses. I am one of the handful that find statistical analysis fascinating, which is why I was drawn to the Data Analytics track.
“When is a result significant” is a question that students need to be able to answer in the first semester at the Bloustein School. Needless to say, there are quite a few instances when we found questionable statistically significant results. For example, if you examine the relationship of the divorce rate in Maine and per capita consumption of margarine in the U.S. or honey-producing bee colonies and juvenile arrests for possession of marijuana in the U.S., these relationships had a stronger correlation than any that I had seen at that point. Although these are all entertaining examples of spurious correlations examined by a student at Harvard, many of us faced similar puzzles when working through our research questions and analyzing data.
If you missed the roundtable, there were several practical points to consider when discussing significance:
It is important to know who is asking the question, and are they asking about statistical significance or if a program works. If you are looking into the pilot testing of a small program, you may not have a large enough N to run statistics. Instead of asking “is the result significant”, you should ask “Can I provide enough evidence that the program is probably working, and if so, I can continue running the program and continue to gather more evidence”. –Melissa Kovacs
Significance tests are a restraining device to discourage people from jumping to conclusions. Sensitivity analysis helps. We should be doing that more in our evaluations. –Winston Lin
We need better estimates of variability and understand if it’s applicable to the real world problem we are trying to solve. –Austin Nichols
We need to look at statistical significance AND practical significance. We need to look at a p-value and what is driving that. When looking into the meaning of a significance test in an evaluation, we would want to do several studies in different places to see if the results match…but this is a very expensive endeavor.- Jeffrey Smith
I have about four pages of notes from this discussion which also included transitioning from p-values to confidence intervals and whether or not, in practice, statistically significance in program evaluations makes any difference. The panelists kept a fast-paced, lively conversation and I look forward to more conversations regarding data analytics at next year’s Fall Research Conference. To fellow students, if you are scared or interested in data analytics, make sure to check out the new track next year!