How to Evaluate the Quality of a Research Study
In my TechWell stories, I frequently reference research studies, but I describe the findings as “studies suggest” rather than “studies show” or “studies prove.” Even if the findings seem definitive, I prefer “studies suggest” because a lot of scientific research has proven to be unreliable. In fact, there’s a growing belief among scientists and others that a great deal of reported research is false.
False as used here doesn’t mean fraudulent, as in the data was fabricated (although that does happen). More often, false means the findings couldn’t be replicated, too small a sample was tested, or the results of studies that generated contrary results weren’t published. And then there’s misleading statistics, or at least demonstrating some creativity with the data. Most researchers know the saying “They tortured the data until it confessed.”
It’s wise not to accept the findings of a research report without asking some questions. So if you read an article claiming that 89 percent of people in a recent study preferred Product A over Product B, you might ask how many people participated in the study. (2,300 would be better than 23.) Were they randomly selected, or were they people who responded to a web survey? How was the study conducted? And what were the questions? The wording of a question can influence the way people respond.
Several other factors can come into play in evaluating a research finding, such as how well the study controlled for confounding variables and the suitability of the research design for the question being researched. And it’s always worth asking who conducted the research and who is reporting it.
The fact that the vendor of Product A sponsored or conducted research that found Product A superior to its competitors doesn’t necessarily invalidate the findings. But when the organization funding or conducting a research study has a vested interest in the outcome, it’s grounds for a healthy dose of skepticism.
Of course, you rarely have the option to read an entire research report in order to evaluate the findings. More often, you come across a newspaper article or a blog that cites the result of a research study, and these summaries present an oversimplified version of the research. Therefore, it’s wise to be aware of how little you actually know about the study, and to let that realization keep you from automatically or unthinkingly accepting the findings.
