For academic researchers, “Impact Factors” is a familiar term – it is designed to measure the impact of your publications. If you are outside of the academic world, you have probably never encountered it, but how much do these metrics really tell us?
Impact factors emerged as a way to compare research papers. Academia often measures a researcher’s success by the number of publications, which influences tenure and promotion decisions at universities. The problem was that not all journals and papers are of the same quality. Universities needed a way to determine the caliber of research without having to actually read every paper.
The solution? If a paper was cited by other researcher within a few years of publication, it is assumed that the paper and the journal it appears in must have an impact. Over time, this has evolved into a formal system for calculating Impact Factors, with journals proudly listing their scores.
To judge individual papers, metrics like “categorized-normalized citation impact” were developed using similar logic along with a “percentile group” score to compare a researcher’s work with papers in a similar field.
But I question: “impact on whom”? I feel that the impact of a paper in science and engineering would be a metric of the adoption of the methods in the paper to solve real-world problems. It’s possible to write a paper with a catchy title on a popular topic that gets the paper listed in the references in other papers. Some of these papers may rely on unrealistic assumptions and yield ridiculous results such that a real, practicing engineer would consider the paper laughable.
It takes a fairly long time for concepts to evolve from a research paper to test applications to inclusion in manuals of practice for professional organizations to adoption by practitioners as best practices. The real impact of a paper isn’t known for years.
The Reality Behind Research Funding
Meanwhile, funding organizations are pouring vast amounts of money into water projects. Judging by the quality of many of the papers, it is clear that these organizations don’t always do a very good job vetting the proposals thoroughly. As a result, the number of mediocre papers is skyrocketing, and with it the number of journals eager to publish them.
Funding of projects is justified based on the huge promises of benefits to society. Once the cash is in hand in the university checking account, the goals change from producing a major breakthrough for society to being “publishable.”
What Truly Matters
Most papers die on the vine while only a few go on to be strong branches in our profession. Every research project can’t be amazing. But the goal should be to advance the state-of-the-art, and publishing a paper is only one small step along that path. The impactful papers are those that lead to implementation.
The gatekeepers to journals are paper reviewers who usually do a good job in calling out questionable papers. However, many have never worked in the real world and can’t distinguish between quality and mediocre papers based on prior mediocre papers. Some journals, eager to fill pages, will publish almost anything as long as the authors are willing to pay the page charge.
Research is essential, but it is becoming increasingly difficult to separate valuable insight from poor quality work. The Impact Factor for any journal is at best a poor indication of the quality of any paper. Ultimately, the real measure of a paper’s impact should be how it shapes the world beyond academia.
Read more of Tom’s blogs here, and you can contact him at tom.walski@bentley.com.
Want to learn more from our resident water and wastewater expert? Join the Dr. Tom Walski Newsletter today!
