- The ability to document the impact of scholarly work is critical for your academic career.
- Scholarly work was traditionally confined to peer-reviewed publications in journals by assessing impact through the counting of peer-reviewed publications and grants.
- The next evolution of metrics was on quality of publications as defined by JIF and h-index.
- Social media has revolutionized the timeliness and ease of dissemination of work, which has resulted in the emergence of altmetrics as mechanisms for assessing global impact in this new digital landscape.
Publish or perish.
We’ve heard these words throughout our careers: If there is no record or proof your work has impact, does it matter?
In this blog post, we’ll take a brief journey through the peer-review process and the changing dynamics of the academic world, as well as examine the variety of metrics for your scholarly work.
Evolution of Peer Review
In 1665, the Royal Society of London and the Paris Academy of Sciences published the first two scientific journals: The Philosophical Transactions of the Royal Society of London and Journal des Scavans. It wasn’t until the 1830s that a peer-review process was formalized. In 1937, the National Cancer Institute became the first US entity to use peer review as a method for awarding grants to researchers. Shortly thereafter, the concept of peer review became the standard method for most research communities to place a figurative “stamp of approval” on publications and decisions for grant funding.
Since then, criticism has emerged about the peer-review process, including:
- It can stifle innovation as unorthodox work is not often published.
- If the review committees are stacked with cronies, then grants don’t get funded to those applying from outside the inner circle.
- If the content reviewers are in fact competing for the same publications and grants as those applying, then expertise and objectivity are at direct odds with each other.
- The peer-review process is not set up to detect fraud and research misconduct (as evidenced in recent years with the rising number of publication retractions).
At its core, the peer-review process relies on the integrity of the process and those who are involved. To help address some of the concerns outlined above, the Committee on Publication Ethics has generated guidelines in recent years for editors and reviewers that are focused on the content areas of professionalism, confidentiality, reliability, fairness of the review process, and conflict of interest.
At its core, the peer-review process relies on the integrity of the process and those who are involved.
Emerging Forces in the Academic World
Whether one chooses to pursue an academic career or not, exposure to academicians is commonplace during college, medical school, and residency.
Universities have evolved to become more like corporations and are, thus, subject to public funding cuts, which have resulted in a change in the culture and environment for academicians. In this environment, intellectual recognition is driven by academic peers. A scientist’s impact is measured by publications and citations of their work by other scientists.
Areas that connect people working in academic environments across the globe include peer-reviewed publications and extramural grants, promotion and tenure (P&T) guidelines, and membership/recognition in academic societies.
As a colleague who wished to remain anonymous, stated to me:
“One can have a debate about the impact of your work and whether publishing in a specific journal or not makes a difference. But no matter the issue at hand, there is one certainty in every P&T committee—its members know how to count.”
Metrics in an Unconventional World
The era of social media has highlighted the ease of dissemination of thoughts and work in a globally interconnected ecosystem. The creation and distribution of information has increased the accessibility of data and the democratization of media beyond the traditional firewalls of journals. Some of the emerging methods of communication include:
- Blog Posts
- Podcasts
- Instructional Educational Videos
- Webinars
The creation and distribution of information has increased the accessibility of data and the democratization of media beyond the traditional firewalls of journals.
Some P&T committees have started to recognize this new arena of engagement. But how does one differentiate those who are evangelists or indoctrinators from those who are committed to advancing science in an insightful, innovative, and scholarly fashion utilizing the social media matrix? There are several ways to measure one’s works.
Traditional metrics for journal publications include Journal Impact Factor (JIF) and the h-index. JIF is a quantification metric of a journal’s influence.
The h-index combines an author’s productivity (number of articles published) and the impact of those articles (number of citations). It is defined as the maximum value such that the given author has published h papers that have been cited at least h times. It thus allows the ability for comparing scientists who work in the same field.
The term altmetrics, proposed in 2010, measures both the dissemination and potential impact of not only individual articles, as but also webinars, instructional videos, and blogs. Altmetrics use public application programming interfaces across platforms to gather data with algorithms. A classification of altmetrics was proposed by ImpactStory in 2012 on the different methods of impact:
- Viewed: HTML views and PDF downloads
- Discussed: Journal comments, science blogs, Wikipedia, Facebook, Twitter, and other social media
- Saved: Social media bookmarks (e.g., Mendeley, CiteULike)
- Cited: Citations in scholarly literature (e.g., CrossRef, Scopus, Web of Science)
- Recommended
There is debate on the interpretation of altmetrics. Some say the focus of altmetrics is on the attention or engagement rather than the quality of impact on the progress of science. Despite this limitation, altmetrics has the ability to expand the measurement of scholarly impact and reach a broader audience.
Some say the focus of altmetrics is on the attention or engagement rather than the quality of impact on the progress of science.
Summary
The ability to document the impact of scholarly work is critical for your academic career. Scholarly work was traditionally confined to peer-reviewed publications in journals by assessing impact through the counting of peer-reviewed publications and grants. The next evolution of metrics was on quality of publications as defined by JIF and h-index. Social media has revolutionized the timeliness and ease of dissemination of work, which has resulted in the emergence of altmetrics as mechanisms for assessing global impact in this new digital landscape.
Connect with Dr. Varghese @TomVargheseJr.
The opinions expressed in this article are those of the author and do not necessarily reflect the views of The Society of Thoracic Surgeons.