When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.
- PLOS Biology
- PLOS Climate
- PLOS Complex Systems
- PLOS Computational Biology
- PLOS Digital Health
- PLOS Genetics
- PLOS Global Public Health
- PLOS Medicine
- PLOS Mental Health
- PLOS Neglected Tropical Diseases
- PLOS Pathogens
- PLOS Sustainability and Transformation
- PLOS Collections
- About This Blog
- Official PLOS Blog
- EveryONE Blog
- Speaking of Medicine
- PLOS Biologue
- Absolutely Maybe
- DNA Science
- PLOS ECR Community
- All Models Are Wrong
- About PLOS Blogs
Impact AND credibility matter when researchers evaluate research
by Veronique Kiermer, Iain Hrynaszkiewicz, & James Harney.
Today we’ve posted a report , along with accompanying data, on qualitative research we conducted about how researchers assess the credibility and impact of research. This study, which has not yet been peer reviewed, was supported by a grant from the Alfred P. Sloan Foundation and conducted with the assistance of the American Society for Cell Biology . The findings will inform future PLOS activities to support improved research assessment practices — specifically to support efforts to move emphasis towards individual research outputs and away from journal-level metrics.
As we wrote in October 2020, we are interested in how researchers evaluate research outputs when (1) conducting their own research, and (2) when they take part in committees for hiring or grant review. In particular, we were interested in how researchers make judgments about the credibility and impact of the research outputs — including papers, preprints, research data — that they encounter in these contexts.
We interviewed 52 cell biology researchers. Our approach focused on the goals they are trying to achieve (e.g.”identify impactful research to read”), rather than the tools they are presently using to carry out these tasks. By focusing on researchers’ goals (the what ) rather than how they are achieving them, we sought to better understand how we might influence those practices. This qualitative research will be followed by survey work to better quantify our findings. This will provide insights into opportunities for better solutions for improved research assessment. In particular, we’ll understand what signals of credibility and impact might provide researchers with more useful ways than journal impact factor or journal prestige to assess the quality and credibility of individual studies and individual researchers.
Our results confirmed our initial hypothesis that the credibility (or trustworthiness) of research outputs is the central concern for researchers when conducting their own research, and that impact was a strong focus when researchers are part of hiring or grant review committees. But we established that researchers also assess attributes of research outputs related to reproducibility, quality, and novelty.
In addition, we found that researchers said they assessed credibility in committees more frequently than we anticipated, given that impact considerations — including journal impact factor — are prevalent in committee guidance and research assessment objectives (see for example McKiernan et al . (2019), Niles et al. (2020), Alperin et al. (2020), and Sugimoto & Larivière (2018)).
Our interviews confirmed that convenient proxies for credibility and impact, usually those based on journals, are used pervasively and are common in both research discovery and committee activities.
Our research also indicates that when researchers inspect publications to evaluate credibility they try to minimize the amount of time they spend reading and understanding publications. Their tactics included selective reading of the abstracts, figures, and methods sections. Sometimes they said that they also look for signals such as whether data was available and had been reused, whether peer-reviewed versions of preprints have been published, and whether open peer review reports were available.
Insights that help us better understand what researchers’ goals are and how they make judgements about credibility when discovering and reading research may offer opportunities to provide more reliable signals that help them with these tasks, yet are better tailored for credibility judgments than journal-level metrics. The stated importance of assessing credibility by researchers who participate in research assessment committees also suggests an opportunity for funders and institutions to better align their guidelines with the practice and motivations of committee members.
After our follow-up survey work to validate these preliminary findings, we will report back and hope that this research will help others in the understanding and development of better methods of research assessment.
Author: Iain Hrynaszkiewicz, Director, Open Research Solutions, PLOS; and Chris Heid, Head of Product, PLOS Summary Research by PLOS and Research Consulting…
We’re pleased to once again annual reporting from the Plan S Price & Service Transparency Framework for 2022…
Written by Lindsay Morton Over 4 years: 74k+ eligible articles. Nearly 85k signed reviews. More than 30k published peer review history packages…
IMAGES
VIDEO
COMMENTS
There are four main types of reliability. Each can be estimated by comparing different sets of results produced by the same method. Type of reliability. Measures the consistency of…. Test-retest. The same test over time. Interrater. The same test conducted by different people. Parallel forms.
Insights that help us better understand what researchers’ goals are and how they make judgements about credibility when discovering and reading research may offer opportunities to provide more reliable signals that help them with these tasks, yet are better tailored for credibility judgments than journal-level metrics.