Analysis of Peer Review Offers Deep Insights into Research Productivity

Investigators with the American Institute of Biological Sciences report findings in a paper published in the journal PLOS One, from an analysis of the research output from a series of biomedical research grants funded after undergoing a scientific peer review process.

The results, reported in 'The Validation of Peer Review Through Research Impact Measures and the Implications for Funding Strategies,' offer insights for future research on peer review and potential models for increasing research productivity.

"Some form of peer review is used at the majority of research funding organizations to determine the best research to fund," said Dr. Joseph Travis, President of AIBS and a biologist at Florida State University. "Peer review makes a significant contribution to how billions of dollars in research grants from government and private sources are awarded," said Travis, a coauthor of the study.

In recent years, this process has been questioned, particularly with regard to how well peer review predicts the ultimate impact of the funded research.

"We conducted a retrospective analysis of peer review and project output data for 2,063 projects from an eight year period. Of these, 227 were funded and we examined whether correlations exist among the assessment of scientific merit using a peer review system and the scientific output from this program," said Dr. Steve Gallo, Technical Operations Manager for AIBS and the lead investigator on the study.

Citation impact, or the number of times a research paper is referenced by others, is a common way to assess research impact. Analysis revealed that peer review scores associated with individual applications were correlated with the total time-adjusted citation output of these funded projects.

Gallo states, "citation impact did not correlate with the amount of funds awarded per application or with the total annual programmatic budget." The number of funded applications per year did correlate well with total annual citation impact, suggesting that improving funding success rates by reducing the size of awards may be one strategy to optimize the scientific impact of research program portfolios.

"This strategy must be weighed against the need for a balanced research portfolio and the inherently high costs of some kinds of research," said Travis.

The relationship observed between peer review scores and publication output lays the groundwork for establishing a model system for future prospective testing of the validity of peer review formats and procedures.

"This is something AIBS is looking at now," said Gallo.

Source: Eurekalert