Reviewing the review: a qualitative assessment of the peer review process in surgical journals

Catherine H Davis; Barbara L Bass; Kevin E Behrns; Keith D Lillemoe; O James Garden; Mark S Roh; Jeffrey E Lee; Charles M Balch; Thomas A Aloia

doi:10.1186/s41073-018-0048-0

Reviewing the review: a qualitative assessment of the peer review process in surgical journals

Catherine H Davis, Barbara L Bass, Kevin E Behrns, Keith D Lillemoe, O James Garden, Mark S Roh, Jeffrey E Lee, Charles M Balch, Thomas A Aloia

Research output: Contribution to journal › Article › peer-review

Abstract / Description of output

Background: Despite rapid growth of the scientific literature, no consensus guidelines have emerged to define the optimal criteria for editors to grade submitted manuscripts. The purpose of this project was to assess the peer reviewer metrics currently used in the surgical literature to evaluate original manuscript submissions.

Methods: Manuscript grading forms for 14 of the highest circulation general surgery-related journals were evaluated for content, including the type and number of quantitative and qualitative questions asked of peer reviewers. Reviewer grading forms for the seven surgical journals with the higher impact factors were compared to the seven surgical journals with lower impact factors using Fisher's exact tests.

Results: Impact factors of the studied journals ranged from 1.73 to 8.57, with a median impact factor of 4.26 in the higher group and 2.81 in the lower group. The content of the grading forms was found to vary considerably. Relatively few journals asked reviewers to grade specific components of a manuscript. Higher impact factor journal manuscript grading forms more frequently addressed statistical analysis, ethical considerations, and conflict of interest. In contrast, lower impact factor journals more commonly requested reviewers to make qualitative assessments of novelty/originality, scientific validity, and scientific importance.

Conclusion: Substantial variation exists in the grading criteria used to evaluate original manuscripts submitted to the surgical literature for peer review, with differential emphasis placed on certain criteria correlated to journal impact factors.

Original language	English
Pages (from-to)	4
Journal	Research integrity and peer review
Volume	3
DOIs	https://doi.org/10.1186/s41073-018-0048-0
Publication status	Published - 23 May 2018

Keywords / Materials (for Non-textual outputs)

Journal Article

Access to Document

10.1186/s41073-018-0048-0

document-1
This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Final published version, 566 KBLicence: Creative Commons: Attribution (CC-BY)

Cite this

@article{6a419162aedb4658867524cd9508b9cb,

title = "Reviewing the review: a qualitative assessment of the peer review process in surgical journals",

abstract = "Background: Despite rapid growth of the scientific literature, no consensus guidelines have emerged to define the optimal criteria for editors to grade submitted manuscripts. The purpose of this project was to assess the peer reviewer metrics currently used in the surgical literature to evaluate original manuscript submissions.Methods: Manuscript grading forms for 14 of the highest circulation general surgery-related journals were evaluated for content, including the type and number of quantitative and qualitative questions asked of peer reviewers. Reviewer grading forms for the seven surgical journals with the higher impact factors were compared to the seven surgical journals with lower impact factors using Fisher's exact tests.Results: Impact factors of the studied journals ranged from 1.73 to 8.57, with a median impact factor of 4.26 in the higher group and 2.81 in the lower group. The content of the grading forms was found to vary considerably. Relatively few journals asked reviewers to grade specific components of a manuscript. Higher impact factor journal manuscript grading forms more frequently addressed statistical analysis, ethical considerations, and conflict of interest. In contrast, lower impact factor journals more commonly requested reviewers to make qualitative assessments of novelty/originality, scientific validity, and scientific importance.Conclusion: Substantial variation exists in the grading criteria used to evaluate original manuscripts submitted to the surgical literature for peer review, with differential emphasis placed on certain criteria correlated to journal impact factors.",

keywords = "Journal Article",

author = "Davis, {Catherine H} and Bass, {Barbara L} and Behrns, {Kevin E} and Lillemoe, {Keith D} and Garden, {O James} and Roh, {Mark S} and Lee, {Jeffrey E} and Balch, {Charles M} and Aloia, {Thomas A}",

year = "2018",

month = may,

day = "23",

doi = "10.1186/s41073-018-0048-0",

language = "English",

volume = "3",

pages = "4",

journal = "Research integrity and peer review",

issn = "2058-8615",

}

TY - JOUR

T1 - Reviewing the review

T2 - a qualitative assessment of the peer review process in surgical journals

AU - Davis, Catherine H

AU - Bass, Barbara L

AU - Behrns, Kevin E

AU - Lillemoe, Keith D

AU - Garden, O James

AU - Roh, Mark S

AU - Lee, Jeffrey E

AU - Balch, Charles M

AU - Aloia, Thomas A

PY - 2018/5/23

Y1 - 2018/5/23

N2 - Background: Despite rapid growth of the scientific literature, no consensus guidelines have emerged to define the optimal criteria for editors to grade submitted manuscripts. The purpose of this project was to assess the peer reviewer metrics currently used in the surgical literature to evaluate original manuscript submissions.Methods: Manuscript grading forms for 14 of the highest circulation general surgery-related journals were evaluated for content, including the type and number of quantitative and qualitative questions asked of peer reviewers. Reviewer grading forms for the seven surgical journals with the higher impact factors were compared to the seven surgical journals with lower impact factors using Fisher's exact tests.Results: Impact factors of the studied journals ranged from 1.73 to 8.57, with a median impact factor of 4.26 in the higher group and 2.81 in the lower group. The content of the grading forms was found to vary considerably. Relatively few journals asked reviewers to grade specific components of a manuscript. Higher impact factor journal manuscript grading forms more frequently addressed statistical analysis, ethical considerations, and conflict of interest. In contrast, lower impact factor journals more commonly requested reviewers to make qualitative assessments of novelty/originality, scientific validity, and scientific importance.Conclusion: Substantial variation exists in the grading criteria used to evaluate original manuscripts submitted to the surgical literature for peer review, with differential emphasis placed on certain criteria correlated to journal impact factors.

AB - Background: Despite rapid growth of the scientific literature, no consensus guidelines have emerged to define the optimal criteria for editors to grade submitted manuscripts. The purpose of this project was to assess the peer reviewer metrics currently used in the surgical literature to evaluate original manuscript submissions.Methods: Manuscript grading forms for 14 of the highest circulation general surgery-related journals were evaluated for content, including the type and number of quantitative and qualitative questions asked of peer reviewers. Reviewer grading forms for the seven surgical journals with the higher impact factors were compared to the seven surgical journals with lower impact factors using Fisher's exact tests.Results: Impact factors of the studied journals ranged from 1.73 to 8.57, with a median impact factor of 4.26 in the higher group and 2.81 in the lower group. The content of the grading forms was found to vary considerably. Relatively few journals asked reviewers to grade specific components of a manuscript. Higher impact factor journal manuscript grading forms more frequently addressed statistical analysis, ethical considerations, and conflict of interest. In contrast, lower impact factor journals more commonly requested reviewers to make qualitative assessments of novelty/originality, scientific validity, and scientific importance.Conclusion: Substantial variation exists in the grading criteria used to evaluate original manuscripts submitted to the surgical literature for peer review, with differential emphasis placed on certain criteria correlated to journal impact factors.

KW - Journal Article

U2 - 10.1186/s41073-018-0048-0

DO - 10.1186/s41073-018-0048-0

M3 - Article

C2 - 29850109

SN - 2058-8615

VL - 3

SP - 4

JO - Research integrity and peer review

JF - Research integrity and peer review

ER -

Reviewing the review: a qualitative assessment of the peer review process in surgical journals

Abstract / Description of output

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this