Skip to main content
SearchLoginLogin or Signup
Draft Report
Published on Mar 28, 2018

Appendix 1: A Taxonomy of Types of Peer Review and Objects Reviewed

Published onMar 28, 2018
Appendix 1: A Taxonomy of Types of Peer Review and Objects Reviewed
·

We propose below a set of definitions for the types of peer review. Note that our objective here is limited to the question of forms of review implemented by publishers—those who take intellectual responsibility for making a decision as to what to publish under their imprimatur. While other forms of peer review certainly exist—notably, commentary on pre-prints and formal reviews of published material—we here keep our focus on those parts of the review and evaluation process a publisher controls and implements as part of its decision-making process.

In proposing a delineation between closed and open forms of review, our organizing principle is the information available to the reader about the content of review. This means that if the identities of authors and reviewers are known to each other, but not to readers, we regard that as a form of closed review.

Theoretically, this could be seen as suggesting a third category of “collaborative review”—one in which authors and reviewers who are part of the process implemented for a publisher to make a publishing decision are known to each other, but not to subsequent readers. For purposes of simplicity however, we propose regarding these collaborative forms of review as “closed” when the identity of reviewers and the content of their review is not disclosed to the readers and users of the published work.

I. Closed forms of review

Closed forms of peer review are those in which the identity of reviewers is hidden from the readers of the work once it is published.  In traditional review systems, the identities of reviewers and authors are also hidden from each other in various ways.

We propose definitions of four kinds of closed review:

  1. Partly closed (Single-blind) Review. Reviewers may be informed of the author’s identity, but the author is not informed of the identity of the reviewers. Publication occurs after the author’s revisions in response to reviewers’ comments satisfy the editors and the Editorial Board.

  2. Fully closed (Double-blind) Review. The identity of the author is not disclosed to the reviewers, and the identity of the reviewers is not disclosed to the author. Publication is contingent on the author responding to the critiques and commentary offered by reviewers to the satisfaction of the editors and the Editorial Board.

  3. Exchanged review. Reviewers are shown each others’ work, under conditions of anonymity, and respond to the work of other reviewers as well as to the work of the author. The editor(s) take in view both the individual reviewers’ reports and their responses to other reviewers’ observations in evaluating the overall work.

  4. Peer-to-Peer Review. The identities of both author and reviewers are disclosed each to the other, but not to readers of the work. The process may result in more substantial exchanges and revisions to the work. Such a review process may eventuate from a process that began as fully or partly closed; in other cases, it may be employed for interdisciplinary work in which authors collaborate and review each other’s contributions. A peer-to-peer review process is often utilized as one form of review implemented in the case of an edited volume, in which the authors of various chapters have read and commented on each others’ chapters.

II. Open forms of review

Open forms of review are processes of peer review in which the identity of reviewers and the substance of their reviews of earlier stages of the work are disclosed to both the authors (or editors) of a scholarly work and, in some fashion, to readers of the published work.

This can take a variety of forms. In some cases, both the final work and the texts of reviews are made available to readers. In others, readers themselves are given means of offering their own reviews of a work prior to it attaining “version of record” status. These reviews, often in the form of comments or annotations (utilizing such platforms as CommentPress or hypothes.is) may be maintained in perpetuity in some form (say, on a web page dedicated to early versions of the work). A variety of means may be implemented to calibrate who may comment on work in progress (for example, permitting anonymous comment; requiring the use of actual names; requiring an indication of professional affiliation).

Published Review. Commissioned reviewers work independently or collaboratively on their review, which is then made available to the author with reviewers’ names disclosed. Both reviewer comments and author replies, once reviewed and accepted by the editors, are all made part of the final published object. The publisher determines who is invited to serve as a reviewer.

Crowd review. All interested commentators may read and offer reviews on content; all comments are made visible as part of the object under review. Commenting becomes part of the author’s writing process.

Managed Crowd Review. The object is openly accessible for comment to all readers, with the publisher identifying a small number of readers to function in something more like the role of traditional reviewer—but within the open comments. The publisher may or may not disclose to readers who the invited reviewers are.

III. Scholarly objects that publishers review

Just as there are many forms of review that take place before and after the evaluation process utilized by publishers,  so there are a variety of objects created by scholars that are read, shared, cited, and used by readers. That said, in deciding whether or not to publish a work, scholarly publishers typically focus on a narrow set of specific objects. Over time, as digital forms of scholarly expression become more broadly accepted within fields and disciplines, new forms of these objects may emerge; for the moment, however, we feel that the work of scholarly publishers is focused in the main on three specific sorts of objects:

Manuscripts, the text of a scholarly argument, regardless of length.

Proposals, the initial statement of argument(s), intended audience, engagement with other existing texts, and proposed structure of the work (more typical in monograph publishing)

Datasets, the accumulation of evidence by scholars upon which an argument is based, and which can be shared with other scholars who seek to replicate an author’s argument or explore further pathways of inquiry (more typical in the natural and social sciences, but increasingly found in digital humanities fields).

License
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
Comments
7
?
Login to discuss
?
Angela Gibson:

I would hope that these forms could be expanded to include “author anonymous” review, where the author is always anonymous to the reviewers and the reviewers are anonymous to the author unless they elect to reveal their identities. This is a version of single-blind review practiced by some humanities journals, including the MLA’s.

?
Sabina Alam:

Maybe I’m splitting hairs, but I find the phrase ‘ the text’ oddly specific. What about the figures and tables associated with the text. These are also often shared, cited, used by readers. Reviewers also have a role in ensuring the associated figures and tables are accurate

?
Sabina Alam:

Assuming that your intention here is that articles are published after peer review, there’s no mention here about what factors determine publication (which is discussed in the section above on closed peer review, including the role of editors and Editorial Board)

Also, what about post-publication open peer review as operated on F1000 Platforms? There the articles are published first (with a DOI), and then formal peer review takes place (‘Published review’). However, as the article is already live readers can also comment on the article at the same time (‘Crowd review’). So, this is a hybrid model which is fully open.

Mark Edington:

Two important points here — thanks fo them.

I think the “factors that determine publication” vary so much from publisher to publisher that no system focused on making peer review more transparent could possibly encompass them. That’s to say, for some publishers the question of marketability is critical in the decision of whether to publish; for others, it isn’t. So it seems that really a theme connected to, but not part of, this effort is establishing the norm of publishers disclosing their evaluative criteria in some way.

Post-publication open peer review is, obviously, increasingly important. And it needs to be captured, somehow, in any information made available about a given scholarly object. The problem, or perhaps conundrum, is that publishers can’t be made responsible for including in the signals they offer on a given work they’ve published reviews over which they’ve exerted no editorial control. Imposing that responsibility would, almost certainly, mean that far fewer publishers would take up the challenge of participating at all in any signaling regime. Most publishers will only be willing to give warrants for those aspects of a publishing process over which they’ve exerted editorial control. That doesn’t mean, however, that a sufficiently modular system could be created under which post-publication reviews could somehow be “connected” to the signals a publisher had applied to any given work…

?
Nick Michal:

Do these defined “forms of review” constitute a helpful framework to identify peer review by?

o   Research that open reviews (the idea of them) not statistically different than closed review (van Rooyen, doi: 10.1136/bmj.c5729)

o   Research that ‘Open Interaction,” “Open Reports,” “Open final-version commenting,” and “Open Participation” are perceived by majority as making peer review “better” (Ross-Hellauer, doi: 10.1371/journal.pone.0189311)

o   Four questions to focus on (Tennant, doi: 10.12688/f1000research.12037.3):

§  How can referees receive credit or recognition for their work, and what form should this take?

§  Should referee reports be published alongside manuscripts?

§  Should referees remain anonymous or have their identities disclosed?

§  Should peer review occur prior or subsequent to the publication process (i.e., publish then filter)?

?
Nick Michal:

Why not use traits of openness, rather than slightly flexible definitions (like those identified in Ross-Hellauer, doi: 10.12688/f1000research.11369.2)? As it stands, for example, a "Crowd review" would need to be moderated or else it could technically be closed via the identities of the commentators.

Mark Edington:

Interesting. The difficulty here is that “open” can be an adjective covering many domains — the question of rights (open access) the question of process (open review) or the question of outcomes (content of reviews open to the reader). For the purposes here, we’ll probably leave aside the question of rights — it is theoretically conceivable that there could be an open review process open to the reader that was behind a paywall. We’ve taken the approach of understanding openness from the point of view of the reader, if only because what publishers ultimately do is connect authors to readers.

Andy Collings:

“We propose definitions of four kinds of closed review”

Is the idea that a publisher’s peer review process would fit into one of the four definitions? It seems that some review processes would fall with definitions 1 and 3, or within 2 and 3, for example.

Also, it’s not clear how the definitions here account for situations where some, but not all, reviewers share their name with the authors. In those cases, isn’t the form of peer review somewhere between single-blind and double-blind?

Mark Edington:

The idea is that this is a proposal — not a dictat. Our trajectory here is to provide a set of suggestions to the convening groups of publishers — whether they are presses or scholarly societies — to carry forward the conversation we’ve begun. In the background work we undertook and in the conversations we’ve had with publishers, we didn’t come across such a hybrid as you describe here — some reviewers disclosed to the author, some not. But you’re right, this would be a combination of fully closed and partially closed.

Andy Collings:

“Exchanged review. Reviewers are shown each others’ work, under conditions of anonymity, and respond to the work of other reviewers as well as to the work of the author.”

What is meant by “conditions of anonymity“? (The reviewers wouldn’t necessarily remain anonymous.)

Also, I’m not sure many people would be familiar with the term “exchanged review“.

Mark Edington:

This is a review practice used by, for example, Science. The reviewers are shown the work of all reviewers, but they aren’t told the identity of either the author or the other reviewers.

Peer Review Transparency
Published with