Appendix 1: A Taxonomy of Types of Peer Review and Objects Reviewed

We propose below a set of definitions for the types of peer review. Note that our objective here is limited to the question of forms of review implemented by publishers—those who take intellectual responsibility for making a decision as to what to publish under their imprimatur. While other forms of peer review certainly exist—notably, commentary on preprints and formal reviews of published material—we here keep our focus on those parts of the review and evaluation process a publisher controls and implements as part of its decision-making process.

In proposing a delineation between closed and open forms of review, our organizing principle is the information available to the reader about the content of review. This means that if the identities of authors and reviewers are known to each other, but not to readers, we regard that as a form of closed review.

Theoretically, this could be seen as suggesting a third category of “collaborative review”—one in which authors and reviewers who are part of the process implemented for a publisher to make a publishing decision are known to each other, but not to subsequent readers. For purposes of simplicity however, we propose regarding these collaborative forms of review as “closed” when the identity of reviewers and the content of their review is not disclosed to the readers and users of the published work.

I. Closed forms of review

Closed forms of peer review are those in which the identity of reviewers is hidden from the readers of the work once it is published. In traditional review systems, the identities of reviewers and authors are also hidden from each other in various ways.

We propose definitions of four kinds of closed review:

• Partly closed (Single-blind) Review. Reviewers may be informed of the author’s identity, but the author is not informed of the identity of the reviewers. Publication occurs after the author’s revisions in response to reviewers’ comments satisfy the editors and the Editorial Board.

• Fully closed (Double-blind) Review. The identity of the author is not disclosed to the reviewers, and the identity of the reviewers is not disclosed to the author. Publication is contingent on the author responding to the critiques and commentary offered by reviewers to the satisfaction of the editors and the Editorial Board.

• Cross review. Reviewers are shown each others’ work, under conditions of anonymity, and respond to the work of other reviewers as well as to the work of the author(s). The editor(s) take in view both the individual reviewers’ reports and their responses to other reviewers’ observations in evaluating the overall work.

• Peer-to-Peer Review. The identities of both author and reviewers are disclosed each to the other, but not to readers of the work. The process may result in more substantial exchanges and revisions to the work. Such a review process may eventuate from a process that began as fully or partly closed; in other cases, it may be employed for interdisciplinary work in which authors collaborate and review each other’s contributions. A peer-to-peer review process is often utilized as one form of review implemented in the case of an edited volume, in which the authors of various chapters have read and commented on each others’ chapters.

II. Open forms of review

Open forms of review are processes of peer review in which the identity of reviewers and the substance of their reviews of earlier stages of the work are disclosed to both the authors (or editors) of a scholarly work and, in some fashion, to readers of the published work.

This can take a variety of forms. In some cases, both the final work and the texts of reviews are made available to readers. In others, readers themselves are given means of offering their own reviews of a work prior to it attaining “version of record” status. These reviews, often in the form of comments or annotations (utilizing such platforms as CommentPress or hypothes.is) may be maintained in perpetuity in some form (say, on a web page dedicated to early versions of the work). A variety of means may be implemented to calibrate who may comment on work in progress (for example, permitting anonymous comment; requiring the use of actual names; requiring an indication of professional affiliation).

• Published Review. Commissioned reviewers work independently or collaboratively on their review, which is then made available to the author with reviewers’ names disclosed. Both reviewer comments and author replies, once reviewed and accepted by the editors, are all made part of the final published object. The publisher determines who is invited to serve as a reviewer.

• Crowd review. All interested commentators may read and offer reviews on content; all comments are made visible as part of the object under review. Commenting becomes part of the author’s writing process.

• Managed Crowd Review. The object is openly accessible for comment to all readers, with the publisher identifying a small number of readers to function in something more like the role of traditional reviewer—but within the open comments. The publisher may or may not disclose to readers who the invited reviewers are.

III. Scholarly objects that publishers review

Just as there are many forms of review that take place before and after the evaluation process utilized by publishers, so there are a variety of objects created by scholars that are read, shared, cited, and used by readers. That said, in deciding whether or not to publish a work, scholarly publishers typically focus on a narrow set of specific objects. Over time, as digital forms of scholarly expression become more broadly accepted within fields and disciplines, new forms of these objects may emerge; for the moment, however, we feel that the work of scholarly publishers is focused in the main on three specific sorts of objects:

• Manuscripts, the text of a scholarly argument, regardless of length, together with any figures, images, tables, or other affordances critical to the making of the argument the text contains.

• Proposals, the initial statement of argument(s), intended audience, engagement with other existing texts, and proposed structure of the work (more typical in monograph publishing)

• Datasets, the accumulation of evidence by scholars upon which an argument is based, and which can be shared with other scholars who seek to replicate an author’s argument or explore further pathways of inquiry (more typical in the natural and social sciences, but increasingly found in digital humanities fields).