Skip to Main Content Carnegie Mellon University Libraries

A guide to Sysrev for collaborative literature and document reviews

Overview of labels and reviewing documents

The fundamental building blocks of Sysrev are labels. Labels are customizable and can be Boolean Yes/No labels, categorical labels, or open free text labels. There are also more complex labels called Group labels and an annotation feature for extracting text from PDFs into a label answer. You can also import labels from another project.

These labels can be used in the eligibility screening, data extraction and coding steps of systematic literature reviews, or for helping you organize and categorize items in your Sysrev project.

Below are some tutorials for using these different types of labels.

[Note that you will also see a label type called "Relationship" in Sysrev. This label type is currently non-functional.]

Reviewing documents with one or more reviewers

Before we go over how to create and use labels, let's take a look at how labels facilitate the process of reviewing or screening documents along with the different review options. In Sysrev, the review of documents can be done by a single reviewer or collaboratively. For collaborations, Sysrev makes possible blind review (where reviewers do not see the decisions or labels of other reviewers), and provides different review modes to facilitate single or double screening of documents. It also makes it easy to identify where conflicts between reviewers exist, so those conflicts can be resolved by discussion, consensus or a third reviewer.

 

Review options and settings

There are a number of settings that you can adjust to manage your screening and review process. These are found under Manage -> Settings, in the Options box.

  • Article Review Priority: When reviewers start to look at articles in the Articles tab, they will be automatically presented with an article. The Article Review Priority will determine if the articles provided to reviewers are not yet screened by anyone (Single), screened by one reviewer already (Full), or a mix of these two (Balanced). In this way, you can manage a collaborative review with either single or double screening.
  • Allow Unlimited Reviews: This setting (Yes/No) can allow for reviews in which you want more than two reviewers (i.e. unlimited reviewers) per document.
  • Automatically Save Review Labels: This option (Yes/No) can be used when reviewers prefer to not have to click Save after completing the required labels. If Yes is chosen, once all REQUIRED labels are completed, they will automatically be saved and the next record will appear. This is probably not a good option if you have some optional labels, or a complex label structure.
  • Label Blinding: Turning this label on will make the decisions of other reviewers invisible. This is a good option for most reviews in which double screening is carried out. This setting can be turned off during conflict resolution.

We will cover the other review Options in other sections of this guide. Once you have adjusted your review settings, don't forget to click the Save Changes button at the bottom of the Options box!

 

How to review documents

Any member of the project can review documents (i.e. assign labels to documents). To do so, from the main project page, click the Review tab. On the left-hand side, all of the enabled labels will appear. Once all required labels are answered by the reviewer, the blue Save button will become available. Click Save to record label answers. The next record for review will automatically appear in the window. Reviewers can also choose to skip the record.

Note that depending on the Article Review Priority settings described above, the records will be prioritized by whether or not they have already been reviewed by another reviewer. Other than this sorting, the records are randomly assigned.

 

How to review documents in Sysrev

 

 

Boolean, categorical and string labels

The main types of labels in Sysrev are Boolean, Categorical and String:

  • Boolean labels are for labels that would require a yes or no answer.
  • Categorical labels can have one of a limited number of possible values. Reviewers can choose from a list, and this can be set so reviewers can choose only one or multiple answers.
  • String labels are 'free text' labels. These can be constrained by number of characters or can be set to take only a certain pattern (using regular expressions).

 

Each label has a number of different settings including:

  • whether an answer is required for review or if the label can be skipped,
  • whether reviewer consensus is required (i.e., if differing answers between reviewers should be flagged as conflicts),
  • whether the answer should align with the inclusion decision or not--if yes, a reviewer will be alerted to conflicts between their inclusion decision and their label answer.

There are also settings related to auto-labeling, which are covered in the auto-labeling section of this guide. The following video demonstrates creating labels and walks through some of these settings in more detail.

A note about editing labels

Label questions can be edited at anytime. However, for categorical labels, categories that have been used in prior labeling in the project cannot be edited or deleted. That said, new categories can be added at any time.

 

Group labels

Group labels in Sysrev are essentially labels within labels. They facilitate tabular data extraction by allowing users to extract multiple sets of the same combination of labels. In this way, users can create a nested data extraction structure for more complex projects. The video below describes how to create and use Group labels.

 

 

Annotation labels

Annotation labels allow users to highlight text in attached PDFs, extracting the highlighted text to a label, and then tag that text with a label option. For example, if you wanted to extract information about the outcomes measured in a study and then assign that extracted text to a defined group of outcome types, you could do this with an annotation label. Or you could simply use this label type to extract highlighted text from PDFs, without applying the additional tag.

Note that you can only do this with attached PDFs. Annotation labels do not work for document metadata (e.g., title/abstract).

 

annotation label example

 

SysRev Analytics

Sysrev's Analytics feature provides an in-depth exploration of reviewer decisions across all of the labels in the project. Understanding concordance between reviewers can help identify labels that may not be clearly defined or are being interpreted differently by different reviewers, leading to many conflicts. The following video provides an overview of using the Sysrev Analytics module.