# Evaluation criteria:

Evaluation criteria for this task falls into two parts: (1) language modeling of the CORD-19 dataset,
and (2) QA performance on the curated dataset.

**First part (language modeling):** due to the small dataset size, effective modeling of the language
found in COVID-related scientific articles is essential to performing well on the QA task.
Participants may choose to use a pretrained base language model (e.g., BERT [10] or BioBERT
[11]), in which case they will receive a baseline score on the language modeling task. However,
they may instead choose to domain-tune their language model of choice on the CORD-19 dataset,
which if done well will give them an edge both by increasing their language modeling score, but
also potentially by improving the quality of their QA model. Language modeling quality will be
determined using the perplexity metric on a the “Test references” shown above. All participants
should report the perplexity metric measured on each of the test references with a detailed
description of the evaluation procedure.

**Second part (QA performance):** an ideal QA system first ranks and retrieves articles related to a
given question from the literature database, and then extracts answers from the selected articles.
To evaluate the participant’s system on this test environment, the participant may build an endto-end model (document retrieval and answer extraction) and check if the list of answers includes
the answers in the test sets. However, given that the test data has not been generated using the
entire collection of articles in COVID-19, the retrieved articles by the QA system developed by
the participant may not contain the test articles. We ask the participants to perform the following
tasks:

1. Relevant sentence prediction: from the Test references, identify top 3 sentences that may
contain answers to the queries. Provide top 3 sentences for each of the 59 queries in the
test set (also indicate the articles from which each sentence was extracted).

2. QA pair label prediction: for each of the 59 question-sentence pairs in the test set, label
each pair as “relevant”, “partially relevant”, or “irrelevant”. For this task, the participant
will need to build a classifier that identifies whether a given sentence is relevant to a given
query or not.

For participants who intend to build an “end-to-end model”, the TREC (Text REtrieval Conference)
workshops can provide useful resources for finding the most relevant documents to a given query.
TREC aims to support research in information retrieval and provide materials for large-scale
evaluation of text retrieval methodologies. Recently, researchers, clinicians, and policy makers
have hosted a TREC workshop for COVID-19. Please refer to TREC-COVID [9].

The evaluation will be done **OFFLINE**.