{
  "fetched_at": "2026-10-24T23:30:50Z",
  "source_url": "https://www.kaggle.com/competitions/covid-19-nlp-challenge/data",
  "pages": [
    {
      "name": "data-description",
      "content": "# Datasets:\n\nResearchers at the Brookhaven Nation Laboratory (BNL), Computational Science Initiative (CSI),\nand the Oak Ridge National Laboratory (ORNL), Biophysics group, have collected a question-answer dataset annotated by a biomedical domain expert to evaluate the system on specialized\ninformation acquisition for protein-related questions. Due to the limited size of the collected QA\ndata for training a full-fledged NLP model, we recommend that participants leverage external\nresources to pre-train and fine-tune their models. The most significant of these is the COVID-19\nOpen Research Dataset (CORD-19) [1], provided by AllenAI’s Semantic Scholar team. Other\npotential resources include publicly-available QA datasets and Natural Language Inference (NLI)\ndatasets (NLI tasks determine semantic relations between two sentences – a premise and a\nconclusion):\n\nQuestion answering (QA) datasets:\n\n1. The Stanford Question Answering Dataset (SQuAD) [2]: a large collection of question-answer pairs created by crowd workers on Wikipedia articles, SQuAD 2.0 contains more than 150k pairs of question answers.\n\n2. COVID-QA [3]: a SQuAD-like dataset consisting of 2,019 COVID related questions and answers to build a COVID-specific QA system\n\n3. SearchQA [4]: more than 140k general question-answer pairs from the popular television show Jeopardy.\n\n4. BioASQ [5]: domain-specific data consisting of 1,504 question-answer pairs created by biomedical domain experts.\n\nNatural Language Inference (NLI) datasets:\n\n1. Stanford NLI (SNLI) [6]\n\n2. Multi-Genre NLI (MultiNLI) [7]\n\n3. Medical NLI (MedNLI) [8]\n\n4. Science text entailment (SciTail) [9]\n\nThe BNL-curated QA validation/test datasets that will be provided to the participants will include\n113 question/answer pairs for the following four questions:\n\n1. What are the oligomeric states of coronavirus structural proteins?\n\n2. What are the oligomeric states of non-structural coronavirus proteins?\n\n3. What are the catalytic domains (active sites) of coronavirus proteins?\n\n4. Are there antivirals that target structural viral proteins?\n\nThe answers are sentences from papers related to COVID and are labeled as one of the three\ncategories (relevant, partially relevant, and not relevant). The QA datasets listed above may not\nbe directly applicable to the BNL-curated datasets due to the different formats. The QA datasets\nabove are pairs of questions and passages (contexts), and the purpose is to find an answer\n(mostly very short answer e.g., a word) in the passage. On the other hand, the BNL-curated\ndatasets are composed of pairs of questions and sentences, and it aims to determine the\nrelevance between them. The QA datasets can be still used to deliver general QA knowledge to\na model. The NLI datasets are useful for semantic relation analysis between sentences. The list is\nnot exclusive, and participants may utilize any other resources for model training.\n\n**Test references:** The list of DOIs of the articles used for the QA dataset generation as well as\npre-processed versions of these articles (in JSON format that is more structured and\nconvenient to use compared to the raw PDF document) will be provided.\n\n**QA data:** Question & Answer pairs will be provided for validation of the developed NLP model\n(by the participant) and for assessing its performance (by the organizer)\n\n1. Validation set: the validation set consists of 54 queries and 54 sentences identified from\nthe test references mentioned above that may provide answers to the queries. Labels\n(relevant, partially relevant, irrelevant) are provided for all 54 pairs. This validation set\ncan be used by the participant to validate and optimize the performance of their NLP\nmodel during development.\n\n2. Test set: the test set consist of 59 queries and 59 sentences identified from the test\nreferences that may (or may not) provide answers to the queries. Labels are not provided\nfor this test set, and the participants are expected to predict the labels for all query-sentence pairs.\n\nThe test references and the BNL-curated QA datasets can be obtained from the following GitHub\nrepo: https://github.com/BNLNLP/IHS-NLP-Data-Hackathon\n\n# References:\n\n[1] Wang, L. L. et al. (2020). Cord-19: The covid-19 open research dataset. arXiv preprint\narXiv:2004.10706.\n\n[2] Rajpurkar, P. et al. (2016). Squad: 100,000+ questions for machine comprehension of text.\narXiv preprint arXiv:1606.05250.\n\n[3] Möller, T. et al. (2020). Covid-qa: A question & answering dataset for covid-19.\n\n[4] Dunn, M. et al. (2017). Searchqa: A new q&a dataset augmented with context from a search\nengine. arXiv preprint arXiv:1704.05179.\n\n[5] Tsatsaronis, G. et al. (2012). Bioasq: A challenge on large-scale biomedical semantic\nindexing and question answering. In AAAI fall symposium: Information retrieval and\nknowledge discovery in biomedical text. Citeseer.\n\n[6] Bowman, S. R. et al. (2015). A large annotated corpus for learning natural language\ninference. arXiv preprint arXiv:1508.05326.\n\n[7] Williams, A. et al. (2017). Abroad-coverage challenge corpus for sentence understanding\nthrough inference. arXiv preprint arXiv:1704.05426.\n\n[8] Shivade, C. (2019). Mednli – a natural language inference dataset for the clinical domain\n(version 1.0.0). PhysioNet.\n\n[9] Khot, T. et al. (2018). Scitail: A textual entailment dataset from science question answering.\nIn AAAI, volume 17, pages 41–42.\n\n[10] Devlin, J., et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language\nUnderstanding.” Proceedings of NAACL-HLT. 2019.\n\n[11] Lee, J., et al. “BioBERT: a pre-trained biomedical language representation model for\nbiomedical text mining.” Bioinformatics 36.4 (2020): 1234-1240.\n\n# Other relevant resources:\n\n[1] Soto, C., Park, G., Chen, Y.C., Sedova, A., Pouchard, L. and Yoo, S., 2020. Applying Natural\nLanguage Processing (NLP) techniques on the scientific literature to accelerate drug\ndiscovery for COVID-19. F1000Research, 9.\n\n[2] Acharya, Atanu, Rupesh Agarwal, Matthew B. Baker, Jerome Baudry, Debsindhu Bhowmik,\nSwen Boehm, Kendall G. Byler et al. “Supercomputer-based ensemble docking drug\ndiscovery pipeline with application to COVID-19.” Journal of chemical information and\nmodeling 60, no. 12 (2020): 5832-5852.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "Description",
      "content": "# ***Email Competition Organizers***\n\nIf you haven't done so already, send an email to the organizing team (Bobak Mortazavi (bobakm@tamu.edu), Ryan King (kingrc15@tamu.edu), Subhamoy Mandal (s.mandal@ieee.org) and Tayo Obafemi-Ajayi (tayoobafemiajayi@missouristate.edu)) for information about the next step.\n\n# Challenge Description:\n\nTens of thousands of research papers on the SARS-CoV-2 virus and the COVID-19 illness have flooded journals and preprint servers, with many more added every day. This publication rate is leaving researchers unable to keep up with new findings and insights, even as they rely on these to inform their own work in understanding and combating the disease. In particular, research groups investigating novel drug candidates with in-silico drug discovery efforts are in essence searching for a proverbial needle in a haystack, and so rely on scientific insights from the literature to guide their efforts toward the most fruitful directions. Thus, they are left to navigate the current publication deluge for their primary guidance. In order to keep pace with the vast amount of relevant literature that continues to grow, researchers need more capable tools to locate and filter the information they seek in these publications. To meet this pressing need, it is desirable to develop and apply the latest Natural Language Processing (NLP) techniques for efficiently locating and filtering relevant information from a growing dataset of COVID-related literature, to help accelerate drug discovery research. The proposed NLP challenge task is framed towards designing and building a Question-Answering (QA) system to find answers to COVID-related questions in the scientific literature.\n\n# Goal:\n\nA practical usage of NLP models with COVID relevant papers might be automated information extraction from literature to facilitate drug discovery efforts. One of the crucial elements that can inform these efforts is the knowledge about viral proteins. The goal of this data challenge to build an NLP model to identify answers to protein-related questions from scientific papers.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "Evaluation",
      "content": "# Evaluation criteria:\n\nEvaluation criteria for this task falls into two parts: (1) language modeling of the CORD-19 dataset,\nand (2) QA performance on the curated dataset.\n\n**First part (language modeling):** due to the small dataset size, effective modeling of the language\nfound in COVID-related scientific articles is essential to performing well on the QA task.\nParticipants may choose to use a pretrained base language model (e.g., BERT [10] or BioBERT\n[11]), in which case they will receive a baseline score on the language modeling task. However,\nthey may instead choose to domain-tune their language model of choice on the CORD-19 dataset,\nwhich if done well will give them an edge both by increasing their language modeling score, but\nalso potentially by improving the quality of their QA model. Language modeling quality will be\ndetermined using the perplexity metric on a the “Test references” shown above. All participants\nshould report the perplexity metric measured on each of the test references with a detailed\ndescription of the evaluation procedure.\n\n**Second part (QA performance):** an ideal QA system first ranks and retrieves articles related to a\ngiven question from the literature database, and then extracts answers from the selected articles.\nTo evaluate the participant’s system on this test environment, the participant may build an endto-end model (document retrieval and answer extraction) and check if the list of answers includes\nthe answers in the test sets. However, given that the test data has not been generated using the\nentire collection of articles in COVID-19, the retrieved articles by the QA system developed by\nthe participant may not contain the test articles. We ask the participants to perform the following\ntasks:\n\n1. Relevant sentence prediction: from the Test references, identify top 3 sentences that may\ncontain answers to the queries. Provide top 3 sentences for each of the 59 queries in the\ntest set (also indicate the articles from which each sentence was extracted).\n\n2. QA pair label prediction: for each of the 59 question-sentence pairs in the test set, label\neach pair as “relevant”, “partially relevant”, or “irrelevant”. For this task, the participant\nwill need to build a classifier that identifies whether a given sentence is relevant to a given\nquery or not.\n\nFor participants who intend to build an “end-to-end model”, the TREC (Text REtrieval Conference)\nworkshops can provide useful resources for finding the most relevant documents to a given query.\nTREC aims to support research in information retrieval and provide materials for large-scale\nevaluation of text retrieval methodologies. Recently, researchers, clinicians, and policy makers\nhave hosted a TREC workshop for COVID-19. Please refer to TREC-COVID [9].\n\nThe evaluation will be done **OFFLINE**.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "rules",
      "content": "# Submission Requirements \n\n- **In order to accept the award for a top place submission, participants must purchase a registration ticket for BHI (https://bhi.embs.org/2023/) and attend the conference. Participants who do not wish to attend the conference are still free to enter their submission but will not be considered for conference awards**",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "Submission",
      "content": "# Submission Requirements \n\n- **In order to accept the award for a top place submission, participants must purchase a registration ticket for BHI (https://bhi.embs.org/2023/) and attend the conference. Participants who do not wish to attend the conference are still free to enter their submission but will not be considered for conference awards**\n\n# What to Submit\n\n- Technical description of your method\n\n- Perplexity metric measured on each of the test references with a detailed description of the evaluation procedure\n\n- Relevant sentence prediction results\n\n- QA pair label prediction results\n\n- URL to your GitHub repo, where your code can be accessed and the prediction results can be reproduced.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    }
  ]
}