{"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"name":"R","codemirror_mode":"r","pygments_lexer":"r","mimetype":"text/x-r-source","file_extension":".r","version":"4.0.5"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Applications of Artificial Intelligence and Machine Learning Models within the Biosciences\n\n## Category: Other\n\n*What have we learnt about use of Machine Learning and Artificial Intelligence for biological data problems ? and how have these helped with modelling and analysing COVID-19 and SARS-CoV-2 data ?*\n\n","metadata":{}},{"cell_type":"markdown","source":"# Introduction\n\nIn December 2019 there started to be reports of a severe respiratory disease affecting people in Wuhan, China ([1](https://pubmed.ncbi.nlm.nih.gov/31978945/),[2](https://pubmed.ncbi.nlm.nih.gov/32015508/)).  By January 2020 this was recognised as a new emergence of a human infecting betacoronavirus (2019-nCoV, now known as Sars-Cov-2) ([2](https://pubmed.ncbi.nlm.nih.gov/32015508/), [3](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8393069/), [4](https://pubmed.ncbi.nlm.nih.gov/31995857/)) and the genome sequence of the new virus was generated ([2](https://pubmed.ncbi.nlm.nih.gov/32015508/), [3](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8393069/)) and published ([2](https://pubmed.ncbi.nlm.nih.gov/32015508/), [reference sequence](https://www.ncbi.nlm.nih.gov/nuccore/NC_045512)).  Since then, scientists, biotech and pharmaceutical industries, medial and public health professionals, governments and others have been generating and analysing data across a wide spectrum of topics, for example: understanding the virus at a molecular and cellular level; its epidemiology and spread a population level; infection risk; disease risk; immune responses; and vaccine development.  Traditionally, these analyses would be performed using bioinformatic, statistical and dynamical modelling frameworks, however some types of data and studies are also amenable to Machine Learning (ML) and Artificial Intelligence (AI) techniques.\n\nArtificial Intelligence as a subject has been receiving coverage in the general media over the last few years, with popular news outlets having special sections of their websites showing the latest stories (e.g. [The Independent](https://www.independent.co.uk/topic/artificial-intelligence)).  Gauging the worldwide interest in Artificial Intelligence as a topic in the news using [Google trends](https://trends.google.com/trends/) shows steady interest from January 2020 until September 2022, followed by increasing interest to June 2023.  In contrast the Pandemic as a topic has been decreasing in news interest from an understandable high in March 2020.\n\nIn this article, we wanted to explore the use of Artificial Intelligence and machine learning models within the Biosciences, with a particular focus on recent developments in the last two years relating to data and modelling for the COVID-19 pandemic.\n","metadata":{}},{"cell_type":"markdown","source":"# Pandemic epochs and pandemic data\n\nSince the start of the pandemic, there have been over 700 million reported cases to World Health Organisation (WHO) (June 2023) and approximately 7 million reported deaths.  This data is collected at a country level and reported to the WHO and it has been collated to a [global dashboard](https://covid19.who.int/).  In the three and a half years from its emergence in humans, the virus has evolved, acquiring a variety of mutations and undergone diversification (e.g. see the [WHO page on variants](https://www.who.int/activities/tracking-SARS-CoV-2-variants)).  Simultaneously we as a global population have been trying to control its spread or at least the occurrence of severe disease using control measures such as non-pharmaceutical interventions (mask wearing, social distancing rules), movement and travel restrictions, and vaccination.  *Table 1* indicates an approximate time line showing different epochs of the pandemic and some modelling related questions for each.\n\n| Dates | Epoch | Modelling related questions |\n| ----- | ----- | ----- |\n| Jan - Mar 2020 | Early Pandemic | Estimates of transmissibility and case fatality ratios |\n| April - Sep 2020 | Lock-downs & restrictions | How is the virus spreading between regions, how effective are the non-pharmaceutical interventions at reducing spread |\n| Sep - Dec 2020 | Rise of Alpha Variant | First mass vaccinations and vaccine efficacy |\n| Dec 2020 - Mar 2021 | Alpha Variant | What makes a Variant of Concern ? Phenotypic effect of mutations in spike protein |\n| Mar - Nov 2021 | Delta and other VOCs | Severity and transmissibility of new VOCs, where are the new VOCs coming from and how are they spreading |\n| Nov 2021 - present |\tRise and diversification of Omicrons | Vaccine efficacy with primers and boosters, antigenic evolution and updating vaccine strains |\n\n*Table 1: Pandemic epochs and modelling related questions*\n\nAt the population scale, in addition to time-series case data (perhaps coupled with further metadata on more precise location, age or risk group category, travel data and vaccination status), many countries were able to sequence a proportion of the positive samples.  For some countries this would be largely restricted to hospitalised, severe or travel associated cases, but in others employing mass testing more generalised viral [genomic surveillance](https://www.cdc.gov/coronavirus/2019-ncov/variants/genomic-surveillance.html) was possible.  This sequence data could then be used to monitor viral evolution, the rise of potentially concerning mutations and spreading patterns of variants.  In previous epidemics viral sequence data has also been used in this way, however during the SARS-CoV-2 pandemic an unprecedented number of sequences have been generated and shared worldwide (far exceeding previous large collections of HIV and Influenza sequences).  As of the end of June 2023, there over 7.5 million openly accessible sequences in [NCBI Virus SARS-CoV-2 DataHub (Genbank)](https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType_s=Nucleotide&VirusLineage_ss=taxid:2697049) and over 15 million available in [GISAID](https://gisaid.org/).  Together with these sequence collections, automated bioinformatic algorithms and pipelines were created to name, classify and further analyse the data, including the [Pango lineage scheme](https://cov-lineages.org/index.html) ([5](https://doi.org/10.1038/s41564-020-0770-5),[6](https://doi.org/10.1093/ve/veab064)) and [NextStrain](https://nextstrain.org/) ([7](https://doi.org/10.1093/bioinformatics/bty407)).\n\nAdditionally, almost as soon as it became apparent that virus containment or ellimination would not be possible on a global scale (although some countries were able to keep case numbers very low), consideration was given to how and when the pandemic would end, and the role of naturally acquired immunity (recovery from infection) and vaccine induced immunity would play in the eventual transition from pandemic to endemicity ([8](https://doi.org/10.1126/science.abe6522),[9](https://doi.org/10.1371/journal.ppat.1010591)).\n\nGiven the amount of data collected and course of the pandemic in the last two years, it might be possible to have some ability to predict variant dynamics.  This could be done by coupling population immunity data and population scale viral variant data with experimental studies where the effects of viral mutations on viral fitness (e.g. [10](https://doi.org/10.1371/journal.ppat.1010951)) and cross reaction to vaccine strains ([11](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9273038/),[12](https://doi.org/10.1172/JCI162192),[13](https://pubmed.ncbi.nlm.nih.gov/36657480/\n)) can be measured.","metadata":{}},{"cell_type":"markdown","source":"# Kaggle competitions, data sets and COVID-19\n[Kaggle](https://www.kaggle.com/) runs a variety of competitions in which data scientists can apply their skills to interesting problems and data sets.  The competitors can develop and run their code via the online platform, and the winning solutions, commentaries and discussions are posted online.\n\nFigure 1 shows a word cloud of the common words used in the competition titles to give an indication of the types and variety of activities undertaken.  Figure 2 shows the number of number of competitions since 2010, with competitions relating to biological problems, epidemiology problems and specifically COVID-19 problems highlighted.\n\nIn 2020 and 2021, Kaggle ran five COVID-19 related competitions, three of them were in Spring 2020 and were about global or local forecasting of numbers of cases and risk factors. The remaining two were related to and radiography (images) and vaccines (RNA sequences).","metadata":{}},{"cell_type":"code","source":"###########################################\n# load packages - these are included with the utility script\n# library(wordcloud)\n# library(tm)\n# library(rjson)\n\n# use functions in utility script\nsource(\"../usr/lib/rutils_for_kaggleai2023_competition/rutils_for_kaggleai2023_competition.R\")\n\n# load Kaggle Writeups\nkw_meta <- get_KaggleWriteups()\n\n# all unique competition titles\nucomps <- get_UniqueCompetitionTitles(kw_meta)\ndf <- txt_to_word_cloud_df(ucomps,doStem=FALSE)\n\n# kaggle write up titles\nflags=process_KaggleWriteups_titles(kw_meta)","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-15T18:32:34.627533Z","iopub.execute_input":"2023-07-15T18:32:34.666397Z","iopub.status.idle":"2023-07-15T18:32:35.857282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Figure 1\nplot_word_cloud(df,legtxt1=\"Competition Title Words\",legtxt2=\"All Time\") ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-15T18:32:35.863766Z","iopub.execute_input":"2023-07-15T18:32:35.865645Z","iopub.status.idle":"2023-07-15T18:32:38.481499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Figure 2\nbarplot_KaggleWriteups_titles(flags)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-15T18:32:38.499565Z","iopub.execute_input":"2023-07-15T18:32:38.502391Z","iopub.status.idle":"2023-07-15T18:32:38.866999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Forecasting challenges\nIn 2020, the three forecasting challenges: [Week 1](https://www.kaggle.com/c/19548), [Week 4](https://www.kaggle.com/c/20010) and [Week 5](https://www.kaggle.com/c/20475) involved using time series data (numbers of confirmed cases and fatalities) collated by [Johns Hopkins University Center for Systems Science and Engineering](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_time_series), partly to make accurate predictions, but the underlying purpose was to shed light on some important epidemiological and scientific questions as set out by the USA’s National Academies of Sciences, Engineering, and Medicine (NASEM) and the World Health Organization (WHO).  These questions related to the expected impacts that social or environmental risk factors could have on disease spread and severity in the population (e.g. deaths, hospitalisations), and answers to which might point to policies for control measures appropriate for the studied regions.\n\nFrom the write-ups of entries in these competitions, gradient boosted machines, random forests, neural networks and forms of linear regression are mentioned.  Both gradient boost machines and random forests are types of ensemble model using decision trees, they differ in the training mechanism, and gradient boosted machines from [lightbgm](https://lightgbm.readthedocs.io/en/latest/) were thought to be particularly useful (see [Week 4 discussion](https://www.kaggle.com/c/20010/discussion/146006), and [Week 5 discussion](https://www.kaggle.com/competitions/covid19-global-forecasting-week-5/discussion/155638)).  In general it was encouraging that ML and statistical approaches could be used for this type of problem, and approaches that were able to explain which features of the data were included in the models did well.  \n\nNotably, the Kaggle COVID-19 forecasting challenges were conducted early in the first year of the pandemic, which means that (a) the data was of very limited time depth (b) reporting biases were likely an issue especially considering availability of testing (although perhaps this applies less to reported fatalities) and (c) this was a case of a novel virus entering populations with no prior immunity, hence the disease dynamics would be driven by number of viral imports and growth rate in the human population impacted by connectivity both in space and between age groups.  In a systematic review of modelling techniques for COVID-19 dynamics, covering until November 2020 ([14](https://pubmed.ncbi.nlm.nih.gov/33458453/)), the authors find that the majority of the studies use the traditional compartmental epidemiological models or statistical growth rate models,  whilst less than 10% used artificial intelligence type models.  Of the artificial intelligence type models that were reviewed, the model types included neural networks, Gated Recurrent Units (GRU) and Long Short-Term Memory (LSTM) models, auto-encoders, random forests and support vector machines.\n","metadata":{"execution":{"iopub.status.busy":"2023-07-01T17:58:30.006093Z","iopub.execute_input":"2023-07-01T17:58:30.008120Z","iopub.status.idle":"2023-07-01T17:58:30.023155Z"}}},{"cell_type":"markdown","source":"# Radiography challenge\n\nIn 2021, Kaggle ran a [competition](https://www.kaggle.com/competitions/siim-covid19-detection/overview) for the Society for Imaging Informatics in Medicine (SIIM) together with the Foundation for the Promotion of Health and Biomedical Research of Valencia Region (FISABIO), Medical Imaging Databank of the Valencia Region (BIMCV) and the Radiological Society of North America (RSNA).  In this competition, the goal was to use chest radiography images and classify them into discrete categories, including typical and atypical for COVID-19. The images had some bounding box segmentations to assist with object detection, and the training set had been classified by experts.  Similarly to the forecasting and vaccine challenges, random forests and gradient boosted machines were amongst popular solution entries, but several top entries used convolutional neural networks (e.g. see [1st](https://www.kaggle.com/c/26680/discussion/263658), [2nd](https://www.kaggle.com/c/26680/discussion/263674), [3rd](https://www.kaggle.com/c/26680/discussion/263654) place solutions).  These involved pre-training on one image set and fine tuning on another, and use of You Only Look Once algorithms, such as [Yolo5](https://pytorch.org/hub/ultralytics_yolov5/) and/or YoloX ([15](https://arxiv.org/abs/2107.08430)), or [EfficientNet](https://arxiv.org/abs/1905.11946)([16](http://proceedings.mlr.press/v97/tan19a.html?ref=jina-ai-gmbh.ghost.io)).  Figure 3 shows some of the common model terms used in the write-ups for this image based challenge.  This task was challenging, but use of pre-training image sets and model architectures previously used for general image segmentation and classification tasks seems to have significantly improved the model performances.  This in itself is a useful finding because it means that using a combination of pre-training data, existing models and then fine tuning towards a specific medical question along with expert guidance can rapidly yield good medical image classification models.","metadata":{}},{"cell_type":"code","source":"# Figure 3\nmatrix_plot_kaggle_writeup(kw_meta=kw_meta,flags=flags,j=5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-15T18:32:38.871690Z","iopub.execute_input":"2023-07-15T18:32:38.874173Z","iopub.status.idle":"2023-07-15T18:32:39.877068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Text Mining the Biomedical literature\n\nThe [COVID-19 Open Research Dataset (CORD-19)](https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge) is a vast collection of articles about COVID-19 (disease), SARS-CoV-2 (virus) and other related coronaviruses.  This data set was made available on [Kaggle](https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge), and is also available from the Allen Institute for AI at https://github.com/allenai/cord19.  The collection was assembled to assist with collation, curation and text mining of the hundreds of thousands of scientific papers ([17](https://pubmed.ncbi.nlm.nih.gov/33279995/)), and the final update was made on June 2022.\n\nAs an indication of the ongoing scientific effort, and the unfeasibility of any single researcher reviewing a significant proportion of the literature, searching on medRxiv and bioRxiv on 1st July 2023 returns 27,303 COVID-19 SARS-CoV-2 preprint articles (20,512 medRxiv, 6,791 bioRxiv).  Pubmed returns 281,688 articles under SARS-CoV-2 or COVID-19 from 2021, Pubmed also finds 124,526 results for SARS-CoV-2, COVID-19 and artificial intelligence or machine learning, with approximately half of those from 2021 onwards (68,268 articles).  \n\nConsequently,  natural language processing techniques especially including large language models have become very useful to classify and analyse COVID-19 and biomedical literature, allowing extraction of relevant primary studies to combine results and to contribute to evidence based decision making ([17](https://pubmed.ncbi.nlm.nih.gov/33279995/)). In particular Bidirectional Encoder Representations from Transformers (BERT) models have been used ([18](https://arxiv.org/abs/1810.04805)).  These are neural network type models with a specific architecture including recurrence ([19](https://proceedings.neurips.cc/paper/7181-attention-is-all)) and make internal representations of word positions in sentences resulting in models which include context.  They can be (pre-)trained on large corpora of general text in an unsupervised mode, and then fine tuned for more specific domains.  For example, this has been done on the CORD-19 dataset to create ClinicalCovidBERT and BioCovidBERT (https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge/discussion/138250, https://github.com/manueltonneau/covid-berts).  It is also now thought that the biomedical field texts are sufficiently different from generalised text, and large in size, to merit pre-training on biomedical texts, for example PubMedBERT ([20](https://doi.org/10.1145/3458754)) and the effects of using different corpora for biomedical tasks are compared in ([21](https://doi.org/10.1186/s12859-022-04642-w)).  Additionally, since there are now a variety of deep learning language models which are trained or tuned on different data sets it is possible to combine their results in an ensemble model to enable rapid updates to the knowledge base for specialised topics and 'living systematic reviews’ ([22](https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-023-02247-9)).","metadata":{}},{"cell_type":"markdown","source":"# Vaccine Challenge\n\nIn Autumn 2020, prior to the first COVID-19 vaccine role out, Kaggle launched  the [Open Vaccine COVID-19 mRNA Vaccine Degradation Prediction](https://www.kaggle.com/c/22111) challenge along with [Eterna](https://eternagame.org/).  In this challenge, the goal was to be able to predict how stable RNA molecules were against degradation.  The general idea is that RNA molecules are long sequences of nucleotides represented by four letters, and that due to nucleotide-nucleotide interactions they can fold up into 2 or 3 dimensional shapes which affect the stability of the molecule, and the accessibility of particular regions of the RNA to enzymes which can cut the molecule at the positions.  This type of problem is very similar to Natural Language Processing in where words are put in context in sentences (here words are short motifs or kmers of RNA letters)([19](https://proceedings.neurips.cc/paper/7181-attention-is-all)), and in the solutions submitted many participants opted for model solutions involving recurrence and/or attention and transformer neural network architecture models (see [Youhan Lee discussion](https://www.kaggle.com/c/22111/discussion/189845)), including LSTM models.  Figure 4 shows some of the common model terms used in the write-ups for this sequence based challenge.","metadata":{}},{"cell_type":"code","source":"# Figure 4\nmatrix_plot_kaggle_writeup(kw_meta=kw_meta,flags=flags,j=4)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-15T18:32:39.881656Z","iopub.execute_input":"2023-07-15T18:32:39.886567Z","iopub.status.idle":"2023-07-15T18:32:40.688634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Use of Natural Language Processing models for Biological sequence data\n\nThe Kaggle Vaccine challenge mentioned above was one example of using language models for biological sequence data.  Biological sequence data is generically a string of nucleotide letters (4 types A,C,G,T/U for DNA or RNA) or amino acid letters (20 types).  For protein coding regions of genomes (which are DNA or RNA), the amino acid sequence can be easily calculated since each group of 3 nucleotides corresponds to an amino acid (but some amino acids are coded for different groups of 3 nucleotides).  Creating and analysing models to associate sequence data with biological phenotype has long been a key goal in bioinformatics (23,[24](https://doi.org/10.1093/bib/bbk007)), and for the SARS-CoV-2 virus important genotype-phenotype problems include which variants (or specific mutations) are associated with increased transmissibility, severity or vaccine escape ([25](https://doi.org/10.1038/s41579-021-00573-0),[26](https://doi.org/10.1186/s11658-022-00352-6), [27](https://doi.org/10.1038/s41579-022-00841-7)).  In viruses these biological properties are usually governed by viral protein structures and their interactions with host proteins (for SARS-CoV-2 see for example [28](https://www.science.org/doi/10.1126/science.abb2762),[29](https://www.nature.com/articles/s41423-020-0400-4)).  So to understand and possibly predict viral properties in-silico in addition to direct association of viral sequence to a numeric or discrete phenotype, the ability to obtain a protein structure from a viral protein sequence is an important goal since this may be used in further structural biology analyses such as host protein docking calculations (eg ([30](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7784839/))).\n\nIn 2020, DeepMind published a paper on AlphaFold ([31](https://doi.org/10.1038/s41586-019-1923-7)) which predicts 3D protein structures from protein sequences.  This deep learning model, based on a convolutional neural network was trained on Protein Data Bank experimentally determined structures and performed well in the Critical Assessment of Protein Structure Prediction competition experiment (CASP13)(32).  However, since it was trained on general protein structures, the accuracy needed for specific binding studies was not thought to be sufficient (e.g. see [commentary](https://www.nature.com/articles/d41586-019-03951-0)).  Nevertheless, this was still a significant achievement.  But in 2021 DeepMind published a new improved model AlphaFold2([33](https://www.nature.com/articles/s41586-021-03819-2)) to great excitement in the field since it had performed even better in the CASP14([34](https://www.predictioncenter.org/casp14/doc/CASP14_Abstracts.pdf)) experiment, and this was considered a ‘breakthrough’ moment([35](https://doi.org/10.3389/frai.2022.875587)).  AlphaFold2 (hereafter ‘AlphaFold’) considers the relative positioning of residues (amino acids) in 3D space to be a graph inference problem, where the links represent nearby residues.  To infer the links in a new input sequence, that sequence is compared with other sequences in a multiple sequence alignment (MSA) and the links are inferred by reference to the other sequences.  The AlphaFold model is complex, and consists of a pre-processing step where the MSA is created and similar sequences (with structures) found, the Evoformer - a type of transformer model including encoding-decoding, recurrence and attention (BERT-like), and a subsequent protein 3D structure model also using attention to refine the predictions (see [33](https://www.nature.com/articles/s41586-021-03819-2), [35](https://doi.org/10.3389/frai.2022.875587), [36](https://pubmed.ncbi.nlm.nih.gov/34342271/), and [explainer by Carlos Outeiral Rubiera](https://www.blopig.com/blog/2021/07/alphafold-2-is-here-whats-behind-the-structure-prediction-miracle/)).  Additionally one of the ways in which AlphaFold improved its performance was the use of self-supervised learning on large amounts of unlabelled protein sequences.  \n\nBesides 3D protein structure prediction itself, models describing protein sequences are important in biology.  In these models, the idea is to have a means of assessing whether a sequence is ‘possible’ without necessarily generating the actual 3D structure. This problem is very akin to Natural Language Processing, and very similar methods can and have been used ([37](https://pubmed.ncbi.nlm.nih.gov/33897979/)).  Of particular note, and similarly to the examples above, are the use of transformer models with attention and self-supervised learning for protein sequence and structure analysis ([38](https://doi.org/10.1093/bioadv/vbad001),[39](https://doi.org/10.48550/arXiv.2006.15222)).  For SARS-CoV-2 Spike protein (this is the surface protein and most important for cell entry and therefore vaccine targets) and other viral surface proteins, Hie et al showed that language models could be used to predict viral escape mutations ([40](https://www.science.org/doi/10.1126/science.abd7331)).  This is important because it enables an assessment of the likely biological fitness of yet to be generated (or discovered) viral variants in nature, such as SARS-CoV-2 variants of concern.  Additionally, like natural language models, protein language models can also be used to generate new ‘fit’ sequences.  For example, in the recent pre-print by Gema et al, they propose the Vaxformer model ([41](https://arxiv.org/abs/2305.11194)), which is able to generate SARS-CoV-2 spike protein sequences that look like they have evolved antigenically from the circulating strains.  These sequences could then be used to help inform the next generation of vaccine design which incorporates sequence fragments.","metadata":{}},{"cell_type":"markdown","source":"# Summary and Conclusion\n\nIn this brief review of applications of artificial intelligence / machine learning models in the biosciences, we focussed on what has been used recently for biological data relating to the coronavirus pandemic.  Not unexpectedly, we have seen that techniques such as gradient boosted machines or random forests can be used for temporal-spatial disease occurrence problems, and that convolutional neural networks for image classification tasks can be fine tuned for specific medical imaging applications.  However, over the past few years Natural Language Processing models including Transformer models with attention, pre-training and self-supervised training have significantly progressed.  These advances have also translated into the Biosciences: by organising and synthesising the vast COVID-19 literature; and by using variants of Natural Language Processing models to model biological sequences including protein sequences and structures.  The use of NLP techniques for protein modelling is particularly scientifically significant, as it progresses the ability to predict viral genotypes and phenotypes in-silico, and consequently assists effective vaccine and drug design not only for the variants which exist now, but for possible future variants as well, with the ultimate goal being to be able to predict and mitigate against the next pandemic.","metadata":{}},{"cell_type":"markdown","source":"# References\n\n### Introduction\n([1](https://pubmed.ncbi.nlm.nih.gov/31978945/)) Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, Zhao X, Huang B, Shi W, Lu R, Niu P. A novel coronavirus from patients with pneumonia in China, 2019. New England journal of medicine. 2020 Jan 24. DOI: 10.1056/NEJMoa2001017\nhttps://www.nejm.org/doi/full/10.1056/nejmoa2001017\n\n([2](https://pubmed.ncbi.nlm.nih.gov/32015508/)) Wu F, Zhao S, Yu B, Chen YM, Wang W, Song ZG, Hu Y, Tao ZW, Tian JH, Pei YY, Yuan ML. A new coronavirus associated with human respiratory disease in China. Nature. 2020 Mar;579(7798):265-9.\nhttps://www.nature.com/articles/s41586-020-2008-3%7D \nhttps://doi.org/10.1038/s41586-020-2008-3 \n\n([3](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8393069/)) Tan W, Zhao X, Ma X, Wang W, Niu P, Xu W, Gao GF, Wu G. A novel coronavirus genome identified in a cluster of pneumonia cases—Wuhan, China 2019− 2020. China CDC weekly. 2020 Jan 1;2(4):61-2. [Weekly China CDC](https://weekly.chinacdc.cn/en/article/id/a3907201-f64f-4154-a19e-4253b453d10c?mod=article_inline) \n\n([4](https://pubmed.ncbi.nlm.nih.gov/31995857/)) Li Q, Guan X, Wu P, Wang X, Zhou L, Tong Y, Ren R, Leung KS, Lau EH, Wong JY, Xing X. Early transmission dynamics in Wuhan, China, of novel coronavirus–infected pneumonia. New England journal of medicine. 2020 Jan 29 \nhttps://www.nejm.org/doi/full/10.1056/NEJMOa2001316 \n\n### Pandemic epochs and pandemic data\n([5](https://doi.org/10.1038/s41564-020-0770-5)) Rambaut, A., Holmes, E.C., O’Toole, Á. et al. A dynamic nomenclature proposal for SARS-CoV-2 lineages to assist genomic epidemiology. Nat Microbiol 5, 1403–1407 (2020). https://doi.org/10.1038/s41564-020-0770-5\nhttps://www.nature.com/articles/s41564-020-0770-5\n\n([6](https://doi.org/10.1093/ve/veab064)) Áine O’Toole and others, Assignment of epidemiological lineages in an emerging pandemic using the pangolin tool, Virus Evolution, Volume 7, Issue 2, December 2021, veab064, https://doi.org/10.1093/ve/veab064\n\n([7](https://doi.org/10.1093/bioinformatics/bty407)) James Hadfield and others, Nextstrain: real-time tracking of pathogen evolution, Bioinformatics, Volume 34, Issue 23, December 2018, Pages 4121–4123, https://doi.org/10.1093/bioinformatics/bty407\n\n([8](https://doi.org/10.1126/science.abe6522)) Lavine JS, Bjornstad ON, Antia R. Immunological characteristics govern the transition of COVID-19 to endemicity. Science. 2021 Feb 12;371(6530):741-5. https://doi.org/10.1126/science.abe6522\n\n([9](https://doi.org/10.1371/journal.ppat.1010591)) Cohen LE, Spiro DJ, Viboud C. Projecting the SARS-CoV-2 transition from pandemicity to endemicity: Epidemiological and immunological considerations. PLoS Pathogens. 2022 Jun 30;18(6):e1010591 https://doi.org/10.1371/journal.ppat.1010591\n\n([10](https://doi.org/10.1371/journal.ppat.1010951)) Starr TN, Greaney AJ, Stewart CM, Walls AC, Hannon WW, Veesler D, Bloom JD. Deep mutational scans for ACE2 binding, RBD expression, and antibody escape in the SARS-CoV-2 Omicron BA. 1 and BA. 2 receptor-binding domains. PLoS pathogens. 2022 Nov 18;18(11):e1010951. https://doi.org/10.1371/journal.ppat.1010951\n\n([11](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9273038/)) Mykytyn, A.Z., Rissmann, M., Kok, A., Rosu, M.E., Schipper, D., Breugem, T.I., van den Doel, P.B., Chandler, F., Bestebroer, T.M., de Wit, M., van Royen, M.E., Molenkamp, R., Oude Munnink, B.B., de Vries, R.D., Geurtsvankessel, C., Smith, D.J., Koopmans, M.P., Rockx, B., Lamers, M.M., Fouchier, R.A., & Haagmans, B.L. (2022). Antigenic cartography of SARS-CoV-2 reveals that Omicron BA.1 and BA.2 are antigenically distinct. Science Immunology.  https://doi.org/10.1126%2Fsciimmunol.abq4450\n\n([12](https://doi.org/10.1172/JCI162192)) Aguilar-Bretones, M., Fouchier, R.A., Koopmans, M.P., & van Nierop, G.P. (2023). Impact of antigenic evolution and original antigenic sin on SARS-CoV-2 immunity. The Journal of Clinical Investigation, 133. https://doi.org/10.1172/JCI162192\n\n([13](https://pubmed.ncbi.nlm.nih.gov/36657480/)) Mykytyn, A.Z., Rosu, M.E., Kok, A., Rissmann, M., van Amerongen, G., Geurtsvankessel, C., de Vries, R.D., Munnink, B.B., Smith, D.J., Koopmans, M.P., Lamers, M.M., Fouchier, R.A., & Haagmans, B.L. (2023). Antigenic mapping of emerging SARS-CoV-2 omicron variants BM.1.1.1, BQ.1.1, and XBB.1. The Lancet. Microbe, 4, e294 - e295.\nhttps://doi.org/10.1016/s2666-5247(22)00384-6\n\n### Forecasting challenges\n([14](https://pubmed.ncbi.nlm.nih.gov/33458453/)) Gnanvi JE, Salako KV, Kotanmi GB, Glèlè Kakaï R. On the reliability of predictions on Covid-19 dynamics: A systematic and critical review of modelling techniques. Infect Dis Model. 2021;6:258-272. doi: 10.1016/j.idm.2020.12.008. Epub 2021 Jan 12. PMID: 33458453; PMCID: PMC7802527\n\n### Image Challenges\n([15](https://arxiv.org/abs/2107.08430)) Ge Z, Liu S, Wang F, Li Z, Sun J. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430. 2021 Jul 18.\nhttps://doi.org/10.48550/arXiv.2107.08430 \n\n([16](http://proceedings.mlr.press/v97/tan19a.html?ref=jina-ai-gmbh.ghost.io)) Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning 2019 May 24 (pp. 6105-6114). PMLR.\nhttp://proceedings.mlr.press/v97/tan19a.html?ref=jina-ai-gmbh.ghost.io pre-print: https://arxiv.org/abs/1905.11946\n\n### Text Mining the Biomedical literature\n([17](https://pubmed.ncbi.nlm.nih.gov/33279995/)) Wang LL, Lo K. Text mining approaches for dealing with the rapidly expanding literature on COVID-19. Brief Bioinform. 2021 Mar 22;22(2):781-799. doi: 10.1093/bib/bbaa296. PMID: 33279995; PMCID: PMC7799291. https://doi.org/10.1093/bib/bbaa296\n\n([18](https://arxiv.org/abs/1810.04805)) Devlin J, Chang MW, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 [cs.CL]. 2018 Oct 11, 24 May 2019. https://doi.org/10.48550/arXiv.1810.04805\n\n([19](https://proceedings.neurips.cc/paper/7181-attention-is-all)) Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. In: Advances in neural information processing systems. 2017, pp. 5998–6008\nPre-print version: https://doi.org/10.48550/arXiv.1706.03762\nhttps://proceedings.neurips.cc/paper/7181-attention-is-all \n\n([20](https://doi.org/10.1145/3458754)) Gu Y, Tinn R, Cheng H, Lucas M, Usuyama N, Liu X, Naumann T, Gao J, Poon H. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH). 2021 Oct 15;3(1):1-23.\nPre-print version: https://doi.org/10.48550/arXiv.2007.15779\nhttps://doi.org/10.1145/3458754 \n\n([21](https://doi.org/10.1186/s12859-022-04642-w)) Su, P., Vijay-Shanker, K. Investigation of improving the pre-training and fine-tuning of BERT model for biomedical relation extraction. BMC Bioinformatics 23, 120 (2022). https://doi.org/10.1186/s12859-022-04642-w\n\n([22](https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-023-02247-9)) Knafou, J., Haas, Q., Borissov, N. et al. Ensemble of deep learning language models to support the creation of living systematic reviews for the COVID-19 literature. Syst Rev 12, 94 (2023). https://doi.org/10.1186/s13643-023-02247-9\n\n### Models for Biological Sequences and Proteins\n(23) Durbin R, Eddy SR, Krogh A, Mitchison G. Biological sequence analysis: probabilistic models of proteins and nucleic acids. Cambridge university press; 1998 Apr 23\n\n([24](https://doi.org/10.1093/bib/bbk007)) Pedro Larrañaga and others, Machine learning in bioinformatics, Briefings in Bioinformatics, Volume 7, Issue 1, March 2006, Pages 86–112, https://doi.org/10.1093/bib/bbk007\n\n([25](https://doi.org/10.1038/s41579-021-00573-0)) Harvey, W.T., Carabelli, A.M., Jackson, B. et al. SARS-CoV-2 variants, spike mutations and immune escape. Nat Rev Microbiol 19, 409–424 (2021). https://doi.org/10.1038/s41579-021-00573-0\n\n([26](https://doi.org/10.1186/s11658-022-00352-6)) Saberiyan, M., Karimi, E., Khademi, Z. et al. SARS-CoV-2: phenotype, genotype, and characterization of different variants. Cell Mol Biol Lett 27, 50 (2022). https://doi.org/10.1186/s11658-022-00352-6\n\n([27](https://doi.org/10.1038/s41579-022-00841-7)) Carabelli, A.M., Peacock, T.P., Thorne, L.G. et al. SARS-CoV-2 variant biology: immune escape, transmission and fitness. Nat Rev Microbiol 21, 162–177 (2023). https://doi.org/10.1038/s41579-022-00841-7\n\n([28](https://www.science.org/doi/10.1126/science.abb2762)) Renhong Yan et al. ,Structural basis for the recognition of SARS-CoV-2 by full-length human ACE2.Science367,1444-1448(2020).DOI:10.1126/science.abb2762\n\n([29](https://doi.org/10.1038/s41423-020-0400-4)) Tai, W., He, L., Zhang, X. et al. Characterization of the receptor-binding domain (RBD) of 2019 novel coronavirus: implication for development of RBD protein as a viral attachment inhibitor and vaccine. Cell Mol Immunol 17, 613–620 (2020). https://doi.org/10.1038/s41423-020-0400-4\n\n([30](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7784839/)) Rezaei S, Sefidbakht Y, Uskoković V. Comparative molecular dynamics study of the receptor-binding domains in SARS-CoV-2 and SARS-CoV and the effects of mutations on the binding affinity. J Biomol Struct Dyn. 2022 Jul;40(10):4662-4681. doi: 10.1080/07391102.2020.1860829. Epub 2020 Dec 17. PMID: 33331243; PMCID: PMC7784839\n\n([31](https://doi.org/10.1038/s41586-019-1923-7)) Senior, A.W., Evans, R., Jumper, J. et al. Improved protein structure prediction using potentials from deep learning. Nature 577, 706–710 (2020). https://doi.org/10.1038/s41586-019-1923-7\n\n(32) Kryshtafovych A, Schwede T, Topf M, Fidelis K, Moult J. Critical assessment of methods of protein structure prediction (CASP)—Round XIII. Proteins: Structure, Function, and Bioinformatics. 2019 Dec;87(12):1011-20.\n\n([33](https://www.nature.com/articles/s41586-021-03819-2)) Jumper, J., Evans, R., Pritzel, A. et al. Highly accurate protein structure prediction with AlphaFold. Nature 596, 583–589 (2021). https://doi.org/10.1038/s41586-021-03819-2\n\n([34](https://www.predictioncenter.org/casp14/doc/CASP14_Abstracts.pdf)) Moult, J., Fidelis, K., Kryshtafovych, A., Schwede, T. & Topf, M. Critical assessment of techniques for protein structure prediction, fourteenth round. CASP 14 Abstract Book https://www.predictioncenter.org/casp14/doc/CASP14_Abstracts.pdf (2020).\n\n([35](https://doi.org/10.3389/frai.2022.875587)) Marcu ŞB, Tăbîrcă S, Tangney M. An overview of Alphafold's breakthrough. Frontiers in Artificial Intelligence. 2022 Jun 9;5:875587. https://doi.org/10.3389/frai.2022.875587 \n\n([36](https://pubmed.ncbi.nlm.nih.gov/34342271/))Bouatta N, Sorger P, AlQuraishi M. Protein structure prediction by AlphaFold2: are attention and symmetries all you need? Acta Crystallogr D Struct Biol. 2021 Aug 1;77(Pt 8):982-991. doi: 10.1107/S2059798321007531. Epub 2021 Jul 29. PMID: 34342271; PMCID: PMC8329862. https://pubmed.ncbi.nlm.nih.gov/34342271/\n\n([37](https://pubmed.ncbi.nlm.nih.gov/33897979/)) Ofer D, Brandes N, Linial M. The language of proteins: NLP, machine learning & protein sequences. Computational and Structural Biotechnology Journal. 2021 Jan 1;19:1750-8. https://pubmed.ncbi.nlm.nih.gov/33897979/\n\n([38](https://doi.org/10.1093/bioadv/vbad001)) Zhang S, Fan R, Liu Y, Chen S, Liu Q, Zeng W. Applications of transformer-based language models in bioinformatics: a survey. Bioinform Adv. 2023 Jan 11;3(1):vbad001. doi: 10.1093/bioadv/vbad001. PMID: 36845200; PMCID: PMC9950855. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9950855/ https://doi.org/10.1093/bioadv/vbad001\n\n([39](https://doi.org/10.48550/arXiv.2006.15222)) Vig, Jesse, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2020. “BERTology Meets Biology: Interpreting Attention in Protein Language Models,” June. http://arxiv.org/abs/2006.15222.\nhttps://doi.org/10.48550/arXiv.2006.15222\n\n([40](https://www.science.org/doi/10.1126/science.abd7331)) Brian Hie et al. ,Learning the language of viral evolution and escape.Science371,284-288(2021).DOI:10.1126/science.abd7331 https://www.science.org/doi/10.1126/science.abd7331\n\n([41](https://arxiv.org/abs/2305.11194)) Gema AP, Kobiela M, Fraisse A, et al. Vaxformer: Antigenicity-controlled Transformer for Vaccine Design Against SARS-CoV-2. arXiv; 2023.\nhttps://arxiv.org/abs/2305.11194\n\n\n","metadata":{}}]}