{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":" 0. Outline
\n 1. Introduction
\n 2. Data and Approach: an Overview
\n 3. Background Context
\n\n3.1 \tText Classification
\n3.2 \tSentiment Analysis
\n3.3 \tQuestion Answering
\n3.4 \tNamed Entity Recognition
\n3.5 \tText Summarization
\n3.6 \tLanguage Modelling
\n
\n 4. Recent Advances
\n 5. Innovation Pace
\n 6. Attacking Textual Datasets in Kaggle Competitions
\n 7. Conclusions and Future Directions
\n References
\n","metadata":{}},{"cell_type":"code","source":"'''Libraries to manipulate data'''\nimport numpy as np\nimport pandas as pd\nimport os\nimport math\nimport copy\nimport glob\n\n'''Labraries for Data Visualization'''\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nfrom wordcloud import WordCloud, STOPWORDS\nimport plotly.graph_objs as go \nfrom plotly.subplots import make_subplots\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nimport plotly.figure_factory as ff\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:42:57.839910Z","iopub.execute_input":"2023-07-04T16:42:57.841395Z","iopub.status.idle":"2023-07-04T16:43:01.215614Z","shell.execute_reply.started":"2023-07-04T16:42:57.841297Z","shell.execute_reply":"2023-07-04T16:43:01.214357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Introduction
\n\nThe Artificial Intelligence and Machine Learning fields are experiencing unprecedented developments that, whilst creating interesting prospects and opportunities, appear to generate concerns under several perspectives, especially for that susbtantial slice of the population that is not acquainted and at ease with the background and inner functioning of such emerging technologies. We will now delve deep into one of the most discussed topics in the recent years, i.e., Textual Data Processing (very closely related to Natural Language Processing), to hopefully make some clarity on a controversial yet astonishingly growing subfield of Machine Learning.
\n","metadata":{}},{"cell_type":"markdown","source":"# 2. Data and Approach: an Overview
\n\nFor a thorough investigation of both past and present pictures with an emphasis on recent developments we will make extensive use of data on benchmark datasets taken from Papers with Code [1], a watchfully curated and up to date collection of benchmarking analysis data and research paper results. Each dataset comprises a list of models/methodologies to process Text Data for the specific task the dataset belongs to, each model associated to a score and a date (i.e., the publication date of the paper where the model was tested on the given dataset).
\n\n
\n\nTo construct a clear and intelligible framework, we will focus on those datasets displaying benchmarks for models in the last two years (i.e., from 2021 onwards) [Section 4]. The big picture of recent developments is going to gradually take shape and, for each task facing notable developments in the Textual Data & NLP setting, we will complement such a picture by drawing parallels with developments experienced in the past decade, analyzing the pace at which innovation in the textual data processing setting was and is currently being achieved [Section 5]. Furthermore, we will explore the contribution of Kaggle community to the trends in the field by leveraging the numerous brilliant solutions to competitions that came with text data [Section 6].
\n\n
\n\nWhere are we? At what pace is the Machine Learning community learning with respect to textual Natural Language Processing, and where are we heading? We are soon unveiling this.
","metadata":{}},{"cell_type":"markdown","source":"# 3. Background Context
\n\nTextual Data Processing comes with a wide range of associated tasks, all following a common development path but each deviating and appearing with its own facets. Let us think about the advent of Transformer-based models that revolutionized the entire field of Natural Language Processing, but whose development unfolded following numerous ramifications depending on the specific task they had to tackle. If it is in fact true that a breakthrough-like discovery has the potential to impact several distinct fields of Machine Learning, it is equally actual that each downstream task comes with its own specifics and associated requirements.
\n\n
\n\nThe main tasks that involve text data and that we are going to tackle throughout our journey are the following:
\n\n- Text Classification
\n- Sentiment Analysis
\n- Question Answering
\n- Named Entity Recognition
\n- Text Summarization
\n- Language Modelling
\n\nLet us briefly overview these subfields of Textual Data Processing and the related most commonly-used assessment metrics and benchmarking datasets for the sake of a more accessible and thorough understanding of the whole essay; you can skip this section and proceed straight to Section 4 if you are acquainted with Natural Language Processing and Machine Learning performance assessment, or go through it for a more contextualized reading.
\n\n### 3.1 Text Classification
\n\nOne of the widest text-related downstream tasks, Text Classification consists in the categorization of textual data, either small sequences or large documents.
\n\n\n\n### 3.2 Sentiment Analysis
\n\nAlso known as Opinion Mining, Sentiment Analysis constitutes a magnificent example of real-world text data processing application: a technique that aims at detecting and classifying the emotional status and feeling expressed in a chunk of text. Various degrees of sentiment discrimination exist, but they generally consist of binary classification problems (Positive/Negative) or three-class multinomial classification (Positive/Negative/Neutral).
\n\n\n\n### 3.3 Question Answering
\n\nIntuitively, Question Answering consists in the automatic generation of possibly accurate answers to given questions.
\n\n\n\n### 3.4 Named Entity Recognition
\n\nNER, short for Named Entity Recognition, involves the identification and classification of named entities in text documents, where such named entities may consist of locations, dates, person names, or other predefined categories.
\n\n\n\n### 3.5 Text Summarization
\n\nProduces a shorter version of a document whilst (aiming at) keeping its main information and meaning.
\n\n\n\n### 3.6 Language Modelling
\n\nLanguage Modelling is the main engine behind a lot of today's innovations in the field of text data processing. It consists in the prediction of the next word/character in a document, and constitutes the most popular task to pretrain transformer-based models (transfer learning).
\n\n\n\n","metadata":{}},{"cell_type":"code","source":"def create_scatter(task, dataset_name, remove_samples):\n dataset_name = dataset_name\n try:\n dataset = pd.read_csv(f\"/kaggle/input/nlp-sota-benchmarks/{task}_{dataset_name}.csv\")\n except FileNotFoundError:\n task = 'named-entity-recognition-ner'\n dataset = pd.read_csv(f\"/kaggle/input/nlp-sota-benchmarks/{task}_{dataset_name}.csv\")\n dataset = dataset.dropna(subset=['score'])\n if remove_samples != 0:\n dataset = dataset.iloc[:-remove_samples,:] ### Depends on dataset ###\n dataset['SOTA'] = \"Non-SOTA\"\n dataset['year'] = pd.DatetimeIndex(dataset['date']).year\n dataset['date'] = pd.to_datetime(dataset['date'])\n #dataset = dataset.drop_duplicates('year').sort_values(by='year') ### If performed, only best model for each year is kept\n df_sota = dataset.drop_duplicates('year').sort_values(by='year')\n for idx, row in dataset.iterrows():\n if row['model'] in (df_sota['model'].tolist()):\n dataset['SOTA'][idx] = \"SOTA\"\n metric = dataset['metric'].iloc[0]\n scatter = px.line(dataset, x='date', y=['score'],\n hover_data=\"model\",color='SOTA'\n )\n\n return scatter, dataset_name, dataset, metric, df_sota\n\n\ndef create_plot(task, objects, plot_number, title_shift=1.30):\n fig.layout.title = f\"Recent Advancements - {task}\"\n fig.layout.title.font.size = 26\n fig.layout.title.font.color = \"black\"\n\n for object_idx in range(len(objects)):\n fig.add_trace(objects[object_idx][0].data[0], row=object_idx + 1, col=1)\n fig.add_trace(objects[object_idx][0].data[1], row=object_idx + 1, col=1)\n\n fig.add_hline(y=objects[object_idx][2]['score'][0], \n line_color='black', line_dash='dash', line_width=1,\n annotation_text='State-Of-The-Art Score', annotation_font_size=12,\n annotation_font_color='black', annotation_position='top left',\n row=object_idx + 1, col=1)\n\n\n fig.data[object_idx*2]['hovertemplate'] = fig.data[object_idx*2]['hovertemplate'].replace(\"value\", f\"{objects[object_idx][3]}\")\n fig.data[object_idx*2 + 1]['hovertemplate'] = fig.data[object_idx*2 + 1]['hovertemplate'].replace(\"value\", f\"{objects[object_idx][3]}\")\n\n for i, row in objects[object_idx][4].iterrows():\n fig.add_annotation(x=row['date'], y=row['score'],\n text=f\"{row['model']}\", xref=\"x\",yref=\"y\",\n showarrow=True,\n row=object_idx+1, col=1)\n\n fig.add_annotation(x=0.5, y=title_shift,\n text=f\"[Subplot {plot_number}.{object_idx + 1}] Dataset: {objects[object_idx][1]}\",\n showarrow=False, xref='x domain',yref='y domain', row=object_idx+1, col=1,\n font=dict(\n family=\"sans serif\",\n size=24,\n color=\"black\"\n ))\n\n for subplot_idx in range(len(fig.data)):\n fig.data[subplot_idx]['mode'] = 'markers'\n fig.data[subplot_idx]['showlegend'] = False\n if subplot_idx % 2 == 0:\n fig.data[subplot_idx]['marker']['size'] = 16\n fig.data[subplot_idx]['marker']['color'] = 'lime'\n else:\n fig.data[subplot_idx]['marker']['size'] = 4\n fig.data[subplot_idx]['marker']['color'] = 'black'\n \n return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:43:01.218851Z","iopub.execute_input":"2023-07-04T16:43:01.219768Z","iopub.status.idle":"2023-07-04T16:43:01.254083Z","shell.execute_reply.started":"2023-07-04T16:43:01.219685Z","shell.execute_reply":"2023-07-04T16:43:01.252691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Recent Advances
\n\nLet us now begin our true journey to unveil recent advances in the setting of textual data processing; as anticipated in section 2, only datasets displaying benchmarks for models developed in the last two years will be leveraged now, in section 4, to identify key insights in the current status of text data processing and related new developments.
\n\n### 4.1.1 Text Classification
\n\n","metadata":{}},{"cell_type":"code","source":"task = \"text-classification\"\nnum_datasets = 3\nplot_number = 1\n\nfig = make_subplots(rows=num_datasets, cols=1, vertical_spacing = 0.15,\n specs=[\n [{\"type\": \"scatter\"}] for dataset in range(num_datasets)\n ],\n )\n\nscatter1, dataset_name1, dataset1, metric1, df_sota1 = create_scatter(task=task, dataset_name='20news', remove_samples=2)\nscatter2, dataset_name2, dataset2, metric2, df_sota2 = create_scatter(task=task, dataset_name='trec-6', remove_samples=2)\nscatter3, dataset_name3, dataset3, metric3, df_sota3 = create_scatter(task=task, dataset_name='r8', remove_samples=2)\n\nobjects = [(scatter1, dataset_name1, dataset1, metric1, df_sota1),\n (scatter2, dataset_name2, dataset2, metric2, df_sota2),\n (scatter3, dataset_name3, dataset3, metric3, df_sota3)]\n\nfig = create_plot(task, objects, plot_number)\n \nfig.layout.xaxis['showgrid'] = False\nfig.layout.xaxis2['showgrid'] = False\nfig.layout.xaxis3['showgrid'] = False\n\nfig.layout.yaxis.title = f\"{metric1}\"\nfig.layout.yaxis2.title = f\"{metric2}\"\nfig.layout.yaxis3.title = f\"{metric3}\"\n\nfig.update_layout(\n template='plotly_white', height=750\n)\n\nfig.layout['annotations'][1].ay = 30\nfig.layout['annotations'][3].ay = 30\n\nfig.show()\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:43:01.256048Z","iopub.execute_input":"2023-07-04T16:43:01.256585Z","iopub.status.idle":"2023-07-04T16:43:04.007145Z","shell.execute_reply.started":"2023-07-04T16:43:01.256534Z","shell.execute_reply":"2023-07-04T16:43:04.005987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can notice a continuous improvement of the State-Of-The-Art benchmarked on 20news dataset [Subplot 1.1] in the last 5 years, reaching the apex in November 2022. Surprisingly, such a SOTA result is held by a Support Vector Machine -based methodology instead of a transformer-based model (very common contingency for a wide set of tasks, as we will soon see); the SVM+TF-IDF methodology [2] (A Comparison of SVM against Pre-trained Language Models (PLMs) for Text Classification Tasks, 2022, Wahba et al.) is specifically tested against several pretrained language models (BERT, DistilBERT, XLM, RoBERTa) and outperforms all of those much more complex architectures. \nAnother model, anyway, previously held the SOTA for one year and a half: presented in May 2021, RoBERTaGCN [3] (BertGCN: Transductive Text Classification by Combining GCN and BERT, Lin et al.) combines large-scale pretraining, one of the war-horses behind transformer-based models, with transductive learning, which consists in learning representations for both labelled and unlabelled data by propagating label information through a graph constructed upon the whole dataset, whose nodes are constituted by textual documents. Despite having being recently outperformed, such an approach poses new interesting perspectives and potential research directions, both for text classification tasks and broad textual data processing exploration.
\n","metadata":{}},{"cell_type":"markdown","source":"The Text Retrieval Conference 6 (TREC-6) dataset [Subplot 1.2] benchmarks multiple older CNN and LSTM -based approaches versus transformer-based machines. The SOTA is held by a Universal Sentence Encoder complemented with a Convolutional Neural Network, developed in 2018 (Universal Sentence Encoder, Cer et al.) [4]. A more recent approach is represented by Dual Contrastive Learning, presented by Chen et al. at the beginning of 2022 in the paper \"Dual Contrastive Learning: Text Classification via Label-Aware Data Augmentation\" [5]. Specifically, DualCL is a framework that combines contrastive learning and supervised learning by simultaneously learning the features of input samples and the parameters of classifiers in the same space. By treating classifier parameters as augmented samples, such a framework leverages contrastive learning to improve classification accuracy and learn enhanced representations. Despite not outperforming the State-of-the-Art, this novel approach spurs innovation towards a potentially fruitful direction, similarly as what happened for the previous benchmark with RoBERTaGCN.
","metadata":{}},{"cell_type":"markdown","source":"The paper \"Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasets\" (Karl and Scherp, 2022) [6] compares various techniques against the R8 text classification dataset [Subplot 1.3]; despite being the various analyzed models developed in the previous years, they are all associated to the year of paper publication (i.e., 2022); DeBERTa stands out among all those models, achieving the State-Of-The-Art both against them and the previously-tested ones. RoBERTa+GCN held SOTA from 2021 onwards, and some techniques involving Graph Neural Networks (ESGNN, SGNN) showed comparable results.
","metadata":{}},{"cell_type":"markdown","source":"\n
\n\n KEY INSIGHTS:\n\n\n \n - Performance superiority of transformer-based models over simpler methodologies is not guaranteed
\n - For some domain-specific settings and datasets (e.g., 20news), classical algorithms (e.g., SVM+TF-IDF) may outperform more complex models (e.g., RoBERTa) if well-implemented
\n - The novel approach of Dual Contrastive Learning shows potential for improving classification accuracy and learning enhanced representations
\n - Beating SOTA in certain high-performance settings (e.g., TREC-6) displays increasing complexity
\n - Techniques involving Graph Neural Networks show strong Text Classification performance, especially when coupled with Transformer-based models
\n
\n\n\n
\n\n\n
\n\n WHAT'S NEW?\n\n\n \n - RoBERTaGCN
\n - RoBERTa+DualCL
\n
\n\n\n
4.1.2 Sentiment Analysis
\n","metadata":{}},{"cell_type":"code","source":"task = \"sentiment-analysis\"\nnum_datasets = 3\nplot_number = 2\n\nfig = make_subplots(rows=num_datasets, cols=1, vertical_spacing = 0.15,\n specs=[\n [{\"type\": \"scatter\"}] for dataset in range(num_datasets)\n ],\n )\n\nscatter1, dataset_name1, dataset1, metric1, df_sota1 = create_scatter(task=task, dataset_name='sst-5-fine-grained', remove_samples=2)\nscatter2, dataset_name2, dataset2, metric2, df_sota2 = create_scatter(task=task, dataset_name='mr', remove_samples=0)\nscatter3, dataset_name3, dataset3, metric3, df_sota3 = create_scatter(task=task, dataset_name='imdb', remove_samples=2)\n\nobjects = [(scatter1, dataset_name1, dataset1, metric1, df_sota1),\n (scatter2, dataset_name2, dataset2, metric2, df_sota2),\n (scatter3, dataset_name3, dataset3, metric3, df_sota3)]\n\nfig = create_plot(task, objects, plot_number)\n \nfig.layout.xaxis['showgrid'] = False\nfig.layout.xaxis2['showgrid'] = False\nfig.layout.xaxis3['showgrid'] = False\n\nfig.layout.yaxis.title = f\"{metric1}\"\nfig.layout.yaxis2.title = f\"{metric2}\"\nfig.layout.yaxis3.title = f\"{metric3}\"\n\nfig.update_layout(\n template='plotly_white', height=750\n)\n\nfig.layout['annotations'][5].ay = -35\nfig.layout['annotations'][7].ay = -20\nfig.layout['annotations'][13].ay = 40\nfig.layout['annotations'][13].ax = 40\n\nfig.layout['annotations'][20].ay = -20\nfig.layout['annotations'][27].ay = -20\nfig.layout['annotations'][29].ay = -40\nfig.layout['annotations'][33].ay = 40\n\nfig.show()\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:44:43.489931Z","iopub.execute_input":"2023-07-04T16:44:43.490475Z","iopub.status.idle":"2023-07-04T16:44:44.454092Z","shell.execute_reply.started":"2023-07-04T16:44:43.490437Z","shell.execute_reply":"2023-07-04T16:44:44.452434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can easily notice that the models upon which state of the art performance on sst-5-fine-grained dataset [Subplot 2.1] is built are extremely similar to those seen for Text Classification and, again, the constantly-improving SOTA appears to reach its current apex with RoBERTa-involving techniques: RoBERTa Large + Self Explaining and RoBERTa Large + Heinsen Routing. Interesting to notice how, so far, RoBERTa was a crucial building block in multiple innovative developments in the world of textual data processing. The latter was proposed in 2022 by F.A. Heinsen (An Algorithm for Routing Vectors in Sequences) [7], and basically consists in the transformation, by means of a routing algorithm, of a sequence of vectors into a new sequence with a specific length and vector size. The proposed implementation is enough optimized that it significantly reduces parameters, computation, and memory usage, allowing relatively long sequence routing and better results, eventually yielding a SOTA model that outperforms all the previous approaches tested against this benchmark, including another RoBERTa-Large.
","metadata":{}},{"cell_type":"markdown","source":"Moving on to the MR (Movie Reviews) dataset [Subplot 2.2], the SOTA is currently held by VLAWE [8], and humongous progress has been done from Feb 2017 (78% Accuracy) to Feb 2019 (93% Accuracy).\nA new approach called EFL (Entailment as Few-Shot Learner, Wang et al.) [9] has been proposed in 2021 that challenges the VLAWE technique and that, despite not outperforming the previous SOTA, yields comparable capabilities. Such an approach appears to be so successful by reformulating NLP tasks as entailment problems, then fine-tuning the backbone Language Model with a much lower amount of samples; overall, entailment involves the analysis of the relationship between two sentences (a Premise and a Hypothesis) and classifying it as Entailment (Hypothesis can be inferred from Premise), Contradiction (the Hypothesis contradicts the Premise), or Neutral (no logical relationships between Hypothesis and Premise). According to the authors, this methodology allows to expand textual learning in new NLP subfields with much lower computational overload and training requirements and, according to the previous chart, the results in terms of performance are promising.
","metadata":{}},{"cell_type":"markdown","source":"Results on the imdb benchmark [Subplot 2.3] show smaller gaps among models due to the accuracy being already above 90% in the end of 2014; XLNet holds SOTA with extremely small detachment from two recently-presented approaches that we already observed in the previous sentiment analysis datasets, i.e., EFL and RoBERTa Large + Heinsen Routing.
","metadata":{}},{"cell_type":"markdown","source":"\n
\n\n KEY INSIGHTS:\n\n\n \n - RoBERTa-based techniques, (e.g., RoBERTa Large + Self Explaining, RoBERTa Large + Heinsen Routing, RoBERTa + EFL) appear to be playing a crucial role in achieving state-of-the-art performance in sentiment analysis
\n - The innovative approach of Heinsen Routing algorithm has achieved strong results by optimizing parameters, computation, and memory usage
\n - Sentiment Analysis performance improved steadily and consistently until the end of the previous decade
\n - Some innovative and diverse approches are being developed that challenge previous SOTA, but benchmarks are becoming difficult to outperform
\n - Formulating NLP tasks as entailment problems shows promising results by fine-tuning a backbone Language Model with fewer samples, enabling expansion in NLP subfields with lower computational overload
\n - Few-shot learning becoming increasingly popular and displaying promising performances
\n
\n\n\n
\n\n\n
\n\n WHAT'S NEW?\n\n\n \n - RoBERTa + Heinsen Routing
\n - RoBERTa + EFL
\n
\n\n\n
4.1.3 Question Answering
\n\nA larger amount of QA datasets exists on which recent approaches have been tested, and whose SOTA has been recently updated, compared to other tasks.
","metadata":{}},{"cell_type":"code","source":"task = \"question-answering\"\nnum_datasets = 8\nplot_number = 3\n\nfig = make_subplots(rows=num_datasets, cols=1,\n specs=[\n [{\"type\": \"scatter\"}] for dataset in range(num_datasets)\n ],\n )\n\nscatter1, dataset_name1, dataset1, metric1, df_sota1 = create_scatter(task=task, dataset_name='wikiqa', remove_samples=0)\nscatter2, dataset_name2, dataset2, metric2, df_sota2 = create_scatter(task=task, dataset_name='webquestions', remove_samples=0)\nscatter3, dataset_name3, dataset3, metric3, df_sota3 = create_scatter(task=task, dataset_name='triviaqa', remove_samples=0)\nscatter4, dataset_name4, dataset4, metric4, df_sota4 = create_scatter(task=task, dataset_name='squad11', remove_samples=7)\nscatter5, dataset_name5, dataset5, metric5, df_sota5 = create_scatter(task=task, dataset_name='pubmedqa', remove_samples=1)\nscatter6, dataset_name6, dataset6, metric6, df_sota6 = create_scatter(task=task, dataset_name='natural-questions', remove_samples=0)\nscatter7, dataset_name7, dataset7, metric7, df_sota7 = create_scatter(task=task, dataset_name='boolq', remove_samples=0)\nscatter8, dataset_name8, dataset8, metric8, df_sota8 = create_scatter(task=task, dataset_name='hotpotqa', remove_samples=26)\n\nobjects = [(scatter1, dataset_name1, dataset1, metric1, df_sota1),\n (scatter2, dataset_name2, dataset2, metric2, df_sota2),\n (scatter3, dataset_name3, dataset3, metric3, df_sota3),\n (scatter4, dataset_name4, dataset4, metric4, df_sota4),\n (scatter5, dataset_name5, dataset5, metric5, df_sota5),\n (scatter6, dataset_name6, dataset6, metric6, df_sota6),\n (scatter7, dataset_name7, dataset7, metric7, df_sota7),\n (scatter8, dataset_name8, dataset8, metric8, df_sota8),]\n\nfig = create_plot(task, objects, plot_number, title_shift=1.20)\n \nfig.layout.xaxis['showgrid'] = False\nfig.layout.xaxis2['showgrid'] = False\nfig.layout.xaxis3['showgrid'] = False\nfig.layout.xaxis4['showgrid'] = False\nfig.layout.xaxis5['showgrid'] = False\nfig.layout.xaxis6['showgrid'] = False\nfig.layout.xaxis7['showgrid'] = False\nfig.layout.xaxis8['showgrid'] = False\n\nfig.layout.yaxis.title = f\"{metric1}\"\nfig.layout.yaxis2.title = f\"{metric2}\"\nfig.layout.yaxis3.title = f\"{metric3}\"\nfig.layout.yaxis4.title = f\"{metric4}\"\nfig.layout.yaxis5.title = f\"{metric5}\"\nfig.layout.yaxis6.title = f\"{metric6}\"\nfig.layout.yaxis7.title = f\"{metric7}\"\nfig.layout.yaxis8.title = f\"{metric8}\"\n\nfig.update_layout(\n template='plotly_white', height=2000\n)\n\nfig.layout['annotations'][3].ay = 40\nfig.layout['annotations'][16].ay = 30\nfig.layout['annotations'][18].ay = 30\nfig.layout['annotations'][35].ay = 30\nfig.layout['annotations'][44].ay = 80\nfig.layout['annotations'][48].ay = 60\nfig.layout['annotations'][57].ay = -35\nfig.layout['annotations'][65].ay = -40\nfig.layout['annotations'][68].ay = 40\n\nfig.show()\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:44:55.854246Z","iopub.execute_input":"2023-07-04T16:44:55.854786Z","iopub.status.idle":"2023-07-04T16:45:00.036649Z","shell.execute_reply.started":"2023-07-04T16:44:55.854748Z","shell.execute_reply":"2023-07-04T16:45:00.035405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"According to the picture witnessed by this chart, QA performance drastically improved after 2018, probably due to the already mentioned revolutionary advent of transformer-based language models. Exactly 1 year ago, in 2022, a new QA SOTA has been established on wikiqa dataset [Subplot 3.1] by RLAS-BIABC [10], a novel approach proposed by Gharagozlou et al. in the paper \"RLAS-BIABC: A Reinforcement Learning-Based Answer Selection Using the BERT Model Boosted by an Improved ABC Algorithm\"; such a methodology combines attention-based LSTM, BERT word embedding, an improved artificial bee colony (ABC) algorithm for pretraining, and a reinforcement learning-based algorithm for training, and challenges (by an extremely tiny margin) the previous state-of-the-art achieved by TANDA-RoBERTa in the end of 2019. Apart from the State-of-the-Art performance achieved by this technique, what makes it innovative is that, unlike previously-existing approaches, it treats Answer Selection as a sequential decision-making process to address the issue of imbalanced classification.
","metadata":{}},{"cell_type":"markdown","source":"The WebQuestions QA dataset [Subplot 3.2] shows a disparate picture, with some recent approaches outperforming older ones (FiE+PAQ holding the SOTA) and some others displaying much lower performance. The gap in performance between the less recent models and newer ones is not humongous this time since the very first approach tested on this dataset involved the use of transformer-based models. Also worth noticing the relatively low performance of PaLM-like approaches, become popular between 2022 and 2023 and probably performing better in different benchmarks. At the SOTA level stand, close to FiE+PAQ, FiE and FiDO. Specifically, FiE+PAQ [11] (FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering, Kedia et al., 2022) stands out by combining Fusion-in-Encoder (a technique for the ingestion, in transformer decoders, of combined information from multiple passages) with PAQ [12], a data augmentation technique proposed by Lewis et al. (PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them, 2021) that provides 65M automatically-generated question-answer pairs. Combining the innovative and already SOTA Fusion-in-Encoder with such a beneficial augmentation yields a 3.9 improvement in Exact Match score over the raw FiE.
","metadata":{}},{"cell_type":"markdown","source":"TriviaQA benchmark [Subplot 3.3] is dominated by PaLM 2-L one-shot, the SOTA being achieved in 2023. Before that, the SOTA was held by PaLM-540B few-shot in 2022, after dethroning GLaM (rose to prominence in the end of 2021). But what are all these techniques about? Let us briefly overview them to better understand what led them to current performances.
\n \n- GLaM [13] - Proposed by Du et al. in 2021 in the paper GLaM: Efficient Scaling of Language Models with Mixture-of-Experts, the described family of models employs a sparsely-activated mixture-of-experts architecture (i.e., an ensemble of various diverse models each specialized in a particular subset or aspect of the overall problem) and achieves increased model capacity while significantly reducing training costs compared to dense models. According to the authors, the largest GLaM model has 1.2 trillion parameters, 7 times larger than GPT-3, but consumes only 1/3 of the energy for training and requires half the computation for inference!
\n- PaLM [14] - Initially proposed by a wide cohort of Google Research scientists (Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, 2022), PaLM leverages few-shot learning to reap the benefits of large language models without humongous computational overload, allowing for a facilitated adaptation to downstream tasks
\n- PaLM 2 [15] - Introduced in the PaLM 2 Technical Report by Google Research in May 2023, this model is the direct evolution of its predecessor PaLM, and it is pretrained on a wider mixture of tasks. Not only the paper shows much stronger performance than its predecessor, but a faster inference execution is exhibited, as well
\n\nThe rapid unfolding of these approaches denotes a dynamic environment, with an increase of 10 EM points in just one year and a half, and a 5 EM points increase in 1 year with a more advanced version of the same model.
","metadata":{}},{"cell_type":"markdown","source":"SQuAD dataset [Subplot 3.4] is undeniably one of the most adopted to test new Question Answering systems, with hundreds of approaches assessed against it between 2016 and 2020. Despite performance improved extremely slightly in the very recent years, current SOTA is held by ANNA [16], a novel approach that outperformed LUKE (the previous top-performing model) in 2021 and kept the State-of-the-Art against the more recent LinkBERT [17], proposed in 2022. Where does the particularly strong performance demonstrated by ANNA come from? Proposed by Jun et al. (ANNA: Enhanced Language Representation for Question Answering), this approach couples a novel pre-training task with a modified Attention mechanism that prioritizes neighboring tokens to capture richer context in language modelling upon pre-training; indeed, ANNA stands for Approach of Noun-phrase based language representation with Neighbor-aware Attention. The paper also investigates the individual usage of improvement techniques (data processing, pre-training task, attention mechanisms) and highlights how combining said techniques is beneficial in terms of performance. Again about pre-training, in their paper \"LinkBERT: Pretraining Language Models with Document Links\" Yasunaga et al. presented a novel pre-training approach that exploits the existence of links between documents (such as hyperlinks), treating a textual corpus as a graph of interconnected documents and generating LM inputs by placing linked documents in the same context.
\n\n
\n\n→ It looks like pre-training plays a non-negligible role in shaping the recent advancements in Question Answering.
","metadata":{}},{"cell_type":"markdown","source":"The PubMedQA benchmark [Subplot 3.5] is relatively recent and draws a comparison between machine learning methodologies and human performance. The latter was indeed beaten in July 2022 by Codex 5-shot CoT, and several methodologies were developed afterwards, the most remarkable being BioGPT [18] (whose large version currently holds the SOTA) and Med-PaLM [19]. The former was presented by Luo et al. in the paper \"BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining\" and constitute a pretty successfull example of domain-specific generative transformer; BioGPT indeed uses the increasingly popular GPT architecture, pre-trained on large amounts of biomedical literature. Such a model provides for the need of generation ability in the biomedical setting, missing from existing domain-specific models (such as BioBERT). Flan-PaLM [20] preceded BioGPT both in terms of release and in performance, and both by a relatively small margin. Again developed by Google Research at DeepMind, Flan-PaLM (\"Large Language Models Encode Clinical Knowledge\") was proposed towards the end of 2022 and constitutes an instruction-tuned variant of PaLM. Extremely recent is the proposal of Med-PaLM 2 by a cohort of DeepMind researchers (Google) in the paper \"Towards Expert-Level Medical Question Answering with Large Language Models\"; its predecessor was the very first automated question answering system to achieve a 'passing' level score in US Medical Licensing Examination (USMLE) style questions, and the newer version conveys to the biomedical domain the enhancements attained by PaLM-2.
\n","metadata":{}},{"cell_type":"markdown","source":"State-of-the-Art on Natural Questions benchmark [Subplot 3.6] consistently improved until the development of Atlas [21], the approach that dethronized the innovative four-stage R2-D2 [22] methodology and holding the SOTA since August 2022. With the exception of Atlas, all the models tested against this specific dataset after R2-D2 (2021) displayed much lower performance compared to the two aforementioned SOTA-level techniques. R2-D2 was initially proposed by Fajcik et al. in the paper \"R2-D2: A Modular Baseline for Open-Domain Question Answering\" and consists in a four-stage open-domain QA methodology, specifically composed by the following four modules: Retriever, Passage Reranker, Extractive Reader, Generative Reader; an ad-hoc mechanism eventually combines the predictions of all of the aforementioned components into a unified final output that, as we observed in the chart, yields a considerably strong and SOTA-level open-domain QA performance. Presented by Izacard et al. (Atlas: Few-shot Learning with Retrieval Augmented Language Models), Atlas consistutes a well-made example of few-shot QA approach, a pre-trained retrieval language model that exhibited the capability to execute complex QA tasks with a minimal amount of training samples. This last example higlights the shifting focus of today's research efforts, in the world of automated textual data processing systems, towards increasing accessibility and reducing computational requirements coming with such systems.
\n","metadata":{}},{"cell_type":"markdown","source":"Performance on BoolQ [Subplot 3.7] dataset is much flatter compared to previously-analyzed QA datasets, also compared to other datasets whose records begin after the advent of transformer-based models (such as PubMedQA and Natural Questions). Apart from the already-observed PaLM approaches, an interesting result is constituted by ST-MoE-32B [23] achieving SOTA, despite by a small margin from its predecessor (T5), in 2022. Proposed by Zoph et al. in ST-MoE: Designing Stable and Transferable Sparse Expert Models, such an approach is devised as a potential solution to the high computational cost brought by the advent of Scale approaches; this technique adopts Mixture-of-Experts (MoE), similarly as we saw for GLaM, and, studying the instability issues affecting Switch Transformers (sparse humongous models such as GLaM itself), specifically employs strategies to address such issues and drawbacks encountered with said approaches: the major technique employed to improve stability being Router Z-Loss, yielding a \"Stable Transferable\" model (ST-MoE).
\n","metadata":{}},{"cell_type":"markdown","source":"The last QA dataset we are tackling is HotPotQA [Subplot 3.8] and, despite the SOTA has not been beaten since 2020 (currently held by BigBird), the two highest-performing approaches after BigBird are relatively recent: AISO [24] and Chain-of-Skills [25]. Developed by Zhu et al. (Adaptive Information Seeking for Open-Domain Question Answering), the former is constituted by an innovative adaptive information-seeking strategy, specifically suited for open-domain QA. According to the paper, the authors model the entire retrieval and answer process as a partially observed Markov decision process, where some retrieval operations and one answer operation are defined as actions and, by learning a policy, AISO dynamically selects the most suitable retrieval action at each step based on the collected evidence and reformulated query whilst, in the event sufficient evidence is available, it directly provides an answer. Compared to the already-existing approaches that rely on predefined strategies, either using the same retrieval function repeatedly or following a fixed order of different retrieval functions, this methodology cater to the diverse requirements of a wide variety of questions, explaining the satisfying performance. The extremely recent Chain-of-Skills was presented by Ma et al. in the paper \"A Configurable Model for Open-domain Question Answering\", and tries to address a non-negligible problem emerged very often in the recent past: an excessive focus on customized methods for different datasets, resulting in limited model transferability and scalability. It indeed consists of a modular retriever that incorporates individual modules representing key skills, which can be reused across datasets (hence the name): this allows flexible skill configurations tailored to the target domain, thereby enhancing performance. Novelties are yet to run out: the parameterization adopted by Chain-of-Skills has been specifically drawn from the sparse Transformer mechanism, leading to more effective modularization; eventually, performance boosts are achieved combined the aforementioned innovations with self-supervised pre-training on Wikipedia corpus and multi-task fine-tuning on several Open-Domain Question-Answering datasets. Despite standing slightly lower than the SOTA BigBird in the overall benchmark picture, these two innovative developments pave the way for previously unthought directions and research opportunities.
","metadata":{}},{"cell_type":"markdown","source":"\n
\n\n KEY INSIGHTS:\n\n\n \n - The advent of transformer-based language models has led to significant improvements in QA performance since 2018
\n - RLAS-BIABC, a novel approach combining attention-based LSTM, BERT word embedding, improved ABC algorithm, and reinforcement learning, achieved promising results
\n - Treating Answer Selection as a sequential decision-making process to address the issue of imbalanced classification
\n - Fusion-in-Encoder and Fusion-in-Decoder approaches consistently showed SOTA and close-to-SOTA performance despite the already-existing transformer-based Question Answering systems; combininb FiE with PAQ augmentation improves results even further
\n - The same models can display diversified performance depending on the benchmark: PaLM-like approaches achieve below-average scores on WebQuestions and NaturalQuestions, SOTA on triviaqa, close to SOTA in BoolQ and PubMedQA
\n - Propitious results achieved by MoE (mixture-of-experts) architecture: increases model capacity while reducing training costs compared to dense models; anyway, drawbacks in terms of stability are observed
\n - Achieving State-of-the-Art in recent days involves more than ever novel pre-training (e.g., ANNA) and language representation (e.g., LinkBERT) approaches
\n - Router Z-Loss to enhance stability and overcome drawbacks proved effective in Question Answering
\n - Adaptive information-seeking strategy based on a partially observed Markov decision process effectively allows dynamic selection of retrieval actions and provides answers when sufficient evidence is available
\n - Adopting a modular retriever with individual modules representing key skills addresses the problem of limited transferability and scalability in QA models (Chain-of-Skills)
\n - Machine learning -based automated textual data processing systems outperformed humans in medical question answering, with BioGPT being the most valiant champion of such a phenomenon
\n - Consistent improvement in Natural Questions until 2022, with Atlas (from Meta AI) holding the SOTA from then on
\n - The rapid advancements observed overall showcase a dynamic environment, with significant performance improvements in a short period
\n
\n\n\n
\n\n\n
\n\n WHAT'S NEW?\n\n\n \n - RLAS-BIABC
\n - FiE+PAQ, FiE, FiDO
\n - PaLM approaches (PaLM, PaLM 2, Flan-PaLM, Med-PaLM, Med-PaLM 2)
\n - ANNA
\n - LinkBERT
\n - BioGPT
\n - R2-D2
\n - Atlas
\n - AISO
\n - Chain-of-Skills
\n
\n\n\n
4.1.4 Named Entity Recognition
\n\nThe NER task has become increasingly popular due to its strong utility at extracting information from textual data, that is unstructured. Its growing popularity goes hand in hand with several benchmarking datasets from various domains, many of them displaying performance improvements in the very last years.
","metadata":{}},{"cell_type":"code","source":"task = \"named-entity-recognition\"\nnum_datasets = 6\nplot_number = 4\n\nfig = make_subplots(rows=num_datasets, cols=1, vertical_spacing=0.05,\n specs=[\n [{\"type\": \"scatter\"}] for dataset in range(num_datasets)\n ],\n )\n\nscatter1, dataset_name1, dataset1, metric1, df_sota1 = create_scatter(task=task, dataset_name='wnut-2017', remove_samples=0)\nscatter2, dataset_name2, dataset2, metric2, df_sota2 = create_scatter(task=task, dataset_name='genia', remove_samples=1)\nscatter3, dataset_name3, dataset3, metric3, df_sota3 = create_scatter(task=task, dataset_name='ontonotes-v5', remove_samples=0)\nscatter4, dataset_name4, dataset4, metric4, df_sota4 = create_scatter(task=task, dataset_name='bc5cdr-chemical', remove_samples=0)\nscatter5, dataset_name5, dataset5, metric5, df_sota5 = create_scatter(task=task, dataset_name='bc5cdr', remove_samples=0)\nscatter6, dataset_name6, dataset6, metric6, df_sota6 = create_scatter(task=task, dataset_name='conll-2003', remove_samples=4)\n\nobjects = [(scatter1, dataset_name1, dataset1, metric1, df_sota1),\n (scatter2, dataset_name2, dataset2, metric2, df_sota2),\n (scatter3, dataset_name3, dataset3, metric3, df_sota3),\n (scatter4, dataset_name4, dataset4, metric4, df_sota4),\n (scatter5, dataset_name5, dataset5, metric5, df_sota5),\n (scatter6, dataset_name6, dataset6, metric6, df_sota6)]\n\nfig = create_plot(task, objects, plot_number, title_shift=1.20)\n \nfig.layout.xaxis['showgrid'] = False\nfig.layout.xaxis2['showgrid'] = False\nfig.layout.xaxis3['showgrid'] = False\nfig.layout.xaxis4['showgrid'] = False\nfig.layout.xaxis5['showgrid'] = False\nfig.layout.xaxis6['showgrid'] = False\n\nfig.layout.yaxis.title = f\"{metric1}\"\nfig.layout.yaxis2.title = f\"{metric2}\"\nfig.layout.yaxis3.title = f\"{metric3}\"\nfig.layout.yaxis4.title = f\"{metric4}\"\nfig.layout.yaxis5.title = f\"{metric5}\"\nfig.layout.yaxis6.title = f\"{metric6}\"\n\nfig.update_layout(\n template='plotly_white', height=1500\n)\n\nfig.layout['annotations'][33].ay = -45\nfig.layout['annotations'][68].ay = -45\n\nfig.show()\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:56:35.189401Z","iopub.execute_input":"2023-07-04T16:56:35.189851Z","iopub.status.idle":"2023-07-04T16:56:37.784759Z","shell.execute_reply.started":"2023-07-04T16:56:35.189818Z","shell.execute_reply":"2023-07-04T16:56:37.783555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The first dataset we are observing (WNUT 2017) [Subplot 4.1] clearly shows the consistent increments in NER performance in the last years. The SOTA is currently held by CL-KL [26], tested and presented in 2021 in the paper \"Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning\" by Wang et al. Such an approach leverages external contexts by retrieving and selecting semantically relevant texts through a search engine, using the original sentence as the query; according to the authors' findings, by constructing a retrieval-based input view, which involves concatenating the sentence with its retrieved external contexts, significantly improved performance can be achieved compared to using only one input view constituted by the sentence. This is only a part of what makes CL-KL so strong in Named Entity Recognition: a training method called Cooperative Learning is exploited that encourages both input views to generate similar contextual representations or output label distributions, thereby further boosting the performance of the models for both of the aforementioned views. Cooperative Learning achieves remarkable results when applied to other NER benchmarking datasets, as well, and we will indeed encounter it again soon. The very last record is from end 2022 and belongs to T-NER XLM-RoBERTa Large [27], a NER framework developed by Ushio et al. (An All-Round Python Library for Transformer-based Named Entity Recognition). Anyway, it doesn't manage to outperform the aforementioned approach.
","metadata":{}},{"cell_type":"markdown","source":"The Genia dataset [Subplot 4.2] shows the DeepStruct multi-task [28] approach, presented in mid 2022, beating the previous State-of-the-Art Biaffine-NER. DeepStruct was proposed by Wang et al. in the paper \"DEEPSTRUCT: Pretraining of Language Models for Structure Prediction\" and represents a groundbreaking methodology that aims at enhancing the capability of language models to comprehend structures. Departing from conventional practices that depend on task-specific adjustments during model fine-tuning, this novel approach revolves around the pretraining of language models on corpora that are independent of specific tasks. This enables the models to generate structures from text, consequently facilitating the transfer of acquired knowledge pertaining to structural tasks without the requirement for explicit fine-tuning.\nThe model coupled with such a pretraining technique is about 10B parameters large and, whilst the non-finetuned version displays slightly lower performance than previous SOTA, successfully outperforms it upon finetuning. The improvement is slight, as is the improvement in overall performance that, albeit steady, consists in around 8% increase in F1 score since July 2018.
","metadata":{}},{"cell_type":"markdown","source":"The OntoNotes V5 dataset [Subplot 4.3] allows us to clearly delineate a picture of Named Entity Recognition progress throughout the last 10 years: with a more dense concentration of tested approaches in the very recent years, humongous advance has been made since the advent of BERT-leveraging approaches, the apex being reached in November 2019 by BERT-MRC+DSC. Despite not beating SOTA, comparable performance has been reached in the last couple of years by two different approaches: PL-Marker [29] and Baseline+BS [30]. PL-Marker was presented by Ye et al. in the paper \"Packed Levitated Marker for Entity and Relation Extraction\" with the purpose of addressing the disregard for the crucial interrelation between spans or pairs, one of the main drawbacks that characterize the recent approaches in entity extraction, mostly focused on improving span representations within pre-trained encoders. The authors introduce a neighborhood-oriented packing strategy, which takes into account the neighboring spans as a whole to effectively model entity boundary information: this holistic consideration of neighboring spans allows for a more comprehensive understanding of the context. Additionally, for more intricate span pair classification tasks, a subject-oriented packing strategy is proposed that groups each subject with its corresponding objects, enabling the modeling of interrelations specifically among span pairs sharing the same subject. Another problem often encountered in Named Entity Recognition is the over-confidence suffered from the models, an issue that can easily hamper both model performance and calibration. Proposed by Zhu et al., Baseline+BS is constituted by a novel regularization technique called Boundary Smoothing (BS), inspired by Label Smoothing and aiming at addressing the inherent ambiguity in NER boundary annotation. The overall approach involves the reallocation of entity probabilities from annotated spans to the surrounding ones, effectively spreading the confidence and reducing the overemphasis on specific boundaries. The paper couples this technique with a strong baseline model, indeed achieving results that challenge the State-of-the-Art.
","metadata":{}},{"cell_type":"markdown","source":"State-of-the-Art is currently held by Spark NLP (an Apache Spark-based BiLSTM-CNN model) on the BC5CDR Chemical benchmark [Subplot 4.4], reached at the end of 2020 and never overcome thenceforth. Anyway, two interesting methodologies have been tested hereafter, reaching very good performance and potential would-be SOTA without Spark NLP dominating the benchmark: SciFive-Large [31] and HGN [32]. SciFive-Large was first proposed in the paper \"SciFive: a text-to-text transformer model for biomedical literature\" (Phan et al.), a T5-based model that has been adapted to the biomedical domain by undergoing domain-specific pre-training on large-scale biomedical corpora to answer the growing demand for NLP methods specifically tailored to the unique characteristics of biomedical language derived from the increasing necessity to leverage NLP methods for biomedical purposes (e.g., extract insights from scientific literature and analyzing electronic health records).\nHGN was tested against the wnut-2017 dataset, as well, with a very similar pattern: strong but below-SOTA results. Presented by Hu et al. in the paper \"Hero-Gang Neural Model For Named Entity Recognition\", HGN comprises two key modules: the Hero module and the Gang module. The Hero module employs a Transformer-based encoder to leverage the benefits of self-attention, enabling effective capturing of global context information. On the other hand, the Gang module utilizes a multi-window recurrent module designed to extract local features and position information, guided by the insights from the Hero module. At the end of 2022 several Bio-BERT-like approaches (e.g., BioDistilBERT, BioMobileBERT, etc.) were tested against this dataset, as well, but none of them outperformed neither Spark NLP nor the two aforementioned lower-performance approaches.
\n","metadata":{}},{"cell_type":"markdown","source":"Proposed in the second half of 2022, BINDER [33] dominates the non-chemical BC5CDR dataset [Subplot 4.5], beating the previous SOTA-holding approach CL-L2 [26], also relatively recent; CL-L2 is a cooperative learning-based approach similar to the previously-observed CL-KL and if for the WNUT-2017 benchmark such a family of models was able to hold its throne, in this case cooperative learning was instead ousted by BINDER. The latter was proposed by Zhang et al. in the paper \"Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning\" and consists in a framework built upon contrastive learning to map candidate text spans and entity types into a shared vector representation space. Unlike conventional methods that focus on sequence labeling or span classification, BINDER tackles NER as a representation learning problem, with the goal of maximizing the similarity between the vector representations of an entity mention and its corresponding type. This unique perspective leads such an approach to handle both nested and flat NER cases efficiently and effectively utilize self-supervision signals, even in the presence of noise. Another innovation brought by this methodology is the substitution of the conventional approach of labeling all non-entity spans as the same class \"Outside\" (O) with a novel dynamic thresholding loss that is learned alongside the standard contrastive loss. This novel approach enhances the model's ability to discern between entities and non-entities more accurately.
","metadata":{}},{"cell_type":"markdown","source":"One of the most-adopted benchmarking datasets for model assessment, CoNLL [Subplot 4.6] presents a wide range of architectures and techniques tested against. The State-of-the-Art is recent and held by ACE+document-context [34], that beat the previous SOTA by FLERT XLM-R; this approach was proposed by Wang et al. (Automated Concatenation of Embeddings for Structured Prediction) in February 2021 with the goal of addressing the issue of embedding type selection: indeed, the authors said, recent research has shown that combining different types of embeddings can yield even better results, but selecting the most effective combination of embeddings is a challenging task, as it depends on the specific task and the available embedding options. ACE automates the process of finding optimal concatenations for structured prediction tasks. Inspired by recent advances in neural architecture search, ACE employs a controller that samples different concatenations based on its current belief about the effectiveness of individual embedding types; then, the controller updates its belief using a reward signal computed from the accuracy of a task model. Leveraging reinforcement learning techniques, the controller's parameters are effectively optimized to determine the best concatenation of embeddings. At the end of 2022, ASP+T5 [35] approaches achieved results comparable to the previous SOTA, with ASP+T5-3B slightly outperforming FLERT XLM-R. Presented by Liu et al. (Autoregressive Structured Prediction with Language Models), this approach revolves around constructing a conditional language model that operates on sequences of structure-building actions rather than traditional string-based representations. Coupled with this approach, the underlying model is successfully empowered to more effectively capture and exploit intra-structure dependencies: the final outcome, according to the presented results, is a system that preserves the inherent structure of the data whilst explicitly capturing intricate dependencies that exist within the structure.
","metadata":{}},{"cell_type":"markdown","source":"\n
\n\n KEY INSIGHTS:\n\n\n \n - There is a wide range of different well-performing NER models, depending on the task
\n - Biaffine approaches proved to be particularly effective against several benchmarks, also in recent tests (e.g., Baseline+BS)
\n - Pretraining on corpora independent of specific tasks allows for knowledge transfer without explicit fine-tuning, enhancing language models' comprehension of structures
\n - Novel pre-training approaches (e.g., DeepStruct) can outperform previous approaches in specific settings/datasets (e.g., Genia)
\n - Addressed the disregard for interrelations between spans or pairs in entity extraction approaches by introducing neighborhood-oriented and subject-oriented packing strategies, enabling a more comprehensive understanding of entity boundary information and interrelations among span pairs sharing the same subject
\n - Addressed the issue of over-confidence in NER models by employing a novel regularization technique called Boundary Smoothing (BS)
\n - Adapting T5-like models to the biomedical domain through domain-specific pre-training on large-scale biomedical corpora achieves promising results
\n - The recent HGN was recently testeg against several datasets and often achieves satisfying (but not SOTA) performance
\n - Less sophisticated architectures such as BiLSTM-CNN can attain SOTA even in the very last years for some domains, if built properly (e.g., SparkNLP, built on top of Apache Spark)
\n - Cooperative Learning (CL) approaches manage to handle NER tasks pretty competently, often achieving State-of-the-Art results
\n - Contrastive Learning emerges as an alternative problem formulation, achieving outstanding performances in some settings
\n - ACE+Document Context as an attempt to address the challenge of selecting the most effective combination of embeddings
\n - Conditional language models operating on sequences of structure-building actions manage to explicitly capture intricate dependencies within the textual data
\n
\n\n\n
\n\n\n
\n\n WHAT'S NEW?\n\n\n \n - Cooperative Learning: CL-KL, CL-L2
\n - T-NER (XLM-RoBERTa-Large)
\n - DeepStruct multi-task
\n - PL-Marker
\n - Baseline+BS: Pretrained Transformer Embeddings and Biaffine Decoder + Boundary Smoothing
\n - SparkNLP: Apache Spark-based BiLSTM-CNN
\n - HGN
\n - SciFive Large
\n - BINDER
\n - ACE + Document-context
\n - ASP+T5 approaches
\n
\n\n\n
4.1.5 Text Summarization
","metadata":{}},{"cell_type":"code","source":"task = \"text-summarization\"\nnum_datasets = 4\nplot_number = 5\n\nfig = make_subplots(rows=num_datasets, cols=1, vertical_spacing=0.1,\n specs=[\n [{\"type\": \"scatter\"}] for dataset in range(num_datasets)\n ],\n )\n\nscatter1, dataset_name1, dataset1, metric1, df_sota1 = create_scatter(task=task, dataset_name='x-sum', remove_samples=0)\nscatter2, dataset_name2, dataset2, metric2, df_sota2 = create_scatter(task=task, dataset_name='pubmed-1', remove_samples=0)\nscatter3, dataset_name3, dataset3, metric3, df_sota3 = create_scatter(task=task, dataset_name='gigaword', remove_samples=1)\nscatter4, dataset_name4, dataset4, metric4, df_sota4 = create_scatter(task=task, dataset_name='arxiv', remove_samples=0)\n\nobjects = [(scatter1, dataset_name1, dataset1, metric1, df_sota1),\n (scatter2, dataset_name2, dataset2, metric2, df_sota2),\n (scatter3, dataset_name3, dataset3, metric3, df_sota3),\n (scatter4, dataset_name4, dataset4, metric4, df_sota4)]\n\nfig = create_plot(task, objects, plot_number)\n \nfig.layout.xaxis['showgrid'] = False\nfig.layout.xaxis2['showgrid'] = False\nfig.layout.xaxis3['showgrid'] = False\nfig.layout.xaxis4['showgrid'] = False\n\nfig.layout.yaxis.title = f\"{metric1}\"\nfig.layout.yaxis2.title = f\"{metric2}\"\nfig.layout.yaxis3.title = f\"{metric3}\"\nfig.layout.yaxis4.title = f\"{metric4}\"\n\nfig.update_layout(\n template='plotly_white', height=1000\n)\n\nfig.layout['annotations'][18].ay = 45\nfig.layout['annotations'][27].ay = -45\nfig.layout['annotations'][29].ay = -45\nfig.layout['annotations'][35].ay = -45\nfig.layout['annotations'][44].ay = -60\nfig.layout['annotations'][52].ay = 45\n\nfig.show()\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T17:03:46.989980Z","iopub.execute_input":"2023-07-04T17:03:46.990500Z","iopub.status.idle":"2023-07-04T17:03:48.444973Z","shell.execute_reply.started":"2023-07-04T17:03:46.990463Z","shell.execute_reply":"2023-07-04T17:03:48.443785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Immense progress has been achieved with the advent of Pegasus models for what concerns Text Summarization and, focusing on X-Sum dataset [Subplot 5.1], we see Pegasus Large brutally achieving State-of-the-Art towards the end of 2019. SOTA has been slightly improved thenceforth, with Pegasus+SimCLS [36] in 2021 and Pegasus 2B+SLIC [37] in 2022. The former was presented by Carnegie Mellon University researchers in the paper \"SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization\" by Liu and Liu. It consists of a novel framework that offers a paradigm shift in abstractive summarization: it addresses the disparity between the learning objective and evaluation metrics inherent in the prevailing sequence-to-sequence learning framework by reframing text generation as a reference-free evaluation problem, leveraging contrastive learning techniques. The latter was introduced by Zhao et al. (Calibrating Sequence Likelihood Improves Conditional Language Generation) and aims at aligning the likelihood of sequences generated by the model with reference sequences in the latent space. By employing SLiC, the reliance on decoding heuristics diminishes, and the quality of decoding candidates significantly improves, regardless of the chosen decoding method. Importantly, the authors report that SLiC exhibits no diminishing returns with larger model sizes, offering alternative avenues for quality enhancement even with limited training and inference resources. We can easily notice that Pegasus is a recurring building block when it comes to text summarization advancements, at least for what concerns the X-Sum dataset. Worth of attention is also BRIO [38], proposed by Liu et al. (the same contributors of SimCLS plus two Yale University researchers) in the paper \"BRIO: Bringing Order to Abstractive Summarization\" in a time frame between the two aforementioned Pegasus-embedded approaches, yielding intermediate results. The authors devised this approach to address the challenges brought by the assumption, during inference when comparing system-generated candidate summaries that deviate from the reference, of a deterministic target distribution where all probability mass is concentrated on the reference summary. BRIO thus constitutes a novel training paradigm that embraces a non-deterministic distribution, allowing the assignment of probability mass to candidate summaries based on their quality.
\n","metadata":{}},{"cell_type":"markdown","source":"On the PubMed-1 benchmark [Subplot 5.2], two approaches clearly stand out: LongT5 [39] and Top Down Transformer (AdaPool) [40]. They followed each other between the end of 2021 and the beginning of 2022, determining two new SOTA results that are still unbeaten one year and a half later. Anyway, several distinct approaches anticipated and followed many of them displaying slightly lower-than-SOTA performance. Specifically, LongT5 was proposed by Pang et al. (LongT5: Efficient Text-To-Text Transformer for Long Sequences) to explore the combined effects of scaling both the input length and the model size. To achieve this, they incorporate attention mechanisms inspired by long-input transformers (ETC) and leverage pretraining strategies from summarization pretraining (PEGASUS) within the flexible T5 architecture. Another key contribution mentioned in the paper is the introduction of Transient Global (TGlobal), a novel attention mechanism that replicates the local/global attention mechanism of ETC but without the need for additional side-inputs. The latter is instead presented in the paper \"Long Document Summarization with Top-Down and Bottom-Up Inference\" (Pang et al., Salesforce research) to enhance summarization models in two key aspects. The proposed framework indeed assumes a hierarchical latent structure of the document, with a top-level representation capturing long-range dependencies and a bottom-level representation preserving details. This hierarchical structure allows token representations to be updated both bottom-up and top-down. During the bottom-up pass, token representations are inferred using local self-attention, which improves computational efficiency. To incorporate long-range dependencies, a top-down correction is applied, enabling tokens to capture broader contextual information. These recent approaches provide interesting innovations whose effectiveness translates into new State-of-the-Art performance being achieved.
\n","metadata":{}},{"cell_type":"markdown","source":"Gigaword dataset [Subplot 5.3] again displays State-of-the-Art held by a Pegasus-embedded approach, i.e., Pegasus+DotProd [41], introduced by Kedia et al. in 2021 in the paper \"Beyond Reptile: Meta-Learned Dot-Product Maximization between Gradients for Improved Single-Task Regularization\". It consists of a novel method that utilizes the finite differences first-order algorithm to calculate the gradient derived from the dot-product of gradients. This approach offers explicit control over the weight assigned to this gradient component relative to standard gradients and, by leveraging the proposed gradient as a regularization technique, the model encourages aligned gradients across different batches. Unlike naively calculating the Hessian matrix, the aforementioned methodology avoids the memory-intensive O(n^2) complexity, making it suitable for large models with sizable batch sizes. The previous SOTA was held by BART-RXF (inferior by a very small margin), while a well-performing result was recently achieved by OFA [42] in February 2022, still below SOTA though. This last approach was presented by Wang et al. (OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework) with the aim of decreasing the need for complex task and modality-specific customization. Such a framework focuses on achieving Task Comprehensiveness, bringing together a diverse range of crossmodal and unimodal tasks such as image generation, visual grounding, image captioning, image classification, and language modeling, within a simple sequence-to-sequence learning framework. Besides, OFA adopts an instruction-based learning approach in both the pretraining and finetuning stages, eliminating the requirement for additional task-specific layers in downstream tasks. In contrast to recent vision and language models that heavily rely on large-scale cross-modal datasets, OFA is pretrained using just 20 million publicly available image-text pairs. Despite its simplicity and relatively smaller training data, OFA achieves new state-of-the-art results across multiple cross-modal tasks, while also demonstrating highly competitive performance in uni-modal tasks.\nThe authors also emphasize that OFA exhibits effective transfer learning capabilities, even for unseen tasks and domains, highlighting its ability to generalize and adapt well to different contexts. Despite not breaking the SOTA wall, this last approach appears to be particularly innovative due to its unifying nature and multimodality.
","metadata":{}},{"cell_type":"markdown","source":"The last Text Summarization dataset we are going to tackle is arXiv [Subplot 5.4]: similarly as we saw with PubMed-1, SOTA is held by the Top Down Transformer (AdaPool) model, outperforming several other approaches (some of which we already encountered) such as Long T5, BartLS, Pegasus-X. The previous SOTA belonged to MemSum (extractive) [43], relatively recent as well (July 2021), whilst an extremely recent approach is constituted by ExtSum + Oracle Segmentation (extractive) [44], tested against arXiv benchmark in January 2023. The former was developed by ETH Zurich researchers (Gu et al., MemSum: Extractive Summarization of Long Documents Using Multi-Step Episodic Markov Decision Processes) and consists of a reinforcement learning-based approach for extractive summarization that takes advantage of detailed extraction history. By considering sentence content, document context, and past extraction decisions, MemSum produces superior summaries for lengthy documents and its performance, backed by state-of-the-art results and a human evaluation, highlights the effectiveness of incorporating extraction history in the summarization process. ExtSum + Oracle Segmentation, once again extractive, was presented by Miculicich and Han (Document Summarization with Test Segmentation) and involves the development of two text segmentation models, aided by the systematical exploration of the optimal strategy for integrating their output predictions into an extractive summarization model. According to the authors, a significant improvement is observed in documents where the most relevant information is not situated at the beginning: this finding suggests that the incorporation of segmentation helps address the lead bias problem, where summaries tend to favor information presented early in the document.
\n","metadata":{}},{"cell_type":"markdown","source":"\n
\n\n KEY INSIGHTS:\n\n\n \n - Text Summarization methodologies are much more similar to each other than NER ones, with similar (if not the same) names recurring among benchmarking datasets
\n - Transformer-based models command recent State-of-the-Art results: Top Down Transformer, Pegasus, BART
\n - Incorporating extraction history in the summarization process through Reinforcement Learning for improved summarization performance
\n - Task comprehensiveness and reduction of task-specific customization are a focus when tackling Text Summarization as they were in previous tasks
\n - Pegasus is an important building block of many recent Text Summarization advancements
\n
\n\n\n
\n\n\n
\n\n WHAT'S NEW?\n\n\n \n - Pegasus+SimCLS
\n - Pegasus 2B + SLiC
\n - Top-Down Transformer (AdaPool)
\n - LongT5
\n - Pegasus + DotProd
\n - OFA
\n - MemSum (extractive)
\n - ExtSum + Oracle Segmentation (extractive)
\n
\n\n\n
4.1.6 Language Modelling
\n","metadata":{}},{"cell_type":"code","source":"task = \"language-modelling\"\nnum_datasets = 4\nplot_number = 6\n\nfig = make_subplots(rows=num_datasets, cols=1, vertical_spacing=0.1,\n specs=[\n [{\"type\": \"scatter\"}] for dataset in range(num_datasets)\n ],\n )\n\nscatter1, dataset_name1, dataset1, metric1, df_sota1 = create_scatter(task=task, dataset_name='lambada', remove_samples=1)\nscatter2, dataset_name2, dataset2, metric2, df_sota2 = create_scatter(task=task, dataset_name='one-billion-word', remove_samples=0)\nscatter3, dataset_name3, dataset3, metric3, df_sota3 = create_scatter(task=task, dataset_name='wikitext-103', remove_samples=0)\nscatter4, dataset_name4, dataset4, metric4, df_sota4 = create_scatter(task=task, dataset_name='wikitext-2', remove_samples=1)\n\nobjects = [(scatter1, dataset_name1, dataset1, metric1, df_sota1),\n (scatter2, dataset_name2, dataset2, metric2, df_sota2),\n (scatter3, dataset_name3, dataset3, metric3, df_sota3),\n (scatter4, dataset_name4, dataset4, metric4, df_sota4)]\n\nfig = create_plot(task, objects, plot_number)\n \nfig.layout.xaxis['showgrid'] = False\nfig.layout.xaxis2['showgrid'] = False\nfig.layout.xaxis3['showgrid'] = False\nfig.layout.xaxis4['showgrid'] = False\n\nfig.layout.yaxis.title = f\"{metric1}\"\nfig.layout.yaxis2.title = f\"{metric2}\"\nfig.layout.yaxis3.title = f\"{metric3}\"\nfig.layout.yaxis4.title = f\"{metric4}\"\n\nfig.update_layout(\n template='plotly_white', height=1000\n)\n\nfig.layout['annotations'][13].ay = 45\nfig.layout['annotations'][16].ay = -20\nfig.layout['annotations'][16].ay = -20\nfig.layout['annotations'][22].ay = -45\nfig.layout['annotations'][24].ay = -45\nfig.layout['annotations'][31].ay = 35\nfig.layout['annotations'][33].ay = -75\nfig.layout['annotations'][35].ay = -75\nfig.layout['annotations'][37].ay = -75\nfig.layout['annotations'][39].ay = -75\nfig.layout['annotations'][41].ay = -75\nfig.layout['annotations'][43].ay = -75\nfig.layout['annotations'][48].ay = 20\nfig.layout['annotations'][50].ay = -45\nfig.layout['annotations'][56].ay = -45\n\nfig.show()\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T17:16:55.440940Z","iopub.execute_input":"2023-07-04T17:16:55.441493Z","iopub.status.idle":"2023-07-04T17:16:57.252519Z","shell.execute_reply.started":"2023-07-04T17:16:55.441455Z","shell.execute_reply":"2023-07-04T17:16:57.251269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The LAMBADA dataset [Subplot 6.1] has a few-shot PaLM (540B) as SOTA holder, followed by a more recent one-shot PaLM 2-L: these models were already encountered in other tasks (such as Question Answering) and have once again proved to be a powerful recent development for the purpose of automated textual data processing. Preceding the SOTA is Megatron-Turing NLG 530B [45], while less recent is the previous State-of-the-Art constituted by few-shot GPT-3 175B. The paper \"Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model\" by Smith et al. (a cohort of Microsoft and NVIDIA researchers) presents the joint leverage of DeepSpeed, a deep learning optimization framework from Microsoft, and Megatron, a torch-based framework developed by NVIDIA to train large transformer-based models; such a combined effort yields a groundbreaking monolithic transformer-based language model, boasting an impressive scale of 530 billion parameters. The paper delves into the infrastructure and 3D parallelism methodology, utilizing the aforementioned frameworks DeepSpeed and Megatron, which enabled the successful training of this colossal model. The focus of the joint cohort of researchers extends beyond the technical aspects of training infrastructure, emphasizing the significance of the training corpus design and data curation techniques, which played a crucial role in the model's exceptional performance, the authors remark.
\n","metadata":{}},{"cell_type":"markdown","source":"OmniNetT Large [46] dominates the One-Billion-Word benchmark [Subplot 6.2] beating, in 2021, the SOTA held by Transformer-XL Large. Other OmniNet variants display comparable performanc and, just one month before their disclosure, SRU++ approached the previous State-of-the-Art. Relatively more recent and with slightly worse results is Cohere Large. What characterizes the methodology proposed by Tay et al. in the paper \"OmniNet: Omnidirectional Representations from Transformers\" is allowing, unlike conventional models with limited receptive fields, each token to attend to all tokens throughout the entire network. This approach can be seen as an extreme form of attention mechanism, encompassing the entire width and depth of the network. To facilitate the learning of omnidirectional attention while mitigating computational costs, a meta-learner is employed which, based on efficient self-attention models like kernel-based, low-rank attention, and Big Bird, assists in acquiring the omnidirectional representation.
","metadata":{}},{"cell_type":"markdown","source":"WikiText-103 dataset [Subplot 6.3] comes with a much larger amount of techniques tested upon, reaching the current SOTA at the end of 2022 with Hybrid H3 2.7B [47] that scores just slightly better than Megatron-LM, the model holding the State-of-the-Art since September 2019. Such an approach was presented by Fu et al. (\"Hungry Hungry Hippos: Towards Language Modeling with State Space Models\") to address the performance gap and hardware utilization challenges between state space models (SSMs) and attention models in the context of language modeling. While SSMs have shown impressive results in certain domains, they have lagged behind attention models in language modeling tasks. Specifically, the authors found that existing SSMs struggle with two crucial abilities: recalling earlier tokens in a sequence and comparing tokens across the sequence; they proposed a novel SSM layer called H3, which exactly focuses on improving these capabilities, and introduced FlashConv, a technique that improves the training efficiency of SSMs on modern hardware. FlashConv leverages a fused block FFT algorithm to enhance efficiency for sequences up to 8K in length, and also incorporates an innovative state passing algorithm that leverages the recurrent properties of SSMs to scale to longer sequences. GLM-XXLarge (bidirectional) [48] challenged such a SOTA in March 2021, yielding comparable (but not superior) results. Presented by Du et al. (GLM: General Language Model Pretraining with Autoregressive Blank Infilling), this technique enhances blank filling pretraining by incorporating 2D positional encodings and introducing the flexibility to predict spans in any order. These enhancements result in improved performance compared to BERT and T5 specifically in NLU tasks. Besides, GLM's versatility enables it to be pretrained for various task types by adjusting the number and lengths of blanks.
\n\n","metadata":{}},{"cell_type":"markdown","source":"Recent advancements measured against the WikiText-2 benchmark [Subplot 6.4] include mainly GPT-like models: the SOTA has been held by GPT-2 since February 2019, then overcome by the fine-tuned GPT-2 version in October 2021 to be, eventually, outperformed by SparseGPT-175B [49] in January 2023. Proposed by Frantar and Alistarh (SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot), SparseGPT is specifically designed to handle the complexities of massive GPT-family models, enabling to rapidly execute the pruning process on the largest available open-source models, including OPT-175B and BLOOM-176B. Through SparseGPT, the paper mentions, up to 60% unstructured sparsity can be achieved without significantly impacting perplexity and this, in turn, implies that more than 100 billion weights from these models can be effectively disregarded during inference, leading to substantial computational savings. Furthermore, SparseGPT is compatible with weight quantization approaches, opening up to directions that accomodate for the additional enhancement of its versatility and potential for optimization. These findings represent a groundbreaking advancement in the field of model pruning: by significantly reducing computational and memory requirements without compromising performance, SparseGPT opens up new avenues for the efficient deployment of large-scale language models. This discovery has profound implications for resource-efficient and sustainable AI applications, potentially heralding a new era of streamlined and optimized model architectures.
","metadata":{}},{"cell_type":"markdown","source":"\n
\n\n KEY INSIGHTS:\n\n\n \n - Radical improvements in Language Modelling over almost a decade
\n - Language models appear to be continuously advancing, despite the margin of improvement has been decreasing dramatically
\n - Enhanced attention mechanism that encircles tokens throughout the whole neural network for stronger (\"Omnidirectional\") representations and increased performance (OmniNet)
\n - Fine-tuning seemingly allows for substantial improvements in language modelling performance (GPT-2 vs Fine-tuned GPT-2)
\n - Apart from the performance advancements, very recent language modelling innovations pose a focus on reducing computational requirements
\n
\n\n\n
\n\n\n
\n\n WHAT'S NEW?\n\n\n \n - OmniNet models
\n - SRU++
\n - GLM-XXLarge (Bidirectional)
\n - Hybrid H3
\n - SparseGPT (175B)
\n
\n\n\n
5. Innovation Pace
\n","metadata":{}},{"cell_type":"markdown","source":"We observed humongous advancements in the field of textual data processing in the last years, and several innovative techniques and machine learning approaches to the main NLP tasks existing saw the light from 2021 onwards. Anyway, we noticed that the rate at which improvements are made radically decreased, posing a question of how much better approaches to textual data -driven ML could become. Hence, we will now investigate the innovation pace followed by this specific field of Artificial Intelligence to shed light on such a widely debated question.
\n\n
\n\nWe will now envision the innovation pace testified by the benchmarking datasets at stake using a proportional yearly statistic: Improvement over Missing. We define it as the improvement over missing performance from top performance (e.g., 0.5 to 0.75 improvement in F1 score translates into a 50% Improvement over Missing, since 0.5 was the gap between current performance and top performance and 0.25 (precisely 50% of the gap) was achieved upon improvement. The proportional nature of this metric makes it suitable to assess dynamicity of performance improvements also when the current state is relatively advanced, when considerable discoveries and innovations have to be made to attain comparatively small performance increases over total performance.
","metadata":{}},{"cell_type":"code","source":"'''Compute marginal improvement by year and averaged over all the datasets of a specific task'''\n\npath = \"/kaggle/input/nlp-sota-benchmarks\"\n\ndatasets = ['question-answering_triviaqa.csv','question-answering_webquestions.csv','text-summarization_gigaword.csv','named-entity-recognition-ner_conll-2003.csv',\n'question-answering_squad11.csv','question-answering_boolq.csv','text-summarization_x-sum.csv','text-summarization_pubmed-1.csv',\n'question-answering_hotpotqa.csv','named-entity-recognition_genia.csv','text-classification_ag-news.csv','sentiment-analysis_sst-2-binary.csv',\n'named-entity-recognition-ner_ontonotes-v5.csv','question-answering_wikiqa.csv','language-modelling_wikitext-103.csv','language-modelling_wikitext-2.csv',\n'named-entity-recognition_bc5cdr-chemical.csv','sentiment-analysis_sst-5-fine-grained.csv','language-modelling_one-billion-word.csv','question-answering_pubmedqa.csv',\n'named-entity-recognition-ner_bc5cdr.csv','question-answering_natural-questions.csv','text-classification_imdb.csv','named-entity-recognition_wnut-2017.csv',\n'text-classification_trec-6.csv','text-summarization_arxiv.csv','sentiment-analysis_yelp-binary.csv','sentiment-analysis_imdb.csv',\n'sentiment-analysis_mr.csv','text-classification_20news.csv','language-modelling_lambada.csv','text-classification_r8.csv']\n\n\ndef get_statistics(task):\n yearly_statistics_overall = {}\n yearly_statistics_proportional = {}\n innovation_pace = {}\n for dataset_name in datasets:\n if task in dataset_name:\n dataset = pd.read_csv(f\"{path}/{dataset_name}\")\n dataset = dataset[~dataset['score'].isna()] \n\n higher_better = dataset['score'].iloc[0] > dataset['score'].iloc[1]\n if dataset['score'].iloc[0] < 1:\n dataset['score'] = dataset['score'] * 100\n score_refactored = dataset['score'] if higher_better else 100 - dataset['score']\n dataset['score_refactored'] = score_refactored\n\n dataset['year'] = pd.DatetimeIndex(dataset['date']).year\n dataset = dataset.drop_duplicates('year').sort_values(by='year') ### If performed, only best model for each year is kept\n\n dataset_records = {}\n\n previous_score = 0\n previous_year = np.min(dataset['year'])\n\n for idx, yearly_best in dataset.reset_index(drop=True).iterrows():\n year = yearly_best['year']\n if year - previous_year > 1:\n for gap_year in range(previous_year, year):\n dataset_records[gap_year] = previous_score\n score = yearly_best['score_refactored']\n\n if idx != 0:\n if year not in yearly_statistics_overall.keys():\n yearly_statistics_overall[year] = []\n yearly_statistics_proportional[year] = []\n\n if score < previous_score:\n score = previous_score\n\n yearly_statistics_overall[year].append( (score - previous_score) / (100 - previous_score) )\n yearly_statistics_proportional[year].append( (score - previous_score) / (100 - dataset[dataset['year'] == np.min(dataset['year'])]['score'].iloc[0]) )\n\n if score > previous_score:\n dataset_records[year] = score\n previous_score = score\n else:\n dataset_records[year] = previous_score\n\n previous_year = year\n innovation_pace[dataset_name] = dataset_records\n\n years = [key for key in yearly_statistics_overall.keys()]\n statistics = [np.mean(yearly_statistics_overall[key]) for key in yearly_statistics_overall.keys()]\n\n df = pd.DataFrame( ([year, statistic] for year, statistic in zip(years, statistics)), columns=['year', 'improvement_over_missing']).sort_values('year', ascending=True)\n\n fig = px.line(df, x='year', y=['improvement_over_missing'],\n color_discrete_map={ \n \"improvement_over_missing\": \"lime\",\n },\n template=\"plotly_white\"\n )\n\n fig.layout.legend.x = 0.75\n fig.layout.legend.y = 1.15\n fig.layout.legend.title.text = ''\n\n return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T12:27:32.983550Z","iopub.execute_input":"2023-07-04T12:27:32.984290Z","iopub.status.idle":"2023-07-04T12:27:33.006675Z","shell.execute_reply.started":"2023-07-04T12:27:32.984248Z","shell.execute_reply":"2023-07-04T12:27:33.005533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"task = \"text-classification\"\n\nfig = make_subplots(rows=3, cols=2, vertical_spacing = 0.15, horizontal_spacing=0.1,\n specs=[\n [{\"type\": \"scatter\"},{\"type\": \"scatter\"}], \n [{\"type\": \"scatter\"},{\"type\": \"scatter\"}],\n [{\"type\": \"scatter\"},{\"type\": \"scatter\"}]\n ],\n )\n\nfig.layout.title = f\"Innovation Pace\"\nfig.layout.title.font.size = 26\nfig.layout.title.font.color = \"black\"\n\nline1 = get_statistics('text-classification')\nline2 = get_statistics('sentiment-analysis')\nline3 = get_statistics('question-answering')\nline4 = get_statistics('named-entity-recognition')\nline5 = get_statistics('text-summarization')\nline6 = get_statistics('language-modelling')\n\n\nfig.add_trace(line1.data[0], row=1, col=1)\nfig.add_trace(line2.data[0], row=1, col=2)\nfig.add_trace(line3.data[0], row=2, col=1)\nfig.add_trace(line4.data[0], row=2, col=2)\nfig.add_trace(line5.data[0], row=3, col=1)\nfig.add_trace(line6.data[0], row=3, col=2)\n\nfig.update_layout(\n template='plotly_white', height=750\n)\n\nfor subplot_idx in range(len(fig.data)):\n fig.data[subplot_idx].showlegend = False\n\nfig.add_annotation(x=0.5, y=1.30,\n text=f\"Text Classification\",showarrow=False, xref='x domain',yref='y domain', row=1, col=1,\n font=dict(family=\"sans serif\",size=18,color=\"black\"))\nfig.add_annotation(x=0.5, y=1.30,\n text=f\"Sentiment Analysis\",showarrow=False, xref='x domain',yref='y domain', row=1, col=2,\n font=dict(family=\"sans serif\",size=18,color=\"black\"))\nfig.add_annotation(x=0.5, y=1.30,\n text=f\"Question Answering\",showarrow=False, xref='x domain',yref='y domain', row=2, col=1,\n font=dict(family=\"sans serif\",size=18,color=\"black\"))\nfig.add_annotation(x=0.5, y=1.30,\n text=f\"Named Entity Recognition\",showarrow=False, xref='x domain',yref='y domain', row=2, col=2,\n font=dict(family=\"sans serif\",size=18,color=\"black\"))\nfig.add_annotation(x=0.5, y=1.30,\n text=f\"Text Summarization\",showarrow=False, xref='x domain',yref='y domain', row=3, col=1,\n font=dict(family=\"sans serif\",size=18,color=\"black\"))\nfig.add_annotation(x=0.5, y=1.30,\n text=f\"Language Modelling\",showarrow=False, xref='x domain',yref='y domain', row=3, col=2,\n font=dict(family=\"sans serif\",size=18,color=\"black\"))\n\nfig.add_vline(x=\"2021\", line_color='black', line_dash='dash', line_width=1,row=1, col=1)\nfig.add_vline(x=\"2021\", line_color='black', line_dash='dash', line_width=1,row=1, col=2)\nfig.add_vline(x=\"2021\", line_color='black', line_dash='dash', line_width=1,row=2, col=1)\nfig.add_vline(x=\"2021\", line_color='black', line_dash='dash', line_width=1,row=2, col=2)\nfig.add_vline(x=\"2021\", line_color='black', line_dash='dash', line_width=1,row=3, col=1)\nfig.add_vline(x=\"2021\", line_color='black', line_dash='dash', line_width=1,row=3, col=2)\n\nfig.layout.xaxis['showgrid'] = False\nfig.layout.yaxis['showgrid'] = False\nfig.layout.xaxis2['showgrid'] = False\nfig.layout.yaxis2['showgrid'] = False\nfig.layout.xaxis3['showgrid'] = False\nfig.layout.yaxis3['showgrid'] = False\nfig.layout.xaxis4['showgrid'] = False\nfig.layout.yaxis4['showgrid'] = False\nfig.layout.xaxis5['showgrid'] = False\nfig.layout.yaxis5['showgrid'] = False\nfig.layout.xaxis6['showgrid'] = False\nfig.layout.yaxis6['showgrid'] = False\n\nfig.show()\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T12:27:33.008404Z","iopub.execute_input":"2023-07-04T12:27:33.008782Z","iopub.status.idle":"2023-07-04T12:27:33.950064Z","shell.execute_reply.started":"2023-07-04T12:27:33.008753Z","shell.execute_reply":"2023-07-04T12:27:33.948856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As already observed in Section 4, proportional improvement in assessment metrics met its peek prior to the very last years. Specifically, considerable dynamicity is shown around 2016, whilst the majority of global peaks appear to have being reached between 2018 and 2019. Trends from 2021 onwards also appear to reflect the overall pace identified in Section 4, confirming that marginal performance improvements are becoming increasingly difficult to achieve. This insight coupled with the beneficial effects of some of the recent innovations we previously encountered further validates what we mentioned throughout our journey, i.e., that the focus is shifting from performance enhancement in terms of optimizing a metric to the reduction of computational needs whilst maintaining or improving performance on metrics. Nevertheless, it is interesting to note how reduced computational requirements are ideally associated not only with increased accessibility, but also with prospects for performance improvements deriving both from the capability to process larger amounts of data and to leverage more complex architectures.
","metadata":{}},{"cell_type":"markdown","source":"# 6. Attacking Textual Datasets in Kaggle Competitions
\n","metadata":{}},{"cell_type":"markdown","source":" State-of-the-Art achieved by novel techniques tested on benchmarking datasets constitute only one side of the medal of Machine Learning innovation, the other being real-world actionability of such and other techniques. Many Kaggle Competitions represent applied machine learning par excellence, hence it is desirable for us to account for top solutions at competitions involving text data for the overall picture to be more rounded and complete.
\n\n Kaggle specifically provides a comprehensive collection of competitions' writeups that we can leverage to identify particular trends adopted by competitors in challenges involving textual datasets throughout different periods. We will divide such a pool of competitions in three groups, according to their launch year:
\n \n- 2015-2017 \n- 2018-2020\n- 2021-2023 ","metadata":{}},{"cell_type":"code","source":"'''Import Competition Data'''\n\nkaggle_writeups = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv\", parse_dates=['Competition Launch Date'])\n\nkaggle_writeups_21_23 = kaggle_writeups[kaggle_writeups['Competition Launch Date'] >= '2021-01-01']\nkaggle_writeups_18_20 = kaggle_writeups[kaggle_writeups['Competition Launch Date'] >= '2018-01-01']\nkaggle_writeups_15_17 = kaggle_writeups[kaggle_writeups['Competition Launch Date'] >= '2015-01-01']\n\nnlp_competitions_21_23 = ['Coleridge Initiative - Show US the Data',\n 'CommonLit Readability Prize',\n 'Jigsaw Rate Severity of Toxic Comments',\n 'Feedback Prize - Evaluating Student Writing',\n 'NBME - Score Clinical Patient Notes',\n 'U.S. Patent Phrase to Phrase Matching',\n 'Feedback Prize - Predicting Effective Arguments',\n 'Feedback Prize - English Language Learning',\n 'Learning Equality - Curriculum Recommendations']\n\nnlp_competitions_18_20 = ['Jigsaw Multilingual Toxic Comment Classification',\n 'Tweet Sentiment Extraction',\n 'Google Quest Q&A Labeling',\n 'Tensor Flow 2.0 Question Answering',\n 'Jigsaw Unintended Bias in Toxicity Classification',\n 'Gendered Pronoun Resolution',\n 'PetFinder.my Adoption Prediction',\n 'Quora Insincere Questions Classification',\n ]\n\nnlp_competitions_15_17 = ['Toxic Comment Classification Challenge',\n 'Text Normalization Challenge - Russian Language',\n 'Text Normalization Challenge - English Language',\n 'Quora Question Pairs',\n 'The Allen AI Science Challenge',\n 'What\\'s Cooking?',\n 'Bag of Words Meets Bags of Popcorn',\n 'Random Acts of Pizza',\n 'Billion Word Imputation',\n 'Sentiment Analysis on Movie Reviews']\n\n\nkaggle_writeups_21_23_nlp = kaggle_writeups_21_23[kaggle_writeups_21_23['Title of Competition'].isin(nlp_competitions_21_23)]\nkaggle_writeups_18_20_nlp = kaggle_writeups_18_20[kaggle_writeups_18_20['Title of Competition'].isin(nlp_competitions_18_20)]\nkaggle_writeups_15_17_nlp = kaggle_writeups_15_17[kaggle_writeups_15_17['Title of Competition'].isin(nlp_competitions_15_17)]\n\nallwriteups = []\nfor writeup in kaggle_writeups_15_17_nlp['Writeup']:\n allwriteups.append(writeup)\nallwriteups = \" \".join(allwriteups)\n\nallwriteups2 = []\nfor writeup in kaggle_writeups_18_20_nlp['Writeup']:\n allwriteups2.append(writeup)\nallwriteups2 = \" \".join(allwriteups2)\n\nallwriteups3 = []\nfor writeup in kaggle_writeups_21_23_nlp['Writeup']:\n allwriteups3.append(writeup)\nallwriteups3 = \" \".join(allwriteups3)\n\ndataframes = [\n allwriteups,\n allwriteups2,\n allwriteups3\n]\n\nidx = 0\nfor writeup in dataframes:\n writeup_clean = []\n for subset in writeup.split(\"<\"):\n subset = subset.split('>')\n try:\n writeup_clean.append(subset[1])\n except:\n writeup_clean.append(subset[0])\n writeup = ' '.join(writeup_clean)\n writeup = writeup.replace('kaggle','')\n writeup = writeup.replace('used','')\n writeup = writeup.replace('thanks','')\n writeup = writeup.replace('thank','')\n writeup = writeup.replace('models','')\n writeup = writeup.replace('model','')\n dataframes[idx] = writeup\n idx += 1\n\nfig = plt.figure()\nfig.set_size_inches(30, 10)\ncolors = ['cyan', 'lime', 'orange']\ntitles = [\n '2015-2017',\n '2018-2020',\n '2021-2023'\n]\nfor i in range(3):\n wordcloud = WordCloud(width = 1200, height = 1200, background_color=\"black\",\n color_func=lambda *args, **kwargs: colors[i], relative_scaling = 0.7).generate(dataframes[i])\n ax = fig.add_subplot(1,3,i+1)\n ax.imshow(wordcloud, interpolation = 'bilinear')\n ax.axis('off')\n ax.set_title(titles[i], size=48, pad=50)\n \n \nplt.tight_layout(pad=0)\nplt.show()\n ","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:27:34.363935Z","iopub.execute_input":"2023-07-04T12:27:34.364309Z","iopub.status.idle":"2023-07-04T12:27:44.879402Z","shell.execute_reply.started":"2023-07-04T12:27:34.364277Z","shell.execute_reply":"2023-07-04T12:27:44.878123Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The previous wordclouds show both the common and the changing trends in the three year groups of writeups. Between 2015 and 2017, for example, amongst the recurring words appear: Feature, Token, Word, Level, Embedding.\nThis denotes a focus on feature engineering & feature extraction techniques and preprocessing, and mostly word-level approaches (outside of the context-embedded world). In the second time frame, between 2018 and 2020, appears BERT: extensively mentioned in the previous sections, it revolutionized all the facets of textual data processing, and both previously-analyzed benchmarks and these wordclouds reflect such a breakthrough.\nThe war-horse for the third time frame is DeBERTa-v3, the most mentioned model in NLP Kaggle Competitions in recent years. Besides, the emphasis on Ensembles has increased over the years becoming substantial, and this can be justified by the decreasing effectiveness of preprocessing and feature engineering in NLP: if not obsolete, transformer-based models made these approaches applied to textual data processing much less effective, reason why competitors in applied machine learning settings devote their efforts towards more potentially rewarding directions.
","metadata":{}},{"cell_type":"markdown","source":"# 7. Conclusions and Future Directions
\n\n Throughout our brief but intense journey in the world of textual data processing, we discovered a wide range of recent innovations and novel approaches that emerged and challenged previous State-of-the-Art techniques to push the current understanding of textual data processing towards new borders. Our era is witnessing changes and developments comparable to those chronicled in history books. Most likely, these monumental strides in technology and data processing systems, particularly in the realm of textual data (in these very last years), will be studied in the future as nothing short of epochal advancements, extending beyond the narrow confines of Machine Learning.
\n\n
\n\nWhile marginal improvements tend to diminish over time (after all, how does one improve, in some settings, upon close perfection?), the community's focus is now shifting towards new frontiers: this entails expediting processes currently lacking speed, adopting novel approaches to streamline technology and enhance its accessibility, and even venturing into activities once deemed out-of-scope for machines. The progress achieved in the most prevalent and popular downstream tasks of Natural Language Processing exerts a beneficial influence on a remarkably diverse range of aspects within Textual Data Processing itself. Consider, for instance, the humongous impact of the transformer, which gained prominence in 2018 and laid the foundations for one of today's most powerful systems, namely GPT.
\n\n
\n\nWe can thus expect future efforts to attain more notable achievements in those settings still holding consistent room for improvement, and to bridge the gap between current performance and close perfection in those more highly-studied, advanced and, why not, maybe relatively more trivial domains. Research appears to be moving towards a combined effort for the sake of accessibility and computational relief, and it is not unreasonable that those systems currently resembling humongous monsters of computing tomorrow will be more easily adapted to relaxed computational settings, both in terms of inference and training.
\n\n
\n\nWe hopefully shed a light on where we are today, the difference with where we were yesterday, and the compelling prospective of tomorrow's automatic textual data processing systems.
\n","metadata":{}},{"cell_type":"markdown","source":"# References
\n\n[1] Papers With Code: https://paperswithcode.com/
\nA watchfully curated and up to date collection of research paper results for benchmarking purposes;
\nDisclaimer:\nThe data used is publicly accessible on the aforementioned source for free, and was collected for educational and research purposes.\n\n[2] Wahba, Y., Madhavji, N., Steinbacher, J. (2022). A Comparison of SVM against Pre-trained Language Models (PLMs) for Text Classification Tasks.
\nArXiv paper: https://arxiv.org/abs/2211.02563v1\n\n[3] Lin, Y., Meng, Y., Sun, X., Han, Q., Kuang, K., Li, J., Wu, F. (2021) BertGCN: Transductive Text Classification by Combining GCN and BERT.
\nArXiv paper: https://arxiv.org/pdf/2105.05727v4\n\n[4] Cer, D., Yang, Y., Kong, S., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Sung, Y., Strope, B., Kurzweil, R. (2018). Universal Sentence Encoder.
\nArXiv paper: https://arxiv.org/pdf/1803.11175v2\n\n[5] Chen, Q., Zhang, R., Zheng, Y., Mao, Y. (2022). Dual Contrastive Learning: Text Classification via Label-Aware Data Augmentation.
\nArXiv paper: https://arxiv.org/pdf/2201.08702v1\n\n[6] Karl, F., Scherp, A. (2022). Transformers are Short Text Classifiers: A Study of Inductive\nShort Text Classifiers on Benchmarks and Real-world Datasets.
\nArXiv paper: https://arxiv.org/pdf/2211.16878v2\n\n[7] Heinsen, F. A., (2022). An Algorithm for Routing Vectors in Sequences.
\nArXiv paper: https://arxiv.org/pdf/2211.11754v3\n\n[8] Ionescu, R. T.,Butnaru, A. M. (2019). Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation.
\nArXiv paper: https://arxiv.org/pdf/1902.08850v3\n\n[9] Wang, S., Fang, H., Khabsa, M., Mao, H., Ma, H. (2021). Entailment as Few-Shot Learner.
\nArXiv paper: https://arxiv.org/pdf/2104.14690v1\n\n[10] Gharagozlou, H., Mohammadzadeh, J., Bastanfard, A., Ghidary, S. S. (2022). RLAS-BIABC: A Reinforcement Learning-Based Answer Selection Using the BERT Model Boosted by an Improved ABC Algorithm.
\nArXiv paper: https://arxiv.org/pdf/2301.02807v1\n\n[11] Kedia, A., Zaidi, M. A., Lee, H. (2022). FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering.
\nArXiv paper: https://arxiv.org/pdf/2211.10147v1\n\n[12] Lewis, P., Wu, Y., Liu, L., Minervini, P., Küttler, H., Piktus, A., Stenetorp, P., Riedel, S. (2021). PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them.
\nArXiv paper: https://arxiv.org/pdf/2102.07033\n\n[13] Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., Cui, C. (2021). GLaM: Efficient Scaling of Language Models with Mixture-of-Experts.
\nArXiv paper: https://arxiv.org/pdf/2112.06905v2\n\n[14] Google Research. (2022). PaLM: Scaling Language Modeling with Pathways.
\nArXiv paper: https://arxiv.org/pdf/2204.02311v5\n\n[15] Google Research. (2023). PaLM 2 Technical Report.
\nPaper link: https://ai.google/static/documents/palm2techreport.pdf\n\n[16] Jun, C., Jang, H., Sim, M., Kim, H., Choi, J., Min, K., Bae, K. (2022). ANNA: Enhanced Language Representation for Question Answering.
\nArXiv paper: https://arxiv.org/pdf/2203.14507v2\n\n[17] Yasunaga, M., Leskovec, J., Liang, P. (2022). LinkBERT: Pretraining Language Models with Document Links.
\nArXiV paper: https://arxiv.org/pdf/2203.15827v1\n\n[18] Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., Liu, T. (2022). BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining.
\nArXiv paper: https://arxiv.org/pdf/2210.10341v3\n\n[19] Google Research, DeepMind. (2023). Towards Expert-Level Medical Question Answering with Large Language Models.
\nArXiv paper: https://arxiv.org/pdf/2305.09617v1\n\n[20] Google Research, DeepMind. (2022). Large Language Models Encode Clinical Knowledge.
\nArXiv paper: https://arxiv.org/pdf/2212.13138v1\n\n[21] Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., Grave, E. (2022). Atlas: Few-shot Learning with Retrieval Augmented Language Models.
\nArXiv paper: https://arxiv.org/pdf/2208.03299v3\n\n[22] Fajcik, M., Docekal, M., Ondrej, K., Smrz, P. (2021). R2-D2: A Modular Baseline for Open-Domain Question Answering.
\nArXiv paper: https://arxiv.org/pdf/2109.03502v1\n\n[23] Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., Fedus W. (2022). ST-MoE: Designing Stable and Transferable Sparse Expert Models.
\nArXiv paper: https://arxiv.org/pdf/2202.08906v2\n\n[24] Zhu, Y., Pang, L., Lan, Y., Shen, H., Cheng, X. (2021). Adaptive Information Seeking for Open-Domain Question Answering.
\nArXiv paper: https://arxiv.org/pdf/2109.06747v1\n\n[25] Ma, K., Cheng, H., Zhang, Y., Liu, X., Nyberg, E., Gao, J. (2023). Chain-of-Skills: A Configurable Model for Open-domain Question Answering.
\nArXiv paper: https://arxiv.org/pdf/2305.03130v2\n\n[26] Wang, X., Jiang, Y., Bach, N., Wang, T., Huang, Z., Huang, F., Tu, K. (2021). Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning.
\nArXiv paper: https://arxiv.org/pdf/2105.03654v3\n\n[27] Ushio, A., Camacho-Collado, J. (2021). T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition.
\nArXiv paper: https://arxiv.org/pdf/2209.12616v1\n\n[28] Wang, C., Liu, X., Chen, Z., Hong, H., Tang, J., Song, D. (2022). DeepStruct: Pretraining of Language Models for Structure Prediction.
\nArXiv paper: https://arxiv.org/pdf/2205.10475v2\n\n[29] Ye, D., Lin, Y., Li, P., Sun, M.(2022). Packed Levitated Marker for Entity and Relation Extraction.
\nArXiv paper: https://arxiv.org/pdf/2109.06067v5\n\n[30] Zhu, E., Li, J. (2022). Boundary Smoothing for Named Entity Recognition.
\nArXiv paper: https://arxiv.org/pdf/2204.12031v1\n\n[31] Phan, L. N., Anibal, J. T., Tran, H., Chanana, S., Bahadroglu, E., Peltekian, A., Altan-Bonnet, G. (2021). SciFive: a text-to-text transformer model for biomedical literature.
\nArXiv paper: https://arxiv.org/pdf/2106.03598v1\n\n[32] Hu, J., Shen, Y., Liu, Y., Wan, X., Chang, T. (2022). Hero-Gang Neural Model For Named Entity Recognition.
\nArXiv paper: https://arxiv.org/pdf/2205.07177\n\n[33] Zhang, S., Cheng, H., Gao, J., Poon, H. (2022). Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning.
\nArXiv paper: https://arxiv.org/pdf/2208.14565v2\n\n[34] Wang, X., Jiang, Y., Bach, N., Wang, T., Huang, Z., Huang, F., Tu, K. (2021). Automated Concatenation of Embeddings for Structured Prediction.
\nArXiv paper: https://arxiv.org/pdf/2010.05006v4\n\n[35] Liu, T., Jiang, Y., Monath, N., Cotterell, R., Sachan, M. (2022). Autoregressive Structured Prediction with Language Models.
\nArXiv paper: https://arxiv.org/pdf/2210.14698\n\n[36] Liu, Y., Liu, P. (2021). SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization.
\nArXiv paper: https://arxiv.org/pdf/2106.01890v1\n\n[37] Zhao, Y., Khalman, M., Joshi, R., Narayan, S., Saleh, M., Liu P. J. (2022). Calibrating Sequence likelihood Improves Conditional Language Generation.
\nArXiv paper: https://arxiv.org/pdf/2210.00045v1\n\n[38] Liu, Y., Liu, P., Radev, D., Neubig, G. (2022). BRIO: Bringing Order to Abstractive Summarization.
\nArXiv paper: https://arxiv.org/pdf/2203.16804v1\n\n[39] Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y., Yang, Y. (2022). LongT5: Efficient Text-To-Text Transformer for Long Sequences.
\nArXiv paper: https://arxiv.org/pdf/2112.07916v2\n\n[40] Pang, B., Nijkamp, E., Kryściński, W., Savarese, S., Zhou, Y., Xiong, C. (2022). Long Document Summarization with Top-down and Bottom-up Inference.
\nArXiv paper: https://arxiv.org/pdf/2203.07586v1\n\n[41] Kedia, A., Chinthakindi, S. C., Ryu, W. (2021). Beyond Reptile Meta-Learned Dot-Product Maximization between Gradients for Improved Single-Task Regularization.
\nPaper link: https://aclanthology.org/2021.findings-emnlp.37.pdf\n\n[42] Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., Yang, H. (2022). OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework.
\nArXiv paper: https://arxiv.org/pdf/2202.03052v2\n\n[43] Gu, N., Ash, E., Hahnloser, R. H. R. (2022). MemSum: Extractive Summarization of Long Documents Using Multi-Step Episodic Markov Decision Processes.
\nArXiv paper: https://arxiv.org/pdf/2107.08929v2\n\n[44] Miculicich, L., Han, B. (2023). Document Summarization with Text Segmentation.
\nArXiv paper: https://arxiv.org/pdf/2301.08817v1\n\n[45] Smith, S., Patwary, M., Norick, B., Legresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., Zhang, E., Child, R., Aminabadi, R. Y., Bernauer, J., Song, X., Shoeybi, M., Yuxiong He, Y., Houston, M., Tiwary, S., Catanzaro, B. (2022). Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model.
\nArXiv paper: https://arxiv.org/pdf/2201.11990v3\n\n[46] Tay, Y., Dehghani, M., Aribandi, V., Gupta, J., Pham, P., Qin, Z., Bahri, D., Juan, D., Metzler, D. (2021). OmniNet: Omnidirectional Representations from Transformers.
\nArXiv paper: https://arxiv.org/pdf/2103.01075v1\n\n[47] Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., Ré, C. (2022). Towards Language Modeling with State Space Models.
\nArXiv paper: https://arxiv.org/pdf/2212.14052v3\n\n[48] Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., Tang, J. (2022). GLM: General Language Model Pretraining with Autoregressive Blank Infilling.
\nArXiv paper: https://arxiv.org/pdf/2103.10360v2\n\n[49] Frantar, E., Alistarh, D. SparseGPT. (2023). Massive Language Models Can Be Accurately Pruned in One-Shot.
\nArXiv paper: https://arxiv.org/pdf/2301.00774v3\n","metadata":{}},{"cell_type":"code","source":"data = [\n ['essay_category', 'Text data'],\n ['essay_url', 'https://www.kaggle.com/code/mauromauro/artificially-intelligent-textual-data-processing'],\n ['feedback1_url', 'https://www.kaggle.com/code/abireltaief/contemporary-large-language-models-llms/comments#2338382'],\n ['feedback2_url', 'https://www.kaggle.com/code/sophieb/a-journey-through-text-data-competitions/comments#2342459'],\n ['feedback3_url', 'https://www.kaggle.com/code/datamafia7/a-text-odyssey-the-past-present-and-future-of-nlp/comments#2345700'],\n]\n\nsubmission = pd.DataFrame(data, columns=['type','value']) \nsubmission.to_csv(\"submission.csv\", index=False)\n ","metadata":{"_kg_hide-input":true,"_kg_hide-output":true},"execution_count":null,"outputs":[]}]}