{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#
Competition cards & recent Kaggle NLP competitions
\n####
Topic: Kaggle Competitions
\n####
By Yury Kashnitsky (@kashnitsky) and Andrew Lukyanenko (@artgor)
","metadata":{"papermill":{"duration":0.004491,"end_time":"2023-07-11T13:38:08.914216","exception":false,"start_time":"2023-07-11T13:38:08.909725","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"In this essay, we propose the idea of a **competition card** and review several latest NLP competitions. Akin to model cards (e.g. [Google Model Cards](https://ai.googleblog.com/2020/07/introducing-model-card-toolkit-for.html)) and data cards, competition cards aim at providing concise information on the competition: the task being solved, data at hand, the winners' write-ups summarized, and, most importantly, **what the community has learned through this competition**. \n\nWhile we realize that filling in such competition cards is a bit of a burden (and can realistically be done probably only by competition hosts), for the \"2023 Kaggle AI Report\" competition, we create several competition cards for some of the latest **NLP** competitions. We also review the latest tricks in those competitions. \n\n\n> Sidenote, Yury's motivation for writing this essay: After the victory in the [Google Q&A Quest Labeling competition](https://www.kaggle.com/competitions/google-quest-challenge/leaderboard) in 2019, I was no longer very active on Kaggle. I was curious to see how much progress has been done in Kaggle NLP competitions since then: what the new ideas are, whether LLMs have been adopted, etc. So, the \"2023 Kaggle AI Report\" competition appeared right on time, I take this chance not only to present the idea on competition cards but also to catch up with the latest Kaggle NLP developments myself.\n\n\n# Structure\n\n1. [What is a Competition Card?](#1.-What-is-a-Competition-Card?)\n1. [Competition Card for \"Coleridge Initiative - Show US the Data\"](#2.-Coleridge-Initiative---Show-US-the-Data.-Competition-Card)\n1. [Competition Card for \"CommonLit Readability Prize\"](#3.-CommonLit-Readability-Prize.-Competition-Card)\n1. [Competition Card for \"Jigsaw Rate Severity of Toxic Comments\"](#4.-Jigsaw-Rate-Severity-of-Toxic-Comments.-Competition-Card)\n1. [Competition Card for \"NBME - Score Clinical Patient Notes\"](#5.-NBME---Score-Clinical-Patient-Notes.-Competition-Card)\n1. [Overview of recent Kaggle NLP techniques](#6.-Overview-of-recent-Kaggle-NLP-techniques)\n1. [Appendix](#Appendix) – Glossary and links to tutorials; throught the text, terms explained in this appendix are highlighted in magenta.\n---","metadata":{"papermill":{"duration":0.003654,"end_time":"2023-07-11T13:38:08.922248","exception":false,"start_time":"2023-07-11T13:38:08.918594","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# 1. What is a Competition Card?","metadata":{"papermill":{"duration":0.003566,"end_time":"2023-07-11T13:38:08.929740","exception":false,"start_time":"2023-07-11T13:38:08.926174","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n
\n
\n\nHere is a loose sketch with sticky notes. The competition card provides essential information about the competition:\n\n - **Title** and (optionally) an **Image**\n - The **Task** being solved – what is the competition about, and which metric was used. Showing a training example also helps\n - Essential information about the **Data** used in the competition \n - A list of the **Winners** with links to their write-ups\n - **Main ideas** by the winners and other competitors, what worked and what did not, etc.\n - **Findings** – what has the community learned from that competition?\n - (optionally) there can also be a **Glossary** introducing all critical terms to complete newbies (see also Appendix in this essay)\n ----","metadata":{"papermill":{"duration":0.003525,"end_time":"2023-07-11T13:38:08.937136","exception":false,"start_time":"2023-07-11T13:38:08.933611","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# 2. [Coleridge Initiative - Show US the Data](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data). Competition Card\n\n## Task\n\nDetection of datasets mentioned in scientific articles.\n\n_Example:_\n\n\"This study used data from the National Education Longitudinal Study (NELS:88) to examine the effects of dual enrollment programs for high school students on college degree attainment....\"\n\nThe task is to detect all dataset mentions from the article text. \n\n## Data\n\n~14k publications in JSON format, broken up into sections with section titles.\n\n\n## Winners and write-ups\n\n - [1st place](https://www.kaggle.com/c/25925/discussion/248251) – [Khoi Nguyen](https://www.kaggle.com/suicaokhoailang), and [Nguyen Quan Anh Minh](https://www.kaggle.com/dathudeptrai) (AI Engineer at Zalo; Ho Chi Minh City, Vietnam)\n - [2nd place](https://www.kaggle.com/c/25925/discussion/248296) – [Chun Ming Lee](https://www.kaggle.com/leecming) (Data Scientist at H2O.ai; Singapore)\n - 3rd place (no write-up) – [Mikhail Arkhipov](https://www.kaggle.com/sinxoverx)\n - [4th place](https://www.kaggle.com/c/25925/discussion/251457) – [Naoto Usuyama](https://www.kaggle.com/naotous) & Co.\n - [5th place](https://www.kaggle.com/c/25925/discussion/248300) – [Kramarenko Vladislav](https://www.kaggle.com/vlomme)\n\n## Main ideas\n\n - A simple approach is to use the BERT model to select candidates from text chunks, the BERT classification layer can predict 3 labels: no dataset mention, 1st word of the mention, and last word;\n - The winning team [used](https://www.kaggle.com/competitions/coleridgeinitiative-show-us-the-data/discussion/248251) GPT to predict whether the next token would be the start of a dataset name, given only the previous context, and then predict whether the next token would be the end of the mention;\n - Dataset titles in academic papers often have a regular structure - mixed cap words followed by an acronym. An example is \"Baltimore Longitudinal Study of Aging (BLSA)\", a good old string search algorithm like Schwartz-Hearst is efficient in spotting candidates. Further filtering can be done with a binary BERT-based classifier. This Search+Classify strategy is outlined in the 2nd place team [write-up](https://www.kaggle.com/competitions/coleridgeinitiative-show-us-the-data/discussion/248296);\n - The 5th place heavily [used](https://www.kaggle.com/competitions/coleridgeinitiative-show-us-the-data/discussion/248300) regular expressions (looking for uppercase letters, for nearby words with these letters, memorizing all found names and searching in other documents, etc.) \n\n## Findings\n\n- Regular expressions and extensions (like the Schwartz-Hearst algorithm used by the 2nd place team or acronym detection) are a very strong baseline for this task;\n- Coupling a regexp-based system with a NER pipeline (mainly for detecting dataset name variations) yields a very efficient solution;\n- Matching dataset titles with those from external sources (like the U.S. government's dataset names) was also found helpful;\n- Adding semantic context around the dataset mentions, in general, didn't help. \n---","metadata":{"papermill":{"duration":0.00355,"end_time":"2023-07-11T13:38:08.944565","exception":false,"start_time":"2023-07-11T13:38:08.941015","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# 3. [CommonLit Readability Prize](https://www.kaggle.com/c/commonlitreadabilityprize). Competition Card\n\n## Task\n\nExisting tools identifying the appropriate reading level of a passage of text are either naive, lacking theoretical validity, or non-transparent. The competition aims at building algorithms to rate the complexity of reading passages for grade 3-12 classroom use. The target value is the result of a Bradley-Terry analysis of more than 111k pairwise comparisons between excerpts.\n\nThe task is to predict the reading ease and thus boils down to a regression problem. RMSE was chosen as a competition metric. \n\n## Data\n\nTraining data contains ~2800 passages with excerpts from literature annotated with their reading ease. The test set contains ~1900 similar excerpts including more modern texts. The full corpus was released [on Github](https://github.com/scrosseye/CLEAR-Corpus).\n\n## Winners and write-ups\n\n- [1st place](https://www.kaggle.com/c/25914/discussion/257844) – [Mathis Lucka](https://www.kaggle.com/mathislucka) (Head of Product at DEMOS; Berlin, Germany)\n- [2nd place](https://www.kaggle.com/c/25914/discussion/258328) – [Takoi](https://www.kaggle.com/takoihiraokazu) (Data Scientist at Rist; Fukuoka, Japan)\n- [3rd place](https://www.kaggle.com/c/25914/discussion/258095) – [hxcc](https://www.kaggle.com/potatoc5)\n- [4th place](https://www.kaggle.com/c/25914/discussion/258148) – [datasaurus](https://www.kaggle.com/anjum48) (Lead Data Scientist at McLaren Racing; London, England, United Kingdom)\n- [5th place](https://www.kaggle.com/c/25914/discussion/259729) – [Ruby](https://www.kaggle.com/w5833946) (Algorithm Engineer; Beijing, China)\n\n\n## Main ideas\n\n - The winner utilized a large external dataset compiled from similar sources (Simple Wiki, Wikipedia, BookCorpus, etc.). For each train excerpt, 5 similar examples are found and then scored with a RoBERTa model (pseudo-labeling) to extend the training set. Filtering pseudo-labeled examples based on the deviation in prediction from the training example's target worked well (\"each external sample which had a pseudo-label score that deviated more from the original excerpt than its standard error was removed from the external data selection\", [1st place write-up](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/257844));\n - Adding some \"classic\" features like Flesch reading ease and SMOG index readability score brought a tiny but repeatable improvement ([4th place write-up](https://www.kaggle.com/c/25914/discussion/258148));\n - Some competitors ([example](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258363)) treated the task as a text ranking problem because Bradley Terry scores were essentially computed by comparing pairs of text excerpts;\n - While most competitors still approached the problem as a regression problem, an alternative approach by Chris Deotte was to compare pairs of excerpts and solve the problem in a binary classification setup [discussion](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/257446)).\n\n## Findings\n\n - As the competition data was pretty small and noisy, various ways of leveraging massive external data helped a lot;\n - In general, this was a canonic Kaggle NLP task with large ensembles of BERT-like models, pseudo-labeling, and data augmentation. A lot of discussions and notebooks were around stable training of transformer-based models, utilizing weight decay, small learning rates, layer freezing, gradient clipping, multi-sample dropout, etc. ([example](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/236465) of one such discussion); Yury's personal finding is that the winning receipts were very much akin to those in the [Google Q&A Quest Labeling competition](https://www.kaggle.com/competitions/google-quest-challenge/leaderboard) held back in 2019 (where Yury was lucky to be a part of the winning team);\n - Although RMSE was the competition metric, training instead with MSE yielded a more stable training process;\n - For regression problems, it makes sense to set dropout to 0 because a non-zero dropout may hurt training stability and hence performance ([discussion](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/260729)).\n---","metadata":{"papermill":{"duration":0.003539,"end_time":"2023-07-11T13:38:08.951894","exception":false,"start_time":"2023-07-11T13:38:08.948355","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# 4. [Jigsaw Rate Severity of Toxic Comments](https://www.kaggle.com/competitions/jigsaw-toxic-severity-rating). Competition Card\n\n## Task\n\nThe goal of this competition is to rate the toxicity of comments. Unlike most competitions, the data does not include labels for toxicity. Instead, annotators compared pairs of comments and determined which of the two was more toxic. Our objective is to build a model that produces scores that rank each pair of comments in the same order as the annotators.\n\nIt's important to note that we're predicting the degree or severity of the comment's toxicity, not the probability of it being toxic.\n\n## Data\n\nThere are 7537 comments that need to be scored.\n\nIn total, there are approximately 200k pair ratings. It is important to note that some pairs may have been rated by multiple annotators who could have ordered them differently. We have access to approximately 30k of these pair ratings.\n\nParticipants in the Jigsaw competitions were encouraged to use datasets from previous competitions for training.\n\n\n## Winners and write-ups\n\n- [1st Place](https://www.kaggle.com/c/jigsaw-toxic-severity-rating/discussion/306274) - [Guanshuo Xu](https://www.kaggle.com/wowfattie) (Data Scientist at H2O.ai; New York, the United States)\n\n- [2nd Place](https://www.kaggle.com/competitions/jigsaw-toxic-severity-rating/discussion/308938) - [Rayden Xu](https://www.kaggle.com/oucxxxx) (Graduate Student; Boston, the United States)\n\n- [3rd Place](https://www.kaggle.com/c/jigsaw-toxic-severity-rating/discussion/306235) - [steubk](https://www.kaggle.com/steubk) (Data Analyst at Softlab; Rome, Italy)\n\n- [4th Place](https://www.kaggle.com/c/jigsaw-toxic-severity-rating/discussion/306084) - [Manuel Campos](https://www.kaggle.com/coreacasa)\n\n- [5th Place](https://www.kaggle.com/c/jigsaw-toxic-severity-rating/discussion/306390) - [Alexander](https://www.kaggle.com/alexander1980)\n\n## Main ideas\n\n* The public leaderboard was not reliable due to the small amount of data shown, so the focus was on improving the local validation score;\n* The winners used various models, such as RoBERTa, DeBERTa, and [XLM](https://huggingface.co/docs/transformers/model_doc/xlm), trained on different datasets such as [Jigsaw2018](https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/data) and [Jigsaw2019](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/data). They also used pre-trained models and ensembled them using optimized weights with Optuna and genetic optimization;\n* Another approach was to extract features (for example, predict specific labels with interim models) and train linear or gradient boosting models with them.\n\n## Findings\n\n* It [was discovered](https://www.kaggle.com/competitions/jigsaw-toxic-severity-rating/discussion/306181) that the private leaderboard dataset overlaps with the data from the 2018 competition. This allowed for better scores by training models on that dataset directly;\n* `MarginRankingLoss` was a useful tool in this competition, though MSE was more common;\n* Surprisingly, a simple TF-IDF with ridge regression gave good scores at the beginning of the competition. This may be because the dataset was too small to fine-tune LLMs. After refining the approaches, neural nets became better, but adding models based on tf-idf still improved the score;\n* It is important to process the texts in a way that is relevant to the models. Some models require heavy preprocessing while others require mostly raw text;\n* As usual, it was important to build a leak-free validation scheme, for example, GroupKFold. Many people relied on the public leaderboard and this resulted in a huge shakeup.\n---","metadata":{"papermill":{"duration":0.003509,"end_time":"2023-07-11T13:38:08.959216","exception":false,"start_time":"2023-07-11T13:38:08.955707","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# 5. [NBME - Score Clinical Patient Notes](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/overview). Competition Card\n\n## Task\n\nThe text data presented here is from the USMLE® Step 2 Clinical Skills examination, a medical licensure exam. This exam measures a trainee's ability to recognize pertinent clinical facts during encounters with standardized patients.\n\nDuring the exam, each test taker meets with a Standardized Patient, someone trained to portray a clinical case. After interacting with the patient, the test taker documents the relevant facts of the encounter in a patient note. Each patient note is scored by a trained physician who looks for the presence of certain key concepts or features relevant to the case, as described in a rubric.\n\nThe goal of this competition is to develop an automated way of identifying the relevant features within each patient note. For each instance, we predict a set of character spans. The score is the F1 score over the spans.\n\nThe challenge of the data is that features may be expressed in any way, and there could be other challenges like ambiguous negations.\n\n## Data\n\nWe have a collection of about 40,000 Patient Note history portions. Only a subset of these have features annotated. We have information on the features in the clinical cases and feature annotations for 1000 of the patient notes, 100 for each of the ten cases.\n\nHere is an example of the text and the features within it.\n\n\n\n## Winners\n\n- [1st place](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323095) - [currypurin](https://www.kaggle.com/currypurin), [Ryuichi](https://www.kaggle.com/ryuichigt) (Musician at DentsuDigital; Tokyo, Japan)\n- [2nd place](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323085) - [CPMP](https://www.kaggle.com/cpmpml) (RAPIDS and deep learning at NVIDIA; France)\n- [3rd place](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322832) - [Raja Biswas](https://www.kaggle.com/conjuring92) (Data Scientist at Evonik; Singapore)\n- [4th place](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322799) - [Takoi](https://www.kaggle.com/takoihiraokazu) (Data Scientist at Rist; Fukuoka, Japan), [hakubishin3](https://www.kaggle.com/shuheigoda) (Data Scientist at Wantedly; Tokyo, Japan), [Y.Nakama](https://www.kaggle.com/yasufuminakama), [copasta](https://www.kaggle.com/copasta) (ML Engineer at Hoge; Tokyo, Japan)\n- [5th place](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322875) - [xia](https://www.kaggle.com/xiamaozi11), [DIKW](https://www.kaggle.com/bestpredict), [yuqiang](https://www.kaggle.com/lxf615712)\n- [6th place](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323237) - [Darragh](https://www.kaggle.com/darraghdog) (AI Researcher at DoubleYard; Dublin, Ireland), [Dieter](https://www.kaggle.com/christofhenkel) (Deep Learning Data Scientist at Nvidia; Munich, Germany), [Pascal Pfeiffer](https://www.kaggle.com/ilu000) (Principal Data Scientist at H2O.AI; Cologne, Germany)\n\n## Main ideas\n\n* The correct way to make validation splits was by using StratifiedGroupKFold with patient notes as groups and cases as states;\n* Everyone used an ensemble of diverse models based on DeBERTa variations, ELECTRA, and other LLMs.\n* Adversarial training (and [AWP](https://arxiv.org/abs/2004.05884) in particular) was used in one of the previous competitions and now became a widely used approach to improve robustness;\n* There were a lot of unlabeled data, so MLM pretraining and pseudo-labeling were quite useful\n* It was a good idea to use various approaches to regularization because of the small size of the data: among them augmentations, dropout variations, and token dropout.\n\n## Findings\n\n* As is common in text-based competitions, specific processing was necessary. For example, in some cases, the annotations ended within a token span, creating a conflict between annotations and tokenization. It was also important to process common abbreviations to avoid incorrect processing;\n* An interesting approach involved the use of knowledge distillation. Some large DeBERTa versions are too large for use in Kaggle Notebooks, but knowledge distillation can help with this;\n* It was found that using GRU/LSTM in the head could be helpful. One possible explanation is that annotators could miss repeated occurrences, and RNNs could help address this issue.\n---","metadata":{"papermill":{"duration":0.003532,"end_time":"2023-07-11T13:38:08.966819","exception":false,"start_time":"2023-07-11T13:38:08.963287","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# 6. Overview of recent Kaggle NLP techniques\n\nFinally, we would like to provide an overview of techniques used in the latest NLP competitions. While the full list of approaches used would be too long, we would like to highlight those we deemed important, interesting, or emerging.\n\nMany competitions don't provide sufficient annotated data, leaving Kagglers to deal with the problem on their own (though it's close to real-world NLP). As a result, various approaches to pseudo-labeling are often used (almost all winning teams mentioned it in [NBME](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/overview), [English Language Learning](https://www.kaggle.com/c/38321), [Predicting Effective Arguments](https://www.kaggle.com/c/35308), and [CommonLit](https://www.kaggle.com/c/25914\t)). An additional problem is the quality of annotations, which is sometimes not high due to disagreement between annotators. Various data augmentation (mentioned in #1 write-ups for [Coleridge](https://www.kaggle.com/c/25925/discussion/248251), [NBME](https://www.kaggle.com/c/33607/discussion/323095), [CommonLit Readability Prize](https://www.kaggle.com/c/25914/discussion/257844), [Predicting Effective Arguments](https://www.kaggle.com/c/35308/discussion/347536), and [chaii - Hindi and Tamil Q&A](https://www.kaggle.com/c/30060/discussion/287923)) and text processing techniques are obvious choices, but adversarial training has become a popular solution to this difficulty (see e.g. [#1 write-up](https://www.kaggle.com/c/31779/discussion/313177) for \"Evaluating Student Writing\", [#2 write-up](https://www.kaggle.com/c/33607/discussion/323085) for NBME, many more teams mentioned AWP (Adversarial Weight Perturbation) being used). We would also like to emphasize that data processing is still important in the LLM world. One reason is that different models are trained with different tokenizers and preprocessing, and it is up to Kagglers to process data to suit these models. Good processing was especially important for tasks with non-English text, e.g. in [chaii - Hindi and Tamil Q&A](https://www.kaggle.com/competitions/chaii-hindi-and-tamil-question-answering/). \n\nWhile BERT variations are the most widely used models (with ** DeBERTa ** being one of the best), other models are also used, such as XLM or task-/language-specific models. Interestingly, when competition data has a sequential nature, adding RNN layers on top of transformers is sometimes a good idea. Regarding architecture, we would also like to highlight that training models with auxiliary targets is a good idea, whether to increase the score or to ensure the robustness of the solution.\n\nPrompt learning is a widely used approach in modern Deep Learning papers, and it is starting to be used on Kaggle as well (see, e.g. [#5 write-up](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332418) for \"U.S. Patent Phrase to Phrase Matching\").\n\nLooking forward to the explosion of activity with LLMs in Kaggle competitions! (**upd 16.07.23:** this was written prior to the essay submission deadline, July 5th, since then, Kaggle has launched [\"LLM Science Exam\"](https://www.kaggle.com/competitions/kaggle-llm-science-exam/overview) and [\"CommonLit - Evaluate Student Summaries\"](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/overview) so the prediction turned out to be prophetic albeit being easy to make given all the LLM hype outside Kaggle).\n\n---","metadata":{"papermill":{"duration":0.003676,"end_time":"2023-07-11T13:38:08.974539","exception":false,"start_time":"2023-07-11T13:38:08.970863","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Appendix\n\n## Glossary & Tutorials\n\n#### Transformer\n\nTransformer is a Deep Learning architecture introduced in a seminal [paper](https://arxiv.org/abs/1706.03762) called \"Attention Is All You Need\" by Vaswani et al. in 2017. Transformers have demonstrated impressive capabilities in a range of NLP applications, such as machine translation, text classification, named entity recognition, and question answering, among others. They have emerged as leading models in the field of NLP due to their capacity to capture broader connections, effectively process extensive context, and deliver outstanding performance through parallel computing.\n\nFor an intro, check [\"The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer) by Jay Alammar. \n\n#### BERT\n\nBERT (Bidirectional Encoder Representations from Transformers, [paper](https://arxiv.org/abs/1810.04805)) is a deep learning model introduced by researchers at Google AI in 2018 that has made significant advancements in natural language processing (NLP) tasks. The key innovation of BERT is its bidirectional training strategy. Unlike previous models that only considered the context to the left or right of a given word, BERT learns representations by considering the entire context from both directions. BERT has achieved remarkable results and state-of-the-art performance on various NLP benchmarks and competitions.\n\nFor an intro to BERT, check [this part](https://lilianweng.github.io/posts/2019-01-31-lm/#bert) of Lilian Weng's post \"Generalized Language Models\". \n\n#### RoBERTa\n\nRoBERTa (Robustly optimized BERT approach, [paper](https://arxiv.org/abs/1907.11692)) is an extension and improvement of the BERT model by Facebook AI in 2019. The modifications include (source: [PapersWithCode](https://paperswithcode.com/method/roberta)):\n - training the model longer, with bigger batches, over more data\n - removing the next sentence prediction objective\n - training on longer sequences\n - dynamically changing the masking pattern applied to the training data. The authors also collect a large new dataset (CC-News) of comparable size to other privately used datasets, to better control for training set size effects.\n \nFor an intro to RoBERTa, check [this part](https://lilianweng.github.io/posts/2019-01-31-lm/#roberta) of Lilian Weng's post \"Generalized Language Models\". \n\n\n#### GPT\n\nGPT (Generative Pre-trained Transformer, [paper](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)) is a deep learning model based on transformers, introduced by OpenAI in 2018. GPT is a language model that is pre-trained on a large corpus of text data. The model learns to predict the next word in a sentence based on the context provided by the previous words. This pre-training phase allows GPT to develop a strong understanding of language patterns, grammar, and contextual relationships.\n\nFor an intro to [GPT](https://lilianweng.github.io/posts/2019-01-31-lm/#gpt), [GPT-2](https://lilianweng.github.io/posts/2019-01-31-lm/#gpt-2), and [GPT-3](https://lilianweng.github.io/posts/2019-01-31-lm/#gpt-3), check the corresponding parts of Lilian Weng's post \"Generalized Language Models\". Jay Alammar's posts [The Illustrated GPT-2 (Visualizing Transformer Language Models)](http://jalammar.github.io/illustrated-gpt2/) and [How GPT3 Works - Visualizations and Animations](https://jalammar.github.io/how-gpt3-works-visualizations-animations/) are also marvelous. \n \n#### Pseudo-labeling\n\nPseudo-labeling (explained [here](https://www.kaggle.com/code/vad13irt/semi-supervised-learning) on Kaggle by [Vadim Irtlach](https://www.kaggle.com/vad13irt)) is a semi-supervised learning technique that combines both labeled and unlabeled data to improve the performance of a model. It includes the following steps ([source](https://www.kaggle.com/code/sophieb/a-journey-through-text-data-competitions/notebook)): \n - Train a model using labeled data\n - Predict labels (pseudo-labels) with the trained model using unlabeled data\n - Retrain the model using both labeled and pseudo-labeled datasets\n - Iterate between steps 2 and 3 as necessarily\n - Use the final trained model to predict test data\n \n \n#### DeBERTa\n\nDeBERTa (Decoding-enhanced BERT with disentangled attention, [paper](https://arxiv.org/abs/2006.03654)) is a variant of the BERT model introduced by Microsoft Research in 2020. The key innovation in DeBERTa lies in disentangled attention, which aims to improve the attention mechanism's ability to capture interactions between tokens. Traditional attention mechanisms in BERT have a single attention head that attends to all token positions simultaneously. In DeBERTa, attention is decomposed into multiple, disentangled attention heads, each attending to a specific subset of positions.\n\nFor an intro to DeBERTa, you can check the [blog post](https://wandb.ai/akshayuppal12/DeBERTa/reports/The-Next-Generation-of-Transformers-Leaving-BERT-Behind-With-DeBERTa--VmlldzoyNDM2NTk2) \"The Next Generation of Transformers: Leaving BERT Behind With DeBERTa\" by Akshay Uppal.\n\n#### Data augmentation\n\nData augmentation refers to the technique of artificially creating additional training examples by applying various transformations or modifications to the original dataset. Mostly efficient in Computer Vision, it was also adopted in NLP where the most common techniques include synonym replacement, random swaps, insertions, deletions, back-translation, etc. \n\nFor an intro, you can check [this blog post](https://neptune.ai/blog/data-augmentation-nlp) by [Shahules](https://www.kaggle.com/shahules).\n\n#### Adversarial training\n\nThe main idea behind adversarial training ([paper](https://arxiv.org/pdf/1605.07725.pdf)) is to expose the model to perturbations or modifications in the input data during training, simulating potential attacks or discrepancies that may occur in real-world scenarios. This helps the model learn to generalize better and perform well even when faced with adversarial examples or unseen test data. Some methods are reviewed in [this](https://www.activeloop.ai/resources/glossary/adversarial-training/) blog post by Activeloop.\n\n#### Prompt learning\n\nPrompt learning is a paradigm to adapt pre-trained language models to downstream NLP tasks, which modifies the input text with a textual template and directly uses pre-trained LMs to conduct pre-trained tasks. One of the seminal papers is [\"Parameter-Efficient Transfer Learning for NLP\"](https://arxiv.org/abs/1902.00751) by Google Research which introduced BERT adapter layers ([GitHub](https://github.com/google-research/adapter-bert)). An overview of papers on prompt learning is given [here](https://github.com/thunlp/PromptPapers). \n\n---","metadata":{}},{"cell_type":"markdown","source":"**Preparing the submission file**","metadata":{}},{"cell_type":"code","source":"# preparing the submission file\nimport pandas as pd\n\nsub_df = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\n\nsub_df.value = [\"Kaggle Competitions\", \n \"https://www.kaggle.com/code/kashnitsky/essay-competition-cards-recent-nlp-competitions/notebook\",\n \"https://www.kaggle.com/code/sophieb/a-journey-through-text-data-competitions/comments#2340366\",\n \"https://www.kaggle.com/code/jocelyndumlao/ai-report-kaggle-competitions/comments#2340394\",\n \"https://www.kaggle.com/code/trushk/2023-kaggle-ai-report-generative-ai/comments#2332657\"\n ]\n\nsub_df.to_csv('submission.csv', index=False)\nsub_df","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.execute_input":"2023-07-11T13:38:08.984706Z","iopub.status.busy":"2023-07-11T13:38:08.983969Z","iopub.status.idle":"2023-07-11T13:38:09.048603Z","shell.execute_reply":"2023-07-11T13:38:09.047279Z"},"papermill":{"duration":0.072852,"end_time":"2023-07-11T13:38:09.051295","exception":false,"start_time":"2023-07-11T13:38:08.978443","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]}]}