{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Approximating global minima with very small datasets: Advances from NLP Kaggle competitions\n\n","metadata":{}},{"cell_type":"markdown","source":"One of the greatest successes in NLP has been the advent of the transformer architecture. However, the scalability of transformers has put the onus of size and complexity back on the court of data collection and task pretraining. While advances like instruction fine-tuning [(Wei 2022)](https://arxiv.org/pdf/2109.01652.pdf) have been proven to increase zero-shot and downstream performance, many people have raised objections to the ability of models to generalize to out-of-vocabulary and out-of-distribution tasks ([McCoy 2019](https://arxiv.org/pdf/1902.01007.pdf), [Jin 2019](https://arxiv.org/pdf/1907.11932.pdf), [Muennighoff 2021](https://arxiv.org/pdf/2210.07316.pdf), [Tankur 2021](https://arxiv.org/pdf/2104.08663.pdf), among others). Moreover, besides the monetary imperative in annotating large datasets, NLP practitioners experience model-dependent ideosyncratic instabilities and performance challenges during deployment. \n\nFortunately, Kaggle competition participants have developed a large set of tools to work with small text datasets and different models. Kaggle NLP competitions center around very hard problems, featuring deliverately tough train/test splits. These tools address the high variance and combinatorial nature of language, the extreme discrepancies in performance observed by shuffling the data order or layer initialization [(Dodge 2020)](https://arxiv.org/pdf/2002.06305.pdf), and the fact that unlike computer vision models, NLP transformers are very resistant to input perturbations for reasons that will be explained below. \n\nStill, the question remains of *if and when* the many Kaggle-made strategies to mitigate this challenge are useful. In this paper, we explore and test these ideas and their limits using common NLP benchmarks and toy experiments. All the ideas discussed come from top-scoring formulations as we consider them to be empirically validated in the setting where they were implemented.","metadata":{}},{"cell_type":"markdown","source":"\n\n## 1. Enforcing low kurtosis in the input distribution achieves SOTA\n\n### 1.1 Anonymizing entities improves model performance in some tasks\n \nWide datasets are considered the gold standard due to their generalizing and regularizing properties, so a counter-intuitive empirical finding on Kaggle is that lower kurtosis through anonymization has a positive effect on performance. \n\nOur first example comes from the [Kaggle Gendered Pronoun Resolution challenge](https://www.kaggle.com/competitions/gendered-pronoun-resolution/code), where participants had to identify the person to which a pronoun referred in a sentence and link it to a set of candidate names. Kaggle user [boliu0](https://www.kaggle.com/boliu0) reduces the heterogeneity of the dataset by replacing the target names with a set of 4 name pairs. The mapping looks as follows: \n\n\n [ { 'female':['Alice','Kate'], 'male': ['John','Michael']},\n { 'female':['Elizabeth','Mary'], 'male': ['James','Henry']},\n { 'female':['Kate','Elizabeth'], 'male': ['Michael','James']},\n { 'female':['Mary','Alice'], 'male': ['Henry','John']}]\n\n\n\nThe author reports, *“[F]or any given sentence, the names in the sentence are replaced 4 times, once with each pronoun mapping, producing 4 new identical sentences where only the name is different.“* ([ibid](https://www.kaggle.com/competitions/gendered-pronoun-resolution/discussion/90334)). In their training regime, these sentences are mixed in with the unaltered training data, the rationale being that this forces the model to learn the identity function from the surrounding words. The user experimented with fewer and more names but reported no noticeable improvement. \n\n\n#### Experiment 1: Whether NER replacement works with other tasks and data\n\nConsider the following mapping:\n\n\n anon_map = {\n 'PERSON': 'this person',\n 'NORP': 'this group',\n 'ORG': 'this organization',\n 'GPE': 'this place',\n 'LOC': 'this place',\n 'PRODUCT': 'this thing',\n 'WORK_OF_ART': 'this title',\n 'LAW': 'this rule'\n }\n\n\n\nUsing the [Spacy NER pipeline](https://spacy.io/usage/processing-pipelines), we map unique tokens to their respective strings in `anon_map` and concatenate them with the unaltered dataset as [boliu0](https://www.kaggle.com/boliu0) suggests. We downsample the SNLI and IMDB datasets to 10k and 2.5k examples due to compute restraints and in keeping with the spirit of the low-sample regimes being described. The pipeline provide advantages over the baseline to the following datasets (3-seed averages):\n\n| Model | IMDB 2k | MNLI 10K | MNLI 2K | COLA | Notes |\n| --- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| Deberta-base | 0.90 | **0.90** | 0.8631 | **0.89** | Same number of training steps as models below |\n| Deberta-base + entity anonymizing | 0.87 | **0.90** | **0.8709** | 0.88 | Faster convergence |\n\n","metadata":{}},{"cell_type":"markdown","source":"\n\n### 1.2 The ‘[MASK]’ token increases the signal of important features\n\n\nSome augmentation regimes have been proposed using the mask token ([S Yavuz, 2020](https://aclanthology.org/2020.emnlp-main.412.pdf), [D Park, 2019](https://www.mdpi.com/2073-8994/11/11/1393), [H Zhang, 2020](https://aclanthology.org/2020.coling-main.379.pdf)), but they do not perturb the training input during finetuning with the `[MASK]` token. Kaggle user \n[CPMP](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323085) uses random masking during task finetuning in a medical NER task, where participants were challenged to identify symptoms in patient diagnostics exams. [CPMP](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323085) reports that masking the named entities themselves resulted in lower cross-validation performance (please see 1.3 for a discussion on masking entities). Unfortunately, the user does not specify if the masked tokens were used to compute the loss when they overlapped with entities. However, [CPMP](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323085) concludes that the reduced performance when masking the named entities was “probably because some feature text are very small and could be entirely masked,” which lends credence to the hypothesis that besides a regularization effect of token dropout ([Zhang, 2017](https://arxiv.org/pdf/1611.03530.pdf)), masking increases the saliency of important features. \n\n\n#### Experiment 2: Whether masking features suppresses signal information\n\nTo test the hypothesis put forth by [CPMP](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323085), we conduct monotonically increasing masking rates per epoch starting at 5%, following the concept of increasing difficulty by [Bengio et al., 2009](https://dl.acm.org/doi/10.1145/1553374.1553380). Only masked inputs are used for training, and inference is conducted on unperturbed text. We find that small masking rates improve results for the MNLI dataset but increasing rates hurt performance, which lends credence to [CPMP](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323085)'s theory of signal masking per the transformer probing findings in [Maudslay, 2021](https://arxiv.org/pdf/2106.02559.pdf). Further, MNLI shows great variability per epoch as remasking was done on a per-epoch basis, leaving the masking of important and unimportant features to chance. Interestingly, masking once per seed and training several epochs returned comparable results to unperturbed text, and decreasing masking rates had the worst results on all datasets.\n\n\n| Model | IMDB 2k | MNLI 2k | MNLI 10k | COLA | Notes |\n| --- | ----------- | --- | -- | -- | - |\n| Baseline | 0.90 | 0.8631 | **0.90** | **0.89** | - |\n| Mask uniform | 0.9422 | **0.87566** | 0.90 | 0.89 | Different tasks require different masking rates |\n| Mask increasing | **0.9434** | 0.8513 | 0.8962 | 0.8885 | COLA requires very small increases | \n| Mask decreasing | - | 0.8358 | 0.8962 | 0.8724 | - | \n\n\n#### A last word on masking strategies\n\nTo our knowledge, [Moon, 2021](https://ojs.aaai.org/index.php/AAAI/article/view/17601/17408f) is the only study that has explored masking schedules similar to these. They go beyond random assignment and overfit a model, and then use attention scores to decide which tokens to mask. They also report that random masking outperforms high-tf-idf masking. For a possible answer to the failure of this scheme, please see [Nikolaev and Pado 2023](https://arxiv.org/pdf/2301.13039.pdf). For a similar failure of signals-based masking, please see [Althammer, 2021](https://arxiv.org/pdf/2106.05768.pdf)\n\nIn a different pipeline in the same competition, Kaggle user [Dieter](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323237) reports using random masking along with other tools to increase uniformity in the text, including lowercasing and the use of special tokens for NER targets that had different spellings. The author reports these strategies did not “help significantly” but created diversity in the pipeline. For other ideas that include this technique in different datasets and tasks, please see [discussion 1](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347536) and [discussion 2](https://www.kaggle.com/competitions/tweet-sentiment-extraction/discussion/159505).\n\n\n### 1.3 Can ‘[MASK]’ features prevent the model from learning heuristics?\n\n\n\nDespite all the corroborating evidence we have gathered, empirical results from Kaggle appear to indicate that masking entities is possible — so long as the right training set-up is devised. The [Kaggle Coleridge Initiative](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data) was a challenge to link dataset titles in scientific articles to their corresponding datasets. The participants faced extremely unbalanced training examples that resulted in transformers learning heuristics. For this reason, a lot of winning entries relied on statistical models aided by hand-crafted rules like RegEx. However, the first-prize winner, Kaggle user [Nguyen Quan Anh Minh](https://www.kaggle.com/competitions/coleridgeinitiative-show-us-the-data/discussion/248253), noted that string-matching would fail in out-of-distribution and noisy examples:\n\n\n*“There are some solutions that don’t need semantic context and only needs dataset title. But those solutions are only suitable for this contest data since almost the dataset title includes keywords such as Study, Survey, Database, Dataset, …” *\n\nIn their pipeline, the author masks the feature (the dataset name) in the support set, retrieve an unmasked corresponding example from the query set, and use ArcFace loss to compute the loss for the pair. In doing so, they effectively turn the standard NER pipeline on its head by detecting whether the adjacent tokens, rather than the entity tokens, indicate the identity of the named entity in the sentence.\n\n\n\nSource: [Nguyen Quan Anh Minh](https://www.kaggle.com/competitions/coleridgeinitiative-show-us-the-data/discussion/248253)\n\n\n### 1.4 Error correction improves generalization \n\nPractitioners remember when avoiding `` tokens was paramount to high performance, but avoiding unknown tokens became less important as new tokenizers greatly expanded vocabulary coverage. Many claims exist in the literature about improving robustness through word replacement and misspelling (e.g., [Marivate 2020](https://arxiv.org/pdf/1907.03752.pdf) for an early example) as well as software libraries that offer this type of corruption as \"adversarial training.\" The spectacular lack of success of these strategies on Kaggle indicate they are not suitable for transformers. The many possible explanations include first, the humongous probability space for perturbation, which reduces the possible overlap with real examples; second, the large search space for random perturbation means that a noise-induced reduction of predictive capacity is possible, and, near the limit, will produce dimensional collapse; third, even when used with prudence, transformers can learn noising functions and map them to the original inputs.\n\nMoreover, empirical evidence shows that normalizing the input through error correction improves the ability of the model to generalize, even when the errors could even be considered features. The [Predicting Effective Arguments](https://www.kaggle.com/c/feedback-prize-effectiveness) challenge involved determining whether parts of student essays written by middle school children were “effective,” “adequate,” or “ineffective” in reference to their argumentative quality. In this context, it’s reasonable to assume that misspellings and bad grammar are correlated with a lack of interest in homework or schooling and thus are indicators of argument effectiveness. However, error correction improves accuracy by ~1% percent:\n\n \n| Model | Accuracy |\n| --- | ----------- |\n| Feedback baseline | 0.67 | \n| Feedback proofread (languagetool) | **0.68** | ","metadata":{}},{"cell_type":"markdown","source":"\n\n## 2. Reformulation through auxiliary tasks increases performance and stabilizes training\n\n![](\"https://i.imgur.com/oItdJ0X.jpg\")
\n\nWhile [Gururangan 2020](https://arxiv.org/pdf/2004.10964.pdf) recommends MLM for domain adaptation along with task-adaptive pretraining, Kaggle users have devised a number of pretraining and auxiliary tasks that are competitive with MLM, allow for longer training, and increase model performance. While these gains are many times only a few percentage points higher than baseline methods, they are free gains in the sense that they do not require more labeled data, and unlike adaptive regimes, some of these pipelines allow for the easy deployment of multistage architectures. \n\n### 2.1 Auxiliary tasks for continuous and categorical problems\n\n\n#### Continuous targets\n\nA common winning idea in Kaggle NLP challenges with continuous targets involves reinterpreting the problem as a classification/regression task. In the [Jigsaw Bias in Toxicity](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/overview) challenge, participants had to predict the toxicity of comments targeting different social groups. Kaggle user [Yurybolkonsky](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/discussion/103280) first trained `BERT-large` to predict which social group was being targeted by the comment and then asked it to assign a toxicity score to the sentence, reinterpreting it as a classification -> regression task: \n\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1192776%2F957bfdf462369a6eca31bf4ef4e7942b%2Fbert_aux.PNG?generation=1565469511607218&alt=media\")
\n\n\n*Pipeline from their model. Source: [Yurybolkonsky](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/discussion/103280). For information on how categorical information reduces the vector search space, please see §6.2*\n\nSimilarly, in the [Google QA](https://www.kaggle.com/c/google-quest-challenge) challenge, where users predicted subjective human judgments on Google question-answer pairs, Kaggle user [Yaroshevskiy](https://www.kaggle.com/competitions/google-quest-challenge/discussion/129840) reports, *\"In addition to the common MLM task, the model also predicted 6 auxiliary targets (question_score, question_view_count, question_favorite_count, answer_score, answers_count, is_answer_accepted) with a linear layer on top of pooled LM output.\"* No metrics for the auxiliary task are provided, so we cannot ascertain whether the model learned distributional properties of higher-order semantics from the task or if the extended training allowed for further domain adaptation. As a final note, it must be mentioned that categorical auxiliary tasks for continuous targets allow the practitioner to diversify the loss landscape and utilize dropout when it may not be possible in regression (cf. [Ozgur 2020](https://www.researchgate.net/publication/344274687_Effect_of_Dropout_Layer_on_Classical_Regression_Problems)).\n\n\n#### Categorical targets\n\nSo there is evidence we can learn distributional properties by doing classification on the task data. However, can we make up our own labels when human-made categories are not available? An example is offered by Kaggle user [TK](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369956), who revives a somewhat discredited task formulation for the [Predicting Effective Arguments](https://www.kaggle.com/c/feedback-prize-effectiveness) challenge, where one of their many auxiliary tasks was mapping the labels “Effective” and “Ineffective” to a verbalizer:\n\n\n\n Positive Tokens List: ['Excellent', 'Good', 'A', '5', '10', '100', 'excellent', 'good', 'High', 'high', 'Strong', 'strong']\n Negative Tokens List: ['1', 'C', 'D', 'F', 'Bad', 'Limited', 'limited', 'poor', 'Poor', '0', 'Low', 'low']\n
\n\n\nAgain, it is not clear if the diversity of the tokens allowed the author to find better minima, as mapping the probability of a sentence label to individual tokens arbitrarily changes the loss landscape, as we will see below. Alternatively, the large number of tokens in the verbalizer might have had a regularization effect like the one we saw in (§1). This type of token mapping formulation received considerable attention at the beginning of the transformer era (e.g., [Yo, 2019](https://ieeexplore.ieee.org/document/8903313), [Sun 2019](https://aclanthology.org/N19-1035/)) but faded out as it became unclear whether the deep semantic relationships within the model can be mapped among tokens, and if, given enough data, the model will learn a mapping to any token(s). Because of this, it appears that the choice of vocabulary and verbalizer could only aid in the speed of convergence. Empirical results from Kaggle have further muddied the waters. In the [Patent Phrase to Phrase Matching competition](Patent Phrase to Phrase Matching competition), participants calculated the semantic similarity score for `{anchor: target}` pairs in the patent domain. Kaggle user [Gezi](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332243) \ndevised a verbalizer with a question: \n\n\n Are they similar? __ anchor[SEP]target[SEP]Context,\n
\n\n\n[Gezi](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332243) manually sets the answer to `YES` for words with a cosine score of 1 and uses a pattern-exploiting model ([Schick, 2020](https://arxiv.org/abs/2001.07676)) to verbalize other examples (for more details, please refer to [Deng, 2022](https://arxiv.org/abs/2208.01312)). Although the indicates they gained 1% – 3% micro-f1 score in SEMEVAL 2022 Task 4 ([Deng, 2022](https://arxiv.org/abs/2208.01312), also the first entry) with this idea, the usefulness of the approach remains unclear, as we will see below. Further, on the comments in the Kaggle competition, other users expressed they had tried the same idea and failed, and [Gezi](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332243) states: *\"I think what is more important is not the template and verbalizer, but how to convert the prompt learning from a multilabel classification task to a continuous value prediction task\"* \n\n#### Can newly initialized tokens alleviate the issue of prompt selection?\n\nMore positive results for token mapping were reported by Kaggle user [Tifo](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347386) in [Predicting Effective Arguments](https://www.kaggle.com/c/feedback-prize-effectiveness) by creating new `[CLS]` tokens for category information and the labels to be predicted. The input looks as follows:\n\n\n```\n [CLS CLAIM] [Ineffective] [Adequate] [Effective] I think that the face is a natural landform because… [END CLAIM]\n [CLS EVIDENCE] [Ineffective] [Adequate] [Effective] The reason this is so is because… [END EVIDENCE]\n ```\n\nThey report, *\"This time, the predicted probabilities are not `[CLS]` and `[END]`, but `[Ineffective]`, `[Adequate]`, and `[Effective]`. And the activation function is using Sigmoid but not Softmax. Then the prob of these 3 tags would be scaling to 1.”\n\n\n#### Experiment 3: Whether word choice matters in prompts and if new tokens can ameliorate this source of stochasticity\n\nFor this experiment, we use the 210k-item News Category Dataset ([Misra 2020](https://arxiv.org/abs/2209.11429)) and try to predict the author of a piece (n_authors = 145 after pruning outliers). Using the newspaper section as auxiliary input (§2.1, §6.2), we use first the gold annotations and then shuffle the gold tokens, where labels are assigned incorrecly but consistently to different classes (e.g., *Arts* becomes *Science*, *Science* becomes *UK Politics*, and so on). We also initialize new tokens for each newspaper class to study the method proposed by [Tifo](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347386). The results show that adding category information significantly outperforms the baseline, but the shuffled categories have better performance than the gold labels. New tokens outperform all other methods. These results provide preliminary evidence that prompt reformulations only change the loss landscape of the problem and can be controlled by initializing new tokens.\n\n\n| Model | f1 score | Avg. training loss | Example |\n| --- | ----------- | ----------- | --- |\n| Control (no section information in input)| 0.3076 | 2.8295 | `Hamilton's Schuyler Sisters Are Singing At Super Bowl LI[SEP]We're so lucky to be alive right now!` |\n| Gold | 0.3798 | 2.4401 | `ARTS & CULTURE[SEP]Hamilton's Schuyler Sisters Are Singing At Super Bowl LI[SEP]We're so lucky to be alive right now!` |\n| Shuffle | 0.3622 | 2.5832 | `WORLDPOST[SEP]Hamilton's Schuyler Sisters Are Singing At Super Bowl LI[SEP]We're so lucky to be alive right now!` |\n| Re-Shuffle | 0.3742 | 2.4778 | `TECH[SEP]Hamilton's Schuyler Sisters Are Singing At Super Bowl LI[SEP]We're so lucky to be alive right now!` |\n| New token | **0.3800** | 2.4670 | `[ARTS & CULTURE][SEP]Hamilton's Schuyler Sisters Are Singing At Super Bowl LI[SEP]We're so lucky to be alive right now!` |\n\nInterestingly, training for longer results in better results for the shuffled class tokens, indicating that different tokens only change the loss landscape rather than the task formulation:\n\n\n| Model | f1 score | Avg. training loss | \n| --- | ----------- | ----------- |\n| Gold (longer)| 0.4257 | 1.9740 | \n| Shuffle (longer) | **0.4319** | 1.9151 | \n\n\n### 2.2 Auxiliary loss through second head\n\nMoving beyond token mappings, some users create more difficult tasks formulated for a second classification head. Unfortunately, information about loss or error analysis has not been shared for these tasks, so it remains unclear if they teach the model new information or if the difficulty has a regularizing effect. Here is an example from the [Clinical Patient Notes](https://www.kaggle.com/c/nbme-score-clinical-patient-notes) NER competition:\n\n*\"In addition to the standard QA formulation, https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323095 includes 3 binary classification heads:*\n```\nNon-named entity token: {0, 1}\n(Any) start of named entity token: {0, 1}\n(Any) end of named entity entity token: {0, 1}\n```\n\n\n### 2.3 A cautionary word on leakage\n\nA mandatory note is that the pursuit of auxiliary training can lead to leakage. \n\nThis is good for statistics competitions but obviously bad for industry and commercial uses. Kaggle user [Khyeh](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/discussion/100530) made a very clever finding about the [Jigsaw Bias in Toxicity](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/overview) dataset, where toxicity scores were averages of `[no, yes] -> {0, 1}` binary determinations by annotators:\n>\n[The] NN also predicts toxicity_annotator_count.min()/toxicity_annotator_count. I added this as one of the targets to be predicted since I found the toxicity value is influenced by the number of toxicity_annotator_count as well, training NN to predict how many annotators for toxicity might help predicting toxicity as well. For example: for toxicity=0.1666666 (1/6), the annotator count happens to be 6, which means the toxicity annotator count determine what values are possible for toxicity (in this case: 0/6, 1/6, 2/6…, 6/6 are possible values).\n","metadata":{}},{"cell_type":"markdown","source":"## 3. Empirically validated data augmentation strategies\n\n![](\"https://i.imgur.com/OVWCxyD.png\")
\n\nAcademic papers are rife with data augmentation schemes that don’t work or require a lot of trial and error. The Kaggle methodology to data augmentation inherits the logic from [Bengio 2009](https://dl.acm.org/doi/10.1145/1553374.1553380), [Gururangan 2020](https://arxiv.org/pdf/2004.10964.pdf) and [Phang 2019](https://arxiv.org/pdf/1811.01088.pdf), but it differs in substantial ways that have not been reported. It remains an open question which combination of the methods below constitutes the best pretraining regime.\n\n### 3.1 Few-sample data mining\n\nThe challenges of unsupervised data mining include that signals methods are unable to capture deep semantic relations, and pretrained sentence transformers display serious representation biases described in [Nikolaev and Pado 2023](https://arxiv.org/pdf/2301.13039.pdf) that result in near-identical examples in the upper range of cosine similarity and high heteroscedasticity in the intermediate regions. Kaggle user [Mathis Lucka](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/257844) devised an unsupervised pipeline for the [CommonLit Readability Prize](https://www.kaggle.com/competitions/commonlitreadabilityprize), where the organizers provided 2834 examples with labels rating the reading complexity. After building a corpus that had some overlap with the target distribution, [Mathis Lucka](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/257844) utilized a four-stage retrieval/scoring system that looked as follows:\n\n`Sentence-transformer retriever > Transformer voting > Filter on standard error > New sentence-transformer training`\n\nThe author uses `paraphrase-TinyBERT-L6-v2` to retrieve the top-5 matches and assigns a label using three transformer voters pretrained on the original data. Then, the error *ϵ* of the pseudo-labeled mined examples is the difference between the label of the original and the assigned pseudolabel score. From that, examples where *ϵ* is higher than the standard error of the inter-annotator agreement of the originals are removed. Afterward, a sentence transformer is trained from scratch using `[original, pseudolabel]` pairs as `[u, v]`, and *ϵ* is used as the target metric. This method outperformed large ensembles tuned on different hyperparameters and seeds, indicating the regime provided a better representation of the global loss landscape for this problem. \n\n\nWe now move on to an interesting example where the author skews the distributional characteristics of the dataset. In [Jigsaw Bias in Toxicity](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/overview), Kaggle user [Pronkin Nikita](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/discussion/101927) reports using the following flow to reduce heuristic biases:\n\n>\n* Take an amazing datasdet of (8M senteces from wikipedia)[https://www.kaggle.com/mikeortman/wikipedia-sentences]\n* Run predictions of all 18 targets on them with the best single model\n* Filter out senteces that don't mention any identites\n* Filter out senteces with high toxicity predictions (some movies and music albums ahve \"toxic\" titles like Hang 'em high!)\n* Set the targets for toxicity and subtypes of toxicity to 0 since you know that sentecces from wikipedia should not be toxic\n* Add those senteces to the training data to kill the bias towards some identites\n\n\n### 3.2 Data synthesization\n\nBecause synthesization has been studied in depth in the literature, only two insights from Kaggle competitions are shared here. First, large amounts of synthetic data can match the performance of MLM and pseudolabeling schemes for domain and task adaptation. For an example discussion, please see [this conversation](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347433). Second, compared to autoregressive encoders, seq-to-seq transformers appear to generalize better to out-of-distribution text as well as highly structured data. Consider the following synthetic examples from [Learning Equality](https://www.kaggle.com/competitions/learning-equality-curriculum-recommendations), where text is matched to clustering trees:\n\n| Gold example | Gold tree | Prompt | T5-large (770M) | ChatGPT (20B) |\n| - | - | - | - | - |\n| Proof of the logarithm change of base rule Sal proves the logarithmic change... | 2.1.2 - Logarithms <_topic_> 2.1 - Exponents and Logarithms <_topic_> Unit 2 - Exponential and Logarithmic Functions <_topic_> Mathematics <_topic_> Grade 10 | In mathematics, addition, represented by the symbol +, is an operation... | Addition and subtraction <_topic_level_1> Mathematics II <_topic_level_1> High school <_topic_level_1> Math by grade | 1.3 - Addition in Mathematics <_topic_level_1> 1.2 - Mathematical Operations <_topic_level_1> Mathematics <_topic_level_1> Science <_topic_level_1> Academic Subjects |\n\n\nT5 also appears to replicate higher-order semantics and some human behavioral patterns. Kaggle user [Raja Biswas](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347433) reports: \n>\nThe trained T5 model was pretty amazing, we couldn't distinguish which one was generated text and which one was original. The augmentations copied the student writing styles, identity masks e.g. and included effects such as that mentioned by @hengck23 in the previous competition discussion:\n‘just imagine you are asked to write the essay for homework. you are a bad student and want to copy other people work. you want to do so, such that that your teacher will not know that you are copying other people's work.\nyou can: 1) choose an essay of the train data. replace the claim from another essay. (if the other essay has the same position as the first essay)'\n","metadata":{}},{"cell_type":"markdown","source":"## 4. Model Heads\nThe new state-of-the-art in Kaggle competitions has been redefined by the addition of an RNN layer on top of the last hidden states of transformers. RNNs made sporadic appearances at the beginning of the BERT era, but they seemed to focus on the [CLS] token as a pooling mechanism for the intermediate layers of large models, as was the style at the time in computer vision (for an example, please see [Jiahai 2020](https://arxiv.org/pdf/2002.04815.pdf)). In the following section, we review existing knowledge and conduct a couple of ablation studies to understand when this is the case\n\n### 4.1 RNNs, positional artifacts, and sequential dependencies\n\nSurprisingly, several studies probing the transformer architecture lend strong credence to the theory that distributional semantics explain the success of the models. For example, [Pham 2021](https://arxiv.org/pdf/2012.15180.pdf) found that 75%-90% of GLUE remains invariant to input randomization. [Sinha 2021](https://arxiv.org/pdf/2104.06644.pdf) trained a transformer on shuffled data and got comparable results to the BERT model (see also [O'Connor 2021](https://arxiv.org/pdf/2106.08367.pdf) and [Maudslay 2021](https://arxiv.org/pdf/2106.02559.pdf)). Moreover, as discussed earlier, augmentation by randomizing via rules or neural encoders (e.g., [Xie 2020](https://arxiv.org/pdf/1904.12848.pdf)) has shown poor empirical results. \n\nThus, the question arises if the increased performance of RNN heads comes from the higher modeling capacity of the predictor described in [Belkin 2019](https://arxiv.org/pdf/1812.11118.pdf), better modeling of sequential dependencies, or learning the distribution of the signals in transformer logits ([Zhang 2018](https://arxiv.org/pdf/1809.10040.pdf)). Indeed, in [Patent Phrase to Phrase Matching](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching), Kaggle user [Gezi](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332243) hypothesizes that RNNs allow for basis expansion through their bidirectional mechanism, stating, *Change rnn output dim * 2 from (bert out dim like 1024 to 2048) help a lot for some weak models like bert-for-patents and simcse-bert-for-patent. So for weak models we might need models to be widder.”\n\n\n#### Experiment 4: Whether RNNs model context better\n\nIn this experiment, we use the homonym dataset from [Garcia 2021](https://arxiv.org/pdf/2106.13553.pdf), which has sets of words that have the same form but different meanings along with sets of sentences for each meaning. We hypothesize that if an RNN can model the context of the sentence better than vainilla transformer logits, the cosine similarity of homonyms should be lower after they undergo a transformation by the RNN. We train two models, deberta-base with a neural pooling mechanism and deberta-base with an LSTM layer and the same pooling function, and train these layers on 128k examples from the MNLI dataset following [Reimers 2019](https://arxiv.org/pdf/1908.10084.pdf). We also test training on `{word, definition}` pairs from [WordNet](http://mitpress.mit.edu/books/wordnet), with `{antonym, definition}` pairs used as negative examples. The results are as follows:\n\n\n| Model | Homonym Cosine Similarity (lower is better) | Same word Cosine Similarity (higher is better) | Diff. (higher is better) |\n| --- | ----------- | ---- | ---- |\n| Non-LSTM embeddings | **0.40** | **0.40** | **0.0068** |\n| LSTM embeddings: | 0.59 | 0.59 | 0.0005 |\n*Results for MNLI pretraining. Best epochs are displayed.*\n\n| Model | Homonym Cosine Similarity (lower is better) | Same word Cosine Similarity (higher is better) | Diff. (higher is better) |\n| --- | ----------- | ---- | ---- |\n| Base embeddings | **0.47** | **0.48** | **0.01** |\n| LSTM embeddings: | 0.51 | 0.52 | **0.01** |\n*Results for WordNet pretraining. Best epochs are displayed.*\n\n\nThe results of the experiment does not provide statistically significant evidence but indicates that LSTMs push embeddings closer together. This is more appreciable when comparing the cosine similarity of words and their corresponding sentence embeddings (`{word_i, sentence embedding_i}`), reaching differences as high as -0.12 for the baseline vs. 0.26 for LSTMs on MNLI pretraining. \n\n#### Experiment 5: Whether RNNs model language sequences better\n\nIn a toy dataset of ~3,000 examples containing the following 3 classes of tokens (symbols and verbalized):\n\n1. Ascending monotonic sequences of numbers, lengths 10 and 15 (e.g., “1, 2, 3..”; “one, two, three...”)\n2. Randomized sequences of numbers lengths 10 and 15 (e.g., “16, 11, 13..”; “sixteen, eleven, thirteen...”)\n3. Randomized sequences with 3 consecutive numbers, lengths 10 and 15 (e.g., “51, 52, 53, 59,...”; “fifty-one, fifty-two, fifty-three, fifty-nine...”)\n\nThe same two models described above are used, and the LSTM obtains worse results than the baseline, particularly in category 3, seemingly disproving the hypothesis that RNNs can capture weak signals in long sequences:\n\n\n| Model | Precision | Recall | f1-score |\n| --- | ----------- | ---- | ---- |\n| Baseline | **0.85** | 0.80 | **0.81** |\n| LSTM | 0.79 | 0.80 | 0.79 |\n\n\n### 4.2 RNNs can model noise\n\nIn the [Clinical Patient Notes](https://www.kaggle.com/c/nbme-score-clinical-patient-notes) NER task, Kaggle user [CPMP](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323085) used recurrent heads after making the following observation:\n\n>I quickly saw that annotations were inconsistent, as most of the participants did. My guess is that annotators sometimes only identified some of the occurrence of a given feature…\nI further made the hypothesis that annotators were more likely to miss repeated annotations than first occurrence. It means that the annotations have a sequence dependency\n\nIn the Tweet Sentiment Extraction challenge, participants had to identify the specific words that carried the sentiment description for the whole tweet. Kaggle user [Dieter](https://www.kaggle.com/competitions/tweet-sentiment-extraction/discussion/159910) noted that the character-level annotations had noise and the features did not overlap with word boundaries, with different words having different offsets. To model this, they report:\n\n> Instead of predicting the start and end word now, as done by “standard” models I replicate the hidden state of each word as often as the word has characters, so we get a hidden state for each character. In order to differentiate between characters I add a few 768 -> 768 RNN layers. Finally, two linear layers predict the start and end character of the selected text.”\n\nAs [CPMP](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323085) notes, data cleaning makes more sense in commercial projects, but modeling noise is a reasonable alternative when the noise does not follow a distribution, such as when it comes from user-generated inputs or when the distribution is unknown. ","metadata":{}},{"cell_type":"markdown","source":"## 5. Unsupervised features narrow the optimization space\n\n![](\"https://i.imgur.com/ve4ilJQ.png\")
\n\n\nThe advent of the transformer rendered once-common statistical features irrelevant as a lack of improvement in performance indicated that transformer representations were able to capture these relationships. However, Kaggle users have found new ways to incorporate unsupervised features that increase performance, seemingly by achieving domain reduction, which has appeared in other forms for very large architectures, such as chain of reasoning ([Wei 2023](https://arxiv.org/pdf/2201.11903.pdf)) or task appendages ([Raffel 2019](https://arxiv.org/pdf/1910.10683.pdf)). \n\n\n### 5.1 Information metatokens\n\nIn [Argumentative Writing Elements](https://www.kaggle.com/competitions/feedback-prize-2021) and [Predicting Effective Arguments](https://www.kaggle.com/c/feedback-prize-effectiveness), many Kaggle users performed unsupervised clustering and them thematized the clusters by utilizing term-frequency information as prompts that were appended to the input as tokens, which significantly improved their results. \n\n\n\n*Example prompts generated using agglomerative clustering and term frequency* \n\n\nOn the supervised side of things, category information as new or old tokens can provide disambiguation. In the [Clinical Patient Notes](https://www.kaggle.com/c/nbme-score-clinical-patient-notes) NER challenge, Kaggle user [Raja Biswas](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322832) created new tokens for different medical cases as similar symptoms had different significance depending on the patient prognosis. Similar to §2.1, [Raja Biswas](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322832) makes the model predict which clinical case is about to undergo NER and then performs the task with the special token prepended to the string.:\n>\nI created 10 new tokens for each case e.g. \"[QA CASE=0]\". I thought some of the feature texts from different cases are very close to each other e.g. Feature 314: nausea, Feature 508: Associated-nausea.\n\n\n\n#### Experiment 6: Distilling information from a linear classifier\n\nKaggle experiments show that transformer pre-classification improves performance on downstream tasks, but the deployment these pipelines is often not feasible in production. However, incorporating information from small models is reasonable for commercial applications. On these grounds, we pick up the News Category Dataset ([Misra 2023](https://arxiv.org/abs/2209.11429)) experiment from §2.2. We train a logistic regression model to predict newspaper section information (precision = 59%, n. classes = 27) and distill the prediction by verbalizing the class name and appending it as a prompt to the string. With a single weak classifier, the results improve by 0.9%:\n\n| Model | Accuracy |\n| --- | ----------- |\n| Baseline | 0.3076 | \n| Baseline + linear model | **0.3158** |\n\n\n\n### 5.2 Interclass correlations in annotator judgement can be fed to transformers \n\nIn [Patent Phrase to Phrase Matching](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching), some users found that adding all the targets related to an anchor improved the prediction accuracy of the model (code excerpts and MNLI experiment below). Kaggle user [Zzy](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332234) theorizes that *\"there exits strong correlations between different targets under the same anchor,\"* partly because they observed degraded performance using `GroupKFold` for anchors instead of `StratifiedGroupKFold` for anchors and targets. However, Kaggle user [Saber](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332420) reports contradictory findings, so it remains unclear if adding all targets as input allow the model to better learn the identity function of the similarity scores or if the extra inputs leaked latent relationships present in the production of the original dataset. The baseline vs. reformulated task looks as follows:\n\n\nSigmoid (anchor_x [SEP] target_x1 [SEP])\n\n–>\n\nSigmoid (anchor_x [SEP] targetx_1; target_x2; target_x3; ... traget_xn)\n\nInterestingly, randomizing the order of the auxiliary targets in each training iteration improved the results further. [Zzy](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332234) reports taking the method one step further, adding the out-of-fold cosine scores from a parent teacher as token input:\n\nSigmoid (anchor_x [SEP] targetx_1; target_x2; target_x3; ... traget_xn)\n \n–>\n\nSigmoid (anchor_x [SEP] target; target_x1 23; target_x2 47; traget_xn; 88)\n\n\n#### Experiment 7: Whether this method can model relationships in MNLI pairs\n \nWe test the generalizability of this approach using the MNLI dataset, which is the only widely used benchmark in this study that contains repeated anchor tuples. Following [Saber](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332420), we add the other tuples at the end and randomize their order. We note that the 2.5k downsampled dataset obtains much worse results and has very high variance across seeds while the 10k and 130k downsamples beat the baseline. This seems to indicate that the approach requires a larger amount of data for the model to learn the relationships. Due to compute limitations, we do not test this method on the whole dataset or with other transformer models. The results are as follows:\n \n \n| Model | MNLI 2.5k | MNLI 10k | MNLI 130k | Example |\n| --- | ----------- | ----------- | ----------- | ----------- |\n| Baseline | **0.8686** | 0.8982 | 0.9153 | Children smiling and waving at camera[SEP]The kids are frowning\n| Baseline + all tuples | 0.4574 | **0.9106** | **0.9229** | Children smiling and waving at camera[SEP]The kids are frowning[SEP]There are children present; They are smiling at their parents","metadata":{}},{"cell_type":"markdown","source":"## 6. Miscellaneous\nHere, we explore some miscellaneous tricks and ideas from Kaggle competitions to achieve better convergence with small datasets.\n\n![](\"https://i.imgur.com/uXXcu54.png\")
\n\n*Feedback Argumentation challenge, political bias vs. score. Categorical representations appear to indicate a relationship between right-leaning essays and higher scores, but a continous representation shows the magnitude of right-leaning bias is greater across the range.*\n\n\n### 6.1 Latent ordinal relationships can be captured by binary bins\n\nKaggle user [Qishen Ha](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/discussion/100961) reports that in the Jigsaw toxicity challenge, where binary judgements on toxicity were averaged out to obtain a score, conversion into binary bins improved their results. It is unclear if the success of this technique is each bin representing specific latent characteristics in the continuous variable or the fact it replicates the voting system used by the authors of the dataset:\n\n>>\nFor example, if we have origin label [0.33, 0.05, 1.0, 0.58, 0.0] , and args.bin = 9 [a hyper-parameter defining how many class to generate], we will get output like this:\n>>\n```array([[1, 1, 1, 0, 0, 0, 0, 0, 0], # 0.33\n [0, 0, 0, 0, 0, 0, 0, 0, 0], # 0.05\n [1, 1, 1, 1, 1, 1, 1, 1, 1], # 1.0\n [1, 1, 1, 1, 1, 0, 0, 0, 0], # 0.58\n [0, 0, 0, 0, 0, 0, 0, 0, 0]]) # 0.0\n```\n\n### 6.2 Categorical targets appear to be less prone to metric hacking \n\nIn Kaggle NLP competitions, categorical targets are more robust to metric hacking than continuous targets. See, for example, [[A]](https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/90334), [[B]](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/discussion/100530), [[C]](https://www.kaggle.com/competitions/google-quest-challenge/discussion/130112). In commercial environments, where practitioners actively avoid this form of overfitting, a model of indeterminate complexity may learn the identity function of these loss-minimizing shortcuts anyway. Data collection managers should be aware of this fact and the following:\n\n1. Continuous rubrics may result in granularity that replicate the biases of researchers, reducing the positive effects of crowd-sourcing in bias reduction and limiting the ability of the model to generalize ([Parmar 2023](https://arxiv.org/pdf/2205.00415.pdf)) \n2. The biases that emerge from categorical annotations are interpretable, and human production of categorical noise is well understood and can be mitigated. Numerical biases in continuous rubrics are less understood and harder to mitigate ([Mellers 1982](https://psycnet.apa.org/record/1983-02408-001), [Parduci 1986](https://pubmed.ncbi.nlm.nih.gov/2946806/), [Hutchinson 1997](https://psycnet.apa.org/doiLanding?doi=10.1037%2F0278-7393.23.3.591), [Tversky 1973](https://www.sciencedirect.com/science/article/abs/pii/0010028573900339), [Hutchinson 1991](https://psycnet.apa.org/record/1992-18084-001), [Kahneman 1994](https://psycnet.apa.org/record/1985-05780-001), [Comay 2023](https://www.sciencedirect.com/science/article/abs/pii/S0010027723000112?dgcid=coauthor))\n3. Categories appear to reduce annotator cognitive load and heuristics ([Hutchinson 1991](https://academic.oup.com/jcr/article-abstract/18/3/325/1812873))\n\n\n### 6.3 Some ensembling tricks from Kaggle competitions\nAlthough ensembling has limited commercial applications, these ideas can be used for small models served in tandem in production.\n\n1. Voting can be interpreted as a time series problem where the probability of consecutive models nudges the trend upward or downward (see [[1]](https://www.kaggle.com/competitions/jigsaw-multilingual-toxic-comment-classification/discussion/160986)). Similar to this and reminescent of the neural ensembles of the early 2010s ([Huang 2011](https://link.springer.com/article/10.1007/s13042-011-0019-y)) users have reported gains from feeding the last vector states of their models to a multilayer perceptron or RNN (see [[2]](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/discussion/103280), [[3]](https://www.kaggle.com/competitions/tweet-sentiment-extraction/discussion/159477)).\n2. Genetic algorithms have been used to optimize voting weights in several winning entries. The flexibility of the fitness function and their population diversity appear to have strong resilience against overfitting (see [[4]](https://www.kaggle.com/competitions/jigsaw-toxic-severity-rating/discussion/306274)).\n3. Hill climbing in combination with a grid-search-like algorithm has shown good results for correlated coefficients. Unlike Ridge or Lasso regression, the coefficients are not bounded by an L2 or L1 penalty, and unlike vainilla hill climbing, the grid search component makes it robust against local optima (see [[5]](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369609)).\n4. Weighed box fusion is an algorithm from visual classification that has been applied to NER (see [[6]](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313478), [[7]](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313389)). For another example of an object-identification algorithm from computer vision in NLP, please see [[8]](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/31).\n\n\n### 6.4 Unsupervised features for cross-validation splits\n\nUnsupervised clustering and statistical metrics can be used to improve stratified cross-validation and have greater control over error analysis. For example, Kaggle user [Mauro](https://www.kaggle.com/competitions/coleridgeinitiative-show-us-the-data/discussion/248360) used thematic information for cross-validation splits, and [Khoi Nguyen](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/232964) uses the Jaccard distance to assess model performance on unseen data. \n\n### 6.5 BPE and SentencePiece tokenizers can miss important features\n\nThe statistical mechanics on which tokenizers are built result in idiosyncrasies that may miss important features. Some examples are newline tokens, which can mark long-sequence dependencies (for example, [[9]](https://www.kaggle.com/competitions/feedback-prize-2021/discussion)), expressive punctuation like (!!!) or (???)[[10]](https://www.kaggle.com/competitions/tweet-sentiment-extraction/discussion/159505), and ellipses. The current empirical evidence appears to indicate these semantic relationships are not captured by transformers, failing also to produce meaningful representations of the typografical joke of emphasizing a word by spacing it out: \n```\n# G O J E T S\nG\nO\n\nJ\nE\nT\nS\n```\n\n```\n… stuff that people will find h u m o r o u s …\n```\n\nIn Kaggle, people create new tokens and change strings to ameliorate this problem, but the heavy use of hand-crafted rules indicates there are improvements to be made to unsupervised tokenizers. For a similar discussion and unsupervised solutions, please see [Zhang 2022](https://arxiv.org/pdf/2204.14268.pdf) and [Islam 2022](https://arxiv.org/ftp/arxiv/papers/2204/2204.10815.pdf).\n\n### 6.6 Multilingual models suffer from the faux amis problem\n\nOverlapping tokens between languages are seen as a feature of multilingual language training, but they suffer from the faux amis problem, or words that, *“in spite of similarities in form, have different meanings… , e.g. [Engish] figure vs [French] figure (‘face’)” or English constipated vs. Spanish constipado (“to have a cold”)\"* ([Bussmann 1996](https://www.routledge.com/Routledge-Dictionary-of-Language-and-Linguistics/Bussmann-Kazzazi-Trauth/p/book/9780415203197)). The dominant languages control the vector representations, and these errors were noted by participants in the [Learning Equality Multilingual](https://www.kaggle.com/competitions/learning-equality-curriculum-recommendations) competition. \n\n### 6.7 There is evidence for stochastic weight averaging\nStochastic weight averaging has been questioned due to the difficulty to produce and reproduce results, but several high-scoring Kaggle solutions report successful empirical results([[11]](https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/discussion/103280), [[12]](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258554), [[13]](https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/90392), [[14]](https://www.kaggle.com/competitions/tweet-sentiment-extraction/discussion/159310), [[15]](https://www.kaggle.com/code/rhtsingh/swa-apex-amp-interpreting-transformers-in-torch?scriptVersionId=67169276), [[16]](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/316071)).\n\n### 6.8 Intermediate layer representations\nAn empirical finding that appears to lend some credence to the theory of layer-wise semantic complexity ([Tenney 2019](https://arxiv.org/pdf/1905.05950.pdf)) is the fact that layer-wise decreasing learning rate, layer restart, and the use of lower layers appears in solutions for tasks that require primitive representations ([[17]](https://www.kaggle.com/c/12797/discussion/90336), [[18]](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/241029), [[19]](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322832), [[20]](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/332243)).\n","metadata":{}}]}