{ "id": 228227, "title": "A List Of Kaggle NLP Competitions & solutions", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/228227", "author_name": "", "post_date": "2021-03-23T21:44:06.291079300Z", "votes": 51, "comment_count": 2, "views": 0, "content": "

Hey all!

\n

Looks like Kaggle has another new competition that is related to NLP 😊 Here are a list of previous Kaggle competitions that were heavily NLP related:

\n

[Competition] https://www.kaggle.com/c/tweet-sentiment-extraction

\n\n

[Competition] https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification

\n\n

[Competition] https://www.kaggle.com/c/google-quest-challenge

\n\n

[Competition] https://www.kaggle.com/c/tensorflow2-question-answering

\n\n

For each competition I also took a few of the top solution posts (in no particular order). There is a lot of great content here when it comes to NLP. The Sentiment Extraction competition where the goal was to extract the sentiment and match it with the known label would probably be the closest to this competition. The metric was almost the same there as well and used Jaccard.

", "messages": [ { "id": "1250251", "postDate": "03/23/2021 21:44:06", "content": "

Hey all!

\n

Looks like Kaggle has another new competition that is related to NLP 😊 Here are a list of previous Kaggle competitions that were heavily NLP related:

\n

[Competition] https://www.kaggle.com/c/tweet-sentiment-extraction

\n\n

[Competition] https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification

\n\n

[Competition] https://www.kaggle.com/c/google-quest-challenge

\n\n

[Competition] https://www.kaggle.com/c/tensorflow2-question-answering

\n\n

For each competition I also took a few of the top solution posts (in no particular order). There is a lot of great content here when it comes to NLP. The Sentiment Extraction competition where the goal was to extract the sentiment and match it with the known label would probably be the closest to this competition. The metric was almost the same there as well and used Jaccard.

", "rawMarkdown": "Hey all!\n\nLooks like Kaggle has another new competition that is related to NLP 😊 Here are a list of previous Kaggle competitions that were heavily NLP related:\n\n[Competition] https://www.kaggle.com/c/tweet-sentiment-extraction\n\n- https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/159477\n- https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/159505\n- https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/159310\n- https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/159440\n\n[Competition] https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification\n- https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/176897\n- https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/160862\n- https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/161974\n\n[Competition] https://www.kaggle.com/c/google-quest-challenge\n- https://www.kaggle.com/c/google-quest-challenge/discussion/129978\n- https://www.kaggle.com/c/google-quest-challenge/discussion/129875\n- https://www.kaggle.com/c/google-quest-challenge/discussion/129840\n- https://www.kaggle.com/c/google-quest-challenge/discussion/129896\n\n[Competition] https://www.kaggle.com/c/tensorflow2-question-answering\n- https://www.kaggle.com/c/tensorflow2-question-answering/discussion/127551\n- https://www.kaggle.com/c/tensorflow2-question-answering/discussion/127339\n- https://www.kaggle.com/c/tensorflow2-question-answering/discussion/127259\n\nFor each competition I also took a few of the top solution posts (in no particular order). There is a lot of great content here when it comes to NLP. The Sentiment Extraction competition where the goal was to extract the sentiment and match it with the known label would probably be the closest to this competition. The metric was almost the same there as well and used Jaccard.", "votes": null }, { "id": "1250335", "postDate": "03/24/2021 01:04:36", "content": "

Useful collection, thank you!
\nAnd I collect only prize competitions and their prize solutions. I have 1 more competition than you - see notebook Data Science with DL & NLP: Advanced Techniques

", "rawMarkdown": "Useful collection, thank you!\nAnd I collect only prize competitions and their prize solutions. I have 1 more competition than you - see notebook [Data Science with DL & NLP: Advanced Techniques](https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques)", "votes": null }, { "id": "1250349", "postDate": "03/24/2021 01:23:49", "content": "

These are my semi-structured notes from speed reading through all the past NLP competitions and associated solutions notebooks

\n

Google Quest Challenge

\n

https://www.kaggle.com/c/google-quest-challenge

\n\n

Jigsaw Toxic Comment Classification Challenge

\n

https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge

\n\n

Jigsaw Unintended Bias in Toxicity Classification

\n

https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification

\n\n

NLP Getting Started

\n

https://www.kaggle.com/c/nlp-getting-started

\n\n

Quora Insincere Questions Classification

\n

https://www.kaggle.com/c/quora-insincere-questions-classification

\n\n

Random Acts of Pizza

\n

https://www.kaggle.com/c/random-acts-of-pizza

\n\n

https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews

\n\n

Other NLP Competitions

\n

https://www.kaggle.com/c/stumbleupon
\nhttps://www.kaggle.com/c/tradeshift-text-classification
\nhttps://www.kaggle.com/c/whats-cooking
\nhttps://www.kaggle.com/c/word2vec-nlp-tutorial

", "rawMarkdown": "These are my semi-structured notes from speed reading through all the past NLP competitions and associated solutions notebooks \n\n## Google Quest Challenge\n\nhttps://www.kaggle.com/c/google-quest-challenge\n- 1st place - https://www.kaggle.com/c/google-quest-challenge/discussion/129978\n - pretrained language models\n - pseudo-labeling\n - https://www.analyticsvidhya.com/blog/2017/09/pseudo-labelling-semi-supervised-learning-technique/\n - post-processing (yes we did one)\n\n- 2nd place - https://www.kaggle.com/c/google-quest-challenge/discussion/129840\n - Our final blend consists of 5 transformer models, that have several differences to push diversity. \n\n- 5th place - https://www.kaggle.com/c/google-quest-challenge/discussion/129875\n - https://github.com/cxy229/BDCI2019-SENTIMENT-CLASSIFICATION\n\n- 77th place - https://www.kaggle.com/c/google-quest-challenge/discussion/130065\n - DBSCAN - https://en.wikipedia.org/wiki/DBSCAN \n\n## Jigsaw Toxic Comment Classification Challenge\n\nhttps://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge\n- 1st place - https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557\n - 1. Diverse pre-trained embeddings (baseline public LB of 0.9877)\n - https://paperswithcode.com/method/bigru \n - 2. Translations as train/test-time augmentation (TTA) (boosted LB from 0.9877 to 0.9880)\n- 3. Rough-bore pseudo-labelling (PL) (boosted LB from 0.9880 to 0.9885)\n - We tried a number of PL variants - canonical per-batch updates, altering the loss functions etc.The variant that performed the best was simply labeling the test samples using our best-performing ensemble, adding them to the train set, and training to convergence. There’s been a fair amount of hay made on the forum about the difference in train and test distributions. PL helps with that.\n- 4. Robust CV + stacking framework (boosted LB from 0.9885 to 0.9890)\n - 1. For stacking, we used a weighted mean of arithmetic averaging and stacking, which worked marginally better (~.0001) than either approach alone. For stacking, we used primarily LightGBM, which both was faster than XGBoost and reached slightly better CV scores with heavy bayesian optimization.\n - 2. Parameters were selected by choosing the best out of 250 runs with bayesian optimization; key points in the parameters were small trees with low depth and strong l1 regularization. We bagged 6 runs of both DART and GBDT using different seeds to account for potential variance during stacking.\n - 3. For CV, we tracked accuracy, log loss and AUC. A model was “strong” if when added to our stack it improved CV-log loss and CV-AUC in addition to improving the public board. We discarded a lot of models from our stack+blend for failing to do any of the three above in fear of overfitting.\n\n- 3rd Place - https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52762\n - Blended Model \n - preprocessing\n - feature engineering, text normalizations, and embeddings\n - LogisticRegression, GRUs, LSTMs, etc.\n - A two-bidirectional-GRU-layer neural net, a two-bidirectional-LSTM-layer neural net, and an XGBoost model\n - Attention model. This model used weighted attention layer from DeepMoji project. It calculates weighted attention on input word sequence and also calculated it on input sequence after a few convolutions.\n - Tfidf model\n\t- Model based on their own trained fasttext vectors on wikipedia comment corpus.\n\t- Vowpal Wabbit model (https://en.wikipedia.org/wiki/Vowpal_Wabbit)\n\t\n- 27th Place - https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52719\n - preprocessing\n - original dataset with spellings correction: by comparing the Levenshtein distance and a lot of regular expressions.\n - original dataset with pos taggings: We generate the part of speech (POS) tagging for every comment by TextBlob and concatenate the word embedding and POS embedding as a single one. Since TextBlob drops some tokens and punctuations when generating the POS sequences, that gives our models another view.\n - Riad's dataset: with very heavily data-cleaning, spelling correction and translation\n - Pooled RNN (public: 0.9862, private: 0.9858)\n - Kmax text CNN (public: 0.9856 , private: 0.9849)\n - As many competitors pointed out, dropout and batch-normalization are the keys to prevent overfitting. By applying the dropout on the word embedding directly and behind the pooling does great regularization both on train set and test set. Although model with many dropouts takes about 5 more epochs to coverage, it boosts our scores significantly. For instance, my RNN boosts from 0.9853 (private: 0.9850) to 0.9862 (private: 0.9858) after adding dropout layers.\n - As for the pre-trained embedding, we used:\n - Fasttext: crawl-300d-2M.vec\n - GloVe: glove.840B.300d.txt\n - GloVe: glove.twitter.27B.200d.txt\n - We simply select some best OOFs from each type of models (RNN, CNN, LightGBM, NBSVM) and ensemble them by LightGBM. Most of the meta-features are the same as olivier's awesome kernel. The only difference is we add #typos, #oov_in_fasttext, #oov_in_glove and the sentiment scores trained by twitter comments for showing the degree of positiveness, which gives a little boost to my local AUC from 0.9920 to 0.9928.\n - https://github.com/zake7749/DeepToxic\n\n## Jigsaw Unintended Bias in Toxicity Classification\n\nhttps://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification\n- 1st place - https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/103280\n - Custom head for BERT, XLNet and GPT2\n - Bucket Sequencing Collator\n - Auxiliary tasks for models\n - Custom mimic loss\n - SWA and checkpoint ensemble\n - Rank average ensemble of 2x XLNet, 2x BERT and GPT2 medium\n\n- 9th Place - https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/100530\n - Custom loss function\n - Embeddings\n - Bi-LSTM: .941 (5 fold-cv) v.s. .939,\n - BERT-Small-Uncased: .943 (1 fold-cv) v.s. .942,\n - Bert-Large-Uncased with WWM: .9442 (1 fold-cv) v.s. .9440\n - XLNET: .9437 (1 fold-cv) v.s. .9431\n - Stacking\n - ExtraTreesClassifier\n\n## NLP Getting Started\nhttps://www.kaggle.com/c/nlp-getting-started\n- Glove + BERT - https://www.kaggle.com/salmanhiro/glove-baseline-bert\n- HuggingFace TFBertModel - https://www.kaggle.com/dhruv1234/huggingface-tfbertmodel\n- NLP Naive Bayes - https://www.kaggle.com/jamesmcguigan/nlp-naive-bayes \n- NLP TF-IDF Classifier - https://www.kaggle.com/jamesmcguigan/nlp-tf-idf-classifier \n\n\n## Quora Insincere Questions Classification\nhttps://www.kaggle.com/c/quora-insincere-questions-classification\n- 1st Place - https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/80568\n - Similar to: https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/79824\n - It’s basically a Single Bi-LSTM 128 followed by a Conv1D with kernel size 1 only and GlobalMaxPooling afterwards plus additional dropout layers with minimal dropout. We additionally use a few statistical features.\n - What did not work for us\n - Different optimizers (focal loss was similar though)\n - Label smoothing\n - Auxiliary learning / multitask learning\n - Snapshot learning\n - Pseudo labeling\n - Fitting own embeddings with gensim\n - Spelling correction\n - Taking median/percentile of predicitons instead of average\n - More complex layers and architectures (Attention, QRNN, Capsule, larger/multiple LSTM layers, larger CNN kernel sizes, LGBM or bag of words)\n - Word collocations - Several words put together can bear a completely new meaning, which is not captured by embeddings. Glove turned out to have quite a lot of such collocations with words put together using \"-\" sign. So we replaced examples like \"ethnical cleansing\" with \"ethnical-cleansing\", which is then captured by a more appropriate glove embedding. It showed no improvement on CV.\n - Extra statistical features - Presence of statistical features added a little bit to the accuracy based on CV, but we saw no improvement with other extra features, like sentiment or bag-of-words based variables.\n - Replacement of words with synonyms - An idea of replacing all nationalities (or e.g. political party) with the same word did not work at all.\n - Order the train data by the length of the sentences - This approach gave a dramatic improvement in the fitting time because each batch contained only sentences with similar sizes, but it hurt the accuracy of the model too much.\n \n- 15th Solution - https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/80540\n - Ensemble\n - the first model is RCNN.\n - the second model is LSTM(128) + GRU(96) + maxpooling1D + dropout(0.1).\n - the third model is LSTM(128) + GRU(64) + Conv1D + maxpooling_concatenate.\n - the fourth model is LSTM(128) + GRU(64) + Conv1D + Attention.\n - we used the word vector concatenated by glove and fasttext.\n - we set max_features = None and we set max_len = 57\n \n- https://www.kaggle.com/ishitori/what-if-we-could-finetune-bert-from-apache-mxnet \n - Fine-tune BERT Pretrained model using Apache MXNet and GluonNLP\n - https://mxnet.incubator.apache.org/versions/1.8.0/\n - https://nlp.gluon.ai/\n\n\n## Random Acts of Pizza\n\nhttps://www.kaggle.com/c/random-acts-of-pizza \n- https://www.kaggle.com/alvations/basic-nlp-with-nltk\n- https://www.kaggle.com/jatinraina/random-acts-of-pizza-xgboost\n\nhttps://www.kaggle.com/c/sentiment-analysis-on-movie-reviews\n- Fastai with HuggingFace Transformers (BERT, RoBERTa, XLNet, XLM, DistilBERT)¶\n - https://www.kaggle.com/maroberti/fastai-with-transformers-bert-roberta\n- Sentiment Prediction With NN and Naive bayes\n - https://www.kaggle.com/muhammadayman/sentiment-prediction-with-nn-and-naive-bayes\n- Natural language processing with Bi-GRU [top 10%]\n - https://www.kaggle.com/michalbrezk/natural-language-processing-with-bi-gru-top-10\n- DistilBert\n - https://www.kaggle.com/ankitkumarsaini/distilbert\n- Optimal Probability Thresholds Using PR Curve\n - https://www.kaggle.com/nicholasgah/optimal-probability-thresholds-using-pr-curve\t \t\n - https://www.kaggle.com/nicholasgah/obtain-optimal-probability-threshold-using-roc\n - https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-imbalanced-classification/\n\n\n## Other NLP Competitions\n\nhttps://www.kaggle.com/c/stumbleupon\nhttps://www.kaggle.com/c/tradeshift-text-classification\nhttps://www.kaggle.com/c/whats-cooking\nhttps://www.kaggle.com/c/word2vec-nlp-tutorial", "votes": null } ], "comments": [ { "id": 1250335, "author_name": "vbmokin", "author_url": "", "post_date": "03/24/2021 01:04:36", "content": "

Useful collection, thank you!
\nAnd I collect only prize competitions and their prize solutions. I have 1 more competition than you - see notebook Data Science with DL & NLP: Advanced Techniques

", "votes": null, "replies": [] }, { "id": 1250349, "author_name": "jamesmcguigan", "author_url": "", "post_date": "03/24/2021 01:23:49", "content": "

These are my semi-structured notes from speed reading through all the past NLP competitions and associated solutions notebooks

\n

Google Quest Challenge

\n

https://www.kaggle.com/c/google-quest-challenge

\n\n

Jigsaw Toxic Comment Classification Challenge

\n

https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge

\n\n

Jigsaw Unintended Bias in Toxicity Classification

\n

https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification

\n\n

NLP Getting Started

\n

https://www.kaggle.com/c/nlp-getting-started

\n\n

Quora Insincere Questions Classification

\n

https://www.kaggle.com/c/quora-insincere-questions-classification

\n\n

Random Acts of Pizza

\n

https://www.kaggle.com/c/random-acts-of-pizza

\n\n

https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews

\n\n

Other NLP Competitions

\n

https://www.kaggle.com/c/stumbleupon
\nhttps://www.kaggle.com/c/tradeshift-text-classification
\nhttps://www.kaggle.com/c/whats-cooking
\nhttps://www.kaggle.com/c/word2vec-nlp-tutorial

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1250251": "Hey all!\n\nLooks like Kaggle has another new competition that is related to NLP 😊 Here are a list of previous Kaggle competitions that were heavily NLP related:\n\n[Competition] https://www.kaggle.com/c/tweet-sentiment-extraction\n\n- https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/159477\n- https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/159505\n- https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/159310\n- https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/159440\n\n[Competition] https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification\n- https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/176897\n- https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/160862\n- https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/161974\n\n[Competition] https://www.kaggle.com/c/google-quest-challenge\n- https://www.kaggle.com/c/google-quest-challenge/discussion/129978\n- https://www.kaggle.com/c/google-quest-challenge/discussion/129875\n- https://www.kaggle.com/c/google-quest-challenge/discussion/129840\n- https://www.kaggle.com/c/google-quest-challenge/discussion/129896\n\n[Competition] https://www.kaggle.com/c/tensorflow2-question-answering\n- https://www.kaggle.com/c/tensorflow2-question-answering/discussion/127551\n- https://www.kaggle.com/c/tensorflow2-question-answering/discussion/127339\n- https://www.kaggle.com/c/tensorflow2-question-answering/discussion/127259\n\nFor each competition I also took a few of the top solution posts (in no particular order). There is a lot of great content here when it comes to NLP. The Sentiment Extraction competition where the goal was to extract the sentiment and match it with the known label would probably be the closest to this competition. The metric was almost the same there as well and used Jaccard.", "1250335": "Useful collection, thank you!\nAnd I collect only prize competitions and their prize solutions. I have 1 more competition than you - see notebook [Data Science with DL & NLP: Advanced Techniques](https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques)", "1250349": "These are my semi-structured notes from speed reading through all the past NLP competitions and associated solutions notebooks \n\n## Google Quest Challenge\n\nhttps://www.kaggle.com/c/google-quest-challenge\n- 1st place - https://www.kaggle.com/c/google-quest-challenge/discussion/129978\n - pretrained language models\n - pseudo-labeling\n - https://www.analyticsvidhya.com/blog/2017/09/pseudo-labelling-semi-supervised-learning-technique/\n - post-processing (yes we did one)\n\n- 2nd place - https://www.kaggle.com/c/google-quest-challenge/discussion/129840\n - Our final blend consists of 5 transformer models, that have several differences to push diversity. \n\n- 5th place - https://www.kaggle.com/c/google-quest-challenge/discussion/129875\n - https://github.com/cxy229/BDCI2019-SENTIMENT-CLASSIFICATION\n\n- 77th place - https://www.kaggle.com/c/google-quest-challenge/discussion/130065\n - DBSCAN - https://en.wikipedia.org/wiki/DBSCAN \n\n## Jigsaw Toxic Comment Classification Challenge\n\nhttps://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge\n- 1st place - https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557\n - 1. Diverse pre-trained embeddings (baseline public LB of 0.9877)\n - https://paperswithcode.com/method/bigru \n - 2. Translations as train/test-time augmentation (TTA) (boosted LB from 0.9877 to 0.9880)\n- 3. Rough-bore pseudo-labelling (PL) (boosted LB from 0.9880 to 0.9885)\n - We tried a number of PL variants - canonical per-batch updates, altering the loss functions etc.The variant that performed the best was simply labeling the test samples using our best-performing ensemble, adding them to the train set, and training to convergence. There’s been a fair amount of hay made on the forum about the difference in train and test distributions. PL helps with that.\n- 4. Robust CV + stacking framework (boosted LB from 0.9885 to 0.9890)\n - 1. For stacking, we used a weighted mean of arithmetic averaging and stacking, which worked marginally better (~.0001) than either approach alone. For stacking, we used primarily LightGBM, which both was faster than XGBoost and reached slightly better CV scores with heavy bayesian optimization.\n - 2. Parameters were selected by choosing the best out of 250 runs with bayesian optimization; key points in the parameters were small trees with low depth and strong l1 regularization. We bagged 6 runs of both DART and GBDT using different seeds to account for potential variance during stacking.\n - 3. For CV, we tracked accuracy, log loss and AUC. A model was “strong” if when added to our stack it improved CV-log loss and CV-AUC in addition to improving the public board. We discarded a lot of models from our stack+blend for failing to do any of the three above in fear of overfitting.\n\n- 3rd Place - https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52762\n - Blended Model \n - preprocessing\n - feature engineering, text normalizations, and embeddings\n - LogisticRegression, GRUs, LSTMs, etc.\n - A two-bidirectional-GRU-layer neural net, a two-bidirectional-LSTM-layer neural net, and an XGBoost model\n - Attention model. This model used weighted attention layer from DeepMoji project. It calculates weighted attention on input word sequence and also calculated it on input sequence after a few convolutions.\n - Tfidf model\n\t- Model based on their own trained fasttext vectors on wikipedia comment corpus.\n\t- Vowpal Wabbit model (https://en.wikipedia.org/wiki/Vowpal_Wabbit)\n\t\n- 27th Place - https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52719\n - preprocessing\n - original dataset with spellings correction: by comparing the Levenshtein distance and a lot of regular expressions.\n - original dataset with pos taggings: We generate the part of speech (POS) tagging for every comment by TextBlob and concatenate the word embedding and POS embedding as a single one. Since TextBlob drops some tokens and punctuations when generating the POS sequences, that gives our models another view.\n - Riad's dataset: with very heavily data-cleaning, spelling correction and translation\n - Pooled RNN (public: 0.9862, private: 0.9858)\n - Kmax text CNN (public: 0.9856 , private: 0.9849)\n - As many competitors pointed out, dropout and batch-normalization are the keys to prevent overfitting. By applying the dropout on the word embedding directly and behind the pooling does great regularization both on train set and test set. Although model with many dropouts takes about 5 more epochs to coverage, it boosts our scores significantly. For instance, my RNN boosts from 0.9853 (private: 0.9850) to 0.9862 (private: 0.9858) after adding dropout layers.\n - As for the pre-trained embedding, we used:\n - Fasttext: crawl-300d-2M.vec\n - GloVe: glove.840B.300d.txt\n - GloVe: glove.twitter.27B.200d.txt\n - We simply select some best OOFs from each type of models (RNN, CNN, LightGBM, NBSVM) and ensemble them by LightGBM. Most of the meta-features are the same as olivier's awesome kernel. The only difference is we add #typos, #oov_in_fasttext, #oov_in_glove and the sentiment scores trained by twitter comments for showing the degree of positiveness, which gives a little boost to my local AUC from 0.9920 to 0.9928.\n - https://github.com/zake7749/DeepToxic\n\n## Jigsaw Unintended Bias in Toxicity Classification\n\nhttps://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification\n- 1st place - https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/103280\n - Custom head for BERT, XLNet and GPT2\n - Bucket Sequencing Collator\n - Auxiliary tasks for models\n - Custom mimic loss\n - SWA and checkpoint ensemble\n - Rank average ensemble of 2x XLNet, 2x BERT and GPT2 medium\n\n- 9th Place - https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/100530\n - Custom loss function\n - Embeddings\n - Bi-LSTM: .941 (5 fold-cv) v.s. .939,\n - BERT-Small-Uncased: .943 (1 fold-cv) v.s. .942,\n - Bert-Large-Uncased with WWM: .9442 (1 fold-cv) v.s. .9440\n - XLNET: .9437 (1 fold-cv) v.s. .9431\n - Stacking\n - ExtraTreesClassifier\n\n## NLP Getting Started\nhttps://www.kaggle.com/c/nlp-getting-started\n- Glove + BERT - https://www.kaggle.com/salmanhiro/glove-baseline-bert\n- HuggingFace TFBertModel - https://www.kaggle.com/dhruv1234/huggingface-tfbertmodel\n- NLP Naive Bayes - https://www.kaggle.com/jamesmcguigan/nlp-naive-bayes \n- NLP TF-IDF Classifier - https://www.kaggle.com/jamesmcguigan/nlp-tf-idf-classifier \n\n\n## Quora Insincere Questions Classification\nhttps://www.kaggle.com/c/quora-insincere-questions-classification\n- 1st Place - https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/80568\n - Similar to: https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/79824\n - It’s basically a Single Bi-LSTM 128 followed by a Conv1D with kernel size 1 only and GlobalMaxPooling afterwards plus additional dropout layers with minimal dropout. We additionally use a few statistical features.\n - What did not work for us\n - Different optimizers (focal loss was similar though)\n - Label smoothing\n - Auxiliary learning / multitask learning\n - Snapshot learning\n - Pseudo labeling\n - Fitting own embeddings with gensim\n - Spelling correction\n - Taking median/percentile of predicitons instead of average\n - More complex layers and architectures (Attention, QRNN, Capsule, larger/multiple LSTM layers, larger CNN kernel sizes, LGBM or bag of words)\n - Word collocations - Several words put together can bear a completely new meaning, which is not captured by embeddings. Glove turned out to have quite a lot of such collocations with words put together using \"-\" sign. So we replaced examples like \"ethnical cleansing\" with \"ethnical-cleansing\", which is then captured by a more appropriate glove embedding. It showed no improvement on CV.\n - Extra statistical features - Presence of statistical features added a little bit to the accuracy based on CV, but we saw no improvement with other extra features, like sentiment or bag-of-words based variables.\n - Replacement of words with synonyms - An idea of replacing all nationalities (or e.g. political party) with the same word did not work at all.\n - Order the train data by the length of the sentences - This approach gave a dramatic improvement in the fitting time because each batch contained only sentences with similar sizes, but it hurt the accuracy of the model too much.\n \n- 15th Solution - https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/80540\n - Ensemble\n - the first model is RCNN.\n - the second model is LSTM(128) + GRU(96) + maxpooling1D + dropout(0.1).\n - the third model is LSTM(128) + GRU(64) + Conv1D + maxpooling_concatenate.\n - the fourth model is LSTM(128) + GRU(64) + Conv1D + Attention.\n - we used the word vector concatenated by glove and fasttext.\n - we set max_features = None and we set max_len = 57\n \n- https://www.kaggle.com/ishitori/what-if-we-could-finetune-bert-from-apache-mxnet \n - Fine-tune BERT Pretrained model using Apache MXNet and GluonNLP\n - https://mxnet.incubator.apache.org/versions/1.8.0/\n - https://nlp.gluon.ai/\n\n\n## Random Acts of Pizza\n\nhttps://www.kaggle.com/c/random-acts-of-pizza \n- https://www.kaggle.com/alvations/basic-nlp-with-nltk\n- https://www.kaggle.com/jatinraina/random-acts-of-pizza-xgboost\n\nhttps://www.kaggle.com/c/sentiment-analysis-on-movie-reviews\n- Fastai with HuggingFace Transformers (BERT, RoBERTa, XLNet, XLM, DistilBERT)¶\n - https://www.kaggle.com/maroberti/fastai-with-transformers-bert-roberta\n- Sentiment Prediction With NN and Naive bayes\n - https://www.kaggle.com/muhammadayman/sentiment-prediction-with-nn-and-naive-bayes\n- Natural language processing with Bi-GRU [top 10%]\n - https://www.kaggle.com/michalbrezk/natural-language-processing-with-bi-gru-top-10\n- DistilBert\n - https://www.kaggle.com/ankitkumarsaini/distilbert\n- Optimal Probability Thresholds Using PR Curve\n - https://www.kaggle.com/nicholasgah/optimal-probability-thresholds-using-pr-curve\t \t\n - https://www.kaggle.com/nicholasgah/obtain-optimal-probability-threshold-using-roc\n - https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-imbalanced-classification/\n\n\n## Other NLP Competitions\n\nhttps://www.kaggle.com/c/stumbleupon\nhttps://www.kaggle.com/c/tradeshift-text-classification\nhttps://www.kaggle.com/c/whats-cooking\nhttps://www.kaggle.com/c/word2vec-nlp-tutorial" }, "source": "meta" }