Semi-supervised and transfer learning in NLP is a very hot topic. And I wish we had a \"big\" Kaggle competition like this one (\"Clickbait news detection\"). And more time of course, 1 week is nothing. I decided to spend only ~ 5 hours on this competition (already had ULMFiT and logit-tf-idf pipelines built for other tasks) but would've spent much more if we had at least 6 weeks (got almost the same task at my work). These are the main ideas that came to my mind (not only about SSL but merely concerning the current competition as well):
\n\nBERT fine-tuning. In the recent gendered pronoun resolution competition by Google AI (so far imho the best competition ever) my team missed gold, mostly due to the fact that we didn't manage to fine-tune BERT for classification (quickly overfitted). Still on my way to understand how this shall be done in a simple setting like we have in this clickbait detection task.
Pseudo-labeling. Typically you train a model, make predictions for the unlabeled set and then use these predictions as new labels. Kaggle hacks include leaving only some 10% most confident predictions and performing 3-5 iterations of pseudo-labeling. I enjoy this discussion in Santander where @ddanevskyi (Dmitriy Danevskiy) encouraged all the crowds to try smth new instead of this ruddy stacking. My idea for this competition was to train several different models, make predictions for the unlabeled set, and then select only those instances for which predictions are both confident and coherent across different models.
LSTMs/GRUs with embeddings. The datasets are small and encourage the use of semi-supervised and transfer learning techniques (which is very cool). However, the data is not as scarce as to abandon classic GRUs and LSTMs with embeddings. Here you can just reuse some pipelines shared in Jigsaw and Quora competitions. Like this Kernel by @jhoward (Jeremy Howard) in Jigsaw or this baseline by @sudalairajkumar (SRK). If you use embeddings, then learn how to make your preprocessing suit the vocabulary better - there's a very important Kernel by @christofhenkel (Dieter).
Data augmentation. Data augmentation for NLP is still a hot research topic. So far doesn't work as well as for computer vision. However, some approaches sometimes work. Among really beautiful ones is backtranslation, I first saw it shared by @pavelost (Pavel Ostyakov) in the toxic comments competition. The idea is that you augment the dataset by translating texts to another language (e.g. French, German) and then back to English.
Stacking/blending. Well... I don't enjoy spending time on this shit - much better to learn smth new (like covering basics of machine translation). But this technique is efficient. Typically you can just create OOF predictions with as many models as you can handle, and then train 2-3 simple models (logit, lgb, shallow RF) on top of that. You get a lot of models making a cartesian product of several models, random seeds, feature sets (in this case titles only, texts only, titles + texst etc), hyperparam settings etc.
Encourage you to share more ideas on efficient SSL for NLP. And hope we'll have a separate competition for that. At least I've got a funny dataset with BTC news sentiment (~3k labeled, 50k unlabeled), will share it if allowed.
", "messages": [ { "id": "524264", "postDate": "04/28/2019 10:52:16", "content": "Semi-supervised and transfer learning in NLP is a very hot topic. And I wish we had a \"big\" Kaggle competition like this one (\"Clickbait news detection\"). And more time of course, 1 week is nothing. I decided to spend only ~ 5 hours on this competition (already had ULMFiT and logit-tf-idf pipelines built for other tasks) but would've spent much more if we had at least 6 weeks (got almost the same task at my work). These are the main ideas that came to my mind (not only about SSL but merely concerning the current competition as well):
\n\nBERT fine-tuning. In the recent gendered pronoun resolution competition by Google AI (so far imho the best competition ever) my team missed gold, mostly due to the fact that we didn't manage to fine-tune BERT for classification (quickly overfitted). Still on my way to understand how this shall be done in a simple setting like we have in this clickbait detection task.
Pseudo-labeling. Typically you train a model, make predictions for the unlabeled set and then use these predictions as new labels. Kaggle hacks include leaving only some 10% most confident predictions and performing 3-5 iterations of pseudo-labeling. I enjoy this discussion in Santander where @ddanevskyi (Dmitriy Danevskiy) encouraged all the crowds to try smth new instead of this ruddy stacking. My idea for this competition was to train several different models, make predictions for the unlabeled set, and then select only those instances for which predictions are both confident and coherent across different models.
LSTMs/GRUs with embeddings. The datasets are small and encourage the use of semi-supervised and transfer learning techniques (which is very cool). However, the data is not as scarce as to abandon classic GRUs and LSTMs with embeddings. Here you can just reuse some pipelines shared in Jigsaw and Quora competitions. Like this Kernel by @jhoward (Jeremy Howard) in Jigsaw or this baseline by @sudalairajkumar (SRK). If you use embeddings, then learn how to make your preprocessing suit the vocabulary better - there's a very important Kernel by @christofhenkel (Dieter).
Data augmentation. Data augmentation for NLP is still a hot research topic. So far doesn't work as well as for computer vision. However, some approaches sometimes work. Among really beautiful ones is backtranslation, I first saw it shared by @pavelost (Pavel Ostyakov) in the toxic comments competition. The idea is that you augment the dataset by translating texts to another language (e.g. French, German) and then back to English.
Stacking/blending. Well... I don't enjoy spending time on this shit - much better to learn smth new (like covering basics of machine translation). But this technique is efficient. Typically you can just create OOF predictions with as many models as you can handle, and then train 2-3 simple models (logit, lgb, shallow RF) on top of that. You get a lot of models making a cartesian product of several models, random seeds, feature sets (in this case titles only, texts only, titles + texst etc), hyperparam settings etc.
Encourage you to share more ideas on efficient SSL for NLP. And hope we'll have a separate competition for that. At least I've got a funny dataset with BTC news sentiment (~3k labeled, 50k unlabeled), will share it if allowed.
", "rawMarkdown": "Semi-supervised and transfer learning in NLP is a very hot topic. And I wish we had a \"big\" Kaggle competition like this one ([\"Clickbait news detection\"](https://www.kaggle.com/c/dlinnlp-spring-2019-clf)). And more time of course, 1 week is nothing. I decided to spend only ~ 5 hours on this competition (already had ULMFiT and logit-tf-idf pipelines built for other tasks) but would've spent much more if we had at least 6 weeks (got almost the same task at my work). These are the main ideas that came to my mind (not only about SSL but merely concerning the current competition as well):\n\n1. **BERT fine-tuning.** In the recent gendered pronoun resolution [competition](https://www.kaggle.com/c/gendered-pronoun-resolution) by Google AI (so far imho the best competition ever) my team missed gold, mostly due to the fact that we didn't manage to fine-tune BERT for classification (quickly overfitted). Still on my way to understand how this shall be done in a simple setting like we have in this clickbait detection task.\n\n2. **Pseudo-labeling.** Typically you train a model, make predictions for the unlabeled set and then use these predictions as new labels. Kaggle hacks include leaving only some 10% most confident predictions and performing 3-5 iterations of pseudo-labeling. I enjoy [this discussion](https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/81012) in Santander where @ddanevskyi (Dmitriy Danevskiy) encouraged all the crowds to try smth new instead of this ruddy stacking. My idea for this competition was to train several different models, make predictions for the unlabeled set, and then select only those instances for which predictions are both confident and coherent across different models. \n\n3. **LSTMs/GRUs with embeddings**. The datasets are small and encourage the use of semi-supervised and transfer learning techniques (which is very cool). However, the data is not as scarce as to abandon classic GRUs and LSTMs with embeddings. Here you can just reuse some pipelines shared in Jigsaw and Quora competitions. Like [this Kernel](https://www.kaggle.com/jhoward/improved-lstm-baseline-glove-dropout) by @jhoward (Jeremy Howard) in Jigsaw or [this baseline](https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings) by @sudalairajkumar (SRK). If you use embeddings, then learn how to make your preprocessing suit the vocabulary better - there's a very important [Kernel](https://www.kaggle.com/christofhenkel/how-to-preprocessing-when-using-embeddings) by @christofhenkel (Dieter). \n\n4. **Data augmentation**. Data augmentation for NLP is still a hot research topic. So far doesn't work as well as for computer vision. However, some approaches *sometimes* work. Among really beautiful ones is *backtranslation*, I first saw it [shared](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038#latest-310689) by @pavelost (Pavel Ostyakov) in the toxic comments competition. The idea is that you augment the dataset by translating texts to another language (e.g. French, German) and then back to English. \n\n5. **Stacking/blending.** Well... I don't enjoy spending time on this shit - much better to learn smth new (like covering basics of machine translation). But this technique is efficient. Typically you can just create OOF predictions with as many models as you can handle, and then train 2-3 simple models (logit, lgb, shallow RF) on top of that. You get a lot of models making a cartesian product of several models, random seeds, feature sets (in this case titles only, texts only, titles + texst etc), hyperparam settings etc. \n\nEncourage you to share more ideas on efficient SSL for NLP. And hope we'll have a separate competition for that. At least I've got a funny dataset with BTC news sentiment (~3k labeled, 50k unlabeled), will share it if allowed.", "votes": null }, { "id": "524839", "postDate": "04/29/2019 16:09:17", "content": "Thanks for an informative summary. I spent the last week reading all the research about NLP from AWD_LSTM, ELMO, GPT to BERT. After reading all the theory, I was thinking of finding resources to practice, which I luckily found in your discussion. So thanks for that.
\n\nI have one question. Have you tried using GPT-2 for any of your tasks and seeing how does it compare to BERT (although I think seeing that only small GPT-2 with one-tenth of params was released, it would not work much better than BERT).
", "rawMarkdown": "Thanks for an informative summary. I spent the last week reading all the research about NLP from AWD_LSTM, ELMO, GPT to BERT. After reading all the theory, I was thinking of finding resources to practice, which I luckily found in your discussion. So thanks for that.\n\nI have one question. Have you tried using GPT-2 for any of your tasks and seeing how does it compare to BERT (although I think seeing that only small GPT-2 with one-tenth of params was released, it would not work much better than BERT).", "votes": null }, { "id": "524971", "postDate": "04/29/2019 21:44:27", "content": "No, haven't yet tried GPT-2. Though it's indeed interesting to compare with BERT.
", "rawMarkdown": "No, haven't yet tried GPT-2. Though it's indeed interesting to compare with BERT.", "votes": null }, { "id": "1266670", "postDate": "04/08/2021 01:28:46", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "3523107", "postDate": "09/10/2026 15:48:15", "content": "A fascinating NLP question: how much labeled text does a model really need? 🔎
\nhttps://mlguidance.blogspot.com/2026/09/what-are-models-of-semi-supervised.html
", "rawMarkdown": "A fascinating NLP question: **how much labeled text does a model really need?** 🔎\n\nhttps://mlguidance.blogspot.com/2026/09/what-are-models-of-semi-supervised.html", "votes": null } ], "comments": [ { "id": 1266670, "author_name": "daikinban", "author_url": "", "post_date": "04/08/2021 01:28:46", "content": "Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 3523107, "author_name": "kaustubh994", "author_url": "", "post_date": "09/10/2026 15:48:15", "content": "A fascinating NLP question: how much labeled text does a model really need? 🔎
\nhttps://mlguidance.blogspot.com/2026/09/what-are-models-of-semi-supervised.html
", "votes": null, "replies": [] }, { "id": 524839, "author_name": "kushaj", "author_url": "", "post_date": "04/29/2019 16:09:17", "content": "Thanks for an informative summary. I spent the last week reading all the research about NLP from AWD_LSTM, ELMO, GPT to BERT. After reading all the theory, I was thinking of finding resources to practice, which I luckily found in your discussion. So thanks for that.
\n\nI have one question. Have you tried using GPT-2 for any of your tasks and seeing how does it compare to BERT (although I think seeing that only small GPT-2 with one-tenth of params was released, it would not work much better than BERT).
", "votes": null, "replies": [ { "id": 524971, "author_name": "kashnitsky", "author_url": "", "post_date": "04/29/2019 21:44:27", "content": "No, haven't yet tried GPT-2. Though it's indeed interesting to compare with BERT.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "524264": "Semi-supervised and transfer learning in NLP is a very hot topic. And I wish we had a \"big\" Kaggle competition like this one ([\"Clickbait news detection\"](https://www.kaggle.com/c/dlinnlp-spring-2019-clf)). And more time of course, 1 week is nothing. I decided to spend only ~ 5 hours on this competition (already had ULMFiT and logit-tf-idf pipelines built for other tasks) but would've spent much more if we had at least 6 weeks (got almost the same task at my work). These are the main ideas that came to my mind (not only about SSL but merely concerning the current competition as well):\n\n1. **BERT fine-tuning.** In the recent gendered pronoun resolution [competition](https://www.kaggle.com/c/gendered-pronoun-resolution) by Google AI (so far imho the best competition ever) my team missed gold, mostly due to the fact that we didn't manage to fine-tune BERT for classification (quickly overfitted). Still on my way to understand how this shall be done in a simple setting like we have in this clickbait detection task.\n\n2. **Pseudo-labeling.** Typically you train a model, make predictions for the unlabeled set and then use these predictions as new labels. Kaggle hacks include leaving only some 10% most confident predictions and performing 3-5 iterations of pseudo-labeling. I enjoy [this discussion](https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/81012) in Santander where @ddanevskyi (Dmitriy Danevskiy) encouraged all the crowds to try smth new instead of this ruddy stacking. My idea for this competition was to train several different models, make predictions for the unlabeled set, and then select only those instances for which predictions are both confident and coherent across different models. \n\n3. **LSTMs/GRUs with embeddings**. The datasets are small and encourage the use of semi-supervised and transfer learning techniques (which is very cool). However, the data is not as scarce as to abandon classic GRUs and LSTMs with embeddings. Here you can just reuse some pipelines shared in Jigsaw and Quora competitions. Like [this Kernel](https://www.kaggle.com/jhoward/improved-lstm-baseline-glove-dropout) by @jhoward (Jeremy Howard) in Jigsaw or [this baseline](https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings) by @sudalairajkumar (SRK). If you use embeddings, then learn how to make your preprocessing suit the vocabulary better - there's a very important [Kernel](https://www.kaggle.com/christofhenkel/how-to-preprocessing-when-using-embeddings) by @christofhenkel (Dieter). \n\n4. **Data augmentation**. Data augmentation for NLP is still a hot research topic. So far doesn't work as well as for computer vision. However, some approaches *sometimes* work. Among really beautiful ones is *backtranslation*, I first saw it [shared](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038#latest-310689) by @pavelost (Pavel Ostyakov) in the toxic comments competition. The idea is that you augment the dataset by translating texts to another language (e.g. French, German) and then back to English. \n\n5. **Stacking/blending.** Well... I don't enjoy spending time on this shit - much better to learn smth new (like covering basics of machine translation). But this technique is efficient. Typically you can just create OOF predictions with as many models as you can handle, and then train 2-3 simple models (logit, lgb, shallow RF) on top of that. You get a lot of models making a cartesian product of several models, random seeds, feature sets (in this case titles only, texts only, titles + texst etc), hyperparam settings etc. \n\nEncourage you to share more ideas on efficient SSL for NLP. And hope we'll have a separate competition for that. At least I've got a funny dataset with BTC news sentiment (~3k labeled, 50k unlabeled), will share it if allowed.", "524839": "Thanks for an informative summary. I spent the last week reading all the research about NLP from AWD_LSTM, ELMO, GPT to BERT. After reading all the theory, I was thinking of finding resources to practice, which I luckily found in your discussion. So thanks for that.\n\nI have one question. Have you tried using GPT-2 for any of your tasks and seeing how does it compare to BERT (although I think seeing that only small GPT-2 with one-tenth of params was released, it would not work much better than BERT).", "524971": "No, haven't yet tried GPT-2. Though it's indeed interesting to compare with BERT.", "1266670": "Thanks for sharing!", "3523107": "A fascinating NLP question: **how much labeled text does a model really need?** 🔎\n\nhttps://mlguidance.blogspot.com/2026/09/what-are-models-of-semi-supervised.html" }, "source": "meta" }