{ "id": 257629, "title": "[Private 13th Solution] Pseudo-labeling", "content": "

Thanks the community for this competition!

\n

Overview

\n\n

Note: I'll report my Public scores in this post since Public / Private difference is only around 0.001.

\n

First half

\n

I spent the majority of first half trying different approaches (classification, custom loss) and learning to use hunggingface with TF and Pytorch. But I didn't know how things work so all of the experiments were not reproducible and turning into classification problem basically throws all the ordinal information away so it wasn't effective.

\n

After getting a hang of Pytorch and huggingface, I trained models TF models on train set that was shifted to the left since TF models were overpredicting on val set. Ensembling roberta-large with xlnet-large got me to LB 0.462. Adding Pytorch models and attention head trained with torch's amazing script boosted me to 0.458.

\n

At this point my single best 5 fold model was only 0.468.

\n

Second half

\n

Pseudo-labeling

\n

Inspired after reading Self-training with Noisy Student improves ImageNet classification\n.

\n

I used mainly CMU's book summaries, movie summaries and a little bit of Mark Wijkhuizen's wikipedia abstracts. Datasets were created by cleaning and mixing different ones together. I generated pseudo-labels with my best ensemble at the time. Most datasets have 16k to 26k excerpts.

\n

Models were trained with 80% of train data (first fold) concatenated with the new dataset. This helped me boost my literal single model (1 file) to 0.465 on roberta-large.

\n

After this I spent two weeks struggling to find what makes a good dataset, I ended up with 20+ different datasets and still weren't able to figure it out due to my lack of knowledge.

\n

Last two weeks of the competition, I decided to use only 3 datasets that performed best and simply added xlnet-large, electra-large, deberta-large and my best score becomes 0.455 with 4 files. Adding SVR 0.454 with 4 files. Splitting the new dataset to 4 folds and ensembling deberta-large, electra-large got me to 0.452 with 8 files.

\n

Few findings:

\n\n

Note: I'm not an expert in the field and there might be hindsight bias so feel free to correct me.

\n

Finishing line

\n\n

Summary

\n

What worked

\n\n

What didn't work
\nMost of my experiments revolves around new datasets so I don't have a lot here.

\n", "comments": [ { "id": 1445422, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-08-04T08:50:34.283000", "content": "

Congrats @abcbum on the results :D

", "votes": 3, "replies": [ { "id": 1450192, "author_name": "ABCbum", "author_url": "", "post_date": "2021-08-05T04:04:58.133000", "content": "

@duykhanh99 haha, means a lot coming from you, I look up to you the whole competition!

", "votes": 2, "replies": [] }, { "id": 1450286, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-08-05T04:37:10.557000", "content": "

biết bro người Việt thì mời vô nhóm chơi rồi :))

", "votes": 2, "replies": [] }, { "id": 1450293, "author_name": "ABCbum", "author_url": "", "post_date": "2021-08-05T04:39:41.473000", "content": "

haha, you guys were so high up so I didn't request xD, hope to see you in future competitions! (hơi tiếc tí a ạ :D).

", "votes": 2, "replies": [] }, { "id": 1450363, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-08-05T05:05:37.410000", "content": "

Okay, hy vọng bài khác mình có thể hợp tác :))

", "votes": 2, "replies": [] }, { "id": 1451486, "author_name": "Khuong T G Hieu", "author_url": "", "post_date": "2021-08-05T11:02:43.890000", "content": "

@abcbum Colab Pro nước mình đăng ký được nha bro ơi

", "votes": 1, "replies": [] }, { "id": 1453803, "author_name": "ABCbum", "author_url": "", "post_date": "2021-08-06T02:22:49.837000", "content": "

@ktgiahieu oh wow I didn't know that, thanks for letting me know. I'll follow you up on that ;D.

", "votes": 0, "replies": [] }, { "id": 1454159, "author_name": "HungNT", "author_url": "", "post_date": "2021-08-06T05:42:13.607000", "content": "

@abcbum Are you really studying at RMIT? :v

", "votes": 1, "replies": [] }, { "id": 1454504, "author_name": "ABCbum", "author_url": "", "post_date": "2021-08-06T08:09:21.603000", "content": "

@hungkhoi haha yes I am :D.

", "votes": 0, "replies": [] } ] }, { "id": 1453281, "author_name": "RK5678", "author_url": "", "post_date": "2021-08-05T20:13:15.853000", "content": "

Many congratulations @abcbum . Can you elaborate more on \"Models were trained with 80% of train data (first fold) concatenated with the new dataset.\"

", "votes": 1, "replies": [ { "id": 1453802, "author_name": "ABCbum", "author_url": "", "post_date": "2021-08-06T02:22:02.900000", "content": "

@rk3256 thanks for the question. It means I take the train data of first fold (80% of train set) together with the new dataset to train the model. So in total I train the model with 80% * 2800 + ~16k = ~18k.

", "votes": 1, "replies": [] } ] }, { "id": 1443892, "author_name": "torch", "author_url": "", "post_date": "2021-08-04T04:27:56.900000", "content": "

Many congratulations for your solo silver and thanks for sharing @abcbum.

", "votes": 1, "replies": [ { "id": 1444034, "author_name": "ABCbum", "author_url": "", "post_date": "2021-08-04T04:51:53.670000", "content": "

@rhtsingh xD thanks, great to hear from you, couldn't have made it without you!

", "votes": 1, "replies": [] } ] }, { "id": 1422582, "author_name": "Mighty Rains", "author_url": "", "post_date": "2021-08-03T05:01:17.650000", "content": "

@abcbum, great effort!

", "votes": 1, "replies": [] } ], "comment_count": 14, "messages": [ { "id": 1421592, "postDate": "2021-08-03T04:13:27.147Z", "content": "

Thanks the community for this competition!

\n

Overview

\n\n

Note: I'll report my Public scores in this post since Public / Private difference is only around 0.001.

\n

First half

\n

I spent the majority of first half trying different approaches (classification, custom loss) and learning to use hunggingface with TF and Pytorch. But I didn't know how things work so all of the experiments were not reproducible and turning into classification problem basically throws all the ordinal information away so it wasn't effective.

\n

After getting a hang of Pytorch and huggingface, I trained models TF models on train set that was shifted to the left since TF models were overpredicting on val set. Ensembling roberta-large with xlnet-large got me to LB 0.462. Adding Pytorch models and attention head trained with torch's amazing script boosted me to 0.458.

\n

At this point my single best 5 fold model was only 0.468.

\n

Second half

\n

Pseudo-labeling

\n

Inspired after reading Self-training with Noisy Student improves ImageNet classification\n.

\n

I used mainly CMU's book summaries, movie summaries and a little bit of Mark Wijkhuizen's wikipedia abstracts. Datasets were created by cleaning and mixing different ones together. I generated pseudo-labels with my best ensemble at the time. Most datasets have 16k to 26k excerpts.

\n

Models were trained with 80% of train data (first fold) concatenated with the new dataset. This helped me boost my literal single model (1 file) to 0.465 on roberta-large.

\n

After this I spent two weeks struggling to find what makes a good dataset, I ended up with 20+ different datasets and still weren't able to figure it out due to my lack of knowledge.

\n

Last two weeks of the competition, I decided to use only 3 datasets that performed best and simply added xlnet-large, electra-large, deberta-large and my best score becomes 0.455 with 4 files. Adding SVR 0.454 with 4 files. Splitting the new dataset to 4 folds and ensembling deberta-large, electra-large got me to 0.452 with 8 files.

\n

Few findings:

\n\n

Note: I'm not an expert in the field and there might be hindsight bias so feel free to correct me.

\n

Finishing line

\n\n

Summary

\n

What worked

\n\n

What didn't work
\nMost of my experiments revolves around new datasets so I don't have a lot here.

\n", "rawMarkdown": "Thanks the community for this competition!\n\n### Overview\n- Model architectures: roberta-large, xlnet-large, electra-large, deberta-large.\n- Attention head: [maunish's clrp-roberta-svm](https://www.kaggle.com/maunish/clrp-roberta-svm).\n- Semi-supervised learning, Pseudo-label.\n- Hyperparameters: I didn't tune much and use mostly original setup from [torch's training script] (https://www.kaggle.com/rhtsingh/commonlit-readability-prize-roberta-torch-fit) since I won't have much chance competing in what I'm far behind others.\n\nNote: I'll report my Public scores in this post since Public / Private difference is only around 0.001.\n\n### First half\nI spent the majority of first half trying different approaches (classification, custom loss) and learning to use hunggingface with TF and Pytorch. But I didn't know how things work so all of the experiments were not reproducible and turning into classification problem basically throws all the ordinal information away so it wasn't effective.\n\nAfter getting a hang of Pytorch and huggingface, I trained models TF models on train set that was shifted to the left since TF models were overpredicting on val set. Ensembling roberta-large with xlnet-large got me to LB **0.462**. Adding Pytorch models and attention head trained with [torch's amazing script](https://www.kaggle.com/rhtsingh/commonlit-readability-prize-roberta-torch-fit) boosted me to **0.458**.\n\nAt this point my single best 5 fold model was only **0.468**.\n\n### Second half\n#### Pseudo-labeling\nInspired after reading [Self-training with Noisy Student improves ImageNet classification\n](https://openaccess.thecvf.com/content_CVPR_2020/papers/Xie_Self-Training_With_Noisy_Student_Improves_ImageNet_Classification_CVPR_2020_paper.pdf).\n\nI used mainly [CMU's book summaries](https://www.cs.cmu.edu/~dbamman/booksummaries.html), [movie summaries] (https://www.cs.cmu.edu/~ark/personas/) and a little bit of [Mark Wijkhuizen's wikipedia abstracts](https://www.kaggle.com/markwijkhuizen/simplenormal-wikipedia-abstracts-v1). Datasets were created by cleaning and mixing different ones together. I generated pseudo-labels with my best ensemble at the time. Most datasets have **16k to 26k** excerpts.\n\nModels were trained with 80% of train data (first fold) concatenated with the new dataset. This helped me boost my literal single model (1 file) to **0.465** on roberta-large. \n\nAfter this I spent two weeks struggling to find what makes a good dataset, I ended up with 20+ different datasets and still weren't able to figure it out due to my lack of knowledge. \n\nLast two weeks of the competition, I decided to use only 3 datasets that performed best and simply added xlnet-large, electra-large, deberta-large and my best score becomes **0.455** with 4 files. Adding SVR **0.454** with 4 files. Splitting the new dataset to 4 folds and ensembling deberta-large, electra-large got me to **0.452** with 8 files.\n\nFew findings:\n- Public score got worse after the first generation. I hypothesize this is due to model's learning capacity since this method leans more towards distilling and compressing model rather than the usual setup for most semi-supervised learning methods where a single model's prediction is used to generate pseudo-labels instead of an ensemble. This finding is similar to what was found in [On the Efficacy of Knowledge Distillation](https://openaccess.thecvf.com/content_ICCV_2019/papers/Cho_On_the_Efficacy_of_Knowledge_Distillation_ICCV_2019_paper.pdf) by the authors even though it's a different technique: \"Bigger models are not better teachers\".\n- deberta-large performed best.\n- Adding noise didn't improve CV so I didn't submit.\n\nNote: I'm not an expert in the field and there might be hindsight bias so feel free to correct me.\n\n#### Finishing line\n- After ensembling a bunch of models together, I finished with **0.450 (20th place)** on Public LB.\n- Due to long training time (~2-3 hours/model), I used up to 6 Colab accounts in the last week LOL. I feel bad for this but that's the best I can do with my 8GB RAM laptop and Colab Pro isn't available where I live.\n\n### Summary\n**What worked**\n- Attention head: +0.004.\n- Pseudo-label: +0.015.\n- SVR: +0.001.\n\n**What didn't work**\nMost of my experiments revolves around new datasets so I don't have a lot here.\n- Classification: I couldn't utilize ordinal information for classification task.\n- Iterative pseudo-labeling: Score gets worse from 2nd generation (my thoughts on this above).\n- Post-processing: Interval mean matching, basically I try to match prediction's mean in each interval ([-0.5, 0.5], ...) with the train set. Worked okay on validation set but not LB.\n", "votes": 37 }, { "id": 1445422, "postDate": "2021-08-04T08:50:34.283Z", "content": "

Congrats @abcbum on the results :D

", "rawMarkdown": "Congrats @abcbum on the results :D", "votes": 3, "replies": [ { "id": 1450192, "postDate": "2021-08-05T04:04:58.133Z", "content": "

@duykhanh99 haha, means a lot coming from you, I look up to you the whole competition!

", "rawMarkdown": "@duykhanh99 haha, means a lot coming from you, I look up to you the whole competition!", "votes": 2 }, { "id": 1450286, "postDate": "2021-08-05T04:37:10.557Z", "content": "

biết bro người Việt thì mời vô nhóm chơi rồi :))

", "rawMarkdown": "biết bro người Việt thì mời vô nhóm chơi rồi :))", "votes": 2 }, { "id": 1450293, "postDate": "2021-08-05T04:39:41.473Z", "content": "

haha, you guys were so high up so I didn't request xD, hope to see you in future competitions! (hơi tiếc tí a ạ :D).

", "rawMarkdown": "haha, you guys were so high up so I didn't request xD, hope to see you in future competitions! (hơi tiếc tí a ạ :D).", "votes": 2 }, { "id": 1450363, "postDate": "2021-08-05T05:05:37.410Z", "content": "

Okay, hy vọng bài khác mình có thể hợp tác :))

", "rawMarkdown": "Okay, hy vọng bài khác mình có thể hợp tác :))", "votes": 2 }, { "id": 1451486, "postDate": "2021-08-05T11:02:43.890Z", "content": "

@abcbum Colab Pro nước mình đăng ký được nha bro ơi

", "rawMarkdown": "@abcbum Colab Pro nước mình đăng ký được nha bro ơi", "votes": 1 }, { "id": 1453803, "postDate": "2021-08-06T02:22:49.837Z", "content": "

@ktgiahieu oh wow I didn't know that, thanks for letting me know. I'll follow you up on that ;D.

", "rawMarkdown": "@ktgiahieu oh wow I didn't know that, thanks for letting me know. I'll follow you up on that ;D." }, { "id": 1454159, "postDate": "2021-08-06T05:42:13.607Z", "content": "

@abcbum Are you really studying at RMIT? :v

", "rawMarkdown": "@abcbum Are you really studying at RMIT? :v ", "votes": 1 }, { "id": 1454504, "postDate": "2021-08-06T08:09:21.603Z", "content": "

@hungkhoi haha yes I am :D.

", "rawMarkdown": "@hungkhoi haha yes I am :D." } ] }, { "id": 1453281, "postDate": "2021-08-05T20:13:15.853Z", "content": "

Many congratulations @abcbum . Can you elaborate more on \"Models were trained with 80% of train data (first fold) concatenated with the new dataset.\"

", "rawMarkdown": "Many congratulations @abcbum . Can you elaborate more on \"Models were trained with 80% of train data (first fold) concatenated with the new dataset.\"", "votes": 1, "replies": [ { "id": 1453802, "postDate": "2021-08-06T02:22:02.900Z", "content": "

@rk3256 thanks for the question. It means I take the train data of first fold (80% of train set) together with the new dataset to train the model. So in total I train the model with 80% * 2800 + ~16k = ~18k.

", "rawMarkdown": "@rk3256 thanks for the question. It means I take the train data of first fold (80% of train set) together with the new dataset to train the model. So in total I train the model with 80% * 2800 + ~16k = ~18k.", "votes": 1 } ] }, { "id": 1443892, "postDate": "2021-08-04T04:27:56.900Z", "content": "

Many congratulations for your solo silver and thanks for sharing @abcbum.

", "rawMarkdown": "Many congratulations for your solo silver and thanks for sharing @abcbum.", "votes": 1, "replies": [ { "id": 1444034, "postDate": "2021-08-04T04:51:53.670Z", "content": "

@rhtsingh xD thanks, great to hear from you, couldn't have made it without you!

", "rawMarkdown": "@rhtsingh xD thanks, great to hear from you, couldn't have made it without you!", "votes": 1 } ] }, { "id": 1422582, "postDate": "2021-08-03T05:01:17.650Z", "content": "

@abcbum, great effort!

", "rawMarkdown": "@abcbum, great effort! ", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:44:56Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6803131, "solution_url": "/competitions/commonlitreadabilityprize/writeups/nguyen-duc-minh-private-13th-solution-pseudo-label" }