We will try to be as brief as possible.
\nStep 1 - Training Part:
\nWe have tried all most all the models from Hugging Face, among them, Xlnet for us is the winner.
\na. With original dataset. Our best single model is xlnet large model(lb 0.461,private lb 0.463).
b. Using pseudo labels for training. We blend some of our selected models to get a public LB score of 0.455. Using the blended models to generate pseudo labels for the legally public data as external data and continue training. Our best pseudo label model is Roberta large model(lb 0.460,private lb 0.456).
\nc. A special remark of the training process, We used std as the second loss. e.g.
\nloss = torch.sqrt(loss_fn(logits, labels.view(-1)))+0.6*torch.sqrt(loss_fn(logits, (labels-stds).view(-1))).
Step 2 - Avoiding shake up with multi-level stacking.
\nSince we have used at the end more than 20 models: Half of them trained with the original dataset, half of them with pseudo labeled dataset. The cv scores are much different. The blend helps us only to reach 0.451 and 0.453 on the public and private LB respectively. How to move on?
Stacking. But so many models with much different cv scores, normal stacking works not so good at this point. We applied multi-level stacking. We used Lasso as the base model for stacking. All the low-score models are placed at level 1, pseudo label models at level 2, linear models at level 3. This setup helps us to reach 14th place.
\nHowever, it is not our best score. We selected 12 models and applied 12 Level stackings, i.e. each level just append one model oof/prediction, public and private LB are both 0.448, there is no overfitting. If more models are added, it might be scored higher.
\nStep 3 - detailed statistical data analysis
\nWe worked hard on selecting the final submissions since we noticed there might be a big chance to have a shake-up on the LB. It took us almost 3 days. We did hundreds of plottings and different statistical analyses. It worthed, especially with such a small dataset.
\nThat is it, thanks for reading guys!
", "comments": [ { "id": 1454986, "author_name": "huiqin", "author_url": "", "post_date": "2021-08-06T12:09:18.870000", "content": "@yukia18 We used wiki free data(about 2000 records) and google quest train data (about 5000 records) as the pseudo labels. I think they are not the best choices for pseudo labels, but they are legal and free to use.
", "votes": 3, "replies": [] }, { "id": 1454091, "author_name": "yabea", "author_url": "", "post_date": "2021-08-06T05:08:38.493000", "content": "Thanks for sharing! Very interesting.
\nWhat kind / amount of data did you use for the pseudo-labeling? I would like to know if the genre of the document or the amount of document affected the results.
", "votes": 1, "replies": [] } ], "comment_count": 2, "messages": [ { "id": 1426538, "postDate": "2021-08-03T07:39:41.153Z", "content": "We will try to be as brief as possible.
\nStep 1 - Training Part:
\nWe have tried all most all the models from Hugging Face, among them, Xlnet for us is the winner.
\na. With original dataset. Our best single model is xlnet large model(lb 0.461,private lb 0.463).
b. Using pseudo labels for training. We blend some of our selected models to get a public LB score of 0.455. Using the blended models to generate pseudo labels for the legally public data as external data and continue training. Our best pseudo label model is Roberta large model(lb 0.460,private lb 0.456).
\nc. A special remark of the training process, We used std as the second loss. e.g.
\nloss = torch.sqrt(loss_fn(logits, labels.view(-1)))+0.6*torch.sqrt(loss_fn(logits, (labels-stds).view(-1))).
Step 2 - Avoiding shake up with multi-level stacking.
\nSince we have used at the end more than 20 models: Half of them trained with the original dataset, half of them with pseudo labeled dataset. The cv scores are much different. The blend helps us only to reach 0.451 and 0.453 on the public and private LB respectively. How to move on?
Stacking. But so many models with much different cv scores, normal stacking works not so good at this point. We applied multi-level stacking. We used Lasso as the base model for stacking. All the low-score models are placed at level 1, pseudo label models at level 2, linear models at level 3. This setup helps us to reach 14th place.
\nHowever, it is not our best score. We selected 12 models and applied 12 Level stackings, i.e. each level just append one model oof/prediction, public and private LB are both 0.448, there is no overfitting. If more models are added, it might be scored higher.
\nStep 3 - detailed statistical data analysis
\nWe worked hard on selecting the final submissions since we noticed there might be a big chance to have a shake-up on the LB. It took us almost 3 days. We did hundreds of plottings and different statistical analyses. It worthed, especially with such a small dataset.
\nThat is it, thanks for reading guys!
", "rawMarkdown": "We will try to be as brief as possible.\n\n**Step 1 - Training Part:**\nWe have tried all most all the models from Hugging Face, among them, Xlnet for us is the winner.\na. With original dataset. Our best single model is xlnet large model(lb 0.461,private lb 0.463).\n\nb. Using pseudo labels for training. We blend some of our selected models to get a public LB score of 0.455. Using the blended models to generate pseudo labels for the legally public data as external data and continue training. Our best pseudo label model is Roberta large model(lb 0.460,private lb 0.456).\n\nc. A special remark of the training process, We used std as the second loss. e.g.\nloss = torch.sqrt(loss_fn(logits, labels.view(-1)))+0.6*torch.sqrt(loss_fn(logits, (labels-stds).view(-1))).\n\n**Step 2 - Avoiding shake up with multi-level stacking.**\nSince we have used at the end more than 20 models: Half of them trained with the original dataset, half of them with pseudo labeled dataset. The cv scores are much different. The blend helps us only to reach 0.451 and 0.453 on the public and private LB respectively. How to move on?\n\nStacking. But so many models with much different cv scores, normal stacking works not so good at this point. We applied multi-level stacking. We used Lasso as the base model for stacking. All the low-score models are placed at level 1, pseudo label models at level 2, linear models at level 3. This setup helps us to reach 14th place.\n\nHowever, it is not our best score. We selected 12 models and applied 12 Level stackings, i.e. each level just append one model oof/prediction, public and private LB are both 0.448, there is no overfitting. If more models are added, it might be scored higher.\n\n**Step 3 - detailed statistical data analysis**\n\nWe worked hard on selecting the final submissions since we noticed there might be a big chance to have a shake-up on the LB. It took us almost 3 days. We did hundreds of plottings and different statistical analyses. It worthed, especially with such a small dataset.\n\nThat is it, thanks for reading guys!", "votes": 23 }, { "id": 1454986, "postDate": "2021-08-06T12:09:18.870Z", "content": "@yukia18 We used wiki free data(about 2000 records) and google quest train data (about 5000 records) as the pseudo labels. I think they are not the best choices for pseudo labels, but they are legal and free to use.
", "rawMarkdown": "@yukia18 We used wiki free data(about 2000 records) and google quest train data (about 5000 records) as the pseudo labels. I think they are not the best choices for pseudo labels, but they are legal and free to use. ", "votes": 3 }, { "id": 1454091, "postDate": "2021-08-06T05:08:38.493Z", "content": "Thanks for sharing! Very interesting.
\nWhat kind / amount of data did you use for the pseudo-labeling? I would like to know if the genre of the document or the amount of document affected the results.
", "rawMarkdown": "Thanks for sharing! Very interesting.\n\nWhat kind / amount of data did you use for the pseudo-labeling? I would like to know if the genre of the document or the amount of document affected the results.", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:46:47Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6896908, "solution_url": "/competitions/commonlitreadabilityprize/writeups/lyzd-fintech-14th-place-solution-with-best-single-" }