{ "id": 446818, "title": "14th place Solution (lucky for me)", "content": "

First my HUGE thanks to the competition host. what a fantastic competition! I indeeed had a fun time kaggling!
\nAlso HUGE thanks to the shared notedbooks and the valuable discussions, from which I got a lot of inspiration and enlightment.

\n

I'm not actually an NLP expert with all sorts of experiences and tuning skills, thus I felt very lucky that I could have got this far. Also because of the shake which fortunately turned out to be in my favour this time, doesn't really happen very often, :p.

\n

Like many other top solutions, my model is acutally very simple deberta-v3-large. But the way I got there was rather serpentine.

\n

MY SOLUTION

\n\n
prompt = sep.join([data[], data[], data[]])\ntokenized = tokenizer(\n        data[],\n        prompt,\n        =,\n        =, \n        =config.max_seq_length,\n    )\n
\n\n

Things did NOT work

\n
    \n

  1. MLM did not work
    \nThis was actually a major mistake I made in this competition. I took it for granted that MLM of deberta on previous commonlit datasets would help boosting the CV. With that in mind, I actually spent the whole August and the first half of September only training the models pretrained with MLM. The best model I got with MLM had however, local CVs of 0.5449, 0.4782, 0.5182, 0.4473 and public LB 0.445. This could not then be improved anymore.
    \nIt was not until the third week of September did I overthrow everything and started again without any pretraining using MLM and ended up with the model described above.

    2. \n

  2. Manually generated datasets with pseudo scores could help accelerate the training process but can lead to overfitting
    \nAlmost at the end phase of the competition, I came up with the thoughts to expand the training data for pretraining task. Here is how I did this:

    \n\n

    I hope this explains your concern @kononenko :p.

    3. \n
\n

I used the extended data to do pretraining task. I got better local CV results and earlier converged networks. HOWEVER, the public LBs were not improving and in the end, the private LB turned out to be even worse :(. It could have led to overfitting (good lesson learnt). So my final model was not trained on this dataset.

\n
    \n
  1. Lightgbm did not work
    \nI did not dive deeply enough to understand why lgbm did not work. I actually expected it could boost my CV.
    2. \n
\n

Things that I would like to try if I were given more time

\n
    \n
  1. EMA
    \nI tried EMA in the early phase of this competition, with models pretrained by MLM. It did not help boosting the local CV so I didn't even bother submitting them. I would like to try EMA on the model without MLM pretraining if I had been given more time.
    2. \n
  2. layer-wise discriminative learning rates
    \nsame with EMA, i only applied this in my early models but did not boost local CV.
    3. \n
  3. play with more max length set ups
    4. \n
\n

Here is a summary of the models I tried

\n

*All LBs are the ensembles over 4 folds

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Model nameRemarksLocal CVLBPrivate LB
Model 2308pretrained with MLM, both pretraining and training are with max length 15360.5449, 0.4782, 0.5182, 0.44730.4450.481
Model 2708pretrained with MLM with max length 2048, model training with max length 15360.5778, 0.4598, 0.5305, 0.43690.4510.466
Model 3008pretrained with MLM, both pretraining and training are with max length 20480.5532, 0.4706, 0.5226, 0.43830.4470.471
Model 1009pretrained with MLM. EMA used in training0.5631, 0.4728, 0.5201, 0.4407not submittednot submitted
Model 2609 (best model)no pretraining, max length 16000.5513, 0.4587, 0.5038, 0.43160.4290.458
Model 0810with incomplete extended dataset to pretrain, max length 16000.5363, 0.4454, 0.4974, 0.43140.4310.474
Model 1010with complete extended data to pretrain, max length 16000.5199, 0.4451, 0.4958, 0.43560.4580.469
Model 0810 + lgbmwith complete extended data to pretrain, max length 1600, plus lgbm0.6154, 0.4695, 0.4990, 0.43910.4350.47
\n

As you can tell, with only a few exceptios, my public LBs and private LBs correlate somehow well.

", "comments": [ { "id": 2489548, "author_name": "Azamat Tolegen", "author_url": "", "post_date": "2023-10-20T04:22:33.787000", "content": "

Thanks for sharing and congratulations!
\nWas hyperparameter tuning stable for you? Was is it reproducible? How did you achieve it?

", "votes": 0, "replies": [ { "id": 2490610, "author_name": "Peng Sun", "author_url": "", "post_date": "2023-10-20T20:28:40.447000", "content": "

The tuning was stable. I did not do too many re-runs, but the ones I did were quite ok. The tuning was done in a step-wise manner. After several experiments I found max_length impacted the results the most. So I worked on that one first by trying several candidates for max_length. Then I did a grid search on learning_rate and weight_decay.

", "votes": 0, "replies": [] } ] }, { "id": 2480393, "author_name": "Jason Heesang Lee", "author_url": "", "post_date": "2023-10-13T09:45:15.307000", "content": "

Congratulations! Lot to learn!

", "votes": 0, "replies": [ { "id": 2480463, "author_name": "Peng Sun", "author_url": "", "post_date": "2023-10-13T10:23:39.990000", "content": "

thanks very much !!!

", "votes": 1, "replies": [] } ] }, { "id": 2480323, "author_name": "Adam Montgomerie", "author_url": "", "post_date": "2023-10-13T08:44:46.373000", "content": "

Congratulations on the gold medal!

", "votes": 0, "replies": [ { "id": 2480435, "author_name": "Peng Sun", "author_url": "", "post_date": "2023-10-13T10:07:08.323000", "content": "

congrats to you too!

", "votes": 0, "replies": [] } ] }, { "id": 2480233, "author_name": "Oleksiy Kononenko", "author_url": "", "post_date": "2023-10-13T07:15:05.633000", "content": "

Looking forward to your post, may be you could also reveal the mystery around this dataset :)

", "votes": 0, "replies": [ { "id": 2480464, "author_name": "Peng Sun", "author_url": "", "post_date": "2023-10-13T10:24:39.397000", "content": "

the post is finished. I tried to generate some extended training data. It accelerated the convergence and boosted local CVs but unfortunately not public or private LB :(

", "votes": 1, "replies": [ { "id": 2480482, "author_name": "Oleksiy Kononenko", "author_url": "", "post_date": "2023-10-13T10:36:24.733000", "content": "

I actually had a guess it was some GPT data. Very insightful summary, thanks, @penpen926.

", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 9, "messages": [ { "id": 2480225, "postDate": "2023-10-13T07:10:46.667Z", "content": "

First my HUGE thanks to the competition host. what a fantastic competition! I indeeed had a fun time kaggling!
\nAlso HUGE thanks to the shared notedbooks and the valuable discussions, from which I got a lot of inspiration and enlightment.

\n

I'm not actually an NLP expert with all sorts of experiences and tuning skills, thus I felt very lucky that I could have got this far. Also because of the shake which fortunately turned out to be in my favour this time, doesn't really happen very often, :p.

\n

Like many other top solutions, my model is acutally very simple deberta-v3-large. But the way I got there was rather serpentine.

\n

MY SOLUTION

\n\n
prompt = sep.join([data[], data[], data[]])\ntokenized = tokenizer(\n        data[],\n        prompt,\n        =,\n        =, \n        =config.max_seq_length,\n    )\n
\n\n

Things did NOT work

\n
    \n

  1. MLM did not work
    \nThis was actually a major mistake I made in this competition. I took it for granted that MLM of deberta on previous commonlit datasets would help boosting the CV. With that in mind, I actually spent the whole August and the first half of September only training the models pretrained with MLM. The best model I got with MLM had however, local CVs of 0.5449, 0.4782, 0.5182, 0.4473 and public LB 0.445. This could not then be improved anymore.
    \nIt was not until the third week of September did I overthrow everything and started again without any pretraining using MLM and ended up with the model described above.

    2. \n

  2. Manually generated datasets with pseudo scores could help accelerate the training process but can lead to overfitting
    \nAlmost at the end phase of the competition, I came up with the thoughts to expand the training data for pretraining task. Here is how I did this:

    \n\n

    I hope this explains your concern @kononenko :p.

    3. \n
\n

I used the extended data to do pretraining task. I got better local CV results and earlier converged networks. HOWEVER, the public LBs were not improving and in the end, the private LB turned out to be even worse :(. It could have led to overfitting (good lesson learnt). So my final model was not trained on this dataset.

\n
    \n
  1. Lightgbm did not work
    \nI did not dive deeply enough to understand why lgbm did not work. I actually expected it could boost my CV.
    2. \n
\n

Things that I would like to try if I were given more time

\n
    \n
  1. EMA
    \nI tried EMA in the early phase of this competition, with models pretrained by MLM. It did not help boosting the local CV so I didn't even bother submitting them. I would like to try EMA on the model without MLM pretraining if I had been given more time.
    2. \n
  2. layer-wise discriminative learning rates
    \nsame with EMA, i only applied this in my early models but did not boost local CV.
    3. \n
  3. play with more max length set ups
    4. \n
\n

Here is a summary of the models I tried

\n

*All LBs are the ensembles over 4 folds

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Model nameRemarksLocal CVLBPrivate LB
Model 2308pretrained with MLM, both pretraining and training are with max length 15360.5449, 0.4782, 0.5182, 0.44730.4450.481
Model 2708pretrained with MLM with max length 2048, model training with max length 15360.5778, 0.4598, 0.5305, 0.43690.4510.466
Model 3008pretrained with MLM, both pretraining and training are with max length 20480.5532, 0.4706, 0.5226, 0.43830.4470.471
Model 1009pretrained with MLM. EMA used in training0.5631, 0.4728, 0.5201, 0.4407not submittednot submitted
Model 2609 (best model)no pretraining, max length 16000.5513, 0.4587, 0.5038, 0.43160.4290.458
Model 0810with incomplete extended dataset to pretrain, max length 16000.5363, 0.4454, 0.4974, 0.43140.4310.474
Model 1010with complete extended data to pretrain, max length 16000.5199, 0.4451, 0.4958, 0.43560.4580.469
Model 0810 + lgbmwith complete extended data to pretrain, max length 1600, plus lgbm0.6154, 0.4695, 0.4990, 0.43910.4350.47
\n

As you can tell, with only a few exceptios, my public LBs and private LBs correlate somehow well.

", "rawMarkdown": "First my HUGE thanks to the competition host. what a fantastic competition! I indeeed had a fun time kaggling!\nAlso HUGE thanks to the shared notedbooks and the valuable discussions, from which I got a lot of inspiration and enlightment.\n\nI'm not actually an NLP expert with all sorts of experiences and tuning skills, thus I felt very lucky that I could have got this far. Also because of the shake which fortunately turned out to be in my favour this time, doesn't really happen very often, :p.\n\nLike many other top solutions, my model is acutally very simple deberta-v3-large. But the way I got there was rather serpentine.\n\n### MY SOLUTION \n\n* **Inputs**:\nParied input into tokenizer: _summary text_ and _prompt title_ +[SEP]+ _prompt text_ +[SEP] + _prompt question_\n```\nprompt = sep.join([data[\"prompt_title\"], data[\"prompt_text\"], data[\"prompt_question\"]])\ntokenized = tokenizer(\n data['text'],\n prompt,\n padding=False,\n truncation=True, \n max_length=config.max_seq_length,\n )\n```\n\n* **Model**:\ndeberta-v3-large. Like everybody else :P. I also tried some other models: deberta-v3-large-squad2, deberta-v3-base\n\n* **Max Length**:\nI tried different max lengths, the final model was with 1600.\n\n* **Model set up and Hyperparameters**:\nlearning rate: after many rounds of hyperparameter tuning, I found 6e-6 the best\nnum_epochs: 4 \nweight_decay: after hyperparameter tuning, 1e-4 \nwarmup_ratio: 0\nbatch_size: 2 for training and eval (I bascially cannot afford a larger batch_size due to GPU memory)\nlr_scheduler_type: cosine\noptimizer: adamw\nI also tried freezing some layers, but did not boost local CV so the final model I actually did not freeze anything.\\\nNo drop was used, no extra pooling was used.\nMoreover, I found that local CVs and public LBs positively correlated with max length in training. Later the private LB proved this. Models with larger max lengths tend to have better private LBs.\n\n* **CV Results and Ensemble**:\nthe best privat LB model achieved CV: 0.5513 (fold 814d6b) , 0.4587 (fold 39c16e), 0.5038 (fold 3b9047) and 0.4316 (fold ebad26)\nThe final model is a simple ensemble of the 4 folds. \nI actually considered to train the model using the set up above on the whole training data set. However, in the end I chose to do an ensemble of the models of different folds. This might not be the best practice, but gave me better opportunity to discover the relations between CV and public LB.\nPublic LB: 0.429\nPrivate LB: 0.458\n\n### Things did NOT work\n1. **MLM did not work**\nThis was actually a major mistake I made in this competition. I took it for granted that MLM of deberta on previous commonlit datasets would help boosting the CV. With that in mind, I actually spent the whole August and the first half of September only training the models pretrained with MLM. The best model I got with MLM had however, local CVs of 0.5449, 0.4782, 0.5182, 0.4473 and public LB 0.445. This could not then be improved anymore.\nIt was not until the third week of September did I overthrow everything and started again without any pretraining using MLM and ended up with the model described above.\n\n2. **Manually generated datasets with pseudo scores could help accelerate the training process but can lead to overfitting**\nAlmost at the end phase of the competition, I came up with the thoughts to expand the training data for pretraining task. Here is how I did this:\n * Minor part of the extended training data: I downloaded from commonlit website some texts as prompt texts and questions as prompt questions. I then generated summary texts using gpt-4 by letting the machine mimick the way students write. The pseudo scores were given by the best model I created at that time. That gave me around 300 more training rows.\n * Major part of the extended training data: without any prompt text, I directly used the student written texts in the previous commonlit competitions and let my best model score them. That gave me ~4000 extra rows\n\n I hope this explains your concern @kononenko :p. \n\nI used the extended data to do pretraining task. I got better local CV results and earlier converged networks. **HOWEVER**, the public LBs were not improving and in the end, the private LB turned out to be even worse :(. It could have led to overfitting (good lesson learnt). So my final model was not trained on this dataset.\n\n3. **Lightgbm did not work**\nI did not dive deeply enough to understand why lgbm did not work. I actually expected it could boost my CV.\n\n### Things that I would like to try if I were given more time\n1. EMA\nI tried EMA in the early phase of this competition, with models pretrained by MLM. It did not help boosting the local CV so I didn't even bother submitting them. I would like to try EMA on the model without MLM pretraining if I had been given more time.\n2. layer-wise discriminative learning rates \nsame with EMA, i only applied this in my early models but did not boost local CV.\n3. play with more max length set ups\n\n### Here is a summary of the models I tried\n*All LBs are the ensembles over 4 folds\n| Model name| Remarks | Local CV | LB | Private LB|\n|----------|:-------------:|------:|\n| Model 2308 | pretrained with MLM, both pretraining and training are with max length 1536 | 0.5449, 0.4782, 0.5182, 0.4473 | 0.445 | 0.481|\n| Model 2708 | pretrained with MLM with max length 2048, model training with max length 1536| 0.5778, 0.4598, 0.5305, 0.4369 | 0.451| 0.466|\n| Model 3008 | pretrained with MLM, both pretraining and training are with max length 2048| 0.5532, 0.4706, 0.5226, 0.4383 | 0.447| 0.471|\n| Model 1009 | pretrained with MLM. EMA used in training| 0.5631, 0.4728, 0.5201, 0.4407| not submitted| not submitted|\n| Model 2609 (best model)| no pretraining, max length 1600| 0.5513, 0.4587, 0.5038, 0.4316| 0.429| 0.458|\n| Model 0810| with incomplete extended dataset to pretrain, max length 1600| 0.5363, 0.4454, 0.4974, 0.4314|0.431 | 0.474|\n| Model 1010| with complete extended data to pretrain, max length 1600| 0.5199, 0.4451, 0.4958, 0.4356| 0.458| 0.469|\n|Model 0810 + lgbm| with complete extended data to pretrain, max length 1600, plus lgbm| 0.6154, 0.4695, 0.4990, 0.4391|0.435| 0.47|\n\nAs you can tell, with only a few exceptios, my public LBs and private LBs correlate somehow well.", "votes": 16 }, { "id": 2489548, "postDate": "2023-10-20T04:22:33.787Z", "content": "

Thanks for sharing and congratulations!
\nWas hyperparameter tuning stable for you? Was is it reproducible? How did you achieve it?

", "rawMarkdown": "Thanks for sharing and congratulations! \nWas hyperparameter tuning stable for you? Was is it reproducible? How did you achieve it?", "replies": [ { "id": 2490610, "postDate": "2023-10-20T20:28:40.447Z", "content": "

The tuning was stable. I did not do too many re-runs, but the ones I did were quite ok. The tuning was done in a step-wise manner. After several experiments I found max_length impacted the results the most. So I worked on that one first by trying several candidates for max_length. Then I did a grid search on learning_rate and weight_decay.

", "rawMarkdown": "The tuning was stable. I did not do too many re-runs, but the ones I did were quite ok. The tuning was done in a step-wise manner. After several experiments I found max_length impacted the results the most. So I worked on that one first by trying several candidates for max_length. Then I did a grid search on learning_rate and weight_decay. " } ] }, { "id": 2480393, "postDate": "2023-10-13T09:45:15.307Z", "content": "

Congratulations! Lot to learn!

", "rawMarkdown": "Congratulations! Lot to learn!", "replies": [ { "id": 2480463, "postDate": "2023-10-13T10:23:39.990Z", "content": "

thanks very much !!!

", "rawMarkdown": "thanks very much !!!", "votes": 1 } ] }, { "id": 2480323, "postDate": "2023-10-13T08:44:46.373Z", "content": "

Congratulations on the gold medal!

", "rawMarkdown": "Congratulations on the gold medal!", "replies": [ { "id": 2480435, "postDate": "2023-10-13T10:07:08.323Z", "content": "

congrats to you too!

", "rawMarkdown": "congrats to you too!" } ] }, { "id": 2480233, "postDate": "2023-10-13T07:15:05.633Z", "content": "

Looking forward to your post, may be you could also reveal the mystery around this dataset :)

", "rawMarkdown": "Looking forward to your post, may be you could also reveal the mystery around [this dataset](https://www.kaggle.com/datasets/penpen926/commonlit-new-train-data) :)", "replies": [ { "id": 2480464, "postDate": "2023-10-13T10:24:39.397Z", "content": "

the post is finished. I tried to generate some extended training data. It accelerated the convergence and boosted local CVs but unfortunately not public or private LB :(

", "rawMarkdown": "the post is finished. I tried to generate some extended training data. It accelerated the convergence and boosted local CVs but unfortunately not public or private LB :(", "votes": 1, "replies": [ { "id": 2480482, "postDate": "2023-10-13T10:36:24.733Z", "content": "

I actually had a guess it was some GPT data. Very insightful summary, thanks, @penpen926.

", "rawMarkdown": "I actually had a [guess](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/446189#2476590) it was some GPT data. Very insightful summary, thanks, @penpen926. \n\n" } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:05:06Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10790526, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/peng-sun-14th-place-solution-lucky-for-me" }