{ "id": 447254, "title": "12th Place Solution", "content": "

Thank you for organizing such an interesting competition!
\nI was unable to sleep due to the fear of shakedown in the last part of the competition, but I am happy to have remained in the gold medal position.

\n

I would like to briefly summarize my methodology.

\n

Model & Result

\n

My best private solution is an ensemble of long mode with prompt_text and short model without prompt_text.
\nThe former is intended to evaluate that the text is summarized correctly, and latter to evaluate that sentence structure and word quality is good or bad.

\n

Long Model

\n

\n

Using prompt_text and long max_len worked very well for me. But inference time is long (over 6 hour) and I was not able to ensemble with other long model. So using the output of the Pooling Layer, trained LightGBM to improve robustness.

\n

Short Model

\n

\n

Most of this model is based on a very nice public notebook.
\nThe changes are as follows

\n\n

Ensemble Model

\n

I simply ensemble (Long + Short)/2
\n

\n

Not working for me

\n\n

Thanks.

", "comments": [ { "id": 2483816, "author_name": "anji", "author_url": "", "post_date": "2023-10-16T02:38:04.703000", "content": "

Congratulations on the gold medal, I noticed you used CLS pooling in the long model and achieved a cv of 0.489, I have used the same approach but my cv is still around 0.51, so I was wondering if you used any other optimization strategies and if it's convenient ,I hope you'll be releasing the code, thanks:)

", "votes": 2, "replies": [ { "id": 2485921, "author_name": "htsbrer", "author_url": "", "post_date": "2023-10-17T14:30:53.273000", "content": "

Thank you for your comment.

\n

CLSPooling using only the final layer certainly did not work. Therefore, I used the averaged CLS pooling of the last 4 layers and the accuracy increased.(call CLS Mean Pooling？)

\n

But, since the training was not stable and CV of some seeds were low score, I tried several seeds and used the model with the highest CV.

", "votes": 2, "replies": [ { "id": 2487128, "author_name": "anji", "author_url": "", "post_date": "2023-10-18T11:42:57.090000", "content": "

Thanks for your kind response

", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 3, "messages": [ { "id": 2482420, "postDate": "2023-10-15T03:20:40.857Z", "content": "

Thank you for organizing such an interesting competition!
\nI was unable to sleep due to the fear of shakedown in the last part of the competition, but I am happy to have remained in the gold medal position.

\n

I would like to briefly summarize my methodology.

\n

Model & Result

\n

My best private solution is an ensemble of long mode with prompt_text and short model without prompt_text.
\nThe former is intended to evaluate that the text is summarized correctly, and latter to evaluate that sentence structure and word quality is good or bad.

\n

Long Model

\n

\n

Using prompt_text and long max_len worked very well for me. But inference time is long (over 6 hour) and I was not able to ensemble with other long model. So using the output of the Pooling Layer, trained LightGBM to improve robustness.

\n

Short Model

\n

\n

Most of this model is based on a very nice public notebook.
\nThe changes are as follows

\n\n

Ensemble Model

\n

I simply ensemble (Long + Short)/2
\n

\n

Not working for me

\n\n

Thanks.

", "rawMarkdown": "Thank you for organizing such an interesting competition!\nI was unable to sleep due to the fear of shakedown in the last part of the competition, but I am happy to have remained in the gold medal position.\n\nI would like to briefly summarize my methodology.\n\n# Model & Result\nMy best private solution is an ensemble of long mode with prompt_text and short model without prompt_text.\nThe former is intended to evaluate that the text is summarized correctly, and latter to evaluate that sentence structure and word quality is good or bad.\n\n## Long Model\n\n\nUsing prompt_text and long max_len worked very well for me. But inference time is long (over 6 hour) and I was not able to ensemble with other long model. So using the output of the Pooling Layer, trained LightGBM to improve robustness.\n\n## Short Model\n\n\nMost of this model is based on [a very nice public notebook](https://www.kaggle.com/code/tsunotsuno/debertav3-lgbm-no-autocorrect).\nThe changes are as follows\n* change deberta-base to deberta-v3-large\n* non text cleaning\n* drop prompt_length\n* change definition of \"overlap\"\n* freeze top 12 deberta layers / non layers ensemble\n\n## Ensemble Model\nI simply ensemble (Long + Short)/2\n\n\n# Not working for me\n* Text Cleaning\n* Backbone except deberta-v3-large\n* MLM\n* AWP\n* Augmentation with ChatGPT\n* SVR, xgboost (As an alternative to lightgbm)\n\nThanks.", "votes": 13 }, { "id": 2483816, "postDate": "2023-10-16T02:38:04.703Z", "content": "

Congratulations on the gold medal, I noticed you used CLS pooling in the long model and achieved a cv of 0.489, I have used the same approach but my cv is still around 0.51, so I was wondering if you used any other optimization strategies and if it's convenient ,I hope you'll be releasing the code, thanks:)

", "rawMarkdown": "Congratulations on the gold medal, I noticed you used **CLS pooling** in the **long model** and achieved a cv of 0.489, I have used the same approach but my cv is still around 0.51, so I was wondering if you used any other optimization strategies and if it's convenient ,I hope you'll be releasing the code, thanks:)", "votes": 2, "replies": [ { "id": 2485921, "postDate": "2023-10-17T14:30:53.273Z", "content": "

Thank you for your comment.

\n

CLSPooling using only the final layer certainly did not work. Therefore, I used the averaged CLS pooling of the last 4 layers and the accuracy increased.(call CLS Mean Pooling？)

\n

But, since the training was not stable and CV of some seeds were low score, I tried several seeds and used the model with the highest CV.

", "rawMarkdown": "Thank you for your comment.\n\nCLSPooling using only the final layer certainly did not work. Therefore, I used the averaged CLS pooling of the last 4 layers and the accuracy increased.(call CLS Mean Pooling？)\n\nBut, since the training was not stable and CV of some seeds were low score, I tried several seeds and used the model with the highest CV.", "votes": 2, "replies": [ { "id": 2487128, "postDate": "2023-10-18T11:42:57.090Z", "content": "

Thanks for your kind response

", "rawMarkdown": "Thanks for your kind response" } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:06:35Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10686985, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/htsbrer-12th-place-solution" }