![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8785809%2F985b784a03c245a29e6113d8de4f7957%2FScreen%20Shot%202023-10-12%20at%207.19.37%20AM.png?generation=1697091595551878&alt=media\")
\nCV: 0.505; public LB: 0.456; private LB: 0.474
Firstly, I want to thank Kaggle and the Host for organizing this fun competition, and thanks to my teammates @muhammad4hmed , @mohammad2012191 , @cody11null , @ivanisaev . This wouldn't be possible to achieve without them!
\nOur solution is an ensemble of diverse 1st stage models (Mostly with Deberta-v3-large backbone) feed into a 2nd Stage stacking Model. Diversity came from input, different custom pooling techniques and heads.
\nOur best submission out of the 3 we chose had CV 0.486 and LB 0.425. It gives PB 0.464 which puts us at 27th. The best submission out of all our submissions had LB of 0.459, so it is a silver in both cases :)
We finetuned Deberta Base, Large and Squad using summary texts + question only. We also added other models with some custom pooling ideas which involved using prompts inside, and they worked pretty nice, thanks to @ihebch and @muhammad4hmed work!
\n
\nCV: 0.505; public LB: 0.456; private LB: 0.474
same as Model1 but without cross-attention mechanism
\nCV: 0.49; publicLB: 0.575; privateLB: 0.468 (wow)
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8785809%2F9c30cf54687f2fb2126330e56af09975%2FScreen%20Shot%202023-10-12%20at%207.01.34%20AM.png?generation=1697090518733548&alt=media\")
\nCV: 0.52; PublicLB: 0.457; PrivateLB: 0.499
We then added all these predictions as features for our LGBM+XG+CB models along with the same features published in public notebooks.
\nWe also added FB3 labels (good boost) and used metadata features(specially grade).
\nWe tried hard to add new features/ do feature selection / do extensive hyperparameters tuning, but all these trials ended up with better cv and much worse lb, so we gave them up and focused on improving our 1st stage models.
Hey,
\nCongrats on finishing so high in the lb. The approach you used seems to be very interesting. I would like to ask a few questions:
\n1) What was the intuition behind using Bi-LSTM? Did you try anything else besides it?
\n2) You said you used FB3 labels, how did you actually do it? Creating pseudo labels using a finetuned model from the FB3 competition?
Hey,
\n\n\nWhat was the intuition behind using Bi-LSTM?
\n
using Bidirectional LSTM on top of token embeddings is meant to capture contextual information between student summary and the prompt.
\n\n\nDid you try anything else besides it?
\n
Pooling is the key for this competition as 2nd Place mentioned, adding special tokens and pooling summary and prompts individually did the magic for us.
\n\n\nYou said you used FB3 labels, how did you actually do it?
\n
We just inferred a public model shared in FB3 on train and test.
", "votes": 5, "replies": [] } ] }, { "id": 2479349, "author_name": "", "author_url": "", "post_date": "2023-10-12T15:16:53.690000", "content": "@ihebch great solution and very congrats to you and your team.If possible could you share your code for better understanding.
", "votes": 1, "replies": [ { "id": 2479370, "author_name": "Reacher", "author_url": "", "post_date": "2023-10-12T15:28:33.287000", "content": "my teammate @mohammad2012191 already shared a notebook that contains most of the code where he will explain his idea of 3rd stage in a separate discussion.
", "votes": 1, "replies": [ { "id": 2479445, "author_name": "", "author_url": "", "post_date": "2023-10-12T16:08:27.747000", "content": "Thanks @ihebch
", "votes": 1, "replies": [] }, { "id": 2479590, "author_name": "Mohamed Eltayeb", "author_url": "", "post_date": "2023-10-12T18:09:50.130000", "content": "Thanks @ihebch
\n@asteyagaur here is the idea if you are interested: https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/446711
@mohammad2012191 Thanks I definitely look into it . Congrats to you and your team.
", "votes": 1, "replies": [] } ] } ] } ], "comment_count": 7, "messages": [ { "id": 2478659, "postDate": "2023-10-12T06:22:12.917Z", "content": "Firstly, I want to thank Kaggle and the Host for organizing this fun competition, and thanks to my teammates @muhammad4hmed , @mohammad2012191 , @cody11null , @ivanisaev . This wouldn't be possible to achieve without them!
\nOur solution is an ensemble of diverse 1st stage models (Mostly with Deberta-v3-large backbone) feed into a 2nd Stage stacking Model. Diversity came from input, different custom pooling techniques and heads.
\nOur best submission out of the 3 we chose had CV 0.486 and LB 0.425. It gives PB 0.464 which puts us at 27th. The best submission out of all our submissions had LB of 0.459, so it is a silver in both cases :)
We finetuned Deberta Base, Large and Squad using summary texts + question only. We also added other models with some custom pooling ideas which involved using prompts inside, and they worked pretty nice, thanks to @ihebch and @muhammad4hmed work!
\n
\nCV: 0.505; public LB: 0.456; private LB: 0.474
same as Model1 but without cross-attention mechanism
\nCV: 0.49; publicLB: 0.575; privateLB: 0.468 (wow)
\nCV: 0.52; PublicLB: 0.457; PrivateLB: 0.499
We then added all these predictions as features for our LGBM+XG+CB models along with the same features published in public notebooks.
\nWe also added FB3 labels (good boost) and used metadata features(specially grade).
\nWe tried hard to add new features/ do feature selection / do extensive hyperparameters tuning, but all these trials ended up with better cv and much worse lb, so we gave them up and focused on improving our 1st stage models.
Hey,
\nCongrats on finishing so high in the lb. The approach you used seems to be very interesting. I would like to ask a few questions:
\n1) What was the intuition behind using Bi-LSTM? Did you try anything else besides it?
\n2) You said you used FB3 labels, how did you actually do it? Creating pseudo labels using a finetuned model from the FB3 competition?
Hey,
\n\n\nWhat was the intuition behind using Bi-LSTM?
\n
using Bidirectional LSTM on top of token embeddings is meant to capture contextual information between student summary and the prompt.
\n\n\nDid you try anything else besides it?
\n
Pooling is the key for this competition as 2nd Place mentioned, adding special tokens and pooling summary and prompts individually did the magic for us.
\n\n\nYou said you used FB3 labels, how did you actually do it?
\n
We just inferred a public model shared in FB3 on train and test.
", "rawMarkdown": "Hey,\n>What was the intuition behind using Bi-LSTM? \n\nusing Bidirectional LSTM on top of token embeddings is meant to capture contextual information between student summary and the prompt.\n>Did you try anything else besides it?\n\nPooling is the key for this competition as 2nd Place mentioned, adding special tokens and pooling summary and prompts individually did the magic for us.\n\n>You said you used FB3 labels, how did you actually do it?\n\nWe just inferred a public model shared in FB3 on train and test.", "votes": 5 } ] }, { "id": 2479349, "postDate": "2023-10-12T15:16:53.690Z", "content": "@ihebch great solution and very congrats to you and your team.If possible could you share your code for better understanding.
", "rawMarkdown": "@ihebch great solution and very congrats to you and your team.If possible could you share your code for better understanding.", "votes": 1, "isDeleted": true, "replies": [ { "id": 2479370, "postDate": "2023-10-12T15:28:33.287Z", "content": "my teammate @mohammad2012191 already shared a notebook that contains most of the code where he will explain his idea of 3rd stage in a separate discussion.
", "rawMarkdown": "my teammate @mohammad2012191 already shared a [notebook](https://www.kaggle.com/code/mohammad2012191/inference-notebook-stacking-v1-autocorrect-new-bas) that contains most of the code where he will explain his idea of 3rd stage in a separate discussion.\n", "votes": 1, "replies": [ { "id": 2479445, "postDate": "2023-10-12T16:08:27.747Z", "content": "Thanks @ihebch
", "rawMarkdown": "Thanks @ihebch ", "votes": 1, "isDeleted": true }, { "id": 2479590, "postDate": "2023-10-12T18:09:50.130Z", "content": "Thanks @ihebch
\n@asteyagaur here is the idea if you are interested: https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/446711
@mohammad2012191 Thanks I definitely look into it . Congrats to you and your team.
", "rawMarkdown": "@mohammad2012191 Thanks I definitely look into it . Congrats to you and your team.", "votes": 1, "isDeleted": true } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:01:58Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10811073, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/i-m-m-r-c-27th-place-solution-how-to-use-prompts" }