Thanks to kaggle and competition host.
\nIn last competition (USPPTPM) , i saw that the way we give data input to the transformers matters a lot so in this competition i was decided to experiment on that, in start i started with
\ndeberta-v3-base | 1024
\nI used data this way -> discourse_text + essay this gave me score -> 0.72(on LB)
deberta-v3-large | 1024 (with 5fold )
\nUsing oof ensembling this gave me 0.627(on LB) score data input this way -> discourse_text + essay
deberta-v3-large | 1024 (with 5fold ) + CLS embedding
\nIn this i not used essay instead i created essay by joining discourse text of same essay_id and added tags in the start of discourse text this way
\n[CLS][LEAD] discourse_text [LEAD][END] [POSITION] discourse_text [POSITION] β¦. [CLAIM] discourse_text [CLAIM]
\nthis gave me 0.61(on LB)
deberta-v3-large | 1024 (with 5fold ) + weighted layer pooling
\nIn this i replace cls embedding to weighted layer pooling and got 0.600 on lb
My final model was oof ensemble(just taking avg.) of 4th method. In the end am happy that i tried this instead of ensembling 3-4 model using discourse_text + essay this data.
\n**bad decisions i taken during competition **
\n**lesson learned: **
\n1.must not assume till you experiment
\n2.if something not worked on some model so it is not neccesery that it won't work with different model
\n3.if some one said in dissussion that this is not working don't trust i try and observe is that working or not
In last i am happy that i got bronze, Because last feedback competition was my first nlp competition and In that competition i just submitted sir @cdeotte 's notebook as it is , i was not able to understand that kernel(it was really nice kernel, i was dumb) then participated in NBME and USPPTPM and learned lot and got bronze in this competiton ik this is too much slow, in nbme and in this comp my team mates cheated on me still i learned a lot of things.
\nThanks to @abhishek and @SanyamBhutani for youtube videos and @vadimirtlach for answering my stupid questions it helped me a lot
", "comments": [ { "id": 1913696, "author_name": "Vadim Irtlach", "author_url": "", "post_date": "2022-08-25T12:59:32.233000", "content": "\n\nIn this i replace cls embedding to weighted layer pooling and got 0.600 on lb
\n
May you do pooling of layer's representations (weighted average) and then took CLS embedding, because I watched the implementation in this great notebook and the author did Weighted Layer Pooling and then CLS pooling. Is it, @arvinddevarkonda?
", "votes": 1, "replies": [ { "id": 1914001, "author_name": "Arvind Devarkonda", "author_url": "", "post_date": "2022-08-25T16:23:36.170000", "content": "yes. I used only weighted layer pooling from that notebook,i tried bilstm pooling layer with max_pooling it not worked so i assumed that using more than one pooling layer will not work.(here i made mistake)
", "votes": 1, "replies": [] } ] }, { "id": 1913412, "author_name": "Vadim Irtlach", "author_url": "", "post_date": "2022-08-25T10:11:04.743000", "content": "Good solution, it is similar to our solution. It seems that your Token Classification model is much better than ours, but we jumped due to model diversity, good luck in the next Feedback Prize competition!
\n @vadimirtlach for answering my stupid questions it helped me a lot\nInteresting ππ. Where did I answer your questions? In the Sanyam podcast?
", "votes": 1, "replies": [ { "id": 1913988, "author_name": "Arvind Devarkonda", "author_url": "", "post_date": "2022-08-25T16:12:47.860000", "content": "1.i not used token classification approach , i used normal seq classification approach (discourse_text + other_discourse_text_from_same_essay)
\n2.π Yes, on Sanyam's Podcast and in last uspptpm comp and in nbme too
", "votes": 1, "replies": [] } ] } ], "comment_count": 4, "messages": [ { "id": 1911818, "postDate": "2022-08-24T10:33:30.847Z", "content": "Thanks to kaggle and competition host.
\nIn last competition (USPPTPM) , i saw that the way we give data input to the transformers matters a lot so in this competition i was decided to experiment on that, in start i started with
\ndeberta-v3-base | 1024
\nI used data this way -> discourse_text + essay this gave me score -> 0.72(on LB)
deberta-v3-large | 1024 (with 5fold )
\nUsing oof ensembling this gave me 0.627(on LB) score data input this way -> discourse_text + essay
deberta-v3-large | 1024 (with 5fold ) + CLS embedding
\nIn this i not used essay instead i created essay by joining discourse text of same essay_id and added tags in the start of discourse text this way
\n[CLS][LEAD] discourse_text [LEAD][END] [POSITION] discourse_text [POSITION] β¦. [CLAIM] discourse_text [CLAIM]
\nthis gave me 0.61(on LB)
deberta-v3-large | 1024 (with 5fold ) + weighted layer pooling
\nIn this i replace cls embedding to weighted layer pooling and got 0.600 on lb
My final model was oof ensemble(just taking avg.) of 4th method. In the end am happy that i tried this instead of ensembling 3-4 model using discourse_text + essay this data.
\n**bad decisions i taken during competition **
\n**lesson learned: **
\n1.must not assume till you experiment
\n2.if something not worked on some model so it is not neccesery that it won't work with different model
\n3.if some one said in dissussion that this is not working don't trust i try and observe is that working or not
In last i am happy that i got bronze, Because last feedback competition was my first nlp competition and In that competition i just submitted sir @cdeotte 's notebook as it is , i was not able to understand that kernel(it was really nice kernel, i was dumb) then participated in NBME and USPPTPM and learned lot and got bronze in this competiton ik this is too much slow, in nbme and in this comp my team mates cheated on me still i learned a lot of things.
\nThanks to @abhishek and @SanyamBhutani for youtube videos and @vadimirtlach for answering my stupid questions it helped me a lot
", "rawMarkdown": "Thanks to kaggle and competition host.\n\nIn last competition (USPPTPM) , i saw that the way we give data input to the transformers matters a lot so in this competition i was decided to experiment on that, in start i started with\n\n1. deberta-v3-base | 1024\n\n I used data this way -> discourse_text + essay this gave me score -> 0.72(on LB)\n\n2. deberta-v3-large | 1024 (with 5fold )\n\n Using oof ensembling this gave me 0.627(on LB) score data input this way -> discourse_text + essay\n\n3. deberta-v3-large | 1024 (with 5fold ) + CLS embedding\n \n In this i not used essay instead i created essay by joining discourse text of same essay_id and added tags in the start of discourse text this way\n\n [CLS][LEAD] discourse_text [LEAD][END] [POSITION] discourse_text [POSITION] .... [CLAIM] discourse_text [CLAIM]\nthis gave me 0.61(on LB)\n\n4. deberta-v3-large | 1024 (with 5fold ) + weighted layer pooling\n\n In this i replace cls embedding to weighted layer pooling and got 0.600 on lb \n\n\nMy final model was oof ensemble(just taking avg.) of 4th method. In the end am happy that i tried this instead of ensembling 3-4 model using discourse_text + essay this data. \n\n\n**bad decisions i taken during competition **\n1. Instead of experimenting different things i already assumed that this will not worked \n2. not tried mlm pretraining on deberta because it not gave nice results on roberta i assumed this will not give good result on deberta too.\n3. not tried different models because during training longformers, gradients was exploded.\n\n**lesson learned: **\n1.must not assume till you experiment\n2.if something not worked on some model so it is not neccesery that it won't work with different model\n3.if some one said in dissussion that this is not working don't trust i try and observe is that working or not\n\n\nIn last i am happy that i got bronze, Because last feedback competition was my first nlp competition and In that competition i just submitted sir @cdeotte 's notebook as it is , i was not able to understand that kernel(it was really nice kernel, i was dumb) then participated in NBME and USPPTPM and learned lot and got bronze in this competiton ik this is too much slow, in nbme and in this comp my team mates cheated on me still i learned a lot of things. \n\nThanks to @abhishek and @SanyamBhutani for youtube videos and @vadimirtlach for answering my stupid questions it helped me a lot", "votes": 13 }, { "id": 1913696, "postDate": "2022-08-25T12:59:32.233Z", "content": "\n\nIn this i replace cls embedding to weighted layer pooling and got 0.600 on lb
\n
May you do pooling of layer's representations (weighted average) and then took CLS embedding, because I watched the implementation in this great notebook and the author did Weighted Layer Pooling and then CLS pooling. Is it, @arvinddevarkonda?
", "rawMarkdown": "> In this i replace cls embedding to weighted layer pooling and got 0.600 on lb\n\nMay you do pooling of layer's representations (weighted average) and then took CLS embedding, because I watched the implementation in [this great notebook](https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently) and the author did Weighted Layer Pooling and then CLS pooling. Is it, @arvinddevarkonda?", "votes": 1, "replies": [ { "id": 1914001, "postDate": "2022-08-25T16:23:36.170Z", "content": "yes. I used only weighted layer pooling from that notebook,i tried bilstm pooling layer with max_pooling it not worked so i assumed that using more than one pooling layer will not work.(here i made mistake)
", "rawMarkdown": "yes. I used only weighted layer pooling from that notebook,i tried bilstm pooling layer with max_pooling it not worked so i assumed that using more than one pooling layer will not work.(here i made mistake)", "votes": 1 } ] }, { "id": 1913412, "postDate": "2022-08-25T10:11:04.743Z", "content": "Good solution, it is similar to our solution. It seems that your Token Classification model is much better than ours, but we jumped due to model diversity, good luck in the next Feedback Prize competition!
\n @vadimirtlach for answering my stupid questions it helped me a lot\nInteresting ππ. Where did I answer your questions? In the Sanyam podcast?
", "rawMarkdown": "Good solution, it is similar to our solution. It seems that your Token Classification model is much better than ours, but we jumped due to model diversity, good luck in the next Feedback Prize competition!\n\n```\n @vadimirtlach for answering my stupid questions it helped me a lot\n```\n\nInteresting ππ. Where did I answer your questions? In the Sanyam podcast? ", "votes": 1, "replies": [ { "id": 1913988, "postDate": "2022-08-25T16:12:47.860Z", "content": "1.i not used token classification approach , i used normal seq classification approach (discourse_text + other_discourse_text_from_same_essay)
\n2.π Yes, on Sanyam's Podcast and in last uspptpm comp and in nbme too
", "rawMarkdown": "1.i not used token classification approach , i used normal seq classification approach (discourse_text + other_discourse_text_from_same_essay)\n\n2.π Yes, on Sanyam's Podcast and in last uspptpm comp and in nbme too", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:55:21Z", "competition_id": 35308, "competition_slug": "feedback-prize-effectiveness", "team_id": 8872008, "solution_url": "/competitions/feedback-prize-effectiveness/writeups/team-idli-146-th-palce-solution" }