# 146 th palce solution

Thanks to kaggle and competition host.
In last competition (USPPTPM) , i saw that the way we give data input to the transformers matters a lot so in this competition  i was decided to experiment on that, in start i started with
deberta-v3-base | 1024
I used data this way -> discourse_text + essay this gave me score -> 0.72(on LB)
deberta-v3-large | 1024 (with 5fold )
Using oof ensembling this gave me 0.627(on LB) score data input this way -> discourse_text + essay
deberta-v3-large | 1024 (with 5fold ) + CLS  embedding
In this i not used essay instead i created essay by joining discourse text of  same essay_id and added tags in the start of discourse text this way
[CLS][LEAD] discourse_text [LEAD][END] [POSITION] discourse_text [POSITION] …. [CLAIM] discourse_text     [CLAIM]
this gave me 0.61(on LB)
deberta-v3-large | 1024 (with 5fold ) + weighted layer pooling
In this i replace cls embedding to weighted layer pooling and got 0.600 on lb
My final model was oof ensemble(just taking avg.) of 4th method. In the end am happy that i tried this instead of ensembling 3-4 model using discourse_text + essay this data.
**bad decisions i taken during competition **
Instead of experimenting different things i already assumed that this  will not worked
not tried mlm pretraining on deberta because it not gave nice results on roberta i assumed this  will not give good result on deberta too.
not tried different models because during training longformers,  gradients was exploded.
**lesson learned: **
1.must not assume till you experiment
2.if something not worked on some model so it is not neccesery that it won't work with different model
3.if some one said in dissussion that this is not working don't trust i try and observe is that working or not
In last i am happy that i got bronze, Because last feedback competition was my first nlp competition and In that competition i just submitted sir
[@cdeotte](https://www.kaggle.com/cdeotte)
's notebook as it is , i was not able to understand that kernel(it was really nice kernel, i was dumb) then participated in NBME and USPPTPM and learned lot and got bronze in this competiton ik this is too much slow, in nbme and in this comp my team mates cheated on me still i learned a lot of things.
Thanks to
[@abhishek](https://www.kaggle.com/abhishek)
and
[@SanyamBhutani](https://www.kaggle.com/SanyamBhutani)
for youtube videos and
[@vadimirtlach](https://www.kaggle.com/vadimirtlach)
for answering my stupid questions it helped me a lot