Congrats all and thank you Georgia State University, Kaggle and all the organizers for hosting such an amazing competition. We were unfortunate to finish just short of a prize in both tracks of the competition, but will take the learnings with us!
\nOur solution is based on deberta-v3, mlm, pseudo labeling and stacking with LGB. We found the combination of a number of NLP improvements added up to a solid score.
\nThe core of our solution was to combine, in order, all discourse ids, and then start and finish them with discourse typeids and special tokens.
\nA cls_position was used as start or end of the discourse. In python, see below,
input_ids = []\nessay_df = = trndf.loc[essay_id]\nfor i in range(len(essay_df)):\n text = f'{discourse_types[i]}: {discourse_texts[i]} {self.tokenizer.sep_token} :{discourse_types[i]}'\n i_ids = self.tokenizer.encode(text, add_special_tokens=False)\n input_ids += i_ids\n cls_pos += [0] * (len(i_ids) - 1) + [1]\nAn example of the output can be seen below.
\nLead: Computers can do many things for us. The idea that a computer can read your emotions is amazing [SEP] :Lead Position: I do not belive it is true. I will believe it when I see it. [SEP] :Position Evidence: In paragraph 8 it says \" But in a false smile, the mouth is stretched sideways ising the zygomatic major and different muscle, the risorius.\" however this may be true for most people; there has to be someone out there in our world of 7 billion that smiles there smiles with their zygomatic muscle or their risorius muscle [SEP] :Evidence Claim: Everyone has diffrent emotions and everyone shows them diffrently [SEP] :Claim Counterclaim: The muscles in our face does say a lot about the emotions we are feeling. [SEP] :Counterclaim Concluding Statement: This is why I believe computures can not read your emotions [SEP] :Concluding Statement\nAuxiliary labels such as ranking (Ineffective -> Adequate -> Effective) and essay topic cluster were also used.
\nWe also added the essay text, which was not in annotated discourses to the end, and sometimes positioned in order between discourses.
\nReplacing line breaks with special tokens and cleaning text in some models was found to increase diversity significantly in the overall blend.
We used three different model pipelines, with some smaller changes in data preprocessing, along with different auxiliary labels and model heads. We only used deberta-v3 as the backbone. Other backbones did not help our blend in CV.
It is important to remove deberta hidden layer dropout on all models. We also found it helpful to pretrain the weights on the earlier feedback competition essays for around 20 epochs (low lr, large batchsize).
\nAll models were trained with 2-3 epochs, with backbone set to lr ~2e-5 and the model head was set to ~1e-4.
We created meta pseudo labels on the essays from the first feedback competition (excl. current comp essays). In-fold predictions from all models were used to create an averaged in-fold prediction, which can be used as a leak-free pseudo label.
\nEach model was then retrained on the hard labels from current competition and the soft pseudo labels. Hard labels from current competition were upweighted in the loss function.
All level2 model predictions were averaged and used in a lightgbm stacking model. This was trained at discourse level, along with meta features from the respective essay, such as word count, sentence count, position in essay-topic and lead & lag features.
\nNo special postprocessing was applied after that.
Congrats! your work is very impressive!
\nGreat writeup btw!
Auxiliary labels such as ranking (Ineffective -> Adequate -> Effective)
\nHow did you make rmseloss to rank them?
Create a map, like {'Ineffective': 0, 'Adequate': 1, 'Effective': 2} for the target and now you have one continuous label instead of categorical labels.
Congrats all and thank you Georgia State University, Kaggle and all the organizers for hosting such an amazing competition. We were unfortunate to finish just short of a prize in both tracks of the competition, but will take the learnings with us!
\nOur solution is based on deberta-v3, mlm, pseudo labeling and stacking with LGB. We found the combination of a number of NLP improvements added up to a solid score.
\nThe core of our solution was to combine, in order, all discourse ids, and then start and finish them with discourse typeids and special tokens.
\nA cls_position was used as start or end of the discourse. In python, see below,
input_ids = []\nessay_df = = trndf.loc[essay_id]\nfor i in range(len(essay_df)):\n text = f'{discourse_types[i]}: {discourse_texts[i]} {self.tokenizer.sep_token} :{discourse_types[i]}'\n i_ids = self.tokenizer.encode(text, add_special_tokens=False)\n input_ids += i_ids\n cls_pos += [0] * (len(i_ids) - 1) + [1]\nAn example of the output can be seen below.
\nLead: Computers can do many things for us. The idea that a computer can read your emotions is amazing [SEP] :Lead Position: I do not belive it is true. I will believe it when I see it. [SEP] :Position Evidence: In paragraph 8 it says \" But in a false smile, the mouth is stretched sideways ising the zygomatic major and different muscle, the risorius.\" however this may be true for most people; there has to be someone out there in our world of 7 billion that smiles there smiles with their zygomatic muscle or their risorius muscle [SEP] :Evidence Claim: Everyone has diffrent emotions and everyone shows them diffrently [SEP] :Claim Counterclaim: The muscles in our face does say a lot about the emotions we are feeling. [SEP] :Counterclaim Concluding Statement: This is why I believe computures can not read your emotions [SEP] :Concluding Statement\nAuxiliary labels such as ranking (Ineffective -> Adequate -> Effective) and essay topic cluster were also used.
\nWe also added the essay text, which was not in annotated discourses to the end, and sometimes positioned in order between discourses.
\nReplacing line breaks with special tokens and cleaning text in some models was found to increase diversity significantly in the overall blend.
We used three different model pipelines, with some smaller changes in data preprocessing, along with different auxiliary labels and model heads. We only used deberta-v3 as the backbone. Other backbones did not help our blend in CV.
It is important to remove deberta hidden layer dropout on all models. We also found it helpful to pretrain the weights on the earlier feedback competition essays for around 20 epochs (low lr, large batchsize).
\nAll models were trained with 2-3 epochs, with backbone set to lr ~2e-5 and the model head was set to ~1e-4.
We created meta pseudo labels on the essays from the first feedback competition (excl. current comp essays). In-fold predictions from all models were used to create an averaged in-fold prediction, which can be used as a leak-free pseudo label.
\nEach model was then retrained on the hard labels from current competition and the soft pseudo labels. Hard labels from current competition were upweighted in the loss function.
All level2 model predictions were averaged and used in a lightgbm stacking model. This was trained at discourse level, along with meta features from the respective essay, such as word count, sentence count, position in essay-topic and lead & lag features.
\nNo special postprocessing was applied after that.
Congrats! your work is very impressive!
\nGreat writeup btw!
Auxiliary labels such as ranking (Ineffective -> Adequate -> Effective)
\nHow did you make rmseloss to rank them?
Create a map, like {'Ineffective': 0, 'Adequate': 1, 'Effective': 2} for the target and now you have one continuous label instead of categorical labels.