# 1,782nd place - Multitasking solution

💡 Leaderboard: Top 68% (1,782 out of 2,654)
Task
The task was to score an essay in
6 targets
: cohesion, syntax, vocabulary, phraseology, grammar, and conventions. No other labels are provided. But we can use data from past Feedback competitions. That includes argument segmentation (FB 1.0) and classification (FB 2.0).
Final Solution
I trained a multitasking model using RoBERTa to do three tasks from all FB competitions. That includes:
[FB1.0](https://www.kaggle.com/competitions/feedback-prize-2021/data)
: Detecting discourse boundary. This is like a NER task. The labels are lead, position, evidence, claim, concluding statement, counterclaim, and rebuttal.
[FB2.0](https://www.kaggle.com/competitions/feedback-prize-effectiveness/data)
: Classifying each discourse as being effective, ineffective, and adequate.
[FB3.0](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/overview)
: Overall essay scoring.
Here is the link to the
[notebook](https://www.kaggle.com/code/vitouphy/fb3-multi-task-prediction)
, and here is its performance:
Public: 0.457696
Private: 0.458945
What I learned
All the scores reported below are private scores. I also run some other experiments before this multitasking model. So I’ll document the overall concepts I learned below.
RoBERTa followed by Conv helps a bit
The Conv layer on top of RoBERTa helps boost the performance by
(0.9% better)
.
RoBERTa: 0.465965
RoBERTa-Conv: 0.461717
DistillRoBERTa is not too bad
Fine-tuning DistillRoBERTa can achieve a similar performance to RoBERTa for significantly fewer parameters and inference latency
(0.1% worse)
.
RoBERTa-Conv: 0.461717
DistillRoBERTaConv: 0.462217
Word-Level embedding is essential
We try using SentenceTransformer or fixed RoBERTa to get embedding for the whole essay without being able to update the encoder’s weight. The performance jumps to more than 0.5.
RoBERTa: 0.465965
SentenceTransformer: 0.52097
Fixed pretrained-NER-RoBERTa: 0.515841
Fine-tuning on previous tasks helps
I try to fine-tune RoBERTa on FB1.0 which is to detect the boundary of arguments in the essay. It improves by
0.5%
.
(Pretrained NER)-RoBERTa-Conv: 0.459103
RoBERTa-Conv: 0.461717
I didn’t get to fine-tune on the FB2.0 though.
Use HyperParameter
I started using hyperparameter tuning instead of sitting and tweaking each parameter. It saves a lot of time as it would just run automatically and return the result at the end. Here are some of the experiment results:
RoBERTa-Conv:
Plain: 0.472317
HParams: 0.461717
(+2.2%)
RoBERTa:
Plain: 0.487746
HParams: 0.465965
(+4.4%)
However, I was not able to do this hyperparameter search all the way. For some experiments, I end up doing them manually. For example, I was testing by using the top 4 layers of Transformers or using 4 layers but evenly selected from different layers. I did hyper-parameter tuning for each one, but not all together. I should have just thrown it in hyperparameter and just look at the result all in one go.
Multitasking is about balancing
Conceptually, multitasking makes the encoder more aware of all the tasks and hopefully it trains to use all those knowledge and improve the score altogether. As it turns out, setting the right trade-off parameter can be tricky. Often, it tips off toward one loss function and that just worsens the others. I’m still searching for the right way to balance these objectives. The current result favors the main scoring task and it makes discourse detection way worse. But yes, for this task, the multitasking model was able to achieve higher performance than the RoBERTa one.
MultiTask (3 tasks): 0.458945
RoBERTa-Conv: 0.461717
(+0.6%)
RoBERTa: 0.465965
(+1.5%)