💡 Leaderboard: Top 68% (1,782 out of 2,654)
\nThe task was to score an essay in 6 targets: cohesion, syntax, vocabulary, phraseology, grammar, and conventions. No other labels are provided. But we can use data from past Feedback competitions. That includes argument segmentation (FB 1.0) and classification (FB 2.0).
\nI trained a multitasking model using RoBERTa to do three tasks from all FB competitions. That includes:
\nHere is the link to the notebook, and here is its performance:
\nAll the scores reported below are private scores. I also run some other experiments before this multitasking model. So I’ll document the overall concepts I learned below.
\nRoBERTa followed by Conv helps a bit
\nThe Conv layer on top of RoBERTa helps boost the performance by (0.9% better).
\nDistillRoBERTa is not too bad
\nFine-tuning DistillRoBERTa can achieve a similar performance to RoBERTa for significantly fewer parameters and inference latency (0.1% worse).
\nWord-Level embedding is essential
\nWe try using SentenceTransformer or fixed RoBERTa to get embedding for the whole essay without being able to update the encoder’s weight. The performance jumps to more than 0.5.
\nFine-tuning on previous tasks helps
\nI try to fine-tune RoBERTa on FB1.0 which is to detect the boundary of arguments in the essay. It improves by 0.5%.
\nI didn’t get to fine-tune on the FB2.0 though.
\nUse HyperParameter
\nI started using hyperparameter tuning instead of sitting and tweaking each parameter. It saves a lot of time as it would just run automatically and return the result at the end. Here are some of the experiment results:
\nHowever, I was not able to do this hyperparameter search all the way. For some experiments, I end up doing them manually. For example, I was testing by using the top 4 layers of Transformers or using 4 layers but evenly selected from different layers. I did hyper-parameter tuning for each one, but not all together. I should have just thrown it in hyperparameter and just look at the result all in one go.
\nMultitasking is about balancing
\nConceptually, multitasking makes the encoder more aware of all the tasks and hopefully it trains to use all those knowledge and improve the score altogether. As it turns out, setting the right trade-off parameter can be tricky. Often, it tips off toward one loss function and that just worsens the others. I’m still searching for the right way to balance these objectives. The current result favors the main scoring task and it makes discourse detection way worse. But yes, for this task, the multitasking model was able to achieve higher performance than the RoBERTa one.
\n💡 Leaderboard: Top 68% (1,782 out of 2,654)
\nThe task was to score an essay in 6 targets: cohesion, syntax, vocabulary, phraseology, grammar, and conventions. No other labels are provided. But we can use data from past Feedback competitions. That includes argument segmentation (FB 1.0) and classification (FB 2.0).
\nI trained a multitasking model using RoBERTa to do three tasks from all FB competitions. That includes:
\nHere is the link to the notebook, and here is its performance:
\nAll the scores reported below are private scores. I also run some other experiments before this multitasking model. So I’ll document the overall concepts I learned below.
\nRoBERTa followed by Conv helps a bit
\nThe Conv layer on top of RoBERTa helps boost the performance by (0.9% better).
\nDistillRoBERTa is not too bad
\nFine-tuning DistillRoBERTa can achieve a similar performance to RoBERTa for significantly fewer parameters and inference latency (0.1% worse).
\nWord-Level embedding is essential
\nWe try using SentenceTransformer or fixed RoBERTa to get embedding for the whole essay without being able to update the encoder’s weight. The performance jumps to more than 0.5.
\nFine-tuning on previous tasks helps
\nI try to fine-tune RoBERTa on FB1.0 which is to detect the boundary of arguments in the essay. It improves by 0.5%.
\nI didn’t get to fine-tune on the FB2.0 though.
\nUse HyperParameter
\nI started using hyperparameter tuning instead of sitting and tweaking each parameter. It saves a lot of time as it would just run automatically and return the result at the end. Here are some of the experiment results:
\nHowever, I was not able to do this hyperparameter search all the way. For some experiments, I end up doing them manually. For example, I was testing by using the top 4 layers of Transformers or using 4 layers but evenly selected from different layers. I did hyper-parameter tuning for each one, but not all together. I should have just thrown it in hyperparameter and just look at the result all in one go.
\nMultitasking is about balancing
\nConceptually, multitasking makes the encoder more aware of all the tasks and hopefully it trains to use all those knowledge and improve the score altogether. As it turns out, setting the right trade-off parameter can be tricky. Often, it tips off toward one loss function and that just worsens the others. I’m still searching for the right way to balance these objectives. The current result favors the main scoring task and it makes discourse detection way worse. But yes, for this task, the multitasking model was able to achieve higher performance than the RoBERTa one.
\n