{ "id": 369369, "title": "2nd solution (back-translation & rank-loss)", "content": "

This is a fun game, thanks for the organizer and all pepole who share here.

\n\n

Single model performance (5folds + 1 full train):

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
modelcvlbpb
base(dev3-large maxlen1280)4514442013438818
base+rank_loss4505438912437175
base+rank_loss+trans-nl pretrain4498440225435711
base+rank_loss+feedback2 pretrain4488438583435814
base+feedback1 pseudo train only4469438601436144
base+rank_loss+feedback1 pseudo pretrain4497440576435144
base+feedback1 pseudo&feedback3 train4468438063434809
base+rank_loss+feedback1 pseudo&feedback3 train4456438084434726
\n

Model with pretrain or pseudo or back-trans(model with more backgound knowlege) work better for both CV/PB and some models for CV/LB/PB.And these models tend to have better cv performance on vocabulary.
\nRank loss always help.

\n\n
    \n
  1. Let bert deal with the target relation
    \n\"[CLS] cohesion syntax vocabulary phraseology grammar conventions text [SEP]“
    \nEach target output emb with 1 unique fc added to predict target value.
    2. \n
  2. Add pearson loss
    \nWe could see bert tend to predict all target with similar value, in order to predict well for hard target, I found adding pearson loss help a lot. But it will hurt for easy target performance, I did not find good method to deal, ensembling with person loss rate 0.1 and 0 is what I used.
    \nAlso with rank_loss_rate = 0.1 improve single model cv from 4514 to 4505. Below show add rank loss (name with -crank) help a lot for hard targets(score/max calc for target index np.argmax(label - label.mean())). Here also has improve space, I think maybe we can better ensemble using some specific features and do regression on OOF.
    \n\"\"
    3. \n
  3. Use back translation for pretrain
    \nI used a lot of back translation for ensembling. They are [nl, fr, de, pt, af, cn, ru, fi, sv, ja, ko, el, hr, cy]
    \nNotice back translation data is only used for pretrain, so 2 steps training, the second step is the same as other models.
    \nNo back translation models:
    \n('score', 0.44672287515754533),
    \n('score/vocabulary', 0.4109792609030708),
    \nAdd back translation models, with weight=np.array([1,1,10,1,0.25,0.25], float) * 0.8:
    \n('score', 0.4453322618856002),
    \n('score/vocabulary', 0.4086610628919421),
    \nBack translation models only:
    \n('score', 0.44640808445875324),
    \n('score/vocabulary', 0.408889105739513),
    \nVocabulary improve a lot ! Actually for all backtrans models vocabulary improve so it means backtrans help on learning more vocabulary info!
    \nSingle model (see image above with name .trans-nl) cv improve a lot 4505 to 4497 and also PB gain.
    4. \n
  4. Use feedback2 data for pretrain
    \nJust train feedback2 listwise model (remove any discourse type info) as pretrain model. This will make model more stable and help especially for some targets like convetions and vocabulary.
    \nTODO: I did not use ensemble and pseudo for feedback2 data pretrain, simply using 1 single full train model on fddeback2 data only, so here might have improvement space.
    \nSingle model (see image above with name .ft2) cv improve a lot 4505 to 4488 and also LB/PB improve.
    \nWhy use feedback2? I did not investigate but I think discourse effectiveness has relation with our 6 targets.
    5. \n
  5. Pseudo
    \nPseudo works, pseudo+optuna is much better then no pseudo+optuna, both LB and cv and PB 433811->43363.
    \nPseudo also helped produce best PB(435144) single model.
    \nTODO: I did not try (Mean of Train PL / Actual Labelsas) mentioned in 1st solution it might give boost.
    6. \n
  6. Tune for each target
    \nFor example back translation help a lot for vocabulary but hurt conventions as show on cv image above.
    7. \n
\n", "comments": [ { "id": 2350698, "author_name": "Athar Sayed", "author_url": "", "post_date": "2023-07-19T12:50:45.817000", "content": "

What does awp training means ?

", "votes": 1, "replies": [] }, { "id": 2052091, "author_name": "ReXTeRR55", "author_url": "", "post_date": "2022-12-01T22:54:05.270000", "content": "

Thats a great info

", "votes": 1, "replies": [] }, { "id": 2051505, "author_name": "rw-nue", "author_url": "", "post_date": "2022-12-01T13:23:56.817000", "content": "

Hello. Congrats! and thank you for sharing your solution.

\n

Can I ask you about \"back translation for pretrain\".

\n

1) How did you produced pseudo labeled records.

\n
\n

[nl, fr, de, pt, af, cn, ru, fi, sv, ja, ko, el, hr, cy]

\n
\n

Did you used all of these languages?
\nAnd, did you used all of those records? ( like 3911 x languages pseudo labeled records), or did you select some part of them for pre-training.

\n

2) About weight parameters.

\n
\n

weight=np.array([1,1,10,1,0.25,0.25], float) * 0.8

\n
\n

How did you decide weights value for each targets?
\nand what is *0.8 parameter for?

\n

Thank you.

", "votes": 1, "replies": [ { "id": 2051521, "author_name": "gezi", "author_url": "", "post_date": "2022-12-01T13:35:11.663000", "content": "

for 1) yes I used all these languages, actually more, but some other languages may not improve my ensemble results so not included finally. I transelate and translate back for each language to get a new train dataset, like for nl get the nl-dataset, train on nl-dataset only to get nl-model, then train on original train dataset with model load from the pretrained model nl-model.
\nI tired mix different language datasets to do the first step seems not work well.
\nfor 2), actually nl-model it's self not perform well, but it works well as a pretrained model. I used wandb to log cv score for each target and found, by using back-translation some targets improve a lot (vocabulary) and some others decrease a lot (conventions).
\nAnd this is a common fact, not by chance, I found all language got same conclution, help vocabulary hurt conventions.
\nSo I tried to manually increase model weights for vocabulary and decrease weight on conventions and it show increase on both CV and LB.

", "votes": 3, "replies": [] }, { "id": 2054845, "author_name": "rw-nue", "author_url": "", "post_date": "2022-12-04T13:39:38.593000", "content": "

Thank you for detailed answer. I learned a lot from your solution!

", "votes": 1, "replies": [] } ] }, { "id": 2050331, "author_name": "Vaidehi Savaliya", "author_url": "", "post_date": "2022-11-30T16:58:25.130000", "content": "

Thanks for sharing and congratulations….

", "votes": 1, "replies": [] }, { "id": 2050274, "author_name": "Mark Baushenko", "author_url": "", "post_date": "2022-11-30T16:11:18.020000", "content": "

we too use back translation. Cool shake up, Congrats!

", "votes": 1, "replies": [] }, { "id": 2050130, "author_name": "CroDoc", "author_url": "", "post_date": "2022-11-30T14:40:29.263000", "content": "

Nice jump! I was wondering where you were :P

", "votes": 1, "replies": [ { "id": 2050682, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T23:22:02.727000", "content": "

Thanks @crodoc, congratulation to you also! Interesting game, if I added svr to my ensemble maybe I could be seen on LB😃, just wondering.

", "votes": 1, "replies": [] }, { "id": 2054762, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2022-12-04T12:50:33.273000", "content": "

Gezi is always on top of private LB, regardless of where on private.

\n

Qishen : CV :: Gezi : NLP 🙏

", "votes": 1, "replies": [] } ] }, { "id": 2050125, "author_name": "marc remy", "author_url": "", "post_date": "2022-11-30T14:36:41.853000", "content": "

Great work!

", "votes": 1, "replies": [] }, { "id": 2049818, "author_name": "CPMP", "author_url": "", "post_date": "2022-11-30T11:06:42.087000", "content": "

Interesting that you use rank loss. Congrats on the result!

", "votes": 1, "replies": [] }, { "id": 2049749, "author_name": "Jianw Chen", "author_url": "", "post_date": "2022-11-30T10:04:25.243000", "content": "

Great work and Congratulations! Learn a lot from your solution. Thanks for sharing.

", "votes": 1, "replies": [] }, { "id": 2049424, "author_name": "Feng Qilong", "author_url": "", "post_date": "2022-11-30T06:19:24.060000", "content": "

Congratulations! I do have a question tho, if the models are pretrained with fb2 data, will there be any form of leakage? Since some of the essays from fb2 might be similar to the validation data of different folds. Also, will there be any form of leakage if just pretrain the models with the training data and the fine tune?

", "votes": 1, "replies": [ { "id": 2049797, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T10:49:46.677000", "content": "

There are small number of overlap docids, but it does not affect so much, basicly for safe cv you could remove them.
\nFor me I did not remove them for fb2 pretrain, but for pseudo using fb1 I removed duplicate doc ids.

", "votes": 1, "replies": [] } ] }, { "id": 2049238, "author_name": "dingyan", "author_url": "", "post_date": "2022-11-30T02:24:05.667000", "content": "

Congrats and nice augmentation! Compared with image augmentation, it is hard to do augmentation in text. Could you explain more about pearson loss?

", "votes": 1, "replies": [ { "id": 2049248, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T02:33:54.850000", "content": "

You can refer to my code https://www.kaggle.com/datasets/goldenlock/feedbacken , torch/loss.py FLAGS.rank_loss_rate

", "votes": 1, "replies": [] }, { "id": 2049309, "author_name": "dingyan", "author_url": "", "post_date": "2022-11-30T03:33:44.133000", "content": "

Thank you.

", "votes": 1, "replies": [] } ] }, { "id": 2049169, "author_name": "Xiang Huang", "author_url": "", "post_date": "2022-11-30T01:24:19.980000", "content": "

Nice solution, congrats for the win!

", "votes": 1, "replies": [ { "id": 2049174, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T01:31:03.023000", "content": "

Thanks! @xianghuang666

", "votes": 1, "replies": [] } ] }, { "id": 2049168, "author_name": "Jackson You", "author_url": "", "post_date": "2022-11-30T01:23:00.913000", "content": "

Congratulations @goldenlock. Your idea is so great.

\n

Could you explain the back translation more detail?

", "votes": 1, "replies": [ { "id": 2049171, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T01:25:03.493000", "content": "

Use gooltrans to translate train text to cn text for example and translate back to english, then using the new train data for pretrain.
\nfrom googletrans import Translator
\ntrans = Translator()

", "votes": 1, "replies": [] }, { "id": 2049175, "author_name": "Jackson You", "author_url": "", "post_date": "2022-11-30T01:31:12.670000", "content": "

Nice augumentation. Hence, does your final submission you chose include models with pseudo data? I found that using pseudo data will hurt private score.

", "votes": 0, "replies": [] }, { "id": 2049177, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T01:32:04.313000", "content": "

Best choosen one with pseudo and best score not, I explain more in my post, I think pseudo should work but not improve so much.

", "votes": 0, "replies": [] } ] }, { "id": 2049367, "author_name": "Nick Liu", "author_url": "", "post_date": "2022-11-30T04:45:08.030000", "content": "

Thanks for sharing, back translation is cool idea. congrats!

", "votes": 2, "replies": [] }, { "id": 2049269, "author_name": "E-Max AI", "author_url": "", "post_date": "2022-11-30T03:02:37.287000", "content": "

Aged ginger is more pungent

", "votes": 1, "replies": [] }, { "id": 2903092, "author_name": "Adithya", "author_url": "", "post_date": "2024-07-03T15:48:22.173000", "content": "

So I have a doubt, doesn't deberta model have max len like bert (512 tokens). Like deberta takes how many tokens u specify.

", "votes": 0, "replies": [] }, { "id": 2051055, "author_name": "future_zz", "author_url": "", "post_date": "2022-12-01T07:17:05.557000", "content": "

@goldenlock hi, congrats and thanks for the sharing!

\n

would you like to share some idea about how to select submission for the private testing?

\n

from experiment we can know, some submissions score high in public leaderboard but low in private, so what should we do? how do we choose final submission for private testing？

\n

for example, i have 2 submission, one of them is ensemble whose cv is 0.4503 but public score worse, the other one cv is 0.4511 but public score better…. i am confused

", "votes": 0, "replies": [ { "id": 2051296, "author_name": "gezi", "author_url": "", "post_date": "2022-12-01T10:26:50.487000", "content": "

For this competion as LB dataset is small, you should trust your cv, you can see from my single model results table in my post, and you will find CV and PB is more aligned then CV and LB.
\nOne thing important is you have to make sure your CV is correct, first no leak like when using pseduo.
\nAnother thing is you should take care not to overfit your CV a lot. For example I got cv 4494 using optuna tune model weights for all OOF then my cv is over optimized, the better way will be you do optuna on OOF fold 1-3 and test on fold 4, by doing this you can avoid overfit to cv.
\nAlso if you manually tune weight on the whole OOF you will face the same problem.

", "votes": 2, "replies": [] } ] }, { "id": 2050281, "author_name": "Muhammad Hilmi Asyrofi", "author_url": "", "post_date": "2022-11-30T16:14:16.470000", "content": "

Congrats on your victory @goldenlock !

\n

I'm curious, how do you choose the 3 final submissions as you have 4 convincing different approaches? what's the rationale behind?

", "votes": 0, "replies": [ { "id": 2050681, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T23:20:40.317000", "content": "

I choose by the best cv( over optimized by optuna tune on whole OOF), and they are also the best on LB.

", "votes": 0, "replies": [] } ] }, { "id": 2049218, "author_name": "IronLuu", "author_url": "", "post_date": "2022-11-30T02:03:55.483000", "content": "

Congrats! Thanks for sharing.
\nI found that deberta-v1-large and others with max_length=560 works just a little worse than deberta-v3s with max_length=1428.
\nWhat max_length did you choose?

", "votes": 0, "replies": [ { "id": 2049229, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T02:14:33.037000", "content": "

For deberta-v1 I tried 512 but not used.

", "votes": 0, "replies": [] } ] }, { "id": 2049194, "author_name": "KKY", "author_url": "", "post_date": "2022-11-30T01:43:05.740000", "content": "

Congrats, nice solution. Can you share which translation model do you used?

", "votes": 0, "replies": [ { "id": 2049197, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T01:44:00.967000", "content": "

I used google trans python api.

", "votes": 0, "replies": [] } ] }, { "id": 2049176, "author_name": "Clyde", "author_url": "", "post_date": "2022-11-30T01:31:33.503000", "content": "

Congratulations! May I ask how much does Back Translation help to improve the performance?

", "votes": 0, "replies": [ { "id": 2049185, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T01:37:07.323000", "content": "

Thanks @clyde7470 , updated my post, 4467 to 44533.

", "votes": 1, "replies": [] } ] }, { "id": 2049244, "author_name": "", "author_url": "", "post_date": "2022-11-30T02:30:19.453000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 38, "messages": [ { "id": 2049157, "postDate": "2022-11-30T01:08:08.287Z", "content": "

This is a fun game, thanks for the organizer and all pepole who share here.

\n\n

Single model performance (5folds + 1 full train):

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
modelcvlbpb
base(dev3-large maxlen1280)4514442013438818
base+rank_loss4505438912437175
base+rank_loss+trans-nl pretrain4498440225435711
base+rank_loss+feedback2 pretrain4488438583435814
base+feedback1 pseudo train only4469438601436144
base+rank_loss+feedback1 pseudo pretrain4497440576435144
base+feedback1 pseudo&feedback3 train4468438063434809
base+rank_loss+feedback1 pseudo&feedback3 train4456438084434726
\n

Model with pretrain or pseudo or back-trans(model with more backgound knowlege) work better for both CV/PB and some models for CV/LB/PB.And these models tend to have better cv performance on vocabulary.
\nRank loss always help.

\n\n
    \n
  1. Let bert deal with the target relation
    \n\"[CLS] cohesion syntax vocabulary phraseology grammar conventions text [SEP]“
    \nEach target output emb with 1 unique fc added to predict target value.
    2. \n
  2. Add pearson loss
    \nWe could see bert tend to predict all target with similar value, in order to predict well for hard target, I found adding pearson loss help a lot. But it will hurt for easy target performance, I did not find good method to deal, ensembling with person loss rate 0.1 and 0 is what I used.
    \nAlso with rank_loss_rate = 0.1 improve single model cv from 4514 to 4505. Below show add rank loss (name with -crank) help a lot for hard targets(score/max calc for target index np.argmax(label - label.mean())). Here also has improve space, I think maybe we can better ensemble using some specific features and do regression on OOF.
    \n\"\"
    3. \n
  3. Use back translation for pretrain
    \nI used a lot of back translation for ensembling. They are [nl, fr, de, pt, af, cn, ru, fi, sv, ja, ko, el, hr, cy]
    \nNotice back translation data is only used for pretrain, so 2 steps training, the second step is the same as other models.
    \nNo back translation models:
    \n('score', 0.44672287515754533),
    \n('score/vocabulary', 0.4109792609030708),
    \nAdd back translation models, with weight=np.array([1,1,10,1,0.25,0.25], float) * 0.8:
    \n('score', 0.4453322618856002),
    \n('score/vocabulary', 0.4086610628919421),
    \nBack translation models only:
    \n('score', 0.44640808445875324),
    \n('score/vocabulary', 0.408889105739513),
    \nVocabulary improve a lot ! Actually for all backtrans models vocabulary improve so it means backtrans help on learning more vocabulary info!
    \nSingle model (see image above with name .trans-nl) cv improve a lot 4505 to 4497 and also PB gain.
    4. \n
  4. Use feedback2 data for pretrain
    \nJust train feedback2 listwise model (remove any discourse type info) as pretrain model. This will make model more stable and help especially for some targets like convetions and vocabulary.
    \nTODO: I did not use ensemble and pseudo for feedback2 data pretrain, simply using 1 single full train model on fddeback2 data only, so here might have improvement space.
    \nSingle model (see image above with name .ft2) cv improve a lot 4505 to 4488 and also LB/PB improve.
    \nWhy use feedback2? I did not investigate but I think discourse effectiveness has relation with our 6 targets.
    5. \n
  5. Pseudo
    \nPseudo works, pseudo+optuna is much better then no pseudo+optuna, both LB and cv and PB 433811->43363.
    \nPseudo also helped produce best PB(435144) single model.
    \nTODO: I did not try (Mean of Train PL / Actual Labelsas) mentioned in 1st solution it might give boost.
    6. \n
  6. Tune for each target
    \nFor example back translation help a lot for vocabulary but hurt conventions as show on cv image above.
    7. \n
\n", "rawMarkdown": "This is a fun game, thanks for the organizer and all pepole who share here. \n- CV \nI used abiheshark's cv strategy with 5 folds and seed 42.\nMy pb best choosen one is with **pseudo train and optuna tune CV 44494** PB 43363\n(but this **cv is not accurate** and over optimized since optuna tune for all train OOF, I should have used OOF agian for optuna based CV, unfortuantely I trust LB here and it happen to improve my LB but hurt PB a little or just by chance) \nMy pb best one is without pseudo train and no optuna tune(hand tune rule by per model per target cv) **CV 44543** PB 433541\nBut after game I tested pseudo+hand tune rule the results is 43380 so hand rule is also not very stable and has chance to overfit as you tune weight on local OOF cv.\nTODO: Try Rige, lasso, hill clmb，Nelder-Mead, negative weight of model methods to improve ensemble results\n**Best single modelsingle model is cv around 449** without pseudo train, using backtrans pretrain or feedback2 pretrain. \n**Best single model with pseudo is cv 4469** , for late submission I find better one with cv **4456** pb **434726** see table below \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F42245%2Fd0f805045cca2060a9f0d146a97de7ee%2Fscore.png?generation=1669801234725714&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F42245%2F989409d2e8fd2082cce8c99dc9a49a07%2Fcv.jpg?generation=1669801251545911&alt=media)\nNotice above image, epoch n means first(n + 1) models mean. \n\nSingle model performance (5folds + 1 full train):\n| model | cv | lb | pb | \n| --- | --- | -- | -- | \n| base(dev3-large maxlen1280) | 4514 | 442013 | 438818 | \n| base+rank_loss | 4505 | 438912 | 437175 | \n| base+rank_loss+trans-nl pretrain | **4498** | 440225 | **435711** |\n| base+rank_loss+feedback2 pretrain | **4488** | **438583**| **435814** | \n| base+feedback1 pseudo train only | **4469** | **438601** | **436144** | \n| base+rank_loss+feedback1 pseudo pretrain | **4497** | 440576 | **435144** | \n| base+feedback1 pseudo&feedback3 train | **4468** | **438063** | **434809** | \n| base+rank_loss+feedback1 pseudo&feedback3 train | **4456** | **438084** | **434726** | \n\nModel with pretrain or pseudo or back-trans(**model with more backgound knowlege**) work better for both CV/PB and some models for CV/LB/PB.And these models tend to have better cv performance on vocabulary.\nRank loss always help.\n\n- Backbones\ndeberta-v3-large (lr 2e-5)\ndeberta-v3-base (lr 5e-5)\nOnly the two work, all others much worse then deberta-v3-large\nTODO: I should have add **SVR method ensemble**\nSVR single model reach CV around 450 so good enough to ensemble and it could introduce more knowlege of non-deberta backbones.\nI did not add it as I return to this game late and could not make it safe to add for infer, there are some OOM issues , local cv seems to improve like 445146 to 44496 without tune weight for each target, using this one https://www.kaggle.com/code/quangphm/lb-0-43-simple-ensemble-deberta-base-svr.\n- Train\nText maxlen 1280, batch size 8, 4 epochs, linear lr decay, awp training for the last 2 epochs \nFor back translation pretrain only 2 epochs train without awp. \n\n- Models\nI submit 23 models * 2, 23 full train model, 23 fold(0-4) models. Cost 2h20 min using T4 * 2 infer. \n\n- What works\n1. **Let bert deal with the target relation**\n\"[CLS] cohesion syntax vocabulary phraseology grammar conventions text [SEP]“ \nEach target output emb with 1 unique fc added to predict target value. \n2. **Add pearson loss**\nWe could see bert tend to **predict all target with similar value**, in order to predict well for **hard target**, I found adding pearson loss help a lot. But it will hurt for easy target performance, I did not find good method to deal, ensembling with person loss rate 0.1 and 0 is what I used. \nAlso with rank_loss_rate = 0.1 improve single model cv from 4514 to 4505. Below show add rank loss (name with -crank) help a lot for hard targets(score/max calc for target index np.argmax(label - label.mean())). Here also has improve space, I think maybe we can better ensemble using some specific features and do regression on OOF. \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F42245%2Fd83e3ae6f0a84eaaecff6bf1db7d6ccc%2Fminmax.png?generation=1669801062373059&alt=media)\n3. **Use back translation for pretrain**\nI used a lot of back translation for ensembling. They are [nl, fr, de, pt, af, cn, ru, fi, sv, ja, ko, el, hr, cy] \nNotice back translation data is only used for pretrain, so 2 steps training, the second step is the same as other models.\nNo back translation models: \n('score', 0.44672287515754533),\n ('score/vocabulary', 0.4109792609030708),\nAdd back translation models, with weight=np.array([1,1,10,1,0.25,0.25], float) * 0.8:\n('score', 0.4453322618856002),\n ('score/vocabulary', 0.4086610628919421),\nBack translation models only:\n('score', 0.44640808445875324),\n ('score/vocabulary', 0.408889105739513),\nVocabulary improve a lot ! Actually for all backtrans models vocabulary improve so it means backtrans help on learning more vocabulary info!\nSingle model (see image above with name .trans-nl) cv improve a lot 4505 to 4497 and also PB gain.\n4. **Use feedback2 data for pretrain**\nJust train feedback2 listwise model (remove any discourse type info) as pretrain model. This will make model more stable and help especially for some targets like convetions and vocabulary. \nTODO: I did not use ensemble and pseudo for feedback2 data pretrain, simply using 1 single full train model on fddeback2 data only, so here might have improvement space. \nSingle model (see image above with name .ft2) cv improve a lot 4505 to 4488 and also LB/PB improve.\nWhy use feedback2? I did not investigate but I think discourse effectiveness has relation with our 6 targets.\n5. **Pseudo**\nPseudo works, pseudo+optuna is much better then no pseudo+optuna, both LB and cv and PB 433811->43363. \nPseudo also helped produce best PB(435144) single model. \nTODO: I did not try (Mean of Train PL / Actual Labelsas) mentioned in [1st solution](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369457) it might give boost.\n6. **Tune for each target**\nFor example back translation help a lot for vocabulary but hurt conventions as show on cv image above. \n\n - Code \nhttps://www.kaggle.com/code/goldenlock/best-pb-online-1127?scriptVersionId=112354741 (ensemble infer, best personal PB)\nhttps://www.kaggle.com/code/goldenlock/pb-433630-pseudo-optuna-tune (ensemble infer, selected PB)\nhttps://www.kaggle.com/goldenlock/single-model-infer (single model infer, 5 fold + 1 full train=6 models, 20min)\nhttps://www.kaggle.com/datasets/goldenlock/feedbacken (train)\nhttps://github.com/chenghuige/Feedback-Prize---English-Language-Learning (full code)", "votes": 112 }, { "id": 2350698, "postDate": "2023-07-19T12:50:45.817Z", "content": "

What does awp training means ?

", "rawMarkdown": "What does awp training means ? ", "votes": 1 }, { "id": 2052091, "postDate": "2022-12-01T22:54:05.270Z", "content": "

Thats a great info

", "rawMarkdown": "Thats a great info", "votes": 1 }, { "id": 2051505, "postDate": "2022-12-01T13:23:56.817Z", "content": "

Hello. Congrats! and thank you for sharing your solution.

\n

Can I ask you about \"back translation for pretrain\".

\n

1) How did you produced pseudo labeled records.

\n
\n

[nl, fr, de, pt, af, cn, ru, fi, sv, ja, ko, el, hr, cy]

\n
\n

Did you used all of these languages?
\nAnd, did you used all of those records? ( like 3911 x languages pseudo labeled records), or did you select some part of them for pre-training.

\n

2) About weight parameters.

\n
\n

weight=np.array([1,1,10,1,0.25,0.25], float) * 0.8

\n
\n

How did you decide weights value for each targets?
\nand what is *0.8 parameter for?

\n

Thank you.

", "rawMarkdown": "Hello. Congrats! and thank you for sharing your solution.\n\nCan I ask you about \"back translation for pretrain\".\n\n1) How did you produced pseudo labeled records.\n\n> [nl, fr, de, pt, af, cn, ru, fi, sv, ja, ko, el, hr, cy]\n\nDid you used all of these languages?\nAnd, did you used all of those records? ( like 3911 x languages pseudo labeled records), or did you select some part of them for pre-training.\n\n2) About weight parameters.\n\n> weight=np.array([1,1,10,1,0.25,0.25], float) * 0.8\n\nHow did you decide weights value for each targets?\nand what is *0.8 parameter for?\n\nThank you.", "votes": 1, "replies": [ { "id": 2051521, "postDate": "2022-12-01T13:35:11.663Z", "content": "

for 1) yes I used all these languages, actually more, but some other languages may not improve my ensemble results so not included finally. I transelate and translate back for each language to get a new train dataset, like for nl get the nl-dataset, train on nl-dataset only to get nl-model, then train on original train dataset with model load from the pretrained model nl-model.
\nI tired mix different language datasets to do the first step seems not work well.
\nfor 2), actually nl-model it's self not perform well, but it works well as a pretrained model. I used wandb to log cv score for each target and found, by using back-translation some targets improve a lot (vocabulary) and some others decrease a lot (conventions).
\nAnd this is a common fact, not by chance, I found all language got same conclution, help vocabulary hurt conventions.
\nSo I tried to manually increase model weights for vocabulary and decrease weight on conventions and it show increase on both CV and LB.

", "rawMarkdown": "for 1) yes I used all these languages, actually more, but some other languages may not improve my ensemble results so not included finally. I transelate and translate back for each language to get a new train dataset, like for nl get the nl-dataset, train on nl-dataset only to get nl-model, then train on original train dataset with model load from the pretrained model nl-model. \nI tired mix different language datasets to do the first step seems not work well.\nfor 2), actually nl-model it's self not perform well, but it works well as a pretrained model. I used wandb to log cv score for each target and found, by using back-translation some targets improve a lot (vocabulary) and some others decrease a lot (conventions).\nAnd this is a common fact, not by chance, I found all language got same conclution, help vocabulary hurt conventions.\nSo I tried to manually increase model weights for vocabulary and decrease weight on conventions and it show increase on both CV and LB.", "votes": 3 }, { "id": 2054845, "postDate": "2022-12-04T13:39:38.593Z", "content": "

Thank you for detailed answer. I learned a lot from your solution!

", "rawMarkdown": "Thank you for detailed answer. I learned a lot from your solution!", "votes": 1 } ] }, { "id": 2050331, "postDate": "2022-11-30T16:58:25.130Z", "content": "

Thanks for sharing and congratulations….

", "rawMarkdown": "Thanks for sharing and congratulations....", "votes": 1 }, { "id": 2050274, "postDate": "2022-11-30T16:11:18.020Z", "content": "

we too use back translation. Cool shake up, Congrats!

", "rawMarkdown": "we too use back translation. Cool shake up, Congrats!", "votes": 1 }, { "id": 2050130, "postDate": "2022-11-30T14:40:29.263Z", "content": "

Nice jump! I was wondering where you were :P

", "rawMarkdown": "Nice jump! I was wondering where you were :P", "votes": 1, "replies": [ { "id": 2050682, "postDate": "2022-11-30T23:22:02.727Z", "content": "

Thanks @crodoc, congratulation to you also! Interesting game, if I added svr to my ensemble maybe I could be seen on LB😃, just wondering.

", "rawMarkdown": "Thanks @crodoc, congratulation to you also! Interesting game, if I added svr to my ensemble maybe I could be seen on LB😃, just wondering.", "votes": 1 }, { "id": 2054762, "postDate": "2022-12-04T12:50:33.273Z", "content": "

Gezi is always on top of private LB, regardless of where on private.

\n

Qishen : CV :: Gezi : NLP 🙏

", "rawMarkdown": "Gezi is always on top of private LB, regardless of where on private. \n\nQishen : CV :: Gezi : NLP 🙏", "votes": 1 } ] }, { "id": 2050125, "postDate": "2022-11-30T14:36:41.853Z", "content": "

Great work!

", "rawMarkdown": "Great work!", "votes": 1 }, { "id": 2049818, "postDate": "2022-11-30T11:06:42.087Z", "content": "

Interesting that you use rank loss. Congrats on the result!

", "rawMarkdown": "Interesting that you use rank loss. Congrats on the result!", "votes": 1 }, { "id": 2049749, "postDate": "2022-11-30T10:04:25.243Z", "content": "

Great work and Congratulations! Learn a lot from your solution. Thanks for sharing.

", "rawMarkdown": "Great work and Congratulations! Learn a lot from your solution. Thanks for sharing.", "votes": 1 }, { "id": 2049424, "postDate": "2022-11-30T06:19:24.060Z", "content": "

Congratulations! I do have a question tho, if the models are pretrained with fb2 data, will there be any form of leakage? Since some of the essays from fb2 might be similar to the validation data of different folds. Also, will there be any form of leakage if just pretrain the models with the training data and the fine tune?

", "rawMarkdown": "Congratulations! I do have a question tho, if the models are pretrained with fb2 data, will there be any form of leakage? Since some of the essays from fb2 might be similar to the validation data of different folds. Also, will there be any form of leakage if just pretrain the models with the training data and the fine tune?", "votes": 1, "replies": [ { "id": 2049797, "postDate": "2022-11-30T10:49:46.677Z", "content": "

There are small number of overlap docids, but it does not affect so much, basicly for safe cv you could remove them.
\nFor me I did not remove them for fb2 pretrain, but for pseudo using fb1 I removed duplicate doc ids.

", "rawMarkdown": "There are small number of overlap docids, but it does not affect so much, basicly for safe cv you could remove them.\nFor me I did not remove them for fb2 pretrain, but for pseudo using fb1 I removed duplicate doc ids.", "votes": 1 } ] }, { "id": 2049238, "postDate": "2022-11-30T02:24:05.667Z", "content": "

Congrats and nice augmentation! Compared with image augmentation, it is hard to do augmentation in text. Could you explain more about pearson loss?

", "rawMarkdown": "Congrats and nice augmentation! Compared with image augmentation, it is hard to do augmentation in text. Could you explain more about pearson loss?", "votes": 1, "replies": [ { "id": 2049248, "postDate": "2022-11-30T02:33:54.850Z", "content": "

You can refer to my code https://www.kaggle.com/datasets/goldenlock/feedbacken , torch/loss.py FLAGS.rank_loss_rate

", "rawMarkdown": "You can refer to my code https://www.kaggle.com/datasets/goldenlock/feedbacken , torch/loss.py FLAGS.rank_loss_rate", "votes": 1 }, { "id": 2049309, "postDate": "2022-11-30T03:33:44.133Z", "content": "

Thank you.

", "rawMarkdown": "Thank you.", "votes": 1 } ] }, { "id": 2049169, "postDate": "2022-11-30T01:24:19.980Z", "content": "

Nice solution, congrats for the win!

", "rawMarkdown": "Nice solution, congrats for the win!", "votes": 1, "replies": [ { "id": 2049174, "postDate": "2022-11-30T01:31:03.023Z", "content": "

Thanks! @xianghuang666

", "rawMarkdown": "Thanks! @xianghuang666 ", "votes": 1 } ] }, { "id": 2049168, "postDate": "2022-11-30T01:23:00.913Z", "content": "

Congratulations @goldenlock. Your idea is so great.

\n

Could you explain the back translation more detail?

", "rawMarkdown": "Congratulations @goldenlock. Your idea is so great.\n\nCould you explain the back translation more detail?", "votes": 1, "replies": [ { "id": 2049171, "postDate": "2022-11-30T01:25:03.493Z", "content": "

Use gooltrans to translate train text to cn text for example and translate back to english, then using the new train data for pretrain.
\nfrom googletrans import Translator
\ntrans = Translator()

", "rawMarkdown": "Use gooltrans to translate train text to cn text for example and translate back to english, then using the new train data for pretrain.\nfrom googletrans import Translator\ntrans = Translator()", "votes": 1 }, { "id": 2049175, "postDate": "2022-11-30T01:31:12.670Z", "content": "

Nice augumentation. Hence, does your final submission you chose include models with pseudo data? I found that using pseudo data will hurt private score.

", "rawMarkdown": "Nice augumentation. Hence, does your final submission you chose include models with pseudo data? I found that using pseudo data will hurt private score. " }, { "id": 2049177, "postDate": "2022-11-30T01:32:04.313Z", "content": "

Best choosen one with pseudo and best score not, I explain more in my post, I think pseudo should work but not improve so much.

", "rawMarkdown": "Best choosen one with pseudo and best score not, I explain more in my post, I think pseudo should work but not improve so much." } ] }, { "id": 2049367, "postDate": "2022-11-30T04:45:08.030Z", "content": "

Thanks for sharing, back translation is cool idea. congrats!

", "rawMarkdown": "Thanks for sharing, back translation is cool idea. congrats!", "votes": 2 }, { "id": 2049269, "postDate": "2022-11-30T03:02:37.287Z", "content": "

Aged ginger is more pungent

", "rawMarkdown": "Aged ginger is more pungent", "votes": 1 }, { "id": 2903092, "postDate": "2024-07-03T15:48:22.173Z", "content": "

So I have a doubt, doesn't deberta model have max len like bert (512 tokens). Like deberta takes how many tokens u specify.

", "rawMarkdown": "So I have a doubt, doesn't deberta model have max len like bert (512 tokens). Like deberta takes how many tokens u specify." }, { "id": 2051055, "postDate": "2022-12-01T07:17:05.557Z", "content": "

@goldenlock hi, congrats and thanks for the sharing!

\n

would you like to share some idea about how to select submission for the private testing?

\n

from experiment we can know, some submissions score high in public leaderboard but low in private, so what should we do? how do we choose final submission for private testing？

\n

for example, i have 2 submission, one of them is ensemble whose cv is 0.4503 but public score worse, the other one cv is 0.4511 but public score better…. i am confused

", "rawMarkdown": "@goldenlock hi, congrats and thanks for the sharing! \n\nwould you like to share some idea about how to select submission for the private testing? \n\nfrom experiment we can know, some submissions score high in public leaderboard but low in private, so what should we do? how do we choose final submission for private testing？\n\nfor example, i have 2 submission, one of them is ensemble whose cv is 0.4503 but public score worse, the other one cv is 0.4511 but public score better.... i am confused", "replies": [ { "id": 2051296, "postDate": "2022-12-01T10:26:50.487Z", "content": "

For this competion as LB dataset is small, you should trust your cv, you can see from my single model results table in my post, and you will find CV and PB is more aligned then CV and LB.
\nOne thing important is you have to make sure your CV is correct, first no leak like when using pseduo.
\nAnother thing is you should take care not to overfit your CV a lot. For example I got cv 4494 using optuna tune model weights for all OOF then my cv is over optimized, the better way will be you do optuna on OOF fold 1-3 and test on fold 4, by doing this you can avoid overfit to cv.
\nAlso if you manually tune weight on the whole OOF you will face the same problem.

", "rawMarkdown": "For this competion as LB dataset is small, you should trust your cv, you can see from my single model results table in my post, and you will find CV and PB is more aligned then CV and LB. \nOne thing important is you have to make sure your CV is correct, first no leak like when using pseduo.\nAnother thing is you should take care not to overfit your CV a lot. For example I got cv 4494 using optuna tune model weights for all OOF then my cv is over optimized, the better way will be you do optuna on OOF fold 1-3 and test on fold 4, by doing this you can avoid overfit to cv.\nAlso if you manually tune weight on the whole OOF you will face the same problem. ", "votes": 2 } ] }, { "id": 2050281, "postDate": "2022-11-30T16:14:16.470Z", "content": "

Congrats on your victory @goldenlock !

\n

I'm curious, how do you choose the 3 final submissions as you have 4 convincing different approaches? what's the rationale behind?

", "rawMarkdown": "Congrats on your victory @goldenlock !\n\nI'm curious, how do you choose the 3 final submissions as you have 4 convincing different approaches? what's the rationale behind?", "replies": [ { "id": 2050681, "postDate": "2022-11-30T23:20:40.317Z", "content": "

I choose by the best cv( over optimized by optuna tune on whole OOF), and they are also the best on LB.

", "rawMarkdown": "I choose by the best cv( over optimized by optuna tune on whole OOF), and they are also the best on LB. " } ] }, { "id": 2049218, "postDate": "2022-11-30T02:03:55.483Z", "content": "

Congrats! Thanks for sharing.
\nI found that deberta-v1-large and others with max_length=560 works just a little worse than deberta-v3s with max_length=1428.
\nWhat max_length did you choose?

", "rawMarkdown": "Congrats! Thanks for sharing.\nI found that deberta-v1-large and others with max_length=560 works just a little worse than deberta-v3s with max_length=1428.\nWhat max_length did you choose?", "replies": [ { "id": 2049229, "postDate": "2022-11-30T02:14:33.037Z", "content": "

For deberta-v1 I tried 512 but not used.

", "rawMarkdown": "For deberta-v1 I tried 512 but not used." } ] }, { "id": 2049194, "postDate": "2022-11-30T01:43:05.740Z", "content": "

Congrats, nice solution. Can you share which translation model do you used?

", "rawMarkdown": "Congrats, nice solution. Can you share which translation model do you used?", "replies": [ { "id": 2049197, "postDate": "2022-11-30T01:44:00.967Z", "content": "

I used google trans python api.

", "rawMarkdown": "I used google trans python api." } ] }, { "id": 2049176, "postDate": "2022-11-30T01:31:33.503Z", "content": "

Congratulations! May I ask how much does Back Translation help to improve the performance?

", "rawMarkdown": "Congratulations! May I ask how much does Back Translation help to improve the performance?", "replies": [ { "id": 2049185, "postDate": "2022-11-30T01:37:07.323Z", "content": "

Thanks @clyde7470 , updated my post, 4467 to 44533.

", "rawMarkdown": "Thanks @clyde7470 , updated my post, 4467 to 44533.", "votes": 1 } ] }, { "id": 2049244, "postDate": "2022-11-30T02:30:19.453Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:57:52Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9091204, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/gezi-2nd-solution-back-translation-rank-loss" }