{ "id": 313201, "title": "9th solution, deberta is the king, pure ensemble of bert models ", "content": "

My solution is an simple ensemble of bert models.
\nI have tried to use recall of different models to do l2 rerank using lgb, but failed to make it work, sad story:(. So no stacking here, just simple ensemble of level 1 bert models with optimized post process(search by optuna).

\n\n

My code has open sourced here:
\ninfering:
\nhttps://www.kaggle.com/code/goldenlock/feedback-base
\ntraining:
\nhttps://github.com/chenghuige/feedback-prize-2021

", "comments": [ { "id": 1724144, "author_name": "hengck23", "author_url": "", "post_date": "2022-03-16T03:38:04.153000", "content": "

Good work and congrats for being the top winners!

\n

\"Be sure not to remove '\\n', super important feature, I change '\\n' to new word '[BR]' \"

\n

I find another improvement in my experiments:
\nadd [essay_end] tag (to differentiate those from truncation and non-truncation)
\n(maybe help in conclusion tag?)

\n
non-truncation\n... [xxx][essay_end][eos][pad][pad] ....[pad]\n\ntruncation\n......[xxx][xxx][xxx][xxx][xxx][eos] \n
", "votes": 9, "replies": [ { "id": 1724182, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T04:13:12.133000", "content": "

Good idea. Your ideas and sharing help us a lot:) Thanks @hengck23

", "votes": 0, "replies": [] } ] }, { "id": 1724152, "author_name": "kwchen", "author_url": "", "post_date": "2022-03-16T03:47:52.627000", "content": "

Congratulations! What batch size and learning rate did you use to train deberta-v3-large? I trained deberta v3 large with batch size 2 and different learning rates but got worse CV and LB than longformer models.

", "votes": 1, "replies": [ { "id": 1724159, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T03:50:22.340000", "content": "

With bs 16 and learning rate max 2e-5, for all non bert part using lr max 1e-3.

", "votes": 4, "replies": [] } ] }, { "id": 1724778, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T13:56:13.473000", "content": "

For single model. There some lucky points(as both LB and PB dataset is small) see below. If I lower lr as * 0.5. I got 0.004 LB gain, but lose 0.001 on PB. And I noticed I have one deberta-v3 1536 model with RL, it got 0.7 LB but 0.719 on PB.
\n\"\"

", "votes": 2, "replies": [] }, { "id": 1767012, "author_name": "mafuf", "author_url": "", "post_date": "2022-04-25T02:23:37.123000", "content": "

Thanks for the details, this really helped me a lot.

", "votes": 0, "replies": [] }, { "id": 1736815, "author_name": "Dimanjan Dahal", "author_url": "", "post_date": "2022-03-27T18:51:44.423000", "content": "

What does PB stand for? And LB; leaderboard?

", "votes": 0, "replies": [ { "id": 1738546, "author_name": "gezi", "author_url": "", "post_date": "2022-03-29T11:12:19.763000", "content": "

pb mean private, lb is public

", "votes": 1, "replies": [] } ] }, { "id": 1729612, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-03-20T09:30:58.700000", "content": "

Congratz on the solo gold @goldenlock !

\n
\n

Word level LSTM on top of bert help a lot! (using torch scatter_add, LB +2K, PB +3K)

\n
\n

That's a very cool idea, would you mind sharing the code for that ? :)

", "votes": 0, "replies": [ { "id": 1729881, "author_name": "gezi", "author_url": "", "post_date": "2022-03-20T16:49:46.573000", "content": "

Sure, I shared the fake code here https://www.kaggle.com/code/goldenlock/word-level-lstm-fake-code

", "votes": 3, "replies": [] }, { "id": 1729905, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-03-20T17:12:05.990000", "content": "

Thanks !

", "votes": 0, "replies": [] } ] }, { "id": 1729276, "author_name": "chumajin", "author_url": "", "post_date": "2022-03-19T21:57:43.007000", "content": "

Congratulations on the gold medal. Also, thank you for sharing. I respect you.

\n

After replacing '\\n' to new word '[BR]', do I have to correct the character position of text such as \"discourse start\" ?

", "votes": 0, "replies": [ { "id": 1729428, "author_name": "gezi", "author_url": "", "post_date": "2022-03-20T04:37:18.060000", "content": "

Thanks! I do not use offset interface just use wordids interface of tokenizer, I think this interface is more elegant for this problem which based on words not chars. After changing to 'BR' what you need to take care is wordid, you need to modify wordids to make sure BR is wordid None. Take an example:
\nHe played \\n basketball -> He played [BR] baskedball
\nyou get wordids 0, 1, 2, 3 you need to change it to 0, 1, None, 2.
\nAnother workaround is you do the tokenization by yoursefl word by word, you should get same results.

", "votes": 1, "replies": [] }, { "id": 1730229, "author_name": "chumajin", "author_url": "", "post_date": "2022-03-21T03:37:20.903000", "content": "

@goldenlock I see. Thank you very much! I learned from you in this competition.

", "votes": 1, "replies": [] } ] }, { "id": 1724450, "author_name": "SchenbergZ", "author_url": "", "post_date": "2022-03-16T08:48:29.427000", "content": "

Congrats! Hope to see your code soon.

", "votes": 0, "replies": [] }, { "id": 1724178, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2022-03-16T04:09:31.280000", "content": "

Nice job! I can't see the images you linked

", "votes": 0, "replies": [ { "id": 1724181, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T04:12:24.003000", "content": "

I do not not know how to show image on kaggle… will update them

", "votes": 0, "replies": [] }, { "id": 1724495, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T09:27:44.210000", "content": "

Here want to say deep thanks to you @nbroad I used your corrected.csv it could improve around 0.001 for both LB and PB. And also I used your code in nbme for using deberta-v3 fast tokenizer. Just updated my post. Your notebooks all very useful indeed, learn a lot from you.

", "votes": 1, "replies": [] }, { "id": 1729914, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2022-03-20T17:20:05.220000", "content": "

I don't know how I missed this. Thank you for the kind words -- I'm always happy to share 😊

", "votes": 2, "replies": [] } ] }, { "id": 1724165, "author_name": "resistance0108", "author_url": "", "post_date": "2022-03-16T03:57:58.647000", "content": "

Congrats!

\n
\n

Be sure not to remove '\\n', super important feature, I change '\\n' to new word '[BR]'
\n Word level LSTM on top of bert help a lot! (using torch scatter_add)
\n I used multi obj model, token classfication(8 class) + sep classification(binary)
\n Lovasz loss help a bit

\n
\n

How much improvement was made by these modification?
\nEspecially the latter 2 because I had these ideas but forgot to try or discarded due to LB degradation.

", "votes": 0, "replies": [ { "id": 1724179, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T04:09:38.063000", "content": "
    \n
  1. lovasz .. en I just looked my pb, it did not improve pb but improve lb about 4k.
    2. \n
  2. Multi obj I used it from the very beginning, as it could do post process much easier. I think it may improve around 5k. It is hard to test as I have to make different post rules…
    \n3.Word level lstm.. I did not test with recent models. Will update later after I test it.
    3. \n
", "votes": 2, "replies": [] }, { "id": 1724188, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2022-03-16T04:19:11.853000", "content": "

What do you mean by 4k and 5k?

", "votes": 1, "replies": [] }, { "id": 1724189, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T04:21:57.873000", "content": "

4k means +0.004

", "votes": 0, "replies": [] }, { "id": 1724313, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T07:07:54.483000", "content": "

\"\" 3k gain from word level lstm.

", "votes": 2, "replies": [] } ] }, { "id": 1724147, "author_name": "DaZhuWei", "author_url": "", "post_date": "2022-03-16T03:40:32.327000", "content": "

Congrats for your solo gold~! admire~!
\ndoes deberta-v3-large perform better than longformer?
\nwhat's the score difference between those single models?

", "votes": 0, "replies": [ { "id": 1724150, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T03:45:54.157000", "content": "

Much better then longformer. about 10-15k(longformer 1536 with PB 705 LB690 while deberta-v3 1536 PB 718 LB 705), I only choosed 1 longformer model for ensemble sense.

", "votes": 1, "replies": [] } ] } ], "comment_count": 26, "messages": [ { "id": 1724138, "postDate": "2022-03-16T03:31:57.497Z", "content": "

My solution is an simple ensemble of bert models.
\nI have tried to use recall of different models to do l2 rerank using lgb, but failed to make it work, sad story:(. So no stacking here, just simple ensemble of level 1 bert models with optimized post process(search by optuna).

\n\n

My code has open sourced here:
\ninfering:
\nhttps://www.kaggle.com/code/goldenlock/feedback-base
\ntraining:
\nhttps://github.com/chenghuige/feedback-prize-2021

", "rawMarkdown": "My solution is an simple ensemble of bert models. \nI have tried to use recall of different models to do l2 rerank using lgb, but failed to make it work, sad story:(. So no stacking here, just simple ensemble of level 1 bert models with optimized post process(search by optuna).\n- Best single model is **deberta-v3-large**!\n deberta-v3-large is the king for both feedback-prize and nbme maybe😉\n deberta-v3-large + maxlen 1536 single model(train on all 15k) could get **LB 705, PB 718**\n the submission run is about 34min , I late submit test 5 folds of this model, score is **LB 708, PB 723**\n Longformer maxlen 1536 with the same model structure could only get **LB 690, PB 705**\n- Longformers(maxlen >= 1024) could improve but shortformers(maxlen == 512) can help a lot!\nWe could get **Lb 705, PB 717** using just two seperately train 512 length models.\ndeberta-v1-xlarge first 512 + deberta-v1-xlarge last 512 \nWe could get **LB 712, PB 723** with 10 shortformers only ensemble.\nlist(zip(range(len(model_dirs)), model_dirs, mns, weights)): \n [(0, '../input/feedback-model0', 'base.deberta-v3.start', 1),\n (1, '../input/feedback-model1', 'base.deberta-v3.end', 1),\n (2, '../input/feedback-model2', 'base.deberta-v3.se', 1),\n (3, '../input/feedback-model3', 'large.deberta-v2-xlarge.start', 1),\n (4, '../input/feedback-model4', 'large.deberta-v2-xlarge.se2', 1),\n (5, '../input/feedback-model5', 'base.deberta.start', 1),\n (6, '../input/feedback-model6', 'base.deberta.mid', 1),\n (7, '../input/feedback-model7', 'base.electra.start', 1),\n (8, '../input/feedback-model8', 'large.deberta-v3.start.mui-end-mid', 2),\n (9, '../input/feedback-model9', 'large.electra.start.mui-end-mid', 2)]\n len(model_dirs): 10\n- What improved single model performance? \n Be sure **not to remove '\\n'**, super important feature, I change '\\n' to new word '[BR]' , to make sure all models(like tokenizer of roberta) can handle it correctly.\n **Word level LSTM on top of bert** help a lot! (using torch scatter_add, LB +2K, PB +3K) \n I used multi obj model, **token classfication**(8 class) + **seperator classification**(binary) \n Lovasz loss help a bit (LB 4K, but no gain on PB)\n- Post process \nHow to split is important, I found below rules help a lot ! But it could not beat LGB:)\n![image.png](https://i.postimg.cc/ZYg42wDN/9-E8-CD1-F2-8-B5-A-4047-9-BB9-8-D41-AED02-FD0.jpg)\n- Ensemble \nUse per word prob mean of different models, model weight is choosen from 1-10 with optuna. \nBackbones: deberta-v3-large, deberta-xlarge, deberta-large, longformer-large, bart-large, roberta-larege. (funnel-transformer-large help a bit on PB , but no gain on LB I did not choose it)\nBest model is online **LB 716, PB 729**, **local CV 724**(about **721** without fancy parameter search) \n16 models running 5hour 20min, model names without len. means 512 length models, start means from position 0, end means from the end(last position), se means like first 256 + end 256, mid means from middle position, seq_encoder-0 means not using word level LSTM.\n [(10, 'deberta-v3.start.nwemb-0.mark_end-0'),\n (10, 'deberta-v3.start.len1024.stride-256.seq_encoder-0'),\n (10, 'deberta-v3.se2'),\n (10, 'deberta-v3.se'),\n (10, 'deberta-v3.end.len1024.seq_encoder-0'),\n (9, 'roberta.start.nwemb-0'),\n (9, 'deberta-xlarge.start'),\n (9, 'deberta-xlarge.end'),\n (9, 'bart.start.run2'),\n (8, 'deberta.start'),\n (8, 'deberta-v3.start.len1024.rnn_bi'),\n (8, 'deberta-v3.mid.len1024'),\n (7, 'deberta-v3.start.stride-256.seq_encoder-0'),\n (7, 'deberta-v3.start.len1536'),\n (6, 'longformer.start.len1536'),\n (6, 'deberta-v3.start.len1024.stride-256')]\n![image.png](https://i.postimg.cc/13ymrfj7/image-1.png)\n![image.png](https://i.postimg.cc/ht4n7NXH/image-2.png)\n![image.png](https://i.postimg.cc/52KhQydC/image-3.png)\n![image.png](https://i.postimg.cc/FHYPqx8Y/2022-03-16-12-40-36.png)\n![image.png](https://i.postimg.cc/wTG2hmMg/2022-03-16-12-41-32.png)\n![image.png](https://i.postimg.cc/8cnRB21t/2022-03-16-12-42-17.png)\n- Thanks @nbroad . I used two of your notebooks.\nhttps://www.kaggle.com/nbroad/qa-ner-hybrid-infer-nbme\nhttps://www.kaggle.com/nbroad/corrected-train-csv-feedback-prize\ncorrected train.csv help improve LB and PB around 0.001.\nfast tokenizer of deberta-v3 is a key for best single model.\n\nMy code has open sourced here:\ninfering: \nhttps://www.kaggle.com/code/goldenlock/feedback-base\ntraining:\nhttps://github.com/chenghuige/feedback-prize-2021", "votes": 67 }, { "id": 1724144, "postDate": "2022-03-16T03:38:04.153Z", "content": "

Good work and congrats for being the top winners!

\n

\"Be sure not to remove '\\n', super important feature, I change '\\n' to new word '[BR]' \"

\n

I find another improvement in my experiments:
\nadd [essay_end] tag (to differentiate those from truncation and non-truncation)
\n(maybe help in conclusion tag?)

\n
non-truncation\n... [xxx][essay_end][eos][pad][pad] ....[pad]\n\ntruncation\n......[xxx][xxx][xxx][xxx][xxx][eos] \n
", "rawMarkdown": "Good work and congrats for being the top winners!\n\n\"Be sure not to remove '\\n', super important feature, I change '\\n' to new word '[BR]' \"\n\nI find another improvement in my experiments:\nadd [essay_end] tag (to differentiate those from truncation and non-truncation)\n(maybe help in conclusion tag?)\n\n```\nnon-truncation\n... [xxx][essay_end][eos][pad][pad] ....[pad]\n\ntruncation\n......[xxx][xxx][xxx][xxx][xxx][eos] \n\n```", "votes": 9, "replies": [ { "id": 1724182, "postDate": "2022-03-16T04:13:12.133Z", "content": "

Good idea. Your ideas and sharing help us a lot:) Thanks @hengck23

", "rawMarkdown": "Good idea. Your ideas and sharing help us a lot:) Thanks @hengck23 " } ] }, { "id": 1724152, "postDate": "2022-03-16T03:47:52.627Z", "content": "

Congratulations! What batch size and learning rate did you use to train deberta-v3-large? I trained deberta v3 large with batch size 2 and different learning rates but got worse CV and LB than longformer models.

", "rawMarkdown": "Congratulations! What batch size and learning rate did you use to train deberta-v3-large? I trained deberta v3 large with batch size 2 and different learning rates but got worse CV and LB than longformer models.", "votes": 1, "replies": [ { "id": 1724159, "postDate": "2022-03-16T03:50:22.340Z", "content": "

With bs 16 and learning rate max 2e-5, for all non bert part using lr max 1e-3.

", "rawMarkdown": "With bs 16 and learning rate max 2e-5, for all non bert part using lr max 1e-3.", "votes": 4 } ] }, { "id": 1724778, "postDate": "2022-03-16T13:56:13.473Z", "content": "

For single model. There some lucky points(as both LB and PB dataset is small) see below. If I lower lr as * 0.5. I got 0.004 LB gain, but lose 0.001 on PB. And I noticed I have one deberta-v3 1536 model with RL, it got 0.7 LB but 0.719 on PB.
\n\"\"

", "rawMarkdown": "For single model. There some lucky points(as both LB and PB dataset is small) see below. If I lower lr as * 0.5. I got 0.004 LB gain, but lose 0.001 on PB. And I noticed I have one deberta-v3 1536 model with RL, it got 0.7 LB but 0.719 on PB.\n![](https://i.postimg.cc/wvg28qrb/Wechat-IMG86.jpg)", "votes": 2 }, { "id": 1767012, "postDate": "2022-04-25T02:23:37.123Z", "content": "

Thanks for the details, this really helped me a lot.

", "rawMarkdown": "Thanks for the details, this really helped me a lot." }, { "id": 1736815, "postDate": "2022-03-27T18:51:44.423Z", "content": "

What does PB stand for? And LB; leaderboard?

", "rawMarkdown": "What does PB stand for? And LB; leaderboard?", "replies": [ { "id": 1738546, "postDate": "2022-03-29T11:12:19.763Z", "content": "

pb mean private, lb is public

", "rawMarkdown": "pb mean private, lb is public", "votes": 1 } ] }, { "id": 1729612, "postDate": "2022-03-20T09:30:58.700Z", "content": "

Congratz on the solo gold @goldenlock !

\n
\n

Word level LSTM on top of bert help a lot! (using torch scatter_add, LB +2K, PB +3K)

\n
\n

That's a very cool idea, would you mind sharing the code for that ? :)

", "rawMarkdown": "Congratz on the solo gold @goldenlock !\n\n> Word level LSTM on top of bert help a lot! (using torch scatter_add, LB +2K, PB +3K)\n\nThat's a very cool idea, would you mind sharing the code for that ? :)", "replies": [ { "id": 1729881, "postDate": "2022-03-20T16:49:46.573Z", "content": "

Sure, I shared the fake code here https://www.kaggle.com/code/goldenlock/word-level-lstm-fake-code

", "rawMarkdown": "Sure, I shared the fake code here https://www.kaggle.com/code/goldenlock/word-level-lstm-fake-code\n", "votes": 3 }, { "id": 1729905, "postDate": "2022-03-20T17:12:05.990Z", "content": "

Thanks !

", "rawMarkdown": "Thanks ! " } ] }, { "id": 1729276, "postDate": "2022-03-19T21:57:43.007Z", "content": "

Congratulations on the gold medal. Also, thank you for sharing. I respect you.

\n

After replacing '\\n' to new word '[BR]', do I have to correct the character position of text such as \"discourse start\" ?

", "rawMarkdown": "Congratulations on the gold medal. Also, thank you for sharing. I respect you.\n\nAfter replacing '\\n' to new word '[BR]', do I have to correct the character position of text such as \"discourse start\" ?", "replies": [ { "id": 1729428, "postDate": "2022-03-20T04:37:18.060Z", "content": "

Thanks! I do not use offset interface just use wordids interface of tokenizer, I think this interface is more elegant for this problem which based on words not chars. After changing to 'BR' what you need to take care is wordid, you need to modify wordids to make sure BR is wordid None. Take an example:
\nHe played \\n basketball -> He played [BR] baskedball
\nyou get wordids 0, 1, 2, 3 you need to change it to 0, 1, None, 2.
\nAnother workaround is you do the tokenization by yoursefl word by word, you should get same results.

", "rawMarkdown": "Thanks! I do not use offset interface just use wordids interface of tokenizer, I think this interface is more elegant for this problem which based on words not chars. After changing to 'BR' what you need to take care is wordid, you need to modify wordids to make sure BR is wordid None. Take an example:\nHe played \\n basketball -> He played [BR] baskedball \nyou get wordids 0, 1, 2, 3 you need to change it to 0, 1, None, 2.\nAnother workaround is you do the tokenization by yoursefl word by word, you should get same results.", "votes": 1 }, { "id": 1730229, "postDate": "2022-03-21T03:37:20.903Z", "content": "

@goldenlock I see. Thank you very much! I learned from you in this competition.

", "rawMarkdown": "@goldenlock I see. Thank you very much! I learned from you in this competition.", "votes": 1 } ] }, { "id": 1724450, "postDate": "2022-03-16T08:48:29.427Z", "content": "

Congrats! Hope to see your code soon.

", "rawMarkdown": "Congrats! Hope to see your code soon." }, { "id": 1724178, "postDate": "2022-03-16T04:09:31.280Z", "content": "

Nice job! I can't see the images you linked

", "rawMarkdown": "Nice job! I can't see the images you linked", "replies": [ { "id": 1724181, "postDate": "2022-03-16T04:12:24.003Z", "content": "

I do not not know how to show image on kaggle… will update them

", "rawMarkdown": "I do not not know how to show image on kaggle... will update them" }, { "id": 1724495, "postDate": "2022-03-16T09:27:44.210Z", "content": "

Here want to say deep thanks to you @nbroad I used your corrected.csv it could improve around 0.001 for both LB and PB. And also I used your code in nbme for using deberta-v3 fast tokenizer. Just updated my post. Your notebooks all very useful indeed, learn a lot from you.

", "rawMarkdown": "Here want to say deep thanks to you @nbroad I used your corrected.csv it could improve around 0.001 for both LB and PB. And also I used your code in nbme for using deberta-v3 fast tokenizer. Just updated my post. Your notebooks all very useful indeed, learn a lot from you. ", "votes": 1 }, { "id": 1729914, "postDate": "2022-03-20T17:20:05.220Z", "content": "

I don't know how I missed this. Thank you for the kind words -- I'm always happy to share 😊

", "rawMarkdown": "I don't know how I missed this. Thank you for the kind words -- I'm always happy to share 😊", "votes": 2 } ] }, { "id": 1724165, "postDate": "2022-03-16T03:57:58.647Z", "content": "

Congrats!

\n
\n

Be sure not to remove '\\n', super important feature, I change '\\n' to new word '[BR]'
\n Word level LSTM on top of bert help a lot! (using torch scatter_add)
\n I used multi obj model, token classfication(8 class) + sep classification(binary)
\n Lovasz loss help a bit

\n
\n

How much improvement was made by these modification?
\nEspecially the latter 2 because I had these ideas but forgot to try or discarded due to LB degradation.

", "rawMarkdown": "Congrats!\n\n> Be sure not to remove '\\n', super important feature, I change '\\n' to new word '[BR]'\nWord level LSTM on top of bert help a lot! (using torch scatter_add)\nI used multi obj model, token classfication(8 class) + sep classification(binary)\nLovasz loss help a bit\n\nHow much improvement was made by these modification?\nEspecially the latter 2 because I had these ideas but forgot to try or discarded due to LB degradation.", "replies": [ { "id": 1724179, "postDate": "2022-03-16T04:09:38.063Z", "content": "
    \n
  1. lovasz .. en I just looked my pb, it did not improve pb but improve lb about 4k.
    2. \n
  2. Multi obj I used it from the very beginning, as it could do post process much easier. I think it may improve around 5k. It is hard to test as I have to make different post rules…
    \n3.Word level lstm.. I did not test with recent models. Will update later after I test it.
    3. \n
", "rawMarkdown": "1. lovasz .. en I just looked my pb, it did not improve pb but improve lb about 4k.\n2. Multi obj I used it from the very beginning, as it could do post process much easier. I think it may improve around 5k. It is hard to test as I have to make different post rules... \n3.Word level lstm.. I did not test with recent models. Will update later after I test it.", "votes": 2 }, { "id": 1724188, "postDate": "2022-03-16T04:19:11.853Z", "content": "

What do you mean by 4k and 5k?

", "rawMarkdown": "What do you mean by 4k and 5k?", "votes": 1 }, { "id": 1724189, "postDate": "2022-03-16T04:21:57.873Z", "content": "

4k means +0.004

", "rawMarkdown": "4k means +0.004" }, { "id": 1724313, "postDate": "2022-03-16T07:07:54.483Z", "content": "

\"\" 3k gain from word level lstm.

", "rawMarkdown": "![](https://i.postimg.cc/L6HZbBBS/851647414309-pic.jpg) 3k gain from word level lstm.", "votes": 2 } ] }, { "id": 1724147, "postDate": "2022-03-16T03:40:32.327Z", "content": "

Congrats for your solo gold~! admire~!
\ndoes deberta-v3-large perform better than longformer?
\nwhat's the score difference between those single models?

", "rawMarkdown": "Congrats for your solo gold~! admire~! \ndoes deberta-v3-large perform better than longformer?\nwhat's the score difference between those single models?", "replies": [ { "id": 1724150, "postDate": "2022-03-16T03:45:54.157Z", "content": "

Much better then longformer. about 10-15k(longformer 1536 with PB 705 LB690 while deberta-v3 1536 PB 718 LB 705), I only choosed 1 longformer model for ensemble sense.

", "rawMarkdown": "Much better then longformer. about 10-15k(longformer 1536 with PB 705 LB690 while deberta-v3 1536 PB 718 LB 705), I only choosed 1 longformer model for ensemble sense.", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:36:08Z", "competition_id": 31779, "competition_slug": "feedback-prize-2021", "team_id": 7887262, "solution_url": "/competitions/feedback-prize-2021/writeups/gezi-9th-solution-deberta-is-the-king-pure-ensembl" }