{ "id": 313389, "title": "2nd Place - Weighted Box Fusion and Post Process", "content": "

Second Place Solution

\n

Thank you Georgia State University, The Learning Agency Lab, and Kaggle for an awesome competition. The data was high quality and interesting. The metric and train test split were well done. This competition was a success and will benefit a good cause.

\n

Teaming with Chun Ming Lee and Udbhav Bamba has been wonderful. I learned more about NLP than if I had taken an online course. These two Kagglers are grandmasters at NLP and I now know how to use HuggingFace Trainer efficiently and effectively. I also know how to customize models and losses. Thank you @leecming @ubamba98

\n

Solution Summary

\n

The secret sauce to our solution is powerful post process by Chun Ming (boost CV LB!), a huge variety of models implemented by Udbhav (many long sequence models!), and weighted box fusion by Chris (used in my previous comp here). Everything was fine tuned on local CV and achieved 2nd Place $35,000!

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Hugging Face ModelCVPublic LBPrivate LBspecial
microsoft/deberta-large706710721trained with 100% train data
microsoft/deberta-large699706719add lstm, add jaccard loss
microsoft/deberta-v3-large694697709convert slow tokenizer to fast
microsoft/deberta-xlarge708704713
microsoft/deberta-v2-xlarge699700716convert slow tokenizer to fast
allenai/longformer-large-4096702705716add lstm head
LSG converted roberta703702714convert 512 roberta to 1536
funnel-transformer/large688689708
google/bigbird-roberta-base675676692train 1024 infer 1024
uw-madison/yoso-4096652655668lsh_backward=False
\n

Weighted Box Fusion - CV 741, Public 727, Private 740

\n

Our final solution contained the 10 models listed above. We included 3 out of 10 K-folds for each of the 10 models. Our 8 hour 30 minute submission inferred 27 models! (we removed 3 folds from 30)

\n

Weighted Box Fusion by @zfturbo from his GitHub here was the magic that combined all the model predictions together. Individual models had average CV around 700 and the WBF ensemble achieved CV 741. (Folds from same models were averaged before WBF).

\n

\"\"

\n

Weighted Box Fusion can do something that no other ensemble method can do. If one model has prediction string 8 9 10 11 and another model has 10 11 12 13. Then weighted box fusion takes the average of the two starts, (8 + 10)/2 = 9 and the average of the two ends, (11 + 13)/2 = 12 resulting in 9 10 11 12. Furthermore, WBF reads submission.csv files and doesn't care what tokenizer you used, so its easy to use. (Note we add an extra column to submission dataframe with span confidence).

\n

If we averaged the token/word probabilities, or used a voting classifier, we would either get the union 8 9 10 11 12 13 or the intersection 10 11. Neither of which we want. Also averaging two \"BIO\" predictions gives a new prediction with two different \"B\" which we do not want either.

\n

Post Process

\n

Chun Ming is the mastermind behind our team's strong PP. We applied the same PP to each model before WBF. Our PP code is posted here and Chun Ming will now describe it. Our PP significantly boosted CV LB!

\n

We applied heavy post-processing to the word-level soft predictions each model made (after averaging token probabilities across folds and before the WBF ensembler). Overall CV was improved by ~.008.

\n

In order of descending impact on CV:

\n
    \n

  1. Repairing span predictions
    \nSince we trained using cross-entropy loss at the token level, the raw predictions would often result in broken spans. For example, given a chain of Lead token predictions, there might be a misprediction of an “Other” in the middle. We had a rule to convert that “Other” back to Lead.

    2. \n

  2. Discourse-specific rules
    \nWe came up with common sense heuristics for the discourses. For example, for Lead, Position, and Concluding Statement - there should only be a maximum of one each for a text. We’d predict the most probable candidate for each or merge close duplicate spans.

    3. \n

  3. Adjusting lengths of predicted spans
    \nWe adjusted lengths of spans based on their original predicted length. For example, for a predicted Evidence span that was less than 45 words long, we’d shift the start of the predicted span back by 9 words. These rules and the improvement in LB/CV suggest we were taking advantage of the evaluation metric only requiring a 50% overlap in prediction and label to be counted as correct.

    4. \n
\n

It’s not sexy but sitting down and eye-balling model predictions and comparing them with ground truths using a tool like displacy was rewarding in helping us better understand the task, how our models were behaving, and how to improve model predictions.

\n

Model Details

\n

Surprisingly many more models than LongFormer and BigBird could be trained with sequence lengths greater than 512. All our models were trained with sequence length 1536 and inferred at 1536 (except BigBird-base at 1024).

\n

The model DeBERTa accepts any input size, so we just train with 1536 as is.

\n

The model Funnel-transformer can accept any size after updating its config.

\n
model = AutoModelForTokenClassification.from_pretrained(\n    'funnel-transformer/large', num_labels=15,\n    max_position_embeddings=1536)\n
\n

For BigBird, we used full attention

\n
model = AutoModelForTokenClassification.from_pretrained(\n    'google/bigbird-roberta-base', num_labels=15,\n    attention_type=\"original_full\")\n
\n

For YOSO, we disabled lsh_backward

\n
model = AutoModelForTokenClassification.from_pretrained(\n    'uw-madison/yoso-4096', num_labels=15,\n    lsh_backward=False)\n
\n

Model Training

\n

Thank you Nvidia for providing us some V100 32GB GPUs and A100 40GB GPUs. For each model, we trained 6 out of 10 KFolds and built our CV from the resultant 60% train data OOF. Each fold would take approximately 4 hours on 1xGPU and required between 3 and 5 epochs to maximize val score.

\n

For two models, we used my train public notebooks here and here. For the other eight models, we used Hugging Face Trainer. Most models, used the following parameters. And we added @robikscube Kaggle metric here and our PP directly to the Trainer evaluation, so we could watch the Kaggle metric score every 500 steps and save the model weights with best Kaggle score.

\n

Notice how we use FP16, gradient_accumulation_steps, and gradient_checkpointing. With these settings, you can basically train any size model on any GPU including Kaggle's P100 16GB GPU.

\n
from transformers import TrainingArguments, Trainer\nargs = TrainingArguments( NAME,\n                     PRETRAINED_MODEL,\n                     evaluation_strategy = 'steps',\n                     eval_steps = 500,\n                     dataloader_num_workers=8,\n                     warmup_ratio=0,\n                     lr_scheduler_type = 'linear',\n                     learning_rate = 2e-5,\n                     log_level = 'warning',\n                     fp16 = True,\n                     per_device_train_batch_size = 2,\n                     per_device_eval_batch_size = 2,\n                     gradient_accumulation_steps = 2,\n                     gradient_checkpointing = True,\n                     num_train_epochs = 5,\n                     save_strategy = 'no',\n                     save_total_limit = 1)\n\ntrainer = Trainer(model,\n              args,\n              train_dataset=train_dataset,\n              eval_dataset=valid_dataset,\n              compute_metrics=KaggleMetric(valid_df, valid_dataset),\n              callbacks=[SaveBestModelCallback],\n              data_collator=data_collator,\n              tokenizer=tokenizer)\n
\n

Inference Code with PP with WBF

\n

We posted our final submission inference code on Kaggle here. It includes our post process and weighted box fusion implementation. Enjoy!

\n

Training Code on GitHub

\n

Our training code is published on GitHub here

", "comments": [ { "id": 1725276, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2022-03-17T01:14:46.953000", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Hugging Face ModelCVPublic LBPrivate LBspecial
microsoft/deberta-large706710721trained with 100% train data
\n

How do you get CV when training on 100% of the data?

", "votes": 8, "replies": [ { "id": 1725281, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-17T01:21:33.547000", "content": "

ha, good question. The \"CV score\" is the score when training with 10 K-Fold. First we train with 10 K-Fold, and make note of the learning rate and number of epochs. Second we trained a second time using 100% data. So actually it's two models. The first model was K-Fold and got CV 706. The second model used the same hyperparameters and trained on 100% data and got Public 710 Private 721

", "votes": 9, "replies": [] }, { "id": 1725283, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-17T01:23:32.277000", "content": "

@nbroad thanks for all your great work at Hugging Face. The NLP library transformers and model repository are fantastic. We also used your code to convert deberta-v2 and v3 slow tokenizers to fast tokenizers. Thanks.

", "votes": 9, "replies": [] }, { "id": 1725639, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2022-03-17T09:24:15.813000", "content": "

I greatly appreciate the comment, even though my contributions to transformers are pretty small relative to others.

\n

I'm always happy to share what I learn through Hugging Face with the community here. I think the deberta v2/3 fast tokenizers should be merged to the main branch soon so it won't be so awkward to use on Kaggle.

\n

I forgot to congratulate you! This is another fantastic approach and final rank 😊

", "votes": 4, "replies": [] }, { "id": 1725709, "author_name": "GUNER", "author_url": "", "post_date": "2022-03-17T11:00:46.857000", "content": "

'''The first model was K-Fold and got CV 706. The second model used the same hyperparameters and trained on 100% data and got Public 710 Private 721'''

\n

so 706 is out of CV of 10 models(so 10 models are submitted each being one fold of 10), whereas 710 is from a single model (only a single model is submitted) trained from all the data??

\n

Big congrats again!!

", "votes": 0, "replies": [] }, { "id": 1732556, "author_name": "Devansh Chowdhury", "author_url": "", "post_date": "2022-03-23T14:05:24.470000", "content": "
\n

ha, good question. The \"CV score\" is the score when training with 10 K-Fold. First we train with 10 K-Fold, and make note of the learning rate and number of epochs. Second we trained a second time using 100% data. So actually it's two models. The first model was K-Fold and got CV 706. The second model used the same hyperparameters and trained on 100% data and got Public 710 Private 721

\n
\n

@cdeotte Is this technique subjective to just the NLP area or you have used this approach for Computer vision and tabular data as well?

", "votes": 0, "replies": [] } ] }, { "id": 1725294, "author_name": "Dewei Chen", "author_url": "", "post_date": "2022-03-17T01:36:58.300000", "content": "

Would you mind public all the training models? Because it's too hard for us to Reappearance it.
\nThanks!

", "votes": 6, "replies": [ { "id": 1725298, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-17T01:40:08.153000", "content": "

We published our inference code here. And we made all trained models (i.e. Kaggle datasets) public.

", "votes": 8, "replies": [] }, { "id": 1725338, "author_name": "Dewei Chen", "author_url": "", "post_date": "2022-03-17T02:35:17.913000", "content": "

Got it, cheers :)

", "votes": 1, "replies": [] } ] }, { "id": 1725135, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T20:51:49.637000", "content": "

Great work! The ensemble use WBF of each model only, not using per word soft logits/prob merge?

", "votes": 3, "replies": [ { "id": 1725137, "author_name": "Udbhav Bamba", "author_url": "", "post_date": "2022-03-16T20:53:56.140000", "content": "

For different folds but same architecture, we average the probabilities and for different architectures we use WBF.

", "votes": 3, "replies": [] }, { "id": 1725139, "author_name": "gezi", "author_url": "", "post_date": "2022-03-16T20:55:54.330000", "content": "

Cool! Thanks!

", "votes": 2, "replies": [] } ] }, { "id": 1725234, "author_name": "KhanhVD", "author_url": "", "post_date": "2022-03-17T00:14:50.640000", "content": "

Nice solution! Congrats @cdeotte and team. I will try your WBF method in the next NLP competition ;)

", "votes": 4, "replies": [] }, { "id": 1735972, "author_name": "Sharaf Mukhtar", "author_url": "", "post_date": "2022-03-26T18:56:51.920000", "content": "

Congratulations, thanks for sharing this

", "votes": 1, "replies": [] }, { "id": 1735423, "author_name": "FPiotro", "author_url": "", "post_date": "2022-03-26T08:27:41.333000", "content": "

Congratulations on this solution @cdeotte and team 🧐
\nWell explained as always 👍
\nKeep up the good work!

", "votes": 1, "replies": [ { "id": 1736048, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-26T21:41:42.450000", "content": "

Thanks FPiotro

", "votes": 0, "replies": [] } ] }, { "id": 1734398, "author_name": "Sdilshad", "author_url": "", "post_date": "2022-03-25T09:56:52.950000", "content": "

congrats.. Great work, thanks for sharing

", "votes": 1, "replies": [] }, { "id": 1733310, "author_name": "biubiubiu~", "author_url": "", "post_date": "2022-03-24T07:13:14.743000", "content": "

Congrats and thanks for sharing!!!
\nDid post processing make a big difference to improve final score? Or an enough large model like deberta and with enough training time can bring a impressive result.

", "votes": 1, "replies": [ { "id": 1735359, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-26T06:29:45.980000", "content": "

Post processing made a large difference. Probably around +0.010

", "votes": 0, "replies": [] } ] }, { "id": 1733060, "author_name": "Tom Zheng", "author_url": "", "post_date": "2022-03-24T01:18:38.210000", "content": "

It is really a great code.

", "votes": 1, "replies": [ { "id": 1733064, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-24T01:25:14.227000", "content": "

Thanks Jingyuan

", "votes": 0, "replies": [] } ] }, { "id": 1732976, "author_name": "spin fit", "author_url": "", "post_date": "2022-03-23T22:46:32.777000", "content": "

Wow..
\nCongratulations ~!!

", "votes": 1, "replies": [] }, { "id": 1732589, "author_name": "Jakub Dębski", "author_url": "", "post_date": "2022-03-23T14:34:42.820000", "content": "

Great work, thanks for sharing details. This post process is looking great.

", "votes": 1, "replies": [] }, { "id": 1732005, "author_name": "AtulVerma", "author_url": "", "post_date": "2022-03-22T23:21:57.423000", "content": "

Congratulations @cdeotte

", "votes": 1, "replies": [] }, { "id": 1727990, "author_name": "ONIONCORE", "author_url": "", "post_date": "2022-03-18T14:02:03.570000", "content": "

Great work. Thanks for sharing the code.

", "votes": 1, "replies": [] }, { "id": 1727879, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2022-03-18T12:13:18.350000", "content": "
\n

Adjusting lengths of predicted spans
\n We adjusted lengths of spans based on their original predicted length. For example, for a predicted Evidence span that was less than 45 words long, we’d shift the start of the predicted span back by 9 words. These rules and the improvement in LB/CV suggest we were taking advantage of the evaluation metric only requiring a 50% overlap in prediction and label to be counted as correct.

\n
\n

@leecming ,
\nHow did you decide how many words to shift the predictions by?

", "votes": 1, "replies": [ { "id": 1727885, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2022-03-18T12:19:54.667000", "content": "

Optimized against CV.

\n

I was somewhat paranoid about these rules and pushed for our second sub to use a more conservative subset, excluding the length adjustments. Turns out it was unnecessary as CV/LB showed a strong correlation extending to private LB.

", "votes": 2, "replies": [] }, { "id": 1727904, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2022-03-18T12:35:22.790000", "content": "

So did you just try a bunch of shifts (-1, +1, -2, +2, etc.) and see how that impacted the CV?

", "votes": 1, "replies": [] }, { "id": 1727915, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2022-03-18T12:46:10.590000", "content": "

Broadly yes - we had optimized CV code that used multiprocessing so we could test hyperparam changes very rapidly (<10s per run for 7K OOF samples). I tried using Optuna but found my gut sense better at optimizing :)

\n

Ofc, in hindsight, this post-processing could have been potentially been done by a 2nd level ML model as a number of top teams did.

", "votes": 1, "replies": [] } ] }, { "id": 1726251, "author_name": "Ken Krige", "author_url": "", "post_date": "2022-03-17T19:43:30.263000", "content": "

Great work. Thanks for sharing the code.

", "votes": 1, "replies": [] }, { "id": 1726184, "author_name": "Dranzer", "author_url": "", "post_date": "2022-03-17T18:07:44.137000", "content": "

Congrats Amazing Work

", "votes": 1, "replies": [] }, { "id": 1725666, "author_name": "Khushboo", "author_url": "", "post_date": "2022-03-17T09:55:23.617000", "content": "

Thank you so much Chris. You've got a new follower to go through your work. ;)

", "votes": 1, "replies": [] }, { "id": 1725424, "author_name": "Niek van der Zwaag", "author_url": "", "post_date": "2022-03-17T04:49:03.590000", "content": "

Really appreciate your team's efforts in explaining your thought process and approach - congrats on 2nd 🎉

", "votes": 1, "replies": [] }, { "id": 1725128, "author_name": "Trushant Kalyanpur", "author_url": "", "post_date": "2022-03-16T20:39:47.723000", "content": "

Using WBF for ensembling spans is brilliant! Great work!

", "votes": 1, "replies": [] }, { "id": 1725120, "author_name": "CroDoc", "author_url": "", "post_date": "2022-03-16T20:31:40.967000", "content": "

@cdeotte your last link does not point to the inference kernel?

", "votes": 1, "replies": [ { "id": 1725124, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-16T20:35:47.477000", "content": "

It should work now. It will display a quick save and we're running now too. We will also share the datasets later today. Congratulations CroDoc and team for you great finish and big jump in last few days!

", "votes": 3, "replies": [] }, { "id": 1725132, "author_name": "CroDoc", "author_url": "", "post_date": "2022-03-16T20:49:48.840000", "content": "

It works now. Thank you Chris! It was a crazy ride till the end.

\n

Congrats to your team. Especially to @leecming. Yellow circle is on its way 😎

", "votes": 2, "replies": [] } ] }, { "id": 1731824, "author_name": "Clear n' Simple", "author_url": "", "post_date": "2022-03-22T18:12:12.813000", "content": "

Congratulations @cdeotte for making it into top 10 competitions ranking for 1st time!

\n

Best of luck continuing to rise!
\nIt's nice when the #1 contributor for kagglers of all stages does well.

", "votes": 2, "replies": [ { "id": 1733066, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-24T01:28:43.840000", "content": "

Thanks @jacob34 I'm excited to enter competitions top 10. My favorite thing on Kaggle is discussions and notebooks, but recently I've been working hard on competitions.

", "votes": 1, "replies": [] } ] }, { "id": 1725730, "author_name": "xbustc", "author_url": "", "post_date": "2022-03-17T11:26:40.857000", "content": "

congrats and thanks , learn a lot from you.

", "votes": 2, "replies": [] }, { "id": 1725705, "author_name": "AK", "author_url": "", "post_date": "2022-03-17T10:56:18.980000", "content": "

Kudos to you and your team. Nicely done…
\nThanks for the detailed writeup as always 🖖

", "votes": 2, "replies": [] }, { "id": 1725691, "author_name": "小白Lan", "author_url": "", "post_date": "2022-03-17T10:36:56.153000", "content": "

amazing work, thx for ur selfless sharing

", "votes": 2, "replies": [] }, { "id": 1725475, "author_name": "kwchen", "author_url": "", "post_date": "2022-03-17T06:19:49.223000", "content": "

Congratulations!
\nFor deberta-v3-large I got worse CV than the longformer-large.
\nI trained 5 fold, each fold has CV around 0.680. I trained 5 fold using the same folds from abhishek here:
\nhttps://www.kaggle.com/abhishek/creating-folds-properly-hopefully-p
\nwith his training code here:
\nhttps://github.com/abhishekkrthakur/long-text-token-classification
\nI also converted the deberta tokenizers using @nbroad's code.
\nI used max_len 1536, batch_size 2, and I tried learning rates 1e-5, 2e-6, 1e-6, 2e-7, using argument --epochs 10, and I would stop training after 3~5 epochs.

\n

Any idea why I got worse CV than yours? I am guessing it's either batch size, learning rate, or because I used 5 folds.

", "votes": 2, "replies": [ { "id": 1725624, "author_name": "Udbhav Bamba", "author_url": "", "post_date": "2022-03-17T09:07:35.947000", "content": "

I guess it could be because of the offset mapping each tokenizer has different way of generating offsets (some include spaces and some don't) and it will effect your NER labels and predictions, especially for the B-tag as they are more rare.

", "votes": 4, "replies": [] } ] }, { "id": 1725217, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-03-16T23:21:56.723000", "content": "

Congratz @cdeotte @leecming @ubamba98 !
\nThat 2nd place is no surprise, I was looking forward to your solution :)

", "votes": 2, "replies": [] }, { "id": 1725142, "author_name": "Kishal Mandal", "author_url": "", "post_date": "2022-03-16T20:59:41.450000", "content": "

Congrats Chris, you never fail to amaze :D

", "votes": 2, "replies": [ { "id": 1725144, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-16T21:04:12.647000", "content": "

Thanks! I could not have done it without my awesome teammates Chun Ming Lee and Udbhav Bamba

", "votes": 2, "replies": [] } ] }, { "id": 1840542, "author_name": "Allohvk", "author_url": "", "post_date": "2022-07-02T10:46:56.530000", "content": "

Did you face any issues customizing trainer API? For one, I think it needs valid_batch_size = 1 else there seems to be an error in its code for the last batch. The other issue is in my case the predictions returned to the compute_metrics (via the EvalPrediction) is a blank array. Any suggestions would immensely help

", "votes": 0, "replies": [] }, { "id": 1765966, "author_name": "Mi Jiaxin", "author_url": "", "post_date": "2022-04-24T04:04:11.947000", "content": "

Congratulations, your sharing has helped me a lot. I have a question about how to get the best thresholds of different classes.

", "votes": 0, "replies": [ { "id": 1765970, "author_name": "Udbhav Bamba", "author_url": "", "post_date": "2022-04-24T04:11:57.723000", "content": "

You can choose them based on your validation

", "votes": 1, "replies": [] } ] }, { "id": 1735339, "author_name": "", "author_url": "", "post_date": "2022-03-26T06:10:08.957000", "content": "", "votes": 1, "replies": [] }, { "id": 1727668, "author_name": "", "author_url": "", "post_date": "2022-03-18T06:55:39.203000", "content": "", "votes": 0, "replies": [] }, { "id": 1734353, "author_name": "", "author_url": "", "post_date": "2022-03-25T08:44:40.327000", "content": "", "votes": 1, "replies": [] }, { "id": 1734264, "author_name": "", "author_url": "", "post_date": "2022-03-25T06:43:43.143000", "content": "", "votes": 1, "replies": [] }, { "id": 1734235, "author_name": "", "author_url": "", "post_date": "2022-03-25T05:51:49.607000", "content": "", "votes": 1, "replies": [] }, { "id": 1733106, "author_name": "", "author_url": "", "post_date": "2022-03-24T02:35:36.353000", "content": "", "votes": 1, "replies": [] }, { "id": 1725886, "author_name": "", "author_url": "", "post_date": "2022-03-17T13:50:16.643000", "content": "", "votes": 1, "replies": [] }, { "id": 1787055, "author_name": "", "author_url": "", "post_date": "2022-05-13T14:40:05.043000", "content": "", "votes": 0, "replies": [] }, { "id": 1727849, "author_name": "", "author_url": "", "post_date": "2022-03-18T11:40:40.370000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 59, "messages": [ { "id": 1725117, "postDate": "2022-03-16T20:26:11.873Z", "content": "

Second Place Solution

\n

Thank you Georgia State University, The Learning Agency Lab, and Kaggle for an awesome competition. The data was high quality and interesting. The metric and train test split were well done. This competition was a success and will benefit a good cause.

\n

Teaming with Chun Ming Lee and Udbhav Bamba has been wonderful. I learned more about NLP than if I had taken an online course. These two Kagglers are grandmasters at NLP and I now know how to use HuggingFace Trainer efficiently and effectively. I also know how to customize models and losses. Thank you @leecming @ubamba98

\n

Solution Summary

\n

The secret sauce to our solution is powerful post process by Chun Ming (boost CV LB!), a huge variety of models implemented by Udbhav (many long sequence models!), and weighted box fusion by Chris (used in my previous comp here). Everything was fine tuned on local CV and achieved 2nd Place $35,000!

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Hugging Face ModelCVPublic LBPrivate LBspecial
microsoft/deberta-large706710721trained with 100% train data
microsoft/deberta-large699706719add lstm, add jaccard loss
microsoft/deberta-v3-large694697709convert slow tokenizer to fast
microsoft/deberta-xlarge708704713
microsoft/deberta-v2-xlarge699700716convert slow tokenizer to fast
allenai/longformer-large-4096702705716add lstm head
LSG converted roberta703702714convert 512 roberta to 1536
funnel-transformer/large688689708
google/bigbird-roberta-base675676692train 1024 infer 1024
uw-madison/yoso-4096652655668lsh_backward=False
\n

Weighted Box Fusion - CV 741, Public 727, Private 740

\n

Our final solution contained the 10 models listed above. We included 3 out of 10 K-folds for each of the 10 models. Our 8 hour 30 minute submission inferred 27 models! (we removed 3 folds from 30)

\n

Weighted Box Fusion by @zfturbo from his GitHub here was the magic that combined all the model predictions together. Individual models had average CV around 700 and the WBF ensemble achieved CV 741. (Folds from same models were averaged before WBF).

\n

\"\"

\n

Weighted Box Fusion can do something that no other ensemble method can do. If one model has prediction string 8 9 10 11 and another model has 10 11 12 13. Then weighted box fusion takes the average of the two starts, (8 + 10)/2 = 9 and the average of the two ends, (11 + 13)/2 = 12 resulting in 9 10 11 12. Furthermore, WBF reads submission.csv files and doesn't care what tokenizer you used, so its easy to use. (Note we add an extra column to submission dataframe with span confidence).

\n

If we averaged the token/word probabilities, or used a voting classifier, we would either get the union 8 9 10 11 12 13 or the intersection 10 11. Neither of which we want. Also averaging two \"BIO\" predictions gives a new prediction with two different \"B\" which we do not want either.

\n

Post Process

\n

Chun Ming is the mastermind behind our team's strong PP. We applied the same PP to each model before WBF. Our PP code is posted here and Chun Ming will now describe it. Our PP significantly boosted CV LB!

\n

We applied heavy post-processing to the word-level soft predictions each model made (after averaging token probabilities across folds and before the WBF ensembler). Overall CV was improved by ~.008.

\n

In order of descending impact on CV:

\n
    \n

  1. Repairing span predictions
    \nSince we trained using cross-entropy loss at the token level, the raw predictions would often result in broken spans. For example, given a chain of Lead token predictions, there might be a misprediction of an “Other” in the middle. We had a rule to convert that “Other” back to Lead.

    2. \n

  2. Discourse-specific rules
    \nWe came up with common sense heuristics for the discourses. For example, for Lead, Position, and Concluding Statement - there should only be a maximum of one each for a text. We’d predict the most probable candidate for each or merge close duplicate spans.

    3. \n

  3. Adjusting lengths of predicted spans
    \nWe adjusted lengths of spans based on their original predicted length. For example, for a predicted Evidence span that was less than 45 words long, we’d shift the start of the predicted span back by 9 words. These rules and the improvement in LB/CV suggest we were taking advantage of the evaluation metric only requiring a 50% overlap in prediction and label to be counted as correct.

    4. \n
\n

It’s not sexy but sitting down and eye-balling model predictions and comparing them with ground truths using a tool like displacy was rewarding in helping us better understand the task, how our models were behaving, and how to improve model predictions.

\n

Model Details

\n

Surprisingly many more models than LongFormer and BigBird could be trained with sequence lengths greater than 512. All our models were trained with sequence length 1536 and inferred at 1536 (except BigBird-base at 1024).

\n

The model DeBERTa accepts any input size, so we just train with 1536 as is.

\n

The model Funnel-transformer can accept any size after updating its config.

\n
model = AutoModelForTokenClassification.from_pretrained(\n    'funnel-transformer/large', num_labels=15,\n    max_position_embeddings=1536)\n
\n

For BigBird, we used full attention

\n
model = AutoModelForTokenClassification.from_pretrained(\n    'google/bigbird-roberta-base', num_labels=15,\n    attention_type=\"original_full\")\n
\n

For YOSO, we disabled lsh_backward

\n
model = AutoModelForTokenClassification.from_pretrained(\n    'uw-madison/yoso-4096', num_labels=15,\n    lsh_backward=False)\n
\n

Model Training

\n

Thank you Nvidia for providing us some V100 32GB GPUs and A100 40GB GPUs. For each model, we trained 6 out of 10 KFolds and built our CV from the resultant 60% train data OOF. Each fold would take approximately 4 hours on 1xGPU and required between 3 and 5 epochs to maximize val score.

\n

For two models, we used my train public notebooks here and here. For the other eight models, we used Hugging Face Trainer. Most models, used the following parameters. And we added @robikscube Kaggle metric here and our PP directly to the Trainer evaluation, so we could watch the Kaggle metric score every 500 steps and save the model weights with best Kaggle score.

\n

Notice how we use FP16, gradient_accumulation_steps, and gradient_checkpointing. With these settings, you can basically train any size model on any GPU including Kaggle's P100 16GB GPU.

\n
from transformers import TrainingArguments, Trainer\nargs = TrainingArguments( NAME,\n                     PRETRAINED_MODEL,\n                     evaluation_strategy = 'steps',\n                     eval_steps = 500,\n                     dataloader_num_workers=8,\n                     warmup_ratio=0,\n                     lr_scheduler_type = 'linear',\n                     learning_rate = 2e-5,\n                     log_level = 'warning',\n                     fp16 = True,\n                     per_device_train_batch_size = 2,\n                     per_device_eval_batch_size = 2,\n                     gradient_accumulation_steps = 2,\n                     gradient_checkpointing = True,\n                     num_train_epochs = 5,\n                     save_strategy = 'no',\n                     save_total_limit = 1)\n\ntrainer = Trainer(model,\n              args,\n              train_dataset=train_dataset,\n              eval_dataset=valid_dataset,\n              compute_metrics=KaggleMetric(valid_df, valid_dataset),\n              callbacks=[SaveBestModelCallback],\n              data_collator=data_collator,\n              tokenizer=tokenizer)\n
\n

Inference Code with PP with WBF

\n

We posted our final submission inference code on Kaggle here. It includes our post process and weighted box fusion implementation. Enjoy!

\n

Training Code on GitHub

\n

Our training code is published on GitHub here

", "rawMarkdown": "# Second Place Solution\nThank you Georgia State University, The Learning Agency Lab, and Kaggle for an awesome competition. The data was high quality and interesting. The metric and train test split were well done. This competition was a success and will benefit a good cause.\n\nTeaming with Chun Ming Lee and Udbhav Bamba has been wonderful. I learned more about NLP than if I had taken an online course. These two Kagglers are grandmasters at NLP and I now know how to use HuggingFace Trainer efficiently and effectively. I also know how to customize models and losses. Thank you @leecming @ubamba98 \n\n# Solution Summary\nThe secret sauce to our solution is powerful **post process** by Chun Ming (boost CV LB!), a huge **variety of models** implemented by Udbhav (many long sequence models!), and **weighted box fusion** by Chris (used in my previous comp [here][2]). Everything was fine tuned on local CV and achieved 2nd Place $35,000!\n\n| Hugging Face Model | CV | Public LB | Private LB | special |\n| --- | --- | | | |\n| microsoft/deberta-large | 706 | 710 | 721 | trained with 100% train data |\n| microsoft/deberta-large | 699 | 706 | 719 | add lstm, add jaccard loss |\n| microsoft/deberta-v3-large | 694 | 697 | 709 | [convert slow tokenizer to fast][9] |\n| microsoft/deberta-xlarge | 708 | 704 | 713 | |\n| microsoft/deberta-v2-xlarge | 699 | 700 | 716 | [convert slow tokenizer to fast][9] |\n| allenai/longformer-large-4096 | 702 | 705 | 716 | add lstm head|\n| [LSG converted roberta][1] | 703 | 702 | 714 | convert 512 roberta to 1536 |\n| funnel-transformer/large | 688 | 689 | 708 |\n| google/bigbird-roberta-base | 675 | 676 | 692 | train 1024 infer 1024 |\n| uw-madison/yoso-4096 | 652 | 655 | 668 | lsh_backward=False |\n\n# Weighted Box Fusion - CV 741, Public 727, Private 740\nOur final solution contained the 10 models listed above. We included 3 out of 10 K-folds for each of the 10 models. Our 8 hour 30 minute submission inferred 27 models! (we removed 3 folds from 30)\n\nWeighted Box Fusion by @zfturbo from his GitHub [here][3] was the magic that combined all the model predictions together. Individual models had average CV around 700 and the WBF ensemble achieved CV 741. (Folds from same models were averaged before WBF).\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Mar-2022/wbf.png)\n\nWeighted Box Fusion can do something that no other ensemble method can do. If one model has prediction string `8 9 10 11` and another model has `10 11 12 13`. Then weighted box fusion takes the average of the two starts, `(8 + 10)/2 = 9` and the average of the two ends, `(11 + 13)/2 = 12` resulting in `9 10 11 12`. Furthermore, WBF reads `submission.csv` files and doesn't care what tokenizer you used, so its easy to use. (Note we add an extra column to submission dataframe with span confidence).\n\nIf we averaged the token/word probabilities, or used a voting classifier, we would either get the union `8 9 10 11 12 13` or the intersection `10 11`. Neither of which we want. Also averaging two \"BIO\" predictions gives a new prediction with two different \"B\" which we do not want either.\n\n# Post Process\nChun Ming is the mastermind behind our team's strong PP. We applied the same PP to each model before WBF. Our PP code is posted [here][10] and Chun Ming will now describe it. Our PP significantly boosted CV LB!\n\nWe applied heavy post-processing to the word-level soft predictions each model made (after averaging token probabilities across folds and before the WBF ensembler). Overall CV was improved by ~.008. \n\nIn order of descending impact on CV: \n1. Repairing span predictions\n Since we trained using cross-entropy loss at the token level, the raw predictions would often result in broken spans. For example, given a chain of Lead token predictions, there might be a misprediction of an “Other” in the middle. We had a rule to convert that “Other” back to Lead. \n\n2. Discourse-specific rules\n We came up with common sense heuristics for the discourses. For example, for Lead, Position, and Concluding Statement - there should only be a maximum of one each for a text. We’d predict the most probable candidate for each or merge close duplicate spans. \n\n3. Adjusting lengths of predicted spans\n We adjusted lengths of spans based on their original predicted length. For example, for a predicted Evidence span that was less than 45 words long, we’d shift the start of the predicted span back by 9 words. These rules and the improvement in LB/CV suggest we were taking advantage of the evaluation metric only requiring a 50% overlap in prediction and label to be counted as correct.\n\nIt’s not sexy but sitting down and eye-balling model predictions and comparing them with ground truths using a tool like displacy was rewarding in helping us better understand the task, how our models were behaving, and how to improve model predictions. \n\n# Model Details\nSurprisingly many more models than LongFormer and BigBird could be trained with sequence lengths greater than 512. All our models were trained with sequence length 1536 and inferred at 1536 (except BigBird-base at 1024).\n\nThe model DeBERTa accepts any input size, so we just train with 1536 as is. \n\nThe model Funnel-transformer can accept any size after updating its config.\n\n model = AutoModelForTokenClassification.from_pretrained(\n 'funnel-transformer/large', num_labels=15,\n max_position_embeddings=1536)\n\nFor BigBird, we used full attention\n\n model = AutoModelForTokenClassification.from_pretrained(\n 'google/bigbird-roberta-base', num_labels=15,\n attention_type=\"original_full\")\n\nFor YOSO, we disabled `lsh_backward`\n\n model = AutoModelForTokenClassification.from_pretrained(\n 'uw-madison/yoso-4096', num_labels=15,\n lsh_backward=False)\n\n# Model Training\nThank you Nvidia for providing us some V100 32GB GPUs and A100 40GB GPUs. For each model, we trained 6 out of 10 KFolds and built our CV from the resultant 60% train data OOF. Each fold would take approximately 4 hours on 1xGPU and required between 3 and 5 epochs to maximize val score. \n\nFor two models, we used my train public notebooks [here][7] and [here][8]. For the other eight models, we used Hugging Face Trainer. Most models, used the following parameters. And we added @robikscube Kaggle metric [here][6] and our PP directly to the Trainer evaluation, so we could watch the Kaggle metric score every 500 steps and save the model weights with best Kaggle score.\n\nNotice how we use FP16, gradient_accumulation_steps, and gradient_checkpointing. With these settings, you can basically train any size model on any GPU including Kaggle's P100 16GB GPU.\n\n from transformers import TrainingArguments, Trainer\n args = TrainingArguments( NAME,\n PRETRAINED_MODEL,\n evaluation_strategy = 'steps',\n eval_steps = 500,\n dataloader_num_workers=8,\n warmup_ratio=0,\n lr_scheduler_type = 'linear',\n learning_rate = 2e-5,\n log_level = 'warning',\n fp16 = True,\n per_device_train_batch_size = 2,\n per_device_eval_batch_size = 2,\n gradient_accumulation_steps = 2,\n gradient_checkpointing = True,\n num_train_epochs = 5,\n save_strategy = 'no',\n save_total_limit = 1)\n\n trainer = Trainer(model,\n args,\n train_dataset=train_dataset,\n eval_dataset=valid_dataset,\n compute_metrics=KaggleMetric(valid_df, valid_dataset),\n callbacks=[SaveBestModelCallback],\n data_collator=data_collator,\n tokenizer=tokenizer)\n\n# Inference Code with PP with WBF\nWe posted our final submission inference code on Kaggle [here][10]. It includes our post process and weighted box fusion implementation. Enjoy!\n\n# Training Code on GitHub\nOur training code is published on GitHub [here][11]\n\n[1]: https://github.com/ccdv-ai/convert_checkpoint_to_lsg\n[2]: https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/307609\n[3]: https://github.com/ZFTurbo/Weighted-Boxes-Fusion\n[4]: https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\n[5]: https://www.kaggle.com/cdeotte/pytorch-bigbird-ner-cv-0-615\n[6]: https://www.kaggle.com/robikscube/student-writing-competition-twitch-stream\n[7]: https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\n[8]: https://www.kaggle.com/cdeotte/pytorch-bigbird-ner-cv-0-615\n[9]: https://www.kaggle.com/nbroad/deberta-v2-3-fast-tokenizer\n[10]: https://www.kaggle.com/cdeotte/2nd-place-solution-cv741-public727-private740\n[11]: https://github.com/ubamba98/feedback-prize", "votes": 203 }, { "id": 1725276, "postDate": "2022-03-17T01:14:46.953Z", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Hugging Face ModelCVPublic LBPrivate LBspecial
microsoft/deberta-large706710721trained with 100% train data
\n

How do you get CV when training on 100% of the data?

", "rawMarkdown": "| Hugging Face Model| CV|Public LB |Private LB | special| \n| --- | --- | --- |--- |--- |\n| microsoft/deberta-large | 706 |\t710 | \t721\t| trained with 100% train data |\n\n\nHow do you get CV when training on 100% of the data?", "votes": 8, "replies": [ { "id": 1725281, "postDate": "2022-03-17T01:21:33.547Z", "content": "

ha, good question. The \"CV score\" is the score when training with 10 K-Fold. First we train with 10 K-Fold, and make note of the learning rate and number of epochs. Second we trained a second time using 100% data. So actually it's two models. The first model was K-Fold and got CV 706. The second model used the same hyperparameters and trained on 100% data and got Public 710 Private 721

", "rawMarkdown": "ha, good question. The \"CV score\" is the score when training with 10 K-Fold. First we train with 10 K-Fold, and make note of the learning rate and number of epochs. Second we trained a second time using 100% data. So actually it's two models. The first model was K-Fold and got CV 706. The second model used the same hyperparameters and trained on 100% data and got Public 710 Private 721", "votes": 9 }, { "id": 1725283, "postDate": "2022-03-17T01:23:32.277Z", "content": "

@nbroad thanks for all your great work at Hugging Face. The NLP library transformers and model repository are fantastic. We also used your code to convert deberta-v2 and v3 slow tokenizers to fast tokenizers. Thanks.

", "rawMarkdown": "@nbroad thanks for all your great work at Hugging Face. The NLP library `transformers` and model repository are fantastic. We also used your code to convert deberta-v2 and v3 slow tokenizers to fast tokenizers. Thanks.", "votes": 9 }, { "id": 1725639, "postDate": "2022-03-17T09:24:15.813Z", "content": "

I greatly appreciate the comment, even though my contributions to transformers are pretty small relative to others.

\n

I'm always happy to share what I learn through Hugging Face with the community here. I think the deberta v2/3 fast tokenizers should be merged to the main branch soon so it won't be so awkward to use on Kaggle.

\n

I forgot to congratulate you! This is another fantastic approach and final rank 😊

", "rawMarkdown": "I greatly appreciate the comment, even though my contributions to `transformers` are pretty small relative to others. \n\nI'm always happy to share what I learn through Hugging Face with the community here. I think the deberta v2/3 fast tokenizers should be merged to the main branch soon so it won't be so awkward to use on Kaggle. \n\nI forgot to congratulate you! This is another fantastic approach and final rank 😊", "votes": 4 }, { "id": 1725709, "postDate": "2022-03-17T11:00:46.857Z", "content": "

'''The first model was K-Fold and got CV 706. The second model used the same hyperparameters and trained on 100% data and got Public 710 Private 721'''

\n

so 706 is out of CV of 10 models(so 10 models are submitted each being one fold of 10), whereas 710 is from a single model (only a single model is submitted) trained from all the data??

\n

Big congrats again!!

", "rawMarkdown": "*'''The first model was K-Fold and got CV 706. The second model used the same hyperparameters and trained on 100% data and got Public 710 Private 721'''*\n\nso 706 is out of CV of 10 models(so 10 models are submitted each being one fold of 10), whereas 710 is from a single model (only a single model is submitted) trained from all the data??\n\nBig congrats again!!" }, { "id": 1732556, "postDate": "2022-03-23T14:05:24.470Z", "content": "
\n

ha, good question. The \"CV score\" is the score when training with 10 K-Fold. First we train with 10 K-Fold, and make note of the learning rate and number of epochs. Second we trained a second time using 100% data. So actually it's two models. The first model was K-Fold and got CV 706. The second model used the same hyperparameters and trained on 100% data and got Public 710 Private 721

\n
\n

@cdeotte Is this technique subjective to just the NLP area or you have used this approach for Computer vision and tabular data as well?

", "rawMarkdown": "> ha, good question. The \"CV score\" is the score when training with 10 K-Fold. First we train with 10 K-Fold, and make note of the learning rate and number of epochs. Second we trained a second time using 100% data. So actually it's two models. The first model was K-Fold and got CV 706. The second model used the same hyperparameters and trained on 100% data and got Public 710 Private 721\n\n@cdeotte Is this technique subjective to just the NLP area or you have used this approach for Computer vision and tabular data as well?" } ] }, { "id": 1725294, "postDate": "2022-03-17T01:36:58.300Z", "content": "

Would you mind public all the training models? Because it's too hard for us to Reappearance it.
\nThanks!

", "rawMarkdown": "Would you mind public all the training models? Because it's too hard for us to Reappearance it.\nThanks!", "votes": 6, "replies": [ { "id": 1725298, "postDate": "2022-03-17T01:40:08.153Z", "content": "

We published our inference code here. And we made all trained models (i.e. Kaggle datasets) public.

", "rawMarkdown": "We published our inference code [here][1]. And we made all trained models (i.e. Kaggle datasets) public.\n\n[1]: https://www.kaggle.com/cdeotte/2nd-place-solution-cv741-public727-private740", "votes": 8 }, { "id": 1725338, "postDate": "2022-03-17T02:35:17.913Z", "content": "

Got it, cheers :)

", "rawMarkdown": "Got it, cheers :)", "votes": 1 } ] }, { "id": 1725135, "postDate": "2022-03-16T20:51:49.637Z", "content": "

Great work! The ensemble use WBF of each model only, not using per word soft logits/prob merge?

", "rawMarkdown": "Great work! The ensemble use WBF of each model only, not using per word soft logits/prob merge?", "votes": 3, "replies": [ { "id": 1725137, "postDate": "2022-03-16T20:53:56.140Z", "content": "

For different folds but same architecture, we average the probabilities and for different architectures we use WBF.

", "rawMarkdown": "For different folds but same architecture, we average the probabilities and for different architectures we use WBF.", "votes": 3 }, { "id": 1725139, "postDate": "2022-03-16T20:55:54.330Z", "content": "

Cool! Thanks!

", "rawMarkdown": "Cool! Thanks!", "votes": 2 } ] }, { "id": 1725234, "postDate": "2022-03-17T00:14:50.640Z", "content": "

Nice solution! Congrats @cdeotte and team. I will try your WBF method in the next NLP competition ;)

", "rawMarkdown": "Nice solution! Congrats @cdeotte and team. I will try your WBF method in the next NLP competition ;)", "votes": 4 }, { "id": 1735972, "postDate": "2022-03-26T18:56:51.920Z", "content": "

Congratulations, thanks for sharing this

", "rawMarkdown": "Congratulations, thanks for sharing this ", "votes": 1 }, { "id": 1735423, "postDate": "2022-03-26T08:27:41.333Z", "content": "

Congratulations on this solution @cdeotte and team 🧐
\nWell explained as always 👍
\nKeep up the good work!

", "rawMarkdown": "Congratulations on this solution @cdeotte and team 🧐\nWell explained as always 👍\nKeep up the good work!", "votes": 1, "replies": [ { "id": 1736048, "postDate": "2022-03-26T21:41:42.450Z", "content": "

Thanks FPiotro

", "rawMarkdown": "Thanks FPiotro" } ] }, { "id": 1734398, "postDate": "2022-03-25T09:56:52.950Z", "content": "

congrats.. Great work, thanks for sharing

", "rawMarkdown": "congrats.. Great work, thanks for sharing ", "votes": 1 }, { "id": 1733310, "postDate": "2022-03-24T07:13:14.743Z", "content": "

Congrats and thanks for sharing!!!
\nDid post processing make a big difference to improve final score? Or an enough large model like deberta and with enough training time can bring a impressive result.

", "rawMarkdown": "Congrats and thanks for sharing!!!\nDid post processing make a big difference to improve final score? Or an enough large model like deberta and with enough training time can bring a impressive result.", "votes": 1, "replies": [ { "id": 1735359, "postDate": "2022-03-26T06:29:45.980Z", "content": "

Post processing made a large difference. Probably around +0.010

", "rawMarkdown": "Post processing made a large difference. Probably around +0.010" } ] }, { "id": 1733060, "postDate": "2022-03-24T01:18:38.210Z", "content": "

It is really a great code.

", "rawMarkdown": "It is really a great code.", "votes": 1, "replies": [ { "id": 1733064, "postDate": "2022-03-24T01:25:14.227Z", "content": "

Thanks Jingyuan

", "rawMarkdown": "Thanks Jingyuan" } ] }, { "id": 1732976, "postDate": "2022-03-23T22:46:32.777Z", "content": "

Wow..
\nCongratulations ~!!

", "rawMarkdown": "Wow..\nCongratulations ~!! ", "votes": 1 }, { "id": 1732589, "postDate": "2022-03-23T14:34:42.820Z", "content": "

Great work, thanks for sharing details. This post process is looking great.

", "rawMarkdown": "Great work, thanks for sharing details. This post process is looking great. ", "votes": 1 }, { "id": 1732005, "postDate": "2022-03-22T23:21:57.423Z", "content": "

Congratulations @cdeotte

", "rawMarkdown": "Congratulations @cdeotte", "votes": 1 }, { "id": 1727990, "postDate": "2022-03-18T14:02:03.570Z", "content": "

Great work. Thanks for sharing the code.

", "rawMarkdown": "Great work. Thanks for sharing the code.", "votes": 1 }, { "id": 1727879, "postDate": "2022-03-18T12:13:18.350Z", "content": "
\n

Adjusting lengths of predicted spans
\n We adjusted lengths of spans based on their original predicted length. For example, for a predicted Evidence span that was less than 45 words long, we’d shift the start of the predicted span back by 9 words. These rules and the improvement in LB/CV suggest we were taking advantage of the evaluation metric only requiring a 50% overlap in prediction and label to be counted as correct.

\n
\n

@leecming ,
\nHow did you decide how many words to shift the predictions by?

", "rawMarkdown": ">Adjusting lengths of predicted spans\nWe adjusted lengths of spans based on their original predicted length. For example, for a predicted Evidence span that was less than 45 words long, we’d shift the start of the predicted span back by 9 words. These rules and the improvement in LB/CV suggest we were taking advantage of the evaluation metric only requiring a 50% overlap in prediction and label to be counted as correct.\n\n@leecming ,\nHow did you decide how many words to shift the predictions by?", "votes": 1, "replies": [ { "id": 1727885, "postDate": "2022-03-18T12:19:54.667Z", "content": "

Optimized against CV.

\n

I was somewhat paranoid about these rules and pushed for our second sub to use a more conservative subset, excluding the length adjustments. Turns out it was unnecessary as CV/LB showed a strong correlation extending to private LB.

", "rawMarkdown": "Optimized against CV.\n\nI was somewhat paranoid about these rules and pushed for our second sub to use a more conservative subset, excluding the length adjustments. Turns out it was unnecessary as CV/LB showed a strong correlation extending to private LB. ", "votes": 2 }, { "id": 1727904, "postDate": "2022-03-18T12:35:22.790Z", "content": "

So did you just try a bunch of shifts (-1, +1, -2, +2, etc.) and see how that impacted the CV?

", "rawMarkdown": "So did you just try a bunch of shifts (-1, +1, -2, +2, etc.) and see how that impacted the CV?", "votes": 1 }, { "id": 1727915, "postDate": "2022-03-18T12:46:10.590Z", "content": "

Broadly yes - we had optimized CV code that used multiprocessing so we could test hyperparam changes very rapidly (<10s per run for 7K OOF samples). I tried using Optuna but found my gut sense better at optimizing :)

\n

Ofc, in hindsight, this post-processing could have been potentially been done by a 2nd level ML model as a number of top teams did.

", "rawMarkdown": "Broadly yes - we had optimized CV code that used multiprocessing so we could test hyperparam changes very rapidly (<10s per run for 7K OOF samples). I tried using Optuna but found my gut sense better at optimizing :)\n\nOfc, in hindsight, this post-processing could have been potentially been done by a 2nd level ML model as a number of top teams did. ", "votes": 1 } ] }, { "id": 1726251, "postDate": "2022-03-17T19:43:30.263Z", "content": "

Great work. Thanks for sharing the code.

", "rawMarkdown": "Great work. Thanks for sharing the code.", "votes": 1 }, { "id": 1726184, "postDate": "2022-03-17T18:07:44.137Z", "content": "

Congrats Amazing Work

", "rawMarkdown": "Congrats Amazing Work", "votes": 1 }, { "id": 1725666, "postDate": "2022-03-17T09:55:23.617Z", "content": "

Thank you so much Chris. You've got a new follower to go through your work. ;)

", "rawMarkdown": "Thank you so much Chris. You've got a new follower to go through your work. ;) ", "votes": 1 }, { "id": 1725424, "postDate": "2022-03-17T04:49:03.590Z", "content": "

Really appreciate your team's efforts in explaining your thought process and approach - congrats on 2nd 🎉

", "rawMarkdown": "Really appreciate your team's efforts in explaining your thought process and approach - congrats on 2nd 🎉", "votes": 1 }, { "id": 1725128, "postDate": "2022-03-16T20:39:47.723Z", "content": "

Using WBF for ensembling spans is brilliant! Great work!

", "rawMarkdown": "Using WBF for ensembling spans is brilliant! Great work! ", "votes": 1 }, { "id": 1725120, "postDate": "2022-03-16T20:31:40.967Z", "content": "

@cdeotte your last link does not point to the inference kernel?

", "rawMarkdown": "@cdeotte your last link does not point to the inference kernel?", "votes": 1, "replies": [ { "id": 1725124, "postDate": "2022-03-16T20:35:47.477Z", "content": "

It should work now. It will display a quick save and we're running now too. We will also share the datasets later today. Congratulations CroDoc and team for you great finish and big jump in last few days!

", "rawMarkdown": "It should work now. It will display a quick save and we're running now too. We will also share the datasets later today. Congratulations CroDoc and team for you great finish and big jump in last few days!", "votes": 3 }, { "id": 1725132, "postDate": "2022-03-16T20:49:48.840Z", "content": "

It works now. Thank you Chris! It was a crazy ride till the end.

\n

Congrats to your team. Especially to @leecming. Yellow circle is on its way 😎

", "rawMarkdown": "It works now. Thank you Chris! It was a crazy ride till the end.\n\nCongrats to your team. Especially to @leecming. Yellow circle is on its way 😎", "votes": 2 } ] }, { "id": 1731824, "postDate": "2022-03-22T18:12:12.813Z", "content": "

Congratulations @cdeotte for making it into top 10 competitions ranking for 1st time!

\n

Best of luck continuing to rise!
\nIt's nice when the #1 contributor for kagglers of all stages does well.

", "rawMarkdown": "Congratulations @cdeotte for making it into top 10 competitions ranking for 1st time!\n\nBest of luck continuing to rise!\nIt's nice when the #1 contributor for kagglers of all stages does well.", "votes": 2, "replies": [ { "id": 1733066, "postDate": "2022-03-24T01:28:43.840Z", "content": "

Thanks @jacob34 I'm excited to enter competitions top 10. My favorite thing on Kaggle is discussions and notebooks, but recently I've been working hard on competitions.

", "rawMarkdown": "Thanks @jacob34 I'm excited to enter competitions top 10. My favorite thing on Kaggle is discussions and notebooks, but recently I've been working hard on competitions.", "votes": 1 } ] }, { "id": 1725730, "postDate": "2022-03-17T11:26:40.857Z", "content": "

congrats and thanks , learn a lot from you.

", "rawMarkdown": "congrats and thanks , learn a lot from you.", "votes": 2 }, { "id": 1725705, "postDate": "2022-03-17T10:56:18.980Z", "content": "

Kudos to you and your team. Nicely done…
\nThanks for the detailed writeup as always 🖖

", "rawMarkdown": "Kudos to you and your team. Nicely done…\nThanks for the detailed writeup as always 🖖", "votes": 2 }, { "id": 1725691, "postDate": "2022-03-17T10:36:56.153Z", "content": "

amazing work, thx for ur selfless sharing

", "rawMarkdown": "amazing work, thx for ur selfless sharing", "votes": 2 }, { "id": 1725475, "postDate": "2022-03-17T06:19:49.223Z", "content": "

Congratulations!
\nFor deberta-v3-large I got worse CV than the longformer-large.
\nI trained 5 fold, each fold has CV around 0.680. I trained 5 fold using the same folds from abhishek here:
\nhttps://www.kaggle.com/abhishek/creating-folds-properly-hopefully-p
\nwith his training code here:
\nhttps://github.com/abhishekkrthakur/long-text-token-classification
\nI also converted the deberta tokenizers using @nbroad's code.
\nI used max_len 1536, batch_size 2, and I tried learning rates 1e-5, 2e-6, 1e-6, 2e-7, using argument --epochs 10, and I would stop training after 3~5 epochs.

\n

Any idea why I got worse CV than yours? I am guessing it's either batch size, learning rate, or because I used 5 folds.

", "rawMarkdown": "Congratulations!\nFor deberta-v3-large I got worse CV than the longformer-large.\nI trained 5 fold, each fold has CV around 0.680. I trained 5 fold using the same folds from abhishek here:\nhttps://www.kaggle.com/abhishek/creating-folds-properly-hopefully-p\nwith his training code here:\nhttps://github.com/abhishekkrthakur/long-text-token-classification\nI also converted the deberta tokenizers using @nbroad's code.\nI used max_len 1536, batch_size 2, and I tried learning rates 1e-5, 2e-6, 1e-6, 2e-7, using argument --epochs 10, and I would stop training after 3~5 epochs.\n\nAny idea why I got worse CV than yours? I am guessing it's either batch size, learning rate, or because I used 5 folds.", "votes": 2, "replies": [ { "id": 1725624, "postDate": "2022-03-17T09:07:35.947Z", "content": "

I guess it could be because of the offset mapping each tokenizer has different way of generating offsets (some include spaces and some don't) and it will effect your NER labels and predictions, especially for the B-tag as they are more rare.

", "rawMarkdown": "I guess it could be because of the offset mapping each tokenizer has different way of generating offsets (some include spaces and some don't) and it will effect your NER labels and predictions, especially for the B-tag as they are more rare.", "votes": 4 } ] }, { "id": 1725217, "postDate": "2022-03-16T23:21:56.723Z", "content": "

Congratz @cdeotte @leecming @ubamba98 !
\nThat 2nd place is no surprise, I was looking forward to your solution :)

", "rawMarkdown": "Congratz @cdeotte @leecming @ubamba98 !\nThat 2nd place is no surprise, I was looking forward to your solution :)", "votes": 2 }, { "id": 1725142, "postDate": "2022-03-16T20:59:41.450Z", "content": "

Congrats Chris, you never fail to amaze :D

", "rawMarkdown": "Congrats Chris, you never fail to amaze :D", "votes": 2, "replies": [ { "id": 1725144, "postDate": "2022-03-16T21:04:12.647Z", "content": "

Thanks! I could not have done it without my awesome teammates Chun Ming Lee and Udbhav Bamba

", "rawMarkdown": "Thanks! I could not have done it without my awesome teammates Chun Ming Lee and Udbhav Bamba", "votes": 2 } ] }, { "id": 1840542, "postDate": "2022-07-02T10:46:56.530Z", "content": "

Did you face any issues customizing trainer API? For one, I think it needs valid_batch_size = 1 else there seems to be an error in its code for the last batch. The other issue is in my case the predictions returned to the compute_metrics (via the EvalPrediction) is a blank array. Any suggestions would immensely help

", "rawMarkdown": "Did you face any issues customizing trainer API? For one, I think it needs valid_batch_size = 1 else there seems to be an error in its code for the last batch. The other issue is in my case the predictions returned to the compute_metrics (via the EvalPrediction) is a blank array. Any suggestions would immensely help" }, { "id": 1765966, "postDate": "2022-04-24T04:04:11.947Z", "content": "

Congratulations, your sharing has helped me a lot. I have a question about how to get the best thresholds of different classes.

", "rawMarkdown": "Congratulations, your sharing has helped me a lot. I have a question about how to get the best thresholds of different classes.", "replies": [ { "id": 1765970, "postDate": "2022-04-24T04:11:57.723Z", "content": "

You can choose them based on your validation

", "rawMarkdown": "You can choose them based on your validation ", "votes": 1 } ] }, { "id": 1735339, "postDate": "2022-03-26T06:10:08.957Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1727668, "postDate": "2022-03-18T06:55:39.203Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1734353, "postDate": "2022-03-25T08:44:40.327Z", "content": "

This is great, and thanks for sharing

", "rawMarkdown": "This is great, and thanks for sharing", "votes": 1 }, { "id": 1734264, "postDate": "2022-03-25T06:43:43.143Z", "content": "

Congrats and thanks for sharing.👌

", "rawMarkdown": "Congrats and thanks for sharing.👌", "votes": 1 }, { "id": 1734235, "postDate": "2022-03-25T05:51:49.607Z", "content": "

Thank you for sharing!

", "rawMarkdown": "Thank you for sharing!", "votes": 1 }, { "id": 1733106, "postDate": "2022-03-24T02:35:36.353Z", "content": "

Congrats and thanks for sharing.

", "rawMarkdown": "Congrats and thanks for sharing.", "votes": 1 }, { "id": 1725886, "postDate": "2022-03-17T13:50:16.643Z", "content": "

Thanks for your result, it's amazing!

", "rawMarkdown": "Thanks for your result, it's amazing!", "votes": 1 }, { "id": 1787055, "postDate": "2022-05-13T14:40:05.043Z", "content": "

Thanks for your insight!

", "rawMarkdown": "Thanks for your insight!" }, { "id": 1727849, "postDate": "2022-03-18T11:40:40.370Z", "content": "

Amazing! Thanks a lot for sharing.

", "rawMarkdown": "Amazing! Thanks a lot for sharing. " } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:38:03Z", "competition_id": 31779, "competition_slug": "feedback-prize-2021", "team_id": 7929547, "solution_url": "/competitions/feedback-prize-2021/writeups/feedforward-2nd-place-weighted-box-fusion-and-post" }