{ "id": 287923, "title": "1st Place Solution", "content": "

Thanks to Kaggle and the hosts for this competition! Thanks to everyone in the community I engaged with in the forums and discussions, I learned a lot from you.

\n

CV Setup

\n

When I'm writing this summary, I only know my public LB rank, so this paragraph may turn bitter. I decided to fully trust public LB in this competition and didn't even track local CV. The train dataset was small and confirmed to be noisy, while the public LB was bigger and contained 3-way annotations, which meant higher quality. I expected a higher correlation between public and private LB than between my local CV and private LB.

\n

Architecture

\n

All my models are standard QA models (AutoModelForQuestionAnswering) and the postprocessing is directly reapplied from the HF example notebook by Sylvain Gugger that I also used in my public EDA. I used XLM-Roberta Large, MURIL Large and Rembert, and blended the predictions from several models per backbone. In the next stage, I used majority voting at word level across the 3 backbones to decide which words will make it to the final prediction.

\n

For diversity, I submitted another ensemble without blending single models, only doing majority voting. It had worse score on public LB, but same on private - both my chosen subs are .787 private. My best private sub is .796. I also have a single model/fold (MURIL) that scored .802 on public LB and .783 on private.

\n

Data Recipes

\n

I started the competition, like most of other participants, with XLM-Roberta Large pretrained by Deepset on English SQUAD and finetuned on chaii data. Later, I started to pretrain my own backbone models (I mean QA pretraining, not MLM). I focused on TyDi dataset, because the host mentioned that chaii data was collected in a similar way. I went through several experiments with different pretraining and finetuning protocols and the key insight was that the performance of pretrained backbone often exceeded its finetuned version. Based on this observation, I decided to drop the 2-stage approach and train my models in 1-stage. However, I wanted to control the order in which I provide different datasets to the model. I call this data recipes because it feels like cooking.

\n

Here is an example of a data recipe that I used in my final experiments:

\n
    \n
  1. Take the English, Bengali and Telugu subset of TyDi dataset and 2/3 of chaii train dataset. Tokenize it with sequence length 256 and remove 90% of negative examples (negative sampling). Add tokenized English SQUAD and Hindi parts of MLQA and XQUAD. Shuffle all together and put aside.
    2. \n
  2. Repeat step 1, but replace English SQUAD with subset of Natural Questions. Use sequece length 384 for tokenization (progressive resizing, see below).
    3. \n
  3. Concatenate the data collected in steps 1 and 2.
    4. \n
  4. Train.
    5. \n
\n

Some more details:

\n\n

Image-Like Augmentations

\n

This part is heavily inspired by fastai - during one of the Deep Learning courses @jhoward talked about his work transfering the pretraining method from Computer Vision to NLP (ULMFiT) and mentioned that there are many other opportunities by following this approach. I focused on the augmentation techniques.

\n

Random Cropping

\n

This is a standard augmentation in CV, but I haven't seen it applied to NLP, although it fits perfectly here: we need to chunk a very long text into a fixed token length fragments. Rather than doing it once and showing the same examples in each epoch, why not make it a dynamic and random crop? Initially, I implemented a function that took a random crop with padding around the answer text for positive examples, and a random crop for negative examples. Once I moved to data recipes described above, I realized that the same can be simpler achieved by using a different stride and negative sampling every time I add a dataset to my recipe.

\n

Progressive Resizing

\n

This is another trick that Jeremy teaches in the fastai lectures, applied commonly for images. In case of progressive resizing, I use different sequence lengths during training. E.g. start with 256, then move to 384, then move to 448.

\n

Cutout

\n

In computer vision, we sometimes black out patches of image as augmentation. In NLP, we can do the same by replacing some tokens with [MASK]. I saw that @cdeotte used this technique in Tweet Sentiment competition. I used 0-10% cutout in my models.

\n

Fine-tuning

\n

I didn't experiment too much with different hyperparameters for finetuning, I usually started with the defaults described in each backbone paper and often stuck with that. I did most of my experiments on the XLM-Roberta backbone. MURIL had the best performance on the public LB (I believe due to tokenization), although the training run sometimes collapsed (a restart usually helped).

\n

Here are my default hyperparameters for each backbone (batch size = 8):

\n\n

Post-processing

\n

Other than the standard postprocessing (I didn't modify the default functions from the official HuggingFace QA notebook), there was an opportunity to improve predictions by addressing XLM-Roberta and Rembert tokenization issues. These models tend to merge some punctuation marks with subwords, while annotators often split them. I initially used @abhishek's method of simply stripping punctuation, and in the last day made some modifications similar to the method shared by @nbroad. I add dots after BC/AD, I add parenthesis if included within the stripped prediction etc. I didn't see an impact in my score by adjusting this, but my position shifted from 3 to 2 based on this postprocessing. This didn't work well on private LB though (-0.004). I didn't do any such postprocessing for MURIL because it handled it well with its tokenizer.

\n

Frameworks

\n

I went all in HuggingFace in this challenge and can't imaging doing it another way. Other than AutoModelForQuestionAnswering, I used the HF Trainer initially, and later realized they have a QA version of the trainer, which has EM/F1 measures implemented for validation. I used EM/F1 for early stopping rather than observing the validation loss. Last but not least, HF datasets was a huge enabler for me, its caching works like magic.

\n

Things that Didn't Work

\n

Label Error Corrections

\n

Quite early in the competition, I tried to correct the noisy labels by doing out of fold predictions, looking at the model errors and replacing the label with prediction if - based on Google Translate - I thought it's better than original. However, this degraded my performance on public LB and I dropped this. I missed the opportunity to correct the answer start, I only tried it in the last 2 days after @zacchaeus published his corrected dataset. The models trained with it appeared to perform better, but they didn't help my ensemble at the end.

\n

MT5

\n

I was quite bullish on MT5, especially when I tried a pretrained checkpoint from HF Hub and it scored .713 without any finetuning. Also, it looks like MT5 XXL is SOTA for this type of challenges with limited data. However, the largest model I was able to run in Kaggle kernel was MT5-Large (is it possible to run a bigger one?). I got MT5-Large to train on Google Colab (thanks to AdaFactor) as I was travelling during this time, but I couldn't surpass the .713 score of the pretrained backbone.

\n

Jaccard Loss

\n

I implemented my own version of jaccard loss (Lovasz-Hinge) but it didn't improve my performance, even when combined with CrossEntropy. It's very likely that I made some implementation error, if you'd like to take a look and give me feedback I'd be happy to share that implementation.

\n

XLNet Head

\n

I also tried the XLNet head where end-logits are predicted by taking into account start-logits, using beam-search on inference. It didn't work too well for me, could be another implementation error.

\n

Tokenization

\n

I didn't venture to tamper with the tokenizers, but I tried to split some punctuation marks before running the tokenizer - this didn't help my score either.

\n

Other

\n

Music

\n

I listened to live concert recordings of 2 Polish rock bands during my coding sessions. Here are some samples if you want to check them out:

\n\n

GPU

\n

I have a rig with 1 RTX 3090 and a Colab Pro account.

\n

Break

\n

I started the competition in the first hours and prepared an EDA, baseline and QA tutorial. Then I took a break and then really focused in the last month of the competition. I used all subs I had available during that time except for 2 that I missed when DST change messed up with my submission plans.

\n

Links

\n

Inference Kernel
\nGithub Repo

", "comments": [ { "id": 1798582, "author_name": "Hảo Tiêu Kim", "author_url": "", "post_date": "2022-05-23T06:56:06.597000", "content": "

😮😮😮 Your code is really motivating because it is not hard to understand.

", "votes": 2, "replies": [] }, { "id": 1594756, "author_name": "numpy pandas1234", "author_url": "", "post_date": "2021-11-25T05:33:02.837000", "content": "

congratulations

", "votes": 1, "replies": [] }, { "id": 1594352, "author_name": "James", "author_url": "", "post_date": "2021-11-24T18:08:48.753000", "content": "

congratulations

", "votes": 1, "replies": [] }, { "id": 1593500, "author_name": "Alexandr Gusev", "author_url": "", "post_date": "2021-11-24T02:22:26.857000", "content": "

Congratulations

", "votes": 1, "replies": [] }, { "id": 1593249, "author_name": "NataeSz", "author_url": "", "post_date": "2021-11-23T18:12:53.380000", "content": "

Great work, congratulations!

", "votes": 1, "replies": [] }, { "id": 1592910, "author_name": "Hammad Maqbool", "author_url": "", "post_date": "2021-11-23T13:11:48.430000", "content": "

Congratulations

", "votes": 1, "replies": [] }, { "id": 1592863, "author_name": "Rustam Fakhretdinov", "author_url": "", "post_date": "2021-11-23T12:27:32.317000", "content": "

Great work! Congratulations!!

", "votes": 1, "replies": [] }, { "id": 1592016, "author_name": "Adhish P", "author_url": "", "post_date": "2021-11-22T19:49:50.443000", "content": "

Congrats!!!

", "votes": 1, "replies": [] }, { "id": 1591413, "author_name": "afioni", "author_url": "", "post_date": "2021-11-22T10:03:07.090000", "content": "

Great! Keep on the Good work.

", "votes": 1, "replies": [] }, { "id": 1591325, "author_name": "Vaibhav Channe", "author_url": "", "post_date": "2021-11-22T07:29:43.573000", "content": "

Much informative

", "votes": 1, "replies": [] }, { "id": 1591138, "author_name": "Hangkai Fan", "author_url": "", "post_date": "2021-11-22T02:57:50.150000", "content": "

Congratulation!

", "votes": 1, "replies": [] }, { "id": 1591137, "author_name": "Hangkai Fan", "author_url": "", "post_date": "2021-11-22T02:57:24.467000", "content": "

Congratulation!

", "votes": 1, "replies": [] }, { "id": 1590990, "author_name": "RodrigoAlves", "author_url": "", "post_date": "2021-11-21T20:22:46.600000", "content": "

Congratulations

", "votes": 1, "replies": [] }, { "id": 1590682, "author_name": "MINGOOK KIM", "author_url": "", "post_date": "2021-11-21T15:02:16.417000", "content": "

Congratulations on your achievements.

\n

Congratulations~

", "votes": 1, "replies": [] }, { "id": 1590358, "author_name": "Anandhu H", "author_url": "", "post_date": "2021-11-21T07:00:45.213000", "content": "

Congrats @thedrcat 😃. Nice explanation 👌

", "votes": 1, "replies": [] }, { "id": 1590301, "author_name": "Don D.M. Tadaya", "author_url": "", "post_date": "2021-11-21T04:38:45.693000", "content": "

Whoop whoop!

", "votes": 1, "replies": [] }, { "id": 1589921, "author_name": "ChaX55", "author_url": "", "post_date": "2021-11-20T17:46:00.007000", "content": "

Congratulation!!

", "votes": 1, "replies": [] }, { "id": 1589800, "author_name": "mix060514", "author_url": "", "post_date": "2021-11-20T15:25:00.653000", "content": "

Congratulation!!

", "votes": 1, "replies": [] }, { "id": 1589757, "author_name": "Parv Yadav", "author_url": "", "post_date": "2021-11-20T14:51:44.017000", "content": "

Great work @thedrcat !✨

", "votes": 1, "replies": [] }, { "id": 1589586, "author_name": "Rajarshi Banerjee", "author_url": "", "post_date": "2021-11-20T12:03:18.073000", "content": "

Lovely explanation!

", "votes": 1, "replies": [] }, { "id": 1587723, "author_name": "Sree Vaddi", "author_url": "", "post_date": "2021-11-18T22:18:42.650000", "content": "

nice work, good explanation 👍

", "votes": 1, "replies": [] }, { "id": 1586538, "author_name": "baaka27", "author_url": "", "post_date": "2021-11-18T05:51:47.113000", "content": "

congratulations

", "votes": 1, "replies": [] }, { "id": 1586536, "author_name": "baaka27", "author_url": "", "post_date": "2021-11-18T05:51:25.637000", "content": "

Congratulations!

", "votes": 1, "replies": [] }, { "id": 1586448, "author_name": "Lau Yam Chun", "author_url": "", "post_date": "2021-11-18T05:00:37.410000", "content": "

Just started learning and joining the kaggle things, hope i can understand such thing one day

", "votes": 1, "replies": [ { "id": 1587722, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-11-18T22:17:16.473000", "content": "

When I started on Kaggle, I didn't understand many top solution descriptions to be honest… If you're getting started, I recommend the fastai course.

", "votes": 1, "replies": [] } ] }, { "id": 1585921, "author_name": "Tsuno", "author_url": "", "post_date": "2021-11-17T16:42:32.273000", "content": "

Congratulations

", "votes": 1, "replies": [] }, { "id": 1585890, "author_name": "Ilya Strelnikov", "author_url": "", "post_date": "2021-11-17T16:17:22.547000", "content": "

Congratulations!

", "votes": 1, "replies": [] }, { "id": 1585843, "author_name": "abhi-glitchhg", "author_url": "", "post_date": "2021-11-17T15:36:44.103000", "content": "

congratulations

", "votes": 1, "replies": [] }, { "id": 1585764, "author_name": "s.kyany", "author_url": "", "post_date": "2021-11-17T14:02:58.187000", "content": "

축하해요~!!!!!

", "votes": 1, "replies": [] }, { "id": 1585685, "author_name": "Kazuya Yagihashi", "author_url": "", "post_date": "2021-11-17T12:38:10.903000", "content": "

Congratulation!

", "votes": 1, "replies": [] }, { "id": 1585534, "author_name": "", "author_url": "", "post_date": "2021-11-17T10:10:25.240000", "content": "", "votes": 1, "replies": [] }, { "id": 1585508, "author_name": "", "author_url": "", "post_date": "2021-11-17T09:46:56.947000", "content": "", "votes": 1, "replies": [] }, { "id": 1585194, "author_name": "", "author_url": "", "post_date": "2021-11-17T06:55:37.080000", "content": "", "votes": 1, "replies": [] }, { "id": 1584519, "author_name": "", "author_url": "", "post_date": "2021-11-16T15:13:46.767000", "content": "", "votes": 1, "replies": [] }, { "id": 1584454, "author_name": "", "author_url": "", "post_date": "2021-11-16T14:41:49.150000", "content": "", "votes": 1, "replies": [] }, { "id": 1584291, "author_name": "", "author_url": "", "post_date": "2021-11-16T12:26:11.527000", "content": "", "votes": 1, "replies": [] }, { "id": 1584190, "author_name": "", "author_url": "", "post_date": "2021-11-16T10:37:59.653000", "content": "", "votes": 1, "replies": [] }, { "id": 1584187, "author_name": "", "author_url": "", "post_date": "2021-11-16T10:35:30.787000", "content": "", "votes": 1, "replies": [] }, { "id": 1584137, "author_name": "", "author_url": "", "post_date": "2021-11-16T09:53:47.127000", "content": "", "votes": 1, "replies": [] }, { "id": 1584046, "author_name": "", "author_url": "", "post_date": "2021-11-16T08:30:03.603000", "content": "", "votes": 1, "replies": [ { "id": 1584775, "author_name": "", "author_url": "", "post_date": "2021-11-16T18:54:30.367000", "content": "", "votes": 2, "replies": [] }, { "id": 1585211, "author_name": "", "author_url": "", "post_date": "2021-11-17T07:09:35.133000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1583996, "author_name": "", "author_url": "", "post_date": "2021-11-16T07:35:43.540000", "content": "", "votes": 1, "replies": [] }, { "id": 1583978, "author_name": "", "author_url": "", "post_date": "2021-11-16T07:24:13.620000", "content": "", "votes": 1, "replies": [] }, { "id": 1583922, "author_name": "", "author_url": "", "post_date": "2021-11-16T06:31:21.270000", "content": "", "votes": 1, "replies": [] }, { "id": 1583918, "author_name": "", "author_url": "", "post_date": "2021-11-16T06:27:38.153000", "content": "", "votes": 1, "replies": [ { "id": 1584777, "author_name": "", "author_url": "", "post_date": "2021-11-16T18:56:10.007000", "content": "", "votes": 0, "replies": [] }, { "id": 1585147, "author_name": "", "author_url": "", "post_date": "2021-11-17T05:53:49.217000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1583868, "author_name": "", "author_url": "", "post_date": "2021-11-16T05:43:45.993000", "content": "", "votes": 1, "replies": [] }, { "id": 1583847, "author_name": "", "author_url": "", "post_date": "2021-11-16T05:23:44.427000", "content": "", "votes": 1, "replies": [ { "id": 1584791, "author_name": "", "author_url": "", "post_date": "2021-11-16T19:00:27.143000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1583829, "author_name": "", "author_url": "", "post_date": "2021-11-16T05:10:09.573000", "content": "", "votes": 1, "replies": [] }, { "id": 1583825, "author_name": "", "author_url": "", "post_date": "2021-11-16T05:08:23.633000", "content": "", "votes": 1, "replies": [] }, { "id": 1583758, "author_name": "", "author_url": "", "post_date": "2021-11-16T03:34:29.217000", "content": "", "votes": 1, "replies": [] }, { "id": 1583725, "author_name": "", "author_url": "", "post_date": "2021-11-16T02:59:02.660000", "content": "", "votes": 1, "replies": [] }, { "id": 1583705, "author_name": "", "author_url": "", "post_date": "2021-11-16T02:39:35.827000", "content": "", "votes": 1, "replies": [] }, { "id": 1583692, "author_name": "", "author_url": "", "post_date": "2021-11-16T02:13:28.703000", "content": "", "votes": 1, "replies": [] }, { "id": 1583664, "author_name": "", "author_url": "", "post_date": "2021-11-16T01:42:51.637000", "content": "", "votes": 1, "replies": [] }, { "id": 1583661, "author_name": "", "author_url": "", "post_date": "2021-11-16T01:41:15.327000", "content": "", "votes": 1, "replies": [] }, { "id": 1583642, "author_name": "", "author_url": "", "post_date": "2021-11-16T01:22:42.437000", "content": "", "votes": 1, "replies": [] }, { "id": 1583628, "author_name": "", "author_url": "", "post_date": "2021-11-16T01:14:36.453000", "content": "", "votes": 1, "replies": [] }, { "id": 1583609, "author_name": "", "author_url": "", "post_date": "2021-11-16T01:00:42.757000", "content": "", "votes": 1, "replies": [] }, { "id": 1583603, "author_name": "", "author_url": "", "post_date": "2021-11-16T00:55:11.957000", "content": "", "votes": 1, "replies": [] }, { "id": 1583598, "author_name": "", "author_url": "", "post_date": "2021-11-16T00:47:30.103000", "content": "", "votes": 1, "replies": [] }, { "id": 1587126, "author_name": "", "author_url": "", "post_date": "2021-11-18T13:53:18.547000", "content": "", "votes": 1, "replies": [] }, { "id": 1612010, "author_name": "", "author_url": "", "post_date": "2021-12-08T13:13:11.173000", "content": "", "votes": 2, "replies": [] }, { "id": 1586696, "author_name": "", "author_url": "", "post_date": "2021-11-18T08:18:37.790000", "content": "", "votes": 2, "replies": [ { "id": 1587721, "author_name": "", "author_url": "", "post_date": "2021-11-18T22:14:26.700000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1586396, "author_name": "", "author_url": "", "post_date": "2021-11-18T03:30:45.683000", "content": "", "votes": 2, "replies": [] }, { "id": 1586003, "author_name": "", "author_url": "", "post_date": "2021-11-17T17:50:43.097000", "content": "", "votes": 2, "replies": [ { "id": 1586201, "author_name": "", "author_url": "", "post_date": "2021-11-17T22:14:48.220000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1584395, "author_name": "", "author_url": "", "post_date": "2021-11-16T13:58:46.167000", "content": "", "votes": 2, "replies": [] }, { "id": 1584263, "author_name": "", "author_url": "", "post_date": "2021-11-16T11:55:35.623000", "content": "", "votes": 2, "replies": [] }, { "id": 1583721, "author_name": "", "author_url": "", "post_date": "2021-11-16T02:57:13.387000", "content": "", "votes": 2, "replies": [] }, { "id": 1583686, "author_name": "", "author_url": "", "post_date": "2021-11-16T02:05:01.610000", "content": "", "votes": 2, "replies": [ { "id": 1584770, "author_name": "", "author_url": "", "post_date": "2021-11-16T18:51:56.570000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1583595, "author_name": "", "author_url": "", "post_date": "2021-11-16T00:44:33.020000", "content": "", "votes": 2, "replies": [] }, { "id": 1587109, "author_name": "", "author_url": "", "post_date": "2021-11-18T13:35:47.250000", "content": "", "votes": 0, "replies": [] }, { "id": 1591037, "author_name": "", "author_url": "", "post_date": "2021-11-21T21:59:10.040000", "content": "", "votes": 1, "replies": [] }, { "id": 1584727, "author_name": "", "author_url": "", "post_date": "2021-11-16T18:07:02.897000", "content": "", "votes": 0, "replies": [] }, { "id": 1594339, "author_name": "", "author_url": "", "post_date": "2021-11-24T18:00:21.743000", "content": "", "votes": 1, "replies": [] }, { "id": 1591349, "author_name": "", "author_url": "", "post_date": "2021-11-22T08:19:11.223000", "content": "", "votes": 1, "replies": [] }, { "id": 1587843, "author_name": "", "author_url": "", "post_date": "2021-11-19T01:28:54.360000", "content": "", "votes": 1, "replies": [] }, { "id": 1585643, "author_name": "", "author_url": "", "post_date": "2021-11-17T12:05:19.607000", "content": "", "votes": 1, "replies": [] }, { "id": 1584748, "author_name": "", "author_url": "", "post_date": "2021-11-16T18:16:56.377000", "content": "", "votes": 1, "replies": [] }, { "id": 1583894, "author_name": "", "author_url": "", "post_date": "2021-11-16T06:06:35.277000", "content": "", "votes": 1, "replies": [] }, { "id": 1583853, "author_name": "", "author_url": "", "post_date": "2021-11-16T05:28:25.230000", "content": "", "votes": 1, "replies": [] } ], "comment_count": 87, "messages": [ { "id": 1583590, "postDate": "2021-11-16T00:42:28.250Z", "content": "

Thanks to Kaggle and the hosts for this competition! Thanks to everyone in the community I engaged with in the forums and discussions, I learned a lot from you.

\n

CV Setup

\n

When I'm writing this summary, I only know my public LB rank, so this paragraph may turn bitter. I decided to fully trust public LB in this competition and didn't even track local CV. The train dataset was small and confirmed to be noisy, while the public LB was bigger and contained 3-way annotations, which meant higher quality. I expected a higher correlation between public and private LB than between my local CV and private LB.

\n

Architecture

\n

All my models are standard QA models (AutoModelForQuestionAnswering) and the postprocessing is directly reapplied from the HF example notebook by Sylvain Gugger that I also used in my public EDA. I used XLM-Roberta Large, MURIL Large and Rembert, and blended the predictions from several models per backbone. In the next stage, I used majority voting at word level across the 3 backbones to decide which words will make it to the final prediction.

\n

For diversity, I submitted another ensemble without blending single models, only doing majority voting. It had worse score on public LB, but same on private - both my chosen subs are .787 private. My best private sub is .796. I also have a single model/fold (MURIL) that scored .802 on public LB and .783 on private.

\n

Data Recipes

\n

I started the competition, like most of other participants, with XLM-Roberta Large pretrained by Deepset on English SQUAD and finetuned on chaii data. Later, I started to pretrain my own backbone models (I mean QA pretraining, not MLM). I focused on TyDi dataset, because the host mentioned that chaii data was collected in a similar way. I went through several experiments with different pretraining and finetuning protocols and the key insight was that the performance of pretrained backbone often exceeded its finetuned version. Based on this observation, I decided to drop the 2-stage approach and train my models in 1-stage. However, I wanted to control the order in which I provide different datasets to the model. I call this data recipes because it feels like cooking.

\n

Here is an example of a data recipe that I used in my final experiments:

\n
    \n
  1. Take the English, Bengali and Telugu subset of TyDi dataset and 2/3 of chaii train dataset. Tokenize it with sequence length 256 and remove 90% of negative examples (negative sampling). Add tokenized English SQUAD and Hindi parts of MLQA and XQUAD. Shuffle all together and put aside.
    2. \n
  2. Repeat step 1, but replace English SQUAD with subset of Natural Questions. Use sequece length 384 for tokenization (progressive resizing, see below).
    3. \n
  3. Concatenate the data collected in steps 1 and 2.
    4. \n
  4. Train.
    5. \n
\n

Some more details:

\n\n

Image-Like Augmentations

\n

This part is heavily inspired by fastai - during one of the Deep Learning courses @jhoward talked about his work transfering the pretraining method from Computer Vision to NLP (ULMFiT) and mentioned that there are many other opportunities by following this approach. I focused on the augmentation techniques.

\n

Random Cropping

\n

This is a standard augmentation in CV, but I haven't seen it applied to NLP, although it fits perfectly here: we need to chunk a very long text into a fixed token length fragments. Rather than doing it once and showing the same examples in each epoch, why not make it a dynamic and random crop? Initially, I implemented a function that took a random crop with padding around the answer text for positive examples, and a random crop for negative examples. Once I moved to data recipes described above, I realized that the same can be simpler achieved by using a different stride and negative sampling every time I add a dataset to my recipe.

\n

Progressive Resizing

\n

This is another trick that Jeremy teaches in the fastai lectures, applied commonly for images. In case of progressive resizing, I use different sequence lengths during training. E.g. start with 256, then move to 384, then move to 448.

\n

Cutout

\n

In computer vision, we sometimes black out patches of image as augmentation. In NLP, we can do the same by replacing some tokens with [MASK]. I saw that @cdeotte used this technique in Tweet Sentiment competition. I used 0-10% cutout in my models.

\n

Fine-tuning

\n

I didn't experiment too much with different hyperparameters for finetuning, I usually started with the defaults described in each backbone paper and often stuck with that. I did most of my experiments on the XLM-Roberta backbone. MURIL had the best performance on the public LB (I believe due to tokenization), although the training run sometimes collapsed (a restart usually helped).

\n

Here are my default hyperparameters for each backbone (batch size = 8):

\n\n

Post-processing

\n

Other than the standard postprocessing (I didn't modify the default functions from the official HuggingFace QA notebook), there was an opportunity to improve predictions by addressing XLM-Roberta and Rembert tokenization issues. These models tend to merge some punctuation marks with subwords, while annotators often split them. I initially used @abhishek's method of simply stripping punctuation, and in the last day made some modifications similar to the method shared by @nbroad. I add dots after BC/AD, I add parenthesis if included within the stripped prediction etc. I didn't see an impact in my score by adjusting this, but my position shifted from 3 to 2 based on this postprocessing. This didn't work well on private LB though (-0.004). I didn't do any such postprocessing for MURIL because it handled it well with its tokenizer.

\n

Frameworks

\n

I went all in HuggingFace in this challenge and can't imaging doing it another way. Other than AutoModelForQuestionAnswering, I used the HF Trainer initially, and later realized they have a QA version of the trainer, which has EM/F1 measures implemented for validation. I used EM/F1 for early stopping rather than observing the validation loss. Last but not least, HF datasets was a huge enabler for me, its caching works like magic.

\n

Things that Didn't Work

\n

Label Error Corrections

\n

Quite early in the competition, I tried to correct the noisy labels by doing out of fold predictions, looking at the model errors and replacing the label with prediction if - based on Google Translate - I thought it's better than original. However, this degraded my performance on public LB and I dropped this. I missed the opportunity to correct the answer start, I only tried it in the last 2 days after @zacchaeus published his corrected dataset. The models trained with it appeared to perform better, but they didn't help my ensemble at the end.

\n

MT5

\n

I was quite bullish on MT5, especially when I tried a pretrained checkpoint from HF Hub and it scored .713 without any finetuning. Also, it looks like MT5 XXL is SOTA for this type of challenges with limited data. However, the largest model I was able to run in Kaggle kernel was MT5-Large (is it possible to run a bigger one?). I got MT5-Large to train on Google Colab (thanks to AdaFactor) as I was travelling during this time, but I couldn't surpass the .713 score of the pretrained backbone.

\n

Jaccard Loss

\n

I implemented my own version of jaccard loss (Lovasz-Hinge) but it didn't improve my performance, even when combined with CrossEntropy. It's very likely that I made some implementation error, if you'd like to take a look and give me feedback I'd be happy to share that implementation.

\n

XLNet Head

\n

I also tried the XLNet head where end-logits are predicted by taking into account start-logits, using beam-search on inference. It didn't work too well for me, could be another implementation error.

\n

Tokenization

\n

I didn't venture to tamper with the tokenizers, but I tried to split some punctuation marks before running the tokenizer - this didn't help my score either.

\n

Other

\n

Music

\n

I listened to live concert recordings of 2 Polish rock bands during my coding sessions. Here are some samples if you want to check them out:

\n\n

GPU

\n

I have a rig with 1 RTX 3090 and a Colab Pro account.

\n

Break

\n

I started the competition in the first hours and prepared an EDA, baseline and QA tutorial. Then I took a break and then really focused in the last month of the competition. I used all subs I had available during that time except for 2 that I missed when DST change messed up with my submission plans.

\n

Links

\n

Inference Kernel
\nGithub Repo

", "rawMarkdown": "Thanks to Kaggle and the hosts for this competition! Thanks to everyone in the community I engaged with in the forums and discussions, I learned a lot from you.\n\n## CV Setup\nWhen I'm writing this summary, I only know my public LB rank, so this paragraph may turn bitter. I decided to fully trust public LB in this competition and didn't even track local CV. The train dataset was small and confirmed to be noisy, while the public LB was bigger and contained 3-way annotations, which meant higher quality. I expected a higher correlation between public and private LB than between my local CV and private LB.\n\n## Architecture\nAll my models are standard QA models (AutoModelForQuestionAnswering) and the postprocessing is directly reapplied from the HF example notebook by Sylvain Gugger that I also used in my public EDA. I used XLM-Roberta Large, MURIL Large and Rembert, and blended the predictions from several models per backbone. In the next stage, I used majority voting at word level across the 3 backbones to decide which words will make it to the final prediction.\n\nFor diversity, I submitted another ensemble without blending single models, only doing majority voting. It had worse score on public LB, but same on private - both my chosen subs are .787 private. My best private sub is .796. I also have a single model/fold (MURIL) that scored .802 on public LB and .783 on private.\n\n## Data Recipes\nI started the competition, like most of other participants, with XLM-Roberta Large pretrained by Deepset on English SQUAD and finetuned on chaii data. Later, I started to pretrain my own backbone models (I mean QA pretraining, not MLM). I focused on TyDi dataset, because the host mentioned that chaii data was collected in a similar way. I went through several experiments with different pretraining and finetuning protocols and the key insight was that the performance of pretrained backbone often exceeded its finetuned version. Based on this observation, I decided to drop the 2-stage approach and train my models in 1-stage. However, I wanted to control the order in which I provide different datasets to the model. I call this **data recipes** because it feels like cooking. \n\nHere is an example of a data recipe that I used in my final experiments: \n1. Take the English, Bengali and Telugu subset of TyDi dataset and 2/3 of chaii train dataset. Tokenize it with sequence length 256 and remove 90% of negative examples (negative sampling). Add tokenized English SQUAD and Hindi parts of MLQA and XQUAD. Shuffle all together and put aside. \n2. Repeat step 1, but replace English SQUAD with subset of Natural Questions. Use sequece length 384 for tokenization (progressive resizing, see below). \n3. Concatenate the data collected in steps 1 and 2. \n4. Train. \n\nSome more details:\n- Because I moved the data combination/shuffling logic into the dataset, I used sequential sampler and 1 epoch for training. HuggingFace `datasets` library made this super easy. \n- I used negative sampling, usually 0.1, for TyDi, chaii and NQ datasets because they came with longer contexts.\n- I initially experimented with some auto-translated datasets like Tamil SQUAD but they didn't help my performance, so at the end I only used original language datasets and those translated manually (Hindi MLQA/XQUAD). \n- I tried various languages from TyDi and found that English/Bengali/Telugu worked best.\n- I wanted to add some more data, so I took Google Natural Questions that contained a short answer and trained a simple Roberta-base classifier to distinguish NQ data from TyDi English dataset. I selected a subset of NQ that was most similar to English TyDi. \n- Since I was not tracking CV, I initiatially used all chaii data. Later on, I moved to a 3 fold setup - for diversity and for early stopping. If an interim checkpoint had a much better CV than the final checkpoint, I would usually test it on public LB. \n\n## Image-Like Augmentations\nThis part is heavily inspired by **fastai** - during one of the Deep Learning courses @jhoward talked about his work transfering the pretraining method from Computer Vision to NLP (ULMFiT) and mentioned that there are many other opportunities by following this approach. I focused on the augmentation techniques. \n\n### Random Cropping\nThis is a standard augmentation in CV, but I haven't seen it applied to NLP, although it fits perfectly here: we need to chunk a very long text into a fixed token length fragments. Rather than doing it once and showing the same examples in each epoch, why not make it a dynamic and random crop? Initially, I implemented a function that took a random crop with padding around the answer text for positive examples, and a random crop for negative examples. Once I moved to data recipes described above, I realized that the same can be simpler achieved by using a different stride and negative sampling every time I add a dataset to my recipe. \n\n### Progressive Resizing\nThis is another trick that Jeremy teaches in the fastai lectures, applied commonly for images. In case of progressive resizing, I use different sequence lengths during training. E.g. start with 256, then move to 384, then move to 448. \n\n### Cutout\nIn computer vision, we sometimes black out patches of image as augmentation. In NLP, we can do the same by replacing some tokens with `[MASK]`. I saw that @cdeotte used this technique in Tweet Sentiment competition. I used 0-10% cutout in my models. \n\n## Fine-tuning\nI didn't experiment too much with different hyperparameters for finetuning, I usually started with the defaults described in each backbone paper and often stuck with that. I did most of my experiments on the XLM-Roberta backbone. MURIL had the best performance on the public LB (I believe due to tokenization), although the training run sometimes collapsed (a restart usually helped). \n\nHere are my default hyperparameters for each backbone (batch size = 8):\n* XLM-Roberta: gradient accumulation: 4, learning rate: 1e-5, wd: 0.01\n* MURIL: gradient accumulation: 4, learning rate: 3e-5, wd: 0.01\n* Rembert: gradient accumulation: 16, learning rate: 1e-5, wd: 0.01\n\n## Post-processing\nOther than the standard postprocessing (I didn't modify the default functions from the official HuggingFace QA notebook), there was an opportunity to improve predictions by addressing XLM-Roberta and Rembert tokenization issues. These models tend to merge some punctuation marks with subwords, while annotators often split them. I initially used @abhishek's method of simply stripping punctuation, and in the last day made some modifications similar to the method shared by @nbroad. I add dots after BC/AD, I add parenthesis if included within the stripped prediction etc. I didn't see an impact in my score by adjusting this, but my position shifted from 3 to 2 based on this postprocessing. This didn't work well on private LB though (-0.004). I didn't do any such postprocessing for MURIL because it handled it well with its tokenizer. \n\n## Frameworks\nI went all in HuggingFace in this challenge and can't imaging doing it another way. Other than `AutoModelForQuestionAnswering`, I used the `HF Trainer` initially, and later realized they have a QA version of the trainer, which has EM/F1 measures implemented for validation. I used EM/F1 for early stopping rather than observing the validation loss. Last but not least, HF `datasets` was a huge enabler for me, its caching works like magic.\n\n# Things that Didn't Work\n\n## Label Error Corrections\nQuite early in the competition, I tried to correct the noisy labels by doing out of fold predictions, looking at the model errors and replacing the label with prediction if - based on Google Translate - I thought it's better than original. However, this degraded my performance on public LB and I dropped this. I missed the opportunity to correct the answer start, I only tried it in the last 2 days after @zacchaeus published his corrected dataset. The models trained with it appeared to perform better, but they didn't help my ensemble at the end. \n\n## MT5\nI was quite bullish on MT5, especially when I tried a pretrained checkpoint from HF Hub and it scored .713 without any finetuning. Also, it looks like MT5 XXL is SOTA for this type of challenges with limited data. However, the largest model I was able to run in Kaggle kernel was MT5-Large (is it possible to run a bigger one?). I got MT5-Large to train on Google Colab (thanks to AdaFactor) as I was travelling during this time, but I couldn't surpass the .713 score of the pretrained backbone. \n\n## Jaccard Loss\nI implemented my own version of jaccard loss (Lovasz-Hinge) but it didn't improve my performance, even when combined with CrossEntropy. It's very likely that I made some implementation error, if you'd like to take a look and give me feedback I'd be happy to share that implementation. \n\n## XLNet Head\nI also tried the XLNet head where end-logits are predicted by taking into account start-logits, using beam-search on inference. It didn't work too well for me, could be another implementation error. \n\n## Tokenization\nI didn't venture to tamper with the tokenizers, but I tried to split some punctuation marks before running the tokenizer - this didn't help my score either. \n\n# Other\n\n## Music\nI listened to live concert recordings of 2 Polish rock bands during my coding sessions. Here are some samples if you want to check them out: \n* https://youtu.be/8XszVw_-z60\n* https://youtu.be/0B5gk0QjHUc\n\n## GPU\nI have a rig with 1 RTX 3090 and a Colab Pro account.\n\n## Break\nI started the competition in the first hours and prepared an EDA, baseline and QA tutorial. Then I took a break and then really focused in the last month of the competition. I used all subs I had available during that time except for 2 that I missed when DST change messed up with my submission plans. \n\n# Links\n[Inference Kernel](https://www.kaggle.com/thedrcat/chaii-ensemble-10postuning)\n[Github Repo](https://github.com/kldarek/chaii)\n", "votes": 173 }, { "id": 1798582, "postDate": "2022-05-23T06:56:06.597Z", "content": "

😮😮😮 Your code is really motivating because it is not hard to understand.

", "rawMarkdown": "😮😮😮 Your code is really motivating because it is not hard to understand.", "votes": 2 }, { "id": 1594756, "postDate": "2021-11-25T05:33:02.837Z", "content": "

congratulations

", "rawMarkdown": "congratulations", "votes": 1 }, { "id": 1594352, "postDate": "2021-11-24T18:08:48.753Z", "content": "

congratulations

", "rawMarkdown": "congratulations\n", "votes": 1 }, { "id": 1593500, "postDate": "2021-11-24T02:22:26.857Z", "content": "

Congratulations

", "rawMarkdown": "Congratulations", "votes": 1 }, { "id": 1593249, "postDate": "2021-11-23T18:12:53.380Z", "content": "

Great work, congratulations!

", "rawMarkdown": "Great work, congratulations! ", "votes": 1 }, { "id": 1592910, "postDate": "2021-11-23T13:11:48.430Z", "content": "

Congratulations

", "rawMarkdown": "Congratulations", "votes": 1 }, { "id": 1592863, "postDate": "2021-11-23T12:27:32.317Z", "content": "

Great work! Congratulations!!

", "rawMarkdown": "Great work! Congratulations!!", "votes": 1 }, { "id": 1592016, "postDate": "2021-11-22T19:49:50.443Z", "content": "

Congrats!!!

", "rawMarkdown": "Congrats!!!", "votes": 1 }, { "id": 1591413, "postDate": "2021-11-22T10:03:07.090Z", "content": "

Great! Keep on the Good work.

", "rawMarkdown": "Great! Keep on the Good work.", "votes": 1 }, { "id": 1591325, "postDate": "2021-11-22T07:29:43.573Z", "content": "

Much informative

", "rawMarkdown": "Much informative", "votes": 1 }, { "id": 1591138, "postDate": "2021-11-22T02:57:50.150Z", "content": "

Congratulation!

", "rawMarkdown": "Congratulation!", "votes": 1 }, { "id": 1591137, "postDate": "2021-11-22T02:57:24.467Z", "content": "

Congratulation!

", "rawMarkdown": "Congratulation!", "votes": 1 }, { "id": 1590990, "postDate": "2021-11-21T20:22:46.600Z", "content": "

Congratulations

", "rawMarkdown": "Congratulations", "votes": 1 }, { "id": 1590682, "postDate": "2021-11-21T15:02:16.417Z", "content": "

Congratulations on your achievements.

\n

Congratulations~

", "rawMarkdown": "Congratulations on your achievements.\n\nCongratulations~", "votes": 1 }, { "id": 1590358, "postDate": "2021-11-21T07:00:45.213Z", "content": "

Congrats @thedrcat 😃. Nice explanation 👌

", "rawMarkdown": "Congrats @thedrcat 😃. Nice explanation 👌", "votes": 1 }, { "id": 1590301, "postDate": "2021-11-21T04:38:45.693Z", "content": "

Whoop whoop!

", "rawMarkdown": "Whoop whoop!", "votes": 1 }, { "id": 1589921, "postDate": "2021-11-20T17:46:00.007Z", "content": "

Congratulation!!

", "rawMarkdown": "Congratulation!!", "votes": 1 }, { "id": 1589800, "postDate": "2021-11-20T15:25:00.653Z", "content": "

Congratulation!!

", "rawMarkdown": "Congratulation!!", "votes": 1 }, { "id": 1589757, "postDate": "2021-11-20T14:51:44.017Z", "content": "

Great work @thedrcat !✨

", "rawMarkdown": "Great work @thedrcat !✨", "votes": 1 }, { "id": 1589586, "postDate": "2021-11-20T12:03:18.073Z", "content": "

Lovely explanation!

", "rawMarkdown": "Lovely explanation!", "votes": 1 }, { "id": 1587723, "postDate": "2021-11-18T22:18:42.650Z", "content": "

nice work, good explanation 👍

", "rawMarkdown": "nice work, good explanation 👍", "votes": 1 }, { "id": 1586538, "postDate": "2021-11-18T05:51:47.113Z", "content": "

congratulations

", "rawMarkdown": "congratulations", "votes": 1 }, { "id": 1586536, "postDate": "2021-11-18T05:51:25.637Z", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 1586448, "postDate": "2021-11-18T05:00:37.410Z", "content": "

Just started learning and joining the kaggle things, hope i can understand such thing one day

", "rawMarkdown": "Just started learning and joining the kaggle things, hope i can understand such thing one day", "votes": 1, "replies": [ { "id": 1587722, "postDate": "2021-11-18T22:17:16.473Z", "content": "

When I started on Kaggle, I didn't understand many top solution descriptions to be honest… If you're getting started, I recommend the fastai course.

", "rawMarkdown": "When I started on Kaggle, I didn't understand many top solution descriptions to be honest... If you're getting started, I recommend the fastai course.", "votes": 1 } ] }, { "id": 1585921, "postDate": "2021-11-17T16:42:32.273Z", "content": "

Congratulations

", "rawMarkdown": "Congratulations", "votes": 1 }, { "id": 1585890, "postDate": "2021-11-17T16:17:22.547Z", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 1585843, "postDate": "2021-11-17T15:36:44.103Z", "content": "

congratulations

", "rawMarkdown": "congratulations\n\n", "votes": 1 }, { "id": 1585764, "postDate": "2021-11-17T14:02:58.187Z", "content": "

축하해요~!!!!!

", "rawMarkdown": "축하해요~!!!!!", "votes": 1 }, { "id": 1585685, "postDate": "2021-11-17T12:38:10.903Z", "content": "

Congratulation!

", "rawMarkdown": "Congratulation!", "votes": 1 }, { "id": 1585534, "postDate": "2021-11-17T10:10:25.240Z", "content": "

Congratulation

", "rawMarkdown": "Congratulation", "votes": 1 }, { "id": 1585508, "postDate": "2021-11-17T09:46:56.947Z", "content": "

Thanks a ton @thedrcat. This is a neat guide for NLP beginners like me.

", "rawMarkdown": "Thanks a ton @thedrcat. This is a neat guide for NLP beginners like me. ", "votes": 1 }, { "id": 1585194, "postDate": "2021-11-17T06:55:37.080Z", "content": "

Congratulations

", "rawMarkdown": "Congratulations", "votes": 1 }, { "id": 1584519, "postDate": "2021-11-16T15:13:46.767Z", "content": "

Congratulation

", "rawMarkdown": "Congratulation", "votes": 1 }, { "id": 1584454, "postDate": "2021-11-16T14:41:49.150Z", "content": "

congrats on your first place wish you the best

", "rawMarkdown": "**congrats on your first place wish you the best** ", "votes": 1 }, { "id": 1584291, "postDate": "2021-11-16T12:26:11.527Z", "content": "

Congratulations for winning this competition and thanks for sharing the solution @thedrcat !

", "rawMarkdown": "Congratulations for winning this competition and thanks for sharing the solution @thedrcat !", "votes": 1 }, { "id": 1584190, "postDate": "2021-11-16T10:37:59.653Z", "content": "

Nice work, congrats!

", "rawMarkdown": "Nice work, congrats!", "votes": 1 }, { "id": 1584187, "postDate": "2021-11-16T10:35:30.787Z", "content": "

Congratulation!

", "rawMarkdown": "Congratulation!", "votes": 1 }, { "id": 1584137, "postDate": "2021-11-16T09:53:47.127Z", "content": "

Congratulations @thedrcat. Very well deserved!

", "rawMarkdown": "Congratulations @thedrcat. Very well deserved!", "votes": 1 }, { "id": 1584046, "postDate": "2021-11-16T08:30:03.603Z", "content": "

Well deserved first place! 🎉

\n

The \"Music\" recommendation section is nice. 🎶

\n

I have read about some of these links between computer vision and NLP, do you recommend other resources (in addition to the fastai course of course) to learn more?

", "rawMarkdown": "Well deserved first place! 🎉\n\nThe \"Music\" recommendation section is nice. 🎶\n\nI have read about some of these links between computer vision and NLP, do you recommend other resources (in addition to the fastai course of course) to learn more?", "votes": 1, "replies": [ { "id": 1584775, "postDate": "2021-11-16T18:54:30.367Z", "content": "

Thanks Yassine! My main inspiration for augmentations is fastai course/book, not aware of any research/papers, but might look for it now that I have a bit more free time :)

", "rawMarkdown": "Thanks Yassine! My main inspiration for augmentations is fastai course/book, not aware of any research/papers, but might look for it now that I have a bit more free time :) ", "votes": 2 }, { "id": 1585211, "postDate": "2021-11-17T07:09:35.133Z", "content": "

Yes, the book is really great for this topic and many more. I haven't finished reading it yet but so far the best part is the \"clever\" applications of computer vision models to many problems. For example, turning malware code into an image and using this representation to classify them.

", "rawMarkdown": "Yes, the book is really great for this topic and many more. I haven't finished reading it yet but so far the best part is the \"clever\" applications of computer vision models to many problems. For example, turning malware code into an image and using this representation to classify them.", "votes": 1 } ] }, { "id": 1583996, "postDate": "2021-11-16T07:35:43.540Z", "content": "

Many congratulations @thedrcat. Very well deserved!

", "rawMarkdown": "Many congratulations @thedrcat. Very well deserved! ", "votes": 1 }, { "id": 1583978, "postDate": "2021-11-16T07:24:13.620Z", "content": "

The experience of the competition is very detailed, very good idea.

", "rawMarkdown": "The experience of the competition is very detailed, very good idea.", "votes": 1 }, { "id": 1583922, "postDate": "2021-11-16T06:31:21.270Z", "content": "

very nice work

", "rawMarkdown": "very nice work", "votes": 1 }, { "id": 1583918, "postDate": "2021-11-16T06:27:38.153Z", "content": "

Congrats @thedrcat. This solution looks like a work of art. Very clever approach adaptation from Computer Vision to NLP.

", "rawMarkdown": "Congrats @thedrcat. This solution looks like a work of art. Very clever approach adaptation from Computer Vision to NLP. ", "votes": 1, "replies": [ { "id": 1584777, "postDate": "2021-11-16T18:56:10.007Z", "content": "

Thanks @jdoesv it was great competing with you on this challenge, and thanks for bringing MURIL to the forums!!

", "rawMarkdown": "Thanks @jdoesv it was great competing with you on this challenge, and thanks for bringing MURIL to the forums!! " }, { "id": 1585147, "postDate": "2021-11-17T05:53:49.217Z", "content": "

Likewise :)

", "rawMarkdown": "Likewise :) " } ] }, { "id": 1583868, "postDate": "2021-11-16T05:43:45.993Z", "content": "

Great Work, Congratulations

", "rawMarkdown": "Great Work, Congratulations ", "votes": 1 }, { "id": 1583847, "postDate": "2021-11-16T05:23:44.427Z", "content": "

Great work @thedrcat. It was impressive to see your rise on the LB and maintain your position eventually in such a volatile LB. Congratulations on the first place!

\n

How well did your Random Cropping augmentations work ? I tried it and really hurt my CV and hence gave it up. I figured that the model was too smart to be confused by this in NLP and gave up.

", "rawMarkdown": "Great work @thedrcat. It was impressive to see your rise on the LB and maintain your position eventually in such a volatile LB. Congratulations on the first place!\n\nHow well did your Random Cropping augmentations work ? I tried it and really hurt my CV and hence gave it up. I figured that the model was too smart to be confused by this in NLP and gave up.", "votes": 1, "replies": [ { "id": 1584791, "postDate": "2021-11-16T19:00:27.143Z", "content": "

Thanks @watzisname! I still plan to do some ablation studies, can't really tell the exact impact of various methods, especially that just differentiating seeds already had a big impact on CV/LB. How did you do random cropping? Conceptually, I don't see how it might hurt model's performance…

", "rawMarkdown": "Thanks @watzisname! I still plan to do some ablation studies, can't really tell the exact impact of various methods, especially that just differentiating seeds already had a big impact on CV/LB. How did you do random cropping? Conceptually, I don't see how it might hurt model's performance..." } ] }, { "id": 1583829, "postDate": "2021-11-16T05:10:09.573Z", "content": "

Congratulations on winning this competition. Great work!!!!!

", "rawMarkdown": "Congratulations on winning this competition. Great work!!!!!\n", "votes": 1 }, { "id": 1583825, "postDate": "2021-11-16T05:08:23.633Z", "content": "

You tried a lot of stuffs. Congratulations for the great work !!! Very Well done.

", "rawMarkdown": "You tried a lot of stuffs. Congratulations for the great work !!! Very Well done. ", "votes": 1 }, { "id": 1583758, "postDate": "2021-11-16T03:34:29.217Z", "content": "

Great work!

", "rawMarkdown": "Great work! ", "votes": 1 }, { "id": 1583725, "postDate": "2021-11-16T02:59:02.660Z", "content": "

Thanks for such a detailed solution and congrats on your win and I think you will be a grandmaster as well so congrats on that as well😊.

", "rawMarkdown": "Thanks for such a detailed solution and congrats on your win and I think you will be a grandmaster as well so congrats on that as well😊.", "votes": 1 }, { "id": 1583705, "postDate": "2021-11-16T02:39:35.827Z", "content": "

Congratulations! You are the deserving winner. Your solution is very helpful. Thank you.

", "rawMarkdown": "Congratulations! You are the deserving winner. Your solution is very helpful. Thank you.", "votes": 1 }, { "id": 1583692, "postDate": "2021-11-16T02:13:28.703Z", "content": "

Congratulations for winning this competition

", "rawMarkdown": "Congratulations for winning this competition", "votes": 1 }, { "id": 1583664, "postDate": "2021-11-16T01:42:51.637Z", "content": "

Congrats!

", "rawMarkdown": "Congrats! ", "votes": 1 }, { "id": 1583661, "postDate": "2021-11-16T01:41:15.327Z", "content": "

Beautiful! =)

", "rawMarkdown": "Beautiful! =) ", "votes": 1 }, { "id": 1583642, "postDate": "2021-11-16T01:22:42.437Z", "content": "

Remarkable solution because of your very innovative ideas especially in augmentation. Very interesting and excellent work!

", "rawMarkdown": "Remarkable solution because of your very innovative ideas especially in augmentation. Very interesting and excellent work!", "votes": 1 }, { "id": 1583628, "postDate": "2021-11-16T01:14:36.453Z", "content": "

Congratulations…

", "rawMarkdown": "Congratulations...", "votes": 1 }, { "id": 1583609, "postDate": "2021-11-16T01:00:42.757Z", "content": "

Wow you definitely put everything into this competition. great work!

", "rawMarkdown": "Wow you definitely put everything into this competition. great work! ", "votes": 1 }, { "id": 1583603, "postDate": "2021-11-16T00:55:11.957Z", "content": "

Congratulations for winning this competition and thanks for solution…

", "rawMarkdown": "Congratulations for winning this competition and thanks for solution...", "votes": 1 }, { "id": 1583598, "postDate": "2021-11-16T00:47:30.103Z", "content": "

Congrats and well deserved👍

", "rawMarkdown": "Congrats and well deserved👍", "votes": 1 }, { "id": 1587126, "postDate": "2021-11-18T13:53:18.547Z", "content": "

日本語訳

\n

CV Setup
\nこの要約を書いているとき、私は自分の公開LBランクしか知らないので、この段落は苦いものになるかもしれません。 私はこの大会でパブリックLBを完全に信頼することに決め、ローカルCVさえ追跡しませんでした。 trainのデータセットは小さく、ノイズが多いことが確認されましたが、公共のLBは大きく、3方向の注釈が含まれていたため、品質が高くなりました。 ローカルCVとプライベートLBの間よりも、パブリックLBとプライベートLBの間の相関が高いと予想しました。

\n

Architecture
\n私のモデルはすべて標準のQAモデル（AutoModelForQuestionAnswering）であり、後処理は、パブリックEDAでも使用したSylvainGuggerによるHFサンプルノートブックから直接再適用されます。 XLM-Roberta Large、MURIL Large、Rembertを使用し、バックボーンごとにいくつかのモデルからの予測をブレンドしました。 次の段階では、3つのバックボーン全体で単語レベルの多数決を使用して、最終的な予測に到達する単語を決定しました。

\n

多様性のために、私は単一のモデルをブレンドせずに、多数決のみを行う別のアンサンブルを提出しました。 パブリックLBではスコアが悪くなりましたが、プライベートでも同じです。選択したサブは両方とも.787プライベートです。 私の最高のプライベートサブは.796です。

\n

Data Recipes
\n私は他のほとんどの参加者と同じように、英語のSQUADでDeepsetによって事前トレーニングされ、chaiiデータで微調整されたXLM-RobertaLargeでコンテストを開始しました。 その後、私は自分のバックボーンモデルを事前トレーニングし始めました（つまり、MLMではなくQA事前トレーニングを意味します）。 ホストがchaiiデータが同様の方法で収集されたと述べたので、私はTyDiデータセットに焦点を合わせました。 さまざまな事前トレーニングおよび微調整プロトコルを使用していくつかの実験を行いましたが、重要な洞察は、事前トレーニングされたバックボーンのパフォーマンスが微調整されたバージョンを超えることが多いということでした。 この観察に基づいて、私は2段階のアプローチをやめ、モデルを1段階でトレーニングすることにしました。 ただし、モデルにさまざまなデータセットを提供する順序を制御したかったのです。 料理のように感じるので、このデータレシピと呼んでいます。

\n

これは、最後の実験で使用したデータレシピの例です。

\n

1.TyDiデータセットの英語、ベンガル語、テルグ語のサブセットと、chaiitrainデータセットの2/3を取得します。 シーケンス長256でトークン化し、ネガティブな例の90％を削除します（ネガティブサンプリング）。 MLQAとXQUADのトークン化された英語のSQUADとヒンディー語の部分を追加します。 すべて一緒にシャッフルして脇に置きます。
\n2.手順1を繰り返しますが、英語のSQUADをNaturalQuestionsのサブセットに置き換えます。 トークン化には、シーケンスの長さ384を使用します（プログレッシブサイズ変更、以下を参照）。
\n3.手順1と2で収集したデータを連結します。
\n4.Train.
\n詳細：

\n

データの組み合わせ/シャッフルロジックをデータセットに移動したため、トレーニングにはシーケンシャルサンプラーと1エポックを使用しました。 HuggingFaceデータセットライブラリはこれを非常に簡単にしました。
\nTyDi、chaii、NQのデータセットには、コンテキストが長いため、通常は0.1のネガティブサンプリングを使用しました。
\n最初はタミル語SQUADのような自動翻訳されたデータセットを試しましたが、パフォーマンスに役立たなかったため、最終的には元の言語のデータセットと手動で翻訳されたデータセット（Hindi MLQA / XQUAD）のみを使用しました。
\nTyDiのさまざまな言語を試してみたところ、英語/ベンガル語/テルグ語が最適であることがわかりました。
\nさらにデータを追加したかったので、短い回答を含むGoogle Natural Questionsを使用して、NQデータとTyDi英語データセットを区別するための単純なRobertaベースの分類子をトレーニングしました。英語のTyDiに最も類似したNQのサブセットを選択しました。
\n私はCVを追跡していなかったので、最初はすべてのchaiiデータを使用しました。その後、多様性と早期打ち切りのために、3つのセットアップに移行しました。中間チェックポイントのCVが最終チェックポイントよりもはるかに優れている場合、通常はパブリックLBでテストします。

\n

画像のような拡張
\nこの部分はfastaiに大きく影響を受けています。ディープラーニングコースの1つで、@ jhowardは、事前トレーニング方法をコンピュータービジョンからNLP（ULMFiT）に移行する作業について話し、このアプローチに従うことで他にも多くの機会があると述べました。私は増強技術に焦点を合わせました。

\n

Random Cropping
\nこれはCVの標準的な拡張ですが、ここでは完全に適合しますが、NLPに適用されることはありません。非常に長いテキストを固定トークン長のフラグメントにチャンクする必要があります。 一度実行して各エポックで同じ例を示すのではなく、動的でランダムなトリミングにしてみませんか？ 最初に、ポジティブな例では回答テキストの周りにパディングを付けてランダムにトリミングし、ネガティブな例ではランダムにトリミングする関数を実装しました。 上記のデータレシピに移行すると、レシピにデータセットを追加するたびに異なるストライドとネガティブサンプリングを使用することで、同じことがより簡単に実現できることに気付きました。

\n

Progressive Resizing
\nこれは、ジェレミーがfastaiの講義で教えるもう一つのトリックであり、画像に一般的に適用されます。 プログレッシブサイズ変更の場合、トレーニング中に異なるシーケンス長を使用します。 例えば。 256から始めて、次に384に移動し、次に448に移動します。

\n

Cutout
\nコンピュータビジョンでは、増強として画像のパッチを黒く塗りつぶすことがあります。 NLPでは、いくつかのトークンを[MASK]に置き換えることで同じことができます。 @cdeotteがTweetSentimentコンテストでこのテクニックを使用しているのを見ました。 モデルでは0〜10％のカットアウトを使用しました。

\n

Fine-tuning
\n微調整のためにさまざまなハイパーパラメータをあまり実験しませんでした。通常、各バックボーンペーパーに記載されているデフォルトから始めて、それを使い続けることがよくありました。 私はほとんどの実験をXLM-Robertaバックボーンで行いました。 MURILはパブリックLBで最高のパフォーマンスを示しました（トークン化が原因だと思います）が、トレーニングの実行が失敗することもありました（通常は再起動が役立ちました）。

\n

各バックボーンのデフォルトのハイパーパラメータは次のとおりです(batch size = 8):

\n

XLM-Roberta: gradient accumulation: 4, learning rate: 1e-5, wd: 0.01
\nMURIL: gradient accumulation: 4, learning rate: 3e-5, wd: 0.01
\nRembert: gradient accumulation: 16, learning rate: 1e-5, wd: 0.01

\n

Post-processing
\n標準の後処理（公式のHuggingFace QAノートブックのデフォルト関数は変更していません）以外に、XLM-RobertaおよびRembertのトークン化の問題に対処することで予測を改善する機会がありました。 これらのモデルは、いくつかの句読点をサブワードとマージする傾向がありますが、アノテーターはそれらを分割することがよくあります。 最初は@abhishekの句読点を削除する方法を使用しましたが、最終日には@nbroadで共有されている方法と同様の変更をいくつか行いました。 BC / ADの後にドットを追加し、ストリップされた予測に含まれている場合は括弧を追加します。これを調整してもスコアに影響は見られませんでしたが、この後処理に基づいて位置が3から2にシフトしました。 ただし、これはプライベートLBではうまく機能しませんでした（-0.004）。 MURILはトークナイザーでうまく処理されたため、このような後処理は行いませんでした。

\n

Frameworks
\n私はこのチャレンジですべてHuggingFaceに行きましたが、別の方法でそれを行うことを想像することはできません。 AutoModelForQuestionAnswering以外に、最初はHFトレーナーを使用しましたが、後で検証用にEM / F1メジャーが実装されたトレーナーのQAバージョンがあることに気付きました。 検証の損失を観察するのではなく、早期停止にEM / F1を使用しました。 最後になりましたが、HFデータセットは私にとって大きなイネーブラーであり、そのキャッシュは魔法のように機能します。

\n

Things that Didn't Work
\nLabel Error Corrections
\nコンペティションのかなり早い段階で、フォールド外予測を行い、モデルエラーを調べ、ラベルを予測に置き換えることで、ノイズの多いラベルを修正しようとしました。Google翻訳に基づいて、元のラベルよりも優れていると思いました。 ただし、これによりパブリックLBでのパフォーマンスが低下し、これを削除しました。 回答の開始を修正する機会を逃しました。@ zacchaeusが修正したデータセットを公開してから過去2日間だけ試しました。 それで訓練されたモデルはより良く機能するように見えました、しかしそれらは最後に私のアンサンブルを助けませんでした。

\n

MT5
\n特にHFハブから事前にトレーニングされたチェックポイントを試したとき、MT5はかなり強気でしたが、微調整なしで.713を記録しました。 また、MT5 XXLは、限られたデータでこのタイプの課題に対応するSOTAのようです。 ただし、Kaggleカーネルで実行できた最大のモデルはMT5-Largeでした（より大きなモデルを実行することは可能ですか？）。 この間旅行していたときに、MT5-LargeをGoogle Colabでトレーニングしました（AdaFactorのおかげで）が、事前にトレーニングされたバックボーンの.713スコアを超えることはできませんでした。

\n

Jaccard Loss
\n独自のバージョンのジャッカードロス（Lovasz-Hinge）を実装しましたが、CrossEntropyと組み合わせてもパフォーマンスは向上しませんでした。 実装エラーが発生した可能性が非常に高いです。ご覧になり、フィードバックをお寄せください。その実装を共有させていただきます。

\n

XLNet Head
\nまた、推論のビームサーチを使用して、開始ロジットを考慮して終了ロジットを予測するXLNetヘッドを試しました。 それは私にとってあまりうまく機能しませんでした、別の実装エラーである可能性があります。

\n

Tokenization
\nトークナイザーを改ざんしようとはしませんでしたが、トークナイザーを実行する前に句読点を分割しようとしました。これもスコアに役立ちませんでした。

\n

Other
\nMusic
\nコーディングセッション中に、2つのポーランドのロックバンドのライブコンサートの録音を聴きました。 あなたがそれらをチェックしたいならば、ここにいくつかのサンプルがあります：

\n

https://youtu.be/8XszVw_-z60
\nhttps://youtu.be/0B5gk0QjHUc

\n

GPU
\nI have a rig with 1 RTX 3090 and a Colab Pro account.

\n

Break
\n私は最初の数時間でコンテストを開始し、EDA、ベースライン、QAチュートリアルを準備しました。 それから私は休憩を取り、そしてコンペティションの最後の月に本当に集中しました。 DSTの変更が提出計画を台無しにしたときに見逃した2つを除いて、その間に利用可能だったすべてのサブを使用しました。

", "rawMarkdown": "日本語訳\n\nCV Setup\nこの要約を書いているとき、私は自分の公開LBランクしか知らないので、この段落は苦いものになるかもしれません。 私はこの大会でパブリックLBを完全に信頼することに決め、ローカルCVさえ追跡しませんでした。 trainのデータセットは小さく、ノイズが多いことが確認されましたが、公共のLBは大きく、3方向の注釈が含まれていたため、品質が高くなりました。 ローカルCVとプライベートLBの間よりも、パブリックLBとプライベートLBの間の相関が高いと予想しました。\n\nArchitecture\n私のモデルはすべて標準のQAモデル（AutoModelForQuestionAnswering）であり、後処理は、パブリックEDAでも使用したSylvainGuggerによるHFサンプルノートブックから直接再適用されます。 XLM-Roberta Large、MURIL Large、Rembertを使用し、バックボーンごとにいくつかのモデルからの予測をブレンドしました。 次の段階では、3つのバックボーン全体で単語レベルの多数決を使用して、最終的な予測に到達する単語を決定しました。\n\n多様性のために、私は単一のモデルをブレンドせずに、多数決のみを行う別のアンサンブルを提出しました。 パブリックLBではスコアが悪くなりましたが、プライベートでも同じです。選択したサブは両方とも.787プライベートです。 私の最高のプライベートサブは.796です。\n\nData Recipes\n私は他のほとんどの参加者と同じように、英語のSQUADでDeepsetによって事前トレーニングされ、chaiiデータで微調整されたXLM-RobertaLargeでコンテストを開始しました。 その後、私は自分のバックボーンモデルを事前トレーニングし始めました（つまり、MLMではなくQA事前トレーニングを意味します）。 ホストがchaiiデータが同様の方法で収集されたと述べたので、私はTyDiデータセットに焦点を合わせました。 さまざまな事前トレーニングおよび微調整プロトコルを使用していくつかの実験を行いましたが、重要な洞察は、事前トレーニングされたバックボーンのパフォーマンスが微調整されたバージョンを超えることが多いということでした。 この観察に基づいて、私は2段階のアプローチをやめ、モデルを1段階でトレーニングすることにしました。 ただし、モデルにさまざまなデータセットを提供する順序を制御したかったのです。 料理のように感じるので、このデータレシピと呼んでいます。\n\nこれは、最後の実験で使用したデータレシピの例です。\n\n1.TyDiデータセットの英語、ベンガル語、テルグ語のサブセットと、chaiitrainデータセットの2/3を取得します。 シーケンス長256でトークン化し、ネガティブな例の90％を削除します（ネガティブサンプリング）。 MLQAとXQUADのトークン化された英語のSQUADとヒンディー語の部分を追加します。 すべて一緒にシャッフルして脇に置きます。\n2.手順1を繰り返しますが、英語のSQUADをNaturalQuestionsのサブセットに置き換えます。 トークン化には、シーケンスの長さ384を使用します（プログレッシブサイズ変更、以下を参照）。\n3.手順1と2で収集したデータを連結します。\n4.Train.\n詳細：\n\nデータの組み合わせ/シャッフルロジックをデータセットに移動したため、トレーニングにはシーケンシャルサンプラーと1エポックを使用しました。 HuggingFaceデータセットライブラリはこれを非常に簡単にしました。\nTyDi、chaii、NQのデータセットには、コンテキストが長いため、通常は0.1のネガティブサンプリングを使用しました。\n最初はタミル語SQUADのような自動翻訳されたデータセットを試しましたが、パフォーマンスに役立たなかったため、最終的には元の言語のデータセットと手動で翻訳されたデータセット（Hindi MLQA / XQUAD）のみを使用しました。\nTyDiのさまざまな言語を試してみたところ、英語/ベンガル語/テルグ語が最適であることがわかりました。\nさらにデータを追加したかったので、短い回答を含むGoogle Natural Questionsを使用して、NQデータとTyDi英語データセットを区別するための単純なRobertaベースの分類子をトレーニングしました。英語のTyDiに最も類似したNQのサブセットを選択しました。\n私はCVを追跡していなかったので、最初はすべてのchaiiデータを使用しました。その後、多様性と早期打ち切りのために、3つのセットアップに移行しました。中間チェックポイントのCVが最終チェックポイントよりもはるかに優れている場合、通常はパブリックLBでテストします。\n\n画像のような拡張\nこの部分はfastaiに大きく影響を受けています。ディープラーニングコースの1つで、@ jhowardは、事前トレーニング方法をコンピュータービジョンからNLP（ULMFiT）に移行する作業について話し、このアプローチに従うことで他にも多くの機会があると述べました。私は増強技術に焦点を合わせました。\n\nRandom Cropping\nこれはCVの標準的な拡張ですが、ここでは完全に適合しますが、NLPに適用されることはありません。非常に長いテキストを固定トークン長のフラグメントにチャンクする必要があります。 一度実行して各エポックで同じ例を示すのではなく、動的でランダムなトリミングにしてみませんか？ 最初に、ポジティブな例では回答テキストの周りにパディングを付けてランダムにトリミングし、ネガティブな例ではランダムにトリミングする関数を実装しました。 上記のデータレシピに移行すると、レシピにデータセットを追加するたびに異なるストライドとネガティブサンプリングを使用することで、同じことがより簡単に実現できることに気付きました。\n\nProgressive Resizing\nこれは、ジェレミーがfastaiの講義で教えるもう一つのトリックであり、画像に一般的に適用されます。 プログレッシブサイズ変更の場合、トレーニング中に異なるシーケンス長を使用します。 例えば。 256から始めて、次に384に移動し、次に448に移動します。\n\nCutout\nコンピュータビジョンでは、増強として画像のパッチを黒く塗りつぶすことがあります。 NLPでは、いくつかのトークンを[MASK]に置き換えることで同じことができます。 @cdeotteがTweetSentimentコンテストでこのテクニックを使用しているのを見ました。 モデルでは0〜10％のカットアウトを使用しました。\n\nFine-tuning\n微調整のためにさまざまなハイパーパラメータをあまり実験しませんでした。通常、各バックボーンペーパーに記載されているデフォルトから始めて、それを使い続けることがよくありました。 私はほとんどの実験をXLM-Robertaバックボーンで行いました。 MURILはパブリックLBで最高のパフォーマンスを示しました（トークン化が原因だと思います）が、トレーニングの実行が失敗することもありました（通常は再起動が役立ちました）。\n\n各バックボーンのデフォルトのハイパーパラメータは次のとおりです(batch size = 8):\n\nXLM-Roberta: gradient accumulation: 4, learning rate: 1e-5, wd: 0.01\nMURIL: gradient accumulation: 4, learning rate: 3e-5, wd: 0.01\nRembert: gradient accumulation: 16, learning rate: 1e-5, wd: 0.01\n\nPost-processing\n標準の後処理（公式のHuggingFace QAノートブックのデフォルト関数は変更していません）以外に、XLM-RobertaおよびRembertのトークン化の問題に対処することで予測を改善する機会がありました。 これらのモデルは、いくつかの句読点をサブワードとマージする傾向がありますが、アノテーターはそれらを分割することがよくあります。 最初は@abhishekの句読点を削除する方法を使用しましたが、最終日には@nbroadで共有されている方法と同様の変更をいくつか行いました。 BC / ADの後にドットを追加し、ストリップされた予測に含まれている場合は括弧を追加します。これを調整してもスコアに影響は見られませんでしたが、この後処理に基づいて位置が3から2にシフトしました。 ただし、これはプライベートLBではうまく機能しませんでした（-0.004）。 MURILはトークナイザーでうまく処理されたため、このような後処理は行いませんでした。\n\nFrameworks\n私はこのチャレンジですべてHuggingFaceに行きましたが、別の方法でそれを行うことを想像することはできません。 AutoModelForQuestionAnswering以外に、最初はHFトレーナーを使用しましたが、後で検証用にEM / F1メジャーが実装されたトレーナーのQAバージョンがあることに気付きました。 検証の損失を観察するのではなく、早期停止にEM / F1を使用しました。 最後になりましたが、HFデータセットは私にとって大きなイネーブラーであり、そのキャッシュは魔法のように機能します。\n\nThings that Didn't Work\nLabel Error Corrections\nコンペティションのかなり早い段階で、フォールド外予測を行い、モデルエラーを調べ、ラベルを予測に置き換えることで、ノイズの多いラベルを修正しようとしました。Google翻訳に基づいて、元のラベルよりも優れていると思いました。 ただし、これによりパブリックLBでのパフォーマンスが低下し、これを削除しました。 回答の開始を修正する機会を逃しました。@ zacchaeusが修正したデータセットを公開してから過去2日間だけ試しました。 それで訓練されたモデルはより良く機能するように見えました、しかしそれらは最後に私のアンサンブルを助けませんでした。\n\nMT5\n特にHFハブから事前にトレーニングされたチェックポイントを試したとき、MT5はかなり強気でしたが、微調整なしで.713を記録しました。 また、MT5 XXLは、限られたデータでこのタイプの課題に対応するSOTAのようです。 ただし、Kaggleカーネルで実行できた最大のモデルはMT5-Largeでした（より大きなモデルを実行することは可能ですか？）。 この間旅行していたときに、MT5-LargeをGoogle Colabでトレーニングしました（AdaFactorのおかげで）が、事前にトレーニングされたバックボーンの.713スコアを超えることはできませんでした。\n\nJaccard Loss\n独自のバージョンのジャッカードロス（Lovasz-Hinge）を実装しましたが、CrossEntropyと組み合わせてもパフォーマンスは向上しませんでした。 実装エラーが発生した可能性が非常に高いです。ご覧になり、フィードバックをお寄せください。その実装を共有させていただきます。\n\nXLNet Head\nまた、推論のビームサーチを使用して、開始ロジットを考慮して終了ロジットを予測するXLNetヘッドを試しました。 それは私にとってあまりうまく機能しませんでした、別の実装エラーである可能性があります。\n\nTokenization\nトークナイザーを改ざんしようとはしませんでしたが、トークナイザーを実行する前に句読点を分割しようとしました。これもスコアに役立ちませんでした。\n\nOther\nMusic\nコーディングセッション中に、2つのポーランドのロックバンドのライブコンサートの録音を聴きました。 あなたがそれらをチェックしたいならば、ここにいくつかのサンプルがあります：\n\nhttps://youtu.be/8XszVw_-z60\nhttps://youtu.be/0B5gk0QjHUc\n\nGPU\nI have a rig with 1 RTX 3090 and a Colab Pro account.\n\nBreak\n私は最初の数時間でコンテストを開始し、EDA、ベースライン、QAチュートリアルを準備しました。 それから私は休憩を取り、そしてコンペティションの最後の月に本当に集中しました。 DSTの変更が提出計画を台無しにしたときに見逃した2つを除いて、その間に利用可能だったすべてのサブを使用しました。", "votes": 1 }, { "id": 1612010, "postDate": "2021-12-08T13:13:11.173Z", "content": "

Congratulations!!

", "rawMarkdown": "Congratulations!!", "votes": 2 }, { "id": 1586696, "postDate": "2021-11-18T08:18:37.790Z", "content": "

Congratulations and thanks for sharing!
\nI'm just wondering: Would you usually do this amount of effort for a data science project?!
\nSeems too much but works too great!

", "rawMarkdown": "Congratulations and thanks for sharing!\nI'm just wondering: Would you usually do this amount of effort for a data science project?! \nSeems too much but works too great!", "votes": 2, "replies": [ { "id": 1587721, "postDate": "2021-11-18T22:14:26.700Z", "content": "

Thank you! On Kaggle, it's my 2nd competition where I put so much work, after HPA. Outside Kaggle, it all depends on how big the return on invested effort is…

", "rawMarkdown": "Thank you! On Kaggle, it's my 2nd competition where I put so much work, after HPA. Outside Kaggle, it all depends on how big the return on invested effort is...", "votes": 1 } ] }, { "id": 1586396, "postDate": "2021-11-18T03:30:45.683Z", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "votes": 2 }, { "id": 1586003, "postDate": "2021-11-17T17:50:43.097Z", "content": "

First and foremost congratulations🎉😃
\nI am still trying to wrap around my heard all the techniques you told, and I hope one day I will be able to learn them and such advanced techniques and use them in competitions.
\nCan you please tell me how to know which technique would go best, consider fixed GPU time, so we can brute force?

\n
\n

Music
\n I listened to live concert recordings of 2 Polish rock bands during my coding sessions. Here are some samples if you want to check them out:
\n https://youtu.be/8XszVw_-z60
\n https://youtu.be/0B5gk0QjHUc

\n
\n

Thank you for sharing about them, will surely try to use them.

\n
\n
\n

GPU
\n I have a rig with 1 RTX 3090 and a Colab Pro account.

\n
\n

Sorry, I am a new Ig, can you please tell me what does rig means in this context?

", "rawMarkdown": "First and foremost congratulations🎉😃 \nI am still trying to wrap around my heard all the techniques you told, and I hope one day I will be able to learn them and such advanced techniques and use them in competitions.\nCan you please tell me how to know which technique would go best, consider fixed GPU time, so we can brute force?\n\n> Music\nI listened to live concert recordings of 2 Polish rock bands during my coding sessions. Here are some samples if you want to check them out:\nhttps://youtu.be/8XszVw_-z60\nhttps://youtu.be/0B5gk0QjHUc\n\nThank you for sharing about them, will surely try to use them. \n\n---\n\n> GPU\nI have a rig with 1 RTX 3090 and a Colab Pro account.\n\nSorry, I am a new Ig, can you please tell me what does rig means in this context?", "votes": 2, "replies": [ { "id": 1586201, "postDate": "2021-11-17T22:14:48.220Z", "content": "

Thanks Krish! I didn't really know which technique would work best, except from some intuition by failing in many competitions previously and later on reading about the winning solutions… The more competitions/experiments you run, the better you'll hone your intuition :) Not sure if my music recos will help 😂let me know if you try! Rig = GPU workstation.

", "rawMarkdown": "Thanks Krish! I didn't really know which technique would work best, except from some intuition by failing in many competitions previously and later on reading about the winning solutions... The more competitions/experiments you run, the better you'll hone your intuition :) Not sure if my music recos will help 😂let me know if you try! Rig = GPU workstation. ", "votes": 1 } ] }, { "id": 1584395, "postDate": "2021-11-16T13:58:46.167Z", "content": "

Thanks for your very detailed solution and thanks for your sharings about Polish rock bands 👍

", "rawMarkdown": "Thanks for your very detailed solution and thanks for your sharings about Polish rock bands 👍", "votes": 2 }, { "id": 1584263, "postDate": "2021-11-16T11:55:35.623Z", "content": "

Congratulations. Nice Work

", "rawMarkdown": "Congratulations. Nice Work", "votes": 2 }, { "id": 1583721, "postDate": "2021-11-16T02:57:13.387Z", "content": "

Nice work, congrats @thedrcat on 1st place

", "rawMarkdown": "Nice work, congrats @thedrcat on 1st place", "votes": 2 }, { "id": 1583686, "postDate": "2021-11-16T02:05:01.610Z", "content": "

Congrats Darek! So now we know that Kult and Pidżama Porno are responsible for this 1st place. But then I am also fan of Kult and i didn't win any kaggle comp. Lol. Just joking. Congrats again, well deserved!

", "rawMarkdown": "Congrats Darek! So now we know that Kult and Pidżama Porno are responsible for this 1st place. But then I am also fan of Kult and i didn't win any kaggle comp. Lol. Just joking. Congrats again, well deserved!", "votes": 2, "replies": [ { "id": 1584770, "postDate": "2021-11-16T18:51:56.570Z", "content": "

Thanks Michał!! Have you tried listening to Kult while doing a Kaggle comp? If not, maybe that's the problem 😂😂😂

", "rawMarkdown": "Thanks Michał!! Have you tried listening to Kult while doing a Kaggle comp? If not, maybe that's the problem 😂😂😂" } ] }, { "id": 1583595, "postDate": "2021-11-16T00:44:33.020Z", "content": "

Good hard work ethic and attention to detail paid off. Congratulations on your well deserved #win.

", "rawMarkdown": "Good hard work ethic and attention to detail paid off. Congratulations on your well deserved #win.", "votes": 2 }, { "id": 1587109, "postDate": "2021-11-18T13:35:47.250Z", "content": "

1st comment

", "rawMarkdown": "1st comment" }, { "id": 1591037, "postDate": "2021-11-21T21:59:10.040Z", "content": "

Great Work!!

", "rawMarkdown": "Great Work!!", "votes": 1, "isDeleted": true }, { "id": 1584727, "postDate": "2021-11-16T18:07:02.897Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1594339, "postDate": "2021-11-24T18:00:21.743Z", "content": "

thanks for sharing

", "rawMarkdown": "thanks for sharing\n", "votes": 1 }, { "id": 1591349, "postDate": "2021-11-22T08:19:11.223Z", "content": "

Thank you for your explanation!

", "rawMarkdown": "Thank you for your explanation!", "votes": 1 }, { "id": 1587843, "postDate": "2021-11-19T01:28:54.360Z", "content": "

Thank you for sharing!

", "rawMarkdown": "Thank you for sharing!", "votes": 1 }, { "id": 1585643, "postDate": "2021-11-17T12:05:19.607Z", "content": "

Thanks! Nice guide!

", "rawMarkdown": "Thanks! Nice guide!", "votes": 1 }, { "id": 1584748, "postDate": "2021-11-16T18:16:56.377Z", "content": "

Congrats and thanks for sharing 🎉

", "rawMarkdown": "Congrats and thanks for sharing 🎉", "votes": 1 }, { "id": 1583894, "postDate": "2021-11-16T06:06:35.277Z", "content": "

thanks for sharing.

", "rawMarkdown": "thanks for sharing.", "votes": 1 }, { "id": 1583853, "postDate": "2021-11-16T05:28:25.230Z", "content": "

Great! Thanks for sharing

", "rawMarkdown": "Great! Thanks for sharing", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:23:36Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7251121, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/hold-my-chai-1st-place-solution" }