First of all, my team-mate @mananjhaveri and I would like to thank the competition organizers for organizing this competition.
\nWe would also like to thank @rhtsingh and @andretugan for their amazing notebooks. Special thanks to @abhishek for his awesome book called Approaching Almost Any Machine Learning Problem. The chapter on NLP helped us a lot.
\nOur best single model was RoBERTa-Large with attention head. We also added layer wise learning rate decay and layer reinitialization used from this amazing notebook which boosted our score.
\nConfiguration for our best single model -
\nNUM_EPOCHS = 5\nNUM_FOLDS = 5\nBATCH_SIZE = 12\nMAX_LEN = 256\nLEARNING_RATE = 3e-5\nWEIGHT_DECAY = 1e-2\nNUM_WARMUP_STEPS = 50\nEVAL_SCHEDULE = [(0.51, 50), (0.49, 25)]\nWith this, our best single model reached a score of 0.467 in Public LB and 0.468 in Private LB.
\nWe used 3 models for this -
\nModel 1 = LB 0.467 - https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch
Model 2 = LB 0.466 - https://www.kaggle.com/rajat95gupta/clrp-mean-pooling-inference
Model 3 = Our best single model
| Model | \nRatio | \n
|---|---|
| Model 1 | \n0.35 | \n
| Model 2 | \n0.35 | \n
| Model 3 | \n0.3 | \n
This gave us a score of 0.457 on public as well as private LB.
\nA big thank you to the Kaggle community for being so supportive. This was our first competition where we landed in the medal zone and it was a very exciting journey. We look forward to more competitions like this and hopefully contribute to this community.
", "comments": [ { "id": 1423191, "author_name": "Manan Jhaveri", "author_url": "", "post_date": "2021-08-03T05:28:50.957000", "content": "Team DilBERT
\nFind our final inference notebook here: https://www.kaggle.com/mananjhaveri/rank-88-clrp-lb-0-457-solution
First of all, my team-mate @mananjhaveri and I would like to thank the competition organizers for organizing this competition.
\nWe would also like to thank @rhtsingh and @andretugan for their amazing notebooks. Special thanks to @abhishek for his awesome book called Approaching Almost Any Machine Learning Problem. The chapter on NLP helped us a lot.
\nOur best single model was RoBERTa-Large with attention head. We also added layer wise learning rate decay and layer reinitialization used from this amazing notebook which boosted our score.
\nConfiguration for our best single model -
\nNUM_EPOCHS = 5\nNUM_FOLDS = 5\nBATCH_SIZE = 12\nMAX_LEN = 256\nLEARNING_RATE = 3e-5\nWEIGHT_DECAY = 1e-2\nNUM_WARMUP_STEPS = 50\nEVAL_SCHEDULE = [(0.51, 50), (0.49, 25)]\nWith this, our best single model reached a score of 0.467 in Public LB and 0.468 in Private LB.
\nWe used 3 models for this -
\nModel 1 = LB 0.467 - https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch
Model 2 = LB 0.466 - https://www.kaggle.com/rajat95gupta/clrp-mean-pooling-inference
Model 3 = Our best single model
| Model | \nRatio | \n
|---|---|
| Model 1 | \n0.35 | \n
| Model 2 | \n0.35 | \n
| Model 3 | \n0.3 | \n
This gave us a score of 0.457 on public as well as private LB.
\nA big thank you to the Kaggle community for being so supportive. This was our first competition where we landed in the medal zone and it was a very exciting journey. We look forward to more competitions like this and hopefully contribute to this community.
", "rawMarkdown": "# Acknowledgement\n\nFirst of all, my team-mate @mananjhaveri and I would like to thank the competition organizers for organizing this competition.\n\nWe would also like to thank @rhtsingh and @andretugan for their amazing notebooks. Special thanks to @abhishek for his awesome book called Approaching Almost Any Machine Learning Problem. The chapter on NLP helped us a lot.\n\n# The breakthrough (best single-model)\n\nOur best single model was RoBERTa-Large with attention head. We also added layer wise learning rate decay and layer reinitialization used from [this amazing notebook](https://www.kaggle.com/rhtsingh/on-stability-of-few-sample-transformer-fine-tuning) which boosted our score.\n\nConfiguration for our best single model - \n\n```python\nNUM_EPOCHS = 5\nNUM_FOLDS = 5\nBATCH_SIZE = 12\nMAX_LEN = 256\nLEARNING_RATE = 3e-5\nWEIGHT_DECAY = 1e-2\nNUM_WARMUP_STEPS = 50\nEVAL_SCHEDULE = [(0.51, 50), (0.49, 25)]\n```\n\nWith this, our best single model reached a score of 0.467 in Public LB and 0.468 in Private LB. \n\n# The breakthrough (ensemble)\n\nWe used 3 models for this - \n\n- Model 1 = LB 0.467 - https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch\n\n- Model 2 = LB 0.466 - https://www.kaggle.com/rajat95gupta/clrp-mean-pooling-inference\n\n- Model 3 = Our best single model\n\n\n| Model | Ratio |\n| --- | --- |\n| Model 1 | 0.35 |\n| Model 2 | 0.35 |\n| Model 3 | 0.3 |\n\nThis gave us a score of 0.457 on public as well as private LB. \n\nA big thank you to the Kaggle community for being so supportive. This was our first competition where we landed in the medal zone and it was a very exciting journey. We look forward to more competitions like this and hopefully contribute to this community.\n", "votes": 8 }, { "id": 1423191, "postDate": "2021-08-03T05:28:50.957Z", "content": "Team DilBERT
\nFind our final inference notebook here: https://www.kaggle.com/mananjhaveri/rank-88-clrp-lb-0-457-solution