Thanks everyone for the competition! This is my first NLP kaggle, so I'm quite happy with the results. Thanks my teammates, ods.ai and kaggle community for great ideas. \nNow I would like to tell you about our solution.
\n\nAs a baseline we took this kernel, so text tokenization and model structure was similar, but we changed Head structure of our model. First thought was to make it more \"smooth\". Linear layers were 1024 * 3 -> 1024 -> ... -> 1024 -> 3 for BERT-large-cased. And we made it 1024 * 3 -> 1024 -> 512 -> 64 -> 3. And for BERT-base-cased 768 * 3 -> 768 -> 768 // 2 -> 768 // 8 -> 3. We also put Dropout(0.5) before each linear layer (and bn, relu, of course). \nIn both cases we didn't use the last BERT output layer. For base it was -2 and for large it was -4.
\n\nOne important step: we used cleared data, from here, because some samples were labeled wrongly and it could corrupted results.\nWe used 6 folds. On each model was trained with BertAdam optimizer(lr = 0.0001, weight_decay = 0.01) from pytorch-BERT repo. And ReduceLROnPlateau scheduler with patience = 5 and alpha = 2. When we had no score improvement in 15 epochs we stopped training. We also tried CosineAnnealingLR after with RMSprop, but it didn't improved the score.\nCV is great for stacking, so we decided to build a second layer model, which was LGBM. We added both BERT-large and BERT-base + some features.
\n\nFor features we used distance and url (as in this kernel) and some statistic features, like 'num words', 'num unique words', 'num chars', 'num stopwords', 'num words upper' and encoded pronoun (just by number, ex. 'She': 1, 'he': 2, etc.). And that's it!
\n\nFor parameters search i used a powerful hyperopt framework. This is very useful and effective for randomized search.
\n\nWe also tried BERT fine-tuning, like in examples from this example , but it didn't seem to work, the score was worse. So if you succeeded in BERT fine-tuning, please tell in comments how you did it.
\n\nThanks for reading! 🥈
", "comments": [ { "id": 521735, "author_name": "", "author_url": "", "post_date": "2019-04-23T10:52:00.293000", "content": "Congratulations for the medal and thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 521498, "author_name": "Akash Ravichandran", "author_url": "", "post_date": "2019-04-23T01:21:04.467000", "content": "Congrats and Thanks for sharing @avens8
", "votes": 1, "replies": [] }, { "id": 521494, "author_name": "SRK", "author_url": "", "post_date": "2019-04-23T01:17:12.337000", "content": "Thanks Pavel for the write-up. We used the last 4 layers. It looks like we should have removed the last two layers and used only -3 and -4 in BERT.
", "votes": 1, "replies": [ { "id": 521496, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-04-23T01:19:33.737000", "content": "Yes, it'd be interesting to explore which layer carries which information. We used -4, -5 and -6.
", "votes": 3, "replies": [] }, { "id": 521521, "author_name": "gui_yihan", "author_url": "", "post_date": "2019-04-23T01:54:07.157000", "content": "We used layer -6 of Bert large and our final result was 0.2334 in stage 2.
", "votes": 2, "replies": [] }, { "id": 522335, "author_name": "Artur Ilichev", "author_url": "", "post_date": "2019-04-24T09:03:54.927000", "content": "We tried using average of the last 4 layers, but it didn't improve our score. Concatenation of -3 and -4 layers also didn't work well.
", "votes": 0, "replies": [] } ] }, { "id": 521538, "author_name": "Tsukasa Moritoki", "author_url": "", "post_date": "2019-04-23T02:53:09.130000", "content": "I tried BERT-LARGE fine-tuning. Single model was unstable but average ensemble was good. There are three points for the fine-tuning to work, simple head, lower learning rate and removing top 2 layers. The last one is not necessary, only slight improvement over the full model(24 layers). See Offsets model for detail.
", "votes": 2, "replies": [ { "id": 521668, "author_name": "Pavel Tsai", "author_url": "", "post_date": "2019-04-23T08:23:45.203000", "content": "Lol, simple models without features give 0.20819. Thanks for sharing a very elegant solution!
", "votes": 0, "replies": [] } ] }, { "id": 521877, "author_name": "dhaqui the kaggler", "author_url": "", "post_date": "2019-04-23T15:11:09.623000", "content": "Thanks for sharing! Congratulations!
", "votes": 1, "replies": [] }, { "id": 521491, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-04-23T01:14:45.263000", "content": "Thanks for your overview, Pavel!
", "votes": 1, "replies": [] } ], "comment_count": 10, "messages": [ { "id": 521484, "postDate": "2019-04-23T00:59:59.453Z", "content": "Thanks everyone for the competition! This is my first NLP kaggle, so I'm quite happy with the results. Thanks my teammates, ods.ai and kaggle community for great ideas. \nNow I would like to tell you about our solution.
\n\nAs a baseline we took this kernel, so text tokenization and model structure was similar, but we changed Head structure of our model. First thought was to make it more \"smooth\". Linear layers were 1024 * 3 -> 1024 -> ... -> 1024 -> 3 for BERT-large-cased. And we made it 1024 * 3 -> 1024 -> 512 -> 64 -> 3. And for BERT-base-cased 768 * 3 -> 768 -> 768 // 2 -> 768 // 8 -> 3. We also put Dropout(0.5) before each linear layer (and bn, relu, of course). \nIn both cases we didn't use the last BERT output layer. For base it was -2 and for large it was -4.
\n\nOne important step: we used cleared data, from here, because some samples were labeled wrongly and it could corrupted results.\nWe used 6 folds. On each model was trained with BertAdam optimizer(lr = 0.0001, weight_decay = 0.01) from pytorch-BERT repo. And ReduceLROnPlateau scheduler with patience = 5 and alpha = 2. When we had no score improvement in 15 epochs we stopped training. We also tried CosineAnnealingLR after with RMSprop, but it didn't improved the score.\nCV is great for stacking, so we decided to build a second layer model, which was LGBM. We added both BERT-large and BERT-base + some features.
\n\nFor features we used distance and url (as in this kernel) and some statistic features, like 'num words', 'num unique words', 'num chars', 'num stopwords', 'num words upper' and encoded pronoun (just by number, ex. 'She': 1, 'he': 2, etc.). And that's it!
\n\nFor parameters search i used a powerful hyperopt framework. This is very useful and effective for randomized search.
\n\nWe also tried BERT fine-tuning, like in examples from this example , but it didn't seem to work, the score was worse. So if you succeeded in BERT fine-tuning, please tell in comments how you did it.
\n\nThanks for reading! 🥈
", "rawMarkdown": "Thanks everyone for the competition! This is my first NLP kaggle, so I'm quite happy with the results. Thanks my teammates, [ods.ai](ods.ai) and kaggle community for great ideas. \nNow I would like to tell you about our solution.\n# model\nAs a baseline we took this [kernel](https://www.kaggle.com/gdoteof/pytorch-bert-baseline-wd-epochs-cnn-lstm), so text tokenization and model structure was similar, but we changed **Head** structure of our model. First thought was to make it more \"smooth\". Linear layers were 1024 * 3 -> 1024 -> ... -> 1024 -> 3 for BERT-large-cased. And we made it 1024 * 3 -> 1024 -> 512 -> 64 -> 3. And for BERT-base-cased 768 * 3 -> 768 -> 768 // 2 -> 768 // 8 -> 3. We also put Dropout(0.5) before each linear layer (and bn, relu, of course). \nIn both cases we didn't use the last BERT output layer. For base it was -2 and for large it was -4.\n# training process\nOne important step: we used cleared data, from [here](https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/81331#503094), because some samples were labeled wrongly and it could corrupted results.\nWe used 6 folds. On each model was trained with BertAdam optimizer(lr = 0.0001, weight_decay = 0.01) from [pytorch-BERT repo](https://github.com/huggingface/pytorch-pretrained-BERT). And ReduceLROnPlateau scheduler with patience = 5 and alpha = 2. When we had no score improvement in 15 epochs we stopped training. We also tried CosineAnnealingLR after with RMSprop, but it didn't improved the score.\nCV is great for stacking, so we decided to build a second layer model, which was LGBM. We added both BERT-large and BERT-base + some features.\n# features\nFor features we used distance and url (as in this [kernel](https://www.kaggle.com/chanhu/bert-score-layer-lb-0-475)) and some statistic features, like 'num words', 'num unique words', 'num chars', 'num stopwords', 'num words upper' and encoded pronoun (just by number, ex. 'She': 1, 'he': 2, etc.). And that's it!\n# LGBM\nFor parameters search i used a powerful [hyperopt](https://github.com/hyperopt/hyperopt) framework. This is very useful and effective for randomized search.\n# fine-tuning BERT\nWe also tried BERT fine-tuning, like in examples from this [example ](https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/lm_finetuning), but it didn't seem to work, the score was worse. So if you succeeded in BERT fine-tuning, please tell in comments how you did it.\n\nThanks for reading! 🥈 ", "votes": 13 }, { "id": 521735, "postDate": "2019-04-23T10:52:00.293Z", "content": "Congratulations for the medal and thanks for sharing!
", "rawMarkdown": "Congratulations for the medal and thanks for sharing!", "votes": 1 }, { "id": 521498, "postDate": "2019-04-23T01:21:04.467Z", "content": "Congrats and Thanks for sharing @avens8
", "rawMarkdown": "**Congrats and Thanks for sharing @avens8**", "votes": 1 }, { "id": 521494, "postDate": "2019-04-23T01:17:12.337Z", "content": "Thanks Pavel for the write-up. We used the last 4 layers. It looks like we should have removed the last two layers and used only -3 and -4 in BERT.
", "rawMarkdown": "Thanks Pavel for the write-up. We used the last 4 layers. It looks like we should have removed the last two layers and used only -3 and -4 in BERT. ", "votes": 1, "replies": [ { "id": 521496, "postDate": "2019-04-23T01:19:33.737Z", "content": "Yes, it'd be interesting to explore which layer carries which information. We used -4, -5 and -6.
", "rawMarkdown": "Yes, it'd be interesting to explore which layer carries which information. We used -4, -5 and -6. ", "votes": 3 }, { "id": 521521, "postDate": "2019-04-23T01:54:07.157Z", "content": "We used layer -6 of Bert large and our final result was 0.2334 in stage 2.
", "rawMarkdown": "We used layer -6 of Bert large and our final result was 0.2334 in stage 2.", "votes": 2 }, { "id": 522335, "postDate": "2019-04-24T09:03:54.927Z", "content": "We tried using average of the last 4 layers, but it didn't improve our score. Concatenation of -3 and -4 layers also didn't work well.
", "rawMarkdown": "We tried using average of the last 4 layers, but it didn't improve our score. Concatenation of -3 and -4 layers also didn't work well." } ] }, { "id": 521538, "postDate": "2019-04-23T02:53:09.130Z", "content": "I tried BERT-LARGE fine-tuning. Single model was unstable but average ensemble was good. There are three points for the fine-tuning to work, simple head, lower learning rate and removing top 2 layers. The last one is not necessary, only slight improvement over the full model(24 layers). See Offsets model for detail.
", "rawMarkdown": "I tried BERT-LARGE fine-tuning. Single model was unstable but average ensemble was good. There are three points for the fine-tuning to work, simple head, lower learning rate and removing top 2 layers. The last one is not necessary, only slight improvement over the full model(24 layers). See [Offsets model](https://www.kaggle.com/tks0123456789/offsets-model) for detail. ", "votes": 2, "replies": [ { "id": 521668, "postDate": "2019-04-23T08:23:45.203Z", "content": "Lol, simple models without features give 0.20819. Thanks for sharing a very elegant solution!
", "rawMarkdown": "Lol, simple models without features give 0.20819. Thanks for sharing a very elegant solution!" } ] }, { "id": 521877, "postDate": "2019-04-23T15:11:09.623Z", "content": "Thanks for sharing! Congratulations!
", "rawMarkdown": "Thanks for sharing! Congratulations!", "votes": 1 }, { "id": 521491, "postDate": "2019-04-23T01:14:45.263Z", "content": "Thanks for your overview, Pavel!
", "rawMarkdown": "Thanks for your overview, Pavel!", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:45:17Z", "competition_id": 12797, "competition_slug": "gendered-pronoun-resolution", "team_id": 2962663, "solution_url": "/competitions/gendered-pronoun-resolution/writeups/ods-ai-gg-33rd-place-simple-solution-lb-0-26663" }