My approach was based purely on fine-tuning large Bert models and heavy ensembling to leverage the instability of fine-tuned Bert models. I started by fine-tuning large Bert on several tasks, most of them about pronoun resolution (swag, definite pronouns, 2 flavours of PreCo and WinoBias). Each of those models is then fitted on the GAP dataset a number of times (10+). Afterwards, I ensemble, through simple averaging, the top 50% models by validation loss per task. At this point, I have 5 different set of test predictions (ensemble_swag.csv, ensemble_wino_bias.csv, ensemble_preco.csv ...). Ensembling those, again through simple averaging we arrive at our final prediction set.
\n\nSeveral pre-training task (QA, classification, etc...) have been tried and most hurt results. Bert large vanilla directly tuned on GAP also hurt overall results when added to the ensembling. It seems that, with the exception of Swag, only directly related fine-tuning task have yielded positive results.
\n\nGenerally, the best hyperparameter for any fine-tuning has been: learning_rate = 1e-5, batch_size = 32, warmup_proportion = 0.1, epochs = 2.
\n\nI believe the reason ensembling worked so well is the instability of the model. Different seed could yield an evaluation loss from 0.33 to 0.40, so it seemed likely that those models learnt different things. Ensembling brought the stage 1 test loss from .33 to .296.
\n\nI had issues getting the conll2012 dataset but I believe the coreference task would of ensembled well with the rest.
\n\nCongratulations to all and thank you for organising such a cool competition!
", "comments": [ { "id": 523381, "author_name": "Prashanth Thangavel", "author_url": "", "post_date": "2019-04-26T05:53:17.527000", "content": "Congratulations! Thanks for sharing the approach.
", "votes": 1, "replies": [] }, { "id": 522238, "author_name": "Ceshine Lee", "author_url": "", "post_date": "2019-04-24T04:42:18.200000", "content": "Thanks for the overview and congrats on the gold medal. The sharing of hyperparameters is quite helpful.
", "votes": 1, "replies": [] }, { "id": 522237, "author_name": "Arjun Chandrababu", "author_url": "", "post_date": "2019-04-24T04:39:19.723000", "content": "Congratulations for winning the Gold medal and thanks for sharing hyperparameters content. It's quite helpful!
", "votes": 1, "replies": [] }, { "id": 521810, "author_name": "Andrii Lebukhorskyi", "author_url": "", "post_date": "2019-04-23T13:11:46.673000", "content": "Thx)
", "votes": 1, "replies": [] }, { "id": 523450, "author_name": "Himanshu Soni", "author_url": "", "post_date": "2019-04-26T09:55:22.633000", "content": "Congratulations!!!!!!!!
", "votes": 2, "replies": [] }, { "id": 521557, "author_name": "Matei Ionita", "author_url": "", "post_date": "2019-04-23T03:29:02.577000", "content": "Tuning BERT on other datasets, then fitting on GAP. That's a nice approach, I'll remember it. Congrats!
", "votes": 2, "replies": [] }, { "id": 521525, "author_name": "SRK", "author_url": "", "post_date": "2019-04-23T02:16:40.330000", "content": "Thank you @octthe16th for the solution overview and the best hyper parameters for the BERT tuning. Congrats on the top 10 finish !
", "votes": 2, "replies": [] }, { "id": 522048, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-04-23T19:46:54.167000", "content": "Congratulations and thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 521876, "author_name": "dhaqui the kaggler", "author_url": "", "post_date": "2019-04-23T15:10:52.417000", "content": "Thanks for sharing! Congratulations!
", "votes": 1, "replies": [] }, { "id": 521732, "author_name": "", "author_url": "", "post_date": "2019-04-23T10:50:03.440000", "content": "Congratulations and thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 521552, "author_name": "Tsukasa Moritoki", "author_url": "", "post_date": "2019-04-23T03:18:54.733000", "content": "Thanks for the sharing.
", "votes": 1, "replies": [] }, { "id": 521526, "author_name": "Ilham Firdausi Putra", "author_url": "", "post_date": "2019-04-23T02:18:09.527000", "content": "Thank you for sharing your solution!
", "votes": 1, "replies": [] } ], "comment_count": 12, "messages": [ { "id": 521519, "postDate": "2019-04-23T01:53:20.593Z", "content": "My approach was based purely on fine-tuning large Bert models and heavy ensembling to leverage the instability of fine-tuned Bert models. I started by fine-tuning large Bert on several tasks, most of them about pronoun resolution (swag, definite pronouns, 2 flavours of PreCo and WinoBias). Each of those models is then fitted on the GAP dataset a number of times (10+). Afterwards, I ensemble, through simple averaging, the top 50% models by validation loss per task. At this point, I have 5 different set of test predictions (ensemble_swag.csv, ensemble_wino_bias.csv, ensemble_preco.csv ...). Ensembling those, again through simple averaging we arrive at our final prediction set.
\n\nSeveral pre-training task (QA, classification, etc...) have been tried and most hurt results. Bert large vanilla directly tuned on GAP also hurt overall results when added to the ensembling. It seems that, with the exception of Swag, only directly related fine-tuning task have yielded positive results.
\n\nGenerally, the best hyperparameter for any fine-tuning has been: learning_rate = 1e-5, batch_size = 32, warmup_proportion = 0.1, epochs = 2.
\n\nI believe the reason ensembling worked so well is the instability of the model. Different seed could yield an evaluation loss from 0.33 to 0.40, so it seemed likely that those models learnt different things. Ensembling brought the stage 1 test loss from .33 to .296.
\n\nI had issues getting the conll2012 dataset but I believe the coreference task would of ensembled well with the rest.
\n\nCongratulations to all and thank you for organising such a cool competition!
", "rawMarkdown": "My approach was based purely on fine-tuning large Bert models and heavy ensembling to leverage the instability of fine-tuned Bert models. I started by fine-tuning large Bert on several tasks, most of them about pronoun resolution (swag, definite pronouns, 2 flavours of PreCo and WinoBias). Each of those models is then fitted on the GAP dataset a number of times (10+). Afterwards, I ensemble, through simple averaging, the top 50% models by validation loss per task. At this point, I have 5 different set of test predictions (ensemble_swag.csv, ensemble_wino_bias.csv, ensemble_preco.csv ...). Ensembling those, again through simple averaging we arrive at our final prediction set.\n\nSeveral pre-training task (QA, classification, etc...) have been tried and most hurt results. Bert large vanilla directly tuned on GAP also hurt overall results when added to the ensembling. It seems that, with the exception of Swag, only directly related fine-tuning task have yielded positive results. \n\nGenerally, the best hyperparameter for any fine-tuning has been: learning_rate = 1e-5, batch_size = 32, warmup_proportion = 0.1, epochs = 2. \n\nI believe the reason ensembling worked so well is the instability of the model. Different seed could yield an evaluation loss from 0.33 to 0.40, so it seemed likely that those models learnt different things. Ensembling brought the stage 1 test loss from .33 to .296.\n\nI had issues getting the conll2012 dataset but I believe the coreference task would of ensembled well with the rest.\n\nCongratulations to all and thank you for organising such a cool competition!", "votes": 32 }, { "id": 523381, "postDate": "2019-04-26T05:53:17.527Z", "content": "Congratulations! Thanks for sharing the approach.
", "rawMarkdown": "Congratulations! Thanks for sharing the approach. ", "votes": 1 }, { "id": 522238, "postDate": "2019-04-24T04:42:18.200Z", "content": "Thanks for the overview and congrats on the gold medal. The sharing of hyperparameters is quite helpful.
", "rawMarkdown": "Thanks for the overview and congrats on the gold medal. The sharing of hyperparameters is quite helpful.", "votes": 1 }, { "id": 522237, "postDate": "2019-04-24T04:39:19.723Z", "content": "Congratulations for winning the Gold medal and thanks for sharing hyperparameters content. It's quite helpful!
", "rawMarkdown": "Congratulations for winning the Gold medal and thanks for sharing hyperparameters content. It's quite helpful!", "votes": 1 }, { "id": 521810, "postDate": "2019-04-23T13:11:46.673Z", "content": "Thx)
", "rawMarkdown": "Thx)", "votes": 1 }, { "id": 523450, "postDate": "2019-04-26T09:55:22.633Z", "content": "Congratulations!!!!!!!!
", "rawMarkdown": "Congratulations!!!!!!!! ", "votes": 2 }, { "id": 521557, "postDate": "2019-04-23T03:29:02.577Z", "content": "Tuning BERT on other datasets, then fitting on GAP. That's a nice approach, I'll remember it. Congrats!
", "rawMarkdown": "Tuning BERT on other datasets, then fitting on GAP. That's a nice approach, I'll remember it. Congrats!", "votes": 2 }, { "id": 521525, "postDate": "2019-04-23T02:16:40.330Z", "content": "Thank you @octthe16th for the solution overview and the best hyper parameters for the BERT tuning. Congrats on the top 10 finish !
", "rawMarkdown": "Thank you @octthe16th for the solution overview and the best hyper parameters for the BERT tuning. Congrats on the top 10 finish !", "votes": 2 }, { "id": 522048, "postDate": "2019-04-23T19:46:54.167Z", "content": "Congratulations and thanks for sharing!
", "rawMarkdown": "Congratulations and thanks for sharing!", "votes": 1 }, { "id": 521876, "postDate": "2019-04-23T15:10:52.417Z", "content": "Thanks for sharing! Congratulations!
", "rawMarkdown": "Thanks for sharing! Congratulations!", "votes": 1 }, { "id": 521732, "postDate": "2019-04-23T10:50:03.440Z", "content": "Congratulations and thanks for sharing!
", "rawMarkdown": "Congratulations and thanks for sharing!", "votes": 1 }, { "id": 521552, "postDate": "2019-04-23T03:18:54.733Z", "content": "Thanks for the sharing.
", "rawMarkdown": "Thanks for the sharing.", "votes": 1 }, { "id": 521526, "postDate": "2019-04-23T02:18:09.527Z", "content": "Thank you for sharing your solution!
", "rawMarkdown": "Thank you for sharing your solution!", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:43:59Z", "competition_id": 12797, "competition_slug": "gendered-pronoun-resolution", "team_id": 2773110, "solution_url": "/competitions/gendered-pronoun-resolution/writeups/louisclouatre-8th-place-solution-lb-0-20138" }