![\"\"](\"http://memesmix.net/media/created/834q6o.jpg\")
Many thanks to google and to kaggle for the organization of the competition, and to the awesome community here - @kashnitsky, @mateiionita, @ceshine, @kenkrige and all others.
\n\nThe kernel by Ceshine Lee was a very good starting point - I played with it, using bert-large-cased, added 1d-conv layer with kernel 1, spanextractor, and the best result was around 0.48 on the test_stage1 data. As others, later I discovered that intermediate layers of BERT encoder contain better representation for the model, for my architecture in particular [-5] and [-6]. This was bringing ~0.41 depending on the seed. Afterwards I started adding the features, like syntactic distance between mentions, token distance and so on. It helped a bit, but not much. I was stuck, and at this point finally the guys from ontonotes replied to me, so I started using the external data.
\n\nInitially I though that the data will decide everything in this compeition, so while the models were running I put some efforts putting winobias, winogender, dpr and later ontonotes into gap-format. 1st place solution shows that more data is not always better performance =) Anyway I had the data and wasn't quite sure what to do with them.
\n\nAt first I simply put all external datasets into one big dataset, split it to train and validation and tried to finetune both the bert and the head on it. The idea was to use the learned weights as initalization weights for the 'normal' gap training, where normal is using ~2400 gap datapoints with 5 fold CV. It didn't work, score became worse. Then I only trained a head for a 2 epochs without tuning the bert and it helped to come from ~0.4 to ~0.36. So I had the observation that better initialization of the head weights lead to better minima. I decided to work a bit on this idea. I started to use winobias as validation data, thus minimizing the gender bias. This brought somewhere around 0.35. Then (it was 2 days before the end of the competition) I tried training the head, then freeze the head and fine-tune the bert weights (only one layer either [-5] or [-6]) with very small learning rate. It had some positive effect and put me around 0.33-34.
\n\nAfterwards I had no time for interesting stuff anymore, so I started with ensembling - I run 4 models overall, bert-large-cased with [-5] layer, bert-large-uncased[-5], bert-large-cased[-6] and bert-large-cased[-6]. The final prediction is just the average of all models with clipping at 1e-2.
\n\nHere is the final overview of the architecture I used:\nBERT -> Conv1d with kernel=1 (1024x64) -> SelfAttentiveSpanExtractor(64) * 3 -> BatchNorm1d() -> fully connected (64*3 x 64) -> dropout (0.6) -> fully connected (64 x 81) (here all the manual features come in) -> fully connected (81 x 3).
", "comments": [ { "id": 522005, "author_name": "abzaliev", "author_url": "", "post_date": "2019-04-23T18:41:46.013000", "content": "Things that didn't work: \n- Preco dataset was too noisy and I couldn't make sense out of it\n- convolutional layer with kernel larger than 1 - still struggle to understand how and why. The intuition was that more context for the span would allow for richer representation, but somehow it didn't work\n- finetuning BERT directly on the GAP data\n- parallelism features from the original GAP paper\n- Elmo-style scalar mixing\n- swapping A and B\n- using the scraped information from wikipedia
", "votes": 5, "replies": [] }, { "id": 522024, "author_name": "Matei Ionita", "author_url": "", "post_date": "2019-04-23T19:12:37.873000", "content": "Thanks for the write-up, and congratulations! Do I understand correctly that you're fine-tuning BERT on other datasets, then freezing it and only training the head on GAP? Thsi idea of fine-tuning on other datasets is one of the main takeaways for me.
", "votes": 3, "replies": [ { "id": 522039, "author_name": "abzaliev", "author_url": "", "post_date": "2019-04-23T19:40:44.570000", "content": "Sorry for the confusion: I first fine-tuned the head with frozen BERT on the external dataset, then fine-tuned the embeddings with frozen head on the same external dataset, and afterwards trained only head on the GAP. \nBtw, without training on GAP at all, after all external training I could get the 0.36 on the 'clean' test stage1 data - don't remember the score on the real 'dirty' data, but should be around 0.44 or smth like this.
", "votes": 5, "replies": [] }, { "id": 522042, "author_name": "abzaliev", "author_url": "", "post_date": "2019-04-23T19:42:32.887000", "content": "Glad that can contribute smth to your learning, as I learned a lot from your kernel and ideas 👍
", "votes": 0, "replies": [] }, { "id": 522053, "author_name": "Matei Ionita", "author_url": "", "post_date": "2019-04-23T19:54:56.723000", "content": "In my team, we were using GAP labels all along for training. You show how to get pretty far with pure transfer learning. That's a very big contribution to my learning!
", "votes": 2, "replies": [] } ] }, { "id": 566792, "author_name": "LoGAN", "author_url": "", "post_date": "2019-07-02T15:50:58.090000", "content": "Thanks for sharing and congratulations! Which dataset is the dpr?
", "votes": 0, "replies": [] }, { "id": 533098, "author_name": "Jack", "author_url": "", "post_date": "2019-05-18T13:02:23.130000", "content": "Congratulations! Thank you for your sharing , I'm a new learner and very interested in this competition.Could you share your code for me to learn?
", "votes": 0, "replies": [] }, { "id": 523376, "author_name": "Prashanth Thangavel", "author_url": "", "post_date": "2019-04-26T05:50:38.973000", "content": "Congratulations! Thanks for sharing the approach.
", "votes": 0, "replies": [] }, { "id": 522232, "author_name": "Ceshine Lee", "author_url": "", "post_date": "2019-04-24T04:19:28.683000", "content": "Congratulations! Really excellent work. I had a feeling that external data would be crucial in this competition, but it takes hard work and conscientiousness like yours to do it right.
", "votes": 0, "replies": [] }, { "id": 522023, "author_name": "Ivan Panshin", "author_url": "", "post_date": "2019-04-23T19:11:57.410000", "content": "Hm. I can't seem to understand why clipping works. Could you give a hand with this?
", "votes": 0, "replies": [ { "id": 522036, "author_name": "abzaliev", "author_url": "", "post_date": "2019-04-23T19:34:09.550000", "content": "the best explanation why clipping is helpful was here. I think that extremely good models (like 1st place) can get without it, but for majority of the models it helpful. I assumed that the test data stage2 ground truth will be noisy and with human errors, so the models will do the mistakes inevitably. Clipping reduce the effect of those mistakes.
", "votes": 3, "replies": [] }, { "id": 522050, "author_name": "Ivan Panshin", "author_url": "", "post_date": "2019-04-23T19:49:38.383000", "content": "Great, thanks!
", "votes": 0, "replies": [] }, { "id": 522085, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-04-23T20:58:07.753000", "content": "Did same clipping with 1e-2 threshold. It's a nice precaution. Interesting though, my late submission without clipping did better (~0.22), a bit confusing.
", "votes": 0, "replies": [] }, { "id": 522086, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-04-23T20:58:54.187000", "content": "One more takeaway: to tune clipping threshold via OOF.
", "votes": 1, "replies": [] }, { "id": 522089, "author_name": "Ivan Panshin", "author_url": "", "post_date": "2019-04-23T21:10:51.147000", "content": "Agreed.
\n\n
Congratulations and thanks for sharing!
", "votes": 0, "replies": [] } ], "comment_count": 16, "messages": [ { "id": 522001, "postDate": "2019-04-23T18:35:34.017Z", "content": "Many thanks to google and to kaggle for the organization of the competition, and to the awesome community here - @kashnitsky, @mateiionita, @ceshine, @kenkrige and all others.
\n\nThe kernel by Ceshine Lee was a very good starting point - I played with it, using bert-large-cased, added 1d-conv layer with kernel 1, spanextractor, and the best result was around 0.48 on the test_stage1 data. As others, later I discovered that intermediate layers of BERT encoder contain better representation for the model, for my architecture in particular [-5] and [-6]. This was bringing ~0.41 depending on the seed. Afterwards I started adding the features, like syntactic distance between mentions, token distance and so on. It helped a bit, but not much. I was stuck, and at this point finally the guys from ontonotes replied to me, so I started using the external data.
\n\nInitially I though that the data will decide everything in this compeition, so while the models were running I put some efforts putting winobias, winogender, dpr and later ontonotes into gap-format. 1st place solution shows that more data is not always better performance =) Anyway I had the data and wasn't quite sure what to do with them.
\n\nAt first I simply put all external datasets into one big dataset, split it to train and validation and tried to finetune both the bert and the head on it. The idea was to use the learned weights as initalization weights for the 'normal' gap training, where normal is using ~2400 gap datapoints with 5 fold CV. It didn't work, score became worse. Then I only trained a head for a 2 epochs without tuning the bert and it helped to come from ~0.4 to ~0.36. So I had the observation that better initialization of the head weights lead to better minima. I decided to work a bit on this idea. I started to use winobias as validation data, thus minimizing the gender bias. This brought somewhere around 0.35. Then (it was 2 days before the end of the competition) I tried training the head, then freeze the head and fine-tune the bert weights (only one layer either [-5] or [-6]) with very small learning rate. It had some positive effect and put me around 0.33-34.
\n\nAfterwards I had no time for interesting stuff anymore, so I started with ensembling - I run 4 models overall, bert-large-cased with [-5] layer, bert-large-uncased[-5], bert-large-cased[-6] and bert-large-cased[-6]. The final prediction is just the average of all models with clipping at 1e-2.
\n\nHere is the final overview of the architecture I used:\nBERT -> Conv1d with kernel=1 (1024x64) -> SelfAttentiveSpanExtractor(64) * 3 -> BatchNorm1d() -> fully connected (64*3 x 64) -> dropout (0.6) -> fully connected (64 x 81) (here all the manual features come in) -> fully connected (81 x 3).
", "rawMarkdown": "Many thanks to google and to kaggle for the organization of the competition, and to the awesome community here - @kashnitsky, @mateiionita, @ceshine, @kenkrige and all others.\n\nThe kernel by Ceshine Lee was a very good starting point - I played with it, using bert-large-cased, added 1d-conv layer with kernel 1, spanextractor, and the best result was around 0.48 on the test_stage1 data. As others, later I discovered that intermediate layers of BERT encoder contain better representation for the model, for my architecture in particular [-5] and [-6]. This was bringing ~0.41 depending on the seed. Afterwards I started adding the features, like syntactic distance between mentions, token distance and so on. It helped a bit, but not much. I was stuck, and at this point finally the guys from ontonotes replied to me, so I started using the external data.\n\n### External data\nInitially I though that the data will decide everything in this compeition, so while the models were running I put some efforts putting winobias, winogender, dpr and later ontonotes into gap-format. 1st place solution shows that more data is not always better performance =) Anyway I had the data and wasn't quite sure what to do with them.\n\nAt first I simply put all external datasets into one big dataset, split it to train and validation and tried to finetune both the bert and the head on it. The idea was to use the learned weights as initalization weights for the 'normal' gap training, where normal is using ~2400 gap datapoints with 5 fold CV. It didn't work, score became worse. Then I only trained a head for a 2 epochs without tuning the bert and it helped to come from ~0.4 to ~0.36. So I had the observation that better initialization of the head weights lead to better minima. I decided to work a bit on this idea. I started to use winobias as validation data, thus minimizing the gender bias. This brought somewhere around 0.35. Then (it was 2 days before the end of the competition) I tried training the head, then freeze the head and fine-tune the bert weights (only one layer either [-5] or [-6]) with very small learning rate. It had some positive effect and put me around 0.33-34. \n\nAfterwards I had no time for interesting stuff anymore, so I started with ensembling - I run 4 models overall, bert-large-cased with [-5] layer, bert-large-uncased[-5], bert-large-cased[-6] and bert-large-cased[-6]. The final prediction is just the average of all models with clipping at 1e-2. \n\n### Final Architecture\nHere is the final overview of the architecture I used:\nBERT -> Conv1d with kernel=1 (1024x64) -> SelfAttentiveSpanExtractor(64) * 3 -> BatchNorm1d() -> fully connected (64*3 x 64) -> dropout (0.6) -> fully connected (64 x 81) (here all the manual features come in) -> fully connected (81 x 3). ", "votes": 26 }, { "id": 522005, "postDate": "2019-04-23T18:41:46.013Z", "content": "Things that didn't work: \n- Preco dataset was too noisy and I couldn't make sense out of it\n- convolutional layer with kernel larger than 1 - still struggle to understand how and why. The intuition was that more context for the span would allow for richer representation, but somehow it didn't work\n- finetuning BERT directly on the GAP data\n- parallelism features from the original GAP paper\n- Elmo-style scalar mixing\n- swapping A and B\n- using the scraped information from wikipedia
", "rawMarkdown": "Things that didn't work: \n- Preco dataset was too noisy and I couldn't make sense out of it\n- convolutional layer with kernel larger than 1 - still struggle to understand how and why. The intuition was that more context for the span would allow for richer representation, but somehow it didn't work\n- finetuning BERT directly on the GAP data\n- parallelism features from the original GAP paper\n- Elmo-style scalar mixing\n- swapping A and B\n- using the scraped information from wikipedia", "votes": 5 }, { "id": 522024, "postDate": "2019-04-23T19:12:37.873Z", "content": "Thanks for the write-up, and congratulations! Do I understand correctly that you're fine-tuning BERT on other datasets, then freezing it and only training the head on GAP? Thsi idea of fine-tuning on other datasets is one of the main takeaways for me.
", "rawMarkdown": "Thanks for the write-up, and congratulations! Do I understand correctly that you're fine-tuning BERT on other datasets, then freezing it and only training the head on GAP? Thsi idea of fine-tuning on other datasets is one of the main takeaways for me.", "votes": 3, "replies": [ { "id": 522039, "postDate": "2019-04-23T19:40:44.570Z", "content": "Sorry for the confusion: I first fine-tuned the head with frozen BERT on the external dataset, then fine-tuned the embeddings with frozen head on the same external dataset, and afterwards trained only head on the GAP. \nBtw, without training on GAP at all, after all external training I could get the 0.36 on the 'clean' test stage1 data - don't remember the score on the real 'dirty' data, but should be around 0.44 or smth like this.
", "rawMarkdown": "Sorry for the confusion: I first fine-tuned the head with frozen BERT on the external dataset, then fine-tuned the embeddings with frozen head on the same external dataset, and afterwards trained only head on the GAP. \nBtw, without training on GAP at all, after all external training I could get the 0.36 on the 'clean' test stage1 data - don't remember the score on the real 'dirty' data, but should be around 0.44 or smth like this. ", "votes": 5 }, { "id": 522042, "postDate": "2019-04-23T19:42:32.887Z", "content": "Glad that can contribute smth to your learning, as I learned a lot from your kernel and ideas 👍
", "rawMarkdown": "Glad that can contribute smth to your learning, as I learned a lot from your kernel and ideas 👍 " }, { "id": 522053, "postDate": "2019-04-23T19:54:56.723Z", "content": "In my team, we were using GAP labels all along for training. You show how to get pretty far with pure transfer learning. That's a very big contribution to my learning!
", "rawMarkdown": "In my team, we were using GAP labels all along for training. You show how to get pretty far with pure transfer learning. That's a very big contribution to my learning!", "votes": 2 } ] }, { "id": 566792, "postDate": "2019-07-02T15:50:58.090Z", "content": "Thanks for sharing and congratulations! Which dataset is the dpr?
", "rawMarkdown": "Thanks for sharing and congratulations! Which dataset is the dpr?" }, { "id": 533098, "postDate": "2019-05-18T13:02:23.130Z", "content": "Congratulations! Thank you for your sharing , I'm a new learner and very interested in this competition.Could you share your code for me to learn?
", "rawMarkdown": "Congratulations! Thank you for your sharing , I'm a new learner and very interested in this competition.Could you share your code for me to learn?" }, { "id": 523376, "postDate": "2019-04-26T05:50:38.973Z", "content": "Congratulations! Thanks for sharing the approach.
", "rawMarkdown": "Congratulations! Thanks for sharing the approach. " }, { "id": 522232, "postDate": "2019-04-24T04:19:28.683Z", "content": "Congratulations! Really excellent work. I had a feeling that external data would be crucial in this competition, but it takes hard work and conscientiousness like yours to do it right.
", "rawMarkdown": "Congratulations! Really excellent work. I had a feeling that external data would be crucial in this competition, but it takes hard work and conscientiousness like yours to do it right. " }, { "id": 522023, "postDate": "2019-04-23T19:11:57.410Z", "content": "Hm. I can't seem to understand why clipping works. Could you give a hand with this?
", "rawMarkdown": "Hm. I can't seem to understand why clipping works. Could you give a hand with this? ", "replies": [ { "id": 522036, "postDate": "2019-04-23T19:34:09.550Z", "content": "the best explanation why clipping is helpful was here. I think that extremely good models (like 1st place) can get without it, but for majority of the models it helpful. I assumed that the test data stage2 ground truth will be noisy and with human errors, so the models will do the mistakes inevitably. Clipping reduce the effect of those mistakes.
", "rawMarkdown": "the best explanation why clipping is helpful was [here](https://www.kaggle.com/c/statoil-iceberg-classifier-challenge/discussion/48241#latest-306293). I think that extremely good models (like 1st place) can get without it, but for majority of the models it helpful. I assumed that the test data stage2 ground truth will be noisy and with human errors, so the models will do the mistakes inevitably. Clipping reduce the effect of those mistakes.", "votes": 3 }, { "id": 522050, "postDate": "2019-04-23T19:49:38.383Z", "content": "Great, thanks!
", "rawMarkdown": "Great, thanks! " }, { "id": 522085, "postDate": "2019-04-23T20:58:07.753Z", "content": "Did same clipping with 1e-2 threshold. It's a nice precaution. Interesting though, my late submission without clipping did better (~0.22), a bit confusing.
", "rawMarkdown": "Did same clipping with 1e-2 threshold. It's a nice precaution. Interesting though, my late submission without clipping did better (~0.22), a bit confusing." }, { "id": 522086, "postDate": "2019-04-23T20:58:54.187Z", "content": "One more takeaway: to tune clipping threshold via OOF.
", "rawMarkdown": "One more takeaway: to tune clipping threshold via OOF.", "votes": 1 }, { "id": 522089, "postDate": "2019-04-23T21:10:51.147Z", "content": "Agreed.
\n\n
Congratulations and thanks for sharing!
", "rawMarkdown": "Congratulations and thanks for sharing!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:44:56Z", "competition_id": 12797, "competition_slug": "gendered-pronoun-resolution", "team_id": 2841804, "solution_url": "/competitions/gendered-pronoun-resolution/writeups/ods-ai-abzaliev-3rd-place-solution" }