Our approach was similar to those already described. \nIn a nutshell, the pipeline includes:\n - BERT embeddings\n - hand-crafted features\n - several MLPs
\n\nWe concatenated embeddings for A, B, and Pronoun taken from Cased and Uncased large BERT models - 3 layers (-4, -5, -6 turned out to work best). For each BERT model this yield 9216-dimensional output: 3 (layers) x 3 (entity) x 1024 (BERT embedding size).
\n\nWe ended up with 69 features of different nature (mistake: turns out we should've put more effort on BERT finetuning):\n - Neuralcoref, Stanford NLP and e2e-coref model predictions\n - Predictions of MLP trained with ELMo embeddings\n - Syntactic roles of A, B, and Pronoun (subject, direct object, attribute etc)\n - Positional and frequency-based (distancies between A, B, Pronoun and derivations, whether they all are in the same sentence or Pronoun is in the following one etc.)\n - Dependency tree-based (from this Kernel by @negedng )\n - Named entities predicted for A and B\n - GAP heuristics (from this Kernel by @sattree )
\n\nThe final combined model was a dense classification head built upon output from 5 other models:\n - Two MLPs (like in Matei's kernel) - separate for Cased and Uncased BERTs, both taking 9216-d input and outputing 112-d vectors\n - Two Siamese models with distances between Pronoun and A-embeddings, Pronoun and B-embeddings as inputs and shared weights\n - One more MLP taking 69-d feature vectors as an input
\n\nFinal predictions were clipped with 0.01 threshold (turns out it's better without clipping).
\n\nIf A is the right reference, then substitute B with all other named entities in the sentence.
\n\nExample: [A] John entered the room and saw [B] Mary. [P] She looked so perfect laughing with Jerry and Noah. Btw, Jerry and Noah are Clara's and Julia's best friends\".
\n\nTrue label is B ('She' refers to Mary). I used to augment with 'She', 'Mary' and all other noun phrases in the sentence (according to Spacy POS tagging):
\n\nA | B | Pronoun\nJerry - Mary - She (B is true)\nNoah - Mary - She (B is true)\nClara - Mary - She (B is true)\nJulia - Mary - She (B is true)
\n\nThus the dataset was increased 9x but new instances were much simpler to classify than original ones (~0.15 CV loss, 0.6 test loss). We noticed that the model trained on augmented data tended to make more confident predictions, hence many good answers but some big misses as well. Decided to use it as one more input for stacking.
\n\nThough in the beginning blending helped a lot, when CV loss was ~0.33, for some reason we were not able to have any profit from stacking. We applied it mostly to OOFs built with augmentation (training folds being augmented, validation one - not), maybe we did smth wrong.
\n\nTraining loss would quickly drop to zero, but test loss would be ~0.6. Now we realize that we should've made more attempts.
\n\nOur code (fully reproducible).
", "comments": [ { "id": 523682, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-04-26T18:30:43.877000", "content": "Thanks for sharing the code and well done to your whole team on playing an inspirational role in the competition.
", "votes": 1, "replies": [] }, { "id": 523375, "author_name": "Prashanth Thangavel", "author_url": "", "post_date": "2019-04-26T05:50:31.893000", "content": "Congratulations! Thanks for sharing the approach.
", "votes": 1, "replies": [] }, { "id": 522045, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-04-23T19:46:18.390000", "content": "Congratulations and thanks for this great write-up!
", "votes": 1, "replies": [] }, { "id": 522044, "author_name": "abzaliev", "author_url": "", "post_date": "2019-04-23T19:45:58.737000", "content": "I like the idea about the siamese networks, will take a closer look. And thanks for the code!
", "votes": 1, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 522034, "postDate": "2019-04-23T19:32:55.350Z", "content": "Our approach was similar to those already described. \nIn a nutshell, the pipeline includes:\n - BERT embeddings\n - hand-crafted features\n - several MLPs
\n\nWe concatenated embeddings for A, B, and Pronoun taken from Cased and Uncased large BERT models - 3 layers (-4, -5, -6 turned out to work best). For each BERT model this yield 9216-dimensional output: 3 (layers) x 3 (entity) x 1024 (BERT embedding size).
\n\nWe ended up with 69 features of different nature (mistake: turns out we should've put more effort on BERT finetuning):\n - Neuralcoref, Stanford NLP and e2e-coref model predictions\n - Predictions of MLP trained with ELMo embeddings\n - Syntactic roles of A, B, and Pronoun (subject, direct object, attribute etc)\n - Positional and frequency-based (distancies between A, B, Pronoun and derivations, whether they all are in the same sentence or Pronoun is in the following one etc.)\n - Dependency tree-based (from this Kernel by @negedng )\n - Named entities predicted for A and B\n - GAP heuristics (from this Kernel by @sattree )
\n\nThe final combined model was a dense classification head built upon output from 5 other models:\n - Two MLPs (like in Matei's kernel) - separate for Cased and Uncased BERTs, both taking 9216-d input and outputing 112-d vectors\n - Two Siamese models with distances between Pronoun and A-embeddings, Pronoun and B-embeddings as inputs and shared weights\n - One more MLP taking 69-d feature vectors as an input
\n\nFinal predictions were clipped with 0.01 threshold (turns out it's better without clipping).
\n\nIf A is the right reference, then substitute B with all other named entities in the sentence.
\n\nExample: [A] John entered the room and saw [B] Mary. [P] She looked so perfect laughing with Jerry and Noah. Btw, Jerry and Noah are Clara's and Julia's best friends\".
\n\nTrue label is B ('She' refers to Mary). I used to augment with 'She', 'Mary' and all other noun phrases in the sentence (according to Spacy POS tagging):
\n\nA | B | Pronoun\nJerry - Mary - She (B is true)\nNoah - Mary - She (B is true)\nClara - Mary - She (B is true)\nJulia - Mary - She (B is true)
\n\nThus the dataset was increased 9x but new instances were much simpler to classify than original ones (~0.15 CV loss, 0.6 test loss). We noticed that the model trained on augmented data tended to make more confident predictions, hence many good answers but some big misses as well. Decided to use it as one more input for stacking.
\n\nThough in the beginning blending helped a lot, when CV loss was ~0.33, for some reason we were not able to have any profit from stacking. We applied it mostly to OOFs built with augmentation (training folds being augmented, validation one - not), maybe we did smth wrong.
\n\nTraining loss would quickly drop to zero, but test loss would be ~0.6. Now we realize that we should've made more attempts.
\n\nOur code (fully reproducible).
", "rawMarkdown": "## 1. Our final solution\nOur approach was similar to those already described. \nIn a nutshell, the pipeline includes:\n - BERT embeddings\n - hand-crafted features\n - several MLPs\n\n### 1.1. BERT embeddings\nWe concatenated embeddings for A, B, and Pronoun taken from Cased and Uncased large BERT models - 3 layers (-4, -5, -6 turned out to work best). For each BERT model this yield 9216-dimensional output: 3 (layers) x 3 (entity) x 1024 (BERT embedding size). \n\n### 1.2 Hand-crafted features\nWe ended up with 69 features of different nature (**mistake:** turns out we should've put more effort on BERT finetuning):\n - Neuralcoref, Stanford NLP and e2e-coref model predictions\n - Predictions of MLP trained with ELMo embeddings\n - Syntactic roles of A, B, and Pronoun (subject, direct object, attribute etc)\n - Positional and frequency-based (distancies between A, B, Pronoun and derivations, whether they all are in the same sentence or Pronoun is in the following one etc.)\n - Dependency tree-based (from [this](https://www.kaggle.com/negedng/extracting-features-from-spacy-dependency) Kernel by @negedng )\n - Named entities predicted for A and B\n - GAP heuristics (from [this](https://www.kaggle.com/sattree/2-reproducing-gap-results) Kernel by @sattree )\n\n### 1.3. Models\nThe final combined model was a dense classification head built upon output from 5 other models:\n - Two MLPs (like in Matei's [kernel](https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline)) - separate for Cased and Uncased BERTs, both taking 9216-d input and outputing 112-d vectors\n - Two Siamese models with distances between Pronoun and A-embeddings, Pronoun and B-embeddings as inputs and shared weights\n - One more MLP taking 69-d feature vectors as an input \n\nFinal predictions were clipped with 0.01 threshold (turns out it's better without clipping).\n\n## 2. What didn't work for us\n### 2.1. Augmentation\nIf A is the right reference, then substitute B with all other named entities in the sentence.\n\n*Example: [A] John entered the room and saw [B] Mary. [P] She looked so perfect laughing with Jerry and Noah. Btw, Jerry and Noah are Clara's and Julia's best friends\".*\n\nTrue label is B ('She' refers to Mary). I used to augment with 'She', 'Mary' and all other noun phrases in the sentence (according to Spacy POS tagging):\n\nA | B | Pronoun\nJerry - Mary - She (B is true)\nNoah - Mary - She (B is true)\nClara - Mary - She (B is true)\nJulia - Mary - She (B is true)\n\nThus the dataset was increased 9x but new instances were much simpler to classify than original ones (~0.15 CV loss, 0.6 test loss). We noticed that the model trained on augmented data tended to make more confident predictions, hence many good answers but some big misses as well. Decided to use it as one more input for stacking.\n\n### 2.2. Stacking\nThough in the beginning blending helped a lot, when CV loss was ~0.33, for some reason we were not able to have any profit from stacking. We applied it mostly to OOFs built with augmentation (training folds being augmented, validation one - not), maybe we did smth wrong.\n\n### 2.3. BERT finetuning\nTraining loss would quickly drop to zero, but test loss would be ~0.6. Now we realize that we should've made more attempts. \n\n## 3. Afterthoughts\n - Even though it's very disappointing to miss the golden zone, still these emotions will fade away and the competition is the best one (that I personally took part in), that's why [thanks](https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/87660#latest-507277) as early as 3 weeks before the end. Many thanks to Google AI and Kaggle for organizing this research competition! Thanks also to @juliaelliott and @wcukierski for handling all our questions. Lastly, thanks to my teammates - without @mateiionita the whole competition would probably never be the same, but others @vlarine @dennislo and @atanasova also worked hard\n - Blind LB played a trick wit us. Have to admit we were a bit over-confident about our running relative position :) \n - Some takeaways for us are prototyping all components (including stacking) on the early stage, and putting more efforts on \"killer features\", also finetuning BERT with larger pronoun-related datasets (instead of overfitting to the small GAP dataset) looks like a good idea to reuse. \n\n\n[Our code](https://github.com/vlarine/kaggle/tree/master/gendered-pronoun-resolution) (fully reproducible). \n", "votes": 23 }, { "id": 523682, "postDate": "2019-04-26T18:30:43.877Z", "content": "Thanks for sharing the code and well done to your whole team on playing an inspirational role in the competition.
", "rawMarkdown": "Thanks for sharing the code and well done to your whole team on playing an inspirational role in the competition.", "votes": 1 }, { "id": 523375, "postDate": "2019-04-26T05:50:31.893Z", "content": "Congratulations! Thanks for sharing the approach.
", "rawMarkdown": "Congratulations! Thanks for sharing the approach. ", "votes": 1 }, { "id": 522045, "postDate": "2019-04-23T19:46:18.390Z", "content": "Congratulations and thanks for this great write-up!
", "rawMarkdown": "Congratulations and thanks for this great write-up!", "votes": 1 }, { "id": 522044, "postDate": "2019-04-23T19:45:58.737Z", "content": "I like the idea about the siamese networks, will take a closer look. And thanks for the code!
", "rawMarkdown": "I like the idea about the siamese networks, will take a closer look. And thanks for the code! ", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:42:40Z", "competition_id": 12797, "competition_slug": "gendered-pronoun-resolution", "team_id": 2732134, "solution_url": "/competitions/gendered-pronoun-resolution/writeups/ods-ai-five-zeros-22nd-place-solution-and-some-aft" }