Our approach was based on extracting co-attention scores from pretrained bert model.
\n\nSo this are selected lines for our preprocessing pipeline:
\n\n```yaml\n preprocess: \n - try_search_near: # try to search for a mention of the names that is located closer to pronoun in question\n - replace_unknowns_preprocessor: #replace names that are not the part or pipeline \n gender: true #try to select name of appropriate gender when gender statistics is known\n cased: false # use lower case\n lastNames: true # try to replace last names to well known lastnames\n - bert_encode: #extract attention scores from bert \n maxToken: true #if named entity spans for several tokens take token with maximum attention score\n - disk-cache:\n split: true
\n - extract_token_bert_scores_3class:
\n startLayer: 9 #start from 9 layer of birt (was selected based on cv)
```\nThen after this pipeline we have applied two models:
\n\n1) NN:\n\n net:\n - flatten\n - dropout: 0.5\n - dense: [144, relu]\n - dense: [3, softmax]\n\n2) LGBM - with hyperparameter search
in bose cases we have used startified 10 fold, CV and blended models accross folds.
\n\nBig Error: We forgot to use test augmenation in our final predictions(you may replace names to a different one that are known to BERT and average predictions) , this costed us gold medal.
\n\nRegards,\nPavel
", "comments": [ { "id": 523378, "author_name": "Prashanth Thangavel", "author_url": "", "post_date": "2019-04-26T05:50:57.220000", "content": "Congratulations! Thanks for sharing the approach.
", "votes": 0, "replies": [] }, { "id": 522168, "author_name": "Hai Pham", "author_url": "", "post_date": "2019-04-24T00:26:54.280000", "content": "Hi Pavel, thanks for sharing.
\n\nWe #13 also had similar approach, extracting attention scores from the pretrained BERT. One thing we didn't have time to try is that to finetune BERT to get GAP-oriented attention scores.
", "votes": 0, "replies": [ { "id": 522210, "author_name": "Pavel Petrochenko", "author_url": "", "post_date": "2019-04-24T03:13:22.340000", "content": "Hi Hai,
\n\n\n\n\nOne thing we didn't have time to try is that to fine tune BERT to get GAP-oriented attention scores.
\n
We also don't used fine tuning, but I really believe that our model is still pretty far from what can be achieved without it:
\n\nreserves that we do not used:\n- Test time aug (even extremely basic implementation with one extra option immediately gives 0.20316)\n- Accumulating scores against all mentions of name instead of nearest one\n- Extra features like page name, which may become significant when attention based methods does not give a clear winner.\n- Use more then one pre trained Bert\n- e.t.c
\n\nAny case it was a great competition and a lot of fun.\nRegards,\nPavel
", "votes": 2, "replies": [] } ] }, { "id": 521968, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-04-23T17:41:23.793000", "content": "Thanks for the writeup! And who knows... maybe you'll end up with 🏅
", "votes": 0, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 521962, "postDate": "2019-04-23T17:32:13.957Z", "content": "Our approach was based on extracting co-attention scores from pretrained bert model.
\n\nSo this are selected lines for our preprocessing pipeline:
\n\n```yaml\n preprocess: \n - try_search_near: # try to search for a mention of the names that is located closer to pronoun in question\n - replace_unknowns_preprocessor: #replace names that are not the part or pipeline \n gender: true #try to select name of appropriate gender when gender statistics is known\n cased: false # use lower case\n lastNames: true # try to replace last names to well known lastnames\n - bert_encode: #extract attention scores from bert \n maxToken: true #if named entity spans for several tokens take token with maximum attention score\n - disk-cache:\n split: true
\n - extract_token_bert_scores_3class:
\n startLayer: 9 #start from 9 layer of birt (was selected based on cv)
```\nThen after this pipeline we have applied two models:
\n\n1) NN:\n\n net:\n - flatten\n - dropout: 0.5\n - dense: [144, relu]\n - dense: [3, softmax]\n\n2) LGBM - with hyperparameter search
in bose cases we have used startified 10 fold, CV and blended models accross folds.
\n\nBig Error: We forgot to use test augmenation in our final predictions(you may replace names to a different one that are known to BERT and average predictions) , this costed us gold medal.
\n\nRegards,\nPavel
", "rawMarkdown": "Our approach was based on extracting co-attention scores from pretrained bert model. \n\nSo this are selected lines for our preprocessing pipeline:\n\n```yaml\n preprocess: \n - try_search_near: # try to search for a mention of the names that is located closer to pronoun in question\n - replace_unknowns_preprocessor: #replace names that are not the part or pipeline \n gender: true #try to select name of appropriate gender when gender statistics is known\n cased: false # use lower case\n lastNames: true # try to replace last names to well known lastnames\n - bert_encode: #extract attention scores from bert \n maxToken: true #if named entity spans for several tokens take token with maximum attention score\n - disk-cache:\n split: true \n - extract_token_bert_scores_3class: \n startLayer: 9 #start from 9 layer of birt (was selected based on cv)\n\n```\nThen after this pipeline we have applied two models:\n\n1) NN:\n```\n net:\n - flatten\n - dropout: 0.5\n - dense: [144, relu]\n - dense: [3, softmax]\n```\n2) LGBM - with hyperparameter search\n\nin bose cases we have used startified 10 fold, CV and blended models accross folds.\n\n**Big Error:** We forgot to use test augmenation in our final predictions(you may replace names to a different one that are known to BERT and average predictions) , this costed us gold medal.\n\nRegards,\nPavel\n\n\n\n\n\n", "votes": 9 }, { "id": 523378, "postDate": "2019-04-26T05:50:57.220Z", "content": "Congratulations! Thanks for sharing the approach.
", "rawMarkdown": "Congratulations! Thanks for sharing the approach. " }, { "id": 522168, "postDate": "2019-04-24T00:26:54.280Z", "content": "Hi Pavel, thanks for sharing.
\n\nWe #13 also had similar approach, extracting attention scores from the pretrained BERT. One thing we didn't have time to try is that to finetune BERT to get GAP-oriented attention scores.
", "rawMarkdown": "Hi Pavel, thanks for sharing.\n\nWe #13 also had similar approach, extracting attention scores from the pretrained BERT. One thing we didn't have time to try is that to finetune BERT to get GAP-oriented attention scores.", "replies": [ { "id": 522210, "postDate": "2019-04-24T03:13:22.340Z", "content": "Hi Hai,
\n\n\n\n\nOne thing we didn't have time to try is that to fine tune BERT to get GAP-oriented attention scores.
\n
We also don't used fine tuning, but I really believe that our model is still pretty far from what can be achieved without it:
\n\nreserves that we do not used:\n- Test time aug (even extremely basic implementation with one extra option immediately gives 0.20316)\n- Accumulating scores against all mentions of name instead of nearest one\n- Extra features like page name, which may become significant when attention based methods does not give a clear winner.\n- Use more then one pre trained Bert\n- e.t.c
\n\nAny case it was a great competition and a lot of fun.\nRegards,\nPavel
", "rawMarkdown": "Hi Hai,\n>One thing we didn't have time to try is that to fine tune BERT to get GAP-oriented attention scores.\n\nWe also don't used fine tuning, but I really believe that our model is still pretty far from what can be achieved without it:\n\nreserves that we do not used:\n- Test time aug (even extremely basic implementation with one extra option immediately gives 0.20316)\n- Accumulating scores against all mentions of name instead of nearest one\n- Extra features like page name, which may become significant when attention based methods does not give a clear winner.\n- Use more then one pre trained Bert\n- e.t.c\n\nAny case it was a great competition and a lot of fun.\nRegards,\nPavel", "votes": 2 } ] }, { "id": 521968, "postDate": "2019-04-23T17:41:23.793Z", "content": "Thanks for the writeup! And who knows... maybe you'll end up with 🏅
", "rawMarkdown": "Thanks for the writeup! And who knows... maybe you'll end up with 🏅 " } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:42:33Z", "competition_id": 12797, "competition_slug": "gendered-pronoun-resolution", "team_id": 2727372, "solution_url": "/competitions/gendered-pronoun-resolution/writeups/ods-ai-five-ones-12-place-writeup" }