{ "id": 90366, "title": "15th place solution [0.21074]", "content": "

My approach was quite straightforward for this competition I have used embeddings, extracted by BERT + some other features and fully connected NN to get the final predictions.

\n

Embeddings:\nI have used Bert large uncased (without any finetuning) and extracted embeddings from 9 layers – from -1 to -9. I have concatenated all of them and used as a 1024*9 = 9216-dimension vectors as final embeddings. Other sets of layers gave me worse results.

\n

Additional features:

\n\n

Additional data:

\n\n

Model:

\n
    \n
  1. All other (non-bert) features were passed through the linear layer:\nOther_features (separate A+P and B+P)-> bn -> linear -> relu -> preprocessed other features
    2. \n
  2. Concatenated other and bert features were passed through the FCNN:\n(A/B BERT emb + preprocessed other features) -> linear(1800) -> relu -> dropout -> bn -> linear(500) -> relu -> dropout -> linear(1) -> A/B pred
    3. \n
  3. The final prediction = softmax(A pred, B pred, 0)
    4. \n
\n

Training:

\n", "comments": [ { "id": 523380, "author_name": "Prashanth Thangavel", "author_url": "", "post_date": "2019-04-26T05:52:27.070000", "content": "

Congratulations! Thanks for sharing the approach.

", "votes": 0, "replies": [] }, { "id": 522120, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-04-23T22:04:55.200000", "content": "

Congrats, Ilya! I'm humbled :)

", "votes": 0, "replies": [] }, { "id": 521733, "author_name": "", "author_url": "", "post_date": "2019-04-23T10:50:10.013000", "content": "

Congratulations and thanks for sharing!

", "votes": 1, "replies": [] }, { "id": 522049, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-04-23T19:47:01.603000", "content": "

Congratulations and thanks for sharing!

", "votes": 0, "replies": [] }, { "id": 521875, "author_name": "dhaqui the kaggler", "author_url": "", "post_date": "2019-04-23T15:10:31.457000", "content": "

Thanks for sharing! Congratulations!

", "votes": 0, "replies": [] } ], "comment_count": 5, "messages": [ { "id": 521642, "postDate": "2019-04-23T07:24:29.450Z", "content": "

My approach was quite straightforward for this competition I have used embeddings, extracted by BERT + some other features and fully connected NN to get the final predictions.

\n

Embeddings:\nI have used Bert large uncased (without any finetuning) and extracted embeddings from 9 layers – from -1 to -9. I have concatenated all of them and used as a 1024*9 = 9216-dimension vectors as final embeddings. Other sets of layers gave me worse results.

\n

Additional features:

\n\n

Additional data:

\n\n

Model:

\n
    \n
  1. All other (non-bert) features were passed through the linear layer:\nOther_features (separate A+P and B+P)-> bn -> linear -> relu -> preprocessed other features
    2. \n
  2. Concatenated other and bert features were passed through the FCNN:\n(A/B BERT emb + preprocessed other features) -> linear(1800) -> relu -> dropout -> bn -> linear(500) -> relu -> dropout -> linear(1) -> A/B pred
    3. \n
  3. The final prediction = softmax(A pred, B pred, 0)
    4. \n
\n

Training:

\n", "rawMarkdown": "My approach was quite straightforward for this competition I have used embeddings, extracted by BERT + some other features and fully connected NN to get the final predictions.\n\n**Embeddings:**\nI have used Bert large uncased (without any finetuning) and extracted embeddings from 9 layers – from -1 to -9. I have concatenated all of them and used as a 1024*9 = 9216-dimension vectors as final embeddings. Other sets of layers gave me worse results.\n\n**Additional features:**\n-\tPronoun offset\n-\tPronoun token offset\n-\tA and B tokens offsets\n-\tA and B lengths\n-\tDistance between a/b and pronoun (in symbols and tokens)\n-\tSome features from this kernel https://www.kaggle.com/pheell/look-ma-no-embeddings\n\n**Additional data:**\n-\tManually corrected labels from here https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/81331\n\n**Model:**\n1.\tAll other (non-bert) features were passed through the linear layer:\nOther_features (separate A+P and B+P)-> bn -> linear -> relu -> preprocessed other features\n2.\tConcatenated other and bert features were passed through the FCNN:\n(A/B BERT emb + preprocessed other features) -> linear(1800) -> relu -> dropout -> bn -> linear(500) -> relu -> dropout -> linear(1) -> A/B pred\n3.\tThe final prediction = softmax(A pred, B pred, 0)\n\n**Training:**\n-\tA have trained the model with Adam. Everything was highly regularized (~90% dropout, batch norm, weight decay ~6)\n-\tThe model was trained on all GAP data using 10-fold cv. The final prediction was just a blend of 10 predictions.", "votes": 18 }, { "id": 523380, "postDate": "2019-04-26T05:52:27.070Z", "content": "

Congratulations! Thanks for sharing the approach.

", "rawMarkdown": "Congratulations! Thanks for sharing the approach. " }, { "id": 522120, "postDate": "2019-04-23T22:04:55.200Z", "content": "

Congrats, Ilya! I'm humbled :)

", "rawMarkdown": "Congrats, Ilya! I'm humbled :)" }, { "id": 521733, "postDate": "2019-04-23T10:50:10.013Z", "content": "

Congratulations and thanks for sharing!

", "rawMarkdown": "Congratulations and thanks for sharing!", "votes": 1 }, { "id": 522049, "postDate": "2019-04-23T19:47:01.603Z", "content": "

Congratulations and thanks for sharing!

", "rawMarkdown": "Congratulations and thanks for sharing!" }, { "id": 521875, "postDate": "2019-04-23T15:10:31.457Z", "content": "

Thanks for sharing! Congratulations!

", "rawMarkdown": "Thanks for sharing! Congratulations!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:42:24Z", "competition_id": 12797, "competition_slug": "gendered-pronoun-resolution", "team_id": 2726981, "solution_url": "/competitions/gendered-pronoun-resolution/writeups/ilia-15th-place-solution-0-21074" }