![\"Model\"](\"https://i.imgur.com/WGJ9OPK.png\")
Congratulation to all the winners and thanks to Google AI for hosting the competition. I have learned a lot through the awesome Kaggle community.
\n\nHere is a brief overview of my solution. Since I joined this competition at a late stage, my solution does not involve BERT fine-tuning or data augmentation. What I have done was based on three observations.
\n\nI've played with BERT in other tasks where I found there are some redundancies in BERT vector. Even though we only use a small portion (like 50%) of the BERT vector, we still can get desirable performance.
\n\nBased on this observation, I placed a dropout with a large rate just after the input layer, which can be considered as a kind of model boosting, just like training several prototypes with subsets that are randomly sampled from the BERT vector.
\n\nAs I mentioned in section 1, it might not be suitable to use the output directly because of redundancies. Therefore I use a word encoder to down-project the BERT vector into a lower-dimensional space where I can extract task-related features efficiently.
\n\nThe word encoder is a simple affine transformation with SELU activation and it is shared for A, B, and P. I have tried to design the word encoder for names and pronouns independently or make the word encoder deeper with highway transformations but all of them results in overfitting.
\n\nThis idea is also inspired by the multi-head transformation. I have implemented a multi-head NLI encoder but it only improved the performance by ~0.0005 and took much computation time. So maybe a single head is good enough for this task.
\n\nI consider this task a sub-task of answer selection. Given queries A, B, and an answer P, we can model the relations between queries and answers with heuristic interaction:
\n\n\nI(Q, A) = [[Q; A], Q - A, Q * A]\n\nand then extract features from the interaction vector I(Q, A) with a siamese encoder. The overall architecture would be like this:
Finally, here is a simple performacne report of my models:
\n\n| Model | 5 fold CV on Stage 1|\n| -------- | -------- |\n| Base BERT | 0.50 |\n| Base BERT + input dropout | 0.45 |\n| Base BERT + input dropout + NLI | 0.43 |\n| Base BERT + all | 0.39 |\n| Large BERT + input dropout | 0.39 |\n| Large BERT + all | 0.32 |\n| Ensemble of Base BERT and Large BERT | 0.30 |
\n\nI will make my whole repository public after cleaning up some messy things. Since I am occupied by certain exams currently, it might be updated a week later.
\n\nCongratulations!
", "votes": 0, "replies": [] }, { "id": 540226, "author_name": "Jiahao Huang", "author_url": "", "post_date": "2019-05-31T07:12:34.870000", "content": "Congratulations! I hope to learn some details from your paper. Can you share your paper with me?
", "votes": 0, "replies": [] }, { "id": 528086, "author_name": "Xinyu", "author_url": "", "post_date": "2019-05-07T04:11:35.370000", "content": "Congratulations! I also did Input Dropout, similar to yours. So I assume the improvement mainly comes from NLI architectures?
", "votes": 0, "replies": [ { "id": 528161, "author_name": "Justin Yang", "author_url": "", "post_date": "2019-05-07T07:05:39.357000", "content": "Thanks. I think so, because the common practice based on concatenation was too shallow to propagate the information across inputs efficiently.
", "votes": 0, "replies": [] } ] }, { "id": 524769, "author_name": "Tania J", "author_url": "", "post_date": "2019-04-29T13:41:00.627000", "content": "Congratulations on the gold! I am looking forward to learning from your full code.
", "votes": 0, "replies": [ { "id": 525034, "author_name": "Justin Yang", "author_url": "", "post_date": "2019-04-30T03:52:57.750000", "content": "Thanks. I will update my code this Saturday.
", "votes": 1, "replies": [] } ] }, { "id": 522332, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-04-24T08:52:07.520000", "content": "Thanks for your summary! Nice to see that you can climb so far without additional data or augmentations.
", "votes": 0, "replies": [ { "id": 528174, "author_name": "Justin Yang", "author_url": "", "post_date": "2019-05-07T07:30:50.703000", "content": "I also would like to thank you for the selfless contributions. They are really impressive.
", "votes": 1, "replies": [] } ] } ], "comment_count": 8, "messages": [ { "id": 522303, "postDate": "2019-04-24T07:39:37.913Z", "content": "Congratulation to all the winners and thanks to Google AI for hosting the competition. I have learned a lot through the awesome Kaggle community.
\n\nHere is a brief overview of my solution. Since I joined this competition at a late stage, my solution does not involve BERT fine-tuning or data augmentation. What I have done was based on three observations.
\n\nI've played with BERT in other tasks where I found there are some redundancies in BERT vector. Even though we only use a small portion (like 50%) of the BERT vector, we still can get desirable performance.
\n\nBased on this observation, I placed a dropout with a large rate just after the input layer, which can be considered as a kind of model boosting, just like training several prototypes with subsets that are randomly sampled from the BERT vector.
\n\nAs I mentioned in section 1, it might not be suitable to use the output directly because of redundancies. Therefore I use a word encoder to down-project the BERT vector into a lower-dimensional space where I can extract task-related features efficiently.
\n\nThe word encoder is a simple affine transformation with SELU activation and it is shared for A, B, and P. I have tried to design the word encoder for names and pronouns independently or make the word encoder deeper with highway transformations but all of them results in overfitting.
\n\nThis idea is also inspired by the multi-head transformation. I have implemented a multi-head NLI encoder but it only improved the performance by ~0.0005 and took much computation time. So maybe a single head is good enough for this task.
\n\nI consider this task a sub-task of answer selection. Given queries A, B, and an answer P, we can model the relations between queries and answers with heuristic interaction:
\n\n\nI(Q, A) = [[Q; A], Q - A, Q * A]\n\nand then extract features from the interaction vector I(Q, A) with a siamese encoder. The overall architecture would be like this:
Finally, here is a simple performacne report of my models:
\n\n| Model | 5 fold CV on Stage 1|\n| -------- | -------- |\n| Base BERT | 0.50 |\n| Base BERT + input dropout | 0.45 |\n| Base BERT + input dropout + NLI | 0.43 |\n| Base BERT + all | 0.39 |\n| Large BERT + input dropout | 0.39 |\n| Large BERT + all | 0.32 |\n| Ensemble of Base BERT and Large BERT | 0.30 |
\n\nI will make my whole repository public after cleaning up some messy things. Since I am occupied by certain exams currently, it might be updated a week later.
\n\nCongratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 540226, "postDate": "2019-05-31T07:12:34.870Z", "content": "Congratulations! I hope to learn some details from your paper. Can you share your paper with me?
", "rawMarkdown": "Congratulations! I hope to learn some details from your paper. Can you share your paper with me? " }, { "id": 528086, "postDate": "2019-05-07T04:11:35.370Z", "content": "Congratulations! I also did Input Dropout, similar to yours. So I assume the improvement mainly comes from NLI architectures?
", "rawMarkdown": "Congratulations! I also did Input Dropout, similar to yours. So I assume the improvement mainly comes from NLI architectures?", "replies": [ { "id": 528161, "postDate": "2019-05-07T07:05:39.357Z", "content": "Thanks. I think so, because the common practice based on concatenation was too shallow to propagate the information across inputs efficiently.
", "rawMarkdown": "Thanks. I think so, because the common practice based on concatenation was too shallow to propagate the information across inputs efficiently." } ] }, { "id": 524769, "postDate": "2019-04-29T13:41:00.627Z", "content": "Congratulations on the gold! I am looking forward to learning from your full code.
", "rawMarkdown": "Congratulations on the gold! I am looking forward to learning from your full code.", "replies": [ { "id": 525034, "postDate": "2019-04-30T03:52:57.750Z", "content": "Thanks. I will update my code this Saturday.
", "rawMarkdown": "Thanks. I will update my code this Saturday.", "votes": 1 } ] }, { "id": 522332, "postDate": "2019-04-24T08:52:07.520Z", "content": "Thanks for your summary! Nice to see that you can climb so far without additional data or augmentations.
", "rawMarkdown": "Thanks for your summary! Nice to see that you can climb so far without additional data or augmentations. ", "replies": [ { "id": 528174, "postDate": "2019-05-07T07:30:50.703Z", "content": "I also would like to thank you for the selfless contributions. They are really impressive.
", "rawMarkdown": "I also would like to thank you for the selfless contributions. They are really impressive.", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:42:54Z", "competition_id": 12797, "competition_slug": "gendered-pronoun-resolution", "team_id": 2737214, "solution_url": "/competitions/gendered-pronoun-resolution/writeups/justin-yang-4th-place-solution-overview" }