Congratulations to winners and my team @taniaj, @dskswu, @arigion
\n\nWe didn't manage to fine-tune Bert, but still had a couple of ideas that worked pretty good.
\n\nUsed Bert-large with concatenated embeddings from -3, -4, -5, -6 layers. As @ceshine pointed out in his great kernel (https://www.kaggle.com/ceshine/pytorch-bert-endpointspanextractor-kfold), it's better not to use the last layer. After some experimentation, we settled on -3, -4, -5, and -6 layers based on CV scores.
Used corrected and original data (https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/81331#latest-503495). We couldn't decide which models to choose: the ones that were trained on original or corrected data. Until we tried to blend them and it worked great (0.34-0.35 for original/corrected and 0.33 after the blending for Public LB). Our 2 final submissions were only different in terms of weights in blending. We tried 0.6/0.4 and 0.5/0.5 for corrected/original data. Just like in Public LB, 0.6/0.4 split worked better (0.24838 vs 0.25200 Private LB)
Applied data augmentation inside each fold by simply swapping A and B columns and concatenating with original data.
10 folds, of course.
A simple NN (4 layers with BN, ReLU and dropout) on top of bert embeddings and linguistic features from public kernels. More complicated architectures didn't bring anything to the table.
It's not as sophisticated as @boliu0 solution (great job, btw. Really impressive), but I still hope you find it useful.
", "comments": [ { "id": 521701, "author_name": "Ceshine Lee", "author_url": "", "post_date": "2019-04-23T09:38:47.717000", "content": "Very neat solution! Thanks for sharing.
\n\nDid you extract the Bert embeddings deterministically and feed it the the NN along with linguistic features? I'm asking because my deterministic version seriously underperformed / overfit and I suspect there're some bug in my code.
", "votes": 1, "replies": [ { "id": 521711, "author_name": "Ivan Panshin", "author_url": "", "post_date": "2019-04-23T10:16:42.537000", "content": "You're welcome.
\n\nYes, exactly. It worked better than extracting bert embeddings on the run. Or maybe there was a bug in my non-deterministic version.
", "votes": 1, "replies": [] }, { "id": 521716, "author_name": "Ceshine Lee", "author_url": "", "post_date": "2019-04-23T10:27:59.440000", "content": "Okay. Thanks for the info. Looks like I have some debugging to do then.
", "votes": 1, "replies": [] } ] }, { "id": 521480, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-04-23T00:58:31.847000", "content": "Thanks, Ivan! Actually, very similar to our solution.
", "votes": 1, "replies": [ { "id": 521487, "author_name": "Ivan Panshin", "author_url": "", "post_date": "2019-04-23T01:04:48.297000", "content": "I look forward to reading your solution.
\n\nI was actually trying to catch up with your team this whole competition. 4 places and 0,02276 difference in stage 2 is good enough I guess :)
\n\nCongrats on getting another silver!
", "votes": 2, "replies": [] }, { "id": 521490, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-04-23T01:11:34.103000", "content": "Thanks! We'll come up with an overview probably tomorrow
", "votes": 1, "replies": [] } ] }, { "id": 521479, "author_name": "Ivan Panshin", "author_url": "", "post_date": "2019-04-23T00:57:38.877000", "content": "Btw, special thanks to @kashnitsky. You were really helpful during this competition. And also to @mateiionita and his great kernels that provided a good baseline.
", "votes": 1, "replies": [] }, { "id": 521758, "author_name": "Dimitrij Shulkin", "author_url": "", "post_date": "2019-04-23T11:36:13.987000", "content": "Great work! Congratulations!
", "votes": 2, "replies": [] }, { "id": 521477, "author_name": "SRK", "author_url": "", "post_date": "2019-04-23T00:54:53.110000", "content": "Thanks @ivanpan for sharing. It is indeed useful.
", "votes": 2, "replies": [] }, { "id": 523294, "author_name": "Akash Ravichandran", "author_url": "", "post_date": "2019-04-26T00:35:14.473000", "content": "Thank you, Ivan Panshin for sharing.
", "votes": 0, "replies": [] }, { "id": 521880, "author_name": "dhaqui the kaggler", "author_url": "", "post_date": "2019-04-23T15:12:12.737000", "content": "Thanks for sharing! Congratulations!
", "votes": 0, "replies": [] } ], "comment_count": 11, "messages": [ { "id": 521474, "postDate": "2019-04-23T00:51:52.680Z", "content": "Congratulations to winners and my team @taniaj, @dskswu, @arigion
\n\nWe didn't manage to fine-tune Bert, but still had a couple of ideas that worked pretty good.
\n\nUsed Bert-large with concatenated embeddings from -3, -4, -5, -6 layers. As @ceshine pointed out in his great kernel (https://www.kaggle.com/ceshine/pytorch-bert-endpointspanextractor-kfold), it's better not to use the last layer. After some experimentation, we settled on -3, -4, -5, and -6 layers based on CV scores.
Used corrected and original data (https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/81331#latest-503495). We couldn't decide which models to choose: the ones that were trained on original or corrected data. Until we tried to blend them and it worked great (0.34-0.35 for original/corrected and 0.33 after the blending for Public LB). Our 2 final submissions were only different in terms of weights in blending. We tried 0.6/0.4 and 0.5/0.5 for corrected/original data. Just like in Public LB, 0.6/0.4 split worked better (0.24838 vs 0.25200 Private LB)
Applied data augmentation inside each fold by simply swapping A and B columns and concatenating with original data.
10 folds, of course.
A simple NN (4 layers with BN, ReLU and dropout) on top of bert embeddings and linguistic features from public kernels. More complicated architectures didn't bring anything to the table.
It's not as sophisticated as @boliu0 solution (great job, btw. Really impressive), but I still hope you find it useful.
", "rawMarkdown": "Congratulations to winners and my team @taniaj, @dskswu, @arigion \n\nWe didn't manage to fine-tune Bert, but still had a couple of ideas that worked pretty good. \n\n1. Used Bert-large with concatenated embeddings from -3, -4, -5, -6 layers. As @ceshine pointed out in his great kernel (https://www.kaggle.com/ceshine/pytorch-bert-endpointspanextractor-kfold), it's better not to use the last layer. After some experimentation, we settled on -3, -4, -5, and -6 layers based on CV scores. \n\n2. Used corrected and original data (https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/81331#latest-503495). We couldn't decide which models to choose: the ones that were trained on original or corrected data. Until we tried to blend them and it worked great (0.34-0.35 for original/corrected and 0.33 after the blending for Public LB). Our 2 final submissions were only different in terms of weights in blending. We tried 0.6/0.4 and 0.5/0.5 for corrected/original data. Just like in Public LB, 0.6/0.4 split worked better (0.24838 vs 0.25200 Private LB)\n\n3. Applied data augmentation inside each fold by simply swapping A and B columns and concatenating with original data. \n\n4. 10 folds, of course. \n\n5. A simple NN (4 layers with BN, ReLU and dropout) on top of bert embeddings and linguistic features from public kernels. More complicated architectures didn't bring anything to the table. \n\nIt's not as sophisticated as @boliu0 solution (great job, btw. Really impressive), but I still hope you find it useful. ", "votes": 19 }, { "id": 521701, "postDate": "2019-04-23T09:38:47.717Z", "content": "Very neat solution! Thanks for sharing.
\n\nDid you extract the Bert embeddings deterministically and feed it the the NN along with linguistic features? I'm asking because my deterministic version seriously underperformed / overfit and I suspect there're some bug in my code.
", "rawMarkdown": "Very neat solution! Thanks for sharing. \n\nDid you extract the Bert embeddings deterministically and feed it the the NN along with linguistic features? I'm asking because my deterministic version seriously underperformed / overfit and I suspect there're some bug in my code.", "votes": 1, "replies": [ { "id": 521711, "postDate": "2019-04-23T10:16:42.537Z", "content": "You're welcome.
\n\nYes, exactly. It worked better than extracting bert embeddings on the run. Or maybe there was a bug in my non-deterministic version.
", "rawMarkdown": "You're welcome. \n\nYes, exactly. It worked better than extracting bert embeddings on the run. Or maybe there was a bug in my non-deterministic version. ", "votes": 1 }, { "id": 521716, "postDate": "2019-04-23T10:27:59.440Z", "content": "Okay. Thanks for the info. Looks like I have some debugging to do then.
", "rawMarkdown": "Okay. Thanks for the info. Looks like I have some debugging to do then.", "votes": 1 } ] }, { "id": 521480, "postDate": "2019-04-23T00:58:31.847Z", "content": "Thanks, Ivan! Actually, very similar to our solution.
", "rawMarkdown": "Thanks, Ivan! Actually, very similar to our solution.", "votes": 1, "replies": [ { "id": 521487, "postDate": "2019-04-23T01:04:48.297Z", "content": "I look forward to reading your solution.
\n\nI was actually trying to catch up with your team this whole competition. 4 places and 0,02276 difference in stage 2 is good enough I guess :)
\n\nCongrats on getting another silver!
", "rawMarkdown": "I look forward to reading your solution. \n\nI was actually trying to catch up with your team this whole competition. 4 places and 0,02276 difference in stage 2 is good enough I guess :) \n\nCongrats on getting another silver! ", "votes": 2 }, { "id": 521490, "postDate": "2019-04-23T01:11:34.103Z", "content": "Thanks! We'll come up with an overview probably tomorrow
", "rawMarkdown": "Thanks! We'll come up with an overview probably tomorrow", "votes": 1 } ] }, { "id": 521479, "postDate": "2019-04-23T00:57:38.877Z", "content": "Btw, special thanks to @kashnitsky. You were really helpful during this competition. And also to @mateiionita and his great kernels that provided a good baseline.
", "rawMarkdown": "Btw, special thanks to @kashnitsky. You were really helpful during this competition. And also to @mateiionita and his great kernels that provided a good baseline. ", "votes": 1 }, { "id": 521758, "postDate": "2019-04-23T11:36:13.987Z", "content": "Great work! Congratulations!
", "rawMarkdown": "Great work! Congratulations!", "votes": 2 }, { "id": 521477, "postDate": "2019-04-23T00:54:53.110Z", "content": "Thanks @ivanpan for sharing. It is indeed useful.
", "rawMarkdown": "Thanks @ivanpan for sharing. It is indeed useful. ", "votes": 2 }, { "id": 523294, "postDate": "2019-04-26T00:35:14.473Z", "content": "Thank you, Ivan Panshin for sharing.
", "rawMarkdown": "**Thank you, Ivan Panshin for sharing.**" }, { "id": 521880, "postDate": "2019-04-23T15:12:12.737Z", "content": "Thanks for sharing! Congratulations!
", "rawMarkdown": "Thanks for sharing! Congratulations!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:45:07Z", "competition_id": 12797, "competition_slug": "gendered-pronoun-resolution", "team_id": 2949554, "solution_url": "/competitions/gendered-pronoun-resolution/writeups/ricardo-ml-quick-overview-of-the-26th-place-soluti" }