{ "id": 90392, "title": "1st place solution summary", "content": "

Great competition and thanks to all the participants for keeping the competition alive!

\n\n

I was more active initially but got sidetracked by things at work and international travel during the later half.

\n\n

Since I'm traveling, I'll be able to post only a brief summary now but hopefully update it with more details later.

\n\n

Summary

\n\n

The focus of my work was on 1 architectural contribution and a lot of generalization analysis.

\n\n

Here are the key ingredients-

\n\n
    \n

  1. Data

    \n\n
    2. \n

  2. Architecture

    \n\n
    3. \n

  3. Unbiased model averaging

    \n\n\n\n

    The intractability of hyper parameter tuning for deep learning models and risk of underfitting/overfitting to the validation set (early stopping) make averaging more useful than usual. However, this effect can also be achieved through SWA by investing some research time but at a substantially reduced computational budget.

    4. \n
\n\n

Those are amongst the major contributors that I can think of right now!

\n\n

UPDATE

\n\n

The above description is for submission model 2.\nModel 1 has all the same elements other than the last bullet point in model averaging. Model 1 was trained only on the sanitized version of the labels, in case the organizers end up putting in some extra effort of cleaning the crowd sourced labels.

\n\n

Architecture details

\n\n

Based on the cluster predictions (clusters co-referent with A, B and P) from the coref models, a mask corresponding to each cluster is created from word-piece tokens and applied to BERT embeddings (from the last layer). This masked version of embeddings for each such cluster is then fed to the Evidence Gathering module which systematically aggregates the information and comes up with an evidence vector. The evidence vector is then fed to the decision making layer resulting in a probability distribution over A, B and P.

\n\n

The Evidence Gathering module sits on top of the BERT pre-trained models, but the BERT layers themselves are not frozen and get weakly adapted through error propagation, while, retaining most of the language characteristics necessary for generalization.

\n\n

UPDATE - 06/04/19

\n\n

arxiv paper link - Gendered Ambiguous Pronouns Shared Task: Boosting Model Confidence\nby Evidence Pooling\ngithub code link - https://github.com/sattree/gap

\n\n

\"Architecture\"

", "comments": [ { "id": 543958, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-06-05T01:03:39.593000", "content": "

@rohitmidha23 @hideodeo @sanikamal @kenkrige I've updated the post with links to the paper and the code. Hope you find them useful!

", "votes": 6, "replies": [ { "id": 544208, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-06-05T09:10:28.093000", "content": "

Brilliant. Thank you. I will go through your code in detail to learn as much as I can as soon as I get the chance to spend some time on it. I have also put my full code on https://github.com/kenkrige/BERT-Fine-tune-for-GAP.

", "votes": 0, "replies": [] }, { "id": 544246, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-06-05T09:54:49.757000", "content": "

@sattree This paper is great. You use some fascinating techniques and it is so well written and easy to read. Just from reading it, I feel like I could implement these techniques without even turning to your code, which is not always the case. For my money, the best ML paper I have ever read, but maybe I am a bit biased because of knowing the data set so well.

", "votes": 0, "replies": [] }, { "id": 544320, "author_name": "Sani Kamal", "author_url": "", "post_date": "2019-06-05T11:45:34.543000", "content": "

Thank you @sattree

", "votes": 0, "replies": [] }, { "id": 544427, "author_name": "Rakesh Chada", "author_url": "", "post_date": "2019-06-05T13:54:13.287000", "content": "

Hey @sattree . Thanks for sharing the paper! It's a fantastic read!

", "votes": 0, "replies": [] }, { "id": 544753, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-06-05T21:43:26.780000", "content": "

@kenkrige Thanks for sharing your code.

\n\n

You are correct, knowing the data helps understand the context. I was running out of page limit, so didn't expand much on it and left it to the references and appendix. I tried my best to put down the ideas through writeup and figures, but I'm sure there is enough room for improvement.

", "votes": 0, "replies": [] } ] }, { "id": 521848, "author_name": "Matei Ionita", "author_url": "", "post_date": "2019-04-23T14:30:26.910000", "content": "

Thanks for sharing your approach. I never appreciated how much useful information you can get from the pre-trained coref models. Your description and the kernel you link to are really eye-opening when it comes to unsupervised / distantly supervised coreference resolution.

", "votes": 1, "replies": [ { "id": 521948, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-23T17:07:21.147000", "content": "

@mateiionita glad you found it useful!

\n\n

Given the current status of (nlp) deep learning models, I prefer to leverage the existing architectures and build on top of them. There is sufficient evidence that these models capture the necessary linguistic structure and, thereby obviating the need to go through the painstaking process of creating hand-crafted domain specific features. Since, proref problem is a subset of the general coref problem, I found it reasonable to leverage the structure unravelled by them and it ended up working well for the problem.

", "votes": 2, "replies": [] } ] }, { "id": 521823, "author_name": "Tania J", "author_url": "", "post_date": "2019-04-23T13:36:36.317000", "content": "

Congratulations! I hope you will share more details about how you did data augmentation for neither instances. I tried this too but it didn't work for me.

", "votes": 2, "replies": [ { "id": 521945, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-23T16:59:45.190000", "content": "

@taniaj glad you asked!

\n\n

There are two aspects to this-

\n\n
    \n

  1. I generated neither instances by using the cluster predictions (albeit noisy) from Lee et al and looking for non-overlapping clusters of a pronoun and a mention. However, this approach was only meant to better calibrate the 'neither' probabilities and led to an overall improvement of 1%. Since, there aren't any architectural elements, it still doesn't help the model in understanding neither scenarios. A less laborious and smarter way to achieve the same effect, would be through instance weighting, although, it will require some investigation and runs the risk of overfitting on the validation set (tuning the weights as hyper parameters).

    2. \n

  2. The main contribution comes from evidence gathering module (described above) which allows the model to understand the linguistic structure leading to neither scenarios. The evidence vector encodes information about which mentions a given pronoun is co-referent with and vice-versa.

    3. \n
\n\n

Hope the explanation helps!

", "votes": 2, "replies": [] }, { "id": 521991, "author_name": "Tania J", "author_url": "", "post_date": "2019-04-23T18:23:36.317000", "content": "

Thank you! I am just beginning to dig into the details and digest your approach. I think it will take me some time to fully understand it (NLP newbie here). But let me just say, wow! Really well deserved win.

", "votes": 1, "replies": [] }, { "id": 522106, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-23T21:35:38.357000", "content": "

@taniaj thanks )

", "votes": 0, "replies": [] } ] }, { "id": 530377, "author_name": "Skinish", "author_url": "", "post_date": "2019-05-12T17:03:47.503000", "content": "

What was your approach for label smoothing?

", "votes": 0, "replies": [] }, { "id": 529410, "author_name": "Sani Kamal", "author_url": "", "post_date": "2019-05-09T20:25:14.987000", "content": "

Congratulations! I hope you will share more details about how you did data augmentation.

", "votes": 0, "replies": [] }, { "id": 526786, "author_name": "Rohit Midha", "author_url": "", "post_date": "2019-05-03T18:42:44.400000", "content": "

@sattree can you share the paper link or the code once you're done? \nCongrats on the win!

", "votes": 0, "replies": [] }, { "id": 524352, "author_name": "AoiMakino", "author_url": "", "post_date": "2019-04-28T15:04:20.557000", "content": "

Brilliant !! Congratulation!

", "votes": 0, "replies": [ { "id": 524813, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-29T15:12:23.607000", "content": "

@aoimakino thanks.

", "votes": 0, "replies": [] } ] }, { "id": 524056, "author_name": "Mystery", "author_url": "", "post_date": "2019-04-27T19:41:22.140000", "content": "

@sattree ,Amaze how your approach move from SM to more accurate one thing how much time factor per day, and for whole job?

", "votes": 0, "replies": [ { "id": 524816, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-29T15:15:32.967000", "content": "

@sajidhussainml sorry, I don't know how to answer your question.

", "votes": 0, "replies": [] } ] }, { "id": 523667, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-04-26T17:31:33.047000", "content": "

I just clicked on @sattree 's profile link. 2 competitions. 2 Gold Medals. \"Unranked\". Classic! I wonder if there has ever been such a good score in a kaggle competition in terms of ratio between first and second places.

", "votes": 0, "replies": [ { "id": 524140, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-28T03:38:46.097000", "content": "

@kenkrige not very active on kaggle )

", "votes": 0, "replies": [] }, { "id": 524167, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-04-28T05:56:33.040000", "content": "

Clearly, but so insightful, and I do think they should reconsider the ranking metric if 2 golds is not enough.

", "votes": 0, "replies": [] } ] }, { "id": 523379, "author_name": "Prashanth Thangavel", "author_url": "", "post_date": "2019-04-26T05:51:05.870000", "content": "

Congratulations! Thanks for sharing the approach.

", "votes": 0, "replies": [ { "id": 524812, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-29T15:12:07.950000", "content": "

@prashanththangavel thanks.

", "votes": 0, "replies": [] } ] }, { "id": 523372, "author_name": "CPMP", "author_url": "", "post_date": "2019-04-26T05:48:00.890000", "content": "

Amazing score, congrats!

", "votes": 0, "replies": [ { "id": 524811, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-29T15:11:53.577000", "content": "

@cpmpml thanks.

", "votes": 0, "replies": [] } ] }, { "id": 523193, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-04-25T17:58:32.947000", "content": "

Very cool. Thank you.

\n\n
\n

This masked version of embeddings for each such cluster is then fed to the Evidence Gathering module which systematically aggregates the information and comes up with an evidence vector.

\n
\n\n

When you have time, I would love to hear more about \"systematically aggregates\".

", "votes": 0, "replies": [ { "id": 523546, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-26T13:37:12.470000", "content": "

@kenkrige I'm in the process of writing the paper, will share the link soon.

\n\n

Meanwhile, here are some references-

\n\n

A Nil-Aware Answer Extraction Framework for Question Answering\nCoarse-grain Fine-grain Coattention Network for Multi-evidence Question Answering\nSentence Similarity Learning by Lexical Decomposition and Composition

", "votes": 1, "replies": [] }, { "id": 523656, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-04-26T17:19:25.597000", "content": "

Thank you, and no rush. I'm sure you are under pressure from various angles at the moment. I will start with those references and then very happy to wait for the paper and read that.

", "votes": 0, "replies": [] } ] }, { "id": 522847, "author_name": "Euihyeon Pyeon", "author_url": "", "post_date": "2019-04-25T06:02:41.050000", "content": "

Congratulation!!

", "votes": 0, "replies": [ { "id": 524810, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-29T15:11:38.200000", "content": "

@ehpyeon thanks.

", "votes": 0, "replies": [] } ] }, { "id": 522760, "author_name": "pinkkyone", "author_url": "", "post_date": "2019-04-25T00:56:34.590000", "content": "

Congratulations!

", "votes": 0, "replies": [ { "id": 524809, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-29T15:11:23.633000", "content": "

@pinkkyone thanks.

", "votes": 0, "replies": [] } ] }, { "id": 522236, "author_name": "Ceshine Lee", "author_url": "", "post_date": "2019-04-24T04:37:11.253000", "content": "

Congratulations. The idea of using distant supervision is really powerful. Well deserved.

", "votes": 0, "replies": [ { "id": 522721, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-24T22:15:04.927000", "content": "

@ceshine thanks.

", "votes": 0, "replies": [] } ] }, { "id": 522197, "author_name": "pheel", "author_url": "", "post_date": "2019-04-24T02:30:01.893000", "content": "

Brilliant! Congrats.

", "votes": 0, "replies": [ { "id": 522722, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-24T22:15:20.187000", "content": "

@pheell thanks,

", "votes": 0, "replies": [] } ] }, { "id": 522151, "author_name": "Amit Kumar Jaiswal", "author_url": "", "post_date": "2019-04-23T23:09:45.910000", "content": "

Congrats @sattree for detailing your approach.

", "votes": 0, "replies": [ { "id": 522723, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-24T22:15:31.933000", "content": "

@amitkumarjaiswal thanks.

", "votes": 0, "replies": [] } ] }, { "id": 522046, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-04-23T19:46:35.417000", "content": "

Congratulations and thanks for sharing!

", "votes": 0, "replies": [ { "id": 522109, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-23T21:36:34.963000", "content": "

@stanislavblinov thanks )

", "votes": 0, "replies": [] } ] }, { "id": 521960, "author_name": "KALE", "author_url": "", "post_date": "2019-04-23T17:27:16.793000", "content": "

Congratulations on your strong finish!

", "votes": 0, "replies": [ { "id": 522107, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-23T21:36:11.117000", "content": "

@cczaixian thanks )

", "votes": 0, "replies": [] } ] }, { "id": 521910, "author_name": "Akash Ravichandran", "author_url": "", "post_date": "2019-04-23T16:07:02.630000", "content": "

Congrats and thanks for sharing @sattree

", "votes": 0, "replies": [ { "id": 521936, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-23T16:47:57.753000", "content": "

@akashravichandran thanks )

", "votes": 1, "replies": [] } ] }, { "id": 521898, "author_name": "dhaqui the kaggler", "author_url": "", "post_date": "2019-04-23T15:41:37.677000", "content": "

Congratulations!

", "votes": 0, "replies": [ { "id": 521935, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-23T16:47:47.283000", "content": "

@dhaqui thanks )

", "votes": 0, "replies": [] } ] }, { "id": 521806, "author_name": "SRK", "author_url": "", "post_date": "2019-04-23T13:07:25.850000", "content": "

@sattree Thanks for sharing the details and congrats on the win !

", "votes": 0, "replies": [ { "id": 521934, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-23T16:47:28.363000", "content": "

@sudalairajkumar thanks )

", "votes": 0, "replies": [] } ] }, { "id": 710322, "author_name": "Alex Mourne", "author_url": "", "post_date": "2020-01-04T15:38:03.480000", "content": "

Thanks for sharing!

", "votes": 0, "replies": [] }, { "id": 582332, "author_name": "Vageesha Budanur", "author_url": "", "post_date": "2019-07-23T04:24:30.787000", "content": "

Thank you @sattree

", "votes": 0, "replies": [] }, { "id": 536958, "author_name": "Hideo Kobayashi", "author_url": "", "post_date": "2019-05-25T20:06:42.263000", "content": "

Thank you for sharing this kernel!

", "votes": 0, "replies": [] }, { "id": 525004, "author_name": "Hunter Knighton", "author_url": "", "post_date": "2019-04-30T01:30:44.740000", "content": "

Thanks for sharing!

", "votes": 0, "replies": [] } ], "comment_count": 53, "messages": [ { "id": 521800, "postDate": "2019-04-23T12:49:48.310Z", "content": "

Great competition and thanks to all the participants for keeping the competition alive!

\n\n

I was more active initially but got sidetracked by things at work and international travel during the later half.

\n\n

Since I'm traveling, I'll be able to post only a brief summary now but hopefully update it with more details later.

\n\n

Summary

\n\n

The focus of my work was on 1 architectural contribution and a lot of generalization analysis.

\n\n

Here are the key ingredients-

\n\n
    \n

  1. Data

    \n\n
    2. \n

  2. Architecture

    \n\n
    3. \n

  3. Unbiased model averaging

    \n\n\n\n

    The intractability of hyper parameter tuning for deep learning models and risk of underfitting/overfitting to the validation set (early stopping) make averaging more useful than usual. However, this effect can also be achieved through SWA by investing some research time but at a substantially reduced computational budget.

    4. \n
\n\n

Those are amongst the major contributors that I can think of right now!

\n\n

UPDATE

\n\n

The above description is for submission model 2.\nModel 1 has all the same elements other than the last bullet point in model averaging. Model 1 was trained only on the sanitized version of the labels, in case the organizers end up putting in some extra effort of cleaning the crowd sourced labels.

\n\n

Architecture details

\n\n

Based on the cluster predictions (clusters co-referent with A, B and P) from the coref models, a mask corresponding to each cluster is created from word-piece tokens and applied to BERT embeddings (from the last layer). This masked version of embeddings for each such cluster is then fed to the Evidence Gathering module which systematically aggregates the information and comes up with an evidence vector. The evidence vector is then fed to the decision making layer resulting in a probability distribution over A, B and P.

\n\n

The Evidence Gathering module sits on top of the BERT pre-trained models, but the BERT layers themselves are not frozen and get weakly adapted through error propagation, while, retaining most of the language characteristics necessary for generalization.

\n\n

UPDATE - 06/04/19

\n\n

arxiv paper link - Gendered Ambiguous Pronouns Shared Task: Boosting Model Confidence\nby Evidence Pooling\ngithub code link - https://github.com/sattree/gap

\n\n

\"Architecture\"

", "rawMarkdown": "Great competition and thanks to all the participants for keeping the competition alive!\n\nI was more active initially but got sidetracked by things at work and international travel during the later half.\n\nSince I'm traveling, I'll be able to post only a brief summary now but hopefully update it with more details later.\n\n#Summary\n\nThe focus of my work was on 1 architectural contribution and a lot of generalization analysis.\n\nHere are the key ingredients-\n\n1. Data\n\n - The automated pipeline is illustrated below.\n ![Data pipeline](https://i.imgur.com/4nBm6AH.png)\n - Data augmentation for neither instances - the under-represented category.\n\n\n2. Architecture\n\n - Bert finetuning - no hyper parameter tuning was done.\n - **Evidence gathering by Distant Supervision** - various attention-pooling, self-attention, co-attention and evidence decomposition layers to incorporate noisy information from heuristic and pre-trained coref models. Lee et al is a very powerful coref model and grossly undermined by the GAP paper. Here's [https://www.kaggle.com/sattree/3-a-better-baseline](https://www.kaggle.com/sattree/3-a-better-baseline) a kernel that compares various coref models. It established a better 'unsupervised' baseline than the official bert baseline of .62. Unfortunately, I couldn't make it work on kaggle kernels and by the time I gave up (and subsequently got sidetracked) a better bert baseline of .53 had been established.\n - Label smoothing - to handle noisy labels.\n\n\n3. Unbiased model averaging\n\n - Averaging across folds\n - Averaging across 5 seeds\n - Averaging across LMs (cased and uncased) - cased models are more representative and help identify named entities but at the same time rare entities often have sparse distributional properties. On the other hand, uncased models are quite noisy.\n - Averaging Sanitized and Unsanitized label models - since the labels are crowd sourced, it is reasonable to assume that there will be some pattern to the errors humans make and will translate to unseen stage 2 data as well (a common challenge in language datasets). At the same time, some of those errors will be just due to the noise in the behavior and enthusiasm of the turkers.\n\n The intractability of hyper parameter tuning for deep learning models and risk of underfitting/overfitting to the validation set (early stopping) make averaging more useful than usual. However, this effect can also be achieved through SWA by investing some research time but at a substantially reduced computational budget.\n\nThose are amongst the major contributors that I can think of right now!\n\n##UPDATE\n\nThe above description is for submission model 2.\nModel 1 has all the same elements other than the last bullet point in model averaging. Model 1 was trained only on the sanitized version of the labels, in case the organizers end up putting in some extra effort of cleaning the crowd sourced labels.\n\n###Architecture details\n\nBased on the cluster predictions (clusters co-referent with A, B and P) from the coref models, a mask corresponding to each cluster is created from word-piece tokens and applied to BERT embeddings (from the last layer). This masked version of embeddings for each such cluster is then fed to the *Evidence Gathering* module which systematically aggregates the information and comes up with an *evidence vector*. The evidence vector is then fed to the decision making layer resulting in a probability distribution over A, B and P. \n\nThe *Evidence Gathering* module sits on top of the BERT pre-trained models, but the BERT layers themselves are not frozen and get weakly adapted through error propagation, while, retaining most of the language characteristics necessary for generalization. \n\n##UPDATE - 06/04/19\n\narxiv paper link - [Gendered Ambiguous Pronouns Shared Task: Boosting Model Confidence\nby Evidence Pooling](https://arxiv.org/pdf/1906.00839.pdf)\ngithub code link - https://github.com/sattree/gap\n\n![Architecture](https://i.imgur.com/DPEGIA6.png)", "votes": 70 }, { "id": 543958, "postDate": "2019-06-05T01:03:39.593Z", "content": "

@rohitmidha23 @hideodeo @sanikamal @kenkrige I've updated the post with links to the paper and the code. Hope you find them useful!

", "rawMarkdown": "@rohitmidha23 @hideodeo @sanikamal @kenkrige I've updated the post with links to the paper and the code. Hope you find them useful!", "votes": 6, "replies": [ { "id": 544208, "postDate": "2019-06-05T09:10:28.093Z", "content": "

Brilliant. Thank you. I will go through your code in detail to learn as much as I can as soon as I get the chance to spend some time on it. I have also put my full code on https://github.com/kenkrige/BERT-Fine-tune-for-GAP.

", "rawMarkdown": "Brilliant. Thank you. I will go through your code in detail to learn as much as I can as soon as I get the chance to spend some time on it. I have also put my full code on https://github.com/kenkrige/BERT-Fine-tune-for-GAP." }, { "id": 544246, "postDate": "2019-06-05T09:54:49.757Z", "content": "

@sattree This paper is great. You use some fascinating techniques and it is so well written and easy to read. Just from reading it, I feel like I could implement these techniques without even turning to your code, which is not always the case. For my money, the best ML paper I have ever read, but maybe I am a bit biased because of knowing the data set so well.

", "rawMarkdown": "@sattree This paper is great. You use some fascinating techniques and it is so well written and easy to read. Just from reading it, I feel like I could implement these techniques without even turning to your code, which is not always the case. For my money, the best ML paper I have ever read, but maybe I am a bit biased because of knowing the data set so well." }, { "id": 544320, "postDate": "2019-06-05T11:45:34.543Z", "content": "

Thank you @sattree

", "rawMarkdown": "Thank you @sattree " }, { "id": 544427, "postDate": "2019-06-05T13:54:13.287Z", "content": "

Hey @sattree . Thanks for sharing the paper! It's a fantastic read!

", "rawMarkdown": "Hey @sattree . Thanks for sharing the paper! It's a fantastic read!" }, { "id": 544753, "postDate": "2019-06-05T21:43:26.780Z", "content": "

@kenkrige Thanks for sharing your code.

\n\n

You are correct, knowing the data helps understand the context. I was running out of page limit, so didn't expand much on it and left it to the references and appendix. I tried my best to put down the ideas through writeup and figures, but I'm sure there is enough room for improvement.

", "rawMarkdown": "@kenkrige Thanks for sharing your code.\n\nYou are correct, knowing the data helps understand the context. I was running out of page limit, so didn't expand much on it and left it to the references and appendix. I tried my best to put down the ideas through writeup and figures, but I'm sure there is enough room for improvement." } ] }, { "id": 521848, "postDate": "2019-04-23T14:30:26.910Z", "content": "

Thanks for sharing your approach. I never appreciated how much useful information you can get from the pre-trained coref models. Your description and the kernel you link to are really eye-opening when it comes to unsupervised / distantly supervised coreference resolution.

", "rawMarkdown": "Thanks for sharing your approach. I never appreciated how much useful information you can get from the pre-trained coref models. Your description and the kernel you link to are really eye-opening when it comes to unsupervised / distantly supervised coreference resolution.", "votes": 1, "replies": [ { "id": 521948, "postDate": "2019-04-23T17:07:21.147Z", "content": "

@mateiionita glad you found it useful!

\n\n

Given the current status of (nlp) deep learning models, I prefer to leverage the existing architectures and build on top of them. There is sufficient evidence that these models capture the necessary linguistic structure and, thereby obviating the need to go through the painstaking process of creating hand-crafted domain specific features. Since, proref problem is a subset of the general coref problem, I found it reasonable to leverage the structure unravelled by them and it ended up working well for the problem.

", "rawMarkdown": "@mateiionita glad you found it useful!\n\nGiven the current status of (nlp) deep learning models, I prefer to leverage the existing architectures and build on top of them. There is sufficient evidence that these models capture the necessary linguistic structure and, thereby obviating the need to go through the painstaking process of creating hand-crafted domain specific features. Since, proref problem is a subset of the general coref problem, I found it reasonable to leverage the structure unravelled by them and it ended up working well for the problem.", "votes": 2 } ] }, { "id": 521823, "postDate": "2019-04-23T13:36:36.317Z", "content": "

Congratulations! I hope you will share more details about how you did data augmentation for neither instances. I tried this too but it didn't work for me.

", "rawMarkdown": "Congratulations! I hope you will share more details about how you did data augmentation for neither instances. I tried this too but it didn't work for me.", "votes": 2, "replies": [ { "id": 521945, "postDate": "2019-04-23T16:59:45.190Z", "content": "

@taniaj glad you asked!

\n\n

There are two aspects to this-

\n\n
    \n

  1. I generated neither instances by using the cluster predictions (albeit noisy) from Lee et al and looking for non-overlapping clusters of a pronoun and a mention. However, this approach was only meant to better calibrate the 'neither' probabilities and led to an overall improvement of 1%. Since, there aren't any architectural elements, it still doesn't help the model in understanding neither scenarios. A less laborious and smarter way to achieve the same effect, would be through instance weighting, although, it will require some investigation and runs the risk of overfitting on the validation set (tuning the weights as hyper parameters).

    2. \n

  2. The main contribution comes from evidence gathering module (described above) which allows the model to understand the linguistic structure leading to neither scenarios. The evidence vector encodes information about which mentions a given pronoun is co-referent with and vice-versa.

    3. \n
\n\n

Hope the explanation helps!

", "rawMarkdown": "@taniaj glad you asked!\n\nThere are two aspects to this-\n\n1. I generated neither instances by using the cluster predictions (albeit noisy) from Lee et al and looking for non-overlapping clusters of a pronoun and a mention. However, this approach was only meant to better calibrate the 'neither' probabilities and led to an overall improvement of 1%. Since, there aren't any architectural elements, it still doesn't help the model in understanding neither scenarios. A less laborious and smarter way to achieve the same effect, would be through *instance weighting*, although, it will require some investigation and runs the risk of overfitting on the validation set (tuning the weights as hyper parameters).\n\n2. The main contribution comes from *evidence gathering* module (described above) which allows the model to understand the linguistic structure leading to neither scenarios. The evidence vector encodes information about which mentions a given pronoun is co-referent with and vice-versa.\n\nHope the explanation helps!", "votes": 2 }, { "id": 521991, "postDate": "2019-04-23T18:23:36.317Z", "content": "

Thank you! I am just beginning to dig into the details and digest your approach. I think it will take me some time to fully understand it (NLP newbie here). But let me just say, wow! Really well deserved win.

", "rawMarkdown": "Thank you! I am just beginning to dig into the details and digest your approach. I think it will take me some time to fully understand it (NLP newbie here). But let me just say, wow! Really well deserved win.", "votes": 1 }, { "id": 522106, "postDate": "2019-04-23T21:35:38.357Z", "content": "

@taniaj thanks )

", "rawMarkdown": "@taniaj thanks )" } ] }, { "id": 530377, "postDate": "2019-05-12T17:03:47.503Z", "content": "

What was your approach for label smoothing?

", "rawMarkdown": "What was your approach for label smoothing?" }, { "id": 529410, "postDate": "2019-05-09T20:25:14.987Z", "content": "

Congratulations! I hope you will share more details about how you did data augmentation.

", "rawMarkdown": "Congratulations! I hope you will share more details about how you did data augmentation." }, { "id": 526786, "postDate": "2019-05-03T18:42:44.400Z", "content": "

@sattree can you share the paper link or the code once you're done? \nCongrats on the win!

", "rawMarkdown": "@sattree can you share the paper link or the code once you're done? \nCongrats on the win! " }, { "id": 524352, "postDate": "2019-04-28T15:04:20.557Z", "content": "

Brilliant !! Congratulation!

", "rawMarkdown": "Brilliant !! Congratulation!", "replies": [ { "id": 524813, "postDate": "2019-04-29T15:12:23.607Z", "content": "

@aoimakino thanks.

", "rawMarkdown": "@aoimakino thanks." } ] }, { "id": 524056, "postDate": "2019-04-27T19:41:22.140Z", "content": "

@sattree ,Amaze how your approach move from SM to more accurate one thing how much time factor per day, and for whole job?

", "rawMarkdown": "@sattree ,Amaze how your approach move from SM to more accurate one thing how much time factor per day, and for whole job?", "replies": [ { "id": 524816, "postDate": "2019-04-29T15:15:32.967Z", "content": "

@sajidhussainml sorry, I don't know how to answer your question.

", "rawMarkdown": "@sajidhussainml sorry, I don't know how to answer your question." } ] }, { "id": 523667, "postDate": "2019-04-26T17:31:33.047Z", "content": "

I just clicked on @sattree 's profile link. 2 competitions. 2 Gold Medals. \"Unranked\". Classic! I wonder if there has ever been such a good score in a kaggle competition in terms of ratio between first and second places.

", "rawMarkdown": "I just clicked on @sattree 's profile link. 2 competitions. 2 Gold Medals. \"Unranked\". Classic! I wonder if there has ever been such a good score in a kaggle competition in terms of ratio between first and second places.", "replies": [ { "id": 524140, "postDate": "2019-04-28T03:38:46.097Z", "content": "

@kenkrige not very active on kaggle )

", "rawMarkdown": "@kenkrige not very active on kaggle )" }, { "id": 524167, "postDate": "2019-04-28T05:56:33.040Z", "content": "

Clearly, but so insightful, and I do think they should reconsider the ranking metric if 2 golds is not enough.

", "rawMarkdown": "Clearly, but so insightful, and I do think they should reconsider the ranking metric if 2 golds is not enough." } ] }, { "id": 523379, "postDate": "2019-04-26T05:51:05.870Z", "content": "

Congratulations! Thanks for sharing the approach.

", "rawMarkdown": "Congratulations! Thanks for sharing the approach. ", "replies": [ { "id": 524812, "postDate": "2019-04-29T15:12:07.950Z", "content": "

@prashanththangavel thanks.

", "rawMarkdown": "@prashanththangavel thanks." } ] }, { "id": 523372, "postDate": "2019-04-26T05:48:00.890Z", "content": "

Amazing score, congrats!

", "rawMarkdown": "Amazing score, congrats!", "replies": [ { "id": 524811, "postDate": "2019-04-29T15:11:53.577Z", "content": "

@cpmpml thanks.

", "rawMarkdown": "@cpmpml thanks." } ] }, { "id": 523193, "postDate": "2019-04-25T17:58:32.947Z", "content": "

Very cool. Thank you.

\n\n
\n

This masked version of embeddings for each such cluster is then fed to the Evidence Gathering module which systematically aggregates the information and comes up with an evidence vector.

\n
\n\n

When you have time, I would love to hear more about \"systematically aggregates\".

", "rawMarkdown": "Very cool. Thank you.\n\n> This masked version of embeddings for each such cluster is then fed to the Evidence Gathering module which systematically aggregates the information and comes up with an evidence vector. \n\nWhen you have time, I would love to hear more about \"systematically aggregates\".", "replies": [ { "id": 523546, "postDate": "2019-04-26T13:37:12.470Z", "content": "

@kenkrige I'm in the process of writing the paper, will share the link soon.

\n\n

Meanwhile, here are some references-

\n\n

A Nil-Aware Answer Extraction Framework for Question Answering\nCoarse-grain Fine-grain Coattention Network for Multi-evidence Question Answering\nSentence Similarity Learning by Lexical Decomposition and Composition

", "rawMarkdown": "@kenkrige I'm in the process of writing the paper, will share the link soon.\n\nMeanwhile, here are some references-\n\n[A Nil-Aware Answer Extraction Framework for Question Answering](http://aclweb.org/anthology/D18-1456)\n[Coarse-grain Fine-grain Coattention Network for Multi-evidence Question Answering](https://arxiv.org/abs/1901.00603)\n[Sentence Similarity Learning by Lexical Decomposition and Composition](https://arxiv.org/pdf/1602.07019.pdf)", "votes": 1 }, { "id": 523656, "postDate": "2019-04-26T17:19:25.597Z", "content": "

Thank you, and no rush. I'm sure you are under pressure from various angles at the moment. I will start with those references and then very happy to wait for the paper and read that.

", "rawMarkdown": "Thank you, and no rush. I'm sure you are under pressure from various angles at the moment. I will start with those references and then very happy to wait for the paper and read that." } ] }, { "id": 522847, "postDate": "2019-04-25T06:02:41.050Z", "content": "

Congratulation!!

", "rawMarkdown": "Congratulation!!", "replies": [ { "id": 524810, "postDate": "2019-04-29T15:11:38.200Z", "content": "

@ehpyeon thanks.

", "rawMarkdown": "@ehpyeon thanks." } ] }, { "id": 522760, "postDate": "2019-04-25T00:56:34.590Z", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "replies": [ { "id": 524809, "postDate": "2019-04-29T15:11:23.633Z", "content": "

@pinkkyone thanks.

", "rawMarkdown": "@pinkkyone thanks." } ] }, { "id": 522236, "postDate": "2019-04-24T04:37:11.253Z", "content": "

Congratulations. The idea of using distant supervision is really powerful. Well deserved.

", "rawMarkdown": "Congratulations. The idea of using distant supervision is really powerful. Well deserved.", "replies": [ { "id": 522721, "postDate": "2019-04-24T22:15:04.927Z", "content": "

@ceshine thanks.

", "rawMarkdown": "@ceshine thanks." } ] }, { "id": 522197, "postDate": "2019-04-24T02:30:01.893Z", "content": "

Brilliant! Congrats.

", "rawMarkdown": "Brilliant! Congrats.", "replies": [ { "id": 522722, "postDate": "2019-04-24T22:15:20.187Z", "content": "

@pheell thanks,

", "rawMarkdown": "@pheell thanks," } ] }, { "id": 522151, "postDate": "2019-04-23T23:09:45.910Z", "content": "

Congrats @sattree for detailing your approach.

", "rawMarkdown": "Congrats @sattree for detailing your approach.", "replies": [ { "id": 522723, "postDate": "2019-04-24T22:15:31.933Z", "content": "

@amitkumarjaiswal thanks.

", "rawMarkdown": "@amitkumarjaiswal thanks." } ] }, { "id": 522046, "postDate": "2019-04-23T19:46:35.417Z", "content": "

Congratulations and thanks for sharing!

", "rawMarkdown": "Congratulations and thanks for sharing!", "replies": [ { "id": 522109, "postDate": "2019-04-23T21:36:34.963Z", "content": "

@stanislavblinov thanks )

", "rawMarkdown": "@stanislavblinov thanks )" } ] }, { "id": 521960, "postDate": "2019-04-23T17:27:16.793Z", "content": "

Congratulations on your strong finish!

", "rawMarkdown": "Congratulations on your strong finish!", "replies": [ { "id": 522107, "postDate": "2019-04-23T21:36:11.117Z", "content": "

@cczaixian thanks )

", "rawMarkdown": "@cczaixian thanks )" } ] }, { "id": 521910, "postDate": "2019-04-23T16:07:02.630Z", "content": "

Congrats and thanks for sharing @sattree

", "rawMarkdown": "**Congrats and thanks for sharing @sattree** ", "replies": [ { "id": 521936, "postDate": "2019-04-23T16:47:57.753Z", "content": "

@akashravichandran thanks )

", "rawMarkdown": "@akashravichandran thanks )", "votes": 1 } ] }, { "id": 521898, "postDate": "2019-04-23T15:41:37.677Z", "content": "

Congratulations!

", "rawMarkdown": "Congratulations! ", "replies": [ { "id": 521935, "postDate": "2019-04-23T16:47:47.283Z", "content": "

@dhaqui thanks )

", "rawMarkdown": "@dhaqui thanks )" } ] }, { "id": 521806, "postDate": "2019-04-23T13:07:25.850Z", "content": "

@sattree Thanks for sharing the details and congrats on the win !

", "rawMarkdown": "@sattree Thanks for sharing the details and congrats on the win ! ", "replies": [ { "id": 521934, "postDate": "2019-04-23T16:47:28.363Z", "content": "

@sudalairajkumar thanks )

", "rawMarkdown": "@sudalairajkumar thanks )" } ] }, { "id": 710322, "postDate": "2020-01-04T15:38:03.480Z", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!" }, { "id": 582332, "postDate": "2019-07-23T04:24:30.787Z", "content": "

Thank you @sattree

", "rawMarkdown": "Thank you @sattree " }, { "id": 536958, "postDate": "2019-05-25T20:06:42.263Z", "content": "

Thank you for sharing this kernel!

", "rawMarkdown": "Thank you for sharing this kernel!" }, { "id": 525004, "postDate": "2019-04-30T01:30:44.740Z", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:43:10Z", "competition_id": 12797, "competition_slug": "gendered-pronoun-resolution", "team_id": 2742137, "solution_url": "/competitions/gendered-pronoun-resolution/writeups/sandeep-attree-1st-place-solution-summary" }