{
  "id": 90334,
  "title": "7th place solution",
  "content": "<h2>Scores</h2>\n\n<p>Stage 1 LB: <strong>0.2929</strong> (trained on 2000 Test + 454 Validation, evaluated on 2000 Development)</p>\n\n<p>Stage 1 5 Fold Out-of-fold: <strong>0.2922</strong> (train/evaluated on 2000 Test + 2000 Development + 400 Validation)</p>\n\n<p>Stage 2:  <strong>0.19473</strong> (trained on the same 4400 examples as above, I left the other 54 validation data for sanity check)</p>\n\n<h2>Model</h2>\n\n<p>I have a main model (referred to as “end-2-end” model below) and a secondary model (referred to as “pure bert” model below).</p>\n\n<p>Both models use Bert features based on Matei’s <a href=\"https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline\">strong baseline</a> (Thanks Matei!) Pure bert model has the same architecture as in Matei’s kernel; end-2-end model uses the top level architecture in <a href=\"https://arxiv.org/abs/1707.07045\">https://arxiv.org/abs/1707.07045</a>, as implemented by Chanhu’s <a href=\"https://www.kaggle.com/chanhu/bert-score-layer-lb-0-475\">kernel</a> I also added 10 linguistic features from pheel’s <a href=\"https://www.kaggle.com/pheell/look-ma-no-embeddings\">kernel</a> to the end-2-end model</p>\n\n<p>For pure bert model, I concatenated layer -3 and -4; for end-2-end model, I only used layer -4 embeddings. I used Bert Large (both cased and uncased) with max token length 256 for both models.</p>\n\n<p>I didn’t do any fine-tuning. </p>\n\n<h2>Augmentation</h2>\n\n<p>Bert embeddings of the names (A and B) contain information about both the context and the name itself. For coreference resolution task, it doesn’t matter whether a name is Alice or Betty or Carol or Debby. Therefore, I augmented the input data by replacing the A and B names by 4 sets of placeholder names:\n<code>\n[ { 'female':['Alice','Kate'], 'male': ['John','Michael']},\n  { 'female':['Elizabeth','Mary'], 'male': ['James','Henry']},\n  { 'female':['Kate','Elizabeth'], 'male': ['Michael','James']},\n  { 'female':['Mary','Alice'], 'male': ['Henry','John']}]\n</code></p>\n\n<p>I chose these names by looking at the most common names in training data while making sure not to choose similar sounding names. If you pay attention to popular name trends in the U.S., you will notice all these placeholder names are old school names that are not as popular anymore (see e.g. <a href=\"https://www.behindthename.com/name/mary/top/united-states\">https://www.behindthename.com/name/mary/top/united-states</a> and\n<a href=\"https://www.behindthename.com/name/john/top/united-states\">https://www.behindthename.com/name/john/top/united-states</a>). This is because in the training data, there are a large amount of articles about historical people.</p>\n\n<p>I initially chose these 4 sets of names. Later I tried fewer, more, and different (more modern) names but didn’t find improvements.</p>\n\n<p>For each input data row, I augmented it into 4 variations by replacing A by Alice, B by Kate if female, A by John B by Michael if male, etc. The bert embeddings are extracted 4 more times for these 4 augmented data. Then each training epoch will see each input data 5 times (original and 4 augmentations). This way, the embedding information about the particular names will be averaged out. Only the information about the sentence structure will be left. Or to put it in a different way, let’s say if 200 dimensions out of the 1024 Bert embeddings are about the name itself and the rest 824 dimensions are about the context, then the model will learn to rely on only those 824 dimension to resolve coreference and ignore the other 200 dimensions.</p>\n\n<p>Another potential benefit of replacing all names by these one-word short names is that, if a name is long (first name + last name), it will be tokenized into many word pieces and make it harder for Bert to embed the necessary information into these token. This is only my guess, to be further tested. But I do have an evidence supporting this: in testing time, I did TTA (test time augmentation) the same way as in training. The order of the 4 augmentation’s scores varies depending on the model, but they all outperformed the un-augmented version, always. See the “original” row in the results table below.</p>\n\n<p>When I did the name replacement, I replaced all the occurrences of the name in the document by placeholder names. In order to avoid confusion, I didn’t do the replacement in the following situations:\n1.  If the placeholder name (i.e. Alice or Kate) already appear in original document\n2. If A or B is full name (first + last name), but the first name or last name appear alone elsewhere in the document. For example, if A is “Michael Jordan” but “Jordan” is used to referred to the person later in the doc, if I replace all the “Michael Jordan” in the doc by “Henry” then Bert would think “Jordan” and “Henry” are different people\n3. If the name has more than two words, such as “Elizabeth Frances Zane” or “Jose de Venecia Jr”, I don’t replace it because it would be difficult to implement the rule above</p>\n\n<h2>Incorrect ground truth</h2>\n\n<p>As pointed in the Discussion, there are clearly wrong labels. I fixed 74 of them in development, and 85 in test and validation. (I did this before wayward’s <a href=\"https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/81331#503094\">post</a>. She reported similar number of corrections, so I assumed it would be similar without comparing hers to mine.)</p>\n\n<p>One important thing is that, the model tends to make very confident predictions when trained with “clean” labeled, i.e. the output probabilities are very close to 0 or 1 for some rows. When scored with “dirty” labels, however, correct prediction of very small probability will lead to very large logloss. Therefore, I clipped all the output probabilities by a tuned threshold, usually 0.005 or 0.006</p>\n\n<h2>Other changes</h2>\n\n<p>I did the following changes to the Bert features extraction par of Matei’s kernel. Without them, the extracted token embeddings are misaligned in some cases.\n1. There are two documents (209 in dev and 921 in test) too long to be correctly embedded using max sequence length 256. I wrote a function to throw away first few sentences in such scenarios to make all embeddings valid. But in stage 2 data, there is no such case.\n2. In Matei’s kernel, it always skip the first 2 tokens. I changed it to skipping two tokens only if the second token is <code>\"</code>, because sometimes (I think when there are quotations in document) actual tokens start at the index 1 instead of 2\n3. I added\n <code>if text=='#': return 1</code>\nin functions <code>count_chars_no_special</code> and <code>count_length_no_special</code></p>\n\n<h2>Validation strategy and ensemble</h2>\n\n<p>Final model is <code>0.9 * end2end + 0.1 * pure bert</code></p>\n\n<p>Below is the breakdown of the model’s OOF scores, ensemble components and weights.I tuned all the weights using clean labels, but reported the scores with dirty (i.e. actual) labels. Each column is a separate model. The top 5 rows in any column are the evaluation results on the 5 TTA variations from the same model.</p>\n\n<p><img src=\"https://imgur.com/oXnZrd6.png\" alt=\"results\"></p>\n\n<p>The model is trained on 4400 data in 5 folds. In each fold there are 3520 train data and 880 OOF validation data. Each 3520 training data is further divided into 5 folds, with each fold having 2816 training set and 704 early-stop validation set. As mentioned, the 2816 data is augmented into 14080 data per epoch. </p>\n\n<h2>Code</h2>\n\n<p>Code is available at: <a href=\"https://github.com/boliu61/gendered-pronoun-resolution\">https://github.com/boliu61/gendered-pronoun-resolution</a></p>\n\n<h2>Update on 4/30/2019</h2>\n\n<p>After reading other solutions, I realized layer -5 and -6 are the best. For some reason I thought -1 to -4 were the only available layers to extract features from. I just trained my models again, with everything else unchanged except changing Pure Bert model from layer -3 and -4 to -5 and -6; End2end model from layer -4 to -5.</p>\n\n<p>Stage 1 5fold CV score improved from <strong>0.2922</strong> to <strong>0.2846</strong>\nStage 2 Late submission score improved from <strong>0.19473</strong> to <strong>0.18075</strong> (only one submission, no LB probing)</p>\n\n<p>I think this shows the augmentation approach can be as good as fine-tuning.</p>\n\n<h2>Update on 5/6/2019</h2>\n\n<p>paper is available at <a href=\"https://arxiv.org/abs/1905.01780\">https://arxiv.org/abs/1905.01780</a></p>",
  "comments": [
    {
      "id": 525433,
      "author_name": "Bo",
      "author_url": "",
      "post_date": "2019-05-01T01:20:20.367000",
      "content": "<p><strong>Update on 4/30/2019</strong></p>\n\n<p>After reading other solutions, I realized layer -5 and -6 are the best. For some reason I thought -1 to -4 were the only available layers to extract features from. I just trained my models again, with everything else unchanged except changing Pure Bert model from layer -3 and -4 to -5 and -6; End2end model from layer -4 to -5.</p>\n\n<p>Stage 1 5fold CV score improved from <strong>0.2922</strong> to <strong>0.2846</strong>\nStage 2 Late submission score improved from <strong>0.19473</strong> to <strong>0.18075</strong> (only one submission, no LB probing)</p>\n\n<p>I think this shows the augmentation approach can be as good as fine-tuning.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 525538,
          "author_name": "Yury Kashnitsky",
          "author_url": "",
          "post_date": "2019-05-01T08:36:58.573000",
          "content": "<p>Thanks, this is impressive. We also found -4,-5, and -6 layers working best for us.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 522242,
      "author_name": "Ceshine Lee",
      "author_url": "",
      "post_date": "2019-04-24T04:50:13.170000",
      "content": "<p>Thanks for this well-written solution and the code released in such a short time. Congratulations.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521840,
      "author_name": "Lorenzo Ampil",
      "author_url": "",
      "post_date": "2019-04-23T14:15:09.980000",
      "content": "<p>Thank you for sharing and congratulations!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521738,
      "author_name": "Pedro Finkler",
      "author_url": "",
      "post_date": "2019-04-23T10:54:41.820000",
      "content": "<p>Congrats!! :D</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521736,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-04-23T10:52:28.020000",
      "content": "<p>Indeed a good solution. Congratulations and thanks for sharing!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521704,
      "author_name": "Don Alex",
      "author_url": "",
      "post_date": "2019-04-23T09:47:00.290000",
      "content": "<p>Congrats</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521637,
      "author_name": "user602655",
      "author_url": "",
      "post_date": "2019-04-23T07:15:03.497000",
      "content": "<p>Congrats! </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521626,
      "author_name": "Tania J",
      "author_url": "",
      "post_date": "2019-04-23T06:53:29.127000",
      "content": "<p>Congrats Bo and thanks for sharing your code too, much appreciated!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521582,
      "author_name": "Mekaveli.",
      "author_url": "",
      "post_date": "2019-04-23T04:19:17.907000",
      "content": "<p>Congratulations! What's the hardware/cloud specs you use to tame the BERT? I couldn't do BERT so I gave up early :P</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521505,
      "author_name": "Akash Ravichandran",
      "author_url": "",
      "post_date": "2019-04-23T01:30:49.953000",
      "content": "<p><strong>Congrats Bo and thanks for sharing the detailed overview.</strong></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521503,
      "author_name": "TerenceLiu",
      "author_url": "",
      "post_date": "2019-04-23T01:25:32.003000",
      "content": "<p>Congratulations Bo for both the gold medal and becoming Grandmaster!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521499,
      "author_name": "huiqin",
      "author_url": "",
      "post_date": "2019-04-23T01:21:38.933000",
      "content": "<p>Very good solution!  Thanks for sharing!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521489,
      "author_name": "Seung-jaeBang",
      "author_url": "",
      "post_date": "2019-04-23T01:10:07.400000",
      "content": "<p>Many congrats Bo, this is amazing!!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521478,
      "author_name": "Ilham Firdausi Putra",
      "author_url": "",
      "post_date": "2019-04-23T00:56:02.070000",
      "content": "<p>Thank you. That's amazing, achieving 7th without fine-tuning.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521463,
      "author_name": "Chanhu",
      "author_url": "",
      "post_date": "2019-04-23T00:39:15.950000",
      "content": "<p>Thank you for sharing.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 521467,
          "author_name": "Bo",
          "author_url": "",
          "post_date": "2019-04-23T00:41:49.500000",
          "content": "<p>Thanks Chanhu. And thanks a lot for your kernel!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 521580,
          "author_name": "Chanhu",
          "author_url": "",
          "post_date": "2019-04-23T04:16:09.387000",
          "content": "<p>It’s My Pleasure. I am really impressed by you solution.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 521462,
      "author_name": "SRK",
      "author_url": "",
      "post_date": "2019-04-23T00:34:10.080000",
      "content": "<p>Thank you Bo for sharing the solution. Congrats on the top 10 finish ! </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 528048,
      "author_name": "Bo",
      "author_url": "",
      "post_date": "2019-05-07T01:04:23.407000",
      "content": "<p>paper is available at <a href=\"https://arxiv.org/abs/1905.01780\">https://arxiv.org/abs/1905.01780</a></p>",
      "votes": 2,
      "replies": [
        {
          "id": 528178,
          "author_name": "Yury Kashnitsky",
          "author_url": "",
          "post_date": "2019-05-07T07:45:13.530000",
          "content": "<p>Thanks! We also submitted a paper on this ACL workshop. Wondered if we are allowed to submit it on Arxiv as well. Typically it depends on the particular conference rules. Did you submit the same paper on the workshop as well? Or Arxiv only?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 528323,
          "author_name": "Bo",
          "author_url": "",
          "post_date": "2019-05-07T13:40:29.650000",
          "content": "<p>yeah I submitted to the workshop as well</p>\n\n<p>regular papers which are anonymized for double blind review are prohibited for preprint release, but since this shared-task description paper is not anonymized (we pretty much described what we did in the Discussion here anyway), I think it's okay. See: <a href=\"http://www.acl2019.org/EN/call-for-papers.xhtml\">http://www.acl2019.org/EN/call-for-papers.xhtml</a></p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 521466,
      "author_name": "Matei Ionita",
      "author_url": "",
      "post_date": "2019-04-23T00:41:23.693000",
      "content": "<p>I'm really impressed by how well you did without fine-tuning. Congratulations!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 521471,
          "author_name": "Bo",
          "author_url": "",
          "post_date": "2019-04-23T00:48:58.603000",
          "content": "<p>Thank you Matei. Your kernel was amazing! It gave us such a strong baseline in this competition. </p>\n\n<p>About fine-tuning, I tried, but couldn't figure out a way to make it work in this (i.e. your original) set up. Without public LB in stage1, it was hard to gauge how I place, so I went all-in on augmentation and ensemble without fine-tuning.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 521564,
          "author_name": "Matei Ionita",
          "author_url": "",
          "post_date": "2019-04-23T03:40:42.453000",
          "content": "<p>I'm glad my kernel helped. And thanks to you, we know that BERT out of the box can come pretty close to fine-tuned BERT on this task.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 521465,
      "author_name": "cab",
      "author_url": "",
      "post_date": "2019-04-23T00:40:45.290000",
      "content": "<p>Thank you and Grand Master tier is comming!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 530380,
      "author_name": "Skinish",
      "author_url": "",
      "post_date": "2019-05-12T17:07:16.517000",
      "content": "<p>How did you refactor the wrong labels? Did you do them manually?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 521469,
      "author_name": "Yury Kashnitsky",
      "author_url": "",
      "post_date": "2019-04-23T00:44:16.693000",
      "content": "<p>Thanks for this overview. For me tuning threshold for clipping will be a takeaway.</p>\n\n<p>We used augmentation generating triples A-B-Pronoun where B is changed to some other name if A is a correct reference. It produced some really confident predictions, but often failed. Planned to use these predictions in stacking, but for some reason didn't succeed with stacking at all :)</p>\n\n<p>Indeed, very strong solution without fine-tuning. </p>",
      "votes": 0,
      "replies": [
        {
          "id": 521475,
          "author_name": "Bo",
          "author_url": "",
          "post_date": "2019-04-23T00:52:19.687000",
          "content": "<p>Thanks Yury for all the discussion posts during the competition. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 522047,
      "author_name": "Stanislav Blinov",
      "author_url": "",
      "post_date": "2019-04-23T19:46:47.733000",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 521879,
      "author_name": "dhaqui the kaggler",
      "author_url": "",
      "post_date": "2019-04-23T15:11:37.570000",
      "content": "<p>Thanks for sharing! Congratulations!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 525633,
      "author_name": "JAEHYEONG AN",
      "author_url": "",
      "post_date": "2019-05-01T12:32:55.510000",
      "content": "<p>good thanks</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 32,
  "messages": [
    {
      "id": 521459,
      "postDate": "2019-04-23T00:27:36.137Z",
      "content": "<h2>Scores</h2>\n\n<p>Stage 1 LB: <strong>0.2929</strong> (trained on 2000 Test + 454 Validation, evaluated on 2000 Development)</p>\n\n<p>Stage 1 5 Fold Out-of-fold: <strong>0.2922</strong> (train/evaluated on 2000 Test + 2000 Development + 400 Validation)</p>\n\n<p>Stage 2:  <strong>0.19473</strong> (trained on the same 4400 examples as above, I left the other 54 validation data for sanity check)</p>\n\n<h2>Model</h2>\n\n<p>I have a main model (referred to as “end-2-end” model below) and a secondary model (referred to as “pure bert” model below).</p>\n\n<p>Both models use Bert features based on Matei’s <a href=\"https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline\">strong baseline</a> (Thanks Matei!) Pure bert model has the same architecture as in Matei’s kernel; end-2-end model uses the top level architecture in <a href=\"https://arxiv.org/abs/1707.07045\">https://arxiv.org/abs/1707.07045</a>, as implemented by Chanhu’s <a href=\"https://www.kaggle.com/chanhu/bert-score-layer-lb-0-475\">kernel</a> I also added 10 linguistic features from pheel’s <a href=\"https://www.kaggle.com/pheell/look-ma-no-embeddings\">kernel</a> to the end-2-end model</p>\n\n<p>For pure bert model, I concatenated layer -3 and -4; for end-2-end model, I only used layer -4 embeddings. I used Bert Large (both cased and uncased) with max token length 256 for both models.</p>\n\n<p>I didn’t do any fine-tuning. </p>\n\n<h2>Augmentation</h2>\n\n<p>Bert embeddings of the names (A and B) contain information about both the context and the name itself. For coreference resolution task, it doesn’t matter whether a name is Alice or Betty or Carol or Debby. Therefore, I augmented the input data by replacing the A and B names by 4 sets of placeholder names:\n<code>\n[ { 'female':['Alice','Kate'], 'male': ['John','Michael']},\n  { 'female':['Elizabeth','Mary'], 'male': ['James','Henry']},\n  { 'female':['Kate','Elizabeth'], 'male': ['Michael','James']},\n  { 'female':['Mary','Alice'], 'male': ['Henry','John']}]\n</code></p>\n\n<p>I chose these names by looking at the most common names in training data while making sure not to choose similar sounding names. If you pay attention to popular name trends in the U.S., you will notice all these placeholder names are old school names that are not as popular anymore (see e.g. <a href=\"https://www.behindthename.com/name/mary/top/united-states\">https://www.behindthename.com/name/mary/top/united-states</a> and\n<a href=\"https://www.behindthename.com/name/john/top/united-states\">https://www.behindthename.com/name/john/top/united-states</a>). This is because in the training data, there are a large amount of articles about historical people.</p>\n\n<p>I initially chose these 4 sets of names. Later I tried fewer, more, and different (more modern) names but didn’t find improvements.</p>\n\n<p>For each input data row, I augmented it into 4 variations by replacing A by Alice, B by Kate if female, A by John B by Michael if male, etc. The bert embeddings are extracted 4 more times for these 4 augmented data. Then each training epoch will see each input data 5 times (original and 4 augmentations). This way, the embedding information about the particular names will be averaged out. Only the information about the sentence structure will be left. Or to put it in a different way, let’s say if 200 dimensions out of the 1024 Bert embeddings are about the name itself and the rest 824 dimensions are about the context, then the model will learn to rely on only those 824 dimension to resolve coreference and ignore the other 200 dimensions.</p>\n\n<p>Another potential benefit of replacing all names by these one-word short names is that, if a name is long (first name + last name), it will be tokenized into many word pieces and make it harder for Bert to embed the necessary information into these token. This is only my guess, to be further tested. But I do have an evidence supporting this: in testing time, I did TTA (test time augmentation) the same way as in training. The order of the 4 augmentation’s scores varies depending on the model, but they all outperformed the un-augmented version, always. See the “original” row in the results table below.</p>\n\n<p>When I did the name replacement, I replaced all the occurrences of the name in the document by placeholder names. In order to avoid confusion, I didn’t do the replacement in the following situations:\n1.  If the placeholder name (i.e. Alice or Kate) already appear in original document\n2. If A or B is full name (first + last name), but the first name or last name appear alone elsewhere in the document. For example, if A is “Michael Jordan” but “Jordan” is used to referred to the person later in the doc, if I replace all the “Michael Jordan” in the doc by “Henry” then Bert would think “Jordan” and “Henry” are different people\n3. If the name has more than two words, such as “Elizabeth Frances Zane” or “Jose de Venecia Jr”, I don’t replace it because it would be difficult to implement the rule above</p>\n\n<h2>Incorrect ground truth</h2>\n\n<p>As pointed in the Discussion, there are clearly wrong labels. I fixed 74 of them in development, and 85 in test and validation. (I did this before wayward’s <a href=\"https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/81331#503094\">post</a>. She reported similar number of corrections, so I assumed it would be similar without comparing hers to mine.)</p>\n\n<p>One important thing is that, the model tends to make very confident predictions when trained with “clean” labeled, i.e. the output probabilities are very close to 0 or 1 for some rows. When scored with “dirty” labels, however, correct prediction of very small probability will lead to very large logloss. Therefore, I clipped all the output probabilities by a tuned threshold, usually 0.005 or 0.006</p>\n\n<h2>Other changes</h2>\n\n<p>I did the following changes to the Bert features extraction par of Matei’s kernel. Without them, the extracted token embeddings are misaligned in some cases.\n1. There are two documents (209 in dev and 921 in test) too long to be correctly embedded using max sequence length 256. I wrote a function to throw away first few sentences in such scenarios to make all embeddings valid. But in stage 2 data, there is no such case.\n2. In Matei’s kernel, it always skip the first 2 tokens. I changed it to skipping two tokens only if the second token is <code>\"</code>, because sometimes (I think when there are quotations in document) actual tokens start at the index 1 instead of 2\n3. I added\n <code>if text=='#': return 1</code>\nin functions <code>count_chars_no_special</code> and <code>count_length_no_special</code></p>\n\n<h2>Validation strategy and ensemble</h2>\n\n<p>Final model is <code>0.9 * end2end + 0.1 * pure bert</code></p>\n\n<p>Below is the breakdown of the model’s OOF scores, ensemble components and weights.I tuned all the weights using clean labels, but reported the scores with dirty (i.e. actual) labels. Each column is a separate model. The top 5 rows in any column are the evaluation results on the 5 TTA variations from the same model.</p>\n\n<p><img src=\"https://imgur.com/oXnZrd6.png\" alt=\"results\"></p>\n\n<p>The model is trained on 4400 data in 5 folds. In each fold there are 3520 train data and 880 OOF validation data. Each 3520 training data is further divided into 5 folds, with each fold having 2816 training set and 704 early-stop validation set. As mentioned, the 2816 data is augmented into 14080 data per epoch. </p>\n\n<h2>Code</h2>\n\n<p>Code is available at: <a href=\"https://github.com/boliu61/gendered-pronoun-resolution\">https://github.com/boliu61/gendered-pronoun-resolution</a></p>\n\n<h2>Update on 4/30/2019</h2>\n\n<p>After reading other solutions, I realized layer -5 and -6 are the best. For some reason I thought -1 to -4 were the only available layers to extract features from. I just trained my models again, with everything else unchanged except changing Pure Bert model from layer -3 and -4 to -5 and -6; End2end model from layer -4 to -5.</p>\n\n<p>Stage 1 5fold CV score improved from <strong>0.2922</strong> to <strong>0.2846</strong>\nStage 2 Late submission score improved from <strong>0.19473</strong> to <strong>0.18075</strong> (only one submission, no LB probing)</p>\n\n<p>I think this shows the augmentation approach can be as good as fine-tuning.</p>\n\n<h2>Update on 5/6/2019</h2>\n\n<p>paper is available at <a href=\"https://arxiv.org/abs/1905.01780\">https://arxiv.org/abs/1905.01780</a></p>",
      "rawMarkdown": "## Scores\n\nStage 1 LB: **0.2929** (trained on 2000 Test + 454 Validation, evaluated on 2000 Development)\n\nStage 1 5 Fold Out-of-fold: **0.2922** (train/evaluated on 2000 Test + 2000 Development + 400 Validation)\n\nStage 2:  **0.19473** (trained on the same 4400 examples as above, I left the other 54 validation data for sanity check)\n\n## Model\nI have a main model (referred to as “end-2-end” model below) and a secondary model (referred to as “pure bert” model below).\n\nBoth models use Bert features based on Matei’s [strong baseline](https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline) (Thanks Matei!) Pure bert model has the same architecture as in Matei’s kernel; end-2-end model uses the top level architecture in https://arxiv.org/abs/1707.07045, as implemented by Chanhu’s [kernel](https://www.kaggle.com/chanhu/bert-score-layer-lb-0-475) I also added 10 linguistic features from pheel’s [kernel](https://www.kaggle.com/pheell/look-ma-no-embeddings) to the end-2-end model\n\nFor pure bert model, I concatenated layer -3 and -4; for end-2-end model, I only used layer -4 embeddings. I used Bert Large (both cased and uncased) with max token length 256 for both models.\n\nI didn’t do any fine-tuning. \n\n## Augmentation\nBert embeddings of the names (A and B) contain information about both the context and the name itself. For coreference resolution task, it doesn’t matter whether a name is Alice or Betty or Carol or Debby. Therefore, I augmented the input data by replacing the A and B names by 4 sets of placeholder names:\n```\n[ { 'female':['Alice','Kate'], 'male': ['John','Michael']},\n  { 'female':['Elizabeth','Mary'], 'male': ['James','Henry']},\n  { 'female':['Kate','Elizabeth'], 'male': ['Michael','James']},\n  { 'female':['Mary','Alice'], 'male': ['Henry','John']}]\n```\n\nI chose these names by looking at the most common names in training data while making sure not to choose similar sounding names. If you pay attention to popular name trends in the U.S., you will notice all these placeholder names are old school names that are not as popular anymore (see e.g. https://www.behindthename.com/name/mary/top/united-states and\nhttps://www.behindthename.com/name/john/top/united-states). This is because in the training data, there are a large amount of articles about historical people.\n\nI initially chose these 4 sets of names. Later I tried fewer, more, and different (more modern) names but didn’t find improvements.\n\nFor each input data row, I augmented it into 4 variations by replacing A by Alice, B by Kate if female, A by John B by Michael if male, etc. The bert embeddings are extracted 4 more times for these 4 augmented data. Then each training epoch will see each input data 5 times (original and 4 augmentations). This way, the embedding information about the particular names will be averaged out. Only the information about the sentence structure will be left. Or to put it in a different way, let’s say if 200 dimensions out of the 1024 Bert embeddings are about the name itself and the rest 824 dimensions are about the context, then the model will learn to rely on only those 824 dimension to resolve coreference and ignore the other 200 dimensions.\n\nAnother potential benefit of replacing all names by these one-word short names is that, if a name is long (first name + last name), it will be tokenized into many word pieces and make it harder for Bert to embed the necessary information into these token. This is only my guess, to be further tested. But I do have an evidence supporting this: in testing time, I did TTA (test time augmentation) the same way as in training. The order of the 4 augmentation’s scores varies depending on the model, but they all outperformed the un-augmented version, always. See the “original” row in the results table below.\n\nWhen I did the name replacement, I replaced all the occurrences of the name in the document by placeholder names. In order to avoid confusion, I didn’t do the replacement in the following situations:\n1.  If the placeholder name (i.e. Alice or Kate) already appear in original document\n2. If A or B is full name (first + last name), but the first name or last name appear alone elsewhere in the document. For example, if A is “Michael Jordan” but “Jordan” is used to referred to the person later in the doc, if I replace all the “Michael Jordan” in the doc by “Henry” then Bert would think “Jordan” and “Henry” are different people\n3. If the name has more than two words, such as “Elizabeth Frances Zane” or “Jose de Venecia Jr”, I don’t replace it because it would be difficult to implement the rule above\n\n## Incorrect ground truth\nAs pointed in the Discussion, there are clearly wrong labels. I fixed 74 of them in development, and 85 in test and validation. (I did this before wayward’s [post](https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/81331#503094). She reported similar number of corrections, so I assumed it would be similar without comparing hers to mine.)\n\nOne important thing is that, the model tends to make very confident predictions when trained with “clean” labeled, i.e. the output probabilities are very close to 0 or 1 for some rows. When scored with “dirty” labels, however, correct prediction of very small probability will lead to very large logloss. Therefore, I clipped all the output probabilities by a tuned threshold, usually 0.005 or 0.006\n\n## Other changes\nI did the following changes to the Bert features extraction par of Matei’s kernel. Without them, the extracted token embeddings are misaligned in some cases.\n1. There are two documents (209 in dev and 921 in test) too long to be correctly embedded using max sequence length 256. I wrote a function to throw away first few sentences in such scenarios to make all embeddings valid. But in stage 2 data, there is no such case.\n2. In Matei’s kernel, it always skip the first 2 tokens. I changed it to skipping two tokens only if the second token is `\"`, because sometimes (I think when there are quotations in document) actual tokens start at the index 1 instead of 2\n3. I added\n `if text=='#': return 1`\nin functions `count_chars_no_special` and `count_length_no_special`\n\n\n## Validation strategy and ensemble\nFinal model is `0.9 * end2end + 0.1 * pure bert`\n\nBelow is the breakdown of the model’s OOF scores, ensemble components and weights.I tuned all the weights using clean labels, but reported the scores with dirty (i.e. actual) labels. Each column is a separate model. The top 5 rows in any column are the evaluation results on the 5 TTA variations from the same model.\n\n![results](https://imgur.com/oXnZrd6.png)\n\n\nThe model is trained on 4400 data in 5 folds. In each fold there are 3520 train data and 880 OOF validation data. Each 3520 training data is further divided into 5 folds, with each fold having 2816 training set and 704 early-stop validation set. As mentioned, the 2816 data is augmented into 14080 data per epoch. \n\n\n##Code\nCode is available at: https://github.com/boliu61/gendered-pronoun-resolution\n\n##Update on 4/30/2019\nAfter reading other solutions, I realized layer -5 and -6 are the best. For some reason I thought -1 to -4 were the only available layers to extract features from. I just trained my models again, with everything else unchanged except changing Pure Bert model from layer -3 and -4 to -5 and -6; End2end model from layer -4 to -5.\n\nStage 1 5fold CV score improved from **0.2922** to **0.2846**\nStage 2 Late submission score improved from **0.19473** to **0.18075** (only one submission, no LB probing)\n\nI think this shows the augmentation approach can be as good as fine-tuning.\n\n##Update on 5/6/2019\npaper is available at https://arxiv.org/abs/1905.01780",
      "votes": 54
    },
    {
      "id": 525433,
      "postDate": "2019-05-01T01:20:20.367Z",
      "content": "<p><strong>Update on 4/30/2019</strong></p>\n\n<p>After reading other solutions, I realized layer -5 and -6 are the best. For some reason I thought -1 to -4 were the only available layers to extract features from. I just trained my models again, with everything else unchanged except changing Pure Bert model from layer -3 and -4 to -5 and -6; End2end model from layer -4 to -5.</p>\n\n<p>Stage 1 5fold CV score improved from <strong>0.2922</strong> to <strong>0.2846</strong>\nStage 2 Late submission score improved from <strong>0.19473</strong> to <strong>0.18075</strong> (only one submission, no LB probing)</p>\n\n<p>I think this shows the augmentation approach can be as good as fine-tuning.</p>",
      "rawMarkdown": "**Update on 4/30/2019**\n\nAfter reading other solutions, I realized layer -5 and -6 are the best. For some reason I thought -1 to -4 were the only available layers to extract features from. I just trained my models again, with everything else unchanged except changing Pure Bert model from layer -3 and -4 to -5 and -6; End2end model from layer -4 to -5.\n\nStage 1 5fold CV score improved from **0.2922** to **0.2846**\nStage 2 Late submission score improved from **0.19473** to **0.18075** (only one submission, no LB probing)\n\nI think this shows the augmentation approach can be as good as fine-tuning.",
      "votes": 3,
      "replies": [
        {
          "id": 525538,
          "postDate": "2019-05-01T08:36:58.573Z",
          "content": "<p>Thanks, this is impressive. We also found -4,-5, and -6 layers working best for us.</p>",
          "rawMarkdown": "Thanks, this is impressive. We also found -4,-5, and -6 layers working best for us."
        }
      ]
    },
    {
      "id": 522242,
      "postDate": "2019-04-24T04:50:13.170Z",
      "content": "<p>Thanks for this well-written solution and the code released in such a short time. Congratulations.</p>",
      "rawMarkdown": "Thanks for this well-written solution and the code released in such a short time. Congratulations.",
      "votes": 1
    },
    {
      "id": 521840,
      "postDate": "2019-04-23T14:15:09.980Z",
      "content": "<p>Thank you for sharing and congratulations!</p>",
      "rawMarkdown": "Thank you for sharing and congratulations!",
      "votes": 1
    },
    {
      "id": 521738,
      "postDate": "2019-04-23T10:54:41.820Z",
      "content": "<p>Congrats!! :D</p>",
      "rawMarkdown": "Congrats!! :D",
      "votes": 1
    },
    {
      "id": 521736,
      "postDate": "2019-04-23T10:52:28.020Z",
      "content": "<p>Indeed a good solution. Congratulations and thanks for sharing!</p>",
      "rawMarkdown": "Indeed a good solution. Congratulations and thanks for sharing!",
      "votes": 1
    },
    {
      "id": 521704,
      "postDate": "2019-04-23T09:47:00.290Z",
      "content": "<p>Congrats</p>",
      "rawMarkdown": "Congrats",
      "votes": 1
    },
    {
      "id": 521637,
      "postDate": "2019-04-23T07:15:03.497Z",
      "content": "<p>Congrats! </p>",
      "rawMarkdown": " Congrats! ",
      "votes": 1
    },
    {
      "id": 521626,
      "postDate": "2019-04-23T06:53:29.127Z",
      "content": "<p>Congrats Bo and thanks for sharing your code too, much appreciated!</p>",
      "rawMarkdown": "Congrats Bo and thanks for sharing your code too, much appreciated!",
      "votes": 1
    },
    {
      "id": 521582,
      "postDate": "2019-04-23T04:19:17.907Z",
      "content": "<p>Congratulations! What's the hardware/cloud specs you use to tame the BERT? I couldn't do BERT so I gave up early :P</p>",
      "rawMarkdown": "Congratulations! What's the hardware/cloud specs you use to tame the BERT? I couldn't do BERT so I gave up early :P",
      "votes": 1
    },
    {
      "id": 521505,
      "postDate": "2019-04-23T01:30:49.953Z",
      "content": "<p><strong>Congrats Bo and thanks for sharing the detailed overview.</strong></p>",
      "rawMarkdown": "**Congrats Bo and thanks for sharing the detailed overview.**",
      "votes": 1
    },
    {
      "id": 521503,
      "postDate": "2019-04-23T01:25:32.003Z",
      "content": "<p>Congratulations Bo for both the gold medal and becoming Grandmaster!</p>",
      "rawMarkdown": "Congratulations Bo for both the gold medal and becoming Grandmaster!",
      "votes": 1
    },
    {
      "id": 521499,
      "postDate": "2019-04-23T01:21:38.933Z",
      "content": "<p>Very good solution!  Thanks for sharing!</p>",
      "rawMarkdown": "Very good solution!  Thanks for sharing!",
      "votes": 1
    },
    {
      "id": 521489,
      "postDate": "2019-04-23T01:10:07.400Z",
      "content": "<p>Many congrats Bo, this is amazing!!</p>",
      "rawMarkdown": "Many congrats Bo, this is amazing!!",
      "votes": 1
    },
    {
      "id": 521478,
      "postDate": "2019-04-23T00:56:02.070Z",
      "content": "<p>Thank you. That's amazing, achieving 7th without fine-tuning.</p>",
      "rawMarkdown": "Thank you. That's amazing, achieving 7th without fine-tuning.",
      "votes": 1
    },
    {
      "id": 521463,
      "postDate": "2019-04-23T00:39:15.950Z",
      "content": "<p>Thank you for sharing.</p>",
      "rawMarkdown": "Thank you for sharing.",
      "votes": 1,
      "replies": [
        {
          "id": 521467,
          "postDate": "2019-04-23T00:41:49.500Z",
          "content": "<p>Thanks Chanhu. And thanks a lot for your kernel!</p>",
          "rawMarkdown": "Thanks Chanhu. And thanks a lot for your kernel!"
        },
        {
          "id": 521580,
          "postDate": "2019-04-23T04:16:09.387Z",
          "content": "<p>It’s My Pleasure. I am really impressed by you solution.</p>",
          "rawMarkdown": "It’s My Pleasure. I am really impressed by you solution."
        }
      ]
    },
    {
      "id": 521462,
      "postDate": "2019-04-23T00:34:10.080Z",
      "content": "<p>Thank you Bo for sharing the solution. Congrats on the top 10 finish ! </p>",
      "rawMarkdown": "Thank you Bo for sharing the solution. Congrats on the top 10 finish ! ",
      "votes": 1
    },
    {
      "id": 528048,
      "postDate": "2019-05-07T01:04:23.407Z",
      "content": "<p>paper is available at <a href=\"https://arxiv.org/abs/1905.01780\">https://arxiv.org/abs/1905.01780</a></p>",
      "rawMarkdown": "paper is available at https://arxiv.org/abs/1905.01780",
      "votes": 2,
      "replies": [
        {
          "id": 528178,
          "postDate": "2019-05-07T07:45:13.530Z",
          "content": "<p>Thanks! We also submitted a paper on this ACL workshop. Wondered if we are allowed to submit it on Arxiv as well. Typically it depends on the particular conference rules. Did you submit the same paper on the workshop as well? Or Arxiv only?</p>",
          "rawMarkdown": "Thanks! We also submitted a paper on this ACL workshop. Wondered if we are allowed to submit it on Arxiv as well. Typically it depends on the particular conference rules. Did you submit the same paper on the workshop as well? Or Arxiv only?"
        },
        {
          "id": 528323,
          "postDate": "2019-05-07T13:40:29.650Z",
          "content": "<p>yeah I submitted to the workshop as well</p>\n\n<p>regular papers which are anonymized for double blind review are prohibited for preprint release, but since this shared-task description paper is not anonymized (we pretty much described what we did in the Discussion here anyway), I think it's okay. See: <a href=\"http://www.acl2019.org/EN/call-for-papers.xhtml\">http://www.acl2019.org/EN/call-for-papers.xhtml</a></p>",
          "rawMarkdown": "yeah I submitted to the workshop as well\n\nregular papers which are anonymized for double blind review are prohibited for preprint release, but since this shared-task description paper is not anonymized (we pretty much described what we did in the Discussion here anyway), I think it's okay. See: http://www.acl2019.org/EN/call-for-papers.xhtml",
          "votes": 1
        }
      ]
    },
    {
      "id": 521466,
      "postDate": "2019-04-23T00:41:23.693Z",
      "content": "<p>I'm really impressed by how well you did without fine-tuning. Congratulations!</p>",
      "rawMarkdown": "I'm really impressed by how well you did without fine-tuning. Congratulations!",
      "votes": 2,
      "replies": [
        {
          "id": 521471,
          "postDate": "2019-04-23T00:48:58.603Z",
          "content": "<p>Thank you Matei. Your kernel was amazing! It gave us such a strong baseline in this competition. </p>\n\n<p>About fine-tuning, I tried, but couldn't figure out a way to make it work in this (i.e. your original) set up. Without public LB in stage1, it was hard to gauge how I place, so I went all-in on augmentation and ensemble without fine-tuning.</p>",
          "rawMarkdown": "Thank you Matei. Your kernel was amazing! It gave us such a strong baseline in this competition. \n\nAbout fine-tuning, I tried, but couldn't figure out a way to make it work in this (i.e. your original) set up. Without public LB in stage1, it was hard to gauge how I place, so I went all-in on augmentation and ensemble without fine-tuning.",
          "votes": 1
        },
        {
          "id": 521564,
          "postDate": "2019-04-23T03:40:42.453Z",
          "content": "<p>I'm glad my kernel helped. And thanks to you, we know that BERT out of the box can come pretty close to fine-tuned BERT on this task.</p>",
          "rawMarkdown": "I'm glad my kernel helped. And thanks to you, we know that BERT out of the box can come pretty close to fine-tuned BERT on this task.",
          "votes": 1
        }
      ]
    },
    {
      "id": 521465,
      "postDate": "2019-04-23T00:40:45.290Z",
      "content": "<p>Thank you and Grand Master tier is comming!</p>",
      "rawMarkdown": "Thank you and Grand Master tier is comming!",
      "votes": 2
    },
    {
      "id": 530380,
      "postDate": "2019-05-12T17:07:16.517Z",
      "content": "<p>How did you refactor the wrong labels? Did you do them manually?</p>",
      "rawMarkdown": "How did you refactor the wrong labels? Did you do them manually?"
    },
    {
      "id": 521469,
      "postDate": "2019-04-23T00:44:16.693Z",
      "content": "<p>Thanks for this overview. For me tuning threshold for clipping will be a takeaway.</p>\n\n<p>We used augmentation generating triples A-B-Pronoun where B is changed to some other name if A is a correct reference. It produced some really confident predictions, but often failed. Planned to use these predictions in stacking, but for some reason didn't succeed with stacking at all :)</p>\n\n<p>Indeed, very strong solution without fine-tuning. </p>",
      "rawMarkdown": "Thanks for this overview. For me tuning threshold for clipping will be a takeaway.\n\nWe used augmentation generating triples A-B-Pronoun where B is changed to some other name if A is a correct reference. It produced some really confident predictions, but often failed. Planned to use these predictions in stacking, but for some reason didn't succeed with stacking at all :)\n\nIndeed, very strong solution without fine-tuning. ",
      "replies": [
        {
          "id": 521475,
          "postDate": "2019-04-23T00:52:19.687Z",
          "content": "<p>Thanks Yury for all the discussion posts during the competition. </p>",
          "rawMarkdown": "Thanks Yury for all the discussion posts during the competition. ",
          "votes": 1
        }
      ]
    },
    {
      "id": 522047,
      "postDate": "2019-04-23T19:46:47.733Z",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "rawMarkdown": "Congratulations and thanks for sharing!",
      "votes": 1
    },
    {
      "id": 521879,
      "postDate": "2019-04-23T15:11:37.570Z",
      "content": "<p>Thanks for sharing! Congratulations!</p>",
      "rawMarkdown": "Thanks for sharing! Congratulations!",
      "votes": 1
    },
    {
      "id": 525633,
      "postDate": "2019-05-01T12:32:55.510Z",
      "content": "<p>good thanks</p>",
      "rawMarkdown": "good thanks"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T22:44:30Z",
  "competition_id": 12797,
  "competition_slug": "gendered-pronoun-resolution",
  "team_id": 2837548,
  "solution_url": "/competitions/gendered-pronoun-resolution/writeups/bo-7th-place-solution"
}