{
  "id": 248251,
  "title": "1st place solution: Metric learning and GPT",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/writeups/zalo-ftw-1st-place-solution-metric-learning-and-gp",
  "author_name": "",
  "post_date": "2021-09-02T12:42:18.867Z",
  "votes": 130,
  "comment_count": 31,
  "views": 0,
  "content": "<p>The code is available here: <a href=\"https://github.com/suicao/coleridge-gpt/\" target=\"_blank\">https://github.com/suicao/coleridge-gpt/</a><br>\nMy inference kernel is available here: <a href=\"https://www.kaggle.com/suicaokhoailang/submit-gpt-spacy?scriptVersionId=66488765\" target=\"_blank\">https://www.kaggle.com/suicaokhoailang/submit-gpt-spacy?scriptVersionId=66488765</a>. <br>\n<strong>Overview</strong><br>\nI'll lay out a few notable features from our solution here for now, kudos to my teammate <a href=\"https://www.kaggle.com/dathudeptrai\" target=\"_blank\">@dathudeptrai</a> for his immense effort in this competition, especially in the second half.<br>\nHis solution is metric learning with usual MLM backbones, I'll leave it to him to give more details later. We could not find a way to make our ensemble to look stable so we ended up submitting each of ours separately, my final sub scored 0.565 while his was 0.576.<br>\nMine is a text extraction model with CLM backbone and beamsearch, in this case GPT. I also tried to modify Roberta and Scibert with future masking, basically turning them into CLMs but so far GPT worked the best.</p>\n<p><strong>Why GPT</strong><br>\nThe reason why GPT worked and Roberta failed so hard is that Roberta is too clever for its own good. When you try to build a text extraction model with a single query of finding dataset names, Roberta doesn't need to care about the context and will try to find a substring that most resemble a dataset name, thus lead to overfitting.</p>\n<p>In my GPT + beamsearch model, GPT will be forced to predict <strong>whether the next token would be the start of dataset name</strong>, given only the previous context, and then predict <strong>wherether the next token would be the end of the mentio</strong>n, given the starting point and the extracted content so far. To make it more robust, I used a few sources of dataset and replaced 95% of mentions in the original training set with these random labels. Funnily, <a href=\"https://www.kaggle.com/datasets\" target=\"_blank\">https://www.kaggle.com/datasets</a> was the best source I could find for augmentation, although it has zero overlapping with the public (and possibly private) labels.</p>\n<p><strong>Modeling</strong><br>\nThe architecture is pretty much the same as what I used in the Tweet extraction competition. We reused XLNet’s decoder head for SQuAD, instead of predicting start and end indices independently, we first predict the start index, and use that information to predict the corresponding end.</p>\n<p><img src=\"https://i.imgur.com/hdVe7bV.png\" alt=\"\"></p>\n<p>For training, we predicting the start index normally.<br>\nWe take the hidden representation at the target index (ignoring the predicted) and concat it into the representations at every position. The new presentation is then fed to a MLP to predict the end index:</p>\n<p>Inference is a bit more complicated:</p>\n<ul>\n<li>We take top-k hidden states corresponding to top-k start indices with highest probabilities, this is normalized with sigmoid, not softmax like training.</li>\n<li>Each hidden state is then concatenated into the representations at every position.</li>\n<li>The new representation is fed to a MLP, similar to training. We then select top-k end indices for each selected hidden state, resulting in k*k top start-end pairs.</li>\n<li>We then calculate the joint probablities of every start-end pairs and take any pairs with a score large than 0.8.<br>\n<img src=\"https://i.imgur.com/nyP25it.png\" alt=\"\"></li>\n</ul>\n<p><strong>Final results and some gotchas</strong><br>\nMy selected submission score 0.565 on the private LB, but my best one (that I know of) was 0.594. I only took predictions that appeared at least 4 times in the corpus, the rare ones were only selected if they contained words like 'Data', 'Survey'.. etc. I tried to add more predictions using scispacy's abbreviation detector for my final sub but this actually lowered my score, should have trusted my model in the first place.</p>",
  "messages": [
    {
      "id": "1361617",
      "postDate": "06/23/2021 00:44:19",
      "content": "<p>The code is available here: <a href=\"https://github.com/suicao/coleridge-gpt/\" target=\"_blank\">https://github.com/suicao/coleridge-gpt/</a><br>\nMy inference kernel is available here: <a href=\"https://www.kaggle.com/suicaokhoailang/submit-gpt-spacy?scriptVersionId=66488765\" target=\"_blank\">https://www.kaggle.com/suicaokhoailang/submit-gpt-spacy?scriptVersionId=66488765</a>. <br>\n<strong>Overview</strong><br>\nI'll lay out a few notable features from our solution here for now, kudos to my teammate <a href=\"https://www.kaggle.com/dathudeptrai\" target=\"_blank\">@dathudeptrai</a> for his immense effort in this competition, especially in the second half.<br>\nHis solution is metric learning with usual MLM backbones, I'll leave it to him to give more details later. We could not find a way to make our ensemble to look stable so we ended up submitting each of ours separately, my final sub scored 0.565 while his was 0.576.<br>\nMine is a text extraction model with CLM backbone and beamsearch, in this case GPT. I also tried to modify Roberta and Scibert with future masking, basically turning them into CLMs but so far GPT worked the best.</p>\n<p><strong>Why GPT</strong><br>\nThe reason why GPT worked and Roberta failed so hard is that Roberta is too clever for its own good. When you try to build a text extraction model with a single query of finding dataset names, Roberta doesn't need to care about the context and will try to find a substring that most resemble a dataset name, thus lead to overfitting.</p>\n<p>In my GPT + beamsearch model, GPT will be forced to predict <strong>whether the next token would be the start of dataset name</strong>, given only the previous context, and then predict <strong>wherether the next token would be the end of the mentio</strong>n, given the starting point and the extracted content so far. To make it more robust, I used a few sources of dataset and replaced 95% of mentions in the original training set with these random labels. Funnily, <a href=\"https://www.kaggle.com/datasets\" target=\"_blank\">https://www.kaggle.com/datasets</a> was the best source I could find for augmentation, although it has zero overlapping with the public (and possibly private) labels.</p>\n<p><strong>Modeling</strong><br>\nThe architecture is pretty much the same as what I used in the Tweet extraction competition. We reused XLNet’s decoder head for SQuAD, instead of predicting start and end indices independently, we first predict the start index, and use that information to predict the corresponding end.</p>\n<p><img src=\"https://i.imgur.com/hdVe7bV.png\" alt=\"\"></p>\n<p>For training, we predicting the start index normally.<br>\nWe take the hidden representation at the target index (ignoring the predicted) and concat it into the representations at every position. The new presentation is then fed to a MLP to predict the end index:</p>\n<p>Inference is a bit more complicated:</p>\n<ul>\n<li>We take top-k hidden states corresponding to top-k start indices with highest probabilities, this is normalized with sigmoid, not softmax like training.</li>\n<li>Each hidden state is then concatenated into the representations at every position.</li>\n<li>The new representation is fed to a MLP, similar to training. We then select top-k end indices for each selected hidden state, resulting in k*k top start-end pairs.</li>\n<li>We then calculate the joint probablities of every start-end pairs and take any pairs with a score large than 0.8.<br>\n<img src=\"https://i.imgur.com/nyP25it.png\" alt=\"\"></li>\n</ul>\n<p><strong>Final results and some gotchas</strong><br>\nMy selected submission score 0.565 on the private LB, but my best one (that I know of) was 0.594. I only took predictions that appeared at least 4 times in the corpus, the rare ones were only selected if they contained words like 'Data', 'Survey'.. etc. I tried to add more predictions using scispacy's abbreviation detector for my final sub but this actually lowered my score, should have trusted my model in the first place.</p>",
      "rawMarkdown": "The code is available here: https://github.com/suicao/coleridge-gpt/\nMy inference kernel is available here: https://www.kaggle.com/suicaokhoailang/submit-gpt-spacy?scriptVersionId=66488765. \n**Overview**\nI'll lay out a few notable features from our solution here for now, kudos to my teammate @dathudeptrai for his immense effort in this competition, especially in the second half.\nHis solution is metric learning with usual MLM backbones, I'll leave it to him to give more details later. We could not find a way to make our ensemble to look stable so we ended up submitting each of ours separately, my final sub scored 0.565 while his was 0.576.\nMine is a text extraction model with CLM backbone and beamsearch, in this case GPT. I also tried to modify Roberta and Scibert with future masking, basically turning them into CLMs but so far GPT worked the best.\n\n**Why GPT**\nThe reason why GPT worked and Roberta failed so hard is that Roberta is too clever for its own good. When you try to build a text extraction model with a single query of finding dataset names, Roberta doesn't need to care about the context and will try to find a substring that most resemble a dataset name, thus lead to overfitting.\n\nIn my GPT + beamsearch model, GPT will be forced to predict **whether the next token would be the start of dataset name**, given only the previous context, and then predict **wherether the next token would be the end of the mentio**n, given the starting point and the extracted content so far. To make it more robust, I used a few sources of dataset and replaced 95% of mentions in the original training set with these random labels. Funnily, https://www.kaggle.com/datasets was the best source I could find for augmentation, although it has zero overlapping with the public (and possibly private) labels.\n\n**Modeling**\nThe architecture is pretty much the same as what I used in the Tweet extraction competition. We reused XLNet’s decoder head for SQuAD, instead of predicting start and end indices independently, we first predict the start index, and use that information to predict the corresponding end.\n\n![](https://i.imgur.com/hdVe7bV.png)\n\nFor training, we predicting the start index normally.\nWe take the hidden representation at the target index (ignoring the predicted) and concat it into the representations at every position. The new presentation is then fed to a MLP to predict the end index:\n\nInference is a bit more complicated:\n- We take top-k hidden states corresponding to top-k start indices with highest probabilities, this is normalized with sigmoid, not softmax like training.\n- Each hidden state is then concatenated into the representations at every position.\n- The new representation is fed to a MLP, similar to training. We then select top-k end indices for each selected hidden state, resulting in k*k top start-end pairs.\n- We then calculate the joint probablities of every start-end pairs and take any pairs with a score large than 0.8.\n![](https://i.imgur.com/nyP25it.png)\n\n**Final results and some gotchas**\nMy selected submission score 0.565 on the private LB, but my best one (that I know of) was 0.594. I only took predictions that appeared at least 4 times in the corpus, the rare ones were only selected if they contained words like 'Data', 'Survey'.. etc. I tried to add more predictions using scispacy's abbreviation detector for my final sub but this actually lowered my score, should have trusted my model in the first place.",
      "votes": null
    },
    {
      "id": "1361623",
      "postDate": "06/23/2021 00:54:23",
      "content": "<p>Impressive work! Congrats and thank you for sharing!</p>",
      "rawMarkdown": "Impressive work! Congrats and thank you for sharing!",
      "votes": null
    },
    {
      "id": "1361624",
      "postDate": "06/23/2021 00:58:17",
      "content": "<p>Thank you so much for sharing!</p>",
      "rawMarkdown": "Thank you so much for sharing!",
      "votes": null
    },
    {
      "id": "1361628",
      "postDate": "06/23/2021 01:05:28",
      "content": "<p>Congrats on first place. Thanks for sharing!</p>",
      "rawMarkdown": "Congrats on first place. Thanks for sharing!",
      "votes": null
    },
    {
      "id": "1361643",
      "postDate": "06/23/2021 01:40:01",
      "content": "<p>Big congrats Minh and Khoi!</p>",
      "rawMarkdown": "Big congrats Minh and Khoi!",
      "votes": null
    },
    {
      "id": "1361670",
      "postDate": "06/23/2021 02:09:07",
      "content": "<p>Congrats! Respect to the sheer amount of effort you guys put in from the start. </p>",
      "rawMarkdown": "Congrats! Respect to the sheer amount of effort you guys put in from the start.",
      "votes": null
    },
    {
      "id": "1361675",
      "postDate": "06/23/2021 02:15:31",
      "content": "<p>Congrats on 1st place!</p>\n<blockquote>\n  <p>Roberta is too clever for its own good.</p>\n</blockquote>\n<p>This is interesting for me</p>",
      "rawMarkdown": "Congrats on 1st place!\n\n>Roberta is too clever for its own good.\n\nThis is interesting for me",
      "votes": null
    },
    {
      "id": "1361679",
      "postDate": "06/23/2021 02:22:41",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/suicaokhoailang\" target=\"_blank\">@suicaokhoailang</a> on becoming a grandmaster!<br>\nAnd to <a href=\"https://www.kaggle.com/dathudeptrai\" target=\"_blank\">@dathudeptrai</a> on becoming a master!</p>\n<p>What better way than with a 1st place finish. 😊</p>",
      "rawMarkdown": "Congratulations @suicaokhoailang on becoming a grandmaster!\nAnd to @dathudeptrai on becoming a master!\n\nWhat better way than with a 1st place finish. 😊",
      "votes": null
    },
    {
      "id": "1361692",
      "postDate": "06/23/2021 02:43:21",
      "content": "<p>Thanks for sharing!</p>",
      "rawMarkdown": "Thanks for sharing!",
      "votes": null
    },
    {
      "id": "1361766",
      "postDate": "06/23/2021 03:59:45",
      "content": "<p>Congratulations!!!! </p>",
      "rawMarkdown": "Congratulations!!!!",
      "votes": null
    },
    {
      "id": "1361815",
      "postDate": "06/23/2021 04:35:46",
      "content": "<p>Congrats on 1st place!</p>",
      "rawMarkdown": "Congrats on 1st place!",
      "votes": null
    },
    {
      "id": "1361888",
      "postDate": "06/23/2021 05:35:14",
      "content": "<p>Very nice way to force model learn the context of dataset mentions rather than the mentions themselves.  I thought this was the problem to solve but did not thought of your solution.  Congrats on the win and GM title to come.</p>",
      "rawMarkdown": "Very nice way to force model learn the context of dataset mentions rather than the mentions themselves.  I thought this was the problem to solve but did not thought of your solution.  Congrats on the win and GM title to come.",
      "votes": null
    },
    {
      "id": "1361933",
      "postDate": "06/23/2021 06:09:32",
      "content": "<p>Congratulations!  That's an interesting approach.</p>",
      "rawMarkdown": "Congratulations!  That's an interesting approach.",
      "votes": null
    },
    {
      "id": "1362085",
      "postDate": "06/23/2021 08:09:13",
      "content": "<p>Congratulations Vietnamese team!<br>\nChúc mừng Khoi Nguyen và đồng bọn :))</p>",
      "rawMarkdown": "Congratulations Vietnamese team!\nChúc mừng Khoi Nguyen và đồng bọn :))",
      "votes": null
    },
    {
      "id": "1362107",
      "postDate": "06/23/2021 08:17:03",
      "content": "<p>Thank you for sharing! Congrats :)</p>",
      "rawMarkdown": "Thank you for sharing! Congrats :)",
      "votes": null
    },
    {
      "id": "1362170",
      "postDate": "06/23/2021 08:49:28",
      "content": "<p>What does CLM stand for?</p>",
      "rawMarkdown": "What does CLM stand for?",
      "votes": null
    },
    {
      "id": "1362237",
      "postDate": "06/23/2021 09:43:44",
      "content": "<p>Congratulations! Great insight!</p>",
      "rawMarkdown": "Congratulations! Great insight!",
      "votes": null
    },
    {
      "id": "1362280",
      "postDate": "06/23/2021 10:47:47",
      "content": "<p>Many congrats and thanks for sharing. Great!</p>",
      "rawMarkdown": "Many congrats and thanks for sharing. Great!",
      "votes": null
    },
    {
      "id": "1362309",
      "postDate": "06/23/2021 11:20:42",
      "content": "<p>Congratz for the win &amp; getting competitions GM !</p>",
      "rawMarkdown": "Congratz for the win & getting competitions GM !",
      "votes": null
    },
    {
      "id": "1362344",
      "postDate": "06/23/2021 11:44:55",
      "content": "<p>Congratulations ! Great work</p>",
      "rawMarkdown": "Congratulations ! Great work",
      "votes": null
    },
    {
      "id": "1362578",
      "postDate": "06/23/2021 14:19:00",
      "content": "<p>Thank you for your kind words 😊</p>",
      "rawMarkdown": "Thank you for your kind words 😊",
      "votes": null
    },
    {
      "id": "1362579",
      "postDate": "06/23/2021 14:19:38",
      "content": "<p>Causal Language Model. Every language model was causal before BERT was a thing.</p>",
      "rawMarkdown": "Causal Language Model. Every language model was causal before BERT was a thing.",
      "votes": null
    },
    {
      "id": "1362622",
      "postDate": "06/23/2021 14:41:22",
      "content": "<p>Nice work!</p>",
      "rawMarkdown": "Nice work!",
      "votes": null
    },
    {
      "id": "1362745",
      "postDate": "06/23/2021 15:49:15",
      "content": "<p>Very insightful, didn't think that sometimes contextual models try to cheat by ignoring the context itself. Your way of augmenting and adding random labels also makes it quite a generic solution which can be applied in other tasks. Thanks for sharing. Congrats 🙂</p>",
      "rawMarkdown": "Very insightful, didn't think that sometimes contextual models try to cheat by ignoring the context itself. Your way of augmenting and adding random labels also makes it quite a generic solution which can be applied in other tasks. Thanks for sharing. Congrats 🙂",
      "votes": null
    },
    {
      "id": "1364091",
      "postDate": "06/24/2021 15:54:44",
      "content": "<p>Apparently I'm kind of a rebel here for sharing my own kernel<br>\n<img src=\"https://i.imgur.com/HGt0JfL.png\" alt=\"\"></p>",
      "rawMarkdown": "Apparently I'm kind of a rebel here for sharing my own kernel\n![](https://i.imgur.com/HGt0JfL.png)",
      "votes": null
    },
    {
      "id": "1364459",
      "postDate": "06/25/2021 03:31:13",
      "content": "<p>Very nice work. Congratulation!</p>",
      "rawMarkdown": "Very nice work. Congratulation!",
      "votes": null
    },
    {
      "id": "1364532",
      "postDate": "06/25/2021 04:36:35",
      "content": "<p>Congratulations on the win! I just have one question, I do not understand what you mean by \"Each hidden state is then concatenated into the representations at every position.\" If you have k start indices with highest probabilities and length of your sentence is n, the how many total representations does that make? k*n?</p>\n<p>Thanks in advance!</p>",
      "rawMarkdown": "Congratulations on the win! I just have one question, I do not understand what you mean by \"Each hidden state is then concatenated into the representations at every position.\" If you have k start indices with highest probabilities and length of your sentence is n, the how many total representations does that make? k*n?\n\nThanks in advance!",
      "votes": null
    },
    {
      "id": "1364626",
      "postDate": "06/25/2021 05:36:28",
      "content": "<p>Congratulations, very well deserved!</p>",
      "rawMarkdown": "Congratulations, very well deserved!",
      "votes": null
    },
    {
      "id": "1365699",
      "postDate": "06/26/2021 03:32:33",
      "content": "<p>Congratulations, weekend review with many thanks to you and learn a lot from what you share. Great work!</p>",
      "rawMarkdown": "Congratulations, weekend review with many thanks to you and learn a lot from what you share. Great work!",
      "votes": null
    },
    {
      "id": "1370493",
      "postDate": "06/30/2021 07:40:30",
      "content": "<p>Nice work. Thanks for sharing </p>",
      "rawMarkdown": "Nice work. Thanks for sharing",
      "votes": null
    },
    {
      "id": "1371460",
      "postDate": "07/01/2021 02:58:39",
      "content": "<p>Good Job! Thank you for contributing back to the community with some knowledge!</p>",
      "rawMarkdown": "Good Job! Thank you for contributing back to the community with some knowledge!",
      "votes": null
    },
    {
      "id": "1371501",
      "postDate": "07/01/2021 04:04:03",
      "content": "<p>Nice, thanks for sharing….</p>",
      "rawMarkdown": "Nice, thanks for sharing....",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1361623,
      "author_name": "laevatein",
      "author_url": "",
      "post_date": "06/23/2021 00:54:23",
      "content": "<p>Impressive work! Congrats and thank you for sharing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361624,
      "author_name": "ktgiahieu",
      "author_url": "",
      "post_date": "06/23/2021 00:58:17",
      "content": "<p>Thank you so much for sharing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361628,
      "author_name": "toxicmaze",
      "author_url": "",
      "post_date": "06/23/2021 01:05:28",
      "content": "<p>Congrats on first place. Thanks for sharing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361643,
      "author_name": "mcggood",
      "author_url": "",
      "post_date": "06/23/2021 01:40:01",
      "content": "<p>Big congrats Minh and Khoi!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361670,
      "author_name": "leecming",
      "author_url": "",
      "post_date": "06/23/2021 02:09:07",
      "content": "<p>Congrats! Respect to the sheer amount of effort you guys put in from the start. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361675,
      "author_name": "teyosan1229",
      "author_url": "",
      "post_date": "06/23/2021 02:15:31",
      "content": "<p>Congrats on 1st place!</p>\n<blockquote>\n  <p>Roberta is too clever for its own good.</p>\n</blockquote>\n<p>This is interesting for me</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361679,
      "author_name": "jacob34",
      "author_url": "",
      "post_date": "06/23/2021 02:22:41",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/suicaokhoailang\" target=\"_blank\">@suicaokhoailang</a> on becoming a grandmaster!<br>\nAnd to <a href=\"https://www.kaggle.com/dathudeptrai\" target=\"_blank\">@dathudeptrai</a> on becoming a master!</p>\n<p>What better way than with a 1st place finish. 😊</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361692,
      "author_name": "ssinyu",
      "author_url": "",
      "post_date": "06/23/2021 02:43:21",
      "content": "<p>Thanks for sharing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361766,
      "author_name": "duykhanh99",
      "author_url": "",
      "post_date": "06/23/2021 03:59:45",
      "content": "<p>Congratulations!!!! </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361815,
      "author_name": "youzi2016",
      "author_url": "",
      "post_date": "06/23/2021 04:35:46",
      "content": "<p>Congrats on 1st place!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361888,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "06/23/2021 05:35:14",
      "content": "<p>Very nice way to force model learn the context of dataset mentions rather than the mentions themselves.  I thought this was the problem to solve but did not thought of your solution.  Congrats on the win and GM title to come.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1362578,
          "author_name": "suicaokhoailang",
          "author_url": "",
          "post_date": "06/23/2021 14:19:00",
          "content": "<p>Thank you for your kind words 😊</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1361933,
      "author_name": "vlomme",
      "author_url": "",
      "post_date": "06/23/2021 06:09:32",
      "content": "<p>Congratulations!  That's an interesting approach.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1362085,
      "author_name": "khoaint",
      "author_url": "",
      "post_date": "06/23/2021 08:09:13",
      "content": "<p>Congratulations Vietnamese team!<br>\nChúc mừng Khoi Nguyen và đồng bọn :))</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1362107,
      "author_name": "emphymachine",
      "author_url": "",
      "post_date": "06/23/2021 08:17:03",
      "content": "<p>Thank you for sharing! Congrats :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1362170,
      "author_name": "jackchungchiehyu",
      "author_url": "",
      "post_date": "06/23/2021 08:49:28",
      "content": "<p>What does CLM stand for?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1362579,
          "author_name": "suicaokhoailang",
          "author_url": "",
          "post_date": "06/23/2021 14:19:38",
          "content": "<p>Causal Language Model. Every language model was causal before BERT was a thing.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1362237,
      "author_name": "pndong",
      "author_url": "",
      "post_date": "06/23/2021 09:43:44",
      "content": "<p>Congratulations! Great insight!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1362280,
      "author_name": "sejoongkim",
      "author_url": "",
      "post_date": "06/23/2021 10:47:47",
      "content": "<p>Many congrats and thanks for sharing. Great!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1362309,
      "author_name": "theoviel",
      "author_url": "",
      "post_date": "06/23/2021 11:20:42",
      "content": "<p>Congratz for the win &amp; getting competitions GM !</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1362344,
      "author_name": "tahaazzim",
      "author_url": "",
      "post_date": "06/23/2021 11:44:55",
      "content": "<p>Congratulations ! Great work</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1362622,
      "author_name": "olehkivernyk",
      "author_url": "",
      "post_date": "06/23/2021 14:41:22",
      "content": "<p>Nice work!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1362745,
      "author_name": "shivammiglani",
      "author_url": "",
      "post_date": "06/23/2021 15:49:15",
      "content": "<p>Very insightful, didn't think that sometimes contextual models try to cheat by ignoring the context itself. Your way of augmenting and adding random labels also makes it quite a generic solution which can be applied in other tasks. Thanks for sharing. Congrats 🙂</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1364091,
      "author_name": "suicaokhoailang",
      "author_url": "",
      "post_date": "06/24/2021 15:54:44",
      "content": "<p>Apparently I'm kind of a rebel here for sharing my own kernel<br>\n<img src=\"https://i.imgur.com/HGt0JfL.png\" alt=\"\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1364459,
      "author_name": "nlebang",
      "author_url": "",
      "post_date": "06/25/2021 03:31:13",
      "content": "<p>Very nice work. Congratulation!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1364532,
      "author_name": "gsreyan",
      "author_url": "",
      "post_date": "06/25/2021 04:36:35",
      "content": "<p>Congratulations on the win! I just have one question, I do not understand what you mean by \"Each hidden state is then concatenated into the representations at every position.\" If you have k start indices with highest probabilities and length of your sentence is n, the how many total representations does that make? k*n?</p>\n<p>Thanks in advance!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1364626,
      "author_name": "saurabhbagchi",
      "author_url": "",
      "post_date": "06/25/2021 05:36:28",
      "content": "<p>Congratulations, very well deserved!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1365699,
      "author_name": "nguyenvanlinh",
      "author_url": "",
      "post_date": "06/26/2021 03:32:33",
      "content": "<p>Congratulations, weekend review with many thanks to you and learn a lot from what you share. Great work!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1370493,
      "author_name": "samarthgupta39",
      "author_url": "",
      "post_date": "06/30/2021 07:40:30",
      "content": "<p>Nice work. Thanks for sharing </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1371460,
      "author_name": "nathangjohnson",
      "author_url": "",
      "post_date": "07/01/2021 02:58:39",
      "content": "<p>Good Job! Thank you for contributing back to the community with some knowledge!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1371501,
      "author_name": "laxmankusuma",
      "author_url": "",
      "post_date": "07/01/2021 04:04:03",
      "content": "<p>Nice, thanks for sharing….</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1361617": "The code is available here: https://github.com/suicao/coleridge-gpt/\nMy inference kernel is available here: https://www.kaggle.com/suicaokhoailang/submit-gpt-spacy?scriptVersionId=66488765. \n**Overview**\nI'll lay out a few notable features from our solution here for now, kudos to my teammate @dathudeptrai for his immense effort in this competition, especially in the second half.\nHis solution is metric learning with usual MLM backbones, I'll leave it to him to give more details later. We could not find a way to make our ensemble to look stable so we ended up submitting each of ours separately, my final sub scored 0.565 while his was 0.576.\nMine is a text extraction model with CLM backbone and beamsearch, in this case GPT. I also tried to modify Roberta and Scibert with future masking, basically turning them into CLMs but so far GPT worked the best.\n\n**Why GPT**\nThe reason why GPT worked and Roberta failed so hard is that Roberta is too clever for its own good. When you try to build a text extraction model with a single query of finding dataset names, Roberta doesn't need to care about the context and will try to find a substring that most resemble a dataset name, thus lead to overfitting.\n\nIn my GPT + beamsearch model, GPT will be forced to predict **whether the next token would be the start of dataset name**, given only the previous context, and then predict **wherether the next token would be the end of the mentio**n, given the starting point and the extracted content so far. To make it more robust, I used a few sources of dataset and replaced 95% of mentions in the original training set with these random labels. Funnily, https://www.kaggle.com/datasets was the best source I could find for augmentation, although it has zero overlapping with the public (and possibly private) labels.\n\n**Modeling**\nThe architecture is pretty much the same as what I used in the Tweet extraction competition. We reused XLNet’s decoder head for SQuAD, instead of predicting start and end indices independently, we first predict the start index, and use that information to predict the corresponding end.\n\n![](https://i.imgur.com/hdVe7bV.png)\n\nFor training, we predicting the start index normally.\nWe take the hidden representation at the target index (ignoring the predicted) and concat it into the representations at every position. The new presentation is then fed to a MLP to predict the end index:\n\nInference is a bit more complicated:\n- We take top-k hidden states corresponding to top-k start indices with highest probabilities, this is normalized with sigmoid, not softmax like training.\n- Each hidden state is then concatenated into the representations at every position.\n- The new representation is fed to a MLP, similar to training. We then select top-k end indices for each selected hidden state, resulting in k*k top start-end pairs.\n- We then calculate the joint probablities of every start-end pairs and take any pairs with a score large than 0.8.\n![](https://i.imgur.com/nyP25it.png)\n\n**Final results and some gotchas**\nMy selected submission score 0.565 on the private LB, but my best one (that I know of) was 0.594. I only took predictions that appeared at least 4 times in the corpus, the rare ones were only selected if they contained words like 'Data', 'Survey'.. etc. I tried to add more predictions using scispacy's abbreviation detector for my final sub but this actually lowered my score, should have trusted my model in the first place.",
    "1361623": "Impressive work! Congrats and thank you for sharing!",
    "1361624": "Thank you so much for sharing!",
    "1361628": "Congrats on first place. Thanks for sharing!",
    "1361643": "Big congrats Minh and Khoi!",
    "1361670": "Congrats! Respect to the sheer amount of effort you guys put in from the start.",
    "1361675": "Congrats on 1st place!\n\n>Roberta is too clever for its own good.\n\nThis is interesting for me",
    "1361679": "Congratulations @suicaokhoailang on becoming a grandmaster!\nAnd to @dathudeptrai on becoming a master!\n\nWhat better way than with a 1st place finish. 😊",
    "1361692": "Thanks for sharing!",
    "1361766": "Congratulations!!!!",
    "1361815": "Congrats on 1st place!",
    "1361888": "Very nice way to force model learn the context of dataset mentions rather than the mentions themselves.  I thought this was the problem to solve but did not thought of your solution.  Congrats on the win and GM title to come.",
    "1361933": "Congratulations!  That's an interesting approach.",
    "1362085": "Congratulations Vietnamese team!\nChúc mừng Khoi Nguyen và đồng bọn :))",
    "1362107": "Thank you for sharing! Congrats :)",
    "1362170": "What does CLM stand for?",
    "1362237": "Congratulations! Great insight!",
    "1362280": "Many congrats and thanks for sharing. Great!",
    "1362309": "Congratz for the win & getting competitions GM !",
    "1362344": "Congratulations ! Great work",
    "1362578": "Thank you for your kind words 😊",
    "1362579": "Causal Language Model. Every language model was causal before BERT was a thing.",
    "1362622": "Nice work!",
    "1362745": "Very insightful, didn't think that sometimes contextual models try to cheat by ignoring the context itself. Your way of augmenting and adding random labels also makes it quite a generic solution which can be applied in other tasks. Thanks for sharing. Congrats 🙂",
    "1364091": "Apparently I'm kind of a rebel here for sharing my own kernel\n![](https://i.imgur.com/HGt0JfL.png)",
    "1364459": "Very nice work. Congratulation!",
    "1364532": "Congratulations on the win! I just have one question, I do not understand what you mean by \"Each hidden state is then concatenated into the representations at every position.\" If you have k start indices with highest probabilities and length of your sentence is n, the how many total representations does that make? k*n?\n\nThanks in advance!",
    "1364626": "Congratulations, very well deserved!",
    "1365699": "Congratulations, weekend review with many thanks to you and learn a lot from what you share. Great work!",
    "1370493": "Nice work. Thanks for sharing",
    "1371460": "Good Job! Thank you for contributing back to the community with some knowledge!",
    "1371501": "Nice, thanks for sharing...."
  },
  "source": "meta"
}