{
  "id": 248320,
  "title": "36th Place Solution (and \"could have been\" 22nd solution)",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/248320",
  "author_name": "Moiz Saifee",
  "post_date": "2021-06-23T07:45:46.239000",
  "votes": 9,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Congratulation to all the medal winners and everyone else who learned new things through this competition. Thanks to organizers and Kaggle for hosting the competition. </p>\n<p>We secured 36th using the solution we selected, but  had a submission which would have given us 22nd had we selected it (no regrets, all part of the game). Here is an overview and some high level details about our solution. </p>\n<p><strong>Overview</strong></p>\n<ol>\n<li>NER with Roberta backbone using new and old competition data: ~ 0.5x / 0.3x. One modification we had done was - we trained the NER with dual objective of predicting the probability of a dataset in a sentence along with NER labels (beginning / middle / end). This seemed to help compared to just using single NER objective. </li>\n<li>Ensemble with similar pipeline using different backbones like Scibert, Conll, Biobert: ~0.56 / ~0.414 (the solution we didn't select)</li>\n<li>Ensemble + some post processing that seemed work on Local and public LB but not on Private LB: ~0.56x / 0.368 (the solution we selected - 1/2)</li>\n<li>Ensemble + post processing + String Match using dataset lists: 0.640 / 0.363 (the solution we selected - 2/2)</li>\n</ol>\n<p><strong>Details</strong></p>\n<ul>\n<li>The training / scoring was done at a sentence level, extracted using Spacy Sentencizer </li>\n<li>Scored only ~10% of the total sentences in test publications which were extracted using regular expression to be able to run Ensemble of 4 models in allotted time for the notebooks. This led to no significant deterioration in accuracy on validation / LB. (thanks to my teammate <a href=\"https://www.kaggle.com/soloway\" target=\"_blank\">@soloway</a> for this)</li>\n<li>There were some datasets with a ton of publications in new / old training data, which was causing local CV to be unstable. Used only 5-10 Publications per dataset for training / validation to address this.</li>\n<li>String Matching seemed to work on Public LB, but hurt us slightly on the Private. We knew the risk, so only 1 of our 2 selected solutions had string matching. </li>\n</ul>",
  "messages": [
    {
      "id": 1362062,
      "postDate": "2021-06-23T07:45:46.240Z",
      "content": "<p>Congratulation to all the medal winners and everyone else who learned new things through this competition. Thanks to organizers and Kaggle for hosting the competition. </p>\n<p>We secured 36th using the solution we selected, but  had a submission which would have given us 22nd had we selected it (no regrets, all part of the game). Here is an overview and some high level details about our solution. </p>\n<p><strong>Overview</strong></p>\n<ol>\n<li>NER with Roberta backbone using new and old competition data: ~ 0.5x / 0.3x. One modification we had done was - we trained the NER with dual objective of predicting the probability of a dataset in a sentence along with NER labels (beginning / middle / end). This seemed to help compared to just using single NER objective. </li>\n<li>Ensemble with similar pipeline using different backbones like Scibert, Conll, Biobert: ~0.56 / ~0.414 (the solution we didn't select)</li>\n<li>Ensemble + some post processing that seemed work on Local and public LB but not on Private LB: ~0.56x / 0.368 (the solution we selected - 1/2)</li>\n<li>Ensemble + post processing + String Match using dataset lists: 0.640 / 0.363 (the solution we selected - 2/2)</li>\n</ol>\n<p><strong>Details</strong></p>\n<ul>\n<li>The training / scoring was done at a sentence level, extracted using Spacy Sentencizer </li>\n<li>Scored only ~10% of the total sentences in test publications which were extracted using regular expression to be able to run Ensemble of 4 models in allotted time for the notebooks. This led to no significant deterioration in accuracy on validation / LB. (thanks to my teammate <a href=\"https://www.kaggle.com/soloway\" target=\"_blank\">@soloway</a> for this)</li>\n<li>There were some datasets with a ton of publications in new / old training data, which was causing local CV to be unstable. Used only 5-10 Publications per dataset for training / validation to address this.</li>\n<li>String Matching seemed to work on Public LB, but hurt us slightly on the Private. We knew the risk, so only 1 of our 2 selected solutions had string matching. </li>\n</ul>",
      "rawMarkdown": "Congratulation to all the medal winners and everyone else who learned new things through this competition. Thanks to organizers and Kaggle for hosting the competition. \n\nWe secured 36th using the solution we selected, but  had a submission which would have given us 22nd had we selected it (no regrets, all part of the game). Here is an overview and some high level details about our solution. \n\n**Overview**\n1. NER with Roberta backbone using new and old competition data: ~ 0.5x / 0.3x. One modification we had done was - we trained the NER with dual objective of predicting the probability of a dataset in a sentence along with NER labels (beginning / middle / end). This seemed to help compared to just using single NER objective. \n2. Ensemble with similar pipeline using different backbones like Scibert, Conll, Biobert: ~0.56 / ~0.414 (the solution we didn't select)\n3. Ensemble + some post processing that seemed work on Local and public LB but not on Private LB: ~0.56x / 0.368 (the solution we selected - 1/2)\n4. Ensemble + post processing + String Match using dataset lists: 0.640 / 0.363 (the solution we selected - 2/2)\n\n**Details**\n- The training / scoring was done at a sentence level, extracted using Spacy Sentencizer \n- Scored only ~10% of the total sentences in test publications which were extracted using regular expression to be able to run Ensemble of 4 models in allotted time for the notebooks. This led to no significant deterioration in accuracy on validation / LB. (thanks to my teammate @soloway for this)\n- There were some datasets with a ton of publications in new / old training data, which was causing local CV to be unstable. Used only 5-10 Publications per dataset for training / validation to address this.\n- String Matching seemed to work on Public LB, but hurt us slightly on the Private. We knew the risk, so only 1 of our 2 selected solutions had string matching. ",
      "votes": 9
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1362062": "Congratulation to all the medal winners and everyone else who learned new things through this competition. Thanks to organizers and Kaggle for hosting the competition. \n\nWe secured 36th using the solution we selected, but  had a submission which would have given us 22nd had we selected it (no regrets, all part of the game). Here is an overview and some high level details about our solution. \n\n**Overview**\n1. NER with Roberta backbone using new and old competition data: ~ 0.5x / 0.3x. One modification we had done was - we trained the NER with dual objective of predicting the probability of a dataset in a sentence along with NER labels (beginning / middle / end). This seemed to help compared to just using single NER objective. \n2. Ensemble with similar pipeline using different backbones like Scibert, Conll, Biobert: ~0.56 / ~0.414 (the solution we didn't select)\n3. Ensemble + some post processing that seemed work on Local and public LB but not on Private LB: ~0.56x / 0.368 (the solution we selected - 1/2)\n4. Ensemble + post processing + String Match using dataset lists: 0.640 / 0.363 (the solution we selected - 2/2)\n\n**Details**\n- The training / scoring was done at a sentence level, extracted using Spacy Sentencizer \n- Scored only ~10% of the total sentences in test publications which were extracted using regular expression to be able to run Ensemble of 4 models in allotted time for the notebooks. This led to no significant deterioration in accuracy on validation / LB. (thanks to my teammate @soloway for this)\n- There were some datasets with a ton of publications in new / old training data, which was causing local CV to be unstable. Used only 5-10 Publications per dataset for training / validation to address this.\n- String Matching seemed to work on Public LB, but hurt us slightly on the Private. We knew the risk, so only 1 of our 2 selected solutions had string matching. "
  }
}