{
  "id": 248511,
  "title": "14th Place Solution (with notebooks)",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/writeups/alex-lichen-14th-place-solution-with-notebooks",
  "author_name": "",
  "post_date": "2021-06-23T21:02:12.951615300Z",
  "votes": 15,
  "comment_count": 3,
  "views": 0,
  "content": "<p>One away from gold! I did this competition on Kaggle notebooks only:</p>\n<ul>\n<li>Additional hand-labeled dataset titles (from train only): <a href=\"https://www.kaggle.com/lichena/coleridgehandlabeled\" target=\"_blank\">https://www.kaggle.com/lichena/coleridgehandlabeled</a></li>\n<li>Preprocessing: <a href=\"https://www.kaggle.com/lichena/coleridge-pre-processing-hand\" target=\"_blank\">https://www.kaggle.com/lichena/coleridge-pre-processing-hand</a></li>\n<li>Training: <a href=\"https://www.kaggle.com/lichena/coleridge-ner?scriptVersionId=66238738\" target=\"_blank\">https://www.kaggle.com/lichena/coleridge-ner?scriptVersionId=66238738</a></li>\n<li>Inference: <a href=\"https://www.kaggle.com/lichena/coleridge-ner-inference\" target=\"_blank\">https://www.kaggle.com/lichena/coleridge-ner-inference</a></li>\n</ul>\n<p>My highest chosen private LB submission scored 0.558/.437, while my best scored 0.545/0.444:</p>\n<ul>\n<li>I fine-tuned bert-base-cased + crf on BIO-tagged chunks of ~200-400 words. </li>\n<li>The key for me was hand-annotating a larger training set. I used regex to find an initial set, then went through the list of false positives for my model that I thought were valid and recategorized them into the positive label set. Ultimately, this larger dataset scored 0.601 publicLB when naively submitted, a subset scored 0.621.</li>\n<li>For training, I used 80% chunks which were positive mentions, 20% chunks which had no mentions. I tried doing a naive cv of 20% chunks with positive mentions, but this didn't generalize well to the private lb</li>\n<li>Instead of the default clean_text() function provided, I also kept casing and parentheticals during training and inference, which boosted the LB score. </li>\n</ul>\n<p>If you're like me and are only doing competitions with kaggle notebooks, here's some tips I learned:</p>\n<ul>\n<li>Use CPU as much as you can - I would choose 2-3 files as the training set and run all the code with those test files first to test bugs</li>\n<li>When you have your pipeline, start with distilled models which are faster to train and save you gpu time. Run it on a small sample of data</li>\n<li>Start the competition early</li>\n<li>Look for competitions where there is a systematic advantage to be had - in this case, hand-annotation provided an \"edge\" that I knew many others weren't going to attempt to replicate.</li>\n</ul>",
  "messages": [
    {
      "id": "1363019",
      "postDate": "06/23/2021 21:02:12",
      "content": "<p>One away from gold! I did this competition on Kaggle notebooks only:</p>\n<ul>\n<li>Additional hand-labeled dataset titles (from train only): <a href=\"https://www.kaggle.com/lichena/coleridgehandlabeled\" target=\"_blank\">https://www.kaggle.com/lichena/coleridgehandlabeled</a></li>\n<li>Preprocessing: <a href=\"https://www.kaggle.com/lichena/coleridge-pre-processing-hand\" target=\"_blank\">https://www.kaggle.com/lichena/coleridge-pre-processing-hand</a></li>\n<li>Training: <a href=\"https://www.kaggle.com/lichena/coleridge-ner?scriptVersionId=66238738\" target=\"_blank\">https://www.kaggle.com/lichena/coleridge-ner?scriptVersionId=66238738</a></li>\n<li>Inference: <a href=\"https://www.kaggle.com/lichena/coleridge-ner-inference\" target=\"_blank\">https://www.kaggle.com/lichena/coleridge-ner-inference</a></li>\n</ul>\n<p>My highest chosen private LB submission scored 0.558/.437, while my best scored 0.545/0.444:</p>\n<ul>\n<li>I fine-tuned bert-base-cased + crf on BIO-tagged chunks of ~200-400 words. </li>\n<li>The key for me was hand-annotating a larger training set. I used regex to find an initial set, then went through the list of false positives for my model that I thought were valid and recategorized them into the positive label set. Ultimately, this larger dataset scored 0.601 publicLB when naively submitted, a subset scored 0.621.</li>\n<li>For training, I used 80% chunks which were positive mentions, 20% chunks which had no mentions. I tried doing a naive cv of 20% chunks with positive mentions, but this didn't generalize well to the private lb</li>\n<li>Instead of the default clean_text() function provided, I also kept casing and parentheticals during training and inference, which boosted the LB score. </li>\n</ul>\n<p>If you're like me and are only doing competitions with kaggle notebooks, here's some tips I learned:</p>\n<ul>\n<li>Use CPU as much as you can - I would choose 2-3 files as the training set and run all the code with those test files first to test bugs</li>\n<li>When you have your pipeline, start with distilled models which are faster to train and save you gpu time. Run it on a small sample of data</li>\n<li>Start the competition early</li>\n<li>Look for competitions where there is a systematic advantage to be had - in this case, hand-annotation provided an \"edge\" that I knew many others weren't going to attempt to replicate.</li>\n</ul>",
      "rawMarkdown": "One away from gold! I did this competition on Kaggle notebooks only:\n- Additional hand-labeled dataset titles (from train only): https://www.kaggle.com/lichena/coleridgehandlabeled\n- Preprocessing: https://www.kaggle.com/lichena/coleridge-pre-processing-hand\n- Training: https://www.kaggle.com/lichena/coleridge-ner?scriptVersionId=66238738\n- Inference: https://www.kaggle.com/lichena/coleridge-ner-inference\n\nMy highest chosen private LB submission scored 0.558/.437, while my best scored 0.545/0.444:\n- I fine-tuned bert-base-cased + crf on BIO-tagged chunks of ~200-400 words. \n- The key for me was hand-annotating a larger training set. I used regex to find an initial set, then went through the list of false positives for my model that I thought were valid and recategorized them into the positive label set. Ultimately, this larger dataset scored 0.601 publicLB when naively submitted, a subset scored 0.621.\n- For training, I used 80% chunks which were positive mentions, 20% chunks which had no mentions. I tried doing a naive cv of 20% chunks with positive mentions, but this didn't generalize well to the private lb\n- Instead of the default clean_text() function provided, I also kept casing and parentheticals during training and inference, which boosted the LB score. \n\nIf you're like me and are only doing competitions with kaggle notebooks, here's some tips I learned:\n- Use CPU as much as you can - I would choose 2-3 files as the training set and run all the code with those test files first to test bugs\n- When you have your pipeline, start with distilled models which are faster to train and save you gpu time. Run it on a small sample of data\n- Start the competition early\n- Look for competitions where there is a systematic advantage to be had - in this case, hand-annotation provided an \"edge\" that I knew many others weren't going to attempt to replicate.",
      "votes": null
    },
    {
      "id": "1364423",
      "postDate": "06/25/2021 01:37:08",
      "content": "<p>Great first medal. Strong finish Alex.</p>",
      "rawMarkdown": "Great first medal. Strong finish Alex.",
      "votes": null
    },
    {
      "id": "1365521",
      "postDate": "06/25/2021 20:22:31",
      "content": "<p>Thank you!</p>",
      "rawMarkdown": "Thank you!",
      "votes": null
    },
    {
      "id": "1367850",
      "postDate": "06/28/2021 05:59:35",
      "content": "<p>Amazing, thanks for sharing</p>",
      "rawMarkdown": "Amazing, thanks for sharing",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1364423,
      "author_name": "underwearfitting",
      "author_url": "",
      "post_date": "06/25/2021 01:37:08",
      "content": "<p>Great first medal. Strong finish Alex.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1365521,
          "author_name": "lichena",
          "author_url": "",
          "post_date": "06/25/2021 20:22:31",
          "content": "<p>Thank you!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1367850,
      "author_name": "shenjiaxjtlu",
      "author_url": "",
      "post_date": "06/28/2021 05:59:35",
      "content": "<p>Amazing, thanks for sharing</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1363019": "One away from gold! I did this competition on Kaggle notebooks only:\n- Additional hand-labeled dataset titles (from train only): https://www.kaggle.com/lichena/coleridgehandlabeled\n- Preprocessing: https://www.kaggle.com/lichena/coleridge-pre-processing-hand\n- Training: https://www.kaggle.com/lichena/coleridge-ner?scriptVersionId=66238738\n- Inference: https://www.kaggle.com/lichena/coleridge-ner-inference\n\nMy highest chosen private LB submission scored 0.558/.437, while my best scored 0.545/0.444:\n- I fine-tuned bert-base-cased + crf on BIO-tagged chunks of ~200-400 words. \n- The key for me was hand-annotating a larger training set. I used regex to find an initial set, then went through the list of false positives for my model that I thought were valid and recategorized them into the positive label set. Ultimately, this larger dataset scored 0.601 publicLB when naively submitted, a subset scored 0.621.\n- For training, I used 80% chunks which were positive mentions, 20% chunks which had no mentions. I tried doing a naive cv of 20% chunks with positive mentions, but this didn't generalize well to the private lb\n- Instead of the default clean_text() function provided, I also kept casing and parentheticals during training and inference, which boosted the LB score. \n\nIf you're like me and are only doing competitions with kaggle notebooks, here's some tips I learned:\n- Use CPU as much as you can - I would choose 2-3 files as the training set and run all the code with those test files first to test bugs\n- When you have your pipeline, start with distilled models which are faster to train and save you gpu time. Run it on a small sample of data\n- Start the competition early\n- Look for competitions where there is a systematic advantage to be had - in this case, hand-annotation provided an \"edge\" that I knew many others weren't going to attempt to replicate.",
    "1364423": "Great first medal. Strong finish Alex.",
    "1365521": "Thank you!",
    "1367850": "Amazing, thanks for sharing"
  },
  "source": "meta"
}