{
  "id": 248463,
  "title": "33rd place solution: pre- and postprocessing",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/248463",
  "author_name": "John Doge",
  "post_date": "2021-06-23T15:39:49.967000",
  "votes": 10,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I only had a couple of days to look at this problem, so the main insights I got were:<br>\n1) Pretty much every dataset string transforms into a chain of proper nouns after POS-tagging.<br>\n2) There is a limited amount of datasets both in train and test parts.</p>\n<p>The idea was that even a weak model could show generalization ability if frequency-based filtering is applied after training on 'PROPN' tags instead of original strings. Overall, my pipeline consists of four parts:<br>\n1) replacing every proper noun chain with a single token. \"Entity typing on the Open Entity dataset (Choi et al., 2018)\" becomes \"Entity typing on the PROPN dataset (PROPN, 2018)\".<br>\n2) training a simple token classification model (most transformer-based models will do).<br>\n3) collecting top document candidates by using an output probability threshold.<br>\n4) decoding all collected 'PROPN' tokens to original strings and filtering them out by frequency.</p>\n<p>Based on 4) implementation, this algorithm is capable of achieving a ~0.4 score on the private leaderboard (and frequently finds dataset mentions that look legit but are missing from annotations).</p>",
  "messages": [
    {
      "id": 1362736,
      "postDate": "2021-06-23T15:39:49.967Z",
      "content": "<p>I only had a couple of days to look at this problem, so the main insights I got were:<br>\n1) Pretty much every dataset string transforms into a chain of proper nouns after POS-tagging.<br>\n2) There is a limited amount of datasets both in train and test parts.</p>\n<p>The idea was that even a weak model could show generalization ability if frequency-based filtering is applied after training on 'PROPN' tags instead of original strings. Overall, my pipeline consists of four parts:<br>\n1) replacing every proper noun chain with a single token. \"Entity typing on the Open Entity dataset (Choi et al., 2018)\" becomes \"Entity typing on the PROPN dataset (PROPN, 2018)\".<br>\n2) training a simple token classification model (most transformer-based models will do).<br>\n3) collecting top document candidates by using an output probability threshold.<br>\n4) decoding all collected 'PROPN' tokens to original strings and filtering them out by frequency.</p>\n<p>Based on 4) implementation, this algorithm is capable of achieving a ~0.4 score on the private leaderboard (and frequently finds dataset mentions that look legit but are missing from annotations).</p>",
      "rawMarkdown": "I only had a couple of days to look at this problem, so the main insights I got were:\n1) Pretty much every dataset string transforms into a chain of proper nouns after POS-tagging.\n2) There is a limited amount of datasets both in train and test parts.\n\nThe idea was that even a weak model could show generalization ability if frequency-based filtering is applied after training on 'PROPN' tags instead of original strings. Overall, my pipeline consists of four parts:\n1) replacing every proper noun chain with a single token. \"Entity typing on the Open Entity dataset (Choi et al., 2018)\" becomes \"Entity typing on the PROPN dataset (PROPN, 2018)\".\n2) training a simple token classification model (most transformer-based models will do).\n3) collecting top document candidates by using an output probability threshold.\n4) decoding all collected 'PROPN' tokens to original strings and filtering them out by frequency.\n\nBased on 4) implementation, this algorithm is capable of achieving a ~0.4 score on the private leaderboard (and frequently finds dataset mentions that look legit but are missing from annotations).",
      "votes": 10
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1362736": "I only had a couple of days to look at this problem, so the main insights I got were:\n1) Pretty much every dataset string transforms into a chain of proper nouns after POS-tagging.\n2) There is a limited amount of datasets both in train and test parts.\n\nThe idea was that even a weak model could show generalization ability if frequency-based filtering is applied after training on 'PROPN' tags instead of original strings. Overall, my pipeline consists of four parts:\n1) replacing every proper noun chain with a single token. \"Entity typing on the Open Entity dataset (Choi et al., 2018)\" becomes \"Entity typing on the PROPN dataset (PROPN, 2018)\".\n2) training a simple token classification model (most transformer-based models will do).\n3) collecting top document candidates by using an output probability threshold.\n4) decoding all collected 'PROPN' tokens to original strings and filtering them out by frequency.\n\nBased on 4) implementation, this algorithm is capable of achieving a ~0.4 score on the private leaderboard (and frequently finds dataset mentions that look legit but are missing from annotations)."
  }
}