{
  "id": 249027,
  "title": "38th place solution (NER + Heuristics)",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/249027",
  "author_name": "Sindhu V",
  "post_date": "2021-06-26T08:25:23.274000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I am afraid my solution is way too simple compared to some of the cool models I've seen. But here goes:</p>\n<p>I used Spacy's entity recogniser to filter all possible candidate datasets. Then I wrote a simple CFG which determines if those entities indeed denote a dataset. This part generates candidate datasets that are strict matches, thereby increasing the rate of false negatives. To mitigate this, I did a second pass on all texts with key terms in these candidate datasets to generate the final discovered dataset. That's the model in a nutshell. </p>\n<p>Few additional notes:</p>\n<ul>\n<li>This model is nearly unsupervised. </li>\n<li>I did not use the train dataset for the actual dataset prediction. I only used it to determine what kind of CFG I need to write. </li>\n<li>One very direct observation is, all datasets that are referred in these publications are camel-cased. If nothing else, you can start with these as your candidates and even exclude the named-entity task. (That gave me a private score of 0.29. The final model I submitted with Spacy gave 036).</li>\n<li>These are not free-form texts on which standard large-scale models have been trained. These are publications and they follow a certain standard. If you analyze the training dataset, you can automatically see that most of them are of the form <code>data collected from &lt;some dataset&gt;</code>, <code>Used &lt;some report/study&gt;</code>, and so on. The second pass in my model exists to capture those that don't follow these candidate phrasings.  </li>\n<li>I did not use Roberta for the same reason. See the first place solution post for a much more in-depth discussion on this. </li>\n<li>As for the heuristics, you can just start with \"data\" and look for subtree (in the PCFG) matches that intersect with the matched entities.  I used a few more. Again, the input texts are almost free of grammatical errors and follow very similar patterns while quoting references and external datasets.  </li>\n<li><strong>Spacy 3.0's Transformer gave me much better initial candidates. But it kept throwing some submission errors.</strong> So I used Spacy 2.5 since I didn't have much time to compete in this competition.</li>\n</ul>",
  "messages": [
    {
      "id": 1365905,
      "postDate": "2021-06-26T08:25:23.273Z",
      "content": "<p>I am afraid my solution is way too simple compared to some of the cool models I've seen. But here goes:</p>\n<p>I used Spacy's entity recogniser to filter all possible candidate datasets. Then I wrote a simple CFG which determines if those entities indeed denote a dataset. This part generates candidate datasets that are strict matches, thereby increasing the rate of false negatives. To mitigate this, I did a second pass on all texts with key terms in these candidate datasets to generate the final discovered dataset. That's the model in a nutshell. </p>\n<p>Few additional notes:</p>\n<ul>\n<li>This model is nearly unsupervised. </li>\n<li>I did not use the train dataset for the actual dataset prediction. I only used it to determine what kind of CFG I need to write. </li>\n<li>One very direct observation is, all datasets that are referred in these publications are camel-cased. If nothing else, you can start with these as your candidates and even exclude the named-entity task. (That gave me a private score of 0.29. The final model I submitted with Spacy gave 036).</li>\n<li>These are not free-form texts on which standard large-scale models have been trained. These are publications and they follow a certain standard. If you analyze the training dataset, you can automatically see that most of them are of the form <code>data collected from &lt;some dataset&gt;</code>, <code>Used &lt;some report/study&gt;</code>, and so on. The second pass in my model exists to capture those that don't follow these candidate phrasings.  </li>\n<li>I did not use Roberta for the same reason. See the first place solution post for a much more in-depth discussion on this. </li>\n<li>As for the heuristics, you can just start with \"data\" and look for subtree (in the PCFG) matches that intersect with the matched entities.  I used a few more. Again, the input texts are almost free of grammatical errors and follow very similar patterns while quoting references and external datasets.  </li>\n<li><strong>Spacy 3.0's Transformer gave me much better initial candidates. But it kept throwing some submission errors.</strong> So I used Spacy 2.5 since I didn't have much time to compete in this competition.</li>\n</ul>",
      "rawMarkdown": "I am afraid my solution is way too simple compared to some of the cool models I've seen. But here goes:\n\nI used Spacy's entity recogniser to filter all possible candidate datasets. Then I wrote a simple CFG which determines if those entities indeed denote a dataset. This part generates candidate datasets that are strict matches, thereby increasing the rate of false negatives. To mitigate this, I did a second pass on all texts with key terms in these candidate datasets to generate the final discovered dataset. That's the model in a nutshell. \n\nFew additional notes:\n* This model is nearly unsupervised. \n* I did not use the train dataset for the actual dataset prediction. I only used it to determine what kind of CFG I need to write. \n* One very direct observation is, all datasets that are referred in these publications are camel-cased. If nothing else, you can start with these as your candidates and even exclude the named-entity task. (That gave me a private score of 0.29. The final model I submitted with Spacy gave 036).\n* These are not free-form texts on which standard large-scale models have been trained. These are publications and they follow a certain standard. If you analyze the training dataset, you can automatically see that most of them are of the form `data collected from <some dataset>`, `Used <some report/study>`, and so on. The second pass in my model exists to capture those that don't follow these candidate phrasings.  \n* I did not use Roberta for the same reason. See the first place solution post for a much more in-depth discussion on this. \n* As for the heuristics, you can just start with \"data\" and look for subtree (in the PCFG) matches that intersect with the matched entities.  I used a few more. Again, the input texts are almost free of grammatical errors and follow very similar patterns while quoting references and external datasets.  \n* **Spacy 3.0's Transformer gave me much better initial candidates. But it kept throwing some submission errors.** So I used Spacy 2.5 since I didn't have much time to compete in this competition.",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1365905": "I am afraid my solution is way too simple compared to some of the cool models I've seen. But here goes:\n\nI used Spacy's entity recogniser to filter all possible candidate datasets. Then I wrote a simple CFG which determines if those entities indeed denote a dataset. This part generates candidate datasets that are strict matches, thereby increasing the rate of false negatives. To mitigate this, I did a second pass on all texts with key terms in these candidate datasets to generate the final discovered dataset. That's the model in a nutshell. \n\nFew additional notes:\n* This model is nearly unsupervised. \n* I did not use the train dataset for the actual dataset prediction. I only used it to determine what kind of CFG I need to write. \n* One very direct observation is, all datasets that are referred in these publications are camel-cased. If nothing else, you can start with these as your candidates and even exclude the named-entity task. (That gave me a private score of 0.29. The final model I submitted with Spacy gave 036).\n* These are not free-form texts on which standard large-scale models have been trained. These are publications and they follow a certain standard. If you analyze the training dataset, you can automatically see that most of them are of the form `data collected from <some dataset>`, `Used <some report/study>`, and so on. The second pass in my model exists to capture those that don't follow these candidate phrasings.  \n* I did not use Roberta for the same reason. See the first place solution post for a much more in-depth discussion on this. \n* As for the heuristics, you can just start with \"data\" and look for subtree (in the PCFG) matches that intersect with the matched entities.  I used a few more. Again, the input texts are almost free of grammatical errors and follow very similar patterns while quoting references and external datasets.  \n* **Spacy 3.0's Transformer gave me much better initial candidates. But it kept throwing some submission errors.** So I used Spacy 2.5 since I didn't have much time to compete in this competition."
  }
}