{
  "id": 235752,
  "title": "Problem with training labels",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/235752",
  "author_name": "Moeen Bagheri",
  "post_date": "2021-05-01T05:25:26.387000",
  "votes": 10,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I was investigating a problem in my preprocessing and I found an issue in the training data.</p>\n<p>Many of the articles contain labels (cleaned) that are not actually mentioned in the text (cleaned).</p>\n<p>For example, the article with Id <em>\"14592eb8-a06e-4b26-89c3-939a9ce28698\"</em> has label <em>\"ibtracs\"</em>, however, in the text, the dataset is referred to as <em>\"ibtracks\"</em>.</p>\n<p>As another example, article <em>\"6fa563ab-1ced-40a3-a626-103306098092\"</em> has label <em>\"beginning postsecondary student\"</em>, but this exact phrase cannot be found in the cleaned text of the article. You can only find the phrase <em>\"beginning postsecondary student<strong>s</strong>\"</em>, which is another label for this article.</p>\n<p>Another article with Id <em>\"f70f268a-8653-47ae-a4bc-c41388ba9a7c\"</em> has label <em>\"slosh model\"</em>, but you can only find the text <em>\"slosh model<strong>s</strong>\"</em> in the article.</p>\n<p>These are just a few examples and there are a total of 69 articles with similar issues (including some that the label is only found within a url). I think this is a huge problem. The only thing I can think of is to remove these articles for now.</p>\n<p>However, this also raises a concern for me with the LB datasets. It is unclear to me how this labeling is done. If there are going to be labels that are not in the cleaned text of the articles, its going to be near impossible for a model to be able to consistently predict them. This is particularly an issue when two phrases such as <em>\"slosh model\"</em> and <em>\"slosh models\"</em> have a jaccard score of 0.33, and won't be counted as a match.</p>",
  "messages": [
    {
      "id": 1289522,
      "postDate": "2021-05-01T05:25:26.387Z",
      "content": "<p>I was investigating a problem in my preprocessing and I found an issue in the training data.</p>\n<p>Many of the articles contain labels (cleaned) that are not actually mentioned in the text (cleaned).</p>\n<p>For example, the article with Id <em>\"14592eb8-a06e-4b26-89c3-939a9ce28698\"</em> has label <em>\"ibtracs\"</em>, however, in the text, the dataset is referred to as <em>\"ibtracks\"</em>.</p>\n<p>As another example, article <em>\"6fa563ab-1ced-40a3-a626-103306098092\"</em> has label <em>\"beginning postsecondary student\"</em>, but this exact phrase cannot be found in the cleaned text of the article. You can only find the phrase <em>\"beginning postsecondary student<strong>s</strong>\"</em>, which is another label for this article.</p>\n<p>Another article with Id <em>\"f70f268a-8653-47ae-a4bc-c41388ba9a7c\"</em> has label <em>\"slosh model\"</em>, but you can only find the text <em>\"slosh model<strong>s</strong>\"</em> in the article.</p>\n<p>These are just a few examples and there are a total of 69 articles with similar issues (including some that the label is only found within a url). I think this is a huge problem. The only thing I can think of is to remove these articles for now.</p>\n<p>However, this also raises a concern for me with the LB datasets. It is unclear to me how this labeling is done. If there are going to be labels that are not in the cleaned text of the articles, its going to be near impossible for a model to be able to consistently predict them. This is particularly an issue when two phrases such as <em>\"slosh model\"</em> and <em>\"slosh models\"</em> have a jaccard score of 0.33, and won't be counted as a match.</p>",
      "rawMarkdown": "I was investigating a problem in my preprocessing and I found an issue in the training data.\n\nMany of the articles contain labels (cleaned) that are not actually mentioned in the text (cleaned).\n\nFor example, the article with Id *\"14592eb8-a06e-4b26-89c3-939a9ce28698\"* has label *\"ibtracs\"*, however, in the text, the dataset is referred to as *\"ibtracks\"*.\n\nAs another example, article *\"6fa563ab-1ced-40a3-a626-103306098092\"* has label *\"beginning postsecondary student\"*, but this exact phrase cannot be found in the cleaned text of the article. You can only find the phrase *\"beginning postsecondary student**s**\"*, which is another label for this article.\n\nAnother article with Id *\"f70f268a-8653-47ae-a4bc-c41388ba9a7c\"* has label *\"slosh model\"*, but you can only find the text *\"slosh model**s**\"* in the article.\n\nThese are just a few examples and there are a total of 69 articles with similar issues (including some that the label is only found within a url). I think this is a huge problem. The only thing I can think of is to remove these articles for now.\n\nHowever, this also raises a concern for me with the LB datasets. It is unclear to me how this labeling is done. If there are going to be labels that are not in the cleaned text of the articles, its going to be near impossible for a model to be able to consistently predict them. This is particularly an issue when two phrases such as *\"slosh model\"* and *\"slosh models\"* have a jaccard score of 0.33, and won't be counted as a match.\n",
      "votes": 9
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1289522": "I was investigating a problem in my preprocessing and I found an issue in the training data.\n\nMany of the articles contain labels (cleaned) that are not actually mentioned in the text (cleaned).\n\nFor example, the article with Id *\"14592eb8-a06e-4b26-89c3-939a9ce28698\"* has label *\"ibtracs\"*, however, in the text, the dataset is referred to as *\"ibtracks\"*.\n\nAs another example, article *\"6fa563ab-1ced-40a3-a626-103306098092\"* has label *\"beginning postsecondary student\"*, but this exact phrase cannot be found in the cleaned text of the article. You can only find the phrase *\"beginning postsecondary student**s**\"*, which is another label for this article.\n\nAnother article with Id *\"f70f268a-8653-47ae-a4bc-c41388ba9a7c\"* has label *\"slosh model\"*, but you can only find the text *\"slosh model**s**\"* in the article.\n\nThese are just a few examples and there are a total of 69 articles with similar issues (including some that the label is only found within a url). I think this is a huge problem. The only thing I can think of is to remove these articles for now.\n\nHowever, this also raises a concern for me with the LB datasets. It is unclear to me how this labeling is done. If there are going to be labels that are not in the cleaned text of the articles, its going to be near impossible for a model to be able to consistently predict them. This is particularly an issue when two phrases such as *\"slosh model\"* and *\"slosh models\"* have a jaccard score of 0.33, and won't be counted as a match.\n"
  }
}