{
  "id": 229238,
  "title": "Isn't it the case that each dataset_title MUST be found in the corresponding pub text? It seems it's not!",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/229238",
  "author_name": "",
  "post_date": "2021-03-29T06:28:48.624993Z",
  "votes": 2,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I was checking <a href=\"https://www.kaggle.com/manabendrarout\" target=\"_blank\">@manabendrarout</a> notebook where a dataframe of pub texts and the datatset_title is created. Then by simply checking if the dataset_title can be found in the text, I found that only for 14640 out of 19661 rows this is the case. For other cases, there are lowercase/uppercase mismatches which leads to not finding the exact dataset_title in the corresponding text; for example:</p>\n<p>dataset_title: <code>National Education Longitudinal Study</code><br>\nwhat I found in the text: <code>National Education Longitudinal study</code> (\"study\" is with lowercase s)</p>\n<p>I personally expected that the whole point in providing datatset_title was to make it easy to build dataset pipelines. But now we need to check for those (about 5000 cases) and find the most similar part from the text.</p>",
  "messages": [
    {
      "id": "1255734",
      "postDate": "03/29/2021 06:28:48",
      "content": "<p>I was checking <a href=\"https://www.kaggle.com/manabendrarout\" target=\"_blank\">@manabendrarout</a> notebook where a dataframe of pub texts and the datatset_title is created. Then by simply checking if the dataset_title can be found in the text, I found that only for 14640 out of 19661 rows this is the case. For other cases, there are lowercase/uppercase mismatches which leads to not finding the exact dataset_title in the corresponding text; for example:</p>\n<p>dataset_title: <code>National Education Longitudinal Study</code><br>\nwhat I found in the text: <code>National Education Longitudinal study</code> (\"study\" is with lowercase s)</p>\n<p>I personally expected that the whole point in providing datatset_title was to make it easy to build dataset pipelines. But now we need to check for those (about 5000 cases) and find the most similar part from the text.</p>",
      "rawMarkdown": "I was checking @manabendrarout notebook where a dataframe of pub texts and the datatset_title is created. Then by simply checking if the dataset_title can be found in the text, I found that only for 14640 out of 19661 rows this is the case. For other cases, there are lowercase/uppercase mismatches which leads to not finding the exact dataset_title in the corresponding text; for example:\n\ndataset_title: `National Education Longitudinal Study`\nwhat I found in the text: `National Education Longitudinal study` (\"study\" is with lowercase s)\n\nI personally expected that the whole point in providing datatset_title was to make it easy to build dataset pipelines. But now we need to check for those (about 5000 cases) and find the most similar part from the text.",
      "votes": null
    },
    {
      "id": "1255770",
      "postDate": "03/29/2021 07:15:44",
      "content": "<p>Yeah, you are right <a href=\"https://www.kaggle.com/moeinshariatnia\" target=\"_blank\">@moeinshariatnia</a> , but as far as I know Kaggle, it must be intentional. I have seen such cases many a times in my (rather short) journey on Kaggle.<br>\nThe logic being, it makes it difficult to apply off-the-shelf pipelines and actually focus on cleaning/preparing the dataset. And I guess that what makes it interesting. 😊</p>",
      "rawMarkdown": "Yeah, you are right @moeinshariatnia , but as far as I know Kaggle, it must be intentional. I have seen such cases many a times in my (rather short) journey on Kaggle.\nThe logic being, it makes it difficult to apply off-the-shelf pipelines and actually focus on cleaning/preparing the dataset. And I guess that what makes it interesting. 😊",
      "votes": null
    },
    {
      "id": "1255779",
      "postDate": "03/29/2021 07:22:51",
      "content": "<p>Yeah that's probably the case here! Thanks for your great notebook btw.</p>",
      "rawMarkdown": "Yeah that's probably the case here! Thanks for your great notebook btw.",
      "votes": null
    },
    {
      "id": "1255787",
      "postDate": "03/29/2021 07:40:45",
      "content": "<p>Happy to help! 😊<br>\nHope it was worth your time!</p>",
      "rawMarkdown": "Happy to help! 😊\nHope it was worth your time!",
      "votes": null
    },
    {
      "id": "1255799",
      "postDate": "03/29/2021 08:04:01",
      "content": "<p>Sure it was.</p>",
      "rawMarkdown": "Sure it was.",
      "votes": null
    },
    {
      "id": "1256234",
      "postDate": "03/29/2021 17:18:23",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/moeinshariatnia\" target=\"_blank\">@moeinshariatnia</a> - good question. The <code>dataset_title</code> is not necessarily in the text. The <code>cleaned_label</code> should, however, be in the (similarly cleaned) publication text.</p>",
      "rawMarkdown": "Hi @moeinshariatnia - good question. The `dataset_title` is not necessarily in the text. The `cleaned_label` should, however, be in the (similarly cleaned) publication text.",
      "votes": null
    },
    {
      "id": "1256249",
      "postDate": "03/29/2021 17:28:03",
      "content": "<p>I got it. Thanks for your response. </p>",
      "rawMarkdown": "I got it. Thanks for your response.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1255770,
      "author_name": "manabendrarout",
      "author_url": "",
      "post_date": "03/29/2021 07:15:44",
      "content": "<p>Yeah, you are right <a href=\"https://www.kaggle.com/moeinshariatnia\" target=\"_blank\">@moeinshariatnia</a> , but as far as I know Kaggle, it must be intentional. I have seen such cases many a times in my (rather short) journey on Kaggle.<br>\nThe logic being, it makes it difficult to apply off-the-shelf pipelines and actually focus on cleaning/preparing the dataset. And I guess that what makes it interesting. 😊</p>",
      "votes": null,
      "replies": [
        {
          "id": 1255779,
          "author_name": "moeinshariatnia",
          "author_url": "",
          "post_date": "03/29/2021 07:22:51",
          "content": "<p>Yeah that's probably the case here! Thanks for your great notebook btw.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1255787,
          "author_name": "manabendrarout",
          "author_url": "",
          "post_date": "03/29/2021 07:40:45",
          "content": "<p>Happy to help! 😊<br>\nHope it was worth your time!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1255799,
          "author_name": "moeinshariatnia",
          "author_url": "",
          "post_date": "03/29/2021 08:04:01",
          "content": "<p>Sure it was.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1256234,
      "author_name": "philculliton",
      "author_url": "",
      "post_date": "03/29/2021 17:18:23",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/moeinshariatnia\" target=\"_blank\">@moeinshariatnia</a> - good question. The <code>dataset_title</code> is not necessarily in the text. The <code>cleaned_label</code> should, however, be in the (similarly cleaned) publication text.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1256249,
          "author_name": "moeinshariatnia",
          "author_url": "",
          "post_date": "03/29/2021 17:28:03",
          "content": "<p>I got it. Thanks for your response. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1255734": "I was checking @manabendrarout notebook where a dataframe of pub texts and the datatset_title is created. Then by simply checking if the dataset_title can be found in the text, I found that only for 14640 out of 19661 rows this is the case. For other cases, there are lowercase/uppercase mismatches which leads to not finding the exact dataset_title in the corresponding text; for example:\n\ndataset_title: `National Education Longitudinal Study`\nwhat I found in the text: `National Education Longitudinal study` (\"study\" is with lowercase s)\n\nI personally expected that the whole point in providing datatset_title was to make it easy to build dataset pipelines. But now we need to check for those (about 5000 cases) and find the most similar part from the text.",
    "1255770": "Yeah, you are right @moeinshariatnia , but as far as I know Kaggle, it must be intentional. I have seen such cases many a times in my (rather short) journey on Kaggle.\nThe logic being, it makes it difficult to apply off-the-shelf pipelines and actually focus on cleaning/preparing the dataset. And I guess that what makes it interesting. 😊",
    "1255779": "Yeah that's probably the case here! Thanks for your great notebook btw.",
    "1255787": "Happy to help! 😊\nHope it was worth your time!",
    "1255799": "Sure it was.",
    "1256234": "Hi @moeinshariatnia - good question. The `dataset_title` is not necessarily in the text. The `cleaned_label` should, however, be in the (similarly cleaned) publication text.",
    "1256249": "I got it. Thanks for your response."
  },
  "source": "meta"
}