{
  "id": 233721,
  "title": "section_title/text issues",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/233721",
  "author_name": "",
  "post_date": "2021-04-20T17:51:46.364405400Z",
  "votes": 5,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I've had a look at the forums to try and see if anyone has mentioned this already but can't see anything - some of the texts/section_titles seem to be mixed up. For example I was just looking at some of the data and found this (from paper id aff7adeb-a6c0-46d6-b10f-ed57a679ce03)</p>\n<pre><code>{'section_title': \"years, credit accumulation diverges in the two kinds of institutions, due in part to community college students' greater involvement in employment, and a higher likelihood of stopping out of college, after controlling for their academic performance. Contrary to some earlier claims, we find that a vocational emphasis in community college is not a major factor behind the disparity. One important mechanism is the widespread loss of credits that occurs after undergraduates transfer from a community college to a 4-year institution; the greater the loss, the lower the chances of completing a BA. However, earlier claims that community college students receive lower aid levels after transfer and that transfers disproportionately fail to survive through the senior year are not supported by our analyses.\",\n  'text': 'college fare in terms of credit accumulation and degree completion. Understanding when and why the disparity in graduation emerges has practical importance when many high school graduates are attracted to the stepping stone route to a BA, and when some state legislatures are seeking to reduce expenditures by emphasizing the cheaper community college route over the more expensive 4-year college path (Kelderman, 2010) .'},\n</code></pre>\n<p>I did a quick search and found 492 section_titles over 200 in length - some of them are very weird, this one for example is an image I think(?):</p>\n<pre><code>{'section_title': 'C C C C C C C C C C S C C S S C C C S C S S C S S C C C C C S S S C C C C S S S C C S C C S S S C S C S S C S C C C C C C C C S C S S C C C S C C C S S S C C C C C C C C C C C S C C C C S S C C C C C C C C C C C C C S C S C S S S C C C C S S S C S S C S C C C C C C C C C C C C S C S C S S C C C C C C S C C C S S S C C C C S C C S S S C S S C C S C C C S S C C S C C C C C C S C C S C C C C S C C S C C C C C C C S S C C C C C C C S C C C C C S C C S C C C C S C S S C S C C C C C C S C C S C C C C C C S C C C C S C C C C S C S C C C C S C C C C S C C S C C C C C S C C C C S S S C C C C S C C C C S C C C C C C C C C C C S S S C S C C C S C S S C C S C C S C C C C C C C C S C C S S C C C C C C C C C C S C C C S C C C C S C S C S C S S C S C C C C C S C C C C S C C S C C C C C C S S C C C C C C',\n  'text': 'Figure 3: A hypothetical farm area with corn-corn-soybean rotation. Soybean acres denoted by letter \"S\" are randomly distributed to 1/3 rd of the planted acres. The crushing plant is assumed to be located at the middle of the field.\\nArgonne National Laboratory (2006) estimated the energy involved in soybean transportation using GREET model. The energy required in transporting soybeans from farms to processing plants was estimated to be 213,927 BTU/ton. The estimation was based on the total distance of 800 miles and the modes of transportation used were barge, truck, and rails. Since this distance is adequately larger to account for much larger than 1000 million gallon/year production plants, the value from GREET model was used for this transportation calculation.'}\n</code></pre>\n<p><em>EDIT</em></p>\n<p>also reading that top one again, it's clear that the <code>text</code> starts part way through a sentence</p>\n<p>I guess this makes the competition a little more interesting, but it's probably not what the hosts intended haha</p>",
  "messages": [
    {
      "id": "1279217",
      "postDate": "04/20/2021 17:51:46",
      "content": "<p>I've had a look at the forums to try and see if anyone has mentioned this already but can't see anything - some of the texts/section_titles seem to be mixed up. For example I was just looking at some of the data and found this (from paper id aff7adeb-a6c0-46d6-b10f-ed57a679ce03)</p>\n<pre><code>{'section_title': \"years, credit accumulation diverges in the two kinds of institutions, due in part to community college students' greater involvement in employment, and a higher likelihood of stopping out of college, after controlling for their academic performance. Contrary to some earlier claims, we find that a vocational emphasis in community college is not a major factor behind the disparity. One important mechanism is the widespread loss of credits that occurs after undergraduates transfer from a community college to a 4-year institution; the greater the loss, the lower the chances of completing a BA. However, earlier claims that community college students receive lower aid levels after transfer and that transfers disproportionately fail to survive through the senior year are not supported by our analyses.\",\n  'text': 'college fare in terms of credit accumulation and degree completion. Understanding when and why the disparity in graduation emerges has practical importance when many high school graduates are attracted to the stepping stone route to a BA, and when some state legislatures are seeking to reduce expenditures by emphasizing the cheaper community college route over the more expensive 4-year college path (Kelderman, 2010) .'},\n</code></pre>\n<p>I did a quick search and found 492 section_titles over 200 in length - some of them are very weird, this one for example is an image I think(?):</p>\n<pre><code>{'section_title': 'C C C C C C C C C C S C C S S C C C S C S S C S S C C C C C S S S C C C C S S S C C S C C S S S C S C S S C S C C C C C C C C S C S S C C C S C C C S S S C C C C C C C C C C C S C C C C S S C C C C C C C C C C C C C S C S C S S S C C C C S S S C S S C S C C C C C C C C C C C C S C S C S S C C C C C C S C C C S S S C C C C S C C S S S C S S C C S C C C S S C C S C C C C C C S C C S C C C C S C C S C C C C C C C S S C C C C C C C S C C C C C S C C S C C C C S C S S C S C C C C C C S C C S C C C C C C S C C C C S C C C C S C S C C C C S C C C C S C C S C C C C C S C C C C S S S C C C C S C C C C S C C C C C C C C C C C S S S C S C C C S C S S C C S C C S C C C C C C C C S C C S S C C C C C C C C C C S C C C S C C C C S C S C S C S S C S C C C C C S C C C C S C C S C C C C C C S S C C C C C C',\n  'text': 'Figure 3: A hypothetical farm area with corn-corn-soybean rotation. Soybean acres denoted by letter \"S\" are randomly distributed to 1/3 rd of the planted acres. The crushing plant is assumed to be located at the middle of the field.\\nArgonne National Laboratory (2006) estimated the energy involved in soybean transportation using GREET model. The energy required in transporting soybeans from farms to processing plants was estimated to be 213,927 BTU/ton. The estimation was based on the total distance of 800 miles and the modes of transportation used were barge, truck, and rails. Since this distance is adequately larger to account for much larger than 1000 million gallon/year production plants, the value from GREET model was used for this transportation calculation.'}\n</code></pre>\n<p><em>EDIT</em></p>\n<p>also reading that top one again, it's clear that the <code>text</code> starts part way through a sentence</p>\n<p>I guess this makes the competition a little more interesting, but it's probably not what the hosts intended haha</p>",
      "rawMarkdown": "I've had a look at the forums to try and see if anyone has mentioned this already but can't see anything - some of the texts/section_titles seem to be mixed up. For example I was just looking at some of the data and found this (from paper id aff7adeb-a6c0-46d6-b10f-ed57a679ce03)\n\n```\n{'section_title': \"years, credit accumulation diverges in the two kinds of institutions, due in part to community college students' greater involvement in employment, and a higher likelihood of stopping out of college, after controlling for their academic performance. Contrary to some earlier claims, we find that a vocational emphasis in community college is not a major factor behind the disparity. One important mechanism is the widespread loss of credits that occurs after undergraduates transfer from a community college to a 4-year institution; the greater the loss, the lower the chances of completing a BA. However, earlier claims that community college students receive lower aid levels after transfer and that transfers disproportionately fail to survive through the senior year are not supported by our analyses.\",\n  'text': 'college fare in terms of credit accumulation and degree completion. Understanding when and why the disparity in graduation emerges has practical importance when many high school graduates are attracted to the stepping stone route to a BA, and when some state legislatures are seeking to reduce expenditures by emphasizing the cheaper community college route over the more expensive 4-year college path (Kelderman, 2010) .'},\n```\n\nI did a quick search and found 492 section_titles over 200 in length - some of them are very weird, this one for example is an image I think(?):\n\n```\n{'section_title': 'C C C C C C C C C C S C C S S C C C S C S S C S S C C C C C S S S C C C C S S S C C S C C S S S C S C S S C S C C C C C C C C S C S S C C C S C C C S S S C C C C C C C C C C C S C C C C S S C C C C C C C C C C C C C S C S C S S S C C C C S S S C S S C S C C C C C C C C C C C C S C S C S S C C C C C C S C C C S S S C C C C S C C S S S C S S C C S C C C S S C C S C C C C C C S C C S C C C C S C C S C C C C C C C S S C C C C C C C S C C C C C S C C S C C C C S C S S C S C C C C C C S C C S C C C C C C S C C C C S C C C C S C S C C C C S C C C C S C C S C C C C C S C C C C S S S C C C C S C C C C S C C C C C C C C C C C S S S C S C C C S C S S C C S C C S C C C C C C C C S C C S S C C C C C C C C C C S C C C S C C C C S C S C S C S S C S C C C C C S C C C C S C C S C C C C C C S S C C C C C C',\n  'text': 'Figure 3: A hypothetical farm area with corn-corn-soybean rotation. Soybean acres denoted by letter \"S\" are randomly distributed to 1/3 rd of the planted acres. The crushing plant is assumed to be located at the middle of the field.\\nArgonne National Laboratory (2006) estimated the energy involved in soybean transportation using GREET model. The energy required in transporting soybeans from farms to processing plants was estimated to be 213,927 BTU/ton. The estimation was based on the total distance of 800 miles and the modes of transportation used were barge, truck, and rails. Since this distance is adequately larger to account for much larger than 1000 million gallon/year production plants, the value from GREET model was used for this transportation calculation.'}\n```\n\n*EDIT*\n\nalso reading that top one again, it's clear that the `text` starts part way through a sentence\n\nI guess this makes the competition a little more interesting, but it's probably not what the hosts intended haha",
      "votes": null
    },
    {
      "id": "1279437",
      "postDate": "04/20/2021 22:55:44",
      "content": "<p>I also noticed it. I like your second example :-) Probably we need to also consider the text in the section titles in order not to miss some datasets.</p>",
      "rawMarkdown": "I also noticed it. I like your second example :-) Probably we need to also consider the text in the section titles in order not to miss some datasets.",
      "votes": null
    },
    {
      "id": "1279635",
      "postDate": "04/21/2021 05:52:34",
      "content": "<p>Yeah think you're right but it changes how I was planning to go about it :)</p>\n<p>The fact that a sentence starts part way through is also concerning for me - that could happen for a few different reasons</p>",
      "rawMarkdown": "Yeah think you're right but it changes how I was planning to go about it :)\n\nThe fact that a sentence starts part way through is also concerning for me - that could happen for a few different reasons",
      "votes": null
    },
    {
      "id": "1279890",
      "postDate": "04/21/2021 10:24:16",
      "content": "<p>You are right. The second point is a problem. This seems to be another facet of noise next to the other noise present in the training data..</p>",
      "rawMarkdown": "You are right. The second point is a problem. This seems to be another facet of noise next to the other noise present in the training data..",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1279437,
      "author_name": "hannes82",
      "author_url": "",
      "post_date": "04/20/2021 22:55:44",
      "content": "<p>I also noticed it. I like your second example :-) Probably we need to also consider the text in the section titles in order not to miss some datasets.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1279635,
          "author_name": "hamishdickson",
          "author_url": "",
          "post_date": "04/21/2021 05:52:34",
          "content": "<p>Yeah think you're right but it changes how I was planning to go about it :)</p>\n<p>The fact that a sentence starts part way through is also concerning for me - that could happen for a few different reasons</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1279890,
          "author_name": "hannes82",
          "author_url": "",
          "post_date": "04/21/2021 10:24:16",
          "content": "<p>You are right. The second point is a problem. This seems to be another facet of noise next to the other noise present in the training data..</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1279217": "I've had a look at the forums to try and see if anyone has mentioned this already but can't see anything - some of the texts/section_titles seem to be mixed up. For example I was just looking at some of the data and found this (from paper id aff7adeb-a6c0-46d6-b10f-ed57a679ce03)\n\n```\n{'section_title': \"years, credit accumulation diverges in the two kinds of institutions, due in part to community college students' greater involvement in employment, and a higher likelihood of stopping out of college, after controlling for their academic performance. Contrary to some earlier claims, we find that a vocational emphasis in community college is not a major factor behind the disparity. One important mechanism is the widespread loss of credits that occurs after undergraduates transfer from a community college to a 4-year institution; the greater the loss, the lower the chances of completing a BA. However, earlier claims that community college students receive lower aid levels after transfer and that transfers disproportionately fail to survive through the senior year are not supported by our analyses.\",\n  'text': 'college fare in terms of credit accumulation and degree completion. Understanding when and why the disparity in graduation emerges has practical importance when many high school graduates are attracted to the stepping stone route to a BA, and when some state legislatures are seeking to reduce expenditures by emphasizing the cheaper community college route over the more expensive 4-year college path (Kelderman, 2010) .'},\n```\n\nI did a quick search and found 492 section_titles over 200 in length - some of them are very weird, this one for example is an image I think(?):\n\n```\n{'section_title': 'C C C C C C C C C C S C C S S C C C S C S S C S S C C C C C S S S C C C C S S S C C S C C S S S C S C S S C S C C C C C C C C S C S S C C C S C C C S S S C C C C C C C C C C C S C C C C S S C C C C C C C C C C C C C S C S C S S S C C C C S S S C S S C S C C C C C C C C C C C C S C S C S S C C C C C C S C C C S S S C C C C S C C S S S C S S C C S C C C S S C C S C C C C C C S C C S C C C C S C C S C C C C C C C S S C C C C C C C S C C C C C S C C S C C C C S C S S C S C C C C C C S C C S C C C C C C S C C C C S C C C C S C S C C C C S C C C C S C C S C C C C C S C C C C S S S C C C C S C C C C S C C C C C C C C C C C S S S C S C C C S C S S C C S C C S C C C C C C C C S C C S S C C C C C C C C C C S C C C S C C C C S C S C S C S S C S C C C C C S C C C C S C C S C C C C C C S S C C C C C C',\n  'text': 'Figure 3: A hypothetical farm area with corn-corn-soybean rotation. Soybean acres denoted by letter \"S\" are randomly distributed to 1/3 rd of the planted acres. The crushing plant is assumed to be located at the middle of the field.\\nArgonne National Laboratory (2006) estimated the energy involved in soybean transportation using GREET model. The energy required in transporting soybeans from farms to processing plants was estimated to be 213,927 BTU/ton. The estimation was based on the total distance of 800 miles and the modes of transportation used were barge, truck, and rails. Since this distance is adequately larger to account for much larger than 1000 million gallon/year production plants, the value from GREET model was used for this transportation calculation.'}\n```\n\n*EDIT*\n\nalso reading that top one again, it's clear that the `text` starts part way through a sentence\n\nI guess this makes the competition a little more interesting, but it's probably not what the hosts intended haha",
    "1279437": "I also noticed it. I like your second example :-) Probably we need to also consider the text in the section titles in order not to miss some datasets.",
    "1279635": "Yeah think you're right but it changes how I was planning to go about it :)\n\nThe fact that a sentence starts part way through is also concerning for me - that could happen for a few different reasons",
    "1279890": "You are right. The second point is a problem. This seems to be another facet of noise next to the other noise present in the training data.."
  },
  "source": "meta"
}