{
  "id": 23098,
  "title": "Any advice on how to handle badly formatted data?",
  "url": "/competitions/bosch-production-line-performance/discussion/23098",
  "author_name": "",
  "post_date": "2016-08-25T19:13:34.817Z",
  "votes": null,
  "comment_count": 4,
  "views": 790,
  "content": "<p>So far I have been trying to somehow parse the CSV files into a useful representation and I've encountered a bunch of strange entries, which isn't exactly amazing given the dataset size, but I'm not sure if it is meant to be part of the competition.</p>\n\n<p>Below is a list of issues that I've noticed so far:</p>\n\n<ol>\n<li>A single dash sign in numerical data (i.e. just <code>-</code> in a column). Occurs in train_date.csv and test_numeric.csv</li>\n<li>What appears to a range, occurs in variations with both single and double dash (e.g. <code>0.092--0.059</code> and <code>0.074-0.52</code>; maybe the double dash is a range dash and a minus sign, but it is hard to tell). Occurs in test_numeric.csv</li>\n<li>Use of a slash in numbers (e.g. <code>-0/021</code>). It seems to follow the 1/3 digits pattern, so might just be a mistyped decimal comma. Occurs in train_date.csv and test_numeric.csv</li>\n</ol>",
  "messages": [
    {
      "id": "132462",
      "postDate": "08/25/2016 19:13:34",
      "content": "<p>So far I have been trying to somehow parse the CSV files into a useful representation and I've encountered a bunch of strange entries, which isn't exactly amazing given the dataset size, but I'm not sure if it is meant to be part of the competition.</p>\n\n<p>Below is a list of issues that I've noticed so far:</p>\n\n<ol>\n<li>A single dash sign in numerical data (i.e. just <code>-</code> in a column). Occurs in train_date.csv and test_numeric.csv</li>\n<li>What appears to a range, occurs in variations with both single and double dash (e.g. <code>0.092--0.059</code> and <code>0.074-0.52</code>; maybe the double dash is a range dash and a minus sign, but it is hard to tell). Occurs in test_numeric.csv</li>\n<li>Use of a slash in numbers (e.g. <code>-0/021</code>). It seems to follow the 1/3 digits pattern, so might just be a mistyped decimal comma. Occurs in train_date.csv and test_numeric.csv</li>\n</ol>",
      "rawMarkdown": "So far I have been trying to somehow parse the CSV files into a useful representation and I've encountered a bunch of strange entries, which isn't exactly amazing given the dataset size, but I'm not sure if it is meant to be part of the competition.\r\n\r\nBelow is a list of issues that I've noticed so far:\r\n\r\n1. A single dash sign in numerical data (i.e. just `-` in a column). Occurs in train_date.csv and test_numeric.csv\r\n2. What appears to a range, occurs in variations with both single and double dash (e.g. `0.092--0.059` and `0.074-0.52`; maybe the double dash is a range dash and a minus sign, but it is hard to tell). Occurs in test_numeric.csv\r\n3. Use of a slash in numbers (e.g. `-0/021`). It seems to follow the 1/3 digits pattern, so might just be a mistyped decimal comma. Occurs in train_date.csv and test_numeric.csv",
      "votes": null
    },
    {
      "id": "132474",
      "postDate": "08/25/2016 20:44:46",
      "content": "<p>I guess it'd depend on how many there are.  And whether any of them are associated with failures.  If the numbers is in the 10s or low 100's and none of them have a positive response, I'd consider treating them all as &quot;missing&quot; values and do a simple imputation for both training and test.</p>\n\n<p>I'll have to dig through the data a bit tonight myself.</p>",
      "rawMarkdown": "I guess it'd depend on how many there are.  And whether any of them are associated with failures.  If the numbers is in the 10s or low 100's and none of them have a positive response, I'd consider treating them all as \"missing\" values and do a simple imputation for both training and test.\r\n\r\nI'll have to dig through the data a bit tonight myself.",
      "votes": null
    },
    {
      "id": "132476",
      "postDate": "08/25/2016 21:15:40",
      "content": "<p>I can't find these errors. What are md5s for your files?</p>\n\n<p>737ce2d54a37d9a9083826ed5cd7de93  sample_submission.csv <br>\n009a03b7e5a8b3e01c380a83ad2d27c4  test_categorical.csv <br>\n1d1e2efff7dca5514fd8b0b2dd153df8  test_date.csv <br>\nca3bafaf897942973f4196acc31155a2  test_numeric.csv <br>\nde6812acb0d21279d4dbbd16b4805b80  train_categorical.csv <br>\n00a53bf30678d6b3560a36d135a9c962  train_date.csv <br>\n3a2cdb74bc57fa9d251faf81ee210128  train_numeric.csv  </p>\n\n<p>de5e6f7bed2c4f6cb86b42ebb8d3c2b6  sample_submission.csv.zip <br>\n2075cf807f136d2f741eb9585c0d66b6  test_categorical.csv.zip <br>\nfce334a5b778687efbeb04640da9755f  test_date.csv.zip <br>\n42def2f015f2bc0ebe3f5d959d3813b3  test_numeric.csv.zip <br>\n50648b808f506da58ad1e85e46085f1c  train_categorical.csv.zip <br>\n7230d062badf42d4f1b904746a9b2a16  train_date.csv.zip <br>\n543fb3d19eb4d5ae0c0e4daeff3075fe  train_numeric.csv.zip</p>",
      "rawMarkdown": "I can't find these errors. What are md5s for your files?\r\n\r\n737ce2d54a37d9a9083826ed5cd7de93  sample_submission.csv  \r\n009a03b7e5a8b3e01c380a83ad2d27c4  test_categorical.csv  \r\n1d1e2efff7dca5514fd8b0b2dd153df8  test_date.csv  \r\nca3bafaf897942973f4196acc31155a2  test_numeric.csv  \r\nde6812acb0d21279d4dbbd16b4805b80  train_categorical.csv  \r\n00a53bf30678d6b3560a36d135a9c962  train_date.csv  \r\n3a2cdb74bc57fa9d251faf81ee210128  train_numeric.csv  \r\n\r\nde5e6f7bed2c4f6cb86b42ebb8d3c2b6  sample_submission.csv.zip  \r\n2075cf807f136d2f741eb9585c0d66b6  test_categorical.csv.zip  \r\nfce334a5b778687efbeb04640da9755f  test_date.csv.zip  \r\n42def2f015f2bc0ebe3f5d959d3813b3  test_numeric.csv.zip  \r\n50648b808f506da58ad1e85e46085f1c  train_categorical.csv.zip  \r\n7230d062badf42d4f1b904746a9b2a16  train_date.csv.zip  \r\n543fb3d19eb4d5ae0c0e4daeff3075fe  train_numeric.csv.zip",
      "votes": null
    },
    {
      "id": "132493",
      "postDate": "08/26/2016 02:15:10",
      "content": "<p>also what is the significance of negative number for (PartId, Station) combination. I presume they are processing times (train_numeric.csv) , if so can not understand what the negative number indicates.</p>",
      "rawMarkdown": "also what is the significance of negative number for (PartId, Station) combination. I presume they are processing times (train_numeric.csv) , if so can not understand what the negative number indicates.",
      "votes": null
    },
    {
      "id": "132503",
      "postDate": "08/26/2016 04:45:47",
      "content": "<p>OK, it seems the problem is entirely with me. The ZIP files I have matched the ones posted by pawzaw above and when I extracted them I also got matching CSV files. However for whatever reason the files I was working with before did not match the contents of the zip files... Maybe I need a new hard drive. :)</p>\n\n<p>Update: lots of places simply had the lowest bit in a byte flipped... No idea what is going on, but certainly not a problem with the dataset...</p>",
      "rawMarkdown": "OK, it seems the problem is entirely with me. The ZIP files I have matched the ones posted by pawzaw above and when I extracted them I also got matching CSV files. However for whatever reason the files I was working with before did not match the contents of the zip files... Maybe I need a new hard drive. :)\r\n\r\n\r\nUpdate: lots of places simply had the lowest bit in a byte flipped... No idea what is going on, but certainly not a problem with the dataset...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 132474,
      "author_name": "taffeylewis",
      "author_url": "",
      "post_date": "08/25/2016 20:44:46",
      "content": "<p>I guess it'd depend on how many there are.  And whether any of them are associated with failures.  If the numbers is in the 10s or low 100's and none of them have a positive response, I'd consider treating them all as &quot;missing&quot; values and do a simple imputation for both training and test.</p>\n\n<p>I'll have to dig through the data a bit tonight myself.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132476,
      "author_name": "pzawal",
      "author_url": "",
      "post_date": "08/25/2016 21:15:40",
      "content": "<p>I can't find these errors. What are md5s for your files?</p>\n\n<p>737ce2d54a37d9a9083826ed5cd7de93  sample_submission.csv <br>\n009a03b7e5a8b3e01c380a83ad2d27c4  test_categorical.csv <br>\n1d1e2efff7dca5514fd8b0b2dd153df8  test_date.csv <br>\nca3bafaf897942973f4196acc31155a2  test_numeric.csv <br>\nde6812acb0d21279d4dbbd16b4805b80  train_categorical.csv <br>\n00a53bf30678d6b3560a36d135a9c962  train_date.csv <br>\n3a2cdb74bc57fa9d251faf81ee210128  train_numeric.csv  </p>\n\n<p>de5e6f7bed2c4f6cb86b42ebb8d3c2b6  sample_submission.csv.zip <br>\n2075cf807f136d2f741eb9585c0d66b6  test_categorical.csv.zip <br>\nfce334a5b778687efbeb04640da9755f  test_date.csv.zip <br>\n42def2f015f2bc0ebe3f5d959d3813b3  test_numeric.csv.zip <br>\n50648b808f506da58ad1e85e46085f1c  train_categorical.csv.zip <br>\n7230d062badf42d4f1b904746a9b2a16  train_date.csv.zip <br>\n543fb3d19eb4d5ae0c0e4daeff3075fe  train_numeric.csv.zip</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132493,
      "author_name": "akumthek",
      "author_url": "",
      "post_date": "08/26/2016 02:15:10",
      "content": "<p>also what is the significance of negative number for (PartId, Station) combination. I presume they are processing times (train_numeric.csv) , if so can not understand what the negative number indicates.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132503,
      "author_name": "rdokov",
      "author_url": "",
      "post_date": "08/26/2016 04:45:47",
      "content": "<p>OK, it seems the problem is entirely with me. The ZIP files I have matched the ones posted by pawzaw above and when I extracted them I also got matching CSV files. However for whatever reason the files I was working with before did not match the contents of the zip files... Maybe I need a new hard drive. :)</p>\n\n<p>Update: lots of places simply had the lowest bit in a byte flipped... No idea what is going on, but certainly not a problem with the dataset...</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "132462": "So far I have been trying to somehow parse the CSV files into a useful representation and I've encountered a bunch of strange entries, which isn't exactly amazing given the dataset size, but I'm not sure if it is meant to be part of the competition.\r\n\r\nBelow is a list of issues that I've noticed so far:\r\n\r\n1. A single dash sign in numerical data (i.e. just `-` in a column). Occurs in train_date.csv and test_numeric.csv\r\n2. What appears to a range, occurs in variations with both single and double dash (e.g. `0.092--0.059` and `0.074-0.52`; maybe the double dash is a range dash and a minus sign, but it is hard to tell). Occurs in test_numeric.csv\r\n3. Use of a slash in numbers (e.g. `-0/021`). It seems to follow the 1/3 digits pattern, so might just be a mistyped decimal comma. Occurs in train_date.csv and test_numeric.csv",
    "132474": "I guess it'd depend on how many there are.  And whether any of them are associated with failures.  If the numbers is in the 10s or low 100's and none of them have a positive response, I'd consider treating them all as \"missing\" values and do a simple imputation for both training and test.\r\n\r\nI'll have to dig through the data a bit tonight myself.",
    "132476": "I can't find these errors. What are md5s for your files?\r\n\r\n737ce2d54a37d9a9083826ed5cd7de93  sample_submission.csv  \r\n009a03b7e5a8b3e01c380a83ad2d27c4  test_categorical.csv  \r\n1d1e2efff7dca5514fd8b0b2dd153df8  test_date.csv  \r\nca3bafaf897942973f4196acc31155a2  test_numeric.csv  \r\nde6812acb0d21279d4dbbd16b4805b80  train_categorical.csv  \r\n00a53bf30678d6b3560a36d135a9c962  train_date.csv  \r\n3a2cdb74bc57fa9d251faf81ee210128  train_numeric.csv  \r\n\r\nde5e6f7bed2c4f6cb86b42ebb8d3c2b6  sample_submission.csv.zip  \r\n2075cf807f136d2f741eb9585c0d66b6  test_categorical.csv.zip  \r\nfce334a5b778687efbeb04640da9755f  test_date.csv.zip  \r\n42def2f015f2bc0ebe3f5d959d3813b3  test_numeric.csv.zip  \r\n50648b808f506da58ad1e85e46085f1c  train_categorical.csv.zip  \r\n7230d062badf42d4f1b904746a9b2a16  train_date.csv.zip  \r\n543fb3d19eb4d5ae0c0e4daeff3075fe  train_numeric.csv.zip",
    "132493": "also what is the significance of negative number for (PartId, Station) combination. I presume they are processing times (train_numeric.csv) , if so can not understand what the negative number indicates.",
    "132503": "OK, it seems the problem is entirely with me. The ZIP files I have matched the ones posted by pawzaw above and when I extracted them I also got matching CSV files. However for whatever reason the files I was working with before did not match the contents of the zip files... Maybe I need a new hard drive. :)\r\n\r\n\r\nUpdate: lots of places simply had the lowest bit in a byte flipped... No idea what is going on, but certainly not a problem with the dataset..."
  },
  "source": "meta"
}