{
  "id": 24054,
  "title": "Training data has different column than test data?",
  "url": "/competitions/bosch-production-line-performance/discussion/24054",
  "author_name": "",
  "post_date": "2016-10-03T15:35:29.103Z",
  "votes": null,
  "comment_count": 8,
  "views": 579,
  "content": "<p>Has anyone noticed that the training numerical data has several different columns than the test numerical data? What's the solution for this? Just delete the ones in the training set?</p>",
  "messages": [
    {
      "id": "137637",
      "postDate": "10/03/2016 15:35:29",
      "content": "<p>Has anyone noticed that the training numerical data has several different columns than the test numerical data? What's the solution for this? Just delete the ones in the training set?</p>",
      "rawMarkdown": "Has anyone noticed that the training numerical data has several different columns than the test numerical data? What's the solution for this? Just delete the ones in the training set?",
      "votes": null
    },
    {
      "id": "137644",
      "postDate": "10/03/2016 16:21:32",
      "content": "<p>i guess you are talking about the &quot;Response&quot; column. The rest should be the same.</p>",
      "rawMarkdown": "i guess you are talking about the \"Response\" column. The rest should be the same.",
      "votes": null
    },
    {
      "id": "137647",
      "postDate": "10/03/2016 16:28:09",
      "content": "<blockquote>\n  <p>Has anyone noticed that the training numerical data has several different columns than the test numerical data?</p>\n</blockquote>\n\n<p>The only difference is in the column we are trying to predict. If you are running a Unix/Linux OS, type the following two commands:</p>\n\n<pre><code>head -1 train_numeric.csv | tr &quot;\\,&quot; &quot;\\n&quot; | wc -l\nhead -1 test_numeric.csv | tr &quot;\\,&quot; &quot;\\n&quot; | wc -l\n</code></pre>\n\n<p>The output will tell you the number of columns in each file, which in this case is 970 and 969.</p>",
      "rawMarkdown": "> Has anyone noticed that the training numerical data has several different columns than the test numerical data?\r\n\r\nThe only difference is in the column we are trying to predict. If you are running a Unix/Linux OS, type the following two commands:\r\n\r\n    head -1 train_numeric.csv | tr \"\\,\" \"\\n\" | wc -l\r\n    head -1 test_numeric.csv | tr \"\\,\" \"\\n\" | wc -l\r\n\r\nThe output will tell you the number of columns in each file, which in this case is 970 and 969.",
      "votes": null
    },
    {
      "id": "137648",
      "postDate": "10/03/2016 16:31:48",
      "content": "<p>Thanks guys. I realized I worded this really poorly. What I'm actually seeing now is this:</p>\n\n<p>If I concatenate the numerical, categorical, and date data for the training set, I loop through the categorical column and create dummy variables (this is in Python). I then do the same for the test data, but find that resulting training and test data differ by almost 200 columns. I'm trying to track down where this is coming from. Wasn't sure if anyone else was seeing this as well. It's probably a problem on my end, but I can't figure out where the difference is coming from.</p>",
      "rawMarkdown": "Thanks guys. I realized I worded this really poorly. What I'm actually seeing now is this:\r\n\r\nIf I concatenate the numerical, categorical, and date data for the training set, I loop through the categorical column and create dummy variables (this is in Python). I then do the same for the test data, but find that resulting training and test data differ by almost 200 columns. I'm trying to track down where this is coming from. Wasn't sure if anyone else was seeing this as well. It's probably a problem on my end, but I can't figure out where the difference is coming from.",
      "votes": null
    },
    {
      "id": "137649",
      "postDate": "10/03/2016 16:37:03",
      "content": "<p>[quote=Kyle DeGrave;137648]</p>\n\n<p>If I concatenate the numerical, categorical, and date data for the training set, I loop through the categorical column and create dummy variables (this is in Python).</p>\n\n<p>[/quote]</p>\n\n<p>This most likely means that there is a different number of unique variables in train and test datasets, which sometimes happens. &quot;Dummyfying&quot; variables on train and test datasets individually will give you different column numbers.</p>\n\n<p>I suggest you concatenate train and test before dummyfying, do the transformation, and then split them back into train and test. </p>",
      "rawMarkdown": "[quote=Kyle DeGrave;137648]\r\n\r\nIf I concatenate the numerical, categorical, and date data for the training set, I loop through the categorical column and create dummy variables (this is in Python).\r\n\r\n[/quote]\r\n\r\nThis most likely means that there is a different number of unique variables in train and test datasets, which sometimes happens. \"Dummyfying\" variables on train and test datasets individually will give you different column numbers.\r\n\r\nI suggest you concatenate train and test before dummyfying, do the transformation, and then split them back into train and test.",
      "votes": null
    },
    {
      "id": "266967",
      "postDate": "01/10/2018 08:19:34",
      "content": "<p>Hi，I wondered if you still have the dataset of 'Bosch Production Line Performance' and if you do have ,can you send it to me for this dataset can't be downloaded now.My email address is xujie0822@sjtu.edu.cn. Thank you very much.</p>",
      "rawMarkdown": "Hi，I wondered if you still have the dataset of 'Bosch Production Line Performance' and if you do have ,can you send it to me for this dataset can't be downloaded now.My email address is xujie0822@sjtu.edu.cn. Thank you very much.",
      "votes": null
    },
    {
      "id": "268219",
      "postDate": "01/13/2018 20:14:36",
      "content": "<p>Hi,\nthe data is still downloadable. Is it still possible to submit one's result ? Just for learning purposes?</p>",
      "rawMarkdown": "Hi,\nthe data is still downloadable. Is it still possible to submit one's result ? Just for learning purposes?",
      "votes": null
    },
    {
      "id": "268253",
      "postDate": "01/13/2018 23:16:38",
      "content": "<p>Hi coolroland -</p>\n\n<p>For most contests, you can submit after the competition close, and you will see the score and where it would have placed on the leader board. (Your score won't actually appear on the leader board, though.)</p>",
      "rawMarkdown": "Hi coolroland -\n\nFor most contests, you can submit after the competition close, and you will see the score and where it would have placed on the leader board. (Your score won't actually appear on the leader board, though.)",
      "votes": null
    },
    {
      "id": "268709",
      "postDate": "01/15/2018 10:51:05",
      "content": "<p>Hi, thanks for you answer. \nIs it normal that there are many many NULL values, and if yes: why is it?\nIs there some explanation of the data or are all these attributes just anonymous w/o any description ?</p>\n\n<p>Best regards,\nrolli</p>",
      "rawMarkdown": "Hi, thanks for you answer. \nIs it normal that there are many many NULL values, and if yes: why is it?\nIs there some explanation of the data or are all these attributes just anonymous w/o any description ?\n\nBest regards,\nrolli",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 137644,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "10/03/2016 16:21:32",
      "content": "<p>i guess you are talking about the &quot;Response&quot; column. The rest should be the same.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137647,
      "author_name": "tilii7",
      "author_url": "",
      "post_date": "10/03/2016 16:28:09",
      "content": "<blockquote>\n  <p>Has anyone noticed that the training numerical data has several different columns than the test numerical data?</p>\n</blockquote>\n\n<p>The only difference is in the column we are trying to predict. If you are running a Unix/Linux OS, type the following two commands:</p>\n\n<pre><code>head -1 train_numeric.csv | tr &quot;\\,&quot; &quot;\\n&quot; | wc -l\nhead -1 test_numeric.csv | tr &quot;\\,&quot; &quot;\\n&quot; | wc -l\n</code></pre>\n\n<p>The output will tell you the number of columns in each file, which in this case is 970 and 969.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137648,
      "author_name": "degravek",
      "author_url": "",
      "post_date": "10/03/2016 16:31:48",
      "content": "<p>Thanks guys. I realized I worded this really poorly. What I'm actually seeing now is this:</p>\n\n<p>If I concatenate the numerical, categorical, and date data for the training set, I loop through the categorical column and create dummy variables (this is in Python). I then do the same for the test data, but find that resulting training and test data differ by almost 200 columns. I'm trying to track down where this is coming from. Wasn't sure if anyone else was seeing this as well. It's probably a problem on my end, but I can't figure out where the difference is coming from.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137649,
      "author_name": "tilii7",
      "author_url": "",
      "post_date": "10/03/2016 16:37:03",
      "content": "<p>[quote=Kyle DeGrave;137648]</p>\n\n<p>If I concatenate the numerical, categorical, and date data for the training set, I loop through the categorical column and create dummy variables (this is in Python).</p>\n\n<p>[/quote]</p>\n\n<p>This most likely means that there is a different number of unique variables in train and test datasets, which sometimes happens. &quot;Dummyfying&quot; variables on train and test datasets individually will give you different column numbers.</p>\n\n<p>I suggest you concatenate train and test before dummyfying, do the transformation, and then split them back into train and test. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 266967,
      "author_name": "xujie0822",
      "author_url": "",
      "post_date": "01/10/2018 08:19:34",
      "content": "<p>Hi，I wondered if you still have the dataset of 'Bosch Production Line Performance' and if you do have ,can you send it to me for this dataset can't be downloaded now.My email address is xujie0822@sjtu.edu.cn. Thank you very much.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 268219,
      "author_name": "coolroland",
      "author_url": "",
      "post_date": "01/13/2018 20:14:36",
      "content": "<p>Hi,\nthe data is still downloadable. Is it still possible to submit one's result ? Just for learning purposes?</p>",
      "votes": null,
      "replies": [
        {
          "id": 268253,
          "author_name": "inversion",
          "author_url": "",
          "post_date": "01/13/2018 23:16:38",
          "content": "<p>Hi coolroland -</p>\n\n<p>For most contests, you can submit after the competition close, and you will see the score and where it would have placed on the leader board. (Your score won't actually appear on the leader board, though.)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 268709,
      "author_name": "coolroland",
      "author_url": "",
      "post_date": "01/15/2018 10:51:05",
      "content": "<p>Hi, thanks for you answer. \nIs it normal that there are many many NULL values, and if yes: why is it?\nIs there some explanation of the data or are all these attributes just anonymous w/o any description ?</p>\n\n<p>Best regards,\nrolli</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "137637": "Has anyone noticed that the training numerical data has several different columns than the test numerical data? What's the solution for this? Just delete the ones in the training set?",
    "137644": "i guess you are talking about the \"Response\" column. The rest should be the same.",
    "137647": "> Has anyone noticed that the training numerical data has several different columns than the test numerical data?\r\n\r\nThe only difference is in the column we are trying to predict. If you are running a Unix/Linux OS, type the following two commands:\r\n\r\n    head -1 train_numeric.csv | tr \"\\,\" \"\\n\" | wc -l\r\n    head -1 test_numeric.csv | tr \"\\,\" \"\\n\" | wc -l\r\n\r\nThe output will tell you the number of columns in each file, which in this case is 970 and 969.",
    "137648": "Thanks guys. I realized I worded this really poorly. What I'm actually seeing now is this:\r\n\r\nIf I concatenate the numerical, categorical, and date data for the training set, I loop through the categorical column and create dummy variables (this is in Python). I then do the same for the test data, but find that resulting training and test data differ by almost 200 columns. I'm trying to track down where this is coming from. Wasn't sure if anyone else was seeing this as well. It's probably a problem on my end, but I can't figure out where the difference is coming from.",
    "137649": "[quote=Kyle DeGrave;137648]\r\n\r\nIf I concatenate the numerical, categorical, and date data for the training set, I loop through the categorical column and create dummy variables (this is in Python).\r\n\r\n[/quote]\r\n\r\nThis most likely means that there is a different number of unique variables in train and test datasets, which sometimes happens. \"Dummyfying\" variables on train and test datasets individually will give you different column numbers.\r\n\r\nI suggest you concatenate train and test before dummyfying, do the transformation, and then split them back into train and test.",
    "266967": "Hi，I wondered if you still have the dataset of 'Bosch Production Line Performance' and if you do have ,can you send it to me for this dataset can't be downloaded now.My email address is xujie0822@sjtu.edu.cn. Thank you very much.",
    "268219": "Hi,\nthe data is still downloadable. Is it still possible to submit one's result ? Just for learning purposes?",
    "268253": "Hi coolroland -\n\nFor most contests, you can submit after the competition close, and you will see the score and where it would have placed on the leader board. (Your score won't actually appear on the leader board, though.)",
    "268709": "Hi, thanks for you answer. \nIs it normal that there are many many NULL values, and if yes: why is it?\nIs there some explanation of the data or are all these attributes just anonymous w/o any description ?\n\nBest regards,\nrolli"
  },
  "source": "meta"
}