{
  "id": 21982,
  "title": "trainset and testset - not matching fields",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/21982",
  "author_name": "",
  "post_date": "2016-06-30T20:00:24.853Z",
  "votes": null,
  "comment_count": 6,
  "views": 904,
  "content": "<p>these two should have the same fields? why do they not? </p>\n\n<p><a href=\"https://www.quora.com/If-the-training-dataset-has-more-variables-than-the-test-dataset-what-does-one-do\">https://www.quora.com/If-the-training-dataset-has-more-variables-than-the-test-dataset-what-does-one-do</a></p>",
  "messages": [
    {
      "id": "125604",
      "postDate": "06/30/2016 20:00:24",
      "content": "<p>these two should have the same fields? why do they not? </p>\n\n<p><a href=\"https://www.quora.com/If-the-training-dataset-has-more-variables-than-the-test-dataset-what-does-one-do\">https://www.quora.com/If-the-training-dataset-has-more-variables-than-the-test-dataset-what-does-one-do</a></p>",
      "rawMarkdown": "these two should have the same fields? why do they not? \r\n\r\nhttps://www.quora.com/If-the-training-dataset-has-more-variables-than-the-test-dataset-what-does-one-do",
      "votes": null
    },
    {
      "id": "125605",
      "postDate": "06/30/2016 20:04:21",
      "content": "<p>Your situation sounds a bit strange though as you're saying you know your test set won't have entire features? Then, do not include those features in your training set. --from Quora - Cloudera Data Scientist</p>",
      "rawMarkdown": "Your situation sounds a bit strange though as you're saying you know your test set won't have entire features? Then, do not include those features in your training set. --from Quora - Cloudera Data Scientist",
      "votes": null
    },
    {
      "id": "125687",
      "postDate": "07/01/2016 15:39:10",
      "content": "<p>In this challenge you are trying to predict demand. </p>\n\n<p>The features not available in the test set are directly related to that demand (like sales in units, sales in pesos and sales returned). They could all have been collapsed into a single dependent variable (sales).</p>\n\n<p>I hope this helps.</p>",
      "rawMarkdown": "In this challenge you are trying to predict demand. \r\n\r\nThe features not available in the test set are directly related to that demand (like sales in units, sales in pesos and sales returned). They could all have been collapsed into a single dependent variable (sales).\r\n\r\nI hope this helps.",
      "votes": null
    },
    {
      "id": "125695",
      "postDate": "07/01/2016 16:10:12",
      "content": "<p>Hi GC !\nThanks for your response. I understand that demand is the dependent variable but it is not in the test data set. </p>\n\n<p>I think my understanding is that I have to split the training set and 70/30 and get the model to validate the way I want. What i should be able to to is  use a variable from the test data set and apply the slope and intercept (for example) from my model to that variable. BUT the variable is an IDENTIFIER and I cannot do that. \nInstead I should probably match the product ID with that of the training set, unless they are different. Then it is really flawed and this whole competition is not REALISTIC, because in reality I would have all product IDs where needed to predict demand.</p>",
      "rawMarkdown": "Hi GC !\r\nThanks for your response. I understand that demand is the dependent variable but it is not in the test data set. \r\n\r\nI think my understanding is that I have to split the training set and 70/30 and get the model to validate the way I want. What i should be able to to is  use a variable from the test data set and apply the slope and intercept (for example) from my model to that variable. BUT the variable is an IDENTIFIER and I cannot do that. \r\nInstead I should probably match the product ID with that of the training set, unless they are different. Then it is really flawed and this whole competition is not REALISTIC, because in reality I would have all product IDs where needed to predict demand.",
      "votes": null
    },
    {
      "id": "125706",
      "postDate": "07/01/2016 17:50:16",
      "content": "<p>[quote=EmileeMcWilliams;125695]\nThen it is really flawed and this whole competition is not REALISTIC, because in reality I would have all product IDs where needed to predict demand.\n[/quote]</p>\n\n<p>Hi Emilee, it's normal to have a new product in the following weeks. For example, a new kind of energy bar, or toy.</p>\n\n<p>In this case, you could make a cluster in order to identify similar products and use this information to predict the new product demand. </p>",
      "rawMarkdown": "[quote=EmileeMcWilliams;125695]\r\nThen it is really flawed and this whole competition is not REALISTIC, because in reality I would have all product IDs where needed to predict demand.\r\n[/quote]\r\n\r\nHi Emilee, it's normal to have a new product in the following weeks. For example, a new kind of energy bar, or toy.\r\n\r\nIn this case, you could make a cluster in order to identify similar products and use this information to predict the new product demand.",
      "votes": null
    },
    {
      "id": "125710",
      "postDate": "07/01/2016 18:18:36",
      "content": "<p>Thanks Thiago! Helpful for sure.  </p>",
      "rawMarkdown": "Thanks Thiago! Helpful for sure.",
      "votes": null
    },
    {
      "id": "125875",
      "postDate": "07/03/2016 22:41:39",
      "content": "<p>[quote=EmileeMcWilliams;125695]</p>\n\n<p>Hi GC !\nThanks for your response. I understand that demand is the dependent variable but it is not in the test data set. </p>\n\n<p>I think my understanding is that I have to split the training set and 70/30 and get the model to validate the way I want. </p>\n\n<p>[/quote]</p>\n\n<p>Yes, you will have to do train-test split yourself on train.csv. Create and test your model on that. </p>\n\n<p>Later you apply it (&quot;predict&quot;) to the test.csv dataset. When you extract from that the ID,Demand pairs you will have your submission.</p>",
      "rawMarkdown": "[quote=EmileeMcWilliams;125695]\r\n\r\nHi GC !\r\nThanks for your response. I understand that demand is the dependent variable but it is not in the test data set. \r\n\r\nI think my understanding is that I have to split the training set and 70/30 and get the model to validate the way I want. \r\n\r\n[/quote]\r\n\r\nYes, you will have to do train-test split yourself on train.csv. Create and test your model on that. \r\n\r\nLater you apply it (\"predict\") to the test.csv dataset. When you extract from that the ID,Demand pairs you will have your submission.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 125605,
      "author_name": "ermc23",
      "author_url": "",
      "post_date": "06/30/2016 20:04:21",
      "content": "<p>Your situation sounds a bit strange though as you're saying you know your test set won't have entire features? Then, do not include those features in your training set. --from Quora - Cloudera Data Scientist</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125687,
      "author_name": "gcmartinelli",
      "author_url": "",
      "post_date": "07/01/2016 15:39:10",
      "content": "<p>In this challenge you are trying to predict demand. </p>\n\n<p>The features not available in the test set are directly related to that demand (like sales in units, sales in pesos and sales returned). They could all have been collapsed into a single dependent variable (sales).</p>\n\n<p>I hope this helps.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125695,
      "author_name": "ermc23",
      "author_url": "",
      "post_date": "07/01/2016 16:10:12",
      "content": "<p>Hi GC !\nThanks for your response. I understand that demand is the dependent variable but it is not in the test data set. </p>\n\n<p>I think my understanding is that I have to split the training set and 70/30 and get the model to validate the way I want. What i should be able to to is  use a variable from the test data set and apply the slope and intercept (for example) from my model to that variable. BUT the variable is an IDENTIFIER and I cannot do that. \nInstead I should probably match the product ID with that of the training set, unless they are different. Then it is really flawed and this whole competition is not REALISTIC, because in reality I would have all product IDs where needed to predict demand.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125706,
      "author_name": "thiagogbrito",
      "author_url": "",
      "post_date": "07/01/2016 17:50:16",
      "content": "<p>[quote=EmileeMcWilliams;125695]\nThen it is really flawed and this whole competition is not REALISTIC, because in reality I would have all product IDs where needed to predict demand.\n[/quote]</p>\n\n<p>Hi Emilee, it's normal to have a new product in the following weeks. For example, a new kind of energy bar, or toy.</p>\n\n<p>In this case, you could make a cluster in order to identify similar products and use this information to predict the new product demand. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125710,
      "author_name": "ermc23",
      "author_url": "",
      "post_date": "07/01/2016 18:18:36",
      "content": "<p>Thanks Thiago! Helpful for sure.  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125875,
      "author_name": "gcmartinelli",
      "author_url": "",
      "post_date": "07/03/2016 22:41:39",
      "content": "<p>[quote=EmileeMcWilliams;125695]</p>\n\n<p>Hi GC !\nThanks for your response. I understand that demand is the dependent variable but it is not in the test data set. </p>\n\n<p>I think my understanding is that I have to split the training set and 70/30 and get the model to validate the way I want. </p>\n\n<p>[/quote]</p>\n\n<p>Yes, you will have to do train-test split yourself on train.csv. Create and test your model on that. </p>\n\n<p>Later you apply it (&quot;predict&quot;) to the test.csv dataset. When you extract from that the ID,Demand pairs you will have your submission.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "125604": "these two should have the same fields? why do they not? \r\n\r\nhttps://www.quora.com/If-the-training-dataset-has-more-variables-than-the-test-dataset-what-does-one-do",
    "125605": "Your situation sounds a bit strange though as you're saying you know your test set won't have entire features? Then, do not include those features in your training set. --from Quora - Cloudera Data Scientist",
    "125687": "In this challenge you are trying to predict demand. \r\n\r\nThe features not available in the test set are directly related to that demand (like sales in units, sales in pesos and sales returned). They could all have been collapsed into a single dependent variable (sales).\r\n\r\nI hope this helps.",
    "125695": "Hi GC !\r\nThanks for your response. I understand that demand is the dependent variable but it is not in the test data set. \r\n\r\nI think my understanding is that I have to split the training set and 70/30 and get the model to validate the way I want. What i should be able to to is  use a variable from the test data set and apply the slope and intercept (for example) from my model to that variable. BUT the variable is an IDENTIFIER and I cannot do that. \r\nInstead I should probably match the product ID with that of the training set, unless they are different. Then it is really flawed and this whole competition is not REALISTIC, because in reality I would have all product IDs where needed to predict demand.",
    "125706": "[quote=EmileeMcWilliams;125695]\r\nThen it is really flawed and this whole competition is not REALISTIC, because in reality I would have all product IDs where needed to predict demand.\r\n[/quote]\r\n\r\nHi Emilee, it's normal to have a new product in the following weeks. For example, a new kind of energy bar, or toy.\r\n\r\nIn this case, you could make a cluster in order to identify similar products and use this information to predict the new product demand.",
    "125710": "Thanks Thiago! Helpful for sure.",
    "125875": "[quote=EmileeMcWilliams;125695]\r\n\r\nHi GC !\r\nThanks for your response. I understand that demand is the dependent variable but it is not in the test data set. \r\n\r\nI think my understanding is that I have to split the training set and 70/30 and get the model to validate the way I want. \r\n\r\n[/quote]\r\n\r\nYes, you will have to do train-test split yourself on train.csv. Create and test your model on that. \r\n\r\nLater you apply it (\"predict\") to the test.csv dataset. When you extract from that the ID,Demand pairs you will have your submission."
  },
  "source": "meta"
}