{
  "id": 21786,
  "title": "Handling categorical values",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/21786",
  "author_name": "",
  "post_date": "2016-06-19T20:30:26.897Z",
  "votes": null,
  "comment_count": 6,
  "views": 1066,
  "content": "<p>Most of the values we have to work with are categorical (ex: producto_id, agencia_id, cliente_id, etc.).</p>\n\n<p>My first attempt was to build simple linear regressions. But for that I need to convert these categorical data into actionable dummy variables.</p>\n\n<p>I never worked with such a large dataset and I'm having trouble converting using Pandas.get_dummies() (my computer simply can't handle it).</p>\n\n<p>Is my approach wrong? Is this dataset simply impractical for Linear Regression?</p>",
  "messages": [
    {
      "id": "124541",
      "postDate": "06/19/2016 20:30:26",
      "content": "<p>Most of the values we have to work with are categorical (ex: producto_id, agencia_id, cliente_id, etc.).</p>\n\n<p>My first attempt was to build simple linear regressions. But for that I need to convert these categorical data into actionable dummy variables.</p>\n\n<p>I never worked with such a large dataset and I'm having trouble converting using Pandas.get_dummies() (my computer simply can't handle it).</p>\n\n<p>Is my approach wrong? Is this dataset simply impractical for Linear Regression?</p>",
      "rawMarkdown": "Most of the values we have to work with are categorical (ex: producto_id, agencia_id, cliente_id, etc.).\r\n\r\nMy first attempt was to build simple linear regressions. But for that I need to convert these categorical data into actionable dummy variables.\r\n\r\nI never worked with such a large dataset and I'm having trouble converting using Pandas.get_dummies() (my computer simply can't handle it).\r\n\r\nIs my approach wrong? Is this dataset simply impractical for Linear Regression?",
      "votes": null
    },
    {
      "id": "124545",
      "postDate": "06/19/2016 20:36:11",
      "content": "<p>@gcmartinelli  You can get a sparse matrix of dummy variables by passing sparse=True in pd.get_dummies. It will be very efficient and will take very small amount of memory. Also you need to pass list of columns you want sparse dummy matrix of</p>",
      "rawMarkdown": "gcmartinelli  You can get a sparse matrix of dummy variables by passing sparse=True in pd.get_dummies. It will be very efficient and will take very small amount of memory. Also you need to pass list of columns you want sparse dummy matrix of",
      "votes": null
    },
    {
      "id": "124546",
      "postDate": "06/19/2016 20:41:38",
      "content": "<p>[quote=Shahnawaz Akhtar;124545]</p>\n\n<p>@gcmartinelli  You can get a sparse matrix of dummy variables by passing sparse=True in pd.get_dummies. It will be very efficient and will take very small amount of memory. Also you need to pass list of columns you want sparse dummy matrix of</p>\n\n<p>[/quote]</p>\n\n<p>Thank you very much! I will experiment with that.</p>\n\n<p>If you don't mind sharing more of your insight... In your opinion, is Linear Regression a valid model to work with?</p>",
      "rawMarkdown": "[quote=Shahnawaz Akhtar;124545]\r\n\r\n@gcmartinelli  You can get a sparse matrix of dummy variables by passing sparse=True in pd.get_dummies. It will be very efficient and will take very small amount of memory. Also you need to pass list of columns you want sparse dummy matrix of\r\n\r\n[/quote]\r\n\r\nThank you very much! I will experiment with that.\r\n\r\nIf you don't mind sharing more of your insight... In your opinion, is Linear Regression a valid model to work with?",
      "votes": null
    },
    {
      "id": "124559",
      "postDate": "06/19/2016 23:06:56",
      "content": "<p>I think it should for some products whose demand has been constantly increasing or decreasing. But for other products whose demand is distributed non-linearly or is heteroskedastic over semanas, the prediction will not be so accurate. </p>",
      "rawMarkdown": "I think it should for some products whose demand has been constantly increasing or decreasing. But for other products whose demand is distributed non-linearly or is heteroskedastic over semanas, the prediction will not be so accurate.",
      "votes": null
    },
    {
      "id": "124575",
      "postDate": "06/20/2016 03:44:02",
      "content": "<p>Deleted.</p>",
      "rawMarkdown": "Deleted.",
      "votes": null
    },
    {
      "id": "124624",
      "postDate": "06/20/2016 16:06:06",
      "content": "<p>I am using R and getting out of memory while creating sparse matrix using Matrix package. can anyone suggest something for R?</p>",
      "rawMarkdown": "I am using R and getting out of memory while creating sparse matrix using Matrix package. can anyone suggest something for R?",
      "votes": null
    },
    {
      "id": "124755",
      "postDate": "06/22/2016 01:24:09",
      "content": "<p>I ran a linear regression model with dummy.code()-applied-categorical-variables for 5,000 rows of\nshuffled train data. The multiple R-squared I got from the model is 0.03096 (correlation around 0.18).\nTherefore, I don't recommend to use a linear regression model. </p>",
      "rawMarkdown": "I ran a linear regression model with dummy.code()-applied-categorical-variables for 5,000 rows of\r\nshuffled train data. The multiple R-squared I got from the model is 0.03096 (correlation around 0.18).\r\nTherefore, I don't recommend to use a linear regression model.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 124545,
      "author_name": "shahnawazakhtar",
      "author_url": "",
      "post_date": "06/19/2016 20:36:11",
      "content": "<p>@gcmartinelli  You can get a sparse matrix of dummy variables by passing sparse=True in pd.get_dummies. It will be very efficient and will take very small amount of memory. Also you need to pass list of columns you want sparse dummy matrix of</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124546,
      "author_name": "gcmartinelli",
      "author_url": "",
      "post_date": "06/19/2016 20:41:38",
      "content": "<p>[quote=Shahnawaz Akhtar;124545]</p>\n\n<p>@gcmartinelli  You can get a sparse matrix of dummy variables by passing sparse=True in pd.get_dummies. It will be very efficient and will take very small amount of memory. Also you need to pass list of columns you want sparse dummy matrix of</p>\n\n<p>[/quote]</p>\n\n<p>Thank you very much! I will experiment with that.</p>\n\n<p>If you don't mind sharing more of your insight... In your opinion, is Linear Regression a valid model to work with?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124559,
      "author_name": "shahnawazakhtar",
      "author_url": "",
      "post_date": "06/19/2016 23:06:56",
      "content": "<p>I think it should for some products whose demand has been constantly increasing or decreasing. But for other products whose demand is distributed non-linearly or is heteroskedastic over semanas, the prediction will not be so accurate. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124575,
      "author_name": "njsgrace",
      "author_url": "",
      "post_date": "06/20/2016 03:44:02",
      "content": "<p>Deleted.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124624,
      "author_name": "avinashrai",
      "author_url": "",
      "post_date": "06/20/2016 16:06:06",
      "content": "<p>I am using R and getting out of memory while creating sparse matrix using Matrix package. can anyone suggest something for R?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124755,
      "author_name": "njsgrace",
      "author_url": "",
      "post_date": "06/22/2016 01:24:09",
      "content": "<p>I ran a linear regression model with dummy.code()-applied-categorical-variables for 5,000 rows of\nshuffled train data. The multiple R-squared I got from the model is 0.03096 (correlation around 0.18).\nTherefore, I don't recommend to use a linear regression model. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "124541": "Most of the values we have to work with are categorical (ex: producto_id, agencia_id, cliente_id, etc.).\r\n\r\nMy first attempt was to build simple linear regressions. But for that I need to convert these categorical data into actionable dummy variables.\r\n\r\nI never worked with such a large dataset and I'm having trouble converting using Pandas.get_dummies() (my computer simply can't handle it).\r\n\r\nIs my approach wrong? Is this dataset simply impractical for Linear Regression?",
    "124545": "gcmartinelli  You can get a sparse matrix of dummy variables by passing sparse=True in pd.get_dummies. It will be very efficient and will take very small amount of memory. Also you need to pass list of columns you want sparse dummy matrix of",
    "124546": "[quote=Shahnawaz Akhtar;124545]\r\n\r\n@gcmartinelli  You can get a sparse matrix of dummy variables by passing sparse=True in pd.get_dummies. It will be very efficient and will take very small amount of memory. Also you need to pass list of columns you want sparse dummy matrix of\r\n\r\n[/quote]\r\n\r\nThank you very much! I will experiment with that.\r\n\r\nIf you don't mind sharing more of your insight... In your opinion, is Linear Regression a valid model to work with?",
    "124559": "I think it should for some products whose demand has been constantly increasing or decreasing. But for other products whose demand is distributed non-linearly or is heteroskedastic over semanas, the prediction will not be so accurate.",
    "124575": "Deleted.",
    "124624": "I am using R and getting out of memory while creating sparse matrix using Matrix package. can anyone suggest something for R?",
    "124755": "I ran a linear regression model with dummy.code()-applied-categorical-variables for 5,000 rows of\r\nshuffled train data. The multiple R-squared I got from the model is 0.03096 (correlation around 0.18).\r\nTherefore, I don't recommend to use a linear regression model."
  },
  "source": "meta"
}