{
  "id": 45866,
  "title": "One Hot Encoding While Reading in Chunks",
  "url": "/competitions/favorita-grocery-sales-forecasting/discussion/45866",
  "author_name": "",
  "post_date": "2017-12-16T21:17:30.168997900Z",
  "votes": 5,
  "comment_count": 3,
  "views": 0,
  "content": "<p>While working on this problem, I had trouble one hot encoding the categorical features because I was reading the data in chunks (using the pandas.read_csv chunksize parameter).</p>\n\n<p>I was reading the data in chunks storing the unique levels of the categorical variables and then one hot encoding them using sklearn.preprocessing.CategoricalEncoder.</p>\n\n<p>I found this entire process tedious hence decided to do something. I have created a module that automates all this - <a href=\"https://github.com/yashu-seth/dummyPy\">dummyPy</a>.</p>\n\n<p>Here is a small tutorial on this - <a href=\"https://yashuseth.wordpress.com/2017/12/14/how-to-one-hot-encode-categorical-variables-of-a-large-dataset-in-python/\">How to One Hot Encode Categorical Variables of a Large Dataset in Python?</a></p>",
  "messages": [
    {
      "id": "258714",
      "postDate": "12/16/2017 21:17:30",
      "content": "<p>While working on this problem, I had trouble one hot encoding the categorical features because I was reading the data in chunks (using the pandas.read_csv chunksize parameter).</p>\n\n<p>I was reading the data in chunks storing the unique levels of the categorical variables and then one hot encoding them using sklearn.preprocessing.CategoricalEncoder.</p>\n\n<p>I found this entire process tedious hence decided to do something. I have created a module that automates all this - <a href=\"https://github.com/yashu-seth/dummyPy\">dummyPy</a>.</p>\n\n<p>Here is a small tutorial on this - <a href=\"https://yashuseth.wordpress.com/2017/12/14/how-to-one-hot-encode-categorical-variables-of-a-large-dataset-in-python/\">How to One Hot Encode Categorical Variables of a Large Dataset in Python?</a></p>",
      "rawMarkdown": "While working on this problem, I had trouble one hot encoding the categorical features because I was reading the data in chunks (using the pandas.read_csv chunksize parameter).\n\nI was reading the data in chunks storing the unique levels of the categorical variables and then one hot encoding them using sklearn.preprocessing.CategoricalEncoder.\n\nI found this entire process tedious hence decided to do something. I have created a module that automates all this - [dummyPy](https://github.com/yashu-seth/dummyPy).\n\nHere is a small tutorial on this - [How to One Hot Encode Categorical Variables of a Large Dataset in Python?](https://yashuseth.wordpress.com/2017/12/14/how-to-one-hot-encode-categorical-variables-of-a-large-dataset-in-python/)",
      "votes": null
    },
    {
      "id": "259496",
      "postDate": "12/18/2017 14:14:21",
      "content": "<p>Thanks for the tutorial.</p>",
      "rawMarkdown": "Thanks for the tutorial.",
      "votes": null
    },
    {
      "id": "259816",
      "postDate": "12/19/2017 03:00:27",
      "content": "<p>tks</p>",
      "rawMarkdown": "tks",
      "votes": null
    },
    {
      "id": "260309",
      "postDate": "12/20/2017 02:36:36",
      "content": "<p>With Pandas, you can define your custom Categorical Variable type with CategoricalDtype and apply it to all chunks.</p>",
      "rawMarkdown": "With Pandas, you can define your custom Categorical Variable type with CategoricalDtype and apply it to all chunks.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 259496,
      "author_name": "yatishbn",
      "author_url": "",
      "post_date": "12/18/2017 14:14:21",
      "content": "<p>Thanks for the tutorial.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 259816,
      "author_name": "buaack",
      "author_url": "",
      "post_date": "12/19/2017 03:00:27",
      "content": "<p>tks</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 260309,
      "author_name": "bearsun",
      "author_url": "",
      "post_date": "12/20/2017 02:36:36",
      "content": "<p>With Pandas, you can define your custom Categorical Variable type with CategoricalDtype and apply it to all chunks.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "258714": "While working on this problem, I had trouble one hot encoding the categorical features because I was reading the data in chunks (using the pandas.read_csv chunksize parameter).\n\nI was reading the data in chunks storing the unique levels of the categorical variables and then one hot encoding them using sklearn.preprocessing.CategoricalEncoder.\n\nI found this entire process tedious hence decided to do something. I have created a module that automates all this - [dummyPy](https://github.com/yashu-seth/dummyPy).\n\nHere is a small tutorial on this - [How to One Hot Encode Categorical Variables of a Large Dataset in Python?](https://yashuseth.wordpress.com/2017/12/14/how-to-one-hot-encode-categorical-variables-of-a-large-dataset-in-python/)",
    "259496": "Thanks for the tutorial.",
    "259816": "tks",
    "260309": "With Pandas, you can define your custom Categorical Variable type with CategoricalDtype and apply it to all chunks."
  },
  "source": "meta"
}