Since there are store and item number are categorical and very important to this dataset, I am wondering if using a neural network for the dataset is impossible. Is there anyway you can transform the data so that it can be fit into a neural network? I have tried scikit learn label encoder and the result is pretty frustrating.
", "messages": [ { "id": 234237, "postDate": "2017-10-22T18:16:37.860Z", "content": "There is something like Embedding Layer in Keras, which deals very well with categorical features. You just have to encode them as numbers. It basically does one hot encoding without having to do any transformations. Here you have a nice example, how to use it:\nhttps://www.kaggle.com/lystdo/beat-kkbox-benchmark-without-using-metadata-0-62
", "rawMarkdown": "There is something like Embedding Layer in Keras, which deals very well with categorical features. You just have to encode them as numbers. It basically does one hot encoding without having to do any transformations. Here you have a nice example, how to use it:\nhttps://www.kaggle.com/lystdo/beat-kkbox-benchmark-without-using-metadata-0-62", "votes": 4 }, { "id": 241935, "postDate": "2017-11-10T06:55:10.040Z", "content": "You can use LabelBinarizer from sklearn.preprocessing
", "rawMarkdown": "You can use LabelBinarizer from sklearn.preprocessing", "votes": 1 }, { "id": 259612, "postDate": "2017-12-18T17:58:48.547Z", "content": "I had the same problem. What I did was read the data in chunks storing the unique levels of all the categorical variables. Then using an encoder like the sklearn.preprocessing.CategoricalEncoder or sklearn.preprocessing.LabelBinarizer.
\n\nI found this entire process tedious hence decided to do something. I have created a module that automates all this - dummyPy.
\n\nHere is a small tutorial on this - How to One Hot Encode Categorical Variables of a Large Dataset in Python?
", "rawMarkdown": "I had the same problem. What I did was read the data in chunks storing the unique levels of all the categorical variables. Then using an encoder like the sklearn.preprocessing.CategoricalEncoder or sklearn.preprocessing.LabelBinarizer.\n\nI found this entire process tedious hence decided to do something. I have created a module that automates all this - [dummyPy](https://github.com/yashu-seth/dummyPy).\n\nHere is a small tutorial on this - [How to One Hot Encode Categorical Variables of a Large Dataset in Python?](https://yashuseth.wordpress.com/2017/12/14/how-to-one-hot-encode-categorical-variables-of-a-large-dataset-in-python/)" }, { "id": 236082, "postDate": "2017-10-26T16:51:28.040Z", "content": "thanks kamill!\nseems like Keras is always the easiest way to deal with neural network!
", "rawMarkdown": "thanks kamill!\nseems like Keras is always the easiest way to deal with neural network!" }, { "id": 234214, "postDate": "2017-10-22T17:19:30.187Z", "content": "Well, one hot encoding should come in handy...
", "rawMarkdown": "Well, one hot encoding should come in handy..." }, { "id": 234197, "postDate": "2017-10-22T16:01:29.327Z", "content": "Since there are store and item number are categorical and very important to this dataset, I am wondering if using a neural network for the dataset is impossible. Is there anyway you can transform the data so that it can be fit into a neural network? I have tried scikit learn label encoder and the result is pretty frustrating.
", "rawMarkdown": "Since there are store and item number are categorical and very important to this dataset, I am wondering if using a neural network for the dataset is impossible. Is there anyway you can transform the data so that it can be fit into a neural network? I have tried scikit learn label encoder and the result is pretty frustrating." }, { "id": 259622, "postDate": "2017-12-18T18:27:31.543Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 234237, "author_name": "Kamil Zieliński", "author_url": "", "post_date": "2017-10-22T18:16:37.860000", "content": "There is something like Embedding Layer in Keras, which deals very well with categorical features. You just have to encode them as numbers. It basically does one hot encoding without having to do any transformations. Here you have a nice example, how to use it:\nhttps://www.kaggle.com/lystdo/beat-kkbox-benchmark-without-using-metadata-0-62
", "votes": 4, "replies": [] }, { "id": 241935, "author_name": "Yashu Seth", "author_url": "", "post_date": "2017-11-10T06:55:10.040000", "content": "You can use LabelBinarizer from sklearn.preprocessing
", "votes": 1, "replies": [] }, { "id": 259612, "author_name": "Yashu Seth", "author_url": "", "post_date": "2017-12-18T17:58:48.547000", "content": "I had the same problem. What I did was read the data in chunks storing the unique levels of all the categorical variables. Then using an encoder like the sklearn.preprocessing.CategoricalEncoder or sklearn.preprocessing.LabelBinarizer.
\n\nI found this entire process tedious hence decided to do something. I have created a module that automates all this - dummyPy.
\n\nHere is a small tutorial on this - How to One Hot Encode Categorical Variables of a Large Dataset in Python?
", "votes": 0, "replies": [] }, { "id": 236082, "author_name": "Ben Fung", "author_url": "", "post_date": "2017-10-26T16:51:28.040000", "content": "thanks kamill!\nseems like Keras is always the easiest way to deal with neural network!
", "votes": 0, "replies": [] }, { "id": 234214, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2017-10-22T17:19:30.187000", "content": "Well, one hot encoding should come in handy...
", "votes": 0, "replies": [] }, { "id": 259622, "author_name": "", "author_url": "", "post_date": "2017-12-18T18:27:31.543000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "234237": "There is something like Embedding Layer in Keras, which deals very well with categorical features. You just have to encode them as numbers. It basically does one hot encoding without having to do any transformations. Here you have a nice example, how to use it:\nhttps://www.kaggle.com/lystdo/beat-kkbox-benchmark-without-using-metadata-0-62", "241935": "You can use LabelBinarizer from sklearn.preprocessing", "259612": "I had the same problem. What I did was read the data in chunks storing the unique levels of all the categorical variables. Then using an encoder like the sklearn.preprocessing.CategoricalEncoder or sklearn.preprocessing.LabelBinarizer.\n\nI found this entire process tedious hence decided to do something. I have created a module that automates all this - [dummyPy](https://github.com/yashu-seth/dummyPy).\n\nHere is a small tutorial on this - [How to One Hot Encode Categorical Variables of a Large Dataset in Python?](https://yashuseth.wordpress.com/2017/12/14/how-to-one-hot-encode-categorical-variables-of-a-large-dataset-in-python/)", "236082": "thanks kamill!\nseems like Keras is always the easiest way to deal with neural network!", "234214": "Well, one hot encoding should come in handy...", "234197": "Since there are store and item number are categorical and very important to this dataset, I am wondering if using a neural network for the dataset is impossible. Is there anyway you can transform the data so that it can be fit into a neural network? I have tried scikit learn label encoder and the result is pretty frustrating.", "259622": "" } }