When I try to one-hot encode predictors so that I can do recursive feature elimination, I run out of memory. Should I try a different approach, or does anyone have a solution? I am aware that what I tried may not be possible because there are so many features with many levels in this dataset, I am mostly looking for a similar method and that's the best example I can come up with. Should I be using a forest instead? I plan to use LightGBM.
\n\nSide note: I did drop the features with proportions of missing values above 80% (except for 'totals_transactionRevenue'). What features have you dropped and why?
", "messages": [ { "id": "409387", "postDate": "10/24/2018 08:07:56", "content": "When I try to one-hot encode predictors so that I can do recursive feature elimination, I run out of memory. Should I try a different approach, or does anyone have a solution? I am aware that what I tried may not be possible because there are so many features with many levels in this dataset, I am mostly looking for a similar method and that's the best example I can come up with. Should I be using a forest instead? I plan to use LightGBM.
\n\nSide note: I did drop the features with proportions of missing values above 80% (except for 'totals_transactionRevenue'). What features have you dropped and why?
", "rawMarkdown": "When I try to one-hot encode predictors so that I can do recursive feature elimination, I run out of memory. Should I try a different approach, or does anyone have a solution? I am aware that what I tried may not be possible because there are so many features with many levels in this dataset, I am mostly looking for a similar method and that's the best example I can come up with. Should I be using a forest instead? I plan to use LightGBM.\n\nSide note: I did drop the features with proportions of missing values above 80% (except for 'totals_transactionRevenue'). What features have you dropped and why?", "votes": null }, { "id": "409407", "postDate": "10/24/2018 08:46:02", "content": "Are you storing the OHE version of your dataset as dense objects? If so, switching to sparse representation should help with memory.
", "rawMarkdown": "Are you storing the OHE version of your dataset as dense objects? If so, switching to sparse representation should help with memory.", "votes": null }, { "id": "409412", "postDate": "10/24/2018 08:52:20", "content": "I was using get_dummies from pandas but now I will use sklearn.preprocessing.OneHotEncoder. I'll set it to sparse representation and see what happens.
", "rawMarkdown": "I was using get_dummies from pandas but now I will use sklearn.preprocessing.OneHotEncoder. I'll set it to sparse representation and see what happens.", "votes": null }, { "id": "410400", "postDate": "10/26/2018 01:56:15", "content": "Okay, I have tried the code below and I am still getting a memory error. The label encoding works but the OneHot encoding does not. Any ideas?
\n\nfrom sklearn.preprocessing import LabelEncoder, LabelBinarizer, OneHotEncoder\n\ntrain_predictors = train.drop('totals_transactionRevenue', axis=1)\n\nnon_relevant = [\"visitNumber\", \"date\", \"fullVisitorId\", \"sessionId\", \"visitId\", \"visitStartTime\"]\n\nconstant_columns = []\nfor col in train_predictors.columns:\n if len(train_predictors[col].value_counts()) == 1:\n constant_columns.append(col)\n\ncategorical_columns = [c for c in train_predictors.columns if not c.startswith(\"total\")]\ncategorical_columns = [c for c in categorical_columns if c not in constant_columns + non_relevant]\n\nfor c in categorical_columns:\n\n le = LabelEncoder()\n train_vals = list(train_predictors[c].values.astype(str))\n\n le.fit(train_vals)\n\n train_predictors[c] = le.transform(train_vals)\n\nfor c in categorical_columns:\n\n ohe = OneHotEncoder(sparse=True)\n train_vals = list(train_predictors[c].values.astype(str))\n\n ohe.fit([train_vals])\n\n train_predictors[c] = ohe.transform(train_vals)\nFrom the top of my head, the fact that you are fitting separate instances of ohe per feature and then appending them to the original (replacing the feature, strictly speaking) might be a problem: train_predictors is a dense object and even if your result of ohe is sparse, it will be converted to dense implicitly.
", "rawMarkdown": "From the top of my head, the fact that you are fitting separate instances of ohe per feature and then appending them to the original (replacing the feature, strictly speaking) might be a problem: train_predictors is a dense object and even if your result of ohe is sparse, it will be converted to dense implicitly.", "votes": null }, { "id": "411076", "postDate": "10/27/2018 10:51:52", "content": "You can merge all category values that have no revenue against them before OHE. They have no information viz the target, so might as well all be labelled “other”
\n\nOr you could use a model engine that likes categoricals, like CatBoost or XGboost (convert to integers first).
", "rawMarkdown": "You can merge all category values that have no revenue against them before OHE. They have no information viz the target, so might as well all be labelled “other”\n\nOr you could use a model engine that likes categoricals, like CatBoost or XGboost (convert to integers first).", "votes": null }, { "id": "411181", "postDate": "10/27/2018 15:00:10", "content": "I ran into similar issue however I didn't experiment with after that. I assume, it might be because of the quantity of sections getting made by the encoder. Experiment with Clearing uncommon classifications before you encode.
\n\ndef clearRare(columnname, limit = 1000):\n # you may search for rare categories in train, train&test, or just test\n #vc = pd.concat([train[columnname], test[columnname]], sort=False).value_counts()\n vc = test[columnname].value_counts()\n\n common = vc\n common = set(common.index[common].values)\n print(\"Set\", sum(vc <= limit), columnname, \"categories to 'other';\", end=\" \")\n\n train.loc[train[columnname].map(lambda x: x not in common), columnname] = 'other'\n test.loc[test[columnname].map(lambda x: x not in common), columnname] = 'other'\n print(\"now there are\", train[columnname].nunique(), \"categories in train\")\nReference code copied from this kernel\nhttps://www.kaggle.com/karkun/sergey-ivanov-msu-mmp#
", "rawMarkdown": "I ran into similar issue however I didn't experiment with after that. I assume, it might be because of the quantity of sections getting made by the encoder. Experiment with **Clearing uncommon classifications** before you encode.\n\n def clearRare(columnname, limit = 1000):\n # you may search for rare categories in train, train&test, or just test\n #vc = pd.concat([train[columnname], test[columnname]], sort=False).value_counts()\n vc = test[columnname].value_counts()\n \n common = vc\n common = set(common.index[common].values)\n print(\"Set\", sum(vc <= limit), columnname, \"categories to 'other';\", end=\" \")\n \n train.loc[train[columnname].map(lambda x: x not in common), columnname] = 'other'\n test.loc[test[columnname].map(lambda x: x not in common), columnname] = 'other'\n print(\"now there are\", train[columnname].nunique(), \"categories in train\")\n\nReference code copied from this kernel\nhttps://www.kaggle.com/karkun/sergey-ivanov-msu-mmp#", "votes": null }, { "id": "853439", "postDate": "05/19/2020 07:30:18", "content": "I am facing a similar issue, so curious to know, if you've got the issue resolved, and if yes, how...
", "rawMarkdown": "I am facing a similar issue, so curious to know, if you've got the issue resolved, and if yes, how...", "votes": null } ], "comments": [ { "id": 409407, "author_name": "konradb", "author_url": "", "post_date": "10/24/2018 08:46:02", "content": "Are you storing the OHE version of your dataset as dense objects? If so, switching to sparse representation should help with memory.
", "votes": null, "replies": [ { "id": 409412, "author_name": "bengwalt", "author_url": "", "post_date": "10/24/2018 08:52:20", "content": "I was using get_dummies from pandas but now I will use sklearn.preprocessing.OneHotEncoder. I'll set it to sparse representation and see what happens.
", "votes": null, "replies": [] }, { "id": 410400, "author_name": "bengwalt", "author_url": "", "post_date": "10/26/2018 01:56:15", "content": "Okay, I have tried the code below and I am still getting a memory error. The label encoding works but the OneHot encoding does not. Any ideas?
\n\nfrom sklearn.preprocessing import LabelEncoder, LabelBinarizer, OneHotEncoder\n\ntrain_predictors = train.drop('totals_transactionRevenue', axis=1)\n\nnon_relevant = [\"visitNumber\", \"date\", \"fullVisitorId\", \"sessionId\", \"visitId\", \"visitStartTime\"]\n\nconstant_columns = []\nfor col in train_predictors.columns:\n if len(train_predictors[col].value_counts()) == 1:\n constant_columns.append(col)\n\ncategorical_columns = [c for c in train_predictors.columns if not c.startswith(\"total\")]\ncategorical_columns = [c for c in categorical_columns if c not in constant_columns + non_relevant]\n\nfor c in categorical_columns:\n\n le = LabelEncoder()\n train_vals = list(train_predictors[c].values.astype(str))\n\n le.fit(train_vals)\n\n train_predictors[c] = le.transform(train_vals)\n\nfor c in categorical_columns:\n\n ohe = OneHotEncoder(sparse=True)\n train_vals = list(train_predictors[c].values.astype(str))\n\n ohe.fit([train_vals])\n\n train_predictors[c] = ohe.transform(train_vals)\nFrom the top of my head, the fact that you are fitting separate instances of ohe per feature and then appending them to the original (replacing the feature, strictly speaking) might be a problem: train_predictors is a dense object and even if your result of ohe is sparse, it will be converted to dense implicitly.
", "votes": null, "replies": [] }, { "id": 853439, "author_name": "bkrishna2006", "author_url": "", "post_date": "05/19/2020 07:30:18", "content": "I am facing a similar issue, so curious to know, if you've got the issue resolved, and if yes, how...
", "votes": null, "replies": [] } ] }, { "id": 411076, "author_name": "burritodan", "author_url": "", "post_date": "10/27/2018 10:51:52", "content": "You can merge all category values that have no revenue against them before OHE. They have no information viz the target, so might as well all be labelled “other”
\n\nOr you could use a model engine that likes categoricals, like CatBoost or XGboost (convert to integers first).
", "votes": null, "replies": [] }, { "id": 411181, "author_name": "pnprabakaran", "author_url": "", "post_date": "10/27/2018 15:00:10", "content": "I ran into similar issue however I didn't experiment with after that. I assume, it might be because of the quantity of sections getting made by the encoder. Experiment with Clearing uncommon classifications before you encode.
\n\ndef clearRare(columnname, limit = 1000):\n # you may search for rare categories in train, train&test, or just test\n #vc = pd.concat([train[columnname], test[columnname]], sort=False).value_counts()\n vc = test[columnname].value_counts()\n\n common = vc\n common = set(common.index[common].values)\n print(\"Set\", sum(vc <= limit), columnname, \"categories to 'other';\", end=\" \")\n\n train.loc[train[columnname].map(lambda x: x not in common), columnname] = 'other'\n test.loc[test[columnname].map(lambda x: x not in common), columnname] = 'other'\n print(\"now there are\", train[columnname].nunique(), \"categories in train\")\nReference code copied from this kernel\nhttps://www.kaggle.com/karkun/sergey-ivanov-msu-mmp#
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "409387": "When I try to one-hot encode predictors so that I can do recursive feature elimination, I run out of memory. Should I try a different approach, or does anyone have a solution? I am aware that what I tried may not be possible because there are so many features with many levels in this dataset, I am mostly looking for a similar method and that's the best example I can come up with. Should I be using a forest instead? I plan to use LightGBM.\n\nSide note: I did drop the features with proportions of missing values above 80% (except for 'totals_transactionRevenue'). What features have you dropped and why?", "409407": "Are you storing the OHE version of your dataset as dense objects? If so, switching to sparse representation should help with memory.", "409412": "I was using get_dummies from pandas but now I will use sklearn.preprocessing.OneHotEncoder. I'll set it to sparse representation and see what happens.", "410400": "Okay, I have tried the code below and I am still getting a memory error. The label encoding works but the OneHot encoding does not. Any ideas?\n\n from sklearn.preprocessing import LabelEncoder, LabelBinarizer, OneHotEncoder\n \n train_predictors = train.drop('totals_transactionRevenue', axis=1)\n \n non_relevant = [\"visitNumber\", \"date\", \"fullVisitorId\", \"sessionId\", \"visitId\", \"visitStartTime\"]\n \n constant_columns = []\n for col in train_predictors.columns:\n if len(train_predictors[col].value_counts()) == 1:\n constant_columns.append(col)\n \n categorical_columns = [c for c in train_predictors.columns if not c.startswith(\"total\")]\n categorical_columns = [c for c in categorical_columns if c not in constant_columns + non_relevant]\n \n for c in categorical_columns:\n \n le = LabelEncoder()\n train_vals = list(train_predictors[c].values.astype(str))\n \n le.fit(train_vals)\n \n train_predictors[c] = le.transform(train_vals)\n \n for c in categorical_columns:\n \n ohe = OneHotEncoder(sparse=True)\n train_vals = list(train_predictors[c].values.astype(str))\n \n ohe.fit([train_vals])\n \n train_predictors[c] = ohe.transform(train_vals)", "410570": "From the top of my head, the fact that you are fitting separate instances of ohe per feature and then appending them to the original (replacing the feature, strictly speaking) might be a problem: train_predictors is a dense object and even if your result of ohe is sparse, it will be converted to dense implicitly.", "411076": "You can merge all category values that have no revenue against them before OHE. They have no information viz the target, so might as well all be labelled “other”\n\nOr you could use a model engine that likes categoricals, like CatBoost or XGboost (convert to integers first).", "411181": "I ran into similar issue however I didn't experiment with after that. I assume, it might be because of the quantity of sections getting made by the encoder. Experiment with **Clearing uncommon classifications** before you encode.\n\n def clearRare(columnname, limit = 1000):\n # you may search for rare categories in train, train&test, or just test\n #vc = pd.concat([train[columnname], test[columnname]], sort=False).value_counts()\n vc = test[columnname].value_counts()\n \n common = vc\n common = set(common.index[common].values)\n print(\"Set\", sum(vc <= limit), columnname, \"categories to 'other';\", end=\" \")\n \n train.loc[train[columnname].map(lambda x: x not in common), columnname] = 'other'\n test.loc[test[columnname].map(lambda x: x not in common), columnname] = 'other'\n print(\"now there are\", train[columnname].nunique(), \"categories in train\")\n\nReference code copied from this kernel\nhttps://www.kaggle.com/karkun/sergey-ivanov-msu-mmp#", "853439": "I am facing a similar issue, so curious to know, if you've got the issue resolved, and if yes, how..." }, "source": "meta" }