There are many categorical columns in this dataset where the cardinality is extremely high. Because of this, if you are one-hot encoding, you create a surplus of columns. Also, there are a lot of unique categories in the test set that are not seen in the training set.
\n\nHow are you dealing with columns like geoNetwork.city, geoNetwork.country, geoNetwork.metro, trafficSource.adContent, trafficSource.keyword, etc.
Here are a range of options:
\n\nEasy, but probably not great:
\n\nPretty easy, possibly better:
\n\ncategorical_feature setting, explained here: https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html. This may take some hyperparameter tuning to work well.Some more effort, but possibly more effective:
\n\nOHE/encoding in conjunction with careful preprocessing. Exclude/combine rare category values, drop values/columns that are not informative, determine meaningful larger groupings that allow you to reduce cardinality.
Target encoding: described in early posts (replace categories with mean value of target for that category). Can be tricky to use without overfitting (make sure to generate target-encoded values out-of-sample).
Engineer numerical features that are sufficient for capturing the signal conveyed by a categorical feature, then exclude the original categorical feature. This is conceptually similar to frequency/target encoding, where you're summarizing category information via some numeric quantity. For example, maybe the mean page-views per day of the month would be a reasonable substitute for a day of the month categorical feature.
Harder, more experimental:
\n\nNice overview, thanks!
\n\nI'll add feature hashing (aka 'the hashing trick') to that list.
", "rawMarkdown": "Nice overview, thanks!\n\nI'll add feature hashing (aka 'the hashing trick') to that list." } ] }, { "id": 390113, "postDate": "2018-09-19T18:29:36.807Z", "content": "There are many categorical columns in this dataset where the cardinality is extremely high. Because of this, if you are one-hot encoding, you create a surplus of columns. Also, there are a lot of unique categories in the test set that are not seen in the training set.
\n\nHow are you dealing with columns like geoNetwork.city, geoNetwork.country, geoNetwork.metro, trafficSource.adContent, trafficSource.keyword, etc.
There was already a nice summary by @aquatic. Just to add my 5 cents:
\n\nTarget-mean encoding can be done. It works better with smaller cardinalities, but my guess is it would be better than just a simple \"factorize\" into numbers transformation. What it is, you have a categorical column with values a, b, c, ..., z. You find the mean of the target of all rows where the value is a, and that mean value replaces a. Similarly for b, c, ..., z. Optionally factorize the target means to make it ordinal, but I'm not sure that's really better.
\n\nhttps://www.coursera.org/lecture/competitive-data-science/concept-of-mean-encoding-b5Gxv (where I learned about it--you can audit the course for free and learn more than this too)
\n\nhttp://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-munging/target-encoding.html
\n\nor google \"target mean encoding\" for more.
", "rawMarkdown": "Target-mean encoding can be done. It works better with smaller cardinalities, but my guess is it would be better than just a simple \"factorize\" into numbers transformation. What it is, you have a categorical column with values a, b, c, ..., z. You find the mean of the target of all rows where the value is a, and that mean value replaces a. Similarly for b, c, ..., z. Optionally factorize the target means to make it ordinal, but I'm not sure that's really better.\n\nhttps://www.coursera.org/lecture/competitive-data-science/concept-of-mean-encoding-b5Gxv (where I learned about it--you can audit the course for free and learn more than this too)\n\nhttp://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-munging/target-encoding.html\n\nor google \"target mean encoding\" for more.", "votes": 1, "replies": [ { "id": 390383, "postDate": "2018-09-20T05:41:24.463Z", "content": "Check Olivier's kernel Target encoding and mine Target encoding using variance.
", "rawMarkdown": "Check Olivier's kernel [Target encoding][1] and mine [Target encoding using variance][2].\n\n\n [1]: https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features\n [2]: https://www.kaggle.com/alexfir/target-encoding-using-variance", "votes": 1 } ] }, { "id": 390178, "postDate": "2018-09-19T20:25:51.130Z", "content": "That's a good question. Using OHE for all columns is a very bad ideia, so one option is using just label encoding with the categorical_features parameter (LightGBM). Another option is using binary encoder and a third is mixing ohe and le (ohe for columns with low cardinality maybe).
\n\nFor now, I'm using label encoder with the sklearn LabelEncoder, so you can fit all labels from both train and test and them transform. But there are many options for trying.
", "rawMarkdown": "That's a good question. Using OHE for all columns is a very bad ideia, so one option is using just label encoding with the categorical_features parameter (LightGBM). Another option is using binary encoder and a third is mixing ohe and le (ohe for columns with low cardinality maybe).\n\nFor now, I'm using label encoder with the sklearn LabelEncoder, so you can fit all labels from both train and test and them transform. But there are many options for trying.", "votes": 1 }, { "id": 390124, "postDate": "2018-09-19T18:44:36.107Z", "content": "Label encoding is quite a good way of dealing with categorical variables if you use tree-based models.
", "rawMarkdown": "Label encoding is quite a good way of dealing with categorical variables if you use tree-based models.", "votes": 1, "replies": [ { "id": 390126, "postDate": "2018-09-19T18:46:57.037Z", "content": "Why? Don't you have worries that the model will think that if Zurich = 2*Chicago then the inferences will be bad?
Sometimes it is okay to use label encoding for tree based models, because they are simply looking for places to split the data, they are not multiplying inputs by weights.
", "rawMarkdown": "Sometimes it is okay to use label encoding for tree based models, because they are simply looking for places to split the data, they are not multiplying inputs by weights.", "votes": 1 } ] }, { "id": 395680, "postDate": "2018-09-29T06:02:58.880Z", "content": "I'm using le with category partitioning method, but I get a good improvement when leaving features with high cardinality as simple numeric features (this recomendation is on lgbms documentation and seems to work)
", "rawMarkdown": "I'm using le with category partitioning method, but I get a good improvement when leaving features with high cardinality as simple numeric features (this recomendation is on lgbms documentation and seems to work)" }, { "id": 391148, "postDate": "2018-09-21T10:33:30.220Z", "content": "In general, how do you decide whether to use one-hot encoding or label encoding? Are there any criteria for which one you should use for categorical-encoding? (although I know it depends on the situation)
\n\nI usually used one-hot when encoding categorial variables.\nHowever, I saw some public kernels using label encoding in this competition, so I wondered.
", "rawMarkdown": "In general, how do you decide whether to use one-hot encoding or label encoding? Are there any criteria for which one you should use for categorical-encoding? (although I know it depends on the situation)\n\nI usually used one-hot when encoding categorial variables.\nHowever, I saw some public kernels using label encoding in this competition, so I wondered.", "replies": [ { "id": 393062, "postDate": "2018-09-24T19:26:25.693Z", "content": "I was scared to use One-hot encoding in this competition, because the cardinality was large. In the previous Home Credit Default Risk competition, the largest cardinality was like 25, so I was comfortable using one-hot encoding. However, this competition has some categories, like city, with over 900 unique values. This is a problem for two reasons:
\n\nThis makes me reluctant to use one-hot encoding; the other methods above seem much better suited, albeit [strikethrough]theoretically they shouldn't be as powerful.[/strikethrough]
\n\nJoe Eddy's comment below is a lot better!
", "rawMarkdown": "I was scared to use One-hot encoding in this competition, because the cardinality was large. In the previous Home Credit Default Risk competition, the largest cardinality was like 25, so I was comfortable using one-hot encoding. However, this competition has some categories, like city, with over 900 unique values. This is a problem for two reasons: \n\n1. Many models begin to suffer when the number of columns is too large\n2. Might not have enough RAM to fit every column\n\nThis makes me reluctant to use one-hot encoding; the other methods above seem much better suited, albeit [strikethrough]theoretically they shouldn't be as powerful.[/strikethrough]\n\nJoe Eddy's comment below is a lot better!", "votes": 1 }, { "id": 393161, "postDate": "2018-09-24T22:42:20.573Z", "content": "Thanks for your reply! I'll use one-hot and label encoding properly, and seek to find out some other method to be effective for this competition.
", "rawMarkdown": "Thanks for your reply! I'll use one-hot and label encoding properly, and seek to find out some other method to be effective for this competition.", "votes": 1 }, { "id": 393479, "postDate": "2018-09-25T12:58:49.740Z", "content": "OHE often produces suboptimal results when working with tree-based methods like gradient boosted trees. The reason is that since each individual category corresponds to only 1 column, a tree can only make a single category on/off splitting comparison at a time and would need to grow very deep in order to recreate more sophisticated category partitioning. Naturally this means that the problem is worse when you're working with higher cardinality category features, but even at low cardinality it's usually not ideal (the exception is when some category values matter much more than others and/or the simplicity of the OHE format makes the model more robust to noise in the categories). So I tend not to use OHE when using tree methods, except possibly to achieve some model diversity.
\n\nThis is a good discussion: https://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/
", "rawMarkdown": "OHE often produces suboptimal results when working with tree-based methods like gradient boosted trees. The reason is that since each individual category corresponds to only 1 column, a tree can only make a single category on/off splitting comparison at a time and would need to grow very deep in order to recreate more sophisticated category partitioning. Naturally this means that the problem is worse when you're working with higher cardinality category features, but even at low cardinality it's usually not ideal (the exception is when some category values matter much more than others and/or the simplicity of the OHE format makes the model more robust to noise in the categories). So I tend not to use OHE when using tree methods, except possibly to achieve some model diversity.\n\nThis is a good discussion: https://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/ ", "votes": 5 }, { "id": 393516, "postDate": "2018-09-25T14:03:45.973Z", "content": "@aquatic Thank you for the clarification. On a related topic, it has puzzled me why totals.bounces was not seen as super important to gradient boosted machines. Maybe it's because it's binary, and therefore appears to be like one-hot encoded? How can we forced gradient boost machines to attempt to use binary variables first, that we think are important, before trying to split on other variables?
Hi my first choice as is OHE/encoding in conjunction with careful preprocessing. Exclude/combine rare category values, drop values/columns that are not informative, determine meaningful larger groupings that allow you to reduce cardinality. So far it seems that this will yield some intresting and meaningfull categories.
", "rawMarkdown": "Hi my first choice as is OHE/encoding in conjunction with careful preprocessing. Exclude/combine rare category values, drop values/columns that are not informative, determine meaningful larger groupings that allow you to reduce cardinality. So far it seems that this will yield some intresting and meaningfull categories. " }, { "id": 390144, "postDate": "2018-09-19T19:21:10.233Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 390576, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-09-20T13:07:43.277000", "content": "Here are a range of options:
\n\nEasy, but probably not great:
\n\nPretty easy, possibly better:
\n\ncategorical_feature setting, explained here: https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html. This may take some hyperparameter tuning to work well.Some more effort, but possibly more effective:
\n\nOHE/encoding in conjunction with careful preprocessing. Exclude/combine rare category values, drop values/columns that are not informative, determine meaningful larger groupings that allow you to reduce cardinality.
Target encoding: described in early posts (replace categories with mean value of target for that category). Can be tricky to use without overfitting (make sure to generate target-encoded values out-of-sample).
Engineer numerical features that are sufficient for capturing the signal conveyed by a categorical feature, then exclude the original categorical feature. This is conceptually similar to frequency/target encoding, where you're summarizing category information via some numeric quantity. For example, maybe the mean page-views per day of the month would be a reasonable substitute for a day of the month categorical feature.
Harder, more experimental:
\n\nNice overview, thanks!
\n\nI'll add feature hashing (aka 'the hashing trick') to that list.
", "votes": 0, "replies": [] } ] }, { "id": 393476, "author_name": "Misha Lisovyi", "author_url": "", "post_date": "2018-09-25T12:51:26.953000", "content": "There was already a nice summary by @aquatic. Just to add my 5 cents:
\n\nTarget-mean encoding can be done. It works better with smaller cardinalities, but my guess is it would be better than just a simple \"factorize\" into numbers transformation. What it is, you have a categorical column with values a, b, c, ..., z. You find the mean of the target of all rows where the value is a, and that mean value replaces a. Similarly for b, c, ..., z. Optionally factorize the target means to make it ordinal, but I'm not sure that's really better.
\n\nhttps://www.coursera.org/lecture/competitive-data-science/concept-of-mean-encoding-b5Gxv (where I learned about it--you can audit the course for free and learn more than this too)
\n\nhttp://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-munging/target-encoding.html
\n\nor google \"target mean encoding\" for more.
", "votes": 1, "replies": [ { "id": 390383, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-09-20T05:41:24.463000", "content": "Check Olivier's kernel Target encoding and mine Target encoding using variance.
", "votes": 1, "replies": [] } ] }, { "id": 390178, "author_name": "Aguiar", "author_url": "", "post_date": "2018-09-19T20:25:51.130000", "content": "That's a good question. Using OHE for all columns is a very bad ideia, so one option is using just label encoding with the categorical_features parameter (LightGBM). Another option is using binary encoder and a third is mixing ohe and le (ohe for columns with low cardinality maybe).
\n\nFor now, I'm using label encoder with the sklearn LabelEncoder, so you can fit all labels from both train and test and them transform. But there are many options for trying.
", "votes": 1, "replies": [] }, { "id": 390124, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2018-09-19T18:44:36.107000", "content": "Label encoding is quite a good way of dealing with categorical variables if you use tree-based models.
", "votes": 1, "replies": [ { "id": 390126, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-09-19T18:46:57.037000", "content": "Why? Don't you have worries that the model will think that if Zurich = 2*Chicago then the inferences will be bad?
Sometimes it is okay to use label encoding for tree based models, because they are simply looking for places to split the data, they are not multiplying inputs by weights.
", "votes": 1, "replies": [] } ] }, { "id": 395680, "author_name": "Aguiar", "author_url": "", "post_date": "2018-09-29T06:02:58.880000", "content": "I'm using le with category partitioning method, but I get a good improvement when leaving features with high cardinality as simple numeric features (this recomendation is on lgbms documentation and seems to work)
", "votes": 0, "replies": [] }, { "id": 391148, "author_name": "Tak", "author_url": "", "post_date": "2018-09-21T10:33:30.220000", "content": "In general, how do you decide whether to use one-hot encoding or label encoding? Are there any criteria for which one you should use for categorical-encoding? (although I know it depends on the situation)
\n\nI usually used one-hot when encoding categorial variables.\nHowever, I saw some public kernels using label encoding in this competition, so I wondered.
", "votes": 0, "replies": [ { "id": 393062, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-09-24T19:26:25.693000", "content": "I was scared to use One-hot encoding in this competition, because the cardinality was large. In the previous Home Credit Default Risk competition, the largest cardinality was like 25, so I was comfortable using one-hot encoding. However, this competition has some categories, like city, with over 900 unique values. This is a problem for two reasons:
\n\nThis makes me reluctant to use one-hot encoding; the other methods above seem much better suited, albeit [strikethrough]theoretically they shouldn't be as powerful.[/strikethrough]
\n\nJoe Eddy's comment below is a lot better!
", "votes": 1, "replies": [] }, { "id": 393161, "author_name": "Tak", "author_url": "", "post_date": "2018-09-24T22:42:20.573000", "content": "Thanks for your reply! I'll use one-hot and label encoding properly, and seek to find out some other method to be effective for this competition.
", "votes": 1, "replies": [] }, { "id": 393479, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-09-25T12:58:49.740000", "content": "OHE often produces suboptimal results when working with tree-based methods like gradient boosted trees. The reason is that since each individual category corresponds to only 1 column, a tree can only make a single category on/off splitting comparison at a time and would need to grow very deep in order to recreate more sophisticated category partitioning. Naturally this means that the problem is worse when you're working with higher cardinality category features, but even at low cardinality it's usually not ideal (the exception is when some category values matter much more than others and/or the simplicity of the OHE format makes the model more robust to noise in the categories). So I tend not to use OHE when using tree methods, except possibly to achieve some model diversity.
\n\nThis is a good discussion: https://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/
", "votes": 5, "replies": [] }, { "id": 393516, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-09-25T14:03:45.973000", "content": "@aquatic Thank you for the clarification. On a related topic, it has puzzled me why totals.bounces was not seen as super important to gradient boosted machines. Maybe it's because it's binary, and therefore appears to be like one-hot encoded? How can we forced gradient boost machines to attempt to use binary variables first, that we think are important, before trying to split on other variables?
Hi my first choice as is OHE/encoding in conjunction with careful preprocessing. Exclude/combine rare category values, drop values/columns that are not informative, determine meaningful larger groupings that allow you to reduce cardinality. So far it seems that this will yield some intresting and meaningfull categories.
", "votes": 0, "replies": [] }, { "id": 390144, "author_name": "", "author_url": "", "post_date": "2018-09-19T19:21:10.233000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "390576": "Here are a range of options:\n\n**Easy, but probably not great**:\n\n - One-hot-encoding\n - Simple label-encoding, treat column as numerical\n\n**Pretty easy, possibly better**:\n\n - Frequency encoding: map categories to their count of occurrence, treat column as numerical. In theory this should be strictly better than label-encoding, as it allows categories to still be split based on their specific values, but conveys some information about inter-category relationships (frequency differences) that may provide signal. \n - Assuming tree-methods (e.g. lgb), use a category partitioning method. In the case of lgb, this means using label encoding in conjunction with lgb's `categorical_feature` setting, explained here: https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html. This may take some hyperparameter tuning to work well.\n\n**Some more effort, but possibly more effective**:\n\n - OHE/encoding in conjunction with careful preprocessing. Exclude/combine rare category values, drop values/columns that are not informative, determine meaningful larger groupings that allow you to reduce cardinality.\n\n - Target encoding: described in early posts (replace categories with mean value of target for that category). Can be tricky to use without overfitting (make sure to generate target-encoded values out-of-sample).\n\n - Engineer numerical features that are sufficient for capturing the signal conveyed by a categorical feature, then exclude the original categorical feature. This is conceptually similar to frequency/target encoding, where you're summarizing category information via some numeric quantity. For example, maybe the mean page-views per day of the month would be a reasonable substitute for a day of the month categorical feature.\n\n\n \n\n**Harder, more experimental**:\n\n - Replace category columns with dense embeddings learned from a neural network.", "390113": "There are many categorical columns in this dataset where the cardinality is extremely high. Because of this, if you are one-hot encoding, you create a surplus of columns. Also, there are a lot of unique categories in the test set that are not seen in the training set.\n\nHow are you dealing with columns like `geoNetwork.city`, `geoNetwork.country`, `geoNetwork.metro`, `trafficSource.adContent`, `trafficSource.keyword`, etc.", "393476": "There was already a nice summary by @aquatic. Just to add my 5 cents: \n\n- an example of several target-encoding implementations (some of which already include regularisation to avoid data leaks): https://www.kaggle.com/mlisovyi/9-ways-to-treat-categorical-features-updated;\n- Catboost is your friend, when it comes to categorical features, as it implements several schemes for categorical encoding with minimised/eliminated(?) data leakage;\n- I personally reduce cardinality simply by identifying all classes within a categorical, that appear with frequency below 0.1% (guesstimated number). In practice I merge them into a new class within category.This allows to reduce cardinality down to ~20 per each category (note that this removes somes info, that is there)", "390324": "Target-mean encoding can be done. It works better with smaller cardinalities, but my guess is it would be better than just a simple \"factorize\" into numbers transformation. What it is, you have a categorical column with values a, b, c, ..., z. You find the mean of the target of all rows where the value is a, and that mean value replaces a. Similarly for b, c, ..., z. Optionally factorize the target means to make it ordinal, but I'm not sure that's really better.\n\nhttps://www.coursera.org/lecture/competitive-data-science/concept-of-mean-encoding-b5Gxv (where I learned about it--you can audit the course for free and learn more than this too)\n\nhttp://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-munging/target-encoding.html\n\nor google \"target mean encoding\" for more.", "390178": "That's a good question. Using OHE for all columns is a very bad ideia, so one option is using just label encoding with the categorical_features parameter (LightGBM). Another option is using binary encoder and a third is mixing ohe and le (ohe for columns with low cardinality maybe).\n\nFor now, I'm using label encoder with the sklearn LabelEncoder, so you can fit all labels from both train and test and them transform. But there are many options for trying.", "390124": "Label encoding is quite a good way of dealing with categorical variables if you use tree-based models.", "395680": "I'm using le with category partitioning method, but I get a good improvement when leaving features with high cardinality as simple numeric features (this recomendation is on lgbms documentation and seems to work)", "391148": "In general, how do you decide whether to use one-hot encoding or label encoding? Are there any criteria for which one you should use for categorical-encoding? (although I know it depends on the situation)\n\nI usually used one-hot when encoding categorial variables.\nHowever, I saw some public kernels using label encoding in this competition, so I wondered.", "391091": "Hi my first choice as is OHE/encoding in conjunction with careful preprocessing. Exclude/combine rare category values, drop values/columns that are not informative, determine meaningful larger groupings that allow you to reduce cardinality. So far it seems that this will yield some intresting and meaningfull categories. ", "390144": "" } }