I struggle with the high cardinality of some features.\nAfter googling, I found good advices.
\n\nI want to share them to help people like me who bothered from the features.
\n\n\n\n\n\nhttps://www.kdnuggets.com/2016/08/include-high-cardinality-attributes-predictive-model.html
\n\nAfter reading these articles, \nI'm trying to apply mean encoding and frequency encoding.
\n\nthe script is shown in below kernels\nhttps://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features\nhttps://www.kaggle.com/vchoubey/random-forest-mean-encoding-for-app-channel
\n\nI attatched my code of mean and frequency encoding in reply.\n(Please, be aware of indent!)
\n\nThe choices depend on you. hope this help you!
", "messages": [ { "id": 392132, "postDate": "2018-09-23T04:22:40.100Z", "content": "I struggle with the high cardinality of some features.\nAfter googling, I found good advices.
\n\nI want to share them to help people like me who bothered from the features.
\n\n\n\n\n\nhttps://www.kdnuggets.com/2016/08/include-high-cardinality-attributes-predictive-model.html
\n\nAfter reading these articles, \nI'm trying to apply mean encoding and frequency encoding.
\n\nthe script is shown in below kernels\nhttps://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features\nhttps://www.kaggle.com/vchoubey/random-forest-mean-encoding-for-app-channel
\n\nI attatched my code of mean and frequency encoding in reply.\n(Please, be aware of indent!)
\n\nThe choices depend on you. hope this help you!
", "rawMarkdown": "I struggle with the high cardinality of some features.\nAfter googling, I found good advices.\n\nI want to share them to help people like me who bothered from the features.\n\nhttps://stats.stackexchange.com/questions/263009/encoding-high-cardinality-many-category-categorical-features-when-\n\nhttps://www.quora.com/What-are-some-tricks-to-numerically-encode-high-cardinality-categorical-features-other-than-one-hot-encoding\n\nhttps://www.kdnuggets.com/2016/08/include-high-cardinality-attributes-predictive-model.html\n\nAfter reading these articles, \nI'm trying to apply mean encoding and frequency encoding.\n\nthe script is shown in below kernels\nhttps://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features\nhttps://www.kaggle.com/vchoubey/random-forest-mean-encoding-for-app-channel\n\nI attatched my code of mean and frequency encoding in reply.\n(Please, be aware of indent!)\n\n\nThe choices depend on you. hope this help you!\n", "votes": 9 }, { "id": 400484, "postDate": "2018-10-08T11:53:48.953Z", "content": "Hi, thanks for sharing. I think of using target encoding as well. What models did you use with target encoding (I mean trees, networks) and if you tried this approach with trees models, does it help you improve on cv?
", "rawMarkdown": "Hi, thanks for sharing. I think of using target encoding as well. What models did you use with target encoding (I mean trees, networks) and if you tried this approach with trees models, does it help you improve on cv?", "votes": 1, "replies": [ { "id": 400502, "postDate": "2018-10-08T12:21:43.397Z", "content": "Yes. For now, I'm using both target and frequency encoding with LGBM and catboost.\nI've always checked the improvement from encodings in my CV system. So, both gave me a boost!
", "rawMarkdown": "Yes. For now, I'm using both target and frequency encoding with LGBM and catboost.\nI've always checked the improvement from encodings in my CV system. So, both gave me a boost!" }, { "id": 400512, "postDate": "2018-10-08T12:46:05.467Z", "content": "Thanks) I’ll try to try it))
", "rawMarkdown": "Thanks) I’ll try to try it))" } ] }, { "id": 392164, "postDate": "2018-09-23T06:37:01.643Z", "content": "def frequency_encoding(frame, col):\nfreq_encoding = frame.groupby([col]).size()/frame.shape[0] \nfreq_encoding = freq_encoding.reset_index().rename(columns={0:'{}_Frequency'.format(col)})\nreturn frame.merge(freq_encoding, on=col, how='left')\n+1 for using size instead of count
", "rawMarkdown": "+1 for using size instead of count" } ] }, { "id": 392878, "postDate": "2018-09-24T14:32:18.827Z", "content": "Hello,I have a problem of 617242.....I read the test.csv by using this code:
\n df = pd.read_csv('../input/ga-customer-revenue-prediction/test.csv',\n dtype = {'fullVistorId':'str'})\n df['fullVisitorId']=df['fullVisitorId'].astype(str)\n df['fullVisitorId'].nunique()
and i got the value:620692 but not 617242....
\n\nI really want to know why ......
", "rawMarkdown": "Hello,I have a problem of 617242.....I read the test.csv by using this code: \n df = pd.read_csv('../input/ga-customer-revenue-prediction/test.csv',\n dtype = {'fullVistorId':'str'})\n df['fullVisitorId']=df['fullVisitorId'].astype(str)\n df['fullVisitorId'].nunique()\n\nand i got the value:620692 but not 617242....\n\nI really want to know why ......", "replies": [ { "id": 392883, "postDate": "2018-09-24T14:39:05.640Z", "rawMarkdown": "", "isDeleted": true }, { "id": 392886, "postDate": "2018-09-24T14:45:09.533Z", "content": "Thanks,but i use pd.read_csv(,dtype = {'fullVistorId':'str'})...but i cannot solve it .....can i know how u did it?Thanks!
Just use str, not using 'str'
\n\nThis will fix it!
", "rawMarkdown": "Just use str, not using 'str'\n\nThis will fix it!" }, { "id": 392889, "postDate": "2018-09-24T14:50:29.690Z", "content": "Many thanks....I am so stupid for this problem.....
", "rawMarkdown": "Many thanks....I am so stupid for this problem....." }, { "id": 392890, "postDate": "2018-09-24T14:51:32.310Z", "rawMarkdown": "", "isDeleted": true }, { "id": 392893, "postDate": "2018-09-24T14:55:11.660Z", "content": "I have solved this problem by the help of @YunHan Lee , My error is I use the 'str' but not str when I do cast\nMany thanks！
", "rawMarkdown": "I have solved this problem by the help of @YunHan Lee , My error is I use the 'str' but not str when I do cast\nMany thanks！" } ] }, { "id": 392166, "postDate": "2018-09-23T06:38:32.593Z", "content": "for col in tqdm(categorical_feats):\ntemp_mean_encoding = df_train.groupby([col])[target_name].agg({'{}_Mean'.format(col):'mean', \n '{}_Std'.format(col):'std', \n '{}_Size'.format(col):'size', \n '{}_Sum'.format(col):'sum'}).reset_index()\ndf_train = df_train.merge(temp_mean_encoding, on=col, how='left')\ndf_test = df_test.merge(temp_mean_encoding, on=col, how='left')\nSimple mean encoding is prone to overfitting.
\n\nHere are some examples of the ways to tackle this problem: https://www.kaggle.com/vprokopev/mean-likelihood-encodings-a-comprehensive-study
", "rawMarkdown": "Simple mean encoding is prone to overfitting.\n\nHere are some examples of the ways to tackle this problem: https://www.kaggle.com/vprokopev/mean-likelihood-encodings-a-comprehensive-study" } ] }, { "id": 392739, "postDate": "2018-09-24T08:57:26.937Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 392814, "postDate": "2018-09-24T12:03:30.543Z", "content": "For example, I did target encoding with k-fold \nAnd cross validation.
\n\nBut the random numbers are same in both cases.
\n\nIs this case OK to avoid target leakage?
", "rawMarkdown": "For example, I did target encoding with k-fold \nAnd cross validation.\n\nBut the random numbers are same in both cases.\n\nIs this case OK to avoid target leakage?" }, { "id": 392826, "postDate": "2018-09-24T12:44:44.680Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 400484, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2018-10-08T11:53:48.953000", "content": "Hi, thanks for sharing. I think of using target encoding as well. What models did you use with target encoding (I mean trees, networks) and if you tried this approach with trees models, does it help you improve on cv?
", "votes": 1, "replies": [ { "id": 400502, "author_name": "Youhan Lee", "author_url": "", "post_date": "2018-10-08T12:21:43.397000", "content": "Yes. For now, I'm using both target and frequency encoding with LGBM and catboost.\nI've always checked the improvement from encodings in my CV system. So, both gave me a boost!
", "votes": 0, "replies": [] }, { "id": 400512, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2018-10-08T12:46:05.467000", "content": "Thanks) I’ll try to try it))
", "votes": 0, "replies": [] } ] }, { "id": 392164, "author_name": "Youhan Lee", "author_url": "", "post_date": "2018-09-23T06:37:01.643000", "content": "def frequency_encoding(frame, col):\nfreq_encoding = frame.groupby([col]).size()/frame.shape[0] \nfreq_encoding = freq_encoding.reset_index().rename(columns={0:'{}_Frequency'.format(col)})\nreturn frame.merge(freq_encoding, on=col, how='left')\n+1 for using size instead of count
", "votes": 0, "replies": [] } ] }, { "id": 392878, "author_name": "Neo Zhao", "author_url": "", "post_date": "2018-09-24T14:32:18.827000", "content": "Hello,I have a problem of 617242.....I read the test.csv by using this code:
\n df = pd.read_csv('../input/ga-customer-revenue-prediction/test.csv',\n dtype = {'fullVistorId':'str'})\n df['fullVisitorId']=df['fullVisitorId'].astype(str)\n df['fullVisitorId'].nunique()
and i got the value:620692 but not 617242....
\n\nI really want to know why ......
", "votes": 0, "replies": [ { "id": 392883, "author_name": "", "author_url": "", "post_date": "2018-09-24T14:39:05.640000", "content": "", "votes": 0, "replies": [] }, { "id": 392886, "author_name": "Neo Zhao", "author_url": "", "post_date": "2018-09-24T14:45:09.533000", "content": "Thanks,but i use pd.read_csv(,dtype = {'fullVistorId':'str'})...but i cannot solve it .....can i know how u did it?Thanks!
Just use str, not using 'str'
\n\nThis will fix it!
", "votes": 0, "replies": [] }, { "id": 392889, "author_name": "Neo Zhao", "author_url": "", "post_date": "2018-09-24T14:50:29.690000", "content": "Many thanks....I am so stupid for this problem.....
", "votes": 0, "replies": [] }, { "id": 392890, "author_name": "", "author_url": "", "post_date": "2018-09-24T14:51:32.310000", "content": "", "votes": 0, "replies": [] }, { "id": 392893, "author_name": "Neo Zhao", "author_url": "", "post_date": "2018-09-24T14:55:11.660000", "content": "I have solved this problem by the help of @YunHan Lee , My error is I use the 'str' but not str when I do cast\nMany thanks！
", "votes": 0, "replies": [] } ] }, { "id": 392166, "author_name": "Youhan Lee", "author_url": "", "post_date": "2018-09-23T06:38:32.593000", "content": "for col in tqdm(categorical_feats):\ntemp_mean_encoding = df_train.groupby([col])[target_name].agg({'{}_Mean'.format(col):'mean', \n '{}_Std'.format(col):'std', \n '{}_Size'.format(col):'size', \n '{}_Sum'.format(col):'sum'}).reset_index()\ndf_train = df_train.merge(temp_mean_encoding, on=col, how='left')\ndf_test = df_test.merge(temp_mean_encoding, on=col, how='left')\nSimple mean encoding is prone to overfitting.
\n\nHere are some examples of the ways to tackle this problem: https://www.kaggle.com/vprokopev/mean-likelihood-encodings-a-comprehensive-study
", "votes": 0, "replies": [] } ] }, { "id": 392739, "author_name": "", "author_url": "", "post_date": "2018-09-24T08:57:26.937000", "content": "", "votes": 0, "replies": [ { "id": 392814, "author_name": "Youhan Lee", "author_url": "", "post_date": "2018-09-24T12:03:30.543000", "content": "For example, I did target encoding with k-fold \nAnd cross validation.
\n\nBut the random numbers are same in both cases.
\n\nIs this case OK to avoid target leakage?
", "votes": 0, "replies": [] }, { "id": 392826, "author_name": "", "author_url": "", "post_date": "2018-09-24T12:44:44.680000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "392132": "I struggle with the high cardinality of some features.\nAfter googling, I found good advices.\n\nI want to share them to help people like me who bothered from the features.\n\nhttps://stats.stackexchange.com/questions/263009/encoding-high-cardinality-many-category-categorical-features-when-\n\nhttps://www.quora.com/What-are-some-tricks-to-numerically-encode-high-cardinality-categorical-features-other-than-one-hot-encoding\n\nhttps://www.kdnuggets.com/2016/08/include-high-cardinality-attributes-predictive-model.html\n\nAfter reading these articles, \nI'm trying to apply mean encoding and frequency encoding.\n\nthe script is shown in below kernels\nhttps://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features\nhttps://www.kaggle.com/vchoubey/random-forest-mean-encoding-for-app-channel\n\nI attatched my code of mean and frequency encoding in reply.\n(Please, be aware of indent!)\n\n\nThe choices depend on you. hope this help you!\n", "400484": "Hi, thanks for sharing. I think of using target encoding as well. What models did you use with target encoding (I mean trees, networks) and if you tried this approach with trees models, does it help you improve on cv?", "392164": " def frequency_encoding(frame, col):\n freq_encoding = frame.groupby([col]).size()/frame.shape[0] \n freq_encoding = freq_encoding.reset_index().rename(columns={0:'{}_Frequency'.format(col)})\n return frame.merge(freq_encoding, on=col, how='left')", "392878": "Hello,I have a problem of 617242.....I read the test.csv by using this code: \n df = pd.read_csv('../input/ga-customer-revenue-prediction/test.csv',\n dtype = {'fullVistorId':'str'})\n df['fullVisitorId']=df['fullVisitorId'].astype(str)\n df['fullVisitorId'].nunique()\n\nand i got the value:620692 but not 617242....\n\nI really want to know why ......", "392166": " for col in tqdm(categorical_feats):\n temp_mean_encoding = df_train.groupby([col])[target_name].agg({'{}_Mean'.format(col):'mean', \n '{}_Std'.format(col):'std', \n '{}_Size'.format(col):'size', \n '{}_Sum'.format(col):'sum'}).reset_index()\n df_train = df_train.merge(temp_mean_encoding, on=col, how='left')\n df_test = df_test.merge(temp_mean_encoding, on=col, how='left')", "392739": "" } }