Hello together,\ni computed the coef and intercept (regression) of the purchase_amount by month. It says whether the amount of purchase per card_id goes down or up. This feature has impact in my model. Here is the Code. I also provide you the final regression.csv with the computed data you can join to your train.csv and test.csv.
\n\nhistorical_transactions_df = historical_transactions_df.sort_values(by=['card_id', 'purchase_date'])[['card_id', 'month_lag', 'purchase_amount']]\ncard_ids = list(historical_transactions_df['card_id'].unique())\nout_dfs = [None] * len(card_ids)\na = 0\nagg_func = {'purchase_amount': ['count','mean', 'sum', 'median', 'nunique'],}\nhistorical_transactions_df_by_month_lag = historical_transactions_df.groupby(['card_id', 'month_lag']).agg(agg_func)\nhistorical_transactions_df_by_month_lag.columns = ['_'.join(map(str,col)) for col in historical_transactions_df_by_month_lag.columns]\nregr = LinearRegression()\ncol_names = ['card_id', 'coef', 'intercept']\ndf = pd.DataFrame(columns=col_names)\nfor card_id in card_ids:\n x = list(historical_transactions_df_by_month_lag.loc[card_id].index)\n y = list(historical_transactions_df_by_month_lag.loc[card_id]['purchase_amount_sum'])\n x = [[x] for x in x]\n y = [[x] for x in y]\n reg = regr.fit(x, y)\n coef = (reg.coef_[0][0])\n intercept = (reg.intercept_[0])\n out_dfs[a] = {'card_id': card_id, 'coef': coef, 'intercept': intercept}\n a += 1\nfor col in df.columns:\n if col != 'card_id':\n features.append(col)\ndf_regression = pd.concat([pd.DataFrame([out_dfs[i]], columns=col_names) for i in range(len(list(filter(None, out_dfs))))],ignore_index=True)\nHello together,\ni computed the coef and intercept (regression) of the purchase_amount by month. It says whether the amount of purchase per card_id goes down or up. This feature has impact in my model. Here is the Code. I also provide you the final regression.csv with the computed data you can join to your train.csv and test.csv.
\n\nhistorical_transactions_df = historical_transactions_df.sort_values(by=['card_id', 'purchase_date'])[['card_id', 'month_lag', 'purchase_amount']]\ncard_ids = list(historical_transactions_df['card_id'].unique())\nout_dfs = [None] * len(card_ids)\na = 0\nagg_func = {'purchase_amount': ['count','mean', 'sum', 'median', 'nunique'],}\nhistorical_transactions_df_by_month_lag = historical_transactions_df.groupby(['card_id', 'month_lag']).agg(agg_func)\nhistorical_transactions_df_by_month_lag.columns = ['_'.join(map(str,col)) for col in historical_transactions_df_by_month_lag.columns]\nregr = LinearRegression()\ncol_names = ['card_id', 'coef', 'intercept']\ndf = pd.DataFrame(columns=col_names)\nfor card_id in card_ids:\n x = list(historical_transactions_df_by_month_lag.loc[card_id].index)\n y = list(historical_transactions_df_by_month_lag.loc[card_id]['purchase_amount_sum'])\n x = [[x] for x in x]\n y = [[x] for x in y]\n reg = regr.fit(x, y)\n coef = (reg.coef_[0][0])\n intercept = (reg.intercept_[0])\n out_dfs[a] = {'card_id': card_id, 'coef': coef, 'intercept': intercept}\n a += 1\nfor col in df.columns:\n if col != 'card_id':\n features.append(col)\ndf_regression = pd.concat([pd.DataFrame([out_dfs[i]], columns=col_names) for i in range(len(list(filter(None, out_dfs))))],ignore_index=True)\nIt is a good idea, but they don't improve my model for two reasons. First, each card_id has only two months of transaction so you don't really need linear regression for the slope. All you need is compare the maximum and minimum of monthly sum of the purchases; and the slope is correlated with their difference very strongly. Second, the monthly sum of purchase amounts doesn't really give you the trend. Since the purchase amounts have been normalized, we cannot really tell if someone who makes 100 purchases and each purchase at the price of -0.5 pays less than the other person who pays once at the price of 0.5.
", "rawMarkdown": "It is a good idea, but they don't improve my model for two reasons. First, each card_id has only two months of transaction so you don't really need linear regression for the slope. All you need is compare the maximum and minimum of monthly sum of the purchases; and the slope is correlated with their difference very strongly. Second, the monthly sum of purchase amounts doesn't really give you the trend. Since the purchase amounts have been normalized, we cannot really tell if someone who makes 100 purchases and each purchase at the price of -0.5 pays less than the other person who pays once at the price of 0.5.", "votes": 3, "replies": [ { "id": 465492, "postDate": "2019-02-03T09:07:46.033Z", "content": "agree with ur points
", "rawMarkdown": "agree with ur points", "votes": 1 }, { "id": 465597, "postDate": "2019-02-03T14:50:00.313Z", "content": "Hm.. Actually historical_transactions.csv contains up to 3 months' worth of transactions. The second point i think i didn't unterstand rightly. Why is it a problem that purchase amount were normalized? if the values were only shifted is that no problem. You can always do a regression on it.
\n\nI have specially removed only this feature and run through the model. My LB score got worse by 2 without this feature.
", "rawMarkdown": "Hm.. Actually historical_transactions.csv contains up to 3 months' worth of transactions. The second point i think i didn't unterstand rightly. Why is it a problem that purchase amount were normalized? if the values were only shifted is that no problem. You can always do a regression on it. \n\nI have specially removed only this feature and run through the model. My LB score got worse by 2 without this feature.\n\n" }, { "id": 465612, "postDate": "2019-02-03T15:25:55.930Z", "content": "Yes, your features are important, but I have other features that correlate with your features so they won't help my model. However, I appreciate your creative thinking.\nAfter normalization of purchase amount, -0.5 may mean 50 dollars and 0.5 is 150 dollars. If you purchases 100 times at 50 dollars in the first month, you actually spend $5000. But after normalization, your sum comes up to -50. At the second month, you spend 150 dollars once, so the monthly sum is 0.5 after normalization. It may seem that you purchase more from -50 to 0.5, but you actually spend less in the second month. Your approach shows the trend of numbers, but not the shopping trend of a consumer.
", "rawMarkdown": "Yes, your features are important, but I have other features that correlate with your features so they won't help my model. However, I appreciate your creative thinking.\nAfter normalization of purchase amount, -0.5 may mean 50 dollars and 0.5 is 150 dollars. If you purchases 100 times at 50 dollars in the first month, you actually spend $5000. But after normalization, your sum comes up to -50. At the second month, you spend 150 dollars once, so the monthly sum is 0.5 after normalization. It may seem that you purchase more from -50 to 0.5, but you actually spend less in the second month. Your approach shows the trend of numbers, but not the shopping trend of a consumer.", "votes": 1 }, { "id": 465617, "postDate": "2019-02-03T15:53:27.770Z", "content": "@Tapioca: Wouldn't it help to map the rang EOF purchase amounts to a positive range? E.g. by adding twice the absolute value of the minimum of purchase amounts to purchase amount? Not perfect, though, because if the non-linearity of the scale in your examle, but eventually mitigates the situation of comparing multiple below average transactions with few large value transactions.
", "rawMarkdown": "@Tapioca: Wouldn't it help to map the rang EOF purchase amounts to a positive range? E.g. by adding twice the absolute value of the minimum of purchase amounts to purchase amount? Not perfect, though, because if the non-linearity of the scale in your examle, but eventually mitigates the situation of comparing multiple below average transactions with few large value transactions. " } ] }, { "id": 467411, "postDate": "2019-02-07T03:23:33.087Z", "content": "Hi Oleg,\nThanks for you insight, I tried aggregating purchase_amout sum as well as merchant_id count per month_lag and it gave me a 0.3% improvement in CV (only 0.1% improvement in LB), do you recommend aggregating other features too, (such as sum of installments, or authorised flag)?
", "rawMarkdown": "Hi Oleg,\nThanks for you insight, I tried aggregating purchase_amout sum as well as merchant_id count per month_lag and it gave me a 0.3% improvement in CV (only 0.1% improvement in LB), do you recommend aggregating other features too, (such as sum of installments, or authorised flag)?", "votes": 1, "replies": [ { "id": 467452, "postDate": "2019-02-07T06:44:28.957Z", "content": "Hey,\ngood idea to do that by merchantid. I think it is better to count it unique. You can do the same maybe \"unique\" count of subsector_id, merchant_category_id, category_1, category_2, category_3, category_4, most_recent_purchases_range and merchant_group_id. It could be that a person buys in more and more eg in merchant categories.
", "rawMarkdown": "Hey,\ngood idea to do that by merchantid. I think it is better to count it unique. You can do the same maybe \"unique\" count of subsector_id, merchant_category_id, category_1, category_2, category_3, category_4, most_recent_purchases_range and merchant_group_id. It could be that a person buys in more and more eg in merchant categories. " } ] }, { "id": 466750, "postDate": "2019-02-05T22:15:07.930Z", "content": "@Oleg Knaub, why do you use 'nunique' to agg the purchase amount?
", "rawMarkdown": "@Oleg Knaub, why do you use 'nunique' to agg the purchase amount?", "votes": 1, "replies": [ { "id": 466870, "postDate": "2019-02-06T05:28:53.930Z", "content": "I forgot to delete it but I don't use it. just the sum. Count mean and median I also don't use.
", "rawMarkdown": "I forgot to delete it but I don't use it. just the sum. Count mean and median I also don't use. " } ] }, { "id": 464681, "postDate": "2019-02-01T09:40:51.450Z", "content": "Yes, this is nice feature which is quite useful for time series related problem.
", "rawMarkdown": "Yes, this is nice feature which is quite useful for time series related problem.", "votes": 1, "replies": [ { "id": 464823, "postDate": "2019-02-01T15:15:06.633Z", "content": "It doesn't work for me, how about you?
", "rawMarkdown": "It doesn't work for me, how about you?" }, { "id": 464836, "postDate": "2019-02-01T15:47:14.477Z", "content": "What do you mean it doesn't work for you? The Code or it doesn't have influence on your score?\nFor me importance coef = 1231 and intercept = 1014. Biggest feature has importance of 6000
", "rawMarkdown": "What do you mean it doesn't work for you? The Code or it doesn't have influence on your score?\nFor me importance coef = 1231 and intercept = 1014. Biggest feature has importance of 6000 ", "votes": 1 }, { "id": 465087, "postDate": "2019-02-02T09:45:49.887Z", "content": "maybe the features is similiar to the other u had
", "rawMarkdown": "maybe the features is similiar to the other u had" } ] }, { "id": 464921, "postDate": "2019-02-01T20:04:59.800Z", "content": "Hi! Good idea, but I think more effective to use OLS from statsmodel lib for this case. I sure OLS more faster then your code.
", "rawMarkdown": "Hi! Good idea, but I think more effective to use OLS from statsmodel lib for this case. I sure OLS more faster then your code.", "votes": 2, "replies": [ { "id": 465257, "postDate": "2019-02-02T17:36:10.843Z", "content": "\n\nI tested your theory and don't think it makes any difference.
\n\n%time\n# Using OLS\nimport statsmodels.api as sm\nX = sm.add_constant(x)\nols = sm.OLS(y, X)\nresults = ols.fit()\nresults.params\n\n\n\n\n\nCPU times: user 2 µs, sys: 0 ns, total: 2 µs
\n
\n Wall time: 3.34 µs
\n array([-24.1123882 , -0.51466783])
%time\n# Using linear regression\nregr = LinearRegression()\nreg = regr.fit(x, y)\ncoef = (reg.coef_[0][0])\nintercept = (reg.intercept_[0])\nprint(coef, intercept)\n\n", "rawMarkdown": "@andrilko\n\nI tested your theory and don't think it makes any difference.\n\n\n %time\n # Using OLS\n import statsmodels.api as sm\n X = sm.add_constant(x)\n ols = sm.OLS(y, X)\n results = ols.fit()\n results.params\n\n>> CPU times: user 2 µs, sys: 0 ns, total: 2 µs \n>> Wall time: 3.34 µs \n>> array([-24.1123882 , -0.51466783])\n\n\n %time\n # Using linear regression\n regr = LinearRegression()\n reg = regr.fit(x, y)\n coef = (reg.coef_[0][0])\n intercept = (reg.intercept_[0])\n print(coef, intercept)\n\n>> CPU times: user 2 µs, sys: 0 ns, total: 2 µs \n>> Wall time: 3.58 µs \n>> -0.5146678253844545 -24.112388202122283", "votes": 3 } ] }, { "id": 465185, "postDate": "2019-02-02T14:53:10.253Z", "rawMarkdown": "", "isDeleted": true }, { "id": 464931, "postDate": "2019-02-01T20:46:18.330Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 465407, "author_name": "Tapioca", "author_url": "", "post_date": "2019-02-03T03:00:28.003000", "content": "\n\nCPU times: user 2 µs, sys: 0 ns, total: 2 µs
\n
\n Wall time: 3.58 µs
\n -0.5146678253844545 -24.112388202122283
It is a good idea, but they don't improve my model for two reasons. First, each card_id has only two months of transaction so you don't really need linear regression for the slope. All you need is compare the maximum and minimum of monthly sum of the purchases; and the slope is correlated with their difference very strongly. Second, the monthly sum of purchase amounts doesn't really give you the trend. Since the purchase amounts have been normalized, we cannot really tell if someone who makes 100 purchases and each purchase at the price of -0.5 pays less than the other person who pays once at the price of 0.5.
", "votes": 3, "replies": [ { "id": 465492, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-02-03T09:07:46.033000", "content": "agree with ur points
", "votes": 1, "replies": [] }, { "id": 465597, "author_name": "Oleg Knaub", "author_url": "", "post_date": "2019-02-03T14:50:00.313000", "content": "Hm.. Actually historical_transactions.csv contains up to 3 months' worth of transactions. The second point i think i didn't unterstand rightly. Why is it a problem that purchase amount were normalized? if the values were only shifted is that no problem. You can always do a regression on it.
\n\nI have specially removed only this feature and run through the model. My LB score got worse by 2 without this feature.
", "votes": 0, "replies": [] }, { "id": 465612, "author_name": "Tapioca", "author_url": "", "post_date": "2019-02-03T15:25:55.930000", "content": "Yes, your features are important, but I have other features that correlate with your features so they won't help my model. However, I appreciate your creative thinking.\nAfter normalization of purchase amount, -0.5 may mean 50 dollars and 0.5 is 150 dollars. If you purchases 100 times at 50 dollars in the first month, you actually spend $5000. But after normalization, your sum comes up to -50. At the second month, you spend 150 dollars once, so the monthly sum is 0.5 after normalization. It may seem that you purchase more from -50 to 0.5, but you actually spend less in the second month. Your approach shows the trend of numbers, but not the shopping trend of a consumer.
", "votes": 1, "replies": [] }, { "id": 465617, "author_name": "lennart", "author_url": "", "post_date": "2019-02-03T15:53:27.770000", "content": "@Tapioca: Wouldn't it help to map the rang EOF purchase amounts to a positive range? E.g. by adding twice the absolute value of the minimum of purchase amounts to purchase amount? Not perfect, though, because if the non-linearity of the scale in your examle, but eventually mitigates the situation of comparing multiple below average transactions with few large value transactions.
", "votes": 0, "replies": [] } ] }, { "id": 467411, "author_name": "JinCui", "author_url": "", "post_date": "2019-02-07T03:23:33.087000", "content": "Hi Oleg,\nThanks for you insight, I tried aggregating purchase_amout sum as well as merchant_id count per month_lag and it gave me a 0.3% improvement in CV (only 0.1% improvement in LB), do you recommend aggregating other features too, (such as sum of installments, or authorised flag)?
", "votes": 1, "replies": [ { "id": 467452, "author_name": "Oleg Knaub", "author_url": "", "post_date": "2019-02-07T06:44:28.957000", "content": "Hey,\ngood idea to do that by merchantid. I think it is better to count it unique. You can do the same maybe \"unique\" count of subsector_id, merchant_category_id, category_1, category_2, category_3, category_4, most_recent_purchases_range and merchant_group_id. It could be that a person buys in more and more eg in merchant categories.
", "votes": 0, "replies": [] } ] }, { "id": 466750, "author_name": "LiwenHuang", "author_url": "", "post_date": "2019-02-05T22:15:07.930000", "content": "@Oleg Knaub, why do you use 'nunique' to agg the purchase amount?
", "votes": 1, "replies": [ { "id": 466870, "author_name": "Oleg Knaub", "author_url": "", "post_date": "2019-02-06T05:28:53.930000", "content": "I forgot to delete it but I don't use it. just the sum. Count mean and median I also don't use.
", "votes": 0, "replies": [] } ] }, { "id": 464681, "author_name": "LongYin/杰少", "author_url": "", "post_date": "2019-02-01T09:40:51.450000", "content": "Yes, this is nice feature which is quite useful for time series related problem.
", "votes": 1, "replies": [ { "id": 464823, "author_name": "Ethan", "author_url": "", "post_date": "2019-02-01T15:15:06.633000", "content": "It doesn't work for me, how about you?
", "votes": 0, "replies": [] }, { "id": 464836, "author_name": "Oleg Knaub", "author_url": "", "post_date": "2019-02-01T15:47:14.477000", "content": "What do you mean it doesn't work for you? The Code or it doesn't have influence on your score?\nFor me importance coef = 1231 and intercept = 1014. Biggest feature has importance of 6000
", "votes": 1, "replies": [] }, { "id": 465087, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-02-02T09:45:49.887000", "content": "maybe the features is similiar to the other u had
", "votes": 0, "replies": [] } ] }, { "id": 464921, "author_name": "Alex Abramov", "author_url": "", "post_date": "2019-02-01T20:04:59.800000", "content": "Hi! Good idea, but I think more effective to use OLS from statsmodel lib for this case. I sure OLS more faster then your code.
", "votes": 2, "replies": [ { "id": 465257, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-02-02T17:36:10.843000", "content": "\n\nI tested your theory and don't think it makes any difference.
\n\n%time\n# Using OLS\nimport statsmodels.api as sm\nX = sm.add_constant(x)\nols = sm.OLS(y, X)\nresults = ols.fit()\nresults.params\n\n\n\n\n\nCPU times: user 2 µs, sys: 0 ns, total: 2 µs
\n
\n Wall time: 3.34 µs
\n array([-24.1123882 , -0.51466783])
%time\n# Using linear regression\nregr = LinearRegression()\nreg = regr.fit(x, y)\ncoef = (reg.coef_[0][0])\nintercept = (reg.intercept_[0])\nprint(coef, intercept)\n\n", "votes": 3, "replies": [] } ] }, { "id": 465185, "author_name": "", "author_url": "", "post_date": "2019-02-02T14:53:10.253000", "content": "", "votes": 0, "replies": [] }, { "id": 464931, "author_name": "", "author_url": "", "post_date": "2019-02-01T20:46:18.330000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "464668": "Hello together,\ni computed the coef and intercept (regression) of the purchase_amount by month. It says whether the amount of purchase per card_id goes down or up. This feature has impact in my model. Here is the Code. I also provide you the final regression.csv with the computed data you can join to your train.csv and test.csv. \n\n historical_transactions_df = historical_transactions_df.sort_values(by=['card_id', 'purchase_date'])[['card_id', 'month_lag', 'purchase_amount']]\n card_ids = list(historical_transactions_df['card_id'].unique())\n out_dfs = [None] * len(card_ids)\n a = 0\n agg_func = {'purchase_amount': ['count','mean', 'sum', 'median', 'nunique'],}\n historical_transactions_df_by_month_lag = historical_transactions_df.groupby(['card_id', 'month_lag']).agg(agg_func)\n historical_transactions_df_by_month_lag.columns = ['_'.join(map(str,col)) for col in historical_transactions_df_by_month_lag.columns]\n regr = LinearRegression()\n col_names = ['card_id', 'coef', 'intercept']\n df = pd.DataFrame(columns=col_names)\n for card_id in card_ids:\n x = list(historical_transactions_df_by_month_lag.loc[card_id].index)\n y = list(historical_transactions_df_by_month_lag.loc[card_id]['purchase_amount_sum'])\n x = [[x] for x in x]\n y = [[x] for x in y]\n reg = regr.fit(x, y)\n coef = (reg.coef_[0][0])\n intercept = (reg.intercept_[0])\n out_dfs[a] = {'card_id': card_id, 'coef': coef, 'intercept': intercept}\n a += 1\n for col in df.columns:\n if col != 'card_id':\n features.append(col)\n df_regression = pd.concat([pd.DataFrame([out_dfs[i]], columns=col_names) for i in range(len(list(filter(None, out_dfs))))],ignore_index=True)", "465407": "It is a good idea, but they don't improve my model for two reasons. First, each card_id has only two months of transaction so you don't really need linear regression for the slope. All you need is compare the maximum and minimum of monthly sum of the purchases; and the slope is correlated with their difference very strongly. Second, the monthly sum of purchase amounts doesn't really give you the trend. Since the purchase amounts have been normalized, we cannot really tell if someone who makes 100 purchases and each purchase at the price of -0.5 pays less than the other person who pays once at the price of 0.5.", "467411": "Hi Oleg,\nThanks for you insight, I tried aggregating purchase_amout sum as well as merchant_id count per month_lag and it gave me a 0.3% improvement in CV (only 0.1% improvement in LB), do you recommend aggregating other features too, (such as sum of installments, or authorised flag)?", "466750": "@Oleg Knaub, why do you use 'nunique' to agg the purchase amount?", "464681": "Yes, this is nice feature which is quite useful for time series related problem.", "464921": "Hi! Good idea, but I think more effective to use OLS from statsmodel lib for this case. I sure OLS more faster then your code.", "465185": "", "464931": "" } }\n\nCPU times: user 2 µs, sys: 0 ns, total: 2 µs
\n
\n Wall time: 3.58 µs
\n -0.5146678253844545 -24.112388202122283