Hi Kagglers,\nIt seems that it's quite difficult to predict the target!!\nI tried to run an LGBM classification instead of regression, labeling 1 for all those that purchased in the target period.\nMy AUC score is going around 0.52~, which is quite terrible.
\n\ndo you have any easier time predicting visitors that will come back to purchase?
", "messages": [ { "id": 420441, "postDate": "2018-11-13T16:22:05.557Z", "content": "Hi Kagglers,\nIt seems that it's quite difficult to predict the target!!\nI tried to run an LGBM classification instead of regression, labeling 1 for all those that purchased in the target period.\nMy AUC score is going around 0.52~, which is quite terrible.
\n\ndo you have any easier time predicting visitors that will come back to purchase?
", "rawMarkdown": "Hi Kagglers,\nIt seems that it's quite difficult to predict the target!!\nI tried to run an LGBM classification instead of regression, labeling 1 for all those that purchased in the target period.\nMy AUC score is going around 0.52~, which is quite terrible.\n\ndo you have any easier time predicting visitors that will come back to purchase?\n", "votes": 16 }, { "id": 423371, "postDate": "2018-11-18T03:55:13.730Z", "content": "I've tryed to aggregate data every 5 months and predict the revenue between 45 and 105 days after.
\n\nSo I have the following dates:
\n\nFeatures from 2016-08-01 to 2017-01-15 and predict revenue from 2017-03-01 to 2017-05-01\nFeatures from 2017-05-01 to 2017-10-15 and predict revenue from 2017-12-01 to 2018-02-01\n...
\n\nThis gaves me around 1M examples to train, but only 400 returning customers with revenue!
\n\nUsing min, mean, max and a few ratio features with lgbm classifier I got (using 5-fold):
\n\nAuc: 0.936\nRecall: 0.035\nPrecision: 0.205
\n\nThe classifier predicted 68 positive labels and only 14 were truly positives, so the classifier was able to predict only 14 of the 400 customers that returned and spent some money...
\n\nI got .3473 rmse, but predictions are totally off. The highest predicted revenue (log) is around 10, while the lowest real value (greater than zero) is around 14.
", "rawMarkdown": "I've tryed to aggregate data every 5 months and predict the revenue between 45 and 105 days after.\n\nSo I have the following dates:\n\nFeatures from 2016-08-01 to 2017-01-15 and predict revenue from 2017-03-01 to 2017-05-01\nFeatures from 2017-05-01 to 2017-10-15 and predict revenue from 2017-12-01 to 2018-02-01\n...\n\nHi @Aguiar,\nThanks for sharing!
", "rawMarkdown": "Hi @Aguiar,\nThanks for sharing!" }, { "id": 424998, "postDate": "2018-11-21T01:45:19.177Z", "content": "wow you got .347rmse, i got 0.573rmse. and predicted revenue is around 4.4,
", "rawMarkdown": "wow you got .347rmse, i got 0.573rmse. and predicted revenue is around 4.4, " }, { "id": 426050, "postDate": "2018-11-22T14:12:37.930Z", "content": "Hi @Aguiar,
\n\nAre you sure .3473 is rsme but not mse? In previous comment you written: full-zero answer gives .32.\n~.32 is my mse score for zero-answer which corresponds to 0.573 rmse from comment above.
", "rawMarkdown": "Hi @Aguiar,\n\nAre you sure .3473 is rsme but not mse? In previous comment you written: full-zero answer gives .32.\n~.32 is my mse score for zero-answer which corresponds to 0.573 rmse from comment above." } ] }, { "id": 424390, "postDate": "2018-11-20T03:09:42.197Z", "content": "Hi everyone,
\n\nI am studying an approach very similar to the one some of you have mentioned before:
\n\nhttps://github.com/luisduque440/google_analytics
\n\nSo far, I have mostly focused on doing feature engineering, which basically consists on getting, for each fullvisitorid in the training period, the following features:\n* Fixed fields (for categorical variables): i.e. country, browser, in the trainining period\n* Moving average of the numerical features in the training period, for example: total_transaction_revenue_last month, last_2_months, last_3_months, ...). ( 'last month', 'last two months', ... refer to periods of time computed backwards from the last day of the training period)\n* Average value of the numerical features per month in the training period (i.e. total_transactio_revenue 2 months ago, 3 months ago, 4 months ago, ...). ('two months ago', '3 months ago', are again, computed based on the last day of training period)\n* The total spent in the evaluation data set for each fullvisitor id in the training period
\n\nTo do this, I consider several periods of 9 months and split them in the exact same way the Kaggle competition does,:
\n\n<---------------- train period (168 days)---------------------------><-------gap (46 days)-------><---evaluation(62 days)----->\n|-------------------------------------------------------------------------|---------------------------------|---------------------------------|
\n\nI am still testing this code, but seems to be working well.
\n\nBest,
\n\nLuis
", "rawMarkdown": "Hi everyone, \n\nI am studying an approach very similar to the one some of you have mentioned before:\n\nhttps://github.com/luisduque440/google_analytics\n\n So far, I have mostly focused on doing feature engineering, which basically consists on getting, for each fullvisitorid in the training period, the following features:\n* Fixed fields (for categorical variables): i.e. country, browser, in the trainining period\n* Moving average of the numerical features in the training period, for example: total_transaction_revenue_last month, last_2_months, last_3_months, ...). ( 'last month', 'last two months', ... refer to periods of time computed backwards from the last day of the training period)\n* Average value of the numerical features per month in the training period (i.e. total_transactio_revenue 2 months ago, 3 months ago, 4 months ago, ...). ('two months ago', '3 months ago', are again, computed based on the last day of training period)\n* The total spent in the evaluation data set for each fullvisitor id in the training period\n\nTo do this, I consider several periods of 9 months and split them in the exact same way the Kaggle competition does,: \n\n<---------------- train period (168 days)---------------------------><-------gap (46 days)-------><---evaluation(62 days)----->\n|-------------------------------------------------------------------------|---------------------------------|---------------------------------|\n\nI am still testing this code, but seems to be working well.\n\nBest,\n\nLuis\n", "votes": 6, "replies": [ { "id": 424540, "postDate": "2018-11-20T09:32:19.027Z", "content": "Hi @luisduque440,\nThanks for sharing!\n2 questions:\n1. What is your validation CV RMSE?\n2. in your train-gap-evaluation, which visitors are you training on?
\n\nThanks
", "rawMarkdown": "Hi @luisduque440,\nThanks for sharing!\n2 questions:\n1. What is your validation CV RMSE?\n2. in your train-gap-evaluation, which visitors are you training on?\n\nThanks" } ] }, { "id": 424850, "postDate": "2018-11-20T19:10:49.750Z", "content": "Hi everyone,
\nI want to do the same thing that you guys do, but seems i work not good..
\nthis is my kernel about it, https://www.kaggle.com/super13579/ga-v2-future-purchase-prediction
\nI only got few positive label on my model... is it normal ?(only 3 true positive for 246 customer will back purchase...)
\nThis is my data set :
\nTraining period set 1==> 2016/08/01 ~ 2017/1/15 (5.5 month)
\nTarget period set 1 ==> 2017/03/1 ~ 2017/04/30 (2 month)
\nTraining period set 2==> 2017/06/01 ~ 2017/11/15 (5.5 month)
\nTarget period set 2 ==> 2018/1/1 ~ 2018/02/30 (2 month)
\nconcat two set training period to final training data
Hi All \nUpdate regression result of my CV
\nvalid target period is Dec-1-2017 and Jan-31-2018 same with @AmirH
\nRMSE model prediction : 0.32178
\nRMSE for all zeros: 0.32496
Looks like have same result....
", "rawMarkdown": "Hi All \nUpdate regression result of my CV \nvalid target period is Dec-1-2017 and Jan-31-2018 same with @AmirH \nRMSE model prediction : 0.32178 \nRMSE for all zeros: 0.32496 \n\nLooks like have same result.... " } ] }, { "id": 426053, "postDate": "2018-11-22T14:15:56.303Z", "content": "My results:\nfull-zero rmse: ~0.58\nfull-zero mse: ~0.32
\n\nmodel rmse: ~0.48\nmodel mse: ~0.24
", "rawMarkdown": "My results:\nfull-zero rmse: ~0.58\nfull-zero mse: ~0.32\n\nmodel rmse: ~0.48\nmodel mse: ~0.24", "votes": 1, "replies": [ { "id": 426061, "postDate": "2018-11-22T14:41:27.750Z", "content": "What regression algorithm do you use?
", "rawMarkdown": "What regression algorithm do you use?" }, { "id": 426180, "postDate": "2018-11-22T19:46:00.237Z", "content": "XGB
", "rawMarkdown": "XGB" }, { "id": 426193, "postDate": "2018-11-22T20:35:21.523Z", "content": "I find a bit strange that the mse of our 'full-zero mse' are so different, mine is:\nfull-zero mse = 0.1086 \nMy validation set has 388283 rows, of them, only 133 have non-zero target, and the sum of them is 2364
", "rawMarkdown": "I find a bit strange that the mse of our 'full-zero mse' are so different, mine is:\nfull-zero mse = 0.1086 \nMy validation set has 388283 rows, of them, only 133 have non-zero target, and the sum of them is 2364" }, { "id": 426607, "postDate": "2018-11-23T14:34:42.310Z", "content": "What logic of each row from your validation array?\nDoes is corresponds with visitorId or visitid?
", "rawMarkdown": "What logic of each row from your validation array?\nDoes is corresponds with visitorId or visitid?" }, { "id": 426659, "postDate": "2018-11-23T15:57:43.247Z", "content": "One row per fullvisitorid.
", "rawMarkdown": "One row per fullvisitorid." } ] }, { "id": 422082, "postDate": "2018-11-15T19:01:31.830Z", "content": "If you want us to share CV, you should share how you are calculating CV. On the last 1.5 months of all data? Including test/train data? Removing the gap? Etc. Without sharing this, our \"numbers\" are meaningless . . !
", "rawMarkdown": "If you want us to share CV, you should share how you are calculating CV. On the last 1.5 months of all data? Including test/train data? Removing the gap? Etc. Without sharing this, our \"numbers\" are meaningless . . !", "votes": 1, "replies": [ { "id": 422142, "postDate": "2018-11-15T20:46:38.180Z", "content": "Hi CoreyLevinson,\nI created 2 sets of data:\nTraining + 5\\10 fold CV: Train on data from Aug-1-2016 to Nov-30-2017 with data for visitors that visited between May-1-2017 and Oct-15-2017, with label of whether they purchased between Dec-1-2017 and Jan-31-2018
\n\nthis replicates the actual task, one year back.
\n\nI created a similar set with a bit of different dates that act as a test set to see the prediction score on.\nI used LGBM binary classification of whether there was a purchase in the target period, using AUC metric.
", "rawMarkdown": "Hi CoreyLevinson,\nI created 2 sets of data:\nTraining + 5\\10 fold CV: Train on data from Aug-1-2016 to Nov-30-2017 with data for visitors that visited between May-1-2017 and Oct-15-2017, with label of whether they purchased between Dec-1-2017 and Jan-31-2018\n\nthis replicates the actual task, one year back.\n\nI created a similar set with a bit of different dates that act as a test set to see the prediction score on.\nI used LGBM binary classification of whether there was a purchase in the target period, using AUC metric.\n", "votes": 2 }, { "id": 422149, "postDate": "2018-11-15T20:57:15.070Z", "content": "Ok. I had been doing the same, and my L2 error was around 2e15 (lol) using 3 folds. I'm rerunning my kernel for binary classification now.
", "rawMarkdown": "Ok. I had been doing the same, and my L2 error was around 2e15 (lol) using 3 folds. I'm rerunning my kernel for binary classification now." }, { "id": 422158, "postDate": "2018-11-15T21:15:38.073Z", "content": "valid's auc:0.796595+0.00362031 (this is with 3 folds again; no tuning)
Edit: Because of small dataset it is extremely noisy. Just by rerunning it a few times I can game it:
\n\nvalid's auc:0.811204+0.0107608
Hi Corey,\nThanks for sharing.\nWhat's the ratio between positive and negative labels in your data?\nmaybe i'm doing something wrong =(
", "rawMarkdown": "Hi Corey,\nThanks for sharing.\nWhat's the ratio between positive and negative labels in your data?\nmaybe i'm doing something wrong =(" }, { "id": 422175, "postDate": "2018-11-15T21:52:00.723Z", "content": "OMG Thanks @CoreyLevinson!!\nyour comment got me to scratch my head and dig more and more into my code just to find a stupid bug with mislabeling my data. (sorting issue)\nNow my score is 0.9704 in 3 fold CV\nand 0.9759 in completely unseen data.
\n\nThanks a lot!!
", "rawMarkdown": "OMG Thanks @CoreyLevinson!!\nyour comment got me to scratch my head and dig more and more into my code just to find a stupid bug with mislabeling my data. (sorting issue)\nNow my score is 0.9704 in 3 fold CV\nand 0.9759 in completely unseen data.\n\nThanks a lot!!", "votes": 1 }, { "id": 422178, "postDate": "2018-11-15T21:53:30.587Z", "content": "Yea, I had a bug too. I forgot to do log1p of the sum of transactions, that's why my RMSE was 2e15. But your AUC of 0.97 is very impressive. I thought that my feature engineering was very good, but obviously not good enough. Current L2 is now: valid's l2:12.4344+0.73334
This is my validation:
\n\n6751 rows\n6464 negatives\n287 positives
\n\nDo you match? I am unfortunately including data past Oct 15 all the way to Nov 31 because I coded it without realizing the gap. I need to go back and recorrect it. But do your numbers match this kind of?
", "rawMarkdown": "Yea, I had a bug too. I forgot to do log1p of the sum of transactions, that's why my RMSE was 2e15. But your AUC of 0.97 is very impressive. I thought that my feature engineering was very good, but obviously not good enough. Current L2 is now: `valid's l2:12.4344+0.73334 `\n\nThis is my validation:\n\n6751 rows\n6464 negatives\n287 positives\n\nDo you match? I am unfortunately including data past Oct 15 all the way to Nov 31 because I coded it without realizing the gap. I need to go back and recorrect it. But do your numbers match this kind of?", "votes": 1 }, { "id": 422186, "postDate": "2018-11-15T22:04:23.873Z", "content": "Hi @CoreyLevinson,\nmy training set is 329636 rows, out of those 106 are positive\nQuite different from yours.\nThis is based on the data i wrote in the post:
\n\nTrain on data from Aug-1-2016 to Nov-30-2017 with data for visitors that visited between May-1-2017 and Oct-15-2017, with label of whether they purchased between Dec-1-2017 and Jan-31-2018
", "rawMarkdown": "Hi @CoreyLevinson,\nmy training set is 329636 rows, out of those 106 are positive\nQuite different from yours.\nThis is based on the data i wrote in the post:\n\nTrain on data from Aug-1-2016 to Nov-30-2017 with data for visitors that visited between May-1-2017 and Oct-15-2017, with label of whether they purchased between Dec-1-2017 and Jan-31-2018\n", "votes": 2 }, { "id": 422191, "postDate": "2018-11-15T22:17:42.477Z", "content": "Oh, I know what I'm doing. I'm including all people who visited between Dec 1 2017 - Jan 31 2018. even if they weren't present in May 1 2017 - Oct 15 2017. So like, I was including people whose last visit might have been in January 2017 but just happened to visit in Dec 1 2017 - Jan 31 2018.
", "rawMarkdown": "Oh, I know what I'm doing. I'm including all people who visited between Dec 1 2017 - Jan 31 2018. even if they weren't present in May 1 2017 - Oct 15 2017. So like, I was including people whose last visit might have been in January 2017 but just happened to visit in Dec 1 2017 - Jan 31 2018." }, { "id": 422198, "postDate": "2018-11-15T22:37:18.700Z", "content": "I understand,\nI personally think this method is risky because it is different than our actual train\\\"test\" set distribution.\nBut I might be wrong.
", "rawMarkdown": "I understand,\nI personally think this method is risky because it is different than our actual train\\\"test\" set distribution.\nBut I might be wrong." }, { "id": 422200, "postDate": "2018-11-15T22:45:23.663Z", "content": "I concur; I am changing my kernel to fix it. But at the same time, I think this competition as a whole is risky. Why? We are training on barely any data. And if there is only around 100 people who actually purchase during test period, I feel it's like a lottery.
", "rawMarkdown": "I concur; I am changing my kernel to fix it. But at the same time, I think this competition as a whole is risky. Why? We are training on barely any data. And if there is only around 100 people who actually purchase during test period, I feel it's like a lottery." }, { "id": 422223, "postDate": "2018-11-15T23:43:27.630Z", "content": "That's the problem... 100 positive cases in 330,000 examples. This is more an anomally detection problem than revenue prediction. What's your recall and precision score?
", "rawMarkdown": "That's the problem... 100 positive cases in 330,000 examples. This is more an anomally detection problem than revenue prediction. What's your recall and precision score?", "votes": 3 }, { "id": 422268, "postDate": "2018-11-16T02:03:18.793Z", "content": "In your training data , did you aggregate your fullVisitorId ?
", "rawMarkdown": "In your training data , did you aggregate your fullVisitorId ?" }, { "id": 422415, "postDate": "2018-11-16T07:28:08.157Z", "content": "I have the same configuration than yours amirH (>99% non revenue visitor), rmse is 0.32 (if I submit only zeros, the score is the same :-D).\nRobustness cannot be here\nI think I m going to give up this comp
", "rawMarkdown": "I have the same configuration than yours amirH (>99% non revenue visitor), rmse is 0.32 (if I submit only zeros, the score is the same :-D).\nRobustness cannot be here\nI think I m going to give up this comp" }, { "id": 422426, "postDate": "2018-11-16T07:49:23Z", "content": "Don't give up Vincent!\nI almost gave up until i found that bug in my code. \nI'll run regression with rmse and let you know how it goes in a couple of hours, maybe it'll give you some insight
", "rawMarkdown": "Don't give up Vincent!\nI almost gave up until i found that bug in my code. \nI'll run regression with rmse and let you know how it goes in a couple of hours, maybe it'll give you some insight ", "votes": 3 }, { "id": 422456, "postDate": "2018-11-16T08:47:53.347Z", "content": "Can I ask why do you guys check AUC instead of the original metric? Do I miss something?
", "rawMarkdown": "Can I ask why do you guys check AUC instead of the original metric? Do I miss something?" }, { "id": 422495, "postDate": "2018-11-16T09:56:49.420Z", "content": "Hi Ahmet,\nI'm no pro, but from what i know:
\n\nRMSE can even give you the same prediction for all observations and still be very low.\nturning this into a classification is just one way of giving you information as to how well your model actually distinguishes between visitors that returned to buy and those that didn't (AUC score, confusion matrix..).\nYou can do it with RMSE also by inspecting your predictions after the training, i just like the classification method better.
", "rawMarkdown": "Hi Ahmet,\nI'm no pro, but from what i know:\n\nRMSE can even give you the same prediction for all observations and still be very low.\nturning this into a classification is just one way of giving you information as to how well your model actually distinguishes between visitors that returned to buy and those that didn't (AUC score, confusion matrix..).\nYou can do it with RMSE also by inspecting your predictions after the training, i just like the classification method better." }, { "id": 422500, "postDate": "2018-11-16T10:14:20.017Z", "content": "Hi @Vincent Larmet,
\n\nWell, no good news. all zeros RMSE is 0.3276\nand RMSE of the model is 0.3236
\n\nbut hey, it's something! O_O
", "rawMarkdown": "Hi @Vincent Larmet,\n\nWell, no good news. all zeros RMSE is 0.3276\nand RMSE of the model is 0.3236\n\nbut hey, it's something! O_O", "votes": 2 }, { "id": 422516, "postDate": "2018-11-16T10:34:07.707Z", "content": "yep, same here!\nModel RMSE is very slightly better...
", "rawMarkdown": "yep, same here!\nModel RMSE is very slightly better...\n" }, { "id": 422666, "postDate": "2018-11-16T15:39:29.413Z", "content": "Hi all.
\n\nJust wondering, how is your train set 329636 rows?
\n\ntrain <- fread(\"../input/save-out-csv/train.csv\", nrows=rows_to_read, colClasses=c(date='character'))\ntest <- fread(\"../input/save-out-csv/test.csv\", nrows=rows_to_read, colClasses=c(date='character'))\ntr_te <- rbind(train, test)\n\ntrain <- tr_te[(date <= as.Date('2017-10-15'))]\nvisiting_period <- tr_te[date >= as.Date('2017-05-01') & date <= as.Date('2017-10-15')]\nvalid <- tr_te[(date >= as.Date('2017-12-01') & date <= as.Date('2018-01-31'))]\n\nnrow(train[fullVisitorId %in% unique(visiting_period$fullVisitorId) & fullVisitorId %in% \nunique(valid$fullVisitorId)])\n10116
I have only 10,116 rows of train data. Am I missing something?
", "rawMarkdown": "Hi all.\n\nJust wondering, how is your train set 329636 rows?\n\n train <- fread(\"../input/save-out-csv/train.csv\", nrows=rows_to_read, colClasses=c(date='character'))\n test <- fread(\"../input/save-out-csv/test.csv\", nrows=rows_to_read, colClasses=c(date='character'))\n tr_te <- rbind(train, test)\n\n train <- tr_te[(date <= as.Date('2017-10-15'))]\n visiting_period <- tr_te[date >= as.Date('2017-05-01') & date <= as.Date('2017-10-15')]\n valid <- tr_te[(date >= as.Date('2017-12-01') & date <= as.Date('2018-01-31'))]\n\n nrow(train[fullVisitorId %in% unique(visiting_period$fullVisitorId) & fullVisitorId %in% \n unique(valid$fullVisitorId)])\n\n`10116`\n\nI have only 10,116 rows of train data. Am I missing something?" }, { "id": 422667, "postDate": "2018-11-16T15:42:47.240Z", "content": "And I have only 2,488 unique fullVisitorId's in the valid period. But still, only 106 positive values, the same as you. So maybe you all are counting a lot of extra rows that shouldn't be in training ?
Edit: Oh, I see. you are training on rows all the way up to Nov 30. That implies that you know how to requery the dataset on Nov 30, which I do not know how to do. Hopefully @emorej can release the data on Nov 30, or tell us how to requery so we can do it ourselves.
", "rawMarkdown": "And I have only 2,488 unique fullVisitorId's in the `valid` period. But still, only 106 positive values, the same as you. So maybe you all are counting a lot of extra rows that shouldn't be in training ?\n\nEdit: Oh, I see. you are training on rows all the way up to Nov 30. That implies that you know how to requery the dataset on Nov 30, which I do not know how to do. Hopefully @emorej can release the data on Nov 30, or tell us how to requery so we can do it ourselves." }, { "id": 422711, "postDate": "2018-11-16T16:56:02.820Z", "content": "Corey i think the issue is that you are training only on those that visited in the valid period and you try to predict if they buy or only visit , where i trained on all visitors, even if they didn't come back.
", "rawMarkdown": "Corey i think the issue is that you are training only on those that visited in the valid period and you try to predict if they buy or only visit , where i trained on all visitors, even if they didn't come back. ", "votes": 1 }, { "id": 422732, "postDate": "2018-11-16T17:34:53.497Z", "rawMarkdown": "", "isDeleted": true }, { "id": 423234, "postDate": "2018-11-17T18:33:32.463Z", "content": "@AmirH the second dataset that you have created (LGBM with binary classification) serves the purpose to validate the visit of customers during the \"Testing period\". Please correct me if i am wrong.
", "rawMarkdown": "@AmirH the second dataset that you have created (LGBM with binary classification) serves the purpose to validate the visit of customers during the \"Testing period\". Please correct me if i am wrong." }, { "id": 423316, "postDate": "2018-11-17T22:29:16Z", "content": "It's to classify whether a visitor will purchase in the target period, in comparison to how much he would spend.
", "rawMarkdown": "It's to classify whether a visitor will purchase in the target period, in comparison to how much he would spend.\n" }, { "id": 424763, "postDate": "2018-11-20T16:20:43.773Z", "content": "I have similar results.
\n\nall 0: 0.32554
\n\nmy pred: 0.32322
", "rawMarkdown": "I have similar results.\n\nall 0: 0.32554\n\nmy pred: 0.32322" }, { "id": 425240, "postDate": "2018-11-21T10:22:06.030Z", "content": "Instead of AUC I am using R2. It scales the RMSE by the RMSE you get if you do the baseline (a model that predicts only the mean). My R2 are around 10% though :)
", "rawMarkdown": "Instead of AUC I am using R2. It scales the RMSE by the RMSE you get if you do the baseline (a model that predicts only the mean). My R2 are around 10% though :)" }, { "id": 425791, "postDate": "2018-11-22T05:36:59.437Z", "content": "Using similar CV strategy but I guess have different train/test split.\nNon the less my results are as follows:\nRMSE model prediction: 0.29832\nRMSE for all zeros: 0.29372
\n\nI think in reality this problem is not solvable in any useful manner based on the data we have.\nThe main problem is that barely any one plans apparel purchase 45 to 100 days in advance.\nIn the data set U can see that almost all users visit the site and purchase at most within 20 days.
\n\nI think it's still worth trying :)
", "rawMarkdown": "Using similar CV strategy but I guess have different train/test split.\nNon the less my results are as follows:\nRMSE model prediction: 0.29832\nRMSE for all zeros: 0.29372\n\nI think in reality this problem is not solvable in any useful manner based on the data we have.\nThe main problem is that barely any one plans apparel purchase 45 to 100 days in advance.\nIn the data set U can see that almost all users visit the site and purchase at most within 20 days.\n\nI think it's still worth trying :)", "votes": 1 } ] }, { "id": 421063, "postDate": "2018-11-14T14:24:28.507Z", "content": "I think the score is very related to how you set up your validation. I've tryed to aggregate customers in each 5 months interval and predict the next two - check item 4.
\n\nThe problem was that very few customers purchased in this two months so just predicting zero for every fullVisitorId gives a really good rmse (around .32). I'll try a classifier to see how it does..
", "rawMarkdown": "I think the score is very related to how you set up your validation. I've tryed to aggregate customers in each 5 months interval and predict the next two - [check item 4](https://www.kaggle.com/jsaguiar/v2-quick-eda-what-s-new). \n\nThe problem was that very few customers purchased in this two months so just predicting zero for every fullVisitorId gives a really good rmse (around .32). I'll try a classifier to see how it does..", "votes": 1, "replies": [ { "id": 421209, "postDate": "2018-11-14T18:09:52.927Z", "content": "Hi Aguiar,\nthanks for your comment, Let me know how the classification goes!\nCheck your AUC score.
\n\nAlso - notice that trying to predict next 2 months is not our case, we have data until Oct 15 and we start predicting Dec1st\nso there's a gap there.
", "rawMarkdown": "Hi Aguiar,\nthanks for your comment, Let me know how the classification goes!\nCheck your AUC score.\n\nAlso - notice that trying to predict next 2 months is not our case, we have data until Oct 15 and we start predicting Dec1st\nso there's a gap there.\n", "votes": 1 }, { "id": 421932, "postDate": "2018-11-15T15:23:01.740Z", "content": "For this validation scheme I had only 120 returning customers with revenue... So the classifier predicted only 5 labels 1, but AUC was 0.94 with 0 recall and 0 precision (I don't even know if that is possible). Accuracy was obviously > 0.99.
\n\nI forgot to mention, but I've included the gap before the two months (as in the notebook in the link).
", "rawMarkdown": "For this validation scheme I had only 120 returning customers with revenue... So the classifier predicted only 5 labels 1, but AUC was 0.94 with 0 recall and 0 precision (I don't even know if that is possible). Accuracy was obviously > 0.99.\n\nI forgot to mention, but I've included the gap before the two months (as in the notebook in the link). ", "votes": 1 }, { "id": 422007, "postDate": "2018-11-15T17:01:27.280Z", "content": "Hi aguiar,\nI'm pretty sure something is wrong with your numbers , your auc sounds veeeeery high.
\n\nMaybe it's auc of your training and not of your out of fold validation?
\n\nI also was able to overfit training to get high auc 0.9~, but then out of fold set the model didn't yet see had auc of 0.48, that's awful.
", "rawMarkdown": "Hi aguiar,\nI'm pretty sure something is wrong with your numbers , your auc sounds veeeeery high. \n\nMaybe it's auc of your training and not of your out of fold validation? \n\nI also was able to overfit training to get high auc 0.9~, but then out of fold set the model didn't yet see had auc of 0.48, that's awful. " } ] }, { "id": 430700, "postDate": "2018-11-30T21:27:00.430Z", "content": "train1 = train[(train['date']< '2017-01-15 00:00:00')]\ngap1 = train[(train['date']> '2017-01-15 00:00:00') & (train['date'] < '2017-02-28 00:00:00')]\ntest1 = train[(train['date']> '2017-03-01 00:00:00') & (train['date'] < '2017-05-01 00:00:00')]
\n\ntrain2 = train[(train['date'] > '2017-01-15 00:00:00') & (train['date']< '2017-07-01 00:00:00')]\ngap2 = train[(train['date'] > '2017-07-01 00:00:00') & (train['date'] < '2017-08-15 00:00:00')]\ntest2 = train[(train['date']> '2017-08-15 00:00:00') & (train['date'] < '2017-10-15 00:00:00')]
\n\ntrain3 = train[(train['date'] > '2017-07-01 00:00:00') & (train['date']< '2018-01-15 00:00:00')]\ngap3 = train[(train['date'] > '2018-01-15 00:00:00') & (train['date'] < '2018-03-01 00:00:00')]\ntest3 = train[(train['date']> '2018-03-01 00:00:00') & (train['date'] < '2018-10-15 00:00:00')]
\n\nI took target from testx for trainx( x in [ 1, 2, 3 ] )\nI used 5.5 months for train, 1.5 months or gap, 2 months for test.
\n\nI started train2 on where gap1 started, I started train3 where gap2 started,
\n\nI concated train1, train2, train2 row by row, got 1 million 77k rows for training. After I trained it, I scored on test data.\nApproximately, 0.3% weren't 0.
\n\nI extracted approximately 70 features(not enough but I didn't have so much time)
\n\nFull-zero rmse with 5 fold validation: 0.33134\nSingle lightgbm model with 5 fold validation: 0.3090
", "rawMarkdown": "train1 = train[(train['date']< '2017-01-15 00:00:00')]\ngap1 = train[(train['date']> '2017-01-15 00:00:00') & (train['date'] < '2017-02-28 00:00:00')]\ntest1 = train[(train['date']> '2017-03-01 00:00:00') & (train['date'] < '2017-05-01 00:00:00')]\n\ntrain2 = train[(train['date'] > '2017-01-15 00:00:00') & (train['date']< '2017-07-01 00:00:00')]\ngap2 = train[(train['date'] > '2017-07-01 00:00:00') & (train['date'] < '2017-08-15 00:00:00')]\ntest2 = train[(train['date']> '2017-08-15 00:00:00') & (train['date'] < '2017-10-15 00:00:00')]\n\ntrain3 = train[(train['date'] > '2017-07-01 00:00:00') & (train['date']< '2018-01-15 00:00:00')]\ngap3 = train[(train['date'] > '2018-01-15 00:00:00') & (train['date'] < '2018-03-01 00:00:00')]\ntest3 = train[(train['date']> '2018-03-01 00:00:00') & (train['date'] < '2018-10-15 00:00:00')]\n\nI took target from testx for trainx( x in [ 1, 2, 3 ] )\nI used 5.5 months for train, 1.5 months or gap, 2 months for test.\n\nI started train2 on where gap1 started, I started train3 where gap2 started, \n\nI concated train1, train2, train2 row by row, got 1 million 77k rows for training. After I trained it, I scored on test data.\nApproximately, 0.3% weren't 0.\n\nI extracted approximately 70 features(not enough but I didn't have so much time)\n\nFull-zero rmse with 5 fold validation: 0.33134\nSingle lightgbm model with 5 fold validation: 0.3090\n\n" } ], "comments": [ { "id": 423371, "author_name": "Aguiar", "author_url": "", "post_date": "2018-11-18T03:55:13.730000", "content": "I've tryed to aggregate data every 5 months and predict the revenue between 45 and 105 days after.
\n\nSo I have the following dates:
\n\nFeatures from 2016-08-01 to 2017-01-15 and predict revenue from 2017-03-01 to 2017-05-01\nFeatures from 2017-05-01 to 2017-10-15 and predict revenue from 2017-12-01 to 2018-02-01\n...
\n\nThis gaves me around 1M examples to train, but only 400 returning customers with revenue!
\n\nUsing min, mean, max and a few ratio features with lgbm classifier I got (using 5-fold):
\n\nAuc: 0.936\nRecall: 0.035\nPrecision: 0.205
\n\nThe classifier predicted 68 positive labels and only 14 were truly positives, so the classifier was able to predict only 14 of the 400 customers that returned and spent some money...
\n\nI got .3473 rmse, but predictions are totally off. The highest predicted revenue (log) is around 10, while the lowest real value (greater than zero) is around 14.
", "votes": 7, "replies": [ { "id": 423402, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-18T06:54:29.193000", "content": "Hi @Aguiar,\nThanks for sharing!
", "votes": 0, "replies": [] }, { "id": 424998, "author_name": "Kang0709", "author_url": "", "post_date": "2018-11-21T01:45:19.177000", "content": "wow you got .347rmse, i got 0.573rmse. and predicted revenue is around 4.4,
", "votes": 0, "replies": [] }, { "id": 426050, "author_name": "Egor", "author_url": "", "post_date": "2018-11-22T14:12:37.930000", "content": "Hi @Aguiar,
\n\nAre you sure .3473 is rsme but not mse? In previous comment you written: full-zero answer gives .32.\n~.32 is my mse score for zero-answer which corresponds to 0.573 rmse from comment above.
", "votes": 0, "replies": [] } ] }, { "id": 424390, "author_name": "luisduque440", "author_url": "", "post_date": "2018-11-20T03:09:42.197000", "content": "Hi everyone,
\n\nI am studying an approach very similar to the one some of you have mentioned before:
\n\nhttps://github.com/luisduque440/google_analytics
\n\nSo far, I have mostly focused on doing feature engineering, which basically consists on getting, for each fullvisitorid in the training period, the following features:\n* Fixed fields (for categorical variables): i.e. country, browser, in the trainining period\n* Moving average of the numerical features in the training period, for example: total_transaction_revenue_last month, last_2_months, last_3_months, ...). ( 'last month', 'last two months', ... refer to periods of time computed backwards from the last day of the training period)\n* Average value of the numerical features per month in the training period (i.e. total_transactio_revenue 2 months ago, 3 months ago, 4 months ago, ...). ('two months ago', '3 months ago', are again, computed based on the last day of training period)\n* The total spent in the evaluation data set for each fullvisitor id in the training period
\n\nTo do this, I consider several periods of 9 months and split them in the exact same way the Kaggle competition does,:
\n\n<---------------- train period (168 days)---------------------------><-------gap (46 days)-------><---evaluation(62 days)----->\n|-------------------------------------------------------------------------|---------------------------------|---------------------------------|
\n\nI am still testing this code, but seems to be working well.
\n\nBest,
\n\nLuis
", "votes": 6, "replies": [ { "id": 424540, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-20T09:32:19.027000", "content": "Hi @luisduque440,\nThanks for sharing!\n2 questions:\n1. What is your validation CV RMSE?\n2. in your train-gap-evaluation, which visitors are you training on?
\n\nThanks
", "votes": 0, "replies": [] } ] }, { "id": 424850, "author_name": "0xFunky", "author_url": "", "post_date": "2018-11-20T19:10:49.750000", "content": "Hi everyone,
\nI want to do the same thing that you guys do, but seems i work not good..
\nthis is my kernel about it, https://www.kaggle.com/super13579/ga-v2-future-purchase-prediction
\nI only got few positive label on my model... is it normal ?(only 3 true positive for 246 customer will back purchase...)
\nThis is my data set :
\nTraining period set 1==> 2016/08/01 ~ 2017/1/15 (5.5 month)
\nTarget period set 1 ==> 2017/03/1 ~ 2017/04/30 (2 month)
\nTraining period set 2==> 2017/06/01 ~ 2017/11/15 (5.5 month)
\nTarget period set 2 ==> 2018/1/1 ~ 2018/02/30 (2 month)
\nconcat two set training period to final training data
Hi All \nUpdate regression result of my CV
\nvalid target period is Dec-1-2017 and Jan-31-2018 same with @AmirH
\nRMSE model prediction : 0.32178
\nRMSE for all zeros: 0.32496
Looks like have same result....
", "votes": 0, "replies": [] } ] }, { "id": 426053, "author_name": "Egor", "author_url": "", "post_date": "2018-11-22T14:15:56.303000", "content": "My results:\nfull-zero rmse: ~0.58\nfull-zero mse: ~0.32
\n\nmodel rmse: ~0.48\nmodel mse: ~0.24
", "votes": 1, "replies": [ { "id": 426061, "author_name": "luisduque440", "author_url": "", "post_date": "2018-11-22T14:41:27.750000", "content": "What regression algorithm do you use?
", "votes": 0, "replies": [] }, { "id": 426180, "author_name": "Egor", "author_url": "", "post_date": "2018-11-22T19:46:00.237000", "content": "XGB
", "votes": 0, "replies": [] }, { "id": 426193, "author_name": "luisduque440", "author_url": "", "post_date": "2018-11-22T20:35:21.523000", "content": "I find a bit strange that the mse of our 'full-zero mse' are so different, mine is:\nfull-zero mse = 0.1086 \nMy validation set has 388283 rows, of them, only 133 have non-zero target, and the sum of them is 2364
", "votes": 0, "replies": [] }, { "id": 426607, "author_name": "Egor", "author_url": "", "post_date": "2018-11-23T14:34:42.310000", "content": "What logic of each row from your validation array?\nDoes is corresponds with visitorId or visitid?
", "votes": 0, "replies": [] }, { "id": 426659, "author_name": "luisduque440", "author_url": "", "post_date": "2018-11-23T15:57:43.247000", "content": "One row per fullvisitorid.
", "votes": 0, "replies": [] } ] }, { "id": 422082, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-11-15T19:01:31.830000", "content": "If you want us to share CV, you should share how you are calculating CV. On the last 1.5 months of all data? Including test/train data? Removing the gap? Etc. Without sharing this, our \"numbers\" are meaningless . . !
", "votes": 1, "replies": [ { "id": 422142, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-15T20:46:38.180000", "content": "Hi CoreyLevinson,\nI created 2 sets of data:\nTraining + 5\\10 fold CV: Train on data from Aug-1-2016 to Nov-30-2017 with data for visitors that visited between May-1-2017 and Oct-15-2017, with label of whether they purchased between Dec-1-2017 and Jan-31-2018
\n\nthis replicates the actual task, one year back.
\n\nI created a similar set with a bit of different dates that act as a test set to see the prediction score on.\nI used LGBM binary classification of whether there was a purchase in the target period, using AUC metric.
", "votes": 2, "replies": [] }, { "id": 422149, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-11-15T20:57:15.070000", "content": "Ok. I had been doing the same, and my L2 error was around 2e15 (lol) using 3 folds. I'm rerunning my kernel for binary classification now.
", "votes": 0, "replies": [] }, { "id": 422158, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-11-15T21:15:38.073000", "content": "valid's auc:0.796595+0.00362031 (this is with 3 folds again; no tuning)
Edit: Because of small dataset it is extremely noisy. Just by rerunning it a few times I can game it:
\n\nvalid's auc:0.811204+0.0107608
Hi Corey,\nThanks for sharing.\nWhat's the ratio between positive and negative labels in your data?\nmaybe i'm doing something wrong =(
", "votes": 0, "replies": [] }, { "id": 422175, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-15T21:52:00.723000", "content": "OMG Thanks @CoreyLevinson!!\nyour comment got me to scratch my head and dig more and more into my code just to find a stupid bug with mislabeling my data. (sorting issue)\nNow my score is 0.9704 in 3 fold CV\nand 0.9759 in completely unseen data.
\n\nThanks a lot!!
", "votes": 1, "replies": [] }, { "id": 422178, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-11-15T21:53:30.587000", "content": "Yea, I had a bug too. I forgot to do log1p of the sum of transactions, that's why my RMSE was 2e15. But your AUC of 0.97 is very impressive. I thought that my feature engineering was very good, but obviously not good enough. Current L2 is now: valid's l2:12.4344+0.73334
This is my validation:
\n\n6751 rows\n6464 negatives\n287 positives
\n\nDo you match? I am unfortunately including data past Oct 15 all the way to Nov 31 because I coded it without realizing the gap. I need to go back and recorrect it. But do your numbers match this kind of?
", "votes": 1, "replies": [] }, { "id": 422186, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-15T22:04:23.873000", "content": "Hi @CoreyLevinson,\nmy training set is 329636 rows, out of those 106 are positive\nQuite different from yours.\nThis is based on the data i wrote in the post:
\n\nTrain on data from Aug-1-2016 to Nov-30-2017 with data for visitors that visited between May-1-2017 and Oct-15-2017, with label of whether they purchased between Dec-1-2017 and Jan-31-2018
", "votes": 2, "replies": [] }, { "id": 422191, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-11-15T22:17:42.477000", "content": "Oh, I know what I'm doing. I'm including all people who visited between Dec 1 2017 - Jan 31 2018. even if they weren't present in May 1 2017 - Oct 15 2017. So like, I was including people whose last visit might have been in January 2017 but just happened to visit in Dec 1 2017 - Jan 31 2018.
", "votes": 0, "replies": [] }, { "id": 422198, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-15T22:37:18.700000", "content": "I understand,\nI personally think this method is risky because it is different than our actual train\\\"test\" set distribution.\nBut I might be wrong.
", "votes": 0, "replies": [] }, { "id": 422200, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-11-15T22:45:23.663000", "content": "I concur; I am changing my kernel to fix it. But at the same time, I think this competition as a whole is risky. Why? We are training on barely any data. And if there is only around 100 people who actually purchase during test period, I feel it's like a lottery.
", "votes": 0, "replies": [] }, { "id": 422223, "author_name": "Aguiar", "author_url": "", "post_date": "2018-11-15T23:43:27.630000", "content": "That's the problem... 100 positive cases in 330,000 examples. This is more an anomally detection problem than revenue prediction. What's your recall and precision score?
", "votes": 3, "replies": [] }, { "id": 422268, "author_name": "Kang0709", "author_url": "", "post_date": "2018-11-16T02:03:18.793000", "content": "In your training data , did you aggregate your fullVisitorId ?
", "votes": 0, "replies": [] }, { "id": 422415, "author_name": "Vincent Larmet", "author_url": "", "post_date": "2018-11-16T07:28:08.157000", "content": "I have the same configuration than yours amirH (>99% non revenue visitor), rmse is 0.32 (if I submit only zeros, the score is the same :-D).\nRobustness cannot be here\nI think I m going to give up this comp
", "votes": 0, "replies": [] }, { "id": 422426, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-16T07:49:23", "content": "Don't give up Vincent!\nI almost gave up until i found that bug in my code. \nI'll run regression with rmse and let you know how it goes in a couple of hours, maybe it'll give you some insight
", "votes": 3, "replies": [] }, { "id": 422456, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-11-16T08:47:53.347000", "content": "Can I ask why do you guys check AUC instead of the original metric? Do I miss something?
", "votes": 0, "replies": [] }, { "id": 422495, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-16T09:56:49.420000", "content": "Hi Ahmet,\nI'm no pro, but from what i know:
\n\nRMSE can even give you the same prediction for all observations and still be very low.\nturning this into a classification is just one way of giving you information as to how well your model actually distinguishes between visitors that returned to buy and those that didn't (AUC score, confusion matrix..).\nYou can do it with RMSE also by inspecting your predictions after the training, i just like the classification method better.
", "votes": 0, "replies": [] }, { "id": 422500, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-16T10:14:20.017000", "content": "Hi @Vincent Larmet,
\n\nWell, no good news. all zeros RMSE is 0.3276\nand RMSE of the model is 0.3236
\n\nbut hey, it's something! O_O
", "votes": 2, "replies": [] }, { "id": 422516, "author_name": "Vincent Larmet", "author_url": "", "post_date": "2018-11-16T10:34:07.707000", "content": "yep, same here!\nModel RMSE is very slightly better...
", "votes": 0, "replies": [] }, { "id": 422666, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-11-16T15:39:29.413000", "content": "Hi all.
\n\nJust wondering, how is your train set 329636 rows?
\n\ntrain <- fread(\"../input/save-out-csv/train.csv\", nrows=rows_to_read, colClasses=c(date='character'))\ntest <- fread(\"../input/save-out-csv/test.csv\", nrows=rows_to_read, colClasses=c(date='character'))\ntr_te <- rbind(train, test)\n\ntrain <- tr_te[(date <= as.Date('2017-10-15'))]\nvisiting_period <- tr_te[date >= as.Date('2017-05-01') & date <= as.Date('2017-10-15')]\nvalid <- tr_te[(date >= as.Date('2017-12-01') & date <= as.Date('2018-01-31'))]\n\nnrow(train[fullVisitorId %in% unique(visiting_period$fullVisitorId) & fullVisitorId %in% \nunique(valid$fullVisitorId)])\n10116
I have only 10,116 rows of train data. Am I missing something?
", "votes": 0, "replies": [] }, { "id": 422667, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-11-16T15:42:47.240000", "content": "And I have only 2,488 unique fullVisitorId's in the valid period. But still, only 106 positive values, the same as you. So maybe you all are counting a lot of extra rows that shouldn't be in training ?
Edit: Oh, I see. you are training on rows all the way up to Nov 30. That implies that you know how to requery the dataset on Nov 30, which I do not know how to do. Hopefully @emorej can release the data on Nov 30, or tell us how to requery so we can do it ourselves.
", "votes": 0, "replies": [] }, { "id": 422711, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-16T16:56:02.820000", "content": "Corey i think the issue is that you are training only on those that visited in the valid period and you try to predict if they buy or only visit , where i trained on all visitors, even if they didn't come back.
", "votes": 1, "replies": [] }, { "id": 422732, "author_name": "", "author_url": "", "post_date": "2018-11-16T17:34:53.497000", "content": "", "votes": 0, "replies": [] }, { "id": 423234, "author_name": "pooh", "author_url": "", "post_date": "2018-11-17T18:33:32.463000", "content": "@AmirH the second dataset that you have created (LGBM with binary classification) serves the purpose to validate the visit of customers during the \"Testing period\". Please correct me if i am wrong.
", "votes": 0, "replies": [] }, { "id": 423316, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-17T22:29:16", "content": "It's to classify whether a visitor will purchase in the target period, in comparison to how much he would spend.
", "votes": 0, "replies": [] }, { "id": 424763, "author_name": "atfujita", "author_url": "", "post_date": "2018-11-20T16:20:43.773000", "content": "I have similar results.
\n\nall 0: 0.32554
\n\nmy pred: 0.32322
", "votes": 0, "replies": [] }, { "id": 425240, "author_name": "Burrito Dan", "author_url": "", "post_date": "2018-11-21T10:22:06.030000", "content": "Instead of AUC I am using R2. It scales the RMSE by the RMSE you get if you do the baseline (a model that predicts only the mean). My R2 are around 10% though :)
", "votes": 0, "replies": [] }, { "id": 425791, "author_name": "Michal Gdak", "author_url": "", "post_date": "2018-11-22T05:36:59.437000", "content": "Using similar CV strategy but I guess have different train/test split.\nNon the less my results are as follows:\nRMSE model prediction: 0.29832\nRMSE for all zeros: 0.29372
\n\nI think in reality this problem is not solvable in any useful manner based on the data we have.\nThe main problem is that barely any one plans apparel purchase 45 to 100 days in advance.\nIn the data set U can see that almost all users visit the site and purchase at most within 20 days.
\n\nI think it's still worth trying :)
", "votes": 1, "replies": [] } ] }, { "id": 421063, "author_name": "Aguiar", "author_url": "", "post_date": "2018-11-14T14:24:28.507000", "content": "I think the score is very related to how you set up your validation. I've tryed to aggregate customers in each 5 months interval and predict the next two - check item 4.
\n\nThe problem was that very few customers purchased in this two months so just predicting zero for every fullVisitorId gives a really good rmse (around .32). I'll try a classifier to see how it does..
", "votes": 1, "replies": [ { "id": 421209, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-14T18:09:52.927000", "content": "Hi Aguiar,\nthanks for your comment, Let me know how the classification goes!\nCheck your AUC score.
\n\nAlso - notice that trying to predict next 2 months is not our case, we have data until Oct 15 and we start predicting Dec1st\nso there's a gap there.
", "votes": 1, "replies": [] }, { "id": 421932, "author_name": "Aguiar", "author_url": "", "post_date": "2018-11-15T15:23:01.740000", "content": "For this validation scheme I had only 120 returning customers with revenue... So the classifier predicted only 5 labels 1, but AUC was 0.94 with 0 recall and 0 precision (I don't even know if that is possible). Accuracy was obviously > 0.99.
\n\nI forgot to mention, but I've included the gap before the two months (as in the notebook in the link).
", "votes": 1, "replies": [] }, { "id": 422007, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-15T17:01:27.280000", "content": "Hi aguiar,\nI'm pretty sure something is wrong with your numbers , your auc sounds veeeeery high.
\n\nMaybe it's auc of your training and not of your out of fold validation?
\n\nI also was able to overfit training to get high auc 0.9~, but then out of fold set the model didn't yet see had auc of 0.48, that's awful.
", "votes": 0, "replies": [] } ] }, { "id": 430700, "author_name": "MuhammedBuyukkinaci", "author_url": "", "post_date": "2018-11-30T21:27:00.430000", "content": "train1 = train[(train['date']< '2017-01-15 00:00:00')]\ngap1 = train[(train['date']> '2017-01-15 00:00:00') & (train['date'] < '2017-02-28 00:00:00')]\ntest1 = train[(train['date']> '2017-03-01 00:00:00') & (train['date'] < '2017-05-01 00:00:00')]
\n\ntrain2 = train[(train['date'] > '2017-01-15 00:00:00') & (train['date']< '2017-07-01 00:00:00')]\ngap2 = train[(train['date'] > '2017-07-01 00:00:00') & (train['date'] < '2017-08-15 00:00:00')]\ntest2 = train[(train['date']> '2017-08-15 00:00:00') & (train['date'] < '2017-10-15 00:00:00')]
\n\ntrain3 = train[(train['date'] > '2017-07-01 00:00:00') & (train['date']< '2018-01-15 00:00:00')]\ngap3 = train[(train['date'] > '2018-01-15 00:00:00') & (train['date'] < '2018-03-01 00:00:00')]\ntest3 = train[(train['date']> '2018-03-01 00:00:00') & (train['date'] < '2018-10-15 00:00:00')]
\n\nI took target from testx for trainx( x in [ 1, 2, 3 ] )\nI used 5.5 months for train, 1.5 months or gap, 2 months for test.
\n\nI started train2 on where gap1 started, I started train3 where gap2 started,
\n\nI concated train1, train2, train2 row by row, got 1 million 77k rows for training. After I trained it, I scored on test data.\nApproximately, 0.3% weren't 0.
\n\nI extracted approximately 70 features(not enough but I didn't have so much time)
\n\nFull-zero rmse with 5 fold validation: 0.33134\nSingle lightgbm model with 5 fold validation: 0.3090
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "420441": "Hi Kagglers,\nIt seems that it's quite difficult to predict the target!!\nI tried to run an LGBM classification instead of regression, labeling 1 for all those that purchased in the target period.\nMy AUC score is going around 0.52~, which is quite terrible.\n\ndo you have any easier time predicting visitors that will come back to purchase?\n", "423371": "I've tryed to aggregate data every 5 months and predict the revenue between 45 and 105 days after.\n\nSo I have the following dates:\n\nFeatures from 2016-08-01 to 2017-01-15 and predict revenue from 2017-03-01 to 2017-05-01\nFeatures from 2017-05-01 to 2017-10-15 and predict revenue from 2017-12-01 to 2018-02-01\n...\n\n