Hi Kagglers,
\n\nI'm getting really frustrated with this competition.\nSeems like anything I do that generates better CV, gets worse LB.\nMy position is pretty bad.
\n\nI started out using time based validation with 3 last months as validation set.\nI had some 1.44~ submissions using this validation.
\n\nIt's easier for me to use 5 fold validation as i can also use these folds for mean encodings, OOF stacking, averaged CV score , implementing oof binary classification to use in model etc.
\n\nAs I don't see any reason not to use 5 fold CV here, i generated 5 Folds by group of fullVisitorId. (all rows of a specific fullvisitorId are always in the same fold).\nSince doing that, anything i seem to be doing , without any high risk things such as mean encodings etc. I get scores of 1.45-1.47\neven using only simple features such as:
\n\n\"channelGrouping\", \"visitNumber\", \"browser\", \"operatingSystem\", \"isMobile\", \"deviceCategory\", \"continent\", \"subContinent\", \"country\", \"region\", \"metro\", \"city\", \"networkDomain\", \"hits\", \"pageviews\", \"newVisits\", \"bounces\", \"source\", \"medium\", \"keyword\", \"isTrueDirect\", \"referralPath\", \"VisitMonth\", \"VisitDay\", \"VisitWeekDay\", \"VisitHour\", \"CountVisitorVisits\", \"Location\", \"IsMulti\", \"Maxhits\", \"Meanhits\", \"Medianhits\", \"Maxpageviews\", \"Meanpageviews\", \"Medianpageviews\", \"DaysActive\"
\n\n(categories are label encoded)\nwith LGBM with parameters:
\n\nlist(objective = \"regression\",\n metric = \"RMSE\",\n learning_rate= 0.03,\n num_leaves= 40, #16\n #max_depth= 12, #3\n #max_bin= 255,\n subsample= 0.9,\n subsample_freq= 1,\n colsample_bytree=0.9)
\n\nI use lgbm's lgb.cv with my 5folds and then retrain on entire data with best.iteration.
\n\nAny suggestions?\nI'm really frustrated.
\n\nI don't like forking and using kernels much, I also use R\nI saw @Olivier using 5 fold validation with very simple model and getting better scores.\nI tried session level, user level.. I just don't understand!!\nDoes anyone else experience the same?
\n\nWhat CV method are you using?
How does one do mean encoding \\ OOF predictions when using 1 set of train and 1 set of validation, without having folds?
Thanks!\nAmir.
", "messages": [ { "id": 396235, "postDate": "2018-09-30T09:52:59.003Z", "content": "Hi Kagglers,
\n\nI'm getting really frustrated with this competition.\nSeems like anything I do that generates better CV, gets worse LB.\nMy position is pretty bad.
\n\nI started out using time based validation with 3 last months as validation set.\nI had some 1.44~ submissions using this validation.
\n\nIt's easier for me to use 5 fold validation as i can also use these folds for mean encodings, OOF stacking, averaged CV score , implementing oof binary classification to use in model etc.
\n\nAs I don't see any reason not to use 5 fold CV here, i generated 5 Folds by group of fullVisitorId. (all rows of a specific fullvisitorId are always in the same fold).\nSince doing that, anything i seem to be doing , without any high risk things such as mean encodings etc. I get scores of 1.45-1.47\neven using only simple features such as:
\n\n\"channelGrouping\", \"visitNumber\", \"browser\", \"operatingSystem\", \"isMobile\", \"deviceCategory\", \"continent\", \"subContinent\", \"country\", \"region\", \"metro\", \"city\", \"networkDomain\", \"hits\", \"pageviews\", \"newVisits\", \"bounces\", \"source\", \"medium\", \"keyword\", \"isTrueDirect\", \"referralPath\", \"VisitMonth\", \"VisitDay\", \"VisitWeekDay\", \"VisitHour\", \"CountVisitorVisits\", \"Location\", \"IsMulti\", \"Maxhits\", \"Meanhits\", \"Medianhits\", \"Maxpageviews\", \"Meanpageviews\", \"Medianpageviews\", \"DaysActive\"
\n\n(categories are label encoded)\nwith LGBM with parameters:
\n\nlist(objective = \"regression\",\n metric = \"RMSE\",\n learning_rate= 0.03,\n num_leaves= 40, #16\n #max_depth= 12, #3\n #max_bin= 255,\n subsample= 0.9,\n subsample_freq= 1,\n colsample_bytree=0.9)
\n\nI use lgbm's lgb.cv with my 5folds and then retrain on entire data with best.iteration.
\n\nAny suggestions?\nI'm really frustrated.
\n\nI don't like forking and using kernels much, I also use R\nI saw @Olivier using 5 fold validation with very simple model and getting better scores.\nI tried session level, user level.. I just don't understand!!\nDoes anyone else experience the same?
\n\nWhat CV method are you using?
How does one do mean encoding \\ OOF predictions when using 1 set of train and 1 set of validation, without having folds?
Thanks!\nAmir.
", "rawMarkdown": "Hi Kagglers,\n\nI'm getting really frustrated with this competition.\nSeems like anything I do that generates better CV, gets worse LB.\nMy position is pretty bad.\n\nI started out using time based validation with 3 last months as validation set.\nI had some 1.44~ submissions using this validation.\n\nIt's easier for me to use 5 fold validation as i can also use these folds for mean encodings, OOF stacking, averaged CV score , implementing oof binary classification to use in model etc.\n\nAs I don't see any reason not to use 5 fold CV here, i generated 5 Folds by group of fullVisitorId. (all rows of a specific fullvisitorId are always in the same fold).\nSince doing that, anything i seem to be doing , without any high risk things such as mean encodings etc. I get scores of 1.45-1.47\neven using only simple features such as:\n\n\n\n\"channelGrouping\", \"visitNumber\", \"browser\", \"operatingSystem\", \"isMobile\", \"deviceCategory\", \"continent\", \"subContinent\", \"country\", \"region\", \"metro\", \"city\", \"networkDomain\", \"hits\", \"pageviews\", \"newVisits\", \"bounces\", \"source\", \"medium\", \"keyword\", \"isTrueDirect\", \"referralPath\", \"VisitMonth\", \"VisitDay\", \"VisitWeekDay\", \"VisitHour\", \"CountVisitorVisits\", \"Location\", \"IsMulti\", \"Maxhits\", \"Meanhits\", \"Medianhits\", \"Maxpageviews\", \"Meanpageviews\", \"Medianpageviews\", \"DaysActive\" \n\n\n (categories are label encoded)\nwith LGBM with parameters:\n\nlist(objective = \"regression\",\n metric = \"RMSE\",\n learning_rate= 0.03,\n num_leaves= 40, #16\n #max_depth= 12, #3\n #max_bin= 255,\n subsample= 0.9,\n subsample_freq= 1,\n colsample_bytree=0.9)\n\n\n\nI use lgbm's lgb.cv with my 5folds and then retrain on entire data with best.iteration.\n \nAny suggestions?\nI'm really frustrated.\n\nI don't like forking and using kernels much, I also use R\nI saw @Olivier using 5 fold validation with very simple model and getting better scores.\nI tried session level, user level.. I just don't understand!!\nDoes anyone else experience the same?\n\n\n- What CV method are you using?\n\n- How does one do mean encoding \\ OOF predictions when using 1 set of train and 1 set of validation, without having folds?\n\n\nThanks!\nAmir.\n\n", "votes": 12 }, { "id": 396322, "postDate": "2018-09-30T13:13:43.360Z", "content": "@AmirH
\n\nIt is true that this competition is a little bit strange !
\n\nHere are 'my' best params I normally use on this file :
\n\nparams = {'objective': 'reg:linear',\n 'eval_metric': 'rmse',\n 'eta': 0.001,\n 'max_depth': 8,\n 'subsample': 0.7,\n 'colsample_bytree': 0.8,\n 'alpha':0.001,\n 'num_leaves' : 63,\n 'random_state': 42,\n 'silent': True,\n }
\n\nDon't forget early_stopping : I usually use a low 20 rounds compared to a lot of kernels you can find. The higher the risk of overfitting is possible
\n\n'subsample': 0.7 is the number of random rows you use and 'colsample_bytree': 0.8 the same for colums to use. This add randomness to your evaluation. 0.9 is according to me most of the times too high.
\n\nThere is no standard rule, it depends on your -own- file. Just play and try, and don't forget to keep track of your Local and LB scores.
\n\nGood luck !
", "rawMarkdown": "@AmirH\n\nIt is true that this competition is a little bit strange ! \n\nHere are 'my' best params I normally use on this file : \n\nparams = {'objective': 'reg:linear',\n 'eval_metric': 'rmse',\n 'eta': 0.001,\n 'max_depth': 8,\n 'subsample': 0.7,\n 'colsample_bytree': 0.8,\n 'alpha':0.001,\n 'num_leaves' : 63,\n 'random_state': 42,\n 'silent': True,\n }\n\nDon't forget early_stopping : I usually use a low 20 rounds compared to a lot of kernels you can find. The higher the risk of overfitting is possible\n\n'**subsample**': 0.7 is the number of random rows you use and '**colsample_bytree**': 0.8 the same for colums to use. This add randomness to your evaluation. 0.9 is according to me most of the times too high.\n\nThere is no standard rule, it depends on your -***own***- file. Just play and try, and don't forget to keep track of your Local and LB scores.\n\nGood luck !", "votes": 6, "replies": [ { "id": 399336, "postDate": "2018-10-05T16:17:22.500Z", "content": "How did you choose those LGBM parameters? In particular, the very low learning rate is interesting to me. In combination with low early stopping, are you sure that allows the training to climb out of shallow local minima?
", "rawMarkdown": "How did you choose those LGBM parameters? In particular, the very low learning rate is interesting to me. In combination with low early stopping, are you sure that allows the training to climb out of shallow local minima?", "votes": 1 }, { "id": 399433, "postDate": "2018-10-05T19:16:40.747Z", "content": "@Misha Lisovyi
\n\nA very low learning rate (lr) is ok when you trust your model... BUT don't forget to lower your 'early_stopping' 20/50 is a maximum when you play with a low lr or you will easily overfit.
\n\nThe problem is that the model will train for a long time. The solution to reduce time for training/predicting is using GPU as a device in LGB params : 'device' : 'GPU'
\n\nHTH
", "rawMarkdown": "@Misha Lisovyi\n\nA very low learning rate (lr) is ok when you trust your model... **BUT** don't forget to lower your 'early_stopping' 20/50 is a maximum when you play with a low lr or you will easily overfit.\n\nThe problem is that the model will train for a long time. The solution to reduce time for training/predicting is using GPU as a device in LGB params : 'device' : 'GPU'\n\nHTH" } ] }, { "id": 396811, "postDate": "2018-10-01T12:12:57.353Z", "content": "I've focused on how to interact the category features. After my feature engineering, I concluded that high cardinality is key. In my case, Leaving high cardinality is a good way to gain the good result. In other words, I think that the special peoples who yield revenue have the very specific categories.
", "rawMarkdown": "I've focused on how to interact the category features. After my feature engineering, I concluded that high cardinality is key. In my case, Leaving high cardinality is a good way to gain the good result. In other words, I think that the special peoples who yield revenue have the very specific categories. ", "votes": 3 }, { "id": 396249, "postDate": "2018-09-30T10:33:45.783Z", "content": "Hi. I'm using simple Kfold validation and the improvement of local cv score can always translate to LB. My expericence is that mean encoding can somehow casue overfitting in you local train data, and I suggest you should use local cv and LB together as your feature selection metric from the very beginning. Or there is just something wrong in your post processing when summing up predictions :)
", "rawMarkdown": "Hi. I'm using simple Kfold validation and the improvement of local cv score can always translate to LB. My expericence is that mean encoding can somehow casue overfitting in you local train data, and I suggest you should use local cv and LB together as your feature selection metric from the very beginning. Or there is just something wrong in your post processing when summing up predictions :)", "votes": 3 }, { "id": 397361, "postDate": "2018-10-02T11:51:31.583Z", "content": "@Youhan @dylan_chen @mezoganet @Thomas Nelson @Corey Levinson
\n\nThanks a lot everybody for the emotional and technical support!!\nI managed to pass the hurdle,\nI took a blank notebook and started from scratch and from simple, checking with submissions that nothing is suspicious.\nI managed to implement some of my ideas and submit a single LGBM session based model with score of 1.4285.
\n\nBack to work,\nyou guys are awesome!
", "rawMarkdown": "@Youhan @dylan_chen @mezoganet @Thomas Nelson @Corey Levinson\n\nThanks a lot everybody for the emotional and technical support!!\nI managed to pass the hurdle,\nI took a blank notebook and started from scratch and from simple, checking with submissions that nothing is suspicious.\nI managed to implement some of my ideas and submit a single LGBM session based model with score of 1.4285.\n\nBack to work,\nyou guys are awesome!", "votes": 1, "replies": [ { "id": 397529, "postDate": "2018-10-02T16:06:30.803Z", "content": "Hi, @AmirH..
\n\nLike you, I also use R and I had problems to get a decent submission. I started again yesterday and got 1.4331 using a session level model (Lightgbm). I didn't peform hyper parameter tuning, so I hope to increase a little bit.
", "rawMarkdown": "Hi, @AmirH..\n\nLike you, I also use R and I had problems to get a decent submission. I started again yesterday and got 1.4331 using a session level model (Lightgbm). I didn't peform hyper parameter tuning, so I hope to increase a little bit." }, { "id": 397533, "postDate": "2018-10-02T16:19:05.647Z", "content": "hi Amir , can i know what was the cv score for that 1.4285 submission ?
", "rawMarkdown": "hi Amir , can i know what was the cv score for that 1.4285 submission ?" }, { "id": 397535, "postDate": "2018-10-02T16:22:23.813Z", "content": "Hi Paulo,\nGood to hear you also got it figured out! \nLet's bomb the LB with R solutions!\nThere really isn't enough of them.
", "rawMarkdown": "Hi Paulo,\nGood to hear you also got it figured out! \nLet's bomb the LB with R solutions!\nThere really isn't enough of them. ", "votes": 1 }, { "id": 397536, "postDate": "2018-10-02T16:24:00.743Z", "content": "Sure @suchith.\nIt's 1.589 on the session level
", "rawMarkdown": "Sure @suchith.\nIt's 1.589 on the session level ", "votes": 2 }, { "id": 397538, "postDate": "2018-10-02T16:27:44.543Z", "content": "Thanks, amir, I don't know what I am doing wrong, my cv and LB are in different directions(cv error decreased and lb error increased)
", "rawMarkdown": "Thanks, amir, I don't know what I am doing wrong, my cv and LB are in different directions(cv error decreased and lb error increased)" }, { "id": 397543, "postDate": "2018-10-02T16:38:08.320Z", "content": "@suchith it happens to me also with specific features, especially some time based and the whole page view and hits ratio thing, i also use 5 group kfold by visitorid for cv
", "rawMarkdown": "@suchith it happens to me also with specific features, especially some time based and the whole page view and hits ratio thing, i also use 5 group kfold by visitorid for cv" } ] }, { "id": 396818, "postDate": "2018-10-01T12:34:28.910Z", "content": "@Youhan @dylan_chen @mezoganet\nThanks for your responses.
\n\nI still can't figure out what's wrong =(
\n\nI duplicated in R the same features as @Olivier, everything simple, only categorical and some time feature(day of week etc)\nTraining on session based model\nI tried my LGB and catboost on those and still get 1.459 on catboost, and 1.46 on LGBM,\nWhere @Olivier said he gets 1.44~
\n\nBefore moving forward with FE and a lot of ideas i have, i must clear this issue!!\nI have no idea what to do , i spent 10-15 submissions to try and find out why this happens, also updated LGBM to latest version to make sure it's not anything there.
\n\nThere's no R kernel that reaches 1.44 currently so i got no place to get inspiration from.
\n\nI'm pretty sure it's something about the categories.
\n\nattached is my importance summary,\nand also Olivier's importance summary from this kernel:
\n\nhttps://www.kaggle.com/ogrellier/teach-lightgbm-to-sum-predictions/notebook
\n\nsomething is completely off there =/\nany ideas?
", "rawMarkdown": "@Youhan @dylan_chen @mezoganet\nThanks for your responses.\n\nI still can't figure out what's wrong =(\n\nI duplicated in R the same features as @Olivier, everything simple, only categorical and some time feature(day of week etc)\nTraining on session based model\nI tried my LGB and catboost on those and still get 1.459 on catboost, and 1.46 on LGBM,\nWhere @Olivier said he gets 1.44~\n\nBefore moving forward with FE and a lot of ideas i have, i must clear this issue!!\nI have no idea what to do , i spent 10-15 submissions to try and find out why this happens, also updated LGBM to latest version to make sure it's not anything there.\n\nThere's no R kernel that reaches 1.44 currently so i got no place to get inspiration from.\n\nI'm pretty sure it's something about the categories.\n\nattached is my importance summary,\nand also Olivier's importance summary from this kernel:\n\nhttps://www.kaggle.com/ogrellier/teach-lightgbm-to-sum-predictions/notebook\n\n\nsomething is completely off there =/\nany ideas? \n", "votes": -1, "replies": [ { "id": 396898, "postDate": "2018-10-01T15:05:48.860Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 396915, "postDate": "2018-10-01T15:35:33.560Z", "content": "Fully agree.
\n\nThis data is so weird that nobody has really found the ´Key’, and we are around 2000 people working hard.
\n\n@AmirH
\n\nBe confident, we have 2 months to go ;-)
", "rawMarkdown": "Fully agree.\n\nThis data is so weird that nobody has really found the ´Key’, and we are around 2000 people working hard.\n\n@AmirH\n\nBe confident, we have 2 months to go ;-)", "votes": 2 }, { "id": 396935, "postDate": "2018-10-01T16:35:04.723Z", "content": "@thomasnelson are you modeling nonzero user as a Poisson, and using probability of nonzero as just an extra feature to the regression?
", "rawMarkdown": "@thomasnelson are you modeling nonzero user as a Poisson, and using probability of nonzero as just an extra feature to the regression?" }, { "id": 396939, "postDate": "2018-10-01T16:45:06.023Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 396940, "postDate": "2018-10-01T16:48:38.257Z", "content": "@thomasnelson Here are some thoughts...maybe try number of nonzero transactions as Poisson, so then you can get it discrete?
\n\nAlso, to address the severe unbalance problem, I highly recommend reading this writeup (https://jeddy92.github.io/JEddy92.github.io/seguro/) by Joe Eddy...he uses a sandwiching technique to downsample zero-transactions. I plan to use this for my submission...
", "rawMarkdown": "@thomasnelson Here are some thoughts...maybe try number of nonzero transactions as Poisson, so then you can get it discrete?\n\nAlso, to address the severe unbalance problem, I *highly* recommend reading this writeup (https://jeddy92.github.io/JEddy92.github.io/seguro/) by Joe Eddy...he uses a sandwiching technique to downsample zero-transactions. I plan to use this for my submission...", "votes": 1 }, { "id": 400566, "postDate": "2018-10-08T15:05:30.337Z", "content": "@thomasnelson have you checked tweedie regression? I am not that great in statistics so I just checked this one out and maybe this will generalize better than the others?
", "rawMarkdown": "@thomasnelson have you checked tweedie regression? I am not that great in statistics so I just checked this one out and maybe this will generalize better than the others?" } ] }, { "id": 399437, "postDate": "2018-10-05T19:28:04.513Z", "content": "@AmirH
\n\nBeing '43rd in this Competition' at the time of writing this comment seems to be more than correct ;-)
\n\nNice job, keep on progressing !
", "rawMarkdown": "@AmirH\n\nBeing '43rd in this Competition' at the time of writing this comment seems to be more than correct ;-)\n\nNice job, keep on progressing !\n" }, { "id": 399259, "postDate": "2018-10-05T14:06:07.797Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 396322, "author_name": "mezoganet", "author_url": "", "post_date": "2018-09-30T13:13:43.360000", "content": "@AmirH
\n\nIt is true that this competition is a little bit strange !
\n\nHere are 'my' best params I normally use on this file :
\n\nparams = {'objective': 'reg:linear',\n 'eval_metric': 'rmse',\n 'eta': 0.001,\n 'max_depth': 8,\n 'subsample': 0.7,\n 'colsample_bytree': 0.8,\n 'alpha':0.001,\n 'num_leaves' : 63,\n 'random_state': 42,\n 'silent': True,\n }
\n\nDon't forget early_stopping : I usually use a low 20 rounds compared to a lot of kernels you can find. The higher the risk of overfitting is possible
\n\n'subsample': 0.7 is the number of random rows you use and 'colsample_bytree': 0.8 the same for colums to use. This add randomness to your evaluation. 0.9 is according to me most of the times too high.
\n\nThere is no standard rule, it depends on your -own- file. Just play and try, and don't forget to keep track of your Local and LB scores.
\n\nGood luck !
", "votes": 6, "replies": [ { "id": 399336, "author_name": "Misha Lisovyi", "author_url": "", "post_date": "2018-10-05T16:17:22.500000", "content": "How did you choose those LGBM parameters? In particular, the very low learning rate is interesting to me. In combination with low early stopping, are you sure that allows the training to climb out of shallow local minima?
", "votes": 1, "replies": [] }, { "id": 399433, "author_name": "mezoganet", "author_url": "", "post_date": "2018-10-05T19:16:40.747000", "content": "@Misha Lisovyi
\n\nA very low learning rate (lr) is ok when you trust your model... BUT don't forget to lower your 'early_stopping' 20/50 is a maximum when you play with a low lr or you will easily overfit.
\n\nThe problem is that the model will train for a long time. The solution to reduce time for training/predicting is using GPU as a device in LGB params : 'device' : 'GPU'
\n\nHTH
", "votes": 0, "replies": [] } ] }, { "id": 396811, "author_name": "Youhan Lee", "author_url": "", "post_date": "2018-10-01T12:12:57.353000", "content": "I've focused on how to interact the category features. After my feature engineering, I concluded that high cardinality is key. In my case, Leaving high cardinality is a good way to gain the good result. In other words, I think that the special peoples who yield revenue have the very specific categories.
", "votes": 3, "replies": [] }, { "id": 396249, "author_name": "fortune cookie", "author_url": "", "post_date": "2018-09-30T10:33:45.783000", "content": "Hi. I'm using simple Kfold validation and the improvement of local cv score can always translate to LB. My expericence is that mean encoding can somehow casue overfitting in you local train data, and I suggest you should use local cv and LB together as your feature selection metric from the very beginning. Or there is just something wrong in your post processing when summing up predictions :)
", "votes": 3, "replies": [] }, { "id": 397361, "author_name": "AmirH", "author_url": "", "post_date": "2018-10-02T11:51:31.583000", "content": "@Youhan @dylan_chen @mezoganet @Thomas Nelson @Corey Levinson
\n\nThanks a lot everybody for the emotional and technical support!!\nI managed to pass the hurdle,\nI took a blank notebook and started from scratch and from simple, checking with submissions that nothing is suspicious.\nI managed to implement some of my ideas and submit a single LGBM session based model with score of 1.4285.
\n\nBack to work,\nyou guys are awesome!
", "votes": 1, "replies": [ { "id": 397529, "author_name": "Paulo Felipe", "author_url": "", "post_date": "2018-10-02T16:06:30.803000", "content": "Hi, @AmirH..
\n\nLike you, I also use R and I had problems to get a decent submission. I started again yesterday and got 1.4331 using a session level model (Lightgbm). I didn't peform hyper parameter tuning, so I hope to increase a little bit.
", "votes": 0, "replies": [] }, { "id": 397533, "author_name": "redshellspy", "author_url": "", "post_date": "2018-10-02T16:19:05.647000", "content": "hi Amir , can i know what was the cv score for that 1.4285 submission ?
", "votes": 0, "replies": [] }, { "id": 397535, "author_name": "AmirH", "author_url": "", "post_date": "2018-10-02T16:22:23.813000", "content": "Hi Paulo,\nGood to hear you also got it figured out! \nLet's bomb the LB with R solutions!\nThere really isn't enough of them.
", "votes": 1, "replies": [] }, { "id": 397536, "author_name": "AmirH", "author_url": "", "post_date": "2018-10-02T16:24:00.743000", "content": "Sure @suchith.\nIt's 1.589 on the session level
", "votes": 2, "replies": [] }, { "id": 397538, "author_name": "redshellspy", "author_url": "", "post_date": "2018-10-02T16:27:44.543000", "content": "Thanks, amir, I don't know what I am doing wrong, my cv and LB are in different directions(cv error decreased and lb error increased)
", "votes": 0, "replies": [] }, { "id": 397543, "author_name": "AmirH", "author_url": "", "post_date": "2018-10-02T16:38:08.320000", "content": "@suchith it happens to me also with specific features, especially some time based and the whole page view and hits ratio thing, i also use 5 group kfold by visitorid for cv
", "votes": 0, "replies": [] } ] }, { "id": 396818, "author_name": "AmirH", "author_url": "", "post_date": "2018-10-01T12:34:28.910000", "content": "@Youhan @dylan_chen @mezoganet\nThanks for your responses.
\n\nI still can't figure out what's wrong =(
\n\nI duplicated in R the same features as @Olivier, everything simple, only categorical and some time feature(day of week etc)\nTraining on session based model\nI tried my LGB and catboost on those and still get 1.459 on catboost, and 1.46 on LGBM,\nWhere @Olivier said he gets 1.44~
\n\nBefore moving forward with FE and a lot of ideas i have, i must clear this issue!!\nI have no idea what to do , i spent 10-15 submissions to try and find out why this happens, also updated LGBM to latest version to make sure it's not anything there.
\n\nThere's no R kernel that reaches 1.44 currently so i got no place to get inspiration from.
\n\nI'm pretty sure it's something about the categories.
\n\nattached is my importance summary,\nand also Olivier's importance summary from this kernel:
\n\nhttps://www.kaggle.com/ogrellier/teach-lightgbm-to-sum-predictions/notebook
\n\nsomething is completely off there =/\nany ideas?
", "votes": -1, "replies": [ { "id": 396898, "author_name": "", "author_url": "", "post_date": "2018-10-01T15:05:48.860000", "content": "", "votes": 3, "replies": [] }, { "id": 396915, "author_name": "mezoganet", "author_url": "", "post_date": "2018-10-01T15:35:33.560000", "content": "Fully agree.
\n\nThis data is so weird that nobody has really found the ´Key’, and we are around 2000 people working hard.
\n\n@AmirH
\n\nBe confident, we have 2 months to go ;-)
", "votes": 2, "replies": [] }, { "id": 396935, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-10-01T16:35:04.723000", "content": "@thomasnelson are you modeling nonzero user as a Poisson, and using probability of nonzero as just an extra feature to the regression?
", "votes": 0, "replies": [] }, { "id": 396939, "author_name": "", "author_url": "", "post_date": "2018-10-01T16:45:06.023000", "content": "", "votes": 3, "replies": [] }, { "id": 396940, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-10-01T16:48:38.257000", "content": "@thomasnelson Here are some thoughts...maybe try number of nonzero transactions as Poisson, so then you can get it discrete?
\n\nAlso, to address the severe unbalance problem, I highly recommend reading this writeup (https://jeddy92.github.io/JEddy92.github.io/seguro/) by Joe Eddy...he uses a sandwiching technique to downsample zero-transactions. I plan to use this for my submission...
", "votes": 1, "replies": [] }, { "id": 400566, "author_name": "rafamacalaba", "author_url": "", "post_date": "2018-10-08T15:05:30.337000", "content": "@thomasnelson have you checked tweedie regression? I am not that great in statistics so I just checked this one out and maybe this will generalize better than the others?
", "votes": 0, "replies": [] } ] }, { "id": 399437, "author_name": "mezoganet", "author_url": "", "post_date": "2018-10-05T19:28:04.513000", "content": "@AmirH
\n\nBeing '43rd in this Competition' at the time of writing this comment seems to be more than correct ;-)
\n\nNice job, keep on progressing !
", "votes": 0, "replies": [] }, { "id": 399259, "author_name": "", "author_url": "", "post_date": "2018-10-05T14:06:07.797000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "396235": "Hi Kagglers,\n\nI'm getting really frustrated with this competition.\nSeems like anything I do that generates better CV, gets worse LB.\nMy position is pretty bad.\n\nI started out using time based validation with 3 last months as validation set.\nI had some 1.44~ submissions using this validation.\n\nIt's easier for me to use 5 fold validation as i can also use these folds for mean encodings, OOF stacking, averaged CV score , implementing oof binary classification to use in model etc.\n\nAs I don't see any reason not to use 5 fold CV here, i generated 5 Folds by group of fullVisitorId. (all rows of a specific fullvisitorId are always in the same fold).\nSince doing that, anything i seem to be doing , without any high risk things such as mean encodings etc. I get scores of 1.45-1.47\neven using only simple features such as:\n\n\n\n\"channelGrouping\", \"visitNumber\", \"browser\", \"operatingSystem\", \"isMobile\", \"deviceCategory\", \"continent\", \"subContinent\", \"country\", \"region\", \"metro\", \"city\", \"networkDomain\", \"hits\", \"pageviews\", \"newVisits\", \"bounces\", \"source\", \"medium\", \"keyword\", \"isTrueDirect\", \"referralPath\", \"VisitMonth\", \"VisitDay\", \"VisitWeekDay\", \"VisitHour\", \"CountVisitorVisits\", \"Location\", \"IsMulti\", \"Maxhits\", \"Meanhits\", \"Medianhits\", \"Maxpageviews\", \"Meanpageviews\", \"Medianpageviews\", \"DaysActive\" \n\n\n (categories are label encoded)\nwith LGBM with parameters:\n\nlist(objective = \"regression\",\n metric = \"RMSE\",\n learning_rate= 0.03,\n num_leaves= 40, #16\n #max_depth= 12, #3\n #max_bin= 255,\n subsample= 0.9,\n subsample_freq= 1,\n colsample_bytree=0.9)\n\n\n\nI use lgbm's lgb.cv with my 5folds and then retrain on entire data with best.iteration.\n \nAny suggestions?\nI'm really frustrated.\n\nI don't like forking and using kernels much, I also use R\nI saw @Olivier using 5 fold validation with very simple model and getting better scores.\nI tried session level, user level.. I just don't understand!!\nDoes anyone else experience the same?\n\n\n- What CV method are you using?\n\n- How does one do mean encoding \\ OOF predictions when using 1 set of train and 1 set of validation, without having folds?\n\n\nThanks!\nAmir.\n\n", "396322": "@AmirH\n\nIt is true that this competition is a little bit strange ! \n\nHere are 'my' best params I normally use on this file : \n\nparams = {'objective': 'reg:linear',\n 'eval_metric': 'rmse',\n 'eta': 0.001,\n 'max_depth': 8,\n 'subsample': 0.7,\n 'colsample_bytree': 0.8,\n 'alpha':0.001,\n 'num_leaves' : 63,\n 'random_state': 42,\n 'silent': True,\n }\n\nDon't forget early_stopping : I usually use a low 20 rounds compared to a lot of kernels you can find. The higher the risk of overfitting is possible\n\n'**subsample**': 0.7 is the number of random rows you use and '**colsample_bytree**': 0.8 the same for colums to use. This add randomness to your evaluation. 0.9 is according to me most of the times too high.\n\nThere is no standard rule, it depends on your -***own***- file. Just play and try, and don't forget to keep track of your Local and LB scores.\n\nGood luck !", "396811": "I've focused on how to interact the category features. After my feature engineering, I concluded that high cardinality is key. In my case, Leaving high cardinality is a good way to gain the good result. In other words, I think that the special peoples who yield revenue have the very specific categories. ", "396249": "Hi. I'm using simple Kfold validation and the improvement of local cv score can always translate to LB. My expericence is that mean encoding can somehow casue overfitting in you local train data, and I suggest you should use local cv and LB together as your feature selection metric from the very beginning. Or there is just something wrong in your post processing when summing up predictions :)", "397361": "@Youhan @dylan_chen @mezoganet @Thomas Nelson @Corey Levinson\n\nThanks a lot everybody for the emotional and technical support!!\nI managed to pass the hurdle,\nI took a blank notebook and started from scratch and from simple, checking with submissions that nothing is suspicious.\nI managed to implement some of my ideas and submit a single LGBM session based model with score of 1.4285.\n\nBack to work,\nyou guys are awesome!", "396818": "@Youhan @dylan_chen @mezoganet\nThanks for your responses.\n\nI still can't figure out what's wrong =(\n\nI duplicated in R the same features as @Olivier, everything simple, only categorical and some time feature(day of week etc)\nTraining on session based model\nI tried my LGB and catboost on those and still get 1.459 on catboost, and 1.46 on LGBM,\nWhere @Olivier said he gets 1.44~\n\nBefore moving forward with FE and a lot of ideas i have, i must clear this issue!!\nI have no idea what to do , i spent 10-15 submissions to try and find out why this happens, also updated LGBM to latest version to make sure it's not anything there.\n\nThere's no R kernel that reaches 1.44 currently so i got no place to get inspiration from.\n\nI'm pretty sure it's something about the categories.\n\nattached is my importance summary,\nand also Olivier's importance summary from this kernel:\n\nhttps://www.kaggle.com/ogrellier/teach-lightgbm-to-sum-predictions/notebook\n\n\nsomething is completely off there =/\nany ideas? \n", "399437": "@AmirH\n\nBeing '43rd in this Competition' at the time of writing this comment seems to be more than correct ;-)\n\nNice job, keep on progressing !\n", "399259": "" } }