Hello all, I hope everyone is enjoying a very nice period toward the end of the year, and for me, it is definitely great fun to have some dedicated time to immerse myself in a kaggle competition without thinking about other daily stuff.
\n\nIt is mid-way through this nice competition(or maybe not so nice depends on how your look at it). I would like to share some of my experience so far. Some are successful, some are not, but they are all good experience.
\n\nso here we go:\n- Feature engineering: my personal feeling is that regular feature engineering (typical groupby-agg, counting, categorical encoding) would bring LB score down (i.e. RMSE) to 3.680, but no more. I personally starting to find it difficult to improve beyond the 3.690 mark. Beyond this line, I started to look for other ways that would give me features. Some of these have already been mentioned in the forum, such as count vector, FM/FFM, NN, and out of fold model.
\n\nFeature selection: I have found this dataset to be quite sensitive to 1) selection of features, and 2) order of features in the dataset - especially if you are using algorithms that choose sub-set of features (i.e. GBM). By switching columns order, I have had scores with 0.005 difference on the LB (obviously with very similar CV). Usually I don't do too much feature selection in a kaggle competition, but now I found this (often overlooked) technique to be useful BOTH in improving CV/LB score, and also helps to reduce computation time. I strongly encourage everyone to have a look at the post from @peterhurford titled \"Less is More\". Everything I used so far w.r.t. feature selection comes from that post.
Get your basics right: Each and every kaggle competition shall be treated as a work/study project in which you should setup a proper workflow to structure your work. and I can not emphasise more about this - every model you chunk out, MAKE SURE IT IS REPRODUCIBLE - ok some algorithms/tool are non-deterministic - but then again you should make sure you can reproduce your model with similar baseline performance. Personally, for each model I create, I use a good-old spreadsheet to capture the following: algorithm used, parameters, features used, seed, cross-validation approach(kfold, stratified kfold, etc), CV score, LB score(if submitted), score at each fold, std of CV score. everyone's approach differs, but still I encourage everyone to write these down - in chinese we have the saying \"a good memory is not as good as a used pen\" :)
Think \"out-of-box\": the elephant(s) in the room is that there are 2000+ outliers which contribute to most of the RMSE, and this requires us to take time to think how to address this. there is quite a lot of discussion on this, and the nice kernel about post-processing from @waitingli, and also the discussion thread from @lblhandsome. I don't have much to add to this, but what I would say is make sure you spend time away from your computer(s). Take a walk, spend time with your family/friends, and eat/sleep properly. For me, the several ideas that really work came when I was not facing the screen busy coding.
Manage your code: always keep your code in proper modules or classes, so that they can be reused for your future projects and competitions. A lot of code I am using to push myself up the LB has from my own little repo - so I have probably saved hundreds of hours by reusing my previous stuff. In a sense, you need to find a way to team up with your previous selves :)
What I have not yet managed to figured out is that magical something both @senkin13 and @lblhandsome have found. So actually everything I have done so far have been fully shared by many others in many previous competitions - I can confirm that without anything \"magic\" or \"leaky\" you could get up to at least 3.67x in public LB - well, private LB is another story, and probably worthy of another post to discuss how much shake-up there will be - insert :trollface:
\n\nAnyway, I hope everyone enjoys this competition, and have a great time!
", "messages": [ { "id": 446239, "postDate": "2018-12-27T18:51:20.540Z", "content": "Hello all, I hope everyone is enjoying a very nice period toward the end of the year, and for me, it is definitely great fun to have some dedicated time to immerse myself in a kaggle competition without thinking about other daily stuff.
\n\nIt is mid-way through this nice competition(or maybe not so nice depends on how your look at it). I would like to share some of my experience so far. Some are successful, some are not, but they are all good experience.
\n\nso here we go:\n- Feature engineering: my personal feeling is that regular feature engineering (typical groupby-agg, counting, categorical encoding) would bring LB score down (i.e. RMSE) to 3.680, but no more. I personally starting to find it difficult to improve beyond the 3.690 mark. Beyond this line, I started to look for other ways that would give me features. Some of these have already been mentioned in the forum, such as count vector, FM/FFM, NN, and out of fold model.
\n\nFeature selection: I have found this dataset to be quite sensitive to 1) selection of features, and 2) order of features in the dataset - especially if you are using algorithms that choose sub-set of features (i.e. GBM). By switching columns order, I have had scores with 0.005 difference on the LB (obviously with very similar CV). Usually I don't do too much feature selection in a kaggle competition, but now I found this (often overlooked) technique to be useful BOTH in improving CV/LB score, and also helps to reduce computation time. I strongly encourage everyone to have a look at the post from @peterhurford titled \"Less is More\". Everything I used so far w.r.t. feature selection comes from that post.
Get your basics right: Each and every kaggle competition shall be treated as a work/study project in which you should setup a proper workflow to structure your work. and I can not emphasise more about this - every model you chunk out, MAKE SURE IT IS REPRODUCIBLE - ok some algorithms/tool are non-deterministic - but then again you should make sure you can reproduce your model with similar baseline performance. Personally, for each model I create, I use a good-old spreadsheet to capture the following: algorithm used, parameters, features used, seed, cross-validation approach(kfold, stratified kfold, etc), CV score, LB score(if submitted), score at each fold, std of CV score. everyone's approach differs, but still I encourage everyone to write these down - in chinese we have the saying \"a good memory is not as good as a used pen\" :)
Think \"out-of-box\": the elephant(s) in the room is that there are 2000+ outliers which contribute to most of the RMSE, and this requires us to take time to think how to address this. there is quite a lot of discussion on this, and the nice kernel about post-processing from @waitingli, and also the discussion thread from @lblhandsome. I don't have much to add to this, but what I would say is make sure you spend time away from your computer(s). Take a walk, spend time with your family/friends, and eat/sleep properly. For me, the several ideas that really work came when I was not facing the screen busy coding.
Manage your code: always keep your code in proper modules or classes, so that they can be reused for your future projects and competitions. A lot of code I am using to push myself up the LB has from my own little repo - so I have probably saved hundreds of hours by reusing my previous stuff. In a sense, you need to find a way to team up with your previous selves :)
What I have not yet managed to figured out is that magical something both @senkin13 and @lblhandsome have found. So actually everything I have done so far have been fully shared by many others in many previous competitions - I can confirm that without anything \"magic\" or \"leaky\" you could get up to at least 3.67x in public LB - well, private LB is another story, and probably worthy of another post to discuss how much shake-up there will be - insert :trollface:
\n\nAnyway, I hope everyone enjoys this competition, and have a great time!
", "rawMarkdown": "Hello all, I hope everyone is enjoying a very nice period toward the end of the year, and for me, it is definitely great fun to have some dedicated time to immerse myself in a kaggle competition without thinking about other daily stuff.\n\nIt is mid-way through this nice competition(or maybe not so nice depends on how your look at it). I would like to share some of my experience so far. Some are successful, some are not, but they are all good experience. \n\nso here we go:\n- Feature engineering: my personal feeling is that regular feature engineering (typical groupby-agg, counting, categorical encoding) would bring LB score down (i.e. RMSE) to 3.680, but no more. I personally starting to find it difficult to improve beyond the 3.690 mark. Beyond this line, I started to look for other ways that would give me features. Some of these have already been mentioned in the forum, such as count vector, FM/FFM, NN, and out of fold model.\n\n- Feature selection: I have found this dataset to be quite sensitive to 1) selection of features, and 2) order of features in the dataset - especially if you are using algorithms that choose sub-set of features (i.e. GBM). By switching columns order, I have had scores with 0.005 difference on the LB (obviously with very similar CV). Usually I don't do too much feature selection in a kaggle competition, but now I found this (often overlooked) technique to be useful BOTH in improving CV/LB score, and also helps to reduce computation time. I strongly encourage everyone to have a look at the post from @peterhurford titled \"[Less is More][1]\". Everything I used so far w.r.t. feature selection comes from that post.\n\n- Get your basics right: Each and every kaggle competition shall be treated as a work/study project in which you should setup a proper workflow to structure your work. and I can not emphasise more about this - every model you chunk out, MAKE SURE IT IS REPRODUCIBLE - ok some algorithms/tool are non-deterministic - but then again you should make sure you can reproduce your model with similar baseline performance. Personally, for each model I create, I use a good-old spreadsheet to capture the following: algorithm used, parameters, features used, seed, cross-validation approach(kfold, stratified kfold, etc), CV score, LB score(if submitted), score at each fold, std of CV score. everyone's approach differs, but still I encourage everyone to write these down - in chinese we have the saying \"a good memory is not as good as a used pen\" :)\n\n- Think \"out-of-box\": the elephant(s) in the room is that there are 2000+ outliers which contribute to most of the RMSE, and this requires us to take time to think how to address this. there is quite a lot of discussion on this, and the [nice kernel about post-processing][2] from @waitingli, and also the [discussion thread][3] from @lblhandsome. I don't have much to add to this, but what I would say is make sure you spend time away from your computer(s). Take a walk, spend time with your family/friends, and eat/sleep properly. For me, the several ideas that really work came when I was not facing the screen busy coding. \n\n- Manage your code: always keep your code in proper modules or classes, so that they can be reused for your future projects and competitions. A lot of code I am using to push myself up the LB has from my own little repo - so I have probably saved hundreds of hours by reusing my previous stuff. In a sense, you need to find a way to team up with your previous selves :)\n\nWhat I have not yet managed to figured out is that magical something both @senkin13 and @lblhandsome have found. So actually everything I have done so far have been fully shared by many others in many previous competitions - I can confirm that without anything \"magic\" or \"leaky\" you could get up to at least 3.67x in public LB - well, private LB is another story, and probably worthy of another post to discuss how much shake-up there will be - insert :trollface: \n\nAnyway, I hope everyone enjoys this competition, and have a great time! \n \n\n\n [1]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73937\n [2]: https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier\n [3]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75034#440855", "votes": 206 }, { "id": 475697, "postDate": "2019-02-21T04:43:43.797Z", "content": "@Yifan Xie Thanks for the guide. About Making sure our code is reproducible. Can you share excel sheets related any of past competitions for reference. Also do you follow any style guide for python.
", "rawMarkdown": "@Yifan Xie Thanks for the guide. About Making sure our code is reproducible. Can you share excel sheets related any of past competitions for reference. Also do you follow any style guide for python.", "votes": 3, "replies": [ { "id": 475910, "postDate": "2019-02-21T11:07:48.807Z", "content": "The ultimate Python style guide: https://www.python.org/dev/peps/pep-0008/
", "rawMarkdown": "The ultimate Python style guide: https://www.python.org/dev/peps/pep-0008/", "votes": 6 }, { "id": 476146, "postDate": "2019-02-21T16:53:28.963Z", "content": "👆
", "rawMarkdown": "👆 ", "votes": 2 } ] }, { "id": 457132, "postDate": "2019-01-17T01:50:56.517Z", "content": "Hi Yifan!\ncan i know the meaning of Features Selection for u ? It improve the gap between cv & lb but can't work for me, so I wanna know the meaning for u\nThanks
", "rawMarkdown": "Hi Yifan!\ncan i know the meaning of Features Selection for u ? It improve the gap between cv & lb but can't work for me, so I wanna know the meaning for u\nThanks", "votes": 1, "replies": [ { "id": 457485, "postDate": "2019-01-17T14:38:39.363Z", "content": "interesting question, I mainly do \"feature selection\" to identify subsets of all available features which provide better model performance - in the context of a competition like this, better model performance for me is 1) ideally better CV & LB, 2) at least better CV. In real world scenario, I would consider other criterias such as modelling time, and consistency between folds - you could very well make an argument these should also be included here.
\n\nHope this helps?
", "rawMarkdown": "interesting question, I mainly do \"feature selection\" to identify subsets of all available features which provide better model performance - in the context of a competition like this, better model performance for me is 1) ideally better CV & LB, 2) at least better CV. In real world scenario, I would consider other criterias such as modelling time, and consistency between folds - you could very well make an argument these should also be included here. \n\nHope this helps?", "votes": 1 }, { "id": 457718, "postDate": "2019-01-18T01:21:37.250Z", "content": "Yup! I agree with all your points, the other is prevent the bad-influence of high-dimensions
", "rawMarkdown": "Yup! I agree with all your points, the other is prevent the bad-influence of high-dimensions" } ] }, { "id": 451952, "postDate": "2019-01-08T00:20:06.340Z", "content": "A used pen is better indeed. Thank you for the post. It is very helpful
", "rawMarkdown": "A used pen is better indeed. Thank you for the post. It is very helpful ", "votes": 1 }, { "id": 447752, "postDate": "2018-12-30T13:26:37.940Z", "content": "thanks so much.you said we should takes notes about our own code,it is a very very good tips for me. I often takes time to reproduce last code.And can't remember where i have changed.I will take notes from now
", "rawMarkdown": "thanks so much.you said we should takes notes about our own code,it is a very very good tips for me. I often takes time to reproduce last code.And can't remember where i have changed.I will take notes from now", "votes": 1 }, { "id": 447020, "postDate": "2018-12-29T02:16:19.633Z", "content": "Thank you for sharing your experience! Good food for thought and investigation...
", "rawMarkdown": "Thank you for sharing your experience! Good food for thought and investigation...", "votes": 1 }, { "id": 446608, "postDate": "2018-12-28T11:22:42.773Z", "content": "Nice thoughts!
", "rawMarkdown": "Nice thoughts!", "votes": 1 }, { "id": 446512, "postDate": "2018-12-28T08:07:51.703Z", "content": "Thank you, I have a question: In the best data version that transformed, are you using info from merchants table?
", "rawMarkdown": "Thank you, I have a question: In the best data version that transformed, are you using info from merchants table?", "votes": 1, "replies": [ { "id": 446552, "postDate": "2018-12-28T09:20:13.323Z", "content": "I haven't used features from the merchant table so far, to put it in another way, I have made some features from it, but they always got filtered out by my feature selection mechanism
", "rawMarkdown": "I haven't used features from the merchant table so far, to put it in another way, I have made some features from it, but they always got filtered out by my feature selection mechanism", "votes": 4 } ] }, { "id": 446403, "postDate": "2018-12-28T02:31:23.337Z", "content": "Thanks for your sharing, which has helped & inspired me a lot.
", "rawMarkdown": "Thanks for your sharing, which has helped & inspired me a lot.", "votes": 1 }, { "id": 446293, "postDate": "2018-12-27T20:12:47.520Z", "content": "Thanks for sharing, excellent advice, it will help me to improve my score and workflow.
", "rawMarkdown": "Thanks for sharing, excellent advice, it will help me to improve my score and workflow.", "votes": 1 }, { "id": 446242, "postDate": "2018-12-27T18:55:38.337Z", "content": "Thanks for sharing! What's your CV score, if you don't mind me asking? I'm curious what it takes to get to 3rd place, when the CV-LB diff is so large and variable.
", "rawMarkdown": "Thanks for sharing! What's your CV score, if you don't mind me asking? I'm curious what it takes to get to 3rd place, when the CV-LB diff is so large and variable.", "votes": 1, "replies": [ { "id": 446247, "postDate": "2018-12-27T18:58:40.337Z", "content": "I am on about 3.620 - I am living with the daily fear that I might have overfitted this badly! ¯_(ツ) _/¯
", "rawMarkdown": "I am on about 3.620 - I am living with the daily fear that I might have overfitted this badly! ¯\\_(ツ) _/¯", "votes": 7 }, { "id": 446254, "postDate": "2018-12-27T19:03:04.340Z", "content": "Wow, that's pretty nice. Thanks!
\n\nWhat about single model, no post-processing CV? I'm curious how much is possible from raw feature engineering + feature selection and how much I need to go \"out of the box\".
", "rawMarkdown": "Wow, that's pretty nice. Thanks!\n\nWhat about single model, no post-processing CV? I'm curious how much is possible from raw feature engineering + feature selection and how much I need to go \"out of the box\".", "votes": 1 }, { "id": 446258, "postDate": "2018-12-27T19:08:01.300Z", "content": "for me, with standard supervised-learning - our everyday x_train, y_train, and x_test - I think I am on about 3.640 CV
", "rawMarkdown": "for me, with standard supervised-learning - our everyday x_train, y_train, and x_test - I think I am on about 3.640 CV ", "votes": 5 }, { "id": 446261, "postDate": "2018-12-27T19:10:38.207Z", "content": "Sweet, thanks. That's helpful for my own calibration. :)
", "rawMarkdown": "Sweet, thanks. That's helpful for my own calibration. :)", "votes": 1 }, { "id": 447747, "postDate": "2018-12-30T13:16:59.263Z", "content": "Hi,thank you for the discuss. you only use agg features and get 3.640 CV or agg features and other features?thanks
", "rawMarkdown": "Hi,thank you for the discuss. you only use agg features and get 3.640 CV or agg features and other features?thanks" }, { "id": 447963, "postDate": "2018-12-30T23:28:14.503Z", "content": "that would be agg + other features - that 3.640CV score is just a rough estimation I made to answer Peter's question though. You can for sure improve on that, by some margin
", "rawMarkdown": "that would be agg + other features - that 3.640CV score is just a rough estimation I made to answer Peter's question though. You can for sure improve on that, by some margin " } ] }, { "id": 460799, "postDate": "2019-01-24T12:52:13.863Z", "content": "Friends have you faced any loading issue in historical_transactions.csv
", "rawMarkdown": "Friends have you faced any loading issue in historical_transactions.csv", "votes": 2, "replies": [ { "id": 461007, "postDate": "2019-01-25T01:21:36.640Z", "content": "what‘s ur problem?
", "rawMarkdown": "what‘s ur problem?" } ] }, { "id": 449433, "postDate": "2019-01-03T06:22:13.030Z", "content": "Thx for the guide. I've tried some fancy agg that I learned in LSST but seems only helps a little, currently at cv 3.6507 / lb 3.70. BTW, how many features do have in your best single model?
", "rawMarkdown": "Thx for the guide. I've tried some fancy agg that I learned in LSST but seems only helps a little, currently at cv 3.6507 / lb 3.70. BTW, how many features do have in your best single model? ", "votes": 2, "replies": [ { "id": 449440, "postDate": "2019-01-03T06:29:57.360Z", "content": "Let's define \"best\" first :) \nIn my case I am still in the vain attempt to produce model that improve both on CV and LB. And I have been pretty much stuck in the last few days.
\n\nCurrently my best single model has around 200 features
", "rawMarkdown": "Let's define \"best\" first :) \nIn my case I am still in the vain attempt to produce model that improve both on CV and LB. And I have been pretty much stuck in the last few days.\n \nCurrently my best single model has around 200 features", "votes": 3 }, { "id": 449540, "postDate": "2019-01-03T10:26:06.690Z", "content": "Thanks a lot! and the \"Best\" here I means best CV :))
\nFYI, I'm using over 1000 features with the model I mentioned above, I'll definitely try feature selection after I finished all the FE stuff.
Maybe you can make more model to improve your score now, I used 7 models that their corr-matrix is between 0.95-0.97, it gave me so much help. For example, you can try to the Graph-Model. Yep, try the everythin to vector
", "rawMarkdown": "Maybe you can make more model to improve your score now, I used 7 models that their corr-matrix is between 0.95-0.97, it gave me so much help. For example, you can try to the Graph-Model. Yep, try the everythin to vector", "votes": 3 }, { "id": 449613, "postDate": "2019-01-03T12:36:07.810Z", "content": "I was quite happy with my CV before I got to 3.670 - now CV and LB no longer agree with each other. \nI also observe that my feature selection strategy starting to falter at this level. It is starting to give me feature set that gives better CV score on higher learning rate (i.e. 0.07) than \"reasonable\" lower learning rate (i.e. 0.01) - this is very rarely the case in my previous experience.
\n\nI am now thinking about reducing my fold numbers to re-evaluate some of my key features, and potentially rebuild my whole model generation stack
", "rawMarkdown": "I was quite happy with my CV before I got to 3.670 - now CV and LB no longer agree with each other. \nI also observe that my feature selection strategy starting to falter at this level. It is starting to give me feature set that gives better CV score on higher learning rate (i.e. 0.07) than \"reasonable\" lower learning rate (i.e. 0.01) - this is very rarely the case in my previous experience. \n\nI am now thinking about reducing my fold numbers to re-evaluate some of my key features, and potentially rebuild my whole model generation stack \n\n\n" }, { "id": 450550, "postDate": "2019-01-05T08:19:54.727Z", "content": "Many things to do, hard way to go!
", "rawMarkdown": "Many things to do, hard way to go!" }, { "id": 468296, "postDate": "2019-02-08T16:35:36.207Z", "content": "Hey, Shuo-Jen Chang, you said \"fancy agg that I learned in LSST\", do you have a link? can you share it with me?
", "rawMarkdown": "Hey, Shuo-Jen Chang, you said \"fancy agg that I learned in LSST\", do you have a link? can you share it with me? ", "votes": 1 } ] }, { "id": 446388, "postDate": "2018-12-28T01:41:42.877Z", "content": "I have one question.How are you selecting the features order?\ndid u use ffm here?any sample ffm working would be nice...\nThank u for the post
", "rawMarkdown": "I have one question.How are you selecting the features order?\ndid u use ffm here?any sample ffm working would be nice...\nThank u for the post", "votes": 2, "replies": [ { "id": 446395, "postDate": "2018-12-28T01:51:21.970Z", "content": "I didn't \"select\" feature order, I group my features into different files and load them one by one while starting my model. It happened that I loaded two of them in different order, and I just observed the LB score difference.
\n\nI have not use FFM yet, I would like to read the paper first before trying this algorithm. I have some success with FM, and I suggest you to check out CPMP's implementation of LibFM in keras. He shared it in the Talking Data competition this year, search the forum. I found his implementation to be more flexible for my purpose, but you may find something else more suitable for your need.
", "rawMarkdown": "I didn't \"select\" feature order, I group my features into different files and load them one by one while starting my model. It happened that I loaded two of them in different order, and I just observed the LB score difference.\n\nI have not use FFM yet, I would like to read the paper first before trying this algorithm. I have some success with FM, and I suggest you to check out CPMP's implementation of LibFM in keras. He shared it in the Talking Data competition this year, search the forum. I found his implementation to be more flexible for my purpose, but you may find something else more suitable for your need. ", "votes": 9 }, { "id": 446733, "postDate": "2018-12-28T15:43:12.103Z", "content": "Thank you..trying to understand CPMP's implementation. Thank you.
", "rawMarkdown": "Thank you..trying to understand CPMP's implementation. Thank you.\n", "votes": 1 }, { "id": 446871, "postDate": "2018-12-28T20:08:12.363Z", "content": "@Yifan IMHO, it is by design to have different results while the same feature sets but their sorting order are different of GBMs. Saying data A and A' with feature columns shuffled. Column-wise sampled should have different results at very beginning and then leading to different tree growths. For example, given a random seed therefore the first round sampling to pick the first two columns out of three in A is ['a', 'b', 'c'] and A' is ['a', 'c', 'b']. As a result, A got ['a', 'b'] but A' got ['a', 'c']. Then model build on A and A' should go different directions, respectively.
\n\nFurthermore, to make it persistent, sort columns before model.fit might be a good practice.
", "rawMarkdown": "@Yifan IMHO, it is by design to have different results while the same feature sets but their sorting order are different of GBMs. Saying data A and A' with feature columns shuffled. Column-wise sampled should have different results at very beginning and then leading to different tree growths. For example, given a random seed therefore the first round sampling to pick the first two columns out of three in A is ['a', 'b', 'c'] and A' is ['a', 'c', 'b']. As a result, A got ['a', 'b'] but A' got ['a', 'c']. Then model build on A and A' should go different directions, respectively.\n\nFurthermore, to make it persistent, sort columns before model.fit might be a good practice. " }, { "id": 446875, "postDate": "2018-12-28T20:13:32.200Z", "content": "On the other hands, shuffle columns and rows might be a good way to moderately diversify models and predictions. B-)
", "rawMarkdown": "On the other hands, shuffle columns and rows might be a good way to moderately diversify models and predictions. B-)", "votes": 2 }, { "id": 446884, "postDate": "2018-12-28T20:34:47.723Z", "content": "@cttsai yes, GMB by design will sample columns - but this is not my point. My point was that this kind of random order had a significant impact on LB score, in my case up to 0.005 difference, and I also note that my local CV for the same model with different column orders remained consistant. I am suggesting that we shall be observant to this - how one decides to act on it is entirely each participant's choice.
\n\nI am simply saying this dataset might be sensitive to choice of features used in the modelling - and thatI it had been useful for me during this competition to invest time in feature selection. I hope this help to clarify my original comment.
", "rawMarkdown": "@cttsai yes, GMB by design will sample columns - but this is not my point. My point was that this kind of random order had a significant impact on LB score, in my case up to 0.005 difference, and I also note that my local CV for the same model with different column orders remained consistant. I am suggesting that we shall be observant to this - how one decides to act on it is entirely each participant's choice. \n\nI am simply saying this dataset might be sensitive to choice of features used in the modelling - and thatI it had been useful for me during this competition to invest time in feature selection. I hope this help to clarify my original comment. ", "votes": 1 }, { "id": 446903, "postDate": "2018-12-28T21:03:14.407Z", "content": "@Yifan, thanks for the clarification. It is interesting that models that have similar CV results but output huge LB shakeup, which their only difference are in column sorting order! May I ask what is the range of your column subsample rate? Is it as high as 0.8, 0.9 or much lower?
", "rawMarkdown": "@Yifan, thanks for the clarification. It is interesting that models that have similar CV results but output huge LB shakeup, which their only difference are in column sorting order! May I ask what is the range of your column subsample rate? Is it as high as 0.8, 0.9 or much lower?", "votes": 1 }, { "id": 446911, "postDate": "2018-12-28T21:18:34.527Z", "content": "I was on 0.7 by tree and 0.6 by level for LGB, I have since developed several sets of parameters for different variants of LGB/XGB, and again for different combination of random seed, and feature sets. \nmost of the col sampling I am using these days are between 0.4 to 0.6.
", "rawMarkdown": "I was on 0.7 by tree and 0.6 by level for LGB, I have since developed several sets of parameters for different variants of LGB/XGB, and again for different combination of random seed, and feature sets. \nmost of the col sampling I am using these days are between 0.4 to 0.6. \n", "votes": 4 }, { "id": 446942, "postDate": "2018-12-28T22:31:47.047Z", "content": "@Yifan, thanks for sharing!
", "rawMarkdown": "@Yifan, thanks for sharing!" }, { "id": 449596, "postDate": "2019-01-03T12:13:53.350Z", "content": "I have read CPMP's implementation but I am still confused how to use fm/ffm to make features, you did fm/ffm with binary task or?what's your target in your model?
", "rawMarkdown": "I have read CPMP's implementation but I am still confused how to use fm/ffm to make features, you did fm/ffm with binary task or?what's your target in your model?" }, { "id": 449614, "postDate": "2019-01-03T12:38:54.443Z", "content": "in his implementation, he uses count (i.e. combination of categorical) as feature.
\n\nit is up to you to decide how to do this, for instance, you can do it in train/test level, or you can also do it in transaction level. one thing to look out for, is that you will need to scale the target properly.
\n\nYou can generate feature with his \"model feature\" output as output from the trained network. \nHope this helps.
", "rawMarkdown": "in his implementation, he uses count (i.e. combination of categorical) as feature. \n\nit is up to you to decide how to do this, for instance, you can do it in train/test level, or you can also do it in transaction level. one thing to look out for, is that you will need to scale the target properly.\n\nYou can generate feature with his \"model feature\" output as output from the trained network. \nHope this helps.", "votes": 1 }, { "id": 456031, "postDate": "2019-01-15T02:07:36.770Z", "content": "Thanks for your reply. I have try the feature1,feature2,feature3's in train/test level, and category1, category2, category3 in transaction level, use count as feature. But \"model feature\" dosen't helps my CV and LB. I'm curious about what you said \"you will need to scale the target properly\".
", "rawMarkdown": "Thanks for your reply. I have try the feature1,feature2,feature3's in train/test level, and category1, category2, category3 in transaction level, use count as feature. But \"model feature\" dosen't helps my CV and LB. I'm curious about what you said \"you will need to scale the target properly\".", "votes": 1 }, { "id": 456351, "postDate": "2019-01-15T16:42:00.823Z", "content": "I can only speak from my own experience - I have to normalise the target value in order to make this work.
\n\nWell, evidently you are already doing very well, so it maybe a case that your model already come with useful features that capture information in a way that the FM features fail to contribute to the score. I implement the FM technique relatively early in this competition, so that might explain the different experience.
", "rawMarkdown": "I can only speak from my own experience - I have to normalise the target value in order to make this work. \n\nWell, evidently you are already doing very well, so it maybe a case that your model already come with useful features that capture information in a way that the FM features fail to contribute to the score. I implement the FM technique relatively early in this competition, so that might explain the different experience. " }, { "id": 466729, "postDate": "2019-02-05T21:45:51.030Z", "content": "Hi @Yifan, how do you predict on the test set? Do you take an average of multiple models or use just one model to make the prediction? If it's the last case, it might be possible that the LB and CV deviate.
", "rawMarkdown": "Hi @Yifan, how do you predict on the test set? Do you take an average of multiple models or use just one model to make the prediction? If it's the last case, it might be possible that the LB and CV deviate.", "votes": 1 } ] }, { "id": 466686, "postDate": "2019-02-05T20:12:06.263Z", "content": "This is very helpful! Thank you for sharing.
", "rawMarkdown": "This is very helpful! Thank you for sharing." }, { "id": 463017, "postDate": "2019-01-29T08:41:02.397Z", "content": "Thank you for sharing. May I ask, when you finish your feature selection, how much has cv and lb improved?
", "rawMarkdown": "Thank you for sharing. May I ask, when you finish your feature selection, how much has cv and lb improved? ", "replies": [ { "id": 463183, "postDate": "2019-01-29T15:24:50.943Z", "content": "hard to say, it was not a one-off process. I iterated the cycle of features creation and features selection many times.
", "rawMarkdown": "hard to say, it was not a one-off process. I iterated the cycle of features creation and features selection many times. ", "votes": 2 } ] }, { "id": 462302, "postDate": "2019-01-28T04:28:27.627Z", "content": "This is a great post, thanks for sharing your experiences!
", "rawMarkdown": "This is a great post, thanks for sharing your experiences!" }, { "id": 461312, "postDate": "2019-01-25T18:35:23.100Z", "content": "@yifanxie, thank you so much for your sharing! Getting stuck at a score and found no directions of improvement are no fun at all @@\"
", "rawMarkdown": "@yifanxie, thank you so much for your sharing! Getting stuck at a score and found no directions of improvement are no fun at all @@\"" }, { "id": 457675, "postDate": "2019-01-17T23:01:31.630Z", "content": "\n\nThanks for sharing. Which method did you use for feature selection? RFE with gbm didn't work for me.
", "rawMarkdown": "@yifanxie\n\nThanks for sharing. Which method did you use for feature selection? RFE with gbm didn't work for me.", "replies": [ { "id": 457679, "postDate": "2019-01-17T23:12:50.450Z", "content": "I am using a modified version of this: https://www.kaggle.com/ogrellier/feature-selection-with-null-importances
\n\nfull credits go to @ogrellier
", "rawMarkdown": "I am using a modified version of this: https://www.kaggle.com/ogrellier/feature-selection-with-null-importances\n\nfull credits go to @ogrellier", "votes": 2 }, { "id": 457721, "postDate": "2019-01-18T01:23:58.427Z", "content": "I also used https://www.kaggle.com/ogrellier/feature-selection-with-null-importances but can't work for me. I attribute this to my lack of features(only 150+ and doesn't needed RFE)
", "rawMarkdown": "I also used https://www.kaggle.com/ogrellier/feature-selection-with-null-importances but can't work for me. I attribute this to my lack of features(only 150+ and doesn't needed RFE)" } ] }, { "id": 455639, "postDate": "2019-01-14T09:33:10.087Z", "content": "good discussion!\ncan i know what means 'count vector & NN' in your post?
", "rawMarkdown": "good discussion!\ncan i know what means 'count vector & NN' in your post?", "replies": [ { "id": 455644, "postDate": "2019-01-14T09:38:20.873Z", "content": "count vector - count vectorization (https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)
\n\nNN - Neural Network
", "rawMarkdown": "count vector - count vectorization (https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)\n\nNN - Neural Network", "votes": 1 } ] }, { "id": 454886, "postDate": "2019-01-12T12:34:47.480Z", "content": "Thank you for sharing these ideas, your philosophy of works make the workflow clearer more decent and scientific. Seems like taking a walk really help a lot.
", "rawMarkdown": "Thank you for sharing these ideas, your philosophy of works make the workflow clearer more decent and scientific. Seems like taking a walk really help a lot." }, { "id": 454858, "postDate": "2019-01-12T11:46:13.367Z", "content": "Thanks a lot for all of your insight!
\n\nArgh - what am I missing?
\n\nI did extensive Feature Engineering with 600~ features. i reduced it via selection methods to 36 columns that generate using single LGBM: CV 3.645 and LB 3.694~\n(My NN is doing worse)
\n\nBut that's it - I really cannot find the way to generate that CV of 3.62~ and LB of 3.67~ that you are saying is possible with some Feature Engineering.\nmy head is exploding!
\n\n... so frustrating
", "rawMarkdown": "Thanks a lot for all of your insight!\n\n\nArgh - what am I missing?\n\nI did extensive Feature Engineering with 600~ features. i reduced it via selection methods to 36 columns that generate using single LGBM: CV 3.645 and LB 3.694~\n(My NN is doing worse)\n\n\nBut that's it - I really cannot find the way to generate that CV of 3.62~ and LB of 3.67~ that you are saying is possible with some Feature Engineering.\nmy head is exploding!\n\n... so frustrating\n", "replies": [ { "id": 454950, "postDate": "2019-01-12T15:48:31.867Z", "content": "I have a feeling that you probably can also try to keep a specific branch/version of your features for optimum LB? just in case you are overfitting on CV - which is possible if you are doing lower learning rate (i.e. <0.01) or hight number of folds (i.e. K>5)
\n\nRegarding specific approach to push down RMSE, all I would say is definitely scout previous competition, as well as other literature for new ideas - don't dwell on fine-tuning the same idea that you have tried.
\n\nI have a hunch that most of the top teams are doing something similar - obviously with some exceptions , we will see after the competition finishes :D
", "rawMarkdown": "I have a feeling that you probably can also try to keep a specific branch/version of your features for optimum LB? just in case you are overfitting on CV - which is possible if you are doing lower learning rate (i.e. <0.01) or hight number of folds (i.e. K>5) \n\nRegarding specific approach to push down RMSE, all I would say is definitely scout previous competition, as well as other literature for new ideas - don't dwell on fine-tuning the same idea that you have tried. \n\nI have a hunch that most of the top teams are doing something similar - obviously with some exceptions , we will see after the competition finishes :D", "votes": 1 }, { "id": 455140, "postDate": "2019-01-13T04:02:45.423Z", "content": "Thanks a lot for your input!\nCould you share which competitions you see are similar? I don't know the competition history so much.
\n\nThanks a lot!
", "rawMarkdown": "Thanks a lot for your input!\nCould you share which competitions you see are similar? I don't know the competition history so much.\n\nThanks a lot! " }, { "id": 455178, "postDate": "2019-01-13T06:06:49.943Z", "content": "@Yifan Thank you for sharing your information, may I ask what do you do to organize specific branch/version of features?
", "rawMarkdown": "@Yifan Thank you for sharing your information, may I ask what do you do to organize specific branch/version of features?" }, { "id": 455231, "postDate": "2019-01-13T10:51:00.750Z", "content": "I don't really particular look for \"similar\" competitions. I looked at Homecredit a bit because it also featured a lot of multi-level aggregation/groupby, I also look at Santander Production Recommendation because there is a quite a lot of time-related feature in it.
\n\nMore generally, I look at competitions where the problem and dataset I already have some knowledge about. This way it is easier for me to understand why some of the techniques are used.
\n\nBut there sometime I would just browse randomly on seemingly unrelated competition to see if I can pick-up idea that I would not have thought of otherwise
", "rawMarkdown": "I don't really particular look for \"similar\" competitions. I looked at Homecredit a bit because it also featured a lot of multi-level aggregation/groupby, I also look at Santander Production Recommendation because there is a quite a lot of time-related feature in it. \n\nMore generally, I look at competitions where the problem and dataset I already have some knowledge about. This way it is easier for me to understand why some of the techniques are used. \n\nBut there sometime I would just browse randomly on seemingly unrelated competition to see if I can pick-up idea that I would not have thought of otherwise\n", "votes": 4 }, { "id": 455232, "postDate": "2019-01-13T10:57:36.913Z", "content": "just use google spreadsheet. as I progress in the competition, I've been grouping my features and pickle them into different files, and wrote down what each file is about. When I create feature selection, I then save such selected features as \"baseline feature\", again write down what are the original feature files that I selected from, and what was my selection criteria.
\n\nFor a competition with this data size, this approach works for me. For competition with much larger dataset, such as Talking Data last year, I would just create feature pipeline logic, and use git to configure manage the pipeline
", "rawMarkdown": "just use google spreadsheet. as I progress in the competition, I've been grouping my features and pickle them into different files, and wrote down what each file is about. When I create feature selection, I then save such selected features as \"baseline feature\", again write down what are the original feature files that I selected from, and what was my selection criteria.\n\nFor a competition with this data size, this approach works for me. For competition with much larger dataset, such as Talking Data last year, I would just create feature pipeline logic, and use git to configure manage the pipeline ", "votes": 3 } ] }, { "id": 454164, "postDate": "2019-01-11T07:29:51.403Z", "content": "just one simple question, when I finish feature engineering and use all features in XGB, I found that 1/3 of the features has 0 importance, what's the reason for that? Thanks.
", "rawMarkdown": "just one simple question, when I finish feature engineering and use all features in XGB, I found that 1/3 of the features has 0 importance, what's the reason for that? Thanks.", "replies": [ { "id": 454315, "postDate": "2019-01-11T12:24:02.167Z", "content": "the reason is XGB didn't find them to be useful at all, if you run it through LGB you will find the same, that not all of your features are included in feature importance.
", "rawMarkdown": "the reason is XGB didn't find them to be useful at all, if you run it through LGB you will find the same, that not all of your features are included in feature importance. " } ] }, { "id": 450267, "postDate": "2019-01-04T15:45:37.480Z", "content": "Nice thoughts
", "rawMarkdown": "Nice thoughts" }, { "id": 449683, "postDate": "2019-01-03T15:32:09.963Z", "content": "Wow! I'm Impressed with your work man. How long did it take you to get comfortable with the material?
", "rawMarkdown": "Wow! I'm Impressed with your work man. How long did it take you to get comfortable with the material? ", "replies": [ { "id": 449764, "postDate": "2019-01-03T17:52:11.470Z", "content": "if it is me that you are asking, I have never been comfortable on any of these ML stuffs I am doing - I still need to constantly google the most basic things - and the field is moving so fast you need to keep renewing yourself anyway.
\n\nI guess once you done a few projects/competitions in a specific areas (NLP, Time Series, Image, etc) you start to have a rough idea what would work as a baseline, and what are the more advanced techniques - getting to know other kagglers and exchange with each other HELPS a lot
", "rawMarkdown": "if it is me that you are asking, I have never been comfortable on any of these ML stuffs I am doing - I still need to constantly google the most basic things - and the field is moving so fast you need to keep renewing yourself anyway. \n\nI guess once you done a few projects/competitions in a specific areas (NLP, Time Series, Image, etc) you start to have a rough idea what would work as a baseline, and what are the more advanced techniques - getting to know other kagglers and exchange with each other HELPS a lot", "votes": 8 }, { "id": 449803, "postDate": "2019-01-03T19:19:20.893Z", "content": "Thanks for the reply! Best of luck to you.
", "rawMarkdown": "Thanks for the reply! Best of luck to you.", "votes": 1 } ] }, { "id": 449375, "postDate": "2019-01-03T03:39:48.483Z", "content": "Food for thought!
", "rawMarkdown": "Food for thought!" }, { "id": 448252, "postDate": "2018-12-31T15:30:56.647Z", "content": "Very inspiring!
", "rawMarkdown": "Very inspiring!" }, { "id": 448133, "postDate": "2018-12-31T10:07:42.157Z", "content": ".
", "rawMarkdown": "." }, { "id": 447956, "postDate": "2018-12-30T22:37:20.090Z", "content": "Thanks so much for your kind sharing! Can I ask that when you say \"regular feature engineering would bring LB score down to 3.680\", do you mean a single model before ensembling or not?
", "rawMarkdown": "Thanks so much for your kind sharing! Can I ask that when you say \"regular feature engineering would bring LB score down to 3.680\", do you mean a single model before ensembling or not? ", "replies": [ { "id": 447962, "postDate": "2018-12-30T23:21:28.703Z", "content": "yes, that is what I mean - I have not exactly tested this, my best single model with groupby/count/categorical encoding features brings me somewhere around that line.
\n\nI won't be too surprised if some other top competitors can surpassed that though
", "rawMarkdown": "yes, that is what I mean - I have not exactly tested this, my best single model with groupby/count/categorical encoding features brings me somewhere around that line. \n\nI won't be too surprised if some other top competitors can surpassed that though\n", "votes": 2 }, { "id": 448018, "postDate": "2018-12-31T04:17:54.980Z", "content": "That helps a lot;)
", "rawMarkdown": "That helps a lot;)" } ] }, { "id": 447285, "postDate": "2018-12-29T14:51:49.520Z", "content": "Very cool! Appreciate the time and effort!
", "rawMarkdown": "Very cool! Appreciate the time and effort!" }, { "id": 447167, "postDate": "2018-12-29T09:16:00.433Z", "content": "i'm novice at the kaggle competition, thank u for sharing.
", "rawMarkdown": "i'm novice at the kaggle competition, thank u for sharing." }, { "id": 446853, "postDate": "2018-12-28T19:27:05.923Z", "content": "thank you for sharing. no wonder you are at 3rd position in the competition already.
", "rawMarkdown": "thank you for sharing. no wonder you are at 3rd position in the competition already." }, { "id": 450206, "postDate": "2019-01-04T13:44:43.593Z", "rawMarkdown": "", "isDeleted": true }, { "id": 448777, "postDate": "2019-01-02T04:34:46.523Z", "rawMarkdown": "", "isDeleted": true }, { "id": 446849, "postDate": "2018-12-28T19:18:40.147Z", "rawMarkdown": "", "isDeleted": true }, { "id": 465639, "postDate": "2019-02-03T16:51:35.803Z", "content": "Thanks for sharing your experiences.
", "rawMarkdown": "Thanks for sharing your experiences.\n\n" }, { "id": 463031, "postDate": "2019-01-29T09:31:24.903Z", "content": "Thanks for sharing your experiences.
", "rawMarkdown": "Thanks for sharing your experiences. " }, { "id": 449462, "postDate": "2019-01-03T07:18:17.080Z", "content": "thanks for sharing!
", "rawMarkdown": "thanks for sharing!" }, { "id": 448880, "postDate": "2019-01-02T09:49:51.820Z", "content": "Well said, thanks for sharing
", "rawMarkdown": "Well said, thanks for sharing" }, { "id": 448836, "postDate": "2019-01-02T07:43:00.857Z", "content": "Thanks for sharing your experiences!!
", "rawMarkdown": "Thanks for sharing your experiences!!" }, { "id": 447341, "postDate": "2018-12-29T17:16:46.273Z", "content": "Insightful, thank you!
", "rawMarkdown": "Insightful, thank you!" }, { "id": 447219, "postDate": "2018-12-29T11:42:10.813Z", "content": "Thank you
", "rawMarkdown": "Thank you" }, { "id": 447147, "postDate": "2018-12-29T08:14:15.733Z", "content": "Thanks for sharing!!
", "rawMarkdown": "Thanks for sharing!!" }, { "id": 446963, "postDate": "2018-12-28T23:21:04.733Z", "content": "Thanks for sharing, excellent advice!
", "rawMarkdown": "Thanks for sharing, excellent advice!" }, { "id": 446938, "postDate": "2018-12-28T22:20:26.517Z", "content": "Thank you for sharing
", "rawMarkdown": "Thank you for sharing" }, { "id": 446862, "postDate": "2018-12-28T19:52:34.077Z", "content": "Thanks for sharing !!!
", "rawMarkdown": "Thanks for sharing !!!" }, { "id": 446851, "postDate": "2018-12-28T19:21:09.890Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing" } ], "comments": [ { "id": 475697, "author_name": "Bapireddy", "author_url": "", "post_date": "2019-02-21T04:43:43.797000", "content": "@Yifan Xie Thanks for the guide. About Making sure our code is reproducible. Can you share excel sheets related any of past competitions for reference. Also do you follow any style guide for python.
", "votes": 3, "replies": [ { "id": 475910, "author_name": "CPMP", "author_url": "", "post_date": "2019-02-21T11:07:48.807000", "content": "The ultimate Python style guide: https://www.python.org/dev/peps/pep-0008/
", "votes": 6, "replies": [] }, { "id": 476146, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-02-21T16:53:28.963000", "content": "👆
", "votes": 2, "replies": [] } ] }, { "id": 457132, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-01-17T01:50:56.517000", "content": "Hi Yifan!\ncan i know the meaning of Features Selection for u ? It improve the gap between cv & lb but can't work for me, so I wanna know the meaning for u\nThanks
", "votes": 1, "replies": [ { "id": 457485, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-17T14:38:39.363000", "content": "interesting question, I mainly do \"feature selection\" to identify subsets of all available features which provide better model performance - in the context of a competition like this, better model performance for me is 1) ideally better CV & LB, 2) at least better CV. In real world scenario, I would consider other criterias such as modelling time, and consistency between folds - you could very well make an argument these should also be included here.
\n\nHope this helps?
", "votes": 1, "replies": [] }, { "id": 457718, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-01-18T01:21:37.250000", "content": "Yup! I agree with all your points, the other is prevent the bad-influence of high-dimensions
", "votes": 0, "replies": [] } ] }, { "id": 451952, "author_name": "sergeman", "author_url": "", "post_date": "2019-01-08T00:20:06.340000", "content": "A used pen is better indeed. Thank you for the post. It is very helpful
", "votes": 1, "replies": [] }, { "id": 447752, "author_name": "ying", "author_url": "", "post_date": "2018-12-30T13:26:37.940000", "content": "thanks so much.you said we should takes notes about our own code,it is a very very good tips for me. I often takes time to reproduce last code.And can't remember where i have changed.I will take notes from now
", "votes": 1, "replies": [] }, { "id": 447020, "author_name": "stout", "author_url": "", "post_date": "2018-12-29T02:16:19.633000", "content": "Thank you for sharing your experience! Good food for thought and investigation...
", "votes": 1, "replies": [] }, { "id": 446608, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2018-12-28T11:22:42.773000", "content": "Nice thoughts!
", "votes": 1, "replies": [] }, { "id": 446512, "author_name": "Truoc Pham", "author_url": "", "post_date": "2018-12-28T08:07:51.703000", "content": "Thank you, I have a question: In the best data version that transformed, are you using info from merchants table?
", "votes": 1, "replies": [ { "id": 446552, "author_name": "Yifan Xie", "author_url": "", "post_date": "2018-12-28T09:20:13.323000", "content": "I haven't used features from the merchant table so far, to put it in another way, I have made some features from it, but they always got filtered out by my feature selection mechanism
", "votes": 4, "replies": [] } ] }, { "id": 446403, "author_name": "Manchoi", "author_url": "", "post_date": "2018-12-28T02:31:23.337000", "content": "Thanks for your sharing, which has helped & inspired me a lot.
", "votes": 1, "replies": [] }, { "id": 446293, "author_name": "C4rl05/V", "author_url": "", "post_date": "2018-12-27T20:12:47.520000", "content": "Thanks for sharing, excellent advice, it will help me to improve my score and workflow.
", "votes": 1, "replies": [] }, { "id": 446242, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-12-27T18:55:38.337000", "content": "Thanks for sharing! What's your CV score, if you don't mind me asking? I'm curious what it takes to get to 3rd place, when the CV-LB diff is so large and variable.
", "votes": 1, "replies": [ { "id": 446247, "author_name": "Yifan Xie", "author_url": "", "post_date": "2018-12-27T18:58:40.337000", "content": "I am on about 3.620 - I am living with the daily fear that I might have overfitted this badly! ¯_(ツ) _/¯
", "votes": 7, "replies": [] }, { "id": 446254, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-12-27T19:03:04.340000", "content": "Wow, that's pretty nice. Thanks!
\n\nWhat about single model, no post-processing CV? I'm curious how much is possible from raw feature engineering + feature selection and how much I need to go \"out of the box\".
", "votes": 1, "replies": [] }, { "id": 446258, "author_name": "Yifan Xie", "author_url": "", "post_date": "2018-12-27T19:08:01.300000", "content": "for me, with standard supervised-learning - our everyday x_train, y_train, and x_test - I think I am on about 3.640 CV
", "votes": 5, "replies": [] }, { "id": 446261, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-12-27T19:10:38.207000", "content": "Sweet, thanks. That's helpful for my own calibration. :)
", "votes": 1, "replies": [] }, { "id": 447747, "author_name": "ying", "author_url": "", "post_date": "2018-12-30T13:16:59.263000", "content": "Hi,thank you for the discuss. you only use agg features and get 3.640 CV or agg features and other features?thanks
", "votes": 0, "replies": [] }, { "id": 447963, "author_name": "Yifan Xie", "author_url": "", "post_date": "2018-12-30T23:28:14.503000", "content": "that would be agg + other features - that 3.640CV score is just a rough estimation I made to answer Peter's question though. You can for sure improve on that, by some margin
", "votes": 0, "replies": [] } ] }, { "id": 460799, "author_name": "Muhammad Tahir A.", "author_url": "", "post_date": "2019-01-24T12:52:13.863000", "content": "Friends have you faced any loading issue in historical_transactions.csv
", "votes": 2, "replies": [ { "id": 461007, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-01-25T01:21:36.640000", "content": "what‘s ur problem?
", "votes": 0, "replies": [] } ] }, { "id": 449433, "author_name": "Angus Chang", "author_url": "", "post_date": "2019-01-03T06:22:13.030000", "content": "Thx for the guide. I've tried some fancy agg that I learned in LSST but seems only helps a little, currently at cv 3.6507 / lb 3.70. BTW, how many features do have in your best single model?
", "votes": 2, "replies": [ { "id": 449440, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-03T06:29:57.360000", "content": "Let's define \"best\" first :) \nIn my case I am still in the vain attempt to produce model that improve both on CV and LB. And I have been pretty much stuck in the last few days.
\n\nCurrently my best single model has around 200 features
", "votes": 3, "replies": [] }, { "id": 449540, "author_name": "Angus Chang", "author_url": "", "post_date": "2019-01-03T10:26:06.690000", "content": "Thanks a lot! and the \"Best\" here I means best CV :))
\nFYI, I'm using over 1000 features with the model I mentioned above, I'll definitely try feature selection after I finished all the FE stuff.
Maybe you can make more model to improve your score now, I used 7 models that their corr-matrix is between 0.95-0.97, it gave me so much help. For example, you can try to the Graph-Model. Yep, try the everythin to vector
", "votes": 3, "replies": [] }, { "id": 449613, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-03T12:36:07.810000", "content": "I was quite happy with my CV before I got to 3.670 - now CV and LB no longer agree with each other. \nI also observe that my feature selection strategy starting to falter at this level. It is starting to give me feature set that gives better CV score on higher learning rate (i.e. 0.07) than \"reasonable\" lower learning rate (i.e. 0.01) - this is very rarely the case in my previous experience.
\n\nI am now thinking about reducing my fold numbers to re-evaluate some of my key features, and potentially rebuild my whole model generation stack
", "votes": 0, "replies": [] }, { "id": 450550, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2019-01-05T08:19:54.727000", "content": "Many things to do, hard way to go!
", "votes": 0, "replies": [] }, { "id": 468296, "author_name": "Dylan_Xia", "author_url": "", "post_date": "2019-02-08T16:35:36.207000", "content": "Hey, Shuo-Jen Chang, you said \"fancy agg that I learned in LSST\", do you have a link? can you share it with me?
", "votes": 1, "replies": [] } ] }, { "id": 446388, "author_name": "SubikashPal", "author_url": "", "post_date": "2018-12-28T01:41:42.877000", "content": "I have one question.How are you selecting the features order?\ndid u use ffm here?any sample ffm working would be nice...\nThank u for the post
", "votes": 2, "replies": [ { "id": 446395, "author_name": "Yifan Xie", "author_url": "", "post_date": "2018-12-28T01:51:21.970000", "content": "I didn't \"select\" feature order, I group my features into different files and load them one by one while starting my model. It happened that I loaded two of them in different order, and I just observed the LB score difference.
\n\nI have not use FFM yet, I would like to read the paper first before trying this algorithm. I have some success with FM, and I suggest you to check out CPMP's implementation of LibFM in keras. He shared it in the Talking Data competition this year, search the forum. I found his implementation to be more flexible for my purpose, but you may find something else more suitable for your need.
", "votes": 9, "replies": [] }, { "id": 446733, "author_name": "SubikashPal", "author_url": "", "post_date": "2018-12-28T15:43:12.103000", "content": "Thank you..trying to understand CPMP's implementation. Thank you.
", "votes": 1, "replies": [] }, { "id": 446871, "author_name": "Chia-Ta Tsai", "author_url": "", "post_date": "2018-12-28T20:08:12.363000", "content": "@Yifan IMHO, it is by design to have different results while the same feature sets but their sorting order are different of GBMs. Saying data A and A' with feature columns shuffled. Column-wise sampled should have different results at very beginning and then leading to different tree growths. For example, given a random seed therefore the first round sampling to pick the first two columns out of three in A is ['a', 'b', 'c'] and A' is ['a', 'c', 'b']. As a result, A got ['a', 'b'] but A' got ['a', 'c']. Then model build on A and A' should go different directions, respectively.
\n\nFurthermore, to make it persistent, sort columns before model.fit might be a good practice.
", "votes": 0, "replies": [] }, { "id": 446875, "author_name": "Chia-Ta Tsai", "author_url": "", "post_date": "2018-12-28T20:13:32.200000", "content": "On the other hands, shuffle columns and rows might be a good way to moderately diversify models and predictions. B-)
", "votes": 2, "replies": [] }, { "id": 446884, "author_name": "Yifan Xie", "author_url": "", "post_date": "2018-12-28T20:34:47.723000", "content": "@cttsai yes, GMB by design will sample columns - but this is not my point. My point was that this kind of random order had a significant impact on LB score, in my case up to 0.005 difference, and I also note that my local CV for the same model with different column orders remained consistant. I am suggesting that we shall be observant to this - how one decides to act on it is entirely each participant's choice.
\n\nI am simply saying this dataset might be sensitive to choice of features used in the modelling - and thatI it had been useful for me during this competition to invest time in feature selection. I hope this help to clarify my original comment.
", "votes": 1, "replies": [] }, { "id": 446903, "author_name": "Chia-Ta Tsai", "author_url": "", "post_date": "2018-12-28T21:03:14.407000", "content": "@Yifan, thanks for the clarification. It is interesting that models that have similar CV results but output huge LB shakeup, which their only difference are in column sorting order! May I ask what is the range of your column subsample rate? Is it as high as 0.8, 0.9 or much lower?
", "votes": 1, "replies": [] }, { "id": 446911, "author_name": "Yifan Xie", "author_url": "", "post_date": "2018-12-28T21:18:34.527000", "content": "I was on 0.7 by tree and 0.6 by level for LGB, I have since developed several sets of parameters for different variants of LGB/XGB, and again for different combination of random seed, and feature sets. \nmost of the col sampling I am using these days are between 0.4 to 0.6.
", "votes": 4, "replies": [] }, { "id": 446942, "author_name": "Chia-Ta Tsai", "author_url": "", "post_date": "2018-12-28T22:31:47.047000", "content": "@Yifan, thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 449596, "author_name": "Ethan", "author_url": "", "post_date": "2019-01-03T12:13:53.350000", "content": "I have read CPMP's implementation but I am still confused how to use fm/ffm to make features, you did fm/ffm with binary task or?what's your target in your model?
", "votes": 0, "replies": [] }, { "id": 449614, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-03T12:38:54.443000", "content": "in his implementation, he uses count (i.e. combination of categorical) as feature.
\n\nit is up to you to decide how to do this, for instance, you can do it in train/test level, or you can also do it in transaction level. one thing to look out for, is that you will need to scale the target properly.
\n\nYou can generate feature with his \"model feature\" output as output from the trained network. \nHope this helps.
", "votes": 1, "replies": [] }, { "id": 456031, "author_name": "Ethan", "author_url": "", "post_date": "2019-01-15T02:07:36.770000", "content": "Thanks for your reply. I have try the feature1,feature2,feature3's in train/test level, and category1, category2, category3 in transaction level, use count as feature. But \"model feature\" dosen't helps my CV and LB. I'm curious about what you said \"you will need to scale the target properly\".
", "votes": 1, "replies": [] }, { "id": 456351, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-15T16:42:00.823000", "content": "I can only speak from my own experience - I have to normalise the target value in order to make this work.
\n\nWell, evidently you are already doing very well, so it maybe a case that your model already come with useful features that capture information in a way that the FM features fail to contribute to the score. I implement the FM technique relatively early in this competition, so that might explain the different experience.
", "votes": 0, "replies": [] }, { "id": 466729, "author_name": "lucaskg", "author_url": "", "post_date": "2019-02-05T21:45:51.030000", "content": "Hi @Yifan, how do you predict on the test set? Do you take an average of multiple models or use just one model to make the prediction? If it's the last case, it might be possible that the LB and CV deviate.
", "votes": 1, "replies": [] } ] }, { "id": 466686, "author_name": "Kadeem Baldwin", "author_url": "", "post_date": "2019-02-05T20:12:06.263000", "content": "This is very helpful! Thank you for sharing.
", "votes": 0, "replies": [] }, { "id": 463017, "author_name": "MoreScroes", "author_url": "", "post_date": "2019-01-29T08:41:02.397000", "content": "Thank you for sharing. May I ask, when you finish your feature selection, how much has cv and lb improved?
", "votes": 0, "replies": [ { "id": 463183, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-29T15:24:50.943000", "content": "hard to say, it was not a one-off process. I iterated the cycle of features creation and features selection many times.
", "votes": 2, "replies": [] } ] }, { "id": 462302, "author_name": "Antra Tyagi", "author_url": "", "post_date": "2019-01-28T04:28:27.627000", "content": "This is a great post, thanks for sharing your experiences!
", "votes": 0, "replies": [] }, { "id": 461312, "author_name": "FP", "author_url": "", "post_date": "2019-01-25T18:35:23.100000", "content": "@yifanxie, thank you so much for your sharing! Getting stuck at a score and found no directions of improvement are no fun at all @@\"
", "votes": 0, "replies": [] }, { "id": 457675, "author_name": "kxx", "author_url": "", "post_date": "2019-01-17T23:01:31.630000", "content": "\n\nThanks for sharing. Which method did you use for feature selection? RFE with gbm didn't work for me.
", "votes": 0, "replies": [ { "id": 457679, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-17T23:12:50.450000", "content": "I am using a modified version of this: https://www.kaggle.com/ogrellier/feature-selection-with-null-importances
\n\nfull credits go to @ogrellier
", "votes": 2, "replies": [] }, { "id": 457721, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-01-18T01:23:58.427000", "content": "I also used https://www.kaggle.com/ogrellier/feature-selection-with-null-importances but can't work for me. I attribute this to my lack of features(only 150+ and doesn't needed RFE)
", "votes": 0, "replies": [] } ] }, { "id": 455639, "author_name": "", "author_url": "", "post_date": "2019-01-14T09:33:10.087000", "content": "good discussion!\ncan i know what means 'count vector & NN' in your post?
", "votes": 0, "replies": [ { "id": 455644, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-14T09:38:20.873000", "content": "count vector - count vectorization (https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)
\n\nNN - Neural Network
", "votes": 1, "replies": [] } ] }, { "id": 454886, "author_name": "Kun-Lin Lee", "author_url": "", "post_date": "2019-01-12T12:34:47.480000", "content": "Thank you for sharing these ideas, your philosophy of works make the workflow clearer more decent and scientific. Seems like taking a walk really help a lot.
", "votes": 0, "replies": [] }, { "id": 454858, "author_name": "AmirH", "author_url": "", "post_date": "2019-01-12T11:46:13.367000", "content": "Thanks a lot for all of your insight!
\n\nArgh - what am I missing?
\n\nI did extensive Feature Engineering with 600~ features. i reduced it via selection methods to 36 columns that generate using single LGBM: CV 3.645 and LB 3.694~\n(My NN is doing worse)
\n\nBut that's it - I really cannot find the way to generate that CV of 3.62~ and LB of 3.67~ that you are saying is possible with some Feature Engineering.\nmy head is exploding!
\n\n... so frustrating
", "votes": 0, "replies": [ { "id": 454950, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-12T15:48:31.867000", "content": "I have a feeling that you probably can also try to keep a specific branch/version of your features for optimum LB? just in case you are overfitting on CV - which is possible if you are doing lower learning rate (i.e. <0.01) or hight number of folds (i.e. K>5)
\n\nRegarding specific approach to push down RMSE, all I would say is definitely scout previous competition, as well as other literature for new ideas - don't dwell on fine-tuning the same idea that you have tried.
\n\nI have a hunch that most of the top teams are doing something similar - obviously with some exceptions , we will see after the competition finishes :D
", "votes": 1, "replies": [] }, { "id": 455140, "author_name": "AmirH", "author_url": "", "post_date": "2019-01-13T04:02:45.423000", "content": "Thanks a lot for your input!\nCould you share which competitions you see are similar? I don't know the competition history so much.
\n\nThanks a lot!
", "votes": 0, "replies": [] }, { "id": 455178, "author_name": "pavitsu", "author_url": "", "post_date": "2019-01-13T06:06:49.943000", "content": "@Yifan Thank you for sharing your information, may I ask what do you do to organize specific branch/version of features?
", "votes": 0, "replies": [] }, { "id": 455231, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-13T10:51:00.750000", "content": "I don't really particular look for \"similar\" competitions. I looked at Homecredit a bit because it also featured a lot of multi-level aggregation/groupby, I also look at Santander Production Recommendation because there is a quite a lot of time-related feature in it.
\n\nMore generally, I look at competitions where the problem and dataset I already have some knowledge about. This way it is easier for me to understand why some of the techniques are used.
\n\nBut there sometime I would just browse randomly on seemingly unrelated competition to see if I can pick-up idea that I would not have thought of otherwise
", "votes": 4, "replies": [] }, { "id": 455232, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-13T10:57:36.913000", "content": "just use google spreadsheet. as I progress in the competition, I've been grouping my features and pickle them into different files, and wrote down what each file is about. When I create feature selection, I then save such selected features as \"baseline feature\", again write down what are the original feature files that I selected from, and what was my selection criteria.
\n\nFor a competition with this data size, this approach works for me. For competition with much larger dataset, such as Talking Data last year, I would just create feature pipeline logic, and use git to configure manage the pipeline
", "votes": 3, "replies": [] } ] }, { "id": 454164, "author_name": "Perfect Is Shit", "author_url": "", "post_date": "2019-01-11T07:29:51.403000", "content": "just one simple question, when I finish feature engineering and use all features in XGB, I found that 1/3 of the features has 0 importance, what's the reason for that? Thanks.
", "votes": 0, "replies": [ { "id": 454315, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-11T12:24:02.167000", "content": "the reason is XGB didn't find them to be useful at all, if you run it through LGB you will find the same, that not all of your features are included in feature importance.
", "votes": 0, "replies": [] } ] }, { "id": 450267, "author_name": "Shivam", "author_url": "", "post_date": "2019-01-04T15:45:37.480000", "content": "Nice thoughts
", "votes": 0, "replies": [] }, { "id": 449683, "author_name": "Kevin", "author_url": "", "post_date": "2019-01-03T15:32:09.963000", "content": "Wow! I'm Impressed with your work man. How long did it take you to get comfortable with the material?
", "votes": 0, "replies": [ { "id": 449764, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-01-03T17:52:11.470000", "content": "if it is me that you are asking, I have never been comfortable on any of these ML stuffs I am doing - I still need to constantly google the most basic things - and the field is moving so fast you need to keep renewing yourself anyway.
\n\nI guess once you done a few projects/competitions in a specific areas (NLP, Time Series, Image, etc) you start to have a rough idea what would work as a baseline, and what are the more advanced techniques - getting to know other kagglers and exchange with each other HELPS a lot
", "votes": 8, "replies": [] }, { "id": 449803, "author_name": "Kevin", "author_url": "", "post_date": "2019-01-03T19:19:20.893000", "content": "Thanks for the reply! Best of luck to you.
", "votes": 1, "replies": [] } ] }, { "id": 449375, "author_name": "Arsalan Azmi", "author_url": "", "post_date": "2019-01-03T03:39:48.483000", "content": "Food for thought!
", "votes": 0, "replies": [] }, { "id": 448252, "author_name": "abena", "author_url": "", "post_date": "2018-12-31T15:30:56.647000", "content": "Very inspiring!
", "votes": 0, "replies": [] }, { "id": 448133, "author_name": "Yota.F", "author_url": "", "post_date": "2018-12-31T10:07:42.157000", "content": ".
", "votes": 0, "replies": [] }, { "id": 447956, "author_name": "WA", "author_url": "", "post_date": "2018-12-30T22:37:20.090000", "content": "Thanks so much for your kind sharing! Can I ask that when you say \"regular feature engineering would bring LB score down to 3.680\", do you mean a single model before ensembling or not?
", "votes": 0, "replies": [ { "id": 447962, "author_name": "Yifan Xie", "author_url": "", "post_date": "2018-12-30T23:21:28.703000", "content": "yes, that is what I mean - I have not exactly tested this, my best single model with groupby/count/categorical encoding features brings me somewhere around that line.
\n\nI won't be too surprised if some other top competitors can surpassed that though
", "votes": 2, "replies": [] }, { "id": 448018, "author_name": "WA", "author_url": "", "post_date": "2018-12-31T04:17:54.980000", "content": "That helps a lot;)
", "votes": 0, "replies": [] } ] }, { "id": 447285, "author_name": "AustonLi", "author_url": "", "post_date": "2018-12-29T14:51:49.520000", "content": "Very cool! Appreciate the time and effort!
", "votes": 0, "replies": [] }, { "id": 447167, "author_name": "", "author_url": "", "post_date": "2018-12-29T09:16:00.433000", "content": "", "votes": 0, "replies": [] }, { "id": 446853, "author_name": "", "author_url": "", "post_date": "2018-12-28T19:27:05.923000", "content": "", "votes": 0, "replies": [] }, { "id": 450206, "author_name": "", "author_url": "", "post_date": "2019-01-04T13:44:43.593000", "content": "", "votes": 0, "replies": [] }, { "id": 448777, "author_name": "", "author_url": "", "post_date": "2019-01-02T04:34:46.523000", "content": "", "votes": 0, "replies": [] }, { "id": 446849, "author_name": "", "author_url": "", "post_date": "2018-12-28T19:18:40.147000", "content": "", "votes": 0, "replies": [] }, { "id": 465639, "author_name": "", "author_url": "", "post_date": "2019-02-03T16:51:35.803000", "content": "", "votes": 0, "replies": [] }, { "id": 463031, "author_name": "", "author_url": "", "post_date": "2019-01-29T09:31:24.903000", "content": "", "votes": 0, "replies": [] }, { "id": 449462, "author_name": "", "author_url": "", "post_date": "2019-01-03T07:18:17.080000", "content": "", "votes": 0, "replies": [] }, { "id": 448880, "author_name": "", "author_url": "", "post_date": "2019-01-02T09:49:51.820000", "content": "", "votes": 0, "replies": [] }, { "id": 448836, "author_name": "", "author_url": "", "post_date": "2019-01-02T07:43:00.857000", "content": "", "votes": 0, "replies": [] }, { "id": 447341, "author_name": "", "author_url": "", "post_date": "2018-12-29T17:16:46.273000", "content": "", "votes": 0, "replies": [] }, { "id": 447219, "author_name": "", "author_url": "", "post_date": "2018-12-29T11:42:10.813000", "content": "", "votes": 0, "replies": [] }, { "id": 447147, "author_name": "", "author_url": "", "post_date": "2018-12-29T08:14:15.733000", "content": "", "votes": 0, "replies": [] }, { "id": 446963, "author_name": "", "author_url": "", "post_date": "2018-12-28T23:21:04.733000", "content": "", "votes": 0, "replies": [] }, { "id": 446938, "author_name": "", "author_url": "", "post_date": "2018-12-28T22:20:26.517000", "content": "", "votes": 0, "replies": [] }, { "id": 446862, "author_name": "", "author_url": "", "post_date": "2018-12-28T19:52:34.077000", "content": "", "votes": 0, "replies": [] }, { "id": 446851, "author_name": "", "author_url": "", "post_date": "2018-12-28T19:21:09.890000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "446239": "Hello all, I hope everyone is enjoying a very nice period toward the end of the year, and for me, it is definitely great fun to have some dedicated time to immerse myself in a kaggle competition without thinking about other daily stuff.\n\nIt is mid-way through this nice competition(or maybe not so nice depends on how your look at it). I would like to share some of my experience so far. Some are successful, some are not, but they are all good experience. \n\nso here we go:\n- Feature engineering: my personal feeling is that regular feature engineering (typical groupby-agg, counting, categorical encoding) would bring LB score down (i.e. RMSE) to 3.680, but no more. I personally starting to find it difficult to improve beyond the 3.690 mark. Beyond this line, I started to look for other ways that would give me features. Some of these have already been mentioned in the forum, such as count vector, FM/FFM, NN, and out of fold model.\n\n- Feature selection: I have found this dataset to be quite sensitive to 1) selection of features, and 2) order of features in the dataset - especially if you are using algorithms that choose sub-set of features (i.e. GBM). By switching columns order, I have had scores with 0.005 difference on the LB (obviously with very similar CV). Usually I don't do too much feature selection in a kaggle competition, but now I found this (often overlooked) technique to be useful BOTH in improving CV/LB score, and also helps to reduce computation time. I strongly encourage everyone to have a look at the post from @peterhurford titled \"[Less is More][1]\". Everything I used so far w.r.t. feature selection comes from that post.\n\n- Get your basics right: Each and every kaggle competition shall be treated as a work/study project in which you should setup a proper workflow to structure your work. and I can not emphasise more about this - every model you chunk out, MAKE SURE IT IS REPRODUCIBLE - ok some algorithms/tool are non-deterministic - but then again you should make sure you can reproduce your model with similar baseline performance. Personally, for each model I create, I use a good-old spreadsheet to capture the following: algorithm used, parameters, features used, seed, cross-validation approach(kfold, stratified kfold, etc), CV score, LB score(if submitted), score at each fold, std of CV score. everyone's approach differs, but still I encourage everyone to write these down - in chinese we have the saying \"a good memory is not as good as a used pen\" :)\n\n- Think \"out-of-box\": the elephant(s) in the room is that there are 2000+ outliers which contribute to most of the RMSE, and this requires us to take time to think how to address this. there is quite a lot of discussion on this, and the [nice kernel about post-processing][2] from @waitingli, and also the [discussion thread][3] from @lblhandsome. I don't have much to add to this, but what I would say is make sure you spend time away from your computer(s). Take a walk, spend time with your family/friends, and eat/sleep properly. For me, the several ideas that really work came when I was not facing the screen busy coding. \n\n- Manage your code: always keep your code in proper modules or classes, so that they can be reused for your future projects and competitions. A lot of code I am using to push myself up the LB has from my own little repo - so I have probably saved hundreds of hours by reusing my previous stuff. In a sense, you need to find a way to team up with your previous selves :)\n\nWhat I have not yet managed to figured out is that magical something both @senkin13 and @lblhandsome have found. So actually everything I have done so far have been fully shared by many others in many previous competitions - I can confirm that without anything \"magic\" or \"leaky\" you could get up to at least 3.67x in public LB - well, private LB is another story, and probably worthy of another post to discuss how much shake-up there will be - insert :trollface: \n\nAnyway, I hope everyone enjoys this competition, and have a great time! \n \n\n\n [1]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73937\n [2]: https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier\n [3]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75034#440855", "475697": "@Yifan Xie Thanks for the guide. About Making sure our code is reproducible. Can you share excel sheets related any of past competitions for reference. Also do you follow any style guide for python.", "457132": "Hi Yifan!\ncan i know the meaning of Features Selection for u ? It improve the gap between cv & lb but can't work for me, so I wanna know the meaning for u\nThanks", "451952": "A used pen is better indeed. Thank you for the post. It is very helpful ", "447752": "thanks so much.you said we should takes notes about our own code,it is a very very good tips for me. I often takes time to reproduce last code.And can't remember where i have changed.I will take notes from now", "447020": "Thank you for sharing your experience! Good food for thought and investigation...", "446608": "Nice thoughts!", "446512": "Thank you, I have a question: In the best data version that transformed, are you using info from merchants table?", "446403": "Thanks for your sharing, which has helped & inspired me a lot.", "446293": "Thanks for sharing, excellent advice, it will help me to improve my score and workflow.", "446242": "Thanks for sharing! What's your CV score, if you don't mind me asking? I'm curious what it takes to get to 3rd place, when the CV-LB diff is so large and variable.", "460799": "Friends have you faced any loading issue in historical_transactions.csv", "449433": "Thx for the guide. I've tried some fancy agg that I learned in LSST but seems only helps a little, currently at cv 3.6507 / lb 3.70. BTW, how many features do have in your best single model? ", "446388": "I have one question.How are you selecting the features order?\ndid u use ffm here?any sample ffm working would be nice...\nThank u for the post", "466686": "This is very helpful! Thank you for sharing.", "463017": "Thank you for sharing. May I ask, when you finish your feature selection, how much has cv and lb improved? ", "462302": "This is a great post, thanks for sharing your experiences!", "461312": "@yifanxie, thank you so much for your sharing! Getting stuck at a score and found no directions of improvement are no fun at all @@\"", "457675": "@yifanxie\n\nThanks for sharing. Which method did you use for feature selection? RFE with gbm didn't work for me.", "455639": "good discussion!\ncan i know what means 'count vector & NN' in your post?", "454886": "Thank you for sharing these ideas, your philosophy of works make the workflow clearer more decent and scientific. Seems like taking a walk really help a lot.", "454858": "Thanks a lot for all of your insight!\n\n\nArgh - what am I missing?\n\nI did extensive Feature Engineering with 600~ features. i reduced it via selection methods to 36 columns that generate using single LGBM: CV 3.645 and LB 3.694~\n(My NN is doing worse)\n\n\nBut that's it - I really cannot find the way to generate that CV of 3.62~ and LB of 3.67~ that you are saying is possible with some Feature Engineering.\nmy head is exploding!\n\n... so frustrating\n", "454164": "just one simple question, when I finish feature engineering and use all features in XGB, I found that 1/3 of the features has 0 importance, what's the reason for that? Thanks.", "450267": "Nice thoughts", "449683": "Wow! I'm Impressed with your work man. How long did it take you to get comfortable with the material? ", "449375": "Food for thought!", "448252": "Very inspiring!", "448133": ".", "447956": "Thanks so much for your kind sharing! Can I ask that when you say \"regular feature engineering would bring LB score down to 3.680\", do you mean a single model before ensembling or not? ", "447285": "Very cool! Appreciate the time and effort!", "447167": "i'm novice at the kaggle competition, thank u for sharing.", "446853": "thank you for sharing. no wonder you are at 3rd position in the competition already.", "450206": "", "448777": "", "446849": "", "465639": "Thanks for sharing your experiences.\n\n", "463031": "Thanks for sharing your experiences. ", "449462": "thanks for sharing!", "448880": "Well said, thanks for sharing", "448836": "Thanks for sharing your experiences!!", "447341": "Insightful, thank you!", "447219": "Thank you", "447147": "Thanks for sharing!!", "446963": "Thanks for sharing, excellent advice!", "446938": "Thank you for sharing", "446862": "Thanks for sharing !!!", "446851": "Thanks for sharing" } }