Hello all, firstly congratulation to the winners and top performers, your effort and sharing are great and I have learnt a lot in a very short space of time thanks to you. Also, I want to say well done for people who had been working on this from the beginning, in the end, tenacity will prevail in the form of knowledge gained, medals and good ranking.
\n\nBasically, I have treated this competition as a week-long hackathon and spent one intensive week (50h+) on it. I am very lucky to be ranked within top 100. This wouldn't have been possible without all the sharing. So big thanks from me. Here is the overall sketch of my approach to achieving top 100 ranking in this short space of time:
\n\nShared Features:
\n\nMy own features:\n 6. Lag 1 features - User total products, and user reorder sum in n-1 order (last order in prior)\n 7. Lag difference - User total products difference between orders: n-1 & n-2, n-2 & n-3, n-3 & n-4\n 8. Skew statistic on user product state: so let' say a user had 5 orders, for a given product, the product purchase states in these orders were - for argument sake - [0 0 1 1 1 ]. so I capture this series and calculate the Skew and IPR measure - but only the skew measure was useful.
\n\nThese yields just above 100 features.
\n\nModelling & CV Approach:\nI used GBM tools from LightGBM, XGBoost and Arboretum - reused most parameters from the original sharing with minor changes at learning rate. Time limit means I didn't have time to do any series tuning. I had also use Keras to build NN model, but the result wasn't good enough to be featured in the final blend
\n\nI tried to setup 3-fold OOF pipeline for this. I must have made some silly mistake on setting up user_id into folds or aggregating prediction values. My LB score from the OOF pipeline was way off the normal range, and I decided that I didn't have enough time to get it right (by this time 4 days left). So bye-bye OOF, bye-bye stacking and bye-bye K-fold CV. I then switched to simply validating on the same 25% of train orders, took the risk of overfitting - I was hoping the amount of data in this competition would help to address the risk of overfitting.
\n\nPost-processing - Optimising F1 Score:\nHere is the part that I spent most energy on getting to grip with. I had no idea so much academia efforts (like this & this) had been done behind such a seemingly simple metric! So I took some long time to read those papers, and tried to understand as much as possible the code share by Faron and Sh1ng. I then integrate the Faron's F1 optimisation routine in ways very similar to what is shared by Ivan
\n\nFinal Blending: \nA blending of ARB*0.7+XGB*0.15+LGB*0.15, this ration was checked and valided in the 25 validation data I left out. For each of these individual model I also used a lower learning rate in the last day of the competition to retrain - turned out this gave by far my best private LB results (+0.0002) better than anything else.
\n\nMy final thoughts on this competition: \n - It is fun to treat it as 1-week hackathon - I had to force myself not to dwell on small details, and move as quick as I can along the ML pipeline. \n - I also had too made some tough trade-off - i.e. giving up on fixing my OOF routine, and work on other aspects\n- By the end, I half-anticipated some strong shake-up. I notice that some of my models with similar local log-loss were giving some significant difference in the LB via MF1. So in the last two days, I decided to include some features that were not improving my validation logloss which I still believed to be reasonable features. Would be very interesting to see how other's logloss correlate with their LBs :) \n - I have benefited a lot from sharing in this competition, but I have also faced significant challenges by late sharing, as discussed in-depth in this post by CPMP here on the Quora competition. My view is simply no matter what, don't lose heart. In the long run, most of us would benefit from learning from such sharing.
\n\nThat is it! not a top solution, but quick & dirty to get some decent result, thanks again!
", "comments": [ { "id": 213751, "author_name": "CPMP", "author_url": "", "post_date": "2017-08-15T07:11:59.867000", "content": "Congrats on this hackathon, and on the result! A new Master is born ;)
", "votes": 3, "replies": [ { "id": 213807, "author_name": "Yifan Xie", "author_url": "", "post_date": "2017-08-15T09:22:32.333000", "content": "Thanks! wouldn't have done it without working with you and learning from you :)
", "votes": 3, "replies": [] }, { "id": 213821, "author_name": "CPMP", "author_url": "", "post_date": "2017-08-15T10:33:45.947000", "content": "I doubt it is because of me, but thanks!
", "votes": 2, "replies": [] } ] }, { "id": 213755, "author_name": "A_b", "author_url": "", "post_date": "2017-08-15T07:24:04.960000", "content": "Good job on synthesizing the key ideas and adding a useful personal touch on them! Did you have time to study the impact of the customer segmentation on your score? How many customer segments did you include in your model? -Cheers
", "votes": 1, "replies": [ { "id": 213809, "author_name": "Yifan Xie", "author_url": "", "post_date": "2017-08-15T09:30:52.470000", "content": "Just checked, the impact is significant - 0.0014 in both public and private LB. I used so two sets of customer segments. Based on the kernel by Andrea, I did KMean with K=3 and then K=4
\n\n\n", "votes": 1, "replies": [] }, { "id": 213985, "author_name": "A_b", "author_url": "", "post_date": "2017-08-15T16:36:32.123000", "content": "A^b wrote
\n \n\n\nGood job on synthesizing the key ideas and adding a useful personal touch on them! Did you have time to study the impact of the customer segmentation on your score? How many customer segments did you include in your model? -Cheers
\n
Nice, thanks for checking! I guess the customer segmentation can be made even finer with a further network analysis (Mohd Shadab Alam/LB=75 implemented a raw version), e.g. by grouping user_ids into clusters derived from betweenness centrality measures.
", "votes": 0, "replies": [] }, { "id": 214014, "author_name": "Yifan Xie", "author_url": "", "post_date": "2017-08-15T18:12:03.403000", "content": "yes, definitely we can improve further from what I have present above. Due to the time constraint, there was no time to fine tune on most aspect of my pipeline. We can definitely use more sophisticated approach to perform product and user segmentation, and maybe even product & user pair segmentation. Looking at the solutation sharing threads, there are several other things I really should have tried
\n\nIn fact, I just realised that the LGB model I was using had huge value for data point per leaf -> it was set to 255 (carried over from Sh1ng's sharing). I changed that to a more reasonable had a go this afternoon and it definitely improves the LGB model. This probably would also improve the final blending.
", "votes": 1, "replies": [] } ] }, { "id": 213680, "author_name": "Chia-Ta Tsai", "author_url": "", "post_date": "2017-08-15T03:55:18.777000", "content": "Congrats! It is really intense to do it as a week-long hackathon. I mainly spent the last three weeks on-and-off on it. For me also, this wouldn't have been possible without all the generous sharing and rich discussions. I appreciated that and thank every participant.
", "votes": 1, "replies": [] }, { "id": 213673, "author_name": "StuartLu", "author_url": "", "post_date": "2017-08-15T03:42:07.057000", "content": "congrats
", "votes": 1, "replies": [] }, { "id": 214309, "author_name": "miguel perez", "author_url": "", "post_date": "2017-08-16T13:55:39.550000", "content": "Congratulations Yifan Xie! Such a pragmatic an efficient approach deserved the reward!
\nThank you for sharing it :-)
Thanks Miguel, it was a mad-dash for silver to make master tier, and I am super glad it worked out. Still nowhere close to GMs like Kaza and Stanislav Semenov, so there is always waysssss to raise the limit :)
", "votes": 0, "replies": [] } ] }, { "id": 214120, "author_name": "germayne", "author_url": "", "post_date": "2017-08-16T02:24:32.420000", "content": "congrats on your master title :) remember you from Quora.
\n\ncouple of questions:
\n\nRegarding OOF predictions, I am using caret most of the time. I am guessing most of your part should be done in python. Just curious if there is a method for extracting OOFs using LGBM? Or do you do it manually? I.e split them and predict them one by one on each fold?
\n\nHope you can help :)
", "votes": 0, "replies": [ { "id": 214213, "author_name": "Yifan Xie", "author_url": "", "post_date": "2017-08-16T08:38:21.840000", "content": "I am afraid I am a LightGBM noobs, but a rough glance seems that this is work in progress - the best way I think is to follow up with this LGB github issue and see when they would implement it, or better still, do some contribution yourself :) You can also head over to the KaggleNoobs slack to ask Laurea - he is a contributor to the LGB github R package.
\n\nAside from this, I normally do CV routine manually, this give some extra options to tailor how the folds are separated, and also some feature engineering required in-fold creation.
", "votes": 0, "replies": [] }, { "id": 214337, "author_name": "Yifan Xie", "author_url": "", "post_date": "2017-08-16T15:24:46.263000", "content": "@germayne A bit more news since this morning, this is now officially in v3.0 milestone - did you do something to wake up the gods? ^ ^
", "votes": 1, "replies": [] }, { "id": 214733, "author_name": "germayne", "author_url": "", "post_date": "2017-08-18T03:08:01.677000", "content": "Sorry for the late reply. Was catching up with everything else after the time taken away in Kaggle :O
\n\nAnyway, that is good news for all of us! I know we can do OOF CV manually, but caret is honestly spoiling me.
", "votes": 0, "replies": [] } ] }, { "id": 214023, "author_name": "Jing Lu", "author_url": "", "post_date": "2017-08-15T18:42:43.550000", "content": "Congrats! Very fast implementation. Do you mind to share what tools you use for fast iteration of algorithms? A lot of my time is used to debug my code.. How do you keep track if you are implementing differenct branches of ideas?
", "votes": 0, "replies": [ { "id": 214063, "author_name": "Yifan Xie", "author_url": "", "post_date": "2017-08-15T21:27:12.297000", "content": "I am afraid there is no magic there :)
\n\nfor fast iteration, It is mainly LGB (0.05) and XGB 'Hist' mode (0.1) on higher learning rate - I switched to finer learning rate for my final solution. With this competition, XGB 'Hist' has issues handling this amount of data and crash once out of every few runs, so I relied more on LGB. If I take part in a competition for a longer period of time, sometimes I also use BayesianOptimisation to generate a list of different parameter combination, I can then from this list of combination to pick and choose depending on if I want to go fast or accurate.
\n\nIn terms of debugging or minimising human coding error, I faced the same challenge, and as I mentioned decided it wasn't worth taking to time to debug my k-fold/oof routine. Generally speaking, I rely on functions/routine that I built up from previous competitions - this is very much still an ongoing progress. but if I get to re-use a function that I previously built, I have better confidence that it would work.
\n\nI have only been seriously competing for about 2 years, and only starting to build up my own collection of ML related functions for about 1 years. I hope this collection will grow to be more completed as I take on more Kaggle challenges.
\n\nIn term of implementing different branch of ideas making sure on traceability, I am old fashioned, I use a google spreadsheet to keep my a record on key information like algo., key parameter, early stopping, features, validation methods, val score, PL for each key runs. Additionally, I make backup copies of codes/notebooks.
", "votes": 2, "replies": [] }, { "id": 214065, "author_name": "Jing Lu", "author_url": "", "post_date": "2017-08-15T21:51:15.587000", "content": "Thanks for the explanation! That's very helpful! I am also working on building a pipeline that can automate at least some of the processes. Rewrite code can really take time especially when the time is limited.
\n\nI totally agree we need to keep several versions of parameters for lightgbm: one for testing features (relatively fast), one for final tuning (like your mentioned BayesianOptimisation).
\n\nWe may also need to keep several train test splits (one n-fold, one single hold-out). The single hold-out can be used for testing features or other experiments. n-fold can be used for ensemble.
\n\nI also try to cache intermediate steps, so I don't need to rerun every time.
\n\nGoogle sheet is a pretty nice idea to keep track of what have been done. I just begin to use it.
", "votes": 1, "replies": [] } ] } ], "comment_count": 18, "messages": [ { "id": 213646, "postDate": "2017-08-15T02:51:22.333Z", "content": "Hello all, firstly congratulation to the winners and top performers, your effort and sharing are great and I have learnt a lot in a very short space of time thanks to you. Also, I want to say well done for people who had been working on this from the beginning, in the end, tenacity will prevail in the form of knowledge gained, medals and good ranking.
\n\nBasically, I have treated this competition as a week-long hackathon and spent one intensive week (50h+) on it. I am very lucky to be ranked within top 100. This wouldn't have been possible without all the sharing. So big thanks from me. Here is the overall sketch of my approach to achieving top 100 ranking in this short space of time:
\n\nShared Features:
\n\nMy own features:\n 6. Lag 1 features - User total products, and user reorder sum in n-1 order (last order in prior)\n 7. Lag difference - User total products difference between orders: n-1 & n-2, n-2 & n-3, n-3 & n-4\n 8. Skew statistic on user product state: so let' say a user had 5 orders, for a given product, the product purchase states in these orders were - for argument sake - [0 0 1 1 1 ]. so I capture this series and calculate the Skew and IPR measure - but only the skew measure was useful.
\n\nThese yields just above 100 features.
\n\nModelling & CV Approach:\nI used GBM tools from LightGBM, XGBoost and Arboretum - reused most parameters from the original sharing with minor changes at learning rate. Time limit means I didn't have time to do any series tuning. I had also use Keras to build NN model, but the result wasn't good enough to be featured in the final blend
\n\nI tried to setup 3-fold OOF pipeline for this. I must have made some silly mistake on setting up user_id into folds or aggregating prediction values. My LB score from the OOF pipeline was way off the normal range, and I decided that I didn't have enough time to get it right (by this time 4 days left). So bye-bye OOF, bye-bye stacking and bye-bye K-fold CV. I then switched to simply validating on the same 25% of train orders, took the risk of overfitting - I was hoping the amount of data in this competition would help to address the risk of overfitting.
\n\nPost-processing - Optimising F1 Score:\nHere is the part that I spent most energy on getting to grip with. I had no idea so much academia efforts (like this & this) had been done behind such a seemingly simple metric! So I took some long time to read those papers, and tried to understand as much as possible the code share by Faron and Sh1ng. I then integrate the Faron's F1 optimisation routine in ways very similar to what is shared by Ivan
\n\nFinal Blending: \nA blending of ARB*0.7+XGB*0.15+LGB*0.15, this ration was checked and valided in the 25 validation data I left out. For each of these individual model I also used a lower learning rate in the last day of the competition to retrain - turned out this gave by far my best private LB results (+0.0002) better than anything else.
\n\nMy final thoughts on this competition: \n - It is fun to treat it as 1-week hackathon - I had to force myself not to dwell on small details, and move as quick as I can along the ML pipeline. \n - I also had too made some tough trade-off - i.e. giving up on fixing my OOF routine, and work on other aspects\n- By the end, I half-anticipated some strong shake-up. I notice that some of my models with similar local log-loss were giving some significant difference in the LB via MF1. So in the last two days, I decided to include some features that were not improving my validation logloss which I still believed to be reasonable features. Would be very interesting to see how other's logloss correlate with their LBs :) \n - I have benefited a lot from sharing in this competition, but I have also faced significant challenges by late sharing, as discussed in-depth in this post by CPMP here on the Quora competition. My view is simply no matter what, don't lose heart. In the long run, most of us would benefit from learning from such sharing.
\n\nThat is it! not a top solution, but quick & dirty to get some decent result, thanks again!
", "rawMarkdown": "Hello all, firstly congratulation to the winners and top performers, your effort and sharing are great and I have learnt a lot in a very short space of time thanks to you. Also, I want to say well done for people who had been working on this from the beginning, in the end, tenacity will prevail in the form of knowledge gained, medals and good ranking.\n\nBasically, I have treated this competition as a week-long hackathon and spent one intensive week (50h+) on it. I am very lucky to be ranked within top 100. This wouldn't have been possible without all the sharing. So big thanks from me. Here is the overall sketch of my approach to achieving top 100 ranking in this short space of time:\n\nShared Features:\n\n 1. Features from LB0.3805009 baseline by 🐳鲲(China) \n 2. Features from LB0.4029970 baseline [Sh1ng][1]\n 3. Product W2V PCA features by [Omar Essam][2], I used the top 5 components.\n 4. Customer Segment features by [Andrea][3]\n 5. Order streak by [Faron][4]\n \nMy own features:\n 6. Lag 1 features - User total products, and user reorder sum in n-1 order (last order in prior)\n 7. Lag difference - User total products difference between orders: n-1 & n-2, n-2 & n-3, n-3 & n-4\n 8. Skew statistic on user product state: so let' say a user had 5 orders, for a given product, the product purchase states in these orders were - for argument sake - [0 0 1 1 1 ]. so I capture this series and calculate the Skew and IPR measure - but only the skew measure was useful.\n\nThese yields just above 100 features. \n\nModelling & CV Approach:\nI used GBM tools from LightGBM, XGBoost and Arboretum - reused most parameters from the original sharing with minor changes at learning rate. Time limit means I didn't have time to do any series tuning. I had also use Keras to build NN model, but the result wasn't good enough to be featured in the final blend \n\nI tried to setup 3-fold OOF pipeline for this. I must have made some silly mistake on setting up user_id into folds or aggregating prediction values. My LB score from the OOF pipeline was way off the normal range, and I decided that I didn't have enough time to get it right (by this time 4 days left). So bye-bye OOF, bye-bye stacking and bye-bye K-fold CV. I then switched to simply validating on the same 25% of train orders, took the risk of overfitting - I was hoping the amount of data in this competition would help to address the risk of overfitting. \n\nPost-processing - Optimising F1 Score:\nHere is the part that I spent most energy on getting to grip with. I had no idea so much academia efforts (like [this][5] & [this][6]) had been done behind such a seemingly simple metric! So I took some long time to read those papers, and tried to understand as much as possible the code share by [Faron][7] and [Sh1ng][8]. I then integrate the Faron's F1 optimisation routine in ways very similar to what is shared by [Ivan][9]\n\nFinal Blending: \nA blending of ARB*0.7+XGB*0.15+LGB*0.15, this ration was checked and valided in the 25 validation data I left out. For each of these individual model I also used a lower learning rate in the last day of the competition to retrain - turned out this gave by far my best private LB results (+0.0002) better than anything else.\n\n\nMy final thoughts on this competition: \n - It is fun to treat it as 1-week hackathon - I had to force myself not to dwell on small details, and move as quick as I can along the ML pipeline. \n - I also had too made some tough trade-off - i.e. giving up on fixing my OOF routine, and work on other aspects\n- By the end, I half-anticipated some strong shake-up. I notice that some of my models with similar local log-loss were giving some significant difference in the LB via MF1. So in the last two days, I decided to include some features that were not improving my validation logloss which I still believed to be reasonable features. Would be very interesting to see how other's logloss correlate with their LBs :) \n - I have benefited a lot from sharing in this competition, but I have also faced significant challenges by late sharing, as discussed in-depth in this post by [CPMP][10] here on the Quora competition. My view is simply no matter what, don't lose heart. In the long run, most of us would benefit from learning from such sharing. \n\nThat is it! not a top solution, but quick & dirty to get some decent result, thanks again!\n\n [1]: https://www.kaggle.com/c/instacart-market-basket-analysis/discussion/37697#latest-213541\n [2]: https://www.kaggle.com/omarito/word2vec-for-products-analysis-0-01-lb\n [3]: https://www.kaggle.com/asindico/customer-segments\n [4]: https://www.kaggle.com/mmueller/order-streaks-feature\n [5]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4442797/\n [6]: https://www.comp.nus.edu.sg/~leews/publications/fscore.pdf\n [7]: https://www.kaggle.com/mmueller/f1-score-expectation-maximization-in-o-n\n [8]: https://github.com/sh1ng/imba\n [9]: https://www.kaggle.com/c/instacart-market-basket-analysis/discussion/37874\n [10]: https://www.kaggle.com/c/quora-question-pairs/discussion/33801", "votes": 28 }, { "id": 213751, "postDate": "2017-08-15T07:11:59.867Z", "content": "Congrats on this hackathon, and on the result! A new Master is born ;)
", "rawMarkdown": "Congrats on this hackathon, and on the result! A new Master is born ;)", "votes": 3, "replies": [ { "id": 213807, "postDate": "2017-08-15T09:22:32.333Z", "content": "Thanks! wouldn't have done it without working with you and learning from you :)
", "rawMarkdown": "Thanks! wouldn't have done it without working with you and learning from you :) ", "votes": 3 }, { "id": 213821, "postDate": "2017-08-15T10:33:45.947Z", "content": "I doubt it is because of me, but thanks!
", "rawMarkdown": "I doubt it is because of me, but thanks!", "votes": 2 } ] }, { "id": 213755, "postDate": "2017-08-15T07:24:04.960Z", "content": "Good job on synthesizing the key ideas and adding a useful personal touch on them! Did you have time to study the impact of the customer segmentation on your score? How many customer segments did you include in your model? -Cheers
", "rawMarkdown": "Good job on synthesizing the key ideas and adding a useful personal touch on them! Did you have time to study the impact of the customer segmentation on your score? How many customer segments did you include in your model? -Cheers", "votes": 1, "replies": [ { "id": 213809, "postDate": "2017-08-15T09:30:52.470Z", "content": "Just checked, the impact is significant - 0.0014 in both public and private LB. I used so two sets of customer segments. Based on the kernel by Andrea, I did KMean with K=3 and then K=4
\n\n\n", "rawMarkdown": "Just checked, the impact is significant - 0.0014 in both public and private LB. I used so two sets of customer segments. Based on the kernel by Andrea, I did KMean with K=3 and then K=4\n\n> **A^b wrote**\n> \n> > Good job on synthesizing the key ideas and adding a useful personal touch on them! Did you have time to study the impact of the customer segmentation on your score? How many customer segments did you include in your model? -Cheers", "votes": 1 }, { "id": 213985, "postDate": "2017-08-15T16:36:32.123Z", "content": "A^b wrote
\n \n\n\nGood job on synthesizing the key ideas and adding a useful personal touch on them! Did you have time to study the impact of the customer segmentation on your score? How many customer segments did you include in your model? -Cheers
\n
Nice, thanks for checking! I guess the customer segmentation can be made even finer with a further network analysis (Mohd Shadab Alam/LB=75 implemented a raw version), e.g. by grouping user_ids into clusters derived from betweenness centrality measures.
", "rawMarkdown": "Nice, thanks for checking! I guess the customer segmentation can be made even finer with a further network analysis (Mohd Shadab Alam/LB=75 implemented a raw version), e.g. by grouping user_ids into clusters derived from betweenness centrality measures. " }, { "id": 214014, "postDate": "2017-08-15T18:12:03.403Z", "content": "yes, definitely we can improve further from what I have present above. Due to the time constraint, there was no time to fine tune on most aspect of my pipeline. We can definitely use more sophisticated approach to perform product and user segmentation, and maybe even product & user pair segmentation. Looking at the solutation sharing threads, there are several other things I really should have tried
\n\nIn fact, I just realised that the LGB model I was using had huge value for data point per leaf -> it was set to 255 (carried over from Sh1ng's sharing). I changed that to a more reasonable had a go this afternoon and it definitely improves the LGB model. This probably would also improve the final blending.
", "rawMarkdown": "yes, definitely we can improve further from what I have present above. Due to the time constraint, there was no time to fine tune on most aspect of my pipeline. We can definitely use more sophisticated approach to perform product and user segmentation, and maybe even product & user pair segmentation. Looking at the solutation sharing threads, there are several other things I really should have tried \n\nIn fact, I just realised that the LGB model I was using had huge value for data point per leaf -> it was set to 255 (carried over from Sh1ng's sharing). I changed that to a more reasonable had a go this afternoon and it definitely improves the LGB model. This probably would also improve the final blending. \n\n ", "votes": 1 } ] }, { "id": 213680, "postDate": "2017-08-15T03:55:18.777Z", "content": "Congrats! It is really intense to do it as a week-long hackathon. I mainly spent the last three weeks on-and-off on it. For me also, this wouldn't have been possible without all the generous sharing and rich discussions. I appreciated that and thank every participant.
", "rawMarkdown": "Congrats! It is really intense to do it as a week-long hackathon. I mainly spent the last three weeks on-and-off on it. For me also, this wouldn't have been possible without all the generous sharing and rich discussions. I appreciated that and thank every participant. ", "votes": 1 }, { "id": 213673, "postDate": "2017-08-15T03:42:07.057Z", "content": "congrats
", "rawMarkdown": "congrats", "votes": 1 }, { "id": 214309, "postDate": "2017-08-16T13:55:39.550Z", "content": "Congratulations Yifan Xie! Such a pragmatic an efficient approach deserved the reward!
\nThank you for sharing it :-)
Thanks Miguel, it was a mad-dash for silver to make master tier, and I am super glad it worked out. Still nowhere close to GMs like Kaza and Stanislav Semenov, so there is always waysssss to raise the limit :)
", "rawMarkdown": "Thanks Miguel, it was a mad-dash for silver to make master tier, and I am super glad it worked out. Still nowhere close to GMs like Kaza and Stanislav Semenov, so there is always waysssss to raise the limit :)\n" } ] }, { "id": 214120, "postDate": "2017-08-16T02:24:32.420Z", "content": "congrats on your master title :) remember you from Quora.
\n\ncouple of questions:
\n\nRegarding OOF predictions, I am using caret most of the time. I am guessing most of your part should be done in python. Just curious if there is a method for extracting OOFs using LGBM? Or do you do it manually? I.e split them and predict them one by one on each fold?
\n\nHope you can help :)
", "rawMarkdown": "congrats on your master title :) remember you from Quora. \n\ncouple of questions: \n\nRegarding OOF predictions, I am using caret most of the time. I am guessing most of your part should be done in python. Just curious if there is a method for extracting OOFs using LGBM? Or do you do it manually? I.e split them and predict them one by one on each fold? \n\n\nHope you can help :)", "replies": [ { "id": 214213, "postDate": "2017-08-16T08:38:21.840Z", "content": "I am afraid I am a LightGBM noobs, but a rough glance seems that this is work in progress - the best way I think is to follow up with this LGB github issue and see when they would implement it, or better still, do some contribution yourself :) You can also head over to the KaggleNoobs slack to ask Laurea - he is a contributor to the LGB github R package.
\n\nAside from this, I normally do CV routine manually, this give some extra options to tailor how the folds are separated, and also some feature engineering required in-fold creation.
", "rawMarkdown": "I am afraid I am a LightGBM noobs, but a rough glance seems that this is [work in progress][1] - the best way I think is to follow up with this LGB github issue and see when they would implement it, or better still, do some contribution yourself :) You can also head over to the [KaggleNoobs][2] slack to ask Laurea - he is a contributor to the LGB github R package. \n\nAside from this, I normally do CV routine manually, this give some extra options to tailor how the folds are separated, and also some feature engineering required in-fold creation. \n\n\n [1]: https://github.com/Microsoft/LightGBM/issues/283\n [2]: http://kagglenoobs.herokuapp.com/" }, { "id": 214337, "postDate": "2017-08-16T15:24:46.263Z", "content": "@germayne A bit more news since this morning, this is now officially in v3.0 milestone - did you do something to wake up the gods? ^ ^
", "rawMarkdown": "@germayne A bit more news since this morning, this is now officially in [v3.0][1] milestone - did you do something to wake up the gods? ^ ^ \n\n [1]: https://github.com/Microsoft/LightGBM/milestone/3", "votes": 1 }, { "id": 214733, "postDate": "2017-08-18T03:08:01.677Z", "content": "Sorry for the late reply. Was catching up with everything else after the time taken away in Kaggle :O
\n\nAnyway, that is good news for all of us! I know we can do OOF CV manually, but caret is honestly spoiling me.
", "rawMarkdown": "Sorry for the late reply. Was catching up with everything else after the time taken away in Kaggle :O \n\nAnyway, that is good news for all of us! I know we can do OOF CV manually, but caret is honestly spoiling me.\n\n" } ] }, { "id": 214023, "postDate": "2017-08-15T18:42:43.550Z", "content": "Congrats! Very fast implementation. Do you mind to share what tools you use for fast iteration of algorithms? A lot of my time is used to debug my code.. How do you keep track if you are implementing differenct branches of ideas?
", "rawMarkdown": "Congrats! Very fast implementation. Do you mind to share what tools you use for fast iteration of algorithms? A lot of my time is used to debug my code.. How do you keep track if you are implementing differenct branches of ideas?", "replies": [ { "id": 214063, "postDate": "2017-08-15T21:27:12.297Z", "content": "I am afraid there is no magic there :)
\n\nfor fast iteration, It is mainly LGB (0.05) and XGB 'Hist' mode (0.1) on higher learning rate - I switched to finer learning rate for my final solution. With this competition, XGB 'Hist' has issues handling this amount of data and crash once out of every few runs, so I relied more on LGB. If I take part in a competition for a longer period of time, sometimes I also use BayesianOptimisation to generate a list of different parameter combination, I can then from this list of combination to pick and choose depending on if I want to go fast or accurate.
\n\nIn terms of debugging or minimising human coding error, I faced the same challenge, and as I mentioned decided it wasn't worth taking to time to debug my k-fold/oof routine. Generally speaking, I rely on functions/routine that I built up from previous competitions - this is very much still an ongoing progress. but if I get to re-use a function that I previously built, I have better confidence that it would work.
\n\nI have only been seriously competing for about 2 years, and only starting to build up my own collection of ML related functions for about 1 years. I hope this collection will grow to be more completed as I take on more Kaggle challenges.
\n\nIn term of implementing different branch of ideas making sure on traceability, I am old fashioned, I use a google spreadsheet to keep my a record on key information like algo., key parameter, early stopping, features, validation methods, val score, PL for each key runs. Additionally, I make backup copies of codes/notebooks.
", "rawMarkdown": "I am afraid there is no magic there :)\n\nfor fast iteration, It is mainly LGB (0.05) and XGB 'Hist' mode (0.1) on higher learning rate - I switched to finer learning rate for my final solution. With this competition, XGB 'Hist' has issues handling this amount of data and crash once out of every few runs, so I relied more on LGB. If I take part in a competition for a longer period of time, sometimes I also use BayesianOptimisation to generate a list of different parameter combination, I can then from this list of combination to pick and choose depending on if I want to go fast or accurate. \n\nIn terms of debugging or minimising human coding error, I faced the same challenge, and as I mentioned decided it wasn't worth taking to time to debug my k-fold/oof routine. Generally speaking, I rely on functions/routine that I built up from previous competitions - this is very much still an ongoing progress. but if I get to re-use a function that I previously built, I have better confidence that it would work. \n\nI have only been seriously competing for about 2 years, and only starting to build up my own collection of ML related functions for about 1 years. I hope this collection will grow to be more completed as I take on more Kaggle challenges. \n\nIn term of implementing different branch of ideas making sure on traceability, I am old fashioned, I use a google spreadsheet to keep my a record on key information like algo., key parameter, early stopping, features, validation methods, val score, PL for each key runs. Additionally, I make backup copies of codes/notebooks. \n \n\n", "votes": 2 }, { "id": 214065, "postDate": "2017-08-15T21:51:15.587Z", "content": "Thanks for the explanation! That's very helpful! I am also working on building a pipeline that can automate at least some of the processes. Rewrite code can really take time especially when the time is limited.
\n\nI totally agree we need to keep several versions of parameters for lightgbm: one for testing features (relatively fast), one for final tuning (like your mentioned BayesianOptimisation).
\n\nWe may also need to keep several train test splits (one n-fold, one single hold-out). The single hold-out can be used for testing features or other experiments. n-fold can be used for ensemble.
\n\nI also try to cache intermediate steps, so I don't need to rerun every time.
\n\nGoogle sheet is a pretty nice idea to keep track of what have been done. I just begin to use it.
", "rawMarkdown": "Thanks for the explanation! That's very helpful! I am also working on building a pipeline that can automate at least some of the processes. Rewrite code can really take time especially when the time is limited. \n\nI totally agree we need to keep several versions of parameters for lightgbm: one for testing features (relatively fast), one for final tuning (like your mentioned BayesianOptimisation).\n\nWe may also need to keep several train test splits (one n-fold, one single hold-out). The single hold-out can be used for testing features or other experiments. n-fold can be used for ensemble.\n\nI also try to cache intermediate steps, so I don't need to rerun every time. \n\nGoogle sheet is a pretty nice idea to keep track of what have been done. I just begin to use it.", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T02:01:40Z", "competition_id": 6644, "competition_slug": "basket-analysis", "team_id": 805758, "solution_url": "/competitions/basket-analysis/writeups/one-week-to-silver-lb-86-approach" }