Hi guys, i work in my own code base (custom written frame work, not R or python) and i just wanted to get an idea of what kind of run time some of the other frameworks were taking on this data.
\n\nI ask just cause at least in my case if I want to gbm and have it finish in a relatively short time (say under 30 minutes) I need to take some fraction (like 1-5%) of the data to work with.
\n\nThat being said, I finially got something I thought might produce a competitive result. so i went to run it and wow ... takes along time.
\n\nThat is to say, i didnt think it would be much of a problem to do the processing of the final test data when the time came but dang.... it's been running 22 hours and shows no significant signs of being done (though those cores are all pegged and when i break the running code to see how its doing it IS moving forward).
\n\nbefore now the only submission i've made was a simple format check, so it didnt do any real processing. anyone else seeing the runtime on this big data be that bad? are you avoiding certain processing mechanisms because of that?
\n\ntake all the time you want to answer apparently i've got hours :) possibly days.
", "messages": [ { "id": 125283, "postDate": "2016-06-28T12:18:53.117Z", "content": "I am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.
", "rawMarkdown": "I am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.", "votes": 1 }, { "id": 125194, "postDate": "2016-06-27T13:52:25.413Z", "content": "[quote=ctd11;125190]
\n\nHow are you doing the feature engineering in C++, are you using Rcpp or Vowpal Wabbit?
\n\n[/quote]
\n\nNo, I just write everything on my own, where I heavily use unordered_map<> and some special designed linked list. But it works great (far more efficient than R or Python), because I can generate features on my mac pro (only 4GB mem). However, it is uneasy for me to perform explorative data analysis using C++. So I also dump all data into a database and build indexes for that purpose.
\n\nHope my reply can help someone who don't have large RAM machine. Undoubtably, one need to rent a large machine (Google cloud or Amazon EC2) or use a computer cluster if he/she wants to generate the submission or perform parameter tunings at the full size of data. Given it is at very early stage of competition, it is not hard to move into top 20. This is my first time to participate in Kaggle and I really learnt a lot!!!
", "rawMarkdown": "[quote=ctd11;125190]\r\n\r\nHow are you doing the feature engineering in C++, are you using Rcpp or Vowpal Wabbit?\r\n\r\n[/quote]\r\n\r\nNo, I just write everything on my own, where I heavily use unordered_map<> and some special designed linked list. But it works great (far more efficient than R or Python), because I can generate features on my mac pro (only 4GB mem). However, it is uneasy for me to perform explorative data analysis using C++. So I also dump all data into a database and build indexes for that purpose. \r\n\r\nHope my reply can help someone who don't have large RAM machine. Undoubtably, one need to rent a large machine (Google cloud or Amazon EC2) or use a computer cluster if he/she wants to generate the submission or perform parameter tunings at the full size of data. Given it is at very early stage of competition, it is not hard to move into top 20. This is my first time to participate in Kaggle and I really learnt a lot!!!\r\n\r\n\r\n\r\n\r\n\r\n", "votes": 1 }, { "id": 124862, "postDate": "2016-06-22T20:02:25.183Z", "content": "Just wanted to say; I just jumped to #11 (and falling quickly) so my advice isn't complete crap.
\n\nBut there's the best advice I can give you; focus on having fun and learning, that's what I do. Winning would be nice, but only one person will eventually win. But we can all learn! And I've learned tons from every competition I've done.
\n\nAnd from the sound of it - you're on top of that at so keep going strong!
\n\n/m
", "rawMarkdown": "Just wanted to say; I just jumped to #11 (and falling quickly) so my advice isn't complete crap.\r\n\r\nBut there's the best advice I can give you; focus on having fun and learning, that's what I do. Winning would be nice, but only one person will eventually win. But we can all learn! And I've learned tons from every competition I've done.\r\n\r\nAnd from the sound of it - you're on top of that at so keep going strong!\r\n\r\n/m", "votes": 1 }, { "id": 124963, "postDate": "2016-06-23T23:08:50.840Z", "content": "I currently rank in top20.
\n\nAfter trying a few directions, I decided to completely use C++ for feature engineering and combine with some python script for learning and generating submissions. No hard-core computations are done in python or R. The speedups and memory efficiency of writing in C++ improves the solution iterates.
\n\nNow it takes about 5-10 minutes for generating all features on my laptop, and several minutes to train a predictor (on a rent 24-core computing node with >64GB), and almost no time to create a submission. My current solution is a leverage of my own laptop and a rent computing node.
", "rawMarkdown": "I currently rank in top20.\r\n\r\nAfter trying a few directions, I decided to completely use C++ for feature engineering and combine with some python script for learning and generating submissions. No hard-core computations are done in python or R. The speedups and memory efficiency of writing in C++ improves the solution iterates. \r\n\r\nNow it takes about 5-10 minutes for generating all features on my laptop, and several minutes to train a predictor (on a rent 24-core computing node with >64GB), and almost no time to create a submission. My current solution is a leverage of my own laptop and a rent computing node. ", "votes": 2 }, { "id": 124853, "postDate": "2016-06-22T19:19:18.620Z", "content": "Hi, I'm currently at #63 (and quickly falling) using all scratch built software, and my #63 submission takes about 20 minutes to generate. No-where near 22 hours. Though I've been optimizing code for some time.
\n\nThe first thing I did was to transform the CSV files to binary files, so I can load them in minutes. That's something you might want to look into. Though that won't shave anything significant off 22hours.
\n\nMy latest features generating run took 140minutes (2h 20minutes) on my desktop machine (32GB, 6 cores, 12 hyperthreads). The very first feature of that run ("Adding Demanda_uni_equil by BigDog 659.517s") takes 11 minutes and would, on its own, place at #63. I'm hoping to use that as one of many features in a ML solution. If my score jump, this is what I did. If my score doesn't jump - this is still what I did ;)
\n\nSo; 22+ hours is a very long time indeed. That doesn't mean it'll be bad, but chances are that you'll wish to tweak it. And when you do, you're in for another 22+ hours. So if you can iterate faster, you can develop your code faster. Which is nice. But perhaps more importantly, you can hone your intuition faster. And a good intuition is very important to understanding a problem. A 22+ hour turnaround on an idea will make you forget the next idea you had when you picked that one you moved forward with.
\n\nI iterate in minutes, meaning I change something, generate a new estimated solution and find out if it worked in minutes. It's not a full answer, and it doesn't guarantee me success, but it helps me understand the problem. My best ever placing is #6, so I might be full of shit; but so might anyone. Iterating fast is important, and if you can do that, you'll learn faster.
\n\nAlso 22hours isn't a problem, it's an opportunity. Do you do any sequential searches? Replace all of them by dictionaries (hash tables). There are other tips, but I'm betting that's the one that's killing you. Are you reading from disk more than to move information to RAM? That'll kill your performance, even if you're using a cluster of SSD drives.
\n\nPerformance optimization is fun. If you wish to figure out how to speed up your code, you can publish it/parts of it here on the forum asking for tips for making it run faster. You'll give away a few of your ideas, but you'll get far more in return. Just an idea.
\n\n/m
", "rawMarkdown": "Hi, I'm currently at #63 (and quickly falling) using all scratch built software, and my #63 submission takes about 20 minutes to generate. No-where near 22 hours. Though I've been optimizing code for some time. \r\n\r\nThe first thing I did was to transform the CSV files to binary files, so I can load them in minutes. That's something you might want to look into. Though that won't shave anything significant off 22hours.\r\n\r\nMy latest features generating run took 140minutes (2h 20minutes) on my desktop machine (32GB, 6 cores, 12 hyperthreads). The very first feature of that run (\"Adding Demanda_uni_equil by BigDog 659.517s\") takes 11 minutes and would, on its own, place at #63. I'm hoping to use that as one of many features in a ML solution. If my score jump, this is what I did. If my score doesn't jump - this is still what I did ;)\r\n\r\nThe real problem\r\n----------------\r\n\r\nSo; 22+ hours is a very long time indeed. That doesn't mean it'll be bad, but chances are that you'll wish to tweak it. And when you do, you're in for another 22+ hours. So if you can iterate faster, you can develop your code faster. Which is nice. But perhaps more importantly, you can hone your intuition faster. And a good intuition is very important to understanding a problem. A 22+ hour turnaround on an idea will make you forget the next idea you had when you picked that one you moved forward with.\r\n\r\nI iterate in minutes, meaning I change something, generate a new estimated solution and find out if it worked in minutes. It's not a full answer, and it doesn't guarantee me success, but it helps me understand the problem. My best ever placing is #6, so I might be full of shit; but so might anyone. Iterating fast is important, and if you can do that, you'll learn faster.\r\n\r\nAlso 22hours isn't a problem, it's an opportunity. Do you do *any* sequential searches? Replace *all* of them by dictionaries (hash tables). There are other tips, but I'm betting that's the one that's killing you. Are you reading from disk more than to move information to RAM? That'll kill your performance, even if you're using a cluster of SSD drives. \r\n\r\nPerformance optimization is *fun*. If you wish to figure out how to speed up your code, you can publish it/parts of it here on the forum asking for tips for making it run faster. You'll give away a few of your ideas, but you'll get far more in return. Just an idea. \r\n\r\n/m", "votes": 2 }, { "id": 125310, "postDate": "2016-06-28T14:41:05.027Z", "content": "[quote=Hartmut Ott;125283]
\n\nI am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.
\n\n[/quote]
\n\nOoops, you actually remind me that there exists an alternative setup for validation. That could be important for model evaluation.
\n\nJianbo
", "rawMarkdown": "[quote=Hartmut Ott;125283]\r\n\r\nI am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.\r\n\r\n[/quote]\r\n\r\nOoops, you actually remind me that there exists an alternative setup for validation. That could be important for model evaluation. \r\n\r\nJianbo" }, { "id": 125296, "postDate": "2016-06-28T13:24:53.640Z", "content": "[quote=Hartmut Ott;125283]
\n\nI am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.
\n\n[/quote]
\n\nCould you share a bit more about how you do validation? Is it close to what you get on LB? Thanks!
", "rawMarkdown": "[quote=Hartmut Ott;125283]\r\n\r\nI am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.\r\n\r\n[/quote]\r\n\r\nCould you share a bit more about how you do validation? Is it close to what you get on LB? Thanks!" }, { "id": 125190, "postDate": "2016-06-27T13:11:13.463Z", "content": "How are you doing the feature engineering in C++, are you using Rcpp or Vowpal Wabbit?
", "rawMarkdown": "How are you doing the feature engineering in C++, are you using Rcpp or Vowpal Wabbit?" }, { "id": 124882, "postDate": "2016-06-23T04:25:11.777Z", "content": "That is impressive! One hour including loading from raw text files, means I have work to do!
", "rawMarkdown": "That is impressive! One hour including loading from raw text files, means I have work to do!\r\n\r\n" }, { "id": 124870, "postDate": "2016-06-22T22:31:02.757Z", "content": "I'm ranked 7, I use data under week 8 to create my features and data over week 8 to learn using the past.\nIt takes 1 hour to generate results and I only have 8 GB of RAM and 4 i5 cores on my laptop ^^
\n\nI didn't spent a lot of time on this competition yet so I guess there is a lot of things you can do even if you don't have a big computer
", "rawMarkdown": "I'm ranked 7, I use data under week 8 to create my features and data over week 8 to learn using the past.\r\nIt takes 1 hour to generate results and I only have 8 GB of RAM and 4 i5 cores on my laptop ^^\r\n\r\nI didn't spent a lot of time on this competition yet so I guess there is a lot of things you can do even if you don't have a big computer" }, { "id": 124860, "postDate": "2016-06-22T19:56:51.777Z", "content": "I appreciate what you're saying. believe i do enjoy myself some optimization for performance. I had to do some major recojiggering (read: memory optimization) to just get this data set to load and process in memory.
\n\nbased on what you've said, it sounds like this is just a case of the nature of the algorithm i've choosen and the settings i'm going with. Other people are either tweaking theirs differently to avoid performance problems or choosing an entirely different algoritm entirely. I know I could probably make my agorithm run about 5-10 times faster (for example) if I rewrote it not to do correlation of feature to target scores values at each level of my trees, but that really helps me select great features (and throw away the noise). random feature selection works okay and is way faster but... well we all want to win.
\n\nMaybe i'll try doing the correlation check every other tree node... give up a little accuracy to double the speed. (food for thought) heck it might even help in the long run through some hidden nature in the data (though I wont hold my breath on that). oh well thanks for the response!
", "rawMarkdown": "I appreciate what you're saying. believe i do enjoy myself some optimization for performance. I had to do some major recojiggering (read: memory optimization) to just get this data set to load and process in memory. \r\n\r\nbased on what you've said, it sounds like this is just a case of the nature of the algorithm i've choosen and the settings i'm going with. Other people are either tweaking theirs differently to avoid performance problems or choosing an entirely different algoritm entirely. I know I could probably make my agorithm run about 5-10 times faster (for example) if I rewrote it not to do correlation of feature to target scores values at each level of my trees, but that really helps me select great features (and throw away the noise). random feature selection works okay and is way faster but... well we all want to win.\r\n\r\nMaybe i'll try doing the correlation check every other tree node... give up a little accuracy to double the speed. (food for thought) heck it might even help in the long run through some hidden nature in the data (though I wont hold my breath on that). oh well thanks for the response!\r\n" }, { "id": 124844, "postDate": "2016-06-22T18:35:56.877Z", "content": "Hi guys, i work in my own code base (custom written frame work, not R or python) and i just wanted to get an idea of what kind of run time some of the other frameworks were taking on this data.
\n\nI ask just cause at least in my case if I want to gbm and have it finish in a relatively short time (say under 30 minutes) I need to take some fraction (like 1-5%) of the data to work with.
\n\nThat being said, I finially got something I thought might produce a competitive result. so i went to run it and wow ... takes along time.
\n\nThat is to say, i didnt think it would be much of a problem to do the processing of the final test data when the time came but dang.... it's been running 22 hours and shows no significant signs of being done (though those cores are all pegged and when i break the running code to see how its doing it IS moving forward).
\n\nbefore now the only submission i've made was a simple format check, so it didnt do any real processing. anyone else seeing the runtime on this big data be that bad? are you avoiding certain processing mechanisms because of that?
\n\ntake all the time you want to answer apparently i've got hours :) possibly days.
", "rawMarkdown": "Hi guys, i work in my own code base (custom written frame work, not R or python) and i just wanted to get an idea of what kind of run time some of the other frameworks were taking on this data. \r\n\r\nI ask just cause at least in my case if I want to gbm and have it finish in a relatively short time (say under 30 minutes) I need to take some fraction (like 1-5%) of the data to work with. \r\n\r\nThat being said, I finially got something I thought might produce a competitive result. so i went to run it and wow ... takes along time.\r\n\r\nThat is to say, i didnt think it would be much of a problem to do the processing of the final test data when the time came but dang.... it's been running 22 hours and shows no significant signs of being done (though those cores are all pegged and when i break the running code to see how its doing it IS moving forward).\r\n\r\n before now the only submission i've made was a simple format check, so it didnt do any real processing. anyone else seeing the runtime on this big data be that bad? are you avoiding certain processing mechanisms because of that?\r\n\r\n take all the time you want to answer apparently i've got hours :) possibly days." }, { "id": 124942, "postDate": "2016-06-23T18:32:34.690Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 125283, "author_name": "Hartmut Ott", "author_url": "", "post_date": "2016-06-28T12:18:53.117000", "content": "I am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.
", "votes": 1, "replies": [] }, { "id": 125194, "author_name": "Jianbo Ye", "author_url": "", "post_date": "2016-06-27T13:52:25.413000", "content": "[quote=ctd11;125190]
\n\nHow are you doing the feature engineering in C++, are you using Rcpp or Vowpal Wabbit?
\n\n[/quote]
\n\nNo, I just write everything on my own, where I heavily use unordered_map<> and some special designed linked list. But it works great (far more efficient than R or Python), because I can generate features on my mac pro (only 4GB mem). However, it is uneasy for me to perform explorative data analysis using C++. So I also dump all data into a database and build indexes for that purpose.
\n\nHope my reply can help someone who don't have large RAM machine. Undoubtably, one need to rent a large machine (Google cloud or Amazon EC2) or use a computer cluster if he/she wants to generate the submission or perform parameter tunings at the full size of data. Given it is at very early stage of competition, it is not hard to move into top 20. This is my first time to participate in Kaggle and I really learnt a lot!!!
", "votes": 1, "replies": [] }, { "id": 124862, "author_name": "Mattias Fagerlund", "author_url": "", "post_date": "2016-06-22T20:02:25.183000", "content": "Just wanted to say; I just jumped to #11 (and falling quickly) so my advice isn't complete crap.
\n\nBut there's the best advice I can give you; focus on having fun and learning, that's what I do. Winning would be nice, but only one person will eventually win. But we can all learn! And I've learned tons from every competition I've done.
\n\nAnd from the sound of it - you're on top of that at so keep going strong!
\n\n/m
", "votes": 1, "replies": [] }, { "id": 124963, "author_name": "Jianbo Ye", "author_url": "", "post_date": "2016-06-23T23:08:50.840000", "content": "I currently rank in top20.
\n\nAfter trying a few directions, I decided to completely use C++ for feature engineering and combine with some python script for learning and generating submissions. No hard-core computations are done in python or R. The speedups and memory efficiency of writing in C++ improves the solution iterates.
\n\nNow it takes about 5-10 minutes for generating all features on my laptop, and several minutes to train a predictor (on a rent 24-core computing node with >64GB), and almost no time to create a submission. My current solution is a leverage of my own laptop and a rent computing node.
", "votes": 2, "replies": [] }, { "id": 124853, "author_name": "Mattias Fagerlund", "author_url": "", "post_date": "2016-06-22T19:19:18.620000", "content": "Hi, I'm currently at #63 (and quickly falling) using all scratch built software, and my #63 submission takes about 20 minutes to generate. No-where near 22 hours. Though I've been optimizing code for some time.
\n\nThe first thing I did was to transform the CSV files to binary files, so I can load them in minutes. That's something you might want to look into. Though that won't shave anything significant off 22hours.
\n\nMy latest features generating run took 140minutes (2h 20minutes) on my desktop machine (32GB, 6 cores, 12 hyperthreads). The very first feature of that run ("Adding Demanda_uni_equil by BigDog 659.517s") takes 11 minutes and would, on its own, place at #63. I'm hoping to use that as one of many features in a ML solution. If my score jump, this is what I did. If my score doesn't jump - this is still what I did ;)
\n\nSo; 22+ hours is a very long time indeed. That doesn't mean it'll be bad, but chances are that you'll wish to tweak it. And when you do, you're in for another 22+ hours. So if you can iterate faster, you can develop your code faster. Which is nice. But perhaps more importantly, you can hone your intuition faster. And a good intuition is very important to understanding a problem. A 22+ hour turnaround on an idea will make you forget the next idea you had when you picked that one you moved forward with.
\n\nI iterate in minutes, meaning I change something, generate a new estimated solution and find out if it worked in minutes. It's not a full answer, and it doesn't guarantee me success, but it helps me understand the problem. My best ever placing is #6, so I might be full of shit; but so might anyone. Iterating fast is important, and if you can do that, you'll learn faster.
\n\nAlso 22hours isn't a problem, it's an opportunity. Do you do any sequential searches? Replace all of them by dictionaries (hash tables). There are other tips, but I'm betting that's the one that's killing you. Are you reading from disk more than to move information to RAM? That'll kill your performance, even if you're using a cluster of SSD drives.
\n\nPerformance optimization is fun. If you wish to figure out how to speed up your code, you can publish it/parts of it here on the forum asking for tips for making it run faster. You'll give away a few of your ideas, but you'll get far more in return. Just an idea.
\n\n/m
", "votes": 2, "replies": [] }, { "id": 125310, "author_name": "Jianbo Ye", "author_url": "", "post_date": "2016-06-28T14:41:05.027000", "content": "[quote=Hartmut Ott;125283]
\n\nI am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.
\n\n[/quote]
\n\nOoops, you actually remind me that there exists an alternative setup for validation. That could be important for model evaluation.
\n\nJianbo
", "votes": 0, "replies": [] }, { "id": 125296, "author_name": "Sangxia", "author_url": "", "post_date": "2016-06-28T13:24:53.640000", "content": "[quote=Hartmut Ott;125283]
\n\nI am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.
\n\n[/quote]
\n\nCould you share a bit more about how you do validation? Is it close to what you get on LB? Thanks!
", "votes": 0, "replies": [] }, { "id": 125190, "author_name": "ctd11", "author_url": "", "post_date": "2016-06-27T13:11:13.463000", "content": "How are you doing the feature engineering in C++, are you using Rcpp or Vowpal Wabbit?
", "votes": 0, "replies": [] }, { "id": 124882, "author_name": "Mattias Fagerlund", "author_url": "", "post_date": "2016-06-23T04:25:11.777000", "content": "That is impressive! One hour including loading from raw text files, means I have work to do!
", "votes": 0, "replies": [] }, { "id": 124870, "author_name": "JonathanSlomka", "author_url": "", "post_date": "2016-06-22T22:31:02.757000", "content": "I'm ranked 7, I use data under week 8 to create my features and data over week 8 to learn using the past.\nIt takes 1 hour to generate results and I only have 8 GB of RAM and 4 i5 cores on my laptop ^^
\n\nI didn't spent a lot of time on this competition yet so I guess there is a lot of things you can do even if you don't have a big computer
", "votes": 0, "replies": [] }, { "id": 124860, "author_name": "j_scheibel", "author_url": "", "post_date": "2016-06-22T19:56:51.777000", "content": "I appreciate what you're saying. believe i do enjoy myself some optimization for performance. I had to do some major recojiggering (read: memory optimization) to just get this data set to load and process in memory.
\n\nbased on what you've said, it sounds like this is just a case of the nature of the algorithm i've choosen and the settings i'm going with. Other people are either tweaking theirs differently to avoid performance problems or choosing an entirely different algoritm entirely. I know I could probably make my agorithm run about 5-10 times faster (for example) if I rewrote it not to do correlation of feature to target scores values at each level of my trees, but that really helps me select great features (and throw away the noise). random feature selection works okay and is way faster but... well we all want to win.
\n\nMaybe i'll try doing the correlation check every other tree node... give up a little accuracy to double the speed. (food for thought) heck it might even help in the long run through some hidden nature in the data (though I wont hold my breath on that). oh well thanks for the response!
", "votes": 0, "replies": [] }, { "id": 124942, "author_name": "", "author_url": "", "post_date": "2016-06-23T18:32:34.690000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "125283": "I am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.", "125194": "[quote=ctd11;125190]\r\n\r\nHow are you doing the feature engineering in C++, are you using Rcpp or Vowpal Wabbit?\r\n\r\n[/quote]\r\n\r\nNo, I just write everything on my own, where I heavily use unordered_map<> and some special designed linked list. But it works great (far more efficient than R or Python), because I can generate features on my mac pro (only 4GB mem). However, it is uneasy for me to perform explorative data analysis using C++. So I also dump all data into a database and build indexes for that purpose. \r\n\r\nHope my reply can help someone who don't have large RAM machine. Undoubtably, one need to rent a large machine (Google cloud or Amazon EC2) or use a computer cluster if he/she wants to generate the submission or perform parameter tunings at the full size of data. Given it is at very early stage of competition, it is not hard to move into top 20. This is my first time to participate in Kaggle and I really learnt a lot!!!\r\n\r\n\r\n\r\n\r\n\r\n", "124862": "Just wanted to say; I just jumped to #11 (and falling quickly) so my advice isn't complete crap.\r\n\r\nBut there's the best advice I can give you; focus on having fun and learning, that's what I do. Winning would be nice, but only one person will eventually win. But we can all learn! And I've learned tons from every competition I've done.\r\n\r\nAnd from the sound of it - you're on top of that at so keep going strong!\r\n\r\n/m", "124963": "I currently rank in top20.\r\n\r\nAfter trying a few directions, I decided to completely use C++ for feature engineering and combine with some python script for learning and generating submissions. No hard-core computations are done in python or R. The speedups and memory efficiency of writing in C++ improves the solution iterates. \r\n\r\nNow it takes about 5-10 minutes for generating all features on my laptop, and several minutes to train a predictor (on a rent 24-core computing node with >64GB), and almost no time to create a submission. My current solution is a leverage of my own laptop and a rent computing node. ", "124853": "Hi, I'm currently at #63 (and quickly falling) using all scratch built software, and my #63 submission takes about 20 minutes to generate. No-where near 22 hours. Though I've been optimizing code for some time. \r\n\r\nThe first thing I did was to transform the CSV files to binary files, so I can load them in minutes. That's something you might want to look into. Though that won't shave anything significant off 22hours.\r\n\r\nMy latest features generating run took 140minutes (2h 20minutes) on my desktop machine (32GB, 6 cores, 12 hyperthreads). The very first feature of that run (\"Adding Demanda_uni_equil by BigDog 659.517s\") takes 11 minutes and would, on its own, place at #63. I'm hoping to use that as one of many features in a ML solution. If my score jump, this is what I did. If my score doesn't jump - this is still what I did ;)\r\n\r\nThe real problem\r\n----------------\r\n\r\nSo; 22+ hours is a very long time indeed. That doesn't mean it'll be bad, but chances are that you'll wish to tweak it. And when you do, you're in for another 22+ hours. So if you can iterate faster, you can develop your code faster. Which is nice. But perhaps more importantly, you can hone your intuition faster. And a good intuition is very important to understanding a problem. A 22+ hour turnaround on an idea will make you forget the next idea you had when you picked that one you moved forward with.\r\n\r\nI iterate in minutes, meaning I change something, generate a new estimated solution and find out if it worked in minutes. It's not a full answer, and it doesn't guarantee me success, but it helps me understand the problem. My best ever placing is #6, so I might be full of shit; but so might anyone. Iterating fast is important, and if you can do that, you'll learn faster.\r\n\r\nAlso 22hours isn't a problem, it's an opportunity. Do you do *any* sequential searches? Replace *all* of them by dictionaries (hash tables). There are other tips, but I'm betting that's the one that's killing you. Are you reading from disk more than to move information to RAM? That'll kill your performance, even if you're using a cluster of SSD drives. \r\n\r\nPerformance optimization is *fun*. If you wish to figure out how to speed up your code, you can publish it/parts of it here on the forum asking for tips for making it run faster. You'll give away a few of your ideas, but you'll get far more in return. Just an idea. \r\n\r\n/m", "125310": "[quote=Hartmut Ott;125283]\r\n\r\nI am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.\r\n\r\n[/quote]\r\n\r\nOoops, you actually remind me that there exists an alternative setup for validation. That could be important for model evaluation. \r\n\r\nJianbo", "125296": "[quote=Hartmut Ott;125283]\r\n\r\nI am ranked 4th currently and my best solution took about 1h to run in total on a $700 laptop with 16gb. I got lucky on this one by using the right validation set up right from the get go. I fully expect a lot of people to catch up though soon. You should spend most of your time feature engineering.\r\n\r\n[/quote]\r\n\r\nCould you share a bit more about how you do validation? Is it close to what you get on LB? Thanks!", "125190": "How are you doing the feature engineering in C++, are you using Rcpp or Vowpal Wabbit?", "124882": "That is impressive! One hour including loading from raw text files, means I have work to do!\r\n\r\n", "124870": "I'm ranked 7, I use data under week 8 to create my features and data over week 8 to learn using the past.\r\nIt takes 1 hour to generate results and I only have 8 GB of RAM and 4 i5 cores on my laptop ^^\r\n\r\nI didn't spent a lot of time on this competition yet so I guess there is a lot of things you can do even if you don't have a big computer", "124860": "I appreciate what you're saying. believe i do enjoy myself some optimization for performance. I had to do some major recojiggering (read: memory optimization) to just get this data set to load and process in memory. \r\n\r\nbased on what you've said, it sounds like this is just a case of the nature of the algorithm i've choosen and the settings i'm going with. Other people are either tweaking theirs differently to avoid performance problems or choosing an entirely different algoritm entirely. I know I could probably make my agorithm run about 5-10 times faster (for example) if I rewrote it not to do correlation of feature to target scores values at each level of my trees, but that really helps me select great features (and throw away the noise). random feature selection works okay and is way faster but... well we all want to win.\r\n\r\nMaybe i'll try doing the correlation check every other tree node... give up a little accuracy to double the speed. (food for thought) heck it might even help in the long run through some hidden nature in the data (though I wont hold my breath on that). oh well thanks for the response!\r\n", "124844": "Hi guys, i work in my own code base (custom written frame work, not R or python) and i just wanted to get an idea of what kind of run time some of the other frameworks were taking on this data. \r\n\r\nI ask just cause at least in my case if I want to gbm and have it finish in a relatively short time (say under 30 minutes) I need to take some fraction (like 1-5%) of the data to work with. \r\n\r\nThat being said, I finially got something I thought might produce a competitive result. so i went to run it and wow ... takes along time.\r\n\r\nThat is to say, i didnt think it would be much of a problem to do the processing of the final test data when the time came but dang.... it's been running 22 hours and shows no significant signs of being done (though those cores are all pegged and when i break the running code to see how its doing it IS moving forward).\r\n\r\n before now the only submission i've made was a simple format check, so it didnt do any real processing. anyone else seeing the runtime on this big data be that bad? are you avoiding certain processing mechanisms because of that?\r\n\r\n take all the time you want to answer apparently i've got hours :) possibly days.", "124942": "" } }