A very basic question. This would be my first Kaggle competition. \nThe data seems huge and my local python is taking close to 10 mins for a simple regression for just one chunk of data. \nAre you guys working on an AWS cluster or some kind of HPC cluster?
\n\nIf you are running on your analysis on a local setup, please let me know how long it takes.
", "messages": [ { "id": "121165", "postDate": "05/24/2016 17:50:57", "content": "A very basic question. This would be my first Kaggle competition. \nThe data seems huge and my local python is taking close to 10 mins for a simple regression for just one chunk of data. \nAre you guys working on an AWS cluster or some kind of HPC cluster?
\n\nIf you are running on your analysis on a local setup, please let me know how long it takes.
", "rawMarkdown": "A very basic question. This would be my first Kaggle competition. \r\nThe data seems huge and my local python is taking close to 10 mins for a simple regression for just one chunk of data. \r\nAre you guys working on an AWS cluster or some kind of HPC cluster?\r\n\r\nIf you are running on your analysis on a local setup, please let me know how long it takes.", "votes": null }, { "id": "121175", "postDate": "05/24/2016 18:52:52", "content": "I'm working on a local machine with 6 cores. A 24 hour run got me almost to 0.58. Jared Turkewitz was able to finish in 15 hours on a macbook pro. So it's possible, but it still takes awhile, and you've got to use some of the tricks discussed in the forum.
", "rawMarkdown": "I'm working on a local machine with 6 cores. A 24 hour run got me almost to 0.58. Jared Turkewitz was able to finish in 15 hours on a macbook pro. So it's possible, but it still takes awhile, and you've got to use some of the tricks discussed in the forum.", "votes": null }, { "id": "121190", "postDate": "05/24/2016 19:44:59", "content": "What kind of algorithmus/techniques/packages are you using, BreakfastPirate?
\n\nThe huge data set breaks my laptop. Need to find out some tricky ways in the next days... :'D
", "rawMarkdown": "What kind of algorithmus/techniques/packages are you using, BreakfastPirate?\r\n\r\nThe huge data set breaks my laptop. Need to find out some tricky ways in the next days... :'D", "votes": null }, { "id": "121192", "postDate": "05/24/2016 20:00:22", "content": "I'm using XGBoost on R. The main trick is to break the data down into chunks by dividing the data into a grid as described in https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21003/aws-usage
", "rawMarkdown": "I'm using XGBoost on R. The main trick is to break the data down into chunks by dividing the data into a grid as described in https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21003/aws-usage", "votes": null }, { "id": "121196", "postDate": "05/24/2016 20:20:02", "content": "I'm on a quad core macbook pro using Python and my xgboost run is expected to take about 36 hours. It only takes about 2.05 gb of ram after it gets running though.
\n\nThe nice thing is that you can make submissions with partially ran output to see if the results are as you expected. That way you can check for major mistakes quickly without having to wait for the full run to complete.
", "rawMarkdown": "I'm on a quad core macbook pro using Python and my xgboost run is expected to take about 36 hours. It only takes about 2.05 gb of ram after it gets running though. \r\n\r\nThe nice thing is that you can make submissions with partially ran output to see if the results are as you expected. That way you can check for major mistakes quickly without having to wait for the full run to complete.", "votes": null }, { "id": "121213", "postDate": "05/24/2016 22:54:07", "content": "If you want something simple try doing a k nearest neighbors. This should take less than an hour with 20 neighbors and say 3 features (x, y and scaled time of day).
\n\nIf you use R, try using the FNN implementation - using this you can cut the time to roughly 30 minutes (I'm using a 8GB dual core Macbook Pro) and you should be able to get something like .52 - .53 on the leaderboard.
", "rawMarkdown": "If you want something simple try doing a k nearest neighbors. This should take less than an hour with 20 neighbors and say 3 features (x, y and scaled time of day). \r\n\r\nIf you use R, try using the FNN implementation - using this you can cut the time to roughly 30 minutes (I'm using a 8GB dual core Macbook Pro) and you should be able to get something like .52 - .53 on the leaderboard.", "votes": null }, { "id": "121269", "postDate": "05/25/2016 08:25:27", "content": "I am working on a 32 GB, 4 core PC in Python (I don't use all that memory here). I have two methods implemented at the moment, one achieves around 0.565 in 1 hour, the other 0.57 in 3 hours.
\n\nIn short it is possible to generate decent solutions, in a reasonable time, on a normal computer :)
", "rawMarkdown": "I am working on a 32 GB, 4 core PC in Python (I don't use all that memory here). I have two methods implemented at the moment, one achieves around 0.565 in 1 hour, the other 0.57 in 3 hours. \r\n\r\nIn short it is possible to generate decent solutions, in a reasonable time, on a normal computer :)", "votes": null }, { "id": "121320", "postDate": "05/25/2016 16:22:40", "content": "My last submission (0.5905) was generated in around 40 mins on a 16 GB laptop (single-threaded). It uses ~6 GB memory at peak.
", "rawMarkdown": "My last submission (0.5905) was generated in around 40 mins on a 16 GB laptop (single-threaded). It uses ~6 GB memory at peak.", "votes": null }, { "id": "121322", "postDate": "05/25/2016 16:26:42", "content": "[quote=Markus Kliegl;121320]
\n\nMy last submission (0.5905) was generated in around 40 mins on a 16 GB laptop (single-threaded). It uses ~6 GB memory at peak.
\n\n[/quote]
\n\nThat's awesome Markus! You've obviously figured out something the rest of us have not!
", "rawMarkdown": "[quote=Markus Kliegl;121320]\r\n\r\nMy last submission (0.5905) was generated in around 40 mins on a 16 GB laptop (single-threaded). It uses ~6 GB memory at peak.\r\n\r\n[/quote]\r\n\r\nThat's awesome Markus! You've obviously figured out something the rest of us have not!", "votes": null }, { "id": "121348", "postDate": "05/25/2016 19:16:15", "content": "I think the basic ideas are out there already: learning various joint or individual distributions for the space and time variables for each place_id (and the overall relative popularities). The challenge is doing this in a computationally efficient manner without losing too much information.
", "rawMarkdown": "I think the basic ideas are out there already: learning various joint or individual distributions for the space and time variables for each place_id (and the overall relative popularities). The challenge is doing this in a computationally efficient manner without losing too much information.", "votes": null }, { "id": "121389", "postDate": "05/26/2016 02:05:35", "content": "Well, my latest XGBoost run got 0.5858, but it took 50 hours to run. :-( The CPU got so hot I had to get a portable fan out and blow it in the side of the computer to cool it down. I'm using 8 features. I could cut eta in half and double the number of rounds, but that would take 100 hours. So I think I need to come up with a different approach.
", "rawMarkdown": "Well, my latest XGBoost run got 0.5858, but it took 50 hours to run. :-( The CPU got so hot I had to get a portable fan out and blow it in the side of the computer to cool it down. I'm using 8 features. I could cut eta in half and double the number of rounds, but that would take 100 hours. So I think I need to come up with a different approach.", "votes": null }, { "id": "121393", "postDate": "05/26/2016 02:47:54", "content": "[quote=BreakfastPirate;121389]
\n\nThe CPU got so hot I had to get a portable fan out and blow it in the side of the computer to cool it down.
\n\n[/quote]
\n\nEvery time I left my laptop running for hours, I have a fear of saying goodbye to it forever XP
", "rawMarkdown": "[quote=BreakfastPirate;121389]\r\n\r\nThe CPU got so hot I had to get a portable fan out and blow it in the side of the computer to cool it down. \r\n\r\n[/quote]\r\n\r\nEvery time I left my laptop running for hours, I have a fear of saying goodbye to it forever XP", "votes": null }, { "id": "121541", "postDate": "05/27/2016 05:51:07", "content": "@Markus Kliegl. Are you sure it is 40 mins, not 40 hours? I highly doubt.
", "rawMarkdown": "Markus Kliegl. Are you sure it is 40 mins, not 40 hours? I highly doubt.", "votes": null }, { "id": "121542", "postDate": "05/27/2016 06:03:54", "content": "Yes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.
", "rawMarkdown": "Yes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.", "votes": null }, { "id": "121543", "postDate": "05/27/2016 06:40:11", "content": "@Markus Kliegl. wow, that's amazing. You must have come up with something never post publicly.
", "rawMarkdown": "Markus Kliegl. wow, that's amazing. You must have come up with something never post publicly.", "votes": null }, { "id": "121619", "postDate": "05/27/2016 23:06:53", "content": "Hey, guys. I am a new bee here. Does any think of using things like discriminant analysis to analysis the data? boosting is not quite right here I can guess, as you may need a lot of computation to get the right results
", "rawMarkdown": "Hey, guys. I am a new bee here. Does any think of using things like discriminant analysis to analysis the data? boosting is not quite right here I can guess, as you may need a lot of computation to get the right results", "votes": null }, { "id": "123977", "postDate": "06/14/2016 19:36:34", "content": "@BreakfastPirate There're 100000+ classes. How do you apply xgb model to such problem? Use multi:softprob or rank:pariwise? Thank you.
", "rawMarkdown": "BreakfastPirate There're 100000+ classes. How do you apply xgb model to such problem? Use multi:softprob or rank:pariwise? Thank you.", "votes": null }, { "id": "123980", "postDate": "06/14/2016 20:06:22", "content": "I used multi:softprob. If you use a grid to break the problem down into many smaller problems, then each has much fewer than 100,000 classes.
", "rawMarkdown": "I used multi:softprob. If you use a grid to break the problem down into many smaller problems, then each has much fewer than 100,000 classes.", "votes": null }, { "id": "125603", "postDate": "06/30/2016 19:59:59", "content": "[quote=Markus Kliegl;121542]
\n\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.
\n\n[/quote]
\n\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (score_samples function from sklearn) takes extremely long time.\nThank you!
", "rawMarkdown": "[quote=Markus Kliegl;121542]\r\n\r\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.\r\n\r\n[/quote]\r\n\r\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (*score_samples* function from sklearn) takes extremely long time.\r\nThank you!", "votes": null }, { "id": "125631", "postDate": "07/01/2016 00:35:54", "content": "[quote=Arcady27;125603]
\n\n[quote=Markus Kliegl;121542]
\n\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.
\n\n[/quote]
\n\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (score_samples function from sklearn) takes extremely long time.\nThank you!
\n\n[/quote]
\n\n40 minutes is from earlier in the competition. At that point, I was not using any precomputed information. Everything was learned and predicted based on a single pass through train and test data. My current solution is more complex and takes longer.
\n\nI think for this competition thinking of ways to efficiently approximate something like KDE is a good approach. You could speed up KDE by e.g. applying it to a histogram instead of the actual data points. You could also use something like GMM with not too many components.
", "rawMarkdown": "[quote=Arcady27;125603]\r\n\r\n[quote=Markus Kliegl;121542]\r\n\r\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.\r\n\r\n[/quote]\r\n\r\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (*score_samples* function from sklearn) takes extremely long time.\r\nThank you!\r\n\r\n[/quote]\r\n\r\n40 minutes is from earlier in the competition. At that point, I was not using any precomputed information. Everything was learned and predicted based on a single pass through train and test data. My current solution is more complex and takes longer.\r\n\r\nI think for this competition thinking of ways to efficiently approximate something like KDE is a good approach. You could speed up KDE by e.g. applying it to a histogram instead of the actual data points. You could also use something like GMM with not too many components.", "votes": null }, { "id": "125663", "postDate": "07/01/2016 12:32:33", "content": "That is very interesting. I managed to efficiently (more or less) create histograms for all places, however I was not able to improve my score. While using histograms there are many additional hyper parameters (equivalent of number of bins in histogram). Maybe if I would calculate KDE instead of histograms I would be able to optimize the results. Anyway thank you for sharing your views.
", "rawMarkdown": "That is very interesting. I managed to efficiently (more or less) create histograms for all places, however I was not able to improve my score. While using histograms there are many additional hyper parameters (equivalent of number of bins in histogram). Maybe if I would calculate KDE instead of histograms I would be able to optimize the results. Anyway thank you for sharing your views.", "votes": null }, { "id": "125665", "postDate": "07/01/2016 13:28:25", "content": "", "rawMarkdown": "", "votes": null }, { "id": "125666", "postDate": "07/01/2016 13:29:43", "content": "[quote=Arcady27;125603]
\n\n[quote=Markus Kliegl;121542]
\n\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.
\n\n[/quote]
\n\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (score_samples function from sklearn) takes extremely long time.\nThank you!
\n\n[/quote]
\n\nHi Arcady
\n\nIf your are computing likelyhood over all the data in one go then there are a very large number of likelyhoods (KDEs) that need to be computed because there are a very large number of classes (100,000? or more from memory).
\n\nMy thoughts are to divide the x,y features and consider a 2D grid (say 10x10) over the spatial locations to perform Bayes models on each cell. That way the number of classes in each cell will be much lower, resulting in far less likelyhood (KDEs) for each model. This should also greatly speed up the test time, since only a smaller number of posterior probabilities need to be evaluated per test sample.
\n\nI haven't had time to try this out yet but would be glad to here from anyone who has.
", "rawMarkdown": "[quote=Arcady27;125603]\r\n\r\n[quote=Markus Kliegl;121542]\r\n\r\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.\r\n\r\n[/quote]\r\n\r\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (*score_samples* function from sklearn) takes extremely long time.\r\nThank you!\r\n\r\n[/quote]\r\n\r\nHi Arcady\r\n\r\nIf your are computing likelyhood over all the data in one go then there are a very large number of likelyhoods (KDEs) that need to be computed because there are a very large number of classes (100,000? or more from memory).\r\n\r\nMy thoughts are to divide the x,y features and consider a 2D grid (say 10x10) over the spatial locations to perform Bayes models on each cell. That way the number of classes in each cell will be much lower, resulting in far less likelyhood (KDEs) for each model. This should also greatly speed up the test time, since only a smaller number of posterior probabilities need to be evaluated per test sample.\r\n\r\nI haven't had time to try this out yet but would be glad to here from anyone who has.", "votes": null }, { "id": "430890", "postDate": "12/01/2018 06:11:36", "content": "You can get pretty fast multiple GPU machine here https://goo.gl/TcDcAo
", "rawMarkdown": "You can get pretty fast multiple GPU machine here https://goo.gl/TcDcAo", "votes": null }, { "id": "479662", "postDate": "02/27/2019 10:24:34", "content": "Wow
", "rawMarkdown": "Wow", "votes": null } ], "comments": [ { "id": 121175, "author_name": "breakfastpirate", "author_url": "", "post_date": "05/24/2016 18:52:52", "content": "I'm working on a local machine with 6 cores. A 24 hour run got me almost to 0.58. Jared Turkewitz was able to finish in 15 hours on a macbook pro. So it's possible, but it still takes awhile, and you've got to use some of the tricks discussed in the forum.
", "votes": null, "replies": [] }, { "id": 121190, "author_name": "antsek", "author_url": "", "post_date": "05/24/2016 19:44:59", "content": "What kind of algorithmus/techniques/packages are you using, BreakfastPirate?
\n\nThe huge data set breaks my laptop. Need to find out some tricky ways in the next days... :'D
", "votes": null, "replies": [] }, { "id": 121192, "author_name": "breakfastpirate", "author_url": "", "post_date": "05/24/2016 20:00:22", "content": "I'm using XGBoost on R. The main trick is to break the data down into chunks by dividing the data into a grid as described in https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21003/aws-usage
", "votes": null, "replies": [] }, { "id": 121196, "author_name": "qwang88", "author_url": "", "post_date": "05/24/2016 20:20:02", "content": "I'm on a quad core macbook pro using Python and my xgboost run is expected to take about 36 hours. It only takes about 2.05 gb of ram after it gets running though.
\n\nThe nice thing is that you can make submissions with partially ran output to see if the results are as you expected. That way you can check for major mistakes quickly without having to wait for the full run to complete.
", "votes": null, "replies": [] }, { "id": 121213, "author_name": "apapiu", "author_url": "", "post_date": "05/24/2016 22:54:07", "content": "If you want something simple try doing a k nearest neighbors. This should take less than an hour with 20 neighbors and say 3 features (x, y and scaled time of day).
\n\nIf you use R, try using the FNN implementation - using this you can cut the time to roughly 30 minutes (I'm using a 8GB dual core Macbook Pro) and you should be able to get something like .52 - .53 on the leaderboard.
", "votes": null, "replies": [] }, { "id": 121269, "author_name": "marekkowalski", "author_url": "", "post_date": "05/25/2016 08:25:27", "content": "I am working on a 32 GB, 4 core PC in Python (I don't use all that memory here). I have two methods implemented at the moment, one achieves around 0.565 in 1 hour, the other 0.57 in 3 hours.
\n\nIn short it is possible to generate decent solutions, in a reasonable time, on a normal computer :)
", "votes": null, "replies": [] }, { "id": 121320, "author_name": "mkliegl", "author_url": "", "post_date": "05/25/2016 16:22:40", "content": "My last submission (0.5905) was generated in around 40 mins on a 16 GB laptop (single-threaded). It uses ~6 GB memory at peak.
", "votes": null, "replies": [ { "id": 479662, "author_name": "samtimberlan", "author_url": "", "post_date": "02/27/2019 10:24:34", "content": "Wow
", "votes": null, "replies": [] } ] }, { "id": 121322, "author_name": "breakfastpirate", "author_url": "", "post_date": "05/25/2016 16:26:42", "content": "[quote=Markus Kliegl;121320]
\n\nMy last submission (0.5905) was generated in around 40 mins on a 16 GB laptop (single-threaded). It uses ~6 GB memory at peak.
\n\n[/quote]
\n\nThat's awesome Markus! You've obviously figured out something the rest of us have not!
", "votes": null, "replies": [] }, { "id": 121348, "author_name": "mkliegl", "author_url": "", "post_date": "05/25/2016 19:16:15", "content": "I think the basic ideas are out there already: learning various joint or individual distributions for the space and time variables for each place_id (and the overall relative popularities). The challenge is doing this in a computationally efficient manner without losing too much information.
", "votes": null, "replies": [] }, { "id": 121389, "author_name": "breakfastpirate", "author_url": "", "post_date": "05/26/2016 02:05:35", "content": "Well, my latest XGBoost run got 0.5858, but it took 50 hours to run. :-( The CPU got so hot I had to get a portable fan out and blow it in the side of the computer to cool it down. I'm using 8 features. I could cut eta in half and double the number of rounds, but that would take 100 hours. So I think I need to come up with a different approach.
", "votes": null, "replies": [] }, { "id": 121393, "author_name": "leonlu", "author_url": "", "post_date": "05/26/2016 02:47:54", "content": "[quote=BreakfastPirate;121389]
\n\nThe CPU got so hot I had to get a portable fan out and blow it in the side of the computer to cool it down.
\n\n[/quote]
\n\nEvery time I left my laptop running for hours, I have a fear of saying goodbye to it forever XP
", "votes": null, "replies": [] }, { "id": 121541, "author_name": "ma350365879", "author_url": "", "post_date": "05/27/2016 05:51:07", "content": "@Markus Kliegl. Are you sure it is 40 mins, not 40 hours? I highly doubt.
", "votes": null, "replies": [] }, { "id": 121542, "author_name": "mkliegl", "author_url": "", "post_date": "05/27/2016 06:03:54", "content": "Yes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.
", "votes": null, "replies": [] }, { "id": 121543, "author_name": "ma350365879", "author_url": "", "post_date": "05/27/2016 06:40:11", "content": "@Markus Kliegl. wow, that's amazing. You must have come up with something never post publicly.
", "votes": null, "replies": [] }, { "id": 121619, "author_name": "raingstar", "author_url": "", "post_date": "05/27/2016 23:06:53", "content": "Hey, guys. I am a new bee here. Does any think of using things like discriminant analysis to analysis the data? boosting is not quite right here I can guess, as you may need a lot of computation to get the right results
", "votes": null, "replies": [] }, { "id": 123977, "author_name": "beedata", "author_url": "", "post_date": "06/14/2016 19:36:34", "content": "@BreakfastPirate There're 100000+ classes. How do you apply xgb model to such problem? Use multi:softprob or rank:pariwise? Thank you.
", "votes": null, "replies": [] }, { "id": 123980, "author_name": "breakfastpirate", "author_url": "", "post_date": "06/14/2016 20:06:22", "content": "I used multi:softprob. If you use a grid to break the problem down into many smaller problems, then each has much fewer than 100,000 classes.
", "votes": null, "replies": [] }, { "id": 125603, "author_name": "arcady27", "author_url": "", "post_date": "06/30/2016 19:59:59", "content": "[quote=Markus Kliegl;121542]
\n\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.
\n\n[/quote]
\n\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (score_samples function from sklearn) takes extremely long time.\nThank you!
", "votes": null, "replies": [] }, { "id": 125631, "author_name": "mkliegl", "author_url": "", "post_date": "07/01/2016 00:35:54", "content": "[quote=Arcady27;125603]
\n\n[quote=Markus Kliegl;121542]
\n\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.
\n\n[/quote]
\n\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (score_samples function from sklearn) takes extremely long time.\nThank you!
\n\n[/quote]
\n\n40 minutes is from earlier in the competition. At that point, I was not using any precomputed information. Everything was learned and predicted based on a single pass through train and test data. My current solution is more complex and takes longer.
\n\nI think for this competition thinking of ways to efficiently approximate something like KDE is a good approach. You could speed up KDE by e.g. applying it to a histogram instead of the actual data points. You could also use something like GMM with not too many components.
", "votes": null, "replies": [] }, { "id": 125663, "author_name": "mwilam", "author_url": "", "post_date": "07/01/2016 12:32:33", "content": "That is very interesting. I managed to efficiently (more or less) create histograms for all places, however I was not able to improve my score. While using histograms there are many additional hyper parameters (equivalent of number of bins in histogram). Maybe if I would calculate KDE instead of histograms I would be able to optimize the results. Anyway thank you for sharing your views.
", "votes": null, "replies": [] }, { "id": 125665, "author_name": "srimalj", "author_url": "", "post_date": "07/01/2016 13:28:25", "content": "", "votes": null, "replies": [] }, { "id": 125666, "author_name": "srimalj", "author_url": "", "post_date": "07/01/2016 13:29:43", "content": "[quote=Arcady27;125603]
\n\n[quote=Markus Kliegl;121542]
\n\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.
\n\n[/quote]
\n\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (score_samples function from sklearn) takes extremely long time.\nThank you!
\n\n[/quote]
\n\nHi Arcady
\n\nIf your are computing likelyhood over all the data in one go then there are a very large number of likelyhoods (KDEs) that need to be computed because there are a very large number of classes (100,000? or more from memory).
\n\nMy thoughts are to divide the x,y features and consider a 2D grid (say 10x10) over the spatial locations to perform Bayes models on each cell. That way the number of classes in each cell will be much lower, resulting in far less likelyhood (KDEs) for each model. This should also greatly speed up the test time, since only a smaller number of posterior probabilities need to be evaluated per test sample.
\n\nI haven't had time to try this out yet but would be glad to here from anyone who has.
", "votes": null, "replies": [] }, { "id": 430890, "author_name": "alx1379", "author_url": "", "post_date": "12/01/2018 06:11:36", "content": "You can get pretty fast multiple GPU machine here https://goo.gl/TcDcAo
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "121165": "A very basic question. This would be my first Kaggle competition. \r\nThe data seems huge and my local python is taking close to 10 mins for a simple regression for just one chunk of data. \r\nAre you guys working on an AWS cluster or some kind of HPC cluster?\r\n\r\nIf you are running on your analysis on a local setup, please let me know how long it takes.", "121175": "I'm working on a local machine with 6 cores. A 24 hour run got me almost to 0.58. Jared Turkewitz was able to finish in 15 hours on a macbook pro. So it's possible, but it still takes awhile, and you've got to use some of the tricks discussed in the forum.", "121190": "What kind of algorithmus/techniques/packages are you using, BreakfastPirate?\r\n\r\nThe huge data set breaks my laptop. Need to find out some tricky ways in the next days... :'D", "121192": "I'm using XGBoost on R. The main trick is to break the data down into chunks by dividing the data into a grid as described in https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21003/aws-usage", "121196": "I'm on a quad core macbook pro using Python and my xgboost run is expected to take about 36 hours. It only takes about 2.05 gb of ram after it gets running though. \r\n\r\nThe nice thing is that you can make submissions with partially ran output to see if the results are as you expected. That way you can check for major mistakes quickly without having to wait for the full run to complete.", "121213": "If you want something simple try doing a k nearest neighbors. This should take less than an hour with 20 neighbors and say 3 features (x, y and scaled time of day). \r\n\r\nIf you use R, try using the FNN implementation - using this you can cut the time to roughly 30 minutes (I'm using a 8GB dual core Macbook Pro) and you should be able to get something like .52 - .53 on the leaderboard.", "121269": "I am working on a 32 GB, 4 core PC in Python (I don't use all that memory here). I have two methods implemented at the moment, one achieves around 0.565 in 1 hour, the other 0.57 in 3 hours. \r\n\r\nIn short it is possible to generate decent solutions, in a reasonable time, on a normal computer :)", "121320": "My last submission (0.5905) was generated in around 40 mins on a 16 GB laptop (single-threaded). It uses ~6 GB memory at peak.", "121322": "[quote=Markus Kliegl;121320]\r\n\r\nMy last submission (0.5905) was generated in around 40 mins on a 16 GB laptop (single-threaded). It uses ~6 GB memory at peak.\r\n\r\n[/quote]\r\n\r\nThat's awesome Markus! You've obviously figured out something the rest of us have not!", "121348": "I think the basic ideas are out there already: learning various joint or individual distributions for the space and time variables for each place_id (and the overall relative popularities). The challenge is doing this in a computationally efficient manner without losing too much information.", "121389": "Well, my latest XGBoost run got 0.5858, but it took 50 hours to run. :-( The CPU got so hot I had to get a portable fan out and blow it in the side of the computer to cool it down. I'm using 8 features. I could cut eta in half and double the number of rounds, but that would take 100 hours. So I think I need to come up with a different approach.", "121393": "[quote=BreakfastPirate;121389]\r\n\r\nThe CPU got so hot I had to get a portable fan out and blow it in the side of the computer to cool it down. \r\n\r\n[/quote]\r\n\r\nEvery time I left my laptop running for hours, I have a fear of saying goodbye to it forever XP", "121541": "Markus Kliegl. Are you sure it is 40 mins, not 40 hours? I highly doubt.", "121542": "Yes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.", "121543": "Markus Kliegl. wow, that's amazing. You must have come up with something never post publicly.", "121619": "Hey, guys. I am a new bee here. Does any think of using things like discriminant analysis to analysis the data? boosting is not quite right here I can guess, as you may need a lot of computation to get the right results", "123977": "BreakfastPirate There're 100000+ classes. How do you apply xgb model to such problem? Use multi:softprob or rank:pariwise? Thank you.", "123980": "I used multi:softprob. If you use a grid to break the problem down into many smaller problems, then each has much fewer than 100,000 classes.", "125603": "[quote=Markus Kliegl;121542]\r\n\r\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.\r\n\r\n[/quote]\r\n\r\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (*score_samples* function from sklearn) takes extremely long time.\r\nThank you!", "125631": "[quote=Arcady27;125603]\r\n\r\n[quote=Markus Kliegl;121542]\r\n\r\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.\r\n\r\n[/quote]\r\n\r\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (*score_samples* function from sklearn) takes extremely long time.\r\nThank you!\r\n\r\n[/quote]\r\n\r\n40 minutes is from earlier in the competition. At that point, I was not using any precomputed information. Everything was learned and predicted based on a single pass through train and test data. My current solution is more complex and takes longer.\r\n\r\nI think for this competition thinking of ways to efficiently approximate something like KDE is a good approach. You could speed up KDE by e.g. applying it to a histogram instead of the actual data points. You could also use something like GMM with not too many components.", "125663": "That is very interesting. I managed to efficiently (more or less) create histograms for all places, however I was not able to improve my score. While using histograms there are many additional hyper parameters (equivalent of number of bins in histogram). Maybe if I would calculate KDE instead of histograms I would be able to optimize the results. Anyway thank you for sharing your views.", "125665": "", "125666": "[quote=Arcady27;125603]\r\n\r\n[quote=Markus Kliegl;121542]\r\n\r\nYes, 40 mins. I'm not using XGBoost - it's custom code that just makes one pass through the train and test data.\r\n\r\n[/quote]\r\n\r\nExcuse for my maybe stupid question, but do you use some precomputed KDE scores for all objects ? (besides the 40 minutes of running script) of. I use KDE from sklearn to determine joint probabilities of some features to places_ids (one estimation for one place_id) and calculating likelihoods for objects (*score_samples* function from sklearn) takes extremely long time.\r\nThank you!\r\n\r\n[/quote]\r\n\r\nHi Arcady\r\n\r\nIf your are computing likelyhood over all the data in one go then there are a very large number of likelyhoods (KDEs) that need to be computed because there are a very large number of classes (100,000? or more from memory).\r\n\r\nMy thoughts are to divide the x,y features and consider a 2D grid (say 10x10) over the spatial locations to perform Bayes models on each cell. That way the number of classes in each cell will be much lower, resulting in far less likelyhood (KDEs) for each model. This should also greatly speed up the test time, since only a smaller number of posterior probabilities need to be evaluated per test sample.\r\n\r\nI haven't had time to try this out yet but would be glad to here from anyone who has.", "430890": "You can get pretty fast multiple GPU machine here https://goo.gl/TcDcAo", "479662": "Wow" }, "source": "meta" }