I am thinking of joining this competition, but unsure whether my computer will be able to run effective models given the large size of the dataset.
\n\nI wonder how much memory and processing capacity would be enough?
\n\nPS. Currently I have 4 cores and 16GB RAM.
", "messages": [ { "id": 122978, "postDate": "2016-06-08T23:43:08.383Z", "content": "I wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.
", "rawMarkdown": "I wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.", "votes": 9 }, { "id": 123737, "postDate": "2016-06-13T19:41:01.520Z", "content": "The problem with medium sized datasets is that you can just XGBoost them to death. These kinds of competitions at least introduce some variety into ultimate solutions.
", "rawMarkdown": "The problem with medium sized datasets is that you can just XGBoost them to death. These kinds of competitions at least introduce some variety into ultimate solutions.", "votes": 5 }, { "id": 123698, "postDate": "2016-06-13T15:17:57.447Z", "content": "If you believe that the latest client behaviour in the train set will be more valuable to predict the behaviour in the test set why don't you keep the last x entries per user - this will significantly reduce the size of the data. You can check this by initially trying to predict on the last week of the training data. Big data isn't an issue if you come up with a way of making it a small data problem - it certainly makes people think rather than going "100,000 rows? - xgb here I come!" ;)
", "rawMarkdown": "If you believe that the latest client behaviour in the train set will be more valuable to predict the behaviour in the test set why don't you keep the last x entries per user - this will significantly reduce the size of the data. You can check this by initially trying to predict on the last week of the training data. Big data isn't an issue if you come up with a way of making it a small data problem - it certainly makes people think rather than going \"100,000 rows? - xgb here I come!\" ;) ", "votes": 5 }, { "id": 123646, "postDate": "2016-06-13T09:02:44.293Z", "content": "I think we'll have to get used to big data sort of issues... and running into hardware limitations will become a daily headache for most of us. \nI have 4 cores/16 GB memory and the last Expedia competition was a nice experience with XGBoost running for hours... It was like working in the punched card era where error codes are a killer ;-))\nMy advice would be to not use RFs or XTCs on the entire data set or you will run into disk swapping issues but you should be able to use XGBoost or vowpal wabbit. The latter is really fast for online learning tasks. Looking into spark is another option.\nIn any case downsampling is your best friend.\nGood luck
", "rawMarkdown": "I think we'll have to get used to big data sort of issues... and running into hardware limitations will become a daily headache for most of us. \r\nI have 4 cores/16 GB memory and the last Expedia competition was a nice experience with XGBoost running for hours... It was like working in the punched card era where error codes are a killer ;-))\r\nMy advice would be to not use RFs or XTCs on the entire data set or you will run into disk swapping issues but you should be able to use XGBoost or vowpal wabbit. The latter is really fast for online learning tasks. Looking into spark is another option.\r\nIn any case downsampling is your best friend.\r\nGood luck", "votes": 5 }, { "id": 122971, "postDate": "2016-06-08T22:50:39.050Z", "content": "people have won with less
", "rawMarkdown": "people have won with less", "votes": 5 }, { "id": 122964, "postDate": "2016-06-08T22:01:11.690Z", "content": "I am thinking of joining this competition, but unsure whether my computer will be able to run effective models given the large size of the dataset.
\n\nI wonder how much memory and processing capacity would be enough?
\n\nPS. Currently I have 4 cores and 16GB RAM.
", "rawMarkdown": "I am thinking of joining this competition, but unsure whether my computer will be able to run effective models given the large size of the dataset.\r\n\r\nI wonder how much memory and processing capacity would be enough?\r\n\r\nPS. Currently I have 4 cores and 16GB RAM.", "votes": 5 }, { "id": 130010, "postDate": "2016-08-03T10:36:49.590Z", "content": "Hello everybody,
\nI just wanted to share. I am an absolute beginner and faced the same issues about hardware limitations talked about here.
I was away from main computer (on a nice vacation) and had with me but an old core 2 lappy with 4 GB.
\n\nI just decided to sample the test and train files in order to be able to play with the data. The gist follows; it was enough to help me wrap my head around the data until I had a more strudy system to fiddle around in.
\n\n$ head -1 test.csv > test_sample.csv && tail -n +2 test.csv | shuf -n 349963 >> test_sample.csv\n$ head -1 train.csv > train_sample.csv && tail -n +2 train.csv | shuf -n 3709023 >> train_sample.csv
\n\nI think I decided to use 5% of data so that laptop wouldn't blow up. Good luck.
\n\nYIKES, now I see how old these posts are... Still posting so it may help someone in the future.
", "rawMarkdown": "Hello everybody, \r\nI just wanted to share. I am an absolute beginner and faced the same issues about hardware limitations talked about here.\r\n\r\nI was away from main computer (on a nice vacation) and had with me but an old core 2 lappy with 4 GB.\r\n\r\nI just decided to sample the test and train files in order to be able to play with the data. The gist follows; it was enough to help me wrap my head around the data until I had a more strudy system to fiddle around in. \r\n\r\n$ head -1 test.csv > test_sample.csv && tail -n +2 test.csv | shuf -n 349963 >> test_sample.csv\r\n$ head -1 train.csv > train_sample.csv && tail -n +2 train.csv | shuf -n 3709023 >> train_sample.csv\r\n\r\nI think I decided to use 5% of data so that laptop wouldn't blow up. Good luck.\r\n\r\nYIKES, now I see how old these posts are... Still posting so it may help someone in the future.", "votes": 3 }, { "id": 123004, "postDate": "2016-06-09T05:21:41.573Z", "content": "Yup, you can definitely get a lot of understanding with prefiltered data, or even just parts of the data. If you've got enough of it figured out with partial data, it's probably worth it to spin up an EC2 if you have to.
\n\nGert kicked butt on Rossman with just a laptop with limited/no net access, for instance... http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/
\n\nIn any case you should be able to get pretty far with 16GB.
\n\nAlso, if you're willing to use old hardware, large memory systems are actually pretty cheap ;) I was able (thanks to a local sale) get a dual E5405 Xeon last week for $40, and got it to 32GB for $35 (and then added a GF770 I already had)
", "rawMarkdown": "Yup, you can definitely get a lot of understanding with prefiltered data, or even just parts of the data. If you've got enough of it figured out with partial data, it's probably worth it to spin up an EC2 if you have to.\r\n\r\nGert kicked butt on Rossman with just a laptop with limited/no net access, for instance... http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/\r\n\r\nIn any case you should be able to get pretty far with 16GB.\r\n\r\nAlso, if you're willing to use old hardware, large memory systems are actually pretty cheap ;) I was able (thanks to a local sale) get a dual E5405 Xeon last week for $40, and got it to 32GB for $35 (and then added a GF770 I already had)", "votes": 4 }, { "id": 125357, "postDate": "2016-06-28T21:35:35.833Z", "content": "[quote=Bindukumar J;125300]
\n\n[quote=CarltonChen;122987]
\n\n[quote=Jared Turkewitz;122978]
\n\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.
\n\n[/quote]
\n\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.
\n\n[/quote]\nsame is the case with me. i am struck at how to handle this dataset.
\n\n[/quote]
\n\nFor me this has been the greatest challenge of this competition.
\n\nThe path I've taken is to build a random 25% sample of train.csv. This random sample is further split between train and test datasets. All training is done on that 25%.
\n\nAlthough still slow, I am able to work with this on my 8GB RAM laptop.
", "rawMarkdown": "[quote=Bindukumar J;125300]\r\n\r\n[quote=CarltonChen;122987]\r\n\r\n[quote=Jared Turkewitz;122978]\r\n\r\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.\r\n\r\n[/quote]\r\n\r\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.\r\n\r\n[/quote]\r\nsame is the case with me. i am struck at how to handle this dataset.\r\n\r\n[/quote]\r\n\r\nFor me this has been the greatest challenge of this competition.\r\n\r\nThe path I've taken is to build a random 25% sample of train.csv. This random sample is further split between train and test datasets. All training is done on that 25%.\r\n\r\nAlthough still slow, I am able to work with this on my 8GB RAM laptop.", "votes": 1 }, { "id": 123604, "postDate": "2016-06-13T04:22:09.047Z", "content": "Hi!\nWith feature engineering and merging the size of the datasets can anyway be much bigger than the 3 GB at the end ...
", "rawMarkdown": "Hi!\r\nWith feature engineering and merging the size of the datasets can anyway be much bigger than the 3 GB at the end ...", "votes": 1 }, { "id": 123547, "postDate": "2016-06-12T16:22:04.233Z", "content": "Try AWS or some similar tool.
", "rawMarkdown": "Try AWS or some similar tool.", "votes": 1 }, { "id": 123040, "postDate": "2016-06-09T10:29:03.660Z", "content": "Hi,
\n\nI will try to sample 10-20% of data from Train.csv for cross-validation analysis. Then go into the cloud to model on the whole training data. Does that make sense?\nI have 32GB machine and it takes just too long to work on the whole set. Have not tried Random Forest or something but this should kill it.
", "rawMarkdown": "Hi,\r\n\r\nI will try to sample 10-20% of data from Train.csv for cross-validation analysis. Then go into the cloud to model on the whole training data. Does that make sense?\r\nI have 32GB machine and it takes just too long to work on the whole set. Have not tried Random Forest or something but this should kill it.\r\n", "votes": 1 }, { "id": 124590, "postDate": "2016-06-20T06:19:35.780Z", "content": "If you want a hassle free setup on AWS, you may want to use docker on a linux EC2. You will then be able to import docker images created by kaggle teams and available on docker hub. This way you replicate any Kaggle environment (python, R, Julia...) with all the good tools like xgboost, theano already there for you. If you want to know more on docker containers you can have a look at this post on kaggle's blog.\nHope this helps.
", "rawMarkdown": "If you want a hassle free setup on AWS, you may want to use [docker on a linux EC2][1]. You will then be able to import docker images created by kaggle teams and available on [docker hub][2]. This way you replicate any Kaggle environment (python, R, Julia...) with all the good tools like xgboost, theano already there for you. If you want to know more on docker containers you can have a look at [this post][3] on kaggle's blog.\r\nHope this helps.\r\n\r\n\r\n [1]: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html\r\n [2]: https://hub.docker.com/u/kaggle/\r\n [3]: http://blog.kaggle.com/2016/02/05/how-to-get-started-with-data-science-in-containers/", "votes": 2 }, { "id": 123742, "postDate": "2016-06-13T20:25:04.683Z", "content": "[quote=Dmitriy Guller;123737]
\n\nThe problem with medium sized datasets is that you can just XGBoost them to death. These kinds of competitions at least introduce some variety into ultimate solutions.
\n\n[/quote]
\n\nBut datasets of this size or bigger also get xgboosted to death by the top solutions, it just requires more patience and better hardware. This dataset is actually not big in the sense of fitting on one machine, it just requires a good machine. The main advantage I see is that it is less likely to be scripted to death because the script run time is too long.
", "rawMarkdown": "[quote=Dmitriy Guller;123737]\r\n\r\nThe problem with medium sized datasets is that you can just XGBoost them to death. These kinds of competitions at least introduce some variety into ultimate solutions.\r\n\r\n[/quote]\r\n\r\nBut datasets of this size or bigger also get xgboosted to death by the top solutions, it just requires more patience and better hardware. This dataset is actually not big in the sense of fitting on one machine, it just requires a good machine. The main advantage I see is that it is less likely to be scripted to death because the script run time is too long.", "votes": 2 }, { "id": 123610, "postDate": "2016-06-13T05:04:53.407Z", "content": "[quote=CarltonChen;122987]
\n\n[quote=Jared Turkewitz;122978]
\n\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.
\n\n[/quote]
\n\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.
\n\n[/quote]
\n\nThe problem is that it isn't about knowledge, it is about hardware. While AWS is a potential solution, it is very expensive for individual use. With that said, you can probably just scale the dataset dramatically down without losing that much accuracy, because percent statistical uncertainty scales like sqrt(N) / N and will likely be much smaller than the systematic uncertainty. Although it will still be a huge pain to make submissions.
", "rawMarkdown": "[quote=CarltonChen;122987]\r\n\r\n[quote=Jared Turkewitz;122978]\r\n\r\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.\r\n\r\n[/quote]\r\n\r\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.\r\n\r\n[/quote]\r\n\r\nThe problem is that it isn't about knowledge, it is about hardware. While AWS is a potential solution, it is very expensive for individual use. With that said, you can probably just scale the dataset dramatically down without losing that much accuracy, because percent statistical uncertainty scales like sqrt(N) / N and will likely be much smaller than the systematic uncertainty. Although it will still be a huge pain to make submissions.", "votes": -2 }, { "id": 130111, "postDate": "2016-08-03T23:20:58.513Z", "content": "Hello princelySid,
\n\nI wrote in more detail about sampling here: https://mtalavera.wordpress.com/2016/07/18/kaggle-grupo-bimbo-preparing-the-data/
\n\nIn a nutshell, when I run this:
\n\nhead -1 test.csv > test_sample.csv && tail -n +2 test.csv | shuf -n 349963 >> test_sample.csv
\n\nI am first grabbing the header of the original file and creating a new file, then get all the rows from the original and grab a random 349963 (5%) rows in this case. I do the same for the train set. I then use this sample file instead of original. Command is written in such a way that you get a different sample each time you run it.
\n\nSay you do your analysis using few iterations of sampling as described, once you find the best performing solution, you can always run this on the whole set. For this, you will definitely need to resort to better firepower than a 4GB machine. Perhaps spin up a box at Amazon or similar?
\n\nBest of luck,
\n\nMario
", "rawMarkdown": "Hello princelySid,\r\n\r\nI wrote in more detail about sampling here: https://mtalavera.wordpress.com/2016/07/18/kaggle-grupo-bimbo-preparing-the-data/\r\n\r\nIn a nutshell, when I run this:\r\n\r\nhead -1 test.csv > test_sample.csv && tail -n +2 test.csv | shuf -n 349963 >> test_sample.csv\r\n\r\nI am first grabbing the header of the original file and creating a new file, then get all the rows from the original and grab a random 349963 (5%) rows in this case. I do the same for the train set. I then use this sample file instead of original. Command is written in such a way that you get a different sample each time you run it.\r\n\r\nSay you do your analysis using few iterations of sampling as described, once you find the best performing solution, you can always run this on the whole set. For this, you will definitely need to resort to better firepower than a 4GB machine. Perhaps spin up a box at Amazon or similar?\r\n\r\nBest of luck,\r\n\r\nMario\r\n" }, { "id": 130109, "postDate": "2016-08-03T22:06:47.463Z", "content": "Seems I'm not the only one struggling with the size of this data. My laptop only has about 4gb of memory.:(
\nThis is my first competition so I have a couple of questions:
[quote=CarltonChen;122987]
\n\n[quote=Jared Turkewitz;122978]
\n\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.
\n\n[/quote]
\n\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.
\n\n[/quote]\nsame is the case with me. i am struck at how to handle this dataset.
", "rawMarkdown": "[quote=CarltonChen;122987]\r\n\r\n[quote=Jared Turkewitz;122978]\r\n\r\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.\r\n\r\n[/quote]\r\n\r\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.\r\n\r\n[/quote]\r\nsame is the case with me. i am struck at how to handle this dataset." }, { "id": 124402, "postDate": "2016-06-18T03:35:48.177Z", "content": "[quote=FredR;123584]
\n\nThis is the first competition I've been active in and yea, my computer can't handle it at allll. It's not even a bad computer. I'm gonna see what I can do with my limited resources and then, like ddrbcn suggested, use aws (the EC2 section) to see what it's like if I try my methods on a computer that can handle it. Shame that I'll have to waste money on this but what can yuh do?
\n\n[/quote]
\n\nAny suggestions of tutorials on how to setup a datascience environment on AWS?
", "rawMarkdown": "[quote=FredR;123584]\r\n\r\nThis is the first competition I've been active in and yea, my computer can't handle it at allll. It's not even a bad computer. I'm gonna see what I can do with my limited resources and then, like ddrbcn suggested, use aws (the EC2 section) to see what it's like if I try my methods on a computer that can handle it. Shame that I'll have to waste money on this but what can yuh do?\r\n\r\n[/quote]\r\n\r\nAny suggestions of tutorials on how to setup a datascience environment on AWS?" }, { "id": 124040, "postDate": "2016-06-15T06:01:30.020Z", "content": "I beleive H2O package (for R users) can also prove a nice option for such huge datasets. I've seen kaggelers using H2O in one of the previous competitions.
\n\nEven, someone has posted some initial solution of this ongoing competition in scripts section.
", "rawMarkdown": "I beleive H2O package (for R users) can also prove a nice option for such huge datasets. I've seen kaggelers using H2O in one of the previous competitions.\r\n\r\nEven, someone has posted some initial solution of this ongoing competition in scripts section." }, { "id": 123648, "postDate": "2016-06-13T09:32:31.377Z", "content": "yea I am also going to downsample to have a go in this competition :-( sad I wasted a lot of time but It didnt built the model
", "rawMarkdown": "yea I am also going to downsample to have a go in this competition :-( sad I wasted a lot of time but It didnt built the model " }, { "id": 123606, "postDate": "2016-06-13T04:44:56.837Z", "content": "Hi, I think we can use command line tools to explore the data and possibly use a single node hadoop or spark deployment to make use of its programming model which allows you to handle more data than your RAM supports, which in a general framework will surely fail due to out of memory errors...
", "rawMarkdown": "Hi, I think we can use command line tools to explore the data and possibly use a single node hadoop or spark deployment to make use of its programming model which allows you to handle more data than your RAM supports, which in a general framework will surely fail due to out of memory errors..." }, { "id": 124408, "postDate": "2016-06-18T04:50:29.713Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 123584, "postDate": "2016-06-12T21:39:46.667Z", "rawMarkdown": "", "isDeleted": true }, { "id": 122987, "postDate": "2016-06-09T01:53:57.677Z", "rawMarkdown": "", "isDeleted": true }, { "id": 124410, "postDate": "2016-06-18T06:44:08.907Z", "content": "Thank you @FredR
", "rawMarkdown": "Thank you @FredR" } ], "comments": [ { "id": 122978, "author_name": "Jared Turkewitz", "author_url": "", "post_date": "2016-06-08T23:43:08.383000", "content": "I wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.
", "votes": 9, "replies": [] }, { "id": 123737, "author_name": "Dmitriy Guller", "author_url": "", "post_date": "2016-06-13T19:41:01.520000", "content": "The problem with medium sized datasets is that you can just XGBoost them to death. These kinds of competitions at least introduce some variety into ultimate solutions.
", "votes": 5, "replies": [] }, { "id": 123698, "author_name": "Scirpus", "author_url": "", "post_date": "2016-06-13T15:17:57.447000", "content": "If you believe that the latest client behaviour in the train set will be more valuable to predict the behaviour in the test set why don't you keep the last x entries per user - this will significantly reduce the size of the data. You can check this by initially trying to predict on the last week of the training data. Big data isn't an issue if you come up with a way of making it a small data problem - it certainly makes people think rather than going "100,000 rows? - xgb here I come!" ;)
", "votes": 5, "replies": [] }, { "id": 123646, "author_name": "olivier", "author_url": "", "post_date": "2016-06-13T09:02:44.293000", "content": "I think we'll have to get used to big data sort of issues... and running into hardware limitations will become a daily headache for most of us. \nI have 4 cores/16 GB memory and the last Expedia competition was a nice experience with XGBoost running for hours... It was like working in the punched card era where error codes are a killer ;-))\nMy advice would be to not use RFs or XTCs on the entire data set or you will run into disk swapping issues but you should be able to use XGBoost or vowpal wabbit. The latter is really fast for online learning tasks. Looking into spark is another option.\nIn any case downsampling is your best friend.\nGood luck
", "votes": 5, "replies": [] }, { "id": 122971, "author_name": "SteveKane", "author_url": "", "post_date": "2016-06-08T22:50:39.050000", "content": "people have won with less
", "votes": 5, "replies": [] }, { "id": 130010, "author_name": "Mario Talavera", "author_url": "", "post_date": "2016-08-03T10:36:49.590000", "content": "Hello everybody,
\nI just wanted to share. I am an absolute beginner and faced the same issues about hardware limitations talked about here.
I was away from main computer (on a nice vacation) and had with me but an old core 2 lappy with 4 GB.
\n\nI just decided to sample the test and train files in order to be able to play with the data. The gist follows; it was enough to help me wrap my head around the data until I had a more strudy system to fiddle around in.
\n\n$ head -1 test.csv > test_sample.csv && tail -n +2 test.csv | shuf -n 349963 >> test_sample.csv\n$ head -1 train.csv > train_sample.csv && tail -n +2 train.csv | shuf -n 3709023 >> train_sample.csv
\n\nI think I decided to use 5% of data so that laptop wouldn't blow up. Good luck.
\n\nYIKES, now I see how old these posts are... Still posting so it may help someone in the future.
", "votes": 3, "replies": [] }, { "id": 123004, "author_name": "happycube", "author_url": "", "post_date": "2016-06-09T05:21:41.573000", "content": "Yup, you can definitely get a lot of understanding with prefiltered data, or even just parts of the data. If you've got enough of it figured out with partial data, it's probably worth it to spin up an EC2 if you have to.
\n\nGert kicked butt on Rossman with just a laptop with limited/no net access, for instance... http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/
\n\nIn any case you should be able to get pretty far with 16GB.
\n\nAlso, if you're willing to use old hardware, large memory systems are actually pretty cheap ;) I was able (thanks to a local sale) get a dual E5405 Xeon last week for $40, and got it to 32GB for $35 (and then added a GF770 I already had)
", "votes": 4, "replies": [] }, { "id": 125357, "author_name": "martinelli", "author_url": "", "post_date": "2016-06-28T21:35:35.833000", "content": "[quote=Bindukumar J;125300]
\n\n[quote=CarltonChen;122987]
\n\n[quote=Jared Turkewitz;122978]
\n\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.
\n\n[/quote]
\n\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.
\n\n[/quote]\nsame is the case with me. i am struck at how to handle this dataset.
\n\n[/quote]
\n\nFor me this has been the greatest challenge of this competition.
\n\nThe path I've taken is to build a random 25% sample of train.csv. This random sample is further split between train and test datasets. All training is done on that 25%.
\n\nAlthough still slow, I am able to work with this on my 8GB RAM laptop.
", "votes": 1, "replies": [] }, { "id": 123604, "author_name": "Toulouse", "author_url": "", "post_date": "2016-06-13T04:22:09.047000", "content": "Hi!\nWith feature engineering and merging the size of the datasets can anyway be much bigger than the 3 GB at the end ...
", "votes": 1, "replies": [] }, { "id": 123547, "author_name": "ddrbcn", "author_url": "", "post_date": "2016-06-12T16:22:04.233000", "content": "Try AWS or some similar tool.
", "votes": 1, "replies": [] }, { "id": 123040, "author_name": "Beluga", "author_url": "", "post_date": "2016-06-09T10:29:03.660000", "content": "Hi,
\n\nI will try to sample 10-20% of data from Train.csv for cross-validation analysis. Then go into the cloud to model on the whole training data. Does that make sense?\nI have 32GB machine and it takes just too long to work on the whole set. Have not tried Random Forest or something but this should kill it.
", "votes": 1, "replies": [] }, { "id": 124590, "author_name": "olivier", "author_url": "", "post_date": "2016-06-20T06:19:35.780000", "content": "If you want a hassle free setup on AWS, you may want to use docker on a linux EC2. You will then be able to import docker images created by kaggle teams and available on docker hub. This way you replicate any Kaggle environment (python, R, Julia...) with all the good tools like xgboost, theano already there for you. If you want to know more on docker containers you can have a look at this post on kaggle's blog.\nHope this helps.
", "votes": 2, "replies": [] }, { "id": 123742, "author_name": "Jared Turkewitz", "author_url": "", "post_date": "2016-06-13T20:25:04.683000", "content": "[quote=Dmitriy Guller;123737]
\n\nThe problem with medium sized datasets is that you can just XGBoost them to death. These kinds of competitions at least introduce some variety into ultimate solutions.
\n\n[/quote]
\n\nBut datasets of this size or bigger also get xgboosted to death by the top solutions, it just requires more patience and better hardware. This dataset is actually not big in the sense of fitting on one machine, it just requires a good machine. The main advantage I see is that it is less likely to be scripted to death because the script run time is too long.
", "votes": 2, "replies": [] }, { "id": 123610, "author_name": "Jared Turkewitz", "author_url": "", "post_date": "2016-06-13T05:04:53.407000", "content": "[quote=CarltonChen;122987]
\n\n[quote=Jared Turkewitz;122978]
\n\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.
\n\n[/quote]
\n\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.
\n\n[/quote]
\n\nThe problem is that it isn't about knowledge, it is about hardware. While AWS is a potential solution, it is very expensive for individual use. With that said, you can probably just scale the dataset dramatically down without losing that much accuracy, because percent statistical uncertainty scales like sqrt(N) / N and will likely be much smaller than the systematic uncertainty. Although it will still be a huge pain to make submissions.
", "votes": -2, "replies": [] }, { "id": 130111, "author_name": "Mario Talavera", "author_url": "", "post_date": "2016-08-03T23:20:58.513000", "content": "Hello princelySid,
\n\nI wrote in more detail about sampling here: https://mtalavera.wordpress.com/2016/07/18/kaggle-grupo-bimbo-preparing-the-data/
\n\nIn a nutshell, when I run this:
\n\nhead -1 test.csv > test_sample.csv && tail -n +2 test.csv | shuf -n 349963 >> test_sample.csv
\n\nI am first grabbing the header of the original file and creating a new file, then get all the rows from the original and grab a random 349963 (5%) rows in this case. I do the same for the train set. I then use this sample file instead of original. Command is written in such a way that you get a different sample each time you run it.
\n\nSay you do your analysis using few iterations of sampling as described, once you find the best performing solution, you can always run this on the whole set. For this, you will definitely need to resort to better firepower than a 4GB machine. Perhaps spin up a box at Amazon or similar?
\n\nBest of luck,
\n\nMario
", "votes": 0, "replies": [] }, { "id": 130109, "author_name": "princelySid", "author_url": "", "post_date": "2016-08-03T22:06:47.463000", "content": "Seems I'm not the only one struggling with the size of this data. My laptop only has about 4gb of memory.:(
\nThis is my first competition so I have a couple of questions:
[quote=CarltonChen;122987]
\n\n[quote=Jared Turkewitz;122978]
\n\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.
\n\n[/quote]
\n\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.
\n\n[/quote]\nsame is the case with me. i am struck at how to handle this dataset.
", "votes": 0, "replies": [] }, { "id": 124402, "author_name": "martinelli", "author_url": "", "post_date": "2016-06-18T03:35:48.177000", "content": "[quote=FredR;123584]
\n\nThis is the first competition I've been active in and yea, my computer can't handle it at allll. It's not even a bad computer. I'm gonna see what I can do with my limited resources and then, like ddrbcn suggested, use aws (the EC2 section) to see what it's like if I try my methods on a computer that can handle it. Shame that I'll have to waste money on this but what can yuh do?
\n\n[/quote]
\n\nAny suggestions of tutorials on how to setup a datascience environment on AWS?
", "votes": 0, "replies": [] }, { "id": 124040, "author_name": "Neeraj", "author_url": "", "post_date": "2016-06-15T06:01:30.020000", "content": "I beleive H2O package (for R users) can also prove a nice option for such huge datasets. I've seen kaggelers using H2O in one of the previous competitions.
\n\nEven, someone has posted some initial solution of this ongoing competition in scripts section.
", "votes": 0, "replies": [] }, { "id": 123648, "author_name": "DANYAL KHALIQ", "author_url": "", "post_date": "2016-06-13T09:32:31.377000", "content": "yea I am also going to downsample to have a go in this competition :-( sad I wasted a lot of time but It didnt built the model
", "votes": 0, "replies": [] }, { "id": 123606, "author_name": "AnandKumar", "author_url": "", "post_date": "2016-06-13T04:44:56.837000", "content": "Hi, I think we can use command line tools to explore the data and possibly use a single node hadoop or spark deployment to make use of its programming model which allows you to handle more data than your RAM supports, which in a general framework will surely fail due to out of memory errors...
", "votes": 0, "replies": [] }, { "id": 124408, "author_name": "", "author_url": "", "post_date": "2016-06-18T04:50:29.713000", "content": "", "votes": 1, "replies": [] }, { "id": 123584, "author_name": "", "author_url": "", "post_date": "2016-06-12T21:39:46.667000", "content": "", "votes": 0, "replies": [] }, { "id": 122987, "author_name": "", "author_url": "", "post_date": "2016-06-09T01:53:57.677000", "content": "", "votes": 0, "replies": [] }, { "id": 124410, "author_name": "martinelli", "author_url": "", "post_date": "2016-06-18T06:44:08.907000", "content": "Thank you @FredR
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "122978": "I wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.", "123737": "The problem with medium sized datasets is that you can just XGBoost them to death. These kinds of competitions at least introduce some variety into ultimate solutions.", "123698": "If you believe that the latest client behaviour in the train set will be more valuable to predict the behaviour in the test set why don't you keep the last x entries per user - this will significantly reduce the size of the data. You can check this by initially trying to predict on the last week of the training data. Big data isn't an issue if you come up with a way of making it a small data problem - it certainly makes people think rather than going \"100,000 rows? - xgb here I come!\" ;) ", "123646": "I think we'll have to get used to big data sort of issues... and running into hardware limitations will become a daily headache for most of us. \r\nI have 4 cores/16 GB memory and the last Expedia competition was a nice experience with XGBoost running for hours... It was like working in the punched card era where error codes are a killer ;-))\r\nMy advice would be to not use RFs or XTCs on the entire data set or you will run into disk swapping issues but you should be able to use XGBoost or vowpal wabbit. The latter is really fast for online learning tasks. Looking into spark is another option.\r\nIn any case downsampling is your best friend.\r\nGood luck", "122971": "people have won with less", "122964": "I am thinking of joining this competition, but unsure whether my computer will be able to run effective models given the large size of the dataset.\r\n\r\nI wonder how much memory and processing capacity would be enough?\r\n\r\nPS. Currently I have 4 cores and 16GB RAM.", "130010": "Hello everybody, \r\nI just wanted to share. I am an absolute beginner and faced the same issues about hardware limitations talked about here.\r\n\r\nI was away from main computer (on a nice vacation) and had with me but an old core 2 lappy with 4 GB.\r\n\r\nI just decided to sample the test and train files in order to be able to play with the data. The gist follows; it was enough to help me wrap my head around the data until I had a more strudy system to fiddle around in. \r\n\r\n$ head -1 test.csv > test_sample.csv && tail -n +2 test.csv | shuf -n 349963 >> test_sample.csv\r\n$ head -1 train.csv > train_sample.csv && tail -n +2 train.csv | shuf -n 3709023 >> train_sample.csv\r\n\r\nI think I decided to use 5% of data so that laptop wouldn't blow up. Good luck.\r\n\r\nYIKES, now I see how old these posts are... Still posting so it may help someone in the future.", "123004": "Yup, you can definitely get a lot of understanding with prefiltered data, or even just parts of the data. If you've got enough of it figured out with partial data, it's probably worth it to spin up an EC2 if you have to.\r\n\r\nGert kicked butt on Rossman with just a laptop with limited/no net access, for instance... http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/\r\n\r\nIn any case you should be able to get pretty far with 16GB.\r\n\r\nAlso, if you're willing to use old hardware, large memory systems are actually pretty cheap ;) I was able (thanks to a local sale) get a dual E5405 Xeon last week for $40, and got it to 32GB for $35 (and then added a GF770 I already had)", "125357": "[quote=Bindukumar J;125300]\r\n\r\n[quote=CarltonChen;122987]\r\n\r\n[quote=Jared Turkewitz;122978]\r\n\r\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.\r\n\r\n[/quote]\r\n\r\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.\r\n\r\n[/quote]\r\nsame is the case with me. i am struck at how to handle this dataset.\r\n\r\n[/quote]\r\n\r\nFor me this has been the greatest challenge of this competition.\r\n\r\nThe path I've taken is to build a random 25% sample of train.csv. This random sample is further split between train and test datasets. All training is done on that 25%.\r\n\r\nAlthough still slow, I am able to work with this on my 8GB RAM laptop.", "123604": "Hi!\r\nWith feature engineering and merging the size of the datasets can anyway be much bigger than the 3 GB at the end ...", "123547": "Try AWS or some similar tool.", "123040": "Hi,\r\n\r\nI will try to sample 10-20% of data from Train.csv for cross-validation analysis. Then go into the cloud to model on the whole training data. Does that make sense?\r\nI have 32GB machine and it takes just too long to work on the whole set. Have not tried Random Forest or something but this should kill it.\r\n", "124590": "If you want a hassle free setup on AWS, you may want to use [docker on a linux EC2][1]. You will then be able to import docker images created by kaggle teams and available on [docker hub][2]. This way you replicate any Kaggle environment (python, R, Julia...) with all the good tools like xgboost, theano already there for you. If you want to know more on docker containers you can have a look at [this post][3] on kaggle's blog.\r\nHope this helps.\r\n\r\n\r\n [1]: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html\r\n [2]: https://hub.docker.com/u/kaggle/\r\n [3]: http://blog.kaggle.com/2016/02/05/how-to-get-started-with-data-science-in-containers/", "123742": "[quote=Dmitriy Guller;123737]\r\n\r\nThe problem with medium sized datasets is that you can just XGBoost them to death. These kinds of competitions at least introduce some variety into ultimate solutions.\r\n\r\n[/quote]\r\n\r\nBut datasets of this size or bigger also get xgboosted to death by the top solutions, it just requires more patience and better hardware. This dataset is actually not big in the sense of fitting on one machine, it just requires a good machine. The main advantage I see is that it is less likely to be scripted to death because the script run time is too long.", "123610": "[quote=CarltonChen;122987]\r\n\r\n[quote=Jared Turkewitz;122978]\r\n\r\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.\r\n\r\n[/quote]\r\n\r\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.\r\n\r\n[/quote]\r\n\r\nThe problem is that it isn't about knowledge, it is about hardware. While AWS is a potential solution, it is very expensive for individual use. With that said, you can probably just scale the dataset dramatically down without losing that much accuracy, because percent statistical uncertainty scales like sqrt(N) / N and will likely be much smaller than the systematic uncertainty. Although it will still be a huge pain to make submissions.", "130111": "Hello princelySid,\r\n\r\nI wrote in more detail about sampling here: https://mtalavera.wordpress.com/2016/07/18/kaggle-grupo-bimbo-preparing-the-data/\r\n\r\nIn a nutshell, when I run this:\r\n\r\nhead -1 test.csv > test_sample.csv && tail -n +2 test.csv | shuf -n 349963 >> test_sample.csv\r\n\r\nI am first grabbing the header of the original file and creating a new file, then get all the rows from the original and grab a random 349963 (5%) rows in this case. I do the same for the train set. I then use this sample file instead of original. Command is written in such a way that you get a different sample each time you run it.\r\n\r\nSay you do your analysis using few iterations of sampling as described, once you find the best performing solution, you can always run this on the whole set. For this, you will definitely need to resort to better firepower than a 4GB machine. Perhaps spin up a box at Amazon or similar?\r\n\r\nBest of luck,\r\n\r\nMario\r\n", "130109": "Seems I'm not the only one struggling with the size of this data. My laptop only has about 4gb of memory.:( \r\nThis is my first competition so I have a couple of questions:\r\n\r\n - It anyone else doing this in python? Most of the solutions seem to be in R\r\n - If you're to sample what's the best way to do that? \r\n - What's the best sample size? How is this determined?", "125300": "[quote=CarltonChen;122987]\r\n\r\n[quote=Jared Turkewitz;122978]\r\n\r\nI wish Kaggle would stop using datasets that are needlessly large like this one looks to be. There is usually not a good reason to have over 1 million rows for a Kaggle competition, as this increases the amount of time spent waiting without significantly decreasing the statistical uncertainty.\r\n\r\n[/quote]\r\n\r\nI think big datasets can block people who do not really know how to do data science... like me... I spent a lot of time to search how to manipulate the over 3 GB dataset.\r\n\r\n[/quote]\r\nsame is the case with me. i am struck at how to handle this dataset.", "124402": "[quote=FredR;123584]\r\n\r\nThis is the first competition I've been active in and yea, my computer can't handle it at allll. It's not even a bad computer. I'm gonna see what I can do with my limited resources and then, like ddrbcn suggested, use aws (the EC2 section) to see what it's like if I try my methods on a computer that can handle it. Shame that I'll have to waste money on this but what can yuh do?\r\n\r\n[/quote]\r\n\r\nAny suggestions of tutorials on how to setup a datascience environment on AWS?", "124040": "I beleive H2O package (for R users) can also prove a nice option for such huge datasets. I've seen kaggelers using H2O in one of the previous competitions.\r\n\r\nEven, someone has posted some initial solution of this ongoing competition in scripts section.", "123648": "yea I am also going to downsample to have a go in this competition :-( sad I wasted a lot of time but It didnt built the model ", "123606": "Hi, I think we can use command line tools to explore the data and possibly use a single node hadoop or spark deployment to make use of its programming model which allows you to handle more data than your RAM supports, which in a general framework will surely fail due to out of memory errors...", "124408": "", "123584": "", "122987": "", "124410": "Thank you @FredR" } }