Hello,
\n\nI think my PC is pretty good. (i7-6700k, 48GB RAM)
\n\nBut sometimes,... is not enough to analyze large data.
\n\nOf course, a good idea is the greatest wealth. (not hardware)
\n\nI'm just wondering, Is that enough to get a good grade on my computer?
\n\nThanks.
", "messages": [ { "id": "137097", "postDate": "09/28/2016 06:45:27", "content": "Hello,
\n\nI think my PC is pretty good. (i7-6700k, 48GB RAM)
\n\nBut sometimes,... is not enough to analyze large data.
\n\nOf course, a good idea is the greatest wealth. (not hardware)
\n\nI'm just wondering, Is that enough to get a good grade on my computer?
\n\nThanks.
", "rawMarkdown": "Hello,\r\n\r\nI think my PC is pretty good. (i7-6700k, 48GB RAM) \r\n\r\nBut sometimes,... is not enough to analyze large data.\r\n\r\nOf course, a good idea is the greatest wealth. (not hardware)\r\n\r\nI'm just wondering, Is that enough to get a good grade on my computer?\r\n\r\nThanks.", "votes": null }, { "id": "137106", "postDate": "09/28/2016 08:22:26", "content": "Me: notebook, 32GB RAM.\nI hope the memory is larger, though :(
", "rawMarkdown": "Me: notebook, 32GB RAM.\r\nI hope the memory is larger, though :(", "votes": null }, { "id": "137123", "postDate": "09/28/2016 12:51:02", "content": "It is, I am using 8GB ;)
", "rawMarkdown": "It is, I am using 8GB ;)", "votes": null }, { "id": "137133", "postDate": "09/28/2016 14:57:24", "content": "Maybe you should go for 480GB instead of 48GB so you can play Overwatch and stream it while your code is running (also a couple GTX 1080s). Supersize me please. Then again you can get above 0.4 on 8GB.
", "rawMarkdown": "Maybe you should go for 480GB instead of 48GB so you can play Overwatch and stream it while your code is running (also a couple GTX 1080s). Supersize me please. Then again you can get above 0.4 on 8GB.", "votes": null }, { "id": "137144", "postDate": "09/28/2016 16:48:25", "content": "I definitely feel the pinch at 48GB for what I'm doing now, but as you can see above me better Kagglers are making do with much less ;) So I'll definitely need to improve my code.
\n\nIn your case desktop Skylake is limited to 64GB, and it would cost $120 to get there, so it's probably not worth an upgrade. If you're running Linux look into using zram to do in-memory compressed swap, that can help.
", "rawMarkdown": "I definitely feel the pinch at 48GB for what I'm doing now, but as you can see above me better Kagglers are making do with much less ;) So I'll definitely need to improve my code.\r\n\r\nIn your case desktop Skylake is limited to 64GB, and it would cost $120 to get there, so it's probably not worth an upgrade. If you're running Linux look into using zram to do in-memory compressed swap, that can help.", "votes": null }, { "id": "137156", "postDate": "09/28/2016 18:59:16", "content": "All of our code runs in 48GB a-ok. That's enough to get a good score, although you might be waiting a little long to run models :)
\n\n@Mike: Does your model run in 8GB? That's impressive
", "rawMarkdown": "All of our code runs in 48GB a-ok. That's enough to get a good score, although you might be waiting a little long to run models :)\r\n\r\n@Mike: Does your model run in 8GB? That's impressive", "votes": null }, { "id": "137160", "postDate": "09/28/2016 21:58:51", "content": "@NxGTR: You make me feel ashamed. ^^. I will try harder.
\n\n@Mike : I don`t play any games, I just enjoy data analysis.
\n\n@anokas : Thank you for reply. I just heard that your model run in 128GB, so i think it is hard to get good score on my PC. But I misunderstood. your feedback helpful for me. I do my best.
", "rawMarkdown": "NxGTR: You make me feel ashamed. ^^. I will try harder.\r\n\r\n@Mike : I don`t play any games, I just enjoy data analysis. \r\n\r\n@anokas : Thank you for reply. I just heard that your model run in 128GB, so i think it is hard to get good score on my PC. But I misunderstood. your feedback helpful for me. I do my best.", "votes": null }, { "id": "137198", "postDate": "09/29/2016 08:18:51", "content": "[quote=NxGTR;137123]
\n\nIt is, I am using 8GB ;)
\n\n[/quote]
\n\nHi NxGTR,
\n\nCan you kindly share some tips / tricks on how to handle these large volumes of Data on 8 GB Machine? I have an 8GB machine and had real problems with Grupo and similar competitions.
\n\nI use Python when I need to use Chunking (have not found any reliable method on R).
\n\nWhen I need to do Data Manipulation that needs say the whole datasets for columns and chunking is not an option, I resort to R ff and ffdf packages and disk based approaches (not sure how to achieve using Python).
\n\nThese processes are very slow. For Grupo, the entire Data Manipulation for me used to take upto 6-8 days for generating some basic statistical features.
\n\nI have also tried SQLlite, but that does not improve the speed either.
\n\nAny suggestion tips will be of great help.
\n\nThanks and regards\nSR
", "rawMarkdown": "[quote=NxGTR;137123]\r\n\r\nIt is, I am using 8GB ;)\r\n\r\n[/quote]\r\n\r\nHi NxGTR,\r\n\r\nCan you kindly share some tips / tricks on how to handle these large volumes of Data on 8 GB Machine? I have an 8GB machine and had real problems with Grupo and similar competitions.\r\n\r\nI use Python when I need to use Chunking (have not found any reliable method on R). \r\n\r\nWhen I need to do Data Manipulation that needs say the whole datasets for columns and chunking is not an option, I resort to R ff and ffdf packages and disk based approaches (not sure how to achieve using Python).\r\n\r\nThese processes are very slow. For Grupo, the entire Data Manipulation for me used to take upto 6-8 days for generating some basic statistical features.\r\n\r\nI have also tried SQLlite, but that does not improve the speed either.\r\n\r\nAny suggestion tips will be of great help.\r\n\r\nThanks and regards\r\nSR", "votes": null }, { "id": "137206", "postDate": "09/29/2016 09:01:30", "content": "I am far from the top, but I am also using 8 GB of RAM. I don't know what the experts are doing but I am training my model on part of the data, not all of it... There is a kernel here that shows that you don't need to use the entire train data for it to be representative.
\n\nI agree that I would use more data if I had more RAM because that could definitely simplify my model and potentially give me a higher score, but I don't think that's the factor limiting my performance in this competition.
", "rawMarkdown": "I am far from the top, but I am also using 8 GB of RAM. I don't know what the experts are doing but I am training my model on part of the data, not all of it... There is a kernel [here][1] that shows that you don't need to use the entire train data for it to be representative.\r\n\r\nI agree that I would use more data if I had more RAM because that could definitely simplify my model and potentially give me a higher score, but I don't think that's the factor limiting my performance in this competition.\r\n\r\n\r\n [1]: https://www.kaggle.com/juliancienfuegos/bosch-production-line-performance/finding-the-smallest-minimum-sample/code", "votes": null }, { "id": "137246", "postDate": "09/29/2016 15:20:25", "content": "[quote=SR;137198]\nHi NxGTR,\nCan you kindly share some tips / tricks on how to handle these large volumes of Data on 8 GB Machine? \n[/quote]
\n\nThere are no fancy tricks, the key is to avoid using expensive libs (Pandas, or whatever), just use the basics, for instance, I have some routines in Python, do you know how many imports I have? 0
\n\nGive it a try. It works for any language, the more libs you use, the more memory hungry your solution will be as they are designed for "generic problems" not for THIS problem.
", "rawMarkdown": "[quote=SR;137198]\r\nHi NxGTR,\r\nCan you kindly share some tips / tricks on how to handle these large volumes of Data on 8 GB Machine? \r\n[/quote]\r\n\r\nThere are no fancy tricks, the key is to avoid using expensive libs (Pandas, or whatever), just use the basics, for instance, I have some routines in Python, do you know how many imports I have? 0 \r\n\r\nGive it a try. It works for any language, the more libs you use, the more memory hungry your solution will be as they are designed for \"generic problems\" not for THIS problem.", "votes": null }, { "id": "137253", "postDate": "09/29/2016 16:41:20", "content": "I'm struggling a bit on memory with about 50GB in RAM from AWS. Although I have not done much to be memory efficient yet. I presume to use the current data set on an 8GB machine like NxGTR and Mike mentioned a subset of columns is used, especially for the date file (raw files loaded and row binded is 12.9GB in data.table for me, sparse not a whole lot less).
\nOn the categorical file I checked on about 200K rows and after removing constant columns and duplicates it was reduced down to about 190 columns.
If you are running xgboost, then you can convert you csv to a libsvm format which allows your fit the complete numeric dataset in your memory with 8 GB without any slowdowns. That's one approach you can do.
", "rawMarkdown": "If you are running xgboost, then you can convert you csv to a libsvm format which allows your fit the complete numeric dataset in your memory with 8 GB without any slowdowns. That's one approach you can do.", "votes": null }, { "id": "137262", "postDate": "09/29/2016 17:50:08", "content": "I'm using R and a 48GB machine to build my current solution. So you need to be memory efficient ;-D
", "rawMarkdown": "I'm using R and a 48GB machine to build my current solution. So you need to be memory efficient ;-D", "votes": null }, { "id": "137264", "postDate": "09/29/2016 17:58:30", "content": "[quote=Gilberto Titericz Junior;137262]
\n\nI'm using R and a 48GB machine to build my current solution. So you need to be memory efficient ;-D
\n\n[/quote]
\n\nCurious. How long does it take for your model to run?
", "rawMarkdown": "[quote=Gilberto Titericz Junior;137262]\r\n\r\nI'm using R and a 48GB machine to build my current solution. So you need to be memory efficient ;-D\r\n\r\n[/quote]\r\n\r\nCurious. How long does it take for your model to run?", "votes": null }, { "id": "137445", "postDate": "10/01/2016 17:53:42", "content": "BTW Scirpus's script is not only very good, but fits in ~4GB. He uses pandas chunking and filters out select columns which allows it to fit into low memory even as dense matrices.
\n\nedit: Looking at things a bit more, definitely use a reduced set of features, you can refine your models even with 8-16GB of memory that way. The set of features in Scirpus's script (+ the two I mention) make a solid starter IMO and are good for >.24 even without probability conversion.
\n\nHowever, i expect the silver medal threshold to be at least 0.4 by the time this competition ends...
", "rawMarkdown": "BTW Scirpus's script is not only very good, but fits in ~4GB. He uses pandas chunking and filters out select columns which allows it to fit into low memory even as dense matrices.\r\n\r\nedit: Looking at things a bit more, *definitely* use a reduced set of features, you can refine your models even with 8-16GB of memory that way. The set of features in Scirpus's script (+ the two I mention) make a solid starter IMO and are good for >.24 even without probability conversion.\r\n\r\nHowever, i expect the silver medal threshold to be *at least* 0.4 by the time this competition ends...", "votes": null }, { "id": "137477", "postDate": "10/01/2016 22:28:58", "content": "Hi @niks2000,
\n\nThere are many ways to handle using a small RAM memory.
\n\nOf course, you will need more time and, if possible, a good processor.
\n\nHave a look at buffering techniques.
\n\nRegards
", "rawMarkdown": "Hi @niks2000,\r\n\r\nThere are many ways to handle using a small RAM memory.\r\n\r\nOf course, you will need more time and, if possible, a good processor.\r\n \r\nHave a look at buffering techniques.\r\n\r\nRegards", "votes": null }, { "id": "137495", "postDate": "10/02/2016 04:00:39", "content": "Hi, I am a novice. This is my second project. I cant even see the data of training or testing sets. All of the comments above are very high level information for me. Can someone tell/help me with how to approach this big data sets?
", "rawMarkdown": "Hi, I am a novice. This is my second project. I cant even see the data of training or testing sets. All of the comments above are very high level information for me. Can someone tell/help me with how to approach this big data sets?", "votes": null }, { "id": "137535", "postDate": "10/02/2016 16:53:25", "content": "@Apoorva
\n\nI am only a beginner myself and have neither a good machine or a lot of experience.
\n\nFor the average human being with a computer of 8-16GB of memory, you will probably need to analyze the data in chunks like this published script:
\n\n\n\nDon't worry about trying to "win" try and understand what the script is doing and why--the author does a nice job explaining what is going on. As you begin to understand what is going on you can tweak the script with your own ideas. Just keep building piece by piece and eventually you catch on.
", "rawMarkdown": "Apoorva\r\n\r\nI am only a beginner myself and have neither a good machine or a lot of experience.\r\n\r\nFor the average human being with a computer of 8-16GB of memory, you will probably need to analyze the data in chunks like this published script:\r\n\r\nhttps://www.kaggle.com/joconnor/bosch-production-line-performance/python-xgboost-starter-0-209-public-mcc\r\n\r\nDon't worry about trying to \"win\" try and understand what the script is doing and why--the author does a nice job explaining what is going on. As you begin to understand what is going on you can tweak the script with your own ideas. Just keep building piece by piece and eventually you catch on.", "votes": null }, { "id": "137539", "postDate": "10/02/2016 17:10:45", "content": "@Bob, Lord of Cats
\n\nThanks for sharing the link. I ll try to understand these.
", "rawMarkdown": "Bob, Lord of Cats\r\n\r\nThanks for sharing the link. I ll try to understand these.", "votes": null }, { "id": "137559", "postDate": "10/02/2016 20:56:38", "content": "Must... Make... my swap big enough so i can fit the whole dataset and wait forever for the computer to finish modeling swapping 100 pages. Kidding
\n\nTrue asnwer:
\n\nBeware that this will put a lot of pressure on SSD's.. BUT, who really cares?
", "rawMarkdown": "Must... Make... my swap big enough so i can fit the whole dataset and wait forever for the computer to finish modeling swapping 100 pages. Kidding\r\n\r\nTrue asnwer: \r\n\r\n - Set a 100GB swap partition on linux. \r\n - Default swap on W10 is locked to 23GB(at least on my computer). Set custom size.\r\n\r\n\r\nBeware that this will put a lot of pressure on SSD's.. BUT, who really cares?", "votes": null }, { "id": "137572", "postDate": "10/02/2016 22:29:31", "content": "I never used R, but I wonder if cannot use some kind of mini-batches of data?
", "rawMarkdown": "I never used R, but I wonder if cannot use some kind of mini-batches of data?", "votes": null } ], "comments": [ { "id": 137106, "author_name": "nbdyz0", "author_url": "", "post_date": "09/28/2016 08:22:26", "content": "Me: notebook, 32GB RAM.\nI hope the memory is larger, though :(
", "votes": null, "replies": [] }, { "id": 137123, "author_name": "carloshuertas", "author_url": "", "post_date": "09/28/2016 12:51:02", "content": "It is, I am using 8GB ;)
", "votes": null, "replies": [] }, { "id": 137133, "author_name": "mikeskim", "author_url": "", "post_date": "09/28/2016 14:57:24", "content": "Maybe you should go for 480GB instead of 48GB so you can play Overwatch and stream it while your code is running (also a couple GTX 1080s). Supersize me please. Then again you can get above 0.4 on 8GB.
", "votes": null, "replies": [] }, { "id": 137144, "author_name": "happycube", "author_url": "", "post_date": "09/28/2016 16:48:25", "content": "I definitely feel the pinch at 48GB for what I'm doing now, but as you can see above me better Kagglers are making do with much less ;) So I'll definitely need to improve my code.
\n\nIn your case desktop Skylake is limited to 64GB, and it would cost $120 to get there, so it's probably not worth an upgrade. If you're running Linux look into using zram to do in-memory compressed swap, that can help.
", "votes": null, "replies": [] }, { "id": 137156, "author_name": "anokas", "author_url": "", "post_date": "09/28/2016 18:59:16", "content": "All of our code runs in 48GB a-ok. That's enough to get a good score, although you might be waiting a little long to run models :)
\n\n@Mike: Does your model run in 8GB? That's impressive
", "votes": null, "replies": [] }, { "id": 137160, "author_name": "niks2000", "author_url": "", "post_date": "09/28/2016 21:58:51", "content": "@NxGTR: You make me feel ashamed. ^^. I will try harder.
\n\n@Mike : I don`t play any games, I just enjoy data analysis.
\n\n@anokas : Thank you for reply. I just heard that your model run in 128GB, so i think it is hard to get good score on my PC. But I misunderstood. your feedback helpful for me. I do my best.
", "votes": null, "replies": [] }, { "id": 137198, "author_name": "sayantan", "author_url": "", "post_date": "09/29/2016 08:18:51", "content": "[quote=NxGTR;137123]
\n\nIt is, I am using 8GB ;)
\n\n[/quote]
\n\nHi NxGTR,
\n\nCan you kindly share some tips / tricks on how to handle these large volumes of Data on 8 GB Machine? I have an 8GB machine and had real problems with Grupo and similar competitions.
\n\nI use Python when I need to use Chunking (have not found any reliable method on R).
\n\nWhen I need to do Data Manipulation that needs say the whole datasets for columns and chunking is not an option, I resort to R ff and ffdf packages and disk based approaches (not sure how to achieve using Python).
\n\nThese processes are very slow. For Grupo, the entire Data Manipulation for me used to take upto 6-8 days for generating some basic statistical features.
\n\nI have also tried SQLlite, but that does not improve the speed either.
\n\nAny suggestion tips will be of great help.
\n\nThanks and regards\nSR
", "votes": null, "replies": [] }, { "id": 137206, "author_name": "yamiomar88", "author_url": "", "post_date": "09/29/2016 09:01:30", "content": "I am far from the top, but I am also using 8 GB of RAM. I don't know what the experts are doing but I am training my model on part of the data, not all of it... There is a kernel here that shows that you don't need to use the entire train data for it to be representative.
\n\nI agree that I would use more data if I had more RAM because that could definitely simplify my model and potentially give me a higher score, but I don't think that's the factor limiting my performance in this competition.
", "votes": null, "replies": [] }, { "id": 137246, "author_name": "carloshuertas", "author_url": "", "post_date": "09/29/2016 15:20:25", "content": "[quote=SR;137198]\nHi NxGTR,\nCan you kindly share some tips / tricks on how to handle these large volumes of Data on 8 GB Machine? \n[/quote]
\n\nThere are no fancy tricks, the key is to avoid using expensive libs (Pandas, or whatever), just use the basics, for instance, I have some routines in Python, do you know how many imports I have? 0
\n\nGive it a try. It works for any language, the more libs you use, the more memory hungry your solution will be as they are designed for "generic problems" not for THIS problem.
", "votes": null, "replies": [] }, { "id": 137253, "author_name": "darraghdog", "author_url": "", "post_date": "09/29/2016 16:41:20", "content": "I'm struggling a bit on memory with about 50GB in RAM from AWS. Although I have not done much to be memory efficient yet. I presume to use the current data set on an 8GB machine like NxGTR and Mike mentioned a subset of columns is used, especially for the date file (raw files loaded and row binded is 12.9GB in data.table for me, sparse not a whole lot less).
\nOn the categorical file I checked on about 200K rows and after removing constant columns and duplicates it was reduced down to about 190 columns.
If you are running xgboost, then you can convert you csv to a libsvm format which allows your fit the complete numeric dataset in your memory with 8 GB without any slowdowns. That's one approach you can do.
", "votes": null, "replies": [] }, { "id": 137262, "author_name": "titericz", "author_url": "", "post_date": "09/29/2016 17:50:08", "content": "I'm using R and a 48GB machine to build my current solution. So you need to be memory efficient ;-D
", "votes": null, "replies": [] }, { "id": 137264, "author_name": "notaapple", "author_url": "", "post_date": "09/29/2016 17:58:30", "content": "[quote=Gilberto Titericz Junior;137262]
\n\nI'm using R and a 48GB machine to build my current solution. So you need to be memory efficient ;-D
\n\n[/quote]
\n\nCurious. How long does it take for your model to run?
", "votes": null, "replies": [] }, { "id": 137445, "author_name": "happycube", "author_url": "", "post_date": "10/01/2016 17:53:42", "content": "BTW Scirpus's script is not only very good, but fits in ~4GB. He uses pandas chunking and filters out select columns which allows it to fit into low memory even as dense matrices.
\n\nedit: Looking at things a bit more, definitely use a reduced set of features, you can refine your models even with 8-16GB of memory that way. The set of features in Scirpus's script (+ the two I mention) make a solid starter IMO and are good for >.24 even without probability conversion.
\n\nHowever, i expect the silver medal threshold to be at least 0.4 by the time this competition ends...
", "votes": null, "replies": [] }, { "id": 137477, "author_name": "brandao", "author_url": "", "post_date": "10/01/2016 22:28:58", "content": "Hi @niks2000,
\n\nThere are many ways to handle using a small RAM memory.
\n\nOf course, you will need more time and, if possible, a good processor.
\n\nHave a look at buffering techniques.
\n\nRegards
", "votes": null, "replies": [] }, { "id": 137495, "author_name": "apoorvaverma", "author_url": "", "post_date": "10/02/2016 04:00:39", "content": "Hi, I am a novice. This is my second project. I cant even see the data of training or testing sets. All of the comments above are very high level information for me. Can someone tell/help me with how to approach this big data sets?
", "votes": null, "replies": [] }, { "id": 137535, "author_name": "rdslater", "author_url": "", "post_date": "10/02/2016 16:53:25", "content": "@Apoorva
\n\nI am only a beginner myself and have neither a good machine or a lot of experience.
\n\nFor the average human being with a computer of 8-16GB of memory, you will probably need to analyze the data in chunks like this published script:
\n\n\n\nDon't worry about trying to "win" try and understand what the script is doing and why--the author does a nice job explaining what is going on. As you begin to understand what is going on you can tweak the script with your own ideas. Just keep building piece by piece and eventually you catch on.
", "votes": null, "replies": [] }, { "id": 137539, "author_name": "apoorvaverma", "author_url": "", "post_date": "10/02/2016 17:10:45", "content": "@Bob, Lord of Cats
\n\nThanks for sharing the link. I ll try to understand these.
", "votes": null, "replies": [] }, { "id": 137559, "author_name": "abriosi", "author_url": "", "post_date": "10/02/2016 20:56:38", "content": "Must... Make... my swap big enough so i can fit the whole dataset and wait forever for the computer to finish modeling swapping 100 pages. Kidding
\n\nTrue asnwer:
\n\nBeware that this will put a lot of pressure on SSD's.. BUT, who really cares?
", "votes": null, "replies": [] }, { "id": 137572, "author_name": "gadgysaidoff", "author_url": "", "post_date": "10/02/2016 22:29:31", "content": "I never used R, but I wonder if cannot use some kind of mini-batches of data?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "137097": "Hello,\r\n\r\nI think my PC is pretty good. (i7-6700k, 48GB RAM) \r\n\r\nBut sometimes,... is not enough to analyze large data.\r\n\r\nOf course, a good idea is the greatest wealth. (not hardware)\r\n\r\nI'm just wondering, Is that enough to get a good grade on my computer?\r\n\r\nThanks.", "137106": "Me: notebook, 32GB RAM.\r\nI hope the memory is larger, though :(", "137123": "It is, I am using 8GB ;)", "137133": "Maybe you should go for 480GB instead of 48GB so you can play Overwatch and stream it while your code is running (also a couple GTX 1080s). Supersize me please. Then again you can get above 0.4 on 8GB.", "137144": "I definitely feel the pinch at 48GB for what I'm doing now, but as you can see above me better Kagglers are making do with much less ;) So I'll definitely need to improve my code.\r\n\r\nIn your case desktop Skylake is limited to 64GB, and it would cost $120 to get there, so it's probably not worth an upgrade. If you're running Linux look into using zram to do in-memory compressed swap, that can help.", "137156": "All of our code runs in 48GB a-ok. That's enough to get a good score, although you might be waiting a little long to run models :)\r\n\r\n@Mike: Does your model run in 8GB? That's impressive", "137160": "NxGTR: You make me feel ashamed. ^^. I will try harder.\r\n\r\n@Mike : I don`t play any games, I just enjoy data analysis. \r\n\r\n@anokas : Thank you for reply. I just heard that your model run in 128GB, so i think it is hard to get good score on my PC. But I misunderstood. your feedback helpful for me. I do my best.", "137198": "[quote=NxGTR;137123]\r\n\r\nIt is, I am using 8GB ;)\r\n\r\n[/quote]\r\n\r\nHi NxGTR,\r\n\r\nCan you kindly share some tips / tricks on how to handle these large volumes of Data on 8 GB Machine? I have an 8GB machine and had real problems with Grupo and similar competitions.\r\n\r\nI use Python when I need to use Chunking (have not found any reliable method on R). \r\n\r\nWhen I need to do Data Manipulation that needs say the whole datasets for columns and chunking is not an option, I resort to R ff and ffdf packages and disk based approaches (not sure how to achieve using Python).\r\n\r\nThese processes are very slow. For Grupo, the entire Data Manipulation for me used to take upto 6-8 days for generating some basic statistical features.\r\n\r\nI have also tried SQLlite, but that does not improve the speed either.\r\n\r\nAny suggestion tips will be of great help.\r\n\r\nThanks and regards\r\nSR", "137206": "I am far from the top, but I am also using 8 GB of RAM. I don't know what the experts are doing but I am training my model on part of the data, not all of it... There is a kernel [here][1] that shows that you don't need to use the entire train data for it to be representative.\r\n\r\nI agree that I would use more data if I had more RAM because that could definitely simplify my model and potentially give me a higher score, but I don't think that's the factor limiting my performance in this competition.\r\n\r\n\r\n [1]: https://www.kaggle.com/juliancienfuegos/bosch-production-line-performance/finding-the-smallest-minimum-sample/code", "137246": "[quote=SR;137198]\r\nHi NxGTR,\r\nCan you kindly share some tips / tricks on how to handle these large volumes of Data on 8 GB Machine? \r\n[/quote]\r\n\r\nThere are no fancy tricks, the key is to avoid using expensive libs (Pandas, or whatever), just use the basics, for instance, I have some routines in Python, do you know how many imports I have? 0 \r\n\r\nGive it a try. It works for any language, the more libs you use, the more memory hungry your solution will be as they are designed for \"generic problems\" not for THIS problem.", "137253": "I'm struggling a bit on memory with about 50GB in RAM from AWS. Although I have not done much to be memory efficient yet. I presume to use the current data set on an 8GB machine like NxGTR and Mike mentioned a subset of columns is used, especially for the date file (raw files loaded and row binded is 12.9GB in data.table for me, sparse not a whole lot less). \r\nOn the categorical file I checked on about 200K rows and after removing constant columns and duplicates it was reduced down to about 190 columns.", "137260": "If you are running xgboost, then you can convert you csv to a libsvm format which allows your fit the complete numeric dataset in your memory with 8 GB without any slowdowns. That's one approach you can do.", "137262": "I'm using R and a 48GB machine to build my current solution. So you need to be memory efficient ;-D", "137264": "[quote=Gilberto Titericz Junior;137262]\r\n\r\nI'm using R and a 48GB machine to build my current solution. So you need to be memory efficient ;-D\r\n\r\n[/quote]\r\n\r\nCurious. How long does it take for your model to run?", "137445": "BTW Scirpus's script is not only very good, but fits in ~4GB. He uses pandas chunking and filters out select columns which allows it to fit into low memory even as dense matrices.\r\n\r\nedit: Looking at things a bit more, *definitely* use a reduced set of features, you can refine your models even with 8-16GB of memory that way. The set of features in Scirpus's script (+ the two I mention) make a solid starter IMO and are good for >.24 even without probability conversion.\r\n\r\nHowever, i expect the silver medal threshold to be *at least* 0.4 by the time this competition ends...", "137477": "Hi @niks2000,\r\n\r\nThere are many ways to handle using a small RAM memory.\r\n\r\nOf course, you will need more time and, if possible, a good processor.\r\n \r\nHave a look at buffering techniques.\r\n\r\nRegards", "137495": "Hi, I am a novice. This is my second project. I cant even see the data of training or testing sets. All of the comments above are very high level information for me. Can someone tell/help me with how to approach this big data sets?", "137535": "Apoorva\r\n\r\nI am only a beginner myself and have neither a good machine or a lot of experience.\r\n\r\nFor the average human being with a computer of 8-16GB of memory, you will probably need to analyze the data in chunks like this published script:\r\n\r\nhttps://www.kaggle.com/joconnor/bosch-production-line-performance/python-xgboost-starter-0-209-public-mcc\r\n\r\nDon't worry about trying to \"win\" try and understand what the script is doing and why--the author does a nice job explaining what is going on. As you begin to understand what is going on you can tweak the script with your own ideas. Just keep building piece by piece and eventually you catch on.", "137539": "Bob, Lord of Cats\r\n\r\nThanks for sharing the link. I ll try to understand these.", "137559": "Must... Make... my swap big enough so i can fit the whole dataset and wait forever for the computer to finish modeling swapping 100 pages. Kidding\r\n\r\nTrue asnwer: \r\n\r\n - Set a 100GB swap partition on linux. \r\n - Default swap on W10 is locked to 23GB(at least on my computer). Set custom size.\r\n\r\n\r\nBeware that this will put a lot of pressure on SSD's.. BUT, who really cares?", "137572": "I never used R, but I wonder if cannot use some kind of mini-batches of data?" }, "source": "meta" }