I tried loading the numerical training set but my computer could not handle it, does anyone have any tips for loading the data into R? I have a 64 bit laptop with 8 gigs of RAM.
", "messages": [ { "id": "133222", "postDate": "09/01/2016 00:15:05", "content": "I tried loading the numerical training set but my computer could not handle it, does anyone have any tips for loading the data into R? I have a 64 bit laptop with 8 gigs of RAM.
", "rawMarkdown": "I tried loading the numerical training set but my computer could not handle it, does anyone have any tips for loading the data into R? I have a 64 bit laptop with 8 gigs of RAM.", "votes": null }, { "id": "133227", "postDate": "09/01/2016 00:47:22", "content": "Look up the command:
\n\nmemory.limit(size=60000)
\n\nI'm not sure all the limitations. I use this on a Windows 10 computer. 8 GB memory.\nIt seems to work.\nMake sure your page file is huge (like 60 GB). My page file is on a solid-state disk. A spinning disk might be too slow.
\n\nAlso, right before running a large data load, I reboot to get "cleaner" memory. I don't know if that actually benefits, or is just superstition.
\n\nAnother thread on this forum showed how to use R with the data broken into multiple chunks. I break the train numeric and train categorical files into 3 files with about 400,000 rows each. I haven't tried to use the Date files yet.
\n\nUsing the above memory option, I can fit the entire train_numeric file in memory, but not the train_numeric and the train_categorical together.
\n\n-Rich
", "rawMarkdown": "Look up the command:\r\n\r\nmemory.limit(size=60000)\r\n\r\nI'm not sure all the limitations. I use this on a Windows 10 computer. 8 GB memory.\r\nIt seems to work.\r\nMake sure your page file is huge (like 60 GB). My page file is on a solid-state disk. A spinning disk might be too slow.\r\n\r\nAlso, right before running a large data load, I reboot to get \"cleaner\" memory. I don't know if that actually benefits, or is just superstition.\r\n\r\nAnother thread on this forum showed how to use R with the data broken into multiple chunks. I break the train numeric and train categorical files into 3 files with about 400,000 rows each. I haven't tried to use the Date files yet.\r\n\r\nUsing the above memory option, I can fit the entire train_numeric file in memory, but not the train_numeric and the train_categorical together.\r\n\r\n-Rich", "votes": null }, { "id": "133267", "postDate": "09/01/2016 06:40:43", "content": "[quote=quadcore;133227]
\n\nLook up the command:
\n\nmemory.limit(size=60000)
\n\nI'm not sure all the limitations. I use this on a Windows 10 computer. 8 GB memory.\nIt seems to work.\nMake sure your page file is huge (like 60 GB). My page file is on a solid-state disk. A spinning disk might be too slow.
\n\nAlso, right before running a large data load, I reboot to get "cleaner" memory. I don't know if that actually benefits, or is just superstition.
\n\n[/quote]
\n\nDoesnt memory.limit set the amount of RAM to use? how can you then go up to 60GB if you only have 8 GB? Think im missing something here.\nOtherwise i also have the feeling that rebooting helps quite a bit before loading, does anyone know exactly why that is? Would be awesome to be able to automatize the memory cleaning in some way.
", "rawMarkdown": "[quote=quadcore;133227]\r\n\r\nLook up the command:\r\n\r\nmemory.limit(size=60000)\r\n\r\nI'm not sure all the limitations. I use this on a Windows 10 computer. 8 GB memory.\r\nIt seems to work.\r\nMake sure your page file is huge (like 60 GB). My page file is on a solid-state disk. A spinning disk might be too slow.\r\n\r\nAlso, right before running a large data load, I reboot to get \"cleaner\" memory. I don't know if that actually benefits, or is just superstition.\r\n\r\n[/quote]\r\n\r\nDoesnt memory.limit set the amount of RAM to use? how can you then go up to 60GB if you only have 8 GB? Think im missing something here.\r\nOtherwise i also have the feeling that rebooting helps quite a bit before loading, does anyone know exactly why that is? Would be awesome to be able to automatize the memory cleaning in some way.", "votes": null }, { "id": "133286", "postDate": "09/01/2016 10:30:34", "content": "[quote=Malte Nalenz;133267]
\n\nDoesnt memory.limit set the amount of RAM to use? how can you then go up to 60GB if you only have 8 GB? Think im missing something here.
\n\n[/quote]
\n\nSwap is virtual RAM on disk. On a SSD, it is still pretty fast (~500MB/s, on a PCI-E SSD you can over 1GB/s easily).
\n\n[quote=Malte Nalenz;133267]
\n\nOtherwise i also have the feeling that rebooting helps quite a bit before loading, does anyone know exactly why that is? Would be awesome to be able to automatize the memory cleaning in some way.
\n\n[/quote]
\n\nMost programs are leaking memory, including Windows itself. Also, the more RAM you have, the more cache the programs want, and the more they will not release the cache for your programs which need RAM (well, they get written on disk / deleted by priority but you don't want that in the first place). It's the reason rebooting before doing a RAM-heavy task on a limited RAM computer is always helping: it unallows other programs to make their own cache (only memory leaks/demands will go through), as nearing the RAM limit forbids to cache.
\n\nTypically with a fast SSD for swap, you want your committed RAM (current RAM usage + current swap usage) as low as possible (not only your RAM used).
", "rawMarkdown": "[quote=Malte Nalenz;133267]\r\n\r\nDoesnt memory.limit set the amount of RAM to use? how can you then go up to 60GB if you only have 8 GB? Think im missing something here.\r\n\r\n[/quote]\r\n\r\nSwap is virtual RAM on disk. On a SSD, it is still pretty fast (~500MB/s, on a PCI-E SSD you can over 1GB/s easily).\r\n\r\n[quote=Malte Nalenz;133267]\r\n\r\nOtherwise i also have the feeling that rebooting helps quite a bit before loading, does anyone know exactly why that is? Would be awesome to be able to automatize the memory cleaning in some way.\r\n\r\n[/quote]\r\n\r\nMost programs are leaking memory, including Windows itself. Also, the more RAM you have, the more cache the programs want, and the more they will not release the cache for your programs which need RAM (well, they get written on disk / deleted by priority but you don't want that in the first place). It's the reason rebooting before doing a RAM-heavy task on a limited RAM computer is always helping: it unallows other programs to make their own cache (only memory leaks/demands will go through), as nearing the RAM limit forbids to cache.\r\n\r\nTypically with a fast SSD for swap, you want your committed RAM (current RAM usage + current swap usage) as low as possible (not only your RAM used).", "votes": null }, { "id": "133399", "postDate": "09/01/2016 21:32:59", "content": "If rebooting does not provide you with enough space to load your data batch by batch you can stop some memory consuming services like MS SQL server service.\nalso if you are familiar with python, you can use that because python does memory management or at least use python to break up your data into smaller chunks
", "rawMarkdown": "If rebooting does not provide you with enough space to load your data batch by batch you can stop some memory consuming services like MS SQL server service.\r\nalso if you are familiar with python, you can use that because python does memory management or at least use python to break up your data into smaller chunks", "votes": null } ], "comments": [ { "id": 133227, "author_name": "richardepstein", "author_url": "", "post_date": "09/01/2016 00:47:22", "content": "Look up the command:
\n\nmemory.limit(size=60000)
\n\nI'm not sure all the limitations. I use this on a Windows 10 computer. 8 GB memory.\nIt seems to work.\nMake sure your page file is huge (like 60 GB). My page file is on a solid-state disk. A spinning disk might be too slow.
\n\nAlso, right before running a large data load, I reboot to get "cleaner" memory. I don't know if that actually benefits, or is just superstition.
\n\nAnother thread on this forum showed how to use R with the data broken into multiple chunks. I break the train numeric and train categorical files into 3 files with about 400,000 rows each. I haven't tried to use the Date files yet.
\n\nUsing the above memory option, I can fit the entire train_numeric file in memory, but not the train_numeric and the train_categorical together.
\n\n-Rich
", "votes": null, "replies": [] }, { "id": 133267, "author_name": "malten", "author_url": "", "post_date": "09/01/2016 06:40:43", "content": "[quote=quadcore;133227]
\n\nLook up the command:
\n\nmemory.limit(size=60000)
\n\nI'm not sure all the limitations. I use this on a Windows 10 computer. 8 GB memory.\nIt seems to work.\nMake sure your page file is huge (like 60 GB). My page file is on a solid-state disk. A spinning disk might be too slow.
\n\nAlso, right before running a large data load, I reboot to get "cleaner" memory. I don't know if that actually benefits, or is just superstition.
\n\n[/quote]
\n\nDoesnt memory.limit set the amount of RAM to use? how can you then go up to 60GB if you only have 8 GB? Think im missing something here.\nOtherwise i also have the feeling that rebooting helps quite a bit before loading, does anyone know exactly why that is? Would be awesome to be able to automatize the memory cleaning in some way.
", "votes": null, "replies": [] }, { "id": 133286, "author_name": "laurae2", "author_url": "", "post_date": "09/01/2016 10:30:34", "content": "[quote=Malte Nalenz;133267]
\n\nDoesnt memory.limit set the amount of RAM to use? how can you then go up to 60GB if you only have 8 GB? Think im missing something here.
\n\n[/quote]
\n\nSwap is virtual RAM on disk. On a SSD, it is still pretty fast (~500MB/s, on a PCI-E SSD you can over 1GB/s easily).
\n\n[quote=Malte Nalenz;133267]
\n\nOtherwise i also have the feeling that rebooting helps quite a bit before loading, does anyone know exactly why that is? Would be awesome to be able to automatize the memory cleaning in some way.
\n\n[/quote]
\n\nMost programs are leaking memory, including Windows itself. Also, the more RAM you have, the more cache the programs want, and the more they will not release the cache for your programs which need RAM (well, they get written on disk / deleted by priority but you don't want that in the first place). It's the reason rebooting before doing a RAM-heavy task on a limited RAM computer is always helping: it unallows other programs to make their own cache (only memory leaks/demands will go through), as nearing the RAM limit forbids to cache.
\n\nTypically with a fast SSD for swap, you want your committed RAM (current RAM usage + current swap usage) as low as possible (not only your RAM used).
", "votes": null, "replies": [] }, { "id": 133399, "author_name": "afshinamiri", "author_url": "", "post_date": "09/01/2016 21:32:59", "content": "If rebooting does not provide you with enough space to load your data batch by batch you can stop some memory consuming services like MS SQL server service.\nalso if you are familiar with python, you can use that because python does memory management or at least use python to break up your data into smaller chunks
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "133222": "I tried loading the numerical training set but my computer could not handle it, does anyone have any tips for loading the data into R? I have a 64 bit laptop with 8 gigs of RAM.", "133227": "Look up the command:\r\n\r\nmemory.limit(size=60000)\r\n\r\nI'm not sure all the limitations. I use this on a Windows 10 computer. 8 GB memory.\r\nIt seems to work.\r\nMake sure your page file is huge (like 60 GB). My page file is on a solid-state disk. A spinning disk might be too slow.\r\n\r\nAlso, right before running a large data load, I reboot to get \"cleaner\" memory. I don't know if that actually benefits, or is just superstition.\r\n\r\nAnother thread on this forum showed how to use R with the data broken into multiple chunks. I break the train numeric and train categorical files into 3 files with about 400,000 rows each. I haven't tried to use the Date files yet.\r\n\r\nUsing the above memory option, I can fit the entire train_numeric file in memory, but not the train_numeric and the train_categorical together.\r\n\r\n-Rich", "133267": "[quote=quadcore;133227]\r\n\r\nLook up the command:\r\n\r\nmemory.limit(size=60000)\r\n\r\nI'm not sure all the limitations. I use this on a Windows 10 computer. 8 GB memory.\r\nIt seems to work.\r\nMake sure your page file is huge (like 60 GB). My page file is on a solid-state disk. A spinning disk might be too slow.\r\n\r\nAlso, right before running a large data load, I reboot to get \"cleaner\" memory. I don't know if that actually benefits, or is just superstition.\r\n\r\n[/quote]\r\n\r\nDoesnt memory.limit set the amount of RAM to use? how can you then go up to 60GB if you only have 8 GB? Think im missing something here.\r\nOtherwise i also have the feeling that rebooting helps quite a bit before loading, does anyone know exactly why that is? Would be awesome to be able to automatize the memory cleaning in some way.", "133286": "[quote=Malte Nalenz;133267]\r\n\r\nDoesnt memory.limit set the amount of RAM to use? how can you then go up to 60GB if you only have 8 GB? Think im missing something here.\r\n\r\n[/quote]\r\n\r\nSwap is virtual RAM on disk. On a SSD, it is still pretty fast (~500MB/s, on a PCI-E SSD you can over 1GB/s easily).\r\n\r\n[quote=Malte Nalenz;133267]\r\n\r\nOtherwise i also have the feeling that rebooting helps quite a bit before loading, does anyone know exactly why that is? Would be awesome to be able to automatize the memory cleaning in some way.\r\n\r\n[/quote]\r\n\r\nMost programs are leaking memory, including Windows itself. Also, the more RAM you have, the more cache the programs want, and the more they will not release the cache for your programs which need RAM (well, they get written on disk / deleted by priority but you don't want that in the first place). It's the reason rebooting before doing a RAM-heavy task on a limited RAM computer is always helping: it unallows other programs to make their own cache (only memory leaks/demands will go through), as nearing the RAM limit forbids to cache.\r\n\r\nTypically with a fast SSD for swap, you want your committed RAM (current RAM usage + current swap usage) as low as possible (not only your RAM used).", "133399": "If rebooting does not provide you with enough space to load your data batch by batch you can stop some memory consuming services like MS SQL server service.\r\nalso if you are familiar with python, you can use that because python does memory management or at least use python to break up your data into smaller chunks" }, "source": "meta" }