I am very interested in this competition because i work in manufacturing and is a data analyst. This is the first time i have seen a leading manufacturer in the world using machine learning to improve quality and this is definitively something that my company can learn from. \nThis competition shows that the manufacturing industry has extended their business excellence efforts from traditional Lean and Six Sigma methodologies to machine learning or predictive modeling.\nFor now, the biggest challenge for me is that i cannot find a computer that has big enough memory that can handle this big dataset. I am thinking using subsamples to for modeling but loading the dataset into R is already a challenge for me.\nI use nothing else but R.\nCan anyone provide any advice of loading maybe partial of the original dataset into R for modeling process?
", "messages": [ { "id": "132313", "postDate": "08/24/2016 13:28:14", "content": "I am very interested in this competition because i work in manufacturing and is a data analyst. This is the first time i have seen a leading manufacturer in the world using machine learning to improve quality and this is definitively something that my company can learn from. \nThis competition shows that the manufacturing industry has extended their business excellence efforts from traditional Lean and Six Sigma methodologies to machine learning or predictive modeling.\nFor now, the biggest challenge for me is that i cannot find a computer that has big enough memory that can handle this big dataset. I am thinking using subsamples to for modeling but loading the dataset into R is already a challenge for me.\nI use nothing else but R.\nCan anyone provide any advice of loading maybe partial of the original dataset into R for modeling process?
", "rawMarkdown": "I am very interested in this competition because i work in manufacturing and is a data analyst. This is the first time i have seen a leading manufacturer in the world using machine learning to improve quality and this is definitively something that my company can learn from. \r\nThis competition shows that the manufacturing industry has extended their business excellence efforts from traditional Lean and Six Sigma methodologies to machine learning or predictive modeling.\r\nFor now, the biggest challenge for me is that i cannot find a computer that has big enough memory that can handle this big dataset. I am thinking using subsamples to for modeling but loading the dataset into R is already a challenge for me.\r\nI use nothing else but R.\r\nCan anyone provide any advice of loading maybe partial of the original dataset into R for modeling process?", "votes": null }, { "id": "132318", "postDate": "08/24/2016 14:18:33", "content": "Hi Felix,
\n\ngenerally I would recommend you to invest some time to learn python. It usually works smoother with larger datasets, has a much cleaner syntax, and (imho) a better developed framework at least for common supervised machine learning tasks (sci-kit learn). It might take some time to get you started, but it will definitely make you a better data analyst (and kaggler).
\n\nYour question was about R though ;). I would start off with the numerical features only, that's about two gigs which you should be able to read into R using the fread function from the data.table package ( the base R read.csv function is usually too slow for datasets this size). Once you have your data in R, I would recommend you have a look at the h2o package, that acts basically as an interface to a JVM (java virtual machine) and handles larger datasets much better. If you have medium memory (8-16 GB) you should be fine. If you have much less, you can still subsample the data in R and train a few models on the subsample. Then if you developed some ideas, you can scale them up using AWS (there are some tutorials on the web which make it really easy to install RStudio Server on an EC2 instance).
\n\nHope this helps,
\n\nTim
", "rawMarkdown": "Hi Felix, \r\n\r\ngenerally I would recommend you to invest some time to learn python. It usually works smoother with larger datasets, has a much cleaner syntax, and (imho) a better developed framework at least for common supervised machine learning tasks (sci-kit learn). It might take some time to get you started, but it will definitely make you a better data analyst (and kaggler). \r\n\r\nYour question was about R though ;). I would start off with the numerical features only, that's about two gigs which you should be able to read into R using the fread function from the data.table package ( the base R read.csv function is usually too slow for datasets this size). Once you have your data in R, I would recommend you have a look at the h2o package, that acts basically as an interface to a JVM (java virtual machine) and handles larger datasets much better. If you have medium memory (8-16 GB) you should be fine. If you have much less, you can still subsample the data in R and train a few models on the subsample. Then if you developed some ideas, you can scale them up using AWS (there are some tutorials on the web which make it really easy to install RStudio Server on an EC2 instance).\r\n\r\nHope this helps, \r\n\r\nTim", "votes": null }, { "id": "132330", "postDate": "08/24/2016 15:53:32", "content": "Another option might be renting an online computing resource (AWS, Google Cloud, Azure). You could probably get a pretty good rate, especially if you are just interested in using it for a few hours to explore and subsample the data.
", "rawMarkdown": "Another option might be renting an online computing resource (AWS, Google Cloud, Azure). You could probably get a pretty good rate, especially if you are just interested in using it for a few hours to explore and subsample the data.", "votes": null }, { "id": "132337", "postDate": "08/24/2016 16:40:49", "content": "Well, for low memory consumption I'd suggest looking at Vowpal Wabbit. I'm giving it a try, but yet didn't get any result.
\n\nThe only think you should do is to read the csv files from Kaggle and transform it to the VW format. You can do it as a stream of data using Python's zipfile, gzip and csv. This way, the memory footprint is minimal (you probably can run it in a 4Gb machine).
", "rawMarkdown": "Well, for low memory consumption I'd suggest looking at [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit). I'm giving it a try, but yet didn't get any result.\r\n\r\nThe only think you should do is to read the csv files from Kaggle and transform it to the VW format. You can do it as a stream of data using Python's [zipfile](https://docs.python.org/2/library/zipfile.html), [gzip](https://docs.python.org/2/library/gzip.html) and [csv](https://docs.python.org/2/library/csv.html). This way, the memory footprint is minimal (you probably can run it in a 4Gb machine).", "votes": null }, { "id": "132343", "postDate": "08/24/2016 17:26:49", "content": "Hello felix,
\n\nI agree with suggestions above, yet if you still wish to sample the dataset and prototype on that subset with R you can still do so using fread from data.table library.
\n\nUse nrow parameter to define number of rows to read and select parameter to define desired columns.
\n\nLet me know if you need more assistence (I available @ nathanie at post dot bgu dot ac dot il)
\n\nNati
", "rawMarkdown": "Hello felix,\r\n\r\n\r\nI agree with suggestions above, yet if you still wish to sample the dataset and prototype on that subset with R you can still do so using fread from data.table library.\r\n\r\nUse nrow parameter to define number of rows to read and select parameter to define desired columns.\r\n\r\nLet me know if you need more assistence (I available @ nathanie at post dot bgu dot ac dot il)\r\n\r\nNati", "votes": null }, { "id": "132346", "postDate": "08/24/2016 18:23:26", "content": "Hi Tim,\nI tried H2O
\n\nptm <- proc.time()\ntrain_date <- h2o.importFile(path="train_date.csv")\ntrain_categorical <- h2o.importFile(path="train_categorical.csv")\ntrain_numeric <- h2o.importFile(path="train_numeric.csv")\nproc.time() - ptm
\n\nworked for some of the training data but then ran into java.lang.OutOfMemoryError, see below. Any suggestions how to fix that?
\n\nThanks
\n\nUwe
\n\n\n\n\ntrain_date <- h2o.importFile(path="train_date.csv")\n |================================================== | 50%
\n
DistributedException from /127.0.0.1:54321, caused by java.lang.OutOfMemoryError: Java heap space
\n\nError: DistributedException from /127.0.0.1:54321, caused by java.lang.OutOfMemoryError: Java heap space
\n\n\n", "rawMarkdown": "Hi Tim,\r\nI tried H2O \r\n\r\n ptm <- proc.time()\r\ntrain_date <- h2o.importFile(path=\"train_date.csv\")\r\ntrain_categorical <- h2o.importFile(path=\"train_categorical.csv\")\r\ntrain_numeric <- h2o.importFile(path=\"train_numeric.csv\")\r\nproc.time() - ptm\r\n\r\nworked for some of the training data but then ran into java.lang.OutOfMemoryError, see below. Any suggestions how to fix that?\r\n\r\nThanks\r\n\r\nUwe\r\n\r\n> train_date <- h2o.importFile(path=\"train_date.csv\")\r\n |================================================== | 50%\r\n\r\nDistributedException from /127.0.0.1:54321, caused by java.lang.OutOfMemoryError: Java heap space\r\n\r\nError: DistributedException from /127.0.0.1:54321, caused by java.lang.OutOfMemoryError: Java heap space\r\n> train_categorical <- h2o.importFile(path=\"train_categorical.csv\")\r\n |=====================================================================================================| 100%\r\n> train_numeric <- h2o.importFile(path=\"train_numeric.csv\")\r\n |=====================================================================================================| 100%\r\n> proc.time() - ptm\r\n user system elapsed \r\n 2.943 0.181 75.119", "votes": null }, { "id": "132349", "postDate": "08/24/2016 18:30:15", "content": "train_categorical <- h2o.importFile(path="train_categorical.csv")\n |=====================================================================================================| 100%\n train_numeric <- h2o.importFile(path="train_numeric.csv")\n |=====================================================================================================| 100%\n proc.time() - ptm\n user system elapsed \n 2.943 0.181 75.119
\n
sorry for the messy formatting, my first post with code...
", "rawMarkdown": "sorry for the messy formatting, my first post with code...", "votes": null }, { "id": "132351", "postDate": "08/24/2016 18:43:29", "content": "Well, seems like I found a solution, lets see if i manage to achieve a better formatting as well\nI increased the memory for H2o using max_mem_size
\nA character string specifying the maximum size, in bytes, of the memory allocation pool to H2O. This value must a multiple of 1024 greater than 2MB. Append the letter m or M to indicate megabytes, or g or G to indicate gigabytes. This value is only used when R starts H2O.
h2oServer <- h2o.init(nthreads = -1, max_mem_size = "7G")\nptm <- proc.time()\ntrain_date <- h2o.importFile(path="train_date.csv")\ntrain_categorical <- h2o.importFile(path="train_categorical.csv")\ntrain_numeric <- h2o.importFile(path="train_numeric.csv")\nproc.time() - ptm\nH2O starts now with H2O cluster total memory: 6.22 GB \nAnd therefore I can read in the training data in no time
\n\nuser system elapsed \n 3.857 0.208 80.088
\n\nUsing\nR version 3.3.1 (2016-06-21)\nPlatform: x86_64-apple-darwin13.4.0 (64-bit)\nRunning under: OS X 10.11.6 (El Capitan)
\n\nHopefully Felix has success as well!\nUwe
", "rawMarkdown": "Well, seems like I found a solution, lets see if i manage to achieve a better formatting as well\r\nI increased the memory for H2o using **max_mem_size**\t\r\nA character string specifying the maximum size, in bytes, of the memory allocation pool to H2O. This value must a multiple of 1024 greater than 2MB. Append the letter m or M to indicate megabytes, or g or G to indicate gigabytes. This value is only used when R starts H2O.\r\n\r\n h2oServer <- h2o.init(nthreads = -1, max_mem_size = \"7G\")\r\n ptm <- proc.time()\r\n train_date <- h2o.importFile(path=\"train_date.csv\")\r\n train_categorical <- h2o.importFile(path=\"train_categorical.csv\")\r\n train_numeric <- h2o.importFile(path=\"train_numeric.csv\")\r\n proc.time() - ptm\r\n\r\nH2O starts now with **H2O cluster total memory: 6.22 GB** \r\nAnd therefore I can read in the training data in no time\r\n\r\n user system elapsed \r\n 3.857 0.208 80.088\r\n\r\nUsing\r\nR version 3.3.1 (2016-06-21)\r\nPlatform: x86_64-apple-darwin13.4.0 (64-bit)\r\nRunning under: OS X 10.11.6 (El Capitan)\r\n\r\nHopefully Felix has success as well!\r\nUwe", "votes": null }, { "id": "132372", "postDate": "08/24/2016 23:46:29", "content": "Thanks to Tim Kreienkamp，Bojan Tunguz，Bruno G. do Amaral，Nathaniel Shimoni，Uwe Sterr!\nYou guys have provided me with multiple options to work with data. I will give them a try and hopefully one of them can work for me.
\n\np.s to Tim: actually I always wanted to learn Python because it is considered faster than R, but it seems very challenging for me because I started my career as a Six Sigma Black Belt and is statistics guy and R seemed natural to me while Python seemed unintuitive. But I have decided to follow your advice to learn Python, especially numpy,pandas and sci-kit learn packages. Do you have any recommendations on books or any other online resources that you think is suitable for beginners like me?
", "rawMarkdown": "Thanks to Tim Kreienkamp，Bojan Tunguz，Bruno G. do Amaral，Nathaniel Shimoni，Uwe Sterr!\r\nYou guys have provided me with multiple options to work with data. I will give them a try and hopefully one of them can work for me.\r\n\r\np.s to Tim: actually I always wanted to learn Python because it is considered faster than R, but it seems very challenging for me because I started my career as a Six Sigma Black Belt and is statistics guy and R seemed natural to me while Python seemed unintuitive. But I have decided to follow your advice to learn Python, especially numpy,pandas and sci-kit learn packages. Do you have any recommendations on books or any other online resources that you think is suitable for beginners like me?", "votes": null }, { "id": "132399", "postDate": "08/25/2016 06:15:44", "content": "FelixZhao besides books I would highly recommend looking at PyCon (python general), SciPy (scientific python) and PyData (python Data Science) conferences. They provide video recordings of all of their sessions. Very helpful.
\n\nSciPy 2016 https://www.youtube.com/playlist?list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6\nSciPy 2015 https://www.youtube.com/playlist?list=PLYx7XA2nY5Gcpabmu61kKcToLz0FapmHu
\n\nPyData London 2016 https://www.youtube.com/watch?v=4Ulrsb_MSFo&list=PLGVZCDnMOq0qfJkoiFj-hN7lSHgQzXtqQ
\n\nHere is a quick introduction to Machine Learning with Python\nPart 1\nhttps://www.youtube.com/watch?v=OB1reY6IX-o&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6&index=4\nPart 2\nhttps://www.youtube.com/watch?v=Cte8FYCpylk&index=18&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6
\n\nSubscribe to Enthought, PyData, NextDay video channels.
\n\nRegards
", "rawMarkdown": "FelixZhao besides books I would highly recommend looking at PyCon (python general), SciPy (scientific python) and PyData (python Data Science) conferences. They provide video recordings of all of their sessions. Very helpful.\r\n\r\nSciPy 2016 https://www.youtube.com/playlist?list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6\r\nSciPy 2015 https://www.youtube.com/playlist?list=PLYx7XA2nY5Gcpabmu61kKcToLz0FapmHu\r\n\r\nPyData London 2016 https://www.youtube.com/watch?v=4Ulrsb_MSFo&list=PLGVZCDnMOq0qfJkoiFj-hN7lSHgQzXtqQ\r\n\r\nHere is a quick introduction to Machine Learning with Python\r\nPart 1\r\nhttps://www.youtube.com/watch?v=OB1reY6IX-o&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6&index=4\r\nPart 2\r\nhttps://www.youtube.com/watch?v=Cte8FYCpylk&index=18&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6\r\n\r\nSubscribe to Enthought, PyData, NextDay video channels.\r\n\r\nRegards", "votes": null }, { "id": "132414", "postDate": "08/25/2016 10:35:43", "content": "[quote=FelixZhao;132372]
\n\np.s to Tim: actually I always wanted to learn Python because it is considered faster than R, but it seems very challenging for me because I started my career as a Six Sigma Black Belt and is statistics guy and R seemed natural to me while Python seemed unintuitive. But I have decided to follow your advice to learn Python, especially numpy,pandas and sci-kit learn packages. Do you have any recommendations on books or any other online resources that you think is suitable for beginners like me?
\n\n[/quote]
\n\nHi Felix, once you start spending some time with it you'll see it's actually much easier than R :). \nI personally started learning the basics with the udacity CS 101 course and then learned python for ML 'by kaggle'. The rest is continuous improvement on the job. \nIf I would start to learn it again, I'll probably choose learnpythonthehardway.org (heard from many people, including one Econ professor of mine, they had really good experiences with it). After you developed a basic understanding of the language, turn to the excellent scikit-learn documentation. Also go through the scripts / kernels here on kaggle, download them and execute them line by line in an ipython notebook to really understand whats happening. Finally, build on those ideas and do your kaggle projects in python.
\n\nHope this helps
\n\nTim
", "rawMarkdown": "[quote=FelixZhao;132372]\r\n\r\np.s to Tim: actually I always wanted to learn Python because it is considered faster than R, but it seems very challenging for me because I started my career as a Six Sigma Black Belt and is statistics guy and R seemed natural to me while Python seemed unintuitive. But I have decided to follow your advice to learn Python, especially numpy,pandas and sci-kit learn packages. Do you have any recommendations on books or any other online resources that you think is suitable for beginners like me?\r\n\r\n[/quote]\r\n\r\nHi Felix, once you start spending some time with it you'll see it's actually much easier than R :). \r\nI personally started learning the basics with the udacity CS 101 course and then learned python for ML 'by kaggle'. The rest is continuous improvement on the job. \r\nIf I would start to learn it again, I'll probably choose learnpythonthehardway.org (heard from many people, including one Econ professor of mine, they had really good experiences with it). After you developed a basic understanding of the language, turn to the excellent scikit-learn documentation. Also go through the scripts / kernels here on kaggle, download them and execute them line by line in an ipython notebook to really understand whats happening. Finally, build on those ideas and do your kaggle projects in python. \r\n\r\nHope this helps\r\n\r\nTim", "votes": null }, { "id": "132525", "postDate": "08/26/2016 13:21:26", "content": "I tried to increase the memory for H2o using max_mem_size as above. Couldn't be successful
\n\nInvalid maximum heap size: -Xmx7G
\n\n================================================================================
\n\n> h2oServer <- h2o.init(nthreads = -1, max_mem_size = "7G")\n\nH2O is not running yet, starting it now...\n\nNote: In case of errors look at the following log files:\n C:\\Users\\User\\AppData\\Local\\Temp\\RtmpKa7XEy/h2o_GNS313_started_from_r.out\n C:\\Users\\User\\AppData\\Local\\Temp\\RtmpKa7XEy/h2o_GNS313_started_from_r.err\n\nError: Could not create the Java Virtual Machine.\nError: A fatal exception has occurred. Program will exit.\nInvalid maximum heap size: -Xmx7G\nThe specified size exceeds the maximum representable size.\n\nStarting H2O JVM and connecting: ............................................................ \n[1] "localhost"\n[1] 54321\n[1] TRUE\n[1] -1\n[1] "Failed to connect to localhost port 54321: Connection refused"\n[1] 127\nError in h2o.init(nthreads = -1, max_mem_size = "7G") : \n H2O failed to start, stopping execution.\nIn addition: Warning messages:\n1: In .h2o.startJar(nthreads = nthreads, max_memory = max_mem_size, :\n You have a 32-bit version of Java. H2O works best with 64-bit Java.\nPlease download the latest Java SE JDK 7 from the following URL:\nhttp://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html\n2: running command '"C:\\Program Files (x86)\\Java\\jdk1.8.0_74\\bin\\java.exe" -Xmx7G -version' had status 1 \n3: running command 'curl 'http://localhost:54321'' had status 127 \nWhen started default h2o its working but no use as space is less 0.97GB
\n\n===========================================================
\n\n> localH2O = h2o.init(ip = 'localhost', port = 54321)\n\nH2O is not running yet, starting it now...\n\nNote: In case of errors look at the following log files:\n C:\\Users\\User\\AppData\\Local\\Temp\\RtmpKa7XEy/h2o_GNS313_started_from_r.out\n C:\\Users\\User\\AppData\\Local\\Temp\\RtmpKa7XEy/h2o_GNS313_started_from_r.err\n\njava version "1.8.0_74"\nJava(TM) SE Runtime Environment (build 1.8.0_74-b02)\nJava HotSpot(TM) Client VM (build 25.74-b02, mixed mode)\n\nStarting H2O JVM and connecting: . Connection successful!\n\nR is connected to the H2O cluster: \n H2O cluster uptime: 2 seconds 827 milliseconds \n H2O cluster version: 3.8.3.3 \n H2O cluster name: H2O_started_from_R_GNS313_sex025 \n H2O cluster total nodes: 1 \n H2O cluster total memory: 0.97 GB \n H2O cluster total cores: 4 \n H2O cluster allowed cores: 2 \n H2O cluster healthy: TRUE \n H2O Connection ip: localhost \n H2O Connection port: 54321 \n H2O Connection proxy: NA \n R Version: R version 3.3.0 (2016-05-03) \n[quote=Tim Kreienkamp;132414]
\n\nHi Felix, once you start spending some time with it you'll see it's actually much easier than R :).
\n\n[/quote]
\n\nIt's not. It's just different. Both are awesome. Let's not go into the 'what is better, R or python' topic and start a programming language war (I know that was not your intention, Tim).
", "rawMarkdown": "[quote=Tim Kreienkamp;132414]\r\n\r\nHi Felix, once you start spending some time with it you'll see it's actually much easier than R :). \r\n\r\n[/quote]\r\n\r\nIt's not. It's just different. Both are awesome. Let's not go into the 'what is better, R or python' topic and start a programming language war (I know that was not your intention, Tim).", "votes": null }, { "id": "132537", "postDate": "08/26/2016 15:28:20", "content": "[quote=Sumegh Bhalshankar;132527]
\n\nWhen started default h2o its working but no use as space is less 0.97GB
\n\n[/quote]
\n\nThe verbose error print gave you all the elements to solve your issue. Step by step solving:
\n\n\n\n\nh2oServer <- h2o.init(nthreads = -1, max_mem_size = "7G")
\n
\n\n\nThe specified size exceeds the maximum representable size.
\n
\n\n\nYou have a 32-bit version of Java. H2O works best with 64-bit Java.\n Please download the latest Java SE JDK 7 from the following URL:\n http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
\n
Hi Felix,
\n\ngenerally I would recommend you to invest some time to learn python. It usually works smoother with larger datasets, has a much cleaner syntax, and (imho) a better developed framework at least for common supervised machine learning tasks (sci-kit learn). It might take some time to get you started, but it will definitely make you a better data analyst (and kaggler).
\n\nYour question was about R though ;). I would start off with the numerical features only, that's about two gigs which you should be able to read into R using the fread function from the data.table package ( the base R read.csv function is usually too slow for datasets this size). Once you have your data in R, I would recommend you have a look at the h2o package, that acts basically as an interface to a JVM (java virtual machine) and handles larger datasets much better. If you have medium memory (8-16 GB) you should be fine. If you have much less, you can still subsample the data in R and train a few models on the subsample. Then if you developed some ideas, you can scale them up using AWS (there are some tutorials on the web which make it really easy to install RStudio Server on an EC2 instance).
\n\nHope this helps,
\n\nTim
", "votes": null, "replies": [] }, { "id": 132330, "author_name": "tunguz", "author_url": "", "post_date": "08/24/2016 15:53:32", "content": "Another option might be renting an online computing resource (AWS, Google Cloud, Azure). You could probably get a pretty good rate, especially if you are just interested in using it for a few hours to explore and subsample the data.
", "votes": null, "replies": [] }, { "id": 132337, "author_name": "bguberfain", "author_url": "", "post_date": "08/24/2016 16:40:49", "content": "Well, for low memory consumption I'd suggest looking at Vowpal Wabbit. I'm giving it a try, but yet didn't get any result.
\n\nThe only think you should do is to read the csv files from Kaggle and transform it to the VW format. You can do it as a stream of data using Python's zipfile, gzip and csv. This way, the memory footprint is minimal (you probably can run it in a 4Gb machine).
", "votes": null, "replies": [] }, { "id": 132343, "author_name": "interesting", "author_url": "", "post_date": "08/24/2016 17:26:49", "content": "Hello felix,
\n\nI agree with suggestions above, yet if you still wish to sample the dataset and prototype on that subset with R you can still do so using fread from data.table library.
\n\nUse nrow parameter to define number of rows to read and select parameter to define desired columns.
\n\nLet me know if you need more assistence (I available @ nathanie at post dot bgu dot ac dot il)
\n\nNati
", "votes": null, "replies": [] }, { "id": 132346, "author_name": "nappon", "author_url": "", "post_date": "08/24/2016 18:23:26", "content": "Hi Tim,\nI tried H2O
\n\nptm <- proc.time()\ntrain_date <- h2o.importFile(path="train_date.csv")\ntrain_categorical <- h2o.importFile(path="train_categorical.csv")\ntrain_numeric <- h2o.importFile(path="train_numeric.csv")\nproc.time() - ptm
\n\nworked for some of the training data but then ran into java.lang.OutOfMemoryError, see below. Any suggestions how to fix that?
\n\nThanks
\n\nUwe
\n\n\n\n\ntrain_date <- h2o.importFile(path="train_date.csv")\n |================================================== | 50%
\n
DistributedException from /127.0.0.1:54321, caused by java.lang.OutOfMemoryError: Java heap space
\n\nError: DistributedException from /127.0.0.1:54321, caused by java.lang.OutOfMemoryError: Java heap space
\n\n\n", "votes": null, "replies": [] }, { "id": 132349, "author_name": "nappon", "author_url": "", "post_date": "08/24/2016 18:30:15", "content": "train_categorical <- h2o.importFile(path="train_categorical.csv")\n |=====================================================================================================| 100%\n train_numeric <- h2o.importFile(path="train_numeric.csv")\n |=====================================================================================================| 100%\n proc.time() - ptm\n user system elapsed \n 2.943 0.181 75.119
\n
sorry for the messy formatting, my first post with code...
", "votes": null, "replies": [] }, { "id": 132351, "author_name": "nappon", "author_url": "", "post_date": "08/24/2016 18:43:29", "content": "Well, seems like I found a solution, lets see if i manage to achieve a better formatting as well\nI increased the memory for H2o using max_mem_size
\nA character string specifying the maximum size, in bytes, of the memory allocation pool to H2O. This value must a multiple of 1024 greater than 2MB. Append the letter m or M to indicate megabytes, or g or G to indicate gigabytes. This value is only used when R starts H2O.
h2oServer <- h2o.init(nthreads = -1, max_mem_size = "7G")\nptm <- proc.time()\ntrain_date <- h2o.importFile(path="train_date.csv")\ntrain_categorical <- h2o.importFile(path="train_categorical.csv")\ntrain_numeric <- h2o.importFile(path="train_numeric.csv")\nproc.time() - ptm\nH2O starts now with H2O cluster total memory: 6.22 GB \nAnd therefore I can read in the training data in no time
\n\nuser system elapsed \n 3.857 0.208 80.088
\n\nUsing\nR version 3.3.1 (2016-06-21)\nPlatform: x86_64-apple-darwin13.4.0 (64-bit)\nRunning under: OS X 10.11.6 (El Capitan)
\n\nHopefully Felix has success as well!\nUwe
", "votes": null, "replies": [] }, { "id": 132372, "author_name": "felixzhao", "author_url": "", "post_date": "08/24/2016 23:46:29", "content": "Thanks to Tim Kreienkamp，Bojan Tunguz，Bruno G. do Amaral，Nathaniel Shimoni，Uwe Sterr!\nYou guys have provided me with multiple options to work with data. I will give them a try and hopefully one of them can work for me.
\n\np.s to Tim: actually I always wanted to learn Python because it is considered faster than R, but it seems very challenging for me because I started my career as a Six Sigma Black Belt and is statistics guy and R seemed natural to me while Python seemed unintuitive. But I have decided to follow your advice to learn Python, especially numpy,pandas and sci-kit learn packages. Do you have any recommendations on books or any other online resources that you think is suitable for beginners like me?
", "votes": null, "replies": [] }, { "id": 132399, "author_name": "sakvaua", "author_url": "", "post_date": "08/25/2016 06:15:44", "content": "FelixZhao besides books I would highly recommend looking at PyCon (python general), SciPy (scientific python) and PyData (python Data Science) conferences. They provide video recordings of all of their sessions. Very helpful.
\n\nSciPy 2016 https://www.youtube.com/playlist?list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6\nSciPy 2015 https://www.youtube.com/playlist?list=PLYx7XA2nY5Gcpabmu61kKcToLz0FapmHu
\n\nPyData London 2016 https://www.youtube.com/watch?v=4Ulrsb_MSFo&list=PLGVZCDnMOq0qfJkoiFj-hN7lSHgQzXtqQ
\n\nHere is a quick introduction to Machine Learning with Python\nPart 1\nhttps://www.youtube.com/watch?v=OB1reY6IX-o&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6&index=4\nPart 2\nhttps://www.youtube.com/watch?v=Cte8FYCpylk&index=18&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6
\n\nSubscribe to Enthought, PyData, NextDay video channels.
\n\nRegards
", "votes": null, "replies": [] }, { "id": 132414, "author_name": "timkreienkamp", "author_url": "", "post_date": "08/25/2016 10:35:43", "content": "[quote=FelixZhao;132372]
\n\np.s to Tim: actually I always wanted to learn Python because it is considered faster than R, but it seems very challenging for me because I started my career as a Six Sigma Black Belt and is statistics guy and R seemed natural to me while Python seemed unintuitive. But I have decided to follow your advice to learn Python, especially numpy,pandas and sci-kit learn packages. Do you have any recommendations on books or any other online resources that you think is suitable for beginners like me?
\n\n[/quote]
\n\nHi Felix, once you start spending some time with it you'll see it's actually much easier than R :). \nI personally started learning the basics with the udacity CS 101 course and then learned python for ML 'by kaggle'. The rest is continuous improvement on the job. \nIf I would start to learn it again, I'll probably choose learnpythonthehardway.org (heard from many people, including one Econ professor of mine, they had really good experiences with it). After you developed a basic understanding of the language, turn to the excellent scikit-learn documentation. Also go through the scripts / kernels here on kaggle, download them and execute them line by line in an ipython notebook to really understand whats happening. Finally, build on those ideas and do your kaggle projects in python.
\n\nHope this helps
\n\nTim
", "votes": null, "replies": [] }, { "id": 132525, "author_name": "sumegh", "author_url": "", "post_date": "08/26/2016 13:21:26", "content": "I tried to increase the memory for H2o using max_mem_size as above. Couldn't be successful
\n\nInvalid maximum heap size: -Xmx7G
\n\n================================================================================
\n\n> h2oServer <- h2o.init(nthreads = -1, max_mem_size = "7G")\n\nH2O is not running yet, starting it now...\n\nNote: In case of errors look at the following log files:\n C:\\Users\\User\\AppData\\Local\\Temp\\RtmpKa7XEy/h2o_GNS313_started_from_r.out\n C:\\Users\\User\\AppData\\Local\\Temp\\RtmpKa7XEy/h2o_GNS313_started_from_r.err\n\nError: Could not create the Java Virtual Machine.\nError: A fatal exception has occurred. Program will exit.\nInvalid maximum heap size: -Xmx7G\nThe specified size exceeds the maximum representable size.\n\nStarting H2O JVM and connecting: ............................................................ \n[1] "localhost"\n[1] 54321\n[1] TRUE\n[1] -1\n[1] "Failed to connect to localhost port 54321: Connection refused"\n[1] 127\nError in h2o.init(nthreads = -1, max_mem_size = "7G") : \n H2O failed to start, stopping execution.\nIn addition: Warning messages:\n1: In .h2o.startJar(nthreads = nthreads, max_memory = max_mem_size, :\n You have a 32-bit version of Java. H2O works best with 64-bit Java.\nPlease download the latest Java SE JDK 7 from the following URL:\nhttp://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html\n2: running command '"C:\\Program Files (x86)\\Java\\jdk1.8.0_74\\bin\\java.exe" -Xmx7G -version' had status 1 \n3: running command 'curl 'http://localhost:54321'' had status 127 \nWhen started default h2o its working but no use as space is less 0.97GB
\n\n===========================================================
\n\n> localH2O = h2o.init(ip = 'localhost', port = 54321)\n\nH2O is not running yet, starting it now...\n\nNote: In case of errors look at the following log files:\n C:\\Users\\User\\AppData\\Local\\Temp\\RtmpKa7XEy/h2o_GNS313_started_from_r.out\n C:\\Users\\User\\AppData\\Local\\Temp\\RtmpKa7XEy/h2o_GNS313_started_from_r.err\n\njava version "1.8.0_74"\nJava(TM) SE Runtime Environment (build 1.8.0_74-b02)\nJava HotSpot(TM) Client VM (build 25.74-b02, mixed mode)\n\nStarting H2O JVM and connecting: . Connection successful!\n\nR is connected to the H2O cluster: \n H2O cluster uptime: 2 seconds 827 milliseconds \n H2O cluster version: 3.8.3.3 \n H2O cluster name: H2O_started_from_R_GNS313_sex025 \n H2O cluster total nodes: 1 \n H2O cluster total memory: 0.97 GB \n H2O cluster total cores: 4 \n H2O cluster allowed cores: 2 \n H2O cluster healthy: TRUE \n H2O Connection ip: localhost \n H2O Connection port: 54321 \n H2O Connection proxy: NA \n R Version: R version 3.3.0 (2016-05-03) \n[quote=Tim Kreienkamp;132414]
\n\nHi Felix, once you start spending some time with it you'll see it's actually much easier than R :).
\n\n[/quote]
\n\nIt's not. It's just different. Both are awesome. Let's not go into the 'what is better, R or python' topic and start a programming language war (I know that was not your intention, Tim).
", "votes": null, "replies": [] }, { "id": 132537, "author_name": "laurae2", "author_url": "", "post_date": "08/26/2016 15:28:20", "content": "[quote=Sumegh Bhalshankar;132527]
\n\nWhen started default h2o its working but no use as space is less 0.97GB
\n\n[/quote]
\n\nThe verbose error print gave you all the elements to solve your issue. Step by step solving:
\n\n\n\n\nh2oServer <- h2o.init(nthreads = -1, max_mem_size = "7G")
\n
\n\n\nThe specified size exceeds the maximum representable size.
\n
\n\n\nYou have a 32-bit version of Java. H2O works best with 64-bit Java.\n Please download the latest Java SE JDK 7 from the following URL:\n http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
\n