Hi guys,
\n\nI am thinking about attacking this problem using R and Spark.\nI understand that there is a possibility to use Amazone AMR to set up the Spark cluster. It was described as very easy. However I don't know much about the whole issue. When I google to find a useful guide I find too many guides that describe all sorts of different tools and topics, throwing arround terms like HDFS, Hadoop, Hive, Zepelin and I don't know what.
\n\nDoes anybody know a good guide to start with when you are really just hoping to start a Spark cluster, upload the data and run your R commands in that cluster to speed up things?\nOr is that not how it works in the first place. I envision A Spark cluster to be basically the same than my computer, just that there is several computers working together in the background.
\n\nThanks in advance,\nFelix
", "messages": [ { "id": "133491", "postDate": "09/02/2016 14:18:40", "content": "Hi guys,
\n\nI am thinking about attacking this problem using R and Spark.\nI understand that there is a possibility to use Amazone AMR to set up the Spark cluster. It was described as very easy. However I don't know much about the whole issue. When I google to find a useful guide I find too many guides that describe all sorts of different tools and topics, throwing arround terms like HDFS, Hadoop, Hive, Zepelin and I don't know what.
\n\nDoes anybody know a good guide to start with when you are really just hoping to start a Spark cluster, upload the data and run your R commands in that cluster to speed up things?\nOr is that not how it works in the first place. I envision A Spark cluster to be basically the same than my computer, just that there is several computers working together in the background.
\n\nThanks in advance,\nFelix
", "rawMarkdown": "Hi guys,\r\n\r\nI am thinking about attacking this problem using R and Spark.\r\nI understand that there is a possibility to use Amazone AMR to set up the Spark cluster. It was described as very easy. However I don't know much about the whole issue. When I google to find a useful guide I find too many guides that describe all sorts of different tools and topics, throwing arround terms like HDFS, Hadoop, Hive, Zepelin and I don't know what.\r\n\r\nDoes anybody know a good guide to start with when you are really just hoping to start a Spark cluster, upload the data and run your R commands in that cluster to speed up things?\r\nOr is that not how it works in the first place. I envision A Spark cluster to be basically the same than my computer, just that there is several computers working together in the background.\r\n\r\nThanks in advance,\r\nFelix", "votes": null }, { "id": "133519", "postDate": "09/02/2016 17:32:11", "content": "Databricks platform could be helpful for your Spark-R cluster need.
\n\nhttps://databricks.com/try-databricks
\n\nCheers
", "rawMarkdown": "Databricks platform could be helpful for your Spark-R cluster need.\r\n\r\nhttps://databricks.com/try-databricks\r\n\r\nCheers", "votes": null }, { "id": "134151", "postDate": "09/05/2016 12:47:41", "content": "Manuel has excellent tutorials and he also competes here.\nThis is probably what you are looking for:\nhttps://www.youtube.com/watch?v=ISsnKm2mAx4
", "rawMarkdown": "Manuel has excellent tutorials and he also competes here.\r\nThis is probably what you are looking for:\r\nhttps://www.youtube.com/watch?v=ISsnKm2mAx4", "votes": null }, { "id": "134161", "postDate": "09/05/2016 13:24:25", "content": "I wrote this\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/\nbut using python and pyspark on a local machine.
", "rawMarkdown": "I wrote this\r\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/\r\nbut using python and pyspark on a local machine.", "votes": null }, { "id": "134826", "postDate": "09/08/2016 18:34:31", "content": "You might also want to check out Databricks' short (2-week) Edx Spark course ('BerkeleyX CS105x') using Python 2.7/PySpark.
\n\nhttps://courses.edx.org/courses/course-v1%3ABerkeleyX%2BCS105x%2B1T2016/info
", "rawMarkdown": "You might also want to check out Databricks' short (2-week) Edx Spark course ('BerkeleyX CS105x') using Python 2.7/PySpark.\r\n\r\nhttps://courses.edx.org/courses/course-v1%3ABerkeleyX%2BCS105x%2B1T2016/info", "votes": null }, { "id": "136536", "postDate": "09/22/2016 18:28:36", "content": "so i take it that using Spark (pyspark) is allowed for Kaggle competitions?
", "rawMarkdown": "so i take it that using Spark (pyspark) is allowed for Kaggle competitions?", "votes": null }, { "id": "136626", "postDate": "09/23/2016 14:52:32", "content": "Yes no probs.
", "rawMarkdown": "Yes no probs.", "votes": null }, { "id": "136688", "postDate": "09/24/2016 00:22:40", "content": "I've set up an Amazon EMR cluster and have used DataBricks, so feel free to ask me any questions. DataBricks basically gives you a managed EMR cluster and a Notebook UI for writing scripts using Spark.
\n\nOur team used Elena's code to get up and running, which allowed us to focus on just getting something working end-to-end without having to know much about Spark at the time.
\n\nIf you happen to live near SF or NYC, Amazon has pop-up lofts where you can ask Amazon engineers questions. They could walk you through the process of setting up an EMR cluster as well!
", "rawMarkdown": "I've set up an Amazon EMR cluster and have used DataBricks, so feel free to ask me any questions. DataBricks basically gives you a managed EMR cluster and a Notebook UI for writing scripts using Spark. \r\n\r\nOur team used Elena's code to get up and running, which allowed us to focus on just getting something working end-to-end without having to know much about Spark at the time. \r\n\r\nIf you happen to live near SF or NYC, Amazon has pop-up lofts where you can ask Amazon engineers questions. They could walk you through the process of setting up an EMR cluster as well!", "votes": null }, { "id": "136954", "postDate": "09/26/2016 20:48:38", "content": "Great job HueyKwik!\nAre your data files living on the clusters? If yes then do you let the clusters (billing) run continuously?
", "rawMarkdown": "Great job HueyKwik!\r\nAre your data files living on the clusters? If yes then do you let the clusters (billing) run continuously?", "votes": null }, { "id": "136976", "postDate": "09/27/2016 03:32:40", "content": "Can anyone suggest me how to save a model in spark, when I am using pipline?\nI am using pyspark. I want to load the model & use rather than always training it, if I am not changing anything before training.
", "rawMarkdown": "Can anyone suggest me how to save a model in spark, when I am using pipline?\r\nI am using pyspark. I want to load the model & use rather than always training it, if I am not changing anything before training.", "votes": null }, { "id": "136988", "postDate": "09/27/2016 06:52:02", "content": "@AbhijitDas You can't if you are using spark < 2.0. \nThey have fixed that in Spark 2.0
", "rawMarkdown": "AbhijitDas You can't if you are using spark < 2.0. \r\nThey have fixed that in Spark 2.0", "votes": null }, { "id": "137015", "postDate": "09/27/2016 13:12:14", "content": "Felix^6 - did you make any progress on AWS? I decided to take the plunge and learned a few things so far:
\n\nYou can save time by starting with the EMR service. On the AWS Management Console, it's listed toward the bottom in the Analytics section. EMR has menu options that let you quickly spin up multiple EC2 instances with Hadoop and Spark properly configured. It also has options to install Hive, Pig, Zeppelin and other cool sounding things.
For setup, I recommend the advanced setup vs. the quick setup. Mainly because you can then use spot requests for your EC2 instances. I use default settings for most everything else. EMR will configure your security policy to work with Spark, which is nice. I found that I still had to go in and open Port 8787 for R-Studio and Port 22 for SSH.
Even so it's pretty amazing that a semi-trained windows user can get up and running with R-Studio or Jupyter Notebook over Spark! Here are some references that helped.
\n\n\n\n", "rawMarkdown": "Felix^6 - did you make any progress on AWS? I decided to take the plunge and learned a few things so far:\r\n\r\n- You can save time by starting with the EMR service. On the AWS Management Console, it's listed toward the bottom in the Analytics section. EMR has menu options that let you quickly spin up multiple EC2 instances with Hadoop and Spark properly configured. It also has options to install Hive, Pig, Zeppelin and other cool sounding things.\r\n\r\n- For setup, I recommend the advanced setup vs. the quick setup. Mainly because you can then use spot requests for your EC2 instances. I use default settings for most everything else. EMR will configure your security policy to work with Spark, which is nice. I found that I still had to go in and open Port 8787 for R-Studio and Port 22 for SSH.\r\n- The thing I like least about it is that the instances are all based on Amazon Linux. Things don't seem nearly as easy as with Ubuntu and such. And the R-version that's installed is 3.2.x. Maybe not a big deal, but I like to run updated software. I spent many hours building R-3.3.1 from source and finally got it to run. And I get R-Studio up and running with the old R. But I have yet to bring both together - maybe today....\r\n\r\nEven so it's pretty amazing that a semi-trained windows user can get up and running with R-Studio or Jupyter Notebook over Spark! Here are some references that helped.\r\n\r\nhttps://blogs.aws.amazon.com/bigdata/post/TxIMFT77KLF6TA/Crunching-Statistics-at-Scale-with-SparkR-on-Amazon-EMR \r\n\r\nhttp://blogs.aws.amazon.com/bigdata/post/Tx37RSKRFDQNTSL/Statistical-Analysis-with-Open-Source-R-and-RStudio-on-Amazon-EMR", "votes": null }, { "id": "137019", "postDate": "09/27/2016 14:08:43", "content": "@Can Arel, Thanks for the reply. Could you pls post the code to save the model?\nI am using spark 2.0
", "rawMarkdown": "Can Arel, Thanks for the reply. Could you pls post the code to save the model?\r\nI am using spark 2.0", "votes": null }, { "id": "137020", "postDate": "09/27/2016 14:17:17", "content": "I have been trying out the new RStudio package, sparklyR. It was easy to set up on my local machine\nhttp://spark.rstudio.com/
\n\nI just need to get it working on AWS now.
", "rawMarkdown": "I have been trying out the new RStudio package, sparklyR. It was easy to set up on my local machine\r\nhttp://spark.rstudio.com/\r\n\r\nI just need to get it working on AWS now.", "votes": null }, { "id": "137021", "postDate": "09/27/2016 14:29:49", "content": "@Akash Saha\nEach model has a save/load function now. \nCheck out e.g. org.apache.spark.ml.classification Classes
", "rawMarkdown": "Akash Saha\r\nEach model has a save/load function now. \r\nCheck out e.g. org.apache.spark.ml.classification Classes", "votes": null }, { "id": "137028", "postDate": "09/27/2016 16:51:47", "content": "@PrashantDeep - I leave the data files on S3, so while they are being billed continuously, it is relatively cheap.
", "rawMarkdown": "PrashantDeep - I leave the data files on S3, so while they are being billed continuously, it is relatively cheap.", "votes": null }, { "id": "137047", "postDate": "09/27/2016 18:31:07", "content": "@HueyKwik\nThats a good option but when ever we begin the analysis, we will have to load the data files from S3 to the new clusters. That would take up lot of time every-time the analysis is started on new clusters especially in this case where the data files are really huge. \n@all - Whats are the good options to avoid this delay
", "rawMarkdown": "HueyKwik\r\nThats a good option but when ever we begin the analysis, we will have to load the data files from S3 to the new clusters. That would take up lot of time every-time the analysis is started on new clusters especially in this case where the data files are really huge. \r\n@all - Whats are the good options to avoid this delay", "votes": null }, { "id": "138461", "postDate": "10/08/2016 22:45:05", "content": "[quote=Elena Cuoco;134161]
\n\nI wrote this\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/\nbut using python and pyspark on a local machine.
\n\n[/quote]
\n\n@Elena, I read your post in the "Magical Feature" thread about spark+xgboost. I'm learning Spark, and having plan to practice spark in this dataset, also in single local machine. However in Windows OS the XGBoost4J is not supported yet. I would like to know, are you in Linux, Mac or Windows?
\n\nAlso would be interesting to read your forum/kernels report on spark+xgboost experiences. For me, this is more interesting than the leakages ;)
", "rawMarkdown": "[quote=Elena Cuoco;134161]\r\n\r\nI wrote this\r\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/\r\nbut using python and pyspark on a local machine.\r\n\r\n[/quote]\r\n\r\n@Elena, I read your post in the \"Magical Feature\" thread about spark+xgboost. I'm learning Spark, and having plan to practice spark in this dataset, also in single local machine. However in Windows OS the XGBoost4J is not supported yet. I would like to know, are you in Linux, Mac or Windows? \r\n\r\nAlso would be interesting to read your forum/kernels report on spark+xgboost experiences. For me, this is more interesting than the leakages ;)", "votes": null }, { "id": "138488", "postDate": "10/09/2016 05:48:57", "content": "[quote=aldente;138461]
\n\n@Elena, I read your post in the "Magical Feature" thread about spark+xgboost. I'm learning Spark, and having plan to practice spark in this dataset, also in single local machine. However in Windows OS the XGBoost4J is not supported yet. I would like to know, are you in Linux, Mac or Windows?
\n\nAlso would be interesting to read your forum/kernels report on spark+xgboost experiences. For me, this is more interesting than the leakages ;)
\n\n[/quote]\n@aldente. I run on Linux and install each package from recent github release. Reading different forum these days to look for solutions and help for scala, it seems that someone did something for windows. \nI cannot help. I'm an linux and open source software addicted user since many years :)\nxgboost recently delivered an interface to java and scala for spark, not yet in python. I asked also to have a direct interface to Spark >2.0, where RDD are not more supported, but there are the DataFrame and they release un updated just one week ago.\nIt was an hard work for me to learn new language (scala) and write the code from reading csv data to write results in parquet and have xgboost running. I plan to write a post on my website on this experience. I was waiting for the end of competition, but It seems now that the competition will be focused on how much leaks are in the data, and not anymore on ML modeling, I will share my code in the next days. \nI will be a bit busy in the following day at work, but I'll try anyhow to write a fast post during next evening. I'll let you know when it is published.
", "rawMarkdown": "[quote=aldente;138461]\r\n\r\n\r\n@Elena, I read your post in the \"Magical Feature\" thread about spark+xgboost. I'm learning Spark, and having plan to practice spark in this dataset, also in single local machine. However in Windows OS the XGBoost4J is not supported yet. I would like to know, are you in Linux, Mac or Windows? \r\n\r\nAlso would be interesting to read your forum/kernels report on spark+xgboost experiences. For me, this is more interesting than the leakages ;) \r\n\r\n\r\n[/quote]\r\n@aldente. I run on Linux and install each package from recent github release. Reading different forum these days to look for solutions and help for scala, it seems that someone did something for windows. \r\nI cannot help. I'm an linux and open source software addicted user since many years :)\r\nxgboost recently delivered an interface to java and scala for spark, not yet in python. I asked also to have a direct interface to Spark >2.0, where RDD are not more supported, but there are the DataFrame and they release un updated just one week ago.\r\nIt was an hard work for me to learn new language (scala) and write the code from reading csv data to write results in parquet and have xgboost running. I plan to write a post on my website on this experience. I was waiting for the end of competition, but It seems now that the competition will be focused on how much leaks are in the data, and not anymore on ML modeling, I will share my code in the next days. \r\nI will be a bit busy in the following day at work, but I'll try anyhow to write a fast post during next evening. I'll let you know when it is published.", "votes": null }, { "id": "138492", "postDate": "10/09/2016 06:26:04", "content": "@Elena. Great, thank you. Spark & scala is working fine in Windows, only I have difficulty to compile XGboost4J, lots of error. I also wonder Xgboost moved to Scala first, while moving to Pyspark is closer, but I can understand it's an open source project with volunteers working in there. Your hardwork will eventually pay off, Spark will be shiny in the future as data getting bigger and bigger ;)
", "rawMarkdown": "Elena. Great, thank you. Spark & scala is working fine in Windows, only I have difficulty to compile XGboost4J, lots of error. I also wonder Xgboost moved to Scala first, while moving to Pyspark is closer, but I can understand it's an open source project with volunteers working in there. Your hardwork will eventually pay off, Spark will be shiny in the future as data getting bigger and bigger ;)", "votes": null }, { "id": "138681", "postDate": "10/10/2016 11:47:13", "content": "@aldente I write a fast post on my website\nhttp://www.elenacuoco.com/2016/10/10/scala-spark-xgboost-classification/\nIn the next days I will add more details for the setup
", "rawMarkdown": "aldente I write a fast post on my website\r\nhttp://www.elenacuoco.com/2016/10/10/scala-spark-xgboost-classification/\r\nIn the next days I will add more details for the setup", "votes": null }, { "id": "138758", "postDate": "10/10/2016 19:55:24", "content": "@Elena, I appreciate it very very much, thank you.
", "rawMarkdown": "Elena, I appreciate it very very much, thank you.", "votes": null }, { "id": "768587", "postDate": "03/11/2020 02:13:11", "content": "Hi.\nI'm sending this link:\nhttps://spark.rstudio.com/\nhttps://blog.rstudio.com/2016/09/27/sparklyr-r-interface-for-apache-spark/
\n\nregards
", "rawMarkdown": "Hi.\nI'm sending this link:\nhttps://spark.rstudio.com/\nhttps://blog.rstudio.com/2016/09/27/sparklyr-r-interface-for-apache-spark/\n\nregards", "votes": null } ], "comments": [ { "id": 133519, "author_name": "", "author_url": "", "post_date": "09/02/2016 17:32:11", "content": "Databricks platform could be helpful for your Spark-R cluster need.
\n\nhttps://databricks.com/try-databricks
\n\nCheers
", "votes": null, "replies": [] }, { "id": 134151, "author_name": "", "author_url": "", "post_date": "09/05/2016 12:47:41", "content": "Manuel has excellent tutorials and he also competes here.\nThis is probably what you are looking for:\nhttps://www.youtube.com/watch?v=ISsnKm2mAx4
", "votes": null, "replies": [] }, { "id": 134161, "author_name": "elenacuoco", "author_url": "", "post_date": "09/05/2016 13:24:25", "content": "I wrote this\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/\nbut using python and pyspark on a local machine.
", "votes": null, "replies": [] }, { "id": 134826, "author_name": "maxpowerwastaken", "author_url": "", "post_date": "09/08/2016 18:34:31", "content": "You might also want to check out Databricks' short (2-week) Edx Spark course ('BerkeleyX CS105x') using Python 2.7/PySpark.
\n\nhttps://courses.edx.org/courses/course-v1%3ABerkeleyX%2BCS105x%2B1T2016/info
", "votes": null, "replies": [] }, { "id": 136536, "author_name": "kmaayah", "author_url": "", "post_date": "09/22/2016 18:28:36", "content": "so i take it that using Spark (pyspark) is allowed for Kaggle competitions?
", "votes": null, "replies": [] }, { "id": 136626, "author_name": "", "author_url": "", "post_date": "09/23/2016 14:52:32", "content": "Yes no probs.
", "votes": null, "replies": [] }, { "id": 136688, "author_name": "", "author_url": "", "post_date": "09/24/2016 00:22:40", "content": "I've set up an Amazon EMR cluster and have used DataBricks, so feel free to ask me any questions. DataBricks basically gives you a managed EMR cluster and a Notebook UI for writing scripts using Spark.
\n\nOur team used Elena's code to get up and running, which allowed us to focus on just getting something working end-to-end without having to know much about Spark at the time.
\n\nIf you happen to live near SF or NYC, Amazon has pop-up lofts where you can ask Amazon engineers questions. They could walk you through the process of setting up an EMR cluster as well!
", "votes": null, "replies": [] }, { "id": 136954, "author_name": "prashantddeeep", "author_url": "", "post_date": "09/26/2016 20:48:38", "content": "Great job HueyKwik!\nAre your data files living on the clusters? If yes then do you let the clusters (billing) run continuously?
", "votes": null, "replies": [] }, { "id": 136976, "author_name": "daskaggle", "author_url": "", "post_date": "09/27/2016 03:32:40", "content": "Can anyone suggest me how to save a model in spark, when I am using pipline?\nI am using pyspark. I want to load the model & use rather than always training it, if I am not changing anything before training.
", "votes": null, "replies": [] }, { "id": 136988, "author_name": "", "author_url": "", "post_date": "09/27/2016 06:52:02", "content": "@AbhijitDas You can't if you are using spark < 2.0. \nThey have fixed that in Spark 2.0
", "votes": null, "replies": [] }, { "id": 137015, "author_name": "jpmiller", "author_url": "", "post_date": "09/27/2016 13:12:14", "content": "Felix^6 - did you make any progress on AWS? I decided to take the plunge and learned a few things so far:
\n\nYou can save time by starting with the EMR service. On the AWS Management Console, it's listed toward the bottom in the Analytics section. EMR has menu options that let you quickly spin up multiple EC2 instances with Hadoop and Spark properly configured. It also has options to install Hive, Pig, Zeppelin and other cool sounding things.
For setup, I recommend the advanced setup vs. the quick setup. Mainly because you can then use spot requests for your EC2 instances. I use default settings for most everything else. EMR will configure your security policy to work with Spark, which is nice. I found that I still had to go in and open Port 8787 for R-Studio and Port 22 for SSH.
Even so it's pretty amazing that a semi-trained windows user can get up and running with R-Studio or Jupyter Notebook over Spark! Here are some references that helped.
\n\n\n\n", "votes": null, "replies": [] }, { "id": 137019, "author_name": "akashsahaa", "author_url": "", "post_date": "09/27/2016 14:08:43", "content": "@Can Arel, Thanks for the reply. Could you pls post the code to save the model?\nI am using spark 2.0
", "votes": null, "replies": [] }, { "id": 137020, "author_name": "pavittd", "author_url": "", "post_date": "09/27/2016 14:17:17", "content": "I have been trying out the new RStudio package, sparklyR. It was easy to set up on my local machine\nhttp://spark.rstudio.com/
\n\nI just need to get it working on AWS now.
", "votes": null, "replies": [] }, { "id": 137021, "author_name": "", "author_url": "", "post_date": "09/27/2016 14:29:49", "content": "@Akash Saha\nEach model has a save/load function now. \nCheck out e.g. org.apache.spark.ml.classification Classes
", "votes": null, "replies": [] }, { "id": 137028, "author_name": "", "author_url": "", "post_date": "09/27/2016 16:51:47", "content": "@PrashantDeep - I leave the data files on S3, so while they are being billed continuously, it is relatively cheap.
", "votes": null, "replies": [] }, { "id": 137047, "author_name": "prashantddeeep", "author_url": "", "post_date": "09/27/2016 18:31:07", "content": "@HueyKwik\nThats a good option but when ever we begin the analysis, we will have to load the data files from S3 to the new clusters. That would take up lot of time every-time the analysis is started on new clusters especially in this case where the data files are really huge. \n@all - Whats are the good options to avoid this delay
", "votes": null, "replies": [] }, { "id": 138461, "author_name": "aldente", "author_url": "", "post_date": "10/08/2016 22:45:05", "content": "[quote=Elena Cuoco;134161]
\n\nI wrote this\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/\nbut using python and pyspark on a local machine.
\n\n[/quote]
\n\n@Elena, I read your post in the "Magical Feature" thread about spark+xgboost. I'm learning Spark, and having plan to practice spark in this dataset, also in single local machine. However in Windows OS the XGBoost4J is not supported yet. I would like to know, are you in Linux, Mac or Windows?
\n\nAlso would be interesting to read your forum/kernels report on spark+xgboost experiences. For me, this is more interesting than the leakages ;)
", "votes": null, "replies": [] }, { "id": 138488, "author_name": "elenacuoco", "author_url": "", "post_date": "10/09/2016 05:48:57", "content": "[quote=aldente;138461]
\n\n@Elena, I read your post in the "Magical Feature" thread about spark+xgboost. I'm learning Spark, and having plan to practice spark in this dataset, also in single local machine. However in Windows OS the XGBoost4J is not supported yet. I would like to know, are you in Linux, Mac or Windows?
\n\nAlso would be interesting to read your forum/kernels report on spark+xgboost experiences. For me, this is more interesting than the leakages ;)
\n\n[/quote]\n@aldente. I run on Linux and install each package from recent github release. Reading different forum these days to look for solutions and help for scala, it seems that someone did something for windows. \nI cannot help. I'm an linux and open source software addicted user since many years :)\nxgboost recently delivered an interface to java and scala for spark, not yet in python. I asked also to have a direct interface to Spark >2.0, where RDD are not more supported, but there are the DataFrame and they release un updated just one week ago.\nIt was an hard work for me to learn new language (scala) and write the code from reading csv data to write results in parquet and have xgboost running. I plan to write a post on my website on this experience. I was waiting for the end of competition, but It seems now that the competition will be focused on how much leaks are in the data, and not anymore on ML modeling, I will share my code in the next days. \nI will be a bit busy in the following day at work, but I'll try anyhow to write a fast post during next evening. I'll let you know when it is published.
", "votes": null, "replies": [] }, { "id": 138492, "author_name": "aldente", "author_url": "", "post_date": "10/09/2016 06:26:04", "content": "@Elena. Great, thank you. Spark & scala is working fine in Windows, only I have difficulty to compile XGboost4J, lots of error. I also wonder Xgboost moved to Scala first, while moving to Pyspark is closer, but I can understand it's an open source project with volunteers working in there. Your hardwork will eventually pay off, Spark will be shiny in the future as data getting bigger and bigger ;)
", "votes": null, "replies": [] }, { "id": 138681, "author_name": "elenacuoco", "author_url": "", "post_date": "10/10/2016 11:47:13", "content": "@aldente I write a fast post on my website\nhttp://www.elenacuoco.com/2016/10/10/scala-spark-xgboost-classification/\nIn the next days I will add more details for the setup
", "votes": null, "replies": [] }, { "id": 138758, "author_name": "aldente", "author_url": "", "post_date": "10/10/2016 19:55:24", "content": "@Elena, I appreciate it very very much, thank you.
", "votes": null, "replies": [] }, { "id": 768587, "author_name": "juancarlosmanotas", "author_url": "", "post_date": "03/11/2020 02:13:11", "content": "Hi.\nI'm sending this link:\nhttps://spark.rstudio.com/\nhttps://blog.rstudio.com/2016/09/27/sparklyr-r-interface-for-apache-spark/
\n\nregards
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "133491": "Hi guys,\r\n\r\nI am thinking about attacking this problem using R and Spark.\r\nI understand that there is a possibility to use Amazone AMR to set up the Spark cluster. It was described as very easy. However I don't know much about the whole issue. When I google to find a useful guide I find too many guides that describe all sorts of different tools and topics, throwing arround terms like HDFS, Hadoop, Hive, Zepelin and I don't know what.\r\n\r\nDoes anybody know a good guide to start with when you are really just hoping to start a Spark cluster, upload the data and run your R commands in that cluster to speed up things?\r\nOr is that not how it works in the first place. I envision A Spark cluster to be basically the same than my computer, just that there is several computers working together in the background.\r\n\r\nThanks in advance,\r\nFelix", "133519": "Databricks platform could be helpful for your Spark-R cluster need.\r\n\r\nhttps://databricks.com/try-databricks\r\n\r\nCheers", "134151": "Manuel has excellent tutorials and he also competes here.\r\nThis is probably what you are looking for:\r\nhttps://www.youtube.com/watch?v=ISsnKm2mAx4", "134161": "I wrote this\r\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/\r\nbut using python and pyspark on a local machine.", "134826": "You might also want to check out Databricks' short (2-week) Edx Spark course ('BerkeleyX CS105x') using Python 2.7/PySpark.\r\n\r\nhttps://courses.edx.org/courses/course-v1%3ABerkeleyX%2BCS105x%2B1T2016/info", "136536": "so i take it that using Spark (pyspark) is allowed for Kaggle competitions?", "136626": "Yes no probs.", "136688": "I've set up an Amazon EMR cluster and have used DataBricks, so feel free to ask me any questions. DataBricks basically gives you a managed EMR cluster and a Notebook UI for writing scripts using Spark. \r\n\r\nOur team used Elena's code to get up and running, which allowed us to focus on just getting something working end-to-end without having to know much about Spark at the time. \r\n\r\nIf you happen to live near SF or NYC, Amazon has pop-up lofts where you can ask Amazon engineers questions. They could walk you through the process of setting up an EMR cluster as well!", "136954": "Great job HueyKwik!\r\nAre your data files living on the clusters? If yes then do you let the clusters (billing) run continuously?", "136976": "Can anyone suggest me how to save a model in spark, when I am using pipline?\r\nI am using pyspark. I want to load the model & use rather than always training it, if I am not changing anything before training.", "136988": "AbhijitDas You can't if you are using spark < 2.0. \r\nThey have fixed that in Spark 2.0", "137015": "Felix^6 - did you make any progress on AWS? I decided to take the plunge and learned a few things so far:\r\n\r\n- You can save time by starting with the EMR service. On the AWS Management Console, it's listed toward the bottom in the Analytics section. EMR has menu options that let you quickly spin up multiple EC2 instances with Hadoop and Spark properly configured. It also has options to install Hive, Pig, Zeppelin and other cool sounding things.\r\n\r\n- For setup, I recommend the advanced setup vs. the quick setup. Mainly because you can then use spot requests for your EC2 instances. I use default settings for most everything else. EMR will configure your security policy to work with Spark, which is nice. I found that I still had to go in and open Port 8787 for R-Studio and Port 22 for SSH.\r\n- The thing I like least about it is that the instances are all based on Amazon Linux. Things don't seem nearly as easy as with Ubuntu and such. And the R-version that's installed is 3.2.x. Maybe not a big deal, but I like to run updated software. I spent many hours building R-3.3.1 from source and finally got it to run. And I get R-Studio up and running with the old R. But I have yet to bring both together - maybe today....\r\n\r\nEven so it's pretty amazing that a semi-trained windows user can get up and running with R-Studio or Jupyter Notebook over Spark! Here are some references that helped.\r\n\r\nhttps://blogs.aws.amazon.com/bigdata/post/TxIMFT77KLF6TA/Crunching-Statistics-at-Scale-with-SparkR-on-Amazon-EMR \r\n\r\nhttp://blogs.aws.amazon.com/bigdata/post/Tx37RSKRFDQNTSL/Statistical-Analysis-with-Open-Source-R-and-RStudio-on-Amazon-EMR", "137019": "Can Arel, Thanks for the reply. Could you pls post the code to save the model?\r\nI am using spark 2.0", "137020": "I have been trying out the new RStudio package, sparklyR. It was easy to set up on my local machine\r\nhttp://spark.rstudio.com/\r\n\r\nI just need to get it working on AWS now.", "137021": "Akash Saha\r\nEach model has a save/load function now. \r\nCheck out e.g. org.apache.spark.ml.classification Classes", "137028": "PrashantDeep - I leave the data files on S3, so while they are being billed continuously, it is relatively cheap.", "137047": "HueyKwik\r\nThats a good option but when ever we begin the analysis, we will have to load the data files from S3 to the new clusters. That would take up lot of time every-time the analysis is started on new clusters especially in this case where the data files are really huge. \r\n@all - Whats are the good options to avoid this delay", "138461": "[quote=Elena Cuoco;134161]\r\n\r\nI wrote this\r\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/\r\nbut using python and pyspark on a local machine.\r\n\r\n[/quote]\r\n\r\n@Elena, I read your post in the \"Magical Feature\" thread about spark+xgboost. I'm learning Spark, and having plan to practice spark in this dataset, also in single local machine. However in Windows OS the XGBoost4J is not supported yet. I would like to know, are you in Linux, Mac or Windows? \r\n\r\nAlso would be interesting to read your forum/kernels report on spark+xgboost experiences. For me, this is more interesting than the leakages ;)", "138488": "[quote=aldente;138461]\r\n\r\n\r\n@Elena, I read your post in the \"Magical Feature\" thread about spark+xgboost. I'm learning Spark, and having plan to practice spark in this dataset, also in single local machine. However in Windows OS the XGBoost4J is not supported yet. I would like to know, are you in Linux, Mac or Windows? \r\n\r\nAlso would be interesting to read your forum/kernels report on spark+xgboost experiences. For me, this is more interesting than the leakages ;) \r\n\r\n\r\n[/quote]\r\n@aldente. I run on Linux and install each package from recent github release. Reading different forum these days to look for solutions and help for scala, it seems that someone did something for windows. \r\nI cannot help. I'm an linux and open source software addicted user since many years :)\r\nxgboost recently delivered an interface to java and scala for spark, not yet in python. I asked also to have a direct interface to Spark >2.0, where RDD are not more supported, but there are the DataFrame and they release un updated just one week ago.\r\nIt was an hard work for me to learn new language (scala) and write the code from reading csv data to write results in parquet and have xgboost running. I plan to write a post on my website on this experience. I was waiting for the end of competition, but It seems now that the competition will be focused on how much leaks are in the data, and not anymore on ML modeling, I will share my code in the next days. \r\nI will be a bit busy in the following day at work, but I'll try anyhow to write a fast post during next evening. I'll let you know when it is published.", "138492": "Elena. Great, thank you. Spark & scala is working fine in Windows, only I have difficulty to compile XGboost4J, lots of error. I also wonder Xgboost moved to Scala first, while moving to Pyspark is closer, but I can understand it's an open source project with volunteers working in there. Your hardwork will eventually pay off, Spark will be shiny in the future as data getting bigger and bigger ;)", "138681": "aldente I write a fast post on my website\r\nhttp://www.elenacuoco.com/2016/10/10/scala-spark-xgboost-classification/\r\nIn the next days I will add more details for the setup", "138758": "Elena, I appreciate it very very much, thank you.", "768587": "Hi.\nI'm sending this link:\nhttps://spark.rstudio.com/\nhttps://blog.rstudio.com/2016/09/27/sparklyr-r-interface-for-apache-spark/\n\nregards" }, "source": "meta" }