With data being so huge, I afraid my low-performance computer could not catch up data of such size. Any recommendations on using Hadoop clusters to do the work? Any tutorials would be highly appreciated.
", "messages": [ { "id": "131819", "postDate": "08/19/2016 21:38:56", "content": "With data being so huge, I afraid my low-performance computer could not catch up data of such size. Any recommendations on using Hadoop clusters to do the work? Any tutorials would be highly appreciated.
", "rawMarkdown": "With data being so huge, I afraid my low-performance computer could not catch up data of such size. Any recommendations on using Hadoop clusters to do the work? Any tutorials would be highly appreciated.", "votes": null }, { "id": "131940", "postDate": "08/20/2016 19:57:31", "content": "Hi,
\n\nI don't want to endorse stuff here. But my normal procedure is:
\n\n=> explore small subsets of the data until I understand what I want to look at. By that I mean to have a specific set of questions that can only be answered when looking at the whole data (this doesn't ahve to be something fancy: simple descriptive stats of the whole data sets are probably the first use case) For this you can and should use the tools you are comfortable with.
\n\n=> prototype these questions in Spark version on your local computer
\n\n=> move the data to the AWS and execute your prototyped stuff there. You can quite easily roll out Spark there using scripts that are distributed with Spark (even though it might be considered a downside that you use the Spark standalone cluster here...and not yarn).
\n\nI am personally not very sophisticated in my approach here :). I just use amazon S3 as data storage and query it via Spark (I know I know ...less performant but easy to handle).
\n\nIf I have the time I will do it with this data set too and post the steps here...but can't make any promises right now.
\n\nI used this tutorial back then to get started with Spark on EC2\nhttps://spark.apache.org/docs/1.6.2/ec2-scripts.html
\n\nObviously the huge downside of this approach is: it costs money (even though if you protopying i we talking a couple of dollars here)\nHTH
", "rawMarkdown": "Hi, \r\n\r\nI don't want to endorse stuff here. But my normal procedure is:\r\n\r\n=> explore small subsets of the data until I understand what I want to look at. By that I mean to have a specific set of questions that can only be answered when looking at the whole data (this doesn't ahve to be something fancy: simple descriptive stats of the whole data sets are probably the first use case) For this you can and should use the tools you are comfortable with.\r\n\r\n=> prototype these questions in Spark version on your local computer\r\n\r\n=> move the data to the AWS and execute your prototyped stuff there. You can quite easily roll out Spark there using scripts that are distributed with Spark (even though it might be considered a downside that you use the Spark standalone cluster here...and not yarn). \r\n\r\nI am personally not very sophisticated in my approach here :). I just use amazon S3 as data storage and query it via Spark (I know I know ...less performant but easy to handle). \r\n\r\nIf I have the time I will do it with this data set too and post the steps here...but can't make any promises right now. \r\n\r\nI used this tutorial back then to get started with Spark on EC2\r\nhttps://spark.apache.org/docs/1.6.2/ec2-scripts.html\r\n\r\n**Obviously the huge downside of this approach is: it costs money** (even though if you protopying i we talking a couple of dollars here)\r\nHTH", "votes": null }, { "id": "132133", "postDate": "08/22/2016 20:32:39", "content": "Hi Ying,
\n\nI am using AWS EMR which ease quite a lot the installation part of spark on multiple machines. Moreover, you can install zeppelin which is a good alternative to ipython notebook. Other than that, I would follow Maggette suggestion and use a batch of data for prototyping and then test your idea with bigger machines.
\n\nCheers
", "rawMarkdown": "Hi Ying,\r\n\r\nI am using AWS EMR which ease quite a lot the installation part of spark on multiple machines. Moreover, you can install [zeppelin][1] which is a good alternative to ipython notebook. Other than that, I would follow Maggette suggestion and use a batch of data for prototyping and then test your idea with bigger machines. \r\n\r\nCheers\r\n\r\n [1]: https://zeppelin.apache.org/", "votes": null }, { "id": "132506", "postDate": "08/26/2016 06:34:00", "content": "If it helps, I use kdb+. Here I melt down the dataset from 4000 columns into 7 columns by storing line, station and feature numbers in its column. The table is partitioned by their Id number and parted attribute is added to Id. As the picture shows I need around 6 gb to store the train_numeric table instead 2gb using csv files. On the other hand I can query the data now.
\n\nThe other picture is actually showing that we can 'group' the Id by their feature sets and build models according to it?
\n\nCheers
", "rawMarkdown": "If it helps, I use kdb+. Here I melt down the dataset from 4000 columns into 7 columns by storing line, station and feature numbers in its column. The table is partitioned by their Id number and parted attribute is added to Id. As the picture shows I need around 6 gb to store the train_numeric table instead 2gb using csv files. On the other hand I can query the data now.\r\n\r\nThe other picture is actually showing that we can 'group' the Id by their feature sets and build models according to it?\r\n\r\nCheers", "votes": null }, { "id": "132589", "postDate": "08/27/2016 05:34:46", "content": "I'm not sure what language you're using, but if you're using Python, you can get a free Databricks Community Edition account which will give you access to a cluster with 6GB of RAM.
\n\nhttps://accounts.cloud.databricks.com/registration.html#signup/community
", "rawMarkdown": "I'm not sure what language you're using, but if you're using Python, you can get a free Databricks Community Edition account which will give you access to a cluster with 6GB of RAM. \r\n\r\nhttps://accounts.cloud.databricks.com/registration.html#signup/community", "votes": null }, { "id": "132636", "postDate": "08/27/2016 20:14:28", "content": "For initial exploratory analysis, I would rather use the docker image with any mpp database. I am using Vertica. The setting up and data loading process is relatively straight forward. Below is the link that you can follow.
\n\nhttps://github.com/sumitchawla/docker-vertica
", "rawMarkdown": "For initial exploratory analysis, I would rather use the docker image with any mpp database. I am using Vertica. The setting up and data loading process is relatively straight forward. Below is the link that you can follow.\r\n\r\nhttps://github.com/sumitchawla/docker-vertica", "votes": null }, { "id": "132721", "postDate": "08/28/2016 17:56:22", "content": "I started using Apache Spark. Here you can find a simple post to start with PySpark.\n I'm still working on how efficiently manage all the categorical features\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/
", "rawMarkdown": "I started using Apache Spark. Here you can find a simple post to start with PySpark.\r\n I'm still working on how efficiently manage all the categorical features\r\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/", "votes": null }, { "id": "132753", "postDate": "08/29/2016 02:19:40", "content": "Hi \nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.
", "rawMarkdown": "Hi \r\nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.", "votes": null }, { "id": "132755", "postDate": "08/29/2016 03:29:27", "content": "[quote=Haozhen;132753]
\n\nHi \nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.
\n\n[/quote]
\n\nGood point.
", "rawMarkdown": "[quote=Haozhen;132753]\r\n\r\nHi \r\nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.\r\n\r\n[/quote]\r\n\r\nGood point.", "votes": null }, { "id": "132842", "postDate": "08/29/2016 17:42:38", "content": "[quote=Haozhen;132753]
\n\nHi \nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.
\n\n[/quote]
\n\nI haven't a 64G machine and for the moment I don't want to use AWS. I want to learn Spark and PySpark and this competition can represents a good opportunity to learn how to work with Spark. That's the main reason for me :)
", "rawMarkdown": "[quote=Haozhen;132753]\r\n\r\nHi \r\nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.\r\n\r\n[/quote]\r\n\r\nI haven't a 64G machine and for the moment I don't want to use AWS. I want to learn Spark and PySpark and this competition can represents a good opportunity to learn how to work with Spark. That's the main reason for me :)", "votes": null }, { "id": "134046", "postDate": "09/04/2016 20:33:40", "content": "Hi\nWhich configuration you recommend for AWS-EMR ?, time, analysis, instance, etc-\nM.Blaschke
", "rawMarkdown": "Hi\r\nWhich configuration you recommend for AWS-EMR ?, time, analysis, instance, etc-\r\nM.Blaschke", "votes": null } ], "comments": [ { "id": 131940, "author_name": "maggette", "author_url": "", "post_date": "08/20/2016 19:57:31", "content": "Hi,
\n\nI don't want to endorse stuff here. But my normal procedure is:
\n\n=> explore small subsets of the data until I understand what I want to look at. By that I mean to have a specific set of questions that can only be answered when looking at the whole data (this doesn't ahve to be something fancy: simple descriptive stats of the whole data sets are probably the first use case) For this you can and should use the tools you are comfortable with.
\n\n=> prototype these questions in Spark version on your local computer
\n\n=> move the data to the AWS and execute your prototyped stuff there. You can quite easily roll out Spark there using scripts that are distributed with Spark (even though it might be considered a downside that you use the Spark standalone cluster here...and not yarn).
\n\nI am personally not very sophisticated in my approach here :). I just use amazon S3 as data storage and query it via Spark (I know I know ...less performant but easy to handle).
\n\nIf I have the time I will do it with this data set too and post the steps here...but can't make any promises right now.
\n\nI used this tutorial back then to get started with Spark on EC2\nhttps://spark.apache.org/docs/1.6.2/ec2-scripts.html
\n\nObviously the huge downside of this approach is: it costs money (even though if you protopying i we talking a couple of dollars here)\nHTH
", "votes": null, "replies": [] }, { "id": 132133, "author_name": "francesco90", "author_url": "", "post_date": "08/22/2016 20:32:39", "content": "Hi Ying,
\n\nI am using AWS EMR which ease quite a lot the installation part of spark on multiple machines. Moreover, you can install zeppelin which is a good alternative to ipython notebook. Other than that, I would follow Maggette suggestion and use a batch of data for prototyping and then test your idea with bigger machines.
\n\nCheers
", "votes": null, "replies": [] }, { "id": 132506, "author_name": "kimktang", "author_url": "", "post_date": "08/26/2016 06:34:00", "content": "If it helps, I use kdb+. Here I melt down the dataset from 4000 columns into 7 columns by storing line, station and feature numbers in its column. The table is partitioned by their Id number and parted attribute is added to Id. As the picture shows I need around 6 gb to store the train_numeric table instead 2gb using csv files. On the other hand I can query the data now.
\n\nThe other picture is actually showing that we can 'group' the Id by their feature sets and build models according to it?
\n\nCheers
", "votes": null, "replies": [] }, { "id": 132589, "author_name": "jmwoloso", "author_url": "", "post_date": "08/27/2016 05:34:46", "content": "I'm not sure what language you're using, but if you're using Python, you can get a free Databricks Community Edition account which will give you access to a cluster with 6GB of RAM.
\n\nhttps://accounts.cloud.databricks.com/registration.html#signup/community
", "votes": null, "replies": [] }, { "id": 132636, "author_name": "mtiwari", "author_url": "", "post_date": "08/27/2016 20:14:28", "content": "For initial exploratory analysis, I would rather use the docker image with any mpp database. I am using Vertica. The setting up and data loading process is relatively straight forward. Below is the link that you can follow.
\n\nhttps://github.com/sumitchawla/docker-vertica
", "votes": null, "replies": [] }, { "id": 132721, "author_name": "elenacuoco", "author_url": "", "post_date": "08/28/2016 17:56:22", "content": "I started using Apache Spark. Here you can find a simple post to start with PySpark.\n I'm still working on how efficiently manage all the categorical features\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/
", "votes": null, "replies": [] }, { "id": 132753, "author_name": "alanwuxm", "author_url": "", "post_date": "08/29/2016 02:19:40", "content": "Hi \nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.
", "votes": null, "replies": [] }, { "id": 132755, "author_name": "jmwoloso", "author_url": "", "post_date": "08/29/2016 03:29:27", "content": "[quote=Haozhen;132753]
\n\nHi \nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.
\n\n[/quote]
\n\nGood point.
", "votes": null, "replies": [] }, { "id": 132842, "author_name": "elenacuoco", "author_url": "", "post_date": "08/29/2016 17:42:38", "content": "[quote=Haozhen;132753]
\n\nHi \nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.
\n\n[/quote]
\n\nI haven't a 64G machine and for the moment I don't want to use AWS. I want to learn Spark and PySpark and this competition can represents a good opportunity to learn how to work with Spark. That's the main reason for me :)
", "votes": null, "replies": [] }, { "id": 134046, "author_name": "mblaschke", "author_url": "", "post_date": "09/04/2016 20:33:40", "content": "Hi\nWhich configuration you recommend for AWS-EMR ?, time, analysis, instance, etc-\nM.Blaschke
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "131819": "With data being so huge, I afraid my low-performance computer could not catch up data of such size. Any recommendations on using Hadoop clusters to do the work? Any tutorials would be highly appreciated.", "131940": "Hi, \r\n\r\nI don't want to endorse stuff here. But my normal procedure is:\r\n\r\n=> explore small subsets of the data until I understand what I want to look at. By that I mean to have a specific set of questions that can only be answered when looking at the whole data (this doesn't ahve to be something fancy: simple descriptive stats of the whole data sets are probably the first use case) For this you can and should use the tools you are comfortable with.\r\n\r\n=> prototype these questions in Spark version on your local computer\r\n\r\n=> move the data to the AWS and execute your prototyped stuff there. You can quite easily roll out Spark there using scripts that are distributed with Spark (even though it might be considered a downside that you use the Spark standalone cluster here...and not yarn). \r\n\r\nI am personally not very sophisticated in my approach here :). I just use amazon S3 as data storage and query it via Spark (I know I know ...less performant but easy to handle). \r\n\r\nIf I have the time I will do it with this data set too and post the steps here...but can't make any promises right now. \r\n\r\nI used this tutorial back then to get started with Spark on EC2\r\nhttps://spark.apache.org/docs/1.6.2/ec2-scripts.html\r\n\r\n**Obviously the huge downside of this approach is: it costs money** (even though if you protopying i we talking a couple of dollars here)\r\nHTH", "132133": "Hi Ying,\r\n\r\nI am using AWS EMR which ease quite a lot the installation part of spark on multiple machines. Moreover, you can install [zeppelin][1] which is a good alternative to ipython notebook. Other than that, I would follow Maggette suggestion and use a batch of data for prototyping and then test your idea with bigger machines. \r\n\r\nCheers\r\n\r\n [1]: https://zeppelin.apache.org/", "132506": "If it helps, I use kdb+. Here I melt down the dataset from 4000 columns into 7 columns by storing line, station and feature numbers in its column. The table is partitioned by their Id number and parted attribute is added to Id. As the picture shows I need around 6 gb to store the train_numeric table instead 2gb using csv files. On the other hand I can query the data now.\r\n\r\nThe other picture is actually showing that we can 'group' the Id by their feature sets and build models according to it?\r\n\r\nCheers", "132589": "I'm not sure what language you're using, but if you're using Python, you can get a free Databricks Community Edition account which will give you access to a cluster with 6GB of RAM. \r\n\r\nhttps://accounts.cloud.databricks.com/registration.html#signup/community", "132636": "For initial exploratory analysis, I would rather use the docker image with any mpp database. I am using Vertica. The setting up and data loading process is relatively straight forward. Below is the link that you can follow.\r\n\r\nhttps://github.com/sumitchawla/docker-vertica", "132721": "I started using Apache Spark. Here you can find a simple post to start with PySpark.\r\n I'm still working on how efficiently manage all the categorical features\r\nhttp://www.elenacuoco.com/2016/08/28/pyspark-first-approaches-ml-classification/", "132753": "Hi \r\nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.", "132755": "[quote=Haozhen;132753]\r\n\r\nHi \r\nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.\r\n\r\n[/quote]\r\n\r\nGood point.", "132842": "[quote=Haozhen;132753]\r\n\r\nHi \r\nI think this data size doesn't need to use distributed system. An AWS EC2 m4.2xlarge instance should be enough and should let you work on the problem faster without worrying the configuration and resource management of Hadoop and Spark. Moreover, distributed package usually sacrifice accuracy. If the data is too large that cannot fit into a single 64G memory instance, I would definitely use distributed system.\r\n\r\n[/quote]\r\n\r\nI haven't a 64G machine and for the moment I don't want to use AWS. I want to learn Spark and PySpark and this competition can represents a good opportunity to learn how to work with Spark. That's the main reason for me :)", "134046": "Hi\r\nWhich configuration you recommend for AWS-EMR ?, time, analysis, instance, etc-\r\nM.Blaschke" }, "source": "meta" }