I am trying to use Pyspark(on AWS cluster) on this dataset but I am facing challenges, like feature selection and feature engineering. And I have doubt can this technique be used for much bigger datasets(suppose > 1 TB)?Any Suggestions are greatly appreciated.
", "messages": [ { "id": "125000", "postDate": "06/24/2016 12:53:02", "content": "I am trying to use Pyspark(on AWS cluster) on this dataset but I am facing challenges, like feature selection and feature engineering. And I have doubt can this technique be used for much bigger datasets(suppose > 1 TB)?Any Suggestions are greatly appreciated.
", "rawMarkdown": "I am trying to use Pyspark(on AWS cluster) on this dataset but I am facing challenges, like **feature selection and feature engineering**. And I have doubt can this technique be used for much bigger datasets(suppose > 1 TB)?Any Suggestions are greatly appreciated.", "votes": null }, { "id": "145395", "postDate": "11/18/2016 02:53:16", "content": "As of Nov'16, Pyspark doesn't have any feature selection algorithms integrated with it. If you want to use the Spark environment for feature selection then i would recommend using the Scala API which has a feature importance algorithm integrated(1). But, if you're not comfortable with Scala and would like to use Python instead, then you can make use of the Scikit-learn - Spark integration package from the link mentioned below(2). Scikit has an extra-trees classifier which can be used to calculate feature importance(3). By integrating Scikit and Spark you can perform feature engineering.
\n\n1) Feature selection - Scala - https://issues.apache.org/jira/browse/SPARK-5133.
\n\n2) https://github.com/databricks/spark-sklearn.
\n\n3) http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html
\n\nCheers!
\n\nRishi
", "rawMarkdown": "As of Nov'16, Pyspark doesn't have any feature selection algorithms integrated with it. If you want to use the Spark environment for feature selection then i would recommend using the Scala API which has a feature importance algorithm integrated(1). But, if you're not comfortable with Scala and would like to use Python instead, then you can make use of the Scikit-learn - Spark integration package from the link mentioned below(2). Scikit has an extra-trees classifier which can be used to calculate feature importance(3). By integrating Scikit and Spark you can perform feature engineering. \r\n\r\n1) Feature selection - Scala - https://issues.apache.org/jira/browse/SPARK-5133.\r\n\r\n2) https://github.com/databricks/spark-sklearn.\r\n\r\n3) http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html\r\n\r\nCheers!\r\n\r\nRishi", "votes": null }, { "id": "148206", "postDate": "12/03/2016 16:29:55", "content": "Great, Thanks Rishi for the suggestion. I will definitely try that.
", "rawMarkdown": "Great, Thanks Rishi for the suggestion. I will definitely try that.", "votes": null }, { "id": "150679", "postDate": "12/16/2016 10:28:38", "content": "I am using PySpark for preprocessing and building the feature vectors for a similar competition. In this EDA, I show some features of PySpark SQL DataFrame: https://www.kaggle.com/gspmoreira/outbrain-click-prediction/unveiling-page-views-csv-with-pyspark
", "rawMarkdown": "I am using PySpark for preprocessing and building the feature vectors for a similar competition. In this EDA, I show some features of PySpark SQL DataFrame: https://www.kaggle.com/gspmoreira/outbrain-click-prediction/unveiling-page-views-csv-with-pyspark", "votes": null } ], "comments": [ { "id": 145395, "author_name": "rishisankineni", "author_url": "", "post_date": "11/18/2016 02:53:16", "content": "As of Nov'16, Pyspark doesn't have any feature selection algorithms integrated with it. If you want to use the Spark environment for feature selection then i would recommend using the Scala API which has a feature importance algorithm integrated(1). But, if you're not comfortable with Scala and would like to use Python instead, then you can make use of the Scikit-learn - Spark integration package from the link mentioned below(2). Scikit has an extra-trees classifier which can be used to calculate feature importance(3). By integrating Scikit and Spark you can perform feature engineering.
\n\n1) Feature selection - Scala - https://issues.apache.org/jira/browse/SPARK-5133.
\n\n2) https://github.com/databricks/spark-sklearn.
\n\n3) http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html
\n\nCheers!
\n\nRishi
", "votes": null, "replies": [] }, { "id": 148206, "author_name": "vinodpathak1991", "author_url": "", "post_date": "12/03/2016 16:29:55", "content": "Great, Thanks Rishi for the suggestion. I will definitely try that.
", "votes": null, "replies": [] }, { "id": 150679, "author_name": "gspmoreira", "author_url": "", "post_date": "12/16/2016 10:28:38", "content": "I am using PySpark for preprocessing and building the feature vectors for a similar competition. In this EDA, I show some features of PySpark SQL DataFrame: https://www.kaggle.com/gspmoreira/outbrain-click-prediction/unveiling-page-views-csv-with-pyspark
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "125000": "I am trying to use Pyspark(on AWS cluster) on this dataset but I am facing challenges, like **feature selection and feature engineering**. And I have doubt can this technique be used for much bigger datasets(suppose > 1 TB)?Any Suggestions are greatly appreciated.", "145395": "As of Nov'16, Pyspark doesn't have any feature selection algorithms integrated with it. If you want to use the Spark environment for feature selection then i would recommend using the Scala API which has a feature importance algorithm integrated(1). But, if you're not comfortable with Scala and would like to use Python instead, then you can make use of the Scikit-learn - Spark integration package from the link mentioned below(2). Scikit has an extra-trees classifier which can be used to calculate feature importance(3). By integrating Scikit and Spark you can perform feature engineering. \r\n\r\n1) Feature selection - Scala - https://issues.apache.org/jira/browse/SPARK-5133.\r\n\r\n2) https://github.com/databricks/spark-sklearn.\r\n\r\n3) http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html\r\n\r\nCheers!\r\n\r\nRishi", "148206": "Great, Thanks Rishi for the suggestion. I will definitely try that.", "150679": "I am using PySpark for preprocessing and building the feature vectors for a similar competition. In this EDA, I show some features of PySpark SQL DataFrame: https://www.kaggle.com/gspmoreira/outbrain-click-prediction/unveiling-page-views-csv-with-pyspark" }, "source": "meta" }