{
  "id": 45061,
  "title": "Spark notebooks",
  "url": "/competitions/favorita-grocery-sales-forecasting/discussion/45061",
  "author_name": "Daniel Sobrado",
  "post_date": "2017-12-06T04:13:22.446000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>The availability of Spark (pyspark in python) will be very useful, Spark is introducing a lot of models (MLLib library for RDDs y ML for Dataframes) that can run in parallel and use less memory given that the executing es split into stages.</p>\n\n<p>For large datasets, the availability of Spark even in a single node with multiple cores node will be really helpful, for EDA tasks and also for modeling.</p>\n\n<p>Probably the best idea will be to use it in a cluster and have a quota of resources per use, depending on the characteristics of each competition.</p>",
  "messages": [
    {
      "id": 254027,
      "postDate": "2017-12-06T04:13:22.447Z",
      "content": "<p>The availability of Spark (pyspark in python) will be very useful, Spark is introducing a lot of models (MLLib library for RDDs y ML for Dataframes) that can run in parallel and use less memory given that the executing es split into stages.</p>\n\n<p>For large datasets, the availability of Spark even in a single node with multiple cores node will be really helpful, for EDA tasks and also for modeling.</p>\n\n<p>Probably the best idea will be to use it in a cluster and have a quota of resources per use, depending on the characteristics of each competition.</p>",
      "rawMarkdown": "The availability of Spark (pyspark in python) will be very useful, Spark is introducing a lot of models (MLLib library for RDDs y ML for Dataframes) that can run in parallel and use less memory given that the executing es split into stages.\n\nFor large datasets, the availability of Spark even in a single node with multiple cores node will be really helpful, for EDA tasks and also for modeling.\n\nProbably the best idea will be to use it in a cluster and have a quota of resources per use, depending on the characteristics of each competition.",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "254027": "The availability of Spark (pyspark in python) will be very useful, Spark is introducing a lot of models (MLLib library for RDDs y ML for Dataframes) that can run in parallel and use less memory given that the executing es split into stages.\n\nFor large datasets, the availability of Spark even in a single node with multiple cores node will be really helpful, for EDA tasks and also for modeling.\n\nProbably the best idea will be to use it in a cluster and have a quota of resources per use, depending on the characteristics of each competition."
  }
}