{
  "id": 43239,
  "title": "Kernel dies when merging train and items data",
  "url": "/competitions/favorita-grocery-sales-forecasting/discussion/43239",
  "author_name": "",
  "post_date": "2017-11-12T00:28:29.589984100Z",
  "votes": null,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Unable to merge data from train.csv and items.csv by pd.merge function possibly due to the  memory limit of the kernel. Any alternate method to go about it?</p>",
  "messages": [
    {
      "id": "242565",
      "postDate": "11/12/2017 00:28:29",
      "content": "<p>Unable to merge data from train.csv and items.csv by pd.merge function possibly due to the  memory limit of the kernel. Any alternate method to go about it?</p>",
      "rawMarkdown": "Unable to merge data from train.csv and items.csv by pd.merge function possibly due to the  memory limit of the kernel. Any alternate method to go about it?",
      "votes": null
    },
    {
      "id": "242578",
      "postDate": "11/12/2017 01:49:17",
      "content": "<p>Same here. \ntrain.csv is too large for this operation, so I only use a portion of it.</p>",
      "rawMarkdown": "Same here. \ntrain.csv is too large for this operation, so I only use a portion of it.",
      "votes": null
    },
    {
      "id": "242597",
      "postDate": "11/12/2017 04:02:55",
      "content": "<p>Wouldnt that be fallacious?...rather than looking at the whole data set at once</p>",
      "rawMarkdown": "Wouldnt that be fallacious?...rather than looking at the whole data set at once",
      "votes": null
    },
    {
      "id": "243388",
      "postDate": "11/13/2017 23:15:49",
      "content": "<p>Trim the train set, i.e. don't use the complete data going back all the way to 2013. </p>",
      "rawMarkdown": "Trim the train set, i.e. don't use the complete data going back all the way to 2013.",
      "votes": null
    },
    {
      "id": "243599",
      "postDate": "11/14/2017 14:03:31",
      "content": "<p>You won't have that problem if you use a local instance of Spark, 4GB in the driver will be enough to avoid disk spilling:</p>\n\n<pre><code>\nimport pyspark\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession \\\n    .builder \\\n    .appName(\"Test Comp\") \\\n    .config(\"spark.driver.memory\",\"4g\")  \\\n    .getOrCreate()\n\ntrain=spark.read.csv(\"file:///D:/Python/Kaggle/train.csv\", inferSchema=True, header=True).cache()\nitems=spark.read.csv(\"file:///D:/Python/Kaggle/items.csv\", inferSchema=True, header=True).cache()\ntrain.createOrReplaceTempView(\"train\")\nitems.createOrReplaceTempView(\"items\")\n\ndata = spark.sql(\"\"\"SELECT  \n                    weekofyear(train.date) woy, \n                    dayofmonth(train.date) dom,\n                    date_format(train.date, 'E') dow,\n                    month(train.date) month, \n                    train.*,\n                    items.*\n             FROM train, items\n             WHERE train.item_nbr = items.item_nbr\n             \"\"\"\n         )\n</code></pre>",
      "rawMarkdown": "You won't have that problem if you use a local instance of Spark, 4GB in the driver will be enough to avoid disk spilling:\n<pre><code>\nimport pyspark\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession \\\n    .builder \\\n    .appName(\"Test Comp\") \\\n    .config(\"spark.driver.memory\",\"4g\")  \\\n    .getOrCreate()\n\ntrain=spark.read.csv(\"file:///D:/Python/Kaggle/train.csv\", inferSchema=True, header=True).cache()\nitems=spark.read.csv(\"file:///D:/Python/Kaggle/items.csv\", inferSchema=True, header=True).cache()\ntrain.createOrReplaceTempView(\"train\")\nitems.createOrReplaceTempView(\"items\")\n\ndata = spark.sql(\"\"\"SELECT  \n                    weekofyear(train.date) woy, \n                    dayofmonth(train.date) dom,\n                    date_format(train.date, 'E') dow,\n                    month(train.date) month, \n                    train.*,\n                    items.*\n             FROM train, items\n             WHERE train.item_nbr = items.item_nbr\n             \"\"\"\n         )\n</code></pre>",
      "votes": null
    },
    {
      "id": "243892",
      "postDate": "11/15/2017 05:07:35",
      "content": "<p>That sounds like a good option, Il definitely try it</p>",
      "rawMarkdown": "That sounds like a good option, Il definitely try it",
      "votes": null
    },
    {
      "id": "265944",
      "postDate": "01/07/2018 06:37:48",
      "content": "<p>Sorry ...but just couldn't stop asking ...I am quite a beginner here and have joined this competition to learn  analysing large data sets ... \nIn order to trim the  train data ... don't we have to remember that sales has trend and that it is cyclic in nature ...aren't we going to miss in those details ? so thats means can not sample train data and need to look into a year of train data . I am doing the analysis in R and struggling at every other step in terms of memory requirment ... </p>",
      "rawMarkdown": "Sorry ...but just couldn't stop asking ...I am quite a beginner here and have joined this competition to learn  analysing large data sets ... \nIn order to trim the  train data ... don't we have to remember that sales has trend and that it is cyclic in nature ...aren't we going to miss in those details ? so thats means can not sample train data and need to look into a year of train data . I am doing the analysis in R and struggling at every other step in terms of memory requirment ...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 242578,
      "author_name": "gaogao",
      "author_url": "",
      "post_date": "11/12/2017 01:49:17",
      "content": "<p>Same here. \ntrain.csv is too large for this operation, so I only use a portion of it.</p>",
      "votes": null,
      "replies": [
        {
          "id": 242597,
          "author_name": "miteshyadav",
          "author_url": "",
          "post_date": "11/12/2017 04:02:55",
          "content": "<p>Wouldnt that be fallacious?...rather than looking at the whole data set at once</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 243388,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "11/13/2017 23:15:49",
      "content": "<p>Trim the train set, i.e. don't use the complete data going back all the way to 2013. </p>",
      "votes": null,
      "replies": [
        {
          "id": 265944,
          "author_name": "vidyams",
          "author_url": "",
          "post_date": "01/07/2018 06:37:48",
          "content": "<p>Sorry ...but just couldn't stop asking ...I am quite a beginner here and have joined this competition to learn  analysing large data sets ... \nIn order to trim the  train data ... don't we have to remember that sales has trend and that it is cyclic in nature ...aren't we going to miss in those details ? so thats means can not sample train data and need to look into a year of train data . I am doing the analysis in R and struggling at every other step in terms of memory requirment ... </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 243599,
      "author_name": "danielsobrado",
      "author_url": "",
      "post_date": "11/14/2017 14:03:31",
      "content": "<p>You won't have that problem if you use a local instance of Spark, 4GB in the driver will be enough to avoid disk spilling:</p>\n\n<pre><code>\nimport pyspark\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession \\\n    .builder \\\n    .appName(\"Test Comp\") \\\n    .config(\"spark.driver.memory\",\"4g\")  \\\n    .getOrCreate()\n\ntrain=spark.read.csv(\"file:///D:/Python/Kaggle/train.csv\", inferSchema=True, header=True).cache()\nitems=spark.read.csv(\"file:///D:/Python/Kaggle/items.csv\", inferSchema=True, header=True).cache()\ntrain.createOrReplaceTempView(\"train\")\nitems.createOrReplaceTempView(\"items\")\n\ndata = spark.sql(\"\"\"SELECT  \n                    weekofyear(train.date) woy, \n                    dayofmonth(train.date) dom,\n                    date_format(train.date, 'E') dow,\n                    month(train.date) month, \n                    train.*,\n                    items.*\n             FROM train, items\n             WHERE train.item_nbr = items.item_nbr\n             \"\"\"\n         )\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 243892,
          "author_name": "miteshyadav",
          "author_url": "",
          "post_date": "11/15/2017 05:07:35",
          "content": "<p>That sounds like a good option, Il definitely try it</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "242565": "Unable to merge data from train.csv and items.csv by pd.merge function possibly due to the  memory limit of the kernel. Any alternate method to go about it?",
    "242578": "Same here. \ntrain.csv is too large for this operation, so I only use a portion of it.",
    "242597": "Wouldnt that be fallacious?...rather than looking at the whole data set at once",
    "243388": "Trim the train set, i.e. don't use the complete data going back all the way to 2013.",
    "243599": "You won't have that problem if you use a local instance of Spark, 4GB in the driver will be enough to avoid disk spilling:\n<pre><code>\nimport pyspark\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession \\\n    .builder \\\n    .appName(\"Test Comp\") \\\n    .config(\"spark.driver.memory\",\"4g\")  \\\n    .getOrCreate()\n\ntrain=spark.read.csv(\"file:///D:/Python/Kaggle/train.csv\", inferSchema=True, header=True).cache()\nitems=spark.read.csv(\"file:///D:/Python/Kaggle/items.csv\", inferSchema=True, header=True).cache()\ntrain.createOrReplaceTempView(\"train\")\nitems.createOrReplaceTempView(\"items\")\n\ndata = spark.sql(\"\"\"SELECT  \n                    weekofyear(train.date) woy, \n                    dayofmonth(train.date) dom,\n                    date_format(train.date, 'E') dow,\n                    month(train.date) month, \n                    train.*,\n                    items.*\n             FROM train, items\n             WHERE train.item_nbr = items.item_nbr\n             \"\"\"\n         )\n</code></pre>",
    "243892": "That sounds like a good option, Il definitely try it",
    "265944": "Sorry ...but just couldn't stop asking ...I am quite a beginner here and have joined this competition to learn  analysing large data sets ... \nIn order to trim the  train data ... don't we have to remember that sales has trend and that it is cyclic in nature ...aren't we going to miss in those details ? so thats means can not sample train data and need to look into a year of train data . I am doing the analysis in R and struggling at every other step in terms of memory requirment ..."
  },
  "source": "meta"
}