{
  "id": 71042,
  "title": "Has this become a \"big data\" competition ?",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/71042",
  "author_name": "Y Assous",
  "post_date": "2018-11-09T14:21:41.698000",
  "votes": 6,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hello Everyone !</p>\n\n<p>Finally the new CSV has arrived and what a surprise !.. 30,76 gb of data..</p>\n\n<p>At first, this competition was nice because we could work with the Kaggel's Kernels.. But now it seems that Kernels doesn't have enough ram (16gRAM), so do my computer.. (32gRAM). </p>\n\n<p>Does anyone have a solution to continue to use kernels, or computer with less than 32 Ram ? I believe that dropping columns with only NaN values could help, but would it be enough ?</p>",
  "messages": [
    {
      "id": 418238,
      "postDate": "2018-11-09T14:21:41.700Z",
      "content": "<p>Hello Everyone !</p>\n\n<p>Finally the new CSV has arrived and what a surprise !.. 30,76 gb of data..</p>\n\n<p>At first, this competition was nice because we could work with the Kaggel's Kernels.. But now it seems that Kernels doesn't have enough ram (16gRAM), so do my computer.. (32gRAM). </p>\n\n<p>Does anyone have a solution to continue to use kernels, or computer with less than 32 Ram ? I believe that dropping columns with only NaN values could help, but would it be enough ?</p>",
      "rawMarkdown": "Hello Everyone !\n\nFinally the new CSV has arrived and what a surprise !.. 30,76 gb of data..\n\nAt first, this competition was nice because we could work with the Kaggel's Kernels.. But now it seems that Kernels doesn't have enough ram (16gRAM), so do my computer.. (32gRAM). \n\nDoes anyone have a solution to continue to use kernels, or computer with less than 32 Ram ? I believe that dropping columns with only NaN values could help, but would it be enough ?\n\n",
      "votes": 6
    },
    {
      "id": 419872,
      "postDate": "2018-11-12T17:16:15.410Z",
      "content": "<p>Kaggle should increase the Memory of their kernels, now a lot of people can't deal anymore with this competition :(</p>",
      "rawMarkdown": "Kaggle should increase the Memory of their kernels, now a lot of people can't deal anymore with this competition :(",
      "votes": 1
    },
    {
      "id": 418995,
      "postDate": "2018-11-11T03:33:12.063Z",
      "content": "<p>Hi!, Y Assous, previously I have used the recommendations from this Kernel:\n<a href=\"https://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows\">https://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows</a></p>",
      "rawMarkdown": "Hi!, Y Assous, previously I have used the recommendations from this Kernel:\nhttps://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows",
      "votes": 1
    },
    {
      "id": 418246,
      "postDate": "2018-11-09T14:32:10.497Z",
      "content": "<p>Hi,\nIn R, the train dataset is 21.1Gb. \nThe size of the data.frame falls to ~800Mo by removing 'hits' column.</p>",
      "rawMarkdown": "Hi,\nIn R, the train dataset is 21.1Gb. \nThe size of the data.frame falls to ~800Mo by removing 'hits' column.",
      "votes": 2,
      "replies": [
        {
          "id": 418255,
          "postDate": "2018-11-09T14:40:29.203Z",
          "content": "<p>Hello Vincent ! \nThanks for the information ! Still 5,1 Gb too much for Kernels.. But that's a progress !</p>",
          "rawMarkdown": "Hello Vincent ! \nThanks for the information ! Still 5,1 Gb too much for Kernels.. But that's a progress !"
        },
        {
          "id": 418290,
          "postDate": "2018-11-09T15:47:02.390Z",
          "rawMarkdown": "",
          "votes": 1,
          "isDeleted": true
        },
        {
          "id": 418293,
          "postDate": "2018-11-09T15:53:00.407Z",
          "content": "<p>Thanks pratik ! I've seen this post too. I'm wondering if it still has constant values in the new dataset.</p>",
          "rawMarkdown": "Thanks pratik ! I've seen this post too. I'm wondering if it still has constant values in the new dataset."
        }
      ]
    },
    {
      "id": 420001,
      "postDate": "2018-11-12T22:53:55.947Z",
      "content": "<p>Probably we should pre-process each Json column individually and only keep what is really important, I'm going to start working on a function</p>",
      "rawMarkdown": "Probably we should pre-process each Json column individually and only keep what is really important, I'm going to start working on a function"
    }
  ],
  "comments": [
    {
      "id": 419872,
      "author_name": "PierreCo",
      "author_url": "",
      "post_date": "2018-11-12T17:16:15.410000",
      "content": "<p>Kaggle should increase the Memory of their kernels, now a lot of people can't deal anymore with this competition :(</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 418995,
      "author_name": "C4rl05/V",
      "author_url": "",
      "post_date": "2018-11-11T03:33:12.063000",
      "content": "<p>Hi!, Y Assous, previously I have used the recommendations from this Kernel:\n<a href=\"https://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows\">https://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows</a></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 418246,
      "author_name": "Vincent Larmet",
      "author_url": "",
      "post_date": "2018-11-09T14:32:10.497000",
      "content": "<p>Hi,\nIn R, the train dataset is 21.1Gb. \nThe size of the data.frame falls to ~800Mo by removing 'hits' column.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 418255,
          "author_name": "Y Assous",
          "author_url": "",
          "post_date": "2018-11-09T14:40:29.203000",
          "content": "<p>Hello Vincent ! \nThanks for the information ! Still 5,1 Gb too much for Kernels.. But that's a progress !</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 418290,
          "author_name": "",
          "author_url": "",
          "post_date": "2018-11-09T15:47:02.390000",
          "content": "",
          "votes": 1,
          "replies": []
        },
        {
          "id": 418293,
          "author_name": "Y Assous",
          "author_url": "",
          "post_date": "2018-11-09T15:53:00.407000",
          "content": "<p>Thanks pratik ! I've seen this post too. I'm wondering if it still has constant values in the new dataset.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 420001,
      "author_name": "C4rl05/V",
      "author_url": "",
      "post_date": "2018-11-12T22:53:55.947000",
      "content": "<p>Probably we should pre-process each Json column individually and only keep what is really important, I'm going to start working on a function</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "418238": "Hello Everyone !\n\nFinally the new CSV has arrived and what a surprise !.. 30,76 gb of data..\n\nAt first, this competition was nice because we could work with the Kaggel's Kernels.. But now it seems that Kernels doesn't have enough ram (16gRAM), so do my computer.. (32gRAM). \n\nDoes anyone have a solution to continue to use kernels, or computer with less than 32 Ram ? I believe that dropping columns with only NaN values could help, but would it be enough ?\n\n",
    "419872": "Kaggle should increase the Memory of their kernels, now a lot of people can't deal anymore with this competition :(",
    "418995": "Hi!, Y Assous, previously I have used the recommendations from this Kernel:\nhttps://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows",
    "418246": "Hi,\nIn R, the train dataset is 21.1Gb. \nThe size of the data.frame falls to ~800Mo by removing 'hits' column.",
    "420001": "Probably we should pre-process each Json column individually and only keep what is really important, I'm going to start working on a function"
  }
}