{
  "id": 71030,
  "title": "Loading train_v2 and test_v2 takes long time in R",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/71030",
  "author_name": "",
  "post_date": "2018-11-09T11:44:14.175320100Z",
  "votes": null,
  "comment_count": 7,
  "views": 0,
  "content": "<p>I tried loading the train_v2 and test_v2 using R with the below code.\ntrain&lt;-read.csv(\"../input/train_v2.csv\")\ntest&lt;-read.csv(\"../input/test_v2.csv\")\nIt takes very long time. I could not get the result.\nIs the number of rows much higher than 9lakh + that was available in previous dataset train &amp; test.</p>",
  "messages": [
    {
      "id": "418161",
      "postDate": "11/09/2018 11:44:14",
      "content": "<p>I tried loading the train_v2 and test_v2 using R with the below code.\ntrain&lt;-read.csv(\"../input/train_v2.csv\")\ntest&lt;-read.csv(\"../input/test_v2.csv\")\nIt takes very long time. I could not get the result.\nIs the number of rows much higher than 9lakh + that was available in previous dataset train &amp; test.</p>",
      "rawMarkdown": "I tried loading the train_v2 and test_v2 using R with the below code.\ntrain&lt;-read.csv(\"../input/train_v2.csv\")\ntest&lt;-read.csv(\"../input/test_v2.csv\")\nIt takes very long time. I could not get the result.\nIs the number of rows much higher than 9lakh + that was available in previous dataset train &amp; test.",
      "votes": null
    },
    {
      "id": "418181",
      "postDate": "11/09/2018 12:37:48",
      "content": "<p>Hi, \nYes it is, if you look at your available RAM when running the kernel you will see that you run out of memory when trying to load the datasets which wasn't the case with the previous datasets.</p>",
      "rawMarkdown": "Hi, \nYes it is, if you look at your available RAM when running the kernel you will see that you run out of memory when trying to load the datasets which wasn't the case with the previous datasets.",
      "votes": null
    },
    {
      "id": "418200",
      "postDate": "11/09/2018 13:20:07",
      "content": "<p>Use this to begin : read.csv(file=\"my.large.file.csv\",nrows=2000)\nSo you can manipulate the first thousands of rows</p>",
      "rawMarkdown": "Use this to begin : read.csv(file=\"my.large.file.csv\",nrows=2000)\nSo you can manipulate the first thousands of rows",
      "votes": null
    },
    {
      "id": "418303",
      "postDate": "11/09/2018 16:10:20",
      "content": "<p>One easy change you can make is to use read_csv() from the readr package.  The base read.csv function is extremely slow.</p>",
      "rawMarkdown": "One easy change you can make is to use read_csv() from the readr package.  The base read.csv function is extremely slow.",
      "votes": null
    },
    {
      "id": "418337",
      "postDate": "11/09/2018 17:20:32",
      "content": "<p>You may try <code>train &lt;- data.table::fread(\"../input/trainv2.csv\", drop = \"hits\", nrows =1e4)</code></p>\n\n<p>upd: this doesn't work either.</p>",
      "rawMarkdown": "You may try `train &lt;- data.table::fread(\"../input/trainv2.csv\", drop = \"hits\", nrows =1e4)`\n\nupd: this doesn't work either.",
      "votes": null
    },
    {
      "id": "418582",
      "postDate": "11/10/2018 07:08:48",
      "content": "<p>Try this:\n<a href=\"https://www.kaggle.com/burritodan/gstore-1-flatten\">https://www.kaggle.com/burritodan/gstore-1-flatten</a></p>",
      "rawMarkdown": "Try this:\nhttps://www.kaggle.com/burritodan/gstore-1-flatten",
      "votes": null
    },
    {
      "id": "418756",
      "postDate": "11/10/2018 15:04:24",
      "content": "<p>The problem is that all methods first take in the full 22 GByte before converting it to a decent size. Readr and fread  are fast when you predefine the data types of the columns, but you need the ram. </p>",
      "rawMarkdown": "The problem is that all methods first take in the full 22 GByte before converting it to a decent size. Readr and fread  are fast when you predefine the data types of the columns, but you need the ram.",
      "votes": null
    },
    {
      "id": "419357",
      "postDate": "11/11/2018 19:35:16",
      "content": "<p>This works for my old R scripts:</p>\n\n<pre><code>library(tidyverse)\nctypes = cols(fullVisitorId = col_character(),\n              channelGrouping = col_character(),\n              date = col_datetime(),\n              device = col_character(),\n              geoNetwork = col_character(),\n              socialEngagementType = col_skip(), \n              totals = col_character(),\n              trafficSource = col_character(),\n              visitId = col_integer(), \n              visitNumber = col_integer(),\n              visitStartTime = col_integer(),\n              hits = col_skip(),\n              customDimensions = col_skip())\ntr = read_csv(\"../input/train_v2.csv\", col_types = ctypes)\nte = read_csv(\"../input/test_v2.csv\", col_types = ctypes)\n</code></pre>",
      "rawMarkdown": "This works for my old R scripts:\n\n\n    library(tidyverse)\n    ctypes = cols(fullVisitorId = col_character(),\n                  channelGrouping = col_character(),\n                  date = col_datetime(),\n                  device = col_character(),\n                  geoNetwork = col_character(),\n                  socialEngagementType = col_skip(), \n                  totals = col_character(),\n                  trafficSource = col_character(),\n                  visitId = col_integer(), \n                  visitNumber = col_integer(),\n                  visitStartTime = col_integer(),\n                  hits = col_skip(),\n                  customDimensions = col_skip())\n    tr = read_csv(\"../input/train_v2.csv\", col_types = ctypes)\n    te = read_csv(\"../input/test_v2.csv\", col_types = ctypes)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 418181,
      "author_name": "antonylavoye",
      "author_url": "",
      "post_date": "11/09/2018 12:37:48",
      "content": "<p>Hi, \nYes it is, if you look at your available RAM when running the kernel you will see that you run out of memory when trying to load the datasets which wasn't the case with the previous datasets.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 418200,
      "author_name": "xtelima",
      "author_url": "",
      "post_date": "11/09/2018 13:20:07",
      "content": "<p>Use this to begin : read.csv(file=\"my.large.file.csv\",nrows=2000)\nSo you can manipulate the first thousands of rows</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 418303,
      "author_name": "dmitriyguller",
      "author_url": "",
      "post_date": "11/09/2018 16:10:20",
      "content": "<p>One easy change you can make is to use read_csv() from the readr package.  The base read.csv function is extremely slow.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 418337,
      "author_name": "kailex",
      "author_url": "",
      "post_date": "11/09/2018 17:20:32",
      "content": "<p>You may try <code>train &lt;- data.table::fread(\"../input/trainv2.csv\", drop = \"hits\", nrows =1e4)</code></p>\n\n<p>upd: this doesn't work either.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 418582,
      "author_name": "burritodan",
      "author_url": "",
      "post_date": "11/10/2018 07:08:48",
      "content": "<p>Try this:\n<a href=\"https://www.kaggle.com/burritodan/gstore-1-flatten\">https://www.kaggle.com/burritodan/gstore-1-flatten</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 418756,
          "author_name": "mrooijer",
          "author_url": "",
          "post_date": "11/10/2018 15:04:24",
          "content": "<p>The problem is that all methods first take in the full 22 GByte before converting it to a decent size. Readr and fread  are fast when you predefine the data types of the columns, but you need the ram. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 419357,
      "author_name": "kailex",
      "author_url": "",
      "post_date": "11/11/2018 19:35:16",
      "content": "<p>This works for my old R scripts:</p>\n\n<pre><code>library(tidyverse)\nctypes = cols(fullVisitorId = col_character(),\n              channelGrouping = col_character(),\n              date = col_datetime(),\n              device = col_character(),\n              geoNetwork = col_character(),\n              socialEngagementType = col_skip(), \n              totals = col_character(),\n              trafficSource = col_character(),\n              visitId = col_integer(), \n              visitNumber = col_integer(),\n              visitStartTime = col_integer(),\n              hits = col_skip(),\n              customDimensions = col_skip())\ntr = read_csv(\"../input/train_v2.csv\", col_types = ctypes)\nte = read_csv(\"../input/test_v2.csv\", col_types = ctypes)\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "418161": "I tried loading the train_v2 and test_v2 using R with the below code.\ntrain&lt;-read.csv(\"../input/train_v2.csv\")\ntest&lt;-read.csv(\"../input/test_v2.csv\")\nIt takes very long time. I could not get the result.\nIs the number of rows much higher than 9lakh + that was available in previous dataset train &amp; test.",
    "418181": "Hi, \nYes it is, if you look at your available RAM when running the kernel you will see that you run out of memory when trying to load the datasets which wasn't the case with the previous datasets.",
    "418200": "Use this to begin : read.csv(file=\"my.large.file.csv\",nrows=2000)\nSo you can manipulate the first thousands of rows",
    "418303": "One easy change you can make is to use read_csv() from the readr package.  The base read.csv function is extremely slow.",
    "418337": "You may try `train &lt;- data.table::fread(\"../input/trainv2.csv\", drop = \"hits\", nrows =1e4)`\n\nupd: this doesn't work either.",
    "418582": "Try this:\nhttps://www.kaggle.com/burritodan/gstore-1-flatten",
    "418756": "The problem is that all methods first take in the full 22 GByte before converting it to a decent size. Readr and fread  are fast when you predefine the data types of the columns, but you need the ram.",
    "419357": "This works for my old R scripts:\n\n\n    library(tidyverse)\n    ctypes = cols(fullVisitorId = col_character(),\n                  channelGrouping = col_character(),\n                  date = col_datetime(),\n                  device = col_character(),\n                  geoNetwork = col_character(),\n                  socialEngagementType = col_skip(), \n                  totals = col_character(),\n                  trafficSource = col_character(),\n                  visitId = col_integer(), \n                  visitNumber = col_integer(),\n                  visitStartTime = col_integer(),\n                  hits = col_skip(),\n                  customDimensions = col_skip())\n    tr = read_csv(\"../input/train_v2.csv\", col_types = ctypes)\n    te = read_csv(\"../input/test_v2.csv\", col_types = ctypes)"
  },
  "source": "meta"
}