{
  "id": 23138,
  "title": "Chunked processing R",
  "url": "/competitions/bosch-production-line-performance/discussion/23138",
  "author_name": "",
  "post_date": "2016-08-28T13:48:12.317Z",
  "votes": null,
  "comment_count": 4,
  "views": 809,
  "content": "<p>I've written a blog post about getting started with this competition in R: </p>\n\n<p><a href=\"http://www.diaryofakaggler.com/2016/bosch-production-line-performance-first-steps/\">http://www.diaryofakaggler.com/2016/bosch-production-line-performance-first-steps/</a></p>\n\n<p>The basic idea is to read small chunks from the data files without decompressing them:</p>\n\n<pre><code>conn &lt;- unz(&quot;train_numeric.csv.zip&quot;,&quot;train_numeric.csv&quot;,&quot;r&quot;)\ncolumns &lt;- scan(conn,what=character(0),sep=&quot;,&quot;,nlines=1)\nread_chunk &lt;- function(chunk_size=50)\n{\n  raw &lt;- scan(conn,\n              what=numeric(0),\n              sep=&quot;,&quot;,\n              nlines=chunk_size,\n              quiet=TRUE)\n\n  chunk &lt;- as.data.frame(t(matrix(raw,nrow=length(columns))))\n  names(chunk) &lt;- columns\n  return(chunk)\n}\n</code></pre>\n\n<p>Feedback welcome.</p>",
  "messages": [
    {
      "id": "132690",
      "postDate": "08/28/2016 13:48:12",
      "content": "<p>I've written a blog post about getting started with this competition in R: </p>\n\n<p><a href=\"http://www.diaryofakaggler.com/2016/bosch-production-line-performance-first-steps/\">http://www.diaryofakaggler.com/2016/bosch-production-line-performance-first-steps/</a></p>\n\n<p>The basic idea is to read small chunks from the data files without decompressing them:</p>\n\n<pre><code>conn &lt;- unz(&quot;train_numeric.csv.zip&quot;,&quot;train_numeric.csv&quot;,&quot;r&quot;)\ncolumns &lt;- scan(conn,what=character(0),sep=&quot;,&quot;,nlines=1)\nread_chunk &lt;- function(chunk_size=50)\n{\n  raw &lt;- scan(conn,\n              what=numeric(0),\n              sep=&quot;,&quot;,\n              nlines=chunk_size,\n              quiet=TRUE)\n\n  chunk &lt;- as.data.frame(t(matrix(raw,nrow=length(columns))))\n  names(chunk) &lt;- columns\n  return(chunk)\n}\n</code></pre>\n\n<p>Feedback welcome.</p>",
      "rawMarkdown": "I've written a blog post about getting started with this competition in R: \r\n\r\nhttp://www.diaryofakaggler.com/2016/bosch-production-line-performance-first-steps/\r\n\r\nThe basic idea is to read small chunks from the data files without decompressing them:\r\n\r\n    conn <- unz(\"train_numeric.csv.zip\",\"train_numeric.csv\",\"r\")\r\n    columns <- scan(conn,what=character(0),sep=\",\",nlines=1)\r\n    read_chunk <- function(chunk_size=50)\r\n    {\r\n      raw <- scan(conn,\r\n                  what=numeric(0),\r\n                  sep=\",\",\r\n                  nlines=chunk_size,\r\n                  quiet=TRUE)\r\n  \r\n      chunk <- as.data.frame(t(matrix(raw,nrow=length(columns))))\r\n      names(chunk) <- columns\r\n      return(chunk)\r\n    }\r\n\r\nFeedback welcome.",
      "votes": null
    },
    {
      "id": "132692",
      "postDate": "08/28/2016 14:38:25",
      "content": "<p>Hi EndInTears,</p>\n\n<p>Your idea is similar to <em>read_csv_chunked()</em> in readr library.<br>\n<a href=\"https://blog.rstudio.org/2016/08/05/readr-1-0-0/\">https://blog.rstudio.org/2016/08/05/readr-1-0-0/</a>\n[quote]</p>\n\n<ul>\n<li>readr contains an experimental API for reading a file in chunks, e.g. read_csv_chunked() and read_lines_chunked(). These allow you to work with files that are bigger than memory. We haven&#8217;t yet finalised the API so please use with care, and send us your feedback.\n[/quote]\nThis function may help your code to become more simple.</li>\n</ul>",
      "rawMarkdown": "Hi EndInTears,\r\n\r\nYour idea is similar to *read_csv_chunked()* in readr library.<br>\r\nhttps://blog.rstudio.org/2016/08/05/readr-1-0-0/\r\n[quote]\r\n\r\n - readr contains an experimental API for reading a file in chunks, e.g. read_csv_chunked() and read_lines_chunked(). These allow you to work with files that are bigger than memory. We haven’t yet finalised the API so please use with care, and send us your feedback.\r\n[/quote]\r\nThis function may help your code to become more simple.",
      "votes": null
    },
    {
      "id": "132705",
      "postDate": "08/28/2016 16:06:11",
      "content": "<p>Thanks for the recommendation toshi_k. Looks like a useful package at the very least for not having to type stringsAsFactors=FALSE all the time...</p>",
      "rawMarkdown": "Thanks for the recommendation toshi_k. Looks like a useful package at the very least for not having to type stringsAsFactors=FALSE all the time...",
      "votes": null
    },
    {
      "id": "133040",
      "postDate": "08/31/2016 01:43:48",
      "content": "<p>Thanks for sharing this EndInTears, I was not able to put my head around where to start ... </p>",
      "rawMarkdown": "Thanks for sharing this EndInTears, I was not able to put my head around where to start ...",
      "votes": null
    },
    {
      "id": "133282",
      "postDate": "09/01/2016 09:05:03",
      "content": "<p>Need help with read_csv_chunked(). What is the default for chunked attribute ?</p>",
      "rawMarkdown": "Need help with read_csv_chunked(). What is the default for chunked attribute ?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 132692,
      "author_name": "toshik",
      "author_url": "",
      "post_date": "08/28/2016 14:38:25",
      "content": "<p>Hi EndInTears,</p>\n\n<p>Your idea is similar to <em>read_csv_chunked()</em> in readr library.<br>\n<a href=\"https://blog.rstudio.org/2016/08/05/readr-1-0-0/\">https://blog.rstudio.org/2016/08/05/readr-1-0-0/</a>\n[quote]</p>\n\n<ul>\n<li>readr contains an experimental API for reading a file in chunks, e.g. read_csv_chunked() and read_lines_chunked(). These allow you to work with files that are bigger than memory. We haven&#8217;t yet finalised the API so please use with care, and send us your feedback.\n[/quote]\nThis function may help your code to become more simple.</li>\n</ul>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132705,
      "author_name": "endintears",
      "author_url": "",
      "post_date": "08/28/2016 16:06:11",
      "content": "<p>Thanks for the recommendation toshi_k. Looks like a useful package at the very least for not having to type stringsAsFactors=FALSE all the time...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133040,
      "author_name": "patternproject",
      "author_url": "",
      "post_date": "08/31/2016 01:43:48",
      "content": "<p>Thanks for sharing this EndInTears, I was not able to put my head around where to start ... </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133282,
      "author_name": "arcluke",
      "author_url": "",
      "post_date": "09/01/2016 09:05:03",
      "content": "<p>Need help with read_csv_chunked(). What is the default for chunked attribute ?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "132690": "I've written a blog post about getting started with this competition in R: \r\n\r\nhttp://www.diaryofakaggler.com/2016/bosch-production-line-performance-first-steps/\r\n\r\nThe basic idea is to read small chunks from the data files without decompressing them:\r\n\r\n    conn <- unz(\"train_numeric.csv.zip\",\"train_numeric.csv\",\"r\")\r\n    columns <- scan(conn,what=character(0),sep=\",\",nlines=1)\r\n    read_chunk <- function(chunk_size=50)\r\n    {\r\n      raw <- scan(conn,\r\n                  what=numeric(0),\r\n                  sep=\",\",\r\n                  nlines=chunk_size,\r\n                  quiet=TRUE)\r\n  \r\n      chunk <- as.data.frame(t(matrix(raw,nrow=length(columns))))\r\n      names(chunk) <- columns\r\n      return(chunk)\r\n    }\r\n\r\nFeedback welcome.",
    "132692": "Hi EndInTears,\r\n\r\nYour idea is similar to *read_csv_chunked()* in readr library.<br>\r\nhttps://blog.rstudio.org/2016/08/05/readr-1-0-0/\r\n[quote]\r\n\r\n - readr contains an experimental API for reading a file in chunks, e.g. read_csv_chunked() and read_lines_chunked(). These allow you to work with files that are bigger than memory. We haven’t yet finalised the API so please use with care, and send us your feedback.\r\n[/quote]\r\nThis function may help your code to become more simple.",
    "132705": "Thanks for the recommendation toshi_k. Looks like a useful package at the very least for not having to type stringsAsFactors=FALSE all the time...",
    "133040": "Thanks for sharing this EndInTears, I was not able to put my head around where to start ...",
    "133282": "Need help with read_csv_chunked(). What is the default for chunked attribute ?"
  },
  "source": "meta"
}