{
  "id": 24140,
  "title": "What is going on with R dada.table::fread and the datafiles ?",
  "url": "/competitions/bosch-production-line-performance/discussion/24140",
  "author_name": "",
  "post_date": "2016-10-06T14:27:31.903Z",
  "votes": null,
  "comment_count": 6,
  "views": 458,
  "content": "<p>I have a basic problem with &quot;fread&quot; after reading the data files...</p>\n\n<p>When I write the following: <em>fread('train_numeric.csv', select=1, nrow = 3)</em> ... I get:</p>\n\n<p>Id</p>\n\n<p>1:  4</p>\n\n<p>2:  6</p>\n\n<p>3:  7</p>\n\n<p>When I write : <em>fread('train_numeric.csv', select=1, skip= 3, nrow = 3)</em> ... I get:</p>\n\n<p>V1</p>\n\n<p>1:  7</p>\n\n<p>2:  9</p>\n\n<p>3: 11</p>\n\n<p><strong>Why is the third value (i.e. 7)  repeated ?</strong> <br>\nThis is not what should happen according to the data.table definition, and that is not either what I get when I perform the same operations on a data.table of my own making.\nI must be doing something very wrong... Thanks for your help !</p>",
  "messages": [
    {
      "id": "137998",
      "postDate": "10/06/2016 14:27:31",
      "content": "<p>I have a basic problem with &quot;fread&quot; after reading the data files...</p>\n\n<p>When I write the following: <em>fread('train_numeric.csv', select=1, nrow = 3)</em> ... I get:</p>\n\n<p>Id</p>\n\n<p>1:  4</p>\n\n<p>2:  6</p>\n\n<p>3:  7</p>\n\n<p>When I write : <em>fread('train_numeric.csv', select=1, skip= 3, nrow = 3)</em> ... I get:</p>\n\n<p>V1</p>\n\n<p>1:  7</p>\n\n<p>2:  9</p>\n\n<p>3: 11</p>\n\n<p><strong>Why is the third value (i.e. 7)  repeated ?</strong> <br>\nThis is not what should happen according to the data.table definition, and that is not either what I get when I perform the same operations on a data.table of my own making.\nI must be doing something very wrong... Thanks for your help !</p>",
      "rawMarkdown": "I have a basic problem with \"fread\" after reading the data files...\r\n\r\nWhen I write the following: *fread('train_numeric.csv', select=1, nrow = 3)* ... I get:\r\n   \r\nId\r\n\r\n1:  4\r\n\r\n2:  6\r\n\r\n3:  7\r\n\r\nWhen I write : *fread('train_numeric.csv', select=1, skip= 3, nrow = 3)* ... I get:\r\n  \r\n V1\r\n\r\n1:  7\r\n\r\n2:  9\r\n\r\n3: 11\r\n\r\n**Why is the third value (i.e. 7)  repeated ?**  \r\nThis is not what should happen according to the data.table definition, and that is not either what I get when I perform the same operations on a data.table of my own making.\r\nI must be doing something very wrong... Thanks for your help !",
      "votes": null
    },
    {
      "id": "138001",
      "postDate": "10/06/2016 14:33:29",
      "content": "<p>Look at the column name for each. Row 0 is the header when using skip, but reads differently when not using the skip parameter. </p>",
      "rawMarkdown": "Look at the column name for each. Row 0 is the header when using skip, but reads differently when not using the skip parameter.",
      "votes": null
    },
    {
      "id": "138002",
      "postDate": "10/06/2016 14:51:25",
      "content": "<p>OK but when I create my own data.table, it does not works teh same</p>\n\n<p>A &lt;- 1:10</p>\n\n<p>B &lt;- LETTERS[1:10]</p>\n\n<p>DT &lt;- as.data.table(cbind(A,B))</p>\n\n<p>write.csv(DT, 'DT.csv', row.names= FALSE)</p>\n\n<p><em>fread('DT.csv', select=1, nrow = 3)</em></p>\n\n<p>A</p>\n\n<p>1: 1</p>\n\n<p>2: 2</p>\n\n<p>3: 3</p>\n\n<p><em>fread('DT.csv', select=1, skip= 3, nrow = 3)</em>\n   3</p>\n\n<p>1: 4</p>\n\n<p>2: 5</p>\n\n<p>3: 6</p>",
      "rawMarkdown": "OK but when I create my own data.table, it does not works teh same\r\n\r\nA <- 1:10\r\n\r\nB <- LETTERS[1:10]\r\n\r\nDT <- as.data.table(cbind(A,B))\r\n\r\nwrite.csv(DT, 'DT.csv', row.names= FALSE)\r\n\r\n*fread('DT.csv', select=1, nrow = 3)*\r\n\r\n   A\r\n\r\n1: 1\r\n\r\n2: 2\r\n\r\n3: 3\r\n\r\n*fread('DT.csv', select=1, skip= 3, nrow = 3)*\r\n   3\r\n\r\n1: 4\r\n\r\n2: 5\r\n\r\n3: 6",
      "votes": null
    },
    {
      "id": "138007",
      "postDate": "10/06/2016 15:11:58",
      "content": "<p>Hi Claude - I had the same issue. Row 0 was the header when reading from the top, but it was counted as a data row when using skip. So for the second chunk, it's really only skipping 2 rows of data, the 3rd skipped row being the header.</p>\n\n<p>There is a 'header' parameter in fread, but I was having trouble getting it to work. I just went with something like this:</p>\n\n<pre><code> a &lt;- fread('train_numeric.csv', nrow = 3, skip = 0)\n b &lt;- fread('train_numeric.csv', nrow = 3, skip = 4)\n\n headnames &lt;- names(a)\n setnames(b, headnames)\n</code></pre>\n\n<p>Also, data.table can do rbind based on position without needing to match names. You may be able to go without named columns in all the chunks. </p>",
      "rawMarkdown": "Hi Claude - I had the same issue. Row 0 was the header when reading from the top, but it was counted as a data row when using skip. So for the second chunk, it's really only skipping 2 rows of data, the 3rd skipped row being the header.\r\n\r\nThere is a 'header' parameter in fread, but I was having trouble getting it to work. I just went with something like this:\r\n\r\n     a <- fread('train_numeric.csv', nrow = 3, skip = 0)\r\n     b <- fread('train_numeric.csv', nrow = 3, skip = 4)\r\n\r\n     headnames <- names(a)\r\n     setnames(b, headnames)\r\n\r\nAlso, data.table can do rbind based on position without needing to match names. You may be able to go without named columns in all the chunks.",
      "votes": null
    },
    {
      "id": "138014",
      "postDate": "10/06/2016 16:06:33",
      "content": "<p>OK JohnM, I get that, but why is it different on the small file that I create ? (see prior message)</p>",
      "rawMarkdown": "OK JohnM, I get that, but why is it different on the small file that I create ? (see prior message)",
      "votes": null
    },
    {
      "id": "138021",
      "postDate": "10/06/2016 16:59:19",
      "content": "<p>I was thinking it's because fread gets that your table has headers and doesn't count the header row as data to skip. But with the big ol train file, it doesn't see it that way. Obviously I don't know the root cause of this behavior, which is why I needed a work around. Good luck with it!</p>",
      "rawMarkdown": "I was thinking it's because fread gets that your table has headers and doesn't count the header row as data to skip. But with the big ol train file, it doesn't see it that way. Obviously I don't know the root cause of this behavior, which is why I needed a work around. Good luck with it!",
      "votes": null
    },
    {
      "id": "138023",
      "postDate": "10/06/2016 17:03:53",
      "content": "<p>@JohnM  Thanks !  </p>",
      "rawMarkdown": "JohnM  Thanks !",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 138001,
      "author_name": "fredtony",
      "author_url": "",
      "post_date": "10/06/2016 14:33:29",
      "content": "<p>Look at the column name for each. Row 0 is the header when using skip, but reads differently when not using the skip parameter. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138002,
      "author_name": "cdamp350",
      "author_url": "",
      "post_date": "10/06/2016 14:51:25",
      "content": "<p>OK but when I create my own data.table, it does not works teh same</p>\n\n<p>A &lt;- 1:10</p>\n\n<p>B &lt;- LETTERS[1:10]</p>\n\n<p>DT &lt;- as.data.table(cbind(A,B))</p>\n\n<p>write.csv(DT, 'DT.csv', row.names= FALSE)</p>\n\n<p><em>fread('DT.csv', select=1, nrow = 3)</em></p>\n\n<p>A</p>\n\n<p>1: 1</p>\n\n<p>2: 2</p>\n\n<p>3: 3</p>\n\n<p><em>fread('DT.csv', select=1, skip= 3, nrow = 3)</em>\n   3</p>\n\n<p>1: 4</p>\n\n<p>2: 5</p>\n\n<p>3: 6</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138007,
      "author_name": "jpmiller",
      "author_url": "",
      "post_date": "10/06/2016 15:11:58",
      "content": "<p>Hi Claude - I had the same issue. Row 0 was the header when reading from the top, but it was counted as a data row when using skip. So for the second chunk, it's really only skipping 2 rows of data, the 3rd skipped row being the header.</p>\n\n<p>There is a 'header' parameter in fread, but I was having trouble getting it to work. I just went with something like this:</p>\n\n<pre><code> a &lt;- fread('train_numeric.csv', nrow = 3, skip = 0)\n b &lt;- fread('train_numeric.csv', nrow = 3, skip = 4)\n\n headnames &lt;- names(a)\n setnames(b, headnames)\n</code></pre>\n\n<p>Also, data.table can do rbind based on position without needing to match names. You may be able to go without named columns in all the chunks. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138014,
      "author_name": "cdamp350",
      "author_url": "",
      "post_date": "10/06/2016 16:06:33",
      "content": "<p>OK JohnM, I get that, but why is it different on the small file that I create ? (see prior message)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138021,
      "author_name": "jpmiller",
      "author_url": "",
      "post_date": "10/06/2016 16:59:19",
      "content": "<p>I was thinking it's because fread gets that your table has headers and doesn't count the header row as data to skip. But with the big ol train file, it doesn't see it that way. Obviously I don't know the root cause of this behavior, which is why I needed a work around. Good luck with it!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138023,
      "author_name": "cdamp350",
      "author_url": "",
      "post_date": "10/06/2016 17:03:53",
      "content": "<p>@JohnM  Thanks !  </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "137998": "I have a basic problem with \"fread\" after reading the data files...\r\n\r\nWhen I write the following: *fread('train_numeric.csv', select=1, nrow = 3)* ... I get:\r\n   \r\nId\r\n\r\n1:  4\r\n\r\n2:  6\r\n\r\n3:  7\r\n\r\nWhen I write : *fread('train_numeric.csv', select=1, skip= 3, nrow = 3)* ... I get:\r\n  \r\n V1\r\n\r\n1:  7\r\n\r\n2:  9\r\n\r\n3: 11\r\n\r\n**Why is the third value (i.e. 7)  repeated ?**  \r\nThis is not what should happen according to the data.table definition, and that is not either what I get when I perform the same operations on a data.table of my own making.\r\nI must be doing something very wrong... Thanks for your help !",
    "138001": "Look at the column name for each. Row 0 is the header when using skip, but reads differently when not using the skip parameter.",
    "138002": "OK but when I create my own data.table, it does not works teh same\r\n\r\nA <- 1:10\r\n\r\nB <- LETTERS[1:10]\r\n\r\nDT <- as.data.table(cbind(A,B))\r\n\r\nwrite.csv(DT, 'DT.csv', row.names= FALSE)\r\n\r\n*fread('DT.csv', select=1, nrow = 3)*\r\n\r\n   A\r\n\r\n1: 1\r\n\r\n2: 2\r\n\r\n3: 3\r\n\r\n*fread('DT.csv', select=1, skip= 3, nrow = 3)*\r\n   3\r\n\r\n1: 4\r\n\r\n2: 5\r\n\r\n3: 6",
    "138007": "Hi Claude - I had the same issue. Row 0 was the header when reading from the top, but it was counted as a data row when using skip. So for the second chunk, it's really only skipping 2 rows of data, the 3rd skipped row being the header.\r\n\r\nThere is a 'header' parameter in fread, but I was having trouble getting it to work. I just went with something like this:\r\n\r\n     a <- fread('train_numeric.csv', nrow = 3, skip = 0)\r\n     b <- fread('train_numeric.csv', nrow = 3, skip = 4)\r\n\r\n     headnames <- names(a)\r\n     setnames(b, headnames)\r\n\r\nAlso, data.table can do rbind based on position without needing to match names. You may be able to go without named columns in all the chunks.",
    "138014": "OK JohnM, I get that, but why is it different on the small file that I create ? (see prior message)",
    "138021": "I was thinking it's because fread gets that your table has headers and doesn't count the header row as data to skip. But with the big ol train file, it doesn't see it that way. Obviously I don't know the root cause of this behavior, which is why I needed a work around. Good luck with it!",
    "138023": "JohnM  Thanks !"
  },
  "source": "meta"
}