{
  "id": 7728,
  "title": "reading transactions.csv hiccup ",
  "url": "/competitions/acquire-valued-shoppers-challenge/discussion/7728",
  "author_name": "",
  "post_date": "2014-04-17T01:03:51.543Z",
  "votes": null,
  "comment_count": 2,
  "views": 1350,
  "content": "<p>Hi All,</p>\n<p>Since&nbsp;transactions.csv is huge, I wrote my analysis program to read it line by line. It worked fine until it got to (I think) shopper ID&nbsp;4294967295. According to my data-reading script, it is a very long record, at least 3,506,000&nbsp;lines. After that I got a read error indicating that an unexpected (non-text) character started the next line. I used Hex Fiend to try and examine the transactions.csv file in the region of the error, however I could find no record with that ID using Hex Fiend.</p>\n<p>It is puzzling, perhaps my file format spec is off, or not robust, but it worked for many earlier records ('%u,%*u,%u,%u,%u,%u,%*u-%*u-%*u,%*u,%*s,%*u,%*f'). (At this point I'm only interested in who and what.)</p>\n<p>Perhaps Hex Fiend can't be trusted? I am also assuming that the IDs reside in a rank order.</p>\n<p>Any input would be great. I want to have a clue before I re-run a line-by-line re-read of the transactions file.</p>\n<p>Many thanks.</p>",
  "messages": [
    {
      "id": "42117",
      "postDate": "04/17/2014 01:03:51",
      "content": "<p>Hi All,</p>\n<p>Since&nbsp;transactions.csv is huge, I wrote my analysis program to read it line by line. It worked fine until it got to (I think) shopper ID&nbsp;4294967295. According to my data-reading script, it is a very long record, at least 3,506,000&nbsp;lines. After that I got a read error indicating that an unexpected (non-text) character started the next line. I used Hex Fiend to try and examine the transactions.csv file in the region of the error, however I could find no record with that ID using Hex Fiend.</p>\n<p>It is puzzling, perhaps my file format spec is off, or not robust, but it worked for many earlier records ('%u,%*u,%u,%u,%u,%u,%*u-%*u-%*u,%*u,%*s,%*u,%*f'). (At this point I'm only interested in who and what.)</p>\n<p>Perhaps Hex Fiend can't be trusted? I am also assuming that the IDs reside in a rank order.</p>\n<p>Any input would be great. I want to have a clue before I re-run a line-by-line re-read of the transactions file.</p>\n<p>Many thanks.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42121",
      "postDate": "04/17/2014 01:18:18",
      "content": "<p>coincidentally 2^32 - 4294967295 = 1</p>\n<p>:)&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42185",
      "postDate": "04/17/2014 12:09:39",
      "content": "<p>Got it, thanks! I'll retry reading as a&nbsp;64 bit unsigned integer.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 42121,
      "author_name": "grizzli3k",
      "author_url": "",
      "post_date": "04/17/2014 01:18:18",
      "content": "<p>coincidentally 2^32 - 4294967295 = 1</p>\n<p>:)&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42185,
      "author_name": "jay12185259",
      "author_url": "",
      "post_date": "04/17/2014 12:09:39",
      "content": "<p>Got it, thanks! I'll retry reading as a&nbsp;64 bit unsigned integer.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "42117": "",
    "42121": "",
    "42185": ""
  },
  "source": "meta"
}