{
  "id": 8249,
  "title": "Python Script to read the transactions.csv",
  "url": "/competitions/acquire-valued-shoppers-challenge/discussion/8249",
  "author_name": "",
  "post_date": "2014-05-22T08:15:11.283Z",
  "votes": 1,
  "comment_count": 2,
  "views": 1867,
  "content": "<p>I have been struggling to read the mega file and do some data manipulation.&nbsp;Here is a jump start python script that I wrote to read the transactions.csv. I have added functions to convert the date into julian format and also the changed the product measurement unit to an integer. This can be used as an input to the read.big.matrix (bigmemory) in R.</p>\n<p>import csv<br>import sys<br>import time<br>from jdcal import *<br>import hashlib</p>\n<p>i_f = open( 'transactions.csv', 'r' )<br>reader = csv.reader( i_f )</p>\n<p>outFile = open('transMod.csv','wb')<br>writer = csv.writer(outFile)</p>\n<p>headerline = reader.next()<br>writer.writerow(headerline)<br>lineNumber = 0<br>stack = [0]</p>\n<p>for line in reader:<br> print('lineNumber ',lineNumber)<br> (year, month, date) = line[6].strip().split('-')<br> lineNumber = lineNumber + 1 <br> (longjd, jd) = (gcal2jd(year, month, date))<br> stack.append(line[8])<br> line[6] = jd<br> line[8] = stack.index(line[8])<br> writer.writerow(line)</p>",
  "messages": [
    {
      "id": "46047",
      "postDate": "05/22/2014 08:15:11",
      "content": "<p>I have been struggling to read the mega file and do some data manipulation.&nbsp;Here is a jump start python script that I wrote to read the transactions.csv. I have added functions to convert the date into julian format and also the changed the product measurement unit to an integer. This can be used as an input to the read.big.matrix (bigmemory) in R.</p>\n<p>import csv<br>import sys<br>import time<br>from jdcal import *<br>import hashlib</p>\n<p>i_f = open( 'transactions.csv', 'r' )<br>reader = csv.reader( i_f )</p>\n<p>outFile = open('transMod.csv','wb')<br>writer = csv.writer(outFile)</p>\n<p>headerline = reader.next()<br>writer.writerow(headerline)<br>lineNumber = 0<br>stack = [0]</p>\n<p>for line in reader:<br> print('lineNumber ',lineNumber)<br> (year, month, date) = line[6].strip().split('-')<br> lineNumber = lineNumber + 1 <br> (longjd, jd) = (gcal2jd(year, month, date))<br> stack.append(line[8])<br> line[6] = jd<br> line[8] = stack.index(line[8])<br> writer.writerow(line)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "46069",
      "postDate": "05/22/2014 16:24:44",
      "content": "<p>Thanks for sharing .</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "116219",
      "postDate": "04/22/2016 15:24:28",
      "content": "<p>thanks for sharing!</p>",
      "rawMarkdown": "thanks for sharing!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 46069,
      "author_name": "pythonomic",
      "author_url": "",
      "post_date": "05/22/2014 16:24:44",
      "content": "<p>Thanks for sharing .</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116219,
      "author_name": "leyue260351",
      "author_url": "",
      "post_date": "04/22/2016 15:24:28",
      "content": "<p>thanks for sharing!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "46047": "",
    "46069": "",
    "116219": "thanks for sharing!"
  },
  "source": "meta"
}