{ "id": 5692, "title": "Really Fast PYPY code to load contest data", "url": "/competitions/accelerometer-biometric-competition/discussion/5692", "author_name": "", "post_date": "2013-09-07T19:18:47.157Z", "votes": null, "comment_count": 4, "views": 1799, "content": "

After waiting countless hours trying to re-load and re-process the train and test files for this contest, I investigated faster versions of Python and discovered PYPY.  

\n

PYPY has a JIT compiler for raw python code (not all libraries are supported).  But it runs REALLY fast (on par with straight c code)

\n

For anyone still struggling with this  Here is an excerpt from my generate features code.  I just cut and pasted some of the key lines, so you will have to tailor it for your needs...

\n

What you need to run this:

\n

1.Install PYPY

\n

2.Change permissions on the file to allow execute

\n

3.Run it from the command line:  ./gen_features.py

\n

 

\n

 

\n

#!/usr/bin/pypy

import os
import csv

parseStr = lambda x: float(x) if '.' in x else int(x)

allData = []
os.nice(9)

\n

f=open('train.csv')
reader = csv.reader(f)
for row in reader:
    if row[0]== 'T':  # skip first row.  TODO: move this before the loop
        continue
    else:   
        convertedRow = []
        for token in row:
            try:
                newTok = parseStr(token)
            except (ValueError):
                print token
                raise ValueError
            convertedRow.append(newTok)
        allData.append(convertedRow)
f.close()    


", "messages": [ { "id": "30445", "postDate": "09/07/2013 19:18:47", "content": "

After waiting countless hours trying to re-load and re-process the train and test files for this contest, I investigated faster versions of Python and discovered PYPY.  

\n

PYPY has a JIT compiler for raw python code (not all libraries are supported).  But it runs REALLY fast (on par with straight c code)

\n

For anyone still struggling with this  Here is an excerpt from my generate features code.  I just cut and pasted some of the key lines, so you will have to tailor it for your needs...

\n

What you need to run this:

\n

1.Install PYPY

\n

2.Change permissions on the file to allow execute

\n

3.Run it from the command line:  ./gen_features.py

\n

 

\n

 

\n

#!/usr/bin/pypy

import os
import csv

parseStr = lambda x: float(x) if '.' in x else int(x)

allData = []
os.nice(9)

\n

f=open('train.csv')
reader = csv.reader(f)
for row in reader:
    if row[0]== 'T':  # skip first row.  TODO: move this before the loop
        continue
    else:   
        convertedRow = []
        for token in row:
            try:
                newTok = parseStr(token)
            except (ValueError):
                print token
                raise ValueError
            convertedRow.append(newTok)
        allData.append(convertedRow)
f.close()    


", "rawMarkdown": "", "votes": null }, { "id": "30463", "postDate": "09/08/2013 02:07:11", "content": "

I suppose you meant:

\n

...

\n

os.nice(9)

\n

f = open('train.csv')

\n

...

", "rawMarkdown": "", "votes": null }, { "id": "30467", "postDate": "09/08/2013 03:28:53", "content": "

Yes...good catch.  (I had copy and paste issue.)

\n

 fixed above

", "rawMarkdown": "", "votes": null }, { "id": "30515", "postDate": "09/09/2013 02:38:05", "content": "

Another way of having very fast IO is to use the HDFS pandas support, eg:

\n

 

\n

import pandas as pd
store = pd.HDFStore('data_store.h5')
train = pd.read_csv('train.csv')
train.sort(['Device', 'T'], ascending = [True, True], inplace = True)
store['train'] = train
store.close()

", "rawMarkdown": "", "votes": null }, { "id": "30529", "postDate": "09/09/2013 13:32:44", "content": "

 

\n

 

\n

Thanks, I looked into HDF a bit but have not used it.  Seems like it saves a lot of time parsing and data conversion each time you want to load.

\n

Of course, the real benefit of  PYPY is the high speed number crunching AFTER your data is loaded.  The code I posted is actually slower in loading than pandas. 

\n

However, once loaded, PYPY can do high speed processing of the data.  I was able to reduce my feature generation from 4 hours down to 4 minutes (on test.csv)!  

\n

 

\n

 

\n

 

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 30463, "author_name": "ghegoo", "author_url": "", "post_date": "09/08/2013 02:07:11", "content": "

I suppose you meant:

\n

...

\n

os.nice(9)

\n

f = open('train.csv')

\n

...

", "votes": null, "replies": [] }, { "id": 30467, "author_name": "geringer", "author_url": "", "post_date": "09/08/2013 03:28:53", "content": "

Yes...good catch.  (I had copy and paste issue.)

\n

 fixed above

", "votes": null, "replies": [] }, { "id": 30515, "author_name": "ghegoo", "author_url": "", "post_date": "09/09/2013 02:38:05", "content": "

Another way of having very fast IO is to use the HDFS pandas support, eg:

\n

 

\n

import pandas as pd
store = pd.HDFStore('data_store.h5')
train = pd.read_csv('train.csv')
train.sort(['Device', 'T'], ascending = [True, True], inplace = True)
store['train'] = train
store.close()

", "votes": null, "replies": [] }, { "id": 30529, "author_name": "geringer", "author_url": "", "post_date": "09/09/2013 13:32:44", "content": "

 

\n

 

\n

Thanks, I looked into HDF a bit but have not used it.  Seems like it saves a lot of time parsing and data conversion each time you want to load.

\n

Of course, the real benefit of  PYPY is the high speed number crunching AFTER your data is loaded.  The code I posted is actually slower in loading than pandas. 

\n

However, once loaded, PYPY can do high speed processing of the data.  I was able to reduce my feature generation from 4 hours down to 4 minutes (on test.csv)!  

\n

 

\n

 

\n

 

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "30445": "", "30463": "", "30467": "", "30515": "", "30529": "" }, "source": "meta" }