{ "id": 10086, "title": "Memory error executing vanilla avito_ProhibitedContent_SampleCode.py", "url": "/competitions/avito-prohibited-content/discussion/10086", "author_name": "", "post_date": "2014-08-20T16:48:20.497Z", "votes": null, "comment_count": 2, "views": 3611, "content": "

Hi,

\n

After reading avito_ProhibitedContent_SampleCode I understood that I needed a "train_data.pkl".

\n

I noticed that the main procedure have four commented lines, followed by one uncommented line.

\n

I thought that these uncommented lines are the way to get "train_data.pkl" in the first execution. So the .tsv file can be processed, serialized and stored into "train_data.pkl"

\n

In order to get that, I inverted the comments, and executed the following main procedure:

\n

featureIndexes = processData(os.path.join(dataFolder,"avito_train.tsv"), itemsLimit=300000)
trainFeatures,trainTargets, trainItemIds=processData(os.path.join(dataFolder,"avito_train.tsv"), featureIndexes, itemsLimit=300000)
testFeatures, testItemIds=processData(os.path.join(dataFolder,"avito_test.tsv"), featureIndexes)
joblib.dump((trainFeatures, trainTargets, trainItemIds, testFeatures, testItemIds), os.path.join(dataFolder,"train_data.pkl"))
# trainFeatures, trainTargets, trainItemIds, testFeatures, testItemIds = joblib.load(os.path.join(dataFolder,"train_data.pkl"))

\n

But after the execution I got a memory error:

\n

[LINE:109]# DEBUG [2014-08-20 17:54:17,323] Generate features for avito_test
.tsv: 1350000 items done
[LINE:109]# DEBUG [2014-08-20 17:54:17,611] Generate features for avito_test
.tsv: 1351000 items done
Traceback (most recent call last):
File "avito_ProhibitedContent_SampleCode.py", line 158, in <module>
main()
File "avito_ProhibitedContent_SampleCode.py", line 131, in main
testFeatures, testItemIds=processData(os.path.join(dataFolder,"avito_test.ts
v"), featureIndexes)
File "avito_ProhibitedContent_SampleCode.py", line 120, in processData
features = sp.csr_matrix((np.ones(len(row)),(row,col)), shape=(cur_row, len(
featureIndexes)), dtype=np.float64)
File "W:\\Python27\\lib\\site-packages\\scipy\\sparse\\compressed.py", line 48, in _
_init__
other = self.__class__(coo_matrix(arg1, shape=shape))
File "W:\\Python27\\lib\\site-packages\\scipy\\sparse\\coo.py", line 139, in __init_
_
self.row = np.array(ij[0], copy=copy)
MemoryError

\n


My laptop is a 32 bit Windows Vista with 4GB (but using only 3GB as Vista does). I started executing it with 2GB free memory, and it seems like it has crashed with 1GB free memory (according to task manager graph but not very sure)

\n

What have I done wrong?

\n

Thanks in advance

", "messages": [ { "id": "52241", "postDate": "08/20/2014 16:48:20", "content": "

Hi,

\n

After reading avito_ProhibitedContent_SampleCode I understood that I needed a "train_data.pkl".

\n

I noticed that the main procedure have four commented lines, followed by one uncommented line.

\n

I thought that these uncommented lines are the way to get "train_data.pkl" in the first execution. So the .tsv file can be processed, serialized and stored into "train_data.pkl"

\n

In order to get that, I inverted the comments, and executed the following main procedure:

\n

featureIndexes = processData(os.path.join(dataFolder,"avito_train.tsv"), itemsLimit=300000)
trainFeatures,trainTargets, trainItemIds=processData(os.path.join(dataFolder,"avito_train.tsv"), featureIndexes, itemsLimit=300000)
testFeatures, testItemIds=processData(os.path.join(dataFolder,"avito_test.tsv"), featureIndexes)
joblib.dump((trainFeatures, trainTargets, trainItemIds, testFeatures, testItemIds), os.path.join(dataFolder,"train_data.pkl"))
# trainFeatures, trainTargets, trainItemIds, testFeatures, testItemIds = joblib.load(os.path.join(dataFolder,"train_data.pkl"))

\n

But after the execution I got a memory error:

\n

[LINE:109]# DEBUG [2014-08-20 17:54:17,323] Generate features for avito_test
.tsv: 1350000 items done
[LINE:109]# DEBUG [2014-08-20 17:54:17,611] Generate features for avito_test
.tsv: 1351000 items done
Traceback (most recent call last):
File "avito_ProhibitedContent_SampleCode.py", line 158, in <module>
main()
File "avito_ProhibitedContent_SampleCode.py", line 131, in main
testFeatures, testItemIds=processData(os.path.join(dataFolder,"avito_test.ts
v"), featureIndexes)
File "avito_ProhibitedContent_SampleCode.py", line 120, in processData
features = sp.csr_matrix((np.ones(len(row)),(row,col)), shape=(cur_row, len(
featureIndexes)), dtype=np.float64)
File "W:\\Python27\\lib\\site-packages\\scipy\\sparse\\compressed.py", line 48, in _
_init__
other = self.__class__(coo_matrix(arg1, shape=shape))
File "W:\\Python27\\lib\\site-packages\\scipy\\sparse\\coo.py", line 139, in __init_
_
self.row = np.array(ij[0], copy=copy)
MemoryError

\n


My laptop is a 32 bit Windows Vista with 4GB (but using only 3GB as Vista does). I started executing it with 2GB free memory, and it seems like it has crashed with 1GB free memory (according to task manager graph but not very sure)

\n

What have I done wrong?

\n

Thanks in advance

", "rawMarkdown": "", "votes": null }, { "id": "52288", "postDate": "08/21/2014 13:35:01", "content": "

I've noticed in the past windows having a limit of ~ 1GB ram per process.

\n

It might sound strange, but running a unix virtual machine instead may work for you.

\n

On a Win 7 host with a Virtual Box guest the benchmark runs fine for me.

\n

But I am on 64 bit.

", "rawMarkdown": "", "votes": null }, { "id": "53684", "postDate": "09/14/2014 10:09:02", "content": "

Problem solved using a different operating system

\n\n

Thanks a lot!

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 52288, "author_name": "devcon", "author_url": "", "post_date": "08/21/2014 13:35:01", "content": "

I've noticed in the past windows having a limit of ~ 1GB ram per process.

\n

It might sound strange, but running a unix virtual machine instead may work for you.

\n

On a Win 7 host with a Virtual Box guest the benchmark runs fine for me.

\n

But I am on 64 bit.

", "votes": null, "replies": [] }, { "id": 53684, "author_name": "virilo", "author_url": "", "post_date": "09/14/2014 10:09:02", "content": "

Problem solved using a different operating system

\n\n

Thanks a lot!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "52241": "", "52288": "", "53684": "" }, "source": "meta" }