{ "id": 5846, "title": "Trouble using Train.csv data in MATLAB", "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5846", "author_name": "", "post_date": "2013-09-22T22:19:26.933Z", "votes": null, "comment_count": 4, "views": 2659, "content": "

I am running: 8GB RAM, Windows 8, MATLAB 2013a

\n

1) I cannot open the train.csv file directly in excel / notepad (probably too big)

\n

2) Within MATLAB, I cannot right click-> Import Data (this method tries to load everything into memory which is not smart for this large a file)

\n

3) Using built in MATLAB functions "fgetl" or "fgets" does not return a complete row so it is difficult to parse

\n

4) I have tried using higher level MATLAB functions such as xlsread and csvread. However, this results in running out of memory (even though I have the java heap size set to MAX (2GB). 

\n

 

\n

So, I am not even able to properly access the data at this point or print something like:

\n

id = 1

\n

title = "some questions title"

\n

question text = "i have a question blah blah blah blah blah blah and so on and on"

\n

tags = "jpeg mime some keywords here"

\n

 

\n

So:

\n

1) Has anyone been successful at using the Train.csv file in MATLAB?

\n

2) Can someone try to duplicate the point 3) above i.e. does MATLAB return a complete row when you use fgets or fgetl?

\n

Is my data corrupted perhaps?

\n

 

\n

Note: Having similar issues with Test.csv. These files were downloaded at different times. So I am thinking / hoping it is not that the file is corrupt.

", "messages": [ { "id": "31360", "postDate": "09/22/2013 22:19:26", "content": "

I am running: 8GB RAM, Windows 8, MATLAB 2013a

\n

1) I cannot open the train.csv file directly in excel / notepad (probably too big)

\n

2) Within MATLAB, I cannot right click-> Import Data (this method tries to load everything into memory which is not smart for this large a file)

\n

3) Using built in MATLAB functions "fgetl" or "fgets" does not return a complete row so it is difficult to parse

\n

4) I have tried using higher level MATLAB functions such as xlsread and csvread. However, this results in running out of memory (even though I have the java heap size set to MAX (2GB). 

\n

 

\n

So, I am not even able to properly access the data at this point or print something like:

\n

id = 1

\n

title = "some questions title"

\n

question text = "i have a question blah blah blah blah blah blah and so on and on"

\n

tags = "jpeg mime some keywords here"

\n

 

\n

So:

\n

1) Has anyone been successful at using the Train.csv file in MATLAB?

\n

2) Can someone try to duplicate the point 3) above i.e. does MATLAB return a complete row when you use fgets or fgetl?

\n

Is my data corrupted perhaps?

\n

 

\n

Note: Having similar issues with Test.csv. These files were downloaded at different times. So I am thinking / hoping it is not that the file is corrupt.

", "rawMarkdown": "", "votes": null }, { "id": "31401", "postDate": "09/23/2013 14:25:57", "content": "

MATLAB has a few problems with this data file.  MATLAB struggles with text data sometimes and doesn't have the native types necessary to support it.  It also doesn't support csv files with newlines as part of the text entry. Getting the data into a useful format using MATLAB would be challenging.

\n

There is a workaround that I'd suggest.  I've used Python and a variant of the Bag-of-words demo found here to get the text data into easily usable numeric data.  From there I'm using MATLAB to prototype a few algorithms (still haven't submitted anything).

", "rawMarkdown": "", "votes": null }, { "id": "31525", "postDate": "09/25/2013 02:27:19", "content": "

Andrew, I like your approach better than mine. Definately much cleaner. I am going to follow the same route.

\n

 

\n

Here is an alternative that uses only MATLAB incase anyone is python shy.

\n

I was able to import the data using a single command:

\n

A = dataset('File', 'Train.csv');

\n

This produces a dataset with a single dimension (i.e. a string containing the id, title, question text and keywords). You can then split it up based on the locations of double quotes). dq_locs = strfind(string,'"') should do the trick. Caution: question text may have double quotes within them.

\n

Also, it is useful to note that the dataset command managed the memory by itself. I do not have sufficient memory to store the entire file in memory. However, I was okay. I was monitoring my memory as it was working. It would reach about 95% and then suddenly drop to about 16%. This pattern repeated itself several times.

\n

In the end, the data was stored in memory though so I was able to operate on it quickly! 

\n

Thanks

\n

Nikhil 

", "rawMarkdown": "", "votes": null }, { "id": "32418", "postDate": "10/18/2013 18:44:54", "content": "

Hello,

\n

I use python's CSV reader to read/parse the file.  Here is a little crappy python script:

\n

 

\n

https://github.com/philipnee/fbrecruit3

", "rawMarkdown": "", "votes": null }, { "id": "32419", "postDate": "10/18/2013 18:45:02", "content": "

Hello,

\n

I use python's CSV reader to read/parse the file.  Here is a little crappy python script:

\n

 

\n

https://github.com/philipnee/fbrecruit3

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 31401, "author_name": "andrewmatteson", "author_url": "", "post_date": "09/23/2013 14:25:57", "content": "

MATLAB has a few problems with this data file.  MATLAB struggles with text data sometimes and doesn't have the native types necessary to support it.  It also doesn't support csv files with newlines as part of the text entry. Getting the data into a useful format using MATLAB would be challenging.

\n

There is a workaround that I'd suggest.  I've used Python and a variant of the Bag-of-words demo found here to get the text data into easily usable numeric data.  From there I'm using MATLAB to prototype a few algorithms (still haven't submitted anything).

", "votes": null, "replies": [] }, { "id": 31525, "author_name": "nikhilagarwal1", "author_url": "", "post_date": "09/25/2013 02:27:19", "content": "

Andrew, I like your approach better than mine. Definately much cleaner. I am going to follow the same route.

\n

 

\n

Here is an alternative that uses only MATLAB incase anyone is python shy.

\n

I was able to import the data using a single command:

\n

A = dataset('File', 'Train.csv');

\n

This produces a dataset with a single dimension (i.e. a string containing the id, title, question text and keywords). You can then split it up based on the locations of double quotes). dq_locs = strfind(string,'"') should do the trick. Caution: question text may have double quotes within them.

\n

Also, it is useful to note that the dataset command managed the memory by itself. I do not have sufficient memory to store the entire file in memory. However, I was okay. I was monitoring my memory as it was working. It would reach about 95% and then suddenly drop to about 16%. This pattern repeated itself several times.

\n

In the end, the data was stored in memory though so I was able to operate on it quickly! 

\n

Thanks

\n

Nikhil 

", "votes": null, "replies": [] }, { "id": 32418, "author_name": "philipnee722", "author_url": "", "post_date": "10/18/2013 18:44:54", "content": "

Hello,

\n

I use python's CSV reader to read/parse the file.  Here is a little crappy python script:

\n

 

\n

https://github.com/philipnee/fbrecruit3

", "votes": null, "replies": [] }, { "id": 32419, "author_name": "philipnee722", "author_url": "", "post_date": "10/18/2013 18:45:02", "content": "

Hello,

\n

I use python's CSV reader to read/parse the file.  Here is a little crappy python script:

\n

 

\n

https://github.com/philipnee/fbrecruit3

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "31360": "", "31401": "", "31525": "", "32418": "", "32419": "" }, "source": "meta" }