{ "id": 5594, "title": "Number of Train/Test Cases? (nested CSV issues)", "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5594", "author_name": "", "post_date": "2013-08-30T21:43:43.307Z", "votes": null, "comment_count": 9, "views": 3633, "content": "

Does anyone have a definitive count on train and test cases?
It seems as there might be some nested CSV data that's causing some unintended parsing.

", "messages": [ { "id": "29809", "postDate": "08/30/2013 21:43:43", "content": "

Does anyone have a definitive count on train and test cases?
It seems as there might be some nested CSV data that's causing some unintended parsing.

", "rawMarkdown": "", "votes": null }, { "id": "29812", "postDate": "08/30/2013 22:27:27", "content": "

The sample count should align with the Ids (starts at 1 in the train set and ends with 8047532 in the test set).

\n

This is a beast of a csv file, so we expect there to be a teething period and possibly some nasty edge cases.  Bear with us if the dataset has  bugs within.

\n

Mac users - you may have to use a program like Keka if the default unzipping  does not spit out a csv file for Train.zip.

", "rawMarkdown": "", "votes": null }, { "id": "29813", "postDate": "08/30/2013 22:42:53", "content": "

There's a few really nasty cases involving MATLAB data and code, it's mostly just nested \\n and " that throw things for a loop. Maybe URL encoding them, or using a uniquely guaranteed separator instead of "field","field"? I.e. remove all ~ from content, then use ~ as separators.

It might also be worthwhile to include a sample_train.csv and sample_test.csv, so participants can preview the data without potentially locking up slower (or virtual) machines.

", "rawMarkdown": "", "votes": null }, { "id": "29814", "postDate": "08/30/2013 22:44:44", "content": "

[quote=William Cukierski;29812]Mac users - you may have to use a program like Keka if the default unzipping  does not spit out a csv file for Train.zip.[/quote]

\n

Archiver.app seems to work. Produces:

\n

MD5 (Train.csv) = 880a8869d61bcce1f26072d9f4797cb9

\n

MD5 (Test.csv) = cac7cb73cf4061d8f74f9b5a094e66ab

", "rawMarkdown": "", "votes": null }, { "id": "29815", "postDate": "08/30/2013 22:47:19", "content": "

Adrian:

\n

I'm sure you know this, but just for the record a simple one-liner like

\n

head -n [number of lines] Train.csv > sample_train.csv

\n

should do just that.

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "29817", "postDate": "08/30/2013 23:18:02", "content": "

Archiver didn't quite work for me, for a reason, the following did:

\n

 

\n

 $ sudo port install p7zip
 $ 7za x Train.zip

", "rawMarkdown": "", "votes": null }, { "id": "29840", "postDate": "08/31/2013 11:36:03", "content": "

[quote=Rohit Sivaprasad;29815]

\n

head -n [number of lines] Train.csv > sample_train.csv

\n

[/quote]

\n

Had no idea you could do this, thanks!

", "rawMarkdown": "", "votes": null }, { "id": "29853", "postDate": "08/31/2013 15:31:44", "content": "

Unfortunately, traditional line based Unix tools for working with text files like head, tail, wc, etc., are not very useful for this data, because of new line characters embedded in records.

\n

So, one should use a library for parsing CSV which will take care of quoting, embedded new line characters, etc. I have attached a simple replacement of the Unix head command written in Python which will correctly copy n records from the competition's CSV files.

\n

$ python head.py Train.csv 2
"Id","Title","Body","Tags"
"1","How to check if an uploaded file is an image without mime type?","<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem is that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p><p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>
","php image-processing file-upload upload mime-types"
 

", "rawMarkdown": "", "votes": null }, { "id": "29857", "postDate": "08/31/2013 18:06:01", "content": "

I noticed that at the end of each record there is a \\r character, so what I did was just replace all the \\n with spaces and the replaced all the \\r with \\n. Let me know if there's something wrong with this approach.

\n

#!/bin/bash

\n

if [ -z "$1" ] ; then
echo "First replaces all the \\\\n with spaces then replaces all the \\\\r with \\\\n"
echo "usage: $0 input.csv output.csv"
exit 1;
fi

\n

tr '\\n' ' ' < "$1" | tr '\\r' '\\n' > "$2"

", "rawMarkdown": "", "votes": null }, { "id": "29865", "postDate": "08/31/2013 21:18:53", "content": "

Thanks Dmitrim, that's very useful!

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 29812, "author_name": "wcukierski", "author_url": "", "post_date": "08/30/2013 22:27:27", "content": "

The sample count should align with the Ids (starts at 1 in the train set and ends with 8047532 in the test set).

\n

This is a beast of a csv file, so we expect there to be a teething period and possibly some nasty edge cases.  Bear with us if the dataset has  bugs within.

\n

Mac users - you may have to use a program like Keka if the default unzipping  does not spit out a csv file for Train.zip.

", "votes": null, "replies": [] }, { "id": 29813, "author_name": "", "author_url": "", "post_date": "08/30/2013 22:42:53", "content": "

There's a few really nasty cases involving MATLAB data and code, it's mostly just nested \\n and " that throw things for a loop. Maybe URL encoding them, or using a uniquely guaranteed separator instead of "field","field"? I.e. remove all ~ from content, then use ~ as separators.

It might also be worthwhile to include a sample_train.csv and sample_test.csv, so participants can preview the data without potentially locking up slower (or virtual) machines.

", "votes": null, "replies": [] }, { "id": 29814, "author_name": "tafkas", "author_url": "", "post_date": "08/30/2013 22:44:44", "content": "

[quote=William Cukierski;29812]Mac users - you may have to use a program like Keka if the default unzipping  does not spit out a csv file for Train.zip.[/quote]

\n

Archiver.app seems to work. Produces:

\n

MD5 (Train.csv) = 880a8869d61bcce1f26072d9f4797cb9

\n

MD5 (Test.csv) = cac7cb73cf4061d8f74f9b5a094e66ab

", "votes": null, "replies": [] }, { "id": 29815, "author_name": "rohits", "author_url": "", "post_date": "08/30/2013 22:47:19", "content": "

Adrian:

\n

I'm sure you know this, but just for the record a simple one-liner like

\n

head -n [number of lines] Train.csv > sample_train.csv

\n

should do just that.

\n

 

", "votes": null, "replies": [] }, { "id": 29817, "author_name": "piotrkuchta", "author_url": "", "post_date": "08/30/2013 23:18:02", "content": "

Archiver didn't quite work for me, for a reason, the following did:

\n

 

\n

 $ sudo port install p7zip
 $ 7za x Train.zip

", "votes": null, "replies": [] }, { "id": 29840, "author_name": "", "author_url": "", "post_date": "08/31/2013 11:36:03", "content": "

[quote=Rohit Sivaprasad;29815]

\n

head -n [number of lines] Train.csv > sample_train.csv

\n

[/quote]

\n

Had no idea you could do this, thanks!

", "votes": null, "replies": [] }, { "id": 29853, "author_name": "piotrkuchta", "author_url": "", "post_date": "08/31/2013 15:31:44", "content": "

Unfortunately, traditional line based Unix tools for working with text files like head, tail, wc, etc., are not very useful for this data, because of new line characters embedded in records.

\n

So, one should use a library for parsing CSV which will take care of quoting, embedded new line characters, etc. I have attached a simple replacement of the Unix head command written in Python which will correctly copy n records from the competition's CSV files.

\n

$ python head.py Train.csv 2
"Id","Title","Body","Tags"
"1","How to check if an uploaded file is an image without mime type?","<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem is that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p><p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>
","php image-processing file-upload upload mime-types"
 

", "votes": null, "replies": [] }, { "id": 29857, "author_name": "dmitrim", "author_url": "", "post_date": "08/31/2013 18:06:01", "content": "

I noticed that at the end of each record there is a \\r character, so what I did was just replace all the \\n with spaces and the replaced all the \\r with \\n. Let me know if there's something wrong with this approach.

\n

#!/bin/bash

\n

if [ -z "$1" ] ; then
echo "First replaces all the \\\\n with spaces then replaces all the \\\\r with \\\\n"
echo "usage: $0 input.csv output.csv"
exit 1;
fi

\n

tr '\\n' ' ' < "$1" | tr '\\r' '\\n' > "$2"

", "votes": null, "replies": [] }, { "id": 29865, "author_name": "piotrkuchta", "author_url": "", "post_date": "08/31/2013 21:18:53", "content": "

Thanks Dmitrim, that's very useful!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "29809": "", "29812": "", "29813": "", "29814": "", "29815": "", "29817": "", "29840": "", "29853": "", "29857": "", "29865": "" }, "source": "meta" }