wc -l avito_trian.tsv indicates that there are ~4m training points. However, my code which is based on the provided python code seems fail to read training data after 1562936th row. Here it is:
\n with open(tsv_file) as tsvReader:
itemReader = DictReader(tsvReader, delimiter='\\t', quotechar = '"')
for i, item in enumerate(itemReader):
item_dict = {featureName:featureValue.decode('utf-8') \\
for featureName,featureValue in item.iteritems() \\
if featureValue is not None}
In specific, for the 1562936th row, some keys, e.g., is_blocked and prices is even missing in item_dict. Does anyone know what goes wrong?
", "messages": [ { "id": "51635", "postDate": "08/05/2014 15:26:48", "content": "wc -l avito_trian.tsv indicates that there are ~4m training points. However, my code which is based on the provided python code seems fail to read training data after 1562936th row. Here it is:
\n with open(tsv_file) as tsvReader:
itemReader = DictReader(tsvReader, delimiter='\\t', quotechar = '"')
for i, item in enumerate(itemReader):
item_dict = {featureName:featureValue.decode('utf-8') \\
for featureName,featureValue in item.iteritems() \\
if featureValue is not None}
In specific, for the 1562936th row, some keys, e.g., is_blocked and prices is even missing in item_dict. Does anyone know what goes wrong?
", "rawMarkdown": "", "votes": null }, { "id": "51636", "postDate": "08/05/2014 15:27:58", "content": "bummer :P
", "rawMarkdown": "", "votes": null }, { "id": "51637", "postDate": "08/05/2014 15:34:08", "content": "Still, I was happy to find out that my model has actually been trained on ~40% data, which makes me think there might be some room for improvement...
", "rawMarkdown": "", "votes": null }, { "id": "51638", "postDate": "08/05/2014 15:35:18", "content": "well, it works for me..
", "rawMarkdown": "", "votes": null }, { "id": "51650", "postDate": "08/05/2014 18:55:04", "content": "Beware that "wc -l" gives a line count. .csv or .tsv files could have samples with text that span multiple lines:
\n"0","15","description \\n\\n buy this!"
", "rawMarkdown": "", "votes": null }, { "id": "51653", "postDate": "08/05/2014 23:16:24", "content": "[quote=Triskelion;51650]
\nBeware that "wc -l" gives a line count. .csv or .tsv files could have samples with text that span multiple lines:
\n"0","15","description \\n\\n buy this!"
\n[/quote]
\nBut it gives the right number of the testing set, i.e., 1351243. PS. what's the number of training data on you side?
", "rawMarkdown": "", "votes": null }, { "id": "51662", "postDate": "08/06/2014 10:23:30", "content": "1.562.936 samples in train.
\n1.562.937 lines in train.
", "rawMarkdown": "", "votes": null }, { "id": "51663", "postDate": "08/06/2014 10:36:51", "content": "omg! you have two train sets!
", "rawMarkdown": "", "votes": null }, { "id": "51664", "postDate": "08/06/2014 10:37:34", "content": "[quote=Triskelion;51662]
\n1.562.936 samples in train.
\n1.562.937 lines in train.
\n[/quote]
\nLOL... I am now confused why Zygmunt said there are 4m points here: http://www.kaggle.com/c/avito-prohibited-content/forums/t/9667/millions-of-training-examples/50147#post50147
", "rawMarkdown": "", "votes": null }, { "id": "51669", "postDate": "08/06/2014 12:27:41", "content": "I have 3995803 samples in train and 1351242 in test. Data loaded just by pandas.read_csv("./avito_train.tsv", sep="\\t", encoding='utf8') without any problem.
\nRaw data has:
\n# cat avito_train.tsv | wc -l
3995804
# cat avito_test.tsv | wc -l
1351243
By the way,
\n# cat test
"0","15","description \\n\\n buy this!"
# cat test | wc -l
1
*upd: sorry for duplicating =)
", "rawMarkdown": "", "votes": null }, { "id": "51671", "postDate": "08/06/2014 12:38:39", "content": "I think I made a mistake similar to yr. Just looking at the file sizes of the train and test set, the train set should be more like 4 million samples.
\nPerhaps there is some bug or incorrect usage of csv.DictReader. Or Pandas has a better parser for edge cases. Anyway it is strange.
\npd.read_csv gives me 3.995.803 records too.
\nThanks Mikhail and yr, I now have more data to train on, will post here if it makes any difference.
", "rawMarkdown": "", "votes": null }, { "id": "51672", "postDate": "08/06/2014 12:41:05", "content": "[quote=Mikhail Trofimov;51670]
\nBy the way,
\n# cat test
"0","15","description \\n\\n buy this!"
# cat test | wc -l
1
*upd: sorry for duplicating =)
\n[/quote]
\nIt is not happening here, but beware that "wc -l" is not always reliable to count samples in .csv files. For example the StackOverflow dataset had multi-line text columns:
\n"0","15","description
\nbuy this!"
", "rawMarkdown": "", "votes": null }, { "id": "51673", "postDate": "08/06/2014 13:07:19", "content": "Thanks all you guys! I will try pandas then. Hopefully it will make a difference with more data :-)
", "rawMarkdown": "", "votes": null }, { "id": "51674", "postDate": "08/06/2014 13:34:11", "content": "yr, can you run the following:
\nprint len(open("avito_train.tsv").readlines())
and report back the number and your OS and Python version?
\nWindows 7, Python 2.7: 1.562.938
\nMac OSX 10, Python 2.7: 3.995.804
\nIs this an encoding issue which Pandas auto-solves? Never heard about encoding issues for line counts in this way.
", "rawMarkdown": "", "votes": null }, { "id": "51675", "postDate": "08/06/2014 13:59:27", "content": "Here it is:
\nWindows 7 64bit, Python 2.7.6 (Anaconda 1.9.0 (64-bit)): 1,562,938
\nBut I am able to reproduce the benchmark. It might also be the reason while others can not: http://www.kaggle.com/c/avito-prohibited-content/forums/t/9722/sample-code-different-from-benchmark
", "rawMarkdown": "", "votes": null }, { "id": "51708", "postDate": "08/07/2014 11:49:57", "content": "@Triskelion,
\nTry this:
\nlen((open(train_file_csv,"rb").readlines()))
It now gives 3,995,804.
\nI learned it from this post: http://stackoverflow.com/questions/1170214/pythons-csv-writer-produces-wrong-line-terminator
", "rawMarkdown": "", "votes": null }, { "id": "51733", "postDate": "08/08/2014 02:18:34", "content": "Seems more data does help: 0.97691 -> 0.98000 :-)
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 51636, "author_name": "abhishek", "author_url": "", "post_date": "08/05/2014 15:27:58", "content": "bummer :P
", "votes": null, "replies": [] }, { "id": 51637, "author_name": "chenglongchen", "author_url": "", "post_date": "08/05/2014 15:34:08", "content": "Still, I was happy to find out that my model has actually been trained on ~40% data, which makes me think there might be some room for improvement...
", "votes": null, "replies": [] }, { "id": 51638, "author_name": "abhishek", "author_url": "", "post_date": "08/05/2014 15:35:18", "content": "well, it works for me..
", "votes": null, "replies": [] }, { "id": 51650, "author_name": "triskelion", "author_url": "", "post_date": "08/05/2014 18:55:04", "content": "Beware that "wc -l" gives a line count. .csv or .tsv files could have samples with text that span multiple lines:
\n"0","15","description \\n\\n buy this!"
", "votes": null, "replies": [] }, { "id": 51653, "author_name": "chenglongchen", "author_url": "", "post_date": "08/05/2014 23:16:24", "content": "[quote=Triskelion;51650]
\nBeware that "wc -l" gives a line count. .csv or .tsv files could have samples with text that span multiple lines:
\n"0","15","description \\n\\n buy this!"
\n[/quote]
\nBut it gives the right number of the testing set, i.e., 1351243. PS. what's the number of training data on you side?
", "votes": null, "replies": [] }, { "id": 51662, "author_name": "triskelion", "author_url": "", "post_date": "08/06/2014 10:23:30", "content": "1.562.936 samples in train.
\n1.562.937 lines in train.
", "votes": null, "replies": [] }, { "id": 51663, "author_name": "abhishek", "author_url": "", "post_date": "08/06/2014 10:36:51", "content": "omg! you have two train sets!
", "votes": null, "replies": [] }, { "id": 51664, "author_name": "chenglongchen", "author_url": "", "post_date": "08/06/2014 10:37:34", "content": "[quote=Triskelion;51662]
\n1.562.936 samples in train.
\n1.562.937 lines in train.
\n[/quote]
\nLOL... I am now confused why Zygmunt said there are 4m points here: http://www.kaggle.com/c/avito-prohibited-content/forums/t/9667/millions-of-training-examples/50147#post50147
", "votes": null, "replies": [] }, { "id": 51669, "author_name": "mikhailtrofimov", "author_url": "", "post_date": "08/06/2014 12:27:41", "content": "I have 3995803 samples in train and 1351242 in test. Data loaded just by pandas.read_csv("./avito_train.tsv", sep="\\t", encoding='utf8') without any problem.
\nRaw data has:
\n# cat avito_train.tsv | wc -l
3995804
# cat avito_test.tsv | wc -l
1351243
By the way,
\n# cat test
"0","15","description \\n\\n buy this!"
# cat test | wc -l
1
*upd: sorry for duplicating =)
", "votes": null, "replies": [] }, { "id": 51671, "author_name": "triskelion", "author_url": "", "post_date": "08/06/2014 12:38:39", "content": "I think I made a mistake similar to yr. Just looking at the file sizes of the train and test set, the train set should be more like 4 million samples.
\nPerhaps there is some bug or incorrect usage of csv.DictReader. Or Pandas has a better parser for edge cases. Anyway it is strange.
\npd.read_csv gives me 3.995.803 records too.
\nThanks Mikhail and yr, I now have more data to train on, will post here if it makes any difference.
", "votes": null, "replies": [] }, { "id": 51672, "author_name": "triskelion", "author_url": "", "post_date": "08/06/2014 12:41:05", "content": "[quote=Mikhail Trofimov;51670]
\nBy the way,
\n# cat test
"0","15","description \\n\\n buy this!"
# cat test | wc -l
1
*upd: sorry for duplicating =)
\n[/quote]
\nIt is not happening here, but beware that "wc -l" is not always reliable to count samples in .csv files. For example the StackOverflow dataset had multi-line text columns:
\n"0","15","description
\nbuy this!"
", "votes": null, "replies": [] }, { "id": 51673, "author_name": "chenglongchen", "author_url": "", "post_date": "08/06/2014 13:07:19", "content": "Thanks all you guys! I will try pandas then. Hopefully it will make a difference with more data :-)
", "votes": null, "replies": [] }, { "id": 51674, "author_name": "triskelion", "author_url": "", "post_date": "08/06/2014 13:34:11", "content": "yr, can you run the following:
\nprint len(open("avito_train.tsv").readlines())
and report back the number and your OS and Python version?
\nWindows 7, Python 2.7: 1.562.938
\nMac OSX 10, Python 2.7: 3.995.804
\nIs this an encoding issue which Pandas auto-solves? Never heard about encoding issues for line counts in this way.
", "votes": null, "replies": [] }, { "id": 51675, "author_name": "chenglongchen", "author_url": "", "post_date": "08/06/2014 13:59:27", "content": "Here it is:
\nWindows 7 64bit, Python 2.7.6 (Anaconda 1.9.0 (64-bit)): 1,562,938
\nBut I am able to reproduce the benchmark. It might also be the reason while others can not: http://www.kaggle.com/c/avito-prohibited-content/forums/t/9722/sample-code-different-from-benchmark
", "votes": null, "replies": [] }, { "id": 51708, "author_name": "chenglongchen", "author_url": "", "post_date": "08/07/2014 11:49:57", "content": "@Triskelion,
\nTry this:
\nlen((open(train_file_csv,"rb").readlines()))
It now gives 3,995,804.
\nI learned it from this post: http://stackoverflow.com/questions/1170214/pythons-csv-writer-produces-wrong-line-terminator
", "votes": null, "replies": [] }, { "id": 51733, "author_name": "chenglongchen", "author_url": "", "post_date": "08/08/2014 02:18:34", "content": "Seems more data does help: 0.97691 -> 0.98000 :-)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "51635": "", "51636": "", "51637": "", "51638": "", "51650": "", "51653": "", "51662": "", "51663": "", "51664": "", "51669": "", "51670": "", "51671": "", "51672": "", "51673": "", "51674": "", "51675": "", "51708": "", "51733": "" }, "source": "meta" }