{ "id": 11018, "title": "Shuffling lines of big data files", "url": "/competitions/avazu-ctr-prediction/discussion/11018", "author_name": "", "post_date": "2014-11-22T16:26:20.563Z", "votes": 12, "comment_count": 13, "views": 5770, "content": "

Hi,

\n

To anyone who's interested in shuffled train data I would like to impudently promote my own opensource tool which is available here under LGPL.

\n

It's written in C++ with Qt framework and specially made for files bigger than available RAM size. More technical details could be found in readme file and this discussion.

\n

Binaries are available for Linux (.deb, x64) and Win x32/x64.

\n

I have tried it on avazu's train file preliminary converted to VW format so it become 15.5 Gb (i used long prefixes for categorical values). It was shuffled for ~ 1h 40 min with 1Gb buffer (up to 2Gb total RAM used). And original ~6Gb train csv was shuffled in less than 20 min with following command:

\n

[code]$ shuf-t -t 1 train -o train_shuffled
random seed: 1416668999
searching line offsets: ...........
offsets found: 40428967
2 min 2.71 sec
shuffling line offsets: ..........
2.89 sec
writing lines to output: ..........
15 min 6.34 sec
[/code]

\n

The command line format is taken from GNU Shuf util.

\n

It's still a beta so feedback is highly appreciated.

", "messages": [ { "id": "58689", "postDate": "11/22/2014 16:26:20", "content": "

Hi,

\n

To anyone who's interested in shuffled train data I would like to impudently promote my own opensource tool which is available here under LGPL.

\n

It's written in C++ with Qt framework and specially made for files bigger than available RAM size. More technical details could be found in readme file and this discussion.

\n

Binaries are available for Linux (.deb, x64) and Win x32/x64.

\n

I have tried it on avazu's train file preliminary converted to VW format so it become 15.5 Gb (i used long prefixes for categorical values). It was shuffled for ~ 1h 40 min with 1Gb buffer (up to 2Gb total RAM used). And original ~6Gb train csv was shuffled in less than 20 min with following command:

\n

[code]$ shuf-t -t 1 train -o train_shuffled
random seed: 1416668999
searching line offsets: ...........
offsets found: 40428967
2 min 2.71 sec
shuffling line offsets: ..........
2.89 sec
writing lines to output: ..........
15 min 6.34 sec
[/code]

\n

The command line format is taken from GNU Shuf util.

\n

It's still a beta so feedback is highly appreciated.

", "rawMarkdown": "", "votes": null }, { "id": "58699", "postDate": "11/22/2014 20:10:42", "content": "

Thanks a lot! Great tool.

\n

Shuffled in just over 4 min :D

", "rawMarkdown": "", "votes": null }, { "id": "58800", "postDate": "11/24/2014 18:10:47", "content": "

When using exe file for win64， where can I enter the command line (including input file for shuffling)？

\n

thanks a lot!

", "rawMarkdown": "", "votes": null }, { "id": "58810", "postDate": "11/24/2014 19:56:04", "content": "

[quote=tung chne;58800]

\n

When using exe file for win64， where can I enter the command line (including input file for shuffling)？

\n

thanks a lot!

\n

[/quote]

\n

Well, it's a console tool without user interface. If you just launch it then it will expect that you type in opened console all your data and press ctrl+D. Shuffled lines will be outputted on screen. You can't specify more options as you already launched the app. To launch it with parameters you shall use windows console and if you need more details then this videolesson seems fine.

", "rawMarkdown": "", "votes": null }, { "id": "58950", "postDate": "11/25/2014 20:15:01", "content": "

Quite curious for the shuffling idea working or not. Tried reversing the order but no luck. Probably time sequence matters. Any idea?

", "rawMarkdown": "", "votes": null }, { "id": "58952", "postDate": "11/25/2014 21:38:50", "content": "

[quote=simeng;58950]

\n

Quite curious for the shuffling idea working or not. Tried reversing the order but no luck. Probably time sequence matters. Any idea?

\n

[/quote]

\n

I don't know if order matters for final model training, but in my opinion defining hyper-parameters and playing with features (making ngrams etc) is better while using shuffled data. 

\n

For example, VW implements online learning algorithms and outputs average train / holdout losses every N learned examples, so you may realize that your last model tweak was a bad idea much earlier than its training ends just by comparing VW output with previous result. And this is easier to do if sequence of average loss values fits in some trend and has small variance. From this perspective shuffled data behave better. If I find good enough model with shuffled data, of course, I'll give it a try with original dataset too to choose better one.

", "rawMarkdown": "", "votes": null }, { "id": "59042", "postDate": "11/26/2014 20:11:03", "content": "

What kind of expected improvement on LB compare shuffle vs non-shuffle? 

", "rawMarkdown": "", "votes": null }, { "id": "59074", "postDate": "11/27/2014 08:27:20", "content": "

[quote=superfan123;59042]

\n

What kind of expected improvement on LB compare shuffle vs non-shuffle? 

\n

[/quote]

\n\n

Ok, I've tried basic VW. Shuffled data was used to find its hyperparameters. Then I used shuffled and non-shuffled data to train 2 models. The model trained on non-shufled data gave -0.002 on public LB vs model trained on random data.

", "rawMarkdown": "", "votes": null }, { "id": "59088", "postDate": "11/27/2014 11:04:08", "content": "

does it have any option to keep header intact?

\n

EDIT: found it 

", "rawMarkdown": "", "votes": null }, { "id": "59089", "postDate": "11/27/2014 11:51:17", "content": "

shuffling did not give me a better score...

", "rawMarkdown": "", "votes": null }, { "id": "59136", "postDate": "11/28/2014 04:39:57", "content": "

Think the vendor will also not favour strategies with shuffling as data will stream in according to time.  One of the advantage of Online algo is in adapting to the time nature and user shifts in preference with the correct features.  Guess more strategic hunting needed....  

", "rawMarkdown": "", "votes": null }, { "id": "62557", "postDate": "01/10/2015 12:42:02", "content": "

I attach here a script in pure python to shuffle big files. The algorithm creates temporary files to do the sorting, so depending on the number of lines you want to shuffle, you may need to increase your limit on open file descriptors. I used 15000 in my computer to shuffle the 40M lines of train.

\n

Usage:

\n

pypy/python shuffle_ooc.py file.txt [N] > file_shuffled.txt

\n

where N is optional and it´s the number of lines in the file, otherwise the scripts counts them

\n

Hope it helps!

", "rawMarkdown": "", "votes": null }, { "id": "356587", "postDate": "07/13/2018 23:03:30", "content": "

You could also use the shuf command in a bash shell, which is very quick.

\n\n

Strip the header line from train.csv: tail -40428967 train.csv > train.csv

\n\n

Then shuffle in place:

\n\n

shuf -o train.csv < train.csv

", "rawMarkdown": "You could also use the shuf command in a bash shell, which is very quick.\n\nStrip the header line from train.csv: tail -40428967 train.csv > train.csv\n\nThen shuffle in place:\n\nshuf -o train.csv < train.csv", "votes": null }, { "id": "928896", "postDate": "07/14/2020 10:05:29", "content": "

I downloaded the shuffle_ooc.py file. from Guillem Palou.\nbut I don't know how to run this script on my dataset.... how to introduce the file path and the file itself.\nany one can help me.

", "rawMarkdown": "I downloaded the shuffle_ooc.py file. from Guillem Palou.\nbut I don't know how to run this script on my dataset.... how to introduce the file path and the file itself.\nany one can help me.", "votes": null } ], "comments": [ { "id": 58699, "author_name": "lukemetz", "author_url": "", "post_date": "11/22/2014 20:10:42", "content": "

Thanks a lot! Great tool.

\n

Shuffled in just over 4 min :D

", "votes": null, "replies": [] }, { "id": 58800, "author_name": "tungchne", "author_url": "", "post_date": "11/24/2014 18:10:47", "content": "

When using exe file for win64， where can I enter the command line (including input file for shuffling)？

\n

thanks a lot!

", "votes": null, "replies": [] }, { "id": 58810, "author_name": "trufanov", "author_url": "", "post_date": "11/24/2014 19:56:04", "content": "

[quote=tung chne;58800]

\n

When using exe file for win64， where can I enter the command line (including input file for shuffling)？

\n

thanks a lot!

\n

[/quote]

\n

Well, it's a console tool without user interface. If you just launch it then it will expect that you type in opened console all your data and press ctrl+D. Shuffled lines will be outputted on screen. You can't specify more options as you already launched the app. To launch it with parameters you shall use windows console and if you need more details then this videolesson seems fine.

", "votes": null, "replies": [] }, { "id": 58950, "author_name": "simengy82", "author_url": "", "post_date": "11/25/2014 20:15:01", "content": "

Quite curious for the shuffling idea working or not. Tried reversing the order but no luck. Probably time sequence matters. Any idea?

", "votes": null, "replies": [] }, { "id": 58952, "author_name": "trufanov", "author_url": "", "post_date": "11/25/2014 21:38:50", "content": "

[quote=simeng;58950]

\n

Quite curious for the shuffling idea working or not. Tried reversing the order but no luck. Probably time sequence matters. Any idea?

\n

[/quote]

\n

I don't know if order matters for final model training, but in my opinion defining hyper-parameters and playing with features (making ngrams etc) is better while using shuffled data. 

\n

For example, VW implements online learning algorithms and outputs average train / holdout losses every N learned examples, so you may realize that your last model tweak was a bad idea much earlier than its training ends just by comparing VW output with previous result. And this is easier to do if sequence of average loss values fits in some trend and has small variance. From this perspective shuffled data behave better. If I find good enough model with shuffled data, of course, I'll give it a try with original dataset too to choose better one.

", "votes": null, "replies": [] }, { "id": 59042, "author_name": "superfan123", "author_url": "", "post_date": "11/26/2014 20:11:03", "content": "

What kind of expected improvement on LB compare shuffle vs non-shuffle? 

", "votes": null, "replies": [] }, { "id": 59074, "author_name": "trufanov", "author_url": "", "post_date": "11/27/2014 08:27:20", "content": "

[quote=superfan123;59042]

\n

What kind of expected improvement on LB compare shuffle vs non-shuffle? 

\n

[/quote]

\n\n

Ok, I've tried basic VW. Shuffled data was used to find its hyperparameters. Then I used shuffled and non-shuffled data to train 2 models. The model trained on non-shufled data gave -0.002 on public LB vs model trained on random data.

", "votes": null, "replies": [] }, { "id": 59088, "author_name": "abhishek", "author_url": "", "post_date": "11/27/2014 11:04:08", "content": "

does it have any option to keep header intact?

\n

EDIT: found it 

", "votes": null, "replies": [] }, { "id": 59089, "author_name": "abhishek", "author_url": "", "post_date": "11/27/2014 11:51:17", "content": "

shuffling did not give me a better score...

", "votes": null, "replies": [] }, { "id": 59136, "author_name": "drpatrickchan", "author_url": "", "post_date": "11/28/2014 04:39:57", "content": "

Think the vendor will also not favour strategies with shuffling as data will stream in according to time.  One of the advantage of Online algo is in adapting to the time nature and user shifts in preference with the correct features.  Guess more strategic hunting needed....  

", "votes": null, "replies": [] }, { "id": 62557, "author_name": "guillempalou", "author_url": "", "post_date": "01/10/2015 12:42:02", "content": "

I attach here a script in pure python to shuffle big files. The algorithm creates temporary files to do the sorting, so depending on the number of lines you want to shuffle, you may need to increase your limit on open file descriptors. I used 15000 in my computer to shuffle the 40M lines of train.

\n

Usage:

\n

pypy/python shuffle_ooc.py file.txt [N] > file_shuffled.txt

\n

where N is optional and it´s the number of lines in the file, otherwise the scripts counts them

\n

Hope it helps!

", "votes": null, "replies": [] }, { "id": 356587, "author_name": "jcatanza", "author_url": "", "post_date": "07/13/2018 23:03:30", "content": "

You could also use the shuf command in a bash shell, which is very quick.

\n\n

Strip the header line from train.csv: tail -40428967 train.csv > train.csv

\n\n

Then shuffle in place:

\n\n

shuf -o train.csv < train.csv

", "votes": null, "replies": [] }, { "id": 928896, "author_name": "samirafotuhi", "author_url": "", "post_date": "07/14/2020 10:05:29", "content": "

I downloaded the shuffle_ooc.py file. from Guillem Palou.\nbut I don't know how to run this script on my dataset.... how to introduce the file path and the file itself.\nany one can help me.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "58689": "", "58699": "", "58800": "", "58810": "", "58950": "", "58952": "", "59042": "", "59074": "", "59088": "", "59089": "", "59136": "", "62557": "", "356587": "You could also use the shuf command in a bash shell, which is very quick.\n\nStrip the header line from train.csv: tail -40428967 train.csv > train.csv\n\nThen shuffle in place:\n\nshuf -o train.csv < train.csv", "928896": "I downloaded the shuffle_ooc.py file. from Guillem Palou.\nbut I don't know how to run this script on my dataset.... how to introduce the file path and the file itself.\nany one can help me." }, "source": "meta" }