Does anyone have a method to download datasets from a remote instance that works for them?
\n\nI've been trying the methods posted on other Kaggle forums with no luck. I've resorted to rural internet service to sftp this dataset and it's uploading at ~50KB/s -- so maybe it'll be done in a week after restarts?
\n\nSurely other people have done this before for AWS instances. I've tried wget with a cookies file, Lynx, Lynx+wget and none of them work for me.
\n\nDoes Kaggle provide a direct means to download the files? Why don't they just set up security keys and be done with it? Where's the harm if my Kaggle profile held my public key?
", "messages": [ { "id": "137748", "postDate": "10/04/2016 16:21:20", "content": "Does anyone have a method to download datasets from a remote instance that works for them?
\n\nI've been trying the methods posted on other Kaggle forums with no luck. I've resorted to rural internet service to sftp this dataset and it's uploading at ~50KB/s -- so maybe it'll be done in a week after restarts?
\n\nSurely other people have done this before for AWS instances. I've tried wget with a cookies file, Lynx, Lynx+wget and none of them work for me.
\n\nDoes Kaggle provide a direct means to download the files? Why don't they just set up security keys and be done with it? Where's the harm if my Kaggle profile held my public key?
", "rawMarkdown": "Does anyone have a method to download datasets from a remote instance that works for them?\r\n\r\nI've been trying the methods posted on other Kaggle forums with no luck. I've resorted to rural internet service to sftp this dataset and it's uploading at ~50KB/s -- so maybe it'll be done in a week after restarts?\r\n\r\nSurely other people have done this before for AWS instances. I've tried wget with a cookies file, Lynx, Lynx+wget and none of them work for me.\r\n\r\nDoes Kaggle provide a direct means to download the files? Why don't they just set up security keys and be done with it? Where's the harm if my Kaggle profile held my public key?", "votes": null }, { "id": "137750", "postDate": "10/04/2016 16:51:14", "content": "There are a few methods. You can download and put the data to Dropbox then use wget to download the Dropbox shared URL. For example, given Dropbox URL
\n\nhttps://www.dropbox.com/s/7ob8wvcdd5k1bnf/helloworld.PNG?dl=0\nyou can run
\n\nwget https://www.dropbox.com/s/7ob8wvcdd5k1bnf/helloworld.PNG\nNote that ?dl=0 is removed.
Also you can convert csv data to binary data and put them into dropbox for faster upload/download.
", "rawMarkdown": "There are a few methods. You can download and put the data to Dropbox then use wget to download the Dropbox shared URL. For example, given Dropbox URL \r\n\r\nhttps://www.dropbox.com/s/7ob8wvcdd5k1bnf/helloworld.PNG?dl=0\r\nyou can run \r\n\r\n`wget https://www.dropbox.com/s/7ob8wvcdd5k1bnf/helloworld.PNG`\r\nNote that ?dl=0 is removed.\r\n\r\nAlso you can convert csv data to binary data and put them into dropbox for faster upload/download.", "votes": null }, { "id": "137751", "postDate": "10/04/2016 16:54:38", "content": "Check this thread.\nI've used the solution described in the first four posts in the past. It should work.
", "rawMarkdown": "Check [this][1] thread.\r\nI've used the solution described in the first four posts in the past. It should work.\r\n\r\n [1]: https://www.kaggle.com/c/belkin-energy-disaggregation-competition/forums/t/5118/downloading-data-via-wget/112602", "votes": null }, { "id": "137758", "postDate": "10/04/2016 21:00:49", "content": "Thanks @Banana Kitty. I have downloaded the Bosch dataset to to my computer, it's the upload rate that's my problem. So whether I upload to Dropbox or my remote instance will take the same amount of time. I would need a direct way of getting the datasets to Dropbox for that to work.
\n\n@Rohan, thanks for the link. That's one of them that I tried and I couldn't get any of the methods to work. On my browser I logged into Kaggle, started the download and accepted the terms. Then took the cookie info, uploaded it to my remote and tried the wget method. No luck...
", "rawMarkdown": "Thanks @Banana Kitty. I have downloaded the Bosch dataset to to my computer, it's the upload rate that's my problem. So whether I upload to Dropbox or my remote instance will take the same amount of time. I would need a direct way of getting the datasets to Dropbox for that to work.\r\n\r\n@Rohan, thanks for the link. That's one of them that I tried and I couldn't get any of the methods to work. On my browser I logged into Kaggle, started the download and accepted the terms. Then took the cookie info, uploaded it to my remote and tried the wget method. No luck...", "votes": null }, { "id": "137765", "postDate": "10/04/2016 22:06:06", "content": "This failed to work for me too, initially. I fixed it by changing %2F in the third column of the cookies file to /
\n\ne.g.
\n\n![https://www.dropbox.com/s/7ob8wvcdd5k1bnf/helloworld.PNG?dl=0](\"https://www.dropbox.com/s/7ob8wvcdd5k1bnf/helloworld.PNG?dl=0\"){kind=link}
.kaggle.com TRUE %2F FALSE 1467890123 ...\nto
\n\n.kaggle.com TRUE / FALSE 1467890123 ...\nThanks @James. That didn't turn out to be my problem.
\n\nActually I don't know what my problem was. I tried it again and it worked. Using the Chrome cookie.txt export extension I did these steps,
\n\nThe files will be in a sequence of folders starting at www.kaggle.com
", "rawMarkdown": "Thanks @James. That didn't turn out to be my problem.\r\n\r\nActually I don't know what my problem was. I tried it again and it worked. Using the Chrome *cookie.txt export* extension I did these steps,\r\n\r\n 1. on the download page, right click on one file to download and copy the link\r\n 2. open a new tab and paste the link into it \r\n 3. after the download starts, pause it from the lower download bar\r\n 4. click on the cookie.txt export extension icon on the upper right bar\r\n 5. copy the text from the pop up and follow the instructions\r\n 6. of course, make sure the cookies.txt file is in the current directory when you run wget\r\n 7. repeat for the other files, changing the file name\r\n\r\nThe files will be in a sequence of folders starting at www.kaggle.com", "votes": null }, { "id": "137778", "postDate": "10/05/2016 01:46:16", "content": "You can upload files using Cyberduck. http://cyberduck.en.softonic.com/mac
", "rawMarkdown": "You can upload files using Cyberduck. http://cyberduck.en.softonic.com/mac", "votes": null }, { "id": "137787", "postDate": "10/05/2016 03:51:50", "content": "Hi r3m,
\n\nDid you include https? Not sure if it will make any difference for you.
\n\nGood luck.
", "rawMarkdown": "Hi r3m,\r\n\r\nDid you include https? Not sure if it will make any difference for you.\r\n\r\nGood luck.", "votes": null }, { "id": "137793", "postDate": "10/05/2016 04:52:10", "content": "To download to an Amazon EC2 node, I just use links on the remote node (I use ubuntu-based AMIs). You don't have to supply a file with the cookies, you can just enter your login username and password from within links. (Indeed, I don't recall ever actually getting the cookie-file approach to work myself.)
It can be a bit confusing because the site is not exactly designed to be read in a text-only browser, so you may fumble a bit at first with the login and password fields. Also, you will probably want to get past the "accepting the competition rules" dialog on your local machine before you attempt the download through links. But beyond that it has always worked for me.
Hi @Banana Kitty. Yeah I did. Actually I got it working (see my previous post). Thanks!
", "rawMarkdown": "Hi @Banana Kitty. Yeah I did. Actually I got it working (see my previous post). Thanks!", "votes": null }, { "id": "137795", "postDate": "10/05/2016 04:58:45", "content": "@Tod. Yeah I couldn't get past the login with lynx. I tried a few times. Mostly my fault there since I don't know lynx well.
\n\nCookies/wget worked after I opened a download session on Chrome (using the direct link to the file) in my local computer, paused the download, and used wget on the remote. Maybe having the open session did something??
", "rawMarkdown": "Tod. Yeah I couldn't get past the login with lynx. I tried a few times. Mostly my fault there since I don't know lynx well. \r\n\r\nCookies/wget worked after I opened a download session on Chrome (using the direct link to the file) in my local computer, paused the download, and used wget on the remote. Maybe having the open session did something??", "votes": null }, { "id": "137796", "postDate": "10/05/2016 05:02:22", "content": "@Scott. I needed a solution running on my remote machine, so Cyberduck wouldn't help me there. But it does look interesting. Thanks!
", "rawMarkdown": "Scott. I needed a solution running on my remote machine, so Cyberduck wouldn't help me there. But it does look interesting. Thanks!", "votes": null }, { "id": "137799", "postDate": "10/05/2016 05:13:32", "content": "@r3m I mean literally links, which is a text-based browser distinct from lynx. I haven't attempted it with lynx, so it is possible that lynx is less agreeable for this task, I wouldn't know. Sounds like you got it sorted anyway...
Ohh... sorry @Tod, I just assumed that's the way you spelled it. Never heard of it before.
", "rawMarkdown": "Ohh... sorry @Tod, I just assumed that's the way you spelled it. Never heard of it before.", "votes": null }, { "id": "137836", "postDate": "10/05/2016 11:57:42", "content": "If you have the datasets on one machine, you could send them to a remote machine, check the scp or rsync command.
or you could use the remote machine to download the datasets directly. I recommend to use the unofficial kaggle command line tool. It has some problems at the moment but you are able to download the datasets.
", "rawMarkdown": "If you have the datasets on one machine, you could send them to a remote machine, check the `scp` or `rsync` command.\r\n\r\nor you could use the remote machine to download the datasets directly. I recommend to use the [unofficial kaggle command line tool][1]. It has some problems at the moment but you are able to download the datasets. \r\n\r\n [1]: https://github.com/floydwch/kaggle-cli", "votes": null }, { "id": "137838", "postDate": "10/05/2016 12:01:31", "content": "If you have the datasets on one machine, you could send them to a remote machine, check the scp or rsync command.
or you could use the remote machine to download the datasets directly. I recommend to use the unofficial kaggle command line tool. It has some problems at the moment but you are able to download the datasets.
", "rawMarkdown": "If you have the datasets on one machine, you could send them to a remote machine, check the `scp` or `rsync` command.\r\n\r\nor you could use the remote machine to download the datasets directly. I recommend to use the [unofficial kaggle command line tool][1]. It has some problems at the moment but you are able to download the datasets. \r\n\r\n [1]: https://github.com/floydwch/kaggle-cli", "votes": null }, { "id": "137839", "postDate": "10/05/2016 12:01:36", "content": "If you have the datasets on one machine, you could send them to a remote machine, check the scp or rsync command.
or you could use the remote machine to download the datasets directly. I recommend to use the unofficial kaggle command line tool. It has some problems at the moment but you are able to download the datasets.
", "rawMarkdown": "If you have the datasets on one machine, you could send them to a remote machine, check the `scp` or `rsync` command.\r\n\r\nor you could use the remote machine to download the datasets directly. I recommend to use the [unofficial kaggle command line tool][1]. It has some problems at the moment but you are able to download the datasets. \r\n\r\n [1]: https://github.com/floydwch/kaggle-cli", "votes": null }, { "id": "138049", "postDate": "10/06/2016 20:10:22", "content": "[quote=YiTang;137839]
\n\n...or you could use the remote machine to download the datasets directly. I recommend to use the unofficial kaggle command line tool. It has some problems at the moment but you are able to download the datasets.
\n\n[/quote]
\n\nThanks @YiTang. Kaggle-CLI may be exactly what I'm looking for.
", "rawMarkdown": "[quote=YiTang;137839]\r\n\r\n...or you could use the remote machine to download the datasets directly. I recommend to use the [unofficial kaggle command line tool][1]. It has some problems at the moment but you are able to download the datasets. \r\n\r\n [1]: https://github.com/floydwch/kaggle-cli\r\n\r\n[/quote]\r\n\r\nThanks @YiTang. Kaggle-CLI may be exactly what I'm looking for.", "votes": null }, { "id": "138832", "postDate": "10/11/2016 03:44:05", "content": "cookies.txt in your data folder.wget -x --load-cookies cookies.txt urlThere are a few methods. You can download and put the data to Dropbox then use wget to download the Dropbox shared URL. For example, given Dropbox URL
\n\nhttps://www.dropbox.com/s/7ob8wvcdd5k1bnf/helloworld.PNG?dl=0\nyou can run
\n\nwget https://www.dropbox.com/s/7ob8wvcdd5k1bnf/helloworld.PNG\nNote that ?dl=0 is removed.
Also you can convert csv data to binary data and put them into dropbox for faster upload/download.
", "votes": null, "replies": [] }, { "id": 137751, "author_name": "rohanrao", "author_url": "", "post_date": "10/04/2016 16:54:38", "content": "Check this thread.\nI've used the solution described in the first four posts in the past. It should work.
", "votes": null, "replies": [] }, { "id": 137758, "author_name": "rshpeley", "author_url": "", "post_date": "10/04/2016 21:00:49", "content": "Thanks @Banana Kitty. I have downloaded the Bosch dataset to to my computer, it's the upload rate that's my problem. So whether I upload to Dropbox or my remote instance will take the same amount of time. I would need a direct way of getting the datasets to Dropbox for that to work.
\n\n@Rohan, thanks for the link. That's one of them that I tried and I couldn't get any of the methods to work. On my browser I logged into Kaggle, started the download and accepted the terms. Then took the cookie info, uploaded it to my remote and tried the wget method. No luck...
", "votes": null, "replies": [] }, { "id": 137765, "author_name": "jtrotman", "author_url": "", "post_date": "10/04/2016 22:06:06", "content": "This failed to work for me too, initially. I fixed it by changing %2F in the third column of the cookies file to /
\n\ne.g.
\n\n.kaggle.com TRUE %2F FALSE 1467890123 ...\nto
\n\n.kaggle.com TRUE / FALSE 1467890123 ...\nThanks @James. That didn't turn out to be my problem.
\n\nActually I don't know what my problem was. I tried it again and it worked. Using the Chrome cookie.txt export extension I did these steps,
\n\nThe files will be in a sequence of folders starting at www.kaggle.com
", "votes": null, "replies": [] }, { "id": 137778, "author_name": "liouscott", "author_url": "", "post_date": "10/05/2016 01:46:16", "content": "You can upload files using Cyberduck. http://cyberduck.en.softonic.com/mac
", "votes": null, "replies": [] }, { "id": 137787, "author_name": "", "author_url": "", "post_date": "10/05/2016 03:51:50", "content": "Hi r3m,
\n\nDid you include https? Not sure if it will make any difference for you.
\n\nGood luck.
", "votes": null, "replies": [] }, { "id": 137793, "author_name": "tmwright", "author_url": "", "post_date": "10/05/2016 04:52:10", "content": "To download to an Amazon EC2 node, I just use links on the remote node (I use ubuntu-based AMIs). You don't have to supply a file with the cookies, you can just enter your login username and password from within links. (Indeed, I don't recall ever actually getting the cookie-file approach to work myself.)
It can be a bit confusing because the site is not exactly designed to be read in a text-only browser, so you may fumble a bit at first with the login and password fields. Also, you will probably want to get past the "accepting the competition rules" dialog on your local machine before you attempt the download through links. But beyond that it has always worked for me.
Hi @Banana Kitty. Yeah I did. Actually I got it working (see my previous post). Thanks!
", "votes": null, "replies": [] }, { "id": 137795, "author_name": "rshpeley", "author_url": "", "post_date": "10/05/2016 04:58:45", "content": "@Tod. Yeah I couldn't get past the login with lynx. I tried a few times. Mostly my fault there since I don't know lynx well.
\n\nCookies/wget worked after I opened a download session on Chrome (using the direct link to the file) in my local computer, paused the download, and used wget on the remote. Maybe having the open session did something??
", "votes": null, "replies": [] }, { "id": 137796, "author_name": "rshpeley", "author_url": "", "post_date": "10/05/2016 05:02:22", "content": "@Scott. I needed a solution running on my remote machine, so Cyberduck wouldn't help me there. But it does look interesting. Thanks!
", "votes": null, "replies": [] }, { "id": 137799, "author_name": "tmwright", "author_url": "", "post_date": "10/05/2016 05:13:32", "content": "@r3m I mean literally links, which is a text-based browser distinct from lynx. I haven't attempted it with lynx, so it is possible that lynx is less agreeable for this task, I wouldn't know. Sounds like you got it sorted anyway...
Ohh... sorry @Tod, I just assumed that's the way you spelled it. Never heard of it before.
", "votes": null, "replies": [] }, { "id": 137836, "author_name": "yimacs", "author_url": "", "post_date": "10/05/2016 11:57:42", "content": "If you have the datasets on one machine, you could send them to a remote machine, check the scp or rsync command.
or you could use the remote machine to download the datasets directly. I recommend to use the unofficial kaggle command line tool. It has some problems at the moment but you are able to download the datasets.
", "votes": null, "replies": [] }, { "id": 137838, "author_name": "yimacs", "author_url": "", "post_date": "10/05/2016 12:01:31", "content": "If you have the datasets on one machine, you could send them to a remote machine, check the scp or rsync command.
or you could use the remote machine to download the datasets directly. I recommend to use the unofficial kaggle command line tool. It has some problems at the moment but you are able to download the datasets.
", "votes": null, "replies": [] }, { "id": 137839, "author_name": "yimacs", "author_url": "", "post_date": "10/05/2016 12:01:36", "content": "If you have the datasets on one machine, you could send them to a remote machine, check the scp or rsync command.
or you could use the remote machine to download the datasets directly. I recommend to use the unofficial kaggle command line tool. It has some problems at the moment but you are able to download the datasets.
", "votes": null, "replies": [] }, { "id": 138049, "author_name": "rshpeley", "author_url": "", "post_date": "10/06/2016 20:10:22", "content": "[quote=YiTang;137839]
\n\n...or you could use the remote machine to download the datasets directly. I recommend to use the unofficial kaggle command line tool. It has some problems at the moment but you are able to download the datasets.
\n\n[/quote]
\n\nThanks @YiTang. Kaggle-CLI may be exactly what I'm looking for.
", "votes": null, "replies": [] }, { "id": 138832, "author_name": "th3o6a1d", "author_url": "", "post_date": "10/11/2016 03:44:05", "content": "cookies.txt in your data folder.wget -x --load-cookies cookies.txt url