Is there any way to load the data faster ??
", "messages": [ { "id": "393793", "postDate": "09/25/2018 22:31:32", "content": "Is there any way to load the data faster ??
", "rawMarkdown": "Is there any way to load the data faster ??", "votes": null }, { "id": "393794", "postDate": "09/25/2018 22:36:30", "content": "Could use a little more information. Where are you trying to load it and what have you tried?
", "rawMarkdown": "Could use a little more information. Where are you trying to load it and what have you tried?", "votes": null }, { "id": "393807", "postDate": "09/25/2018 22:52:24", "content": "by loading I meant reading the dataFrame, because it take more than 4 minutes to read the training and test set
", "rawMarkdown": "by loading I meant reading the dataFrame, because it take more than 4 minutes to read the training and test set", "votes": null }, { "id": "393808", "postDate": "09/25/2018 23:01:00", "content": "I know at least in R it's about 10x faster to work with RDS files, so I usually read data in from CSVs once and save in that format. All subsequent loads will use the RDS file.
", "rawMarkdown": "I know at least in R it's about 10x faster to work with RDS files, so I usually read data in from CSVs once and save in that format. All subsequent loads will use the RDS file.", "votes": null }, { "id": "393831", "postDate": "09/26/2018 00:22:36", "content": "If you are working with R, try using fread() from data.table
", "rawMarkdown": "If you are working with R, try using fread() from data.table", "votes": null }, { "id": "393858", "postDate": "09/26/2018 01:29:25", "content": "BTW, if you're using python, it looks like you can read the CSV once and then store as a numpy object. Reading the numpy object from then on should be much faster.
", "rawMarkdown": "BTW, if you're using python, it looks like you can [read the CSV once and then store as a numpy object][1]. Reading the numpy object from then on should be much faster.\n\n\n [1]: https://towardsdatascience.com/why-you-should-start-using-npy-file-more-often-df2a13cc0161", "votes": null }, { "id": "393942", "postDate": "09/26/2018 05:19:18", "content": "You can drop useless columns and save the dataframe in memory with pickle extension. Check this kernel, it takes like 20s to load (with ssd). You can also use feather, which is even faster.
", "rawMarkdown": "You can drop useless columns and save the dataframe in memory with pickle extension. Check [this kernel][1], it takes like 20s to load (with ssd). You can also use feather, which is even faster.\n\n [1]: https://www.kaggle.com/jsaguiar/python-quick-start-clean-and-pickle-data", "votes": null }, { "id": "393969", "postDate": "09/26/2018 06:01:44", "content": "I always use pandas to_hdf / read_hdf, it is much faster than working with plain csv.
", "rawMarkdown": "I always use pandas to_hdf / read_hdf, it is much faster than working with plain csv.", "votes": null } ], "comments": [ { "id": 393794, "author_name": "nnnnick", "author_url": "", "post_date": "09/25/2018 22:36:30", "content": "Could use a little more information. Where are you trying to load it and what have you tried?
", "votes": null, "replies": [ { "id": 393807, "author_name": "djaballah", "author_url": "", "post_date": "09/25/2018 22:52:24", "content": "by loading I meant reading the dataFrame, because it take more than 4 minutes to read the training and test set
", "votes": null, "replies": [] }, { "id": 393808, "author_name": "nnnnick", "author_url": "", "post_date": "09/25/2018 23:01:00", "content": "I know at least in R it's about 10x faster to work with RDS files, so I usually read data in from CSVs once and save in that format. All subsequent loads will use the RDS file.
", "votes": null, "replies": [] }, { "id": 393831, "author_name": "captcalculator", "author_url": "", "post_date": "09/26/2018 00:22:36", "content": "If you are working with R, try using fread() from data.table
", "votes": null, "replies": [] }, { "id": 393858, "author_name": "nnnnick", "author_url": "", "post_date": "09/26/2018 01:29:25", "content": "BTW, if you're using python, it looks like you can read the CSV once and then store as a numpy object. Reading the numpy object from then on should be much faster.
", "votes": null, "replies": [] } ] }, { "id": 393942, "author_name": "jsaguiar", "author_url": "", "post_date": "09/26/2018 05:19:18", "content": "You can drop useless columns and save the dataframe in memory with pickle extension. Check this kernel, it takes like 20s to load (with ssd). You can also use feather, which is even faster.
", "votes": null, "replies": [] }, { "id": 393969, "author_name": "alexfir", "author_url": "", "post_date": "09/26/2018 06:01:44", "content": "I always use pandas to_hdf / read_hdf, it is much faster than working with plain csv.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "393793": "Is there any way to load the data faster ??", "393794": "Could use a little more information. Where are you trying to load it and what have you tried?", "393807": "by loading I meant reading the dataFrame, because it take more than 4 minutes to read the training and test set", "393808": "I know at least in R it's about 10x faster to work with RDS files, so I usually read data in from CSVs once and save in that format. All subsequent loads will use the RDS file.", "393831": "If you are working with R, try using fread() from data.table", "393858": "BTW, if you're using python, it looks like you can [read the CSV once and then store as a numpy object][1]. Reading the numpy object from then on should be much faster.\n\n\n [1]: https://towardsdatascience.com/why-you-should-start-using-npy-file-more-often-df2a13cc0161", "393942": "You can drop useless columns and save the dataframe in memory with pickle extension. Check [this kernel][1], it takes like 20s to load (with ssd). You can also use feather, which is even faster.\n\n [1]: https://www.kaggle.com/jsaguiar/python-quick-start-clean-and-pickle-data", "393969": "I always use pandas to_hdf / read_hdf, it is much faster than working with plain csv." }, "source": "meta" }