Is it time to ditch csv files? Dealing with some large files lately got me looking around – and Python/pandas has “to_pickle” and “read_pickle” functions that will work with your favorite file compression scheme. Just choose “.zip” / “.bz2” / “.gzip”, etc for a file extension and the methods will infer the compression type and act accordingly. Seems there is little reason for “.csv” files except for the submission file Kaggle needs. These methods appear to run fast enough to be worth the saved disc space.
\n\ntrain.to_pickle(os.path.join(FEATURES_DIR, 'myfile.zip'))
\n\ntrain = pd.read_pickle(os.path.join(FEATURES_DIR, 'myfile.zip'))
", "messages": [ { "id": "559200", "postDate": "06/23/2019 16:31:04", "content": "Is it time to ditch csv files? Dealing with some large files lately got me looking around – and Python/pandas has “to_pickle” and “read_pickle” functions that will work with your favorite file compression scheme. Just choose “.zip” / “.bz2” / “.gzip”, etc for a file extension and the methods will infer the compression type and act accordingly. Seems there is little reason for “.csv” files except for the submission file Kaggle needs. These methods appear to run fast enough to be worth the saved disc space.
\n\ntrain.to_pickle(os.path.join(FEATURES_DIR, 'myfile.zip'))
\n\ntrain = pd.read_pickle(os.path.join(FEATURES_DIR, 'myfile.zip'))
", "rawMarkdown": "Is it time to ditch csv files? Dealing with some large files lately got me looking around – and Python/pandas has “to\\_pickle” and “read\\_pickle” functions that will work with your favorite file compression scheme. Just choose “.zip” / “.bz2” / “.gzip”, etc for a file extension and the methods will infer the compression type and act accordingly. Seems there is little reason for “.csv” files except for the submission file Kaggle needs. These methods appear to run fast enough to be worth the saved disc space. \n\ntrain.to\\_pickle(os.path.join(FEATURES_DIR, 'myfile.zip')) \n\ntrain = pd.read\\_pickle(os.path.join(FEATURES_DIR, 'myfile.zip'))", "votes": null }, { "id": "559206", "postDate": "06/23/2019 16:36:04", "content": "Never tried this (will this improve our visualisation speed) ? And thanks for sharing I'll try this method 🙌
", "rawMarkdown": "Never tried this (will this improve our visualisation speed) ? And thanks for sharing I'll try this method 🙌", "votes": null }, { "id": "559248", "postDate": "06/23/2019 18:21:22", "content": "I actually prefer HDF5 which can be compressed and retains all the data types\nmagic_feature_df.to_hdf(\"magic_features.h5\", key=\"magic\", complevel=9)
Also for submission files, you can create a zip and upload that directly to Kaggle\nsub.to_csv('submission.zip', index=False, compression='zip')
Any zipped CSV can also be read in the same way without any additional arguments:\ndf = pd.read_csv('other_magic_features.zip')
I also use HDF5 format for this competition, it is fast and takes less disk space.
", "rawMarkdown": "I also use HDF5 format for this competition, it is fast and takes less disk space.", "votes": null }, { "id": "559251", "postDate": "06/23/2019 18:31:39", "content": "my previous teammate has hooked me on .parquet files. smaller, faster to read
", "rawMarkdown": "my previous teammate has hooked me on .parquet files. smaller, faster to read", "votes": null }, { "id": "559315", "postDate": "06/23/2019 22:00:49", "content": "I've been using parquet files. They keep the schema.
\n\nAlso, pandas has the handy read_parquet and to_parquet methods.
", "rawMarkdown": "I've been using parquet files. They keep the schema. \n\nAlso, pandas has the handy read_parquet and to_parquet methods.", "votes": null }, { "id": "559351", "postDate": "06/24/2019 00:56:26", "content": "parquet FTW!
", "rawMarkdown": "parquet FTW!", "votes": null }, { "id": "559394", "postDate": "06/24/2019 04:23:39", "content": "Great shout - I'm going to try it out now!
", "rawMarkdown": "Great shout - I'm going to try it out now!", "votes": null }, { "id": "559438", "postDate": "06/24/2019 05:50:06", "content": "Thanks, all. I will look into HDF5 / parquet options.
", "rawMarkdown": "Thanks, all. I will look into HDF5 / parquet options.", "votes": null }, { "id": "559669", "postDate": "06/24/2019 12:02:42", "content": "Yep, .pkl is great, given that .csv takes an eternity to read. You may also want to try .feather.
", "rawMarkdown": "Yep, .pkl is great, given that .csv takes an eternity to read. You may also want to try .feather.", "votes": null }, { "id": "564112", "postDate": "06/29/2019 01:57:23", "content": "Here is a nice article about the comparison between different file formats for saving pandas data frames.
\n\nSpoiler: 'feather' is the winner.
", "rawMarkdown": "Here is a [nice article](https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d) about the comparison between different file formats for saving pandas data frames. \n\nSpoiler: 'feather' is the winner.", "votes": null }, { "id": "564139", "postDate": "06/29/2019 03:03:22", "content": "Agree with you. I always convert csv to pkl at the beginning. 2 main reasons:\n1) Loading CSV files is really slow.\n2) CSV files won't reserve datatype, I always have more than one scripts. Having inconsistent datatype is not ideal.
", "rawMarkdown": "Agree with you. I always convert csv to pkl at the beginning. 2 main reasons:\n1) Loading CSV files is really slow.\n2) CSV files won't reserve datatype, I always have more than one scripts. Having inconsistent datatype is not ideal.", "votes": null } ], "comments": [ { "id": 559206, "author_name": "iluvmahheart", "author_url": "", "post_date": "06/23/2019 16:36:04", "content": "Never tried this (will this improve our visualisation speed) ? And thanks for sharing I'll try this method 🙌
", "votes": null, "replies": [] }, { "id": 559248, "author_name": "anjum48", "author_url": "", "post_date": "06/23/2019 18:21:22", "content": "I actually prefer HDF5 which can be compressed and retains all the data types\nmagic_feature_df.to_hdf(\"magic_features.h5\", key=\"magic\", complevel=9)
Also for submission files, you can create a zip and upload that directly to Kaggle\nsub.to_csv('submission.zip', index=False, compression='zip')
Any zipped CSV can also be read in the same way without any additional arguments:\ndf = pd.read_csv('other_magic_features.zip')
I also use HDF5 format for this competition, it is fast and takes less disk space.
", "votes": null, "replies": [] } ] }, { "id": 559251, "author_name": "returnofsputnik", "author_url": "", "post_date": "06/23/2019 18:31:39", "content": "my previous teammate has hooked me on .parquet files. smaller, faster to read
", "votes": null, "replies": [ { "id": 559351, "author_name": "robikscube", "author_url": "", "post_date": "06/24/2019 00:56:26", "content": "parquet FTW!
", "votes": null, "replies": [] }, { "id": 559394, "author_name": "anjum48", "author_url": "", "post_date": "06/24/2019 04:23:39", "content": "Great shout - I'm going to try it out now!
", "votes": null, "replies": [] } ] }, { "id": 559315, "author_name": "marcogorelli", "author_url": "", "post_date": "06/23/2019 22:00:49", "content": "I've been using parquet files. They keep the schema.
\n\nAlso, pandas has the handy read_parquet and to_parquet methods.
", "votes": null, "replies": [] }, { "id": 559438, "author_name": "vettejeep", "author_url": "", "post_date": "06/24/2019 05:50:06", "content": "Thanks, all. I will look into HDF5 / parquet options.
", "votes": null, "replies": [] }, { "id": 559669, "author_name": "stanislavblinov", "author_url": "", "post_date": "06/24/2019 12:02:42", "content": "Yep, .pkl is great, given that .csv takes an eternity to read. You may also want to try .feather.
", "votes": null, "replies": [] }, { "id": 564112, "author_name": "graf10a", "author_url": "", "post_date": "06/29/2019 01:57:23", "content": "Here is a nice article about the comparison between different file formats for saving pandas data frames.
\n\nSpoiler: 'feather' is the winner.
", "votes": null, "replies": [] }, { "id": 564139, "author_name": "kingychiu", "author_url": "", "post_date": "06/29/2019 03:03:22", "content": "Agree with you. I always convert csv to pkl at the beginning. 2 main reasons:\n1) Loading CSV files is really slow.\n2) CSV files won't reserve datatype, I always have more than one scripts. Having inconsistent datatype is not ideal.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "559200": "Is it time to ditch csv files? Dealing with some large files lately got me looking around – and Python/pandas has “to\\_pickle” and “read\\_pickle” functions that will work with your favorite file compression scheme. Just choose “.zip” / “.bz2” / “.gzip”, etc for a file extension and the methods will infer the compression type and act accordingly. Seems there is little reason for “.csv” files except for the submission file Kaggle needs. These methods appear to run fast enough to be worth the saved disc space. \n\ntrain.to\\_pickle(os.path.join(FEATURES_DIR, 'myfile.zip')) \n\ntrain = pd.read\\_pickle(os.path.join(FEATURES_DIR, 'myfile.zip'))", "559206": "Never tried this (will this improve our visualisation speed) ? And thanks for sharing I'll try this method 🙌", "559248": "I actually prefer HDF5 which can be compressed and retains all the data types\n`magic_feature_df.to_hdf(\"magic_features.h5\", key=\"magic\", complevel=9)`\n\nAlso for submission files, you can create a zip and upload that directly to Kaggle\n`sub.to_csv('submission.zip', index=False, compression='zip')`\n\nAny zipped CSV can also be read in the same way without any additional arguments:\n`df = pd.read_csv('other_magic_features.zip')`", "559250": "I also use HDF5 format for this competition, it is fast and takes less disk space.", "559251": "my previous teammate has hooked me on .parquet files. smaller, faster to read", "559315": "I've been using parquet files. They keep the schema. \n\nAlso, pandas has the handy read_parquet and to_parquet methods.", "559351": "parquet FTW!", "559394": "Great shout - I'm going to try it out now!", "559438": "Thanks, all. I will look into HDF5 / parquet options.", "559669": "Yep, .pkl is great, given that .csv takes an eternity to read. You may also want to try .feather.", "564112": "Here is a [nice article](https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d) about the comparison between different file formats for saving pandas data frames. \n\nSpoiler: 'feather' is the winner.", "564139": "Agree with you. I always convert csv to pkl at the beginning. 2 main reasons:\n1) Loading CSV files is really slow.\n2) CSV files won't reserve datatype, I always have more than one scripts. Having inconsistent datatype is not ideal." }, "source": "meta" }