Since the data provided for the competition is relatively large, loading it as pd.read_csv takes extremely long (around 1 minute and 10 seconds). However, converting the data into .jay files speeds up the process by a lot (about 19 seconds).
# Converting to Jay Files\nimport pandas as pd\ntrain = pd.read_csv(\"../input/g-research-crypto-forecasting/train.csv\")\nimport datatable as dt\ndt.Frame(train).to_jay(\"g_research_crypto_forecasting.jay\")\nOnce converted to Jay files, read in the data as follow
\nimport datatable as dt\ndata = dt.fread(\"../input/g-research-train-in-jay-format/g_research_crypto_forecasting.jay\").to_pandas()\n#Gives you a Pandas DataFrame\nI hope this method can speed up your data reading process without having to wait for a whole minute just to read in the data.
\nDoes anyone know other ways that can read the data faster? (19 seconds is still pretty slow π I've tried Pickle and Feather)
Since the data provided for the competition is relatively large, loading it as pd.read_csv takes extremely long (around 1 minute and 10 seconds). However, converting the data into .jay files speeds up the process by a lot (about 19 seconds).
# Converting to Jay Files\nimport pandas as pd\ntrain = pd.read_csv(\"../input/g-research-crypto-forecasting/train.csv\")\nimport datatable as dt\ndt.Frame(train).to_jay(\"g_research_crypto_forecasting.jay\")\nOnce converted to Jay files, read in the data as follow
\nimport datatable as dt\ndata = dt.fread(\"../input/g-research-train-in-jay-format/g_research_crypto_forecasting.jay\").to_pandas()\n#Gives you a Pandas DataFrame\nI hope this method can speed up your data reading process without having to wait for a whole minute just to read in the data.
\nDoes anyone know other ways that can read the data faster? (19 seconds is still pretty slow π I've tried Pickle and Feather)
Dear @andy1010
\nMay I heartily recommend the \"Tutorial on reading large datasets\" written by Vopani, introducing alternatives such as Dask, Datatable and the the Rapids cudf, which harnesses the power of the GPU's, and all affording considerable speedups w.r.t. pandas.
All the best,
\ncarl
Hi @andy1010
\nThis might help.
\nhttps://www.kaggle.com/pedrocouto39/fast-reading-w-pickle-feather-parquet-jay
\nhttps://www.kaggle.com/c/jane-street-market-prediction/discussion/199233
\nhttps://www.kaggle.com/pedrocouto39/jane-street-market-train-data-best-formats
wow, jay seems super fast. Thanks for the links @faisalalsrheed
", "rawMarkdown": "wow, jay seems super fast. Thanks for the links @faisalalsrheed", "votes": null }, { "id": "1570764", "postDate": "11/04/2021 11:02:55", "content": "Already read that notebook, but still the best in this topic. Thanks @carlmcbrideellis
", "rawMarkdown": "Already read that notebook, but still the best in this topic. Thanks @carlmcbrideellis", "votes": null }, { "id": "1570832", "postDate": "11/04/2021 12:09:26", "content": "Alright, this question made me curious and I had to try. I made a notebook with the suggestions from: @faisalalsrheed and @carlmcbrideellis using the training data from the competition: https://www.kaggle.com/ajpass/reading-speed-comparation-between-file-formats . Hope it helps you :D
\nAbstract: Jay format is the fastest but uses a library called datatable which doesn't have a very exhaustive set of data processing functions (as said in https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets).
\nP.S.: I also added all the advice given by @jpmiller which I think it's fantastic.
", "rawMarkdown": "Alright, this question made me curious and I had to try. I made a notebook with the suggestions from: @faisalalsrheed and @carlmcbrideellis using the training data from the competition: https://www.kaggle.com/ajpass/reading-speed-comparation-between-file-formats . Hope it helps you :D\n\nAbstract: Jay format is the fastest but uses a library called datatable which doesn't have a very exhaustive set of data processing functions (as said in https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets).\n\nP.S.: I also added all the advice given by @jpmiller which I think it's fantastic.", "votes": null }, { "id": "1570849", "postDate": "11/04/2021 12:30:34", "content": "You might want to build a smaller data set for debugging purposes.
", "rawMarkdown": "You might want to build a smaller data set for debugging purposes.", "votes": null }, { "id": "1570995", "postDate": "11/04/2021 14:34:45", "content": "I'm not sure where you getting 19sec from, but your method is pretty fast for me, thanks!
\nstats from kaggle cpu kernel:
\n%%time
\ndata_folder = \"../input/g-research-crypto-forecasting/\"
\ndata_folder2 = \"../input/gresearch-crypto-forcasting-train/\"
\nβ
\nasset_details = pd.read_csv(data_folder + 'asset_details.csv')
\ncrypto_df = dt.fread(data_folder2 + \"train.jay\").to_pandas()
CPU times: user 908 ms, sys: 490 ms, total: 1.4 s
\nWall time: 987 ms
Hmm, that's quite interesting. For me, reading in the data only takes about 25 milliseconds, however the conversion to pandas takes 15~ seconds.
", "rawMarkdown": "Hmm, that's quite interesting. For me, reading in the data only takes about 25 milliseconds, however the conversion to pandas takes 15~ seconds.", "votes": null }, { "id": "1571046", "postDate": "11/04/2021 14:57:56", "content": "Thanks for the nice notebook, very helpful resource!
", "rawMarkdown": "Thanks for the nice notebook, very helpful resource!", "votes": null }, { "id": "1571052", "postDate": "11/04/2021 14:59:49", "content": "Yes, I did get similar results using other data formats and Jay was the fastest. However, the process that's taking the longest for me is converting to pandas DataFrame since as mentioned, datatable does not have a very wide range of data processing functions. Thanks for the help anyways!
", "rawMarkdown": "Yes, I did get similar results using other data formats and Jay was the fastest. However, the process that's taking the longest for me is converting to pandas DataFrame since as mentioned, datatable does not have a very wide range of data processing functions. Thanks for the help anyways!", "votes": null }, { "id": "1571787", "postDate": "11/05/2021 06:48:55", "content": "Using pickle is also a good way.
", "rawMarkdown": "Using pickle is also a good way.", "votes": null }, { "id": "1572190", "postDate": "11/05/2021 13:46:24", "content": "Just putting it out thereβ¦
\nvroom package in R: 12.562 seconds
You might try flipping the operations here.
\npd.read_csv())pd.to_parquet() or feather, hd5Pickle is easy, versatile and fine for personal use. There may be issues though in more general applications.
\nThanks for reminding ! π
", "rawMarkdown": "Thanks for reminding ! π", "votes": null }, { "id": "1573111", "postDate": "11/06/2021 09:32:46", "content": "Thank you so much for taking your time for making this post. Also, When pd.read_csv take 1 minute 19s for me, jay made it in 2 seconds. Insane difference.
", "rawMarkdown": "Thank you so much for taking your time for making this post. Also, When pd.read_csv take 1 minute 19s for me, jay made it in 2 seconds. Insane difference.", "votes": null }, { "id": "1573200", "postDate": "11/06/2021 11:44:27", "content": "Don't know why i didn't think about that, nice point @jpmiller
\nAs suggested:
\nGood advice, I will read data this way for my next notebook.
\n@andy1010 you can check this page for a comparison between datatable and pandas
", "rawMarkdown": "Don't know why i didn't think about that, nice point @jpmiller \n\nAs suggested:\n- Install datatable: ~9seconds\n- csvDT = dt.fread(mainFolder + 'train.csv'): ~9 seconds\n- df = csvDT.to_pandas(): ~2 seconds \n\nGood advice, I will read data this way for my next notebook.\n\n@andy1010 you can check this page for a[ comparison between datatable and pandas](https://datatable.readthedocs.io/en/latest/manual/comparison_with_pandas.html)", "votes": null }, { "id": "1573231", "postDate": "11/06/2021 12:14:11", "content": "Didn't know about that @jpmiller I will keep it in mind thanks.
", "rawMarkdown": "Didn't know about that @jpmiller I will keep it in mind thanks.", "votes": null }, { "id": "1574421", "postDate": "11/07/2021 14:27:40", "content": "You're right. You could also divide the dataset by assets. And later on you may work with the assets relationships.
", "rawMarkdown": "You're right. You could also divide the dataset by assets. And later on you may work with the assets relationships.", "votes": null } ], "comments": [ { "id": 1570401, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "11/04/2021 05:43:30", "content": "Dear @andy1010
\nMay I heartily recommend the \"Tutorial on reading large datasets\" written by Vopani, introducing alternatives such as Dask, Datatable and the the Rapids cudf, which harnesses the power of the GPU's, and all affording considerable speedups w.r.t. pandas.
All the best,
\ncarl
Already read that notebook, but still the best in this topic. Thanks @carlmcbrideellis
", "votes": null, "replies": [] }, { "id": 1571046, "author_name": "andy1010", "author_url": "", "post_date": "11/04/2021 14:57:56", "content": "Thanks for the nice notebook, very helpful resource!
", "votes": null, "replies": [] } ] }, { "id": 1570595, "author_name": "faisalalsrheed", "author_url": "", "post_date": "11/04/2021 08:53:36", "content": "Hi @andy1010
\nThis might help.
\nhttps://www.kaggle.com/pedrocouto39/fast-reading-w-pickle-feather-parquet-jay
\nhttps://www.kaggle.com/c/jane-street-market-prediction/discussion/199233
\nhttps://www.kaggle.com/pedrocouto39/jane-street-market-train-data-best-formats
wow, jay seems super fast. Thanks for the links @faisalalsrheed
", "votes": null, "replies": [] } ] }, { "id": 1570832, "author_name": "ajpass", "author_url": "", "post_date": "11/04/2021 12:09:26", "content": "Alright, this question made me curious and I had to try. I made a notebook with the suggestions from: @faisalalsrheed and @carlmcbrideellis using the training data from the competition: https://www.kaggle.com/ajpass/reading-speed-comparation-between-file-formats . Hope it helps you :D
\nAbstract: Jay format is the fastest but uses a library called datatable which doesn't have a very exhaustive set of data processing functions (as said in https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets).
\nP.S.: I also added all the advice given by @jpmiller which I think it's fantastic.
", "votes": null, "replies": [ { "id": 1571052, "author_name": "andy1010", "author_url": "", "post_date": "11/04/2021 14:59:49", "content": "Yes, I did get similar results using other data formats and Jay was the fastest. However, the process that's taking the longest for me is converting to pandas DataFrame since as mentioned, datatable does not have a very wide range of data processing functions. Thanks for the help anyways!
", "votes": null, "replies": [] }, { "id": 1572852, "author_name": "jpmiller", "author_url": "", "post_date": "11/06/2021 03:19:25", "content": "You might try flipping the operations here.
\npd.read_csv())pd.to_parquet() or feather, hd5Don't know why i didn't think about that, nice point @jpmiller
\nAs suggested:
\nGood advice, I will read data this way for my next notebook.
\n@andy1010 you can check this page for a comparison between datatable and pandas
", "votes": null, "replies": [] } ] }, { "id": 1570849, "author_name": "lucasmorin", "author_url": "", "post_date": "11/04/2021 12:30:34", "content": "You might want to build a smaller data set for debugging purposes.
", "votes": null, "replies": [ { "id": 1574421, "author_name": "ajpass", "author_url": "", "post_date": "11/07/2021 14:27:40", "content": "You're right. You could also divide the dataset by assets. And later on you may work with the assets relationships.
", "votes": null, "replies": [] } ] }, { "id": 1570995, "author_name": "danielbruintjies", "author_url": "", "post_date": "11/04/2021 14:34:45", "content": "I'm not sure where you getting 19sec from, but your method is pretty fast for me, thanks!
\nstats from kaggle cpu kernel:
\n%%time
\ndata_folder = \"../input/g-research-crypto-forecasting/\"
\ndata_folder2 = \"../input/gresearch-crypto-forcasting-train/\"
\nβ
\nasset_details = pd.read_csv(data_folder + 'asset_details.csv')
\ncrypto_df = dt.fread(data_folder2 + \"train.jay\").to_pandas()
CPU times: user 908 ms, sys: 490 ms, total: 1.4 s
\nWall time: 987 ms
Hmm, that's quite interesting. For me, reading in the data only takes about 25 milliseconds, however the conversion to pandas takes 15~ seconds.
", "votes": null, "replies": [] } ] }, { "id": 1571787, "author_name": "pbokieee", "author_url": "", "post_date": "11/05/2021 06:48:55", "content": "Using pickle is also a good way.
", "votes": null, "replies": [ { "id": 1572864, "author_name": "jpmiller", "author_url": "", "post_date": "11/06/2021 03:30:40", "content": "Pickle is easy, versatile and fine for personal use. There may be issues though in more general applications.
\nThanks for reminding ! π
", "votes": null, "replies": [] }, { "id": 1573231, "author_name": "ajpass", "author_url": "", "post_date": "11/06/2021 12:14:11", "content": "Didn't know about that @jpmiller I will keep it in mind thanks.
", "votes": null, "replies": [] } ] }, { "id": 1572190, "author_name": "jaredsavage", "author_url": "", "post_date": "11/05/2021 13:46:24", "content": "Just putting it out thereβ¦
\nvroom package in R: 12.562 seconds
Thank you so much for taking your time for making this post. Also, When pd.read_csv take 1 minute 19s for me, jay made it in 2 seconds. Insane difference.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1570343": "Since the data provided for the competition is relatively large, loading it as `pd.read_csv` takes extremely long (around 1 minute and 10 seconds). However, converting the data into `.jay` files speeds up the process by a lot (about 19 seconds).\n```\n# Converting to Jay Files\nimport pandas as pd\ntrain = pd.read_csv(\"../input/g-research-crypto-forecasting/train.csv\")\nimport datatable as dt\ndt.Frame(train).to_jay(\"g_research_crypto_forecasting.jay\")\n```\nOnce converted to Jay files, read in the data as follow\n```\nimport datatable as dt\ndata = dt.fread(\"../input/g-research-train-in-jay-format/g_research_crypto_forecasting.jay\").to_pandas()\n#Gives you a Pandas DataFrame\n```\nI hope this method can speed up your data reading process without having to wait for a whole minute just to read in the data.\nDoes anyone know other ways that can read the data faster? (19 seconds is still pretty slow π I've tried Pickle and Feather)", "1570401": "Dear @andy1010 \n\n May I heartily recommend the [\"Tutorial on reading large datasets\"](https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets) written by [Vopani](https://www.kaggle.com/rohanrao), introducing alternatives such as Dask, Datatable and the the Rapids `cudf`, which harnesses the power of the GPU's, and all affording considerable speedups w.r.t. pandas. \n\nAll the best,\ncarl", "1570595": "Hi @andy1010 \n\nThis might help. \nhttps://www.kaggle.com/pedrocouto39/fast-reading-w-pickle-feather-parquet-jay\nhttps://www.kaggle.com/c/jane-street-market-prediction/discussion/199233\nhttps://www.kaggle.com/pedrocouto39/jane-street-market-train-data-best-formats", "1570748": "wow, jay seems super fast. Thanks for the links @faisalalsrheed", "1570764": "Already read that notebook, but still the best in this topic. Thanks @carlmcbrideellis", "1570832": "Alright, this question made me curious and I had to try. I made a notebook with the suggestions from: @faisalalsrheed and @carlmcbrideellis using the training data from the competition: https://www.kaggle.com/ajpass/reading-speed-comparation-between-file-formats . Hope it helps you :D\n\nAbstract: Jay format is the fastest but uses a library called datatable which doesn't have a very exhaustive set of data processing functions (as said in https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets).\n\nP.S.: I also added all the advice given by @jpmiller which I think it's fantastic.", "1570849": "You might want to build a smaller data set for debugging purposes.", "1570995": "I'm not sure where you getting 19sec from, but your method is pretty fast for me, thanks!\n\nstats from kaggle cpu kernel:\n\n%%time\ndata_folder = \"../input/g-research-crypto-forecasting/\"\ndata_folder2 = \"../input/gresearch-crypto-forcasting-train/\"\nβ\nasset_details = pd.read_csv(data_folder + 'asset_details.csv')\ncrypto_df = dt.fread(data_folder2 + \"train.jay\").to_pandas()\n\nCPU times: user 908 ms, sys: 490 ms, total: 1.4 s\nWall time: 987 ms", "1571044": "Hmm, that's quite interesting. For me, reading in the data only takes about 25 milliseconds, however the conversion to pandas takes 15~ seconds.", "1571046": "Thanks for the nice notebook, very helpful resource!", "1571052": "Yes, I did get similar results using other data formats and Jay was the fastest. However, the process that's taking the longest for me is converting to pandas DataFrame since as mentioned, datatable does not have a very wide range of data processing functions. Thanks for the help anyways!", "1571787": "Using pickle is also a good way.", "1572190": "Just putting it out there... \nvroom package in R: 12.562 seconds", "1572852": "You might try flipping the operations here.\n- Use datatable to read in the csv and convert to pandas (much faster than a `pd.read_csv()`)\n- Do your thing with the data in pandas\n- Save the file back as `pd.to_parquet()` or feather, hd5", "1572864": "Pickle is easy, versatile and fine for personal use. There may be issues though in more general applications.\n- https://www.benfrederickson.com/dont-pickle-your-data/\n- https://nedbatchelder.com/blog/202006/pickles_nine_flaws.html", "1572874": "Thanks for reminding ! π", "1573111": "Thank you so much for taking your time for making this post. Also, When pd.read_csv take 1 minute 19s for me, jay made it in 2 seconds. Insane difference.", "1573200": "Don't know why i didn't think about that, nice point @jpmiller \n\nAs suggested:\n- Install datatable: ~9seconds\n- csvDT = dt.fread(mainFolder + 'train.csv'): ~9 seconds\n- df = csvDT.to_pandas(): ~2 seconds \n\nGood advice, I will read data this way for my next notebook.\n\n@andy1010 you can check this page for a[ comparison between datatable and pandas](https://datatable.readthedocs.io/en/latest/manual/comparison_with_pandas.html)", "1573231": "Didn't know about that @jpmiller I will keep it in mind thanks.", "1574421": "You're right. You could also divide the dataset by assets. And later on you may work with the assets relationships." }, "source": "meta" }