{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook was born from this discussion: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285291\n\nI just followed the notebooks cited by:\n* [@faisalalsrheed](https://www.kaggle.com/faisalalsrheed) : https://www.kaggle.com/pedrocouto39/fast-reading-w-pickle-feather-parquet-jay basic explanation\n* [@carlmcbrideellis](https://www.kaggle.com/carlmcbrideellis) : https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets more in-depth knowledge\n\nCheck those notebooks and upvote them as they are gems. ","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-11-04T11:04:42.536942Z","iopub.execute_input":"2021-11-04T11:04:42.537657Z","iopub.status.idle":"2021-11-04T11:04:42.580539Z","shell.execute_reply.started":"2021-11-04T11:04:42.537522Z","shell.execute_reply":"2021-11-04T11:04:42.579362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# datatable installation with internet\n!pip install datatable==0.11.0 > /dev/null","metadata":{"execution":{"iopub.status.busy":"2021-11-04T11:14:05.129145Z","iopub.execute_input":"2021-11-04T11:14:05.129592Z","iopub.status.idle":"2021-11-04T11:14:19.234006Z","shell.execute_reply.started":"2021-11-04T11:14:05.12953Z","shell.execute_reply":"2021-11-04T11:14:19.232987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import datatable as dt","metadata":{"execution":{"iopub.status.busy":"2021-11-04T11:26:55.586807Z","iopub.execute_input":"2021-11-04T11:26:55.587189Z","iopub.status.idle":"2021-11-04T11:26:55.703612Z","shell.execute_reply.started":"2021-11-04T11:26:55.587153Z","shell.execute_reply":"2021-11-04T11:26:55.702763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Format: CSV","metadata":{}},{"cell_type":"code","source":"%%time\n\ndata = pd.read_csv(\"/kaggle/input/g-research-crypto-forecasting/train.csv\")\n\nprint(\"Train size:\", data.shape)","metadata":{"execution":{"iopub.status.busy":"2021-11-04T11:15:28.892436Z","iopub.execute_input":"2021-11-04T11:15:28.892846Z","iopub.status.idle":"2021-11-04T11:16:31.673512Z","shell.execute_reply.started":"2021-11-04T11:15:28.89281Z","shell.execute_reply":"2021-11-04T11:16:31.672452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"But as suggested by [@jpmiller](https://www.kaggle.com/jpmiller) you could also try this:\n\n> You might try flipping the operations here.\n> * Use datatable to read in the csv and convert to pandas (much faster than a pd.read_csv())\n> * Do your thing with the data in pandas\n> * Save the file back as pd.to_parquet() or feather, hd5","metadata":{}},{"cell_type":"code","source":"%%time\ncsvDT = dt.fread(\"/kaggle/input/g-research-crypto-forecasting/train.csv\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \n# Convert datatable to pandas\ndf = csvDT.to_pandas()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Wrtiting data to diferent file formats","metadata":{}},{"cell_type":"code","source":"%%time\n# Writing dataset as hdf5\ndata.to_hdf(\"gCryptoTrain.h5\", \"gCryptoTrain\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Writing dataset as feather\ndata.to_feather(\"gCryptoTrain.feather\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Writing dataset as parquet\ndata.to_parquet(\"gCryptoTrain.parquet\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Writing dataset as pickle\ndata.to_pickle(\"gCryptoTrain.pkl.gzip\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Writing dataset as jay\ndt.Frame(data).to_jay(\"gCryptoTrain.jay\")","metadata":{"execution":{"iopub.status.busy":"2021-11-04T11:26:57.94281Z","iopub.execute_input":"2021-11-04T11:26:57.943749Z","iopub.status.idle":"2021-11-04T11:28:12.854525Z","shell.execute_reply.started":"2021-11-04T11:26:57.943696Z","shell.execute_reply":"2021-11-04T11:28:12.853369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Format: hdf5","metadata":{}},{"cell_type":"code","source":"%%time\n\ndata = pd.read_hdf(\"gCryptoTrain.h5\", \"gCryptoTrain\")\n\nprint(\"Train size:\", data.shape)","metadata":{"execution":{"iopub.status.busy":"2021-11-04T11:32:47.679424Z","iopub.execute_input":"2021-11-04T11:32:47.679874Z","iopub.status.idle":"2021-11-04T11:33:08.197159Z","shell.execute_reply.started":"2021-11-04T11:32:47.679832Z","shell.execute_reply":"2021-11-04T11:33:08.195876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Format: feather","metadata":{}},{"cell_type":"code","source":"%%time\n\ndata = pd.read_feather(\"gCryptoTrain.feather\")\n\nprint(\"Train size:\", data.shape)","metadata":{"execution":{"iopub.status.busy":"2021-11-04T11:34:15.850435Z","iopub.execute_input":"2021-11-04T11:34:15.850804Z","iopub.status.idle":"2021-11-04T11:34:21.846068Z","shell.execute_reply.started":"2021-11-04T11:34:15.850769Z","shell.execute_reply":"2021-11-04T11:34:21.844765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Format: parquet","metadata":{}},{"cell_type":"code","source":"%%time\n\ndata = pd.read_parquet(\"gCryptoTrain.parquet\")\n\nprint(\"Train size:\", data.shape)","metadata":{"execution":{"iopub.status.busy":"2021-11-04T11:36:10.633729Z","iopub.execute_input":"2021-11-04T11:36:10.634795Z","iopub.status.idle":"2021-11-04T11:36:18.198577Z","shell.execute_reply.started":"2021-11-04T11:36:10.634737Z","shell.execute_reply":"2021-11-04T11:36:18.197347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Format: pickle","metadata":{}},{"cell_type":"code","source":"%%time\n\ndata = pd.read_pickle(\"gCryptoTrain.pkl.gzip\")\n\nprint(\"Train size:\", data.shape)","metadata":{"execution":{"iopub.status.busy":"2021-11-04T11:38:19.626863Z","iopub.execute_input":"2021-11-04T11:38:19.627798Z","iopub.status.idle":"2021-11-04T11:38:33.414908Z","shell.execute_reply.started":"2021-11-04T11:38:19.627734Z","shell.execute_reply":"2021-11-04T11:38:33.413803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I would like to point out what [@jpmiller](https://www.kaggle.com/jpmiller) said about pickle. \n\n> Pickle is easy, versatile and fine for personal use. There may be issues though in more general applications.\n> \n> * https://www.benfrederickson.com/dont-pickle-your-data/\n> * https://nedbatchelder.com/blog/202006/pickles_nine_flaws.html","metadata":{}},{"cell_type":"markdown","source":"# Format: jay","metadata":{}},{"cell_type":"code","source":"%%time\n\ndata = dt.fread(\"gCryptoTrain.jay\")\n\nprint(\"Train size:\", data.shape)","metadata":{"execution":{"iopub.status.busy":"2021-11-04T11:39:11.4311Z","iopub.execute_input":"2021-11-04T11:39:11.431546Z","iopub.status.idle":"2021-11-04T11:39:11.542623Z","shell.execute_reply.started":"2021-11-04T11:39:11.431478Z","shell.execute_reply":"2021-11-04T11:39:11.541451Z"},"trusted":true},"execution_count":null,"outputs":[]}]}