{"nbformat_minor": 1, "nbformat": 4, "metadata": {"kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}, "language_info": {"name": "python", "nbconvert_exporter": "python", "mimetype": "text/x-python", "version": "3.6.3", "pygments_lexer": "ipython3", "codemirror_mode": {"version": 3, "name": "ipython"}, "file_extension": ".py"}}, "cells": [{"source": ["# The Problem:\n", "#### The training data, `train.csv` is a large file (~5 GB) - this can be problematic if you have relatively low RAM (8 GB).\n", "# The Solution:\n", "## Set `low_memory=True` in Pandas' `read_csv`\n", "On a machine with relatively low RAM, attempting to load the entire file in a pandas DataFrame can lead to failure caused by running out of memory.  One way of fixing this issue is to make use of the `low_memory=True` argument of `read_csv`.  With this method, the csv file is processed in chunks requiring lower memory usage, while at the same time reading the csv's contents into a single DataFrame.\n", "## But the Jupyter kernel still keeps restarting even with `low_memory=True`....why?\n", "The dtypes of the columns of the DataFrame must be specified in `read_csv` if we wish to set `low_memory=True`.  This is because not specifying dtypes forces pandas to guess column dtypes - which is a memory-intensive task.  Please see this Stack Overflow answer for a additional explanation:\n", "https://stackoverflow.com/a/27232309\n", "\n", "# The Complete Solution\n", "We first create a new file called `small_train.csv` using only the first row of data from `train.csv`:\n"], "cell_type": "markdown", "metadata": {"_uuid": "4258145e50f3814f07c45d8fbb6e0e50c516ee91", "_cell_guid": "ef6030cf-ab59-4be8-8cc7-faf2715b20ef"}}, {"source": ["!head -2 train.csv > small_train.csv"], "outputs": [], "execution_count": null, "cell_type": "code", "metadata": {"collapsed": true, "_kg_hide-output": true, "_uuid": "ffada2307da93888255c99fd53e47fcab02ce941", "_cell_guid": "7fba4035-2aaa-4380-90da-3124f38c64d6"}}, {"source": ["import pandas as pd"], "outputs": [], "execution_count": null, "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "58fcad59f711203e9b684c6d2fcdf1de0f78bcb2", "_cell_guid": "2cc41c75-e8e2-4cd6-8ec5-70b25607fb6a"}}, {"source": ["small_train = pd.read_csv('../input/small-train/small_train.csv')\n", "print(small_train)"], "outputs": [], "execution_count": null, "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "c01fd34d513bb84d236676785961b0424fbacfb8", "_cell_guid": "4f2a7b6e-f0ee-432a-bccc-e84ead6d1251"}}, {"source": ["types_dict = small_train.dtypes.to_dict()\n", "types_dict"], "outputs": [], "execution_count": null, "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "535627a3d54e8a7525826041337b4dde319bd9fc", "_cell_guid": "270eb4de-b367-48ca-b6b3-9123cdec9169"}}, {"source": ["Next, let's update types of some columns to make them more memory efficient.  This is based on information shared in the following kernel:\n", "\n", "https://www.kaggle.com/jagangupta/memory-optimization-and-eda-on-entire-dataset\n", "\n", "I highly recommend looking at the link above - it shows additonal steps for making your dataframe even more memory efficient."], "cell_type": "markdown", "metadata": {"_uuid": "b0475a3184c1c9b00e14432bb18c60ebf14d9d6b", "_cell_guid": "53509fd8-e230-41b7-baa7-4e061a4f2255"}}, {"source": ["types_dict = {'id': 'int32',\n", "             'item_nbr': 'int32',\n", "             'store_nbr': 'int8',\n", "             'unit_sales': 'float32'}"], "outputs": [], "execution_count": null, "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "fad1f97aa8874e3877daed34394c34569cc1eac4", "_cell_guid": "ce0ae4d8-fb05-4eb6-997b-d5fc94b4eddb"}}, {"source": ["Now, we can use `types_dict` to specify the dtypes of each column of the DataFrame we are loading the `train.csv` file into:"], "cell_type": "markdown", "metadata": {"_uuid": "4f7f758e6c5d7a10cc3cc1260e1d60405ec766e9", "_cell_guid": "78cc2c29-71ad-4439-895d-3116222a4f57"}}, {"source": ["grocery_train = pd.read_csv('train.csv', low_memory=True, dtype=types_dict)"], "outputs": [], "execution_count": null, "cell_type": "code", "metadata": {"collapsed": true, "_kg_hide-output": true, "_uuid": "ffa1b2d4517c0c5121b45258235cb173dba303c7", "_cell_guid": "41869637-f53a-42ce-8be9-721b4214d153"}}, {"source": ["The steps above will let you load the entire 5 GB file in memory without crashing the Jupyter kernel."], "cell_type": "markdown", "metadata": {"_uuid": "9bc385deacbc2f64eda75fa15270521fbb78a432", "_cell_guid": "bcf27399-0864-4148-ab90-9316ad71fdd9"}}, {"source": ["# Feather Format: Quickly Reloading Saved Training Dataframe"], "cell_type": "markdown", "metadata": {"_uuid": "90cff02b05be594f12b416a606dcf0d90b354357", "_cell_guid": "6d1d67c9-109c-41a0-9768-80fe99794b95"}}, {"source": ["Every time you reopen your Jupyter notebook, you need not rerun the steps shown in the previous section.  Instead, simply use the **feather** format to save the `grocery_train` dataframe after you load it in memory the first time.  The feather format enables very fast read and write access for working with dataframes, both in `R` and `Python` (read more here: https://blog.rstudio.com/2016/03/29/feather/).  Note that your pandas version must be 0.20.0 or newer for the code below to work."], "cell_type": "markdown", "metadata": {"_uuid": "c1f67f8d8aa223570ae83eca7fa657be3070494b", "_cell_guid": "d29e2d25-abec-4cf5-bab3-220354b16068"}}, {"source": ["os.makedirs('tmp', exist_ok=True)  # Make a temp dir for storing the feather file\n", "# Save feather file, requires pandas 0.20.0 at least:\n", "grocery_train.to_feather('./tmp/grocery_train_raw')"], "outputs": [], "execution_count": null, "cell_type": "code", "metadata": {"collapsed": true, "_kg_hide-output": true, "_uuid": "b1af06f5e7142af304e26670f30918846f6522f4", "_cell_guid": "4079c008-dedf-4754-910c-fd6c6eb0c14c"}}, {"source": ["### Going forward, you can read the `grocery_train` dataframe directly from the feather file as shown below:"], "cell_type": "markdown", "metadata": {"_uuid": "19298fe83de0e1a0599f2d7b15d169ddd8b51858", "_cell_guid": "8cc49164-6bb8-4362-8f93-7dc53c503ed0"}}, {"source": ["grocery_train = pd.read_feather('./tmp/train_sub_raw')"], "outputs": [], "execution_count": null, "cell_type": "code", "metadata": {"collapsed": true, "_kg_hide-output": true, "_uuid": "e52a56e9c29beedb33042d9cb12291a737e5f603", "_cell_guid": "31cb312f-41bf-4517-baf5-06c61f3f7e26"}}]}