{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"In this notebook, I will show you an easy workflow to get the historical transactions dataset \nfrom a **13.1GB loaded dataset** to a **much smaller one**. Notice that the techniques showcased in what follows are applicable to a wide range of datasets of course. \nLet's get started!"},{"metadata":{"trusted":true,"_uuid":"3a94b32b87ccfe6a4070983ea135c9fea018d035"},"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/big-to-small-filepng/big_to_small_file.png\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3dcd9816106c9a294cc9843f076390b3e4bb4aa8"},"cell_type":"markdown","source":"<center><h1>Sometimes, small is better!</h1></center>\nsource (with some adaptation): https://bulbapedia.bulbagarden.net"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Why should you bother?"},{"metadata":{"_uuid":"9a2823fa562fd7106d9f003c255f6a7018531c5d"},"cell_type":"markdown","source":"First, it is \"fun\" to do it, in the sense that it is challenging, interesting to learn \nto do it, and finally, could be useful. \nHow could it be useful?\nWell, let's see: \n\n* you need to train a gradient boosted trees model with every possible training dataset you have. Unfortunately, your laptop has only 8GB of RAM. \n\n* you have access to cloud resources but to get your model working, you need a much bigger instance which costs 3 times more.\n\n* you have access to a large cloud instance that fits everything but the training time is high and you want to reduce it. \n\nIn all cases, your boss will be happy that you have managed to train with the largest amount of data, using the least amout of resources necessary, and finished the training in a reasonable time. \n\n"},{"metadata":{"_uuid":"e8796835416cdcc2d253972773944d94a73b9810"},"cell_type":"markdown","source":"# Before optimization"},{"metadata":{"trusted":true,"_uuid":"18610c8fcc67711a1bd1fc20091d14d418a9d2f9"},"cell_type":"code","source":"# Some imports\nimport pandas as pd\nDATA_PATH = \"../input/elo-merchant-category-recommendation/historical_transactions.csv\"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5baa374d9cc835b65d2bb56f56b8203eff32660e"},"cell_type":"markdown","source":"Let's start by loading the dataset using the good old `pd.read_csv` and timeit (using the `%%timeit` magic command). "},{"metadata":{"trusted":true,"_uuid":"ed59317cc734191f90864cfb3ae106fb0bb64851"},"cell_type":"code","source":"df = pd.read_csv(DATA_PATH)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e9ccae532fdb8a4cc069982915c87a0c64d6541d"},"cell_type":"markdown","source":"Let's also time the loading process"},{"metadata":{"trusted":true,"_uuid":"fa72694c912e8ce63bad3004397fb6e753715466"},"cell_type":"code","source":"%%timeit\ndf = pd.read_csv(DATA_PATH)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66ec1be096b8e269cda03605a40ab7d61821d97e"},"cell_type":"markdown","source":"Around **1 minute** to load the historical transactions dataset!\nThat's not negligible. Alright, let's check how much space it takes on \ndisk first. For that, will issue the following `bash` command: `ls -lh` (the `h` flag is for getting a human-readable output). \n\nIf you don't know it, you can issue `bash` commands right from a[ jupyter notebook](https://jupyter.org/) (with or without the `!` sign before the command since `automagic` is turned on by default). Check this great [blog post](https://jakevdp.github.io/PythonDataScienceHandbook/01.05-ipython-and-shell-commands.html) for more details.\n                                        "},{"metadata":{"trusted":true,"_uuid":"de0547bdbd9f89df01f179065bb3e9ce8c0ab938"},"cell_type":"code","source":"ls -lh {DATA_PATH}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3c31b520acf43b38f5a5cf8ee9667215fc1a4dbf"},"cell_type":"markdown","source":"**2.7GB** on disk. That's a large dataset! Not yet \"big data\" but could make older computers flinch. \n\nAlright, the next question to ask is: how to get the size of this dataset when loaded into memory?"},{"metadata":{"_uuid":"3f3c9fc2c4290e67186492391fc64801bffeefcb"},"cell_type":"markdown","source":"# Memory footprint"},{"metadata":{"_uuid":"ae59211295abcc28fbeeb0db10f149d66c58d413"},"cell_type":"markdown","source":"For that, I will be using a pandas method (of course, there is usually a pandas method for almost everything): [`pandas.DataFrame.info`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.info.html). Let's see what we get. "},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"ea656afcc49ddeb078a484dc92d217ba29aa0acc"},"cell_type":"code","source":"# verbose is set to False here to avoid the metadata information\ndf.info(verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f1a7a8833c78cc3aa59367b11e378ccf8e0f9335"},"cell_type":"markdown","source":"**3GB**. That's not that bad. \n\nBut wait, is this really the memory size? **Why is there a `+` sign at the end?** That looks suspecious..."},{"metadata":{"_uuid":"a5fa2613dd18de4de716078a87f49be081c32290"},"cell_type":"markdown","source":"# Real memory footprint"},{"metadata":{"_uuid":"6e8c52904c25665f41d936848742aab3b13e8c0f"},"cell_type":"markdown","source":"The answer to the previous questions is: no, it isn't!\nAnd the `+` sign is here to indicate that the returned value is an **estimation**. \n\nOk, so why is that?\n\nThe answer to that could be summed up in one word: `object`.\nIn fact, when calling the `.info` method, one doesn't get the \"real\" memory footprint bur rather an estimation. \n\nFrom the [`pandas.DataFrame.memory`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.info.html#pandas.DataFrame.info) documentation, here is how it is computed: \n\n> True always show memory usage. False never shows memory usage. A value of ‘deep’ is equivalent to “True with deep introspection”. Memory usage is shown in human-readable units (base-2 representation). Without deep introspection a memory estimation is made based in column dtype and number of rows assuming values consume the same memory amount for corresponding dtypes. With deep memory introspection, a real memory usage calculation is performed at the cost of computational resources.\n\n\nSo what is the correct way to get it? \nIt isn't that hard either, just use the same method again but this time setting `memory_usage=\"deep\"`. By doing so, `pandas` will do the real memory usage computation (thus computing how much space `object` data takes). \nSimple!"},{"metadata":{"trusted":true,"_uuid":"28fea4e8eea72bf7d562137a90d33e10dc41425a","scrolled":true},"cell_type":"code","source":"df.info(memory_usage=\"deep\", verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"976859ab727a03fd1fb04c758c072a5ea930875d"},"cell_type":"markdown","source":"The correct answer is thus **13.1GB** (but you knew it already if you paid attention to the introduction). That's a huge DataFrame loaded into memory, one that is more than 4 times bigger than the original estimate. \n\nCan we do something about it? Of course, otherwise this notebook won't make sense. ;)"},{"metadata":{"_uuid":"aad604c976133a016740c58fe21539929dc7dc3b"},"cell_type":"markdown","source":"# Dtypes"},{"metadata":{"_uuid":"da32c0759763e8425c5e262ac25f0c182243f678"},"cell_type":"markdown","source":"![df_blocks.png](https://www.dataquest.io/blog/content/images/df_blocks.png)\n<center><h1>Pandas block representation</h1></center>\nsource: https://www.dataquest.io/blog/pandas-big-data/ "},{"metadata":{"_uuid":"5b7ebdf372cd19fa0e4731dcaee7fb53010719de"},"cell_type":"markdown","source":"As mentionned earlier, the \"heavy\" load comes mostly from the `object` type (and the associated `ObjectBlock`). In simpler words, an `object` dtype is how pandas stores strings. For that, it uses python and not numpy (contrary to all the other types). Check this [thread](https://stackoverflow.com/questions/34881079/pandas-distinction-between-str-and-object-types) for some explanations why.  \n\nLet's see which columns have this type and see how much they contribute to the overall memory footprint. "},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"47b22cbd1f2378a8eac07109db2d54ef39d04bba"},"cell_type":"code","source":"# Verbose is left to the default True here since we want the columns metadata.\ndf.select_dtypes('object').info(memory_usage='deep')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b33769a1fc0853f0f2d2002fc038355dbc57356"},"cell_type":"markdown","source":"Waw, 11.3GB! That's around **86%** of the total memory footprint!\nAlright, what can we do to reduce it?\n\nThere is probably only one solution I can think of, that is casting the `object`colums to another, **more efficient **type representation (for example, integer) ** while preserving the information**.  Let's do this. "},{"metadata":{"_uuid":"d57baa1de87e6270925c863b10fe0519a42af367"},"cell_type":"markdown","source":"# Exploring the object columns"},{"metadata":{"trusted":true,"_uuid":"4edc7cd9a8e662ac61176cbbee09ec5515311c9b"},"cell_type":"markdown","source":"Before casting to the appropriate type, we need to explore the columns to find out the best \none. In what follows, I will display few values of each column, count the number of unique values and compare it to the length of the column. "},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"38d65aca832e8403b5c40a91223c62993a6b5a87"},"cell_type":"code","source":"for col in df.select_dtypes('object'):\n    print(df[col].sample(5))\n    print(f\"{df[col].nunique()} unique values for {col}, which has {len(df[col])} rows.\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b9e90619feacf6dc577bdd07346cdcf60221246"},"cell_type":"markdown","source":"# Timestamps anyone?"},{"metadata":{"_uuid":"c0d4cfa122429373294c623b92244939576c6c31"},"cell_type":"markdown","source":"Alright, the `purchase_date` contains temporal information, so let's turn it into a `datetime` type using the [`pandas.to_datetime`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html) function. "},{"metadata":{"trusted":true,"_uuid":"28cda6c5ef66cd89a889af7fef16cf9a9b2d78e7"},"cell_type":"code","source":"df.purchase_date = pd.to_datetime(df.purchase_date)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3d718488357b0b35b37ef350c81707ab03c6f24"},"cell_type":"markdown","source":"Alright, what is the new memory footprint?"},{"metadata":{"trusted":true,"_uuid":"f148b0f9ec6c4ef22b8027943090feed63a8e8e3"},"cell_type":"code","source":"df.info(memory_usage=\"deep\", verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3cb7417a72f225f9753d3627909e9ebe5b10e3c"},"cell_type":"markdown","source":"Not bad for a start!"},{"metadata":{"_uuid":"c86aef4b1b95edb77ec399df79bd0edc9c48a488"},"cell_type":"markdown","source":"# Categorical to the rescue"},{"metadata":{"_uuid":"0972dc45c11c5d5b9706fe21d2e0edab6ff3e77e"},"cell_type":"markdown","source":"Next, any column with \"textual\" information and having more than 3 unique values \nand less than, say, 60% of the column length, should be transformed into the [categorical](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Categorical.html) \ntype. Some background information about this type: it is a fairly new addition to pandas \n(since version 0.21.0) and is inspired from the R one. \nTo do so, will use the `astype(\"category\")` method. "},{"metadata":{"_uuid":"c4db26295c8fe2abc4e56f5f70ba3294a443010f","trusted":true},"cell_type":"code","source":"CATEGORICAL_COLS = [\"card_id\", \"category_3\", \"merchant_id\"]\nfor col in[\"card_id\", \"category_3\", \"merchant_id\"]:\n    df[col] = df[col].astype(\"category\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"50fc327af22835d0d2f0e960825bd5b9886aa90f"},"cell_type":"code","source":"df.info(memory_usage=\"deep\", verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9332ae19cb0fe8287a49ba064752b6ce9dceff4"},"cell_type":"markdown","source":"That's a huge gain!"},{"metadata":{"_uuid":"b6137df01ed8e5ec1d7db083846a637913d9eb11"},"cell_type":"markdown","source":"# Binarize some features"},{"metadata":{"_uuid":"2a1072e4eafa715b705bdafd1c591a50d78b3011"},"cell_type":"markdown","source":"What about the other `object` columns? These are neither timestamps and have only 2 unique textual \nvalues. So what to do about thses? \nBinarize them! Let's see how to do it."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"b44da3814c1b1e3d0f51686d060e5eec56771e9e"},"cell_type":"code","source":"for col in [\"authorized_flag\", \"category_1\"]:\n    # Each row having \"Y\" (short for yes) will get the value 1, otherwise, 0.\n    df[col] = pd.np.where(df[col] == \"Y\", 1, 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"4ad5b4c6c65a985b3195d5c7180fd995bb43b671"},"cell_type":"code","source":"df.info(memory_usage=\"deep\", verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36faf4abc1b3f585e194857d05f971690cd10a2d"},"cell_type":"markdown","source":"We are on a roll!"},{"metadata":{"_uuid":"f72f4f5ebaf6bd20a0c9d31feb17a1d2ab2d8ced"},"cell_type":"markdown","source":"# What about other categorical columns?"},{"metadata":{"_uuid":"de98598d6f7bd1185dc138b5f50d3170d091374c"},"cell_type":"markdown","source":"After more exploration, it appears that other columns aren't of `object` type but could be \nturned into categoricals to save some more space. Let's do it.!"},{"metadata":{"trusted":true,"_uuid":"ef8cbe0cabb92cd42a6553a55add6e3716f2eeae"},"cell_type":"code","source":"df.nunique().sort_values(ascending=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af49a495bbf108d9d3dd95d8cf025e6f30846f26"},"cell_type":"code","source":"# Be careful, even though it is tempting to turn the \"purchase_amount\" to\n# categorical to gain more space, \n# it isn't the best thing to do since we will be using this column to compute\n# aggregations!\nfor col in [\"month_lag\", \"installments\", \"state_id\", \"subsector_id\", \n            \"city_id\", \"merchant_category_id\", \"merchant_id\"]:\n    df[col] = df[col].astype(\"category\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"33170764aa4ad8581011dc223dc78cfefbf15918"},"cell_type":"code","source":"df.info(memory_usage=\"deep\", verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"fa5dff4e17ce2b785e6a1185900feb413576465c"},"cell_type":"code","source":"df.dtypes","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0e1bbbee4460e275a0ecf2e80692b0e16679912b"},"cell_type":"markdown","source":"# Integer with NaNs"},{"metadata":{"_uuid":"344a7a911ccde20547334fc5ed0e326a40e9b6ef"},"cell_type":"markdown","source":"One last thing before leaving, there is the `category_2` that is a `float64` column. Why is that? To see why, let's plot the distribution of the unique values. "},{"metadata":{"trusted":true,"_uuid":"b482661398b04e1f319c2c945671c26fc0fb4c7a"},"cell_type":"code","source":"df.category_2.value_counts(dropna=False, normalize=True).plot(kind='bar')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ca6bbcd3931861ec3c9da6df0fb817adb53832e"},"cell_type":"markdown","source":"Alright, all the values are integer ones, except some NaNs. It is possible to cast these to integer of one uses\nthe underlying numpy array. "},{"metadata":{"trusted":true,"_uuid":"290aec6e3fb2ad2fc39d715f346337e6debb80db"},"cell_type":"code","source":"df.category_2 = df.category_2.values.astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"91338341f3c635619201432443a722bc22571b17"},"cell_type":"code","source":"pd.__version__","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a500805ff7268a3d4283ee1a85dc2a830f68aec7"},"cell_type":"markdown","source":"There is a new feature in the [0.24](http://pandas-docs.github.io/pandas-docs-travis/whatsnew/v0.24.0.html#optional-integer-na-support) version that allows to do this \"natively\" but we need to wait \nfor Kaggle to update the [Dockerfile](https://github.com/Kaggle/docker-python/blob/master/Dockerfile) ;)"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"8e442c9ce5426cf1055039613633615b6b4a6836"},"cell_type":"code","source":"df.info(memory_usage=\"deep\", verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f79acf34ee69c8a48b58148dcac91735cc7e656"},"cell_type":"markdown","source":"# Bonus: smaller integer types"},{"metadata":{"_uuid":"d267d531fadb43fcaefddf062bb35dc35cd97db9"},"cell_type":"markdown","source":"No need to use the int64 for binary type, the numpy unit8 or the bool_ type are \nmore than enough. So let's do this!"},{"metadata":{"trusted":true,"_uuid":"e2b20c71dba1c8a66881c2b555241fe72c1d81ce"},"cell_type":"code","source":"# You can also use the \"bool\" type (both take one byte for storage).\ndf.authorized_flag = df.authorized_flag.astype(pd.np.uint8)\ndf.category_1 = df.category_1.astype(pd.np.uint8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"0faeb408ea2b872c133f8c11db247cffcdfa308b"},"cell_type":"code","source":"df.info(memory_usage=\"deep\", verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"214170298d0db6ccd8e5b606f912ca2f50182b50"},"cell_type":"markdown","source":"Same thing for the `category_2` column, where the `NaN` value can be stored as 0\nand the column cast as `np.unit8`."},{"metadata":{"trusted":true,"_uuid":"1473c58e973e629cfe0177ce8247fe8d2f1f6d50"},"cell_type":"code","source":"df.category_2 = df.category_2.astype(pd.np.uint8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0c9156fa7a29c5f4f59fe0ae492f87143d715255"},"cell_type":"code","source":"df.category_2.value_counts(normalize=True, dropna=False).plot(kind='bar')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"bed3a54cf10381c5d5343b8ebaf48606ad91d498"},"cell_type":"code","source":"df.info(memory_usage=\"deep\", verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9cac5a79a68c1eb275db55bd1277718209ca8575"},"cell_type":"markdown","source":"We went from **13.1GB to 1GB**. How awesome is that!"},{"metadata":{"_uuid":"92f44add1e3adf3209ad68ed53abb4587c5f46d5"},"cell_type":"markdown","source":"Remark: I guess it is even possible to get a smaller DataFrame by using smaller integer types for some of the categorical columns. I haven't done it, so let me know in the comments. ;)"},{"metadata":{"_uuid":"8025c755a4ae7bdec128912eb7de6ca562dad9a4"},"cell_type":"markdown","source":"# TL;DR: give me the optimization pipeline"},{"metadata":{"_uuid":"6d19b7eaf9a62054f844a99040c58904a37b2873"},"cell_type":"markdown","source":"For those only interested in the output and how to generate it, here is a function that you can add to your notebook/script. "},{"metadata":{"trusted":true,"_uuid":"c380791f5c707fc4936421f879f28283e3b910fc"},"cell_type":"code","source":"# This function could be made generic to almost any loaded CSV file with\n# pandas. Can you see how to do it?\n\n# Some constants\nPARQUET_ENGINE = \"pyarrow\"\nDATE_COL = \"purchase_date\"\nCATEGORICAL_COLS = [\"card_id\", \"category_3\", \"merchant_id\", \"month_lag\", \n                    \"installments\", \"state_id\", \"subsector_id\", \n                    \"city_id\", \"merchant_category_id\", \"merchant_id\"]\nCATEGORICAL_DTYPES = {col: \"category\" for col in CATEGORICAL_COLS}\nPOSITIVE_LABEL = \"Y\"\nINTEGER_WITH_NAN_COL = \"category_2\"\nBINARY_COLS = [\"authorized_flag\", \"category_1\"]\nINPUT_PATH = \"../input/elo-merchant-category-recommendation/historical_transactions.csv\"\nOUTPUT_PATH = \"historical_transactions.parquet\"\n\n\ndef smaller_historical_transactions(input_path, output_path):\n    # Load the CSV file, parse the datetime column and the categorical ones.\n    df = pd.read_csv(input_path, parse_dates=[DATE_COL], \n                    dtype=CATEGORICAL_DTYPES)\n    # Binarize some columns and cast to the boolean type\n    for col in BINARY_COLS:\n        df[col] = pd.np.where(df[col] == POSITIVE_LABEL, 1, 0).astype('bool')\n    # Cast the category_2 to np.uint8\n    df[INTEGER_WITH_NAN_COL] = df[INTEGER_WITH_NAN_COL].values.astype(pd.np.uint8)\n    # Save as parquet file\n    df.to_parquet(output_path, engine=PARQUET_ENGINE)\n    return df\n    \ndef load_historical_transactions(path=None):\n    if path is None:\n        return smaller_historical_transactions(INPUT_PATH, OUTPUT_PATH)\n    else: \n        df = pd.read_parquet(path, engine=PARQUET_ENGINE)\n        # Categorical columns aren't preserved when doing pandas.to_parquet\n        # (or maybe I am missing something?)\n        for col in CATEGORICAL_COLS:\n            df[col] = df[col].astype('cateogry')\n        return df\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"4c1a993f78fdb594eae71a307590a2d348568437"},"cell_type":"code","source":"optimized_df = smaller_historical_transactions(INPUT_PATH, OUTPUT_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"035fa3ecf38666f380a1674618acfc3e9a8687a3"},"cell_type":"code","source":"optimized_df.info(memory_usage=\"deep\", verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c90517e8f9c42466f1d3e953f1062b5a52e77989"},"cell_type":"markdown","source":"Finally, let's time how long it takes to load the dataset from parquet, how much disk space it takes, and how big is its memory footprint. Notice that I need to remove old DataFrames, otherwise the kernel dies."},{"metadata":{"trusted":true,"_uuid":"3e96c0146fdff9e680f60830ec054e80a0c64453"},"cell_type":"code","source":"del df\ndel optimized_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"22b952799a7fd34efa7c011bcd447da4ce80eb92"},"cell_type":"code","source":"# TODO: There is a bug when reading the saved parquet file. Check why and fix it!\n# Is it related to this issue: https://issues.apache.org/jira/browse/ARROW-2369?\n# %%timeit \n# parquet_df = load_historical_transactions(INPUT_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8ca4de44194e507532718d68268ae598c37a9d9b"},"cell_type":"code","source":"# parquet_df.info(memory_usage=\"deep\", verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"d693ef09a35735952dcdd65cfcc008a2485753fb"},"cell_type":"code","source":"ls -lh {OUTPUT_PATH}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"18bc1d213a8f2ae047d97ebb21d0dbc85ed47154"},"cell_type":"markdown","source":"# To go beyond\n\n* I have written a blog post about pandas and there is a section about memory optimization, so check it out [here](https://www.datacamp.com/community/tutorials/pandas-idiomatic).\n* Here is [another](http://pbpython.com/pandas_dtypes.html) blog post about pandas dtypes. \n* Check the pandas [internal archtitecture](https://github.com/pydata/pandas-design/blob/master/source/internal-architecture.rst) document for more details about the `Block` data structure, the `BlockManager`, and their drawbacks. \n* A great [blog post](https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/) to understand how pythons objects are stored. This explains why the pandas `object` has a variable size and can't be accurately estimated without the `memory_usage=\"deep\"`. \n\n"},{"metadata":{"_uuid":"8dba3bae72f29be2d3654c36a4767526293ee4e5"},"cell_type":"markdown","source":"# To wrap up"},{"metadata":{"trusted":true,"_uuid":"363577671b4600c9a91e3f420e3d2291a0294eb0"},"cell_type":"markdown","source":"I hope you have enjoyed reading this memory optimization workflow and have learned something new. Stay tuned for upcoming kernels. ;)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}