{"cells":[{"metadata":{"_uuid":"2b39dc08e2bbb577d1e4d442fc1ab6c7450ad153"},"cell_type":"markdown","source":"In this notebook, I will demonstrate on how to reduce the memory consumption of the Pandas Dataframe which will enable you to carry out the data pre-processing and analysis steps in your less-powerful machine."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"merchants = pd.read_csv('../input/merchants.csv', index_col='merchant_id')\nmerchants.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ec5728e17acd1e3b77f307c703381eceb109ef9"},"cell_type":"markdown","source":"Lets check out the memory usage of the merchants.csv by pandas dataframeb"},{"metadata":{"trusted":true,"_uuid":"c9c84d28ecd5c6fb60aad812cdbe8afd6d46c01a"},"cell_type":"code","source":"print(merchants.info())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d80068a0e85635f1793312cac3d391c966fc2841"},"cell_type":"markdown","source":"Lets make use of the Block Manager to see the data types used across the dataframe"},{"metadata":{"_uuid":"ec2b27b6a847fa4d8b5cf245c39d5993da040752"},"cell_type":"markdown","source":"There are 3 data types class; Integer, Float and Object.\n\nLets check out how much each column takes up the memory usage. \n\nThe float64 type represents each floating point value using 64 bits, or 8 bytes. We can save memory by converting within the same type (from float64 to float32 for example), or by converting between types (from float64 to int32) and so on.."},{"metadata":{"trusted":true,"_uuid":"1153a0c5faaa284d5a5926534de781c07b72dc91"},"cell_type":"code","source":"print(merchants.memory_usage(deep=True))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e2f9b753b62b5d04f89be236ece5fd123314c6b9"},"cell_type":"markdown","source":"We can see that the DataTypes with object takes up the most memory because Pandas uses pointers to access the string values. \n\nLets first try to reduce float columns by downcasting. "},{"metadata":{"trusted":true,"_uuid":"15af95f68f9519d32b3aadc2b48ff1a9d0533d47"},"cell_type":"code","source":"float_cols = merchants.select_dtypes(include=['float'])\n#print(float_cols.dtypes)\n\n\nfor cols in float_cols.columns:\n\tmerchants[cols] = pd.to_numeric(merchants[cols], downcast ='float')\n\nprint(merchants.info())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f95ab60f6d35c56a4048238c54a7eb356bb99ac"},"cell_type":"markdown","source":":Tada: Here we can see the reduce of more than 10MB memory usage by just downcasting the float datatype. It can be significant gain seemingly"},{"metadata":{"trusted":true,"_uuid":"0f870bd6a3b001ade8e5c8f0f7b9dd84559bae8a"},"cell_type":"code","source":"merchants.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff2cff068b8c5d4cfadd0d3daf90a1156d4fedac"},"cell_type":"code","source":"int_cols = merchants.select_dtypes(include=['int'])\n\n\nfor i in int_cols.columns:\n\tmerchants[i] = pd.to_numeric(merchants[i], downcast ='integer')\n\nprint(merchants.info())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b5854e532eb5652d4fcf71145368a3e35dfa96f1"},"cell_type":"code","source":"merchants.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2fb23686d672a3f12fc136971dabd799c8093797"},"cell_type":"markdown","source":"Object takes up the most memory. Below I've identified much more appropriate data types for Object type. Pandas has category type which is much more memory efficient. However keep in mind that you may not be able to apply numeric operations on the category variable."},{"metadata":{"trusted":true,"_uuid":"94e5145a7798945dd0b7e080ae9cf234d9a418ae"},"cell_type":"code","source":"for cols in ['category_1', 'category_4', 'most_recent_purchases_range', 'most_recent_sales_range']:\n\tmerchants[cols] = merchants[cols].astype('category')\n\nprint(merchants.info())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85eb681414b036aad28ba8a8c0f00d97af02201a"},"cell_type":"markdown","source":"As you can see from 56.2+ MB we went down to around 19.5MB memory. This should ease the numeric computation for large MB data size most likely"},{"metadata":{"trusted":true,"_uuid":"5d47a61f9b19858da00937bdcc272cf7c8f5627d"},"cell_type":"code","source":"print(merchants.memory_usage(deep=True))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25a46ccf4b38dad6be62a56505c4a869c08179d8"},"cell_type":"markdown","source":"Much better approach might be to encode the categorical variables as integers. That will further reduce the memory usage by the pandas dataframes of merchant.csv. I hope this notebook has been helpful. Let me know your thoughts on comments. "},{"metadata":{"trusted":true,"_uuid":"d9e3abaaa243105e6038fa2231d2c52845e13879"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}