{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Compressing MNIST Images"},{"metadata":{},"cell_type":"markdown","source":"In this kernel we will compress MNIST images as far as we can without losing information. We will calculate the byte count for each manipulation and compare them with each other.\n\nThis kernel is the 2nd in a series I am doing on MNIST. The first is a kernel on the power of dimensionality reduction combined with simple algorithms such as decision trees. [You can check out the 1st kernel in this series here.](https://www.kaggle.com/carlolepelaars/97-on-mnist-with-a-single-decision-tree-t-sne)\n\nMNIST series:\n\n1. [97% on MNIST with a single decision tree (+ t-SNE)](97% on MNIST with a single decision tree (+ t-SNE))\n2. [Compressing MNIST Images (to 784 bits)](https://www.kaggle.com/carlolepelaars/compressing-mnist-images-to-784-bits)\n3. An end-to-end neural network using only NumPy (Work in progress)"},{"metadata":{},"cell_type":"markdown","source":"## Table of contents"},{"metadata":{},"cell_type":"markdown","source":"- [Dependencies](#1)\n- [Preparation](#2)\n- [Baseline](#3)\n- [Uint8](#4)\n- [TSVD](#5)\n- [Bitstring and ASCII Encoding](#6)\n- [Conclusion](#7)"},{"metadata":{},"cell_type":"markdown","source":"## Dependencies <a id=\"1\"></a>"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Standard libraries\nimport os\nimport sys\nimport pickle\nimport numpy as np\nimport random as rn\nimport pandas as pd\nfrom tqdm import tqdm # Progress Bar\n\n# Dimensionality reduction\nfrom sklearn.decomposition import TruncatedSVD\n\n# Specify Paths\nBASE_PATH = '../input/digit-recognizer/'\nTRAIN_PATH = BASE_PATH + 'train.csv'\nTEST_PATH = BASE_PATH + 'test.csv'\n\n# Seed for reproducability\nseed = 1234\nnp.random.seed(seed)\nrn.seed(seed)\nos.environ['PYTHONHASHSEED'] = str(seed)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# File sizes and specifications\nprint('\\n# Files and file sizes')\nfor file in os.listdir(BASE_PATH):\n    print('{}| {} MB'.format(file.ljust(30), \n                             str(round(os.path.getsize(BASE_PATH + file) / 1000000, 2))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparation <a id=\"2\"></a>"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Load in training and testing data\ntrain_df = pd.read_csv(TRAIN_PATH)\ntest_df = pd.read_csv(TEST_PATH)\nsample_sub = pd.read_csv(BASE_PATH + 'sample_submission.csv')\n\nfeatures = [col for col in train_df.columns if col.startswith('pixel')]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Baseline <a id=\"3\"></a>"},{"metadata":{},"cell_type":"markdown","source":"We will try to reduce the byte count as much as we can before training a model on MNIST. Below are the baselines from where we will be working. Pandas stores every integer automatically as int64 so the initial byte count is quite large."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# The byte count for our baseline (Standard Pandas Dataframe with Int64 types)\nbase_byte_count_train = sys.getsizeof(train_df)\nbase_byte_count_test = sys.getsizeof(test_df[features])\nprint(f'The standard byte count for train_df is: {base_byte_count_train}')\nprint(f'The standard byte count for test_df is: {base_byte_count_test}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"Distribution description for select pixel values:\")\ntrain_df[['pixel100', 'pixel200', 'pixel300', 'pixel500', 'pixel600', 'pixel700']].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Uint8 <a id=\"4\"></a>"},{"metadata":{},"cell_type":"markdown","source":"Since all values in this DataFrame are between 0 and 255 we can store the data as unsigned integers of 8 bytes (uint8). This will save 56 bytes on every value."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"Training Dataframe:\")\ntrain_df.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert all values to unsigned 8-bit integers\ntrain_df = train_df.astype(np.uint8)\ntest_df = test_df.astype(np.uint8)\nlabels = list(train_df['label'])\nlabel_byte_count = sys.getsizeof(labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"uint8_byte_count_train = sys.getsizeof(train_df)\nuint8_byte_count_test = sys.getsizeof(test_df[features])\n\nprint(f'Once we convert all integers to uint8 we are left with a byte count of {uint8_byte_count_train} for train_df and\\n\\\n{uint8_byte_count_test} bytes for test_df.\\n\\n\\\nThis is a reduction of (compared to a basic Pandas DataFrame):\\n\\\n{round(((uint8_byte_count_train - base_byte_count_train)/base_byte_count_train)*100,2)}% for the train_df and\\n\\\n{round(((uint8_byte_count_test - base_byte_count_test)/base_byte_count_test)*100,2)}% for the test_df.')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'The cost of storing the index in a Pandas DataFrame is: {test_df.index.memory_usage()} bytes')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TSVD <a id=\"5\"></a>"},{"metadata":{},"cell_type":"markdown","source":"TSVD is a technique that can be used to compress sparse data, like MNIST Images. We can decide ourselves how much we want to reduce the dimensionality. In this example we reduce the 28x28 images to 15x15 (15 components)\n\nFor more information about dimensionality reduction on MNIST I suggest you check out [this Kaggle kernel](https://www.kaggle.com/carlolepelaars/97-on-mnist-with-a-single-decision-tree-t-sne)."},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"concat_uint8 = pd.concat([train_df, test_df])\nn_components = 15\n# Perform Truncated Singular Value Decomposition (TSVD) on all features\ntsvd = TruncatedSVD(n_components=n_components).fit_transform(concat_uint8[features])\n# Split up the t-SNE results in training and testing data\ncomponents = [f\"Component_{_}\" for _ in range(n_components)]\ntsvd_train = pd.DataFrame(tsvd[:len(train_df)], columns=components)\ntsvd_test = pd.DataFrame(tsvd[len(train_df):], columns=components)\ntsvd_train['label'] = labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('TSVD features: ')\ntsvd_train.head(3)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'Once we perform TSVD we are left with a byte count of\\n\\\n{sys.getsizeof(tsvd_train)} for train_df and\\n\\\n{sys.getsizeof(tsvd_test)} bytes for test_df.\\n\\n\\\nThis is a reduction of (compared to Pandas DataFrame with uint8):\\n\\\n{round(((sys.getsizeof(tsvd_train) - uint8_byte_count_train)/uint8_byte_count_train)*100,2)}% for the train_df and\\n\\\n{round(((sys.getsizeof(tsvd_test) - uint8_byte_count_test)/uint8_byte_count_test)*100,2)}% for the test_df.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Bitstring and ASCII encoding <a id=\"6\"></a>"},{"metadata":{},"cell_type":"markdown","source":"Since different grayscale values don't play a very large role in MNIST we can round all numbers smaller than 127 to 0 and clip all values larger or equal to 127 to 1. In this way we can reduce every image to an array of 784 bytes (0s and 1s). We will also need 42128 bytes more for the training set because we have to store the labels as uint8. From here on out we all lose the Pandas DataFrame format. We save another 128 bytes because we don't have to store the index anymore.\n\nTo reduce the byte count we store every image as a bitstring and then encode it to ASCII."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert all values to 0 and 1\nfor df in [train_df, test_df]:\n    for col in features:\n        df[col] = np.where(df[col]<127, 0, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"train_array = []\ntest_array = []\n# Encode every image to a ascii bitstring\nfor i in tqdm(range(len(train_df))):\n    train_array.append(train_df.iloc[i][features].astype(str).sum().encode('ascii'))\nfor i in tqdm(range(len(test_df))):\n    test_array.append(test_df.iloc[i][features].astype(str).sum().encode('ascii'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"bitcount_train = sys.getsizeof(train_array)\nbitcount_test = sys.getsizeof(test_array)\nprint(f'Once we convert all integers to bits we are left with a byte count of \\n\\\n{bitcount_train} for train_df and\\n\\\n{bitcount_test} bytes for test_df.\\n\\n\\\nThis is a further reduction (compared to Pandas DataFrame with uint8) of:\\n\\\n{round((((bitcount_train+label_byte_count) - uint8_byte_count_train)/uint8_byte_count_train)*100,2)}% for the train_df and\\n\\\n{round(((bitcount_test - uint8_byte_count_test)/uint8_byte_count_test)*100,2)}% for the test_df.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wow! Now we're talking. Note that we can still visualize the digit even though it is stored in an unintuitive way. Here is how we can easily print the representation as a 28x28 representation."},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_digit_bytes(text, lineLength):\n    \"\"\"\n    Visualize a 784-bit representation of an MNIST Image\n    \n    text: The bitstring that you want to represent\n    lineLength: The length at which to jump to a new line \n    (28 in the case of MNIST images)\n    \"\"\"\n    if len(text) <= lineLength:\n        return text\n    else:\n        # Print line and next line recursively\n        print(text[:lineLength])\n        return display_digit_bytes(text[lineLength:], lineLength)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'Label = {labels[3]}\\n')\ndisplay_digit_bytes(train_array[3], 28)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We store we smallest data format we have as a [pickle file](https://docs.python.org/3/library/pickle.html)."},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# Add labels to training data\ntrain_array.append(labels)\n# Save data as pickle files\nwith open('training_data.pkl','wb') as file:\n    pickle.dump(train_array, file)\nwith open('test_data.pkl','wb') as file:\n    pickle.dump(test_array, file)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\"File size for pickled training data: {os.path.getsize('training_data.pkl')}\")\nprint(f\"File size for pickled test data: {os.path.getsize('training_data.pkl')}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusion <a id=\"7\"></a>"},{"metadata":{},"cell_type":"markdown","source":"TL;DR: Use NumPy arrays with uint8. Consider dimensionality reduction to compress data."},{"metadata":{},"cell_type":"markdown","source":"As you can see there are many ways to compress grayscale images without losing much information. However, a bitstring is not really suitable for training machine learning models. NumPy arrays are very practical and therefore I suggest compressing the images so they can be used with the uint8 type (values between 0 and 255). It is also beneficial to consider dimensionality reduction techniques if you need to compress the data even further. [Dimensionality reduction techniques can even increase performance of simple models as illustrated in this Kaggle kernel](https://www.kaggle.com/carlolepelaars/97-on-mnist-with-a-single-decision-tree-t-sne)."},{"metadata":{},"cell_type":"markdown","source":"And we are done! Please let me know if you know more tricks to compress (MNIST) images or to store image data more efficiently.\n\nIf you like this Kaggle kernel, feel free to give an upvote and leave a comment! I will try to implement your suggestions in this kernel!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}