{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reading input data \nIn the first step, we need to read our data from the input the directory (`/kaggle/input/Kannada-MNIST/`).  \nWe are interested in two files:\n* `train.csv` containing samples for training model,\n* `test.csv` containing records that need to be classified in this competition."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.read_csv(os.path.join('/kaggle/input/Kannada-MNIST/test.csv'))\ndf_test.sample(n=5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(os.path.join('/kaggle/input/Kannada-MNIST/train.csv'))\ndf_train.sample(n=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that each row in `train.csv` file consists of a `label` and 784 (28x28) pixels. Pixel values are our training data (called `X`) and labels are our target (classification result, called `y`). In `test.csv` on the other hand, `label` column is missing (because that's what we need to find) but we've got `id` column which is just row index - not really useful so we will get rid of that. Let's now split dataframes into X and y arrays for both training and test sets."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = np.array(df_train.loc[:, df_train.columns != 'label'])\nX_test  = np.array(df_test.loc[:, df_test.columns != 'id'])\n\ny_train = np.array(df_train['label'])\n\nprint(f\"X_train: {X_train.shape}\\nX_test: {X_test.shape}\\ny_train: {y_train.shape}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualization\n\n#### Sample images\n\nWe start with displaying sample images for each of ten labels which is quite simple task but it's good to know what data we are dealing with."},{"metadata":{"trusted":true},"cell_type":"code","source":"rows = 5\ncols = 10\nfig, ax = plt.subplots(nrows=rows, ncols=cols, figsize=(cols, rows))\n\nfor label in range(cols):\n    digits = df_train.loc[df_train['label'] == label]\n    digits = digits.drop('label', axis=1)\n    ax[0][label].set_title(label)\n    for j in range(rows):\n        ax[j][label].axis('off')\n        ax[j][label].imshow(digits.iloc[j, :].to_numpy().astype(np.uint8).reshape(28, 28), cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Distribution\n\nLet's display number of samples representing each label (0-9). You can clearly see that in the training set, every label has 6000 samples. It means we don't have to do anything specific to reduce data imbalance which is a good sign."},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(y_train, kde=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Pixel heatmap\n\nThe next (actually interesting) thing is a pixel heatmap. It shows which pixels are usually used in our data images i.e. aren't black pixels. If you take a look at the image below, you can see that approx. 2-3 pixels from each side of images (and a bit more in the top-left corner) are never used in the training set."},{"metadata":{"trusted":true},"cell_type":"code","source":"pixel_counts = (df_train.loc[:, df_train.columns != 'label'] / 255).astype(int)\npixel_counts = pixel_counts.sum(axis=0).values\npixel_counts = pixel_counts.reshape((28, 28))\nsns.heatmap(pixel_counts)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also plot heatmaps for each label separately, as below."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=2, ncols=5, figsize=(20, 5))\nax = ax.flatten()\n\nfor label in range(10):\n    pixel_counts = (df_train.loc[:, df_train.columns != 'label'] / 255).astype(int)\n    pixel_counts = pixel_counts.loc[df_train['label'] == label]\n    pixel_counts = pixel_counts.sum(axis=0).values\n    pixel_counts = pixel_counts.reshape((28, 28))\n    ax[label].axis('off')\n    sns.heatmap(pixel_counts, ax=ax[label])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Further steps\n\nWhen you already know what kind of data you are dealing with, you can start creating and training your model. You can refer to many existing tutorials and kernels regarding MNIST dataset which is also available on Kaggle as [Digit Recognizer](https://www.kaggle.com/c/digit-recognizer) competition. There you can find exhausting list of visualizations, models and general advices."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}