{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Cats vs Dogs Classification with Visual BOW + KNN","metadata":{}},{"cell_type":"code","source":"import cv2\nimport pandas as pd\n\n# Image Directory\nimport os\nimport pickle\n\n# Math\nimport numpy as np\n\n# Visualization\nimport matplotlib.pyplot as plt\n\n# Models and Preprocessing\nfrom scipy.cluster.vq import *\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.preprocessing import StandardScaler\nfrom scipy.spatial.distance import euclidean","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-01-26T07:34:46.262301Z","iopub.execute_input":"2023-01-26T07:34:46.263484Z","iopub.status.idle":"2023-01-26T07:34:47.630845Z","shell.execute_reply.started":"2023-01-26T07:34:46.263374Z","shell.execute_reply":"2023-01-26T07:34:47.629733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Get images","metadata":{}},{"cell_type":"markdown","source":"First, we unzip the dataset files.","metadata":{}},{"cell_type":"code","source":"!unzip -q \"/kaggle/input/dogs-vs-cats/test1.zip\"\n!unzip -q \"/kaggle/input/dogs-vs-cats/train.zip\"","metadata":{"execution":{"iopub.status.busy":"2023-01-26T07:34:47.634208Z","iopub.execute_input":"2023-01-26T07:34:47.634703Z","iopub.status.idle":"2023-01-26T07:35:04.343268Z","shell.execute_reply.started":"2023-01-26T07:34:47.634654Z","shell.execute_reply":"2023-01-26T07:35:04.341832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, this is a function to read the images in the train folder and test folder and compile them each of them into two dataframes, a train dataframe, and a test dataframe.\n\nAn important thing to note is that we only used 200 images in total from the train folder (100 cats and 100 dogs) and 20 images in total from the test folder, because the original dataset consists of 25k training images and 10k test images, and that is too large. We didn't have enough RAM to compute SIFT descriptors for all 35k images. So, we selected a sample of 200 training images and 20 test images.","metadata":{}},{"cell_type":"code","source":"def get_dataframe(path, mode='train', n_cats=100, n_dogs=100, n_test=20):\n    '''\n        Function to get images from directory and respective labels\n        Since the dataset is very large, we will only use 100 training images\n        (50 cats and 50 dogs) and 10 test images\n    '''\n    images = []\n    labels = []\n    ctr_cats = 0\n    ctr_dogs = 0\n    \n    for filename in os.listdir(path):\n        image = cv2.imread(path + '/' + filename)\n        \n#         dog = 1, cat = 0\n        if mode == 'train':\n            if 'cat' in filename and ctr_cats < n_cats:\n                images.append(image)\n                labels.append(0)\n                ctr_cats += 1\n            elif 'dog' in filename and ctr_dogs < n_dogs:\n                images.append(image)\n                labels.append(1)\n                ctr_dogs += 1\n                \n            if ctr_cats == n_cats and ctr_dogs == n_dogs: break\n        else:\n            images.append(image)\n            if len(images) == n_test: break\n    \n    if mode == 'train':\n        return pd.DataFrame({'image': images, 'label': labels})\n    else:\n        return pd.DataFrame({'image': images})","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:55:47.100777Z","iopub.execute_input":"2023-01-26T09:55:47.101195Z","iopub.status.idle":"2023-01-26T09:55:47.112526Z","shell.execute_reply.started":"2023-01-26T09:55:47.101160Z","shell.execute_reply":"2023-01-26T09:55:47.111030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_DIR = '/kaggle/working/train'\nTEST_DIR = '/kaggle/working/test1'\n\ntrain_df = get_dataframe(TRAIN_DIR)\ntest_df = get_dataframe(TEST_DIR, mode='test')","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:55:47.115328Z","iopub.execute_input":"2023-01-26T09:55:47.115834Z","iopub.status.idle":"2023-01-26T09:55:47.740506Z","shell.execute_reply.started":"2023-01-26T09:55:47.115788Z","shell.execute_reply":"2023-01-26T09:55:47.739159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:55:47.743040Z","iopub.execute_input":"2023-01-26T09:55:47.743516Z","iopub.status.idle":"2023-01-26T09:56:02.184037Z","shell.execute_reply.started":"2023-01-26T09:55:47.743467Z","shell.execute_reply":"2023-01-26T09:56:02.182729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:56:02.187202Z","iopub.execute_input":"2023-01-26T09:56:02.187964Z","iopub.status.idle":"2023-01-26T09:56:02.201462Z","shell.execute_reply.started":"2023-01-26T09:56:02.187927Z","shell.execute_reply":"2023-01-26T09:56:02.200613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize sample images","metadata":{}},{"cell_type":"markdown","source":"Here, we visualized random images in the training dataset.","metadata":{}},{"cell_type":"code","source":"def plot_image(image, label):\n    plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))\n    plt.axis('off')\n    label = 'Cat' if label == 0 else 'Dog'\n    plt.title(label)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:56:02.203192Z","iopub.execute_input":"2023-01-26T09:56:02.203594Z","iopub.status.idle":"2023-01-26T09:56:02.211395Z","shell.execute_reply.started":"2023-01-26T09:56:02.203541Z","shell.execute_reply":"2023-01-26T09:56:02.210366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rand = train_df.sample(1).to_numpy()[0]\n\nplot_image(rand[0], rand[1])","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:56:02.212779Z","iopub.execute_input":"2023-01-26T09:56:02.213091Z","iopub.status.idle":"2023-01-26T09:56:02.421047Z","shell.execute_reply.started":"2023-01-26T09:56:02.213063Z","shell.execute_reply":"2023-01-26T09:56:02.419729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Extract Features","metadata":{}},{"cell_type":"markdown","source":"For this experiment, we will construct a visual bag-of-words and feed it into a K-NN classifier. The first step to building visual vocabulary is to use SIFT to detect the important descriptors or features in each image.","metadata":{}},{"cell_type":"markdown","source":"This function is used to format the descriptors so that we can use in later steps (explained later). What this function does is, it combines all of the descriptors from every image in the train dataframe into one whole list. ","metadata":{}},{"cell_type":"code","source":"def format_descriptors(descriptor_list):\n    stacked_descriptors = descriptor_list[0]\n    for i in descriptor_list[1:]:\n        stacked_descriptors = np.vstack((stacked_descriptors, i))\n\n    stacked_descriptors = np.float32(stacked_descriptors)\n    return stacked_descriptors","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:56:02.422598Z","iopub.execute_input":"2023-01-26T09:56:02.423768Z","iopub.status.idle":"2023-01-26T09:56:02.431130Z","shell.execute_reply.started":"2023-01-26T09:56:02.423723Z","shell.execute_reply":"2023-01-26T09:56:02.429855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is a function we use to get descriptors from each image in the train and test dataframes. We used the SIFT algorithm to get the descriptors.","metadata":{}},{"cell_type":"code","source":"def get_descriptors(dataset, sift):\n    descriptor_list = []\n    ctr = 0\n\n    for image in dataset:\n        keypoints, descriptors = sift.detectAndCompute(image, None)\n        descriptor_list.append(descriptors)\n        if ctr < 3:\n            img_kp = cv2.drawKeypoints(image, keypoints, cv2.DRAW_MATCHES_FLAGS_DEFAULT, color=(255, 0, 0))\n            plt.imshow(img_kp)\n            plt.axis('off')\n            plt.show()\n            ctr += 1\n        \n#     returns original descriptors list and formatted for kmeans\n    return descriptor_list, format_descriptors(descriptor_list)","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:56:02.432993Z","iopub.execute_input":"2023-01-26T09:56:02.434137Z","iopub.status.idle":"2023-01-26T09:56:02.444854Z","shell.execute_reply.started":"2023-01-26T09:56:02.434089Z","shell.execute_reply":"2023-01-26T09:56:02.443833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we also visualized the keypoints that the SIFT algorithm detected, to get an idea of what SIFT is focusing on in each image.","metadata":{}},{"cell_type":"code","source":"SIFT = cv2.SIFT_create()\n\ntrain_descriptors, stacked_train_descriptors = get_descriptors(train_df['image'], SIFT)\ntest_descriptors, _ = get_descriptors(test_df['image'], SIFT)","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:56:02.449574Z","iopub.execute_input":"2023-01-26T09:56:02.449924Z","iopub.status.idle":"2023-01-26T09:56:15.985298Z","shell.execute_reply.started":"2023-01-26T09:56:02.449895Z","shell.execute_reply":"2023-01-26T09:56:15.984206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(stacked_train_descriptors)","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:56:15.986850Z","iopub.execute_input":"2023-01-26T09:56:15.987189Z","iopub.status.idle":"2023-01-26T09:56:15.993247Z","shell.execute_reply.started":"2023-01-26T09:56:15.987159Z","shell.execute_reply":"2023-01-26T09:56:15.991880Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Cluster descriptors with KMeans","metadata":{}},{"cell_type":"markdown","source":"Next, to build a visual vocabulary, we are going to quantize the feature space. Basically, we have a lot of descriptors/features in our hands, and we have to reduce them by clustering the similar ones together, using the K-Means algorithm.\n\n![](https://miro.medium.com/max/828/1*ge_bVqseHxmmb57xXIOJag.webp)\n\nHere, we made 10 clusters over 50 iterations.","metadata":{}},{"cell_type":"code","source":"centroids, _ = kmeans(stacked_train_descriptors, 10, 50)","metadata":{"execution":{"iopub.status.busy":"2023-01-26T09:56:16.010283Z","iopub.execute_input":"2023-01-26T09:56:16.011257Z","iopub.status.idle":"2023-01-26T10:01:33.361390Z","shell.execute_reply.started":"2023-01-26T09:56:16.011210Z","shell.execute_reply":"2023-01-26T10:01:33.359613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create vector for each image visual words","metadata":{}},{"cell_type":"markdown","source":"Next, we make a histogram of each image to count how many of each image's descriptors fall into cluster 1, 2, 3, etc. To see which centroid each descriptor is closest to, we use the function `scipy.cluster.vq.vq()`.\n\nThis function from scipy is used to perform vector quantization, to see which centroid is closest to a descriptor of an image. If a certain descriptor of an image is close to the centroid in cluster 1, then we will assign the label 1 to the descriptor. We keep doing this for all the descriptors in an image until we have successfully categorized all the descriptors into their respective clusters. We are left with a histogram of how many image's descriptors belonging in each cluster.\n\nThen, we repeat this process for every image in the training dataset. We get a vector V which holds every histogram for every image.","metadata":{}},{"cell_type":"code","source":"V = np.zeros((len(train_df), len(centroids)), '\n             float32')\n\nfor img_idx in range(len(train_descriptors)):\n    for desc in train_descriptors[img_idx]:\n        words, _ = vq([desc], centroids)\n        for w in words:\n            V[img_idx][w-1] += 1\n\nprint(V)","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:01:33.364252Z","iopub.execute_input":"2023-01-26T10:01:33.365459Z","iopub.status.idle":"2023-01-26T10:01:40.366733Z","shell.execute_reply.started":"2023-01-26T10:01:33.365389Z","shell.execute_reply":"2023-01-26T10:01:40.365600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To read vector V, we can see that V[0] has 251 descriptors belonging to cluster 1, 241 descriptors belonging to cluster 2, and etc.","metadata":{}},{"cell_type":"markdown","source":"Next, we have to normalize each image's histogram.","metadata":{}},{"cell_type":"code","source":"norm_V = []\nfor v in V:\n    norm_V.append(v / v.sum())\n\nprint(norm_V)","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:01:40.368314Z","iopub.execute_input":"2023-01-26T10:01:40.368700Z","iopub.status.idle":"2023-01-26T10:01:40.417388Z","shell.execute_reply.started":"2023-01-26T10:01:40.368659Z","shell.execute_reply":"2023-01-26T10:01:40.416074Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Do above steps for test images","metadata":{}},{"cell_type":"code","source":"# Create vector V\ntest_V = np.zeros((len(test_df), len(centroids)), 'float32')\n\nfor img_idx in range(len(test_descriptors)):\n    for desc in test_descriptors[img_idx]:\n        words, _ = vq([desc], centroids)\n        for w in words:\n            test_V[img_idx][w-1] += 1\n\n# Normalize V\nnorm_test_V = test_V / test_V.sum()\nprint(norm_test_V)","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:01:40.418914Z","iopub.execute_input":"2023-01-26T10:01:40.419258Z","iopub.status.idle":"2023-01-26T10:01:41.279338Z","shell.execute_reply.started":"2023-01-26T10:01:40.419226Z","shell.execute_reply":"2023-01-26T10:01:41.277940Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we have a vector `test_V` that holds the normalized histogram of each test image. There are 20 images in total. The histogram depicts the number of visual words (centroids) each image has, calculated from each image's descriptors.","metadata":{}},{"cell_type":"markdown","source":"# Train KNN classifier","metadata":{}},{"cell_type":"markdown","source":"Next, we fit the images to a K-NN model from sklearn.","metadata":{}},{"cell_type":"code","source":"knn = KNeighborsClassifier(n_neighbors=11)\nknn.fit(norm_V, train_df['label'])","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:04:18.683136Z","iopub.execute_input":"2023-01-26T10:04:18.683602Z","iopub.status.idle":"2023-01-26T10:04:18.692350Z","shell.execute_reply.started":"2023-01-26T10:04:18.683553Z","shell.execute_reply":"2023-01-26T10:04:18.691248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Its important to use binary mode \nknnPickle = open('knn.pkl', 'wb') \n      \n# source, destination \npickle.dump(knn, knnPickle)  \n\n# close the file\nknnPickle.close()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:04:18.694468Z","iopub.execute_input":"2023-01-26T10:04:18.694805Z","iopub.status.idle":"2023-01-26T10:04:18.702324Z","shell.execute_reply.started":"2023-01-26T10:04:18.694777Z","shell.execute_reply":"2023-01-26T10:04:18.701383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = knn.predict(norm_test_V)\npred","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:04:18.703753Z","iopub.execute_input":"2023-01-26T10:04:18.704055Z","iopub.status.idle":"2023-01-26T10:04:18.717080Z","shell.execute_reply.started":"2023-01-26T10:04:18.704029Z","shell.execute_reply":"2023-01-26T10:04:18.715741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we can see that the model did not do very well. It predicted all the 20 test images as cats. This may be because we didn't use the whole dataset in training (there should be 25k images but we didn't have enough RAM). \n\nAside from that, some of the training images' quality are not that good. There are some pictures that show the only the side profile or body of a dog or a cat, not the face (can be seen from visualization in previous sections). There are other pictures which also doesn't properly zoom in on the face of the dog or cat, but only shows the dog or cat from afar. This has resulted in a lot of noise in the dataset, and it may be the reason why, in the previous section, we see that the SIFT algorithm detects many background pixels as features, instead of the pixels that are on the cat's or dog's body. This problem then probably affected the K-Means algorithm too, as it means that there may be a lot of noise in the clustering process.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16, 10))\nfor i, test in enumerate(test_df['image']):\n    plt.subplot(4, 5, i+1)\n    plt.imshow(cv2.cvtColor(test, cv2.COLOR_BGR2RGB))\n    plt.axis('off')\n    \n    title = 'Cat' if pred[i] == 0 else 'Dog'\n    plt.title('Prediction: ' + title)\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-26T10:04:18.718764Z","iopub.execute_input":"2023-01-26T10:04:18.719261Z","iopub.status.idle":"2023-01-26T10:04:20.408294Z","shell.execute_reply.started":"2023-01-26T10:04:18.719228Z","shell.execute_reply":"2023-01-26T10:04:20.407383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To solve this cats vs dogs recognition problem, it might be best to use other complex methods, more neural-network based methods than feature engineering based methods like this one (Visual BOW).","metadata":{}}]}