{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\nThe goal of this notebook is to show you the \"whole game\" of machine learning. That is, instead of learning concepts 'bottom-up', we are going to learn 'top-down'. Consider the analogy of a basketball game. You learn the game by playing a few pickup games- you learn the rules, how to dribble etc. You may suck at most of the aspects, but you get an idea of what skills are needed. Then, you sharpen your skills in each area and get better over time. \n\nContrast this with learning calculus. You may learn that the derivative of $y = x^2$ is $2x$, but you have no idea what's the point of learning that. \n\nMy hope is that this notebook helps you to see that:\n1. Deep learning is not intimidating, and many terms are made so that deep learning practitioners sound smart\n2. You can be competent in DL from day one.\n\nI will be using `fastai`'s library, so many operations will be abstracted away. No worries, you'll learn more about them over the course of this module.","metadata":{}},{"cell_type":"code","source":"import torch\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:19:27.158716Z","iopub.execute_input":"2023-09-29T09:19:27.159374Z","iopub.status.idle":"2023-09-29T09:19:27.163759Z","shell.execute_reply.started":"2023-09-29T09:19:27.159344Z","shell.execute_reply":"2023-09-29T09:19:27.162857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading data","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-09-29T09:19:29.228560Z","iopub.execute_input":"2023-09-29T09:19:29.229227Z","iopub.status.idle":"2023-09-29T09:19:29.237467Z","shell.execute_reply.started":"2023-09-29T09:19:29.229196Z","shell.execute_reply":"2023-09-29T09:19:29.236300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path = '/kaggle/input/Kannada-MNIST/train.csv'\ndf = pd.read_csv(path)\ndf.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:19:30.590259Z","iopub.execute_input":"2023-09-29T09:19:30.590941Z","iopub.status.idle":"2023-09-29T09:19:33.598908Z","shell.execute_reply.started":"2023-09-29T09:19:30.590889Z","shell.execute_reply":"2023-09-29T09:19:33.597988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I always do the above to see whether:\n1. I have loaded `train_df` properly and\n2. The dimensions of the dataset. In this case, there are 60k images\n\nLet's split data into `train` and `val` set.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain_df, val_df = train_test_split(df, test_size=0.2)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:19:37.985129Z","iopub.execute_input":"2023-09-29T09:19:37.985805Z","iopub.status.idle":"2023-09-29T09:19:38.491668Z","shell.execute_reply.started":"2023-09-29T09:19:37.985773Z","shell.execute_reply":"2023-09-29T09:19:38.490693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_tensor = torch.tensor(train_df.values)\nval_tensor = torch.tensor(val_df.values)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:19:48.754237Z","iopub.execute_input":"2023-09-29T09:19:48.754582Z","iopub.status.idle":"2023-09-29T09:19:49.041319Z","shell.execute_reply.started":"2023-09-29T09:19:48.754555Z","shell.execute_reply":"2023-09-29T09:19:49.040344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_tensor.shape, val_tensor.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:19:52.677402Z","iopub.execute_input":"2023-09-29T09:19:52.677732Z","iopub.status.idle":"2023-09-29T09:19:52.683298Z","shell.execute_reply.started":"2023-09-29T09:19:52.677706Z","shell.execute_reply":"2023-09-29T09:19:52.682443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_x, train_y = train_tensor[:, 1:], train_tensor[:, 0]\nval_x, val_y = val_tensor[:, 1:], val_tensor[:, 0]","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:20:19.872669Z","iopub.execute_input":"2023-09-29T09:20:19.873035Z","iopub.status.idle":"2023-09-29T09:20:19.877595Z","shell.execute_reply.started":"2023-09-29T09:20:19.873008Z","shell.execute_reply":"2023-09-29T09:20:19.876671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(train_x[0].reshape(28,28))","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:20:31.202366Z","iopub.execute_input":"2023-09-29T09:20:31.202696Z","iopub.status.idle":"2023-09-29T09:20:31.430603Z","shell.execute_reply.started":"2023-09-29T09:20:31.202668Z","shell.execute_reply":"2023-09-29T09:20:31.429686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_y[0]","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:20:37.567213Z","iopub.execute_input":"2023-09-29T09:20:37.568297Z","iopub.status.idle":"2023-09-29T09:20:37.575299Z","shell.execute_reply.started":"2023-09-29T09:20:37.568255Z","shell.execute_reply":"2023-09-29T09:20:37.574445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great, we can now visualise an image, along with its corresponding `y` value.\n\nNow, let's quickly get our validation set sorted out as well.\n","metadata":{}},{"cell_type":"markdown","source":"We just need to reshape our `train_x` and `val_x` so that they are in the format of 28 by 28, which represents the image.","metadata":{}},{"cell_type":"code","source":"train_x, val_x = train_x.reshape(-1, 28, 28), val_x.reshape(-1, 28, 28)\ntrain_x.shape, val_x.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:21:00.960692Z","iopub.execute_input":"2023-09-29T09:21:00.961780Z","iopub.status.idle":"2023-09-29T09:21:00.969151Z","shell.execute_reply.started":"2023-09-29T09:21:00.961738Z","shell.execute_reply":"2023-09-29T09:21:00.968214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We need to do some slight modifications- changing the channel of the image from 1 to 3 because our pretrained model later on takes in 3 channels.","metadata":{}},{"cell_type":"code","source":"# Add a channel dimension at index 1\ntrain_x, val_x = train_x.unsqueeze(1), val_x.unsqueeze(1)\n\n# # Expand the single channel to three channels\n# train_x = train_x.expand(-1, 3, -1, -1)\n# val_x = val_x.expand(-1, 3, -1, -1)\n\ntrain_x, val_x = train_x.float(), val_x.float()\n\nif torch.cuda.is_available():\n    train_x, train_y = train_x.to(\"cuda\"), train_y.to(\"cuda\")\n    val_x, val_y = val_x.to(\"cuda\"), val_y.to(\"cuda\")\n\ntrain_x.shape, val_x.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:21:17.910157Z","iopub.execute_input":"2023-09-29T09:21:17.911250Z","iopub.status.idle":"2023-09-29T09:21:18.132248Z","shell.execute_reply.started":"2023-09-29T09:21:17.911212Z","shell.execute_reply":"2023-09-29T09:21:18.131265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we need to transform our `train_x`, `train_y`, `val_x`, `val_y` into PyTorch DataLoaders so that they can be loaded efficiently for our model to finetune on. You can skip the details of implementation here, just know that it's meant to speed things up by processing them in parallel.","metadata":{}},{"cell_type":"code","source":"from torch.utils.data import TensorDataset\n\ntrain_dataset = TensorDataset(train_x, train_y)\nval_dataset = TensorDataset(val_x, val_y)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:21:23.052681Z","iopub.execute_input":"2023-09-29T09:21:23.053447Z","iopub.status.idle":"2023-09-29T09:21:23.058426Z","shell.execute_reply.started":"2023-09-29T09:21:23.053409Z","shell.execute_reply":"2023-09-29T09:21:23.057423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torch.utils.data import DataLoader\n\ntrain_dataloader = DataLoader(train_dataset, shuffle=True, batch_size=256)\neval_dataloader = DataLoader(val_dataset, batch_size=256)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:21:23.879298Z","iopub.execute_input":"2023-09-29T09:21:23.880359Z","iopub.status.idle":"2023-09-29T09:21:23.885655Z","shell.execute_reply.started":"2023-09-29T09:21:23.880318Z","shell.execute_reply":"2023-09-29T09:21:23.884667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Finetuning\nNext, we are going to finetune a model to classify these digits. Finetuning is using a trained model, then slightly training it to fit our specific task. In our case, we are going to take a model that was trained to classify images, then finetune it to classify these digits.\n\nWhy finetune? Finetunining is much faster and cheaper than training a model from scratch. :)\n\n[HuggingFace](https://huggingface.co/models) provides many trained models to be finetuned. I'm also just following [this guide to finetune HuggingFace models](https://huggingface.co/docs/transformers/training#train-in-native-pytorch).","metadata":{}},{"cell_type":"code","source":"from transformers import AutoModelForImageClassification\n\nmodel = AutoModelForImageClassification.from_pretrained(\"fxmarty/resnet-tiny-mnist\", num_labels=10)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:21:26.413595Z","iopub.execute_input":"2023-09-29T09:21:26.413949Z","iopub.status.idle":"2023-09-29T09:21:26.603301Z","shell.execute_reply.started":"2023-09-29T09:21:26.413899Z","shell.execute_reply":"2023-09-29T09:21:26.602351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use stochastic gradient descent\nfrom torch.optim import SGD\n\noptimizer = SGD(model.parameters(), lr=0.001)\ncriterion = torch.nn.CrossEntropyLoss()","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:21:28.132933Z","iopub.execute_input":"2023-09-29T09:21:28.133609Z","iopub.status.idle":"2023-09-29T09:21:28.138735Z","shell.execute_reply.started":"2023-09-29T09:21:28.133578Z","shell.execute_reply":"2023-09-29T09:21:28.137825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import get_scheduler\n\nnum_epochs = 3\nnum_training_steps = num_epochs * len(train_dataloader)\nlr_scheduler = get_scheduler(\n    name=\"linear\", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps\n)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:21:29.966084Z","iopub.execute_input":"2023-09-29T09:21:29.966759Z","iopub.status.idle":"2023-09-29T09:21:29.971681Z","shell.execute_reply.started":"2023-09-29T09:21:29.966729Z","shell.execute_reply":"2023-09-29T09:21:29.970791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\n\ndevice = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\nmodel.to(device)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:21:31.264207Z","iopub.execute_input":"2023-09-29T09:21:31.264872Z","iopub.status.idle":"2023-09-29T09:21:31.276328Z","shell.execute_reply.started":"2023-09-29T09:21:31.264841Z","shell.execute_reply":"2023-09-29T09:21:31.275313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training loop","metadata":{}},{"cell_type":"code","source":"for epoch in range(num_epochs):\n    # Training\n    model.train()\n    total_loss = 0.0\n    for inputs, labels in train_dataloader:\n        optimizer.zero_grad()\n        outputs = model(inputs).logits\n        loss = criterion(outputs, labels)\n        loss.backward()\n        optimizer.step()\n        total_loss += loss.item()\n    avg_train_loss = total_loss / len(train_dataloader)\n\n    # Validation\n    model.eval()\n    with torch.no_grad():\n        total_correct = 0\n        total_samples = 0\n        for inputs, labels in eval_dataloader:\n            outputs = model(inputs).logits\n            _, predicted = torch.max(outputs, 1)\n            total_samples += labels.size(0)\n            total_correct += (predicted == labels).sum().item()\n        val_accuracy = total_correct / total_samples\n\n    # Print training and validation statistics\n    print(f'Epoch [{epoch+1}/{num_epochs}], '\n          f'Loss: {avg_train_loss:.4f}, '\n          f'Val Accuracy: {val_accuracy*100:.2f}%')\n","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:21:58.528743Z","iopub.execute_input":"2023-09-29T09:21:58.529829Z","iopub.status.idle":"2023-09-29T09:22:04.348813Z","shell.execute_reply.started":"2023-09-29T09:21:58.529786Z","shell.execute_reply":"2023-09-29T09:22:04.347885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In 3 epochs we have gotten very high validation accuracy. This is a great job!","metadata":{"execution":{"iopub.status.busy":"2023-09-29T09:15:25.931595Z","iopub.status.idle":"2023-09-29T09:15:25.932397Z","shell.execute_reply.started":"2023-09-29T09:15:25.932163Z","shell.execute_reply":"2023-09-29T09:15:25.932187Z"}}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}