{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30558,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### TinyVGG Model - A Beginner's Approach\n\nThe primary goal of this competition is to develop a model capable of accurately predicting the digit depicted in an image. The dataset consists of a diverse collection of handwritten images.\n\n","metadata":{}},{"cell_type":"markdown","source":"\n ## Contents:\n*

Libraries

\n*

Dataset Transform

\n*

Dataloader and Visualization

\n*

TinyVGG Model

\n \n*

Configs and Training Loop

\n\n*

Inference and Submission

","metadata":{}},{"cell_type":"markdown","source":"
\n

\n Libraries\n

\n
","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport torch\nfrom torch import nn\nimport torchvision\nfrom torchvision import datasets\nfrom torchvision.transforms import ToTensor\nimport matplotlib.pyplot as plt\nimport random\nfrom torchvision import transforms, models\nimport math\nfrom torch.utils.data import Dataset\nfrom torch.utils.data.sampler import SubsetRandomSampler\nfrom tqdm.auto import tqdm\nfrom torchinfo import summary\nfrom torch.utils.data import DataLoader\nimport warnings\nwarnings.simplefilter('ignore')\n%matplotlib inline\nprint(f\"Make sure that the PyTorch version is the same as yours.\")\nprint(f\"PyTorch version: {torch.__version__}\\ntorchvision version: {torchvision.__version__}\")\n","metadata":{"scrolled":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Checking if GPU is available","metadata":{}},{"cell_type":"code","source":"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nif not device:\n print('CPU mode is on...')\nelse:\n print('GPU mode is on...')","metadata":{"scrolled":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Read the training and test datasets. We must predict the numbers depicted in the test dataset. Each row contains the pixel values of a number. The training dataset also includes a label column that represents the corresponding number, whereas the test dataset does not.","metadata":{}},{"cell_type":"code","source":"train_data_csv = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest_data_csv = pd.read_csv(\"../input/digit-recognizer/test.csv\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_csv.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data_csv.head()","metadata":{"scrolled":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Grayscale images must be of size 28x28 (height x width).","metadata":{}},{"cell_type":"code","source":"SIZE = int(math.sqrt(test_data_csv.shape[1]))\nprint(train_data_csv.shape, test_data_csv.shape, SIZE)\ntrain_data_csv.columns[0]\ntrain_data_csv.columns[0] == test_data_csv.columns[0]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

\n Dataset Transform\n

\n
","metadata":{}},{"cell_type":"markdown","source":"The DigitsDataset class is responsible for transforming (preprocessing) images.","metadata":{}},{"cell_type":"code","source":"class DigitsDataset(Dataset):\n def __init__(self, data, transform=None) -> None:\n self.data = data\n self.transform = transform\n \n def __getitem__(self, index):\n \"Iterable function which applies to each row\"\n item = self.data.iloc[index]\n # here we take all pixel columns except label column that's why [1:]\n image = item[1:].values.astype(np.uint8).reshape((SIZE, SIZE))\n # that's the label column\n label = item[0]\n if self.transform:\n image = self.transform(image)\n return image, label\n \n def __len__(self) -> int:\n \"Returns the total number of samples.\"\n return len(self.data)\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TestDigitsDataset(Dataset):\n def __init__(self, data, transform=None) -> None:\n self.data = data\n self.transform = transform\n \n def __getitem__(self, index):\n \"Iterable function which applies to each row\"\n item = self.data.iloc[index]\n # 1 is the channel of an image. In our case it's grayscale. (For rgb is 3).\n image = item[:].values.astype(np.uint8).reshape((SIZE, SIZE, 1))\n if self.transform:\n image = self.transform(image)\n return image\n \n def __len__(self) -> int:\n \"Returns the total number of samples.\"\n return len(self.data)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Necessary transformations, including these normalizations (mean and std), were calculated by taking the mean and standard deviation across all the images in the dataset.","metadata":{}},{"cell_type":"code","source":"transform_train = transforms.Compose([\n transforms.ToTensor(),\n transforms.Normalize(mean=(0.5,), std=(0.5,))\n])\n\ntransform_validation = transforms.Compose([\n transforms.ToTensor(),\n transforms.Normalize(mean=(0.5,), std=(0.5,))\n])\n\ntransform_test = transforms.Compose([\n transforms.ToTensor(),\n transforms.Normalize(mean=(0.5,), std=(0.5,))\n])\n\ntrain_data = DigitsDataset(train_data_csv, transform=transform_train)\nvalidation_data = DigitsDataset(train_data_csv, transform=transform_validation)\ntest_data = TestDigitsDataset(test_data_csv, transform=transform_test)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we divide our training data into validation and training samples. The training subset will contain 80%, and the remaining 20% will be used for validation data.","metadata":{}},{"cell_type":"code","source":"train_size = int(0.8 * len(train_data))\nvalidation_size = len(train_data) - train_size\ntrain_sampler, valid_sampler = torch.utils.data.random_split(train_data, [train_size, validation_size])\nlen(train_sampler), len(valid_sampler)\n\ntrain_sampler = SubsetRandomSampler(train_sampler.indices)\nvalid_sampler = SubsetRandomSampler(valid_sampler.indices)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

\n Dataloader and Visualization\n

\n
","metadata":{}},{"cell_type":"markdown","source":"The dataloader takes training data as an input and samples it by list of indexes with certain batch size. To to speed up processing, it's possible specify the number of workers depending on CPU of your server or pc.","metadata":{}},{"cell_type":"code","source":"train_dataloader = DataLoader(dataset=train_data,\n batch_size=64,\n num_workers=1,\n sampler=train_sampler)\n\nvalidation_dataloader = DataLoader(dataset=train_data, \n batch_size=64,\n num_workers=1, \n sampler=valid_sampler)\n\ntest_dataloader = DataLoader(dataset=test_data, \n batch_size=64, \n num_workers=1, \n shuffle=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Length of train dataloader: {len(train_dataloader)} batches of {32}\")\nprint(f\"Length of test dataloader: {len(test_dataloader)} batches of {32}\")\nprint(f\"Length of validation dataloader: {len(validation_dataloader)} batches of {32}\")","metadata":{"scrolled":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_features_batch, train_labels_batch = next(iter(train_dataloader))\ntrain_features_batch.shape, train_labels_batch.shape","metadata":{"scrolled":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_features_batch, valid_labels_batch = next(iter(validation_dataloader))\nvalid_features_batch.shape, valid_labels_batch.shape","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_features_batch = next(iter(test_dataloader))\nvalid_features_batch.shape","metadata":{"scrolled":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img, label = train_features_batch[0], train_labels_batch[0]\nplt.imshow(img.squeeze(), cmap=\"gray\")\nplt.title(train_labels_batch[0])\nplt.axis(\"Off\");\nprint(f\"Image size: {img.shape}\")\nprint(f\"Label: {label}, label size: {label.shape}\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

\n TinyVGG Model\n

\n
","metadata":{}},{"cell_type":"markdown","source":"Here is the model architecture of TinyVGG. The sequential operator runs the operations defined in each block sequentially. \"Conv\" represents the convolutional operation, \"Relu\" is the activation function that introduces non-linearity to the data, and the \"Max Pooling\" layer reduces the dimension by half. Each operation operates in 2D (Height and Width). The classifier block includes the \"Flatten\" layer, which squeezes the dimensions of the tensor into a single vector, and a linear classification layer with an output shape corresponding to the number of label classes (0-9).","metadata":{}},{"cell_type":"code","source":"class TinyVGG(nn.Module):\n \"\"\"\n Model architecture copying TinyVGG from: \n https://poloclub.github.io/cnn-explainer/\n \"\"\"\n def __init__(self, input_shape: int, hidden_units: int, output_shape: int):\n super().__init__()\n self.block_1 = nn.Sequential(\n nn.Conv2d(in_channels=input_shape, \n out_channels=hidden_units, \n kernel_size=3,\n stride=1,\n padding=1), \n nn.ReLU(),\n nn.Conv2d(in_channels=hidden_units, \n out_channels=hidden_units,\n kernel_size=3,\n stride=1,\n padding=1),\n nn.ReLU(),\n nn.MaxPool2d(kernel_size=2, stride=2))\n \n self.block_2 = nn.Sequential(\n nn.Conv2d(hidden_units, hidden_units, 3, padding=1),\n nn.ReLU(),\n nn.Conv2d(hidden_units, hidden_units, 3, padding=1),\n nn.ReLU(),\n nn.MaxPool2d(2)\n )\n self.classifier = nn.Sequential(\n nn.Flatten(),\n nn.Linear(in_features=hidden_units*7*7, out_features=output_shape))\n \n def forward(self, x: torch.Tensor):\n x = self.block_1(x)\n x = self.block_2(x)\n x = self.classifier(x)\n return x","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_vgg = TinyVGG(input_shape=1, hidden_units=10, output_shape=10).to(device)\nmodel_vgg","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

\n Configs and Training Loop\n

\n
","metadata":{}},{"cell_type":"markdown","source":"Setup loss and optimizer. Try to experiment with learning rate. Training loop consist of training step and validation step.\n","metadata":{}},{"cell_type":"code","source":"LR = 0.01\nEPOCHS = 30\n\nloss_fn = nn.CrossEntropyLoss()\noptimizer = torch.optim.SGD(params=model_vgg.parameters(), lr=LR)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_loss_count, validation_loss_count = [], []\n\ndef accuracy_step(y_true, y_pred):\n correct = torch.eq(y_true, y_pred).sum().item()\n acc = (correct / len(y_pred)) * 100\n return acc\n\n\ndef train_step(model: torch.nn.Module,\n data_loader: torch.utils.data.DataLoader,\n loss_fn: torch.nn.Module,\n optimizer: torch.optim.Optimizer,\n accuracy_step,\n device: torch.device = device):\n train_loss, train_acc = 0, 0\n model.to(device)\n \n for batch, (X, y) in enumerate(data_loader):\n X, y = X.to(device), y.to(device)\n y_pred = model(X)\n loss = loss_fn(y_pred, y)\n train_loss += loss\n train_acc += accuracy_step(y_true=y, y_pred=y_pred.argmax(dim=1))\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n train_loss /= len(data_loader)\n training_loss_count.append(train_loss.cpu().item())\n train_acc /= len(data_loader)\n print(f\"Train loss: {train_loss:.5f} | Train accuracy: {train_acc:.2f}%\")\n\n \ndef validation_step(data_loader: torch.utils.data.DataLoader,\n model: torch.nn.Module,\n loss_fn: torch.nn.Module,\n accuracy_step,\n device: torch.device = device):\n valid_loss, valid_acc = 0, 0\n model.to(device)\n model.eval()\n with torch.inference_mode(): \n for X, y in data_loader:\n X, y = X.to(device), y.to(device)\n valid_pred = model(X)\n valid_loss += loss_fn(valid_pred, y)\n valid_acc += accuracy_step(y_true=y, y_pred=valid_pred.argmax(dim=1))\n valid_loss /= len(data_loader)\n validation_loss_count.append(valid_loss.cpu().item())\n valid_acc /= len(data_loader)\n print(f\"Validation loss: {valid_loss:.5f} | Validation accuracy: {valid_acc:.2f}%\\n\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for epoch in tqdm(range(EPOCHS)):\n print(f\"Epoch: {epoch}\\n---------\")\n train_step(data_loader=train_dataloader, \n model=model_vgg, \n loss_fn=loss_fn,\n optimizer=optimizer,\n accuracy_step=accuracy_step,\n device=device)\n \n validation_step(data_loader=validation_dataloader,\n model=model_vgg,\n loss_fn=loss_fn,\n accuracy_step=accuracy_step,\n device=device)\n ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(training_loss_count, label='Training Loss')\nplt.plot(validation_loss_count, label='Validation Loss')\nplt.legend(frameon=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nsummary(model=model_vgg, \n input_size=(32, 1, 28, 28), # make sure this is \"input_size\", not \"input_shape\"\n col_names=[\"input_size\", \"output_size\", \"num_params\", \"trainable\"],\n col_width=9,\n row_settings=[\"var_names\"]) ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

\n Inference and Submission\n

\n
","metadata":{}},{"cell_type":"markdown","source":"Setup Torch in inference mode and make predictions for the numbers depicted in the images loaded into the test_dataloader. Read the sample_submission.csv file and place predicted labels for each corresponding ImageId. Each ImageId field corresponds to each row in test.csv.","metadata":{}},{"cell_type":"code","source":"submission_df = pd.read_csv(\"../input/digit-recognizer/sample_submission.csv\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = []\nwith torch.inference_mode():\n for X in test_dataloader:\n X = X.to(device)\n y_preds = model_vgg(X)\n y_preds = y_preds.argmax(dim=1)\n for y_pred in y_preds.cpu():\n submission.append(y_pred.item())\nsubmission_df['Label'] = submission","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show a sample\nindex = random.randint(0, len(test_features_batch)-1)\ntest_image = test_features_batch[index]\nplt.imshow(test_image.squeeze(), cmap=\"gray\")\nplt.title(\"Predicted Label: {}\".format(submission_df['Label'][index]),fontsize=20, c=\"black\")\nplt.axis(\"Off\");\n\nprint(f\"Image size: {img.shape}\")\nprint(f\"Label: {label}, label size: {label.shape}\")","metadata":{"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.to_csv('submission.csv', index=False)\nprint('Submitted successfully...')\nprint('Thank you!')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thank you for taking the time to read this notebook. I hope you found it useful and informative. \nIf you enjoyed and gained insights from this kernel, please consider showing your support by UPVOTING. Your support is greatly appreciated🌞.\n","metadata":{}}]}