{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":31234,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 1. Introduction\n\nFor the task of digit recognition we will use one of the foundational neural network architectures founded by Yann LeCunn, Léon Bottou, Yoshua Bengio, and Patrick Haffner known as LeNet-5. \n\nLeNet-5 is aconvolutional neural network (CNN) & it introduced features such as weight sharing, feature classification, spatial invariance etc... \n\nThe LeNet-5 paper [GradientBased Learning Applied to Document\nRecognition](http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf) was published in 1998 & some of the ideas that inspired the network's architecture go back to the 1960's from the work of neurobioligists Hubel & Weisel on simple connections in the neural networks of cats. Furthermore the gradient based learning methods used during back propagation of the network go back to the 1950's. \n\nAnd you thought Ai was new!","metadata":{}},{"cell_type":"markdown","source":"# 2. Key Features","metadata":{}},{"cell_type":"markdown","source":"### 2.1 Feature Mapping & Weight Sharing\n\nTo reduce the cost of fully connecting large inputs such as images the idea of weight sharing & feature maps is implemented in Lenet-5. This idea stems from the work on simple connections by Hubel & Weisel & is the technique used in convolutional layers.\n\nIn a convolutional layer a feature map of size fXf is dragged across an input taking dot products adding a bias & passing the result through an activation function, or convolving the input into a smaller size. This feature map or feature classifier has fxf learned paramaters used to classify features of the input image. In early layers of the network low level features are extracted then combined with later layers to recognize more complicated features.","metadata":{}},{"cell_type":"markdown","source":"### 2.2 Sub Sampling\n\nOnce a feature is detected by any of the feature maps the technique of sub sampling is used to reduce the precision of the encoded image. Reducing the precision will allow the network to disregard the precise position of the feature. You may think this counter intuitive but it can actually be harmful to the accuracy of prediction as these features are likely to show up in different positions for each distinctive hand written digit.\n\nSub sampling or pooling can be achieved by means or maximums. LeNet-5's sub sampling layers perform local averaging multiplied by a trainable coefficient, added to a bias & passes the result through a sigmoid function.\n","metadata":{}},{"cell_type":"markdown","source":"### 2.3 Fully connected outputs\n\nThe output layer of LeNet-5 is composed of Euclidean Radial Basis Function units (RBF). Each output of an RBF unit computes the Euclidean distance between the input vector & its parameter vector. Each output classifying the result of a forward pass through the network.\n\nThe input vectors were chosen in such a way to represent a stylized image of the corresponding character class. \n\nIn modern implemenations of LeNet-5 a SoftMax activation is applied to the final layer to get probabilities of each class & a Cross-Entropy Loss function is minimized. This is the approach that will be further discussed in this notebook.","metadata":{}},{"cell_type":"markdown","source":"### 2.4 Gradient Based Learning & Cost Function\n\nAs stated previously the authors of LeNet-5 sought to minimize the loss of the RBF units using MAP (maximum a posteriori), this is described in the paper but not in this notebook.\n\nThe implementation in this notebook will use gradient descent to minimize a Cross-Entropy loss function corresponding to the output of SoftMax units.","metadata":{}},{"cell_type":"markdown","source":"# 3. Architecture","metadata":{}},{"cell_type":"markdown","source":"### 3.1 Input layer\n\nLeNet-5 excepts an input of size 32x32 pixels. The network is designed to be trained on the MNIST data set, a subset of Nist database consisting of 60,000 handwritten digits & 10,00 test examples of size 28x28 pixels. The input of the network being larger than the training examples forces centering of images so corners and endpoints can be properly considered by the network. Each unit of input is normalized to a value between 0 & 1 corresponding to white and black values respectively.","metadata":{}},{"cell_type":"markdown","source":"### 3.2 Layer C1\n\nA convolutional layer with 6 feature maps. Each feature map, filter or synonymously kernel are of size 5x5 producing a 28x28x6 convolved output. Trainable paramaters including weights & bias amount to 156 with 122,304 connections.","metadata":{}},{"cell_type":"markdown","source":"### 3.3 Layer S2\n\nLayer S2 is a sub-sampling layer with 6 feature maps of size 2x2. This sub-sampling layer also reffered to as a pooling layer 'pools' connections in a 2x2 neighborhood by adding 4 inputs, multiplying by a trainable coefficient & adding a trainable bias. The result is passed through a sigmoid function. Layer S2 has 12 trainable parameters & 5,880 connections with output of size 14x14x6.","metadata":{}},{"cell_type":"markdown","source":"### 3.4 Layer C3\n\nLayer C3 is a convolutional layer with 16 feature maps of size 5x5. Layer C3 is not fully connected forcing a break in symmetry in hopes to extract different complimentary features. Layer C3 has 1,516 trainable parameters, 151,600 connections & produces and output of size 10x10x16.","metadata":{}},{"cell_type":"markdown","source":"### 3.5 Layer S4\n\nLayer S4 is a sub-sampling layer with 16 feature maps. Each feature map or filter is sized 2x2 behaving similarly to C1 & S2 with 32 trainable parameters, 2,000 connections and an output of size 5x5x16.","metadata":{}},{"cell_type":"markdown","source":"### 3.6 Layer C5\n\nLayer C5 is a convolutional layer with 120 feature maps of size 1x1 amounting to a full connection with S4. Layer C5 has 48,120 trainable connections.","metadata":{}},{"cell_type":"markdown","source":"### 3.7 Layer F6\n\nLayer F6 is a fully connected layer conating 84 units fully connected to 120 connections of C5 producing 10,164 trainable connections. ","metadata":{}},{"cell_type":"markdown","source":"### 3.8 Output Layer\n\nInitial approach:\n\nThe output layer is composed of Euclidean Radial Basis Function units (RBF), 1 for each class and 84 inputs each. The outputs of each unit y are computed as the Euclidean distance between it's input vector & its parameter vector.\n\nModernized approach:\n\nA SoftMax output layer with 10 values corresponding to digits 0 through 9.\n","metadata":{}},{"cell_type":"markdown","source":"# 4. Implementation","metadata":{}},{"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport tensorflow as tf\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom tensorflow.keras.utils import to_categorical # Or from keras.utils import to_categorical\nfrom sklearn.model_selection import train_test_split\n\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n\nbatch_size = 64\nnum_classes = 10\nlearning_rate = 0.001\nnum_epochs = 10","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-28T19:48:47.571286Z","iopub.execute_input":"2025-12-28T19:48:47.571560Z","iopub.status.idle":"2025-12-28T19:49:22.688841Z","shell.execute_reply.started":"2025-12-28T19:48:47.571534Z","shell.execute_reply":"2025-12-28T19:49:22.687940Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 4.1 Data Loading & Preprocessing\n\nTraining data consists of 42,000 examples each a flattened column vector of length 785 representing a 28x28 pixel image encoding & a label corresponding to the character represented.\nSimilarly the test set contains 28,000 column vectors but they are not labeled.\n\nFirst we will drop & one-hot-encode labels in train data.\nOur implementation of LeNet-5 excepts a 4d tensor (examples, channels, height, width), height & width of size 32X32 so we will reshape the data. We will also normalize the data as best practice to ensure faster processing.","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\nsample = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")\n\nprint(train.info(), \"\\n\")\nprint(sample.head())\n\nprint(train.shape)\nprint(test.shape)\n\nX_train = train.drop('label',axis=1)\ny_train = train['label']\n\ny_train_encoded = to_categorical(y_train, num_classes=10)\nX_train = X_train/255\n\ny_train_encoded = torch.tensor(y_train_encoded, dtype=torch.float32)\nX_train = torch.tensor(X_train.values,dtype=torch.float32)\n\nX_train = X_train.reshape(-1,1,28,28)\nX_train = F.pad(X_train, (2,2,2,2),mode='constant', value=0)\nprint(X_train.shape)\nprint(y_train_encoded.shape)\n\n\n\nX_train, X_test, y_train, y_test = train_test_split(X_train,y_train_encoded,test_size=0.2,random_state=42)\n\ntrain_dataset = torch.utils.data.TensorDataset(X_train, y_train)\ntest_dataset = torch.utils.data.TensorDataset(X_test, y_test)\nprint(X_test.shape)\nprint(y_test.shape)\n\ntrain_loader = torch.utils.data.DataLoader(dataset=train_dataset,batch_size=64,shuffle=True)\ntest_loader = torch.utils.data.DataLoader(dataset=test_dataset,batch_size=64,shuffle=True)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-28T19:58:20.390425Z","iopub.execute_input":"2025-12-28T19:58:20.390783Z","iopub.status.idle":"2025-12-28T19:58:24.203721Z","shell.execute_reply.started":"2025-12-28T19:58:20.390751Z","shell.execute_reply":"2025-12-28T19:58:24.202792Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 4.2 Model Building\n\nFollowing the LeNet-5 architecture stated above we will build a class model using the framework PyTorch & appropriate modules.","metadata":{}},{"cell_type":"code","source":"class LeNet5(nn.Module):\n    def __init__(self, num_classes=10):\n        super(LeNet5, self).__init__()\n        self.conv1 = nn.Sequential(nn.Conv2d(1,6,kernel_size=5),nn.BatchNorm2d(6))\n        self.pool1 = nn.AvgPool2d(kernel_size=2, stride=2)\n        self.conv2 = nn.Sequential(nn.Conv2d(6,16,kernel_size=5),nn.BatchNorm2d(16))\n        self.pool2 = nn.AvgPool2d(kernel_size=2, stride=2)\n        self.fc1 = nn.Linear(16*5*5, 120)\n        self.fc2 = nn.Linear(120,84)\n        self.out = nn.Linear(84, num_classes)\n\n    def forward(self, x):\n        x = self.pool1(F.relu(self.conv1(x)))\n        x = self.pool2(F.relu(self.conv2(x)))\n        x = x.view(-1,16*5*5)\n        x = F.relu(self.fc1(x))\n        x = F.relu(self.fc2(x))\n        x = self.out(x)\n        return x\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-28T19:53:11.934914Z","iopub.execute_input":"2025-12-28T19:53:11.935306Z","iopub.status.idle":"2025-12-28T19:53:11.945137Z","shell.execute_reply.started":"2025-12-28T19:53:11.935278Z","shell.execute_reply":"2025-12-28T19:53:11.944062Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 4.3 Train Model","metadata":{}},{"cell_type":"code","source":"model = LeNet5(num_classes)\ncriterion = nn.CrossEntropyLoss()\n\noptimizer = torch.optim.Adam(model.parameters(),lr=learning_rate)\n\ndef train_model(model, train_loader, criterion, optimizer, epochs):\n    model.train()\n    for epoch in range(epochs):\n        running_loss = 0.0\n        for images, labels in train_loader:\n\n           optimizer.zero_grad()\n\n           outputs = model(images)\n           loss = criterion(outputs, labels)\n\n           loss.backward()\n           optimizer.step()\n            \n           running_loss += loss.item()\n        print(f\"Epoch {epoch+1}, Loss: {running_loss/len(train_loader)}\")\n\ntrain_model(model, train_loader, criterion, optimizer,epochs=10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-28T19:58:24.205474Z","iopub.execute_input":"2025-12-28T19:58:24.205765Z","iopub.status.idle":"2025-12-28T19:59:39.670995Z","shell.execute_reply.started":"2025-12-28T19:58:24.205740Z","shell.execute_reply":"2025-12-28T19:59:39.669672Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 4.4 Evaluate Model","metadata":{}},{"cell_type":"code","source":"\n\ndef evaluate_model(model, test_loader):\n    model.eval()\n    correct = 0\n    total = 0\n    total_loss = 0.0\n    \n    with torch.no_grad():\n        for images, labels in test_loader:\n\n            outputs = model(images)\n\n            loss = criterion(outputs, labels)\n            total_loss += loss.item() * images.size(0)\n            \n            _, predicted = torch.max(outputs.data, 1)\n            _,labels = torch.max(labels,1)\n            total += len(labels)\n            correct += (predicted == labels).sum().item()\n           \n    avg_loss = total_loss / total\n    accuracy = 100 * correct / total\n    \n    print(f'Accuracy of the network on the {total} test images: {accuracy:.2f} %')\n    print(f'Average Test Loss: {avg_loss:.4f}')\n\n    return accuracy, avg_loss\n\nevaluate_model(model, test_loader)\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-28T19:59:40.286113Z","iopub.execute_input":"2025-12-28T19:59:40.286744Z","iopub.status.idle":"2025-12-28T19:59:40.892339Z","shell.execute_reply.started":"2025-12-28T19:59:40.286716Z","shell.execute_reply":"2025-12-28T19:59:40.891608Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 5. Create Submission","metadata":{}},{"cell_type":"code","source":"submission_test_data = test/255\nsubmission_test_data = torch.tensor(submission_test_data.values,dtype=torch.float32)\nsubmission_test_data = submission_test_data.reshape(-1,1,28,28)\nsubmission_test_data = F.pad(submission_test_data, (2,2,2,2),mode='constant', value=0)\n\nprint(submission_test_data.shape)\n\nsubmission_loader = torch.utils.data.DataLoader(dataset=submission_test_data,batch_size=64,shuffle=False)\n\n\ndef model_predict(submission_loader):\n    model.eval() \n\n    predictions = []\n    with torch.no_grad(): \n        for images in submission_loader:\n            \n            outputs = model(images)\n\n            _, predicted = torch.max(outputs.data, 1)\n            predicted = predicted.numpy()\n            predictions = np.concatenate((predictions,predicted))\n    return predictions\n\npredictions = model_predict(submission_loader)\npredictions = predictions.astype(int)\nimage_id = test.index + 1\nsubmission = pd.DataFrame({'ImageId':image_id,'Label':predictions})\nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()\n\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-28T20:01:01.087728Z","iopub.execute_input":"2025-12-28T20:01:01.088065Z","iopub.status.idle":"2025-12-28T20:01:03.059575Z","shell.execute_reply.started":"2025-12-28T20:01:01.088037Z","shell.execute_reply":"2025-12-28T20:01:03.058553Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}