{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Introduction**\n\nThis notebook is a baseline for training a simple neuronal network on GPU without augmentations and should be easy to understand for anyone. In the Session options -> Accelerator ->  should select a GPU","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:39:58.731108Z","iopub.execute_input":"2025-09-25T19:39:58.731357Z","iopub.status.idle":"2025-09-25T19:40:00.988172Z","shell.execute_reply.started":"2025-09-25T19:39:58.731331Z","shell.execute_reply":"2025-09-25T19:40:00.987484Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# **Import packages**","metadata":{}},{"cell_type":"code","source":"import torch \nimport torch.nn as nn \nimport matplotlib.pyplot as plt # visualization\nfrom torch.utils.data import DataLoader, TensorDataset # load the data\nfrom sklearn.model_selection import StratifiedKFold\nfrom torch.optim import SGD\nfrom tqdm import tqdm # track the progress ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:49:16.509836Z","iopub.execute_input":"2025-09-25T19:49:16.510362Z","iopub.status.idle":"2025-09-25T19:49:16.514645Z","shell.execute_reply.started":"2025-09-25T19:49:16.510339Z","shell.execute_reply":"2025-09-25T19:49:16.513713Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# **Import training data from Kaggle**","metadata":{}},{"cell_type":"code","source":"train_data=pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:06.528202Z","iopub.execute_input":"2025-09-25T19:40:06.528716Z","iopub.status.idle":"2025-09-25T19:40:09.398435Z","shell.execute_reply.started":"2025-09-25T19:40:06.528684Z","shell.execute_reply":"2025-09-25T19:40:09.397582Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's look at the first 5 rows:","metadata":{}},{"cell_type":"code","source":"train_data.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.399299Z","iopub.execute_input":"2025-09-25T19:40:09.399574Z","iopub.status.idle":"2025-09-25T19:40:09.428271Z","shell.execute_reply.started":"2025-09-25T19:40:09.399550Z","shell.execute_reply":"2025-09-25T19:40:09.427482Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The dataset contains the label (0 to 9) in the first column and then either a value between 0 and 255 for each of the 784 pixels. The higher the number, the darker the color. We will have a look at the first picture:","metadata":{}},{"cell_type":"code","source":"# Take first row [0], leave out the label column [1:]\nx1=train_data.iloc[0,1:].values","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.429079Z","iopub.execute_input":"2025-09-25T19:40:09.429396Z","iopub.status.idle":"2025-09-25T19:40:09.433308Z","shell.execute_reply.started":"2025-09-25T19:40:09.429376Z","shell.execute_reply":"2025-09-25T19:40:09.432483Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# For visualization of the image, reshape the first into a 28 x 28 shape:\nx1re=x1.reshape(28,28)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.434053Z","iopub.execute_input":"2025-09-25T19:40:09.434293Z","iopub.status.idle":"2025-09-25T19:40:09.449084Z","shell.execute_reply.started":"2025-09-25T19:40:09.434271Z","shell.execute_reply":"2025-09-25T19:40:09.448421Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot the reshaped image\nplt.imshow(x1re)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.449905Z","iopub.execute_input":"2025-09-25T19:40:09.450127Z","iopub.status.idle":"2025-09-25T19:40:09.728349Z","shell.execute_reply.started":"2025-09-25T19:40:09.450107Z","shell.execute_reply":"2025-09-25T19:40:09.727669Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Split the training data into training and validation set**","metadata":{}},{"cell_type":"code","source":"# Generate a config class:\nclass CFG:\n    batch_size = 32 # Specifies how many images are in one batch\n    lr = 1e-3 # Learning rate, how much the model adapts the parameters in each step\n    num_epochs = 200 # Number of epochs, how long the model is training\n    n_folds = 5 # Number of folds","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.729061Z","iopub.execute_input":"2025-09-25T19:40:09.729280Z","iopub.status.idle":"2025-09-25T19:40:09.733349Z","shell.execute_reply.started":"2025-09-25T19:40:09.729262Z","shell.execute_reply":"2025-09-25T19:40:09.732589Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"x = train_data\ny = train_data['label']\ncv = StratifiedKFold(n_splits=CFG.n_folds) # Stratified k fold preserves number of occurences in dataset\nsplit = list(cv.split(x,y)) # generates a list of the train and validation split for each fold","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.735892Z","iopub.execute_input":"2025-09-25T19:40:09.736086Z","iopub.status.idle":"2025-09-25T19:40:09.759036Z","shell.execute_reply.started":"2025-09-25T19:40:09.736071Z","shell.execute_reply":"2025-09-25T19:40:09.758359Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Add a fold column into the train dataset\ntrain_data['fold'] = -1 #dummy which gets overwriten in next 2 lines\nfor fold,(tr_idx, val_idx) in enumerate(split):\n    train_data.iloc[val_idx,-1] = fold \n# loops over the just generated split list with training and validation index and adds this into the fold column","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.759765Z","iopub.execute_input":"2025-09-25T19:40:09.759962Z","iopub.status.idle":"2025-09-25T19:40:09.765898Z","shell.execute_reply.started":"2025-09-25T19:40:09.759947Z","shell.execute_reply":"2025-09-25T19:40:09.765378Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Here us what the training data looks like now:","metadata":{}},{"cell_type":"code","source":"train_data.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.766517Z","iopub.execute_input":"2025-09-25T19:40:09.766718Z","iopub.status.idle":"2025-09-25T19:40:09.790913Z","shell.execute_reply.started":"2025-09-25T19:40:09.766703Z","shell.execute_reply":"2025-09-25T19:40:09.790053Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Are the folds the same size?\ntrain_data['fold'].value_counts(normalize=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.791803Z","iopub.execute_input":"2025-09-25T19:40:09.791990Z","iopub.status.idle":"2025-09-25T19:40:09.810161Z","shell.execute_reply.started":"2025-09-25T19:40:09.791976Z","shell.execute_reply":"2025-09-25T19:40:09.809618Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Yes, the data is evenly split into 5 folds, each containing 20% of the data.","metadata":{}},{"cell_type":"code","source":"# How is the distribution of the lables in the dataset?\ntrain_data['label'].value_counts(normalize=True).sort_index()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.810770Z","iopub.execute_input":"2025-09-25T19:40:09.810940Z","iopub.status.idle":"2025-09-25T19:40:09.828560Z","shell.execute_reply.started":"2025-09-25T19:40:09.810926Z","shell.execute_reply":"2025-09-25T19:40:09.828027Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.hist(train_data['label'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.829328Z","iopub.execute_input":"2025-09-25T19:40:09.829965Z","iopub.status.idle":"2025-09-25T19:40:09.962796Z","shell.execute_reply.started":"2025-09-25T19:40:09.829936Z","shell.execute_reply":"2025-09-25T19:40:09.961948Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Similar counts for all digits.","metadata":{}},{"cell_type":"markdown","source":"**Prepare the dataloader of training and validation set**","metadata":{}},{"cell_type":"code","source":"def prepare_loaders(fold, batch_size):\n    train_df = train_data[train_data[\"fold\"]!=fold].reset_index(drop=True)\n    valid_df = train_data[train_data[\"fold\"]==fold].reset_index(drop=True)\n\n    X = train_df.iloc[:,1:-1].values / 255 # norm picture to values between 0-1\n    y = train_df.iloc[:,0].values\n    train_dataset = TensorDataset(torch.from_numpy(X), torch.from_numpy(y))\n\n    X_val = valid_df.iloc[:,1:-1].values / 255 # norm picture to values between 0-1\n    y_val = valid_df.iloc[:,0].values\n    valid_dataset = TensorDataset(torch.from_numpy(X_val), torch.from_numpy(y_val))\n\n    train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True, pin_memory=True, drop_last=False)\n    valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=batch_size, shuffle=False, pin_memory=True)\n    \n    return train_loader, valid_loader","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.963572Z","iopub.execute_input":"2025-09-25T19:40:09.963845Z","iopub.status.idle":"2025-09-25T19:40:09.969761Z","shell.execute_reply.started":"2025-09-25T19:40:09.963821Z","shell.execute_reply":"2025-09-25T19:40:09.968869Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Load the training and validation loader for fold 0.","metadata":{}},{"cell_type":"code","source":"train_loader, valid_loader = prepare_loaders(fold=0, batch_size=CFG.batch_size) ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:40:09.970487Z","iopub.execute_input":"2025-09-25T19:40:09.970749Z","iopub.status.idle":"2025-09-25T19:40:10.465442Z","shell.execute_reply.started":"2025-09-25T19:40:09.970733Z","shell.execute_reply":"2025-09-25T19:40:10.464893Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = nn.Sequential(nn.Linear(784, 1024), # 1st linear layer\n                      nn.LeakyReLU(negative_slope=0.05), # Leaky rectified linear unit \n                      nn.Linear(1024, 512), # 2nd linear layer\n                      nn.LeakyReLU(negative_slope=0.05),\n                      nn.Linear(512, 256), # 3rd linear layer\n                      nn.LeakyReLU(negative_slope=0.05),\n                      nn.Linear(256, 10)) # 4th linear layer outputs 10 features\nmodel.cuda() # Push the model to GPU\noptimizer = SGD(model.parameters(), lr=CFG.lr) # Stochastic gradient descent\ncriterion = nn.CrossEntropyLoss() #loss function","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:49:23.716251Z","iopub.execute_input":"2025-09-25T19:49:23.716841Z","iopub.status.idle":"2025-09-25T19:49:26.501712Z","shell.execute_reply.started":"2025-09-25T19:49:23.716817Z","shell.execute_reply":"2025-09-25T19:49:26.500939Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Generate the NN model\nInitiate a very simple NN with 4 linear layers.","metadata":{}},{"cell_type":"markdown","source":"Train the model and calculate validation loss (Cross Entropy Loss) and accuracy. \nSave the model parameter (later loaded in inference notebook). ","metadata":{}},{"cell_type":"code","source":"train_loss = []\nv_loss2 = []\nacc2 = []\nfor i in range(CFG.num_epochs):\n    model.train()\n    for data in tqdm(train_loader):\n        # Set the gradients to zero\n        optimizer.zero_grad()\n        # Run a forward pass\n        feature, target = data\n        prediction = model(feature.float().cuda())\n        prediction = prediction.cpu()\n        # Compute the loss\n        loss = criterion(prediction, target)\n        # Compute the gradients\n        loss.backward()\n        # Update the model's parameters\n        optimizer.step()\n        train_loss += [loss.item()]\n    val_loss = []\n    accuracy = []\n    model.eval()\n    with torch.no_grad():\n        for data in tqdm(valid_loader):\n            feature, target = data\n            prediction = model(feature.float().cuda())\n            prediction = prediction.cpu()\n            loss = criterion(prediction, target)\n            val_loss += [loss.item()]\n            accuracy += [(prediction.softmax(1).argmax(1) == target).float().mean()]\n    v_loss = np.array(val_loss).mean()\n    print(f'Epoch: {i}')\n    print(f'Val_loss: {v_loss}')\n    acc = np.array(accuracy).mean()\n    print(f'Accuracy: {acc}')\n    v_loss2 += [v_loss]\n    acc2 += [acc]\n    PATH = f\"last_epoch-{fold:02d}.bin\"\n    torch.save(model.state_dict(), PATH) # Saves the model weights","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:50:17.995866Z","iopub.execute_input":"2025-09-25T19:50:17.996142Z","iopub.status.idle":"2025-09-25T19:58:19.093334Z","shell.execute_reply.started":"2025-09-25T19:50:17.996120Z","shell.execute_reply":"2025-09-25T19:58:19.092469Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Plot the training and validation loss**","metadata":{}},{"cell_type":"code","source":"all_loss = np.array(train_loss)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:58:36.086390Z","iopub.execute_input":"2025-09-25T19:58:36.086926Z","iopub.status.idle":"2025-09-25T19:58:36.101858Z","shell.execute_reply.started":"2025-09-25T19:58:36.086901Z","shell.execute_reply":"2025-09-25T19:58:36.100931Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.plot(all_loss)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:58:37.543275Z","iopub.execute_input":"2025-09-25T19:58:37.543556Z","iopub.status.idle":"2025-09-25T19:58:37.879487Z","shell.execute_reply.started":"2025-09-25T19:58:37.543536Z","shell.execute_reply":"2025-09-25T19:58:37.878678Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.plot(v_loss2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:58:41.667270Z","iopub.execute_input":"2025-09-25T19:58:41.667767Z","iopub.status.idle":"2025-09-25T19:58:41.801855Z","shell.execute_reply.started":"2025-09-25T19:58:41.667744Z","shell.execute_reply":"2025-09-25T19:58:41.801076Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Plot the accuracy**","metadata":{}},{"cell_type":"code","source":"plt.plot(acc2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-25T19:58:47.520050Z","iopub.execute_input":"2025-09-25T19:58:47.520320Z","iopub.status.idle":"2025-09-25T19:58:47.665481Z","shell.execute_reply.started":"2025-09-25T19:58:47.520299Z","shell.execute_reply":"2025-09-25T19:58:47.664852Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This notebook just trains a model, in order to create a submission to the competition I generated an inference notebook where the trained model / weights are loaded. In this case the same notebook could train the model and predict the unknown labels, as the computational costs are low. However, if the model is big, just importing the weights into a separate [Inference notebook](https://www.kaggle.com/code/bridgeoverwater/gpu-infer-digit-recognizer) is more efficient.\n","metadata":{}},{"cell_type":"markdown","source":"[](http://)","metadata":{}}]}