{"metadata":{"kernelspec":{"display_name":"Python 3","name":"python3","language":"python"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":31011,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Introduction\n\nMNIST (\"Modified National Institute of Standards and Technology\") is the de facto “Hello World” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n\nIn this competition, we aim to correctly identify digits from a dataset of tens of thousands of handwritten images. Kaggle has curated a set of tutorial-style kernels which cover everything from regression to neural networks. They hope to encourage us to experiment with different algorithms to learn first-hand what works well and how techniques compare.\n\n## Approach\n\nFor this competition, we will be using Keras (with TensorFlow as our backend) as the main package to create a simple neural network to predict, as accurately as we can, digits from handwritten images. In particular, we will be calling the Functional Model API of Keras, and creating a 4-layered and 5-layered neural network.\n\nAlso, we will be experimenting with various optimizers: the plain vanilla Stochastic Gradient Descent optimizer and the Adam optimizer. However, there are many other parameters, such as training epochs which will we will not be experimenting with.\n\nIn addition, the choice of hidden layer units are completely arbitrary and may not be optimal. This is yet another parameter which we will not attempt to tinker with. Lastly, we introduce dropout, a form of regularisation, in our neural networks to prevent overfitting.\n\n## Result\n\nFollowing our simulations on the cross validation dataset, it appears that a 4-layered neural network, using 'Adam' as the optimizer along with a learning rate of 0.01, performs best. We proceed to introduce dropout in the model, and use the model to predict for the test set.\n\nThe test predictions (submitted to Kaggle) generated by our model predicts with an accuracy score of 97.600%, which places us at the top 55 percentile of the competition.","metadata":{"_cell_guid":"82418877-8407-45ef-9b76-86ca80e07abf","_uuid":"483f57b296f8d69647bbec1154acfa2da3c6fc2f","_execution_state":"idle","collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Библиотеки\nimport pandas as pd\nimport numpy as np\nimport torch\nfrom torch import nn\nimport torchvision\nfrom torchvision import datasets\nfrom torchvision.transforms import ToTensor\nimport matplotlib.pyplot as plt\n\nnp.random.seed(1212)\n\nprint('Библиотеки загружены.')","metadata":{"_cell_guid":"55ec59e1-8a8c-42d2-8ccc-b6d95ea198b4","_uuid":"a6d625ede9e19ac1f146a5f63d6ca19c4b98e20c","_execution_state":"busy","trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:02:53.079652Z","iopub.execute_input":"2025-05-05T10:02:53.080304Z","iopub.status.idle":"2025-05-05T10:03:05.368775Z","shell.execute_reply.started":"2025-05-05T10:02:53.080279Z","shell.execute_reply":"2025-05-05T10:03:05.368170Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_train = pd.read_csv('../input/train.csv')\ndf_test = pd.read_csv('../input/test.csv')","metadata":{"_cell_guid":"daa0805e-e1dd-4806-b2db-ff0124febe7d","_uuid":"be0a7acbb3bfc83fd3103b4a401337aafe57c00d","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-05-05T10:03:08.154436Z","iopub.execute_input":"2025-05-05T10:03:08.154827Z","iopub.status.idle":"2025-05-05T10:03:12.731022Z","shell.execute_reply.started":"2025-05-05T10:03:08.154806Z","shell.execute_reply":"2025-05-05T10:03:12.730450Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_train.head()","metadata":{"_cell_guid":"dfae6a31-23ea-4d88-9fcd-06b8b705b614","_uuid":"0f7bfb5923e5b4ed5a5149c007a2cd39cb9b8282","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-05-05T10:03:15.896596Z","iopub.execute_input":"2025-05-05T10:03:15.897325Z","iopub.status.idle":"2025-05-05T10:03:15.928039Z","shell.execute_reply.started":"2025-05-05T10:03:15.897299Z","shell.execute_reply":"2025-05-05T10:03:15.927407Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_test.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:03:24.785688Z","iopub.execute_input":"2025-05-05T10:03:24.785943Z","iopub.status.idle":"2025-05-05T10:03:24.797828Z","shell.execute_reply.started":"2025-05-05T10:03:24.785924Z","shell.execute_reply":"2025-05-05T10:03:24.797254Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_features = df_train.iloc[:, 1:785]\ndf_label = df_train.iloc[:, 0]\n\nX_test = df_test.iloc[:, 0:784]\n\nprint(df_features.shape)\nprint(df_test.shape)","metadata":{"_cell_guid":"bb78680a-ce50-4ee9-a77f-265b9ee38aee","_uuid":"16e39b5512ed29c8f9d7db51331693e4689ce306","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-05-05T10:05:28.682607Z","iopub.execute_input":"2025-05-05T10:05:28.683042Z","iopub.status.idle":"2025-05-05T10:05:28.687455Z","shell.execute_reply.started":"2025-05-05T10:05:28.683024Z","shell.execute_reply":"2025-05-05T10:05:28.686849Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Видим, что датасеты содержат 784 фичи и у трейн есть лейбл.\n# Нет валидационного набора, поэтому разделимм df_train на обучающую и валидационную выборки\n\nfrom sklearn.model_selection import train_test_split\n\nX_train, X_cv, y_train, y_cv = train_test_split(df_features, df_label, \n                                                test_size = 0.2,\n                                                random_state = 1212)\n\nX_train = X_train.to_numpy().reshape(33600, 784) #(33600, 784)\nX_cv = X_cv.to_numpy().reshape(8400, 784) #(8400, 784)\n\nX_test = X_test.to_numpy().reshape(28000, 784)","metadata":{"_cell_guid":"a3fb0b37-1edf-42bd-b951-86f254b78fc1","_uuid":"a98808f6188e9c44741ad2b5c25a894106406dfc","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-05-05T10:05:53.926754Z","iopub.execute_input":"2025-05-05T10:05:53.926998Z","iopub.status.idle":"2025-05-05T10:05:54.792324Z","shell.execute_reply.started":"2025-05-05T10:05:53.926983Z","shell.execute_reply":"2025-05-05T10:05:54.791750Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print((min(X_train[1]), max(X_train[1])))","metadata":{"_cell_guid":"4d0e07b5-28dd-42f1-b425-6414aea90cde","_uuid":"cc2daf9f099fb62e3b470f913349428ef396b75b","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-05-05T10:10:44.033246Z","iopub.execute_input":"2025-05-05T10:10:44.033835Z","iopub.status.idle":"2025-05-05T10:10:44.038202Z","shell.execute_reply.started":"2025-05-05T10:10:44.033797Z","shell.execute_reply":"2025-05-05T10:10:44.037552Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Интенсивность пикселей находится в диапазоне от 0 до 255\n# Нормализуем признаки с помощью трансляции\n# Ещё преобразуем метки из вектора класса в двоичный One Hot Encoded\n\nX_train = X_train.astype('float32'); X_cv= X_cv.astype('float32'); X_test = X_test.astype('float32')\nX_train /= 255; X_cv /= 255; X_test /= 255\n\ny_train_tensor = torch.tensor(y_train.values)\ny_cv_tensor = torch.tensor(y_cv.values)   \n\nnum_digits = 10\ny_train_tensor = torch.nn.functional.one_hot(y_train_tensor, num_digits)\ny_cv_tensor = torch.nn.functional.one_hot(y_cv_tensor, num_digits)","metadata":{"_cell_guid":"e9f8eaa0-c77e-4510-9ff1-bb56e1d5f76d","_uuid":"c24261ffd2384078a836a909ea4839a31d51dd66","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-05-05T10:10:47.023109Z","iopub.execute_input":"2025-05-05T10:10:47.023626Z","iopub.status.idle":"2025-05-05T10:10:47.166235Z","shell.execute_reply.started":"2025-05-05T10:10:47.023601Z","shell.execute_reply":"2025-05-05T10:10:47.165435Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(y_train_tensor[0])\nprint(y_cv_tensor[3])","metadata":{"_cell_guid":"6667ca35-44ff-43f3-b88b-27b787be4ee0","_uuid":"3fdf37a538504f4e5288cbb8b70a719c2689caca","_execution_state":"busy","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-05-05T10:11:08.748899Z","iopub.execute_input":"2025-05-05T10:11:08.749628Z","iopub.status.idle":"2025-05-05T10:11:08.754470Z","shell.execute_reply.started":"2025-05-05T10:11:08.749602Z","shell.execute_reply":"2025-05-05T10:11:08.753884Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_cv.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:11:10.877114Z","iopub.execute_input":"2025-05-05T10:11:10.877602Z","iopub.status.idle":"2025-05-05T10:11:10.882267Z","shell.execute_reply.started":"2025-05-05T10:11:10.877579Z","shell.execute_reply":"2025-05-05T10:11:10.881558Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"y_train_tensor.size()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:11:13.260810Z","iopub.execute_input":"2025-05-05T10:11:13.261471Z","iopub.status.idle":"2025-05-05T10:11:13.266557Z","shell.execute_reply.started":"2025-05-05T10:11:13.261436Z","shell.execute_reply":"2025-05-05T10:11:13.265643Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Конвертируем numpy массивы в тензоры PyTorch\nX_train_tensor = torch.tensor(X_train, dtype=torch.float32)\nX_cv_tensor = torch.tensor(X_cv, dtype=torch.float32)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:11:18.978366Z","iopub.execute_input":"2025-05-05T10:11:18.978668Z","iopub.status.idle":"2025-05-05T10:11:19.045823Z","shell.execute_reply.started":"2025-05-05T10:11:18.978648Z","shell.execute_reply":"2025-05-05T10:11:19.044982Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Конвертируем numpy массивы в тензоры PyTorch\nX_test_tensor = torch.tensor(X_test, dtype=torch.float32)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:11:26.318597Z","iopub.execute_input":"2025-05-05T10:11:26.319256Z","iopub.status.idle":"2025-05-05T10:11:26.380521Z","shell.execute_reply.started":"2025-05-05T10:11:26.319235Z","shell.execute_reply":"2025-05-05T10:11:26.379380Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import torch.optim as optim\nfrom torch.utils.data import TensorDataset, DataLoader\n\n# Создадим датасеты и загрузчики\ntrain_dataset = TensorDataset(X_train_tensor, y_train_tensor)\ncv_dataset = TensorDataset(X_cv_tensor, y_cv_tensor)\n\ntrain_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)\ncv_loader = DataLoader(cv_dataset, batch_size=64)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:11:31.118453Z","iopub.execute_input":"2025-05-05T10:11:31.119254Z","iopub.status.idle":"2025-05-05T10:11:31.123203Z","shell.execute_reply.started":"2025-05-05T10:11:31.119230Z","shell.execute_reply":"2025-05-05T10:11:31.122657Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# # Последовательная нейронная сеть\n# model1 = nn.Sequential(\n#     nn.Linear(784, 128),\n#     nn.ReLU(),\n#     nn.Linear(128, 10)  # Выходной слой для 10 классов\n# )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:12:35.522767Z","iopub.execute_input":"2025-05-05T10:12:35.523104Z","iopub.status.idle":"2025-05-05T10:12:35.526824Z","shell.execute_reply.started":"2025-05-05T10:12:35.523070Z","shell.execute_reply":"2025-05-05T10:12:35.526190Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class SimpleNN(nn.Module):\n    def __init__(self):\n        super(SimpleNN, self).__init__()\n        self.fc1 = nn.Linear(784, 128)\n        self.relu = nn.ReLU()\n        self.fc2 = nn.Linear(128, 10)\n    \n    def forward(self, x):\n        x = self.fc1(x)\n        x = self.relu(x)\n        x = self.fc2(x)  # logits\n        return x\n\nmodel = SimpleNN()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:12:37.769630Z","iopub.execute_input":"2025-05-05T10:12:37.770314Z","iopub.status.idle":"2025-05-05T10:12:37.775637Z","shell.execute_reply.started":"2025-05-05T10:12:37.770294Z","shell.execute_reply":"2025-05-05T10:12:37.775056Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Используем кросс-энтропию — она ожидает метки в виде индексов классов,\n# поэтому нужно преобразовать one-hot в индексы классов:\ny_train_labels = torch.argmax(y_train_tensor, dim=1)\ny_cv_labels = torch.argmax(y_cv_tensor, dim=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:14:41.661076Z","iopub.execute_input":"2025-05-05T10:14:41.661648Z","iopub.status.idle":"2025-05-05T10:14:41.686250Z","shell.execute_reply.started":"2025-05-05T10:14:41.661616Z","shell.execute_reply":"2025-05-05T10:14:41.685727Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Для обучения нам нужны лейблы в DataLoader — поэтому\n# создадим новые датасеты с индексами классов:\ntrain_dataset_labels = TensorDataset(X_train_tensor, y_train_labels)\ncv_dataset_labels = TensorDataset(X_cv_tensor, y_cv_labels)\n\ntrain_loader_labels = DataLoader(train_dataset_labels, batch_size=64, shuffle=True)\ncv_loader_labels = DataLoader(cv_dataset_labels, batch_size=64)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:14:52.371282Z","iopub.execute_input":"2025-05-05T10:14:52.371858Z","iopub.status.idle":"2025-05-05T10:14:52.375701Z","shell.execute_reply.started":"2025-05-05T10:14:52.371830Z","shell.execute_reply":"2025-05-05T10:14:52.375100Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"criterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model.parameters(), lr=0.001)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:14:54.976419Z","iopub.execute_input":"2025-05-05T10:14:54.976701Z","iopub.status.idle":"2025-05-05T10:14:54.980874Z","shell.execute_reply.started":"2025-05-05T10:14:54.976681Z","shell.execute_reply":"2025-05-05T10:14:54.980178Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Функция для оценки точности\ndef evaluate(model, data_loader):\n    model.eval()\n    correct = 0\n    total = 0\n    with torch.no_grad():\n        for inputs, labels in data_loader:\n            outputs = model(inputs)\n            _, predicted = torch.max(outputs.data, 1)\n            total += labels.size(0)\n            correct += (predicted == labels).sum().item()\n    return correct / total","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:16:42.248887Z","iopub.execute_input":"2025-05-05T10:16:42.249541Z","iopub.status.idle":"2025-05-05T10:16:42.254153Z","shell.execute_reply.started":"2025-05-05T10:16:42.249520Z","shell.execute_reply":"2025-05-05T10:16:42.253542Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Обучение модели\nnum_epochs = 10\n\nfor epoch in range(num_epochs):\n    model.train()\n    running_loss = 0.0\n    for inputs, labels in train_loader_labels:\n        optimizer.zero_grad()\n        outputs = model(inputs)\n        loss = criterion(outputs, labels)\n        loss.backward()\n        optimizer.step()\n        \n        running_loss += loss.item() * inputs.size(0)\n    \n    epoch_loss = running_loss / len(train_loader_labels.dataset)\n    train_acc = evaluate(model, train_loader_labels)\n    cv_acc = evaluate(model, cv_loader_labels)\n\n    print(f\"Epoch {epoch+1}/{num_epochs}, Loss: {epoch_loss:.4f}, Train Acc: {train_acc:.4f}, CV Acc: {cv_acc:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:16:51.534335Z","iopub.execute_input":"2025-05-05T10:16:51.534945Z","iopub.status.idle":"2025-05-05T10:17:19.609425Z","shell.execute_reply.started":"2025-05-05T10:16:51.534923Z","shell.execute_reply":"2025-05-05T10:17:19.608717Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"torch.save(model.state_dict(), 'model.pth')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:17:19.610733Z","iopub.execute_input":"2025-05-05T10:17:19.611358Z","iopub.status.idle":"2025-05-05T10:17:19.616066Z","shell.execute_reply.started":"2025-05-05T10:17:19.611333Z","shell.execute_reply":"2025-05-05T10:17:19.615326Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Переводим модель в режим оценки\nmodel.eval()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:17:19.616740Z","iopub.execute_input":"2025-05-05T10:17:19.616962Z","iopub.status.idle":"2025-05-05T10:17:19.631767Z","shell.execute_reply.started":"2025-05-05T10:17:19.616945Z","shell.execute_reply":"2025-05-05T10:17:19.631067Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Получаем предсказания на тестовых данных\nwith torch.no_grad():\n    outputs = model(X_test_tensor)\n    _, predicted = torch.max(outputs.data, 1)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:17:19.633084Z","iopub.execute_input":"2025-05-05T10:17:19.633270Z","iopub.status.idle":"2025-05-05T10:17:19.685137Z","shell.execute_reply.started":"2025-05-05T10:17:19.633256Z","shell.execute_reply":"2025-05-05T10:17:19.684545Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Создаем DataFrame для отправки\n# Предполагаем, что ImageId начинается с 1 и идет до len(predicted)\nsubmission_df = pd.DataFrame({\n    'ImageId': range(1, len(predicted) + 1),  # Номера изображений от 1 до N\n    'Label': predicted.numpy()  # Предсказанные классы\n})\n\n# Сохраняем DataFrame в CSV файл для отправки на Kaggle\nsubmission_df.to_csv('submission.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:17:19.685874Z","iopub.execute_input":"2025-05-05T10:17:19.686133Z","iopub.status.idle":"2025-05-05T10:17:19.718346Z","shell.execute_reply.started":"2025-05-05T10:17:19.686109Z","shell.execute_reply":"2025-05-05T10:17:19.717872Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Сохраняем DataFrame в CSV файл для отправки на Kaggle\nsubmission_df.to_csv('submission.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-05T10:17:19.718940Z","iopub.execute_input":"2025-05-05T10:17:19.719109Z","iopub.status.idle":"2025-05-05T10:17:19.741470Z","shell.execute_reply.started":"2025-05-05T10:17:19.719096Z","shell.execute_reply":"2025-05-05T10:17:19.741012Z"}},"outputs":[],"execution_count":null}]}