{"cells":[{"metadata":{},"cell_type":"markdown","source":"2020010216 陈润泽 高级机器学习 期末实验\n\n# MNIST ConvNet 实验\n\n**Goals:**\n\n1. understand Neural Networks and how they are arranged in layered architectures\n1. implement dropout to regularize networks\n1. effectively cross-validate and find the best hyperparameters for Neural Network architecture\n1. understand the architecture of Convolutional Neural Networks and train gain experience with training these models on data\n\n**Hints:**\n\n1. Implement Dropout and explore its effects on model generalization\n1. You will implement several CNN toolkits that are commonly used. You will train a (shallow) convolutional network on MNIST, and it will then be up to you to train the best network that you can.\n\n**Extra:**\n\n1. In the process of training your network, you should feel free to implement anything that you want to get better performance. You can modify the solver, implement additional layers, use different types of regularization, use an ensemble of models, or anything else that comes to mind."},{"metadata":{},"cell_type":"markdown","source":"## 前期准备与数据处理\n\n显示数据文件目录："},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"引入PyTorch库："},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import torch\nimport torchvision as vision\nfrom torch import nn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"读取原始文件："},{"metadata":{"trusted":true},"cell_type":"code","source":"train_source_frame = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest_source_frame = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"读入所有原始数据并调整尺寸："},{"metadata":{"trusted":true},"cell_type":"code","source":"train_label = torch.tensor(train_source_frame.loc[:, 'label'].to_numpy()).reshape(-1).long()\ntrain_features = torch.tensor(train_source_frame.loc[:, tuple(map(lambda idx: f'pixel{idx}', range(train_source_frame.shape[1] - 1)))].to_numpy()).reshape(-1, 1, 28, 28).float()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"读入标签数据尺寸与训练特征尺寸："},{"metadata":{"trusted":true},"cell_type":"code","source":"train_label.shape, train_features.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"引入`pytorch-ignite`库控制训练过程："},{"metadata":{"trusted":true},"cell_type":"code","source":"from ignite.engine import Events, create_supervised_trainer, create_supervised_evaluator\nfrom ignite.metrics import Loss, Accuracy\nfrom ignite.contrib.handlers.tqdm_logger import ProgressBar","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 进行训练\n\n设定公共参数：\n\n- 训练集/总数据集比例\n- 批大小\n- 学习率"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ratio = 0.9\nbatch_size = 2048\nlearning_rate = 5e-5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"封装`Dataset`："},{"metadata":{"trusted":true},"cell_type":"code","source":"class MNISTDataset(torch.utils.data.Dataset):\n    def __init__(self, features, label):\n        assert features.shape[0] == label.shape[0]\n        self._features = features\n        self._label = label\n        \n    def __getitem__(self, item):\n        return self._features[item], self._label[item]\n    \n    def __len__(self):\n        return self._label.shape[0]\n\ntrain_slice = slice(0, int(train_label.shape[0] * train_ratio))\nvalid_slice = slice(int(train_label.shape[0] * train_ratio), None)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"准备训练集与验证集\n\n训练集数据量与验证集数据量："},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = MNISTDataset(train_features[train_slice], train_label[train_slice])\nvalid_dataset = MNISTDataset(train_features[valid_slice], train_label[valid_slice])\nlen(train_dataset), len(valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"组装`DataLoader`："},{"metadata":{"trusted":true},"cell_type":"code","source":"train_loader = torch.utils.data.DataLoader(\n    train_dataset,\n    batch_size=batch_size,\n    shuffle=True,\n    pin_memory=True\n)\n\nvalid_loader = torch.utils.data.DataLoader(\n    valid_dataset,\n    batch_size=batch_size,\n    shuffle=True,\n    pin_memory=True\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 模型结构\n\n本实验模型结构如下所示："},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install torchsummary\nfrom torchsummary import summary ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = nn.Sequential(\n    nn.Conv2d(1, 32, 3, bias=False),\n    nn.BatchNorm2d(32),\n    nn.SELU(inplace=True),\n    nn.Conv2d(32, 128, 1, bias=False),\n    nn.BatchNorm2d(128),\n    nn.SELU(inplace=True),\n    nn.Conv2d(128, 192, 5, bias=False),\n    nn.BatchNorm2d(192),\n    nn.SELU(inplace=True),\n    nn.Conv2d(192, 256, 3, bias=False),\n    nn.BatchNorm2d(256),\n    nn.SELU(inplace=True),\n    nn.Conv2d(256, 32, 20, bias=False),\n    nn.BatchNorm2d(32),\n    nn.SELU(inplace=True),\n    nn.Conv2d(32, 10, 1, bias=True),\n    nn.Flatten(),\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"summary(model.cuda(), (1, 28, 28))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 开始训练"},{"metadata":{"trusted":true},"cell_type":"code","source":"trainer = create_supervised_trainer(\n    model=model,\n    optimizer=torch.optim.Adam(model.parameters(), lr=learning_rate),\n    loss_fn=torch.nn.CrossEntropyLoss(),\n    device=torch.device('cuda:0')\n)\nevaluator = create_supervised_evaluator(\n    model=model,\n    metrics={\n        'accuracy': Accuracy(),\n        'loss': Loss(torch.nn.CrossEntropyLoss())\n    },\n    device=torch.device('cuda:0')\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"训练详细过程如下："},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy = []\nloss = []\n@trainer.on(Events.EPOCH_COMPLETED)\ndef on_epoch_completed(engine):\n    evaluator.run(valid_loader)\n    accuracy.append(evaluator.state.metrics[\"accuracy\"])\n    loss.append(evaluator.state.metrics[\"loss\"])\n    print(f'Epoch {engine.state.epoch} Accuracy {evaluator.state.metrics[\"accuracy\"]} Loss {evaluator.state.metrics[\"loss\"]}')\n\npbar = ProgressBar()\npbar.attach(trainer)\ntrainer.run(train_loader, 50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from matplotlib import pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Accuracy-Loss变化趋势如下："},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(accuracy)\nplt.plot(loss)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"max(accuracy), min(loss)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.cuda.empty_cache()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_features = torch.tensor(test_source_frame.loc[:, tuple(map(lambda idx: f'pixel{idx}', range(test_source_frame.shape[1])))].to_numpy()).reshape(-1, 1, 28, 28).float()\ntest_dataset = torch.utils.data.TensorDataset(test_features)\ntest_loader = torch.utils.data.DataLoader(\n    test_dataset,\n    batch_size=256,\n    shuffle=False,\n    pin_memory=True\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 实验结果\n\n保存实验结果并提交Kaggle平台进行排名："},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import tqdm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.cuda.empty_cache()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"outputs = []\nmodel.eval()\nfor batch in tqdm.tqdm(test_loader):\n    with torch.no_grad():\n        output = model(batch[0].to('cuda:0'))\n        batch[0] = batch[0].to('cpu')\n        outputs.append(output.to('cpu'))\n        del output, batch\n        if len(outputs) % 1 == 0:\n            torch.cuda.empty_cache()\n            gc.collect()\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = torch.cat(outputs).argmax(axis=1).numpy()\nsubmission = pd.DataFrame({'ImageId': list(range(1, target.shape[0] + 1)), 'Label': target})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv('/kaggle/working/submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model, '/kaggle/working/model.pt')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}