{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Изучаем сверточные нейронные сети

","metadata":{}},{"cell_type":"markdown","source":"В этой работе Вам предстоит познакомиться с применением сверточных нейронных сетей к задачам классификации, в частности - к задаче классификации кошек. Вам необходимо дополнить этот ноутбук, чтобы модифицированная представленная или выбранная Вами модели показали хороший результат в поиске на картинках нашей грациозной кошки из трешки. Для получения оценки необходимо набрать значение метрики $f_1$ >0.91.","metadata":{}},{"cell_type":"markdown","source":"

\n \n

","metadata":{}},{"cell_type":"markdown","source":"\n

Оглавление

","metadata":{}},{"cell_type":"markdown","source":"* [Импорт библиотек](#step1)\n* [Работа с данными](#step2)\n* [Построение модели](#step3)\n* [Пример использования модели](#step4)\n* [Kaggle submission](#step5)\n* [HW and tips](#step6)","metadata":{}},{"cell_type":"markdown","source":"\n## Импорт библиотек","metadata":{}},{"cell_type":"code","source":"import torch\nimport numpy as np\nimport random\ntrain_on_gpu = torch.cuda.is_available()\n\nif not train_on_gpu:\n print('CUDA is not available. Training on CPU ...')\nelse:\n print('CUDA is available! Training on GPU ...')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# детерменизм\nSEED = 0xDEAD\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntorch.random.manual_seed(SEED)\ntorch.cuda.random.manual_seed_all(SEED)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!nvidia-smi\nimport torch\ntorch.cuda.is_available()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(action='ignore', category=DeprecationWarning)\nwarnings.filterwarnings(action='ignore', category=UserWarning)\n\nimport pickle\nimport numpy as np\nfrom skimage import io\n\nfrom tqdm import tqdm, tqdm_notebook\nfrom PIL import Image\nfrom pathlib import Path\nimport glob\nfrom torchvision import transforms\nfrom multiprocessing.pool import ThreadPool\nfrom sklearn.preprocessing import LabelEncoder\nfrom torch.utils.data import Dataset, DataLoader\nimport torch.nn as nn\n\nfrom matplotlib import colors, pyplot as plt\n%matplotlib inline","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Работа с данными\n[К оглавлению](#content)","metadata":{}},{"cell_type":"code","source":"# разные режимы датасета \nDATA_MODES = ['train', 'val', 'test']\n# все изображения будут масштабированы к размеру 224x224 px\nRESCALE_SIZE = 224\n# работаем на видеокарте\nDEVICE = torch.device(\"cuda\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ниже мы используем враппер над датасетом для удобной работы. Вам стоит понимать, что происходит с LabelEncoder и с torch.Transformation. \n\nToTensor конвертирует PIL Image с параметрами в диапазоне [0, 255] (как все пиксели) в FloatTensor размера (C x H x W) [0,1] , затем производится масштабирование:\n$\\text{input} = \\frac{\\text{input} - \\mu}{\\text{standard deviation}} $,
константы - средние и дисперсии по каналам на основе ImageNet\n\n\nСтоит также отметить, что мы переопределяем метод __getitem__ для удобства работы с данной структурой данных.\n Также используется LabelEncoder для преобразования строковых меток классов в id и обратно. В описании датасета указано, что картинки разного размера, так как брались напрямую с видео, поэтому следуем привести их к одному размер (это делает метод _prepare_sample) ","metadata":{}},{"cell_type":"code","source":"class ImageDataset(Dataset):\n \"\"\"\n Датасет с картинками, который паралельно подгружает их из папок\n производит скалирование и аугментации, превращение в торчевые тензоры\n \"\"\"\n def __init__(self, files, mode, transform=None):\n super().__init__()\n # список файлов для загрузки\n self.files = sorted(files)\n # режим работы\n self.mode = mode\n self.transform = transform if transform != None else transforms.Compose([\n transforms.Resize((RESCALE_SIZE, RESCALE_SIZE)),\n transforms.ToTensor(),\n transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) \n ])\n\n if self.mode not in DATA_MODES:\n print(f\"{self.mode} is not correct; correct modes: {DATA_MODES}\")\n raise NameError\n\n self.len_ = len(self.files)\n \n self.label_encoder = LabelEncoder()\n\n if self.mode != 'test':\n self.labels = [path.parent.name for path in self.files]\n self.label_encoder.fit(self.labels)\n\n with open('label_encoder.pkl', 'wb') as le_dump_file:\n pickle.dump(self.label_encoder, le_dump_file)\n \n def __len__(self):\n return self.len_\n \n def load_sample(self, file):\n image = Image.open(file).convert('RGB')\n image.load()\n return image\n \n def __getitem__(self, index):\n # для преобразования изображений в тензоры PyTorch и нормализации входа\n x = self.load_sample(self.files[index])\n x = self.transform(x)\n if self.mode == 'test':\n return x\n else:\n label = self.labels[index]\n label_id = self.label_encoder.transform([label])\n y = label_id.item()\n return x, y","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def imshow(inp, title=None, plt_ax=plt, default=False):\n \"\"\"Imshow для тензоров\"\"\"\n inp = inp.numpy().transpose((1, 2, 0))\n mean = np.array([0.485, 0.456, 0.406])\n std = np.array([0.229, 0.224, 0.225])\n inp = std * inp + mean\n inp = np.clip(inp, 0, 1)\n plt_ax.imshow(inp)\n if title is not None:\n plt_ax.set_title(title)\n plt_ax.grid(False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TRAIN_DIR = Path('data/train')\n# TEST_DIR = Path('data/test/test')\n\nTRAIN_DIR = Path('../input/3rd-corpus-cat/data/data/train')\nTEST_DIR = Path('../input/3rd-corpus-cat/data/data/test/test')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Так как изображения были автоматически собраны, среди них могут оказаться битые. Посмотрим, есть ли такие.","metadata":{}},{"cell_type":"code","source":"imgs_names = glob.glob('../input/3rd-corpus-cat/data/data/train/kowka_ne_iz_treshki'+'\\\\*.jpg')\n\nfor imgname in imgs_names: \n img = Image.open(imgname) \n if img is None: \n print(imgname)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_val_files = sorted(list(TRAIN_DIR.rglob('*.jpg')))\ntest_files = sorted(list(TEST_DIR.rglob('*.jpg')))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain_val_labels = [path.parent.name for path in train_val_files]\ntrain_files, val_files = train_test_split(train_val_files, test_size=0.25, \\\n stratify=train_val_labels)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_dataset = ImageDataset(val_files, mode='val')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(nrows=3, ncols=3,figsize=(14, 14), \\\n sharey=True, sharex=True)\nfor fig_x in ax.flatten():\n random_characters = int(np.random.uniform(0,362))\n im_val, label = val_dataset[random_characters]\n img_label = \" \".join(map(lambda x: x.capitalize(),\\\n val_dataset.label_encoder.inverse_transform([label])[0].split('_')))\n imshow(im_val.data.cpu(), \\\n title=img_label,plt_ax=fig_x)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# transform = transforms.Compose([\n# transforms.Resize(size=(RESCALE_SIZE, RESCALE_SIZE)),\n# transforms.RandomRotation(degrees=45),\n# transforms.RandomHorizontalFlip(),\n# transforms.ColorJitter(hue=.1, saturation=.1),\n# transforms.ToTensor(),\n# transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) \n# ])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Построение модели\n[К оглавлению](#content)","metadata":{}},{"cell_type":"markdown","source":"Здесь представлена максимально простая модель для укрепления понимания и, возможно, совершения попыток сдачи submit на kaggle.\n\n*Описание слоев*:\n\n\n\n1. Вход: размерность `3x224x224` \n2. 1ый слой: выходная размерность `8x111x111`\n3. 2й слой: выходная размерность `16x54x54`\n6. выход: `2`","metadata":{}},{"cell_type":"code","source":"# Очень простая сеть\nclass SimpleCnn(nn.Module):\n \n def __init__(self, n_classes):\n super().__init__()\n self.conv1 = nn.Sequential(\n nn.Conv2d(in_channels=3, out_channels=8, kernel_size=3),\n nn.ReLU(),\n nn.MaxPool2d(kernel_size=2)\n )\n self.conv2 = nn.Sequential(\n nn.Conv2d(in_channels=8, out_channels=16, kernel_size=3),\n nn.ReLU(),\n nn.MaxPool2d(kernel_size=2)\n )\n self.out = nn.Linear(16 * 54 * 54, n_classes)\n \n \n def forward(self, x):\n x = self.conv1(x)\n x = self.conv2(x)\n x = x.view(x.size(0), -1)\n logits = self.out(x)\n return logits","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Процесс обучения модели","metadata":{}},{"cell_type":"code","source":"def eval_epoch(model, val_loader, criterion):\n model.eval()\n running_loss = 0.0\n running_corrects = 0\n processed_size = 0\n\n for inputs, labels in val_loader:\n inputs = inputs.to(DEVICE)\n labels = labels.to(DEVICE)\n\n with torch.set_grad_enabled(False):\n outputs = model(inputs)\n loss = criterion(outputs, labels)\n preds = torch.argmax(outputs, 1)\n\n running_loss += loss.item() * inputs.size(0)\n running_corrects += torch.sum(preds == labels.data)\n processed_size += inputs.size(0)\n val_loss = running_loss / processed_size\n val_acc = running_corrects.double() / processed_size\n return val_loss, val_acc\n\ndef fit_epoch(model, tepoch, criterion, optimizer):\n running_loss = 0.0\n running_corrects = 0\n processed_data = 0\n \n for inputs, labels in tepoch:\n inputs = inputs.to(DEVICE)\n labels = labels.to(DEVICE)\n optimizer.zero_grad()\n outputs = model(inputs)\n loss = criterion(outputs, labels)\n loss.backward()\n optimizer.step()\n preds = torch.argmax(outputs, 1)\n running_loss += loss.item() * inputs.size(0)\n running_corrects += torch.sum(preds == labels.data)\n processed_data += inputs.size(0)\n tepoch.set_postfix(loss=loss.item(),\n accuracy=100. * running_corrects.cpu().numpy() / processed_data)\n\n \n train_loss = running_loss / processed_data\n train_acc = running_corrects.cpu().numpy() / processed_data\n return train_loss, train_acc\n\ndef train(train_files, val_files,\n model, epochs, batch_size, \n optimizer=None, criterion=None, scheduler=None):\n train_loader = DataLoader(train_files, batch_size=batch_size, shuffle=True)\n val_loader = DataLoader(val_files, batch_size=batch_size, shuffle=False)\n\n history = []\n log_template = \"\\nEpoch {ep:03d} train_loss: {t_loss:0.4f} \\\n val_loss {v_loss:0.4f} train_acc {t_acc:0.4f} val_acc {v_acc:0.4f}\\n\"\n optimizer = torch.optim.Adam(model.parameters()) if optimizer == None else optimizer\n criterion = nn.CrossEntropyLoss() if criterion == None else criterion\n\n for epoch in range(epochs):\n with tqdm(train_loader, unit=\"batch\") as tepoch:\n tepoch.set_description(f\"Epoch {epoch + 1}\")\n train_loss, train_acc = fit_epoch(model, tepoch, criterion, optimizer)\n val_loss, val_acc = eval_epoch(model, val_loader, criterion)\n history.append((train_loss, train_acc, val_loss, val_acc))\n if scheduler != None:\n scheduler.step()\n tepoch.update(1)\n tqdm.write(log_template.format(ep=epoch+1, t_loss=train_loss,\\\n v_loss=val_loss, t_acc=train_acc, v_acc=val_acc))\n return history","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Функция предсказания","metadata":{}},{"cell_type":"code","source":"def predict(model, test_loader):\n with torch.no_grad():\n logits = []\n \n for inputs in test_loader:\n inputs = inputs.to(DEVICE)\n model.eval()\n outputs = model(inputs).cpu()\n logits.append(outputs)\n \n probs = nn.functional.softmax(torch.cat(logits), dim=-1).numpy()\n return probs","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Создадим модель","metadata":{}},{"cell_type":"code","source":"n_classes = len(np.unique(train_val_labels))\nsimple_cnn = SimpleCnn(n_classes).to(DEVICE)\nprint(\"Классов для классификации: {}\".format(n_classes))\nprint(simple_cnn)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if val_dataset is None:\n val_dataset = ImageDataset(val_files, mode='val')\n \ntrain_dataset = ImageDataset(train_files, mode='train')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Этап обучения модели","metadata":{}},{"cell_type":"code","source":"history = train(train_dataset, val_dataset, model=simple_cnn, epochs=2, batch_size=32)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Построение графиков поведения метрики и функции ошибки в течение обучения","metadata":{}},{"cell_type":"code","source":"def plot_history(history: list):\n fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(16,10))\n loss, acc, val_loss, val_acc = zip(*history) \n try:\n val_acc = [i.cpu().numpy() for i in val_acc]\n except ValueError:\n print('Something went wrong')\n ax[0].plot(loss, label='train_loss')\n ax[0].plot(val_loss, label='val_loss')\n ax[1].plot(acc, label='train_acc')\n ax[1].plot(val_acc, label='val_acc')\n ax[0].set_title('Loss')\n ax[1].set_title('Accuracy')\n\n ax[0].legend(loc='best')\n ax[1].legend(loc='best')\n \n ax[0].set_xlabel(\"epochs\")\n ax[0].set_ylabel(\"loss\")\n \n ax[1].set_xlabel(\"epochs\")\n ax[1].set_ylabel(\"acc\")\n plt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_history(history=history)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Пример использования модели\n[К оглавлению](#content)","metadata":{}},{"cell_type":"markdown","source":"Функция для предсказания одного объекта (изображения)","metadata":{}},{"cell_type":"code","source":"def predict_one_sample(model, inputs, device=DEVICE):\n \"\"\"Предсказание, для одной картинки\"\"\"\n with torch.no_grad():\n inputs = inputs.to(device)\n model.eval()\n logit = model(inputs).cpu()\n probs = torch.nn.functional.softmax(logit, dim=-1).numpy()\n return probs","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Пример предсказания для одного объекта","metadata":{}},{"cell_type":"code","source":"random_characters = int(np.random.uniform(0,len(val_dataset)))\nex_img, true_label = val_dataset[random_characters]\nprobs_im = predict_one_sample(simple_cnn, ex_img.unsqueeze(0))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Вычислим целевую метрику($f_1$) на 20 рандомных объектах валидационного датасета","metadata":{}},{"cell_type":"code","source":"idxs = list(map(int, np.random.uniform(0,len(val_dataset), 20)))\nimgs = [val_dataset[id][0].unsqueeze(0) for id in idxs]\n\nprobs_ims = predict(simple_cnn, imgs)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Используем обратное преобразование label_encoder, чтобы вернуть строковые значения классов. Альтернативное использование - метод `inverse_transform`","metadata":{}},{"cell_type":"code","source":"label_encoder = pickle.load(open(\"label_encoder.pkl\", 'rb'))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = np.argmax(probs_ims,-1)\n\nactual_labels = [label_encoder.classes_[val_dataset[id][1]] for id in idxs]\n\npreds_class = [label_encoder.classes_[i] for i in y_pred]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import f1_score\nf1_score(actual_labels, preds_class, average='micro')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Сделаем визуализацию (взято из курса `DLS`)","metadata":{}},{"cell_type":"code","source":"import matplotlib.patches as patches\nfrom matplotlib.font_manager import FontProperties\n\ndef beautiful_plot(dataset: ImageDataset, model=simple_cnn):\n fig, ax = plt.subplots(nrows=3, ncols=3,figsize=(14, 14), \\\n sharey=True, sharex=True)\n for fig_x in ax.flatten():\n random_characters = int(np.random.uniform(0,len(dataset)))\n im_val, label = dataset[random_characters]\n img_label = \" \".join(map(lambda x: x.capitalize(),\\\n dataset.label_encoder.inverse_transform([label])[0].split('_')))\n \n \n\n imshow(im_val.data.cpu(), \\\n title=img_label,plt_ax=fig_x)\n \n actual_text = \"Actual : {}\".format(img_label)\n \n fig_x.add_patch(patches.Rectangle((0, 53),86,35,color='white'))\n font0 = FontProperties()\n font = font0.copy()\n font.set_family(\"fantasy\")\n prob_pred = predict_one_sample(simple_cnn, im_val.unsqueeze(0))\n predicted_proba = np.max(prob_pred)*100\n y_pred = np.argmax(prob_pred)\n \n predicted_label = label_encoder.classes_[y_pred]\n predicted_label = predicted_label[:len(predicted_label)//2] + '\\n' + predicted_label[len(predicted_label)//2:]\n predicted_text = \"{} : {:.0f}%\".format(predicted_label,predicted_proba)\n \n fig_x.text(1, 59, predicted_text , horizontalalignment='left', fontproperties=font,\n verticalalignment='top',fontsize=10, color='black',fontweight='bold')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"beautiful_plot(val_dataset, simple_cnn)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Сделаем функцию для быстрого поиска значения целевой метрики","metadata":{}},{"cell_type":"code","source":"def f1_scoring(dataset: ImageDataset, model, count=50):\n idxs = list(map(int, np.random.uniform(0, len(dataset), count)))\n imgs = [dataset[id][0].unsqueeze(0) for id in idxs]\n\n probs_ims = predict(model, imgs)\n y_pred = np.argmax(probs_ims,-1)\n\n actual_labels = [label_encoder.classes_[dataset[id][1]] for id in idxs]\n\n preds_class = [label_encoder.classes_[i] for i in y_pred]\n return f1_score(actual_labels, preds_class, average='micro')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_scoring(val_dataset, simple_cnn, count=10)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Kaggle Submission\n[К оглавлению](#content)\n

","metadata":{}},{"cell_type":"markdown","source":"Теперь зададимся вопросом, как сделать Submission на Kaggle. Для этого загрузим тестовые данные. Сделаем предсказания и сохраним всё согласно `sample_submission.csv`","metadata":{}},{"cell_type":"code","source":"test_dataset = ImageDataset(test_files, mode=\"test\")\ntest_loader = DataLoader(test_dataset, shuffle=False, batch_size=64)\nprobs = predict(simple_cnn, test_loader)\n\n\npreds = label_encoder.inverse_transform(np.argmax(probs, axis=1))\ntest_filenames = [path.name for path in test_dataset.files]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"my_submit = pd.DataFrame({'Image_id': test_filenames, 'Expected': preds})\nmy_submit.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Обернем выше сделанное в функцию","metadata":{}},{"cell_type":"code","source":"def to_submit(test_files: list, test_dataset: ImageDataset, model, save=False) -> pd.DataFrame:\n test_dataset = ImageDataset(test_files, mode=\"test\")\n test_loader = DataLoader(test_dataset, shuffle=False, batch_size=64)\n probs = predict(model, test_loader)\n\n\n preds = label_encoder.inverse_transform(np.argmax(probs, axis=1))\n test_filenames = [path.name for path in test_dataset.files]\n result = pd.DataFrame({'Id': test_filenames, 'Expected': preds})\n if save:\n result.to_csv('submission.csv', index=False)\n return result","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = to_submit(test_files=test_files, test_dataset=test_dataset, model=simple_cnn, save=True)\nsubmission","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## HW and tips\n[К оглавлению](#content)","metadata":{}},{"cell_type":"markdown","source":"Что можно сделать для улучшения качества классификации:\n* Поменять гиперпараметры модели (количество слоев, функции активации, размеры батчей, количество эпох);\n* Добавить dropout-регуляризацию и батч-нормализацию;\n* Использовать [другие](https://openreview.net/pdf?id=ryQu7f-RZ) оптимизаторы (SGD, AdamW, Adadelta и прочие);\n* Добавить аугментации в данные, так как их мало;\n* Можно реализовать Transfer Learning для предварительно обученных моделей.\n\nНапомню, что Вашей минимальной целью является результат $f_1$ не ниже 0.91. Как максимум, первое место в leaderboard. Всем удачи!","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}