{"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":5441,"databundleVersionId":38425,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"},"papermill":{"default_parameters":{},"duration":2178.451466,"end_time":"2024-07-14T11:02:54.055227","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-07-14T10:26:35.603761","version":"2.5.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## モジュールのインポート","metadata":{"papermill":{"duration":0.006175,"end_time":"2024-07-14T10:26:38.326236","exception":false,"start_time":"2024-07-14T10:26:38.320061","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import numpy as np \nimport glob\nimport pandas as pd\nimport shutil\nimport os\nimport zipfile\nfrom sklearn.model_selection import train_test_split\nimport time\nimport copy\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.utils.data import DataLoader\nfrom torch.utils import data\nfrom torchvision import datasets, models, transforms\nfrom PIL import Image\nfrom tqdm import tqdm \nimport re, csv, sys\nimport matplotlib.pyplot as plt ","metadata":{"execution":{"iopub.status.busy":"2024-07-22T17:14:29.032919Z","iopub.execute_input":"2024-07-22T17:14:29.033441Z","iopub.status.idle":"2024-07-22T17:14:29.041699Z","shell.execute_reply.started":"2024-07-22T17:14:29.033401Z","shell.execute_reply":"2024-07-22T17:14:29.040437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## データの解凍","metadata":{"papermill":{"duration":0.005658,"end_time":"2024-07-14T10:26:46.494717","exception":false,"start_time":"2024-07-14T10:26:46.489059","status":"completed"},"tags":[]}},{"cell_type":"code","source":"working_dir = '/kaggle/working'\n\n# フォルダ内のすべての内容を削除\nfor item in os.listdir(working_dir):\n    item_path = os.path.join(working_dir, item)\n    shutil.rmtree(item_path, ignore_errors=True)\n    \n# zipファイルを自分の作業するディレクトリに解凍\nimport zipfile\nwith zipfile.ZipFile('/kaggle/input/dogs-vs-cats-redux-kernels-edition/train.zip') as existing_zip:\n    existing_zip.extractall()\nwith zipfile.ZipFile('/kaggle/input/dogs-vs-cats-redux-kernels-edition/test.zip') as existing_zip:\n    existing_zip.extractall()\n\n\n# 自分の作業するディレクトリを確認\nfiles = glob.glob('/kaggle/working/*')\nfor file in files:\n    print(file)\n\n# フォルダの中身を確認\ntrain_files = sorted(glob.glob('/kaggle/working/train/*'))\nprint('trainフォルダ中のtrain用の画像', len(train_files), '枚')\n\ntest_files = sorted(glob.glob('/kaggle/working/test/*'))\nprint('testフォルダ中のtest用の画像', len(test_files), '枚')\n\n","metadata":{"papermill":{"duration":15.347723,"end_time":"2024-07-14T10:27:01.847998","exception":false,"start_time":"2024-07-14T10:26:46.500275","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-07-22T17:14:29.043602Z","iopub.execute_input":"2024-07-22T17:14:29.04431Z","iopub.status.idle":"2024-07-22T17:14:40.602241Z","shell.execute_reply.started":"2024-07-22T17:14:29.044281Z","shell.execute_reply":"2024-07-22T17:14:40.600884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 犬と猫をフォルダ分類","metadata":{}},{"cell_type":"code","source":"# 犬と猫の画像をそれぞれ保存するファイルパスを定義\ntrain_dog = '/kaggle/working/train/dog'\ntrain_cat = '/kaggle/working/train/cat'\ntrain_dir = '/kaggle/working/train'\n\n# ディレクトリ作成\nos.makedirs(train_dog, exist_ok=True)\nos.makedirs(train_cat, exist_ok=True)\n    \n# 犬と猫の画像を分ける\nfor file in train_files:\n    file_name = os.path.basename(file)\n    if 'cat' in file_name:\n        shutil.move(file, os.path.join(train_cat, file_name))\n    else:\n        shutil.move(file, os.path.join(train_dog, file_name))\n\n# 画像数の確認\nprint('学習犬画像枚数:', len(glob.glob(train_dog + '/*.jpg')))\nprint('学習猫画像枚数:', len(glob.glob(train_cat + '/*.jpg')))\n","metadata":{"execution":{"iopub.status.busy":"2024-07-22T17:14:40.603443Z","iopub.execute_input":"2024-07-22T17:14:40.603764Z","iopub.status.idle":"2024-07-22T17:14:41.619956Z","shell.execute_reply.started":"2024-07-22T17:14:40.603737Z","shell.execute_reply":"2024-07-22T17:14:41.618926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## データを学習用とバリデーションに分類","metadata":{}},{"cell_type":"code","source":"val = '/kaggle/working/val'\nval_dog = '/kaggle/working/val/dog'\nval_cat = '/kaggle/working/val/cat'\n\n# バリデーションデータ用のディレクトリを作成\nos.makedirs(val_dog, exist_ok=True)\nos.makedirs(val_cat, exist_ok=True)\n\n# 犬の訓練データの一部をバリデーションデータ用に分割\ndog_files = glob.glob('/kaggle/working/train/dog/*.jpg')\ndog_train, dog_val = train_test_split(dog_files, test_size=0.2, random_state=42)\n\n# 画像の一部をバリデーションフォルダに移動させる\nfor file in dog_val:\n    file_name = os.path.basename(file)\n    shutil.move(file, os.path.join(val_dog, file_name))\n\n# 猫の訓練データの一部をバリデーションデータ用に分割\ncat_files = glob.glob('/kaggle/working/train/cat/*.jpg')\ncat_train, cat_val = train_test_split(cat_files, test_size=0.2, random_state=42)\n\n# バリデーションフォルダに移動させる\nfor file in cat_val:\n    file_name = os.path.basename(file)\n    shutil.move(file, os.path.join(val_cat, file_name))\n    \nprint('学習用犬画像:', len(glob.glob('/kaggle/working/train/dog/*.jpg')))\nprint('学習用猫画像:', len(glob.glob('/kaggle/working/train/cat/*.jpg')))\nprint('バリデーション用犬画像:', len(glob.glob('/kaggle/working/val/dog/*.jpg')))\nprint('バリデーション用猫画像:', len(glob.glob('/kaggle/working/val/cat/*.jpg')))\n","metadata":{"execution":{"iopub.status.busy":"2024-07-22T17:14:41.622267Z","iopub.execute_input":"2024-07-22T17:14:41.622582Z","iopub.status.idle":"2024-07-22T17:14:42.026388Z","shell.execute_reply.started":"2024-07-22T17:14:41.622556Z","shell.execute_reply":"2024-07-22T17:14:42.025404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 学習用データセットの作成","metadata":{}},{"cell_type":"code","source":"# 画像サイズ、ImageNetデータセットの平均値および標準偏差を設定\nsize = 224  # 画像サイズ\nmean = (0.485, 0.456, 0.406)  # ImageNetデータセットの平均値\nstd = (0.229, 0.224, 0.225)  # ImageNetデータセットの標準偏差\n\n# 学習用および検証用のデータ変換を定義\ndata_transforms = {\n    'train': transforms.Compose([\n        transforms.Resize((size, size)),  # 画像を指定サイズにリサイズ\n        transforms.RandomHorizontalFlip(),  # ランダムに水平反転\n        transforms.RandomVerticalFlip(),  # ランダムに垂直反転\n        transforms.ToTensor(),  # テンソルに変換\n        transforms.Normalize(mean, std)  # 平均と標準偏差を用いて正規化\n    ]),\n    'val': transforms.Compose([\n        transforms.Resize((size, size)),  # 画像を指定サイズにリサイズ\n        transforms.ToTensor(),  # テンソルに変換\n        transforms.Normalize(mean, std)  # 平均と標準偏差を用いて正規化\n    ]),\n}\n\n# トレーニング用および検証用のデータセットを作成\ntrain_dataset = datasets.ImageFolder(root='/kaggle/working/train', transform=data_transforms['train'])\nval_dataset = datasets.ImageFolder(root='/kaggle/working/val', transform=data_transforms['val'])\n\n# データセットをバッチ処理するためのデータローダーを作成\nbatch_size = 32\ntrain_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\nval_dataloader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)\n\n# トレーニングと検証のデータローダーを一つの辞書型でまとめる\ndataloader_dict = {'train': train_dataloader, 'val': val_dataloader}","metadata":{"papermill":{"duration":0.015791,"end_time":"2024-07-14T10:27:01.870477","exception":false,"start_time":"2024-07-14T10:27:01.854686","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-07-22T17:14:42.027771Z","iopub.execute_input":"2024-07-22T17:14:42.028183Z","iopub.status.idle":"2024-07-22T17:14:42.16734Z","shell.execute_reply.started":"2024-07-22T17:14:42.028146Z","shell.execute_reply":"2024-07-22T17:14:42.166254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## デバイスの設定とモデル構築","metadata":{}},{"cell_type":"code","source":"# デバイスの設定 (GPUが利用可能な場合はGPUを使用)\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\n# 事前学習済みのVGG16モデルをロード\nmodel = models.vgg16(pretrained=True)\n\n# 最後の全結合層をクラス数に合わせて変更\nmodel.classifier[6] = nn.Linear(4096, 2)\n\n# すべてのパラメータを更新不可に設定\nfor param in model.parameters():\n    param.requires_grad = False\n\n# 新たに追加した全結合層のみ更新可能に設定\nfor param in model.classifier[6].parameters():\n    param.requires_grad = True\n\n# モデルをデバイスに転送\nmodel = model.to(device)\n\n# 損失関数と最適化関数の設定\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model.classifier[6].parameters(), lr=0.0001)\n","metadata":{"papermill":{"duration":0.014851,"end_time":"2024-07-14T10:27:01.890755","exception":false,"start_time":"2024-07-14T10:27:01.875904","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-07-22T17:14:42.168608Z","iopub.execute_input":"2024-07-22T17:14:42.168939Z","iopub.status.idle":"2024-07-22T17:14:44.289558Z","shell.execute_reply.started":"2024-07-22T17:14:42.168911Z","shell.execute_reply":"2024-07-22T17:14:44.2886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## エポック数（学習回数）の設定","metadata":{"papermill":{"duration":0.00541,"end_time":"2024-07-14T10:26:45.365757","exception":false,"start_time":"2024-07-14T10:26:45.360347","status":"completed"},"tags":[]}},{"cell_type":"code","source":"num_epochs = 15","metadata":{"execution":{"iopub.status.busy":"2024-07-22T17:14:44.29109Z","iopub.execute_input":"2024-07-22T17:14:44.291498Z","iopub.status.idle":"2024-07-22T17:14:44.296439Z","shell.execute_reply.started":"2024-07-22T17:14:44.291461Z","shell.execute_reply":"2024-07-22T17:14:44.295408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 学習工程","metadata":{}},{"cell_type":"code","source":"# 学習開始時刻を記録\nsince = time.time()  \n\n# 最良モデルの初期化\nbest_model_wts = copy.deepcopy(model.state_dict())  \nbest_loss = float('inf')  # 最小損失の初期化\n\n# 損失と精度の履歴を保持するための辞書\nhistory = {'train_loss': [], 'val_loss': [], 'train_acc': [], 'val_acc': []}\n\nprint('Training started')\n\n# 各エポックの進捗を表示しながら学習\nfor epoch in range(num_epochs):  \n    print(f'Epoch {epoch+1}/{num_epochs}')\n    print('-' * 20)\n    \n    for phase in ['train', 'val']:\n        # 学習と検証を交互に行う\n        if phase == 'train':\n            model.train()  # 学習フェーズ\n        else:\n            model.eval()  # 検証フェーズ\n            \n        epoch_loss = 0.0\n        epoch_corrects = 0\n        \n        # データローダーからバッチを取得\n        for inputs, labels in tqdm(dataloader_dict[phase]):\n            inputs = inputs.to(device)\n            labels = labels.to(device)\n            optimizer.zero_grad()\n            \n            with torch.set_grad_enabled(phase == 'train'):\n                outputs = model(inputs)  # 出力を算出\n                _, preds = torch.max(outputs, 1)  # 予測値の取得\n                loss = criterion(outputs, labels)  # 損失の計算\n                \n                if phase == 'train':\n                    loss.backward()  # 逆伝播で学習\n                    optimizer.step()\n                    \n                # バッチごとの損失と精度を累積    \n                epoch_loss += loss.item() * inputs.size(0)\n                epoch_corrects += torch.sum(preds == labels.data)\n        \n        # エポックごとの損失と精度の計算\n        epoch_loss = epoch_loss / len(dataloader_dict[phase].dataset)\n        epoch_acc = epoch_corrects.double() / len(dataloader_dict[phase].dataset)\n        \n        print(f'{phase} Loss: {epoch_loss:.4f} Acc: {epoch_acc:.4f}')\n        \n        # 履歴に保存\n        history[f'{phase}_loss'].append(epoch_loss)\n        history[f'{phase}_acc'].append(epoch_acc.cpu().numpy())\n        \n        # 最良モデルの保存\n        if phase == 'val' and epoch_loss < best_loss:\n            best_loss = epoch_loss\n            best_model_wts = copy.deepcopy(model.state_dict())\n            \n# 学習の総時間を計算\ntime_elapsed = time.time() - since  \nprint(f'Training complete in {time_elapsed // 60:.0f}m {time_elapsed % 60:.0f}s')\nprint(f'Best val Loss: {best_loss:.4f}')\n\n# 最良の重みを読み込む\nmodel.load_state_dict(best_model_wts)\n","metadata":{"papermill":{"duration":0.088571,"end_time":"2024-07-14T10:26:45.459829","exception":false,"start_time":"2024-07-14T10:26:45.371258","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-07-22T17:14:44.298253Z","iopub.execute_input":"2024-07-22T17:14:44.29867Z","iopub.status.idle":"2024-07-22T17:17:24.445425Z","shell.execute_reply.started":"2024-07-22T17:14:44.298632Z","shell.execute_reply":"2024-07-22T17:17:24.44432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 学習曲線の可視化","metadata":{"papermill":{"duration":0.616454,"end_time":"2024-07-14T11:00:16.511355","exception":false,"start_time":"2024-07-14T11:00:15.894901","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# matplotlibのインライン表示設定\n%matplotlib inline  \n\n# 図の設定\nplt.figure(figsize=(16, 6))  # 図のサイズ設定\nplt.subplots_adjust(wspace=0.3)  # サブプロット間のスペース調整\n\n# 損失の学習曲線\nplt.subplot(1, 2, 1)  # 1行2列の1番目のプロットを設定\nplt.plot(history['train_loss'], label=\"Train Loss\", marker='o')  # 学習損失のプロット\nplt.plot(history['val_loss'], label=\"Validation Loss\", marker='x')  # 検証損失のプロット\nplt.title(\"Train and Validation Loss\")  # タイトル設定\nplt.xlabel(\"Epochs\")  # x軸のラベル設定\nplt.ylabel(\"Loss\")  # y軸のラベル設定\nplt.legend()  # 凡例の表示\nplt.grid(True)  # グリッドの表示\n\n# 精度の学習曲線\nplt.subplot(1, 2, 2)  # 1行2列の2番目のプロットを設定\nplt.plot(history['train_acc'], label=\"Train Accuracy\", marker='o')  # 学習精度のプロット\nplt.plot(history['val_acc'], label=\"Validation Accuracy\", marker='x')  # 検証精度のプロット\nplt.title(\"Train and Validation Accuracy\")  # タイトル設定\nplt.xlabel(\"Epochs\")  # x軸のラベル設定\nplt.ylabel(\"Accuracy\")  # y軸のラベル設定\nplt.legend()  # 凡例の表示\nplt.grid(True)  # グリッドの表示\n\n# グラフの表示\nplt.show()  # グラフを表示\n","metadata":{"papermill":{"duration":0.014821,"end_time":"2024-07-14T10:27:01.911054","exception":false,"start_time":"2024-07-14T10:27:01.896233","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-07-22T17:19:09.423806Z","iopub.execute_input":"2024-07-22T17:19:09.424661Z","iopub.status.idle":"2024-07-22T17:19:10.072814Z","shell.execute_reply.started":"2024-07-22T17:19:09.424627Z","shell.execute_reply":"2024-07-22T17:19:10.071774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## テストデータの推論・CSVファイル書き込み","metadata":{"papermill":{"duration":0.620276,"end_time":"2024-07-14T11:00:19.561038","exception":false,"start_time":"2024-07-14T11:00:18.940762","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# データ変換の設定\ndata_transforms = transforms.Compose([\n    transforms.Resize((size, size)),  # 画像を指定サイズにリサイズ\n    transforms.ToTensor(),  # テンソルに変換\n    transforms.Normalize(mean, std)  # 平均と標準偏差を用いて正規化\n])\n\n# テストデータの準備\n# テスト画像のリストをID順にソート\np = re.compile(r'\\d+')\ntest_list = sorted(glob.glob('/kaggle/working/test/*.jpg'), key=lambda s: int(p.search(s).group()))\n\n# CSV書き込み\n# 推論結果をCSVファイルに保存\nwith open('/kaggle/working/submission.csv', 'w', newline='') as f:\n    writer = csv.writer(f)\n    writer.writerow(['id', 'label'])  # CSVファイルのヘッダーを書き込み\n    print(\"start\")\n    imcount = 1\n    total = len(test_list)\n    for img_path in tqdm(test_list, file=sys.stdout):\n        img = Image.open(img_path)  # 画像読み込み\n        img = data_transforms(img)  # 画像変換\n        img = img.unsqueeze(0)  # バッチ次元の追加\n        img = img.to(device)  # 画像をGPUに転送\n\n        with torch.no_grad():\n            outputs = model(img)  # モデルによる推論\n            probs = F.softmax(outputs, dim=1).cpu().numpy()  # 確率に変換\n\n        result_predict = probs[0]\n        predicted_class = np.argmax(result_predict)  # クラスの予測\n\n        writer.writerow([imcount, result_predict[1]])  # 犬の確率をCSVに書き込む\n\n        # 進行状況の表示\n        sys.stdout.write(f\"\\rProcessing image {imcount}/{total}\")\n        sys.stdout.flush()\n\n        imcount += 1\n    print(\"\\nend\")\n\nprint(\"Prediction and CSV creation complete.\")\n","metadata":{"papermill":{"duration":150.606935,"end_time":"2024-07-14T11:02:50.828113","exception":false,"start_time":"2024-07-14T11:00:20.221178","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-07-22T17:19:16.294784Z","iopub.execute_input":"2024-07-22T17:19:16.295239Z","iopub.status.idle":"2024-07-22T17:19:20.192763Z","shell.execute_reply.started":"2024-07-22T17:19:16.295205Z","shell.execute_reply":"2024-07-22T17:19:20.191312Z"},"trusted":true},"execution_count":null,"outputs":[]}]}