{"metadata":{"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":5441,"databundleVersionId":38425,"sourceType":"competition"}],"dockerImageVersionId":30747,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## 1.inputファイルを確認¶\n\"/kaggle/input\"ディレクトリに次の3つのファイルが存在することを確認\n\n「sample_submission.csv」\n\n「test.zip」\n\n「train.zip」","metadata":{}},{"cell_type":"code","source":"# 必要ライブラリのインポート\nimport numpy as np\nimport pandas as pd\nimport os\n\n# \"/kaggle/input\"ディレクトリに存在するファイルを表示\nfor dirname, _, filenames in os.walk(\"/kaggle/input\"):\n    for filename in filenames:\n        print(filename)","metadata":{"execution":{"iopub.status.busy":"2024-07-15T10:08:08.708513Z","iopub.execute_input":"2024-07-15T10:08:08.708854Z","iopub.status.idle":"2024-07-15T10:08:09.704814Z","shell.execute_reply.started":"2024-07-15T10:08:08.708827Z","shell.execute_reply":"2024-07-15T10:08:09.703851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.inputファイルの解凍及びパスの確認¶\n「train.zip」「test.zip」を解凍し、「sample_submission.csv」をコピー","metadata":{}},{"cell_type":"code","source":"# 必要ライブラリのインポート\nimport shutil\nimport glob\n\n# 「train.zip」、「test.zip」を解凍\nshutil.unpack_archive(\"/kaggle/input/dogs-vs-cats-redux-kernels-edition/train.zip\", \"/kaggle/working/unpack\")\nshutil.unpack_archive(\"/kaggle/input/dogs-vs-cats-redux-kernels-edition/test.zip\", \"/kaggle/working/unpack\")\n\n# 「sample_submission.csv」を提出用ファイル「submission.csv」としてコピー\nshutil.copyfile(\"/kaggle/input/dogs-vs-cats-redux-kernels-edition/sample_submission.csv\",\"/kaggle/working/submission.csv\")\n     \n# /kaggle/workingディレクトリ以下にファイルが作成されていることを確認\nfiles = glob.glob(\"/kaggle/working/*\")\nfor file in files:\n    print(file)\nfiles = glob.glob(\"/kaggle/working/unpack/*\")\nfor file in files:\n    print(file)","metadata":{"execution":{"iopub.status.busy":"2024-07-15T10:08:09.706796Z","iopub.execute_input":"2024-07-15T10:08:09.707358Z","iopub.status.idle":"2024-07-15T10:08:24.928640Z","shell.execute_reply.started":"2024-07-15T10:08:09.707325Z","shell.execute_reply":"2024-07-15T10:08:24.927760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.学習データとテストデータの画像枚数を確認¶\n学習データ(訓練データと検証データ)とテストデータの画像枚数を確認\n\n・学習データ：「犬：12500枚」「猫：12500枚」　合計25000枚\n\n・テストデータ：犬猫混合　合計12500枚","metadata":{}},{"cell_type":"code","source":"# 解凍した「train」「test」ディレクトリ内の画像（.jpg）ファイルの枚数を確認\nTrain_data_list = glob.glob(os.path.join(\"unpack/train\",\"*.jpg\"))\nprint(f\"学習データ　: {len(Train_data_list)}枚\")\n\nTest_data_list = glob.glob(os.path.join(\"unpack/test\",\"*.jpg\"))\nprint(f\"テストデータ: {len(Test_data_list)}枚\")","metadata":{"execution":{"iopub.status.busy":"2024-07-15T10:08:24.929863Z","iopub.execute_input":"2024-07-15T10:08:24.930176Z","iopub.status.idle":"2024-07-15T10:08:25.067591Z","shell.execute_reply.started":"2024-07-15T10:08:24.930151Z","shell.execute_reply":"2024-07-15T10:08:25.066656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.datasetディレクトリの作成と階層構造の確認\nデータの学習、分類に使用するためのdatasetディレクトリを作成し、階層構造を確認","metadata":{}},{"cell_type":"code","source":"# ディレクトリ構造を描画するためのライブラリをインストール\n!pip install Tree\nprint(\"----------------------\")\n\n# ./datasetディレクトリ以下のパス\ntrain_dir = \"./dataset/trainData\"\nvalid_dir = \"./dataset/validData\"\ntest_dir = \"./dataset/testData\"\n\n# 各ディレクトリの作成↓\n# 訓練データ\nos.makedirs(\"%s/dogs\" %train_dir, exist_ok=True)\nos.makedirs(\"%s/cats\" %train_dir, exist_ok=True)\n# 検証データ\nos.makedirs(\"%s/dogs\" %valid_dir, exist_ok=True)\nos.makedirs(\"%s/cats\" %valid_dir, exist_ok=True)\n# テストデータ\nos.makedirs(\"%s\" %test_dir, exist_ok=True)\n\n# 階層構造を表示\n!tree -d ./","metadata":{"execution":{"iopub.status.busy":"2024-07-15T10:08:25.070257Z","iopub.execute_input":"2024-07-15T10:08:25.070634Z","iopub.status.idle":"2024-07-15T10:08:42.696455Z","shell.execute_reply.started":"2024-07-15T10:08:25.070603Z","shell.execute_reply":"2024-07-15T10:08:42.695408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.学習データとテストデータをdatasetディレクトリに移動¶\n「unpack」から「dataset」にデータを移動","metadata":{}},{"cell_type":"code","source":"# 移動元のパスを指定\nsource_dir = \"./unpack\"\n\n# 最初の10000枚をtrainDataフォルダに格納\nfor i in range(10000):\n  os.rename(\"%s/train/dog.%d.jpg\" %(source_dir, i+0), \"%s/dogs/dog%04d.jpg\" %(train_dir, i+0))\n  os.rename(\"%s/train/cat.%d.jpg\" %(source_dir, i+0), \"%s/cats/cat%04d.jpg\" %(train_dir, i+0))\n\n# 次の2500枚をvalidDataフォルダに格納\nfor i in range(2500):\n  os.rename(\"%s/train/dog.%d.jpg\" %(source_dir, 10000+i+0), \"%s/dogs/dog%04d.jpg\" %(valid_dir, i+0))\n  os.rename(\"%s/train/cat.%d.jpg\" %(source_dir, 10000+i+0), \"%s/cats/cat%04d.jpg\" %(valid_dir, i+0))\n\n# /unpack/test/*.jpgをtestDataフォルダに格納\nread_dir = './unpack/test/'\nread_files = os.listdir(read_dir)\nfor read_name in read_files:\n    copy_from_path = read_dir + read_name\n    copy_to_path = test_dir + '/' + read_name \n    shutil.copy(copy_from_path, copy_to_path)","metadata":{"execution":{"iopub.status.busy":"2024-07-15T10:08:42.697785Z","iopub.execute_input":"2024-07-15T10:08:42.698090Z","iopub.status.idle":"2024-07-15T10:08:44.767881Z","shell.execute_reply.started":"2024-07-15T10:08:42.698064Z","shell.execute_reply":"2024-07-15T10:08:44.767102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.Datasetの確認¶\ndatasetディレクトリの中身を確認","metadata":{}},{"cell_type":"code","source":"# Outputデータ、「train」「test」ディレクトリ内のデータを確認\nTrain_data_dogs_list = glob.glob(os.path.join(\"dataset/trainData/dogs/\",\"*.jpg\"))\nTrain_data_cats_list = glob.glob(os.path.join(\"dataset/trainData/cats/\",\"*.jpg\"))\nprint(f\"Train_data_dogs: {len(Train_data_dogs_list)}\")\nprint(f\"Train_data_cats: {len(Train_data_cats_list)}\")\n\nValid_data_dogs_list = glob.glob(os.path.join(\"dataset/validData/dogs/\",\"*.jpg\"))\nValid_data_cats_list = glob.glob(os.path.join(\"dataset/validData/cats/\",\"*.jpg\"))\nprint(f\"Valid_data_dogs: {len(Valid_data_dogs_list)}\")\nprint(f\"Valid_data_cats: {len(Valid_data_cats_list)}\")\n\nTest_data_list = glob.glob(os.path.join(\"dataset/testData/\",\"*.jpg\"))\nprint(f\"Test_data: {len(Test_data_list)}\")\n\n!tree -d ./dataset","metadata":{"execution":{"iopub.status.busy":"2024-07-15T10:08:44.769008Z","iopub.execute_input":"2024-07-15T10:08:44.769290Z","iopub.status.idle":"2024-07-15T10:08:46.042870Z","shell.execute_reply.started":"2024-07-15T10:08:44.769267Z","shell.execute_reply":"2024-07-15T10:08:46.041708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 7.必要ライブラリのインポート¶\n分類に必要なライブラリをインポート","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n%matplotlib inline \n\nimport math\nimport re\nimport csv\nimport urllib.request\nimport tqdm\n\nfrom tensorflow import keras\nfrom tensorflow.keras import models, optimizers, layers\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nimport tensorflow.keras.layers as layers\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense, GlobalAveragePooling2D\nfrom tensorflow.keras.layers import Activation, BatchNormalization, Input\nfrom tensorflow.keras.applications.vgg16 import VGG16\nfrom tensorflow.keras.models import Sequential, load_model, Model\nfrom tensorflow.keras.optimizers import SGD\nfrom tensorflow.keras.callbacks import EarlyStopping","metadata":{"execution":{"iopub.status.busy":"2024-07-15T10:08:46.044299Z","iopub.execute_input":"2024-07-15T10:08:46.044614Z","iopub.status.idle":"2024-07-15T10:08:58.423208Z","shell.execute_reply.started":"2024-07-15T10:08:46.044585Z","shell.execute_reply":"2024-07-15T10:08:58.422241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 8.ジェネレータの設定¶\nモデルにデータを読み込ませるためのジェネレータを設定","metadata":{}},{"cell_type":"code","source":"# バッチサイズの指定\nbatch_size = 32\n\n# データ拡張しながら，テンソル画像データのバッチを生成\ntrain_data_generator = ImageDataGenerator(rescale=1.0/255,\n                                          shear_range=0.2,\n                                          zoom_range=0.2,\n                                          horizontal_flip=True,\n                                          vertical_flip = True)\n\nvalid_data_generator = ImageDataGenerator(rescale=1.0/255)\n\ntest_data_generator = ImageDataGenerator(rescale=1.0/255)\n\n# ディレクトリからデータを生成\ntrain_generator = train_data_generator.flow_from_directory(\n    train_dir,\n    target_size=(224,224),\n    batch_size=batch_size,\n    class_mode=\"binary\",\n    shuffle=True\n)\n\nvalid_generator = valid_data_generator.flow_from_directory(\n    valid_dir,\n    target_size=(224,224),\n    batch_size=batch_size,\n    class_mode=\"binary\",\n    shuffle=True\n)\n\ntest_generator = test_data_generator.flow_from_directory(\n    valid_dir,\n    target_size=(224,224),\n    batch_size=batch_size,\n    class_mode=\"binary\",\n    shuffle=True\n)\n\n# ----- {'cats': 0, 'dogs': 1} ------#\nprint(train_generator.class_indices)\nprint(valid_generator.class_indices)\nprint(test_generator.class_indices)","metadata":{"execution":{"iopub.status.busy":"2024-07-15T10:08:58.424810Z","iopub.execute_input":"2024-07-15T10:08:58.425731Z","iopub.status.idle":"2024-07-15T10:08:59.730975Z","shell.execute_reply.started":"2024-07-15T10:08:58.425695Z","shell.execute_reply":"2024-07-15T10:08:59.729985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 9.学習モデルの構築\n学習済みモデルである「VGG16」を使用し、犬・猫の2値分類用に全結合層を作成し従来のVGG16に結合","metadata":{}},{"cell_type":"code","source":"# VGG16をインポート\nmodel_vgg16 = VGG16(include_top=False, weights=\"imagenet\", input_tensor=Input(shape=(224,224,3)))\n\n# 全結合層を作成\nx = model_vgg16.output\nx = Flatten()(x)\nx = Dense(256)(x)\nx = Activation(\"relu\")(x)\nx = Dropout(0.1)(x)\nprediction = Dense(1, activation=\"sigmoid\")(x)\n\n# モデル合体\nmodel = Model(inputs=model_vgg16.input, outputs=prediction)\n\n# モデル構築 \nmodel.compile(loss = \"binary_crossentropy\",\n             optimizer = optimizers.Adam(learning_rate = 1e-4),\n             metrics = [\"accuracy\"])\n\n# 構築したモデルの確認\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2024-07-15T10:08:59.732170Z","iopub.execute_input":"2024-07-15T10:08:59.732480Z","iopub.status.idle":"2024-07-15T10:09:02.634266Z","shell.execute_reply.started":"2024-07-15T10:08:59.732453Z","shell.execute_reply":"2024-07-15T10:09:02.633222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 10.モデルの学習\n学習済みモデルである「VGG16」をファインチューニング","metadata":{}},{"cell_type":"code","source":"# block5_pool(18層目)までの重みの更新をキャンセル\nfor layer in model_vgg16.layers[:15]:\n  layer.trainable=False\n\n# 学習\nhistory = model.fit(train_generator,\n                    epochs=5,\n                    verbose=1,\n                    validation_data=valid_generator,\n                    callbacks=[EarlyStopping(patience=10)])\n\n# モデル保存\nmodel.save(\"VGG16_dogs_vs_cats.h5\")\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2024-07-15T10:09:02.636323Z","iopub.execute_input":"2024-07-15T10:09:02.637039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 11.訓練データ,検証データに対するLoss値,Acc値の確認","metadata":{}},{"cell_type":"code","source":"# lossとaccuracyを可視化\nprint(\"-\"*100)\nplt.figure(1, figsize=(13,4))\nplt.subplots_adjust(wspace=0.5)\n\n# 学習曲線\nplt.subplot(1, 2, 1)\nplt.plot(history.history[\"loss\"], label=\"train\")\nplt.plot(history.history[\"val_loss\"], label=\"valid\")\nplt.title(\"train and valid loss\")\nplt.xlabel(\"epoch\")\nplt.ylabel(\"loss\")\nplt.legend()\nplt.grid()\n\n# 精度表示\nplt.subplot(1, 2, 2)\nplt.plot(history.history[\"accuracy\"], label=\"train\")\nplt.plot(history.history[\"val_accuracy\"], label=\"valid\")\nplt.title(\"train and valid accuracy\")\nplt.xlabel(\"epoch\")\nplt.ylabel(\"accuracy\")\nplt.legend()\nplt.grid()\n\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 12.テストデータの分類","metadata":{}},{"cell_type":"code","source":"# testData内の写真を番号順にソート\np = re.compile(r'\\d+')\ntest_list = glob.glob('/kaggle/working/dataset/testData/*.jpg')\ntest_list = sorted(test_list, key = lambda s: int(p.search(s).group()))\n# print(test_list)\nimcount = 1\n\n\n\n#モデルの読み込み\nmodel = models.load_model('VGG16_dogs_vs_cats.h5')\n\n#csvに書き込み\nwith open('/kaggle/working/submission.csv', 'w') as f:\n    writer = csv.writer(f, lineterminator='\\n')\n    writer.writerow(['id', 'label'])\n\n    for i in tqdm.tqdm(range(len(test_list))):\n        img = image.load_img(test_list[i], target_size = (224, 224))\n        x = image.img_to_array(img)\n        x = np.expand_dims(x, axis=0)\n        x = x / 255.0\n        \n        result_predict = model.predict(x,verbose=0)\n        writer.writerow([imcount, result_predict[0][0]])\n#         print([imcount, result_predict[0][0]])\n        imcount += 1","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}