{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","isGpuEnabled":true,"isInternetEnabled":false},"colab":{"name":"dogs_cats_tutorial.ipynb","provenance":[]}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 😺猫🐶狗图像分类\n\n## 这个 notebook 是什么\n\n用深度学习做**二分类任务**：输入一张图片，模型判断它是猫还是狗。这是入门计算机视觉（CV）最经典的练手项目，也是面试/简历里最常聊的项目之一。\n\n## 运行前 3 步（很重要）\n\n1. 点右侧 **+ Add Input**，搜索添加数据集 **dogs-cats-images**（作者 Sabsrp98）；\n2. 右侧 **Settings → Accelerator** 把 `None` 改成 **GPU T4 x2**（不开 GPU 会非常慢）；\n3. 从上到下 **Run All**，预计 20-40 分钟出结果。\n\n## 总流程一览（12 步 + 1 个可选进阶）\n\n| 阶段 | 步骤 | 干什么 |\n|---|---|---|\n| 准备 | 0 环境检查 | 确认 GPU 开没开 |\n| 数据 | 1 配置 / 2 加载+增强 / 3 数据检查 | 读图、造数据、看数据 |\n| 模型 | 4 搭 CNN / 5 编译+回调 | 搭网络、设训练策略 |\n| 训练 | 6 训练 / 7 训练曲线 | 学参数、看收敛 |\n| 评估 | 8 测试评估 / 9 错分分析 / 10 Grad-CAM | 打分、找错、看注意力 |\n| 交付 | 11 单图推理+保存 | 用起来 |\n| 进阶 | 12 迁移学习（可选） | 冲更高准确率 |\n\n> 每一小节都是 **「先讲原理 → 再跑代码 → 最后看结论」** 三段式，方便你一步步对照学习。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 1：全局配置\n\n**这一步在干什么**：把所有\"可调参数\"集中放在最上面，统一管理。\n\n**为什么需要**：训练涉及很多超参数（图片尺寸、批次、轮数、随机种子）。集中定义的好处是：后面想调整时只改这里一处，不用满 notebook 找。\n\n**每个参数是什么**：\n- `IMAGE_SIZE`：图片统一缩放到多少×多少。越大保留信息越多但越慢、越吃显存；猫狗 128 足够。\n- `BATCH_SIZE`：每批喂给网络多少张图。32 是常用值；GPU 显存够可以调大到 64。\n- `EPOCHS`：最多训练多少轮。设 30 是上限，实际会被\"早停\"提前截断（见步骤 5）。\n- `VAL_SPLIT`：从训练集切出 20% 当验证集，用来监控模型有没有学偏（过拟合）。\n- `SEED`：随机种子。固定后每次跑的结果可复现，方便对比实验。\n\n**知识点**：超参数（hyperparameter）是\"训练前你手动定的数\"，和模型自己学的参数（weights）不同\n\n> 数据路径用的是 Kaggle 数据集 `dogs-cats-images`，里面已经按 `training_set` / `test_set` 分好，每个文件夹下再有 `cats/`、`dogs/` 子目录。\n","metadata":{}},{"cell_type":"code","source":"import os, shutil, zipfile\nfrom sklearn.model_selection import train_test_split\n\nWORK = \"/kaggle/working\"\n\n# 1. 解压（已解压过会自动跳过）\nif not os.path.isdir(f\"{WORK}/train\"):\n    print(\"解压中...\")\n    with zipfile.ZipFile(\"/kaggle/input/competitions/dogs-vs-cats/train.zip\") as z:\n        z.extractall(WORK)\n\n# 2. 分猫狗 + 切分\nraw = f\"{WORK}/train\"\ncats = sorted(f for f in os.listdir(raw) if f.startswith('cat.'))\ndogs = sorted(f for f in os.listdir(raw) if f.startswith('dog.'))\nc_tr, c_te = train_test_split(cats, test_size=0.3, random_state=42)\nd_tr, d_te = train_test_split(dogs, test_size=0.3, random_state=42)\n\nbase = f\"{WORK}/dogs_vs_cats\"\nfor split, cs, ds in [('train', c_tr, d_tr), ('test', c_te, d_te)]:\n    for cls, files in [('cats', cs), ('dogs', ds)]:\n        os.makedirs(f\"{base}/{split}/{cls}\", exist_ok=True)\n    for f in cs: shutil.copyfile(f\"{raw}/{f}\", f\"{base}/{split}/cats/{f}\")\n    for f in ds: shutil.copyfile(f\"{raw}/{f}\", f\"{base}/{split}/dogs/{f}\")\n\nTRAIN_DIR = f\"{base}/train\"\nTEST_DIR  = f\"{base}/test\"\nprint(\"完成:\", TRAIN_DIR, \"|\", TEST_DIR)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T12:47:15.340876Z","iopub.execute_input":"2026-09-05T12:47:15.34177Z","iopub.status.idle":"2026-09-05T12:47:18.693213Z","shell.execute_reply.started":"2026-09-05T12:47:15.341725Z","shell.execute_reply":"2026-09-05T12:47:18.69213Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\nIMAGE_SIZE = 128   # 图片统一缩放到 128×128\nBATCH_SIZE = 32    # 每批 32 张图\nEPOCHS     = 30    # 最多训练 30 轮（会被早停提前截断）\nVAL_SPLIT  = 0.2   # 从训练集切 20% 当验证集\nSEED       = 42    # 随机种子\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T09:52:50.745575Z","iopub.execute_input":"2026-09-05T09:52:50.745852Z","iopub.status.idle":"2026-09-05T09:52:50.750358Z","shell.execute_reply.started":"2026-09-05T09:52:50.745828Z","shell.execute_reply":"2026-09-05T09:52:50.74924Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 1 结论**：这段只是\"写参数\"不产生输出（可能只显示最后一个变量的值）。真正有意义的是理解每个参数的作用——上面已经逐个解释。后面所有代码都会引用这些变量。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 2：数据加载 + 数据增强\n\n**这一步在干什么**：把硬盘上的图片变成模型能读的\"数据流\"，并在训练时给每张图随机\"变形\"，变出更多训练样本。\n\n**两个关键函数**：\n- `ImageDataGenerator`：一个\"数据工厂\"，能批量读图、自动归一化、自动做数据增强。\n- `flow_from_directory`：按文件夹名自动生成标签（`cats/` 里的图标签就是 cat），不用自己手动打标签。\n\n**为什么要数据增强**：神经网络靠\"看很多样本\"学习。真实场景里图片千变万化（猫会转头、狗有各种姿势），如果只喂原始图，模型容易\"背答案\"（过拟合）。增强 = 每轮训练给每张图随机旋转/平移/缩放/翻转，等于免费造出更多变体，模型学到的规律更泛化。\n\n**为什么验证集/测试集不增强**：验证集和测试集是用来\"考试\"的，考试不能泄题、也不能把题目改得面目全非，否则分数不真实。所以它们只做归一化。\n\n**归一化（rescale=1./255）是什么**：把像素值从 0~255 缩放到 0~1。数值范围统一后，模型收敛更快、更稳定。\n\n> 这里从训练集内部用 `validation_split=0.2` 切出验证集（省得单独准备验证文件夹），是 sklearn `train_test_split` 分层思想的 Keras 版。\n","metadata":{}},{"cell_type":"code","source":"# ===== 步骤 2：数据加载 + 数据增强 =====\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.15,\n    height_shift_range=0.15,\n    shear_range=0.1,\n    zoom_range=0.15,\n    horizontal_flip=True,\n    fill_mode='nearest',\n    validation_split=0.2\n)\n\ntest_datagen = ImageDataGenerator(rescale=1./255)\n\ntrain_generator = train_datagen.flow_from_directory(\n    TRAIN_DIR,\n    target_size=(128, 128),\n    batch_size=32,\n    class_mode='categorical',\n    subset='training',\n    seed=42\n)\n\nval_generator = train_datagen.flow_from_directory(\n    TRAIN_DIR,\n    target_size=(128, 128),\n    batch_size=32,\n    class_mode='categorical',\n    subset='validation',\n    seed=42\n)\n\ntest_generator = test_datagen.flow_from_directory(\n    TEST_DIR,\n    target_size=(128, 128),\n    batch_size=32,\n    class_mode='categorical',\n    shuffle=False\n)\n\nclass_names = list(train_generator.class_indices.keys())\nprint(\"类别:\", class_names)\nprint(\"训练:\", train_generator.samples, \"| 验证:\", val_generator.samples, \"| 测试:\", test_generator.samples)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T12:48:26.597524Z","iopub.execute_input":"2026-09-05T12:48:26.598147Z","iopub.status.idle":"2026-09-05T12:48:26.857418Z","shell.execute_reply.started":"2026-09-05T12:48:26.598118Z","shell.execute_reply":"2026-09-05T12:48:26.856551Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 2 结论（怎么判断成功）**：\n- 会打印类似 `Found 16000 images belonging to 2 classes.` 的提示，说明数据读进来了；\n- `类别: ['cats', 'dogs']` 正确；\n- 训练/验证/测试三个数字加起来合理（例如 16000 / 4000 / 2000 左右）；\n- **常见坑**：如果某个数字是 0，说明路径或文件夹名不对，检查 Add Input 的数据集名和路径是否一致。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 3：数据快速检查（EDA）\n\n**这一步在干什么**：训练前先\"看一眼\"数据——每类有多少张、图片长什么样。\n\n**为什么需要**：深度学习里有一句老话\"garbage in, garbage out\"。如果类别严重不均衡（比如猫 9000 张、狗 1000 张），模型会偷懒全部猜猫；如果图片本身有问题（损坏、全黑、尺寸异常），训练会白跑。先检查能省下大量时间。\n\n**知识点**：EDA（Exploratory Data Analysis，探索性数据分析）是任何数据项目的标准第一步，不止图像，表格数据同样适用——这是能写进简历的\"流程意识\"。\n","metadata":{}},{"cell_type":"code","source":"# 3.1 每类数量（看类别是否均衡）\nfor name, gen in [(\"训练\", train_generator), (\"验证\", val_generator), (\"测试\", test_generator)]:\n    counts = {c: int(np.sum(gen.classes == i)) for c, i in gen.class_indices.items()}\n    print(name, counts)\n\n# 3.2 随机看 6 张训练图（看图片是否正常）\nimgs, lbls = next(train_generator)\nfig, axes = plt.subplots(2, 3, figsize=(9, 6))\nfor ax, img, lab in zip(axes.flat, imgs, lbls):\n    ax.imshow(img)\n    ax.set_title(class_names[np.argmax(lab)], fontsize=11)\n    ax.axis('off')\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T09:57:15.833177Z","iopub.execute_input":"2026-09-05T09:57:15.833451Z","iopub.status.idle":"2026-09-05T09:57:16.577785Z","shell.execute_reply.started":"2026-09-05T09:57:15.833429Z","shell.execute_reply":"2026-09-05T09:57:16.576997Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 3 结论（怎么判断成功）**：\n- 每类的数量应该接近（说明数据均衡，不用做特殊处理）；\n- 能看到 6 张带标签的猫狗图，图片清晰、标签正确 → 数据没问题，可以开始建模；\n- **如果发现某类数量明显偏少** → 后面模型会偏向多数类，属于可写进简历的\"数据不均衡问题\"，可提一下用加权损失/重采样解决。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 4：搭建 CNN 模型\n\n**这一步在干什么**：用代码\"组装\"一个卷积神经网络（CNN）——这就是项目核心，也是面试最常被追问的部分。\n\n**CNN 是什么**：一种专门处理图像的网络。它像人眼看图一样，先看局部小细节（边缘、纹理），再层层组合成整体形状。核心是 **卷积层（Conv2D）** 用一个个小窗口扫过整张图，提取特征。\n\n**每个部件的作用**：\n\n| 部件 | 作用 | 通俗解释 |\n|---|---|---|\n| Conv2D | 卷积提取特征 | 小窗口扫图，找\"边缘/纹理/形状\" |\n| BatchNormalization | 归一化层输出 | 让每层数据尺度稳定，训练更快更稳 |\n| MaxPooling2D | 下采样缩小 | 保留主要特征、砍掉冗余、减少计算 |\n| Dropout | 随机丢弃神经元 | 防止过拟合 |\n| L2 正则 | 权重加惩罚 | 防止权重过大 → 防过拟合 |\n| GlobalAveragePooling | 特征图压成向量 | 比 Flatten 参数少、不易过拟合 |\n| Dense + Softmax | 输出概率 | 给每个类别打分，加起来=1 |\n\n**为什么卷积核数量逐层翻倍（32→64→128→256）**：浅层特征简单用少量卷积核就够，深层特征更复杂，需要更多卷积核去刻画——这是 CNN 设计的通用规律。\n\n**为什么用 L2 正则**：它给大权重\"罚款\"，逼模型用更小的权重拟合，降低过拟合风险。\n\n> 采用函数式 API（`keras.Input` → 一层层接 → `keras.Model`）而不是 Sequential，因为后面 Grad-CAM 需要能按层名取到中间层的输出。\n","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras import layers\n\ndef build_cnn(input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3), num_classes=2):\n    inputs = keras.Input(shape=input_shape, name='input')\n    reg = keras.regularizers.l2(0.001)   # L2 正则系数\n\n    # Block 1：32 个卷积核\n    x = layers.Conv2D(32, 3, padding='same', activation='relu', kernel_regularizer=reg, name='conv1')(inputs)\n    x = layers.BatchNormalization()(x)\n    x = layers.MaxPooling2D(2)(x)\n    x = layers.Dropout(0.25)(x)\n\n    # Block 2：64 个卷积核\n    x = layers.Conv2D(64, 3, padding='same', activation='relu', kernel_regularizer=reg, name='conv2')(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.MaxPooling2D(2)(x)\n    x = layers.Dropout(0.25)(x)\n\n    # Block 3：128 个卷积核\n    x = layers.Conv2D(128, 3, padding='same', activation='relu', kernel_regularizer=reg, name='conv3')(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.MaxPooling2D(2)(x)\n    x = layers.Dropout(0.3)(x)\n\n    # Block 4：256 个卷积核（特征最丰富，Grad-CAM 就看这一层）\n    x = layers.Conv2D(256, 3, padding='same', activation='relu', kernel_regularizer=reg, name='conv4')(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.MaxPooling2D(2)(x)\n    x = layers.Dropout(0.3)(x)\n\n    # 分类头\n    x = layers.GlobalAveragePooling2D()(x)\n    x = layers.Dense(256, activation='relu', kernel_regularizer=reg)(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.Dropout(0.5)(x)\n    outputs = layers.Dense(num_classes, activation='softmax', name='predictions')(x)\n\n    return keras.Model(inputs, outputs, name='DogsCats_CNN_Combined')\n\nmodel = build_cnn(num_classes=len(class_names))\nmodel.summary()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T10:00:09.457508Z","iopub.execute_input":"2026-09-05T10:00:09.458102Z","iopub.status.idle":"2026-09-05T10:00:10.987467Z","shell.execute_reply.started":"2026-09-05T10:00:09.458069Z","shell.execute_reply":"2026-09-05T10:00:10.98685Z"},"scrolled":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 4 结论（怎么判断成功）**：\n- `model.summary()` 会打印一张\"网络结构表\"，每行是一个层，能看到 `Output Shape` 和 `Param #`；\n- 最下面显示 `Total params`（总参数量，大概几十万到一两百万）→ 说明网络搭好了；\n- **怎么读**：从上往下看，图片从 128×128×3 逐层变小（因为池化），到分类头变成 2 个输出（猫/狗概率）；\n- **常见坑**：如果报 `Input shape mismatch`，说明上面 target_size 和这里 input_shape 不一致，改成一个值即可。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 5：编译模型 + 挂回调\n\n**这一步在干什么**：配置\"训练策略\"——用什么损失函数、用什么优化器、学习率怎么变、训练中途自动干什么。\n\n**三部分拆解**：\n\n**① 损失函数（loss）**：告诉模型\"错了多少\"。`categorical_crossentropy`（多分类交叉熵）是分类任务的标准损失。模型每轮训练的目标就是让这个数越小越好。\n\n**② 优化器（optimizer）**：告诉模型\"怎么根据错误更新参数\"。`Adam` 是最常用的自适应优化器，基本不用调参。这里加了 **指数衰减学习率（ExponentialDecay）**：前期学习率大（学得快），后期变小（学得细），这是 notebook1 的亮点。\n\n**③ 回调（callbacks）**：训练途中的\"自动管家\"：\n\n| 回调 | 功能 |\n|---|---|\n| EarlyStopping | 验证准确率连续 5 轮不涨就提前停，省时间 |\n| ReduceLROnPlateau | 验证损失卡住时自动把学习率减半，帮模型跳出平台期 |\n| ModelCheckpoint | 只保存验证集表现最好那版权重，防止后期过拟合把最好的模型覆盖掉 |\n\n**为什么\"早停\"重要**：训练时间太长会从\"学会\"变成\"背题\"（过拟合）。早停就是\"见好就收\"，这也是工程里非常实用的习惯。\n","metadata":{}},{"cell_type":"code","source":"# ===== 步骤 5：编译 + 回调（修复版）=====\nmodel.compile(\n    optimizer=keras.optimizers.Adam(learning_rate=1e-3),\n    loss='categorical_crossentropy',\n    metrics=['accuracy']\n)\n\ncallbacks = [\n    keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=5,\n                                  restore_best_weights=True, verbose=1),\n    keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5,\n                                      patience=3, min_lr=1e-6, verbose=1),\n    keras.callbacks.ModelCheckpoint('best_model.keras', monitor='val_accuracy',\n                                    save_best_only=True),\n]\nprint(\"✅ 编译完成\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T10:23:52.101505Z","iopub.execute_input":"2026-09-05T10:23:52.101969Z","iopub.status.idle":"2026-09-05T10:23:52.124517Z","shell.execute_reply.started":"2026-09-05T10:23:52.101938Z","shell.execute_reply":"2026-09-05T10:23:52.123815Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 5 结论**：打印 `编译完成，回调已挂好` 即可。这里没有真正的\"结果\"，但配置是否正确会在训练时体现——如果训练 loss 一直不降，多半要回来调这里的初始学习率。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 6：训练（核心环节）\n\n**这一步在干什么**：让模型一遍遍看训练数据、反复更新参数，直到\"学会\"区分猫狗。这是最花时间的一步，也是产出核心结果（模型权重）的一步。\n\n**每轮打印的 4 个数字怎么看**：\n- `loss` / `accuracy` = 训练集表现（模型\"开卷考试\"的成绩）；\n- `val_loss` / `val_accuracy` = 验证集表现（\"闭卷考试\"成绩，更可信）。\n\n**两个重要的判断**：\n1. **过拟合**：训练准确率高、验证准确率低 → 模型在背训练题，没学到通用规律；\n2. **欠拟合**：两个准确率都低 → 模型能力不足或训练不够。\n\n**训练时间怎么估**：第 1 轮结束后，看\"每轮耗时 × 轮数\"。Kaggle GPU 上这个配置大约每轮 20-40 秒。\n\n> 训练时上方会实时显示进度条和 loss 变化，这是观察训练是否健康的第一现场。\n","metadata":{}},{"cell_type":"code","source":"history = model.fit(\n    train_generator,\n    validation_data=val_generator,\n    epochs=EPOCHS,\n    callbacks=callbacks\n)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T10:24:00.024798Z","iopub.execute_input":"2026-09-05T10:24:00.025194Z","iopub.status.idle":"2026-09-05T10:39:15.526347Z","shell.execute_reply.started":"2026-09-05T10:24:00.025164Z","shell.execute_reply":"2026-09-05T10:39:15.525653Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 6 结论（怎么判断成功）**：\n- 看到进度条走完，每轮有 loss/accuracy/val_loss/val_accuracy 四个数；\n- **理想曲线**：训练和验证准确率都稳步上升，两者差距不大；\n- 最终验证准确率大概在 **85%~92%** 区间属于正常（这个简单 CNN + 增强的水平）；\n- **如果 val_accuracy 卡在 50% 附近不动** → 等于瞎猜，检查：数据路径对不对、有没有开 GPU、初始学习率是否太大；\n- **如果 val 明显低于 train** → 过拟合，可以加强数据增强或加大 Dropout。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 7：画训练曲线\n\n**这一步在干什么**：把训练过程画成两条曲线（准确率、损失），直观看到模型是否收敛、有没有过拟合。\n\n**为什么用图而不是看数字**：一张图能同时看到\"整个训练过程\"的趋势，比逐行看数字高效得多。这也是所有深度学习项目必做的可视化。\n\n**怎么看这张图**：\n- 训练/验证两条线越贴近越好；\n- 训练高、验证低，且差距越来越大 = 过拟合信号；\n- 两条线都还在上升就停了 = 还可以多训练（把 EPOCHS 调大或放宽早停）。\n","metadata":{}},{"cell_type":"code","source":"# ===== 步骤 7：训练曲线 =====\nimport matplotlib.pyplot as plt\n\nacc = history.history['accuracy']\nval_acc = history.history['val_accuracy']\nloss = history.history['loss']\nval_loss = history.history['val_loss']\nep = range(1, len(acc) + 1)\n\nfig, axes = plt.subplots(1, 2, figsize=(12, 4))\naxes[0].plot(ep, acc, 'o-', label='Train acc')\naxes[0].plot(ep, val_acc, 's--', label='Val acc')\naxes[0].set_title('Accuracy'); axes[0].set_xlabel('Epoch'); axes[0].legend()\naxes[1].plot(ep, loss, 'o-', label='Train loss')\naxes[1].plot(ep, val_loss, 's--', label='Val loss')\naxes[1].set_title('Loss'); axes[1].set_xlabel('Epoch'); axes[1].legend()\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T10:50:54.402096Z","iopub.execute_input":"2026-09-05T10:50:54.40298Z","iopub.status.idle":"2026-09-05T10:50:54.68917Z","shell.execute_reply.started":"2026-09-05T10:50:54.40293Z","shell.execute_reply":"2026-09-05T10:50:54.688325Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 7 结论（怎么判断成功）**：\n- 左图两条 acc 曲线整体向上、右图两条 loss 曲线整体向下 → 正常收敛；\n- 两条线\"贴合\"程度越好，说明模型越健康；\n- **写进简历的一句话**：通过训练/验证曲线诊断出模型存在过拟合，并通过数据增强 + 早停缓解，最终验证准确率 xx%。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 8：在测试集上最终评估\n\n**这一步在干什么**：用模型**从未见过**的测试集算出最终成绩——这是项目里最重要的\"考试分数\"。\n\n**为什么不用训练/验证集当最终成绩**：训练集模型已经背过，验证集调参时已经\"偷看\"过。只有测试集是完全没见过的，用它评估才真实。\n\n**两个评估工具**：\n- **混淆矩阵（Confusion Matrix）**：2×2 表格，行=真实类别，列=预测类别。对角线越亮越好。能看出模型\"把狗认成猫\"和\"把猫认成狗\"分别错多少。\n- **分类报告（Classification Report）**：给出每类的精确率（Precision）、召回率（Recall）、F1。这些是简历/面试里必考的指标。\n\n**三个指标是什么（背下来）**：\n- **Precision（精确率）**：模型说是狗的里面，有多少真是狗 → \"说对的比例\"；\n- **Recall（召回率）**：所有真狗里，模型找到了多少 → \"找全的比例\"；\n- **F1**：两者的调和平均，综合衡量。\n","metadata":{}},{"cell_type":"code","source":"# ===== 修复中文显示 v2（从系统路径直接注册）=====\nimport matplotlib\nimport matplotlib.pyplot as plt\nfrom matplotlib import font_manager\nimport os\n\n# Kaggle 镜像里常见的中文字体路径，挨个试\ncandidates = [\n    \"/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc\",\n    \"/usr/share/fonts/opentype/noto/NotoSansCJK-Bold.ttc\",\n    \"/usr/share/fonts/truetype/wqy/wqy-microhei.ttc\",\n    \"/usr/share/fonts/truetype/wqy/wqy-zenhei.ttc\",\n]\n\nok = False\nfor p in candidates:\n    if os.path.exists(p):\n        font_manager.fontManager.addfont(p)\n        name = font_manager.FontProperties(fname=p).get_name()\n        plt.rcParams['font.sans-serif'] = [name]\n        plt.rcParams['axes.unicode_minus'] = False\n        print(f\"✅ 已注册中文字体: {name}\")\n        ok = True\n        break\n\nif not ok:\n    # 最后的兜底：下载一个中文字体（需联网）\n    print(\"系统没有现成中文字体，改用下载方案...\")\n    import urllib.request\n    url = \"https://github.com/googlefonts/noto-cjk/raw/main/Sans/SubsetOTF/SC/NotoSansSC-Regular.otf\"\n    dest = \"/kaggle/working/NotoSansSC.otf\"\n    if not os.path.exists(dest):\n        urllib.request.urlretrieve(url, dest)\n    font_manager.fontManager.addfont(dest)\n    name = font_manager.FontProperties(fname=dest).get_name()\n    plt.rcParams['font.sans-serif'] = [name]\n    plt.rcParams['axes.unicode_minus'] = False\n    print(f\"✅ 已下载并注册: {name}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T11:04:22.559202Z","iopub.execute_input":"2026-09-05T11:04:22.559637Z","iopub.status.idle":"2026-09-05T11:04:23.988632Z","shell.execute_reply.started":"2026-09-05T11:04:22.559602Z","shell.execute_reply":"2026-09-05T11:04:23.987928Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, classification_report\nimport seaborn as sns\n\ntest_loss, test_acc = model.evaluate(test_generator, verbose=0)\nprint(f\"🎯 测试集准确率: {test_acc*100:.2f}%\")\n\ntest_generator.reset()\ny_pred = np.argmax(model.predict(test_generator, verbose=0), axis=1)\ny_true = test_generator.classes\n\ncm = confusion_matrix(y_true, y_pred)\nplt.figure(figsize=(5, 4))\nsns.heatmap(cm, annot=True, fmt='d', cmap='Blues',\n            xticklabels=class_names, yticklabels=class_names)\nplt.xlabel('预测'); plt.ylabel('真实'); plt.title('混淆矩阵')\nplt.show()\n\nprint(classification_report(y_true, y_pred, target_names=class_names))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T11:04:28.853001Z","iopub.execute_input":"2026-09-05T11:04:28.85379Z","iopub.status.idle":"2026-09-05T11:04:47.395458Z","shell.execute_reply.started":"2026-09-05T11:04:28.853761Z","shell.execute_reply":"2026-09-05T11:04:47.394805Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 8 结论（怎么判断成功）**：\n- 打印出测试准确率（和训练/验证接近才算健康）；\n- 混淆矩阵对角线上两个数字明显大于非对角线 → 分类可靠；\n- **注意**：这里测试集只有几千张，和训练集同源，所以分数偏高是正常的；\n- **面试表达**：模型在测试集上达到 xx% 准确率，其中猫的 F1 为 xx、狗的 F1 为 xx，并解释了混淆矩阵反映的误判模式。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 9：看哪些图被分错了\n\n**这一步在干什么**：把分错的图片挑出来，看看模型到底错在哪。\n\n**为什么值得做**：模型给你 90% 准确率，但你可能想知道\"剩下 10% 错在哪\"。这属于**误差分析（Error Analysis）**，是真实项目里非常加分的环节——能发现\"深色背景的狗容易被认成猫\"这类规律，然后针对性改进。\n\n**怎么做改进（闭环思维）**：\n- 发现\"模糊图容易错\" → 预处理锐化/去模糊；\n- 发现\"特定姿势错\" → 加强这类数据的增强；\n- 这就是数据质量 → 模型的迭代闭环。\n","metadata":{}},{"cell_type":"code","source":"test_generator.reset()\nwrong = []\nfor imgs, lbls in test_generator:\n    probs = model.predict(imgs, verbose=0)\n    preds = np.argmax(probs, axis=1)\n    trues = np.argmax(lbls, axis=1)\n    for img, t, p, pr in zip(imgs, trues, preds, np.max(probs, axis=1)):\n        if t != p:\n            wrong.append((img, t, p, pr))\n    if len(wrong) >= 9:\n        break\n\nfig, axes = plt.subplots(3, 3, figsize=(8, 8))\nfor ax, (img, t, p, pr) in zip(axes.flat, wrong[:9]):\n    ax.imshow(img)\n    ax.set_title(f'真:{class_names[t]} 测:{class_names[p]}\\n置信{pr:.0%}', color='red', fontsize=9)\n    ax.axis('off')\nplt.tight_layout()\nplt.show()\nprint(f\"共收集到 {len(wrong)} 张错分图（至少展示 9 张）\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T11:08:00.164993Z","iopub.execute_input":"2026-09-05T11:08:00.165695Z","iopub.status.idle":"2026-09-05T11:08:01.606443Z","shell.execute_reply.started":"2026-09-05T11:08:00.165663Z","shell.execute_reply":"2026-09-05T11:08:01.605555Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n## 步骤 12（可选进阶）：迁移学习冲更高准确率\n\n**这一步在干什么**：用预训练好的 **EfficientNet** 做迁移学习，目标是冲到 90%+ 甚至 95%。\n\n**为什么能更高**：EfficientNet 已经在 ImageNet（1400 万张图）上学会了通用图像特征。我们\"借\"它的能力，只换掉最后的分类层重新训练。相当于让一个资深设计师帮你完成大部分工作，你只需微调收尾。\n\n**两个阶段（面试重点）**：\n1. **特征提取（Feature Extraction）**：冻结主干（`trainable=False`），只训练新加的分类头——快；\n2. **微调（Fine-tuning）**：解冻主干，用小学习率（1e-5）整体微调——慢但更准。\n\n**代价**：比自定义 CNN 慢不少，Kaggle GPU 上约多花 20-40 分钟。**新手建议先跑完主流程，这一节以后再试。**\n\n> 上一轮对比时提到的 VGG16 是更经典的教材例子；这里给 EfficientNet（更新的 SOTA 轻量模型），两个选一个练即可。\n","metadata":{}},{"cell_type":"code","source":"# ===== 步骤 12：迁移学习 EfficientNet（冲更高准确率）=====\nfrom tensorflow.keras import layers as kl\n\ndef build_transfer_model(num_classes=2):\n    # 用在 ImageNet 上预训练好的 EfficientNetB0 当\"特征提取器\"\n    base = keras.applications.EfficientNetB0(\n        include_top=False,\n        input_shape=(128, 128, 3),\n        weights='imagenet'\n    )\n    base.trainable = False  # 第一阶段：冻结主干，只训练新加的分类头\n\n    inputs = keras.Input(shape=(128, 128, 3))\n    # generator 已把像素缩到 0~1，EfficientNet 期望 0~255，先还原\n    x = kl.Rescaling(255.0)(inputs)\n    x = base(x, training=False)\n    x = kl.GlobalAveragePooling2D()(x)\n    x = kl.Dropout(0.3)(x)\n    outputs = kl.Dense(num_classes, activation='softmax')(x)\n    return keras.Model(inputs, outputs)\n\ntl_model = build_transfer_model(len(class_names))\ntl_model.compile(\n    optimizer=keras.optimizers.Adam(1e-3),\n    loss='categorical_crossentropy',\n    metrics=['accuracy']\n)\n\nprint(\"===== 阶段1：特征提取（主干冻结，只训分类头，约5-10分钟）=====\")\ntl_model.fit(\n    train_generator,\n    validation_data=val_generator,\n    epochs=10,\n    callbacks=callbacks,\n    verbose=1\n)\n\nprint(\"===== 阶段2：微调（解冻主干，小学习率精调，约15-30分钟）=====\")\ntl_model.trainable = True\nfor layer in tl_model.layers:\n    if isinstance(layer, keras.Model):\n        layer.trainable = True\ntl_model.compile(\n    optimizer=keras.optimizers.Adam(1e-5),  # 微调必须用很小的学习率\n    loss='categorical_crossentropy',\n    metrics=['accuracy']\n)\ntl_model.fit(\n    train_generator,\n    validation_data=val_generator,\n    epochs=10,\n    callbacks=callbacks,\n    verbose=1\n)\n\n# 最终测试集评估\ntl_test_loss, tl_test_acc = tl_model.evaluate(test_generator, verbose=0)\nprint(f\"\\n🎯 迁移学习测试集准确率: {tl_test_acc*100:.2f}%\")\ntl_model.save(\"dogs_cats_efficientnet.keras\")\nprint(\"✅ 模型已保存: dogs_cats_efficientnet.keras\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T11:16:23.839515Z","iopub.execute_input":"2026-09-05T11:16:23.840243Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ===== 迁移学习模型最终评估 =====\nimport numpy as np\nfrom sklearn.metrics import classification_report\n\ntl_test_loss, tl_test_acc = tl_model.evaluate(test_generator, verbose=0)\nprint(f\"🎯 迁移学习测试集准确率: {tl_test_acc*100:.2f}%\")\nprint(f\"（对比：从零训练 CNN 是 90.61%）\")\n\ntest_generator.reset()\ny_pred = np.argmax(tl_model.predict(test_generator, verbose=0), axis=1)\ny_true = test_generator.classes\nprint(classification_report(y_true, y_pred, target_names=class_names))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T11:55:43.623445Z","iopub.execute_input":"2026-09-05T11:55:43.624121Z","iopub.status.idle":"2026-09-05T11:56:15.588322Z","shell.execute_reply.started":"2026-09-05T11:55:43.624091Z","shell.execute_reply":"2026-09-05T11:56:15.587445Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 9 结论（怎么判断成功）**：\n- 能看到 9 张\"真实标签 vs 预测标签\"不同的图，标题是红色；\n- 试着总结规律：这些错图是模糊？光线暗？还是物体不完整？\n- 这就是你的\"误差分析结论\"，写进简历非常有说服力。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 10：Grad-CAM 热力图（模型在看哪里）\n\n**这一步在干什么**：把模型的\"注意力\"画成热力图，告诉我们\"模型判断猫/狗时，到底在看图片的哪个部位\"。\n\n**为什么值得做（可解释性）**：90% 准确率只是一个数，但\"为什么这么判\"才是更难也更有价值的回答。**可解释性（Explainable AI）** 是当前 AI 领域很受关注的方向，也是这个项目区别于\"跑通就跑完\"的加分项。\n\n**Grad-CAM 原理（一句话版）**：对最后一个卷积层（conv4）的输出求\"梯度\"，梯度大的位置说明它对分类贡献大，把这些位置加权画成热力图叠加在原图上。红色=模型重点看的地方。\n\n> 实现依赖函数式 API（步骤 4 特意用了 `keras.Input` + `keras.Model`），因为需要能按 `conv4` 这层名取到中间层输出。\n","metadata":{}},{"cell_type":"code","source":"# ===== 步骤 10：Grad-CAM 热力图（模型在看哪里）=====\nimport cv2\nimport tensorflow as tf\nimport numpy as np\nimport matplotlib.pyplot as plt\n\ndef grad_cam(model, img_array, layer_name='conv4'):\n    # 构建一个能输出中间层特征和最终预测的模型\n    grad_model = keras.Model(\n        inputs=model.input,\n        outputs=[model.get_layer(layer_name).output, model.output])\n    with tf.GradientTape() as tape:\n        conv_out, preds = grad_model(tf.cast(img_array, tf.float32))\n        top_class = tf.argmax(preds[0])\n        loss = preds[:, top_class]\n    # 梯度大的位置 = 对分类贡献大\n    grads = tape.gradient(loss, conv_out)\n    pooled = tf.reduce_mean(grads, axis=(0, 1, 2))\n    heatmap = tf.squeeze(conv_out[0] @ pooled[..., tf.newaxis]).numpy()\n    heatmap = np.maximum(heatmap, 0) / (heatmap.max() + 1e-8)\n    return heatmap, int(top_class.numpy()), float(tf.reduce_max(preds[0]).numpy())\n\ntest_generator.reset()\nimgs, lbls = next(test_generator)\n\nfig, axes = plt.subplots(2, 3, figsize=(10, 7))\nfor i, ax in enumerate(axes.flat):\n    img = imgs[i]\n    hm, pred_idx, conf = grad_cam(model, img[np.newaxis, ...])\n    hm = cv2.resize(hm, (img.shape[1], img.shape[0]))\n    hm = cv2.applyColorMap(np.uint8(255 * hm), cv2.COLORMAP_JET)\n    hm = cv2.cvtColor(hm, cv2.COLOR_BGR2RGB) / 255.\n    overlay = np.clip(img * 0.5 + hm * 0.5, 0, 1)\n    ax.imshow(overlay)\n    ax.set_title(f'Pred:{class_names[pred_idx]} {conf:.0%}\\nTrue:{class_names[np.argmax(lbls[i])]}', fontsize=9)\n    ax.axis('off')\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T12:04:14.107844Z","iopub.execute_input":"2026-09-05T12:04:14.108218Z","iopub.status.idle":"2026-09-05T12:04:15.692585Z","shell.execute_reply.started":"2026-09-05T12:04:14.108189Z","shell.execute_reply":"2026-09-05T12:04:15.690096Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 10 结论（怎么判断成功）**：\n- 6 张图上叠加了红黄蓝热力图；\n- **好现象**：热力图集中在动物主体（脸、身体）上 → 模型学到了\"看猫狗本身\"而非背景等无关信息；\n- **坏现象**：热力图散在背景上 → 模型在\"偷懒\"或数据有偏差，值得记录；\n- 这是\"模型可解释性分析\"的完整演示，简历里写\"使用 Grad-CAM 验证模型关注区域\"会非常亮眼。\n","metadata":{}},{"cell_type":"markdown","source":"---\n## 步骤 11：单张图片预测 + 保存模型\n\n**这一步在干什么**：① 拿一张新图给模型预测；② 把训练好的模型保存下来。\n\n**为什么单图预测重要**：模型训练完要\"能用\"，即对新图片做预测（推理 inference）。这也对应真实业务场景：训练好一次，部署后持续预测。\n\n**为什么保存模型**：训练一次很贵（时间+算力），保存后以后直接加载用，不用重新训练。`dogs_cats_cnn.keras` 就是模型文件。\n\n**知识点**：`load_img` 读图 → `img_to_array` 转成数组 → 归一化 → `model.predict` 得到两个类别的概率 → 取最大者为预测结果。注意单张图也要包一层维度 `[None, ...]`，因为模型默认处理\"一批\"图。\n","metadata":{}},{"cell_type":"code","source":"# ===== 单图预测（从测试集取图，无需文件路径）=====\nimport numpy as np\nimport matplotlib.pyplot as plt\n\ntest_generator.reset()\nimgs, lbls = next(test_generator)\n\n# 取第一张图\nimg = imgs[0]\nprob = model.predict(img[np.newaxis, ...], verbose=0)[0]\npred_idx = int(np.argmax(prob))\ntrue_idx = int(np.argmax(lbls[0]))\n\nprint(f\"预测: {class_names[pred_idx]} (置信度 {prob[pred_idx]:.1%})\")\nprint(f\"真实: {class_names[true_idx]}\")\n\nplt.imshow(img)\nplt.title(f\"Pred: {class_names[pred_idx]} ({prob[pred_idx]:.0%})\\nTrue: {class_names[true_idx]}\")\nplt.axis('off')\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-05T12:09:58.678606Z","iopub.execute_input":"2026-09-05T12:09:58.679253Z","iopub.status.idle":"2026-09-05T12:09:59.596377Z","shell.execute_reply.started":"2026-09-05T12:09:58.679222Z","shell.execute_reply":"2026-09-05T12:09:59.595604Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ===== 生成竞赛提交文件 submission.csv =====\nimport zipfile, os\nimport numpy as np\nimport pandas as pd\nfrom tensorflow.keras.preprocessing.image import load_img, img_to_array\n\n# 1. 解压竞赛测试集 test1.zip（12500张无标签图）\ntest1_zip = \"/kaggle/input/competitions/dogs-vs-cats/test1.zip\"\ntest1_dir = \"/kaggle/working/test1\"\nif not os.path.isdir(test1_dir):\n    print(\"解压 test1.zip ...\")\n    with zipfile.ZipFile(test1_zip) as z:\n        z.extractall(\"/kaggle/working/\")\n\n# 2. 用迁移学习模型预测所有测试图\nimage_ids = []\npredictions = []\nfiles = sorted(os.listdir(test1_dir), key=lambda x: int(x.split('.')[0]))\n\nfor fname in files:\n    img_id = int(fname.split('.')[0])\n    img = load_img(os.path.join(test1_dir, fname), target_size=(128, 128))\n    arr = img_to_array(img) / 255.\n    prob = tl_model.predict(arr[np.newaxis, ...], verbose=0)[0]\n    label = int(np.argmax(prob))  # 0=cat, 1=dog（竞赛标准格式）\n    image_ids.append(img_id)\n    predictions.append(label)\n\n# 3. 生成 submission.csv\nsubmission = pd.DataFrame({\"id\": image_ids, \"label\": predictions})\nsubmission = submission.sort_values(\"id\")\nsubmission.to_csv(\"/kaggle/working/submission.csv\", index=False)\nprint(f\"✅ 已生成 submission.csv，共 {len(submission)} 条预测\")\nprint(submission.head())\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**步骤 11 结论（怎么判断成功）**：\n- 打印出\"预测: cats，置信度 xx%\"且图片正确显示 → 推理成功；\n- 输出 `模型已保存: dogs_cats_cnn.keras` → 模型文件已生成（在左侧 /kaggle/working 里能看到）；\n- **想测自己的图**：点右侧 Add Input 或上传图片到工作区，把 `predict_image(...)` 里的路径换成你的图路径再跑。\n","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}