{
  "id": 248645,
  "title": "初次尝试了 TTA 之后得到更差的结果，有这块有经验的同学帮忙看下吗",
  "url": "/competitions/classify-leaves/discussion/248645",
  "author_name": "",
  "post_date": "2021-06-24T10:30:38.367097300Z",
  "votes": 3,
  "comment_count": 14,
  "views": 0,
  "content": "<p><strong>疑问</strong><br>\n验证集 2753 样本下，原始不做tta，错了 54个。<br>\n单看某次 tta 的预测结果，会错 70~80个<br>\n使用6次投票结果，会错 63 个，效果还是比不做 tta 更差。。</p>\n<p><strong>详情</strong><br>\n我用的 resnest50 + 一些 tricks, 不使用tta单折验证集上精度 0.9803，public board 上 0.978 <br>\n然后在单折验证集上，根据网上思路试验了自己实现的 tta，即在预测每个batch时，重复使用一个 transfrom 多次，进下上下水平翻转、90旋转的操作（和训练时用的 aug 相同）</p>\n<p>即每个批次，除了原始仅进下 Normalize 处理的预测外，额外进下了5次 其他变化（Normalize 仅1次没重复），然后观察到</p>\n<ol>\n<li>做了单此 tta 明显不如不做tta的</li>\n<li>即使用投票结果，也不不错tta差<br>\n3.部分非tta预测错的结果，再做了tta后，有可能得到正确样本</li>\n</ol>\n<p><strong>模型对变化不够稳定</strong><br>\n通过验证集 tta 还观察到，验证集中，有 3% 数据，会因为做了 tta，得到不同分类结果；因为模型精度验证集上已经超过97%，不知道3%这个值是否异常，有哪些可以进一步优化吗</p>\n<p><strong>-- 错误例子</strong><br>\n使用非tta原始结果，预测错了的例子，预计其他tta的结果<br>\nlabel    pre_org pre_tta0    pre_tta1    pre_tta2    pre_tta3    pre_tta4    pre_vote<br>\n1888    153 135 135 9   135 91  9   135<br>\n1605    56  58  58  58  58  58  58  58<br>\n2158    68  12  12  12  12  12  12  12<br>\n1177     33  154 33  33  33  154 33  33<br>\n1135    132 69  69  69  69  69  69  69</p>\n<p>使用投票结果，但预测错误的例子<br>\nlabel    pre_org pre_tta0    pre_tta1    pre_tta2    pre_tta3    pre_tta4    pre_vote<br>\n137    70  137 143 143 137 143 143<br>\n68    12  12  12  12  12  12  12<br>\n140    140 163 163 163 140 163 163<br>\n59    87  87  128 87  128 128 87<br>\n69    95  95  95  95  95  95  95</p>",
  "messages": [
    {
      "id": "1363744",
      "postDate": "06/24/2021 10:30:38",
      "content": "<p><strong>疑问</strong><br>\n验证集 2753 样本下，原始不做tta，错了 54个。<br>\n单看某次 tta 的预测结果，会错 70~80个<br>\n使用6次投票结果，会错 63 个，效果还是比不做 tta 更差。。</p>\n<p><strong>详情</strong><br>\n我用的 resnest50 + 一些 tricks, 不使用tta单折验证集上精度 0.9803，public board 上 0.978 <br>\n然后在单折验证集上，根据网上思路试验了自己实现的 tta，即在预测每个batch时，重复使用一个 transfrom 多次，进下上下水平翻转、90旋转的操作（和训练时用的 aug 相同）</p>\n<p>即每个批次，除了原始仅进下 Normalize 处理的预测外，额外进下了5次 其他变化（Normalize 仅1次没重复），然后观察到</p>\n<ol>\n<li>做了单此 tta 明显不如不做tta的</li>\n<li>即使用投票结果，也不不错tta差<br>\n3.部分非tta预测错的结果，再做了tta后，有可能得到正确样本</li>\n</ol>\n<p><strong>模型对变化不够稳定</strong><br>\n通过验证集 tta 还观察到，验证集中，有 3% 数据，会因为做了 tta，得到不同分类结果；因为模型精度验证集上已经超过97%，不知道3%这个值是否异常，有哪些可以进一步优化吗</p>\n<p><strong>-- 错误例子</strong><br>\n使用非tta原始结果，预测错了的例子，预计其他tta的结果<br>\nlabel    pre_org pre_tta0    pre_tta1    pre_tta2    pre_tta3    pre_tta4    pre_vote<br>\n1888    153 135 135 9   135 91  9   135<br>\n1605    56  58  58  58  58  58  58  58<br>\n2158    68  12  12  12  12  12  12  12<br>\n1177     33  154 33  33  33  154 33  33<br>\n1135    132 69  69  69  69  69  69  69</p>\n<p>使用投票结果，但预测错误的例子<br>\nlabel    pre_org pre_tta0    pre_tta1    pre_tta2    pre_tta3    pre_tta4    pre_vote<br>\n137    70  137 143 143 137 143 143<br>\n68    12  12  12  12  12  12  12<br>\n140    140 163 163 163 140 163 163<br>\n59    87  87  128 87  128 128 87<br>\n69    95  95  95  95  95  95  95</p>",
      "rawMarkdown": "**疑问**\n验证集 2753 样本下，原始不做tta，错了 54个。\n单看某次 tta 的预测结果，会错 70~80个\n使用6次投票结果，会错 63 个，效果还是比不做 tta 更差。。\n\n**详情**\n我用的 resnest50 + 一些 tricks, 不使用tta单折验证集上精度 0.9803，public board 上 0.978 \n然后在单折验证集上，根据网上思路试验了自己实现的 tta，即在预测每个batch时，重复使用一个 transfrom 多次，进下上下水平翻转、90旋转的操作（和训练时用的 aug 相同）\n\n即每个批次，除了原始仅进下 Normalize 处理的预测外，额外进下了5次 其他变化（Normalize 仅1次没重复），然后观察到\n1. 做了单此 tta 明显不如不做tta的\n2. 即使用投票结果，也不不错tta差\n3.部分非tta预测错的结果，再做了tta后，有可能得到正确样本\n\n**模型对变化不够稳定**\n通过验证集 tta 还观察到，验证集中，有 3% 数据，会因为做了 tta，得到不同分类结果；因为模型精度验证集上已经超过97%，不知道3%这个值是否异常，有哪些可以进一步优化吗\n \n\n\n**-- 错误例子**\n使用非tta原始结果，预测错了的例子，预计其他tta的结果\nlabel\tpre_org\tpre_tta0\tpre_tta1\tpre_tta2\tpre_tta3\tpre_tta4\tpre_vote\n1888\t153\t135\t135\t9\t135\t91\t9\t135\n1605\t56\t58\t58\t58\t58\t58\t58\t58\n2158\t68\t12\t12\t12\t12\t12\t12\t12\n1177 \t33\t154\t33\t33\t33\t154\t33\t33\n1135\t132\t69\t69\t69\t69\t69\t69\t69\n\n使用投票结果，但预测错误的例子\nlabel\tpre_org\tpre_tta0\tpre_tta1\tpre_tta2\tpre_tta3\tpre_tta4\tpre_vote\n137\t70\t137\t143\t143\t137\t143\t143\n68\t12\t12\t12\t12\t12\t12\t12\n140\t140\t163\t163\t163\t140\t163\t163\n59\t87\t87\t128\t87\t128\t128\t87\n69\t95\t95\t95\t95\t95\t95\t95",
      "votes": null
    },
    {
      "id": "1363867",
      "postDate": "06/24/2021 12:20:26",
      "content": "<p>前面学习TTA时候看了这一篇论文 <a href=\"https://arxiv.org/pdf/2011.11156.pdf\" target=\"_blank\">When and Why Test-Time Augmentation Works</a> 推荐你可以看一下。 其中就提到了你说的情况，TTA不是单纯的让你错误的预测变成正确，也有 corruption的情况。 不同的模型，不同的数据集，还有不同的TTA方法都可能带来不同的结果。[<em>kaggle把本地图片上传关了，囧，具体你看论文吧，写的很详细的</em>] <br>\n简单来说就是 你模型精度越高，TTA效果越不明显。 你训练的数据集越大TTA越弱。 后面也有对两个不同的分类任务，定性的分析了不同的TTA方法的效果分析.</p>\n<p>BTW, 之前有试过训练一个模型 30epoch，然后训练精度都有0.98以上了. 最后做TTA inference时候发现直接变成0.95. 最后在LB上也是类似的0.95，我觉得也许是我模型过拟合了，TTA也许可以帮忙cap过拟合？ 还要进一步研究下.</p>",
      "rawMarkdown": "前面学习TTA时候看了这一篇论文 [When and Why Test-Time Augmentation Works](https://arxiv.org/pdf/2011.11156.pdf) 推荐你可以看一下。 其中就提到了你说的情况，TTA不是单纯的让你错误的预测变成正确，也有 corruption的情况。 不同的模型，不同的数据集，还有不同的TTA方法都可能带来不同的结果。[*kaggle把本地图片上传关了，囧，具体你看论文吧，写的很详细的*] \n简单来说就是 你模型精度越高，TTA效果越不明显。 你训练的数据集越大TTA越弱。 后面也有对两个不同的分类任务，定性的分析了不同的TTA方法的效果分析.\n\nBTW, 之前有试过训练一个模型 30epoch，然后训练精度都有0.98以上了. 最后做TTA inference时候发现直接变成0.95. 最后在LB上也是类似的0.95，我觉得也许是我模型过拟合了，TTA也许可以帮忙cap过拟合？ 还要进一步研究下.",
      "votes": null
    },
    {
      "id": "1364018",
      "postDate": "06/24/2021 14:43:53",
      "content": "<p>感谢 Xichao👍，这次比赛真的学到挺多</p>\n<p>现在用了 resnest 50 + adamw_weight_decay + cosine学习率下降 + smoothing_label + 常规训练图像增强</p>\n<p>现在除了多模型 essemble 没有试（colab 没gpu额度），感觉单模型其他方法都差不多试过了，单折验证集差不多 0.981就到顶了</p>\n<p>不知道 0.980 -&gt; 0.988 这个跨度是怎么上去的，还有什么优化点能分享下吗 ~</p>",
      "rawMarkdown": "感谢 Xichao👍，这次比赛真的学到挺多\n\n\n现在用了 resnest 50 + adamw_weight_decay + cosine学习率下降 + smoothing_label + 常规训练图像增强\n\n现在除了多模型 essemble 没有试（colab 没gpu额度），感觉单模型其他方法都差不多试过了，单折验证集差不多 0.981就到顶了\n\n不知道 0.980 -> 0.988 这个跨度是怎么上去的，还有什么优化点能分享下吗 ~",
      "votes": null
    },
    {
      "id": "1364613",
      "postDate": "06/25/2021 05:27:20",
      "content": "<p>0.980已经很高了，这个数据下，没必要追求更高了；学到东西目的已经达到了</p>",
      "rawMarkdown": "0.980已经很高了，这个数据下，没必要追求更高了；学到东西目的已经达到了",
      "votes": null
    },
    {
      "id": "1364632",
      "postDate": "06/25/2021 05:42:28",
      "content": "<p>嗯，找一个合适的组合，炼金术启动o.0， colab等一天就行了，或者你关了notebook再试试也行，或者试试转TPU？ 实在不行买pro吧😹. </p>\n<p>essemble还是可以试试的，如果做了k-fold的话，多个模型融合起来的结果我觉得更可靠。 可以瞅瞅<a href=\"https://www.kaggle.com/amrmahmoud123/1-guide-to-ensembling-methods\" target=\"_blank\">这里</a>。 </p>\n<p>我在train的时候，保留模型并不是只保留acc最高的，我是先训练一个fold，通过设置合适的epoch, 看看效果，观察收敛情况，最后选择保留每个fold里，最末尾的几个epoch的模型。 主要是观察train 和 val 的loss的震荡，acc的变化等。也可以设置其他的保存方法, 应该根据你设定的epoch 和 lr_schedule或者其他综合来看. 对于不同lr_scheduler 的performance可以看看<a href=\"https://www.kaggle.com/isbhargav/guide-to-pytorch-learning-rate-scheduling#6.-CosineAnnealingLR\" target=\"_blank\">这里</a>。 我自己做 inference时候就使用不同的 weight 对最后保留的模型做的 average weighted essembling了.</p>\n<p>还有就是也许可以再试试 <a href=\"https://www.kaggle.com/virajbagal/mixup-cutmix-fmix-visualisations\" target=\"_blank\">mixup 和 cutmix等</a>方法 ，看seefun大佬前面也提到过</p>",
      "rawMarkdown": "嗯，找一个合适的组合，炼金术启动o.0， colab等一天就行了，或者你关了notebook再试试也行，或者试试转TPU？ 实在不行买pro吧😹. \n\nessemble还是可以试试的，如果做了k-fold的话，多个模型融合起来的结果我觉得更可靠。 可以瞅瞅[这里](https://www.kaggle.com/amrmahmoud123/1-guide-to-ensembling-methods)。 \n\n我在train的时候，保留模型并不是只保留acc最高的，我是先训练一个fold，通过设置合适的epoch, 看看效果，观察收敛情况，最后选择保留每个fold里，最末尾的几个epoch的模型。 主要是观察train 和 val 的loss的震荡，acc的变化等。也可以设置其他的保存方法, 应该根据你设定的epoch 和 lr_schedule或者其他综合来看. 对于不同lr_scheduler 的performance可以看看[这里](https://www.kaggle.com/isbhargav/guide-to-pytorch-learning-rate-scheduling#6.-CosineAnnealingLR)。 我自己做 inference时候就使用不同的 weight 对最后保留的模型做的 average weighted essembling了.\n\n还有就是也许可以再试试 [mixup 和 cutmix等](https://www.kaggle.com/virajbagal/mixup-cutmix-fmix-visualisations)方法 ，看seefun大佬前面也提到过",
      "votes": null
    },
    {
      "id": "1364640",
      "postDate": "06/25/2021 05:48:13",
      "content": "<p>同意的，我也觉得0.98很高了。 我train的accuracy最高都不过0.98的. 主要还是看你的策略组合，目前的LB过高也有可能有overfitting的风险？我还是担目标测试集里面有noisy。还是相信自己吧.   😆</p>",
      "rawMarkdown": "同意的，我也觉得0.98很高了。 我train的accuracy最高都不过0.98的. 主要还是看你的策略组合，目前的LB过高也有可能有overfitting的风险？我还是担目标测试集里面有noisy。还是相信自己吧.   😆",
      "votes": null
    },
    {
      "id": "1364651",
      "postDate": "06/25/2021 05:55:46",
      "content": "<p>同学能看下你验证集acc曲线吗，我的曲线波动很大，调低学习率之后爬升很慢，resnet卡在0.95了</p>",
      "rawMarkdown": "同学能看下你验证集acc曲线吗，我的曲线波动很大，调低学习率之后爬升很慢，resnet卡在0.95了",
      "votes": null
    },
    {
      "id": "1364705",
      "postDate": "06/25/2021 06:45:53",
      "content": "<p>我用 resnet50的时候，最终验证集准确率也就在 95 左右停止了，后来换成 resnest50 </p>\n<p>模型备注：（resnest50 沐神团队在 resnet 基础上优化，模型思路是对通道进行分组，组内按照每个通道 globalAvagePooling 后的信息，通过liner层变化后，进行attention，即组内每个通道会融合其他通道信息；以及模型训练用了一些tricks，实测其他不变，模型带来验证集1个点多一点的提升）</p>\n<p>现在kaggle 传不了图片和附件, 我把 resnest50 下，前 15 个epoch个验证集 loss 和 acc 放在下面了</p>\n<p>备注1：我用的colab免费版，当时训练一半没gpu限额了，log 只记录前15个epoch，验证集精度到 0.975<br>\n备注2：batch size 64，每训练 20 个 step 在验证集上跑一次，下面的 dev acc 间隔就是这样产生的</p>\n<p>-- dev acc 数据<br>\n[tensor(0.0712, dtype=torch.float64),<br>\n tensor(0.2060, dtype=torch.float64),<br>\n tensor(0.2804, dtype=torch.float64),<br>\n tensor(0.3335, dtype=torch.float64),<br>\n tensor(0.4254, dtype=torch.float64),<br>\n tensor(0.4853, dtype=torch.float64),<br>\n tensor(0.5347, dtype=torch.float64),<br>\n tensor(0.5630, dtype=torch.float64),<br>\n tensor(0.6302, dtype=torch.float64),<br>\n tensor(0.6567, dtype=torch.float64),<br>\n tensor(0.6829, dtype=torch.float64),<br>\n tensor(0.7036, dtype=torch.float64),<br>\n tensor(0.7544, dtype=torch.float64),<br>\n tensor(0.7323, dtype=torch.float64),<br>\n tensor(0.7559, dtype=torch.float64),<br>\n tensor(0.7719, dtype=torch.float64),<br>\n tensor(0.7973, dtype=torch.float64),<br>\n tensor(0.8039, dtype=torch.float64),<br>\n tensor(0.8082, dtype=torch.float64),<br>\n tensor(0.8195, dtype=torch.float64),<br>\n tensor(0.8206, dtype=torch.float64),<br>\n tensor(0.8398, dtype=torch.float64),<br>\n tensor(0.8536, dtype=torch.float64),<br>\n tensor(0.8609, dtype=torch.float64),<br>\n tensor(0.8525, dtype=torch.float64),<br>\n tensor(0.8700, dtype=torch.float64),<br>\n tensor(0.8801, dtype=torch.float64),<br>\n tensor(0.8859, dtype=torch.float64),<br>\n tensor(0.8943, dtype=torch.float64),<br>\n tensor(0.8790, dtype=torch.float64),<br>\n tensor(0.8925, dtype=torch.float64),<br>\n tensor(0.8721, dtype=torch.float64),<br>\n tensor(0.8758, dtype=torch.float64),<br>\n tensor(0.8907, dtype=torch.float64),<br>\n tensor(0.8878, dtype=torch.float64),<br>\n tensor(0.8976, dtype=torch.float64),<br>\n tensor(0.8888, dtype=torch.float64),<br>\n tensor(0.9027, dtype=torch.float64),<br>\n tensor(0.9077, dtype=torch.float64),<br>\n tensor(0.8987, dtype=torch.float64),<br>\n tensor(0.9114, dtype=torch.float64),<br>\n tensor(0.9241, dtype=torch.float64),<br>\n tensor(0.8947, dtype=torch.float64),<br>\n tensor(0.9165, dtype=torch.float64),<br>\n tensor(0.9226, dtype=torch.float64),<br>\n tensor(0.9128, dtype=torch.float64),<br>\n tensor(0.9194, dtype=torch.float64),<br>\n tensor(0.9186, dtype=torch.float64),<br>\n tensor(0.9277, dtype=torch.float64),<br>\n tensor(0.9237, dtype=torch.float64),<br>\n tensor(0.9194, dtype=torch.float64),<br>\n tensor(0.9353, dtype=torch.float64),<br>\n tensor(0.9277, dtype=torch.float64),<br>\n tensor(0.9332, dtype=torch.float64),<br>\n tensor(0.9234, dtype=torch.float64),<br>\n tensor(0.9288, dtype=torch.float64),<br>\n tensor(0.9241, dtype=torch.float64),<br>\n tensor(0.8979, dtype=torch.float64),<br>\n tensor(0.9313, dtype=torch.float64),<br>\n tensor(0.9150, dtype=torch.float64),<br>\n tensor(0.9343, dtype=torch.float64),<br>\n tensor(0.9346, dtype=torch.float64),<br>\n tensor(0.9339, dtype=torch.float64),<br>\n tensor(0.9219, dtype=torch.float64),<br>\n tensor(0.9281, dtype=torch.float64),<br>\n tensor(0.9299, dtype=torch.float64),<br>\n tensor(0.9372, dtype=torch.float64),<br>\n tensor(0.9372, dtype=torch.float64),<br>\n tensor(0.9419, dtype=torch.float64),<br>\n tensor(0.9361, dtype=torch.float64),<br>\n tensor(0.9437, dtype=torch.float64),<br>\n tensor(0.9390, dtype=torch.float64),<br>\n tensor(0.9408, dtype=torch.float64),<br>\n tensor(0.9328, dtype=torch.float64),<br>\n tensor(0.9477, dtype=torch.float64),<br>\n tensor(0.9448, dtype=torch.float64),<br>\n tensor(0.9426, dtype=torch.float64),<br>\n tensor(0.9448, dtype=torch.float64),<br>\n tensor(0.9531, dtype=torch.float64),<br>\n tensor(0.9452, dtype=torch.float64),<br>\n tensor(0.9510, dtype=torch.float64),<br>\n tensor(0.9506, dtype=torch.float64),<br>\n tensor(0.9510, dtype=torch.float64),<br>\n tensor(0.9560, dtype=torch.float64),<br>\n tensor(0.9430, dtype=torch.float64),<br>\n tensor(0.9433, dtype=torch.float64),<br>\n tensor(0.9473, dtype=torch.float64),<br>\n tensor(0.9386, dtype=torch.float64),<br>\n tensor(0.9466, dtype=torch.float64),<br>\n tensor(0.9408, dtype=torch.float64),<br>\n tensor(0.9502, dtype=torch.float64),<br>\n tensor(0.9448, dtype=torch.float64),<br>\n tensor(0.9433, dtype=torch.float64),<br>\n tensor(0.9484, dtype=torch.float64),<br>\n tensor(0.9466, dtype=torch.float64),<br>\n tensor(0.9448, dtype=torch.float64),<br>\n tensor(0.9437, dtype=torch.float64),<br>\n tensor(0.9513, dtype=torch.float64),<br>\n tensor(0.9495, dtype=torch.float64),<br>\n tensor(0.9586, dtype=torch.float64),<br>\n tensor(0.9495, dtype=torch.float64),<br>\n tensor(0.9531, dtype=torch.float64),<br>\n tensor(0.9444, dtype=torch.float64),<br>\n tensor(0.9502, dtype=torch.float64),<br>\n tensor(0.9568, dtype=torch.float64),<br>\n tensor(0.9408, dtype=torch.float64),<br>\n tensor(0.9553, dtype=torch.float64),<br>\n tensor(0.9550, dtype=torch.float64),<br>\n tensor(0.9571, dtype=torch.float64),<br>\n tensor(0.9462, dtype=torch.float64),<br>\n tensor(0.9557, dtype=torch.float64),<br>\n tensor(0.9575, dtype=torch.float64),<br>\n tensor(0.9659, dtype=torch.float64),<br>\n tensor(0.9633, dtype=torch.float64),<br>\n tensor(0.9600, dtype=torch.float64),<br>\n tensor(0.9590, dtype=torch.float64),<br>\n tensor(0.9611, dtype=torch.float64),<br>\n tensor(0.9629, dtype=torch.float64),<br>\n tensor(0.9590, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9608, dtype=torch.float64),<br>\n tensor(0.9655, dtype=torch.float64),<br>\n tensor(0.9608, dtype=torch.float64),<br>\n tensor(0.9648, dtype=torch.float64),<br>\n tensor(0.9593, dtype=torch.float64),<br>\n tensor(0.9586, dtype=torch.float64),<br>\n tensor(0.9669, dtype=torch.float64),<br>\n tensor(0.9648, dtype=torch.float64),<br>\n tensor(0.9648, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9597, dtype=torch.float64),<br>\n tensor(0.9582, dtype=torch.float64),<br>\n tensor(0.9517, dtype=torch.float64),<br>\n tensor(0.9709, dtype=torch.float64),<br>\n tensor(0.9666, dtype=torch.float64),<br>\n tensor(0.9615, dtype=torch.float64),<br>\n tensor(0.9662, dtype=torch.float64),<br>\n tensor(0.9593, dtype=torch.float64),<br>\n tensor(0.9611, dtype=torch.float64),<br>\n tensor(0.9608, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9659, dtype=torch.float64),<br>\n tensor(0.9604, dtype=torch.float64),<br>\n tensor(0.9662, dtype=torch.float64),<br>\n tensor(0.9648, dtype=torch.float64),<br>\n tensor(0.9611, dtype=torch.float64),<br>\n tensor(0.9629, dtype=torch.float64),<br>\n tensor(0.9640, dtype=torch.float64),<br>\n tensor(0.9684, dtype=torch.float64),<br>\n tensor(0.9640, dtype=torch.float64),<br>\n tensor(0.9691, dtype=torch.float64),<br>\n tensor(0.9659, dtype=torch.float64),<br>\n tensor(0.9724, dtype=torch.float64),<br>\n tensor(0.9669, dtype=torch.float64),<br>\n tensor(0.9691, dtype=torch.float64),<br>\n tensor(0.9593, dtype=torch.float64),<br>\n tensor(0.9626, dtype=torch.float64),<br>\n tensor(0.9669, dtype=torch.float64),<br>\n tensor(0.9673, dtype=torch.float64),<br>\n tensor(0.9699, dtype=torch.float64),<br>\n tensor(0.9680, dtype=torch.float64),<br>\n tensor(0.9659, dtype=torch.float64),<br>\n tensor(0.9684, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9666, dtype=torch.float64),<br>\n tensor(0.9699, dtype=torch.float64),<br>\n tensor(0.9695, dtype=torch.float64),<br>\n tensor(0.9731, dtype=torch.float64),<br>\n tensor(0.9691, dtype=torch.float64),<br>\n tensor(0.9702, dtype=torch.float64),<br>\n tensor(0.9691, dtype=torch.float64),<br>\n tensor(0.9699, dtype=torch.float64),<br>\n tensor(0.9702, dtype=torch.float64),<br>\n tensor(0.9677, dtype=torch.float64),<br>\n tensor(0.9651, dtype=torch.float64),<br>\n tensor(0.9655, dtype=torch.float64),<br>\n tensor(0.9699, dtype=torch.float64),<br>\n tensor(0.9724, dtype=torch.float64),<br>\n tensor(0.9713, dtype=torch.float64),<br>\n tensor(0.9706, dtype=torch.float64),<br>\n tensor(0.9764, dtype=torch.float64),<br>\n tensor(0.9742, dtype=torch.float64)]</p>",
      "rawMarkdown": "我用 resnet50的时候，最终验证集准确率也就在 95 左右停止了，后来换成 resnest50 \n\n模型备注：（resnest50 沐神团队在 resnet 基础上优化，模型思路是对通道进行分组，组内按照每个通道 globalAvagePooling 后的信息，通过liner层变化后，进行attention，即组内每个通道会融合其他通道信息；以及模型训练用了一些tricks，实测其他不变，模型带来验证集1个点多一点的提升）\n\n现在kaggle 传不了图片和附件, 我把 resnest50 下，前 15 个epoch个验证集 loss 和 acc 放在下面了\n\n备注1：我用的colab免费版，当时训练一半没gpu限额了，log 只记录前15个epoch，验证集精度到 0.975\n备注2：batch size 64，每训练 20 个 step 在验证集上跑一次，下面的 dev acc 间隔就是这样产生的\n\n-- dev acc 数据\n[tensor(0.0712, dtype=torch.float64),\n tensor(0.2060, dtype=torch.float64),\n tensor(0.2804, dtype=torch.float64),\n tensor(0.3335, dtype=torch.float64),\n tensor(0.4254, dtype=torch.float64),\n tensor(0.4853, dtype=torch.float64),\n tensor(0.5347, dtype=torch.float64),\n tensor(0.5630, dtype=torch.float64),\n tensor(0.6302, dtype=torch.float64),\n tensor(0.6567, dtype=torch.float64),\n tensor(0.6829, dtype=torch.float64),\n tensor(0.7036, dtype=torch.float64),\n tensor(0.7544, dtype=torch.float64),\n tensor(0.7323, dtype=torch.float64),\n tensor(0.7559, dtype=torch.float64),\n tensor(0.7719, dtype=torch.float64),\n tensor(0.7973, dtype=torch.float64),\n tensor(0.8039, dtype=torch.float64),\n tensor(0.8082, dtype=torch.float64),\n tensor(0.8195, dtype=torch.float64),\n tensor(0.8206, dtype=torch.float64),\n tensor(0.8398, dtype=torch.float64),\n tensor(0.8536, dtype=torch.float64),\n tensor(0.8609, dtype=torch.float64),\n tensor(0.8525, dtype=torch.float64),\n tensor(0.8700, dtype=torch.float64),\n tensor(0.8801, dtype=torch.float64),\n tensor(0.8859, dtype=torch.float64),\n tensor(0.8943, dtype=torch.float64),\n tensor(0.8790, dtype=torch.float64),\n tensor(0.8925, dtype=torch.float64),\n tensor(0.8721, dtype=torch.float64),\n tensor(0.8758, dtype=torch.float64),\n tensor(0.8907, dtype=torch.float64),\n tensor(0.8878, dtype=torch.float64),\n tensor(0.8976, dtype=torch.float64),\n tensor(0.8888, dtype=torch.float64),\n tensor(0.9027, dtype=torch.float64),\n tensor(0.9077, dtype=torch.float64),\n tensor(0.8987, dtype=torch.float64),\n tensor(0.9114, dtype=torch.float64),\n tensor(0.9241, dtype=torch.float64),\n tensor(0.8947, dtype=torch.float64),\n tensor(0.9165, dtype=torch.float64),\n tensor(0.9226, dtype=torch.float64),\n tensor(0.9128, dtype=torch.float64),\n tensor(0.9194, dtype=torch.float64),\n tensor(0.9186, dtype=torch.float64),\n tensor(0.9277, dtype=torch.float64),\n tensor(0.9237, dtype=torch.float64),\n tensor(0.9194, dtype=torch.float64),\n tensor(0.9353, dtype=torch.float64),\n tensor(0.9277, dtype=torch.float64),\n tensor(0.9332, dtype=torch.float64),\n tensor(0.9234, dtype=torch.float64),\n tensor(0.9288, dtype=torch.float64),\n tensor(0.9241, dtype=torch.float64),\n tensor(0.8979, dtype=torch.float64),\n tensor(0.9313, dtype=torch.float64),\n tensor(0.9150, dtype=torch.float64),\n tensor(0.9343, dtype=torch.float64),\n tensor(0.9346, dtype=torch.float64),\n tensor(0.9339, dtype=torch.float64),\n tensor(0.9219, dtype=torch.float64),\n tensor(0.9281, dtype=torch.float64),\n tensor(0.9299, dtype=torch.float64),\n tensor(0.9372, dtype=torch.float64),\n tensor(0.9372, dtype=torch.float64),\n tensor(0.9419, dtype=torch.float64),\n tensor(0.9361, dtype=torch.float64),\n tensor(0.9437, dtype=torch.float64),\n tensor(0.9390, dtype=torch.float64),\n tensor(0.9408, dtype=torch.float64),\n tensor(0.9328, dtype=torch.float64),\n tensor(0.9477, dtype=torch.float64),\n tensor(0.9448, dtype=torch.float64),\n tensor(0.9426, dtype=torch.float64),\n tensor(0.9448, dtype=torch.float64),\n tensor(0.9531, dtype=torch.float64),\n tensor(0.9452, dtype=torch.float64),\n tensor(0.9510, dtype=torch.float64),\n tensor(0.9506, dtype=torch.float64),\n tensor(0.9510, dtype=torch.float64),\n tensor(0.9560, dtype=torch.float64),\n tensor(0.9430, dtype=torch.float64),\n tensor(0.9433, dtype=torch.float64),\n tensor(0.9473, dtype=torch.float64),\n tensor(0.9386, dtype=torch.float64),\n tensor(0.9466, dtype=torch.float64),\n tensor(0.9408, dtype=torch.float64),\n tensor(0.9502, dtype=torch.float64),\n tensor(0.9448, dtype=torch.float64),\n tensor(0.9433, dtype=torch.float64),\n tensor(0.9484, dtype=torch.float64),\n tensor(0.9466, dtype=torch.float64),\n tensor(0.9448, dtype=torch.float64),\n tensor(0.9437, dtype=torch.float64),\n tensor(0.9513, dtype=torch.float64),\n tensor(0.9495, dtype=torch.float64),\n tensor(0.9586, dtype=torch.float64),\n tensor(0.9495, dtype=torch.float64),\n tensor(0.9531, dtype=torch.float64),\n tensor(0.9444, dtype=torch.float64),\n tensor(0.9502, dtype=torch.float64),\n tensor(0.9568, dtype=torch.float64),\n tensor(0.9408, dtype=torch.float64),\n tensor(0.9553, dtype=torch.float64),\n tensor(0.9550, dtype=torch.float64),\n tensor(0.9571, dtype=torch.float64),\n tensor(0.9462, dtype=torch.float64),\n tensor(0.9557, dtype=torch.float64),\n tensor(0.9575, dtype=torch.float64),\n tensor(0.9659, dtype=torch.float64),\n tensor(0.9633, dtype=torch.float64),\n tensor(0.9600, dtype=torch.float64),\n tensor(0.9590, dtype=torch.float64),\n tensor(0.9611, dtype=torch.float64),\n tensor(0.9629, dtype=torch.float64),\n tensor(0.9590, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9608, dtype=torch.float64),\n tensor(0.9655, dtype=torch.float64),\n tensor(0.9608, dtype=torch.float64),\n tensor(0.9648, dtype=torch.float64),\n tensor(0.9593, dtype=torch.float64),\n tensor(0.9586, dtype=torch.float64),\n tensor(0.9669, dtype=torch.float64),\n tensor(0.9648, dtype=torch.float64),\n tensor(0.9648, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9597, dtype=torch.float64),\n tensor(0.9582, dtype=torch.float64),\n tensor(0.9517, dtype=torch.float64),\n tensor(0.9709, dtype=torch.float64),\n tensor(0.9666, dtype=torch.float64),\n tensor(0.9615, dtype=torch.float64),\n tensor(0.9662, dtype=torch.float64),\n tensor(0.9593, dtype=torch.float64),\n tensor(0.9611, dtype=torch.float64),\n tensor(0.9608, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9659, dtype=torch.float64),\n tensor(0.9604, dtype=torch.float64),\n tensor(0.9662, dtype=torch.float64),\n tensor(0.9648, dtype=torch.float64),\n tensor(0.9611, dtype=torch.float64),\n tensor(0.9629, dtype=torch.float64),\n tensor(0.9640, dtype=torch.float64),\n tensor(0.9684, dtype=torch.float64),\n tensor(0.9640, dtype=torch.float64),\n tensor(0.9691, dtype=torch.float64),\n tensor(0.9659, dtype=torch.float64),\n tensor(0.9724, dtype=torch.float64),\n tensor(0.9669, dtype=torch.float64),\n tensor(0.9691, dtype=torch.float64),\n tensor(0.9593, dtype=torch.float64),\n tensor(0.9626, dtype=torch.float64),\n tensor(0.9669, dtype=torch.float64),\n tensor(0.9673, dtype=torch.float64),\n tensor(0.9699, dtype=torch.float64),\n tensor(0.9680, dtype=torch.float64),\n tensor(0.9659, dtype=torch.float64),\n tensor(0.9684, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9666, dtype=torch.float64),\n tensor(0.9699, dtype=torch.float64),\n tensor(0.9695, dtype=torch.float64),\n tensor(0.9731, dtype=torch.float64),\n tensor(0.9691, dtype=torch.float64),\n tensor(0.9702, dtype=torch.float64),\n tensor(0.9691, dtype=torch.float64),\n tensor(0.9699, dtype=torch.float64),\n tensor(0.9702, dtype=torch.float64),\n tensor(0.9677, dtype=torch.float64),\n tensor(0.9651, dtype=torch.float64),\n tensor(0.9655, dtype=torch.float64),\n tensor(0.9699, dtype=torch.float64),\n tensor(0.9724, dtype=torch.float64),\n tensor(0.9713, dtype=torch.float64),\n tensor(0.9706, dtype=torch.float64),\n tensor(0.9764, dtype=torch.float64),\n tensor(0.9742, dtype=torch.float64)]",
      "votes": null
    },
    {
      "id": "1364712",
      "postDate": "06/25/2021 06:50:39",
      "content": "<p>非常感谢❤️，我在优化中，借鉴了你提供的 notebook 中的思路，还没试的还剩更复制的 aug、k折以及RangerLars优化器</p>\n<p>想问下你notebook中用的这个优化器，和adamw对比的话，感觉会有提升吗：</p>\n<h1>RAdam + LARS + LookAHead</h1>\n<h1>Lookahead implementation from <a href=\"https://github.com/lonePatient/lookahead_pytorch/blob/master/optimizer.py\" target=\"_blank\">https://github.com/lonePatient/lookahead_pytorch/blob/master/optimizer.py</a></h1>\n<h1>RAdam + LARS implementation from <a href=\"https://gist.github.com/redknightlois/c4023d393eb8f92bb44b2ab582d7ec20\" target=\"_blank\">https://gist.github.com/redknightlois/c4023d393eb8f92bb44b2ab582d7ec20</a></h1>\n<p>def Over9000(params, alpha=0.5, k=6, *args, **kwargs):<br>\n     ralamb = Ralamb(params, *args, **kwargs)<br>\n     return Lookahead(ralamb, alpha, k)</p>\n<p>RangerLars = Over9000</p>",
      "rawMarkdown": "非常感谢❤️，我在优化中，借鉴了你提供的 notebook 中的思路，还没试的还剩更复制的 aug、k折以及RangerLars优化器\n\n想问下你notebook中用的这个优化器，和adamw对比的话，感觉会有提升吗：\n\n# RAdam + LARS + LookAHead\n# Lookahead implementation from https://github.com/lonePatient/lookahead_pytorch/blob/master/optimizer.py\n# RAdam + LARS implementation from https://gist.github.com/redknightlois/c4023d393eb8f92bb44b2ab582d7ec20\n\ndef Over9000(params, alpha=0.5, k=6, *args, **kwargs):\n     ralamb = Ralamb(params, *args, **kwargs)\n     return Lookahead(ralamb, alpha, k)\n\nRangerLars = Over9000",
      "votes": null
    },
    {
      "id": "1364716",
      "postDate": "06/25/2021 06:51:39",
      "content": "<p>再次感谢耐心的解答！</p>",
      "rawMarkdown": "再次感谢耐心的解答！",
      "votes": null
    },
    {
      "id": "1364797",
      "postDate": "06/25/2021 08:02:57",
      "content": "<p>请问你这个模型跑的时候加image augmentation了吗? </p>",
      "rawMarkdown": "请问你这个模型跑的时候加image augmentation了吗?",
      "votes": null
    },
    {
      "id": "1365149",
      "postDate": "06/25/2021 13:30:14",
      "content": "<p>加了水平、垂直翻转，还有正负90度旋转</p>\n<p>transform_train = transforms.Compose([<br>\n        transforms.RandomHorizontalFlip(),<br>\n        transforms.RandomVerticalFlip(),<br>\n        transforms.RandomRotation(90),<br>\n        transforms.ConvertImageDtype(torch.float),<br>\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>\n])</p>",
      "rawMarkdown": "加了水平、垂直翻转，还有正负90度旋转\n\ntransform_train = transforms.Compose([\n        transforms.RandomHorizontalFlip(),\n        transforms.RandomVerticalFlip(),\n        transforms.RandomRotation(90),\n        transforms.ConvertImageDtype(torch.float),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n])",
      "votes": null
    },
    {
      "id": "1365228",
      "postDate": "06/25/2021 14:16:37",
      "content": "<p>谢谢同学 谢谢你 我看看</p>",
      "rawMarkdown": "谢谢同学 谢谢你 我看看",
      "votes": null
    },
    {
      "id": "1365693",
      "postDate": "06/26/2021 03:25:04",
      "content": "<p>优化器可以当成超参数之一来调整，还是应该用Adam和SGD作为baseline，之后再尝试其他优化器。</p>",
      "rawMarkdown": "优化器可以当成超参数之一来调整，还是应该用Adam和SGD作为baseline，之后再尝试其他优化器。",
      "votes": null
    },
    {
      "id": "1367796",
      "postDate": "06/28/2021 05:16:36",
      "content": "<p>个人实测，CutMix效果很好，提升了一个点左右。</p>",
      "rawMarkdown": "个人实测，CutMix效果很好，提升了一个点左右。",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1363867,
      "author_name": "sheepwang",
      "author_url": "",
      "post_date": "06/24/2021 12:20:26",
      "content": "<p>前面学习TTA时候看了这一篇论文 <a href=\"https://arxiv.org/pdf/2011.11156.pdf\" target=\"_blank\">When and Why Test-Time Augmentation Works</a> 推荐你可以看一下。 其中就提到了你说的情况，TTA不是单纯的让你错误的预测变成正确，也有 corruption的情况。 不同的模型，不同的数据集，还有不同的TTA方法都可能带来不同的结果。[<em>kaggle把本地图片上传关了，囧，具体你看论文吧，写的很详细的</em>] <br>\n简单来说就是 你模型精度越高，TTA效果越不明显。 你训练的数据集越大TTA越弱。 后面也有对两个不同的分类任务，定性的分析了不同的TTA方法的效果分析.</p>\n<p>BTW, 之前有试过训练一个模型 30epoch，然后训练精度都有0.98以上了. 最后做TTA inference时候发现直接变成0.95. 最后在LB上也是类似的0.95，我觉得也许是我模型过拟合了，TTA也许可以帮忙cap过拟合？ 还要进一步研究下.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1364018,
          "author_name": "niangaopx",
          "author_url": "",
          "post_date": "06/24/2021 14:43:53",
          "content": "<p>感谢 Xichao👍，这次比赛真的学到挺多</p>\n<p>现在用了 resnest 50 + adamw_weight_decay + cosine学习率下降 + smoothing_label + 常规训练图像增强</p>\n<p>现在除了多模型 essemble 没有试（colab 没gpu额度），感觉单模型其他方法都差不多试过了，单折验证集差不多 0.981就到顶了</p>\n<p>不知道 0.980 -&gt; 0.988 这个跨度是怎么上去的，还有什么优化点能分享下吗 ~</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1364613,
          "author_name": "seefun",
          "author_url": "",
          "post_date": "06/25/2021 05:27:20",
          "content": "<p>0.980已经很高了，这个数据下，没必要追求更高了；学到东西目的已经达到了</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1364632,
          "author_name": "sheepwang",
          "author_url": "",
          "post_date": "06/25/2021 05:42:28",
          "content": "<p>嗯，找一个合适的组合，炼金术启动o.0， colab等一天就行了，或者你关了notebook再试试也行，或者试试转TPU？ 实在不行买pro吧😹. </p>\n<p>essemble还是可以试试的，如果做了k-fold的话，多个模型融合起来的结果我觉得更可靠。 可以瞅瞅<a href=\"https://www.kaggle.com/amrmahmoud123/1-guide-to-ensembling-methods\" target=\"_blank\">这里</a>。 </p>\n<p>我在train的时候，保留模型并不是只保留acc最高的，我是先训练一个fold，通过设置合适的epoch, 看看效果，观察收敛情况，最后选择保留每个fold里，最末尾的几个epoch的模型。 主要是观察train 和 val 的loss的震荡，acc的变化等。也可以设置其他的保存方法, 应该根据你设定的epoch 和 lr_schedule或者其他综合来看. 对于不同lr_scheduler 的performance可以看看<a href=\"https://www.kaggle.com/isbhargav/guide-to-pytorch-learning-rate-scheduling#6.-CosineAnnealingLR\" target=\"_blank\">这里</a>。 我自己做 inference时候就使用不同的 weight 对最后保留的模型做的 average weighted essembling了.</p>\n<p>还有就是也许可以再试试 <a href=\"https://www.kaggle.com/virajbagal/mixup-cutmix-fmix-visualisations\" target=\"_blank\">mixup 和 cutmix等</a>方法 ，看seefun大佬前面也提到过</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1364640,
          "author_name": "sheepwang",
          "author_url": "",
          "post_date": "06/25/2021 05:48:13",
          "content": "<p>同意的，我也觉得0.98很高了。 我train的accuracy最高都不过0.98的. 主要还是看你的策略组合，目前的LB过高也有可能有overfitting的风险？我还是担目标测试集里面有noisy。还是相信自己吧.   😆</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1364712,
          "author_name": "niangaopx",
          "author_url": "",
          "post_date": "06/25/2021 06:50:39",
          "content": "<p>非常感谢❤️，我在优化中，借鉴了你提供的 notebook 中的思路，还没试的还剩更复制的 aug、k折以及RangerLars优化器</p>\n<p>想问下你notebook中用的这个优化器，和adamw对比的话，感觉会有提升吗：</p>\n<h1>RAdam + LARS + LookAHead</h1>\n<h1>Lookahead implementation from <a href=\"https://github.com/lonePatient/lookahead_pytorch/blob/master/optimizer.py\" target=\"_blank\">https://github.com/lonePatient/lookahead_pytorch/blob/master/optimizer.py</a></h1>\n<h1>RAdam + LARS implementation from <a href=\"https://gist.github.com/redknightlois/c4023d393eb8f92bb44b2ab582d7ec20\" target=\"_blank\">https://gist.github.com/redknightlois/c4023d393eb8f92bb44b2ab582d7ec20</a></h1>\n<p>def Over9000(params, alpha=0.5, k=6, *args, **kwargs):<br>\n     ralamb = Ralamb(params, *args, **kwargs)<br>\n     return Lookahead(ralamb, alpha, k)</p>\n<p>RangerLars = Over9000</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1364716,
          "author_name": "niangaopx",
          "author_url": "",
          "post_date": "06/25/2021 06:51:39",
          "content": "<p>再次感谢耐心的解答！</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1365693,
          "author_name": "wangtf96",
          "author_url": "",
          "post_date": "06/26/2021 03:25:04",
          "content": "<p>优化器可以当成超参数之一来调整，还是应该用Adam和SGD作为baseline，之后再尝试其他优化器。</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1367796,
          "author_name": "charlesyyun",
          "author_url": "",
          "post_date": "06/28/2021 05:16:36",
          "content": "<p>个人实测，CutMix效果很好，提升了一个点左右。</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1364651,
      "author_name": "bamboopu1996",
      "author_url": "",
      "post_date": "06/25/2021 05:55:46",
      "content": "<p>同学能看下你验证集acc曲线吗，我的曲线波动很大，调低学习率之后爬升很慢，resnet卡在0.95了</p>",
      "votes": null,
      "replies": [
        {
          "id": 1364705,
          "author_name": "niangaopx",
          "author_url": "",
          "post_date": "06/25/2021 06:45:53",
          "content": "<p>我用 resnet50的时候，最终验证集准确率也就在 95 左右停止了，后来换成 resnest50 </p>\n<p>模型备注：（resnest50 沐神团队在 resnet 基础上优化，模型思路是对通道进行分组，组内按照每个通道 globalAvagePooling 后的信息，通过liner层变化后，进行attention，即组内每个通道会融合其他通道信息；以及模型训练用了一些tricks，实测其他不变，模型带来验证集1个点多一点的提升）</p>\n<p>现在kaggle 传不了图片和附件, 我把 resnest50 下，前 15 个epoch个验证集 loss 和 acc 放在下面了</p>\n<p>备注1：我用的colab免费版，当时训练一半没gpu限额了，log 只记录前15个epoch，验证集精度到 0.975<br>\n备注2：batch size 64，每训练 20 个 step 在验证集上跑一次，下面的 dev acc 间隔就是这样产生的</p>\n<p>-- dev acc 数据<br>\n[tensor(0.0712, dtype=torch.float64),<br>\n tensor(0.2060, dtype=torch.float64),<br>\n tensor(0.2804, dtype=torch.float64),<br>\n tensor(0.3335, dtype=torch.float64),<br>\n tensor(0.4254, dtype=torch.float64),<br>\n tensor(0.4853, dtype=torch.float64),<br>\n tensor(0.5347, dtype=torch.float64),<br>\n tensor(0.5630, dtype=torch.float64),<br>\n tensor(0.6302, dtype=torch.float64),<br>\n tensor(0.6567, dtype=torch.float64),<br>\n tensor(0.6829, dtype=torch.float64),<br>\n tensor(0.7036, dtype=torch.float64),<br>\n tensor(0.7544, dtype=torch.float64),<br>\n tensor(0.7323, dtype=torch.float64),<br>\n tensor(0.7559, dtype=torch.float64),<br>\n tensor(0.7719, dtype=torch.float64),<br>\n tensor(0.7973, dtype=torch.float64),<br>\n tensor(0.8039, dtype=torch.float64),<br>\n tensor(0.8082, dtype=torch.float64),<br>\n tensor(0.8195, dtype=torch.float64),<br>\n tensor(0.8206, dtype=torch.float64),<br>\n tensor(0.8398, dtype=torch.float64),<br>\n tensor(0.8536, dtype=torch.float64),<br>\n tensor(0.8609, dtype=torch.float64),<br>\n tensor(0.8525, dtype=torch.float64),<br>\n tensor(0.8700, dtype=torch.float64),<br>\n tensor(0.8801, dtype=torch.float64),<br>\n tensor(0.8859, dtype=torch.float64),<br>\n tensor(0.8943, dtype=torch.float64),<br>\n tensor(0.8790, dtype=torch.float64),<br>\n tensor(0.8925, dtype=torch.float64),<br>\n tensor(0.8721, dtype=torch.float64),<br>\n tensor(0.8758, dtype=torch.float64),<br>\n tensor(0.8907, dtype=torch.float64),<br>\n tensor(0.8878, dtype=torch.float64),<br>\n tensor(0.8976, dtype=torch.float64),<br>\n tensor(0.8888, dtype=torch.float64),<br>\n tensor(0.9027, dtype=torch.float64),<br>\n tensor(0.9077, dtype=torch.float64),<br>\n tensor(0.8987, dtype=torch.float64),<br>\n tensor(0.9114, dtype=torch.float64),<br>\n tensor(0.9241, dtype=torch.float64),<br>\n tensor(0.8947, dtype=torch.float64),<br>\n tensor(0.9165, dtype=torch.float64),<br>\n tensor(0.9226, dtype=torch.float64),<br>\n tensor(0.9128, dtype=torch.float64),<br>\n tensor(0.9194, dtype=torch.float64),<br>\n tensor(0.9186, dtype=torch.float64),<br>\n tensor(0.9277, dtype=torch.float64),<br>\n tensor(0.9237, dtype=torch.float64),<br>\n tensor(0.9194, dtype=torch.float64),<br>\n tensor(0.9353, dtype=torch.float64),<br>\n tensor(0.9277, dtype=torch.float64),<br>\n tensor(0.9332, dtype=torch.float64),<br>\n tensor(0.9234, dtype=torch.float64),<br>\n tensor(0.9288, dtype=torch.float64),<br>\n tensor(0.9241, dtype=torch.float64),<br>\n tensor(0.8979, dtype=torch.float64),<br>\n tensor(0.9313, dtype=torch.float64),<br>\n tensor(0.9150, dtype=torch.float64),<br>\n tensor(0.9343, dtype=torch.float64),<br>\n tensor(0.9346, dtype=torch.float64),<br>\n tensor(0.9339, dtype=torch.float64),<br>\n tensor(0.9219, dtype=torch.float64),<br>\n tensor(0.9281, dtype=torch.float64),<br>\n tensor(0.9299, dtype=torch.float64),<br>\n tensor(0.9372, dtype=torch.float64),<br>\n tensor(0.9372, dtype=torch.float64),<br>\n tensor(0.9419, dtype=torch.float64),<br>\n tensor(0.9361, dtype=torch.float64),<br>\n tensor(0.9437, dtype=torch.float64),<br>\n tensor(0.9390, dtype=torch.float64),<br>\n tensor(0.9408, dtype=torch.float64),<br>\n tensor(0.9328, dtype=torch.float64),<br>\n tensor(0.9477, dtype=torch.float64),<br>\n tensor(0.9448, dtype=torch.float64),<br>\n tensor(0.9426, dtype=torch.float64),<br>\n tensor(0.9448, dtype=torch.float64),<br>\n tensor(0.9531, dtype=torch.float64),<br>\n tensor(0.9452, dtype=torch.float64),<br>\n tensor(0.9510, dtype=torch.float64),<br>\n tensor(0.9506, dtype=torch.float64),<br>\n tensor(0.9510, dtype=torch.float64),<br>\n tensor(0.9560, dtype=torch.float64),<br>\n tensor(0.9430, dtype=torch.float64),<br>\n tensor(0.9433, dtype=torch.float64),<br>\n tensor(0.9473, dtype=torch.float64),<br>\n tensor(0.9386, dtype=torch.float64),<br>\n tensor(0.9466, dtype=torch.float64),<br>\n tensor(0.9408, dtype=torch.float64),<br>\n tensor(0.9502, dtype=torch.float64),<br>\n tensor(0.9448, dtype=torch.float64),<br>\n tensor(0.9433, dtype=torch.float64),<br>\n tensor(0.9484, dtype=torch.float64),<br>\n tensor(0.9466, dtype=torch.float64),<br>\n tensor(0.9448, dtype=torch.float64),<br>\n tensor(0.9437, dtype=torch.float64),<br>\n tensor(0.9513, dtype=torch.float64),<br>\n tensor(0.9495, dtype=torch.float64),<br>\n tensor(0.9586, dtype=torch.float64),<br>\n tensor(0.9495, dtype=torch.float64),<br>\n tensor(0.9531, dtype=torch.float64),<br>\n tensor(0.9444, dtype=torch.float64),<br>\n tensor(0.9502, dtype=torch.float64),<br>\n tensor(0.9568, dtype=torch.float64),<br>\n tensor(0.9408, dtype=torch.float64),<br>\n tensor(0.9553, dtype=torch.float64),<br>\n tensor(0.9550, dtype=torch.float64),<br>\n tensor(0.9571, dtype=torch.float64),<br>\n tensor(0.9462, dtype=torch.float64),<br>\n tensor(0.9557, dtype=torch.float64),<br>\n tensor(0.9575, dtype=torch.float64),<br>\n tensor(0.9659, dtype=torch.float64),<br>\n tensor(0.9633, dtype=torch.float64),<br>\n tensor(0.9600, dtype=torch.float64),<br>\n tensor(0.9590, dtype=torch.float64),<br>\n tensor(0.9611, dtype=torch.float64),<br>\n tensor(0.9629, dtype=torch.float64),<br>\n tensor(0.9590, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9608, dtype=torch.float64),<br>\n tensor(0.9655, dtype=torch.float64),<br>\n tensor(0.9608, dtype=torch.float64),<br>\n tensor(0.9648, dtype=torch.float64),<br>\n tensor(0.9593, dtype=torch.float64),<br>\n tensor(0.9586, dtype=torch.float64),<br>\n tensor(0.9669, dtype=torch.float64),<br>\n tensor(0.9648, dtype=torch.float64),<br>\n tensor(0.9648, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9597, dtype=torch.float64),<br>\n tensor(0.9582, dtype=torch.float64),<br>\n tensor(0.9517, dtype=torch.float64),<br>\n tensor(0.9709, dtype=torch.float64),<br>\n tensor(0.9666, dtype=torch.float64),<br>\n tensor(0.9615, dtype=torch.float64),<br>\n tensor(0.9662, dtype=torch.float64),<br>\n tensor(0.9593, dtype=torch.float64),<br>\n tensor(0.9611, dtype=torch.float64),<br>\n tensor(0.9608, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9659, dtype=torch.float64),<br>\n tensor(0.9604, dtype=torch.float64),<br>\n tensor(0.9662, dtype=torch.float64),<br>\n tensor(0.9648, dtype=torch.float64),<br>\n tensor(0.9611, dtype=torch.float64),<br>\n tensor(0.9629, dtype=torch.float64),<br>\n tensor(0.9640, dtype=torch.float64),<br>\n tensor(0.9684, dtype=torch.float64),<br>\n tensor(0.9640, dtype=torch.float64),<br>\n tensor(0.9691, dtype=torch.float64),<br>\n tensor(0.9659, dtype=torch.float64),<br>\n tensor(0.9724, dtype=torch.float64),<br>\n tensor(0.9669, dtype=torch.float64),<br>\n tensor(0.9691, dtype=torch.float64),<br>\n tensor(0.9593, dtype=torch.float64),<br>\n tensor(0.9626, dtype=torch.float64),<br>\n tensor(0.9669, dtype=torch.float64),<br>\n tensor(0.9673, dtype=torch.float64),<br>\n tensor(0.9699, dtype=torch.float64),<br>\n tensor(0.9680, dtype=torch.float64),<br>\n tensor(0.9659, dtype=torch.float64),<br>\n tensor(0.9684, dtype=torch.float64),<br>\n tensor(0.9637, dtype=torch.float64),<br>\n tensor(0.9666, dtype=torch.float64),<br>\n tensor(0.9699, dtype=torch.float64),<br>\n tensor(0.9695, dtype=torch.float64),<br>\n tensor(0.9731, dtype=torch.float64),<br>\n tensor(0.9691, dtype=torch.float64),<br>\n tensor(0.9702, dtype=torch.float64),<br>\n tensor(0.9691, dtype=torch.float64),<br>\n tensor(0.9699, dtype=torch.float64),<br>\n tensor(0.9702, dtype=torch.float64),<br>\n tensor(0.9677, dtype=torch.float64),<br>\n tensor(0.9651, dtype=torch.float64),<br>\n tensor(0.9655, dtype=torch.float64),<br>\n tensor(0.9699, dtype=torch.float64),<br>\n tensor(0.9724, dtype=torch.float64),<br>\n tensor(0.9713, dtype=torch.float64),<br>\n tensor(0.9706, dtype=torch.float64),<br>\n tensor(0.9764, dtype=torch.float64),<br>\n tensor(0.9742, dtype=torch.float64)]</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1364797,
          "author_name": "hsiungfei",
          "author_url": "",
          "post_date": "06/25/2021 08:02:57",
          "content": "<p>请问你这个模型跑的时候加image augmentation了吗? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1365149,
          "author_name": "niangaopx",
          "author_url": "",
          "post_date": "06/25/2021 13:30:14",
          "content": "<p>加了水平、垂直翻转，还有正负90度旋转</p>\n<p>transform_train = transforms.Compose([<br>\n        transforms.RandomHorizontalFlip(),<br>\n        transforms.RandomVerticalFlip(),<br>\n        transforms.RandomRotation(90),<br>\n        transforms.ConvertImageDtype(torch.float),<br>\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>\n])</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1365228,
          "author_name": "bamboopu1996",
          "author_url": "",
          "post_date": "06/25/2021 14:16:37",
          "content": "<p>谢谢同学 谢谢你 我看看</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1363744": "**疑问**\n验证集 2753 样本下，原始不做tta，错了 54个。\n单看某次 tta 的预测结果，会错 70~80个\n使用6次投票结果，会错 63 个，效果还是比不做 tta 更差。。\n\n**详情**\n我用的 resnest50 + 一些 tricks, 不使用tta单折验证集上精度 0.9803，public board 上 0.978 \n然后在单折验证集上，根据网上思路试验了自己实现的 tta，即在预测每个batch时，重复使用一个 transfrom 多次，进下上下水平翻转、90旋转的操作（和训练时用的 aug 相同）\n\n即每个批次，除了原始仅进下 Normalize 处理的预测外，额外进下了5次 其他变化（Normalize 仅1次没重复），然后观察到\n1. 做了单此 tta 明显不如不做tta的\n2. 即使用投票结果，也不不错tta差\n3.部分非tta预测错的结果，再做了tta后，有可能得到正确样本\n\n**模型对变化不够稳定**\n通过验证集 tta 还观察到，验证集中，有 3% 数据，会因为做了 tta，得到不同分类结果；因为模型精度验证集上已经超过97%，不知道3%这个值是否异常，有哪些可以进一步优化吗\n \n\n\n**-- 错误例子**\n使用非tta原始结果，预测错了的例子，预计其他tta的结果\nlabel\tpre_org\tpre_tta0\tpre_tta1\tpre_tta2\tpre_tta3\tpre_tta4\tpre_vote\n1888\t153\t135\t135\t9\t135\t91\t9\t135\n1605\t56\t58\t58\t58\t58\t58\t58\t58\n2158\t68\t12\t12\t12\t12\t12\t12\t12\n1177 \t33\t154\t33\t33\t33\t154\t33\t33\n1135\t132\t69\t69\t69\t69\t69\t69\t69\n\n使用投票结果，但预测错误的例子\nlabel\tpre_org\tpre_tta0\tpre_tta1\tpre_tta2\tpre_tta3\tpre_tta4\tpre_vote\n137\t70\t137\t143\t143\t137\t143\t143\n68\t12\t12\t12\t12\t12\t12\t12\n140\t140\t163\t163\t163\t140\t163\t163\n59\t87\t87\t128\t87\t128\t128\t87\n69\t95\t95\t95\t95\t95\t95\t95",
    "1363867": "前面学习TTA时候看了这一篇论文 [When and Why Test-Time Augmentation Works](https://arxiv.org/pdf/2011.11156.pdf) 推荐你可以看一下。 其中就提到了你说的情况，TTA不是单纯的让你错误的预测变成正确，也有 corruption的情况。 不同的模型，不同的数据集，还有不同的TTA方法都可能带来不同的结果。[*kaggle把本地图片上传关了，囧，具体你看论文吧，写的很详细的*] \n简单来说就是 你模型精度越高，TTA效果越不明显。 你训练的数据集越大TTA越弱。 后面也有对两个不同的分类任务，定性的分析了不同的TTA方法的效果分析.\n\nBTW, 之前有试过训练一个模型 30epoch，然后训练精度都有0.98以上了. 最后做TTA inference时候发现直接变成0.95. 最后在LB上也是类似的0.95，我觉得也许是我模型过拟合了，TTA也许可以帮忙cap过拟合？ 还要进一步研究下.",
    "1364018": "感谢 Xichao👍，这次比赛真的学到挺多\n\n\n现在用了 resnest 50 + adamw_weight_decay + cosine学习率下降 + smoothing_label + 常规训练图像增强\n\n现在除了多模型 essemble 没有试（colab 没gpu额度），感觉单模型其他方法都差不多试过了，单折验证集差不多 0.981就到顶了\n\n不知道 0.980 -> 0.988 这个跨度是怎么上去的，还有什么优化点能分享下吗 ~",
    "1364613": "0.980已经很高了，这个数据下，没必要追求更高了；学到东西目的已经达到了",
    "1364632": "嗯，找一个合适的组合，炼金术启动o.0， colab等一天就行了，或者你关了notebook再试试也行，或者试试转TPU？ 实在不行买pro吧😹. \n\nessemble还是可以试试的，如果做了k-fold的话，多个模型融合起来的结果我觉得更可靠。 可以瞅瞅[这里](https://www.kaggle.com/amrmahmoud123/1-guide-to-ensembling-methods)。 \n\n我在train的时候，保留模型并不是只保留acc最高的，我是先训练一个fold，通过设置合适的epoch, 看看效果，观察收敛情况，最后选择保留每个fold里，最末尾的几个epoch的模型。 主要是观察train 和 val 的loss的震荡，acc的变化等。也可以设置其他的保存方法, 应该根据你设定的epoch 和 lr_schedule或者其他综合来看. 对于不同lr_scheduler 的performance可以看看[这里](https://www.kaggle.com/isbhargav/guide-to-pytorch-learning-rate-scheduling#6.-CosineAnnealingLR)。 我自己做 inference时候就使用不同的 weight 对最后保留的模型做的 average weighted essembling了.\n\n还有就是也许可以再试试 [mixup 和 cutmix等](https://www.kaggle.com/virajbagal/mixup-cutmix-fmix-visualisations)方法 ，看seefun大佬前面也提到过",
    "1364640": "同意的，我也觉得0.98很高了。 我train的accuracy最高都不过0.98的. 主要还是看你的策略组合，目前的LB过高也有可能有overfitting的风险？我还是担目标测试集里面有noisy。还是相信自己吧.   😆",
    "1364651": "同学能看下你验证集acc曲线吗，我的曲线波动很大，调低学习率之后爬升很慢，resnet卡在0.95了",
    "1364705": "我用 resnet50的时候，最终验证集准确率也就在 95 左右停止了，后来换成 resnest50 \n\n模型备注：（resnest50 沐神团队在 resnet 基础上优化，模型思路是对通道进行分组，组内按照每个通道 globalAvagePooling 后的信息，通过liner层变化后，进行attention，即组内每个通道会融合其他通道信息；以及模型训练用了一些tricks，实测其他不变，模型带来验证集1个点多一点的提升）\n\n现在kaggle 传不了图片和附件, 我把 resnest50 下，前 15 个epoch个验证集 loss 和 acc 放在下面了\n\n备注1：我用的colab免费版，当时训练一半没gpu限额了，log 只记录前15个epoch，验证集精度到 0.975\n备注2：batch size 64，每训练 20 个 step 在验证集上跑一次，下面的 dev acc 间隔就是这样产生的\n\n-- dev acc 数据\n[tensor(0.0712, dtype=torch.float64),\n tensor(0.2060, dtype=torch.float64),\n tensor(0.2804, dtype=torch.float64),\n tensor(0.3335, dtype=torch.float64),\n tensor(0.4254, dtype=torch.float64),\n tensor(0.4853, dtype=torch.float64),\n tensor(0.5347, dtype=torch.float64),\n tensor(0.5630, dtype=torch.float64),\n tensor(0.6302, dtype=torch.float64),\n tensor(0.6567, dtype=torch.float64),\n tensor(0.6829, dtype=torch.float64),\n tensor(0.7036, dtype=torch.float64),\n tensor(0.7544, dtype=torch.float64),\n tensor(0.7323, dtype=torch.float64),\n tensor(0.7559, dtype=torch.float64),\n tensor(0.7719, dtype=torch.float64),\n tensor(0.7973, dtype=torch.float64),\n tensor(0.8039, dtype=torch.float64),\n tensor(0.8082, dtype=torch.float64),\n tensor(0.8195, dtype=torch.float64),\n tensor(0.8206, dtype=torch.float64),\n tensor(0.8398, dtype=torch.float64),\n tensor(0.8536, dtype=torch.float64),\n tensor(0.8609, dtype=torch.float64),\n tensor(0.8525, dtype=torch.float64),\n tensor(0.8700, dtype=torch.float64),\n tensor(0.8801, dtype=torch.float64),\n tensor(0.8859, dtype=torch.float64),\n tensor(0.8943, dtype=torch.float64),\n tensor(0.8790, dtype=torch.float64),\n tensor(0.8925, dtype=torch.float64),\n tensor(0.8721, dtype=torch.float64),\n tensor(0.8758, dtype=torch.float64),\n tensor(0.8907, dtype=torch.float64),\n tensor(0.8878, dtype=torch.float64),\n tensor(0.8976, dtype=torch.float64),\n tensor(0.8888, dtype=torch.float64),\n tensor(0.9027, dtype=torch.float64),\n tensor(0.9077, dtype=torch.float64),\n tensor(0.8987, dtype=torch.float64),\n tensor(0.9114, dtype=torch.float64),\n tensor(0.9241, dtype=torch.float64),\n tensor(0.8947, dtype=torch.float64),\n tensor(0.9165, dtype=torch.float64),\n tensor(0.9226, dtype=torch.float64),\n tensor(0.9128, dtype=torch.float64),\n tensor(0.9194, dtype=torch.float64),\n tensor(0.9186, dtype=torch.float64),\n tensor(0.9277, dtype=torch.float64),\n tensor(0.9237, dtype=torch.float64),\n tensor(0.9194, dtype=torch.float64),\n tensor(0.9353, dtype=torch.float64),\n tensor(0.9277, dtype=torch.float64),\n tensor(0.9332, dtype=torch.float64),\n tensor(0.9234, dtype=torch.float64),\n tensor(0.9288, dtype=torch.float64),\n tensor(0.9241, dtype=torch.float64),\n tensor(0.8979, dtype=torch.float64),\n tensor(0.9313, dtype=torch.float64),\n tensor(0.9150, dtype=torch.float64),\n tensor(0.9343, dtype=torch.float64),\n tensor(0.9346, dtype=torch.float64),\n tensor(0.9339, dtype=torch.float64),\n tensor(0.9219, dtype=torch.float64),\n tensor(0.9281, dtype=torch.float64),\n tensor(0.9299, dtype=torch.float64),\n tensor(0.9372, dtype=torch.float64),\n tensor(0.9372, dtype=torch.float64),\n tensor(0.9419, dtype=torch.float64),\n tensor(0.9361, dtype=torch.float64),\n tensor(0.9437, dtype=torch.float64),\n tensor(0.9390, dtype=torch.float64),\n tensor(0.9408, dtype=torch.float64),\n tensor(0.9328, dtype=torch.float64),\n tensor(0.9477, dtype=torch.float64),\n tensor(0.9448, dtype=torch.float64),\n tensor(0.9426, dtype=torch.float64),\n tensor(0.9448, dtype=torch.float64),\n tensor(0.9531, dtype=torch.float64),\n tensor(0.9452, dtype=torch.float64),\n tensor(0.9510, dtype=torch.float64),\n tensor(0.9506, dtype=torch.float64),\n tensor(0.9510, dtype=torch.float64),\n tensor(0.9560, dtype=torch.float64),\n tensor(0.9430, dtype=torch.float64),\n tensor(0.9433, dtype=torch.float64),\n tensor(0.9473, dtype=torch.float64),\n tensor(0.9386, dtype=torch.float64),\n tensor(0.9466, dtype=torch.float64),\n tensor(0.9408, dtype=torch.float64),\n tensor(0.9502, dtype=torch.float64),\n tensor(0.9448, dtype=torch.float64),\n tensor(0.9433, dtype=torch.float64),\n tensor(0.9484, dtype=torch.float64),\n tensor(0.9466, dtype=torch.float64),\n tensor(0.9448, dtype=torch.float64),\n tensor(0.9437, dtype=torch.float64),\n tensor(0.9513, dtype=torch.float64),\n tensor(0.9495, dtype=torch.float64),\n tensor(0.9586, dtype=torch.float64),\n tensor(0.9495, dtype=torch.float64),\n tensor(0.9531, dtype=torch.float64),\n tensor(0.9444, dtype=torch.float64),\n tensor(0.9502, dtype=torch.float64),\n tensor(0.9568, dtype=torch.float64),\n tensor(0.9408, dtype=torch.float64),\n tensor(0.9553, dtype=torch.float64),\n tensor(0.9550, dtype=torch.float64),\n tensor(0.9571, dtype=torch.float64),\n tensor(0.9462, dtype=torch.float64),\n tensor(0.9557, dtype=torch.float64),\n tensor(0.9575, dtype=torch.float64),\n tensor(0.9659, dtype=torch.float64),\n tensor(0.9633, dtype=torch.float64),\n tensor(0.9600, dtype=torch.float64),\n tensor(0.9590, dtype=torch.float64),\n tensor(0.9611, dtype=torch.float64),\n tensor(0.9629, dtype=torch.float64),\n tensor(0.9590, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9608, dtype=torch.float64),\n tensor(0.9655, dtype=torch.float64),\n tensor(0.9608, dtype=torch.float64),\n tensor(0.9648, dtype=torch.float64),\n tensor(0.9593, dtype=torch.float64),\n tensor(0.9586, dtype=torch.float64),\n tensor(0.9669, dtype=torch.float64),\n tensor(0.9648, dtype=torch.float64),\n tensor(0.9648, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9597, dtype=torch.float64),\n tensor(0.9582, dtype=torch.float64),\n tensor(0.9517, dtype=torch.float64),\n tensor(0.9709, dtype=torch.float64),\n tensor(0.9666, dtype=torch.float64),\n tensor(0.9615, dtype=torch.float64),\n tensor(0.9662, dtype=torch.float64),\n tensor(0.9593, dtype=torch.float64),\n tensor(0.9611, dtype=torch.float64),\n tensor(0.9608, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9659, dtype=torch.float64),\n tensor(0.9604, dtype=torch.float64),\n tensor(0.9662, dtype=torch.float64),\n tensor(0.9648, dtype=torch.float64),\n tensor(0.9611, dtype=torch.float64),\n tensor(0.9629, dtype=torch.float64),\n tensor(0.9640, dtype=torch.float64),\n tensor(0.9684, dtype=torch.float64),\n tensor(0.9640, dtype=torch.float64),\n tensor(0.9691, dtype=torch.float64),\n tensor(0.9659, dtype=torch.float64),\n tensor(0.9724, dtype=torch.float64),\n tensor(0.9669, dtype=torch.float64),\n tensor(0.9691, dtype=torch.float64),\n tensor(0.9593, dtype=torch.float64),\n tensor(0.9626, dtype=torch.float64),\n tensor(0.9669, dtype=torch.float64),\n tensor(0.9673, dtype=torch.float64),\n tensor(0.9699, dtype=torch.float64),\n tensor(0.9680, dtype=torch.float64),\n tensor(0.9659, dtype=torch.float64),\n tensor(0.9684, dtype=torch.float64),\n tensor(0.9637, dtype=torch.float64),\n tensor(0.9666, dtype=torch.float64),\n tensor(0.9699, dtype=torch.float64),\n tensor(0.9695, dtype=torch.float64),\n tensor(0.9731, dtype=torch.float64),\n tensor(0.9691, dtype=torch.float64),\n tensor(0.9702, dtype=torch.float64),\n tensor(0.9691, dtype=torch.float64),\n tensor(0.9699, dtype=torch.float64),\n tensor(0.9702, dtype=torch.float64),\n tensor(0.9677, dtype=torch.float64),\n tensor(0.9651, dtype=torch.float64),\n tensor(0.9655, dtype=torch.float64),\n tensor(0.9699, dtype=torch.float64),\n tensor(0.9724, dtype=torch.float64),\n tensor(0.9713, dtype=torch.float64),\n tensor(0.9706, dtype=torch.float64),\n tensor(0.9764, dtype=torch.float64),\n tensor(0.9742, dtype=torch.float64)]",
    "1364712": "非常感谢❤️，我在优化中，借鉴了你提供的 notebook 中的思路，还没试的还剩更复制的 aug、k折以及RangerLars优化器\n\n想问下你notebook中用的这个优化器，和adamw对比的话，感觉会有提升吗：\n\n# RAdam + LARS + LookAHead\n# Lookahead implementation from https://github.com/lonePatient/lookahead_pytorch/blob/master/optimizer.py\n# RAdam + LARS implementation from https://gist.github.com/redknightlois/c4023d393eb8f92bb44b2ab582d7ec20\n\ndef Over9000(params, alpha=0.5, k=6, *args, **kwargs):\n     ralamb = Ralamb(params, *args, **kwargs)\n     return Lookahead(ralamb, alpha, k)\n\nRangerLars = Over9000",
    "1364716": "再次感谢耐心的解答！",
    "1364797": "请问你这个模型跑的时候加image augmentation了吗?",
    "1365149": "加了水平、垂直翻转，还有正负90度旋转\n\ntransform_train = transforms.Compose([\n        transforms.RandomHorizontalFlip(),\n        transforms.RandomVerticalFlip(),\n        transforms.RandomRotation(90),\n        transforms.ConvertImageDtype(torch.float),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n])",
    "1365228": "谢谢同学 谢谢你 我看看",
    "1365693": "优化器可以当成超参数之一来调整，还是应该用Adam和SGD作为baseline，之后再尝试其他优化器。",
    "1367796": "个人实测，CutMix效果很好，提升了一个点左右。"
  },
  "source": "meta"
}