{ "id": 245198, "title": "[CV] vs [LB]", "url": "/competitions/classify-leaves/discussion/245198", "author_name": "seefun", "post_date": "2021-06-10T06:19:21.061000", "votes": 11, "comment_count": 12, "views": 0, "content": "

CV stands for Cross Validation Score；LB stands for LeaderBoard Score；
\nCould you share your CV and LB？ Is there huge gap between them？

\n

Here is some of my results:

\n

ResNeSt50 baseline model1 (resize): CV 0.980 LB 0.980
\nResNeSt50 baseline model2 (center crop): CV 0.980 LB 0.984
\nupdate: EfficientnetV2-small CV0.989 (remove duplicated and wrong images) LB 0.983
\nupdate: 感觉LB上，很多人都来到了0.98+，分差都在几个测试样本，大家模型泛化性能可能相差不大，感觉private LB成绩如何全看运气了，感觉没有必要继续更新实验结果了，建议大家上了0.98或0.98附近也没有必要继续去过拟合public LB了，基本已经达到数据极限了(测试集存在错误标注)
\nupdate: EfficientNet-V2 (remove duplicated images) CV 0.9885 LB 0.98795，但我认为过拟合了，不作为最终提交结果。最终提交我选择使用了这些模型的均值融合，LB 0.97772， 虽然LB分更低，但值得相信

\n

LB得分为 5 fold 的5个模型结果(after softmax)均值融合, 并使用TTA (test time augmentation),即在inference使用轻微的数据增强(flip and rotate),并均值融合不同增广下的inference结果。

\n

模型为baseline，未调参且并未完全训练收敛，但已显著超越autogluon的baseline LB 0.948，或许是autogluon的使用姿势不对？期待沐神放出autogluon的baseline

\n

what is TTA | why should believe CV not LB | RandAugment

\n
\n

另: 整理了一些比赛常用trick,将保持更新,欢迎star:
\nhttps://github.com/seefun/TorchUtils

\n

向沐神学习!有问题欢迎询问！我的b站，打算向沐神学习，更新数据比赛经验以及AI落地科普!

", "messages": [ { "id": 1343263, "postDate": "2021-06-10T06:19:21.063Z", "content": "

CV stands for Cross Validation Score；LB stands for LeaderBoard Score；
\nCould you share your CV and LB？ Is there huge gap between them？

\n

Here is some of my results:

\n

ResNeSt50 baseline model1 (resize): CV 0.980 LB 0.980
\nResNeSt50 baseline model2 (center crop): CV 0.980 LB 0.984
\nupdate: EfficientnetV2-small CV0.989 (remove duplicated and wrong images) LB 0.983
\nupdate: 感觉LB上，很多人都来到了0.98+，分差都在几个测试样本，大家模型泛化性能可能相差不大，感觉private LB成绩如何全看运气了，感觉没有必要继续更新实验结果了，建议大家上了0.98或0.98附近也没有必要继续去过拟合public LB了，基本已经达到数据极限了(测试集存在错误标注)
\nupdate: EfficientNet-V2 (remove duplicated images) CV 0.9885 LB 0.98795，但我认为过拟合了，不作为最终提交结果。最终提交我选择使用了这些模型的均值融合，LB 0.97772， 虽然LB分更低，但值得相信

\n

LB得分为 5 fold 的5个模型结果(after softmax)均值融合, 并使用TTA (test time augmentation),即在inference使用轻微的数据增强(flip and rotate),并均值融合不同增广下的inference结果。

\n

模型为baseline，未调参且并未完全训练收敛，但已显著超越autogluon的baseline LB 0.948，或许是autogluon的使用姿势不对？期待沐神放出autogluon的baseline

\n

what is TTA | why should believe CV not LB | RandAugment

\n
\n

另: 整理了一些比赛常用trick,将保持更新,欢迎star:
\nhttps://github.com/seefun/TorchUtils

\n

向沐神学习!有问题欢迎询问！我的b站，打算向沐神学习，更新数据比赛经验以及AI落地科普!

", "rawMarkdown": "CV stands for Cross Validation Score；LB stands for LeaderBoard Score；\nCould you share your CV and LB？ Is there huge gap between them？\n\nHere is some of my results:\n\nResNeSt50 baseline model1 (resize): CV 0.980 LB 0.980 \nResNeSt50 baseline model2 (center crop): CV 0.980 LB 0.984\nupdate: EfficientnetV2-small CV0.989 (remove duplicated and wrong images) LB 0.983\nupdate: 感觉LB上，很多人都来到了0.98+，分差都在几个测试样本，大家模型泛化性能可能相差不大，感觉private LB成绩如何全看运气了，感觉没有必要继续更新实验结果了，建议大家上了0.98或0.98附近也没有必要继续去过拟合public LB了，基本已经达到数据极限了(测试集存在错误标注)\nupdate: EfficientNet-V2 (remove duplicated images) CV 0.9885 LB 0.98795，但我认为过拟合了，不作为最终提交结果。最终提交我选择使用了这些模型的均值融合，LB 0.97772， 虽然LB分更低，但值得相信\n\nLB得分为 5 fold 的5个模型结果(after softmax)均值融合, 并使用TTA (test time augmentation),即在inference使用轻微的数据增强(flip and rotate),并均值融合不同增广下的inference结果。\n\n模型为baseline，未调参且并未完全训练收敛，但已显著超越autogluon的baseline LB 0.948，或许是autogluon的使用姿势不对？期待沐神放出autogluon的baseline\n\n[what is TTA](https://machinelearningmastery.com/how-to-use-test-time-augmentation-to-improve-model-performance-for-image-classification/) | [why should believe CV not LB](https://medium.com/kaggle-blog/profiling-top-kagglers-bestfitting-currently-1-in-the-world-58cc0e187b) | [RandAugment](https://arxiv.org/abs/1909.13719)\n\n---------------------------------\n\n另: 整理了一些比赛常用trick,将保持更新,欢迎star:\nhttps://github.com/seefun/TorchUtils\n\n向沐神学习!有问题欢迎询问！[我的b站](https://space.bilibili.com/388264943)，打算向沐神学习，更新数据比赛经验以及AI落地科普!", "votes": 11 }, { "id": 1343400, "postDate": "2021-06-10T07:27:56.603Z", "content": "

未完全训练收敛是因为训练数据存在277张矛盾的图片，比如images/78.jpg, images/8329.jpg完全相同，但label不同。

", "rawMarkdown": "未完全训练收敛是因为训练数据存在277张矛盾的图片，比如images/78.jpg, images/8329.jpg完全相同，但label不同。", "votes": 2, "replies": [ { "id": 1343453, "postDate": "2021-06-10T08:04:04.513Z", "content": "

我的未完全收敛是因为我的baseline的epoch数目设置的比较小，训练loss和验证loss都还在下降趋势中，就提前结束训练了；

\n

数据duplicate确实是一个问题，使用hash阈值0.98，发现了1117张重复数据，导致本地验证可能存在leak；标注矛盾还没观察，看来训练集还存在一些noise label。代码参考:

\n

https://github.com/seefun/TorchUtils/blob/master/torch_utils/dataset/del_duplicate_image.py

", "rawMarkdown": "我的未完全收敛是因为我的baseline的epoch数目设置的比较小，训练loss和验证loss都还在下降趋势中，就提前结束训练了；\n\n数据duplicate确实是一个问题，使用hash阈值0.98，发现了1117张重复数据，导致本地验证可能存在leak；标注矛盾还没观察，看来训练集还存在一些noise label。代码参考:\n\nhttps://github.com/seefun/TorchUtils/blob/master/torch_utils/dataset/del_duplicate_image.py" }, { "id": 1343680, "postDate": "2021-06-10T11:13:52.817Z", "content": "

感觉可能加个soft label效果能好点？试了，没啥用。

", "rawMarkdown": "感觉可能加个soft label效果能好点？试了，没啥用。", "votes": 1 } ] }, { "id": 2200115, "postDate": "2023-03-28T09:53:01.253Z", "content": "

厉害，新手来学习一下

", "rawMarkdown": "厉害，新手来学习一下" }, { "id": 1362821, "postDate": "2021-06-23T17:05:41.840Z", "content": "

提交的结果不多，CV和LB很接近，目前结果来看5折的LB比CV高0.001-0.002的样子，单折的LB比CV低。同样模型加了TTA(flip and rotate)算CV发现没啥效果，不确定值不值得用于预测。

", "rawMarkdown": "提交的结果不多，CV和LB很接近，目前结果来看5折的LB比CV高0.001-0.002的样子，单折的LB比CV低。同样模型加了TTA(flip and rotate)算CV发现没啥效果，不确定值不值得用于预测。" }, { "id": 1355460, "postDate": "2021-06-18T09:38:57.197Z", "content": "

This is my baseline notebook for this dataset:
\nhttps://github.com/seefun/TorchUtils/blob/master/examples/kaggle_leaves_classification.ipynb

", "rawMarkdown": "This is my baseline notebook for this dataset:\nhttps://github.com/seefun/TorchUtils/blob/master/examples/kaggle_leaves_classification.ipynb" }, { "id": 1343643, "postDate": "2021-06-10T10:47:34.887Z", "content": "

厉害啊老哥，我研究一下怎么用先

", "rawMarkdown": "厉害啊老哥，我研究一下怎么用先" }, { "id": 1343464, "postDate": "2021-06-10T08:15:02.377Z", "content": "

这是我的一些结果
\n\"my\"

", "rawMarkdown": "这是我的一些结果\n![my](https://cdn.jsdelivr.net/gh/nekomiao123/pic/img/image-20210610161414660.png)", "replies": [ { "id": 1343499, "postDate": "2021-06-10T08:36:29.533Z", "content": "

(timm user +1)
\nLB是多折还是单折，加TTA了嘛

", "rawMarkdown": "(timm user +1)\nLB是多折还是单折，加TTA了嘛 \n" }, { "id": 1343999, "postDate": "2021-06-10T14:55:04.877Z", "content": "

单折，已经加了TTA

", "rawMarkdown": "单折，已经加了TTA" } ] }, { "id": 1344403, "postDate": "2021-06-10T22:41:14.360Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1343384, "postDate": "2021-06-10T07:20:10.597Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1343400, "author_name": "xyz", "author_url": "", "post_date": "2021-06-10T07:27:56.603000", "content": "

未完全训练收敛是因为训练数据存在277张矛盾的图片，比如images/78.jpg, images/8329.jpg完全相同，但label不同。

", "votes": 2, "replies": [ { "id": 1343453, "author_name": "seefun", "author_url": "", "post_date": "2021-06-10T08:04:04.513000", "content": "

我的未完全收敛是因为我的baseline的epoch数目设置的比较小，训练loss和验证loss都还在下降趋势中，就提前结束训练了；

\n

数据duplicate确实是一个问题，使用hash阈值0.98，发现了1117张重复数据，导致本地验证可能存在leak；标注矛盾还没观察，看来训练集还存在一些noise label。代码参考:

\n

https://github.com/seefun/TorchUtils/blob/master/torch_utils/dataset/del_duplicate_image.py

", "votes": 0, "replies": [] }, { "id": 1343680, "author_name": "Tianfu Wang (王天夫)", "author_url": "", "post_date": "2021-06-10T11:13:52.817000", "content": "

感觉可能加个soft label效果能好点？试了，没啥用。

", "votes": 1, "replies": [] } ] }, { "id": 2200115, "author_name": "hao yi99", "author_url": "", "post_date": "2023-03-28T09:53:01.253000", "content": "

厉害，新手来学习一下

", "votes": 0, "replies": [] }, { "id": 1362821, "author_name": "Yichao Han", "author_url": "", "post_date": "2021-06-23T17:05:41.840000", "content": "

提交的结果不多，CV和LB很接近，目前结果来看5折的LB比CV高0.001-0.002的样子，单折的LB比CV低。同样模型加了TTA(flip and rotate)算CV发现没啥效果，不确定值不值得用于预测。

", "votes": 0, "replies": [] }, { "id": 1355460, "author_name": "seefun", "author_url": "", "post_date": "2021-06-18T09:38:57.197000", "content": "

This is my baseline notebook for this dataset:
\nhttps://github.com/seefun/TorchUtils/blob/master/examples/kaggle_leaves_classification.ipynb

", "votes": 0, "replies": [] }, { "id": 1343643, "author_name": "迪迦奥特曼本曼", "author_url": "", "post_date": "2021-06-10T10:47:34.887000", "content": "

厉害啊老哥，我研究一下怎么用先

", "votes": 0, "replies": [] }, { "id": 1343464, "author_name": "Neko Kiku", "author_url": "", "post_date": "2021-06-10T08:15:02.377000", "content": "

这是我的一些结果
\n\"my\"

", "votes": 0, "replies": [ { "id": 1343499, "author_name": "seefun", "author_url": "", "post_date": "2021-06-10T08:36:29.533000", "content": "

(timm user +1)
\nLB是多折还是单折，加TTA了嘛

", "votes": 0, "replies": [] }, { "id": 1343999, "author_name": "Neko Kiku", "author_url": "", "post_date": "2021-06-10T14:55:04.877000", "content": "

单折，已经加了TTA

", "votes": 0, "replies": [] } ] }, { "id": 1344403, "author_name": "", "author_url": "", "post_date": "2021-06-10T22:41:14.360000", "content": "", "votes": 0, "replies": [] }, { "id": 1343384, "author_name": "", "author_url": "", "post_date": "2021-06-10T07:20:10.597000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1343263": "CV stands for Cross Validation Score；LB stands for LeaderBoard Score；\nCould you share your CV and LB？ Is there huge gap between them？\n\nHere is some of my results:\n\nResNeSt50 baseline model1 (resize): CV 0.980 LB 0.980 \nResNeSt50 baseline model2 (center crop): CV 0.980 LB 0.984\nupdate: EfficientnetV2-small CV0.989 (remove duplicated and wrong images) LB 0.983\nupdate: 感觉LB上，很多人都来到了0.98+，分差都在几个测试样本，大家模型泛化性能可能相差不大，感觉private LB成绩如何全看运气了，感觉没有必要继续更新实验结果了，建议大家上了0.98或0.98附近也没有必要继续去过拟合public LB了，基本已经达到数据极限了(测试集存在错误标注)\nupdate: EfficientNet-V2 (remove duplicated images) CV 0.9885 LB 0.98795，但我认为过拟合了，不作为最终提交结果。最终提交我选择使用了这些模型的均值融合，LB 0.97772， 虽然LB分更低，但值得相信\n\nLB得分为 5 fold 的5个模型结果(after softmax)均值融合, 并使用TTA (test time augmentation),即在inference使用轻微的数据增强(flip and rotate),并均值融合不同增广下的inference结果。\n\n模型为baseline，未调参且并未完全训练收敛，但已显著超越autogluon的baseline LB 0.948，或许是autogluon的使用姿势不对？期待沐神放出autogluon的baseline\n\n[what is TTA](https://machinelearningmastery.com/how-to-use-test-time-augmentation-to-improve-model-performance-for-image-classification/) | [why should believe CV not LB](https://medium.com/kaggle-blog/profiling-top-kagglers-bestfitting-currently-1-in-the-world-58cc0e187b) | [RandAugment](https://arxiv.org/abs/1909.13719)\n\n---------------------------------\n\n另: 整理了一些比赛常用trick,将保持更新,欢迎star:\nhttps://github.com/seefun/TorchUtils\n\n向沐神学习!有问题欢迎询问！[我的b站](https://space.bilibili.com/388264943)，打算向沐神学习，更新数据比赛经验以及AI落地科普!", "1343400": "未完全训练收敛是因为训练数据存在277张矛盾的图片，比如images/78.jpg, images/8329.jpg完全相同，但label不同。", "2200115": "厉害，新手来学习一下", "1362821": "提交的结果不多，CV和LB很接近，目前结果来看5折的LB比CV高0.001-0.002的样子，单折的LB比CV低。同样模型加了TTA(flip and rotate)算CV发现没啥效果，不确定值不值得用于预测。", "1355460": "This is my baseline notebook for this dataset:\nhttps://github.com/seefun/TorchUtils/blob/master/examples/kaggle_leaves_classification.ipynb", "1343643": "厉害啊老哥，我研究一下怎么用先", "1343464": "这是我的一些结果\n![my](https://cdn.jsdelivr.net/gh/nekomiao123/pic/img/image-20210610161414660.png)", "1344403": "", "1343384": "" } }