{
  "id": 249173,
  "title": "9th Place Solution",
  "url": "/competitions/classify-leaves/writeups/yichao-han-9th-place-solution",
  "author_name": "",
  "post_date": "2021-07-03T01:37:09.253Z",
  "votes": 13,
  "comment_count": 3,
  "views": 0,
  "content": "<p>开个discussion不定期更新下这次比赛的小小心得</p>\n<h2>Single model performance</h2>\n<p>I used late submissions to evaluate all the single models.</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV</th>\n<th>public LB</th>\n<th>private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>resnet50d</td>\n<td>0.9729</td>\n<td>0.97704</td>\n<td>0.98068</td>\n</tr>\n<tr>\n<td>efficientnet_b3</td>\n<td>0.9726</td>\n<td>0.97977</td>\n<td>0.98431</td>\n</tr>\n<tr>\n<td>resnext50_32x4d</td>\n<td>0.9720</td>\n<td>0.97795</td>\n<td>0.97977</td>\n</tr>\n<tr>\n<td>inception_resnet_v2</td>\n<td>0.9748</td>\n<td>0.97886</td>\n<td>0.98340</td>\n</tr>\n<tr>\n<td>vit_base_patch16_224</td>\n<td>0.9717</td>\n<td>0.97568</td>\n<td>0.98000</td>\n</tr>\n<tr>\n<td>tf_efficientnet_b3_ns</td>\n<td>0.9725</td>\n<td>0.97750</td>\n<td>0.98272</td>\n</tr>\n<tr>\n<td>tf_efficientnet_b4_ns</td>\n<td>0.9781</td>\n<td>0.98340</td>\n<td>0.98704</td>\n</tr>\n<tr>\n<td>resnest200e</td>\n<td>0.9777</td>\n<td>0.98318</td>\n<td>0.98613</td>\n</tr>\n<tr>\n<td>mixnet_s</td>\n<td>0.9706</td>\n<td>0.97636</td>\n<td>0.98159</td>\n</tr>\n<tr>\n<td>mixnet_xl</td>\n<td>0.9761</td>\n<td>0.97886</td>\n<td>0.98409</td>\n</tr>\n<tr>\n<td>resnest50d</td>\n<td>0.9741</td>\n<td>0.97863</td>\n<td>0.98386</td>\n</tr>\n</tbody>\n</table>\n<h2>Final submissions</h2>\n<p>The final 2 submissions I used are trained from two different datasets. One is the raw dataset as what we downloaded and another is a clean dataset where I removed all noisy data (the same image with different labels) and duplicated images (the same image and label, I only kept one of them)</p>\n<p>I spent most of my time working on the raw dataset and trained 11 models while I only trained 3 models for the clean dataset.</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV</th>\n<th>public LB</th>\n<th>private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>7 model ensemble_raw</td>\n<td>0.9833</td>\n<td>0.98590</td>\n<td>0.98818</td>\n</tr>\n<tr>\n<td>3 model ensemble_clean</td>\n<td>0.9905</td>\n<td>0.98454</td>\n<td>0.98659</td>\n</tr>\n<tr>\n<td>efficientnet_b4_clean</td>\n<td>0.9872</td>\n<td>0.98477</td>\n<td>0.98704</td>\n</tr>\n</tbody>\n</table>\n<p>It seems that a single model trained with a clean dataset is good enough to achieve about the top 10% performance.</p>\n<h3>How did I pick these two submissions?</h3>\n<p>I was quite lucky that I pick my best submission. In the first CA house price prediction competition, I did not select my best prediction that could rank top 3 (similar performance as the transformer baseline model), which gave me a big lesson for the Kaggle competition. I would like to share this with you.</p>\n<p>Last time, I had an ensemble MLP model and an ensemble tree model. Both of them had a good CV and perform well on public LB (top 20). I tried to further combine these two by averaging them with various weights. The blended model decreases the loss on public LB by a lot, but I am not able to evaluate it using the CV since a lot of settings are different between the MLP and the tree model. The system automatically selected two blended MLP+tree models (one with log transform before average and the other not). The final private LB showed that my tree model is the best, and none of the other blended models with MLP improve anything.</p>\n<p>A similar decision process came to me this time. I had an ensemble model trained by raw data and another trained by clean data. These two models predict different labels for ~50 images. I submitted four blended prediction results using different weights, some of which yield higher scores on public LB. But this time, I made a different decision: don't put all their eggs in one basket. The blended model with some magic weights can overfit the public test set but you may not realize it. </p>\n<h2>Thought about the models</h2>\n<p>Some top performers used Densenet, which I did not explore this time. But I also trained some models for a single fold, including EfficientNet b7 and vit_deit. Training these models is very time-consuming. With the same training settings, EfficientNet b7 performed well and but it did not outperform b4 when comparing the valid acc for the same fold. And the transformer's valid acc converged at 0.965. so the vit may not be good in a small dataset like this.</p>\n<p>After this competition, I think resnet50 is a good start. To quickly set up a baseline model for a much larger dataset, resnet34 and resnet18 are good as well. I used resnet18 to get &gt;0.95 acc when I start this competition and quickly increased acc to  0.97+ by just switching to resnet50. EfficientNet series are worth trying if you want to win a competition. And an ensemble model is nearly a must. Besides EfficientNet, ResNet, ResNeSt (a few students discussed this), I would also recommend MixNet. It has fewer parameters and its performance is as good as many larger models.</p>\n<p>code: <a href=\"https://www.kaggle.com/yichaohan/classify-leaves-11-models\" target=\"_blank\">https://www.kaggle.com/yichaohan/classify-leaves-11-models</a></p>",
  "messages": [
    {
      "id": "1366599",
      "postDate": "06/27/2021 01:28:56",
      "content": "<p>开个discussion不定期更新下这次比赛的小小心得</p>\n<h2>Single model performance</h2>\n<p>I used late submissions to evaluate all the single models.</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV</th>\n<th>public LB</th>\n<th>private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>resnet50d</td>\n<td>0.9729</td>\n<td>0.97704</td>\n<td>0.98068</td>\n</tr>\n<tr>\n<td>efficientnet_b3</td>\n<td>0.9726</td>\n<td>0.97977</td>\n<td>0.98431</td>\n</tr>\n<tr>\n<td>resnext50_32x4d</td>\n<td>0.9720</td>\n<td>0.97795</td>\n<td>0.97977</td>\n</tr>\n<tr>\n<td>inception_resnet_v2</td>\n<td>0.9748</td>\n<td>0.97886</td>\n<td>0.98340</td>\n</tr>\n<tr>\n<td>vit_base_patch16_224</td>\n<td>0.9717</td>\n<td>0.97568</td>\n<td>0.98000</td>\n</tr>\n<tr>\n<td>tf_efficientnet_b3_ns</td>\n<td>0.9725</td>\n<td>0.97750</td>\n<td>0.98272</td>\n</tr>\n<tr>\n<td>tf_efficientnet_b4_ns</td>\n<td>0.9781</td>\n<td>0.98340</td>\n<td>0.98704</td>\n</tr>\n<tr>\n<td>resnest200e</td>\n<td>0.9777</td>\n<td>0.98318</td>\n<td>0.98613</td>\n</tr>\n<tr>\n<td>mixnet_s</td>\n<td>0.9706</td>\n<td>0.97636</td>\n<td>0.98159</td>\n</tr>\n<tr>\n<td>mixnet_xl</td>\n<td>0.9761</td>\n<td>0.97886</td>\n<td>0.98409</td>\n</tr>\n<tr>\n<td>resnest50d</td>\n<td>0.9741</td>\n<td>0.97863</td>\n<td>0.98386</td>\n</tr>\n</tbody>\n</table>\n<h2>Final submissions</h2>\n<p>The final 2 submissions I used are trained from two different datasets. One is the raw dataset as what we downloaded and another is a clean dataset where I removed all noisy data (the same image with different labels) and duplicated images (the same image and label, I only kept one of them)</p>\n<p>I spent most of my time working on the raw dataset and trained 11 models while I only trained 3 models for the clean dataset.</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV</th>\n<th>public LB</th>\n<th>private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>7 model ensemble_raw</td>\n<td>0.9833</td>\n<td>0.98590</td>\n<td>0.98818</td>\n</tr>\n<tr>\n<td>3 model ensemble_clean</td>\n<td>0.9905</td>\n<td>0.98454</td>\n<td>0.98659</td>\n</tr>\n<tr>\n<td>efficientnet_b4_clean</td>\n<td>0.9872</td>\n<td>0.98477</td>\n<td>0.98704</td>\n</tr>\n</tbody>\n</table>\n<p>It seems that a single model trained with a clean dataset is good enough to achieve about the top 10% performance.</p>\n<h3>How did I pick these two submissions?</h3>\n<p>I was quite lucky that I pick my best submission. In the first CA house price prediction competition, I did not select my best prediction that could rank top 3 (similar performance as the transformer baseline model), which gave me a big lesson for the Kaggle competition. I would like to share this with you.</p>\n<p>Last time, I had an ensemble MLP model and an ensemble tree model. Both of them had a good CV and perform well on public LB (top 20). I tried to further combine these two by averaging them with various weights. The blended model decreases the loss on public LB by a lot, but I am not able to evaluate it using the CV since a lot of settings are different between the MLP and the tree model. The system automatically selected two blended MLP+tree models (one with log transform before average and the other not). The final private LB showed that my tree model is the best, and none of the other blended models with MLP improve anything.</p>\n<p>A similar decision process came to me this time. I had an ensemble model trained by raw data and another trained by clean data. These two models predict different labels for ~50 images. I submitted four blended prediction results using different weights, some of which yield higher scores on public LB. But this time, I made a different decision: don't put all their eggs in one basket. The blended model with some magic weights can overfit the public test set but you may not realize it. </p>\n<h2>Thought about the models</h2>\n<p>Some top performers used Densenet, which I did not explore this time. But I also trained some models for a single fold, including EfficientNet b7 and vit_deit. Training these models is very time-consuming. With the same training settings, EfficientNet b7 performed well and but it did not outperform b4 when comparing the valid acc for the same fold. And the transformer's valid acc converged at 0.965. so the vit may not be good in a small dataset like this.</p>\n<p>After this competition, I think resnet50 is a good start. To quickly set up a baseline model for a much larger dataset, resnet34 and resnet18 are good as well. I used resnet18 to get &gt;0.95 acc when I start this competition and quickly increased acc to  0.97+ by just switching to resnet50. EfficientNet series are worth trying if you want to win a competition. And an ensemble model is nearly a must. Besides EfficientNet, ResNet, ResNeSt (a few students discussed this), I would also recommend MixNet. It has fewer parameters and its performance is as good as many larger models.</p>\n<p>code: <a href=\"https://www.kaggle.com/yichaohan/classify-leaves-11-models\" target=\"_blank\">https://www.kaggle.com/yichaohan/classify-leaves-11-models</a></p>",
      "rawMarkdown": "开个discussion不定期更新下这次比赛的小小心得\n\n## Single model performance\nI used late submissions to evaluate all the single models.\n\n|model|CV|public LB|private LB |\n| --- | --- | --- | --- |\n|resnet50d| 0.9729 | 0.97704 | 0.98068 |\n|efficientnet_b3| 0.9726 | 0.97977 | 0.98431 |\n|resnext50_32x4d| 0.9720 | 0.97795 | 0.97977 |\n|inception_resnet_v2| 0.9748 | 0.97886 | 0.98340 |\n|vit_base_patch16_224| 0.9717 | 0.97568 | 0.98000 |\n|tf_efficientnet_b3_ns| 0.9725 | 0.97750 | 0.98272 |\n|tf_efficientnet_b4_ns| 0.9781 | 0.98340 | 0.98704 |\n|resnest200e| 0.9777 | 0.98318 | 0.98613 |\n|mixnet_s| 0.9706 | 0.97636 | 0.98159 |\n|mixnet_xl| 0.9761 | 0.97886 | 0.98409 |\n|resnest50d| 0.9741 | 0.97863 | 0.98386 |\n\n## Final submissions\nThe final 2 submissions I used are trained from two different datasets. One is the raw dataset as what we downloaded and another is a clean dataset where I removed all noisy data (the same image with different labels) and duplicated images (the same image and label, I only kept one of them)\n\nI spent most of my time working on the raw dataset and trained 11 models while I only trained 3 models for the clean dataset.\n\n|model|CV|public LB|private LB |\n| --- | --- | --- | --- |\n|7 model ensemble_raw| 0.9833 |0.98590 | 0.98818 |\n|3 model ensemble_clean| 0.9905 | 0.98454 |0.98659 |\n|efficientnet_b4_clean| 0.9872 | 0.98477 | 0.98704 |\n\nIt seems that a single model trained with a clean dataset is good enough to achieve about the top 10% performance.\n\n### How did I pick these two submissions?\nI was quite lucky that I pick my best submission. In the first CA house price prediction competition, I did not select my best prediction that could rank top 3 (similar performance as the transformer baseline model), which gave me a big lesson for the Kaggle competition. I would like to share this with you.\n\nLast time, I had an ensemble MLP model and an ensemble tree model. Both of them had a good CV and perform well on public LB (top 20). I tried to further combine these two by averaging them with various weights. The blended model decreases the loss on public LB by a lot, but I am not able to evaluate it using the CV since a lot of settings are different between the MLP and the tree model. The system automatically selected two blended MLP+tree models (one with log transform before average and the other not). The final private LB showed that my tree model is the best, and none of the other blended models with MLP improve anything.\n\nA similar decision process came to me this time. I had an ensemble model trained by raw data and another trained by clean data. These two models predict different labels for ~50 images. I submitted four blended prediction results using different weights, some of which yield higher scores on public LB. But this time, I made a different decision: don't put all their eggs in one basket. The blended model with some magic weights can overfit the public test set but you may not realize it. \n\n## Thought about the models\nSome top performers used Densenet, which I did not explore this time. But I also trained some models for a single fold, including EfficientNet b7 and vit_deit. Training these models is very time-consuming. With the same training settings, EfficientNet b7 performed well and but it did not outperform b4 when comparing the valid acc for the same fold. And the transformer's valid acc converged at 0.965. so the vit may not be good in a small dataset like this.\n\nAfter this competition, I think resnet50 is a good start. To quickly set up a baseline model for a much larger dataset, resnet34 and resnet18 are good as well. I used resnet18 to get >0.95 acc when I start this competition and quickly increased acc to  0.97+ by just switching to resnet50. EfficientNet series are worth trying if you want to win a competition. And an ensemble model is nearly a must. Besides EfficientNet, ResNet, ResNeSt (a few students discussed this), I would also recommend MixNet. It has fewer parameters and its performance is as good as many larger models.\n\ncode: https://www.kaggle.com/yichaohan/classify-leaves-11-models",
      "votes": null
    },
    {
      "id": "1367795",
      "postDate": "06/28/2021 05:13:13",
      "content": "<p>居然做了11个模型，有设备真好🙈</p>",
      "rawMarkdown": "居然做了11个模型，有设备真好🙈",
      "votes": null
    },
    {
      "id": "1374019",
      "postDate": "07/02/2021 23:47:05",
      "content": "<p>第一次做计算机视觉，做之前对每个模型大概会有怎么样的表现没啥感觉，所以刷了一波。</p>",
      "rawMarkdown": "第一次做计算机视觉，做之前对每个模型大概会有怎么样的表现没啥感觉，所以刷了一波。",
      "votes": null
    },
    {
      "id": "1387129",
      "postDate": "07/14/2021 01:43:09",
      "content": "<p>你用raw data 训练的最高精度是多少？</p>",
      "rawMarkdown": "你用raw data 训练的最高精度是多少？",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1367795,
      "author_name": "charlesyyun",
      "author_url": "",
      "post_date": "06/28/2021 05:13:13",
      "content": "<p>居然做了11个模型，有设备真好🙈</p>",
      "votes": null,
      "replies": [
        {
          "id": 1374019,
          "author_name": "yichaohan",
          "author_url": "",
          "post_date": "07/02/2021 23:47:05",
          "content": "<p>第一次做计算机视觉，做之前对每个模型大概会有怎么样的表现没啥感觉，所以刷了一波。</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1387129,
      "author_name": "feixu22",
      "author_url": "",
      "post_date": "07/14/2021 01:43:09",
      "content": "<p>你用raw data 训练的最高精度是多少？</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1366599": "开个discussion不定期更新下这次比赛的小小心得\n\n## Single model performance\nI used late submissions to evaluate all the single models.\n\n|model|CV|public LB|private LB |\n| --- | --- | --- | --- |\n|resnet50d| 0.9729 | 0.97704 | 0.98068 |\n|efficientnet_b3| 0.9726 | 0.97977 | 0.98431 |\n|resnext50_32x4d| 0.9720 | 0.97795 | 0.97977 |\n|inception_resnet_v2| 0.9748 | 0.97886 | 0.98340 |\n|vit_base_patch16_224| 0.9717 | 0.97568 | 0.98000 |\n|tf_efficientnet_b3_ns| 0.9725 | 0.97750 | 0.98272 |\n|tf_efficientnet_b4_ns| 0.9781 | 0.98340 | 0.98704 |\n|resnest200e| 0.9777 | 0.98318 | 0.98613 |\n|mixnet_s| 0.9706 | 0.97636 | 0.98159 |\n|mixnet_xl| 0.9761 | 0.97886 | 0.98409 |\n|resnest50d| 0.9741 | 0.97863 | 0.98386 |\n\n## Final submissions\nThe final 2 submissions I used are trained from two different datasets. One is the raw dataset as what we downloaded and another is a clean dataset where I removed all noisy data (the same image with different labels) and duplicated images (the same image and label, I only kept one of them)\n\nI spent most of my time working on the raw dataset and trained 11 models while I only trained 3 models for the clean dataset.\n\n|model|CV|public LB|private LB |\n| --- | --- | --- | --- |\n|7 model ensemble_raw| 0.9833 |0.98590 | 0.98818 |\n|3 model ensemble_clean| 0.9905 | 0.98454 |0.98659 |\n|efficientnet_b4_clean| 0.9872 | 0.98477 | 0.98704 |\n\nIt seems that a single model trained with a clean dataset is good enough to achieve about the top 10% performance.\n\n### How did I pick these two submissions?\nI was quite lucky that I pick my best submission. In the first CA house price prediction competition, I did not select my best prediction that could rank top 3 (similar performance as the transformer baseline model), which gave me a big lesson for the Kaggle competition. I would like to share this with you.\n\nLast time, I had an ensemble MLP model and an ensemble tree model. Both of them had a good CV and perform well on public LB (top 20). I tried to further combine these two by averaging them with various weights. The blended model decreases the loss on public LB by a lot, but I am not able to evaluate it using the CV since a lot of settings are different between the MLP and the tree model. The system automatically selected two blended MLP+tree models (one with log transform before average and the other not). The final private LB showed that my tree model is the best, and none of the other blended models with MLP improve anything.\n\nA similar decision process came to me this time. I had an ensemble model trained by raw data and another trained by clean data. These two models predict different labels for ~50 images. I submitted four blended prediction results using different weights, some of which yield higher scores on public LB. But this time, I made a different decision: don't put all their eggs in one basket. The blended model with some magic weights can overfit the public test set but you may not realize it. \n\n## Thought about the models\nSome top performers used Densenet, which I did not explore this time. But I also trained some models for a single fold, including EfficientNet b7 and vit_deit. Training these models is very time-consuming. With the same training settings, EfficientNet b7 performed well and but it did not outperform b4 when comparing the valid acc for the same fold. And the transformer's valid acc converged at 0.965. so the vit may not be good in a small dataset like this.\n\nAfter this competition, I think resnet50 is a good start. To quickly set up a baseline model for a much larger dataset, resnet34 and resnet18 are good as well. I used resnet18 to get >0.95 acc when I start this competition and quickly increased acc to  0.97+ by just switching to resnet50. EfficientNet series are worth trying if you want to win a competition. And an ensemble model is nearly a must. Besides EfficientNet, ResNet, ResNeSt (a few students discussed this), I would also recommend MixNet. It has fewer parameters and its performance is as good as many larger models.\n\ncode: https://www.kaggle.com/yichaohan/classify-leaves-11-models",
    "1367795": "居然做了11个模型，有设备真好🙈",
    "1374019": "第一次做计算机视觉，做之前对每个模型大概会有怎么样的表现没啥感觉，所以刷了一波。",
    "1387129": "你用raw data 训练的最高精度是多少？"
  },
  "source": "meta"
}