{
  "id": 51202,
  "title": "How to use unsupervised learning (aka knowledge distillation or pseudo label) ",
  "url": "/competitions/data-science-bowl-2018/discussion/51202",
  "author_name": "",
  "post_date": "2018-03-06T13:48:48.095867100Z",
  "votes": 9,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Refer to this paper:</p>\n\n<p>\"Data Distillation: Towards Omni-Supervised Learning\"\n- Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He, CVPR 2018</p>\n\n<p><a href=\"https://arxiv.org/abs/1712.04440\">https://arxiv.org/abs/1712.04440</a></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/291618/8703/semi2.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/291618/8702/semi1.png\" alt=\"enter image description here\"></p>\n\n<p>this is almost a natural way to improve results because:</p>\n\n<ul>\n<li><p>you already have an ensemble of classifiers (especially for kaggle competition)</p></li>\n<li><p>you need to improve detection first (before instance segmentation)</p></li>\n<li><p>there are really many potential training images from the web (especially tissues from histology) but they are no label. You can label a few and use unsupervised learning to label the rest.</p></li>\n<li><p>within a given test image, cell nuclei are similar. You may detect  say 80%. If you can learn online from the most confident ones, you can further improve detection rate by maybe 1 or 2%. Also in the strategy of ensembles, you trained zero \"fp classifier\",  so most confident detectoins are usually correct (giving more training samples).  </p>\n\n<ul><li>Deep learning can tolerate about 10% to 20% label noise.  </li>\n<li>Also test images can be moasic to give a bigger image. Hence test images are correlated. knowledge can be propagated.</li>\n<li>There is additional information to know if your detection is correct or not. Density of detected nuclei. Low density is using correct. High density are prone to error. Also compare mask region of semantic versus instance segmentation results.  </li></ul></li>\n</ul>",
  "messages": [
    {
      "id": "291618",
      "postDate": "03/06/2018 13:48:48",
      "content": "<p>Refer to this paper:</p>\n\n<p>\"Data Distillation: Towards Omni-Supervised Learning\"\n- Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He, CVPR 2018</p>\n\n<p><a href=\"https://arxiv.org/abs/1712.04440\">https://arxiv.org/abs/1712.04440</a></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/291618/8703/semi2.png\" alt=\"enter image description here\"></p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/291618/8702/semi1.png\" alt=\"enter image description here\"></p>\n\n<p>this is almost a natural way to improve results because:</p>\n\n<ul>\n<li><p>you already have an ensemble of classifiers (especially for kaggle competition)</p></li>\n<li><p>you need to improve detection first (before instance segmentation)</p></li>\n<li><p>there are really many potential training images from the web (especially tissues from histology) but they are no label. You can label a few and use unsupervised learning to label the rest.</p></li>\n<li><p>within a given test image, cell nuclei are similar. You may detect  say 80%. If you can learn online from the most confident ones, you can further improve detection rate by maybe 1 or 2%. Also in the strategy of ensembles, you trained zero \"fp classifier\",  so most confident detectoins are usually correct (giving more training samples).  </p>\n\n<ul><li>Deep learning can tolerate about 10% to 20% label noise.  </li>\n<li>Also test images can be moasic to give a bigger image. Hence test images are correlated. knowledge can be propagated.</li>\n<li>There is additional information to know if your detection is correct or not. Density of detected nuclei. Low density is using correct. High density are prone to error. Also compare mask region of semantic versus instance segmentation results.  </li></ul></li>\n</ul>",
      "rawMarkdown": "Refer to this paper:\n\n\"Data Distillation: Towards Omni-Supervised Learning\"\n- Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He, CVPR 2018\n\n\nhttps://arxiv.org/abs/1712.04440\n\n\n ![enter image description here][1]\n\n\n ![enter image description here][2]\n\n\nthis is almost a natural way to improve results because:\n\n-  you already have an ensemble of classifiers (especially for kaggle competition)\n\n- you need to improve detection first (before instance segmentation)\n\n- there are really many potential training images from the web (especially tissues from histology) but they are no label. You can label a few and use unsupervised learning to label the rest.\n\n- within a given test image, cell nuclei are similar. You may detect  say 80%. If you can learn online from the most confident ones, you can further improve detection rate by maybe 1 or 2%. Also in the strategy of ensembles, you trained zero \"fp classifier\",  so most confident detectoins are usually correct (giving more training samples).  \n    - Deep learning can tolerate about 10% to 20% label noise.  \n    -  Also test images can be moasic to give a bigger image. Hence test images are correlated. knowledge can be propagated.\n    - There is additional information to know if your detection is correct or not. Density of detected nuclei. Low density is using correct. High density are prone to error. Also compare mask region of semantic versus instance segmentation results.  \n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/291618/8703/semi2.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/291618/8702/semi1.png",
      "votes": null
    },
    {
      "id": "291812",
      "postDate": "03/06/2018 22:16:23",
      "content": "<p>Nice paper. Easy to read because of no equations at all. Seems easy to adopt too. However, I have a hard time to understand how to covert soft labels (average probabilities from ensembles) into hard labels.  Is something like np.where( a&gt; 0.5, 1, 0)? </p>",
      "rawMarkdown": "Nice paper. Easy to read because of no equations at all. Seems easy to adopt too. However, I have a hard time to understand how to covert soft labels (average probabilities from ensembles) into hard labels.  Is something like np.where( a&gt; 0.5, 1, 0)?",
      "votes": null
    },
    {
      "id": "291871",
      "postDate": "03/07/2018 02:30:00",
      "content": "<p>you can find more information here:</p>\n\n<p><a href=\"https://github.com/kweonwooj/papers/issues/64\">https://github.com/kweonwooj/papers/issues/64</a></p>",
      "rawMarkdown": "you can find more information here:\n\nhttps://github.com/kweonwooj/papers/issues/64",
      "votes": null
    },
    {
      "id": "293197",
      "postDate": "03/09/2018 11:32:22",
      "content": "<p>observation:</p>\n\n<ul>\n<li><p>train and validation LB metric is very high, about 80% for box precision at threshold=0.5 and 70% for mask average precision at threshold=0.5 to 0.9.</p></li>\n<li><p>this is achieved despite that is about 5% label error in kaggle train dataset</p></li>\n<li><p>test time augmentation shoes that nuclei not detected in one image can be detected in another augmented image.</p></li>\n</ul>\n\n<p>problem:</p>\n\n<ul>\n<li>how to apply ensemble?</li>\n</ul>\n\n<hr>\n\n<p>solution:</p>\n\n<ul>\n<li><p>using the distillation paper above, for a LB test image, we make many augmentations. Then we apply our trained network. Hopefully, all nuclei will get label at least once (this is the pesudo label)</p></li>\n<li><p>mix the LB test image (with the pseudo label) with the train image.</p></li>\n<li><p>retrain or fine-tune the network.</p></li>\n</ul>\n\n<p>NOTE: </p>\n\n<ul>\n<li><p>if the label noise is below a certain value, it will work.\n (e.g. instead of training on augmented test image, you can transfer all pseudo label to the original test LB image)</p></li>\n<li><p>need to take care of mixing portion of pseudo labelled image and human labelled image.</p></li>\n</ul>",
      "rawMarkdown": "observation:\n\n- train and validation LB metric is very high, about 80% for box precision at threshold=0.5 and 70% for mask average precision at threshold=0.5 to 0.9.\n\n- this is achieved despite that is about 5% label error in kaggle train dataset\n\n- test time augmentation shoes that nuclei not detected in one image can be detected in another augmented image.\n\nproblem:\n\n- how to apply ensemble?\n\n---\n\nsolution:\n\n- using the distillation paper above, for a LB test image, we make many augmentations. Then we apply our trained network. Hopefully, all nuclei will get label at least once (this is the pesudo label)\n\n- mix the LB test image (with the pseudo label) with the train image.\n\n- retrain or fine-tune the network.\n\nNOTE: \n\n  - if the label noise is below a certain value, it will work.\n     (e.g. instead of training on augmented test image, you can transfer all pseudo label to the original test LB image)\n\n  - need to take care of mixing portion of pseudo labelled image and human labelled image.",
      "votes": null
    },
    {
      "id": "296609",
      "postDate": "03/15/2018 14:15:16",
      "content": "<p><a href=\"https://arxiv.org/pdf/1706.04737.pdf\">https://arxiv.org/pdf/1706.04737.pdf</a></p>\n\n<p>\"Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation\"\nLin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, Danny Z. Chen</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/296609/8772/sugeest1.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "https://arxiv.org/pdf/1706.04737.pdf\n\n\"Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation\"\nLin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, Danny Z. Chen\n\n\n  ![enter image description here][1]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/296609/8772/sugeest1.png",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 291812,
      "author_name": "ybwu01",
      "author_url": "",
      "post_date": "03/06/2018 22:16:23",
      "content": "<p>Nice paper. Easy to read because of no equations at all. Seems easy to adopt too. However, I have a hard time to understand how to covert soft labels (average probabilities from ensembles) into hard labels.  Is something like np.where( a&gt; 0.5, 1, 0)? </p>",
      "votes": null,
      "replies": [
        {
          "id": 291871,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "03/07/2018 02:30:00",
          "content": "<p>you can find more information here:</p>\n\n<p><a href=\"https://github.com/kweonwooj/papers/issues/64\">https://github.com/kweonwooj/papers/issues/64</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 293197,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "03/09/2018 11:32:22",
      "content": "<p>observation:</p>\n\n<ul>\n<li><p>train and validation LB metric is very high, about 80% for box precision at threshold=0.5 and 70% for mask average precision at threshold=0.5 to 0.9.</p></li>\n<li><p>this is achieved despite that is about 5% label error in kaggle train dataset</p></li>\n<li><p>test time augmentation shoes that nuclei not detected in one image can be detected in another augmented image.</p></li>\n</ul>\n\n<p>problem:</p>\n\n<ul>\n<li>how to apply ensemble?</li>\n</ul>\n\n<hr>\n\n<p>solution:</p>\n\n<ul>\n<li><p>using the distillation paper above, for a LB test image, we make many augmentations. Then we apply our trained network. Hopefully, all nuclei will get label at least once (this is the pesudo label)</p></li>\n<li><p>mix the LB test image (with the pseudo label) with the train image.</p></li>\n<li><p>retrain or fine-tune the network.</p></li>\n</ul>\n\n<p>NOTE: </p>\n\n<ul>\n<li><p>if the label noise is below a certain value, it will work.\n (e.g. instead of training on augmented test image, you can transfer all pseudo label to the original test LB image)</p></li>\n<li><p>need to take care of mixing portion of pseudo labelled image and human labelled image.</p></li>\n</ul>",
      "votes": null,
      "replies": []
    },
    {
      "id": 296609,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "03/15/2018 14:15:16",
      "content": "<p><a href=\"https://arxiv.org/pdf/1706.04737.pdf\">https://arxiv.org/pdf/1706.04737.pdf</a></p>\n\n<p>\"Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation\"\nLin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, Danny Z. Chen</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/296609/8772/sugeest1.png\" alt=\"enter image description here\"></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "291618": "Refer to this paper:\n\n\"Data Distillation: Towards Omni-Supervised Learning\"\n- Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He, CVPR 2018\n\n\nhttps://arxiv.org/abs/1712.04440\n\n\n ![enter image description here][1]\n\n\n ![enter image description here][2]\n\n\nthis is almost a natural way to improve results because:\n\n-  you already have an ensemble of classifiers (especially for kaggle competition)\n\n- you need to improve detection first (before instance segmentation)\n\n- there are really many potential training images from the web (especially tissues from histology) but they are no label. You can label a few and use unsupervised learning to label the rest.\n\n- within a given test image, cell nuclei are similar. You may detect  say 80%. If you can learn online from the most confident ones, you can further improve detection rate by maybe 1 or 2%. Also in the strategy of ensembles, you trained zero \"fp classifier\",  so most confident detectoins are usually correct (giving more training samples).  \n    - Deep learning can tolerate about 10% to 20% label noise.  \n    -  Also test images can be moasic to give a bigger image. Hence test images are correlated. knowledge can be propagated.\n    - There is additional information to know if your detection is correct or not. Density of detected nuclei. Low density is using correct. High density are prone to error. Also compare mask region of semantic versus instance segmentation results.  \n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/291618/8703/semi2.png\n  [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/291618/8702/semi1.png",
    "291812": "Nice paper. Easy to read because of no equations at all. Seems easy to adopt too. However, I have a hard time to understand how to covert soft labels (average probabilities from ensembles) into hard labels.  Is something like np.where( a&gt; 0.5, 1, 0)?",
    "291871": "you can find more information here:\n\nhttps://github.com/kweonwooj/papers/issues/64",
    "293197": "observation:\n\n- train and validation LB metric is very high, about 80% for box precision at threshold=0.5 and 70% for mask average precision at threshold=0.5 to 0.9.\n\n- this is achieved despite that is about 5% label error in kaggle train dataset\n\n- test time augmentation shoes that nuclei not detected in one image can be detected in another augmented image.\n\nproblem:\n\n- how to apply ensemble?\n\n---\n\nsolution:\n\n- using the distillation paper above, for a LB test image, we make many augmentations. Then we apply our trained network. Hopefully, all nuclei will get label at least once (this is the pesudo label)\n\n- mix the LB test image (with the pseudo label) with the train image.\n\n- retrain or fine-tune the network.\n\nNOTE: \n\n  - if the label noise is below a certain value, it will work.\n     (e.g. instead of training on augmented test image, you can transfer all pseudo label to the original test LB image)\n\n  - need to take care of mixing portion of pseudo labelled image and human labelled image.",
    "296609": "https://arxiv.org/pdf/1706.04737.pdf\n\n\"Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation\"\nLin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, Danny Z. Chen\n\n\n  ![enter image description here][1]\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/296609/8772/sugeest1.png"
  },
  "source": "meta"
}