{
  "id": 53940,
  "title": "which augmentation works?",
  "url": "/competitions/data-science-bowl-2018/discussion/53940",
  "author_name": "",
  "post_date": "2018-04-07T04:03:32.594333600Z",
  "votes": 21,
  "comment_count": 19,
  "views": 0,
  "content": "<p>A. Gray images</p>\n\n<ol>\n<li><p>noises</p>\n\n<ul><li>surprising, this works and improve the difficult cases in the LB test. I use \"speckle noise: image = image+ gaussian*image\". However, there are some \"side effects\", e.g. the fibre object now become detected. (However, these FP can be removed in other post processing)</li></ul></li>\n<li><p>brightness</p>\n\n<ul><li>CLAHE, gamma, contrast, shift, etc ... there is no significant improvement or degradation</li></ul></li>\n<li><p>geometric</p>\n\n<ul><li>reflect, shift, rotate, scale ... surprising, there is no significant improvement or degradation </li></ul></li>\n<li><p>elastic deformation</p>\n\n<ul><li>haven't tried yet</li></ul></li>\n<li><p>relabel</p>\n\n<ul><li>hand correct some labels ... maybe it work?</li></ul></li>\n<li><p>filter</p>\n\n<ul><li>unsharp mask, gaussian blur,  haven't tried yet</li></ul></li>\n</ol>",
  "messages": [
    {
      "id": "310306",
      "postDate": "04/07/2018 04:03:32",
      "content": "<p>A. Gray images</p>\n\n<ol>\n<li><p>noises</p>\n\n<ul><li>surprising, this works and improve the difficult cases in the LB test. I use \"speckle noise: image = image+ gaussian*image\". However, there are some \"side effects\", e.g. the fibre object now become detected. (However, these FP can be removed in other post processing)</li></ul></li>\n<li><p>brightness</p>\n\n<ul><li>CLAHE, gamma, contrast, shift, etc ... there is no significant improvement or degradation</li></ul></li>\n<li><p>geometric</p>\n\n<ul><li>reflect, shift, rotate, scale ... surprising, there is no significant improvement or degradation </li></ul></li>\n<li><p>elastic deformation</p>\n\n<ul><li>haven't tried yet</li></ul></li>\n<li><p>relabel</p>\n\n<ul><li>hand correct some labels ... maybe it work?</li></ul></li>\n<li><p>filter</p>\n\n<ul><li>unsharp mask, gaussian blur,  haven't tried yet</li></ul></li>\n</ol>",
      "rawMarkdown": "A. Gray images\n\n1. noises\n\n- surprising, this works and improve the difficult cases in the LB test. I use \"speckle noise: image = image+ gaussian*image\". However, there are some \"side effects\", e.g. the fibre object now become detected. (However, these FP can be removed in other post processing)\n\n2. brightness\n\n- CLAHE, gamma, contrast, shift, etc ... there is no significant improvement or degradation\n\n3. geometric\n\n- reflect, shift, rotate, scale ... surprising, there is no significant improvement or degradation \n\n4. elastic deformation\n\n- haven't tried yet\n\n5. relabel\n\n- hand correct some labels ... maybe it work?\n\n6. filter\n\n- unsharp mask, gaussian blur,  haven't tried yet",
      "votes": null
    },
    {
      "id": "310307",
      "postDate": "04/07/2018 04:12:16",
      "content": "<p>Every Heng's posts are very useful. I used geometric and it helps to improve a little than without using it. You may need to add random crop in geometric because it is main augmentation to improve performance. </p>",
      "rawMarkdown": "Every Heng's posts are very useful. I used geometric and it helps to improve a little than without using it. You may need to add random crop in geometric because it is main augmentation to improve performance.",
      "votes": null
    },
    {
      "id": "310353",
      "postDate": "04/07/2018 08:53:37",
      "content": "<p>Do you use size to remove fibre?</p>",
      "rawMarkdown": "Do you use size to remove fibre?",
      "votes": null
    },
    {
      "id": "310381",
      "postDate": "04/07/2018 10:30:05",
      "content": "<p>i have add random crop on my training, it's help a lot. is there any other useful method ?thanks</p>",
      "rawMarkdown": "i have add random crop on my training, it's help a lot. is there any other useful method ?thanks",
      "votes": null
    },
    {
      "id": "310396",
      "postDate": "04/07/2018 11:38:15",
      "content": "<p>@Nino are you using mask RCNN? If so where did you add random crop? Can you possibly point to an example? Thanks. I am using Matterport's mask_RCNN.</p>",
      "rawMarkdown": "Nino are you using mask RCNN? If so where did you add random crop? Can you possibly point to an example? Thanks. I am using Matterport's mask_RCNN.",
      "votes": null
    },
    {
      "id": "310451",
      "postDate": "04/07/2018 15:20:55",
      "content": "<p>yup... CLAHE hurts my performance on color samples, no influence on grayscale ones. Thought it would be useful :(</p>",
      "rawMarkdown": "yup... CLAHE hurts my performance on color samples, no influence on grayscale ones. Thought it would be useful :(",
      "votes": null
    },
    {
      "id": "310465",
      "postDate": "04/07/2018 16:22:38",
      "content": "<p>@YaGana Sheriff-Hussaini add the method in load image function</p>",
      "rawMarkdown": "YaGana Sheriff-Hussaini add the method in load image function",
      "votes": null
    },
    {
      "id": "310491",
      "postDate": "04/07/2018 17:32:31",
      "content": "<p>For color, color normalization may help. I did not try it. <a href=\"https://github.com/Peter554/StainTools\">https://github.com/Peter554/StainTools</a></p>",
      "rawMarkdown": "For color, color normalization may help. I did not try it. https://github.com/Peter554/StainTools",
      "votes": null
    },
    {
      "id": "310503",
      "postDate": "04/07/2018 18:42:13",
      "content": "<p>Given this is really sensitive to single pixel error I reckon, and observed for the case I implemented, that transformations that require interpolation (like rotating by angles not multiple of 90 degrees) degrade performance.</p>\n\n<p>I think it is because the annotations become way less accurate.</p>",
      "rawMarkdown": "Given this is really sensitive to single pixel error I reckon, and observed for the case I implemented, that transformations that require interpolation (like rotating by angles not multiple of 90 degrees) degrade performance.\n\nI think it is because the annotations become way less accurate.",
      "votes": null
    },
    {
      "id": "310556",
      "postDate": "04/08/2018 00:52:49",
      "content": "<p>I tried using color shift only. It actually helps.</p>",
      "rawMarkdown": "I tried using color shift only. It actually helps.",
      "votes": null
    },
    {
      "id": "310560",
      "postDate": "04/08/2018 01:03:25",
      "content": "<p>Thanks @Jiafin. I am looking at color normalization codes posted above. Are you able to share your color shift code at this time only bacuase I feel like I am doing it wrong? </p>",
      "rawMarkdown": "Thanks @Jiafin. I am looking at color normalization codes posted above. Are you able to share your color shift code at this time only bacuase I feel like I am doing it wrong?",
      "votes": null
    },
    {
      "id": "310611",
      "postDate": "04/08/2018 05:04:12",
      "content": "<p>I used the Gray-level co-ocurrence matrix to get some information about the images. Then I used hundred of random generated filters to see how they change pixel correlation and contrast. \nTo my model (which is not a neural network) the increase of correlation b/w pixels reduced overall performance.\nThat may be useful statistics to \"automatic\"  hyperparameter tunning which may help on stage-two (since hyperparameters can't be adjusted once first stage algorithm is submitted)</p>",
      "rawMarkdown": "I used the Gray-level co-ocurrence matrix to get some information about the images. Then I used hundred of random generated filters to see how they change pixel correlation and contrast. \nTo my model (which is not a neural network) the increase of correlation b/w pixels reduced overall performance.\nThat may be useful statistics to \"automatic\"  hyperparameter tunning which may help on stage-two (since hyperparameters can't be adjusted once first stage algorithm is submitted)",
      "votes": null
    },
    {
      "id": "310662",
      "postDate": "04/08/2018 07:41:17",
      "content": "<p>how to merge the results by different augment images.</p>",
      "rawMarkdown": "how to merge the results by different augment images.",
      "votes": null
    },
    {
      "id": "310942",
      "postDate": "04/09/2018 04:38:22",
      "content": "<p>Here is how I do color shifting for now:</p>\n\n<p>color_delta = [np.random.randint(-50,50),np.random.randint(-50,50),np.random.randint(-50,50)]</p>\n\n<p>image = image + color_delta</p>\n\n<p>image = np.clip(image, 0, 255).astype(np.uint8)</p>",
      "rawMarkdown": "Here is how I do color shifting for now:\n\n\ncolor_delta = [np.random.randint(-50,50),np.random.randint(-50,50),np.random.randint(-50,50)]\n\nimage = image + color_delta\n\nimage = np.clip(image, 0, 255).astype(np.uint8)",
      "votes": null
    },
    {
      "id": "310952",
      "postDate": "04/09/2018 05:00:01",
      "content": "<p>@Jiafin, thanks so much. Did you use this in loading the train as well as test data?</p>",
      "rawMarkdown": "Jiafin, thanks so much. Did you use this in loading the train as well as test data?",
      "votes": null
    },
    {
      "id": "311627",
      "postDate": "04/10/2018 11:39:30",
      "content": "<p>I was augmenting so hard I don't even know how my networks managed to learn anything.</p>",
      "rawMarkdown": "I was augmenting so hard I don't even know how my networks managed to learn anything.",
      "votes": null
    },
    {
      "id": "312018",
      "postDate": "04/11/2018 03:57:26",
      "content": "<p>I only used for training.</p>",
      "rawMarkdown": "I only used for training.",
      "votes": null
    },
    {
      "id": "312032",
      "postDate": "04/11/2018 04:33:39",
      "content": "<p>Thanks @Jiafin. Is that with the original training data for Konstantin Lopuhin's fixed drop in training data?</p>",
      "rawMarkdown": "Thanks @Jiafin. Is that with the original training data for Konstantin Lopuhin's fixed drop in training data?",
      "votes": null
    },
    {
      "id": "312035",
      "postDate": "04/11/2018 04:36:58",
      "content": "<p>I used the fixed training data. I don't remember its name.</p>",
      "rawMarkdown": "I used the fixed training data. I don't remember its name.",
      "votes": null
    },
    {
      "id": "1095279",
      "postDate": "11/29/2020 12:41:48",
      "content": "<p>bro?<br>\nhow to calculate clahe manually?</p>",
      "rawMarkdown": "bro?\nhow to calculate clahe manually?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1095279,
      "author_name": "felixindra46",
      "author_url": "",
      "post_date": "11/29/2020 12:41:48",
      "content": "<p>bro?<br>\nhow to calculate clahe manually?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 310307,
      "author_name": "john1231983",
      "author_url": "",
      "post_date": "04/07/2018 04:12:16",
      "content": "<p>Every Heng's posts are very useful. I used geometric and it helps to improve a little than without using it. You may need to add random crop in geometric because it is main augmentation to improve performance. </p>",
      "votes": null,
      "replies": [
        {
          "id": 310381,
          "author_name": "ninocy",
          "author_url": "",
          "post_date": "04/07/2018 10:30:05",
          "content": "<p>i have add random crop on my training, it's help a lot. is there any other useful method ?thanks</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 310396,
          "author_name": "sheriytm",
          "author_url": "",
          "post_date": "04/07/2018 11:38:15",
          "content": "<p>@Nino are you using mask RCNN? If so where did you add random crop? Can you possibly point to an example? Thanks. I am using Matterport's mask_RCNN.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 310465,
          "author_name": "ermiao",
          "author_url": "",
          "post_date": "04/07/2018 16:22:38",
          "content": "<p>@YaGana Sheriff-Hussaini add the method in load image function</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 310662,
          "author_name": "liyawei",
          "author_url": "",
          "post_date": "04/08/2018 07:41:17",
          "content": "<p>how to merge the results by different augment images.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 310353,
      "author_name": "t831011",
      "author_url": "",
      "post_date": "04/07/2018 08:53:37",
      "content": "<p>Do you use size to remove fibre?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 310451,
      "author_name": "atm584",
      "author_url": "",
      "post_date": "04/07/2018 15:20:55",
      "content": "<p>yup... CLAHE hurts my performance on color samples, no influence on grayscale ones. Thought it would be useful :(</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 310491,
      "author_name": "john1231983",
      "author_url": "",
      "post_date": "04/07/2018 17:32:31",
      "content": "<p>For color, color normalization may help. I did not try it. <a href=\"https://github.com/Peter554/StainTools\">https://github.com/Peter554/StainTools</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 310503,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "04/07/2018 18:42:13",
      "content": "<p>Given this is really sensitive to single pixel error I reckon, and observed for the case I implemented, that transformations that require interpolation (like rotating by angles not multiple of 90 degrees) degrade performance.</p>\n\n<p>I think it is because the annotations become way less accurate.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 310556,
      "author_name": "zhujf553",
      "author_url": "",
      "post_date": "04/08/2018 00:52:49",
      "content": "<p>I tried using color shift only. It actually helps.</p>",
      "votes": null,
      "replies": [
        {
          "id": 310560,
          "author_name": "sheriytm",
          "author_url": "",
          "post_date": "04/08/2018 01:03:25",
          "content": "<p>Thanks @Jiafin. I am looking at color normalization codes posted above. Are you able to share your color shift code at this time only bacuase I feel like I am doing it wrong? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 310942,
          "author_name": "zhujf553",
          "author_url": "",
          "post_date": "04/09/2018 04:38:22",
          "content": "<p>Here is how I do color shifting for now:</p>\n\n<p>color_delta = [np.random.randint(-50,50),np.random.randint(-50,50),np.random.randint(-50,50)]</p>\n\n<p>image = image + color_delta</p>\n\n<p>image = np.clip(image, 0, 255).astype(np.uint8)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 310952,
          "author_name": "sheriytm",
          "author_url": "",
          "post_date": "04/09/2018 05:00:01",
          "content": "<p>@Jiafin, thanks so much. Did you use this in loading the train as well as test data?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 312018,
          "author_name": "zhujf553",
          "author_url": "",
          "post_date": "04/11/2018 03:57:26",
          "content": "<p>I only used for training.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 312032,
          "author_name": "sheriytm",
          "author_url": "",
          "post_date": "04/11/2018 04:33:39",
          "content": "<p>Thanks @Jiafin. Is that with the original training data for Konstantin Lopuhin's fixed drop in training data?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 312035,
          "author_name": "zhujf553",
          "author_url": "",
          "post_date": "04/11/2018 04:36:58",
          "content": "<p>I used the fixed training data. I don't remember its name.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 310611,
      "author_name": "phmonforte",
      "author_url": "",
      "post_date": "04/08/2018 05:04:12",
      "content": "<p>I used the Gray-level co-ocurrence matrix to get some information about the images. Then I used hundred of random generated filters to see how they change pixel correlation and contrast. \nTo my model (which is not a neural network) the increase of correlation b/w pixels reduced overall performance.\nThat may be useful statistics to \"automatic\"  hyperparameter tunning which may help on stage-two (since hyperparameters can't be adjusted once first stage algorithm is submitted)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 311627,
      "author_name": "thenickname",
      "author_url": "",
      "post_date": "04/10/2018 11:39:30",
      "content": "<p>I was augmenting so hard I don't even know how my networks managed to learn anything.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "310306": "A. Gray images\n\n1. noises\n\n- surprising, this works and improve the difficult cases in the LB test. I use \"speckle noise: image = image+ gaussian*image\". However, there are some \"side effects\", e.g. the fibre object now become detected. (However, these FP can be removed in other post processing)\n\n2. brightness\n\n- CLAHE, gamma, contrast, shift, etc ... there is no significant improvement or degradation\n\n3. geometric\n\n- reflect, shift, rotate, scale ... surprising, there is no significant improvement or degradation \n\n4. elastic deformation\n\n- haven't tried yet\n\n5. relabel\n\n- hand correct some labels ... maybe it work?\n\n6. filter\n\n- unsharp mask, gaussian blur,  haven't tried yet",
    "310307": "Every Heng's posts are very useful. I used geometric and it helps to improve a little than without using it. You may need to add random crop in geometric because it is main augmentation to improve performance.",
    "310353": "Do you use size to remove fibre?",
    "310381": "i have add random crop on my training, it's help a lot. is there any other useful method ?thanks",
    "310396": "Nino are you using mask RCNN? If so where did you add random crop? Can you possibly point to an example? Thanks. I am using Matterport's mask_RCNN.",
    "310451": "yup... CLAHE hurts my performance on color samples, no influence on grayscale ones. Thought it would be useful :(",
    "310465": "YaGana Sheriff-Hussaini add the method in load image function",
    "310491": "For color, color normalization may help. I did not try it. https://github.com/Peter554/StainTools",
    "310503": "Given this is really sensitive to single pixel error I reckon, and observed for the case I implemented, that transformations that require interpolation (like rotating by angles not multiple of 90 degrees) degrade performance.\n\nI think it is because the annotations become way less accurate.",
    "310556": "I tried using color shift only. It actually helps.",
    "310560": "Thanks @Jiafin. I am looking at color normalization codes posted above. Are you able to share your color shift code at this time only bacuase I feel like I am doing it wrong?",
    "310611": "I used the Gray-level co-ocurrence matrix to get some information about the images. Then I used hundred of random generated filters to see how they change pixel correlation and contrast. \nTo my model (which is not a neural network) the increase of correlation b/w pixels reduced overall performance.\nThat may be useful statistics to \"automatic\"  hyperparameter tunning which may help on stage-two (since hyperparameters can't be adjusted once first stage algorithm is submitted)",
    "310662": "how to merge the results by different augment images.",
    "310942": "Here is how I do color shifting for now:\n\n\ncolor_delta = [np.random.randint(-50,50),np.random.randint(-50,50),np.random.randint(-50,50)]\n\nimage = image + color_delta\n\nimage = np.clip(image, 0, 255).astype(np.uint8)",
    "310952": "Jiafin, thanks so much. Did you use this in loading the train as well as test data?",
    "311627": "I was augmenting so hard I don't even know how my networks managed to learn anything.",
    "312018": "I only used for training.",
    "312032": "Thanks @Jiafin. Is that with the original training data for Konstantin Lopuhin's fixed drop in training data?",
    "312035": "I used the fixed training data. I don't remember its name.",
    "1095279": "bro?\nhow to calculate clahe manually?"
  },
  "source": "meta"
}