{
  "id": 337697,
  "title": "Experiments to use post processing to improve result (it did).",
  "url": "/competitions/denoising-shabby-pages/discussion/337697",
  "author_name": "",
  "post_date": "2022-07-17T08:45:33.288872900Z",
  "votes": 5,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi guys,</p>\n<p>From my previous post, i tried with several post processing methods and it did improved the final result. Here's the setup:</p>\n<p>Model - <strong>Autoencoder</strong><br>\nCode - <a href=\"https://www.kaggle.com/code/weisinx7/denoising-by-autoencoder-simple-data\" target=\"_blank\">https://www.kaggle.com/code/weisinx7/denoising-by-autoencoder-simple-data</a></p>\n<p>Raw result without any post-processing method:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Ffc8b1facbdf200dfcedcc97d9f5ecbd1%2F11111.png?generation=1658045446138540&amp;alt=media\" alt=\"\"></p>\n<p><strong>Post processing methods:</strong></p>\n<ol>\n<li><p><strong>Median filter</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Fe249e2fede0ef66f499898fac152605d%2F11111.png?generation=1658045678686725&amp;alt=media\" alt=\"\"></p></li>\n<li><p><strong>FFT filter</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F83dffa1d7dd7a92b81353c109555eca3%2F11111.png?generation=1658046166800925&amp;alt=media\" alt=\"\"></p></li>\n<li><p><strong>Remove small blobs</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F6eb5ae35c1fed96f1f931cf613624c57%2F11111.png?generation=1658045947994516&amp;alt=media\" alt=\"\"></p></li>\n<li><p><strong>Simple black-white thresholding</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F695cab317e3494aaec0303e6ed203852%2F11111.png?generation=1658045993699894&amp;alt=media\" alt=\"\"></p></li>\n<li><p><strong>Crimmins Speckle removal</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Fa4442b1a3ec82368bfc0f24c9802e203%2F11111.png?generation=1658046052796827&amp;alt=media\" alt=\"\"></p></li>\n</ol>\n<p>Fusing several post-processing methods:</p>\n<ol>\n<li><p><strong>Median filter  + Remove small blobs</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F5db84e6135b00f95256b7061f44b3686%2F11111.png?generation=1658046326253912&amp;alt=media\" alt=\"\"></p></li>\n<li><p><strong>Median filter  + Crimmins Speckle removal</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F73201fdfe04c5a608241267498f0259a%2F11111.png?generation=1658046399429369&amp;alt=media\" alt=\"\"></p></li>\n</ol>\n<p>So for single filter, median filter actually work the best, while the others are just average and some are not even able to improve the result. Fusing several methods are able to further improve the result too.</p>\n<p>Here are the snippet of code:</p>\n<ol>\n<li><p><strong>Median filter:</strong><br>\n<code>from scipy import ndimage</code><br>\n<code>img_med = ndimage.median_filter(img, size=2)</code></p></li>\n<li><p><strong>FFT filter</strong><br>\n<code>keep_fraction = 0.25</code><br>\n<code>img_fft = fftpack.fft2(img)</code><br>\n<code>r, c = img_fft.shape</code></p>\n<p><code># Set to zero all rows with indices between r*keep_fraction and</code><br>\n<code># r*(1-keep_fraction):</code><br>\n<code>img_fft[int(r*keep_fraction):int(r*(1-keep_fraction))] = 0</code></p>\n<p><code># Similarly with the columns:</code><br>\n<code>img_fft[:, int(c*keep_fraction):int(c*(1-keep_fraction))] = 0</code><br>\n<code>img_fft_filter = fftpack.ifft2(img_fft).real</code></p></li>\n<li><p><strong>Remove small blobs</strong><br>\n<code># simple thresholding</code><br>\n<code>threshold = 245</code><br>\n<code>img_binary = img.copy()</code><br>\n<code>img_binary[img_binary&lt;threshold] = 0</code><br>\n<code>img_binary[img_binary&gt;=threshold] = 255</code></p>\n<p><code># invert so that small blob becomes white</code><br>\n<code>img_binary = 255-img_binary</code></p>\n<p><code># get connected components</code><br>\n<code>output = cv2.connectedComponentsWithStats(img_binary, 4, cv2.CV_32S)</code></p>\n<p><code># get stats</code><br>\n<code>num_labels, labels, stats, centroids = output</code></p>\n<p><code># remove area &lt; 5 pixels</code><br>\n<code>for i in range(num_labels):</code><br>\n<code>area = stats[i, cv2.CC_STAT_AREA]</code><br>\n<code>if area&lt;5:</code><br>\n<code>img[np.where(labels==i)] = 255</code></p></li>\n<li><p><strong>Simple black-white thresholding</strong><br>\n<code>thres_val = 224</code><br>\n<code>img[img&gt;thres_val] = 255</code><br>\n<code>img[img&lt;=thres_val] = 0</code></p></li>\n<li><p><strong>Crimmins Speckle removal</strong><br>\nIt's too long to post here, refer here instead: <a href=\"https://github.com/m4nv1r/medium_articles/blob/master/Image_Filters_in_Python.ipynb\" target=\"_blank\">https://github.com/m4nv1r/medium_articles/blob/master/Image_Filters_in_Python.ipynb</a></p></li>\n</ol>\n<p>I hope this will be helpful for you guys and please let me know if you have some interesting post processing method to try out too.</p>",
  "messages": [
    {
      "id": "1858838",
      "postDate": "07/17/2022 08:45:33",
      "content": "<p>Hi guys,</p>\n<p>From my previous post, i tried with several post processing methods and it did improved the final result. Here's the setup:</p>\n<p>Model - <strong>Autoencoder</strong><br>\nCode - <a href=\"https://www.kaggle.com/code/weisinx7/denoising-by-autoencoder-simple-data\" target=\"_blank\">https://www.kaggle.com/code/weisinx7/denoising-by-autoencoder-simple-data</a></p>\n<p>Raw result without any post-processing method:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Ffc8b1facbdf200dfcedcc97d9f5ecbd1%2F11111.png?generation=1658045446138540&amp;alt=media\" alt=\"\"></p>\n<p><strong>Post processing methods:</strong></p>\n<ol>\n<li><p><strong>Median filter</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Fe249e2fede0ef66f499898fac152605d%2F11111.png?generation=1658045678686725&amp;alt=media\" alt=\"\"></p></li>\n<li><p><strong>FFT filter</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F83dffa1d7dd7a92b81353c109555eca3%2F11111.png?generation=1658046166800925&amp;alt=media\" alt=\"\"></p></li>\n<li><p><strong>Remove small blobs</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F6eb5ae35c1fed96f1f931cf613624c57%2F11111.png?generation=1658045947994516&amp;alt=media\" alt=\"\"></p></li>\n<li><p><strong>Simple black-white thresholding</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F695cab317e3494aaec0303e6ed203852%2F11111.png?generation=1658045993699894&amp;alt=media\" alt=\"\"></p></li>\n<li><p><strong>Crimmins Speckle removal</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Fa4442b1a3ec82368bfc0f24c9802e203%2F11111.png?generation=1658046052796827&amp;alt=media\" alt=\"\"></p></li>\n</ol>\n<p>Fusing several post-processing methods:</p>\n<ol>\n<li><p><strong>Median filter  + Remove small blobs</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F5db84e6135b00f95256b7061f44b3686%2F11111.png?generation=1658046326253912&amp;alt=media\" alt=\"\"></p></li>\n<li><p><strong>Median filter  + Crimmins Speckle removal</strong><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F73201fdfe04c5a608241267498f0259a%2F11111.png?generation=1658046399429369&amp;alt=media\" alt=\"\"></p></li>\n</ol>\n<p>So for single filter, median filter actually work the best, while the others are just average and some are not even able to improve the result. Fusing several methods are able to further improve the result too.</p>\n<p>Here are the snippet of code:</p>\n<ol>\n<li><p><strong>Median filter:</strong><br>\n<code>from scipy import ndimage</code><br>\n<code>img_med = ndimage.median_filter(img, size=2)</code></p></li>\n<li><p><strong>FFT filter</strong><br>\n<code>keep_fraction = 0.25</code><br>\n<code>img_fft = fftpack.fft2(img)</code><br>\n<code>r, c = img_fft.shape</code></p>\n<p><code># Set to zero all rows with indices between r*keep_fraction and</code><br>\n<code># r*(1-keep_fraction):</code><br>\n<code>img_fft[int(r*keep_fraction):int(r*(1-keep_fraction))] = 0</code></p>\n<p><code># Similarly with the columns:</code><br>\n<code>img_fft[:, int(c*keep_fraction):int(c*(1-keep_fraction))] = 0</code><br>\n<code>img_fft_filter = fftpack.ifft2(img_fft).real</code></p></li>\n<li><p><strong>Remove small blobs</strong><br>\n<code># simple thresholding</code><br>\n<code>threshold = 245</code><br>\n<code>img_binary = img.copy()</code><br>\n<code>img_binary[img_binary&lt;threshold] = 0</code><br>\n<code>img_binary[img_binary&gt;=threshold] = 255</code></p>\n<p><code># invert so that small blob becomes white</code><br>\n<code>img_binary = 255-img_binary</code></p>\n<p><code># get connected components</code><br>\n<code>output = cv2.connectedComponentsWithStats(img_binary, 4, cv2.CV_32S)</code></p>\n<p><code># get stats</code><br>\n<code>num_labels, labels, stats, centroids = output</code></p>\n<p><code># remove area &lt; 5 pixels</code><br>\n<code>for i in range(num_labels):</code><br>\n<code>area = stats[i, cv2.CC_STAT_AREA]</code><br>\n<code>if area&lt;5:</code><br>\n<code>img[np.where(labels==i)] = 255</code></p></li>\n<li><p><strong>Simple black-white thresholding</strong><br>\n<code>thres_val = 224</code><br>\n<code>img[img&gt;thres_val] = 255</code><br>\n<code>img[img&lt;=thres_val] = 0</code></p></li>\n<li><p><strong>Crimmins Speckle removal</strong><br>\nIt's too long to post here, refer here instead: <a href=\"https://github.com/m4nv1r/medium_articles/blob/master/Image_Filters_in_Python.ipynb\" target=\"_blank\">https://github.com/m4nv1r/medium_articles/blob/master/Image_Filters_in_Python.ipynb</a></p></li>\n</ol>\n<p>I hope this will be helpful for you guys and please let me know if you have some interesting post processing method to try out too.</p>",
      "rawMarkdown": "Hi guys,\n\nFrom my previous post, i tried with several post processing methods and it did improved the final result. Here's the setup:\n\nModel - **Autoencoder**\nCode - https://www.kaggle.com/code/weisinx7/denoising-by-autoencoder-simple-data\n\nRaw result without any post-processing method:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Ffc8b1facbdf200dfcedcc97d9f5ecbd1%2F11111.png?generation=1658045446138540&alt=media)\n\n**Post processing methods:**\n\n1. **Median filter**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Fe249e2fede0ef66f499898fac152605d%2F11111.png?generation=1658045678686725&alt=media)\n\n2. **FFT filter**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F83dffa1d7dd7a92b81353c109555eca3%2F11111.png?generation=1658046166800925&alt=media)\n\n3. **Remove small blobs**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F6eb5ae35c1fed96f1f931cf613624c57%2F11111.png?generation=1658045947994516&alt=media)\n\n4. **Simple black-white thresholding**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F695cab317e3494aaec0303e6ed203852%2F11111.png?generation=1658045993699894&alt=media)\n\n5. **Crimmins Speckle removal**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Fa4442b1a3ec82368bfc0f24c9802e203%2F11111.png?generation=1658046052796827&alt=media)\n\nFusing several post-processing methods:\n\n6. **Median filter  + Remove small blobs**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F5db84e6135b00f95256b7061f44b3686%2F11111.png?generation=1658046326253912&alt=media)\n\n7. **Median filter  + Crimmins Speckle removal**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F73201fdfe04c5a608241267498f0259a%2F11111.png?generation=1658046399429369&alt=media)\n\nSo for single filter, median filter actually work the best, while the others are just average and some are not even able to improve the result. Fusing several methods are able to further improve the result too.\n\nHere are the snippet of code:\n\n1. **Median filter:**\n`from scipy import ndimage`\n`img_med = ndimage.median_filter(img, size=2)`\n\n2.  **FFT filter**\n    `keep_fraction = 0.25`\n    `img_fft = fftpack.fft2(img)`\n    `r, c = img_fft.shape`\n    \n    `# Set to zero all rows with indices between r*keep_fraction and`\n    `# r*(1-keep_fraction):`\n    `img_fft[int(r*keep_fraction):int(r*(1-keep_fraction))] = 0`\n    \n    `# Similarly with the columns:`\n    `img_fft[:, int(c*keep_fraction):int(c*(1-keep_fraction))] = 0`\n    `img_fft_filter = fftpack.ifft2(img_fft).real`\n\n3. **Remove small blobs**\n    `# simple thresholding`\n    `threshold = 245`\n    `img_binary = img.copy()`\n    `img_binary[img_binary<threshold] = 0`\n    `img_binary[img_binary>=threshold] = 255`\n   \n    `# invert so that small blob becomes white`\n    `img_binary = 255-img_binary`\n    \n    `# get connected components`\n    `output = cv2.connectedComponentsWithStats(img_binary, 4, cv2.CV_32S)`\n    \n    `# get stats`\n    `num_labels, labels, stats, centroids = output`\n    \n    `# remove area < 5 pixels`\n    `for i in range(num_labels):`\n    `    area = stats[i, cv2.CC_STAT_AREA]`\n    `    if area<5:`\n    `        img[np.where(labels==i)] = 255`\n\n4. **Simple black-white thresholding**\n    `thres_val = 224 `\n    `img[img>thres_val] = 255`\n    `img[img<=thres_val] = 0`\n\n5. **Crimmins Speckle removal**\nIt's too long to post here, refer here instead: https://github.com/m4nv1r/medium_articles/blob/master/Image_Filters_in_Python.ipynb\n\nI hope this will be helpful for you guys and please let me know if you have some interesting post processing method to try out too.",
      "votes": null
    },
    {
      "id": "1858843",
      "postDate": "07/17/2022 08:47:34",
      "content": "<p>Wonderful post! Keep it up!</p>",
      "rawMarkdown": "Wonderful post! Keep it up!",
      "votes": null
    },
    {
      "id": "1865192",
      "postDate": "07/21/2022 15:46:55",
      "content": "<p>Wow great work.</p>",
      "rawMarkdown": "Wow great work.",
      "votes": null
    },
    {
      "id": "1885418",
      "postDate": "08/05/2022 07:07:17",
      "content": "<p>I added an article with more details here:<br>\n<a href=\"https://medium.com/@ck91wei/post-processing-methods-to-improve-denoised-images-eb7c293cc8a3\" target=\"_blank\">https://medium.com/@ck91wei/post-processing-methods-to-improve-denoised-images-eb7c293cc8a3</a></p>",
      "rawMarkdown": "I added an article with more details here:\nhttps://medium.com/@ck91wei/post-processing-methods-to-improve-denoised-images-eb7c293cc8a3",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1858843,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "07/17/2022 08:47:34",
      "content": "<p>Wonderful post! Keep it up!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1865192,
      "author_name": "deepdreamx",
      "author_url": "",
      "post_date": "07/21/2022 15:46:55",
      "content": "<p>Wow great work.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1885418,
      "author_name": "weisinx7",
      "author_url": "",
      "post_date": "08/05/2022 07:07:17",
      "content": "<p>I added an article with more details here:<br>\n<a href=\"https://medium.com/@ck91wei/post-processing-methods-to-improve-denoised-images-eb7c293cc8a3\" target=\"_blank\">https://medium.com/@ck91wei/post-processing-methods-to-improve-denoised-images-eb7c293cc8a3</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1858838": "Hi guys,\n\nFrom my previous post, i tried with several post processing methods and it did improved the final result. Here's the setup:\n\nModel - **Autoencoder**\nCode - https://www.kaggle.com/code/weisinx7/denoising-by-autoencoder-simple-data\n\nRaw result without any post-processing method:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Ffc8b1facbdf200dfcedcc97d9f5ecbd1%2F11111.png?generation=1658045446138540&alt=media)\n\n**Post processing methods:**\n\n1. **Median filter**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Fe249e2fede0ef66f499898fac152605d%2F11111.png?generation=1658045678686725&alt=media)\n\n2. **FFT filter**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F83dffa1d7dd7a92b81353c109555eca3%2F11111.png?generation=1658046166800925&alt=media)\n\n3. **Remove small blobs**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F6eb5ae35c1fed96f1f931cf613624c57%2F11111.png?generation=1658045947994516&alt=media)\n\n4. **Simple black-white thresholding**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F695cab317e3494aaec0303e6ed203852%2F11111.png?generation=1658045993699894&alt=media)\n\n5. **Crimmins Speckle removal**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2Fa4442b1a3ec82368bfc0f24c9802e203%2F11111.png?generation=1658046052796827&alt=media)\n\nFusing several post-processing methods:\n\n6. **Median filter  + Remove small blobs**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F5db84e6135b00f95256b7061f44b3686%2F11111.png?generation=1658046326253912&alt=media)\n\n7. **Median filter  + Crimmins Speckle removal**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3363018%2F73201fdfe04c5a608241267498f0259a%2F11111.png?generation=1658046399429369&alt=media)\n\nSo for single filter, median filter actually work the best, while the others are just average and some are not even able to improve the result. Fusing several methods are able to further improve the result too.\n\nHere are the snippet of code:\n\n1. **Median filter:**\n`from scipy import ndimage`\n`img_med = ndimage.median_filter(img, size=2)`\n\n2.  **FFT filter**\n    `keep_fraction = 0.25`\n    `img_fft = fftpack.fft2(img)`\n    `r, c = img_fft.shape`\n    \n    `# Set to zero all rows with indices between r*keep_fraction and`\n    `# r*(1-keep_fraction):`\n    `img_fft[int(r*keep_fraction):int(r*(1-keep_fraction))] = 0`\n    \n    `# Similarly with the columns:`\n    `img_fft[:, int(c*keep_fraction):int(c*(1-keep_fraction))] = 0`\n    `img_fft_filter = fftpack.ifft2(img_fft).real`\n\n3. **Remove small blobs**\n    `# simple thresholding`\n    `threshold = 245`\n    `img_binary = img.copy()`\n    `img_binary[img_binary<threshold] = 0`\n    `img_binary[img_binary>=threshold] = 255`\n   \n    `# invert so that small blob becomes white`\n    `img_binary = 255-img_binary`\n    \n    `# get connected components`\n    `output = cv2.connectedComponentsWithStats(img_binary, 4, cv2.CV_32S)`\n    \n    `# get stats`\n    `num_labels, labels, stats, centroids = output`\n    \n    `# remove area < 5 pixels`\n    `for i in range(num_labels):`\n    `    area = stats[i, cv2.CC_STAT_AREA]`\n    `    if area<5:`\n    `        img[np.where(labels==i)] = 255`\n\n4. **Simple black-white thresholding**\n    `thres_val = 224 `\n    `img[img>thres_val] = 255`\n    `img[img<=thres_val] = 0`\n\n5. **Crimmins Speckle removal**\nIt's too long to post here, refer here instead: https://github.com/m4nv1r/medium_articles/blob/master/Image_Filters_in_Python.ipynb\n\nI hope this will be helpful for you guys and please let me know if you have some interesting post processing method to try out too.",
    "1858843": "Wonderful post! Keep it up!",
    "1865192": "Wow great work.",
    "1885418": "I added an article with more details here:\nhttps://medium.com/@ck91wei/post-processing-methods-to-improve-denoised-images-eb7c293cc8a3"
  },
  "source": "meta"
}