![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1202828%2F141e471009fb0f3a0358c6e647eff3ee%2FCycleGAN%201.png?generation=1596267020743984&alt=media\")
Hey everybody, I wanted to share something that didn't end up working out for me, but I think is still pretty interesting.
\n\nBack at the beginning of the contest I was trying to generate more training examples using CycleGAN. My thinking was that CycleGAN should preserve the location of the wheat heads so I wouldn't need to worry about the bounding boxes.
\n\nI tried to transform images from the Arvalis_1 subset to the Ethz_1 subset. I chose those two because they had the most examples. In hindsight, they may not have been the best choices since there seems to be a lot of variation in Arvalis_1 and I'm not sure more images that look like Ethz_1 would be helpful.
\n\nIt looked like it was working really well:
\n\n
But unfortunately, when I used fake/generated images into train my object detector, it was slower to converge and performed worse on the validation set. It turns out the GAN was hallucinating wheat textures in brighter areas where they didn't belong and removing the wheat texture in other places. So the bounding boxes weren't correct:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1202828%2F529e794153878e6fe746e94d4a657deb%2FCycleGan2.jpg?generation=1596267666137025&alt=media\")
So I tried adding a second discriminator and masking out the wheat heads / background. So discriminator 1 only sees wheat heads and discriminator 2 only sees background. That way, if the GAN puts wheat heads in the wrong spot or vice versa, the discriminator should catch it.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1202828%2F82443fccce8207e20a55583e80cc3f3e%2FCycleGan3.jpg?generation=1596268417491010&alt=media\")
Unfortunately, the discriminator learned that the bounding box distributions for the two subsets... and to compensate the GAN stated adding grey boxes around the bounding boxes to fool the discriminators! Pretty cool, but not super useful for the contest.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1202828%2F82e1b36d34d3250a3e59006f0240609a%2FCycleGan4.jpg?generation=1596268761505192&alt=media\")
I tried adding a third discriminator that used the whole (unmasked) image but at that point I think the whole thing was over-constrained and it failed to make any progress on the losses.
\n\nI retried the 2 discriminator approach but I stopped training before the GAN started adding the grey boxes. It improved my local validation results but didn't improve my LB score. By then I had spent WAY too much time trying to generate training images and I moved on to other ideas.
\n\nAdditional ideas I didn't get to try:\n- Randomly crop/scale the images so the bounding boxes are less predictable.\n- Use different subsets.\n- Allow the GAN to hallucinate new wheat heads and then re-label the generated images using a trained detector.
\n\nDid anybody else try to generate new training images? How did it go? I'd love to hear about it!
", "messages": [ { "id": 953879, "postDate": "2020-08-01T08:00:43.390Z", "content": "Hey everybody, I wanted to share something that didn't end up working out for me, but I think is still pretty interesting.
\n\nBack at the beginning of the contest I was trying to generate more training examples using CycleGAN. My thinking was that CycleGAN should preserve the location of the wheat heads so I wouldn't need to worry about the bounding boxes.
\n\nI tried to transform images from the Arvalis_1 subset to the Ethz_1 subset. I chose those two because they had the most examples. In hindsight, they may not have been the best choices since there seems to be a lot of variation in Arvalis_1 and I'm not sure more images that look like Ethz_1 would be helpful.
\n\nIt looked like it was working really well:
\n\n
But unfortunately, when I used fake/generated images into train my object detector, it was slower to converge and performed worse on the validation set. It turns out the GAN was hallucinating wheat textures in brighter areas where they didn't belong and removing the wheat texture in other places. So the bounding boxes weren't correct:
\n\n
So I tried adding a second discriminator and masking out the wheat heads / background. So discriminator 1 only sees wheat heads and discriminator 2 only sees background. That way, if the GAN puts wheat heads in the wrong spot or vice versa, the discriminator should catch it.
\n\n
Unfortunately, the discriminator learned that the bounding box distributions for the two subsets... and to compensate the GAN stated adding grey boxes around the bounding boxes to fool the discriminators! Pretty cool, but not super useful for the contest.
\n\n
I tried adding a third discriminator that used the whole (unmasked) image but at that point I think the whole thing was over-constrained and it failed to make any progress on the losses.
\n\nI retried the 2 discriminator approach but I stopped training before the GAN started adding the grey boxes. It improved my local validation results but didn't improve my LB score. By then I had spent WAY too much time trying to generate training images and I moved on to other ideas.
\n\nAdditional ideas I didn't get to try:\n- Randomly crop/scale the images so the bounding boxes are less predictable.\n- Use different subsets.\n- Allow the GAN to hallucinate new wheat heads and then re-label the generated images using a trained detector.
\n\nDid anybody else try to generate new training images? How did it go? I'd love to hear about it!
", "rawMarkdown": "Hey everybody, I wanted to share something that didn't end up working out for me, but I think is still pretty interesting.\n\nBack at the beginning of the contest I was trying to generate more training examples using [CycleGAN](https://junyanz.github.io/CycleGAN/). My thinking was that CycleGAN should preserve the location of the wheat heads so I wouldn't need to worry about the bounding boxes.\n\nI tried to transform images from the Arvalis_1 subset to the Ethz_1 subset. I chose those two because they had the most examples. In hindsight, they may not have been the best choices since there seems to be a lot of variation in Arvalis_1 and I'm not sure more images that look like Ethz_1 would be helpful.\n\nIt looked like it was working really well:\n\n\n\nBut unfortunately, when I used fake/generated images into train my object detector, it was slower to converge and performed worse on the validation set. It turns out the GAN was hallucinating wheat textures in brighter areas where they didn't belong and removing the wheat texture in other places. So the bounding boxes weren't correct:\n\n\n\nSo I tried adding a second discriminator and masking out the wheat heads / background. So discriminator 1 only sees wheat heads and discriminator 2 only sees background. That way, if the GAN puts wheat heads in the wrong spot or vice versa, the discriminator should catch it.\n\n\n\nUnfortunately, the discriminator learned that the bounding box distributions for the two subsets... and to compensate the GAN stated adding grey boxes around the bounding boxes to fool the discriminators! Pretty cool, but not super useful for the contest. \n\n\n\n\nI tried adding a third discriminator that used the whole (unmasked) image but at that point I think the whole thing was over-constrained and it failed to make any progress on the losses. \n\nI retried the 2 discriminator approach but I stopped training before the GAN started adding the grey boxes. It improved my local validation results but didn't improve my LB score. By then I had spent WAY too much time trying to generate training images and I moved on to other ideas. \n\n**Additional ideas I didn't get to try:**\n- Randomly crop/scale the images so the bounding boxes are less predictable.\n- Use different subsets.\n- Allow the GAN to hallucinate new wheat heads and then re-label the generated images using a trained detector.\n\n**Did anybody else try to generate new training images? How did it go? I'd love to hear about it!**\n\n\n", "votes": 44 }, { "id": 954071, "postDate": "2020-08-01T11:03:30.223Z", "content": "I have not tried cycle-gan but I used style-transfer and pix2pix GAN to generate images.\nHere are some examples:
\n\nStyle Transfer:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1231059%2F961c77efee13d5d7a74842d7ab51c270%2Fstyles.jpg?generation=1596279696685115&alt=media\")
Pix2Pix: (Not that great)\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1231059%2Fe62db128712e2ea7384e714d99009650%2Fpix2pix.jpg?generation=1596279779235784&alt=media\")
Nice! The style transfer images look great. Did they improve your lb score? For pix2pix, don't you need matching pairs of images? How did you choose the input/output pairs?
", "rawMarkdown": "Nice! The style transfer images look great. Did they improve your lb score? For pix2pix, don't you need matching pairs of images? How did you choose the input/output pairs?", "votes": 1 }, { "id": 954879, "postDate": "2020-08-02T06:59:09.077Z", "content": "Very nice experiments. My approach was to generate the images without labels since I thought it would be an easier task. Despite the low scores we are getting in the competition, the task itself is actually not a hard one so I didn't worry much about labeling, I would let my model do the work. I got some pretty decent results but in the end my GAN was too weak, and not very diverse. A handpicked example: ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1110596%2Ffa03febaae8ec390f2f2615a46f7d0f1%2Fseed8469.png?generation=1596270961097189&alt=media\")
That looks really good! Did you use the full dataset or a subset? What did your setup look like?
\n\nIt looks like we may have run into a similar issue with the GAN creating half-wheat-head-half-leaf creatures. Haha.
", "rawMarkdown": "That looks really good! Did you use the full dataset or a subset? What did your setup look like?\n\nIt looks like we may have run into a similar issue with the GAN creating half-wheat-head-half-leaf creatures. Haha." }, { "id": 957764, "postDate": "2020-08-04T14:49:54.083Z", "content": "I used a manually curated subset, removing ground images and low quality ones like Arvalis_2. By doing this I was able to lower my fid from around 16 to 11. I used Stylegan2 with augmentations, image size of 512, running on 4 1080ti's, I trained until 8 million real images were shown to the discriminator (this is when the quality started to deteriorate). I then used a super-resolution model to upsample the images from 512-1024.
", "rawMarkdown": "I used a manually curated subset, removing ground images and low quality ones like Arvalis_2. By doing this I was able to lower my fid from around 16 to 11. I used Stylegan2 with augmentations, image size of 512, running on 4 1080ti's, I trained until 8 million real images were shown to the discriminator (this is when the quality started to deteriorate). I then used a super-resolution model to upsample the images from 512-1024. \n", "votes": 1 } ] }, { "id": 953886, "postDate": "2020-08-01T08:06:47.040Z", "content": "Very nice work! I also tried out CycleGAN but abandoned it pretty quickly. I'm glad you were able to generate some nice images out of it, though thats unfortunate it didn't boost your score
", "rawMarkdown": "Very nice work! I also tried out CycleGAN but abandoned it pretty quickly. I'm glad you were able to generate some nice images out of it, though thats unfortunate it didn't boost your score", "votes": 1 }, { "id": 958036, "postDate": "2020-08-04T18:00:06.270Z", "content": "The problem of applying GANs as of now is it too computational costly to get that kind of image quality also in order to increase quality we can probably join it with auto-encoders but again too costly .
", "rawMarkdown": "The problem of applying GANs as of now is it too computational costly to get that kind of image quality also in order to increase quality we can probably join it with auto-encoders but again too costly .", "votes": 2 }, { "id": 954169, "postDate": "2020-08-01T13:14:12.597Z", "content": "Great!Can you public your code after this game finish?
", "rawMarkdown": "Great!Can you public your code after this game finish?", "votes": 2 }, { "id": 954199, "postDate": "2020-08-01T13:40:56.693Z", "content": "Interesting results 👍\nSome of the generated images look quite realistic.
", "rawMarkdown": "Interesting results 👍\nSome of the generated images look quite realistic. " }, { "id": 958088, "postDate": "2020-08-04T18:58:51.433Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 954071, "author_name": "bendang", "author_url": "", "post_date": "2020-08-01T11:03:30.223000", "content": "I have not tried cycle-gan but I used style-transfer and pix2pix GAN to generate images.\nHere are some examples:
\n\nStyle Transfer:\n
Pix2Pix: (Not that great)\n
Nice! The style transfer images look great. Did they improve your lb score? For pix2pix, don't you need matching pairs of images? How did you choose the input/output pairs?
", "votes": 1, "replies": [] }, { "id": 954879, "author_name": "bendang", "author_url": "", "post_date": "2020-08-02T06:59:09.077000", "content": "Very nice experiments. My approach was to generate the images without labels since I thought it would be an easier task. Despite the low scores we are getting in the competition, the task itself is actually not a hard one so I didn't worry much about labeling, I would let my model do the work. I got some pretty decent results but in the end my GAN was too weak, and not very diverse. A handpicked example:
That looks really good! Did you use the full dataset or a subset? What did your setup look like?
\n\nIt looks like we may have run into a similar issue with the GAN creating half-wheat-head-half-leaf creatures. Haha.
", "votes": 0, "replies": [] }, { "id": 957764, "author_name": "Max", "author_url": "", "post_date": "2020-08-04T14:49:54.083000", "content": "I used a manually curated subset, removing ground images and low quality ones like Arvalis_2. By doing this I was able to lower my fid from around 16 to 11. I used Stylegan2 with augmentations, image size of 512, running on 4 1080ti's, I trained until 8 million real images were shown to the discriminator (this is when the quality started to deteriorate). I then used a super-resolution model to upsample the images from 512-1024.
", "votes": 1, "replies": [] } ] }, { "id": 953886, "author_name": "Matt", "author_url": "", "post_date": "2020-08-01T08:06:47.040000", "content": "Very nice work! I also tried out CycleGAN but abandoned it pretty quickly. I'm glad you were able to generate some nice images out of it, though thats unfortunate it didn't boost your score
", "votes": 1, "replies": [] }, { "id": 958036, "author_name": "Vishal singh", "author_url": "", "post_date": "2020-08-04T18:00:06.270000", "content": "The problem of applying GANs as of now is it too computational costly to get that kind of image quality also in order to increase quality we can probably join it with auto-encoders but again too costly .
", "votes": 2, "replies": [] }, { "id": 954169, "author_name": "Zekun", "author_url": "", "post_date": "2020-08-01T13:14:12.597000", "content": "Great!Can you public your code after this game finish?
", "votes": 2, "replies": [] }, { "id": 954199, "author_name": "Kuo-Hsin Tu", "author_url": "", "post_date": "2020-08-01T13:40:56.693000", "content": "Interesting results 👍\nSome of the generated images look quite realistic.
", "votes": 0, "replies": [] }, { "id": 958088, "author_name": "", "author_url": "", "post_date": "2020-08-04T18:58:51.433000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "953879": "Hey everybody, I wanted to share something that didn't end up working out for me, but I think is still pretty interesting.\n\nBack at the beginning of the contest I was trying to generate more training examples using [CycleGAN](https://junyanz.github.io/CycleGAN/). My thinking was that CycleGAN should preserve the location of the wheat heads so I wouldn't need to worry about the bounding boxes.\n\nI tried to transform images from the Arvalis_1 subset to the Ethz_1 subset. I chose those two because they had the most examples. In hindsight, they may not have been the best choices since there seems to be a lot of variation in Arvalis_1 and I'm not sure more images that look like Ethz_1 would be helpful.\n\nIt looked like it was working really well:\n\n\n\nBut unfortunately, when I used fake/generated images into train my object detector, it was slower to converge and performed worse on the validation set. It turns out the GAN was hallucinating wheat textures in brighter areas where they didn't belong and removing the wheat texture in other places. So the bounding boxes weren't correct:\n\n\n\nSo I tried adding a second discriminator and masking out the wheat heads / background. So discriminator 1 only sees wheat heads and discriminator 2 only sees background. That way, if the GAN puts wheat heads in the wrong spot or vice versa, the discriminator should catch it.\n\n\n\nUnfortunately, the discriminator learned that the bounding box distributions for the two subsets... and to compensate the GAN stated adding grey boxes around the bounding boxes to fool the discriminators! Pretty cool, but not super useful for the contest. \n\n\n\n\nI tried adding a third discriminator that used the whole (unmasked) image but at that point I think the whole thing was over-constrained and it failed to make any progress on the losses. \n\nI retried the 2 discriminator approach but I stopped training before the GAN started adding the grey boxes. It improved my local validation results but didn't improve my LB score. By then I had spent WAY too much time trying to generate training images and I moved on to other ideas. \n\n**Additional ideas I didn't get to try:**\n- Randomly crop/scale the images so the bounding boxes are less predictable.\n- Use different subsets.\n- Allow the GAN to hallucinate new wheat heads and then re-label the generated images using a trained detector.\n\n**Did anybody else try to generate new training images? How did it go? I'd love to hear about it!**\n\n\n", "954071": "I have not tried cycle-gan but I used style-transfer and pix2pix GAN to generate images.\nHere are some examples:\n\n1. Style Transfer:\n\n\n2. Pix2Pix: (Not that great)\n\n\n ", "953915": "Very nice experiments. My approach was to generate the images without labels since I thought it would be an easier task. Despite the low scores we are getting in the competition, the task itself is actually not a hard one so I didn't worry much about labeling, I would let my model do the work. I got some pretty decent results but in the end my GAN was too weak, and not very diverse. A handpicked example: \n", "953886": "Very nice work! I also tried out CycleGAN but abandoned it pretty quickly. I'm glad you were able to generate some nice images out of it, though thats unfortunate it didn't boost your score", "958036": "The problem of applying GANs as of now is it too computational costly to get that kind of image quality also in order to increase quality we can probably join it with auto-encoders but again too costly .", "954169": "Great!Can you public your code after this game finish?", "954199": "Interesting results 👍\nSome of the generated images look quite realistic. ", "958088": "" } }