![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2601421%2F051843cf1ec024617bea19491f2d3e3d%2FScreenshot%20from%202019-11-15%2010-28-55.png?generation=1573788839142791&alt=media\")
First of all, thanks Kaggle team to have such a new competition and congrats to all the winners ! And special thanks to @cdeotte, @dvorobiev, @jesucristo, who made this competition more interesting. I have learned many things about GANs through this competition.
\n\nTo be honest, I’m very surprised at my final position. I thought I could not be in the gold medal zone, since my public score was not under 30.
\n\nHere is my solution(Public 31.09602, Private 55.42142).
\n\n・Preprocessing&Augmentations\n - exclude images with extreme aspect ratio (y/x < 0.2 and y/x > 4.0)\n - exclude images with intruders (thanks @korovai for sharing the kernel https://www.kaggle.com/korovai/dogs-images-intruders-extraction-tf-gan)\n - use BoundingBox (no modification)\n - Resize 64 (one side) and then RandomCrop to image size (64,64)\n - HorizontalFlip(p=0.5)
\n\n・Model: BigGAN\n - number of parameters G:10M, D:8M\n - input noise from normal distribution (nz=120)\n - use LeakyReLU\n - attention on size 32 feature map \n - use truncated trick (threshold=0.8)\n - no EMA
\n\n・Loss: BCE loss\n・Optimizer: Adam (lrG=3e-4, lrD=3e-4, beta1=0.0, beta2=0.999)\n・Batch Size: 32 \n・Epochs: 130 (maximum kernel time limit, i.e. 32400sec)\n・Others\n - label smoothing 0.9
\n\nAt the beginning of this competition, I played with standard DCGANs, which got Public scores around 55. Then I used AC-GAN and got around 50. With ACGAN-projection, I could get around 39. Then I switched to BigGAN and jumped to around 31. But after that, I struggled to improve my score and could not get under 30. I have checked all of my submitted scores and found that there are some randomness. So I'm lucky to be in the final position. Anyway, I enjoyed this competition very much. Thank you!
\n\nEDIT:\nI made my kernel public https://www.kaggle.com/tikutiku/gan-dogs-starter-biggan\nversion32 is the 1st place solution
", "messages": [ { "id": 611107, "postDate": "2019-08-29T05:30:05.003Z", "content": "First of all, thanks Kaggle team to have such a new competition and congrats to all the winners ! And special thanks to @cdeotte, @dvorobiev, @jesucristo, who made this competition more interesting. I have learned many things about GANs through this competition.
\n\nTo be honest, I’m very surprised at my final position. I thought I could not be in the gold medal zone, since my public score was not under 30.
\n\nHere is my solution(Public 31.09602, Private 55.42142).
\n\n・Preprocessing&Augmentations\n - exclude images with extreme aspect ratio (y/x < 0.2 and y/x > 4.0)\n - exclude images with intruders (thanks @korovai for sharing the kernel https://www.kaggle.com/korovai/dogs-images-intruders-extraction-tf-gan)\n - use BoundingBox (no modification)\n - Resize 64 (one side) and then RandomCrop to image size (64,64)\n - HorizontalFlip(p=0.5)
\n\n・Model: BigGAN\n - number of parameters G:10M, D:8M\n - input noise from normal distribution (nz=120)\n - use LeakyReLU\n - attention on size 32 feature map \n - use truncated trick (threshold=0.8)\n - no EMA
\n\n・Loss: BCE loss\n・Optimizer: Adam (lrG=3e-4, lrD=3e-4, beta1=0.0, beta2=0.999)\n・Batch Size: 32 \n・Epochs: 130 (maximum kernel time limit, i.e. 32400sec)\n・Others\n - label smoothing 0.9
\n\nAt the beginning of this competition, I played with standard DCGANs, which got Public scores around 55. Then I used AC-GAN and got around 50. With ACGAN-projection, I could get around 39. Then I switched to BigGAN and jumped to around 31. But after that, I struggled to improve my score and could not get under 30. I have checked all of my submitted scores and found that there are some randomness. So I'm lucky to be in the final position. Anyway, I enjoyed this competition very much. Thank you!
\n\nEDIT:\nI made my kernel public https://www.kaggle.com/tikutiku/gan-dogs-starter-biggan\nversion32 is the 1st place solution
", "rawMarkdown": "First of all, thanks Kaggle team to have such a new competition and congrats to all the winners ! And special thanks to @cdeotte, @dvorobiev, @jesucristo, who made this competition more interesting. I have learned many things about GANs through this competition. \n\nTo be honest, I’m very surprised at my final position. I thought I could not be in the gold medal zone, since my public score was not under 30.\n\nHere is my solution(Public 31.09602, Private 55.42142).\n\n・Preprocessing&Augmentations\n\t- exclude images with extreme aspect ratio (y/x < 0.2 and y/x > 4.0)\n\t- exclude images with intruders (thanks @korovai for sharing the kernel https://www.kaggle.com/korovai/dogs-images-intruders-extraction-tf-gan)\n\t- use BoundingBox (no modification)\n\t- Resize 64 (one side) and then RandomCrop to image size (64,64)\n\t- HorizontalFlip(p=0.5)\n\n・Model: BigGAN\n\t- number of parameters G:10M, D:8M\n\t- input noise from normal distribution (nz=120)\n\t- use LeakyReLU\n\t- attention on size 32 feature map \n\t- use truncated trick (threshold=0.8)\n\t- no EMA\n\n・Loss: BCE loss\n・Optimizer: Adam (lrG=3e-4, lrD=3e-4, beta1=0.0, beta2=0.999)\n・Batch Size: 32 \n・Epochs: 130 (maximum kernel time limit, i.e. 32400sec)\n・Others\n\t- label smoothing 0.9 \n\t\nAt the beginning of this competition, I played with standard DCGANs, which got Public scores around 55. Then I used AC-GAN and got around 50. With ACGAN-projection, I could get around 39. Then I switched to BigGAN and jumped to around 31. But after that, I struggled to improve my score and could not get under 30. I have checked all of my submitted scores and found that there are some randomness. So I'm lucky to be in the final position. Anyway, I enjoyed this competition very much. Thank you!\n\n-----------\nEDIT:\nI made my kernel public https://www.kaggle.com/tikutiku/gan-dogs-starter-biggan\nversion32 is the 1st place solution", "votes": 55 }, { "id": 613717, "postDate": "2019-08-30T18:57:18.530Z", "content": "Congratulations! Thanks for sharing! Do you think we could improve on this model by experimenting with activations like Elu or Celu?
", "rawMarkdown": "Congratulations! Thanks for sharing! Do you think we could improve on this model by experimenting with activations like Elu or Celu?", "votes": 3, "replies": [ { "id": 613923, "postDate": "2019-08-31T00:20:48.297Z", "content": "Thanks @carlolepelaars ! For me LeakyReLU was slightly better than ReLU, but there were randomness on the score. I didn't try other activations but it's worth trying.
", "rawMarkdown": "Thanks @carlolepelaars ! For me LeakyReLU was slightly better than ReLU, but there were randomness on the score. I didn't try other activations but it's worth trying. " } ] }, { "id": 614371, "postDate": "2019-08-31T12:03:38.230Z", "content": "Congrats. Great work.
", "rawMarkdown": "Congrats. Great work. ", "votes": 2 }, { "id": 633231, "postDate": "2019-09-24T15:20:57.837Z", "content": "Thanks for sharing very interesting work
", "rawMarkdown": "Thanks for sharing very interesting work", "votes": 1 }, { "id": 616935, "postDate": "2019-09-03T15:53:55.497Z", "content": "Thanks for sharing the approach..!!\nVery impressive..!!!
", "rawMarkdown": "Thanks for sharing the approach..!!\nVery impressive..!!!", "votes": 1 }, { "id": 613582, "postDate": "2019-08-30T16:27:23.767Z", "content": "Congrats!! Thanks for sharing your solution.
", "rawMarkdown": "Congrats!! Thanks for sharing your solution.", "votes": 1 }, { "id": 613576, "postDate": "2019-08-30T16:22:42.310Z", "content": "Congratulations !
", "rawMarkdown": "Congratulations !", "votes": 1 }, { "id": 613275, "postDate": "2019-08-30T11:36:00.450Z", "content": "Great job. Impressive work
", "rawMarkdown": "Great job. Impressive work", "votes": 1 }, { "id": 612940, "postDate": "2019-08-30T05:26:11.700Z", "content": "Congrats @tikutiku ! Thanks for sharing your solution, some neat ideas there.
", "rawMarkdown": "Congrats @tikutiku ! Thanks for sharing your solution, some neat ideas there.", "votes": 1 }, { "id": 611822, "postDate": "2019-08-29T13:02:07.390Z", "content": "Congrats @tikutiku \nThank you for sharing your approach.
", "rawMarkdown": "Congrats @tikutiku \nThank you for sharing your approach.", "votes": 1 }, { "id": 611758, "postDate": "2019-08-29T12:20:20.250Z", "content": "Congrats! Interesting settings.. Have you used breed classes for labeling?
", "rawMarkdown": "Congrats! Interesting settings.. Have you used breed classes for labeling?", "votes": 1, "replies": [ { "id": 611970, "postDate": "2019-08-29T14:35:22.567Z", "content": "Thanks @dvorobiev ! In G, I concatenated the breed classes (one-hot-vector) with split z and feed it into the ConditionalNorm in each layers. Also in D, I used the classes for projection (add label embedding to the output before sigmoid activation).
", "rawMarkdown": "Thanks @dvorobiev ! In G, I concatenated the breed classes (one-hot-vector) with split z and feed it into the ConditionalNorm in each layers. Also in D, I used the classes for projection (add label embedding to the output before sigmoid activation). ", "votes": 1 }, { "id": 612499, "postDate": "2019-08-29T20:30:25.030Z", "content": "Thanks. I didn't know label smoothing is applicable in multi-class setups.
", "rawMarkdown": "Thanks. I didn't know label smoothing is applicable in multi-class setups." } ] }, { "id": 611552, "postDate": "2019-08-29T10:16:27.740Z", "content": "Congratulations! You taught me to pre-process my data better.
", "rawMarkdown": "Congratulations! You taught me to pre-process my data better.", "votes": 1 }, { "id": 611347, "postDate": "2019-08-29T08:11:37.750Z", "content": "Great Work... Congrats.... @tikutiku \nMany thanks for sharing.
", "rawMarkdown": "Great Work... Congrats.... @tikutiku \nMany thanks for sharing.", "votes": 1 }, { "id": 611246, "postDate": "2019-08-29T07:23:32.117Z", "content": "Congrats @tikutiku
", "rawMarkdown": "Congrats @tikutiku ", "votes": 1 }, { "id": 611171, "postDate": "2019-08-29T06:31:56.430Z", "content": "Congratz ! I guess the preprocessing you did fit the private dataset quite well. Simple yet effective solution :)
", "rawMarkdown": "Congratz ! I guess the preprocessing you did fit the private dataset quite well. Simple yet effective solution :) ", "votes": 1, "replies": [ { "id": 611184, "postDate": "2019-08-29T06:40:57.577Z", "content": "Thanks @theoviel ! I agree with your guess.
", "rawMarkdown": "Thanks @theoviel ! I agree with your guess.", "votes": 1 } ] }, { "id": 611137, "postDate": "2019-08-29T06:01:26.537Z", "content": "Congrats! @tikutiku , thanks your solution!!\nNice Job!
", "rawMarkdown": "Congrats! @tikutiku , thanks your solution!!\nNice Job!", "votes": 1 }, { "id": 616130, "postDate": "2019-09-02T18:08:11.753Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 2 }, { "id": 615355, "postDate": "2019-09-01T20:05:33.047Z", "content": "congratulations .U did a awesome work
", "rawMarkdown": "congratulations .U did a awesome work", "votes": 2 }, { "id": 611467, "postDate": "2019-08-29T09:22:51.207Z", "content": "@tikutiku , Congrats
\n\nthanks for sharing your approach.
", "rawMarkdown": "@tikutiku , Congrats\n\nthanks for sharing your approach.", "votes": 2 }, { "id": 611136, "postDate": "2019-08-29T06:01:20.440Z", "content": "Congrats! Well-done!
", "rawMarkdown": "Congrats! Well-done!", "votes": 2 }, { "id": 673489, "postDate": "2019-11-15T03:41:58.200Z", "content": "Hello, I would like to ask if you guys encountered a phenomenon similar to the screen shot below, which I call \"light-turn-off effect\" since if those almost-all-gray pictures were gray-replaced-by-black then we'd see black-out effect and actually there might be dogs hiding behind it, just that we don't have enough light to see it.
\n\n
I was wondering why this happened. Was it due to the fact that I made the following modifications?\n[I found the notebook's data loading process posed a bottleneck to the overall speed, so I went to have a look at pytorch's online tutorial, and realized the code version 32 did not fully utilize the parallel data processing ability, so roughly I changed things as below]
\n\nG = nn.DataParallel(G); D = nn.DataParallel(D)
\n```python\nclass DogDataset(Dataset):\n def init(self, path, img_list, transform1=None, transform2=None):\n self.path = path\n self.img_list = img_list\n self.transform1 = transform1\n self.transform2 = transform2
def __len__(self):\n return len(self.img_list)\n\ndef __getitem__(self, idx):\n full_img_path = self.img_list[idx]\n img_path = full_img_path[:-6]+'.jpg'\n bbx_idx = int(full_img_path[-5])\n img = data_preprocessing(img_path, bbx_idx)\n if self.transform1:\n img = self.transform1(img) #output shape=(ch,h,w)\n if self.transform2:\n img = self.transform2(img)\n #label\n label = breed_map_2[img_path.split('_')[0]]\n return {'img':img, 'label':label}\n```
\n\npython\ntrain_set = DogDataset(path=PATH,\n img_list=list(set(img_filenames_th) - set(intruders)),\n transform1=transform1,\n transform2=transform2,\n )\n
```python\ndef run(lr_G=3e-4,lr_D=3e-4, beta1=0.0, beta2=0.999, nz=120, epochs=2, n_ite_G=1,\n n_ite_D=1, ema_decay_rate=0.999, show_epoch_list=None, output_freq=10, continue_on=None, pth_dir=\".\"):\n # Modified code: netG, netD\n G = Generator(n_feat=36, codes_dim=24, n_classes=n_classes)\n D = Discriminator(n_feat=42, n_classes=n_classes)\n netG = G.to(device)\n netD = D.to(device)\n if continue_on:\n try:\n pth_G = os.path.join(pth_dir, \"generator_epoch_{}.pth\".format(continue_on))\n pth_D = os.path.join(pth_dir, \"discriminator_epoch_{}.pth\".format(continue_on))\n from_epoch = continue_on + 1\n to_epoch = max(continue_on + 50, epochs)\n show_epoch_list = np.arange(continue_on + 10, to_epoch + 1, 10)\n if torch.cuda.is_available():\n # GPU-mode load\n netG.load_state_dict(torch.load(pth_G))\n netD.load_state_dict(torch.load(pth_D))\n else:\n # CPU-mode load\n netG.load_state_dict(torch.load(pth_G, map_location=torch.device('cpu')))\n netD.load_state_dict(torch.load(pth_D, map_location=torch.device('cpu')))\n netG.eval()\n netD.eval()\n except:\n print(\"Wrongly specified continue_on/pth_dir when calling run().\")\n print(\"Training from zero.\")\n from_epoch = 1\n to_epoch = epochs\n show_epoch_list = np.arange(10, to_epoch + 1, 10)\n else:\n from_epoch = 1\n to_epoch = epochs\n show_epoch_list = np.arange(10, to_epoch + 1, 10)
\n\n```
", "rawMarkdown": "Hello, I would like to ask if you guys encountered a phenomenon similar to the screen shot below, which I call \"light-turn-off effect\" since if those almost-all-gray pictures were gray-replaced-by-black then we'd see black-out effect and actually there might be dogs hiding behind it, just that we don't have enough light to see it.\n\n\n\nI was wondering why this happened. Was it due to the fact that I made the following modifications?\n[I found the notebook's data loading process posed a bottleneck to the overall speed, so I went to have a look at pytorch's online tutorial, and realized the code version 32 did not fully utilize the parallel data processing ability, so roughly I changed things as below]\n\n`G = nn.DataParallel(G); D = nn.DataParallel(D)`Did you change some hyperparameters? Generally speaking, training of GAN is unstable and sensitive to the hyperparameters. You can check a discussion for the instability in the BigGAN paper.
\n\nI'm not familiar with the behavior of DataParallel, but if it modifies the training procedure then you need to change some hyperparameters so that the training becomes stable.
", "rawMarkdown": "Did you change some hyperparameters? Generally speaking, training of GAN is unstable and sensitive to the hyperparameters. You can check a discussion for the instability in the BigGAN paper.\n\nI'm not familiar with the behavior of DataParallel, but if it modifies the training procedure then you need to change some hyperparameters so that the training becomes stable.", "votes": 1 }, { "id": 673928, "postDate": "2019-11-15T17:02:20.297Z", "content": "No, the only changes I made was those I had included. Because at work I do have access only to a laptop without GPU and its CPU isn't particularly decent, so loading the data in the original code, that is, as a whole in one single take, instead of on the fly, would take several to a dozen of minutes compared to a few milli-seconds after the modification. But thanks, I didn't think of tuning the parameters you had found. I should probably give it a try. Do you have some initial suggestions/directions?
\n\nAh, by the way, later on I decided to explore how to use multiple GPUs in pytorch, so I further added nn.DataParallel and tested it on aws ec2 instances, which costed me a little fortune :P
", "rawMarkdown": "No, the only changes I made was those I had included. Because at work I do have access only to a laptop without GPU and its CPU isn't particularly decent, so loading the data in the original code, that is, as a whole in one single take, instead of on the fly, would take several to a dozen of minutes compared to a few milli-seconds after the modification. But thanks, I didn't think of tuning the parameters you had found. I should probably give it a try. Do you have some initial suggestions/directions?\n\nAh, by the way, later on I decided to explore how to use multiple GPUs in pytorch, so I further added nn.DataParallel and tested it on aws ec2 instances, which costed me a little fortune :P" }, { "id": 674337, "postDate": "2019-11-16T09:48:13.770Z", "content": "For parameter tuning, my suggestion is 1. learning rate, 2. number of features in G and D. \nFor further tuning, trying other loss functions maybe helpful. You can check other solutions shared in this competition.\nBTW, how many epochs do you train ? It was only 9 hours (kaggle kernel's time limit) we could use in this competition. If you train much longer than I did in ver32 (about 130epochs), you may encounter training instability.
", "rawMarkdown": "For parameter tuning, my suggestion is 1. learning rate, 2. number of features in G and D. \nFor further tuning, trying other loss functions maybe helpful. You can check other solutions shared in this competition.\nBTW, how many epochs do you train ? It was only 9 hours (kaggle kernel's time limit) we could use in this competition. If you train much longer than I did in ver32 (about 130epochs), you may encounter training instability.", "votes": 1 }, { "id": 674435, "postDate": "2019-11-16T14:02:43.837Z", "content": "Indeed, I also suspected that it was due to the fact I trained longer (I have trained all over again using your original code to see if similar effect takes place). According to what I observed, obvious beginning of the effect happens around epoch 250-300. I hope tomorrow I will get to know if the original code also encounter such instability at that high epoch number. Thanks for the suggestions (y)
", "rawMarkdown": "Indeed, I also suspected that it was due to the fact I trained longer (I have trained all over again using your original code to see if similar effect takes place). According to what I observed, obvious beginning of the effect happens around epoch 250-300. I hope tomorrow I will get to know if the original code also encounter such instability at that high epoch number. Thanks for the suggestions (y)", "votes": 1 } ] }, { "id": 663486, "postDate": "2019-11-02T06:25:31.933Z", "content": "Hello, Tom. I'd like to ask how would one generate images using your notebook on a dataset without labels (for example, images of a lot of fire extinguishers)?
\n\nAt first, I had tried to get rid of anything related to breed classes, but then I realized that my understanding about BigGAN was too little to allow me to do so in a rapid manner. Then I realized that I could just adjust n_classes to 1, couldn't I?
I read your code and made n_classes = 1, and in class DogDataset: I modified self.labels = [0]*len(self.imgs), but I got an exception saying
```python\n9857593
\n\nRuntimeError: invalid argument 0: Tensors must have same number of dimensions: got 1 and 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:74\n```
", "rawMarkdown": "Hello, Tom. I'd like to ask **how would one generate images using your notebook on a dataset without labels** (for example, images of a lot of fire extinguishers)?\n\nAt first, I had tried to get rid of anything related to breed classes, but then I realized that my understanding about BigGAN was too little to allow me to do so in a rapid manner. Then I realized that I could just adjust `n_classes` to `1`, couldn't I?\n\nI read your code and made `n_classes = 1`, and in `class DogDataset:` I modified `self.labels = [0]*len(self.imgs)`, but I got an exception saying\n\n```python\n9857593\n8417306\n---------------------------------------------------------------------------\nRuntimeError: invalid argument 0: Tensors must have same number of dimensions: got 1 and 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:74\n```", "replies": [ { "id": 663502, "postDate": "2019-11-02T07:37:13.583Z", "content": "When n_classes=1, the squeeze() operation in class Generator causes the error because it deletes all the dimensions with length 1. \nYou need to change \nlabel_ohe = label_ohe.squeeze()\nto\nlabel_ohe = label_ohe.view(-1,1)
Yes, I am reading that part of the code, too. Thank you (y), I am going to try the suggested modification from you right away.
", "rawMarkdown": "Yes, I am reading that part of the code, too. Thank you (y), I am going to try the suggested modification from you right away." }, { "id": 663602, "postDate": "2019-11-02T11:57:23.750Z", "content": "Many thanks @tikutiku \nYour spontaneous answer saved me a lot of work, since I'm very inexperienced in pytorch. I got the code up and running, it's cool. Cheers 💯 :)
", "rawMarkdown": "Many thanks @tikutiku \nYour spontaneous answer saved me a lot of work, since I'm very inexperienced in pytorch. I got the code up and running, it's cool. Cheers 💯 :)", "votes": 1 } ] }, { "id": 617587, "postDate": "2019-09-04T09:24:09.333Z", "content": "I don't understand why you use HorizontalFlip(p=0.5), it must be cancelled by pooling layers right ?
", "rawMarkdown": "I don't understand why you use HorizontalFlip(p=0.5), it must be cancelled by pooling layers right ?" }, { "id": 616881, "postDate": "2019-09-03T14:53:50.517Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 633156, "postDate": "2019-09-24T13:48:52.240Z", "content": "Very impressive, thanks!
", "rawMarkdown": "Very impressive, thanks!", "votes": 1 }, { "id": 617648, "postDate": "2019-09-04T10:50:47.457Z", "content": "Thanks for sharing. Great work!!
", "rawMarkdown": "Thanks for sharing. Great work!!", "votes": 1 }, { "id": 616908, "postDate": "2019-09-03T15:22:25.457Z", "content": "Congrats ! thanks for sharing
", "rawMarkdown": "Congrats ! thanks for sharing", "votes": 1 }, { "id": 611166, "postDate": "2019-08-29T06:28:50.217Z", "content": "Congrats,\nMany thanks for sharing.
", "rawMarkdown": "Congrats,\nMany thanks for sharing.", "votes": 1 }, { "id": 611150, "postDate": "2019-08-29T06:14:28.377Z", "content": "Very insightful, thanks!
", "rawMarkdown": "Very insightful, thanks!", "votes": 1 } ], "comments": [ { "id": 613717, "author_name": "Carlo", "author_url": "", "post_date": "2019-08-30T18:57:18.530000", "content": "Congratulations! Thanks for sharing! Do you think we could improve on this model by experimenting with activations like Elu or Celu?
", "votes": 3, "replies": [ { "id": 613923, "author_name": "Tom", "author_url": "", "post_date": "2019-08-31T00:20:48.297000", "content": "Thanks @carlolepelaars ! For me LeakyReLU was slightly better than ReLU, but there were randomness on the score. I didn't try other activations but it's worth trying.
", "votes": 0, "replies": [] } ] }, { "id": 614371, "author_name": "Manoj Prabhakar", "author_url": "", "post_date": "2019-08-31T12:03:38.230000", "content": "Congrats. Great work.
", "votes": 2, "replies": [] }, { "id": 633231, "author_name": "Anthony Wynne", "author_url": "", "post_date": "2019-09-24T15:20:57.837000", "content": "Thanks for sharing very interesting work
", "votes": 1, "replies": [] }, { "id": 616935, "author_name": "Saikat Biswas", "author_url": "", "post_date": "2019-09-03T15:53:55.497000", "content": "Thanks for sharing the approach..!!\nVery impressive..!!!
", "votes": 1, "replies": [] }, { "id": 613582, "author_name": "dhaqui the kaggler", "author_url": "", "post_date": "2019-08-30T16:27:23.767000", "content": "Congrats!! Thanks for sharing your solution.
", "votes": 1, "replies": [] }, { "id": 613576, "author_name": "Utku_Kubilay", "author_url": "", "post_date": "2019-08-30T16:22:42.310000", "content": "Congratulations !
", "votes": 1, "replies": [] }, { "id": 613275, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-08-30T11:36:00.450000", "content": "Great job. Impressive work
", "votes": 1, "replies": [] }, { "id": 612940, "author_name": "Vopani", "author_url": "", "post_date": "2019-08-30T05:26:11.700000", "content": "Congrats @tikutiku ! Thanks for sharing your solution, some neat ideas there.
", "votes": 1, "replies": [] }, { "id": 611822, "author_name": "Joel Hanson", "author_url": "", "post_date": "2019-08-29T13:02:07.390000", "content": "Congrats @tikutiku \nThank you for sharing your approach.
", "votes": 1, "replies": [] }, { "id": 611758, "author_name": "Dmitry Vorobiev", "author_url": "", "post_date": "2019-08-29T12:20:20.250000", "content": "Congrats! Interesting settings.. Have you used breed classes for labeling?
", "votes": 1, "replies": [ { "id": 611970, "author_name": "Tom", "author_url": "", "post_date": "2019-08-29T14:35:22.567000", "content": "Thanks @dvorobiev ! In G, I concatenated the breed classes (one-hot-vector) with split z and feed it into the ConditionalNorm in each layers. Also in D, I used the classes for projection (add label embedding to the output before sigmoid activation).
", "votes": 1, "replies": [] }, { "id": 612499, "author_name": "Dmitry Vorobiev", "author_url": "", "post_date": "2019-08-29T20:30:25.030000", "content": "Thanks. I didn't know label smoothing is applicable in multi-class setups.
", "votes": 0, "replies": [] } ] }, { "id": 611552, "author_name": "timetraveller", "author_url": "", "post_date": "2019-08-29T10:16:27.740000", "content": "Congratulations! You taught me to pre-process my data better.
", "votes": 1, "replies": [] }, { "id": 611347, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-08-29T08:11:37.750000", "content": "Great Work... Congrats.... @tikutiku \nMany thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 611246, "author_name": "Akash Ravichandran", "author_url": "", "post_date": "2019-08-29T07:23:32.117000", "content": "Congrats @tikutiku
", "votes": 1, "replies": [] }, { "id": 611171, "author_name": "Theo Viel", "author_url": "", "post_date": "2019-08-29T06:31:56.430000", "content": "Congratz ! I guess the preprocessing you did fit the private dataset quite well. Simple yet effective solution :)
", "votes": 1, "replies": [ { "id": 611184, "author_name": "Tom", "author_url": "", "post_date": "2019-08-29T06:40:57.577000", "content": "Thanks @theoviel ! I agree with your guess.
", "votes": 1, "replies": [] } ] }, { "id": 611137, "author_name": "0xFunky", "author_url": "", "post_date": "2019-08-29T06:01:26.537000", "content": "Congrats! @tikutiku , thanks your solution!!\nNice Job!
", "votes": 1, "replies": [] }, { "id": 616130, "author_name": "Daniel Li", "author_url": "", "post_date": "2019-09-02T18:08:11.753000", "content": "Congrats!
", "votes": 2, "replies": [] }, { "id": 615355, "author_name": "Bhanu_Chennam", "author_url": "", "post_date": "2019-09-01T20:05:33.047000", "content": "congratulations .U did a awesome work
", "votes": 2, "replies": [] }, { "id": 611467, "author_name": "Prashant Banerjee", "author_url": "", "post_date": "2019-08-29T09:22:51.207000", "content": "@tikutiku , Congrats
\n\nthanks for sharing your approach.
", "votes": 2, "replies": [] }, { "id": 611136, "author_name": "Mark Peng", "author_url": "", "post_date": "2019-08-29T06:01:20.440000", "content": "Congrats! Well-done!
", "votes": 2, "replies": [] }, { "id": 673489, "author_name": "phunc20", "author_url": "", "post_date": "2019-11-15T03:41:58.200000", "content": "Hello, I would like to ask if you guys encountered a phenomenon similar to the screen shot below, which I call \"light-turn-off effect\" since if those almost-all-gray pictures were gray-replaced-by-black then we'd see black-out effect and actually there might be dogs hiding behind it, just that we don't have enough light to see it.
\n\n
I was wondering why this happened. Was it due to the fact that I made the following modifications?\n[I found the notebook's data loading process posed a bottleneck to the overall speed, so I went to have a look at pytorch's online tutorial, and realized the code version 32 did not fully utilize the parallel data processing ability, so roughly I changed things as below]
\n\nG = nn.DataParallel(G); D = nn.DataParallel(D)
\n```python\nclass DogDataset(Dataset):\n def init(self, path, img_list, transform1=None, transform2=None):\n self.path = path\n self.img_list = img_list\n self.transform1 = transform1\n self.transform2 = transform2
def __len__(self):\n return len(self.img_list)\n\ndef __getitem__(self, idx):\n full_img_path = self.img_list[idx]\n img_path = full_img_path[:-6]+'.jpg'\n bbx_idx = int(full_img_path[-5])\n img = data_preprocessing(img_path, bbx_idx)\n if self.transform1:\n img = self.transform1(img) #output shape=(ch,h,w)\n if self.transform2:\n img = self.transform2(img)\n #label\n label = breed_map_2[img_path.split('_')[0]]\n return {'img':img, 'label':label}\n```
\n\npython\ntrain_set = DogDataset(path=PATH,\n img_list=list(set(img_filenames_th) - set(intruders)),\n transform1=transform1,\n transform2=transform2,\n )\n
```python\ndef run(lr_G=3e-4,lr_D=3e-4, beta1=0.0, beta2=0.999, nz=120, epochs=2, n_ite_G=1,\n n_ite_D=1, ema_decay_rate=0.999, show_epoch_list=None, output_freq=10, continue_on=None, pth_dir=\".\"):\n # Modified code: netG, netD\n G = Generator(n_feat=36, codes_dim=24, n_classes=n_classes)\n D = Discriminator(n_feat=42, n_classes=n_classes)\n netG = G.to(device)\n netD = D.to(device)\n if continue_on:\n try:\n pth_G = os.path.join(pth_dir, \"generator_epoch_{}.pth\".format(continue_on))\n pth_D = os.path.join(pth_dir, \"discriminator_epoch_{}.pth\".format(continue_on))\n from_epoch = continue_on + 1\n to_epoch = max(continue_on + 50, epochs)\n show_epoch_list = np.arange(continue_on + 10, to_epoch + 1, 10)\n if torch.cuda.is_available():\n # GPU-mode load\n netG.load_state_dict(torch.load(pth_G))\n netD.load_state_dict(torch.load(pth_D))\n else:\n # CPU-mode load\n netG.load_state_dict(torch.load(pth_G, map_location=torch.device('cpu')))\n netD.load_state_dict(torch.load(pth_D, map_location=torch.device('cpu')))\n netG.eval()\n netD.eval()\n except:\n print(\"Wrongly specified continue_on/pth_dir when calling run().\")\n print(\"Training from zero.\")\n from_epoch = 1\n to_epoch = epochs\n show_epoch_list = np.arange(10, to_epoch + 1, 10)\n else:\n from_epoch = 1\n to_epoch = epochs\n show_epoch_list = np.arange(10, to_epoch + 1, 10)
\n\n```
", "votes": 0, "replies": [ { "id": 673830, "author_name": "Tom", "author_url": "", "post_date": "2019-11-15T14:35:59.943000", "content": "Did you change some hyperparameters? Generally speaking, training of GAN is unstable and sensitive to the hyperparameters. You can check a discussion for the instability in the BigGAN paper.
\n\nI'm not familiar with the behavior of DataParallel, but if it modifies the training procedure then you need to change some hyperparameters so that the training becomes stable.
", "votes": 1, "replies": [] }, { "id": 673928, "author_name": "phunc20", "author_url": "", "post_date": "2019-11-15T17:02:20.297000", "content": "No, the only changes I made was those I had included. Because at work I do have access only to a laptop without GPU and its CPU isn't particularly decent, so loading the data in the original code, that is, as a whole in one single take, instead of on the fly, would take several to a dozen of minutes compared to a few milli-seconds after the modification. But thanks, I didn't think of tuning the parameters you had found. I should probably give it a try. Do you have some initial suggestions/directions?
\n\nAh, by the way, later on I decided to explore how to use multiple GPUs in pytorch, so I further added nn.DataParallel and tested it on aws ec2 instances, which costed me a little fortune :P
", "votes": 0, "replies": [] }, { "id": 674337, "author_name": "Tom", "author_url": "", "post_date": "2019-11-16T09:48:13.770000", "content": "For parameter tuning, my suggestion is 1. learning rate, 2. number of features in G and D. \nFor further tuning, trying other loss functions maybe helpful. You can check other solutions shared in this competition.\nBTW, how many epochs do you train ? It was only 9 hours (kaggle kernel's time limit) we could use in this competition. If you train much longer than I did in ver32 (about 130epochs), you may encounter training instability.
", "votes": 1, "replies": [] }, { "id": 674435, "author_name": "phunc20", "author_url": "", "post_date": "2019-11-16T14:02:43.837000", "content": "Indeed, I also suspected that it was due to the fact I trained longer (I have trained all over again using your original code to see if similar effect takes place). According to what I observed, obvious beginning of the effect happens around epoch 250-300. I hope tomorrow I will get to know if the original code also encounter such instability at that high epoch number. Thanks for the suggestions (y)
", "votes": 1, "replies": [] } ] }, { "id": 663486, "author_name": "phunc20", "author_url": "", "post_date": "2019-11-02T06:25:31.933000", "content": "Hello, Tom. I'd like to ask how would one generate images using your notebook on a dataset without labels (for example, images of a lot of fire extinguishers)?
\n\nAt first, I had tried to get rid of anything related to breed classes, but then I realized that my understanding about BigGAN was too little to allow me to do so in a rapid manner. Then I realized that I could just adjust n_classes to 1, couldn't I?
I read your code and made n_classes = 1, and in class DogDataset: I modified self.labels = [0]*len(self.imgs), but I got an exception saying
```python\n9857593
\n\nRuntimeError: invalid argument 0: Tensors must have same number of dimensions: got 1 and 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:74\n```
", "votes": 0, "replies": [ { "id": 663502, "author_name": "Tom", "author_url": "", "post_date": "2019-11-02T07:37:13.583000", "content": "When n_classes=1, the squeeze() operation in class Generator causes the error because it deletes all the dimensions with length 1. \nYou need to change \nlabel_ohe = label_ohe.squeeze()\nto\nlabel_ohe = label_ohe.view(-1,1)
Yes, I am reading that part of the code, too. Thank you (y), I am going to try the suggested modification from you right away.
", "votes": 0, "replies": [] }, { "id": 663602, "author_name": "phunc20", "author_url": "", "post_date": "2019-11-02T11:57:23.750000", "content": "Many thanks @tikutiku \nYour spontaneous answer saved me a lot of work, since I'm very inexperienced in pytorch. I got the code up and running, it's cool. Cheers 💯 :)
", "votes": 1, "replies": [] } ] }, { "id": 617587, "author_name": "Sylvain L", "author_url": "", "post_date": "2019-09-04T09:24:09.333000", "content": "I don't understand why you use HorizontalFlip(p=0.5), it must be cancelled by pooling layers right ?
", "votes": 0, "replies": [] }, { "id": 616881, "author_name": "", "author_url": "", "post_date": "2019-09-03T14:53:50.517000", "content": "", "votes": 1, "replies": [] }, { "id": 633156, "author_name": "Lance Zhang", "author_url": "", "post_date": "2019-09-24T13:48:52.240000", "content": "Very impressive, thanks!
", "votes": 1, "replies": [] }, { "id": 617648, "author_name": "Bsilva", "author_url": "", "post_date": "2019-09-04T10:50:47.457000", "content": "Thanks for sharing. Great work!!
", "votes": 1, "replies": [] }, { "id": 616908, "author_name": "Wael Dahech", "author_url": "", "post_date": "2019-09-03T15:22:25.457000", "content": "Congrats ! thanks for sharing
", "votes": 1, "replies": [] }, { "id": 611166, "author_name": "Eric Vos", "author_url": "", "post_date": "2019-08-29T06:28:50.217000", "content": "Congrats,\nMany thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 611150, "author_name": "Shay Guterman", "author_url": "", "post_date": "2019-08-29T06:14:28.377000", "content": "Very insightful, thanks!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "611107": "First of all, thanks Kaggle team to have such a new competition and congrats to all the winners ! And special thanks to @cdeotte, @dvorobiev, @jesucristo, who made this competition more interesting. I have learned many things about GANs through this competition. \n\nTo be honest, I’m very surprised at my final position. I thought I could not be in the gold medal zone, since my public score was not under 30.\n\nHere is my solution(Public 31.09602, Private 55.42142).\n\n・Preprocessing&Augmentations\n\t- exclude images with extreme aspect ratio (y/x < 0.2 and y/x > 4.0)\n\t- exclude images with intruders (thanks @korovai for sharing the kernel https://www.kaggle.com/korovai/dogs-images-intruders-extraction-tf-gan)\n\t- use BoundingBox (no modification)\n\t- Resize 64 (one side) and then RandomCrop to image size (64,64)\n\t- HorizontalFlip(p=0.5)\n\n・Model: BigGAN\n\t- number of parameters G:10M, D:8M\n\t- input noise from normal distribution (nz=120)\n\t- use LeakyReLU\n\t- attention on size 32 feature map \n\t- use truncated trick (threshold=0.8)\n\t- no EMA\n\n・Loss: BCE loss\n・Optimizer: Adam (lrG=3e-4, lrD=3e-4, beta1=0.0, beta2=0.999)\n・Batch Size: 32 \n・Epochs: 130 (maximum kernel time limit, i.e. 32400sec)\n・Others\n\t- label smoothing 0.9 \n\t\nAt the beginning of this competition, I played with standard DCGANs, which got Public scores around 55. Then I used AC-GAN and got around 50. With ACGAN-projection, I could get around 39. Then I switched to BigGAN and jumped to around 31. But after that, I struggled to improve my score and could not get under 30. I have checked all of my submitted scores and found that there are some randomness. So I'm lucky to be in the final position. Anyway, I enjoyed this competition very much. Thank you!\n\n-----------\nEDIT:\nI made my kernel public https://www.kaggle.com/tikutiku/gan-dogs-starter-biggan\nversion32 is the 1st place solution", "613717": "Congratulations! Thanks for sharing! Do you think we could improve on this model by experimenting with activations like Elu or Celu?", "614371": "Congrats. Great work. ", "633231": "Thanks for sharing very interesting work", "616935": "Thanks for sharing the approach..!!\nVery impressive..!!!", "613582": "Congrats!! Thanks for sharing your solution.", "613576": "Congratulations !", "613275": "Great job. Impressive work", "612940": "Congrats @tikutiku ! Thanks for sharing your solution, some neat ideas there.", "611822": "Congrats @tikutiku \nThank you for sharing your approach.", "611758": "Congrats! Interesting settings.. Have you used breed classes for labeling?", "611552": "Congratulations! You taught me to pre-process my data better.", "611347": "Great Work... Congrats.... @tikutiku \nMany thanks for sharing.", "611246": "Congrats @tikutiku ", "611171": "Congratz ! I guess the preprocessing you did fit the private dataset quite well. Simple yet effective solution :) ", "611137": "Congrats! @tikutiku , thanks your solution!!\nNice Job!", "616130": "Congrats!", "615355": "congratulations .U did a awesome work", "611467": "@tikutiku , Congrats\n\nthanks for sharing your approach.", "611136": "Congrats! Well-done!", "673489": "Hello, I would like to ask if you guys encountered a phenomenon similar to the screen shot below, which I call \"light-turn-off effect\" since if those almost-all-gray pictures were gray-replaced-by-black then we'd see black-out effect and actually there might be dogs hiding behind it, just that we don't have enough light to see it.\n\n\n\nI was wondering why this happened. Was it due to the fact that I made the following modifications?\n[I found the notebook's data loading process posed a bottleneck to the overall speed, so I went to have a look at pytorch's online tutorial, and realized the code version 32 did not fully utilize the parallel data processing ability, so roughly I changed things as below]\n\n`G = nn.DataParallel(G); D = nn.DataParallel(D)`