![https://raw.githubusercontent.com/bestfitting/kaggle/master/gandogs/samples/samples_mode8.jpg](\"https://raw.githubusercontent.com/bestfitting/kaggle/master/gandogs/samples/samples_mode8.jpg\"){kind=link}
![\"image\"](\"http://playagricola.com/Kaggle/dogs1b81419.png\")
It's an interesting competition, and this is the first time I write a solution without knowing private LB score. Great thanks to kaggle for the efforts to host this competition and quick response to the community by @juliaelliott, @wendykan.
\n\nMy solution may be described as a mix of StyleGAN and BigGAN.
\n\nRespect to the authors of the papers and these two github repositories:\nhttps://github.com/ajbrock/BigGAN-PyTorch\nhttps://github.com/rosinality/style-based-gan-pytorch
\n\nChallenges:\n1. Limited resources and limited time to train a GAN from scratch.\n2. A new kind of challenge with new rules and settings. It took me 2 weeks to choose between this one and the Youbute 8M video challenge.
\n\nSolution:
\n\nI tried DCGAN and ACGAN to get familiar with this competition at first , and then select papers related, I selected about 40 papers which might be useful.
\n\nI was impressed by the results of the StyleGAN and tried to make it work more efficiently so I could make it run in a kernel, but I found Progressive GAN and its followers were inefficient to train, but I still spent about 10 days to tune it and got FID about 35 , I guessed that this score was not enough to top 10, but I thought I should spend more time on it to get understanding of every detail of it before I stepped forward.
\n\nThen I tried VQ-VAE2 and a lot of other mechanisms and I was not satisfied with the efficiency and the scores.
\n\nBigGAN was listed on the top of my most promising solutions, but the BIG, DEEP in the model name prevent me from trying it in a hurry and the paper stated that using more channels was important, I worried that it might be inefficient in this competition, so I planed to try it after I have tested others. To my surprise, the channels was not so important, when I decreased the G_ch and D_ch from 64 to 24, the FID score did not decrease too much and the FID score can easily reach below 30.
\n\nWhat's more, the project structure of the https://github.com/ajbrock/BigGAN-PyTorch is very good, we can do a lot of experiments by just modifying some parameters, so I could start many experiments and went out have a bicycle driving or swimming :) After several experiments, my score reached about 24.
\n\nAfter I checked and compared the architecture and mechanisms of the Big and Style GAN, I found that Spectral normalization and Class-conditional Batch normalization were important to BigAN, so we need not use progressive training, and BigGAN's G_shared and skip_z mechanisms were similar to the StyleGAN's style mechanism to some extend, so I made a lot of modifications to BigGAN and tried to move almost all important mechanisms of StyleGAN to it. After many experiments, I found: Style mechanism could improve the score significantly and coverage more quickly, the Blur was also important, Noise was not so important, to save time, I did not add it to my networks. Since I had more efficient and powerful tools for the BigGAN, so I did not use skip_z. I found mix-style mechanism of StyleGAN was not important in this competition, so I did not use it in my final submission. After these efforts and some hyper-parameters tuning, the score can reach about 17.
\n\nI thought I might use large image size to train the network, so I designed a 96x96 model instead of 64x64, I found it could improve the score.
\n\nI tried to use Attention including BigGAN's attention and CBAM in the network, it increased the training time and the improvement to the score was not so big, so I did not use them
\n\nTo my surprise, although the paper of BigGAN stated that the larger batch-size, the better result, I found 32 was the best batch-size. So the two most important results--large channels and bigger batch-size of the BigGAN were not suitable of this dataset, so my solution should be called Style-Small-GAN ?? :smile::smile:
\n\nMy best parameters can be described as the following command:
\n\npython train.py --shuffle --batch_size 32 --num_G_accumulations 1 --num_D_accumulations 1 --num_epochs 200 --num_D_steps 1 --G_lr 1e-4 --D_lr 6e-4 --dataset DogOrigin96 --bottom_width 6 --G_ortho 0.0 --G_attn 0 --D_attn 0 --G_init ortho --D_init ortho --ema --use_ema --ema_start 2000 --test_every 25 --save_every 10 --num_best_copies 5 --num_save_copies 2 --G_ch 24 --D_ch 24 --seed 0 --augment 1 --add_blur --add_style --crop_mode 8 --on_kaggle --experiment_name i96_ch24_hinge_ema_dstep1_bs32_noatt_glr0001_glr0006_aug_init_ortho_blur_style_origin_crop_mode8\n\n\n
The score improved to 12.4 on public LB after these steps
\n\nIn the last several days of this competition, I turned to the model's generalization and the data itself, I found that if I cropped square box containing the dogs only, it performed better than adding resized origin images, but, as kaggle will use another datasets and my models were trained on all-dogs datasets, they might over-fit to the cropped all-dogs to some extend, when my models were applied on other images, the dogs might not be cropped, so I added resized origin images to the datasets with some probability. The score on public LB of this augmentation is 13.4, but I guess this model may perform better than 12.4 one.
\n\nI also added all kinds of mechanisms , such as mode seeking, all kinds of loss functions such as such as GDPP, normalizations, regularizations , initializations, and most of them did not work.
\n\nMost of mechanism I talked above are kept in the kernels and can be tested by using the parameters. If you want to have a try, 100 epochs in 4.5h can reach FID 18.x, and 125 epochs can reach FID 15.x
\n\nWhat can we do to improve the model if there are no limits on the resources and time?\n1. I designed a mechanism to add dog count in an image to the model, that's to say, dog count embedding. It works, but it took more time. \n2. We can still cluster the images to 30-40 clusters, use these cluster id as other embedding, so, the dog on the grass, floor..... ( I did not test it )\n3. We can use large sized images.
\n\nSo the FID less than 10 is not so difficult to achieve if we have two GPUs or double time. \n1 and 2 can be described as Multi-Classes-Conditional Style GAN, with the mixture of the Style and BigGAN, after some well designed experiments we may make some progress to the state of the art GANs?\nYou can check some of images, a type of dogs per line:\nhttps://raw.githubusercontent.com/bestfitting/kaggle/master/gandogs/samples/samples_mode8.jpg
\n\nThe link to my kernels:\nhttps://www.kaggle.com/bestfitting/i96-1e-4-6e-4-bs32-195\nhttps://www.kaggle.com/bestfitting/i96-crop-mode3-dstep1-1e-4-6e-4-bs32
\n\nUpdated\nThe source codes in the kennels above are not easy to read, you can check the normal python project structure version here https://github.com/bestfitting/kaggle/tree/master/gandogs
\n\nPlease note that the progresses and the scores of my models are not a formal ablation study in a paper and I don't plan to write a paper either, I modified the parameters and other setting during these progresses, they were results of a series of complex experiment setting.
", "messages": [ { "id": "600170", "postDate": "08/15/2019 18:40:24", "content": "It's an interesting competition, and this is the first time I write a solution without knowing private LB score. Great thanks to kaggle for the efforts to host this competition and quick response to the community by @juliaelliott, @wendykan.
\n\nMy solution may be described as a mix of StyleGAN and BigGAN.
\n\nRespect to the authors of the papers and these two github repositories:\nhttps://github.com/ajbrock/BigGAN-PyTorch\nhttps://github.com/rosinality/style-based-gan-pytorch
\n\nChallenges:\n1. Limited resources and limited time to train a GAN from scratch.\n2. A new kind of challenge with new rules and settings. It took me 2 weeks to choose between this one and the Youbute 8M video challenge.
\n\nSolution:
\n\nI tried DCGAN and ACGAN to get familiar with this competition at first , and then select papers related, I selected about 40 papers which might be useful.
\n\nI was impressed by the results of the StyleGAN and tried to make it work more efficiently so I could make it run in a kernel, but I found Progressive GAN and its followers were inefficient to train, but I still spent about 10 days to tune it and got FID about 35 , I guessed that this score was not enough to top 10, but I thought I should spend more time on it to get understanding of every detail of it before I stepped forward.
\n\nThen I tried VQ-VAE2 and a lot of other mechanisms and I was not satisfied with the efficiency and the scores.
\n\nBigGAN was listed on the top of my most promising solutions, but the BIG, DEEP in the model name prevent me from trying it in a hurry and the paper stated that using more channels was important, I worried that it might be inefficient in this competition, so I planed to try it after I have tested others. To my surprise, the channels was not so important, when I decreased the G_ch and D_ch from 64 to 24, the FID score did not decrease too much and the FID score can easily reach below 30.
\n\nWhat's more, the project structure of the https://github.com/ajbrock/BigGAN-PyTorch is very good, we can do a lot of experiments by just modifying some parameters, so I could start many experiments and went out have a bicycle driving or swimming :) After several experiments, my score reached about 24.
\n\nAfter I checked and compared the architecture and mechanisms of the Big and Style GAN, I found that Spectral normalization and Class-conditional Batch normalization were important to BigAN, so we need not use progressive training, and BigGAN's G_shared and skip_z mechanisms were similar to the StyleGAN's style mechanism to some extend, so I made a lot of modifications to BigGAN and tried to move almost all important mechanisms of StyleGAN to it. After many experiments, I found: Style mechanism could improve the score significantly and coverage more quickly, the Blur was also important, Noise was not so important, to save time, I did not add it to my networks. Since I had more efficient and powerful tools for the BigGAN, so I did not use skip_z. I found mix-style mechanism of StyleGAN was not important in this competition, so I did not use it in my final submission. After these efforts and some hyper-parameters tuning, the score can reach about 17.
\n\nI thought I might use large image size to train the network, so I designed a 96x96 model instead of 64x64, I found it could improve the score.
\n\nI tried to use Attention including BigGAN's attention and CBAM in the network, it increased the training time and the improvement to the score was not so big, so I did not use them
\n\nTo my surprise, although the paper of BigGAN stated that the larger batch-size, the better result, I found 32 was the best batch-size. So the two most important results--large channels and bigger batch-size of the BigGAN were not suitable of this dataset, so my solution should be called Style-Small-GAN ?? :smile::smile:
\n\nMy best parameters can be described as the following command:
\n\npython train.py --shuffle --batch_size 32 --num_G_accumulations 1 --num_D_accumulations 1 --num_epochs 200 --num_D_steps 1 --G_lr 1e-4 --D_lr 6e-4 --dataset DogOrigin96 --bottom_width 6 --G_ortho 0.0 --G_attn 0 --D_attn 0 --G_init ortho --D_init ortho --ema --use_ema --ema_start 2000 --test_every 25 --save_every 10 --num_best_copies 5 --num_save_copies 2 --G_ch 24 --D_ch 24 --seed 0 --augment 1 --add_blur --add_style --crop_mode 8 --on_kaggle --experiment_name i96_ch24_hinge_ema_dstep1_bs32_noatt_glr0001_glr0006_aug_init_ortho_blur_style_origin_crop_mode8\n\n\n
The score improved to 12.4 on public LB after these steps
\n\nIn the last several days of this competition, I turned to the model's generalization and the data itself, I found that if I cropped square box containing the dogs only, it performed better than adding resized origin images, but, as kaggle will use another datasets and my models were trained on all-dogs datasets, they might over-fit to the cropped all-dogs to some extend, when my models were applied on other images, the dogs might not be cropped, so I added resized origin images to the datasets with some probability. The score on public LB of this augmentation is 13.4, but I guess this model may perform better than 12.4 one.
\n\nI also added all kinds of mechanisms , such as mode seeking, all kinds of loss functions such as such as GDPP, normalizations, regularizations , initializations, and most of them did not work.
\n\nMost of mechanism I talked above are kept in the kernels and can be tested by using the parameters. If you want to have a try, 100 epochs in 4.5h can reach FID 18.x, and 125 epochs can reach FID 15.x
\n\nWhat can we do to improve the model if there are no limits on the resources and time?\n1. I designed a mechanism to add dog count in an image to the model, that's to say, dog count embedding. It works, but it took more time. \n2. We can still cluster the images to 30-40 clusters, use these cluster id as other embedding, so, the dog on the grass, floor..... ( I did not test it )\n3. We can use large sized images.
\n\nSo the FID less than 10 is not so difficult to achieve if we have two GPUs or double time. \n1 and 2 can be described as Multi-Classes-Conditional Style GAN, with the mixture of the Style and BigGAN, after some well designed experiments we may make some progress to the state of the art GANs?\nYou can check some of images, a type of dogs per line:\nhttps://raw.githubusercontent.com/bestfitting/kaggle/master/gandogs/samples/samples_mode8.jpg
\n\nThe link to my kernels:\nhttps://www.kaggle.com/bestfitting/i96-1e-4-6e-4-bs32-195\nhttps://www.kaggle.com/bestfitting/i96-crop-mode3-dstep1-1e-4-6e-4-bs32
\n\nUpdated\nThe source codes in the kennels above are not easy to read, you can check the normal python project structure version here https://github.com/bestfitting/kaggle/tree/master/gandogs
\n\nPlease note that the progresses and the scores of my models are not a formal ablation study in a paper and I don't plan to write a paper either, I modified the parameters and other setting during these progresses, they were results of a series of complex experiment setting.
", "rawMarkdown": "It's an interesting competition, and this is the first time I write a solution without knowing private LB score. Great thanks to kaggle for the efforts to host this competition and quick response to the community by @juliaelliott, @wendykan. \n\nMy solution may be described as a mix of StyleGAN and BigGAN.\n\nRespect to the authors of the papers and these two github repositories:\nhttps://github.com/ajbrock/BigGAN-PyTorch\nhttps://github.com/rosinality/style-based-gan-pytorch\n\n\n**Challenges**:\n1. Limited resources and limited time to train a GAN from scratch.\n2. A new kind of challenge with new rules and settings. It took me 2 weeks to choose between this one and the Youbute 8M video challenge.\n\n**Solution**:\n\nI tried DCGAN and ACGAN to get familiar with this competition at first , and then select papers related, I selected about 40 papers which might be useful.\n\nI was impressed by the results of the StyleGAN and tried to make it work more efficiently so I could make it run in a kernel, but I found Progressive GAN and its followers were inefficient to train, but I still spent about 10 days to tune it and got FID about 35 , I guessed that this score was not enough to top 10, but I thought I should spend more time on it to get understanding of every detail of it before I stepped forward.\n\nThen I tried VQ-VAE2 and a lot of other mechanisms and I was not satisfied with the efficiency and the scores.\n\nBigGAN was listed on the top of my most promising solutions, but the **BIG**, **DEEP** in the model name prevent me from trying it in a hurry and the paper stated that using more channels was important, I worried that it might be inefficient in this competition, so I planed to try it after I have tested others. To my surprise, the channels was not so important, when I decreased the G\\_ch and D\\_ch from 64 to 24, the FID score did not decrease too much and the FID score can easily reach below 30.\n\nWhat's more, the project structure of the https://github.com/ajbrock/BigGAN-PyTorch is very good, we can do a lot of experiments by just modifying some parameters, so I could start many experiments and went out have a bicycle driving or swimming :) After several experiments, my score reached about 24. \n\nAfter I checked and compared the architecture and mechanisms of the Big and Style GAN, I found that Spectral normalization and Class-conditional Batch normalization were important to BigAN, so we need not use progressive training, and BigGAN's G_shared and skip\\_z mechanisms were similar to the StyleGAN's style mechanism to some extend, so I made a lot of modifications to BigGAN and tried to move almost all important mechanisms of StyleGAN to it. After many experiments, I found: Style mechanism could improve the score significantly and coverage more quickly, the Blur was also important, Noise was not so important, to save time, I did not add it to my networks. Since I had more efficient and powerful tools for the BigGAN, so I did not use skip\\_z. I found mix-style mechanism of StyleGAN was not important in this competition, so I did not use it in my final submission. After these efforts and some hyper-parameters tuning, the score can reach about 17. \n\nI thought I might use large image size to train the network, so I designed a 96x96 model instead of 64x64, I found it could improve the score.\n\nI tried to use Attention including BigGAN's attention and CBAM in the network, it increased the training time and the improvement to the score was not so big, so I did not use them\n\nTo my surprise, although the paper of BigGAN stated that the larger batch-size, the better result, I found 32 was the best batch-size. So the two most important results--large channels and bigger batch-size of the BigGAN were not suitable of this dataset, so my solution should be called **Style-Small-GAN** ?? :smile::smile:\n\nMy best parameters can be described as the following command:\npython train.py --shuffle --batch_size 32 --num_G_accumulations 1 --num_D_accumulations 1 --num_epochs 200 --num_D_steps 1 --G_lr 1e-4 --D_lr 6e-4 --dataset DogOrigin96 --bottom_width 6 --G_ortho 0.0 --G_attn 0 --D_attn 0 --G_init ortho --D_init ortho --ema --use_ema --ema_start 2000 --test_every 25 --save_every 10 --num_best_copies 5 --num_save_copies 2 --G_ch 24 --D_ch 24 --seed 0 --augment 1 --add_blur --add_style --crop_mode 8 --on_kaggle --experiment_name i96_ch24_hinge_ema_dstep1_bs32_noatt_glr0001_glr0006_aug_init_ortho_blur_style_origin_crop_mode8\n\nThe score improved to **12.4** on public LB after these steps\n\nIn the last several days of this competition, I turned to the model's generalization and the data itself, I found that if I cropped square box containing the dogs only, it performed better than adding resized origin images, but, as kaggle will use another datasets and my models were trained on all-dogs datasets, they might over-fit to the cropped all-dogs to some extend, when my models were applied on other images, the dogs might not be cropped, so I added resized origin images to the datasets with some probability. The score on public LB of this augmentation is **13.4**, but I guess this model may perform better than 12.4 one.\n\nI also added all kinds of mechanisms , such as mode seeking, all kinds of loss functions such as such as GDPP, normalizations, regularizations , initializations, and most of them did not work.\n\nMost of mechanism I talked above are kept in the kernels and can be tested by using the parameters. If you want to have a try, 100 epochs in 4.5h can reach FID 18.x, and 125 epochs can reach FID 15.x\n\n**What can we do to improve the model if there are no limits on the resources and time?**\n1. I designed a mechanism to add dog count in an image to the model, that's to say, dog count embedding. It works, but it took more time. \n2. We can still cluster the images to 30-40 clusters, use these cluster id as other embedding, so, the dog on the grass, floor..... ( I did not test it )\n3. We can use large sized images.\n\nSo the FID less than 10 is not so difficult to achieve if we have two GPUs or double time. \n1 and 2 can be described as Multi-Classes-Conditional Style GAN, with the mixture of the Style and BigGAN, after some well designed experiments we may make some progress to the state of the art GANs?\nYou can check some of images, a type of dogs per line:\nhttps://raw.githubusercontent.com/bestfitting/kaggle/master/gandogs/samples/samples_mode8.jpg\n\nThe link to my kernels:\nhttps://www.kaggle.com/bestfitting/i96-1e-4-6e-4-bs32-195\nhttps://www.kaggle.com/bestfitting/i96-crop-mode3-dstep1-1e-4-6e-4-bs32\n\nUpdated\nThe source codes in the kennels above are not easy to read, you can check the normal python project structure version here https://github.com/bestfitting/kaggle/tree/master/gandogs\n\n\n
10 thumbs up!! Amazing work Bestfitting. I like how you combined the best features of multiple top performing GANs and invented your own GAN.
\n\nFor those that are interested, below are images from Bestfitting's two posted solutions. Check out the double headed dog in the bottom images in row 5 column 8 :-)
\n\n
![\"image\"](\"http://playagricola.com/Kaggle/dogs2b81419.png\")
Nice solution, thanks!\nI have similar findings about batch size and the number of channels in BigGAN model
", "rawMarkdown": "Nice solution, thanks!\nI have similar findings about batch size and the number of channels in BigGAN model", "votes": null }, { "id": "600240", "postDate": "08/15/2019 20:35:30", "content": "I have just put a link in my main post, you can check the images, a class of dogs per line. When the metric is FID, the variety is important.
", "rawMarkdown": "I have just put a link in my main post, you can check the images, a class of dogs per line. When the metric is FID, the variety is important.", "votes": null }, { "id": "600244", "postDate": "08/15/2019 20:40:48", "content": "Could you tell us please, how did you implement the code used in your kernels? Was it generated from notebooks?
\n\nIt looks like a dump of many Python files/notebooks into \"procedural\", C-like plain file. Did you iterate through your modules/py-files and gathered everything relevant making proper prefixes? So probably you had some \"source\" that was compiled into this \"concise\" representation.
", "rawMarkdown": "Could you tell us please, how did you implement the code used in your kernels? Was it generated from notebooks? \n\nIt looks like a dump of many Python files/notebooks into \"procedural\", C-like plain file. Did you iterate through your modules/py-files and gathered everything relevant making proper prefixes? So probably you had some \"source\" that was compiled into this \"concise\" representation.", "votes": null }, { "id": "600250", "postDate": "08/15/2019 20:45:12", "content": "It's inconvenient to edit the source codes in a kernel, so I wrote codes in my local computer and converted them to a file and paste it to the kernel. I am cleaning the codes.
", "rawMarkdown": "It's inconvenient to edit the source codes in a kernel, so I wrote codes in my local computer and converted them to a file and paste it to the kernel. I am cleaning the codes.", "votes": null }, { "id": "600399", "postDate": "08/16/2019 04:11:16", "content": "Amazing Work... Thanks for Sharing...!! @bestfitting
", "rawMarkdown": "Amazing Work... Thanks for Sharing...!! @bestfitting", "votes": null }, { "id": "600483", "postDate": "08/16/2019 06:46:49", "content": "Yes, absolutely agree, but did you do it manually or using some converter script? Because you're totally right and it is not that suitable to run scripts on kernels, so I am thinking how to better organize my pipelines, like write a proper Python structure with folders and modules, and then have something like export.py that could traverse that structure and paste into a single file to submit on Kaggle. I think that it could be a cool tool: point to the repo on GitHub, and convert it into single, standalone, plain .py file that can be run anywhere.
Thank you for taking your time and explaining all this. There is so much to learn here for me 👌
", "rawMarkdown": "Thank you for taking your time and explaining all this. There is so much to learn here for me 👌", "votes": null }, { "id": "600697", "postDate": "08/16/2019 13:16:27", "content": "I wrote a script to convert the project to a .py file indeed.
", "rawMarkdown": "I wrote a script to convert the project to a .py file indeed.", "votes": null }, { "id": "600706", "postDate": "08/16/2019 13:31:39", "content": "Thanks for Sharing
", "rawMarkdown": "Thanks for Sharing", "votes": null }, { "id": "600833", "postDate": "08/16/2019 16:05:13", "content": "Thanks for sharing.\nIs there any way one can get reliable tricks about StyleGAN training ?
", "rawMarkdown": "Thanks for sharing.\nIs there any way one can get reliable tricks about StyleGAN training ?", "votes": null }, { "id": "600843", "postDate": "08/16/2019 16:20:57", "content": "If you read my source codes, you will find I have moved many interesting mechanisms to BigGAN framework, for example, you can use my mix-style parameter to train a model to get similar effects of the SytleGAN. so this may be one of tricks to train StyleGAN efficiently by removing progressive training? But I have not tested on larger datasets and large image sizes, I believe it will work.
", "rawMarkdown": "If you read my source codes, you will find I have moved many interesting mechanisms to BigGAN framework, for example, you can use my mix-style parameter to train a model to get similar effects of the SytleGAN. so this may be one of tricks to train StyleGAN efficiently by removing progressive training? But I have not tested on larger datasets and large image sizes, I believe it will work.", "votes": null }, { "id": "600854", "postDate": "08/16/2019 16:32:34", "content": "Thanks for sharing @bestfitting .
", "rawMarkdown": "Thanks for sharing @bestfitting .", "votes": null }, { "id": "600901", "postDate": "08/16/2019 18:03:51", "content": "this is out of the world !
", "rawMarkdown": "this is out of the world !", "votes": null }, { "id": "601048", "postDate": "08/17/2019 00:14:34", "content": "I have read you solution, and I found we did a lot of similar experiments and got similar results, it confirmed a lot of effective and not so effective parts of BigGAN and others architecture, it's very interesting and useful.
", "rawMarkdown": "I have read you solution, and I found we did a lot of similar experiments and got similar results, it confirmed a lot of effective and not so effective parts of BigGAN and others architecture, it's very interesting and useful.", "votes": null }, { "id": "601093", "postDate": "08/17/2019 02:22:04", "content": "thank u :)
", "rawMarkdown": "thank u :)", "votes": null }, { "id": "601095", "postDate": "08/17/2019 02:23:25", "content": "Good info !
", "rawMarkdown": "Good info !", "votes": null }, { "id": "601116", "postDate": "08/17/2019 03:28:47", "content": "what a wonderful Name. Nice solution. I will try your method next time.
", "rawMarkdown": "what a wonderful Name. Nice solution. I will try your method next time.", "votes": null }, { "id": "601280", "postDate": "08/17/2019 10:19:28", "content": "Thanks for Sharing
", "rawMarkdown": "Thanks for Sharing", "votes": null }, { "id": "601296", "postDate": "08/17/2019 11:24:50", "content": "informative
", "rawMarkdown": "informative", "votes": null }, { "id": "601527", "postDate": "08/17/2019 18:41:02", "content": "informative
", "rawMarkdown": "informative", "votes": null }, { "id": "601559", "postDate": "08/17/2019 20:07:16", "content": "so this is the level of the best in the Kaggle... impresive! ;)
", "rawMarkdown": "so this is the level of the best in the Kaggle... impresive! ;)", "votes": null }, { "id": "602095", "postDate": "08/18/2019 15:06:00", "content": "many thanks for sharing. amazing work
", "rawMarkdown": "many thanks for sharing. amazing work", "votes": null }, { "id": "602140", "postDate": "08/18/2019 16:30:55", "content": "Without knowledge of real LB is might interesting, but not useful. Sometimes to know what happens is good to know how much effort should be added. And hidden LB pushed everybody to not share ideas, which is also not good.
", "rawMarkdown": "Without knowledge of real LB is might interesting, but not useful. Sometimes to know what happens is good to know how much effort should be added. And hidden LB pushed everybody to not share ideas, which is also not good.", "votes": null }, { "id": "602260", "postDate": "08/18/2019 20:42:59", "content": "Under the inspiration of this kernel, I used the following code. Since it is not readable anyway, this solution seemed more convenient to me.
\n\n```python\nimport base64\nimport io\nimport os\nimport shutil\nimport subprocess\nimport sys\nimport zipfile
\n\ndef unzip_code(code_b64):\n z = zipfile.ZipFile(io.BytesIO(base64.b64decode(code_b64)))\n z.extractall('.')
\n\nROOT_PROJECT_DIR = '../working/'\nPROJECT_DIR = 'dog_gan'\nsys.path.append('/kaggle/working')\nprint(sys.path)
\n\ncode_b64 = b'YOUR_BASE64_ZIP_HASHED_REPOSITORY'\ntry:\n shutil.rmtree(PROJECT_DIR)\nexcept Exception as e:\n pass\nunzip_code(code_b64)
\n\nfrom dog_gan.run import main
\n\nfrom dog_gan.configuration import KernelConfig
\n\nk = KernelConfig()
\n\nmain('submit')
\n\n```
", "rawMarkdown": "Under the inspiration of [this kernel](https://www.kaggle.com/lopuhin/imet-2019-submission), I used the following code. Since it is not readable anyway, this solution seemed more convenient to me.\n\n```python\nimport base64\nimport io\nimport os\nimport shutil\nimport subprocess\nimport sys\nimport zipfile\n\n\ndef unzip_code(code_b64):\n z = zipfile.ZipFile(io.BytesIO(base64.b64decode(code_b64)))\n z.extractall('.')\n\n\nROOT_PROJECT_DIR = '../working/'\nPROJECT_DIR = 'dog_gan'\nsys.path.append('/kaggle/working')\nprint(sys.path)\n\ncode_b64 = b'YOUR_BASE64_ZIP_HASHED_REPOSITORY'\ntry:\n shutil.rmtree(PROJECT_DIR)\nexcept Exception as e:\n pass\nunzip_code(code_b64)\n\nfrom dog_gan.run import main\n# Select your config (OOP-like hierarchy)\nfrom dog_gan.configuration import KernelConfig\n\nk = KernelConfig()\n# Change your config atributes here\nmain('submit')\n\n```", "votes": null }, { "id": "602368", "postDate": "08/19/2019 02:14:12", "content": "Thanks for sharing this info
", "rawMarkdown": "Thanks for sharing this info", "votes": null }, { "id": "602372", "postDate": "08/19/2019 02:27:17", "content": "Thank you for your sharing
", "rawMarkdown": "Thank you for your sharing", "votes": null }, { "id": "602404", "postDate": "08/19/2019 03:40:51", "content": "thanks
", "rawMarkdown": "thanks", "votes": null }, { "id": "602690", "postDate": "08/19/2019 11:50:03", "content": "Thanks
", "rawMarkdown": "Thanks", "votes": null }, { "id": "603144", "postDate": "08/19/2019 23:37:04", "content": "Great!
", "rawMarkdown": "Great!", "votes": null }, { "id": "603245", "postDate": "08/20/2019 03:48:52", "content": "awesome code. your work is so impresive! I really with have fun with this. thanks for sharing. I have a question for you? how many hours do you speed making this?
", "rawMarkdown": "awesome code. your work is so impresive! I really with have fun with this. thanks for sharing. I have a question for you? how many hours do you speed making this?", "votes": null }, { "id": "603410", "postDate": "08/20/2019 08:26:10", "content": "Good
", "rawMarkdown": "Good", "votes": null }, { "id": "603874", "postDate": "08/20/2019 18:53:43", "content": "thanks
", "rawMarkdown": "thanks", "votes": null }, { "id": "605974", "postDate": "08/23/2019 03:19:57", "content": "Great. Thanks for sharing 😊
", "rawMarkdown": "Great. Thanks for sharing 😊", "votes": null }, { "id": "609132", "postDate": "08/27/2019 12:37:22", "content": "thanks for the explanation :)
", "rawMarkdown": "thanks for the explanation :)", "votes": null }, { "id": "610546", "postDate": "08/28/2019 21:20:26", "content": "Nice, really good work!
", "rawMarkdown": "Nice, really good work!", "votes": null }, { "id": "646385", "postDate": "10/11/2019 07:44:34", "content": "Thank you for your sharing. I learned a lot from your explaination.
", "rawMarkdown": "Thank you for your sharing. I learned a lot from your explaination.", "votes": null }, { "id": "1000896", "postDate": "09/06/2020 22:04:36", "content": "What is the difference Between This Code And The Original BigGAN
", "rawMarkdown": "What is the difference Between This Code And The Original BigGAN", "votes": null }, { "id": "1084385", "postDate": "11/20/2020 01:47:43", "content": "Thank you for this. I am curious, what kind of hardware did you need to run and how long did training typically take?
", "rawMarkdown": "Thank you for this. I am curious, what kind of hardware did you need to run and how long did training typically take?", "votes": null }, { "id": "2644287", "postDate": "02/09/2024 11:12:00", "content": "سلام الله عليكم
", "rawMarkdown": "سلام الله عليكم", "votes": null } ], "comments": [ { "id": 1000896, "author_name": "amirahalsulami", "author_url": "", "post_date": "09/06/2020 22:04:36", "content": "What is the difference Between This Code And The Original BigGAN
", "votes": null, "replies": [] }, { "id": 1084385, "author_name": "matthewchung", "author_url": "", "post_date": "11/20/2020 01:47:43", "content": "Thank you for this. I am curious, what kind of hardware did you need to run and how long did training typically take?
", "votes": null, "replies": [] }, { "id": 2644287, "author_name": "khalidamin77", "author_url": "", "post_date": "02/09/2024 11:12:00", "content": "سلام الله عليكم
", "votes": null, "replies": [] }, { "id": 600211, "author_name": "cdeotte", "author_url": "", "post_date": "08/15/2019 19:52:37", "content": "10 thumbs up!! Amazing work Bestfitting. I like how you combined the best features of multiple top performing GANs and invented your own GAN.
\n\nFor those that are interested, below are images from Bestfitting's two posted solutions. Check out the double headed dog in the bottom images in row 5 column 8 :-)
\n\n
I have just put a link in my main post, you can check the images, a class of dogs per line. When the metric is FID, the variety is important.
", "votes": null, "replies": [] } ] }, { "id": 600212, "author_name": "dvorobiev", "author_url": "", "post_date": "08/15/2019 19:56:53", "content": "Nice solution, thanks!\nI have similar findings about batch size and the number of channels in BigGAN model
", "votes": null, "replies": [ { "id": 601048, "author_name": "bestfitting", "author_url": "", "post_date": "08/17/2019 00:14:34", "content": "I have read you solution, and I found we did a lot of similar experiments and got similar results, it confirmed a lot of effective and not so effective parts of BigGAN and others architecture, it's very interesting and useful.
", "votes": null, "replies": [] } ] }, { "id": 600244, "author_name": "purplejester", "author_url": "", "post_date": "08/15/2019 20:40:48", "content": "Could you tell us please, how did you implement the code used in your kernels? Was it generated from notebooks?
\n\nIt looks like a dump of many Python files/notebooks into \"procedural\", C-like plain file. Did you iterate through your modules/py-files and gathered everything relevant making proper prefixes? So probably you had some \"source\" that was compiled into this \"concise\" representation.
", "votes": null, "replies": [ { "id": 600250, "author_name": "bestfitting", "author_url": "", "post_date": "08/15/2019 20:45:12", "content": "It's inconvenient to edit the source codes in a kernel, so I wrote codes in my local computer and converted them to a file and paste it to the kernel. I am cleaning the codes.
", "votes": null, "replies": [] }, { "id": 600483, "author_name": "purplejester", "author_url": "", "post_date": "08/16/2019 06:46:49", "content": "Yes, absolutely agree, but did you do it manually or using some converter script? Because you're totally right and it is not that suitable to run scripts on kernels, so I am thinking how to better organize my pipelines, like write a proper Python structure with folders and modules, and then have something like export.py that could traverse that structure and paste into a single file to submit on Kaggle. I think that it could be a cool tool: point to the repo on GitHub, and convert it into single, standalone, plain .py file that can be run anywhere.
I wrote a script to convert the project to a .py file indeed.
", "votes": null, "replies": [] }, { "id": 602260, "author_name": "lucidyan", "author_url": "", "post_date": "08/18/2019 20:42:59", "content": "Under the inspiration of this kernel, I used the following code. Since it is not readable anyway, this solution seemed more convenient to me.
\n\n```python\nimport base64\nimport io\nimport os\nimport shutil\nimport subprocess\nimport sys\nimport zipfile
\n\ndef unzip_code(code_b64):\n z = zipfile.ZipFile(io.BytesIO(base64.b64decode(code_b64)))\n z.extractall('.')
\n\nROOT_PROJECT_DIR = '../working/'\nPROJECT_DIR = 'dog_gan'\nsys.path.append('/kaggle/working')\nprint(sys.path)
\n\ncode_b64 = b'YOUR_BASE64_ZIP_HASHED_REPOSITORY'\ntry:\n shutil.rmtree(PROJECT_DIR)\nexcept Exception as e:\n pass\nunzip_code(code_b64)
\n\nfrom dog_gan.run import main
\n\nfrom dog_gan.configuration import KernelConfig
\n\nk = KernelConfig()
\n\nmain('submit')
\n\n```
", "votes": null, "replies": [] } ] }, { "id": 600399, "author_name": "veeralakrishna", "author_url": "", "post_date": "08/16/2019 04:11:16", "content": "Amazing Work... Thanks for Sharing...!! @bestfitting
", "votes": null, "replies": [] }, { "id": 600533, "author_name": "matthiaspilz", "author_url": "", "post_date": "08/16/2019 08:35:52", "content": "Thank you for taking your time and explaining all this. There is so much to learn here for me 👌
", "votes": null, "replies": [] }, { "id": 600706, "author_name": "peacemakertsp", "author_url": "", "post_date": "08/16/2019 13:31:39", "content": "Thanks for Sharing
", "votes": null, "replies": [] }, { "id": 600833, "author_name": "penanklihicyrille", "author_url": "", "post_date": "08/16/2019 16:05:13", "content": "Thanks for sharing.\nIs there any way one can get reliable tricks about StyleGAN training ?
", "votes": null, "replies": [ { "id": 600843, "author_name": "bestfitting", "author_url": "", "post_date": "08/16/2019 16:20:57", "content": "If you read my source codes, you will find I have moved many interesting mechanisms to BigGAN framework, for example, you can use my mix-style parameter to train a model to get similar effects of the SytleGAN. so this may be one of tricks to train StyleGAN efficiently by removing progressive training? But I have not tested on larger datasets and large image sizes, I believe it will work.
", "votes": null, "replies": [] } ] }, { "id": 600854, "author_name": "joelhanson", "author_url": "", "post_date": "08/16/2019 16:32:34", "content": "Thanks for sharing @bestfitting .
", "votes": null, "replies": [] }, { "id": 600901, "author_name": "phoenix9032", "author_url": "", "post_date": "08/16/2019 18:03:51", "content": "this is out of the world !
", "votes": null, "replies": [] }, { "id": 601093, "author_name": "hsmin065", "author_url": "", "post_date": "08/17/2019 02:22:04", "content": "thank u :)
", "votes": null, "replies": [] }, { "id": 601095, "author_name": "easylife93", "author_url": "", "post_date": "08/17/2019 02:23:25", "content": "Good info !
", "votes": null, "replies": [] }, { "id": 601116, "author_name": "topchaebol", "author_url": "", "post_date": "08/17/2019 03:28:47", "content": "what a wonderful Name. Nice solution. I will try your method next time.
", "votes": null, "replies": [] }, { "id": 601280, "author_name": "sumansahoo16", "author_url": "", "post_date": "08/17/2019 10:19:28", "content": "Thanks for Sharing
", "votes": null, "replies": [] }, { "id": 601296, "author_name": "troublem1", "author_url": "", "post_date": "08/17/2019 11:24:50", "content": "informative
", "votes": null, "replies": [] }, { "id": 601527, "author_name": "abhinavsinha1", "author_url": "", "post_date": "08/17/2019 18:41:02", "content": "informative
", "votes": null, "replies": [] }, { "id": 601559, "author_name": "jesucristo", "author_url": "", "post_date": "08/17/2019 20:07:16", "content": "so this is the level of the best in the Kaggle... impresive! ;)
", "votes": null, "replies": [] }, { "id": 602095, "author_name": "jazzzzm", "author_url": "", "post_date": "08/18/2019 15:06:00", "content": "many thanks for sharing. amazing work
", "votes": null, "replies": [] }, { "id": 602140, "author_name": "muhakabartay", "author_url": "", "post_date": "08/18/2019 16:30:55", "content": "Without knowledge of real LB is might interesting, but not useful. Sometimes to know what happens is good to know how much effort should be added. And hidden LB pushed everybody to not share ideas, which is also not good.
", "votes": null, "replies": [] }, { "id": 602368, "author_name": "naveev", "author_url": "", "post_date": "08/19/2019 02:14:12", "content": "Thanks for sharing this info
", "votes": null, "replies": [] }, { "id": 602372, "author_name": "elsa1717", "author_url": "", "post_date": "08/19/2019 02:27:17", "content": "Thank you for your sharing
", "votes": null, "replies": [] }, { "id": 602404, "author_name": "tuanakka", "author_url": "", "post_date": "08/19/2019 03:40:51", "content": "thanks
", "votes": null, "replies": [] }, { "id": 602690, "author_name": "dbfud3516", "author_url": "", "post_date": "08/19/2019 11:50:03", "content": "Thanks
", "votes": null, "replies": [] }, { "id": 603144, "author_name": "lzz888", "author_url": "", "post_date": "08/19/2019 23:37:04", "content": "Great!
", "votes": null, "replies": [] }, { "id": 603245, "author_name": "marcovasquez", "author_url": "", "post_date": "08/20/2019 03:48:52", "content": "awesome code. your work is so impresive! I really with have fun with this. thanks for sharing. I have a question for you? how many hours do you speed making this?
", "votes": null, "replies": [] }, { "id": 603410, "author_name": "shubhangi23", "author_url": "", "post_date": "08/20/2019 08:26:10", "content": "Good
", "votes": null, "replies": [] }, { "id": 603874, "author_name": "amirahalsulami", "author_url": "", "post_date": "08/20/2019 18:53:43", "content": "thanks
", "votes": null, "replies": [] }, { "id": 605974, "author_name": "vishnurapps", "author_url": "", "post_date": "08/23/2019 03:19:57", "content": "Great. Thanks for sharing 😊
", "votes": null, "replies": [] }, { "id": 609132, "author_name": "benru89", "author_url": "", "post_date": "08/27/2019 12:37:22", "content": "thanks for the explanation :)
", "votes": null, "replies": [] }, { "id": 610546, "author_name": "ekolodin", "author_url": "", "post_date": "08/28/2019 21:20:26", "content": "Nice, really good work!
", "votes": null, "replies": [] }, { "id": 646385, "author_name": "doantientai", "author_url": "", "post_date": "10/11/2019 07:44:34", "content": "Thank you for your sharing. I learned a lot from your explaination.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "600170": "It's an interesting competition, and this is the first time I write a solution without knowing private LB score. Great thanks to kaggle for the efforts to host this competition and quick response to the community by @juliaelliott, @wendykan. \n\nMy solution may be described as a mix of StyleGAN and BigGAN.\n\nRespect to the authors of the papers and these two github repositories:\nhttps://github.com/ajbrock/BigGAN-PyTorch\nhttps://github.com/rosinality/style-based-gan-pytorch\n\n\n**Challenges**:\n1. Limited resources and limited time to train a GAN from scratch.\n2. A new kind of challenge with new rules and settings. It took me 2 weeks to choose between this one and the Youbute 8M video challenge.\n\n**Solution**:\n\nI tried DCGAN and ACGAN to get familiar with this competition at first , and then select papers related, I selected about 40 papers which might be useful.\n\nI was impressed by the results of the StyleGAN and tried to make it work more efficiently so I could make it run in a kernel, but I found Progressive GAN and its followers were inefficient to train, but I still spent about 10 days to tune it and got FID about 35 , I guessed that this score was not enough to top 10, but I thought I should spend more time on it to get understanding of every detail of it before I stepped forward.\n\nThen I tried VQ-VAE2 and a lot of other mechanisms and I was not satisfied with the efficiency and the scores.\n\nBigGAN was listed on the top of my most promising solutions, but the **BIG**, **DEEP** in the model name prevent me from trying it in a hurry and the paper stated that using more channels was important, I worried that it might be inefficient in this competition, so I planed to try it after I have tested others. To my surprise, the channels was not so important, when I decreased the G\\_ch and D\\_ch from 64 to 24, the FID score did not decrease too much and the FID score can easily reach below 30.\n\nWhat's more, the project structure of the https://github.com/ajbrock/BigGAN-PyTorch is very good, we can do a lot of experiments by just modifying some parameters, so I could start many experiments and went out have a bicycle driving or swimming :) After several experiments, my score reached about 24. \n\nAfter I checked and compared the architecture and mechanisms of the Big and Style GAN, I found that Spectral normalization and Class-conditional Batch normalization were important to BigAN, so we need not use progressive training, and BigGAN's G_shared and skip\\_z mechanisms were similar to the StyleGAN's style mechanism to some extend, so I made a lot of modifications to BigGAN and tried to move almost all important mechanisms of StyleGAN to it. After many experiments, I found: Style mechanism could improve the score significantly and coverage more quickly, the Blur was also important, Noise was not so important, to save time, I did not add it to my networks. Since I had more efficient and powerful tools for the BigGAN, so I did not use skip\\_z. I found mix-style mechanism of StyleGAN was not important in this competition, so I did not use it in my final submission. After these efforts and some hyper-parameters tuning, the score can reach about 17. \n\nI thought I might use large image size to train the network, so I designed a 96x96 model instead of 64x64, I found it could improve the score.\n\nI tried to use Attention including BigGAN's attention and CBAM in the network, it increased the training time and the improvement to the score was not so big, so I did not use them\n\nTo my surprise, although the paper of BigGAN stated that the larger batch-size, the better result, I found 32 was the best batch-size. So the two most important results--large channels and bigger batch-size of the BigGAN were not suitable of this dataset, so my solution should be called **Style-Small-GAN** ?? :smile::smile:\n\nMy best parameters can be described as the following command:\npython train.py --shuffle --batch_size 32 --num_G_accumulations 1 --num_D_accumulations 1 --num_epochs 200 --num_D_steps 1 --G_lr 1e-4 --D_lr 6e-4 --dataset DogOrigin96 --bottom_width 6 --G_ortho 0.0 --G_attn 0 --D_attn 0 --G_init ortho --D_init ortho --ema --use_ema --ema_start 2000 --test_every 25 --save_every 10 --num_best_copies 5 --num_save_copies 2 --G_ch 24 --D_ch 24 --seed 0 --augment 1 --add_blur --add_style --crop_mode 8 --on_kaggle --experiment_name i96_ch24_hinge_ema_dstep1_bs32_noatt_glr0001_glr0006_aug_init_ortho_blur_style_origin_crop_mode8\n\nThe score improved to **12.4** on public LB after these steps\n\nIn the last several days of this competition, I turned to the model's generalization and the data itself, I found that if I cropped square box containing the dogs only, it performed better than adding resized origin images, but, as kaggle will use another datasets and my models were trained on all-dogs datasets, they might over-fit to the cropped all-dogs to some extend, when my models were applied on other images, the dogs might not be cropped, so I added resized origin images to the datasets with some probability. The score on public LB of this augmentation is **13.4**, but I guess this model may perform better than 12.4 one.\n\nI also added all kinds of mechanisms , such as mode seeking, all kinds of loss functions such as such as GDPP, normalizations, regularizations , initializations, and most of them did not work.\n\nMost of mechanism I talked above are kept in the kernels and can be tested by using the parameters. If you want to have a try, 100 epochs in 4.5h can reach FID 18.x, and 125 epochs can reach FID 15.x\n\n**What can we do to improve the model if there are no limits on the resources and time?**\n1. I designed a mechanism to add dog count in an image to the model, that's to say, dog count embedding. It works, but it took more time. \n2. We can still cluster the images to 30-40 clusters, use these cluster id as other embedding, so, the dog on the grass, floor..... ( I did not test it )\n3. We can use large sized images.\n\nSo the FID less than 10 is not so difficult to achieve if we have two GPUs or double time. \n1 and 2 can be described as Multi-Classes-Conditional Style GAN, with the mixture of the Style and BigGAN, after some well designed experiments we may make some progress to the state of the art GANs?\nYou can check some of images, a type of dogs per line:\nhttps://raw.githubusercontent.com/bestfitting/kaggle/master/gandogs/samples/samples_mode8.jpg\n\nThe link to my kernels:\nhttps://www.kaggle.com/bestfitting/i96-1e-4-6e-4-bs32-195\nhttps://www.kaggle.com/bestfitting/i96-crop-mode3-dstep1-1e-4-6e-4-bs32\n\nUpdated\nThe source codes in the kennels above are not easy to read, you can check the normal python project structure version here https://github.com/bestfitting/kaggle/tree/master/gandogs\n\n\n