# 1st Place - TPU Leaderboard Prizes Submission Notebooks I am delighted to share with you the winning submission in these notebooks, [Training-Densenet201-Bagging1](https://www.kaggle.com/coreacasa/flower-class-tpu-densenet-training-v2) [Training-Densenet201-Bagging2](https://www.kaggle.com/coreacasa/flower-class-tpu-densenet-training-v3) [Training-EfficientnetB7-Bagging1](https://www.kaggle.com/coreacasa/flower-class-tpu-efficientnet-training-v2) [Training-EfficientnetB7-Bagging2](https://www.kaggle.com/coreacasa/flower-class-tpu-efficientnet-training-v3) [Inference-Flower-Classification-TPU](https://www.kaggle.com/coreacasa/flower-classification-tpu-inference-v3) Your links are pinned to the key versions. Although you will probably find some parts of the code redundant or not quite efficient, I didn't want to touch a comma to show the composition of the winning solution as it is here. This way, you can check both the clean inputs included in the inference notebooks and the automatic generation of the final solution. I just want to add 'a little' to my summary of the competition below and highlight the small details that perhaps made my score stronger. Acknowledgements Finally the TPU Leaderboard Prizes were settled and as you can imagine I am very happy with the results. Thank you very much to everyone. Thanks to all the Kaggle team and Google-Cloud-TPU for making this competition possible and thanks to the 848 teams that participated in this challenge. Whether it is explicitly (sharing notebooks and posting comments and discussions) or implicitly (pushing the leaderboard up with high scores) I think Kagglers are the ones that give a unique value to any challenge. What you can't miss I believe that more than a few Data Scientists will come to review the material in 'Flower Classification with TPUs' to get initiated or indeed improve the use of this powerful technology in their projects. If you are one of them here I detail, without order of preference or priority, a series of authors (sorry because I surely left some behind) of kernels whose work in this competition you should review (all Tensorflow). Thanks [@mgornergoogle](/mgornergoogle) (Martin Görner), [@xiejialun](/xiejialun) (Xie29), [@yihdarshieh](/yihdarshieh) (Yih-Dar SHIEH), [@cdeotte](/cdeotte) (Chris Deotte), [@calebeverett](/calebeverett) (Caleb), [@romanweilguny](/romanweilguny) (Roman Weilguny) but also don't forget to look at everything else if needed. I would state that the playground objective, which was aimed at facilitating the use of TPU technology, has been more than achieved. The Funny Part There are three key kernels at the beinning of the competition, - [ratan123/densenet201-flower-classification-with-tpus](https://www.kaggle.com/ratan123/densenet201-flower-classification-with-tpus) - [xhlulu/flowers-tpu-concise-efficientnet-b7](https://www.kaggle.com/xhlulu/flowers-tpu-concise-efficientnet-b7) - [wrrosa/tpu-enet-b7-densenet](https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet) The baseline of these notebooks is enough, for with the allowed competition data, to obtain the limit in the range of 0.9700 score. I am referring to the limit because of what we have seen in practice later on. On how to beat the limit of that score: the use of external data. [@cdeotte](/cdeotte) (Chris Deotte) in this discussions (1) [Oxford Flowers - Kaggle Dataset for Everyone](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/140964) . (2) [How to Score Over LB 0.970+](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/148329) and [@hengck23](/hengck23) (Heng CherKeng) in [[placeholder] external data and how to use them](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/140866) made public the manner. In my opinion this was the exciting part, I mean the rush to include these new datasets in training and see how the performance of the models improved very fast. I have to make a super special mention here for [@kirillblinov](/kirillblinov) (Kirill Blinov) and its shared external dataset [tf_flower_photo_tfrec](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec) , great work. In this way, [@szacho](/szacho) (Michal Szachniewicz) y [@calebeverett](/calebeverett) (Caleb) made an interesting contribution. By the way, one of the external dataset did not work for me, openimage . The Controversial Part I am referring here to all the final polemics about whether or not to use external datasets. The host never tried to hide the composition of the images in the test set and their origin. In my particular opinion I would have let the competition run until the end by allowing any source of data. It was also a very hard challenge to work on these new data and in any case the final obstacle is not so much the difficulty of the objective itself but the competition between the participants to overcome a score. The later, however, as long as everyone can access any advantage as happened here. You can be more or less in line with the direction that was finally decided, but you don't have to lose your commitment to respect and follow the rules of the competition for this reason. I expressed my disagreement with the final treatment of the Private Leaderboard and explained it [here](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/151494#84954) . Training Topics If you have enjoyed the competition from the beginning, participating in the published notebooks and discussions I think you won't find anything new in the kernels I have shared but you will probably be interested in knowing what could be missing or left over from your solution to achieve a higher score in the Public or Private Leaderboard. Augmentation I just used the files for an image size of 512x512 , I did not preprocess original images. On them, Random Blockout y Horizontal Random Flip were the best choices. Validation Since the initial validation dataset was very extensive and its information very useful in terms of performance, I divided the validation files into two subsets generating an additional Fold. Something like that looking for cross validation, Fold1>>> VALIDATIONFILENAMES1=VALIDATIONFILENAMES[:int(len(VALIDATIONFILENAMES)/2)] Fold2>>> VALIDATIONFILENAMES2=VALIDATIONFILENAMES[int(len(VALIDATIONFILENAMES)/2):] TRAININGFILENAMES1=TRAININGFILENAMES+VALIDATIONFILENAMES1 TRAININGFILENAMES2=TRAININGFILENAMES+VALIDATIONFILENAMES2 DenseNet-201 Imagenet Weights with tf.keras.layers.GlobalAveragePooling2D() . Little or no adjustment here, I kept BATCH_SIZE=128 and I brought to 4 the LR-RAMPUP-EPOCHS and LR-SUSTAIN-EPOCHS parameters of the well-tuned 'CustomLRSchedule' from [@mgornergoogle](/mgornergoogle) (Martin Görner) in his ' Five 'Flowers' Notebook. EfficientNet-B7 Noisy-Student Weights with tf.keras.layers.GlobalAveragePooling2D() . The rest is the same as Densenet-201 except for 5 the epochs number ramp-up and sustain. Loss y Optimizer Shared by both models, Loss with 'sparse_categorical_crossentropy' and Optimizer with Adam(lr='CustomLRSchedule') . Model Checkpoints The following callback was used to save the best weights according to the loss in validation, mc=tf.keras.callbacks.ModelCheckpoint('name.h5',verbose=1,monitor='val_loss', mode='min',save_best_only=True,save_weights_only=True) 2-Baggings While the use of the Blockout made the training results more stable, in general the performance of the models varied significantly (+/- 0.001) from run to run. Considering this problem of randomness I trained each model twice with the same basic setting. You can presume that the random augmentations are relevant here but even without them the variability of the TPU workouts is something important to consider. NUM_EPOCHS number 35, 50 to DenseNet and 35, 40 to EfficientNet. Inference Topics Therefore with the details above I actually had 8 trained models available with which to perform the classification inference. 1. DenseNet (4), 2xFold with best-checkpoints epochs 24-18, Fold1 and 25-24, Fold2 . 2. EfficientNet (4), 2xFold with best-checkpoints on epochs 17-16, Fold1 and 14-13, Fold2 . Test Time Augmentation The strategy here was to explode the training augmentated with 2 types of prediction. One of them (1) with the original 512x512 images without any transformation and the other (2) with 100% of the images flipped using the function tf.image.flip_left_right(image) as the only augment. Averaging Out of Folds I join the oofs of each bagging and model with a simple average of the logits , not probabilities. Following TTA strategy I perform the same operation with the full-flip images and let's say I get the following sets, 1 oof-non-flip-densenet 1 oof-non-flip-efficientnet 1 oof-full-flip-densenet 1 oof-full-flip-efficientnet Weighting Models OOFs Non-Flip With the logit transformation still applied, I use a simple optimizer to linearly maximize the weight of each model in the non-flip context. Best weighting to EfficientNet-B7: 0.71 Validation F1-Score: 0.969903 Weighting Models OOFs Full-Flip Following the same procedure I get now, Best weighting for EfficientNet-B7: 0.65 Validation F1-Score: 0.970014 One Last Optimization A final step was to obtain a new weighting but now to calibrate the power of the TTA strategy between full-flip and non-flip derived classifications. Best weighting for validation Non-Flip: 0.05 (any value 0.05 to 0.21) Validation F1-Score: 0.970367 That's all folks !!!