# Solution Arhitecture Pipeline for Bronze Medal 

It had been a very interesting competition filled with of a lot of challenges, starting with the noise from the dataset, the possibility of using images from the previous competition dataset, and the diversity of possible solutions.
I would like to share to the community one of the designed architectures that lead me and my team to the bronze medal
![](assets/2abd7511e4e80769fbc460ac.jpg)
Presented above is the general schema of the architecture which was designed from 4 steps.
First step: Finding the best 3 models for the original provided data
There were tuned:
architecture type: EfficientNet B2, EfficientNet B3, EfficientNet B4
input image size: 480x480, 512x512, 520x520, 600x600
training epochs number: 30, 40
learning rate: 0.0001, 0.00015, 0.0002. 0.0003
learning rate scheduler: Cosine Anealing w/wo warmup, One Cycle, Reduce on Plateau
aditional layers on top of the network
random resize crop paramers
cutout probability, cutout size, cutout number, random brightness, random contrast parameters,     - hue&saturation
label smoothing w/wo
cutmix w/wo and the proportions of cutting ([0.3, 0.4, 0.5, 0.6, 0.7])
imagenet vs noisy student initial weights
optimizer: Adam, RangerLars, AdamW
The best 3 models configurations were:
Model 1:
Model architecture: EfficientNet B2 (imagenet weights)
Image size: 600x600
Augmentations: Cutout(nr:32, size:32, prob:0.5), Random Brightness, Random Contrast, Transpose + Flip, Cutmix
Adam + One cycle (max value: 0.0003)
Model 2:
Model architecture: EfficientNet B4 (noisy student weights)
Image size: 512x512
Augmentations: Cutout(nr:32, size:32, prob:0.5), Random Brightness, Random Contrast, Transpose + Flip, Cutmix,
Adam + One cycle (max value: 0.0002) + Label Smoothing
Model 3:
Model architecture: EfficientNet B3 (imagenet weights)
Image size: 512x512
Augmentations: Cutout(nr:32, size:32, prob:0.5), Random Brightness, Random Contrast, Transpose + Flip, Cutmix,
Adam + One cycle (max value: 0.0002) + Label Smoothing
Second step: Denoising as much as possible the original data
![](assets/c353e600cc48b62e6284e908.jpg)
The OOF prediction of the presented 3 models using 8 TTA combined using a weighted mean(weighted using the OOF score) was used to eliminate the samples that most likely were wrongly label. So we use the best OOF accuracy 3 models, each with 8 TTA added for extra robustness. We eliminated data where the predicted probabily for ground truth is lower than 0.20 (random chance).
Using the new clean dataset, we tune the parameters and find the best possible model on this data
Third step: Adding useful data from the last year similar competition
![](assets/516be7dbdb1a77423c05cb77.jpg)
The OOF prediction of the original 3 models using 8 TTA combined using a weighted mean(weighted using the OOF score) was used to determine what data are most likely to be correct from the last year dataset. So we use the best OOF accuracy 3 models, each with 8 TTA added for extra robustness. We added data where the predicted probabily for ground truth is higher than 0.90.
Using the new clean dataset, we tune the parameters and find the best possible model on this data
Forth step: Designing a meta classifier on top of the 3 models (original, cleaned and extended dataset)
![](assets/50608906830d08693d432ecb.jpg)
The input for the final step of the pipeline is the predictions of 3 models(original, cleaned and extended dataset) each with 4 TTA. So in total 12 predictions.
There had been tested bagging, boosting and classical stacking and the best result is the one presented below
![](assets/1075bad2631a3e7b773f9468.jpg)
So, we have a bagging of 100 meta classifiers, where each of the classifier is a two layer system with Random Forest, SVC, LGBM and Ridge at the bottom and SVC on the second layer. The main parameters of the bagging system is max_features = 0.3 and max_sample = 0.1 .
The important parameters were tuned with a Bayesian approach.
An interesting comparison was made between bagging and boosting methodology. As it you know, boosting is an algorithm which lead to more good results but prone to overfitting and bagging is more robust, generalize better but can give lower accuracy. And our results confirm this hypothesis. In the validation set, boosting provide much better accuracy but on the leaderboard due to the fact that we don't have a very good alignment between validation and learboard, bagging wins due to extra robustness.
In the end I want to congratulate the winners and to all participants !