# Our solution [5th place]

The key idea was to train several different convolutional neural networks and then build an ensemble.
We trained many different CNNs: Resnet50, Resnet101, Resnet152, InceptionResnetV2, DenseNet161. Each one gives 74-75 LB score except of DenseNet161 which gives about 73. Using simple averaging we got 77.5. However, it was not enough and we decided to train a second-layer model.
We had only one CNN which was trained using KFolds split. It was Resnet-50. For each product and for each image of this product we extracted TOP-5 probabilities and 5 relevant classes. So, there were 5 categorical and 5 float features for each image. We also noticed that one image could be present in the data multiple times, thus, we computed MD5 hash for each picture and matched it with the most common class IDs. So, we built another 10 features for each image: 5 TOP most common class IDs with such hash and the number of images with such hash.
After this procedure, we had 80 raw features (40 numerical and 40 categorical). We preprocessed categorical features using different approaches. Since it is impossible to train xgboost on 5270 classes we added a new feature "possible_class_id" and changed the multiclassification problem to binary classification. Next, we were training a model which predicts 'is it true that this product has class_id which is equal to "possible_class_id"'. Therefore, a new dataset consisted of samples (product_features, possible_class_id, binary_target). We only used samples with possible_class_id which was one of the TOP5 predictions of CNN or one of the possible classes based on hash trick.
Using the new dataset it was possible to train a second-layer model. We tried different models: XGBoost, LightGBM, CatBoost, Random Forest, Extra Trees. They gave similiar results and averaging of their predictions improved our score. Since we didn't want to spend too much time to finding the best coefficient for each model in our ensemble, we decided to train a third-layer model.
For third layer we used:
The features which were using for train second-layer models
Predictions of second-layer models. We also computed several statistics of predictions on a set of possible class ids.
Priors for categorical features. For each class_id we computed a frequency of this class in the whole dataset and a frequency of this class among the closest products based on their IDs (50k closest products). It was helpful because the dataset has a slight leak.
Our third-layer models are three different LightGBM models and XGBoost one. Then, we used weighted arifmetic mean of these models to compute predictions for the test set. It seems that we could improve our score by building a fourth layer model :)
For test predictions we used two different approaches:
First, average probabilities extracted from CNNs, then build features and use them in second and third layer models.
Extract predictions of each CNN, build features, use it for predictions on second and third layers and only then merge predictions of different CNNs.
These two approaches gave almost the same score. We averaged their predictions and got a slight improvement.
Some notes:
I also tried to use predictions of the test set for training CNN. It improved the score of my single Resnet101 from 0.749 to 0.754. I stopped training after 1.5 epochs because such model showed worse results when I tried to put the predictions to XGBoost. However, I think it was possible to get 76+ from this single model using this approach. I also tried to apply this trick to second-layer models, the score also improved but it decreased the score of the third-layer model. I do believe that we could get better results applying this trick to our third-layer models. However, we didn't have extra time for doing it :(
For training CNN we (Alexey Kharlamov and me) used such approach: first, initalize a model using pretrained weights from Imagenet, there were not frozen layers, the augmentation was disabled. Then, as soon as the validation score stopped growing, we added augmentation and doubled batch size. After this procedure, the score began to grow sharply. But, after a while the growth stopped, so we repeated this procedure again and again.
Congratulations to the winners and thanks for all the participants!