# Silver medal solution and approaches we tried on the way

First of all, we are very happy to be a part of such unordinary and interesting competition, many thanks to the organizers and
[@andrefaraujo](https://www.kaggle.com/andrefaraujo)
, we learnt a lot and did a wonderful work, and many thanks to my great team QuData
[@synset](https://www.kaggle.com/synset)
,
[@kerrit](https://www.kaggle.com/kerrit)
and
[@vaslav](https://www.kaggle.com/vaslav)
, you are awesome!
Pretrained Models and AvgPool
From our first encounter with the competition, we decided to try different models for image encoding: SwinTransformer, ConvNeXt, Dino VisionTransformer and EfficientNet. However, none of them gave a boost compared to CLIP from OpenAI (ViT_L_14_336px). Based on this model, with the compression of the CLIP output vector (768 elements in this model, later in the retrained - 1024) using AvgPool, we got the first starting point in this competition with a metric of 0.447. It is important to clarify that all the time we used the last CLIP layer, and experiments on the penultimate layer did not improve the result.
Experiments with different preprocessing variations (such as the original CLIP preprocessing) did not increase the metric either. Similarly, experiments with standard PCA and AutoEncoder instead of AvgPool gave negative results. Although the result was better locally, but after model submission, the metric turned out to be significantly lower.
Curiously, a noticeable boost at this stage was achieved by analyzing and optimizing the CLIP output vector. It turned out that in the feature space after encoders (both text and image) there are strongly correlated outliers on pairs of some vector components. This fact does not depend on the dataset and the model (although the outliers, of course, are in different places in different models). So, below is a plot of the typical means of each component of the feature vector (in red) and their standard deviations (in green).
mean, std = vectors.mean(axis=0), vectors.std(axis=0)
![](assets/5330ff230ac3fb5105b3d2f8.png)
Removing these components allowed us to win in terms of quality, which is how we managed to achieve an increase of +0.020 points, and compressing these components using Whitening PCA with training based on Caltech + ImageNet1k + self-assembled Apparel datasets gave an increase of +0.038. A similar effect was given by the normalization of vectors:
vectors = (vectors - mean) / std
Text-based PCA
At the beginning of the competition, there were quite strict requirements for the origin of datasets. Since the CLIP model was trained on the cosine proximity between the text and image encoder vectors, we decided to try using a set of text descriptions instead of image datasets. With the help of a text generator based on templates, about 200 thousand balanced texts on the topics of the competition domains were obtained. The vectors of these texts (after the text encoder with subsequent normalization) made it possible to build a fairly good PCA model for this stage:
CLIP Backbone Text (output 768) => Whitening PCA (output 128) => AvgPool (output 64).
This approach achieved 0.534 points, and we did not return to it again. Note that the use of a PCA-based projector for various image datasets usually did not lead to satisfactory results.
Training a Projector for a classification
After the release of
[@motono0223](https://www.kaggle.com/motono0223)
's notebook (special thanks for that), we decided to train the projector as part of a classification task with contrast learning based on ArcMargin. Experiments were also carried out with other methods of contrast learning, but for this problem ArcMargin with parameters m = 0.33 and s = 22 turned out to be better.
Due to the lack of powerful hardware, we immediately discarded the idea of ​​retraining the backbone CLIP. To conduct experiments with hyperparameters and datasets, we decided to get feature vectors once by passing dataset images through the image encoder CLIP. Then these vectors were stored in separate datasets and loaded immediately before training the projector. This made it possible to significantly speed up the learning process (several seconds per epoch on a standard colab GPU machine).
To eliminate overfitting, we used feature space augmentation rather than image space. To do this, a random bias with a normal distribution was added to the feature vector (its variance was a rather important hyperparameter). Also, a dropout layer in front of the projector turned out to be a good way to augment in the feature space.
As a projector, a linear layer was usually used, followed by normalization of the vector. In this way, we have achieved an improvement in the metric to about 0.555. Note that the use of a nonlinear projector always led to a deterioration in the metric (although the metric improved significantly on validation data in local datasets).
Experimenting with Datasets
It seemed to us that it was important to have datasets reflecting the composition of the test dataset by domain. Therefore, at first we manually compiled the following datasets:
Packaged Goods (1000 classes),
Landmarks (210 classes),
Dishes (67 classes),
Furniture (60 classes).
Although rather unbalanced, this selection of datasets was enough to reach approximately 0.600 points.
Next, we decided to expand the list of datasets and select the correct options and formats for the datasets themselves.
According to our observations, in this competition, not class-type datasets, but instance-type datasets work better - these are those where one class represents one specific entity - Landmarks is an ideal example, and Dishes is a good example of a class-based dataset.
As a result, we came to the following list of datasets:
Apparel & accessories:
[https://www.kaggle.com/datasets/andreybeyn/qudata-gembed-hm-products-apparel-acc-187](https://www.kaggle.com/datasets/andreybeyn/qudata-gembed-hm-products-apparel-acc-187)
[https://mmlab.ie.cuhk.edu.hk/projects/DeepFashion.html](https://mmlab.ie.cuhk.edu.hk/projects/DeepFashion.html)
[https://github.com/switchablenorms/DeepFashion2](https://github.com/switchablenorms/DeepFashion2)
[https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/data](https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/data)
[https://www.kaggle.com/datasets/mayukh18/fashion200k-dataset](https://www.kaggle.com/datasets/mayukh18/fashion200k-dataset)
Packaged goods:
[https://www.kaggle.com/competitions/products-10k](https://www.kaggle.com/competitions/products-10k)
[https://www.pinlandata.com/rp2k_dataset/](https://www.pinlandata.com/rp2k_dataset/)
Landmarks:
[https://www.kaggle.com/datasets/andreybeyn/qudata-gembed-landmarks-210](https://www.kaggle.com/datasets/andreybeyn/qudata-gembed-landmarks-210)
[https://www.kaggle.com/datasets/kerrit/landmarks-416](https://www.kaggle.com/datasets/kerrit/landmarks-416)
Furniture & decor:
[https://www.kaggle.com/datasets/kerrit/furniture-101](https://www.kaggle.com/datasets/kerrit/furniture-101)
[https://www.kaggle.com/datasets/andreybeyn/qudata-gembed-furniture-180/](https://www.kaggle.com/datasets/andreybeyn/qudata-gembed-furniture-180/)
[https://www.kaggle.com/datasets/promptcloud/amazon-home-furnishings-dataset](https://www.kaggle.com/datasets/promptcloud/amazon-home-furnishings-dataset)
Storefronts:
[https://www.kaggle.com/datasets/kerrit/storefront-146](https://www.kaggle.com/datasets/kerrit/storefront-146)
Dishes:
[https://www.kaggle.com/datasets/kerrit/dishes-67](https://www.kaggle.com/datasets/kerrit/dishes-67)
Artwork:
[http://cmp.felk.cvut.cz/met/](http://cmp.felk.cvut.cz/met/)
[https://www.kaggle.com/datasets/kerrit/artwork-51](https://www.kaggle.com/datasets/kerrit/artwork-51)
Toys:
[https://www.kaggle.com/datasets/andreybeyn/qudata-gembed-toys-46](https://www.kaggle.com/datasets/andreybeyn/qudata-gembed-toys-46)
After assembling the datasets, we realized that the datasets themselves would be difficult to balance in accordance with the Google domain distribution schedule, so we decided to achieve this balancing by another method - by assigning to each of the datasets its own weight in the model pipeline.
Change to CLIP model ViT-H/14
About 2-3 weeks before the end of the competition, we decided to change the CLIP model to a slightly different version: CLIP ViT-H-14 laion2b_s32b_b79k. We quickly switched to this model by re-vectorizing all datasets. This pre-trained version of CLIP with the correct list of datasets: Apparel (187), Packaged Goods (3k), RP (2k), Storefront (146), Landmarks (1k), Furniture (135), Food (101), Artwork (51) allowed us to temporarily take 4th place, reaching a score of 0.646 points without additional training backbone CLIP!
Unfortunately, since we did not have the opportunity to use our datasets and approach along with the backbone CLIP retraining, and as a result we were quickly overtaken, but this competition was interesting, we learned a lot and gained valuable experience in such an interesting task as Image Embeddings. Thanks to all!