![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5116959%2Fec7ebd70ca3c485ef7d1f2606845fc1f%2FViT-Model.jpg?generation=1665563516204970&alt=media\")
A quick thank you to my teammates @simjeg and Marcos it was great working with you. It was a lot of fun exploring different datasets and dimensionality reduction techniques as a part of a team. Finally, thank you to the team of people that hosted this competition.
\nRepository : G-Universal-CLIP 🔥 🚀
\nWe use as baseline the content from the famous repo https://github.com/mlfoundations/open_clip
\nOur solution is an ensemble of 2 CLIP models trained using sub-center ArcFace with adaptive margin loss, each model produces a 512D descriptor. A total of 9 models were trained consisting of 4 ViT-L-14-336 models and 5 ViT-H-14 models.
\nThe weights of these 9 models were then averaged to make 2 “model soups”.
We created a 1024 dimensional embedding using the concatenated outputs of both models. We then used PCA to reduce the dimensionality from 1024 to 64.
\nPre-trained models from open_clip
\nViT-L-14-336 : openai
\nViT-H/14 on LAION-2B laion_s32b_b79k
Google Landmarks 2020 : https://www.kaggle.com/competitions/landmark-recognition-2020/data
\nProducts-10k : https://www.kaggle.com/competitions/products-10k/data
\nWe did not find it beneficial to use more data for finetuning. Products and landmarks already represent 50% of this challenge data distribution.
\nWe provide the details in G-Universal-CLIP and our paper! Stay tuned
\nKeep in mind we use ObjectNet for offline validation only, as discussed in this post : https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/352644
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5116959%2F0bf93302aa021538c8e454e742778477%2FViT-Model-Ensemble.jpg?generation=1665563545588984&alt=media\")
When creating the model soup we had to find a way to diversify the models before averaging their weights. It was decided that we would train each model with a different augmentation technique on each run, then train one model without augmentations. For the final ensemble we used the following augmentation methods:
\nThese augmentations were taken from this solution : https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/319896
\ndef transforms_happy_whale(image_path, image_size):\n image = cv2.imread(image_path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.uint8)\n aug8p3 = A.OneOf([\n A.Sharpen(p=0.3),\n A.ToGray(p=0.3),\n A.CLAHE(p=0.3),\n ], p=0.5)\n\n train_transforms = A.Compose([\n A.ShiftScaleRotate(rotate_limit=15, scale_limit=0.1, border_mode=cv2.BORDER_REFLECT, p=0.5),\n A.Resize(image_size, image_size),\n aug8p3,\n A.HorizontalFlip(p=0.5),\n A.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1),\n ToTensorV2(),\n ])\n return train_transforms(image=image)['image']\nThese augmentations were taken from this solution : https://www.kaggle.com/competitions/landmark-recognition-2020/discussion/187757
\ndef transforms_cutout(image_path, image_size):\n image = cv2.imread(image_path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.uint8)\n train_transforms = A.Compose([\n A.HorizontalFlip(p=0.5),\n A.ImageCompression(quality_lower=99, quality_upper=100),\n A.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=10, border_mode=0, p=0.7),\n A.Resize(image_size, image_size),\n A.Cutout(max_h_size=int(image_size * 0.4), max_w_size=int(image_size * 0.4), num_holes=1, p=0.5),\n ToTensorV2(),\n ])\n return train_transforms(image=image)['image']\nThese augmentations are taken from this paper : https://arxiv.org/abs/1805.09501
\ndef transforms_auto_augment(image_path, image_size):\n image = Image.open(image_path).convert('RGB')\n train_transforms = transforms.Compose([transforms.AutoAugment(transforms.AutoAugmentPolicy.IMAGENET), transforms.PILToTensor()])\n return train_transforms(image)\nWe created 4 ViT-L-14-336 models using the above augmentation methods with one model trained without augmentations (all 1 epoch).
\nWe created 5 ViT-H-14 models using the above augmentation methods with one model trained without augmentations (all 1 epoch) and an extra model trained on the auto augment transforms for 2 epochs.
\nWe then averaged these weights and were left with a single ViT-L-14-336 and a single ViT-H-14.
\nCLIP has a lot of potential in the future and it will be interesting to see where it goes. I am excited to share with everyone the full details in the paper that will soon be published. If you want to learn more, please make sure you check out the repository we have set up. It contains the training code used by our team.
\nThe paper will be available by Monday : https://arxiv.org/abs/2210.11141
\nRepository : G-Universal-CLIP
Great solution, how much time do you train for one model?
", "votes": 1, "replies": [ { "id": 1984811, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2022-10-12T22:03:33.463000", "content": "Roughly 2.5 hours for ViT-H-14 and a little over 3 hours for ViT-L-14-336. This is with a single RTX 3090.
", "votes": 1, "replies": [ { "id": 2077083, "author_name": "", "author_url": "", "post_date": "2022-12-27T08:16:46.907000", "content": "", "votes": 0, "replies": [] }, { "id": 2077085, "author_name": "", "author_url": "", "post_date": "2022-12-27T08:18:45.760000", "content": "", "votes": 0, "replies": [] }, { "id": 2077118, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2022-12-27T08:47:31.947000", "content": "Shouldn't you be using captions as the labels?
\nIf you have the captions you can just train Clip the normal way.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2103730, "author_name": "ZavodRobotov", "author_url": "", "post_date": "2023-01-17T10:41:42.223000", "content": "Great solution! How to implement PCA on embeddings? Should it be calculated on full dataset, or on one example?
", "votes": 0, "replies": [ { "id": 2200986, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-03-29T00:24:26.047000", "content": "Sorry, this reply is a little late.
\nPCA should be calculated on the training portion of the dataset. Then validated on the validation portion.
", "votes": 0, "replies": [] } ] } ], "comment_count": 7, "messages": [ { "id": 1983808, "postDate": "2022-10-12T08:47:58.177Z", "content": "A quick thank you to my teammates @simjeg and Marcos it was great working with you. It was a lot of fun exploring different datasets and dimensionality reduction techniques as a part of a team. Finally, thank you to the team of people that hosted this competition.
\nRepository : G-Universal-CLIP 🔥 🚀
\nWe use as baseline the content from the famous repo https://github.com/mlfoundations/open_clip
\nOur solution is an ensemble of 2 CLIP models trained using sub-center ArcFace with adaptive margin loss, each model produces a 512D descriptor. A total of 9 models were trained consisting of 4 ViT-L-14-336 models and 5 ViT-H-14 models.
\nThe weights of these 9 models were then averaged to make 2 “model soups”.
We created a 1024 dimensional embedding using the concatenated outputs of both models. We then used PCA to reduce the dimensionality from 1024 to 64.
\nPre-trained models from open_clip
\nViT-L-14-336 : openai
\nViT-H/14 on LAION-2B laion_s32b_b79k
Google Landmarks 2020 : https://www.kaggle.com/competitions/landmark-recognition-2020/data
\nProducts-10k : https://www.kaggle.com/competitions/products-10k/data
\nWe did not find it beneficial to use more data for finetuning. Products and landmarks already represent 50% of this challenge data distribution.
\nWe provide the details in G-Universal-CLIP and our paper! Stay tuned
\nKeep in mind we use ObjectNet for offline validation only, as discussed in this post : https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/352644
\n
When creating the model soup we had to find a way to diversify the models before averaging their weights. It was decided that we would train each model with a different augmentation technique on each run, then train one model without augmentations. For the final ensemble we used the following augmentation methods:
\nThese augmentations were taken from this solution : https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/319896
\ndef transforms_happy_whale(image_path, image_size):\n image = cv2.imread(image_path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.uint8)\n aug8p3 = A.OneOf([\n A.Sharpen(p=0.3),\n A.ToGray(p=0.3),\n A.CLAHE(p=0.3),\n ], p=0.5)\n\n train_transforms = A.Compose([\n A.ShiftScaleRotate(rotate_limit=15, scale_limit=0.1, border_mode=cv2.BORDER_REFLECT, p=0.5),\n A.Resize(image_size, image_size),\n aug8p3,\n A.HorizontalFlip(p=0.5),\n A.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1),\n ToTensorV2(),\n ])\n return train_transforms(image=image)['image']\nThese augmentations were taken from this solution : https://www.kaggle.com/competitions/landmark-recognition-2020/discussion/187757
\ndef transforms_cutout(image_path, image_size):\n image = cv2.imread(image_path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.uint8)\n train_transforms = A.Compose([\n A.HorizontalFlip(p=0.5),\n A.ImageCompression(quality_lower=99, quality_upper=100),\n A.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=10, border_mode=0, p=0.7),\n A.Resize(image_size, image_size),\n A.Cutout(max_h_size=int(image_size * 0.4), max_w_size=int(image_size * 0.4), num_holes=1, p=0.5),\n ToTensorV2(),\n ])\n return train_transforms(image=image)['image']\nThese augmentations are taken from this paper : https://arxiv.org/abs/1805.09501
\ndef transforms_auto_augment(image_path, image_size):\n image = Image.open(image_path).convert('RGB')\n train_transforms = transforms.Compose([transforms.AutoAugment(transforms.AutoAugmentPolicy.IMAGENET), transforms.PILToTensor()])\n return train_transforms(image)\nWe created 4 ViT-L-14-336 models using the above augmentation methods with one model trained without augmentations (all 1 epoch).
\nWe created 5 ViT-H-14 models using the above augmentation methods with one model trained without augmentations (all 1 epoch) and an extra model trained on the auto augment transforms for 2 epochs.
\nWe then averaged these weights and were left with a single ViT-L-14-336 and a single ViT-H-14.
\nCLIP has a lot of potential in the future and it will be interesting to see where it goes. I am excited to share with everyone the full details in the paper that will soon be published. If you want to learn more, please make sure you check out the repository we have set up. It contains the training code used by our team.
\nThe paper will be available by Monday : https://arxiv.org/abs/2210.11141
\nRepository : G-Universal-CLIP
Great solution, how much time do you train for one model?
", "rawMarkdown": "Great solution, how much time do you train for one model?", "votes": 1, "replies": [ { "id": 1984811, "postDate": "2022-10-12T22:03:33.463Z", "content": "Roughly 2.5 hours for ViT-H-14 and a little over 3 hours for ViT-L-14-336. This is with a single RTX 3090.
", "rawMarkdown": "Roughly 2.5 hours for ViT-H-14 and a little over 3 hours for ViT-L-14-336. This is with a single RTX 3090. ", "votes": 1, "replies": [ { "id": 2077083, "postDate": "2022-12-27T08:16:46.907Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2077085, "postDate": "2022-12-27T08:18:45.760Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2077118, "postDate": "2022-12-27T08:47:31.947Z", "content": "Shouldn't you be using captions as the labels?
\nIf you have the captions you can just train Clip the normal way.
", "rawMarkdown": "Shouldn't you be using captions as the labels?\n\nIf you have the captions you can just train Clip the normal way." } ] } ] }, { "id": 2103730, "postDate": "2023-01-17T10:41:42.223Z", "content": "Great solution! How to implement PCA on embeddings? Should it be calculated on full dataset, or on one example?
", "rawMarkdown": "Great solution! How to implement PCA on embeddings? Should it be calculated on full dataset, or on one example?", "replies": [ { "id": 2200986, "postDate": "2023-03-29T00:24:26.047Z", "content": "Sorry, this reply is a little late.
\nPCA should be calculated on the training portion of the dataset. Then validated on the validation portion.
", "rawMarkdown": "Sorry, this reply is a little late. \n\nPCA should be calculated on the training portion of the dataset. Then validated on the validation portion." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-24T13:24:16Z", "competition_id": 36414, "competition_slug": "google-universal-image-embedding", "team_id": 9060098, "solution_url": "/competitions/google-universal-image-embedding/writeups/ivan-simjeg-clip-art-guie-4th-place-solution" }