![\"model_architecture\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1694617%2F0a169ad601296aabab75c6e1a80fd619%2Fmodel_architecture_loss_change_meta_info_del.png?generation=1665458129592589&alt=media\")
※2022/10/12 Title changed due to change in ranking.
\n※2022/10/18 Add solution code: https://github.com/riron1206/kaggle-Google-Universal-Image-Embedding-Competition-5th-Place-Solution
\n※2022/10/19 Add arXiv paper: https://arxiv.org/abs/2210.09495
Thank you to the hosts and the organizers for having such a challenging competition!
\nThinking about good image embedding was a lot of fun. And thank you to my teammates and all the participants for hard working!
StratifiedGroupKFold(n_splits=20).split(X, y=< used Dataset >, groups=< class_label >))StratifiedKFold(n_splits=20).split(X, y=< class_label >))It is very important to use ViT trained with LAION-2B from OpenCLIP for the backbone of ArcFace. OpenCLIP is more powerful than the official openai/CLIP.
\nAlso, adding image size information to embedding improved public lb by about 0.003.
\n
It is important to apply strong regularization(weight decay) to the model parameters to avoid overfitting the training data.
\nfreeze backbone parameters and train only head
optimizer: SAM(AdamW(weight_decay=0.1))
scheduler: warmup 0-3epoch(lr=0.0001->0.01) + Cosine Annealing 3-1000epoch(lr=0.01->0.0001)
augmentation: we used Albumentations follow OpenCLIP augmentations
A.Compose([\n A.RandomResizedCrop(height=224, width=224, scale=(0.9, 1.0), ratio=(1.0, 1.0)),\n A.Normalize(mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711], max_pixel_value=255.0, p=1.0),\n ToTensorV2(),\n ])\nSince lb improved when images with different heights and widths were resized keeping the aspect ratio, TTA was switched according to the aspect ratio of the image as follows.
\nIn addition, we switched TTA with a classification model. The classification model was trained with 130k-images-512x512-universal-image-embeddings.
\nimport torch\nimport torch.nn as nn\nfrom torchvision import transforms\n\nclass CLIPTTA(nn.Module):\n def __init__(self, encoder, neck, classifier):\n super().__init__()\n self.encoder = encoder # torchscript OpenClip ViT\n self.neck = neck # Model neck(head)\n self.classifier = classifier # Classifier Model\n\n ...\n\n def concat_image_info(self, emb, H: int, W: int):\n subfeature = torch.tensor([[\n float(H/W),\n float(H/224.0), \n float(W/224.0),\n ]], device=emb.device).float()\n return torch.cat([emb, subfeature], dim=1)\n\n def forward(self, x):\n B,C,H,W = x.shape\n ratio = W/H\n\n if ratio==1.0:\n x_resize = self.ccrop_resize(x,cratio=0.9) # center_crop_resize\n feat = self.encoder(x_resize) # OpenClip ViT embedding\n feat = self.concat_image_info(feat, H, W)\n feat = self.head(feat)\n return torch.nn.functional.normalize(feat)\n\n else:\n x_resize = self.ccrop_resize(x,cratio=1.0) # resize\n feat1 = self.encoder(x_resize) # OpenClip ViT embedding\n cls = torch.argmax(self.classifier(feat1)) # Classification\n feat1 = self.concat_image_info(feat1, H, W)\n feat1 = self.head(feat1)\n\n # cls: apparel=0, packaged=8, toys=10\n if cls == 0 or cls == 8 or cls == 10:\n x_pad = self.pad_ccrop_resize(x,cratio=0.9) # keep aspect center_crop_resize\n feat2 = self.encoder(x_pad) # OpenClip ViT embedding\n feat2 = self.concat_image_info(feat2, H, W)\n feat2 = self.head(feat2)\n\n x_pad = self.pad_ccrop_resize(x,cratio=1.0) # keep aspect resize\n feat3 = self.encoder(x_pad) # OpenClip ViT embedding\n feat3 = self.concat_image_info(feat3, H, W)\n feat3 = self.head(feat3)\n return torch.nn.functional.normalize((feat1+feat2+feat3)/3.0)\n else:\n x_pad = self.pad_ccrop_resize(x,cratio=0.9) # keep aspect center_crop_resize\n feat2 = self.encoder(x_pad) # OpenClip ViT embedding\n feat2 = self.concat_image_info(feat2, H, W)\n feat2 = self.head(feat2)\n\n return torch.nn.functional.normalize((feat1+feat2)/2.0)\nAlso, setting antialias=True in torchvision's resize improved lb. This is because antialias is active when the openclip model is trained.
torchvision.transforms.functional.resize(x, [224, 224],interpolation=transforms.InterpolationMode.BICUBIC, antialias=True)\nThe yellow line is the border of the GOLD MEDAL.
\n![\"lb_update\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1694617%2F576cf28ca33ef0699ce3f8a86512c4f9%2Flb_update_goldline_meta_info_del.png?generation=1665458200593557&alt=media\")
Congrats! lol , The not work list are very similar to us.
\nOur models seems not convergence … freeze train with 100ep + id uniform + 1e-3 cosine deday, then unfreeze 20ep + 1e-5 cosine decay.
", "votes": 1, "replies": [] }, { "id": 1982673, "author_name": "Nanashi", "author_url": "", "post_date": "2022-10-11T14:50:03.543000", "content": "@anonamename (and your team) congratulations! I like the idea of the add_features residual connection
Congrats! Thank you for sharing solution.
\n\n\nnum_features is set to +3 because normalized original height, original width, original aspect ratio are concatenated in the embedding of the backbone
\n
This is very interesting idea. Is this your original idea or used in other method?
", "votes": 0, "replies": [ { "id": 1982390, "author_name": "hirune924", "author_url": "", "post_date": "2022-10-11T11:59:03.220000", "content": "This is our original idea, invented by @sinpcw
", "votes": 3, "replies": [] }, { "id": 1982426, "author_name": "imori", "author_url": "", "post_date": "2022-10-11T12:22:10.727000", "content": "Thank you
", "votes": 0, "replies": [] } ] }, { "id": 1982143, "author_name": "alvitawa", "author_url": "", "post_date": "2022-10-11T09:28:20.217000", "content": "Thanks for your solution! How did you sample instances for the arcface (and what was your batch size)?
", "votes": 0, "replies": [ { "id": 1982188, "author_name": "anonamename", "author_url": "", "post_date": "2022-10-11T10:08:39.153000", "content": "sample instances are sampled with .sample(random_state=0) as follows
# glr2021 sampling\nlabels_glr = df_glr[\"label\"].unique()\nlabels_glr = pd.Series(labels_glr).sample(n=len(labels_glr) - 2000, random_state=0)\ndf_glr = df_glr[df_glr[\"label\"].isin(labels_glr)]\n\n# GPR1200 sampling\ndf_GPR1200 = df_GPR1200[df_GPR1200[\"category\"] != \"iNat\"]\n\n# food101 sampling\ndf_food101 = df_food101.groupby(\"label\").sample(n=30, random_state=0)\nbatch size is 256
", "votes": 2, "replies": [] } ] }, { "id": 1982070, "author_name": "Man of the year", "author_url": "", "post_date": "2022-10-11T08:33:52.947000", "content": "Also, what do you mean by neck in your model? And how did you avoid notebooks timeout, when you used Vit huge? As I see, sometimes you run encoder like 3-4 times for an image. My notebook with just vit huge took about 5-6 hours, so I decided not to run encoder several times.
", "votes": 0, "replies": [ { "id": 1982089, "author_name": "SiNpcw", "author_url": "", "post_date": "2022-10-11T08:53:10.310000", "content": "Sorry for the lack of clarity. Neck means the same as the Head of the model.
\nAs you pointed out, inference of ViT-H took about 290 minutes for full data. We tried to speed up the process by using classification model and image aspect ratio and applying TTA only partially. Performing TTA on a non-square image and not performing TTA on a square image took 1.5 times longer (approximately 7 hours).
", "votes": 3, "replies": [] } ] }, { "id": 1981980, "author_name": "khyeh", "author_url": "", "post_date": "2022-10-11T07:02:30.743000", "content": "Congrats and thanks for sharing, learned a lot! It is a good idea to have different preprocessing for different domains!
", "votes": 0, "replies": [] }, { "id": 1981831, "author_name": "Man of the year", "author_url": "", "post_date": "2022-10-11T04:45:49.993000", "content": "Congragulations on winning, your strategy and solution is very cool.
", "votes": 0, "replies": [] }, { "id": 1981979, "author_name": "", "author_url": "", "post_date": "2022-10-11T07:02:25.497000", "content": "", "votes": 0, "replies": [] }, { "id": 1981978, "author_name": "", "author_url": "", "post_date": "2022-10-11T07:01:32.013000", "content": "", "votes": 0, "replies": [] }, { "id": 2039693, "author_name": "vvoo", "author_url": "", "post_date": "2022-11-22T12:27:47.247000", "content": "Thanks for sharing solution.
", "votes": 0, "replies": [] }, { "id": 1984937, "author_name": "Mingyu Jeon", "author_url": "", "post_date": "2022-10-13T01:41:16.323000", "content": "Thanks for sharing solution
", "votes": 0, "replies": [] }, { "id": 1984004, "author_name": "Vaidehi Savaliya", "author_url": "", "post_date": "2022-10-12T11:59:06.383000", "content": "Thanks for sharing solution
", "votes": 0, "replies": [] }, { "id": 1981736, "author_name": "MiHu", "author_url": "", "post_date": "2022-10-11T03:07:31.720000", "content": "Thank you. I have learned a lot.
", "votes": 0, "replies": [] } ], "comment_count": 17, "messages": [ { "id": 1981727, "postDate": "2022-10-11T02:59:29.393Z", "content": "※2022/10/12 Title changed due to change in ranking.
\n※2022/10/18 Add solution code: https://github.com/riron1206/kaggle-Google-Universal-Image-Embedding-Competition-5th-Place-Solution
\n※2022/10/19 Add arXiv paper: https://arxiv.org/abs/2210.09495
Thank you to the hosts and the organizers for having such a challenging competition!
\nThinking about good image embedding was a lot of fun. And thank you to my teammates and all the participants for hard working!
StratifiedGroupKFold(n_splits=20).split(X, y=< used Dataset >, groups=< class_label >))StratifiedKFold(n_splits=20).split(X, y=< class_label >))It is very important to use ViT trained with LAION-2B from OpenCLIP for the backbone of ArcFace. OpenCLIP is more powerful than the official openai/CLIP.
\nAlso, adding image size information to embedding improved public lb by about 0.003.
\n
It is important to apply strong regularization(weight decay) to the model parameters to avoid overfitting the training data.
\nfreeze backbone parameters and train only head
optimizer: SAM(AdamW(weight_decay=0.1))
scheduler: warmup 0-3epoch(lr=0.0001->0.01) + Cosine Annealing 3-1000epoch(lr=0.01->0.0001)
augmentation: we used Albumentations follow OpenCLIP augmentations
A.Compose([\n A.RandomResizedCrop(height=224, width=224, scale=(0.9, 1.0), ratio=(1.0, 1.0)),\n A.Normalize(mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711], max_pixel_value=255.0, p=1.0),\n ToTensorV2(),\n ])\nSince lb improved when images with different heights and widths were resized keeping the aspect ratio, TTA was switched according to the aspect ratio of the image as follows.
\nIn addition, we switched TTA with a classification model. The classification model was trained with 130k-images-512x512-universal-image-embeddings.
\nimport torch\nimport torch.nn as nn\nfrom torchvision import transforms\n\nclass CLIPTTA(nn.Module):\n def __init__(self, encoder, neck, classifier):\n super().__init__()\n self.encoder = encoder # torchscript OpenClip ViT\n self.neck = neck # Model neck(head)\n self.classifier = classifier # Classifier Model\n\n ...\n\n def concat_image_info(self, emb, H: int, W: int):\n subfeature = torch.tensor([[\n float(H/W),\n float(H/224.0), \n float(W/224.0),\n ]], device=emb.device).float()\n return torch.cat([emb, subfeature], dim=1)\n\n def forward(self, x):\n B,C,H,W = x.shape\n ratio = W/H\n\n if ratio==1.0:\n x_resize = self.ccrop_resize(x,cratio=0.9) # center_crop_resize\n feat = self.encoder(x_resize) # OpenClip ViT embedding\n feat = self.concat_image_info(feat, H, W)\n feat = self.head(feat)\n return torch.nn.functional.normalize(feat)\n\n else:\n x_resize = self.ccrop_resize(x,cratio=1.0) # resize\n feat1 = self.encoder(x_resize) # OpenClip ViT embedding\n cls = torch.argmax(self.classifier(feat1)) # Classification\n feat1 = self.concat_image_info(feat1, H, W)\n feat1 = self.head(feat1)\n\n # cls: apparel=0, packaged=8, toys=10\n if cls == 0 or cls == 8 or cls == 10:\n x_pad = self.pad_ccrop_resize(x,cratio=0.9) # keep aspect center_crop_resize\n feat2 = self.encoder(x_pad) # OpenClip ViT embedding\n feat2 = self.concat_image_info(feat2, H, W)\n feat2 = self.head(feat2)\n\n x_pad = self.pad_ccrop_resize(x,cratio=1.0) # keep aspect resize\n feat3 = self.encoder(x_pad) # OpenClip ViT embedding\n feat3 = self.concat_image_info(feat3, H, W)\n feat3 = self.head(feat3)\n return torch.nn.functional.normalize((feat1+feat2+feat3)/3.0)\n else:\n x_pad = self.pad_ccrop_resize(x,cratio=0.9) # keep aspect center_crop_resize\n feat2 = self.encoder(x_pad) # OpenClip ViT embedding\n feat2 = self.concat_image_info(feat2, H, W)\n feat2 = self.head(feat2)\n\n return torch.nn.functional.normalize((feat1+feat2)/2.0)\nAlso, setting antialias=True in torchvision's resize improved lb. This is because antialias is active when the openclip model is trained.
torchvision.transforms.functional.resize(x, [224, 224],interpolation=transforms.InterpolationMode.BICUBIC, antialias=True)\nThe yellow line is the border of the GOLD MEDAL.
\n
Congrats! lol , The not work list are very similar to us.
\nOur models seems not convergence … freeze train with 100ep + id uniform + 1e-3 cosine deday, then unfreeze 20ep + 1e-5 cosine decay.
", "rawMarkdown": "Congrats! lol , The not work list are very similar to us. \n\nOur models seems not convergence ... freeze train with 100ep + id uniform + 1e-3 cosine deday, then unfreeze 20ep + 1e-5 cosine decay.", "votes": 1 }, { "id": 1982673, "postDate": "2022-10-11T14:50:03.543Z", "content": "@anonamename (and your team) congratulations! I like the idea of the add_features residual connection
Congrats! Thank you for sharing solution.
\n\n\nnum_features is set to +3 because normalized original height, original width, original aspect ratio are concatenated in the embedding of the backbone
\n
This is very interesting idea. Is this your original idea or used in other method?
", "rawMarkdown": "Congrats! Thank you for sharing solution.\n\n> num_features is set to +3 because normalized original height, original width, original aspect ratio are concatenated in the embedding of the backbone\n\nThis is very interesting idea. Is this your original idea or used in other method?", "replies": [ { "id": 1982390, "postDate": "2022-10-11T11:59:03.220Z", "content": "This is our original idea, invented by @sinpcw
", "rawMarkdown": "This is our original idea, invented by @sinpcw ", "votes": 3 }, { "id": 1982426, "postDate": "2022-10-11T12:22:10.727Z", "content": "Thank you
", "rawMarkdown": "Thank you " } ] }, { "id": 1982143, "postDate": "2022-10-11T09:28:20.217Z", "content": "Thanks for your solution! How did you sample instances for the arcface (and what was your batch size)?
", "rawMarkdown": "Thanks for your solution! How did you sample instances for the arcface (and what was your batch size)?", "replies": [ { "id": 1982188, "postDate": "2022-10-11T10:08:39.153Z", "content": "sample instances are sampled with .sample(random_state=0) as follows
# glr2021 sampling\nlabels_glr = df_glr[\"label\"].unique()\nlabels_glr = pd.Series(labels_glr).sample(n=len(labels_glr) - 2000, random_state=0)\ndf_glr = df_glr[df_glr[\"label\"].isin(labels_glr)]\n\n# GPR1200 sampling\ndf_GPR1200 = df_GPR1200[df_GPR1200[\"category\"] != \"iNat\"]\n\n# food101 sampling\ndf_food101 = df_food101.groupby(\"label\").sample(n=30, random_state=0)\nbatch size is 256
", "rawMarkdown": "sample instances are sampled with ```.sample(random_state=0)``` as follows\n```\n# glr2021 sampling\nlabels_glr = df_glr[\"label\"].unique()\nlabels_glr = pd.Series(labels_glr).sample(n=len(labels_glr) - 2000, random_state=0)\ndf_glr = df_glr[df_glr[\"label\"].isin(labels_glr)]\n\n# GPR1200 sampling\ndf_GPR1200 = df_GPR1200[df_GPR1200[\"category\"] != \"iNat\"]\n\n# food101 sampling\ndf_food101 = df_food101.groupby(\"label\").sample(n=30, random_state=0)\n```\n\nbatch size is 256", "votes": 2 } ] }, { "id": 1982070, "postDate": "2022-10-11T08:33:52.947Z", "content": "Also, what do you mean by neck in your model? And how did you avoid notebooks timeout, when you used Vit huge? As I see, sometimes you run encoder like 3-4 times for an image. My notebook with just vit huge took about 5-6 hours, so I decided not to run encoder several times.
", "rawMarkdown": "Also, what do you mean by neck in your model? And how did you avoid notebooks timeout, when you used Vit huge? As I see, sometimes you run encoder like 3-4 times for an image. My notebook with just vit huge took about 5-6 hours, so I decided not to run encoder several times. ", "replies": [ { "id": 1982089, "postDate": "2022-10-11T08:53:10.310Z", "content": "Sorry for the lack of clarity. Neck means the same as the Head of the model.
\nAs you pointed out, inference of ViT-H took about 290 minutes for full data. We tried to speed up the process by using classification model and image aspect ratio and applying TTA only partially. Performing TTA on a non-square image and not performing TTA on a square image took 1.5 times longer (approximately 7 hours).
", "rawMarkdown": "Sorry for the lack of clarity. Neck means the same as the Head of the model.\n\nAs you pointed out, inference of ViT-H took about 290 minutes for full data. We tried to speed up the process by using classification model and image aspect ratio and applying TTA only partially. Performing TTA on a non-square image and not performing TTA on a square image took 1.5 times longer (approximately 7 hours).", "votes": 3 } ] }, { "id": 1981980, "postDate": "2022-10-11T07:02:30.743Z", "content": "Congrats and thanks for sharing, learned a lot! It is a good idea to have different preprocessing for different domains!
", "rawMarkdown": "Congrats and thanks for sharing, learned a lot! It is a good idea to have different preprocessing for different domains!" }, { "id": 1981831, "postDate": "2022-10-11T04:45:49.993Z", "content": "Congragulations on winning, your strategy and solution is very cool.
", "rawMarkdown": "Congragulations on winning, your strategy and solution is very cool. " }, { "id": 1981979, "postDate": "2022-10-11T07:02:25.497Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1981978, "postDate": "2022-10-11T07:01:32.013Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2039693, "postDate": "2022-11-22T12:27:47.247Z", "content": "Thanks for sharing solution.
", "rawMarkdown": "Thanks for sharing solution." }, { "id": 1984937, "postDate": "2022-10-13T01:41:16.323Z", "content": "Thanks for sharing solution
", "rawMarkdown": "Thanks for sharing solution" }, { "id": 1984004, "postDate": "2022-10-12T11:59:06.383Z", "content": "Thanks for sharing solution
", "rawMarkdown": "Thanks for sharing solution\n" }, { "id": 1981736, "postDate": "2022-10-11T03:07:31.720Z", "content": "Thank you. I have learned a lot.
", "rawMarkdown": "Thank you. I have learned a lot." } ], "complete": true, "source": "live", "fetched_at": "2026-10-24T13:23:05Z", "competition_id": 36414, "competition_slug": "google-universal-image-embedding", "team_id": 8888698, "solution_url": "/competitions/google-universal-image-embedding/writeups/ns-embedding-5th-place-solution-ns-embedding" }