First of all we would like to thank Kaggle / Google for a very interresting competition. Also many thanks to my teammate @mightyrains for the excellent cooperation and teamwork.
\nOur solution
\nOur solution consisted of Tensorflow training pipeline that ran on the Kaggle TPU's. We also used Colab Pro additionally because of the Kaggle 20 hours TPU limit. This proved to be a very good combination todo many experiments.
We used the CLIP-ViT-H-14 model as backbone. As projection layer we use a 64 dense units linear layer that was trained with ArcFace (s = 30, m = 0.30). Augmentation consisted of the usual stuff like: Horizontal/vertical flips, rotations and randomly modifying saturation, brightness, hue and contrast.
\nThe majority of training runs consisted of 30 to 40 epochs where each epoch consisted of 20% of all training data.
\nFor datasets the core consisted of Products10K data and the Google Landmarks data. In our final solution we added some small datasets such as the Food Recognition, Stanford Cars dataset and the Best Artsworks of All Time. Looking at the private LB the Products10K and Landmarks on itself already seem to give a very good result. Depending on the training run the additional datasets could give a slight increase or decrease of the results. This is likely caused by the randomness in each training run.
\nFor the Tensorflow CLIP VIT H-14 model we ran into the issue eventually that after submission it would error on compute resources used. We solved this in a simple way by exporting the weights from the projection layer and importing these into a similar setup PyTorch model that we used for Inference and submission.
\nWe looked at setting up a complete training pipeline in PyTorch but with the easy way we could use the Tensorflow/Pytorch combination we now had we decided to keep it that way.
\nOur selected solutions scored 0.653 on the Private LB. Our best solution on the Private LB scored 0.658.
\nWhat didn't work / didn't seem to work / partially failed ;-)
\nOverall we can look back at a very exciting and challenging competition which we enjoyed very much.
", "comments": [ { "id": 2161382, "author_name": "KKY", "author_url": "", "post_date": "2023-02-27T12:58:42.100000", "content": "Sorry for be late here. Respect! You guys only use the Kaggle & colab to get such high score. BTW, you use the openai clip , not the open_clip model weight?
", "votes": 0, "replies": [ { "id": 2161483, "author_name": "KKY", "author_url": "", "post_date": "2023-02-27T14:40:42.740000", "content": "@rsmits look forward your reply
", "votes": 0, "replies": [ { "id": 2163119, "author_name": "Robin Smits", "author_url": "", "post_date": "2023-02-28T15:40:11.640000", "content": "Hi @evilpsycho42 Thanks! Yes combination of TPU and GPU from Kaggle + Colab. We used the model weights as available in https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K
", "votes": 1, "replies": [] } ] } ] }, { "id": 1986102, "author_name": "Rasoul Mojtahedzadeh", "author_url": "", "post_date": "2022-10-13T19:46:19.707000", "content": "Could you share your solution of converting tensorflow model weights to pytorch for submission?
", "votes": 0, "replies": [ { "id": 1986170, "author_name": "Robin Smits", "author_url": "", "post_date": "2022-10-13T21:11:24.980000", "content": "\nOur TF model consisted - besides the input layers and CLIP model itself - of only the 64 units dense layers that was used as embedding layer.
\ntf.keras.layers.Dense(64, name = 'embedding_dense_layer')
On each epoch we would save the weights of this layer through a Callback with custom code.
\nclass SaveModelCallback(tf.keras.callbacks.Callback):\n def on_epoch_end(self, epoch, logs=None):\n\n embed_layer_object = self.model.get_layer('embedding_dense_layer') \n\n embed_layer_kernel0 = embed_layer_object.weights[0]\n np.save('embed_layer_kernel0.npy', embed_layer_kernel0.numpy())\n\n embed_layer_bias0 = embed_layer_object.weights[1]\n np.save('embed_layer_bias0.npy', embed_layer_bias0.numpy())\nIn our PyTorch Inference model we would then setup the model in a similar way and we would load the exported weights into the Pytorch layer.
\nso something like this.
\nclass EmbeddingModel(nn.Module):\n def __init__(self, target_size = [224, 224], normalize = True):\n super().__init__()\n self.target_size = target_size \n self.model = CLIPVisionModel.from_pretrained(\"laion/CLIP-ViT-H-14-laion2B-s32B-b79K\")\n self.final = nn.Linear(1280, 64, bias = True)\n self.normalize = normalize\n\n def forward(self, x):\n x = x.to(device)\n x = x / 255.0\n x = transforms.functional.normalize(x, mean = [0.48145466, 0.4578275, 0.40821073], std = [0.26862954, 0.26130258, 0.27577711])\n x = transforms.functional.resize(x, size=self.target_size, interpolation = InterpolationMode.BICUBIC) \n model_output = self.model(x)\n image_embeds = model_output.pooler_output\n image_embeds = image_embeds / image_embeds.norm(p=2, dim=-1, keepdim=True) \n x = image_embeds \n\n x = self.final(x)\n if self.normalize:\n x = torch.nn.functional.normalize(x)\n\n return x\nThe tensorflow exported weights would then be loaded as follows.
\ntf_embed_layer_bias = np.load(\"../input/embeddingfiles/embed_layer_bias0.npy\")\ntf_embed_layer_weights = np.load(\"../input/embeddingfiles/embed_layer_kernel0.npy\")\n\nmodel = EmbeddingModel(target_size = [224, 224], normalize = True)\n\nmodel.final.weight.data = torch.from_numpy(tf_embed_layer_weights.T).to(device)\nmodel.final.bias.data = torch.from_numpy(tf_embed_layer_bias).to(device)\nWe could use it this way since we didn't further finetune the CLIP backbone.
\nHopes this answers your question.
", "votes": 2, "replies": [] }, { "id": 1986426, "author_name": "Rasoul Mojtahedzadeh", "author_url": "", "post_date": "2022-10-14T05:44:27.970000", "content": "Thank you! I had no clue that pytorch could solve out-of-memory issue with CLIP-ViT-H-14.
", "votes": 0, "replies": [] }, { "id": 1986799, "author_name": "Robin Smits", "author_url": "", "post_date": "2022-10-14T09:38:35.207000", "content": "Hi @rasoulmojtahedzadeh Yes fortunately it does. It is even able to run the G-14 version on inference time. That one has over a billion parameters.
\nThe performance is however less thatn that of the H-14 model. The G-14 model was only trained for 1/3 of the epochs.
", "votes": 1, "replies": [] }, { "id": 1986882, "author_name": "Rasoul Mojtahedzadeh", "author_url": "", "post_date": "2022-10-14T11:20:00.787000", "content": "Wow! I have missed pytorch in this competition:(
\nDo you know why pytorch is fit to submission notebook but not tf?
@rasoulmojtahedzadeh Oh I think Pytorch was used plenty enough….just may'be less visible in public kernels this time…but I haven't checked.
\nI think Pytorch is some more efficient with memory which makes the difference. We noticed when doing Inference with the TF version of the H-14 model that everytime it would get errors after about 8hours…so way within the time limit. We did some local checks with the speed that TF could achieve and it seemed that errors were around the time the embeddings for all images where generated. Presumable after that the KNN proces in the Kaggle submission would start.
\nWe tried to solve it with TF but switching that part to Pytorch was relatively ease.
", "votes": 1, "replies": [] } ] } ], "comment_count": 10, "messages": [ { "id": 1984684, "postDate": "2022-10-12T19:49:17.627Z", "content": "First of all we would like to thank Kaggle / Google for a very interresting competition. Also many thanks to my teammate @mightyrains for the excellent cooperation and teamwork.
\nOur solution
\nOur solution consisted of Tensorflow training pipeline that ran on the Kaggle TPU's. We also used Colab Pro additionally because of the Kaggle 20 hours TPU limit. This proved to be a very good combination todo many experiments.
We used the CLIP-ViT-H-14 model as backbone. As projection layer we use a 64 dense units linear layer that was trained with ArcFace (s = 30, m = 0.30). Augmentation consisted of the usual stuff like: Horizontal/vertical flips, rotations and randomly modifying saturation, brightness, hue and contrast.
\nThe majority of training runs consisted of 30 to 40 epochs where each epoch consisted of 20% of all training data.
\nFor datasets the core consisted of Products10K data and the Google Landmarks data. In our final solution we added some small datasets such as the Food Recognition, Stanford Cars dataset and the Best Artsworks of All Time. Looking at the private LB the Products10K and Landmarks on itself already seem to give a very good result. Depending on the training run the additional datasets could give a slight increase or decrease of the results. This is likely caused by the randomness in each training run.
\nFor the Tensorflow CLIP VIT H-14 model we ran into the issue eventually that after submission it would error on compute resources used. We solved this in a simple way by exporting the weights from the projection layer and importing these into a similar setup PyTorch model that we used for Inference and submission.
\nWe looked at setting up a complete training pipeline in PyTorch but with the easy way we could use the Tensorflow/Pytorch combination we now had we decided to keep it that way.
\nOur selected solutions scored 0.653 on the Private LB. Our best solution on the Private LB scored 0.658.
\nWhat didn't work / didn't seem to work / partially failed ;-)
\nOverall we can look back at a very exciting and challenging competition which we enjoyed very much.
", "rawMarkdown": "First of all we would like to thank Kaggle / Google for a very interresting competition. Also many thanks to my teammate @mightyrains for the excellent cooperation and teamwork.\n\n**Our solution**\nOur solution consisted of Tensorflow training pipeline that ran on the Kaggle TPU's. We also used Colab Pro additionally because of the Kaggle 20 hours TPU limit. This proved to be a very good combination todo many experiments. \n\nWe used the CLIP-ViT-H-14 model as backbone. As projection layer we use a 64 dense units linear layer that was trained with ArcFace (s = 30, m = 0.30). Augmentation consisted of the usual stuff like: Horizontal/vertical flips, rotations and randomly modifying saturation, brightness, hue and contrast.\n\nThe majority of training runs consisted of 30 to 40 epochs where each epoch consisted of 20% of all training data.\n\nFor datasets the core consisted of Products10K data and the Google Landmarks data. In our final solution we added some small datasets such as the Food Recognition, Stanford Cars dataset and the Best Artsworks of All Time. Looking at the private LB the Products10K and Landmarks on itself already seem to give a very good result. Depending on the training run the additional datasets could give a slight increase or decrease of the results. This is likely caused by the randomness in each training run.\n\nFor the Tensorflow CLIP VIT H-14 model we ran into the issue eventually that after submission it would error on compute resources used. We solved this in a simple way by exporting the weights from the projection layer and importing these into a similar setup PyTorch model that we used for Inference and submission.\n\nWe looked at setting up a complete training pipeline in PyTorch but with the easy way we could use the Tensorflow/Pytorch combination we now had we decided to keep it that way.\n\nOur selected solutions scored 0.653 on the Private LB. Our best solution on the Private LB scored 0.658. \n\n**What didn't work / didn't seem to work / partially failed ;-)**\n1. We had a hard time getting a good local validation strategy to work. There was to little consistency with the LB score. Because of little time available in the end we didn't further invest time in it.\n2. We tried various datasets like Omni Benchmark and webscraped images. With embeddings generated by a base CLIP model we tried to create clusters of images on which to further train a CLIP model with the idea to further increase performance. In general this didn't work…although it is interresting to note that performance also not really dropped. Using clean and clear datasets with proper labels was the best.\n3. We did some tests with ensembles of models. This didn't seem to work on LB….with hindsight this did work however. In the next competition this will defintely get more focus.\n4. Early in the competition we already did experiments with also training the backbone. At that moment it didn't give us proper results so we didn't follow up on that. Reading the various other solutions where this was applied succesfully this is something we will retry with a few Late Submissions.\n\nOverall we can look back at a very exciting and challenging competition which we enjoyed very much.\n", "votes": 15 }, { "id": 2161382, "postDate": "2023-02-27T12:58:42.100Z", "content": "Sorry for be late here. Respect! You guys only use the Kaggle & colab to get such high score. BTW, you use the openai clip , not the open_clip model weight?
", "rawMarkdown": "Sorry for be late here. Respect! You guys only use the Kaggle & colab to get such high score. BTW, you use the openai clip , not the open_clip model weight?", "replies": [ { "id": 2161483, "postDate": "2023-02-27T14:40:42.740Z", "content": "@rsmits look forward your reply
", "rawMarkdown": "@rsmits look forward your reply", "replies": [ { "id": 2163119, "postDate": "2023-02-28T15:40:11.640Z", "content": "Hi @evilpsycho42 Thanks! Yes combination of TPU and GPU from Kaggle + Colab. We used the model weights as available in https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K
", "rawMarkdown": "Hi @evilpsycho42 Thanks! Yes combination of TPU and GPU from Kaggle + Colab. We used the model weights as available in https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K\n", "votes": 1 } ] } ] }, { "id": 1986102, "postDate": "2022-10-13T19:46:19.707Z", "content": "Could you share your solution of converting tensorflow model weights to pytorch for submission?
", "rawMarkdown": "Could you share your solution of converting tensorflow model weights to pytorch for submission?", "replies": [ { "id": 1986170, "postDate": "2022-10-13T21:11:24.980Z", "content": "\nOur TF model consisted - besides the input layers and CLIP model itself - of only the 64 units dense layers that was used as embedding layer.
\ntf.keras.layers.Dense(64, name = 'embedding_dense_layer')
On each epoch we would save the weights of this layer through a Callback with custom code.
\nclass SaveModelCallback(tf.keras.callbacks.Callback):\n def on_epoch_end(self, epoch, logs=None):\n\n embed_layer_object = self.model.get_layer('embedding_dense_layer') \n\n embed_layer_kernel0 = embed_layer_object.weights[0]\n np.save('embed_layer_kernel0.npy', embed_layer_kernel0.numpy())\n\n embed_layer_bias0 = embed_layer_object.weights[1]\n np.save('embed_layer_bias0.npy', embed_layer_bias0.numpy())\nIn our PyTorch Inference model we would then setup the model in a similar way and we would load the exported weights into the Pytorch layer.
\nso something like this.
\nclass EmbeddingModel(nn.Module):\n def __init__(self, target_size = [224, 224], normalize = True):\n super().__init__()\n self.target_size = target_size \n self.model = CLIPVisionModel.from_pretrained(\"laion/CLIP-ViT-H-14-laion2B-s32B-b79K\")\n self.final = nn.Linear(1280, 64, bias = True)\n self.normalize = normalize\n\n def forward(self, x):\n x = x.to(device)\n x = x / 255.0\n x = transforms.functional.normalize(x, mean = [0.48145466, 0.4578275, 0.40821073], std = [0.26862954, 0.26130258, 0.27577711])\n x = transforms.functional.resize(x, size=self.target_size, interpolation = InterpolationMode.BICUBIC) \n model_output = self.model(x)\n image_embeds = model_output.pooler_output\n image_embeds = image_embeds / image_embeds.norm(p=2, dim=-1, keepdim=True) \n x = image_embeds \n\n x = self.final(x)\n if self.normalize:\n x = torch.nn.functional.normalize(x)\n\n return x\nThe tensorflow exported weights would then be loaded as follows.
\ntf_embed_layer_bias = np.load(\"../input/embeddingfiles/embed_layer_bias0.npy\")\ntf_embed_layer_weights = np.load(\"../input/embeddingfiles/embed_layer_kernel0.npy\")\n\nmodel = EmbeddingModel(target_size = [224, 224], normalize = True)\n\nmodel.final.weight.data = torch.from_numpy(tf_embed_layer_weights.T).to(device)\nmodel.final.bias.data = torch.from_numpy(tf_embed_layer_bias).to(device)\nWe could use it this way since we didn't further finetune the CLIP backbone.
\nHopes this answers your question.
", "rawMarkdown": "Hi @rasoulmojtahedzadeh,\n\nOur TF model consisted - besides the input layers and CLIP model itself - of only the 64 units dense layers that was used as embedding layer.\n\n`tf.keras.layers.Dense(64, name = 'embedding_dense_layer')`\n\nOn each epoch we would save the weights of this layer through a Callback with custom code.\n\n```\nclass SaveModelCallback(tf.keras.callbacks.Callback):\n def on_epoch_end(self, epoch, logs=None):\n\n embed_layer_object = self.model.get_layer('embedding_dense_layer') \n \n embed_layer_kernel0 = embed_layer_object.weights[0]\n np.save('embed_layer_kernel0.npy', embed_layer_kernel0.numpy())\n\n embed_layer_bias0 = embed_layer_object.weights[1]\n np.save('embed_layer_bias0.npy', embed_layer_bias0.numpy())\n```\n\nIn our PyTorch Inference model we would then setup the model in a similar way and we would load the exported weights into the Pytorch layer.\n\nso something like this.\n\n```\nclass EmbeddingModel(nn.Module):\n def __init__(self, target_size = [224, 224], normalize = True):\n super().__init__()\n self.target_size = target_size \n self.model = CLIPVisionModel.from_pretrained(\"laion/CLIP-ViT-H-14-laion2B-s32B-b79K\")\n self.final = nn.Linear(1280, 64, bias = True)\n self.normalize = normalize\n\n def forward(self, x):\n x = x.to(device)\n x = x / 255.0\n x = transforms.functional.normalize(x, mean = [0.48145466, 0.4578275, 0.40821073], std = [0.26862954, 0.26130258, 0.27577711])\n x = transforms.functional.resize(x, size=self.target_size, interpolation = InterpolationMode.BICUBIC) \n model_output = self.model(x)\n image_embeds = model_output.pooler_output\n image_embeds = image_embeds / image_embeds.norm(p=2, dim=-1, keepdim=True) \n x = image_embeds \n\n x = self.final(x)\n if self.normalize:\n x = torch.nn.functional.normalize(x)\n\n return x\n```\n\nThe tensorflow exported weights would then be loaded as follows.\n\n```\ntf_embed_layer_bias = np.load(\"../input/embeddingfiles/embed_layer_bias0.npy\")\ntf_embed_layer_weights = np.load(\"../input/embeddingfiles/embed_layer_kernel0.npy\")\n\nmodel = EmbeddingModel(target_size = [224, 224], normalize = True)\n\nmodel.final.weight.data = torch.from_numpy(tf_embed_layer_weights.T).to(device)\nmodel.final.bias.data = torch.from_numpy(tf_embed_layer_bias).to(device)\n```\nWe could use it this way since we didn't further finetune the CLIP backbone.\n\nHopes this answers your question.", "votes": 2 }, { "id": 1986426, "postDate": "2022-10-14T05:44:27.970Z", "content": "Thank you! I had no clue that pytorch could solve out-of-memory issue with CLIP-ViT-H-14.
", "rawMarkdown": "Thank you! I had no clue that pytorch could solve out-of-memory issue with CLIP-ViT-H-14." }, { "id": 1986799, "postDate": "2022-10-14T09:38:35.207Z", "content": "Hi @rasoulmojtahedzadeh Yes fortunately it does. It is even able to run the G-14 version on inference time. That one has over a billion parameters.
\nThe performance is however less thatn that of the H-14 model. The G-14 model was only trained for 1/3 of the epochs.
", "rawMarkdown": "Hi @rasoulmojtahedzadeh Yes fortunately it does. It is even able to run the G-14 version on inference time. That one has over a billion parameters.\n\nThe performance is however less thatn that of the H-14 model. The G-14 model was only trained for 1/3 of the epochs.", "votes": 1 }, { "id": 1986882, "postDate": "2022-10-14T11:20:00.787Z", "content": "Wow! I have missed pytorch in this competition:(
\nDo you know why pytorch is fit to submission notebook but not tf?
@rasoulmojtahedzadeh Oh I think Pytorch was used plenty enough….just may'be less visible in public kernels this time…but I haven't checked.
\nI think Pytorch is some more efficient with memory which makes the difference. We noticed when doing Inference with the TF version of the H-14 model that everytime it would get errors after about 8hours…so way within the time limit. We did some local checks with the speed that TF could achieve and it seemed that errors were around the time the embeddings for all images where generated. Presumable after that the KNN proces in the Kaggle submission would start.
\nWe tried to solve it with TF but switching that part to Pytorch was relatively ease.
", "rawMarkdown": "@rasoulmojtahedzadeh Oh I think Pytorch was used plenty enough....just may'be less visible in public kernels this time...but I haven't checked.\n\nI think Pytorch is some more efficient with memory which makes the difference. We noticed when doing Inference with the TF version of the H-14 model that everytime it would get errors after about 8hours...so way within the time limit. We did some local checks with the speed that TF could achieve and it seemed that errors were around the time the embeddings for all images where generated. Presumable after that the KNN proces in the Kaggle submission would start.\n\nWe tried to solve it with TF but switching that part to Pytorch was relatively ease.", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-22T08:32:52Z", "competition_id": 36414, "competition_slug": "google-universal-image-embedding", "team_id": 8991392, "solution_url": "/competitions/google-universal-image-embedding/writeups/the-kaggle-whisperers-28th-place-solution" }