So the competition is finalized now, and we are so glad to get the 1st place in both public LB and private one. We really appreciate the amazing competition held by the host @andrefaraujo and the support provided by Kaggle staffs @maggiemd Contestants who share their wonderful ideas and resources also help me a lot. I may forget to tag some persons but I want to say you guys did really a wonderful job!
\nHere I will not only share how we get the final performance, but I also go through the whole competition, for what difficulties we faced and what approaches we chose to overcome, in the order of time. I hope it will help people better learn the practical things one may face and alternative ways to get over them in future competitions.
\nHonestly We felt doubtful about joining this competition or not. It is so rare for both of us to see a competition w/o a provided datasets. But we finally took it as a challenge for us to get over this task, and decided to join in (Luckily we did so!).
\nAt the beginning, @khabel and @skyyy93 were way ahead of the rest of people in LB, making us eagerly want to know what general orientation they were walking toward to lead to such a giant score gap. We decided to go through the discussion carefully to find some tips or hints to reach a superior score. https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/340043 and a deleted thread claims high score of only using pre-trained model, even like 0.47+, but not claiming the actual name of the pre-trained weights. At the same time we found some thread like [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/338505], indicating their bad performances after training or fine-tuning. So the first idea was assuming maybe @khabel and other contestants with higher ranking just uses pre-trained weights w/o training or fine-tuning. So we decided to collect powerful pretrained weights first. From the https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/340043, we can tell that larger datasets result in better score, since weights trained on ImageNet-22K is better than 1K ones. The best weights of ImageNet-22K gives 0.405, so the 0.47+ one should be trained on much more larger datasets. Searching pre-trained weights on super large datasets, we found a wonderful work CLIP[https://arxiv.org/abs/2103.00020]. For the reproductive code, [https://github.com/mlfoundations/open_clip] was found to be a good choice to begin with. We tried almost every available weights for VIT-L, realizing the best performance come from Laion-400M 31ep, with a score of 0.499. We thought we almost find the most powerful pretrained weight, which still have a gap of like .050 to @khabel Considering notebooks and discussion like https://www.kaggle.com/code/moeinshariatnia/random-vs-avg-max-dim-reduction, we want to find whether some tricks may help. We found that random projection works better for weak weights like efficientnet-b0 trained on ImageNet-1K, but harm the performance of strong weights like VIT-L Laion-400M 31ep. The rest methods like PCA, t-SNE is not working in our experiments. We did find some seemingly robust methods to improve the score w/o training, simply using less values from embedding vectors to calculate the mean will boost ~0.010 on both public LB and private LB. The reason we guess is that calculating average from a large amount of values in embedding will let the real feature get vanished. However It seems that is the best we can get, to reach ~0.510 w/o training and fine-tuning. We need to go further, and since that time we believe @khabel must train on something.
\nTraining is not that easy, not only because many contestants claimed they get much lower score by training, but also the strict rule of this competition forbidding people from using datasets with no commercial use guarantee. We tried our best, and went through https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/337384, but it is still rather hard to find datasets perfectly meet the requirements, and also be believed to help in this competition. Realizing this competition is an extend of the previous series of competitions Google Landmark. We decided to try to first train on GLDv2-clean, unsure if it legal though. Intuitively, the best way may be simply adding a projection layer to downsize the embedding to 64-d, and only fine-tuning the last layer. Since that we used GLDv2, it was reasonable to get to know how the winners of that competition did to reach the best performance. From [https://www.kaggle.com/competitions/landmark-retrieval-2020/discussion/176037] and [https://arxiv.org/abs/2009.05132], we supposed arcface may play a vital role here. So we just went simple first, fine-tuning the last layer using arcface with a margin of 0.5 and s of 30. It was surprising that we reached 0.560 after 6 epochs of fine-tuning, pointing out that we are in the right direction.
\nBut we realized it is hard to move on, if we supposed GLD-v2 is on the boarder line, the rest of datasets is just obvious illegal from our view. We strongly hesitate that most teams, including us, may be disqualified because of datasets issue. We opened a thread here [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/344643], and we glad to see @maggiemd and @andrefaraujo response in time to make any public datasets available as long as they are mentioned in the forum.
\nGetting free from the previous rules, we can try a lot of stuffs to get improved. We always think we should follow this order: Chose datasets -> Decide model, training-related things. So we decided to test various datasets. To maintain a good performance and save time, we added datasets to out training list iteratively instead of training on each datasets from very beginning. We then added Products-10k, Shopee, MET Artwork Dataset, Alibaba goods, H&M Personalized Fashion, GPR1200, GLDv2-Full, DeepFashion - Consumer-to-shop Clothes Retrieval Benchmark part, boosting the score above 0.610. I felt like the score most likely converged since more epochs of fine-tuning did not help improve then. So it is time to do something about the model and training part. I felt like it is hard to do some stuffs about model structure, because a pre-trained weights is very vulnerable to any extra structure added, and even if we come up with a way of that, we don’t think it is a good trade-off regarding computing resources and time.
\nSo moving to the training tricks, we suppose leaving the backbone weights as it is, which was definitely not a good idea. Since now the linear head is well trained, we did not need to worry about the random linear head would affect the backbone weights. So we unfreeze the backbone part, using 10 times lower initial learning rate training the whole part. Just as @anonamename mentioned in [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/359161], we found it is very likely to cause over-fitting, if we also trained on 6 or more epochs as we did on the linear head. Investigating on the phenomenons, we found that when we train on the whole models, the weights of linear projection has jumped sharply. It is weird since linear projection has already converged when we fine-tune on it individually, reminding us of thinking about what is actually the weights of linear projection is.
\nGiven a set of weights of a projection layer F(C, X),
\nwhere C denotes the central embedding of each class from class pool X.
From above, the shaking of the F(C, X) reveals that the central embedding of each classes changed rapidly, which results in the Euclidean distance between classes has been largely changed. The Euclidean distance between each class is meaningful indeed, since it indicates the relationships between each classes. The rapid changing of central embedding may be a regardless-of-the-cost adaption to fit for the training loss. We supposed this was one of the root cause of the over-fitting. So we decided to freeze the final fc layer when train on backbone.
\nAdding dropout to the full connection layer is already a well-known trick to avoid over-fitting of training CNNs or vision transformers. It is not always work in most time, but here we supposed it is worthy to try, because 1) Our model have 2 fc layers, 1 to get embedding in the length of 512, provided originally by VIT, and we added another to downsize it to 64. More fc layers tend not to lose that much information because of dropout. 2) The over-fitting is rather serious here.
\nWe combined the two methods above, with only 3 epochs, and get the model boost to 0.650-0.660.
\nIn the previous training, we found that products-10k is most likely give the most improvement, so we extracted products-10k to fine-tune individually followed the order of “first fc, then backbone”. It reached 0.671.
\nIt seems that it is hard to continue improving, what should kagglers do now:)
\nBut ensemble seems not working in this competition, as mentioned in [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/337360] and [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/339554]. We need to find the reason if we want to solve an issue. Why did ensemble not work here? I fully agree with @manwithaflower ‘s point of view indicated in [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/340546#1876558]. Recalling the stuffs we found about the F(C, X), if we got similar F(C, X) in different models, then the ensemble of those models should work.
\nHow to achieve this? Basically we have two ways:
\nFirst, if we fine-tune models in different hyper-parameters, like model soups [https://arxiv.org/abs/2203.05482], but in the models ensemble style, it will likely work since the F(C, X) won’t change a lot. That is what we tried, and we got boost to 0.680 with ensemble of the resolution 224 and 280.
\nSecond, if we tried to keep F(C, X) the same with a maintaining in performance, we can also get models ensemble work, and even achieve better performance than the first way, because the variance of the models can be huge which help ensemble. The way to did it is based on an
\nassumption that the different spaces represented by F(C, X) of different models can be aligned through a linear operation. Like below:
If we have,
\n y^ = F1(C, X) * B1(W, x), and,
\n y^ = F2(C, X) * B2(W, x),
\nThen ∃ G(C1, C2), such that,
\n y^ = F1(C, X) * G(C1,C2) * B2(W, x)
So all we need is to train a G(*) on the models to do the ensemble. This is what we want to further investigate after this competition.
\nThings is really interesting till here, then a good catch by @dschettler8845 in [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/353768] breaks that. LOL (but it is really an informative catch!)
\nI need to redo EVERY THING MENTIONED ABOVE to the new laion-2b VIT-H model thanks to this weight:(, except several changes: 1) drop model ensemble, VIT-H is really a huge guy 2) train on all the datasets at the same time, drop products-10k, leave products-10k as the final fine-tuning datasets.
\nThen it reaches a score of 0.703. After fine-tuning on 280, it reaches 0.705. Here are some tricks, training again through all the process from last layer to backbone in 280 with less epochs and lower learning rate, then it gives 0.723. Here is an alternative way not to turn higher the resolution, but gives some overlap to the patch. It gives the best performance in the last day to reach public LB with 0.732 and private LB 0.728 with 290 resolution and 4 pixels overlapping.
\nHope this will help someone to better perform in Kaggle Competitions. See you guys in the next comp, Happy Kaggling!
\nGithub repository: https://github.com/LouieShao/1st-Place-Solution-in-Google-Universal-Image-Embedding
\nPaper: https://arxiv.org/abs/2210.08473
Congratulations!
\ntrain on all the datasets at the same time, drop products-10k, leave products-10k as the final fine-tuning datasets\nCan you clarify how you trained on all of the datasets? Did you combine all of them and train considering all of their labels in the ArcFace loss? Or did you maybe train on them one by one, in some particular order? Or did you somehow train on all of them at the same time taking batches from one dataset at a time? Or maybe even something else? :)
\nThanks!
", "votes": 2, "replies": [ { "id": 1997025, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-20T15:21:12.117000", "content": "Hi Andre!
\nI trained on all the datasets all at once, since many papers especially about network adaption claims that it is worth to re-train the whole model after we found the optimal architecture through training. Same things here, but for datasets. We added different datasets to our training list one by one ONLY in the datasets selection process. We only keep datasets that leads to boosting on our score. In this way we got our final list of datasets to be used. After we got this, we re-trained the model on all the datasets to get the best performance, only to leave products-10k to be the last part training.
\nAgain, thanks for hosting such amazing competition! (Will we have a season 2?)
", "votes": 2, "replies": [ { "id": 2129095, "author_name": "Pranav Belhekar", "author_url": "", "post_date": "2023-02-04T10:20:28.683000", "content": "Thanks alot!
", "votes": 0, "replies": [] } ] }, { "id": 1997065, "author_name": "Andre Araujo", "author_url": "", "post_date": "2022-10-20T15:53:45.097000", "content": "Thanks! Got it, so you are really basically concatenating the label sets of each dataset, and training a very large classifier. Very interesting. Would you know what is the resulting number of training images and classes?
\nAnd during training, I guess you are mixing data from all datasets, and shuffling the entire set of images so that each batch could have images from several different domains (is this correct?). Did you encounter any issues where maybe one dataset would be much larger than another, and lead to biasing the embedding too much towards some image types?
\nThanks again!
", "votes": 4, "replies": [] }, { "id": 1997097, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-20T16:29:59.470000", "content": "Would you know what is the resulting number of training images and classes?
\nWill check tommorow when I can reach my laptop. But you can simply summarize them up, since no sampling stuffs happen here.
And during training, I guess you are mixing data from all datasets, and shuffling the entire set of images so that each batch could have images from several different domains (is this correct?)
\nThat’s correct. We sample each mini-batch randomly.
Did you encounter any issues where maybe one dataset would be much larger than another, and lead to biasing the embedding too much towards some image types?
We did not dive into this. But for domains things, nearly 30% of domains we even did not select any datasets, since they are too small to affect score.
\nSo here goes an assumption. If given a dataset A is ‘much larger’ than B, then B should be smaller than like 0.1M. (Since we don’t have datasets like Laion-2B, a datasets is much larger than another only occur when a datasets is super small) And for a super tiny datasets, we don’t care if it got influenced, since it basically has no contribution to the final score. ( maybe a little boost, even in that case we will also first guarantee the big datasets get well trained, regardless the influences to small ones)
", "votes": 2, "replies": [] }, { "id": 1997254, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-20T18:09:17.613000", "content": "And feel free to let me know if we can help in your future work about this:)
", "votes": 1, "replies": [] }, { "id": 2000189, "author_name": "Andre Araujo", "author_url": "", "post_date": "2022-10-23T05:49:57.983000", "content": "Thanks for all the answers, Shihao! Looking forward to your talk in the workshop tomorrow :)
", "votes": 2, "replies": [] } ] }, { "id": 1984873, "author_name": "Tonghui Li", "author_url": "", "post_date": "2022-10-13T00:30:17.097000", "content": "I didn't participate in this competition but I will say this is one of the most comprehensive write-ups I have read. It is most educational to me to learn about how people get around what didn't work and improve, compared to only reading about what works.
\nCongrats and great job!
Thanks for your kind words!
", "votes": 1, "replies": [] } ] }, { "id": 1983834, "author_name": "ConnorKevin", "author_url": "", "post_date": "2022-10-12T09:14:48.070000", "content": "Very brilliant work! I am a new bee in Kaggle, could I know the computing resource this work has cost? I really want to try it but I don't know if I can afford it… thank you very much!
", "votes": 2, "replies": [ { "id": 1983845, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-12T09:22:28.350000", "content": "Sure. Single node with 4 A100 GPUs. The whole training process should cost ~20 days.
", "votes": 0, "replies": [ { "id": 2832978, "author_name": "Kilaru Vasudeva", "author_url": "", "post_date": "2024-05-24T02:15:09.170000", "content": "Did you use cloud GPUs? If yes, which platform? What is the total training hours if you could recall?
\nThank you very much for your work!
", "votes": 0, "replies": [] } ] } ] }, { "id": 1988001, "author_name": "Zi-Yi ZHANG", "author_url": "", "post_date": "2022-10-15T02:01:15.990000", "content": "Congratulations and thanks for your detailed explanation, quite useful for new bees like me😁. One question, when you say
\n\n\nSo we decided to freeze the final fc layer when train on backbone.
\n
Did you freeze the 64-D linear projection layer or the last 512-D fc layer in ViT-L? I got a little confused here.
", "votes": 1, "replies": [ { "id": 1988062, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-15T04:00:24.297000", "content": "Thank you! For your question, both, since they are both linear operation, which can be reduced to one single operation, and there has no non-linear activation between them.
", "votes": 0, "replies": [] } ] }, { "id": 1985307, "author_name": "Zhang Feifan", "author_url": "", "post_date": "2022-10-13T08:20:16.090000", "content": "Congratulations! These details are just like the ConvNeXt article, both are helpful to us new bees.
\nAnd I have a little question it will take a long time to score your submitted model， and what will you do in that period? I just change some parameters and submit five times every day, and then get asleep waiting for another five chances to submit.
I will keep training other idea, since my training time is far longer than 9 hrs. But in most cases, I recommend you to test various idea during that time if you have robust CV.
", "votes": 0, "replies": [] } ] }, { "id": 1985235, "author_name": "Hasbi azi faisya", "author_url": "", "post_date": "2022-10-13T07:35:01.803000", "content": "test my comment
", "votes": 1, "replies": [] }, { "id": 1984242, "author_name": "Will04", "author_url": "", "post_date": "2022-10-12T14:33:51.640000", "content": "Thanks for the detailed explanation, the ensemble part is very educational.
", "votes": 1, "replies": [] }, { "id": 1984215, "author_name": "YNG2020", "author_url": "", "post_date": "2022-10-12T14:17:02.647000", "content": "Great! That is totally brilliant!
", "votes": 1, "replies": [] }, { "id": 1983293, "author_name": "Godsent Abode", "author_url": "", "post_date": "2022-10-12T00:08:54.690000", "content": "Wow, that was very detailed. Congratulations @louieshao Very well deserved!
", "votes": 1, "replies": [] }, { "id": 1983286, "author_name": "Gabriel Lins", "author_url": "", "post_date": "2022-10-11T23:50:52.050000", "content": "Wow! Thanks for such an explanation, that was so rich in detail, I learned a lot reading your approach to this competition.
", "votes": 1, "replies": [] }, { "id": 1983035, "author_name": "Nanashi", "author_url": "", "post_date": "2022-10-11T18:41:45.110000", "content": "interesting @louieshao we also did model soup and other tricks. Looking forward to read your paper :)
", "votes": 1, "replies": [ { "id": 1983418, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-12T03:19:01.627000", "content": "Thanks! I’m working on that, for another 2K words😂
", "votes": 0, "replies": [] } ] }, { "id": 1982816, "author_name": "Ankit Maurya", "author_url": "", "post_date": "2022-10-11T16:36:40.653000", "content": "Congrtas and
\nThanks for wonderfull explanation.
Thank you:D
", "votes": 0, "replies": [] } ] }, { "id": 1982808, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-10-11T16:32:09.790000", "content": "Hearty congratulations for the first place! Thanks for sharing the approach too! I am amazed at the detailing and baroque approach @louieshao!
", "votes": 1, "replies": [ { "id": 1982810, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-11T16:33:09.497000", "content": "Glad to hear that help!
", "votes": 0, "replies": [] } ] }, { "id": 1982777, "author_name": "Starry", "author_url": "", "post_date": "2022-10-11T16:13:19.170000", "content": "Congrats! solid work with so many useful training tricks, theoretical analysis and experiments. I also use the similar fine-tune and frozen training strategy.
", "votes": 1, "replies": [ { "id": 1982781, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-11T16:16:46.410000", "content": "Thank you! See you in the next comp🙂
", "votes": 0, "replies": [] } ] }, { "id": 3305915, "author_name": "Muhammad Ehsan", "author_url": "", "post_date": "2025-10-23T17:46:03.710000", "content": "Many congratulations to all the team 🎉🎉🎉
", "votes": 0, "replies": [] }, { "id": 3024702, "author_name": "LI PEIHANG", "author_url": "", "post_date": "2024-10-21T22:23:53.207000", "content": "congradulations
", "votes": 0, "replies": [] }, { "id": 2836995, "author_name": "Kilaru Vasudeva", "author_url": "", "post_date": "2024-05-26T08:16:22.407000", "content": "I see you have different training scripts like s1… s13. What are these scripts, are they training scripts for different experiments/stages, or do each script correspond to each training data?
", "votes": 0, "replies": [] }, { "id": 1997248, "author_name": "Tarek Atassi", "author_url": "", "post_date": "2022-10-20T18:04:26.173000", "content": "Congratulations! Thanks for sharing this :)
", "votes": 0, "replies": [] }, { "id": 1996953, "author_name": "Himanshu Dongre", "author_url": "", "post_date": "2022-10-20T14:20:56.387000", "content": "Congratulations and Thanks for sharing your amazing thought process. This was a really tricky competition and your post really gave me a lot to think about how to approach such open-ended problems.
", "votes": 0, "replies": [] }, { "id": 1994795, "author_name": "Rahul Pandey", "author_url": "", "post_date": "2022-10-19T08:36:16.127000", "content": "Congratulations! These details are amazing, both are helpful to us new gigs. I thinks it took a lot of ram and time for external large datasets.
", "votes": 0, "replies": [] }, { "id": 1990650, "author_name": "Arpan 1291", "author_url": "", "post_date": "2022-10-16T17:08:24.303000", "content": "Congratulations and thanks for your detailed explanation, quite useful for new bees like me😁. One question, when you say
", "votes": 0, "replies": [] }, { "id": 1990641, "author_name": "Matt OPP", "author_url": "", "post_date": "2022-10-16T17:06:26.920000", "content": "Congratulations and thanks for your detailed explanation, quite useful for new bees like me😁. One question, when you say
", "votes": 0, "replies": [] }, { "id": 1990083, "author_name": "Rahulkishan Gupta", "author_url": "", "post_date": "2022-10-16T10:47:10.470000", "content": "Congratulations And Thanks For the Detailed Explanation.
", "votes": 0, "replies": [] }, { "id": 1989554, "author_name": "Tnsor Boy", "author_url": "", "post_date": "2022-10-16T04:15:43.943000", "content": "osm sir nice work
", "votes": 0, "replies": [] }, { "id": 1988695, "author_name": "Aurora Zhi", "author_url": "", "post_date": "2022-10-15T13:41:24.057000", "content": "Thank you for sharing in this detail! REALLY HELPFUL
", "votes": 0, "replies": [] }, { "id": 1988689, "author_name": "Aurora Zhi", "author_url": "", "post_date": "2022-10-15T13:38:57.067000", "content": "Thank you for sharing the whole procedure, really helpful!
", "votes": 0, "replies": [] }, { "id": 1987269, "author_name": "Balram Agnihotri", "author_url": "", "post_date": "2022-10-14T16:08:25.420000", "content": "Great work!!!
", "votes": 0, "replies": [] }, { "id": 1986361, "author_name": "Shukur A.", "author_url": "", "post_date": "2022-10-14T04:00:06.550000", "content": "Congratulations!
\narticle wiil be helpfull to me in my feature projects
thanks a lot for sharing your work in such a didactic and detailed form.
", "votes": 0, "replies": [] }, { "id": 1985487, "author_name": "", "author_url": "", "post_date": "2022-10-13T11:09:22.927000", "content": "", "votes": 0, "replies": [] }, { "id": 1985454, "author_name": "", "author_url": "", "post_date": "2022-10-13T10:35:01.943000", "content": "", "votes": 0, "replies": [] }, { "id": 2038255, "author_name": "", "author_url": "", "post_date": "2022-11-21T09:36:24.230000", "content": "", "votes": 1, "replies": [] }, { "id": 1990120, "author_name": "", "author_url": "", "post_date": "2022-10-16T11:16:05.253000", "content": "", "votes": 0, "replies": [] }, { "id": 1987397, "author_name": "", "author_url": "", "post_date": "2022-10-14T17:22:14.867000", "content": "", "votes": 0, "replies": [] }, { "id": 1990376, "author_name": "", "author_url": "", "post_date": "2022-10-16T13:50:46.400000", "content": "", "votes": 1, "replies": [] }, { "id": 1986401, "author_name": "", "author_url": "", "post_date": "2022-10-14T05:12:49.300000", "content": "", "votes": 1, "replies": [] }, { "id": 1986246, "author_name": "", "author_url": "", "post_date": "2022-10-14T00:21:40.763000", "content": "", "votes": 1, "replies": [] }, { "id": 1985007, "author_name": "", "author_url": "", "post_date": "2022-10-13T03:27:17.030000", "content": "", "votes": 1, "replies": [] }, { "id": 1984412, "author_name": "", "author_url": "", "post_date": "2022-10-12T16:36:01.637000", "content": "", "votes": 1, "replies": [] }, { "id": 1983360, "author_name": "", "author_url": "", "post_date": "2022-10-12T02:18:16.773000", "content": "", "votes": 1, "replies": [] }, { "id": 2045108, "author_name": "", "author_url": "", "post_date": "2022-11-27T04:58:45.677000", "content": "", "votes": 0, "replies": [] }, { "id": 1994803, "author_name": "", "author_url": "", "post_date": "2022-10-19T08:45:27.843000", "content": "", "votes": 0, "replies": [] }, { "id": 1989616, "author_name": "", "author_url": "", "post_date": "2022-10-16T05:22:33.150000", "content": "", "votes": 0, "replies": [] }, { "id": 1987431, "author_name": "", "author_url": "", "post_date": "2022-10-14T17:33:52.593000", "content": "", "votes": 0, "replies": [] }, { "id": 1987225, "author_name": "", "author_url": "", "post_date": "2022-10-14T15:29:06.840000", "content": "", "votes": 0, "replies": [] }, { "id": 1987120, "author_name": "", "author_url": "", "post_date": "2022-10-14T14:14:11.210000", "content": "", "votes": 0, "replies": [] }, { "id": 1985990, "author_name": "", "author_url": "", "post_date": "2022-10-13T18:29:32.050000", "content": "", "votes": 0, "replies": [] }, { "id": 1985689, "author_name": "", "author_url": "", "post_date": "2022-10-13T13:45:55.040000", "content": "", "votes": 0, "replies": [] }, { "id": 1985344, "author_name": "", "author_url": "", "post_date": "2022-10-13T09:02:50.060000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 64, "messages": [ { "id": 1982755, "postDate": "2022-10-11T15:56:23.377Z", "content": "So the competition is finalized now, and we are so glad to get the 1st place in both public LB and private one. We really appreciate the amazing competition held by the host @andrefaraujo and the support provided by Kaggle staffs @maggiemd Contestants who share their wonderful ideas and resources also help me a lot. I may forget to tag some persons but I want to say you guys did really a wonderful job!
\nHere I will not only share how we get the final performance, but I also go through the whole competition, for what difficulties we faced and what approaches we chose to overcome, in the order of time. I hope it will help people better learn the practical things one may face and alternative ways to get over them in future competitions.
\nHonestly We felt doubtful about joining this competition or not. It is so rare for both of us to see a competition w/o a provided datasets. But we finally took it as a challenge for us to get over this task, and decided to join in (Luckily we did so!).
\nAt the beginning, @khabel and @skyyy93 were way ahead of the rest of people in LB, making us eagerly want to know what general orientation they were walking toward to lead to such a giant score gap. We decided to go through the discussion carefully to find some tips or hints to reach a superior score. https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/340043 and a deleted thread claims high score of only using pre-trained model, even like 0.47+, but not claiming the actual name of the pre-trained weights. At the same time we found some thread like [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/338505], indicating their bad performances after training or fine-tuning. So the first idea was assuming maybe @khabel and other contestants with higher ranking just uses pre-trained weights w/o training or fine-tuning. So we decided to collect powerful pretrained weights first. From the https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/340043, we can tell that larger datasets result in better score, since weights trained on ImageNet-22K is better than 1K ones. The best weights of ImageNet-22K gives 0.405, so the 0.47+ one should be trained on much more larger datasets. Searching pre-trained weights on super large datasets, we found a wonderful work CLIP[https://arxiv.org/abs/2103.00020]. For the reproductive code, [https://github.com/mlfoundations/open_clip] was found to be a good choice to begin with. We tried almost every available weights for VIT-L, realizing the best performance come from Laion-400M 31ep, with a score of 0.499. We thought we almost find the most powerful pretrained weight, which still have a gap of like .050 to @khabel Considering notebooks and discussion like https://www.kaggle.com/code/moeinshariatnia/random-vs-avg-max-dim-reduction, we want to find whether some tricks may help. We found that random projection works better for weak weights like efficientnet-b0 trained on ImageNet-1K, but harm the performance of strong weights like VIT-L Laion-400M 31ep. The rest methods like PCA, t-SNE is not working in our experiments. We did find some seemingly robust methods to improve the score w/o training, simply using less values from embedding vectors to calculate the mean will boost ~0.010 on both public LB and private LB. The reason we guess is that calculating average from a large amount of values in embedding will let the real feature get vanished. However It seems that is the best we can get, to reach ~0.510 w/o training and fine-tuning. We need to go further, and since that time we believe @khabel must train on something.
\nTraining is not that easy, not only because many contestants claimed they get much lower score by training, but also the strict rule of this competition forbidding people from using datasets with no commercial use guarantee. We tried our best, and went through https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/337384, but it is still rather hard to find datasets perfectly meet the requirements, and also be believed to help in this competition. Realizing this competition is an extend of the previous series of competitions Google Landmark. We decided to try to first train on GLDv2-clean, unsure if it legal though. Intuitively, the best way may be simply adding a projection layer to downsize the embedding to 64-d, and only fine-tuning the last layer. Since that we used GLDv2, it was reasonable to get to know how the winners of that competition did to reach the best performance. From [https://www.kaggle.com/competitions/landmark-retrieval-2020/discussion/176037] and [https://arxiv.org/abs/2009.05132], we supposed arcface may play a vital role here. So we just went simple first, fine-tuning the last layer using arcface with a margin of 0.5 and s of 30. It was surprising that we reached 0.560 after 6 epochs of fine-tuning, pointing out that we are in the right direction.
\nBut we realized it is hard to move on, if we supposed GLD-v2 is on the boarder line, the rest of datasets is just obvious illegal from our view. We strongly hesitate that most teams, including us, may be disqualified because of datasets issue. We opened a thread here [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/344643], and we glad to see @maggiemd and @andrefaraujo response in time to make any public datasets available as long as they are mentioned in the forum.
\nGetting free from the previous rules, we can try a lot of stuffs to get improved. We always think we should follow this order: Chose datasets -> Decide model, training-related things. So we decided to test various datasets. To maintain a good performance and save time, we added datasets to out training list iteratively instead of training on each datasets from very beginning. We then added Products-10k, Shopee, MET Artwork Dataset, Alibaba goods, H&M Personalized Fashion, GPR1200, GLDv2-Full, DeepFashion - Consumer-to-shop Clothes Retrieval Benchmark part, boosting the score above 0.610. I felt like the score most likely converged since more epochs of fine-tuning did not help improve then. So it is time to do something about the model and training part. I felt like it is hard to do some stuffs about model structure, because a pre-trained weights is very vulnerable to any extra structure added, and even if we come up with a way of that, we don’t think it is a good trade-off regarding computing resources and time.
\nSo moving to the training tricks, we suppose leaving the backbone weights as it is, which was definitely not a good idea. Since now the linear head is well trained, we did not need to worry about the random linear head would affect the backbone weights. So we unfreeze the backbone part, using 10 times lower initial learning rate training the whole part. Just as @anonamename mentioned in [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/359161], we found it is very likely to cause over-fitting, if we also trained on 6 or more epochs as we did on the linear head. Investigating on the phenomenons, we found that when we train on the whole models, the weights of linear projection has jumped sharply. It is weird since linear projection has already converged when we fine-tune on it individually, reminding us of thinking about what is actually the weights of linear projection is.
\nGiven a set of weights of a projection layer F(C, X),
\nwhere C denotes the central embedding of each class from class pool X.
From above, the shaking of the F(C, X) reveals that the central embedding of each classes changed rapidly, which results in the Euclidean distance between classes has been largely changed. The Euclidean distance between each class is meaningful indeed, since it indicates the relationships between each classes. The rapid changing of central embedding may be a regardless-of-the-cost adaption to fit for the training loss. We supposed this was one of the root cause of the over-fitting. So we decided to freeze the final fc layer when train on backbone.
\nAdding dropout to the full connection layer is already a well-known trick to avoid over-fitting of training CNNs or vision transformers. It is not always work in most time, but here we supposed it is worthy to try, because 1) Our model have 2 fc layers, 1 to get embedding in the length of 512, provided originally by VIT, and we added another to downsize it to 64. More fc layers tend not to lose that much information because of dropout. 2) The over-fitting is rather serious here.
\nWe combined the two methods above, with only 3 epochs, and get the model boost to 0.650-0.660.
\nIn the previous training, we found that products-10k is most likely give the most improvement, so we extracted products-10k to fine-tune individually followed the order of “first fc, then backbone”. It reached 0.671.
\nIt seems that it is hard to continue improving, what should kagglers do now:)
\nBut ensemble seems not working in this competition, as mentioned in [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/337360] and [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/339554]. We need to find the reason if we want to solve an issue. Why did ensemble not work here? I fully agree with @manwithaflower ‘s point of view indicated in [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/340546#1876558]. Recalling the stuffs we found about the F(C, X), if we got similar F(C, X) in different models, then the ensemble of those models should work.
\nHow to achieve this? Basically we have two ways:
\nFirst, if we fine-tune models in different hyper-parameters, like model soups [https://arxiv.org/abs/2203.05482], but in the models ensemble style, it will likely work since the F(C, X) won’t change a lot. That is what we tried, and we got boost to 0.680 with ensemble of the resolution 224 and 280.
\nSecond, if we tried to keep F(C, X) the same with a maintaining in performance, we can also get models ensemble work, and even achieve better performance than the first way, because the variance of the models can be huge which help ensemble. The way to did it is based on an
\nassumption that the different spaces represented by F(C, X) of different models can be aligned through a linear operation. Like below:
If we have,
\n y^ = F1(C, X) * B1(W, x), and,
\n y^ = F2(C, X) * B2(W, x),
\nThen ∃ G(C1, C2), such that,
\n y^ = F1(C, X) * G(C1,C2) * B2(W, x)
So all we need is to train a G(*) on the models to do the ensemble. This is what we want to further investigate after this competition.
\nThings is really interesting till here, then a good catch by @dschettler8845 in [https://www.kaggle.com/competitions/google-universal-image-embedding/discussion/353768] breaks that. LOL (but it is really an informative catch!)
\nI need to redo EVERY THING MENTIONED ABOVE to the new laion-2b VIT-H model thanks to this weight:(, except several changes: 1) drop model ensemble, VIT-H is really a huge guy 2) train on all the datasets at the same time, drop products-10k, leave products-10k as the final fine-tuning datasets.
\nThen it reaches a score of 0.703. After fine-tuning on 280, it reaches 0.705. Here are some tricks, training again through all the process from last layer to backbone in 280 with less epochs and lower learning rate, then it gives 0.723. Here is an alternative way not to turn higher the resolution, but gives some overlap to the patch. It gives the best performance in the last day to reach public LB with 0.732 and private LB 0.728 with 290 resolution and 4 pixels overlapping.
\nHope this will help someone to better perform in Kaggle Competitions. See you guys in the next comp, Happy Kaggling!
\nGithub repository: https://github.com/LouieShao/1st-Place-Solution-in-Google-Universal-Image-Embedding
\nPaper: https://arxiv.org/abs/2210.08473
Congratulations!
\ntrain on all the datasets at the same time, drop products-10k, leave products-10k as the final fine-tuning datasets\nCan you clarify how you trained on all of the datasets? Did you combine all of them and train considering all of their labels in the ArcFace loss? Or did you maybe train on them one by one, in some particular order? Or did you somehow train on all of them at the same time taking batches from one dataset at a time? Or maybe even something else? :)
\nThanks!
", "rawMarkdown": "Congratulations!\n\n```\ntrain on all the datasets at the same time, drop products-10k, leave products-10k as the final fine-tuning datasets\n```\n\nCan you clarify how you trained on all of the datasets? Did you combine all of them and train considering all of their labels in the ArcFace loss? Or did you maybe train on them one by one, in some particular order? Or did you somehow train on all of them at the same time taking batches from one dataset at a time? Or maybe even something else? :)\n\nThanks!", "votes": 2, "replies": [ { "id": 1997025, "postDate": "2022-10-20T15:21:12.117Z", "content": "Hi Andre!
\nI trained on all the datasets all at once, since many papers especially about network adaption claims that it is worth to re-train the whole model after we found the optimal architecture through training. Same things here, but for datasets. We added different datasets to our training list one by one ONLY in the datasets selection process. We only keep datasets that leads to boosting on our score. In this way we got our final list of datasets to be used. After we got this, we re-trained the model on all the datasets to get the best performance, only to leave products-10k to be the last part training.
\nAgain, thanks for hosting such amazing competition! (Will we have a season 2?)
", "rawMarkdown": "Hi Andre! \n\nI trained on all the datasets **all at once**, since many papers especially about network adaption claims **that it is worth to re-train the whole model after we found the optimal architecture through training.** Same things here, but for **datasets**. We added different datasets to our training list one by one ONLY in the datasets selection process. We only keep datasets that leads to boosting on our score. In this way we got our final list of datasets to be used. After we got this, we re-trained the model on all the datasets to get the best performance, only to leave products-10k to be the last part training.\n\nAgain, thanks for hosting such amazing competition! (Will we have a season 2?)", "votes": 2, "replies": [ { "id": 2129095, "postDate": "2023-02-04T10:20:28.683Z", "content": "Thanks alot!
", "rawMarkdown": "Thanks alot!" } ] }, { "id": 1997065, "postDate": "2022-10-20T15:53:45.097Z", "content": "Thanks! Got it, so you are really basically concatenating the label sets of each dataset, and training a very large classifier. Very interesting. Would you know what is the resulting number of training images and classes?
\nAnd during training, I guess you are mixing data from all datasets, and shuffling the entire set of images so that each batch could have images from several different domains (is this correct?). Did you encounter any issues where maybe one dataset would be much larger than another, and lead to biasing the embedding too much towards some image types?
\nThanks again!
", "rawMarkdown": "Thanks! Got it, so you are really basically concatenating the label sets of each dataset, and training a very large classifier. Very interesting. Would you know what is the resulting number of training images and classes?\n\nAnd during training, I guess you are mixing data from all datasets, and shuffling the entire set of images so that each batch could have images from several different domains (is this correct?). Did you encounter any issues where maybe one dataset would be much larger than another, and lead to biasing the embedding too much towards some image types?\n\nThanks again! ", "votes": 4 }, { "id": 1997097, "postDate": "2022-10-20T16:29:59.470Z", "content": "Would you know what is the resulting number of training images and classes?
\nWill check tommorow when I can reach my laptop. But you can simply summarize them up, since no sampling stuffs happen here.
And during training, I guess you are mixing data from all datasets, and shuffling the entire set of images so that each batch could have images from several different domains (is this correct?)
\nThat’s correct. We sample each mini-batch randomly.
Did you encounter any issues where maybe one dataset would be much larger than another, and lead to biasing the embedding too much towards some image types?
We did not dive into this. But for domains things, nearly 30% of domains we even did not select any datasets, since they are too small to affect score.
\nSo here goes an assumption. If given a dataset A is ‘much larger’ than B, then B should be smaller than like 0.1M. (Since we don’t have datasets like Laion-2B, a datasets is much larger than another only occur when a datasets is super small) And for a super tiny datasets, we don’t care if it got influenced, since it basically has no contribution to the final score. ( maybe a little boost, even in that case we will also first guarantee the big datasets get well trained, regardless the influences to small ones)
", "rawMarkdown": "`Would you know what is the resulting number of training images and classes?`\nWill check tommorow when I can reach my laptop. But you can simply summarize them up, since no sampling stuffs happen here. \n\n`And during training, I guess you are mixing data from all datasets, and shuffling the entire set of images so that each batch could have images from several different domains (is this correct?)`\nThat’s correct. We sample each mini-batch randomly. \n\n`Did you encounter any issues where maybe one dataset would be much larger than another, and lead to biasing the embedding too much towards some image types?`\n\nWe did not dive into this. But for domains things, nearly 30% of domains we even did not select any datasets, since they are too small to affect score. \n\nSo here goes an assumption. If given a dataset A is ‘much larger’ than B, then B should be smaller than like 0.1M. (Since we don’t have datasets like Laion-2B, a datasets is much larger than another only occur when a datasets is super small) And for a super tiny datasets, we don’t care if it got influenced, since it basically has no contribution to the final score. ( maybe a little boost, even in that case we will also first guarantee the big datasets get well trained, regardless the influences to small ones)", "votes": 2 }, { "id": 1997254, "postDate": "2022-10-20T18:09:17.613Z", "content": "And feel free to let me know if we can help in your future work about this:)
", "rawMarkdown": "And feel free to let me know if we can help in your future work about this:)", "votes": 1 }, { "id": 2000189, "postDate": "2022-10-23T05:49:57.983Z", "content": "Thanks for all the answers, Shihao! Looking forward to your talk in the workshop tomorrow :)
", "rawMarkdown": "Thanks for all the answers, Shihao! Looking forward to your talk in the workshop tomorrow :)", "votes": 2 } ] }, { "id": 1984873, "postDate": "2022-10-13T00:30:17.097Z", "content": "I didn't participate in this competition but I will say this is one of the most comprehensive write-ups I have read. It is most educational to me to learn about how people get around what didn't work and improve, compared to only reading about what works.
\nCongrats and great job!
Thanks for your kind words!
", "rawMarkdown": "Thanks for your kind words!", "votes": 1 } ] }, { "id": 1983834, "postDate": "2022-10-12T09:14:48.070Z", "content": "Very brilliant work! I am a new bee in Kaggle, could I know the computing resource this work has cost? I really want to try it but I don't know if I can afford it… thank you very much!
", "rawMarkdown": "Very brilliant work! I am a new bee in Kaggle, could I know the computing resource this work has cost? I really want to try it but I don't know if I can afford it... thank you very much!", "votes": 2, "replies": [ { "id": 1983845, "postDate": "2022-10-12T09:22:28.350Z", "content": "Sure. Single node with 4 A100 GPUs. The whole training process should cost ~20 days.
", "rawMarkdown": "Sure. Single node with 4 A100 GPUs. The whole training process should cost ~20 days.", "replies": [ { "id": 2832978, "postDate": "2024-05-24T02:15:09.170Z", "content": "Did you use cloud GPUs? If yes, which platform? What is the total training hours if you could recall?
\nThank you very much for your work!
", "rawMarkdown": "Did you use cloud GPUs? If yes, which platform? What is the total training hours if you could recall?\n\nThank you very much for your work!" } ] } ] }, { "id": 1988001, "postDate": "2022-10-15T02:01:15.990Z", "content": "Congratulations and thanks for your detailed explanation, quite useful for new bees like me😁. One question, when you say
\n\n\nSo we decided to freeze the final fc layer when train on backbone.
\n
Did you freeze the 64-D linear projection layer or the last 512-D fc layer in ViT-L? I got a little confused here.
", "rawMarkdown": "Congratulations and thanks for your detailed explanation, quite useful for new bees like me😁. One question, when you say\n\n> So we decided to freeze the final fc layer when train on backbone.\n\nDid you freeze the 64-D linear projection layer or the last 512-D fc layer in ViT-L? I got a little confused here.", "votes": 1, "replies": [ { "id": 1988062, "postDate": "2022-10-15T04:00:24.297Z", "content": "Thank you! For your question, both, since they are both linear operation, which can be reduced to one single operation, and there has no non-linear activation between them.
", "rawMarkdown": "Thank you! For your question, both, since they are both linear operation, which can be reduced to one single operation, and there has no non-linear activation between them. " } ] }, { "id": 1985307, "postDate": "2022-10-13T08:20:16.090Z", "content": "Congratulations! These details are just like the ConvNeXt article, both are helpful to us new bees.
\nAnd I have a little question it will take a long time to score your submitted model， and what will you do in that period? I just change some parameters and submit five times every day, and then get asleep waiting for another five chances to submit.
I will keep training other idea, since my training time is far longer than 9 hrs. But in most cases, I recommend you to test various idea during that time if you have robust CV.
", "rawMarkdown": "I will keep training other idea, since my training time is far longer than 9 hrs. But in most cases, I recommend you to test various idea during that time if you have robust CV." } ] }, { "id": 1985235, "postDate": "2022-10-13T07:35:01.803Z", "content": "test my comment
", "rawMarkdown": "test my comment", "votes": 1 }, { "id": 1984242, "postDate": "2022-10-12T14:33:51.640Z", "content": "Thanks for the detailed explanation, the ensemble part is very educational.
", "rawMarkdown": "Thanks for the detailed explanation, the ensemble part is very educational.\n\n", "votes": 1 }, { "id": 1984215, "postDate": "2022-10-12T14:17:02.647Z", "content": "Great! That is totally brilliant!
", "rawMarkdown": "Great! That is totally brilliant!", "votes": 1 }, { "id": 1983293, "postDate": "2022-10-12T00:08:54.690Z", "content": "Wow, that was very detailed. Congratulations @louieshao Very well deserved!
", "rawMarkdown": "Wow, that was very detailed. Congratulations @louieshao Very well deserved!", "votes": 1 }, { "id": 1983286, "postDate": "2022-10-11T23:50:52.050Z", "content": "Wow! Thanks for such an explanation, that was so rich in detail, I learned a lot reading your approach to this competition.
", "rawMarkdown": "Wow! Thanks for such an explanation, that was so rich in detail, I learned a lot reading your approach to this competition.", "votes": 1 }, { "id": 1983035, "postDate": "2022-10-11T18:41:45.110Z", "content": "interesting @louieshao we also did model soup and other tricks. Looking forward to read your paper :)
", "rawMarkdown": "interesting @louieshao we also did model soup and other tricks. Looking forward to read your paper :) ", "votes": 1, "replies": [ { "id": 1983418, "postDate": "2022-10-12T03:19:01.627Z", "content": "Thanks! I’m working on that, for another 2K words😂
", "rawMarkdown": "Thanks! I’m working on that, for another 2K words😂" } ] }, { "id": 1982816, "postDate": "2022-10-11T16:36:40.653Z", "content": "Congrtas and
\nThanks for wonderfull explanation.
Thank you:D
", "rawMarkdown": "Thank you:D" } ] }, { "id": 1982808, "postDate": "2022-10-11T16:32:09.790Z", "content": "Hearty congratulations for the first place! Thanks for sharing the approach too! I am amazed at the detailing and baroque approach @louieshao!
", "rawMarkdown": "Hearty congratulations for the first place! Thanks for sharing the approach too! I am amazed at the detailing and baroque approach @louieshao! ", "votes": 1, "replies": [ { "id": 1982810, "postDate": "2022-10-11T16:33:09.497Z", "content": "Glad to hear that help!
", "rawMarkdown": "Glad to hear that help!" } ] }, { "id": 1982777, "postDate": "2022-10-11T16:13:19.170Z", "content": "Congrats! solid work with so many useful training tricks, theoretical analysis and experiments. I also use the similar fine-tune and frozen training strategy.
", "rawMarkdown": "Congrats! solid work with so many useful training tricks, theoretical analysis and experiments. I also use the similar fine-tune and frozen training strategy.", "votes": 1, "replies": [ { "id": 1982781, "postDate": "2022-10-11T16:16:46.410Z", "content": "Thank you! See you in the next comp🙂
", "rawMarkdown": "Thank you! See you in the next comp🙂" } ] }, { "id": 3305915, "postDate": "2025-10-23T17:46:03.710Z", "content": "Many congratulations to all the team 🎉🎉🎉
", "rawMarkdown": "Many congratulations to all the team 🎉🎉🎉" }, { "id": 3024702, "postDate": "2024-10-21T22:23:53.207Z", "content": "congradulations
", "rawMarkdown": "congradulations" }, { "id": 2836995, "postDate": "2024-05-26T08:16:22.407Z", "content": "I see you have different training scripts like s1… s13. What are these scripts, are they training scripts for different experiments/stages, or do each script correspond to each training data?
", "rawMarkdown": "I see you have different training scripts like s1... s13. What are these scripts, are they training scripts for different experiments/stages, or do each script correspond to each training data?" }, { "id": 1997248, "postDate": "2022-10-20T18:04:26.173Z", "content": "Congratulations! Thanks for sharing this :)
", "rawMarkdown": "Congratulations! Thanks for sharing this :)" }, { "id": 1996953, "postDate": "2022-10-20T14:20:56.387Z", "content": "Congratulations and Thanks for sharing your amazing thought process. This was a really tricky competition and your post really gave me a lot to think about how to approach such open-ended problems.
", "rawMarkdown": "Congratulations and Thanks for sharing your amazing thought process. This was a really tricky competition and your post really gave me a lot to think about how to approach such open-ended problems." }, { "id": 1994795, "postDate": "2022-10-19T08:36:16.127Z", "content": "Congratulations! These details are amazing, both are helpful to us new gigs. I thinks it took a lot of ram and time for external large datasets.
", "rawMarkdown": "Congratulations! These details are amazing, both are helpful to us new gigs. I thinks it took a lot of ram and time for external large datasets." }, { "id": 1990650, "postDate": "2022-10-16T17:08:24.303Z", "content": "Congratulations and thanks for your detailed explanation, quite useful for new bees like me😁. One question, when you say
", "rawMarkdown": "Congratulations and thanks for your detailed explanation, quite useful for new bees like me😁. One question, when you say" }, { "id": 1990641, "postDate": "2022-10-16T17:06:26.920Z", "content": "Congratulations and thanks for your detailed explanation, quite useful for new bees like me😁. One question, when you say
", "rawMarkdown": "Congratulations and thanks for your detailed explanation, quite useful for new bees like me😁. One question, when you say" }, { "id": 1990083, "postDate": "2022-10-16T10:47:10.470Z", "content": "Congratulations And Thanks For the Detailed Explanation.
", "rawMarkdown": "Congratulations And Thanks For the Detailed Explanation." }, { "id": 1989554, "postDate": "2022-10-16T04:15:43.943Z", "content": "osm sir nice work
", "rawMarkdown": "osm sir nice work\n\n" }, { "id": 1988695, "postDate": "2022-10-15T13:41:24.057Z", "content": "Thank you for sharing in this detail! REALLY HELPFUL
", "rawMarkdown": "Thank you for sharing in this detail! REALLY HELPFUL" }, { "id": 1988689, "postDate": "2022-10-15T13:38:57.067Z", "content": "Thank you for sharing the whole procedure, really helpful!
", "rawMarkdown": "Thank you for sharing the whole procedure, really helpful!" }, { "id": 1987269, "postDate": "2022-10-14T16:08:25.420Z", "content": "Great work!!!
", "rawMarkdown": "Great work!!!" }, { "id": 1986361, "postDate": "2022-10-14T04:00:06.550Z", "content": "Congratulations!
\narticle wiil be helpfull to me in my feature projects
thanks a lot for sharing your work in such a didactic and detailed form.
", "rawMarkdown": "thanks a lot for sharing your work in such a didactic and detailed form." }, { "id": 1985487, "postDate": "2022-10-13T11:09:22.927Z", "content": "Very Detailed and nice Explanation, Congrats!!
", "rawMarkdown": "Very Detailed and nice Explanation, Congrats!!" }, { "id": 1985454, "postDate": "2022-10-13T10:35:01.943Z", "content": "Wow thanks for the detailed explanation. Learned Something new today.
", "rawMarkdown": "Wow thanks for the detailed explanation. Learned Something new today." }, { "id": 2038255, "postDate": "2022-11-21T09:36:24.230Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1990120, "postDate": "2022-10-16T11:16:05.253Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1987397, "postDate": "2022-10-14T17:22:14.867Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1990376, "postDate": "2022-10-16T13:50:46.400Z", "content": "Congratulations! Thanks for sharing.
", "rawMarkdown": "Congratulations! Thanks for sharing.", "votes": 1 }, { "id": 1986401, "postDate": "2022-10-14T05:12:49.300Z", "content": "Thank you very much for sharing
", "rawMarkdown": "Thank you very much for sharing ", "votes": 1 }, { "id": 1986246, "postDate": "2022-10-14T00:21:40.763Z", "content": "Thanks a lot
", "rawMarkdown": "Thanks a lot", "votes": 1 }, { "id": 1985007, "postDate": "2022-10-13T03:27:17.030Z", "content": "thanks for sharing this
", "rawMarkdown": " thanks for sharing this\n\n", "votes": 1 }, { "id": 1984412, "postDate": "2022-10-12T16:36:01.637Z", "content": "Wow.. thanks for sharing this
", "rawMarkdown": "Wow.. thanks for sharing this ", "votes": 1 }, { "id": 1983360, "postDate": "2022-10-12T02:18:16.773Z", "content": "Very nice explanation, thanks a lot
", "rawMarkdown": "Very nice explanation, thanks a lot ", "votes": 1 }, { "id": 2045108, "postDate": "2022-11-27T04:58:45.677Z", "content": "Thanks for sharing.
\nCongratulations!
Thanks for sharing!🙌
", "rawMarkdown": "Thanks for sharing!🙌" }, { "id": 1989616, "postDate": "2022-10-16T05:22:33.150Z", "content": "Thank you for sharing!!!
", "rawMarkdown": "Thank you for sharing!!!" }, { "id": 1987431, "postDate": "2022-10-14T17:33:52.593Z", "content": "thanks a lot
", "rawMarkdown": "thanks a lot" }, { "id": 1987225, "postDate": "2022-10-14T15:29:06.840Z", "content": "Thanks for the detailed explanation
", "rawMarkdown": "Thanks for the detailed explanation" }, { "id": 1987120, "postDate": "2022-10-14T14:14:11.210Z", "content": "Thank you for your time!
", "rawMarkdown": "Thank you for your time!" }, { "id": 1985990, "postDate": "2022-10-13T18:29:32.050Z", "content": "Thanks for sharing this.
", "rawMarkdown": "Thanks for sharing this." }, { "id": 1985689, "postDate": "2022-10-13T13:45:55.040Z", "content": "thanks for sharing this :)
", "rawMarkdown": "thanks for sharing this :)" }, { "id": 1985344, "postDate": "2022-10-13T09:02:50.060Z", "content": "thanks for this
", "rawMarkdown": "thanks for this\n\n" } ], "complete": true, "source": "live", "fetched_at": "2026-10-24T13:23:46Z", "competition_id": 36414, "competition_slug": "google-universal-image-embedding", "team_id": 8893543, "solution_url": "/competitions/google-universal-image-embedding/writeups/cuilab-ai-1st-place-solution" }