Congrats to winners. Little pity just silver medal.
\nMy solution can be summarized as following:
Distillation can bring 0.015~0.03 LB compared to directly adaptive average pooling.
\nVit-L avgpooling: 0.608, distillation: 0.623
\nVit-L-336 avgpooling: 0.628, distillation: 0.643
\nOpenclip-Vit-H avgpooling: 0.641, distillation: 0.663
\n(all in publice LB)
\n( BUT!!!! Maybe directly trained 64 projection layer also work well, I had no time to do more experiments…☹️). (solved)
\nupdate: code (naive version without collating) available at https://github.com/StarrySky-SHT/GUIE-13th-place-solution
\nupdate: directly train 64 projection layer scored 0.644/654 in public/private LB while distillation scored 0.663/0.666 in public/private LB, so it means knowledge distillation do improve the performance of the model.✔️
congrats @shtljw ! KD is really nice but I feel like really depends on the dataset you use for training, in this case it worked (I also did some similar tests). I think training (learning) directly a 64D representation is better.
", "votes": 1, "replies": [ { "id": 1983393, "author_name": "Starry", "author_url": "", "post_date": "2022-10-12T02:56:47.250000", "content": "Thx for your response, I will do extra experiments and check which solution (training directly 64D/knowledge distillation) is better in my case.
", "votes": 0, "replies": [] } ] }, { "id": 1982879, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-11T17:13:28.437000", "content": "Great works bro! I can bet you are almost the only person who use knowledge distillation. Awesome try! Anyway, congrats for becoming an expert, and only 1 medal to be a master!
", "votes": 1, "replies": [ { "id": 1982888, "author_name": "Starry", "author_url": "", "post_date": "2022-10-11T17:21:17.790000", "content": "Thx. Your solution also impresses me a lot.
\nTake lots of time in past few days with little sleep, time to sleep haha.😴
But it is not knowledge distillation though. Knowledge distillation is taking fewer parameters to produce the same output, while you actually added (256 + 1)x64 parameters to produce different outputs (of lower dimensionality).
", "votes": 1, "replies": [] }, { "id": 1983771, "author_name": "Starry", "author_url": "", "post_date": "2022-10-12T08:17:03.187000", "content": "Yes, strictly speaking, it can't be interpreted as knowledge distillation in this case since I add parameters to the model. But I think the essence of the methods are same, they both use a student model to mimic a teacher model and then get a more powerful student network.
", "votes": 0, "replies": [] }, { "id": 1983870, "author_name": "Danylo Kasianenko", "author_url": "", "post_date": "2022-10-12T09:58:15.103000", "content": "This seems like a regular bottleneck. Because from your description I see no mimicking of the original 256d embeddings structure, just their usage for constructing lower dimensionality embeddings. BTW if you didn't put an activation function between the 256d linear layer and the last 64d, then model representation power is mathematically the same as just using 64d linear layer instead of 256d one.
", "votes": 1, "replies": [] }, { "id": 1983915, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-12T10:45:16.353000", "content": "@kasyanenko As for presentation power part, I think the weights of linear layer that produces 256-d is trained from openai or laion, which should contain pre-training information that make it different from just set a 64-d linear layer. Let me give an example here, when you put GAP after the 1024-d classification vector and the 512-d embedding layer, respectively, the performance should be different that the latter one works better than the former one. The case that their representation is same only occurs when they are both trained from scratch.
", "votes": 1, "replies": [] }, { "id": 1984134, "author_name": "Danylo Kasianenko", "author_url": "", "post_date": "2022-10-12T13:34:34.133000", "content": "What you said is correct, but I meant the case where you have some N features output from the backbone and then you have 2 possible heads:
\nI claim that both scenarios have the same representational power because in the first case though we seemingly have 2 layers it is actually equivalent to having one layer. Because you can perform dot product in any order you can simplify W1 ⋅ W2 to be W3 which would contain all transformations done by the 2 layers in single matrix multiplication and be of the shape [N, 256] x [256, 64] = [N, 64]. The same shape as the W in the second case, so the same representational power.
\nHowever, it is just my assumption that there is no activation layer in between. Anyways, adding a new 64d linear layer indeed may and probably would improve finetuning speed as it uses more high-level features. So even with a random initialization, it would by default demonstrate better performance as has been shown in other discussions in this competition. But in a few finetuning steps, my guess is that they would match their performance because the depth difference of 1 layer in such a huge model isn't really that big.
", "votes": 1, "replies": [] }, { "id": 1984151, "author_name": "Shihao Shao", "author_url": "", "post_date": "2022-10-12T13:43:15.703000", "content": "@kasyanenko I agree your idea for the upper limit. It is obvious series connection of linear operation can be simplized as single one. You are right they will be almost same since only one layer difference between them. Only converging speed can be another story.
", "votes": 1, "replies": [] }, { "id": 1984183, "author_name": "Starry", "author_url": "", "post_date": "2022-10-12T13:56:58.100000", "content": "The mimic I mentioned above, means that the process of knowledge distillation is a mimicry process between student model and teacher model. My understanding is that we can use the KD to make the student model get the comparable performance with original high dimension (256 or larger) even its embed dimension is lower (64). Refer you read the RKD paper. And as for the linear layer you mention, I agree with you, I just simply add a fc layer without thinking too much.
", "votes": 0, "replies": [] } ] }, { "id": 1984237, "author_name": "Xiao", "author_url": "", "post_date": "2022-10-12T14:31:36.490000", "content": "Now I kind of understand why bn doesn't work in my experiments! Very impressive solution, thanks!
", "votes": 0, "replies": [ { "id": 1984955, "author_name": "Starry", "author_url": "", "post_date": "2022-10-13T02:16:04.783000", "content": "Pleasure it helps！
", "votes": 0, "replies": [] } ] } ], "comment_count": 13, "messages": [ { "id": 1982857, "postDate": "2022-10-11T17:02:23.050Z", "content": "Congrats to winners. Little pity just silver medal.
\nMy solution can be summarized as following:
Distillation can bring 0.015~0.03 LB compared to directly adaptive average pooling.
\nVit-L avgpooling: 0.608, distillation: 0.623
\nVit-L-336 avgpooling: 0.628, distillation: 0.643
\nOpenclip-Vit-H avgpooling: 0.641, distillation: 0.663
\n(all in publice LB)
\n( BUT!!!! Maybe directly trained 64 projection layer also work well, I had no time to do more experiments…☹️). (solved)
\nupdate: code (naive version without collating) available at https://github.com/StarrySky-SHT/GUIE-13th-place-solution
\nupdate: directly train 64 projection layer scored 0.644/654 in public/private LB while distillation scored 0.663/0.666 in public/private LB, so it means knowledge distillation do improve the performance of the model.✔️
congrats @shtljw ! KD is really nice but I feel like really depends on the dataset you use for training, in this case it worked (I also did some similar tests). I think training (learning) directly a 64D representation is better.
", "rawMarkdown": "congrats @shtljw ! KD is really nice but I feel like really depends on the dataset you use for training, in this case it worked (I also did some similar tests). I think training (learning) directly a 64D representation is better.", "votes": 1, "replies": [ { "id": 1983393, "postDate": "2022-10-12T02:56:47.250Z", "content": "Thx for your response, I will do extra experiments and check which solution (training directly 64D/knowledge distillation) is better in my case.
", "rawMarkdown": "Thx for your response, I will do extra experiments and check which solution (training directly 64D/knowledge distillation) is better in my case." } ] }, { "id": 1982879, "postDate": "2022-10-11T17:13:28.437Z", "content": "Great works bro! I can bet you are almost the only person who use knowledge distillation. Awesome try! Anyway, congrats for becoming an expert, and only 1 medal to be a master!
", "rawMarkdown": "Great works bro! I can bet you are almost the only person who use knowledge distillation. Awesome try! Anyway, congrats for becoming an expert, and only 1 medal to be a master!", "votes": 1, "replies": [ { "id": 1982888, "postDate": "2022-10-11T17:21:17.790Z", "content": "Thx. Your solution also impresses me a lot.
\nTake lots of time in past few days with little sleep, time to sleep haha.😴
But it is not knowledge distillation though. Knowledge distillation is taking fewer parameters to produce the same output, while you actually added (256 + 1)x64 parameters to produce different outputs (of lower dimensionality).
", "rawMarkdown": "But it is not knowledge distillation though. Knowledge distillation is taking fewer parameters to produce the same output, while you actually added (256 + 1)x64 parameters to produce different outputs (of lower dimensionality).", "votes": 1 }, { "id": 1983771, "postDate": "2022-10-12T08:17:03.187Z", "content": "Yes, strictly speaking, it can't be interpreted as knowledge distillation in this case since I add parameters to the model. But I think the essence of the methods are same, they both use a student model to mimic a teacher model and then get a more powerful student network.
", "rawMarkdown": "Yes, strictly speaking, it can't be interpreted as knowledge distillation in this case since I add parameters to the model. But I think the essence of the methods are same, they both use a student model to mimic a teacher model and then get a more powerful student network." }, { "id": 1983870, "postDate": "2022-10-12T09:58:15.103Z", "content": "This seems like a regular bottleneck. Because from your description I see no mimicking of the original 256d embeddings structure, just their usage for constructing lower dimensionality embeddings. BTW if you didn't put an activation function between the 256d linear layer and the last 64d, then model representation power is mathematically the same as just using 64d linear layer instead of 256d one.
", "rawMarkdown": "This seems like a regular bottleneck. Because from your description I see no mimicking of the original 256d embeddings structure, just their usage for constructing lower dimensionality embeddings. BTW if you didn't put an activation function between the 256d linear layer and the last 64d, then model representation power is mathematically the same as just using 64d linear layer instead of 256d one.", "votes": 1 }, { "id": 1983915, "postDate": "2022-10-12T10:45:16.353Z", "content": "@kasyanenko As for presentation power part, I think the weights of linear layer that produces 256-d is trained from openai or laion, which should contain pre-training information that make it different from just set a 64-d linear layer. Let me give an example here, when you put GAP after the 1024-d classification vector and the 512-d embedding layer, respectively, the performance should be different that the latter one works better than the former one. The case that their representation is same only occurs when they are both trained from scratch.
", "rawMarkdown": "@kasyanenko As for presentation power part, I think the weights of linear layer that produces 256-d is trained from openai or laion, which should contain pre-training information that make it different from just set a 64-d linear layer. Let me give an example here, when you put GAP after the 1024-d classification vector and the 512-d embedding layer, respectively, the performance should be different that the latter one works better than the former one. The case that their representation is same only occurs when they are both trained from scratch.", "votes": 1 }, { "id": 1984134, "postDate": "2022-10-12T13:34:34.133Z", "content": "What you said is correct, but I meant the case where you have some N features output from the backbone and then you have 2 possible heads:
\nI claim that both scenarios have the same representational power because in the first case though we seemingly have 2 layers it is actually equivalent to having one layer. Because you can perform dot product in any order you can simplify W1 ⋅ W2 to be W3 which would contain all transformations done by the 2 layers in single matrix multiplication and be of the shape [N, 256] x [256, 64] = [N, 64]. The same shape as the W in the second case, so the same representational power.
\nHowever, it is just my assumption that there is no activation layer in between. Anyways, adding a new 64d linear layer indeed may and probably would improve finetuning speed as it uses more high-level features. So even with a random initialization, it would by default demonstrate better performance as has been shown in other discussions in this competition. But in a few finetuning steps, my guess is that they would match their performance because the depth difference of 1 layer in such a huge model isn't really that big.
", "rawMarkdown": "What you said is correct, but I meant the case where you have some N features output from the backbone and then you have 2 possible heads:\n1. Original 256-dimensional linear layer and then you immediately (without activation) add another 64-dimensional layer to match the competition restriction. The function they calculate can be represented as X ⋅ W1 ⋅ W2, where X is the backbone features of shape [N], W1 is of shape [N, 256] and W2 is of shape [256, 64].\n2. Replace the original 256-dimensional head with a 64-dimensional linear layer. The function it calculates is X ⋅ W, where X is the backbone features of shape [N] and W is of shape **[N, 64]**.\n\nI claim that both scenarios have the same representational power because in the first case though we seemingly have 2 layers it is actually equivalent to having one layer. Because you can perform dot product in any order you can simplify W1 ⋅ W2 to be W3 which would contain all transformations done by the 2 layers in single matrix multiplication and be of the shape [N, 256] x [256, 64] = **[N, 64]**. The same shape as the W in the second case, so the same representational power. \n\nHowever, it is just my assumption that there is no activation layer in between. Anyways, adding a new 64d linear layer indeed may and probably would improve finetuning speed as it uses more high-level features. So even with a random initialization, it would by default demonstrate better performance as has been shown in other discussions in this competition. But in a few finetuning steps, my guess is that they would match their performance because the depth difference of 1 layer in such a huge model isn't really that big.", "votes": 1 }, { "id": 1984151, "postDate": "2022-10-12T13:43:15.703Z", "content": "@kasyanenko I agree your idea for the upper limit. It is obvious series connection of linear operation can be simplized as single one. You are right they will be almost same since only one layer difference between them. Only converging speed can be another story.
", "rawMarkdown": "@kasyanenko I agree your idea for the upper limit. It is obvious series connection of linear operation can be simplized as single one. You are right they will be almost same since only one layer difference between them. Only converging speed can be another story.", "votes": 1 }, { "id": 1984183, "postDate": "2022-10-12T13:56:58.100Z", "content": "The mimic I mentioned above, means that the process of knowledge distillation is a mimicry process between student model and teacher model. My understanding is that we can use the KD to make the student model get the comparable performance with original high dimension (256 or larger) even its embed dimension is lower (64). Refer you read the RKD paper. And as for the linear layer you mention, I agree with you, I just simply add a fc layer without thinking too much.
", "rawMarkdown": "The mimic I mentioned above, means that the process of knowledge distillation is a mimicry process between student model and teacher model. My understanding is that we can use the KD to make the student model get the comparable performance with original high dimension (256 or larger) even its embed dimension is lower (64). Refer you read the RKD paper. And as for the linear layer you mention, I agree with you, I just simply add a fc layer without thinking too much." } ] }, { "id": 1984237, "postDate": "2022-10-12T14:31:36.490Z", "content": "Now I kind of understand why bn doesn't work in my experiments! Very impressive solution, thanks!
", "rawMarkdown": "Now I kind of understand why bn doesn't work in my experiments! Very impressive solution, thanks!", "replies": [ { "id": 1984955, "postDate": "2022-10-13T02:16:04.783Z", "content": "Pleasure it helps！
", "rawMarkdown": "Pleasure it helps！" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-24T13:23:57Z", "competition_id": 36414, "competition_slug": "google-universal-image-embedding", "team_id": 9076098, "solution_url": "/competitions/google-universal-image-embedding/writeups/little-pity-just-silver-medal-13-th-place-solution" }