Congrats to all competitors, and although it had a strange ending, I am confident that some techniques of this competition can be re-used in future.
\nMy solution is based on a two-stage approach utilizing NLP deep learning models. The first-stage model is a metric learning model with ArcFace loss trained on the original input data predicting the point-of-interest of a record and consequently putting similar records into a similar embedding space. The second stage additionally trains a bi-encoder NLP model based on the proposed candidates from the first stage. The final sub is a blend of both first- and second-stage models.
\nIn the first stage, I fit ArcFace model on most of the input columns separated by SEP tokens. I directly feed the longitude and latitude into the string but split it up every 3rd digit so that the tokenizers can work better with it. A sample record looks like follows:
\n<s> Tesco HQ</s></s> Offices</s></s> 18 963 587</s></s> 47 450 562</s></s> 2040</s></s> Budaörs</s></s> Kinizsi út 1-3.</s>
If a field was NAN, I just filled in an empty string, and no special tokens or other alterations were helpful.
\nIt was still quite tricky to get ArcFace models to work, and some effort was necessary to get the training to stabilize and converge. I only rely on xlm-roberta-large models here. In the beginning I experimented with Triplet-like models, which were easier to get going, but ArcFace was clearly superior after making these models work.
\nI then calculate all-pairs similarity and cut-off below a certain threshold. All pairs above the threshold make it into the second stage and we also keep the predicted probability for blending.
\nFor training, I use in-sample results from the first stage as candidate pairs. I use various bi-encoder models to train for TP/FP of pairs. A record looks like follows:
\n<s> Tesco HQ</s></s> Novo Nordisk</s></s> 18 963 587</s></s> 12 449 657</s></s> 47 450 562</s></s> 55 751 928</s></s> Offices</s></s> Offices</s></s> HU</s></s> DK</s></s> Pest</s></s> Bagsværd</s></s> Budaörs</s></s> Bagsværd</s></s> 2040</s></s> 2880</s></s> Kinizsi út 1-3.</s></s> Novo Allé</s></s></s></s></s></s></s></s>
So pasting each column of both records next to each other and then training a bi-encoder model. I always added all TPs to the training data, and then increasingly added more FPs if time allowed.
\nThe test set in this competition contained ~600k records, and the whole training data contained approximately twice the data. Retrieval/Recognition type problems are always heavily dependent on the list of candidates to choose from, so naturally having the possibility of more unknown candidates will deteriorate the score.
\nHence, the most reasonable CV setup was to split out 600k records with unique points of interest as a validation set, and train models on the remaining data. When doing so I saw good correlation between CV and LB however there is one big issue with this approach, namely that you miss training models on half the data. Consequently, I decided around six weeks before competition deadline to only evaluate on public LB which from my point of view was the best set to evaluate my models due to the following reasons:
\nUnfortunately, I only learned after competition ended, that the test data has a leak that was already discussed thoroughly in the forums. So I definitely biased my model choices towards that leak (e.g., longer training, more overfitting - which definitely works very well with my data and model setup). However, I still made sure that longer training would not hurt my CV score. Yet, I stopped trying to explicitly optimize my CV score, and in retrospective this is a bit unfortunate as I am fully convinced that my general approach could further be improved by working towards a different direction.
\nI blend both probabilities from first and second stage to get the final prediction and cut-off by a certain threshold. I applied some minor QE post processing, but not much was helping additionally.
\nI personally never suspected the leak, due to the dataset description and nature of the problem, so I also never actively searched for it. That said, I also noticed that longer training leads to better LB, now in retrospective I understand why. I was rather suspecting that the models learn / overfit the individual datasets (there are different synthetic data sources) which can help the matching tremendously as it reduces the possible candidates.
\nThe post-analysis revealed that my solution also scores very competitively on non-overlapping data which shows how well my pipeline can generalize to both scenarios.
I still learned a lot in this competition and am very happy to have solved it with deep learning only. It is most fun to me personally, and I am sure this general approach will come in handy again in the future.
", "comments": [ { "id": 1862389, "author_name": "dropout", "author_url": "", "post_date": "2022-07-19T16:46:37.513000", "content": "Impressive solution. One question: What do you mean by \"calculate all-pairs similarity\"? How is that implemented? Do you use ANN (Approximate Nearest Neighbor) to find neighbors? Thanks in advance.
", "votes": 2, "replies": [ { "id": 1862410, "author_name": "Psi", "author_url": "", "post_date": "2022-07-19T17:02:10.243000", "content": "I am calculating it directly in Pytorch on GPU in batch-wise fashion on the embedding output from the ArcFace model. It is very quick.
", "votes": 0, "replies": [] }, { "id": 1862430, "author_name": "dropout", "author_url": "", "post_date": "2022-07-19T17:14:23.600000", "content": "Still kinda confused. Do you mean do sim-score calculations 600k*600k times and you implement that in pytorch in batch-manner?
", "votes": 0, "replies": [] }, { "id": 1862467, "author_name": "Psi", "author_url": "", "post_date": "2022-07-19T18:01:03.587000", "content": "You split the source in batches, let's say 32, and keep the target full. So you calculate 32x600k for 600k/32 times. It is very fast. Let me know if it doesn't make sense and I will try to elaborate more detailed.
", "votes": 1, "replies": [] }, { "id": 1862704, "author_name": "Bilzard", "author_url": "", "post_date": "2022-07-19T23:01:20.893000", "content": "I am also wondering how distance(=similarity) is calculated. Since ArcFace’s embedding vectors are point-wise, we should calculate distance pair-wise, which needs 600K x 600K = 360G samples (suppose we don’t use kNN search). I think this number of samples are intractable.
", "votes": 0, "replies": [] }, { "id": 1862714, "author_name": "dropout", "author_url": "", "post_date": "2022-07-19T23:19:12.917000", "content": "It is more than clear now. I have heard of similar candidate-retrieval pipeline in recommendation systems where people do brute-force metric calculation on all pairs instead of ANN (Approximate Nearest Neighbor), thus to avoid possible precision loss, and all that are done real-time at query. At first I just did not believe it is computationally feasible. We should give it a try 👀
", "votes": 1, "replies": [] }, { "id": 1862719, "author_name": "dropout", "author_url": "", "post_date": "2022-07-19T23:36:42.210000", "content": "@tatamikenn I think the total number of sim-score calculation is indeed 360G. It is still computationally feasible though, since in every single sample pair only vector product calculation is needed (no model inference needed). And that could be further batched into matrix calculation.
", "votes": 1, "replies": [] }, { "id": 1862739, "author_name": "dropout", "author_url": "", "post_date": "2022-07-20T00:36:36.250000", "content": "@tatamikenn try this code snippet, where I simulate the sim-score calculation in batch on GPU:
\nimport numpy as np\nimport torch as torch\ndef mock_batch_distance(batch_size=256,embedding_dimension=384):\n matrix1 = torch.rand(batch_size,embedding_dimension,dtype=torch.float16,device='cuda')\n matrix2 = torch.rand(batch_size,embedding_dimension,dtype=torch.float16,device='cuda')\n result = matrix1.matmul(matrix2.T)\n return result\n%timeit mock_batch_distance(batch_size=256,embedding_dimension=384)\n\n\nOut [1]: 46.3 µs ± 980 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
\n
You see that for each batch-batch distance calculation, only 46.3 µs is needed. And the total time spent should be:
\nbatch_size = 256\ntime_per_function_call = 46/1000000\nnum_total_function_call = (600000/batch_size) * (600000/batch_size)\ntotal_time = num_total_function_call * time_per_function_call\ntotal_time\n\n\nOut [2]: 252.685546875
\n
252s, just a few minutes.
", "votes": 1, "replies": [] }, { "id": 1862745, "author_name": "Bilzard", "author_url": "", "post_date": "2022-07-20T01:10:26.703000", "content": "@peltor Thanks for simulation. Now my question is clear.
", "votes": 0, "replies": [] }, { "id": 1862748, "author_name": "Bilzard", "author_url": "", "post_date": "2022-07-20T01:22:00.093000", "content": "I found a mistake on the simulation.
\n\n\n(matrix1*matrix2).sum(axis=-1)
\n
This gives us only batch_size samples (corresponds with self pair). What we want is batch_size x batch_size matrix.
\nHowever, I'm thinking it's more likely to be feasible even if in the fixed simulation.
You are right, I have modified my code. The result indicates no difference though.
", "votes": 0, "replies": [] }, { "id": 1862773, "author_name": "Bilzard", "author_url": "", "post_date": "2022-07-20T02:07:29.210000", "content": "I tried the fixed version. It was quite feasible (19.3 sec for 360G pairs).
\ndef mock_batch_distance(generator, batch_size, embedding_dimension):\n embeddings1 = torch.rand(\n batch_size,\n embedding_dimension,\n dtype=torch.float32,\n device=\"cuda\",\n requires_grad=False,\n generator=generator,\n )\n embeddings2 = torch.rand(\n batch_size,\n embedding_dimension,\n dtype=torch.float32,\n device=\"cuda\",\n requires_grad=False,\n generator=generator,\n )\n result = embeddings1.matmul(embeddings2.T)\n return result\n\n\ngenerator = torch.Generator(device=\"cuda\")\ngenerator.manual_seed(123)\n%timeit -n 100 mock_batch_distance(generator, batch_size=1024 * 32, embedding_dimension=384)\n\n>>> The slowest run took 3287.43 times longer than the fastest. This could mean that an intermediate result is being cached.\n>>> 57.4 ms ± 41.9 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)\nbatch_size = 1024 * 32\ntime_per_function_call = 57.6 * 0.001\nnum_total_function_call = (600_000 / batch_size) * (600_000 / batch_size)\ntotal_time = num_total_function_call * time_per_function_call\ntotal_time\n>>> 19.311904907226566\nc.f. https://www.kaggle.com/code/tatamikenn/4sq-simulation-of-3rd-solution?scriptVersionId=101281202
", "votes": 1, "replies": [] }, { "id": 1862780, "author_name": "Bilzard", "author_url": "", "post_date": "2022-07-20T02:15:25.617000", "content": "\n\nI took that into consideration in num_total_function_call, where i do square calculation
\n
No, since the first version calculates batch_size element-wise multiplication, the total calculation result gives (600_000/batch_size) * (600_000/batch_size) * batch_size pairs, which isn't equal to 600K x 600K.
Besides, computation in CUDA should be different between item-wise multiplication and matrix multiplication. I think we should use matrix multiplication when we simulate exactly (suppose the author used it).
", "votes": 1, "replies": [] }, { "id": 1863088, "author_name": "Psi", "author_url": "", "post_date": "2022-07-20T06:36:41.183000", "content": "# embeddings 600k x embedding dimension - already normalized\nchunk = 512\nembeddings_chunks = embeddings.split(chunk)\n\nfor idx in range(len(embeddings_chunks)):\n cos_sim_chunk = torch.mm(embeddings_chunks[idx], embeddings.transpose(0, 1))\n # here you will only want to keep the topk indices in memory for further use\nBTW: it is crazy how far we have got. I remember struggling with this problem on tfidf features ~10 years ago. It was sheer impossible to do on CPU and memory restrictions.
", "votes": 8, "replies": [] } ] }, { "id": 1861868, "author_name": "charmq", "author_url": "", "post_date": "2022-07-19T09:57:41.927000", "content": "Your solution using ArcFace is really impressive.
\nIt seems to demonstrate the broad applicability of ArcFace.
\nIn fact, my team gave up using ArcFace due to the difficulty of tuning.
\nAnd, congratulations on your win!
I also nearly gave up on it, and I still wasted many runs where the loss exploded. It is very hard to tune in general and needs some patience. I started with ArcFace, couldnt make it work, and switched to TripletLoss which worked. So I thought it has to work and went back to ArcFace.
", "votes": 7, "replies": [] } ] }, { "id": 1861866, "author_name": "213tubo", "author_url": "", "post_date": "2022-07-19T09:55:37.860000", "content": "Thank you for sharing your solution.
\nYour deep learning approach is very impressive.
\nI respect your solution with good rank in both overlap and no-overlap.
I could not succeed with ArcFace. So I would like to read your training code.
\nDo you plan to share your code?
There is no magic in the code that would be interesting. Rather certain training tricks to make the training stable such as: long warmup, playing with the neck of the model, trying different margins and scale combinations, etc.
", "votes": 12, "replies": [] }, { "id": 1863214, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2022-07-20T08:09:33.803000", "content": "@philippsinger did differential learning rate work? or some other trick that really helped
\nand did you use any type of augmentations for the texts?
Another query:
\nFor the Bidirectional Encoder,(really impressive trick) you concatenated the sequences I guess it was a binary classification task? where the sequences having more cosine similarity (than some threshold) is considered label 1 else 0. Is it like that?
Diff LR was also helpful, yes
\nIt was a binary classification task, and label 1 or 0 was defined by ground truth data. And I extracted the candidates from in-sample similarities.
", "votes": 0, "replies": [] } ] }, { "id": 1861983, "author_name": "colum2131", "author_url": "", "post_date": "2022-07-19T11:23:56.030000", "content": "Thank you for sharing your solution. It was very inspiring!
\nI have one question.
\nThere were many poi with only one class size in this competition data.
\nIn training ArcFace, did you do anything special to the data whose class size was 1?
I tried quite a few things with loss weighting, dynamic margins, and other things, but in the end I did not do anything special with them as overall the class distribution (in terms of nr of records per class) was not too diverse and just leaving it as-is was best or at least not worse.
", "votes": 3, "replies": [] } ] }, { "id": 2132283, "author_name": "Trushant Kalyanpur", "author_url": "", "post_date": "2023-02-06T18:02:06.723000", "content": "Great work @philippsinger ! Did you use arcface as a loss function or incorporate something like ArcMarginProduct as part of the model arch?
", "votes": 0, "replies": [] }, { "id": 1864016, "author_name": "Bilal Suppal", "author_url": "", "post_date": "2022-07-20T17:00:30.457000", "content": "Congrats! thanks for sharing👍 @philippsinger
", "votes": 0, "replies": [] }, { "id": 1862676, "author_name": "Bilzard", "author_url": "", "post_date": "2022-07-19T22:22:21.737000", "content": "Shanks for sharing.
\nI’m interested in what is the efficiency of first stage, i.e. recall and positive sample ratio.
", "votes": 0, "replies": [ { "id": 1863217, "author_name": "Psi", "author_url": "", "post_date": "2022-07-20T08:11:20.257000", "content": "I will have to check on old models on cv, I don't remember it fully. But it was somewhere in range of 0.97 recall maybe? But by taking more I could get higher.
\nWhat I want to mention is that the whole solution is quite quick. A single first stage + single second stage model scores in 90 minutes on LB with 600k records.
", "votes": 3, "replies": [] } ] }, { "id": 1862570, "author_name": "user315", "author_url": "", "post_date": "2022-07-19T19:52:41.673000", "content": "Well done!
", "votes": 0, "replies": [] }, { "id": 1861799, "author_name": "Gerwyn", "author_url": "", "post_date": "2022-07-19T08:50:56.323000", "content": "Congrats on your win!
\n\n\nThe first-stage model is a metric learning model with ArcFace loss trained on the original input data predicting the point-of-interest of a record
\n
does this mean you had ~740k classes in last dense layer, using the whole training data?
\n(asking as i did not have success training arcface model due to the large number of classes)
", "votes": 0, "replies": [ { "id": 1861808, "author_name": "Psi", "author_url": "", "post_date": "2022-07-19T08:54:06.970000", "content": "ArcFace is explicitly tailored towards large number of classes. You have this dimension in the margin product of ArcFace.
", "votes": 0, "replies": [] } ] } ], "comment_count": 30, "messages": [ { "id": 1861774, "postDate": "2022-07-19T08:34:27.747Z", "content": "Congrats to all competitors, and although it had a strange ending, I am confident that some techniques of this competition can be re-used in future.
\nMy solution is based on a two-stage approach utilizing NLP deep learning models. The first-stage model is a metric learning model with ArcFace loss trained on the original input data predicting the point-of-interest of a record and consequently putting similar records into a similar embedding space. The second stage additionally trains a bi-encoder NLP model based on the proposed candidates from the first stage. The final sub is a blend of both first- and second-stage models.
\nIn the first stage, I fit ArcFace model on most of the input columns separated by SEP tokens. I directly feed the longitude and latitude into the string but split it up every 3rd digit so that the tokenizers can work better with it. A sample record looks like follows:
\n<s> Tesco HQ</s></s> Offices</s></s> 18 963 587</s></s> 47 450 562</s></s> 2040</s></s> Budaörs</s></s> Kinizsi út 1-3.</s>
If a field was NAN, I just filled in an empty string, and no special tokens or other alterations were helpful.
\nIt was still quite tricky to get ArcFace models to work, and some effort was necessary to get the training to stabilize and converge. I only rely on xlm-roberta-large models here. In the beginning I experimented with Triplet-like models, which were easier to get going, but ArcFace was clearly superior after making these models work.
\nI then calculate all-pairs similarity and cut-off below a certain threshold. All pairs above the threshold make it into the second stage and we also keep the predicted probability for blending.
\nFor training, I use in-sample results from the first stage as candidate pairs. I use various bi-encoder models to train for TP/FP of pairs. A record looks like follows:
\n<s> Tesco HQ</s></s> Novo Nordisk</s></s> 18 963 587</s></s> 12 449 657</s></s> 47 450 562</s></s> 55 751 928</s></s> Offices</s></s> Offices</s></s> HU</s></s> DK</s></s> Pest</s></s> Bagsværd</s></s> Budaörs</s></s> Bagsværd</s></s> 2040</s></s> 2880</s></s> Kinizsi út 1-3.</s></s> Novo Allé</s></s></s></s></s></s></s></s>
So pasting each column of both records next to each other and then training a bi-encoder model. I always added all TPs to the training data, and then increasingly added more FPs if time allowed.
\nThe test set in this competition contained ~600k records, and the whole training data contained approximately twice the data. Retrieval/Recognition type problems are always heavily dependent on the list of candidates to choose from, so naturally having the possibility of more unknown candidates will deteriorate the score.
\nHence, the most reasonable CV setup was to split out 600k records with unique points of interest as a validation set, and train models on the remaining data. When doing so I saw good correlation between CV and LB however there is one big issue with this approach, namely that you miss training models on half the data. Consequently, I decided around six weeks before competition deadline to only evaluate on public LB which from my point of view was the best set to evaluate my models due to the following reasons:
\nUnfortunately, I only learned after competition ended, that the test data has a leak that was already discussed thoroughly in the forums. So I definitely biased my model choices towards that leak (e.g., longer training, more overfitting - which definitely works very well with my data and model setup). However, I still made sure that longer training would not hurt my CV score. Yet, I stopped trying to explicitly optimize my CV score, and in retrospective this is a bit unfortunate as I am fully convinced that my general approach could further be improved by working towards a different direction.
\nI blend both probabilities from first and second stage to get the final prediction and cut-off by a certain threshold. I applied some minor QE post processing, but not much was helping additionally.
\nI personally never suspected the leak, due to the dataset description and nature of the problem, so I also never actively searched for it. That said, I also noticed that longer training leads to better LB, now in retrospective I understand why. I was rather suspecting that the models learn / overfit the individual datasets (there are different synthetic data sources) which can help the matching tremendously as it reduces the possible candidates.
\nThe post-analysis revealed that my solution also scores very competitively on non-overlapping data which shows how well my pipeline can generalize to both scenarios.
I still learned a lot in this competition and am very happy to have solved it with deep learning only. It is most fun to me personally, and I am sure this general approach will come in handy again in the future.
", "rawMarkdown": "Congrats to all competitors, and although it had a strange ending, I am confident that some techniques of this competition can be re-used in future.\n\n### Summary\n\nMy solution is based on a two-stage approach utilizing NLP deep learning models. The first-stage model is a metric learning model with ArcFace loss trained on the original input data predicting the point-of-interest of a record and consequently putting similar records into a similar embedding space. The second stage additionally trains a bi-encoder NLP model based on the proposed candidates from the first stage. The final sub is a blend of both first- and second-stage models.\n\n### First stage: ArcFace\n\nIn the first stage, I fit ArcFace model on most of the input columns separated by SEP tokens. I directly feed the longitude and latitude into the string but split it up every 3rd digit so that the tokenizers can work better with it. A sample record looks like follows:\n\n`Impressive solution. One question: What do you mean by \"calculate all-pairs similarity\"? How is that implemented? Do you use ANN (Approximate Nearest Neighbor) to find neighbors? Thanks in advance.
", "rawMarkdown": "Impressive solution. One question: What do you mean by \"calculate all-pairs similarity\"? How is that implemented? Do you use ANN (Approximate Nearest Neighbor) to find neighbors? Thanks in advance.", "votes": 2, "replies": [ { "id": 1862410, "postDate": "2022-07-19T17:02:10.243Z", "content": "I am calculating it directly in Pytorch on GPU in batch-wise fashion on the embedding output from the ArcFace model. It is very quick.
", "rawMarkdown": "I am calculating it directly in Pytorch on GPU in batch-wise fashion on the embedding output from the ArcFace model. It is very quick." }, { "id": 1862430, "postDate": "2022-07-19T17:14:23.600Z", "content": "Still kinda confused. Do you mean do sim-score calculations 600k*600k times and you implement that in pytorch in batch-manner?
", "rawMarkdown": "Still kinda confused. Do you mean do sim-score calculations 600k*600k times and you implement that in pytorch in batch-manner?" }, { "id": 1862467, "postDate": "2022-07-19T18:01:03.587Z", "content": "You split the source in batches, let's say 32, and keep the target full. So you calculate 32x600k for 600k/32 times. It is very fast. Let me know if it doesn't make sense and I will try to elaborate more detailed.
", "rawMarkdown": "You split the source in batches, let's say 32, and keep the target full. So you calculate 32x600k for 600k/32 times. It is very fast. Let me know if it doesn't make sense and I will try to elaborate more detailed.", "votes": 1 }, { "id": 1862704, "postDate": "2022-07-19T23:01:20.893Z", "content": "I am also wondering how distance(=similarity) is calculated. Since ArcFace’s embedding vectors are point-wise, we should calculate distance pair-wise, which needs 600K x 600K = 360G samples (suppose we don’t use kNN search). I think this number of samples are intractable.
", "rawMarkdown": "I am also wondering how distance(=similarity) is calculated. Since ArcFace’s embedding vectors are point-wise, we should calculate distance pair-wise, which needs 600K x 600K = 360G samples (suppose we don’t use kNN search). I think this number of samples are intractable." }, { "id": 1862714, "postDate": "2022-07-19T23:19:12.917Z", "content": "It is more than clear now. I have heard of similar candidate-retrieval pipeline in recommendation systems where people do brute-force metric calculation on all pairs instead of ANN (Approximate Nearest Neighbor), thus to avoid possible precision loss, and all that are done real-time at query. At first I just did not believe it is computationally feasible. We should give it a try 👀
", "rawMarkdown": "It is more than clear now. I have heard of similar candidate-retrieval pipeline in recommendation systems where people do brute-force metric calculation on all pairs instead of ANN (Approximate Nearest Neighbor), thus to avoid possible precision loss, and all that are done real-time at query. At first I just did not believe it is computationally feasible. We should give it a try 👀", "votes": 1 }, { "id": 1862719, "postDate": "2022-07-19T23:36:42.210Z", "content": "@tatamikenn I think the total number of sim-score calculation is indeed 360G. It is still computationally feasible though, since in every single sample pair only vector product calculation is needed (no model inference needed). And that could be further batched into matrix calculation.
", "rawMarkdown": "@tatamikenn I think the total number of sim-score calculation is indeed 360G. It is still computationally feasible though, since in every single sample pair only vector product calculation is needed (no model inference needed). And that could be further batched into matrix calculation.", "votes": 1 }, { "id": 1862739, "postDate": "2022-07-20T00:36:36.250Z", "content": "@tatamikenn try this code snippet, where I simulate the sim-score calculation in batch on GPU:
\nimport numpy as np\nimport torch as torch\ndef mock_batch_distance(batch_size=256,embedding_dimension=384):\n matrix1 = torch.rand(batch_size,embedding_dimension,dtype=torch.float16,device='cuda')\n matrix2 = torch.rand(batch_size,embedding_dimension,dtype=torch.float16,device='cuda')\n result = matrix1.matmul(matrix2.T)\n return result\n%timeit mock_batch_distance(batch_size=256,embedding_dimension=384)\n\n\nOut [1]: 46.3 µs ± 980 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
\n
You see that for each batch-batch distance calculation, only 46.3 µs is needed. And the total time spent should be:
\nbatch_size = 256\ntime_per_function_call = 46/1000000\nnum_total_function_call = (600000/batch_size) * (600000/batch_size)\ntotal_time = num_total_function_call * time_per_function_call\ntotal_time\n\n\nOut [2]: 252.685546875
\n
252s, just a few minutes.
", "rawMarkdown": "@tatamikenn try this code snippet, where I simulate the sim-score calculation in batch on GPU:\n\n```python\nimport numpy as np\nimport torch as torch\ndef mock_batch_distance(batch_size=256,embedding_dimension=384):\n matrix1 = torch.rand(batch_size,embedding_dimension,dtype=torch.float16,device='cuda')\n matrix2 = torch.rand(batch_size,embedding_dimension,dtype=torch.float16,device='cuda')\n result = matrix1.matmul(matrix2.T)\n return result\n%timeit mock_batch_distance(batch_size=256,embedding_dimension=384)\n\n```\n\n> Out [1]: 46.3 µs ± 980 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n\nYou see that for each batch-batch distance calculation, only 46.3 µs is needed. And the total time spent should be:\n\n```python\nbatch_size = 256\ntime_per_function_call = 46/1000000\nnum_total_function_call = (600000/batch_size) * (600000/batch_size)\ntotal_time = num_total_function_call * time_per_function_call\ntotal_time\n```\n\n> Out [2]: 252.685546875\n\n252s, just a few minutes.\n", "votes": 1 }, { "id": 1862745, "postDate": "2022-07-20T01:10:26.703Z", "content": "@peltor Thanks for simulation. Now my question is clear.
", "rawMarkdown": "@peltor Thanks for simulation. Now my question is clear." }, { "id": 1862748, "postDate": "2022-07-20T01:22:00.093Z", "content": "I found a mistake on the simulation.
\n\n\n(matrix1*matrix2).sum(axis=-1)
\n
This gives us only batch_size samples (corresponds with self pair). What we want is batch_size x batch_size matrix.
\nHowever, I'm thinking it's more likely to be feasible even if in the fixed simulation.
You are right, I have modified my code. The result indicates no difference though.
", "rawMarkdown": "You are right, I have modified my code. The result indicates no difference though." }, { "id": 1862773, "postDate": "2022-07-20T02:07:29.210Z", "content": "I tried the fixed version. It was quite feasible (19.3 sec for 360G pairs).
\ndef mock_batch_distance(generator, batch_size, embedding_dimension):\n embeddings1 = torch.rand(\n batch_size,\n embedding_dimension,\n dtype=torch.float32,\n device=\"cuda\",\n requires_grad=False,\n generator=generator,\n )\n embeddings2 = torch.rand(\n batch_size,\n embedding_dimension,\n dtype=torch.float32,\n device=\"cuda\",\n requires_grad=False,\n generator=generator,\n )\n result = embeddings1.matmul(embeddings2.T)\n return result\n\n\ngenerator = torch.Generator(device=\"cuda\")\ngenerator.manual_seed(123)\n%timeit -n 100 mock_batch_distance(generator, batch_size=1024 * 32, embedding_dimension=384)\n\n>>> The slowest run took 3287.43 times longer than the fastest. This could mean that an intermediate result is being cached.\n>>> 57.4 ms ± 41.9 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)\nbatch_size = 1024 * 32\ntime_per_function_call = 57.6 * 0.001\nnum_total_function_call = (600_000 / batch_size) * (600_000 / batch_size)\ntotal_time = num_total_function_call * time_per_function_call\ntotal_time\n>>> 19.311904907226566\nc.f. https://www.kaggle.com/code/tatamikenn/4sq-simulation-of-3rd-solution?scriptVersionId=101281202
", "rawMarkdown": "I tried the fixed version. It was quite feasible (19.3 sec for 360G pairs).\n```\ndef mock_batch_distance(generator, batch_size, embedding_dimension):\n embeddings1 = torch.rand(\n batch_size,\n embedding_dimension,\n dtype=torch.float32,\n device=\"cuda\",\n requires_grad=False,\n generator=generator,\n )\n embeddings2 = torch.rand(\n batch_size,\n embedding_dimension,\n dtype=torch.float32,\n device=\"cuda\",\n requires_grad=False,\n generator=generator,\n )\n result = embeddings1.matmul(embeddings2.T)\n return result\n\n\ngenerator = torch.Generator(device=\"cuda\")\ngenerator.manual_seed(123)\n%timeit -n 100 mock_batch_distance(generator, batch_size=1024 * 32, embedding_dimension=384)\n\n>>> The slowest run took 3287.43 times longer than the fastest. This could mean that an intermediate result is being cached.\n>>> 57.4 ms ± 41.9 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)\n```\n\n```\nbatch_size = 1024 * 32\ntime_per_function_call = 57.6 * 0.001\nnum_total_function_call = (600_000 / batch_size) * (600_000 / batch_size)\ntotal_time = num_total_function_call * time_per_function_call\ntotal_time\n>>> 19.311904907226566\n```\n\nc.f. https://www.kaggle.com/code/tatamikenn/4sq-simulation-of-3rd-solution?scriptVersionId=101281202", "votes": 1 }, { "id": 1862780, "postDate": "2022-07-20T02:15:25.617Z", "content": "\n\nI took that into consideration in num_total_function_call, where i do square calculation
\n
No, since the first version calculates batch_size element-wise multiplication, the total calculation result gives (600_000/batch_size) * (600_000/batch_size) * batch_size pairs, which isn't equal to 600K x 600K.
Besides, computation in CUDA should be different between item-wise multiplication and matrix multiplication. I think we should use matrix multiplication when we simulate exactly (suppose the author used it).
", "rawMarkdown": "> I took that into consideration in num_total_function_call, where i do square calculation\n\nNo, since the first version calculates `batch_size` element-wise multiplication, the total calculation result gives `(600_000/batch_size) * (600_000/batch_size) * batch_size` pairs, which isn't equal to 600K x 600K.\n\nBesides, computation in CUDA should be different between item-wise multiplication and matrix multiplication. I think we should use matrix multiplication when we simulate exactly (suppose the author used it).", "votes": 1 }, { "id": 1863088, "postDate": "2022-07-20T06:36:41.183Z", "content": "# embeddings 600k x embedding dimension - already normalized\nchunk = 512\nembeddings_chunks = embeddings.split(chunk)\n\nfor idx in range(len(embeddings_chunks)):\n cos_sim_chunk = torch.mm(embeddings_chunks[idx], embeddings.transpose(0, 1))\n # here you will only want to keep the topk indices in memory for further use\nBTW: it is crazy how far we have got. I remember struggling with this problem on tfidf features ~10 years ago. It was sheer impossible to do on CPU and memory restrictions.
", "rawMarkdown": "BTW: it is crazy how far we have got. I remember struggling with this problem on tfidf features ~10 years ago. It was sheer impossible to do on CPU and memory restrictions. ", "votes": 8 } ] }, { "id": 1861868, "postDate": "2022-07-19T09:57:41.927Z", "content": "Your solution using ArcFace is really impressive.
\nIt seems to demonstrate the broad applicability of ArcFace.
\nIn fact, my team gave up using ArcFace due to the difficulty of tuning.
\nAnd, congratulations on your win!
I also nearly gave up on it, and I still wasted many runs where the loss exploded. It is very hard to tune in general and needs some patience. I started with ArcFace, couldnt make it work, and switched to TripletLoss which worked. So I thought it has to work and went back to ArcFace.
", "rawMarkdown": "I also nearly gave up on it, and I still wasted many runs where the loss exploded. It is very hard to tune in general and needs some patience. I started with ArcFace, couldnt make it work, and switched to TripletLoss which worked. So I thought it has to work and went back to ArcFace.", "votes": 7 } ] }, { "id": 1861866, "postDate": "2022-07-19T09:55:37.860Z", "content": "Thank you for sharing your solution.
\nYour deep learning approach is very impressive.
\nI respect your solution with good rank in both overlap and no-overlap.
I could not succeed with ArcFace. So I would like to read your training code.
\nDo you plan to share your code?
There is no magic in the code that would be interesting. Rather certain training tricks to make the training stable such as: long warmup, playing with the neck of the model, trying different margins and scale combinations, etc.
", "rawMarkdown": "There is no magic in the code that would be interesting. Rather certain training tricks to make the training stable such as: long warmup, playing with the neck of the model, trying different margins and scale combinations, etc.", "votes": 12 }, { "id": 1863214, "postDate": "2022-07-20T08:09:33.803Z", "content": "@philippsinger did differential learning rate work? or some other trick that really helped
\nand did you use any type of augmentations for the texts?
Another query:
\nFor the Bidirectional Encoder,(really impressive trick) you concatenated the sequences I guess it was a binary classification task? where the sequences having more cosine similarity (than some threshold) is considered label 1 else 0. Is it like that?
Diff LR was also helpful, yes
\nIt was a binary classification task, and label 1 or 0 was defined by ground truth data. And I extracted the candidates from in-sample similarities.
", "rawMarkdown": "Diff LR was also helpful, yes\n\nIt was a binary classification task, and label 1 or 0 was defined by ground truth data. And I extracted the candidates from in-sample similarities." } ] }, { "id": 1861983, "postDate": "2022-07-19T11:23:56.030Z", "content": "Thank you for sharing your solution. It was very inspiring!
\nI have one question.
\nThere were many poi with only one class size in this competition data.
\nIn training ArcFace, did you do anything special to the data whose class size was 1?
I tried quite a few things with loss weighting, dynamic margins, and other things, but in the end I did not do anything special with them as overall the class distribution (in terms of nr of records per class) was not too diverse and just leaving it as-is was best or at least not worse.
", "rawMarkdown": "I tried quite a few things with loss weighting, dynamic margins, and other things, but in the end I did not do anything special with them as overall the class distribution (in terms of nr of records per class) was not too diverse and just leaving it as-is was best or at least not worse.", "votes": 3 } ] }, { "id": 2132283, "postDate": "2023-02-06T18:02:06.723Z", "content": "Great work @philippsinger ! Did you use arcface as a loss function or incorporate something like ArcMarginProduct as part of the model arch?
", "rawMarkdown": "Great work @philippsinger ! Did you use arcface as a loss function or incorporate something like ArcMarginProduct as part of the model arch? " }, { "id": 1864016, "postDate": "2022-07-20T17:00:30.457Z", "content": "Congrats! thanks for sharing👍 @philippsinger
", "rawMarkdown": "Congrats! thanks for sharing👍 @philippsinger " }, { "id": 1862676, "postDate": "2022-07-19T22:22:21.737Z", "content": "Shanks for sharing.
\nI’m interested in what is the efficiency of first stage, i.e. recall and positive sample ratio.
", "rawMarkdown": "Shanks for sharing.\n\nI’m interested in what is the efficiency of first stage, i.e. recall and positive sample ratio.", "replies": [ { "id": 1863217, "postDate": "2022-07-20T08:11:20.257Z", "content": "I will have to check on old models on cv, I don't remember it fully. But it was somewhere in range of 0.97 recall maybe? But by taking more I could get higher.
\nWhat I want to mention is that the whole solution is quite quick. A single first stage + single second stage model scores in 90 minutes on LB with 600k records.
", "rawMarkdown": "I will have to check on old models on cv, I don't remember it fully. But it was somewhere in range of 0.97 recall maybe? But by taking more I could get higher.\n\nWhat I want to mention is that the whole solution is quite quick. A single first stage + single second stage model scores in 90 minutes on LB with 600k records. ", "votes": 3 } ] }, { "id": 1862570, "postDate": "2022-07-19T19:52:41.673Z", "content": "Well done!
", "rawMarkdown": "Well done!" }, { "id": 1861799, "postDate": "2022-07-19T08:50:56.323Z", "content": "Congrats on your win!
\n\n\nThe first-stage model is a metric learning model with ArcFace loss trained on the original input data predicting the point-of-interest of a record
\n
does this mean you had ~740k classes in last dense layer, using the whole training data?
\n(asking as i did not have success training arcface model due to the large number of classes)
", "rawMarkdown": "Congrats on your win!\n\n> The first-stage model is a metric learning model with ArcFace loss trained on the original input data predicting the point-of-interest of a record\n\ndoes this mean you had ~740k classes in last dense layer, using the whole training data?\n\n(asking as i did not have success training arcface model due to the large number of classes)", "replies": [ { "id": 1861808, "postDate": "2022-07-19T08:54:06.970Z", "content": "ArcFace is explicitly tailored towards large number of classes. You have this dimension in the margin product of ArcFace.
", "rawMarkdown": "ArcFace is explicitly tailored towards large number of classes. You have this dimension in the margin product of ArcFace." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T05:25:07Z", "competition_id": 35476, "competition_slug": "foursquare-location-matching", "team_id": 8502052, "solution_url": "/competitions/foursquare-location-matching/writeups/psi-3rd-place-solution" }