First, thanks to Kaggle and Foursquare for providing a very interesting challenge. The leakage, though, is rather frustrating. This would have been an easy leak to avoid. I can't imagine how this slipped through. Looking for leaks is a normal part of Kaggle competitions, but when the instructions explicitly state that a particular leak does not exist, then it really shouldn't be there.
\nCongrats to the winners, especially @psi for his late surge and winning solution (if we don't include explicit use of the leak). This was a tough competition that required a lot of ingenuity to do well.
\nMy big take-aways from this competition are:
\nWhy are most of the top teams from Japan? Is there something about the data that gave teams from Japan an advantage?
\nBelow is a brief outline of my solution (does not explicitly use the leak).
\nCV scoring was performed on a holdout set of 20% of the candidates, grouped by point_of_interest. From the beginning, the LB scores were slightly higher than the CV scores. By the end of the competition, they were considerably higher:
\nCV: 0.908 (single model)
\nLB: 0.935 (ensemble of 10 models)
I expected some increase due to the ensembling but this was more than expected. We now know that this is due to the leak. Essentially, the difference in CV/LB scores represents how much our models overfit. So, it's true, as some have written, that we all used the leak implicitly, but that's still a lot different from using it explicitly.
", "comments": [ { "id": 1847888, "author_name": "qyxs", "author_url": "", "post_date": "2022-07-08T08:24:07.893000", "content": "Some team find the leak early, some near deadline and some in same nations.
", "votes": 3, "replies": [ { "id": 1847897, "author_name": "heng", "author_url": "", "post_date": "2022-07-08T08:32:08.090000", "content": "This reminds me of the previous nbme competition when the top five were all Chinese teams, \"That is an interesting fact.\" lol
", "votes": 0, "replies": [] } ] }, { "id": 1849868, "author_name": "Matt Motoki", "author_url": "", "post_date": "2022-07-09T22:45:07.680000", "content": "Thanks for sharing and congratulations on a strong solo performance! Old-school NLP and good engineering were definitely important for us too.
\n\n\nWhy are most of the top teams from Japan? Is there something about the data that gave teams from Japan an advantage?
\n
I was wondering about that too. I feel like Japan usually has a strong presence on the leaderboard. In this competition, it seemed like there were fewer competitors from other parts of the world. So another question to ask is:
\n\n", "votes": 1, "replies": [] }, { "id": 1849772, "author_name": "steubk", "author_url": "", "post_date": "2022-07-09T19:18:46.657000", "content": "Where are all the Masters and Grandmasters from other parts of the world?
\n
Thank you for the sharing !
\n\n\nAs ML toolsets mature, ML is becoming more and more just a software engineering specialty. In this competition especially, setting up the problem correctly to process the data in chunks using minimal resources was just as important as the specific ML algorithms employed.44
\n
This was my same feeling during the competition: it seemed to me that I was using ML as (just another) programming technique to solve a complex problem.
\nBut perhaps this is also due to the fact that (due to the overlap between train and test) everything that worked in the train also worked in the test.
Thank you for the sharing ! Great work. Do you have an idea of the time you spent on this competition ?
", "votes": 0, "replies": [ { "id": 1847988, "author_name": "Silogram", "author_url": "", "post_date": "2022-07-08T09:41:41.580000", "content": "I didn't keep track, but it wouldn't surprise me if it was more than 100 hours. The full pipeline took a long time to run, and then the submissions took even longer, so there were long periods where I was working on other stuff while I waited for some code to run. How about you?
", "votes": 0, "replies": [] }, { "id": 1848178, "author_name": "Vincent Schuler", "author_url": "", "post_date": "2022-07-08T12:31:53.063000", "content": "Ok, thank you. It's more or less the same for me, ~100/150 hours depending on how you count the time (coding vs passive waiting).
\nBut I'm sure that I spent more than 100 hours in front of my computer for this competition ^^
great stuff. step number 4/5/6 is kind of cool! greetings from winterthur
", "votes": 0, "replies": [ { "id": 1847964, "author_name": "Silogram", "author_url": "", "post_date": "2022-07-08T09:25:31.847000", "content": "Thanks. Nice to see another Swiss Kaggler here.
", "votes": 0, "replies": [] } ] }, { "id": 1848162, "author_name": "Marek Nurzynski", "author_url": "", "post_date": "2022-07-08T12:16:28.573000", "content": "Great work, Thank you for the sharing !
", "votes": 0, "replies": [] } ], "comment_count": 10, "messages": [ { "id": 1847808, "postDate": "2022-07-08T06:57:03.383Z", "content": "First, thanks to Kaggle and Foursquare for providing a very interesting challenge. The leakage, though, is rather frustrating. This would have been an easy leak to avoid. I can't imagine how this slipped through. Looking for leaks is a normal part of Kaggle competitions, but when the instructions explicitly state that a particular leak does not exist, then it really shouldn't be there.
\nCongrats to the winners, especially @psi for his late surge and winning solution (if we don't include explicit use of the leak). This was a tough competition that required a lot of ingenuity to do well.
\nMy big take-aways from this competition are:
\nWhy are most of the top teams from Japan? Is there something about the data that gave teams from Japan an advantage?
\nBelow is a brief outline of my solution (does not explicitly use the leak).
\nCV scoring was performed on a holdout set of 20% of the candidates, grouped by point_of_interest. From the beginning, the LB scores were slightly higher than the CV scores. By the end of the competition, they were considerably higher:
\nCV: 0.908 (single model)
\nLB: 0.935 (ensemble of 10 models)
I expected some increase due to the ensembling but this was more than expected. We now know that this is due to the leak. Essentially, the difference in CV/LB scores represents how much our models overfit. So, it's true, as some have written, that we all used the leak implicitly, but that's still a lot different from using it explicitly.
", "rawMarkdown": "\nFirst, thanks to Kaggle and Foursquare for providing a very interesting challenge. The leakage, though, is rather frustrating. This would have been an easy leak to avoid. I can't imagine how this slipped through. Looking for leaks is a normal part of Kaggle competitions, but when the instructions explicitly state that a particular leak does not exist, then it really shouldn't be there. \n \nCongrats to the winners, especially @psi for his late surge and winning solution (if we don't include explicit use of the leak). This was a tough competition that required a lot of ingenuity to do well.\n \nMy big take-aways from this competition are:\n \n1. RAPIDS is a very mature and useful library. This was my first time using RAPIDS and I was really impressed with how much sklearn and pandas functionality can be run on the GPU with virtually no code changes. The speed-up was impressive, espcially in cases where the non-GPU version uses only 1 CPU. Also, many thanks to @steubk for showing how to do inference using ForestInference (https://www.kaggle.com/competitions/foursquare-location-matching/discussion/328242).\n\n2. Old-school nlp techniques such as text normalization, tfidf, keyword extraction, etc. are still relevant in isolated cases. In this competition, most of the text were names in multiple languages, which reduced the usefulness of pre-trained text models. \n\n3. As ML toolsets mature, ML is becoming more and more just a software engineering specialty. In this competition especially, setting up the problem correctly to process the data in chunks using minimal resources was just as important as the specific ML algorithms employed.\n\n# Big question about this competition\n \nWhy are most of the top teams from Japan? Is there something about the data that gave teams from Japan an advantage?\n \nBelow is a brief outline of my solution (does not explicitly use the leak).\n \n## Training Pipeline\n \n1. Preprocess training data:\n\ta) Normalize text fields (lowercase, remove punctuation, normalize common spelling variations, convert to ASCII)\n\tb) Select frequent name keywords and encode as features\n\tc) Separate multiple categories and encode as single categories.\n\n2. Randomly divide training set into two equal parts, grouped by point_of_interest, producing two sets -- train0 and train1.\n\n3. Process the two training sets separately, as follows:\n\n a. Get term frequency vectors for text fields\n\tb. For each id, find 200 closest neighbors by lat/lon.\n\tc. Generate features in groups of 10 neighbors (e.g., first 1-10 closest neighbors, then 11-20, 21-30, etc.). Each id is paired with the 10 neighbors in the group to generate pairwise features. Altogether, about 50 features are generated, the most important of which are:\n\t\t1) Cosine similarities between text fields (especially normalized names, name_nonkey, and address)\n\t\t2) Distance rank (e.g., 2nd closest neighbor vs. 150th closest)\n\t\t3) Prev and Next distance (e.g., if we're processing the 5th closest neighbor, what are the distances of the 4th and 6th neighbors)\n\t\t4) Jaccard similarity between names\n\t\t5) Raw latitude/longitude\n\t\t6) Encoded categories\n\t\t7) number of matching name words in pair \n\t\t8) length in characters of longest name\n\t\t9) Min and max ordinal values of characters in name (proxy for character set / language)\n\t\t10) Number of category matches\n\t\t11) Encoded country\n\n4. Again, processing in groups of 10 neighbors, train shallow LGB models to predict pair matches.\n\n5. Use the match predictions from #4 to select the most likely matches. The goal here is to reduce the candidate universe and get a more balanced training set without throwing out too many positives. For each group of 10 nearest neighbors, we train 2 shallow Lightgbm models -- one for train0 and one for train1. We use the model from one set to predict matches for the other set so that we get oof predictions for both sets. We save the 40 models (2 each for 20 groups of 10 neighbors) for inference.\n\n6. Using the predictions from step 5, we set thresholds to filter out candidates that have almost no chance of being matches. I found that setting the threshold at about 30-1 (positives to negatives) worked well. It captured about 95% of the positives (the ones in the 200 nearest-neighbor pairs) while eliminating about 90% of the negatives. The threshold values were set using different percentiles in each group of 10. For example, in the neighbor group 1-10, we keep the top 15% of pairs, whereas in the group containing neighbors 190-199, we keep only the top 0.05% of pairs.\n\n7. Use two additional heuristics to collect more candiates:\n\ta) Round latitude and longitude values and then merge by the rounded lat/lon and specific fields, such as name, address, phone, url, etc. Rounding the lat/lon values is equivalent to dividing the map into quadrants and assigning the same lat/lon values to all ids in the quadrant. Then if we merge by name, for example, we're finding all pairs sharing the same name within each quadrant.\n\tb) Merge by city and name.\n\n8. Combine all candidates from steps 6-7. Recall that steps 3-7 are applied seprately to the two training sets generated in step #2, so in this step we also combine the candidates from those 2 sets. This gives us our master set of training candidates -- about 20m records, with a negative-positive ratio of about 15-1. Of the potential 1.9m positive pairs in the training set, we capture about 70%. \n\n9. Train Lightgbm and Xgboost models and save the models for inference.\n\n10. Repeat steps 2-9 five times to get different lgb/xgb models.\n\n## Inference pipeline\n\n1. Apply steps 1-10 (excluding step #2) to the test set. For the modeling steps, we do inference only using the models generated from the train set.\n2. Post process predictions to add two types of pairs that may be missing from the candidate set:\n\ta) inverses (e.g., if AB is a match, then BA should also be a match)\n\tb) graph traversal pairs (e.g., if AB and AC are pairs, then BC should also be a pair). For traversal pairs, we use a threshold of 0.65, meaning that subgraphs are formed only from pairs whose the probability of being a match is greater than 0.65.\n\n## CV/LB Agreement\n \nCV scoring was performed on a holdout set of 20% of the candidates, grouped by point_of_interest. From the beginning, the LB scores were slightly higher than the CV scores. By the end of the competition, they were considerably higher:\n \nCV: 0.908 (single model)\nLB: 0.935 (ensemble of 10 models)\n \nI expected some increase due to the ensembling but this was more than expected. We now know that this is due to the leak. Essentially, the difference in CV/LB scores represents how much our models overfit. So, it's true, as some have written, that we all used the leak implicitly, but that's still a lot different from using it explicitly.", "votes": 38 }, { "id": 1847888, "postDate": "2022-07-08T08:24:07.893Z", "content": "Some team find the leak early, some near deadline and some in same nations.
", "rawMarkdown": "Some team find the leak early, some near deadline and some in same nations.", "votes": 3, "replies": [ { "id": 1847897, "postDate": "2022-07-08T08:32:08.090Z", "content": "This reminds me of the previous nbme competition when the top five were all Chinese teams, \"That is an interesting fact.\" lol
", "rawMarkdown": "This reminds me of the previous nbme competition when the top five were all Chinese teams, \"That is an interesting fact.\" lol" } ] }, { "id": 1849868, "postDate": "2022-07-09T22:45:07.680Z", "content": "Thanks for sharing and congratulations on a strong solo performance! Old-school NLP and good engineering were definitely important for us too.
\n\n\nWhy are most of the top teams from Japan? Is there something about the data that gave teams from Japan an advantage?
\n
I was wondering about that too. I feel like Japan usually has a strong presence on the leaderboard. In this competition, it seemed like there were fewer competitors from other parts of the world. So another question to ask is:
\n\n", "rawMarkdown": "Thanks for sharing and congratulations on a strong solo performance! Old-school NLP and good engineering were definitely important for us too. \n\n> Why are most of the top teams from Japan? Is there something about the data that gave teams from Japan an advantage?\n\n\nI was wondering about that too. I feel like Japan usually has a strong presence on the leaderboard. In this competition, it seemed like there were fewer competitors from other parts of the world. So another question to ask is:\n\n> Where are all the Masters and Grandmasters from other parts of the world? ", "votes": 1 }, { "id": 1849772, "postDate": "2022-07-09T19:18:46.657Z", "content": "Where are all the Masters and Grandmasters from other parts of the world?
\n
Thank you for the sharing !
\n\n\nAs ML toolsets mature, ML is becoming more and more just a software engineering specialty. In this competition especially, setting up the problem correctly to process the data in chunks using minimal resources was just as important as the specific ML algorithms employed.44
\n
This was my same feeling during the competition: it seemed to me that I was using ML as (just another) programming technique to solve a complex problem.
\nBut perhaps this is also due to the fact that (due to the overlap between train and test) everything that worked in the train also worked in the test.
Thank you for the sharing ! Great work. Do you have an idea of the time you spent on this competition ?
", "rawMarkdown": "Thank you for the sharing ! Great work. Do you have an idea of the time you spent on this competition ?", "replies": [ { "id": 1847988, "postDate": "2022-07-08T09:41:41.580Z", "content": "I didn't keep track, but it wouldn't surprise me if it was more than 100 hours. The full pipeline took a long time to run, and then the submissions took even longer, so there were long periods where I was working on other stuff while I waited for some code to run. How about you?
", "rawMarkdown": "I didn't keep track, but it wouldn't surprise me if it was more than 100 hours. The full pipeline took a long time to run, and then the submissions took even longer, so there were long periods where I was working on other stuff while I waited for some code to run. How about you?" }, { "id": 1848178, "postDate": "2022-07-08T12:31:53.063Z", "content": "Ok, thank you. It's more or less the same for me, ~100/150 hours depending on how you count the time (coding vs passive waiting).
\nBut I'm sure that I spent more than 100 hours in front of my computer for this competition ^^
great stuff. step number 4/5/6 is kind of cool! greetings from winterthur
", "rawMarkdown": "great stuff. step number 4/5/6 is kind of cool! greetings from winterthur", "replies": [ { "id": 1847964, "postDate": "2022-07-08T09:25:31.847Z", "content": "Thanks. Nice to see another Swiss Kaggler here.
", "rawMarkdown": "Thanks. Nice to see another Swiss Kaggler here. " } ] }, { "id": 1848162, "postDate": "2022-07-08T12:16:28.573Z", "content": "Great work, Thank you for the sharing !
", "rawMarkdown": "Great work, Thank you for the sharing !" } ], "complete": true, "source": "live", "fetched_at": "2026-10-22T08:24:34Z", "competition_id": 35476, "competition_slug": "foursquare-location-matching", "team_id": 8607166, "solution_url": "/competitions/foursquare-location-matching/writeups/silogram-a-few-notes-on-16-solution" }