![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F623370%2F00c6255ee515c26709b1107d59a95937%2F4square_pipeline.jpg?generation=1657345465981057&alt=media\")
\nWe divided the generation of candidates and the determination of whether they were the same POI into four separate stages.
First of all, thanks to the host for an interesting competition and congratulations to all the winners!
\nWhile we learned many things from this competition, it is truly regrettable that there was leakage. It was very sad that we had to spend so much time looking for leaks in the last few days.
\n
\nWe divided the generation of candidates and the determination of whether they were the same POI into four separate stages.
Next, since there was the same data in train and test, we tried to link them and deleted the pairs that would have been incorrect.
\nFor the past few weeks, we have been suspicious of leak for many reasons:
\nIn the few days just before the deadline, we focused on thinking about how to find and utilize leakage. And fortunately, we succeeded in discovering how to take advantage of leakage.
\nhttps://www.kaggle.com/code/takoihiraokazu/sub-ex73-74-75-ex104-115-90-101-merge-train3
", "comments": [ { "id": 1854450, "author_name": "Dogukan Celik", "author_url": "", "post_date": "2022-07-13T17:38:23.197000", "content": "Great approach, thanks for sharing the solution with us! Lot to learn from this.
", "votes": 2, "replies": [] }, { "id": 1849037, "author_name": "Subhajit", "author_url": "", "post_date": "2022-07-09T06:21:56.833000", "content": "Great blog and nice approach! i certainly learned a lot from this. Thanks @takoihiraokazu
", "votes": 2, "replies": [] }, { "id": 1852509, "author_name": "Old Monk", "author_url": "", "post_date": "2022-07-12T05:45:19.260000", "content": "Great approach, thanks for sharing the solution with us! Lot to learn from this.
", "votes": 1, "replies": [] }, { "id": 1852166, "author_name": "zhehao liang", "author_url": "", "post_date": "2022-07-11T21:00:19.190000", "content": "Thank you its a good approach. Use kaggle like code to solve kaggle problem.. :>
", "votes": 1, "replies": [] }, { "id": 1851634, "author_name": "Man of the year", "author_url": "", "post_date": "2022-07-11T11:58:35.250000", "content": "\n\nxlm-roberta-large
\n
\n text : name + categories + address + city + state
\n Combine some features of 2nd ( about 70)
\n 3 epochs
How do you add features to the roberta model?
", "votes": 1, "replies": [ { "id": 1851669, "author_name": "Takoi", "author_url": "", "post_date": "2022-07-11T12:37:54.363000", "content": "I have concated it to the output of roberta.
\nclass FourSquare_model(nn.Module):\n def __init__(self):\n super(FourSquare_model, self).__init__()\n self.model = AutoModel.from_pretrained(THIRD_BERT_MODEL1)\n self.ln1 = nn.LayerNorm(1024)\n self.linear1 = nn.Sequential(\n nn.Linear(1024,128),\n nn.LayerNorm(128),\n nn.ReLU(),\n nn.Dropout(0.2))\n\n self.linear2 = nn.Sequential(\n nn.Linear(76,128),\n nn.LayerNorm(128),\n nn.ReLU(),\n nn.Dropout(0.2))\n\n self.linear3 = nn.Sequential(\n nn.Linear(128 + 128,64),\n nn.LayerNorm(64),\n nn.ReLU(),\n nn.Dropout(0.2),\n nn.Linear(64,1),\n )\n\n\n\n def forward(self, ids, mask, token_type_ids,num_features):\n # pooler\n out = self.model(ids, attention_mask=mask,token_type_ids=token_type_ids)['last_hidden_state'][:,0,:]\n out = self.ln1(out)\n out = self.linear1(out)\n out2 = self.linear2(num_features)\n out = torch.cat([out,out2],axis=-1)\n out = self.linear3(out)\n return out\nthx for sharing the code
", "votes": 1, "replies": [] }, { "id": 1851061, "author_name": "shanshilei", "author_url": "", "post_date": "2022-07-11T02:19:10.793000", "content": "How to use ForestInference，and How to convert the model to Treelite-compatible format
", "votes": 1, "replies": [] }, { "id": 1849925, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-10T00:02:09.300000", "content": "Thank you for sharing your code with us. Your team's feature engineering is really impressive!
\nCongrats!
Interesting approach, it was certainly helpful! Thanks.
", "votes": 1, "replies": [] }, { "id": 1849734, "author_name": "Naman Sharma", "author_url": "", "post_date": "2022-07-09T18:36:51.290000", "content": "Really Nice Work!!
", "votes": 1, "replies": [] }, { "id": 1849355, "author_name": "alu arjun", "author_url": "", "post_date": "2022-07-09T12:49:26.803000", "content": "great approach! thanks for sharing with us
", "votes": 1, "replies": [] }, { "id": 1849081, "author_name": "Phaedrus", "author_url": "", "post_date": "2022-07-09T07:32:52.617000", "content": "Great writeup and solution. One question - Why did you choose such low threshold for 2nd stage (0.01), and even after choosing low value the candidates decreased by 90%? I assume threshold = predicted probability from lgbm with rich features.
", "votes": 1, "replies": [ { "id": 1849313, "author_name": "Takoi", "author_url": "", "post_date": "2022-07-09T12:07:58.600000", "content": "Thanks!
\nYes, I was able to eliminate almost 90% of the data with a threshold of 0.01. The threshold was determined by looking at the balance between the number of remaining data and the max iou. At the end of the 2nd stage, max iou was 0.974.
Perfect, thanks! Congrats again.
", "votes": 1, "replies": [] } ] }, { "id": 1849058, "author_name": "qyxs", "author_url": "", "post_date": "2022-07-09T07:13:44.057000", "content": "Congratulation and one question, can you share the journey of finding the leak? I think it is very hard because the online train file is different with offline.
", "votes": 1, "replies": [ { "id": 1849331, "author_name": "Takoi", "author_url": "", "post_date": "2022-07-09T12:31:16.940000", "content": "Thanks!
\nHere's how we found leak.
From the above, we were almost certain that there was data overlap between train and test.
\nNext, we experimented with the train merge described in the solution using online train data and found that LB did not increase.
\nHowever, since we were convinced that train and test had the same data, we did not give up and wondered what the problem was. We thought that the contents of train.csv might have been switched, so we used another data that we had uploaded for debugging in advance. When we did so, LB went up quite a bit and we noticed that the contents of train.csv had been replaced.
Admire your team, especially the spirit of never giving up.
", "votes": 2, "replies": [] } ] }, { "id": 1855522, "author_name": "Bilal Suppal", "author_url": "", "post_date": "2022-07-14T16:53:02.850000", "content": "Congratulation and one question, can you share the journey of finding the leak? I think it is very hard because the online train file is different with offline.
", "votes": 0, "replies": [] }, { "id": 1855175, "author_name": "OJASWA YADAV", "author_url": "", "post_date": "2022-07-14T10:47:32.153000", "content": "Appreciate this approach, kudos
", "votes": 0, "replies": [] }, { "id": 1850803, "author_name": "Muhammad Rejwan", "author_url": "", "post_date": "2022-07-10T18:56:42.760000", "content": "thank you for sharing the code
", "votes": 1, "replies": [] }, { "id": 1849694, "author_name": "HendZB", "author_url": "", "post_date": "2022-07-09T18:01:41.390000", "content": "Very interesting, thanks for sharing.
", "votes": 1, "replies": [] } ], "comment_count": 22, "messages": [ { "id": 1849020, "postDate": "2022-07-09T05:58:36.040Z", "content": "First of all, thanks to the host for an interesting competition and congratulations to all the winners!
\nWhile we learned many things from this competition, it is truly regrettable that there was leakage. It was very sad that we had to spend so much time looking for leaks in the last few days.
\n
\nWe divided the generation of candidates and the determination of whether they were the same POI into four separate stages.
Next, since there was the same data in train and test, we tried to link them and deleted the pairs that would have been incorrect.
\nFor the past few weeks, we have been suspicious of leak for many reasons:
\nIn the few days just before the deadline, we focused on thinking about how to find and utilize leakage. And fortunately, we succeeded in discovering how to take advantage of leakage.
\nhttps://www.kaggle.com/code/takoihiraokazu/sub-ex73-74-75-ex104-115-90-101-merge-train3
", "rawMarkdown": "First of all, thanks to the host for an interesting competition and congratulations to all the winners!\n\nWhile we learned many things from this competition, it is truly regrettable that there was leakage. It was very sad that we had to spend so much time looking for leaks in the last few days.\n\n## Summary\n\nWe divided the generation of candidates and the determination of whether they were the same POI into four separate stages.\n- 1st stage : Create candidates\n- 2nd stage : Feature engineering and LightGBM\n- 3rd stage : xlm-roberta-large and mdeberta-v3-base and Catboost\n- 4th stage : Post Process and Predicting newly emerged candidates with xlm-roberta-large\n\nNext, since there was the same data in train and test, we tried to link them and deleted the pairs that would have been incorrect.\n\n## Cross validation strategy\n- Local evaluation was done with 2 fold cross validation. In the training for submit, we used all of the 2 fold data.\n\n## 1st stage\n- For each id, we selected 100 candidates for each of the following two methods\n - Euclidean distance in latitude and longitude\n - Cos similarity of name embedding\n - model : bert-base-multilingual-uncased\n - knn : cuml\n - Generate features for candidates. Here, to save memory, only a few features were generated, such as jaro distance for names and jaro distance for categories.\n - Using LightGBM, each of the above two patterns was predicted, leaving the top 20 candidates each, for a total of about 40 candidates. To reduce inference time, we used Forestinference.\n - Max IOU: 0.979\n\n## 2nd stage\n- We created about 120 features based on the following notebook.\n - https://www.kaggle.com/code/ryotayoshinobu/foursquare-lightgbm-baseline\n - Character similarity features such as Levenshtein distance and Jaro-winkler Distance.\n - Character similarity statistics (maximum, minimum, average) using id as key. \nAlso, the ratio of those statistics.\n - Euclidean distance using latitude and longitude\n - Embedding of name with svd for dimensionality reduction\n - model : bert-base-multilingual-uncased\n- LightGBM ( Inference : Forestinference)\n - CV : 0.875\n- Threshold value : 0.01\n - The number of candidates was reduced to about 10%.\n\n## 3rd stage\n- We had suspected overlap between train and test, but had no direct evidence, so we created train overfit models to increase LB and models to increase CV at the same time. After finding evidence of train and test overlap, we only needed to concentrate on improving the accuracy of the data that existed only in test, so we finally used models that had the highest CV and did not overfit train data(e.g. do not increase the number of epochs, use FGM, etc.)\n - catboost\n - The same features were used as in the 2nd\n - CV : 0.878\n - xlm-roberta-large\n - text : name + categories + address + city + state\n - Combine some features of 2nd ( about 70)\n - 3 epochs\n - mdeberta-v3-base\n - text : name + categories + address + city + state\n - Combine some features of 2nd and new features ( about 90 ) \n - manhattan distance, harversinie distance, etc \n - Trained with FGM and EMA\n - 4 epochs\n - CV : 0.907\n- Ensemble : 2nd lgb * 0.01 + 3rd catboost * 0.32 + 3rd xlm-roberta-large * 0.29 + 3rd medebeta-v3-base * 0.38\n - CV : 0.911\n- Threshold value : 0.5\n## 4th stage\n- Post procress\n - Compare the matches of two ids and merge the matches of two ids if the common id exceeds 50% from either side.\n - Prediction with xlm-roberta-large for the newly created pair above.\n - Threshold value : 0.02\n - CV : 0.9166\n\n## Merge train (Leak)\n- Tie train and test by name,lat,lon\n- Add true positive\n - Create all pairs of true positive with POI as key (1)\n- Remove false positive\n - Remove Pairs of id's tied with train data (2)\n - It was possible to determine whether the id associated with the train data was false positive or not\n - Remove pairs of id tied with training data and id not tied with training data(3)\n- By using LB, we validated the effects of (1) add train-train tp, (2) remove train-train fp, (3) remove train-test fp, which are explained above.\n - We compared LB with submission at stage2.\n - Without merge train : 0.900\n - (1) : 0.943\n - (1) + (2) + (3) : 0.971\n- Since we were not sure whether (3) would work for private data, we chose both (1) + (2) and (1) + (2) + (3) for the final submission.\n\n## Private Score / Public Score\n- Over fit version\n - mdeberta-v3-base(no dropout) 8epoch + xlm-roberta-large 8epoch\n - without train merge : 0.946 / 0.947\n - with train merge : 0.976 / 0.976\n- Non over fit version1\n - as explained above\n - without train merge : 0.941 / 0.941\n - with train merge : 0.977 / 0.978\n - (1) + (2) + (3)\n- Non over fit version2\n - Slightly changed model compared to non over fit version1\n - with train merge : 0.966 / 0.966\n - (1) + (2)\n\n## About Leakage\nFor the past few weeks, we have been suspicious of leak for many reasons:\n- There is a large gap between CV and LB. Max IOU is already high, so a gap of this magnitude is unnatural.\n- LB score is higher with overfitted models. For example, no dropout or many epochs training. Training with many epochs leads to a worse CV score.\n- As the CV increased, the CV/LB correlation disappeared. We had a big improvement of CV by training of NNs with FGM but LB was worse.\n- The host explained that \"we include a few example instances selected from the test set.\" and the two of the five sample records exist in training data.\n\nIn the few days just before the deadline, we focused on thinking about how to find and utilize leakage. And fortunately, we succeeded in discovering how to take advantage of leakage.\n\n## Inference code\nhttps://www.kaggle.com/code/takoihiraokazu/sub-ex73-74-75-ex104-115-90-101-merge-train3\n\n\n", "votes": 92 }, { "id": 1854450, "postDate": "2022-07-13T17:38:23.197Z", "content": "Great approach, thanks for sharing the solution with us! Lot to learn from this.
", "rawMarkdown": "Great approach, thanks for sharing the solution with us! Lot to learn from this.", "votes": 2 }, { "id": 1849037, "postDate": "2022-07-09T06:21:56.833Z", "content": "Great blog and nice approach! i certainly learned a lot from this. Thanks @takoihiraokazu
", "rawMarkdown": "Great blog and nice approach! i certainly learned a lot from this. Thanks @takoihiraokazu ", "votes": 2 }, { "id": 1852509, "postDate": "2022-07-12T05:45:19.260Z", "content": "Great approach, thanks for sharing the solution with us! Lot to learn from this.
", "rawMarkdown": "Great approach, thanks for sharing the solution with us! Lot to learn from this.", "votes": 1 }, { "id": 1852166, "postDate": "2022-07-11T21:00:19.190Z", "content": "Thank you its a good approach. Use kaggle like code to solve kaggle problem.. :>
", "rawMarkdown": "Thank you its a good approach. Use kaggle like code to solve kaggle problem.. :>", "votes": 1 }, { "id": 1851634, "postDate": "2022-07-11T11:58:35.250Z", "content": "\n\nxlm-roberta-large
\n
\n text : name + categories + address + city + state
\n Combine some features of 2nd ( about 70)
\n 3 epochs
How do you add features to the roberta model?
", "rawMarkdown": ">xlm-roberta-large\n>text : name + categories + address + city + state\n>Combine some features of 2nd ( about 70)\n>3 epochs\n\nHow do you add features to the roberta model?", "votes": 1, "replies": [ { "id": 1851669, "postDate": "2022-07-11T12:37:54.363Z", "content": "I have concated it to the output of roberta.
\nclass FourSquare_model(nn.Module):\n def __init__(self):\n super(FourSquare_model, self).__init__()\n self.model = AutoModel.from_pretrained(THIRD_BERT_MODEL1)\n self.ln1 = nn.LayerNorm(1024)\n self.linear1 = nn.Sequential(\n nn.Linear(1024,128),\n nn.LayerNorm(128),\n nn.ReLU(),\n nn.Dropout(0.2))\n\n self.linear2 = nn.Sequential(\n nn.Linear(76,128),\n nn.LayerNorm(128),\n nn.ReLU(),\n nn.Dropout(0.2))\n\n self.linear3 = nn.Sequential(\n nn.Linear(128 + 128,64),\n nn.LayerNorm(64),\n nn.ReLU(),\n nn.Dropout(0.2),\n nn.Linear(64,1),\n )\n\n\n\n def forward(self, ids, mask, token_type_ids,num_features):\n # pooler\n out = self.model(ids, attention_mask=mask,token_type_ids=token_type_ids)['last_hidden_state'][:,0,:]\n out = self.ln1(out)\n out = self.linear1(out)\n out2 = self.linear2(num_features)\n out = torch.cat([out,out2],axis=-1)\n out = self.linear3(out)\n return out\nthx for sharing the code
", "rawMarkdown": "thx for sharing the code", "votes": 1 }, { "id": 1851061, "postDate": "2022-07-11T02:19:10.793Z", "content": "How to use ForestInference，and How to convert the model to Treelite-compatible format
", "rawMarkdown": "How to use ForestInference，and How to convert the model to Treelite-compatible format", "votes": 1 }, { "id": 1849925, "postDate": "2022-07-10T00:02:09.300Z", "content": "Thank you for sharing your code with us. Your team's feature engineering is really impressive!
\nCongrats!
Interesting approach, it was certainly helpful! Thanks.
", "rawMarkdown": "Interesting approach, it was certainly helpful! Thanks.", "votes": 1 }, { "id": 1849734, "postDate": "2022-07-09T18:36:51.290Z", "content": "Really Nice Work!!
", "rawMarkdown": "Really Nice Work!!", "votes": 1 }, { "id": 1849355, "postDate": "2022-07-09T12:49:26.803Z", "content": "great approach! thanks for sharing with us
", "rawMarkdown": "great approach! thanks for sharing with us", "votes": 1 }, { "id": 1849081, "postDate": "2022-07-09T07:32:52.617Z", "content": "Great writeup and solution. One question - Why did you choose such low threshold for 2nd stage (0.01), and even after choosing low value the candidates decreased by 90%? I assume threshold = predicted probability from lgbm with rich features.
", "rawMarkdown": "Great writeup and solution. One question - Why did you choose such low threshold for 2nd stage (0.01), and even after choosing low value the candidates decreased by 90%? I assume threshold = predicted probability from lgbm with rich features. ", "votes": 1, "replies": [ { "id": 1849313, "postDate": "2022-07-09T12:07:58.600Z", "content": "Thanks!
\nYes, I was able to eliminate almost 90% of the data with a threshold of 0.01. The threshold was determined by looking at the balance between the number of remaining data and the max iou. At the end of the 2nd stage, max iou was 0.974.
Perfect, thanks! Congrats again.
", "rawMarkdown": "Perfect, thanks! Congrats again. ", "votes": 1 } ] }, { "id": 1849058, "postDate": "2022-07-09T07:13:44.057Z", "content": "Congratulation and one question, can you share the journey of finding the leak? I think it is very hard because the online train file is different with offline.
", "rawMarkdown": "Congratulation and one question, can you share the journey of finding the leak? I think it is very hard because the online train file is different with offline.", "votes": 1, "replies": [ { "id": 1849331, "postDate": "2022-07-09T12:31:16.940Z", "content": "Thanks!
\nHere's how we found leak.
From the above, we were almost certain that there was data overlap between train and test.
\nNext, we experimented with the train merge described in the solution using online train data and found that LB did not increase.
\nHowever, since we were convinced that train and test had the same data, we did not give up and wondered what the problem was. We thought that the contents of train.csv might have been switched, so we used another data that we had uploaded for debugging in advance. When we did so, LB went up quite a bit and we noticed that the contents of train.csv had been replaced.
Admire your team, especially the spirit of never giving up.
", "rawMarkdown": "Admire your team, especially the spirit of never giving up.", "votes": 2 } ] }, { "id": 1855522, "postDate": "2022-07-14T16:53:02.850Z", "content": "Congratulation and one question, can you share the journey of finding the leak? I think it is very hard because the online train file is different with offline.
", "rawMarkdown": "Congratulation and one question, can you share the journey of finding the leak? I think it is very hard because the online train file is different with offline.\n\n" }, { "id": 1855175, "postDate": "2022-07-14T10:47:32.153Z", "content": "Appreciate this approach, kudos
", "rawMarkdown": "Appreciate this approach, kudos" }, { "id": 1850803, "postDate": "2022-07-10T18:56:42.760Z", "content": "thank you for sharing the code
", "rawMarkdown": "thank you for sharing the code", "votes": 1 }, { "id": 1849694, "postDate": "2022-07-09T18:01:41.390Z", "content": "Very interesting, thanks for sharing.
", "rawMarkdown": "Very interesting, thanks for sharing.", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T05:24:10Z", "competition_id": 35476, "competition_slug": "foursquare-location-matching", "team_id": 8600383, "solution_url": "/competitions/foursquare-location-matching/writeups/re-waiwai-1st-place-solution" }