{
  "id": 335924,
  "title": "11th place code and solution",
  "content": "<p>First of all I would like to thank the team member and, participants and Foursquare.<br>\nExcept the data leak, the problem design was very interesting and a good competition with many possible approaches.</p>\n<p>Our team may have been too pure to suspect the data leak, (because the host explicitly declared that there was no overlaps).</p>\n<p>ATTENTION: This notes was written before the data leak was known, and some points may seem silly for now, but we will share it.</p>\n<h3>Links</h3>\n<ul>\n<li>Training(XGB) notebook: <a href=\"https://www.kaggle.com/code/iiyamaiiyama/fs-training-4-ensemble-0705/notebook?scriptVersionId=100082426\" target=\"_blank\">https://www.kaggle.com/code/iiyamaiiyama/fs-training-4-ensemble-0705/notebook?scriptVersionId=100082426</a></li>\n<li>Inference notebook: <a href=\"https://www.kaggle.com/code/iiyamaiiyama/fs-inference-no54-spatial-no-country-07045/notebook?scriptVersionId=100093618\" target=\"_blank\">https://www.kaggle.com/code/iiyamaiiyama/fs-inference-no54-spatial-no-country-07045/notebook?scriptVersionId=100093618</a> </li>\n<li>NN models traing code (github): <a href=\"https://github.com/heartkilla/mcd-kaggle-fsq\" target=\"_blank\">https://github.com/heartkilla/mcd-kaggle-fsq</a></li>\n</ul>\n<h3>Summary</h3>\n<ul>\n<li>Metric learning multi-input(concatenated text -&gt; BERT + lat/lon) NN model.</li>\n<li>Generate candidate by NN model embeddings and spatial nearest neighbours.</li>\n<li>Create 2nd stage binary classification XGB models for each candidate rank.</li>\n</ul>\n<h3>Validation strategy</h3>\n<ul>\n<li>In the beginning we split training data in 4 folds, and trained BERT(train:valid = 3:1). Then we pick up one fold(validation), and split it further in 4 folds for 2nd stage XGB model training.</li>\n<li>After a while, we found that we can get much better results if we trained BERT using all the data. In this case we train XGB with leaked data and it may perform badly for the test data. Maybe the quality of embeddings is more important than traning a leak-free XGB model.</li>\n</ul>\n<h3>Preprocessing</h3>\n<ul>\n<li>Fill <code>NaN</code> texts with nearest(in longitude and latitude) 5 points for BERT models.</li>\n</ul>\n<h3>1st stage: Create candidates</h3>\n<ul>\n<li>BERT embeding candidates<ul>\n<li>Architechture(BERT + lat/lon)<ul>\n<li>Concatenate BERT model output with normalized lat/lon values and connected to the FC layer(320).</li>\n<li>FC layer(320) embeddings were used for similarity.</li>\n<li>Loss: ArcMarginProduct</li>\n<li>Training<ul>\n<li>Gradually increase Margin during training(0.2 to 0.8)</li>\n<li>40 epoch training(24-48 hours to train)</li></ul></li></ul></li>\n<li>Emsemble<ul>\n<li>Concatenate multiple BERT models embeddings<ul>\n<li><code>xlm-roBERTa-large</code>, <code>sentence-transformers/LaBSE</code>, <code>sentence-transformers/paraphrase-multilingual-mpnet-base-v2</code>, <code>remBERT</code></li></ul></li></ul></li>\n<li>DBA/QE<ul>\n<li>Weighted DBA/QE for better embedding.</li></ul></li>\n<li>Create candidates<ul>\n<li>For each id, 50 candidates were created using cosine similarity (faiss)</li></ul></li></ul></li>\n<li>Spatial candidates<ul>\n<li>Added candidates from lat/lon using sklearn.neighbors.BallTree.</li>\n<li>Hoping to add candidates that were missed by BERT embeddings.</li></ul></li>\n</ul>\n<h3>2nd stage: GBDT models for each candidate rank</h3>\n<ul>\n<li>Many public kernels train GDBT model after taking N(e.g.20) neighbors from each point by Bert embeddings and creating a dataset with size len(test_df)*N. We create XGB models for each rank of the candidates. We first search the rank point(excluding itself) from each point and create a dataset with a size of len(test_df) to train one XGB model. Similarly, we create another XGB model from the second rank point, and so on. Finally we created 50 models. This is very efficient approach in memory, training time and accuracy.</li>\n<li>Sample weights were set according to the number of POI matches.</li>\n<li>ForestInference was used during inference. It was very fast and helpful.</li>\n</ul>\n<h3>Post process(dijkstra)</h3>\n<ul>\n<li>To generate the set of matched point, we defined a graph from the output of XGB models and then used Dijkstra's algorithm. We add edges x -&gt; y for each point x and its candidate y and set the weight to 1.0 - (prediction of XGB model). We then generate the set of matched point by collecting all points whose distances from the focussing point are smaller than a threshold. This process reduces the number of FN.</li>\n<li>Before executing Dijkstra's algorithm, we make the graph undirected by averaging the weigths of  forward edge x -&gt; y and inverse edge y -&gt; x.</li>\n<li>We tried various binary operations (sum, prod, max, etc…) of weights to change the distance on the graph. The usual summation works best for us. </li>\n</ul>\n<h3>LB vs CV</h3>\n<ul>\n<li>As we discussed before, our approach is pretty leaked. Our best CV is 0.997. But we can see almost linear CV-LB relationship with this leaked pipeline. This approach caused problem, our score stopped increasing when the CV score was close to 1.0</li>\n<li>LB vs CV plot<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F194365%2F8a61a2976e19f08c444f0a11e881e856%2F05926d0c5dca1180991ecc1ad152ad17.png?generation=1657291410405512&amp;alt=media\" alt=\"\"></li>\n</ul>\n<h3>Hardware</h3>\n<ul>\n<li>Our team did not have sufficient GPU, and Google Colab(pro) was not powerful enough for Bert training. So we employed Vast.ai, which is relatively inexpensive compared to GCE or AWS, and allowed us to use good GPU.</li>\n<li>For XGB training, Kaggle Kernel was enough. We can finish XGB training within one hour.</li>\n</ul>\n<h3>Score Timeline</h3>\n<ul>\n<li>0.922 baseline: bert-base-multilingual-cased</li>\n<li>0.928 +ensemble xlm-roberta-base(2 ensemble total)</li>\n<li>0.931 +add XGB feature</li>\n<li>0.940 +fill na</li>\n<li>0.945 +add bert models(4 ensemble total)</li>\n<li>0.947 +dijkstra postprocessing</li>\n</ul>\n<h3>Not worked for us</h3>\n<ul>\n<li>shuffle text augmentation</li>\n<li>reverse geocoding text augmentation</li>\n<li>text unidecode -&gt; deberta-large-v3</li>\n</ul>\n<h3>Postscript: When should we notice the leak?</h3>\n<p>In retrospect, we had several opportunities to know the data leaks.</p>\n<ul>\n<li>We noticed that there are a overlap items of <code>train.csv</code> in sample <code>test.csv</code>.</li>\n<li>BERT training on all data scored much higher than the single fold model.</li>\n<li>XGBoost parameters which are more aggressive, gave better scores.</li>\n</ul>",
  "comments": [
    {
      "id": 1849906,
      "author_name": "The Devastator",
      "author_url": "",
      "post_date": "2022-07-09T23:36:33.550000",
      "content": "<p>Thanks for sharing your approach! This is really helpful.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1848452,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-07-08T16:24:15.720000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 1848320,
      "postDate": "2022-07-08T14:14:47.430Z",
      "content": "<p>First of all I would like to thank the team member and, participants and Foursquare.<br>\nExcept the data leak, the problem design was very interesting and a good competition with many possible approaches.</p>\n<p>Our team may have been too pure to suspect the data leak, (because the host explicitly declared that there was no overlaps).</p>\n<p>ATTENTION: This notes was written before the data leak was known, and some points may seem silly for now, but we will share it.</p>\n<h3>Links</h3>\n<ul>\n<li>Training(XGB) notebook: <a href=\"https://www.kaggle.com/code/iiyamaiiyama/fs-training-4-ensemble-0705/notebook?scriptVersionId=100082426\" target=\"_blank\">https://www.kaggle.com/code/iiyamaiiyama/fs-training-4-ensemble-0705/notebook?scriptVersionId=100082426</a></li>\n<li>Inference notebook: <a href=\"https://www.kaggle.com/code/iiyamaiiyama/fs-inference-no54-spatial-no-country-07045/notebook?scriptVersionId=100093618\" target=\"_blank\">https://www.kaggle.com/code/iiyamaiiyama/fs-inference-no54-spatial-no-country-07045/notebook?scriptVersionId=100093618</a> </li>\n<li>NN models traing code (github): <a href=\"https://github.com/heartkilla/mcd-kaggle-fsq\" target=\"_blank\">https://github.com/heartkilla/mcd-kaggle-fsq</a></li>\n</ul>\n<h3>Summary</h3>\n<ul>\n<li>Metric learning multi-input(concatenated text -&gt; BERT + lat/lon) NN model.</li>\n<li>Generate candidate by NN model embeddings and spatial nearest neighbours.</li>\n<li>Create 2nd stage binary classification XGB models for each candidate rank.</li>\n</ul>\n<h3>Validation strategy</h3>\n<ul>\n<li>In the beginning we split training data in 4 folds, and trained BERT(train:valid = 3:1). Then we pick up one fold(validation), and split it further in 4 folds for 2nd stage XGB model training.</li>\n<li>After a while, we found that we can get much better results if we trained BERT using all the data. In this case we train XGB with leaked data and it may perform badly for the test data. Maybe the quality of embeddings is more important than traning a leak-free XGB model.</li>\n</ul>\n<h3>Preprocessing</h3>\n<ul>\n<li>Fill <code>NaN</code> texts with nearest(in longitude and latitude) 5 points for BERT models.</li>\n</ul>\n<h3>1st stage: Create candidates</h3>\n<ul>\n<li>BERT embeding candidates<ul>\n<li>Architechture(BERT + lat/lon)<ul>\n<li>Concatenate BERT model output with normalized lat/lon values and connected to the FC layer(320).</li>\n<li>FC layer(320) embeddings were used for similarity.</li>\n<li>Loss: ArcMarginProduct</li>\n<li>Training<ul>\n<li>Gradually increase Margin during training(0.2 to 0.8)</li>\n<li>40 epoch training(24-48 hours to train)</li></ul></li></ul></li>\n<li>Emsemble<ul>\n<li>Concatenate multiple BERT models embeddings<ul>\n<li><code>xlm-roBERTa-large</code>, <code>sentence-transformers/LaBSE</code>, <code>sentence-transformers/paraphrase-multilingual-mpnet-base-v2</code>, <code>remBERT</code></li></ul></li></ul></li>\n<li>DBA/QE<ul>\n<li>Weighted DBA/QE for better embedding.</li></ul></li>\n<li>Create candidates<ul>\n<li>For each id, 50 candidates were created using cosine similarity (faiss)</li></ul></li></ul></li>\n<li>Spatial candidates<ul>\n<li>Added candidates from lat/lon using sklearn.neighbors.BallTree.</li>\n<li>Hoping to add candidates that were missed by BERT embeddings.</li></ul></li>\n</ul>\n<h3>2nd stage: GBDT models for each candidate rank</h3>\n<ul>\n<li>Many public kernels train GDBT model after taking N(e.g.20) neighbors from each point by Bert embeddings and creating a dataset with size len(test_df)*N. We create XGB models for each rank of the candidates. We first search the rank point(excluding itself) from each point and create a dataset with a size of len(test_df) to train one XGB model. Similarly, we create another XGB model from the second rank point, and so on. Finally we created 50 models. This is very efficient approach in memory, training time and accuracy.</li>\n<li>Sample weights were set according to the number of POI matches.</li>\n<li>ForestInference was used during inference. It was very fast and helpful.</li>\n</ul>\n<h3>Post process(dijkstra)</h3>\n<ul>\n<li>To generate the set of matched point, we defined a graph from the output of XGB models and then used Dijkstra's algorithm. We add edges x -&gt; y for each point x and its candidate y and set the weight to 1.0 - (prediction of XGB model). We then generate the set of matched point by collecting all points whose distances from the focussing point are smaller than a threshold. This process reduces the number of FN.</li>\n<li>Before executing Dijkstra's algorithm, we make the graph undirected by averaging the weigths of  forward edge x -&gt; y and inverse edge y -&gt; x.</li>\n<li>We tried various binary operations (sum, prod, max, etc…) of weights to change the distance on the graph. The usual summation works best for us. </li>\n</ul>\n<h3>LB vs CV</h3>\n<ul>\n<li>As we discussed before, our approach is pretty leaked. Our best CV is 0.997. But we can see almost linear CV-LB relationship with this leaked pipeline. This approach caused problem, our score stopped increasing when the CV score was close to 1.0</li>\n<li>LB vs CV plot<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F194365%2F8a61a2976e19f08c444f0a11e881e856%2F05926d0c5dca1180991ecc1ad152ad17.png?generation=1657291410405512&amp;alt=media\" alt=\"\"></li>\n</ul>\n<h3>Hardware</h3>\n<ul>\n<li>Our team did not have sufficient GPU, and Google Colab(pro) was not powerful enough for Bert training. So we employed Vast.ai, which is relatively inexpensive compared to GCE or AWS, and allowed us to use good GPU.</li>\n<li>For XGB training, Kaggle Kernel was enough. We can finish XGB training within one hour.</li>\n</ul>\n<h3>Score Timeline</h3>\n<ul>\n<li>0.922 baseline: bert-base-multilingual-cased</li>\n<li>0.928 +ensemble xlm-roberta-base(2 ensemble total)</li>\n<li>0.931 +add XGB feature</li>\n<li>0.940 +fill na</li>\n<li>0.945 +add bert models(4 ensemble total)</li>\n<li>0.947 +dijkstra postprocessing</li>\n</ul>\n<h3>Not worked for us</h3>\n<ul>\n<li>shuffle text augmentation</li>\n<li>reverse geocoding text augmentation</li>\n<li>text unidecode -&gt; deberta-large-v3</li>\n</ul>\n<h3>Postscript: When should we notice the leak?</h3>\n<p>In retrospect, we had several opportunities to know the data leaks.</p>\n<ul>\n<li>We noticed that there are a overlap items of <code>train.csv</code> in sample <code>test.csv</code>.</li>\n<li>BERT training on all data scored much higher than the single fold model.</li>\n<li>XGBoost parameters which are more aggressive, gave better scores.</li>\n</ul>",
      "rawMarkdown": "First of all I would like to thank the team member and, participants and Foursquare.\nExcept the data leak, the problem design was very interesting and a good competition with many possible approaches.\n\nOur team may have been too pure to suspect the data leak, (because the host explicitly declared that there was no overlaps).\n\nATTENTION: This notes was written before the data leak was known, and some points may seem silly for now, but we will share it.\n\n### Links\n- Training(XGB) notebook: https://www.kaggle.com/code/iiyamaiiyama/fs-training-4-ensemble-0705/notebook?scriptVersionId=100082426\n- Inference notebook: https://www.kaggle.com/code/iiyamaiiyama/fs-inference-no54-spatial-no-country-07045/notebook?scriptVersionId=100093618 \n- NN models traing code (github): https://github.com/heartkilla/mcd-kaggle-fsq\n\n### Summary\n- Metric learning multi-input(concatenated text -> BERT + lat/lon) NN model.\n- Generate candidate by NN model embeddings and spatial nearest neighbours.\n- Create 2nd stage binary classification XGB models for each candidate rank.\n\n### Validation strategy\n- In the beginning we split training data in 4 folds, and trained BERT(train:valid = 3:1). Then we pick up one fold(validation), and split it further in 4 folds for 2nd stage XGB model training.\n- After a while, we found that we can get much better results if we trained BERT using all the data. In this case we train XGB with leaked data and it may perform badly for the test data. Maybe the quality of embeddings is more important than traning a leak-free XGB model.\n\n### Preprocessing\n- Fill `NaN` texts with nearest(in longitude and latitude) 5 points for BERT models.\n\n### 1st stage: Create candidates\n- BERT embeding candidates\n    - Architechture(BERT + lat/lon)\n        - Concatenate BERT model output with normalized lat/lon values and connected to the FC layer(320).\n        - FC layer(320) embeddings were used for similarity.\n  \t- Loss: ArcMarginProduct\n  \t- Training\n \t\t- Gradually increase Margin during training(0.2 to 0.8)\n \t    - 40 epoch training(24-48 hours to train)\n    - Emsemble\n        - Concatenate multiple BERT models embeddings\n \t\t\t- `xlm-roBERTa-large`, `sentence-transformers/LaBSE`, `sentence-transformers/paraphrase-multilingual-mpnet-base-v2`, `remBERT`\n    - DBA/QE\n        - Weighted DBA/QE for better embedding.\n    - Create candidates\n        - For each id, 50 candidates were created using cosine similarity (faiss)\n- Spatial candidates\n    - Added candidates from lat/lon using sklearn.neighbors.BallTree.\n    - Hoping to add candidates that were missed by BERT embeddings.\n\n### 2nd stage: GBDT models for each candidate rank\n- Many public kernels train GDBT model after taking N(e.g.20) neighbors from each point by Bert embeddings and creating a dataset with size len(test_df)*N. We create XGB models for each rank of the candidates. We first search the rank point(excluding itself) from each point and create a dataset with a size of len(test_df) to train one XGB model. Similarly, we create another XGB model from the second rank point, and so on. Finally we created 50 models. This is very efficient approach in memory, training time and accuracy.\n- Sample weights were set according to the number of POI matches.\n- ForestInference was used during inference. It was very fast and helpful.\n\n### Post process(dijkstra)\n- To generate the set of matched point, we defined a graph from the output of XGB models and then used Dijkstra's algorithm. We add edges x -> y for each point x and its candidate y and set the weight to 1.0 - (prediction of XGB model). We then generate the set of matched point by collecting all points whose distances from the focussing point are smaller than a threshold. This process reduces the number of FN.\n- Before executing Dijkstra's algorithm, we make the graph undirected by averaging the weigths of  forward edge x -> y and inverse edge y -> x.\n- We tried various binary operations (sum, prod, max, etc...) of weights to change the distance on the graph. The usual summation works best for us. \n\n### LB vs CV\n- As we discussed before, our approach is pretty leaked. Our best CV is 0.997. But we can see almost linear CV-LB relationship with this leaked pipeline. This approach caused problem, our score stopped increasing when the CV score was close to 1.0\n- LB vs CV plot\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F194365%2F8a61a2976e19f08c444f0a11e881e856%2F05926d0c5dca1180991ecc1ad152ad17.png?generation=1657291410405512&alt=media)\n### Hardware\n- Our team did not have sufficient GPU, and Google Colab(pro) was not powerful enough for Bert training. So we employed Vast.ai, which is relatively inexpensive compared to GCE or AWS, and allowed us to use good GPU.\n- For XGB training, Kaggle Kernel was enough. We can finish XGB training within one hour.\n\n### Score Timeline\n- 0.922 baseline: bert-base-multilingual-cased\n- 0.928 +ensemble xlm-roberta-base(2 ensemble total)\n- 0.931 +add XGB feature\n- 0.940 +fill na\n- 0.945 +add bert models(4 ensemble total)\n- 0.947 +dijkstra postprocessing\n\n###  Not worked for us\n- shuffle text augmentation\n- reverse geocoding text augmentation\n- text unidecode -> deberta-large-v3\n\n### Postscript: When should we notice the leak?\nIn retrospect, we had several opportunities to know the data leaks.\n- We noticed that there are a overlap items of `train.csv` in sample `test.csv`.\n- BERT training on all data scored much higher than the single fold model.\n- XGBoost parameters which are more aggressive, gave better scores.\n",
      "votes": 35
    },
    {
      "id": 1849906,
      "postDate": "2022-07-09T23:36:33.550Z",
      "content": "<p>Thanks for sharing your approach! This is really helpful.</p>",
      "rawMarkdown": "Thanks for sharing your approach! This is really helpful.\n",
      "votes": 1
    },
    {
      "id": 1848452,
      "postDate": "2022-07-08T16:24:15.720Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-22T08:26:08Z",
  "competition_id": 35476,
  "competition_slug": "foursquare-location-matching",
  "team_id": 8551952,
  "solution_url": "/competitions/foursquare-location-matching/writeups/mc-digital-11th-place-code-and-solution"
}