{
  "id": 335883,
  "title": "23rd place solution (arcmargin)",
  "content": "<p>Congratulations to all medal winners and thanks to Kaggle and Foursquare for such an interesting competition. </p>\n<p>My solution is based on arcmargin models to extract candidate pairs and LigthGBM for matching. </p>\n<p><strong>Blocking</strong>: Generate pair candidates with arcmargin model. This model is trained with arcmargin loss (<a href=\"url\" target=\"_blank\">https://arxiv.org/pdf/1801.07698.pdf</a>)  </p>\n<p>Features: </p>\n<ul>\n<li><p>TFIDF (char level of (1, 3) ngrams) of name, address, zip code. </p></li>\n<li><p>TFIDF (word level) of categories. </p></li>\n<li><p>Onehot of city, country, state and the domain of the url. </p></li>\n<li><p>Binary features: has_phone, has_url, etc. </p></li>\n<li><p>Latitude and longitude encoded with a trick explained below. </p></li>\n</ul>\n<p>Model: </p>\n<p>All the inputs are concatenated and followed by two blocks of Dense-PReLU-BN-Dropout. The output of the last block is the input to the arcmargin layer. </p>\n<ul>\n<li><p>Size of embedding is 512.  </p></li>\n<li><p>It’s important to use L2 regularization in the arcmargin layer (it has a lot of parameters, &gt;200M when is trained with &gt;500k points of interest). I use L2=1e-4 </p></li>\n<li><p>Margin warmup from 0.2 to 0.8. I think it’s not very important, but the model converges faster. </p></li>\n</ul>\n<p>Results: </p>\n<p>These models generate a set of pairs with a max IOU of &gt;0.98.  </p>\n<p>Public LB: 0.89. </p>\n<p><strong>Matching: LightGBM models</strong> </p>\n<p>Features: </p>\n<ul>\n<li><p>Cosine distance between embeddings.  </p></li>\n<li><p>Euclidean, Manhattan, Harvesine distances between lat, lon. </p></li>\n<li><p>Levenhstein, Jaro-Winkler, LCS, distances/similarities (<a href=\"url\" target=\"_blank\">https://github.com/seatgeek/thefuzz</a>). </p></li>\n</ul>\n<p>Results: </p>\n<p>Public LB: 0.908 </p>\n<p><strong>Final submission</strong>: </p>\n<ul>\n<li><p>New arcmargin model based only on text features with XLM-Roberta. Public LB: 0.917 </p></li>\n<li><p>New features in LGB models: Embedding rankings, cosine distance to the nearest neighbor, etc. Public LB: 0.923. </p></li>\n</ul>\n<p><strong>Latitude and longitude encoding trick</strong>: Because the latitude and longitude features are of high precision, it is necessary to encode them in some way to make them useful for the arcmargin model. I have used a technique like positional encoding of transformers to embed these high precision features in vectors. The code used is: </p>\n<pre><code>emb_size = 20 \nprecision = 1e6 \n\nlatlon = np.expand_dims(df[[\"latitude\", \"longitude\"]].values, axis=-1) \n\nm = np.exp(np.log(precision) / emb_size) \nangle_freq = m ** np.arange(emb_size) \nangle_freq = angle_freq.reshape(1, 1, emb_size) \n\nlatlon = latlon * angle_freq \nlatlon[..., 0::2] = np.cos(latlon[..., 0::2]) \nlatlon[..., 1::2] = np.sin(latlon[..., 1::2]) \nlatlon = latlon.reshape(-1, 2 * emb_size) \n</code></pre>",
  "comments": [
    {
      "id": 1848731,
      "author_name": "The Devastator",
      "author_url": "",
      "post_date": "2022-07-08T21:14:54.450000",
      "content": "<p>Thanks for sharing your solution! Your blocking method looks interesting. Can you elaborate more on the 'latitude and longitude encoding trick'?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1888606,
          "author_name": "PLA",
          "author_url": "",
          "post_date": "2022-08-07T17:34:06.190000",
          "content": "<p>Hi, sorry for the late reply. I finished the competition a bit burnt out and have been away on vacation for a while. </p>\n<p>I have prepared a notebook explaining in more detail the 'latitude and longitude encoding trick': <a href=\"https://www.kaggle.com/code/joulazas/latitude-and-longitude-encoding-for-arcmargin\" target=\"_blank\">notebook</a></p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1847977,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-07-08T09:36:44.187000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 3,
  "messages": [
    {
      "id": 1847970,
      "postDate": "2022-07-08T09:32:21.677Z",
      "content": "<p>Congratulations to all medal winners and thanks to Kaggle and Foursquare for such an interesting competition. </p>\n<p>My solution is based on arcmargin models to extract candidate pairs and LigthGBM for matching. </p>\n<p><strong>Blocking</strong>: Generate pair candidates with arcmargin model. This model is trained with arcmargin loss (<a href=\"url\" target=\"_blank\">https://arxiv.org/pdf/1801.07698.pdf</a>)  </p>\n<p>Features: </p>\n<ul>\n<li><p>TFIDF (char level of (1, 3) ngrams) of name, address, zip code. </p></li>\n<li><p>TFIDF (word level) of categories. </p></li>\n<li><p>Onehot of city, country, state and the domain of the url. </p></li>\n<li><p>Binary features: has_phone, has_url, etc. </p></li>\n<li><p>Latitude and longitude encoded with a trick explained below. </p></li>\n</ul>\n<p>Model: </p>\n<p>All the inputs are concatenated and followed by two blocks of Dense-PReLU-BN-Dropout. The output of the last block is the input to the arcmargin layer. </p>\n<ul>\n<li><p>Size of embedding is 512.  </p></li>\n<li><p>It’s important to use L2 regularization in the arcmargin layer (it has a lot of parameters, &gt;200M when is trained with &gt;500k points of interest). I use L2=1e-4 </p></li>\n<li><p>Margin warmup from 0.2 to 0.8. I think it’s not very important, but the model converges faster. </p></li>\n</ul>\n<p>Results: </p>\n<p>These models generate a set of pairs with a max IOU of &gt;0.98.  </p>\n<p>Public LB: 0.89. </p>\n<p><strong>Matching: LightGBM models</strong> </p>\n<p>Features: </p>\n<ul>\n<li><p>Cosine distance between embeddings.  </p></li>\n<li><p>Euclidean, Manhattan, Harvesine distances between lat, lon. </p></li>\n<li><p>Levenhstein, Jaro-Winkler, LCS, distances/similarities (<a href=\"url\" target=\"_blank\">https://github.com/seatgeek/thefuzz</a>). </p></li>\n</ul>\n<p>Results: </p>\n<p>Public LB: 0.908 </p>\n<p><strong>Final submission</strong>: </p>\n<ul>\n<li><p>New arcmargin model based only on text features with XLM-Roberta. Public LB: 0.917 </p></li>\n<li><p>New features in LGB models: Embedding rankings, cosine distance to the nearest neighbor, etc. Public LB: 0.923. </p></li>\n</ul>\n<p><strong>Latitude and longitude encoding trick</strong>: Because the latitude and longitude features are of high precision, it is necessary to encode them in some way to make them useful for the arcmargin model. I have used a technique like positional encoding of transformers to embed these high precision features in vectors. The code used is: </p>\n<pre><code>emb_size = 20 \nprecision = 1e6 \n\nlatlon = np.expand_dims(df[[\"latitude\", \"longitude\"]].values, axis=-1) \n\nm = np.exp(np.log(precision) / emb_size) \nangle_freq = m ** np.arange(emb_size) \nangle_freq = angle_freq.reshape(1, 1, emb_size) \n\nlatlon = latlon * angle_freq \nlatlon[..., 0::2] = np.cos(latlon[..., 0::2]) \nlatlon[..., 1::2] = np.sin(latlon[..., 1::2]) \nlatlon = latlon.reshape(-1, 2 * emb_size) \n</code></pre>",
      "rawMarkdown": "Congratulations to all medal winners and thanks to Kaggle and Foursquare for such an interesting competition. \n\nMy solution is based on arcmargin models to extract candidate pairs and LigthGBM for matching. \n\n**Blocking**: Generate pair candidates with arcmargin model. This model is trained with arcmargin loss ([https://arxiv.org/pdf/1801.07698.pdf](url))  \n\nFeatures: \n\n- TFIDF (char level of (1, 3) ngrams) of name, address, zip code. \n\n- TFIDF (word level) of categories. \n\n- Onehot of city, country, state and the domain of the url. \n\n- Binary features: has_phone, has_url, etc. \n\n- Latitude and longitude encoded with a trick explained below. \n\nModel: \n\nAll the inputs are concatenated and followed by two blocks of Dense-PReLU-BN-Dropout. The output of the last block is the input to the arcmargin layer. \n\n- Size of embedding is 512.  \n\n- It’s important to use L2 regularization in the arcmargin layer (it has a lot of parameters, >200M when is trained with >500k points of interest). I use L2=1e-4 \n\n- Margin warmup from 0.2 to 0.8. I think it’s not very important, but the model converges faster. \n\nResults: \n\nThese models generate a set of pairs with a max IOU of >0.98.  \n\nPublic LB: 0.89. \n\n**Matching: LightGBM models** \n\nFeatures: \n\n- Cosine distance between embeddings.  \n\n- Euclidean, Manhattan, Harvesine distances between lat, lon. \n\n- Levenhstein, Jaro-Winkler, LCS, distances/similarities ([https://github.com/seatgeek/thefuzz](url)). \n\nResults: \n\nPublic LB: 0.908 \n\n**Final submission**: \n\n- New arcmargin model based only on text features with XLM-Roberta. Public LB: 0.917 \n\n- New features in LGB models: Embedding rankings, cosine distance to the nearest neighbor, etc. Public LB: 0.923. \n\n\n**Latitude and longitude encoding trick**: Because the latitude and longitude features are of high precision, it is necessary to encode them in some way to make them useful for the arcmargin model. I have used a technique like positional encoding of transformers to embed these high precision features in vectors. The code used is: \n\n````\nemb_size = 20 \nprecision = 1e6 \n \nlatlon = np.expand_dims(df[[\"latitude\", \"longitude\"]].values, axis=-1) \n \nm = np.exp(np.log(precision) / emb_size) \nangle_freq = m ** np.arange(emb_size) \nangle_freq = angle_freq.reshape(1, 1, emb_size) \n \nlatlon = latlon * angle_freq \nlatlon[..., 0::2] = np.cos(latlon[..., 0::2]) \nlatlon[..., 1::2] = np.sin(latlon[..., 1::2]) \nlatlon = latlon.reshape(-1, 2 * emb_size) \n````",
      "votes": 10
    },
    {
      "id": 1848731,
      "postDate": "2022-07-08T21:14:54.450Z",
      "content": "<p>Thanks for sharing your solution! Your blocking method looks interesting. Can you elaborate more on the 'latitude and longitude encoding trick'?</p>",
      "rawMarkdown": "Thanks for sharing your solution! Your blocking method looks interesting. Can you elaborate more on the 'latitude and longitude encoding trick'?\n",
      "replies": [
        {
          "id": 1888606,
          "postDate": "2022-08-07T17:34:06.190Z",
          "content": "<p>Hi, sorry for the late reply. I finished the competition a bit burnt out and have been away on vacation for a while. </p>\n<p>I have prepared a notebook explaining in more detail the 'latitude and longitude encoding trick': <a href=\"https://www.kaggle.com/code/joulazas/latitude-and-longitude-encoding-for-arcmargin\" target=\"_blank\">notebook</a></p>",
          "rawMarkdown": "Hi, sorry for the late reply. I finished the competition a bit burnt out and have been away on vacation for a while. \n\nI have prepared a notebook explaining in more detail the 'latitude and longitude encoding trick': [notebook](https://www.kaggle.com/code/joulazas/latitude-and-longitude-encoding-for-arcmargin)"
        }
      ]
    },
    {
      "id": 1847977,
      "postDate": "2022-07-08T09:36:44.187Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-23T05:23:21Z",
  "competition_id": 35476,
  "competition_slug": "foursquare-location-matching",
  "team_id": 8547797,
  "solution_url": "/competitions/foursquare-location-matching/writeups/pla-23rd-place-solution-arcmargin"
}