{
  "id": 348399,
  "title": "6th place solution",
  "content": "<p>First of all, thank you to everyone who organized the competition.<br>\nThe competition was ended over a month ago, I share my solution.<br>\nThis competition's task and data were interesting, but I regretted about the leakage.</p>\n<h1>1. Preprocessing</h1>\n<hr>\n<ul>\n<li>Remove symbols that seem to be noise.<ul>\n<li>About zip and phone, leave only numbers.</li></ul></li>\n<li>Apply pykakasi to JP and unidecode to all data.</li>\n</ul>\n<h1>2. Candidate Generation</h1>\n<hr>\n<ul>\n<li>Generate 28 candidates per id, IoU=0.978.</li>\n<li>Dataset Creation Process<ul>\n<li>Split the data into two parts by poi.</li>\n<li>Create candidates and features for each data.</li>\n<li>Change seed and split the data multiple times.</li>\n<li>Combine them to form a dataset.</li></ul></li>\n</ul>\n<h4>Phase1</h4>\n<ul>\n<li>Extract 6500 points for each point by coordinates, using kdtree.</li>\n</ul>\n<h4>Phase2</h4>\n<ul>\n<li>Calculate the similarity with each point, and select top28 as a candidate by the following weighted sum.<ul>\n<li>square root of the coordinate distance</li>\n<li>1 - jarowinkler: name, categories</li>\n<li>normalized levenshtein: name, categories</li>\n<li>1 - simpson: name splited by ' ', categories splited by ', '</li></ul></li>\n</ul>\n<h1>3. Matching Prediction</h1>\n<hr>\n<h4>Feature</h4>\n<ul>\n<li>coordinate distance rank of kdtree</li>\n<li>language identification from name<ul>\n<li>using top3 labels and probs</li></ul></li>\n<li>coordinate distance<ul>\n<li>haversine, chebyshev, euclidean, manhattan</li></ul></li>\n<li>similarity of the categorical feature<ul>\n<li>LCS(Subsequence/Substring), levenshtein, jarowinkler</li>\n<li>About categories splited by ', ', use similarity, calculated on a brute force basis, mean, min and max.</li></ul></li>\n<li>string length of categorical features</li>\n<li>features about sets based on string and word<ul>\n<li>jaccard, dice, simpson</li></ul></li>\n<li>count encoding<ul>\n<li>coordinate, name, address, url, phone</li></ul></li>\n<li>tf-idf</li>\n<li>rank features about frequency of occurrence<ul>\n<li>country, categories, categories splited by ', ', categories splited by ' '(word)</li>\n<li>Except country, labeling in both of ascending and descending order.</li>\n<li>Labels less than 1% are to be removed.</li></ul></li>\n<li>rank features for each candidate<ul>\n<li>Create a rank of between 28 candidates per id for each feature.</li></ul></li>\n</ul>\n<h4>Model</h4>\n<ul>\n<li>catboost, single model</li>\n<li>remove early stopping &amp; increase iterations (overfit)</li>\n<li>train on all data</li>\n</ul>\n<h1>4. Postprocessing</h1>\n<hr>\n<ul>\n<li>graph probability convolution\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2126876%2F2e034453f5dbb06da3b079ab9326f2cf%2Fgraph%20probability%20convolution.png?generation=1661677435456298&amp;alt=media\" alt=\"\"><ul>\n<li>ex) Updating the prob of the connection from A to B (red line).<ul>\n<li>Extract A's candidates that have B as a candidate.</li>\n<li>Limit to connections having 0.5 or more prob to A.</li>\n<li>Calcurate a weighted average of the probs of the connection with B, using the probs of the connection with A as the weight.</li>\n<li>AB’<br>\n = (AC * CB + AD * DB + AA * AB) / (AC + AD + AA)<br>\n = (0.7 * 0.7 + 0.9 * 0.1 + 1 * 0.6) / (0.7 + 0.9 + 1)<br>\n = 0.45</li></ul></li></ul></li>\n<li>Add nodes that are connected with a prob of 0.5 or more at the adjacent nodes to the prediction.<ul>\n<li>Repeat twice.</li></ul></li>\n</ul>\n<h1>5. Other Tips</h1>\n<hr>\n<ul>\n<li>Build with dict and numpy basically.</li>\n<li>Batch processing of feature creation and prediction to avoid memory errors.</li>\n<li>Parallel processing of the candidate generation with 4 cpu at submission.</li>\n<li>Convert id dtype from str to int.</li>\n</ul>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 1916913,
      "postDate": "2022-08-28T09:13:35.207Z",
      "content": "<p>First of all, thank you to everyone who organized the competition.<br>\nThe competition was ended over a month ago, I share my solution.<br>\nThis competition's task and data were interesting, but I regretted about the leakage.</p>\n<h1>1. Preprocessing</h1>\n<hr>\n<ul>\n<li>Remove symbols that seem to be noise.<ul>\n<li>About zip and phone, leave only numbers.</li></ul></li>\n<li>Apply pykakasi to JP and unidecode to all data.</li>\n</ul>\n<h1>2. Candidate Generation</h1>\n<hr>\n<ul>\n<li>Generate 28 candidates per id, IoU=0.978.</li>\n<li>Dataset Creation Process<ul>\n<li>Split the data into two parts by poi.</li>\n<li>Create candidates and features for each data.</li>\n<li>Change seed and split the data multiple times.</li>\n<li>Combine them to form a dataset.</li></ul></li>\n</ul>\n<h4>Phase1</h4>\n<ul>\n<li>Extract 6500 points for each point by coordinates, using kdtree.</li>\n</ul>\n<h4>Phase2</h4>\n<ul>\n<li>Calculate the similarity with each point, and select top28 as a candidate by the following weighted sum.<ul>\n<li>square root of the coordinate distance</li>\n<li>1 - jarowinkler: name, categories</li>\n<li>normalized levenshtein: name, categories</li>\n<li>1 - simpson: name splited by ' ', categories splited by ', '</li></ul></li>\n</ul>\n<h1>3. Matching Prediction</h1>\n<hr>\n<h4>Feature</h4>\n<ul>\n<li>coordinate distance rank of kdtree</li>\n<li>language identification from name<ul>\n<li>using top3 labels and probs</li></ul></li>\n<li>coordinate distance<ul>\n<li>haversine, chebyshev, euclidean, manhattan</li></ul></li>\n<li>similarity of the categorical feature<ul>\n<li>LCS(Subsequence/Substring), levenshtein, jarowinkler</li>\n<li>About categories splited by ', ', use similarity, calculated on a brute force basis, mean, min and max.</li></ul></li>\n<li>string length of categorical features</li>\n<li>features about sets based on string and word<ul>\n<li>jaccard, dice, simpson</li></ul></li>\n<li>count encoding<ul>\n<li>coordinate, name, address, url, phone</li></ul></li>\n<li>tf-idf</li>\n<li>rank features about frequency of occurrence<ul>\n<li>country, categories, categories splited by ', ', categories splited by ' '(word)</li>\n<li>Except country, labeling in both of ascending and descending order.</li>\n<li>Labels less than 1% are to be removed.</li></ul></li>\n<li>rank features for each candidate<ul>\n<li>Create a rank of between 28 candidates per id for each feature.</li></ul></li>\n</ul>\n<h4>Model</h4>\n<ul>\n<li>catboost, single model</li>\n<li>remove early stopping &amp; increase iterations (overfit)</li>\n<li>train on all data</li>\n</ul>\n<h1>4. Postprocessing</h1>\n<hr>\n<ul>\n<li>graph probability convolution\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2126876%2F2e034453f5dbb06da3b079ab9326f2cf%2Fgraph%20probability%20convolution.png?generation=1661677435456298&amp;alt=media\" alt=\"\"><ul>\n<li>ex) Updating the prob of the connection from A to B (red line).<ul>\n<li>Extract A's candidates that have B as a candidate.</li>\n<li>Limit to connections having 0.5 or more prob to A.</li>\n<li>Calcurate a weighted average of the probs of the connection with B, using the probs of the connection with A as the weight.</li>\n<li>AB’<br>\n = (AC * CB + AD * DB + AA * AB) / (AC + AD + AA)<br>\n = (0.7 * 0.7 + 0.9 * 0.1 + 1 * 0.6) / (0.7 + 0.9 + 1)<br>\n = 0.45</li></ul></li></ul></li>\n<li>Add nodes that are connected with a prob of 0.5 or more at the adjacent nodes to the prediction.<ul>\n<li>Repeat twice.</li></ul></li>\n</ul>\n<h1>5. Other Tips</h1>\n<hr>\n<ul>\n<li>Build with dict and numpy basically.</li>\n<li>Batch processing of feature creation and prediction to avoid memory errors.</li>\n<li>Parallel processing of the candidate generation with 4 cpu at submission.</li>\n<li>Convert id dtype from str to int.</li>\n</ul>",
      "rawMarkdown": "First of all, thank you to everyone who organized the competition.\nThe competition was ended over a month ago, I share my solution.\nThis competition's task and data were interesting, but I regretted about the leakage.\n\n# 1. Preprocessing\n\n---\n\n- Remove symbols that seem to be noise.\n  - About zip and phone, leave only numbers.\n- Apply pykakasi to JP and unidecode to all data.\n\n# 2. Candidate Generation\n\n---\n\n- Generate 28 candidates per id, IoU=0.978.\n- Dataset Creation Process\n    - Split the data into two parts by poi.\n    - Create candidates and features for each data.\n    - Change seed and split the data multiple times.\n    - Combine them to form a dataset.\n\n#### Phase1\n- Extract 6500 points for each point by coordinates, using kdtree.\n\n#### Phase2\n- Calculate the similarity with each point, and select top28 as a candidate by the following weighted sum.\n    - square root of the coordinate distance\n    - 1 - jarowinkler: name, categories\n    - normalized levenshtein: name, categories\n    - 1 - simpson: name splited by ' ', categories splited by ', '\n\n# 3. Matching Prediction\n\n---\n\n#### Feature\n\n- coordinate distance rank of kdtree\n- language identification from name\n    - using top3 labels and probs\n- coordinate distance\n    - haversine, chebyshev, euclidean, manhattan\n- similarity of the categorical feature\n    - LCS(Subsequence/Substring), levenshtein, jarowinkler\n    - About categories splited by ', ', use similarity, calculated on a brute force basis, mean, min and max.\n- string length of categorical features\n- features about sets based on string and word\n    - jaccard, dice, simpson\n- count encoding\n    - coordinate, name, address, url, phone\n- tf-idf\n- rank features about frequency of occurrence\n    - country, categories, categories splited by ', ', categories splited by ' '(word)\n    - Except country, labeling in both of ascending and descending order.\n    - Labels less than 1% are to be removed.\n- rank features for each candidate\n    - Create a rank of between 28 candidates per id for each feature.\n\n#### Model\n\n- catboost, single model\n- remove early stopping & increase iterations (overfit)\n- train on all data\n\n# 4. Postprocessing\n\n---\n\n- graph probability convolution\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2126876%2F2e034453f5dbb06da3b079ab9326f2cf%2Fgraph%20probability%20convolution.png?generation=1661677435456298&alt=media)\n    - ex) Updating the prob of the connection from A to B (red line).\n        - Extract A's candidates that have B as a candidate.\n        - Limit to connections having 0.5 or more prob to A.\n        - Calcurate a weighted average of the probs of the connection with B, using the probs of the connection with A as the weight.\n        - AB’\n             = (AC * CB + AD * DB + AA * AB) / (AC + AD + AA)\n             = (0.7 * 0.7 + 0.9 * 0.1 + 1 * 0.6) / (0.7 + 0.9 + 1)\n             = 0.45\n- Add nodes that are connected with a prob of 0.5 or more at the adjacent nodes to the prediction.\n    - Repeat twice.\n\n# 5. Other Tips\n\n---\n\n- Build with dict and numpy basically.\n- Batch processing of feature creation and prediction to avoid memory errors.\n- Parallel processing of the candidate generation with 4 cpu at submission.\n- Convert id dtype from str to int.",
      "votes": 7
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-23T05:25:25Z",
  "competition_id": 35476,
  "competition_slug": "foursquare-location-matching",
  "team_id": 8626025,
  "solution_url": "/competitions/foursquare-location-matching/writeups/tomo20180402-6th-place-solution"
}