First of all thanks to Kaggle and the host for this competition. I enjoyed it quite a bit as this competition needed a lot of memory management (OOM is painful), NLP and GBDT related pipelines.
\nIt was fun working with @locbaop who put in lots of hard work in this.
\nOur solution is reasonably simple:
\nPre-process
\nWe preprocess the text by Pykakashi for JP text and unidecode+lower for other countries.
Blocking
\nAt the start I was mostly iterating using distance measure, and that kept us in 84-85 LB range. Then we tried TFIDF - 3 ngrams+char_analyzer and it immediately gave boost.
Most of our effort was spend on pretraining the SBERT models, in total we trained 5 SBERT models and used the concat embeddings in KNN for blocking. We could get a MAX IOU of > 99 with concatenation of paraphrase-xlm-r-multilingual-v1 & msmarco-bert-base-dot-v5 embeddings. We train SBERT for 1 epoch on full data.
In total we created 20 neighs per ID.
\nBinary Classification Features & Model
\nLightGBM model with heavy regularization.
We considered name, address, category, and another one based on ceil and floor of latitude/longitude, like below
\ndf['text'] = df['name'] + '[SEP]' + df['address'] + '[SEP]' + df['categories']+ '[SEP]'+ \\\n lat_int_floor + '[SEP]' +long_int_floor + '[SEP]'+ lat_int_ceil + '[SEP]' +long_int_ceil
âď¸ This is also the text on which SBERT is trained.
\nWe used embedding similarity for all 5 SBERT models in train. Giving us local CV on 20% valid data ~94.3.
\nI also added counts as a last try and saw immediate boost to local logloss and CV improving to 94.5. Unfortunately I didn't have enough time to do full training and submit.
\nPost Processing
\n Same as public notebooks.
Inference
\nWe used inference forest from cuML to do infer, I think there is some accuracy drop due to lower casting (float64 to float32), but I was really impressed with cuML's abilities on GPU. Will explore this more.
What did not work
\nThats it, goes without saying we couldn't figure out the leak!
\nThanks,
", "comments": [ { "id": 1847870, "author_name": "Psi", "author_url": "", "post_date": "2022-07-08T08:06:16.410000", "content": "\n\nCouldn't get mDebertaV3 to work, lots of NAN weights in model state dict (lol)
\n
There is a bug with float16 / mixed precision for MDeberta, need to train it in float32 in huggingface.
", "votes": 2, "replies": [ { "id": 1848174, "author_name": "Phaedrus", "author_url": "", "post_date": "2022-07-08T12:29:55.193000", "content": "I didn't know about this, thanks!
", "votes": 0, "replies": [] } ] }, { "id": 1847804, "author_name": "Bao Loc Pham", "author_url": "", "post_date": "2022-07-08T06:53:56.030000", "content": "Thanks @pheadrus, I'm learned a lots when competing with you.
", "votes": 1, "replies": [ { "id": 1847817, "author_name": "Phaedrus", "author_url": "", "post_date": "2022-07-08T07:02:42.163000", "content": "Was fun competing together! All the best. đŞ
", "votes": 1, "replies": [] } ] }, { "id": 1848721, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-08T21:04:45.457000", "content": "Thanks for sharing your solution! How did you select to use SBERT? did any try any alternative?
", "votes": 0, "replies": [ { "id": 1849031, "author_name": "Phaedrus", "author_url": "", "post_date": "2022-07-09T06:11:57.157000", "content": "I arrived at SBERT by reading through similar cases, my hypothesis was that sentence models should work better here since I was combining multiple features to make a sentence. I didn't try other LMs except mDeberta but failed to train it properly due to fp16 bugs (as per Psi).
", "votes": 0, "replies": [] } ] }, { "id": 1848042, "author_name": "BUUMOO", "author_url": "", "post_date": "2022-07-08T10:09:22.457000", "content": "Thanks for sharing!
\nMay I ask if you are using ArcFace loss/CurricularFace loss to pretrain the sbert? If yes, do you mind sharing some more details on the training tricks you use? As I tried this solution as well and found it very diffcuilt to make it converge.
I trained as shown in SBERT training overview, specifically I used the simple ContrastiveLoss. Moreover, I was not trying to make it converge, but just do an epoch of pretrain on domain text. Check here --> https://www.sbert.net/docs/training/overview.html
", "votes": 0, "replies": [] }, { "id": 1848210, "author_name": "BUUMOO", "author_url": "", "post_date": "2022-07-08T12:58:45.460000", "content": "Thanks for the info. Learned a lot.
", "votes": 1, "replies": [] } ] }, { "id": 1847862, "author_name": "Muhammed Tausif", "author_url": "", "post_date": "2022-07-08T07:56:52.647000", "content": "Thanks for sharing insights @pheadrus.
", "votes": 0, "replies": [] } ], "comment_count": 10, "messages": [ { "id": 1847801, "postDate": "2022-07-08T06:51:55.407Z", "content": "First of all thanks to Kaggle and the host for this competition. I enjoyed it quite a bit as this competition needed a lot of memory management (OOM is painful), NLP and GBDT related pipelines.
\nIt was fun working with @locbaop who put in lots of hard work in this.
\nOur solution is reasonably simple:
\nPre-process
\nWe preprocess the text by Pykakashi for JP text and unidecode+lower for other countries.
Blocking
\nAt the start I was mostly iterating using distance measure, and that kept us in 84-85 LB range. Then we tried TFIDF - 3 ngrams+char_analyzer and it immediately gave boost.
Most of our effort was spend on pretraining the SBERT models, in total we trained 5 SBERT models and used the concat embeddings in KNN for blocking. We could get a MAX IOU of > 99 with concatenation of paraphrase-xlm-r-multilingual-v1 & msmarco-bert-base-dot-v5 embeddings. We train SBERT for 1 epoch on full data.
In total we created 20 neighs per ID.
\nBinary Classification Features & Model
\nLightGBM model with heavy regularization.
We considered name, address, category, and another one based on ceil and floor of latitude/longitude, like below
\ndf['text'] = df['name'] + '[SEP]' + df['address'] + '[SEP]' + df['categories']+ '[SEP]'+ \\\n lat_int_floor + '[SEP]' +long_int_floor + '[SEP]'+ lat_int_ceil + '[SEP]' +long_int_ceil
âď¸ This is also the text on which SBERT is trained.
\nWe used embedding similarity for all 5 SBERT models in train. Giving us local CV on 20% valid data ~94.3.
\nI also added counts as a last try and saw immediate boost to local logloss and CV improving to 94.5. Unfortunately I didn't have enough time to do full training and submit.
\nPost Processing
\n Same as public notebooks.
Inference
\nWe used inference forest from cuML to do infer, I think there is some accuracy drop due to lower casting (float64 to float32), but I was really impressed with cuML's abilities on GPU. Will explore this more.
What did not work
\nThats it, goes without saying we couldn't figure out the leak!
\nThanks,
", "rawMarkdown": "First of all thanks to Kaggle and the host for this competition. I enjoyed it quite a bit as this competition needed a lot of memory management (OOM is painful), NLP and GBDT related pipelines.\n\nIt was fun working with @locbaop who put in lots of hard work in this. \n\nOur solution is reasonably simple:\n\n**Pre-process**\nWe preprocess the text by Pykakashi for JP text and unidecode+lower for other countries. \n\n**Blocking**\nAt the start I was mostly iterating using distance measure, and that kept us in 84-85 LB range. Then we tried TFIDF - 3 ngrams+char_analyzer and it immediately gave boost.\n\nMost of our effort was spend on pretraining the SBERT models, in total we trained **5 SBERT models** and used the concat embeddings in KNN for blocking. We could get a MAX IOU of > 99 with concatenation of `paraphrase-xlm-r-multilingual-v1` & `msmarco-bert-base-dot-v5` embeddings. We train SBERT for 1 epoch on full data.\n\nIn total we created 20 neighs per ID.\n\n**Binary Classification Features & Model**\nLightGBM model with heavy regularization.\n\nWe considered name, address, category, and another one based on ceil and floor of latitude/longitude, like below\n\n`df['text'] = df['name'] + '[SEP]' + df['address'] + '[SEP]' + df['categories']+ '[SEP]'+ \\\n lat_int_floor + '[SEP]' +long_int_floor + '[SEP]'+ lat_int_ceil + '[SEP]' +long_int_ceil`\n\nâď¸ This is also the text on which SBERT is trained. \n\nWe used embedding similarity for all 5 SBERT models in train. Giving us local CV on 20% valid data ~94.3.\n\nI also added counts as a last try and saw immediate boost to local logloss and CV improving to 94.5. Unfortunately I didn't have enough time to do full training and submit.\n\n**Post Processing**\n Same as public notebooks.\n\n**Inference**\nWe used inference forest from cuML to do infer, I think there is some accuracy drop due to lower casting (float64 to float32), but I was really impressed with cuML's abilities on GPU. Will explore this more.\n\n**What did not work**\n- Couldn't get mDebertaV3 to work, lots of NAN weights in model state dict (lol)\n- Adding City, State for SBERT training\n- XGB didn't work for us\n\nThats it, goes without saying we couldn't figure out the leak! \n\nThanks,", "votes": 26 }, { "id": 1847870, "postDate": "2022-07-08T08:06:16.410Z", "content": "\n\nCouldn't get mDebertaV3 to work, lots of NAN weights in model state dict (lol)
\n
There is a bug with float16 / mixed precision for MDeberta, need to train it in float32 in huggingface.
", "rawMarkdown": "> Couldn't get mDebertaV3 to work, lots of NAN weights in model state dict (lol)\n\nThere is a bug with float16 / mixed precision for MDeberta, need to train it in float32 in huggingface.", "votes": 2, "replies": [ { "id": 1848174, "postDate": "2022-07-08T12:29:55.193Z", "content": "I didn't know about this, thanks!
", "rawMarkdown": "I didn't know about this, thanks! " } ] }, { "id": 1847804, "postDate": "2022-07-08T06:53:56.030Z", "content": "Thanks @pheadrus, I'm learned a lots when competing with you.
", "rawMarkdown": "Thanks @pheadrus, I'm learned a lots when competing with you. ", "votes": 1, "replies": [ { "id": 1847817, "postDate": "2022-07-08T07:02:42.163Z", "content": "Was fun competing together! All the best. đŞ
", "rawMarkdown": "Was fun competing together! All the best. đŞ", "votes": 1 } ] }, { "id": 1848721, "postDate": "2022-07-08T21:04:45.457Z", "content": "Thanks for sharing your solution! How did you select to use SBERT? did any try any alternative?
", "rawMarkdown": "Thanks for sharing your solution! How did you select to use SBERT? did any try any alternative?\n", "replies": [ { "id": 1849031, "postDate": "2022-07-09T06:11:57.157Z", "content": "I arrived at SBERT by reading through similar cases, my hypothesis was that sentence models should work better here since I was combining multiple features to make a sentence. I didn't try other LMs except mDeberta but failed to train it properly due to fp16 bugs (as per Psi).
", "rawMarkdown": "I arrived at SBERT by reading through similar cases, my hypothesis was that sentence models should work better here since I was combining multiple features to make a sentence. I didn't try other LMs except mDeberta but failed to train it properly due to fp16 bugs (as per Psi). " } ] }, { "id": 1848042, "postDate": "2022-07-08T10:09:22.457Z", "content": "Thanks for sharing!
\nMay I ask if you are using ArcFace loss/CurricularFace loss to pretrain the sbert? If yes, do you mind sharing some more details on the training tricks you use? As I tried this solution as well and found it very diffcuilt to make it converge.
I trained as shown in SBERT training overview, specifically I used the simple ContrastiveLoss. Moreover, I was not trying to make it converge, but just do an epoch of pretrain on domain text. Check here --> https://www.sbert.net/docs/training/overview.html
", "rawMarkdown": "I trained as shown in SBERT training overview, specifically I used the simple ContrastiveLoss. Moreover, I was not trying to make it converge, but just do an epoch of pretrain on domain text. Check here --> https://www.sbert.net/docs/training/overview.html" }, { "id": 1848210, "postDate": "2022-07-08T12:58:45.460Z", "content": "Thanks for the info. Learned a lot.
", "rawMarkdown": "Thanks for the info. Learned a lot.", "votes": 1 } ] }, { "id": 1847862, "postDate": "2022-07-08T07:56:52.647Z", "content": "Thanks for sharing insights @pheadrus.
", "rawMarkdown": "\nThanks for sharing insights @pheadrus." } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T22:20:02Z", "competition_id": 35476, "competition_slug": "foursquare-location-matching", "team_id": 8502566, "solution_url": "/competitions/foursquare-location-matching/writeups/bulian-ai-21st-place-solution" }