Hello.
\nHere is the solution that got us to 8th place.
\nI apologize if the English is incorrect because I am using an automatic translation.
\nAlso, this is my first time posting in a discussion forum so I may not be able to summarize it well. If you have any questions, please comment.
Like many other competitors, our method consists of three steps.
\nThe basic process is copied from @guoyonfan's baseline. (Thank you for the great baseline.)
\nBelow are some of the points we devised for each step
\nIn addition to the distance-based kNearestNeighbor (distance kNN) used in the baseline, kNearestNeighbor (embedding kNN) using embedding created from fine-tuned SentenceBERT Added.
\nThe k parameters used for the final submission are 10 for distance kNN and 20 for embedding kNN.
\nThe ideal IoU simulated on the training data was 0.986.
The embedding kNN is very important and played a major role as a feature in the binary classification described below.
\nThe embedding model was fine-tuned based on sentence-transformers/all-MiniLM-L12-v2.
The key points that were important to get a good performance with the embedding kNN are
\nThe input of the model is a concatenated string of name, address, city, state, country, category, and latlon.
\nSince the base model is not a multilingual model, names and addresses are normalized by pykakasi for Japanese and by unidecode for other languages.
The reason why we did not use the multilingual model is that the English-based model with normalized inputs simply performs better, but we believe that the reason is that the number of parameters in the multilingual model is very large and the amount of data is insufficient for fine-tuning.
\nUsually, a large amount of training data is required for metric learning, but the number of bases available for this competition was only about 1.1 million, which is not enough data to obtain invariant expressions from data expressed in dozens of languages, let alone a single language.
Another reason why we could not successfully train multilingual models may be that we had to reduce the size of the mini-batches that could be set during training due to the memory limitation of the GPU.
\nThe models were trained on Google Colab using a single GPU, and if more resources and time were available, the multilingual models might perform better.
\nFor the loss function of fine-tuning, we used Contrastive Loss implemented in sentence-transformer. library
\nWe chose this function because it has the best performance compared to TripletLoss and ConsineSimilarityLoss.
\nTo obtain better performance, two stages of fine-tuning were performed.
\nIn the first stage, fine-tuning is performed using the pair set created by distance kNN as training data.
\nIn the second stage, fine-tuning is performed on the training data, using the pair set created by embedding kNN using the model learned in the first stage, in addition to the distance kNN.
We believe that the second stage of fine-tuning contributed as a hard negative to the training data, since the locations with similar names and addresses added by embedding KNN were added to the training data.
\nComparing the first and second stage models, the ideal IoU(recall) improved from 0.97 to 0.986.
\nFor the binary classification, we used 7 LightGBMs created by kFold.
\nBelow are the features that were important to us and our training efforts.
In addition to the features used in the baseline, we added the following
\nfeature importance is as shown in the attached image.
\nLightGBM was trained by optimizing logloss and applying early stopping.
\nThe training data were the same pairs created by distance kNN and embedding kNN.
One of the major findings is that the larger the number of id and the number of pairs in the training data, the larger the stopping iteration of early stopping becomes and the better the score improves.
\nInitially, the public LB was 0.908 after training with 550,000 id, but it improved to 0.928 with 700,000 id, and to 0.948 with 1.1 million id.
\nIn the final submission, seven models were trained using 25 million pairs created from 1.1 million locations.
\nThe number of stop iterations for each model was about 1000.
\n(Unfortunately, due to memory limitations, the kaggle environment did not work, so we used Google Colabe pro+ for training.)
Finally, the model became so large that inference on the CPU took too long, so we used Rapids to speed up the process when submitting. (The inference time was reduced by a factor of 40, which was very impressive.)
\nOmitted because no changes have been made from the baseline.
\nThis was the first competition that I seriously worked on, and there were many points that required a lot of trial and error, and I learned a lot about how to learn the model and how to make the resources more efficient.
\nI would like to thank the competition organizers and Kaggle, a great platform, for giving me this opportunity.
Last but not least, thank you to my delightful teammates.
", "comments": [ { "id": 1849908, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-09T23:39:11.037000", "content": "Thanks for sharing your solution!
\nGreat job on getting to 8th place!
Hello.
\nHere is the solution that got us to 8th place.
\nI apologize if the English is incorrect because I am using an automatic translation.
\nAlso, this is my first time posting in a discussion forum so I may not be able to summarize it well. If you have any questions, please comment.
Like many other competitors, our method consists of three steps.
\nThe basic process is copied from @guoyonfan's baseline. (Thank you for the great baseline.)
\nBelow are some of the points we devised for each step
\nIn addition to the distance-based kNearestNeighbor (distance kNN) used in the baseline, kNearestNeighbor (embedding kNN) using embedding created from fine-tuned SentenceBERT Added.
\nThe k parameters used for the final submission are 10 for distance kNN and 20 for embedding kNN.
\nThe ideal IoU simulated on the training data was 0.986.
The embedding kNN is very important and played a major role as a feature in the binary classification described below.
\nThe embedding model was fine-tuned based on sentence-transformers/all-MiniLM-L12-v2.
The key points that were important to get a good performance with the embedding kNN are
\nThe input of the model is a concatenated string of name, address, city, state, country, category, and latlon.
\nSince the base model is not a multilingual model, names and addresses are normalized by pykakasi for Japanese and by unidecode for other languages.
The reason why we did not use the multilingual model is that the English-based model with normalized inputs simply performs better, but we believe that the reason is that the number of parameters in the multilingual model is very large and the amount of data is insufficient for fine-tuning.
\nUsually, a large amount of training data is required for metric learning, but the number of bases available for this competition was only about 1.1 million, which is not enough data to obtain invariant expressions from data expressed in dozens of languages, let alone a single language.
Another reason why we could not successfully train multilingual models may be that we had to reduce the size of the mini-batches that could be set during training due to the memory limitation of the GPU.
\nThe models were trained on Google Colab using a single GPU, and if more resources and time were available, the multilingual models might perform better.
\nFor the loss function of fine-tuning, we used Contrastive Loss implemented in sentence-transformer. library
\nWe chose this function because it has the best performance compared to TripletLoss and ConsineSimilarityLoss.
\nTo obtain better performance, two stages of fine-tuning were performed.
\nIn the first stage, fine-tuning is performed using the pair set created by distance kNN as training data.
\nIn the second stage, fine-tuning is performed on the training data, using the pair set created by embedding kNN using the model learned in the first stage, in addition to the distance kNN.
We believe that the second stage of fine-tuning contributed as a hard negative to the training data, since the locations with similar names and addresses added by embedding KNN were added to the training data.
\nComparing the first and second stage models, the ideal IoU(recall) improved from 0.97 to 0.986.
\nFor the binary classification, we used 7 LightGBMs created by kFold.
\nBelow are the features that were important to us and our training efforts.
In addition to the features used in the baseline, we added the following
\nfeature importance is as shown in the attached image.
\nLightGBM was trained by optimizing logloss and applying early stopping.
\nThe training data were the same pairs created by distance kNN and embedding kNN.
One of the major findings is that the larger the number of id and the number of pairs in the training data, the larger the stopping iteration of early stopping becomes and the better the score improves.
\nInitially, the public LB was 0.908 after training with 550,000 id, but it improved to 0.928 with 700,000 id, and to 0.948 with 1.1 million id.
\nIn the final submission, seven models were trained using 25 million pairs created from 1.1 million locations.
\nThe number of stop iterations for each model was about 1000.
\n(Unfortunately, due to memory limitations, the kaggle environment did not work, so we used Google Colabe pro+ for training.)
Finally, the model became so large that inference on the CPU took too long, so we used Rapids to speed up the process when submitting. (The inference time was reduced by a factor of 40, which was very impressive.)
\nOmitted because no changes have been made from the baseline.
\nThis was the first competition that I seriously worked on, and there were many points that required a lot of trial and error, and I learned a lot about how to learn the model and how to make the resources more efficient.
\nI would like to thank the competition organizers and Kaggle, a great platform, for giving me this opportunity.
Last but not least, thank you to my delightful teammates.
", "rawMarkdown": "Hello.\n\nHere is the solution that got us to 8th place. \nI apologize if the English is incorrect because I am using an automatic translation. \nAlso, this is my first time posting in a discussion forum so I may not be able to summarize it well. If you have any questions, please comment.\n\n### Overview\n\nLike many other competitors, our method consists of three steps. \n\n1. Creation of candidate matching pairs. \n2. Binary classification to determine if the locations are identical or not. \n3. Post-processing\n\nThe basic process is copied from [@guoyonfan's baseline](https://www.kaggle.com/code/guoyonfan/binary-lgb-baseline-0-834). (Thank you for the great baseline.)\n\nBelow are some of the points we devised for each step\n\n### Creation of candidate matching pairs\n\nIn addition to the distance-based kNearestNeighbor (distance kNN) used in the baseline, kNearestNeighbor (embedding kNN) using embedding created from fine-tuned SentenceBERT Added.\n\nThe k parameters used for the final submission are 10 for distance kNN and 20 for embedding kNN.\nThe ideal IoU simulated on the training data was 0.986.\n\nThe embedding kNN is very important and played a major role as a feature in the binary classification described below.\n\n#### SentenceBERT embedding details\n\nThe embedding model was fine-tuned based on `sentence-transformers/all-MiniLM-L12-v2`. \n\nThe key points that were important to get a good performance with the embedding kNN are\n- Does not use multilingual model\n - In fact, we also use multilingual models to create features for the binary classification step, but the English-based models are by far the most important.\n- Loss selection\n- Two steps fine-tuning\n\n##### does not use a multilingual model.\nThe input of the model is a concatenated string of name, address, city, state, country, category, and latlon. \nSince the base model is not a multilingual model, names and addresses are normalized by pykakasi for Japanese and by unidecode for other languages. \n\nThe reason why we did not use the multilingual model is that the English-based model with normalized inputs simply performs better, but we believe that the reason is that the number of parameters in the multilingual model is very large and the amount of data is insufficient for fine-tuning. \nUsually, a large amount of training data is required for metric learning, but the number of bases available for this competition was only about 1.1 million, which is not enough data to obtain invariant expressions from data expressed in dozens of languages, let alone a single language. \n\nAnother reason why we could not successfully train multilingual models may be that we had to reduce the size of the mini-batches that could be set during training due to the memory limitation of the GPU. \n\nThe models were trained on Google Colab using a single GPU, and if more resources and time were available, the multilingual models might perform better. \n\n##### Loss Selection\nFor the loss function of fine-tuning, we used [Contrastive Loss](https://www.sbert.net/docs/package_reference/losses.html) implemented in sentence-transformer. library \n\nWe chose this function because it has the best performance compared to TripletLoss and ConsineSimilarityLoss.\n\n##### 2-step fine-tuning\n\nTo obtain better performance, two stages of fine-tuning were performed. \n\nIn the first stage, fine-tuning is performed using the pair set created by distance kNN as training data. \nIn the second stage, fine-tuning is performed on the training data, using the pair set created by embedding kNN using the model learned in the first stage, in addition to the distance kNN. \n\nWe believe that the second stage of fine-tuning contributed as a hard negative to the training data, since the locations with similar names and addresses added by embedding KNN were added to the training data. \n\nComparing the first and second stage models, the ideal IoU(recall) improved from 0.97 to 0.986. \n\n### Binary classification to determine if they are the same location\n\nFor the binary classification, we used 7 LightGBMs created by kFold. \nBelow are the features that were important to us and our training efforts. \n\n#### Features that were effective for prediction\n\nIn addition to the features used in the baseline, we added the following\n\n- cosine similarity of the embedding vectors created by the three types of SentenceBERT\n - 1st : multi-lingual model with name, address, latlon, etc. as input + fine-tuning with triplet loss\n - 2nd : same as embedding kNN\n - 3rd: multi-lingual model with only names as input + fine-tuning with contrastive loss\n- Country and category label encoding\n - Label encoding of country and category associated with the id that is the key of the matching pair\n - To avoid over-learning, labels with high frequency in train.csv are selected, and the rest are replaced with \"other\".\n- Land use characteristics by area\n - The following statistics were created using latlon rounded to two decimal places as the key\n - Number of POIs included\n - A 5-dimensional PCA was applied to the frequency distribution of the category labels.\n - The above statistic was combined with the latlon of one id of the matching pair\n- Normalized name tfidf similarity\n - Normalized names in unidecode and pykakasi converted to character base tfidf vector and cosine similarity calculated\n\nfeature importance is as shown in the attached image.\n\n#### Learning LightGBM\n\nLightGBM was trained by optimizing logloss and applying early stopping. \nThe training data were the same pairs created by distance kNN and embedding kNN. \n\nOne of the major findings is that the larger the number of id and the number of pairs in the training data, the larger the stopping iteration of early stopping becomes and the better the score improves. \n\nInitially, the public LB was 0.908 after training with 550,000 id, but it improved to 0.928 with 700,000 id, and to 0.948 with 1.1 million id.\n\nIn the final submission, seven models were trained using 25 million pairs created from 1.1 million locations. \nThe number of stop iterations for each model was about 1000. \n(Unfortunately, due to memory limitations, the kaggle environment did not work, so we used Google Colabe pro+ for training.)\n\nFinally, the model became so large that inference on the CPU took too long, so we used Rapids to speed up the process when submitting. (The inference time was reduced by a factor of 40, which was very impressive.)\n\n### Post-processing\nOmitted because no changes have been made from the baseline.\n\n### Conclusion\nThis was the first competition that I seriously worked on, and there were many points that required a lot of trial and error, and I learned a lot about how to learn the model and how to make the resources more efficient. \nI would like to thank the competition organizers and Kaggle, a great platform, for giving me this opportunity. \n\nLast but not least, thank you to my delightful teammates. \n", "votes": 31 }, { "id": 1849908, "postDate": "2022-07-09T23:39:11.037Z", "content": "Thanks for sharing your solution!
\nGreat job on getting to 8th place!