Thank you very much for organizing this interesting and exciting competition, and thank you for teaming up with me, @no9more9ria10, @tomyanabe, @tkm2261, @yukia18 .
\nThis post is part of the 2nd place solution in LB 0.949 (LB 0.971 with leak).
\n\nOther part
\n\nUsing the candidate matches created in the previous stage, we ensemble three models: GBDT (XGBoost, lightGBM), BERT(xlm-roberta-base).
\nFor this id-pair (e.g., src_id, dst_id), GBDT mainly used the edit distance feature, while BERT used text created in particular columns. These models classified whether the pair matched or not.
XGBoost features were created by colum2131 ( @columbia2131 ), and lightGBM features by ria ( @no9more9ria10 ).
\nAlthough there are some differences in each feature, the following XGBoost’s features were below:
The cross-validation of the train data was purposely split so that leakage would occur.
\nSpecifically, instead of the folds with GroupKFold(n_splits=2, group=‘point_of_interest’), we created new folds with GroupKFold(n_splits=5, group=‘src_id’).
\nThis leakage improved the CV and Public LB because train and test data contained common records.
\nWe suspect that the latitude and longitude variables and the ordinal categories worked effectively as features to represent the common id’s records.
The text was defined as each lowercased name and cateogires connected by a SEP-token and country.
\nAdditionally, the numerical feature was logarithmic variables of haversine distance.
def get_basearray(cfg, df):\n cfg.idx2id = df['id'].fillna('').to_numpy()\n cfg.id2idx = {c: i for i, c in enumerate(cfg.idx2id)}\n\n cfg.idx2name = df['name'].fillna('').str.lower().to_numpy()\n cfg.idx2categories = df['categories'].fillna('').str.lower().to_numpy()\n cfg.idx2address = df['address'].fillna('').str.lower().to_numpy() \n return cfg\n\ndef get_text(cfg, i, j): # {i: index of src_id, j: index of dst_id}\n text = cfg.idx2country[i]+cfg.sep+\\\n 'name1 '+cfg.idx2name[i]+cfg.sep+'category1 '+cfg.idx2categories[i]+\\\n cfg.sep + cfg.sep +\\\n 'name2 '+cfg.idx2name[j]+cfg.sep+'category2 '+cfg.idx2categories[j]\n return text\nThe architecture of BERT is below:
\nclass CustomModel(nn.Module):\n def __init__(self, cfg, criterion):\n super().__init__()\n self.cfg = cfg\n self.criterion = criterion\n self.config = AutoConfig.from_pretrained(\n cfg.MODEL_PATH,\n output_hidden_states=True\n )\n self.config.attention_probs_dropout_prob = 0.0\n self.config.hidden_dropout_prob = 0.0\n self.backbone = AutoModel.from_pretrained(\n cfg.MODEL_PATH, # xlm-roberta-base\n config=self.config\n )\n self.linear1 = nn.Sequential(\n nn.Linear(self.config.hidden_size+len(cfg.num_cols), 1024),\n nn.SELU(),\n nn.Linear(1024, 1)\n )\n\n def forward(self, inputs, labels=None):\n outputs = self.backbone(\n inputs['input_ids'],\n attention_mask=inputs['attention_mask']\n )[\"last_hidden_state\"]\n outputs = outputs[:, 0, :]\n outputs = torch.cat((\n outputs,\n inputs['num_features']\n ), dim=1)\n logits = self.linear1(outputs).flatten()\n loss = self.criterion(logits, labels)\n return logits, loss\nThis model was trained for 5 epochs on all train data.
\nAs mentioned in other discussions, there was a duplication of train data in the test data.
\nTherefore, the some test data was merged with the train data, and matches were created with the same poi. The prediction of LB0.949 was overwritten.
\nWe processed this matches overwriting the matches of LB0.949.
\nNote that train.csv is replaced on submit, so we created a separate train.csv dataset.
import cudf\ntrain = cudf.read_feather(‘../input/4sq-train-data-copy-feather/train.ftr’).fillna(‘[NAN]‘)\ntest = cudf.read_csv(‘../input/foursquare-location-matching/test.csv’, dtype=str).fillna(‘[NAN]’)\ncols = [‘name’, ‘latitude’, ‘longitude’]\ntest = test.merge(train[cols+[‘point_of_interest’]], on=cols, how=‘left’)\ntest[[‘id’, ‘point_of_interest’]].to_pandas().to_csv(‘id2poi.csv’, index=False)\n\nid2poi = pd.read_csv('id2poi.csv')\n\nmatch_dict = {}\nfor poi, group_df in id2poi.groupby('point_of_interest'):\n ids = set(group_df['id'].tolist())\n for id_str in ids:\n match_dict[id_str] = ids\n\nsub = pd.DataFrame({\n ‘id’: sub_dict.keys(),\n ‘matches’: [\n match_dict.get(key, value)\n for key, value in sub_dict.items()\n ]\n})\nsub[‘matches’] = sub[‘matches’].map(lambda x: ' ’.join(x))\nhttp://sub.to_csv(‘submission.csv’, index=False)\nThanks for sharing your insights on this competition!
\nGreat job on your success in this competition!
I couldn't find the RapidFuzz library during the competition.
\nThank you for sharing the solution and congrats for your 2nd place in this competition.
Thank you very much for organizing this interesting and exciting competition, and thank you for teaming up with me, @no9more9ria10, @tomyanabe, @tkm2261, @yukia18 .
\nThis post is part of the 2nd place solution in LB 0.949 (LB 0.971 with leak).
\n\nOther part
\n\nUsing the candidate matches created in the previous stage, we ensemble three models: GBDT (XGBoost, lightGBM), BERT(xlm-roberta-base).
\nFor this id-pair (e.g., src_id, dst_id), GBDT mainly used the edit distance feature, while BERT used text created in particular columns. These models classified whether the pair matched or not.
XGBoost features were created by colum2131 ( @columbia2131 ), and lightGBM features by ria ( @no9more9ria10 ).
\nAlthough there are some differences in each feature, the following XGBoost’s features were below:
The cross-validation of the train data was purposely split so that leakage would occur.
\nSpecifically, instead of the folds with GroupKFold(n_splits=2, group=‘point_of_interest’), we created new folds with GroupKFold(n_splits=5, group=‘src_id’).
\nThis leakage improved the CV and Public LB because train and test data contained common records.
\nWe suspect that the latitude and longitude variables and the ordinal categories worked effectively as features to represent the common id’s records.
The text was defined as each lowercased name and cateogires connected by a SEP-token and country.
\nAdditionally, the numerical feature was logarithmic variables of haversine distance.
def get_basearray(cfg, df):\n cfg.idx2id = df['id'].fillna('').to_numpy()\n cfg.id2idx = {c: i for i, c in enumerate(cfg.idx2id)}\n\n cfg.idx2name = df['name'].fillna('').str.lower().to_numpy()\n cfg.idx2categories = df['categories'].fillna('').str.lower().to_numpy()\n cfg.idx2address = df['address'].fillna('').str.lower().to_numpy() \n return cfg\n\ndef get_text(cfg, i, j): # {i: index of src_id, j: index of dst_id}\n text = cfg.idx2country[i]+cfg.sep+\\\n 'name1 '+cfg.idx2name[i]+cfg.sep+'category1 '+cfg.idx2categories[i]+\\\n cfg.sep + cfg.sep +\\\n 'name2 '+cfg.idx2name[j]+cfg.sep+'category2 '+cfg.idx2categories[j]\n return text\nThe architecture of BERT is below:
\nclass CustomModel(nn.Module):\n def __init__(self, cfg, criterion):\n super().__init__()\n self.cfg = cfg\n self.criterion = criterion\n self.config = AutoConfig.from_pretrained(\n cfg.MODEL_PATH,\n output_hidden_states=True\n )\n self.config.attention_probs_dropout_prob = 0.0\n self.config.hidden_dropout_prob = 0.0\n self.backbone = AutoModel.from_pretrained(\n cfg.MODEL_PATH, # xlm-roberta-base\n config=self.config\n )\n self.linear1 = nn.Sequential(\n nn.Linear(self.config.hidden_size+len(cfg.num_cols), 1024),\n nn.SELU(),\n nn.Linear(1024, 1)\n )\n\n def forward(self, inputs, labels=None):\n outputs = self.backbone(\n inputs['input_ids'],\n attention_mask=inputs['attention_mask']\n )[\"last_hidden_state\"]\n outputs = outputs[:, 0, :]\n outputs = torch.cat((\n outputs,\n inputs['num_features']\n ), dim=1)\n logits = self.linear1(outputs).flatten()\n loss = self.criterion(logits, labels)\n return logits, loss\nThis model was trained for 5 epochs on all train data.
\nAs mentioned in other discussions, there was a duplication of train data in the test data.
\nTherefore, the some test data was merged with the train data, and matches were created with the same poi. The prediction of LB0.949 was overwritten.
\nWe processed this matches overwriting the matches of LB0.949.
\nNote that train.csv is replaced on submit, so we created a separate train.csv dataset.
import cudf\ntrain = cudf.read_feather(‘../input/4sq-train-data-copy-feather/train.ftr’).fillna(‘[NAN]‘)\ntest = cudf.read_csv(‘../input/foursquare-location-matching/test.csv’, dtype=str).fillna(‘[NAN]’)\ncols = [‘name’, ‘latitude’, ‘longitude’]\ntest = test.merge(train[cols+[‘point_of_interest’]], on=cols, how=‘left’)\ntest[[‘id’, ‘point_of_interest’]].to_pandas().to_csv(‘id2poi.csv’, index=False)\n\nid2poi = pd.read_csv('id2poi.csv')\n\nmatch_dict = {}\nfor poi, group_df in id2poi.groupby('point_of_interest'):\n ids = set(group_df['id'].tolist())\n for id_str in ids:\n match_dict[id_str] = ids\n\nsub = pd.DataFrame({\n ‘id’: sub_dict.keys(),\n ‘matches’: [\n match_dict.get(key, value)\n for key, value in sub_dict.items()\n ]\n})\nsub[‘matches’] = sub[‘matches’].map(lambda x: ' ’.join(x))\nhttp://sub.to_csv(‘submission.csv’, index=False)\nThanks for sharing your insights on this competition!
\nGreat job on your success in this competition!
I couldn't find the RapidFuzz library during the competition.
\nThank you for sharing the solution and congrats for your 2nd place in this competition.