Thanks to all of you how shared your work and ideas during the competition!
\n\nI started this competition late and without any knowledge about BERT or the current transformers architectures. My goal was just to learn about BERT, so I'm very happy to be in the silver zone.
\n\nMy base models were just variants of what was shared in the public kernels:
\n\nbert-large trained over question_titile + question_body + answer.bert-base trained over question_title+question_body and answer, to predict answer and question targets separately.bert-base trained over question_title+question_body, question_body+answer and answer, to predict all targets jointly. I used 5-GroupKFold grouped by question_title. For the training process I first trained the regression head with BERT fixed, and then trained the whole network for 3 epochs.
My final submission was just the average of these three models.
\n\nTwo days previous to the competition end, I was at ~800th position in public leaderboard with ~0.39 score with no chances to enter the medal zone. Thanks to @corochann in this discussion I found out why I wasn't able to go up in the leaderboard: Even when my models were pretty good at predicting scores, the only thing that matter for the evaluation metric is the order. The solution: find groups of target predictions which are very close and set them with the same value, i.e., cluster nearby target predictions, and DBSCAN is pretty good at this task.
\n\n\n\n\nDBSCAN is a density-based clustering non-parametric algorithm: given a set of points in some space, it groups together points that are closely packed together (points with many nearby neighbors), marking as outliers points that lie alone in low-density regions (whose nearest neighbors are too far away).
\n
This algorithm requires two parameters: eps the maximum distance for two points to be considered neighbors (and be assigned to the same cluster), and min_samples the number of samples in a neighborhood for a point to be considered as a core point.
Thus, my post-processing routine was:
\n\npython\ndef cluster_predictions(submission, q_value=0.95):\n submission = submission.copy(deep=True)\n for column in submission.columns[1:]:\n print(column)\n max_dist = (submission[column]\n .sort_values()\n .diff()\n .quantile(q_value))\n print(max_dist)\n if max_dist == 0: continue\n X = submission[column].values.reshape((-1,1))\n clustering = DBSCAN(eps=max_dist, min_samples=2).fit(X)\n print(np.unique(clustering.labels_))\n for cluster_id in np.unique(clustering.labels_):\n if cluster_id == -1: continue\n cluster_idx = np.argwhere(clustering.labels_ == cluster_id)\n cluster_median = np.median(submission.loc[cluster_idx, column])\n submission.loc[cluster_idx, column] = cluster_median\n return submission\n\nfor each target column, I compute the differences between the sorted predictions, and set eps as a high quantile of these differences. I tuned q_value through the 5-GroupKFold predictions and found 0.95 to be a good value. The post-processing gave me a boost of 0.39X -> 0.42X in public LB and 0.37X -> 0.39X in private LB.
With more time and a better ensemble, I think this approach could have achieved a much better correlation in the private leaderboard.
", "comments": [ { "id": 743310, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-12T01:17:22.960000", "content": "Awesome use of DBSCAN. Congrats on winning a Silver medal. I wanted to try DBSCAN too.
\n\nWhen I realized we needed to discretize the continuous predictions, I was going to try different clustering algorithms. I played around with doing it visually first by looking at scatter plots of OOF rank versus True target rank. I could never improve Spearman's Rank correlation by clustering anything besides the left and right tail, so I just used numpy.clip() in the end.
Thanks!! I also thought to use numpy.clip but that meant having to find the cut-off values. I decided to go for DBSCAN as it requires much less tuning, and gave me a great boost in score.
Congrats @mavillan in using DBSCAN to a high jump. I was introduced to it during the TrackML competition but did not think of using it here for post-processing☹️
", "votes": 1, "replies": [] }, { "id": 744576, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-02-13T00:10:01.883000", "content": "Nice approach mavillan. DBSCAN is the main method to get gold medals in the TrackML competition. It is extremely powerful if used properly.
", "votes": 1, "replies": [] }, { "id": 743374, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-02-12T02:38:03.057000", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @mavillan
", "votes": 1, "replies": [] }, { "id": 1044983, "author_name": "howard zhou ", "author_url": "", "post_date": "2020-10-10T09:06:47.263000", "content": "What is this function?
", "votes": 0, "replies": [] }, { "id": 971973, "author_name": "", "author_url": "", "post_date": "2020-08-16T06:15:37.477000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 7, "messages": [ { "id": 743280, "postDate": "2020-02-12T00:37:55.170Z", "content": "Thanks to all of you how shared your work and ideas during the competition!
\n\nI started this competition late and without any knowledge about BERT or the current transformers architectures. My goal was just to learn about BERT, so I'm very happy to be in the silver zone.
\n\nMy base models were just variants of what was shared in the public kernels:
\n\nbert-large trained over question_titile + question_body + answer.bert-base trained over question_title+question_body and answer, to predict answer and question targets separately.bert-base trained over question_title+question_body, question_body+answer and answer, to predict all targets jointly. I used 5-GroupKFold grouped by question_title. For the training process I first trained the regression head with BERT fixed, and then trained the whole network for 3 epochs.
My final submission was just the average of these three models.
\n\nTwo days previous to the competition end, I was at ~800th position in public leaderboard with ~0.39 score with no chances to enter the medal zone. Thanks to @corochann in this discussion I found out why I wasn't able to go up in the leaderboard: Even when my models were pretty good at predicting scores, the only thing that matter for the evaluation metric is the order. The solution: find groups of target predictions which are very close and set them with the same value, i.e., cluster nearby target predictions, and DBSCAN is pretty good at this task.
\n\n\n\n\nDBSCAN is a density-based clustering non-parametric algorithm: given a set of points in some space, it groups together points that are closely packed together (points with many nearby neighbors), marking as outliers points that lie alone in low-density regions (whose nearest neighbors are too far away).
\n
This algorithm requires two parameters: eps the maximum distance for two points to be considered neighbors (and be assigned to the same cluster), and min_samples the number of samples in a neighborhood for a point to be considered as a core point.
Thus, my post-processing routine was:
\n\npython\ndef cluster_predictions(submission, q_value=0.95):\n submission = submission.copy(deep=True)\n for column in submission.columns[1:]:\n print(column)\n max_dist = (submission[column]\n .sort_values()\n .diff()\n .quantile(q_value))\n print(max_dist)\n if max_dist == 0: continue\n X = submission[column].values.reshape((-1,1))\n clustering = DBSCAN(eps=max_dist, min_samples=2).fit(X)\n print(np.unique(clustering.labels_))\n for cluster_id in np.unique(clustering.labels_):\n if cluster_id == -1: continue\n cluster_idx = np.argwhere(clustering.labels_ == cluster_id)\n cluster_median = np.median(submission.loc[cluster_idx, column])\n submission.loc[cluster_idx, column] = cluster_median\n return submission\n\nfor each target column, I compute the differences between the sorted predictions, and set eps as a high quantile of these differences. I tuned q_value through the 5-GroupKFold predictions and found 0.95 to be a good value. The post-processing gave me a boost of 0.39X -> 0.42X in public LB and 0.37X -> 0.39X in private LB.
With more time and a better ensemble, I think this approach could have achieved a much better correlation in the private leaderboard.
", "rawMarkdown": "Thanks to all of you how shared your work and ideas during the competition!\n\nI started this competition late and without any knowledge about BERT or the current transformers architectures. My goal was just to learn about BERT, so I'm very happy to be in the silver zone. \n\n## Base models\nMy base models were just variants of what was shared in the public kernels:\n\n1. `bert-large` trained over `question_titile` + `question_body` + `answer`.\n2. Two `bert-base` trained over `question_title`+`question_body` and `answer`, to predict answer and question targets separately.\n3. Three `bert-base` trained over `question_title`+`question_body`, `question_body`+`answer` and `answer`, to predict all targets jointly. \n\nI used 5-GroupKFold grouped by `question_title`. For the training process I first trained the regression head with BERT fixed, and then trained the whole network for 3 epochs.\n\nMy final submission was just the average of these three models.\n\n## Post-processing\n\nTwo days previous to the competition end, I was at ~800th position in public leaderboard with ~0.39 score with no chances to enter the medal zone. Thanks to @corochann in this [discussion](https://www.kaggle.com/c/google-quest-challenge/discussion/118724) I found out why I wasn't able to go up in the leaderboard: Even when my models were pretty good at predicting scores, the only thing that matter for the evaluation metric is the order. **The solution**: find groups of target predictions which are very close and set them with the same value, i.e., **cluster** nearby target predictions, and [DBSCAN](https://en.wikipedia.org/wiki/DBSCAN) is pretty good at this task.\n\n> DBSCAN is a density-based clustering non-parametric algorithm: given a set of points in some space, it groups together points that are closely packed together (points with many nearby neighbors), marking as outliers points that lie alone in low-density regions (whose nearest neighbors are too far away).\n\nThis algorithm requires two parameters: `eps` the maximum distance for two points to be considered neighbors (and be assigned to the same cluster), and `min_samples` the number of samples in a neighborhood for a point to be considered as a core point.\n\nThus, my post-processing routine was:\n\n```python\ndef cluster_predictions(submission, q_value=0.95):\n\tsubmission = submission.copy(deep=True)\n\tfor column in submission.columns[1:]:\n\t\tprint(column)\n\t\tmax_dist = (submission[column]\n\t\t\t\t .sort_values()\n\t\t\t\t .diff()\n\t\t\t\t .quantile(q_value))\n\t\tprint(max_dist)\n\t\tif max_dist == 0: continue\n\t\tX = submission[column].values.reshape((-1,1))\n\t\tclustering = DBSCAN(eps=max_dist, min_samples=2).fit(X)\n\t\tprint(np.unique(clustering.labels_))\n\t\tfor cluster_id in np.unique(clustering.labels_):\n\t\t\tif cluster_id == -1: continue\n\t\t\tcluster_idx = np.argwhere(clustering.labels_ == cluster_id)\n\t\t\tcluster_median = np.median(submission.loc[cluster_idx, column])\n\t\t\tsubmission.loc[cluster_idx, column] = cluster_median\n\treturn submission\n```\nfor each target column, I compute the differences between the sorted predictions, and set `eps` as a high quantile of these differences. I tuned `q_value` through the 5-GroupKFold predictions and found 0.95 to be a good value. The post-processing gave me a boost of `0.39X -> 0.42X` in public LB and `0.37X -> 0.39X` in private LB. \n\nWith more time and a better ensemble, I think this approach could have achieved a much better correlation in the private leaderboard. \n\n", "votes": 13 }, { "id": 743310, "postDate": "2020-02-12T01:17:22.960Z", "content": "Awesome use of DBSCAN. Congrats on winning a Silver medal. I wanted to try DBSCAN too.
\n\nWhen I realized we needed to discretize the continuous predictions, I was going to try different clustering algorithms. I played around with doing it visually first by looking at scatter plots of OOF rank versus True target rank. I could never improve Spearman's Rank correlation by clustering anything besides the left and right tail, so I just used numpy.clip() in the end.
Thanks!! I also thought to use numpy.clip but that meant having to find the cut-off values. I decided to go for DBSCAN as it requires much less tuning, and gave me a great boost in score.
Congrats @mavillan in using DBSCAN to a high jump. I was introduced to it during the TrackML competition but did not think of using it here for post-processing☹️
", "rawMarkdown": "Congrats @mavillan in using DBSCAN to a high jump. I was introduced to it during the TrackML competition but did not think of using it here for post-processing☹️ ", "votes": 1 }, { "id": 744576, "postDate": "2020-02-13T00:10:01.883Z", "content": "Nice approach mavillan. DBSCAN is the main method to get gold medals in the TrackML competition. It is extremely powerful if used properly.
", "rawMarkdown": "Nice approach mavillan. DBSCAN is the main method to get gold medals in the TrackML competition. It is extremely powerful if used properly. ", "votes": 1 }, { "id": 743374, "postDate": "2020-02-12T02:38:03.057Z", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @mavillan
", "rawMarkdown": "Congratulations!\nThanks for Sharing your Approach & Insights!! @mavillan ", "votes": 1 }, { "id": 1044983, "postDate": "2020-10-10T09:06:47.263Z", "content": "What is this function?
", "rawMarkdown": "What is this function?" }, { "id": 971973, "postDate": "2020-08-16T06:15:37.477Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:08:24Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4230949, "solution_url": "/competitions/google-quest-challenge/writeups/mavillan-77th-place-post-processing-method-dbscan" }