Our post-processing strategy is very similar to the 1st place team.
\n\nThe observation is that spearman is related to the ranks of samples, i.e. it based on the relative values but not the absolute values of the predictions and it prefer predictions that has the similar distribution to the labels.
\n\nOur assumptions are:
\n\nThe model we built cannot predict the absolute values well for those very imbalanced columns like question_type_spelling, but it can, hopefully, differentiate the examples of higher labels and lower labels.
For each column, the distribution of labels of the public/private test set are close to that of the trainning set.
So what we have to do is just help our model to align the predictions according to the distribution of this column in the trainning set. It turns out that our assumptions may not hold true for every column, so we analyze the performance of this method on every column and ban the post-processing on several of them.
\n\nThe related notebook is Here
\n\nOur final solution consists of 5 models: bert-base-uncased, bert-wwm-uncased, bert-wwm-cased, bert-large-uncased, roberta-large. This schema lacks the diversity, but we cannot get good results on many other pre-trained models. We train each model with 5 folds and 5 epochs.
\n\nWe noticed that generally the better the original prediction, the more benefits we can gain from post-processing, so for each fold, we simply average predictions of 5 models to get a more accurate prediction and then apply PP on it, thus giving us 5 predictions for the test set. Then average the prediction of all folds and get only one prediction.
\n\nIt turns out that the average process, though very powerful and necessary, will change the distribution of the prediction to some extent, so we just heuristically apply the PP again to the prediction.
", "comments": [ { "id": 744700, "author_name": "zidaoziyan", "author_url": "", "post_date": "2020-02-13T03:36:50.140000", "content": "Is anyone preparing for internship recruitment, and if so, preparing for progress together, my WeChat is zhengyc1025
", "votes": 0, "replies": [] }, { "id": 743647, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-02-12T06:50:06.480000", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @pyxiea
", "votes": 0, "replies": [ { "id": 743660, "author_name": "pyxiea", "author_url": "", "post_date": "2020-02-12T07:03:57.823000", "content": "Thanks!
", "votes": 0, "replies": [] } ] } ], "comment_count": 3, "messages": [ { "id": 743464, "postDate": "2020-02-12T04:44:08.803Z", "content": "Our post-processing strategy is very similar to the 1st place team.
\n\nThe observation is that spearman is related to the ranks of samples, i.e. it based on the relative values but not the absolute values of the predictions and it prefer predictions that has the similar distribution to the labels.
\n\nOur assumptions are:
\n\nThe model we built cannot predict the absolute values well for those very imbalanced columns like question_type_spelling, but it can, hopefully, differentiate the examples of higher labels and lower labels.
For each column, the distribution of labels of the public/private test set are close to that of the trainning set.
So what we have to do is just help our model to align the predictions according to the distribution of this column in the trainning set. It turns out that our assumptions may not hold true for every column, so we analyze the performance of this method on every column and ban the post-processing on several of them.
\n\nThe related notebook is Here
\n\nOur final solution consists of 5 models: bert-base-uncased, bert-wwm-uncased, bert-wwm-cased, bert-large-uncased, roberta-large. This schema lacks the diversity, but we cannot get good results on many other pre-trained models. We train each model with 5 folds and 5 epochs.
\n\nWe noticed that generally the better the original prediction, the more benefits we can gain from post-processing, so for each fold, we simply average predictions of 5 models to get a more accurate prediction and then apply PP on it, thus giving us 5 predictions for the test set. Then average the prediction of all folds and get only one prediction.
\n\nIt turns out that the average process, though very powerful and necessary, will change the distribution of the prediction to some extent, so we just heuristically apply the PP again to the prediction.
", "rawMarkdown": "## Post Processing\n\nOur post-processing strategy is very similar to [the 1st place team](https://www.kaggle.com/c/google-quest-challenge/discussion/129840). \n\nThe observation is that spearman is related to the ranks of samples, i.e. it based on the relative values but not the absolute values of the predictions and it prefer predictions that has the similar distribution to the labels.\n\nOur assumptions are:\n\n1. The model we built cannot predict the absolute values well for those very imbalanced columns like `question_type_spelling`, but it can, hopefully, differentiate the examples of higher labels and lower labels.\n\n2. For each column, the distribution of labels of the public/private test set are close to that of the trainning set.\n\nSo what we have to do is just help our model to **align** the predictions according to the distribution of this column in the trainning set. It turns out that our assumptions may not hold true for every column, so we analyze the performance of this method on every column and ban the post-processing on several of them.\n\nThe related notebook is [Here](https://www.kaggle.com/pyxiea/postprocessing-for-google-quest)\n\n## Ensembling\n\nOur final solution consists of 5 models: bert-base-uncased, bert-wwm-uncased, bert-wwm-cased, bert-large-uncased, roberta-large. This schema lacks the diversity, but we cannot get good results on many other pre-trained models. We train each model with 5 folds and 5 epochs. \n\nWe noticed that generally the better the original prediction, the more benefits we can gain from post-processing, so for each fold, we simply average predictions of 5 models to get a more accurate prediction and then apply PP on it, thus giving us 5 predictions for the test set. Then average the prediction of all folds and get only one prediction.\n\nIt turns out that the average process, though very powerful and necessary, will change the distribution of the prediction to some extent, so we just heuristically apply the PP again to the prediction.", "votes": 15 }, { "id": 744700, "postDate": "2020-02-13T03:36:50.140Z", "content": "Is anyone preparing for internship recruitment, and if so, preparing for progress together, my WeChat is zhengyc1025
", "rawMarkdown": "Is anyone preparing for internship recruitment, and if so, preparing for progress together, my WeChat is zhengyc1025\n" }, { "id": 743647, "postDate": "2020-02-12T06:50:06.480Z", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @pyxiea
", "rawMarkdown": "Congratulations!\nThanks for Sharing your Approach & Insights!! @pyxiea ", "replies": [ { "id": 743660, "postDate": "2020-02-12T07:03:57.823Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:08:46Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4139129, "solution_url": "/competitions/google-quest-challenge/writeups/7th-place-post-processing-and-ensembling" }