Congratulations to all the winners! Learnt more about BERT-based model architectures in this competition, and would like to share some of my takeaways.
\n\nI only joined the competition in the last two weeks, therefore I didn't spend too much time on the preprocessing step, but some simple tricks did improve the score. Assume that the built-in tokenizers from pretrained BERTs could do a good job for further preprocessing.
\n\n& lt;, & equals;, & gt ;, ...get_text() of BeautifulSoup)\\ and $ (mostly for Latex keywords and some syntax keywords)As a fan of ensemble learning, I always believe that the model diversity is the key for success. I trained 5 BERT-based models with slightly different custom layers, but they share the same structure for BERT embeddings: [CLS] title [SEP] question [SEP] for question text and [CLS] answer [SEP] for the answer text. I only used the last hidden state output embeddings of [CLS] token from BERT models to combine with other layers (pooler output performed worse).
Topology 1: Roberta-Base, Xlnet-Base-Cased
\n\nTopology 2: Roberta-Base, Bert-Base-Uncased, Bert-Base-Cased
\n\nI also discovered that splitting questions and answers into two separate fully-connected layer paths works better than mixing both. It makes sense to me as the labeling of classes by the voters may focus on the content of title+question and answer separately. Categorical embedding layers for host, 1st token of host and category columns contributed the ensemble score.
The learning rate of all models is fixed to 2e-5, also applied ReduceLROnPlateau for LR decay (factor=0.1) and a custom early stopping callback based on validation Spearman score.
The final model is a weighted average of those models with a post processing to optimize ranks.
\n\nReproducibility had been an issue for tensorflow/keras, but this repo from Nvidia helped me to control the determinism to a great deal! Now we can get almost the same result in multiple runs using the same random seed.\nThis gives us a clear view about the relative performance of all experiments, and then we can gradually improve the models by the right setup and approaches.\nhttps://github.com/NVIDIA/tensorflow-determinism
\n\nLost of people were discussing about what is the actual trick/magic that can boost the Spearman correlation score. I was originally having no clues about it, but after studying the definition of Spearman correlation and the patterns inside the training set labels, I discovered that we could utilize fixed percentiles of label values to approximate to the optimal rank in each class.
\n\nI searched from 1 to 100 as the divisor for fixed percentile intervals using out-of-fold prediction from one of the best ensembles. I finally chose 60 as the fixed divisor because it consistently boosted the score on both local CV and public LB (+~0.03-0.05).
\n\nThe code is very simple, given the unique labels of training set as the distribution samples:\n```\ny_labels = df_train[output_categories].copy()\ny_labels = y_labels.values.flatten()\nunique_labels = np.array(sorted(np.unique(y_labels)))\nunique_labels
\n\narray([0. , 0.2 , 0.26666667, 0.3 , 0.33333333,\n 0.33333333, 0.4 , 0.44444444, 0.46666667, 0.5 ,\n 0.53333333, 0.55555556, 0.6 , 0.66666667, 0.66666667,\n 0.7 , 0.73333333, 0.77777778, 0.8 , 0.83333333,\n 0.86666667, 0.88888889, 0.9 , 0.93333333, 1. ])\n```
\n\nI created 60 optimal percentiles:\n```\ndenominator = 60\nq = np.arange(0, 101, 100 / denominator)\nexp_labels = np.percentile(unique_labels, q)\nexp_labels
\n\narray([0. , 0.08 , 0.16 , 0.21333333, 0.24 ,\n 0.26666667, 0.28 , 0.29333333, 0.30666667, 0.32 ,\n 0.33333333, 0.33333333, 0.33333333, 0.34666667, 0.37333333,\n 0.4 , 0.41777778, 0.43555556, 0.44888889, 0.45777778,\n 0.46666667, 0.48 , 0.49333333, 0.50666667, 0.52 ,\n 0.53333333, 0.54222222, 0.55111111, 0.56444444, 0.58222222,\n 0.6 , 0.62666667, 0.65333333, 0.66666667, 0.66666667,\n 0.66666667, 0.68 , 0.69333333, 0.70666667, 0.72 ,\n 0.73333333, 0.75111111, 0.76888889, 0.78222222, 0.79111111,\n 0.8 , 0.81333333, 0.82666667, 0.84 , 0.85333333,\n 0.86666667, 0.87555556, 0.88444444, 0.89111111, 0.89555556,\n 0.9 , 0.91333333, 0.92666667, 0.94666667, 0.97333333,\n 1. ])\n```
\n\nAnd a mapping function to align BERT outputs to the closest percentile value.\n```\ndef optimize_ranks(preds, unique_labels):\n new_preds = np.zeros(preds.shape)\n for i in range(preds.shape[1]):\n interpolate_bins = np.digitize(preds[:, i],\n bins=unique_labels,\n right=False)
\n\n if len(np.unique(interpolate_bins)) == 1:\n # Use original preds\n new_preds[:, i] = preds[:, i]\n else:\n new_preds[:, i] = unique_labels[interpolate_bins]\n\nreturn new_preds\nweights = [1.0, 1.0, 1.0, 1.0, 1.0]\noof_preds = val_ensemble_preds(all_val_preds, weights)\nmagic_preds = optimize_ranks(oof_preds, exp_labels)\nblend_score = compute_spearmanr(outputs, magic_preds)\n```
\n\nThe Spearman correlation will become NaN if the output column contains 1 unique value, because in this case the standard deviation will be zero and caused divide-by-zero problem (submission error). The trick I used is to use original predictions from BERT models for that column.
\n\nHere is a summary table of original scores versus magic-boosted scores:\n| Model | Local CV without Magic | Local CV with Magic | Public LB with Magic | Private LB with Magic |\n|------------------------|-----------------------:|--------------------:|---------------------:|----------------------:|\n| Roberta-Base (T1) | 0.395972 | 0.414739 | 0.43531 | 0.40019 |\n| Xlnet-Base-Cased (T1) | 0.392654 | 0.407847 | 0.42771 | 0.39609 |\n| Roberta-Base (T2) | 0.398664 | 0.422453 | 0.43522 | 0.40242 |\n| Bert-Base-Uncased (T2) | 0.389013 | 0.398852 | 0.41844 | 0.39075 |\n| Bert-Base-Cased (T2) | 0.387040 | 0.400199 | 0.42026 | 0.38455 |\n| Final Ensemble | 0.392669 | 0.438232 | 0.44238 | 0.41208 |
\n\nThey produced worse results on both local CV and LBs\n- SpatialDropout1D for embeddings and Dense dropouts\n- Separate BERT embeddings for title and question\n- Batch normalizations for embeddings and dense layers
\n\nFinal Submission Inference Code:\nhttps://www.kaggle.com/markpeng/ensemble-5models-v4-v7-magic
\n\nFull Code:\nhttps://github.com/guitarmind/kaggle_google_quest_qa_labeling
", "comments": [ { "id": 743775, "author_name": "0xFunky", "author_url": "", "post_date": "2020-02-12T09:44:32.727000", "content": "Congrats! Mark!!!
", "votes": 1, "replies": [] }, { "id": 742823, "author_name": "Manan", "author_url": "", "post_date": "2020-02-11T15:04:45.870000", "content": "Congrats and thanks for sharing your approach.I will try your post processing technique on my model..😀
", "votes": 1, "replies": [] }, { "id": 742651, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-02-11T12:48:22.970000", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @markpeng
", "votes": 1, "replies": [] } ], "comment_count": 3, "messages": [ { "id": 742492, "postDate": "2020-02-11T10:19:49.573Z", "content": "Congratulations to all the winners! Learnt more about BERT-based model architectures in this competition, and would like to share some of my takeaways.
\n\nI only joined the competition in the last two weeks, therefore I didn't spend too much time on the preprocessing step, but some simple tricks did improve the score. Assume that the built-in tokenizers from pretrained BERTs could do a good job for further preprocessing.
\n\n& lt;, & equals;, & gt ;, ...get_text() of BeautifulSoup)\\ and $ (mostly for Latex keywords and some syntax keywords)As a fan of ensemble learning, I always believe that the model diversity is the key for success. I trained 5 BERT-based models with slightly different custom layers, but they share the same structure for BERT embeddings: [CLS] title [SEP] question [SEP] for question text and [CLS] answer [SEP] for the answer text. I only used the last hidden state output embeddings of [CLS] token from BERT models to combine with other layers (pooler output performed worse).
Topology 1: Roberta-Base, Xlnet-Base-Cased
\n\nTopology 2: Roberta-Base, Bert-Base-Uncased, Bert-Base-Cased
\n\nI also discovered that splitting questions and answers into two separate fully-connected layer paths works better than mixing both. It makes sense to me as the labeling of classes by the voters may focus on the content of title+question and answer separately. Categorical embedding layers for host, 1st token of host and category columns contributed the ensemble score.
The learning rate of all models is fixed to 2e-5, also applied ReduceLROnPlateau for LR decay (factor=0.1) and a custom early stopping callback based on validation Spearman score.
The final model is a weighted average of those models with a post processing to optimize ranks.
\n\nReproducibility had been an issue for tensorflow/keras, but this repo from Nvidia helped me to control the determinism to a great deal! Now we can get almost the same result in multiple runs using the same random seed.\nThis gives us a clear view about the relative performance of all experiments, and then we can gradually improve the models by the right setup and approaches.\nhttps://github.com/NVIDIA/tensorflow-determinism
\n\nLost of people were discussing about what is the actual trick/magic that can boost the Spearman correlation score. I was originally having no clues about it, but after studying the definition of Spearman correlation and the patterns inside the training set labels, I discovered that we could utilize fixed percentiles of label values to approximate to the optimal rank in each class.
\n\nI searched from 1 to 100 as the divisor for fixed percentile intervals using out-of-fold prediction from one of the best ensembles. I finally chose 60 as the fixed divisor because it consistently boosted the score on both local CV and public LB (+~0.03-0.05).
\n\nThe code is very simple, given the unique labels of training set as the distribution samples:\n```\ny_labels = df_train[output_categories].copy()\ny_labels = y_labels.values.flatten()\nunique_labels = np.array(sorted(np.unique(y_labels)))\nunique_labels
\n\narray([0. , 0.2 , 0.26666667, 0.3 , 0.33333333,\n 0.33333333, 0.4 , 0.44444444, 0.46666667, 0.5 ,\n 0.53333333, 0.55555556, 0.6 , 0.66666667, 0.66666667,\n 0.7 , 0.73333333, 0.77777778, 0.8 , 0.83333333,\n 0.86666667, 0.88888889, 0.9 , 0.93333333, 1. ])\n```
\n\nI created 60 optimal percentiles:\n```\ndenominator = 60\nq = np.arange(0, 101, 100 / denominator)\nexp_labels = np.percentile(unique_labels, q)\nexp_labels
\n\narray([0. , 0.08 , 0.16 , 0.21333333, 0.24 ,\n 0.26666667, 0.28 , 0.29333333, 0.30666667, 0.32 ,\n 0.33333333, 0.33333333, 0.33333333, 0.34666667, 0.37333333,\n 0.4 , 0.41777778, 0.43555556, 0.44888889, 0.45777778,\n 0.46666667, 0.48 , 0.49333333, 0.50666667, 0.52 ,\n 0.53333333, 0.54222222, 0.55111111, 0.56444444, 0.58222222,\n 0.6 , 0.62666667, 0.65333333, 0.66666667, 0.66666667,\n 0.66666667, 0.68 , 0.69333333, 0.70666667, 0.72 ,\n 0.73333333, 0.75111111, 0.76888889, 0.78222222, 0.79111111,\n 0.8 , 0.81333333, 0.82666667, 0.84 , 0.85333333,\n 0.86666667, 0.87555556, 0.88444444, 0.89111111, 0.89555556,\n 0.9 , 0.91333333, 0.92666667, 0.94666667, 0.97333333,\n 1. ])\n```
\n\nAnd a mapping function to align BERT outputs to the closest percentile value.\n```\ndef optimize_ranks(preds, unique_labels):\n new_preds = np.zeros(preds.shape)\n for i in range(preds.shape[1]):\n interpolate_bins = np.digitize(preds[:, i],\n bins=unique_labels,\n right=False)
\n\n if len(np.unique(interpolate_bins)) == 1:\n # Use original preds\n new_preds[:, i] = preds[:, i]\n else:\n new_preds[:, i] = unique_labels[interpolate_bins]\n\nreturn new_preds\nweights = [1.0, 1.0, 1.0, 1.0, 1.0]\noof_preds = val_ensemble_preds(all_val_preds, weights)\nmagic_preds = optimize_ranks(oof_preds, exp_labels)\nblend_score = compute_spearmanr(outputs, magic_preds)\n```
\n\nThe Spearman correlation will become NaN if the output column contains 1 unique value, because in this case the standard deviation will be zero and caused divide-by-zero problem (submission error). The trick I used is to use original predictions from BERT models for that column.
\n\nHere is a summary table of original scores versus magic-boosted scores:\n| Model | Local CV without Magic | Local CV with Magic | Public LB with Magic | Private LB with Magic |\n|------------------------|-----------------------:|--------------------:|---------------------:|----------------------:|\n| Roberta-Base (T1) | 0.395972 | 0.414739 | 0.43531 | 0.40019 |\n| Xlnet-Base-Cased (T1) | 0.392654 | 0.407847 | 0.42771 | 0.39609 |\n| Roberta-Base (T2) | 0.398664 | 0.422453 | 0.43522 | 0.40242 |\n| Bert-Base-Uncased (T2) | 0.389013 | 0.398852 | 0.41844 | 0.39075 |\n| Bert-Base-Cased (T2) | 0.387040 | 0.400199 | 0.42026 | 0.38455 |\n| Final Ensemble | 0.392669 | 0.438232 | 0.44238 | 0.41208 |
\n\nThey produced worse results on both local CV and LBs\n- SpatialDropout1D for embeddings and Dense dropouts\n- Separate BERT embeddings for title and question\n- Batch normalizations for embeddings and dense layers
\n\nFinal Submission Inference Code:\nhttps://www.kaggle.com/markpeng/ensemble-5models-v4-v7-magic
\n\nFull Code:\nhttps://github.com/guitarmind/kaggle_google_quest_qa_labeling
", "rawMarkdown": "Congratulations to all the winners! Learnt more about BERT-based model architectures in this competition, and would like to share some of my takeaways.\n\n## Text Preprocessing\n\nI only joined the competition in the last two weeks, therefore I didn't spend too much time on the preprocessing step, but some simple tricks did improve the score. Assume that the built-in tokenizers from pretrained BERTs could do a good job for further preprocessing.\n\n - Remove extra white spaces to make text more dense\n - Unescape html entities, like `& lt;`, `& equals;`, `& gt ;`, ...\n - Extract plain text from html tags if found (using `get_text()` of BeautifulSoup)\n - Remove words starting with `\\` and `$` (mostly for Latex keywords and some syntax keywords)\n\n## Custom BERT Models for Ensembles\n\nAs a fan of ensemble learning, I always believe that the **model diversity** is the key for success. I trained 5 BERT-based models with slightly different custom layers, but they share the same structure for BERT embeddings: `[CLS] title [SEP] question [SEP]` for question text and `[CLS] answer [SEP]` for the answer text. I only used the last hidden state output embeddings of `[CLS]` token from BERT models to combine with other layers (pooler output performed worse).\n\n - **Topology 1: Roberta-Base, Xlnet-Base-Cased**\n\n - 2 BERT embeddings (CLS): q_embed, a_embed\n - 3 categorical embeddings: Concat(cate_embeds) -> Dense(128, relu)\n - 2 separate FC layer paths\n - Concat(q_embed, cate_embeds_dense) -> Dense(256, relu) -> Dense(21, sigmoid)\n - Concat(a_embed, cate_embeds_dense) -> Dense(256, relu) -> Dense(9, sigmoid)\n\n - **Topology 2: Roberta-Base, Bert-Base-Uncased, Bert-Base-Cased**\n\n - 2 BERT embeddings (CLS): q_embed, a_embed\n - 2 separate FC layer paths\n - q_embed -> Dense(256, relu) -> Dense(21, sigmoid)\n - a_embed -> Dense(256, relu) -> Dense(9, sigmoid)\n\nI also discovered that splitting questions and answers into two separate fully-connected layer paths works better than mixing both. It makes sense to me as the labeling of classes by the voters may focus on the content of `title+question` and `answer` separately. Categorical embedding layers for `host`, 1st token of `host` and `category` columns contributed the ensemble score.\n\nThe learning rate of all models is fixed to `2e-5`, also applied `ReduceLROnPlateau` for LR decay (factor=0.1) and a custom early stopping callback based on validation Spearman score.\n\nThe final model is a weighted average of those models with a post processing to optimize ranks.\n\n\n## Determinism on TensorFlow 2.1\n\nReproducibility had been an issue for tensorflow/keras, but this repo from Nvidia helped me to control the determinism to a great deal! Now we can get almost the same result in multiple runs using the same random seed.\nThis gives us a clear view about the relative performance of all experiments, and then we can gradually improve the models by the right setup and approaches.\nhttps://github.com/NVIDIA/tensorflow-determinism\n\n## Post Processing Magic\n\nLost of people were discussing about what is the actual trick/magic that can boost the Spearman correlation score. I was originally having no clues about it, but after studying the definition of Spearman correlation and the patterns inside the training set labels, I discovered that we could utilize fixed percentiles of label values to approximate to the optimal rank in each class.\n\nI searched from 1 to 100 as the divisor for fixed percentile intervals using out-of-fold prediction from one of the best ensembles. I finally chose 60 as the fixed divisor because it consistently boosted the score on both local CV and public LB (+~0.03-0.05).\n\nThe code is very simple, given the unique labels of training set as the distribution samples:\n```\ny_labels = df_train[output_categories].copy()\ny_labels = y_labels.values.flatten()\nunique_labels = np.array(sorted(np.unique(y_labels)))\nunique_labels\n\narray([0. , 0.2 , 0.26666667, 0.3 , 0.33333333,\n 0.33333333, 0.4 , 0.44444444, 0.46666667, 0.5 ,\n 0.53333333, 0.55555556, 0.6 , 0.66666667, 0.66666667,\n 0.7 , 0.73333333, 0.77777778, 0.8 , 0.83333333,\n 0.86666667, 0.88888889, 0.9 , 0.93333333, 1. ])\n```\n\nI created 60 optimal percentiles:\n```\ndenominator = 60\nq = np.arange(0, 101, 100 / denominator)\nexp_labels = np.percentile(unique_labels, q)\nexp_labels\n\narray([0. , 0.08 , 0.16 , 0.21333333, 0.24 ,\n 0.26666667, 0.28 , 0.29333333, 0.30666667, 0.32 ,\n 0.33333333, 0.33333333, 0.33333333, 0.34666667, 0.37333333,\n 0.4 , 0.41777778, 0.43555556, 0.44888889, 0.45777778,\n 0.46666667, 0.48 , 0.49333333, 0.50666667, 0.52 ,\n 0.53333333, 0.54222222, 0.55111111, 0.56444444, 0.58222222,\n 0.6 , 0.62666667, 0.65333333, 0.66666667, 0.66666667,\n 0.66666667, 0.68 , 0.69333333, 0.70666667, 0.72 ,\n 0.73333333, 0.75111111, 0.76888889, 0.78222222, 0.79111111,\n 0.8 , 0.81333333, 0.82666667, 0.84 , 0.85333333,\n 0.86666667, 0.87555556, 0.88444444, 0.89111111, 0.89555556,\n 0.9 , 0.91333333, 0.92666667, 0.94666667, 0.97333333,\n 1. ])\n```\n\nAnd a mapping function to align BERT outputs to the closest percentile value.\n```\ndef optimize_ranks(preds, unique_labels):\n new_preds = np.zeros(preds.shape)\n for i in range(preds.shape[1]):\n interpolate_bins = np.digitize(preds[:, i],\n bins=unique_labels,\n right=False)\n \n if len(np.unique(interpolate_bins)) == 1:\n # Use original preds\n new_preds[:, i] = preds[:, i]\n else:\n new_preds[:, i] = unique_labels[interpolate_bins]\n\n return new_preds\n\nweights = [1.0, 1.0, 1.0, 1.0, 1.0]\noof_preds = val_ensemble_preds(all_val_preds, weights)\nmagic_preds = optimize_ranks(oof_preds, exp_labels)\nblend_score = compute_spearmanr(outputs, magic_preds)\n```\n\nThe Spearman correlation will become NaN if the output column contains 1 unique value, because in this case the standard deviation will be zero and caused divide-by-zero problem (submission error). The trick I used is to use original predictions from BERT models for that column.\n\nHere is a summary table of original scores versus magic-boosted scores:\n| Model | Local CV without Magic | Local CV with Magic | Public LB with Magic | Private LB with Magic |\n|------------------------|-----------------------:|--------------------:|---------------------:|----------------------:|\n| Roberta-Base (T1) | 0.395972 | 0.414739 | 0.43531 | 0.40019 |\n| Xlnet-Base-Cased (T1) | 0.392654 | 0.407847 | 0.42771 | 0.39609 |\n| Roberta-Base (T2) | 0.398664 | 0.422453 | 0.43522 | 0.40242 |\n| Bert-Base-Uncased (T2) | 0.389013 | 0.398852 | 0.41844 | 0.39075 |\n| Bert-Base-Cased (T2) | 0.387040 | 0.400199 | 0.42026 | 0.38455 |\n| Final Ensemble | 0.392669 | 0.438232 | 0.44238 | 0.41208 |\n\n\n## Things That Didn't Work for Me\n\nThey produced worse results on both local CV and LBs\n- SpatialDropout1D for embeddings and Dense dropouts\n- Separate BERT embeddings for title and question\n- Batch normalizations for embeddings and dense layers\n\n## Source Code\n\n**Final Submission Inference Code:**\nhttps://www.kaggle.com/markpeng/ensemble-5models-v4-v7-magic\n\n**Full Code:**\nhttps://github.com/guitarmind/kaggle_google_quest_qa_labeling", "votes": 18 }, { "id": 743775, "postDate": "2020-02-12T09:44:32.727Z", "content": "Congrats! Mark!!!
", "rawMarkdown": "Congrats! Mark!!!", "votes": 1 }, { "id": 742823, "postDate": "2020-02-11T15:04:45.870Z", "content": "Congrats and thanks for sharing your approach.I will try your post processing technique on my model..😀
", "rawMarkdown": "Congrats and thanks for sharing your approach.I will try your post processing technique on my model..😀", "votes": 1 }, { "id": 742651, "postDate": "2020-02-11T12:48:22.970Z", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @markpeng
", "rawMarkdown": "Congratulations!\nThanks for Sharing your Approach & Insights!! @markpeng ", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:08:11Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4020723, "solution_url": "/competitions/google-quest-challenge/writeups/mark-peng-30th-sliver-solution-post-processing-mag" }