Thank you Kaggle and hosts for providing us with this challenging competition and congratulations to all winners! Also I'd like to thank my teammates @aerdem4, @zhesun and @jingqinnl for their hard work and great insights. This was my first time teaming up and definitely made the competition process all the more enjoyable for me.
\n\nIn short, our final submission is based on a weighted blend of 4 Siamese/double transformer architectures and one USE + feature engineering model coupled with a rounding based post-processing approach.
\n\nI'll start of describing our post-processing strategy, given that as for many other teams this had a massive impact on our performance. In combination with weighted ensembling it improved our 10 fold GroupKFold CV by ~0.05. The general idea is based on rounding predictions downwards to a multiple of some fraction 1/d: \n\ndef scale(x, d):\n if d:\n return (x//(1/d))/d\n return x\n\nSo if d=4 and x = [0.12, 0.3, 0.31, 0.24, 0.7] these values will get rounded to [0.0, 0.25, 0.25, 0.0, 0.5]. For each target column we did a grid search for values of d in [4, 8, 16, 32, 64, None].
In our ensembling we exploited this technique even further, applying the rounding first to individual model predictions and again after taking a linear combination of model predictions. In doing so we did find that using a separate rounding parameter for each model, OOF score improvements would no longer translate to LB. We addressed this by reducing the number of rounding parameters using the same d_local across all models:\n\ny_temp = 0\nfor pred, w in zip(model_preds, ws):\n y_temp += w * scale(pred, d_local) / sum(ws)\ny_temp = scale(y_temp, d_global)\n\nAll ensembling parameters - 2 rounding parameters and 5 model weights - were set using a small grid search optimising the spearman rho metric on OOFs while ignoring question targets for rows with duplicate questions. For all these smart stacking and post-processing tricks the credit goes to @aerdem4.
Our final ensemble consists of:\n- Siamese Roberta base (CV 0.416)\n- Siamese XLNet base (CV 0.414)\n- Double Albert base V2 (CV 0.413)\n- Siamese Bert base uncased (CV 0.410)\n- USE + Feature Engineering model (CV 0.393)
\n\nListed CV scores are 10 fold GroupKFold w/o post-processing. Although, the transformer models scored significantly higher in terms of CV, the USE + Feature Engineering still contributed significantly in the stack (about 0.005 boost on CV and LB).
\n\nAll transformer models were implemented using Pytorch and used the pretrained models from the huggingface Transformers library as backbones. Transformer models were trained locally on one RTX 2080Ti. The USE + feature engineering model was implemented with Keras and trained using Kaggle kernels (code available here https://www.kaggle.com/aerdem4/qa-use-save-model-weights. As this model was developed by my teammates I will rely on them to provide more details regarding features, architecture and training in the comment section if needed.
\n\nApart from the pretrained backbones all transformer architectures were very similar:\n- question_title + question_body and question_title + answer are fed separately as input to a transformer. As for other top teams, this was easily the biggest difference maker in terms of architecture, adding up to 0.01 to CV scores.\n- Average pooling. This improved CV for some models (~ 0.002), but was similar to CLS output for other models.\n- Custom 2 layer deep regression head also taking one hot encoded category feature as input. Improved CV ~0.005 relative to simpler linear regression heads.
Only difference between the 4 transformer architectures is that Roberta, XLNet and Bert all used a Siamese design - i.e. the same transformer (shared weights) is used for both question and answer inputs. For Albert using a separate transformer (non-shared weights) worked better.
\n\nThe training followed the exact same format for all four transformers and consisted of 2 stages.
\n\nFirst stage:\n\nFreeze transformer backbone and fine-tune the regression head for an additional 5 epochs with constant LR of 1e-5. Added about 0.002 to CV for most models.
\n\nNo special pre-processing on text inputs, just the default model specific tokenisers provided by huggingface. For target variables we did find a trick. First rank transform and follow up with min-max scaling. This made it so that target values were much more evenly distributed between 0 and 1, which played well with BCE loss. Gave 0.003 - 0.005 boost on CV.
\n\nCode for training the transformer models and post-processing/ensembling is available here: https://github.com/robinniesert/kaggle-google-quest
", "comments": [ { "id": 749632, "author_name": "", "author_url": "", "post_date": "2020-02-18T20:25:34.787000", "content": "tq u for sharing iam trying to understand code
", "votes": 15, "replies": [] }, { "id": 863344, "author_name": "哈尔的移动城堡", "author_url": "", "post_date": "2020-05-27T08:52:20.713000", "content": "Wow oh , your code are so clean and clearly , you must be very good at programming.
", "votes": 1, "replies": [] }, { "id": 745008, "author_name": "", "author_url": "", "post_date": "2020-02-13T11:43:36.590000", "content": "Thank you and congratulations on your medal.
", "votes": 1, "replies": [] }, { "id": 744667, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-02-13T03:01:25.680000", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @robin111
", "votes": 2, "replies": [] }, { "id": 832068, "author_name": "Bibek", "author_url": "", "post_date": "2020-05-03T19:53:12.810000", "content": "@robin111 Congratulations for your result and thank you for sharing your approach and codes. I'm trying to replicate your results with your codes and was able to train all the models with CV as mentioned in your post. For full replication of your result, could you please open-source your inference kernel(if possible)?
", "votes": 0, "replies": [ { "id": 832102, "author_name": "Robin N", "author_url": "", "post_date": "2020-05-03T21:09:59.330000", "content": "Thank you @bibek777 . I've made the inference kernel public for you here: https://www.kaggle.com/robin111/bert-use-fe-stacking-full-pipeline-v2. The code isn't as clean as on the github repo, but I hope you're still able to make some sense of it.
", "votes": 1, "replies": [] } ] }, { "id": 750227, "author_name": "", "author_url": "", "post_date": "2020-02-19T07:59:39.270000", "content": "tq u for sharing
", "votes": 0, "replies": [] }, { "id": 745030, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-02-13T12:05:48.237000", "content": "Congrats @robin111 and team. Thanks for sharing your solution.
\n\nWould you be sharing your training code as well?
", "votes": 0, "replies": [ { "id": 745047, "author_name": "Robin N", "author_url": "", "post_date": "2020-02-13T12:31:59.473000", "content": "Thanks @sheriytm. Training code for transformer models can be found at the github repo linked at the bottom of the post. For training code for the USE + FE model follow the link to Ahmet's kernel.
", "votes": 1, "replies": [] }, { "id": 745059, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-02-13T12:42:17.753000", "content": "Thanks so much @robin111 .
", "votes": 0, "replies": [] }, { "id": 745063, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-02-13T12:44:26.850000", "content": "@robin111 I just check the link to Ahmet's kernel, it is only inference but no training code.
", "votes": 0, "replies": [] }, { "id": 745094, "author_name": "Robin N", "author_url": "", "post_date": "2020-02-13T13:24:02.970000", "content": "Not sure how you got that impression, it definitely does contain the training code.
", "votes": 1, "replies": [] }, { "id": 745119, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-02-13T14:11:32.010000", "content": "@robin111, wow this is amazing. Sorry I looked at the run time of 724.8 seconds and wrongly assumed it is an inference kernel. My bad. Thanks.
", "votes": 2, "replies": [] }, { "id": 745138, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-02-13T14:43:30.507000", "content": "Inference on public test set takes 2 minutes or so:)
", "votes": 0, "replies": [] } ] } ], "comment_count": 14, "messages": [ { "id": 744584, "postDate": "2020-02-13T00:21:28.473Z", "content": "Thank you Kaggle and hosts for providing us with this challenging competition and congratulations to all winners! Also I'd like to thank my teammates @aerdem4, @zhesun and @jingqinnl for their hard work and great insights. This was my first time teaming up and definitely made the competition process all the more enjoyable for me.
\n\nIn short, our final submission is based on a weighted blend of 4 Siamese/double transformer architectures and one USE + feature engineering model coupled with a rounding based post-processing approach.
\n\nI'll start of describing our post-processing strategy, given that as for many other teams this had a massive impact on our performance. In combination with weighted ensembling it improved our 10 fold GroupKFold CV by ~0.05. The general idea is based on rounding predictions downwards to a multiple of some fraction 1/d: \n\ndef scale(x, d):\n if d:\n return (x//(1/d))/d\n return x\n\nSo if d=4 and x = [0.12, 0.3, 0.31, 0.24, 0.7] these values will get rounded to [0.0, 0.25, 0.25, 0.0, 0.5]. For each target column we did a grid search for values of d in [4, 8, 16, 32, 64, None].
In our ensembling we exploited this technique even further, applying the rounding first to individual model predictions and again after taking a linear combination of model predictions. In doing so we did find that using a separate rounding parameter for each model, OOF score improvements would no longer translate to LB. We addressed this by reducing the number of rounding parameters using the same d_local across all models:\n\ny_temp = 0\nfor pred, w in zip(model_preds, ws):\n y_temp += w * scale(pred, d_local) / sum(ws)\ny_temp = scale(y_temp, d_global)\n\nAll ensembling parameters - 2 rounding parameters and 5 model weights - were set using a small grid search optimising the spearman rho metric on OOFs while ignoring question targets for rows with duplicate questions. For all these smart stacking and post-processing tricks the credit goes to @aerdem4.
Our final ensemble consists of:\n- Siamese Roberta base (CV 0.416)\n- Siamese XLNet base (CV 0.414)\n- Double Albert base V2 (CV 0.413)\n- Siamese Bert base uncased (CV 0.410)\n- USE + Feature Engineering model (CV 0.393)
\n\nListed CV scores are 10 fold GroupKFold w/o post-processing. Although, the transformer models scored significantly higher in terms of CV, the USE + Feature Engineering still contributed significantly in the stack (about 0.005 boost on CV and LB).
\n\nAll transformer models were implemented using Pytorch and used the pretrained models from the huggingface Transformers library as backbones. Transformer models were trained locally on one RTX 2080Ti. The USE + feature engineering model was implemented with Keras and trained using Kaggle kernels (code available here https://www.kaggle.com/aerdem4/qa-use-save-model-weights. As this model was developed by my teammates I will rely on them to provide more details regarding features, architecture and training in the comment section if needed.
\n\nApart from the pretrained backbones all transformer architectures were very similar:\n- question_title + question_body and question_title + answer are fed separately as input to a transformer. As for other top teams, this was easily the biggest difference maker in terms of architecture, adding up to 0.01 to CV scores.\n- Average pooling. This improved CV for some models (~ 0.002), but was similar to CLS output for other models.\n- Custom 2 layer deep regression head also taking one hot encoded category feature as input. Improved CV ~0.005 relative to simpler linear regression heads.
Only difference between the 4 transformer architectures is that Roberta, XLNet and Bert all used a Siamese design - i.e. the same transformer (shared weights) is used for both question and answer inputs. For Albert using a separate transformer (non-shared weights) worked better.
\n\nThe training followed the exact same format for all four transformers and consisted of 2 stages.
\n\nFirst stage:\n\nFreeze transformer backbone and fine-tune the regression head for an additional 5 epochs with constant LR of 1e-5. Added about 0.002 to CV for most models.
\n\nNo special pre-processing on text inputs, just the default model specific tokenisers provided by huggingface. For target variables we did find a trick. First rank transform and follow up with min-max scaling. This made it so that target values were much more evenly distributed between 0 and 1, which played well with BCE loss. Gave 0.003 - 0.005 boost on CV.
\n\nCode for training the transformer models and post-processing/ensembling is available here: https://github.com/robinniesert/kaggle-google-quest
", "rawMarkdown": "Thank you Kaggle and hosts for providing us with this challenging competition and congratulations to all winners! Also I'd like to thank my teammates @aerdem4, @zhesun and @jingqinnl for their hard work and great insights. This was my first time teaming up and definitely made the competition process all the more enjoyable for me.\n\nIn short, our final submission is based on a weighted blend of 4 Siamese/double transformer architectures and one USE + feature engineering model coupled with a rounding based post-processing approach.\n\n### 1. Post-processing/Ensembling\nI'll start of describing our post-processing strategy, given that as for many other teams this had a massive impact on our performance. In combination with weighted ensembling it improved our 10 fold GroupKFold CV by ~0.05. The general idea is based on rounding predictions downwards to a multiple of some fraction `1/d`: \n```\ndef scale(x, d):\n if d:\n return (x//(1/d))/d\n return x\n```\nSo if `d=4` and `x = [0.12, 0.3, 0.31, 0.24, 0.7]` these values will get rounded to `[0.0, 0.25, 0.25, 0.0, 0.5]`. For each target column we did a grid search for values of `d` in `[4, 8, 16, 32, 64, None]`. \n\nIn our ensembling we exploited this technique even further, applying the rounding first to individual model predictions and again after taking a linear combination of model predictions. In doing so we did find that using a separate rounding parameter for each model, OOF score improvements would no longer translate to LB. We addressed this by reducing the number of rounding parameters using the same `d_local` across all models:\n```\ny_temp = 0\nfor pred, w in zip(model_preds, ws):\n y_temp += w * scale(pred, d_local) / sum(ws)\ny_temp = scale(y_temp, d_global)\n```\nAll ensembling parameters - 2 rounding parameters and 5 model weights - were set using a small grid search optimising the spearman rho metric on OOFs while ignoring question targets for rows with duplicate questions. For all these smart stacking and post-processing tricks the credit goes to @aerdem4. \n\n### 2. Models\nOur final ensemble consists of:\n- Siamese Roberta base (CV 0.416)\n- Siamese XLNet base (CV 0.414)\n- Double Albert base V2 (CV 0.413)\n- Siamese Bert base uncased (CV 0.410)\n- USE + Feature Engineering model (CV 0.393)\n\nListed CV scores are 10 fold GroupKFold w/o post-processing. Although, the transformer models scored significantly higher in terms of CV, the USE + Feature Engineering still contributed significantly in the stack (about 0.005 boost on CV and LB).\n\nAll transformer models were implemented using Pytorch and used the pretrained models from the huggingface Transformers library as backbones. Transformer models were trained locally on one RTX 2080Ti. The USE + feature engineering model was implemented with Keras and trained using Kaggle kernels (code available here [https://www.kaggle.com/aerdem4/qa-use-save-model-weights](https://www.kaggle.com/aerdem4/qa-use-save-model-weights). As this model was developed by my teammates I will rely on them to provide more details regarding features, architecture and training in the comment section if needed. \n\nApart from the pretrained backbones all transformer architectures were very similar:\n- `question_title` + `question_body` and `question_title` + `answer` are fed separately as input to a transformer. As for other top teams, this was easily the biggest difference maker in terms of architecture, adding up to 0.01 to CV scores.\n- Average pooling. This improved CV for some models (~ 0.002), but was similar to CLS output for other models.\n- Custom 2 layer deep regression head also taking one hot encoded category feature as input. Improved CV ~0.005 relative to simpler linear regression heads.\n\nOnly difference between the 4 transformer architectures is that Roberta, XLNet and Bert all used a Siamese design - i.e. the same transformer (shared weights) is used for both question and answer inputs. For Albert using a separate transformer (non-shared weights) worked better. \n\n### 3. Training\nThe training followed the exact same format for all four transformers and consisted of 2 stages.\n\n##### First stage:\n- Train for 4 epochs with huggingface AdamW optimiser.\n- Binary cross-entropy loss.\n- One-cycle LR schedule. Uses cosine warmup, followed by cosine decay, whilst having a mirrored schedule for momentum (i.e. cosine decay followed by cosine warmup). \n- Max LR of 1e-3 for the regression head, max LR of 1e-5 for transformer backbones.\n- Accumulated batch size of 8\n\n##### Second stage:\nFreeze transformer backbone and fine-tune the regression head for an additional 5 epochs with constant LR of 1e-5. Added about 0.002 to CV for most models.\n\n### 4. Pre-processing\nNo special pre-processing on text inputs, just the default model specific tokenisers provided by huggingface. For target variables we did find a trick. First rank transform and follow up with min-max scaling. This made it so that target values were much more evenly distributed between 0 and 1, which played well with BCE loss. Gave 0.003 - 0.005 boost on CV.\n\n\nCode for training the transformer models and post-processing/ensembling is available here: [https://github.com/robinniesert/kaggle-google-quest](https://github.com/robinniesert/kaggle-google-quest)", "votes": 30 }, { "id": 749632, "postDate": "2020-02-18T20:25:34.787Z", "content": "tq u for sharing iam trying to understand code
", "rawMarkdown": "tq u for sharing iam trying to understand code", "votes": 15 }, { "id": 863344, "postDate": "2020-05-27T08:52:20.713Z", "content": "Wow oh , your code are so clean and clearly , you must be very good at programming.
", "rawMarkdown": "Wow oh , your code are so clean and clearly , you must be very good at programming.", "votes": 1 }, { "id": 745008, "postDate": "2020-02-13T11:43:36.590Z", "content": "Thank you and congratulations on your medal.
", "rawMarkdown": "Thank you and congratulations on your medal.", "votes": 1 }, { "id": 744667, "postDate": "2020-02-13T03:01:25.680Z", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @robin111
", "rawMarkdown": "Congratulations!\nThanks for Sharing your Approach & Insights!! @robin111 ", "votes": 2 }, { "id": 832068, "postDate": "2020-05-03T19:53:12.810Z", "content": "@robin111 Congratulations for your result and thank you for sharing your approach and codes. I'm trying to replicate your results with your codes and was able to train all the models with CV as mentioned in your post. For full replication of your result, could you please open-source your inference kernel(if possible)?
", "rawMarkdown": "@robin111 Congratulations for your result and thank you for sharing your approach and codes. I'm trying to replicate your results with your codes and was able to train all the models with CV as mentioned in your post. For full replication of your result, could you please open-source your inference kernel(if possible)?", "replies": [ { "id": 832102, "postDate": "2020-05-03T21:09:59.330Z", "content": "Thank you @bibek777 . I've made the inference kernel public for you here: https://www.kaggle.com/robin111/bert-use-fe-stacking-full-pipeline-v2. The code isn't as clean as on the github repo, but I hope you're still able to make some sense of it.
", "rawMarkdown": "Thank you @bibek777 . I've made the inference kernel public for you here: [https://www.kaggle.com/robin111/bert-use-fe-stacking-full-pipeline-v2](https://www.kaggle.com/robin111/bert-use-fe-stacking-full-pipeline-v2). The code isn't as clean as on the github repo, but I hope you're still able to make some sense of it.", "votes": 1 } ] }, { "id": 750227, "postDate": "2020-02-19T07:59:39.270Z", "content": "tq u for sharing
", "rawMarkdown": "tq u for sharing" }, { "id": 745030, "postDate": "2020-02-13T12:05:48.237Z", "content": "Congrats @robin111 and team. Thanks for sharing your solution.
\n\nWould you be sharing your training code as well?
", "rawMarkdown": "Congrats @robin111 and team. Thanks for sharing your solution.\n\nWould you be sharing your training code as well?", "replies": [ { "id": 745047, "postDate": "2020-02-13T12:31:59.473Z", "content": "Thanks @sheriytm. Training code for transformer models can be found at the github repo linked at the bottom of the post. For training code for the USE + FE model follow the link to Ahmet's kernel.
", "rawMarkdown": "Thanks @sheriytm. Training code for transformer models can be found at the github repo linked at the bottom of the post. For training code for the USE + FE model follow the link to Ahmet's kernel.", "votes": 1 }, { "id": 745059, "postDate": "2020-02-13T12:42:17.753Z", "content": "Thanks so much @robin111 .
", "rawMarkdown": "Thanks so much @robin111 ." }, { "id": 745063, "postDate": "2020-02-13T12:44:26.850Z", "content": "@robin111 I just check the link to Ahmet's kernel, it is only inference but no training code.
", "rawMarkdown": "@robin111 I just check the link to Ahmet's kernel, it is only inference but no training code." }, { "id": 745094, "postDate": "2020-02-13T13:24:02.970Z", "content": "Not sure how you got that impression, it definitely does contain the training code.
", "rawMarkdown": "Not sure how you got that impression, it definitely does contain the training code.", "votes": 1 }, { "id": 745119, "postDate": "2020-02-13T14:11:32.010Z", "content": "@robin111, wow this is amazing. Sorry I looked at the run time of 724.8 seconds and wrongly assumed it is an inference kernel. My bad. Thanks.
", "rawMarkdown": "@robin111, wow this is amazing. Sorry I looked at the run time of 724.8 seconds and wrongly assumed it is an inference kernel. My bad. Thanks.", "votes": 2 }, { "id": 745138, "postDate": "2020-02-13T14:43:30.507Z", "content": "Inference on public test set takes 2 minutes or so:)
", "rawMarkdown": "Inference on public test set takes 2 minutes or so:)" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:09:22Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4074053, "solution_url": "/competitions/google-quest-challenge/writeups/coolkidsofutrecht-6th-place-solution" }