Thanks everyone for posting the elegant solutions . Thanks to the organizers and my team mates \n@iru538 @tarandro @mobassir and @shahules
\n\nI had joined the competition early and then left it for PKU .After failing to secure a medal spot in PKU , I joined back the competition and teamed up with Shahul and Mobassir . Both of them had by that time already trained around 13+ vanilla (public Kernel) TF bert + xlnet models and doing prediction at the range of .385 to .387 , but we were not able to creack .400 mark . Then I applied the below post processing (collected from one of the public Kernel )
\n\n```\ntest_preds = final_predictions\ny_train = df_train[output_categories].values\nfor column_ind in range(30):\n curr_column = y_train[:, column_ind]\n values = np.unique(curr_column)\n map_quantiles = []\n for val in values:\n occurrence = np.mean(curr_column == val)\n cummulative = sum(el['occurrence'] for el in map_quantiles)\n map_quantiles.append({'value': val, 'occurrence': occurrence, 'cummulative': cummulative})
\n\nfor quant in map_quantiles:\n pred_col = test_preds[:, column_ind]\n q1, q2 = np.quantile(pred_col, quant['cummulative']), np.quantile(pred_col, min(quant['cummulative'] + quant['occurrence'], 1))\n pred_col[(pred_col >= q1) & (pred_col <= q2)] = quant['value']\n test_preds[:, column_ind] = pred_col\n```\nWe crossed the .401 first time and was able to extend it further with XLNET and Roberta etc till .417
\n\nIn parallel Toru joined our team and had a great single pytorch model with .409 score without post processing . \nSince I was more comfortable with pytorch , I used that model as base and started training the GPT2 and other pytroch BERT models . This took us to .410 based on pytorch models .
\n\nAfter Alexis joined he pointed us to a post processing trick used in Data Sceience Bowl 2018 1st place solution .\"Data Science is Important- Bert says\" . Here is our OptimizedRounder
\n\n```\nfrom functools import partial\nimport scipy as sp
\n\nclass OptimizedRounder(object):\n def init(self,correlation):\n self.correlation = correlation\n self.coef_ = 0\n self.score = 0
\n\ndef _kappa_loss(self, coef, X, y):\n a= X.copy()\n b=y.copy()\n X_p = pd.cut(a, [-np.inf] + list(np.sort(coef)) + [np.inf], labels = [0,1,2])\n\n a[X_p == 0] = 0\n a[X_p == 2] = 1 \n\n #print(\"validation score = {}\".format(spearmanr(a, b).correlation))\n if spearmanr(a, b).correlation < self.correlation:\n self.score = spearmanr(a, b).correlation\n return - spearmanr(a, b).correlation + (self.correlation - spearmanr(a, b).correlation + 1)**10\n else:\n self.score = spearmanr(a, b).correlation\n return - spearmanr(a, b).correlation\n\ndef fit(self, X, y,coef_ini):\n loss_partial = partial(self._kappa_loss, X=X, y=y)\n initial_coef = coef_ini\n self.coef_ = sp.optimize.minimize(loss_partial, initial_coef, method='nelder-mead')\n\ndef coefficients(self):\n return self.coef_['x']\ndef score_fc(self):\n return self.score\n```\nOnce Alexis was onboarded he started applying post processing to our pytorch models and it gave a lot of boost . So we scrapped the TF models (As it was taking more GPU as well) .
\n\nOur final selected solution consists of \nEnsemble of : \n1.BERT_Base 5 Folds (Question Model and Answer Model separate)(Tried three different architecture) + Statistical Features (Boost the score a lot).\n2. Sequence model with Statistical Features \n2. Distillbert Embedding Sequence Model \n3. Distillbert Embedding LSTM model
\n\nWe got a slightly better Private Score using GPT2 in place of Sequential Model , but we didnt select that as final score . Our highest scoring Kernel has : \n1.BERT_Base 5 Folds (Question Model and Answer Model separate)(Tried three different architecture)\n2. GPT2 Base (10 Folds Question and Answer Separate Model)\n3. Simple LSTM model \n4.Sequence Model with USE features
\n\nHere is a Kernel with the Inference which will give some idea of the architectures : \nhttps://www.kaggle.com/phoenix9032/threshold-base-4toru?scriptVersionId=28450007
\n\nWe have not used any trick in training time .
", "comments": [ { "id": 743438, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-12T04:05:43.217000", "content": "Awesome job. Great work Nirjah and team. Your OptimizedRounder looks interesting. I'll need to play with that and see how it works.
", "votes": 1, "replies": [ { "id": 743440, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-02-12T04:08:30.913000", "content": "This was originally posted here @cdeotte
\n\nhttps://www.kaggle.com/naveenasaithambi/optimizedrounder-improved
", "votes": 1, "replies": [] } ] } ], "comment_count": 2, "messages": [ { "id": 743396, "postDate": "2020-02-12T02:59:59Z", "content": "Thanks everyone for posting the elegant solutions . Thanks to the organizers and my team mates \n@iru538 @tarandro @mobassir and @shahules
\n\nI had joined the competition early and then left it for PKU .After failing to secure a medal spot in PKU , I joined back the competition and teamed up with Shahul and Mobassir . Both of them had by that time already trained around 13+ vanilla (public Kernel) TF bert + xlnet models and doing prediction at the range of .385 to .387 , but we were not able to creack .400 mark . Then I applied the below post processing (collected from one of the public Kernel )
\n\n```\ntest_preds = final_predictions\ny_train = df_train[output_categories].values\nfor column_ind in range(30):\n curr_column = y_train[:, column_ind]\n values = np.unique(curr_column)\n map_quantiles = []\n for val in values:\n occurrence = np.mean(curr_column == val)\n cummulative = sum(el['occurrence'] for el in map_quantiles)\n map_quantiles.append({'value': val, 'occurrence': occurrence, 'cummulative': cummulative})
\n\nfor quant in map_quantiles:\n pred_col = test_preds[:, column_ind]\n q1, q2 = np.quantile(pred_col, quant['cummulative']), np.quantile(pred_col, min(quant['cummulative'] + quant['occurrence'], 1))\n pred_col[(pred_col >= q1) & (pred_col <= q2)] = quant['value']\n test_preds[:, column_ind] = pred_col\n```\nWe crossed the .401 first time and was able to extend it further with XLNET and Roberta etc till .417
\n\nIn parallel Toru joined our team and had a great single pytorch model with .409 score without post processing . \nSince I was more comfortable with pytorch , I used that model as base and started training the GPT2 and other pytroch BERT models . This took us to .410 based on pytorch models .
\n\nAfter Alexis joined he pointed us to a post processing trick used in Data Sceience Bowl 2018 1st place solution .\"Data Science is Important- Bert says\" . Here is our OptimizedRounder
\n\n```\nfrom functools import partial\nimport scipy as sp
\n\nclass OptimizedRounder(object):\n def init(self,correlation):\n self.correlation = correlation\n self.coef_ = 0\n self.score = 0
\n\ndef _kappa_loss(self, coef, X, y):\n a= X.copy()\n b=y.copy()\n X_p = pd.cut(a, [-np.inf] + list(np.sort(coef)) + [np.inf], labels = [0,1,2])\n\n a[X_p == 0] = 0\n a[X_p == 2] = 1 \n\n #print(\"validation score = {}\".format(spearmanr(a, b).correlation))\n if spearmanr(a, b).correlation < self.correlation:\n self.score = spearmanr(a, b).correlation\n return - spearmanr(a, b).correlation + (self.correlation - spearmanr(a, b).correlation + 1)**10\n else:\n self.score = spearmanr(a, b).correlation\n return - spearmanr(a, b).correlation\n\ndef fit(self, X, y,coef_ini):\n loss_partial = partial(self._kappa_loss, X=X, y=y)\n initial_coef = coef_ini\n self.coef_ = sp.optimize.minimize(loss_partial, initial_coef, method='nelder-mead')\n\ndef coefficients(self):\n return self.coef_['x']\ndef score_fc(self):\n return self.score\n```\nOnce Alexis was onboarded he started applying post processing to our pytorch models and it gave a lot of boost . So we scrapped the TF models (As it was taking more GPU as well) .
\n\nOur final selected solution consists of \nEnsemble of : \n1.BERT_Base 5 Folds (Question Model and Answer Model separate)(Tried three different architecture) + Statistical Features (Boost the score a lot).\n2. Sequence model with Statistical Features \n2. Distillbert Embedding Sequence Model \n3. Distillbert Embedding LSTM model
\n\nWe got a slightly better Private Score using GPT2 in place of Sequential Model , but we didnt select that as final score . Our highest scoring Kernel has : \n1.BERT_Base 5 Folds (Question Model and Answer Model separate)(Tried three different architecture)\n2. GPT2 Base (10 Folds Question and Answer Separate Model)\n3. Simple LSTM model \n4.Sequence Model with USE features
\n\nHere is a Kernel with the Inference which will give some idea of the architectures : \nhttps://www.kaggle.com/phoenix9032/threshold-base-4toru?scriptVersionId=28450007
\n\nWe have not used any trick in training time .
", "rawMarkdown": "Thanks everyone for posting the elegant solutions . Thanks to the organizers and my team mates \n@iru538 @tarandro @mobassir and @shahules \n\nI had joined the competition early and then left it for PKU .After failing to secure a medal spot in PKU , I joined back the competition and teamed up with Shahul and Mobassir . Both of them had by that time already trained around 13+ vanilla (public Kernel) TF bert + xlnet models and doing prediction at the range of .385 to .387 , but we were not able to creack .400 mark . Then I applied the below post processing (collected from one of the public Kernel ) \n\n```\ntest_preds = final_predictions\ny_train = df_train[output_categories].values\nfor column_ind in range(30):\n curr_column = y_train[:, column_ind]\n values = np.unique(curr_column)\n map_quantiles = []\n for val in values:\n occurrence = np.mean(curr_column == val)\n cummulative = sum(el['occurrence'] for el in map_quantiles)\n map_quantiles.append({'value': val, 'occurrence': occurrence, 'cummulative': cummulative})\n \n for quant in map_quantiles:\n pred_col = test_preds[:, column_ind]\n q1, q2 = np.quantile(pred_col, quant['cummulative']), np.quantile(pred_col, min(quant['cummulative'] + quant['occurrence'], 1))\n pred_col[(pred_col >= q1) & (pred_col <= q2)] = quant['value']\n test_preds[:, column_ind] = pred_col\n```\nWe crossed the .401 first time and was able to extend it further with XLNET and Roberta etc till .417 \n\nIn parallel Toru joined our team and had a great single pytorch model with .409 score without post processing . \nSince I was more comfortable with pytorch , I used that model as base and started training the GPT2 and other pytroch BERT models . This took us to .410 based on pytorch models . \n\nAfter Alexis joined he pointed us to a post processing trick used in Data Sceience Bowl 2018 1st place solution .\"Data Science is Important- Bert says\" . Here is our OptimizedRounder\n\n```\nfrom functools import partial\nimport scipy as sp\n\nclass OptimizedRounder(object):\n def __init__(self,correlation):\n self.correlation = correlation\n self.coef_ = 0\n self.score = 0\n\n def _kappa_loss(self, coef, X, y):\n a= X.copy()\n b=y.copy()\n X_p = pd.cut(a, [-np.inf] + list(np.sort(coef)) + [np.inf], labels = [0,1,2])\n \n a[X_p == 0] = 0\n a[X_p == 2] = 1 \n\n #print(\"validation score = {}\".format(spearmanr(a, b).correlation))\n if spearmanr(a, b).correlation < self.correlation:\n self.score = spearmanr(a, b).correlation\n return - spearmanr(a, b).correlation + (self.correlation - spearmanr(a, b).correlation + 1)**10\n else:\n self.score = spearmanr(a, b).correlation\n return - spearmanr(a, b).correlation\n\n def fit(self, X, y,coef_ini):\n loss_partial = partial(self._kappa_loss, X=X, y=y)\n initial_coef = coef_ini\n self.coef_ = sp.optimize.minimize(loss_partial, initial_coef, method='nelder-mead')\n\n def coefficients(self):\n return self.coef_['x']\n def score_fc(self):\n return self.score\n```\nOnce Alexis was onboarded he started applying post processing to our pytorch models and it gave a lot of boost . So we scrapped the TF models (As it was taking more GPU as well) . \n\nOur final selected solution consists of \nEnsemble of : \n1.BERT_Base 5 Folds (Question Model and Answer Model separate)(Tried three different architecture) + Statistical Features (Boost the score a lot).\n2. Sequence model with Statistical Features \n2. Distillbert Embedding Sequence Model \n3. Distillbert Embedding LSTM model \n\nWe got a slightly better Private Score using GPT2 in place of Sequential Model , but we didnt select that as final score . Our highest scoring Kernel has : \n1.BERT_Base 5 Folds (Question Model and Answer Model separate)(Tried three different architecture)\n2. GPT2 Base (10 Folds Question and Answer Separate Model)\n3. Simple LSTM model \n4.Sequence Model with USE features \n\nHere is a Kernel with the Inference which will give some idea of the architectures : \nhttps://www.kaggle.com/phoenix9032/threshold-base-4toru?scriptVersionId=28450007\n\n\nWe have not used any trick in training time .", "votes": 10 }, { "id": 743438, "postDate": "2020-02-12T04:05:43.217Z", "content": "Awesome job. Great work Nirjah and team. Your OptimizedRounder looks interesting. I'll need to play with that and see how it works.
", "rawMarkdown": "Awesome job. Great work Nirjah and team. Your OptimizedRounder looks interesting. I'll need to play with that and see how it works.", "votes": 1, "replies": [ { "id": 743440, "postDate": "2020-02-12T04:08:30.913Z", "content": "This was originally posted here @cdeotte
\n\nhttps://www.kaggle.com/naveenasaithambi/optimizedrounder-improved
", "rawMarkdown": "This was originally posted here @cdeotte \n\nhttps://www.kaggle.com/naveenasaithambi/optimizedrounder-improved", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:08:36Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4295935, "solution_url": "/competitions/google-quest-challenge/writeups/the-unstoppables-45th-place-simple-solution-4-mode" }