\n2. > Averaging 4 BERT base-uncased model\nQ. What is the difference of the four models?\n\n-> Used the models described as the following figure.\n![\"fig\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F291723%2Fff85f6338fa415af7375021586f8704b%2F2020-02-19%2022.21.16.png?generation=1582119385493091&alt=media\")
Thank you for Google as a host and Kaggle team! I really learned many NLP technics especially BERT form this competition.
\n\n```\nclass OptimPreds(object):\n def init(self, df_train):\n self.score_range_dict = {}\n for i, c in enumerate(df_train.columns[11:]):\n cnt = df_train[c].value_counts(normalize=True).sort_index()\n self.score_range_dict[i] = [cnt.index.values.tolist(), cnt.values.tolist()]\n def predict(self, preds, i):\n return pd.cut(rank_average(preds), [-np.inf] + np.cumsum(self.score_range_dict[i][1])[:-1].tolist() + [np.inf], labels = self.score_range_dict[i][0])
\n\ndef optim_predict(pred, do_round=True, target=[ 2, 5, 7, 9, 11, 12, 13, 14, 15, 16, 19]):\n for i in range(pred.shape[1]):\n if i in target:\n pred[:,i] = optim.predict(pred[:,i], i)\n return pred\n```
", "comments": [ { "id": 750199, "author_name": "guchio3", "author_url": "", "post_date": "2020-02-19T07:34:48.157000", "content": "@kenmatsu4 Thanks for sharing a great solution :D\nCan I ask you some questions?
\n\n```\n1. > Use first and last part of text if length exceed max_seq_length \nQ. How did you set the ratio for the first and last? (ex. 1 : 1, or 2 : 1 or etc …)
\n\n> Averaging 4 BERT base-uncased model \nQ. What is the difference of the four models?
> Fit target distribution with train data (a part of target columns)
\n
Q. How did you choose the part of the target columns? Did you use oof?
> Freeze half of BertLayer for reducing model complexity
\n
Q. Which BertLayers did you freeze?
\n```
Thanks!
", "votes": 1, "replies": [ { "id": 750532, "author_name": "kenmatsu4", "author_url": "", "post_date": "2020-02-19T13:37:49.467000", "content": "Thank you for your question, @guchio3 !\nThe followings are my answer to your questions :D
\n\n\n1. > Use first and last part of text if length exceed max_seq_length\nQ. How did you set the ratio for the first and last? (ex. 1 : 1, or 2 : 1 or etc …)
\n
-> I tried various type of the ratio, however 1 : 1 was the best for me.
\n\n\n2. > Averaging 4 BERT base-uncased model\nQ. What is the difference of the four models?\n\n-> Used the models described as the following figure.\n
\n3. > Fit target distribution with train data (a part of target columns)
Q. How did you choose the part of the target columns? Did you use oof? \n-> Yes, I tried all of the columns and then, pick-up columns which has improvement with the post processing.
\n
\n4. > Freeze half of BertLayer for reducing model complexity
Q. Which BertLayers did you freeze? \n-> Set trainable as False for first half on the list of TFBertLayer class.
\n
Thank you for the quick and detailed answer.\nNow it's clear for me!
", "votes": 1, "replies": [] } ] }, { "id": 742303, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-02-11T08:17:10.937000", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @kenmatsu4
", "votes": 1, "replies": [ { "id": 742345, "author_name": "kenmatsu4", "author_url": "", "post_date": "2020-02-11T08:47:19.600000", "content": "Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 742186, "author_name": "Dracarys", "author_url": "", "post_date": "2020-02-11T06:13:20.450000", "content": "You really did a great job, thanks for sharing.
", "votes": 1, "replies": [ { "id": 742341, "author_name": "kenmatsu4", "author_url": "", "post_date": "2020-02-11T08:46:23.430000", "content": "Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 742278, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2020-02-11T07:48:25.330000", "content": "Let us continue our learning and exciting journey ! @kenmatsu4
", "votes": 2, "replies": [ { "id": 742343, "author_name": "kenmatsu4", "author_url": "", "post_date": "2020-02-11T08:47:11.823000", "content": "Thanks ratthachat! Let's get gold medal next time!
", "votes": 1, "replies": [] } ] } ], "comment_count": 9, "messages": [ { "id": 742168, "postDate": "2020-02-11T05:49:11.387Z", "content": "Thank you for Google as a host and Kaggle team! I really learned many NLP technics especially BERT form this competition.
\n\n```\nclass OptimPreds(object):\n def init(self, df_train):\n self.score_range_dict = {}\n for i, c in enumerate(df_train.columns[11:]):\n cnt = df_train[c].value_counts(normalize=True).sort_index()\n self.score_range_dict[i] = [cnt.index.values.tolist(), cnt.values.tolist()]\n def predict(self, preds, i):\n return pd.cut(rank_average(preds), [-np.inf] + np.cumsum(self.score_range_dict[i][1])[:-1].tolist() + [np.inf], labels = self.score_range_dict[i][0])
\n\ndef optim_predict(pred, do_round=True, target=[ 2, 5, 7, 9, 11, 12, 13, 14, 15, 16, 19]):\n for i in range(pred.shape[1]):\n if i in target:\n pred[:,i] = optim.predict(pred[:,i], i)\n return pred\n```
", "rawMarkdown": "Thank you for Google as a host and Kaggle team! I really learned many NLP technics especially BERT form this competition.\n\n# Key point of final model\n- Use first and last part of text if length exceed max\\_seq\\_length\n- Averaging 4 BERT base-uncased model\n- Postprocessing: Fit target distribution with train data (a part of target columns)\n - Detail is in below.\n- Concatenate pool\\_output & sequence\\_output from bert\\_layer for GlobalAveragePooling1D\n- 10 fold with MultilabelStratifiedKFold (thanks @ratthachat !)\n- Strong heart for solo participation\n\n# Didn’t work for me\n- Pre-training with stackoverflow data (150,000 sentences)\n- Multi sample dropout\n- The other models\n - Roberta\n - Albert\n - XLNet\n- Concatenate question only output & answer only model \n- Concatenate category MLP with BERT model\n- LSTM head instead of Dense with BERT model\n- Freeze half of BertLayer for reducing model complexity\n- Skip half of BertLayer for reducing model complexity\n- USE + MLP\n- LSTM model with gensim embedding\n- custom loss\n - BCE & MSE\n - focal loss\n- Word count feature\n- Concat title and question\\_body as a one block (removing [\"SEP\"] between them)\n- Up-sampling for imbalance target column\n\n# My postprocessing class\n```\nclass OptimPreds(object):\n def __init__(self, df_train):\n self.score_range_dict = {}\n for i, c in enumerate(df_train.columns[11:]):\n cnt = df_train[c].value_counts(normalize=True).sort_index()\n self.score_range_dict[i] = [cnt.index.values.tolist(), cnt.values.tolist()]\n def predict(self, preds, i):\n return pd.cut(rank_average(preds), [-np.inf] + np.cumsum(self.score_range_dict[i][1])[:-1].tolist() + [np.inf], labels = self.score_range_dict[i][0])\n\ndef optim_predict(pred, do_round=True, target=[ 2, 5, 7, 9, 11, 12, 13, 14, 15, 16, 19]):\n for i in range(pred.shape[1]):\n if i in target:\n pred[:,i] = optim.predict(pred[:,i], i)\n return pred\n```", "votes": 25 }, { "id": 750199, "postDate": "2020-02-19T07:34:48.157Z", "content": "@kenmatsu4 Thanks for sharing a great solution :D\nCan I ask you some questions?
\n\n```\n1. > Use first and last part of text if length exceed max_seq_length \nQ. How did you set the ratio for the first and last? (ex. 1 : 1, or 2 : 1 or etc …)
\n\n> Averaging 4 BERT base-uncased model \nQ. What is the difference of the four models?
> Fit target distribution with train data (a part of target columns)
\n
Q. How did you choose the part of the target columns? Did you use oof?
> Freeze half of BertLayer for reducing model complexity
\n
Q. Which BertLayers did you freeze?
\n```
Thanks!
", "rawMarkdown": "@kenmatsu4 Thanks for sharing a great solution :D\nCan I ask you some questions?\n\n```\n1. > Use first and last part of text if length exceed max_seq_length \nQ. How did you set the ratio for the first and last? (ex. 1 : 1, or 2 : 1 or etc …) \n\n \n2. > Averaging 4 BERT base-uncased model \nQ. What is the difference of the four models? \n\n \n3. > Fit target distribution with train data (a part of target columns) \n Q. How did you choose the part of the target columns? Did you use oof? \n\n \n4. > Freeze half of BertLayer for reducing model complexity \n Q. Which BertLayers did you freeze? \n``` \n\nThanks!", "votes": 1, "replies": [ { "id": 750532, "postDate": "2020-02-19T13:37:49.467Z", "content": "Thank you for your question, @guchio3 !\nThe followings are my answer to your questions :D
\n\n\n1. > Use first and last part of text if length exceed max_seq_length\nQ. How did you set the ratio for the first and last? (ex. 1 : 1, or 2 : 1 or etc …)
\n
-> I tried various type of the ratio, however 1 : 1 was the best for me.
\n\n\n2. > Averaging 4 BERT base-uncased model\nQ. What is the difference of the four models?\n\n-> Used the models described as the following figure.\n
\n3. > Fit target distribution with train data (a part of target columns)
Q. How did you choose the part of the target columns? Did you use oof? \n-> Yes, I tried all of the columns and then, pick-up columns which has improvement with the post processing.
\n
\n4. > Freeze half of BertLayer for reducing model complexity
Q. Which BertLayers did you freeze? \n-> Set trainable as False for first half on the list of TFBertLayer class.
\n
Thank you for the quick and detailed answer.\nNow it's clear for me!
", "rawMarkdown": "Thank you for the quick and detailed answer.\nNow it's clear for me!", "votes": 1 } ] }, { "id": 742303, "postDate": "2020-02-11T08:17:10.937Z", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @kenmatsu4
", "rawMarkdown": "Congratulations!\nThanks for Sharing your Approach & Insights!! @kenmatsu4 ", "votes": 1, "replies": [ { "id": 742345, "postDate": "2020-02-11T08:47:19.600Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] }, { "id": 742186, "postDate": "2020-02-11T06:13:20.450Z", "content": "You really did a great job, thanks for sharing.
", "rawMarkdown": "You really did a great job, thanks for sharing.", "votes": 1, "replies": [ { "id": 742341, "postDate": "2020-02-11T08:46:23.430Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] }, { "id": 742278, "postDate": "2020-02-11T07:48:25.330Z", "content": "Let us continue our learning and exciting journey ! @kenmatsu4
", "rawMarkdown": "Let us continue our learning and exciting journey ! @kenmatsu4 ", "votes": 2, "replies": [ { "id": 742343, "postDate": "2020-02-11T08:47:11.823Z", "content": "Thanks ratthachat! Let's get gold medal next time!
", "rawMarkdown": "Thanks ratthachat! Let's get gold medal next time!", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:07:43Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4034555, "solution_url": "/competitions/google-quest-challenge/writeups/kenmatsu4-47th-place-solution" }