Congrats to the winners and Thanks to my teammates. ( @theoviel @cl2ev1 @mathurinache @titericz ). Here's a brief overview of things that matter in our solution
\n\n\nloss_fct = nn.BCEWithLogitsLoss(pos_weight=torch.Tensor([\n0.9, 1, 1.5, 0.8, 0.8, 0.8, 0.96, 1.1, 1.1, 3, 1, 1.1, 2, 3, 3, 2, 1, 2, 1, 2, 0.9, 0.75, 0.9, 0.75, 0.75, 0.7, 1, 2.5, 1, 0.75]))\n
betas=(0.5, 0.999) and a no bias decay of 1Our solution is a Ensemble of 4 models, (3 BERT-Large + 1 BERT-Base)
We made most experiments using the bert-base-uncased architecture, and managed to build a strong pipeline about 1 week before the end of the competition. This enabled us to switch easily to bigger ones which in the end made the strength of our ensemble.
We build 4 different architectures on top of the transformer, and used two of them overall.
Here is the idea about the custom arch one we picked for our bert-base approach :
\n\n[CLS] title [Q] question [A] answer [SEP] and [0, … 0, 1, …., 1, 2, …, 2] for ids. We initialed special tokens with values of the [SEP] token. Custom ids were also initialized with the model values.[CLS] token of the last n=8 layers, apply a dense of size m=1024 and a tanh activation, then concatenate everything. categoryand host column (+ tanh). We concatenate them with the output of the custom pooler and obtain the logits.\nSome text cleaning (latex, urls, spaces, backslashes) was also appliedThis model is the only one that uses text cleaning and embeddings, it helps for diversity I guess.
for tokens and[0, … 0, 1, …., 1]for ids</li>\n<li>[A] title [SEP] answer[SEP]<code>for tokens and</code>[0, … 0, 1, …., 1] for idsnand mare below. Depending on the column, we either predict the value with only the pooled [Q] token, only the *
\n\n\nself.mix = [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 2] # 0 = both, 1 = q, 2 = a\nConcerning the pooler, it differs a bit :
\n\nn = 24, m = 128n = 24, m = 128n = 8, m = 768Detailed here :
\n\nhttps://www.kaggle.com/c/google-quest-challenge/discussion/129901
\n\nOur best model is the BERT-Large Whole Word Masking one, trained using Weighted BCE Loss gave :
With Post-Processing:
\n\nWithout post-processing :
\n\nWhich is a +0.06 boost on public and +0.04 on private. As you can see our single models are not that strong.
Thanks for reading, happy kaggling !
", "comments": [ { "id": 743392, "author_name": "Saikrishna Mukkamala", "author_url": "", "post_date": "2020-02-12T02:47:34.833000", "content": "Congratulations !! Why did you not try roberta?
", "votes": 1, "replies": [ { "id": 743725, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-02-12T08:33:10.447000", "content": "We tried it but had worse results overall (- 0.01 I'd say)
", "votes": 0, "replies": [] } ] }, { "id": 742423, "author_name": "ilovescience", "author_url": "", "post_date": "2020-02-11T09:36:42.260000", "content": "Congrats on your gold medal! Thanks for sharing your solution.
\n\n\n\n\nand [0, … 0, 1, …., 1, 2, …, 2] for ids.
\n
Are you talking about the token_type_ids here? If so, how did you increase the vocab from two to three? Huggingface did not have a function like that like it did for regular tokens (add_tokens and resize_embeddings). It didn't seem like there was a simple way of doing this for already pretrained models that you want to fine-tune. Please share how you were able to do this.
Quick and dirty\n\nw = self.transformer.embeddings.token_type_embeddings.weight\nself.transformer.embeddings.token_type_embeddings = nn.Embedding.from_pretrained(\n torch.cat([\n w[0].view(1, -1).detach().clone(),\n w[0].detach().view(1, -1).clone(),\n w[1].detach().view(1, -1).clone(),],\n 0,\n )\n)\n
Thank you so much for sharing this. This fixes the embeddings, but will the tokenizer properly return the correct token_type_ids (through encode_plus)? Or do you manually set the token_type_ids?
We don't use encode_plus, we manually set the token_type_ids
Ok sounds good. Thanks for sharing and congrats again on your gold!
", "votes": 1, "replies": [] } ] }, { "id": 742414, "author_name": "Zhuang Jia", "author_url": "", "post_date": "2020-02-11T09:34:12.570000", "content": "Congratulations! Very nice solution! I have a few questions.\n(1) How did you add Custom ids(the original is just 0 and 1)?\n(2) How did you init your custom tokens and ids?(some code may help me know how to implement it, thanks)\n(3) How did you add host info into bert model?
Will you sharing your code to us? Thanks very much!
", "votes": 1, "replies": [ { "id": 742441, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-02-11T09:49:44.893000", "content": "(1) \n\nw = self.transformer.embeddings.token_type_embeddings.weight\nself.transformer.embeddings.token_type_embeddings = nn.Embedding.from_pretrained(\n torch.cat([\n w[0].view(1, -1).detach().clone(),\n w[0].detach().view(1, -1).clone(),\n w[1].detach().view(1, -1).clone(),],\n 0,\n )\n)\n\n(2)\n```\nself.tokenizer.add_tokens([\"[q]\", \"[a]\"])
w = self.transformer.embeddings.word_embeddings.weight\nsep_w = w[102].view(1, -1).detach()\nself.transformer.embeddings.word_embeddings = nn.Embedding.from_pretrained(\n torch.cat([w, sep_w.clone(), sep_w.clone()])\n)\n```
\n\n(3) Using a simple nn.Embedding() layer
I will probably share my code on GitHub soon or later :)
", "votes": 1, "replies": [] } ] }, { "id": 742381, "author_name": "RB", "author_url": "", "post_date": "2020-02-11T09:14:24.473000", "content": "Congratulations and thank you for this post. How did you come up the pos_weights for loss function ?
Credits for Weighted Loss goes to @cl2ev1. He figured out based on some analysis of data and experiments with weights. I think he'll add more to answer your question.
", "votes": 3, "replies": [] }, { "id": 742386, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-02-11T09:16:52.520000", "content": "@cl2ev1 Spent some (a lot of actually) time optimizing them for optimal post-processed CV.
", "votes": 3, "replies": [] }, { "id": 742394, "author_name": "Ram Ramrakhya", "author_url": "", "post_date": "2020-02-11T09:23:13.223000", "content": "Yes, I agree he gave in a lot of effort to 2 of the most important parts of our solutions.
", "votes": 0, "replies": [] }, { "id": 742412, "author_name": "", "author_url": "", "post_date": "2020-02-11T09:32:07.677000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 744205, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-02-12T16:46:54.547000", "content": "Congrats @axel81 and team.
", "votes": 0, "replies": [] }, { "id": 742407, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-02-11T09:30:13.093000", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @axel81
", "votes": 0, "replies": [] }, { "id": 744563, "author_name": "corochann", "author_url": "", "post_date": "2020-02-12T23:46:44.207000", "content": "Congrats and thank you for sharing.
", "votes": 0, "replies": [] } ], "comment_count": 17, "messages": [ { "id": 742368, "postDate": "2020-02-11T09:07:21.290Z", "content": "Congrats to the winners and Thanks to my teammates. ( @theoviel @cl2ev1 @mathurinache @titericz ). Here's a brief overview of things that matter in our solution
\n\n\nloss_fct = nn.BCEWithLogitsLoss(pos_weight=torch.Tensor([\n0.9, 1, 1.5, 0.8, 0.8, 0.8, 0.96, 1.1, 1.1, 3, 1, 1.1, 2, 3, 3, 2, 1, 2, 1, 2, 0.9, 0.75, 0.9, 0.75, 0.75, 0.7, 1, 2.5, 1, 0.75]))\n
betas=(0.5, 0.999) and a no bias decay of 1Our solution is a Ensemble of 4 models, (3 BERT-Large + 1 BERT-Base)
We made most experiments using the bert-base-uncased architecture, and managed to build a strong pipeline about 1 week before the end of the competition. This enabled us to switch easily to bigger ones which in the end made the strength of our ensemble.
We build 4 different architectures on top of the transformer, and used two of them overall.
Here is the idea about the custom arch one we picked for our bert-base approach :
\n\n[CLS] title [Q] question [A] answer [SEP] and [0, … 0, 1, …., 1, 2, …, 2] for ids. We initialed special tokens with values of the [SEP] token. Custom ids were also initialized with the model values.[CLS] token of the last n=8 layers, apply a dense of size m=1024 and a tanh activation, then concatenate everything. categoryand host column (+ tanh). We concatenate them with the output of the custom pooler and obtain the logits.\nSome text cleaning (latex, urls, spaces, backslashes) was also appliedThis model is the only one that uses text cleaning and embeddings, it helps for diversity I guess.
for tokens and[0, … 0, 1, …., 1]for ids</li>\n<li>[A] title [SEP] answer[SEP]<code>for tokens and</code>[0, … 0, 1, …., 1] for idsnand mare below. Depending on the column, we either predict the value with only the pooled [Q] token, only the *
\n\n\nself.mix = [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 2] # 0 = both, 1 = q, 2 = a\nConcerning the pooler, it differs a bit :
\n\nn = 24, m = 128n = 24, m = 128n = 8, m = 768Detailed here :
\n\nhttps://www.kaggle.com/c/google-quest-challenge/discussion/129901
\n\nOur best model is the BERT-Large Whole Word Masking one, trained using Weighted BCE Loss gave :
With Post-Processing:
\n\nWithout post-processing :
\n\nWhich is a +0.06 boost on public and +0.04 on private. As you can see our single models are not that strong.
Thanks for reading, happy kaggling !
", "rawMarkdown": "Congrats to the winners and Thanks to my teammates. ( @theoviel @cl2ev1 @mathurinache @titericz ). Here's a brief overview of things that matter in our solution\n\n\n### Loss function\n\n* We sticked for a long time with the BCE and couldn’t improve results by changing it. We however managed to find weights that nicely fitted our post-processing policy.\n\n```\nloss_fct = nn.BCEWithLogitsLoss(pos_weight=torch.Tensor([\n0.9, 1, 1.5, 0.8, 0.8, 0.8, 0.96, 1.1, 1.1, 3, 1, 1.1, 2, 3, 3, 2, 1, 2, 1, 2, 0.9, 0.75, 0.9, 0.75, 0.75, 0.7, 1, 2.5, 1, 0.75]))\n```\n\n* Multiple other experiments with custom loss functions were conducted, but BCE performed the best.\n\n### Training stuff\n\n* 1 epoch with frozen encoder + 2 epochs with with everything unfrozen\n* Linear lr scheduling, with a custom learning rate depending on the layer. The transformer has a lower one as it is pretrained, and the closer to the output, the larger the learning rate\n* AdamW with `betas=(0.5, 0.999)` and a no bias decay of 1\n* Batch size is 64 for the first epoch, and then the larger we can fit on our gpus with sometimes an accumulation step\n\n## Models\n\n* Our solution is a Ensemble of 4 models, (3 BERT-Large + 1 BERT-Base)\n\n* We made most experiments using the bert-base-uncased architecture, and managed to build a strong pipeline about 1 week before the end of the competition. This enabled us to switch easily to bigger ones which in the end made the strength of our ensemble. \n\n* We build 4 different architectures on top of the transformer, and used two of them overall.\n\n### Bert Base Uncased\n\n* Here is the idea about the custom arch one we picked for our bert-base approach :\n\n * Input is `[CLS] title [Q] question [A] answer [SEP]` and `[0, … 0, 1, …., 1, 2, …, 2]` for ids. We initialed special tokens with values of the `[SEP]` token. Custom ids were also initialized with the model values.\n * Custom head : take the `[CLS]` token of the last `n=8` layers, apply a dense of size `m=1024` and a `tanh` activation, then concatenate everything. \n * Embeddings for the `category`and host `column` (+ `tanh`). We concatenate them with the output of the custom pooler and obtain the logits.\nSome text cleaning (latex, urls, spaces, backslashes) was also applied\n\n* This model is the only one that uses text cleaning and embeddings, it helps for diversity I guess. \n\n### Bert Larges\n\n* They repose on the same architecture. We use two inputs : \n - [Q] title [SEP] question [SEP]` for tokens and `[0, … 0, 1, …., 1]` for ids\n - [A] title [SEP] answer[SEP]` for tokens and `[0, … 0, 1, …., 1]` for ids\n - [Q] and [A] start with the value of the [CLS] token\n* Again, custom pooling head. Values of `n`and `m`are below. \n* Depending on the column, we either predict the value with only the pooled [Q] token, only the * \n - [A] token or both. The policy chosen is the following : \n ```\nself.mix = [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 2] # 0 = both, 1 = q, 2 = a\n```\n\n* Concerning the pooler, it differs a bit :\n - Bert large cased : `n = 24`, `m = 128`\n - Bert large uncased : `n = 24`, `m = 128`\n - Bert large uncased wwm : `n = 8`, `m = 768`\n\n## Post-processing\n\nDetailed here : \n\nhttps://www.kaggle.com/c/google-quest-challenge/discussion/129901\n\n## Results\n\n### Single model\n\n* Our best model is the BERT-Large Whole Word Masking one, trained using Weighted BCE Loss gave : \n\n* With Post-Processing:\n - Private LB : **0.41919**\n - Public LB : **0.46428**\n - CV: **0.454**\n\n* Without post-processing :\n - Private LB : **0.38127**\n - Public LB : **0.40737**\n - CV: **0.415**\n\n* Which is a +0.06 boost on public and +0.04 on private. As you can see our single models are not that strong.\n\n\n### Ensemble\n\n* Our best selected solution is a simple average of the 4 mentioned model. \n - Private LB : **0.42430**\n - Public LB : **0.47259**\n\n\nThanks for reading, happy kaggling !", "votes": 31 }, { "id": 743392, "postDate": "2020-02-12T02:47:34.833Z", "content": "Congratulations !! Why did you not try roberta?
", "rawMarkdown": "Congratulations !! Why did you not try roberta?", "votes": 1, "replies": [ { "id": 743725, "postDate": "2020-02-12T08:33:10.447Z", "content": "We tried it but had worse results overall (- 0.01 I'd say)
", "rawMarkdown": "We tried it but had worse results overall (- 0.01 I'd say)" } ] }, { "id": 742423, "postDate": "2020-02-11T09:36:42.260Z", "content": "Congrats on your gold medal! Thanks for sharing your solution.
\n\n\n\n\nand [0, … 0, 1, …., 1, 2, …, 2] for ids.
\n
Are you talking about the token_type_ids here? If so, how did you increase the vocab from two to three? Huggingface did not have a function like that like it did for regular tokens (add_tokens and resize_embeddings). It didn't seem like there was a simple way of doing this for already pretrained models that you want to fine-tune. Please share how you were able to do this.
Quick and dirty\n\nw = self.transformer.embeddings.token_type_embeddings.weight\nself.transformer.embeddings.token_type_embeddings = nn.Embedding.from_pretrained(\n torch.cat([\n w[0].view(1, -1).detach().clone(),\n w[0].detach().view(1, -1).clone(),\n w[1].detach().view(1, -1).clone(),],\n 0,\n )\n)\n
Thank you so much for sharing this. This fixes the embeddings, but will the tokenizer properly return the correct token_type_ids (through encode_plus)? Or do you manually set the token_type_ids?
We don't use encode_plus, we manually set the token_type_ids
Ok sounds good. Thanks for sharing and congrats again on your gold!
", "rawMarkdown": "Ok sounds good. Thanks for sharing and congrats again on your gold!", "votes": 1 } ] }, { "id": 742414, "postDate": "2020-02-11T09:34:12.570Z", "content": "Congratulations! Very nice solution! I have a few questions.\n(1) How did you add Custom ids(the original is just 0 and 1)?\n(2) How did you init your custom tokens and ids?(some code may help me know how to implement it, thanks)\n(3) How did you add host info into bert model?
Will you sharing your code to us? Thanks very much!
", "rawMarkdown": "Congratulations! Very nice solution! I have a few questions.\n(1) How did you add `Custom ids`(the original is just 0 and 1)?\n(2) How did you init your custom tokens and ids?(some code may help me know how to implement it, thanks)\n(3) How did you add `host` info into bert model?\n\nWill you sharing your code to us? Thanks very much!", "votes": 1, "replies": [ { "id": 742441, "postDate": "2020-02-11T09:49:44.893Z", "content": "(1) \n\nw = self.transformer.embeddings.token_type_embeddings.weight\nself.transformer.embeddings.token_type_embeddings = nn.Embedding.from_pretrained(\n torch.cat([\n w[0].view(1, -1).detach().clone(),\n w[0].detach().view(1, -1).clone(),\n w[1].detach().view(1, -1).clone(),],\n 0,\n )\n)\n\n(2)\n```\nself.tokenizer.add_tokens([\"[q]\", \"[a]\"])
w = self.transformer.embeddings.word_embeddings.weight\nsep_w = w[102].view(1, -1).detach()\nself.transformer.embeddings.word_embeddings = nn.Embedding.from_pretrained(\n torch.cat([w, sep_w.clone(), sep_w.clone()])\n)\n```
\n\n(3) Using a simple nn.Embedding() layer
I will probably share my code on GitHub soon or later :)
", "rawMarkdown": "(1) \n```\nw = self.transformer.embeddings.token_type_embeddings.weight\nself.transformer.embeddings.token_type_embeddings = nn.Embedding.from_pretrained(\n torch.cat([\n w[0].view(1, -1).detach().clone(),\n w[0].detach().view(1, -1).clone(),\n w[1].detach().view(1, -1).clone(),],\n 0,\n )\n)\n```\n(2)\n```\nself.tokenizer.add_tokens([\"[q]\", \"[a]\"])\n\nw = self.transformer.embeddings.word_embeddings.weight\nsep_w = w[102].view(1, -1).detach()\nself.transformer.embeddings.word_embeddings = nn.Embedding.from_pretrained(\n torch.cat([w, sep_w.clone(), sep_w.clone()])\n)\n```\n\n(3) Using a simple `nn.Embedding()` layer\n\nI will probably share my code on GitHub soon or later :) ", "votes": 1 } ] }, { "id": 742381, "postDate": "2020-02-11T09:14:24.473Z", "content": "Congratulations and thank you for this post. How did you come up the pos_weights for loss function ?
Credits for Weighted Loss goes to @cl2ev1. He figured out based on some analysis of data and experiments with weights. I think he'll add more to answer your question.
", "rawMarkdown": "Credits for Weighted Loss goes to @cl2ev1. He figured out based on some analysis of data and experiments with weights. I think he'll add more to answer your question.", "votes": 3 }, { "id": 742386, "postDate": "2020-02-11T09:16:52.520Z", "content": "@cl2ev1 Spent some (a lot of actually) time optimizing them for optimal post-processed CV.
", "rawMarkdown": "@cl2ev1 Spent some (a lot of actually) time optimizing them for optimal post-processed CV. ", "votes": 3 }, { "id": 742394, "postDate": "2020-02-11T09:23:13.223Z", "content": "Yes, I agree he gave in a lot of effort to 2 of the most important parts of our solutions.
", "rawMarkdown": "Yes, I agree he gave in a lot of effort to 2 of the most important parts of our solutions." }, { "id": 742412, "postDate": "2020-02-11T09:32:07.677Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ] }, { "id": 744205, "postDate": "2020-02-12T16:46:54.547Z", "content": "Congrats @axel81 and team.
", "rawMarkdown": "Congrats @axel81 and team." }, { "id": 742407, "postDate": "2020-02-11T09:30:13.093Z", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @axel81
", "rawMarkdown": "Congratulations!\nThanks for Sharing your Approach & Insights!! @axel81 " }, { "id": 744563, "postDate": "2020-02-12T23:46:44.207Z", "content": "Congrats and thank you for sharing.
", "rawMarkdown": "Congrats and thank you for sharing." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:08:02Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4014164, "solution_url": "/competitions/google-quest-challenge/writeups/what-else-private-lb-10th-place-solution" }