Congrats to the winners. It is a hard and funny competition. Thanks my teammates. ( @Yang Zhang @chizhu @chensheng @Guan_Yuhang ).
\n\nWe got some magics in this competition.
\n\nOne is question part, another is answer part. The question part has 21 targets. The answer part has 9 targets. We train two model to predict the two part.\nIt let us get a higer score(0.39x) in the very beginning.
\n\na) Add the additional layers. For roberta base model, we add the 13th layer with random initialize. For roberta large model, we add the 25th layer with random initialize.\nb) concate CLS token embedding and SEP token embedding together.
\n\nc) concate the category features and length features
\n\nWe tried many pre-process methods for transformers. Only one is useful.\ndef cln(x): return \" \".join(x.split())
\n\nThe very simple function boost 0.005 for the roberta base lb score .
\n\nIt comes from https://www.kaggle.com/stackoverflow/stackoverflow. \nWe got the ideas from paper 'Self-training with Noisy Student improves ImageNet classification'. The teacher gives big size weak pesudo labels for the students.The students use the weak pesudo labels for train.Firstly we use one model to predict the psuedo labels for the unlabeled datas.\nThen we use the psuedo labels to train our models.
\n\nFor 6k pesuedo labels, we boost the lstm model lb score from 0.365 to 0.372. (private score from 0.332 to 0.343)\nFor 50k pesuedo labels, we boost the roberta base lb score from 0.405 to 0.415.\nFor 100k pesuedo labels, the roberta base lb score is still 0.415.
\n\nWe do not have too much time for training powerful student models.
\n\nIt is the big magic for this competition.\nWithout post-process, our roberta large mode got the 0.411 lb score.\nWhen we use the simple round post-process, it jumped 0.448 lb score.
\n\nThen we combine several optimization methods for post-process, these make roberta large jumped 0.46x lb score. Very big boosting!
\n\nThanks my teemates again. You are genius.
\n\nWe ensembled 6 models(roberta large, roberta base, bert base and xlnet. The best model is roberta base lb 0.415). We used post-process for them and got the final score.
\n\nCongrats on an amazing job Huiqin. I didn't realize that Roberta tokenizer doesn't remove white space itself. Thanks for sharing that.
\n\nFor others, I will emphasize this. If you type RobertaTokenizer.from_pretrained('roberta-base').tokenize(\"extra (with 5 spaces between \"extra\" and \"space\"), then you will get
\n space\")[\"extra\", \" \", \" \", \" \", \" \", \"space\"]. This is different than BertTokenizer.from_pretrained('bert-base-cased').tokenize(\"extra . space\") which removes white space and returns [\"extra\", \"space\"].
I discovered this accidentally when I started stripping code blocks out of my text. Some of my models had started overfitting on multiple whitespace inadvertently.
", "votes": 1, "replies": [] }, { "id": 743664, "author_name": "stout", "author_url": "", "post_date": "2020-02-12T07:10:05.843000", "content": "Great info for newbies. Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 742487, "author_name": "sakami", "author_url": "", "post_date": "2020-02-11T10:17:37.913000", "content": "I published my LSTM code here.
", "votes": 3, "replies": [ { "id": 742501, "author_name": "huiqin", "author_url": "", "post_date": "2020-02-11T10:25:42.340000", "content": "Thanks for sharing. It's a very strong model !
", "votes": 0, "replies": [] } ] }, { "id": 742730, "author_name": "jionie", "author_url": "", "post_date": "2020-02-11T14:11:23", "content": "Congrats!!! Very impressive solution💯 , especially the customed model and pseudo labeling. May I ask how could you come up with the idea of adding the additional layers? I have never thought about doing this 🙌 .
", "votes": 1, "replies": [ { "id": 743318, "author_name": "huiqin", "author_url": "", "post_date": "2020-02-12T01:29:41.427000", "content": "I got this idea in 'Jigsaw Unintended Bias in Toxicity Classification' competition. I want to copy one of the all 12 encoder layers and extend bert base model into 13 encoders. But it did not work. Tried many times, it suddenly worked when I added a new encoder layer with random initialize.
", "votes": 1, "replies": [] }, { "id": 744058, "author_name": "jionie", "author_url": "", "post_date": "2020-02-12T14:21:37.250000", "content": "Wow, that's interesting! Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 742042, "author_name": "Serigne ", "author_url": "", "post_date": "2020-02-11T03:41:38.357000", "content": "\n\n\nOne is question part, another is answer part. The question part has 21 targets. The answer part has 9 targets. We train two model to predict the two part.
\n
I did exactly the same thing in one of my bert model
\n\nCongrats @qinhui1999 and your teams !
\n\nIt would be great to team up with you one day !
", "votes": 1, "replies": [ { "id": 742068, "author_name": "huiqin", "author_url": "", "post_date": "2020-02-11T04:03:53.567000", "content": "@Serigne You are welcome. :)
", "votes": 1, "replies": [] } ] }, { "id": 743433, "author_name": "oohara", "author_url": "", "post_date": "2020-02-12T03:52:09.420000", "content": "Could you explain more about what the pre-processing code is doing?\ndef cln(x): return \" \".join(x.split())
That removes white space. If there are 5 spaces between two words, that function converts the 5 spaces to 1 space.
", "votes": 4, "replies": [] }, { "id": 743473, "author_name": "huiqin", "author_url": "", "post_date": "2020-02-12T05:09:55.760000", "content": "Thanks for your good explanations. In fact , we just want to remove many '\\n' tokens and let the sentences shorter. And this function works. : )
", "votes": 1, "replies": [] } ] }, { "id": 743688, "author_name": "stout", "author_url": "", "post_date": "2020-02-12T07:39:22.327000", "content": "Thanks for sharing, and congrats!
\n\nFor everyone following the 2 part training:\nI asume that for the answer part you used as features only the \"answer\". Did you try to use a Q-A input form to BERT, where features for the answer part are \"question_title\"+\"question_body\"+[SEP]+\"answer\"? In my experiments I observed a big improvement over the \"answer\" only feature. Looking at targets it seems logical that some answer targets are correlated with questions features...
I would appreciate very much to hear if anybody confirms the above observation :-)
", "votes": 0, "replies": [] }, { "id": 742738, "author_name": "CPMP", "author_url": "", "post_date": "2020-02-11T14:15:29.430000", "content": "Thanks for sharing, and congrats on the result. We should have tried pseudo labeling more ;)
", "votes": 0, "replies": [] }, { "id": 742255, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-02-11T07:28:47.127000", "content": "Congrats @qinhui1999 and team. Well done! Thanks for sharing your solution.
", "votes": 0, "replies": [] }, { "id": 742231, "author_name": "yukiya", "author_url": "", "post_date": "2020-02-11T07:03:46.480000", "content": "Congratulations and thanks for the share.
\n\n\n\n\nWe want to see @sakami lstm solutions
\n
I second this request :)
", "votes": 0, "replies": [ { "id": 742254, "author_name": "sakami", "author_url": "", "post_date": "2020-02-11T07:28:24.903000", "content": "I'll share it later. ;)
", "votes": 1, "replies": [] } ] }, { "id": 742185, "author_name": "MaChaogong", "author_url": "", "post_date": "2020-02-11T06:11:25.237000", "content": "really amazing work. Congratulations!
", "votes": 0, "replies": [] }, { "id": 742108, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-02-11T04:40:16.837000", "content": "How did you guys used external data? Can we see the snip or pseudo code?\nThanks And Congratulations for bagging a gold medal!!
", "votes": 0, "replies": [ { "id": 742116, "author_name": "huiqin", "author_url": "", "post_date": "2020-02-11T04:47:41.893000", "content": "As I mentioned, firstly you need a teacher model to predict the unlabeled datas and get the pseudo labels. Then you can add the pseudo labels into the train data for your student models. You can see the details from paper ''Self-training with Noisy Student improves ImageNet classification'' .
", "votes": 1, "replies": [] } ] }, { "id": 741999, "author_name": "XY", "author_url": "", "post_date": "2020-02-11T03:07:19.433000", "content": "Question and Answer labels have correlation. Din't you afraid split bring bad score ? I was very much afraid did'n do that 😂
", "votes": 0, "replies": [ { "id": 742014, "author_name": "huiqin", "author_url": "", "post_date": "2020-02-11T03:18:06.197000", "content": "Question asker is the same one , but the answer people are quite different. Split two parts, we can use more sentences for bert models.
", "votes": 1, "replies": [] } ] }, { "id": 741994, "author_name": "huiqin", "author_url": "", "post_date": "2020-02-11T03:01:34.740000", "content": "", "votes": 0, "replies": [] }, { "id": 741991, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2020-02-11T02:59:50.560000", "content": "So you didn’t use post-processing? That amazing Hui!\nEDIT Okay, I can see the full post now :D
", "votes": 0, "replies": [ { "id": 741995, "author_name": "huiqin", "author_url": "", "post_date": "2020-02-11T03:03:08.263000", "content": "No. We use the post-processing. My post can not display all informations. :(
", "votes": 0, "replies": [] }, { "id": 742005, "author_name": "huiqin", "author_url": "", "post_date": "2020-02-11T03:13:29.490000", "content": "Can you share your solution? You teem is powerful too.
", "votes": 0, "replies": [] } ] }, { "id": 741982, "author_name": "liuze", "author_url": "", "post_date": "2020-02-11T02:54:10.593000", "content": "Congratulations, may I ask a question? what SEP token embedding is ?
last encoder SEP like these:\nlayer, _,encoded_layer = self.mdl(x, attention_mask=attn_msk)\npooled_last_21 = layer[:, -1]
", "votes": 1, "replies": [] }, { "id": 742566, "author_name": "liuze", "author_url": "", "post_date": "2020-02-11T11:32:57.183000", "content": "thanks, got it
", "votes": 0, "replies": [] } ] }, { "id": 744558, "author_name": "corochann", "author_url": "", "post_date": "2020-02-12T23:42:22.617000", "content": "Thanks for sharing.
", "votes": 0, "replies": [] }, { "id": 743507, "author_name": "Sanku Vishnu Darshan", "author_url": "", "post_date": "2020-02-12T05:44:46.017000", "content": "Thanks for sharing, and congrats.
", "votes": 0, "replies": [] } ], "comment_count": 32, "messages": [ { "id": 741975, "postDate": "2020-02-11T02:49:47.660Z", "content": "Congrats to the winners. It is a hard and funny competition. Thanks my teammates. ( @Yang Zhang @chizhu @chensheng @Guan_Yuhang ).
\n\nWe got some magics in this competition.
\n\nOne is question part, another is answer part. The question part has 21 targets. The answer part has 9 targets. We train two model to predict the two part.\nIt let us get a higer score(0.39x) in the very beginning.
\n\na) Add the additional layers. For roberta base model, we add the 13th layer with random initialize. For roberta large model, we add the 25th layer with random initialize.\nb) concate CLS token embedding and SEP token embedding together.
\n\nc) concate the category features and length features
\n\nWe tried many pre-process methods for transformers. Only one is useful.\ndef cln(x): return \" \".join(x.split())
\n\nThe very simple function boost 0.005 for the roberta base lb score .
\n\nIt comes from https://www.kaggle.com/stackoverflow/stackoverflow. \nWe got the ideas from paper 'Self-training with Noisy Student improves ImageNet classification'. The teacher gives big size weak pesudo labels for the students.The students use the weak pesudo labels for train.Firstly we use one model to predict the psuedo labels for the unlabeled datas.\nThen we use the psuedo labels to train our models.
\n\nFor 6k pesuedo labels, we boost the lstm model lb score from 0.365 to 0.372. (private score from 0.332 to 0.343)\nFor 50k pesuedo labels, we boost the roberta base lb score from 0.405 to 0.415.\nFor 100k pesuedo labels, the roberta base lb score is still 0.415.
\n\nWe do not have too much time for training powerful student models.
\n\nIt is the big magic for this competition.\nWithout post-process, our roberta large mode got the 0.411 lb score.\nWhen we use the simple round post-process, it jumped 0.448 lb score.
\n\nThen we combine several optimization methods for post-process, these make roberta large jumped 0.46x lb score. Very big boosting!
\n\nThanks my teemates again. You are genius.
\n\nWe ensembled 6 models(roberta large, roberta base, bert base and xlnet. The best model is roberta base lb 0.415). We used post-process for them and got the final score.
\n\nCongrats on an amazing job Huiqin. I didn't realize that Roberta tokenizer doesn't remove white space itself. Thanks for sharing that.
\n\nFor others, I will emphasize this. If you type RobertaTokenizer.from_pretrained('roberta-base').tokenize(\"extra (with 5 spaces between \"extra\" and \"space\"), then you will get
\n space\")[\"extra\", \" \", \" \", \" \", \" \", \"space\"]. This is different than BertTokenizer.from_pretrained('bert-base-cased').tokenize(\"extra . space\") which removes white space and returns [\"extra\", \"space\"].
I discovered this accidentally when I started stripping code blocks out of my text. Some of my models had started overfitting on multiple whitespace inadvertently.
", "rawMarkdown": "I discovered this accidentally when I started stripping code blocks out of my text. Some of my models had started overfitting on multiple whitespace inadvertently. ", "votes": 1 }, { "id": 743664, "postDate": "2020-02-12T07:10:05.843Z", "content": "Great info for newbies. Thanks!
", "rawMarkdown": "Great info for newbies. Thanks!" } ] }, { "id": 742487, "postDate": "2020-02-11T10:17:37.913Z", "content": "I published my LSTM code here.
", "rawMarkdown": "I published my LSTM code [here](https://www.kaggle.com/sakami/google-quest-single-lstm?scriptVersionId=28487242).", "votes": 3, "replies": [ { "id": 742501, "postDate": "2020-02-11T10:25:42.340Z", "content": "Thanks for sharing. It's a very strong model !
", "rawMarkdown": "Thanks for sharing. It's a very strong model !" } ] }, { "id": 742730, "postDate": "2020-02-11T14:11:23Z", "content": "Congrats!!! Very impressive solution💯 , especially the customed model and pseudo labeling. May I ask how could you come up with the idea of adding the additional layers? I have never thought about doing this 🙌 .
", "rawMarkdown": "Congrats!!! Very impressive solution💯 , especially the customed model and pseudo labeling. May I ask how could you come up with the idea of adding the additional layers? I have never thought about doing this 🙌 .", "votes": 1, "replies": [ { "id": 743318, "postDate": "2020-02-12T01:29:41.427Z", "content": "I got this idea in 'Jigsaw Unintended Bias in Toxicity Classification' competition. I want to copy one of the all 12 encoder layers and extend bert base model into 13 encoders. But it did not work. Tried many times, it suddenly worked when I added a new encoder layer with random initialize.
", "rawMarkdown": "I got this idea in 'Jigsaw Unintended Bias in Toxicity Classification' competition. I want to copy one of the all 12 encoder layers and extend bert base model into 13 encoders. But it did not work. Tried many times, it suddenly worked when I added a new encoder layer with random initialize.", "votes": 1 }, { "id": 744058, "postDate": "2020-02-12T14:21:37.250Z", "content": "Wow, that's interesting! Thanks!
", "rawMarkdown": "Wow, that's interesting! Thanks!" } ] }, { "id": 742042, "postDate": "2020-02-11T03:41:38.357Z", "content": "\n\n\nOne is question part, another is answer part. The question part has 21 targets. The answer part has 9 targets. We train two model to predict the two part.
\n
I did exactly the same thing in one of my bert model
\n\nCongrats @qinhui1999 and your teams !
\n\nIt would be great to team up with you one day !
", "rawMarkdown": "> One is question part, another is answer part. The question part has 21 targets. The answer part has 9 targets. We train two model to predict the two part.\n\nI did exactly the same thing in one of my bert model \n\n\nCongrats @qinhui1999 and your teams !\n\nIt would be great to team up with you one day !", "votes": 1, "replies": [ { "id": 742068, "postDate": "2020-02-11T04:03:53.567Z", "content": "@Serigne You are welcome. :)
", "rawMarkdown": "@Serigne You are welcome. :)", "votes": 1 } ] }, { "id": 743433, "postDate": "2020-02-12T03:52:09.420Z", "content": "Could you explain more about what the pre-processing code is doing?\ndef cln(x): return \" \".join(x.split())
That removes white space. If there are 5 spaces between two words, that function converts the 5 spaces to 1 space.
", "rawMarkdown": "That removes white space. If there are 5 spaces between two words, that function converts the 5 spaces to 1 space.", "votes": 4 }, { "id": 743473, "postDate": "2020-02-12T05:09:55.760Z", "content": "Thanks for your good explanations. In fact , we just want to remove many '\\n' tokens and let the sentences shorter. And this function works. : )
", "rawMarkdown": "Thanks for your good explanations. In fact , we just want to remove many '\\n' tokens and let the sentences shorter. And this function works. : )", "votes": 1 } ] }, { "id": 743688, "postDate": "2020-02-12T07:39:22.327Z", "content": "Thanks for sharing, and congrats!
\n\nFor everyone following the 2 part training:\nI asume that for the answer part you used as features only the \"answer\". Did you try to use a Q-A input form to BERT, where features for the answer part are \"question_title\"+\"question_body\"+[SEP]+\"answer\"? In my experiments I observed a big improvement over the \"answer\" only feature. Looking at targets it seems logical that some answer targets are correlated with questions features...
I would appreciate very much to hear if anybody confirms the above observation :-)
", "rawMarkdown": "Thanks for sharing, and congrats! \n\nFor everyone following the 2 part training:\nI asume that for the answer part you used as features only the \"answer\". Did you try to use a Q-A input form to BERT, where features for the answer part are `\"question_title\"+\"question_body\"+[SEP]+\"answer\"`? In my experiments I observed a big improvement over the `\"answer\"` only feature. Looking at targets it seems logical that some answer targets are correlated with questions features...\n\nI would appreciate very much to hear if anybody confirms the above observation :-)" }, { "id": 742738, "postDate": "2020-02-11T14:15:29.430Z", "content": "Thanks for sharing, and congrats on the result. We should have tried pseudo labeling more ;)
", "rawMarkdown": "Thanks for sharing, and congrats on the result. We should have tried pseudo labeling more ;)" }, { "id": 742255, "postDate": "2020-02-11T07:28:47.127Z", "content": "Congrats @qinhui1999 and team. Well done! Thanks for sharing your solution.
", "rawMarkdown": "Congrats @qinhui1999 and team. Well done! Thanks for sharing your solution." }, { "id": 742231, "postDate": "2020-02-11T07:03:46.480Z", "content": "Congratulations and thanks for the share.
\n\n\n\n\nWe want to see @sakami lstm solutions
\n
I second this request :)
", "rawMarkdown": "Congratulations and thanks for the share.\n\n> We want to see @sakami lstm solutions\n\nI second this request :) ", "replies": [ { "id": 742254, "postDate": "2020-02-11T07:28:24.903Z", "content": "I'll share it later. ;)
", "rawMarkdown": "I'll share it later. ;)", "votes": 1 } ] }, { "id": 742185, "postDate": "2020-02-11T06:11:25.237Z", "content": "really amazing work. Congratulations!
", "rawMarkdown": "really amazing work. Congratulations! " }, { "id": 742108, "postDate": "2020-02-11T04:40:16.837Z", "content": "How did you guys used external data? Can we see the snip or pseudo code?\nThanks And Congratulations for bagging a gold medal!!
", "rawMarkdown": "How did you guys used external data? Can we see the snip or pseudo code?\nThanks And Congratulations for bagging a gold medal!!", "replies": [ { "id": 742116, "postDate": "2020-02-11T04:47:41.893Z", "content": "As I mentioned, firstly you need a teacher model to predict the unlabeled datas and get the pseudo labels. Then you can add the pseudo labels into the train data for your student models. You can see the details from paper ''Self-training with Noisy Student improves ImageNet classification'' .
", "rawMarkdown": "As I mentioned, firstly you need a teacher model to predict the unlabeled datas and get the pseudo labels. Then you can add the pseudo labels into the train data for your student models. You can see the details from paper ''[Self-training with Noisy Student improves ImageNet classification'](https://arxiv.org/abs/1911.04252)' . ", "votes": 1 } ] }, { "id": 741999, "postDate": "2020-02-11T03:07:19.433Z", "content": "Question and Answer labels have correlation. Din't you afraid split bring bad score ? I was very much afraid did'n do that 😂
", "rawMarkdown": "Question and Answer labels have correlation. Din't you afraid split bring bad score ? I was very much afraid did'n do that 😂 ", "replies": [ { "id": 742014, "postDate": "2020-02-11T03:18:06.197Z", "content": "Question asker is the same one , but the answer people are quite different. Split two parts, we can use more sentences for bert models.
", "rawMarkdown": "Question asker is the same one , but the answer people are quite different. Split two parts, we can use more sentences for bert models. ", "votes": 1 } ] }, { "id": 741994, "postDate": "2020-02-11T03:01:34.740Z", "rawMarkdown": "" }, { "id": 741991, "postDate": "2020-02-11T02:59:50.560Z", "content": "So you didn’t use post-processing? That amazing Hui!\nEDIT Okay, I can see the full post now :D
", "rawMarkdown": "So you didn’t use post-processing? That amazing Hui!\n*EDIT* Okay, I can see the full post now :D", "replies": [ { "id": 741995, "postDate": "2020-02-11T03:03:08.263Z", "content": "No. We use the post-processing. My post can not display all informations. :(
", "rawMarkdown": "No. We use the post-processing. My post can not display all informations. :(" }, { "id": 742005, "postDate": "2020-02-11T03:13:29.490Z", "content": "Can you share your solution? You teem is powerful too.
", "rawMarkdown": "Can you share your solution? You teem is powerful too. " } ] }, { "id": 741982, "postDate": "2020-02-11T02:54:10.593Z", "content": "Congratulations, may I ask a question? what SEP token embedding is ?
last encoder SEP like these:\nlayer, _,encoded_layer = self.mdl(x, attention_mask=attn_msk)\npooled_last_21 = layer[:, -1]
", "rawMarkdown": "last encoder SEP like these:\nlayer, _,encoded_layer = self.mdl(x, attention_mask=attn_msk)\npooled_last_21 = layer[:, -1]\n", "votes": 1 }, { "id": 742566, "postDate": "2020-02-11T11:32:57.183Z", "content": "thanks, got it
", "rawMarkdown": "thanks, got it" } ] }, { "id": 744558, "postDate": "2020-02-12T23:42:22.617Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." }, { "id": 743507, "postDate": "2020-02-12T05:44:46.017Z", "content": "Thanks for sharing, and congrats.
", "rawMarkdown": "Thanks for sharing, and congrats." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:07:32Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4014952, "solution_url": "/competitions/google-quest-challenge/writeups/on-the-way-our-magics" }