First, I want to thanks my teammates @brightertiger @khyeh0719 @arvissu @qbenrosb00, we have great works. And thanks google and kaggle to host the good competition. I will briefly summarize our methods
\n\nWe tried some preprocessing before trained BERT model and found the CV and LB are similar to no preprocessing. For the clean code, we didn't do any preprocessing to train SoTA models. Moreover, We used head and tail part of the texts as model input.
\n\nAs other teams, We train two BERT models for question labels(21) and answer labels(9).\n* Question : [cls]+ title+[sep]+question+[sep]\n* Answer : [cls]+title+[sep]+question+[sep]+answer[sep]
We tried BERT with different architectures.\n* Vanilla BERT: 0.35x LB\n* Customized BERT+head and tail part of the texts: 0.392 LB\n* Separate two BERT for question and answer: 0.396 LB\n* Separate two BERT for the question and answer with special tokens: CV 415 LB 0.405
\n\nWe find out using two BERT and special token get the best result on LB.\nTherefore we apply the method to train GPT2 and RoBERTa.
\n\nWe also used BERT outputs as embedding to train the LSTM and got great results\nCV: 0.418, LB: 0.396
\n\nThe summary of models we tried\n* BERT-base CV 415 LB 0.405\n* RoBERTa CV 0.411 LB 0.398\n* GPT2 CV: 0.418 LB: 0.396\n* BERT-large\n* BERT-large-wwm\n* BERT-large-wwm-squad\n* BERT-RNN CV: 0.418 LB: 0.396\n* Pretrained Embedding + NN\n* RNN+USE
\n\nThe models we used for ensemble are BERT, RoBERTa, GPT2, BERT-RNN. We only used simple average for ensemble.
\n\nWe only used the threshold method for postprocessing and got ~0.004 improvement in LB. I don't like the postpocessing method. However, in this competition, postprocessing is all you need.
\n\n```\ndef postProcessing(x):
\n\nx = np.where(x>=0.9241, 1.0, x)\nx = np.where(x<=0.0759, 0.0, x)\n\nreturn x\ntargets = ['question_conversational',\n 'question_type_compare', \n 'question_type_consequence', \n 'question_type_definition', \n 'question_type_entity', \n 'question_type_choice']
\n\nsub.loc[:, targets] = postProcessing(sub.loc[:, targets].values)\n```
\n\n恭喜~~
", "votes": 1, "replies": [ { "id": 746632, "author_name": "Dean", "author_url": "", "post_date": "2020-02-15T09:42:33.993000", "content": "謝謝^^
", "votes": 0, "replies": [] } ] }, { "id": 745142, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-02-13T14:47:57.777000", "content": "Congratulations!!\nThanks for SHaring your APproach & Insights!! @m10515009
", "votes": 1, "replies": [ { "id": 745159, "author_name": "Dean", "author_url": "", "post_date": "2020-02-13T15:07:01.557000", "content": "Thank you
", "votes": 0, "replies": [] } ] }, { "id": 745225, "author_name": "arvis", "author_url": "", "post_date": "2020-02-13T16:23:53.577000", "content": "we obeserved extreme labels such as question conversational, question type definition, etc.
\n\nand we are only adjust prediction sigmoid to hard_sigmoid in some columns, like @m10515009 postprocessing function.
\n\nfinally, we got a good score in the end of the competition. I really appreciate all the teammate. I have been learning a lot in this competition 😊
", "votes": 2, "replies": [] } ], "comment_count": 5, "messages": [ { "id": 745081, "postDate": "2020-02-13T13:06:22.867Z", "content": "First, I want to thanks my teammates @brightertiger @khyeh0719 @arvissu @qbenrosb00, we have great works. And thanks google and kaggle to host the good competition. I will briefly summarize our methods
\n\nWe tried some preprocessing before trained BERT model and found the CV and LB are similar to no preprocessing. For the clean code, we didn't do any preprocessing to train SoTA models. Moreover, We used head and tail part of the texts as model input.
\n\nAs other teams, We train two BERT models for question labels(21) and answer labels(9).\n* Question : [cls]+ title+[sep]+question+[sep]\n* Answer : [cls]+title+[sep]+question+[sep]+answer[sep]
We tried BERT with different architectures.\n* Vanilla BERT: 0.35x LB\n* Customized BERT+head and tail part of the texts: 0.392 LB\n* Separate two BERT for question and answer: 0.396 LB\n* Separate two BERT for the question and answer with special tokens: CV 415 LB 0.405
\n\nWe find out using two BERT and special token get the best result on LB.\nTherefore we apply the method to train GPT2 and RoBERTa.
\n\nWe also used BERT outputs as embedding to train the LSTM and got great results\nCV: 0.418, LB: 0.396
\n\nThe summary of models we tried\n* BERT-base CV 415 LB 0.405\n* RoBERTa CV 0.411 LB 0.398\n* GPT2 CV: 0.418 LB: 0.396\n* BERT-large\n* BERT-large-wwm\n* BERT-large-wwm-squad\n* BERT-RNN CV: 0.418 LB: 0.396\n* Pretrained Embedding + NN\n* RNN+USE
\n\nThe models we used for ensemble are BERT, RoBERTa, GPT2, BERT-RNN. We only used simple average for ensemble.
\n\nWe only used the threshold method for postprocessing and got ~0.004 improvement in LB. I don't like the postpocessing method. However, in this competition, postprocessing is all you need.
\n\n```\ndef postProcessing(x):
\n\nx = np.where(x>=0.9241, 1.0, x)\nx = np.where(x<=0.0759, 0.0, x)\n\nreturn x\ntargets = ['question_conversational',\n 'question_type_compare', \n 'question_type_consequence', \n 'question_type_definition', \n 'question_type_entity', \n 'question_type_choice']
\n\nsub.loc[:, targets] = postProcessing(sub.loc[:, targets].values)\n```
\n\n恭喜~~
", "rawMarkdown": "恭喜~~", "votes": 1, "replies": [ { "id": 746632, "postDate": "2020-02-15T09:42:33.993Z", "content": "謝謝^^
", "rawMarkdown": "謝謝^^" } ] }, { "id": 745142, "postDate": "2020-02-13T14:47:57.777Z", "content": "Congratulations!!\nThanks for SHaring your APproach & Insights!! @m10515009
", "rawMarkdown": "Congratulations!!\nThanks for SHaring your APproach & Insights!! @m10515009 ", "votes": 1, "replies": [ { "id": 745159, "postDate": "2020-02-13T15:07:01.557Z", "content": "Thank you
", "rawMarkdown": "Thank you" } ] }, { "id": 745225, "postDate": "2020-02-13T16:23:53.577Z", "content": "we obeserved extreme labels such as question conversational, question type definition, etc.
\n\nand we are only adjust prediction sigmoid to hard_sigmoid in some columns, like @m10515009 postprocessing function.
\n\nfinally, we got a good score in the end of the competition. I really appreciate all the teammate. I have been learning a lot in this competition 😊
", "rawMarkdown": "we obeserved extreme labels such as question conversational, question type definition, etc. \n\nand we are only adjust prediction sigmoid to hard_sigmoid in some columns, like @m10515009 postprocessing function.\n\nfinally, we got a good score in the end of the competition. I really appreciate all the teammate. I have been learning a lot in this competition 😊 \n\t\n\n\n\n\n", "votes": 2 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:09:27Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4016328, "solution_url": "/competitions/google-quest-challenge/writeups/kun-hao-yeh-21st-place-solution" }