This is my first NLP competition and I am grateful to have gained my first gold. It was a great pleasure working with my talented and extremely hard working teammates @chanhu @xiaojiu1414 @serigne @seif95.
\n\nOur ensemble is nothing different from what other teams have shared, which includes bert base, bert large, xlnet, and roberta. I wanted to share a little bit my thought process on the submission trick.
\n\na few modeling tricks we have used are \n1) separate regression heads for questions and answers, \n2) head and tail tokenization \n3) concatenated [CLS] tokens as regression input\n4) USE difference and entity embedding as external features in Bert
\n\nI personally thought the magic was going to be clever loss functions and I tried many different losses including customized BCE based on ranking, but nothing significantly outperformed vanilla BCE. One thing worth noting is that, in the early stage of the competition, I trained an MSE model, which gave me a decent boost after simple blending with my BCE models, but it does not work well with the \"magic\".
\n\nMy teammate has shared our post-processing code. I am going to explain how we came up with each one of them.
\n\nThe first small trick is the culture list. This part was only meant for the 19th column, which is question_type_spelling. This came from the observation that this column is only non-zero when the host is english.stackexchange.com or ell.stackexchange.com.
The second (and the more significant) part of the trick is building out equalities among the predictions, based on this discussion.
\n\nThe very first attempt was to round the output assuming there were 90 raters. I believe many of you might be familiar with this assumption. This gave us a big boost about 0.03. This was when I found out that my blending of BCE and MSE did not work so well when applying this trick. The blended model performed far worse than my single BCE model after post-processing.
\n\nAn improvement on this trick came from a deeper look at the train labels. Assuming the 90-rater assumption were true, if I were among the raters, I would have given random points to some of the questions. This means if these scores are really given by humans, there should be 90 (or close) unique values for each column. It was easy to deduct the actual number of raters for each column based on the number of unique values. One comforting fact is that the lists of unique values are always the same for the columns with the same number of raters.
\n\nThis improvement did NOT work as I expected initially. But I really believed this was the way to go from my observations. What I then realized was that because the number of raters was too sparse and the original predictions of some columns were really small, the rounding might have squeezed many of the predictions to 0. Min-max scaling these predictions followed naturally considering the competition metric only cares about ranking rather than the actual values. Scaling gave us another ~0.02 boost.
\n\nAgain, I had great time working on this competition and learning about NLP. But as you can see, the main boost came from post-processing. I guess one takeaway is that if you hit the bottleneck of a competition, always take another look at the data.
\n\nCheers!
", "comments": [ { "id": 743154, "author_name": "CPMP", "author_url": "", "post_date": "2020-02-11T20:55:47.760000", "content": "congrats for the gold finish!
", "votes": 1, "replies": [ { "id": 743156, "author_name": "Tonghui Li", "author_url": "", "post_date": "2020-02-11T20:57:45.650000", "content": "Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 743151, "author_name": "Serigne ", "author_url": "", "post_date": "2020-02-11T20:54:26.330000", "content": "It was great pleasure to work with such talented guy like you @raphael1123
\n\nYour ideas with the post-processing gave us huge boost.
\n\nThanks again to all my teammates @xiaojiu1414 @chanhu @seif95 @raphael1123
", "votes": 1, "replies": [ { "id": 743158, "author_name": "Tonghui Li", "author_url": "", "post_date": "2020-02-11T21:00:12.750000", "content": "The feeling is mutual!
", "votes": 0, "replies": [] } ] }, { "id": 743530, "author_name": "Manan", "author_url": "", "post_date": "2020-02-12T06:10:08.050000", "content": "Congrats for gold and thanks for sharing appraoch
\n\n\" USE difference and entity embedding as external features in Bert..\"
\n\nexternal fetures as input to bert or concatinated with output [CLS] token ..\nand if it's input to bert then what was the structure .(E.g. [ cls ] q_title [ SEP ] external_f [ SEP ]...)
", "votes": 0, "replies": [ { "id": 743536, "author_name": "Tonghui Li", "author_url": "", "post_date": "2020-02-12T06:13:56.527000", "content": "Thanks. \nExternal features concatenated with CLS tokens then passed into regression heads.
", "votes": 0, "replies": [] } ] }, { "id": 743282, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-02-12T00:38:18.953000", "content": "Congratulations, and thanks for the overview, those last lines are very important.
", "votes": 0, "replies": [] } ], "comment_count": 7, "messages": [ { "id": 743117, "postDate": "2020-02-11T19:59:27.963Z", "content": "This is my first NLP competition and I am grateful to have gained my first gold. It was a great pleasure working with my talented and extremely hard working teammates @chanhu @xiaojiu1414 @serigne @seif95.
\n\nOur ensemble is nothing different from what other teams have shared, which includes bert base, bert large, xlnet, and roberta. I wanted to share a little bit my thought process on the submission trick.
\n\na few modeling tricks we have used are \n1) separate regression heads for questions and answers, \n2) head and tail tokenization \n3) concatenated [CLS] tokens as regression input\n4) USE difference and entity embedding as external features in Bert
\n\nI personally thought the magic was going to be clever loss functions and I tried many different losses including customized BCE based on ranking, but nothing significantly outperformed vanilla BCE. One thing worth noting is that, in the early stage of the competition, I trained an MSE model, which gave me a decent boost after simple blending with my BCE models, but it does not work well with the \"magic\".
\n\nMy teammate has shared our post-processing code. I am going to explain how we came up with each one of them.
\n\nThe first small trick is the culture list. This part was only meant for the 19th column, which is question_type_spelling. This came from the observation that this column is only non-zero when the host is english.stackexchange.com or ell.stackexchange.com.
The second (and the more significant) part of the trick is building out equalities among the predictions, based on this discussion.
\n\nThe very first attempt was to round the output assuming there were 90 raters. I believe many of you might be familiar with this assumption. This gave us a big boost about 0.03. This was when I found out that my blending of BCE and MSE did not work so well when applying this trick. The blended model performed far worse than my single BCE model after post-processing.
\n\nAn improvement on this trick came from a deeper look at the train labels. Assuming the 90-rater assumption were true, if I were among the raters, I would have given random points to some of the questions. This means if these scores are really given by humans, there should be 90 (or close) unique values for each column. It was easy to deduct the actual number of raters for each column based on the number of unique values. One comforting fact is that the lists of unique values are always the same for the columns with the same number of raters.
\n\nThis improvement did NOT work as I expected initially. But I really believed this was the way to go from my observations. What I then realized was that because the number of raters was too sparse and the original predictions of some columns were really small, the rounding might have squeezed many of the predictions to 0. Min-max scaling these predictions followed naturally considering the competition metric only cares about ranking rather than the actual values. Scaling gave us another ~0.02 boost.
\n\nAgain, I had great time working on this competition and learning about NLP. But as you can see, the main boost came from post-processing. I guess one takeaway is that if you hit the bottleneck of a competition, always take another look at the data.
\n\nCheers!
", "rawMarkdown": "This is my first NLP competition and I am grateful to have gained my first gold. It was a great pleasure working with my talented and extremely hard working teammates @chanhu @xiaojiu1414 @serigne @seif95.\n\nOur ensemble is nothing different from what other teams have shared, which includes bert base, bert large, xlnet, and roberta. I wanted to share a little bit my thought process on the submission trick. \n\n### **Before that**\na few modeling tricks we have used are \n1) separate regression heads for questions and answers, \n2) head and tail tokenization \n3) concatenated [CLS] tokens as regression input\n4) USE difference and entity embedding as external features in Bert\n\nI personally thought the magic was going to be clever loss functions and I tried many different losses including customized BCE based on ranking, but nothing significantly outperformed vanilla BCE. One thing worth noting is that, in the early stage of the competition, I trained an MSE model, which gave me a decent boost after simple blending with my BCE models, but it does not work well with the \"magic\".\n\n### **Back to the \"magic\"** \n\nMy teammate has shared [our post-processing code](https://www.kaggle.com/c/google-quest-challenge/discussion/129926). I am going to explain how we came up with each one of them. \n\nThe first small trick is the culture list. This part was only meant for the 19th column, which is `question_type_spelling`. This came from the observation that this column is only non-zero when the host is `english.stackexchange.com` or `ell.stackexchange.com`.\n\nThe second (and the more significant) part of the trick is building out equalities among the predictions, based on [this discussion](https://www.kaggle.com/c/google-quest-challenge/discussion/118724). \n\n***The very first attempt*** was to round the output assuming there were 90 raters. I believe many of you might be familiar with this assumption. This gave us a big boost about 0.03. This was when I found out that my blending of BCE and MSE did not work so well when applying this trick. The blended model performed far worse than my single BCE model after post-processing. \n \n***An improvement*** on this trick came from a deeper look at the train labels. Assuming the 90-rater assumption were true, if I were among the raters, I would have given random points to some of the questions. This means if these scores are really given by humans, there should be 90 (or close) unique values for each column. It was easy to deduct the actual number of raters for each column based on the number of unique values. One comforting fact is that the lists of unique values are always the same for the columns with the same number of raters. \n\n***This improvement did NOT work as I expected initially.*** But I really believed this was the way to go from my observations. What I then realized was that because the number of raters was too sparse and the original predictions of some columns were really small, the rounding might have squeezed many of the predictions to 0. ***Min-max scaling*** these predictions followed naturally considering the competition metric only cares about ranking rather than the actual values. Scaling gave us another ~0.02 boost. \n\n### **At the end**\nAgain, I had great time working on this competition and learning about NLP. But as you can see, the main boost came from post-processing. I guess one takeaway is that if you hit the bottleneck of a competition, always take another look at the data. \n\nCheers!", "votes": 16 }, { "id": 743154, "postDate": "2020-02-11T20:55:47.760Z", "content": "congrats for the gold finish!
", "rawMarkdown": "congrats for the gold finish!", "votes": 1, "replies": [ { "id": 743156, "postDate": "2020-02-11T20:57:45.650Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] }, { "id": 743151, "postDate": "2020-02-11T20:54:26.330Z", "content": "It was great pleasure to work with such talented guy like you @raphael1123
\n\nYour ideas with the post-processing gave us huge boost.
\n\nThanks again to all my teammates @xiaojiu1414 @chanhu @seif95 @raphael1123
", "rawMarkdown": "It was great pleasure to work with such talented guy like you @raphael1123 \n\nYour ideas with the post-processing gave us huge boost. \n\nThanks again to all my teammates @xiaojiu1414 @chanhu @seif95 @raphael1123 ", "votes": 1, "replies": [ { "id": 743158, "postDate": "2020-02-11T21:00:12.750Z", "content": "The feeling is mutual!
", "rawMarkdown": "The feeling is mutual! " } ] }, { "id": 743530, "postDate": "2020-02-12T06:10:08.050Z", "content": "Congrats for gold and thanks for sharing appraoch
\n\n\" USE difference and entity embedding as external features in Bert..\"
\n\nexternal fetures as input to bert or concatinated with output [CLS] token ..\nand if it's input to bert then what was the structure .(E.g. [ cls ] q_title [ SEP ] external_f [ SEP ]...)
", "rawMarkdown": "Congrats for gold and thanks for sharing appraoch\n\n\" USE difference and entity embedding as external features in Bert..\" \n\nexternal fetures as input to bert or concatinated with output [CLS] token ..\nand if it's input to bert then what was the structure .(E.g. [ cls ] q_title [ SEP ] external_f [ SEP ]...)", "replies": [ { "id": 743536, "postDate": "2020-02-12T06:13:56.527Z", "content": "Thanks. \nExternal features concatenated with CLS tokens then passed into regression heads.
", "rawMarkdown": "Thanks. \nExternal features concatenated with CLS tokens then passed into regression heads." } ] }, { "id": 743282, "postDate": "2020-02-12T00:38:18.953Z", "content": "Congratulations, and thanks for the overview, those last lines are very important.
", "rawMarkdown": "Congratulations, and thanks for the overview, those last lines are very important." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:08:16Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4014617, "solution_url": "/competitions/google-quest-challenge/writeups/wubba-lubba-dub-dub-how-we-found-magic-13th-soluti" }