![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F273250fcf82c1639aa2504e0ab874d50%2Fbody_crit.png?generation=1581440405894463&alt=media\")
Thank you everyone for sharing your great solutions. I'm learning so much!
\n\nI had only a few days to work on this competition and I didn't know anything about NLP. So I spent a few days reading about BERT and a few days exploring meta features and analyzing the Spearman's rank correlation metric.
\n\nI learned a lot from Akensert's great public kernel here. If you start with that and incorporate meta features and apply a Spearman rank transformation, you can achieve one position short of a Silver medal a.k.a The Best Bronze Solution :-)
\n\nConsider the following meta features\n* question_user_name\n* question_title_word_count, question_title_first_word, question_title_last_word\n* question_body_word_count, question_body_first_word, question_body_last_word\n* answer_user_name\n* answer_word_count, answer_first_word, answer_last_word\n* category, host\n* frequency encode all the above
If we build LGBM models for each target using these 26 meta features and display the top 3 feature importances, we discover some interesting relationships. Above each image below is the target name and LGBM Spearman's rank correlation validation score:
\n\n
Question body word count helps predict question_body_critical. That makes sense.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Fa9828b713cfe7aaf7bd3c4e6649ed813%2Finteresting_others.png?generation=1581440530534709&alt=media\")
Meta feature host helps predict whether a question is interesting to others. It's amazing that these meta features can achieve a score of 0.314 which is better than public BERT notebooks! Let's see how good host is by itself:
train, validate = train_test_split(train, test_size=0.2, random_state=42)\nmm = train.groupby('host')['question_interestingness_others'].mean().to_dict()\nvalidate['cat'] = validate.host.map(mm)\nspearmanr(validate.question_interestingness_others,validate.cat).correlation\n# This prints 0.311\nThis code shows that we can build a model from just the feature host and achieve a validation score of 0.311 predicting question_interestingness_others ! Wow
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Fee6342f07a2ee54aa08cc170746098cc%2Finteresting_self.png?generation=1581441808653553&alt=media\")
Once again, the meta feature host does a great job predicting question_interesting_self. By itself it scores 0.442 which does as good as public notebook BERTs!
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F520ced1e879a9661265024c4b9dda477%2Fspelling.png?generation=1581441968092095&alt=media\")
LGBM says that the most important feature to predict question_type_spelling is answer_word_count. From public kernels, we know that host=='ell.stackexchange.com' and host=='english.stackexchange.com' are very predictive too. (LGBM didn't find this which shows that LGBM is not optimizing Spearman's rank correlation). Using host and answer_word_count, we can predict question_type_spelling with score 0.300 ! Wow
validate['h_type'] = validate.host=='ell.stackexchange.com'\nvalidate['h_type'] |= validate.host=='english.stackexchange.com'\nvalidate['h_type'] &= validate.answer.map(lambda x: len(x.split()))>95\nspearmanr(validate.question_type_spelling,validate.h_type).correlation\n# This prints 0.300\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F0dfbb4bf8645dd9f043f7a61f751fcfa%2Fdefinition.png?generation=1581443385397518&alt=media\")
Question first word helps predict whether we have question_type_definition.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F69285f93b55a135fd87ba01f1d264754%2Fwell_written.png?generation=1581442904056045&alt=media\")
Question first word frequency helps predict whether a question is well written. That's interesting. If your writing is similar to how other people write, it is deemed well written.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Fd2b111e2e208ff739dbce612b5444f5d%2Finfo.png?generation=1581443008426365&alt=media\")
Answer word count helps predict answer_level_of_information. That makes sense. The code below shows that this single feature alone can predict better than public BERT notebooks!
validate['a_ct'] = validate.answer.map(lambda x: len(x.split()))\nspearmanr(validate.answer_level_of_information,validate.a_ct).correlation\n# This prints 0.392\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Fbad5aaee8d43f49f3a9282c3381ae25f%2Fplausible.png?generation=1581443262147882&alt=media\")
Answer user name frequency helps predict whether an answer is plausible. This makes sense. Active users are probably more plausible.
\n\nNow that we know important meta features, we can either incorporate them into our BERT model inputs, or we can ensemble them with BERT's outputs. Using my OOF, I ensembled meta features to my BERT outputs. For example, the two best CV/LB increases came from replacing BERT's outputs for question_type_spelling with
test['a_ct'] = test.answer.map(lambda x: len(x.split()))\ntest['h_type'] = test.host=='ell.stackexchange.com'\ntest['h_type'] |= test.host=='english.stackexchange.com'\ntest['h_type'] &= test.a_ct>95\nsub.question_type_spelling = test.h_type.values \nAnd updating answer_level_of_information with:
sub.answer_level_of_information += 7 * test.a_ct.values/8158\nsub.answer_level_of_information /= sub.answer_level_of_information.max() \nAs pointed out in this discussion post here, the Spearman's rank correlation metric can be improved if you convert continuous predictions into more discrete predictions if the true targets are discrete. By applying a grid search on my OOF, I found that applying numpy.clip( pred, clippings[col][0], clippings[col][1] ) can maximize this metric using
clippings = {\n 'question_has_commonly_accepted_answer':[0,0.6],\n 'question_conversational':[0.15,1],\n 'question_multi_intent':[0.1,1],\n 'question_type_choice':[0.1,1],\n 'question_type_compare':[0.1,1],\n 'question_type_consequence':[0.08,1],\n 'question_type_definition':[0.1,1],\n 'question_type_entity':[0.13,1]\n}\nDuring the competition, I only built and trained one bert-base-uncased model. Other models didn't have good CVs. But after reading all the great solutions, I just built a roberta-base model by first applying def cln(x): return \" \".join(x.split()) which removes white space (as described in 8th place solution here).
It turns out that RobertaTokenizer.from_pretrained('roberta-base') does not remove white space by itself. (whereas the bert-base-uncased tokenizer does). Using the same architecture as my bert-base, my Roberta has CV 0.395. When I ensemble it with my bert-base CV 0.392, the new CV is 0.412. That's an increase of 0.020 over my bert-base.
I just submitted to the leaderboard and it increased my LB by 0.015 and my rank by 40 spots! Wow! I just learned how to make better NLP models. I can't wait for the next competition !
", "comments": [ { "id": 750188, "author_name": "", "author_url": "", "post_date": "2020-02-19T07:22:38.873000", "content": "tq u for sharing
", "votes": 1, "replies": [] }, { "id": 745579, "author_name": "oohara", "author_url": "", "post_date": "2020-02-14T00:38:22.843000", "content": "Very impressive feature analysis! Did you use the meta data as a input of your model? If you did, what many it boost your score?
", "votes": 1, "replies": [ { "id": 745604, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-14T01:33:34.853000", "content": "Thanks. I tried using them as inputs but I couldn't get that to work well. Instead I ensembled them with BERT outputs. The boosts were as follows: question_type_spelling gave +0.007, answer_level_of_info gave +0.003, question_interesting_self and question_interesting_others gave +0.001. And post process gave +0.030. So In total it increased CV/LB by +0.041.
Congrats on the new job and the Top Bronze Score!\nI hope Nvidia encourages Kaggle participation.
", "votes": 1, "replies": [] }, { "id": 743645, "author_name": "Massoud Hosseinali", "author_url": "", "post_date": "2020-02-12T06:49:55.620000", "content": "Congratulations Chris! It is great to see you competing again. I missed your informative analyses/discussions in recent competitions.
\n\nBtw, congratulations on your new role at Nvidia too!
", "votes": 1, "replies": [ { "id": 743656, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-12T06:56:13.640000", "content": "Thanks Massoud. It's great to be back playing at Kaggle.
", "votes": 1, "replies": [] } ] }, { "id": 743386, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-02-12T02:44:46.673000", "content": "Congratulations!\nGreat Write-Up\nThanks for Sharing your Approach & Insights!! @cdeotte
", "votes": 1, "replies": [] }, { "id": 743332, "author_name": "Zhuang Jia", "author_url": "", "post_date": "2020-02-12T01:51:43.737000", "content": "Very impressive @cdeotte! Can always learn a lot from you!
", "votes": 1, "replies": [] }, { "id": 743265, "author_name": "Gerwyn", "author_url": "", "post_date": "2020-02-12T00:23:08.253000", "content": "Thanks for sharing your findings!! I did not know that removing white space is needed for roberta base... Same as you, i tried various bert models but couldnt beat the baseline bert base uncased seems like special attention is required for each bert...
\n\nBtw, are you using tf or pytorch?
", "votes": 1, "replies": [ { "id": 743313, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-12T01:20:45.247000", "content": "I did everything in TensorFlow. I'm training a XLNet model now. I'm gonna ensemble Bert, Roberta, and XLNet and see how well that does. It looks like ensembling different Berts was the way to go. (And then add post process and meta feature updates).
", "votes": 1, "replies": [] }, { "id": 743340, "author_name": "Gerwyn", "author_url": "", "post_date": "2020-02-12T02:02:02.240000", "content": "what's your trick to make results deterministic?
", "votes": 1, "replies": [] }, { "id": 743353, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-12T02:18:27.630000", "content": "I just use the same 5-Folds for every experiment to compare CV scores. The rest of the randomness, I don't care about.
", "votes": 1, "replies": [] } ] }, { "id": 744197, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-02-12T16:42:23.760000", "content": "@cdeotte double congratulations on your performance here and your new Nvidia gig. Well done.
", "votes": 2, "replies": [] }, { "id": 851177, "author_name": "", "author_url": "", "post_date": "2020-05-17T12:25:11.800000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 14, "messages": [ { "id": 743135, "postDate": "2020-02-11T20:24:46.037Z", "content": "Thank you everyone for sharing your great solutions. I'm learning so much!
\n\nI had only a few days to work on this competition and I didn't know anything about NLP. So I spent a few days reading about BERT and a few days exploring meta features and analyzing the Spearman's rank correlation metric.
\n\nI learned a lot from Akensert's great public kernel here. If you start with that and incorporate meta features and apply a Spearman rank transformation, you can achieve one position short of a Silver medal a.k.a The Best Bronze Solution :-)
\n\nConsider the following meta features\n* question_user_name\n* question_title_word_count, question_title_first_word, question_title_last_word\n* question_body_word_count, question_body_first_word, question_body_last_word\n* answer_user_name\n* answer_word_count, answer_first_word, answer_last_word\n* category, host\n* frequency encode all the above
If we build LGBM models for each target using these 26 meta features and display the top 3 feature importances, we discover some interesting relationships. Above each image below is the target name and LGBM Spearman's rank correlation validation score:
\n\n
Question body word count helps predict question_body_critical. That makes sense.
Meta feature host helps predict whether a question is interesting to others. It's amazing that these meta features can achieve a score of 0.314 which is better than public BERT notebooks! Let's see how good host is by itself:
train, validate = train_test_split(train, test_size=0.2, random_state=42)\nmm = train.groupby('host')['question_interestingness_others'].mean().to_dict()\nvalidate['cat'] = validate.host.map(mm)\nspearmanr(validate.question_interestingness_others,validate.cat).correlation\n# This prints 0.311\nThis code shows that we can build a model from just the feature host and achieve a validation score of 0.311 predicting question_interestingness_others ! Wow
Once again, the meta feature host does a great job predicting question_interesting_self. By itself it scores 0.442 which does as good as public notebook BERTs!
LGBM says that the most important feature to predict question_type_spelling is answer_word_count. From public kernels, we know that host=='ell.stackexchange.com' and host=='english.stackexchange.com' are very predictive too. (LGBM didn't find this which shows that LGBM is not optimizing Spearman's rank correlation). Using host and answer_word_count, we can predict question_type_spelling with score 0.300 ! Wow
validate['h_type'] = validate.host=='ell.stackexchange.com'\nvalidate['h_type'] |= validate.host=='english.stackexchange.com'\nvalidate['h_type'] &= validate.answer.map(lambda x: len(x.split()))>95\nspearmanr(validate.question_type_spelling,validate.h_type).correlation\n# This prints 0.300\n
Question first word helps predict whether we have question_type_definition.
Question first word frequency helps predict whether a question is well written. That's interesting. If your writing is similar to how other people write, it is deemed well written.
\n\n
Answer word count helps predict answer_level_of_information. That makes sense. The code below shows that this single feature alone can predict better than public BERT notebooks!
validate['a_ct'] = validate.answer.map(lambda x: len(x.split()))\nspearmanr(validate.answer_level_of_information,validate.a_ct).correlation\n# This prints 0.392\n
Answer user name frequency helps predict whether an answer is plausible. This makes sense. Active users are probably more plausible.
\n\nNow that we know important meta features, we can either incorporate them into our BERT model inputs, or we can ensemble them with BERT's outputs. Using my OOF, I ensembled meta features to my BERT outputs. For example, the two best CV/LB increases came from replacing BERT's outputs for question_type_spelling with
test['a_ct'] = test.answer.map(lambda x: len(x.split()))\ntest['h_type'] = test.host=='ell.stackexchange.com'\ntest['h_type'] |= test.host=='english.stackexchange.com'\ntest['h_type'] &= test.a_ct>95\nsub.question_type_spelling = test.h_type.values \nAnd updating answer_level_of_information with:
sub.answer_level_of_information += 7 * test.a_ct.values/8158\nsub.answer_level_of_information /= sub.answer_level_of_information.max() \nAs pointed out in this discussion post here, the Spearman's rank correlation metric can be improved if you convert continuous predictions into more discrete predictions if the true targets are discrete. By applying a grid search on my OOF, I found that applying numpy.clip( pred, clippings[col][0], clippings[col][1] ) can maximize this metric using
clippings = {\n 'question_has_commonly_accepted_answer':[0,0.6],\n 'question_conversational':[0.15,1],\n 'question_multi_intent':[0.1,1],\n 'question_type_choice':[0.1,1],\n 'question_type_compare':[0.1,1],\n 'question_type_consequence':[0.08,1],\n 'question_type_definition':[0.1,1],\n 'question_type_entity':[0.13,1]\n}\nDuring the competition, I only built and trained one bert-base-uncased model. Other models didn't have good CVs. But after reading all the great solutions, I just built a roberta-base model by first applying def cln(x): return \" \".join(x.split()) which removes white space (as described in 8th place solution here).
It turns out that RobertaTokenizer.from_pretrained('roberta-base') does not remove white space by itself. (whereas the bert-base-uncased tokenizer does). Using the same architecture as my bert-base, my Roberta has CV 0.395. When I ensemble it with my bert-base CV 0.392, the new CV is 0.412. That's an increase of 0.020 over my bert-base.
I just submitted to the leaderboard and it increased my LB by 0.015 and my rank by 40 spots! Wow! I just learned how to make better NLP models. I can't wait for the next competition !
", "rawMarkdown": "Thank you everyone for sharing your great solutions. I'm learning so much!\n\nI had only a few days to work on this competition and I didn't know anything about NLP. So I spent a few days reading about BERT and a few days exploring meta features and analyzing the Spearman's rank correlation metric. \n\nI learned a lot from Akensert's great public kernel [here][1]. If you start with that and incorporate meta features and apply a Spearman rank transformation, you can achieve one position short of a Silver medal a.k.a The Best Bronze Solution :-)\n\n# Meta Features\nConsider the following meta features\n* `question_user_name`\n* `question_title_word_count`, `question_title_first_word`, `question_title_last_word`\n* `question_body_word_count`, `question_body_first_word`, `question_body_last_word`\n* `answer_user_name`\n* `answer_word_count`, `answer_first_word`, `answer_last_word`\n* `category`, `host`\n* frequency encode all the above\n\nIf we build LGBM models for each target using these 26 meta features and display the top 3 feature importances, we discover some interesting relationships. Above each image below is the target name and LGBM Spearman's rank correlation validation score:\n\n## Question Body Critical\n\n\n\nQuestion body word count helps predict `question_body_critical`. That makes sense.\n\n## Question Interesting Others\n\n\n\nMeta feature `host` helps predict whether a question is interesting to others. It's amazing that these meta features can achieve a score of 0.314 which is better than public BERT notebooks! Let's see how good `host` is by itself:\n\n train, validate = train_test_split(train, test_size=0.2, random_state=42)\n mm = train.groupby('host')['question_interestingness_others'].mean().to_dict()\n validate['cat'] = validate.host.map(mm)\n spearmanr(validate.question_interestingness_others,validate.cat).correlation\n # This prints 0.311\n\nThis code shows that we can build a model from just the feature host and achieve a validation score of 0.311 predicting `question_interestingness_others` ! Wow\n\n## Question Interesting Self\n\n\n\nOnce again, the meta feature `host` does a great job predicting `question_interesting_self`. By itself it scores 0.442 which does as good as public notebook BERTs!\n\n## Question Type Spelling\n\n\n\nLGBM says that the most important feature to predict `question_type_spelling` is `answer_word_count`. From public kernels, we know that `host=='ell.stackexchange.com'` and `host=='english.stackexchange.com'` are very predictive too. (LGBM didn't find this which shows that LGBM is not optimizing Spearman's rank correlation). Using `host` and `answer_word_count`, we can predict `question_type_spelling` with score 0.300 ! Wow\n\n validate['h_type'] = validate.host=='ell.stackexchange.com'\n validate['h_type'] |= validate.host=='english.stackexchange.com'\n validate['h_type'] &= validate.answer.map(lambda x: len(x.split()))>95\n spearmanr(validate.question_type_spelling,validate.h_type).correlation\n # This prints 0.300\n\n## Question Type Definition\n\n\n\nQuestion first word helps predict whether we have `question_type_definition`.\n\n## Question Well Written\n\n\n\nQuestion first word frequency helps predict whether a question is well written. That's interesting. If your writing is similar to how other people write, it is deemed well written.\n\n## Answer Level of Information\n\n\n\nAnswer word count helps predict `answer_level_of_information`. That makes sense. The code below shows that this single feature alone can predict better than public BERT notebooks!\n\n validate['a_ct'] = validate.answer.map(lambda x: len(x.split()))\n spearmanr(validate.answer_level_of_information,validate.a_ct).correlation\n # This prints 0.392\n\n## Answer Plausible\n\n \nAnswer user name frequency helps predict whether an answer is plausible. This makes sense. Active users are probably more plausible.\n\n# Meta Features Applied\nNow that we know important meta features, we can either incorporate them into our BERT model inputs, or we can ensemble them with BERT's outputs. Using my OOF, I ensembled meta features to my BERT outputs. For example, the two best CV/LB increases came from replacing BERT's outputs for `question_type_spelling` with \n\n test['a_ct'] = test.answer.map(lambda x: len(x.split()))\n test['h_type'] = test.host=='ell.stackexchange.com'\n test['h_type'] |= test.host=='english.stackexchange.com'\n test['h_type'] &= test.a_ct>95\n sub.question_type_spelling = test.h_type.values \n\nAnd updating `answer_level_of_information` with:\n\n sub.answer_level_of_information += 7 * test.a_ct.values/8158\n sub.answer_level_of_information /= sub.answer_level_of_information.max() \n\n# Spearman's Rank Correlation\nAs pointed out in this discussion post [here][2], the Spearman's rank correlation metric can be improved if you convert continuous predictions into more discrete predictions if the true targets are discrete. By applying a grid search on my OOF, I found that applying `numpy.clip( pred, clippings[col][0], clippings[col][1] )` can maximize this metric using\n\n clippings = {\n 'question_has_commonly_accepted_answer':[0,0.6],\n 'question_conversational':[0.15,1],\n 'question_multi_intent':[0.1,1],\n 'question_type_choice':[0.1,1],\n 'question_type_compare':[0.1,1],\n 'question_type_consequence':[0.08,1],\n 'question_type_definition':[0.1,1],\n 'question_type_entity':[0.13,1]\n }\n\n# UPDATE\nDuring the competition, I only built and trained one `bert-base-uncased` model. Other models didn't have good CVs. But after reading all the great solutions, I just built a `roberta-base` model by first applying `def cln(x): return \" \".join(x.split())` which removes white space (as described in 8th place solution [here][3]). \n\nIt turns out that `RobertaTokenizer.from_pretrained('roberta-base')` does not remove white space by itself. (whereas the bert-base-uncased tokenizer does). Using the same architecture as my bert-base, my Roberta has CV 0.395. When I ensemble it with my bert-base CV 0.392, the new CV is 0.412. That's an increase of 0.020 over my bert-base. \n\nI just submitted to the leaderboard and it increased my LB by 0.015 and my rank by 40 spots! Wow! I just learned how to make better NLP models. I can't wait for the next competition ! \n\n[1]: https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer\n[2]: https://www.kaggle.com/c/google-quest-challenge/discussion/118724\n[3]: https://www.kaggle.com/c/google-quest-challenge/discussion/129857", "votes": 41 }, { "id": 750188, "postDate": "2020-02-19T07:22:38.873Z", "content": "tq u for sharing
", "rawMarkdown": "tq u for sharing", "votes": 1 }, { "id": 745579, "postDate": "2020-02-14T00:38:22.843Z", "content": "Very impressive feature analysis! Did you use the meta data as a input of your model? If you did, what many it boost your score?
", "rawMarkdown": "Very impressive feature analysis! Did you use the meta data as a input of your model? If you did, what many it boost your score?", "votes": 1, "replies": [ { "id": 745604, "postDate": "2020-02-14T01:33:34.853Z", "content": "Thanks. I tried using them as inputs but I couldn't get that to work well. Instead I ensembled them with BERT outputs. The boosts were as follows: question_type_spelling gave +0.007, answer_level_of_info gave +0.003, question_interesting_self and question_interesting_others gave +0.001. And post process gave +0.030. So In total it increased CV/LB by +0.041.
Congrats on the new job and the Top Bronze Score!\nI hope Nvidia encourages Kaggle participation.
", "rawMarkdown": "Congrats on the new job and the Top Bronze Score!\nI hope Nvidia encourages Kaggle participation.", "votes": 1 }, { "id": 743645, "postDate": "2020-02-12T06:49:55.620Z", "content": "Congratulations Chris! It is great to see you competing again. I missed your informative analyses/discussions in recent competitions.
\n\nBtw, congratulations on your new role at Nvidia too!
", "rawMarkdown": "Congratulations Chris! It is great to see you competing again. I missed your informative analyses/discussions in recent competitions.\n\nBtw, congratulations on your new role at Nvidia too!", "votes": 1, "replies": [ { "id": 743656, "postDate": "2020-02-12T06:56:13.640Z", "content": "Thanks Massoud. It's great to be back playing at Kaggle.
", "rawMarkdown": "Thanks Massoud. It's great to be back playing at Kaggle.", "votes": 1 } ] }, { "id": 743386, "postDate": "2020-02-12T02:44:46.673Z", "content": "Congratulations!\nGreat Write-Up\nThanks for Sharing your Approach & Insights!! @cdeotte
", "rawMarkdown": "Congratulations!\nGreat Write-Up\nThanks for Sharing your Approach & Insights!! @cdeotte ", "votes": 1 }, { "id": 743332, "postDate": "2020-02-12T01:51:43.737Z", "content": "Very impressive @cdeotte! Can always learn a lot from you!
", "rawMarkdown": "Very impressive @cdeotte! Can always learn a lot from you!", "votes": 1 }, { "id": 743265, "postDate": "2020-02-12T00:23:08.253Z", "content": "Thanks for sharing your findings!! I did not know that removing white space is needed for roberta base... Same as you, i tried various bert models but couldnt beat the baseline bert base uncased seems like special attention is required for each bert...
\n\nBtw, are you using tf or pytorch?
", "rawMarkdown": "Thanks for sharing your findings!! I did not know that removing white space is needed for roberta base... Same as you, i tried various bert models but couldnt beat the baseline bert base uncased seems like special attention is required for each bert...\n\nBtw, are you using tf or pytorch? ", "votes": 1, "replies": [ { "id": 743313, "postDate": "2020-02-12T01:20:45.247Z", "content": "I did everything in TensorFlow. I'm training a XLNet model now. I'm gonna ensemble Bert, Roberta, and XLNet and see how well that does. It looks like ensembling different Berts was the way to go. (And then add post process and meta feature updates).
", "rawMarkdown": "I did everything in TensorFlow. I'm training a XLNet model now. I'm gonna ensemble Bert, Roberta, and XLNet and see how well that does. It looks like ensembling different Berts was the way to go. (And then add post process and meta feature updates).", "votes": 1 }, { "id": 743340, "postDate": "2020-02-12T02:02:02.240Z", "content": "what's your trick to make results deterministic?
", "rawMarkdown": "what's your trick to make results deterministic? ", "votes": 1 }, { "id": 743353, "postDate": "2020-02-12T02:18:27.630Z", "content": "I just use the same 5-Folds for every experiment to compare CV scores. The rest of the randomness, I don't care about.
", "rawMarkdown": "I just use the same 5-Folds for every experiment to compare CV scores. The rest of the randomness, I don't care about.", "votes": 1 } ] }, { "id": 744197, "postDate": "2020-02-12T16:42:23.760Z", "content": "@cdeotte double congratulations on your performance here and your new Nvidia gig. Well done.
", "rawMarkdown": "@cdeotte double congratulations on your performance here and your new Nvidia gig. Well done.", "votes": 2 }, { "id": 851177, "postDate": "2020-05-17T12:25:11.800Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:08:17Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4338635, "solution_url": "/competitions/google-quest-challenge/writeups/chris-deotte-79th-place-meta-feature-analysis" }