![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F512750%2F0555fde7612232dd8de8c25c2f6f5aa1%2Fscores_hist.png?generation=1581430695438632&alt=media\")
Congratulations to all teams and especially to daring solo players! Thanks to organizers as well!
\n\nTremendous feeling for the whole team. I happened to finish one place away from the golden zone in 2 NLP competitions, and there it is. A few posts below I wrote down \"respect your efforts, not results\" and you'll be there. There is always a room for such motivational quotes. I don't know whether it's harder to compete with yourself or with such great minds. Thanks to the team ( @ddanevskiy , @kashnitsky , @dmitriyab ) for this thrilling race. And thanks to the whole community for ideas, criticism, and a sense of humor.
\n\nEDIT
\n\nThe community seems to be waiting for a killer feature from us. I don’t reckon we’ve got a single one. Our solution is comprised of three cornerstones:\n- pretrained language models\n- pseudo-labeling\n- post-processing (yes we did one)
\n\nThe basic pipeline
\n\nLess than 3 weeks ago just when we were done with the TF Q&A competition, we jumped into this one. Oh yes, there were already so many great NLP folks seen on the leaderboard. We started from a public pytorch baseline [(0.377 I guess)] (https://www.kaggle.com/phoenix9032/pytorch-bert-plain) (kudos @phoenix9032) and managed to update it effectively (0.396 for BERT-base and 0.402 for BERT-large) with a couple of tricks:\n- GroupKFold with question_title groups\n- Multi-Sample Dropout for Accelerated Training and Better Generalization, paper\n- Different learning rate settings for encoder and head (deserves a paper @ddanevskyi)\n- Using CLS outputs from all BERT layers rather than using only the last one. We did this by using a weighted sum of these outputs, where the weights were learnable and constrained to be positive and sum to 1
\n\nWe experimented with separate weights for different targets and with various loss functions but eventually ended up sticking to simple BCE loss.
\n\nPretrained language models
\n\nWhen @dmitriyab joined our team, he introduced stackexchange (SE) pretrained LMs with 110k-long dictionary including code and other domain tokens. In addition to the common MLM task, the model also predicted 6 auxiliary targets (question_score, question_view_count, question_favorite_count, answer_score, answers_count, is_answer_accepted) with a linear layer on top of pooled LM output. That looked beneficial enough for me to drop my code replacement experiments and update the existing bert pipeline with a huge pretrained embedding layer -> 0.414. In the end @dmitriyab and @kashnitsky trained a few LMs (bert-large, roberta, roberta-large) on 7M SE data using whole-word-masking, but the initial model was still the best.
Pseudo-labeling
\n\nPseudo-labeling is a well known semi-supervised learning technique within the Kaggle community and sometimes I wonder how they reinvent stuff in academic papers like this one (btw a great result). So we labeled additional 100k samples using our three basic bert models. First experiments with pseudo-labels showed huge improvement for SE pretrained bert (0.414 -> 0.445). That was huge. But the leaderboard did not agree.
\n\nPreviously, my teammates and I used models from all folds to create pseudolabels. This strategy worked well in many cases including recent Recursion competition, where my team finished 6th. @ddanevskyi used the same approach for his 3rd place on Freesound. This way of creating pseudolabels gives quite accurate predictions as you are using a fold ensemble, but has an unobvious pitfall. If the pseudolabeled set contains a lot of similar samples to the training set, you could expect that some actual training labels could leak through those samples, even if there is no “direct” target leakage.
\n\nImagine that you’ve trained a 5-fold model on the training set then used those 5 folds to create pseudolabels on some external dataset. Then you add those pseudolabels to the training part of each of 5 folds. Very likely you will see an overoptimistic score on the validation part as 4 out of 5 models used to generate pseudolabels have “seen” the current validation set through their training process.
\n\nWe could expect a lot of very similar questions/answers in SO&SE data, so we clearly couldn’t rely on that sort of pseudolabels. In order to fix the problem, we generated 5 different sets of pseudolabels where for each train/val split we used only those models that were trained using only the current train set. This gave much less accurate pseudolabels, but eliminated any possible source of target leakage.
\n\nAs expected, it didn’t bring such an improvement compared to a “leaked” approach (0.414 -> 0.422) but that was a wise decision in terms of tackling overfitting.
\n\nBART
\n\nOne week prior to the deadline, we noticed @christofhenkel ’s post in the external data thread - the one about fairseq BART . Autoregressive sequence-to-sequence model for text classification? Looked strange to me but at the same time that was something new and daring, and we wouldn’t ignore a hint by such a prominent kaggler. It took us three whole days to train BART on 1080ti with batch size 2. The model scored 0.427 locally and it’s predictions decorrelated well with those of BERT.
\n\nWe also trained a few other bert-base, bert-large, roberta-base, and roberta-large models. They all scored around 0.420 locally but didn’t improve drastically our bert+bart baseline.
\n\nPost-processing
\n\nAnd finally, post-processing. Yes we did one. Based on this discussion we found that discretization of predictions for some challenging targets led to better spearman corr. So we used no thresholds but discretized such columns based on their distribution in the training set. That simple heuristics boosted score by 0.027-0.030 for almost every single model (up to 0.448 bert-base, 0.454 bart).
\n\nFinish
\n\nOur local validation and public score stayed the same for the whole last week - 0.468. Yesterday, since early morning with all this berts, barts, robertas we were not able to reproduce public score :harold:, having only 0.462 submissions but they seem to work the best -> 0.430-0.433 private leaderboard (unfortunately we didn’t select the best one). But that was good enough.
\n\nWhat didn’t happen to work well
\n\nWonderful teamwork and team patience (not really haha). Thanks to you all. We are making our submission kernels and datasets public. By the end of the day I’m releasing our code on GitHub.
\n\nEDIT\nCode released here. There are 3 separate branches. I'll update with a nice readme soon.
\n\nOleg Yaroshevskiy, Dmitriy Danevskiy, Yury Kashnitsky, Dmitriy Abulkhanov
", "comments": [ { "id": 742757, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-02-11T14:25:47.420000", "content": "I would also like to share two useful things.
\n\nThe first one is quite simple. At some point, I decided to figure out how reliable the public leaderboard could be. To do so, I created 150 random train/test splits of the OOF scores such that the test part of each split contained around 450 samples. Then, I computed the target metric on each of those \"test\" datasets and computed the mean and standard deviation of these scores. The standard deviation appeared to be around 0.02 which made us think that we could trust the public score only up to +- 0.02. When I used 8x bigger test size (which is roughly the size of the private test set), the standard deviation lowered to around 0.003.
\n\nHere you can see a distribution of the scores for 150 splits:
\n\n
The second thing is about proper tooling. About 2 years ago I spent a couple of evenings to create a very tiny python library called mag that addressed the problem of machine learning experiment management. Since then, I used it in every single competition including this one. This really helped our team to don't lose track of all the models we trained for these two weeks. I hope some of you might find it useful too.
", "votes": 24, "replies": [ { "id": 742764, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2020-02-11T14:29:28.017000", "content": "I confirm this - using mag turned out to be very straightforward. Although I'm typically frustrated with trying to use \"yet another bicycle\" (like poutyne used by my teammate monty to manage LM training).
", "votes": 3, "replies": [] }, { "id": 742768, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-02-11T14:31:59.490000", "content": "I totally agree with this \"yet another bicycle\" concern. That's why I came up with my own ;)
", "votes": 2, "replies": [] }, { "id": 742774, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2020-02-11T14:34:03.937000", "content": ":slow: \n![](\"https://habrastorage.org/webt/lc/pe/tg/lcpetgdnrzhrbc4i-vfwjq-dtwo.png\")
What are the other bicycles? Any hint for the term to use on Google to search for that? Maybe \"machine learning experiment management\"?
", "votes": 0, "replies": [] }, { "id": 743291, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2020-02-12T00:57:06.150000", "content": "MLFlow is the most prominent one.
", "votes": 0, "replies": [] }, { "id": 743963, "author_name": "liuze", "author_url": "", "post_date": "2020-02-12T12:47:38.990000", "content": "mag is good, I would try, thanks
", "votes": 0, "replies": [] }, { "id": 755943, "author_name": "Yu-ray Li", "author_url": "", "post_date": "2020-02-25T09:41:29.130000", "content": "@ddanevskyi Thank you for sharing!\nI need to ask a dumb question. You said you created 150 random train/test splits of the OOF scores. Does that mean you trained these data 150 times?\nThank you!!
", "votes": 0, "replies": [] }, { "id": 756192, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-02-25T13:53:47.483000", "content": "Not really, I just use the same OOF predictions
", "votes": 1, "replies": [] } ] }, { "id": 742515, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-02-11T10:35:14.300000", "content": "On behalf of our team, I have to apologize we didn't post the solution yet. The reason is we slept for 2-3 hours at the very most and now we need to get some rest. You might expect the solution to be posted in several hours. The code will be posted a bit later as we need to add some READMEs and do a bit of refactoring.
", "votes": 8, "replies": [ { "id": 742516, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-02-11T10:37:02.280000", "content": "We can wait patiently for that minimal post-processing sol :)
", "votes": 0, "replies": [] } ] }, { "id": 753654, "author_name": "Chirag Samal", "author_url": "", "post_date": "2020-02-22T14:05:31.183000", "content": "Congratulations @yaroshevskiy and thanks for sharing
", "votes": 6, "replies": [] }, { "id": 743732, "author_name": "Firas Baba", "author_url": "", "post_date": "2020-02-12T08:41:36.200000", "content": "Congarts! I can say this is the best solution and it deserves 1st place\nRegarding the pseudo-labeling, We used a similar idea and it turns out that using 5 predictions (from 5 fold) separately wont leak the target because when we averaged the predictions of 5 models we were getting 0.53+ CV with the distilled data.\nUsing this technique correctly introduced improvement for CV in training folds (+0.005 per fold) for each fold without postprocessing but less score on oof which was so weird (got same score on LB) and we decided to leave the idea.\nMy question is: did you post-process the distilled labels? or you trained with it as it was given by the teacher model?
", "votes": 6, "replies": [ { "id": 744163, "author_name": "Oleg Yaroshevskiy", "author_url": "", "post_date": "2020-02-12T16:14:16.287000", "content": "Oh that's a nice question. We decided that post-processing leads only to better spearman corr but not to better classification distribution so we left them untouched.
", "votes": 2, "replies": [] }, { "id": 744176, "author_name": "Firas Baba", "author_url": "", "post_date": "2020-02-12T16:33:36.880000", "content": "same for us. When we post-process the distilled data spearman drop without PP (from 0.41x to 0.38x) but increase when we apply PP (from 0.45 to 0.48). We couldn't trust this since the model makes predictions so weird and we were worried that model is overfiting on the validation (PP improves with 0.1+ lol).\nI am so glad that this worked for you. Chapeau bas!
", "votes": 2, "replies": [] } ] }, { "id": 747982, "author_name": "AnandYadav", "author_url": "", "post_date": "2020-02-17T04:19:26.070000", "content": "Congratulations !!!! and thanks for sharing.
", "votes": 3, "replies": [] }, { "id": 744334, "author_name": "James Ingram", "author_url": "", "post_date": "2020-02-12T18:44:49.670000", "content": "Congratulations and many thanks for you and your team for the insights!
\n\nВітаємо та дякуємо вашій команді за ідеї
\n\nПоздравления и большое спасибо вам и вашей команде за идеи
", "votes": 3, "replies": [] }, { "id": 742742, "author_name": "CPMP", "author_url": "", "post_date": "2020-02-11T14:17:41.460000", "content": "Congrats on the result. It would be interesting to see how far we could go by combining our methods.
", "votes": 3, "replies": [] }, { "id": 742268, "author_name": "Vitaliy Fedorovych", "author_url": "", "post_date": "2020-02-11T07:42:07.673000", "content": "Wow, congrats @yaroshevskiy! I'm so glad you and your team won this competition! It made my day)
", "votes": 3, "replies": [] }, { "id": 744173, "author_name": "Muhammet Ikbal Elek", "author_url": "", "post_date": "2020-02-12T16:32:53.603000", "content": "Congrats!
", "votes": 4, "replies": [] }, { "id": 742720, "author_name": "Oleg Yaroshevskiy", "author_url": "", "post_date": "2020-02-11T14:02:29.153000", "content": "Updated with the solution. Thanks for your patience!\nSolution kernel. GitHub code is coming!
", "votes": 4, "replies": [ { "id": 742723, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-02-11T14:04:57.050000", "content": "Wow all !python flags Really cool And Clean solution ! And A great write-up!
Congratulations !!!
", "votes": 1, "replies": [] }, { "id": 747914, "author_name": "Vaibhav Saxena", "author_url": "", "post_date": "2020-02-17T02:23:37.457000", "content": "Powerful stuff. Congratulations 👍
", "votes": 1, "replies": [] }, { "id": 747894, "author_name": "Aman Sharma", "author_url": "", "post_date": "2020-02-17T01:27:54.990000", "content": "Congratulations !!
", "votes": 1, "replies": [] }, { "id": 747798, "author_name": "Nader A.Rasoul", "author_url": "", "post_date": "2020-02-16T21:41:34.363000", "content": "Congratulations! awesome work!
", "votes": 1, "replies": [] }, { "id": 747784, "author_name": "Paulo Castro", "author_url": "", "post_date": "2020-02-16T21:01:14.313000", "content": "Good topic!
", "votes": 1, "replies": [] }, { "id": 747738, "author_name": "Ramil", "author_url": "", "post_date": "2020-02-16T19:30:56.120000", "content": "Congratulations!
", "votes": 1, "replies": [] }, { "id": 747203, "author_name": "MonsirData", "author_url": "", "post_date": "2020-02-16T05:30:08.140000", "content": "Congratulations and thank you for your insight!
", "votes": 1, "replies": [] }, { "id": 747141, "author_name": "Mohammad Shahnawaz", "author_url": "", "post_date": "2020-02-16T02:38:09.420000", "content": "Great
", "votes": 1, "replies": [] }, { "id": 746969, "author_name": "AcidentlGenius", "author_url": "", "post_date": "2020-02-15T19:45:46.960000", "content": "Congratulations
", "votes": 1, "replies": [] }, { "id": 746884, "author_name": "Guilherme Cadori", "author_url": "", "post_date": "2020-02-15T17:28:38.183000", "content": "well done, buddy!
", "votes": 1, "replies": [] }, { "id": 746763, "author_name": "Venkatesh Yadava", "author_url": "", "post_date": "2020-02-15T14:06:16.973000", "content": "Congratulations!!!
", "votes": 1, "replies": [] }, { "id": 746759, "author_name": "Ismail Adamu", "author_url": "", "post_date": "2020-02-15T14:03:18.490000", "content": "congratulations
", "votes": 1, "replies": [] }, { "id": 746536, "author_name": "Chandra Shekhar", "author_url": "", "post_date": "2020-02-15T06:47:39.983000", "content": "Congrats
", "votes": 1, "replies": [] }, { "id": 746483, "author_name": "Anirban Pal", "author_url": "", "post_date": "2020-02-15T04:39:01.663000", "content": "Congrats!
", "votes": 1, "replies": [] }, { "id": 746380, "author_name": "", "author_url": "", "post_date": "2020-02-14T23:10:38.300000", "content": "Congratulation!!
", "votes": 1, "replies": [] }, { "id": 745873, "author_name": "Abhishek Anand", "author_url": "", "post_date": "2020-02-14T09:59:44.510000", "content": "Congratulations and Many Thanks for sharing these
", "votes": 1, "replies": [] }, { "id": 745667, "author_name": "Vishal Deo Prasad", "author_url": "", "post_date": "2020-02-14T04:04:28.593000", "content": "Congratulations!!
", "votes": 1, "replies": [] }, { "id": 745563, "author_name": "kaggleforfun", "author_url": "", "post_date": "2020-02-13T23:46:38.667000", "content": "congrats!
", "votes": 1, "replies": [] }, { "id": 745560, "author_name": "Joe Eskimo", "author_url": "", "post_date": "2020-02-13T23:36:33.510000", "content": "Nice, congrats!
", "votes": 1, "replies": [] }, { "id": 745546, "author_name": "", "author_url": "", "post_date": "2020-02-13T23:00:11.247000", "content": "", "votes": 1, "replies": [] }, { "id": 745400, "author_name": "", "author_url": "", "post_date": "2020-02-13T19:30:58.157000", "content": "", "votes": 1, "replies": [] }, { "id": 745218, "author_name": "", "author_url": "", "post_date": "2020-02-13T16:10:59.997000", "content": "", "votes": 1, "replies": [] }, { "id": 745111, "author_name": "", "author_url": "", "post_date": "2020-02-13T14:04:24.913000", "content": "", "votes": 1, "replies": [] }, { "id": 745092, "author_name": "", "author_url": "", "post_date": "2020-02-13T13:23:16.597000", "content": "", "votes": 1, "replies": [] }, { "id": 744978, "author_name": "", "author_url": "", "post_date": "2020-02-13T10:53:42.953000", "content": "", "votes": 1, "replies": [] }, { "id": 744845, "author_name": "", "author_url": "", "post_date": "2020-02-13T07:39:49.053000", "content": "", "votes": 1, "replies": [] }, { "id": 744731, "author_name": "", "author_url": "", "post_date": "2020-02-13T04:45:03.787000", "content": "", "votes": 1, "replies": [] }, { "id": 744707, "author_name": "", "author_url": "", "post_date": "2020-02-13T03:53:19.173000", "content": "", "votes": 1, "replies": [] }, { "id": 744621, "author_name": "", "author_url": "", "post_date": "2020-02-13T01:47:58.560000", "content": "", "votes": 1, "replies": [] }, { "id": 744527, "author_name": "", "author_url": "", "post_date": "2020-02-12T22:54:03.903000", "content": "", "votes": 1, "replies": [] }, { "id": 744454, "author_name": "", "author_url": "", "post_date": "2020-02-12T21:03:28.747000", "content": "", "votes": 1, "replies": [] }, { "id": 744441, "author_name": "", "author_url": "", "post_date": "2020-02-12T20:42:51.693000", "content": "", "votes": 1, "replies": [] }, { "id": 744395, "author_name": "", "author_url": "", "post_date": "2020-02-12T19:41:03.260000", "content": "", "votes": 1, "replies": [] }, { "id": 744145, "author_name": "", "author_url": "", "post_date": "2020-02-12T16:00:43.227000", "content": "", "votes": 1, "replies": [] }, { "id": 744144, "author_name": "", "author_url": "", "post_date": "2020-02-12T15:53:16.377000", "content": "", "votes": 1, "replies": [] }, { "id": 742746, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:19:45.057000", "content": "", "votes": 1, "replies": [ { "id": 742759, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:27:49.267000", "content": "", "votes": 1, "replies": [] }, { "id": 742769, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:32:07.397000", "content": "", "votes": 0, "replies": [] }, { "id": 742773, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:33:58.603000", "content": "", "votes": 2, "replies": [] }, { "id": 742776, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:36:25.750000", "content": "", "votes": 0, "replies": [] }, { "id": 742784, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:38:14.133000", "content": "", "votes": 0, "replies": [] }, { "id": 742787, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:39:22.620000", "content": "", "votes": 2, "replies": [] }, { "id": 742850, "author_name": "", "author_url": "", "post_date": "2020-02-11T15:39:07.217000", "content": "", "votes": 0, "replies": [] }, { "id": 742907, "author_name": "", "author_url": "", "post_date": "2020-02-11T16:30:47.720000", "content": "", "votes": 4, "replies": [] }, { "id": 742919, "author_name": "", "author_url": "", "post_date": "2020-02-11T16:36:09.123000", "content": "", "votes": 3, "replies": [] }, { "id": 742927, "author_name": "", "author_url": "", "post_date": "2020-02-11T16:38:52.237000", "content": "", "votes": 0, "replies": [] }, { "id": 742940, "author_name": "", "author_url": "", "post_date": "2020-02-11T16:49:48.723000", "content": "", "votes": 1, "replies": [] }, { "id": 742954, "author_name": "", "author_url": "", "post_date": "2020-02-11T17:00:34.960000", "content": "", "votes": 4, "replies": [] }, { "id": 743128, "author_name": "", "author_url": "", "post_date": "2020-02-11T20:12:59.843000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 742434, "author_name": "", "author_url": "", "post_date": "2020-02-11T09:44:56.247000", "content": "", "votes": 1, "replies": [] }, { "id": 742348, "author_name": "", "author_url": "", "post_date": "2020-02-11T08:49:57.410000", "content": "", "votes": 1, "replies": [ { "id": 742374, "author_name": "", "author_url": "", "post_date": "2020-02-11T09:11:06.683000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 742191, "author_name": "", "author_url": "", "post_date": "2020-02-11T06:18:43.140000", "content": "", "votes": 1, "replies": [ { "id": 742379, "author_name": "", "author_url": "", "post_date": "2020-02-11T09:13:40.110000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 741916, "author_name": "", "author_url": "", "post_date": "2020-02-11T02:08:57.793000", "content": "", "votes": 1, "replies": [] }, { "id": 741848, "author_name": "", "author_url": "", "post_date": "2020-02-11T01:02:45.310000", "content": "", "votes": 1, "replies": [] }, { "id": 741841, "author_name": "", "author_url": "", "post_date": "2020-02-11T00:59:48.317000", "content": "", "votes": 1, "replies": [] }, { "id": 747303, "author_name": "", "author_url": "", "post_date": "2020-02-16T08:18:58.250000", "content": "", "votes": 2, "replies": [] }, { "id": 746499, "author_name": "", "author_url": "", "post_date": "2020-02-15T05:24:40", "content": "", "votes": 2, "replies": [] }, { "id": 745043, "author_name": "", "author_url": "", "post_date": "2020-02-13T12:28:17.347000", "content": "", "votes": 2, "replies": [] }, { "id": 743350, "author_name": "", "author_url": "", "post_date": "2020-02-12T02:15:25.557000", "content": "", "votes": 2, "replies": [ { "id": 744487, "author_name": "", "author_url": "", "post_date": "2020-02-12T21:52:54.373000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 742737, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:15:26.483000", "content": "", "votes": 2, "replies": [ { "id": 742747, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:20:45.350000", "content": "", "votes": 0, "replies": [] }, { "id": 742758, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:26:10.093000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 742260, "author_name": "", "author_url": "", "post_date": "2020-02-11T07:33:17.097000", "content": "", "votes": 2, "replies": [] }, { "id": 741898, "author_name": "", "author_url": "", "post_date": "2020-02-11T01:58:57.403000", "content": "", "votes": 2, "replies": [ { "id": 741911, "author_name": "", "author_url": "", "post_date": "2020-02-11T02:04:41.853000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1044979, "author_name": "", "author_url": "", "post_date": "2020-10-10T09:04:28.380000", "content": "", "votes": 0, "replies": [] }, { "id": 921748, "author_name": "", "author_url": "", "post_date": "2020-07-09T14:13:19.197000", "content": "", "votes": 0, "replies": [] }, { "id": 850684, "author_name": "", "author_url": "", "post_date": "2020-05-16T22:51:37.030000", "content": "", "votes": 0, "replies": [] }, { "id": 759595, "author_name": "", "author_url": "", "post_date": "2020-02-29T08:50:22.510000", "content": "", "votes": 0, "replies": [] }, { "id": 756885, "author_name": "", "author_url": "", "post_date": "2020-02-26T07:02:18.267000", "content": "", "votes": 0, "replies": [] }, { "id": 755875, "author_name": "", "author_url": "", "post_date": "2020-02-25T08:11:01.007000", "content": "", "votes": 0, "replies": [] }, { "id": 754983, "author_name": "", "author_url": "", "post_date": "2020-02-24T09:54:18.310000", "content": "", "votes": 0, "replies": [] }, { "id": 754780, "author_name": "", "author_url": "", "post_date": "2020-02-24T03:31:31.277000", "content": "", "votes": 0, "replies": [] }, { "id": 754761, "author_name": "", "author_url": "", "post_date": "2020-02-24T02:57:29.907000", "content": "", "votes": 0, "replies": [] }, { "id": 754179, "author_name": "", "author_url": "", "post_date": "2020-02-23T07:07:54.420000", "content": "", "votes": 0, "replies": [] }, { "id": 753712, "author_name": "", "author_url": "", "post_date": "2020-02-22T15:30:53.933000", "content": "", "votes": 0, "replies": [] }, { "id": 753402, "author_name": "", "author_url": "", "post_date": "2020-02-22T06:09:25.213000", "content": "", "votes": 0, "replies": [] }, { "id": 752543, "author_name": "", "author_url": "", "post_date": "2020-02-21T06:09:57.083000", "content": "", "votes": 0, "replies": [] }, { "id": 752409, "author_name": "", "author_url": "", "post_date": "2020-02-21T02:12:40.863000", "content": "", "votes": 0, "replies": [] }, { "id": 752303, "author_name": "", "author_url": "", "post_date": "2020-02-20T21:18:51.360000", "content": "", "votes": 0, "replies": [] }, { "id": 751925, "author_name": "", "author_url": "", "post_date": "2020-02-20T16:37:53.703000", "content": "", "votes": 0, "replies": [] }, { "id": 751867, "author_name": "", "author_url": "", "post_date": "2020-02-20T15:28:16.340000", "content": "", "votes": 0, "replies": [] }, { "id": 751558, "author_name": "", "author_url": "", "post_date": "2020-02-20T09:21:32.357000", "content": "", "votes": 0, "replies": [] }, { "id": 751134, "author_name": "", "author_url": "", "post_date": "2020-02-20T02:39:03.877000", "content": "", "votes": 0, "replies": [] }, { "id": 751088, "author_name": "", "author_url": "", "post_date": "2020-02-20T01:47:30.603000", "content": "", "votes": 0, "replies": [] }, { "id": 751068, "author_name": "", "author_url": "", "post_date": "2020-02-20T01:19:30.340000", "content": "", "votes": 0, "replies": [] }, { "id": 751012, "author_name": "", "author_url": "", "post_date": "2020-02-19T23:29:09.060000", "content": "", "votes": 0, "replies": [] }, { "id": 750847, "author_name": "", "author_url": "", "post_date": "2020-02-19T18:38:25.273000", "content": "", "votes": 0, "replies": [] }, { "id": 750584, "author_name": "", "author_url": "", "post_date": "2020-02-19T14:32:26.577000", "content": "", "votes": 0, "replies": [] }, { "id": 750466, "author_name": "", "author_url": "", "post_date": "2020-02-19T12:12:13.290000", "content": "", "votes": 0, "replies": [] }, { "id": 750247, "author_name": "", "author_url": "", "post_date": "2020-02-19T08:19:04.350000", "content": "", "votes": 0, "replies": [] }, { "id": 750106, "author_name": "", "author_url": "", "post_date": "2020-02-19T06:25:40.993000", "content": "", "votes": 0, "replies": [] }, { "id": 750098, "author_name": "", "author_url": "", "post_date": "2020-02-19T06:13:59.440000", "content": "", "votes": 0, "replies": [] }, { "id": 750090, "author_name": "", "author_url": "", "post_date": "2020-02-19T05:59:35.723000", "content": "", "votes": 0, "replies": [] }, { "id": 750052, "author_name": "", "author_url": "", "post_date": "2020-02-19T05:03:12.510000", "content": "", "votes": 0, "replies": [] }, { "id": 749799, "author_name": "", "author_url": "", "post_date": "2020-02-18T22:53:41.663000", "content": "", "votes": 0, "replies": [] }, { "id": 749312, "author_name": "", "author_url": "", "post_date": "2020-02-18T15:10:22.640000", "content": "", "votes": 0, "replies": [] }, { "id": 749216, "author_name": "", "author_url": "", "post_date": "2020-02-18T12:57:30.180000", "content": "", "votes": 0, "replies": [] }, { "id": 749076, "author_name": "", "author_url": "", "post_date": "2020-02-18T09:16:33.417000", "content": "", "votes": 0, "replies": [] }, { "id": 749065, "author_name": "", "author_url": "", "post_date": "2020-02-18T08:55:15.303000", "content": "", "votes": 0, "replies": [] }, { "id": 749049, "author_name": "", "author_url": "", "post_date": "2020-02-18T08:25:58.053000", "content": "", "votes": 0, "replies": [] }, { "id": 748793, "author_name": "", "author_url": "", "post_date": "2020-02-18T02:05:13.610000", "content": "", "votes": 0, "replies": [] }, { "id": 748781, "author_name": "", "author_url": "", "post_date": "2020-02-18T01:16:24.960000", "content": "", "votes": 0, "replies": [] }, { "id": 748778, "author_name": "", "author_url": "", "post_date": "2020-02-18T01:04:47.527000", "content": "", "votes": 0, "replies": [] }, { "id": 748749, "author_name": "", "author_url": "", "post_date": "2020-02-17T23:52:31.053000", "content": "", "votes": 0, "replies": [] }, { "id": 748594, "author_name": "", "author_url": "", "post_date": "2020-02-17T17:58:56.477000", "content": "", "votes": 0, "replies": [] }, { "id": 748160, "author_name": "", "author_url": "", "post_date": "2020-02-17T07:52:46.530000", "content": "", "votes": 0, "replies": [] }, { "id": 747731, "author_name": "", "author_url": "", "post_date": "2020-02-16T19:21:25.917000", "content": "", "votes": 0, "replies": [] }, { "id": 743874, "author_name": "", "author_url": "", "post_date": "2020-02-12T10:52:08.870000", "content": "", "votes": 0, "replies": [] }, { "id": 743528, "author_name": "", "author_url": "", "post_date": "2020-02-12T06:06:18.127000", "content": "", "votes": 0, "replies": [] }, { "id": 743526, "author_name": "", "author_url": "", "post_date": "2020-02-12T06:05:15.840000", "content": "", "votes": 0, "replies": [] }, { "id": 743416, "author_name": "", "author_url": "", "post_date": "2020-02-12T03:36:14.510000", "content": "", "votes": 0, "replies": [] }, { "id": 743399, "author_name": "", "author_url": "", "post_date": "2020-02-12T03:08:19.850000", "content": "", "votes": 0, "replies": [] }, { "id": 743261, "author_name": "", "author_url": "", "post_date": "2020-02-12T00:10:05.157000", "content": "", "votes": 0, "replies": [ { "id": 743305, "author_name": "", "author_url": "", "post_date": "2020-02-12T01:13:26.700000", "content": "", "votes": 2, "replies": [] }, { "id": 745098, "author_name": "", "author_url": "", "post_date": "2020-02-13T13:31:03.953000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 742988, "author_name": "", "author_url": "", "post_date": "2020-02-11T17:23:15.737000", "content": "", "votes": 0, "replies": [ { "id": 857496, "author_name": "", "author_url": "", "post_date": "2020-05-22T16:46:24.683000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 742841, "author_name": "", "author_url": "", "post_date": "2020-02-11T15:25:32.397000", "content": "", "votes": 0, "replies": [] }, { "id": 742782, "author_name": "", "author_url": "", "post_date": "2020-02-11T14:37:34.877000", "content": "", "votes": 0, "replies": [ { "id": 742846, "author_name": "", "author_url": "", "post_date": "2020-02-11T15:32:26.917000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 742597, "author_name": "", "author_url": "", "post_date": "2020-02-11T12:03:04.717000", "content": "", "votes": 0, "replies": [] }, { "id": 742585, "author_name": "", "author_url": "", "post_date": "2020-02-11T11:51:45.703000", "content": "", "votes": 0, "replies": [] }, { "id": 742540, "author_name": "", "author_url": "", "post_date": "2020-02-11T11:14:00.290000", "content": "", "votes": 0, "replies": [] }, { "id": 742442, "author_name": "", "author_url": "", "post_date": "2020-02-11T09:49:46.633000", "content": "", "votes": 0, "replies": [] }, { "id": 742352, "author_name": "", "author_url": "", "post_date": "2020-02-11T08:53:15.997000", "content": "", "votes": 0, "replies": [] }, { "id": 742323, "author_name": "", "author_url": "", "post_date": "2020-02-11T08:28:10", "content": "", "votes": 0, "replies": [] }, { "id": 742216, "author_name": "", "author_url": "", "post_date": "2020-02-11T06:52:08.007000", "content": "", "votes": 0, "replies": [] }, { "id": 741932, "author_name": "", "author_url": "", "post_date": "2020-02-11T02:22:01.063000", "content": "", "votes": 0, "replies": [] }, { "id": 741915, "author_name": "", "author_url": "", "post_date": "2020-02-11T02:07:12.003000", "content": "", "votes": 0, "replies": [] }, { "id": 741879, "author_name": "", "author_url": "", "post_date": "2020-02-11T01:39:10.970000", "content": "", "votes": 0, "replies": [] }, { "id": 741839, "author_name": "", "author_url": "", "post_date": "2020-02-11T00:58:04.070000", "content": "", "votes": 0, "replies": [] }, { "id": 1821936, "author_name": "", "author_url": "", "post_date": "2022-06-15T23:59:20.637000", "content": "", "votes": 0, "replies": [] }, { "id": 795813, "author_name": "", "author_url": "", "post_date": "2020-04-03T04:26:12.490000", "content": "", "votes": 0, "replies": [] }, { "id": 753987, "author_name": "", "author_url": "", "post_date": "2020-02-22T22:35:13.550000", "content": "", "votes": 2, "replies": [] }, { "id": 752614, "author_name": "", "author_url": "", "post_date": "2020-02-21T08:08:54.980000", "content": "", "votes": 0, "replies": [] }, { "id": 749190, "author_name": "", "author_url": "", "post_date": "2020-02-18T12:18:00.410000", "content": "", "votes": 0, "replies": [] }, { "id": 749001, "author_name": "", "author_url": "", "post_date": "2020-02-18T06:49:20.197000", "content": "", "votes": 0, "replies": [] }, { "id": 748838, "author_name": "", "author_url": "", "post_date": "2020-02-18T03:23:44.203000", "content": "", "votes": 0, "replies": [] }, { "id": 748586, "author_name": "", "author_url": "", "post_date": "2020-02-17T17:42:50.157000", "content": "", "votes": 0, "replies": [] }, { "id": 748565, "author_name": "", "author_url": "", "post_date": "2020-02-17T17:08:37.310000", "content": "", "votes": 0, "replies": [] }, { "id": 747681, "author_name": "", "author_url": "", "post_date": "2020-02-16T18:08:28.983000", "content": "", "votes": 1, "replies": [] }, { "id": 746487, "author_name": "", "author_url": "", "post_date": "2020-02-15T04:53:28.067000", "content": "", "votes": 1, "replies": [] }, { "id": 744697, "author_name": "", "author_url": "", "post_date": "2020-02-13T03:31:44.030000", "content": "", "votes": 1, "replies": [] }, { "id": 1019145, "author_name": "", "author_url": "", "post_date": "2020-09-20T08:14:06.167000", "content": "", "votes": 2, "replies": [] }, { "id": 747593, "author_name": "", "author_url": "", "post_date": "2020-02-16T16:01:45.523000", "content": "", "votes": 1, "replies": [] }, { "id": 746545, "author_name": "", "author_url": "", "post_date": "2020-02-15T07:20:10.180000", "content": "", "votes": 1, "replies": [] }, { "id": 746239, "author_name": "", "author_url": "", "post_date": "2020-02-14T19:20:15.930000", "content": "", "votes": 1, "replies": [] }, { "id": 753602, "author_name": "", "author_url": "", "post_date": "2020-02-22T12:55:02.197000", "content": "", "votes": 0, "replies": [] }, { "id": 753435, "author_name": "", "author_url": "", "post_date": "2020-02-22T07:16:47.843000", "content": "", "votes": 0, "replies": [] }, { "id": 752736, "author_name": "", "author_url": "", "post_date": "2020-02-21T10:56:04.487000", "content": "", "votes": 0, "replies": [] }, { "id": 751772, "author_name": "", "author_url": "", "post_date": "2020-02-20T13:53:52.493000", "content": "", "votes": 0, "replies": [] }, { "id": 751110, "author_name": "", "author_url": "", "post_date": "2020-02-20T02:09:33.740000", "content": "", "votes": 0, "replies": [] }, { "id": 750620, "author_name": "", "author_url": "", "post_date": "2020-02-19T14:48:14.907000", "content": "", "votes": 0, "replies": [] }, { "id": 749388, "author_name": "", "author_url": "", "post_date": "2020-02-18T16:47:22.323000", "content": "", "votes": 0, "replies": [] }, { "id": 749141, "author_name": "", "author_url": "", "post_date": "2020-02-18T10:56:37.933000", "content": "", "votes": 0, "replies": [] }, { "id": 748975, "author_name": "", "author_url": "", "post_date": "2020-02-18T06:00:29.397000", "content": "", "votes": 0, "replies": [] }, { "id": 748252, "author_name": "", "author_url": "", "post_date": "2020-02-17T10:04:14.890000", "content": "", "votes": 0, "replies": [] }, { "id": 743820, "author_name": "", "author_url": "", "post_date": "2020-02-12T10:24:03.177000", "content": "", "votes": 0, "replies": [] }, { "id": 743738, "author_name": "", "author_url": "", "post_date": "2020-02-12T08:48:29.183000", "content": "", "votes": 0, "replies": [] }, { "id": 741875, "author_name": "", "author_url": "", "post_date": "2020-02-11T01:30:27.057000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 186, "messages": [ { "id": 741834, "postDate": "2020-02-11T00:55:46.693Z", "content": "Congratulations to all teams and especially to daring solo players! Thanks to organizers as well!
\n\nTremendous feeling for the whole team. I happened to finish one place away from the golden zone in 2 NLP competitions, and there it is. A few posts below I wrote down \"respect your efforts, not results\" and you'll be there. There is always a room for such motivational quotes. I don't know whether it's harder to compete with yourself or with such great minds. Thanks to the team ( @ddanevskiy , @kashnitsky , @dmitriyab ) for this thrilling race. And thanks to the whole community for ideas, criticism, and a sense of humor.
\n\nEDIT
\n\nThe community seems to be waiting for a killer feature from us. I don’t reckon we’ve got a single one. Our solution is comprised of three cornerstones:\n- pretrained language models\n- pseudo-labeling\n- post-processing (yes we did one)
\n\nThe basic pipeline
\n\nLess than 3 weeks ago just when we were done with the TF Q&A competition, we jumped into this one. Oh yes, there were already so many great NLP folks seen on the leaderboard. We started from a public pytorch baseline [(0.377 I guess)] (https://www.kaggle.com/phoenix9032/pytorch-bert-plain) (kudos @phoenix9032) and managed to update it effectively (0.396 for BERT-base and 0.402 for BERT-large) with a couple of tricks:\n- GroupKFold with question_title groups\n- Multi-Sample Dropout for Accelerated Training and Better Generalization, paper\n- Different learning rate settings for encoder and head (deserves a paper @ddanevskyi)\n- Using CLS outputs from all BERT layers rather than using only the last one. We did this by using a weighted sum of these outputs, where the weights were learnable and constrained to be positive and sum to 1
\n\nWe experimented with separate weights for different targets and with various loss functions but eventually ended up sticking to simple BCE loss.
\n\nPretrained language models
\n\nWhen @dmitriyab joined our team, he introduced stackexchange (SE) pretrained LMs with 110k-long dictionary including code and other domain tokens. In addition to the common MLM task, the model also predicted 6 auxiliary targets (question_score, question_view_count, question_favorite_count, answer_score, answers_count, is_answer_accepted) with a linear layer on top of pooled LM output. That looked beneficial enough for me to drop my code replacement experiments and update the existing bert pipeline with a huge pretrained embedding layer -> 0.414. In the end @dmitriyab and @kashnitsky trained a few LMs (bert-large, roberta, roberta-large) on 7M SE data using whole-word-masking, but the initial model was still the best.
Pseudo-labeling
\n\nPseudo-labeling is a well known semi-supervised learning technique within the Kaggle community and sometimes I wonder how they reinvent stuff in academic papers like this one (btw a great result). So we labeled additional 100k samples using our three basic bert models. First experiments with pseudo-labels showed huge improvement for SE pretrained bert (0.414 -> 0.445). That was huge. But the leaderboard did not agree.
\n\nPreviously, my teammates and I used models from all folds to create pseudolabels. This strategy worked well in many cases including recent Recursion competition, where my team finished 6th. @ddanevskyi used the same approach for his 3rd place on Freesound. This way of creating pseudolabels gives quite accurate predictions as you are using a fold ensemble, but has an unobvious pitfall. If the pseudolabeled set contains a lot of similar samples to the training set, you could expect that some actual training labels could leak through those samples, even if there is no “direct” target leakage.
\n\nImagine that you’ve trained a 5-fold model on the training set then used those 5 folds to create pseudolabels on some external dataset. Then you add those pseudolabels to the training part of each of 5 folds. Very likely you will see an overoptimistic score on the validation part as 4 out of 5 models used to generate pseudolabels have “seen” the current validation set through their training process.
\n\nWe could expect a lot of very similar questions/answers in SO&SE data, so we clearly couldn’t rely on that sort of pseudolabels. In order to fix the problem, we generated 5 different sets of pseudolabels where for each train/val split we used only those models that were trained using only the current train set. This gave much less accurate pseudolabels, but eliminated any possible source of target leakage.
\n\nAs expected, it didn’t bring such an improvement compared to a “leaked” approach (0.414 -> 0.422) but that was a wise decision in terms of tackling overfitting.
\n\nBART
\n\nOne week prior to the deadline, we noticed @christofhenkel ’s post in the external data thread - the one about fairseq BART . Autoregressive sequence-to-sequence model for text classification? Looked strange to me but at the same time that was something new and daring, and we wouldn’t ignore a hint by such a prominent kaggler. It took us three whole days to train BART on 1080ti with batch size 2. The model scored 0.427 locally and it’s predictions decorrelated well with those of BERT.
\n\nWe also trained a few other bert-base, bert-large, roberta-base, and roberta-large models. They all scored around 0.420 locally but didn’t improve drastically our bert+bart baseline.
\n\nPost-processing
\n\nAnd finally, post-processing. Yes we did one. Based on this discussion we found that discretization of predictions for some challenging targets led to better spearman corr. So we used no thresholds but discretized such columns based on their distribution in the training set. That simple heuristics boosted score by 0.027-0.030 for almost every single model (up to 0.448 bert-base, 0.454 bart).
\n\nFinish
\n\nOur local validation and public score stayed the same for the whole last week - 0.468. Yesterday, since early morning with all this berts, barts, robertas we were not able to reproduce public score :harold:, having only 0.462 submissions but they seem to work the best -> 0.430-0.433 private leaderboard (unfortunately we didn’t select the best one). But that was good enough.
\n\nWhat didn’t happen to work well
\n\nWonderful teamwork and team patience (not really haha). Thanks to you all. We are making our submission kernels and datasets public. By the end of the day I’m releasing our code on GitHub.
\n\nEDIT\nCode released here. There are 3 separate branches. I'll update with a nice readme soon.
\n\nOleg Yaroshevskiy, Dmitriy Danevskiy, Yury Kashnitsky, Dmitriy Abulkhanov
", "rawMarkdown": "Congratulations to all teams and especially to daring solo players! Thanks to organizers as well!\n\nTremendous feeling for the whole team. I happened to finish one place away from the golden zone in 2 NLP competitions, and there it is. A few posts below I wrote down \"respect your efforts, not results\" and you'll be there. There is always a room for such motivational quotes. I don't know whether it's harder to compete with yourself or with such great minds. Thanks to the team ( @ddanevskiy , @kashnitsky , @dmitriyab ) for this thrilling race. And thanks to the whole community for ideas, criticism, and a sense of humor. \n\n\n**EDIT**\n\nThe community seems to be waiting for a killer feature from us. I don’t reckon we’ve got a single one. Our solution is comprised of three cornerstones:\n- pretrained language models\n- pseudo-labeling\n- post-processing (yes we did one)\n\n\n**The basic pipeline**\n\nLess than 3 weeks ago just when we were done with the TF Q&A competition, we jumped into this one. Oh yes, there were already so many great NLP folks seen on the leaderboard. We started from a public pytorch baseline [(0.377 I guess)] (https://www.kaggle.com/phoenix9032/pytorch-bert-plain) (kudos @phoenix9032) and managed to update it effectively (0.396 for BERT-base and 0.402 for BERT-large) with a couple of tricks:\n- GroupKFold with question_title groups\n- Multi-Sample Dropout for Accelerated Training and Better Generalization, [paper](https://arxiv.org/pdf/1905.09788.pdf)\n- Different learning rate settings for encoder and head (deserves a paper @ddanevskyi)\n- Using CLS outputs from all BERT layers rather than using only the last one. We did this by using a weighted sum of these outputs, where the weights were learnable and constrained to be positive and sum to 1\n\nWe experimented with separate weights for different targets and with various loss functions but eventually ended up sticking to simple BCE loss. \n\n\n**Pretrained language models**\n\nWhen @dmitriyab joined our team, he introduced stackexchange (SE) pretrained LMs with 110k-long dictionary including code and other domain tokens. In addition to the common MLM task, the model also predicted 6 auxiliary targets (`question_score, question_view_count, question_favorite_count, answer_score, answers_count, is_answer_accepted`) with a linear layer on top of pooled LM output. That looked beneficial enough for me to drop my code replacement experiments and update the existing bert pipeline with a huge pretrained embedding layer -> 0.414. In the end @dmitriyab and @kashnitsky trained a few LMs (bert-large, roberta, roberta-large) on 7M SE data using whole-word-masking, but the initial model was still the best. \n\n\n**Pseudo-labeling**\n\nPseudo-labeling is a well known semi-supervised learning technique within the Kaggle community and sometimes I wonder how they reinvent stuff in academic papers like [this one](https://arxiv.org/pdf/1911.04252.pdf) (btw a great result). So we labeled additional 100k samples using our three basic bert models. First experiments with pseudo-labels showed huge improvement for SE pretrained bert (0.414 -> 0.445). That was huge. But the leaderboard did not agree.\n \nPreviously, my teammates and I used models from all folds to create pseudolabels. This strategy worked well in many cases including recent Recursion competition, where my team finished 6th. @ddanevskyi used the same approach for his 3rd place on Freesound. This way of creating pseudolabels gives quite accurate predictions as you are using a fold ensemble, but has an unobvious pitfall. If the pseudolabeled set contains a lot of similar samples to the training set, you could expect that some actual training labels could leak through those samples, even if there is no “direct” target leakage. \n\nImagine that you’ve trained a 5-fold model on the training set then used those 5 folds to create pseudolabels on some external dataset. Then you add those pseudolabels to the training part of each of 5 folds. Very likely you will see an overoptimistic score on the validation part as 4 out of 5 models used to generate pseudolabels have “seen” the current validation set through their training process.\n\nWe could expect a lot of very similar questions/answers in SO&SE data, so we clearly couldn’t rely on that sort of pseudolabels. In order to fix the problem, we generated 5 different sets of pseudolabels where for each train/val split we used only those models that were trained using only the current train set. This gave much less accurate pseudolabels, but eliminated any possible source of target leakage.\n\nAs expected, it didn’t bring such an improvement compared to a “leaked” approach (0.414 -> 0.422) but that was a wise decision in terms of tackling overfitting. \n\n\n**BART**\n\nOne week prior to the deadline, we noticed @christofhenkel ’s post in the external data thread - the one about [fairseq BART](https://github.com/pytorch/fairseq/tree/master/examples/bart) . Autoregressive sequence-to-sequence model for text classification? Looked strange to me but at the same time that was something new and daring, and we wouldn’t ignore a hint by such a prominent kaggler. It took us three whole days to train BART on 1080ti with batch size 2. The model scored 0.427 locally and it’s predictions decorrelated well with those of BERT. \n\nWe also trained a few other bert-base, bert-large, roberta-base, and roberta-large models. They all scored around 0.420 locally but didn’t improve drastically our bert+bart baseline. \n\n\n**Post-processing**\n\nAnd finally, post-processing. Yes we did one. Based on [this discussion](https://www.kaggle.com/c/google-quest-challenge/discussion/118724) we found that discretization of predictions for some challenging targets led to better spearman corr. So we used no thresholds but discretized such columns based on their distribution in the training set. That simple heuristics boosted score by 0.027-0.030 for almost every single model (up to 0.448 bert-base, 0.454 bart). \n\n\n**Finish**\n\nOur local validation and public score stayed the same for the whole last week - 0.468. Yesterday, since early morning with all this berts, barts, robertas we were not able to reproduce public score :harold:, having only 0.462 submissions but they seem to work the best -> 0.430-0.433 private leaderboard (unfortunately we didn’t select the best one). But that was good enough. \n\n\n**What didn’t happen to work well**\n\n- StackExchange meta-features used to boost our score in the very beginning but in the end they made no significant difference\n- GPT, GPT2-medium, GPT2-large showed mediocre scores\n- backtranslation\n- stacking. Instead, linear blends worked unreasonably well\n\nWonderful teamwork and team patience (not really haha). Thanks to you all. We are making our submission kernels and datasets public. By the end of the day I’m releasing our code on GitHub. \n\n**EDIT**\nCode released [here](https://github.com/oleg-yaroshevskiy/quest_qa_labeling). There are 3 separate branches. I'll update with a nice readme soon. \n\n*Oleg Yaroshevskiy, Dmitriy Danevskiy, Yury Kashnitsky, Dmitriy Abulkhanov*", "votes": 328 }, { "id": 742757, "postDate": "2020-02-11T14:25:47.420Z", "content": "I would also like to share two useful things.
\n\nThe first one is quite simple. At some point, I decided to figure out how reliable the public leaderboard could be. To do so, I created 150 random train/test splits of the OOF scores such that the test part of each split contained around 450 samples. Then, I computed the target metric on each of those \"test\" datasets and computed the mean and standard deviation of these scores. The standard deviation appeared to be around 0.02 which made us think that we could trust the public score only up to +- 0.02. When I used 8x bigger test size (which is roughly the size of the private test set), the standard deviation lowered to around 0.003.
\n\nHere you can see a distribution of the scores for 150 splits:
\n\n
The second thing is about proper tooling. About 2 years ago I spent a couple of evenings to create a very tiny python library called mag that addressed the problem of machine learning experiment management. Since then, I used it in every single competition including this one. This really helped our team to don't lose track of all the models we trained for these two weeks. I hope some of you might find it useful too.
", "rawMarkdown": "I would also like to share two useful things. \n\nThe first one is quite simple. At some point, I decided to figure out how reliable the public leaderboard could be. To do so, I created 150 random train/test splits of the OOF scores such that the test part of each split contained around 450 samples. Then, I computed the target metric on each of those \"test\" datasets and computed the mean and standard deviation of these scores. The standard deviation appeared to be around 0.02 which made us think that we could trust the public score only up to +- 0.02. When I used 8x bigger test size (which is roughly the size of the private test set), the standard deviation lowered to around 0.003. \n\nHere you can see a distribution of the scores for 150 splits:\n\n\n\nThe second thing is about proper tooling. About 2 years ago I spent a couple of evenings to create a very tiny python library called [mag](https://github.com/ex4sperans/mag) that addressed the problem of machine learning experiment management. Since then, I used it in every single competition including this one. This really helped our team to don't lose track of all the models we trained for these two weeks. I hope some of you might find it useful too.", "votes": 24, "replies": [ { "id": 742764, "postDate": "2020-02-11T14:29:28.017Z", "content": "I confirm this - using mag turned out to be very straightforward. Although I'm typically frustrated with trying to use \"yet another bicycle\" (like poutyne used by my teammate monty to manage LM training).
", "rawMarkdown": "I confirm this - using mag turned out to be very straightforward. Although I'm typically frustrated with trying to use \"yet another bicycle\" (like poutyne used by my teammate monty to manage LM training). ", "votes": 3 }, { "id": 742768, "postDate": "2020-02-11T14:31:59.490Z", "content": "I totally agree with this \"yet another bicycle\" concern. That's why I came up with my own ;)
", "rawMarkdown": "I totally agree with this \"yet another bicycle\" concern. That's why I came up with my own ;)", "votes": 2 }, { "id": 742774, "postDate": "2020-02-11T14:34:03.937Z", "content": ":slow: \n
",
"votes": 6
},
{
"id": 743226,
"postDate": "2020-02-11T22:34:03.387Z",
"content": "What are the other bicycles? Any hint for the term to use on Google to search for that? Maybe \"machine learning experiment management\"?
", "rawMarkdown": "What are the other bicycles? Any hint for the term to use on Google to search for that? Maybe \"machine learning experiment management\"?" }, { "id": 743291, "postDate": "2020-02-12T00:57:06.150Z", "content": "MLFlow is the most prominent one.
", "rawMarkdown": "MLFlow is the most prominent one. " }, { "id": 743963, "postDate": "2020-02-12T12:47:38.990Z", "content": "mag is good, I would try, thanks
", "rawMarkdown": "mag is good, I would try, thanks" }, { "id": 755943, "postDate": "2020-02-25T09:41:29.130Z", "content": "@ddanevskyi Thank you for sharing!\nI need to ask a dumb question. You said you created 150 random train/test splits of the OOF scores. Does that mean you trained these data 150 times?\nThank you!!
", "rawMarkdown": "@ddanevskyi Thank you for sharing!\nI need to ask a dumb question. You said you created 150 random train/test splits of the OOF scores. Does that mean you trained these data 150 times?\nThank you!!" }, { "id": 756192, "postDate": "2020-02-25T13:53:47.483Z", "content": "Not really, I just use the same OOF predictions
", "rawMarkdown": "Not really, I just use the same OOF predictions", "votes": 1 } ] }, { "id": 742515, "postDate": "2020-02-11T10:35:14.300Z", "content": "On behalf of our team, I have to apologize we didn't post the solution yet. The reason is we slept for 2-3 hours at the very most and now we need to get some rest. You might expect the solution to be posted in several hours. The code will be posted a bit later as we need to add some READMEs and do a bit of refactoring.
", "rawMarkdown": "On behalf of our team, I have to apologize we didn't post the solution yet. The reason is we slept for 2-3 hours at the very most and now we need to get some rest. You might expect the solution to be posted in several hours. The code will be posted a bit later as we need to add some READMEs and do a bit of refactoring.", "votes": 8, "replies": [ { "id": 742516, "postDate": "2020-02-11T10:37:02.280Z", "content": "We can wait patiently for that minimal post-processing sol :)
", "rawMarkdown": "We can wait patiently for that minimal post-processing sol :)" } ] }, { "id": 753654, "postDate": "2020-02-22T14:05:31.183Z", "content": "Congratulations @yaroshevskiy and thanks for sharing
", "rawMarkdown": "Congratulations @yaroshevskiy and thanks for sharing", "votes": 6 }, { "id": 743732, "postDate": "2020-02-12T08:41:36.200Z", "content": "Congarts! I can say this is the best solution and it deserves 1st place\nRegarding the pseudo-labeling, We used a similar idea and it turns out that using 5 predictions (from 5 fold) separately wont leak the target because when we averaged the predictions of 5 models we were getting 0.53+ CV with the distilled data.\nUsing this technique correctly introduced improvement for CV in training folds (+0.005 per fold) for each fold without postprocessing but less score on oof which was so weird (got same score on LB) and we decided to leave the idea.\nMy question is: did you post-process the distilled labels? or you trained with it as it was given by the teacher model?
", "rawMarkdown": "Congarts! I can say this is the best solution and it deserves 1st place\nRegarding the pseudo-labeling, We used a similar idea and it turns out that using 5 predictions (from 5 fold) separately wont leak the target because when we averaged the predictions of 5 models we were getting 0.53+ CV with the distilled data.\nUsing this technique correctly introduced improvement for CV in training folds (+0.005 per fold) for each fold without postprocessing but less score on oof which was so weird (got same score on LB) and we decided to leave the idea.\nMy question is: did you post-process the distilled labels? or you trained with it as it was given by the teacher model?", "votes": 6, "replies": [ { "id": 744163, "postDate": "2020-02-12T16:14:16.287Z", "content": "Oh that's a nice question. We decided that post-processing leads only to better spearman corr but not to better classification distribution so we left them untouched.
", "rawMarkdown": "Oh that's a nice question. We decided that post-processing leads only to better spearman corr but not to better classification distribution so we left them untouched. ", "votes": 2 }, { "id": 744176, "postDate": "2020-02-12T16:33:36.880Z", "content": "same for us. When we post-process the distilled data spearman drop without PP (from 0.41x to 0.38x) but increase when we apply PP (from 0.45 to 0.48). We couldn't trust this since the model makes predictions so weird and we were worried that model is overfiting on the validation (PP improves with 0.1+ lol).\nI am so glad that this worked for you. Chapeau bas!
", "rawMarkdown": "same for us. When we post-process the distilled data spearman drop without PP (from 0.41x to 0.38x) but increase when we apply PP (from 0.45 to 0.48). We couldn't trust this since the model makes predictions so weird and we were worried that model is overfiting on the validation (PP improves with 0.1+ lol).\nI am so glad that this worked for you. Chapeau bas!", "votes": 2 } ] }, { "id": 747982, "postDate": "2020-02-17T04:19:26.070Z", "content": "Congratulations !!!! and thanks for sharing.
", "rawMarkdown": "Congratulations !!!! and thanks for sharing.", "votes": 3 }, { "id": 744334, "postDate": "2020-02-12T18:44:49.670Z", "content": "Congratulations and many thanks for you and your team for the insights!
\n\nВітаємо та дякуємо вашій команді за ідеї
\n\nПоздравления и большое спасибо вам и вашей команде за идеи
", "rawMarkdown": "Congratulations and many thanks for you and your team for the insights!\n\nВітаємо та дякуємо вашій команді за ідеї\n\nПоздравления и большое спасибо вам и вашей команде за идеи \n\n", "votes": 3 }, { "id": 742742, "postDate": "2020-02-11T14:17:41.460Z", "content": "Congrats on the result. It would be interesting to see how far we could go by combining our methods.
", "rawMarkdown": "Congrats on the result. It would be interesting to see how far we could go by combining our methods. ", "votes": 3 }, { "id": 742268, "postDate": "2020-02-11T07:42:07.673Z", "content": "Wow, congrats @yaroshevskiy! I'm so glad you and your team won this competition! It made my day)
", "rawMarkdown": "Wow, congrats @yaroshevskiy! I'm so glad you and your team won this competition! It made my day)", "votes": 3 }, { "id": 744173, "postDate": "2020-02-12T16:32:53.603Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 4 }, { "id": 742720, "postDate": "2020-02-11T14:02:29.153Z", "content": "Updated with the solution. Thanks for your patience!\nSolution kernel. GitHub code is coming!
", "rawMarkdown": "Updated with the solution. Thanks for your patience!\n[Solution kernel](https://www.kaggle.com/ddanevskyi/1st-place-solution). GitHub code is coming!", "votes": 4, "replies": [ { "id": 742723, "postDate": "2020-02-11T14:04:57.050Z", "content": "Wow all !python flags Really cool And Clean solution ! And A great write-up!
Congratulations !!!
", "rawMarkdown": "Congratulations !!!", "votes": 1 }, { "id": 747914, "postDate": "2020-02-17T02:23:37.457Z", "content": "Powerful stuff. Congratulations 👍
", "rawMarkdown": "Powerful stuff. Congratulations 👍 ", "votes": 1 }, { "id": 747894, "postDate": "2020-02-17T01:27:54.990Z", "content": "Congratulations !!
", "rawMarkdown": "Congratulations !!", "votes": 1 }, { "id": 747798, "postDate": "2020-02-16T21:41:34.363Z", "content": "Congratulations! awesome work!
", "rawMarkdown": "Congratulations! awesome work!", "votes": 1 }, { "id": 747784, "postDate": "2020-02-16T21:01:14.313Z", "content": "Good topic!
", "rawMarkdown": "Good topic!", "votes": 1 }, { "id": 747738, "postDate": "2020-02-16T19:30:56.120Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 747203, "postDate": "2020-02-16T05:30:08.140Z", "content": "Congratulations and thank you for your insight!
", "rawMarkdown": "Congratulations and thank you for your insight!", "votes": 1 }, { "id": 747141, "postDate": "2020-02-16T02:38:09.420Z", "content": "Great
", "rawMarkdown": "Great ", "votes": 1 }, { "id": 746969, "postDate": "2020-02-15T19:45:46.960Z", "content": "Congratulations
", "rawMarkdown": "Congratulations\n\n", "votes": 1 }, { "id": 746884, "postDate": "2020-02-15T17:28:38.183Z", "content": "well done, buddy!
", "rawMarkdown": "well done, buddy!", "votes": 1 }, { "id": 746763, "postDate": "2020-02-15T14:06:16.973Z", "content": "Congratulations!!!
", "rawMarkdown": "Congratulations!!!", "votes": 1 }, { "id": 746759, "postDate": "2020-02-15T14:03:18.490Z", "content": "congratulations
", "rawMarkdown": "congratulations\n", "votes": 1 }, { "id": 746536, "postDate": "2020-02-15T06:47:39.983Z", "content": "Congrats
", "rawMarkdown": "Congrats", "votes": 1 }, { "id": 746483, "postDate": "2020-02-15T04:39:01.663Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 1 }, { "id": 746380, "postDate": "2020-02-14T23:10:38.300Z", "content": "Congratulation!!
", "rawMarkdown": "Congratulation!!", "votes": 1 }, { "id": 745873, "postDate": "2020-02-14T09:59:44.510Z", "content": "Congratulations and Many Thanks for sharing these
", "rawMarkdown": "Congratulations and Many Thanks for sharing these", "votes": 1 }, { "id": 745667, "postDate": "2020-02-14T04:04:28.593Z", "content": "Congratulations!!
", "rawMarkdown": "Congratulations!!", "votes": 1 }, { "id": 745563, "postDate": "2020-02-13T23:46:38.667Z", "content": "congrats!
", "rawMarkdown": "congrats!", "votes": 1 }, { "id": 745560, "postDate": "2020-02-13T23:36:33.510Z", "content": "Nice, congrats!
", "rawMarkdown": "Nice, congrats!", "votes": 1 }, { "id": 745546, "postDate": "2020-02-13T23:00:11.247Z", "content": "Congrats!!!
", "rawMarkdown": "Congrats!!!", "votes": 1 }, { "id": 745400, "postDate": "2020-02-13T19:30:58.157Z", "content": "Congrats!!
", "rawMarkdown": "Congrats!!", "votes": 1 }, { "id": 745218, "postDate": "2020-02-13T16:10:59.997Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 1 }, { "id": 745111, "postDate": "2020-02-13T14:04:24.913Z", "content": "Wow! Congrats!
", "rawMarkdown": "Wow! Congrats!", "votes": 1 }, { "id": 745092, "postDate": "2020-02-13T13:23:16.597Z", "content": "Congratulations !
", "rawMarkdown": "Congratulations !", "votes": 1 }, { "id": 744978, "postDate": "2020-02-13T10:53:42.953Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 1 }, { "id": 744845, "postDate": "2020-02-13T07:39:49.053Z", "content": "Congratulations on the win!
", "rawMarkdown": "Congratulations on the win!", "votes": 1 }, { "id": 744731, "postDate": "2020-02-13T04:45:03.787Z", "content": "Congrats.. superb work!
", "rawMarkdown": "Congrats.. superb work!", "votes": 1 }, { "id": 744707, "postDate": "2020-02-13T03:53:19.173Z", "content": "COngratulations !!
", "rawMarkdown": "COngratulations !!", "votes": 1 }, { "id": 744621, "postDate": "2020-02-13T01:47:58.560Z", "content": "Congratulations to the team! The ideas are great, it will help a lot ...
", "rawMarkdown": "Congratulations to the team! The ideas are great, it will help a lot ...", "votes": 1 }, { "id": 744527, "postDate": "2020-02-12T22:54:03.903Z", "content": "Congratulations! I'm glad to know that postprocessing trick worked well 😄
", "rawMarkdown": "Congratulations! I'm glad to know that postprocessing trick worked well 😄 ", "votes": 1 }, { "id": 744454, "postDate": "2020-02-12T21:03:28.747Z", "content": "Interesting approach, congrats!
", "rawMarkdown": "Interesting approach, congrats!", "votes": 1 }, { "id": 744441, "postDate": "2020-02-12T20:42:51.693Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 1 }, { "id": 744395, "postDate": "2020-02-12T19:41:03.260Z", "content": "congrats
", "rawMarkdown": "congrats", "votes": 1 }, { "id": 744145, "postDate": "2020-02-12T16:00:43.227Z", "content": "Cool! Красава 👍
", "rawMarkdown": "Cool! Красава 👍 ", "votes": 1 }, { "id": 744144, "postDate": "2020-02-12T15:53:16.377Z", "content": "Thanks for sharing and congratulations to the team!
", "rawMarkdown": "Thanks for sharing and congratulations to the team!", "votes": 1 }, { "id": 742746, "postDate": "2020-02-11T14:19:45.057Z", "content": "Congrats, great job! I just want to add that thresholding and fitting to distribution is very similar, so you also need some postprocessing to reach such a score.
", "rawMarkdown": "Congrats, great job! I just want to add that thresholding and fitting to distribution is very similar, so you also need some postprocessing to reach such a score.", "votes": 1, "replies": [ { "id": 742759, "postDate": "2020-02-11T14:27:49.267Z", "content": "Exactly :)
", "rawMarkdown": "Exactly :)", "votes": 1 }, { "id": 742769, "postDate": "2020-02-11T14:32:07.397Z", "content": "True, we never denied doing post-processing. We denied using thesholding :)
\n\nThesholding looks too crude compared to the bucketing technique introduced here, i.e. it looks more promising to analyze the whole distribution rather than tweeking a single threshold.
", "rawMarkdown": "True, we never denied doing post-processing. We denied using thesholding :)\n\nThesholding looks too crude compared to the bucketing technique introduced here, i.e. it looks more promising to analyze the whole distribution rather than tweeking a single threshold. " }, { "id": 742773, "postDate": "2020-02-11T14:33:58.603Z", "content": "But the threshold does a very similar thing. I actually tried both approaches on a very robust CV setup and fitting to distribution was always way worse for us. This was also confirmed when using rank-based approaches which inherenctly care more about the distribution.
\n\nThere is also one fundamental issue I see, namely that the distribution of train is inherently different to test, because it includes multiples of the same question. But apparently it worked, so what can I say :)
\n\nI would be curious if our PP would improve your results even more. We will post solution very soon.
", "rawMarkdown": "But the threshold does a very similar thing. I actually tried both approaches on a very robust CV setup and fitting to distribution was always way worse for us. This was also confirmed when using rank-based approaches which inherenctly care more about the distribution. \n\nThere is also one fundamental issue I see, namely that the distribution of train is inherently different to test, because it includes multiples of the same question. But apparently it worked, so what can I say :)\n\nI would be curious if our PP would improve your results even more. We will post solution very soon.", "votes": 2 }, { "id": 742776, "postDate": "2020-02-11T14:36:25.750Z", "content": "Hm. then it might be a \"kaggle case\" when a certain technique works for some teams but not for others. @ddanevskyi did we try simple thresholds? I doubt that, I guess we only checked the mentioned technique.
", "rawMarkdown": "Hm. then it might be a \"kaggle case\" when a certain technique works for some teams but not for others. @ddanevskyi did we try simple thresholds? I doubt that, I guess we only checked the mentioned technique. " }, { "id": 742784, "postDate": "2020-02-11T14:38:14.133Z", "content": "I'm not sure we tried any threshold-based approaches.
\n\nBut in general I agree those technics are very similar.
", "rawMarkdown": "I'm not sure we tried any threshold-based approaches.\n\nBut in general I agree those technics are very similar." }, { "id": 742787, "postDate": "2020-02-11T14:39:22.620Z", "content": "Whatever works on private LB is the right approach :)
", "rawMarkdown": "Whatever works on private LB is the right approach :) ", "votes": 2 }, { "id": 742850, "postDate": "2020-02-11T15:39:07.217Z", "content": "\n\n\nWe denied using thesholding :)
\n
No you didn't. One of your team mate wrote:
\n\n\n\n\nActually I didn't get how we got this score. I guess I just arbitrary played with thresholds…\n See https://www.kaggle.com/c/google-quest-challenge/discussion/127529#733065
\n
That's the only negative I can see in what your team did. You mislead people here and also when you said you don't use trick to get to 46+.
", "rawMarkdown": "> We denied using thesholding :)\n\nNo you didn't. One of your team mate wrote:\n> Actually I didn't get how we got this score. I guess I just arbitrary played with thresholds…\nSee https://www.kaggle.com/c/google-quest-challenge/discussion/127529#733065\n\nThat's the only negative I can see in what your team did. You mislead people here and also when you said you don't use trick to get to 46+." }, { "id": 742907, "postDate": "2020-02-11T16:30:47.720Z", "content": "Well, probably I have to explain that comment :)
\n\nI certainly didn't mean to confuse anyone. I'm so sorry this comment misled some of the fellow kagglers.
\n\nThat post was actually a pun. This \"arbitrary played with thresholds\" thing was a reference to the @wowfattie solution for TF2.0 competition, where he used it to describe some kind of \"magic\" he did in the last couple of hours to win the competition. In my comment, I decided to use the same phrase to describe some \"magical\" stuff we did in order to drastically improve our score. I didn't really think it will confuse anyone.
\n\nIt's just a sort of a bad coincidence that happened to (probably) affect some teams.
", "rawMarkdown": "Well, probably I have to explain that comment :) \n\nI certainly didn't mean to confuse anyone. I'm so sorry this comment misled some of the fellow kagglers.\n\nThat post was actually a pun. This \"arbitrary played with thresholds\" thing was a reference to the @wowfattie solution for TF2.0 competition, where he used it to describe some kind of \"magic\" he did in the last couple of hours to win the competition. In my comment, I decided to use the same phrase to describe some \"magical\" stuff we did in order to drastically improve our score. I didn't really think it will confuse anyone.\n\nIt's just a sort of a bad coincidence that happened to (probably) affect some teams.", "votes": 4 }, { "id": 742919, "postDate": "2020-02-11T16:36:09.123Z", "content": "I missed the pun, but it is a good one indeed ;)
\n\nIt didn't misled us, but I don't know for others. ANyway, binning and thresholding are quite similar. Not the same, but similar.
", "rawMarkdown": "I missed the pun, but it is a good one indeed ;)\n\nIt didn't misled us, but I don't know for others. ANyway, binning and thresholding are quite similar. Not the same, but similar.", "votes": 3 }, { "id": 742927, "postDate": "2020-02-11T16:38:52.237Z", "content": "Yeah, the comment about not using any thresholding was a bit confusing, I knew it is not possible, but others might have been quite mislead by it.
", "rawMarkdown": "Yeah, the comment about not using any thresholding was a bit confusing, I knew it is not possible, but others might have been quite mislead by it." }, { "id": 742940, "postDate": "2020-02-11T16:49:48.723Z", "content": "Actually, at the time that comment was made, we were so upset with the results of that TF Q&A competition, that I immediately understood that comment as a reference to Guanshuo's amazing finish. At that point I didn't recognize thresholding as smth boosting score here in this competition.
", "rawMarkdown": "Actually, at the time that comment was made, we were so upset with the results of that TF Q&A competition, that I immediately understood that comment as a reference to Guanshuo's amazing finish. At that point I didn't recognize thresholding as smth boosting score here in this competition. ", "votes": 1 }, { "id": 742954, "postDate": "2020-02-11T17:00:34.960Z", "content": "Rather refering to the posts yesterday where you teased a few times that you use no tricks and no thresholding. But it really does not matter. I am just happy that you used some sort of postprocessing and we did not miss something essential :)
", "rawMarkdown": "Rather refering to the posts yesterday where you teased a few times that you use no tricks and no thresholding. But it really does not matter. I am just happy that you used some sort of postprocessing and we did not miss something essential :)", "votes": 4 }, { "id": 743128, "postDate": "2020-02-11T20:12:59.843Z", "content": "Yes, sorry, then the question on how we define a trick is indeed relevant. I thought once the observation was shared everyone was doing smth like our post-processing. Everyone also discussed post-processing. So I no longer treated that as a trick.
", "rawMarkdown": "Yes, sorry, then the question on how we define a trick is indeed relevant. I thought once the observation was shared everyone was doing smth like our post-processing. Everyone also discussed post-processing. So I no longer treated that as a trick. " } ] }, { "id": 742434, "postDate": "2020-02-11T09:44:56.247Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 742348, "postDate": "2020-02-11T08:49:57.410Z", "content": "Well done @kashnitsky and team. So excited to see this result.
", "rawMarkdown": "Well done @kashnitsky and team. So excited to see this result.", "votes": 1, "replies": [ { "id": 742374, "postDate": "2020-02-11T09:11:06.683Z", "content": "Thanks, Ken! I’d refactor it as @ddanevskyi & team :)
", "rawMarkdown": "Thanks, Ken! I’d refactor it as @ddanevskyi & team :) ", "votes": 2 } ] }, { "id": 742191, "postDate": "2020-02-11T06:18:43.140Z", "content": "looking forward to the only \"no post-process\" solution. Congratulations
", "rawMarkdown": "looking forward to the only \"no post-process\" solution. Congratulations", "votes": 1, "replies": [ { "id": 742379, "postDate": "2020-02-11T09:13:40.110Z", "content": "Well, there is some post-processing. But tuning thresholds for each target was clearly seen as a straight way to overfitting, especially with 30 targets and so small public test size.
", "rawMarkdown": "Well, there is some post-processing. But tuning thresholds for each target was clearly seen as a straight way to overfitting, especially with 30 targets and so small public test size. ", "votes": 2 } ] }, { "id": 741916, "postDate": "2020-02-11T02:08:57.793Z", "content": "Congratulation !! Looking forword to your codes!
", "rawMarkdown": "Congratulation !! Looking forword to your codes!", "votes": 1 }, { "id": 741848, "postDate": "2020-02-11T01:02:45.310Z", "content": "Great! I started to feel excited after the competition is over -- because I can see non-threshold solutions!!!
", "rawMarkdown": "Great! I started to feel excited after the competition is over -- because I can see non-threshold solutions!!!", "votes": 1 }, { "id": 741841, "postDate": "2020-02-11T00:59:48.317Z", "content": "Wow!! We will have already one thresholding free solution!! Thanks for the open-source..
", "rawMarkdown": "Wow!! We will have already one thresholding free solution!! Thanks for the open-source..", "votes": 1 }, { "id": 747303, "postDate": "2020-02-16T08:18:58.250Z", "content": "Incredible article!
", "rawMarkdown": "Incredible article!", "votes": 2 }, { "id": 746499, "postDate": "2020-02-15T05:24:40Z", "content": "What a great explanation! Congrats!
", "rawMarkdown": "What a great explanation! Congrats!", "votes": 2 }, { "id": 745043, "postDate": "2020-02-13T12:28:17.347Z", "content": "Congratulations and many thanks for you and your team for the insights!
", "rawMarkdown": "Congratulations and many thanks for you and your team for the insights!", "votes": 2 }, { "id": 743350, "postDate": "2020-02-12T02:15:25.557Z", "content": "Congratulations for your work and thanks for sharing!\nI am trying to run your kernel and learn. So excuse me if i make stupid questions, but:\n1. What about all those missing files like \n'/kaggle/input/old-bert-code/predict_test.py'\n'../input/roberta-base-code/infer.py'\netc.\n1. Using stackexchange (SE), you reference:
\n\n\n\n\n... he introduced stackexchange (SE) pretrained LMs with 110k-long dictionary including code and other domain tokens. In addition to the common MLM task, the model also predicted 6 auxiliary targets (question_score, question_view_count, question_favorite_count, answer_score, answers_count, is_answer_accepted) with a linear layer on top of pooled LM output.\n ..This model is initialized with original BERT weights, then finetuning with SX data, finally reusing the checkpoint in our pipeline
\n
Could you elaborate a little more on funetuning BERT with SE data? How many training SE examples did you use? It would be nice to see code for LM pretraining on stackexchange data (as refeenced in https://github.com/oleg-yaroshevskiy/quest_qa_labeling). I tried a similar pipeline - to finetune BERT-base with SE data on Masked LM task (using Tensorflow 2.0) and then continue fine-tuning on our Google QA data - playing with a single GPU and using 50k training SE examples for not so many epochs. The performance was worst than simple BERT-base so I decided to stay with that.
\n\nAnd finally (I think a basic point for everyone interested) what resources did you use (how many GPUs, TPUs)?
", "rawMarkdown": "Congratulations for your work and thanks for sharing!\nI am trying to run your kernel and learn. So excuse me if i make stupid questions, but:\n1. What about all those missing files like \n'/kaggle/input/old-bert-code/predict_test.py'\n'../input/roberta-base-code/infer.py'\netc.\n1. Using stackexchange (SE), you reference:\n> ... he introduced stackexchange (SE) pretrained LMs with 110k-long dictionary including code and other domain tokens. In addition to the common MLM task, the model also predicted 6 auxiliary targets (question_score, question_view_count, question_favorite_count, answer_score, answers_count, is_answer_accepted) with a linear layer on top of pooled LM output.\n..This model is initialized with original BERT weights, then finetuning with SX data, finally reusing the checkpoint in our pipeline\n\nCould you elaborate a little more on funetuning BERT with SE data? How many training SE examples did you use? It would be nice to see code for LM pretraining on stackexchange data (as refeenced in https://github.com/oleg-yaroshevskiy/quest_qa_labeling). I tried a similar pipeline - to finetune BERT-base with SE data on Masked LM task (using Tensorflow 2.0) and then continue fine-tuning on our Google QA data - playing with a single GPU and using 50k training SE examples for not so many epochs. The performance was worst than simple BERT-base so I decided to stay with that.\n\nAnd finally (I think a basic point for everyone interested) what resources did you use (how many GPUs, TPUs)?", "votes": 2, "replies": [ { "id": 744487, "postDate": "2020-02-12T21:52:54.373Z", "content": "@ddanevskyi can you check missing files and datasets left private?
The LM finetuning code is there in the same repository, branch “monty”. We are waiting for a detailed post from Dmitry A. specifically about that.
As for GPUs, we had pretty limited resources. One Quadro P6000 for me, a couple of 1080ti for both Dmitry D. and Oleg (correct me if I’m wrong). Well, Dmitry A. had several GPUs for LM finetuning.
", "rawMarkdown": "1. @ddanevskyi can you check missing files and datasets left private?\n\n2. The LM finetuning code is there in the same repository, branch “monty”. We are waiting for a detailed post from Dmitry A. specifically about that. \n\nAs for GPUs, we had pretty limited resources. One Quadro P6000 for me, a couple of 1080ti for both Dmitry D. and Oleg (correct me if I’m wrong). Well, Dmitry A. had several GPUs for LM finetuning. ", "votes": 1 } ] }, { "id": 742737, "postDate": "2020-02-11T14:15:26.483Z", "content": "Awesome. Thanks for sharing your approach and congrats on fantastic performance!
\n\nDid you train your stack exchange pretrained models yourself? i.e. you randomized the weights of a transformer and downloaded the stack exchange data and trained all weights yourselves from scratch?
\n\nThen how exactly did you use the pretrained stack exchange model? i.e. did you use the output of that model as word embeddings to another BERT model? Or did you finetune that stack exchange model?
", "rawMarkdown": "Awesome. Thanks for sharing your approach and congrats on fantastic performance!\n\nDid you train your stack exchange pretrained models yourself? i.e. you randomized the weights of a transformer and downloaded the stack exchange data and trained all weights yourselves from scratch?\n\nThen how exactly did you use the pretrained stack exchange model? i.e. did you use the output of that model as word embeddings to another BERT model? Or did you finetune that stack exchange model? ", "votes": 2, "replies": [ { "id": 742747, "postDate": "2020-02-11T14:20:45.350Z", "content": "Thanks!
\n\nWe basically used the same BERT model with a linear layer, first fine-tuning it with StackExchange data, then - with our competition data.
", "rawMarkdown": "Thanks!\n\nWe basically used the same BERT model with a linear layer, first fine-tuning it with StackExchange data, then - with our competition data. " }, { "id": 742758, "postDate": "2020-02-11T14:26:10.093Z", "content": "```\nclass BertPretrain(BertForPreTraining):\n def init(self, config, num_labels):\n super(BertPretrain, self).init(config, )
\n\n self.dropout = nn.Dropout(config.hidden_dropout_prob)\n self.classifier = nn.Linear(config.hidden_size, num_labels)\n\ndef forward(self, input_ids, token_type_ids=None, attention_mask=None, masked_lm_labels=None,\n next_sentence_label=None, position_ids=None, head_mask=None):\n outputs = self.bert(input_ids, position_ids=position_ids, token_type_ids=token_type_ids,\n attention_mask=attention_mask, head_mask=head_mask)\n\n sequence_output, pooled_output = outputs[:2]\n prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)\n\n mean_pooled_output = torch.mean(sequence_output, dim=1)\n mean_pooled_output = self.dropout(mean_pooled_output)\n logits = self.classifier(mean_pooled_output)\n\n outputs = (prediction_scores, seq_relationship_score, logits)\n return outputs\n```
\n\nThis model is initialized with original BERT weights, then finetuning with SX data, finally reusing the checkpoint in our pipeline. A subtle intermediate step was to delete classifier weights and biases from the SX checkpoint, because we had to switch from 6 SX targets to 30.
", "rawMarkdown": "```\nclass BertPretrain(BertForPreTraining):\n def __init__(self, config, num_labels):\n super(BertPretrain, self).__init__(config, )\n\n self.dropout = nn.Dropout(config.hidden_dropout_prob)\n self.classifier = nn.Linear(config.hidden_size, num_labels)\n\n def forward(self, input_ids, token_type_ids=None, attention_mask=None, masked_lm_labels=None,\n next_sentence_label=None, position_ids=None, head_mask=None):\n outputs = self.bert(input_ids, position_ids=position_ids, token_type_ids=token_type_ids,\n attention_mask=attention_mask, head_mask=head_mask)\n\n sequence_output, pooled_output = outputs[:2]\n prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)\n\n mean_pooled_output = torch.mean(sequence_output, dim=1)\n mean_pooled_output = self.dropout(mean_pooled_output)\n logits = self.classifier(mean_pooled_output)\n\n outputs = (prediction_scores, seq_relationship_score, logits)\n return outputs\n```\n\nThis model is initialized with original BERT weights, then finetuning with SX data, finally reusing the checkpoint in our pipeline. A subtle intermediate step was to delete classifier weights and biases from the SX checkpoint, because we had to switch from 6 SX targets to 30. ", "votes": 3 } ] }, { "id": 742260, "postDate": "2020-02-11T07:33:17.097Z", "content": "Congrats @yaroshevskiy and team. You really deserve this win and I look forward to your solution.
", "rawMarkdown": "Congrats @yaroshevskiy and team. You really deserve this win and I look forward to your solution.", "votes": 2 }, { "id": 741898, "postDate": "2020-02-11T01:58:57.403Z", "content": "congrats! looking forward to your writeup.
", "rawMarkdown": "congrats! looking forward to your writeup.", "votes": 2, "replies": [ { "id": 741911, "postDate": "2020-02-11T02:04:41.853Z", "content": "and code ? :D
", "rawMarkdown": "and code ? :D " } ] }, { "id": 1044979, "postDate": "2020-10-10T09:04:28.380Z", "content": "What is this function?
", "rawMarkdown": "What is this function?" }, { "id": 921748, "postDate": "2020-07-09T14:13:19.197Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 850684, "postDate": "2020-05-16T22:51:37.030Z", "content": "@yaroshevskiy Just making sure, did you pseudo label and pretrain on the same set?
", "rawMarkdown": "@yaroshevskiy Just making sure, did you pseudo label and pretrain on the same set?" }, { "id": 759595, "postDate": "2020-02-29T08:50:22.510Z", "content": "@yaroshevskiy Congratulations.
", "rawMarkdown": "@yaroshevskiy Congratulations." }, { "id": 756885, "postDate": "2020-02-26T07:02:18.267Z", "content": "Congratulations !!!
", "rawMarkdown": "Congratulations !!!" }, { "id": 755875, "postDate": "2020-02-25T08:11:01.007Z", "content": "Congratulations !!!! and thanks for sharing.
", "rawMarkdown": "Congratulations !!!! and thanks for sharing." }, { "id": 754983, "postDate": "2020-02-24T09:54:18.310Z", "content": "congrats!
", "rawMarkdown": "congrats!" }, { "id": 754780, "postDate": "2020-02-24T03:31:31.277Z", "content": "Congrats!
", "rawMarkdown": "Congrats!" }, { "id": 754761, "postDate": "2020-02-24T02:57:29.907Z", "content": "Congratulations!!!!
", "rawMarkdown": "Congratulations!!!!" }, { "id": 754179, "postDate": "2020-02-23T07:07:54.420Z", "content": "Nice
", "rawMarkdown": "Nice" }, { "id": 753712, "postDate": "2020-02-22T15:30:53.933Z", "content": "brilliant
", "rawMarkdown": "brilliant" }, { "id": 753402, "postDate": "2020-02-22T06:09:25.213Z", "content": "Congratulations !!
", "rawMarkdown": "Congratulations !!" }, { "id": 752543, "postDate": "2020-02-21T06:09:57.083Z", "content": "Congratulations!!!
", "rawMarkdown": "Congratulations!!!" }, { "id": 752409, "postDate": "2020-02-21T02:12:40.863Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 752303, "postDate": "2020-02-20T21:18:51.360Z", "content": "Great
", "rawMarkdown": "Great" }, { "id": 751925, "postDate": "2020-02-20T16:37:53.703Z", "content": "Congratulations !!!
", "rawMarkdown": "Congratulations !!!" }, { "id": 751867, "postDate": "2020-02-20T15:28:16.340Z", "content": "brilliant!
", "rawMarkdown": "brilliant!" }, { "id": 751558, "postDate": "2020-02-20T09:21:32.357Z", "content": ".
", "rawMarkdown": "." }, { "id": 751134, "postDate": "2020-02-20T02:39:03.877Z", "content": "Good topic! !!
", "rawMarkdown": "Good topic! !!" }, { "id": 751088, "postDate": "2020-02-20T01:47:30.603Z", "content": "Congratulations and thanks for sharing!!!
", "rawMarkdown": "Congratulations and thanks for sharing!!!" }, { "id": 751068, "postDate": "2020-02-20T01:19:30.340Z", "content": "Congratz!
", "rawMarkdown": "Congratz!" }, { "id": 751012, "postDate": "2020-02-19T23:29:09.060Z", "content": "Congrats !
", "rawMarkdown": "Congrats !" }, { "id": 750847, "postDate": "2020-02-19T18:38:25.273Z", "content": "Congratulations!!! Thanks for sharing the Code
", "rawMarkdown": "Congratulations!!! Thanks for sharing the Code" }, { "id": 750584, "postDate": "2020-02-19T14:32:26.577Z", "content": "congrats
", "rawMarkdown": "congrats" }, { "id": 750466, "postDate": "2020-02-19T12:12:13.290Z", "content": "wow!
", "rawMarkdown": "wow!\n" }, { "id": 750247, "postDate": "2020-02-19T08:19:04.350Z", "content": "Master Piece, Congratulations..!!
", "rawMarkdown": "Master Piece, Congratulations..!!" }, { "id": 750106, "postDate": "2020-02-19T06:25:40.993Z", "content": "Respect your efforts, not results - is appropriate motto and following it is a hallmark of success.
", "rawMarkdown": "Respect your efforts, not results - is appropriate motto and following it is a hallmark of success." }, { "id": 750098, "postDate": "2020-02-19T06:13:59.440Z", "content": "cool
", "rawMarkdown": "cool" }, { "id": 750090, "postDate": "2020-02-19T05:59:35.723Z", "content": "Congratulations
", "rawMarkdown": "Congratulations" }, { "id": 750052, "postDate": "2020-02-19T05:03:12.510Z", "content": "Congratulations!!
", "rawMarkdown": "Congratulations!!" }, { "id": 749799, "postDate": "2020-02-18T22:53:41.663Z", "content": "Great work and Congratulations!!!
", "rawMarkdown": "Great work and Congratulations!!!" }, { "id": 749312, "postDate": "2020-02-18T15:10:22.640Z", "content": "Congratulation and thanks for your sharing
", "rawMarkdown": "Congratulation and thanks for your sharing" }, { "id": 749216, "postDate": "2020-02-18T12:57:30.180Z", "content": "Congratulations !!
", "rawMarkdown": "Congratulations !!" }, { "id": 749076, "postDate": "2020-02-18T09:16:33.417Z", "content": "Congratulations!!
", "rawMarkdown": "Congratulations!!" }, { "id": 749065, "postDate": "2020-02-18T08:55:15.303Z", "content": "Congratulations
", "rawMarkdown": "Congratulations" }, { "id": 749049, "postDate": "2020-02-18T08:25:58.053Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 748793, "postDate": "2020-02-18T02:05:13.610Z", "content": "👍
", "rawMarkdown": "👍 " }, { "id": 748781, "postDate": "2020-02-18T01:16:24.960Z", "content": "Congrats!
", "rawMarkdown": "Congrats!" }, { "id": 748778, "postDate": "2020-02-18T01:04:47.527Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 748749, "postDate": "2020-02-17T23:52:31.053Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 748594, "postDate": "2020-02-17T17:58:56.477Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 748160, "postDate": "2020-02-17T07:52:46.530Z", "content": "congrats
", "rawMarkdown": "congrats" }, { "id": 747731, "postDate": "2020-02-16T19:21:25.917Z", "content": "With such a great solution toward problems like this, helps the community to tackle problem for the day.
", "rawMarkdown": "With such a great solution toward problems like this, helps the community to tackle problem for the day. " }, { "id": 743874, "postDate": "2020-02-12T10:52:08.870Z", "content": "Thanks for sharing and releasing the code, really really nice solution, looking forward to going through it!
", "rawMarkdown": "Thanks for sharing and releasing the code, really really nice solution, looking forward to going through it!" }, { "id": 743528, "postDate": "2020-02-12T06:06:18.127Z", "content": "Congratulations bibimorph!!!!
", "rawMarkdown": "Congratulations bibimorph!!!!" }, { "id": 743526, "postDate": "2020-02-12T06:05:15.840Z", "content": "Nice👍
", "rawMarkdown": "Nice👍 " }, { "id": 743416, "postDate": "2020-02-12T03:36:14.510Z", "content": "Just Wow ! . There are so many Novel Ideas here !! .. Thanks for the tag too :)
", "rawMarkdown": "Just Wow ! . There are so many Novel Ideas here !! .. Thanks for the tag too :) " }, { "id": 743399, "postDate": "2020-02-12T03:08:19.850Z", "content": "Hi @yaroshevskiy, thanks for sharing the solution code, it will help me in understanding NLP questions from a better perspective.
", "rawMarkdown": "Hi @yaroshevskiy, thanks for sharing the solution code, it will help me in understanding NLP questions from a better perspective." }, { "id": 743261, "postDate": "2020-02-12T00:10:05.157Z", "content": "Could you provide more details of Different learning rate settings for encoder and head?
I think they must have grouped those parameters just like we do for LayerNorm etc? And given them a different learning rate... \nEdit - Yep they did it that way! Ref
", "rawMarkdown": "I think they must have grouped those parameters just like we do for LayerNorm etc? And given them a different learning rate... \nEdit - Yep they did it that way! [Ref](https://github.com/oleg-yaroshevskiy/quest_qa_labeling/blob/master/model.py#L88)", "votes": 2 }, { "id": 745098, "postDate": "2020-02-13T13:31:03.953Z", "content": "@oohara read about differential learning rate.\nthe code exists in fastai files
", "rawMarkdown": "@oohara read about differential learning rate.\nthe code exists in fastai files" } ] }, { "id": 742988, "postDate": "2020-02-11T17:23:15.737Z", "content": "Congratulations guys! Great work, thanks for sharing!
", "rawMarkdown": "Congratulations guys! Great work, thanks for sharing!", "replies": [ { "id": 857496, "postDate": "2020-05-22T16:46:24.683Z", "content": "Agree!
", "rawMarkdown": "Agree!" } ] }, { "id": 742841, "postDate": "2020-02-11T15:25:32.397Z", "content": "Thank you so much for sharing your approach! Congratulations!
", "rawMarkdown": "Thank you so much for sharing your approach! Congratulations!" }, { "id": 742782, "postDate": "2020-02-11T14:37:34.877Z", "content": "Congrats! Sharing your strategies helps newer Kaggle members and we greatly appreciate it
", "rawMarkdown": "Congrats! Sharing your strategies helps newer Kaggle members and we greatly appreciate it", "replies": [ { "id": 742846, "postDate": "2020-02-11T15:32:26.917Z", "content": "This comment was deleted.
", "rawMarkdown": "This comment was deleted." } ] }, { "id": 742597, "postDate": "2020-02-11T12:03:04.717Z", "content": "Amazing
", "rawMarkdown": "Amazing" }, { "id": 742585, "postDate": "2020-02-11T11:51:45.703Z", "content": "Congratulations! Looking forward to your update.
", "rawMarkdown": "Congratulations! Looking forward to your update." }, { "id": 742540, "postDate": "2020-02-11T11:14:00.290Z", "content": "Congratulation to team for all the efforts you have put for given competition.
", "rawMarkdown": "Congratulation to team for all the efforts you have put for given competition.\n" }, { "id": 742442, "postDate": "2020-02-11T09:49:46.633Z", "content": "Congratulations, I need sleep too:)
", "rawMarkdown": "Congratulations, I need sleep too:) " }, { "id": 742352, "postDate": "2020-02-11T08:53:15.997Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 742323, "postDate": "2020-02-11T08:28:10Z", "content": "congratulations !!! looking forward to a cool writeup.
", "rawMarkdown": "congratulations !!! looking forward to a cool writeup." }, { "id": 742216, "postDate": "2020-02-11T06:52:08.007Z", "content": "Congrats!\nI'm waiting for your sharing!
", "rawMarkdown": "Congrats!\nI'm waiting for your sharing!" }, { "id": 741932, "postDate": "2020-02-11T02:22:01.063Z", "content": "Congrats!!! looking forward to your result and writeup!!\n\"not about threshold\" << really expect that
", "rawMarkdown": "Congrats!!! looking forward to your result and writeup!!\n\"not about threshold\" << really expect that " }, { "id": 741915, "postDate": "2020-02-11T02:07:12.003Z", "content": "Congratulations! Looking forward to your non-thresholding solution!
", "rawMarkdown": "Congratulations! Looking forward to your non-thresholding solution!" }, { "id": 741879, "postDate": "2020-02-11T01:39:10.970Z", "content": "Congratulations! Waiting for your approach 🤟
", "rawMarkdown": "Congratulations! Waiting for your approach 🤟" }, { "id": 741839, "postDate": "2020-02-11T00:58:04.070Z", "content": "Congratulations!! I look forward to reading about input transformation your teammate mentioned at some point on the forums.
", "rawMarkdown": "Congratulations!! I look forward to reading about input transformation your teammate mentioned at some point on the forums. " }, { "id": 1821936, "postDate": "2022-06-15T23:59:20.637Z", "rawMarkdown": "", "isDeleted": true }, { "id": 795813, "postDate": "2020-04-03T04:26:12.490Z", "rawMarkdown": "", "isDeleted": true }, { "id": 753987, "postDate": "2020-02-22T22:35:13.550Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 752614, "postDate": "2020-02-21T08:08:54.980Z", "rawMarkdown": "", "isDeleted": true }, { "id": 749190, "postDate": "2020-02-18T12:18:00.410Z", "rawMarkdown": "", "isDeleted": true }, { "id": 749001, "postDate": "2020-02-18T06:49:20.197Z", "rawMarkdown": "", "isDeleted": true }, { "id": 748838, "postDate": "2020-02-18T03:23:44.203Z", "rawMarkdown": "", "isDeleted": true }, { "id": 748586, "postDate": "2020-02-17T17:42:50.157Z", "rawMarkdown": "", "isDeleted": true }, { "id": 748565, "postDate": "2020-02-17T17:08:37.310Z", "content": "well done!
", "rawMarkdown": "well done!", "isDeleted": true }, { "id": 747681, "postDate": "2020-02-16T18:08:28.983Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 746487, "postDate": "2020-02-15T04:53:28.067Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 744697, "postDate": "2020-02-13T03:31:44.030Z", "content": "congratulations!
", "rawMarkdown": "congratulations!", "votes": 1, "isDeleted": true }, { "id": 1019145, "postDate": "2020-09-20T08:14:06.167Z", "content": "@Yaroshevskiy congo thanks for sharing
", "rawMarkdown": "@Yaroshevskiy congo thanks for sharing", "votes": 2 }, { "id": 747593, "postDate": "2020-02-16T16:01:45.523Z", "content": "Great work. Thanks for sharing!
", "rawMarkdown": "Great work. Thanks for sharing!", "votes": 1 }, { "id": 746545, "postDate": "2020-02-15T07:20:10.180Z", "content": "Congrats and thanks for sharing!
", "rawMarkdown": "Congrats and thanks for sharing!", "votes": 1 }, { "id": 746239, "postDate": "2020-02-14T19:20:15.930Z", "content": "Great work ! Thanks for sharing :D
", "rawMarkdown": "Great work ! Thanks for sharing :D", "votes": 1 }, { "id": 753602, "postDate": "2020-02-22T12:55:02.197Z", "content": "Congrates thanks for sharing
", "rawMarkdown": "Congrates thanks for sharing" }, { "id": 753435, "postDate": "2020-02-22T07:16:47.843Z", "content": "Great work! thanks!
", "rawMarkdown": "Great work! thanks!" }, { "id": 752736, "postDate": "2020-02-21T10:56:04.487Z", "content": "Congratulations!\nThanks for sharing.
", "rawMarkdown": "Congratulations!\nThanks for sharing." }, { "id": 751772, "postDate": "2020-02-20T13:53:52.493Z", "content": "thank you for sharing!
", "rawMarkdown": "thank you for sharing!" }, { "id": 751110, "postDate": "2020-02-20T02:09:33.740Z", "content": "thanks for this 🎉
", "rawMarkdown": "thanks for this 🎉 " }, { "id": 750620, "postDate": "2020-02-19T14:48:14.907Z", "content": "Nice, Thanks.
", "rawMarkdown": "Nice, Thanks." }, { "id": 749388, "postDate": "2020-02-18T16:47:22.323Z", "content": "Congratulations!! Thanks for sharing
", "rawMarkdown": "Congratulations!! Thanks for sharing" }, { "id": 749141, "postDate": "2020-02-18T10:56:37.933Z", "content": "Congratulations and thank you!
", "rawMarkdown": "Congratulations and thank you!" }, { "id": 748975, "postDate": "2020-02-18T06:00:29.397Z", "content": "Amazing! Thanks for sharing.
", "rawMarkdown": "Amazing! Thanks for sharing." }, { "id": 748252, "postDate": "2020-02-17T10:04:14.890Z", "content": "Thanks
", "rawMarkdown": "Thanks" }, { "id": 743820, "postDate": "2020-02-12T10:24:03.177Z", "content": "Congratulations, thanks for sharing.
", "rawMarkdown": "Congratulations, thanks for sharing." }, { "id": 743738, "postDate": "2020-02-12T08:48:29.183Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 741875, "postDate": "2020-02-11T01:30:27.057Z", "content": "Congratulations!\nThanks for sharing.
", "rawMarkdown": "Congratulations!\nThanks for sharing." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:07:28Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4132452, "solution_url": "/competitions/google-quest-challenge/writeups/bibimorph-1st-place-solution-with-code" }