![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1435684%2F0bb9b707b885e27cb1a09b0df625a0dd%2FBildschirmfoto%202020-02-11%20um%2012.11.51.png?generation=1581431173679092&alt=media\")
Thanks to the sponsors and kaggle for hosting such an interesting and challenging competition! Our work is an amazing team effort and sprint over a very short time period of two weeks during which we made progress each day.
\nWhy two weeks? Because some of us just finished the TF2 QA competition and decided to reuse whatever NLP knowledge that was acquired there. Thanks to all teammates: @christofhenkel @cpmpml @philippsinger @dott1718 @maxjeblick.
Please also give them upvotes, we made a lottery roll of who will post the solution and everyone contributed equally to it.
\n\nOur only regret is that the final score depends too heavily on the question_type_spelling column that has only 11 non zero target values in train data. Late submission experiments show we could easily improve, or degrade, our final rank by little tweaks on how we handle this column.
\n\nLike most teams, we used pretrained models from huggingface implemented in pytorch. Our final blend consists of 5 transformer models, that have several differences to push diversity. However, they all have in common to use 2 separate inputs of up to 512 tokens length representing questions respectively answers. These 2 inputs go either through the same transformer or through 2 separate transformers. We changed the targets to an ordinal representation (setting >target to 1) resulting in 170 target columns with each original column having n-unique target columns. Predictions were then generated by calculating the expected value.
\n\nMost models were trained with differential learning rate, where the transformer gets a lr of 3-e5 and the model head(s) a lr of 0.005 and used a cosine schedule for training 3 epochs including warmup over one epoch. Optimizer were either AdamW or RAdam with weight decay of 0.01 using an effective batchsize of 8 (gradient accumulation).
\n\nDual roberta-base:\n- double roberta-base\n- added question title as input to answer-transformer\n- individual head for each target\n- mean of last layer output
\n\nDual roberta-base:\n- double roberta-base\n- added question title as input to answer-transformer\n- individual head for each target\n- mean of last layer output
\n\nSiamese roberta-base with softmax layer weight:\n- double roberta-base\n- added question title as input to answer-transformer\n- individual head for each target\n- mean of last layer output
\n\nDual roberta-large:\n- only use 256 tokens for question / answer\n- double roberta-large\n- only fits in memory with batchsize 1 and fp16
\n\nDual xlnet:\n- same as dual roberta-base but with xlnet-base backbone\n- We used a simple average of probabiity predictions to ensemble our models.
\n\nGiven the metric is rank-based, and given targets are not binary, it seemed important to be able to predict values that are neither 0 or 1 correctly. We tried mse loss and other variants but results were not satisfactory. We then decided to use binary cross-entropy with binary targets. The first try was to use on hot encoding of targets, given that targets have a small number of distinct values. This wasn’t satisfactory either, because this representation loses the ordering of values. \nWe ended up using an encoding of the form (t > v) for all v values of target t, except last value. For instance, if a target t has unique values [0, ⅓, ⅔, 1] then we would get 3 binary targets: t > 0, t> ⅓, and t > ⅔. Assuming we get perfect probability predictions p(0), p(⅓ ), and p(⅔) for the binary targets, we can compute predictions for each value of the original target as:\nt(0)= 1 - p(0)\nt(⅓)= p(0) - p(⅓)\nt(⅔)= p(⅓) - p(⅔)\nt(1)= p(⅔) - 0\nAnd the original target value as\nt = 0 * t(0) + ⅓ * t(⅓) + ⅔ * t(⅔) + 1*t(1)\nWe keep that computation with the actual predictions p(0), p(⅓ ), and p(⅔) for the binary targets.
\nWhen the values are evenly spaced, as in this example, then the formula simplifies into:\nt = mean(p(0), p(⅓ ), p(⅔))\nIf we assume that what matters is only the order of values, then we can stick to this simplified form.\nWe used the simplified form throughout the competition as it was simpler to code, and faster. In our final two selected subs we reverted to the exact computation because it was better on both cv and public LB, but this didn’t change private LB significantly (0.0004 difference).
Similar to most teams, we use a 5-fold GroupKFold on question body for fitting our models and evaluation. After a bunch of experiments we saw though, that this is not sufficient mainly due to the following aspects:\n- Test data is different to training data in the sense that it only has one question-answer pair sampled out of a group of questions in train data. There can be stark noise for labels of the same question, which is why this needs to be addressed robustly.\n- There are a few columns with very rare events and also a lot of noise within those rare events. The prime example is spelling which has a huge impact on CV and LB, but can completely blind your view when trying to judge the overall strength of your model.
\n\nThat is why we settled after some time on the following full validation scheme:\n- Use 5-fold GroupKFold on question body\n- For each validation fold, sample 100 times randomly a single question-answer pair out of multiple questions.\n- Calculate the median score across these 100 samples and report.\n- Ignore spelling column.\n- Final CV is a mean of 5 folds.\nThis setup also allowed us to properly test any type of postprocessing in a realistic manner as described next.
\n\nEven though we had really strong models, post-processing was very important to us and we spent significant time on it to find a robust way that we can trust. It is based on threshold clipping. \nFor each target column, we separately attempt to find optimal thresholds from both sides (starting from lowest predictions and highest predictions) based on which we clip the data. So let’s assume our predictions look like x=[0.01, 0.015, 0.02, 0.03, 0.04] and our optimal thresholds are coefs=[0.016, 0.029] then we would clip the data with np.clip(x, coefs[0], coefs[1]) leading to x=[0.016, 0.016, 0.02, 0.029, 0.029] effectively generating ties at the edges.
The CV setup from above gave us a perfect way to fully validate any of our approaches by simply generating the thresholds on training folds, and applying them on truncated samples of the validation folds. For final PP, we get the thresholds on full oof and apply them on the test set.\nIn a nutshell, our routine looks like the following:\n- Sample 1000 times single question-answer pairs from multiple questions\n- Generate thresholds that optimize the median score of all 1000 samples
\n\nWe tested quite a few different strategies, but this one was the most robust one we found and we are quite happy with it. Only for spelling, it was still a bit shaky, which is why our two final subs just differ in how we handle spelling column, one uses the calculated thresholds, and one hard-sets the 6 highest predictions in private LB to 1 and rest to 0 which is based on experiments on samples.\nUnfortunately, if we would have post-processed all columns except spelling and would have kept spelling as is, we would have reached 0.432 on private LB as apparently spelling is very differently distributed on private. No CV experiment would have let us make this decision though. It would have been so much better though to not include this column into the competition.
\n\nSimilar to other recent competitions at first it was quite difficult to beat public kernels, which we normally see as a kind of baseline. All “normal” tricks that worked in past computer vision or other NLP competitions (e.g. concat of Max and Mean pooling) did not improve cv. Also using a sliding window approach to capture more text did not work.\nWe saw a first big improvement when using 2 transformers instead of one (+0.01), but still, any slightly more complex architectures lead to a worse result. That changed when we tried to freeze the transformers and only train the model head for 1 epoch, before fine-tuning. With this approach, we were able to try more fancy things and that is also how we finally came to the two main architectures. \nWe further improved than this 2 step approach by using different learning rates for transformer and head, together with a warm up schedule, which enabled us to get rid of the freezing step in general. We illustrate the two main architectures in the following figure (For the sake of simplicity we show the architecture for the original 30 targets. It was adapted to work with the binarized targets.). \n
The upper architecture shows the dual transformer model, a combo model where one transformer handles the question text, while a second transformer handles the answer text. the output of the last layer is then averaged over the 512 tokens and both resulting tensors are concatenated. The resulting representation is then fed into 30 little 2 fully connected layers which result in the target predictions. 30 little heads enable each target to have its own head and gain individuality.
\n\nIt is inspired by the Elmo paper, where the final embedding representation is a weighted average of all LSTM layers. We use the output of every layer of a single transformer model in which we put 512 question and 512 answer tokens. For roberta-base that will give us twelve 512x768 tensors (for roberta large it would be 24 512x1024 tensors). We then average over the 512 tokens for each layer which results in twelve 768 representations. We then take a weighted sum of these 12 representations (where the weights for adding the representations are trainable!). This results in a 768 representation. The weighted average of all layer outputs enables to capture low level features in the final representation which was quite important for some answer related targets. Finally, we add a single prediction head for getting our targets.
\n\nOur final ensemble contains 5 models which belong to the one or the other architecture, and only differ in used pretrained backbone. \nWe ended up with \n- 2x dual roberta-base\n- dual roberta-large (2x 256 tokens)\n- dual xnet-base\n- siamese roberta-large with weighted averaged layers\nas this combination had the best cv score while fitting in the 2h kernel runtime requirement.
\n\nWe even trained one dual roberta-large model which on a V100 can only be trained when using a batch size of 1 and fp16. and was our best single model. Although this model could not fit into the ensemble due to runtime reasons, the promising cv results let us to a daring experiment: reducing the number of tokens used. There we saw something quite surprising. We could reduce the number of tokens to 256 for question as well as for answer without losing much quality.
\n\nMost models were trained with differential learning rate, where the transformer gets a lr of 3-e5 and the model head(s) a lr of 0.005 and used a cosine schedule for training 3 epochs including warmup over one epoch. Optimizer were either AdamW or RAdam with weight decay of 0.01 using an effective batchsize of 8 (gradient accumulation).
\n\nOur submission includes an equally weighted blend between all of above mentioned 5 models (leading to 25 test predictions with 5-fold). The blend is conducted on raw expected values calculated as described above. We experimented quite a bit with ranked blending as it seemed natural, but results were always worse.\nOur final blend is both our best one on CV, public LB and private LB meaning that we had a robust CV setup in the end and our selection was solid.
\n\nTraining was conducted offline and inference in kernel. Our final runtime was close to two hours and we had to spend some efforts to squeeze the 5 models in. All our models are implemented in Pytorch using the amazing Huggingface library. Fitting was done either locally or on cloud providers.
\n\nThanks for reading.
", "comments": [ { "id": 742868, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-11T15:54:46.687000", "content": "I really like your binary encoding trick. I tried something similar and it nearly performed as well as post processing. For each target, you predict 3 values.
\n\nfor i in range(3):\n x = Dense(30, activation='sigmoid')(bert)\n heads.append(x)\nmodel = Model(inputs=inputs, outputs=heads)\nAnd you train with
\n\nmodel.fit(inputs, [targets, (targets==1).astype(int), (targets==0).astype(int)])\nThen your final predictions are
\n\npreds, t1, t0 = model.predict(inputs)\npreds[t1>0.8] = 1\npreds[t0>0.8] = 0\nNice, this is a simplified version of what we used, but it may perform as well in practice. Thanks for sharing.
", "votes": 2, "replies": [] } ] }, { "id": 742803, "author_name": "Dieter", "author_url": "", "post_date": "2020-02-11T14:54:57.267000", "content": "Really cool sprint. And to be honest, after TF2.0QA I thought I already know everything about transformers. Quite the opposite - I learned a lot in the short time. Thank you all for that.
", "votes": 12, "replies": [] }, { "id": 742852, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-11T15:40:16.443000", "content": "Great job Max, Psi, CPMP, Dieter, Dott. Awesome models.
\n\nI want to make sure I understand your architecture. For your dual transformer (original 30 targets) are the last layers the following?
\n\nx = Concatenate([bert_1,bert_2])\nheads = []\nfor i in range(30):\n h = Dense(64, activation='tanh')(x)\n h = Dense(1, activation='sigmoid')(h)\n heads.append(h)\nmodel = Model(inputs=inputs, outputs=heads)\nAnd your Siamese is
\n\npoolings = []\nfor i in range(12):\n embeddings = bert[2][i]\n x = GlobalAveragePooling1D(embeddings)\n poolings.append(x)\nstacked = Lambda(lambda x: K.stack(x, axis=-1))(poolings)\nweighted = Dense(1, use_bias=False)(stacked)\nweighted = Lambda(lambda x: K.squeeze(x, axis=-1))(weighted)\nheads = Dense(64, activation='tanh')(weighted)\nheads = Dense(30, activation='sigmoid')(x)\nmodel = Model(inputs=inputs, outputs=heads)\nYes, in principle. We left out some details like activation function, dropout and layernorm though.
", "votes": 3, "replies": [] } ] }, { "id": 742822, "author_name": "dott", "author_url": "", "post_date": "2020-02-11T15:04:36.967000", "content": "Congrats to the winners and thanks a lot to the amazing teammates!
", "votes": 7, "replies": [] }, { "id": 742790, "author_name": "CPMP", "author_url": "", "post_date": "2020-02-11T14:43:13.063000", "content": "I was so happy to be on that dream team, learned a lot, had fun, and got a great result!
", "votes": 8, "replies": [] }, { "id": 742778, "author_name": "Psi", "author_url": "", "post_date": "2020-02-11T14:36:31.373000", "content": "Thanks again for the great team work and I am super stoked about what we could achieve in such a team in such a short time period of two weeks.
", "votes": 6, "replies": [] }, { "id": 742993, "author_name": "TJ Klein", "author_url": "", "post_date": "2020-02-11T17:27:54.983000", "content": "Dream team indeed...congrats guys, well earned!
", "votes": 3, "replies": [] }, { "id": 742956, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-02-11T17:00:49.677000", "content": "Thank you for such a detailed writeup! Congrats on your result guys :)
", "votes": 3, "replies": [ { "id": 742968, "author_name": "Dieter", "author_url": "", "post_date": "2020-02-11T17:07:59.290000", "content": "I am amazed how similar our models are. Weighted sum of all layers and differential learning rate :)
", "votes": 3, "replies": [] } ] }, { "id": 742854, "author_name": "Firas Baba", "author_url": "", "post_date": "2020-02-11T15:42:20.643000", "content": "your solution is so similar to ours, The 2 differences is that we used 3 transformers instead of 2 + we backpropagate in every transformer using 4 losses separately ( loss1(answers_target, output from bert_answers) , loss2(question_targets, output from bert_question), loss3(targets, output from bert3 which combines answer+question), loss4 is final loss after concatenating all logits from different transofmers))\nAn epoch takes 40 minutes on a V100 since we're not destroying the graph after each backward().\nCongrats you did a good job!
", "votes": 3, "replies": [ { "id": 742926, "author_name": "CPMP", "author_url": "", "post_date": "2020-02-11T16:38:46.557000", "content": "Congrats to you as well. We tried the same 3 berts as you and did not find it to help. Maybe we didn't try hard enough ;) Congrats on your result too!
", "votes": 1, "replies": [] } ] }, { "id": 744517, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-12T22:38:09.570000", "content": "Wow, I just read how your private score would have been 0.432 if you didn't post process question_type_spelling column. That's a bummer to figure that out now. I agree that that was a poor column to add when the competition metric is Spearman's rank correlation, because just a few correct and incorrect predictions radically change that column's score.
Indeed, this is the lottery part of the competition.
", "votes": -1, "replies": [] } ] }, { "id": 743753, "author_name": "OmarSalman", "author_url": "", "post_date": "2020-02-12T09:14:09.680000", "content": "Congratulations! Excellent solution.
", "votes": 1, "replies": [] }, { "id": 743003, "author_name": "Gerwyn", "author_url": "", "post_date": "2020-02-11T17:43:24.170000", "content": "Congrats to dream team!! Thanks for the detailed write up!
", "votes": 1, "replies": [] }, { "id": 742842, "author_name": "liuze", "author_url": "", "post_date": "2020-02-11T15:27:43.457000", "content": "Congratulations! good solution
", "votes": 1, "replies": [] }, { "id": 744196, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2020-02-12T16:42:18.087000", "content": "Guys, did you try pretraining with additional SO/SX data? I'm a bit surprised (and humbled) with such a significant gap and the fact that no teams actually used pretraining (as far as I analyzed a bunch of top solutions).
\n\nThis was one of our first ideas, and by the time I collected those ruddy TF records to fine-tune a couple of MLMs with TPUs and 7kk SX records, our would-be teammate @dmitriyab came up with a finetuned model with auxiliary targets which made the whole difference. Well, it actually needs a careful ablation study (no time for that) but 99% that it's about that pretrained model.
\n\nDmitry is going to share his approach tonight, with an additional post, our code is already shared. So again, all kudos go to Dmitry, I'm just curious whether you considered finetuning as well.
", "votes": 2, "replies": [ { "id": 744220, "author_name": "Dieter", "author_url": "", "post_date": "2020-02-12T17:03:06.817000", "content": "Using the SO data was the first thing I tried. But it gave no improvement to our score and would be quite some effort, so in light of the tight timeline I discarded the idea.
", "votes": 1, "replies": [] }, { "id": 744228, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2020-02-12T17:07:23.940000", "content": "I see, thanks. Maybe indeed a regular MLM is not as beneficial, and it's auxiliary targets that make the difference. Probably we'll find time for a short ablation study.
", "votes": 2, "replies": [] } ] }, { "id": 1064621, "author_name": "Umer", "author_url": "", "post_date": "2020-10-30T10:44:03.060000", "content": "Thanks for sharing your ideas
\n\n\nWe ended up using an encoding of the form (t > v) for all v values of target t, except last value. For instance, if a target t has unique values [0, ⅓, ⅔, 1] then we would get 3 binary targets: t > 0, t> ⅓, and t > ⅔. Assuming we get perfect probability predictions p(0), p(⅓ ), and p(⅔) for the binary targets, we can compute predictions for each value of the original target as:
\n
\n t(0)= 1 - p(0) t(⅓)= p(0) - p(⅓) t(⅔)= p(⅓) - p(⅔) t(1)= p(⅔) - 0
\n And the original target value as
\n t = 0 * t(0) + ⅓ * t(⅓) + ⅔ * t(⅔) + 1*t(1)
\n We keep that computation with the actual predictions p(0), p(⅓ ), and p(⅔) for the binary targets.
\n When the values are evenly spaced, as in this example, then the formula simplifies into:
\n t = mean(p(0), p(⅓ ), p(⅔))
@christofhenkel @cpmpml @philippsinger @dott1718 @maxjeblick ,
\nI am having difficulty understanding this concept, how did you arrive at these formulas, would it be possible to explain it little more
1) t(⅓)= p(0) - p(⅓), how does t > 1/3 = p(0) - p(⅓), tried figuring out , not able to get it
\n2) t = 0 * t(0) + ⅓ * t(⅓) + ⅔ * t(⅔) + 1*t(1) , is this similar to expected value calculation ?
\n3) t = mean(p(0), p(⅓ ), p(⅔)) , how does the formula simplify to this
Is there any other place where this is explained ? I hope these are not too naive questions
\nThanks in advance
", "votes": 0, "replies": [ { "id": 1064714, "author_name": "CPMP", "author_url": "", "post_date": "2020-10-30T12:51:23.277000", "content": "Let me answer the first question, hopefully you'll get how it works with it.
\np(0) is the prediction probability that the actual target t is such that t > 0.
\nThe probability that t = 0 is t(0) and the probability that t > 0 is p(0).
\nBy definition t = 0 or t > 0 then t(0) + p(0) = 1
Similarly, the probability that t > 0 is p(0), the probability that t > ⅓ is p(⅓), and the probability that t = ⅓ is t(⅓)
\nt > 0 if and only if t > ⅓ or t = ⅓
\nThen we have that p(0) = p(⅓) + t(⅓)
\nand so on.
\nThis way of encoding ordinal target has been published but I don't have a handy reference.
", "votes": 2, "replies": [] }, { "id": 1064881, "author_name": "Umer", "author_url": "", "post_date": "2020-10-30T16:25:25.347000", "content": "thanks @cpmpml
", "votes": 0, "replies": [] } ] }, { "id": 750015, "author_name": "Hassan Amin", "author_url": "", "post_date": "2020-02-19T04:12:35.747000", "content": "Great work, I am totally impressed. Hats off !\nGreat community contribution as well.
", "votes": 0, "replies": [] }, { "id": 744772, "author_name": "Tonghui Li", "author_url": "", "post_date": "2020-02-13T06:06:26.777000", "content": "Thanks a lot for sharing! I have a question about clipping.
\ngenerating ties at the edges Did you guys do anything to generate ties in between the edges?
We tried, and we didn't found anything really working. We haven't tried rounding using estimated number of annotators though. Certainly worth a try.
", "votes": 0, "replies": [] } ] }, { "id": 744743, "author_name": "MarioGzSl", "author_url": "", "post_date": "2020-02-13T05:08:58.127000", "content": "Congrats, a very interesting solution!
", "votes": 0, "replies": [] }, { "id": 744546, "author_name": "corochann", "author_url": "", "post_date": "2020-02-12T23:27:52.267000", "content": "Congratulations and thank you for precise write up. learned a lot.
", "votes": 0, "replies": [] }, { "id": 744540, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-02-12T23:17:52.973000", "content": "Awesome solution, congratulations!
", "votes": 0, "replies": [] }, { "id": 744491, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-12T22:02:19.110000", "content": "I just realized something. I wanted to make a dual transformer like you guys did. I used the code from the popular public Huggingface notebook here. (This code is simplied by removing attention mask, and work token inputs).
\n\ndef create_model():\n q_id = tf.keras.layers.Input((512,), dtype=tf.int32)\n a_id = tf.keras.layers.Input((512,), dtype=tf.int32)\n\n bert_model = TFBertModel.from_pretrained('bert-base-uncased') \n\n q_embedding = bert_model(q_id)[0]\n a_embedding = bert_model(a_id)[0]\n\n q = tf.keras.layers.GlobalAveragePooling1D()(q_embedding)\n a = tf.keras.layers.GlobalAveragePooling1D()(a_embedding)\n\n x = tf.keras.layers.Concatenate()([q, a])\n x = tf.keras.layers.Dense(30, activation='sigmoid')(x)\n\n model = tf.keras.models.Model(inputs=[q_id, a_id], outputs=x)\n\nreturn model\nUpon refection, this doesn't use 2 BERTs does it? It seems that 1 BERT is acting similar to a \"convolution filter with shared weights\". This 1 BERT acts on input1 and it acts on input2 producing \"2 channels\" and the back propagation update has a contribution from each input flowing through the model. And that updates 1 set of weights.
\n\nDid you build your dual transformer in PyTorch or TensorFlow?
\n\nUPDATE: This post was originally a question, but I think i figured it out.
", "votes": 0, "replies": [ { "id": 744502, "author_name": "Psi", "author_url": "", "post_date": "2020-02-12T22:20:28.830000", "content": "What you are showing here is exactly our second type of model that we call \"siamese transformer\" in the figure above. Basically you just feed question and answer twice through the same transformer. In order to get a full dual transformer model, you need to define it separately:
\n\n\nbert_model_q = TFBertModel.from_pretrained('bert-base-uncased') \nbert_model_a = TFBertModel.from_pretrained('bert-base-uncased') \n
Thanks for the reply. As you were typing, I just figured it out and was updating my post.
\n\nWhich model had higher CV, your dual transformer or siamese transformer?
", "votes": 0, "replies": [] }, { "id": 744734, "author_name": "Dieter", "author_url": "", "post_date": "2020-02-13T04:48:20.240000", "content": "roughly the same. However they had different strength. Especially the usage of all layers for the siamese one improved diversity.
", "votes": 0, "replies": [] } ] }, { "id": 742872, "author_name": "morg", "author_url": "", "post_date": "2020-02-11T16:02:59.123000", "content": "Well done all!! Do you plan on releasing your training code? Would love to step through it line by line and learn a little more :)
", "votes": 0, "replies": [ { "id": 742874, "author_name": "Dieter", "author_url": "", "post_date": "2020-02-11T16:07:41.527000", "content": "We'd love to, but @abhishek asked everybody not to share code
", "votes": 7, "replies": [] }, { "id": 742977, "author_name": "Mycroft Holmes", "author_url": "", "post_date": "2020-02-11T17:17:02.100000", "content": "could somebody explain to me this joke?
", "votes": 0, "replies": [] }, { "id": 742984, "author_name": "Dieter", "author_url": "", "post_date": "2020-02-11T17:20:00.763000", "content": "https://www.kaggle.com/c/google-quest-challenge/discussion/129850
", "votes": 3, "replies": [] }, { "id": 743783, "author_name": "morg", "author_url": "", "post_date": "2020-02-12T10:00:38.440000", "content": "Haha hes a tough guy
", "votes": 0, "replies": [] } ] }, { "id": 752625, "author_name": "MachineLP", "author_url": "", "post_date": "2020-02-21T08:21:39.837000", "content": "Thanks for sharing!
", "votes": 2, "replies": [] }, { "id": 755990, "author_name": "Swathy Murali Mohan", "author_url": "", "post_date": "2020-02-25T10:33:35.107000", "content": "Thank you for Sharing!!
", "votes": 0, "replies": [] } ], "comment_count": 41, "messages": [ { "id": 742775, "postDate": "2020-02-11T14:35:15.737Z", "content": "Thanks to the sponsors and kaggle for hosting such an interesting and challenging competition! Our work is an amazing team effort and sprint over a very short time period of two weeks during which we made progress each day.
\nWhy two weeks? Because some of us just finished the TF2 QA competition and decided to reuse whatever NLP knowledge that was acquired there. Thanks to all teammates: @christofhenkel @cpmpml @philippsinger @dott1718 @maxjeblick.
Please also give them upvotes, we made a lottery roll of who will post the solution and everyone contributed equally to it.
\n\nOur only regret is that the final score depends too heavily on the question_type_spelling column that has only 11 non zero target values in train data. Late submission experiments show we could easily improve, or degrade, our final rank by little tweaks on how we handle this column.
\n\nLike most teams, we used pretrained models from huggingface implemented in pytorch. Our final blend consists of 5 transformer models, that have several differences to push diversity. However, they all have in common to use 2 separate inputs of up to 512 tokens length representing questions respectively answers. These 2 inputs go either through the same transformer or through 2 separate transformers. We changed the targets to an ordinal representation (setting >target to 1) resulting in 170 target columns with each original column having n-unique target columns. Predictions were then generated by calculating the expected value.
\n\nMost models were trained with differential learning rate, where the transformer gets a lr of 3-e5 and the model head(s) a lr of 0.005 and used a cosine schedule for training 3 epochs including warmup over one epoch. Optimizer were either AdamW or RAdam with weight decay of 0.01 using an effective batchsize of 8 (gradient accumulation).
\n\nDual roberta-base:\n- double roberta-base\n- added question title as input to answer-transformer\n- individual head for each target\n- mean of last layer output
\n\nDual roberta-base:\n- double roberta-base\n- added question title as input to answer-transformer\n- individual head for each target\n- mean of last layer output
\n\nSiamese roberta-base with softmax layer weight:\n- double roberta-base\n- added question title as input to answer-transformer\n- individual head for each target\n- mean of last layer output
\n\nDual roberta-large:\n- only use 256 tokens for question / answer\n- double roberta-large\n- only fits in memory with batchsize 1 and fp16
\n\nDual xlnet:\n- same as dual roberta-base but with xlnet-base backbone\n- We used a simple average of probabiity predictions to ensemble our models.
\n\nGiven the metric is rank-based, and given targets are not binary, it seemed important to be able to predict values that are neither 0 or 1 correctly. We tried mse loss and other variants but results were not satisfactory. We then decided to use binary cross-entropy with binary targets. The first try was to use on hot encoding of targets, given that targets have a small number of distinct values. This wasn’t satisfactory either, because this representation loses the ordering of values. \nWe ended up using an encoding of the form (t > v) for all v values of target t, except last value. For instance, if a target t has unique values [0, ⅓, ⅔, 1] then we would get 3 binary targets: t > 0, t> ⅓, and t > ⅔. Assuming we get perfect probability predictions p(0), p(⅓ ), and p(⅔) for the binary targets, we can compute predictions for each value of the original target as:\nt(0)= 1 - p(0)\nt(⅓)= p(0) - p(⅓)\nt(⅔)= p(⅓) - p(⅔)\nt(1)= p(⅔) - 0\nAnd the original target value as\nt = 0 * t(0) + ⅓ * t(⅓) + ⅔ * t(⅔) + 1*t(1)\nWe keep that computation with the actual predictions p(0), p(⅓ ), and p(⅔) for the binary targets.
\nWhen the values are evenly spaced, as in this example, then the formula simplifies into:\nt = mean(p(0), p(⅓ ), p(⅔))\nIf we assume that what matters is only the order of values, then we can stick to this simplified form.\nWe used the simplified form throughout the competition as it was simpler to code, and faster. In our final two selected subs we reverted to the exact computation because it was better on both cv and public LB, but this didn’t change private LB significantly (0.0004 difference).
Similar to most teams, we use a 5-fold GroupKFold on question body for fitting our models and evaluation. After a bunch of experiments we saw though, that this is not sufficient mainly due to the following aspects:\n- Test data is different to training data in the sense that it only has one question-answer pair sampled out of a group of questions in train data. There can be stark noise for labels of the same question, which is why this needs to be addressed robustly.\n- There are a few columns with very rare events and also a lot of noise within those rare events. The prime example is spelling which has a huge impact on CV and LB, but can completely blind your view when trying to judge the overall strength of your model.
\n\nThat is why we settled after some time on the following full validation scheme:\n- Use 5-fold GroupKFold on question body\n- For each validation fold, sample 100 times randomly a single question-answer pair out of multiple questions.\n- Calculate the median score across these 100 samples and report.\n- Ignore spelling column.\n- Final CV is a mean of 5 folds.\nThis setup also allowed us to properly test any type of postprocessing in a realistic manner as described next.
\n\nEven though we had really strong models, post-processing was very important to us and we spent significant time on it to find a robust way that we can trust. It is based on threshold clipping. \nFor each target column, we separately attempt to find optimal thresholds from both sides (starting from lowest predictions and highest predictions) based on which we clip the data. So let’s assume our predictions look like x=[0.01, 0.015, 0.02, 0.03, 0.04] and our optimal thresholds are coefs=[0.016, 0.029] then we would clip the data with np.clip(x, coefs[0], coefs[1]) leading to x=[0.016, 0.016, 0.02, 0.029, 0.029] effectively generating ties at the edges.
The CV setup from above gave us a perfect way to fully validate any of our approaches by simply generating the thresholds on training folds, and applying them on truncated samples of the validation folds. For final PP, we get the thresholds on full oof and apply them on the test set.\nIn a nutshell, our routine looks like the following:\n- Sample 1000 times single question-answer pairs from multiple questions\n- Generate thresholds that optimize the median score of all 1000 samples
\n\nWe tested quite a few different strategies, but this one was the most robust one we found and we are quite happy with it. Only for spelling, it was still a bit shaky, which is why our two final subs just differ in how we handle spelling column, one uses the calculated thresholds, and one hard-sets the 6 highest predictions in private LB to 1 and rest to 0 which is based on experiments on samples.\nUnfortunately, if we would have post-processed all columns except spelling and would have kept spelling as is, we would have reached 0.432 on private LB as apparently spelling is very differently distributed on private. No CV experiment would have let us make this decision though. It would have been so much better though to not include this column into the competition.
\n\nSimilar to other recent competitions at first it was quite difficult to beat public kernels, which we normally see as a kind of baseline. All “normal” tricks that worked in past computer vision or other NLP competitions (e.g. concat of Max and Mean pooling) did not improve cv. Also using a sliding window approach to capture more text did not work.\nWe saw a first big improvement when using 2 transformers instead of one (+0.01), but still, any slightly more complex architectures lead to a worse result. That changed when we tried to freeze the transformers and only train the model head for 1 epoch, before fine-tuning. With this approach, we were able to try more fancy things and that is also how we finally came to the two main architectures. \nWe further improved than this 2 step approach by using different learning rates for transformer and head, together with a warm up schedule, which enabled us to get rid of the freezing step in general. We illustrate the two main architectures in the following figure (For the sake of simplicity we show the architecture for the original 30 targets. It was adapted to work with the binarized targets.). \n
The upper architecture shows the dual transformer model, a combo model where one transformer handles the question text, while a second transformer handles the answer text. the output of the last layer is then averaged over the 512 tokens and both resulting tensors are concatenated. The resulting representation is then fed into 30 little 2 fully connected layers which result in the target predictions. 30 little heads enable each target to have its own head and gain individuality.
\n\nIt is inspired by the Elmo paper, where the final embedding representation is a weighted average of all LSTM layers. We use the output of every layer of a single transformer model in which we put 512 question and 512 answer tokens. For roberta-base that will give us twelve 512x768 tensors (for roberta large it would be 24 512x1024 tensors). We then average over the 512 tokens for each layer which results in twelve 768 representations. We then take a weighted sum of these 12 representations (where the weights for adding the representations are trainable!). This results in a 768 representation. The weighted average of all layer outputs enables to capture low level features in the final representation which was quite important for some answer related targets. Finally, we add a single prediction head for getting our targets.
\n\nOur final ensemble contains 5 models which belong to the one or the other architecture, and only differ in used pretrained backbone. \nWe ended up with \n- 2x dual roberta-base\n- dual roberta-large (2x 256 tokens)\n- dual xnet-base\n- siamese roberta-large with weighted averaged layers\nas this combination had the best cv score while fitting in the 2h kernel runtime requirement.
\n\nWe even trained one dual roberta-large model which on a V100 can only be trained when using a batch size of 1 and fp16. and was our best single model. Although this model could not fit into the ensemble due to runtime reasons, the promising cv results let us to a daring experiment: reducing the number of tokens used. There we saw something quite surprising. We could reduce the number of tokens to 256 for question as well as for answer without losing much quality.
\n\nMost models were trained with differential learning rate, where the transformer gets a lr of 3-e5 and the model head(s) a lr of 0.005 and used a cosine schedule for training 3 epochs including warmup over one epoch. Optimizer were either AdamW or RAdam with weight decay of 0.01 using an effective batchsize of 8 (gradient accumulation).
\n\nOur submission includes an equally weighted blend between all of above mentioned 5 models (leading to 25 test predictions with 5-fold). The blend is conducted on raw expected values calculated as described above. We experimented quite a bit with ranked blending as it seemed natural, but results were always worse.\nOur final blend is both our best one on CV, public LB and private LB meaning that we had a robust CV setup in the end and our selection was solid.
\n\nTraining was conducted offline and inference in kernel. Our final runtime was close to two hours and we had to spend some efforts to squeeze the 5 models in. All our models are implemented in Pytorch using the amazing Huggingface library. Fitting was done either locally or on cloud providers.
\n\nThanks for reading.
", "rawMarkdown": "Thanks to the sponsors and kaggle for hosting such an interesting and challenging competition! Our work is an amazing team effort and sprint over a very short time period of two weeks during which we made progress each day. \nWhy two weeks? Because some of us just finished the TF2 QA competition and decided to reuse whatever NLP knowledge that was acquired there. Thanks to all teammates: @christofhenkel @cpmpml @philippsinger @dott1718 @maxjeblick. \n\n**Please** also give them upvotes, we made a lottery roll of who will post the solution and everyone contributed equally to it.\n\nOur only regret is that the final score depends too heavily on the question_type_spelling column that has only 11 non zero target values in train data. Late submission experiments show we could easily improve, or degrade, our final rank by little tweaks on how we handle this column.\n\n## Brief Summary\nLike most teams, we used pretrained models from huggingface implemented in pytorch. Our final blend consists of 5 transformer models, that have several differences to push diversity. However, they all have in common to use 2 separate inputs of up to 512 tokens length representing questions respectively answers. These 2 inputs go either through the same transformer or through 2 separate transformers. We changed the targets to an ordinal representation (setting >target to 1) resulting in 170 target columns with each original column having n-unique target columns. Predictions were then generated by calculating the expected value.\n\nMost models were trained with differential learning rate, where the transformer gets a lr of 3-e5 and the model head(s) a lr of 0.005 and used a cosine schedule for training 3 epochs including warmup over one epoch. Optimizer were either AdamW or RAdam with weight decay of 0.01 using an effective batchsize of 8 (gradient accumulation).\n\n**Dual roberta-base:**\n- double roberta-base\n- added question title as input to answer-transformer\n- individual head for each target\n- mean of last layer output\n\n**Dual roberta-base:**\n- double roberta-base\n- added question title as input to answer-transformer\n- individual head for each target\n- mean of last layer output\n\n**Siamese roberta-base with softmax layer weight:**\n- double roberta-base\n- added question title as input to answer-transformer\n- individual head for each target\n- mean of last layer output\n\n**Dual roberta-large:**\n- only use 256 tokens for question / answer\n- double roberta-large\n- only fits in memory with batchsize 1 and fp16\n\n**Dual xlnet:**\n- same as dual roberta-base but with xlnet-base backbone\n- We used a simple average of probabiity predictions to ensemble our models. \n\n## Longer Summary\n\n### Binary Encoded Targets\n\nGiven the metric is rank-based, and given targets are not binary, it seemed important to be able to predict values that are neither 0 or 1 correctly. We tried mse loss and other variants but results were not satisfactory. We then decided to use binary cross-entropy with binary targets. The first try was to use on hot encoding of targets, given that targets have a small number of distinct values. This wasn’t satisfactory either, because this representation loses the ordering of values. \nWe ended up using an encoding of the form `(t > v)` for all v values of target t, except last value. For instance, if a target t has unique values `[0, ⅓, ⅔, 1]` then we would get 3 binary targets: `t > 0, t> ⅓`, and `t > ⅔`. Assuming we get perfect probability predictions `p(0), p(⅓ )`, and `p(⅔)` for the binary targets, we can compute predictions for each value of the original target as:\n```t(0)= 1 - p(0)\nt(⅓)= p(0) - p(⅓)\nt(⅔)= p(⅓) - p(⅔)\nt(1)= p(⅔) - 0```\nAnd the original target value as\n`t = 0 * t(0) + ⅓ * t(⅓) + ⅔ * t(⅔) + 1*t(1)`\nWe keep that computation with the actual predictions `p(0), p(⅓ )`, and `p(⅔)` for the binary targets. \nWhen the values are evenly spaced, as in this example, then the formula simplifies into:\n`t = mean(p(0), p(⅓ ), p(⅔))`\nIf we assume that what matters is only the order of values, then we can stick to this simplified form.\nWe used the simplified form throughout the competition as it was simpler to code, and faster. In our final two selected subs we reverted to the exact computation because it was better on both cv and public LB, but this didn’t change private LB significantly (0.0004 difference). \n\n### Validation scheme\n\nSimilar to most teams, we use a 5-fold GroupKFold on question body for fitting our models and evaluation. After a bunch of experiments we saw though, that this is not sufficient mainly due to the following aspects:\n- Test data is different to training data in the sense that it only has one question-answer pair sampled out of a group of questions in train data. There can be stark noise for labels of the same question, which is why this needs to be addressed robustly.\n- There are a few columns with very rare events and also a lot of noise within those rare events. The prime example is spelling which has a huge impact on CV and LB, but can completely blind your view when trying to judge the overall strength of your model.\n\nThat is why we settled after some time on the following full validation scheme:\n- Use 5-fold GroupKFold on question body\n- For each validation fold, sample 100 times randomly a single question-answer pair out of multiple questions.\n- Calculate the median score across these 100 samples and report.\n- Ignore spelling column.\n- Final CV is a mean of 5 folds.\nThis setup also allowed us to properly test any type of postprocessing in a realistic manner as described next.\n\n### Postprocessing\n\nEven though we had really strong models, post-processing was very important to us and we spent significant time on it to find a robust way that we can trust. It is based on threshold clipping. \nFor each target column, we separately attempt to find optimal thresholds from both sides (starting from lowest predictions and highest predictions) based on which we clip the data. So let’s assume our predictions look like `x=[0.01, 0.015, 0.02, 0.03, 0.04]` and our optimal thresholds are `coefs=[0.016, 0.029]` then we would clip the data with `np.clip(x, coefs[0], coefs[1])` leading to `x=[0.016, 0.016, 0.02, 0.029, 0.029]` effectively generating ties at the edges.\n\nThe CV setup from above gave us a perfect way to fully validate any of our approaches by simply generating the thresholds on training folds, and applying them on truncated samples of the validation folds. For final PP, we get the thresholds on full oof and apply them on the test set.\nIn a nutshell, our routine looks like the following:\n- Sample 1000 times single question-answer pairs from multiple questions\n- Generate thresholds that optimize the median score of all 1000 samples\n\nWe tested quite a few different strategies, but this one was the most robust one we found and we are quite happy with it. Only for spelling, it was still a bit shaky, which is why our two final subs just differ in how we handle spelling column, one uses the calculated thresholds, and one hard-sets the 6 highest predictions in private LB to 1 and rest to 0 which is based on experiments on samples.\nUnfortunately, if we would have post-processed all columns except spelling and would have kept spelling as is, we would have reached 0.432 on private LB as apparently spelling is very differently distributed on private. No CV experiment would have let us make this decision though. It would have been so much better though to not include this column into the competition.\n\n### Architectures and training models\n\nSimilar to other recent competitions at first it was quite difficult to beat public kernels, which we normally see as a kind of baseline. All “normal” tricks that worked in past computer vision or other NLP competitions (e.g. concat of Max and Mean pooling) did not improve cv. Also using a sliding window approach to capture more text did not work.\nWe saw a first big improvement when using 2 transformers instead of one (+0.01), but still, any slightly more complex architectures lead to a worse result. That changed when we tried to freeze the transformers and only train the model head for 1 epoch, before fine-tuning. With this approach, we were able to try more fancy things and that is also how we finally came to the two main architectures. \nWe further improved than this 2 step approach by using different learning rates for transformer and head, together with a warm up schedule, which enabled us to get rid of the freezing step in general. We illustrate the two main architectures in the following figure (For the sake of simplicity we show the architecture for the original 30 targets. It was adapted to work with the binarized targets.). \n\n\n### Dual Transformer: \nThe upper architecture shows the dual transformer model, a combo model where one transformer handles the question text, while a second transformer handles the answer text. the output of the last layer is then averaged over the 512 tokens and both resulting tensors are concatenated. The resulting representation is then fed into 30 little 2 fully connected layers which result in the target predictions. 30 little heads enable each target to have its own head and gain individuality.\n\n### Siamese Transformer with soft weighted layers:\nIt is inspired by the *Elmo paper*, where the final embedding representation is a weighted average of all LSTM layers. We use the output of every layer of a single transformer model in which we put 512 question and 512 answer tokens. For roberta-base that will give us twelve 512x768 tensors (for roberta large it would be 24 512x1024 tensors). We then average over the 512 tokens for each layer which results in twelve 768 representations. We then take a weighted sum of these 12 representations (where the weights for adding the representations are trainable!). This results in a 768 representation. The weighted average of all layer outputs enables to capture low level features in the final representation which was quite important for some answer related targets. Finally, we add a single prediction head for getting our targets. \n\n\nOur final ensemble contains 5 models which belong to the one or the other architecture, and only differ in used pretrained backbone. \nWe ended up with \n- 2x dual roberta-base\n- dual roberta-large (2x 256 tokens)\n- dual xnet-base\n- siamese roberta-large with weighted averaged layers\nas this combination had the best cv score while fitting in the 2h kernel runtime requirement. \n\nWe even trained one dual roberta-large model which on a V100 can only be trained when using a batch size of 1 and fp16. and was our best single model. Although this model could not fit into the ensemble due to runtime reasons, the promising cv results let us to a daring experiment: reducing the number of tokens used. There we saw something quite surprising. We could reduce the number of tokens to 256 for question as well as for answer without losing much quality. \n\nMost models were trained with differential learning rate, where the transformer gets a lr of 3-e5 and the model head(s) a lr of 0.005 and used a cosine schedule for training 3 epochs including warmup over one epoch. Optimizer were either AdamW or RAdam with weight decay of 0.01 using an effective batchsize of 8 (gradient accumulation).\n\n### Blending\nOur submission includes an equally weighted blend between all of above mentioned 5 models (leading to 25 test predictions with 5-fold). The blend is conducted on raw expected values calculated as described above. We experimented quite a bit with ranked blending as it seemed natural, but results were always worse.\nOur final blend is both our best one on CV, public LB and private LB meaning that we had a robust CV setup in the end and our selection was solid.\n\n### Wrapping things up and putting into kernel\nTraining was conducted offline and inference in kernel. Our final runtime was close to two hours and we had to spend some efforts to squeeze the 5 models in. All our models are implemented in Pytorch using the amazing Huggingface library. Fitting was done either locally or on cloud providers. \n\n\nThanks for reading.\n\n\n", "votes": 112 }, { "id": 742868, "postDate": "2020-02-11T15:54:46.687Z", "content": "I really like your binary encoding trick. I tried something similar and it nearly performed as well as post processing. For each target, you predict 3 values.
\n\nfor i in range(3):\n x = Dense(30, activation='sigmoid')(bert)\n heads.append(x)\nmodel = Model(inputs=inputs, outputs=heads)\nAnd you train with
\n\nmodel.fit(inputs, [targets, (targets==1).astype(int), (targets==0).astype(int)])\nThen your final predictions are
\n\npreds, t1, t0 = model.predict(inputs)\npreds[t1>0.8] = 1\npreds[t0>0.8] = 0\nNice, this is a simplified version of what we used, but it may perform as well in practice. Thanks for sharing.
", "rawMarkdown": "Nice, this is a simplified version of what we used, but it may perform as well in practice. Thanks for sharing.", "votes": 2 } ] }, { "id": 742803, "postDate": "2020-02-11T14:54:57.267Z", "content": "Really cool sprint. And to be honest, after TF2.0QA I thought I already know everything about transformers. Quite the opposite - I learned a lot in the short time. Thank you all for that.
", "rawMarkdown": "Really cool sprint. And to be honest, after TF2.0QA I thought I already know everything about transformers. Quite the opposite - I learned a lot in the short time. Thank you all for that.", "votes": 12 }, { "id": 742852, "postDate": "2020-02-11T15:40:16.443Z", "content": "Great job Max, Psi, CPMP, Dieter, Dott. Awesome models.
\n\nI want to make sure I understand your architecture. For your dual transformer (original 30 targets) are the last layers the following?
\n\nx = Concatenate([bert_1,bert_2])\nheads = []\nfor i in range(30):\n h = Dense(64, activation='tanh')(x)\n h = Dense(1, activation='sigmoid')(h)\n heads.append(h)\nmodel = Model(inputs=inputs, outputs=heads)\nAnd your Siamese is
\n\npoolings = []\nfor i in range(12):\n embeddings = bert[2][i]\n x = GlobalAveragePooling1D(embeddings)\n poolings.append(x)\nstacked = Lambda(lambda x: K.stack(x, axis=-1))(poolings)\nweighted = Dense(1, use_bias=False)(stacked)\nweighted = Lambda(lambda x: K.squeeze(x, axis=-1))(weighted)\nheads = Dense(64, activation='tanh')(weighted)\nheads = Dense(30, activation='sigmoid')(x)\nmodel = Model(inputs=inputs, outputs=heads)\nYes, in principle. We left out some details like activation function, dropout and layernorm though.
", "rawMarkdown": "Yes, in principle. We left out some details like activation function, dropout and layernorm though. ", "votes": 3 } ] }, { "id": 742822, "postDate": "2020-02-11T15:04:36.967Z", "content": "Congrats to the winners and thanks a lot to the amazing teammates!
", "rawMarkdown": "Congrats to the winners and thanks a lot to the amazing teammates!", "votes": 7 }, { "id": 742790, "postDate": "2020-02-11T14:43:13.063Z", "content": "I was so happy to be on that dream team, learned a lot, had fun, and got a great result!
", "rawMarkdown": "I was so happy to be on that dream team, learned a lot, had fun, and got a great result! ", "votes": 8 }, { "id": 742778, "postDate": "2020-02-11T14:36:31.373Z", "content": "Thanks again for the great team work and I am super stoked about what we could achieve in such a team in such a short time period of two weeks.
", "rawMarkdown": "Thanks again for the great team work and I am super stoked about what we could achieve in such a team in such a short time period of two weeks.", "votes": 6 }, { "id": 742993, "postDate": "2020-02-11T17:27:54.983Z", "content": "Dream team indeed...congrats guys, well earned!
", "rawMarkdown": "Dream team indeed...congrats guys, well earned!", "votes": 3 }, { "id": 742956, "postDate": "2020-02-11T17:00:49.677Z", "content": "Thank you for such a detailed writeup! Congrats on your result guys :)
", "rawMarkdown": "Thank you for such a detailed writeup! Congrats on your result guys :)", "votes": 3, "replies": [ { "id": 742968, "postDate": "2020-02-11T17:07:59.290Z", "content": "I am amazed how similar our models are. Weighted sum of all layers and differential learning rate :)
", "rawMarkdown": "I am amazed how similar our models are. Weighted sum of all layers and differential learning rate :)", "votes": 3 } ] }, { "id": 742854, "postDate": "2020-02-11T15:42:20.643Z", "content": "your solution is so similar to ours, The 2 differences is that we used 3 transformers instead of 2 + we backpropagate in every transformer using 4 losses separately ( loss1(answers_target, output from bert_answers) , loss2(question_targets, output from bert_question), loss3(targets, output from bert3 which combines answer+question), loss4 is final loss after concatenating all logits from different transofmers))\nAn epoch takes 40 minutes on a V100 since we're not destroying the graph after each backward().\nCongrats you did a good job!
", "rawMarkdown": "your solution is so similar to ours, The 2 differences is that we used 3 transformers instead of 2 + we backpropagate in every transformer using 4 losses separately ( loss1(answers_target, output from bert_answers) , loss2(question_targets, output from bert_question), loss3(targets, output from bert3 which combines answer+question), loss4 is final loss after concatenating all logits from different transofmers))\nAn epoch takes 40 minutes on a V100 since we're not destroying the graph after each backward().\nCongrats you did a good job!", "votes": 3, "replies": [ { "id": 742926, "postDate": "2020-02-11T16:38:46.557Z", "content": "Congrats to you as well. We tried the same 3 berts as you and did not find it to help. Maybe we didn't try hard enough ;) Congrats on your result too!
", "rawMarkdown": "Congrats to you as well. We tried the same 3 berts as you and did not find it to help. Maybe we didn't try hard enough ;) Congrats on your result too!", "votes": 1 } ] }, { "id": 744517, "postDate": "2020-02-12T22:38:09.570Z", "content": "Wow, I just read how your private score would have been 0.432 if you didn't post process question_type_spelling column. That's a bummer to figure that out now. I agree that that was a poor column to add when the competition metric is Spearman's rank correlation, because just a few correct and incorrect predictions radically change that column's score.
Indeed, this is the lottery part of the competition.
", "rawMarkdown": "Indeed, this is the lottery part of the competition.", "votes": -1 } ] }, { "id": 743753, "postDate": "2020-02-12T09:14:09.680Z", "content": "Congratulations! Excellent solution.
", "rawMarkdown": "Congratulations! Excellent solution.", "votes": 1 }, { "id": 743003, "postDate": "2020-02-11T17:43:24.170Z", "content": "Congrats to dream team!! Thanks for the detailed write up!
", "rawMarkdown": "Congrats to dream team!! Thanks for the detailed write up! ", "votes": 1 }, { "id": 742842, "postDate": "2020-02-11T15:27:43.457Z", "content": "Congratulations! good solution
", "rawMarkdown": "Congratulations! good solution", "votes": 1 }, { "id": 744196, "postDate": "2020-02-12T16:42:18.087Z", "content": "Guys, did you try pretraining with additional SO/SX data? I'm a bit surprised (and humbled) with such a significant gap and the fact that no teams actually used pretraining (as far as I analyzed a bunch of top solutions).
\n\nThis was one of our first ideas, and by the time I collected those ruddy TF records to fine-tune a couple of MLMs with TPUs and 7kk SX records, our would-be teammate @dmitriyab came up with a finetuned model with auxiliary targets which made the whole difference. Well, it actually needs a careful ablation study (no time for that) but 99% that it's about that pretrained model.
\n\nDmitry is going to share his approach tonight, with an additional post, our code is already shared. So again, all kudos go to Dmitry, I'm just curious whether you considered finetuning as well.
", "rawMarkdown": "Guys, did you try pretraining with additional SO/SX data? I'm a bit surprised (and humbled) with such a significant gap and the fact that no teams actually used pretraining (as far as I analyzed a bunch of top solutions). \n\nThis was one of our first ideas, and by the time I collected those ruddy TF records to fine-tune a couple of MLMs with TPUs and 7kk SX records, our would-be teammate @dmitriyab came up with a finetuned model with auxiliary targets which made the whole difference. Well, it actually needs a careful ablation study (no time for that) but 99% that it's about that pretrained model. \n\nDmitry is going to share his approach tonight, with an additional post, our code is already shared. So again, all kudos go to Dmitry, I'm just curious whether you considered finetuning as well. ", "votes": 2, "replies": [ { "id": 744220, "postDate": "2020-02-12T17:03:06.817Z", "content": "Using the SO data was the first thing I tried. But it gave no improvement to our score and would be quite some effort, so in light of the tight timeline I discarded the idea.
", "rawMarkdown": "Using the SO data was the first thing I tried. But it gave no improvement to our score and would be quite some effort, so in light of the tight timeline I discarded the idea.", "votes": 1 }, { "id": 744228, "postDate": "2020-02-12T17:07:23.940Z", "content": "I see, thanks. Maybe indeed a regular MLM is not as beneficial, and it's auxiliary targets that make the difference. Probably we'll find time for a short ablation study.
", "rawMarkdown": "I see, thanks. Maybe indeed a regular MLM is not as beneficial, and it's auxiliary targets that make the difference. Probably we'll find time for a short ablation study. ", "votes": 2 } ] }, { "id": 1064621, "postDate": "2020-10-30T10:44:03.060Z", "content": "Thanks for sharing your ideas
\n\n\nWe ended up using an encoding of the form (t > v) for all v values of target t, except last value. For instance, if a target t has unique values [0, ⅓, ⅔, 1] then we would get 3 binary targets: t > 0, t> ⅓, and t > ⅔. Assuming we get perfect probability predictions p(0), p(⅓ ), and p(⅔) for the binary targets, we can compute predictions for each value of the original target as:
\n
\n t(0)= 1 - p(0) t(⅓)= p(0) - p(⅓) t(⅔)= p(⅓) - p(⅔) t(1)= p(⅔) - 0
\n And the original target value as
\n t = 0 * t(0) + ⅓ * t(⅓) + ⅔ * t(⅔) + 1*t(1)
\n We keep that computation with the actual predictions p(0), p(⅓ ), and p(⅔) for the binary targets.
\n When the values are evenly spaced, as in this example, then the formula simplifies into:
\n t = mean(p(0), p(⅓ ), p(⅔))
@christofhenkel @cpmpml @philippsinger @dott1718 @maxjeblick ,
\nI am having difficulty understanding this concept, how did you arrive at these formulas, would it be possible to explain it little more
1) t(⅓)= p(0) - p(⅓), how does t > 1/3 = p(0) - p(⅓), tried figuring out , not able to get it
\n2) t = 0 * t(0) + ⅓ * t(⅓) + ⅔ * t(⅔) + 1*t(1) , is this similar to expected value calculation ?
\n3) t = mean(p(0), p(⅓ ), p(⅔)) , how does the formula simplify to this
Is there any other place where this is explained ? I hope these are not too naive questions
\nThanks in advance
", "rawMarkdown": "Thanks for sharing your ideas\n\n> We ended up using an encoding of the form (t > v) for all v values of target t, except last value. For instance, if a target t has unique values [0, ⅓, ⅔, 1] then we would get 3 binary targets: t > 0, t> ⅓, and t > ⅔. Assuming we get perfect probability predictions p(0), p(⅓ ), and p(⅔) for the binary targets, we can compute predictions for each value of the original target as:\nt(0)= 1 - p(0) t(⅓)= p(0) - p(⅓) t(⅔)= p(⅓) - p(⅔) t(1)= p(⅔) - 0\nAnd the original target value as\nt = 0 * t(0) + ⅓ * t(⅓) + ⅔ * t(⅔) + 1*t(1)\nWe keep that computation with the actual predictions p(0), p(⅓ ), and p(⅔) for the binary targets.\nWhen the values are evenly spaced, as in this example, then the formula simplifies into:\nt = mean(p(0), p(⅓ ), p(⅔))\n\n@christofhenkel @cpmpml @philippsinger @dott1718 @maxjeblick , \nI am having difficulty understanding this concept, how did you arrive at these formulas, would it be possible to explain it little more \n\n1) t(⅓)= p(0) - p(⅓), how does t > 1/3 = p(0) - p(⅓), tried figuring out , not able to get it \n2) t = 0 * t(0) + ⅓ * t(⅓) + ⅔ * t(⅔) + 1*t(1) , is this similar to expected value calculation ? \n3) t = mean(p(0), p(⅓ ), p(⅔)) , how does the formula simplify to this \n\nIs there any other place where this is explained ? I hope these are not too naive questions\n\nThanks in advance ", "replies": [ { "id": 1064714, "postDate": "2020-10-30T12:51:23.277Z", "content": "Let me answer the first question, hopefully you'll get how it works with it.
\np(0) is the prediction probability that the actual target t is such that t > 0.
\nThe probability that t = 0 is t(0) and the probability that t > 0 is p(0).
\nBy definition t = 0 or t > 0 then t(0) + p(0) = 1
Similarly, the probability that t > 0 is p(0), the probability that t > ⅓ is p(⅓), and the probability that t = ⅓ is t(⅓)
\nt > 0 if and only if t > ⅓ or t = ⅓
\nThen we have that p(0) = p(⅓) + t(⅓)
\nand so on.
\nThis way of encoding ordinal target has been published but I don't have a handy reference.
", "rawMarkdown": "Let me answer the first question, hopefully you'll get how it works with it.\n\np(0) is the prediction probability that the actual target t is such that t > 0.\n\nThe probability that t = 0 is t(0) and the probability that t > 0 is p(0). \nBy definition t = 0 or t > 0 then t(0) + p(0) = 1\n\nSimilarly, the probability that t > 0 is p(0), the probability that t > ⅓ is p(⅓), and the probability that t = ⅓ is t(⅓)\n\nt > 0 if and only if t > ⅓ or t = ⅓\n\nThen we have that p(0) = p(⅓) + t(⅓)\n\nand so on.\n\nThis way of encoding ordinal target has been published but I don't have a handy reference.", "votes": 2 }, { "id": 1064881, "postDate": "2020-10-30T16:25:25.347Z", "content": "thanks @cpmpml
", "rawMarkdown": "thanks @cpmpml " } ] }, { "id": 750015, "postDate": "2020-02-19T04:12:35.747Z", "content": "Great work, I am totally impressed. Hats off !\nGreat community contribution as well.
", "rawMarkdown": "Great work, I am totally impressed. Hats off !\nGreat community contribution as well." }, { "id": 744772, "postDate": "2020-02-13T06:06:26.777Z", "content": "Thanks a lot for sharing! I have a question about clipping.
\ngenerating ties at the edges Did you guys do anything to generate ties in between the edges?
We tried, and we didn't found anything really working. We haven't tried rounding using estimated number of annotators though. Certainly worth a try.
", "rawMarkdown": "We tried, and we didn't found anything really working. We haven't tried rounding using estimated number of annotators though. Certainly worth a try." } ] }, { "id": 744743, "postDate": "2020-02-13T05:08:58.127Z", "content": "Congrats, a very interesting solution!
", "rawMarkdown": "Congrats, a very interesting solution!" }, { "id": 744546, "postDate": "2020-02-12T23:27:52.267Z", "content": "Congratulations and thank you for precise write up. learned a lot.
", "rawMarkdown": "Congratulations and thank you for precise write up. learned a lot." }, { "id": 744540, "postDate": "2020-02-12T23:17:52.973Z", "content": "Awesome solution, congratulations!
", "rawMarkdown": "Awesome solution, congratulations!" }, { "id": 744491, "postDate": "2020-02-12T22:02:19.110Z", "content": "I just realized something. I wanted to make a dual transformer like you guys did. I used the code from the popular public Huggingface notebook here. (This code is simplied by removing attention mask, and work token inputs).
\n\ndef create_model():\n q_id = tf.keras.layers.Input((512,), dtype=tf.int32)\n a_id = tf.keras.layers.Input((512,), dtype=tf.int32)\n\n bert_model = TFBertModel.from_pretrained('bert-base-uncased') \n\n q_embedding = bert_model(q_id)[0]\n a_embedding = bert_model(a_id)[0]\n\n q = tf.keras.layers.GlobalAveragePooling1D()(q_embedding)\n a = tf.keras.layers.GlobalAveragePooling1D()(a_embedding)\n\n x = tf.keras.layers.Concatenate()([q, a])\n x = tf.keras.layers.Dense(30, activation='sigmoid')(x)\n\n model = tf.keras.models.Model(inputs=[q_id, a_id], outputs=x)\n\nreturn model\nUpon refection, this doesn't use 2 BERTs does it? It seems that 1 BERT is acting similar to a \"convolution filter with shared weights\". This 1 BERT acts on input1 and it acts on input2 producing \"2 channels\" and the back propagation update has a contribution from each input flowing through the model. And that updates 1 set of weights.
\n\nDid you build your dual transformer in PyTorch or TensorFlow?
\n\nUPDATE: This post was originally a question, but I think i figured it out.
", "rawMarkdown": "I just realized something. I wanted to make a dual transformer like you guys did. I used the code from the popular public Huggingface notebook [here][1]. (This code is simplied by removing attention mask, and work token inputs).\n\n def create_model():\n q_id = tf.keras.layers.Input((512,), dtype=tf.int32)\n a_id = tf.keras.layers.Input((512,), dtype=tf.int32)\n \n bert_model = TFBertModel.from_pretrained('bert-base-uncased') \n \n q_embedding = bert_model(q_id)[0]\n a_embedding = bert_model(a_id)[0]\n \n q = tf.keras.layers.GlobalAveragePooling1D()(q_embedding)\n a = tf.keras.layers.GlobalAveragePooling1D()(a_embedding)\n\n x = tf.keras.layers.Concatenate()([q, a])\n x = tf.keras.layers.Dense(30, activation='sigmoid')(x)\n\n model = tf.keras.models.Model(inputs=[q_id, a_id], outputs=x)\n \n return model\n\nUpon refection, this doesn't use 2 BERTs does it? It seems that 1 BERT is acting similar to a \"convolution filter with shared weights\". This 1 BERT acts on input1 and it acts on input2 producing \"2 channels\" and the back propagation update has a contribution from each input flowing through the model. And that updates 1 set of weights.\n\nDid you build your dual transformer in PyTorch or TensorFlow?\n\nUPDATE: This post was originally a question, but I think i figured it out.\n\n[1]: https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer", "replies": [ { "id": 744502, "postDate": "2020-02-12T22:20:28.830Z", "content": "What you are showing here is exactly our second type of model that we call \"siamese transformer\" in the figure above. Basically you just feed question and answer twice through the same transformer. In order to get a full dual transformer model, you need to define it separately:
\n\n\nbert_model_q = TFBertModel.from_pretrained('bert-base-uncased') \nbert_model_a = TFBertModel.from_pretrained('bert-base-uncased') \n
Thanks for the reply. As you were typing, I just figured it out and was updating my post.
\n\nWhich model had higher CV, your dual transformer or siamese transformer?
", "rawMarkdown": "Thanks for the reply. As you were typing, I just figured it out and was updating my post.\n\nWhich model had higher CV, your dual transformer or siamese transformer? " }, { "id": 744734, "postDate": "2020-02-13T04:48:20.240Z", "content": "roughly the same. However they had different strength. Especially the usage of all layers for the siamese one improved diversity.
", "rawMarkdown": "roughly the same. However they had different strength. Especially the usage of all layers for the siamese one improved diversity." } ] }, { "id": 742872, "postDate": "2020-02-11T16:02:59.123Z", "content": "Well done all!! Do you plan on releasing your training code? Would love to step through it line by line and learn a little more :)
", "rawMarkdown": "Well done all!! Do you plan on releasing your training code? Would love to step through it line by line and learn a little more :)", "replies": [ { "id": 742874, "postDate": "2020-02-11T16:07:41.527Z", "content": "We'd love to, but @abhishek asked everybody not to share code
", "rawMarkdown": "We'd love to, but @abhishek asked everybody not to share code\n\n", "votes": 7 }, { "id": 742977, "postDate": "2020-02-11T17:17:02.100Z", "content": "could somebody explain to me this joke?
", "rawMarkdown": "\n\ncould somebody explain to me this joke?" }, { "id": 742984, "postDate": "2020-02-11T17:20:00.763Z", "content": "https://www.kaggle.com/c/google-quest-challenge/discussion/129850
", "rawMarkdown": "https://www.kaggle.com/c/google-quest-challenge/discussion/129850", "votes": 3 }, { "id": 743783, "postDate": "2020-02-12T10:00:38.440Z", "content": "Haha hes a tough guy
", "rawMarkdown": "Haha hes a tough guy" } ] }, { "id": 752625, "postDate": "2020-02-21T08:21:39.837Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!\n", "votes": 2 }, { "id": 755990, "postDate": "2020-02-25T10:33:35.107Z", "content": "Thank you for Sharing!!
", "rawMarkdown": "Thank you for Sharing!!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:08:12Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4018152, "solution_url": "/competitions/google-quest-challenge/writeups/berts-and-the-holy-grail-two-berts-are-better-than" }