My model was based on fine-tuning BERT-large with rudimentary output layer and no additional statistical features. The code in run_classifier.py() provided with BERT was modified to achieve this. I will post the full code here once I've neatened it up, but for now an overview of the main elements.
For the input layer, I sorted the 3 char offsets and stored the corresponding permutation of P, A, B and split the text into 4 segments at these offsets. Each segment was Wordpiece tokenized and the token segments were truncated to match the required sequence length. The choice of truncating algorithim made a big difference to the results. The token offsets were calculated as the cumulative sum of tokens in each segment. The token offsets were sorted back into P, A, B order and used to make binary mask “features” for the positions of P, A, B.
\n\nThe output from BERT was masked to retrieve the 1024 dimensional vectors for P, A, B. Three new “probability embeddings” were computed as (element-wise multiplication):
\n\nThe idea of the first two was to represent the similarity between P&A or P&B as abstract vector embeddings. For ‘Neither” the abstract embedding in 3 is supposed to represent the extent to which A&B are similar to each other, but differ from P. These embeddings (1024 dim vectors) were then reduced to a scalar probability by a trainable (1024, 1) tensor. The same tensor was used for the probs of A and B to prevent the model learning from whether “True” or “False” appeared first, as my own pseudo-data (not used in the end) was somewhat unbalanced in this regard. A seperate trainable tensor was applied to (AB – PP).
\n\nThis architecture on a 64 token sequence gave a stage 1 score (corrected gap data) of 0.30 to 0.38 (single model), which ensembled to 0.275 by simple averaging of the predictions from 6 models. I assume the reason for the range of results has to do with the architecture being a bit unstable with respect to random initialisation.
\n\nWithin run_classifier.py(), changes were made to the input methods to suite the GAP data. A detailed explanation is done via this short kernel, which illustrates how a single example is prepared for the input layer. Pay particular attention to the way text was truncated to fit the chosen maximum token sequence. Changes to this algorithm made a big difference to results. I think that doing this truncating at a sentence level might bring further gains.
Vanilla BERT-large.
\n\nUnfortunately, the 1024 dimensional vectors make it impractical to do a similar demonstration of the output. Instead I will give a short explanation of some of the code.
\n\nFirstly, the output was extracted and masked to get the P, A, B embeddings.
\n\n\nall_out = model.get_sequence_output()\nP = tf.boolean_mask(all_out, P_mask)\nA = tf.boolean_mask(all_out, A_mask)\nB = tf.boolean_mask(all_out, B_mask)\n
This worked fine on a GPU, but tf.boolean_mask() is not implemented for TPU, which I stepped up to for BERT-large. It took a good few hours to figure out a messy workaround, but it did the job.
The probability embeddings were then calculated as explained above.
\n\n\nPA = tf.multiply(P, A)\nPB = tf.multiply(P, B)\nPP = tf.multiply(P, P)\nAB = tf.multiply(A, B)\nN = tf.subtract(PP, AB)\n
And finally reduced to probabilities by trainable tensors and a softmax.
\n\n```\nAB_weights = tf.get_variable(\n \"AB_weights\", [1, hidden_size],\n initializer=tf.truncated_normal_initializer(stddev=0.02))
\n\nN_weights = tf.get_variable(\n \"N_weights\", [1, hidden_size],\n initializer=tf.truncated_normal_initializer(stddev=0.02))
\n\nA_out = tf.matmul(PA, AB_weights, transpose_b=True)\nB_out = tf.matmul(PB, AB_weights, transpose_b=True)\nN_out = tf.matmul(N, N_weights, transpose_b=True)
\n\noutput_bias = tf.get_variable(\n \"output_bias\", [num_labels], initializer=tf.zeros_initializer())
\n\nlogits = tf.concat([A_out, B_out, N_out], axis=1)\nlogits = tf.nn.bias_add(logits, output_bias)\nprobabilities = tf.nn.softmax(logits, axis=-1)\nlog_probs = tf.nn.log_softmax(logits, axis=-1)\n``\nThat was about it. The rest of the code was pretty routine or already included inrun_classifier.py`.
Fantastic! Simple but effective. Thanks for sharing. The prob embeddings were genius. Bit-wise multiplication is underrated.
", "votes": 1, "replies": [ { "id": 523345, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-04-26T03:43:40.457000", "content": "Actually, I should have said element-wise multiplication, as in tf.multiply(). Apologies for the error. I will correct it in the post.
", "votes": 0, "replies": [] }, { "id": 523401, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-04-26T07:05:19.897000", "content": "The idea of the element-wise multiplication was to achieve something similar to a dot product, as described in the original BERT Paper. If the trainable output tensor were all 1s it would be exactly dot product.
\n\nP dot A would give some measure of how similar the embeddings of the pronoun and name A are.
", "votes": 1, "replies": [] }, { "id": 523671, "author_name": "Kepler456b", "author_url": "", "post_date": "2019-04-26T17:52:06.157000", "content": "I thought bit-wise was the DS slang for element-wise :) If the rows are summed up it indeed is the dot product. This could be done in hidden layers too. From my understanding Attention achieves its results by making use of multiplication, whereas in vanilla nets, outputs of hidden layers do not get multiplied. I am also curious how truncating in the middle of sentences didn’t reduce the score much.
\n\nBy the way, I am in the process of publishing a paper on AGI: solving word problems. It just needs a review and a few suggestions. I would be happy if you would like to take a look or join as an author. Cheers.
", "votes": 1, "replies": [] }, { "id": 523674, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-04-26T18:02:26.253000", "content": "@isikkuntay I would be very happy to take a look, but I'm a newcomer to this field so might not be able to make much of a contribution.
", "votes": 0, "replies": [] }, { "id": 523768, "author_name": "Kepler456b", "author_url": "", "post_date": "2019-04-27T01:15:06.710000", "content": "Thanks! Here is the link to the paper: https://github.com/isikkuntay/AIND/blob/master/updated%20CRN_paper-04-22-19.pdf
\n\nNew comer or not, come to think of it, you did 3x better than a group of million dollar google engineers in your first ML project :) Didn’t make you principal for nothing ;))
", "votes": 0, "replies": [] } ] }, { "id": 523247, "author_name": "LuEE", "author_url": "", "post_date": "2019-04-25T20:22:36.137000", "content": "Thanks for sharing! So if I understand correctly what made the biggest difference for you was the truncation?
", "votes": 1, "replies": [ { "id": 523346, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-04-26T03:51:44.567000", "content": "That certainly made a big difference. Curiously, shorter sequences, all the way down to about 64 tokens, gave me the best single model results. Instinctively I had assumed that retaining as much information as possible would give the best results. The only explanation I can think of is that tokens close together influence each other's embeddings more, giving stronger relationship between P, A, B in the output layer. I need help in understanding this and would love to hear alternative explanations.
", "votes": 1, "replies": [] }, { "id": 523582, "author_name": "Sandeep Attree", "author_url": "", "post_date": "2019-04-26T14:58:53.910000", "content": "@kenkrige even today, sota language models suffer from noise in long-range dependencies. Add to that, the noise inherent in text.
\n\nIn case of the GPR problem, there are only very few examples that require long-range context pooling. You may have made these worse by breaking the syntactic structure while truncating. If the truncation retains phrasal structure, then it won't be as detrimental. As you mentioned, truncating at a sentence-level will be better, since a sentence has self-contained structure.
\n\nFor the remaining majority examples where the model is correct, you should see more confident probabilities for the shortened version compared to the original text version. Hence, a bump in the overall score.
", "votes": 2, "replies": [] }, { "id": 523651, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-04-26T17:16:04.873000", "content": "Thanks @sattree. I suppose your coref pooling based on other models would have shortened the dependencies thereby reducing some of that noise. Well done again on your insightful solution.
", "votes": 0, "replies": [] } ] }, { "id": 524614, "author_name": "Tania J", "author_url": "", "post_date": "2019-04-29T07:36:50.373000", "content": "Congrats on the gold and thanks for sharing. I'm looking forward to seeing the full code.
", "votes": 0, "replies": [ { "id": 525814, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-05-01T19:00:25.433000", "content": "@taniaj sorry about the delay. I was planning to polish my code but time has not been on my side. There is not much of a readme so feel free to ask questions if you have problems running it.
", "votes": 3, "replies": [] }, { "id": 526087, "author_name": "Tania J", "author_url": "", "post_date": "2019-05-02T10:23:56.003000", "content": "No worries at all. I got it running (with reduced batch size) on my GPUs but the result wasn't great. I guess you ended up using a Colab TPU for the final result? Your code is very clean and readable but my Tensorflow knowledge is lacking so I'm using this as motivation to learn it well enough to understand your code. Thank you!
", "votes": 0, "replies": [] }, { "id": 526117, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-05-02T11:33:54.557000", "content": "I'm also new to tensorflow and to machine learning as a whole, which is why started with their run_cassifier.py and modified that. I was very lucky to do so well in this competition.
I'm glad you got it running. I found large batch sizes helped results a lot, so did using BERT large instead of BERT base. For both of those you will need a TPU I think. I didn't use colab, I used google cloud platform. When you sign up, they give you one year and $300 free. A preemptible TPU is about $1.40 per hour and each training run only takes about 6 min for that data. There are a few configuration tricks, because you have to use google storage gs://your_bucket/... for all your files. Please ask if you get stuck.
Another thing I learned was that some bad logloss results don't really matter as long as the accuracy is good. Then when you average the output csv files from a few different training runs and clip the results that are too close to 0 or 1 then the logloss improves drastically. Two training runs with the same hyper-parameters can give significantly different log-loss results, but prediction accuracy is quite consistent. I got about 85% accuracy from BERT base and 89% for BERT large. I just did the averaging in pandas afterwards.
", "votes": 2, "replies": [] }, { "id": 526120, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-05-02T11:37:28.040000", "content": "By the way, @taniaj thanks again for sharing your data corrections. Those also helped my results.
", "votes": 1, "replies": [] }, { "id": 526290, "author_name": "Kepler456b", "author_url": "", "post_date": "2019-05-02T18:17:58.970000", "content": "@kenkrige Thanks for taking time off your holiday to post the code.
", "votes": 0, "replies": [] }, { "id": 526571, "author_name": "Tania J", "author_url": "", "post_date": "2019-05-03T09:45:45.920000", "content": "I got it running with BERT-Large on Google Cloud TPU. :) Private LB score was 0.26718 but I didn't play with any parameters at all.
\n\nI do have a question about how you do the token segment truncation though. If I understand your code correctly it looks like you are allowing the middle 2 segments to be larger than the first and last (2x as large) and truncating from the middle of the middle segments instead of the end. Is there a particular reason for doing this instead of allowing all segments to be equally large and always truncating from the end (except for the first segment)?
", "votes": 0, "replies": [] }, { "id": 526588, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-05-03T10:29:54.737000", "content": "Well done! That's an excellent score. If you run training again with exactly the same parameters, you might get a worse (or better) logloss score. For my final leaderboard score 0f 0.19, I used 3 training runs with sequence length of 64 and 3 with length 128. Then I averaged all 6 prediction files. The averaging improves logloss score dramatically, so 0.26 should be fine as one of the runs.
\n\nYou are exactly right in your explanation of my truncating algorithm. My thinking was that I wanted to truncate as far away from P, A, B tokens as possible. So you could actually think of it as 6 segments:
\n\n`\n x --- P --- x --- A --- x --- B --- x #x represents a delete point.\n
The end segments were truncated on the outsides and the inner segments were truncated in the middle, away from P, A, B. I'm not sure if that sketch explains why the 2 middle segments were longer.
\n\nChanging the truncation algorithm seemed to have a big impact on results. So this would be an area for further research. Instinct tells me that truncating at the end of sentences where possible would be better than just counting tokens like I did.
", "votes": 1, "replies": [] }, { "id": 526590, "author_name": "Tania J", "author_url": "", "post_date": "2019-05-03T10:40:14.397000", "content": "I see, that makes sense.
\n\nWell, thanks again for sharing so much, I've definitely learned a lot from you! Coolest school principal ever. :)
", "votes": 0, "replies": [] }, { "id": 526599, "author_name": "Ken Krige", "author_url": "", "post_date": "2019-05-03T11:07:44.873000", "content": "Only a pleasure. Enjoy playing with hyper parameters, and don't forget to turn off server and tpu on google cloud when you are done.
", "votes": 1, "replies": [] } ] } ], "comment_count": 20, "messages": [ { "id": 523224, "postDate": "2019-04-25T18:50:43.207Z", "content": "My model was based on fine-tuning BERT-large with rudimentary output layer and no additional statistical features. The code in run_classifier.py() provided with BERT was modified to achieve this. I will post the full code here once I've neatened it up, but for now an overview of the main elements.
For the input layer, I sorted the 3 char offsets and stored the corresponding permutation of P, A, B and split the text into 4 segments at these offsets. Each segment was Wordpiece tokenized and the token segments were truncated to match the required sequence length. The choice of truncating algorithim made a big difference to the results. The token offsets were calculated as the cumulative sum of tokens in each segment. The token offsets were sorted back into P, A, B order and used to make binary mask “features” for the positions of P, A, B.
\n\nThe output from BERT was masked to retrieve the 1024 dimensional vectors for P, A, B. Three new “probability embeddings” were computed as (element-wise multiplication):
\n\nThe idea of the first two was to represent the similarity between P&A or P&B as abstract vector embeddings. For ‘Neither” the abstract embedding in 3 is supposed to represent the extent to which A&B are similar to each other, but differ from P. These embeddings (1024 dim vectors) were then reduced to a scalar probability by a trainable (1024, 1) tensor. The same tensor was used for the probs of A and B to prevent the model learning from whether “True” or “False” appeared first, as my own pseudo-data (not used in the end) was somewhat unbalanced in this regard. A seperate trainable tensor was applied to (AB – PP).
\n\nThis architecture on a 64 token sequence gave a stage 1 score (corrected gap data) of 0.30 to 0.38 (single model), which ensembled to 0.275 by simple averaging of the predictions from 6 models. I assume the reason for the range of results has to do with the architecture being a bit unstable with respect to random initialisation.
\n\nWithin run_classifier.py(), changes were made to the input methods to suite the GAP data. A detailed explanation is done via this short kernel, which illustrates how a single example is prepared for the input layer. Pay particular attention to the way text was truncated to fit the chosen maximum token sequence. Changes to this algorithm made a big difference to results. I think that doing this truncating at a sentence level might bring further gains.
Vanilla BERT-large.
\n\nUnfortunately, the 1024 dimensional vectors make it impractical to do a similar demonstration of the output. Instead I will give a short explanation of some of the code.
\n\nFirstly, the output was extracted and masked to get the P, A, B embeddings.
\n\n\nall_out = model.get_sequence_output()\nP = tf.boolean_mask(all_out, P_mask)\nA = tf.boolean_mask(all_out, A_mask)\nB = tf.boolean_mask(all_out, B_mask)\n
This worked fine on a GPU, but tf.boolean_mask() is not implemented for TPU, which I stepped up to for BERT-large. It took a good few hours to figure out a messy workaround, but it did the job.
The probability embeddings were then calculated as explained above.
\n\n\nPA = tf.multiply(P, A)\nPB = tf.multiply(P, B)\nPP = tf.multiply(P, P)\nAB = tf.multiply(A, B)\nN = tf.subtract(PP, AB)\n
And finally reduced to probabilities by trainable tensors and a softmax.
\n\n```\nAB_weights = tf.get_variable(\n \"AB_weights\", [1, hidden_size],\n initializer=tf.truncated_normal_initializer(stddev=0.02))
\n\nN_weights = tf.get_variable(\n \"N_weights\", [1, hidden_size],\n initializer=tf.truncated_normal_initializer(stddev=0.02))
\n\nA_out = tf.matmul(PA, AB_weights, transpose_b=True)\nB_out = tf.matmul(PB, AB_weights, transpose_b=True)\nN_out = tf.matmul(N, N_weights, transpose_b=True)
\n\noutput_bias = tf.get_variable(\n \"output_bias\", [num_labels], initializer=tf.zeros_initializer())
\n\nlogits = tf.concat([A_out, B_out, N_out], axis=1)\nlogits = tf.nn.bias_add(logits, output_bias)\nprobabilities = tf.nn.softmax(logits, axis=-1)\nlog_probs = tf.nn.log_softmax(logits, axis=-1)\n``\nThat was about it. The rest of the code was pretty routine or already included inrun_classifier.py`.
Fantastic! Simple but effective. Thanks for sharing. The prob embeddings were genius. Bit-wise multiplication is underrated.
", "rawMarkdown": "Fantastic! Simple but effective. Thanks for sharing. The prob embeddings were genius. Bit-wise multiplication is underrated.", "votes": 1, "replies": [ { "id": 523345, "postDate": "2019-04-26T03:43:40.457Z", "content": "Actually, I should have said element-wise multiplication, as in tf.multiply(). Apologies for the error. I will correct it in the post.
", "rawMarkdown": "Actually, I should have said element-wise multiplication, as in tf.multiply(). Apologies for the error. I will correct it in the post." }, { "id": 523401, "postDate": "2019-04-26T07:05:19.897Z", "content": "The idea of the element-wise multiplication was to achieve something similar to a dot product, as described in the original BERT Paper. If the trainable output tensor were all 1s it would be exactly dot product.
\n\nP dot A would give some measure of how similar the embeddings of the pronoun and name A are.
", "rawMarkdown": "The idea of the element-wise multiplication was to achieve something similar to a dot product, as described in the original BERT Paper. If the trainable output tensor were all 1s it would be exactly dot product.\n\n**P dot A** would give some measure of how similar the embeddings of the pronoun and name A are.", "votes": 1 }, { "id": 523671, "postDate": "2019-04-26T17:52:06.157Z", "content": "I thought bit-wise was the DS slang for element-wise :) If the rows are summed up it indeed is the dot product. This could be done in hidden layers too. From my understanding Attention achieves its results by making use of multiplication, whereas in vanilla nets, outputs of hidden layers do not get multiplied. I am also curious how truncating in the middle of sentences didn’t reduce the score much.
\n\nBy the way, I am in the process of publishing a paper on AGI: solving word problems. It just needs a review and a few suggestions. I would be happy if you would like to take a look or join as an author. Cheers.
", "rawMarkdown": "I thought bit-wise was the DS slang for element-wise :) If the rows are summed up it indeed is the dot product. This could be done in hidden layers too. From my understanding Attention achieves its results by making use of multiplication, whereas in vanilla nets, outputs of hidden layers do not get multiplied. I am also curious how truncating in the middle of sentences didn’t reduce the score much.\n\nBy the way, I am in the process of publishing a paper on AGI: solving word problems. It just needs a review and a few suggestions. I would be happy if you would like to take a look or join as an author. Cheers.\n\n", "votes": 1 }, { "id": 523674, "postDate": "2019-04-26T18:02:26.253Z", "content": "@isikkuntay I would be very happy to take a look, but I'm a newcomer to this field so might not be able to make much of a contribution.
", "rawMarkdown": "@isikkuntay I would be very happy to take a look, but I'm a newcomer to this field so might not be able to make much of a contribution." }, { "id": 523768, "postDate": "2019-04-27T01:15:06.710Z", "content": "Thanks! Here is the link to the paper: https://github.com/isikkuntay/AIND/blob/master/updated%20CRN_paper-04-22-19.pdf
\n\nNew comer or not, come to think of it, you did 3x better than a group of million dollar google engineers in your first ML project :) Didn’t make you principal for nothing ;))
", "rawMarkdown": "Thanks! Here is the link to the paper: https://github.com/isikkuntay/AIND/blob/master/updated%20CRN_paper-04-22-19.pdf\n\nNew comer or not, come to think of it, you did 3x better than a group of million dollar google engineers in your first ML project :) Didn’t make you principal for nothing ;))" } ] }, { "id": 523247, "postDate": "2019-04-25T20:22:36.137Z", "content": "Thanks for sharing! So if I understand correctly what made the biggest difference for you was the truncation?
", "rawMarkdown": "Thanks for sharing! So if I understand correctly what made the biggest difference for you was the truncation?", "votes": 1, "replies": [ { "id": 523346, "postDate": "2019-04-26T03:51:44.567Z", "content": "That certainly made a big difference. Curiously, shorter sequences, all the way down to about 64 tokens, gave me the best single model results. Instinctively I had assumed that retaining as much information as possible would give the best results. The only explanation I can think of is that tokens close together influence each other's embeddings more, giving stronger relationship between P, A, B in the output layer. I need help in understanding this and would love to hear alternative explanations.
", "rawMarkdown": "That certainly made a big difference. Curiously, shorter sequences, all the way down to about 64 tokens, gave me the best single model results. Instinctively I had assumed that retaining as much information as possible would give the best results. The only explanation I can think of is that tokens close together influence each other's embeddings more, giving stronger relationship between P, A, B in the output layer. I need help in understanding this and would love to hear alternative explanations.", "votes": 1 }, { "id": 523582, "postDate": "2019-04-26T14:58:53.910Z", "content": "@kenkrige even today, sota language models suffer from noise in long-range dependencies. Add to that, the noise inherent in text.
\n\nIn case of the GPR problem, there are only very few examples that require long-range context pooling. You may have made these worse by breaking the syntactic structure while truncating. If the truncation retains phrasal structure, then it won't be as detrimental. As you mentioned, truncating at a sentence-level will be better, since a sentence has self-contained structure.
\n\nFor the remaining majority examples where the model is correct, you should see more confident probabilities for the shortened version compared to the original text version. Hence, a bump in the overall score.
", "rawMarkdown": "@kenkrige even today, sota language models suffer from noise in long-range dependencies. Add to that, the noise inherent in text.\n\nIn case of the GPR problem, there are only very few examples that require long-range context pooling. You may have made these worse by breaking the syntactic structure while truncating. If the truncation retains phrasal structure, then it won't be as detrimental. As you mentioned, truncating at a sentence-level will be better, since a sentence has self-contained structure.\n\nFor the remaining majority examples where the model is correct, you should see more confident probabilities for the shortened version compared to the original text version. Hence, a bump in the overall score.", "votes": 2 }, { "id": 523651, "postDate": "2019-04-26T17:16:04.873Z", "content": "Thanks @sattree. I suppose your coref pooling based on other models would have shortened the dependencies thereby reducing some of that noise. Well done again on your insightful solution.
", "rawMarkdown": "Thanks @sattree. I suppose your coref pooling based on other models would have shortened the dependencies thereby reducing some of that noise. Well done again on your insightful solution." } ] }, { "id": 524614, "postDate": "2019-04-29T07:36:50.373Z", "content": "Congrats on the gold and thanks for sharing. I'm looking forward to seeing the full code.
", "rawMarkdown": "Congrats on the gold and thanks for sharing. I'm looking forward to seeing the full code.", "replies": [ { "id": 525814, "postDate": "2019-05-01T19:00:25.433Z", "content": "@taniaj sorry about the delay. I was planning to polish my code but time has not been on my side. There is not much of a readme so feel free to ask questions if you have problems running it.
", "rawMarkdown": "@taniaj sorry about the delay. I was planning to polish [my code](https://github.com/kenkrige/BERT-Fine-tune-for-GAP) but time has not been on my side. There is not much of a readme so feel free to ask questions if you have problems running it.", "votes": 3 }, { "id": 526087, "postDate": "2019-05-02T10:23:56.003Z", "content": "No worries at all. I got it running (with reduced batch size) on my GPUs but the result wasn't great. I guess you ended up using a Colab TPU for the final result? Your code is very clean and readable but my Tensorflow knowledge is lacking so I'm using this as motivation to learn it well enough to understand your code. Thank you!
", "rawMarkdown": "No worries at all. I got it running (with reduced batch size) on my GPUs but the result wasn't great. I guess you ended up using a Colab TPU for the final result? Your code is very clean and readable but my Tensorflow knowledge is lacking so I'm using this as motivation to learn it well enough to understand your code. Thank you!" }, { "id": 526117, "postDate": "2019-05-02T11:33:54.557Z", "content": "I'm also new to tensorflow and to machine learning as a whole, which is why started with their run_cassifier.py and modified that. I was very lucky to do so well in this competition.
I'm glad you got it running. I found large batch sizes helped results a lot, so did using BERT large instead of BERT base. For both of those you will need a TPU I think. I didn't use colab, I used google cloud platform. When you sign up, they give you one year and $300 free. A preemptible TPU is about $1.40 per hour and each training run only takes about 6 min for that data. There are a few configuration tricks, because you have to use google storage gs://your_bucket/... for all your files. Please ask if you get stuck.
Another thing I learned was that some bad logloss results don't really matter as long as the accuracy is good. Then when you average the output csv files from a few different training runs and clip the results that are too close to 0 or 1 then the logloss improves drastically. Two training runs with the same hyper-parameters can give significantly different log-loss results, but prediction accuracy is quite consistent. I got about 85% accuracy from BERT base and 89% for BERT large. I just did the averaging in pandas afterwards.
", "rawMarkdown": "I'm also new to tensorflow and to machine learning as a whole, which is why started with their `run_cassifier.py` and modified that. I was very lucky to do so well in this competition.\n\nI'm glad you got it running. I found large batch sizes helped results a lot, so did using BERT large instead of BERT base. For both of those you will need a TPU I think. I didn't use colab, I used google cloud platform. When you sign up, they give you one year and $300 free. A preemptible TPU is about $1.40 per hour and each training run only takes about 6 min for that data. There are a few configuration tricks, because you have to use google storage `gs://your_bucket/...` for all your files. Please ask if you get stuck.\n\n Another thing I learned was that some bad logloss results don't really matter as long as the accuracy is good. Then when you average the output csv files from a few different training runs and clip the results that are too close to 0 or 1 then the logloss improves drastically. Two training runs with the same hyper-parameters can give significantly different log-loss results, but prediction accuracy is quite consistent. I got about 85% accuracy from BERT base and 89% for BERT large. I just did the averaging in pandas afterwards.", "votes": 2 }, { "id": 526120, "postDate": "2019-05-02T11:37:28.040Z", "content": "By the way, @taniaj thanks again for sharing your data corrections. Those also helped my results.
", "rawMarkdown": "By the way, @taniaj thanks again for sharing your data corrections. Those also helped my results.", "votes": 1 }, { "id": 526290, "postDate": "2019-05-02T18:17:58.970Z", "content": "@kenkrige Thanks for taking time off your holiday to post the code.
", "rawMarkdown": "@kenkrige Thanks for taking time off your holiday to post the code." }, { "id": 526571, "postDate": "2019-05-03T09:45:45.920Z", "content": "I got it running with BERT-Large on Google Cloud TPU. :) Private LB score was 0.26718 but I didn't play with any parameters at all.
\n\nI do have a question about how you do the token segment truncation though. If I understand your code correctly it looks like you are allowing the middle 2 segments to be larger than the first and last (2x as large) and truncating from the middle of the middle segments instead of the end. Is there a particular reason for doing this instead of allowing all segments to be equally large and always truncating from the end (except for the first segment)?
", "rawMarkdown": "I got it running with BERT-Large on Google Cloud TPU. :) Private LB score was 0.26718 but I didn't play with any parameters at all.\n\nI do have a question about how you do the token segment truncation though. If I understand your code correctly it looks like you are allowing the middle 2 segments to be larger than the first and last (2x as large) and truncating from the middle of the middle segments instead of the end. Is there a particular reason for doing this instead of allowing all segments to be equally large and always truncating from the end (except for the first segment)?" }, { "id": 526588, "postDate": "2019-05-03T10:29:54.737Z", "content": "Well done! That's an excellent score. If you run training again with exactly the same parameters, you might get a worse (or better) logloss score. For my final leaderboard score 0f 0.19, I used 3 training runs with sequence length of 64 and 3 with length 128. Then I averaged all 6 prediction files. The averaging improves logloss score dramatically, so 0.26 should be fine as one of the runs.
\n\nYou are exactly right in your explanation of my truncating algorithm. My thinking was that I wanted to truncate as far away from P, A, B tokens as possible. So you could actually think of it as 6 segments:
\n\n`\n x --- P --- x --- A --- x --- B --- x #x represents a delete point.\n
The end segments were truncated on the outsides and the inner segments were truncated in the middle, away from P, A, B. I'm not sure if that sketch explains why the 2 middle segments were longer.
\n\nChanging the truncation algorithm seemed to have a big impact on results. So this would be an area for further research. Instinct tells me that truncating at the end of sentences where possible would be better than just counting tokens like I did.
", "rawMarkdown": "Well done! That's an excellent score. If you run training again with exactly the same parameters, you might get a worse (or better) logloss score. For my final leaderboard score 0f 0.19, I used 3 training runs with sequence length of 64 and 3 with length 128. Then I averaged all 6 prediction files. The averaging improves logloss score dramatically, so 0.26 should be fine as one of the runs.\n\nYou are exactly right in your explanation of my truncating algorithm. My thinking was that I wanted to truncate as far away from P, A, B tokens as possible. So you could actually think of it as 6 segments:\n\n```\n x --- P --- x --- A --- x --- B --- x #x represents a delete point.\n``\n\nThe end segments were truncated on the outsides and the inner segments were truncated in the middle, away from P, A, B. I'm not sure if that sketch explains why the 2 middle segments were longer.\n\nChanging the truncation algorithm seemed to have a big impact on results. So this would be an area for further research. Instinct tells me that truncating at the end of sentences where possible would be better than just counting tokens like I did.", "votes": 1 }, { "id": 526590, "postDate": "2019-05-03T10:40:14.397Z", "content": "I see, that makes sense.
\n\nWell, thanks again for sharing so much, I've definitely learned a lot from you! Coolest school principal ever. :)
", "rawMarkdown": "I see, that makes sense. \n\nWell, thanks again for sharing so much, I've definitely learned a lot from you! Coolest school principal ever. :)" }, { "id": 526599, "postDate": "2019-05-03T11:07:44.873Z", "content": "Only a pleasure. Enjoy playing with hyper parameters, and don't forget to turn off server and tpu on google cloud when you are done.
", "rawMarkdown": "Only a pleasure. Enjoy playing with hyper parameters, and don't forget to turn off server and tpu on google cloud when you are done.", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:42:47Z", "competition_id": 12797, "competition_slug": "gendered-pronoun-resolution", "team_id": 2735513, "solution_url": "/competitions/gendered-pronoun-resolution/writeups/ken-krige-5th-placed-solution" }