{
  "id": 130056,
  "title": "41-th Place Solution",
  "content": "<h1><strong>DATA PREPROCESSING</strong></h1>\n\n<p><strong>NER</strong>: we pretrained NER-model (bert-base) for code blocks replacement with special token [CODE]\n<strong>Reg.exp.</strong>: we cleaned up LaTeX snippets and replaced them with [MATH] token\n<strong>Crop strategy</strong>: from both side (we cut the middle part, if sequence was longer than 450 tokens)\n<strong>Max. seq. lengths</strong>: 450</p>\n\n<h1><strong>MODEL PARAMETERS</strong></h1>\n\n<p><strong>Model architecture</strong>: roberta-large\n<strong>Pretrained</strong>: Yes (from transformers library)\n<strong>Model design</strong>: siamese bi-encoder (dot-product of question-answer pair -&gt; sigmoid)\n<strong>Pooling strategy</strong>: [CLS] token</p>\n\n<h1><strong>TRAINING PARAMETERS</strong></h1>\n\n<p><strong>Optimizer</strong>: AdamW (from transformers library)\n<strong>Max grad. norm</strong>: 5 \n<strong>Epochs</strong>: 5\n<strong>Batch size</strong>: 8 (1 * 8 accum. steps)\n<strong>LR</strong>: 2.5e-05\n<strong>LR-scheduler</strong>: linear\n<strong>Warmup steps</strong>: 50\n<strong>Seed</strong>: 228 :)</p>\n\n<h1><strong>ENSEMBLE PARAMETERS</strong></h1>\n\n<p><strong>Blend of</strong>: 7-fold models\n<strong>Fold strategy</strong>: GroupKFold\n<strong>Blend strategy</strong>: probabilities averaging</p>\n\n<h1><strong>POST-PROCESSING</strong></h1>\n\n<p>For each target:\n1. Sort values (ascending)\n2. Initialize an anchor value (the smallest one)\n3. Iterate over each value and calculate the difference between it and anchor\n4. If difference &gt; TARGET-EPSILON, then collapse all values between anchor and current value into one bin\n5. Update anchor (set current value)\n6. Repeat from 3 until end\nFinally, normalize bins to (0, 1) range</p>\n\n<p>TARGET-EPSILONs (for each) target estimated via out of fold predictions (from 7 folds)</p>\n\n<h1><strong>ANOTHER TRICKS</strong></h1>\n\n<p><strong>SWA</strong>: implemented, but did not have time to submit\n<strong>Categorical features</strong>: was concatenated with bi-encoder output. No improvements\n<strong>Additional data</strong>: few additional data was added to fill up some flawed targets. Did not help\n<strong>Combine crop strategies</strong>: different crop strategies (from left, rigth, both and middle parts) were combined as a TT augmentations. Did not help\n<strong>Different pooling strategies</strong>: average pooling, average pooling of N last layers. [CLS] was the best one\n<strong>Freezing</strong> differential, full, only n layers, etc. No freezing was the best</p>\n\n<h1><strong>Reproducibility</strong></h1>\n\n<p>All code to reproduce our experiments could be found here:\n<a href=\"https://github.com/alexeykarnachev/kaggle_google_qa_labeling\">https://github.com/alexeykarnachev/kaggle_google_qa_labeling</a></p>",
  "comments": [
    {
      "id": 743376,
      "author_name": "Ailurophile",
      "author_url": "",
      "post_date": "2020-02-12T02:38:32.503000",
      "content": "<p>Congratulations!\nGreat Write-Up\nThanks for Sharing your Approach &amp; Insights!! <a href=\"/alexeykarnachev\">@alexeykarnachev</a> </p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 1,
  "messages": [
    {
      "id": 743217,
      "postDate": "2020-02-11T22:18:47.897Z",
      "content": "<h1><strong>DATA PREPROCESSING</strong></h1>\n\n<p><strong>NER</strong>: we pretrained NER-model (bert-base) for code blocks replacement with special token [CODE]\n<strong>Reg.exp.</strong>: we cleaned up LaTeX snippets and replaced them with [MATH] token\n<strong>Crop strategy</strong>: from both side (we cut the middle part, if sequence was longer than 450 tokens)\n<strong>Max. seq. lengths</strong>: 450</p>\n\n<h1><strong>MODEL PARAMETERS</strong></h1>\n\n<p><strong>Model architecture</strong>: roberta-large\n<strong>Pretrained</strong>: Yes (from transformers library)\n<strong>Model design</strong>: siamese bi-encoder (dot-product of question-answer pair -&gt; sigmoid)\n<strong>Pooling strategy</strong>: [CLS] token</p>\n\n<h1><strong>TRAINING PARAMETERS</strong></h1>\n\n<p><strong>Optimizer</strong>: AdamW (from transformers library)\n<strong>Max grad. norm</strong>: 5 \n<strong>Epochs</strong>: 5\n<strong>Batch size</strong>: 8 (1 * 8 accum. steps)\n<strong>LR</strong>: 2.5e-05\n<strong>LR-scheduler</strong>: linear\n<strong>Warmup steps</strong>: 50\n<strong>Seed</strong>: 228 :)</p>\n\n<h1><strong>ENSEMBLE PARAMETERS</strong></h1>\n\n<p><strong>Blend of</strong>: 7-fold models\n<strong>Fold strategy</strong>: GroupKFold\n<strong>Blend strategy</strong>: probabilities averaging</p>\n\n<h1><strong>POST-PROCESSING</strong></h1>\n\n<p>For each target:\n1. Sort values (ascending)\n2. Initialize an anchor value (the smallest one)\n3. Iterate over each value and calculate the difference between it and anchor\n4. If difference &gt; TARGET-EPSILON, then collapse all values between anchor and current value into one bin\n5. Update anchor (set current value)\n6. Repeat from 3 until end\nFinally, normalize bins to (0, 1) range</p>\n\n<p>TARGET-EPSILONs (for each) target estimated via out of fold predictions (from 7 folds)</p>\n\n<h1><strong>ANOTHER TRICKS</strong></h1>\n\n<p><strong>SWA</strong>: implemented, but did not have time to submit\n<strong>Categorical features</strong>: was concatenated with bi-encoder output. No improvements\n<strong>Additional data</strong>: few additional data was added to fill up some flawed targets. Did not help\n<strong>Combine crop strategies</strong>: different crop strategies (from left, rigth, both and middle parts) were combined as a TT augmentations. Did not help\n<strong>Different pooling strategies</strong>: average pooling, average pooling of N last layers. [CLS] was the best one\n<strong>Freezing</strong> differential, full, only n layers, etc. No freezing was the best</p>\n\n<h1><strong>Reproducibility</strong></h1>\n\n<p>All code to reproduce our experiments could be found here:\n<a href=\"https://github.com/alexeykarnachev/kaggle_google_qa_labeling\">https://github.com/alexeykarnachev/kaggle_google_qa_labeling</a></p>",
      "rawMarkdown": "**DATA PREPROCESSING**\n=======================================\n**NER**: we pretrained NER-model (bert-base) for code blocks replacement with special token [CODE]\n**Reg.exp.**: we cleaned up LaTeX snippets and replaced them with [MATH] token\n**Crop strategy**: from both side (we cut the middle part, if sequence was longer than 450 tokens)\n**Max. seq. lengths**: 450\n\n**MODEL PARAMETERS**\n=======================================\n**Model architecture**: roberta-large\n**Pretrained**: Yes (from transformers library)\n**Model design**: siamese bi-encoder (dot-product of question-answer pair -&gt; sigmoid)\n**Pooling strategy**: [CLS] token\n\n**TRAINING PARAMETERS**\n=======================================\n**Optimizer**: AdamW (from transformers library)\n**Max grad. norm**: 5 \n**Epochs**: 5\n**Batch size**: 8 (1 * 8 accum. steps)\n**LR**: 2.5e-05\n**LR-scheduler**: linear\n**Warmup steps**: 50\n**Seed**: 228 :)\n\n**ENSEMBLE PARAMETERS**\n=======================================\n**Blend of**: 7-fold models\n**Fold strategy**: GroupKFold\n**Blend strategy**: probabilities averaging\n\n**POST-PROCESSING**\n=======================================\nFor each target:\n1. Sort values (ascending)\n2. Initialize an anchor value (the smallest one)\n3. Iterate over each value and calculate the difference between it and anchor\n4. If difference &gt; TARGET-EPSILON, then collapse all values between anchor and current value into one bin\n5. Update anchor (set current value)\n6. Repeat from 3 until end\nFinally, normalize bins to (0, 1) range\n\nTARGET-EPSILONs (for each) target estimated via out of fold predictions (from 7 folds)\n\n**ANOTHER TRICKS**\n=======================================\n**SWA**: implemented, but did not have time to submit\n**Categorical features**: was concatenated with bi-encoder output. No improvements\n**Additional data**: few additional data was added to fill up some flawed targets. Did not help\n**Combine crop strategies**: different crop strategies (from left, rigth, both and middle parts) were combined as a TT augmentations. Did not help\n**Different pooling strategies**: average pooling, average pooling of N last layers. [CLS] was the best one\n**Freezing** differential, full, only n layers, etc. No freezing was the best\n\n**Reproducibility**\n=======================================\nAll code to reproduce our experiments could be found here:\nhttps://github.com/alexeykarnachev/kaggle_google_qa_labeling",
      "votes": 7
    },
    {
      "id": 743376,
      "postDate": "2020-02-12T02:38:32.503Z",
      "content": "<p>Congratulations!\nGreat Write-Up\nThanks for Sharing your Approach &amp; Insights!! <a href=\"/alexeykarnachev\">@alexeykarnachev</a> </p>",
      "rawMarkdown": "Congratulations!\nGreat Write-Up\nThanks for Sharing your Approach &amp; Insights!! @alexeykarnachev "
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:08:23Z",
  "competition_id": 7968,
  "competition_slug": "google-quest-challenge",
  "team_id": 4028075,
  "solution_url": "/competitions/google-quest-challenge/writeups/popustivshiesya-41-th-place-solution"
}