{
  "id": 668619,
  "title": "Probing the Best Leaderboard Public Models",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/668619",
  "author_name": "Aaron Bornstein",
  "post_date": "2026-01-17T20:05:52.915000",
  "votes": 42,
  "comment_count": 12,
  "views": 0,
  "content": "<p>I’ve been digging into the high-scoring baseline models (34+ LB) with a systematic probing framework I developed to understand the strengths and weaknesses of these models.</p>\n<p>I wanted to share some evidence suggesting we might be seriously overfitting to the public leaderboard. After running 104 paired perturbation probes, my analysis shows the best ByT5 models are effectively functioning as a sophisticated pattern matchers (~60% translation, ~40% memorization) rather than a general translator. </p>\n<p>While they handle negation and numbers perfectly, the best public models fail on many isolated vocabulary and often fail to successfully copies novel proper names, hallucinating generic trade sentences around single words like \"Disregard that an unclean man…\". </p>\n<p>This strongly suggests the model is exploiting corpus-specific templates rather than learning core semantics, it's important to note that high scores on the visible 33% of the test set are likely to degrade significantly on the hidden 67% if they are out of domain.</p>\n<p>Sharing the notebook below hope these probes can help identitfy new techniques for generalizing the models beyond data cleaning tricks.</p>\n<p><a href=\"https://www.kaggle.com/code/aaronbornstein/probing-leader-board-overfitting-34-baseline\" target=\"_blank\">https://www.kaggle.com/code/aaronbornstein/probing-leader-board-overfitting-34-baseline</a></p>",
  "messages": [
    {
      "id": 3392924,
      "postDate": "2026-01-17T20:05:52.917Z",
      "content": "<p>I’ve been digging into the high-scoring baseline models (34+ LB) with a systematic probing framework I developed to understand the strengths and weaknesses of these models.</p>\n<p>I wanted to share some evidence suggesting we might be seriously overfitting to the public leaderboard. After running 104 paired perturbation probes, my analysis shows the best ByT5 models are effectively functioning as a sophisticated pattern matchers (~60% translation, ~40% memorization) rather than a general translator. </p>\n<p>While they handle negation and numbers perfectly, the best public models fail on many isolated vocabulary and often fail to successfully copies novel proper names, hallucinating generic trade sentences around single words like \"Disregard that an unclean man…\". </p>\n<p>This strongly suggests the model is exploiting corpus-specific templates rather than learning core semantics, it's important to note that high scores on the visible 33% of the test set are likely to degrade significantly on the hidden 67% if they are out of domain.</p>\n<p>Sharing the notebook below hope these probes can help identitfy new techniques for generalizing the models beyond data cleaning tricks.</p>\n<p><a href=\"https://www.kaggle.com/code/aaronbornstein/probing-leader-board-overfitting-34-baseline\" target=\"_blank\">https://www.kaggle.com/code/aaronbornstein/probing-leader-board-overfitting-34-baseline</a></p>",
      "rawMarkdown": "I’ve been digging into the high-scoring baseline models (34+ LB) with a systematic probing framework I developed to understand the strengths and weaknesses of these models.\n\nI wanted to share some evidence suggesting we might be seriously overfitting to the public leaderboard. After running 104 paired perturbation probes, my analysis shows the best ByT5 models are effectively functioning as a sophisticated pattern matchers (~60% translation, ~40% memorization) rather than a general translator. \n\nWhile they handle negation and numbers perfectly, the best public models fail on many isolated vocabulary and often fail to successfully copies novel proper names, hallucinating generic trade sentences around single words like \"Disregard that an unclean man...\". \n\nThis strongly suggests the model is exploiting corpus-specific templates rather than learning core semantics, it's important to note that high scores on the visible 33% of the test set are likely to degrade significantly on the hidden 67% if they are out of domain.\n\nSharing the notebook below hope these probes can help identitfy new techniques for generalizing the models beyond data cleaning tricks.\n\nhttps://www.kaggle.com/code/aaronbornstein/probing-leader-board-overfitting-34-baseline",
      "votes": 42
    },
    {
      "id": 3392935,
      "postDate": "2026-01-17T21:00:52.233Z",
      "content": "<p>This is a brilliant and devastating analysis. It confirms exactly what I've been experiencing in my latest experiments. I want to share a concrete example that validates your \"Pattern Matcher vs. Translator\" hypothesis.</p>\n<p>I analyzed two distinct models:</p>\n<p><strong>Model A (The Parrot):</strong></p>\n<ul>\n<li>Method: Standard ByT5 with aggressive gap-filling regex.</li>\n<li>Behavior: It hallucinates standard phrases like \"The City,\" \"Silver,\" and \"Merchant\" to fill gaps. It often repeats n-grams.</li>\n<li>Public LB Score: 34.2</li>\n</ul>\n<p>Examples:\nid,translation</p>\n<ul>\n<li>0,\"Thus Kanesh colony, say to the  of our messengers, every single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single day.\"</li>\n<li>1,\"In the tablet of the City you wrote to me in the tablet of the City. This day whoever receives the silver, Daur or Ewa, will take the colony of my colony.\"</li>\n</ul>\n<p><strong>Model B (The Domain Expert):</strong></p>\n<ul>\n<li>Method: Reference-aligned training with minimal normalization, focused on semantic accuracy.</li>\n<li>Behavior: It correctly translates specific Old Assyrian trade terms.</li>\n<li>- shaddutum --&gt; \"Transport tariff\" (Instead of just \"silver\" or \"tax\"), </li>\n<li>- emārum ṣalmum --&gt; \"Black donkey\" (Instead of just \"donkey\" or \"goods\")</li>\n<li>- wabartum --&gt; \"Affiliated trading station\" (Instead of leaving it as \"Wabartum\")</li>\n<li>Public LB Score: 11.8 (!)</li>\n</ul>\n<p>Examples:\nid,translation</p>\n<ul>\n<li>0,From the Kanesh colony to a merchant: In accordance with our instructions we have seized us for an expense of black donkey.</li>\n<li>1,In accordance with our instructions we have received the silver on behalf of a merchant.</li>\n</ul>\n<p>The current Public LB metric (BLEU/CHRF) seems to heavily penalize Semantic Fidelity (Model B) while rewarding Structural Conformity (Model A).</p>\n<p>When Model B translates shaddutum as \"transport tariff,\" it likely gets a 0 score for that n-gram if the reference translation uses a generic term like \"tax\" or \"silver.\" meanwhile, Model A gets points for simply outputting the most statistically probable words (\"silver\", \"give\", \"city\"), even if the sentence meaning is hallucinated.</p>\n<p>Your probing framework proves that we are optimizing for a metric that measures \"Vibes\" rather than \"Translation.\" As you suggested, this creates a massive risk for the Private LB (hidden 67%), where out-of-distribution texts will likely break the pattern-matching models (Model A), while the semantically grounded models (Model B) should theoretically perform better.</p>\n<p>Thank you for providing the tools to diagnose this \"Metric Trap.\"</p>",
      "rawMarkdown": "This is a brilliant and devastating analysis. It confirms exactly what I've been experiencing in my latest experiments. I want to share a concrete example that validates your \"Pattern Matcher vs. Translator\" hypothesis.\n\nI analyzed two distinct models:\n\n**Model A (The Parrot):**\n- Method: Standard ByT5 with aggressive gap-filling regex.\n- Behavior: It hallucinates standard phrases like \"The City,\" \"Silver,\" and \"Merchant\" to fill gaps. It often repeats n-grams.\n- Public LB Score: 34.2\n\nExamples:\nid,translation\n- 0,\"Thus Kanesh colony, say to the <big_gap> of our messengers, every single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single day.\"\n- 1,\"In the tablet of the City you wrote to me in the tablet of the City. This day whoever receives the silver, Daur or Ewa, will take the colony of my colony.\"\n\n**Model B (The Domain Expert):**\n- Method: Reference-aligned training with minimal normalization, focused on semantic accuracy.\n- Behavior: It correctly translates specific Old Assyrian trade terms.\n- - shaddutum --> \"Transport tariff\" (Instead of just \"silver\" or \"tax\"), \n- - emārum ṣalmum --> \"Black donkey\" (Instead of just \"donkey\" or \"goods\")\n- - wabartum --> \"Affiliated trading station\" (Instead of leaving it as \"Wabartum\")\n- Public LB Score: 11.8 (!)\n\nExamples:\nid,translation\n- 0,From the Kanesh colony to a merchant: In accordance with our instructions we have seized us for an expense of black donkey.\n- 1,In accordance with our instructions we have received the silver on behalf of a merchant.\n\n\nThe current Public LB metric (BLEU/CHRF) seems to heavily penalize Semantic Fidelity (Model B) while rewarding Structural Conformity (Model A).\n\nWhen Model B translates shaddutum as \"transport tariff,\" it likely gets a 0 score for that n-gram if the reference translation uses a generic term like \"tax\" or \"silver.\" meanwhile, Model A gets points for simply outputting the most statistically probable words (\"silver\", \"give\", \"city\"), even if the sentence meaning is hallucinated.\n\nYour probing framework proves that we are optimizing for a metric that measures \"Vibes\" rather than \"Translation.\" As you suggested, this creates a massive risk for the Private LB (hidden 67%), where out-of-distribution texts will likely break the pattern-matching models (Model A), while the semantically grounded models (Model B) should theoretically perform better.\n\nThank you for providing the tools to diagnose this \"Metric Trap.\"",
      "votes": 13,
      "replies": [
        {
          "id": 3393091,
          "postDate": "2026-01-18T08:08:47.777Z",
          "content": "<p>I agree I’m starting to think that some of the gains from data cleaning are also misleading they are essentially minimizing the false positive character match rate on the test distribution but not necessarily increasing the quality of the translations.</p>",
          "rawMarkdown": "I agree I’m starting to think that some of the gains from data cleaning are also misleading they are essentially minimizing the false positive character match rate on the test distribution but not necessarily increasing the quality of the translations.",
          "votes": 3
        },
        {
          "id": 3393147,
          "postDate": "2026-01-18T10:26:46.257Z",
          "content": "<p>BLEU is simply not suitable for this task. Private LB can also be OOD in many ways, there is no guarantee that a semantically grounded model will perform better under metrics like this. The hosts should consider replacing BLEU with BLEURT - there is still enough time.</p>",
          "rawMarkdown": "BLEU is simply not suitable for this task. Private LB can also be OOD in many ways, there is no guarantee that a semantically grounded model will perform better under metrics like this. The hosts should consider replacing BLEU with BLEURT - there is still enough time.",
          "votes": 11,
          "replies": [
            {
              "id": 3393344,
              "postDate": "2026-01-18T20:41:26.290Z",
              "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> </p>",
              "rawMarkdown": "@deeppast "
            }
          ]
        },
        {
          "id": 3393301,
          "postDate": "2026-01-18T18:18:43.440Z",
          "content": "<p>Is Model B (the domain expert) a ByT5 model?</p>",
          "rawMarkdown": "Is Model B (the domain expert) a ByT5 model?",
          "replies": [
            {
              "id": 3393334,
              "postDate": "2026-01-18T20:04:20.610Z",
              "content": "<p>I used a ByT5-based sequence-to-sequence model with a reference-aligned preprocessing strategy. To improve robustness, I applied character-level noise augmentation and lexicon-based augmentation, along with a few additional optimization techniques during training.</p>\n<table>\n<thead>\n<tr>\n<th>Epoch</th>\n<th>Training Loss</th>\n<th>Validation Loss</th>\n<th>BLEU</th>\n<th>CHRF</th>\n<th>Geom Mean</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>4.039800</td>\n<td>3.053655</td>\n<td>0.308448</td>\n<td>6.956536</td>\n<td>1.464831</td>\n</tr>\n<tr>\n<td>2</td>\n<td>2.566600</td>\n<td>2.046551</td>\n<td>0.394575</td>\n<td>11.933742</td>\n<td>2.169968</td>\n</tr>\n<tr>\n<td>3</td>\n<td>2.003100</td>\n<td>1.728552</td>\n<td>3.788404</td>\n<td>17.906485</td>\n<td>8.236322</td>\n</tr>\n<tr>\n<td>…</td>\n<td>…</td>\n<td>…</td>\n<td>…</td>\n<td>…</td>\n<td>…</td>\n</tr>\n<tr>\n<td>23</td>\n<td>1.350800</td>\n<td>1.292178</td>\n<td>28.180715</td>\n<td>50.800508</td>\n<td>37.836420</td>\n</tr>\n<tr>\n<td>24</td>\n<td>1.347100</td>\n<td>1.290001</td>\n<td>28.441435</td>\n<td>51.086752</td>\n<td>38.117981</td>\n</tr>\n<tr>\n<td>25</td>\n<td>1.345700</td>\n<td>1.289753</td>\n<td>27.886274</td>\n<td>50.658614</td>\n<td>37.585635</td>\n</tr>\n</tbody>\n</table>\n<p>Despite promising offline metrics, the model did not generalize well to the leaderboard evaluation, leading to lower-than-expected scores. </p>",
              "rawMarkdown": "I used a ByT5-based sequence-to-sequence model with a reference-aligned preprocessing strategy. To improve robustness, I applied character-level noise augmentation and lexicon-based augmentation, along with a few additional optimization techniques during training.\n\n| Epoch | Training Loss | Validation Loss | BLEU      | CHRF      | Geom Mean |\n| ----- | ------------- | --------------- | --------- | --------- | --------- |\n| 1     | 4.039800      | 3.053655        | 0.308448  | 6.956536  | 1.464831  |\n| 2     | 2.566600      | 2.046551        | 0.394575  | 11.933742 | 2.169968  |\n| 3     | 2.003100      | 1.728552        | 3.788404  | 17.906485 | 8.236322  |\n| ...   | ...           | ...             | ...       | ...       | ...       |\n| 23    | 1.350800      | 1.292178        | 28.180715 | 50.800508 | 37.836420 |\n| 24    | 1.347100      | 1.290001        | 28.441435 | 51.086752 | 38.117981 |\n| 25    | 1.345700      | 1.289753        | 27.886274 | 50.658614 | 37.585635 |\n\nDespite promising offline metrics, the model did not generalize well to the leaderboard evaluation, leading to lower-than-expected scores. \n\n",
              "votes": 2
            },
            {
              "id": 3393346,
              "postDate": "2026-01-18T20:56:09.547Z",
              "rawMarkdown": "",
              "isDeleted": true
            },
            {
              "id": 3393350,
              "postDate": "2026-01-18T21:15:06.090Z",
              "content": "<p>This approach is not sentence-aligned. Training is done on full transliteration–translation pairs, without explicit sentence or segment boundary alignment.\nA lexicon-based augmentation step is used to normalize or substitute source-side tokens, but this operates at the token/word level, not at the sentence or segment level.\nThe main focus is reference-format alignment (punctuation, ellipsis, brackets, fractions) and ByT5-safe character handling, rather than semantic or sentence-level alignment.</p>",
              "rawMarkdown": "This approach is not sentence-aligned. Training is done on full transliteration–translation pairs, without explicit sentence or segment boundary alignment.\nA lexicon-based augmentation step is used to normalize or substitute source-side tokens, but this operates at the token/word level, not at the sentence or segment level.\nThe main focus is reference-format alignment (punctuation, ellipsis, brackets, fractions) and ByT5-safe character handling, rather than semantic or sentence-level alignment.",
              "votes": 2
            },
            {
              "id": 3393355,
              "postDate": "2026-01-18T21:19:31.803Z",
              "content": "<p>Thanks for your reply! Got it now</p>",
              "rawMarkdown": "Thanks for your reply! Got it now",
              "votes": 1
            },
            {
              "id": 3405501,
              "postDate": "2026-02-13T08:09:41.467Z",
              "content": "<p>Great analysis on the 'Pattern Matcher' trap. I'm curious about the training stability you observed. Did you encounter significant volatility in your validation scores (e.g., BLEU/ChrF) as the model began to overfit on templates? Specifically, did you notice a correlation between a sudden jump in scores and the emergence of the 'parrot' behavior (repetitive n-grams) in your validation samples?</p>",
              "rawMarkdown": "Great analysis on the 'Pattern Matcher' trap. I'm curious about the training stability you observed. Did you encounter significant volatility in your validation scores (e.g., BLEU/ChrF) as the model began to overfit on templates? Specifically, did you notice a correlation between a sudden jump in scores and the emergence of the 'parrot' behavior (repetitive n-grams) in your validation samples?"
            }
          ]
        }
      ]
    },
    {
      "id": 3393486,
      "postDate": "2026-01-19T05:49:17.807Z",
      "content": "<p>How can we redesign evaluation metrics to reward semantic fidelity instead of superficial n-gram conformity?</p>",
      "rawMarkdown": "How can we redesign evaluation metrics to reward semantic fidelity instead of superficial n-gram conformity?",
      "votes": 2
    },
    {
      "id": 3393924,
      "postDate": "2026-01-20T06:34:42.990Z",
      "content": "<p>High scores on the visible 33%?</p>",
      "rawMarkdown": " High scores on the visible 33%?"
    }
  ],
  "comments": [
    {
      "id": 3392935,
      "author_name": "Musa Peker",
      "author_url": "",
      "post_date": "2026-01-17T21:00:52.233000",
      "content": "<p>This is a brilliant and devastating analysis. It confirms exactly what I've been experiencing in my latest experiments. I want to share a concrete example that validates your \"Pattern Matcher vs. Translator\" hypothesis.</p>\n<p>I analyzed two distinct models:</p>\n<p><strong>Model A (The Parrot):</strong></p>\n<ul>\n<li>Method: Standard ByT5 with aggressive gap-filling regex.</li>\n<li>Behavior: It hallucinates standard phrases like \"The City,\" \"Silver,\" and \"Merchant\" to fill gaps. It often repeats n-grams.</li>\n<li>Public LB Score: 34.2</li>\n</ul>\n<p>Examples:\nid,translation</p>\n<ul>\n<li>0,\"Thus Kanesh colony, say to the  of our messengers, every single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single day.\"</li>\n<li>1,\"In the tablet of the City you wrote to me in the tablet of the City. This day whoever receives the silver, Daur or Ewa, will take the colony of my colony.\"</li>\n</ul>\n<p><strong>Model B (The Domain Expert):</strong></p>\n<ul>\n<li>Method: Reference-aligned training with minimal normalization, focused on semantic accuracy.</li>\n<li>Behavior: It correctly translates specific Old Assyrian trade terms.</li>\n<li>- shaddutum --&gt; \"Transport tariff\" (Instead of just \"silver\" or \"tax\"), </li>\n<li>- emārum ṣalmum --&gt; \"Black donkey\" (Instead of just \"donkey\" or \"goods\")</li>\n<li>- wabartum --&gt; \"Affiliated trading station\" (Instead of leaving it as \"Wabartum\")</li>\n<li>Public LB Score: 11.8 (!)</li>\n</ul>\n<p>Examples:\nid,translation</p>\n<ul>\n<li>0,From the Kanesh colony to a merchant: In accordance with our instructions we have seized us for an expense of black donkey.</li>\n<li>1,In accordance with our instructions we have received the silver on behalf of a merchant.</li>\n</ul>\n<p>The current Public LB metric (BLEU/CHRF) seems to heavily penalize Semantic Fidelity (Model B) while rewarding Structural Conformity (Model A).</p>\n<p>When Model B translates shaddutum as \"transport tariff,\" it likely gets a 0 score for that n-gram if the reference translation uses a generic term like \"tax\" or \"silver.\" meanwhile, Model A gets points for simply outputting the most statistically probable words (\"silver\", \"give\", \"city\"), even if the sentence meaning is hallucinated.</p>\n<p>Your probing framework proves that we are optimizing for a metric that measures \"Vibes\" rather than \"Translation.\" As you suggested, this creates a massive risk for the Private LB (hidden 67%), where out-of-distribution texts will likely break the pattern-matching models (Model A), while the semantically grounded models (Model B) should theoretically perform better.</p>\n<p>Thank you for providing the tools to diagnose this \"Metric Trap.\"</p>",
      "votes": 13,
      "replies": [
        {
          "id": 3393091,
          "author_name": "Aaron Bornstein",
          "author_url": "",
          "post_date": "2026-01-18T08:08:47.777000",
          "content": "<p>I agree I’m starting to think that some of the gains from data cleaning are also misleading they are essentially minimizing the false positive character match rate on the test distribution but not necessarily increasing the quality of the translations.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 3393147,
          "author_name": "John Doge",
          "author_url": "",
          "post_date": "2026-01-18T10:26:46.257000",
          "content": "<p>BLEU is simply not suitable for this task. Private LB can also be OOD in many ways, there is no guarantee that a semantically grounded model will perform better under metrics like this. The hosts should consider replacing BLEU with BLEURT - there is still enough time.</p>",
          "votes": 11,
          "replies": [
            {
              "id": 3393344,
              "author_name": "Aaron Bornstein",
              "author_url": "",
              "post_date": "2026-01-18T20:41:26.290000",
              "content": "<p><a href=\"https://www.kaggle.com/deeppast\" target=\"_blank\">@deeppast</a> </p>",
              "votes": 0,
              "replies": []
            }
          ]
        },
        {
          "id": 3393301,
          "author_name": "hongan",
          "author_url": "",
          "post_date": "2026-01-18T18:18:43.440000",
          "content": "<p>Is Model B (the domain expert) a ByT5 model?</p>",
          "votes": 0,
          "replies": [
            {
              "id": 3393334,
              "author_name": "Musa Peker",
              "author_url": "",
              "post_date": "2026-01-18T20:04:20.610000",
              "content": "<p>I used a ByT5-based sequence-to-sequence model with a reference-aligned preprocessing strategy. To improve robustness, I applied character-level noise augmentation and lexicon-based augmentation, along with a few additional optimization techniques during training.</p>\n<table>\n<thead>\n<tr>\n<th>Epoch</th>\n<th>Training Loss</th>\n<th>Validation Loss</th>\n<th>BLEU</th>\n<th>CHRF</th>\n<th>Geom Mean</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>4.039800</td>\n<td>3.053655</td>\n<td>0.308448</td>\n<td>6.956536</td>\n<td>1.464831</td>\n</tr>\n<tr>\n<td>2</td>\n<td>2.566600</td>\n<td>2.046551</td>\n<td>0.394575</td>\n<td>11.933742</td>\n<td>2.169968</td>\n</tr>\n<tr>\n<td>3</td>\n<td>2.003100</td>\n<td>1.728552</td>\n<td>3.788404</td>\n<td>17.906485</td>\n<td>8.236322</td>\n</tr>\n<tr>\n<td>…</td>\n<td>…</td>\n<td>…</td>\n<td>…</td>\n<td>…</td>\n<td>…</td>\n</tr>\n<tr>\n<td>23</td>\n<td>1.350800</td>\n<td>1.292178</td>\n<td>28.180715</td>\n<td>50.800508</td>\n<td>37.836420</td>\n</tr>\n<tr>\n<td>24</td>\n<td>1.347100</td>\n<td>1.290001</td>\n<td>28.441435</td>\n<td>51.086752</td>\n<td>38.117981</td>\n</tr>\n<tr>\n<td>25</td>\n<td>1.345700</td>\n<td>1.289753</td>\n<td>27.886274</td>\n<td>50.658614</td>\n<td>37.585635</td>\n</tr>\n</tbody>\n</table>\n<p>Despite promising offline metrics, the model did not generalize well to the leaderboard evaluation, leading to lower-than-expected scores. </p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 3393346,
              "author_name": "",
              "author_url": "",
              "post_date": "2026-01-18T20:56:09.547000",
              "content": "",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3393350,
              "author_name": "Musa Peker",
              "author_url": "",
              "post_date": "2026-01-18T21:15:06.090000",
              "content": "<p>This approach is not sentence-aligned. Training is done on full transliteration–translation pairs, without explicit sentence or segment boundary alignment.\nA lexicon-based augmentation step is used to normalize or substitute source-side tokens, but this operates at the token/word level, not at the sentence or segment level.\nThe main focus is reference-format alignment (punctuation, ellipsis, brackets, fractions) and ByT5-safe character handling, rather than semantic or sentence-level alignment.</p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 3393355,
              "author_name": "hongan",
              "author_url": "",
              "post_date": "2026-01-18T21:19:31.803000",
              "content": "<p>Thanks for your reply! Got it now</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3405501,
              "author_name": "cswwp",
              "author_url": "",
              "post_date": "2026-02-13T08:09:41.467000",
              "content": "<p>Great analysis on the 'Pattern Matcher' trap. I'm curious about the training stability you observed. Did you encounter significant volatility in your validation scores (e.g., BLEU/ChrF) as the model began to overfit on templates? Specifically, did you notice a correlation between a sudden jump in scores and the emergence of the 'parrot' behavior (repetitive n-grams) in your validation samples?</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3393486,
      "author_name": "Soham_Deshpande",
      "author_url": "",
      "post_date": "2026-01-19T05:49:17.807000",
      "content": "<p>How can we redesign evaluation metrics to reward semantic fidelity instead of superficial n-gram conformity?</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 3393924,
      "author_name": "Navneet",
      "author_url": "",
      "post_date": "2026-01-20T06:34:42.990000",
      "content": "<p>High scores on the visible 33%?</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3392924": "I’ve been digging into the high-scoring baseline models (34+ LB) with a systematic probing framework I developed to understand the strengths and weaknesses of these models.\n\nI wanted to share some evidence suggesting we might be seriously overfitting to the public leaderboard. After running 104 paired perturbation probes, my analysis shows the best ByT5 models are effectively functioning as a sophisticated pattern matchers (~60% translation, ~40% memorization) rather than a general translator. \n\nWhile they handle negation and numbers perfectly, the best public models fail on many isolated vocabulary and often fail to successfully copies novel proper names, hallucinating generic trade sentences around single words like \"Disregard that an unclean man...\". \n\nThis strongly suggests the model is exploiting corpus-specific templates rather than learning core semantics, it's important to note that high scores on the visible 33% of the test set are likely to degrade significantly on the hidden 67% if they are out of domain.\n\nSharing the notebook below hope these probes can help identitfy new techniques for generalizing the models beyond data cleaning tricks.\n\nhttps://www.kaggle.com/code/aaronbornstein/probing-leader-board-overfitting-34-baseline",
    "3392935": "This is a brilliant and devastating analysis. It confirms exactly what I've been experiencing in my latest experiments. I want to share a concrete example that validates your \"Pattern Matcher vs. Translator\" hypothesis.\n\nI analyzed two distinct models:\n\n**Model A (The Parrot):**\n- Method: Standard ByT5 with aggressive gap-filling regex.\n- Behavior: It hallucinates standard phrases like \"The City,\" \"Silver,\" and \"Merchant\" to fill gaps. It often repeats n-grams.\n- Public LB Score: 34.2\n\nExamples:\nid,translation\n- 0,\"Thus Kanesh colony, say to the <big_gap> of our messengers, every single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single single day.\"\n- 1,\"In the tablet of the City you wrote to me in the tablet of the City. This day whoever receives the silver, Daur or Ewa, will take the colony of my colony.\"\n\n**Model B (The Domain Expert):**\n- Method: Reference-aligned training with minimal normalization, focused on semantic accuracy.\n- Behavior: It correctly translates specific Old Assyrian trade terms.\n- - shaddutum --> \"Transport tariff\" (Instead of just \"silver\" or \"tax\"), \n- - emārum ṣalmum --> \"Black donkey\" (Instead of just \"donkey\" or \"goods\")\n- - wabartum --> \"Affiliated trading station\" (Instead of leaving it as \"Wabartum\")\n- Public LB Score: 11.8 (!)\n\nExamples:\nid,translation\n- 0,From the Kanesh colony to a merchant: In accordance with our instructions we have seized us for an expense of black donkey.\n- 1,In accordance with our instructions we have received the silver on behalf of a merchant.\n\n\nThe current Public LB metric (BLEU/CHRF) seems to heavily penalize Semantic Fidelity (Model B) while rewarding Structural Conformity (Model A).\n\nWhen Model B translates shaddutum as \"transport tariff,\" it likely gets a 0 score for that n-gram if the reference translation uses a generic term like \"tax\" or \"silver.\" meanwhile, Model A gets points for simply outputting the most statistically probable words (\"silver\", \"give\", \"city\"), even if the sentence meaning is hallucinated.\n\nYour probing framework proves that we are optimizing for a metric that measures \"Vibes\" rather than \"Translation.\" As you suggested, this creates a massive risk for the Private LB (hidden 67%), where out-of-distribution texts will likely break the pattern-matching models (Model A), while the semantically grounded models (Model B) should theoretically perform better.\n\nThank you for providing the tools to diagnose this \"Metric Trap.\"",
    "3393486": "How can we redesign evaluation metrics to reward semantic fidelity instead of superficial n-gram conformity?",
    "3393924": " High scores on the visible 33%?"
  }
}