{
  "id": 680874,
  "title": "Local validation vs public LB mismatch after recent updates — clarification on normalization/eval details?",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/680874",
  "author_name": "",
  "post_date": "2026-03-11T13:04:03.324165500Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n<p>I've noticed a significant discrepancy between local validation and public rating, even while maintaining the determinism of the output and adhering to the published recommendations for normalisation .</p>\n<p>Without going into details related to a specific model, I would like to receive clarification on several points related to evaluation:</p>\n<p>Are there any additional normalization details used on the competition side beyond the published  / ellipsis guidance?</p>\n<p>Should participants assume that local validation built from the provided training data may differ substantially from the distribution used in the public leaderboard?</p>\n<p>Are there any known preprocessing specifics that are especially important for matching leaderboard behavior (for example around whitespace, repeated gap markers, punctuation variants, or similar edge cases)?</p>\n<p>Is the current public leaderboard slice particularly small/noisy, such that local gains may not transfer reliably even when the local metric improves consistently?</p>\n<p>I am not asking about hidden labels, only whether there are any evaluation or normalization details that competitors should mirror more closely in order to make local validation more predictive.</p>\n<p>Thanks a lot.</p>",
  "messages": [
    {
      "id": "3419694",
      "postDate": "03/11/2026 13:04:03",
      "content": "<p>Hi everyone,</p>\n<p>I've noticed a significant discrepancy between local validation and public rating, even while maintaining the determinism of the output and adhering to the published recommendations for normalisation .</p>\n<p>Without going into details related to a specific model, I would like to receive clarification on several points related to evaluation:</p>\n<p>Are there any additional normalization details used on the competition side beyond the published  / ellipsis guidance?</p>\n<p>Should participants assume that local validation built from the provided training data may differ substantially from the distribution used in the public leaderboard?</p>\n<p>Are there any known preprocessing specifics that are especially important for matching leaderboard behavior (for example around whitespace, repeated gap markers, punctuation variants, or similar edge cases)?</p>\n<p>Is the current public leaderboard slice particularly small/noisy, such that local gains may not transfer reliably even when the local metric improves consistently?</p>\n<p>I am not asking about hidden labels, only whether there are any evaluation or normalization details that competitors should mirror more closely in order to make local validation more predictive.</p>\n<p>Thanks a lot.</p>",
      "rawMarkdown": "Hi everyone,\n\nI've noticed a significant discrepancy between local validation and public rating, even while maintaining the determinism of the output and adhering to the published recommendations for normalisation <gap>.\n\nWithout going into details related to a specific model, I would like to receive clarification on several points related to evaluation:\n\nAre there any additional normalization details used on the competition side beyond the published <gap> / ellipsis guidance?\n\nShould participants assume that local validation built from the provided training data may differ substantially from the distribution used in the public leaderboard?\n\nAre there any known preprocessing specifics that are especially important for matching leaderboard behavior (for example around whitespace, repeated gap markers, punctuation variants, or similar edge cases)?\n\nIs the current public leaderboard slice particularly small/noisy, such that local gains may not transfer reliably even when the local metric improves consistently?\n\nI am not asking about hidden labels, only whether there are any evaluation or normalization details that competitors should mirror more closely in order to make local validation more predictive.\n\nThanks a lot.",
      "votes": null
    },
    {
      "id": "3419819",
      "postDate": "03/11/2026 18:47:48",
      "content": "<p>I think one issue you should consider is dataset contamination. If you use part of the provided training dataset as your validation set, then you must choose your val set carefully, otherwise there will be leakage, giving the illusion that your model is doing well on your local val set.</p>",
      "rawMarkdown": "I think one issue you should consider is dataset contamination. If you use part of the provided training dataset as your validation set, then you must choose your val set carefully, otherwise there will be leakage, giving the illusion that your model is doing well on your local val set.",
      "votes": null
    },
    {
      "id": "3420926",
      "postDate": "03/14/2026 07:39:15",
      "content": "<p>if you will try on local validation and your input are normal text eg -&gt;\"ITI-AB UD 21-KAM₂ 1-d-AG—šal-lim-šu₂-nu LU₂-DUB.SAR—LUGAL LU₂-DUB.SAR—LUGAL\" your score will will mostly be high but text with … have low score like of public LB eg -&gt; \"… i-pad-dan-ni ki-ša₂-di i-ru-… BAL.TIL KI a-na e-re-bi e-ti-qu pi-i-šu₂ ip-te-e-ma ki-ma LU₂ it-ti-šu₂ …\" and also post process generated text from … to &lt; gap &gt; (no space in btw &lt;&gt;)it will boost your score</p>\n<p>i have tried this on many local validation set, with … it give score mostly around my LB score but without … it give 45+ score</p>",
      "rawMarkdown": "if you will try on local validation and your input are normal text eg ->\"ITI-AB UD 21-KAM₂ 1-d-AG—šal-lim-šu₂-nu LU₂-DUB.SAR—LUGAL LU₂-DUB.SAR—LUGAL\" your score will will mostly be high but text with ... have low score like of public LB eg -> \"... i-pad-dan-ni ki-ša₂-di i-ru-... BAL.TIL KI a-na e-re-bi e-ti-qu pi-i-šu₂ ip-te-e-ma ki-ma LU₂ it-ti-šu₂ ...\" and also post process generated text from ... to < gap > (no space in btw <>)it will boost your score\n\ni have tried this on many local validation set, with ... it give score mostly around my LB score but without ... it give 45+ score",
      "votes": null
    },
    {
      "id": "3420994",
      "postDate": "03/14/2026 11:45:01",
      "content": "<p>Thanks, this is helpful. I see something similar: examples with ellipses / gaps seem to behave much closer to the public LB than “clean” lines, so a local split that under-represents those cases can definitely look over-optimistic.</p>\n<p>That said, I’m not sure this is the whole story by itself. In our case the mismatch may also come from a combination of normalization details, split composition, and the fact that the public slice is quite small/noisy.</p>\n<p>So my current takeaway is: ellipsis-heavy / gap-heavy examples probably deserve their own local subset, rather than relying on a single aggregate validation score. Thanks for pointing that out.</p>",
      "rawMarkdown": "Thanks, this is helpful. I see something similar: examples with ellipses / gaps seem to behave much closer to the public LB than “clean” lines, so a local split that under-represents those cases can definitely look over-optimistic.\n\nThat said, I’m not sure this is the whole story by itself. In our case the mismatch may also come from a combination of normalization details, split composition, and the fact that the public slice is quite small/noisy.\n\nSo my current takeaway is: ellipsis-heavy / gap-heavy examples probably deserve their own local subset, rather than relying on a single aggregate validation score. Thanks for pointing that out.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3419819,
      "author_name": "anaphase21",
      "author_url": "",
      "post_date": "03/11/2026 18:47:48",
      "content": "<p>I think one issue you should consider is dataset contamination. If you use part of the provided training dataset as your validation set, then you must choose your val set carefully, otherwise there will be leakage, giving the illusion that your model is doing well on your local val set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3420926,
      "author_name": "adarsh2626",
      "author_url": "",
      "post_date": "03/14/2026 07:39:15",
      "content": "<p>if you will try on local validation and your input are normal text eg -&gt;\"ITI-AB UD 21-KAM₂ 1-d-AG—šal-lim-šu₂-nu LU₂-DUB.SAR—LUGAL LU₂-DUB.SAR—LUGAL\" your score will will mostly be high but text with … have low score like of public LB eg -&gt; \"… i-pad-dan-ni ki-ša₂-di i-ru-… BAL.TIL KI a-na e-re-bi e-ti-qu pi-i-šu₂ ip-te-e-ma ki-ma LU₂ it-ti-šu₂ …\" and also post process generated text from … to &lt; gap &gt; (no space in btw &lt;&gt;)it will boost your score</p>\n<p>i have tried this on many local validation set, with … it give score mostly around my LB score but without … it give 45+ score</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3420994,
      "author_name": "alessiosantos",
      "author_url": "",
      "post_date": "03/14/2026 11:45:01",
      "content": "<p>Thanks, this is helpful. I see something similar: examples with ellipses / gaps seem to behave much closer to the public LB than “clean” lines, so a local split that under-represents those cases can definitely look over-optimistic.</p>\n<p>That said, I’m not sure this is the whole story by itself. In our case the mismatch may also come from a combination of normalization details, split composition, and the fact that the public slice is quite small/noisy.</p>\n<p>So my current takeaway is: ellipsis-heavy / gap-heavy examples probably deserve their own local subset, rather than relying on a single aggregate validation score. Thanks for pointing that out.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3419694": "Hi everyone,\n\nI've noticed a significant discrepancy between local validation and public rating, even while maintaining the determinism of the output and adhering to the published recommendations for normalisation <gap>.\n\nWithout going into details related to a specific model, I would like to receive clarification on several points related to evaluation:\n\nAre there any additional normalization details used on the competition side beyond the published <gap> / ellipsis guidance?\n\nShould participants assume that local validation built from the provided training data may differ substantially from the distribution used in the public leaderboard?\n\nAre there any known preprocessing specifics that are especially important for matching leaderboard behavior (for example around whitespace, repeated gap markers, punctuation variants, or similar edge cases)?\n\nIs the current public leaderboard slice particularly small/noisy, such that local gains may not transfer reliably even when the local metric improves consistently?\n\nI am not asking about hidden labels, only whether there are any evaluation or normalization details that competitors should mirror more closely in order to make local validation more predictive.\n\nThanks a lot.",
    "3419819": "I think one issue you should consider is dataset contamination. If you use part of the provided training dataset as your validation set, then you must choose your val set carefully, otherwise there will be leakage, giving the illusion that your model is doing well on your local val set.",
    "3420926": "if you will try on local validation and your input are normal text eg ->\"ITI-AB UD 21-KAM₂ 1-d-AG—šal-lim-šu₂-nu LU₂-DUB.SAR—LUGAL LU₂-DUB.SAR—LUGAL\" your score will will mostly be high but text with ... have low score like of public LB eg -> \"... i-pad-dan-ni ki-ša₂-di i-ru-... BAL.TIL KI a-na e-re-bi e-ti-qu pi-i-šu₂ ip-te-e-ma ki-ma LU₂ it-ti-šu₂ ...\" and also post process generated text from ... to < gap > (no space in btw <>)it will boost your score\n\ni have tried this on many local validation set, with ... it give score mostly around my LB score but without ... it give 45+ score",
    "3420994": "Thanks, this is helpful. I see something similar: examples with ellipses / gaps seem to behave much closer to the public LB than “clean” lines, so a local split that under-represents those cases can definitely look over-optimistic.\n\nThat said, I’m not sure this is the whole story by itself. In our case the mismatch may also come from a combination of normalization details, split composition, and the fact that the public slice is quite small/noisy.\n\nSo my current takeaway is: ellipsis-heavy / gap-heavy examples probably deserve their own local subset, rather than relying on a single aggregate validation score. Thanks for pointing that out."
  },
  "source": "meta"
}