{
  "id": 674342,
  "title": "Cleaning Translations in Training vs Test Data",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/674342",
  "author_name": "",
  "post_date": "2026-02-19T23:10:54.187387900Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>It is easy to clean the translation column in the training data so the model is not affected by academic additions like:</p>\n<ol>\n<li>… indicating missing parts</li>\n<li>\"\" at the end of lines or tablets</li>\n<li>Incomplete words like ru-x</li>\n</ol>\n<p>If the model is trained on clean translations, I wonder if it is possible to modify the translation column in the test data before applying the model to the samples.\nSince we do not have access to the test labels directly and only see them upon submission, my question is: can the translation column be modified using only the path to test.csv durring submission?</p>",
  "messages": [
    {
      "id": "3408158",
      "postDate": "02/19/2026 23:10:54",
      "content": "<p>It is easy to clean the translation column in the training data so the model is not affected by academic additions like:</p>\n<ol>\n<li>… indicating missing parts</li>\n<li>\"\" at the end of lines or tablets</li>\n<li>Incomplete words like ru-x</li>\n</ol>\n<p>If the model is trained on clean translations, I wonder if it is possible to modify the translation column in the test data before applying the model to the samples.\nSince we do not have access to the test labels directly and only see them upon submission, my question is: can the translation column be modified using only the path to test.csv durring submission?</p>",
      "rawMarkdown": "It is easy to clean the translation column in the training data so the model is not affected by academic additions like:\n\n1. ... indicating missing parts\n2. \"\" at the end of lines or tablets\n3. Incomplete words like ru-x\n\nIf the model is trained on clean translations, I wonder if it is possible to modify the translation column in the test data before applying the model to the samples.\nSince we do not have access to the test labels directly and only see them upon submission, my question is: can the translation column be modified using only the path to test.csv durring submission?",
      "votes": null
    },
    {
      "id": "3408253",
      "postDate": "02/20/2026 07:02:53",
      "content": "<p>I don't think we have access to the test labels upon submission</p>",
      "rawMarkdown": "I don't think we have access to the test labels upon submission",
      "votes": null
    },
    {
      "id": "3408386",
      "postDate": "02/20/2026 14:14:38",
      "content": "<p>Before testing our model on the test data, we do not let it make predictions directly. We first apply the same preprocessing steps that were used on the training data. Even in production, for example with GPT, the model does not process raw text as plain letters. It first converts the text into embeddings. So I am wondering if we normally write code that explicitly applies the same preprocessing to the test dataset by calling the test data directory during submission..</p>",
      "rawMarkdown": "Before testing our model on the test data, we do not let it make predictions directly. We first apply the same preprocessing steps that were used on the training data. Even in production, for example with GPT, the model does not process raw text as plain letters. It first converts the text into embeddings. So I am wondering if we normally write code that explicitly applies the same preprocessing to the test dataset by calling the test data directory during submission..",
      "votes": null
    },
    {
      "id": "3408399",
      "postDate": "02/20/2026 14:41:22",
      "content": "<p>You can do whatever you want with the test transliterations. But you dont have an access to the labels, i.e. translations</p>",
      "rawMarkdown": "You can do whatever you want with the test transliterations. But you dont have an access to the labels, i.e. translations",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3408253,
      "author_name": "steubk",
      "author_url": "",
      "post_date": "02/20/2026 07:02:53",
      "content": "<p>I don't think we have access to the test labels upon submission</p>",
      "votes": null,
      "replies": [
        {
          "id": 3408386,
          "author_name": "tarekziad",
          "author_url": "",
          "post_date": "02/20/2026 14:14:38",
          "content": "<p>Before testing our model on the test data, we do not let it make predictions directly. We first apply the same preprocessing steps that were used on the training data. Even in production, for example with GPT, the model does not process raw text as plain letters. It first converts the text into embeddings. So I am wondering if we normally write code that explicitly applies the same preprocessing to the test dataset by calling the test data directory during submission..</p>",
          "votes": null,
          "replies": [
            {
              "id": 3408399,
              "author_name": "samson8",
              "author_url": "",
              "post_date": "02/20/2026 14:41:22",
              "content": "<p>You can do whatever you want with the test transliterations. But you dont have an access to the labels, i.e. translations</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3408158": "It is easy to clean the translation column in the training data so the model is not affected by academic additions like:\n\n1. ... indicating missing parts\n2. \"\" at the end of lines or tablets\n3. Incomplete words like ru-x\n\nIf the model is trained on clean translations, I wonder if it is possible to modify the translation column in the test data before applying the model to the samples.\nSince we do not have access to the test labels directly and only see them upon submission, my question is: can the translation column be modified using only the path to test.csv durring submission?",
    "3408253": "I don't think we have access to the test labels upon submission",
    "3408386": "Before testing our model on the test data, we do not let it make predictions directly. We first apply the same preprocessing steps that were used on the training data. Even in production, for example with GPT, the model does not process raw text as plain letters. It first converts the text into embeddings. So I am wondering if we normally write code that explicitly applies the same preprocessing to the test dataset by calling the test data directory during submission..",
    "3408399": "You can do whatever you want with the test transliterations. But you dont have an access to the labels, i.e. translations"
  },
  "source": "meta"
}