{
  "id": 670046,
  "title": "English pre-processing ",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/670046",
  "author_name": "Alexandru Craevschi",
  "post_date": "2026-01-25T18:43:12.269000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hey everyone, </p>\n<p>So far, there have been s lot of discussion on how to preprocess the Old Assyrian text to make it cleaner and make it match the test set. </p>\n<p>I wonder what have others done (if anything) to pre-process the English text. I imagine that arguably matching the style of the translated language is arguably even more important for the metric than the source text. </p>\n<p>What I imagine many people are already doing is: </p>\n<ul>\n<li>transforming different characters that represent gaps into <code>&lt;gap&gt;</code> or <code>&lt;big_gap&gt;</code></li>\n<li>transform fractions into Unicode characters </li>\n<li>remove comments like <code>(plural)</code> or <code>(fem.)</code>. </li>\n</ul>\n<p>The last point is actually tricky as sometimes the output text might contain something like an implicitly meant determiner, like <code>(this) house</code>. So it's probably not a good idea to remove everything in round brackets. </p>\n<p>Is there something else that you do that you noticed boosts the metric? </p>",
  "messages": [
    {
      "id": 3396748,
      "postDate": "2026-01-25T18:43:12.270Z",
      "content": "<p>Hey everyone, </p>\n<p>So far, there have been s lot of discussion on how to preprocess the Old Assyrian text to make it cleaner and make it match the test set. </p>\n<p>I wonder what have others done (if anything) to pre-process the English text. I imagine that arguably matching the style of the translated language is arguably even more important for the metric than the source text. </p>\n<p>What I imagine many people are already doing is: </p>\n<ul>\n<li>transforming different characters that represent gaps into <code>&lt;gap&gt;</code> or <code>&lt;big_gap&gt;</code></li>\n<li>transform fractions into Unicode characters </li>\n<li>remove comments like <code>(plural)</code> or <code>(fem.)</code>. </li>\n</ul>\n<p>The last point is actually tricky as sometimes the output text might contain something like an implicitly meant determiner, like <code>(this) house</code>. So it's probably not a good idea to remove everything in round brackets. </p>\n<p>Is there something else that you do that you noticed boosts the metric? </p>",
      "rawMarkdown": "Hey everyone, \n\nSo far, there have been s lot of discussion on how to preprocess the Old Assyrian text to make it cleaner and make it match the test set. \n\nI wonder what have others done (if anything) to pre-process the English text. I imagine that arguably matching the style of the translated language is arguably even more important for the metric than the source text. \n\nWhat I imagine many people are already doing is: \n- transforming different characters that represent gaps into `<gap>` or `<big_gap>`\n- transform fractions into Unicode characters \n- remove comments like `(plural)` or `(fem.)`. \n\nThe last point is actually tricky as sometimes the output text might contain something like an implicitly meant determiner, like `(this) house`. So it's probably not a good idea to remove everything in round brackets. \n\nIs there something else that you do that you noticed boosts the metric? ",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3396748": "Hey everyone, \n\nSo far, there have been s lot of discussion on how to preprocess the Old Assyrian text to make it cleaner and make it match the test set. \n\nI wonder what have others done (if anything) to pre-process the English text. I imagine that arguably matching the style of the translated language is arguably even more important for the metric than the source text. \n\nWhat I imagine many people are already doing is: \n- transforming different characters that represent gaps into `<gap>` or `<big_gap>`\n- transform fractions into Unicode characters \n- remove comments like `(plural)` or `(fem.)`. \n\nThe last point is actually tricky as sometimes the output text might contain something like an implicitly meant determiner, like `(this) house`. So it's probably not a good idea to remove everything in round brackets. \n\nIs there something else that you do that you noticed boosts the metric? "
  }
}