Our approach is based on an ensemble of ByT5 models supported by a multi-stage, Gemini-driven data construction pipeline. Final predictions were generated using a Weighted MBR Decoding strategy.
\nCore Differentiators:
\nWe evaluated multiple architectures, including decoder-only LLMs (e.g., Qwen series), but they underperformed due to:
\nBest-performing models:
\nByT5 variants: We experimented with base, large, and XL models. Although larger models sometimes achieved higher private scores, they were less stable and underperformed on validation. We therefore prioritized the robustness of ByT5-base.
\nPerformance was primarily bottlenecked by data quality and alignment, not model capacity. We built a unified corpus from multiple heterogeneous sources using a Gemini-based pipeline.
\nTraining Data (Document → Sentence Alignment):\nThe original dataset was provided at the document level. We used Gemini to segment it into sentence-level pairs, significantly improving alignment and increasing the number of effective training samples.
Publications (OCR Data):\nNoisy OCR text was segmented and normalized using Gemini, improving coverage of rare tokens despite high noise levels.
Published Text (Pseudo-Labeling):\nWe generated pseudo-parallel data via few-shot prompting with injected Akkadian lexicons, improving consistency for rare words and named entities.
PDF Sources (Multimodal Extraction):\nFor image-heavy PDFs, we designed a prompt pipeline (few-shot + lexicon) to directly extract aligned sentence pairs.
Cross-Lingual Normalization:\nNon-English segments (e.g., Turkish) were translated and aligned into English using Gemini.
Transforming heterogeneous, noisy, document-level data into sentence-aligned pairs was the single most impactful improvement in our pipeline.
\nWhile preprocessing had limited direct impact on validation scores, it was essential for:
\nInstead of relying solely on standard cross-validation, we focused on data-centric validation:
\nThis was critical due to:
\nObservation:
\nSingle-model plateau: ~38 LB score
Increasing sequence length did not help → bottleneck was data quality
Discarded methods:
\nFinal system: 3-model ensemble with cross-model candidate pooling
\nPipeline:
\nCandidate Generation
\nCross-Pooling
\nWe implemented a metric-aware MBR decoding strategy, optimized using Optuna.
\nTuned parameters:
\nKey idea:
\nInstead of selecting the highest-probability output, we select the candidate that maximizes agreement under evaluation metrics across models.
\nResult:
\nThis competition reinforced a key lesson:
\n\n\nIn low-resource settings, data quality and alignment dominate model scaling.
\n
Our strongest gains came from:
\nOur approach is based on an ensemble of ByT5 models supported by a multi-stage, Gemini-driven data construction pipeline. Final predictions were generated using a Weighted MBR Decoding strategy.
\nCore Differentiators:
\nWe evaluated multiple architectures, including decoder-only LLMs (e.g., Qwen series), but they underperformed due to:
\nBest-performing models:
\nByT5 variants: We experimented with base, large, and XL models. Although larger models sometimes achieved higher private scores, they were less stable and underperformed on validation. We therefore prioritized the robustness of ByT5-base.
\nPerformance was primarily bottlenecked by data quality and alignment, not model capacity. We built a unified corpus from multiple heterogeneous sources using a Gemini-based pipeline.
\nTraining Data (Document → Sentence Alignment):\nThe original dataset was provided at the document level. We used Gemini to segment it into sentence-level pairs, significantly improving alignment and increasing the number of effective training samples.
Publications (OCR Data):\nNoisy OCR text was segmented and normalized using Gemini, improving coverage of rare tokens despite high noise levels.
Published Text (Pseudo-Labeling):\nWe generated pseudo-parallel data via few-shot prompting with injected Akkadian lexicons, improving consistency for rare words and named entities.
PDF Sources (Multimodal Extraction):\nFor image-heavy PDFs, we designed a prompt pipeline (few-shot + lexicon) to directly extract aligned sentence pairs.
Cross-Lingual Normalization:\nNon-English segments (e.g., Turkish) were translated and aligned into English using Gemini.
Transforming heterogeneous, noisy, document-level data into sentence-aligned pairs was the single most impactful improvement in our pipeline.
\nWhile preprocessing had limited direct impact on validation scores, it was essential for:
\nInstead of relying solely on standard cross-validation, we focused on data-centric validation:
\nThis was critical due to:
\nObservation:
\nSingle-model plateau: ~38 LB score
Increasing sequence length did not help → bottleneck was data quality
Discarded methods:
\nFinal system: 3-model ensemble with cross-model candidate pooling
\nPipeline:
\nCandidate Generation
\nCross-Pooling
\nWe implemented a metric-aware MBR decoding strategy, optimized using Optuna.
\nTuned parameters:
\nKey idea:
\nInstead of selecting the highest-probability output, we select the candidate that maximizes agreement under evaluation metrics across models.
\nResult:
\nThis competition reinforced a key lesson:
\n\n\nIn low-resource settings, data quality and alignment dominate model scaling.
\n
Our strongest gains came from:
\n