{ "id": 684866, "title": "[12th] NotebookLM & Single ByT5-Base Model", "url": "/competitions/deep-past-initiative-machine-translation/writeups/12th-place-notebooklm-and-single-byt5-base-model", "author_name": "", "post_date": "2026-03-26T10:11:32.493Z", "votes": 5, "comment_count": 1, "views": 0, "content": "

First of all , I want to thank the organizers for such an interesting competition and I also want to congratulate the winners for the hard work they've done.
\nSecond, I also want to thank my family for the support, and thank my teammate and brother @hasinarazafindrazaka for his ML experience which has given us a substantial boost to the score.

\n

Our Humble Approach

\n

We have, extensively, made use of Google's NotebookLM platform for data extraction, data preprocessing, and pseudo-labelling.

\n

We used a single Byt5-base model fine tuned with three stages:

\n\n

How we built our training data

\n

Creating a csv of sentence pairs with NotebookLM is quite straightforward with its reasoning model of 1 million token context length, and an OCR of a 'refined quality'.

\n

How extracted data from PDFs

\n

For most of the pdf that we had to deal with (like the AKT series), they had to be provided as image inputs due to their quality.

\n

So, we basically screenshotted every single tablet, and processed them with NotebookLM with a small batch of 1-5 tablets (1-10 images) at a time, hopefully allowing the model to focus more and avoid hallucinations.
\nDoing so resulted in more sentence pairs than giving larger batches of tablets.
\n\"\"

\n

We had found out that we could speed up the process by opening multiple notebook tabs, we just needed to be organized.

\n

How we tried to recover original transliteration from publications.csv and CAD

\n

After extracting some tablets and OA sentences, we made use of their tablet reference (eg: BIN 4 , CCT 2) to merge them with their original transliteration found in published_text.csv.

\n

To do so, we had to ask NotebookLM to find the 'aliases' of the tablet reference in published_text.csv, as hard coding a solution wasn't obvious… We gave it as input 2 files , one with 'aliases' and 'note' columns from published_text.csv and the other the list of tablet reference.

\n

Then we batched tablets/sentences with ocr_transliteration, ocr_translation merged with original transliteration, and processed them with NotebookLM

\n

How we built a small custom data

\n

As we processed data, we 've got a little bit familiar with some basic akkadian formulas, so we had decided to create a custom data that we filled with random names from onomasticon.csv and random numbers with a 0.2 probability of getting < gap > for both
\nThe custom data was mostly about Seal, Witness, From … To , month eponym, and 'owes' statements

\n

We also asked NotebookLM to split some of the training data (train.csv), but as we already had sentences from PDFs, we asked him to replace names with PN and numbers with NUM that we randomly replaced later

\n

Pseudo-labelling:

\n

We made use of the untranslated texts from published_texts.csv that we asked NotebookLM to translate and split.
\nThe translation quality was not like a 'SIG5 copper' but we believed there could be some value that we could get out of it

\n

ŠU.NÍGIN, we got a total of ~81K unique pairs: ~28K human-labelled (from PDFs, train.csv included), ~11K of custom data, ~3K (from publications.csv, … we didn't have much time), and pseudo-labelled data for the rest

\n

Training and Model

\n

We used a single ByT5-base model

\n

We have split the fine tuning into 3 stages:

\n
- 1st stage with all available data , we called it 'pretraining':
\n

With 81K pairs, a batch size of 8, learning rate of 5e-5, and 5 epoches
\nThe purpose was that the model would learn akkadian in general and have some good 'momentum'

\n

Submitting at this stage would give us ~35+, which was less than a model that was trained on human labelled only which could get up to ~37 for 25K pairs (at that time).

\n
- 2nd stage with only human labelled data:
\n

With 28K pairs, batch size of 2, lr 1e-5 , and 2 epochs
\nThe hope was that the model output would align more with human labelled data and correct all biases from pseudo-labelling.

\n

Submitting at this stage would give us 38+

\n

On our local validation with these 2 stages we could see a +2 to +3 BLEU score boost compared to a model trained only on human labelled data

\n
- 3rd stage LoRA with only the competition training data:
\n

With 1.5K , the main purpose was to align the output vocabulary of the model with the competition train.
\nSubmitting at this stage would give us ~39

\n

Some notes

\n\n

To finish , we want to give tribute to all Assyriologists out there for the truly amazing work they've done , and the Google Team for providing us advanced AI tool and really powerful pretrained models to work with.

", "messages": [ { "id": "3429056", "postDate": "03/26/2026 09:48:22", "content": "

First of all , I want to thank the organizers for such an interesting competition and I also want to congratulate the winners for the hard work they've done.
\nSecond, I also want to thank my family for the support, and thank my teammate and brother @hasinarazafindrazaka for his ML experience which has given us a substantial boost to the score.

\n

Our Humble Approach

\n

We have, extensively, made use of Google's NotebookLM platform for data extraction, data preprocessing, and pseudo-labelling.

\n

We used a single Byt5-base model fine tuned with three stages:

\n\n

How we built our training data

\n

Creating a csv of sentence pairs with NotebookLM is quite straightforward with its reasoning model of 1 million token context length, and an OCR of a 'refined quality'.

\n

How extracted data from PDFs

\n

For most of the pdf that we had to deal with (like the AKT series), they had to be provided as image inputs due to their quality.

\n

So, we basically screenshotted every single tablet, and processed them with NotebookLM with a small batch of 1-5 tablets (1-10 images) at a time, hopefully allowing the model to focus more and avoid hallucinations.
\nDoing so resulted in more sentence pairs than giving larger batches of tablets.
\n\"\"

\n

We had found out that we could speed up the process by opening multiple notebook tabs, we just needed to be organized.

\n

How we tried to recover original transliteration from publications.csv and CAD

\n

After extracting some tablets and OA sentences, we made use of their tablet reference (eg: BIN 4 , CCT 2) to merge them with their original transliteration found in published_text.csv.

\n

To do so, we had to ask NotebookLM to find the 'aliases' of the tablet reference in published_text.csv, as hard coding a solution wasn't obvious… We gave it as input 2 files , one with 'aliases' and 'note' columns from published_text.csv and the other the list of tablet reference.

\n

Then we batched tablets/sentences with ocr_transliteration, ocr_translation merged with original transliteration, and processed them with NotebookLM

\n

How we built a small custom data

\n

As we processed data, we 've got a little bit familiar with some basic akkadian formulas, so we had decided to create a custom data that we filled with random names from onomasticon.csv and random numbers with a 0.2 probability of getting < gap > for both
\nThe custom data was mostly about Seal, Witness, From … To , month eponym, and 'owes' statements

\n

We also asked NotebookLM to split some of the training data (train.csv), but as we already had sentences from PDFs, we asked him to replace names with PN and numbers with NUM that we randomly replaced later

\n

Pseudo-labelling:

\n

We made use of the untranslated texts from published_texts.csv that we asked NotebookLM to translate and split.
\nThe translation quality was not like a 'SIG5 copper' but we believed there could be some value that we could get out of it

\n

ŠU.NÍGIN, we got a total of ~81K unique pairs: ~28K human-labelled (from PDFs, train.csv included), ~11K of custom data, ~3K (from publications.csv, … we didn't have much time), and pseudo-labelled data for the rest

\n

Training and Model

\n

We used a single ByT5-base model

\n

We have split the fine tuning into 3 stages:

\n
- 1st stage with all available data , we called it 'pretraining':
\n

With 81K pairs, a batch size of 8, learning rate of 5e-5, and 5 epoches
\nThe purpose was that the model would learn akkadian in general and have some good 'momentum'

\n

Submitting at this stage would give us ~35+, which was less than a model that was trained on human labelled only which could get up to ~37 for 25K pairs (at that time).

\n
- 2nd stage with only human labelled data:
\n

With 28K pairs, batch size of 2, lr 1e-5 , and 2 epochs
\nThe hope was that the model output would align more with human labelled data and correct all biases from pseudo-labelling.

\n

Submitting at this stage would give us 38+

\n

On our local validation with these 2 stages we could see a +2 to +3 BLEU score boost compared to a model trained only on human labelled data

\n
- 3rd stage LoRA with only the competition training data:
\n

With 1.5K , the main purpose was to align the output vocabulary of the model with the competition train.
\nSubmitting at this stage would give us ~39

\n

Some notes

\n\n

To finish , we want to give tribute to all Assyriologists out there for the truly amazing work they've done , and the Google Team for providing us advanced AI tool and really powerful pretrained models to work with.

", "rawMarkdown": "First of all , I want to thank the organizers for such an interesting competition and I also want to congratulate the winners for the hard work they've done. \n\nSecond, I also want to thank my family for the support, and thank my teammate and brother @hasinarazafindrazaka for his ML experience which has given us a substantial boost to the score. \n \n\n### Our Humble Approach \n \nWe have, extensively, made use of **Google's NotebookLM platform** for data extraction, data preprocessing, and pseudo-labelling. \n \nWe used a single Byt5-base model fine tuned with three stages: \n- 1st stage with all available data (human + pseudo labelled and small custom data) of 81K unique pairs \n- 2nd stage with only human labelled data, ~28K pairs\n- 3rd stage LoRA with only the competition training data ~1.5K\n\n\n\n### How we built our training data\n\nCreating a csv of sentence pairs with NotebookLM is quite straightforward with its reasoning model of 1 million token context length, and **an OCR of a 'refined quality'.** \n\n \n #### How extracted data from PDFs\n\nFor most of the pdf that we had to deal with (like the AKT series), they had to be provided as image inputs due to their quality.\n \nSo, we basically screenshotted every single tablet, and processed them with NotebookLM with a small batch of 1-5 tablets (1-10 images) at a time, hopefully allowing the model to focus more and avoid hallucinations. \nDoing so resulted in more sentence pairs than giving larger batches of tablets. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15932745%2Fa5c415f06cf239393f6c15c65d7868a8%2Fpdf%20captures.JPG?generation=1774516558283023&alt=media) \n \nWe had found out that we could speed up the process by opening multiple notebook tabs, we just needed to be organized. \n\n#### How we tried to recover original transliteration from publications.csv and CAD\n \nAfter extracting some tablets and OA sentences, we made use of their tablet reference (eg: BIN 4 , CCT 2) to merge them with their original transliteration found in published_text.csv. \n \nTo do so, we had to ask NotebookLM to find the 'aliases' of the tablet reference in published_text.csv, as hard coding a solution wasn't obvious... We gave it as input 2 files , one with 'aliases' and 'note' columns from published_text.csv and the other the list of tablet reference.\n \nThen we batched tablets/sentences with ocr_transliteration, ocr_translation merged with original transliteration, and processed them with NotebookLM \n\n#### How we built a small custom data\n \nAs we processed data, we 've got a little bit familiar with some basic akkadian formulas, so we had decided to create a custom data that we filled with random names from onomasticon.csv and random numbers with a 0.2 probability of getting < gap > for both \nThe custom data was mostly about Seal, Witness, From ... To , month eponym, and 'owes' statements\n \nWe also asked NotebookLM to split some of the training data (train.csv), but as we already had sentences from PDFs, we asked him to replace names with PN and numbers with NUM that we randomly replaced later\n\n#### Pseudo-labelling:\n\nWe made use of the untranslated texts from published_texts.csv that we asked NotebookLM to translate and split. \nThe translation quality was not like a 'SIG5 copper' but we believed there could be some value that we could get out of it \n\n \nŠU.NÍGIN, we got a total of ~81K unique pairs: ~28K human-labelled (from PDFs, train.csv included), ~11K of custom data, ~3K (from publications.csv, ... we didn't have much time), and pseudo-labelled data for the rest \n \n### Training and Model\n \nWe used a **single ByT5-base model**\n\n \nWe have split the fine tuning into 3 stages:\n\n##### - 1st stage with all available data , we called it 'pretraining': \nWith 81K pairs, a batch size of 8, learning rate of 5e-5, and 5 epoches \nThe purpose was that the model would learn akkadian in general and have some good 'momentum'\n \nSubmitting at this stage would give us ~35+, which was less than a model that was trained on human labelled only which could get up to ~37 for 25K pairs (at that time). \n \n##### - 2nd stage with only human labelled data: \nWith 28K pairs, batch size of 2, lr 1e-5 , and 2 epochs \n\nThe hope was that the model output would align more with human labelled data and correct all biases from pseudo-labelling. \n \nSubmitting at this stage would give us 38+\n \nOn our local validation with these 2 stages we could see a +2 to +3 BLEU score boost compared to a model trained only on human labelled data\n\n##### - 3rd stage LoRA with only the competition training data: \nWith 1.5K , the main purpose was to align the output vocabulary of the model with the competition train. \nSubmitting at this stage would give us ~39\n\n#### Some notes\n\n- We considered using APIs, but our ... 'working capital' was quite limited\n- We only did very few unsuccessful experiments with ByT5-large\n- We tried to extract data from 'Sentences_Oare_FirstWord_LinNum.csv' but result got surprisingly worse either on local validation or public leaderboard\n- We plateaued at 38.1 for a long time, even if we added more data (eg ATHE CAD)... Break through only happened after we added some data from publications.csv (CMK, TPAK, APU,...) into the pretraining phase , and further refinement of our dataset \n \nTo finish , we want to give tribute to all Assyriologists out there for the truly amazing work they've done , and the Google Team for providing us advanced AI tool and really powerful pretrained models to work with.", "votes": null }, { "id": "3429128", "postDate": "03/26/2026 12:16:47", "content": "
\n

We had found out that we could speed up the process by opening multiple notebook tabs, we just needed to be organized.

\n
\n

How the sausage is made. Great work!

", "rawMarkdown": ">We had found out that we could speed up the process by opening multiple notebook tabs, we just needed to be organized.\n\nHow the sausage is made. Great work!", "votes": null } ], "comments": [ { "id": 3429128, "author_name": "ottpocket", "author_url": "", "post_date": "03/26/2026 12:16:47", "content": "
\n

We had found out that we could speed up the process by opening multiple notebook tabs, we just needed to be organized.

\n
\n

How the sausage is made. Great work!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3429056": "First of all , I want to thank the organizers for such an interesting competition and I also want to congratulate the winners for the hard work they've done. \n\nSecond, I also want to thank my family for the support, and thank my teammate and brother @hasinarazafindrazaka for his ML experience which has given us a substantial boost to the score. \n \n\n### Our Humble Approach \n \nWe have, extensively, made use of **Google's NotebookLM platform** for data extraction, data preprocessing, and pseudo-labelling. \n \nWe used a single Byt5-base model fine tuned with three stages: \n- 1st stage with all available data (human + pseudo labelled and small custom data) of 81K unique pairs \n- 2nd stage with only human labelled data, ~28K pairs\n- 3rd stage LoRA with only the competition training data ~1.5K\n\n\n\n### How we built our training data\n\nCreating a csv of sentence pairs with NotebookLM is quite straightforward with its reasoning model of 1 million token context length, and **an OCR of a 'refined quality'.** \n\n \n #### How extracted data from PDFs\n\nFor most of the pdf that we had to deal with (like the AKT series), they had to be provided as image inputs due to their quality.\n \nSo, we basically screenshotted every single tablet, and processed them with NotebookLM with a small batch of 1-5 tablets (1-10 images) at a time, hopefully allowing the model to focus more and avoid hallucinations. \nDoing so resulted in more sentence pairs than giving larger batches of tablets. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15932745%2Fa5c415f06cf239393f6c15c65d7868a8%2Fpdf%20captures.JPG?generation=1774516558283023&alt=media) \n \nWe had found out that we could speed up the process by opening multiple notebook tabs, we just needed to be organized. \n\n#### How we tried to recover original transliteration from publications.csv and CAD\n \nAfter extracting some tablets and OA sentences, we made use of their tablet reference (eg: BIN 4 , CCT 2) to merge them with their original transliteration found in published_text.csv. \n \nTo do so, we had to ask NotebookLM to find the 'aliases' of the tablet reference in published_text.csv, as hard coding a solution wasn't obvious... We gave it as input 2 files , one with 'aliases' and 'note' columns from published_text.csv and the other the list of tablet reference.\n \nThen we batched tablets/sentences with ocr_transliteration, ocr_translation merged with original transliteration, and processed them with NotebookLM \n\n#### How we built a small custom data\n \nAs we processed data, we 've got a little bit familiar with some basic akkadian formulas, so we had decided to create a custom data that we filled with random names from onomasticon.csv and random numbers with a 0.2 probability of getting < gap > for both \nThe custom data was mostly about Seal, Witness, From ... To , month eponym, and 'owes' statements\n \nWe also asked NotebookLM to split some of the training data (train.csv), but as we already had sentences from PDFs, we asked him to replace names with PN and numbers with NUM that we randomly replaced later\n\n#### Pseudo-labelling:\n\nWe made use of the untranslated texts from published_texts.csv that we asked NotebookLM to translate and split. \nThe translation quality was not like a 'SIG5 copper' but we believed there could be some value that we could get out of it \n\n \nŠU.NÍGIN, we got a total of ~81K unique pairs: ~28K human-labelled (from PDFs, train.csv included), ~11K of custom data, ~3K (from publications.csv, ... we didn't have much time), and pseudo-labelled data for the rest \n \n### Training and Model\n \nWe used a **single ByT5-base model**\n\n \nWe have split the fine tuning into 3 stages:\n\n##### - 1st stage with all available data , we called it 'pretraining': \nWith 81K pairs, a batch size of 8, learning rate of 5e-5, and 5 epoches \nThe purpose was that the model would learn akkadian in general and have some good 'momentum'\n \nSubmitting at this stage would give us ~35+, which was less than a model that was trained on human labelled only which could get up to ~37 for 25K pairs (at that time). \n \n##### - 2nd stage with only human labelled data: \nWith 28K pairs, batch size of 2, lr 1e-5 , and 2 epochs \n\nThe hope was that the model output would align more with human labelled data and correct all biases from pseudo-labelling. \n \nSubmitting at this stage would give us 38+\n \nOn our local validation with these 2 stages we could see a +2 to +3 BLEU score boost compared to a model trained only on human labelled data\n\n##### - 3rd stage LoRA with only the competition training data: \nWith 1.5K , the main purpose was to align the output vocabulary of the model with the competition train. \nSubmitting at this stage would give us ~39\n\n#### Some notes\n\n- We considered using APIs, but our ... 'working capital' was quite limited\n- We only did very few unsuccessful experiments with ByT5-large\n- We tried to extract data from 'Sentences_Oare_FirstWord_LinNum.csv' but result got surprisingly worse either on local validation or public leaderboard\n- We plateaued at 38.1 for a long time, even if we added more data (eg ATHE CAD)... Break through only happened after we added some data from publications.csv (CMK, TPAK, APU,...) into the pretraining phase , and further refinement of our dataset \n \nTo finish , we want to give tribute to all Assyriologists out there for the truly amazing work they've done , and the Google Team for providing us advanced AI tool and really powerful pretrained models to work with.", "3429128": ">We had found out that we could speed up the process by opening multiple notebook tabs, we just needed to be organized.\n\nHow the sausage is made. Great work!" }, "source": "meta" }