{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":121150,"databundleVersionId":15061024,"sourceType":"competition"},{"sourceId":14523367,"sourceType":"datasetVersion","datasetId":9275788},{"sourceId":14523528,"sourceType":"datasetVersion","datasetId":9275882}],"dockerImageVersionId":31260,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\n> The biggest and heartful thanks to [MPWARE](https://www.kaggle.com/mpware) for starting a thread on OCR experiments with The Assur-nada Archive (Larsen 2002), sharing [their results as a public dataset](https://www.kaggle.com/datasets/mpware/dpc-larsen), starting the discussion ([Any luck with letters from Larsen PDF?](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/670040)) and being an inspiration for using the GLM-OCR model. Without all of the above this work would not be possible.\n\nMy deepest hope is that by making this notebook public, more people get their hands on a tool to retrieve the transliterations/translations. If you do, please consider sharing your results. We can work better and faster as a community.\n\nThe following notebook constitutes a minimum working example for anyone attempting to OCR the PDF documents for the [Deep Past Challenge](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview) by using nothing more than a T4 machine provided by Kaggle. The code base was created with the help of Gemini 3 on Google Colab. It is not the cleanest, not the prettiest but I strived to make it leaner and faster than it would be for loading an entire page. Current output is pretty good and typically needs only a bit of manual work to be fully compliant.\n\nThis code can be greatly improved in terms of parallel use of T4 machines, PDF preprocessing and output cleaning/post-processing.\n\n## Base model\nThis model uses a fine-tuned version of zai-org/GLM-OCR, more precisely unsloth/GLM-OCR. \n\n## Known issues\nThe GLM-OCR struggles to capture emphatic s (ṣ) and t (ṭ), and sometimes even vowels. It will typically use HTML tags for sub/superscripts (`<sub></sub>` and `<sup></sup>`) but may use LaTeX notation as well (e.g., ` $ gu_{5} $ ` for `gu₅`). It's also not consitent about it, so be on the lookout.\n\n## Output format\nThe OCR result is a list of translations for each block. This is intentional and is meant to be the most general to work with most PDF documents. In some cases like The Assur-nada Archive (Larsen 2002) the transliterations form a leftmost column, while translations for a right-most column. For such cases it is trivial to use the extracted block position metadata (saved to a `metadata.csv` file) to filter out the transliteration/translation blocks and re-arrange/join the text. It is not the case for the AKT series, for which the both transliterations and translations come after one another and occupy the full width (at least for AKT 1990; I have not checked all of them).\n\n## Final remarks\nGood luck with the OCR!","metadata":{}},{"cell_type":"markdown","source":"# Download and install the necessary packages","metadata":{}},{"cell_type":"code","source":"!pip install -U git+https://github.com/huggingface/transformers.git\n!pip install pymupdf\n!pip install numba\n\n!apt-get install poppler-utils\n!pip install pytesseract opencv-python pdf2image pillow numpy\n!pip install pylatexenc","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:19:34.157495Z","iopub.execute_input":"2026-02-14T21:19:34.157961Z","iopub.status.idle":"2026-02-14T21:21:04.004947Z","shell.execute_reply.started":"2026-02-14T21:19:34.157929Z","shell.execute_reply":"2026-02-14T21:21:04.003618Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Download the unsloth/GLM-OCR LLM model","metadata":{}},{"cell_type":"code","source":"!git clone https://huggingface.co/unsloth/GLM-OCR ./unsloth/GLM-OCR","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:21:04.007708Z","iopub.execute_input":"2026-02-14T21:21:04.008261Z","iopub.status.idle":"2026-02-14T21:21:56.330579Z","shell.execute_reply.started":"2026-02-14T21:21:04.008220Z","shell.execute_reply":"2026-02-14T21:21:56.329310Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Handle the imports","metadata":{}},{"cell_type":"code","source":"import glob\n# import math\nimport io\nimport shutil\nimport os\nimport csv\nimport json\nimport requests\n\nimport cv2\nfrom PIL import Image, ImageDraw, ImageFont\nfrom pdf2image import convert_from_path\nimport numpy as np\nimport pytesseract\nfrom pytesseract import Output\nimport pymupdf\n\nfrom tqdm.notebook import tqdm\n\nimport matplotlib.pyplot as plt\n\nimport torch\nfrom numba import cuda\nfrom transformers import AutoProcessor, AutoModelForImageTextToText\n\nfrom pylatexenc.latex2text import LatexNodes2Text","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:21:56.332904Z","iopub.execute_input":"2026-02-14T21:21:56.333374Z","iopub.status.idle":"2026-02-14T21:22:40.729759Z","shell.execute_reply.started":"2026-02-14T21:21:56.333320Z","shell.execute_reply":"2026-02-14T21:22:40.728242Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Initialize the constants","metadata":{}},{"cell_type":"code","source":"larsen_pdf='/kaggle/input/old-assyrian-grammars-and-other-resources/Larsen 2002 - The Assur-nada Archive. PIHANS 96 2002.pdf'\nakt_1_1990_pdf='/kaggle/input/datasets/deeppast/old-assyrian-kltepe-tablets-in-pdf/AKT 1 1990.pdf'\nLibertinusMathRegular_font = 'https://github.com/googlefonts/libertinus/raw/refs/heads/main/fonts/ttf/LibertinusMath-Regular.ttf'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:24:02.355990Z","iopub.execute_input":"2026-02-14T21:24:02.356902Z","iopub.status.idle":"2026-02-14T21:24:02.367478Z","shell.execute_reply.started":"2026-02-14T21:24:02.356851Z","shell.execute_reply":"2026-02-14T21:24:02.365384Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Download the font","metadata":{}},{"cell_type":"code","source":"font = requests.get(LibertinusMathRegular_font).content\nwith open(\"/kaggle/working/LibertinusMath-Regular.ttf\", mode='wb') as f:\n    f.write(font)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:24:03.559436Z","iopub.execute_input":"2026-02-14T21:24:03.560749Z","iopub.status.idle":"2026-02-14T21:24:04.227849Z","shell.execute_reply.started":"2026-02-14T21:24:03.560668Z","shell.execute_reply":"2026-02-14T21:24:04.226211Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Define the functions","metadata":{}},{"cell_type":"code","source":"def get_pdf_page_image(pdf_path, page_number, dpi=300):\n    \"\"\"Extracts a page from a PDF as a PIL Image.\"\"\"\n    try:\n        doc = pymupdf.open(pdf_path)\n        # PyMuPDF is 0-indexed\n        page = doc.load_page(page_number - 1)\n        pix = page.get_pixmap(dpi=dpi)\n        img_data = pix.tobytes(\"png\")\n        return Image.open(io.BytesIO(img_data))\n    except Exception as e:\n        print(f\"PyMuPDF failed, falling back to pdf2image: {e}\")\n        try:\n            images = convert_from_path(pdf_path, first_page=page_number, last_page=page_number, dpi=dpi)\n            return images[0]\n        except Exception as e2:\n            print(f\"pdf2image failed: {e2}\")\n            return None","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:24:05.627544Z","iopub.execute_input":"2026-02-14T21:24:05.627895Z","iopub.status.idle":"2026-02-14T21:24:05.640581Z","shell.execute_reply.started":"2026-02-14T21:24:05.627865Z","shell.execute_reply":"2026-02-14T21:24:05.637485Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def extract_fields_mod(pdf_path, page_num, output_dir, psm=3, show_bounds=True):\n    # Setup Output Directory\n    if os.path.exists(output_dir):\n        shutil.rmtree(output_dir)\n    os.makedirs(output_dir)\n\n    # 1. Load Image\n    pil_img = get_pdf_page_image(pdf_path, page_num)\n    if pil_img is None:\n        print(\"Error: Could not load image.\")\n        return None\n\n    img = np.array(pil_img)\n    # Handle Grayscale\n    if len(img.shape) == 2:\n        img = cv2.cvtColor(img, cv2.COLOR_GRAY2RGB)\n\n    # Prepare inverted binary image for ink detection (Text=255, Bg=0)\n    img_gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n    _, bin_inv = cv2.threshold(img_gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)\n    h_img, w_img = bin_inv.shape\n\n    # --- Preprocessing for Tesseract (Adaptive Threshold) ---\n    # Use img_gray as input (must be single channel uint8)\n    img = cv2.adaptiveThreshold(img_gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,\n                                   cv2.THRESH_BINARY, 11, 2)\n\n    # 2. Tesseract Layout Analysis\n    print(\"Running Tesseract layout analysis...\")\n    custom_config = r'' + f'--psm {psm}'\n    try:\n        d = pytesseract.image_to_data(img, config=custom_config, output_type=Output.DICT)\n    except Exception as e:\n        print(f\"Tesseract Error: {e}\")\n        return\n\n    # 3. Group Text into Fields\n    # Group by (Block, Para, Line) to respect column splits\n    lines_map = {}\n    n_boxes = len(d['level'])\n\n    for i in range(n_boxes):\n        if d['level'][i] != 5: # Word level\n            continue\n        text = d['text'][i].strip()\n        if not text:\n            continue\n\n        key = (d['block_num'][i], d['par_num'][i], d['line_num'][i])\n\n        if key not in lines_map:\n            lines_map[key] = []\n\n        lines_map[key].append({\n            'left': d['left'][i],\n            'top': d['top'][i],\n            'width': d['width'][i],\n            'height': d['height'][i]\n        })\n\n    raw_fields = []\n    for key, boxes in lines_map.items():\n        x1 = min(b['left'] for b in boxes)\n        y1 = min(b['top'] for b in boxes)\n        x2 = max(b['left'] + b['width'] for b in boxes)\n        y2 = max(b['top'] + b['height'] for b in boxes)\n        raw_fields.append((x1, y1, x2 - x1, y2 - y1))\n\n    # 4. Refine Fields: Expand borders if cutting through ink\n    refined_fields = []\n    for (x, y, w, h) in raw_fields:\n        # Calculate max pixel value for half-height along the entire length\n        mid_y = min(max(y + h // 2, 0), h_img - 1)\n        mid_row = bin_inv[mid_y, x:x+w]\n\n        if len(mid_row) == 0:\n            refined_fields.append((x, y, w, h))\n            continue\n\n        max_px_val = np.max(mid_row)\n\n        # If the center line has no ink (value 0), do not expand\n        if max_px_val == 0:\n            refined_fields.append((x, y, w, h))\n            continue\n\n        cur_x, cur_y, cur_w, cur_h = x, y, w, h\n\n        # --- TIGHTENING LOGIC: Shrink box to fit text ---\n        # Top\n        while cur_h > 1:\n            check_row = bin_inv[cur_y, cur_x : cur_x + cur_w]\n            if not np.any(check_row == max_px_val):\n                cur_y += 1\n                cur_h -= 1\n            else:\n                break\n\n        # Bottom\n        while cur_h > 1:\n            check_row = bin_inv[cur_y + cur_h - 1, cur_x : cur_x + cur_w]\n            if not np.any(check_row == max_px_val):\n                cur_h -= 1\n            else:\n                break\n\n        # Left\n        while cur_w > 1:\n            check_col = bin_inv[cur_y : cur_y + cur_h, cur_x]\n            if not np.any(check_col == max_px_val):\n                cur_x += 1\n                cur_w -= 1\n            else:\n                break\n\n        # Right\n        while cur_w > 1:\n            check_col = bin_inv[cur_y : cur_y + cur_h, cur_x + cur_w - 1]\n            if not np.any(check_col == max_px_val):\n                cur_w -= 1\n            else:\n                break\n\n        # --- EXPANSION LOGIC ---\n        # Recursively increase border by 2px (max 10px)\n        max_add = 10\n        step_size = 2\n\n        # Top\n        for _ in range(0, max_add, step_size):\n            if cur_y <= 0: break\n            check_row = bin_inv[cur_y, cur_x : cur_x + cur_w]\n            if np.any(check_row == max_px_val):\n                step = min(step_size, cur_y)\n                cur_y -= step\n                cur_h += step\n            else:\n                break\n\n        # Bottom\n        for _ in range(0, max_add, step_size):\n            if cur_y + cur_h >= h_img: break\n            check_row = bin_inv[cur_y + cur_h - 1, cur_x : cur_x + cur_w]\n            if np.any(check_row == max_px_val):\n                step = min(step_size, h_img - (cur_y + cur_h))\n                cur_h += step\n            else:\n                break\n\n        # Left\n        for _ in range(0, max_add, step_size):\n            if cur_x <= 0: break\n            check_col = bin_inv[cur_y : cur_y + cur_h, cur_x]\n            if np.any(check_col == max_px_val):\n                step = min(step_size, cur_x)\n                cur_x -= step\n                cur_w += step\n            else:\n                break\n\n        # Right\n        for _ in range(0, max_add, step_size):\n            if cur_x + cur_w >= w_img: break\n            check_col = bin_inv[cur_y : cur_y + cur_h, cur_x + cur_w - 1]\n            if np.any(check_col == max_px_val):\n                step = min(step_size, w_img - (cur_x + cur_w))\n                cur_w += step\n            else:\n                break\n\n        refined_fields.append((cur_x, cur_y, cur_w, cur_h))\n\n    # 5. Sort Fields (Top-to-Bottom, then Left-to-Right)\n    refined_fields.sort(key=lambda box: (box[1] // 20, box[0]))\n\n    # 6. Process: Save & visualize\n    vis_img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR) # Create BGR for visualization since img is now gray/binary\n\n    csv_path = os.path.join(output_dir, \"metadata.csv\")\n    with open(csv_path, 'w', newline='') as csvfile:\n        fieldnames = ['page_num', 'field_num', 'x', 'y', 'w', 'h']\n        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n        writer.writeheader()\n\n        writer.writerow({'page_num': page_num, 'field_num': -1, 'x': 0, 'y': 0, 'w': img.shape[1], 'h': img.shape[0]})\n\n        for idx, (x, y, w, h) in enumerate(refined_fields):\n            # Save crop\n            filename = f\"p-{page_num}_f-{idx}.png\"\n            filepath = os.path.join(output_dir, filename)\n            crop = img[y:y+h, x:x+w]\n            cv2.imwrite(filepath, crop)\n\n            # Save metadata\n            writer.writerow({'page_num': page_num, 'field_num': idx, 'x': x, 'y': y, 'w': w, 'h': h})\n\n            if show_bounds:\n                # Visualize: Box\n                cv2.rectangle(vis_img, (x, y), (x + w, y + h), (0, 255, 0), 2)\n\n                # Visualize: Label (Outside Right-Side Corner)\n                label_text = f\"f-{idx}\"\n                label_x = x + w + 5\n                label_y = y + 15\n                cv2.putText(vis_img, label_text, (label_x, label_y),\n                            cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255, 0, 0), 2)\n\n    print(f\"Extracted {len(refined_fields)} fields using PyTesseract to '{output_dir}'.\")\n    print(f\"Metadata saved to {csv_path}\")\n\n    # Display\n    plt.figure(figsize=(12, 16))\n    plt.imshow(cv2.cvtColor(vis_img, cv2.COLOR_BGR2RGB))\n    plt.axis('off')\n    plt.title(f\"Page {page_num} Fields (Tesseract) - Expanded\")\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:24:08.244584Z","iopub.execute_input":"2026-02-14T21:24:08.244997Z","iopub.status.idle":"2026-02-14T21:24:08.286430Z","shell.execute_reply.started":"2026-02-14T21:24:08.244964Z","shell.execute_reply":"2026-02-14T21:24:08.285200Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def batch_process_glm_ocr(input_dir, model=None, processor=None, model_path=\"unsloth/GLM-OCR\"):\n    \"\"\"\n    Batch process images using unsloth/GLM-OCR.\n    Returns a dictionary {filename_no_ext: recognized_text}.\n    \"\"\"\n    device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\n    # Load model if not provided\n    if model is None or processor is None:\n        print(f\"Loading model from {model_path}...\")\n        processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=False)\n        model = AutoModelForImageTextToText.from_pretrained(\n            model_path,\n            dtype=torch.bfloat16,\n            device_map=\"auto\",\n            trust_remote_code=False\n        )\n\n    results = {}\n    image_files = sorted(glob.glob(os.path.join(input_dir, \"*.png\")))\n\n    print(f\"Processing {len(image_files)} images from '{input_dir}' using {device}...\")\n\n    for img_path in tqdm(image_files, desc=\"OCR Processing\"):\n        filename = os.path.basename(img_path)\n        key, _ = os.path.splitext(filename)\n\n        try:\n            # Convert image to greyscale\n            image = Image.open(img_path).convert(\"L\")\n\n            # GLM-OCR Chat Template\n            messages=[]\n            messages.append( {\n                    \"role\": \"user\",\n                    \"content\": [\n                        {\"type\": \"image\", \"image\": image},\n                        {\"type\": \"text\", \"text\": \"Text Recognition:\"}\n                    ]\n            })\n\n            inputs = processor.apply_chat_template(\n                messages,\n                add_generation_prompt=True,\n                tokenize=True,\n                return_dict=True,\n                return_tensors=\"pt\"\n            ).to(model.device)\n            \n            # Generate text\n            with torch.no_grad():\n                outputs = model.generate(**inputs, max_new_tokens=1024)\n\n            # Decode output\n            generated_text = processor.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)\n\n            results[key] = generated_text\n\n        except Exception as e:\n            print(f\"Error processing {filename}: {e}\")\n            results[key] = \"\"\n\n    return results","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:24:10.831178Z","iopub.execute_input":"2026-02-14T21:24:10.831508Z","iopub.status.idle":"2026-02-14T21:24:10.843836Z","shell.execute_reply.started":"2026-02-14T21:24:10.831480Z","shell.execute_reply":"2026-02-14T21:24:10.842559Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Break the PDF into smaller blocks\nThis is essential for faster image processing and lower GPU consumption.","metadata":{}},{"cell_type":"markdown","source":"## The Assur-nada Archive (Larsen 2002)","metadata":{}},{"cell_type":"code","source":"# Constants\nPDF_FILEPATH = larsen_pdf\nPAGE_NUMBER = 73\nRAW_EXTRACTED_LINES = \"/kaggle/working/extracted_fields_mod\"\n\n# Run\nextract_fields_mod(PDF_FILEPATH,\n                   PAGE_NUMBER,\n                   RAW_EXTRACTED_LINES,\n                   psm=11,\n                   show_bounds=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:24:17.124004Z","iopub.execute_input":"2026-02-14T21:24:17.124525Z","iopub.status.idle":"2026-02-14T21:24:23.237615Z","shell.execute_reply.started":"2026-02-14T21:24:17.124489Z","shell.execute_reply":"2026-02-14T21:24:23.235986Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Assyrian Kultepe Tablets (AKT 1990)","metadata":{}},{"cell_type":"code","source":"# Constants\nPDF_FILEPATH = akt_1_1990_pdf\nPAGE_NUMBER = 34\nRAW_EXTRACTED_LINES_AKT = \"/kaggle/working/extracted_fields_mod_akt_1_1990\"\n\n# Run\nextract_fields_mod(PDF_FILEPATH,\n                   PAGE_NUMBER,\n                   RAW_EXTRACTED_LINES_AKT,\n                   psm=11,\n                   show_bounds=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:24:46.144239Z","iopub.execute_input":"2026-02-14T21:24:46.144915Z","iopub.status.idle":"2026-02-14T21:24:51.250938Z","shell.execute_reply.started":"2026-02-14T21:24:46.144881Z","shell.execute_reply":"2026-02-14T21:24:51.249689Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Use unsloth/GLM-OCR model to OCR the extracted blocks of text","metadata":{}},{"cell_type":"markdown","source":"## The Assur-nada Archive (Larsen 2002)","metadata":{}},{"cell_type":"code","source":"MODEL_HF_PATH='unsloth/GLM-OCR'\n\nocr_results_raw = batch_process_glm_ocr(input_dir=RAW_EXTRACTED_LINES,\n                                        model=None,\n                                        processor=None,\n                                        model_path=MODEL_HF_PATH)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:24:58.707023Z","iopub.execute_input":"2026-02-14T21:24:58.707599Z","iopub.status.idle":"2026-02-14T21:37:53.128695Z","shell.execute_reply.started":"2026-02-14T21:24:58.707560Z","shell.execute_reply":"2026-02-14T21:37:53.127004Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Assyrian Kultepe Tablets (1990)","metadata":{}},{"cell_type":"code","source":"MODEL_HF_PATH='unsloth/GLM-OCR'\n\nocr_results_raw_akt = batch_process_glm_ocr(input_dir=RAW_EXTRACTED_LINES_AKT,\n                                        model=None,\n                                        processor=None,\n                                        model_path=MODEL_HF_PATH)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:37:53.131038Z","iopub.execute_input":"2026-02-14T21:37:53.131469Z","iopub.status.idle":"2026-02-14T21:55:35.422116Z","shell.execute_reply.started":"2026-02-14T21:37:53.131438Z","shell.execute_reply":"2026-02-14T21:55:35.420735Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Reconstruct the page using the OCR-ed blocks of text","metadata":{}},{"cell_type":"code","source":"def reconstruct_page_text(metadata_csv_path, ocr_results, output_filename=None):\n    # 1. Load Metadata\n    fields = []\n    if os.path.exists(metadata_csv_path):\n        with open(metadata_csv_path, 'r') as f:\n            reader = csv.DictReader(f)\n            for row in reader:\n                fields.append(row)\n    else:\n        print(f\"Error: Metadata file {metadata_csv_path} not found.\")\n        return None\n\n    # 2. Determine Page Size\n    w, h = int(fields[0]['w']), int(fields[0]['h'])\n\n    # Create white canvas (RGB for PIL)\n    pil_image = Image.new(\"RGB\", (w, h), (255, 255, 255))\n    draw = ImageDraw.Draw(pil_image)\n    \n    selected_font='./LibertinusMath-Regular.ttf'\n    \n\n    if not selected_font:\n        print(\"Warning: No suitable TTF font found. Using default PIL font (bitmap, unicode support poor).\")\n\n    for field in fields[1:]:\n        # Parse metadata\n        page_num = field['page_num']\n        field_idx = field['field_num']\n        x, y, w, h = int(field['x']), int(field['y']), int(field['w']), int(field['h'])\n\n        # Retrieve Text\n        base_key = f\"p-{page_num}_f-{field_idx}\"\n        pre_key = f\"{base_key}\"\n\n        text = \"\"\n        if isinstance(ocr_results, dict):\n            text = ocr_results.get(base_key, ocr_results.get(pre_key, \"\"))\n            if '$' in text:\n                text = LatexNodes2Text().latex_to_text(text)\n            elif ('<sup>' in text) & ('</sup>' in text):\n                text = text.replace('<sup>', '^').replace('</sup>', '')\n            elif ('<sub>' in text) & ('</sub>' in text):\n                text = text.replace('<sub>', '_').replace('</sub>', '')\n\n        if not text:\n            continue\n\n        # Draw Box (faintly)\n        draw.rectangle([x, y, x+w, y+h], outline=(220, 220, 220), width=1)\n\n        if selected_font:\n            # Estimate Font Scale\n            # 1. Measure at arbitrary size\n            test_size = 20\n            font = ImageFont.truetype(selected_font, test_size)\n\n            # Get text size\n            try:\n                left, top, right, bottom = draw.textbbox((0, 0), text, font=font)\n                tw = right - left\n                th = bottom - top\n            except AttributeError:\n                # Fallback for older PIL\n                tw, th = font.getsize(text)\n\n            if tw == 0 or th == 0:\n                 continue\n\n            # Calculate scaling\n            scale_w = w / tw\n            scale_h = h / th\n            scale = min(scale_w, scale_h)\n\n            # Apply scale to font size\n            new_size = int(test_size * scale)\n            if new_size < 8: new_size = 8 # Minimum readable size\n\n            font = ImageFont.truetype(selected_font, new_size)\n\n            # Recalculate size for centering\n            try:\n                left, top, right, bottom = draw.textbbox((0, 0), text, font=font)\n                tw_final = right - left\n                th_final = bottom - top\n            except:\n                tw_final, th_final = font.getsize(text)\n\n            # Center\n            text_x = x + (w - tw_final) // 2\n            text_y = y + (h - th_final) // 2\n\n            draw.text((text_x, text_y), text, font=font, fill=(0, 0, 0))\n        else:\n             draw.text((x, y), text, fill=(0,0,0)) # Default font\n\n    # Convert to NumPy for compatibility (RGB)\n    canvas_rgb = np.array(pil_image)\n\n    if output_filename:\n        # Convert RGB to BGR for OpenCV saving\n        canvas_bgr = cv2.cvtColor(canvas_rgb, cv2.COLOR_RGB2BGR)\n        cv2.imwrite(output_filename, canvas_bgr)\n\n    return canvas_rgb","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:56:56.521565Z","iopub.execute_input":"2026-02-14T21:56:56.522624Z","iopub.status.idle":"2026-02-14T21:56:56.542410Z","shell.execute_reply.started":"2026-02-14T21:56:56.522585Z","shell.execute_reply":"2026-02-14T21:56:56.541251Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Larsen 2002 - The Assur-nada Archive","metadata":{}},{"cell_type":"code","source":"# Usage\ncsv_path = \"extracted_fields_mod/metadata.csv\"\n\nreconstruction = reconstruct_page_text(csv_path,\n                                       ocr_results_raw,\n                                       \"reconstructed_page.png\")\n\nplt.figure(figsize=(15, 20))\nplt.imshow(reconstruction)\nplt.axis('off')\nplt.title(\"Reconstructed OCR Text (PIL/Serif)\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:57:02.051354Z","iopub.execute_input":"2026-02-14T21:57:02.052506Z","iopub.status.idle":"2026-02-14T21:57:03.791215Z","shell.execute_reply.started":"2026-02-14T21:57:02.052469Z","shell.execute_reply":"2026-02-14T21:57:03.790211Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Assyrian Kultepe Tablets (1990)","metadata":{}},{"cell_type":"code","source":"# Usage\ncsv_path = \"extracted_fields_mod_akt_1_1990/metadata.csv\"\n\nreconstruction = reconstruct_page_text(csv_path,\n                                       ocr_results_raw_akt,\n                                       \"reconstructed_page_akt.png\")\n\nplt.figure(figsize=(15, 20))\nplt.imshow(reconstruction)\nplt.axis('off')\nplt.title(\"Reconstructed OCR Text (PIL/Serif)\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-14T21:58:01.629934Z","iopub.execute_input":"2026-02-14T21:58:01.630541Z","iopub.status.idle":"2026-02-14T21:58:03.458980Z","shell.execute_reply.started":"2026-02-14T21:58:01.630508Z","shell.execute_reply":"2026-02-14T21:58:03.457366Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}