{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":94147,"databundleVersionId":11390004,"sourceType":"competition"}],"dockerImageVersionId":31012,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Project Report: A 28.7% Score Improvement Through Post-Competition Debugging\n\nThis notebook represents the final, optimized version of my entry for the Global AI Hackathon '25.\n\n## The Journey:\n\n**Initial Submission (Version 7):** My model submitted at the competition deadline achieved a private score of 0.19677.\n\n**Final Version (This Notebook, V15):** After the deadline, I performed a deep-dive analysis and systematic debugging of the data pipeline. This resulted in a final score of 0.25311.\n\n## Key Achievement: A 28.7% Increase in Performance\n\nThis significant improvement was achieved by identifying and fixing several critical bugs in the initial data loading and processing pipeline. This notebook documents not only the final, high-performing solution but also the analytical process used to find and resolve these issues.\n\n## Key Learnings:\n\n\n* The importance of data validation at every step.\n* Robustly handling data types and column names from HDF5 files.\n* Correctly diagnosing and fixing data normalization issues for image data.\n","metadata":{}},{"cell_type":"markdown","source":"## Problem Overview\n\nHistology is a century-old clinical tool, but its analysis is subjective and coarse. This competition aimed to bring **automation and molecular-level inference** to histopathology using deep learning.\n\nGiven:\n- **HE-stained slide images** for training\n- A final test image (`S_7`) for evaluation\n\nGoal:\n- Predict 35 cell-type abundances at various spatial coordinates (spots) across the test tissue slide.\n- Score based on **Spearman correlation** with ground truth values.","metadata":{}},{"cell_type":"markdown","source":"## Problem Overview – Made Simple\n\nYou’re given a large histology image (like a zoomed-in tissue photo under a microscope) with 2000+ circular spots. For each spot, you must predict the proportions of 35 different cell types using just the image pixels — no genomic data.\n\n**🍕 Analogy:**\n\n**Imagine you have a pizza (the tissue image), and 2000 small circles marked on it. You need to say, for each circle:**\n\n**“There’s 10% type C1, 5% type C2, 0% type C3… etc.”**\n\n🔍 What You’re Given:\n\n6 training tissue images\n\nFor each training image, the 2000+ spot coordinates and true cell type distributions (answers).\n\n1 test tissue image (called S_7) with 2000+ spot coordinates, but no answers. You must predict the cell type mix for each test spot.\n\n📸 In Simple Steps:\n\nCut small image patches around each training spot from the tissue images.\n\nBuild a deep learning model (like a CNN or ViT) that takes an image patch and outputs a 35-dimensional vector (cell type proportions).\n\nTrain your model using the training patches + their real answers.\n\nPredict the 35-dimensional vectors for the test image patches.\n\nSubmit your predictions in the required format.\n\n🎯 The Goal:\n\nYour model should match the real cell type distributions as closely as possible. The scoring is done using Spearman correlation, which checks how well your predicted ranks match the actual ranks for each spot.\n\n🤖 Analogy:\n\nIt’s like giving a model a zoomed-in photo of a small part of a forest and asking:\n\n\"How many oak trees? How many pine? How many birch?\"\nBased only on the photo, with no species names on the trees.","metadata":{}},{"cell_type":"markdown","source":"## Dataset & Challenges\n\n- High-resolution pathology images (HE stain)\n- Labels: Cell-type proportions for each spatial spot\n- No external data allowed\n- Runtime ≤ 9 hours (CPU or GPU)\n\nKey challenges:\n- Large image size, tiled spatially\n- Weak supervision: Only spot-level cell-type proportions\n- No genomics input — pure image-based prediction","metadata":{"execution":{"iopub.status.busy":"2025-08-08T04:15:32.421659Z","iopub.execute_input":"2025-08-08T04:15:32.422275Z","iopub.status.idle":"2025-08-08T04:15:32.430570Z","shell.execute_reply.started":"2025-08-08T04:15:32.422248Z","shell.execute_reply":"2025-08-08T04:15:32.429819Z"}}},{"cell_type":"markdown","source":"## My Approach\n\n- Used PyTorch to implement a CNN-based model\n- Tiled large images into smaller patches centered on each spot\n- Applied basic image augmentations (flip, rotate)\n- Used pretrained **ResNet18** backbone to extract features\n- Used a small fully connected head to output 35 cell-type proportions\n- Trained using MSE loss (regression task)\n\nThis was a baseline approach — no attention, segmentation, or stain normalization yet.","metadata":{}},{"cell_type":"markdown","source":"## Results\n\n- **Private Leaderboard Rank**: 305/355  \n- **Score (Spearman Correlation)**: 0.19677  \n- **Top score**: 0.71803\n\nThough the model captured some spatial patterns, the performance was far from ideal — likely due to underfitting, lack of domain-specific augmentations, and limited training cycles.","metadata":{}},{"cell_type":"markdown","source":"# 1. Data Loading and Visualization\n\nBelow, we load the HE images (both Train and Test) from the HDF5 file and overlay the spot coordinates on the images.","metadata":{}},{"cell_type":"code","source":"import h5py\nimport numpy as np\nimport matplotlib.pyplot as plt","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:35:39.712133Z","iopub.execute_input":"2025-08-11T13:35:39.712430Z","iopub.status.idle":"2025-08-11T13:35:40.087489Z","shell.execute_reply.started":"2025-08-11T13:35:39.712407Z","shell.execute_reply":"2025-08-11T13:35:40.086843Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import h5py\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport math\n\ndef visualize_slides_with_spots(h5_path, image_group, spot_group, max_slides_to_show=5, cols=1, plot_size_per_image=(10, 8), dot_color='red', dot_size=15, dot_alpha=0.6):\n    \"\"\"\n    Visualizes histology slides with spatial transcriptomics spot overlays.\n\n    Args:\n        h5_path (str): Path to the HDF5 file.\n        image_group (str): The HDF5 group for images (e.g., 'images/Train').\n        spot_group (str): The HDF5 group for spot data (e.g., 'spots/Train').\n        max_slides_to_show (int): The maximum number of slides to display.\n        cols (int): The number of columns in the subplot grid.\n        plot_size_per_image (tuple): The (width, height) in inches for each subplot.\n        dot_color (str): The color of the spot markers.\n        dot_size (int): The size of the spot markers.\n        dot_alpha (float): The transparency of the spot markers.\n    \"\"\"\n    h5file = None\n    try:\n        h5file = h5py.File(h5_path, \"r\")\n        images_data = h5file[image_group]\n        spots_data = h5file[spot_group]\n\n        slide_names = list(images_data.keys())\n        slides_to_show = slide_names[:min(len(slide_names), max_slides_to_show)]\n        num_slides = len(slides_to_show)\n\n        if num_slides == 0:\n            print(f\"No slides found in group '{image_group}'.\")\n            return\n\n        # Calculate rows and figure dimensions based on desired plot size\n        rows = math.ceil(num_slides / cols)\n        fig_width = plot_size_per_image[0] * cols\n        fig_height = plot_size_per_image[1] * rows\n        \n        fig, axes = plt.subplots(rows, cols, figsize=(fig_width, fig_height), squeeze=False)\n        axes = axes.flatten()\n\n        for i, slide_name in enumerate(slides_to_show):\n            image = np.array(images_data[slide_name])\n            spots = np.array(spots_data[slide_name])\n            \n            axes[i].imshow(image, aspect=\"equal\") # 'equal' aspect ratio is often better for images\n            axes[i].scatter(spots[\"x\"], spots[\"y\"], color=dot_color, s=dot_size, alpha=dot_alpha)\n            axes[i].set_title(slide_name, fontsize=12, pad=10)\n            axes[i].axis('off')\n\n        # Hide unused subplots\n        for j in range(num_slides, len(axes)):\n            axes[j].axis('off')\n            \n        plt.tight_layout()\n        plt.show()\n\n    except KeyError as e:\n        print(f\"Error: Could not find HDF5 group {e}. Please check the path.\")\n    except Exception as e:\n        print(f\"An unexpected error occurred: {e}\")\n    finally:\n        if h5file:\n            h5file.close()\n            print(\"HDF5 file closed successfully.\")\n\n\n# --- How to use the function to get the desired 5x1 layout ---\nH5_FILE_PATH = \"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\"\n\n# Visualize 5 training slides in 1 column and 5 rows with a large view\nvisualize_slides_with_spots(\n    h5_path=H5_FILE_PATH,\n    image_group=\"images/Train\",\n    spot_group=\"spots/Train\",\n    max_slides_to_show=5,\n    cols=1,  # <-- Set to 1 column\n    plot_size_per_image=(10, 8),  # <-- Set a large size for each plot (width, height)\n    dot_size=15,  # <-- Increased dot size for visibility\n    dot_alpha=0.6  # <-- Increased opacity for more prominent spots\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:35:40.088947Z","iopub.execute_input":"2025-08-11T13:35:40.089542Z","iopub.status.idle":"2025-08-11T13:35:47.041000Z","shell.execute_reply.started":"2025-08-11T13:35:40.089518Z","shell.execute_reply":"2025-08-11T13:35:47.039840Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Visualize Test slide ('S_7')\nimport h5py\nimport numpy as np\nimport matplotlib.pyplot as plt\n\ndef visualize_single_slide(h5_path, slide_id, image_group, spot_group, figsize=(10, 8), dot_color='red', dot_size=15, dot_alpha=0.6):\n    \"\"\"\n    Visualizes a single histology slide and its corresponding spot overlay.\n\n    Args:\n        h5_path (str): Path to the HDF5 file.\n        slide_id (str): The name/ID of the specific slide to visualize (e.g., 'S_7').\n        image_group (str): The HDF5 group for images (e.g., 'images/Test').\n        spot_group (str): The HDF5 group for spot data (e.g., 'spots/Test').\n        figsize (tuple): The (width, height) in inches for the figure.\n        dot_color (str): The color of the spot markers.\n        dot_size (int): The size of the spot markers.\n        dot_alpha (float): The transparency of the spot markers.\n    \"\"\"\n    try:\n        with h5py.File(h5_path, \"r\") as h5file:\n            images_data = h5file[image_group]\n            spots_data = h5file[spot_group]\n\n            # --- Robustness Check: Ensure the slide exists before trying to access it ---\n            if slide_id not in images_data:\n                print(f\"Error: Slide ID '{slide_id}' not found in image group '{image_group}'.\")\n                print(f\"Available slides are: {list(images_data.keys())}\")\n                return\n\n            # Retrieve data for the specific slide\n            image = np.array(images_data[slide_id])\n            spots = np.array(spots_data[slide_id])\n            \n            # Plotting\n            plt.figure(figsize=figsize)\n            plt.imshow(image, aspect='equal') # 'equal' is often better for preserving tissue shape\n            plt.scatter(spots[\"x\"], spots[\"y\"], color=dot_color, s=dot_size, alpha=dot_alpha)\n            plt.title(f\"Slide: {slide_id}\", fontsize=14)\n            plt.axis('off')\n            plt.show()\n\n    except KeyError as e:\n        print(f\"Error: Could not find HDF5 group {e}. Please check the path.\")\n    except Exception as e:\n        print(f\"An unexpected error occurred: {e}\")\n\n\n# --- How to use the new, flexible function ---\nH5_FILE_PATH = \"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\"\n\n# Example 1: Replicate the original visualization for 'S_7' but with more prominent spots\nprint(\"Visualizing Test slide 'S_7' with prominent spots...\")\nvisualize_single_slide(\n    h5_path=H5_FILE_PATH,\n    slide_id='S_7',\n    image_group='images/Test',\n    spot_group='spots/Test',\n    figsize=(12, 10), # Slightly larger figure\n    dot_size=20       # Larger dots\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:35:47.041850Z","iopub.execute_input":"2025-08-11T13:35:47.042053Z","iopub.status.idle":"2025-08-11T13:35:48.422434Z","shell.execute_reply.started":"2025-08-11T13:35:47.042031Z","shell.execute_reply":"2025-08-11T13:35:48.421694Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. Loading and Displaying Spots Tables\n\nThe following cells load and display the (x, y) coordinates and cell type annotations for Train slides and the spot coordinates for the Test slide.  ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\n# Load and display (x,y) spot locations and cell type annotation table for Train slides\nwith h5py.File(\"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\", \"r\") as f:\n    train_spots = f[\"spots/Train\"]\n    \n    # Dictionary to store DataFrames for each slide\n    train_spot_tables = {}\n    \n    for slide_name in train_spots.keys():\n        # Load dataset as NumPy structured array\n        spot_array = np.array(train_spots[slide_name])\n        \n        # Convert to DataFrame\n        df = pd.DataFrame(spot_array)\n        \n        # Store in dictionary\n        train_spot_tables[slide_name] = df\n\n# Example: Display the spots table for slide 'S_1'\ntrain_spot_tables['S_1']","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:35:48.424433Z","iopub.execute_input":"2025-08-11T13:35:48.424649Z","iopub.status.idle":"2025-08-11T13:35:48.881892Z","shell.execute_reply.started":"2025-08-11T13:35:48.424631Z","shell.execute_reply":"2025-08-11T13:35:48.881107Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Display spot table for Test slide (only the spot coordinates on 2D array)\nwith h5py.File(\"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\", \"r\") as f:\n    test_spots = f[\"spots/Test\"]\n    spot_array = np.array(test_spots['S_7'])\n    test_spot_table = pd.DataFrame(spot_array)\n    \n# Show the test spots coordinates for slide 'S_7'\ntest_spot_table","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:35:48.882805Z","iopub.execute_input":"2025-08-11T13:35:48.883248Z","iopub.status.idle":"2025-08-11T13:35:48.905247Z","shell.execute_reply.started":"2025-08-11T13:35:48.883219Z","shell.execute_reply":"2025-08-11T13:35:48.904657Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. Sanity Check: Random Predictions\n\nIn this section, we generate random predictions for the 35 cell types.  \nThe predictions are random floats between 0 and 2 (without any normalization) for each spot in the Test slide.  \nThe order of spots is preserved as in the test spots table.","metadata":{}},{"cell_type":"code","source":"# Create a random submission\n# (predictions of cell type abundances for 35 classes across the Test slide spots;\n# spot order should be same as in the 'Test' spots table)\n\n# Use the cell type columns from the train spots table; assuming first two columns are (x, y)\ncell_type_columns = train_spot_tables['S_1'].columns[2:].values  # Expecting 35 cell types here\nindices = test_spot_table.index.values  # All spots on the Test slide\n\n# Create a 2D array of random floats between 0 and 2 for each spot and cell type\nprediction_matrix = 2 * np.random.rand(len(indices), len(cell_type_columns))\npredicted_labels = pd.DataFrame(prediction_matrix, columns=cell_type_columns, index=indices)\n\npredicted_labels.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:35:48.905895Z","iopub.execute_input":"2025-08-11T13:35:48.906132Z","iopub.status.idle":"2025-08-11T13:35:48.925492Z","shell.execute_reply.started":"2025-08-11T13:35:48.906109Z","shell.execute_reply":"2025-08-11T13:35:48.924932Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. Submission File Generation\n\nFinally, we generate the submission file in the required format.  \nEach row corresponds to a test spot with its identifier (constructed here as the index) followed by 35 predictions.","metadata":{}},{"cell_type":"code","source":"# Prepare submission DataFrame: spot_id column and then predictions for each cell type\nsubmission_df = predicted_labels.copy()\nsubmission_df.insert(0, 'ID', submission_df.index)\n\n# Save the submission file as submission.csv\nsubmission_df.to_csv(\"./submission.csv\", index=False)\nprint(\"Submission file 'submission.csv' created!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:35:48.926081Z","iopub.execute_input":"2025-08-11T13:35:48.926315Z","iopub.status.idle":"2025-08-11T13:35:49.060782Z","shell.execute_reply.started":"2025-08-11T13:35:48.926296Z","shell.execute_reply":"2025-08-11T13:35:49.059943Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Functionality of the Starter Notebook:\n\n**Loads Libraries:** Imports h5py (for HDF5 files), numpy, pandas, and matplotlib.\n\n**Visualizes Data (Section 1):**\nDemonstrates how to open the elucidata_ai_challenge_data.h5 file.\n\nShows how to access the image data (h5file[\"images/Train\"][slide_name]) and spot data (h5file[\"spots/Train\"][slide_name]).\n\nLoads the image data directly into NumPy arrays (np.array(...)).\n\nLoads spot data as NumPy structured arrays and extracts x, y coordinates.\n\nUses matplotlib to display the full H&E images for training slides and the test slide, overlaying \nthe spot locations as red dots to visually confirm alignment.\n\n**Loads Spot Data into DataFrames (Section 2):**\n\nCrucially, shows how to convert the NumPy structured arrays containing spot coordinates and cell abundances (C1...C35) into easily usable pandas DataFrames. This is done for both training (train_spot_tables dictionary) and test (test_spot_table) spots. This will be very useful for your Dataset implementation.\n\n**Generates Random Predictions (Section 3):**\n\nDemonstrates the structure of the required predictions.\nGets the cell type column names (C1...C35) dynamically from one of the training spot tables.\nGets the indices (which serve as spot IDs) from the test_spot_table.\nCreates a pandas DataFrame (predicted_labels) where rows are spot indices and columns are the 35 cell types, filled with random values.\n\n**Creates Submission File (Section 4):**\n\nTakes the random prediction DataFrame.\nAdds an 'ID' column using the DataFrame's index (submission_df.insert(0, 'ID', submission_df.index)). This confirms the spot's row index in the test table should be used as its ID.\nSaves this DataFrame to submission.csv in the correct format using to_csv(..., index=False).\n\n**Key Code Snippets & Learnings for Your Pipeline:**\n\n**HDF5 Loading:** The pattern with h5py.File(...) as f: and accessing data via keys like f[\"images/Train\"][slide_id] and f[\"spots/Train\"][slide_id] is how you'll get your raw data.\nSpot Data Handling: Converting the spot data to pandas DataFrames (pd.DataFrame(np.array(f[\"spots/...\"]))) is the most convenient way to work with coordinates and target abundances.\n\n**Image Data Handling:** The images are directly available as NumPy arrays (np.array(f[\"images/...\"][slide_id])). This means you don't need complex WSI libraries like openslide. You can directly use NumPy slicing.\n\n**Submission Format:** Sections 3 and 4 provide the exact template for creating your final \n\nsubmission.csv from your model's predictions. You need a DataFrame with an 'ID' column (spot index) and columns C1 through C35.\n\n# What's MISSING (What You Need to Implement Next):\n\n**Patch Extraction:** The notebook loads the entire image. It does not show how to extract a specific image patch (e.g., 256x256 pixels) centered around a given (x, y) spot coordinate from the large image NumPy array. This will be the core of your Dataset's __getitem__ method using NumPy slicing.\n\n**Dataset Class:** A PyTorch Dataset class to handle loading data, extracting patches, applying transforms, and returning (patch_tensor, target_tensor) or (patch_tensor, spot_id).\nModel Definition: A neural network model (likely CNN-based) that takes an image patch tensor and outputs 35 predicted abundance values.\n\n**Training Loop:** The code to iterate through data, calculate loss (e.g., MSE), perform backpropagation, update weights, and evaluate using the average Spearman correlation metric.\n\n**Inference Loop:** The code to process the test spots, get predictions, and format them for submission.","metadata":{}},{"cell_type":"markdown","source":"# 1. Understanding the Data and Problem:\n\nGoal: Predict the abundance of 35 cell types (C1-C35) for specific spatial spots on H&E tissue slides.\nInput: HDF5 file containing:\nWhole-slide H&E images as NumPy arrays.\nSpot coordinates (x, y) and, for training, corresponding cell type abundances.\n\nOutput: A submission.csv file with ID (spot identifier) and 35 predicted abundance columns.\nEvaluation: Average per-spot Spearman rank correlation. This means the relative order of cell type abundances within each spot is more important than their absolute values.\n\n# 2. Data Loading and Preprocessing Pipeline (PyTorch Dataset):\n\n**HDF5 Handling:** Use h5py to read data. The starter notebook confirmed the structure.\n\n**Spot Metadata Management:**\nLoad spot coordinates and (for train/val) target abundances into a list or structure that can be easily indexed. Pandas DataFrames were used initially to parse this.\nStore essential info: (slide_id, spot_original_idx, x, y, targets_or_id).\n\n**Image Caching:**\nLoad full slide images (NumPy arrays) into an in-memory cache (self.images_cache) within the Dataset class to avoid repeated HDF5 reads for spots on the same slide, significantly speeding up data loading.\n\n**Patch Extraction (extract_patch function):**\nGiven a full slide image (NumPy array) and a spot's (center_x, center_y) coordinates, extract a square image patch (e.g., 256x256 pixels) centered on the spot.\nImplement boundary handling (e.g., zero-padding) if the patch extends beyond the image edges.\n\n**Data Type Conversion (Crucial Fix):**\nThe HDF5 images were loaded as float32. The extract_patch function preserved this.\ntorchvision.transforms (especially Image.fromarray for PIL conversion) expect uint8 [0-255] image data.\n\n**Solution:** In the Dataset.__getitem__, convert the float32 patch to uint8 (clipping to 0-255 and then astype(np.uint8)) before passing it to Image.fromarray() or other transforms.\n\n**Transformations (torchvision.transforms):**\nConvert the uint8 NumPy patch to a PIL Image (Image.fromarray()).\n\n**Training Augmentations:** RandomHorizontalFlip, RandomVerticalFlip, ColorJitter. (Could be expanded with RandomAffine, Gaussian Blur, or specialized stain augmentation).\n\n**Common Transforms:** Resize (if needed for consistency), ToTensor() (converts PIL to (C,H,W) tensor and scales to [0,1]), Normalize() (using ImageNet stats as a starting point).\n\n**Mode Handling:** The Dataset distinguishes between 'train', 'val', and 'test' modes to load targets or spot IDs accordingly.\n\n# 3. Model Architecture:\n   \n**Pretrained CNN Backbone:** Use a standard pretrained model like ResNet18 (loaded with IMAGENET1K_V1 weights) for transfer learning. This leverages features learned from a large dataset.\n\n**Regression Head:** Modify the final fully connected layer (model.fc) of the ResNet to output N_CELL_TYPES (35) continuous values.\n\n**Regularization (Addressing Overfitting):**\n\nDropout layer added before the final linear layer.\nWeight decay (L2 regularization) included in the optimizer.\n\n# 4. Training and Validation Strategy:\n\n**Loss Function:** nn.MSELoss() (Mean Squared Error) was chosen as a standard regression loss.\n\n**Optimizer:** optim.AdamW (Adam with decoupled weight decay).\n\n**Learning Rate Scheduler:** optim.lr_scheduler.ReduceLROnPlateau to reduce the learning rate when the validation Spearman average stops improving.\n\n**Evaluation Metric Function (calculate_average_spearman):**\nCustom function to compute the per-spot Spearman correlation and then average these scores, matching the competition metric.\nHandles edge cases like constant prediction/target arrays for a spot.\n\n**Training Loop:** Standard epoch-based training with batch iteration.\n\n**Validation Loop:** Evaluate on a separate validation set after each training epoch to monitor generalization.\n\n**Early Stopping:** Stop training if the validation Spearman average doesn't improve for a specified PATIENCE number of epochs. Saves the model weights corresponding to the best validation metric.\n\n# 5. Prediction and Submission:\n   \n**Load Best Model:** After training, load the model weights that achieved the best validation Spearman score.\n\n**Inference:** Run the model on the test_loader in evaluation mode (model.eval()) with torch.no_grad().\n\n**Collect Predictions and IDs:** Store the model's output predictions and the corresponding spot IDs.\n\n**ID Formatting (Crucial Fix):**\nThe spot_ids from the DataLoader were PyTorch tensors.\n\n**Solution:** Convert the tensor batch of spot IDs to a Python list of integers (.cpu().tolist()) before adding them to the master list of IDs.\n\n**Generate submission.csv:** Create a Pandas DataFrame with an 'ID' column and columns for each of the 35 cell types (C1...C35), then save it to submission.csv with index=False.\n\n# 6. Iterative Development and Debugging:\n\nThe process involved identifying and fixing key issues:\n\nIncorrect data type for image transforms.\nIncorrect formatting of spot IDs in the submission file.\nRecognizing and planning strategies to address overfitting (though the very short training run didn't fully expose the extent, it's a common problem in such tasks).\n\nThis structured approach, combining standard deep learning practices with specific adaptations for the competition's data format and evaluation metric, forms the core of the solution.","metadata":{}},{"cell_type":"code","source":"!pip install -U albumentations","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:35:49.061647Z","iopub.execute_input":"2025-08-11T13:35:49.061892Z","iopub.status.idle":"2025-08-11T13:35:53.705858Z","shell.execute_reply.started":"2025-08-11T13:35:49.061876Z","shell.execute_reply":"2025-08-11T13:35:53.705147Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Imports and Global Setup","metadata":{}},{"cell_type":"code","source":"import torch\nfrom torch.utils.data import Dataset, DataLoader\nimport h5py\nimport numpy as np\nimport pandas as pd\nimport logging\nfrom collections import OrderedDict\nfrom typing import Dict, List, Tuple, Optional\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\nfrom sklearn.model_selection import GroupKFold","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:35:53.706856Z","iopub.execute_input":"2025-08-11T13:35:53.707057Z","iopub.status.idle":"2025-08-11T13:36:03.334303Z","shell.execute_reply.started":"2025-08-11T13:35:53.707038Z","shell.execute_reply":"2025-08-11T13:36:03.333698Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Core Utility Functions and Classes","metadata":{}},{"cell_type":"code","source":"# Set up logging\nlogging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\nlogger = logging.getLogger(__name__)\n\ndef extract_patch(image_array, center_x, center_y, patch_size):\n    \"\"\"\n    Extracts a square patch from a large image array centered at (center_x, center_y).\n    Handles boundary conditions by padding with zeros if the patch extends beyond the image.\n\n    Args:\n        image_array (np.ndarray): The large image array (H, W, C).\n        center_x (float or int): The x-coordinate (column) of the patch center.\n        center_y (float or int): The y-coordinate (row) of the patch center.\n        patch_size (int): The side length of the square patch (e.g., 256).\n\n    Returns:\n        np.ndarray: The extracted patch (patch_size, patch_size, C)\n                    or None if dimensions are invalid.\n    \"\"\"\n    try:\n        img_h, img_w, img_c = image_array.shape\n\n        # Ensure patch_size is positive\n        if patch_size <= 0:\n            logger.error(\"Error: patch_size must be positive.\")\n            return None\n\n        half_patch = patch_size // 2\n\n        # Calculate integer center coordinates\n        center_y_int = int(round(center_y))\n        center_x_int = int(round(center_x))\n\n        # Calculate boundaries of the patch in the large image coordinate system\n        y_start_ideal = center_y_int - half_patch\n        y_end_ideal = y_start_ideal + patch_size  # End is exclusive in slicing\n        x_start_ideal = center_x_int - half_patch\n        x_end_ideal = x_start_ideal + patch_size  # End is exclusive in slicing\n\n        # Create the output patch, initialized with zeros\n        patch = np.zeros((patch_size, patch_size, img_c), dtype=image_array.dtype)\n\n        # Determine the intersection of the ideal patch and the actual image boundaries\n        y_start_img = max(0, y_start_ideal)\n        y_end_img = min(img_h, y_end_ideal)\n        x_start_img = max(0, x_start_ideal)\n        x_end_img = min(img_w, x_end_ideal)\n\n        # Calculate where to place this intersection within the output patch\n        y_offset_patch = max(0, -y_start_ideal)\n        x_offset_patch = max(0, -x_start_ideal)\n\n        # Calculate the height and width of the region to copy\n        copy_height = y_end_img - y_start_img\n        copy_width = x_end_img - x_start_img\n\n        # Check if there's any valid region to copy\n        if copy_height > 0 and copy_width > 0:\n            # Copy the valid region from the image array to the correct position in the patch\n            patch[\n                y_offset_patch : y_offset_patch + copy_height,\n                x_offset_patch : x_offset_patch + copy_width,\n                :\n            ] = image_array[\n                y_start_img : y_end_img,\n                x_start_img : x_end_img,\n                :\n            ]\n\n        return patch\n    except Exception as e:\n        logger.error(f\"Error in extract_patch: {e}\")\n        return None\n\n\nclass LRUCache(OrderedDict):\n    \"\"\"\n    Limit-size cache that discards the least recently used items.\n    \"\"\"\n    def __init__(self, maxsize=128):\n        self.maxsize = maxsize\n        super().__init__()\n\n    def __getitem__(self, key):\n        value = super().__getitem__(key)\n        self.move_to_end(key)\n        return value\n\n    def __setitem__(self, key, value):\n        if key in self:\n            self.move_to_end(key)\n        super().__setitem__(key, value)\n        if len(self) > self.maxsize:\n            oldest = next(iter(self))\n            del self[oldest]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:36:03.336477Z","iopub.execute_input":"2025-08-11T13:36:03.336827Z","iopub.status.idle":"2025-08-11T13:36:03.345989Z","shell.execute_reply.started":"2025-08-11T13:36:03.336812Z","shell.execute_reply":"2025-08-11T13:36:03.345338Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# The DataFactory Class Definition\n\n## Purpose: Define the class responsible for the heavy, one-time data preparation.","metadata":{}},{"cell_type":"code","source":"# --- MODULARITY IMPROVEMENT 1: DATA PREPARATION FACTORY ---\n# This class handles all the one-time data loading and preparation.\nclass DataFactory:\n    \"\"\"A builder class to prepare and split the dataset metadata efficiently.\"\"\"\n    def __init__(self, h5_path: str):\n        self.h5_path = h5_path\n        logger.info(f\"DataFactory initialized with H5 file: {h5_path}\")\n        self.full_spots_df = self._load_all_spots()\n        self.target_cols = [col for col in self.full_spots_df.columns if col.startswith('C')]\n        if self.target_cols:\n            logger.info(f\"Identified {len(self.target_cols)} target columns: {self.target_cols}\")\n            # Ensure correct dtype at the source\n            self.full_spots_df[self.target_cols] = self.full_spots_df[self.target_cols].astype(np.float32)\n            logger.info(\"Converted target columns to float32 dtype.\")\n        else:\n            logger.error(\"CRITICAL ERROR: No target columns (starting with 'c') found! Check data loading.\")\n\n\n    def _load_all_spots(self) -> pd.DataFrame:\n        \"\"\"\n        Loads all spot data from all training slides into a single, efficient DataFrame.\n        \"\"\"\n        all_spots_list = []\n        with h5py.File(self.h5_path, \"r\") as f:\n            train_spots_group = f['spots/Train']\n            slide_ids = list(train_spots_group.keys())\n            logger.info(\"Loading all training spots into a unified DataFrame...\")\n            for slide_id in slide_ids:\n                # Pass the structured H5 dataset directly to pd.DataFrame to preserve column names.\n                # The [()] ensures the data is fully read into memory.\n                spots_df = pd.DataFrame(train_spots_group[slide_id][()])\n                \n                spots_df['slide_id'] = slide_id\n                all_spots_list.append(spots_df)\n        \n        full_df = pd.concat(all_spots_list, ignore_index=True)\n        logger.info(f\"Total spots loaded from all slides: {len(full_df)}\")\n        # For debugging, let's see the columns\n        logger.info(f\"DataFrame columns loaded: {full_df.columns.tolist()}\")\n        return full_df\n    \n    # ... (the rest of your DataFactory class is fine) ...\n    def get_split_data(self, train_ids: List[str], val_ids: List[str]) -> Tuple[pd.DataFrame, pd.DataFrame]:\n        train_df = self.full_spots_df[self.full_spots_df['slide_id'].isin(train_ids)].reset_index(drop=True)\n        val_df = self.full_spots_df[self.full_spots_df['slide_id'].isin(val_ids)].reset_index(drop=True)\n        logger.info(f\"Train set created with {len(train_df)} spots from {len(train_ids)} slides.\")\n        logger.info(f\"Validation set created with {len(val_df)} spots from {len(val_ids)} slides.\")\n        return train_df, val_df\n        \n    def get_test_data(self, test_slide_ids: List[str]) -> pd.DataFrame:\n        test_spots_list = []\n        with h5py.File(self.h5_path, \"r\") as f:\n            test_spots_group = f['spots/Test']\n            for slide_id in test_slide_ids:\n                if slide_id in test_spots_group:\n                    # Apply the same fix for test data\n                    spots_df = pd.DataFrame(test_spots_group[slide_id][()])\n                    spots_df['slide_id'] = slide_id\n                    test_spots_list.append(spots_df)\n        \n        test_df = pd.concat(test_spots_list, ignore_index=True)\n        logger.info(f\"Test set created with {len(test_df)} spots.\")\n        return test_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:36:03.346969Z","iopub.execute_input":"2025-08-11T13:36:03.347500Z","iopub.status.idle":"2025-08-11T13:36:03.378227Z","shell.execute_reply.started":"2025-08-11T13:36:03.347478Z","shell.execute_reply":"2025-08-11T13:36:03.377621Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# The HESpotDataset and Augmentation Functions\n\n## Purpose: Define the main PyTorch Dataset class and the helper function that creates the augmentation pipelines. This cell defines how data will be served.","metadata":{}},{"cell_type":"code","source":"# --- MODULARITY IMPROVEMENT 2: A LEAN AND EFFICIENT DATASET CLASS ---\n# This class's only job is to serve data. It's fast and clean.\nclass HESpotDataset(Dataset):\n    def __init__(self, h5_path: str, spots_df: pd.DataFrame, patch_size: int, \n                 target_cols: List[str], transforms: A.Compose, mode: str = 'train'):\n        self.h5_path = h5_path\n        self.spots_df = spots_df\n        self.patch_size = patch_size\n        self.target_cols = target_cols\n        self.transforms = transforms\n        self.mode = mode\n        \n        # We will use a simple cache for the full images\n        self.image_cache = LRUCache(maxsize=8) # LRUCache class should be defined in a prior cell\n\n    def __len__(self) -> int:\n        return len(self.spots_df)\n\n    def __getitem__(self, idx: int) -> Tuple[torch.Tensor, torch.Tensor]:\n        spot_info = self.spots_df.iloc[idx]\n        slide_id = spot_info['slide_id']\n        \n        # Load full image using the LRU cache\n        try:\n            image_array = self.image_cache[slide_id]\n        except KeyError:\n            # If not in cache, open the file, read the image, and close.\n            # This is simpler and more robust for notebook environments.\n            with h5py.File(self.h5_path, 'r') as h5file:\n                group = 'Train' if self.mode in ['train', 'val'] else 'Test'\n                image_array = np.array(h5file[f'images/{group}'][slide_id])\n                self.image_cache[slide_id] = image_array\n        \n        # Extract patch\n        patch = extract_patch(image_array, spot_info['x'], spot_info['y'], self.patch_size)\n        \n        # Apply transforms using Albumentations\n        if self.transforms:\n            transformed = self.transforms(image=patch)\n            patch_tensor = transformed['image']\n        else: # If no transforms, just convert to tensor\n            patch_tensor = torch.from_numpy(patch).permute(2, 0, 1)\n\n        # Prepare target\n        if self.mode in ['train', 'val']:\n            target_np = spot_info[self.target_cols].values.astype(np.float32)\n            target = torch.tensor(target_np, dtype=torch.float32)\n        else:\n            target = torch.tensor([spot_info['x'], spot_info['y']], dtype=torch.float32)\n\n        return patch_tensor, target\n        \ndef get_transforms(mode: str, patch_size: int) -> A.Compose:\n    \"\"\"\n    Final, correct transforms. The data is already float32 in [0, 1] range.\n    We only need geometric augmentations and conversion to tensor. NO NORMALIZATION.\n    \"\"\"\n    if mode == 'train':\n        return A.Compose([\n            A.HorizontalFlip(p=0.5),\n            A.VerticalFlip(p=0.5),\n            A.RandomRotate90(p=0.5),\n            A.ShiftScaleRotate(shift_limit=0.05, scale_limit=0.1, rotate_limit=15, p=0.3),\n            A.Resize(patch_size, patch_size),\n            # NO A.Normalize() NEEDED!\n            ToTensorV2(), # Converts numpy (H,W,C) [0,1] to tensor (C,H,W) [0,1]\n        ])\n    else: # For validation and test\n        return A.Compose([\n            A.Resize(patch_size, patch_size),\n            # NO A.Normalize() NEEDED!\n            ToTensorV2(),\n        ])\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:36:03.378900Z","iopub.execute_input":"2025-08-11T13:36:03.379059Z","iopub.status.idle":"2025-08-11T13:36:03.401820Z","shell.execute_reply.started":"2025-08-11T13:36:03.379047Z","shell.execute_reply":"2025-08-11T13:36:03.401309Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Configuration and Data Preparation\n\nPurpose: This the main control panel. All hyperparameters are set here. \n\nThen instantiate the DataFactory and run the slow data preparation step just once. \n\nYou can change hyperparameters and re-run this cell without having to re-define all the classes.","metadata":{}},{"cell_type":"code","source":"# --- Main Execution Block ---\nif __name__ == '__main__':\n    # --- Configuration ---\n    H5_FILE_PATH = \"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\"\n    PATCH_SIZE = 256\n    BATCH_SIZE = 32\n    NUM_WORKERS = 2 \n\n    # 1. Create the DataFactory (it now discovers slide names and column names automatically)\n    data_factory = DataFactory(H5_FILE_PATH)\n    \n    # 2. Define slide splits dynamically based on discovered slides\n    all_train_slides = data_factory.full_spots_df['slide_id'].unique().tolist()\n    \n    # Use a robust split (e.g., last slide for validation)\n    if len(all_train_slides) > 1:\n        train_ids = all_train_slides[:-1]\n        val_ids = [all_train_slides[-1]]\n    else: # Handle case with only one slide\n        train_ids = all_train_slides\n        val_ids = all_train_slides\n        \n    # Assume 'S_7' is the correct test slide name\n    test_ids = ['S_7'] \n\n    # 3. Get the pre-processed DataFrames\n    train_df, val_df = data_factory.get_split_data(train_ids, val_ids)\n    test_df = data_factory.get_test_data(test_ids)\n\n    # 4. Create the lean Dataset objects\n    # (HESpotDataset definition should be in a cell before this)\n    train_dataset = HESpotDataset(\n        h5_path=H5_FILE_PATH,\n        spots_df=train_df,\n        patch_size=PATCH_SIZE,\n        target_cols=data_factory.target_cols,\n        transforms=get_transforms('train', PATCH_SIZE), # get_transforms should also be defined\n        mode='train'\n    )\n    val_dataset = HESpotDataset(\n        h5_path=H5_FILE_PATH,\n        spots_df=val_df,\n        patch_size=PATCH_SIZE,\n        target_cols=data_factory.target_cols,\n        transforms=get_transforms('val', PATCH_SIZE),\n        mode='val'\n    )\n\n    # 5. Create DataLoaders\n    train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKERS)\n    val_loader = DataLoader(val_dataset, batch_size=BATCH_SIZE * 2, shuffle=False, num_workers=NUM_WORKERS)\n\n    logger.info(f\"Setup complete. Training on {len(train_dataset)} spots, validating on {len(val_dataset)} spots.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:36:03.402473Z","iopub.execute_input":"2025-08-11T13:36:03.402720Z","iopub.status.idle":"2025-08-11T13:36:03.525264Z","shell.execute_reply.started":"2025-08-11T13:36:03.402704Z","shell.execute_reply":"2025-08-11T13:36:03.524552Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Create Datasets and DataLoaders\n\nPurpose: Use the DataFrames from the previous cell to create the final Dataset and DataLoader objects.","metadata":{}},{"cell_type":"code","source":"# 5. Create DataLoaders\ntrain_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKERS)\nval_loader = DataLoader(val_dataset, batch_size=BATCH_SIZE * 2, shuffle=False, num_workers=NUM_WORKERS)\n\nlogger.info(f\"Setup complete. Training on {len(train_dataset)} spots, validating on {len(val_dataset)} spots.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:36:03.525978Z","iopub.execute_input":"2025-08-11T13:36:03.526251Z","iopub.status.idle":"2025-08-11T13:36:03.530489Z","shell.execute_reply.started":"2025-08-11T13:36:03.526232Z","shell.execute_reply":"2025-08-11T13:36:03.529796Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Sanity Check and Visualization\n\nPurpose: To visualize a batch of my data to ensure the entire pipeline is working as expected. To check that images look correct, augmentations are being applied, and tensor shapes are right.","metadata":{}},{"cell_type":"code","source":"# --- Final Visualization Cell ---\n\nlogger.info(\"Visualizing one batch from the training loader...\")\npatches, targets = next(iter(train_loader))\n\nprint(f\"Patches batch shape: {patches.shape}\")\nprint(f\"Targets batch shape: {targets.shape}\")\n\nfig, axes = plt.subplots(4, 4, figsize=(10, 10))\nfor i, ax in enumerate(axes.flat):\n    if i >= len(patches): break\n    \n    # The data is already in the [0, 1] range. No un-normalization needed.\n    patch = patches[i].permute(1, 2, 0).numpy()\n    patch = np.clip(patch, 0, 1) # Clipping is a good safety measure\n    \n    ax.imshow(patch)\n    ax.set_title(f\"Target C1: {targets[i][0]:.2f}\", fontsize=8)\n    ax.axis('off')\n    \nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:36:03.531469Z","iopub.execute_input":"2025-08-11T13:36:03.531680Z","iopub.status.idle":"2025-08-11T13:36:06.939653Z","shell.execute_reply.started":"2025-08-11T13:36:03.531664Z","shell.execute_reply":"2025-08-11T13:36:06.938436Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model Definition","metadata":{}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nimport torchvision.transforms as transforms\nimport torchvision.models as models\nfrom PIL import Image\nimport h5py\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm # Use tqdm for non-notebook envs\nimport os\nimport time\nfrom scipy.stats import spearmanr # For evaluation metric\n\n# --- Configuration & Device ---\nPATCH_SIZE = 256\nBATCH_SIZE = 32 # Adjust based on GPU memory\nNUM_WORKERS = 2\nLEARNING_RATE = 1e-4\nWEIGHT_DECAY = 1e-4\nEPOCHS = 15 # Adjust as needed, use early stopping\nPATIENCE = 5 # For early stopping\nMODEL_SAVE_PATH = \"best_model.pth\"\nN_CELL_TYPES = 35 # Number of output values\n\n# Set device\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(f\"Using device: {device}\")\n\n# Define Train/Val/Test Slide IDs\nall_train_slides = [f'S_{i}' for i in range(1, 7)] # S_1 to S_6\n# Simple split for demonstration: S_1-S_5 train, S_6 val\ntrain_slide_ids = all_train_slides[:-1]\nval_slide_ids = [all_train_slides[-1]]\ntest_slide_ids = ['S_7']\n\n\n# --- Model Definition ---\ndef get_model(num_classes=N_CELL_TYPES, pretrained=True, dropout_p=0.4): # Added dropout_p argument\n    \"\"\"Loads a pretrained ResNet, adds dropout, and modifies the final layer for regression.\"\"\"\n    # Use weights argument for newer torchvision versions\n    if pretrained:\n        weights = models.ResNet18_Weights.IMAGENET1K_V1\n        model = models.resnet18(weights=weights)\n    else:\n        model = models.resnet18(weights=None)\n\n    # Get the number of input features for the original fc layer\n    num_ftrs = model.fc.in_features\n\n    # Replace the original fc layer with a new sequence including Dropout\n    model.fc = nn.Sequential(\n        nn.Dropout(p=dropout_p),      # Dropout layer\n        nn.Linear(num_ftrs, num_classes) # Final Linear layer for regression\n    )\n\n    return model\n\n# --- Instantiate the model with dropout ---\nDROPOUT_RATE = 0.5 # Define your desired dropout rate\nmodel = get_model(dropout_p=DROPOUT_RATE).to(device) # Pass the dropout rate\nprint(\"\\n--- Model Loaded (with Dropout) ---\")\n# print(model) # Optional: print model structure to see the new Dropout layer\n\n# --- Loss Function, Optimizer, Scheduler ---\ncriterion = nn.MSELoss() # Mean Squared Error is common for regression tasks like this\noptimizer = optim.AdamW(model.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)\n# Scheduler reduces learning rate when validation metric stops improving\nscheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', factor=0.1, patience=PATIENCE // 3, verbose=True) # Reduce on plateau of avg spearman\n\n# --- Evaluation Metric Function ---\ndef calculate_average_spearman(y_preds_all, y_targets_all):\n    \"\"\"Calculates the competition metric: average per-spot Spearman correlation.\"\"\"\n    if not isinstance(y_preds_all, np.ndarray):\n        y_preds_all = np.array(y_preds_all)\n    if not isinstance(y_targets_all, np.ndarray):\n        y_targets_all = np.array(y_targets_all)\n\n    if y_preds_all.shape != y_targets_all.shape:\n        print(f\"Shape mismatch: preds {y_preds_all.shape}, targets {y_targets_all.shape}\")\n        return 0.0 # Or raise error\n\n    num_spots = y_preds_all.shape[0]\n    if num_spots == 0:\n        return 0.0\n\n    spot_spearman_scores = []\n    for i in range(num_spots):\n        pred_spot = y_preds_all[i, :]\n        target_spot = y_targets_all[i, :]\n\n        # Check for constant arrays - spearmanr returns nan\n        if np.all(pred_spot == pred_spot[0]) or np.all(target_spot == target_spot[0]):\n             # If either prediction or target is constant for a spot, correlation is undefined.\n             # Assign 0 or handle as per competition rules (often 0 is reasonable).\n             # Or check if prediction matches target constant value -> 1.0? Let's use 0 for undefined.\n             score = 0.0\n        else:\n            correlation, p_value = spearmanr(pred_spot, target_spot)\n            if np.isnan(correlation):\n                # Handle potential NaNs from spearmanr if variance is zero (should be caught above mostly)\n                score = 0.0\n            else:\n                score = correlation\n        spot_spearman_scores.append(score)\n\n    return np.mean(spot_spearman_scores)\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:36:06.940864Z","iopub.execute_input":"2025-08-11T13:36:06.941144Z","iopub.status.idle":"2025-08-11T13:36:11.669014Z","shell.execute_reply.started":"2025-08-11T13:36:06.941119Z","shell.execute_reply":"2025-08-11T13:36:11.668222Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Trainer Class\n\nPurpose : Just as we created a DataFactory, we can create a Trainer class to encapsulate all the logic related to training, validation, and evaluation. This keeps your main notebook clean and makes your training pipeline reusable.\n","metadata":{}},{"cell_type":"code","source":"# A Trainer Class ---\nclass Trainer:\n    def __init__(self, model, train_loader, val_loader, criterion, optimizer, scheduler, device, config):\n        self.model = model\n        self.train_loader = train_loader\n        self.val_loader = val_loader\n        self.criterion = criterion\n        self.optimizer = optimizer\n        self.scheduler = scheduler\n        self.device = device\n        self.config = config\n        \n        self.history = {'train_loss': [], 'val_loss': [], 'val_spearman': []}\n        self.best_spearman = -1.0\n        self.epochs_no_improve = 0\n        \n    def _train_one_epoch(self):\n        \"\"\"Runs a single training epoch.\"\"\"\n        self.model.train()\n        total_loss = 0.0\n        \n        for patches, targets in tqdm(self.train_loader, desc=\"Training\"):\n            patches = patches.to(self.device)\n            targets = targets.to(self.device)\n            \n            # Forward pass\n            outputs = self.model(patches)\n            loss = self.criterion(outputs, targets)\n            \n            # Backward pass and optimization\n            self.optimizer.zero_grad()\n            loss.backward()\n            \n            # --- IMPROVEMENT: GRADIENT CLIPPING ---\n            # Prevents exploding gradients for more stable training\n            torch.nn.utils.clip_grad_norm_(self.model.parameters(), max_norm=1.0)\n            \n            self.optimizer.step()\n            \n            total_loss += loss.item()\n            \n        return total_loss / len(self.train_loader)\n\n    def _validate_one_epoch(self):\n        \"\"\"Runs a single validation epoch.\"\"\"\n        self.model.eval()\n        total_loss = 0.0\n        all_preds = []\n        all_targets = []\n        \n        with torch.no_grad():\n            for patches, targets in tqdm(self.val_loader, desc=\"Validating\"):\n                patches = patches.to(self.device)\n                targets = targets.to(self.device)\n                \n                outputs = self.model(patches)\n                loss = self.criterion(outputs, targets)\n                \n                total_loss += loss.item()\n                all_preds.extend(outputs.cpu().numpy())\n                all_targets.extend(targets.cpu().numpy())\n                \n        val_loss = total_loss / len(self.val_loader)\n        val_spearman = calculate_average_spearman(all_preds, all_targets)\n        return val_loss, val_spearman\n        \n    def fit(self):\n        \"\"\"The main training loop with early stopping.\"\"\"\n        start_time = time.time()\n        \n        for epoch in range(self.config['EPOCHS']):\n            print(f\"\\n--- Epoch {epoch + 1}/{self.config['EPOCHS']} ---\")\n            \n            train_loss = self._train_one_epoch()\n            val_loss, val_spearman = self._validate_one_epoch()\n            \n            self.history['train_loss'].append(train_loss)\n            self.history['val_loss'].append(val_loss)\n            self.history['val_spearman'].append(val_spearman)\n            \n            print(f\"Epoch Summary: Train Loss: {train_loss:.4f} | Val Loss: {val_loss:.4f} | Val Spearman: {val_spearman:.4f}\")\n            \n            # Learning rate scheduler step\n            self.scheduler.step(val_spearman)\n            \n            # --- IMPROVEMENT: EARLY STOPPING ---\n            if val_spearman > self.best_spearman:\n                print(f\"Validation Spearman improved from {self.best_spearman:.4f} to {val_spearman:.4f}. Saving model...\")\n                self.best_spearman = val_spearman\n                torch.save(self.model.state_dict(), self.config['MODEL_SAVE_PATH'])\n                self.epochs_no_improve = 0\n            else:\n                self.epochs_no_improve += 1\n                print(f\"No improvement in validation Spearman for {self.epochs_no_improve} epoch(s).\")\n                \n            if self.epochs_no_improve >= self.config['PATIENCE']:\n                print(f\"\\nEarly stopping triggered after {self.config['PATIENCE']} epochs without improvement.\")\n                break\n                \n        end_time = time.time()\n        print(f\"\\nTraining finished in {(end_time - start_time) / 60:.2f} minutes.\")\n        print(f\"Best validation Spearman score: {self.best_spearman:.4f}\")\n        \n        return self.history\n\n# --- Main Execution Block ---\nif __name__ == '__main__':\n    # (Assuming all data loading cells have been run and train_loader/val_loader exist)\n    \n    # 1. Define Model, Loss, Optimizer, etc.\n    model = get_model(num_classes=N_CELL_TYPES, dropout_p=0.5).to(device)\n    criterion = nn.MSELoss()\n    optimizer = optim.AdamW(model.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)\n    scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', factor=0.2, patience=2, verbose=True)\n    \n    # 2. Bundle config into a dictionary\n    config = {\n        \"EPOCHS\": EPOCHS,\n        \"PATIENCE\": PATIENCE,\n        \"MODEL_SAVE_PATH\": MODEL_SAVE_PATH\n    }\n\n    # 3. Instantiate and run the trainer\n    trainer = Trainer(model, train_loader, val_loader, criterion, optimizer, scheduler, device, config)\n    history = trainer.fit()\n\n    # 4. Plot training history\n    plt.figure(figsize=(12, 5))\n    plt.subplot(1, 2, 1)\n    plt.plot(history['train_loss'], label='Train Loss')\n    plt.plot(history['val_loss'], label='Validation Loss')\n    plt.title('Loss History')\n    plt.legend()\n    plt.subplot(1, 2, 2)\n    plt.plot(history['val_spearman'], label='Validation Spearman', color='orange')\n    plt.title('Validation Spearman Correlation History')\n    plt.legend()\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:36:11.669994Z","iopub.execute_input":"2025-08-11T13:36:11.670436Z","iopub.status.idle":"2025-08-11T13:40:10.788228Z","shell.execute_reply.started":"2025-08-11T13:36:11.670417Z","shell.execute_reply":"2025-08-11T13:40:10.787365Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Generating Test Set Predictions","metadata":{}},{"cell_type":"code","source":"import torch\nimport pandas as pd\nfrom tqdm.notebook import tqdm\nimport numpy as np\n\n# --- Configuration ---\n# Ensure these variables are consistent with your training setup\nMODEL_SAVE_PATH = \"best_model.pth\"\nSUBMISSION_PATH = \"submission.csv\"\nH5_FILE_PATH = \"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\"\nPATCH_SIZE = 256\nBATCH_SIZE = 64 # You can often use a larger batch size for inference\nNUM_WORKERS = 2\nN_CELL_TYPES = 35 # Must match the model's output\ntest_ids = ['S_7'] # The slide IDs for the test set\n\n# Use the same device as training\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(f\"Using device: {device} for inference.\")\n\n\n# --- 1. Load the Trained Model ---\nprint(\"\\n--- Loading best model weights ---\")\n# First, instantiate the model architecture (must be identical to the one you trained)\nmodel = get_model(num_classes=N_CELL_TYPES, pretrained=False) # pretrained=False is fine, we are loading our own weights\n\n# Load the saved state dictionary\nmodel.load_state_dict(torch.load(MODEL_SAVE_PATH))\n\n# Move the model to the appropriate device and set to evaluation mode\nmodel.to(device)\nmodel.eval()\nprint(\"Model loaded successfully.\")\n\n\n# --- 2. Create the Test DataLoader ---\n# Use the same DataFactory and Dataset classes as before\n# The DataFactory should already be instantiated as 'data_factory' from training cells\nprint(\"\\n--- Preparing test data ---\")\ntest_df = data_factory.get_test_data(test_ids)\n\n# We need the 'spot_id' for the submission file. The test DataFrame from the HDF5 file\n# might have different column names. Let's assume the spot identifier column is the first one\n# if it's not 'x' or 'y'. A more robust way is to check the dtype names.\nwith h5py.File(H5_FILE_PATH, \"r\") as f:\n    # Let's get the exact names from the test set file\n    test_col_names = [name.decode('utf-8') if isinstance(name, bytes) else name for name in f[f'spots/Test/{test_ids[0]}'].dtype.names]\n\n# Let's assume the submission ID is the 'Test set' column, or another unique identifier.\n# For this competition, let's create a unique ID from slide and spot index.\nif 'spot_id' not in test_df.columns:\n    test_df['spot_id'] = test_df.index.astype(str) # Create a default spot ID if not present\n\ntest_dataset = HESpotDataset(\n    h5_path=H5_FILE_PATH,\n    spots_df=test_df,\n    patch_size=PATCH_SIZE,\n    target_cols=[], # No target columns for test set\n    transforms=get_transforms('test', PATCH_SIZE), # Use 'val' or 'test' mode transforms\n    mode='test'\n)\n\ntest_loader = DataLoader(\n    test_dataset,\n    batch_size=BATCH_SIZE,\n    shuffle=False, # Never shuffle the test set!\n    num_workers=NUM_WORKERS\n)\nprint(f\"Test data loader created with {len(test_dataset)} spots.\")\n\n# --- 3. Prediction Loop ---\nprint(\"\\n--- Generating predictions on the test set ---\")\nall_preds = []\n\nwith torch.no_grad(): # Disable gradient calculations for speed and memory efficiency\n    for patches, _ in tqdm(test_loader, desc=\"Predicting\"):\n        patches = patches.to(device)\n        \n        # Get model outputs\n        outputs = model(patches)\n        \n        # Move predictions to CPU and append to our list\n        all_preds.extend(outputs.cpu().numpy())\n\n# Convert list of predictions to a NumPy array\npredictions_np = np.array(all_preds)\nprint(f\"Predictions generated. Shape: {predictions_np.shape}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:40:10.789453Z","iopub.execute_input":"2025-08-11T13:40:10.789882Z","iopub.status.idle":"2025-08-11T13:40:14.794877Z","shell.execute_reply.started":"2025-08-11T13:40:10.789846Z","shell.execute_reply":"2025-08-11T13:40:14.794143Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Creating the Submission File","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\n# --- Configuration ---\nSUBMISSION_PATH = \"submission.csv\"\nSAMPLE_SUBMISSION_PATH = \"/kaggle/working/submission.csv\" \nN_CELL_TYPES = 35\n\n# --- Assume these variables exist from your prediction loop ---\n# predictions_np: A NumPy array of shape (num_test_spots, 35) containing your model's predictions.\n# test_loader: The DataLoader used for the test set, with shuffle=False.\n\n# Let's create dummy variables for demonstration if they don't exist yet\nif 'predictions_np' not in locals():\n    print(\"Creating dummy `predictions_np` for demonstration.\")\n    # This assumes a test_df DataFrame is available to get the number of spots\n    if 'test_df' in locals():\n        num_test_spots = len(test_df)\n    else: # Fallback if test_df doesn't exist\n        print(\"Warning: `test_df` not found, creating predictions for 1000 spots as an example.\")\n        num_test_spots = 1000\n    predictions_np = np.random.rand(num_test_spots, N_CELL_TYPES)\n\n# --- 4. Create the Submission File (The Final, Correct Way) ---\nprint(\"\\n--- Creating submission.csv file using sample_submission.csv as a template ---\")\n\ntry:\n    # Step 4.1: Load the sample submission file\n    sample_df = pd.read_csv(SAMPLE_SUBMISSION_PATH)\n    print(\"Sample submission file loaded successfully.\")\n    print(\"Columns found in sample submission:\", sample_df.columns.tolist())\n    \n    # Step 4.2: Verify that the number of predictions matches the number of rows\n    if len(sample_df) != len(predictions_np):\n        raise ValueError(f\"CRITICAL ERROR: Mismatch in prediction rows. \"\n                         f\"Model predicted for {len(predictions_np)} spots, but sample submission has {len(sample_df)} rows.\")\n\n    # Step 4.3: Get the names of the target columns from the sample file\n    # --- THIS IS THE CORRECTED LINE ---\n    # We now look for the column named 'ID' (all caps) to exclude it.\n    target_cols = [col for col in sample_df.columns if col != 'ID']\n    \n    if len(target_cols) == 0:\n        raise KeyError(\"Could not find any target columns in the sample submission file. \"\n                       \"Check that the ID column is actually named 'ID'.\")\n        \n    if len(target_cols) != N_CELL_TYPES:\n         raise ValueError(f\"CRITICAL ERROR: Mismatch in number of target columns. \"\n                         f\"Model outputs {N_CELL_TYPES}, but sample file expects {len(target_cols)}.\")\n\n    print(f\"Found ID column 'ID' and {len(target_cols)} target columns.\")\n\n    # Step 4.4: Assign your predictions to the target columns\n    # The order is preserved because shuffle=False was used in the test DataLoader.\n    sample_df[target_cols] = predictions_np\n    \n    # The 'ID' column from the sample_df is preserved perfectly.\n    submission_df = sample_df\n\n    # Step 4.5: Save the final DataFrame to a .csv file\n    submission_df.to_csv(SUBMISSION_PATH, index=False)\n\n    print(f\"\\nSubmission file created successfully at: {SUBMISSION_PATH}\")\n    print(\"Final submission file head:\")\n    print(submission_df.head())\n\nexcept FileNotFoundError:\n    print(f\"ERROR: The sample submission file was not found at '{SAMPLE_SUBMISSION_PATH}'. \"\n          \"Please check the file path.\")\nexcept KeyError:\n    print(\"ERROR: The column 'ID' was not found in the sample submission file. \"\n          \"Please double-check the exact column name for the spot identifiers in sample_submission.csv.\")\nexcept Exception as e:\n    print(f\"An unexpected error occurred: {e}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:40:14.796478Z","iopub.execute_input":"2025-08-11T13:40:14.796713Z","iopub.status.idle":"2025-08-11T13:40:14.930539Z","shell.execute_reply.started":"2025-08-11T13:40:14.796691Z","shell.execute_reply":"2025-08-11T13:40:14.929677Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!pip freeze > requitements.txt","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-11T13:40:14.931468Z","iopub.execute_input":"2025-08-11T13:40:14.931999Z","iopub.status.idle":"2025-08-11T13:40:16.469051Z","shell.execute_reply.started":"2025-08-11T13:40:14.931972Z","shell.execute_reply":"2025-08-11T13:40:16.468141Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}