{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":94147,"databundleVersionId":11390004,"sourceType":"competition"}],"dockerImageVersionId":30918,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# AI Challenge - Prediction of Spatial Cellular Organization from Histology Images\n\n## Problem Statement\nMap the spatial distribution of biological cell types in a tissue slice from routine Histology HE slides.\n\n## Introduction & Background\nHistology is a century-old technique critical for clinical diagnosis. However, its interpretation is subjective, time-consuming, and limited to coarse-grained features. In contrast, genomic deep sequencing and spatial transcriptomics provide rich molecular details but are cost-prohibitive and technically demanding.\n\nThis challenge invites participants to bridge this gap by predicting cell-type composition directly from HE images, leveraging machine learning to infer spatial patterns of biological signals from routine data.\n\n## Core Challenge\nDesign a computational pipeline to predict the spatially resolved cell-type composition of a tissue sample. For each spot (~55 μm in diameter), the goal is to predict the abundances of **35 cell types** using image data alone.\n\n## Submission Requirements\n- Notebook submissions with run-time ≤ 9 hours (CPU or GPU).\n- No external data – only the provided training set and public models/libraries.\n- The submission file must be named `submission.csv` and follow the format:\n  `ID,C1,C2,...,C35`\n  where C1-C35 are the predicted cell-type abundances on the test slide 'S_7'.\n\n## Evaluation\nSubmissions will be scored by the **Spearman correlation** between predicted and ground truth cell-type abundances.","metadata":{}},{"cell_type":"markdown","source":"# 1. Data Loading and Visualization\n\nBelow, we load the HE images (both Train and Test) from the HDF5 file and overlay the spot coordinates on the images.","metadata":{}},{"cell_type":"code","source":"import h5py\nimport numpy as np\nimport matplotlib.pyplot as plt","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-12T18:51:32.899385Z","iopub.execute_input":"2025-03-12T18:51:32.899885Z","iopub.status.idle":"2025-03-12T18:51:32.942053Z","shell.execute_reply.started":"2025-03-12T18:51:32.899843Z","shell.execute_reply":"2025-03-12T18:51:32.940552Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Visualize Training slides with spot overlays\nwith h5py.File(\"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\", \"r\") as h5file:\n    train_images = h5file[\"images/Train\"]\n    train_spots = h5file[\"spots/Train\"]\n\n    num_train_slides = len(train_images)\n    fig, ax = plt.subplots(1, num_train_slides, figsize=(14, 3))\n    for i, slide_name in enumerate(train_images.keys()):\n        image = np.array(train_images[slide_name])\n        spots = np.array(train_spots[slide_name])\n        x, y = spots[\"x\"], spots[\"y\"]\n\n        ax[i].imshow(image, aspect=\"auto\")\n        ax[i].scatter(x, y, color=\"red\", s=1, alpha=0.4)  # Overlay spot locations\n        ax[i].set_title(slide_name)\n        ax[i].axis('off')\n\n    plt.tight_layout()\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-12T18:51:33.929315Z","iopub.execute_input":"2025-03-12T18:51:33.929853Z","iopub.status.idle":"2025-03-12T18:51:40.766615Z","shell.execute_reply.started":"2025-03-12T18:51:33.929816Z","shell.execute_reply":"2025-03-12T18:51:40.764313Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Visualize Test slide ('S_7')\nwith h5py.File(\"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\", \"r\") as h5file:\n    test_images = h5file[\"images/Test\"]\n    test_spots = h5file[\"spots/Test\"]\n    \n    sample = 'S_7'\n    image = np.array(test_images[sample])\n    spots = np.array(test_spots[sample])\n    x, y = spots[\"x\"], spots[\"y\"]\n\n    plt.figure(figsize=(6,6))\n    plt.imshow(image, aspect=\"auto\")\n    plt.scatter(x, y, color=\"red\", s=1, alpha=0.4)\n    plt.axis('off')\n    plt.title(sample)\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-12T18:51:50.097168Z","iopub.execute_input":"2025-03-12T18:51:50.097527Z","iopub.status.idle":"2025-03-12T18:51:51.361729Z","shell.execute_reply.started":"2025-03-12T18:51:50.097501Z","shell.execute_reply":"2025-03-12T18:51:51.360502Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. Loading and Displaying Spots Tables\n\nThe following cells load and display the (x, y) coordinates and cell type annotations for Train slides and the spot coordinates for the Test slide.  ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\n# Load and display (x,y) spot locations and cell type annotation table for Train slides\nwith h5py.File(\"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\", \"r\") as f:\n    train_spots = f[\"spots/Train\"]\n    \n    # Dictionary to store DataFrames for each slide\n    train_spot_tables = {}\n    \n    for slide_name in train_spots.keys():\n        # Load dataset as NumPy structured array\n        spot_array = np.array(train_spots[slide_name])\n        \n        # Convert to DataFrame\n        df = pd.DataFrame(spot_array)\n        \n        # Store in dictionary\n        train_spot_tables[slide_name] = df\n\n# Example: Display the spots table for slide 'S_1'\ntrain_spot_tables['S_1']","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-12T18:52:04.104820Z","iopub.execute_input":"2025-03-12T18:52:04.105200Z","iopub.status.idle":"2025-03-12T18:52:04.220240Z","shell.execute_reply.started":"2025-03-12T18:52:04.105168Z","shell.execute_reply":"2025-03-12T18:52:04.219083Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Display spot table for Test slide (only the spot coordinates on 2D array)\nwith h5py.File(\"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\", \"r\") as f:\n    test_spots = f[\"spots/Test\"]\n    spot_array = np.array(test_spots['S_7'])\n    test_spot_table = pd.DataFrame(spot_array)\n    \n# Show the test spots coordinates for slide 'S_7'\ntest_spot_table","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-12T18:52:05.015224Z","iopub.execute_input":"2025-03-12T18:52:05.015593Z","iopub.status.idle":"2025-03-12T18:52:05.036176Z","shell.execute_reply.started":"2025-03-12T18:52:05.015560Z","shell.execute_reply":"2025-03-12T18:52:05.034807Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. Creating a Random Submission\n\nIn this section, we generate random predictions for the 35 cell types.  \nThe predictions are random floats between 0 and 2 (without any normalization) for each spot in the Test slide.  \nThe order of spots is preserved as in the test spots table.","metadata":{}},{"cell_type":"code","source":"# Create a random submission\n# (predictions of cell type abundances for 35 classes across the Test slide spots;\n# spot order should be same as in the 'Test' spots table)\n\n# Use the cell type columns from the train spots table; assuming first two columns are (x, y)\ncell_type_columns = train_spot_tables['S_1'].columns[2:].values  # Expecting 35 cell types here\nindices = test_spot_table.index.values  # All spots on the Test slide\n\n# Create a 2D array of random floats between 0 and 2 for each spot and cell type\nprediction_matrix = 2 * np.random.rand(len(indices), len(cell_type_columns))\npredicted_labels = pd.DataFrame(prediction_matrix, columns=cell_type_columns, index=indices)\n\npredicted_labels.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-12T18:52:06.144406Z","iopub.execute_input":"2025-03-12T18:52:06.144850Z","iopub.status.idle":"2025-03-12T18:52:06.169988Z","shell.execute_reply.started":"2025-03-12T18:52:06.144815Z","shell.execute_reply":"2025-03-12T18:52:06.168831Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. Submission File Generation\n\nFinally, we generate the submission file in the required format.  \nEach row corresponds to a test spot with its identifier (constructed here as the index) followed by 35 predictions.","metadata":{}},{"cell_type":"code","source":"# Prepare submission DataFrame: spot_id column and then predictions for each cell type\nsubmission_df = predicted_labels.copy()\nsubmission_df.insert(0, 'ID', submission_df.index)\n\n# Save the submission file as submission.csv\nsubmission_df.to_csv(\"./submission.csv\", index=False)\nprint(\"Submission file 'submission.csv' created!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-12T18:52:07.247122Z","iopub.execute_input":"2025-03-12T18:52:07.247484Z","iopub.status.idle":"2025-03-12T18:52:07.383187Z","shell.execute_reply.started":"2025-03-12T18:52:07.247457Z","shell.execute_reply":"2025-03-12T18:52:07.382004Z"}},"outputs":[],"execution_count":null}]}