{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":94147,"databundleVersionId":11390004,"sourceType":"competition"}],"dockerImageVersionId":30918,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EL Hackathon 2025: Spatial Cell-Type Prediction EDA","metadata":{"_uuid":"2b4a07e0-dabc-4f7d-8978-13deb254754d","_cell_guid":"2a5d8c54-4dc5-4e54-ae2a-e8fb7952613f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# 1. Introduction\nThis notebook provides an exploratory data analysis (EDA) of the EL Hackathon 2025 dataset, which consists of:\n\n* H&E-stained histology images (6 training slides, 1 test slide)\n\n* Spatial transcriptomics (SRT)-derived cell-type compositions (~10k spots, 35 cell types)\n\n* Goal: Predict cell-type abundances (C1–C35) for each spot (~55 µm diameter) using only H&E images","metadata":{}},{"cell_type":"code","source":"# Import libraries\nimport h5py\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm import tqdm\n\n# Set style\nplt.style.use('ggplot')\nsns.set_palette('viridis')","metadata":{"_uuid":"75eb2110-30d8-40f2-8615-5b0186bb988f","_cell_guid":"6b5004eb-6bbc-4b74-b314-6d5010ac2700","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-04-09T22:35:34.364413Z","iopub.execute_input":"2025-04-09T22:35:34.364835Z","iopub.status.idle":"2025-04-09T22:35:34.373419Z","shell.execute_reply.started":"2025-04-09T22:35:34.364802Z","shell.execute_reply":"2025-04-09T22:35:34.371812Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Data Loading & Overview\nWe first load the data from the provided HDF5 file and inspect its structure","metadata":{"_uuid":"2cd34d81-5a05-4b44-93d4-e0d7f2528016","_cell_guid":"02686b11-1820-4eb9-848d-04952108bbf7","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Load and inspect HDF5 file structure\ndata_path = \"/kaggle/input/el-hackathon-2025/elucidata_ai_challenge_data.h5\"\n\nwith h5py.File(data_path, \"r\") as f:\n    print(\"HDF5 Groups:\", list(f.keys()))\n    train_slides = list(f[\"images/Train\"].keys())\n    test_slide = list(f[\"images/Test\"].keys())\n    \nprint(\"\\nTraining Slides:\", train_slides)\nprint(\"Test Slide:\", test_slide)","metadata":{"_uuid":"9267142a-d25c-4567-83db-79da210cf3d0","_cell_guid":"23d980c0-07ea-4dd8-908c-bacd67c88663","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-04-09T22:35:36.484798Z","iopub.execute_input":"2025-04-09T22:35:36.485183Z","iopub.status.idle":"2025-04-09T22:35:36.523439Z","shell.execute_reply.started":"2025-04-09T22:35:36.485150Z","shell.execute_reply":"2025-04-09T22:35:36.522306Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"***Observations:***\n\n* 6 training slides (S_1 to S_6) and 1 test slide (S_7)\n\n* Data is stored in two groups:\n\n  - images/: Contains H&E-stained tissue images\n  - spots/: Contains spot coordinates and cell-type abundances","metadata":{}},{"cell_type":"markdown","source":"## 3. Visualizing H&E Slides with Spots\nWe visualize H&E images with overlaid spot locations to understand tissue structure","metadata":{"_uuid":"9c34d30f-8c52-4050-9681-24e0a094f3aa","_cell_guid":"c50846d1-c6ba-43ce-94d2-76a53d510a06","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def plot_slide_with_spots(slide_name, h5_file, zoom_region=None):\n    \"\"\"Plot H&E image with spot coordinates, optionally zoomed to a region\"\"\"\n    with h5py.File(h5_file, \"r\") as f:\n        image = np.array(f[f\"images/Train/{slide_name}\"])\n        spots = pd.DataFrame(np.array(f[f\"spots/Train/{slide_name}\"]))\n    \n    fig, ax = plt.subplots(1, 1 + int(zoom_region is not None), \n                         figsize=(18, 6) if zoom_region else (10, 10))\n    \n    if zoom_region:\n        x1, x2, y1, y2 = zoom_region\n        zoom_img = image[y1:y2, x1:x2]\n        zoom_spots = spots[(spots['x'].between(x1, x2)) & \n                          (spots['y'].between(y1, y2))]\n        \n        ax[0].imshow(image)\n        ax[0].set_title(f\"Full Slide {slide_name}\")\n        ax[1].imshow(zoom_img)\n        ax[1].scatter(zoom_spots[\"x\"]-x1, zoom_spots[\"y\"]-y1, \n                     s=10, c=\"red\", alpha=0.7)\n        ax[1].set_title(f\"Zoomed Region ({x1}-{x2}, {y1}-{y2})\")\n    else:\n        ax.imshow(image)\n        ax.scatter(spots[\"x\"], spots[\"y\"], s=5, c=\"red\", alpha=0.5)\n        ax.set_title(f\"Slide {slide_name} | Spots: {len(spots)}\")\n    \n    for a in fig.axes:\n        a.axis('off')\n    plt.tight_layout()\n    plt.show()\nplot_slide_with_spots(train_slides[0], data_path, \n                     zoom_region=(1000, 1500, 1000, 1500))","metadata":{"_uuid":"9719a26d-9781-41f5-b5ae-4e21a3cfcac3","_cell_guid":"0e044f67-d1a8-4bc1-829b-754f704461c1","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-04-09T22:35:39.330822Z","iopub.execute_input":"2025-04-09T22:35:39.331125Z","iopub.status.idle":"2025-04-09T22:35:41.018644Z","shell.execute_reply.started":"2025-04-09T22:35:39.331101Z","shell.execute_reply":"2025-04-09T22:35:41.017643Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"***Observations:***\n\n* Spots are uniformly distributed across tissue regions\n\n* Some slides show higher spot density \n\n* Test slide (S_7) has a similar distribution","metadata":{}},{"cell_type":"markdown","source":"## 4. Analyzing Cell-type Composition\nWe analyze the distribution of cell-type abundances (C1–C35) across spots","metadata":{"_uuid":"d698bb67-c98c-48d3-a440-a3e8935ac45b","_cell_guid":"6febc48b-0f14-442e-bc49-ab955528c509","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Load cell-type data for all slides\ncell_data = []\nfor slide in train_slides:\n    with h5py.File(data_path, \"r\") as f:\n        spots = pd.DataFrame(np.array(f[f\"spots/Train/{slide}\"]))\n        cell_data.append(spots.iloc[:, 2:37])  # Columns C1-C35\n\ncell_types = [f\"C{i}\" for i in range(1, 36)]\nall_cell_data = pd.concat(cell_data)\n\n# Plot distribution of mean abundances\nplt.figure(figsize=(15, 6))\nmean_abundance = all_cell_data.mean().sort_values(ascending=False)\nsns.barplot(x=mean_abundance.index, y=mean_abundance.values)\nplt.xticks(rotation=90)\nplt.title(\"Mean Cell-Type Abundance Across All Slides\")\nplt.ylabel(\"Mean Abundance\")\nplt.show()","metadata":{"_uuid":"824d0afb-6a89-4d29-b71f-dd17cac825f1","_cell_guid":"17febf44-1021-4883-a8f6-be7420f9c8b3","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-04-09T22:35:42.661194Z","iopub.execute_input":"2025-04-09T22:35:42.661521Z","iopub.status.idle":"2025-04-09T22:35:43.141086Z","shell.execute_reply.started":"2025-04-09T22:35:42.661494Z","shell.execute_reply":"2025-04-09T22:35:43.139911Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"***Observations:***\n\n* Some cell types (e.g., C16, C1, C5, C3) are highly abundant\n\n* Others (e.g., C13, C8, C28...) are rare\n\n* *Class imbalance exists = Consider weighted loss in modeling*","metadata":{}},{"cell_type":"markdown","source":"## 5. Spatial Distribution of Cell Types","metadata":{"_uuid":"04abf881-4a79-46ba-9b5f-62c883353e46","_cell_guid":"a6e3d098-5d32-449e-b182-37a86d1573a0","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Plot spatial distribution of top 5 cell types\ntop_cell_types = mean_abundance.index[:5]\n\nwith h5py.File(data_path, \"r\") as f:\n    image = np.array(f[\"images/Train/S_1\"])\n    spots = pd.DataFrame(np.array(f[\"spots/Train/S_1\"]))\n\nfig, axes = plt.subplots(2, 3, figsize=(18, 12))\naxes = axes.ravel()\n\nfor i, cell_type in enumerate(top_cell_types):\n    sc = axes[i].scatter(\n        spots[\"x\"], \n        spots[\"y\"], \n        c=spots[cell_type], \n        cmap=\"viridis\", \n        s=10,\n        alpha=0.7\n    )\n    axes[i].set_title(f\"{cell_type} Distribution\")\n    plt.colorbar(sc, ax=axes[i])\n    axes[i].axis('off')\n\n# Remove empty subplot\nfig.delaxes(axes[-1])\nplt.tight_layout()\nplt.show()","metadata":{"_uuid":"0cbc04e4-057f-4f6d-b22b-34ddc106c7bd","_cell_guid":"4028a0e6-5916-4e38-ae0f-aa006c72110e","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-04-09T22:35:43.142301Z","iopub.execute_input":"2025-04-09T22:35:43.142551Z","iopub.status.idle":"2025-04-09T22:35:44.764264Z","shell.execute_reply.started":"2025-04-09T22:35:43.142526Z","shell.execute_reply":"2025-04-09T22:35:44.762796Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"***Observations:***\n\n* Some cell types (e.g. C16) form clusters\n\n* Others (e.g., C4) are diffusely distributed\n\n* Spatial patterns suggest local correlations = Useful for graph neural networks","metadata":{}},{"cell_type":"markdown","source":"## 6. Cell-Type Correlations","metadata":{"_uuid":"1d30248d-6739-4a03-8f4e-62e6a066ac32","_cell_guid":"266c0369-a89f-4529-a272-8832f8a608ff","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Compute and plot correlation matrix\ncorr_matrix = all_cell_data.corr()\n\nplt.figure(figsize=(15, 12))\nsns.heatmap(\n    corr_matrix, \n    cmap=\"coolwarm\", \n    vmin=-0.5, \n    vmax=0.5,\n    center=0,\n    square=True,\n    annot=False,\n    cbar_kws={\"shrink\": 0.8}\n)\nplt.title(\"Cell-Type Correlation Matrix Across All Slides\")\nplt.show()","metadata":{"_uuid":"1e339a5e-b6df-46a8-915b-149a49d7a4e5","_cell_guid":"ff2fa3b9-4ae3-49f7-9aae-55de562bd289","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-04-09T22:36:53.292758Z","iopub.execute_input":"2025-04-09T22:36:53.293130Z","iopub.status.idle":"2025-04-09T22:36:53.831849Z","shell.execute_reply.started":"2025-04-09T22:36:53.293101Z","shell.execute_reply":"2025-04-09T22:36:53.830735Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"***Observations:***\n\n* Some cell types (e.g., C1 & C35) are positively correlated\n\n* Others (e.g., C1 & C33) are negatively correlated.\n\n* Multi-task learning may help leverage these relationships.","metadata":{}},{"cell_type":"markdown","source":"## 7. Key Takeaways & Discussion","metadata":{"_uuid":"82a1842a-f8d5-483d-aec6-9cb74204eaee","_cell_guid":"1b3d8ab1-72a9-4672-8e3e-a835d432651b","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"###  Key Observations:\n1. **Data Structure**: \n   - 6 training slides with ~2000 spots each\n   - 35 cell types with varying abundances\n\n2. **Cell-Type Distribution**:\n   - Significant class imbalance (some types very rare)\n   - Spatial clustering evident for certain types\n\n3. **Correlations**:\n   - Both positive and negative correlations exist between cell types\n\n###  Discussion Points:\n- How should we handle rare cell types?\n- What spatial modeling approaches work best?\n- Should we use domain-specific pretrained models?","metadata":{"_uuid":"5e3acda2-7ac1-4ed9-94bf-64c8e7f41c19","_cell_guid":"7a43eab5-dd18-4d04-9637-e3a52341c519","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}