{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":112509,"databundleVersionId":14254895,"sourceType":"competition"}],"dockerImageVersionId":31192,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🌾 CSIRO Image2Biomass: Comprehensive EDA & Image Explorer\n\n**Competition:** [CSIRO Image2Biomass Prediction](https://www.kaggle.com/competitions/csiro-biomass)\n\n---\n\n## 📋 **Notebook Overview**\n\nThis notebook provides a **complete exploratory data analysis** for the CSIRO Image2Biomass competition, with a special focus on understanding the relationship between images and their corresponding biomass measurements.\n\n### **What You'll Get:**\n\n1. **📊 Data Quality Assessment**\n   - Missing value analysis\n   - Outlier detection (IQR & Z-score methods)\n   - Physical constraint validation\n\n2. **📈 Statistical Analysis**\n   - Target variable distributions\n   - Correlation patterns\n   - Key statistics and metrics\n\n3. **🗺️ Metadata Exploration**\n   - Geographic distribution (States)\n   - Temporal patterns (Seasons, Years)\n   - Species composition\n   - NDVI and Height relationships\n\n4. **🖼️ Interactive Image Viewer** ⭐\n   - Sort images by any metric\n   - Filter by State, Season, Species\n   - View detailed information\n   - Compare high/low biomass samples\n\n5. **🔍 Insights for Modeling**\n   - Feature importance indicators\n   - Data imbalances\n   - Potential challenges\n\n---\n\n### **Key Features:**\n\n- ✅ Clean, modular, reusable code\n- ✅ Comprehensive visualizations\n- ✅ Interactive image exploration\n- ✅ Physical constraint checking\n- ✅ Easy to adapt and extend\n\n---\n\n### **Competition Context:**\n\n**Goal:** Predict 5 biomass components from pasture images:\n- `Dry_Green_g` (10% weight)\n- `Dry_Dead_g` (10% weight)\n- `Dry_Clover_g` (10% weight)\n- `GDM_g` (20% weight)\n- `Dry_Total_g` (50% weight) ⭐\n\n**Evaluation:** Weighted R² score\n\n---\n\n**Let's dive in! 🚀**","metadata":{"_uuid":"4a21c458-1071-4412-9d96-af5f4522d7c6","_cell_guid":"ebf26341-7142-4367-8a21-112716b82eab","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## 1️⃣ Setup & Configuration","metadata":{"_uuid":"ef8d590b-dc06-4955-84ec-7d58a297ec09","_cell_guid":"28969eb4-d6c3-44d5-97c6-561333cc8d65","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Import libraries\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom pathlib import Path\nfrom PIL import Image\nfrom scipy import stats\nimport warnings\n\nwarnings.filterwarnings('ignore')\n\n# Set plotting style\nplt.style.use('seaborn-v0_8-darkgrid')\nsns.set_palette(\"husl\")\n\n# Display settings\npd.set_option('display.max_columns', 100)\npd.set_option('display.max_rows', 100)\npd.set_option('display.width', 1000)\npd.set_option('display.float_format', '{:.2f}'.format)\n\nprint(\"✅ Libraries imported successfully\")","metadata":{"_uuid":"066fc9f3-9553-4b4f-aa81-2bd8b28c3fa6","_cell_guid":"276341ce-823b-4b5d-bc8f-896fe6567958","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:49.213331Z","iopub.execute_input":"2025-11-11T12:31:49.213578Z","iopub.status.idle":"2025-11-11T12:31:53.668689Z","shell.execute_reply.started":"2025-11-11T12:31:49.213556Z","shell.execute_reply":"2025-11-11T12:31:53.667528Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Configuration\nCSV_PATH = \"/kaggle/input/csiro-biomass/train.csv\"\nIMAGE_DIR = \"/kaggle/input/csiro-biomass\"\n\n# Target columns\nTARGET_COLS = ['Dry_Green_g', 'Dry_Dead_g', 'Dry_Clover_g', 'GDM_g', 'Dry_Total_g']\n\n# Competition weights\nWEIGHTS = {\n    'Dry_Green_g': 0.1,\n    'Dry_Dead_g': 0.1,\n    'Dry_Clover_g': 0.1,\n    'GDM_g': 0.2,\n    'Dry_Total_g': 0.5\n}\n\nprint(\"✅ Configuration set\")","metadata":{"_uuid":"092d9421-039d-49e3-9d50-70223bab371d","_cell_guid":"6f87bf1c-64f0-4811-94ae-fb997edc38f1","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:53.671149Z","iopub.execute_input":"2025-11-11T12:31:53.671601Z","iopub.status.idle":"2025-11-11T12:31:53.680910Z","shell.execute_reply.started":"2025-11-11T12:31:53.671575Z","shell.execute_reply":"2025-11-11T12:31:53.679613Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2️⃣ Data Loading & Preprocessing\n\n### What we'll do:\n1. Load the CSV data (long format: 5 rows per image)\n2. Convert to wide format (1 row per image)\n3. Add temporal features (season, year, month)\n4. Add derived features (ratios, physical constraints)","metadata":{"_uuid":"da319783-8749-4cd4-91eb-ebeeac477f0a","_cell_guid":"ab9d52ef-6932-4d37-bf76-d328a85cd258","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def load_and_preprocess_data(csv_path: str) -> tuple[pd.DataFrame, pd.DataFrame]:\n    \"\"\"\n    Load and preprocess biomass data\n    \n    Returns:\n        Tuple of (long_format_df, wide_format_df)\n    \"\"\"\n    # Load data\n    print(\"📂 Loading data...\")\n    df_long = pd.read_csv(csv_path)\n    print(f\"✅ Loaded {len(df_long):,} rows × {len(df_long.columns)} columns\")\n    \n    # Extract image ID\n    df_long['image_id'] = df_long['sample_id'].str.split('__').str[0]\n    \n    # Convert to wide format\n    print(\"\\n🔄 Converting to wide format...\")\n    meta_cols = ['image_path', 'Sampling_Date', 'State', 'Species', \n                 'Pre_GSHH_NDVI', 'Height_Ave_cm']\n    \n    df_wide = df_long.pivot_table(\n        index=['image_id'] + meta_cols,\n        columns='target_name',\n        values='target',\n        aggfunc='first'\n    ).reset_index()\n    \n    n_images = len(df_wide)\n    print(f\"✅ Wide format: {n_images:,} images\")\n    \n    # Add temporal features\n    print(\"\\n📅 Adding temporal features...\")\n    df_wide['date'] = pd.to_datetime(df_wide['Sampling_Date'], format='%Y/%m/%d')\n    df_wide['year'] = df_wide['date'].dt.year\n    df_wide['month'] = df_wide['date'].dt.month\n    df_wide['day'] = df_wide['date'].dt.day\n    df_wide['dayofweek'] = df_wide['date'].dt.dayofweek\n    df_wide['dayofyear'] = df_wide['date'].dt.dayofyear\n    \n    # Add season (Southern Hemisphere)\n    def get_season(month: int) -> str:\n        if month in [9, 10, 11]:\n            return \"Spring\"\n        elif month in [12, 1, 2]:\n            return \"Summer\"\n        elif month in [3, 4, 5]:\n            return \"Autumn\"\n        else:\n            return \"Winter\"\n    \n    df_wide['season'] = df_wide['month'].apply(get_season)\n    \n    # Add derived features\n    print(\"🧮 Adding derived features...\")\n    \n    # Component ratios\n    total = df_wide['Dry_Total_g'] + 1e-8\n    df_wide['green_ratio'] = df_wide['Dry_Green_g'] / total\n    df_wide['dead_ratio'] = df_wide['Dry_Dead_g'] / total\n    df_wide['clover_ratio'] = df_wide['Dry_Clover_g'] / total\n    \n    # Physical constraint check\n    df_wide['sum_components'] = (\n        df_wide['Dry_Green_g'] + \n        df_wide['Dry_Dead_g'] + \n        df_wide['Dry_Clover_g']\n    )\n    df_wide['constraint_diff'] = df_wide['Dry_Total_g'] - df_wide['sum_components']\n    df_wide['constraint_error'] = np.abs(df_wide['constraint_diff'])\n    \n    print(\"✅ Preprocessing complete!\\n\")\n    \n    return df_long, df_wide\n\n# Execute\ndf_long, df = load_and_preprocess_data(CSV_PATH)","metadata":{"_uuid":"9a9819e5-2b9e-4ab0-88ed-b9b96a955172","_cell_guid":"d0504dcb-08c7-4982-b28f-844b4a3cba8a","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:53.682332Z","iopub.execute_input":"2025-11-11T12:31:53.682701Z","iopub.status.idle":"2025-11-11T12:31:53.890441Z","shell.execute_reply.started":"2025-11-11T12:31:53.682667Z","shell.execute_reply":"2025-11-11T12:31:53.889231Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Quick peek at the data\nprint(\"=\"*80)\nprint(\"📊 DATA OVERVIEW\")\nprint(\"=\"*80)\nprint(f\"\\nShape: {df.shape[0]:,} rows × {df.shape[1]} columns\")\nprint(f\"Memory: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB\")\nprint(f\"\\nDate range: {df['date'].min().date()} to {df['date'].max().date()}\")\nprint(f\"Number of states: {df['State'].nunique()}\")\nprint(f\"Number of species combinations: {df['Species'].nunique()}\")","metadata":{"_uuid":"1a64f672-8320-4f84-a29d-d07af3badb35","_cell_guid":"a3245299-14a0-4d31-853d-08e65c683a2f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:53.891555Z","iopub.execute_input":"2025-11-11T12:31:53.891930Z","iopub.status.idle":"2025-11-11T12:31:53.907362Z","shell.execute_reply.started":"2025-11-11T12:31:53.891899Z","shell.execute_reply":"2025-11-11T12:31:53.905928Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Display first few rows\ndf.head()","metadata":{"_uuid":"1e36aa1d-4d78-43bd-8a2f-ea865ec7f6db","_cell_guid":"071fbeee-9c01-4ca1-8cf7-49711cd8285e","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:53.908433Z","iopub.execute_input":"2025-11-11T12:31:53.908801Z","iopub.status.idle":"2025-11-11T12:31:53.959417Z","shell.execute_reply.started":"2025-11-11T12:31:53.908755Z","shell.execute_reply":"2025-11-11T12:31:53.958067Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3️⃣ Data Quality Assessment\n\nLet's check for:\n- Missing values\n- Data types\n- Basic statistics","metadata":{"_uuid":"3f104b42-26b3-4170-baa7-77df84aa3acb","_cell_guid":"3fee1d53-589a-4ca6-9435-93e050232b8f","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def analyze_data_quality(df: pd.DataFrame) -> None:\n    \"\"\"Comprehensive data quality analysis\"\"\"\n    \n    print(\"\\n\" + \"=\"*80)\n    print(\"🔍 DATA QUALITY ANALYSIS\")\n    print(\"=\"*80)\n    \n    # Missing values\n    print(\"\\n【Missing Values】\")\n    missing = pd.DataFrame({\n        'Missing_Count': df.isnull().sum(),\n        'Missing_Pct': (df.isnull().sum() / len(df) * 100).round(2)\n    })\n    missing = missing[missing['Missing_Count'] > 0].sort_values('Missing_Count', ascending=False)\n    \n    if len(missing) == 0:\n        print(\"✅ No missing values!\")\n    else:\n        print(missing)\n    \n    # Data types\n    print(\"\\n【Data Types】\")\n    print(df.dtypes.value_counts())\n    \n    # Duplicates\n    n_duplicates = df.duplicated(subset='image_id').sum()\n    print(f\"\\n【Duplicates】\")\n    print(f\"Duplicate image IDs: {n_duplicates}\")\n\nanalyze_data_quality(df)","metadata":{"_uuid":"44f6a54e-bde4-4fe6-82fd-6f3b7df26276","_cell_guid":"b6ac5d35-4857-430f-a8fe-653de8e822b2","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:53.960767Z","iopub.execute_input":"2025-11-11T12:31:53.961159Z","iopub.status.idle":"2025-11-11T12:31:53.979296Z","shell.execute_reply.started":"2025-11-11T12:31:53.961129Z","shell.execute_reply":"2025-11-11T12:31:53.977659Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4️⃣ Target Variable Analysis\n\n### Key Questions:\n- What are the distributions of each target?\n- Are they normally distributed?\n- What are the correlations between targets?\n- Are there any outliers?","metadata":{"_uuid":"3c7b9f43-7315-4584-81c2-518e2fd61e4e","_cell_guid":"4c78ce04-f119-41ac-babc-56bbc9e28619","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def analyze_target_statistics(df: pd.DataFrame, target_cols: list) -> pd.DataFrame:\n    \"\"\"Compute comprehensive target statistics\"\"\"\n    \n    print(\"\\n\" + \"=\"*80)\n    print(\"📈 TARGET VARIABLE STATISTICS\")\n    print(\"=\"*80)\n    \n    stats = df[target_cols].describe().T\n    \n    # Additional metrics\n    stats['cv'] = stats['std'] / stats['mean']  # Coefficient of variation\n    stats['skewness'] = df[target_cols].skew()\n    stats['kurtosis'] = df[target_cols].kurtosis()\n    stats['comp_weight'] = [WEIGHTS[col] for col in target_cols]\n    \n    return stats.round(3)\n\ntarget_stats = analyze_target_statistics(df, TARGET_COLS)\ntarget_stats","metadata":{"_uuid":"4dd915f6-a9fc-4f8c-8f65-b55c6bd35e1e","_cell_guid":"cf6f44b9-ba58-49b6-8c40-6b911f5e6c30","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:53.982968Z","iopub.execute_input":"2025-11-11T12:31:53.983376Z","iopub.status.idle":"2025-11-11T12:31:54.044033Z","shell.execute_reply.started":"2025-11-11T12:31:53.983350Z","shell.execute_reply":"2025-11-11T12:31:54.042528Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Interpretation\nprint(\"\\n💡 Key Observations:\")\nprint(f\"• Dry_Total_g has the highest mean ({target_stats.loc['Dry_Total_g', 'mean']:.1f}g) \" +\n      \"and is the primary target (50% weight)\")\nprint(f\"• Dry_Clover_g has high variability (CV: {target_stats.loc['Dry_Clover_g', 'cv']:.2f})\")\nprint(f\"• All targets show positive skewness (right-tailed distributions)\")","metadata":{"_uuid":"cad3fd05-cfb1-42cd-a343-459b5d551bc6","_cell_guid":"a9d3c64c-6302-4a56-b787-4a4055d2f8b3","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:54.045217Z","iopub.execute_input":"2025-11-11T12:31:54.045568Z","iopub.status.idle":"2025-11-11T12:31:54.054215Z","shell.execute_reply.started":"2025-11-11T12:31:54.045534Z","shell.execute_reply":"2025-11-11T12:31:54.052913Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5️⃣ Target Distribution Visualization\n\nLet's visualize the distributions with:\n- Histograms (with mean/median lines)\n- Box plots (for outlier detection)\n- Q-Q plots (for normality check)","metadata":{"_uuid":"6d68bf3f-5ab8-4446-86c7-53ebe753dd91","_cell_guid":"e8167022-2fe0-4d6f-ae9c-be4e02df1d30","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def plot_target_distributions(df: pd.DataFrame, target_cols: list) -> None:\n    \"\"\"Plot comprehensive target distributions\"\"\"\n    \n    fig, axes = plt.subplots(3, 5, figsize=(20, 12))\n    \n    for i, target in enumerate(target_cols):\n        data = df[target].dropna()\n        \n        # Histogram\n        axes[0, i].hist(data, bins=50, edgecolor='black', alpha=0.7, color='steelblue')\n        axes[0, i].axvline(data.mean(), color='red', linestyle='--', \n                          linewidth=2, label=f'Mean: {data.mean():.1f}g')\n        axes[0, i].axvline(data.median(), color='green', linestyle='--', \n                          linewidth=2, label=f'Median: {data.median():.1f}g')\n        axes[0, i].set_title(f'{target}\\nDistribution', fontsize=11, fontweight='bold')\n        axes[0, i].set_xlabel('Value (g)')\n        axes[0, i].set_ylabel('Frequency')\n        axes[0, i].legend(fontsize=8)\n        axes[0, i].grid(True, alpha=0.3)\n        \n        # Box plot\n        bp = axes[1, i].boxplot(data, vert=True, patch_artist=True)\n        bp['boxes'][0].set_facecolor('lightblue')\n        axes[1, i].set_title(f'{target}\\nBox Plot', fontsize=11, fontweight='bold')\n        axes[1, i].set_ylabel('Value (g)')\n        axes[1, i].grid(True, alpha=0.3)\n        \n        # Q-Q plot\n        stats.probplot(data, dist=\"norm\", plot=axes[2, i])\n        axes[2, i].set_title(f'{target}\\nQ-Q Plot', fontsize=11, fontweight='bold')\n        axes[2, i].grid(True, alpha=0.3)\n    \n    plt.tight_layout()\n    plt.show()\n\nplot_target_distributions(df, TARGET_COLS)","metadata":{"_uuid":"d13707fd-7302-485f-9d6e-7ac75aa8a82d","_cell_guid":"039fed09-ae35-4c35-84cf-1c20662b036d","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:54.055494Z","iopub.execute_input":"2025-11-11T12:31:54.056080Z","iopub.status.idle":"2025-11-11T12:31:57.641969Z","shell.execute_reply.started":"2025-11-11T12:31:54.056048Z","shell.execute_reply":"2025-11-11T12:31:57.640841Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"\\n💡 Distribution Insights:\")\nprint(\"• All targets show right-skewed distributions (more low values than high)\")\nprint(\"• Q-Q plots show deviations from normality, especially at the tails\")\nprint(\"• Consider log transformation for modeling\")\nprint(\"• Several outliers present in all targets\")","metadata":{"_uuid":"3046f5bb-a76a-4d4e-a7c4-06a1e68097a3","_cell_guid":"4244354f-285d-4025-98f5-9f209f74e513","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:57.643224Z","iopub.execute_input":"2025-11-11T12:31:57.643531Z","iopub.status.idle":"2025-11-11T12:31:57.649993Z","shell.execute_reply.started":"2025-11-11T12:31:57.643504Z","shell.execute_reply":"2025-11-11T12:31:57.648554Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6️⃣ Correlation Analysis\n\nUnderstanding relationships between targets is crucial for:\n- Multi-task learning strategies\n- Feature engineering\n- Model architecture design","metadata":{"_uuid":"467386ac-b7e8-495d-8cb8-4f7eed1013a9","_cell_guid":"77215eb1-2f78-40c7-ad7c-ce89f1090d90","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def plot_correlation_analysis(df: pd.DataFrame, target_cols: list) -> None:\n    \"\"\"Plot correlation heatmap and key insights\"\"\"\n    \n    corr = df[target_cols].corr()\n    \n    plt.figure(figsize=(10, 8))\n    sns.heatmap(corr, annot=True, fmt='.3f', cmap='coolwarm', \n               center=0, square=True, linewidths=2,\n               cbar_kws={\"shrink\": 0.8}, vmin=-1, vmax=1)\n    plt.title('Target Variable Correlations', fontsize=14, fontweight='bold', pad=20)\n    plt.tight_layout()\n    plt.show()\n    \n    # Print key correlations\n    print(\"\\n💡 Key Correlations:\")\n    \n    # Get upper triangle\n    corr_flat = corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool))\n    corr_pairs = corr_flat.stack().sort_values(ascending=False)\n    \n    print(\"\\nStrongest positive correlations:\")\n    for (var1, var2), corr_val in corr_pairs.head(3).items():\n        print(f\"  • {var1} ↔ {var2}: {corr_val:.3f}\")\n    \n    print(\"\\nWeakest correlations:\")\n    for (var1, var2), corr_val in corr_pairs.tail(3).items():\n        print(f\"  • {var1} ↔ {var2}: {corr_val:.3f}\")\n\nplot_correlation_analysis(df, TARGET_COLS)","metadata":{"_uuid":"cf5474ea-e46e-412f-ac22-752eac2a8cb2","_cell_guid":"0da00229-b1ba-4e45-a9b3-466e10d6f5f8","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:57.651054Z","iopub.execute_input":"2025-11-11T12:31:57.651491Z","iopub.status.idle":"2025-11-11T12:31:58.092447Z","shell.execute_reply.started":"2025-11-11T12:31:57.651468Z","shell.execute_reply":"2025-11-11T12:31:58.091227Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 7️⃣ Metadata Exploration\n\nLet's explore the metadata features:\n- Geographic (State)\n- Temporal (Season, Year)\n- Physical (NDVI, Height)\n- Biological (Species)","metadata":{"_uuid":"256b627d-97ca-4ad8-ae3d-403e5f5b6a54","_cell_guid":"3d4588e5-eddf-448c-af22-cc187819cbf0","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def plot_metadata_distributions(df: pd.DataFrame) -> None:\n    \"\"\"Plot metadata feature distributions\"\"\"\n    \n    fig, axes = plt.subplots(2, 3, figsize=(18, 10))\n    axes = axes.flatten()\n    \n    # State distribution\n    state_counts = df['State'].value_counts()\n    bars = axes[0].bar(state_counts.index, state_counts.values, \n                      edgecolor='black', color='skyblue', alpha=0.8)\n    axes[0].set_title('State Distribution', fontsize=12, fontweight='bold')\n    axes[0].set_xlabel('State')\n    axes[0].set_ylabel('Sample Count')\n    axes[0].grid(True, alpha=0.3, axis='y')\n    for bar, count in zip(bars, state_counts.values):\n        axes[0].text(bar.get_x() + bar.get_width()/2, count, \n                    str(count), ha='center', va='bottom', fontweight='bold')\n    \n    # Top 10 species\n    species_counts = df['Species'].value_counts().head(10)\n    axes[1].barh(range(len(species_counts)), species_counts.values, \n                edgecolor='black', color='lightgreen', alpha=0.8)\n    axes[1].set_yticks(range(len(species_counts)))\n    axes[1].set_yticklabels([s[:30] for s in species_counts.index], fontsize=9)\n    axes[1].set_title('Top 10 Species Combinations', fontsize=12, fontweight='bold')\n    axes[1].set_xlabel('Sample Count')\n    axes[1].invert_yaxis()\n    axes[1].grid(True, alpha=0.3, axis='x')\n    \n    # NDVI distribution\n    ndvi_data = df['Pre_GSHH_NDVI'].dropna()\n    axes[2].hist(ndvi_data, bins=30, edgecolor='black', alpha=0.7, color='green')\n    axes[2].axvline(ndvi_data.mean(), color='red', linestyle='--', \n                   linewidth=2, label=f'Mean: {ndvi_data.mean():.2f}')\n    axes[2].axvline(ndvi_data.median(), color='orange', linestyle='--', \n                   linewidth=2, label=f'Median: {ndvi_data.median():.2f}')\n    axes[2].set_title('NDVI Distribution', fontsize=12, fontweight='bold')\n    axes[2].set_xlabel('NDVI Value')\n    axes[2].set_ylabel('Frequency')\n    axes[2].legend()\n    axes[2].grid(True, alpha=0.3)\n    \n    # Height distribution\n    height_data = df['Height_Ave_cm'].dropna()\n    axes[3].hist(height_data, bins=30, edgecolor='black', alpha=0.7, color='orange')\n    axes[3].axvline(height_data.mean(), color='red', linestyle='--', \n                   linewidth=2, label=f'Mean: {height_data.mean():.1f}cm')\n    axes[3].axvline(height_data.median(), color='blue', linestyle='--', \n                   linewidth=2, label=f'Median: {height_data.median():.1f}cm')\n    axes[3].set_title('Height Distribution', fontsize=12, fontweight='bold')\n    axes[3].set_xlabel('Height (cm)')\n    axes[3].set_ylabel('Frequency')\n    axes[3].legend()\n    axes[3].grid(True, alpha=0.3)\n    \n    # Season distribution\n    season_counts = df['season'].value_counts()\n    colors = ['#90EE90', '#FFD700', '#FF6347', '#87CEEB']\n    wedges, texts, autotexts = axes[4].pie(\n        season_counts.values, labels=season_counts.index, \n        autopct='%1.1f%%', startangle=90, colors=colors\n    )\n    for autotext in autotexts:\n        autotext.set_color('white')\n        autotext.set_fontweight('bold')\n    axes[4].set_title('Season Distribution', fontsize=12, fontweight='bold')\n    \n    # Yearly sample counts\n    year_counts = df['year'].value_counts().sort_index()\n    axes[5].plot(year_counts.index, year_counts.values, \n                marker='o', linewidth=2, markersize=8, color='purple')\n    axes[5].set_title('Samples by Year', fontsize=12, fontweight='bold')\n    axes[5].set_xlabel('Year')\n    axes[5].set_ylabel('Sample Count')\n    axes[5].grid(True, alpha=0.3)\n    for x, y in zip(year_counts.index, year_counts.values):\n        axes[5].text(x, y, str(y), ha='center', va='bottom', fontweight='bold')\n    \n    plt.tight_layout()\n    plt.show()\n\nplot_metadata_distributions(df)","metadata":{"_uuid":"76a2f03b-de74-401a-b325-bbf946979280","_cell_guid":"d6379db5-a9fc-41ce-a223-98b61e5835d2","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:58.093767Z","iopub.execute_input":"2025-11-11T12:31:58.094166Z","iopub.status.idle":"2025-11-11T12:31:59.647568Z","shell.execute_reply.started":"2025-11-11T12:31:58.094134Z","shell.execute_reply":"2025-11-11T12:31:59.646407Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"\\n💡 Metadata Insights:\")\nprint(f\"• Data collected across {df['State'].nunique()} Australian states\")\nprint(f\"• {df['Species'].nunique()} unique species combinations\")\nprint(f\"• NDVI range: {df['Pre_GSHH_NDVI'].min():.2f} to {df['Pre_GSHH_NDVI'].max():.2f}\")\nprint(f\"• Height range: {df['Height_Ave_cm'].min():.1f} to {df['Height_Ave_cm'].max():.1f} cm\")\nprint(f\"• Data spans {df['year'].nunique()} years ({df['year'].min()}-{df['year'].max()})\")\nprint(f\"• Most samples from: {df['State'].value_counts().index[0]}\")","metadata":{"_uuid":"ae5b3946-f6a5-48b5-8fc1-a970c32e6535","_cell_guid":"b4d117f4-6e53-45bf-875a-b9abbf461800","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:59.648328Z","iopub.execute_input":"2025-11-11T12:31:59.648588Z","iopub.status.idle":"2025-11-11T12:31:59.659939Z","shell.execute_reply.started":"2025-11-11T12:31:59.648566Z","shell.execute_reply":"2025-11-11T12:31:59.658738Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 8️⃣ Target vs Metadata Relationships\n\nUnderstanding how targets vary across:\n- States\n- Seasons\n- NDVI values\n- Height measurements\n- Species","metadata":{"_uuid":"eec3d37c-9d61-402f-89f2-cc440bd89f4e","_cell_guid":"b507f04f-077e-4969-9957-38731f55adbe","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def plot_target_by_metadata(df: pd.DataFrame, target: str = 'Dry_Total_g') -> None:\n    \"\"\"Plot target relationships with metadata\"\"\"\n    \n    fig, axes = plt.subplots(2, 3, figsize=(18, 12))\n    axes = axes.flatten()\n    \n    # By State\n    df.boxplot(column=target, by='State', ax=axes[0])\n    axes[0].set_title(f'{target} by State', fontsize=12, fontweight='bold')\n    axes[0].set_xlabel('State')\n    axes[0].set_ylabel(f'{target} (g)')\n    plt.sca(axes[0])\n    plt.xticks(rotation=0)\n    \n    # By Season\n    season_order = ['Spring', 'Summer', 'Autumn', 'Winter']\n    season_data = [df[df['season'] == s][target].dropna() for s in season_order]\n    bp = axes[1].boxplot(season_data, labels=season_order, patch_artist=True)\n    colors = ['#90EE90', '#FFD700', '#FF6347', '#87CEEB']\n    for patch, color in zip(bp['boxes'], colors):\n        patch.set_facecolor(color)\n    axes[1].set_title(f'{target} by Season', fontsize=12, fontweight='bold')\n    axes[1].set_xlabel('Season')\n    axes[1].set_ylabel(f'{target} (g)')\n    axes[1].grid(True, alpha=0.3)\n    \n    # vs NDVI\n    corr_ndvi = df[[target, 'Pre_GSHH_NDVI']].corr().iloc[0, 1]\n    axes[2].scatter(df['Pre_GSHH_NDVI'], df[target], alpha=0.5, s=30, edgecolor='black')\n    axes[2].set_title(f'{target} vs NDVI (r={corr_ndvi:.3f})', fontsize=12, fontweight='bold')\n    axes[2].set_xlabel('NDVI')\n    axes[2].set_ylabel(f'{target} (g)')\n    axes[2].grid(True, alpha=0.3)\n    \n    # vs Height\n    corr_height = df[[target, 'Height_Ave_cm']].corr().iloc[0, 1]\n    axes[3].scatter(df['Height_Ave_cm'], df[target], alpha=0.5, s=30, \n                   edgecolor='black', color='orange')\n    axes[3].set_title(f'{target} vs Height (r={corr_height:.3f})', fontsize=12, fontweight='bold')\n    axes[3].set_xlabel('Height (cm)')\n    axes[3].set_ylabel(f'{target} (g)')\n    axes[3].grid(True, alpha=0.3)\n    \n    # By Top 5 Species\n    top_species = df['Species'].value_counts().head(5).index\n    df_top = df[df['Species'].isin(top_species)]\n    species_data = [df_top[df_top['Species'] == s][target].dropna() for s in top_species]\n    bp = axes[4].boxplot(species_data, labels=[s[:20] for s in top_species])\n    axes[4].set_title(f'{target} by Top 5 Species', fontsize=12, fontweight='bold')\n    axes[4].set_xlabel('Species')\n    axes[4].set_ylabel(f'{target} (g)')\n    plt.sca(axes[4])\n    plt.xticks(rotation=45, ha='right')\n    axes[4].grid(True, alpha=0.3)\n    \n    # Monthly trend\n    monthly_mean = df.groupby('month')[target].mean()\n    axes[5].plot(monthly_mean.index, monthly_mean.values, \n                marker='o', linewidth=2, markersize=8, color='green')\n    axes[5].set_title(f'{target} Monthly Average', fontsize=12, fontweight='bold')\n    axes[5].set_xlabel('Month')\n    axes[5].set_ylabel(f'{target} Average (g)')\n    axes[5].set_xticks(range(1, 13))\n    axes[5].grid(True, alpha=0.3)\n    \n    plt.tight_layout()\n    plt.show()\n\n# Plot for main targets\nfor target in ['Dry_Total_g', 'Dry_Green_g', 'GDM_g']:\n    plot_target_by_metadata(df, target)\n    print(f\"\\n💡 {target} shows variation across states, seasons, and metadata\")","metadata":{"_uuid":"3124a17d-ec2c-4afc-beb1-fef73da51645","_cell_guid":"4169c77b-cf19-4074-ae9e-def81a09bcc5","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:31:59.661053Z","iopub.execute_input":"2025-11-11T12:31:59.661390Z","iopub.status.idle":"2025-11-11T12:32:04.123363Z","shell.execute_reply.started":"2025-11-11T12:31:59.661363Z","shell.execute_reply":"2025-11-11T12:32:04.122216Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 9️⃣ Physical Constraint Validation\n\n### Important Physical Relationship:\n```\nDry_Total = Dry_Green + Dry_Dead + Dry_Clover\n```\n\nLet's check if this constraint holds in the data!","metadata":{"_uuid":"54ce44e6-3f00-49bd-890d-53698eecae50","_cell_guid":"2255a8cc-4aa8-4441-8db6-ac7f5c1b7f04","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def check_physical_constraints(df: pd.DataFrame, threshold: float = 5.0) -> None:\n    \"\"\"Check and visualize physical constraint violations\"\"\"\n    \n    violations = df[df['constraint_error'] > threshold]\n    \n    print(\"\\n\" + \"=\"*80)\n    print(\"🔍 PHYSICAL CONSTRAINT CHECK\")\n    print(\"=\"*80)\n    print(f\"\\nConstraint: Dry_Total ≈ Dry_Green + Dry_Dead + Dry_Clover\")\n    print(f\"Threshold: {threshold}g\")\n    print(f\"Violations: {len(violations):,} / {len(df):,} ({len(violations)/len(df)*100:.1f}%)\")\n    \n    print(f\"\\nConstraint Error Statistics:\")\n    print(df['constraint_error'].describe())\n    \n    if len(violations) > 0:\n        print(f\"\\n⚠️ Top 10 Violations:\")\n        top_viol = violations.nlargest(10, 'constraint_error')[[\n            'image_id', 'Dry_Total_g', 'Dry_Green_g', 'Dry_Dead_g', \n            'Dry_Clover_g', 'sum_components', 'constraint_error'\n        ]]\n        print(top_viol.to_string(index=False))\n    \n    # Visualization\n    fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n    \n    # Error distribution\n    axes[0].hist(df['constraint_error'], bins=50, edgecolor='black', alpha=0.7, color='red')\n    axes[0].axvline(threshold, color='blue', linestyle='--', linewidth=2, \n                   label=f'Threshold: {threshold}g')\n    axes[0].set_title('Physical Constraint Error Distribution', fontsize=12, fontweight='bold')\n    axes[0].set_xlabel('Error (g)')\n    axes[0].set_ylabel('Frequency')\n    axes[0].legend()\n    axes[0].grid(True, alpha=0.3)\n    \n    # Dry_Total vs sum_components\n    axes[1].scatter(df['Dry_Total_g'], df['sum_components'], alpha=0.5, s=20)\n    axes[1].plot([df['Dry_Total_g'].min(), df['Dry_Total_g'].max()],\n                [df['Dry_Total_g'].min(), df['Dry_Total_g'].max()],\n                'r--', linewidth=2, label='Perfect Match')\n    axes[1].set_title('Dry_Total vs Component Sum', fontsize=12, fontweight='bold')\n    axes[1].set_xlabel('Dry_Total (g)')\n    axes[1].set_ylabel('Green + Dead + Clover (g)')\n    axes[1].legend()\n    axes[1].grid(True, alpha=0.3)\n    \n    plt.tight_layout()\n    plt.show()\n\ncheck_physical_constraints(df)","metadata":{"_uuid":"b045d975-ae3d-41a7-b991-c2bd744c0fc0","_cell_guid":"929bcd01-7f61-4fbe-9467-13385b9f9d95","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:04.124355Z","iopub.execute_input":"2025-11-11T12:32:04.124639Z","iopub.status.idle":"2025-11-11T12:32:04.780177Z","shell.execute_reply.started":"2025-11-11T12:32:04.124612Z","shell.execute_reply":"2025-11-11T12:32:04.778479Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🔟 Outlier Detection\n\nWe'll use two methods:\n1. **IQR Method**: Q1 - 1.5×IQR to Q3 + 1.5×IQR\n2. **Z-Score Method**: |Z| > 3.0","metadata":{"_uuid":"5cb3d41d-2001-46a4-8616-9092fc38b974","_cell_guid":"bc3b9cf5-d967-45a6-b525-42f59ca86e80","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def detect_outliers(df: pd.DataFrame, target: str) -> dict:\n    \"\"\"Detect outliers using IQR and Z-score methods\"\"\"\n    \n    print(f\"\\n🔍 Outlier Detection: {target}\")\n    print(\"=\"*60)\n    \n    # IQR method\n    Q1 = df[target].quantile(0.25)\n    Q3 = df[target].quantile(0.75)\n    IQR = Q3 - Q1\n    lower_bound = Q1 - 1.5 * IQR\n    upper_bound = Q3 + 1.5 * IQR\n    \n    iqr_outliers = df[(df[target] < lower_bound) | (df[target] > upper_bound)]\n    \n    print(f\"\\n【IQR Method】\")\n    print(f\"Bounds: [{lower_bound:.2f}g, {upper_bound:.2f}g]\")\n    print(f\"Outliers: {len(iqr_outliers):,} ({len(iqr_outliers)/len(df)*100:.1f}%)\")\n    \n    # Z-score method\n    mean = df[target].mean()\n    std = df[target].std()\n    z_scores = np.abs((df[target] - mean) / std)\n    zscore_outliers = df[z_scores > 3.0]\n    \n    print(f\"\\n【Z-Score Method】(threshold=3.0)\")\n    print(f\"Mean: {mean:.2f}g, Std: {std:.2f}g\")\n    print(f\"Outliers: {len(zscore_outliers):,} ({len(zscore_outliers)/len(df)*100:.1f}%)\")\n    \n    # Visualization\n    fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n    \n    # Box plot\n    bp = axes[0].boxplot(df[target], vert=True, patch_artist=True)\n    bp['boxes'][0].set_facecolor('lightblue')\n    axes[0].axhline(upper_bound, color='red', linestyle='--', linewidth=2, label='Upper')\n    axes[0].axhline(lower_bound, color='red', linestyle='--', linewidth=2, label='Lower')\n    axes[0].set_title(f'{target} - IQR Outlier Detection', fontsize=12, fontweight='bold')\n    axes[0].set_ylabel(f'{target} (g)')\n    axes[0].legend()\n    axes[0].grid(True, alpha=0.3)\n    \n    # Z-score plot\n    axes[1].scatter(range(len(df)), z_scores, alpha=0.5, s=20)\n    axes[1].axhline(3.0, color='red', linestyle='--', linewidth=2, label='Threshold: 3.0')\n    axes[1].set_title(f'{target} - Z-Score Plot', fontsize=12, fontweight='bold')\n    axes[1].set_xlabel('Sample Index')\n    axes[1].set_ylabel('|Z-Score|')\n    axes[1].legend()\n    axes[1].grid(True, alpha=0.3)\n    \n    plt.tight_layout()\n    plt.show()\n    \n    return {\n        'iqr_outliers': iqr_outliers,\n        'zscore_outliers': zscore_outliers\n    }\n\n# Detect outliers for all targets\noutlier_results = {}\nfor target in TARGET_COLS:\n    outlier_results[target] = detect_outliers(df, target)","metadata":{"_uuid":"6f372c0c-7001-4010-aad1-9108903ce3a8","_cell_guid":"2c37dde4-24a7-4da9-a427-cae59a71bac8","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:04.781451Z","iopub.execute_input":"2025-11-11T12:32:04.781752Z","iopub.status.idle":"2025-11-11T12:32:07.113507Z","shell.execute_reply.started":"2025-11-11T12:32:04.781727Z","shell.execute_reply":"2025-11-11T12:32:07.112443Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1️⃣1️⃣ 🖼️ Interactive Image Viewer\n\n**This is the most powerful feature of this notebook!**\n\nYou can:\n- Sort images by any metric (Dry_Total, NDVI, Height, etc.)\n- Filter by State, Season, Species\n- View detailed information for each image\n- Compare high/low biomass samples visually","metadata":{"_uuid":"9f406bde-43b9-4ca1-a754-f9b5235daaf2","_cell_guid":"a50a8729-2b3a-4c74-ba69-5cac4e19475b","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def show_images_by_condition(\n    df: pd.DataFrame,\n    image_base_dir: str,\n    sort_by: str = 'Dry_Total_g',\n    ascending: bool = False,\n    n_images: int = 9,\n    filters: dict = None,\n    ncols: int = 3\n) -> None:\n    \"\"\"\n    Display images with filtering and sorting\n    \n    Args:\n        df: DataFrame with image info\n        image_base_dir: Base directory for images\n        sort_by: Column to sort by\n        ascending: Sort order\n        n_images: Number of images to display\n        filters: Dictionary of filters (e.g., {'State': 'Vic'})\n        ncols: Number of columns\n    \"\"\"\n    # Apply filters\n    df_filtered = df.copy()\n    if filters:\n        for col, value in filters.items():\n            if col in df_filtered.columns:\n                df_filtered = df_filtered[df_filtered[col] == value]\n                print(f\"✅ Filter: {col} == {value}\")\n    \n    print(f\"📊 Filtered samples: {len(df_filtered):,}\")\n    \n    if len(df_filtered) == 0:\n        print(\"❌ No samples match filters\")\n        return\n    \n    # Sort\n    df_sorted = df_filtered.sort_values(sort_by, ascending=ascending)\n    \n    # Display\n    nrows = (n_images + ncols - 1) // ncols\n    fig, axes = plt.subplots(nrows, ncols, figsize=(ncols*5, nrows*5))\n    axes = axes.flatten() if n_images > 1 else [axes]\n    \n    for idx, (i, row) in enumerate(df_sorted.head(n_images).iterrows()):\n        if idx >= len(axes):\n            break\n            \n        img_path = Path(image_base_dir) / row['image_path']\n        \n        if img_path.exists():\n            img = Image.open(img_path)\n            axes[idx].imshow(img)\n            axes[idx].axis('off')\n            \n            title = f\"ID: {row['image_id']}\\n{sort_by}: {row[sort_by]:.1f}g\\n\" + \\\n                   f\"NDVI: {row['Pre_GSHH_NDVI']:.2f} | Height: {row['Height_Ave_cm']:.1f}cm\\n\" + \\\n                   f\"{row['State']} | {row['season']}\"\n            axes[idx].set_title(title, fontsize=9)\n        else:\n            axes[idx].text(0.5, 0.5, 'Image not found', \n                         ha='center', va='center', color='red')\n            axes[idx].axis('off')\n    \n    for idx in range(n_images, len(axes)):\n        axes[idx].axis('off')\n    \n    plt.tight_layout()\n    plt.show()","metadata":{"_uuid":"937522e0-f2dc-41cc-b642-cc1c9c26b579","_cell_guid":"ea7fc89e-e069-409e-982d-ee9680f3998c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:07.114501Z","iopub.execute_input":"2025-11-11T12:32:07.114751Z","iopub.status.idle":"2025-11-11T12:32:07.129610Z","shell.execute_reply.started":"2025-11-11T12:32:07.114731Z","shell.execute_reply":"2025-11-11T12:32:07.128027Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def show_single_image_detail(\n    df: pd.DataFrame,\n    image_base_dir: str,\n    image_id: str\n) -> None:\n    \"\"\"Display single image with detailed info\"\"\"\n    \n    matching = df[df['image_id'] == image_id]\n    if len(matching) == 0:\n        print(f\"❌ Image ID '{image_id}' not found\")\n        return\n    \n    row = matching.iloc[0]\n    img_path = Path(image_base_dir) / row['image_path']\n    \n    fig = plt.figure(figsize=(16, 6))\n    gs = fig.add_gridspec(1, 2, width_ratios=[1.5, 1])\n    ax_img = fig.add_subplot(gs[0])\n    ax_info = fig.add_subplot(gs[1])\n    \n    # Image\n    if img_path.exists():\n        img = Image.open(img_path)\n        ax_img.imshow(img)\n        ax_img.axis('off')\n        ax_img.set_title(f\"Image ID: {row['image_id']}\", fontsize=14, fontweight='bold')\n    else:\n        ax_img.text(0.5, 0.5, 'Not found', ha='center', va='center', color='red')\n        ax_img.axis('off')\n    \n    # Info\n    ax_info.axis('off')\n    \n    info_text = f\"\"\"\nIMAGE INFORMATION\n{'='*50}\n\n【Metadata】\nDate: {row['Sampling_Date']}\nState: {row['State']}\nSpecies: {row['Species']}\nNDVI: {row['Pre_GSHH_NDVI']:.3f}\nHeight: {row['Height_Ave_cm']:.2f} cm\nSeason: {row['season']}\n\n【Target Values】\nDry_Total: {row['Dry_Total_g']:.2f} g (100%)\n  ├─ Dry_Green: {row['Dry_Green_g']:.2f} g ({row['green_ratio']*100:.1f}%)\n  ├─ Dry_Dead: {row['Dry_Dead_g']:.2f} g ({row['dead_ratio']*100:.1f}%)\n  └─ Dry_Clover: {row['Dry_Clover_g']:.2f} g ({row['clover_ratio']*100:.1f}%)\nGDM: {row['GDM_g']:.2f} g\n\n【Constraint Check】\nSum(G+D+C): {row['sum_components']:.2f} g\nError: {row['constraint_error']:.2f} g\n    \"\"\"\n    \n    ax_info.text(0.1, 0.5, info_text, fontsize=10, \n                verticalalignment='center', family='monospace',\n                bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.3))\n    \n    plt.tight_layout()\n    plt.show()","metadata":{"_uuid":"b1f0fe32-b9d9-4328-bc08-d775ee7f890a","_cell_guid":"24ffbe52-2205-4a24-82bb-073e43ef16ec","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:07.130903Z","iopub.execute_input":"2025-11-11T12:32:07.131285Z","iopub.status.idle":"2025-11-11T12:32:07.190554Z","shell.execute_reply.started":"2025-11-11T12:32:07.131260Z","shell.execute_reply":"2025-11-11T12:32:07.189215Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📸 Image Viewer Examples","metadata":{"_uuid":"f1add27b-0357-4d35-a63d-6902107913ef","_cell_guid":"7e58ae0d-5935-4456-8177-317b9ac6b2e5","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Example 1: Highest Dry_Total images\nprint(\"=\"*80)\nprint(\"【Example 1】Top 9 Images by Dry_Total\")\nprint(\"=\"*80)\nshow_images_by_condition(\n    df, IMAGE_DIR,\n    sort_by='Dry_Total_g',\n    ascending=False,\n    n_images=9,\n    ncols=3\n)","metadata":{"_uuid":"6e028c40-d481-451c-9718-154c13a95c9b","_cell_guid":"cad5b76f-cfeb-4eb6-a033-964951dbec52","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:07.191620Z","iopub.execute_input":"2025-11-11T12:32:07.191894Z","iopub.status.idle":"2025-11-11T12:32:11.852083Z","shell.execute_reply.started":"2025-11-11T12:32:07.191870Z","shell.execute_reply":"2025-11-11T12:32:11.850813Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Example 2: Lowest Dry_Total images\nprint(\"=\"*80)\nprint(\"【Example 2】Bottom 9 Images by Dry_Total\")\nprint(\"=\"*80)\nshow_images_by_condition(\n    df, IMAGE_DIR,\n    sort_by='Dry_Total_g',\n    ascending=True,\n    n_images=9,\n    ncols=3\n)","metadata":{"_uuid":"4c6d6531-967b-47c3-a764-c0ce343a22c6","_cell_guid":"253395e1-5bbe-4d6f-bf5b-4e24bdb4c177","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:11.853468Z","iopub.execute_input":"2025-11-11T12:32:11.853878Z","iopub.status.idle":"2025-11-11T12:32:16.379687Z","shell.execute_reply.started":"2025-11-11T12:32:11.853842Z","shell.execute_reply":"2025-11-11T12:32:16.378480Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Example 3: High NDVI in Spring\nprint(\"=\"*80)\nprint(\"【Example 3】Top 6 Spring Images by NDVI\")\nprint(\"=\"*80)\nshow_images_by_condition(\n    df, IMAGE_DIR,\n    sort_by='Pre_GSHH_NDVI',\n    ascending=False,\n    n_images=6,\n    filters={'season': 'Spring'},\n    ncols=3\n)","metadata":{"_uuid":"6f9822d7-b705-4a30-8226-b1eec2071755","_cell_guid":"c0e32ac3-8874-49f7-8a55-5aba47bdc55e","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:16.381331Z","iopub.execute_input":"2025-11-11T12:32:16.381602Z","iopub.status.idle":"2025-11-11T12:32:19.232202Z","shell.execute_reply.started":"2025-11-11T12:32:16.381581Z","shell.execute_reply":"2025-11-11T12:32:19.230936Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Example 4: Victoria state, high height\nprint(\"=\"*80)\nprint(\"【Example 4】Top 6 Victoria Images by Height\")\nprint(\"=\"*80)\nshow_images_by_condition(\n    df, IMAGE_DIR,\n    sort_by='Height_Ave_cm',\n    ascending=False,\n    n_images=6,\n    filters={'State': 'Vic'},\n    ncols=3\n)","metadata":{"_uuid":"ec53c18a-992a-4db2-96ac-2a253fec12f8","_cell_guid":"c1786632-205e-4ca8-9f8f-c5ac772e35cb","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:19.235950Z","iopub.execute_input":"2025-11-11T12:32:19.236368Z","iopub.status.idle":"2025-11-11T12:32:22.056418Z","shell.execute_reply.started":"2025-11-11T12:32:19.236344Z","shell.execute_reply":"2025-11-11T12:32:22.054397Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Example 5: Detailed view of specific image\nprint(\"=\"*80)\nprint(\"【Example 5】Detailed View\")\nprint(\"=\"*80)\nfirst_image_id = df.iloc[0]['image_id']\nshow_single_image_detail(df, IMAGE_DIR, first_image_id)","metadata":{"_uuid":"d8677f88-8ac4-4fc3-a731-2f881ba2631a","_cell_guid":"668d9bd6-c3e6-4303-a5e2-68c968541b1c","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:22.058332Z","iopub.execute_input":"2025-11-11T12:32:22.058831Z","iopub.status.idle":"2025-11-11T12:32:22.878343Z","shell.execute_reply.started":"2025-11-11T12:32:22.058795Z","shell.execute_reply":"2025-11-11T12:32:22.876660Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Example 6: Largest constraint violations\nprint(\"=\"*80)\nprint(\"【Example 6】Top 6 Constraint Violations\")\nprint(\"=\"*80)\nshow_images_by_condition(\n    df, IMAGE_DIR,\n    sort_by='constraint_error',\n    ascending=False,\n    n_images=6,\n    ncols=3\n)","metadata":{"_uuid":"3e85561a-43b9-441a-9081-13570f70f8d6","_cell_guid":"161a4ffb-1df6-464f-a334-d9508ec6e779","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:22.879856Z","iopub.execute_input":"2025-11-11T12:32:22.880241Z","iopub.status.idle":"2025-11-11T12:32:25.927018Z","shell.execute_reply.started":"2025-11-11T12:32:22.880211Z","shell.execute_reply":"2025-11-11T12:32:25.925706Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1️⃣2️⃣ Key Insights & Recommendations\n\n### 📊 Data Characteristics:\n1. **Target Distributions**\n   - All targets are right-skewed\n   - Consider log transformation: `log(1 + target)`\n   - Outliers present in all targets\n\n2. **Strong Correlations**\n   - High correlation between related targets\n   - Consider multi-task learning\n   - GDM = Dry_Green + Dry_Clover (by definition)\n\n3. **Physical Constraints**\n   - Dry_Total ≈ Green + Dead + Clover\n   - Can be used as regularization\n   - Small violations exist (<5g for most samples)\n\n### 🗺️ Metadata Patterns:\n1. **Geographic Variation**\n   - Different biomass levels across states\n   - Consider state as categorical feature\n\n2. **Temporal Patterns**\n   - Seasonal variation exists\n   - Year effects present\n   - Consider temporal features\n\n3. **Physical Measurements**\n   - NDVI shows positive correlation with biomass\n   - Height correlates with total biomass\n   - Both are valuable features\n\n### 🎯 Modeling Recommendations:\n\n1. **Target Transformation**\n   ```python\n   y_transformed = np.log1p(y)  # log(1 + y)\n   ```\n\n2. **Cross-Validation Strategy**\n   - Stratify by State or Season\n   - GroupKFold by date/location\n   - Watch for temporal leakage\n\n3. **Feature Engineering**\n   - Use NDVI, Height as features\n   - One-hot encode State\n   - Cyclical encoding for month/season\n   - Consider species embeddings\n\n4. **Model Architecture**\n   - Multi-head architecture (one head per target)\n   - Shared backbone with task-specific heads\n   - Add physical constraint loss:\n     ```python\n     constraint_loss = |Dry_Total - (Green + Dead + Clover)|\n     ```\n\n5. **Data Augmentation**\n   - Standard image augmentations\n   - Be careful with spatial augmentations\n   - Consider Mixup/CutMix\n\n### ⚠️ Potential Challenges:\n1. Small dataset (357 images)\n2. Class imbalance in some species\n3. Outliers need handling\n4. Test set may have different distribution\n\n---\n\n**Good luck with your modeling! 🚀**","metadata":{"_uuid":"a044eeaf-45c1-41c6-8564-31be6b39b06d","_cell_guid":"3c377787-fbdb-4737-92bd-de40f6d41e64","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## 1️⃣3️⃣ Custom Analysis Space\n\nUse the cells below for your own custom analysis!","metadata":{"_uuid":"de5aa14b-703f-4e54-b24b-680512367cd1","_cell_guid":"7629ff69-a105-451d-a475-d94e499068bb","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Your custom analysis here\n# Example: Filter and view images\n# show_images_by_condition(\n#     df, IMAGE_DIR,\n#     sort_by='YOUR_COLUMN',\n#     filters={'YOUR_FILTER': 'VALUE'},\n#     n_images=9\n# )","metadata":{"_uuid":"c4c5b813-f98b-4acb-9f81-671fd7898b37","_cell_guid":"f05048f2-8f9f-4692-86e4-fba05686a40a","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2025-11-11T12:32:25.928523Z","iopub.execute_input":"2025-11-11T12:32:25.929681Z","iopub.status.idle":"2025-11-11T12:32:25.934129Z","shell.execute_reply.started":"2025-11-11T12:32:25.929644Z","shell.execute_reply":"2025-11-11T12:32:25.932867Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n\n## 📝 Summary\n\nThis notebook has provided:\n- ✅ Complete data quality assessment\n- ✅ Comprehensive statistical analysis\n- ✅ Rich visualizations\n- ✅ Interactive image viewer\n- ✅ Physical constraint validation\n- ✅ Outlier detection\n- ✅ Modeling recommendations\n\n### Next Steps:\n1. Feature engineering\n2. Model selection\n3. Cross-validation strategy\n4. Baseline model training\n\n---\n\n**If you found this notebook helpful, please upvote! 👍**\n\n**Feel free to fork and customize for your own analysis! 🎯**\n\n---","metadata":{"_uuid":"d4a0fff9-8ded-4843-9de5-e1fdaf31a254","_cell_guid":"e06bd5ab-5320-47fc-bc67-9249380543b5","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}}]}