{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":112509,"databundleVersionId":14254895,"sourceType":"competition"}],"dockerImageVersionId":31153,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n# CSIRO Image2Biomass — Deep EDA (No Modeling)\n\nThis notebook provides a comprehensive exploration to help you understand:\n- The structure and diversity of the training data\n- Relationships between features (e.g., NDVI, height, species) and biomass targets\n- Distribution and variability across states, species, and sampling dates\n- Visual patterns in pasture images related to biomass conditions\n\nThis baseline providing an 0.5 LB provide weak results from two target\nhttps://www.kaggle.com/code/mathieuduverne/csiro-simple-baseline\n\n## Additional EDA – Investigating Weak Targets (Dry_Dead_g & Dry_Clover_g)\n\nThe model performs poorly on **Dry_Dead_g** and **Dry_Clover_g** compared to the other biomass components.  \nIn this section, we investigate potential reasons:\n1. Check their distributions and outliers.  \n2. Explore correlations with NDVI and height.  \n3. Visualize sample images.  \n4. Evaluate label consistency and potential noise.  \n5. Examine model prediction patterns later (optional link to modeling notebook).  \n","metadata":{}},{"cell_type":"markdown","source":"## 1. Setup & Data Loading","metadata":{}},{"cell_type":"code","source":"\nimport os\nimport sys\nimport gc\nimport math\nimport json\nimport random\nfrom pathlib import Path\nfrom collections import Counter, defaultdict\nfrom datetime import datetime\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom PIL import Image\n\nplt.rcParams[\"figure.figsize\"] = (8, 5)\nplt.rcParams[\"axes.grid\"] = True\nplt.rcParams[\"figure.dpi\"] = 110\n\nRANDOM_SEED = 42\nrandom.seed(RANDOM_SEED)\nnp.random.seed(RANDOM_SEED)\n\nDATA_DIR = Path('/kaggle/input/csiro-biomass')\nTRAIN_IMG_DIR = DATA_DIR / 'train'\nTEST_IMG_DIR = DATA_DIR / 'test'\nTRAIN_CSV = DATA_DIR / 'train.csv'\nTEST_CSV = DATA_DIR / 'test.csv'\nSAMPLE_SUB_CSV = DATA_DIR / 'sample_submission.csv'\n\nprint(\"Data directory exists:\", DATA_DIR.exists())\nprint(\"Train images dir exists:\", TRAIN_IMG_DIR.exists())\nprint(\"Test images dir exists:\", TEST_IMG_DIR.exists())\nprint(\"CSV files present:\", TRAIN_CSV.exists(), TEST_CSV.exists(), SAMPLE_SUB_CSV.exists())\n\ndef safe_read_csv(path, **kwargs):\n    try:\n        df = pd.read_csv(path, **kwargs)\n        print(f\"Loaded: {path.name} → shape={df.shape}\")\n        return df\n    except Exception as e:\n        print(f\"Could not load {path}: {e}\")\n        return pd.DataFrame()\n\ntrain_df = safe_read_csv(TRAIN_CSV)\ntest_df  = safe_read_csv(TEST_CSV)\nsub_df   = safe_read_csv(SAMPLE_SUB_CSV)\n\n# Basic overview\ndef df_overview(df, name):\n    print(f\"\\n{name} — shape {df.shape}\")\n    display(df.head(3))\n    print(\"\\nColumns:\", list(df.columns))\n    if df.select_dtypes(include=[np.number]).shape[1] > 0:\n        display(df.describe(include='all').T)\n\ndf_overview(train_df, \"train.csv\")\ndf_overview(test_df, \"test.csv\")\ndf_overview(sub_df, \"sample_submission.csv\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-30T14:45:11.855352Z","iopub.execute_input":"2025-10-30T14:45:11.855649Z","iopub.status.idle":"2025-10-30T14:45:13.286093Z","shell.execute_reply.started":"2025-10-30T14:45:11.855626Z","shell.execute_reply":"2025-10-30T14:45:13.285175Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Data Dictionary & Context","metadata":{}},{"cell_type":"markdown","source":"\n**Files & Folders**\n- `train/` — All training images (JPEG)\n- `test/` — All test images (JPEG)\n- `train.csv` — Metadata + ground-truth biomass per sample\n- `test.csv` — Metadata + target names for required predictions\n- `sample_submission.csv` — Submission format example\n\n**Columns in `train.csv`**\n\n| Column | Description |\n|---|---|\n| `sample_id` | Unique identifier for each sample (one per image). |\n| `image_path` | Relative path to the image (e.g., `train/ID1098771283.jpg`). |\n| `Sampling_Date` | Date of data collection. |\n| `State` | Australian state where the sample was collected. |\n| `Species` | Pasture species present, ordered by biomass (underscore-separated). |\n| `Pre_GSHH_NDVI` | GreenSeeker NDVI reading. |\n| `Height_Ave_cm` | Average pasture height (cm) measured by falling plate. |\n| `target_name` | Biomass component type: `Dry_Green_g`, `Dry_Dead_g`, `Dry_Clover_g`, `GDM_g`, `Dry_Total_g`. |\n| `target` | Ground-truth biomass value (grams). |\n\n**Biomass components**\n- `Dry_Green_g`: mass of green vegetation (excluding clover)\n- `Dry_Dead_g`: mass of dry/dead material\n- `Dry_Clover_g`: mass of dry clover biomass\n- `GDM_g`: green dry matter\n- `Dry_Total_g`: total dry biomass\n\n**Ecological relevance.** These components track forage availability and composition. Green matter and clover relate to nutritional quality, while dead material reflects senescence and seasonal dynamics. Understanding their balance helps optimise grazing decisions for animal welfare and soil health.\n","metadata":{}},{"cell_type":"markdown","source":"## 3. Missing Values & Data Integrity","metadata":{}},{"cell_type":"code","source":"\n# Unique checks & path consistency\ndef integrity_checks(df):\n    out = {}\n    if 'sample_id' in df.columns:\n        out['unique_sample_ids'] = df['sample_id'].is_unique\n        out['n_sample_ids'] = df['sample_id'].nunique()\n    if 'image_path' in df.columns:\n        out['image_path_nulls'] = int(df['image_path'].isna().sum())\n        out['image_path_examples'] = df['image_path'].dropna().head(3).tolist()\n    return out\n\nprint(\"train integrity:\", integrity_checks(train_df))\nprint(\"test integrity:\", integrity_checks(test_df))\n\n# Missing values\ndef missing_table(df):\n    miss = df.isna().sum()\n    prop = (miss / len(df)).round(4) if len(df) > 0 else miss * 0\n    m = pd.DataFrame({\"missing\": miss, \"proportion\": prop})\n    return m[m[\"missing\"] > 0].sort_values(\"missing\", ascending=False)\n\nprint(\"\\nMissing values in train:\")\ndisplay(missing_table(train_df))\n\nprint(\"\\nMissing values in test:\")\ndisplay(missing_table(test_df))\n\n# Duplicates\nif not train_df.empty:\n    dup_rows = train_df.duplicated().sum()\n    dup_sample = train_df.duplicated(subset=['sample_id']).sum() if 'sample_id' in train_df.columns else np.nan\n    print(f\"\\nDuplicate full rows in train: {dup_rows}\")\n    print(f\"Duplicate sample_id in train: {dup_sample}\")\n\n# Verify that image paths exist\ndef count_existing_paths(paths):\n    ok = 0\n    total = 0\n    for p in paths:\n        total += 1\n        full = DATA_DIR / p\n        if full.exists():\n            ok += 1\n    return ok, total\n\nif 'image_path' in train_df.columns and len(train_df) > 0:\n    sample_paths = train_df['image_path'].dropna().sample(min(500, len(train_df)), random_state=RANDOM_SEED)\n    ok, total = count_existing_paths(sample_paths)\n    print(f\"\\nTrain image path existence (sample {total}): {ok}/{total} found.\")\n\nif 'image_path' in test_df.columns and len(test_df) > 0:\n    sample_paths = test_df['image_path'].dropna().sample(min(500, len(test_df)), random_state=RANDOM_SEED)\n    ok, total = count_existing_paths(sample_paths)\n    print(f\"Test image path existence (sample {total}): {ok}/{total} found.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-30T14:45:13.287766Z","iopub.execute_input":"2025-10-30T14:45:13.288173Z","iopub.status.idle":"2025-10-30T14:45:13.684124Z","shell.execute_reply.started":"2025-10-30T14:45:13.288148Z","shell.execute_reply":"2025-10-30T14:45:13.683105Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Target Distribution Analysis","metadata":{}},{"cell_type":"code","source":"\n# Pivot so each target_name becomes a column (one row per sample_id)\nif not train_df.empty:\n    piv = (train_df\n           .pivot_table(index='sample_id', columns='target_name', values='target', aggfunc='first')\n           .reset_index())\n    display(piv.head())\n\n    target_cols = [c for c in piv.columns if c != 'sample_id']\n    print(\"Target columns:\", target_cols)\n\n    # Histograms for each target\n    import matplotlib.pyplot as plt\n    for col in target_cols:\n        plt.figure()\n        plt.hist(piv[col].dropna(), bins=50)\n        plt.title(f\"Histogram: {col}\")\n        plt.xlabel(\"value (g)\")\n        plt.ylabel(\"count\")\n        plt.show()\n\n    # Boxplots\n    plt.figure()\n    piv[target_cols].plot(kind='box', rot=45)\n    plt.title(\"Boxplots of biomass targets\")\n    plt.ylabel(\"value (g)\")\n    plt.show()\n\n    # Skewness\n    skewness = piv[target_cols].skew(numeric_only=True).sort_values(ascending=False)\n    display(pd.DataFrame({\"skew\": skewness}))\nelse:\n    print(\"train_df is empty; skipping target distribution analysis.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-30T14:45:13.685082Z","iopub.execute_input":"2025-10-30T14:45:13.685386Z","iopub.status.idle":"2025-10-30T14:45:15.324087Z","shell.execute_reply.started":"2025-10-30T14:45:13.685360Z","shell.execute_reply":"2025-10-30T14:45:15.323049Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Metadata Exploration","metadata":{}},{"cell_type":"code","source":"\nmeta_cols = ['Sampling_Date','State','Species','Pre_GSHH_NDVI','Height_Ave_cm','target_name','target']\ndisplay(train_df[meta_cols].head() if set(meta_cols).issubset(train_df.columns) else train_df.head())\n\n# Parse date & add helpers\nif 'Sampling_Date' in train_df.columns:\n    train_df['Sampling_Date'] = pd.to_datetime(train_df['Sampling_Date'], errors='coerce')\n    train_df['Year'] = train_df['Sampling_Date'].dt.year\n    train_df['Month'] = train_df['Sampling_Date'].dt.month\n    train_df['MonthName'] = train_df['Sampling_Date'].dt.month_name()\n    train_df['Quarter'] = train_df['Sampling_Date'].dt.to_period('Q').astype(str)\n    train_df['Week'] = train_df['Sampling_Date'].dt.isocalendar().week\n\n# State-wise target distributions (per target_name)\nif not train_df.empty and {'State', 'target_name','target'}.issubset(train_df.columns):\n    top_states = train_df['State'].value_counts().head(10).index.tolist()\n    subset = train_df[train_df['State'].isin(top_states)]\n    plt.figure(figsize=(10,6))\n    sns.boxplot(data=subset, x='State', y='target', hue='target_name', showfliers=False)\n    plt.title(\"Target distributions by State (top states)\")\n    plt.xticks(rotation=45)\n    plt.tight_layout()\n    plt.show()\n\n# Month-wise trends (aggregate mean)\nif not train_df.empty and {'Month','target'}.issubset(train_df.columns):\n    month_mean = (train_df.groupby(['Month','target_name'])['target']\n                  .mean().reset_index().sort_values(['target_name','Month']))\n    display(month_mean.head())\n    g = sns.FacetGrid(month_mean, col=\"target_name\", col_wrap=3, sharey=False, height=3.2)\n    g.map_dataframe(sns.lineplot, x=\"Month\", y=\"target\")\n    g.set_titles(col_template=\"{col_name}\")\n    g.fig.suptitle(\"Mean biomass by Month\", y=1.05)\n    plt.show()\n\n# NDVI & Height relationships with targets\nif not train_df.empty:\n    # Average across target_name by sample to align features\n    feats = (train_df\n             .pivot_table(index='sample_id',\n                          values=['Pre_GSHH_NDVI','Height_Ave_cm'],\n                          aggfunc='first')\n            )\n    targs = (train_df\n             .pivot_table(index='sample_id', columns='target_name', values='target', aggfunc='first'))\n    merged = feats.join(targs, how='inner').reset_index()\n    display(merged.head())\n\n    # Scatter plots: NDVI vs each target\n    targ_cols = [c for c in merged.columns if c not in ['sample_id','Pre_GSHH_NDVI','Height_Ave_cm']]\n    for col in targ_cols:\n        plt.figure()\n        plt.scatter(merged['Pre_GSHH_NDVI'], merged[col], s=10, alpha=0.6)\n        plt.title(f\"Pre_GSHH_NDVI vs {col}\")\n        plt.xlabel(\"Pre_GSHH_NDVI\")\n        plt.ylabel(col)\n        plt.show()\n\n    # Scatter plots: Height vs target\n    for col in targ_cols:\n        plt.figure()\n        plt.scatter(merged['Height_Ave_cm'], merged[col], s=10, alpha=0.6)\n        plt.title(f\"Height_Ave_cm vs {col}\")\n        plt.xlabel(\"Height_Ave_cm (cm)\")\n        plt.ylabel(col)\n        plt.show()\n\n    # Correlation heatmap including features\n    corr_cols = ['Pre_GSHH_NDVI','Height_Ave_cm'] + targ_cols\n    corr2 = merged[corr_cols].corr()\n    display(corr2)\n    plt.figure()\n    sns.heatmap(corr2, annot=True, fmt=\".2f\", cmap=\"viridis\")\n    plt.title(\"Correlation: features & targets\")\n    plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-30T14:45:15.325181Z","iopub.execute_input":"2025-10-30T14:45:15.326176Z","iopub.status.idle":"2025-10-30T14:45:19.561377Z","shell.execute_reply.started":"2025-10-30T14:45:15.326146Z","shell.execute_reply":"2025-10-30T14:45:19.559938Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6. Image Data Exploration","metadata":{}},{"cell_type":"code","source":"\n# Helper: open image safely\ndef open_image(rel_path, max_size=768):\n    # Open image by relative path within DATA_DIR and optionally thumbnail it.\n    path = DATA_DIR / rel_path\n    with Image.open(path) as img:\n        img = img.convert(\"RGB\")\n        # Resize (keep aspect) for display\n        img.thumbnail((max_size, max_size))\n        return img\n\n# Random sample of images with captions (metadata + targets)\nif 'image_path' in train_df.columns and len(train_df) > 0:\n    print(\"Displaying a few random training images…\")\n    sample_ids = train_df['sample_id'].dropna().unique()\n    show_sample = np.random.choice(sample_ids, size=min(9, len(sample_ids)), replace=False)\n\n    # Collect metadata per sample_id\n    meta_cols = ['State','Species','Sampling_Date','Pre_GSHH_NDVI','Height_Ave_cm']\n    targ_piv = (train_df.pivot_table(index='sample_id', columns='target_name', values='target', aggfunc='first'))\n    info = (train_df.drop_duplicates('sample_id')[['sample_id','image_path'] + [c for c in meta_cols if c in train_df.columns]]\n            .set_index('sample_id').join(targ_piv, how='left'))\n\n    ncols = 3\n    nrows = int(math.ceil(len(show_sample) / ncols))\n    fig, axes = plt.subplots(nrows, ncols, figsize=(12, 4*nrows))\n    axes = axes.flatten() if isinstance(axes, np.ndarray) else [axes]\n\n    for ax, sid in zip(axes, show_sample):\n        try:\n            rec = info.loc[sid]\n            img = open_image(rec['image_path'])\n            ax.imshow(img)\n            ndvi = rec.get('Pre_GSHH_NDVI', np.nan)\n            ndvi_str = f\"{ndvi:.3f}\" if pd.notna(ndvi) else \"NA\"\n            h = rec.get('Height_Ave_cm', np.nan)\n            h_str = f\"{h:.1f}\" if pd.notna(h) else \"NA\"\n            caption = f\"{rec.get('State','?')} | NDVI={ndvi_str} | H={h_str} cm\"\n            if 'Dry_Total_g' in rec.index and not pd.isna(rec['Dry_Total_g']):\n                caption += f\" | Dry_Total_g={rec['Dry_Total_g']:.1f}\"\n            ax.set_title(caption, fontsize=10)\n            ax.axis('off')\n        except Exception as e:\n            ax.axis('off')\n    for k in range(len(show_sample), len(axes)):\n        axes[k].axis('off')\n    plt.tight_layout()\n    plt.show()\n\n# Image statistics: RGB histograms and basic stats for a sample\ndef image_stats(rel_paths, max_n=200):\n    vals = []\n    for p in rel_paths[:max_n]:\n        try:\n            img = open_image(p, max_size=512)\n            arr = np.asarray(img).astype(np.float32)\n            vals.append({\n                \"path\": p,\n                \"mean_R\": float(arr[:,:,0].mean()),\n                \"mean_G\": float(arr[:,:,1].mean()),\n                \"mean_B\": float(arr[:,:,2].mean()),\n                \"std_R\": float(arr[:,:,0].std()),\n                \"std_G\": float(arr[:,:,1].std()),\n                \"std_B\": float(arr[:,:,2].std()),\n                \"mean_intensity\": float(arr.mean()),\n                \"std_intensity\": float(arr.std()),\n            })\n        except Exception as e:\n            continue\n    return pd.DataFrame(vals)\n\nif 'image_path' in train_df.columns and len(train_df) > 0:\n    unique_paths = train_df.drop_duplicates('sample_id')['image_path'].dropna().tolist()\n    img_df = image_stats(unique_paths, max_n=200)\n    print(\"Image stats (sample):\", img_df.shape)\n    display(img_df.head())\n\n    # Channel means\n    for ch in [\"mean_R\",\"mean_G\",\"mean_B\",\"mean_intensity\"]:\n        plt.figure()\n        plt.hist(img_df[ch].dropna(), bins=40)\n        plt.title(f\"Histogram: {ch}\")\n        plt.xlabel(ch)\n        plt.ylabel(\"count\")\n        plt.show()\n\n    # Compare low vs high biomass (Dry_Total_g if available)\n    if 'Dry_Total_g' in train_df['target_name'].unique():\n        piv = train_df.pivot_table(index='sample_id', columns='target_name', values='target', aggfunc='first').reset_index()\n        merged_img = (train_df.drop_duplicates('sample_id')[['sample_id','image_path']]\n                      .merge(piv, on='sample_id', how='left'))\n        merged_img = merged_img.merge(img_df.rename(columns={\"path\":\"image_path\"}), on='image_path', how='left')\n        if 'Dry_Total_g' in merged_img.columns:\n            q_low, q_high = merged_img['Dry_Total_g'].quantile([0.1, 0.9])\n            low = merged_img[merged_img['Dry_Total_g'] <= q_low].dropna(subset=['mean_intensity']).head(12)\n            high = merged_img[merged_img['Dry_Total_g'] >= q_high].dropna(subset=['mean_intensity']).head(12)\n\n            def show_grid(df, title):\n                n = len(df)\n                ncols = 4\n                nrows = int(math.ceil(n / ncols))\n                fig, axes = plt.subplots(nrows, ncols, figsize=(12, 3*nrows))\n                axes = axes.flatten() if isinstance(axes, np.ndarray) else [axes]\n                for ax, (_, row) in zip(axes, df.iterrows()):\n                    try:\n                        img = open_image(row['image_path'])\n                        ax.imshow(img)\n                        ax.set_title(f\"Dry_Total_g={row['Dry_Total_g']:.0f}\", fontsize=9)\n                        ax.axis('off')\n                    except Exception:\n                        ax.axis('off')\n                for k in range(n, len(axes)):\n                    axes[k].axis('off')\n                fig.suptitle(title, y=0.98)\n                plt.tight_layout()\n                plt.show()\n\n            show_grid(low, \"Examples: Low Dry_Total_g (bottom 10%)\")\n            show_grid(high, \"Examples: High Dry_Total_g (top 10%)\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-30T14:45:19.563414Z","iopub.execute_input":"2025-10-30T14:45:19.563688Z","iopub.status.idle":"2025-10-30T14:45:46.237810Z","shell.execute_reply.started":"2025-10-30T14:45:19.563664Z","shell.execute_reply":"2025-10-30T14:45:46.236986Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 7. Combined Feature Insights","metadata":{}},{"cell_type":"code","source":"\n# Merge features & targets (one row per sample)\nif not train_df.empty:\n    features = (train_df\n                .pivot_table(index='sample_id',\n                             values=['Pre_GSHH_NDVI','Height_Ave_cm','State','Species','Sampling_Date'],\n                             aggfunc='first'))\n    targets = (train_df.pivot_table(index='sample_id', columns='target_name', values='target', aggfunc='first'))\n    dfm = features.join(targets, how='left').reset_index()\n\n    # Grouped means by State\n    if 'State' in dfm.columns:\n        g_state = (dfm.groupby('State')[targets.columns.tolist()].mean().sort_index())\n        display(g_state.head())\n        g_state.plot(kind='bar', subplots=True, layout=(math.ceil(len(targets.columns)/2), 2), figsize=(12,8), legend=False, sharex=True, sharey=False)\n        plt.suptitle(\"Mean biomass targets by State\")\n        plt.tight_layout()\n        plt.show()\n\n    # Species frequency & contribution\n    if 'Species' in dfm.columns:\n        species_counts = dfm['Species'].value_counts().head(25)\n        plt.figure(figsize=(10,6))\n        species_counts.sort_values(ascending=True).plot(kind='barh')\n        plt.title(\"Top Species combinations (by count)\")\n        plt.xlabel(\"count\")\n        plt.tight_layout()\n        plt.show()\n\n        # Per-species (top 10) mean Dry_Total_g, if available\n        top10 = species_counts.head(10).index.tolist()\n        if 'Dry_Total_g' in dfm.columns:\n            sp_mean = dfm[dfm['Species'].isin(top10)].groupby('Species')['Dry_Total_g'].mean().sort_values()\n            plt.figure(figsize=(10,6))\n            sp_mean.plot(kind='barh')\n            plt.title(\"Mean Dry_Total_g for top-10 Species combos\")\n            plt.xlabel(\"Dry_Total_g (g)\")\n            plt.tight_layout()\n            plt.show()\n\n    # Sampling period comparisons\n    if 'Sampling_Date' in dfm.columns:\n        dfm['Year'] = pd.to_datetime(dfm['Sampling_Date'], errors='coerce').dt.year\n        dfm['Month'] = pd.to_datetime(dfm['Sampling_Date'], errors='coerce').dt.month\n        # Multi-panel comparison across targets\n        ncols = 2\n        tcols = [c for c in targets.columns]\n        nrows = int(math.ceil(len(tcols) / ncols))\n        fig, axes = plt.subplots(nrows, ncols, figsize=(12, 4*nrows))\n        axes = axes.flatten() if isinstance(axes, np.ndarray) else [axes]\n        for ax, col in zip(axes, tcols):\n            tmp = dfm.groupby('Month')[col].mean()\n            ax.plot(tmp.index, tmp.values, marker='o')\n            ax.set_title(f\"Monthly mean of {col}\")\n            ax.set_xlabel(\"Month\")\n            ax.set_ylabel(col)\n        for k in range(len(tcols), len(axes)):\n            axes[k].axis('off')\n        plt.tight_layout()\n        plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-30T14:45:46.238700Z","iopub.execute_input":"2025-10-30T14:45:46.238994Z","iopub.status.idle":"2025-10-30T14:45:48.718467Z","shell.execute_reply.started":"2025-10-30T14:45:46.238970Z","shell.execute_reply":"2025-10-30T14:45:48.717515Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 8. Investigate on weak targets","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\nsubset = train_df[train_df[\"target_name\"].isin([\"Dry_Dead_g\",\"Dry_Clover_g\"])]\n\nplt.figure(figsize=(10,4))\nsns.histplot(data=subset, x=\"target\", hue=\"target_name\", kde=True, bins=30)\nplt.title(\"Target Distribution – Dry_Dead_g vs Dry_Clover_g\")\nplt.xlabel(\"Target value (grams)\")\nplt.show()\n\n# Boxplot for outliers\nplt.figure(figsize=(8,4))\nsns.boxplot(data=subset, x=\"target_name\", y=\"target\")\nplt.title(\"Boxplot – Target spread & outliers\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-30T14:47:28.710950Z","iopub.execute_input":"2025-10-30T14:47:28.711271Z","iopub.status.idle":"2025-10-30T14:47:29.250475Z","shell.execute_reply.started":"2025-10-30T14:47:28.711250Z","shell.execute_reply":"2025-10-30T14:47:29.249498Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"corrs = (\n    subset.groupby(\"target_name\")[[\"Pre_GSHH_NDVI\",\"Height_Ave_cm\",\"target\"]]\n    .corr()\n    .iloc[::3, -1]  # correlation with target\n)\nprint(\"Correlation with target:\")\nprint(corrs)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-30T14:47:29.252207Z","iopub.execute_input":"2025-10-30T14:47:29.252454Z","iopub.status.idle":"2025-10-30T14:47:29.264021Z","shell.execute_reply.started":"2025-10-30T14:47:29.252436Z","shell.execute_reply":"2025-10-30T14:47:29.262963Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import lightgbm as lgb\nfrom sklearn.metrics import r2_score\n\nfor target in [\"Dry_Dead_g\",\"Dry_Clover_g\"]:\n    df_t = train_df[train_df[\"target_name\"] == target]\n    X = df_t[[\"Pre_GSHH_NDVI\",\"Height_Ave_cm\"]]\n    y = df_t[\"target\"]\n    model = lgb.LGBMRegressor().fit(X, y)\n    r2 = r2_score(y, model.predict(X))\n    print(f\"{target} | R² from meta-features only: {r2:.3f}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-30T14:48:35.416349Z","iopub.execute_input":"2025-10-30T14:48:35.416702Z","iopub.status.idle":"2025-10-30T14:48:41.519430Z","shell.execute_reply.started":"2025-10-30T14:48:35.416679Z","shell.execute_reply":"2025-10-30T14:48:41.518466Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\nsummary\n- **Highly skewed or sparse distributions** → possible data imbalance.  \n- **Weak correlations** → NDVI/height not predictive for these targets.   Dry_Clover_g  Pre_GSHH_NDVI    0.224150 Dry_Dead_g    Pre_GSHH_NDVI   -0.122818\n","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}