{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":112509,"databundleVersionId":14254895,"sourceType":"competition"}],"dockerImageVersionId":31154,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n---\n\nThe final Python code is structured into four separate cells to ensure a clear, repeatable, and highly competitive workflow for the CSIRO - Image2Biomass Prediction competition.\n\n## 1. Training Cell Explanation (3-Target Model)\n\nThis cell handles all data preparation and model training, incorporating the competitive necessity of log-transformation and target subsetting.\n\n* **Target Selection (3 Units):** The code extracts only the **three independent targets** to predict: `Dry_Total_g`, `GDM_g`, and `Dry_Green_g`. This strategy ensures the final two dependent components are mathematically derived later, guaranteeing adherence to physical constraints. The model's output layer is set to **3 units**.\n* **Log Transformation:** Target values are transformed using $\\mathbf{\\log(target + EPS)}$. This technique is essential for stabilizing the variance of highly skewed mass measurements and improving the performance of the Mean Squared Error (MSE) loss function used for training.\n* **Data Structure:** The code converts the **long-format** $\\text{train.csv}$ (one row per image-target pair) into a **wide format** (one row per image) required for the deep learning model.\n* **Architecture:** A simple **Convolutional Neural Network (CNN)** is used as a baseline to process the image data. The model is compiled with MSE loss, appropriate for the log-transformed regression task.\n\n---\n\n## 2. Inference Cell Explanation (Constraint Enforcement)\n\nThis cell applies the trained 3-target model to the test data and critically enforces the competition's physical rules via the custom inference function.\n\n* **The `enforce_biological_constraints_corrected` Function:** This is the core of the competitive solution. It accepts the model's 3 log-predictions and performs three steps:\n    1.  **Inverse Log-Transformation:** Converts the predictions back to grams using $\\mathbf{\\exp(\\text{prediction}) - EPS}$.\n    2.  **Hierarchy Enforcement:** Ensures the predicted masses respect the principle of inclusion: $\\mathbf{\\text{Dry\\_Total\\_g} \\ge \\text{GDM\\_g}}$ and $\\mathbf{\\text{GDM\\_g} \\ge \\text{Dry\\_Green\\_g}}$.\n    3.  **Dependent Target Derivation:** It calculates the final two components, ensuring perfect physical compliance:\n        * $\\mathbf{\\text{Dry\\_Dead\\_g} = \\text{Dry\\_Total\\_g} - \\text{GDM\\_g}}$\n        * $\\mathbf{\\text{Dry\\_Clover\\_g} = \\text{GDM\\_g} - \\text{Dry\\_Green\\_g}}$\n* **Submission Formatting:** The final 5-column wide prediction array is converted back into the required **long format** (columns: $\\text{sample\\_id}$ and $\\text{target}$) using $\\text{melt}$ and $\\text{merge}$ operations to produce the $\\text{submission.csv}$ file.\n\n---\n\n## 3. Analysis Cell Explanation\n\nThis cell validates the structural and logical integrity of the final output.\n\n* **Format Check:** Confirms the output file adheres to the required **two-column long format** ($\\text{sample\\_id}$ and $\\text{target}$).\n* **Constraint Check:** Mathematically verifies the two primary derivation rules using the predicted values for the first test sample. This step is the ultimate proof that the derivation logic in the inference cell worked correctly.\n\n---\n\n## 4. EDA Cell Explanation (Exploratory Data Analysis)\n\nThis cell helps visualize and understand the characteristics of the competition data before complex modeling.\n\n* **Target Distribution:** It plots histograms and box plots of the raw biomass targets ($\\text{target}$ column) to show their distribution and range. This visualization confirms the **high skewness** of the data, which justifies the use of the **log transformation** in the training cell.\n* **Image Visualization:** It loads and plots sample images from both the **training** and **test** sets. This confirms that the image loading paths are correct for both data sets and helps visually assess the quality and characteristics of the pasture images.\n* **Warning Suppression:** Includes the $\\text{warnings}$ module to suppress specific `FutureWarning` messages from the $\\text{seaborn}$ library, resulting in a clean notebook execution.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, Dropout\nfrom tensorflow.keras.preprocessing.image import load_img, img_to_array\nfrom sklearn.model_selection import train_test_split\nfrom tqdm.auto import tqdm \nimport os\n\n# --- A. CONFIGURATION ---\nDATA_PATH = '/kaggle/input/csiro-biomass/'\nTRAIN_CSV = os.path.join(DATA_PATH, 'train.csv')\nTEST_CSV = os.path.join(DATA_PATH, 'test.csv') \nTRAIN_IMG_DIR = DATA_PATH \nIMG_SIZE = (128, 128) \nEPS = 1e-6 \n\n# 🛑 Targets the model WILL predict (The 3 independent components)\nPREDICTED_TARGETS = ['Dry_Total_g', 'GDM_g', 'Dry_Green_g']\n# All 5 targets are used for the final submission column list\nTARGET_NAMES = ['Dry_Green_g', 'Dry_Dead_g', 'Dry_Clover_g', 'GDM_g', 'Dry_Total_g']\n\nIMAGE_PATH_COL = 'image_path'\nTARGET_COL = 'target' \n\n# --- B. DATA LOADING & PREPARATION ---\n\nprint(\"--- Data Loading and Log-Transformation (3 Targets) ---\")\n\ndf_train_long = pd.read_csv(TRAIN_CSV)\n\n# Pivot the long data into wide format\ndf_train_wide = df_train_long.pivot_table(\n    index=[IMAGE_PATH_COL],\n    columns='target_name',\n    values=TARGET_COL\n).reset_index()\n\n# Drop NaNs based ONLY on the three targets we plan to predict\ndf_train_wide.dropna(subset=PREDICTED_TARGETS, inplace=True)\n\n# Log-Transformation of Targets (Only the 3 predicted targets)\ny_targets = df_train_wide[PREDICTED_TARGETS].values\ny_targets_log = np.log(y_targets + EPS)\n\n# Load Images\nX_images = []\nfor index, row in tqdm(df_train_wide.iterrows(), total=len(df_train_wide), desc=\"Loading Train Images\"):\n    image_path = os.path.join(TRAIN_IMG_DIR, row[IMAGE_PATH_COL])\n    try:\n        img = load_img(image_path, target_size=IMG_SIZE)\n        img_array = img_to_array(img) / 255.0 \n        X_images.append(img_array)\n    except FileNotFoundError:\n        continue \nX_images = np.array(X_images)\n\n# Split Data - Train the model using the LOG-TRANSFORMED targets\nX_train_img, X_val_img, y_train_log, y_val_log = train_test_split(\n    X_images, y_targets_log, test_size=0.2, random_state=42\n)\n\n# --- C. MODEL DEFINITION & TRAINING ---\n\ndef create_image_model(img_shape, num_targets):\n    input_img = Input(shape=img_shape, name='image_input')\n    x = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img)\n    x = MaxPooling2D((2, 2))(x)\n    x = Flatten()(x)\n    x = Dense(128, activation='relu')(x)\n    x = Dropout(0.5)(x)\n    \n    # Output layer is Dense(3) for the 3 predicted components\n    output_biomass = Dense(num_targets, activation='linear', name='biomass_output')(x)\n    model = Model(inputs=input_img, outputs=output_biomass)\n    return model\n\n# Initialize and compile the model\nbiomass_model = create_image_model(\n    img_shape=X_train_img.shape[1:], \n    num_targets=len(PREDICTED_TARGETS)\n)\n\nbiomass_model.compile(optimizer='adam', loss='mse', metrics=['mae']) \n\nn_epoch=40\nprint(f\"\\n--- Starting Model Training ({n_epoch} Epochs) ---\")\nbiomass_model.fit(\n    X_train_img, y_train_log, \n    validation_data=(X_val_img, y_val_log),\n    epochs=n_epoch, batch_size=32, verbose=1\n)\n\n# Store necessary variables globally for the next cell\nglobal TEST_CSV_VAR, TARGET_NAMES_VAR, PREDICTED_TARGETS_VAR, IMAGE_PATH_COL_VAR, SUBMISSION_ID_COL_VAR, TARGET_COL_VAR, EPS_VAR, TEST_IMG_DIR_VAR, IMG_SIZE_VAR, biomass_model_var\nTEST_CSV_VAR = TEST_CSV\nTARGET_NAMES_VAR = TARGET_NAMES\nPREDICTED_TARGETS_VAR = PREDICTED_TARGETS\nIMAGE_PATH_COL_VAR = IMAGE_PATH_COL\nSUBMISSION_ID_COL_VAR = 'sample_id'\nTARGET_COL_VAR = TARGET_COL\nEPS_VAR = EPS\nTEST_IMG_DIR_VAR = os.path.join(DATA_PATH)\nIMG_SIZE_VAR = IMG_SIZE\nbiomass_model_var = biomass_model\n\nprint('training complete')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-01T23:02:44.619316Z","iopub.execute_input":"2025-11-01T23:02:44.619632Z","iopub.status.idle":"2025-11-01T23:03:22.500101Z","shell.execute_reply.started":"2025-11-01T23:02:44.619610Z","shell.execute_reply":"2025-11-01T23:03:22.499536Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom tensorflow.keras.preprocessing.image import load_img, img_to_array\nfrom tqdm.auto import tqdm \nimport os\n\n# --- A. INFERENCE FUNCTIONS ---\n\ndef enforce_biological_constraints_corrected(predictions_wide_log, PREDICTED_TARGETS, TARGET_NAMES, EPS):\n    \"\"\"\n    Applies constraints by predicting 3 targets and deriving the other 2.\n    \"\"\"\n    \n    # 1. Inverse Log-Transformation\n    predictions_wide_g = np.exp(predictions_wide_log) - EPS\n    \n    # Create a DataFrame using only the 3 PREDICTED targets\n    df_pred = pd.DataFrame(predictions_wide_g, columns=PREDICTED_TARGETS)\n\n    # 2. Enforces Hierarchy on PREDICTED components\n    df_pred['GDM_g'] = np.maximum(df_pred['GDM_g'].values, df_pred['Dry_Green_g'].values) # GDM_g >= Dry_Green_g\n    df_pred['Dry_Total_g'] = np.maximum(df_pred['Dry_Total_g'].values, df_pred['GDM_g'].values) # Dry_Total_g >= GDM_g\n    \n    # Ensure all predictions are non-negative before derivation\n    for col in PREDICTED_TARGETS:\n        df_pred[col] = np.maximum(0, df_pred[col])\n\n    # 3. Derives Missing Targets (CRITICAL)\n    df_pred['Dry_Clover_g'] = np.maximum(0, df_pred['GDM_g'].values - df_pred['Dry_Green_g'].values)\n    df_pred['Dry_Dead_g'] = np.maximum(0, df_pred['Dry_Total_g'].values - df_pred['GDM_g'].values)\n    \n    # Reorder columns to match the required final TARGET_NAMES order\n    final_order = TARGET_NAMES \n    predictions_final = df_pred[final_order].values\n\n    return predictions_final\n\ndef run_inference(model, X_test_images, PREDICTED_TARGETS, TARGET_NAMES, EPS):\n    # Get log-transformed predictions (3 outputs)\n    predictions_log = model.predict(X_test_images)\n    \n    # Enforce constraints and transform back to grams (5 outputs)\n    predictions_g = enforce_biological_constraints_corrected(predictions_log, PREDICTED_TARGETS, TARGET_NAMES, EPS)\n    \n    return predictions_g\n\n# --- B. EXECUTION ---\n\nprint(\"\\n--- Generating Submission File ---\")\n\n# Load configuration from global variables\nTEST_CSV = TEST_CSV_VAR\nTARGET_NAMES = TARGET_NAMES_VAR\nPREDICTED_TARGETS = PREDICTED_TARGETS_VAR\nIMAGE_PATH_COL = IMAGE_PATH_COL_VAR\nSUBMISSION_ID_COL = SUBMISSION_ID_COL_VAR\nTARGET_COL = TARGET_COL_VAR\nEPS = EPS_VAR\nTEST_IMG_DIR = TEST_IMG_DIR_VAR\nIMG_SIZE = IMG_SIZE_VAR\nbiomass_model = biomass_model_var\n\ndf_test_long = pd.read_csv(TEST_CSV)\ndf_test_wide_meta = df_test_long.drop_duplicates(subset=[IMAGE_PATH_COL]).reset_index(drop=True)\n\n# Prepare wide test data for prediction\nX_test_images_wide = []\nfor index, row in tqdm(df_test_wide_meta.iterrows(), total=len(df_test_wide_meta), desc=\"Loading Test Images\"):\n    image_path_full = os.path.join(TEST_IMG_DIR, row[IMAGE_PATH_COL])\n    try:\n        img = load_img(image_path_full, target_size=IMG_SIZE)\n        X_test_images_wide.append(img_to_array(img) / 255.0) \n    except FileNotFoundError:\n        continue \nX_test_images_wide = np.array(X_test_images_wide)\n\n\nif len(X_test_images_wide) > 0:\n    predictions_wide_g = run_inference(biomass_model, X_test_images_wide, PREDICTED_TARGETS, TARGET_NAMES, EPS)\n\n    # Convert wide predictions (in grams) back to a long DataFrame\n    df_pred_wide = pd.DataFrame(predictions_wide_g, columns=TARGET_NAMES)\n    df_pred_wide.insert(0, IMAGE_PATH_COL, df_test_wide_meta[IMAGE_PATH_COL])\n\n    # Melt the wide prediction table back to long format \n    df_pred_long = df_pred_wide.melt(\n        id_vars=[IMAGE_PATH_COL],\n        value_vars=TARGET_NAMES,\n        var_name='target_name',\n        value_name=TARGET_COL\n    )\n\n    # Merge predictions back to the original test structure\n    df_submission = df_test_long[[SUBMISSION_ID_COL, IMAGE_PATH_COL, 'target_name']].merge(\n        df_pred_long,\n        on=[IMAGE_PATH_COL, 'target_name'],\n        how='left'\n    )\n    \n    # Final submission structure\n    df_submission = df_submission[[SUBMISSION_ID_COL, TARGET_COL]]\n\n    # Save to CSV\n    SUBMISSION_FILE = 'submission.csv'\n    df_submission.to_csv(SUBMISSION_FILE, index=False)\n\n    print(f\"Successfully generated submission file: {SUBMISSION_FILE}\")\nelse:\n    print(\"No test images were loaded. Submission file not created.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-01T23:03:52.638907Z","iopub.execute_input":"2025-11-01T23:03:52.639174Z","iopub.status.idle":"2025-11-01T23:03:52.948135Z","shell.execute_reply.started":"2025-11-01T23:03:52.639154Z","shell.execute_reply":"2025-11-01T23:03:52.947558Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\n\n# --- CONFIGURATION (Load from global) ---\nSUBMISSION_FILE = 'submission.csv'\nSUBMISSION_ID_COL = SUBMISSION_ID_COL_VAR\nTARGET_COL = TARGET_COL_VAR\nEPS = EPS_VAR\n\n# --- FILE VERIFICATION ---\n\nprint(\"\\n--- Final Submission File Verification and Content Analysis ---\")\n\nif not os.path.exists(SUBMISSION_FILE):\n    print(f\"FATAL ERROR: Submission file '{SUBMISSION_FILE}' not found.\")\nelse:\n    df_submission = pd.read_csv(SUBMISSION_FILE)\n\n    # 1. Validation Checks\n    expected_cols = [SUBMISSION_ID_COL, TARGET_COL]\n    if df_submission.columns.tolist() != expected_cols:\n        print(f\"❌ FAIL: Expected columns {expected_cols}, found {df_submission.columns.tolist()}.\")\n    else:\n        print(\"✅ PASS: Submission file has the correct columns and order.\")\n\n    # 2. Print Structure\n    print(\"-\" * 50)\n    print(f\"Shape: {df_submission.shape}\")\n    \n    print(\"\\nSubmission Head (First 10 rows, showing constrained predictions):\")\n    print(df_submission.head(10).to_markdown(index=False))\n    \n    # 3. Post-Processing Constraint Check (Validation based on the first sample)\n    \n    if len(df_submission) >= 5:\n        # Sort the first 5 rows to ensure correct mapping for constraint check\n        df_check = df_submission.head(5).sort_values(by=SUBMISSION_ID_COL)\n        \n        # Mapping values based on the component name in sample_id\n        T = df_check[df_check[SUBMISSION_ID_COL].str.contains('Total_g')]['target'].iloc[0]\n        M = df_check[df_check[SUBMISSION_ID_COL].str.contains('GDM_g')]['target'].iloc[0]\n        G = df_check[df_check[SUBMISSION_ID_COL].str.contains('Green_g')]['target'].iloc[0]\n        D = df_check[df_check[SUBMISSION_ID_COL].str.contains('Dead_g')]['target'].iloc[0]\n        C = df_check[df_check[SUBMISSION_ID_COL].str.contains('Clover_g')]['target'].iloc[0]\n        \n        # Check Total Derivation: T = M + D\n        total_derived_check = M + D\n        \n        # Check GDM Derivation: M = G + C\n        gdm_derived_check = G + C\n        \n        print(\"\\n--- Biological Constraint Check (First Sample) ---\")\n        print(f\"Dry_Total_g (T): {T:.4f} | GDM_g (M): {M:.4f} | Dry_Green_g (G): {G:.4f}\")\n        \n        # Check if derived components match the total/GDM:\n        if np.isclose(T, total_derived_check, atol=EPS * 10):\n            print(f\"✅ PASS: Dry_Total_g (T={T:.4f}) matches GDM + Dry_Dead ({total_derived_check:.4f})\")\n        else:\n            print(f\"❌ FAIL: Dry_Total_g ({T:.4f}) should equal GDM + Dry_Dead ({total_derived_check:.4f})\")\n\n        if np.isclose(M, gdm_derived_check, atol=EPS * 10):\n            print(f\"✅ PASS: GDM_g (M={M:.4f}) matches Dry_Green + Dry_Clover ({gdm_derived_check:.4f})\")\n        else:\n            print(f\"❌ FAIL: GDM_g ({M:.4f}) should equal Dry_Green + Dry_Clover ({gdm_derived_check:.4f})\")\n    \n    print(\"-\" * 50)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-01T23:04:00.939374Z","iopub.execute_input":"2025-11-01T23:04:00.940084Z","iopub.status.idle":"2025-11-01T23:04:00.954720Z","shell.execute_reply.started":"2025-11-01T23:04:00.940058Z","shell.execute_reply":"2025-11-01T23:04:00.953930Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nimport pandas as pd\nimport numpy as np\nimport os\nfrom tensorflow.keras.preprocessing.image import load_img, img_to_array\nimport warnings # <-- ADDED\n\n# Suppress the specific FutureWarning from seaborn\nwarnings.filterwarnings(\"ignore\", category=FutureWarning, module=\"seaborn\")\n\n# --- Configuration (Reloaded from globals) ---\nDATA_PATH = '/kaggle/input/csiro-biomass/'\nTRAIN_CSV = os.path.join(DATA_PATH, 'train.csv')\nTEST_CSV = os.path.join(DATA_PATH, 'test.csv') \nIMAGE_PATH_COL = 'image_path'\nTARGET_COL = 'target'\nTARGET_NAMES = ['Dry_Green_g', 'Dry_Dead_g', 'Dry_Clover_g', 'GDM_g', 'Dry_Total_g']\nIMG_SIZE = (128, 128)\n\n# --- 1. Load Data (If not already loaded in the environment) ---\ntry:\n    df_train_long = pd.read_csv(TRAIN_CSV)\n    df_test_long = pd.read_csv(TEST_CSV)\nexcept NameError:\n    # If the environment was reset, load data again\n    df_train_long = pd.read_csv(os.path.join(DATA_PATH, 'train.csv'))\n    df_test_long = pd.read_csv(os.path.join(DATA_PATH, 'test.csv'))\n\n# --- 2. Target Variable Analysis (Training Data) ---\nprint(\"--- Target Variable Distribution Analysis (Training Data) ---\")\n\n# Plot the distribution of the 'target' column (all components combined)\nplt.figure(figsize=(12, 5))\n# This plotting call generates the FutureWarning\nsns.histplot(df_train_long[TARGET_COL], bins=50, kde=True, log_scale=False)\nplt.title(f'Distribution of Raw Biomass Target ({TARGET_COL})')\nplt.xlabel('Biomass (grams)')\nplt.show()\n\n# Print descriptive statistics for raw targets\nprint(\"\\nDescriptive Statistics for Raw Targets (All Components):\")\nprint(df_train_long[TARGET_COL].describe().to_markdown())\n\n# Plot individual distributions for the five components\ndf_pivot = df_train_long.pivot_table(\n    index=IMAGE_PATH_COL,\n    columns='target_name',\n    values=TARGET_COL\n)\n\nplt.figure(figsize=(15, 8))\ndf_pivot[TARGET_NAMES].boxplot()\nplt.title('Box Plot of Biomass Distribution by Component')\nplt.ylabel('Biomass (grams)')\nplt.grid(True, axis='y')\nplt.show()\n\n# --- 3. Image Visualization (Train and Test) ---\nprint(\"\\n--- Sample Image Visualization ---\")\n\n# Get a sample train image path\ntrain_image_path = os.path.join(DATA_PATH, df_train_long[IMAGE_PATH_COL].iloc[0])\n# Get the single test image path (it's the first and likely only one)\ntest_image_path = os.path.join(DATA_PATH, df_test_long[IMAGE_PATH_COL].iloc[0])\n\nfig, axes = plt.subplots(1, 2, figsize=(12, 6))\n\n# Load and plot a TRAIN image\ntry:\n    img_train = load_img(train_image_path)\n    axes[0].imshow(img_train)\n    axes[0].set_title(f'Sample Train Image\\n{os.path.basename(train_image_path)}')\n    axes[0].axis('off')\nexcept FileNotFoundError:\n    axes[0].set_title('Train Image Not Found')\n\n# Load and plot the TEST image\ntry:\n    img_test = load_img(test_image_path)\n    axes[1].imshow(img_test)\n    axes[1].set_title(f'Sample Test Image\\n{os.path.basename(test_image_path)}')\n    axes[1].axis('off')\nexcept FileNotFoundError:\n    axes[1].set_title('Test Image Not Found')\n\nplt.tight_layout()\nplt.show()\n\nprint(\"\\nEDA complete. Review the plots for distribution analysis.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-01T23:04:20.793329Z","iopub.execute_input":"2025-11-01T23:04:20.793626Z","iopub.status.idle":"2025-11-01T23:04:21.949071Z","shell.execute_reply.started":"2025-11-01T23:04:20.793603Z","shell.execute_reply":"2025-11-01T23:04:21.948238Z"}},"outputs":[],"execution_count":null}]}