{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":112509,"databundleVersionId":14254895,"isSourceIdPinned":false,"sourceType":"competition"},{"sourceId":6158,"sourceType":"modelInstanceVersion","isSourceIdPinned":false,"modelInstanceId":4608,"modelId":2797}],"dockerImageVersionId":31153,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Catalysing Scientific Discovery with Kaggle - advanced\n## CSIRO Image2Biomass competition with the [GDG AI for Science](https://gdg.community.dev/gdg-ai-for-science-australia/) \n\nIn this more advanced Notebook we will continue to explore [CSIRO's Image2Biomass Prediction](https://www.kaggle.com/competitions/csiro-biomass/overview) competition using computer vision and machine learning. The goal of the competition is to estimate how much biomass (i.e grass for cows) there is in a set of images.\n\nReminder, we are tackling a Multi-Output Regression problem. We aren't predicting just one thing (like \"Grass vs No-grass\"). We are predicting 5 different numerical values representing 5 different biomass components. For each, we are predicting continuous numbers (regression), NOT categorical or binary classes.\n\nIn this Notebook we will use a pre-trained model to extract high-dimensional features from each image. And use that feature vector to build a typical machine learning classifier. \n\nSee our intro Notebook for a primer: [https://www.kaggle.com/code/astrobutter/gdg-kaggle-csiro-image2biomass-intro](https://www.kaggle.com/code/astrobutter/gdg-kaggle-csiro-image2biomass-intro)\n\n### Workflow:\n1. Load the data - read images and csv data.\n2. Use EfficientNet CV model to exctract key \"features\" or attributes (colour, height, shape, etc) in images.\n3. Train 5 Random Forest ML models (one per biomass class) to find how the features predict each of our 5 biomass classes.\n4. Use the trained ML model to predict on images we do not have the Biomass target amounts for.\n\n### What this workflow is missing\nOften, Kaggle datasets may be messy! We have no defensive programming in here - we are assuming our code will work on the unseen test images (submissions will fail if it does not). Often you must deal with this defensively by removing NaNs, using Try/Except blocks, etc. Thankfully, it appears like the dataset is consistent and THIS code works on THIS dataset.\n\n### Setting up the Environment\n\nAs before, we import our required libraries...","metadata":{}},{"cell_type":"code","source":"# For data manipulation\nimport os\nimport pandas as pd\nimport numpy as np \nfrom tqdm import tqdm\n\n# For machine learning model\nimport sklearn\nfrom sklearn.ensemble import RandomForestRegressor\n\n# For computer vision\nimport tensorflow as tf\n\nprint(\"Setup complete. TensorFlow version:\", tf.__version__, \"SciKit-LEarn version:\", sklearn.__version__)\n\ngpus = tf.config.list_physical_devices('GPU')\n\nif gpus:\n    print(f\"GPU Detected: {gpus[0]}\")\nelse:\n    print(\"No GPU Detected. Training will be very slow. Please enable an accelerator.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-26T23:37:14.419936Z","iopub.execute_input":"2025-11-26T23:37:14.420634Z","iopub.status.idle":"2025-11-26T23:37:29.625171Z","shell.execute_reply.started":"2025-11-26T23:37:14.420608Z","shell.execute_reply":"2025-11-26T23:37:29.624529Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1. Load the data\n\nFor this competition - we cannot use the internet. So all data and models we use must be available within the Kaggle platform. Here we will import pre-trained models and call them directly from our Kaggle's Virtual Machine file path.","metadata":{}},{"cell_type":"code","source":"# Load the datasets\nBASE_PATH = '/kaggle/input/csiro-biomass/'\nTRAIN_CSV_PATH = os.path.join(BASE_PATH, 'train.csv')\nTEST_CSV_PATH = os.path.join(BASE_PATH, 'test.csv')\n\ntrain_df = pd.read_csv(TRAIN_CSV_PATH)\ntest_df = pd.read_csv(TEST_CSV_PATH)\n\n# Wrangle the dataset to a \"wide\" format\ntrain_wide_df = train_df.pivot(\n    index='image_path',\n    columns='target_name',\n    values='target'\n).reset_index()\n\ndisplay(train_wide_df)\n\n# Set the path to our local CV model - import on the right hand side ->\nMODEL_PATH = \"/kaggle/input/efficientnetv2/keras/efficientnetv2_s_imagenet_classifier/2\"\n# Often, the size we smoosh the 2000x1000 pixel images into must fit the pre-trained model we use. \n# Here we don't strictly need to because of the architechure, but we must consider it.\nIMG_SIZE = 512\n# Set a random seed for reproducibility\nRANDOM_SEED = 999","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-26T23:38:09.600629Z","iopub.execute_input":"2025-11-26T23:38:09.601463Z","iopub.status.idle":"2025-11-26T23:38:09.625473Z","shell.execute_reply.started":"2025-11-26T23:38:09.601433Z","shell.execute_reply":"2025-11-26T23:38:09.624882Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 2. Define Pre-trained Model as a \"Feature Extractor\"\nWe will use a pre-trained Computer Vision model to \"see\" the images and extract meaningful \"features\". For this we use [EfficientNetV2](https://www.kaggle.com/models/keras/efficientnetv2), a specific architecture of a Convolutional Neural Network (CNN) designed to identify features and patterns within images, serving as a sophisticated mathematical framework for visual analysis. The architechture was trained on the massive [ImageNet](https://www.image-net.org/) database (millions of labeled everyday images) to create the weights and output classifier.\n\nWe don't want the final classifier layer though, just the features it has learned before it. The layer before the final 'Dense' classifier is the \"Global Average Pooling\" layer. So we can create a *new* model that shares the same input as the full model, but outputs the features from that pooling layer.\n\nWe are cutting off the original output \"head\" of EfficientNet. We don't care if EfficientNet thinks the image is a \"cat\" or a \"dog\" (its original purpose). We only want the feature-rich, internal description it generated right before it made that decision. It knows what edges, textures, shadows, and shapes, etc look like (in an abstract sense). We will use the understanding of those features to classify our grass!","metadata":{}},{"cell_type":"code","source":"# Load the model\nfull_model = tf.keras.models.load_model(MODEL_PATH)\n\n# 'full_model.layers[-2]' refers to the second-to-last layer.\nbase_model = tf.keras.Model(\n    inputs=full_model.input,\n    outputs=full_model.layers[-2].output\n)\n\n# We \"freeze\" the model weights as we are not training (fine-tuning) it,\n# we are only using it for inference (as a feature extractor).\nbase_model.trainable = False\n\nprint(f\"Model loaded. Feature vector output shape: {base_model.output_shape}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-26T23:45:16.275835Z","iopub.execute_input":"2025-11-26T23:45:16.276455Z","iopub.status.idle":"2025-11-26T23:45:31.838776Z","shell.execute_reply.started":"2025-11-26T23:45:16.276429Z","shell.execute_reply":"2025-11-26T23:45:31.838148Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Image Feature Engineering Function\n\nWe are simply getting the image into the correct format so we can run [Keras' predict method](https://keras.io/api/models/model_training_apis/#predict-method) - which runs the image through the model and outputs the feature vector (in our modified case).","metadata":{}},{"cell_type":"code","source":"# Now we define our new feature extraction function.\ndef extract_deep_features(image_path_suffix):\n    \"\"\"\n    Reads an image, resizes it, preprocesses it, and uses\n    the EfficientNet model to extract a 1D feature vector.\n    \"\"\"\n    # Create the full path to the image\n    full_path = os.path.join(BASE_PATH, image_path_suffix)\n    \n    # Load the image using Keras's helper function.\n    # This automatically resizes the image to our target_size. \n    image = tf.keras.preprocessing.image.load_img(full_path, target_size=(IMG_SIZE, IMG_SIZE))\n\n    # Convert from PIL image format to a numpy array\n    image_array = tf.keras.preprocessing.image.img_to_array(image)\n    \n    # Expand dimensions to create a \"batch\" of 1 from (512, 5122, 3) to (1, 512, 5122, 3)\n    image_batch = np.expand_dims(image_array, axis=0)\n    \n    # Pre-process the image (scales pixel values appropriately for EfficientNet)\n    # This is *required* for the model.\n    image_preprocessed = tf.keras.applications.efficientnet.preprocess_input(image_batch)\n    \n    # Get features. 'base_model.predict' runs the image through\n    # the network. This returns a 2D array, e.g., [[f1, f2, ...]]\n    # We set verbose=0 to avoid printing progress bars for every image.\n    features = base_model.predict(image_preprocessed, verbose=0)\n    \n    # Return just the 1D feature vector (the first item in the batch)\n    return features[0]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-26T23:49:52.994264Z","iopub.execute_input":"2025-11-26T23:49:52.994949Z","iopub.status.idle":"2025-11-26T23:49:53.000401Z","shell.execute_reply.started":"2025-11-26T23:49:52.994924Z","shell.execute_reply":"2025-11-26T23:49:52.999584Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"###  Apply Feature Extraction to All Training Images","metadata":{}},{"cell_type":"code","source":"# Now we run our *new* function on every image.\nprint(\"Starting training feature extraction... \")\n# This list will contain our 1D feature vectors (each of length 1280)\nimage_paths = train_wide_df['image_path'].values\nimage_features_list = [extract_deep_features(p) for p in tqdm(image_paths, desc=\"Extracting Features\")]\nprint(\"Feature extraction complete.\")\n\n# Get the number of features from our model (e.g., 1280)\nnum_features = base_model.output_shape[1]\n# Create column names like 'feature_0', 'feature_1', ... 'feature_1279'\nfeature_column_names = [f'feature_{i}' for i in range(num_features)]\n\n# Convert our list of lists into a new pandas DataFrame\nimage_features_df = pd.DataFrame(\n    image_features_list,\n    columns=feature_column_names\n)\n\n# Now, we combine our new features with our wide training data.\ntrain_data = pd.concat([train_wide_df, image_features_df], axis=1)\n\ndisplay(train_data)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-26T23:50:15.008962Z","iopub.execute_input":"2025-11-26T23:50:15.009415Z","iopub.status.idle":"2025-11-26T23:51:33.413425Z","shell.execute_reply.started":"2025-11-26T23:50:15.009391Z","shell.execute_reply":"2025-11-26T23:51:33.412644Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Clean Data and Define Features (X) and Targets (y)","metadata":{}},{"cell_type":"code","source":"# These are the five biomass components we need to predict \n#['Dry_Green_g', 'Dry_Dead_g', 'Dry_Clover_g', 'GDM_g', 'Dry_Total_g']\nTARGET_COLS = test_df['target_name'].unique()\nprint(\"Target columns:\", TARGET_COLS)\n\n# By convention, the feature matrix (our inputs) is called 'X'\nX = train_data[feature_column_names]\n\n# The target matrix (what we want to predict) is called 'y'\ny = train_data[TARGET_COLS]\n\nprint(\"Features (X):\")\ndisplay(X) # This will now be a very wide table\nprint(\"Targets (y):\")\ndisplay(y)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-26T23:51:41.269361Z","iopub.execute_input":"2025-11-26T23:51:41.269853Z","iopub.status.idle":"2025-11-26T23:51:41.301632Z","shell.execute_reply.started":"2025-11-26T23:51:41.269830Z","shell.execute_reply":"2025-11-26T23:51:41.301044Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# --- Feature Correlation ---\n# Do similar biomass values cluster together?\nplt.figure(figsize=(10, 8))\nscatter = plt.scatter(train_data['feature_355'], train_data['feature_645'], \n                      c=train_data['Dry_Total_g'], \n                      cmap='viridis', \n                      alpha=0.7)\nplt.colorbar(scatter, label='Total dry biomass (g) - Dry_Total_g')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-26T23:57:47.885840Z","iopub.execute_input":"2025-11-26T23:57:47.886454Z","iopub.status.idle":"2025-11-26T23:57:48.101282Z","shell.execute_reply.started":"2025-11-26T23:57:47.886430Z","shell.execute_reply":"2025-11-26T23:57:48.100681Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# --- Feature Correlation ---\n# # Let's see which specific abstract features correlate most with one of the target variables.\n\n# Calculate correlations between all features and the target\ncorrelations = train_data[feature_column_names].corrwith(train_data['Dry_Green_g'])\n\n# Get the top 5 most positive and top 5 most negative correlations\ntop_pos = correlations.nlargest(5)\ntop_neg = correlations.nsmallest(5)\n\nplt.figure(figsize=(10, 4))\ntop_features = pd.concat([top_pos, top_neg])\ntop_features.plot(kind='bar', color=['green']*5 + ['red']*5)\nplt.title('Feature Correlations with Dry_Total_g Biomass')\nplt.ylabel('Correlation Coefficient')\nplt.xlabel('Feature Index (Abstract)')\nplt.grid(axis='y')\nplt.show()\n\nprint(\"These are the specific 'neurons' in EfficientNet that are most useful.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-26T23:54:17.361398Z","iopub.execute_input":"2025-11-26T23:54:17.361996Z","iopub.status.idle":"2025-11-26T23:54:17.781105Z","shell.execute_reply.started":"2025-11-26T23:54:17.361974Z","shell.execute_reply":"2025-11-26T23:54:17.780427Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# --- Visualise the Features ---\n\n# Pick an example image path from the training set\n# sample_img_path = train_wide_df['image_path'].iloc[1]\n\n# sample_rows = df_train.sample(5)\n\nsample_img_path = train_wide_df['image_path'].sample(3)\n\nplt.figure(figsize=(20, 15))\nfor i, row in enumerate(sample_img_path):\n    # Show the original image\n    plt.subplot(3, 2, 2 * i + 1)\n    img = tf.keras.preprocessing.image.load_img(os.path.join(BASE_PATH, row))\n    plt.imshow(img)\n    plt.title(f\"Original Image: {row}\")\n    plt.axis('off')\n\n    # Extract the features\n    sample_features = extract_deep_features(row)\n    # We plot the 1280 numbers as a barcode/heatmap\n    \n    plt.subplot(3, 2, 2 * i + 2)\n    sns.heatmap([sample_features], cmap='viridis', cbar=True, yticklabels=False)\n    plt.title(f\"EfficientNet Representation\\n(1x{len(sample_features)} Vector)\")\n    plt.xlabel(\"Feature Index (0-1279)\")\n    \n    \nplt.tight_layout()\nplt.show()\n\nprint(f\"The raw image has been transformed into {len(sample_features)} abstract numbers!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-27T00:11:27.442191Z","iopub.execute_input":"2025-11-27T00:11:27.442817Z","iopub.status.idle":"2025-11-27T00:11:31.494865Z","shell.execute_reply.started":"2025-11-27T00:11:27.442797Z","shell.execute_reply":"2025-11-27T00:11:31.493899Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 3. Machine Learning model training\nNow we must convert those features of the image into an actual classifiction! Or, that is, a target value. Here we can use any classification algorithm, but for simplicity we can go with a [Random Forest](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html). Our approach is to train 5 different RF models - 1 for each target class, as each image requires 5 different target values for the solution.","metadata":{}},{"cell_type":"code","source":"print(\"Training models on ALL data for submission...\")\n\n# We will store our 5 final, trained models in this dictionary\nfinal_models = {}\n\nfor target_col in TARGET_COLS:\n    print(f\"Training for: {target_col}\")\n\n    # Get the full set of target values\n    y_target = y[target_col]\n\n    # Define a new model\n    model = RandomForestRegressor(n_estimators=100, random_state=RANDOM_SEED, n_jobs=-1)\n\n    # Train it on the full dataset (X and y)\n    model.fit(X, y_target)\n\n    # Store the trained model in our dictionary\n    final_models[target_col] = model\n\nprint(\"All 5 final models are ready!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-27T00:15:16.952877Z","iopub.execute_input":"2025-11-27T00:15:16.953836Z","iopub.status.idle":"2025-11-27T00:15:51.241949Z","shell.execute_reply.started":"2025-11-27T00:15:16.953810Z","shell.execute_reply":"2025-11-27T00:15:51.241142Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Many Kaggle competitions have \"run-time\" limits. For a \"big-data\" competition, or if you are using a process that takes a long time, we might save our models at this point. Then create a new Notebook, that simply reads in all our hard training and runs the predictions on the test data. This can save a lot of time. \n\n### 4. Make predictions on the test data!","metadata":{}},{"cell_type":"code","source":"# Get unique images to process - in some kaggle datasets there may be unseen classes in the test set!\nunique_test_images = test_df['image_path'].unique()\nprint(f\"Found {len(unique_test_images)} unique test images to process.\")\nprint(\"Now extracting test features...\")\ntest_features_list = [extract_deep_features(p) for p in unique_test_images]\n\n# Create the test feature DataFrame\ntest_features_df = pd.DataFrame(\n    test_features_list,\n    columns=feature_column_names # ['feature_0', 'feature_1', ...]\n)\n\n# Add the image_path back so we can map predictions\ntest_features_df['image_path'] = unique_test_images\n\n# Make Predictions on Test Data\npredictions_df = test_features_df[['image_path']].copy()\n# Get just the feature columns for prediction\ntest_X = test_features_df[feature_column_names]\n\nfor target_col, model in final_models.items():\n    print(f\"Predicting for: {target_col}\")\n    predictions = model.predict(test_X)\n    predictions_df[target_col] = predictions\n\nprint(\"\\nPredictions for all unique images (wide format):\")\ndisplay(predictions_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-27T00:15:51.243169Z","iopub.execute_input":"2025-11-27T00:15:51.243642Z","iopub.status.idle":"2025-11-27T00:15:51.648800Z","shell.execute_reply.started":"2025-11-27T00:15:51.243624Z","shell.execute_reply":"2025-11-27T00:15:51.648167Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# --- Model Inspection ---\n\n# Let's visualise the results for just one target to keep it simple\ntarget_to_viz = 'Dry_Total_g'\nmodel = final_models[target_to_viz]\ny_true = y[target_to_viz]\ny_pred = model.predict(X) # Predicting on training data for diagnostic\n\nfig, ax = plt.subplots(1, 2, figsize=(16, 6))\n\n# Plot A: Actual vs Predicted (Goodness of Fit)\n# Ideally, all points lie on the red diagonal line\nax[0].scatter(y_true, y_pred, alpha=0.5)\nax[0].plot([y_true.min(), y_true.max()], [y_true.min(), y_true.max()], 'r--', lw=2)\nax[0].set_xlabel('Actual Biomass')\nax[0].set_ylabel('Predicted Biomass')\nax[0].set_title(f'Goodness of Fit: {target_to_viz}')\nr2 = sklearn.metrics.r2_score(y_true, y_pred)\nax[0].text(0.05, 0.9, f'$R^2 = {r2:.2f}$', transform=ax[0].transAxes, fontsize=12)\n\n# Plot B: Feature Importances\n# Which of the 1280 features mattered most?\nimportances = model.feature_importances_\nindices = np.argsort(importances)[-20:] # Top 20 features\n\nax[1].barh(range(len(indices)), importances[indices], align='center')\nax[1].set_yticks(range(len(indices)))\nax[1].set_yticklabels([f\"Feature {i}\" for i in indices])\nax[1].set_xlabel('Relative Importance')\nax[1].set_title(f'Top 20 Most Important EfficientNet Features for {target_to_viz}')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-27T00:19:06.011931Z","iopub.execute_input":"2025-11-27T00:19:06.012522Z","iopub.status.idle":"2025-11-27T00:19:06.505191Z","shell.execute_reply.started":"2025-11-27T00:19:06.012500Z","shell.execute_reply":"2025-11-27T00:19:06.504539Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Format and create submission file!","metadata":{}},{"cell_type":"code","source":"# 'pd.melt' is the opposite of 'pivot'. It turns columns into rows.\npredictions_long_df = predictions_df.melt(\n    id_vars=['image_path'],\n    value_vars=TARGET_COLS,\n    var_name='target_name',     \n    value_name='target'        \n)\n\n# Now, we just need to merge this with the original test_df to get the 'sample_id'\n# and keep only the two columns ('sample_id', 'target') required for submission\nsubmission_df = pd.merge(\n    test_df[['sample_id', 'image_path', 'target_name']],\n    predictions_long_df,\n    on=['image_path', 'target_name']\n)[['sample_id', 'target']]\n\ndisplay(submission_df)\n\nsubmission_df.to_csv('submission.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-27T00:15:51.649423Z","iopub.execute_input":"2025-11-27T00:15:51.649601Z","iopub.status.idle":"2025-11-27T00:15:51.675167Z","shell.execute_reply.started":"2025-11-27T00:15:51.649587Z","shell.execute_reply":"2025-11-27T00:15:51.674684Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Optional: Model Validation","metadata":{}},{"cell_type":"code","source":"# We split our data to check if our features are any good.\n# X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, test_size=0.2, random_state=3)\n\n# print(f\"Training features shape: {X_train.shape}\")\n# print(f\"Validation features shape: {X_val.shape}\")\n\n# # We will store our validation scores in a dictionary\n# validation_scores = {}\n\n# # We loop over each target column and train a *separate* model for it.\n# for target_col in TARGET_COLS:\n#     print(f\"--- Validating model for: {target_col} ---\")\n\n#     # Get the specific target values for this model\n#     y_train_target = y_train[target_col]\n#     y_val_target = y_val[target_col]\n\n#     model = sklearn.ensemble.RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1)\n\n#     # Train the model on the training data\n#     model.fit(X_train, y_train_target)\n\n#     # Make predictions on the validation data\n#     predictions = model.predict(X_val)\n\n#     # Calculate the error (Root Mean Squared Error)\n#     rmse = np.sqrt(sklearn.metrics.mean_squared_error(y_val_target, predictions))\n#     print(f\"Validation RMSE for {target_col}: {rmse:.2f} g\")\n#     validation_scores[target_col] = rmse\n\n# print(\"\\nValidation complete.\")","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}