{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":112509,"databundleVersionId":14254895,"sourceType":"competition"}],"dockerImageVersionId":31153,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Catalysing Scientific Discovery with Kaggle\n## CSIRO Image2Biomass competition with the [GDG AI for Science](https://gdg.community.dev/gdg-ai-for-science-australia/) \n\nKaggle Notebooks are deeply integrated with the platform. They automatically enable access to Kaggle-hosted datasets & models, plus we can leverage compute & GPU/TPU resources, and make submissions to competitions all from a Python (or R) environment.\n\nIn this introductory Notebook we will explore CSIRO's Image2Biomass Prediction competition. The goal of the competition is to estimate how much biomass (i.e grass for cows) there is a set of images.\n\n### Workflow:\n1. Load the data - read images and csv data.\n2. Exploratory Data Analysis - understand the data with some plots and thinking\n3. Build a model - constuct a workflow to predict unseen images\n4. Apply the model to the test images\n   \n### Setting up the Environment\n\nKaggle Notebooks come with a whole bunch of preinstalled packages. If you need another package you can `!pip install` it, but be careful, if you want to submit this Notebook to a competition you may have to \"turn internet off\".","metadata":{}},{"cell_type":"code","source":"# For data wrangling\nimport pandas as pd \n\n# For visualisation and image handling\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom PIL import Image","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-11-20T04:19:00.176952Z","iopub.execute_input":"2025-11-20T04:19:00.177283Z","iopub.status.idle":"2025-11-20T04:19:00.593978Z","shell.execute_reply.started":"2025-11-20T04:19:00.177257Z","shell.execute_reply":"2025-11-20T04:19:00.592773Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1. Load the data\n\nWhen working with a competition the data will be accessible from the path `/kaggle/input`. Additional models and datasets can be loaded into this path as required! Note: you can use a `!` prefix in a Notebook code cell to execute typical Unix commands `!ls -l /kaggle/input/csiro-biomass`.\n\nLet's read in the competition data and see what we are working with - step 1 for any exploratory data analysis and for solving any science problem.","metadata":{}},{"cell_type":"code","source":"# Load the datasets\nBASE_PATH = '/kaggle/input/csiro-biomass/'\nTRAIN_CSV_PATH = BASE_PATH + 'train.csv'\nTEST_CSV_PATH = BASE_PATH + 'test.csv'\n\ndf_train = pd.read_csv(TRAIN_CSV_PATH)\ndf_test = pd.read_csv(TEST_CSV_PATH)\n\nprint(TRAIN_CSV_PATH)\ndisplay(df_train)\n\nprint(TEST_CSV_PATH)\ndisplay(df_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-20T04:19:00.595817Z","iopub.execute_input":"2025-11-20T04:19:00.596254Z","iopub.status.idle":"2025-11-20T04:19:00.640914Z","shell.execute_reply.started":"2025-11-20T04:19:00.596230Z","shell.execute_reply":"2025-11-20T04:19:00.639649Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 2. Exploratory Data Analyis - visualise the data\n\nWe could jump straight into machine learning, training, etc, but let's explore our data more. Many ways to do this - but ultimately we want to understand our problem!","metadata":{}},{"cell_type":"code","source":"# Visualise some images\n# sample_rows = df_train.iloc[3:8]\nsample_rows = df_train.sample(5)\n\nplt.figure(figsize=(20, 5))\nfor i, (idx, row) in enumerate(sample_rows.iterrows()):\n    # Construct the full path to the images\n    full_path = BASE_PATH + row['image_path']\n    \n    img = Image.open(full_path)\n    plt.subplot(1, 5, i + 1)\n    plt.imshow(img)\n    plt.title(f\"\"\"\n        sample_id: {row['sample_id']}\n        image_path: {row['image_path']}\n        Sampling_Date: {row['Sampling_Date']} State: {row['State']}\n        Species: {row['Species']}\n        Pre_GSHH_NDVI: {row['Pre_GSHH_NDVI']}\n        Height_Ave_cm: {row['Height_Ave_cm']} cm\n        target_name: {row['target_name']}\n        target: {row['target']:.1f} g\n        \"\"\", loc='left', fontsize=8)\n    plt.axis('off')\n\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-20T04:45:18.499707Z","iopub.execute_input":"2025-11-20T04:45:18.500070Z","iopub.status.idle":"2025-11-20T04:45:20.658215Z","shell.execute_reply.started":"2025-11-20T04:45:18.500046Z","shell.execute_reply":"2025-11-20T04:45:20.657083Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.scatterplot(data=df_train, x='Height_Ave_cm', y='target', hue='target_name', alpha=0.5)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-20T04:19:03.514131Z","iopub.execute_input":"2025-11-20T04:19:03.514932Z","iopub.status.idle":"2025-11-20T04:19:03.951113Z","shell.execute_reply.started":"2025-11-20T04:19:03.514893Z","shell.execute_reply":"2025-11-20T04:19:03.950022Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We have image data and we have some meta-data about each image. Think about what we are trying to solve (how much pasture is in each image) and start thinking of many different ways we could try to solve this problem.\n\nFor (most) Machine Leanring problems we are given a \"target\" predictor feature. This is the value we want to predict to solve the challenge! The more accurate we can predict it, the better score we will get in to compeition (and hopefully the more scientifically useful the workflow will be). Here, we are given pictures of grasses, our challenge is to build a workflow to take the information in those images to predict the amount (mass in grams) of 5 different classes of \"biomass components\" which are:\n\n\n* Dry green vegetation (excluding clover)\n* Dry dead material\n* Dry clover biomass\n* Green dry matter (GDM)\n* Total dry biomass\n\nA human expert has performed pixel-wise estimation from the series of training images for us. Again, most ML problems require this fundamental training set that have a one or more predictor-variables (features) and a target we are trying to predict. To make sure we build a useful and robust workflow we are also given a set of \"test\" images that we DO NOT know the answer to (i.e. the target biomass component amounts). And this is what is used to score our submission!\n\n### 3. Build a model \n\nOnce we have a good idea about our data and have the data into a nice format we can implement whatever Machine Learning algorithm and Models that are approriate for our probelm (image + meta data input -> multi-class regression class output).\n\nFor an example we will build the most elegant and simple model I have seen (credit to krabhijeet for this). Let's take the average value of all training images, and apply that to all the test images.","metadata":{}},{"cell_type":"code","source":"# Calculate the average of 'target' grouped by 'target_name'\ntarget_by_name = df_train.groupby('target_name')['target'].mean().reset_index()\ndisplay(target_by_name)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-20T04:19:03.952154Z","iopub.execute_input":"2025-11-20T04:19:03.952436Z","iopub.status.idle":"2025-11-20T04:19:03.966760Z","shell.execute_reply.started":"2025-11-20T04:19:03.952414Z","shell.execute_reply":"2025-11-20T04:19:03.965294Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 4. Apply the model\n\nHere we are simply just saying that every single test image we could possibly ever face has the exact values of the average biomass from all the training set. A completley naieve, but great baseline approach to begin any Kaggle challenge. Other simple metrics we could benchmark could be median, all 1's, all 0's, etc.","metadata":{}},{"cell_type":"code","source":"df_submission = df_test.merge(\n    target_by_name,\n    on='target_name',\n    how='left'\n)[[\"sample_id\",\"target\"]]\n\ndisplay(df_submission)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-20T04:19:03.967938Z","iopub.execute_input":"2025-11-20T04:19:03.968380Z","iopub.status.idle":"2025-11-20T04:19:03.987024Z","shell.execute_reply.started":"2025-11-20T04:19:03.968356Z","shell.execute_reply":"2025-11-20T04:19:03.985803Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The final step is producing a submission file. In this case it must be called \"submission.csv\" and have exactly two columns with headings \"sample_id\" and \"target\". Pandas did most of the formatting in the previous step.","metadata":{}},{"cell_type":"code","source":"df_submission.to_csv('submission.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-20T04:19:03.988352Z","iopub.execute_input":"2025-11-20T04:19:03.989066Z","iopub.status.idle":"2025-11-20T04:19:04.011077Z","shell.execute_reply.started":"2025-11-20T04:19:03.989026Z","shell.execute_reply":"2025-11-20T04:19:04.009790Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}