{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":64733,"databundleVersionId":7762802,"sourceType":"competition"}],"dockerImageVersionId":30673,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# import numpy as np # linear algebra\n# import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     print(len(filenames))\n#     print(dirname)\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-03-25T22:45:39.301911Z","iopub.execute_input":"2024-03-25T22:45:39.303143Z","iopub.status.idle":"2024-03-25T22:54:36.829512Z","shell.execute_reply.started":"2024-03-25T22:45:39.303106Z","shell.execute_reply":"2024-03-25T22:54:36.827885Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **INPUT DATA**\n\n1. **EnvironmentalRasters**\n\nThese folders contain test and train CSV files containing data on vairous different environmental factors\n\n* **Climate**\n* * Contains two folders for the monthly climate values as well as the average climate values\n\n* **Elevation**\n* * only contains one columns worth of data for each survey with the elevation of the sample\n\n* **Human Footprint**\n* * \n\n* **LandCover**\n* * Only contains one columns worth of data for each survey representing the landcover for each survey\n\n* **SoilGrids**\n* * for each survey it gives a breakdown of what type of natural materials are present in each sample (sand, silt, nitrogen etc...)\n\n**OPINION:**\nfor each survey initially we should concatenate each of the environmental raster data tables. Each table may need to be preprocessed differently as we cannot insure the consistency of the data between tables, but I suggest we concatenate each of the individual tables into one larger table\n- See final observation 1\n\n\n2. **Landsat_time_series**\n\nThere are 6 different files within the landsat_time_series folder. They represent the amount of different types of light that was present in the surveys for each season from 2000-2020\nThe 6 different types of light are. Each file contains 84 columns worth of seasonal data from 2000-2020\n\n* Blue - Wavelength 450-520nm\n* Green - Wavelength 500-600nm\n* Red - Wavelength 620-750nm\n* NIR (Near infrared) - Wavelength 750-1400nm\n* Swir1 (Short-wave infrared) - Wavelength 1500-1750nm\n* Swir2 (Short-wave infrared) - Wavelength 2000-2300nm\n\n**OPINION:** \nWe should probably learn more about what each of these 6 wavelengths of light represent in terms of how they effect vegetation land cover. We will also need to do a bit of work on how to best handle this timeseries data before feeding it into a model.\n\n\n3. **Satellite Patches**\n\nThere are two folders of satellite patches present, normal RGB photos which contain photos of each sample site in full colour. There is also NIR photos of the same sample site. Each photo is a .jpeg photo\n\nthere is a weird convention for finding the photo of each survey site.  \nTo load the data -> '…/CD/AB/XXXXABCD.jpeg'. For example, the image location for the surveyId 3013575 is \"./75/85/3018575.jpeg\"\n\n**OPINION:** \nWe would need to develop a good preprocessing pipeline to handle these jpeg images before feeding them into our model. Would it be best to \n\n\n4. **TimeSeries-Cubes**\n\nThese files contain preprocessed and aggregated satelitte data which has been converted into a 3d Tensor. There are two folders in here one which represents bioclimatic monthly data, the other is a landsat timeseries.\n\nThe two types of files present in the Timeseries-Cubes folder are:\n* GLC24-PA-test-landsat_time_series_1001507_cube: shape=[6,4,21]\n* GLC24-PA-test-bioclimatic_monthly_1001507_cube: shape=[4,19,12]\n\n\n\n**ASSUMPTIONS**\nI am not fully sure what each dimension represents for both of these timeseries tensors. But I would assume...\n* GLC24-PA-test-landsat_time_series_1001507_cube: shape=[6,4,21]\n* * 6 - each one of these would represent a different one of the 6 different types of light(red, blue, nir, swir1...)\n* * 4 - for each of the different seasons\n* * 21 - for each year from 2000-2020\n* GLC24-PA-test-bioclimatic_monthly_1001507_cube: shape=[4,19,12]\n* * 4 - Raster-type (What this actually means I'm not quite sure\n* * 19 - Year\n* * 12 - Month\n\n**QUESTIONS**\n* There is an index at the very end of all tensors within this folder. I am not sure what this tensor represents. It is definitely a way to locate which sample the timeseries represents, and it might be the same as the indexing for the satellite patches. But i am not sure\n\n\n5. **MetaData**\n\nThe metadata is split into two different files: Presence-Absence(PA), and Presence-Occurance(PO)\n\nPA surveys - Contains 1.5 million samples (rows) of data \nColumns:\n* Lon - Longitude coordinate of the survey location.\n* Lat - Latitude coordinate of the survey location.\n* Year - Year survey was taken\n* geoUncertaintyInM - Geographic uncertainty in meters, which may indicate the level of uncertainty associated with the recorded coordinates. **IMPORTANT COLUMN**\n* areaInM2 - Area covered by the survey in square meters, which may provide information about the spatial extent of the survey.\n* Region - Region where the survey was conducted\n* Country - Country where the survey was conducted\n* speciesId - Identifier for the species observed during the survey\n* surveyId - Identifier for the survey\n\nPO surveys - contains 5 million samples (rows) of data\nColumns:\n* Year - Year survey took place\n* Month - Month survey took place\n* Day - Day survey took place\n* lat - Latitude coordinate of the survey location.\n* lon - Longitude coordinate of the survey location.\n* geoUncertaintyInM - Geographic uncertainty in meters, which may indicate the level of uncertainty associated with the recorded coordinates. **IMPORTANT COLUMN**\n* taxonRank - gives us an indication of the taxonomic rank of the species (species, genus family etc...)\n* date - date survey took place\n* dayOfYear - day of the year the survey took place (does this matter???)\n* speciesId - SpeciesId that we found a positive sample of\n* surveyId - ID of the survey (can help us link everything together)\n\nThis data contains all the positive occurances of species at a certain location, at a certain time. They have provided the PA survey so that we can address false absences in the PA data (to be honest what this means or how it would avoid the false abscenses I'm not sure. There is no specific column within the PA dataset that indicates if a species IS present in the sample)\n\n**QUESTIONS**\n1. I believed that a Presence-Absence survey would have some way of indicating if the species identified in the sample was present or absent for the specific survey. Does this mean we are to assume that the data in the PA survey will contain only positive sample of species found during surveys? \n2. I am not really sure about the difference between these two surveys: to my understanding the PO survey contains all the positive occurances of samples - but then what is the point of the PA survey, as it doesn't have any columns that explicitly state the presence or abscence of a species in a sample.\n\n\n** **\n\n**FINAL OBSERVATIONS** \nThere is so much we WILL have to do with this data before we can even get to the point of building a model\n1. The first thing we will have to do is traceback through the *timeSeries-Cubes* dataframe and find out the bioclimatic conditions for each positive occurance of a species in the dataset (at a certain time of the year)\n* This will give us an indication of the conditions and location that this species can grow in so if similar conditions are met at the same place, we should expect the same species to appear\n* I mentioned before it might be useful to concatenate all the columns into one large table for the environmental conditions but it will probably be usefull to traceback through each surveys (in metadata) date and location before pulling the relevent data from the timeseries tensor (timeSeries-Cubes)\n\n2. There is also a notebook (https://www.kaggle.com/code/gobyeonggeon/preprocess-visualize-spatial-data-eda-xgb/notebook) which contains another folder of data called climateclef and contains all the .tif (Geotiff) files - I do not know where it was found and haven't been able to fully investigate the data in this. However I understand it to be raster data of the \"EnvironmentalRasters\" folder, and the data found in the EnvironmentalRasters folder is just a summary of the full raster.\n* We could initially build a baseline model which uses this summary csv data before trying to improve our model by making it work for the full climateclef data.\n\n\n**PROPOSED SOLUTION**\nthis is a very basic solution that doesn't take into account things like the full rasters and image data.\n\nwe will need a table which contains a column for each species, initially fill with 0's - This will be what we are predicting for. We will need to fill this table out for each new training sample\n\nFor each sample in the PO_metadata survey we will need to loop through the timeseries data and build out a list of all the environmental conditions, landsat data and location into a  table which will be the table we train our model on. \nFor each training sample we then fill the species table with the species that was present. We can then train a model to try predict which species would be present given a certain location and environmental conditions\n\n- In know this is probably badly worded and not the ideal solution (it might not work) but I'm begining to think in the right direction which is more than I could say about yesterday\n\n**POTENTIAL PROBLEMS**\nAccording to the 2023 GeoCLEF winner (https://ceur-ws.org/Vol-3497/paper-181.pdf) the main challenge they found was that for 98% of data only had a single positive label - we would need to investigate this (if it is the case for geoCLEF2024) and discuss potential solutions\n- 2023 winner was able to mitage this challenge by three step learning method. They first trained the model on just the PA data with binary-cross entropy loss. Then they continue the training with PA data and changed the loss to CE loss instead. This allowed it to learn a single class per input sample. They then continued to train the models from the second step on the PO data with BCE loss utilising the features learned from the second step","metadata":{}},{"cell_type":"code","source":"import torch\n\n# Load a .pt file\nfile_path = \"/kaggle/input/geolifeclef-2024/TimeSeries-Cubes/GLC24-PA-test-bioclimatic_monthly/GLC24-PA-test-bioclimatic_monthly_1001507_cube.pt\"\nfile_path_ts = '/kaggle/input/geolifeclef-2024/TimeSeries-Cubes/GLC24-PA-test-landsat_time_series/GLC24-PA-test-landsat_time_series_1001507_cube.pt'\ndata = torch.load(file_path)\ndata_ts = torch.load(file_path_ts)\n\n# Print information about the loaded data\nprint(\"Data Shape:\", data.shape)\nprint('Time Series data shape: ', data_ts.shape)\ntemp = data[0,:,:]\nprint(\"Data Contents:\")\nprint(temp)","metadata":{"execution":{"iopub.status.busy":"2024-03-26T09:27:33.607801Z","iopub.execute_input":"2024-03-26T09:27:33.608693Z","iopub.status.idle":"2024-03-26T09:27:33.628843Z","shell.execute_reply.started":"2024-03-26T09:27:33.608658Z","shell.execute_reply":"2024-03-26T09:27:33.628167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\nPO = pd.read_csv('/kaggle/input/geolifeclef-2024/GLC24_P0_metadata_train.csv')\nPA = pd.read_csv('/kaggle/input/geolifeclef-2024/GLC24_PA_metadata_train.csv')","metadata":{"execution":{"iopub.status.busy":"2024-03-26T10:08:15.130492Z","iopub.execute_input":"2024-03-26T10:08:15.131114Z","iopub.status.idle":"2024-03-26T10:08:23.956874Z","shell.execute_reply.started":"2024-03-26T10:08:15.131075Z","shell.execute_reply":"2024-03-26T10:08:23.955769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(PA.shape)\nprint(PO.shape)\nprint(PA.columns)\nprint(PO.columns)","metadata":{"execution":{"iopub.status.busy":"2024-03-26T10:13:06.542718Z","iopub.execute_input":"2024-03-26T10:13:06.543614Z","iopub.status.idle":"2024-03-26T10:13:06.549087Z","shell.execute_reply.started":"2024-03-26T10:13:06.543579Z","shell.execute_reply":"2024-03-26T10:13:06.548029Z"},"trusted":true},"execution_count":null,"outputs":[]}]}