{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":64733,"databundleVersionId":8171035,"sourceType":"competition"}],"dockerImageVersionId":30699,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Simple baseline with Landsat and Bioclimatic Cubes + Sentinel images [0.31626]\n\nFollowing the three provided baselies with different modalities, we have provide a multimodal approch based on \"siamiese\" network with multiple inputs and simple shared \"decoder\". The links for the separated baselines are as follows:\n\n- [Baseline with Bioclimatic Cubes [0.25784]](https://www.kaggle.com/code/picekl/baseline-with-bioclimatic-cubes-0-25784)\n- [Baseline with Landsat Cubes [0.26424]](https://www.kaggle.com/code/picekl/baseline-with-landsat-cubes-0-26424)\n- [Baseline with Sentinel Images [0.23594]](https://www.kaggle.com/code/picekl/baseline-with-sentinel-images-0-23594)\n\n**Considering the significant extent for enhancing performance of this baseline, we encourage you to experiment with various techniques, architectures, losses, etc.**\n\n#### **Have Fun!**","metadata":{}},{"cell_type":"markdown","source":"# Data description\n\n## Landsat time series\n\nSatellite time series data includes over 20 years of Landsat satellite imagery extracted from [Ecodatacube](https://stac.ecodatacube.eu/).\nThe data was acquired through the Landsat satellite program and pre-processed by Ecodatacube to produce raster files scaled to the entire European continent and projected into a unique CRS.\n\nSince the original rasters require a high amount of disk space, we extracted the data points from each spectral band corresponding to all PA and PO locations (i.e., GPS coordinates) and aggregated them in (i) CSV files and (ii) data cubes as tensor objects. Each data point corresponds to the mean value of Landsat's observations at the given location for three months before the given time; e.g., the value of a time series element under column 2012_4 will represent the mean value for that element from October 2012 to December 2012.\n\nIn this notebook, we will work with just the cubes. The cubes are structured as follows.\n**Shape**: `(n_bands, n_quarters, n_years)` where:\n- `n_bands` = 6 comprising [`red`, `green`, `blue`, `nir`, `swir1`, `swir2`]\n- `n_quarters` = 4 \n    - *Quarter 1*: December 2 of previous year until March 20 of current year (winter season proxy),\n    - *Quarter 2*: March 21 until June 24 of current year (spring season proxy),\n    - *Quarter 3*: June 25 until September 12 of current year (summer season proxy),\n    - *Quarter 4*: September 13 until December 1 of current year (fall season proxy).\n- `n_years` = 21 (ranging from 2000 to 2020)\n\nThe datacubes can simply be loaded as tensors using PyTorch with the following command :\n\n```python\nimport torch\ntorch.load('path_to_file.pt')\n```\n\n**References:**\n- *Traceability (lineage): This dataset is a seasonally aggregated and gapfilled version of the Landsat GLAD analysis-ready data product presented by Potapov et al., 2020 ( https://doi.org/10.3390/rs12030426 ).*\n- *Scientific methodology: The Landsat GLAD ARD dataset was aggregated and harmonized using the eumap python package (available at https://eumap.readthedocs.io/en/latest/ ). The full process of gapfilling and harmonization is described in detail in Witjes et al., 2022 (in review, preprint available at https://doi.org/10.21203/rs.3.rs-561383/v3 ).*\n- *Ecodatacube.eu: Analysis-ready open environmental data cube for Europe (https://doi.org/10.21203/rs.3.rs-2277090/v3).*\n\n\n## Bioclimatic time series\n\nThe Bioclimatic Cubes are created from **four** monthly GeoTIFF CHELSA (https://chelsa-climate.org/timeseries/) time series climatic rasters with a resolution of 30 arc seconds, i.e. approximately 1km. The four variables are the precipitation (pr), maximum- (taxmax), minimum- (tasmin), and mean (tax) daily temperatures per month from January 2000 to June 2019. We provide the data in three forms: (i) raw rasters (GeoTiff images), (ii) CSV file with pre-extracted values for each location, i.e., surveyId, and (iii) data cubes as tensor object (.pt).\n\nIn this notebook, we will work with just the cubes. The cubes are structured as follows.\n**Shape**: `(n_year, n_month, n_bio)` where:\n- `n_year` = 19 (ranging from 2000 to 2018)\n- `n_month` = 12 (ranging from January 01 to December 12)\n- `n_bio` = 4 comprising [`pr` (precipitation), `tas` (mean daily air temperature), `tasmin`, `tasmax`]\n\nThe datacubes can simply be loaded as tensors using PyTorch with the following command :\n\n```python\nimport torch\ntorch.load('path_to_file.pt')\n```\n\n**References:**\n- *Karger, D.N., Conrad, O., Böhner, J., Kawohl, T., Kreft, H., Soria-Auza, R.W., Zimmermann, N.E., Linder, P., Kessler, M. (2017): Climatologies at high resolution for the Earth land surface areas. Scientific Data. 4 170122. https://doi.org/10.1038/sdata.2017.122*\n\n- *Karger D.N., Conrad, O., Böhner, J., Kawohl, T., Kreft, H., Soria-Auza, R.W., Zimmermann, N.E, Linder, H.P., Kessler, M. Data from: Climatologies at high resolution for the earth’s land surface areas. Dryad Digital Repository. http://dx.doi.org/doi:10.5061/dryad.kd1d4*\n\n\n## Sentinel Image Patches\n\nThe Sentinel Image data was acquired through the Sentinel2 satellite program and pre-processed by [Ecodatacube](https://stac.ecodatacube.eu/) to produce raster files scaled to the entire European continent and projected into a unique CRS. We filtered the data in order to pick patches from each spectral band corresponding to a location ((lon, lat) GPS coordinates) and a date matching that of our occurrences', and split them into JPEG files (RGB in 3-channels .jpeg files and NIR in single-channel .jpeg files) with a 128x128 resolution. The images were converted from sentinel uint15 to uint8 by clipping data pixel values over 10000 and applying a gamma correction of 2.5.\n\nThe data can simply be loaded using the following method:\n\n```python\ndef construct_patch_path(output_path, survey_id):\n    \"\"\"Construct the patch file path based on survey_id as './CD/AB/XXXXABCD.jpeg'\"\"\"\n    path = output_path\n    for d in (str(survey_id)[-2:], str(survey_id)[-4:-2]):\n        path = os.path.join(path, d)\n\n    path = os.path.join(path, f\"{survey_id}.jpeg\")\n\n    return path\n```\n\n**References:**\n- *Traceability (lineage): The dataset was produced entirely by mosaicking and seasonally aggregating imagery from the Sentinel-2 Level-2A product (https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-2-msi/product-types/level-2a)*\n- *Ecodatacube.eu: Analysis-ready open environmental data cube for Europe (https://doi.org/10.21203/rs.3.rs-2277090/v3)*","metadata":{"execution":{"iopub.status.busy":"2024-05-01T13:30:07.053659Z","iopub.execute_input":"2024-05-01T13:30:07.054038Z","iopub.status.idle":"2024-05-01T13:30:07.058148Z","shell.execute_reply.started":"2024-05-01T13:30:07.054008Z","shell.execute_reply":"2024-05-01T13:30:07.057269Z"}}},{"cell_type":"code","source":"import os\nimport torch\nimport tqdm\nimport numpy as np\nimport pandas as pd\nimport torchvision.models as models\nimport torchvision.transforms as transforms\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\nfrom sklearn.metrics import precision_recall_fscore_support","metadata":{"ExecuteTime":{"end_time":"2024-04-30T21:25:07.29831Z","start_time":"2024-04-30T21:25:05.354584Z"},"tags":[],"execution":{"iopub.status.busy":"2024-05-06T09:06:57.259571Z","iopub.execute_input":"2024-05-06T09:06:57.259928Z","iopub.status.idle":"2024-05-06T09:07:04.755868Z","shell.execute_reply.started":"2024-05-06T09:06:57.259898Z","shell.execute_reply":"2024-05-06T09:07:04.754947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare custom dataset loader\n\nWe have to slightly update the Dataset to provide the relevant data in the appropriate format.","metadata":{}},{"cell_type":"code","source":"from PIL import Image\n\ndef construct_patch_path(data_path, survey_id):\n    \"\"\"Construct the patch file path based on plot_id as './CD/AB/XXXXABCD.jpeg'\"\"\"\n    path = data_path\n    for d in (str(survey_id)[-2:], str(survey_id)[-4:-2]):\n        path = os.path.join(path, d)\n\n    path = os.path.join(path, f\"{survey_id}.jpeg\")\n\n    return path\n\nclass TrainDataset(Dataset):\n    def __init__(self, bioclim_data_dir, landsat_data_dir, sentinel_data_dir, metadata, transform=None):\n        self.transform = transform\n        self.sentinel_transform = transform = transforms.Compose([\n            transforms.ToTensor(),\n            transforms.Normalize(mean=(0.5, 0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5, 0.5)),\n        ])\n      \n        self.bioclim_data_dir = bioclim_data_dir\n        self.landsat_data_dir = landsat_data_dir\n        self.sentinel_data_dir = sentinel_data_dir\n        self.metadata = metadata\n        self.metadata = self.metadata.dropna(subset=\"speciesId\").reset_index(drop=True)\n        self.metadata['speciesId'] = self.metadata['speciesId'].astype(int)\n        self.label_dict = self.metadata.groupby('surveyId')['speciesId'].apply(list).to_dict()\n        \n        self.metadata = self.metadata.drop_duplicates(subset=\"surveyId\").reset_index(drop=True)\n\n    def __len__(self):\n        return len(self.metadata)\n\n    def __getitem__(self, idx):\n        \n        survey_id = self.metadata.surveyId[idx]\n        \n        landsat_sample = torch.nan_to_num(torch.load(os.path.join(self.landsat_data_dir, f\"GLC24-PA-train-landsat-time-series_{survey_id}_cube.pt\")))\n        bioclim_sample = torch.nan_to_num(torch.load(os.path.join(self.bioclim_data_dir, f\"GLC24-PA-train-bioclimatic_monthly_{survey_id}_cube.pt\")))\n\n        rgb_sample = np.array(Image.open(construct_patch_path(self.sentinel_data_dir, survey_id)))\n        nir_sample = np.array(Image.open(construct_patch_path(self.sentinel_data_dir.replace(\"rgb\", \"nir\").replace(\"RGB\", \"NIR\"), survey_id)))\n        sentinel_sample = np.concatenate((rgb_sample, nir_sample[...,None]), axis=2)\n\n        species_ids = self.label_dict.get(survey_id, [])  # Get list of species IDs for the survey ID\n        label = torch.zeros(num_classes)  # Initialize label tensor\n        for species_id in species_ids:\n            label_id = species_id\n            label[label_id] = 1  # Set the corresponding class index to 1 for each species\n        \n        if isinstance(landsat_sample, torch.Tensor):\n            landsat_sample = landsat_sample.permute(1, 2, 0)  # Change tensor shape from (C, H, W) to (H, W, C)\n            landsat_sample = landsat_sample.numpy()  # Convert tensor to numpy array\n            \n        if isinstance(bioclim_sample, torch.Tensor):\n            bioclim_sample = bioclim_sample.permute(1, 2, 0)  # Change tensor shape from (C, H, W) to (H, W, C)\n            bioclim_sample = bioclim_sample.numpy()  # Convert tensor to numpy array   \n        \n        if self.transform:\n            landsat_sample = self.transform(landsat_sample)\n            bioclim_sample = self.transform(bioclim_sample)\n            sentinel_sample = self.sentinel_transform(sentinel_sample)\n\n        return landsat_sample, bioclim_sample, sentinel_sample, label, survey_id\n\nclass TestDataset(TrainDataset):\n    def __init__(self, bioclim_data_dir, landsat_data_dir, sentinel_data_dir, metadata, transform=None):\n        self.transform = transform\n        self.sentinel_transform = transform = transforms.Compose([\n            transforms.ToTensor(),\n            transforms.Normalize(mean=(0.5, 0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5, 0.5)),\n        ])\n      \n        self.bioclim_data_dir = bioclim_data_dir\n        self.landsat_data_dir = landsat_data_dir\n        self.sentinel_data_dir = sentinel_data_dir\n        self.metadata = metadata\n        \n    def __getitem__(self, idx):\n        \n        survey_id = self.metadata.surveyId[idx]\n        landsat_sample = torch.nan_to_num(torch.load(os.path.join(self.landsat_data_dir, f\"GLC24-PA-test-landsat_time_series_{survey_id}_cube.pt\")))\n        bioclim_sample = torch.nan_to_num(torch.load(os.path.join(self.bioclim_data_dir, f\"GLC24-PA-test-bioclimatic_monthly_{survey_id}_cube.pt\")))\n        \n        rgb_sample = np.array(Image.open(construct_patch_path(self.sentinel_data_dir, survey_id)))\n        nir_sample = np.array(Image.open(construct_patch_path(self.sentinel_data_dir.replace(\"rgb\", \"nir\").replace(\"RGB\", \"NIR\"), survey_id)))\n        sentinel_sample = np.concatenate((rgb_sample, nir_sample[...,None]), axis=2)\n\n        if isinstance(landsat_sample, torch.Tensor):\n            landsat_sample = landsat_sample.permute(1, 2, 0)  # Change tensor shape from (C, H, W) to (H, W, C)\n            landsat_sample = landsat_sample.numpy()  # Convert tensor to numpy array\n            \n        if isinstance(bioclim_sample, torch.Tensor):\n            bioclim_sample = bioclim_sample.permute(1, 2, 0)  # Change tensor shape from (C, H, W) to (H, W, C)\n            bioclim_sample = bioclim_sample.numpy()  # Convert tensor to numpy array   \n        \n        if self.transform:\n            landsat_sample = self.transform(landsat_sample)\n            bioclim_sample = self.transform(bioclim_sample)\n            sentinel_sample = self.sentinel_transform(sentinel_sample)\n\n        return landsat_sample, bioclim_sample, sentinel_sample, survey_id","metadata":{"ExecuteTime":{"end_time":"2024-04-30T21:25:32.627928Z","start_time":"2024-04-30T21:25:32.612131Z"},"collapsed":false,"jupyter":{"outputs_hidden":false},"tags":[],"execution":{"iopub.status.busy":"2024-05-06T09:07:04.757772Z","iopub.execute_input":"2024-05-06T09:07:04.758324Z","iopub.status.idle":"2024-05-06T09:07:04.783050Z","shell.execute_reply.started":"2024-05-06T09:07:04.758293Z","shell.execute_reply":"2024-05-06T09:07:04.782005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load metadata and prepare data loaders","metadata":{}},{"cell_type":"code","source":"# Dataset and DataLoader\nbatch_size = 64\ntransform = transforms.Compose([\n    transforms.ToTensor()\n])\n\n# Load Training metadata\ntrain_landsat_data_path = \"/kaggle/input/geolifeclef-2024/TimeSeries-Cubes/TimeSeries-Cubes/GLC24-PA-train-landsat_time_series/\"\ntrain_bioclim_data_path = \"/kaggle/input/geolifeclef-2024/TimeSeries-Cubes/TimeSeries-Cubes/GLC24-PA-train-bioclimatic_monthly/\"\ntrain_sentinel_data_path=\"/kaggle/input/geolifeclef-2024/PA_Train_SatellitePatches_RGB/pa_train_patches_rgb/\"\ntrain_metadata_path = \"/kaggle/input/geolifeclef-2024/GLC24_PA_metadata_train.csv\"\n\ntrain_metadata = pd.read_csv(train_metadata_path)\ndataset_alpine = TrainDataset(train_bioclim_data_path, train_landsat_data_path, train_sentinel_data_path, train_metadata, transform=transform)\ntrain_loader = DataLoader(dataset_alpine, batch_size=batch_size, shuffle=True, num_workers=4)\n\n# Load Test metadata\ntest_landsat_data_path = \"/kaggle/input/geolifeclef-2024/TimeSeries-Cubes/TimeSeries-Cubes/GLC24-PA-test-landsat_time_series/\"\ntest_bioclim_data_path = \"/kaggle/input/geolifeclef-2024/TimeSeries-Cubes/TimeSeries-Cubes/GLC24-PA-test-bioclimatic_monthly/\"\ntest_sentinel_data_path = \"/kaggle/input/geolifeclef-2024/PA_Test_SatellitePatches_RGB/pa_test_patches_rgb/\"\ntest_metadata_path = \"/kaggle/input/geolifeclef-2024/GLC24_PA_metadata_test.csv\"\n\ntest_metadata = pd.read_csv(test_metadata_path)\ntest_dataset = TestDataset(test_bioclim_data_path, test_landsat_data_path, test_sentinel_data_path, test_metadata, transform=transform)\ntest_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False, num_workers=4)","metadata":{"ExecuteTime":{"end_time":"2024-04-30T21:25:34.532017Z","start_time":"2024-04-30T21:25:32.615562Z"},"collapsed":false,"jupyter":{"outputs_hidden":false},"tags":[],"execution":{"iopub.status.busy":"2024-05-06T09:07:04.784374Z","iopub.execute_input":"2024-05-06T09:07:04.784747Z","iopub.status.idle":"2024-05-06T09:07:10.575053Z","shell.execute_reply.started":"2024-05-06T09:07:04.784703Z","shell.execute_reply":"2024-05-06T09:07:10.574175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define and initialize a Multimodal Model\n\nTo process multiple inputs with different modalities and formats we use so-call siamiese approach where each modality is processed with different backbone (i.e., encoder). Data encoded into a 1d vector are concatenated and classified with a simple fully connected neural network. Short recap from previous notebooks.\n- The Landsat cubes have a shape of [6,4,21] (BANDs, QUARTERs, and YEARs).\n- The Bioclimatic cubes have a shape of [4,19,12] (RASTER-TYPE, YEAR, and MONTH)\n- The Sentinel Image Patches have a shape od [128, 128, 4] (R, G, B, NIR)","metadata":{}},{"cell_type":"code","source":"import torch.nn.functional as F\n\nclass MultimodalEnsemble(nn.Module):\n    def __init__(self, num_classes):\n        super(MultimodalEnsemble, self).__init__()\n        \n        self.landsat_norm = nn.LayerNorm([6,4,21])\n        self.landsat_model = models.resnet18(weights=None)\n        # Modify the first convolutional layer to accept 6 channels instead of 3\n        self.landsat_model.conv1 = nn.Conv2d(6, 64, kernel_size=3, stride=1, padding=1, bias=False)\n        self.landsat_model.maxpool = nn.Identity()\n        \n        self.bioclim_norm = nn.LayerNorm([4,19,12])\n        self.bioclim_model = models.resnet18(weights=None)  \n        # Modify the first convolutional layer to accept 4 channels instead of 3\n        self.bioclim_model.conv1 = nn.Conv2d(4, 64, kernel_size=3, stride=1, padding=1, bias=False)\n        self.bioclim_model.maxpool = nn.Identity()\n        \n        self.sentinel_model = models.swin_t(weights=\"IMAGENET1K_V1\")\n        # Modify the first layer to accept 4 channels instead of 3\n        self.sentinel_model.features[0][0] = nn.Conv2d(4, 96, kernel_size=(4, 4), stride=(4, 4))\n        self.sentinel_model.head = nn.Identity()\n        \n        self.ln1 = nn.LayerNorm(1000)\n        self.ln2 = nn.LayerNorm(1000)\n        self.fc1 = nn.Linear(2768, 4096)\n        self.fc2 = nn.Linear(4096, num_classes)\n        \n        self.dropout = nn.Dropout(p=0.1)\n        \n    def forward(self, x, y, z):\n        \n        x = self.landsat_norm(x)\n        x = self.landsat_model(x)\n        x = self.ln1(x)\n        \n        y = self.bioclim_norm(y)\n        y = self.bioclim_model(y)\n        y = self.ln2(y)\n        \n        z = self.sentinel_model(z)\n        \n        xyz = torch.cat((x, y, z), dim=1)\n        xyz = self.fc1(xyz)\n        xyz = self.dropout(xyz)\n        out = self.fc2(xyz)\n        return out","metadata":{"ExecuteTime":{"end_time":"2024-04-30T21:25:31.014067Z","start_time":"2024-04-30T21:25:31.01006Z"},"collapsed":false,"jupyter":{"outputs_hidden":false},"tags":[],"execution":{"iopub.status.busy":"2024-05-06T09:08:57.920217Z","iopub.execute_input":"2024-05-06T09:08:57.920574Z","iopub.status.idle":"2024-05-06T09:08:57.933714Z","shell.execute_reply.started":"2024-05-06T09:08:57.920548Z","shell.execute_reply":"2024-05-06T09:08:57.932748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def set_seed(seed):\n    # Set seed for Python's built-in random number generator\n    torch.manual_seed(seed)\n    # Set seed for numpy\n    np.random.seed(seed)\n    # Set seed for CUDA if available\n    if torch.cuda.is_available():\n        torch.cuda.manual_seed_all(seed)\n        # Set cuDNN's random number generator seed for deterministic behavior\n        torch.backends.cudnn.deterministic = True\n        torch.backends.cudnn.benchmark = False\n\nset_seed(69)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2024-05-06T09:08:58.387461Z","iopub.execute_input":"2024-05-06T09:08:58.387818Z","iopub.status.idle":"2024-05-06T09:08:58.395133Z","shell.execute_reply.started":"2024-05-06T09:08:58.387791Z","shell.execute_reply":"2024-05-06T09:08:58.394142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check if cuda is available\ndevice = torch.device(\"cpu\")\n\nif torch.cuda.is_available():\n    device = torch.device(\"cuda\")\n    print(\"DEVICE = CUDA\")\n\nnum_classes = 11255 # Number of all unique classes within the PO and PA data.\nmodel = MultimodalEnsemble(num_classes).to(device)","metadata":{"ExecuteTime":{"end_time":"2024-04-30T21:25:31.611823Z","start_time":"2024-04-30T21:25:31.607373Z"},"collapsed":false,"jupyter":{"outputs_hidden":false},"tags":[],"execution":{"iopub.status.busy":"2024-05-06T09:08:58.916410Z","iopub.execute_input":"2024-05-06T09:08:58.917225Z","iopub.status.idle":"2024-05-06T09:09:01.804079Z","shell.execute_reply.started":"2024-05-06T09:08:58.917179Z","shell.execute_reply":"2024-05-06T09:09:01.803256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Loop\n\nNothing special, just a standard Pytorch training loop.","metadata":{}},{"cell_type":"code","source":"# Hyperparameters\nlearning_rate = 0.00025\nnum_epochs = 10\npositive_weigh_factor = 1.0\n\noptimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)\nscheduler = CosineAnnealingLR(optimizer, T_max=25, verbose=True)","metadata":{"ExecuteTime":{"end_time":"2024-04-30T21:25:32.181927Z","start_time":"2024-04-30T21:25:32.177073Z"},"collapsed":false,"jupyter":{"outputs_hidden":false},"tags":[],"execution":{"iopub.status.busy":"2024-05-06T09:09:04.721477Z","iopub.execute_input":"2024-05-06T09:09:04.721855Z","iopub.status.idle":"2024-05-06T09:09:04.730311Z","shell.execute_reply.started":"2024-05-06T09:09:04.721824Z","shell.execute_reply":"2024-05-06T09:09:04.729183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Training for {num_epochs} epochs started.\")\n\nfor epoch in range(num_epochs):\n    model.train()\n    \n    for batch_idx, (data1, data2, data3, targets, _) in enumerate(train_loader):\n\n        data1 = data1.to(device)\n        data2 = data2.to(device)\n        data3 = data3.to(device)\n        targets = targets.to(device)\n\n        optimizer.zero_grad()\n        outputs = model(data1, data2, data3)\n\n        pos_weight = targets*positive_weigh_factor  # All positive weights are equal to 10\n        criterion = torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight)\n        loss = criterion(outputs, targets)\n\n        loss.backward()\n        optimizer.step()\n\n        if batch_idx % 278 == 0:\n            print(f\"Epoch {epoch+1}/{num_epochs}, Batch {batch_idx}/{len(train_loader)}, Loss: {loss.item()}\")\n\n    scheduler.step()\n    print(\"Scheduler:\",scheduler.state_dict())\n\n# Save the trained model\nmodel.eval()\ntorch.save(model.state_dict(), \"multimodal-model.pth\")","metadata":{"ExecuteTime":{"start_time":"2024-04-30T21:25:34.536634Z"},"collapsed":false,"is_executing":true,"jupyter":{"outputs_hidden":false},"tags":[],"execution":{"iopub.status.busy":"2024-05-06T09:09:08.277588Z","iopub.execute_input":"2024-05-06T09:09:08.277946Z","iopub.status.idle":"2024-05-06T10:34:36.512218Z","shell.execute_reply.started":"2024-05-06T09:09:08.277917Z","shell.execute_reply":"2024-05-06T10:34:36.511070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.eval()\nprint(\"Done\")","metadata":{"execution":{"iopub.status.busy":"2024-05-06T11:43:07.217927Z","iopub.execute_input":"2024-05-06T11:43:07.218737Z","iopub.status.idle":"2024-05-06T11:43:07.548284Z","shell.execute_reply.started":"2024-05-06T11:43:07.218705Z","shell.execute_reply":"2024-05-06T11:43:07.546977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test Loop\n\nAgain, nothing special, just a standard inference.","metadata":{}},{"cell_type":"code","source":"with torch.no_grad():\n    all_predictions = []\n    surveys = []\n    top_k_indices = None\n    for batch_idx, (data1, data2, data3, surveyID) in enumerate(test_loader):\n\n        data1 = data1.to(device)\n        data2 = data2.to(device)\n        data3 = data3.to(device)\n        targets = targets.to(device)\n\n        outputs = model(data1, data2, data3)\n        predictions = torch.sigmoid(outputs).cpu().numpy()\n\n        # Sellect top-25 values as predictions\n        top_25 = np.argsort(-predictions, axis=1)[:, :25] \n        if top_k_indices is None:\n            top_k_indices = top_25\n        else:\n            top_k_indices = np.concatenate((top_k_indices, top_25), axis=0)\n\n        surveys.extend(surveyID.cpu().numpy())","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"tags":[],"execution":{"iopub.status.busy":"2024-05-06T09:07:11.048053Z","iopub.status.idle":"2024-05-06T09:07:11.048510Z","shell.execute_reply.started":"2024-05-06T09:07:11.048283Z","shell.execute_reply":"2024-05-06T09:07:11.048302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Save prediction file! 🎉🥳🙌🤗","metadata":{}},{"cell_type":"code","source":"data_concatenated = [' '.join(map(str, row)) for row in top_k_indices]\n\npd.DataFrame(\n    {'surveyId': surveys,\n     'predictions': data_concatenated,\n    }).to_csv(\"submission.csv\", index = False)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2024-05-06T09:07:11.050549Z","iopub.status.idle":"2024-05-06T09:07:11.051316Z","shell.execute_reply.started":"2024-05-06T09:07:11.051034Z","shell.execute_reply":"2024-05-06T09:07:11.051053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}