{"cells":[{"metadata":{"trusted":true},"cell_type":"markdown","source":"\n## Automated Wheat Counter\nby Marsh [ @vbookshelf ] <br>\n30 May 2020","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Why do we need an automated wheat counter?\n\nBecause manual counting is a path to the dark side. Whether it's counting cells on a microscope slide or counting wheat heads on a photo - the process is tiring and frustrating. The scientists in this music video express it best.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"https://www.youtube.com/watch?v=qgu865oCYw4\n\n<div align=\"center\"><iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/qgu865oCYw4?rel=0\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe></div>","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Objective\n\nMy objective for this project was to build and deploy a flask web app that can automatically count wheat heads on wheat photos.\n\nThis tool could help researchers get quick estimates of wheat density.\n\n<br>\n\n\n<img src=\"http://wheatcounter.test.woza.work/assets/holdout_image_11.png\" width=\"350\"></img>\n\n<h5 align=\"center\">Fig. 1 - Prediction on image from holdout set</h5>\n\n\n<br>\n\nFig. 1 above is an actual prediction that the model made on image 667b4a999.jpg. This image is part of the holodout set. Also, it's from the arvalis_2 source - arvalis_2 images were not part of the training set. Counting the number of wheat heads on this image would be hard for a human to do accurately. The correct number of wheat heads is 11. The app predicted 12.\n\n<br>\n\n> Live Demo<br>\n> http://wheatcounter.test.woza.work/<br>\n> \n> Github<br>\n> https://github.com/vbookshelf/Automated-Wheat-Counter\n\nThe demo will be live until 30 June 2020.\n\nThis app accepts only one image at a time. However, the code could be modified to support batch analysis. A user could then select multiple images. The app would output a prediction for each image and a total count off all wheat heads in the batch.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Approach\n\nThis solution uses two models - a segmentation model and a regression model. Both models were trained at the same time. \n\n\n\n<img src=\"http://wheatcounter.test.woza.work/assets/images_and_masks.png\" height=\"200\"></img>\n\n<h5 align=\"center\">Fig. 2 - Segmentation and Regression model images and masks</h5>\n\n<br>\n<br>\n**For each training loop the process is as follows**:\n\nA pre-processed 512x512x3 RGB image is input into the segmentation model. This model outputs a 512x512x1 segmentation mask with matrix values that are probabilities between zero and one. This mask is then thresholded to convert it into a binary mask i.e. the matrix now has values that are either zero or one. \n\nThis binary mask and the pre-processed RGB input image are then multiplied. As a result any areas of the input image that are not inside the segmentation areas (i.e. not wheat heads) get assigned a value of zero. In other words they become black. In this way image patterns that are specific to certain sources (world regions where the photos were taken) are removed.\n\nThis multiplied image is then fed into the regression model. The regression model predicts the number of wheat heads.\n\nThe segmentation model is a Unet with a pre-trained resnet34 encoder. The regression model is a pre-trained resnet34. I used image augmentation to help the models generalize. The models were trained for 30 epochs with a constant learning rate of 0.0001 - dice loss for the segmentation model and mse loss for the regression model.\n\nAlthough there are two models, for the rest of this notebook I will be referring to them both as \"the model\".\n\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Validation Strategy\n\nIn order to check how well the solution was generalizing I split the training and validation images based on their source.\n\nThe model was trained on images from these sources:<br>\n(2969 images)\n\n- arvalis_1 (France)\n- arvalis_3 (France)\n- inrae_1 (France)\n- ethz_1 (Switzerland)\n- rres_1 (UK)\n\nIt was validated on images from these sources:<br>\n(389 images)\n\n- usask_1 (Canada)\n- arvalis_2 (France)\n\n\nHoldout set:<br>\n(15 images)\n- usask_1 (Canada)\n- arvalis_2 (France)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Validation Results\n\n> **Total val set percentage error**: 20.5 %\n> \n> This is the sum of the absolute counting error for all 389 validation images divided by the total number of wheat heads in the vaidation set.\n\n> **Val set MAE**: 5.03<br>\n>\n> **usask_1 MAE**: 3.89<br>\n> **arvalis_2 MAE**: 6.16<br>\n\n\n\n> Could this model be a helpful tool? A practical way to determine this would be to compare it's percentage error to human-level performance.\n>\n> Let's define human-level performance as the percentage error that could be achieved by a typical human who is counting wheat heads on a dataset of images. A typical human could achieve 0% error on one image but, because of the tedious nature of counting, the error would be higher when a human has to manually analyze a large batch of images.\n>\n> **If a human were to manually count the wheat heads in the validation set (389 images), what would be his or her percentage error? **\n>\n> If this model's error is lower, then we could conclude that it is a viable counting tool. This is because even though the model's counting accuracy is not perfect, it's still better and faster than a typical human. The model's ability to generalize to regions that were not part of the training or validation data would still need to be tested.\n\n\n\n<br>\n\n\n*[Note: The above values change each time this notebook is run. They may not match what you see below.]*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Deployment Strategy\n\nThe app frontend and backend are on different servers. They communicate using API request/response calls.\n\n### 1- Frontend\n\nThe frontend is a simple website. It takes a user image as input. It then sends this image to the API server (backend) for prediction. When the prediction (response) is received, the frontend website displays it.\n\n### 2- Backend\n\nMachine learning web apps work nicely when tested on a local machine (localhost). But the true test comes when the model is deployed to a real production server. This process can be frustrating due package dependency issues.\n\nThrough experimentation I've found that using a rented server plus Docker plus Pytorch is a robust deployment approach. Although it's not cheap, this backend strategy can be set up without headaches and it works reliably. It's also easy to scale i.e. to support high workloads it's easy to run an app on mutiple servers with load balancing.\n\nThe server for this app has 2 vCPU's and 4GB of RAM. It's running Ubuntu 16.04. \n\nThe setup is a Docker container that's made up of Flask, Nginx and uWSgi. Flask is where the python code is stored. Nginx is the web server and uWSGI allows Flask and Nginx to talk. By using Docker, the model and all it's dependencies (incl Flask and Nginx) are in the same container.\n\nThe trend now is to outsource the backend to third party companies. However, as server side code becomes highly complex, I believe that in order to ensure service reliability the owner of the app needs to control the entire workflow - from the server to the end user. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Install Packages\n\nWe'll be using the excellent segmentation-models-pytorch package created by Pavel Yakubovsky.<br>\nhttps://github.com/qubvel/segmentation_models.pytorch","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"! pip install segmentation-models-pytorch","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\n\nimport ast\n\n\nimport cv2\nimport time\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nimport torch.nn.functional as F\n\nimport torchvision\nimport torchvision.transforms as transforms\n\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms, utils\n\n# set a seed value\ntorch.manual_seed(555)\n\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nfrom torch.optim.lr_scheduler import StepLR\n\n\nimport albumentations as albu\nfrom albumentations import Compose\n\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.utils import shuffle\n\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n# Don't Show Warning Messages\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Note: Pytorch uses a channels-first format:\n# [batch_size, num_channels, height, width]\n\nprint(torch.__version__)\nprint(torchvision.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('../input/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BACKBONE = 'resnet34'\n\nIMAGE_HEIGHT_ORIG = 1024\nIMAGE_WIDTH_ORIG = 1024\nIMAGE_CHANNELS_ORIG = 3\n\nIMAGE_HEIGHT = 512\nIMAGE_WIDTH = 512\nIMAGE_CHANNELS = 3\n\nBATCH_SIZE = 8\n\nSAMPLE_SIZE = 15\n\nNUM_EPOCHS = 30\n\nTHRESHOLD = 0.7\n\nLRATE = 0.0001\n\n# Check the number of available cpu cores.\n# This variable is used to set the num workers in the data loader.\nNUM_CORES = os.cpu_count()\n\nNUM_CORES","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define the device","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\nprint(device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef create_mask(image_id, image_height, image_width):\n    \"\"\"\n    Input: image_id\n    Output: Image mask containing all the bbox masks.\n    \"\"\"\n    \n    # filter out all rows with the image_id\n    df = df_data[df_data['image_id'] == image_id]\n\n    # create blank mask\n    mask = np.zeros((image_height, image_width, 1))\n\n    # get a list of all bbox values\n    bbox_list = list(df['bbox'])\n\n\n    # loop over the bbox list\n    for bbox_string in bbox_list:\n\n        bbox = ast.literal_eval(bbox_string) \n        \n        # First we check if a mask exists.\n        # If not a blank mask is returned.\n        if (bbox_string[0] != '[') or (len(bbox) == 0):\n\n            pass\n\n        else:\n\n            # get the bounding box\n            # ast.literal_eval converts '[...]' to [...]\n            bbox = ast.literal_eval(bbox_string) \n\n            x = int(bbox[0])\n            y = int(bbox[1])\n            w = int(bbox[2])\n            h = int(bbox[3])\n\n            # add the bbox mask to the blank mask created above\n            mask[y:y+h, x:x+w] = 1\n    \n    return mask\n\n\n\n\n\ndef multiply_masks_and_images(images, thresh_masks):\n    \n    \"\"\"\n    Trying to do this multiplication with Pytorch tensors\n    did not produce the result that I wanted. Therefore, here I am\n    converting the tensors to numpy, doing the multiplication, and \n    then converting back to pytorch.\n    \n    \"\"\"\n\n    # convert from torch tensors to numpy\n    np_images = images.cpu().numpy()\n    np_thresh_masks = thresh_masks.cpu().numpy()\n\n    # reshape\n    np_images = np_images.reshape((-1, 512, 512, 3))\n    np_thresh_masks = np_thresh_masks.reshape((-1, 512, 512, 1))\n\n\n    # multiply the mask by the image\n    modified_images = np_thresh_masks * np_images\n\n    # change shape to channels first to suit pytorch\n    #modified_images = modified_images.transpose((2, 0, 1))\n    modified_images = modified_images.reshape((-1, 3, 512, 512))\n\n    # convert to torch tensor\n    modified_images = torch.tensor(modified_images, dtype=torch.float)\n\n    return modified_images\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare the Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# I saved the original competition data as a Kaggle dataset.\n# Here I'm using that dataset as the data source.\n\nbase_path = '../input/wheat-detection-comp-original-data/global-wheat-detection/'\n\nos.listdir(base_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = base_path + 'train.csv'\n\ndf_data = pd.read_csv(path)\n\nprint(df_data.shape)\n\ndf_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Create new columns","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# [xmin, ymin, width, height]\n\n\ndef extract_width(x):\n    \n    # convert the string to a python list\n    bbox = ast.literal_eval(x) \n    \n    # get the width\n    w = int(bbox[2])\n    \n    return w\n\n\n\ndef extract_height(x):\n    \n    # convert the string to a python list\n    bbox = ast.literal_eval(x) \n    \n    # get the width\n    h = int(bbox[3])\n    \n    return h\n\n\n\n# Create new columns\ndf_data['w'] = df_data['bbox'].apply(extract_width)\ndf_data['h'] = df_data['bbox'].apply(extract_height)\n\ndf_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Clean the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filter out the big masks.\n# Use 500px as the limit.\n\ndf_data = df_data[df_data['w'] < 500]\ndf_data = df_data[df_data['h'] < 500]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filter out the very small masks.\n# Use 20px as the limit.\n\ndf_data = df_data[df_data['w'] >= 20]\ndf_data = df_data[df_data['h'] >= 20]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check how many rows now exist\n\ndf_data.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Get the num wheat heads on each image\n\nWe had to clean the data first before doing this. The resulting dataframe will need to be merged with the train and val data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_masks = df_data.copy()\n\n# Create a new column\ndf_masks['num_masks'] = 1\n\ndf_masks = df_masks[['image_id', 'num_masks']]\n\nprint(df_masks.shape)\n\ndf_masks.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a dataframe showing num wheat heads on each unique image\n\ndf_masks = df_masks.groupby('image_id').sum()\n\ndf_masks = df_masks.reset_index()\n\ndf_masks.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Get lists of unique images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# List the data sources\n\ndf_data['source'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filter out all images\ndf_usask_1 = df_data[df_data['source'] == 'usask_1']\n# Drop duplicate image_id values\ndf_usask_1 = df_usask_1.drop_duplicates(subset='image_id', keep='first', inplace=False)\n\n# Filter out all images\ndf_arvalis_1 = df_data[df_data['source'] == 'arvalis_1']\n# Drop duplicate image_id values\ndf_arvalis_1 = df_arvalis_1.drop_duplicates(subset='image_id', keep='first', inplace=False)\n\n# Filter out all images\ndf_arvalis_2 = df_data[df_data['source'] == 'arvalis_2']\n# Drop duplicate image_id values\ndf_arvalis_2 = df_arvalis_2.drop_duplicates(subset='image_id', keep='first', inplace=False)\n\n# Filter out all images\ndf_arvalis_3 = df_data[df_data['source'] == 'arvalis_3']\n# Drop duplicate image_id values\ndf_arvalis_3 = df_arvalis_3.drop_duplicates(subset='image_id', keep='first', inplace=False)\n\n# Filter out all images\ndf_inrae_1 = df_data[df_data['source'] == 'inrae_1']\n# Drop duplicate image_id values\ndf_inrae_1 = df_inrae_1.drop_duplicates(subset='image_id', keep='first', inplace=False)\n\n# Filter out all images\ndf_ethz_1 = df_data[df_data['source'] == 'ethz_1']\n# Drop duplicate image_id values\ndf_ethz_1 = df_ethz_1.drop_duplicates(subset='image_id', keep='first', inplace=False)\n\n# Filter out all images\ndf_rres_1 = df_data[df_data['source'] == 'rres_1']\n# Drop duplicate image_id values\ndf_rres_1 = df_rres_1.drop_duplicates(subset='image_id', keep='first', inplace=False)\n\n\n\nprint('usask_1:', df_usask_1.shape) # Use for validation\nprint('arvalis_1:', df_arvalis_1.shape)\nprint('arvalis_2:', df_arvalis_2.shape) # Use for validation\nprint('arvalis_3:', df_arvalis_3.shape)\nprint('inrae_1:', df_inrae_1.shape)\nprint('ethz_1:', df_ethz_1.shape)\nprint('rres_1:', df_rres_1.shape)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Display one image and mask","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Print one image\n\nimage_id = '42e6efaaa'\n\npath = base_path + 'train/' + image_id + '.jpg'\n\n\nmask = create_mask(image_id, 1024, 1024)\nmask = mask[:,:,0]\n\nimage = plt.imread(path)\n\nprint(image.shape)\n\nplt.imshow(image, cmap='Greys')\nplt.imshow(mask, cmap='Reds', alpha=0.3)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Test Split","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train\n    # df_arvalis_1\n    # df_arvalis_3\n    # df_inrae_1\n    # df_ethz_1\n    # df_rres_1\n\n\n\n# Val\n    # df_usask_1\n    # df_arvalis_2\n\n    \n    \n# Create the train set\ndf_train = pd.concat([df_arvalis_1, df_arvalis_3, df_inrae_1, df_ethz_1, df_rres_1], axis=0)\n\n\n# Create the val set\ndf_val = pd.concat([df_usask_1, df_arvalis_2], axis=0)\n\n\nprint(df_train.shape)\nprint(df_val.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Merge df_masks with df_train and df_val\n\nThis is so that we have a column showing num wheat heads on each image.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# We perform an inner join (intersection).\n# Only image_id values that are common to both dataframes will be kept. Other rows will be deleted.\n# Codebasics tutorial: https://www.youtube.com/watch?v=h4hOPGo4UVU\n\ndf_train = pd.merge(df_train, df_masks, on='image_id', how='inner')\n\ndf_val = pd.merge(df_val, df_masks, on='image_id', how='inner')\n\n# Select the columns we want.\ncols = ['image_id', 'source', 'num_masks']\ndf_train = df_train[cols]\ndf_val = df_val[cols]\n\nprint(df_train.shape)\nprint(df_val.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_val.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create a holdout test set\n\nThis will be set aside and won't be used during training and validation. We could use these images later to check how the trained model performs on unseen data and to test the app. We will set aside 15 images from df_val as a holdout set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = df_val.sample(n=SAMPLE_SIZE, random_state=101)\n\nprint(df_test.shape)\n\ndf_test.head(15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_val.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Remove the test images from df_val\n\ntest_images_list = list(df_test['image_id'])\n \ndf_val = df_val[~df_val['image_id'].isin(test_images_list)] # This line means: is not in\n\ndf_val.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Shuffle\n\ndf_train = shuffle(df_train)\n\ndf_val = shuffle(df_val)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Save the dataframes as compressed csv files","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_data.to_csv('df_data.csv.gz', compression='gzip', index=False)\n\ndf_train.to_csv('df_train.csv.gz', compression='gzip', index=False)\ndf_val.to_csv('df_val.csv.gz', compression='gzip', index=False)\n\ndf_test.to_csv('df_test.csv.gz', compression='gzip', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check that the dataframes have been saved.\n!ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Set up and test the augmentations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nimport albumentations as albu\n\n# Define the augmentations\n\ndef get_training_augmentation():\n    \n    train_transform = [\n    albu.Flip(always_apply=False, p=0.8),\n    albu.RandomRotate90(always_apply=False, p=0.8),\n    albu.Blur(blur_limit=7, always_apply=False, p=0.3),\n    albu.OneOf([\n        albu.RandomContrast(),\n        albu.RandomGamma(),\n        albu.RandomBrightness(),\n        ], p=0.5),\n    albu.OneOf([\n        albu.ElasticTransform(alpha=120, sigma=120 * 0.05, alpha_affine=120 * 0.03),\n        albu.GridDistortion(),\n        albu.OpticalDistortion(distort_limit=2, shift_limit=0.5),\n        ], p=0.5),\n   \n    ]\n  \n    return albu.Compose(train_transform)\n\n\n\n\ndef get_preprocessing(preprocessing_fn):\n    \"\"\"Construct preprocessing transform\n    \n    Args:\n        preprocessing_fn (callbale): data normalization function \n            (can be specific for each pretrained neural network)\n    Return:\n        transform: albumentations.Compose\n    \n    \"\"\"\n    \n    _transform = [\n        albu.Lambda(image=preprocessing_fn),\n    ]\n    return albu.Compose(_transform)\n   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display an image and mask\n# ==========================\n\nimage_id = '7d5af5b74'\n\npath = '../input/global-wheat-detection/train/' + image_id + '.jpg'\n\n\nmask = create_mask(image_id, 1024, 1024)\nmask = mask[:,:,0]\n\nimage = plt.imread(path)\n\nprint(image.min())\nprint(image.max())\nprint(image.shape)\n\nplt.imshow(image, cmap='Greys')\nplt.imshow(mask, cmap='Reds', alpha=0.3)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display an AUGMENTED image\n# =============================\n\nimage_id = '7d5af5b74'\npath = '../input/global-wheat-detection/train/' + image_id + '.jpg'\n\nimage = plt.imread(path)\nmask = create_mask(image_id, 1024, 1024)\n\naugmentation = get_training_augmentation()\n\nsample = augmentation(image=image, mask=mask)\nimage, mask = sample['image'], sample['mask']\n\n\nprint(image.min())\nprint(image.max())\n\nprint(image.shape)\nprint(mask.shape)\n\n\n\nplt.imshow(image, cmap='Greys')\nplt.imshow(np.squeeze(mask), cmap='Reds', alpha=0.3)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display a PRE-PROCESSED image\n# ==============================\n\n\nimage_id = '7d5af5b74'\npath = '../input/global-wheat-detection/train/' + image_id + '.jpg'\n\nimage = plt.imread(path)\nmask = create_mask(image_id, 1024, 1024)\n\n\n\n\nfrom segmentation_models_pytorch.encoders import get_preprocessing_fn\n\n# Initialize the preprocessing function\npreprocessing_fn = get_preprocessing_fn(BACKBONE, pretrained='imagenet')\n\npreprocessing = get_preprocessing(preprocessing_fn)\n\nsample = preprocessing(image=image, mask=mask)\nimage, mask = sample['image'], sample['mask']\n\n\n\nprint(image.min())\nprint(image.max())\n\nprint(image.shape)\nprint(mask.shape)\n\n\n\nplt.imshow(image)\n\n# Uncomment the next line to see a mask overlayed on the pre-processed image.\n# plt.imshow(np.squeeze(mask), cmap='Reds', alpha=0.3)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Set up the DataLoader","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reset the indices\n\ndf_train = df_train.reset_index(drop=True)\ndf_val = df_val.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class CompDataset(Dataset):\n    \n    def __init__(self, df, augmentation=None, preprocessing=None):\n        self.df_data = df\n        self.augmentation = augmentation\n        self.preprocessing = preprocessing\n        \n        \n        \n    def __getitem__(self, index):\n        image_id = self.df_data.loc[index, 'image_id']\n        \n        image_path = base_path + 'train/'\n        \n\n        # set the path to the image\n        path = image_path + image_id + '.jpg'\n        \n        # Create the image\n        # ------------------\n\n\n        # read the image\n        image = cv2.imread(path)\n\n        # convert to from BGR to RGB\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n        # resize the image\n        image = cv2.resize(image, (IMAGE_WIDTH, IMAGE_HEIGHT))\n\n        \n        \n        # Create the mask\n        # ------------------\n        \n        # create the mask\n        mask = create_mask(image_id, 1024, 1024)\n\n        # resize the mask\n        mask = cv2.resize(mask, (IMAGE_WIDTH, IMAGE_HEIGHT))\n        \n        # create a channel dimension\n        mask = np.expand_dims(mask, axis=-1)\n        \n  \n        \n        # apply augmentations\n        if self.augmentation:\n            sample = self.augmentation(image=image, mask=mask)\n            image, mask = sample['image'], sample['mask']\n        \n        # apply preprocessing\n        if self.preprocessing:\n            sample = self.preprocessing(image=image, mask=mask)\n            image, mask = sample['image'], sample['mask']\n            \n\n    \n        # Swap color axis\n        # numpy image: H x W x C\n        # torch image: C X H X W\n        image = image.transpose((2, 0, 1))\n        mask = mask.transpose((2, 0, 1))\n        \n        # convert to torch tensor\n        image = torch.tensor(image, dtype=torch.float)\n        mask = torch.tensor(mask, dtype=torch.float)\n        \n        \n        # Create the regression target.\n        cols = ['num_masks']\n        \n        target = torch.tensor(self.df_data.loc[index, cols], dtype=torch.float)\n        \n        \n        sample = (image, mask, target)\n        \n        \n        return sample\n    \n    \n    def __len__(self):\n        return len(self.df_data)\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Test the dataloaders\n\nfrom segmentation_models_pytorch.encoders import get_preprocessing_fn\n\n# Initialize the preprocessing function\npreprocessing_fn = get_preprocessing_fn(BACKBONE, pretrained='imagenet')\n\n\ntrain_data = CompDataset(df_train, augmentation=get_training_augmentation(), \n                        preprocessing=get_preprocessing(preprocessing_fn))\n\nval_data = CompDataset(df_val, augmentation=None, \n                        preprocessing=get_preprocessing(preprocessing_fn))\n\n\ntrain_loader = torch.utils.data.DataLoader(train_data,\n                                            batch_size=BATCH_SIZE,\n                                            shuffle=True,\n                                           num_workers=NUM_CORES\n                                            )\n\nval_loader = torch.utils.data.DataLoader(val_data,\n                                            batch_size=BATCH_SIZE,\n                                            shuffle=False,\n                                           num_workers=NUM_CORES\n                                            )\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get one of train batch\nimages, masks, targets = next(iter(train_loader))\n\nprint(images.shape)\nprint(masks.shape)\nprint(targets.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# display an image from the batch\nimage = images[0, 0, :, :]\nmask = masks[0, 0, :, :]\n\nprint(image.min())\nprint(image.max())\n\nplt.imshow(image)\nplt.imshow(mask, cmap='Reds', alpha=0.3)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display the image batch in a grid\n\ngrid = torchvision.utils.make_grid(images) # nrows is the number of classes\n\nplt.figure(figsize=(15,15))\nplt.imshow(np.transpose(grid, (1,2,0)))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display the mask batch in a grid\n\ngrid = torchvision.utils.make_grid(masks) # nrows is the number of classes\n\nplt.figure(figsize=(15,15))\nplt.imshow(np.transpose(grid, (1,2,0)))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Architecture\n\nHere we will set up two models - a segmentation model and a regression model. We will train both models at the same time. The output from the first model will be processed and then used as an input for the second model. The final output will be a prediction of how many wheat heads are on each image.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Model Workflow\n\n\n# (1) Segmentation Model\n# .......................\n\n# seg_model Input: \n# 3x512x512 RGB pre-processed image\n\n# seg_model Output:\n# Mask with float values in range 0 to 1\n\n\n\n# (2) Intermediate Step\n# .......................\n\n# Threshold the seg_model output to obtain a binary mask.\n# Take the image that was used as the input to the seg_model and multiply it by the binary mask.\n\n\n\n# (3) Regression Model\n# .......................\n# Resnet34 was used as the seg_model encoder therefore, we are using resnet34 in the reg_model also\n# so that the image pre-processing will not need to be done again.\n\n# reg_model Input: \n# 3x512x512 pre-processed image that has been multiplied by a binary mask.\n\n# reg_model Output: \n# Number of wheat heads on the image.\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## seg_model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import segmentation_models_pytorch as smp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# seg_model Input: \n# .................\n\n# 3x512x512 RGB pre-processed image\n\n\n# seg_model Output:\n# .................\n\n# Mask with values in range 0 to 1\n\n\nENCODER_WEIGHTS = 'imagenet'\nACTIVATION = 'sigmoid' \n\n# create segmentation model with pretrained encoder\nseg_model = smp.Unet(\n    encoder_name=BACKBONE, \n    encoder_weights=ENCODER_WEIGHTS, \n    classes=1, \n    activation=ACTIVATION,\n)\n\npreprocessing_fn = smp.encoders.get_preprocessing_fn(BACKBONE, ENCODER_WEIGHTS)\n\nprint(seg_model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## reg_model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision import models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# reg_model Input: \n# .................\n\n# 3x512x512 pre-processed image that has been multiplied by a binary mask.\n\n# reg_model Output: \n# ..................\n\n# Number of wheat heads on the image.\n\n\nreg_model = models.resnet34(pretrained=True)\nin_features = reg_model.fc.in_features # If we print the architecture we see this number in the last layer.\n\nreg_model.fc = nn.Linear(in_features, 1)\n#reg_model.out = nn.ReLU(inplace=True)\n\n\nprint(reg_model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test the models\n\nThis step is not essential but I've found that it helps to visualize what will be happening inside the model during training.\n\nIn Pytorch layers and models are simply functions that take an input and produce an output. Here we will simply pass a batch of training images through the model. We will be able to see:\n\n- Does the model work?\n- What is the output shape?\n- What is the range of output values?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# get one train batch\nimages, masks, targets = next(iter(train_loader))\n\nprint(images.shape)\nprint(masks.shape)\nprint(targets.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### seg_model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Here we simply pass the input images through the model like we are making a prediction.\nseg_preds = seg_model(images)\n\n# loss\nseg_criterion = smp.utils.losses.DiceLoss()\nseg_loss = seg_criterion(seg_preds, masks).item()\n\n\nprint(seg_preds.shape)\nprint(seg_loss)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np_seg_preds = seg_preds.detach().numpy()\n\n\n# reshape\nnp_seg_preds = np_seg_preds.reshape(-1, 512, 512, 1)\n\nprint(type(np_seg_preds))\nprint(np_seg_preds.min())\nprint(np_seg_preds.max())\nprint(np_seg_preds.shape)\n\n# The range is 0 to 1 because of the sigmoid layer.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Print a predicted mask.\n\npred_mask = np_seg_preds[1, :, :, 0]\n\nprint(pred_mask.shape)\n\nplt.imshow(pred_mask)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Process the seg_model output","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Threshold the predicted mask\n# Note that seg_preds is of type torch tensor because we will use it as input for the reg_model.\n\nthreshold = 0.7\n\nthresh_masks = (seg_preds >= threshold).int() # change the dtype of the torch tensor to int32\n\nprint(thresh_masks.min())\nprint(thresh_masks.max())\nprint(thresh_masks.shape)\n\nplt.imshow(thresh_masks[1, 0, :, :])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create the input by multiplying the thresh_masks and the seg_model input images.\n# When we mutiply we are selecting only parts of the image that are inside the predicted segmentations.\n# All other parts of the image, outside the segmentations, are set to zero i.e. they become black.\n\n# reg_input = thresh_masks * images # This line multiplies torch tensors. \nreg_input = multiply_masks_and_images(images, thresh_masks) # Here the multiplication is done using numpy.\n\nprint(reg_input.min())\nprint(reg_input.max())\nprint(reg_input.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert to numpy so we can use plt to display an image\n\nnp_reg_input = reg_input.numpy()\n\nnp_reg_input = np_reg_input.reshape((-1, 512, 512, 3))\n\nprint(np_reg_input.shape)\n\nimage = np_reg_input[1, :, :, :]\n\nplt.imshow(image)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### reg_model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Here we pass the processed seg_model preds through reg_model.\n\nreg_preds = reg_model(reg_input)\n\n# define the layer as a function\n#reg_output = nn.ReLU(inplace=True)\n#postive_preds = reg_output(reg_preds)\n\nprint(reg_preds.shape)\n\nreg_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# loss\nreg_criterion = nn.MSELoss()\nreg_loss = reg_criterion(reg_preds, targets).item()\n\nreg_loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have tested both models and are sure that they work. Now we need to combine both models into a single training loop that includes the loss functions and the optimizers.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Model Architecture","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We will train two models at the same time but independently. The ouput from the seg model will be thresholded, multiplied by the input image and then fed into the reg model. Each model will have its own loss function and optimizer. The models will be trained inside the same training loop.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Set up the Training Loop","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Initialize the dataloaders\n\ntrain_data = CompDataset(df_train, augmentation=get_training_augmentation(), \n                        preprocessing=get_preprocessing(preprocessing_fn))\n\n# Note that we are not augmenting the validation images.\nval_data = CompDataset(df_val, augmentation=None, \n                        preprocessing=get_preprocessing(preprocessing_fn))\n\n\ntrain_loader = torch.utils.data.DataLoader(train_data,\n                                            batch_size=BATCH_SIZE,\n                                            shuffle=True,\n                                           num_workers=NUM_CORES\n                                            )\n\nval_loader = torch.utils.data.DataLoader(val_data,\n                                            batch_size=BATCH_SIZE,\n                                            shuffle=False,\n                                           num_workers=NUM_CORES\n                                            )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# send the model to the device\nseg_model.to(device)\nreg_model.to(device)\n\n# instantiate the optimizers\nseg_optimizer = torch.optim.Adam(seg_model.parameters(), lr=LRATE)\nreg_optimizer = torch.optim.Adam(reg_model.parameters(), lr=LRATE)\n\n# define the loss functions\nseg_criterion = smp.utils.losses.DiceLoss()\nreg_criterion = nn.MSELoss()\n\n\nseg_val_loss_list = []\nreg_val_loss_list = []\n\n\nfor epoch in range(0, NUM_EPOCHS):\n    \n    print('\\n') \n    print('Epoch:', epoch)\n    print('Train steps:', len(train_loader))\n    \n    \n    # ====================\n    # TRAINING\n    # ====================\n    \n    # Set the Mode\n    seg_model.train()\n    reg_model.train()\n    \n    # Turn gradient calculations on.\n    torch.set_grad_enabled(True)\n    \n    epoch_loss = 0\n    seg_epoch_loss = 0\n    reg_epoch_loss = 0\n    \n    for i, batch in enumerate(train_loader):\n        \n        \n        print(i, end=\"\\r\") \n\n        \n        # Get a batch and send to device\n        images, masks, reg_targets = batch\n        \n        images = images.to(device, dtype=torch.float)\n        masks = masks.to(device, dtype=torch.float)\n        reg_targets = reg_targets.squeeze(dim=1)\n        reg_targets = reg_targets.to(device, dtype=torch.float)\n        \n        \n        \n        # (1) SEGMENTATION MODEL\n        # .......................\n        \n        \n        # pass the input through the model\n        seg_preds = seg_model(images)\n        \n        # calculate the loss\n        seg_loss = seg_criterion(seg_preds, masks)\n        \n        \n        seg_optimizer.zero_grad()\n        \n        seg_loss.backward() # Calculate Gradients\n        seg_optimizer.step() # Update Weights\n        \n        # accumulate the loss for the epoch\n        seg_epoch_loss = seg_epoch_loss + seg_loss.item()\n        \n        \n        # (2) PROCESS SEGMENTATION MODEL OUTPUT\n        # ......................................\n        \n        # threshold the predicted segmentation masks\n        thresh_masks = (seg_preds >= THRESHOLD).int()\n        \n        # Multiply the thresholded masks by the RGB images.\n        # The result will be an image with 3 channels.\n        # The wheat heads that are inside the segmentation will\n        # be visible. Everything else on the image will be black.\n        #seg_output_masks = thresh_masks * images\n        \n        # do the multiplication with numpy\n        seg_output_masks = multiply_masks_and_images(images, thresh_masks)\n        \n        # send the masks to the device\n        seg_output_masks = seg_output_masks.to(device, dtype=torch.float)\n        \n        \n        # (3) REGRESSION MODEL\n        # .....................\n        \n        # pass the input through the model\n        reg_preds = reg_model(seg_output_masks)\n        \n        # calculate the loss\n        reg_preds = reg_preds.squeeze(dim=1)\n        \n        reg_loss = reg_criterion(reg_preds, reg_targets)\n        \n        \n        reg_optimizer.zero_grad()\n        \n        reg_loss.backward() # Calculate Gradients\n        reg_optimizer.step() # Update Weights\n        \n        # accumulate the loss for the epoch\n        reg_epoch_loss = reg_epoch_loss + reg_loss.item()\n        \n      \n    # get the avarage loss for the epoch\n    seg_avg_loss = seg_epoch_loss/len(train_loader)\n    reg_avg_loss = reg_epoch_loss/len(train_loader)\n\n    print('Train dice loss:', seg_avg_loss)\n    print('Train mse loss:', reg_avg_loss)\n    #print('\\n')\n    \n    \n    \n    \n    \n    \n    # ====================\n    # VALIDATION\n    # ====================\n    \n    # Set the Mode\n    seg_model.eval()\n    reg_model.eval()\n    \n    # Turn gradient calculations off.\n    torch.set_grad_enabled(False)\n    \n    epoch_loss = 0\n    seg_epoch_loss = 0\n    reg_epoch_loss = 0\n    \n    print('---')\n    print('Val steps:', len(val_loader))\n    \n    for i, batch in enumerate(val_loader):\n        \n        print(i, end=\"\\r\") \n\n        \n        # Get a batch and send to device\n        images, masks, reg_targets = batch\n        \n        images = images.to(device, dtype=torch.float)\n        masks = masks.to(device, dtype=torch.float)\n        reg_targets = reg_targets.squeeze(dim=1)\n        reg_targets = reg_targets.to(device, dtype=torch.float)\n        \n        \n        \n        # (1) SEGMENTATION MODEL\n        # .......................\n        \n        \n        # pass the input through the model\n        seg_preds = seg_model(images)\n        \n        # calculate the loss\n        seg_loss = seg_criterion(seg_preds, masks)\n        \n        \n        # accumulate the loss for the epoch\n        seg_epoch_loss = seg_epoch_loss + seg_loss.item()\n        \n        \n        # (2) PROCESS SEGMENTATION MODEL OUTPUT\n        # ......................................\n        \n        # threshold the predicted segmentation masks\n        thresh_masks = (seg_preds >= THRESHOLD).int()\n        \n        # Multiply the thresholded masks by the RGB images.\n        # The result will be an image with 3 channels.\n        # The wheat heads that are inside the segmentation will\n        # be visible. Everything else on the image will be black.\n        #seg_output_masks = thresh_masks * images\n        \n        # do the multiplication with numpy\n        seg_output_masks = multiply_masks_and_images(images, thresh_masks)\n        \n        # send the masks to the device\n        seg_output_masks = seg_output_masks.to(device, dtype=torch.float)\n        \n        \n        # (3) REGRESSION MODEL\n        # .....................\n        \n        # pass the input through the model\n        reg_preds = reg_model(seg_output_masks)\n        \n        # calculate the loss\n        reg_preds = reg_preds.squeeze(dim=1)\n        \n        reg_loss = reg_criterion(reg_preds, reg_targets)\n        \n        \n        # accumulate the loss for the epoch\n        reg_epoch_loss = reg_epoch_loss + reg_loss.item()\n        \n      \n    # get the avarage loss for the epoch\n    seg_avg_loss = seg_epoch_loss/len(val_loader)\n    reg_avg_loss = reg_epoch_loss/len(val_loader)\n    \n\n    print('Val dice loss:', seg_avg_loss)\n    print('Val mse loss:', reg_avg_loss)\n    \n    \n    \n    # Save the models\n    # ----------------\n    \n    if epoch == 0:\n        \n        # save both models\n        torch.save(seg_model.state_dict(), 'seg_model.pt')\n        torch.save(reg_model.state_dict(), 'reg_model.pt')\n        print('Both models saved.')\n        \n        \n    if epoch != 0:\n        \n        # Be sure to calculate these variables before \n        # appending the new loss values to the lists.\n        best_val_seg_avg_loss = min(seg_val_loss_list)\n        best_val_reg_avg_loss = min(reg_val_loss_list)\n \n        if seg_avg_loss < best_val_seg_avg_loss:\n            # save the model\n            torch.save(seg_model.state_dict(), 'seg_model.pt')\n            print('Val dice loss improved. Saved model as seg_model.pt')\n      \n        if reg_avg_loss < best_val_reg_avg_loss:\n            # save the model\n            torch.save(reg_model.state_dict(), 'reg_model.pt')\n            print('Val mse loss improved. Saved model as reg_model.pt')\n            \n            \n    # append the loss values to the lists   \n    seg_val_loss_list.append(seg_avg_loss)\n    reg_val_loss_list.append(reg_avg_loss)\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check that the models have been saved.\n\n!ls","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make a prediction on the val set","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the saved models\nseg_model.load_state_dict(torch.load('seg_model.pt'))\nreg_model.load_state_dict(torch.load('reg_model.pt'))\n\n# Make a prediction on the val set\nfor i, batch in enumerate(val_loader):\n        \n        print(i, end=\"\\r\") \n\n        \n        # Get a batch and send to device\n        images, masks, reg_targets = batch\n        \n        images = images.to(device, dtype=torch.float)\n        masks = masks.to(device, dtype=torch.float)\n        reg_targets = reg_targets.squeeze(dim=1)\n        reg_targets = reg_targets.to(device, dtype=torch.float)\n        \n        \n        \n        # (1) SEGMENTATION MODEL\n        # .......................\n        \n        \n        # pass the input through the model\n        seg_preds = seg_model(images)\n        \n        \n        \n        # (2) PROCESS SEGMENTATION MODEL OUTPUT\n        # ......................................\n        \n        # threshold the predicted segmentation masks\n        thresh_masks = (seg_preds >= THRESHOLD).int()\n        \n        # Multiply the thresholded masks by the RGB images.\n        # The result will be an image with 3 channels.\n        # The wheat heads that are inside the segmentation will\n        # be visible. Everything else on the image will be black.\n        #seg_output_masks = thresh_masks * images\n        \n        # do the multiplication with numpy\n        seg_output_masks = multiply_masks_and_images(images, thresh_masks)\n        \n        # send the masks to the device\n        seg_output_masks = seg_output_masks.to(device, dtype=torch.float)\n        \n        \n        # (3) REGRESSION MODEL\n        # .....................\n        \n        # pass the input through the model\n        reg_preds = reg_model(seg_output_masks)\n        \n        \n        \n         # Stack the predictions from each batch\n        if i == 0:\n            stacked_images = images\n            stacked_masks = masks\n            #stacked_thresh_masks = thresh_masks\n            stacked_reg_targets = reg_targets\n            \n            \n            stacked_seg_preds = seg_preds\n            stacked_seg_output_masks = seg_output_masks\n            stacked_reg_preds = reg_preds\n            \n        else:\n            \n            stacked_images = torch.cat((stacked_images, images), dim=0)\n            stacked_masks = torch.cat((stacked_masks, masks), dim=0)\n            #stacked_thresh_masks = torch.cat((stacked_thresh_masks, thresh_masks), dim=0)\n            \n            stacked_reg_targets = torch.cat((stacked_reg_targets, reg_targets), dim=0)\n            \n            stacked_seg_preds = torch.cat((stacked_seg_preds, seg_preds), dim=0)\n            stacked_seg_output_masks = torch.cat((stacked_seg_output_masks, seg_output_masks), dim=0)\n            stacked_reg_preds = torch.cat((stacked_reg_preds, reg_preds), dim=0)\n            \n# True    \nprint(stacked_images.shape)\nprint(stacked_masks.shape)\n#print(stacked_thresh_masks.shape)\nprint(stacked_reg_targets.shape)\nprint('\\n')\n\n# Predicted\nprint(stacked_seg_preds.shape)\nprint(stacked_seg_output_masks.shape)\nprint(stacked_reg_preds.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Convert to numpy\n\nnp_stacked_images = stacked_images.cpu().numpy() ##\nnp_stacked_masks = stacked_masks.cpu().numpy() ##\n#np_stacked_thresh_masks = stacked_thresh_masks.cpu().numpy()\n\nnp_stacked_seg_preds = stacked_seg_preds.cpu().numpy()\nnp_stacked_seg_output_masks = stacked_seg_output_masks.cpu().numpy() ##\n\n# reshape to channels first\nnp_stacked_images = np_stacked_images.reshape((-1, 512, 512, 3)) ##\nnp_stacked_masks = np_stacked_masks.reshape((-1, 512, 512, 1)) ##\n#np_stacked_thresh_masks = np_stacked_thresh_masks.reshape((-1, 512, 512, 1))\n\nnp_stacked_seg_preds = np_stacked_seg_preds.reshape((-1, 512, 512, 1))\nnp_stacked_seg_output_masks = np_stacked_seg_output_masks.reshape((-1, 512, 512, 3)) ##\n\n#print(np_stacked_images.shape)\n#print(np_stacked_masks.shape)\n#print(np_stacked_thresh_masks.shape)\n#print(np_stacked_seg_preds.shape)\n#print(np_stacked_seg_output_masks.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Display the seg_model predicted masks and the reg_model input images","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for index in range(1, 4):\n\n    # set up the canvas for the subplots\n    plt.figure(figsize=(15,15))\n    plt.tight_layout()\n    plt.axis('off')\n\n\n\n    plt.subplot(1,4,1)\n    true_image = np_stacked_images[index, :, :, :]\n    plt.imshow(true_image)\n    plt.title('Pre-processed Image', fontsize=14)\n    plt.axis('off')\n\n    plt.subplot(1,4,2)\n    true_mask = np_stacked_masks[index, :, :, 0]\n    plt.imshow(true_mask, cmap='Reds', alpha=0.3)\n    plt.title('True Mask', fontsize=14)\n    plt.axis('off')\n\n    plt.subplot(1,4,3)\n    pred_mask = np_stacked_seg_preds[index, :, :, 0]\n    plt.imshow(pred_mask, cmap='Blues', alpha=0.3)\n    plt.title('Seg model Pred Mask', fontsize=14)\n    plt.axis('off')\n    \n    plt.subplot(1,4,4)\n    pred_mask = np_stacked_seg_output_masks[index, :, :, :]\n    #thresh_mask = np_stacked_thresh_masks[index, :, :, 0]\n    \n    plt.imshow(pred_mask)\n    #plt.imshow(thresh_mask, cmap='Reds', alpha=0.3)\n    plt.title('Reg model input', fontsize=14)\n    plt.axis('off')\n\n    plt.show()\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analyze the regression predictions","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Change to numpy\n\nnp_stacked_reg_targets = stacked_reg_targets.cpu().numpy()\n\nnp_stacked_reg_preds = stacked_reg_preds.cpu().numpy()\nnp_stacked_reg_preds = np_stacked_reg_preds.squeeze()\n\n#print(np_stacked_reg_targets.shape)\n#print(np_stacked_reg_preds.shape)\n\n\n\n# Add the predictions to a dataframe\n\ndf_val['true_count'] = np_stacked_reg_targets\n\ndf_val['pred_count'] = np_stacked_reg_preds\n\n\n# Create more columns\n\ndf_val['pred_error'] = df_val['true_count'] - df_val['pred_count']\n\ndf_val['abs_error'] = df_val['pred_error'].apply(abs)\n\ndf_val['percent_error'] = (df_val['abs_error']/df_val['num_masks'])*100\n\n\n\ncols = ['image_id', 'num_masks', 'true_count', 'pred_count', 'pred_error', 'abs_error']\ndf = df_val[cols]\n\n#df.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images with < 10% count error","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = df_val[df_val['percent_error'] < 10]\n\nmessage = str(len(df)) + ' of 389 val images have < 10% error.'\n\nprint(message)\n\ndf = df.reset_index(drop=True)\n\n# set up the canvas for the subplots\nplt.figure(figsize=(15,15))\n\nplt.subplot(3,4,1)\n\n# Our subplot will contain 3 rows and 3 columns\n# plt.subplot(nrows, ncols, plot_number)\n\n\nfor i in range(1, 10):\n    \n    # image\n    plt.subplot(3,3,i)\n\n    path = base_path + 'train/' + df.loc[i, 'image_id'] + '.jpg'\n    image = plt.imread(path)\n\n    true = df.loc[i, 'num_masks']\n\n    # round the pred\n    pred = round(df.loc[i, 'pred_count'], 0)\n    # convert to type int\n    pred = int(pred)\n\n    source = df.loc[i, 'source']\n    \n    result = 'True: ' + str(true) + ' Pred: ' + str(pred) + ' -- ' + source\n    \n    plt.imshow(image)\n    plt.title(result, fontsize=18)\n    #plt.xlabel(source, fontsize=10)\n    plt.tight_layout()\n    plt.axis('off')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The results above demonstrate that this model is capable of counting with high levels of accuracy.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Images with > 50% count error","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = df_val[df_val['percent_error'] > 50]\n\nmessage = str(len(df)) + ' of 389 val images have > 50% error.'\n\nprint(message)\n\ndf = df.reset_index(drop=True)\n\n# set up the canvas for the subplots\nplt.figure(figsize=(15,15))\n\nplt.subplot(3,4,1)\n\n# Our subplot will contain 3 rows and 3 columns\n# plt.subplot(nrows, ncols, plot_number)\n\n\nfor i in range(1, 10):\n    \n    # image\n    plt.subplot(3,3,i)\n\n    path = base_path + 'train/' + df.loc[i, 'image_id'] + '.jpg'\n    image = plt.imread(path)\n\n    true = df.loc[i, 'num_masks']\n\n    # round the pred\n    pred = round(df.loc[i, 'pred_count'], 0)\n    # convert to type int\n    pred = int(pred)\n\n    source = df.loc[i, 'source']\n    \n    result = 'True: ' + str(true) + ' Pred: ' + str(pred) + ' -- ' + source\n    \n    plt.imshow(image)\n    plt.title(result, fontsize=18)\n    #plt.xlabel(source, fontsize=10)\n    plt.tight_layout()\n    plt.axis('off')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On the above images the wheat heads are not clearly distinguishable at first glance. This could be why the model is producing a counting error greater than 50%. Even a human would struggle to produce accurate wheat head counts for many of these images. What's encouraging is that the number of validation images with an error greater than 50% is low. On different experiements I've observed that this number is usually less than 60 of the 389 validation images.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Val Set Percentage Error","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Total val percentage error:', (df_val['abs_error'].sum()/df_val['true_count'].sum()) * 100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Mean Absolute Error (MAE)\n\nMAE is a metric that's easy to understand and to explain because the scale of the error is the same as the scale of the target.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Put the rows from each region into separate dataframes\n\ndf_usask_1 = df_val[df_val['source'] == 'usask_1']\ndf_arvalis_2 = df_val[df_val['source'] == 'arvalis_2']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check the prediction error stats for the entire val set\nprint('MAE for all validation data:')\nprint('Val set MAE:', df_val['abs_error'].mean())\nprint('')\nprint('MAE by source:')\nprint('usask_1 MAE:', df_usask_1['abs_error'].mean())\nprint('arvalis_2 MAE:', df_arvalis_2['abs_error'].mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save df_val for future analysis\n\ndf_val.to_csv('df_val.csv.gz', compression='gzip', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check if the dataframe was saved.\n!ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reference Notebooks\n\n- Segmentation tutorial by Pavel Yakubovsky<br>\nhttps://github.com/qubvel/segmentation_models.pytorch/blob/master/examples/cars%20segmentation%20(camvid).ipynb\n\n- Simple Cell Counter - Keras CNN<br>\nhttps://www.kaggle.com/vbookshelf/simple-cell-counter-with-web-interface\n\n- Bert as a Microservice - Flask App<br>\nhttps://www.kaggle.com/vbookshelf/bert-as-a-microservice-flask-app","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Helpful Resources\n\n- Segmentation Models Pytorch package<br>\nhttps://github.com/qubvel/segmentation_models.pytorch\n\n- Forum post by Chris Deotte on the segmentation models package.<br>\nhttps://www.kaggle.com/c/severstal-steel-defect-detection/discussion/103367\n\n- Codebasics tutorial on dataframe merging<br>\nhttps://www.youtube.com/watch?v=h4hOPGo4UVU\n\n- Deeplizard Pytorch tutorial<br>\nhttps://deeplizard.com/learn/video/v5cngxo4mIg\n\n- Deeplizard flask video tutorial<br>\nhttps://deeplizard.com/learn/video/SI1hVGvbbZ4\n\n- Julian Nash docker and flask video tutorial<br>\nhttps://www.youtube.com/watch?v=dVEjSmKFUVI\n\n- How to use SSH to connect to a web server<br>\nhttps://www.youtube.com/watch?v=B_lZt9_9UCc\n\n- Free Udemy Docker course<br>\nhttps://www.udemy.com/course/docker-essentials/\n\n\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Conclusion\n\nThis project has been very challenging. Many thanks to the competition hosts for presenting this interesting problem to the Kaggle community. Thanks Kaggle for the free GPU.\n\nThank you for reading.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}