{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook implements a simple starter convolution neural network for the [glasses prediction Kaggle](https://www.kaggle.com/competitions/applications-of-deep-learning-wustl-fall-2023). We begin by creating a simple neural network and detect if we should use a GPU.\n\nI created this notebook from the following prompt in ChatGPT.\n\n```\nCreate a PyTorch GPU-enabled neural network for a Kaggle competition that asks me to predict if people have glasses or not.  The images are stored at this path: /kaggle/input/applications-of-deep-learning-wustl-fall-2023/ . The training data is in the file: /kaggle/input/applications-of-deep-learning-wustl-fall-2023/train.csv. The training data has 3 columns, id, file, and glasses. The field glasses is the target, to be predicted, 0=no glasses, 1=glasses. The file contains the filename of the image that corresponds to each row, the images are named 1.jpg, 2.jpg, etc, which corrisponds to both the id and the file fields. There is also a test dataset that we must generate a submission dataframe for. The test data is in the file /kaggle/input/applications-of-deep-learning-wustl-fall-2023/test.csv, and has the id and file columns, but we need to generate a submit dataframe with just id and glasses(the prediction). Train the neural network, use early stopping and generate the submit dataframe. \n```\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms\nfrom PIL import Image\n\n# Load data\ntrain_df = pd.read_csv('/kaggle/input/applications-of-deep-learning-wustl-fall-2023/train.csv')\ntest_df = pd.read_csv('/kaggle/input/applications-of-deep-learning-wustl-fall-2023/test.csv')\n\nclass GlassesDataset(Dataset):\n    def __init__(self, dataframe, root_dir, transform=None, is_test=False):\n        self.dataframe = dataframe\n        self.root_dir = root_dir\n        self.transform = transform\n        self.is_test = is_test\n    \n    def __len__(self):\n        return len(self.dataframe)\n    \n    def __getitem__(self, idx):\n        img_name = self.root_dir + self.dataframe.iloc[idx, 1]\n        image = Image.open(img_name)\n        if self.transform:\n            image = self.transform(image)\n        \n        if self.is_test:\n            return image\n        else:\n            label = self.dataframe.iloc[idx, 2]\n            return image, label\n\ntransform = transforms.Compose([\n    transforms.Resize((128, 128)),\n    transforms.ToTensor(),\n])","metadata":{"execution":{"iopub.status.busy":"2023-10-08T00:17:55.612479Z","iopub.execute_input":"2023-10-08T00:17:55.612838Z","iopub.status.idle":"2023-10-08T00:18:00.828688Z","shell.execute_reply.started":"2023-10-08T00:17:55.612814Z","shell.execute_reply":"2023-10-08T00:18:00.827716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Structure\n\nNext, we create the structure of the neural network. This neural network is fairly simple.","metadata":{}},{"cell_type":"code","source":"import torch.nn as nn\nimport torch.optim as optim\nimport torch\n\n# CNN model definition\nclass SimpleCNN(nn.Module):\n    def __init__(self):\n        super(SimpleCNN, self).__init__()\n        self.conv1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1)\n        self.fc1 = nn.Linear(16 * 64 * 64, 1)\n\n    def forward(self, x):\n        x = nn.functional.relu(self.conv1(x))\n        x = x.view(-1, 16 * 64 * 64)\n        x = torch.sigmoid(self.fc1(x))\n        return x\n\n# Detect device\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\nmodel = SimpleCNN().to(device)\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model.parameters(), lr=0.001)\n\nprint(f\"Using device: {device}\")\n","metadata":{"execution":{"iopub.status.busy":"2023-10-08T00:18:00.830391Z","iopub.execute_input":"2023-10-08T00:18:00.830915Z","iopub.status.idle":"2023-10-08T00:18:04.110415Z","shell.execute_reply.started":"2023-10-08T00:18:00.830881Z","shell.execute_reply":"2023-10-08T00:18:04.109447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will make use of early stopping. This early stopping class was generated by ChatGPT.","metadata":{}},{"cell_type":"code","source":"import torch\n\nclass EarlyStopping:\n    \"\"\"Early stops the training if validation loss doesn't improve after a given patience.\"\"\"\n    def __init__(self, patience=7, verbose=False, delta=0, path='checkpoint.pt'):\n        \"\"\"\n        Args:\n            patience (int): How long to wait after last time validation loss improved.\n            verbose (bool): If True, prints a message for each validation loss improvement.\n            delta (float): Minimum change in the monitored quantity to qualify as an improvement.\n            path (str): Path for the checkpoint to be saved to.\n        \"\"\"\n        self.patience = patience\n        self.verbose = verbose\n        self.counter = 0\n        self.best_score = None\n        self.early_stop = False\n        self.val_loss_min = float('inf')\n        self.delta = delta\n        self.path = path\n\n    def __call__(self, val_loss, model):\n        score = -val_loss\n\n        if self.best_score is None:\n            self.best_score = score\n            self.save_checkpoint(val_loss, model)\n        elif score < self.best_score + self.delta:\n            self.counter += 1\n            print(f'EarlyStopping counter: {self.counter} out of {self.patience}')\n            if self.counter >= self.patience:\n                self.early_stop = True\n        else:\n            self.best_score = score\n            self.save_checkpoint(val_loss, model)\n            self.counter = 0\n\n    def save_checkpoint(self, val_loss, model):\n        '''Saves model when validation loss decrease.'''\n        if self.verbose:\n            print(f'Validation loss decreased ({self.val_loss_min:.6f} --> {val_loss:.6f}).  Saving model ...')\n        torch.save(model.state_dict(), self.path)\n        self.val_loss_min = val_loss\n","metadata":{"execution":{"iopub.status.busy":"2023-10-08T00:18:04.111982Z","iopub.execute_input":"2023-10-08T00:18:04.112606Z","iopub.status.idle":"2023-10-08T00:18:04.121391Z","shell.execute_reply.started":"2023-10-08T00:18:04.112571Z","shell.execute_reply":"2023-10-08T00:18:04.120507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Model\nNow we train the model, with early stopping.","metadata":{}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader, DataLoader, random_split\nfrom torchvision import transforms\nfrom PIL import Image\nimport pandas as pd\nimport os\n\nbatch_size = 32\n\n# Instantiate dataset, model, and early stopping\ntransform = transforms.Compose([\n    transforms.Resize((64, 64)),\n    transforms.ToTensor(),\n])\n\n# Load your datasets\nfull_dataset = GlassesDataset(\n    dataframe=train_df, \n    root_dir='/kaggle/input/applications-of-deep-learning-wustl-fall-2023/', \n    transform=transform)\n\n# Split into training and validation datasets\ntrain_size = int(0.8 * len(full_dataset))\nvalid_size = len(full_dataset) - train_size\ntrain_dataset, valid_dataset = random_split(full_dataset, [train_size, valid_size])\n\ntrain_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\nvalidation_loader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False)\n\n\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\nmodel = SimpleCNN().to(device)\nprint(f\"Using device: {device}\")\n\ncriterion = nn.BCELoss()\noptimizer = optim.Adam(model.parameters(), lr=0.001)\nearly_stopping = EarlyStopping(patience=3, verbose=True)\n\n# Training loop with early stopping\nnum_epochs = 25\nfor epoch in range(num_epochs):\n    model.train()\n    running_loss = 0.0\n    \n    for inputs, labels in train_loader:\n        inputs, labels = inputs.to(device), labels.to(device)\n        optimizer.zero_grad()\n        outputs = model(inputs)\n        loss = criterion(outputs, labels.float().view(-1, 1))\n        loss.backward()\n        optimizer.step()\n        running_loss += loss.item() * inputs.size(0)\n    \n    epoch_loss = running_loss / len(train_loader.dataset)\n    \n    # Validation phase\n    model.eval()  # Set model to evaluation mode\n    \n    val_loss = 0.0\n    with torch.no_grad():  # Turn off gradients for validation\n        for inputs, labels in validation_loader:\n            inputs, labels = inputs.to(device), labels.to(device)\n            \n            outputs = model(inputs)  # Forward pass\n            loss = criterion(outputs, labels.float().view(-1, 1))\n            \n            val_loss += loss.item() * inputs.size(0)\n    \n    val_loss = val_loss / len(validation_loader.dataset)\n    \n    print(f\"Epoch {epoch}/{num_epochs} => \"\n        f\"Validation loss: {val_loss:.4f}\")\n    \n    # early stopping\n    early_stopping(val_loss, model)\n    if early_stopping.early_stop:\n        print(\"Early stopping\")\n        break\n        \n\n# Load the best model weights\nmodel.load_state_dict(torch.load('checkpoint.pt'))\n","metadata":{"execution":{"iopub.status.busy":"2023-10-08T00:18:04.123774Z","iopub.execute_input":"2023-10-08T00:18:04.124095Z","iopub.status.idle":"2023-10-08T00:37:00.440178Z","shell.execute_reply.started":"2023-10-08T00:18:04.124062Z","shell.execute_reply":"2023-10-08T00:37:00.439197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generate Submit File\nWe can now generate the submission file that you will send to Kaggle","metadata":{}},{"cell_type":"code","source":"submit_dataset = GlassesDataset(\n    dataframe=test_df, \n    root_dir='/kaggle/input/applications-of-deep-learning-wustl-fall-2023/', \n    transform=transform,\n    is_test=True,\n)\nsubmit_loader = DataLoader(submit_dataset, batch_size=batch_size, shuffle=False)\n\n# Set model to evaluation mode\nmodel.eval()\n\n# Store predictions\npredictions = []\n\n# Disable gradient calculation and make predictions\nwith torch.no_grad():\n    for inputs in submit_loader:\n        inputs = inputs.to(device)  # Send inputs to the device\n        outputs = model(inputs).flatten().cpu().numpy()\n        predictions.extend(outputs)\n\n# Create a DataFrame and save to CSV\nsubmission_df = pd.DataFrame({\n    'id': test_df.id,\n    'glasses': predictions\n})\n\nsubmission_df.to_csv('submission.csv', index=False) \n","metadata":{"execution":{"iopub.status.busy":"2023-10-08T00:37:00.441580Z","iopub.execute_input":"2023-10-08T00:37:00.442514Z","iopub.status.idle":"2023-10-08T00:37:50.429418Z","shell.execute_reply.started":"2023-10-08T00:37:00.442476Z","shell.execute_reply":"2023-10-08T00:37:50.428095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df","metadata":{"execution":{"iopub.status.busy":"2023-10-08T00:37:50.430927Z","iopub.execute_input":"2023-10-08T00:37:50.431387Z","iopub.status.idle":"2023-10-08T00:37:50.447529Z","shell.execute_reply.started":"2023-10-08T00:37:50.431331Z","shell.execute_reply":"2023-10-08T00:37:50.446400Z"},"trusted":true},"execution_count":null,"outputs":[]}]}