{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## PolyNeuronNet","metadata":{}},{"cell_type":"markdown","source":"### A new architecture just experimenting with(explanation is given below), if you like this please upvote\n","metadata":{}},{"cell_type":"markdown","source":"### Imports and Libraries\n\n","metadata":{}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.optim as optim\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.cuda.amp import autocast, GradScaler  # For mixed precision training","metadata":{"execution":{"iopub.status.busy":"2024-10-11T18:49:27.707711Z","iopub.execute_input":"2024-10-11T18:49:27.708237Z","iopub.status.idle":"2024-10-11T18:49:27.715658Z","shell.execute_reply.started":"2024-10-11T18:49:27.708177Z","shell.execute_reply":"2024-10-11T18:49:27.714242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Custom Dataset Class\n","metadata":{}},{"cell_type":"code","source":"# Custom dataset class for loading data from CSV\nclass DigitRecognizerDataset(Dataset):\n    def __init__(self, data, labels=None):\n        self.data = torch.tensor(data.values, dtype=torch.float32) / 255.0  # Normalize pixel values to [0,1]\n        self.labels = torch.tensor(labels.values, dtype=torch.long) if labels is not None else None\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        if self.labels is not None:\n            return self.data[idx], self.labels[idx]\n        else:\n            return self.data[idx]\n","metadata":{"execution":{"iopub.status.busy":"2024-10-11T18:49:27.718337Z","iopub.execute_input":"2024-10-11T18:49:27.718849Z","iopub.status.idle":"2024-10-11T18:49:27.730195Z","shell.execute_reply.started":"2024-10-11T18:49:27.718794Z","shell.execute_reply":"2024-10-11T18:49:27.729025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Polynomial Activation Function","metadata":{}},{"cell_type":"code","source":"# Define the custom polynomial activation functions\nclass PolynomialActivation(nn.Module):\n    def __init__(self, degree=1):\n        super(PolynomialActivation, self).__init__()\n        self.degree = degree\n        self.coefficients = nn.Parameter(torch.randn(degree + 1))\n\n    def forward(self, x):\n        powers = [x ** i for i in range(self.degree, -1, -1)]  # Create the powers in one go\n        powers = torch.stack(powers, dim=-1)  # Stack them along a new dimension\n        return torch.sum(powers * self.coefficients, dim=-1)  # Apply the coefficients and sum","metadata":{"execution":{"iopub.status.busy":"2024-10-11T18:49:27.732071Z","iopub.execute_input":"2024-10-11T18:49:27.732946Z","iopub.status.idle":"2024-10-11T18:49:27.74292Z","shell.execute_reply.started":"2024-10-11T18:49:27.732875Z","shell.execute_reply":"2024-10-11T18:49:27.741704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Input Transformation:** The input images are flattened into a 784-dimensional vector, one vector per image in the batch.\n\n**Hidden Layer Calculation:**\nFor each input feature, a polynomial activation is applied across the hidden neurons. This is done in a vectorized manner using torch.stack(), ensuring efficient computation.\nThe resulting hidden outputs are reshaped for use in the next layer.\n\n**Output Calculation:**\nThe hidden layer outputs are then multiplied by the learned output weights using matrix multiplication (@).\nThis operation computes the final output, which represents the logits (unscaled probabilities) for each of the 10 digits (0-9).","metadata":{}},{"cell_type":"markdown","source":"Each input feature is transformed using a custom activation function (PolynomialActivation). This activation function learns coefficients for each degree of the polynomial, allowing for complex transformations. The polynomial is computed as:\n\n$$f(x)=c_{0}+c_{1}x+c_{2}x^{2}+...+c_{d}x^{d}$$\n\nWhere d is the degree of the polynomial and c_0, c_1, c_2, \\ldots are the learned coefficients. This provides the model with the ability to capture highly non-linear relationships in the input data.","metadata":{}},{"cell_type":"markdown","source":"### Custom MNIST Model\n","metadata":{}},{"cell_type":"code","source":"# Define the custom neural network architecture\nclass CustomMNISTModel(nn.Module):\n    def __init__(self, input_size=784, hidden_multiplier=3, output_size=10, degree=3):\n        super(CustomMNISTModel, self).__init__()\n        self.n = input_size  # Number of input neurons (784 for 28x28 pixels)\n        self.m = hidden_multiplier  # Number of hidden neurons per input\n        self.degree = degree  # Degree of polynomial activation function\n\n        # Hidden layers with polynomial activations for each input neuron\n        self.hidden_layers = nn.ModuleList()\n        for i in range(self.n):\n            self.hidden_layers.append(PolynomialActivation(degree=self.degree))\n\n        # Output layer weights\n        self.output_weights = nn.ParameterList([nn.Parameter(torch.randn(self.n)) for _ in range(output_size)])\n\n    def forward(self, x):\n        x = x.view(-1, self.n)  # Flatten input to shape [batch_size, n]\n\n        # Apply the polynomial activation to all neurons at once (no for-loop needed)\n        hidden_outputs = torch.stack([self.hidden_layers[i](x[:, i:i + 1]) for i in range(self.n)], dim=1)\n        hidden_outputs = hidden_outputs.view(x.size(0), -1)\n\n        # Output layer calculation\n        outputs = hidden_outputs @ torch.stack([self.output_weights[i] for i in range(10)], dim=1)\n\n        return outputs","metadata":{"execution":{"iopub.status.busy":"2024-10-11T18:49:27.745389Z","iopub.execute_input":"2024-10-11T18:49:27.745826Z","iopub.status.idle":"2024-10-11T18:49:27.759111Z","shell.execute_reply.started":"2024-10-11T18:49:27.745764Z","shell.execute_reply":"2024-10-11T18:49:27.757819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Input Size (784):** Since MNIST images are 28x28 pixels, the input size to the model is flattened into a 784-dimensional vector.\n\n**Hidden Multiplier:** Controls how many hidden neurons are created per input feature. In this case, the number of hidden neurons per feature is determined by m.\n\n**Polynomial Activation:** Each input feature is passed through a custom activation function (polynomial) that can learn complex non-linear transformations. The degree of the polynomial is adjustable.\n\n**Output Layer:** The model has 10 output neurons (one for each digit), and the output layer is fully connected, with learned weights for each connection between hidden and output neurons.","metadata":{}},{"cell_type":"markdown","source":"### Training Function with Mixed Precision\n","metadata":{}},{"cell_type":"code","source":"# Training function with mixed precision\ndef train(model, device, train_loader, optimizer, scaler, epoch, log_interval=40):\n    model.train()\n    loss_fn = nn.CrossEntropyLoss()\n    train_loss = 0\n    correct = 0\n    total = 0\n\n    for batch_idx, (data, target) in enumerate(train_loader):\n        data, target = data.to(device), target.to(device)\n        optimizer.zero_grad()\n\n        # Mixed precision training\n        with autocast():\n            output = model(data)\n            loss = loss_fn(output, target)\n\n        scaler.scale(loss).backward()  # Gradient scaling for stability\n        scaler.step(optimizer)\n        scaler.update()\n\n        train_loss += loss.item() * data.size(0)\n        pred = output.argmax(dim=1, keepdim=True)\n        correct += pred.eq(target.view_as(pred)).sum().item()\n        total += target.size(0)\n\n        if batch_idx % log_interval == 0:\n            print(f'Train Epoch: {epoch} [{batch_idx * len(data)}/{len(train_loader.dataset)} ({100. * batch_idx / len(train_loader):.0f}%)]\\tLoss: {loss.item():.6f}')\n\n    avg_loss = train_loss / total\n    accuracy = 100. * correct / total\n    return avg_loss, accuracy","metadata":{"execution":{"iopub.status.busy":"2024-10-11T18:49:27.760984Z","iopub.execute_input":"2024-10-11T18:49:27.761776Z","iopub.status.idle":"2024-10-11T18:49:27.775864Z","shell.execute_reply.started":"2024-10-11T18:49:27.761718Z","shell.execute_reply":"2024-10-11T18:49:27.774654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Testing Function\n","metadata":{}},{"cell_type":"code","source":"# Testing function\ndef test(model, device, test_loader):\n    model.eval()\n    test_loss = 0\n    correct = 0\n    loss_fn = nn.CrossEntropyLoss(reduction='sum')\n    with torch.no_grad():\n        for data, target in test_loader:\n            data, target = data.to(device), target.to(device)\n            output = model(data)\n            test_loss += loss_fn(output, target).item()\n            pred = output.argmax(dim=1, keepdim=True)\n            correct += pred.eq(target.view_as(pred)).sum().item()\n\n    test_loss /= len(test_loader.dataset)\n    accuracy = 100. * correct / len(test_loader.dataset)\n    print(f'\\nTest set: Average loss: {test_loss:.4f}, Accuracy: {correct}/{len(test_loader.dataset)} ({accuracy:.0f}%)\\n')\n    return test_loss, accuracy","metadata":{"execution":{"iopub.status.busy":"2024-10-11T18:49:27.777479Z","iopub.execute_input":"2024-10-11T18:49:27.777866Z","iopub.status.idle":"2024-10-11T18:49:27.793363Z","shell.execute_reply.started":"2024-10-11T18:49:27.777824Z","shell.execute_reply":"2024-10-11T18:49:27.791975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Submission Generation for Kaggle\n","metadata":{}},{"cell_type":"code","source":"# Prepare submission on test data for Kaggle competition\ndef create_submission(model, device, test_loader):\n    model.eval()\n    test_preds = []\n    with torch.no_grad():\n        for data in test_loader:\n            data = data.to(device)\n            output = model(data)\n            preds = output.argmax(dim=1, keepdim=True).cpu().numpy()\n            test_preds.extend(preds)\n\n    submission = pd.DataFrame({\"ImageId\": np.arange(1, len(test_preds) + 1), \"Label\": np.array(test_preds).flatten()})\n    submission.to_csv(\"submission.csv\", index=False)\n    print(\"Submission file created: submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-10-11T18:49:27.79485Z","iopub.execute_input":"2024-10-11T18:49:27.795337Z","iopub.status.idle":"2024-10-11T18:49:27.807474Z","shell.execute_reply.started":"2024-10-11T18:49:27.795293Z","shell.execute_reply":"2024-10-11T18:49:27.806426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the CSV data and return as DataLoader\ndef load_data_from_csv(batch_size, train=True):\n    if train:\n        data = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\n        labels = data['label']\n        data = data.drop(columns=['label'])\n        dataset = DigitRecognizerDataset(data, labels)\n    else:\n        data = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\n        dataset = DigitRecognizerDataset(data)\n\n    loader = DataLoader(dataset, batch_size=batch_size, shuffle=train, pin_memory=True, num_workers=4)\n    return loader","metadata":{"execution":{"iopub.status.busy":"2024-10-11T18:49:27.91668Z","iopub.execute_input":"2024-10-11T18:49:27.917135Z","iopub.status.idle":"2024-10-11T18:49:27.924916Z","shell.execute_reply.started":"2024-10-11T18:49:27.917089Z","shell.execute_reply":"2024-10-11T18:49:27.923684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Main Function\n","metadata":{}},{"cell_type":"code","source":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n# Main function to set up training\ndef main():\n    batch_size = 256  # Increase batch size for speed\n    epochs = 100\n    learning_rate = 0.0005  # Adjust learning rate\n    hidden_multiplier = 5  # m\n    degree = 3  # Polynomial degree\n\n    # Initialize model, optimizer, and device\n    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n    model = CustomMNISTModel(input_size=784, hidden_multiplier=hidden_multiplier, degree=degree).to(device)\n    optimizer = optim.Adam(model.parameters(), lr=learning_rate)  # Faster optimizer (Adam)\n    scaler = GradScaler()  # Mixed precision scaler\n\n    # Load data from CSV\n    train_loader = load_data_from_csv(batch_size=batch_size, train=True)\n    test_loader = load_data_from_csv(batch_size=batch_size, train=False)\n\n    train_losses, train_accuracies = [], []\n    test_losses, test_accuracies = [], []\n\n    # Training and testing loop\n    for epoch in range(1, epochs + 1):\n        train_loss, train_acc = train(model, device, train_loader, optimizer, scaler, epoch)\n        test_loss, test_acc = test(model, device, train_loader)\n\n        # Store the results for plotting\n        train_losses.append(train_loss)\n        train_accuracies.append(train_acc)\n        test_losses.append(test_loss)\n        test_accuracies.append(test_acc)\n\n    # Plot accuracy and loss graphs\n    epochs_range = range(1, epochs + 1)\n    plt.figure(figsize=(12, 5))\n    plt.subplot(1, 2, 1)\n    plt.plot(epochs_range, train_losses, label='Training Loss')\n    plt.plot(epochs_range, test_losses, label='Test Loss')\n    plt.xlabel('Epochs')\n    plt.ylabel('Loss')\n    plt.title('Loss over Epochs')\n    plt.legend()\n\n    plt.subplot(1, 2, 2)\n    plt.plot(epochs_range, train_accuracies, label='Training Accuracy')\n    plt.plot(epochs_range, test_accuracies, label='Test Accuracy')\n    plt.xlabel('Epochs')\n    plt.ylabel('Accuracy (%)')\n    plt.title('Accuracy over Epochs')\n    plt.legend()\n\n    plt.show()\n\n    # Create submission CSV for Kaggle\n    create_submission(model, device, test_loader)\n","metadata":{"execution":{"iopub.status.busy":"2024-10-11T18:49:27.927338Z","iopub.execute_input":"2024-10-11T18:49:27.927785Z","iopub.status.idle":"2024-10-11T18:49:27.944141Z","shell.execute_reply.started":"2024-10-11T18:49:27.927741Z","shell.execute_reply":"2024-10-11T18:49:27.942903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training Starts here","metadata":{}},{"cell_type":"code","source":"main()","metadata":{"execution":{"iopub.status.busy":"2024-10-11T18:49:27.945622Z","iopub.execute_input":"2024-10-11T18:49:27.945991Z","iopub.status.idle":"2024-10-11T19:03:07.585971Z","shell.execute_reply.started":"2024-10-11T18:49:27.945938Z","shell.execute_reply":"2024-10-11T19:03:07.584127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}