\nThis project presents a comprehensive study on training Convolutional Neural Networks (CNNs) for image classification using the CIFAR-10 dataset. \nThe objective is to evaluate and compare the performance of various optimization algorithms, including SGD, SGD with Momentum, Nesterov Accelerated Gradient (NAG), Adam, AdamW, Adagrad, and RMSprop. \n\nA consistent CNN architecture is employed across all experiments to ensure fair comparison. Each optimizer is analyzed based on convergence speed, training stability, and validation accuracy. \nExperimental results demonstrate that momentum-based optimizers, particularly SGD with NAG, achieve superior performance with faster convergence and higher accuracy. \nAdaptive optimizers such as Adam and AdamW provide rapid initial learning and competitive results, while Adagrad shows strong early progress but plateaus due to aggressive learning rate decay. \nRMSprop exhibits unstable behavior under the chosen hyperparameters.\n\nThis study highlights the critical role of optimizer selection in deep learning, showing that momentum-based methods offer the best balance between efficiency, stability, and generalization performance for image classification tasks.\n
\n\n\nA random seed generator is used to control randomness in machine learning experiments. \nIn deep learning, processes such as weight initialization, data shuffling, and dropout depend on random numbers, which can lead to different results each time the model is trained. \nBy fixing a random seed, we ensure that the same sequence of random numbers is generated every run, making the experiment reproducible. \nThis is particularly important in this study because it guarantees a fair comparison between different optimization algorithms, ensuring that all models start from identical initial conditions.\n
\n\n\nIn this step, the CIFAR-10 dataset is loaded using torchvision. \nData augmentation techniques such as random horizontal flipping and random cropping are applied to improve the model’s ability to generalize by exposing it to slightly varied versions of the same images. \nThe images are then converted to tensors and normalized using the CIFAR-10 dataset statistics to stabilize training. \nFinally, the dataset is split into training (80%) and validation (20%) sets, and PyTorch DataLoader is used to efficiently load batches of data during training.\n
\n\n\nA Convolutional Neural Network (CNN) is designed to extract hierarchical features from images. \nIn this architecture, multiple convolutional layers are used to learn spatial patterns such as edges, textures, and shapes from the CIFAR-10 images. \nEach convolution layer is followed by Batch Normalization to stabilize and accelerate training, and ReLU activation to introduce non-linearity. \nMax Pooling layers reduce spatial dimensions and help the network focus on the most important features. \nTo reduce overfitting, Dropout layers are applied before the fully connected layers. \nFinally, the extracted features are passed through fully connected layers to produce predictions for the 10 CIFAR-10 classes.\n
\n\n| Layer | \nOperation | \nOutput Shape | \nTrainable Parameters | \n
|---|---|---|---|
| Conv Block 1 | \nConv2D (3 → 32, 3×3) + BatchNorm + ReLU | \n32 × 32 × 32 | \n960 | \n
| Conv Block 2 | \nConv2D (32 → 64, 3×3) + BatchNorm + ReLU + MaxPool | \n64 × 16 × 16 | \n18624 | \n
| Conv Block 3 | \nConv2D (64 → 128, 3×3) + BatchNorm + ReLU | \n128 × 16 × 16 | \n74112 | \n
| Conv Block 4 | \nConv2D (128 → 256, 3×3) + BatchNorm + ReLU + MaxPool | \n256 × 8 × 8 | \n295680 | \n
| Flatten | \nFeature Flattening | \n16384 | \n0 | \n
| FC Layer 1 | \nLinear (16384 → 512) + Dropout | \n512 | \n8,389,120 | \n
| FC Layer 2 | \nLinear (512 → 128) + Dropout | \n128 | \n65,664 | \n
| Output Layer | \nLinear (128 → 10) | \n10 | \n1,290 | \n
| Component | \nConfiguration | \nPurpose | \n
|---|---|---|
| Model Architecture | \nCustom CNN with BatchNorm and Dropout | \nFeature extraction and classification | \n
| Loss Function | \nCrossEntropyLoss | \nHandles multi-class classification | \n
| Optimizers | \nSGD, SGD+Momentum, NAG, Adam, AdamW, RMSprop, Adagrad | \nCompare optimization techniques | \n
| Maximum Epochs | \n100 | \nUpper bound for training iterations | \n
| Early Stopping | \nPatience = 5 | \nStops training when validation loss stops improving | \n
| Model Checkpointing | \nBest model saved per optimizer | \nEnsures best validation performance is preserved | \n
| Evaluation Metrics | \nValidation Loss and Accuracy | \nMeasure model performance | \n
| Optimizer | \nConvergence Behaviour | \nBest Validation Accuracy | \nObservation | \n
|---|---|---|---|
| SGD | \nSlow convergence | \n~65% | \nStable learning but requires many epochs | \n
| SGD + Momentum | \nFast and stable convergence | \n~84–85% | \nBest performance due to accelerated gradients | \n
| SGD + NAG | \nFaster than SGD | \n~71–72% | \nPredictive gradient improves update direction | \n
| Adam | \nFast initial learning | \n~80% | \nAdaptive learning rates speed up convergence | \n
| AdamW | \nFast and stable | \n~80% | \nImproved weight decay regularization | \n
| Adagrad | \nModerate convergence | \n~67% | \nLearning rate decays quickly | \n
| RMSprop | \nUnstable training | \n~10% | \nFailed to converge with current settings | \n
\nFrom the comparison, SGD with Momentum achieved the best overall performance, reaching the highest validation accuracy while maintaining stable convergence. \nAdaptive optimizers such as Adam and AdamW showed faster initial learning but slightly lower final accuracy compared to momentum-based SGD. \nAdagrad demonstrated moderate performance due to aggressive learning rate decay, while RMSprop failed to converge effectively with the chosen hyperparameters. \nThese results highlight the importance of momentum in accelerating gradient descent and improving training efficiency for CNN models.\n
\n\n\nAfter training and comparing different optimization algorithms, the final step is to evaluate the model on the \nCIFAR-10 test dataset. This dataset contains unseen images that were not used during training or validation, \nallowing us to measure the model’s true generalization performance. The images are converted into tensors and \nnormalized using the CIFAR-10 dataset statistics to maintain consistency with the training pipeline. \nThe dataset is then loaded using a PyTorch DataLoader, which enables efficient batch-wise processing during \nmodel evaluation.\n
\n\n| Optimizer | \nAccuracy | \nPrecision | \nRecall | \nF1 Score | \nAUC | \n
|---|---|---|---|---|---|
| SGD | 0.6486 | 0.6445 | 0.6486 | 0.6445 | 0.9426 |
| SGD + Momentum | 0.8458 | 0.8478 | 0.8458 | 0.8456 | 0.9861 |
| SGD + NAG | 0.7194 | 0.7264 | 0.7194 | 0.7201 | 0.9612 |
| Adam | 0.8021 | 0.8035 | 0.8021 | 0.8007 | 0.9779 |
| AdamW | 0.8002 | 0.8045 | 0.8002 | 0.8005 | 0.9778 |
| Adagrad | 0.6756 | 0.6967 | 0.6756 | 0.6805 | 0.9515 |
| RMSprop | 0.1000 | 0.0100 | 0.1000 | 0.0182 | 0.5000 |
\n\nThe final comparison shows clear differences in the performance of optimization algorithms when applied to the CNN model on the CIFAR-10 dataset.\n\n
\n\nThe confusion matrices provide deeper insight into how different optimization algorithms affect the classification behavior of the CNN model on the CIFAR-10 dataset. Each matrix illustrates how often images from a true class are predicted as another class.\n\n
\n\n\n\nOverall, the confusion matrices confirm that SGD with Momentum provides the most balanced and accurate classification performance for this CNN architecture. Adaptive optimizers such as Adam and AdamW also perform well, while vanilla SGD and RMSprop show comparatively weaker class separation.\n\n
\n\n\n\nThis study compared multiple optimization algorithms for training a Convolutional Neural Network on the CIFAR-10 image classification dataset. \nThe results demonstrate that optimization strategy plays a crucial role in model convergence, classification accuracy, and overall training stability.\n\n
\n\n\n\nOverall, the experiments show that incorporating momentum-based optimization significantly improves training efficiency and classification performance for CNN models on CIFAR-10. These findings emphasize the importance of selecting an appropriate optimizer when training deep learning models.\n\n
\n\n