{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":31192,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pathlib\n\nimport numpy as np\nimport pandas as pd\nfrom sklearn import decomposition, model_selection, preprocessing, pipeline\nimport torch\nfrom torch import nn, utils\nimport torchmetrics","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:15:05.628611Z","iopub.execute_input":"2025-12-12T11:15:05.628831Z","iopub.status.idle":"2025-12-12T11:15:18.941785Z","shell.execute_reply.started":"2025-12-12T11:15:05.628811Z","shell.execute_reply":"2025-12-12T11:15:18.941129Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 0. Load the data (5 pts)\n\nComplete the code in the cells below to load the MNIST dataset in to a Pandas `DataFrame` using the `pd.read_csv` function. Be sure to use the `np.uint8` data-type to represent the pixel values as non-negative integers.","metadata":{}},{"cell_type":"code","source":"INPUT_DIR = pathlib.Path(\"/kaggle/input\")\nOUTPUT_DIR = pathlib.Path(\"/kaggle/working\")\nTEMP_DIR = pathlib.Path(\"/kaggle/temp\")\nDATA_DIR = INPUT_DIR / \"digit-recognizer\"\n\nRANDOM_STATE = np.random.RandomState(42)\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:15:18.943286Z","iopub.execute_input":"2025-12-12T11:15:18.943658Z","iopub.status.idle":"2025-12-12T11:15:19.023743Z","shell.execute_reply.started":"2025-12-12T11:15:18.943639Z","shell.execute_reply":"2025-12-12T11:15:19.023000Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pd.read_csv?","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"mnist_df = pd.read_csv(\n    DATA_DIR / \"train.csv\",\n    dtype=np.uint8,\n)\n\ntest_features_df = pd.read_csv(\n    DATA_DIR / \"test.csv\",\n    dtype=np.uint8,\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:15:19.027388Z","iopub.execute_input":"2025-12-12T11:15:19.027640Z","iopub.status.idle":"2025-12-12T11:15:22.629610Z","shell.execute_reply.started":"2025-12-12T11:15:19.027613Z","shell.execute_reply":"2025-12-12T11:15:22.628755Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1. Simulate a partially labeled dataset (5 pts)\n\nComplete the code in the cell below to create a simulated, partially labeled version of the MNIST dataset.\n\nYou must use the `model_selection.train_test_split` function from Scikit-Learn to split the provided training data into \"labeled\" and \"unlabeled\" pieces using stratified sampling. The unlabeled piece should contain 80% of the original training data. ","metadata":{}},{"cell_type":"code","source":"model_selection.train_test_split?","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"labeled_mnist_df, unlabeled_mnist_df = (\n    model_selection.train_test_split(\n        mnist_df,\n        stratify=mnist_df[\"label\"],\n        test_size=0.8,\n        random_state=RANDOM_STATE\n    )\n)\n\n# we drop the label to simulate an unlabeled subset of data\nunlabeled_features_df = unlabeled_mnist_df.drop(\"label\", axis=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:16:13.763867Z","iopub.execute_input":"2025-12-12T11:16:13.764540Z","iopub.status.idle":"2025-12-12T11:16:13.932898Z","shell.execute_reply.started":"2025-12-12T11:16:13.764510Z","shell.execute_reply":"2025-12-12T11:16:13.932031Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# should have 8400 rows and 785 columns!\nlabeled_mnist_df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:16:18.023489Z","iopub.execute_input":"2025-12-12T11:16:18.024242Z","iopub.status.idle":"2025-12-12T11:16:18.057929Z","shell.execute_reply.started":"2025-12-12T11:16:18.024176Z","shell.execute_reply":"2025-12-12T11:16:18.057119Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# should have 33600 rows and 784 cols!\nunlabeled_features_df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:16:21.935117Z","iopub.execute_input":"2025-12-12T11:16:21.935866Z","iopub.status.idle":"2025-12-12T11:16:21.960835Z","shell.execute_reply.started":"2025-12-12T11:16:21.935838Z","shell.execute_reply":"2025-12-12T11:16:21.960162Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Create a train/val split for the labeled data (5 pts)\n\nComplete the code in the cell below to split your labeled training data into training and validation sets using stratified sampling. The validation set should contain 10% of the labeled training data. ","metadata":{}},{"cell_type":"code","source":"train_df, val_df = (\n    model_selection.train_test_split(\n        labeled_mnist_df,\n        stratify=labeled_mnist_df[\"label\"],\n        test_size=0.1,\n        random_state=RANDOM_STATE\n    )\n)\n\ntrain_features_df = train_df.drop(\"label\", axis=1)\ntrain_labels = train_df.loc[:, \"label\"]\n\nval_features_df = val_df.drop(\"label\", axis=1)\nval_labels = val_df.loc[:, \"label\"]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:16:58.031750Z","iopub.execute_input":"2025-12-12T11:16:58.032411Z","iopub.status.idle":"2025-12-12T11:16:58.059903Z","shell.execute_reply.started":"2025-12-12T11:16:58.032375Z","shell.execute_reply":"2025-12-12T11:16:58.059044Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# should have 7560 rows and 784 cols!\ntrain_features_df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:17:00.159344Z","iopub.execute_input":"2025-12-12T11:17:00.160053Z","iopub.status.idle":"2025-12-12T11:17:00.183797Z","shell.execute_reply.started":"2025-12-12T11:17:00.160024Z","shell.execute_reply":"2025-12-12T11:17:00.183048Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# should have 840 rows and 784 cols!\nval_features_df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:17:03.243419Z","iopub.execute_input":"2025-12-12T11:17:03.244098Z","iopub.status.idle":"2025-12-12T11:17:03.269212Z","shell.execute_reply.started":"2025-12-12T11:17:03.244074Z","shell.execute_reply":"2025-12-12T11:17:03.268503Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Prepare the datasets (20 pts)","metadata":{}},{"cell_type":"markdown","source":"### Converting DataFrames to Tensors (10 pts)\n\nComplete the code in the cell below in order to define a Scikit Learn preprocessing pipeline for your features that has the following steps.\n\n1. Uses a `StandardScaler` to standardize the features.\n2. Uses a `FunctionTransformer` to convert a Numpy Array to a Torch Tensor with a `torch.float32` dtype.\n\nNext, define a Scikit Learn preprocessing pipeline for your target that has the following steps.\n\n1. Uses a `FunctionTransformer` to convert a Pandas Series to a Numpy Array.\n2. Uses a `FunctionTransformer` to convert a NumPy Array to a Torch Tensor with a `torch.int64` dtype.\n\n\n\n","metadata":{}},{"cell_type":"code","source":"features_preprocessing_pipeline = pipeline.make_pipeline(\n    preprocessing.StandardScaler(),\n    preprocessing.FunctionTransformer(\n        func=lambda x: torch.tensor(x, dtype=torch.float32)\n    )\n)\n\ntarget_preprocessing_pipeline = pipeline.make_pipeline(\n    preprocessing.FunctionTransformer(\n        func=lambda x: x.values\n    ),\n    preprocessing.FunctionTransformer(\n        func=lambda x: torch.tensor(x, dtype=torch.int64)\n    )\n)\n\ntrain_features_tensor = (\n    features_preprocessing_pipeline.fit_transform(\n        train_features_df\n    )\n)\n\ntrain_target_tensor = (\n    target_preprocessing_pipeline.fit_transform(\n        train_labels\n    )\n)\n\nval_features_tensor = (\n    features_preprocessing_pipeline.transform(\n        val_features_df\n    )\n)\n\nval_target_tensor = (\n    target_preprocessing_pipeline.transform(\n        val_labels\n    )\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:17:43.323672Z","iopub.execute_input":"2025-12-12T11:17:43.324291Z","iopub.status.idle":"2025-12-12T11:17:43.483823Z","shell.execute_reply.started":"2025-12-12T11:17:43.324267Z","shell.execute_reply":"2025-12-12T11:17:43.483176Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# should be 7560 by 784!\nprint(train_features_tensor.shape)\n\n# should be float32!\nprint(train_features_tensor.dtype)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:17:46.080932Z","iopub.execute_input":"2025-12-12T11:17:46.081766Z","iopub.status.idle":"2025-12-12T11:17:46.086019Z","shell.execute_reply.started":"2025-12-12T11:17:46.081740Z","shell.execute_reply":"2025-12-12T11:17:46.085247Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# should be 7560!\nprint(train_target_tensor.shape)\n\n# should be int64!\nprint(train_target_tensor.dtype)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:17:51.801603Z","iopub.execute_input":"2025-12-12T11:17:51.802336Z","iopub.status.idle":"2025-12-12T11:17:51.806511Z","shell.execute_reply.started":"2025-12-12T11:17:51.802309Z","shell.execute_reply":"2025-12-12T11:17:51.805692Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# should be 840 by 784!\nprint(val_features_tensor.shape)\n\n# should be float32!\nprint(val_features_tensor.dtype)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:17:54.501534Z","iopub.execute_input":"2025-12-12T11:17:54.501816Z","iopub.status.idle":"2025-12-12T11:17:54.506489Z","shell.execute_reply.started":"2025-12-12T11:17:54.501796Z","shell.execute_reply":"2025-12-12T11:17:54.505355Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# should be 840!\nprint(val_target_tensor.shape)\n\n# should be int64!\nprint(val_target_tensor.dtype)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:17:56.911854Z","iopub.execute_input":"2025-12-12T11:17:56.912153Z","iopub.status.idle":"2025-12-12T11:17:56.916521Z","shell.execute_reply.started":"2025-12-12T11:17:56.912133Z","shell.execute_reply":"2025-12-12T11:17:56.915782Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Create TensorDatasets (5 pts)\n\nComplete the code in the cell below to create `utils.data.TensorDatasets` from the training and validation tensors.","metadata":{}},{"cell_type":"code","source":"train_dataset = utils.data.TensorDataset(\n    train_features_tensor,\n    train_target_tensor\n)\n\nval_dataset = utils.data.TensorDataset(\n    val_features_tensor,\n    val_target_tensor\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:18:46.030872Z","iopub.execute_input":"2025-12-12T11:18:46.031204Z","iopub.status.idle":"2025-12-12T11:18:46.036821Z","shell.execute_reply.started":"2025-12-12T11:18:46.031164Z","shell.execute_reply":"2025-12-12T11:18:46.036058Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Create DataLoaders (5 pts)\n\nComplete the code in the cell below to create `utils.data.DataLoaders` for the training and validation datasets.\n\n","metadata":{}},{"cell_type":"code","source":"train_dataloader = utils.data.DataLoader(\n    train_dataset,\n    batch_size=64,\n    shuffle=True\n)\n\nval_dataloader = utils.data.DataLoader(\n    val_dataset,\n    batch_size=64,\n    shuffle=False\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:18:55.375562Z","iopub.execute_input":"2025-12-12T11:18:55.375833Z","iopub.status.idle":"2025-12-12T11:18:55.380291Z","shell.execute_reply.started":"2025-12-12T11:18:55.375812Z","shell.execute_reply":"2025-12-12T11:18:55.379422Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Define a model, optimizer, and LR scheduler (20 pts)\n\nUsing only the functions provided below define a self-normalizing MLP with three hidden layers. Configure the `optim.AdamW` optimizer, and configure the `optim.lr_scheduler.ReduceLROnPlateau` learning rate scheduler.","metadata":{}},{"cell_type":"code","source":"def initialize_linear_layer(\n    in_features,\n    out_features,\n    init_strategy_=nn.init.kaiming_uniform_,\n    init_strategy_kwargs=None,\n    ):\n    linear_layer = nn.Linear(in_features, out_features)\n\n    if init_strategy_ is not None:\n        if init_strategy_kwargs is None:\n            init_strategy_kwargs = {}\n        init_strategy_(linear_layer.weight, **init_strategy_kwargs)\n        linear_layer.bias.data.fill_(0.0)\n\n    return linear_layer\n\n\ndef make_mlp_classifier(\n    input_size,\n    hidden_sizes=None,\n    output_size=2,\n    activation_fn=None,\n    init_strategy_=nn.init.kaiming_uniform_,\n    init_strategy_kwargs=None,\n    batch_normalization=False\n    ):\n    \n    modules = []\n    hidden_sizes = [] if hidden_sizes is None else hidden_sizes\n    for hidden_size in hidden_sizes:\n        hidden_layer = initialize_linear_layer(\n            input_size,\n            hidden_size,\n            init_strategy_,\n            init_strategy_kwargs,\n        )\n        modules.append(hidden_layer)\n\n        # batch normalization goes after the linear layer...\n        if batch_normalization:\n            modules.append(nn.BatchNorm1d(hidden_size))\n\n        # ...but before the activation_fn!\n        if activation_fn is not None:\n            modules.append(activation_fn)\n        input_size=hidden_size\n\n    output_layer = initialize_linear_layer(\n            input_size,\n            output_size,\n            init_strategy_,\n            init_strategy_kwargs,\n    )\n    modules.append(output_layer)\n    model_fn = nn.Sequential(*modules)\n    return nn.CrossEntropyLoss(), model_fn","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:35:19.850228Z","iopub.execute_input":"2025-12-12T11:35:19.850487Z","iopub.status.idle":"2025-12-12T11:35:19.857568Z","shell.execute_reply.started":"2025-12-12T11:35:19.850470Z","shell.execute_reply":"2025-12-12T11:35:19.856761Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"criterion, model_fn = make_mlp_classifier(\n    input_size=784,\n    hidden_sizes=[256, 128, 64],\n    output_size=10,\n    activation_fn=nn.SELU(),\n    init_strategy_=nn.init.kaiming_normal_,\n    init_strategy_kwargs=None,\n)\nmodel_fn = model_fn.to(DEVICE)\n\noptimizer = torch.optim.AdamW(\n    model_fn.parameters(),\n    lr=1e-3,\n    weight_decay=1e-5\n)\n\nlr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n    optimizer,\n    mode='min',\n    factor=0.5,\n    patience=5,\n    verbose=True\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:35:22.437389Z","iopub.execute_input":"2025-12-12T11:35:22.437946Z","iopub.status.idle":"2025-12-12T11:35:22.448756Z","shell.execute_reply.started":"2025-12-12T11:35:22.437914Z","shell.execute_reply":"2025-12-12T11:35:22.448096Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Train your model (20 pts)\n\nUsing the training loop provided below, train your MLP for 100 epochs and plot the training and validation curves as well as the learning rate.","metadata":{}},{"cell_type":"code","source":"def compute_average_loss_per_batch(dataloader, criterion, model_fn, device=\"cpu\"):\n    total_loss = torch.zeros(1, 1, device=device)\n    num_batches = len(dataloader)\n    for features, targets in dataloader:\n        features, targets = features.to(device), targets.to(device)\n        predictions = model_fn(features)\n        batch_loss = criterion(predictions, targets)\n        total_loss += batch_loss\n    average_loss_per_batch = total_loss / num_batches\n    return average_loss_per_batch\n\n\ndef clip_gradients_(\n    clip_grad_strategy,\n    model_fn,\n    clip_value=None,\n    error_if_nonfinite=False,\n    max_norm=None,\n    norm_type=2.0):\n    if clip_grad_strategy == \"value\" and clip_value is not None:\n        nn.utils.clip_grad_value_(\n            model_fn.parameters(),\n            clip_value\n        )\n    elif clip_grad_strategy == \"norm\" and max_norm is not None:\n        nn.utils.clip_grad_norm_(\n            model_fn.parameters(),\n            max_norm,\n            norm_type,\n            error_if_nonfinite\n        )\n    else:\n        raise NotImplementedError()\n\n\ndef evaluate(model_fn, dataloader, metric, device=\"cpu\"):\n    model_fn.eval()\n    metric.reset()\n    with torch.inference_mode():\n        for X_batch, y_batch in dataloader:\n            X_batch, y_batch = X_batch.to(device), y_batch.to(device)\n            y_pred = model_fn(X_batch)\n            metric.update(y_pred, y_batch)\n    return metric.compute()\n\n\ndef fit(\n    criterion,\n    model_fn,\n    optimizer,\n    train_dataloader,\n    val_dataloader,\n    device=\"cpu\",\n    clip_grad_strategy=None,\n    clip_value=None,\n    error_if_nonfinite=False,\n    lr_scheduler=None,\n    log_epochs=1,\n    max_epochs=1,\n    max_norm=None,\n    norm_type=2.0\n    ):\n\n    history = {\n        \"epoch\": [],\n        \"average_train_loss\": [],\n        \"average_val_loss\": [],\n        \"lr\": [],\n    }\n    num_train_batches = len(train_dataloader)\n    for epoch in range(max_epochs):\n        total_train_loss = torch.zeros(1, 1, device=device)\n        model_fn = model_fn.train()\n        for features, targets in train_dataloader:\n\n            # forward pass\n            features, targets = features.to(device), targets.to(device)\n            predictions = model_fn(features)\n            batch_loss = criterion(predictions, targets)\n            total_train_loss += batch_loss\n\n            # backward pass\n            optimizer.zero_grad()\n            batch_loss.backward()\n            if clip_grad_strategy is not None:\n                clip_gradients_(\n                    clip_grad_strategy,\n                    model_fn,\n                    clip_value,\n                    error_if_nonfinite,\n                    max_norm,\n                    norm_type\n                )\n            optimizer.step()\n\n        history[\"epoch\"].append(epoch)\n\n        average_train_loss_per_batch = total_train_loss / num_train_batches\n        history[\"average_train_loss\"].append(average_train_loss_per_batch.item())\n\n        model_fn = model_fn.eval()\n        with torch.inference_mode():\n            average_val_loss_per_batch = compute_average_loss_per_batch(\n                val_dataloader,\n                criterion,\n                model_fn,\n                device\n            )\n        history[\"average_val_loss\"].append(average_val_loss_per_batch.item())\n\n        # update the learning rate after every training epoch\n        if lr_scheduler is not None:\n            lr_scheduler.step(average_val_loss_per_batch.item())\n            history[\"lr\"].append(lr_scheduler._last_lr[-1])\n    \n        if (epoch + 1) % log_epochs == 0:\n            print(\n                f\"Epoch {epoch},\",\n                f\"Average train Loss {average_train_loss_per_batch.item():.4f},\",\n                f\"Average val Loss {average_val_loss_per_batch.item():.4f}\"\n            )\n\n    history_df = (\n        pd.DataFrame.from_dict(history)\n                    .set_index(\"epoch\")\n    )\n\n    return history_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:35:25.413414Z","iopub.execute_input":"2025-12-12T11:35:25.414144Z","iopub.status.idle":"2025-12-12T11:35:25.425862Z","shell.execute_reply.started":"2025-12-12T11:35:25.414121Z","shell.execute_reply":"2025-12-12T11:35:25.425011Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"history_df = fit(\n    criterion=criterion,\n    model_fn=model_fn,\n    optimizer=optimizer,\n    train_dataloader=train_dataloader,\n    val_dataloader=val_dataloader,\n    device=DEVICE,\n    clip_grad_strategy=None,\n    clip_value=None,\n    error_if_nonfinite=False,\n    lr_scheduler=lr_scheduler,\n    log_epochs=1,  # Print every 10 epochs for less clutter\n    max_epochs=100,\n    max_norm=None,\n    norm_type=2.0\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:35:29.265814Z","iopub.execute_input":"2025-12-12T11:35:29.266561Z","iopub.status.idle":"2025-12-12T11:36:00.022842Z","shell.execute_reply.started":"2025-12-12T11:35:29.266533Z","shell.execute_reply":"2025-12-12T11:36:00.022203Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(10, 6))\nplt.plot(history_df.index, history_df['average_train_loss'], label='Training Loss', marker='o', markersize=3)\nplt.plot(history_df.index, history_df['average_val_loss'], label='Validation Loss', marker='s', markersize=3)\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.title('Training and Validation Loss Curves')\nplt.legend()\nplt.grid(True, alpha=0.3)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:36:00.024391Z","iopub.execute_input":"2025-12-12T11:36:00.024901Z","iopub.status.idle":"2025-12-12T11:36:00.229718Z","shell.execute_reply.started":"2025-12-12T11:36:00.024881Z","shell.execute_reply":"2025-12-12T11:36:00.228997Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6. Prepare the test dataset (10 pts)\n\nComplete the code in the cell below to prepare the test features and create the test dataloader.","metadata":{}},{"cell_type":"code","source":"test_dataset = features_preprocessing_pipeline.transform(\n    test_features_df\n)\n\ntest_dataloader = utils.data.DataLoader(\n    test_dataset,\n    batch_size=128,\n    shuffle=False\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:40:16.981921Z","iopub.execute_input":"2025-12-12T11:40:16.982726Z","iopub.status.idle":"2025-12-12T11:40:17.151779Z","shell.execute_reply.started":"2025-12-12T11:40:16.982700Z","shell.execute_reply":"2025-12-12T11:40:17.151131Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 7. Generate predictions (10 pts)\n\nUsing the function defined below, complete the code to generate predictions for the test dataset using your trained model.","metadata":{}},{"cell_type":"code","source":"def predict(model_fn, dataloader, device=\"cpu\"):\n    model_fn = model_fn.eval()\n    \n    batch_predictions = []\n    with torch.inference_mode():\n        for X in dataloader:\n            X = X.to(device)\n            logits = model_fn(X)\n\n            # convert logits to probabilities\n            probs = logits.softmax(dim=1)\n            preds = torch.argmax(probs, dim=1)\n            batch_predictions.append(preds.cpu())\n\n    predictions = torch.cat(batch_predictions, dim=0)\n    \n    return predictions.numpy()\n        ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:40:47.162636Z","iopub.execute_input":"2025-12-12T11:40:47.163287Z","iopub.status.idle":"2025-12-12T11:40:47.168530Z","shell.execute_reply.started":"2025-12-12T11:40:47.163263Z","shell.execute_reply":"2025-12-12T11:40:47.167704Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_predictions = predict(\n    model_fn=model_fn,\n    dataloader=test_dataloader,\n    device=DEVICE\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:40:54.303472Z","iopub.execute_input":"2025-12-12T11:40:54.304031Z","iopub.status.idle":"2025-12-12T11:40:54.559755Z","shell.execute_reply.started":"2025-12-12T11:40:54.304005Z","shell.execute_reply":"2025-12-12T11:40:54.559168Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 9. Submit predictions to the competition (5 pts)","metadata":{}},{"cell_type":"code","source":"_ = (\n    pd.read_csv(\n        DATA_DIR / \"sample_submission.csv\",\n        index_col=\"ImageId\",\n    ).assign(\n        Label=test_predictions\n    ).to_csv(\n        OUTPUT_DIR / \"submission.csv\",\n        index=True,\n    )\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:41:00.325288Z","iopub.execute_input":"2025-12-12T11:41:00.326140Z","iopub.status.idle":"2025-12-12T11:41:00.370826Z","shell.execute_reply.started":"2025-12-12T11:41:00.326112Z","shell.execute_reply":"2025-12-12T11:41:00.370103Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%bash\n\ncat submission.csv | head -n 5","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:41:02.259292Z","iopub.execute_input":"2025-12-12T11:41:02.259881Z","iopub.status.idle":"2025-12-12T11:41:02.272787Z","shell.execute_reply.started":"2025-12-12T11:41:02.259843Z","shell.execute_reply":"2025-12-12T11:41:02.271874Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 8. Unsupervised pretraining (25 pts, Optional)\n\nLeverage the unlabeled dataset created above to perform unsupervised pretraining and assess whether using unsupervised pretraining improves the performance of your model.\n\nPlot the training and validation loss curves for your original model, and plot the training and validation loss curves for your new model that uses unsupervised pretraining. Which model do you prefer and why?\n","metadata":{}},{"cell_type":"code","source":"# Use PCA for unsupervised feature learning\nencoder = decomposition.PCA(\n    n_components=100,  # Reduce 784 features to 100\n    random_state=RANDOM_STATE\n)\n\n# Fit on the unlabeled features (33,600 samples)\n_ = encoder.fit(unlabeled_features_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:44:40.977946Z","iopub.execute_input":"2025-12-12T11:44:40.978400Z","iopub.status.idle":"2025-12-12T11:44:43.431629Z","shell.execute_reply.started":"2025-12-12T11:44:40.978376Z","shell.execute_reply":"2025-12-12T11:44:43.431070Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Transform the labeled training and validation data\nencoded_train_features_df = pd.DataFrame(\n    encoder.transform(train_features_df)\n)\n\nencoded_val_features_df = pd.DataFrame(\n    encoder.transform(val_features_df)\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:44:52.939874Z","iopub.execute_input":"2025-12-12T11:44:52.940369Z","iopub.status.idle":"2025-12-12T11:44:53.058960Z","shell.execute_reply.started":"2025-12-12T11:44:52.940345Z","shell.execute_reply":"2025-12-12T11:44:53.057966Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create a simpler preprocessing pipeline (no StandardScaler needed after PCA)\nencoded_features_preprocessing_pipeline = pipeline.make_pipeline(\n    preprocessing.StandardScaler(),\n    preprocessing.FunctionTransformer(\n        func=lambda x: torch.tensor(x, dtype=torch.float32)\n    )\n)\n\nencoded_train_features_tensor = (\n    encoded_features_preprocessing_pipeline.fit_transform(\n        encoded_train_features_df\n    )\n)\n\nencoded_val_features_tensor = (\n    encoded_features_preprocessing_pipeline.transform(\n        encoded_val_features_df\n    )\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:45:13.292385Z","iopub.execute_input":"2025-12-12T11:45:13.292679Z","iopub.status.idle":"2025-12-12T11:45:13.318153Z","shell.execute_reply.started":"2025-12-12T11:45:13.292660Z","shell.execute_reply":"2025-12-12T11:45:13.317491Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"_encoded_train_dataset = utils.data.TensorDataset(\n    encoded_train_features_tensor,\n    train_target_tensor\n)\n\n_encoded_val_dataset = utils.data.TensorDataset(\n    encoded_val_features_tensor,\n    val_target_tensor\n)\n\nencoded_train_dataloader = utils.data.DataLoader(\n    _encoded_train_dataset,\n    batch_size=64,\n    shuffle=True\n)\n\nencoded_val_dataloader = utils.data.DataLoader(\n    _encoded_val_dataset,\n    batch_size=64,\n    shuffle=False\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:45:24.302516Z","iopub.execute_input":"2025-12-12T11:45:24.302797Z","iopub.status.idle":"2025-12-12T11:45:24.308266Z","shell.execute_reply.started":"2025-12-12T11:45:24.302777Z","shell.execute_reply":"2025-12-12T11:45:24.307483Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"criterion_encoded, model_fn_encoded = make_mlp_classifier(\n    input_size=100,  # Now 100 instead of 784!\n    hidden_sizes=[64, 32],  # Smaller architecture\n    output_size=10,\n    activation_fn=nn.SELU(),\n    init_strategy_=nn.init.kaiming_normal_,\n    init_strategy_kwargs=None,\n)\nmodel_fn_encoded = model_fn_encoded.to(DEVICE)\n\noptimizer_encoded = torch.optim.AdamW(\n    model_fn_encoded.parameters(),\n    lr=1e-3,\n    weight_decay=1e-5\n)\n\nlr_scheduler_encoded = torch.optim.lr_scheduler.ReduceLROnPlateau(\n    optimizer_encoded,\n    mode='min',\n    factor=0.5,\n    patience=5,\n    verbose=True\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:45:34.412775Z","iopub.execute_input":"2025-12-12T11:45:34.413496Z","iopub.status.idle":"2025-12-12T11:45:34.420328Z","shell.execute_reply.started":"2025-12-12T11:45:34.413471Z","shell.execute_reply":"2025-12-12T11:45:34.419705Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"unsupervised_pretraining_history_df = fit(\n    criterion=criterion_encoded,\n    model_fn=model_fn_encoded,\n    optimizer=optimizer_encoded,\n    train_dataloader=encoded_train_dataloader,\n    val_dataloader=encoded_val_dataloader,\n    device=DEVICE,\n    clip_grad_strategy=None,\n    clip_value=None,\n    error_if_nonfinite=False,\n    lr_scheduler=lr_scheduler_encoded,\n    log_epochs=1,\n    max_epochs=100,\n    max_norm=None,\n    norm_type=2.0\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:45:51.099752Z","iopub.execute_input":"2025-12-12T11:45:51.100432Z","iopub.status.idle":"2025-12-12T11:46:14.965647Z","shell.execute_reply.started":"2025-12-12T11:45:51.100407Z","shell.execute_reply":"2025-12-12T11:46:14.964943Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))\n\n# Original model\nax1.plot(history_df.index, history_df['average_train_loss'], label='Training Loss', marker='o', markersize=3)\nax1.plot(history_df.index, history_df['average_val_loss'], label='Validation Loss', marker='s', markersize=3)\nax1.set_xlabel('Epoch')\nax1.set_ylabel('Loss')\nax1.set_title('Original Model (784 features)')\nax1.legend()\nax1.grid(True, alpha=0.3)\n\n# Encoded model with unsupervised pretraining\nax2.plot(unsupervised_pretraining_history_df.index, unsupervised_pretraining_history_df['average_train_loss'], label='Training Loss', marker='o', markersize=3)\nax2.plot(unsupervised_pretraining_history_df.index, unsupervised_pretraining_history_df['average_val_loss'], label='Validation Loss', marker='s', markersize=3)\nax2.set_xlabel('Epoch')\nax2.set_ylabel('Loss')\nax2.set_title('Model with Unsupervised Pretraining (100 PCA features)')\nax2.legend()\nax2.grid(True, alpha=0.3)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:46:21.613120Z","iopub.execute_input":"2025-12-12T11:46:21.613821Z","iopub.status.idle":"2025-12-12T11:46:22.036327Z","shell.execute_reply.started":"2025-12-12T11:46:21.613798Z","shell.execute_reply":"2025-12-12T11:46:22.035699Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Evaluate original model\noriginal_accuracy = evaluate(\n    model_fn=model_fn,\n    dataloader=val_dataloader,\n    metric=torchmetrics.Accuracy(task='multiclass', num_classes=10).to(DEVICE),\n    device=DEVICE\n)\n\n# Evaluate encoded model\nencoded_accuracy = evaluate(\n    model_fn=model_fn_encoded,\n    dataloader=encoded_val_dataloader,\n    metric=torchmetrics.Accuracy(task='multiclass', num_classes=10).to(DEVICE),\n    device=DEVICE\n)\n\nprint(f\"Original Model Validation Accuracy: {original_accuracy:.4f}\")\nprint(f\"Encoded Model Validation Accuracy: {encoded_accuracy:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-12T11:47:49.770660Z","iopub.execute_input":"2025-12-12T11:47:49.771361Z","iopub.status.idle":"2025-12-12T11:47:49.890692Z","shell.execute_reply.started":"2025-12-12T11:47:49.771336Z","shell.execute_reply":"2025-12-12T11:47:49.889870Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Analysis: Comparison of Original vs. Unsupervised Pretraining\n\n### Key Findings:\n\n**Original Model (784 features):**\n- Validation Accuracy: **94.17%**\n- Training loss drops to ~0.0004, validation loss plateaus at ~0.31\n- **Severe overfitting**: Massive gap between training and validation loss\n- Model has memorized the training data\n\n**Encoded Model with PCA (100 features):**\n- Validation Accuracy: **90.60%**\n- Training loss drops to ~0.012, validation loss plateaus at ~0.36\n- **Less severe overfitting**: Smaller gap between training and validation curves\n- Better generalization behavior but slightly lower accuracy\n\n### Trade-offs:\n\n**Advantages of Unsupervised Pretraining:**\n-  Reduced dimensionality (784 → 100 features)\n-  Faster training and inference\n-  Smaller model with fewer parameters\n-  Better regularization (less overfitting)\n\n**Disadvantages:**\n-  Lower validation accuracy (3.57% drop)\n-  Information loss from dimensionality reduction\n\n### Conclusion:\nThe **original model performs better** with 94.17% accuracy, but suffers from severe overfitting. The **encoded model offers a good balance** between efficiency and performance at 90.60% accuracy with significantly fewer features. For production deployment where model size and speed matter, the encoded model is preferable. For maximum accuracy, the original model should be used with early stopping (around epoch 10-20).","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}