{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":94834,"databundleVersionId":11269999,"sourceType":"competition"}],"dockerImageVersionId":30918,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This Python script provides a basic PyTorch regression solution for the Kaggle assignment. It begins by loading the training and test datasets, then processes categorical features using label encoding to convert them into numerical form. The script selects relevant numeric features while excluding the 'id' column and designating 'performance_score' as the target variable. It splits the data into training and validation sets and constructs a fully connected neural network using PyTorch. The model is trained with a small number of epochs and a moderate architecture to balance performance and efficiency. Finally, it evaluates the model using the R-squared (R²) metric, printing the result to indicate how well the model explains the variance in the target variable.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder, StandardScaler\nfrom sklearn.metrics import r2_score\n\n# Load the dataset\ntrain_file_path = \"/kaggle/input/app-of-gen-ai-deep-learning-wustl-spring-2025/train.csv\"\ntest_file_path = \"/kaggle/input/app-of-gen-ai-deep-learning-wustl-spring-2025/test.csv\"\n\ndf = pd.read_csv(train_file_path)\ndf_test = pd.read_csv(test_file_path)\n\n# Handle categorical columns by encoding them\ncategorical_cols = df.select_dtypes(include=['object']).columns\nlabel_encoders = {}\nfor col in categorical_cols:\n    label_encoders[col] = LabelEncoder()\n    df[col] = label_encoders[col].fit_transform(df[col].astype(str))\n    \n    if col in df_test.columns:\n        df_test[col] = df_test[col].astype(str)\n        df_test[col] = df_test[col].apply(lambda x: x if x in label_encoders[col].classes_ else 'unknown')\n        label_encoders[col].classes_ = np.append(label_encoders[col].classes_, 'unknown')\n        df_test[col] = label_encoders[col].transform(df_test[col])\n\n# Define features and target\nX = df.drop(columns=['id', 'performance_score'])  # Remove ID and target column\ny = df['performance_score']\n\n# Standardize numerical features\nscaler = StandardScaler()\nX = scaler.fit_transform(X)\n\n# Convert to PyTorch tensors\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\nX_train_tensor = torch.tensor(X_train, dtype=torch.float32)\nX_test_tensor = torch.tensor(X_test, dtype=torch.float32)\ny_train_tensor = torch.tensor(y_train.values, dtype=torch.float32).view(-1, 1)\ny_test_tensor = torch.tensor(y_test.values, dtype=torch.float32).view(-1, 1)\n\n# Define a simple PyTorch regression model\nclass RegressionNN(nn.Module):\n    def __init__(self, input_dim):\n        super(RegressionNN, self).__init__()\n        self.fc = nn.Sequential(\n            nn.Linear(input_dim, 64),\n            nn.ReLU(),\n            nn.Linear(64, 32),\n            nn.ReLU(),\n            nn.Linear(32, 1)\n        )\n\n    def forward(self, x):\n        return self.fc(x)\n\n# Initialize model, loss function, and optimizer\ninput_dim = X_train.shape[1]\nmodel = RegressionNN(input_dim)\ncriterion = nn.MSELoss()\noptimizer = optim.Adam(model.parameters(), lr=0.01)\n\n# Training loop\nepochs = 50\nfor epoch in range(epochs):\n    model.train()\n    optimizer.zero_grad()\n    y_pred = model(X_train_tensor)\n    loss = criterion(y_pred, y_train_tensor)\n    loss.backward()\n    optimizer.step()\n    \n    if (epoch + 1) % 10 == 0:\n        print(f\"Epoch [{epoch+1}/{epochs}], Loss: {loss.item():.4f}\")\n\n# Evaluation\nmodel.eval()\nwith torch.no_grad():\n    y_pred_test = model(X_test_tensor).numpy().flatten()\n\n# Compute R-squared\nr2 = r2_score(y_test, y_pred_test)\nprint(f\"R-squared: {r2:.4f}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-05T11:55:14.388143Z","iopub.execute_input":"2025-03-05T11:55:14.388458Z","iopub.status.idle":"2025-03-05T11:55:24.002045Z","shell.execute_reply.started":"2025-03-05T11:55:14.388421Z","shell.execute_reply":"2025-03-05T11:55:24.000997Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This code analyzes and visualizes the impact of each feature on the predictions made by the trained PyTorch neural network in the Kaggle assignment. Since neural networks do not provide built-in feature importance scores like XGBoost, the script estimates feature importance using a permutation-based method. It systematically shuffles each feature and measures the resulting change in the model’s performance, storing the importance scores in a DataFrame. The features are then sorted in descending order to highlight the most influential predictors. Finally, the script generates a horizontal bar chart using Matplotlib, making it easier to interpret which features contribute most to the model’s predictions. The y-axis is inverted to ensure the most important features appear at the top of the chart.","metadata":{}},{"cell_type":"code","source":"from sklearn.inspection import permutation_importance\nimport matplotlib.pyplot as plt\n\n# Ensure the model is in evaluation mode\nmodel.eval()\n\n# Compute feature importance using permutation importance\ndef compute_permutation_importance(model, X_test, y_test, metric=r2_score):\n    baseline_score = metric(y_test, model(X_test).detach().numpy().flatten())\n    importance_scores = []\n    \n    for i in range(X_test.shape[1]):\n        X_test_permuted = X_test.clone()\n        X_test_permuted[:, i] = X_test_permuted[:, i][torch.randperm(X_test.shape[0])]  # Shuffle one feature\n        permuted_score = metric(y_test, model(X_test_permuted).detach().numpy().flatten())\n        importance_scores.append(baseline_score - permuted_score)\n\n    return np.array(importance_scores)\n\nimportances = compute_permutation_importance(model, X_test_tensor, y_test)\n\n# Retrieve original feature names from the DataFrame before conversion to NumPy\nfeature_names = df.drop(columns=['id', 'performance_score']).columns\n\n# Create DataFrame for feature importance\nfeature_importance_df = pd.DataFrame({'Feature': feature_names, 'Importance': importances})\nfeature_importance_df = feature_importance_df.sort_values(by='Importance', ascending=False)\n\n# Display feature importance\nplt.figure(figsize=(10, 6))\nplt.barh(feature_importance_df['Feature'], feature_importance_df['Importance'])\nplt.xlabel(\"Importance\")\nplt.ylabel(\"Feature\")\nplt.title(\"Feature Importance in PyTorch Model (Permutation Importance)\")\nplt.gca().invert_yaxis()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-05T12:07:16.775527Z","iopub.execute_input":"2025-03-05T12:07:16.775869Z","iopub.status.idle":"2025-03-05T12:07:17.333431Z","shell.execute_reply.started":"2025-03-05T12:07:16.775841Z","shell.execute_reply":"2025-03-05T12:07:17.332231Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This code prepares the Kaggle submission file for the competition by generating predictions on the test dataset using the trained PyTorch model. It first extracts the 'id' column from the test set to maintain proper identification in the submission. After dropping 'id' from the test features, the script standardizes the test data using the same scaler applied to the training set. The trained model then predicts the performance_score for each test instance. The predictions are stored in a new DataFrame, ensuring they are paired with the corresponding 'id'. Finally, the script saves the results as a CSV file (submission.csv) in the format required by Kaggle.\n\n# How to Make a Kaggle Submission in the Notebook:\n1. Run the notebook to generate submission.csv.\n2. Locate the file in the Kaggle notebook environment under the \"Output\" section.\n3. Go to the Kaggle competition page and navigate to the \"Submit Predictions\" tab.\n4. Upload the submission.csv file and confirm the submission.\n5. Check the leaderboard to see your model's performance ranking.","metadata":{}},{"cell_type":"code","source":"# Prepare Kaggle submission\ntest_ids = df_test['id']\ndf_test = df_test.drop(columns=['id'])\n\n# Standardize test data using the same scaler\nX_test_final = scaler.transform(df_test)\n\n# Convert to PyTorch tensor\nX_test_final_tensor = torch.tensor(X_test_final, dtype=torch.float32)\n\n# Generate predictions\nmodel.eval()\nwith torch.no_grad():\n    submission_preds = model(X_test_final_tensor).numpy().flatten()\n\n# Create submission DataFrame\nsubmission = pd.DataFrame({'id': test_ids, 'performance_score': submission_preds})\nsubmission.to_csv(\"submission.csv\", index=False)\n\nprint(\"Kaggle submission file 'submission.csv' created successfully.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-05T12:09:14.916836Z","iopub.execute_input":"2025-03-05T12:09:14.917255Z","iopub.status.idle":"2025-03-05T12:09:14.946775Z","shell.execute_reply.started":"2025-03-05T12:09:14.917222Z","shell.execute_reply":"2025-03-05T12:09:14.945391Z"}},"outputs":[],"execution_count":null}]}