{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceType":"competition","sourceId":131037,"databundleVersionId":15734623}],"dockerImageVersionId":31286,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This Python script provides a baseline XGBoost regression solution for the “Applications of Deep Learning/GenAI (WUSTL, Spring 2026)” Kaggle competition. It begins by loading the competition’s train.csv and test.csv files from the directory. Categorical features are encoded as numeric via LabelEncoder, and the remaining numeric columns are selected, with the id column dropped and preservation_score set as the target variable. The data is then split into training and validation sets. An XGBRegressor is trained with n_estimators=25 and max_depth=5 to balance model complexity and efficiency. Finally, the model is evaluated on the validation set using Root Mean Squared Error (RMSE), with the resulting score printed to indicate the average prediction error for artifact preservation scores","metadata":{}},{"cell_type":"code","source":"import os\nfrom pathlib import Path\n\ndef find_dataset_dir(required=(\"train.csv\", \"test.csv\")) -> Path:\n    \"\"\"Locate the dataset directory in Kaggle or locally, with helpful diagnostics.\"\"\"\n    # 0) Env var overrides\n    env_dir = os.getenv(\"DATASET_DIR\")\n    if env_dir:\n        p = Path(env_dir)\n        if p.is_dir() and all((p/f).exists() for f in required):\n            print(f\"🔧 Using DATASET_DIR override: {p}\")\n            return p\n        else:\n            print(f\"⚠️ DATASET_DIR set but invalid: {p}\")\n\n    env_name = os.getenv(\"DATASET_NAME\")\n    kaggle_input = Path(\"/kaggle/input\")\n    if env_name and kaggle_input.exists():\n        cand = kaggle_input / env_name\n        if cand.is_dir() and all((cand/f).exists() for f in required):\n            print(f\"🔧 Using DATASET_NAME under /kaggle/input: {cand}\")\n            return cand\n        else:\n            print(f\"⚠️ DATASET_NAME set but not found or missing files: {cand}\")\n\n    candidates = []\n\n    # 1) Exact files at /kaggle/input (rare, but check)\n    if kaggle_input.exists():\n        if all((kaggle_input/f).exists() for f in required):\n            candidates.append(kaggle_input)\n\n        # 2) Any subdir of /kaggle/input containing required files (non-recursive, then recursive)\n        for d in kaggle_input.iterdir():\n            if d.is_dir() and all((d/f).exists() for f in required):\n                candidates.append(d)\n\n        # 3) Recursive search as a last resort (handles nested or renamed datasets)\n        if not candidates:\n            trains = list(kaggle_input.rglob(required[0]))\n            tests  = set(p.parent for p in kaggle_input.rglob(required[1]))\n            for t in trains:\n                if t.parent in tests:\n                    candidates.append(t.parent)\n\n    # 4) Local working directory fallback\n    cwd = Path(\".\").resolve()\n    if all((cwd/f).exists() for f in required):\n        candidates.append(cwd)\n\n    # Deduplicate while preserving order\n    seen = set()\n    unique = []\n    for c in candidates:\n        if str(c) not in seen:\n            seen.add(str(c))\n            unique.append(c)\n\n    if not unique:\n        # Helpful diagnostics\n        print(\"❌ Could not find a folder containing both train.csv and test.csv.\")\n        if kaggle_input.exists():\n            print(\"📁 Top-level entries in /kaggle/input:\")\n            for d in kaggle_input.iterdir():\n                print(\"  -\", d.name)\n        print(\"🔎 Searched:\", kaggle_input if kaggle_input.exists() else \"(no /kaggle/input)\")\n        print(\"🔎 Also checked working dir:\", cwd)\n        raise FileNotFoundError(\"Could not find dataset directory containing train.csv and test.csv\")\n\n    # Pick the most recently modified candidate\n    chosen = max(unique, key=lambda p: p.stat().st_mtime if p.exists() else 0)\n    # Print a short diagnostic list\n    print(\"✅ Candidates that contain the required files:\")\n    for c in unique:\n        marker = \" (chosen)\" if c == chosen else \"\"\n        print(f\"  - {c}{marker}\")\n    return chosen\n\n# Usage:\ndataset_dir = find_dataset_dir()\nprint(f\"📂 Using dataset directory: {dataset_dir}\")\n\ntrain_path = dataset_dir / \"train.csv\"\ntest_path  = dataset_dir / \"test.csv\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-02T03:55:41.258568Z","iopub.execute_input":"2026-03-02T03:55:41.258892Z","iopub.status.idle":"2026-03-02T03:55:41.284774Z","shell.execute_reply.started":"2026-03-02T03:55:41.258865Z","shell.execute_reply":"2026-03-02T03:55:41.283522Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Train XGBoost\n\nThis script builds and evaluates a regression model using **XGBoost** within a **scikit-learn pipeline**. It starts by loading the training and test datasets from CSV files and automatically infers the **ID column** (first column) and the **target column** (last column). The features (`X`) consist of all columns between them, while the target (`y`) is the final column. Data preprocessing is handled through a `ColumnTransformer` that imputes missing numeric values with the median and categorical values with the most frequent category. Categorical columns are then ordinally encoded, allowing the model to handle non-numeric data. An `XGBRegressor` is trained with 300 estimators, moderate depth (`max_depth=6`), and a learning rate of 0.05, using 90% subsampling for both rows and columns to reduce overfitting. Finally, the dataset is split into training and validation sets (80/20), the model is fitted, predictions are made, and **Root Mean Squared Error (RMSE)** is printed as the evaluation metric.\n\n---\n\n### 🚀 How to Get Better Scores\n\n- **Use cross-validation** instead of a single 80/20 split to reduce randomness and get a more stable estimate of performance.  \n- **Add early stopping** with more estimators (e.g., 2000+) and a smaller learning rate (e.g., 0.02) to prevent overfitting.  \n- **Tune hyperparameters** such as `max_depth`, `min_child_weight`, `subsample`, `colsample_bytree`, and regularization terms (`reg_alpha`, `reg_lambda`) using `GridSearchCV`, `RandomizedSearchCV`, or **Optuna**.  \n- **Replace `OrdinalEncoder` with `OneHotEncoder`** for categorical variables, as ordinal encoding imposes an artificial order that can hurt tree-based models.  \n- **Add feature engineering** — log-transform skewed numeric features, create interaction features, or encode domain relationships.  \n- **Handle outliers** and normalize or scale numeric data where appropriate.  \n- **Match the competition’s validation strategy**, such as grouped or time-based splits, to avoid overestimating performance.  \n- **Ensemble multiple models** (e.g., XGBoost, LightGBM, CatBoost) for improved robustness.  \n- **Train on full data after tuning** with the best-found parameters and early stopping round count, then generate test predictions for submission.  \n\nThese refinements help improve generalization and typically yield lower RMSE or higher leaderboard placement in real-world or Kaggle-style competitions.\n","metadata":{}},{"cell_type":"code","source":"# --- Model train + submit (drop in after dataset_dir/train_path/test_path) ---\nimport pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import OrdinalEncoder\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.metrics import mean_squared_error\nfrom xgboost import XGBRegressor\n\n# 1) Load\ndf      = pd.read_csv(train_path)\ndf_test = pd.read_csv(test_path)\n\n# 2) Infer columns by position\nid_col     = df.columns[0]     # leftmost\ntarget_col = df.columns[-1]    # rightmost\nprint(f\"🪪 ID column: {id_col}\")\nprint(f\"🎯 Target column: {target_col}\")\n\n# 3) Split features/target\nX = df.iloc[:, 1:-1]   # everything between ID and target\ny = df.iloc[:, -1]\n\n# 4) Preprocess (numeric + categorical)\ncat_cols = X.select_dtypes(include=[\"object\"]).columns.tolist()\nnum_cols = X.select_dtypes(include=[np.number, \"bool\"]).columns.tolist()\n\npreprocess = ColumnTransformer(\n    transformers=[\n        (\"num\", SimpleImputer(strategy=\"median\"), num_cols),\n        (\"cat\", Pipeline(steps=[\n            (\"impute\", SimpleImputer(strategy=\"most_frequent\")),\n            (\"encode\", OrdinalEncoder(handle_unknown=\"use_encoded_value\", unknown_value=-1)),\n        ]), cat_cols),\n    ],\n    remainder=\"drop\"\n)\n\n# 5) Model\nmodel = XGBRegressor(\n    objective=\"reg:squarederror\",\n    n_estimators=300,\n    max_depth=6,\n    learning_rate=0.05,\n    subsample=0.9,\n    colsample_bytree=0.9,\n    random_state=42,\n    n_jobs=-1,\n)\n\npipe = Pipeline(steps=[(\"prep\", preprocess), (\"model\", model)])\n\n# 6) Train/validate\nX_train, X_val, y_train, y_val = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)\npipe.fit(X_train, y_train)\ny_pred = pipe.predict(X_val)\nrmse = np.sqrt(mean_squared_error(y_val, y_pred))\nprint(f\"✅ RMSE: {rmse:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-02T03:56:55.523579Z","iopub.execute_input":"2026-03-02T03:56:55.523913Z","iopub.status.idle":"2026-03-02T03:56:59.628155Z","shell.execute_reply.started":"2026-03-02T03:56:55.523889Z","shell.execute_reply":"2026-03-02T03:56:59.627341Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Feature Importance\n\nThis code extracts and visualizes the feature‐importance scores from the XGBoost regressor trained for the Kaggle competition. It begins by calling model.featureimportances on the fitted XGBRegressor to retrieve each input feature’s relative importance in predicting the preservation_score. Those scores are paired with their corresponding feature names in a pandas DataFrame, then sorted in descending order so the most influential predictors rise to the top. Finally, the script uses Matplotlib to plot a horizontal bar chart—annotating each bar with its importance score and inverting the y-axis so the highest‐importance features appear at the top—making it easy to see which variables contribute most to the model’s predictions.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Feature importance\nimportances = model.feature_importances_\nfeature_importance_df = pd.DataFrame({'Feature': X.columns, 'Importance': importances})\nfeature_importance_df = feature_importance_df.sort_values(by='Importance', ascending=False)\n\n# Display feature importance\nplt.figure(figsize=(10, 6))\nplt.barh(feature_importance_df['Feature'], feature_importance_df['Importance'])\nplt.xlabel(\"Importance\")\nplt.ylabel(\"Feature\")\nplt.title(\"Feature Importance in XGBoost Model\")\nplt.gca().invert_yaxis()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-02T03:57:54.874707Z","iopub.execute_input":"2026-03-02T03:57:54.875026Z","iopub.status.idle":"2026-03-02T03:57:55.495822Z","shell.execute_reply.started":"2026-03-02T03:57:54.875001Z","shell.execute_reply":"2026-03-02T03:57:55.494379Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This code prepares the Kaggle submission file for the competition by generating predictions on the test dataset using the trained XGBoost model. It first extracts the 'id' column from the test set to maintain proper identification in the submission. After dropping 'id' from the test features, it applies the trained model to predict performance_score for each test instance. The predictions are then stored in a new DataFrame, ensuring they are paired with the corresponding 'id'. Finally, the script saves the results as a CSV file (\"submission.csv\") in the format required by Kaggle.\n\n# How to Make a Kaggle Submission in the Notebook\n1. Run the notebook to generate submission.csv.\n2. Locate the file in the Kaggle notebook environment under the \"Output\" section.\n3. Go to the Kaggle competition page and navigate to the \"Submit Predictions\" tab.\n4. Upload the submission.csv file and confirm the submission.\n5. Check the leaderboard to see your model's performance ranking.","metadata":{}},{"cell_type":"code","source":"\n# 7) Predict test + save submission\nX_sub = df_test.iloc[:, 1:]  # skip ID\npred = pipe.predict(X_sub)\n\nsubmission = pd.DataFrame({\n    id_col: df_test[id_col],\n    target_col: pred\n})\nprint(\"\\n📄 Submission preview:\")\nprint(submission.head())\n\nsubmission.to_csv(\"submission.csv\", index=False)\nprint(\"\\n💾 Saved to submission.csv\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-02T03:58:39.806751Z","iopub.execute_input":"2026-03-02T03:58:39.807306Z","iopub.status.idle":"2026-03-02T03:58:39.892005Z","shell.execute_reply.started":"2026-03-02T03:58:39.807273Z","shell.execute_reply":"2026-03-02T03:58:39.891051Z"}},"outputs":[],"execution_count":null}]}