{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceType":"competition","sourceId":130062,"databundleVersionId":15606741}],"dockerImageVersionId":31286,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <div align = \"center\">Bidding Prediction for Construction</div>\n\n---\n\n<div align = \"center\" style = \"background-color:gray;color:black;font-weight:bold\">Made by Stable Space</div>\n\n---\n\n## 1. About Competition\n\n### Competition Overview\n- Task: Predict **contractor total bid submissions** for public construction projects.  \n- Data: Historical projects with **pre-letting job characteristics** and corresponding bid totals.  \n- Two datasets:\n  - **train_summary** → contractor’s total bid for a project.  \n  - **train_raw** → individual line-item bids (auxiliary data for feature engineering).  \n- External data allowed: inflation metrics, cost indices, etc., if public and time-consistent.  \n- Collaboration and ensembling are encouraged.\n\n### Goal\n- Build the **most accurate model** for predicting total bid amounts.  \n- Focus is on **job-level prediction**, not individual line-item prices.\n\n### Timeline\n- Competition started: **16 days ago**.  \n- Closing: **8 days remaining**.\n\n### Evaluation\n- Metric: **Root Mean Squared Logarithmic Error (RMSLE)**.  \n- Properties:\n  - Unitless measure.  \n  - Lower RMSLE = better performance.  \n  - Sensitive to relative differences (helps with skewed distributions).  \n\n---\n\n## 2. About Dataset\n\nThis dataset is about **contractors bidding on jobs**.\n\nEach **job**:\n\n- Has multiple **contractors** bidding.\n- Each contractor submits a **total bid**.\n- That total bid is made up of multiple **pay-items** (line items).\n\nYour task is evaluated at:\n\n>  **Contractor–Job level**\n> Meaning: For each contractor bidding on a job, predict their **total bid**.\n\n\n## 3. What Is the Goal? (Output)\n\n### OUTPUT (Target Variable)\n\n```\ntotal_bid\n```\n\nThis is the **final total amount** a contractor bids for a job.\n\nEvaluation is done at:\n\n```\nrow_id = job_id + contractor_id\n```\n\nSo the model predicts:\n\n> For each contractor bidding on a job → what will be their total bid?\n\n## 4. Two Data Views\n\nThere are two ways data is structured:\n\n### Pay-Item Level (Detailed Level)\n\nEach job is made of many pay-items.\n\nFor example:\n\nJob A:\n\n- Cement work\n- Steel work\n- Labor\n- Excavation\n\nEach has:\n\n- Quantity\n- Unit price\n- Description\n- Unit type\n\nThese combine to form the total bid.\n\nFiles:\n\n- `raw_train.csv`\n- `raw_test.csv`\n\n\n### Contractor–Job Level (Aggregated Level)\n\nThis shows:\n\n- One row per contractor per job\n- Includes total_bid (in train)\n\nFiles:\n\n- `train_summary.csv`\n- `test.csv`\n\nThis is the format used for **evaluation**.\n\n## 5. Important Columns Explained Simply\n\n### Identifiers\n\n| Column        | Meaning                                          |\n| ------------- | ------------------------------------------------ |\n| job_id        | Unique job                                       |\n| contractor_id | Unique contractor                                |\n| row_id        | Unique contractor-job pair (used for submission) |\n\n\n### Pay-Item Level Features\n\nThese exist in `raw_train.csv`:\n\n| Column               | Meaning                        |\n| -------------------- | ------------------------------ |\n| pay_item_id          | Unique pay-item                |\n| pay_item_description | What work is done              |\n| unit_english_id      | Unit (kg, m, ton, etc.)        |\n| quantity             | How much of that item          |\n| amount               | Unit price submitted           |\n| total_bid            | Sum of all (quantity × amount) |\n\n\n### Job-Level Features\n\n| Column                   | Meaning             |\n| ------------------------ | ------------------- |\n| job_category_description | Type of job         |\n| category_id              | Category ID         |\n| primary_location         | Where job is        |\n| bid_date                 | When bid was placed |\n\n\n## 5. What Are the Inputs?\n\nDepending on how you model:\n\n\n### Option A: Use Job-Level Dataset (`train_summary.csv`)\n\n### Input Features:\n\n- contractor_id\n- job_category_description\n- category_id\n- primary_location\n- bid_date\n- other aggregated statistics\n\n### Output:\n\n* total_bid\n\n### Option B: Use Pay-Item Dataset (`raw_train.csv`)\n\nYou can engineer features like:\n\n- Total quantity per job\n- Average unit price\n- Number of pay-items\n- Contractor-specific average pricing\n- Historical bidding patterns\n\nThen aggregate to contractor–job level.\n\n### Input:\n\nEverything except:\n\n```\ntotal_bid\n```\n\n### Output:\n\n```\ntotal_bid\n```\n\n\n## 6. What Happens in Test?\n\nIn test:\n\n- You are given contractor–job combinations.\n- You must predict their total bid.\n- Submit:\n\n```\nrow_id, total_bid\n```\n\n### Simple Real-World Analogy\n\nImagine:\n\nGovernment announces a road construction job.\n\n5 contractors submit bids.\n\nEach contractor:\n\n- Lists materials\n- Quantities\n- Unit prices\n\nThose sum to a final total bid.\n\nYour model must predict:\n\n>  What total bid each contractor will submit for each job.\n\n\n### Summary\n\n####  Input (X)\n\n- Contractor info\n- Job info\n- Location\n- Category\n- Pay-item details (optional if you engineer features)\n- Historical bidding behavior\n\n#### Output (y)\n\n```\ntotal_bid\n```\n\nEvaluation unit:\n\n```\ncontractor + job\n```","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom pathlib import Path\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nimport lightgbm as lgb\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:45:42.286795Z","iopub.execute_input":"2026-02-22T12:45:42.287811Z","iopub.status.idle":"2026-02-22T12:45:45.962391Z","shell.execute_reply.started":"2026-02-22T12:45:42.287774Z","shell.execute_reply":"2026-02-22T12:45:45.961615Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ROOT_DIR = r\"/kaggle/input/bidding-predictions-for-construction\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T11:41:41.428246Z","iopub.execute_input":"2026-02-22T11:41:41.428962Z","iopub.status.idle":"2026-02-22T11:41:41.433510Z","shell.execute_reply.started":"2026-02-22T11:41:41.428924Z","shell.execute_reply":"2026-02-22T11:41:41.432562Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"TRAIN_RAW = Path(ROOT_DIR) / \"raw_train.csv\"\nTRAIN_SUMMARY = Path(ROOT_DIR) / \"train_summary.csv\"\n\nTEST_RAW = Path(ROOT_DIR) / \"raw_test.csv\"\nTEST = Path(ROOT_DIR) / \"test.csv\"\n\nSAMPLE_SUBMISSION = Path(ROOT_DIR) / \"sample_submission.csv\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T11:45:49.593908Z","iopub.execute_input":"2026-02-22T11:45:49.594693Z","iopub.status.idle":"2026-02-22T11:45:49.599142Z","shell.execute_reply.started":"2026-02-22T11:45:49.594659Z","shell.execute_reply":"2026-02-22T11:45:49.598342Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def load_csv(path:Path, n:int = 5):\n    df = pd.read_csv(path)\n    display(df.head(n))\n    print(df.info())\n    print(\"-\" * 50)\n    print(f\"NULL Values: \\n{df.isnull().sum()}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:06:38.565918Z","iopub.execute_input":"2026-02-22T12:06:38.567117Z","iopub.status.idle":"2026-02-22T12:06:38.572081Z","shell.execute_reply.started":"2026-02-22T12:06:38.567027Z","shell.execute_reply":"2026-02-22T12:06:38.571131Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"load_csv(TRAIN_RAW)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:06:41.174105Z","iopub.execute_input":"2026-02-22T12:06:41.175128Z","iopub.status.idle":"2026-02-22T12:06:44.218838Z","shell.execute_reply.started":"2026-02-22T12:06:41.175090Z","shell.execute_reply":"2026-02-22T12:06:44.218120Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"load_csv(TEST_RAW)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:06:55.714190Z","iopub.execute_input":"2026-02-22T12:06:55.715208Z","iopub.status.idle":"2026-02-22T12:06:56.119466Z","shell.execute_reply.started":"2026-02-22T12:06:55.715176Z","shell.execute_reply":"2026-02-22T12:06:56.118468Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"load_csv(TRAIN_SUMMARY)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:07:11.054731Z","iopub.execute_input":"2026-02-22T12:07:11.055338Z","iopub.status.idle":"2026-02-22T12:07:11.094880Z","shell.execute_reply.started":"2026-02-22T12:07:11.055300Z","shell.execute_reply":"2026-02-22T12:07:11.094154Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"load_csv(TEST)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:07:16.347266Z","iopub.execute_input":"2026-02-22T12:07:16.348269Z","iopub.status.idle":"2026-02-22T12:07:16.370858Z","shell.execute_reply.started":"2026-02-22T12:07:16.348234Z","shell.execute_reply":"2026-02-22T12:07:16.369746Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"load_csv(SAMPLE_SUBMISSION)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:07:20.270118Z","iopub.execute_input":"2026-02-22T12:07:20.270753Z","iopub.status.idle":"2026-02-22T12:07:20.291778Z","shell.execute_reply.started":"2026-02-22T12:07:20.270717Z","shell.execute_reply":"2026-02-22T12:07:20.290809Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_raw = pd.read_csv(TRAIN_RAW)\ntest_raw = pd.read_csv(TEST_RAW)\ntrain_summary = pd.read_csv(TRAIN_SUMMARY)\ntest = pd.read_csv(TEST)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:48:30.470787Z","iopub.execute_input":"2026-02-22T12:48:30.471339Z","iopub.status.idle":"2026-02-22T12:48:32.904955Z","shell.execute_reply.started":"2026-02-22T12:48:30.471307Z","shell.execute_reply":"2026-02-22T12:48:32.904184Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Convert date\ntrain_raw[\"bid_date\"] = pd.to_datetime(train_raw[\"bid_date\"])\ntest_raw[\"bid_date\"] = pd.to_datetime(test_raw[\"bid_date\"])\ntrain_summary[\"bid_date\"] = pd.to_datetime(train_summary[\"bid_date\"])\ntest[\"bid_date\"] = pd.to_datetime(test[\"bid_date\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:49:42.017754Z","iopub.execute_input":"2026-02-22T12:49:42.018561Z","iopub.status.idle":"2026-02-22T12:49:42.077109Z","shell.execute_reply.started":"2026-02-22T12:49:42.018531Z","shell.execute_reply":"2026-02-22T12:49:42.076421Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ===============================\n# STEP 1: HISTORICAL FEATURES\n# ===============================\n\n# 1️⃣ Contractor historical avg unit price\ncontractor_avg_price = (\n    train_raw.groupby(\"contractor_id\")[\"amount\"]\n    .mean()\n    .reset_index()\n    .rename(columns={\"amount\": \"contractor_avg_unit_price\"})\n)\n\n# 2️⃣ Contractor-category avg unit price\ncontractor_category_avg = (\n    train_raw.groupby([\"contractor_id\", \"category_id\"])[\"amount\"]\n    .mean()\n    .reset_index()\n    .rename(columns={\"amount\": \"contractor_category_avg_price\"})\n)\n\n# 3️⃣ Contractor-location avg total bid\ncontractor_location_avg = (\n    train_summary.groupby([\"contractor_id\", \"primary_location\"])[\"total_bid\"]\n    .mean()\n    .reset_index()\n    .rename(columns={\"total_bid\": \"contractor_location_avg_bid\"})\n)\n\n# 4️⃣ Category avg total bid\n# First create job-level category mapping from raw data\njob_category_map = (\n    train_raw[[\"job_id\", \"category_id\"]]\n    .drop_duplicates()\n)\n\n# Merge into train_summary\ntrain_summary = train_summary.merge(job_category_map, on=\"job_id\", how=\"left\")\n\n# Now compute category avg bid\ncategory_avg_bid = (\n    train_summary.groupby(\"category_id\")[\"total_bid\"]\n    .mean()\n    .reset_index()\n    .rename(columns={\"total_bid\": \"category_avg_bid\"})\n)\n\n# 5️⃣ Location avg total bid\nlocation_avg_bid = (\n    train_summary.groupby(\"primary_location\")[\"total_bid\"]\n    .mean()\n    .reset_index()\n    .rename(columns={\"total_bid\": \"location_avg_bid\"})\n)\n\n# 6️⃣ Pay-item frequency per contractor-job\npayitem_counts = (\n    train_raw.groupby([\"job_id\", \"contractor_id\"])\n    .size()\n    .reset_index(name=\"num_pay_items_calc\")\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:51:22.223525Z","iopub.execute_input":"2026-02-22T12:51:22.224041Z","iopub.status.idle":"2026-02-22T12:51:22.621275Z","shell.execute_reply.started":"2026-02-22T12:51:22.224010Z","shell.execute_reply":"2026-02-22T12:51:22.620427Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\n# ===============================\n# STEP 2: AGGREGATE TO JOB LEVEL\n# ===============================\n\n# Aggregate pay-item quantities\njob_level_agg = (\n    train_raw.groupby([\"job_id\", \"contractor_id\"])\n    .agg(\n        total_quantity=(\"quantity\", \"sum\"),\n        avg_quantity=(\"quantity\", \"mean\"),\n        unique_pay_items=(\"pay_item_id\", \"nunique\")\n    )\n    .reset_index()\n)\n\n# Merge all features into train_summary\ntrain = train_summary.copy()\n\ntrain = train.merge(job_level_agg, on=[\"job_id\", \"contractor_id\"], how=\"left\")\ntrain = train.merge(payitem_counts, on=[\"job_id\", \"contractor_id\"], how=\"left\")\ntrain = train.merge(contractor_avg_price, on=\"contractor_id\", how=\"left\")\ntrain = train.merge(contractor_category_avg, on=[\"contractor_id\"], how=\"left\")\ntrain = train.merge(contractor_location_avg, on=[\"contractor_id\", \"primary_location\"], how=\"left\")\njob_category_map = (\n    train_raw[[\"job_id\", \"category_id\"]]\n    .drop_duplicates()\n)\n\ntrain = train_summary.copy()\n\ntrain = train.merge(job_category_map, on=\"job_id\", how=\"left\")\ntrain = train.merge(location_avg_bid, on=\"primary_location\", how=\"left\")\n\n# ===============================\n# PREPARE TEST FEATURES\n# ===============================\n\n# Aggregate test_raw same way\ntest_job_level_agg = (\n    test_raw.groupby([\"job_id\", \"contractor_id\"])\n    .agg(\n        total_quantity=(\"quantity\", \"sum\"),\n        avg_quantity=(\"quantity\", \"mean\"),\n        unique_pay_items=(\"pay_item_id\", \"nunique\")\n    )\n    .reset_index()\n)\n\ntest_payitem_counts = (\n    test_raw.groupby([\"job_id\", \"contractor_id\"])\n    .size()\n    .reset_index(name=\"num_pay_items_calc\")\n)\n\ntest_data = test.copy()\n\ntest_data = test_data.merge(test_job_level_agg, on=[\"job_id\", \"contractor_id\"], how=\"left\")\ntest_data = test_data.merge(test_payitem_counts, on=[\"job_id\", \"contractor_id\"], how=\"left\")\ntest_data = test_data.merge(contractor_avg_price, on=\"contractor_id\", how=\"left\")\ntest_data = test_data.merge(contractor_category_avg, on=[\"contractor_id\"], how=\"left\")\ntest_data = test_data.merge(contractor_location_avg, on=[\"contractor_id\", \"primary_location\"], how=\"left\")\ntest_data = test_data.merge(category_avg_bid, on=\"category_id\", how=\"left\")\ntest_data = test_data.merge(location_avg_bid, on=\"primary_location\", how=\"left\")\n\n# Fill missing values\ntrain.fillna(-1, inplace=True)\ntest_data.fillna(-1, inplace=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T12:53:30.132302Z","iopub.execute_input":"2026-02-22T12:53:30.132840Z","iopub.status.idle":"2026-02-22T12:53:35.030626Z","shell.execute_reply.started":"2026-02-22T12:53:30.132810Z","shell.execute_reply":"2026-02-22T12:53:35.029825Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ===============================\n# STEP 3: TRAIN MODEL (LightGBM)\n# ===============================\n\ntarget = \"total_bid\"\n\ndrop_cols = [\"total_bid\", \"job_id\", \"contractor_id\", \"row_id\", \"bid_date\"]\nfeatures = [col for col in train.columns if col not in drop_cols]\n\n\ncategorical_cols = [\n    \"primary_location\",\n    \"job_category_description\",\n    \"contractor_id\"\n]\n\nfor col in categorical_cols:\n    train[col] = train[col].astype(\"category\")\n    test_data[col] = test_data[col].astype(\"category\")\n\n\nX = train[features]\ny = np.log1p(train[\"total_bid\"])\n\n\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)\n\nimport lightgbm as lgb\n\nmodel = lgb.LGBMRegressor(\n    n_estimators=2000,\n    learning_rate=0.01,\n    max_depth=6,\n    num_leaves=31,\n    subsample=0.8,\n    colsample_bytree=0.8,\n    random_state=42\n)\n\nmodel.fit(\n    X_train,\n    y_train,\n    eval_set=[(X_val, y_val)],\n    eval_metric=\"rmse\",\n    callbacks=[\n        lgb.early_stopping(100),\n        lgb.log_evaluation(200)\n    ]\n)\n\n# Validation score\nval_pred = model.predict(X_val)\nrmse = np.sqrt(mean_squared_error(y_val, val_pred))\nprint(\"Validation RMSE:\", rmse)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T13:02:29.120617Z","iopub.execute_input":"2026-02-22T13:02:29.121041Z","iopub.status.idle":"2026-02-22T13:06:38.789005Z","shell.execute_reply.started":"2026-02-22T13:02:29.121004Z","shell.execute_reply":"2026-02-22T13:06:38.787888Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T13:11:06.580345Z","iopub.execute_input":"2026-02-22T13:11:06.581171Z","iopub.status.idle":"2026-02-22T13:11:06.588689Z","shell.execute_reply.started":"2026-02-22T13:11:06.581137Z","shell.execute_reply":"2026-02-22T13:11:06.587291Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Get exact training features\nfeature_names = model.feature_name_\n\n# Align test\nX_test = test_data.reindex(columns=feature_names, fill_value=0)\n\n# Ensure categorical dtype\nfor col in categorical_cols:\n    if col in X_test.columns:\n        X_test[col] = X_test[col].astype(\"category\")\n\n# Predict\ntest_pred = model.predict(\n    X_test,\n    validate_features=False\n)\n\n# Cleanup\ntest_pred = np.maximum(test_pred, 0)\ntest_pred = np.nan_to_num(test_pred)\n\nsubmission = pd.DataFrame({\n    \"row_id\": test_data[\"row_id\"],\n    \"total_bid\": test_pred\n})","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T13:15:53.803470Z","iopub.execute_input":"2026-02-22T13:15:53.804210Z","iopub.status.idle":"2026-02-22T13:15:57.934249Z","shell.execute_reply.started":"2026-02-22T13:15:53.804175Z","shell.execute_reply":"2026-02-22T13:15:57.933230Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Collapse duplicates by averaging\nsubmission = submission.groupby(\"row_id\", as_index=False)[\"total_bid\"].mean()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T13:28:35.707193Z","iopub.execute_input":"2026-02-22T13:28:35.708272Z","iopub.status.idle":"2026-02-22T13:28:35.720197Z","shell.execute_reply.started":"2026-02-22T13:28:35.708233Z","shell.execute_reply":"2026-02-22T13:28:35.718959Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ===============================\n# CREATE SUBMISSION\n# ===============================\n\nsubmission.to_csv(\"submission.csv\", index=False)\n\nprint(\"Submission file created!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T13:28:39.683160Z","iopub.execute_input":"2026-02-22T13:28:39.683558Z","iopub.status.idle":"2026-02-22T13:28:39.695758Z","shell.execute_reply.started":"2026-02-22T13:28:39.683528Z","shell.execute_reply":"2026-02-22T13:28:39.694829Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(len(test_data))\nprint(len(submission))\nprint(submission[\"row_id\"].nunique())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-22T13:27:28.430622Z","iopub.execute_input":"2026-02-22T13:27:28.431240Z","iopub.status.idle":"2026-02-22T13:27:28.440336Z","shell.execute_reply.started":"2026-02-22T13:27:28.431208Z","shell.execute_reply":"2026-02-22T13:27:28.439187Z"}},"outputs":[],"execution_count":null}]}