### 🧩 Challenge 02 — Work Plan

**From Errors to Integrity: Step-by-Step**

This work plan guides participants through the practical stages of **Challenge 02 – Errors → Integrity**, the second milestone in the *Grocery Retail – Driven by Data* series.
It defines a clear roadmap—from raw audit through validation—showing how to transform a “structured but messy” dataset into a **reproducible, trustworthy foundation** for modeling.

---

#### 🗺️ Workflow Overview

| Phase                           | Focus                                                                              | Outcome                                                                          |
| :------------------------------ | :--------------------------------------------------------------------------------- | :------------------------------------------------------------------------------- |
| 1️⃣ Audit Setup                 | Load the database from Challenge 01; profile missingness, duplicates, and drift.   | Data-quality notebook initialized with profiling visuals + summary stats.        |
| 2️⃣ Data Profiling              | Use `pandas_profiling`, `missingno`, and SQL checks for range and type integrity.  | Complete audit log and missingness matrix exported to `reports/data_quality.md`. |
| 3️⃣ Rules & Standardization     | Deduplicate keys, canonicalize categories, normalize units, fix timezone drift.    | Clean intermediate tables with validated primary + foreign keys.                 |
| 4️⃣ Imputation Methods          | Apply mean/median, kNN, regression, and time-aware interpolation. Compare results. | Imputation notebook with diagnostics and side-by-side distributions.             |
| 5️⃣ Validation & Leakage Checks | Run `evaluation.py` tests for referential integrity and distributional stability.  | Validation report confirming no suspicious metric jumps or bias shifts.          |
| 6️⃣ Baseline Re-Test            | Retrain the Challenge 01 baseline model on the cleaned data.                       | Comparison metrics (MAE/MAPE/RMSE) logged and explained.                         |
| 7️⃣ Documentation & Reporting   | Summarize audits, fixes, and results in Markdown + PDF executive summary.          | Transparent, auditable report ready for peer or AI-assisted review.              |
| 8️⃣ Presentation Summary        | Create a 6–10 slide deck showing process, before/after visuals, and key insights.  | Final presentation uploaded to `slides/presentation_deck.pptx`.                  |
| 9️⃣ Submission Package          | Assemble all artifacts in the standard folder layout.                              | Reproducible zip ready for submission and series continuation.                   |

---

#### 🧰 Practical Notes

* **Environment:** Python 3 + PostgreSQL 15 recommended.
* **Libraries:** `pandas`, `sqlalchemy`, `missingno`, `scikit-learn`, `matplotlib`, `seaborn`.
* **Version Control:** Use GitHub or Kaggle Code Versions for traceability.
* **Reproducibility:** All scripts must run end-to-end without manual edits.
* **Validation Checks:** Row counts ✓ Foreign keys ✓ No duplicate IDs ✓ Type coercion ✓

---

#### ✅ Deliverables Checklist

| Category       | Files / Outputs                                                                  |
| :------------- | :------------------------------------------------------------------------------- |
| Docs & Reports | `docs/change_log.md`, `reports/data_quality.md`, `reports/executive_summary.pdf` |
| Notebooks      | `02_data_quality.ipynb`, `02_model_retest.ipynb`                                 |
| SQL / Scripts  | `src/evaluation.py`, optional validation queries                                 |
| Slides         | `slides/presentation_deck.pptx`                                                  |
| Archive        | `submission_package.zip` with reproducible folder structure                      |

---

#### 🧭 Outcome

By following this plan, participants will produce a **validated, auditable dataset** and a **fully documented workflow** proving that each repair was justified and reproducible.
This stage turns data stewardship into an analytic discipline — laying the groundwork for **Challenge 03 (Data → Signal)**, where clean data evolves into predictive insight.