{ "fetched_at": "2026-10-20T07:23:50Z", "source_url": "https://www.kaggle.com/competitions/challenge-02-errors-integrity-2026/data", "pages": [ { "name": "data-description", "content": "## 🧱 1. Overall structure target\n\nYou want the data tab for Challenge 02 to have **three layers**, like Challenge 01:\n\n1. **Dataset Description block** – a clean narrative summary (1–2 screens max)\n2. **Files table** – already auto-generated by Kaggle\n3. **About this directory** – a short sidebar paragraph (optional but recommended)\n4. **README.txt** – downloadable, mirrors the Dataset Description text for offline viewing\n\nRight now, Challenge 02 has a *full technical README inside the Dataset Description*, which makes the page long and redundant.\n\n---\n\n## 🧩 2. What to change in Kaggle\n\n| Section | What to do | Why |\n| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------- |\n| **Dataset Description (main editable block)** | Replace the long markdown README with a concise version like below. Keep headings 🧹 and 📁 for continuity. | Matches Challenge 01’s tone and length. |\n| **README.txt file** | Keep the detailed technical README you already wrote — that becomes the downloadable file. | Keeps depth without cluttering the web layout. |\n| **About this directory (sidebar)** | Add a short 3-line summary (I’ll provide it). | Gives context in file panel, like Challenge 01. |\n\n---\n\n## ✍️ 3. Replacement text for the main “Dataset Description” box\n\nYou can paste this directly in Kaggle’s **Edit Dataset Description**:\n\n---\n\n### 🧹 Dataset Description\n\nThis dataset accompanies **Challenge 02 – Errors → Integrity** in the *Grocery Retail – Driven by Data* series.\nIt provides a structured but imperfect grocery database — complete yet noisy — for practicing systematic **data auditing, repair, and validation**.\n\n#### 📁 Files Included\n\n| File | Description |\n| ---------------- | ------------------------------------------------------------------------- |\n| `products.csv` | Product master list with IDs, names, and vendor links. |\n| `vendors.csv` | Supplier reference table containing duplicates and mismatched keys. |\n| `stores.csv` | Store metadata with regional and time-zone variations. |\n| `sales.csv` | Transaction-level data with missing totals and duplicates for validation. |\n| `categories.csv` | Product category hierarchy, with a few orphaned entries. |\n| `README.txt` | Full technical notes and workflow guide. |\n\n#### 🧾 Objective\n\nAudit, repair, and validate the data to build a **trustworthy foundation** for analytics and forecasting in later challenges.\nParticipants identify missing or inconsistent values, apply principled fixes, and document every change transparently.\n\n#### ⚖️ License\n\n**CC BY-NC-SA 4.0** — free for non-commercial educational and research use with attribution.\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "🧩 This builds on Challenge 01 (Spaghetti → Schema) by focusing on detecting, repairing, and validating data to ensure integrity and reproducibility.\n\n**🧭 Rules – Challenge 02 Errors → Integrity**\n\nThis is an educational and prototype competition created to explore the process of building real-world data challenges on Kaggle. \nIt is not an official or prize-bearing competition. Participation is open to anyone for learning and collaboration.\n\nThis is the second challenge in the *Grocery Retail – Driven by Data* series. Like the others, it is educational and open for experimentation. \nNo prizes or formal rankings apply.\n\n\n**1️⃣ Conduct**\n\nAll participants must follow the [Kaggle Code of Conduct](https://www.kaggle.com/community-guidelines) and [Terms of Service](https://www.kaggle.com/terms). \nBe respectful, cite external sources appropriately, and share constructive feedback with others.\n\n**2️⃣ Data Usage**\n\nThe dataset provided is for educational and demonstration purposes only. \nYou may download, transform, and share derivative work for personal or academic learning, but no commercial use is implied or permitted.\n\n**3️⃣ Collaboration**\n\nTeams and discussion are encouraged. \nPublic sharing of general approaches, code snippets, or data auditing and validation ideas is welcome. \nDo not share direct solutions or private datasets that would invalidate learning goals.\n\n**4️⃣ Submissions**\n\nSince this challenge focuses on data validation and reproducibility rather than scoring metrics: \nSubmissions may include `.zip` folders, `.ipynb` notebooks, or `.sql` files. \nEvaluation will be manual and qualitative.\n\n**5️⃣ Timeline**\n\nThis challenge is currently in prototype mode and will remain open while materials and data are being refined. \nNo prizes, deadlines, or rankings apply at this stage. \nFuture versions may transition into formal scored competitions as data pipelines mature.\n\n**6️⃣ Acknowledgment**\n\nBy joining this project, you acknowledge that participation is voluntary and intended for educational experimentation and public learning.\n\n---\n\n*This challenge leads into Challenge 03 – Data → Signal, where validated data becomes predictive insights.*\n\n---", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "abstract", "content": "## 🧭 Overview — Challenge 02 Errors → Integrity\n\nA grocery chain now runs on a clean relational database — but reality isn’t clean.\nMissing transactions, drifted product codes, timestamp quirks, and duplicate records all erode confidence in analytics.\nYour mission: **detect, repair, and validate** these flaws so every metric and model rests on data that can be trusted.\n\n**Dr. V's Slide Decks :** [The Architecture of Validity](https://starshiptutor.substack.com/p/the-architecture-of-validity), [Challenge 01](https://starshiptutor.substack.com/p/challenge-01-spaghetti-schema-294), [Challenge 02](https://starshiptutor.substack.com/p/challenge-02-errors-integrity), [Challenge 03](https://starshiptutor.substack.com/p/challenge-03-data-signal)
\n**Meet the Avatars :** [Dr. Elena Vasquez](https://starshiptutor.substack.com/p/introducing-dr-elena-vasquez), [Evan Morales](https://starshiptutor.substack.com/p/introducing-evan-morales), [Maya Chen](https://starshiptutor.substack.com/p/introducing-maya-chen), [Daniel Reeve](https://starshiptutor.substack.com/p/introducing-daniel-reeve)
\n**Kaggle Pages :** [Challenge 01](https://www.kaggle.com/competitions/challenge-01-spaghetti-schema-2026), Challenge 02, [Challenge 03](https://www.kaggle.com/competitions/challenge-03-data-signal-2026)\n\nThis second challenge in the *Grocery Retail – Driven by Data* series transforms “structured” data into **reliable** data.\nYou’ll move from schema design to **systematic data auditing and principled imputation** — learning how transparency, documentation, and statistical rigor turn noise into signal without distortion.\n\nBy the end, you’ll deliver a reproducible data-quality workflow that others can run and verify: audits, repair logs, and post-imputation validations.\nIntegrity becomes the bridge between architecture and analysis — the moment when data science grows from *organized* to *trustworthy*.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "foundational-rules", "content": "The following Kaggle Competition Foundational Rules (“ Foundational Rules ”) apply to every competition regardless of whether the Sponsor creates competition-specific rules. Any competition-specific rules provided by the Sponsor are in addition to these rules, and in the case of any conflict or inconsistency, these Foundational Rules control and nullify contrary competition-specific rules.\n###GENERAL COMPETITION RULES - BINDING AGREEMENT\n####1. ELIGIBILITY\na. To be eligible to enter the Competition, you must be:\n1. a registered account holder at Kaggle.com; \n2. the older of 18 years old or the age of majority in your jurisdiction of residence (unless otherwise agreed to by Competition Sponsor and appropriate parental/guardian consents have been obtained by Competition Sponsor); \n3. not a resident of Crimea, so-called Donetsk People's Republic (DNR) or Luhansk People's Republic (LNR), Cuba, Iran, Syria, or North Korea; and\n4. not a person or representative of an entity under U.S. export controls or sanctions (see: [https://www.treasury.gov/resourcecenter/sanctions/Programs/Pages/Programs.aspx][1]).\n\nb. Competitions are open to residents of the United States and worldwide, except that if you are a resident of Crimea, so-called Donetsk People's Republic (DNR) or Luhansk People's Republic (LNR), Cuba, Iran, Syria, North Korea, or are subject to U.S. export controls or sanctions, you may not enter the Competition. Other local rules and regulations may apply to you, so please check your local laws to ensure that you are eligible to participate in skills-based competitions. The Competition Host reserves the right to forego or award alternative Prizes where needed to comply with local laws. If a winner is located in a country where prizes cannot be awarded, then they are not eligible to receive a prize.\n\nc. If you are entering as a representative of a company, educational institution or other legal entity, or on behalf of your employer, these rules are binding on you, individually, and the entity you represent or where you are an employee. If you are acting within the scope of your employment, or as an agent of another party, you warrant that such party or your employer has full knowledge of your actions and has consented thereto, including your potential receipt of a Prize. You further warrant that your actions do not violate your employer's or entity's policies and procedures. \n\nd. The Competition Sponsor reserves the right to verify eligibility and to adjudicate on any dispute at any time. If you provide any false information relating to the Competition concerning your identity, residency, mailing address, telephone number, email address, ownership of right, or information required for entering the Competition, you may be immediately disqualified from the Competition.\n\n####2. SPONSOR AND HOSTING PLATFORM\n\na. The Competition is sponsored by Competition Sponsor named above. The Competition is hosted on behalf of Competition Sponsor by Kaggle Inc. (\"Kaggle\"). Kaggle is an independent contractor of Competition Sponsor, and is not a party to this or any agreement between you and Competition Sponsor. You understand that Kaggle has no responsibility with respect to selecting the potential Competition winner(s) or awarding any Prizes. Kaggle will perform certain administrative functions relating to hosting the Competition, and you agree to abide by the provisions relating to Kaggle under these Rules. As a Kaggle.com account holder and user of the Kaggle competition platform, remember you have accepted and are subject to the Kaggle Terms of Service at [www.kaggle.com/terms][2] in addition to these Rules.\n\n####3. COMPETITION PERIOD\na. For the purposes of Prizes, the Competition will run from the Start Date and time to the Final Submission Deadline (such duration the “Competition Period”). The Competition Timeline is subject to change, and Competition Sponsor may introduce additional hurdle deadlines during the Competition Period. Any updated or additional deadlines will be publicized on the Competition Website. It is your responsibility to check the Competition Website regularly to stay informed of any deadline changes. YOU ARE RESPONSIBLE FOR DETERMINING THE CORRESPONDING TIME ZONE IN YOUR LOCATION.\n\n####4. COMPETITION ENTRY\na. NO PURCHASE NECESSARY TO ENTER OR WIN. To enter the Competition, you must register on the Competition Website prior to the Entry Deadline, and follow the instructions for developing and entering your Submission through the Competition Website. Your Submissions must be made in the manner and format, and in compliance with all other requirements, stated on the Competition Website (the \"Requirements\"). Submissions must be received before any Submission deadlines stated on the Competition Website. Submissions not received by the stated deadlines will not be eligible to receive a Prize.\nb. Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\nc. If the Competition is a multi-stage competition with temporally separate training and/or test data, one or more valid Submissions may be required during each Competition stage in the manner described on the Competition Website in order for the Submissions to be Prize eligible.\nd. Submissions are void if they are in whole or part illegible, incomplete, damaged, altered, counterfeit, obtained through fraud, or late. Competition Sponsor reserves the right to disqualify any entrant who does not follow these Rules, including making a Submission that does not meet the Requirements. \n\n####5. INDIVIDUALS AND TEAMS\na. Individual Account. You may make Submissions only under one, unique Kaggle.com account. You will be disqualified if you make Submissions through more than one Kaggle account, or attempt to falsify an account to act as your proxy. You may submit up to the maximum number of Submissions per day as specified on the Competition Website. \nb. Teams. If permitted under the Competition Website guidelines, multiple individuals may collaborate as a Team; however, you may join or form only one Team. Each Team member must be a single individual with a separate Kaggle account. You must register individually for the Competition before joining a Team. You must confirm your Team membership to make it official by responding to the Team notification message sent to your Kaggle account. Team membership may not exceed the Maximum Team Size stated on the Competition Website.\nc. Team Merger. Teams may request to merge via the Competition Website. Team mergers may be allowed provided that: (i) the combined Team does not exceed the Maximum Team Size; (ii) the number of Submissions made by the merging Teams does not exceed the number of Submissions permissible for one Team at the date of the merger request; (iii) the merger is completed before the earlier of: any merger deadline or the Competition deadline; and (iv) the proposed combined Team otherwise meets all the requirements of these Rules. \nd. Private Sharing. No private sharing outside of Teams. Privately sharing code or data outside of Teams is not permitted. It's okay to share code if made available to all Participants on the forums.\n\n####6. SUBMISSION CODE REQUIREMENTS\na. Private Code Sharing. Unless otherwise specifically permitted under the Competition Website or Competition Specific Rules above, during the Competition Period, you are not allowed to privately share source or executable code developed in connection with or based upon the Competition Data or other source or executable code relevant to the Competition (“Competition Code”). This prohibition includes sharing Competition Code between separate Teams, unless a Team merger occurs. Any such sharing of Competition Code is a breach of these Competition Rules and may result in disqualification.\nb. Public Code Sharing. You are permitted to publicly share Competition Code, provided that such public sharing does not violate the intellectual property rights of any third party. If you do choose to share Competition Code or other such code, you are required to share it on Kaggle.com on the discussion forum or notebooks associated specifically with the Competition for the benefit of all competitors. By so sharing, you are deemed to have licensed the shared code under an Open Source Initiative-approved license (see [www.opensource.org][3]) that in no event limits commercial use of such Competition Code or model containing or depending on such Competition Code.\nc. Use of Open Source. Unless otherwise stated in the Specific Competition Rules above, if open source code is used in the model to generate the Submission, then you must only use open source code licensed under an Open Source Initiative-approved license (see [www.opensource.org][4]) that in no event limits commercial use of such code or model containing or depending on such code.\n\n####7. DETERMINING WINNERS\na. Each Submission will be scored and ranked by the evaluation metric stated on the Competition Website. During the Competition Period, the current ranking will be visible on the Competition Website's Public Leaderboard. The potential winner(s) are determined solely by the leaderboard ranking on the Private Leaderboard, subject to compliance with these Rules. The Public Leaderboard will be based on the public test set and the Private Leaderboard will be based on the private test set.\nb. In the event of a tie, the Submission that was entered first to the Competition will be the winner. In the event a potential winner is disqualified for any reason, the Submission that received the next highest score rank will be chosen as the potential winner.\n\n####8. NOTIFICATION OF WINNERS & DISQUALIFICATION\na. The potential winner(s) will be notified by email. \nb. If a potential winner (i) does not respond to the notification attempt within one (1) week from the first notification attempt or (ii) notifies Kaggle within one week after the Final Submission Deadline that the potential winner does not want to be nominated as a winner or does not want to receive a Prize, then, in each case (i) and (ii) such potential winner will not receive any Prize, and an alternate potential winner will be selected from among all eligible entries received based on the Competition’s judging criteria.\nc. In case (i) and (ii) above Kaggle may disqualify the Participant. However, in case (ii) above, if requested by Kaggle, such potential winner may provide code and documentation to verify the Participant’s compliance with these Rules. If the potential winner provides code and documentation to the satisfaction of Kaggle, the Participant will not be disqualified pursuant to this paragraph.\nd. Competition Sponsor reserves the right to disqualify any Participant from the Competition if the Competition Sponsor reasonably believes that the Participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other Participants, Competition Sponsor or Kaggle.\ne. A disqualified Participant may be removed from the Competition leaderboard, at Kaggle's sole discretion. If a Participant is removed from the Competition Leaderboard, additional winning features associated with the Kaggle competition platform, for example Kaggle points or medals, may also not be awarded.\nf. The final leaderboard list will be publicly displayed at Kaggle.com. Determinations of Competition Sponsor are final and binding.\n\n####9. PRIZES\na. Prize(s) are as described on the Competition Website and are only available for winning during the time period described on the Competition Website. The odds of winning any Prize depends on the number of eligible Submissions received during the Competition Period and the skill of the Participants. \nb. All Prizes are subject to Competition Sponsor's review and verification of the Participant’s eligibility and compliance with these Rules, and the compliance of the winning Submissions with the Submissions Requirements. In the event that the Submission demonstrates non-compliance with these Competition Rules, Competition Sponsor may at its discretion take either of the following actions: (i) disqualify the Submission(s); or (ii) require the potential winner to remediate within one week after notice all issues identified in the Submission(s) (including, without limitation, the resolution of license conflicts, the fulfillment of all obligations required by software licenses, and the removal of any software that violates the software restrictions).\nc. A potential winner may decline to be nominated as a Competition winner in accordance with Section 3.8.\nd. Potential winners must return all required Prize acceptance documents within two (2) weeks following notification of such required documents, or such potential winner will be deemed to have forfeited the prize and another potential winner will be selected. Prize(s) will be awarded within approximately thirty (30) days after receipt by Competition Sponsor or Kaggle of the required Prize acceptance documents. Transfer or assignment of a Prize is not allowed. \ne. You are not eligible to receive any Prize if you do not meet the Eligibility requirements in Section 2.7 and Section 3.1 above.\nf. If a Team wins a monetary Prize, the Prize money will be allocated in even shares between the eligible Team members, unless the Team unanimously opts for a different Prize split and notifies Kaggle before Prizes are issued.\n\n####10. TAXES\na. ALL TAXES IMPOSED ON PRIZES ARE THE SOLE RESPONSIBILITY OF THE WINNERS. Payments to potential winners are subject to the express requirement that they submit all documentation requested by Competition Sponsor or Kaggle for compliance with applicable state, federal, local and foreign (including provincial) tax reporting and withholding requirements. Prizes will be net of any taxes that Competition Sponsor is required by law to withhold. If a potential winner fails to provide any required documentation or comply with applicable laws, the Prize may be forfeited and Competition Sponsor may select an alternative potential winner. Any winners who are U.S. residents will receive an IRS Form-1099 in the amount of their Prize.\n\n####11. GENERAL CONDITIONS\na. All federal, state, provincial and local laws and regulations apply.\n\n####12. PUBLICITY\na. You agree that Competition Sponsor, Kaggle and its affiliates may use your name and likeness for advertising and promotional purposes without additional compensation, unless prohibited by law.\n\n####13. PRIVACY\na. You acknowledge and agree that Competition Sponsor and Kaggle may collect, store, share and otherwise use personally identifiable information provided by you during the Kaggle account registration process and the Competition, including but not limited to, name, mailing address, phone number, and email address (“Personal Information”). Kaggle acts as an independent controller with regard to its collection, storage, sharing, and other use of this Personal Information, and will use this Personal Information in accordance with its Privacy Policy <[www.kaggle.com/privacy][6]>, including for administering the Competition. As a Kaggle.com account holder, you have the right to request access to, review, rectification, portability or deletion of any personal data held by Kaggle about you by logging into your account and/or contacting Kaggle Support at <[www.kaggle.com/contact][7]>.\nb. As part of Competition Sponsor performing this contract between you and the Competition Sponsor, Kaggle will transfer your Personal Information to Competition Sponsor, which acts as an independent controller with regard to this Personal Information. As a controller of such Personal Information, Competition Sponsor agrees to comply with all U.S. and foreign data protection obligations with regard to your Personal Information. Kaggle will transfer your Personal Information to Competition Sponsor in the country specified in the Competition Sponsor Address listed above, which may be a country outside the country of your residence. Such country may not have privacy laws and regulations similar to those of the country of your residence.\n\n####14. WARRANTY, INDEMNITY AND RELEASE \na. You warrant that your Submission is your own original work and, as such, you are the sole and exclusive owner and rights holder of the Submission, and you have the right to make the Submission and grant all required licenses. You agree not to make any Submission that: (i) infringes any third party proprietary rights, intellectual property rights, industrial property rights, personal or moral rights or any other rights, including without limitation, copyright, trademark, patent, trade secret, privacy, publicity or confidentiality obligations, or defames any person; or (ii) otherwise violates any applicable U.S. or foreign state or federal law.\nb. To the maximum extent permitted by law, you indemnify and agree to keep indemnified Competition Entities at all times from and against any liability, claims, demands, losses, damages, costs and expenses resulting from any of your acts, defaults or omissions and/or a breach of any warranty set forth herein. To the maximum extent permitted by law, you agree to defend, indemnify and hold harmless the Competition Entities from and against any and all claims, actions, suits or proceedings, as well as any and all losses, liabilities, damages, costs and expenses (including reasonable attorneys fees) arising out of or accruing from: (a) your Submission or other material uploaded or otherwise provided by you that infringes any third party proprietary rights, intellectual property rights, industrial property rights, personal or moral rights or any other rights, including without limitation, copyright, trademark, patent, trade secret, privacy, publicity or confidentiality obligations, or defames any person; (b) any misrepresentation made by you in connection with the Competition; (c) any non-compliance by you with these Rules or any applicable U.S. or foreign state or federal law; (d) claims brought by persons or entities other than the parties to these Rules arising from or related to your involvement with the Competition; and (e) your acceptance, possession, misuse or use of any Prize, or your participation in the Competition and any Competition-related activity.\nc. You hereby release Competition Entities from any liability associated with: (a) any malfunction or other problem with the Competition Website; (b) any error in the collection, processing, or retention of any Submission; or (c) any typographical or other error in the printing, offering or announcement of any Prize or winners.\n\n####15. INTERNET\na. Competition Entities are not responsible for any malfunction of the Competition Website or any late, lost, damaged, misdirected, incomplete, illegible, undeliverable, or destroyed Submissions or entry materials due to system errors, failed, incomplete or garbled computer or other telecommunication transmission malfunctions, hardware or software failures of any kind, lost or unavailable network connections, typographical or system/human errors and failures, technical malfunction(s) of any telephone network or lines, cable connections, satellite transmissions, servers or providers, or computer equipment, traffic congestion on the Internet or at the Competition Website, or any combination thereof, which may limit a Participant’s ability to participate.\n\n####16. RIGHT TO CANCEL, MODIFY OR DISQUALIFY\na. If for any reason the Competition is not capable of running as planned, including infection by computer virus, bugs, tampering, unauthorized intervention, fraud, technical failures, or any other causes which corrupt or affect the administration, security, fairness, integrity, or proper conduct of the Competition, Competition Sponsor reserves the right to cancel, terminate, modify or suspend the Competition. Competition Sponsor further reserves the right to disqualify any Participant who tampers with the submission process or any other part of the Competition or Competition Website. Any attempt by a Participant to deliberately damage any website, including the Competition Website, or undermine the legitimate operation of the Competition is a violation of criminal and civil laws. Should such an attempt be made, Competition Sponsor and Kaggle each reserves the right to seek damages from any such Participant to the fullest extent of the applicable law.\n\n####17. NOT AN OFFER OR CONTRACT OF EMPLOYMENT\na. Under no circumstances will the entry of a Submission, the awarding of a Prize, or anything in these Rules be construed as an offer or contract of employment with Competition Sponsor or any of the Competition Entities. You acknowledge that you have submitted your Submission voluntarily and not in confidence or in trust. You acknowledge that no confidential, fiduciary, agency, employment or other similar relationship is created between you and Competition Sponsor or any of the Competition Entities by your acceptance of these Rules or your entry of your Submission.\n\n####18. DEFINITIONS\na. \"Competition Data\" are the data or datasets available from the Competition Website for the purpose of use in the Competition, including any prototype or executable code provided on the Competition Website. The Competition Data will contain private and public test sets. Which data belongs to which set will not be made available to Participants. \nb. An “Entry” is when a Participant has joined, signed up, or accepted the rules of a competition. Entry is required to make a Submission to a competition.\nc. A “Final Submission” is the Submission selected by the user, or automatically selected by Kaggle in the event not selected by the user, that is/are used for final placement on the competition leaderboard.\nd. A “Participant” or “Participant User” is an individual who participates in a competition by entering the competition and making a Submission.\ne. The “Private Leaderboard” is a ranked display of Participants’ Submission scores against the private test set. The Private Leaderboard determines the final standing in the competition.\nf. The “Public Leaderboard” is a ranked display of Participants’ Submission scores against a representative sample of the test data. This leaderboard is visible throughout the competition.\ng. A “Sponsor” is responsible for hosting the competition, which includes but is not limited to providing the data for the competition, determining winners, and enforcing competition rules.\nh. A “Submission” is anything provided by the Participant to the Sponsor to be evaluated for competition purposes and determine leaderboard position. A Submission may be made as a model, notebook, prediction file, or other format as determined by the Sponsor.\ni. A “Team” is one or more Participants participating together in a Kaggle competition, by officially merging together as a Team within the competition platform.\n\n [1]: https://www.treasury.gov/resource-center/sanctions/Programs/Pages/Programs.aspx\n [2]: http://www.kaggle.com/terms\n [3]: http://www.opensource.org\n [4]: http://www.opensource.org\n [5]: https://www.kaggle.com/WinningModelDocumentationGuidelines\n [6]: http://www.kaggle.com/privacy\n [7]: http://www.kaggle.com/contact\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Series Overview", "content": "🛒 **Series Overview**\n\n**Grocery Retail – Driven by Data** is a four-part open learning and research series that follows a grocery chain’s evolution from spreadsheet chaos to data-driven strategy. \nEach challenge builds on the last—turning messy operations into trusted analytics and predictive insight.\n\nYou’ll move step by step through the modern data journey — from **structure → trust → signal → strategy**:\n\n**1️⃣ Spaghetti → Schema** \n🧩 Untangle inconsistent spreadsheets and design a normalized, analytics-ready database.\n👉 [Visit Challenge 01](https://www.kaggle.com/competitions/challenge-01-spaghetti-schema-2026)\n\n**2️⃣ Errors → Integrity** \n🧹 Audit, repair, and validate data quality so models rest on trustworthy foundations.\n\n**3️⃣ Data → Signal** \n🌤️ Reveal patterns, seasonality, and growth trends through forecasting and decomposition.\n👉 [Visit Challenge 03](https://www.kaggle.com/competitions/challenge-03-data-signal-2026)\n\n**4️⃣ Shoppers → Strategy** \n🧾 Model loyalty, simulate promotions, and measure ROI for real-world business impact.\n👉 ( Challenge Under Development )\n\n---\n\n### 🧩 The Complete Learning Arc\n\nTogether, these challenges trace a full analytical lifecycle:\n**structure → integrity → insight → impact**\n\n* **Data Engineering** (schemas, ETL, pipelines)\n* **Machine Learning** (forecasting, imputation, uplift models)\n* **Applied Business Intelligence** (ROI simulation, decision support)\nEach challenge stands alone as a reproducible case study, but together they form a living experiment in applied data science—where design, ethics, and machine learning meet the realities of modern retail.\n\nWhether you’re a student, data professional, or researcher, you can join at any stage. \nThe series is designed to grow with you—from your first clean schema to your first production-ready predictive model.\n\nJoin any challenge individually, or follow the complete arc to build a portfolio that bridges:\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "02 — About", "content": "### 🧹 Challenge 02 — About\n\n**From Structure to Trust**\n\nThe grocery chain now has a solid database — but real-world data is rarely perfect.\nIn this stage, you’ll learn how to **detect, repair, and document** common data issues so the models you build later rest on reliable foundations.\n\nAfter importing the schema from Challenge 01, you’ll run a complete **data-quality audit**: finding missing values, duplicates, unit mix-ups, and timestamp drift. Then you’ll apply **principled imputation techniques** to fix them while keeping the data’s meaning intact.\n\nBy the end, you’ll have a **clean, validated dataset** — one you can trust for forecasting, experimentation, and ROI modeling in future challenges.\n\n---\n\n#### 🧭 Series Context\n\nChallenge 02 builds directly on the schema created in Challenge 01 (Spaghetti → Schema).\nHere, you move from *organized data* to *trustworthy data* — learning the professional habits that separate working pipelines from fragile ones:\n\n* documenting every change,\n* checking assumptions, and\n* proving reproducibility.\n\nThese same practices form the bridge to **Challenge 03 (Data → Signal)**, where trustworthy data becomes the foundation for time-series forecasting.\n\n---\n\n#### 🎯 Learning Goals\n\n* Identify and describe common data-quality issues.\n* Apply multiple imputation methods and justify your choices.\n* Validate that cleaning hasn’t distorted key patterns.\n* Create transparent documentation others can audit or extend.\n\n---\n\n#### 💡 Why It Matters\n\nEvery great model depends on clean, believable data.\nChallenge 02 helps you build that trust — turning “structured but messy” datasets into solid ground for analytics and machine learning.\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "02 — Objectives", "content": "### 🎯 Challenge 02 — Objectives\n\n**From Data Chaos to Data Confidence**\n\nIn this stage, you’ll learn how to **detect, repair, and validate** data-quality issues so every downstream analysis and model rests on a trustworthy foundation.\nYou’ll practice the essential skills of a modern data scientist: turning “structured but messy” data into something reproducible, auditable, and ready for AI.\n\n---\n\n#### 🧭 Core Objectives\n\nYou’ll expand on the schema from **Challenge 01 (Spaghetti → Schema)** and build the habits of professional data stewardship:\n\n1. **Audit for errors** — Profile missing values, duplicates, drifted categories, time-zone inconsistencies, and outlier spikes.\n2. **Repair with integrity** — Apply multiple imputation methods (mean, median, regression, kNN, time-aware) while preserving data meaning.\n3. **Validate results** — Confirm that cleaning hasn’t introduced bias or distorted key distributions.\n4. **Document transparently** — Produce clear audit logs, change summaries, and reproducibility notes others can verify.\n\n---\n\n#### 🧰 Broader Learning Outcomes\n\nBy completing this challenge, you’ll learn to:\n\n* Treat **data quality as part of modeling**, not a separate cleanup task.\n* Create **reproducible audit notebooks** and data-quality reports that show your reasoning.\n* Use **Python tooling** (pandas, missingno, scikit-learn imputers) to automate checks and fixes.\n* Communicate results clearly — in both technical notebooks and concise summaries.\n\n---\n\n#### 🪜 Looking Ahead\n\nChallenge 02 is where data science becomes accountable.\nThe practices you build here — documentation, reproducibility, and statistical validation — prepare you for **Challenge 03 (Data → Signal)**, where you’ll use this trusted dataset to model patterns and forecast the future.\n\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "02 — Deliverables", "content": "### 📦 Challenge 02 — Deliverables\n\n**From Messy Reality to Reproducible Trust**\n\nIn this stage, clarity and reproducibility matter more than complexity.\nYou’ll produce a complete, auditable record showing **how messy data became trustworthy data** — from raw audit logs to the final validation report.\n\n---\n\n#### 🧩 Required Deliverables\n\n📄 **Problem Statement & Notebook Stub**\nA concise description of the business context and data-quality goals.\nInclude a Jupyter notebook skeleton with standard sections: *Setup → Audit → Repair → Validate → Summary.*\n\n📚 **Data Quality Report**\nA Markdown or HTML summary (`reports/data_quality.md`) listing:\n\n* detected issues (missing, duplicates, drift, outliers)\n* applied fixes and rationale\n* diagnostics or before/after comparisons\n\n🧮 **Audit & Repair Notebook**\nA reproducible Jupyter notebook (`notebooks/02_data_quality.ipynb`) that:\n\n* profiles missingness and drift (e.g., `missingno`, `pandas_profiling`)\n* applies and documents imputation techniques\n* runs validation queries and visual checks\n\n🧱 **Validation Scripts**\nStandalone Python utilities in `src/` to test dataset integrity:\n`evaluation.py` for referential-integrity checks, outlier detection, and simple schema validation.\n\n📊 **Baseline Re-Test Notebook**\nRetrain or rerun your baseline model from Challenge 01 on the cleaned dataset.\nCompare metrics (MAE, MAPE, RMSE) and note any improvement or variance.\n\n🗂️ **File Structure Checklist**\n\n| Folder | Files / Outputs |\n| :----------- | :------------------------------------------- |\n| `docs/` | architecture.md, change_log.md |\n| `notebooks/` | 02_data_quality.ipynb, 02_model_retest.ipynb |\n| `reports/` | data_quality.md, executive_summary.pdf |\n| `src/` | evaluation.py, helper_utils.py |\n| `slides/` | presentation_deck.pptx |\n\n---\n\n#### ✅ Acceptance Criteria\n\n* All introduced issues are **detected and either fixed or explicitly justified.**\n* Imputation steps are **documented with rationale and diagnostics.**\n* The cleaned dataset **re-trains the baseline without suspicious metric jumps.**\n* All notebooks & scripts **run end-to-end without manual intervention.**\n\n---\n\n#### 💡 Outcome\n\nA **validated, auditable, and reproducible dataset** that transforms “structured but messy” records into a reliable analytical asset — the trusted foundation for Challenge 03 (Data → Signal).\n\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "02 — Evaluation", "content": "### ⚖️ Challenge 02 — Evaluation\n\n**Accuracy, Integrity, and Reproducibility**\n\nThis challenge is about **building trust**, not chasing leaderboard points.\nYour goal is to show that your workflow can *detect, repair, and validate* data issues transparently—turning a messy dataset into something reproducible and reliable.\n\n---\n\n#### 🧩 Evaluation Principles\n\nSubmissions are reviewed on five professional data-science dimensions:\n\n| Area | What Reviewers Look For |\n| :-------------------------- | :---------------------------------------------------------------------------------------------------------------- |\n| **Audit Completeness** | All major data-quality problems identified (missing, duplicates, drift, outliers). Clear summary of findings. |\n| **Repair Integrity** | Fixes are principled and justified—imputation methods documented with rationale and diagnostics. |\n| **Validation Rigor** | Evidence that cleaning preserved statistical meaning (distributional checks, leakage tests, re-trained baseline). |\n| **Reproducibility** | Scripts and notebooks run end-to-end without manual edits; outputs match documentation. |\n| **Clarity & Documentation** | Readable reports, organized folder structure, concise executive summary. |\n\n---\n\n#### 🧮 Scoring Framework\n\nInstead of numeric scores, each submission receives a qualitative review focused on craftsmanship and learning:\n\n| Rating | Description |\n| :---------------------- | :----------------------------------------------------------------------------------------- |\n| 🥇 **Gold Integrity** | Exceptional audit depth and documentation; all fixes reproducible and statistically sound. |\n| 🥈 **Silver Integrity** | Complete, functional workflow; minor clarity or reproducibility improvements possible. |\n| 🥉 **Bronze Integrity** | Core requirements met; clear learning progress and honest methodology. |\n| 🧭 **Needs Revision** | Missing audits, unclear rationale, or non-reproducible pipeline—resubmission encouraged. |\n\n---\n\n#### 📊 What Counts as Success\n\n* All injected issues found and either fixed or explained.\n* Audit and repair steps clearly logged and repeatable.\n* Post-cleaning dataset passes validation and baseline model re-test without suspicious metric jumps.\n* Reports and notebooks readable by someone new to the project.\n\n---\n\n#### 💬 Peer and AI-Assisted Review\n\nParticipants are encouraged to review one another’s notebooks and logs.\nAI tools may suggest repairs or summaries — your job is to verify them and explain your reasoning.\nIntegrity is earned through transparent collaboration, not automation alone.\n\n---\n\n#### 🔮 Looking Ahead\n\nChallenge 03 (Data → Signal) will introduce automated metrics and forecast validation suites.\nThe standards you build here — reproducibility, clarity, and honest documentation — will become the benchmark for every future challenge in the series.\n\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "02 — Work Plan", "content": "### 🧩 Challenge 02 — Work Plan\n\n**From Errors to Integrity: Step-by-Step**\n\nThis work plan guides participants through the practical stages of **Challenge 02 – Errors → Integrity**, the second milestone in the *Grocery Retail – Driven by Data* series.\nIt defines a clear roadmap—from raw audit through validation—showing how to transform a “structured but messy” dataset into a **reproducible, trustworthy foundation** for modeling.\n\n---\n\n#### 🗺️ Workflow Overview\n\n| Phase | Focus | Outcome |\n| :------------------------------ | :--------------------------------------------------------------------------------- | :------------------------------------------------------------------------------- |\n| 1️⃣ Audit Setup | Load the database from Challenge 01; profile missingness, duplicates, and drift. | Data-quality notebook initialized with profiling visuals + summary stats. |\n| 2️⃣ Data Profiling | Use `pandas_profiling`, `missingno`, and SQL checks for range and type integrity. | Complete audit log and missingness matrix exported to `reports/data_quality.md`. |\n| 3️⃣ Rules & Standardization | Deduplicate keys, canonicalize categories, normalize units, fix timezone drift. | Clean intermediate tables with validated primary + foreign keys. |\n| 4️⃣ Imputation Methods | Apply mean/median, kNN, regression, and time-aware interpolation. Compare results. | Imputation notebook with diagnostics and side-by-side distributions. |\n| 5️⃣ Validation & Leakage Checks | Run `evaluation.py` tests for referential integrity and distributional stability. | Validation report confirming no suspicious metric jumps or bias shifts. |\n| 6️⃣ Baseline Re-Test | Retrain the Challenge 01 baseline model on the cleaned data. | Comparison metrics (MAE/MAPE/RMSE) logged and explained. |\n| 7️⃣ Documentation & Reporting | Summarize audits, fixes, and results in Markdown + PDF executive summary. | Transparent, auditable report ready for peer or AI-assisted review. |\n| 8️⃣ Presentation Summary | Create a 6–10 slide deck showing process, before/after visuals, and key insights. | Final presentation uploaded to `slides/presentation_deck.pptx`. |\n| 9️⃣ Submission Package | Assemble all artifacts in the standard folder layout. | Reproducible zip ready for submission and series continuation. |\n\n---\n\n#### 🧰 Practical Notes\n\n* **Environment:** Python 3 + PostgreSQL 15 recommended.\n* **Libraries:** `pandas`, `sqlalchemy`, `missingno`, `scikit-learn`, `matplotlib`, `seaborn`.\n* **Version Control:** Use GitHub or Kaggle Code Versions for traceability.\n* **Reproducibility:** All scripts must run end-to-end without manual edits.\n* **Validation Checks:** Row counts ✓ Foreign keys ✓ No duplicate IDs ✓ Type coercion ✓\n\n---\n\n#### ✅ Deliverables Checklist\n\n| Category | Files / Outputs |\n| :------------- | :------------------------------------------------------------------------------- |\n| Docs & Reports | `docs/change_log.md`, `reports/data_quality.md`, `reports/executive_summary.pdf` |\n| Notebooks | `02_data_quality.ipynb`, `02_model_retest.ipynb` |\n| SQL / Scripts | `src/evaluation.py`, optional validation queries |\n| Slides | `slides/presentation_deck.pptx` |\n| Archive | `submission_package.zip` with reproducible folder structure |\n\n---\n\n#### 🧭 Outcome\n\nBy following this plan, participants will produce a **validated, auditable dataset** and a **fully documented workflow** proving that each repair was justified and reproducible.\nThis stage turns data stewardship into an analytic discipline — laying the groundwork for **Challenge 03 (Data → Signal)**, where clean data evolves into predictive insight.\n\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "02 — Tagline", "content": "### 🏷️ Challenge 02 — Tagline\n\n**From errors to integrity — building trust in every dataset.**\n\nThis challenge is where structure becomes confidence.\nYou’ll learn to see data not just as numbers, but as evidence — audited, repaired, and verified.\nEach fix, each log, and each validation step strengthens the foundation for forecasting, strategy, and AI-ready analytics to come.\n\nIntegrity isn’t an afterthought; it’s the proof that your data — and your work — can be trusted.\n\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Project Vision - Grocery Retail – Driven by Data", "content": "### 🚀 Project Vision — Grocery Retail – Driven by Data\n\n**Why this series exists**\nModern grocery retail is where data complexity meets human decision-making.\nEvery day, inventory, pricing, loyalty, weather, and promotions overlap — creating the perfect environment for analytics and AI to make measurable impact.\n\nGrocery Retail – Driven by Data turns this reality into a living laboratory — four open challenges that trace the evolution from spreadsheet chaos to predictive strategy.\n\n---\n\n**What we aim to achieve**\nOver the next several years, this initiative will:\n\n* Demonstrate how organizations evolve from ad-hoc spreadsheets to reproducible, AI-assisted analytics.\n* Provide open, end-to-end examples of data engineering, modeling, and business interpretation.\n* Explore how human reasoning, domain expertise, and AI collaboration combine to produce trustworthy insight.\n* Build an educational and research framework connecting students, professionals, and industry mentors.\n\n---\n\n**The bigger picture**\nEach challenge represents one step in the modern analytics journey:\n\n1. **Structure** – Design clean relational schemas from chaos.\n2. **Integrity** – Validate, repair, and document data quality.\n3. **Signal** – Model temporal and behavioral patterns for forecasting.\n4. **Strategy** – Quantify ROI and simulate decisions that drive profit and loyalty.\n\nTogether, they form a reproducible playbook for data-driven retail — balancing technical rigor, business context, and ethical transparency.\n\n---\n\n**Why it matters**\nBy embedding research inside realistic, open competitions, we can observe how teams learn, collaborate, and govern their analytics as AI tools mature.\nThe goal is not automation for its own sake, but shared understanding — how humans and machines co-create insight responsibly, reproducibly, and at scale.\n\nEvery step — from *Spaghetti → Schema* onward — contributes to a more interpretable, human-centered data future, where data, AI, and decision-making evolve together toward clarity and trust.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Project Roadmap", "content": "## 🗺️ Project Roadmap — Grocery Retail – Driven by Data\n\n**Purpose**\nThe *Grocery Retail – Driven by Data* roadmap defines how this multi-year project grows from classroom challenges to an open research ecosystem on applied data science and human–AI collaboration.\nEach phase expands scope, automation, and interpretability—moving from manual craftsmanship to guided collaboration and, eventually, AI-assisted autonomy.\n\n---\n\n### **Phase 1 — Foundations (Year 1)**\n\n**Goal:** Build a shared baseline for structure, trust, and reproducibility.\n\n* Launch the four core challenges covering schema design, data quality, forecasting, and ROI modeling.\n* Establish standard file structures, naming conventions, and validation scripts.\n* Collect reproducible artifacts (ERDs, ETL pipelines, notebooks, evaluation reports).\n* Observe how participants use AI as a **Tutor**—seeking guidance, explanations, and code feedback.\n\n**Outcome:**\nA complete, transparent workflow from spreadsheets to structured data, serving as the foundation for all future phases.\n\n---\n\n### **Phase 2 — Collaboration (Years 2–3)**\n\n**Goal:** Expand from assisted learning to productive co-creation.\n\n* Transition into the **AI Coder** mode, where AI generates candidate code, models, and diagnostics while humans validate logic and ethics.\n* Introduce more complex, multi-store and multi-year datasets with realistic data quality issues.\n* Compare human-initiated vs. AI-initiated pipelines for accuracy, interpretability, and reproducibility.\n* Partner with universities and research labs to analyze learning patterns, revision strategies, and error correction.\n\n**Outcome:**\nDocumented design patterns for human–AI collaboration that balance creativity, quality assurance, and accountability.\n\n---\n\n### **Phase 3 — Autonomy (Years 4–5)**\n\n**Goal:** Explore responsible automation under human oversight.\n\n* Advance to the **AI Agent** phase—AI executes ingestion, transformation, modeling, and reporting based on human-defined intent.\n* Develop reproducibility and explainability metrics to evaluate AI-led pipelines.\n* Conduct controlled studies on co-validation, where humans audit AI-generated analyses for logic, bias, and implications.\n* Publish governance guidelines for mixed human–AI analytical teams.\n\n**Outcome:**\nA transparent benchmark for AI-assisted analytics—showing how automation, validation, and ethics can coexist in real-world data science.\n\n---\n\n### **Deliverable Vision**\n\nBy the end of the roadmap, the series will provide:\n\n* A longitudinal dataset tracing human–AI collaboration.\n* Reproducible benchmarks for schema quality, forecasting accuracy, and ROI modeling.\n* Educational modules that map directly into academic and industry training.\n* A practical framework for ethical, interpretable, and collaborative AI-assisted analytics.\n\n**Long-term goal:**\nNot automation for its own sake—but collaboration.\nThe project seeks to build shared understanding of how humans and AI co-create insight responsibly, reproducibly, and at scale.\n\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }