# SmilyAI LLM Competition - Official Rules ## ENTRY IN THIS COMPETITION CONSTITUTES YOUR ACCEPTANCE OF THESE OFFICIAL COMPETITION RULES. ## 1. COMPETITION-SPECIFIC TERMS ### 1.1 COMPETITION TITLE SmilyAI LLM Challenge: Advancing Reasoning Capabilities ### 1.2 COMPETITION SPONSOR SmilyAI Labs ### 1.3 COMPETITION WEBSITE https://www.kaggle.com/competitions/smilyai-llm-challenge ### 1.6 WINNER LICENSE TYPE Open Source ### 1.7 DATA ACCESS AND USE Competition Use and Commercial - participants may use data for any purpose --- ## 2. COMPETITION-SPECIFIC RULES ### 2.1 COMPETITION OBJECTIVE Develop a large language model that achieves a score of 40+ on MMLU (Massive Multitask Language Understanding) reasoning benchmarks. Participants are encouraged to push the boundaries of reasoning capabilities in language models. ### 2.2 TEAM LIMITS a. The maximum Team size is five (5). b. Team mergers are allowed and can be performed by the Team leader. In order to merge, the combined Team must have a total Submission count less than or equal to the maximum allowed as of the Team Merger Deadline. ### 2.3 SUBMISSION LIMITS a. You may submit a maximum of five (5) Submissions per day. b. You may select up to two (2) Final Submissions for judging. ### 2.4 COMPETITION TIMELINE a. Competition Timeline dates (including Entry Deadline, Final Submission Deadline, Start Date, and Team Merger Deadline) are reflected on the competition's Overview > Timeline page. ### 2.5 COMPETITION DATA #### a. Data Access and Use You may access and use the Competition Data for any purpose, whether commercial or non-commercial, including for participating in the Competition and on Kaggle.com forums, and for academic research and education. The Competition Sponsor reserves the right to disqualify any Participant who uses the Competition Data other than as permitted by the Competition Website and these Rules. #### b. Data Security You agree to use reasonable and suitable measures to prevent persons who have not formally agreed to these Rules from gaining access to the Competition Data. You agree not to transmit, duplicate, publish, redistribute or otherwise provide or make available the Competition Data to any party not participating in the Competition. ### 2.6 EVALUATION METRIC Models will be evaluated on MMLU reasoning tasks. The primary metric is accuracy on MMLU reasoning subtasks, with a minimum threshold of 40% required for prize eligibility. ### 2.7 WINNER LICENSE a. You hereby license and will license your winning Submission and the source code used to generate the Submission under an Open Source Initiative-approved license (see www.opensource.org) that in no event limits commercial use of such code or model containing or depending on such code. b. For generally commercially available software that you used to generate your Submission that is not owned by you, but that can be procured by the Competition Sponsor without undue expense, you do not need to grant the license in the preceding Section for that software. c. In the event that input data or pretrained models with an incompatible license are used to generate your winning solution, you do not need to grant an open source license in the preceding Section for that data and/or model(s). d. You may be required by the Sponsor to provide a detailed description of how the winning Submission was generated, including methodology, architecture, preprocessing, training details, and hyperparameters, with a link to a code repository with complete instructions for reproduction. ### 2.8 EXTERNAL DATA AND TOOLS #### a. External Data - OPEN POLICY **You may use ANY publicly available datasets or external data sources** to develop and test your Submissions. This includes but is not limited to: - Public datasets (Common Crawl, Wikipedia, books, academic papers, etc.) - Synthetic data you generate - Proprietary datasets you have access to - Any other data sources you can legally access The only requirement is that you document what data you used in your final submission documentation. #### b. Pretrained Models - OPEN POLICY **You may use ANY pretrained models as a starting point**, including: - Open source models (LLaMA, Mistral, Falcon, etc.) - Models from commercial providers (if licensing permits) - Your own pretrained models - Fine-tuned versions of existing models #### c. Reasonableness Standard The use of external data and models is fully acceptable. While there are no strict cost limitations, the Reasonableness Standard applies: tools and data should be accessible enough that winners can document their approach for reproducibility. #### d. Automated Machine Learning Tools (AMLT) Individual Participants and Teams may use any automated machine learning tools or frameworks to create a Submission, provided that they ensure compliance with licensing requirements and can document their methodology. ### 2.9 ELIGIBILITY a. Unless otherwise stated or prohibited by internal policies of the Competition Entities, employees, interns, contractors, officers and directors of Competition Entities may enter and participate in the Competition, but are not eligible to win any Prizes. ### 2.10 WINNER'S OBLIGATIONS As a condition to being awarded a Prize, a Prize winner must fulfill the following obligations: a. **Code Delivery**: Deliver to the Competition Sponsor the final model's software code as used to generate the winning Submission and associated documentation, including: - Training code - Inference code - Description of computational environment and resources required - Documentation following standard best practices b. **Model Documentation**: Provide a detailed write-up describing: - Data sources used (with appropriate citations) - Model architecture and design decisions - Training methodology and hyperparameters - Preprocessing and post-processing steps - Any pretrained models or external tools used c. **Reproducibility**: Ensure that your documentation enables others to understand and reproduce your approach, within reasonable computational constraints. d. **License Grant**: Grant to the Competition Sponsor the open source license stated in Section 2.7. e. **Legal Documentation**: Sign and return all Prize acceptance documents as may be required by Competition Sponsor or Kaggle, including eligibility certifications, releases, and tax forms. ### 2.11 GOVERNING LAW All claims arising out of or relating to these Rules will be governed by California law, excluding its conflict of laws rules, and will be litigated exclusively in the Federal or State courts of Santa Clara County, California, USA. --- ## 3. GENERAL PROVISIONS ### 3.1 NO PURCHASE NECESSARY No purchase is necessary to enter or win. Entry is free. ### 3.2 MULTIPLE ACCOUNTS PROHIBITED You cannot sign up to Kaggle from multiple accounts and therefore you cannot enter or submit from multiple accounts. ### 3.3 FAIR PLAY All Participants must compete fairly and ethically. Any attempt to manipulate the competition through fraudulent means will result in disqualification. --- **Good luck to all participants! We're excited to see innovative approaches to advancing reasoning capabilities in language models.**