{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Kaggle AI Report 2023:
The Impact of Kaggle Competitions on the ML Community: Lessons Learned and Advancements Achieved
\nAuthor: Juan Ricardo P.
\n ","metadata":{}},{"cell_type":"markdown","source":"Preparation Phase
\n\nWe accessed a dataset of 59 competitions with their first solutions write-ups via Farid.one/kaggle-solutions. We compiled an unstructured dataset using this and Carlens 'The State of Competitive Machine Learning' as a reference.\n\nThis dataset, including an overview of each competition and its solution, was converted to a structured, tabular format using OpenAI. We sorted competitions by industry, and summaries of each were generated for analysis.\n\nWe also referenced Kenjee's 2023 research on AI/ML paper submission growth and Carlens' framework throughout the process. This process allowed us to understand competition solutions and AI/ML development trends.","metadata":{}},{"cell_type":"markdown","source":"# 1. Introduction
\nKaggle Competitions, a vital platform for machine learning(ML), enable collaboration among data scientists, researchers, and enthusiasts to solve real-world problems (Kaggle, 2021). These competitions have significantly influenced the ML landscape, driving advancements over the past two years. Our analysis of competition writeups, shared code, and academic references aims to explore the lessons learned and evaluate the broader impact on the ML community. \n\nThrough this analysis, we aim to address a fundamental research question: how have Kaggle Competitions shaped the machine learning community and contributed to advancements in the field? We delve into how Kaggle Competitions have shaped the field and contributed to its advancements. This includes highlighting key lessons such as feature engineering, ensemble methods, hyperparameter optimization, and collaborative learning. We also examine specific sectors like computer vision, healthcare, finance, and NLP, revealing how these competitions have profoundly influenced diverse industries.\n\n Additionally, we address potential drawbacks like a performance-centric focus and reproducibility challenges. By exploring these aspects, we assess the overall benefits and contributions of Kaggle Competitions to the ML community.","metadata":{}},{"cell_type":"markdown","source":"# 2. Key Trends and Insights in Competitive Machine Learning
\nCompetitive machine learning has witnessed several prominent trends and insights that have shaped the strategies and approaches of participants.\n\nOne key trend is the convergence of successful competitors and core tools. Python, PyData, PyTorch, TensorFlow, and gradient-boosted decision trees have become core tools among successful competitors. Deep learning, while not entirely replacing gradient-boosted decision trees, has demonstrated added value when combined with boosting methods (Carlens, 2023).\n\nAnother trend is the dominance of Transformers in Natural Language Processing (NLP) and computer vision. Transformers have established their dominance in NLP tasks and are beginning to rival traditional Convolutional Neural Networks (CNNs) in computer vision tasks (Carlens, 2023).\n\nMachine learning competitions also cover a wide range of research areas. These competitions span diverse domains such as computer vision, healthcare/medical, finance and quantitative trading, NLP, games, and more. Each area presents unique challenges and opportunities for innovation.\n\nThe success of ensembles and single-model solutions is another notable trend. While large ensembles remain common among competition winners, single-model solutions have also achieved success. Ensembles, whether implicit or explicit, continue to play a pivotal role in achieving high-performance models (Carlens, 2023).\n\nBy summarizing these key trends and insights, we gain a better understanding of the advancements and strategies that have emerged within the ML community through their participation in Kaggle Competitions. These insights provide a foundation for further exploration and analysis of the impact of these competitions on the broader field of machine learning.\n","metadata":{}},{"cell_type":"markdown","source":"\n\nFig. (Carlens, 2023)","metadata":{}},{"cell_type":"markdown","source":"# 3. Lessons Learned from Kaggle Competitions
Kaggle Competitions have enabled the ML community to gain valuable insights and knowledge. Here are some key lessons learned:\n\n3.1 Feature Engineering and Preprocessing: Participants have discovered the importance of feature engineering and data preprocessing techniques. Through Kaggle Competitions, practitioners have explored various strategies such as feature selection, dimensionality reduction, outlier detection, and data augmentation. This emphasis on data manipulation has led to more robust ML models and improved performance (Kaggle, 2021).\n\n3.2 Model Selection and Stacking: Competitions have fostered an environment where participants can experiment with a wide range of ML models. The community has gained insights into the strengths and weaknesses of different algorithms, enabling them to make informed choices for specific tasks. Additionally, participants have explored the power of model stacking and ensemble methods to further enhance predictive performance (Pavan, 2021).\n\n3.3 Hyperparameter Optimization: Kaggle Competitions have highlighted the significance of hyperparameter tuning. Researchers and practitioners have experimented with different optimization algorithms and search strategies, leveraging techniques such as grid search, random search, Bayesian optimization, and evolutionary algorithms. This emphasis on hyperparameter optimization has led to improved model performance and generalization (Brownlee, 2021).\n\n3.4 Collaborative Learning and Open Source: Kaggle Competitions have encouraged collaboration and knowledge-sharing within the ML community. Participants often publish their code, methodologies, and insights, fostering a culture of open-source contributions. This collective effort has accelerated learning, allowed for peer review, and facilitated the reproducibility of state-of-the-art models.","metadata":{}},{"cell_type":"markdown","source":"# 4. Summary of Learnings from Kaggle Competitions
\nIn order to provide a comprehensive overview of the learnings from Kaggle Competitions, the information gathered from competition write-ups, solution approaches, and available data has been organized and grouped by different sectors. This systematic approach allows for a clearer understanding of the impact of these competitions within specific industries and research areas. The following sections provide a brief overview of each sector before delving into specific examples.\n\nI. Computer Vision Competitions:
\n - This sector encompasses competitions related to computer vision tasks, such as image recognition and object detection.
\n1. H&M Personalized Fashion Recommendations: (Kaggle, (2022, February 7)).
\nThe competition aimed to develop personalized product recommendations in the fashion domain. The winning solution involved retrieval strategies, feature engineering, and gradient boosting decision tree models. Participants discovered the importance of extensive interaction features between users and items for recommendation problems. They also learned the significance of incorporating recent trends, seasonality, and collaborative filtering scores in feature engineering. \nThe solution's performance showcased the effectiveness of ensemble models and the benefits of utilizing high-performance hardware resources, such as NVIDIA A100 GPUs, to train a deep learning model that could recommend fashion items to customers more accurately (Senkin13, 2022).
\n\n2. Image Matching Challenge 2022: (Kaggle, (2022, April 4)).
\nThe competition aimed to register and match images taken from different viewpoints. Participants utilized ensemble methods and multiple models such as LoFTR, SuperGlue, and DKM for image matching. They learned the significance of multi-resolution matching, clustering, and post-processing techniques for accurate registration. The competition emphasized the importance of leveraging pre-trained models, such as LoFTR, and applying fusion techniques to improve overall performance (Yumuneko, 2022).
\n\n3. Google Landmark Recognition 2021: (Kaggle, (2021, August 10)).
\nThe competition involved labeling landmarks in images. Participants utilized ensemble models combining DOLG models and hybrid Swin transformers. They gained insights into deep learning architectures, ensemble techniques, and image recognition for landmark identification. The ensemble models underwent training and optimization on a high-performance computing setup, utilizing distributed data parallel across 8xV100 NVIDIA GPUs. The competition highlighted the significance of image analysis and classification in the field of computer vision and demonstrated the potential of ML for preserving cultural heritage (Dieter, 2021).
\n\nII. Healthcare Competitions::
\n - Competitions in this sector focus on healthcare-related challenges, ranging from clinical concept prediction to brain tumor classification.
\n1. NBME - Score Clinical Patient Notes: (Kaggle, (2022, February 1)).
\nIn this competition, participants focused on predicting clinical concepts from patient notes. The winning solution approach involved blending multiple models and applying techniques such as MLM, pseudo-labeling, augmentation, and ensemble blending. The ML community learned the importance of diverse model architectures, data augmentation, and ensembling for achieving high performance. The competition highlighted the potential of transfer learning and model blending in the medical domain (Currypurin, 2022).
\n\n2. RSNA-MICCAI Brain Tumor Radiogenomic Classification: (kaggle, (2021, July 13)).
\nThe competition aimed to predict the genetic subtype of glioblastoma using MRI scans. The winning solution employed a 3D Convolutional Neural Network (CNN) with a ResNet10 architecture. It showcased the importance of deep learning models and advanced training strategies in leveraging imaging data for genetic classification of brain tumors (Firas Baba, 2021).
\n\n3. Mayo Clinic - STRIP AI - Healthcare: (Kaggle, (2022, July 6)).
\nThe competition focused on classifying the blood clot origins in ischemic stroke using digital pathology images. Participants utilized Swin Transformer models, attention pooling, and ensemble techniques. The competition demonstrated the potential of ML in improving diagnostic capabilities and therapeutic management in stroke patients (Khyeh, 2022).
\n\nIII. Quantitative Trading Competitions:
\n - This sector involves competitions centered around quantitative trading and market prediction models. Prominent competitions include:
\n \n1. Jane Street Market Prediction: (Kaggle, (2021, February 15)).
\nThe competition aimed to develop a quantitative trading model for predicting market movements. Participants utilized deep learning models such as supervised autoencoder MLP and LSTM. They learned the importance of feature engineering, model optimization, and ensemble methods for accurate market prediction. The competition showcased the significance of ML in quantitative finance and highlighted the potential of machine learning for trading strategies (Yirun Zhang, 2021).
\n\n2. G-Research Crypto Forecasting: (Kaggle, (2021, November 2)).
\nIn this competition, participants focused on forecasting short-term returns in cryptocurrencies using machine learning techniques. The winning solution approach involved a supervised autoencoder and MLP ensemble. The community learned that leveraging autoencoders for denoising and feature generation can improve prediction accuracy. Additionally, using data augmentation techniques like adding Gaussian noise and incorporating target information in the autoencoder proved effective for preventing overfitting and generating more relevant features (Yam Peleg, 2021)).
\n\n3. Optiver Realized Volatility Prediction: (Kaggle, (2021, June 28)).
\nThe competition focused on predicting short-term volatility for stocks. Participants utilized optimization algorithms and heuristics to design models that accurately forecasted volatility over 10-minute periods. The competition demonstrated the ML community's understanding of volatility prediction in financial markets and their ability to leverage optimization techniques for improved predictions (Nyanp, 2021).
\nIV. Finance Competitions:
\n- Competitions in the finance sector concentrate on tasks like credit default prediction and building recommender systems. Examples include:
\n\n1. American Express - Default Prediction: (Kaggle, (2022, May 25)).
\nIn the competition, the ML community learned about credit default prediction models in the finance sector. Participants utilized machine learning techniques, including ensemble models such as LightGBM and neural networks, to improve the existing credit default prediction models used in consumer lending. The competition emphasized the importance of risk management and customer experience in the finance industry (Daishu, 2022). Additionally, other participants benefited from Nvidia's GPU compute resources, specifically 4xV100 32GB, which expedited their experimentation process.
\n\nV. Natural Language Processing (NLP) Competitions:
\n- This sector covers competitions related to NLP tasks such as sentiment analysis and question answering. Prominent competitions include:
\n\n1. Jigsaw Rate Severity of Toxic Comments: (Kaggle, (2021, November 8)).
\nThis competition focused on ranking the severity of toxicity in comments. Participants utilized RoBERTa and DeBERTa models and trained linear models on predicted probabilities. The community learned that combining transformer-based models with linear models can effectively rank and predict the toxicity levels. The use of genetic algorithms for model optimization and ensembling multiple models further improved performance (Guanshuo Xu, 2021).
\n\n2. Chaii - Hindi and Tamil Question Answering: (Kaggle, (2021, August 11).
\nIn this competition, the ML community focused on improving natural language understanding models in Indian languages, specifically Hindi and Tamil. Participants utilized standard question answering models like Auto Mode lFor Question Answering and employed multiple backbone models such as XLM-Roberta Large, MURIL Large, and Rembert. They learned the significance of data preprocessing, feature engineering, and ensemble methods in achieving accurate question answering predictions. The competition showcased the importance of catering to diverse languages and provided insights into multilingual NLP and question answering research (Kłeczek, 2021).
\n\nVI. Education Competitions:
\n - Competitions in this sector focus on developing automated tools for evaluating student writing and providing feedback. Notable examples include:
\n1. Feedback Prize - Evaluating Student Writing: (kaggle, (2021, December 14)).
\nThe competition aimed to develop automated writing feedback tools for evaluating student writing. Participants leveraged BERT-based models such as Longformer-Large, RoBERTa-Large, DeBERTa-XXLarge, distilbart_mnli_12_9, and BART_large_finetuned_squadv1 for ensemble modeling. They learned the importance of feature engineering, cross-validation, and ensemble techniques for accurate essay evaluation. The competition highlighted the significance of NLP in the education sector and showcased the potential of machine learning for automated feedback systems (Wht1996, 2021).
\n\nVII. Retail and Shopping Competitions:
\n- Competitions in this sector revolve around tasks like predicting the appeal of pet photos and building recommender systems for e-commerce. Examples include:
\n\n1. PetFinder.my - Pawpularity Contest: (Kaggle, (2021, September 23)).
\nThis competition focused on predicting the pawpularity (appeal) of pet photos. Participants utilized transfer learning, SVR, CNN, and Vision Transformer models. The ML community learned the importance of feature extraction using pretrained architectures, ensembling models, and applying augmentations during training and inference. They also gained insights into leveraging transfer learning and combining multiple models to achieve accurate predictions (GibA, 2021).
\n\n2. OTTO - Multi-Objective Recommender System: (Kaggle, (2022, November 1)).
\nThe competition targeted the retail and shopping sector, specifically building a multi-objective recommender system for e-commerce sessions. ML practitioners employed neural networks (MLP or Transformer) for candidate generation and LightGBM Ranker for reranking recommendations. Through extensive experimentation and optimization of ranking features, these models improved the accuracy and relevance of recommendations. This competition provided insights into recommender systems, neural networks, and session embeddings, showcasing the ML community's progress in the retail industry (MrKMaKr, 2022).
\n\nVIII. Sports Competitions:
\n- Competitions in the sports sector involve tasks like predicting outcomes of basketball tournaments and forecasting fan engagement. Examples include:
\n1. March Machine Learning Mania 2022 (Men's and Women's): (Kaggle, (2022, March 12-13)).
\nThese competitions involved predicting the outcomes of the US men's and women's college basketball tournaments. Participants used a variety of models such as LGBM, CNN, and SVR, along with ensemble techniques. The ML community discovered the effectiveness of transfer learning, feature engineering, and ensemble methods for accurate tournament outcome prediction. They also learned the significance of hyperparameter tuning, augmentations, and data splits for improving model performance (Shin Koshi 2022), (Koki Ando 2022).
\n\n2. MLB Player Digital Engagement Forecasting: (Kaggle, (2021, June 10)).
\nIn this competition, the ML community focused on predicting fan engagement with MLB players' digital content. Participants developed neural network models, including GRU, for predicting engagement. They gained insights into feature engineering, model architecture design, and post-processing techniques for accurate engagement forecasting. The competition emphasized the significance of ML in sports analytics and fan interaction (Ph0921, 2021).
\nIX. Video Games and Simulation Competitions:
\n - This sector covers competitions related to video games and simulated environments.
\n \n 1. Kore 2022: (Kaggle, (2022, April 12)).
\nIn the competition, the ML community tackled a turn-based simulation game that involved fleet management, resource gathering, and strategic decision-making. Participants applied reinforcement learning and game theory techniques to optimize fleet movements and maximize resource collection. The competition highlighted the ML community's understanding of reinforcement learning algorithms and their application in simulated gaming environments (Harm Buisman, 2022).
\n\n2. Lux AI Challenge: (Kaggle, ((2023, January 25)).
\nThe Lux AI Challenge provided an opportunity for the ML community to showcase their expertise in reinforcement learning and game theory. Participants designed agents to optimize resource gathering and allocation in a simulated 1v1 scenario. The competition emphasized the use of deep reinforcement learning techniques, including convolutional neural networks (CNNs), to develop agents that could outsmart opponents and maximize resource collection. The challenge highlighted the importance of strategic thinking and decision-making skills in complex game environments (Ryan Anderson 2023).
\n\nThese are just a few examples of the competitions grouped by sector. There were several more competitions that covered various other domains and applications.
","metadata":{}},{"cell_type":"markdown","source":"# 5. Positive and Negative Aspects
\nKaggle competitions have brought significant benefits to the ML community, but it is also important to acknowledge the potential drawbacks.\n\n5.1 Positive Impact:\n\n Kaggle competitions have had a profound positive impact on the ML community in several ways. Firstly, participants have the opportunity to hone their ML skills and gain practical experience by working on real-world problems. Through competition challenges, participants learn new techniques, explore innovative approaches, and broaden their expertise. \n\nSecondly, competitions provide a platform for participants to connect and collaborate with data scientists, researchers, and industry professionals. These networking opportunities foster knowledge exchange, partnerships, and mutual learning, contributing to professional growth and community-building. \n\nLastly, the availability of public leaderboards and evaluation metrics allows participants to benchmark their models against others. This fosters healthy competition, encourages continuous improvement, and pushes participants to strive for higher performance.\n\n5.2 Negative Impact:\n\n Firstly, in some cases, the competitive nature of Kaggle competitions may lead to a singular focus on maximizing performance metrics. This narrow focus may overshadow other important considerations such as model interpretability, fairness, and ethical implications. \n\nSecondly, while Kaggle offers a wide range of competitions, it is important to ensure diverse representation in the datasets. There is a need to improve the availability of datasets that encompass underrepresented domains or populations, promoting inclusivity and reducing potential biases. \n\nLastly, reproducing winning solutions can be challenging. Despite efforts to encourage code sharing and documentation, limited access to the full dataset, time constraints, or proprietary algorithms may hinder the reproducibility of state-of-the-art models.\n\nBy clearly delineating the positive and negative aspects, we can better appreciate the benefits of Kaggle competitions while also acknowledging the challenges and areas for improvement. This balanced analysis contributes to a more nuanced understanding of the impact of Kaggle competitions on the ML community.\n","metadata":{}},{"cell_type":"markdown","source":"# 6. Overall Progress and Advancements in the Field of Machine Learning
\n6.1. Strategies and Tools: \n\nThe field of machine learning (ML) has witnessed remarkable progress and advancements, driven by the rapid growth and evolution of artificial intelligence (AI) (Krenn, 2022). This expansion is evident from the increasing number of AI and ML paper submissions, reflecting the growing interest and participation in this field (Kenjee, 2023). This surge in research and innovation serves as a powerful motivator for the ML community to embrace a mindset of continuous learning and exploration.\n \n\n\n\nFig. (Kenjee, 2023)\n \n To stay at the forefront of ML, practitioners are encouraged to actively engage with the latest advancements, stay updated on cutting-edge techniques, and actively participate in academic and research communities. By doing so, ML professionals can cultivate a deeper understanding of AI and contribute to the advancement of the field (Carlens, 2023).\n\nInterdisciplinary collaboration has emerged as a key factor in driving ML progress. Collaborating with experts from diverse fields such as computer science, mathematics, and psychology allows ML professionals to adopt a holistic approach to problem-solving. By leveraging insights and techniques from different disciplines, they can unlock new possibilities in the realm of artificial intelligence (Carlens, 2023).\n\nIn the world of competitive machine learning, specific strategies have emerged as winning approaches. The utilization of key Python packages, such as the PyData stack, has become a common denominator among successful competitors. Deep learning, with PyTorch emerging as a dominant framework, continues to drive advancements in ML (Carlens, 2023).\n\nTransformers have emerged as a powerful tool, particularly in natural language processing (NLP) tasks, challenging the traditional convolutional neural networks (CNNs). This trend highlights the potential of transformers to revolutionize not only NLP but also computer vision tasks (Carlens, 2023).\n\nIn 2022, CNNs continued to dominate as the most popular neural net architecture in computer vision competitions (Carlens, 2023), appearing in the top solutions of prominent challenges such as the RSNA-MICCAI Brain Tumor Radiogenomic Classification, PetFinder.my - Pawpularity Contest, US men's and women's college basketball tournaments, and Lux AI Challenge.\n \n\n \nFig. (Carlens, 2023) \n\nEnsembles, whether implicit or explicit, remain popular and effective in achieving success. Whether through gradient boosting methods like LightGBM and XGBoost or explicit ensembles that aggregate results from multiple models, ensembles play a pivotal role in achieving high-performance models (Carlens, 2023).\n\n6.2 The Role of Hardware and Infrastructure: \n\nExamining the computer and hardware infrastructure used by winners provides valuable insights into the resources employed in groundbreaking solutions. GPUs have been a crucial component, significantly enhancing the performance of gradient boosted trees and deep neural networks (Carlens, 2023). Winners often had access to GPUs through employer or university clusters. The NVIDIA A100 was the preferred high-end GPU among the competition winners. For instance, the H&M Personalized Fashion Recommendations competition and the Image Matching Challenge 2022 both experienced significant benefits by utilizing NVIDIA A100 GPUs for faster and more efficient model training.\n\n\n \nFig. (Carlens, 2023)\n\nComparing computing power is challenging, but dataset size and training time are useful indicators. Dataset sizes varied from small to massive (30 MB to 500 GB), and training times ranged from short with free resources to extensive with cloud compute. Kaggle competitions use a variety of compute resources depending on dataset size, complexity, model type, and desired accuracy. Examples of free compute competitions include NBME Score Clinical Patient Notes and Jigsaw Toxic Severity Rating, while examples of extensive cloud compute competitions include Mayo Clinic Strip AI and Landmark Recognition 2021.\n\n\n\nOverall, the progress and advancements in ML are driven by a combination of technological breakthroughs, emerging techniques, interdisciplinary collaboration, and the utilization of robust hardware infrastructure. By embracing these advancements and continuing to push the boundaries, the ML community can further drive innovation and contribute to the development of AI and ML (Carlens, 2023).\n\n","metadata":{}},{"cell_type":"markdown","source":"# 7. Conclusion
\nIn conclusion, competitive machine learning, particularly through Kaggle Competitions, has had a significant impact on the machine learning community, driving advancements in the field. Core tools and techniques such as Python, PyTorch, TensorFlow, ensemble methods, and feature engineering have played a crucial role in achieving competitive performance. Transformers have emerged as dominant models in NLP tasks and are challenging traditional CNNs in computer vision, demonstrating their transformative influence. \n\nKaggle Competitions have influenced various sectors, fostering innovation and practical solutions in computer vision, healthcare, finance, NLP, and more. However, there are challenges in terms of interpretability, ethics, and reproducibility of winning solutions in real-world scenarios. Nonetheless, Kaggle Competitions have promoted collaboration, knowledge sharing, and innovation in the machine learning community. It is important to responsibly leverage the insights gained from these competitions to drive impactful advancements in machine learning for the benefit of society.","metadata":{}},{"cell_type":"markdown","source":"\n# 8. References
\n\n\"Disclaimer: This essay incorporates information, reports, and summaries provided to the AI language model (OpenAI's GPT-3.5) as input. While efforts have been made to review and refine the generated content, it is important to acknowledge that the AI model's responses are based on patterns learned from the training data, including the provided input. Please consider this when evaluating the information presented, as it may not reflect personal expertise or firsthand knowledge.\"\n\nOpenAI. (2023). ChatGPT (Version 3.5, Version 4.0) [AI-generated response]. Retrieved July 5, 2023, from https://www.openai.com\n\nPrompts:\n\"Please provide competition name, a concise description and solution for a given problem in the form of bullet points. Identify the industry/sector related to the problem, research area, describe the solution approach and model used, specify the data split, testing and training techniques used in machine learning, mention the software and hardware resources utilized for solving the problem. Later, this information will be used to classify and cluster the data and draw meaningful inferences from it.\"\n\nMooney, P. (n.d.). Prompt: Describe what the ML community has learned over the past 2 years of working and experimenting with Kaggle Competitions. Retrieved from https://www.kaggle.com/paultimothymooney\n\nKaggle. (2021). Kaggle Competitions. Retrieved from https://www.kaggle.com/competitions \n\nCarlens, H. (2023). State of Competitive Machine Learning in 2022. ML Contests Research\n.\nKaggle. (2021). Feature Engineering. Retrieved from https://www.kaggle.com/learn/feature-engineering\n\nPavan Sanagapati. (2021). Ensemble Learning Techniques Tutorial. Retrieved from https://www.kaggle.com/code/pavansanagapati/ensemble-learning-techniques-tutorial\n\nBrownlee, J. (2021). Hyperparameter Optimization for Deep Learning Models. Retrieved from https://machinelearningmastery.com/hyperparameter-optimization-for-deep-learning-models/\n\nKrenn, M. (2022, October 3). Tweet. Retrieved from twitter.com/MarioKrenn6240/status/1577102743927652354 \n\nKenjee. (2023, May). \"Growth in AI/ML Paper Submissions.\" Retrieved from www.kaggle.com/code/kenjee/growth-in-ai-ml-paper-submissions\n\nKaggle Competitions. (n.d.). Retrieved from https://www.kaggle.com/competitions\nKaggle. (2023). 2023 Kaggle AI Report. Retrieved from https://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/400084\n\n\n\nML Contests. (2022). State of Competitive Machine Learning 2022 Edition. Retrieved from https://mlcontests.com/state-of-competitive-machine-learning-2022/#competitive-machine-learning-toolkit\n\nKrenn, Mario. Tweet. 2 July 2023, 9:15 a.m., twitter.com/MarioKrenn6240/status/1577102743927652354.\n\nKenjee. \"Growth in AI/ML Paper Submissions.\" n.d., www.kaggle.com/code/kenjee/growth-in-ai-ml-paper-submissions.\n","metadata":{}},{"cell_type":"markdown","source":"**Referenced Solutionss**\n\nDaishu. (2022, May 25). American Express - Default Prediction. Kaggle. Retrieved from https://www.kaggle.com/c/amex-default-prediction/discussion/348111\n\nDarek Kłeczek. (2021, August 11). Chaii - Hindi and Tamil Question Answering. Kaggle. Retrieved from https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/287923\n\nDieter. (2021, August 10). Google Landmark Recognition 2021. Kaggle. Retrieved from https://www.kaggle.com/c/landmark-recognition-2021/discussion/277098\n\nCurrypurin. (2022, February 1). NBME - Score Clinical Patient Notes. Kaggle. Retrieved from https://www.kaggle.com/c/nbme-score-clinical-patient-notes/discussion/323095\n\nFiras Baba. (2021, July 13). RSNA-MICCAI Brain Tumor Radiogenomic Classification - Medical Imaging. Kaggle. Retrieved from https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/281347\n\nGibA. (2021, September 23). PetFinder.my - Pawpularity Contest. Kaggle. Retrieved from https://www.kaggle.com/c/petfinder-pawpularity-score/discussion/301686\n\nGuanshuo Xu. (2021, November 8). Jigsaw Rate Severity of Toxic Comments. Kaggle. Retrieved from https://www.kaggle.com/c/jigsaw-toxic-severity-rating/discussion/306274\n\nHarm Buisman. (2022, April 12). Kore 2022. Kaggle. Retrieved from https://www.kaggle.com/c/kore-2022/discussion/340035\n\nKhyeh. (2022, July 6). Mayo Clinic - STRIP AI - Healthcare. Kaggle. Retrieved from https://www.kaggle.com/c/mayo-clinic-strip-ai/discussion/357892 \n\nDarek Kłeczek. (2021, August 11). Chaii - Hindi and Tamil Question Answering. Kaggle. Retrieved from https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/287923\n\nMrKMaKr. (2022, November 1). OTTO - Multi-Objective Recommender System. Kaggle. Retrieved from https://www.kaggle.com/c/otto-recommender-system/discussion/384022\n\nNyanP. (2021, June 28). Optiver Realized Volatility Prediction. Kaggle. Retrieved from https://www.kaggle.com/c/optiver-realized-volatility-prediction/discussion/274970\n\nPh0921. (2021, June 10). MLB Player Digital Engagement Forecasting. Kaggle. Retrieved from https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/discussion/274255\n\nRyan Anderson. (2023, January 25). Lux AI Challenge. Kaggle. Retrieved from https://www.kaggle.com/c/lux-ai-season-2/discussion/407982\n\nSenkin13. (2022, February 7). H&M Personalized Fashion Recommendations. Kaggle. Retrieved from https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/324070\n\nHarm Buisman. (2022, April 12). Kore 2022. Kaggle. Retrieved from https://www.kaggle.com/c/kore-2022/discussion/340035\n\nWht1996. (2021, December 14). Feedback Prize - Evaluating Student Writing. Kaggle. Retrieved from https://www.kaggle.com/c/feedback-prize-2021/discussion/313177\n\nYam Peleg. (2021, November 2). G-Research Crypto Forecasting. Kaggle. Retrieved from https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286676\n\nYirun Zhang. (2021, February 15). Jane Street Market Prediction. Kaggle. Retrieved from https://www.kaggle.com/c/jane-street-market-prediction/discussion/224348\n\nYumuneko. (2022, April 4). Image Matching Challenge 2022. Kaggle. Retrieved from https://www.kaggle.com/c/image-matching-challenge-2022/discussion/329131\n\nShin Koshi. (2022, Mar 12). March Machine Learning Mania 2022 Men's. Retrieved from https://www.kaggle.com/c/mens-march-mania-2022/discussion/318779\n\nKoki Ando. (2022, Mar 13). March Machine Learning Mania 2022 Women's. Retrieved from https://www.kaggle.com/c/womens-march-mania-2022/discussion/317817\n\n","metadata":{}},{"cell_type":"markdown","source":"**References for Kaggle Competitions**","metadata":{}},{"cell_type":"markdown","source":"\n1. Kaggle Competition: (2022, February 7). H&M Personalized Fashion Recommendations. Retrieved from https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations\n2. Kaggle Competition: (2022, April 4). Image Matching Challenge 2022. Retrieved from https://www.kaggle.com/c/image-matching-challenge-2022\n3. Kaggle Competition: (2021, August 10). Google Landmark Recognition 2021. Retrieved from https://www.kaggle.com/c/landmark-recognition-2021\nII. Healthcare Competitions:\n1. Kaggle Competition: (2022, February 1). NBME - Score Clinical Patient Notes. Retrieved from https://www.kaggle.com/c/nbme-score-clinical-patient-notes\n2. Kaggle Competition: (2021, July 13). RSNA-MICCAI Brain Tumor Radiogenomic Classification - Medical Imaging. Retrieved from https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification\n3. Kaggle Competition: (2022, July 6). Mayo Clinic - STRIP AI - Healthcare. Retrieved from https://www.kaggle.com/c/mayo-clinic-strip-ai\nIII. Quantitative Trading Competitions:\n1. Kaggle Competition: (2021, February 15). Jane Street Market Prediction. Retrieved from https://www.kaggle.com/c/jane-street-market-prediction/overview\n2. Kaggle Competition: (2021, November 2). G-Research Crypto Forecasting. Retrieved from https://www.kaggle.com/c/g-research-crypto-forecasting\n3. Kaggle Competition: (2021, June 28). Optiver Realized Volatility Prediction. Retrieved from https://www.kaggle.com/c/optiver-realized-volatility-prediction\nIV. Finance Competitions:\n1. Kaggle Competition: (2022, May 25). American Express - Default Prediction. Retrieved from https://www.kaggle.com/c/amex-default-prediction\nV. Natural Language Processing (NLP) Competitions:\n1. Kaggle Competition: (2021, November 8). Jigsaw Rate Severity of Toxic Comments. Retrieved from https://www.kaggle.com/c/jigsaw-toxic-severity-rating\n2. Kaggle Competition: (2021, August 11). Chaii - Hindi and Tamil Question Answering. Retrieved from https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering\nVI. Education Competitions:\n1. Kaggle Competition: (2021, December 14). Feedback Prize - Evaluating Student Writing. Retrieved from https://www.kaggle.com/c/feedback-prize-2021\nVII. Retail and Shopping Competitions:\n1. Kaggle Competition: (2021, September 23). PetFinder.my - Pawpularity Contest. Retrieved from https://www.kaggle.com/c/petfinder-pawpularity-score\n2. Kaggle Competition: (2022, November 1). OTTO - Multi-Objective Recommender System. Retrieved from https://www.kaggle.com/c/otto-recommender-system\nVIII. Sports Competitions:\n1. Kaggle Competition: (2022, March 12-13). March Machine Learning Mania 2022 (Men's and Women's). Retrieved from https://www.kaggle.com/c/mens-march-mania-2022 and https://www.kaggle.com/c/womens-march-mania-2022\n2. Kaggle Competition: (2021, June 10). MLB Player Digital Engagement Forecasting. Retrieved from https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting\nIX. Video Games and Simulation Competitions:\n1. Kaggle Competition: (2022, April 12). Kore 2022. Retrieved from https://www.kaggle.com/c/kore-2022\n2. Kaggle Competition: (2023, January 25). Lux AI Challenge. Retrieved from https://www.kaggle.com/competitions/lux-ai-season-2","metadata":{}},{"cell_type":"markdown","source":"**Code**","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:45:41.749122Z","iopub.execute_input":"2023-07-16T11:45:41.749580Z","iopub.status.idle":"2023-07-16T11:45:41.761149Z","shell.execute_reply.started":"2023-07-16T11:45:41.749547Z","shell.execute_reply":"2023-07-16T11:45:41.759724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport datetime as dt\nimport matplotlib.pyplot as plt\n\n#retrieve Kaggle writeups data\ncomps = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv')\n\n# extract competitions in the past two years\n#the starting date is the date this essay was first drafted\nstart_date = dt.datetime(2021, 6, 4) \nend_date = dt.datetime(2023, 6, 4)\ncomps['Competition Launch Date'] = pd.to_datetime(comps['Competition Launch Date'])\nfiltered = comps[(comps['Competition Launch Date'] >= start_date) & (comps['Competition Launch Date'] <= end_date)]\nprint(\"Number of competitions with writeups in the past 2 years: \", len(filtered['Title of Competition'].unique()))\n#Number of competitions with writeups in the past 2 years: 57","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:45:49.328343Z","iopub.execute_input":"2023-07-16T11:45:49.328796Z","iopub.status.idle":"2023-07-16T11:45:49.809376Z","shell.execute_reply.started":"2023-07-16T11:45:49.328761Z","shell.execute_reply":"2023-07-16T11:45:49.807773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get the last ten competitions\nlatest_comps = filtered['Title of Competition'].unique() # [-10:]\nprint(\"\\nThe latest 57 competitions:\")\n#number the outputs\nfor i, name in enumerate(latest_comps):\n print(f\"{i+1}) {name}\")","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:46:20.020214Z","iopub.execute_input":"2023-07-16T11:46:20.020662Z","iopub.status.idle":"2023-07-16T11:46:20.029283Z","shell.execute_reply.started":"2023-07-16T11:46:20.020621Z","shell.execute_reply":"2023-07-16T11:46:20.027562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\ncompetitions = [\n \"Mayo Clinic Strip AI\",\n \"Rsna Miccai Brain Tumor Radiogenomic\",\n \"Landmark Recognition 2021\",\n \"Amex Default Prediction\",\n \"H&M Personalized Fashion Recommendations\",\n \"Otto Recommender System\",\n \"Mlb Player Digital Engagement Forecasting\",\n \"G-Research Crypto Forecasting\",\n \"Image Matching Challenge 2022\",\n \"Optiver Realized Volatility Prediction\",\n \"Petfinder Pawpularity Score\",\n \"Feedback Prize 2021\",\n \"Jigsaw Toxic Severity Rating\",\n \"Nbme Score Clinical Patient Notes\",\n \"Women's March Madness 2022\",\n \"Men's March Madness 2022\",\n \"Chaii Hindi and Tamil Question Answering\"\n]\n\ndataset_sizes = [\n 395.36,\n 136.85,\n 105.52,\n 50.31,\n 34.56,\n 11.89,\n 8.35,\n 3.12,\n 2.65,\n 2.73,\n 1.04,\n 0.10442,\n 0.03537,\n 0.03573,\n 0.02501,\n 0.23826,\n 0.0318\n]\n\n# Sort competitions and dataset sizes in descending order\ncompetitions_sorted = [comp for _, comp in sorted(zip(dataset_sizes, competitions), reverse=True)]\ndataset_sizes_sorted = sorted(dataset_sizes, reverse=False)\n\n# Convert dataset sizes to GB for better readability\ndataset_sizes_gb = [size for size in dataset_sizes_sorted]\n\n# Define color map\ncmap = plt.get_cmap('Blues')\ncolors = cmap(np.linspace(0.2, 0.5, len(competitions)))\n\n# Plotting the bar chart\nplt.figure(figsize=(6, 6))\nbars = plt.barh(competitions_sorted, dataset_sizes_gb, color=colors)\n\n# Adding data labels to the bars\nfor i, bar in enumerate(bars):\n plt.text(bar.get_width() + 0.1, bar.get_y() + bar.get_height() / 2, f\"{dataset_sizes_gb[i]} GB\", va='center')\n\n# Setting labels and title\nplt.xlabel('Dataset Size (GB)')\nplt.ylabel('Competition')\nplt.title('Dataset Sizes for Competitions')\n\n# Adjusting layout and adding gridlines\nplt.grid(axis='x', linestyle='--')\n\n# Removing spines\nplt.gca().spines['top'].set_visible(False)\nplt.gca().spines['right'].set_visible(False)\n\n# Tight layout\nplt.tight_layout()\n\n# Displaying the bar chart\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:50:32.375597Z","iopub.execute_input":"2023-07-16T11:50:32.376056Z","iopub.status.idle":"2023-07-16T11:50:32.956441Z","shell.execute_reply.started":"2023-07-16T11:50:32.376022Z","shell.execute_reply":"2023-07-16T11:50:32.954885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nsub=pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:39:49.776508Z","iopub.execute_input":"2023-07-16T11:39:49.777029Z","iopub.status.idle":"2023-07-16T11:39:49.832486Z","shell.execute_reply.started":"2023-07-16T11:39:49.776989Z","shell.execute_reply":"2023-07-16T11:39:49.831017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vals = [\"'Kaggle Competitions'\", \n \"'https://www.kaggle.com/juanricardop/2023-ai-report-kaggle-competitions'\",\n \"'https://www.kaggle.com/code/adriandip/approximating-global-minima-with-very-small-datase/comments#2342747'\",\n \"'https://www.kaggle.com/code/pandoraalias/unleashing-the-competitive-spirit-k-competitions/comments#2345492'\",\n \"'https://www.kaggle.com/code/pinkythomas/the-state-of-time-series-in-kaggle-competitions/comments#2334778'\"]\nsub.value = vals\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T11:39:54.218008Z","iopub.execute_input":"2023-07-16T11:39:54.218471Z","iopub.status.idle":"2023-07-16T11:39:54.232215Z","shell.execute_reply.started":"2023-07-16T11:39:54.218436Z","shell.execute_reply":"2023-07-16T11:39:54.230866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-07-07T10:10:50.027944Z","iopub.execute_input":"2023-07-07T10:10:50.028566Z","iopub.status.idle":"2023-07-07T10:10:50.035947Z","shell.execute_reply.started":"2023-07-07T10:10:50.028534Z","shell.execute_reply":"2023-07-07T10:10:50.034635Z"},"trusted":true},"execution_count":null,"outputs":[]}]}