{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## **Relevance of Kaggle competitions for real life**\n#### Author: [Andrija Miličević](https://www.linkedin.com/in/andrijamilicevic) ([CroDoc](https://www.kaggle.com/crodoc))\n\n## Introduction\n\n[Kaggle](https://www.kaggle.com/) is an online data science community platform that hosts data science competitions. Kaggle was [founded in April 2010](https://en.wikipedia.org/wiki/Kaggle). Since then, the number of registered users per year has been growing. More than 3 million users registered in 2022.","metadata":{}},{"cell_type":"code","source":"# imports\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\nimport matplotlib.ticker as ticker\n\nfrom IPython.display import Image","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-14T11:04:56.277788Z","iopub.execute_input":"2023-07-14T11:04:56.278201Z","iopub.status.idle":"2023-07-14T11:04:56.314859Z","shell.execute_reply.started":"2023-07-14T11:04:56.278170Z","shell.execute_reply":"2023-07-14T11:04:56.313437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"../input/meta-kaggle/Users.csv\")\nusers_registered_year = pd.to_datetime(df['RegisterDate']).map(lambda d: d.year)\n\ntotal_users_per_year = users_registered_year.groupby(users_registered_year).count()\n\nfor i in range(1, len(total_users_per_year)):\n    total_users_per_year.iloc[i] += total_users_per_year.iloc[i-1]\n\nusers_registered_year = users_registered_year[users_registered_year < 2023]\nregistered_users_per_year = users_registered_year.groupby(users_registered_year).count()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-14T11:04:56.316738Z","iopub.execute_input":"2023-07-14T11:04:56.317095Z","iopub.status.idle":"2023-07-14T11:06:24.458726Z","shell.execute_reply.started":"2023-07-14T11:04:56.317067Z","shell.execute_reply":"2023-07-14T11:06:24.457662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_, ax = plt.subplots()\nax.yaxis.set_major_formatter(ticker.FuncFormatter(lambda y, _: '{0:g}'.format(y/1e6)))\nplt.title('Registered users per year')\nax.set_xlabel('Year')\nax.set_ylabel('Number of registerd users in millions')\nplt.plot(registered_users_per_year)\npass","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-14T11:06:24.459991Z","iopub.execute_input":"2023-07-14T11:06:24.460380Z","iopub.status.idle":"2023-07-14T11:06:24.816419Z","shell.execute_reply.started":"2023-07-14T11:06:24.460351Z","shell.execute_reply":"2023-07-14T11:06:24.815237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On June 23, 2023, Kaggle [announced](https://www.linkedin.com/posts/kaggle_weve-officially-passed-14m-community-members-activity-7078063761030291456-39tW) that the platform had over 14 million \nregistered users. This makes Kaggle the largest data science community out there.","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots()\nax.yaxis.set_major_formatter(ticker.FuncFormatter(lambda y, _: '{0:g}'.format(y/1e6)))\nplt.title('Total registered users')\nax.set_xlabel('Year')\nax.set_ylabel('Number of registerd users in millions')\nplt.plot(total_users_per_year)\npass","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-14T11:06:24.818991Z","iopub.execute_input":"2023-07-14T11:06:24.819354Z","iopub.status.idle":"2023-07-14T11:06:25.166725Z","shell.execute_reply.started":"2023-07-14T11:06:24.819323Z","shell.execute_reply":"2023-07-14T11:06:25.165818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Throughout the years, there were always questions regarding the relevance of experiences and learnings acquired through Kaggle competitions to scientific and real-world data science problems. The biggest concern was that solutions and approaches from Kaggle competitions are not applicable outside of the Kaggle platform (e.g., the winning solutions can be [huge ensembles of large models](https://www.kaggle.com/code/cdeotte/3rd-place-solution-lb-0-4337-cv-0-4420/) and a lot of time can be spent on solving edge cases and squeezing out the data to gain a small boost in performance - [fourth decimal boosts can even win competitions](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/leaderboard)). While some approaches to solving problems are unique to Kaggle (e.g. spending a lot of time for a potential small gain), there is a lot the scientific and industry communities can learn.\n\nThis essay aims to give some examples and insights that knowledge, solutions, and results from Kaggle competitions are indeed valuable for scientific and real-world problems.","metadata":{}},{"cell_type":"markdown","source":"## Validation\n\nOne of the most popular strategies for training a robust model that generalizes well is:\n1. split a dataset into a train, test and validation set\n2. train a model on the train set while using the validation set to select hyperparameters\n3. test the final model on the test set","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/kaggle-ai-report-images/standard_validation.png\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-14T11:06:25.168117Z","iopub.execute_input":"2023-07-14T11:06:25.168633Z","iopub.status.idle":"2023-07-14T11:06:25.200226Z","shell.execute_reply.started":"2023-07-14T11:06:25.168604Z","shell.execute_reply":"2023-07-14T11:06:25.199236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In practice, too often the test set is used more than once and therefore it becomes an extra validation set instead. It might not be obvious to inexperienced data scientists, but one can overfit on the validation set although the model never saw these examples during training. By selecting hyperparameters based on the validation set, knowledge about the validation set is given to the model. Treating the test set as an extra validation by doing too many validations on the test set (instead of only once) can lead to inflated scores of a model. Since labels for the test set are often available, it can be tempting to even peek into the test set and check where the model is making mistakes. The image above shows that although the test set should only be used for scoring the final model, the accessibility of the test set allows unlimited scoring/validation.\n\nTo prevent this, Kaggle has a slightly different approach. Instead of having a single test set, Kaggle introduced the idea of using two test sets instead. The test set is split into a public and private test set. Competitors don't have access to the test set labels. In most competitions, competitors don't have access to the test sets at all. Competitors submit models in an environment similar to jupyter notebooks which are then evaluated on the Kaggle platform. During the competitions, competitors see only the score on the public test set (public leaderboard). The number of submissions is usually limited to five times per day. Before the competition ends, competitors can select a few submissions (usually two) which are used for the final ranking (private leaderboard). The ranking is based on the score on the private test set. This way there is no peeking into the private test set. This helps to get a better sense of the generalizability of a model.","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/kaggle-ai-report-images/kaggle_validation.png\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-14T11:06:25.201486Z","iopub.execute_input":"2023-07-14T11:06:25.202398Z","iopub.status.idle":"2023-07-14T11:06:25.217575Z","shell.execute_reply.started":"2023-07-14T11:06:25.202354Z","shell.execute_reply":"2023-07-14T11:06:25.216600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On Kaggle, there is a popular term \"shake-up\" which stands for overfitting to the public test set and then dropping on the private leaderboard after the competition ends and the scores on the private test set are revealed. Every experienced Kagglers knows it is possible to overfit on a dataset that was never available during training and that was never even seen by a Kaggler.","metadata":{}},{"cell_type":"markdown","source":"## Benchmarking\n\nMost scientific papers report results on a dataset that is not a standard benchmark. This can make it harder to reproduce or verify the results. Although this is changing and although there is more emphasis on reproducibility, it still seems to be an unsolved issue in the scientific community. Kaggle competitions usually attract hundreds of even thousands of teams that compete in a single competition. By default, the winning solutions are benchmarked and their quality is compared to the solutions of other teams.\n\nIn May 2021, [American Journal of Neuroradiology](https://www.ajnr.org/) published a paper titled \"[MRI-Based Deep-Learning Method for Determining Glioma MGMT Promoter Methylation Status](https://www.ajnr.org/content/42/5/845)\". The paper presented an approach to determining the presence of methylated MGMT promoter for gliomas using MRI images. The presence of methylated MGMT promoter has a high impact on the prognosis and treatment of such tumors. The authors claimed that their model had an accuracy of 94.73% and a mean area under the curve of 0.93. [American Journal of Neuroradiology](https://www.ajnr.org/) was [ranked 10th](https://www.scimagojr.com/journalrank.php?category=2741&order=h&ord=desc&year=2021) in the category of Radiology, Nuclear Medicine and Imaging out of 329 indexed journals. The journal was a high-impact one and the results sounded amazing.\n\nIn July 2021, Kaggle hosted a competition called \"[RSNA-MICCAI Brain Tumor Radiogenomic Classification](https://www.kaggle.com/competitions/rsna-miccai-brain-tumor-radiogenomic-classification)\" whose goal was to predict MGMT promoter methylation from MRI images of brain tumors. During the competition, a kaggler [Théo Viel](https://www.kaggle.com/theoviel) created a [discussion post](https://www.kaggle.com/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/279820) that indicated there is a high chance that the signal for predicting the task is very low and that a significant \"shake-up\" could be expected. At the end of the competition, the same kaggler [posted something very interesting](https://www.kaggle.com/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/279823). The model that scored the worst on the private test set had the best discriminative power (if it predicted the opposite status).\n\nThe [highest score](https://www.kaggle.com/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/leaderboard) on the private test set was 0.62174 ([area under the ROC curve](https://www.kaggle.com/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/overview/evaluation)). This was very different from the results reported in the previously mentioned paper.\n\nIn January 2023, an [erratum](https://www.ajnr.org/content/44/1/E1) was published in the same journal. The erratum stated that there was an error in the cross-validation procedure and that the train set was used as the test set which caused the inflated results. The final corrected score was 0.5802 which should that their approach was not miraculous and that it lacked proper validation and craved for benchmarking.\n\nKaggle has been recognized as the go-to place for benchmarking. The most prestigious computer science conferences are hosting competitions on Kaggle and are reporting the results. [CVPR](https://cvpr2023.thecvf.com/) launched [Image Matching Challenge 2023](https://www.kaggle.com/competitions/image-matching-challenge-2023/overview/cvpr-2023-workshop) and [NeurIPS](https://nips.cc/) is launching [the first Machine Unlearning Challenge](https://ai.googleblog.com/2023/06/announcing-first-machine-unlearning.html).","metadata":{}},{"cell_type":"markdown","source":"## How to train a model for production\n\n[Darek Kłeczek](https://www.kaggle.com/thedrcat), a Kaggle grandmaster, [analyzed winning solutions](https://www.kaggle.com/code/thedrcat/how-to-win-a-kaggle-competition) of Kaggle competitions. The most popular method mentioned was ensembling. Ensembling means using multiple models to solve a problem. The idea is similar to the wisdom of the crowd. As [wikipedia states](https://en.wikipedia.org/wiki/Wisdom_of_the_crowd): \"The wisdom of the crowd is the collective opinion of a diverse independent group of individuals rather than that of a single expert. A large group's aggregated answers to questions involving quantity estimation, general world knowledge, and spatial reasoning has generally been found to be as good as, but often superior to, the answer given by any of the individuals within the group\". The same usually holds true for problems on Kaggle. An ensemble of diverse models in most cases outperforms single model solutions.\n\nIn a lot of Kaggle competitions, the solution submitted in an environment similar to jupyter notebooks gets 9 hours of inference time on GPU(s). Because of this, winning solutions can be an ensemble of large models that are impractical for production usage. This was often one of the common arguments why kaggling is not that important for real-world data science or machine learning.\n\nIn May 2022, Kaggle hosted a competition ([Feedback Prize - Predicting Effective Arguments](https://www.kaggle.com/competitions/feedback-prize-effectiveness/overview/description)) that changed a lot. Along with the standard 9 hours of GPU inference time leaderboard ranking, Kaggle hosted a parallel [efficiency track](https://www.kaggle.com/competitions/feedback-prize-effectiveness/overview/efficiency-prize-evaluation). The efficiency ranking was based both on the score of the solution and inference time according to a [predefined formula](https://www.kaggle.com/competitions/feedback-prize-effectiveness/overview/efficiency-prize-evaluation). The team \"Now You See Me\" won [7th place](https://www.kaggle.com/competitions/feedback-prize-effectiveness/leaderboard) on the standard leaderboard with [nine 5-fold models](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347891) and a submission time of around 9 hours. On the [efficiency leaderboard](https://www.kaggle.com/code/ryanholbrook/feedback-prize-efficiency-leaderboard) they got [2nd place](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347488) with a 8-minute single-model submission that had similar score (<0.3% difference).","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/competitions/feedback-prize-effectiveness/leaderboard\n# https://www.kaggle.com/code/ryanholbrook/feedback-prize-efficiency-leaderboard\n# https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347891\n# https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347488\n\ncolors = ['lightcoral', 'dodgerblue']\n\nplt.figure(figsize=(12, 3))\nplt.subplot(1, 3, 1)\nx = ['ensemble', 'single\\nmodel']\ny = [45, 1]\nplt.bar(x, y, color=colors)\nplt.ylabel('models')\nplt.title('Number of models')\n\nplt.subplot(1, 3, 2)\nx = ['ensemble', 'single\\nmodel']\ny = [9, 474/(60*60)]\nplt.bar(x, y, color=colors)\nplt.ylabel('hours')\nplt.title('Inference time')\n\nplt.subplot(1, 3, 3)\nx = ['ensemble', 'single\\nmodel']\ny = [0.56153, 0.56289]\nplt.bar(x, y, color=colors)\nplt.ylabel('score')\nplt.title('Score')\nplt.ylim(top=1, bottom=0)\npass","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-14T11:06:25.219611Z","iopub.execute_input":"2023-07-14T11:06:25.220240Z","iopub.status.idle":"2023-07-14T11:06:25.798617Z","shell.execute_reply.started":"2023-07-14T11:06:25.220208Z","shell.execute_reply":"2023-07-14T11:06:25.797436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[This solution shows](https://wandb.ai/darek/fbck/reports/How-To-Build-an-Efficient-NLP-Model--VmlldzoyNTE5MDEx?accessToken=pmm41mpdkxif0lsbm927tfxj947to4gbd0nrgjjq9rdoq1c4jr3kruf993ys5kpg) that knowledge distillation and pseudolabels is one of the ways how to turn an inference-heavy Kaggle competition solution into a production-friendly model. Doing a stable k-fold cross-validation to select hyperparameters and then training on all data is one of the things that can boost the score of a single production model. [Another solution shows](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/418420) that in some cases it is beneficial to train a single model with a high learning rate and do a [model soup](https://proceedings.mlr.press/v162/wortsman22a.html). Yet, [another solution shows](https://www.kaggle.com/competitions/hpa-single-cell-image-classification/discussion/239071) that improving the dataset can get great results even with smaller models.\n\nKaggle started hosting more and more competitions that had \"production grade\" constraints. [CPU-only competitions](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/overview/code-requirements), [competitions that accept only small and fast tflite models](https://www.kaggle.com/competitions/asl-fingerspelling/overview/evaluation) and [competitions that require fast model that can work in real-time](https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/overview/description) are a few of them. These competitions are hosted with the sole purpose of putting the winning solutions straight into production.","metadata":{}},{"cell_type":"markdown","source":"## Conclusion\n\nKaggle is becoming more and more popular. The number of registered users per year is higher than ever. Scientific problems are hosted on the platform. Production models are being developed. Kaggle shows us that there are best practices for a lot of problems and each of them usually has unique opportunities for improvement. These opportunities are found by doing a lot of experiments and testing a lot of ideas. The best way to get better at discovering these opportunities is by solving a lot of different problems and by reading what was discovered by others. Kaggle is the best place for this. Kaggle is more relevant than ever.","metadata":{}},{"cell_type":"markdown","source":"## Submission","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ndf = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\ndf.iloc[0,1] = 'Kaggle Competitions'\ndf.iloc[1,1] = 'https://www.kaggle.com/crodoc/relevance-of-kaggle-competitions-for-real-life'\ndf.iloc[2,1] = 'https://www.kaggle.com/code/naturalray/the-change-in-kaggle-competitions/comments#2338083'\ndf.iloc[3,1] = 'https://www.kaggle.com/code/arunodhayan/kaggle-ai-report-birdclef-series-2019-2023/comments#2338213'\ndf.iloc[4,1] = 'https://www.kaggle.com/code/thedrcat/how-to-win-a-kaggle-competition/comments#2338257'\ndf.to_csv('submission.csv', index=False)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-14T11:06:25.800200Z","iopub.execute_input":"2023-07-14T11:06:25.800614Z","iopub.status.idle":"2023-07-14T11:06:26.720924Z","shell.execute_reply.started":"2023-07-14T11:06:25.800582Z","shell.execute_reply":"2023-07-14T11:06:26.719697Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]}]}