{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Examination of Impact: Kaggle Competitions over the Last Two Years

\n

Kaggle and Crowdsourcing ML

\nSince its conception in 2010, Kaggle has emerged as the largest data science community in the world. With over 14 million users presently [1], Kaggle has rapidly evolved into a hub for data science and machine learning enthusiasts, fostering discovery, knowledge dissemination, and collaboration through its famed data science competitions. These competitions lie at the heart of Kaggle’s appeal, offering participants lucrative prizes, prestige, and the chance to have a hand in solving significant real-world problems.\n\nThe impact of the competitions extends far and wide, with winning solutions often being integrated directly or in part by host organizations [2]. Also, by facilitating the development of novel methods and extensive datasets, these competitions have been instrumental in advancing ML over the past decade. These datasets, numbering over 230,000 to date [1], have led to significant innovations, even outside of the competitions themselves.\n\nAdditionally, the open sourcing of code and the sharing of work on public forums have had immense educational value. Competition winners often reference or draw upon ideas from previous submissions in their solutions. Besides, innumerable blog posts and articles are written on websites discussing the competitions and results. Numerous scientific papers are published about the solutions themselves or citing winning solutions and datasets. Upon examining just one open research platform, arxiv.org [3], we found over 300 papers that directly reference Kaggle.\n\nKaggle competitions also cultivate a culture of healthy competition and collaboration among participants, which is central to creating an inclusive and transparent community. Numerous forums exist on Kaggle where users publicly discuss opinions and share approaches and ideas. Remarkably, hosts have even taken tips from the discussion forums and incorporated them directly into production before the competition was even completed [2].\n\nThroughout its history, the Kaggle platform has consistently demonstrated that crowdsourcing ML works wonders. In this essay, we narrow our focus and delve into an in-depth analysis of Kaggle competitions over the past two years. We will uncover the remarkable impacts that these competitions have had, highlighting their effects on the community and their contributions to advancing ML research and applications.\n\nThe rest of this essay is organized into the following sections:\n\n1. [Trends and Emerging Technologies](#trends)\n2. [Not For Profit and Social Good](#nonprofit)\n3. [Sustainability and Preservation](#sustainability)\n4. [Healthcare](#healthcare)\n5. [Developing Solutions](#sols)\n6. [Recruiting AI Experts, Data Scientists](#recruitment)\n7. [Collaboration and Knowledge Sharing](#collabs)\n8. [Conclusion](#conclusion)\n\n

Trends and Emerging Technologies

\nIn the fast-paced world of data science, staying at the forefront of trends and technologies is vital for driving innovation. By analyzing competitions, examining winning solutions, and studying the strategies employed by top participants over the past two years, we aim to provide valuable insights into the prevailing trends and emerging technologies that are propelling the competition landscape forward.\n
Machines
\nWe were interested in finding out more about the kinds of machines used by participants. After surveying the solution writeups and code, we have generated the graph below depicting GPUs vs the number of mentions:","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport plotly.express as px\nimport re \n\ndf=pd.read_csv(\"/kaggle/input/all-20212023-competitions/kaggle_writeups_20230510 3.csv\")\na=df['Writeup'].str.count('v100',flags=re.IGNORECASE).sum()\nb=df['Writeup'].str.count('rtx3090',flags=re.IGNORECASE).sum()+df['Writeup'].str.count('rtx 3090',flags=re.IGNORECASE).sum()+df['Writeup'].str.count(' 3090 ',flags=re.IGNORECASE).sum()\nc=df['Writeup'].str.count('a100',flags=re.IGNORECASE).sum()\nd=df['Writeup'].str.count('p100',flags=re.IGNORECASE).sum()\ne=df['Writeup'].str.count('a6000',flags=re.IGNORECASE).sum()\n\ndata = [['V100', a],['3090', b],['P100', d],['A100', c],['A6000', e]]\n\n# Create the pandas DataFrame\ndf1 = pd.DataFrame(data, columns=['GPU', 'No. of Mentions'])\nfig = px.bar(df1, x=\"No. of Mentions\", y=\"GPU\",color='GPU', orientation='h')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:21:31.344884Z","iopub.execute_input":"2023-07-16T21:21:31.345361Z","iopub.status.idle":"2023-07-16T21:21:32.162508Z","shell.execute_reply.started":"2023-07-16T21:21:31.345329Z","shell.execute_reply":"2023-07-16T21:21:32.161033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One thing is clear: Nvidia dominates. Although we did find mention of AMD machines, the huge majority mentioned having used Nvidia GPUs.\n\nSurveying writeups and codes of winning solutions, we found that the Nvidia RTX 3090, despite not being as costly as the V100 or A6000 models, was one of the most popular machines. It is comforting for participants to note that more affordable GPUs can often be enough to produce a winning solution, and you don't have to spend thousands of dollars on compute resources. \n\nAdding to this, it is also encouraging that we noticed many cases where first-time participants and even participants whose day jobs had nothing to do with ML or data science emerged victorious. You don't need to be a seasoned, full-time practitioner to bag the gold. Anyone has the potential to succeed here.\n
Deep Learning Frameworks
\nWhen it comes to the deep learning frameworks used by winners, PyTorch was the clear winner. Examining the writeups and the codes of solutions, we found that PyTorch was much more popular than TensorFlow among participants. What surprised us the most is that in the Help Protect the Great Barrier Reef competition, which offered an additional $50,000 in prizes to participants using Tensorflow, the top 3 solutions all used PyTorch. Our findings are in line with the 2022 Kaggle Machine Learning & Data Science Survey [4], which shows PyTorch increasing in popularity in contrast to TensorFlow’s more or less steady or declining rate.\n
Backbones
\nFor those seeking state-of-the-art (SOTA) backbones to tackle problems, Kaggle competitions can be a great place to look. \n\nWe first examined the computer vision challenges and present our findings: ","metadata":{}},{"cell_type":"code","source":"df=pd.read_csv(\"/kaggle/input/image-20212023-competitions/image-writeups.csv\")\na=df['Writeup'].str.count('Efficientnet',flags=re.IGNORECASE).sum()\nb=df['Writeup'].str.count('Transformer',flags=re.IGNORECASE).sum()+df['Writeup'].str.count(' vit ',flags=re.IGNORECASE).sum()+df['Writeup'].str.count(' swin ',flags=re.IGNORECASE).sum()\nc=df['Writeup'].str.count('resnet',flags=re.IGNORECASE).sum()\nd=df['Writeup'].str.count('resnext',flags=re.IGNORECASE).sum()\ne=df['Writeup'].str.count('nfnet',flags=re.IGNORECASE).sum()\nf=df['Writeup'].str.count('convnext',flags=re.IGNORECASE).sum()\ndata = [['Efficientnet', a], ['Transformer/ViT/Swin', b], ['ResNet', c],['ResNeXt', d],['ConvNEXT', f],['NFNet', e]]\n\n# Create the pandas DataFrame\ndf1 = pd.DataFrame(data, columns=['BackBone', 'No. of Mentions'])\nfig = px.bar(df1, x=\"No. of Mentions\", y=\"BackBone\",color='BackBone', orientation='h')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:21:46.706798Z","iopub.execute_input":"2023-07-16T21:21:46.707407Z","iopub.status.idle":"2023-07-16T21:21:47.086623Z","shell.execute_reply.started":"2023-07-16T21:21:46.707375Z","shell.execute_reply":"2023-07-16T21:21:47.085765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Upon surveying the writeups and codes of winning solutions, we found that the most popular backbones were the EfficientNets [5] and different variants of image and vision transformers. Notably, in the SIIM-FISABIO-RSNA COVID-19 Detection competition, the solutions of all 10 winners either relied solely on these models or worked by incorporating them into their ensembles.\n\nDespite only being introduced in 2019, EfficientNets have already become the most popular backbone in computer vision competitions. We observed this trend extending even into the audio-based challenges, where EfficientNets came out on top by a large margin. Kaggle contests often require optimizing performance with limited computational resources. As their name implies, EfficientNets are one of the most accurate yet efficient models available, which is clearly demonstrated by their popularity in these challenges.\n\nNot far behind are the image and vision transformer models. It is a testament to the power of transformers that their influence has transcended the boundaries of NLP to claim the position of runner-up here. Maybe Attention is All You Need after all. \n\nHaving been first extensively evaluated in the 2020 paper “An image is worth 16x16 words'' [6], their rapid rise in fame may be attributed to their ability to capture global context and long-range dependencies. Some of the most popular models among participants were the ViT [7] and Swin [8] transformer models. Apart from these renowned models, lesser known and more recent versions such as PoolFormer [9], CvT [10], BLIP-2 [11], SegFormer [12], CoaT [13], XCiT [14], and BEiT [15] were some alternatives.\n\nIn the case of NLP competitions, transformers are unrivaled. We found that all the winning solutions in these competitions relied primarily on transformers, especially pre-trained models from the Hugging Face model hub [16]. \n\nWhile a wide variety of models were used, the overwhelmingly popular choice were variants of Microsoft’s DeBERTa [17]. Despite having been introduced only in 2020, this model has quickly been adopted as SOTA in NLP competitions due to its advanced features and performance. Particularly in the NBME - Score Clinical Patient Notes competition, all 6 winning solutions relied purely on ensembles of various DeBERTa models.\n\nFacebook’s RoBERTa [18] was also found to be referenced quite often. Debuted in 2019, this model showcased several improvements and better performance as compared to the standard BERT transformer [19]. \n\nUpon examining the models used for tabular data-based challenges, we made the following discovery:","metadata":{}},{"cell_type":"code","source":"df=pd.read_csv(\"/kaggle/input/tabular20212023competitions/tabular_writeups.csv\")\na=df['Writeup'].str.count('lightgbm',flags=re.IGNORECASE).sum()+df['Writeup'].str.count('lgbm',flags=re.IGNORECASE).sum()\nb=df['Writeup'].str.count('neural net',flags=re.IGNORECASE).sum()+df['Writeup'].str.count('mlp',flags=re.IGNORECASE).sum()\ne=df['Writeup'].str.count('xgboost',flags=re.IGNORECASE).sum()\nf=df['Writeup'].str.count('catboost',flags=re.IGNORECASE).sum()\ndata = [['LightGBM/LGBM', a], ['Neural Net/MLP', b], ['XGBoost', e],['CatBoost', f]]\n\n# Create the pandas DataFrame\ndf1 = pd.DataFrame(data, columns=['Method', 'No. of Mentions'])\nfig = px.bar(df1, x=\"No. of Mentions\", y=\"Method\",color='Method', orientation='h')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:22:35.706096Z","iopub.execute_input":"2023-07-16T21:22:35.706999Z","iopub.status.idle":"2023-07-16T21:22:35.984227Z","shell.execute_reply.started":"2023-07-16T21:22:35.706957Z","shell.execute_reply":"2023-07-16T21:22:35.983044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Kaggle competitions have unequivocally established the superiority of gradient-boosting algorithms when it comes to tackling tabular data challenges. LightGBM [20], XGBoost [21], and CatBoost [22] were found to be used very often by winning solutions. \n\nAs evident from the graph above and from our analysis, we found that LightGBM showed significantly greater usage in recent competitions compared to XGBoost. Gaining popularity over time due to its fast training speed and efficiency in handling large datasets, LightGBM has clearly replaced XGBoost as a competition favorite.\n\nWhile most of the tabular data challenges were won using machine learning models and algorithms, deep learning models, including Neural Nets, LSTMs, 1D CNNs, and RNNs, were also used often. \n\nFurthermore, it was notable to observe that the influence of transformers has extended to these competitions as well. For example, in the IceCube - Neutrinos in Deep Ice challenge, all the winning solutions relied heavily on transformers, showcasing the remarkable and versatile capabilities of attention models.\n\n
Object Detection
\nWhen it comes to object detection, we noticed that, even almost a decade later, YOLO still dominates. Specifically, the YOLOV5 [23] and the recent YOLOx [24] models were the most popular by a large margin among winning solutions. Alternate models that we found mentioned but not as popular include Facebook’s Detectron [25], EfficientDet [26], and Faster R-CNN [27]. Microsoft’s MegaDetector [28], a specialized model developed to detect animals, people, and vehicles in camera trap images, was the most popular choice for the iWildCam 2021 and 2022 challenges due to its relevance.\n\n
Segmentation
\nIn the case of segmentation challenges, the U-Net [29] architecture and its variants, such as the UNet++ [30], appear unmatched. Despite being less popular, alternative models like Mask R-CNN [31], DeepLabv3+ [32], UPerNet [33], LinkNet [34], and even transformer models like the SegFormer [12] were still used and proved successful in winning challenges.\n\n
The Power of Ensembling
\nKaggle competitions consistently showcase the power of ensemble learning. In fact, ensembles were the single most common element that we observed among most of the winning solutions. We even saw ensembles as large as around 50 models, as in the case of the first prize winner of the Happywhale - Whale and Dolphin Identification competition. \n\nBut, this is not to say that single-model approaches never win contests. For instance, in the RSNA-MICCAI Brain Tumor Radiogenomic Classification competition, the first position was snagged by a simple yet clever solution that did not involve any ensembling.\n\nWhile simple weighted averaging was the most common ensembling technique used, we also found significant mention of the WBF/ Weighted Boxes Fusion [35] method. Specifically, for ensembling bounding box outputs in the case of object detection models, this seems to be the way to go. \n\n
Data Augmentation
\nKaggle competitions also serve to demonstrate the effectiveness of data augmentation during training as well as during testing. In the case of computer vision challenges, while standard augmentation methods such as flip, transpose, crop, scale, and rotate were extensively used to boost performance, more elaborate methods such as CutMix [36], MixUp [37], AugMix [38], and CLAHE were also common. Of significant note is the Hotel-ID to Combat Human Trafficking 2022 competition, where Kaggle Grandmaster David Austin bagged the first position by inventing a completely new augmentation method, BlendFlip. His novel approach showed better performance than other traditional methods in dealing with images containing masks or occlusions. \n\nWhen it came to NLP competitions, several solutions drew inspiration from computer vision challenges in their augmentation methods, incorporating techniques such as random cropping, progressive resizing, Cutout [39], and CutMix [36]. We also saw pure NLP approaches to augmentation such as masking, back translation, SentAugment [40], Augmented SBERT [41], and using pre-trained transformers [42].\n\n
Feature-Engineering
\nThe competitions involving tabular data provided undeniable evidence of the power of feature selection and feature engineering. After carefully examining the write-ups, it became apparent that many winning solutions outperformed their counterparts not due to the choice of models employed, but rather due to the clever techniques they used in engineering the provided features. \n\nKaggle competitions often involve complex and real-world problems where raw data may not directly lead to optimal model performance. By generating aggregated statistics, employing polynomial transformations, or making use of domain-specific insights combined with experimentation and iterative improvement, participants have demonstrated the significant impact that well-crafted and relevant features can have on performance.\n\n
Semi-supervised Learning
\nKaggle competitions from this time frame clearly highlighted the power of semi-supervised learning methods, specifically pseudo-labelling. Pseudo-labelling is the process of adding confidently predicted test data to the training data, allowing the model to learn from both the labeled and pseudo-labeled data. As an example, in the HuBMAP + HPA Hacking the Human Body challenge, pseudo-labelling was the key to winning. All the winning solutions pseudo-labeled the provided dataset extensively, some even twice.\n

Not For Profit and Social Good

\nOver the past two years, Kaggle has held several competitions that promoted humanitarian efforts and nonprofit causes. In this section, we aim to provide a well-rounded view of the learnings and advancements within the ML community through these challenges, showcasing not only technical insights but also the broader societal impact and ethical dimensions of ML applications. \n\nWe must begin by discussing the Feedback Prize series of competitions. These were NLP challenges designed by The Learning Agency Lab with the goal of helping struggling students improve their writing skills. The first competition in this series was the Evaluating Student Writing challenge , where participants were tasked with developing models for the automatic segmentation and classification of elements in essays. Impressively, in a case study following the competition [43], hosts stated that the winning solution’s accuracy rate mirrored the consensus reached by human readers when assessing the annotated data. Furthermore, they verified that they were working to ensure that the winning solutions could be integrated into open-source automated writing assessment tools, allowing educational organizations worldwide to make use of them.\n\nThe next two challenges in this series were the Predicting Effective Arguments and the English Language Learning competitions . What set these competitions apart was their dual-prize track, which included both a traditional accuracy track and a computational efficiency track. The second track was introduced to encourage the development of environmentally friendly solutions suitable for low-resource environments, in contrast to large computational models with high carbon footprints.\n\nIn the Predicting Effective Arguments challenge, it was the same team that bagged the first position in both tracks. It was really intriguing because, although their efficiency track model’s score was virtually the same as their accuracy track model’s, its runtime was only 6 minutes as opposed to almost 6 hours for the latter, illustrating without doubt that it is possible for efficient models to perform on par with large models.\n\nThe competition hosts confirmed that the models generated within these competitions too would be fully open-sourced and offered to school systems, non-profit organizations, and researchers worldwide. They also expressed their gratitude to Kagglers by stating that they could never have achieved this without the hard work and dedication of the Kaggle community [44].\n\nApart from the Feedback Prize series of competitions, The Learning Agency Lab, in collaboration with Learning Equality, had also organized the Curriculum Recommendations challenge during this timeframe. Here, participants were required to develop models capable of matching educational content to specific topics in a curriculum. This challenge too, consisted of two tracks: accuracy and computational efficiency. \n\nThe hosts verified that the models generated within this competition would be made fully open-source and offered to Ministries of Education and UN agencies. The outcome of this competition holds great potential to facilitate quality learning worldwide, particularly for refugees.\nIn quoting Nestori, a former refugee teacher at the Kakuma Refugee Camp: “It's a great milestone towards fulfilling the dreams of thousands of refugee students in accessing quality education.” [45]\n\nIn this regard, the hosts affirmed that Learning Equality would be collaborating closely with the UN High Commissioner for Refugees. They aim to build upon the competition's winning algorithms to prototype an automated alignment feature within their curriculum alignment tool, Kolibri Studio [46]. This feature represents a significant stride forward in delivering the quality education that refugee students rightfully deserve.\n\nThe Fine-Grained Visual Categorization (FGVC) series of competitions stands out as another set of competitions with significant impact. 9 FGVC competitions have been conducted and completed on Kaggle over the past two years as a part of workshops at CVPR 2021 and 2022.\n\nOne of the major outcomes of these competitions has been the curation of extensive benchmark datasets. The Herbarium 2021 Half–Earth dataset [47] is definitely one of the most impressive, consisting of over 2 million images and nearly 65,000 taxa. The dataset for the Cassava Leaf Disease Classification contest, on the other hand, consisted of fewer images that were crowdsourced from farmers in Uganda taking photos of their gardens. Following the previous iteration of this challenge, researchers had open-sourced a new model for Cassava disease classification on TFHub. This model, dubbed CropNet [48], was widely used by the solutions in the 2021 iteration. \n\nIn this series, we also identified the Hotel-ID to Combat Human Trafficking challenges as being particularly significant. This is because aspects of the top solutions are integrated into an image search tool that is deployed at the National Center for Missing and Exploited Children [49], playing a vital role in combating human trafficking.\n\n

Sustainability and Preservation

\nIn considering competitions impacting sustainability and preservation, we first discuss the TensorFlow - Help Protect the Great Barrier Reef competition. This challenge was aimed at developing models for the real-time detection of crown-of-thorns starfish (COTS) that pose a threat to Australia’s Great Barrier Reef. In an article on the TensorFlow Blog [50], hosts pointed out that this competition provided them with invaluable insights in developing their own model. \n\n“We ran hundreds of experiments based on what we learned from the Kaggle submissions to determine which techniques in combination were most effective in increasing our model’s accuracy.”\n\n

-Megha Malpani, Google Product Manager and Ard Oerlemans, Google Software Engineer

\n\nAs a result, they were able to develop and open-source both 1080p and 720p COTS object detection models [51], even implementing an optimized inference pipeline for edge ML platforms.\n\nNext, we highlight the BirdCLEF series of challenges hosted by the Cornell Lab of Ornithology. The goal of these challenges is to develop models for the automatic acoustic identification of birds from soundscape recordings. The models generated within these challenges are to be used by researchers for the detection and classification of bird sounds, so that they can better monitor the status and trends of biodiversity in important ecosystems. Since its launch in 2014, it has become one of the largest bird sound recognition competitions in terms of dataset size and species diversity.\n\nMost impressively, extending the successful approaches and datasets from this series of challenges, Cornell researchers have made a remarkable advancement with the introduction of BirdNET [52,53]. This is a citizen science research platform based on artificial neural networks that is capable of identifying approximately 3,000 of the world’s most common species. BirdNET has greatly fostered public engagement, enabled collaborations, and advanced the field of machine learning in audio analysis.\n\nThe competitions results were also published in the proceedings of the CLEF 2021 conference [54]. Notably, the inclusive nature of the conference allowed for the submission of solutions by all participants, regardless of their score or rank. This inclusive approach ensured that valuable contributions, even if not reflected in the leaderboard, would be accessible to a wide audience.\n\n

Healthcare

\nKaggle competitions have consistently provided a platform for data scientists, machine learning practitioners, and healthcare professionals to collaborate and tackle critical challenges in the healthcare industry. These competitions have offered the ML community valuable opportunities for real-world application, access to diverse and unique healthcare data, and collaboration with domain experts.\n\nThe Single Cell Classification contest hosted by Human Protein Atlas was one of the most impactful competitions in this sector. The challenge involved a weakly supervised multi-label classification problem. Numerous strategies, including the use of external datasets and new approaches to multi-label classification, were explored here. In fact, the winning solution was bagged by developing a novel network dubbed the Fair Cell Activation Network (FCAN) which was based on Puzzle-CAM [55].\n\nAccording to the analysis conducted post-competition [56], the hosts stated that the winning models far outperformed their previous effort at multi-label classification of protein localization patterns by ~20%. Although they had previously developed a machine learning model called Loc-CAT [57] for this problem, the winning solutions in the competition demonstrated significantly superior performance. They also stated that they intend for the results of the competition to be open-sourced as pre-trained networks for a wide range of biological applications. To this extent, they have built a model zoo [58] to host the source code and trained models from select teams. They even hope that these will replace the standard ImageNet pre-trained models in the context of transfer learning for applications involving biological images.\n\nNext, we consider The Radiological Society of North America (RSNA). Over the course of the last two years, they have actively participated on the platform by hosting or co-hosting a total of four competitions. Out of these, it is impressive to note that the Brain Tumor Radiogenomic Classification challenge resulted in the release of the BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification [59], and one of the most extensive datasets of multi-parametric magnetic resonance imaging (mpMRI) scans to date. \n\nSimilarly, the VinBigData Chest X-ray Abnormalities Detection competition led to the curation of one of the most detailed X-ray datasets [60], consisting of 18,000 images that were labeled by experienced radiologists and also carefully de-identified to protect patient privacy.\n\nFinally, we spotlight the NBME - Score Clinical Patient Notes competition. Here, participants were tasked with developing an automated method to map clinical concepts from an exam rubric to various ways in which these concepts are expressed in clinical patient notes. The winning solutions from the challenge were released under an open-source license, enabling medical educators to adapt them and use them in their own institutions. In an article on the NBME website [61], hosts appreciated the hundreds of forum discussions and publicly shared code, the substantial body of work produced by the competition on state-of-the-art in automated clinical text scoring. They also expressed their gratitude to the Kaggle community:\n\n“We are grateful to all participants for their hard work and dedication in making the first collective effort towards solving an important problem in medical education assessment.”\n\n

-Victoria Yaneva, Senior Data Scientist at NBME

\n\n

Developing Solutions

\nOver the last two years, Kaggle competitions have played a crucial role in fostering research and development efforts towards the creation of SOTA solutions that can be deployed across various sectors, including commercial, social enterprise, and even government domains.\n\nNotably, in an interview [62], Kaggle grandmaster Jean-Francois Puget revealed that data scientists at Nvidia use Kaggle challenges as a learning platform and even incorporate models and approaches from competitions into their own internal projects. \n\nReferring to their experience participating in the BirdCLEF series, he said, “Several of us entered this competition, and we did quite well on average, and now we are reusing similar models internally for audio classification projects at Nvidia. We definitely took what we learned at Kaggle, and our models improved whatever Nvidia was doing before.”\n\nApart from audio classification, he also mentioned that they had been similarly leveraging the experience and knowledge gained from a series of NLP competitions on Kaggle for another one of their projects, enhancing the existing solutions.\n\nIn considering specific competitions, we first discuss the Smartphone Decimeter challenges organized by Google in 2021 and 2022 . These competitions were designed to propel research in smartphone GNSS positioning accuracy. \n\nThe top rank in both iterations of the competition was achieved by the same contestant, who happens to be a researcher specializing in the field of GNSS. It was interesting to note that his solution in both installments of the challenge did not involve machine learning at all. Instead, his winning approach was based on using factor graph optimization (FGO) with accumulated delta range (ADR) measurements from the smartphone. The results of this competition demonstrated how domain knowledge can occasionally surpass the capabilities of even the most exceptional machine learning models.\n\nThe winning solutions from both editions of the competition were showcased at the prestigious ION GNSS+ conferences of 2021 and 2022 [63,64], where they were presented to international leaders in the fields of GNSS and allied positioning, navigation, and timing domains.\n\nYet another series of competitions that deserve to be highlighted are the 2021 Google Landmark Recognition and Retrieval competitions. The first position in both of the competitions was bagged by the same participant, by developing an architecture combining DOLG [65] methods of orthogonal fusion and hybrid Swin Transformers. \n\nAdditionally, another impressive Google competition conducted during this time frame was the 2022 Image Matching Challenge . We noticed that all the winning solutions involved the use of transformers, specifically LoFTR [66], or SuperGlue [67] for feature matching. The second-place winner even published a paper about his novel single model, the SpanFormer [68].\n\nNext, we highlight the Show US the Data challenge, which aimed to use NLP for automating the identification of references to scientific data in publications. The solutions devised by successful participants in this challenge could play a pivotal role in bringing down a critical barrier to the access and use of public data. The winning solutions received well-deserved recognition at the 2021 Show US the Data Conference. In front of a diverse audience comprising federal agencies, publishers, and researchers, the winning models and algorithms were presented and also showcased through initial tool demonstrations, in the form of data usage scorecards [69].\n\nLastly, we present the U.S. Patent Phrase to Phrase Matching competition. As the name suggests, this competition required participants to create models capable of matching key phrases in patent documents. In the 2022 Patent Public Advisory Committee Meeting [70], Mr. Jerry Ma, director of emerging technology at the United States Patent and Trademark Office (USPTO), expressed his awe at how this competition had surpassed all their expectations in terms of both engagement and quantitative outcomes. He further commended the remarkable sense of collaboration exhibited by the Kaggle community throughout the duration of the challenge.\n\nMost impressively, marking a momentous achievement, he stated that this competition gave rise to the establishment of the AI and Emerging Technology Partnership at the USPTO, with the goal of capitalizing on the unprecedented level of engagement witnessed. Furthermore, he confirmed that they even dispatched a delegation to the PatentSemTech conference at SIGIR 2022 in Madrid to showcase the remarkable work stemming from the competition.\n\n

Recruiting AI Experts, Data Scientists

\nKaggle competitions provide a platform for recruiters to assess the technical abilities, problem-solving skills, and collaboration capabilities of data scientists. It serves as a talent marketplace where individuals can showcase their expertise and attract the attention of potential employers.\n\nDuring the past two years, particularly, the Realized Volatility Prediction by Optiver and the Default Prediction challenge by American Express have stood out as competitions where hosts explicitly expressed their interest and actively encouraged participants to submit their resumes or apply to open positions post-competition.\n

Collaboration and Knowledge Sharing

\nKaggle competitions contribute significantly to knowledge sharing, collaboration, and engagement in the ML community. Through forums, code sharing, collaborative environments, and interactions with hosts and experts, participants have ample opportunities to exchange ideas, and collectively push the boundaries of data science and ML.\n\nTo cite a specific example, we consider the March Machine Learning Mania series of competitions. The series encompassed 3 distinct competitions over the past two years: March Machine Learning Mania 2022 - Women's , March Machine Learning Mania 2022 - Men's , and March Machine Learning Mania 2023 . Upon examining the winning solutions, we made an interesting observation. The winning solutions in 2 out of these 3 competitions were created by drawing upon the March Madness winning solution from 5 years ago [71], demonstrating clearly the incredible capacity of the Kaggle community to learn from one another.\n\nWe also see this trend continue in the popular use of ArcFace [72] in computer vision competitions, especially those involving metric learning. A variation of the ArcFace loss, incorporating dynamic margins [73], was introduced by the winners of third place in the Google Landmark Recognition 2020 competition [74]. We observed that over the past two years, participants have regularly borrowed this variant of ArcFace from their fellow Kagglers to boost their own models' performance. \n\nLastly, we consider the MLB Player Digital Engagement Forecasting challenge. This competition stood out due to its distinctive feature—a supplementary prize track known as the Explainability Prize, in addition to the conventional accuracy prize. This encouraged participants to make their solutions and methodologies clearly documented and reproducible, fostering a culture of sharing insights and advancing collective understanding. \n\n

Conclusion

\nIn summary, Kaggle contests have had a significant influence on the ML community and on driving progress in the field over the past two years. These contests have promoted teamwork and collaboration in addition to fostering invention and producing useful solutions with real-world applications. As Helen Keller once said, \"Alone, we can do so little; together, we can do so much.\" The ripple effects of these challenges reach far beyond the individual participants and teams, leaving a lasting imprint on the advancement of ML.","metadata":{}},{"cell_type":"markdown","source":"
References:
\n\nWe made use of AI tools: ChatGPT and Quillbot for running grammar checks, and also for occasionally rectifying phrasing.\n\n1. https://www.kaggle.com/code/carlmcbrideellis/kaggle-in-numbers\n\n2. https://www.kaggle.com/discussions/general/243348\n\n3. https://arxiv.org/\n\n4. https://www.kaggle.com/c/kaggle-survey-2022/data\n\n5. Mingxing Tan, Quoc V. Le. \"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.\" 2020. arXiv:1905.11946 [cs.LG].\n\n6. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.\" 2021. arXiv:2010.11929 [cs.CV].\n\n7. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo. \"Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.\" 2021. arXiv:2103.14030 [cs.CV].\n\n8. Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan. \"MetaFormer Is Actually What You Need for Vision.\" 2022. arXiv:2111.11418 [cs.CV].\n\n9. Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, Lei Zhang. \"CvT: Introducing Convolutions to Vision Transformers.\" 2021. arXiv:2103.15808 [cs.CV].\n\n10. Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. \"BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.\" 2023. arXiv:2301.12597 [cs.CV].\n\n11. Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Ping Luo. \"SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers.\" 2021. arXiv:2105.15203 [cs.CV].\n\n12. Weijian Xu, Yifan Xu, Tyler Chang, Zhuowen Tu. \"Co-Scale Conv-Attentional Image Transformers.\" 2021. arXiv:2104.06399 [cs.CV].\n\n13. Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, Hervé Jegou. \"XCiT: Cross-Covariance Image Transformers.\" 2021. arXiv:2106.09681 [cs.CV].\n\n14. Hangbo Bao, Li Dong, Songhao Piao, Furu Wei. \"BEiT: BERT Pre-Training of Image Transformers.\" 2022. arXiv:2106.08254 [cs.CV].\n\n15. Hangbo Bao, Li Dong, Songhao Piao, Furu Wei. \"BEiT: BERT Pre-Training of Image Transformers.\" 2022. arXiv:2106.08254 [cs.CV].\n\n16. https://huggingface.co/models\n\n17. Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen. \"DeBERTa: Decoding-enhanced BERT with Disentangled Attention.\" 2021. arXiv:2006.03654 [cs.CL].\n\n18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. \"RoBERTa: A Robustly Optimized BERT Pretraining Approach.\" 2019. arXiv:1907.11692 [cs.CL].\n\n19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. \"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.\" 2019. arXiv:1810.04805 [cs.CL].\n\n20. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu. \"LightGBM: A Highly Efficient Gradient Boosting Decision Tree.\" Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17), 2017, pp. 3149-3157.\n\n21. Tianqi Chen, Carlos Guestrin. \"XGBoost.\" Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016. DOI: 10.1145/2939672.2939785.\n\n22. Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, Andrey Gulin. \"CatBoost: unbiased boosting with categorical features.\" 2019. arXiv:1706.09516 [cs.LG].\n\n23. https://github.com/ultralytics/yolov5\n\n24. Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, Jian Sun. \"YOLOX: Exceeding YOLO Series in 2021.\" 2021. arXiv:2107.08430 [cs.CV].\n\n25. https://github.com/facebookresearch/Detectron\n\n26. Mingxing Tan, Ruoming Pang, Quoc V. Le. \"EfficientDet: Scalable and Efficient Object Detection.\" 2020. arXiv:1911.09070 [cs.CV].\n\n27. Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun. \"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.\" 2016. arXiv:1506.01497 [cs.CV].\n\n28. https://github.com/microsoft/CameraTraps/blob/main/megadetector.md\n\n29. Olaf Ronneberger, Philipp Fischer, Thomas Brox. \"U-Net: Convolutional Networks for Biomedical Image Segmentation.\" 2015. arXiv:1505.04597 [cs.CV].\n\n30. Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, Jianming Liang. \"UNet++: A Nested U-Net Architecture for Medical Image Segmentation.\" 2018. arXiv:1807.10165 [cs.CV].\n\n31. Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick. \"Mask R-CNN.\" 2018. arXiv:1703.06870 [cs.CV].\n\n32. Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam. \"Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.\" 2018. arXiv:1802.02611 [cs.CV].\n\n33. Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, Jian Sun. \"Unified Perceptual Parsing for Scene Understanding.\" 2018. arXiv:1807.10221 [cs.CV].\n\n34. Abhishek Chaurasia, Eugenio Culurciello. \"LinkNet: Exploiting encoder representations for efficient semantic segmentation.\" In 2017 IEEE Visual Communications and Image Processing (VCIP). IEEE, 2017. DOI: 10.1109/vcip.2017.8305148.\n\n35. Roman Solovyev, Weimin Wang, Tatiana Gabruseva. \"Weighted boxes fusion: Ensembling boxes from different object detection models.\" Image and Vision Computing, vol. 107, p. 104117, Mar. 2021. DOI: 10.1016/j.imavis.2021.104117.\n\n36. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, Youngjoon Yoo. \"CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features.\" arXiv preprint arXiv:1905.04899, 2019.\n\n37. Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz. \"mixup: Beyond Empirical Risk Minimization.\" arXiv preprint arXiv:1710.09412, 2018.\n\n38. Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, Balaji Lakshminarayanan. \"AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty.\" arXiv preprint arXiv:1912.02781, 2020.\n\n39. DeVries, T., & Taylor, G. W. (2017). Improved Regularization of Convolutional Neural Networks with Cutout. arXiv preprint arXiv:1708.04552.\n\n40. Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, Alexis Conneau. \"Self-training Improves Pre-training for Natural Language Understanding.\" arXiv preprint arXiv:2010.02194, 2020.\n\n41. Nandan Thakur, Nils Reimers, Johannes Daxenberger, Iryna Gurevych. \"Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks.\" arXiv preprint arXiv:2010.08240, 2021.\n\n42. Varun Kumar, Ashutosh Choudhary, Eunah Cho. \"Data Augmentation using Pre-trained Transformer Models.\" arXiv preprint arXiv:2003.02245, 2021.\n\n43. https://www.the-learning-agency-lab.com/the-feedback-prize-case-study/\n\n44. https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369289\n\n45. https://www.kaggle.com/competitions/learning-equality-curriculum-recommendations/discussion/395082\n\n46. https://www.kaggle.com/competitions/learning-equality-curriculum-recommendations/discussion/372362\n\n47. https://www.kaggle.com/competitions/herbarium-2021-fgvc8/data\n\n48. https://www.tensorflow.org/hub/tutorials/cropnet_cassava\n\n49. https://www.kaggle.com/competitions/hotel-id-to-combat-human-trafficking-2022-fgvc9/discussion/316799\n\n50. https://blog.tensorflow.org/2022/05/Kaggle-Great-Barrier-Reef-ML.html\n\n51. https://colab.research.google.com/github/tensorflow/models/blob/master/official/projects/cots_detector/crown_of_thorns_starfish_detection_pipeline.ipynb\n\n52. Stefan Kahl, Connor M. Wood, Maximilian Eibl, Holger Klinck, BirdNET: A deep learning solution for avian diversity monitoring,Ecological Informatics, Volume 61, 2021, 101236, ISSN 1574-9541\n\n53. https://birdnet.cornell.edu/\n\n54. https://ceur-ws.org/Vol-2936/\n\n55. S. Jo and I.-J. Yu. \"Puzzle-CAM: Improved Localization Via Matching Partial And Full Features.\" In Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), September 2021. [Online]. Available: https://doi.org/10.1109/icip42928.2021.9506058\n\n56. Ouyang, W., Winsnes, C.F., Hjelmare, M. et al. Analysis of the Human Protein Atlas Image Classification competition. Nat Methods 16, 1254–1261 (2019). https://doi.org/10.1038/s41592-019-0658-6\n\n57. Sullivan DP, Winsnes CF, Åkesson L, Hjelmare M, Wiking M, Schutten R, Campbell L, Leifsson H, Rhodes S, Nordgren A, Smith K, Revaz B, Finnbogason B, Szantner A, Lundberg E. Deep learning is combined with massive-scale citizen science to improve large-scale image classification. Nat Biotechnol. 2018 Oct;36(9):820-828. doi: 10.1038/nbt.4225. Epub 2018 Aug 20. PMID: 30125267.\n\n58. https://modelzoo.cellprofiling.org/#/\n\n59. U. Baid et al. \"The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification.\" arXiv preprint arXiv:2107.02314, 2021. [Online]. Available: https://arxiv.org/abs/2107.02314\n\n60. Nguyen, H.Q., Lam, K., Le, L.T. et al. VinDr-CXR: An open dataset of chest X-rays with radiologist’s annotations. Sci Data 9, 429 (2022). https://doi.org/10.1038/s41597-022-01498-w\n\n61. https://www.nbme.org/news/six-teams-recognized-nlp-advances-nbmes-patient-note-scoring-competition\n\n62. https://www.nvidia.com/en-us/on-demand/session/gtcfall22-a41132/\n\n63. https://www.ion.org/publications/browse.cfm?proceedingsID=155\n\n64. https://www.ion.org/publications/browse.cfm?proceedingsID=159\n\n65. M. Yang et al. \"DOLG: Single-Stage Image Retrieval with Deep Orthogonal Fusion of Local and Global Features.\" arXiv preprint arXiv:2108.02927, 2021. [Online]. Available: https://arxiv.org/abs/2108.02927\n\n66. J. Sun et al. \"LoFTR: Detector-Free Local Feature Matching with Transformers.\" arXiv preprint arXiv:2104.00680, 2021. [Online]. Available: https://arxiv.org/abs/2104.00680\n\n67. P.-E. Sarlin et al. \"SuperGlue: Learning Feature Matching with Graph Neural Networks.\" arXiv preprint arXiv:1911.11763, 2020. [Online]. Available: https://arxiv.org/abs/1911.11763\n\n68. H. Chen et al. \"ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer.\" arXiv preprint arXiv:2208.14201, 2022. [Online]. Available: https://arxiv.org/abs/2208.14201\n\n69. https://www.kaggle.com/competitions/coleridgeinitiative-show-us-the-data/discussion/243730\n\n70. https://www.uspto.gov/sites/default/files/documents/PPAC-Transcript-20220809.pdf\n\n71. https://www.kaggle.com/competitions/womens-machine-learning-competition-2018/discussion/53597\n\n72. J. Deng et al., \"{{ArcFace}: Additive Angular Margin Loss for Deep Face Recognition},\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 5962-5979, Oct. 2022, doi: 10.1109/TPAMI.2021.3087709.\n\n73. Q. Ha, B. Liu, F. Liu, and P. Liao, \"Google Landmark Recognition 2020 Competition Third Place Solution,\" arXiv:2010.05350 [cs.CV], 2020.\n\n74. https://www.kaggle.com/competitions/landmark-recognition-2020/overview","metadata":{}},{"cell_type":"markdown","source":"
Note:
\nThe dataset of examined competitions and writeups is under: \n\n/kaggle/input/all-20212023-competitions/kaggle_writeups_20230510 3.csv","metadata":{}},{"cell_type":"markdown","source":"
Links to specifically mentioned solution write-ups:
\n\n1. https://www.kaggle.com/competitions/tensorflow-great-barrier-reef/discussion/307878\n2. https://www.kaggle.com/competitions/tensorflow-great-barrier-reef/discussion/307760\n3. https://www.kaggle.com/competitions/tensorflow-great-barrier-reef/discussion/307707\n4. https://www.kaggle.com/competitions/siim-covid19-detection/discussion/263658\n5. https://www.kaggle.com/competitions/siim-covid19-detection/discussion/263674\n6. https://www.kaggle.com/competitions/siim-covid19-detection/discussion/263654\n7. https://www.kaggle.com/competitions/siim-covid19-detection/discussion/264243\n8. https://www.kaggle.com/competitions/siim-covid19-detection/discussion/263945\n9. https://www.kaggle.com/competitions/siim-covid19-detection/discussion/264858\n10. https://www.kaggle.com/competitions/siim-covid19-detection/discussion/263748\n11. https://www.kaggle.com/competitions/siim-covid19-detection/discussion/263734\n12. https://www.kaggle.com/competitions/siim-covid19-detection/discussion/265583\n13. https://www.kaggle.com/competitions/siim-covid19-detection/discussion/263683\n14. https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323095\n15. https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323085\n16. https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322832\n17. https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322799\n18. https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322875\n19. https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323237\n20. https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402976\n21. https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402882\n22. https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/discussion/402888\n23. https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/320192\n24. https://www.kaggle.com/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/281347\n25. https://www.kaggle.com/competitions/hotel-id-to-combat-human-trafficking-2022-fgvc9/discussion/328281\n26. https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/356201\n27. https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/354857\n28. https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/354683\n29. https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177\n30. https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347536\n31. https://www.kaggle.com/competitions/hpa-single-cell-image-classification/discussion/239001\n32. https://www.kaggle.com/competitions/google-smartphone-decimeter-challenge/discussion/262406\n33. https://www.kaggle.com/competitions/smartphone-decimeter-2022/discussion/341111\n34. https://www.kaggle.com/competitions/landmark-recognition-2021/discussion/277098\n35. https://www.kaggle.com/competitions/landmark-retrieval-2021/discussion/277099\n36. https://www.kaggle.com/competitions/image-matching-challenge-2022/discussion/329131\n37. https://www.kaggle.com/competitions/image-matching-challenge-2022/discussion/329317\n38. https://www.kaggle.com/competitions/image-matching-challenge-2022/discussion/329540\n39. https://www.kaggle.com/competitions/mens-march-mania-2022/discussion/317145\n40. https://www.kaggle.com/competitions/march-machine-learning-mania-2023/discussion/399553\n41. https://www.kaggle.com/competitions/womens-machine-learning-competition-2018/discussion/53597\n42. https://www.kaggle.com/competitions/landmark-recognition-2020/discussion/187757","metadata":{}}]}