{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Kaggle Competitions and the AI Community\n\n*Author: [Benjamin Senst](https://www.linkedin.com/in/benjaminsenst)*\n\nArtificial intelligence (AI) has shown impressive development over the past decade. Interest in AI is no longer only found at university centers, but also among the general public. Kaggle is a machine learning online community and offers competitions in the field of artificial intelligence. This article is a contribution to the Kaggle Competition [Kaggle AI report](https://www.kaggle.com/competitions/2023-kaggle-ai-report) and focuses on evaluating the solution descriptions of 310 Kaggle competitions between 2010 and 2023. Using natural language techniques Processing (NLP) you can see how the development of artificial intelligence over this period is reflected in the solutions to the Kaggle competitions.\n\nThis essay is based on several Kaggle datasets. When processing the data records, it should be noted that the size of some tables requires a lot of memory or that alternative methods (e.g. lazy loading with Dask) should be used.","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"import warnings \nwarnings.filterwarnings('ignore')\n\n# import libraries\n! pip install sketch --quiet\nimport sketch\nimport numpy as np\nimport pandas as pd\nimport dask.dataframe as ddf\nimport matplotlib.pyplot as plt","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-15T06:24:43.890979Z","iopub.execute_input":"2023-07-15T06:24:43.891910Z","iopub.status.idle":"2023-07-15T06:24:58.965429Z","shell.execute_reply.started":"2023-07-15T06:24:43.891875Z","shell.execute_reply":"2023-07-15T06:24:58.964321Z"},"_kg_hide-output":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# public dataset https://www.kaggle.com/datasets/kaggle/meta-kaggle\nusers = pd.read_csv(\"/kaggle/input/meta-kaggle/Users.csv\")\nuserregistrationperyear = users.RegisterDate.map(lambda x: x[-4:]).value_counts().sort_index()\n\nforummessages = pd.read_csv(\"/kaggle/input/meta-kaggle/ForumMessages.csv\")\nforummessagesperyear = forummessages.PostDate.map(lambda x: x.split()[0][-4:]).value_counts().sort_index()\nforumtopics = pd.read_csv(\"/kaggle/input/meta-kaggle/ForumTopics.csv\")\n\ntags = pd.read_csv(\"/kaggle/input/meta-kaggle/Tags.csv\")\n\ncompetitions = pd.read_csv(\"/kaggle/input/meta-kaggle/Competitions.csv\")\ncompetitionsperyear = competitions.EnabledDate.map(lambda x: x.split()[0][-4:]).value_counts().sort_index()[:-1]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-15T06:24:58.966910Z","iopub.execute_input":"2023-07-15T06:24:58.967374Z","iopub.status.idle":"2023-07-15T06:26:00.154468Z","shell.execute_reply.started":"2023-07-15T06:24:58.967328Z","shell.execute_reply":"2023-07-15T06:26:00.153365Z"},"_kg_hide-output":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# About Kaggle\n\nKaggle was founded in 2010. It is based in San Francisco, California. In 2017 it was acquired by Google. In June 2023, [Kaggle announced on Twitter](https://twitter.com/kaggle/status/1672297218382938115) that the community exceeded 14 million users. The Kaggle platform includes competitions, data sets, models, notebooks, discussion forums and learning areas.\n\nhttps://en.wikipedia.org/wiki/Kaggle\n\nWhile the user count increased exponentially, more recently the number of forum messages and competitions launched decreased. What the recent drop in launched competitions causes is unclear.","metadata":{}},{"cell_type":"code","source":"userregistrations_2010_to_2022 = list(users.RegisterDate.map(lambda x: x[-4:]).value_counts().sort_index().values)\n\nusercount_total = 0 \nusercount_per_year = []\n\nfor i, userregistrations in enumerate(userregistrations_2010_to_2022):\n    usercount_total += userregistrations\n    usercount_per_year.append(usercount_total)\n\nusercount = pd.DataFrame()\nusercount[\"count\"] = usercount_per_year\nusercount.index = range(2010,2024)\n\nfig, axes = plt.subplots(1, 3, figsize=(12, 2))\n\n# Plot on the first subplot\naxes[0].plot(usercount[:-1])\naxes[0].set_title('User Count Total')\naxes[0].tick_params(axis='x', rotation=90)\n\n# Plot on the second subplot\naxes[1].plot(forummessagesperyear[:-1])\naxes[1].set_title('Forum Messages per Year')\naxes[1].tick_params(axis='x', rotation=90)\n\n# Plot on the second subplot\naxes[2].plot(competitionsperyear)\naxes[2].set_title('Competition Launches per Year')\naxes[2].tick_params(axis='x', rotation=90)\n\n# Adjust the spacing between subplots\nplt.subplots_adjust(wspace=0.4)\n\n# Show the plot\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-15T06:26:00.155798Z","iopub.execute_input":"2023-07-15T06:26:00.156506Z","iopub.status.idle":"2023-07-15T06:26:06.476871Z","shell.execute_reply.started":"2023-07-15T06:26:00.156471Z","shell.execute_reply":"2023-07-15T06:26:06.475770Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Competition Tags: \"Tabular\", \"Beginner\" & \"Binary Classification\"\n\nCustom tags help capture the content of the competitions. The Meta-Kaggle record contains tags for the competitions. Most competitions are labeled as \"Beginner\", \"Tabular\" and \"Binary Classification\" which represents the character of Kaggle competitions, that in particular the large group of beginners find opportunities on the Kaggle platform to apply their skills to specific problems. ","metadata":{}},{"cell_type":"code","source":"from wordcloud import WordCloud\n\ntags_competitions_value_counts = tags[tags.CompetitionCount>0][[\"Name\", \"CompetitionCount\"]].sort_values(\"CompetitionCount\", ascending=False)[:]\n\nword_frq = dict([(el[\"Name\"], el[\"CompetitionCount\"]) for el in tags_competitions_value_counts.to_dict(orient=\"records\")])\nwordcloud = WordCloud(background_color='lightgrey',\n                      colormap='RdYlGn',relative_scaling=0.05).fit_words(word_frq)\n\nfig, axes = plt.subplots(1, 2, figsize=(12, 3))\n\n# todo add title\n\naxes[0].imshow(wordcloud)\naxes[0].axis(\"off\")\naxes[0].set_title(\"Most common Tags WordCloud\")\n\naxes[1].barh(y=tags_competitions_value_counts.Name[:10], width=tags_competitions_value_counts.CompetitionCount[:10])\naxes[1].invert_yaxis()\naxes[1].set_title(\"Most common Tags Barplot\")\n\nplt.subplots_adjust(wspace=0.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:26:06.478762Z","iopub.execute_input":"2023-07-15T06:26:06.479199Z","iopub.status.idle":"2023-07-15T06:26:07.028165Z","shell.execute_reply.started":"2023-07-15T06:26:06.479160Z","shell.execute_reply":"2023-07-15T06:26:07.027348Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Tags of the Competition Subset Align with All Competition Tags","metadata":{}},{"cell_type":"code","source":"# private writeup with keywords dataset\nkeywords = pd.read_json(\"/kaggle/input/kaggle-writeups-scored-keywords/kaggle.json\")\n\n# public ai report competition dataset https://www.kaggle.com/competitions/2023-kaggle-ai-report\ncompetition_date = keywords[[\"Title of Competition\", \"Competition Launch Date\"]].drop_duplicates()\ncompetition_date[\"Competition Launch Date\"] = competition_date[\"Competition Launch Date\"].map(lambda x: x.split()[0][-4:])\n\n# create keywords per competition dataframe\nkeywords_writeup = []\ntitles = []\n\nfor title in keywords[\"Title of Competition\"].unique():\n    titles.append(title)\n    keywords_writeup.append([\" \".join(pd.DataFrame(scores).sequence.values) for scores in keywords[keywords[\"Title of Competition\"]==title].scores])\n\nkeywords_df = pd.DataFrame()\nkeywords_df[\"keywords\"] = keywords_writeup\nkeywords_df[\"Title of Competition\"] = titles\n\n# private ngrams dataset\nngrams = pd.read_json(\"/kaggle/input/kaggle-competition-with-scored-ngrams/kaggle_competition_with_scored_ngrams.json\")\n\n# private llm-generated dataset\nwith open(\"/kaggle/input/sketch-output-kaggle-competitions/sketch_output_kaggle_competitions.txt\") as infile:\n    text = infile.read()\n    \ntext = text.split(\"\\n\")\ntitles = [title for title in text[::3]]\ncategories = [category for category in text[1::3]]\ntechniques = [technique for technique in text[2::3]]\n\nllm_df = pd.DataFrame()\nllm_df[\"titles\"] = titles[:-1]\nllm_df[\"categories\"] = categories\nllm_df[\"techniques\"] = techniques\nllm_df.to_json(\"llm_df.json\")\n\n# create dataset for each collection containing data from prior three datasets\ndf = pd.DataFrame()\ndf[\"Title of Competition\"] = titles[:-1]\ndf[\"categories\"] = categories\ndf[\"techniques\"] = techniques\ndf = pd.merge(df, ngrams, on=\"Title of Competition\", how=\"left\")\ndf = pd.merge(df, keywords_df, on=\"Title of Competition\", how=\"left\")\ndf = pd.merge(df, competition_date, on=\"Title of Competition\", how=\"left\")","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:26:07.029411Z","iopub.execute_input":"2023-07-15T06:26:07.030536Z","iopub.status.idle":"2023-07-15T06:26:12.415353Z","shell.execute_reply.started":"2023-07-15T06:26:07.030503Z","shell.execute_reply":"2023-07-15T06:26:12.414376Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Both the Meta-Kaggle dataset and the dataset provided for the Kaggle AI Report contain information about competitions. The latter contains writeups for the competitions. However, the number of competitions recorded differs significantly. The two data sets also differ in the tags distribution. Not all competitions in the kaggle ai report dataset have corresponding tags in the meta kaggle dataset. Especially the competitions of the early 2010s lack tags. With a large total number of competitions, however, the small number of competitions without tagging has little impact on further analysis.","metadata":{}},{"cell_type":"code","source":"# competition tags from meta kaggle dataset collection\ncompetitiontags = pd.read_csv(\"/kaggle/input/meta-kaggle/CompetitionTags.csv\")\n\nprint(\"Kaggle AI Report Competitions without Tags\")\nprint(df[~df[\"Title of Competition\"].isin(competitiontags[competitiontags.CompetitionId.isin(competitions[competitions.Title.isin(df[\"Title of Competition\"])].Id)]\\\n    .drop(columns=\"Id\")\\\n    .rename(columns={\"TagId\":\"Id\"})\\\n    .merge(tags, on=\"Id\")\\\n    .drop(columns=\"Id\")\\\n    .rename(columns={\"CompetitionId\":\"Id\"})\\\n    .merge(competitions, on=\"Id\")\\\n    [['Title']][\"Title\"].drop_duplicates())][\"Competition Launch Date\"].value_counts().sort_index()\n)","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:26:12.417928Z","iopub.execute_input":"2023-07-15T06:26:12.418234Z","iopub.status.idle":"2023-07-15T06:26:12.460876Z","shell.execute_reply.started":"2023-07-15T06:26:12.418209Z","shell.execute_reply":"2023-07-15T06:26:12.459699Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Kaggle AI Report Dataset represents only a small subset of all Kaggle competitions available in the Meta Kaggle Dataset. To evaluate the topics of the writeups of the competitions in the Kaggle AI Report Dataset it is of interest whether these selected competitions represent a balanced sample of all competitions. If this is the case conclusion can be drawn from the subset for all kaggle competitions. The distribution of tags is similar between these two competition datasets but the tags \"nlp\", \"biology\" and \"healthcare\" are not used among the competitions inside the Kaggle AI Report.","metadata":{}},{"cell_type":"code","source":"outsidecompetitiontags = competitiontags\\\n.drop(columns=\"Id\")\\\n.rename(columns={\"TagId\":\"Id\"})\\\n.merge(tags, on=\"Id\")\\\n.drop(columns=\"Id\")\\\n.rename(columns={\"CompetitionId\":\"Id\"})\\\n.merge(competitions, on=\"Id\")\\\n[['Name', 'FullPath', 'Title']].Name.value_counts()[:30]\noutsidecompetitiontags = round(outsidecompetitiontags/len(outsidecompetitiontags),1)\n\ninsidecompetitiontags = competitiontags[competitiontags.CompetitionId.isin(competitions[competitions.Title.isin(df[\"Title of Competition\"])].Id)]\\\n.drop(columns=\"Id\")\\\n.rename(columns={\"TagId\":\"Id\"})\\\n.merge(tags, on=\"Id\")\\\n.drop(columns=\"Id\")\\\n.rename(columns={\"CompetitionId\":\"Id\"})\\\n.merge(competitions, on=\"Id\")\\\n[['Name', 'FullPath', 'Title']].Name.value_counts()[:30]\ninsidecompetitiontags = round(insidecompetitiontags/len(insidecompetitiontags),1)\n\nfig, axes = plt.subplots(1, 2, figsize=(12, 3))\n\naxes[0].set_title(f\"Number of Competitions (total = {len(competitions)})\")\naxes[0].pie([competitions[~competitions.Title.isin(df[\"Title of Competition\"])].Title.unique().shape[0], \\\n             competitions[competitions.Title.isin(df[\"Title of Competition\"])].Title.unique().shape[0]], \\\n           labels=[\"Meta-Kaggle Dataset\", f\"Kaggle AI Report Dataset\\n(n = {len(df)})\"])\n\naxes[1].set_title(\"Distribution of Competitions Tags (Meta-Kaggle vs AI Report) [%]\")\naxes[1].barh(y=insidecompetitiontags.index[:10], width=insidecompetitiontags.values[:10], alpha=0.5, align=\"center\")\naxes[1].barh(y=outsidecompetitiontags.index[:10], width=outsidecompetitiontags.values[:10], alpha=0.5, align=\"edge\")\naxes[1].invert_yaxis()\n\nplt.subplots_adjust(wspace=0.8)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:26:12.462133Z","iopub.execute_input":"2023-07-15T06:26:12.462447Z","iopub.status.idle":"2023-07-15T06:26:12.895196Z","shell.execute_reply.started":"2023-07-15T06:26:12.462421Z","shell.execute_reply":"2023-07-15T06:26:12.894182Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Tagging remaines almost unchanged over the Years \n\nThe tagging of the competitions in the Kaggle AI Report Dataset remained almost stable over the years and do not represent the evolution of the artificial intelligence. A surprising observation is the more frequent use of the beginner tag in 2023.","metadata":{}},{"cell_type":"code","source":"competitiontagsyear = competitiontags\\\n.drop(columns=\"Id\")\\\n.rename(columns={\"TagId\":\"Id\"})\\\n.merge(tags, on=\"Id\")\\\n.drop(columns=\"Id\")\\\n.rename(columns={\"CompetitionId\":\"Id\"})\\\n.merge(competitions, on=\"Id\")\\\n[['Name', 'FullPath', 'Title', 'EnabledDate']]\ncompetitiontagsyear[\"year\"] = competitiontagsyear.EnabledDate.map(lambda x: x.split()[0][-4:])\ncompetitiontagsyear = competitiontagsyear.groupby(\"year\")[\"Name\"].value_counts().unstack().fillna(0)\ncompetitiontagsyear_common = list(competitiontagsyear.columns[competitiontagsyear.sum()>20])\ncompetitiontagsyear_common.append(\"beginner\")\ncompetitiontagsyear_rare = competitiontagsyear.columns[competitiontagsyear.sum()<=20]\n\n# [col for col in competitiontagsyear.columns if any(competitiontagsyear[col]>10)]\n\nplt.figure(figsize=(12,4))\nplt.plot(competitiontagsyear[competitiontagsyear_common])\nplt.plot(competitiontagsyear[competitiontagsyear_rare], alpha=0.1, color=\"grey\")\nplt.legend(labels=competitiontagsyear_common)\nplt.title(\"Competition Tags over Time\")\nplt.ylabel(\"Tag [n]\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:26:12.896641Z","iopub.execute_input":"2023-07-15T06:26:12.897676Z","iopub.status.idle":"2023-07-15T06:26:13.663017Z","shell.execute_reply.started":"2023-07-15T06:26:12.897634Z","shell.execute_reply":"2023-07-15T06:26:13.662244Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluating Competitions with Natural Language Processing\n\nIt is possible that the competition tags do not provide a complete picture, which is why additional natural language processing techniques based on the writeups were applied. For each competition, the writeups were concatenated and key terms were [extracted using the TF-IDF algorithm](https://github.com/bsenst/data-science/blob/5c48a93d851927ac1b3f7a4ae4760257eb7d6352/competitions/kaggle-ai-report/kaggle-competitions-report.ipynb). In addition, the key terms and writeups were fed into an LLM via the [Sketch library](https://github.com/approximatelabs/sketch) with the question of technology and the area of machine learning. Overall the stemmed terms \"classif\", \"tf\", \"neural\" can be found in most competitions.","metadata":{"execution":{"iopub.status.busy":"2023-06-30T17:54:38.657856Z","iopub.execute_input":"2023-06-30T17:54:38.658343Z","iopub.status.idle":"2023-06-30T17:54:38.685425Z","shell.execute_reply.started":"2023-06-30T17:54:38.658303Z","shell.execute_reply":"2023-06-30T17:54:38.683581Z"}}},{"cell_type":"code","source":"# manually defined terms of interest\nterms = [\n    \"imagenet\", \"dqn\", \"lstm\", \"rnn\", \"cnn\", \"transfer\", \"automl\",\n    \"transformer\", \"tf\", \"tensor\", \"reinforcement\", \"mining\", \"sound\",\n    \"gradient boost\", \"tsp\", \"keras\", \"adversarial\", \"pca\", \"deep\",\n    \"xgb\", \"distance\", \"sequence\", \"factorization\", \"classif\", \"auc\",\n    \"ngram\", \"cluster\", \"vision\", \"nlp\", \"recommend\", \"neural\",\n    \"forest\", \"svm\", \"gbm\", \"bayes\", \"convolutional\", \"regress\", \"knn\"\n]\n\ndef get_topic(term):\n    return pd.Series([term in str(df.loc[i].values).lower() for i in range(len(df))]).apply(lambda x: 0 if x == False else 1).to_list()\n    \nfor term in terms:\n    df[term] = get_topic(term)\n\nplt.figure(figsize=(16,4))\nterm_freq = df[terms].sum().sort_values(ascending=False)\nplt.bar(term_freq.index.values, term_freq.values, width=0.7)\nplt.xticks(rotation=90)\nplt.ylabel(\"Competitions [n]\")\nplt.title(\"Describing Kaggle Competitions with Key Terms\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:26:13.664270Z","iopub.execute_input":"2023-07-15T06:26:13.665155Z","iopub.status.idle":"2023-07-15T06:27:08.857394Z","shell.execute_reply.started":"2023-07-15T06:26:13.665114Z","shell.execute_reply":"2023-07-15T06:27:08.856582Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most competitions are captured by one or two manually defined terms. No competition remains without a manually defined label.","metadata":{}},{"cell_type":"code","source":"# how many terms per ai report competition\nplt.figure(figsize=(16,4))\nterms_per_competition = pd.Series([(sum(row[1])) for row in df[terms].iterrows()]).value_counts().sort_index()\nplt.bar(terms_per_competition.index.values, terms_per_competition.values, width=0.7)\nplt.ylabel(\"Competitions [n]\")\nplt.xlabel(\"Terms [n]\")\nplt.title(\"Terms per Kaggle Competition\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:27:08.859064Z","iopub.execute_input":"2023-07-15T06:27:08.859681Z","iopub.status.idle":"2023-07-15T06:27:09.164859Z","shell.execute_reply.started":"2023-07-15T06:27:08.859647Z","shell.execute_reply":"2023-07-15T06:27:09.163845Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Kaggle Competitions reflect the Evolution of Artificial Intelligence\n\nThe figure shows the number of keywords in the competitions scaled to the total number of the respective keyword. It can be seen that the stemmed term \"gradient boost\", was common in the early 2010s and has become less common thereafter. On the other hand, \"keras\" from 2015, \"tensor\" from 2016, \"reinforcement\" from 2018 and \"automl\" in 2021 are frequently used in the Kaggle competition solutions. Thus Kaggle contest writeups reflect the evolution of artificial intelligence over the past decade.","metadata":{}},{"cell_type":"code","source":"# frequency of terms over time\ntimeseries = dict()\n\nfor term in df[terms]:\n    total = df[term].sum()\n    timeseries[term] = (df.groupby(by=\"Competition Launch Date\")[term].sum()[:-1]/total).to_list()\n\ntimeseries = pd.DataFrame(timeseries)\ntimeseries.index = ([year for year in range(2010, 2023)])\n\nfirst_mention = (timeseries[timeseries.columns[1:]]>0.0)\nfirst_mention[\"first_mention\"] = first_mention.idxmax().value_counts().sort_index()\nfirst_mention[\"first_mention\"] = first_mention[\"first_mention\"].replace(np.nan, 0).astype(int)\nfirst_mention[\"first_mention\"]\n\nts_overall = pd.DataFrame()\nts_overall[\"mean_occ\"] = timeseries.transpose().mean()\nts_overall[\"std_occ\"] = timeseries.transpose().std()\nts_overall = pd.merge(ts_overall, first_mention[\"first_mention\"], left_index=True, right_index=True)\n\nplt.figure(figsize=(16, 6))\n\nfor term in df[terms]:\n    if term in [\"tensor\", \"reinforcement\", \"gradient boost\", \"keras\", \"adversarial\", \"ngram\", \"vision\", \"automl\"]:\n        timeseries[term].plot(label=term, alpha=0.8, linewidth=2.0)\n    else:\n        timeseries[term].plot(label=\"_\"+term, alpha=0.3, color=\"grey\")\n\n(ts_overall.mean_occ+ts_overall.std_occ).plot(label=\"mean+std\", linestyle=(0, (1, 1)), linewidth=3.0, alpha=0.5, color=\"black\")\n(ts_overall.mean_occ-ts_overall.std_occ).plot(label=\"mean-std\", linestyle=(0, (1, 1)), linewidth=3.0, alpha=0.5, color=\"black\")\n\nplt.legend()\nplt.title(\"Manually defined Terms per Competition Writeup over Time\")\nplt.ylabel(\"Term Count Competition / Term Count in all Competitions\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:27:09.166387Z","iopub.execute_input":"2023-07-15T06:27:09.166702Z","iopub.status.idle":"2023-07-15T06:27:10.042818Z","shell.execute_reply.started":"2023-07-15T06:27:09.166676Z","shell.execute_reply":"2023-07-15T06:27:10.041653Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Competitions serve Purposes\n\nCompetitions serve purposes that depend on the clients. Most competitions are created by Kaggle. Essential for the number of participants is the reward that competitions promise. The most common rewards are knowledge gain and prize money.","metadata":{}},{"cell_type":"code","source":"organizations = pd.read_csv(\"/kaggle/input/meta-kaggle/Organizations.csv\")\n\n# competitions.groupby(\"RewardType\").RewardQuantity.value_counts()\n\n# most competitions launched by Kaggle Organization\ncomp_count_orga_dict = competitions.OrganizationId.value_counts().to_dict()\ntopcomporgs = organizations[organizations.Id.isin(competitions.OrganizationId.value_counts().index[:10])]\ntopcomporgs[\"CompetitionCount\"] = topcomporgs.Id.map(lambda x: comp_count_orga_dict[x])\ntopcomporgs.sort_values(\"CompetitionCount\", ascending=False)[[\"Name\", \"Description\", \"CompetitionCount\"]]","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:27:10.045019Z","iopub.execute_input":"2023-07-15T06:27:10.045438Z","iopub.status.idle":"2023-07-15T06:27:10.093713Z","shell.execute_reply.started":"2023-07-15T06:27:10.045400Z","shell.execute_reply":"2023-07-15T06:27:10.092666Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(6,3))\nplt.bar(competitions.RewardType.value_counts().index, height=competitions.RewardType.value_counts().values)\nplt.title(\"Competition Reward Type\")\nplt.ylabel(\"Competitions [n]\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:27:10.095310Z","iopub.execute_input":"2023-07-15T06:27:10.095651Z","iopub.status.idle":"2023-07-15T06:27:10.281842Z","shell.execute_reply.started":"2023-07-15T06:27:10.095623Z","shell.execute_reply":"2023-07-15T06:27:10.280983Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Motivation to participate in Competitions\n\nMost competitions offer only one prize for the winner, which is monetary or intangible such as gaining knowledge. Knowledge and prize money attract most submissions per competitor. Prizes, jobs and money trigger the high contribution to competitions. Only seven percent of competitions qualify for Kaggle Tiers of which most award prize money, prizes and jobs.","metadata":{}},{"cell_type":"code","source":"# people go for USD, keep in mind the number of competition for each reward type\ncompetitions[\"TotalSubmissions/TotalCompetitors\"] = (competitions.TotalSubmissions/competitions.TotalCompetitors).round(1)\nrewardparticipation = competitions.groupby(\"RewardType\")[['TotalTeams', 'TotalCompetitors', 'TotalSubmissions', \"TotalSubmissions/TotalCompetitors\"]].sum()\nrewardparticipation[\"TotalSub/TotalComp normalized for RewardCount\"] = [el[1] for el in sorted([(rewardtype, round(competitions.groupby(\"RewardType\")[\"TotalSubmissions/TotalCompetitors\"].sum().loc[rewardtype]/competitions.RewardType.value_counts().loc[rewardtype],2)) for rewardtype in competitions.RewardType.unique() if rewardtype is not np.nan])]\nrewardparticipation[\"TotalSub/TotalComp normalized for RewardCount\"]","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:27:10.282991Z","iopub.execute_input":"2023-07-15T06:27:10.283561Z","iopub.status.idle":"2023-07-15T06:27:10.311362Z","shell.execute_reply.started":"2023-07-15T06:27:10.283530Z","shell.execute_reply":"2023-07-15T06:27:10.310238Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# User Achievements remain mostly constant throughout the Years\n\nKaggle users can progress to different skill levels based on their ability. These are \"Novice\", \"Contributor\", \"Expert\", \"Master\" and \"Grandmaster\". Points and medals are awarded in the four categories \"Discussion\", \"Competitions\", \"Scripts\" and \"Datasets\". The qualifications awarded have remained almost the same over the years, although the number of users has increased. Thus, in order to achieve a higher qualification, the threshold must have risen continuously.\n\n[https://www.kaggle.com/progression](https://www.kaggle.com/progression)\n\n[https://www.kaggle.com/rankings](https://www.kaggle.com/rankings)","metadata":{}},{"cell_type":"code","source":"userachievements = ddf.read_csv(\"/kaggle/input/meta-kaggle/UserAchievements.csv\")\nuserachievements = userachievements.assign(year=userachievements.TierAchievementDate.map(lambda x: str(x)[-4:]))\nuserachievementsgrouped = userachievements.groupby(\"year\").Tier.value_counts().compute()\n\n# https://stackoverflow.com/questions/25386870/pandas-plotting-with-multi-index\nuserachievementsgrouped = userachievementsgrouped.unstack()[:-1]","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:27:10.312751Z","iopub.execute_input":"2023-07-15T06:27:10.313633Z","iopub.status.idle":"2023-07-15T06:28:18.107412Z","shell.execute_reply.started":"2023-07-15T06:27:10.313591Z","shell.execute_reply":"2023-07-15T06:28:18.106298Z"},"_kg_hide-input":true,"_kg_hide-output":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# https://matplotlib.org/stable/gallery/subplots_axes_and_figures/broken_axis.html\nfig, (ax1, ax2, ax3, ax4) = plt.subplots(4, 1, sharex=True)\nfig.suptitle(\"Kaggle Tiers Awarded Achievements during the Years\")\n\n# plot the same data on both axes\nax1.plot(userachievementsgrouped[0], color='green', label=\"Novice\")\nax1.legend(loc=\"upper left\")\nax2.plot(userachievementsgrouped[1], color='red', label=\"Contributor\")\nax2.legend(loc=\"upper left\")\nax3.plot(userachievementsgrouped[2], color='purple', label=\"Expert\")\nax3.legend(loc=\"upper left\")\nax4.plot(userachievementsgrouped.iloc[:,3:], label=[\"Master\", \"Grandmaster\"])\nax4.legend(loc=\"upper left\")\n\n# zoom-in / limit the view to different portions of the data\nax1.set_ylim(3*10**5, 14*10**6)  # outliers only\nax2.set_ylim(5_000, 3*10**5)  # most of the data\nax3.set_ylim(700, 5_000)\nax4.set_ylim(0, 700)\n\n# hide the spines between ax and ax2\nax1.spines.bottom.set_visible(False)\nax2.spines.top.set_visible(False)\nax2.spines.bottom.set_visible(False)\nax3.spines.top.set_visible(False)\nax3.spines.bottom.set_visible(False)\nax4.spines.top.set_visible(False)\n\nax1.xaxis.tick_top()\nax1.tick_params(labeltop=False)  # don't put tick labels at the top\nax2.tick_params(bottom=False)\nax3.tick_params(bottom=False)\nax4.xaxis.tick_bottom()\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:28:18.110451Z","iopub.execute_input":"2023-07-15T06:28:18.110792Z","iopub.status.idle":"2023-07-15T06:28:18.598338Z","shell.execute_reply.started":"2023-07-15T06:28:18.110765Z","shell.execute_reply":"2023-07-15T06:28:18.597072Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Users achieve Expert/Master/Grandmaster Status with Competitions\n\nMost achievements are made in the competitions. While the proportion of novices and contributors is the same across the achievement types, the proportion of achievements for experts, masters and grandmasters can be observed primarily through the competitions.","metadata":{}},{"cell_type":"code","source":"achievementtypetierusers = userachievements.groupby(\"AchievementType\").Tier.value_counts().compute()\n\nachievementtypetierusers_df = pd.DataFrame(achievementtypetierusers)\nachievementtypetierusers_df.rename(columns={\"Tier\":\"UserCount\"}, inplace=True)\nachievementtypetierusers_df.reset_index(inplace=True)\nachievementtypetierusers_df[\"TierAchievementType/TierTotal\"] = (pd.Series(list(achievementtypetierusers_df.UserCount))/pd.Series([el for five in [list(achievementtypetierusers_df.groupby(\"Tier\").sum().UserCount) for _ in range(4)] for el in five])).round(2)\n\nax = achievementtypetierusers_df.set_index([\"AchievementType\", \"Tier\"])[\"TierAchievementType/TierTotal\"].unstack().plot.bar(stacked=True)\nax.get_yaxis().set_visible(False)\nplt.title(\"Relative Share of Tiers per Achievement Type\")\nplt.legend([\"0 = Novice\", \"1 = Contributor\", \"2 = Expert\", \"3 = Master\", \"4 = Grandmaster\"])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:28:18.602401Z","iopub.execute_input":"2023-07-15T06:28:18.602835Z","iopub.status.idle":"2023-07-15T06:29:20.321301Z","shell.execute_reply.started":"2023-07-15T06:28:18.602796Z","shell.execute_reply":"2023-07-15T06:29:20.320206Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\nThe Kaggle platform offers the machine learning community an attractive environment, which stimulates the continuous growth in user numbers and user participation over the years. In this environment, users can grow in their skills and participate in the progress in the field of artificial intelligence. The developments in AI are reflected in the writeups of the solutions to the Kaggle competitions. Kaggle competitions are essential for users to advance in the field of machine learning.","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\n\nvalue_list = submission.value.to_list()\nvalue_list[0] = 'Kaggle Competitions' # 'copy/paste the exact category that you are submitting to'\nvalue_list[1] = 'https://www.kaggle.com/code/bnzn261029/kaggle-competitions-reflect-the-development-of-ai' # 'http://www.kaggle.com/your_username/your_notebook'\nvalue_list[2] = 'https://www.kaggle.com/code/jotangwq/ai-ethics-through-the-lens-of-immanuel-kant/comments#2330232' # 'http://www.kaggle.com/.../your_1st_peer_feedback'\nvalue_list[3] = 'https://www.kaggle.com/code/dhorvay/kaggle-is-the-global-data-science-classroom/comments#2330279' # 'http://www.kaggle.com/.../your_2nd_peer_feedback'\nvalue_list[4] = 'https://www.kaggle.com/code/sreeharicr/unleashing-the-power-of-kaggle-competitions/comments#2330289' # 'http://www.kaggle.com/.../your_3rd_peer_feedback'\n\nsubmission[\"value\"] = value_list\nsubmission.to_csv(\"/kaggle/working/submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2023-07-15T06:29:20.322361Z","iopub.execute_input":"2023-07-15T06:29:20.323159Z","iopub.status.idle":"2023-07-15T06:29:20.340097Z","shell.execute_reply.started":"2023-07-15T06:29:20.323128Z","shell.execute_reply":"2023-07-15T06:29:20.339194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## References\n* Chen, T., & Guestrin, C. (2016, August). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining (pp. 785-794).\n* Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84-90.\n* Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.\n* Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., ... & Liu, T. Y. (2017). Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30.\n* Zargar, S. (2021). Introduction to sequence learning models: RNN, LSTM, GRU. no. April.\n* Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2020). Generative adversarial networks. Commun. Acm, 63(11), 139-144.\n* Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818-2826).\n* He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).\n* Truong, A., Walters, A., Goodsitt, J., Hines, K., Bruss, C. B., & Farivar, R. (2019, November). Towards automated machine learning: Evaluation and comparison of AutoML approaches and tools. In 2019 IEEE 31st international conference on tools with artificial intelligence (ICTAI) (pp. 1471-1479). IEEE.\n* Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. Advances in neural information processing systems, 31.\n* Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.","metadata":{}}]}