{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
The change in Kaggle competitions in the past 2 years
\n\nCategory: Kaggle Competitions\n \n**1.Introduction**\n\nKaggle is an important web community in data science and artificial intelligence. It was very early to focus on data science and train data scientists. The first Kaggle competition began at 2000-01-01 00:00:00. Artificial Intelligence things are hotting and swarming. And It has brought great change to data science and Kaggle’s community in the past 2 years. How did Kaggle’s community change in the past 2 years? A suitable and efficient procedure is important to find the change in Kaggle competitions.\n\n**2.Procedure**\n\nThis part will introduce the procedure to get ready-to-use data.\n> **2.1 Data Structure**\n\n> For the change of Kaggle competitions, the relative data of the date must be collected and processed. After analyzing Meta-Kaggle, Table 1 was obtained. In the screening process, tree categories are obvious and useful. First one includes Competitions.csv, Datasets.csv, Tags.csv. Tags.csv contains classification, which is important to find the relatives of AI. The second one is Submissions.csv, Users.csv, Teams.csv, which contribute to Kaggle competitions. The last has some diversity expansion, which consists of code, discussion, organization in communities.","metadata":{}},{"cell_type":"markdown","source":"
Table 1. The data structure of Kaggle Competition
\n\n
\n \"Trulli\"\n
\n","metadata":{}},{"cell_type":"markdown","source":"> **2.2 Target Data**\n\n> According to the start date of the Kaggle AI Report (20230511) and 2 years, the target data is obtained by screening the data, including Competitions.csv, Datasets.csv, Submissions.csv, Kernels.csv.\n\n> **2.3 Need to discuss continuously the competitions with AI**\n\n> For Tags.csv, by observing the variables(ParentTagId, FullPath), it can be known that the tag is classified with a tree structure. And the size of Tags.csv is small. Through observing FullPath one by one, the tag of artificial intelligence is found in Table 2.","metadata":{}},{"cell_type":"markdown","source":"
Table 2. The tag of artificial intelligence
\n\n
\n \"Trulli\"\n
\n","metadata":{}},{"cell_type":"markdown","source":"> Then it is easy to get CompetitionId from 'CompetitionTags.csv' and find the detail from 'Competitions.csv' as Table 3.","metadata":{}},{"cell_type":"markdown","source":"
Table 3. The detail of competitions with AI
\n\n
\n \"Trulli\"\n
\n","metadata":{}},{"cell_type":"markdown","source":"**3 Summary**\n\nSome statistical characteristics are needed. This part will introduce some insights into Kaggle competitions in the past two years.\n\nAs shown in Table 4, the competitions’ percentage is the same as submissions percentage. Kernels and datasets percentage are almost 1.5 times and 2.5 times as much as them, respectively. More importantly, it happened in just two years, which is faster than the time from the earliest date.","metadata":{}},{"cell_type":"markdown","source":"
Table 4. The target data in past two years
\n\n
\n \"Trulli\"\n
\n","metadata":{}},{"cell_type":"markdown","source":"In the next tables, the variables are presented in a grouping sort order. \n\nTable 5 shows the top 10 competitions in submissions. The total number of submissions is correlated with competitors’ and teams and does not necessarily correlate with reward quantity and prizes’ number. It indicates that more rewards and prizes produce more submissions and fiercer competition in general. Simultaneously, some special topics have some function.","metadata":{}},{"cell_type":"markdown","source":"
Table 5. Top 10 of Competitions submissions
\n\n
\n \"Trulli\"\n
\n","metadata":{}},{"cell_type":"markdown","source":"Table 6 shows the top 10 datasets in kernels’ number. Usually, competitions come with several datasets so that Table 6 reflects the contribution of competition to community from other side. It found that the total number of views, votes, downloads doesn’t increase as the total number of kernel increase. The total number of votes is correlated with downloads to some degree. It's not necessarily that more submissions lead to more attention.","metadata":{}},{"cell_type":"markdown","source":"
Table 6. Top 10 of Datasets total kernels
\n\n
\n \"Trulli\"\n
\n","metadata":{}},{"cell_type":"markdown","source":"Table 7 shows the top 10 kernels in views. From Table 6, it is not easy to find the relations between the total number of views and votes. But from Table 7, the total number of views doesn’t bring the bigger number of votes and medal. It indicates that kernels’ recognition from community or organizer does not necessarily relate to its attention.","metadata":{}},{"cell_type":"markdown","source":"
Table 7. Top 10 of Kernels total views
\n\n
\n \"Trulli\"\n
\n","metadata":{}},{"cell_type":"markdown","source":"**4 Supporting Material**\n> **4.1 codes**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport time\npath_basic = \"/kaggle/input/meta-kaggle/\"\n\ndef screen_date(data_path, drop_col, drop_v):\n data = pd.read_csv(data_path)\n data = data.drop(drop_col, axis=1)\n data[drop_v] = pd.to_datetime(data[drop_v])\n x = len(data)\n early_date = min(data[drop_v])\n data = data[data[drop_v] > pd.Timestamp(2021, 5, 10)]\n return [early_date, len(data), len(data) / x]\n\nfilenames = ['Competitions.csv', 'Datasets.csv', 'Submissions.csv', 'Kernels.csv']\ndrop_col = {'Competitions.csv':\n ['Slug', 'Subtitle', 'HostSegmentTitle', 'ForumId', 'OrganizationId', 'DeadlineDate',\n 'ProhibitNewEntrantsDeadlineDate', 'TeamMergerDeadlineDate',\n 'TeamModelDeadlineDate', 'ModelSubmissionDeadlineDate',\n 'FinalLeaderboardHasBeenVerified', 'HasKernels', 'OnlyAllowKernelSubmissions', 'HasLeaderboard',\n 'LeaderboardDisplayFormat', 'EvaluationAlgorithmAbbreviation',\n 'EvaluationAlgorithmName', 'EvaluationAlgorithmDescription',\n 'EvaluationAlgorithmIsMax', 'MaxDailySubmissions',\n 'MaxTeamSize', 'BanTeamMergers','EnableTeamModels', 'RewardType', 'RewardQuantity',\n 'UserRankMultiplier', 'CanQualifyTiers', 'ValidationSetName',\n 'ValidationSetValue', 'EnableSubmissionModelHashes', 'EnableSubmissionModelAttachments', 'HostName',\n 'NumScoredSubmissions', 'TotalTeams', 'TotalCompetitors', 'TotalSubmissions', 'CompetitionTypeId'],\n 'Datasets.csv':\n ['OwnerUserId', 'OwnerOrganizationId', 'CurrentDatasetVersionId',\n 'CurrentDatasourceVersionId', 'Type', 'LastActivityDate',\n 'TotalDownloads', 'TotalVotes', 'TotalKernels'],\n 'Submissions.csv':\n ['SourceKernelVersionId', 'ScoreDate', 'IsAfterDeadline',\n 'PublicScoreFullPrecision', 'PrivateScoreFullPrecision'],\n 'Kernels.csv':\n ['CurrentKernelVersionId', 'ForkParentKernelVersionId', 'ForumTopicId',\n 'FirstKernelVersionId', 'EvaluationDate', 'MadePublicDate',\n 'IsProjectLanguageTemplate', 'CurrentUrlSlug']}\ndrop_v = {'Competitions.csv': 'EnabledDate',\n 'Datasets.csv': 'CreationDate',\n 'Submissions.csv': 'SubmissionDate',\n 'Kernels.csv': 'CreationDate'}\nresults = []\nfor filename in filenames:\n data_path = path_basic + filename\n result = screen_date(data_path, drop_col[filename], drop_v[filename])\n results.append(result)\nresults","metadata":{"execution":{"iopub.status.busy":"2023-06-08T08:43:30.855805Z","iopub.execute_input":"2023-06-08T08:43:30.856218Z","iopub.status.idle":"2023-06-08T08:46:18.033711Z","shell.execute_reply.started":"2023-06-08T08:43:30.856186Z","shell.execute_reply":"2023-06-08T08:46:18.031641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport time\npath_basic = \"/kaggle/input/meta-kaggle/\"\n\ndata = pd.read_csv(path_basic + \"Tags.csv\")\ndata = data.drop(['Slug', 'Description',\n 'DatasetCount', 'CompetitionCount', 'KernelCount'], axis=1)\nprint(data.columns, len(data))\nprint(data[data['Name'] == 'artificial intelligence']['Id'].tolist()[0])\nprint(data[data['Name'] == 'artificial intelligence']['FullPath'].tolist()[0])","metadata":{"execution":{"iopub.status.busy":"2023-06-08T08:43:23.364085Z","iopub.execute_input":"2023-06-08T08:43:23.364465Z","iopub.status.idle":"2023-06-08T08:43:23.464669Z","shell.execute_reply.started":"2023-06-08T08:43:23.364435Z","shell.execute_reply":"2023-06-08T08:43:23.463756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\ntable1 = pd.read_csv('/kaggle/input/the-relative-data-in-kaggle-ai-report/The data structure of Kaggle Competition.csv')\ntable2 = pd.read_csv('/kaggle/input/the-relative-data-in-kaggle-ai-report/The tag of artificial intelligence.csv')\ntable3 = pd.read_csv('/kaggle/input/the-relative-data-in-kaggle-ai-report/The detail of competitions with AI.csv')\ntable4 = pd.read_csv('/kaggle/input/the-relative-data-in-kaggle-ai-report/The target data in past two years.csv')\ntable5 = pd.read_csv('/kaggle/input/the-relative-data-in-kaggle-ai-report/Top 10 of Competitions submissions.csv')\ntable6 = pd.read_csv('/kaggle/input/the-relative-data-in-kaggle-ai-report/Top 10 of Datasets total kernels.csv')\ntable7 = pd.read_csv('/kaggle/input/the-relative-data-in-kaggle-ai-report/Top 10 of Kernels total views.csv')\ndata = pd.read_csv(\"/kaggle/input/the-relative-data-in-kaggle-ai-report/submission.csv\")\ndata","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **4.2 Reference**\n\n> [[1] ANTHONY WYNNE. AI report - Meta Kaggle data joins.](https://www.kaggle.com/code/anthony35813/ai-report-meta-kaggle-data-joins)","metadata":{}}]}