![](\"https://i.imgur.com/2Egeb8R.png\")
π΄ The goal of the competition is to summarize the advancements in AI from the past two years by tapping into the expertise of the Kaggle community.
\n\nβͺ Participants will write an essay on one of seven topics related to AI, including text data, image and/or video data, tabular and/or time series data, Kaggle competitions, generative AI, AI ethics, and \"Other\".
\n\nβͺ Participants are free to use any resources to write their essays, but Kaggle has provided a collection of writeups and scholarly articles for reference.
\n\nβͺ Participants will take part in a peer feedback process, where they will review three peer essays, and their submission will be assessed by a panel of seven Kaggle Grandmasters.
\n\nβͺ Winning essays will be aggregated in a post-competition publication and given authorship credit.
\n\nβͺ The competition also includes a ranking system, where eligible participants will be ranked on a leaderboard and awarded points and medals, which will be displayed on their Kaggle profile.
","metadata":{}},{"cell_type":"markdown","source":"#### \n#βͺ In this analytics competition, participants will write an essay on one of the following seven topics, with a prompt to describe what the community has learned over the past 2 years of working and experimenting with:
\n\n*Text data
\n*Image and/or video data
\n*Tabular and/or time series data
\n*Kaggle Competitions
\n*Generative AI
\n*AI ethics
\n*Other (anything that does not fall into any other category)
","metadata":{}},{"cell_type":"markdown","source":"βͺ Kaggle has provided a collection of writeups and a collection of scholarly articles in the official competition dataset, but participants are free to use any resources.
","metadata":{}},{"cell_type":"markdown","source":"\n##βͺ let's first dive into the \"Kaggle Competitions\" topic, as it generally stands apart from other topics and, accordingly, other tools and resources are needed specifically for this topic.
\n\nβͺ In the competition data specified the file kaggle_writeups.csv - metadata from recent Kaggle Competition Writeups. Let's load it and see what it looks like.
βͺ Considering that the terms of the Competishine state that we need to examine data only for the last 2 years, we can easily get rid of everything superfluous. As a result, this data farm was reduced three times, after leaving the most relevant data in it.
","metadata":{}},{"cell_type":"code","source":"kaggle_writeups_df[\"Date of Writeup\"] = pd.to_datetime(kaggle_writeups_df[\"Date of Writeup\"])\n\nfiltered_kaggle_writeups_df = kaggle_writeups_df[kaggle_writeups_df['Date of Writeup'].dt.year >= 2021]","metadata":{"execution":{"iopub.status.busy":"2023-06-14T04:53:12.120445Z","iopub.execute_input":"2023-06-14T04:53:12.121017Z","iopub.status.idle":"2023-06-14T04:53:12.525841Z","shell.execute_reply.started":"2023-06-14T04:53:12.120985Z","shell.execute_reply":"2023-06-14T04:53:12.524855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"filtered_kaggle_writeups_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T04:53:15.267082Z","iopub.execute_input":"2023-06-14T04:53:15.267481Z","iopub.status.idle":"2023-06-14T04:53:15.282653Z","shell.execute_reply.started":"2023-06-14T04:53:15.267453Z","shell.execute_reply":"2023-06-14T04:53:15.281851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"filtered_kaggle_writeups_df.describe()","metadata":{"execution":{"iopub.status.busy":"2023-06-14T04:53:41.889876Z","iopub.execute_input":"2023-06-14T04:53:41.890260Z","iopub.status.idle":"2023-06-14T04:53:41.931670Z","shell.execute_reply.started":"2023-06-14T04:53:41.890233Z","shell.execute_reply":"2023-06-14T04:53:41.930109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(filtered_kaggle_writeups_df)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T04:54:01.552889Z","iopub.execute_input":"2023-06-14T04:54:01.553320Z","iopub.status.idle":"2023-06-14T04:54:01.560814Z","shell.execute_reply.started":"2023-06-14T04:54:01.553288Z","shell.execute_reply":"2023-06-14T04:54:01.559576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"βͺ This dataset is just an extraction from a more general set Meta Kaggle.
\n\n
Meta Kaggle data example
","metadata":{}},{"cell_type":"markdown","source":"βͺ Also I recommend using Kaggle solutions. This is a list of almost all available solutions and ideas shared by top performers in the past Kaggle competitions. This list gets updated as soon as a new competition finishes. If you find a solution besides the ones listed here, I would encourage you to contribute.
\n\nKaggle solutions page
","metadata":{}},{"cell_type":"markdown","source":"\n##βͺ All other topics can be combined under the heading of scientific articles, since they fully relate to this particular topic. In competition data have included a collection of scholarly articles for our convenience, but we are free to use additional resources as well. There is no requirement to refer to any of these files using code.
\n\nβͺ In the competition data specified the file arXiv_metadata.json - metadata for recent arXiv publications. Let's load it and see what it looks like.
βͺ The number of scholarly articles is simply gigantic, so let's first filter out all the articles that were published before 2021.
","metadata":{}},{"cell_type":"code","source":"date_format = \"%a, %d %b %Y %H:%M:%S %Z\"\nlimit_year = datetime.strptime(\"2021\", \"%Y\")\n\nfiltered_arxiv = []\nfor article_meta in dict_arxiv:\n parsed_date = datetime.strptime(\n article_meta[\"versions\"][0][\"created\"], \n date_format\n )\n if parsed_date >= limit_year:\n filtered_arxiv.append(article_meta)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:08:35.711620Z","iopub.execute_input":"2023-06-14T05:08:35.712362Z","iopub.status.idle":"2023-06-14T05:09:26.627510Z","shell.execute_reply.started":"2023-06-14T05:08:35.712308Z","shell.execute_reply":"2023-06-14T05:09:26.626080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(filtered_arxiv)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:09:26.629378Z","iopub.execute_input":"2023-06-14T05:09:26.629722Z","iopub.status.idle":"2023-06-14T05:09:26.637230Z","shell.execute_reply.started":"2023-06-14T05:09:26.629694Z","shell.execute_reply":"2023-06-14T05:09:26.635703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"βͺ The number of articles has drastically decreased, but still there are articles in this set that have nothing to do with AI, ML, etc. In order to select only topics of interest to us, you can use arXiv Category Taxonomy.
\n\narXiv Category Taxonomy page
\n\nβͺ Most of the articles we are interested in are in Computer Science, but you can choose others, for example, from mathematics or statistics, if you wish.
\n","metadata":{}},{"cell_type":"code","source":"categories = [\"cs.AI\", \"cs.DS\", \"cs.LG\", \"cs.NE\", \"cs.RO\", \"cs.CL\"] # cs.CV","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:10:06.228866Z","iopub.execute_input":"2023-06-14T05:10:06.229297Z","iopub.status.idle":"2023-06-14T05:10:06.236748Z","shell.execute_reply.started":"2023-06-14T05:10:06.229266Z","shell.execute_reply":"2023-06-14T05:10:06.234492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categories_ids = {}\nfor category in categories:\n category_ids = []\n for article_meta in filtered_arxiv:\n if category in article_meta[\"categories\"]:\n category_ids.append(article_meta[\"id\"])\n categories_ids[category] = set(category_ids)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:10:08.940830Z","iopub.execute_input":"2023-06-14T05:10:08.941212Z","iopub.status.idle":"2023-06-14T05:10:10.719283Z","shell.execute_reply.started":"2023-06-14T05:10:08.941184Z","shell.execute_reply":"2023-06-14T05:10:10.717810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"βͺ Let's display a graph of the correspondence between the number of articles for each of the selected category.
","metadata":{}},{"cell_type":"code","source":"fig = px.bar(x=categories_ids.keys(), y=[len(x) for x in categories_ids.values()], text_auto='.2s')\nfig.update_layout(\n xaxis_title=\"Category\",\n yaxis_title=\"Articles count\"\n)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-06-14T05:11:10.184474Z","iopub.execute_input":"2023-06-14T05:11:10.185038Z","iopub.status.idle":"2023-06-14T05:11:12.112159Z","shell.execute_reply.started":"2023-06-14T05:11:10.184991Z","shell.execute_reply":"2023-06-14T05:11:12.110936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"βͺ Now let's see which articles use the different categories that were selected earlier. This can show the extent to which one category is interchangeable with another. For example, from the visualization below, it is clear that for half of the articles where the category cs.LG is indicated, the category cs.AI is also indicated.
","metadata":{}},{"cell_type":"code","source":"vals = list(categories_ids.values())\nlabels = list(categories_ids.keys())\nc = ('#3E64AF', '#3EAF5D')\n# subplot indexes\ntxt_indexes = list(range(1, len(labels)*5, 6))\ntitle_indexes = list(range(2, len(labels)*6, 7))\nplot_indexes = [8, 14, 20, 26, 15, 21, 27, 22, 28, 29]\n# combinations of sets\ntitle_sets = [[x, y] for x, y in zip(vals[:len(vals)-1], vals[1:len(vals)])]\nplot_sets = []\nfor i in range(len(vals)-2):\n for j in range(i+2, len(vals)):\n plot_sets.append([vals[i], vals[j]])\n\nfig, ax = plt.subplots(1, figsize=(24,24))\n# plot texts\nfor idx, txt_idx in enumerate(txt_indexes):\n plt.subplot(6, 6, txt_idx)\n plt.text(0.5,0.5,\n labels[idx+1], \n fontsize=24,\n ha=\"center\", va=\"center\", color=\"#1F764B\")\n plt.axis('off')\n# plot top plots (the ones with a title)\nfor idx, title_idx in enumerate(title_indexes):\n plt.subplot(6, 6, title_idx)\n venn2(title_sets[idx], set_colors=c, set_labels = (' ', ' '))\n plt.title(labels[idx], fontsize=24, color=\"#1F4576\")\n# plot the rest of the diagrams\nfor idx, plot_idx in enumerate(plot_indexes):\n plt.subplot(6, 6, plot_idx)\n venn2(plot_sets[idx], set_colors=c, set_labels = (' ', ' '))\nplt.savefig('venn_matrix.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-06-14T05:11:25.709678Z","iopub.execute_input":"2023-06-14T05:11:25.710277Z","iopub.status.idle":"2023-06-14T05:11:27.892219Z","shell.execute_reply.started":"2023-06-14T05:11:25.710232Z","shell.execute_reply":"2023-06-14T05:11:27.890712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"π΄ After we figured out how to work with articles, we can now go directly to the topics. Since the provided dictionary does not contain any information that would help to accurately identify which topic this or that article belongs to, therefore, it is necessary to somehow classify all these articles into 5 remaining topics:
\n\n*Text data
\n*Image and/or video data
\n*Tabular and/or time series data
\n*Generative AI
\n*AI ethics
","metadata":{}},{"cell_type":"markdown","source":"βͺ To solve this problem, I suggest using Zero-Shot Classification.
βͺ Zero-shot text classification is a task in natural language processing where a model is trained on a set of labeled examples but is then able to classify new examples from previously unseen classes.
Zero-Shot Classification illustration
","metadata":{}},{"cell_type":"markdown","source":"π΄ For more information follow this link.
","metadata":{}},{"cell_type":"markdown","source":"βͺ Load the model to perform Zero-Shot Classification.
","metadata":{}},{"cell_type":"code","source":"pipe = pipeline(model=\"facebook/bart-large-mnli\")","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:30:27.889890Z","iopub.execute_input":"2023-06-14T05:30:27.890412Z","iopub.status.idle":"2023-06-14T05:30:53.291727Z","shell.execute_reply.started":"2023-06-14T05:30:27.890376Z","shell.execute_reply":"2023-06-14T05:30:53.289509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"βͺ Indicate candidate labels.
","metadata":{}},{"cell_type":"code","source":"candidate_labels = [\n \"Text data\", \n \"Image and/or video data\", \n \"Tabular and/or time series data\", \n \"Generative AI\", \n \"AI ethics\"\n]","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:30:53.299994Z","iopub.execute_input":"2023-06-14T05:30:53.300493Z","iopub.status.idle":"2023-06-14T05:30:53.314043Z","shell.execute_reply.started":"2023-06-14T05:30:53.300443Z","shell.execute_reply":"2023-06-14T05:30:53.312501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"βͺ Choose a random article from the cs.AI category.
","metadata":{}},{"cell_type":"code","source":"rand_article_id = list(categories_ids[\"cs.LG\"])[0]\nrand_article_id","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:33:15.710226Z","iopub.execute_input":"2023-06-14T05:33:15.711192Z","iopub.status.idle":"2023-06-14T05:33:15.734479Z","shell.execute_reply.started":"2023-06-14T05:33:15.711136Z","shell.execute_reply":"2023-06-14T05:33:15.732722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rand_article = [x for x in filtered_arxiv if x[\"id\"] == rand_article_id][0]\nrand_article","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:33:19.257753Z","iopub.execute_input":"2023-06-14T05:33:19.260063Z","iopub.status.idle":"2023-06-14T05:33:19.482693Z","shell.execute_reply.started":"2023-06-14T05:33:19.260024Z","shell.execute_reply":"2023-06-14T05:33:19.481598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"βͺ And run Zero-Shot Classification.
","metadata":{}},{"cell_type":"code","source":"rand_article_abstract = rand_article[\"abstract\"]","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:33:30.201060Z","iopub.execute_input":"2023-06-14T05:33:30.201507Z","iopub.status.idle":"2023-06-14T05:33:30.206931Z","shell.execute_reply.started":"2023-06-14T05:33:30.201471Z","shell.execute_reply":"2023-06-14T05:33:30.205695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results = pipe(rand_article[\"abstract\"], candidate_labels=candidate_labels)","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:34:36.906345Z","iopub.execute_input":"2023-06-14T05:34:36.906842Z","iopub.status.idle":"2023-06-14T05:34:42.035716Z","shell.execute_reply.started":"2023-06-14T05:34:36.906805Z","shell.execute_reply":"2023-06-14T05:34:42.034557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"list(zip(results[\"labels\"], results[\"scores\"]))","metadata":{"execution":{"iopub.status.busy":"2023-06-14T05:34:42.037819Z","iopub.execute_input":"2023-06-14T05:34:42.038159Z","iopub.status.idle":"2023-06-14T05:34:42.047801Z","shell.execute_reply.started":"2023-06-14T05:34:42.038130Z","shell.execute_reply":"2023-06-14T05:34:42.046630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"π΄ As you can see from the results, the most likely topic is Text data, which is the correct label. Thus, you can go through all the articles and leave only those that belong to the topic you have chosen.
βͺ I'm planning on making a similar set soon to make your life a little easier, so stay tuned :).
","metadata":{}},{"cell_type":"code","source":"# TODO: Go through every article and classify it","metadata":{"execution":{"iopub.status.busy":"2023-05-12T22:52:03.606273Z","iopub.execute_input":"2023-05-12T22:52:03.606657Z","iopub.status.idle":"2023-05-12T22:52:03.620847Z","shell.execute_reply.started":"2023-05-12T22:52:03.606627Z","shell.execute_reply":"2023-05-12T22:52:03.619679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# (β β κβ α΄β κβ ) WORK STILL IN PROGRESS","metadata":{"execution":{"iopub.status.busy":"2023-05-12T22:52:03.622471Z","iopub.execute_input":"2023-05-12T22:52:03.623436Z","iopub.status.idle":"2023-05-12T22:52:03.636822Z","shell.execute_reply.started":"2023-05-12T22:52:03.623393Z","shell.execute_reply":"2023-05-12T22:52:03.635825Z"},"trusted":true},"execution_count":null,"outputs":[]}]}