{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-05-12T08:20:49.512825Z","iopub.execute_input":"2022-05-12T08:20:49.513126Z","iopub.status.idle":"2022-05-12T08:20:49.517118Z","shell.execute_reply.started":"2022-05-12T08:20:49.513095Z","shell.execute_reply":"2022-05-12T08:20:49.516419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"src = '/kaggle/input/AI4Code/'","metadata":{"execution":{"iopub.status.busy":"2022-05-12T07:24:09.845058Z","iopub.execute_input":"2022-05-12T07:24:09.845392Z","iopub.status.idle":"2022-05-12T07:24:09.848652Z","shell.execute_reply.started":"2022-05-12T07:24:09.845360Z","shell.execute_reply":"2022-05-12T07:24:09.848012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Overview","metadata":{}},{"cell_type":"code","source":"!ls /kaggle/input/AI4Code/","metadata":{"execution":{"iopub.status.busy":"2022-05-12T07:24:10.666324Z","iopub.execute_input":"2022-05-12T07:24:10.666633Z","iopub.status.idle":"2022-05-12T07:24:10.959840Z","shell.execute_reply.started":"2022-05-12T07:24:10.666610Z","shell.execute_reply":"2022-05-12T07:24:10.958977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **train/** - A folder comprising about 140,000 JSON files with the filenames corresponding to the id field in the csv files. Each file contains the code and markdown cells of a Kaggle notebook. The code cells are in their original (correct) order. The markdown cells have been shuffled and placed after the code cells.","metadata":{}},{"cell_type":"code","source":"train_jsons = os.listdir(src + '/train')[:10]\ntrain_jsons","metadata":{"execution":{"iopub.status.busy":"2022-05-12T07:24:11.485757Z","iopub.execute_input":"2022-05-12T07:24:11.486052Z","iopub.status.idle":"2022-05-12T07:24:11.541868Z","shell.execute_reply.started":"2022-05-12T07:24:11.486019Z","shell.execute_reply":"2022-05-12T07:24:11.541101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **train_orders.csv** - Gives the correct order of the cells for each notebook in the train/ folder.\n* **id** - The notebook in file {id}.json.\n* **cell_order** - A space delimited list of the correct cell ordering given in terms of the order in {id}.json.","metadata":{}},{"cell_type":"code","source":"train_orders_df = pd.read_csv(src + 'train_orders.csv')\nprint('train_orders:')\ndisplay(train_orders_df.head())","metadata":{"execution":{"iopub.status.busy":"2022-05-12T07:24:12.426106Z","iopub.execute_input":"2022-05-12T07:24:12.426342Z","iopub.status.idle":"2022-05-12T07:24:13.040405Z","shell.execute_reply.started":"2022-05-12T07:24:12.426319Z","shell.execute_reply":"2022-05-12T07:24:13.039639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Most of notebooks have 21 cells (~3100 of 140000)","metadata":{}},{"cell_type":"code","source":"train_orders_df['length'] = train_orders_df['cell_order'].apply(lambda x: len(x.split(' ')))\n\nimport plotly.express as px\nfig = px.histogram(train_orders_df, x=\"length\",title='Number of cells in notebook')\nfig.show()","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-05-12T07:25:22.524850Z","iopub.execute_input":"2022-05-12T07:25:22.525083Z","iopub.status.idle":"2022-05-12T07:25:24.317935Z","shell.execute_reply.started":"2022-05-12T07:25:22.525042Z","shell.execute_reply":"2022-05-12T07:25:24.317139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **train_ancestors.csv** - On Kaggle, a user may \"fork\" (that is, copy) the notebook of another user to create their own version. This file contains the forking history of notebooks in the training set. Note: There is no corresponding file for the test set.\n* **ancestor_id** - Identifies sets of notebooks that have a common origin or \"ancestor\". As no notebook in the test set has an ancestor in the training set, you may find this field to be of use as a grouping factor when constructing validation splits.\n* **parent_id** - Indicates that some version of the notebook id was forked from some version of the notebook parent_id. The notebook parent_id may or may not be present in the training data. (The parent may be missing because someone had forked a private notebook of their own, for instance.)","metadata":{}},{"cell_type":"code","source":"train_ancestors_df = pd.read_csv(src + 'train_ancestors.csv')\nprint('train_ancestors:')\ndisplay(train_ancestors_df.head())","metadata":{"execution":{"iopub.status.busy":"2022-05-12T07:24:16.495253Z","iopub.execute_input":"2022-05-12T07:24:16.495538Z","iopub.status.idle":"2022-05-12T07:24:16.615050Z","shell.execute_reply.started":"2022-05-12T07:24:16.495507Z","shell.execute_reply":"2022-05-12T07:24:16.613902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **sample_submission** looks like this","metadata":{}},{"cell_type":"code","source":"sample_submission_df = pd.read_csv(src + 'sample_submission.csv')\nprint('sample_submission:')\ndisplay(sample_submission_df.head())\n","metadata":{"execution":{"iopub.status.busy":"2022-05-12T07:24:18.807463Z","iopub.execute_input":"2022-05-12T07:24:18.807729Z","iopub.status.idle":"2022-05-12T07:24:18.820393Z","shell.execute_reply.started":"2022-05-12T07:24:18.807700Z","shell.execute_reply":"2022-05-12T07:24:18.819529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------------\n-------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"# Functions to work with notebooks and some findings","metadata":{}},{"cell_type":"code","source":"def read_notebook_from_train_orders(i):\n    \"\"\"\n    This function reads a notebook from i-th line of train_orders.csv with a correct cell order\n    \"\"\"\n    id_, cell_order, length = train_orders_df.iloc[i]\n    path = src + 'train/' + id_ +'.json'\n    cell_order = cell_order.split( )\n    #print(cell_order)\n    notebook_df = pd.read_json(\n                            path,\n                            dtype={'cell_type': 'category', 'source': 'str'}\n                            ).rename_axis('cell_id')\n    return notebook_df.loc[cell_order] # put cells in a correct cell order ","metadata":{"execution":{"iopub.status.busy":"2022-05-12T07:25:37.290090Z","iopub.execute_input":"2022-05-12T07:25:37.290359Z","iopub.status.idle":"2022-05-12T07:25:37.296651Z","shell.execute_reply.started":"2022-05-12T07:25:37.290329Z","shell.execute_reply":"2022-05-12T07:25:37.296135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Example:","metadata":{}},{"cell_type":"code","source":"read_notebook_from_train_orders(1)","metadata":{"execution":{"iopub.status.busy":"2022-05-12T07:25:39.210717Z","iopub.execute_input":"2022-05-12T07:25:39.211580Z","iopub.status.idle":"2022-05-12T07:25:39.240769Z","shell.execute_reply.started":"2022-05-12T07:25:39.211552Z","shell.execute_reply":"2022-05-12T07:25:39.240286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Most notebooks have roughly the same number of code and markdown cells, but there are a lot of notebooks that have a lot of code with small number of markdowns","metadata":{}},{"cell_type":"code","source":"ratios = []\nN = len(train_orders_df)\n\nfor i in tqdm(range(N // 40)):\n    table = read_notebook_from_train_orders(i)\n    id_, cell_order, length = train_orders_df.iloc[i]\n    path = src + 'train/' + id_ +'.json'\n    cell_order = cell_order.split()\n    #print(cell_order)\n    table = pd.read_json(\n                        path,\n                        dtype={'cell_type': 'category'}\n                        ).rename_axis('cell_id')\n    n_code, n_markdown = table.cell_type.value_counts()\n    ratios.append(n_code / n_markdown)\n\npx.histogram(ratios,title='histogram of ratio of number of code cells to markdown cells')\n","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-05-12T08:21:29.642187Z","iopub.execute_input":"2022-05-12T08:21:29.642406Z","iopub.status.idle":"2022-05-12T08:21:44.759855Z","shell.execute_reply.started":"2022-05-12T08:21:29.642384Z","shell.execute_reply":"2022-05-12T08:21:44.758835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_with_parent_id = train_ancestors_df[train_ancestors_df.parent_id.notna()]\ndisplay(df_with_parent_id.head())\nparent_ids = df_with_parent_id.parent_id.tolist()\nprint(f'There are {len(parent_ids)} notebooks with parent_id\\nbut none of them are represented in train_orders.csv')\ntrain_orders_df[train_orders_df.id.isin(parent_ids)]\n","metadata":{"execution":{"iopub.status.busy":"2022-05-11T22:42:51.515297Z","iopub.execute_input":"2022-05-11T22:42:51.515658Z","iopub.status.idle":"2022-05-11T22:42:51.555006Z","shell.execute_reply.started":"2022-05-11T22:42:51.515623Z","shell.execute_reply":"2022-05-11T22:42:51.554086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* \"... you may find **ancestor** field to be of use as a **grouping factor** when constructing validation splits\"","metadata":{}},{"cell_type":"code","source":"train_ancestors_df.ancestor_id.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-11T22:27:56.060733Z","iopub.execute_input":"2022-05-11T22:27:56.061020Z","iopub.status.idle":"2022-05-11T22:27:56.194837Z","shell.execute_reply.started":"2022-05-11T22:27:56.060989Z","shell.execute_reply":"2022-05-11T22:27:56.193840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* There are 65 notebooks with **4569bfc1** as an ancestor","metadata":{}},{"cell_type":"code","source":"train_ancestors_df[train_ancestors_df.ancestor_id == '4569bfc1']","metadata":{"execution":{"iopub.status.busy":"2022-05-11T22:48:08.027962Z","iopub.execute_input":"2022-05-11T22:48:08.028301Z","iopub.status.idle":"2022-05-11T22:48:08.055110Z","shell.execute_reply.started":"2022-05-11T22:48:08.028268Z","shell.execute_reply":"2022-05-11T22:48:08.054378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Here are several examples of these notebooks that starts from the same markdown:","metadata":{}},{"cell_type":"code","source":"read_notebook_from_train_orders(10760)","metadata":{"execution":{"iopub.status.busy":"2022-05-11T22:55:56.202544Z","iopub.execute_input":"2022-05-11T22:55:56.203276Z","iopub.status.idle":"2022-05-11T22:55:56.231187Z","shell.execute_reply.started":"2022-05-11T22:55:56.203239Z","shell.execute_reply":"2022-05-11T22:55:56.229681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"read_notebook_from_train_orders(2613)","metadata":{"execution":{"iopub.status.busy":"2022-05-11T22:49:42.438329Z","iopub.execute_input":"2022-05-11T22:49:42.439235Z","iopub.status.idle":"2022-05-11T22:49:42.470352Z","shell.execute_reply.started":"2022-05-11T22:49:42.439159Z","shell.execute_reply":"2022-05-11T22:49:42.469521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"read_notebook_from_train_orders(4918)","metadata":{"execution":{"iopub.status.busy":"2022-05-11T22:53:43.542869Z","iopub.execute_input":"2022-05-11T22:53:43.543172Z","iopub.status.idle":"2022-05-11T22:53:43.571230Z","shell.execute_reply.started":"2022-05-11T22:53:43.543142Z","shell.execute_reply":"2022-05-11T22:53:43.570295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('2613:', read_notebook_from_train_orders(2613).source[0])\nprint('4918:', read_notebook_from_train_orders(4918).source[0])\nprint('10760:', read_notebook_from_train_orders(10760).source[0])\n","metadata":{"execution":{"iopub.status.busy":"2022-05-11T22:57:56.148277Z","iopub.execute_input":"2022-05-11T22:57:56.149333Z","iopub.status.idle":"2022-05-11T22:57:56.185806Z","shell.execute_reply.started":"2022-05-11T22:57:56.149287Z","shell.execute_reply":"2022-05-11T22:57:56.184300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### By the way, their last markdowns are also the same:","metadata":{}},{"cell_type":"code","source":"print('2613:', read_notebook_from_train_orders(2613).source[-1])\nprint('4918:', read_notebook_from_train_orders(4918).source[-1])\nprint('10760:', read_notebook_from_train_orders(10760).source[-1])","metadata":{"execution":{"iopub.status.busy":"2022-05-11T22:58:35.309651Z","iopub.execute_input":"2022-05-11T22:58:35.310205Z","iopub.status.idle":"2022-05-11T22:58:35.343145Z","shell.execute_reply.started":"2022-05-11T22:58:35.310150Z","shell.execute_reply":"2022-05-11T22:58:35.341809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"--------------------------------------------------","metadata":{}},{"cell_type":"markdown","source":"# Collecting training dataset for your NLP / Ai model","metadata":{}},{"cell_type":"code","source":"N = 1000 # number of notebooks to add\ndata = []\n\nfor i in range(N):\n    data.append(read_notebook_from_train_orders(N)[['cell_type','source']])","metadata":{"execution":{"iopub.status.busy":"2022-05-11T23:09:31.926620Z","iopub.execute_input":"2022-05-11T23:09:31.927155Z","iopub.status.idle":"2022-05-11T23:09:38.239017Z","shell.execute_reply.started":"2022-05-11T23:09:31.927115Z","shell.execute_reply":"2022-05-11T23:09:38.238188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[666]","metadata":{"execution":{"iopub.status.busy":"2022-05-11T23:11:49.246313Z","iopub.execute_input":"2022-05-11T23:11:49.247716Z","iopub.status.idle":"2022-05-11T23:11:49.267568Z","shell.execute_reply.started":"2022-05-11T23:11:49.247658Z","shell.execute_reply":"2022-05-11T23:11:49.265206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}