{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"AI4Code Pytorch DistilBert Baseline¶\nI used a lot of code from Kaggle's starter notebook here: https://www.kaggle.com/code/ryanholbrook/getting-started-with-ai4code\nthis notebook explin how to extract functions,classes,variables....names ,so we can use them in a clean data, and improve score","metadata":{}},{"cell_type":"markdown","source":"# Setup #","metadata":{}},{"cell_type":"code","source":"import json\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nfrom scipy import sparse\nfrom tqdm import tqdm\n\npd.options.display.width = 180\npd.options.display.max_colwidth = 120\n\ndata_dir = Path('../input/AI4Code')","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-06-12T06:36:53.389265Z","iopub.execute_input":"2022-06-12T06:36:53.389607Z","iopub.status.idle":"2022-06-12T06:36:53.501945Z","shell.execute_reply.started":"2022-06-12T06:36:53.389525Z","shell.execute_reply":"2022-06-12T06:36:53.501049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Data #","metadata":{}},{"cell_type":"markdown","source":"The notebooks are stored as individiual JSON files. They've been cleaned of the usual metadata present in Jupyter notebooks, leaving only the `cell_type` and `source`. The [Data](https://www.kaggle.com/competitions/AI4Code/data) page on the competition website has the full documentation of this dataset.\n\nWe'll load the notebooks here and join them into a dataframe for easier processing. The full set of training data takes quite a while to load, so we'll just use a subset for this demonstration.","metadata":{}},{"cell_type":"code","source":"NUM_TRAIN = 200\n\n\ndef read_notebook(path):\n    return (\n        pd.read_json(\n            path,\n            dtype={'cell_type': 'category', 'source': 'str'})\n        .assign(id=path.stem)\n        .rename_axis('cell_id')\n    )\n\n\npaths_train = list((data_dir / 'train').glob('*.json'))[:NUM_TRAIN]\nnotebooks_train = [\n    read_notebook(path) for path in tqdm(paths_train, desc='Train NBs')\n]\ndf = (\n    pd.concat(notebooks_train)\n    .set_index('id', append=True)\n    .swaplevel()\n    .sort_index(level='id', sort_remaining=False)\n)\n\ndf","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-06-12T06:36:53.503502Z","iopub.execute_input":"2022-06-12T06:36:53.503742Z","iopub.status.idle":"2022-06-12T06:36:59.114130Z","shell.execute_reply.started":"2022-06-12T06:36:53.503713Z","shell.execute_reply":"2022-06-12T06:36:59.113284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each notebook has all the code cells given first with the markdown cells following. The code cells are in the correct relative order, while the markdown cells are shuffled. In the next section, we'll see how to recover the correct orderings for notebooks in the training set.","metadata":{}},{"cell_type":"code","source":"# Get an example notebook\nnb_id = df.index.unique('id')[6]\nprint('Notebook:', nb_id)\n\nprint(\"The disordered notebook:\")\nnb = df.loc[nb_id, :]\ndisplay(nb)\nprint()","metadata":{"execution":{"iopub.status.busy":"2022-06-12T06:36:59.115343Z","iopub.execute_input":"2022-06-12T06:36:59.116043Z","iopub.status.idle":"2022-06-12T06:36:59.134081Z","shell.execute_reply.started":"2022-06-12T06:36:59.116000Z","shell.execute_reply":"2022-06-12T06:36:59.133210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Your task in this competition is to predict the correct order of the notebook cells, both code and markdown. Since you're given the relative ordering of the code cells among themselves, you could also think of this as predicting where the markdown cells should be placed among the code cells.\n\nFor example, a disordered notebook might be:\n```\ncode_1\ncode_2\ncode_3\nmarkdown_1\nmarkdown_2\n```\nand the correctly ordered notebook might be:\n```\ncode_1\nmarkdown_2\ncode_2\ncode_3\nmarkdown_1\n```\nThe markdown cells can be in any order, but you would never see `code_2` before `code_1`, for instance.","metadata":{}},{"cell_type":"markdown","source":"# Ordering the Cells #","metadata":{}},{"cell_type":"markdown","source":"In the `train_orders.csv` file we have, for notebooks in the training set, the correct ordering of cells in terms of the cell ids.","metadata":{}},{"cell_type":"code","source":"df_orders = pd.read_csv(\n    data_dir / 'train_orders.csv',\n    index_col='id',\n    squeeze=True,\n).str.split()  # Split the string representation of cell_ids into a list\n\ndf_orders","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-06-12T06:36:59.136610Z","iopub.execute_input":"2022-06-12T06:36:59.136902Z","iopub.status.idle":"2022-06-12T06:37:01.843316Z","shell.execute_reply.started":"2022-06-12T06:36:59.136853Z","shell.execute_reply":"2022-06-12T06:37:01.842652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the correct order\ncell_order = df_orders.loc[nb_id]\n\nprint(\"The ordered notebook:\")\nnb.loc[cell_order, :]","metadata":{"execution":{"iopub.status.busy":"2022-06-12T06:37:01.844510Z","iopub.execute_input":"2022-06-12T06:37:01.844983Z","iopub.status.idle":"2022-06-12T06:37:01.860598Z","shell.execute_reply.started":"2022-06-12T06:37:01.844939Z","shell.execute_reply":"2022-06-12T06:37:01.859651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The correct numeric position of a cell we will call the **rank** of the cell. We can find the ranks of the cells within a notebook by referencing the true ordering of cell ids as given in `train_orders.csv`.","metadata":{}},{"cell_type":"code","source":"def get_ranks(base, derived):\n    return [base.index(d) for d in derived]\n\ncell_ranks = get_ranks(cell_order, list(nb.index))\nnb.insert(0, 'rank', cell_ranks)\n\nnb","metadata":{"execution":{"iopub.status.busy":"2022-06-12T06:37:01.861950Z","iopub.execute_input":"2022-06-12T06:37:01.862251Z","iopub.status.idle":"2022-06-12T06:37:01.881568Z","shell.execute_reply.started":"2022-06-12T06:37:01.862212Z","shell.execute_reply":"2022-06-12T06:37:01.880704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"code=nb.loc[nb.cell_type==\"code\"]","metadata":{"execution":{"iopub.status.busy":"2022-06-12T06:38:45.415125Z","iopub.execute_input":"2022-06-12T06:38:45.415480Z","iopub.status.idle":"2022-06-12T06:38:45.421071Z","shell.execute_reply.started":"2022-06-12T06:38:45.415431Z","shell.execute_reply":"2022-06-12T06:38:45.419818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"code","metadata":{"execution":{"iopub.status.busy":"2022-06-12T06:38:49.171718Z","iopub.execute_input":"2022-06-12T06:38:49.172007Z","iopub.status.idle":"2022-06-12T06:38:49.184795Z","shell.execute_reply.started":"2022-06-12T06:38:49.171976Z","shell.execute_reply":"2022-06-12T06:38:49.184221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tokenize\nimport io","metadata":{"execution":{"iopub.status.busy":"2022-06-12T06:37:01.976970Z","iopub.execute_input":"2022-06-12T06:37:01.977419Z","iopub.status.idle":"2022-06-12T06:37:01.982236Z","shell.execute_reply.started":"2022-06-12T06:37:01.977378Z","shell.execute_reply":"2022-06-12T06:37:01.981322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"code.loc['d3e265f6','source']","metadata":{"execution":{"iopub.status.busy":"2022-06-12T06:40:33.951705Z","iopub.execute_input":"2022-06-12T06:40:33.951968Z","iopub.status.idle":"2022-06-12T06:40:33.958296Z","shell.execute_reply.started":"2022-06-12T06:40:33.951942Z","shell.execute_reply":"2022-06-12T06:40:33.957181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"code_text = tokenize.generate_tokens(io.StringIO(code.loc['d3e265f6','source']).readline)\n[tok for tok in code_text]","metadata":{"execution":{"iopub.status.busy":"2022-06-12T06:47:10.490118Z","iopub.execute_input":"2022-06-12T06:47:10.490381Z","iopub.status.idle":"2022-06-12T06:47:10.497520Z","shell.execute_reply.started":"2022-06-12T06:47:10.490344Z","shell.execute_reply":"2022-06-12T06:47:10.496729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#extract only function names , variables,comments then we can join them\ncode_text = tokenize.generate_tokens(io.StringIO(code.loc['d3e265f6','source']).readline)\n[tok.string for tok in code_text if tok.type==1 or tok.type==55]\n","metadata":{"execution":{"iopub.status.busy":"2022-06-12T06:47:39.603939Z","iopub.execute_input":"2022-06-12T06:47:39.604228Z","iopub.status.idle":"2022-06-12T06:47:39.611683Z","shell.execute_reply.started":"2022-06-12T06:47:39.604201Z","shell.execute_reply":"2022-06-12T06:47:39.610921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#join all strings\ncode_text = tokenize.generate_tokens(io.StringIO(code.loc['d3e265f6','source']).readline)\n' '.join([tok.string for tok in code_text if tok.type==1 or tok.type==55])","metadata":{"execution":{"iopub.status.busy":"2022-06-12T06:58:43.796251Z","iopub.execute_input":"2022-06-12T06:58:43.796549Z","iopub.status.idle":"2022-06-12T06:58:43.803293Z","shell.execute_reply.started":"2022-06-12T06:58:43.796516Z","shell.execute_reply":"2022-06-12T06:58:43.802707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}