{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**TASK: In this competition you're challenged to reconstruct the order of Kaggle notebooks whose cells have been shuffled.**","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"Let's try a simple approach with a simple ranking Model.","metadata":{}},{"cell_type":"markdown","source":"### What we will Learn?","metadata":{}},{"cell_type":"markdown","source":"* Wrangle the competition data and **create validation splits**,\n* Represent the code cell orders **with a feature**,\n* Build a ranking model with **XGBoost**,\n* Evaluate predictions with a Python implementation of the competition metric, and,\n* Format predictions to make a successful submission.","metadata":{}},{"cell_type":"markdown","source":"Objective of the  Model is to learn roughly where a cell should go in a notebook\nbased on what word it contains -- that, for example, cells containing \"Introduction\" or import should usually be near the beginning, while cells containing \"Submit\" or submission.csv should usually be near the end. These simple features are effective at reconstructing the global order of typical data science workflows. ","metadata":{}},{"cell_type":"markdown","source":"For a better Solution an understading of the interactions or relationship between cells is needed.","metadata":{}},{"cell_type":"markdown","source":"# Setup","metadata":{}},{"cell_type":"code","source":"import json\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nfrom scipy import sparse\nfrom tqdm import tqdm\n\n\npd.options.display.width = 180\npd.options.display.max_colwidth = 120\n\ndata_dir = Path('../input/AI4Code')","metadata":{"execution":{"iopub.status.busy":"2022-06-05T11:57:39.675578Z","iopub.execute_input":"2022-06-05T11:57:39.676353Z","iopub.status.idle":"2022-06-05T11:57:39.725511Z","shell.execute_reply.started":"2022-06-05T11:57:39.676247Z","shell.execute_reply":"2022-06-05T11:57:39.724802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Data","metadata":{}},{"cell_type":"markdown","source":"* The notebooks are stored as individiual JSON files.\n* They've been cleaned of the usual metadata present in Jupyter notebooks, leaving only the cell_type and source. \n\n* We'll load the notebooks here and join them **into a dataframe for easier processing**.\n<!-- * The full set of training data takes quite a while to load, so we'll just use a subset for this demonstration. -->","metadata":{}},{"cell_type":"code","source":"# NUM_TRAIN=10000   For a sample of training data\n\ndef read_notebooks(path):\n    return (\n        pd.read_json(\n            path,\n            dtype={'cell_type':'category','source':str})\n        .assign(id=path.stem)\n        .rename_axis('cell_id')\n    )\n\npaths_train=list((data_dir/'train').glob('*.json'))  # find all the files with .json and put them into list\n\nnotebook_train=[read_notebooks(path) for path in paths_train]\n\ndf= (\n    pd.concat(notebook_train)\n.set_index('id',append=True)\n.swaplevel()\n.sort_index(level='id',sort_remaining=False)\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:00:44.368716Z","iopub.execute_input":"2022-06-05T12:00:44.369097Z","iopub.status.idle":"2022-06-05T12:29:40.886192Z","shell.execute_reply.started":"2022-06-05T12:00:44.369067Z","shell.execute_reply":"2022-06-05T12:29:40.885062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(notebook_train))\nprint(notebook_train[0])","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:29:40.888448Z","iopub.execute_input":"2022-06-05T12:29:40.888942Z","iopub.status.idle":"2022-06-05T12:29:40.906231Z","shell.execute_reply.started":"2022-06-05T12:29:40.888898Z","shell.execute_reply":"2022-06-05T12:29:40.905076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:29:40.913193Z","iopub.execute_input":"2022-06-05T12:29:40.913689Z","iopub.status.idle":"2022-06-05T12:29:40.939281Z","shell.execute_reply.started":"2022-06-05T12:29:40.913641Z","shell.execute_reply":"2022-06-05T12:29:40.938433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each notebook has all the code cells given first with the markdown cells following. <br>\n**The code cells are in the correct relative order, while the markdown cells are shuffled.**\n\nIn the next section, we'll see how to recover the correct orderings for notebooks in the training set.","metadata":{}},{"cell_type":"code","source":"print('columns:',df.columns)\nprint(df.index)   # multi index dataframe","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:29:40.940437Z","iopub.execute_input":"2022-06-05T12:29:40.940936Z","iopub.status.idle":"2022-06-05T12:29:40.947584Z","shell.execute_reply.started":"2022-06-05T12:29:40.940903Z","shell.execute_reply":"2022-06-05T12:29:40.946792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get an example notebook\nnb_id=df.index.unique('id')[6]   # Return unique values in the index.\n\n# Print Notebook\nprint('Notebook:',nb_id)\n\nprint(\"The disorderd Notebook:\")\nnb=df.loc[nb_id,:]\ndisplay(nb)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:29:40.948749Z","iopub.execute_input":"2022-06-05T12:29:40.949225Z","iopub.status.idle":"2022-06-05T12:29:41.146163Z","shell.execute_reply.started":"2022-06-05T12:29:40.949192Z","shell.execute_reply":"2022-06-05T12:29:41.145329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our Task in this notebook is to predict the correct order of the  notebook cells, both code and markdown.\n Since you're given the relative ordering of the code cells among themselves, you could also think of this as predicting where the markdown cells should be placed among the code cells.\n\n","metadata":{}},{"cell_type":"markdown","source":"# Ordering the Cells","metadata":{}},{"cell_type":"markdown","source":"In the train_orders.csv file we have, for notebooks in the training set, the correct ordering of cells in terms of the cell ids.","metadata":{}},{"cell_type":"code","source":"df_orders=pd.read_csv(\n    data_dir/'train_orders.csv',\n    index_col='id',\n    squeeze=True\n).str.split()  # Split the string representation of cell_ids into a list\n\ndf_orders","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:29:41.147443Z","iopub.execute_input":"2022-06-05T12:29:41.148013Z","iopub.status.idle":"2022-06-05T12:29:43.549769Z","shell.execute_reply.started":"2022-06-05T12:29:41.147978Z","shell.execute_reply":"2022-06-05T12:29:43.548530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the correct order\ncell_order=df_orders.loc[nb_id]\n\nprint(\"The ordered Notebook:\")\nnb.loc[cell_order,:]   # get all columns for each cell order (id)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:29:43.551354Z","iopub.execute_input":"2022-06-05T12:29:43.552146Z","iopub.status.idle":"2022-06-05T12:29:43.567842Z","shell.execute_reply.started":"2022-06-05T12:29:43.552100Z","shell.execute_reply":"2022-06-05T12:29:43.567129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cell_order  # store the correct order of cell in the form of list of ID's","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:29:43.568881Z","iopub.execute_input":"2022-06-05T12:29:43.569419Z","iopub.status.idle":"2022-06-05T12:29:43.584787Z","shell.execute_reply.started":"2022-06-05T12:29:43.569384Z","shell.execute_reply":"2022-06-05T12:29:43.583733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The correct numeric position of a cell we will call the **rank** of the cell. We can find the ranks of the cells within a notebook by referencing the true ordering of cell ids as given in **train_orders.csv**","metadata":{}},{"cell_type":"code","source":"list(nb.index)   # store the index of ID's in sample notebook","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:29:43.588266Z","iopub.execute_input":"2022-06-05T12:29:43.588674Z","iopub.status.idle":"2022-06-05T12:29:43.599766Z","shell.execute_reply.started":"2022-06-05T12:29:43.588641Z","shell.execute_reply":"2022-06-05T12:29:43.598871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#find the index of '3e551fb7' in cell_order(actual order)\ncell_order.index('3e551fb7')","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:41:20.175248Z","iopub.execute_input":"2022-06-05T12:41:20.175646Z","iopub.status.idle":"2022-06-05T12:41:20.182417Z","shell.execute_reply.started":"2022-06-05T12:41:20.175616Z","shell.execute_reply":"2022-06-05T12:41:20.181050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_ranks(base,derived):\n    return [base.index(d) for d in derived]  #find the index of 'd' in cell_order(actual order)\n\ncell_ranks=get_ranks(cell_order,list(nb.index))\nnb.insert(0,'rank',cell_ranks)    # insert a new column in sample notebook\n\nnb","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:41:20.206876Z","iopub.execute_input":"2022-06-05T12:41:20.207981Z","iopub.status.idle":"2022-06-05T12:41:20.221847Z","shell.execute_reply.started":"2022-06-05T12:41:20.207940Z","shell.execute_reply":"2022-06-05T12:41:20.221117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Sorting a notebook by the cell ranks is another way to order the notebook.","metadata":{"execution":{"iopub.status.busy":"2022-06-03T10:52:28.707227Z","iopub.execute_input":"2022-06-03T10:52:28.707687Z","iopub.status.idle":"2022-06-03T10:52:28.714773Z","shell.execute_reply.started":"2022-06-03T10:52:28.707644Z","shell.execute_reply":"2022-06-03T10:52:28.712696Z"}}},{"cell_type":"code","source":"nb.sort_values('rank')","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:41:20.223142Z","iopub.execute_input":"2022-06-05T12:41:20.223601Z","iopub.status.idle":"2022-06-05T12:41:20.245166Z","shell.execute_reply.started":"2022-06-05T12:41:20.223570Z","shell.execute_reply":"2022-06-05T12:41:20.244523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pandas.testing import assert_frame_equal\n\nassert_frame_equal(nb.loc[cell_order, :], nb.sort_values('rank'))","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:41:20.246561Z","iopub.execute_input":"2022-06-05T12:41:20.247025Z","iopub.status.idle":"2022-06-05T12:41:20.263185Z","shell.execute_reply.started":"2022-06-05T12:41:20.246994Z","shell.execute_reply":"2022-06-05T12:41:20.262258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The algorithm we'll be using for our baseline model uses the cell ranks as the target, so let's create a dataframe of the ranks for each notebook.","metadata":{}},{"cell_type":"code","source":"type(df_orders)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:41:20.264203Z","iopub.execute_input":"2022-06-05T12:41:20.264579Z","iopub.status.idle":"2022-06-05T12:41:20.276909Z","shell.execute_reply.started":"2022-06-05T12:41:20.264546Z","shell.execute_reply":"2022-06-05T12:41:20.276182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_orders.to_frame()","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:41:20.278945Z","iopub.execute_input":"2022-06-05T12:41:20.279968Z","iopub.status.idle":"2022-06-05T12:41:20.307100Z","shell.execute_reply.started":"2022-06-05T12:41:20.279930Z","shell.execute_reply":"2022-06-05T12:41:20.306063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:41:20.308844Z","iopub.execute_input":"2022-06-05T12:41:20.309295Z","iopub.status.idle":"2022-06-05T12:41:20.326049Z","shell.execute_reply.started":"2022-06-05T12:41:20.309251Z","shell.execute_reply":"2022-06-05T12:41:20.325084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert Series to DataFrame.\ndf_orders_ = df_orders.to_frame().join(\n    df.reset_index('cell_id').groupby('id')['cell_id'].apply(list),\n    how='right',\n)\n\n# df_orders\n\nranks = {}\nfor id_, cell_order, cell_id in df_orders_.itertuples():\n    ranks[id_] = {'cell_id': cell_id, 'rank': get_ranks(cell_order, cell_id)}\n\ndf_ranks = (\n    pd.DataFrame\n    .from_dict(ranks, orient='index')\n    .rename_axis('id')\n    .apply(pd.Series.explode)\n    .set_index('cell_id', append=True)\n)\n\ndf_ranks\n","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:41:20.327387Z","iopub.execute_input":"2022-06-05T12:41:20.328237Z","iopub.status.idle":"2022-06-05T12:42:38.544309Z","shell.execute_reply.started":"2022-06-05T12:41:20.328204Z","shell.execute_reply":"2022-06-05T12:42:38.542969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Splits","metadata":{}},{"cell_type":"markdown","source":"The *df_ancestors.csv* file identifies groups of notebooks derived from a common origin, that is, notebooks belonging to the same forking tree.","metadata":{}},{"cell_type":"code","source":"df_ancestors=pd.read_csv(data_dir/'train_ancestors.csv',index_col='id')\ndf_ancestors","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:42:38.545721Z","iopub.execute_input":"2022-06-05T12:42:38.546075Z","iopub.status.idle":"2022-06-05T12:42:40.002758Z","shell.execute_reply.started":"2022-06-05T12:42:38.546043Z","shell.execute_reply":"2022-06-05T12:42:40.001607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To prevent leakage, the test set has no notebook with an ancestor in the training set. We therefore form a validation split using ancestor_id as a grouping factor.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GroupShuffleSplit\n\nNVALID = 0.1  # size of validation set\n\nsplitter=GroupShuffleSplit(n_splits=1,test_size=NVALID, random_state=0)\n\n# Split, keeping notebooks with a common origin (ancestor_id) together\nids = df.index.unique('id')\nancestors =df_ancestors.loc[ids,'ancestor_id']\nids_train, ids_valid = next(splitter.split(ids, groups=ancestors))\nids_train,ids_valid=ids[ids_train],ids[ids_valid]\n\ndf_train=df.loc[ids_train, :]\ndf_valid=df.loc[ids_valid, :]","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:42:40.005235Z","iopub.execute_input":"2022-06-05T12:42:40.005635Z","iopub.status.idle":"2022-06-05T12:42:43.712470Z","shell.execute_reply.started":"2022-06-05T12:42:40.005594Z","shell.execute_reply":"2022-06-05T12:42:43.711307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.describe()","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:42:43.713720Z","iopub.execute_input":"2022-06-05T12:42:43.714081Z","iopub.status.idle":"2022-06-05T12:42:58.539218Z","shell.execute_reply.started":"2022-06-05T12:42:43.714050Z","shell.execute_reply":"2022-06-05T12:42:58.538311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:42:58.540533Z","iopub.execute_input":"2022-06-05T12:42:58.540893Z","iopub.status.idle":"2022-06-05T12:42:58.556702Z","shell.execute_reply.started":"2022-06-05T12:42:58.540862Z","shell.execute_reply":"2022-06-05T12:42:58.555679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering\n\nLet's generate **tf-idf features** to use with our ranking model. These features will help our model learn what kinds of words tend to occur most often at various positions within a notebook.\n\n\n","metadata":{}},{"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer\n\n#Training set\ntfidf=TfidfVectorizer(min_df=0.01)\nX_train=tfidf.fit_transform(df_train['source'].astype(str)) # COnvertinf source into a feature\n\n# Rank of each cell within the notebook\ny_train=df_ranks.loc[ids_train].to_numpy()   #target\n\n# Number of cells in each notebook\ngroups=df_ranks.loc[ids_train].groupby('id').size().to_numpy()","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:42:58.558357Z","iopub.execute_input":"2022-06-05T12:42:58.559303Z","iopub.status.idle":"2022-06-05T12:47:17.112107Z","shell.execute_reply.started":"2022-06-05T12:42:58.559254Z","shell.execute_reply":"2022-06-05T12:47:17.111100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:47:17.113651Z","iopub.execute_input":"2022-06-05T12:47:17.114121Z","iopub.status.idle":"2022-06-05T12:47:17.121132Z","shell.execute_reply.started":"2022-06-05T12:47:17.114076Z","shell.execute_reply":"2022-06-05T12:47:17.120233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Now, we will add Code Cell ordering as a **feature**.\n* **We will append a column that enumerates the code cell in the  correct order, like 1, 2, 3, 4, ..., while having the dummy value 0 for all markdown cells.**\n* This feature will help the model learn to put the code cells in the correct order.","metadata":{}},{"cell_type":"code","source":"# Add code cell ordering\n\n# Stack sparse matrices horizontally (column wise)\nX_train=sparse.hstack((\n    X_train,\n    np.where(\n    df_train['cell_type'] == 'code',\n    df_train.groupby(['id','cell_type']).cumcount().to_numpy()+1,0,).reshape(-1,1)\n    )\n)\n    \nprint(X_train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:47:17.122934Z","iopub.execute_input":"2022-06-05T12:47:17.123672Z","iopub.status.idle":"2022-06-05T12:47:21.908901Z","shell.execute_reply.started":"2022-06-05T12:47:17.123628Z","shell.execute_reply":"2022-06-05T12:47:21.907759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train","metadata":{}},{"cell_type":"code","source":"from xgboost import XGBRanker\n\nmodel=XGBRanker(\nmin_child_weight=10,\nsubsample=0.5,\ntree_method='hist',)\n\nmodel.fit(X_train,y_train,group=groups)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:47:21.910174Z","iopub.execute_input":"2022-06-05T12:47:21.910541Z","iopub.status.idle":"2022-06-05T12:48:58.589756Z","shell.execute_reply.started":"2022-06-05T12:47:21.910502Z","shell.execute_reply":"2022-06-05T12:48:58.588801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluate\n\nNow let's see how well our model learned to order Kaggle notebook cells. We'll evaluate predictions on the validation set with a variant of the Kendall tau correlation.","metadata":{}},{"cell_type":"markdown","source":"## Validation set","metadata":{}},{"cell_type":"markdown","source":"First we'll create features for the validation set just like we did for the training set.","metadata":{}},{"cell_type":"markdown","source":"Let's understand **where(condition, [x, y])**\n\nReturn elements chosen from `x` or `y` depending on `condition`.\n\n.. note::\n    When only `condition` is provided, this function is a shorthand for\n    ``np.asarray(condition).nonzero()``. Using `nonzero` directly should be\n    preferred, as it behaves correctly for subclasses. The rest of this\n    documentation covers only the case where all three arguments are\n    provided.\n\nParameters\n----------\ncondition : array_like, bool\n    Where True, yield `x`, otherwise yield `y`.\nx, y : array_like\n    Values from which to choose. `x`, `y` and `condition` need to be\n    broadcastable to some shape.","metadata":{}},{"cell_type":"code","source":"np.where(df_valid['cell_type'] == 'code',\n    df_valid.groupby(['id','cell_type']).cumcount().to_numpy() +1,0).reshape(-1,1)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:48:58.591575Z","iopub.execute_input":"2022-06-05T12:48:58.592261Z","iopub.status.idle":"2022-06-05T12:48:58.968721Z","shell.execute_reply.started":"2022-06-05T12:48:58.592218Z","shell.execute_reply":"2022-06-05T12:48:58.967747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Validation set\nX_valid=tfidf.transform(df_valid['source'].astype(str))\n\n# The metric uses cell ids\ny_valid=df_orders.loc[ids_valid]\n\n# Stack sparse matrices horizontally (column wise)\nX_valid=sparse.hstack((\nX_valid,\nnp.where(\n    df_valid['cell_type'] == 'code',\n    df_valid.groupby(['id','cell_type']).cumcount().to_numpy() +1,0).reshape(-1,1)\n))","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:48:58.970021Z","iopub.execute_input":"2022-06-05T12:48:58.970374Z","iopub.status.idle":"2022-06-05T12:49:24.901472Z","shell.execute_reply.started":"2022-06-05T12:48:58.970343Z","shell.execute_reply":"2022-06-05T12:49:24.900536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we'll use the model to predict the rank of each cell within its notebook and then convert these ranks into a list of ordered cell ids.","metadata":{}},{"cell_type":"code","source":"y_pred=pd.DataFrame({'rank':model.predict(X_valid)},index=df_valid.index)\ny_pred=(y_pred\n        .sort_values(['id','rank']) # Sort the cells in each notebook by their rank\n        \n        .reset_index('cell_id')   # Convert the cell_id index into a column.\n        \n        .groupby('id')['cell_id'].apply(list)  # Group the cell_ids for each notebook into a list\n        \n       )\n\ny_pred.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:24.903425Z","iopub.execute_input":"2022-06-05T12:49:24.903918Z","iopub.status.idle":"2022-06-05T12:49:27.489684Z","shell.execute_reply.started":"2022-06-05T12:49:24.903872Z","shell.execute_reply":"2022-06-05T12:49:27.488547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's examine a notebook to see how the model did.","metadata":{"execution":{"iopub.status.busy":"2022-05-30T08:34:20.303621Z","iopub.execute_input":"2022-05-30T08:34:20.30444Z","iopub.status.idle":"2022-05-30T08:34:20.323537Z","shell.execute_reply.started":"2022-05-30T08:34:20.304387Z","shell.execute_reply":"2022-05-30T08:34:20.322734Z"}}},{"cell_type":"code","source":"nb_id = df_valid.index.get_level_values('id').unique()[8]\n\ndisplay(df.loc[nb_id])\ndisplay(df.loc[nb_id].loc[y_pred.loc[nb_id]])","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:27.490983Z","iopub.execute_input":"2022-06-05T12:49:27.491432Z","iopub.status.idle":"2022-06-05T12:49:27.707264Z","shell.execute_reply.started":"2022-06-05T12:49:27.491387Z","shell.execute_reply":"2022-06-05T12:49:27.706259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Metric","metadata":{}},{"cell_type":"markdown","source":"This competition uses a variant of the **Kendall tau correlation**, which will measure `how close to the correct order our predicted orderings are.` ","metadata":{}},{"cell_type":"code","source":"from bisect import bisect\n\n\ndef count_inversions(a):\n    inversions = 0\n    sorted_so_far = []\n    for i, u in enumerate(a):\n        j = bisect(sorted_so_far, u)\n        inversions += i - j\n        sorted_so_far.insert(j, u)\n    return inversions\n\n\ndef kendall_tau(ground_truth, predictions):\n    total_inversions = 0\n    total_2max = 0  # twice the maximum possible inversions across all instances\n    for gt, pred in zip(ground_truth, predictions):\n        ranks = [gt.index(x) for x in pred]  # rank predicted order in terms of ground truth\n        total_inversions += count_inversions(ranks)\n        n = len(gt)\n        total_2max += n * (n - 1)\n    return 1 - 4 * total_inversions / total_2max","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:27.711377Z","iopub.execute_input":"2022-06-05T12:49:27.711738Z","iopub.status.idle":"2022-06-05T12:49:27.720703Z","shell.execute_reply.started":"2022-06-05T12:49:27.711708Z","shell.execute_reply":"2022-06-05T12:49:27.719524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's test the metric with a dummy submission created from the ids of the shuffled notebooks.","metadata":{}},{"cell_type":"code","source":"y_dummy = df_valid.reset_index('cell_id').groupby('id')['cell_id'].apply(list)\nkendall_tau(y_valid, y_dummy)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:27.722307Z","iopub.execute_input":"2022-06-05T12:49:27.722832Z","iopub.status.idle":"2022-06-05T12:49:29.904542Z","shell.execute_reply.started":"2022-06-05T12:49:27.722775Z","shell.execute_reply":"2022-06-05T12:49:29.903535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Comparing this to the score on the predictions, we can see that our model was indeed able to improve the cell ordering somewhat.","metadata":{}},{"cell_type":"code","source":"kendall_tau(y_valid, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:29.905810Z","iopub.execute_input":"2022-06-05T12:49:29.906260Z","iopub.status.idle":"2022-06-05T12:49:31.269320Z","shell.execute_reply.started":"2022-06-05T12:49:29.906214Z","shell.execute_reply":"2022-06-05T12:49:31.268168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"markdown","source":"To create a submission for this competition, we'll apply our model to the notebooks in the test set. Note that this is a Code Competition, which means that the test data we see here is only a small sample. When we submit our notebook for scoring, this example data will be replaced with the full test set of about 20,000 notebooks.\n\n","metadata":{}},{"cell_type":"markdown","source":"First we load the data.","metadata":{}},{"cell_type":"code","source":"paths_test = list((data_dir / 'test').glob('*.json'))\nnotebooks_test = [\n    read_notebooks(path) for path in tqdm(paths_test, desc='Test NBs')\n]\ndf_test = (\n    pd.concat(notebooks_test)\n    .set_index('id', append=True)\n    .swaplevel()\n    .sort_index(level='id', sort_remaining=False)\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:31.270822Z","iopub.execute_input":"2022-06-05T12:49:31.272040Z","iopub.status.idle":"2022-06-05T12:49:31.329327Z","shell.execute_reply.started":"2022-06-05T12:49:31.271997Z","shell.execute_reply":"2022-06-05T12:49:31.328329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then create the tf-idf and code cell features.","metadata":{}},{"cell_type":"code","source":"X_test = tfidf.transform(df_test['source'].astype(str))\nX_test = sparse.hstack((\n    X_test,\n    np.where(\n        df_test['cell_type'] == 'code',\n        df_test.groupby(['id', 'cell_type']).cumcount().to_numpy() + 1,\n        0,\n    ).reshape(-1, 1)\n))","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:31.330710Z","iopub.execute_input":"2022-06-05T12:49:31.331159Z","iopub.status.idle":"2022-06-05T12:49:31.350968Z","shell.execute_reply.started":"2022-06-05T12:49:31.331114Z","shell.execute_reply":"2022-06-05T12:49:31.349946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And then create predictions on the test set.","metadata":{}},{"cell_type":"code","source":"y_infer = pd.DataFrame({'rank': model.predict(X_test)}, index=df_test.index)\ny_infer = y_infer.sort_values(['id', 'rank']).reset_index('cell_id').groupby('id')['cell_id'].apply(list)\ny_infer","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:31.352168Z","iopub.execute_input":"2022-06-05T12:49:31.352562Z","iopub.status.idle":"2022-06-05T12:49:31.380053Z","shell.execute_reply.started":"2022-06-05T12:49:31.352530Z","shell.execute_reply":"2022-06-05T12:49:31.379094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The sample_submission.csv file shows what a correctly formatted submission must look like. We'll just use it as a visual check, but you might like to directly modify the values of sample submission instead. (This would help prevent failed submissions due to missing notebook ids or incorrectly named columns, for instance.)","metadata":{}},{"cell_type":"code","source":"y_sample = pd.read_csv(data_dir / 'sample_submission.csv', index_col='id', squeeze=True)\ny_sample","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:31.381404Z","iopub.execute_input":"2022-06-05T12:49:31.381803Z","iopub.status.idle":"2022-06-05T12:49:31.397520Z","shell.execute_reply.started":"2022-06-05T12:49:31.381770Z","shell.execute_reply":"2022-06-05T12:49:31.396446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that a correctly formatted submission needs the index named id and the column of cell orders named cell_order. Moreover, we need to convert the list of cell ids into a space-delimited string of cell ids.","metadata":{}},{"cell_type":"code","source":"y_submit = (\n    y_infer\n    .apply(' '.join)  # list of ids -> string of ids\n    .rename_axis('id')\n    .rename('cell_order')\n)\ny_submit","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:31.399013Z","iopub.execute_input":"2022-06-05T12:49:31.400074Z","iopub.status.idle":"2022-06-05T12:49:31.411159Z","shell.execute_reply.started":"2022-06-05T12:49:31.400027Z","shell.execute_reply":"2022-06-05T12:49:31.410134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_submit.to_csv('submission.csv')","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:31.412292Z","iopub.execute_input":"2022-06-05T12:49:31.412683Z","iopub.status.idle":"2022-06-05T12:49:31.427915Z","shell.execute_reply.started":"2022-06-05T12:49:31.412640Z","shell.execute_reply":"2022-06-05T12:49:31.427118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.display import FileLink \nFileLink(r'./submission.csv')","metadata":{"execution":{"iopub.status.busy":"2022-06-05T12:49:31.429637Z","iopub.execute_input":"2022-06-05T12:49:31.430380Z","iopub.status.idle":"2022-06-05T12:49:31.440607Z","shell.execute_reply.started":"2022-06-05T12:49:31.430337Z","shell.execute_reply":"2022-06-05T12:49:31.439824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}