{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Welcome to the Google AI4Code Competition! #\n\nIn this competition you're challenged to reconstruct the order of Kaggle notebooks whose cells have been shuffled. Check out the [Competition Pages](https://www.kaggle.com/competitions/AI4Code/overview) for a complete overview.\n\nThis notebook will walk you through making a submission with a simple ranking model. We'll look at how to:\n- Wrangle the competition data and create validation splits,\n- Represent the code cell orders with a feature,\n- Build a ranking model with XGBoost,\n- Evaluate predictions with a Python implementation of the competition metric, and,\n- Format predictions to make a successful submission.\n\nOur model will be able to learn roughly where a cell should go in a notebook based on what words it contains -- that, for example, cells containing \"Introduction\" or `import` should usually be near the beginning, while cells containing \"Submit\" or `submission.csv` should usually be near the end. These simple features are effective at reconstructing the global order of typical data science workflows. An understanding of the *interactions* or *relationships between cells*, however, will be required of the most successful solutions. We encourage you therefore to explore things like modern neural network language models for learning the relationships between natural language and computer code.","metadata":{}},{"cell_type":"markdown","source":"# Setup #","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"import json\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nfrom scipy import sparse\nfrom tqdm import tqdm\n\npd.options.display.width = 180\npd.options.display.max_colwidth = 120\n\ndata_dir = Path('../input/AI4Code')","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-03-21T12:50:16.167057Z","iopub.execute_input":"2023-03-21T12:50:16.167380Z","iopub.status.idle":"2023-03-21T12:50:16.211446Z","shell.execute_reply.started":"2023-03-21T12:50:16.167342Z","shell.execute_reply":"2023-03-21T12:50:16.210369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Data #","metadata":{}},{"cell_type":"markdown","source":"The notebooks are stored as individiual JSON files. They've been cleaned of the usual metadata present in Jupyter notebooks, leaving only the `cell_type` and `source`. The [Data](https://www.kaggle.com/competitions/AI4Code/data) page on the competition website has the full documentation of this dataset.\n\nWe'll load the notebooks here and join them into a dataframe for easier processing. The full set of training data takes quite a while to load, so we'll just use a subset for this demonstration.","metadata":{}},{"cell_type":"code","source":"len(list((data_dir / 'train').glob('*.json')))","metadata":{"execution":{"iopub.status.busy":"2023-03-21T09:52:31.037465Z","iopub.execute_input":"2023-03-21T09:52:31.037769Z","iopub.status.idle":"2023-03-21T09:52:32.226508Z","shell.execute_reply.started":"2023-03-21T09:52:31.037739Z","shell.execute_reply":"2023-03-21T09:52:32.225557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"paths = list((data_dir / 'train').glob('*.json'))\n\nasync def get_num_cells(path):\n    return pd.read_json(path).size\n\nnum_cells = []\nfor path in tqdm(paths):\n    res = await get_num_cells(path)\n    num_cells.append(res)\n\n#num_cells = pd.Series([get_num_cells(nb_path) for nb_path in paths])\n#num_cells","metadata":{"execution":{"iopub.status.busy":"2023-03-21T12:50:40.271748Z","iopub.execute_input":"2023-03-21T12:50:40.272069Z","iopub.status.idle":"2023-03-21T13:24:24.408178Z","shell.execute_reply.started":"2023-03-21T12:50:40.272038Z","shell.execute_reply":"2023-03-21T13:24:24.407633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(num_cells)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T13:43:31.960376Z","iopub.execute_input":"2023-03-21T13:43:31.960673Z","iopub.status.idle":"2023-03-21T13:43:31.969679Z","shell.execute_reply.started":"2023-03-21T13:43:31.960641Z","shell.execute_reply":"2023-03-21T13:43:31.968425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(num_cells).describe()","metadata":{"execution":{"iopub.status.busy":"2023-03-21T13:46:03.225454Z","iopub.execute_input":"2023-03-21T13:46:03.225798Z","iopub.status.idle":"2023-03-21T13:46:03.286889Z","shell.execute_reply.started":"2023-03-21T13:46:03.225760Z","shell.execute_reply":"2023-03-21T13:46:03.285743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"hist_data = pd.Series(num_cells).value_counts().to_frame(\"num_notebooks\").reset_index().rename(columns={\"index\": \"num_cells\"})","metadata":{"execution":{"iopub.status.busy":"2023-03-21T13:43:41.517368Z","iopub.execute_input":"2023-03-21T13:43:41.517623Z","iopub.status.idle":"2023-03-21T13:43:41.574558Z","shell.execute_reply.started":"2023-03-21T13:43:41.517595Z","shell.execute_reply":"2023-03-21T13:43:41.573430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import altair as alt\n\nalt.Chart(hist_data, width=1000).mark_bar().encode(\n    x=alt.X(\"num_cells:Q\"),\n    y=alt.Y(\"num_notebooks:Q\", scale=alt.Scale(type=\"log\"))  # Here the scale is applied\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T13:43:44.626770Z","iopub.execute_input":"2023-03-21T13:43:44.627037Z","iopub.status.idle":"2023-03-21T13:43:44.837805Z","shell.execute_reply.started":"2023-03-21T13:43:44.626989Z","shell.execute_reply":"2023-03-21T13:43:44.836871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_TRAIN = 10000\n\n\ndef read_notebook(path):\n    return (\n        pd.read_json(\n            path,\n            dtype={'cell_type': 'category', 'source': 'str'})\n        .assign(id=path.stem)\n        .rename_axis('cell_id')\n    )\n\n\npaths_train = list((data_dir / 'train').glob('*.json'))[:NUM_TRAIN]\nnotebooks_train = [\n    read_notebook(path) for path in tqdm(paths_train, desc='Train NBs')\n]\ndf = (\n    pd.concat(notebooks_train)\n    .set_index('id', append=True)\n    .swaplevel()\n    .sort_index(level='id', sort_remaining=False)\n)\n\ndf","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2023-03-21T09:37:35.842673Z","iopub.execute_input":"2023-03-21T09:37:35.842933Z","iopub.status.idle":"2023-03-21T09:40:39.366183Z","shell.execute_reply.started":"2023-03-21T09:37:35.842894Z","shell.execute_reply":"2023-03-21T09:40:39.365530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series = df.reset_index().groupby(\"id\").cell_id.nunique().to_frame(\"num_cells\")\nhist_data = series.num_cells.value_counts().to_frame(\"num_notebooks\").reset_index().rename(columns={\"index\": \"num_cells\"})","metadata":{"execution":{"iopub.status.busy":"2023-03-21T09:48:45.314993Z","iopub.execute_input":"2023-03-21T09:48:45.315293Z","iopub.status.idle":"2023-03-21T09:48:45.646024Z","shell.execute_reply.started":"2023-03-21T09:48:45.315263Z","shell.execute_reply":"2023-03-21T09:48:45.644935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"hist_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-21T09:51:24.388882Z","iopub.execute_input":"2023-03-21T09:51:24.392153Z","iopub.status.idle":"2023-03-21T09:51:24.410170Z","shell.execute_reply.started":"2023-03-21T09:51:24.392090Z","shell.execute_reply":"2023-03-21T09:51:24.409065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import altair as alt\n\nalt.Chart(hist_data, width=1000).mark_bar().encode(\n    x=alt.X(\"num_cells:Q\"),\n    y=alt.Y(\"num_notebooks:Q\", scale=alt.Scale(type=\"log\"))  # Here the scale is applied\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T09:51:51.130306Z","iopub.execute_input":"2023-03-21T09:51:51.130661Z","iopub.status.idle":"2023-03-21T09:51:51.166655Z","shell.execute_reply.started":"2023-03-21T09:51:51.130626Z","shell.execute_reply":"2023-03-21T09:51:51.165838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import altair as alt\n\nalt.Chart(series[:5000], width=1000).mark_bar().encode(\n    x=alt.X(\"num_cells:Q\"),\n    y=alt.Y(\"count()\", scale=alt.Scale(type=\"log\"))  # Here the scale is applied\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T09:49:41.237515Z","iopub.execute_input":"2023-03-21T09:49:41.237831Z","iopub.status.idle":"2023-03-21T09:49:41.290611Z","shell.execute_reply.started":"2023-03-21T09:49:41.237794Z","shell.execute_reply":"2023-03-21T09:49:41.289754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import altair as alt\n\nalt.Chart(series[5000:], width=1000).mark_bar().encode(\n    x=alt.X(\"num_cells:Q\"),\n    y=alt.Y(\"count()\", scale=alt.Scale(type=\"log\"))  # Here the scale is applied\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T09:49:23.219815Z","iopub.execute_input":"2023-03-21T09:49:23.220115Z","iopub.status.idle":"2023-03-21T09:49:23.277699Z","shell.execute_reply.started":"2023-03-21T09:49:23.220084Z","shell.execute_reply":"2023-03-21T09:49:23.276550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each notebook has all the code cells given first with the markdown cells following. The code cells are in the correct relative order, while the markdown cells are shuffled. In the next section, we'll see how to recover the correct orderings for notebooks in the training set.","metadata":{}},{"cell_type":"code","source":"# Get an example notebook\nnb_id = df.index.unique('id')[6]\nprint('Notebook:', nb_id)\n\nprint(\"The disordered notebook:\")\nnb = df.loc[nb_id, :]\ndisplay(nb)\nprint()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Your task in this competition is to predict the correct order of the notebook cells, both code and markdown. Since you're given the relative ordering of the code cells among themselves, you could also think of this as predicting where the markdown cells should be placed among the code cells.\n\nFor example, a disordered notebook might be:\n```\ncode_1\ncode_2\ncode_3\nmarkdown_1\nmarkdown_2\n```\nand the correctly ordered notebook might be:\n```\ncode_1\nmarkdown_2\ncode_2\ncode_3\nmarkdown_1\n```\nThe markdown cells can be in any order, but you would never see `code_2` before `code_1`, for instance.","metadata":{}},{"cell_type":"markdown","source":"# Ordering the Cells #","metadata":{}},{"cell_type":"markdown","source":"In the `train_orders.csv` file we have, for notebooks in the training set, the correct ordering of cells in terms of the cell ids.","metadata":{}},{"cell_type":"code","source":"df_orders = pd.read_csv(\n    data_dir / 'train_orders.csv',\n    index_col='id',\n    squeeze=True,\n).str.split()  # Split the string representation of cell_ids into a list\n\ndf_orders","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the correct order\ncell_order = df_orders.loc[nb_id]\n\nprint(\"The ordered notebook:\")\nnb.loc[cell_order, :]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The correct numeric position of a cell we will call the **rank** of the cell. We can find the ranks of the cells within a notebook by referencing the true ordering of cell ids as given in `train_orders.csv`.","metadata":{}},{"cell_type":"code","source":"def get_ranks(base, derived):\n    return [base.index(d) for d in derived]\n\ncell_ranks = get_ranks(cell_order, list(nb.index))\nnb.insert(0, 'rank', cell_ranks)\n\nnb","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Sorting a notebook by the cell ranks is another way to order the notebook.","metadata":{}},{"cell_type":"code","source":"from pandas.testing import assert_frame_equal\n\nassert_frame_equal(nb.loc[cell_order, :], nb.sort_values('rank'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The algorithm we'll be using for our baseline model uses the cell ranks as the target, so let's create a dataframe of the ranks for each notebook.","metadata":{}},{"cell_type":"code","source":"df_orders_ = df_orders.to_frame().join(\n    df.reset_index('cell_id').groupby('id')['cell_id'].apply(list),\n    how='right',\n)\n\nranks = {}\nfor id_, cell_order, cell_id in df_orders_.itertuples():\n    ranks[id_] = {'cell_id': cell_id, 'rank': get_ranks(cell_order, cell_id)}\n\ndf_ranks = (\n    pd.DataFrame\n    .from_dict(ranks, orient='index')\n    .rename_axis('id')\n    .apply(pd.Series.explode)\n    .set_index('cell_id', append=True)\n)\n\ndf_ranks","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Splits #","metadata":{}},{"cell_type":"markdown","source":"The `df_ancestors.csv` file identifies groups of notebooks derived from a common origin, that is, notebooks belonging to the same forking tree.","metadata":{}},{"cell_type":"code","source":"df_ancestors = pd.read_csv(data_dir / 'train_ancestors.csv', index_col='id')\ndf_ancestors","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To prevent leakage, the test set has no notebook with an ancestor in the training set. We therefore form a validation split using `ancestor_id` as a grouping factor.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GroupShuffleSplit\n\nNVALID = 0.1  # size of validation set\n\nsplitter = GroupShuffleSplit(n_splits=1, test_size=NVALID, random_state=0)\n\n# Split, keeping notebooks with a common origin (ancestor_id) together\nids = df.index.unique('id')\nancestors = df_ancestors.loc[ids, 'ancestor_id']\nids_train, ids_valid = next(splitter.split(ids, groups=ancestors))\nids_train, ids_valid = ids[ids_train], ids[ids_valid]\n\ndf_train = df.loc[ids_train, :]\ndf_valid = df.loc[ids_valid, :]","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering #\n\nLet's generate [tf-idf features](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html#sklearn.feature_extraction.text.TfidfTransformer) to use with our ranking model. These features will help our model learn what kinds of words tend to occur most often at various positions within a notebook.","metadata":{}},{"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer\n\n# Training set\ntfidf = TfidfVectorizer(min_df=0.01)\nX_train = tfidf.fit_transform(df_train['source'].astype(str))\n# Rank of each cell within the notebook\ny_train = df_ranks.loc[ids_train].to_numpy()\n# Number of cells in each notebook\ngroups = df_ranks.loc[ids_train].groupby('id').size().to_numpy()","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's add the code cell ordering as a feature. We'll append a column that enumerates the code cells in the correct order, like `1, 2, 3, 4, ...`, while having the dummy value `0` for all markdown cells. This feature will help the model learn to put the code cells in the correct order.","metadata":{}},{"cell_type":"code","source":"# Add code cell ordering\nX_train = sparse.hstack((\n    X_train,\n    np.where(\n        df_train['cell_type'] == 'code',\n        df_train.groupby(['id', 'cell_type']).cumcount().to_numpy() + 1,\n        0,\n    ).reshape(-1, 1)\n))\nprint(X_train.shape)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train #","metadata":{}},{"cell_type":"markdown","source":"We'll use the ranking algorithm provided by XGBoost.","metadata":{}},{"cell_type":"code","source":"from xgboost import XGBRanker\n\nmodel = XGBRanker(\n    min_child_weight=10,\n    subsample=0.5,\n    tree_method='hist',\n)\nmodel.fit(X_train, y_train, group=groups)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluate #","metadata":{}},{"cell_type":"markdown","source":"Now let's see how well our model learned to order Kaggle notebook cells. We'll evaluate predictions on the validation set with a variant of the Kendall tau correlation.","metadata":{}},{"cell_type":"markdown","source":"## Validation set ##","metadata":{}},{"cell_type":"markdown","source":"First we'll create features for the validation set just like we did for the training set.","metadata":{}},{"cell_type":"code","source":"# Validation set\nX_valid = tfidf.transform(df_valid['source'].astype(str))\n# The metric uses cell ids\ny_valid = df_orders.loc[ids_valid]\n\nX_valid = sparse.hstack((\n    X_valid,\n    np.where(\n        df_valid['cell_type'] == 'code',\n        df_valid.groupby(['id', 'cell_type']).cumcount().to_numpy() + 1,\n        0,\n    ).reshape(-1, 1)\n))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we'll use the model to predict the rank of each cell within its notebook and then convert these ranks into a list of ordered cell ids.","metadata":{}},{"cell_type":"code","source":"y_pred = pd.DataFrame({'rank': model.predict(X_valid)}, index=df_valid.index)\ny_pred = (\n    y_pred\n    .sort_values(['id', 'rank'])  # Sort the cells in each notebook by their rank.\n                                  # The cell_ids are now in the order the model predicted.\n    .reset_index('cell_id')  # Convert the cell_id index into a column.\n    .groupby('id')['cell_id'].apply(list)  # Group the cell_ids for each notebook into a list.\n)\ny_pred.head(10)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's examine a notebook to see how the model did.","metadata":{}},{"cell_type":"code","source":"nb_id = df_valid.index.get_level_values('id').unique()[8]\n\ndisplay(df.loc[nb_id])\ndisplay(df.loc[nb_id].loc[y_pred.loc[nb_id]])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Metric ##\n\nThis competition uses a variant of the [Kendall tau correlation](https://www.kaggle.com/competitions/AI4Code/overview/evaluation), which will measure how close to the correct order our predicted orderings are. See this notebook for more on this metric: [Competition Metric - Kendall Tau Correlation](https://www.kaggle.com/code/ryanholbrook/competition-metric-kendall-tau-correlation/notebook).","metadata":{}},{"cell_type":"code","source":"from bisect import bisect\n\n\ndef count_inversions(a):\n    inversions = 0\n    sorted_so_far = []\n    for i, u in enumerate(a):\n        j = bisect(sorted_so_far, u)\n        inversions += i - j\n        sorted_so_far.insert(j, u)\n    return inversions\n\n\ndef kendall_tau(ground_truth, predictions):\n    total_inversions = 0\n    total_2max = 0  # twice the maximum possible inversions across all instances\n    for gt, pred in zip(ground_truth, predictions):\n        ranks = [gt.index(x) for x in pred]  # rank predicted order in terms of ground truth\n        total_inversions += count_inversions(ranks)\n        n = len(gt)\n        total_2max += n * (n - 1)\n    return 1 - 4 * total_inversions / total_2max","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's test the metric with a dummy submission created from the ids of the shuffled notebooks.","metadata":{}},{"cell_type":"code","source":"y_dummy = df_valid.reset_index('cell_id').groupby('id')['cell_id'].apply(list)\nkendall_tau(y_valid, y_dummy)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Comparing this to the score on the predictions, we can see that our model was indeed able to improve the cell ordering somewhat.","metadata":{}},{"cell_type":"code","source":"kendall_tau(y_valid, y_pred)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission #","metadata":{}},{"cell_type":"markdown","source":"To create a submission for this competition, we'll apply our model to the notebooks in the test set. Note that this is a **Code Competition**, which means that the test data we see here is only a small sample. When we submit our notebook for scoring, this example data will be replaced with the full test set of about 20,000 notebooks.","metadata":{}},{"cell_type":"markdown","source":"First we load the data.","metadata":{}},{"cell_type":"code","source":"paths_test = list((data_dir / 'test').glob('*.json'))\nnotebooks_test = [\n    read_notebook(path) for path in tqdm(paths_test, desc='Test NBs')\n]\ndf_test = (\n    pd.concat(notebooks_test)\n    .set_index('id', append=True)\n    .swaplevel()\n    .sort_index(level='id', sort_remaining=False)\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then create the tf-idf and code cell features.","metadata":{}},{"cell_type":"code","source":"X_test = tfidf.transform(df_test['source'].astype(str))\nX_test = sparse.hstack((\n    X_test,\n    np.where(\n        df_test['cell_type'] == 'code',\n        df_test.groupby(['id', 'cell_type']).cumcount().to_numpy() + 1,\n        0,\n    ).reshape(-1, 1)\n))","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And then create predictions on the test set.","metadata":{}},{"cell_type":"code","source":"y_infer = pd.DataFrame({'rank': model.predict(X_test)}, index=df_test.index)\ny_infer = y_infer.sort_values(['id', 'rank']).reset_index('cell_id').groupby('id')['cell_id'].apply(list)\ny_infer","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `sample_submission.csv` file shows what a correctly formatted submission must look like. We'll just use it as a visual check, but you might like to directly modify the values of sample submission instead. (This would help prevent failed submissions due to missing notebook ids or incorrectly named columns, for instance.)","metadata":{}},{"cell_type":"code","source":"y_sample = pd.read_csv(data_dir / 'sample_submission.csv', index_col='id', squeeze=True)\ny_sample","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that a correctly formatted submission needs the index named `id` and the column of cell orders named `cell_order`. Moreover, we need to convert the list of cell ids into a space-delimited string of cell ids.","metadata":{}},{"cell_type":"code","source":"y_submit = (\n    y_infer\n    .apply(' '.join)  # list of ids -> string of ids\n    .rename_axis('id')\n    .rename('cell_order')\n)\ny_submit","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And finally we'll write out the formatted submissions to a file `submission.csv`. When we submit our notebook, it will be rerun on the full test data to create the submission file that's actually scored.","metadata":{}},{"cell_type":"code","source":"y_submit.to_csv('submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}