{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nimport os\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.style as style\nstyle.use('fivethirtyeight')\nfrom pathlib import Path\n\npd.options.display.width = 180\npd.options.display.max_colwidth = 120\npd.options.display.max_rows = None\n","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.054235,"end_time":"2022-05-22T09:28:19.055212","exception":false,"start_time":"2022-05-22T09:28:19.000977","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:40:01.138298Z","iopub.execute_input":"2022-05-22T09:40:01.139033Z","iopub.status.idle":"2022-05-22T09:40:01.168560Z","shell.execute_reply.started":"2022-05-22T09:40:01.138936Z","shell.execute_reply":"2022-05-22T09:40:01.167850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fun start: Find your notebooks and references\n\nJust taking the quick lane by loading Rob Mulla's Parquet files.","metadata":{"papermill":{"duration":0.034456,"end_time":"2022-05-22T09:28:19.124791","exception":false,"start_time":"2022-05-22T09:28:19.090335","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\ntrain = pd.read_parquet('../input/ai4code-parquet-tabular/train_all.parquet')\ntrain.head()","metadata":{"papermill":{"duration":26.418756,"end_time":"2022-05-22T09:28:45.578201","exception":false,"start_time":"2022-05-22T09:28:19.159445","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:40:01.170193Z","iopub.execute_input":"2022-05-22T09:40:01.170411Z","iopub.status.idle":"2022-05-22T09:40:29.084648Z","shell.execute_reply.started":"2022-05-22T09:40:01.170384Z","shell.execute_reply":"2022-05-22T09:40:29.084063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So I was curious which of my notebooks are actually in train. Below I made a function that you can use to find them easily.","metadata":{"papermill":{"duration":0.035874,"end_time":"2022-05-22T09:28:45.652578","exception":false,"start_time":"2022-05-22T09:28:45.616704","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def find_notebooks(text_piece, head_cells = 8, return_df = True):\n    notebooks = pd.DataFrame()\n    df = train[train.source.str.contains(text_piece)]\n    ids = df.id.unique()\n    for i in ids:\n        df1 = train.query('id == @i').iloc[:head_cells,:]\n        notebooks = notebooks.append(df1)\n    if len(notebooks) >0:\n        print(f'This piece of text was found {len(df)} times in {len(ids)} unique notebooks, with ids {ids}')\n    else:\n        print('Text not found in any notebook')\n    if return_df and len(df) >0:\n        print(f'The dataframe below prints the first {head_cells} cells of each notebook')\n        return notebooks","metadata":{"papermill":{"duration":0.046755,"end_time":"2022-05-22T09:28:45.735362","exception":false,"start_time":"2022-05-22T09:28:45.688607","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:40:29.086146Z","iopub.execute_input":"2022-05-22T09:40:29.086552Z","iopub.status.idle":"2022-05-22T09:40:29.092606Z","shell.execute_reply.started":"2022-05-22T09:40:29.086519Z","shell.execute_reply":"2022-05-22T09:40:29.092073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you take a piece of text out of your notebook and seems \"unique enough\", you should find it if the notebook is included in the train set. The piece of text below is from https://www.kaggle.com/code/erikbruin/riiid-comprehensive-eda-baseline\n\nOf course you can still get multiple hits if somebody forked your notebook and made it public (hopefully with significant changes). There actually are 7 public forks, but I guess they forked an older version that did not contain this piece of text yet or the forks are just not included in train.","metadata":{"papermill":{"duration":0.035263,"end_time":"2022-05-22T09:28:45.806324","exception":false,"start_time":"2022-05-22T09:28:45.771061","status":"completed"},"tags":[]}},{"cell_type":"code","source":"find_notebooks(\"So we should realize that example_test.csv really is just an example.\")\n","metadata":{"papermill":{"duration":6.715557,"end_time":"2022-05-22T09:28:52.557727","exception":false,"start_time":"2022-05-22T09:28:45.842170","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:40:29.093779Z","iopub.execute_input":"2022-05-22T09:40:29.093979Z","iopub.status.idle":"2022-05-22T09:40:35.611475Z","shell.execute_reply.started":"2022-05-22T09:40:29.093954Z","shell.execute_reply":"2022-05-22T09:40:35.610638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#just checking if this (bad) Dutch word is found somewhere\nfind_notebooks('mietje')","metadata":{"papermill":{"duration":5.185067,"end_time":"2022-05-22T09:28:57.780183","exception":false,"start_time":"2022-05-22T09:28:52.595116","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:40:35.613431Z","iopub.execute_input":"2022-05-22T09:40:35.613668Z","iopub.status.idle":"2022-05-22T09:40:40.690400Z","shell.execute_reply.started":"2022-05-22T09:40:35.613638Z","shell.execute_reply":"2022-05-22T09:40:40.689451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below, I am checking how often I have been referenced ('erikbruin' is always part of the url). I by the way found 2 more notebooks of mine via this route ('8c34197a9f9c1a' and'c1fd4f8cb2fd27'). In one of them I found a reference made by me to a dataset that I published, and in the other one I suggest also reading another notebook published by me ;-).","metadata":{"papermill":{"duration":0.038058,"end_time":"2022-05-22T09:28:57.857324","exception":false,"start_time":"2022-05-22T09:28:57.819266","status":"completed"},"tags":[]}},{"cell_type":"code","source":"find_notebooks('erikbruin', return_df = False)","metadata":{"papermill":{"duration":20.15369,"end_time":"2022-05-22T09:29:18.048022","exception":false,"start_time":"2022-05-22T09:28:57.894332","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:40:40.691730Z","iopub.execute_input":"2022-05-22T09:40:40.692023Z","iopub.status.idle":"2022-05-22T09:41:00.214082Z","shell.execute_reply.started":"2022-05-22T09:40:40.691980Z","shell.execute_reply":"2022-05-22T09:41:00.213399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA","metadata":{"papermill":{"duration":0.03853,"end_time":"2022-05-22T09:29:18.124248","exception":false,"start_time":"2022-05-22T09:29:18.085718","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"As you can see, there is a notebook with over a 1000 cells! I wonder how tidy this one is (perhaps a new cells for every single line? Notebooks of just 2 cells make sense to me: This is likely a (long) script put into one cell.","metadata":{"papermill":{"duration":0.036701,"end_time":"2022-05-22T09:29:18.198159","exception":false,"start_time":"2022-05-22T09:29:18.161458","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cells = train.groupby('id').size()\nprint(f\"\"\"The average number of cells per notebook is {round(cells.mean(),1)}{os.linesep}\nThe longest notebook has {cells.max()} cells{os.linesep}\nThe shortest notebook has {cells.min()} cells\"\"\")","metadata":{"papermill":{"duration":0.883499,"end_time":"2022-05-22T09:29:19.118644","exception":false,"start_time":"2022-05-22T09:29:18.235145","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:00.215510Z","iopub.execute_input":"2022-05-22T09:41:00.216017Z","iopub.status.idle":"2022-05-22T09:41:01.047306Z","shell.execute_reply.started":"2022-05-22T09:41:00.215975Z","shell.execute_reply":"2022-05-22T09:41:01.046439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's now look at the length of cells.","metadata":{"papermill":{"duration":0.037224,"end_time":"2022-05-22T09:29:19.193593","exception":false,"start_time":"2022-05-22T09:29:19.156369","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#new column\ntrain['cell_chars'] = train.source.str.len()\ntrain.head()","metadata":{"papermill":{"duration":4.550252,"end_time":"2022-05-22T09:29:23.781177","exception":false,"start_time":"2022-05-22T09:29:19.230925","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:01.048457Z","iopub.execute_input":"2022-05-22T09:41:01.050323Z","iopub.status.idle":"2022-05-22T09:41:05.422811Z","shell.execute_reply.started":"2022-05-22T09:41:01.050276Z","shell.execute_reply":"2022-05-22T09:41:05.421855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So what are the longest cell? Well, interestingly, they are all markdown cells where images have been inserted!","metadata":{"papermill":{"duration":0.040195,"end_time":"2022-05-22T09:29:23.862001","exception":false,"start_time":"2022-05-22T09:29:23.821806","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train.sort_values(by = \"cell_chars\", ascending = False).head()","metadata":{"papermill":{"duration":7.22005,"end_time":"2022-05-22T09:29:31.120978","exception":false,"start_time":"2022-05-22T09:29:23.900928","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:05.424548Z","iopub.execute_input":"2022-05-22T09:41:05.425111Z","iopub.status.idle":"2022-05-22T09:41:12.011143Z","shell.execute_reply.started":"2022-05-22T09:41:05.425070Z","shell.execute_reply":"2022-05-22T09:41:12.010232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, what if we just look at code cells?","metadata":{"papermill":{"duration":0.039786,"end_time":"2022-05-22T09:29:31.200812","exception":false,"start_time":"2022-05-22T09:29:31.161026","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train.query('cell_type == \"code\"').sort_values(by = \"cell_chars\", ascending = False).head()","metadata":{"papermill":{"duration":5.858363,"end_time":"2022-05-22T09:29:37.098636","exception":false,"start_time":"2022-05-22T09:29:31.240273","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:12.012291Z","iopub.execute_input":"2022-05-22T09:41:12.012528Z","iopub.status.idle":"2022-05-22T09:41:17.379514Z","shell.execute_reply.started":"2022-05-22T09:41:12.012498Z","shell.execute_reply":"2022-05-22T09:41:17.378875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So the fourth one seems the original notebook here, with 3 forks (same anchestor_id). What does this notebook look like? Well, this what I expected; almost the entire code in one cell ;-)","metadata":{"papermill":{"duration":0.040058,"end_time":"2022-05-22T09:29:37.179241","exception":false,"start_time":"2022-05-22T09:29:37.139183","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train.query('id == \"fd4199afd8d8b2\"')","metadata":{"papermill":{"duration":0.316517,"end_time":"2022-05-22T09:29:37.536282","exception":false,"start_time":"2022-05-22T09:29:37.219765","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:17.381763Z","iopub.execute_input":"2022-05-22T09:41:17.382096Z","iopub.status.idle":"2022-05-22T09:41:17.642464Z","shell.execute_reply.started":"2022-05-22T09:41:17.382068Z","shell.execute_reply":"2022-05-22T09:41:17.641880Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's also have a look at the percent of markdown vs code. As you can see, some notebooks are almost entirely markdown only. Those seem hard to predict.","metadata":{"papermill":{"duration":0.040257,"end_time":"2022-05-22T09:29:37.616545","exception":false,"start_time":"2022-05-22T09:29:37.576288","status":"completed"},"tags":[]}},{"cell_type":"code","source":"type_counts = train.groupby(['id', 'cell_type'], as_index = False)['cell_type'].size()\ntype_counts = type_counts.pivot_table('size', 'id', 'cell_type')\ntype_counts['percent_md'] = round(type_counts.markdown/(type_counts.markdown + type_counts.code),2)\ntype_counts = type_counts.sort_values(by = \"percent_md\", ascending = False)\ntype_counts.head()","metadata":{"papermill":{"duration":2.130327,"end_time":"2022-05-22T09:29:39.787250","exception":false,"start_time":"2022-05-22T09:29:37.656923","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:17.643340Z","iopub.execute_input":"2022-05-22T09:41:17.644036Z","iopub.status.idle":"2022-05-22T09:41:19.671362Z","shell.execute_reply.started":"2022-05-22T09:41:17.644002Z","shell.execute_reply":"2022-05-22T09:41:19.670522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's now see what the test set looks like. Since it's very small, would Kaggle have taken a few \"representative\" notebooks? Cannot take the quick lane here (as Rob Mulla only made Parquet files for train; makes sense as these are the ones that take long to load).","metadata":{"papermill":{"duration":0.040914,"end_time":"2022-05-22T09:29:39.868810","exception":false,"start_time":"2022-05-22T09:29:39.827896","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#taking the code to load json published by the organisers in the getting started notebook\n\ndata_dir = Path('../input/AI4Code')\n\ndef read_notebook(path):\n    return (\n        pd.read_json(\n            path,\n            dtype={'cell_type': 'category', 'source': 'str'})\n        .assign(id=path.stem)\n        .rename_axis('cell_id')\n    )\n\n\npaths_test = list((data_dir / 'test').glob('*.json'))\nnotebooks_test = [\n    read_notebook(path) for path in tqdm(paths_test, desc='Train NBs')\n]\ntest = (\n    pd.concat(notebooks_test)\n    .set_index('id', append=True)\n    .swaplevel()\n    .sort_index(level='id', sort_remaining=False)\n)\n\ntest = test.reset_index()","metadata":{"papermill":{"duration":0.117862,"end_time":"2022-05-22T09:29:40.027892","exception":false,"start_time":"2022-05-22T09:29:39.910030","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:19.672297Z","iopub.execute_input":"2022-05-22T09:41:19.672487Z","iopub.status.idle":"2022-05-22T09:41:19.735446Z","shell.execute_reply.started":"2022-05-22T09:41:19.672462Z","shell.execute_reply":"2022-05-22T09:41:19.734614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the description of the competition is stated:\n*You are challenged to reconstruct the order of markdown cells in a given notebook based on the order of the code cells, demonstrating comprehension of which natural language references which code.*\n\nThis confused me a little initially, but in the discussion I found the answer:\n*To clarify, all of the cells need to be ordered, both code and markdown. It's just that you are given the correct relative order of the code cells among themselves. You still need to figure out the correct overall order -- where the markdown cells should be placed among the code cells.*\n\nOk, I finally get it. So in the notebook below with 9 code cells and one markdown cell, we just need to insert the single markdown cell into the ordered list of code cell 1-9!","metadata":{"papermill":{"duration":0.041815,"end_time":"2022-05-22T09:29:40.111612","exception":false,"start_time":"2022-05-22T09:29:40.069797","status":"completed"},"tags":[]}},{"cell_type":"code","source":"type_counts_test = test.groupby(['id', 'cell_type'], as_index = False)['cell_type'].size()\ntype_counts_test = type_counts_test.pivot_table('size', 'id', 'cell_type')\ntype_counts_test","metadata":{"papermill":{"duration":0.069459,"end_time":"2022-05-22T09:29:40.223652","exception":false,"start_time":"2022-05-22T09:29:40.154193","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:19.736542Z","iopub.execute_input":"2022-05-22T09:41:19.736786Z","iopub.status.idle":"2022-05-22T09:41:19.758280Z","shell.execute_reply.started":"2022-05-22T09:41:19.736756Z","shell.execute_reply":"2022-05-22T09:41:19.757749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The 3rd seems most important for the public score. Which notebook is that one with lots of cells in the test set? Well I don't know as the function that I made only works the other way around. However, it's easy to see that it's a Titanic notebook when looking at the source column.\n\n# The Metric; Kendall Tau\n\nBelow, I am selecting one notebook in train with one code cell and a markdown cell.","metadata":{"papermill":{"duration":0.042628,"end_time":"2022-05-22T09:29:40.309971","exception":false,"start_time":"2022-05-22T09:29:40.267343","status":"completed"},"tags":[]}},{"cell_type":"code","source":"nb = type_counts.query('code ==1 and markdown == 1').iloc[[0]].index.values[0]\nnb1 = train.query('id == @nb')\nnb1","metadata":{"papermill":{"duration":0.325433,"end_time":"2022-05-22T09:29:40.678116","exception":false,"start_time":"2022-05-22T09:29:40.352683","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:19.759449Z","iopub.execute_input":"2022-05-22T09:41:19.759872Z","iopub.status.idle":"2022-05-22T09:41:20.030563Z","shell.execute_reply.started":"2022-05-22T09:41:19.759841Z","shell.execute_reply":"2022-05-22T09:41:20.029713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, I am putting together a Pandas series as required as the input of the function given in the Getting Started with AI4Code notebook, and and reverse the order in the predictions.","metadata":{"papermill":{"duration":0.04263,"end_time":"2022-05-22T09:29:40.763588","exception":false,"start_time":"2022-05-22T09:29:40.720958","status":"completed"},"tags":[]}},{"cell_type":"code","source":"gt1 = nb1.groupby('id')['cell'].apply(list)\ngt1","metadata":{"papermill":{"duration":0.054808,"end_time":"2022-05-22T09:29:40.861169","exception":false,"start_time":"2022-05-22T09:29:40.806361","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:20.031618Z","iopub.execute_input":"2022-05-22T09:41:20.031879Z","iopub.status.idle":"2022-05-22T09:41:20.040156Z","shell.execute_reply.started":"2022-05-22T09:41:20.031850Z","shell.execute_reply":"2022-05-22T09:41:20.039319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred1 = gt1.copy()\npred1[0] = ['59c59076', 'd04e6a45'] #reversing the order\npred1","metadata":{"papermill":{"duration":0.053184,"end_time":"2022-05-22T09:29:40.957663","exception":false,"start_time":"2022-05-22T09:29:40.904479","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:20.041364Z","iopub.execute_input":"2022-05-22T09:41:20.041847Z","iopub.status.idle":"2022-05-22T09:41:20.053460Z","shell.execute_reply.started":"2022-05-22T09:41:20.041801Z","shell.execute_reply":"2022-05-22T09:41:20.052912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In code cell below, you can find the kendall_tau function from the organizers. I adjusted the function a little to be able to print more information that just the tau score, and I added a little print function.","metadata":{"papermill":{"duration":0.044273,"end_time":"2022-05-22T09:29:41.046774","exception":false,"start_time":"2022-05-22T09:29:41.002501","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from bisect import bisect\n\n\n# Actually O(N^2), but fast in practice for our data\ndef count_inversions(a):\n    inversions = 0\n    sorted_so_far = []\n    for i, u in enumerate(a):  # O(N)\n        j = bisect(sorted_so_far, u)  # O(log N)\n        inversions += i - j\n        sorted_so_far.insert(j, u)  # O(N)\n    return inversions\n\ndef kendall_tau(ground_truth, predictions, print_2max = False):\n    total_inversions = 0  # total inversions in predicted ranks across all instances\n    total_2max = 0  # maximum possible inversions across all instances\n    for gt, pred in zip(ground_truth, predictions):\n        ranks = [gt.index(x) for x in pred]  # rank predicted order in terms of ground truth\n        total_inversions += count_inversions(ranks)\n        n = len(gt)\n        total_2max += n * (n - 1)\n    if print_2max == False:\n        return 1 - 4 * total_inversions / total_2max, total_inversions\n    else:\n        return 1 - 4 * total_inversions / total_2max, total_inversions, total_2max\n\ndef print_tau(gt, pred, print_2max = False):\n    if print_2max:\n        tau, inversions, total_2max = kendall_tau(gt, pred, print_2max = True)\n        print(f' tau is {round(tau, 2)}, number of inversions is {inversions}, maximum possible inversions is {total_2max}')\n    else:\n        tau, inversions = kendall_tau(gt, pred, print_2max = False)\n        print(f' tau is {round(tau, 2)}, number of inversions is {inversions}')","metadata":{"_kg_hide-input":false,"papermill":{"duration":0.057856,"end_time":"2022-05-22T09:29:41.149561","exception":false,"start_time":"2022-05-22T09:29:41.091705","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:20.054479Z","iopub.execute_input":"2022-05-22T09:41:20.054824Z","iopub.status.idle":"2022-05-22T09:41:20.065241Z","shell.execute_reply.started":"2022-05-22T09:41:20.054782Z","shell.execute_reply":"2022-05-22T09:41:20.064486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Of course predicting the wrong order of just 2 cells gives us the results as printed below.","metadata":{"papermill":{"duration":0.044003,"end_time":"2022-05-22T09:29:41.238259","exception":false,"start_time":"2022-05-22T09:29:41.194256","status":"completed"},"tags":[]}},{"cell_type":"code","source":"print_tau(gt1, pred1)","metadata":{"papermill":{"duration":0.051864,"end_time":"2022-05-22T09:29:41.334556","exception":false,"start_time":"2022-05-22T09:29:41.282692","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:20.066364Z","iopub.execute_input":"2022-05-22T09:41:20.066573Z","iopub.status.idle":"2022-05-22T09:41:20.079639Z","shell.execute_reply.started":"2022-05-22T09:41:20.066546Z","shell.execute_reply":"2022-05-22T09:41:20.078767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What happens to the score if I take a notebook with one markdown cell followed by 4 code cells, and switch the first 2 cells?","metadata":{"papermill":{"duration":0.044228,"end_time":"2022-05-22T09:29:41.423204","exception":false,"start_time":"2022-05-22T09:29:41.378976","status":"completed"},"tags":[]}},{"cell_type":"code","source":"nb = type_counts.query('code ==4 and markdown == 1').iloc[[0]].index.values[0]\nnb2 = train.query('id == @nb')\nnb2","metadata":{"papermill":{"duration":0.326646,"end_time":"2022-05-22T09:29:41.796102","exception":false,"start_time":"2022-05-22T09:29:41.469456","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:20.080956Z","iopub.execute_input":"2022-05-22T09:41:20.081754Z","iopub.status.idle":"2022-05-22T09:41:20.354576Z","shell.execute_reply.started":"2022-05-22T09:41:20.081692Z","shell.execute_reply":"2022-05-22T09:41:20.353772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, the number of inversions is of course the same but the effect on Tau is much less here while we are making kind of the same mistake.","metadata":{"papermill":{"duration":0.045376,"end_time":"2022-05-22T09:29:41.887000","exception":false,"start_time":"2022-05-22T09:29:41.841624","status":"completed"},"tags":[]}},{"cell_type":"code","source":"gt2 = nb2.groupby('id')['cell'].apply(list)\npred2 = gt2.copy()\npred2[0] = ['e2eb8c27','8aa62aa6', 'e1b55174', 'd1dd08ac', '0d9a73d3'] #reversing the order of the first 2 cells\nprint_tau(gt2, pred2)","metadata":{"papermill":{"duration":0.057404,"end_time":"2022-05-22T09:29:41.990458","exception":false,"start_time":"2022-05-22T09:29:41.933054","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:20.355870Z","iopub.execute_input":"2022-05-22T09:41:20.356355Z","iopub.status.idle":"2022-05-22T09:41:20.364438Z","shell.execute_reply.started":"2022-05-22T09:41:20.356311Z","shell.execute_reply":"2022-05-22T09:41:20.363507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What if we put both notebooks through the score function? Is our score now the simple average of the scores of both notebooks? No, fortunately not! If that were the case, getting the short notebooks wrong would be very costly. What the organizers do is combine both ranking orders and calculate one score. As you can see, maximum possible inversions of 22 comes out this time. This is n(n-1) for both notebooks; (2×1) + (5×4) = 22. Is this entirely correct? Well, yes and no. The organizers apply the formula correctly but in this case total_2max is not described correctly (maximum possible inversions across all instances). The maximum possible inversions across all instances should be the binominal coefficient, which is total_2max/2 (so 1 + 10 = 11 maximal total inversions in this case).","metadata":{"papermill":{"duration":0.04603,"end_time":"2022-05-22T09:29:42.083440","exception":false,"start_time":"2022-05-22T09:29:42.037410","status":"completed"},"tags":[]}},{"cell_type":"code","source":"gt = gt1.append(gt2)\npred = pred1.append(pred2)\nprint_tau(gt, pred, print_2max = True)","metadata":{"papermill":{"duration":0.057241,"end_time":"2022-05-22T09:29:42.185497","exception":false,"start_time":"2022-05-22T09:29:42.128256","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-22T09:41:20.365402Z","iopub.execute_input":"2022-05-22T09:41:20.365591Z","iopub.status.idle":"2022-05-22T09:41:20.376181Z","shell.execute_reply.started":"2022-05-22T09:41:20.365567Z","shell.execute_reply":"2022-05-22T09:41:20.375460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Please stay tuned!**","metadata":{"papermill":{"duration":0.046377,"end_time":"2022-05-22T09:29:42.279215","exception":false,"start_time":"2022-05-22T09:29:42.232838","status":"completed"},"tags":[]}}]}