The image above shows how the markdown text can be related to the code cell. We will need to leverage this relationship to determine the ordering of the notebook code/markdown cells.
\n\n
COMPETITION IMPACT STATEMENT\n\nAn understanding of the relationships between code and markdown could lend to fresh improvements across many aspects of AI-assisted development, such as the construction of better data filtering and preprocessing pipelines for model training, or automatic assessments of a notebook's readability.","metadata":{}},{"cell_type":"markdown","source":"
\n\n1.2 COMPETITION EVALUATION
\n\n---\n\n
GENERAL EVALUATION INFORMATION\n\nPredictions are evaluated by the Kendall tau correlation between predicted cell orders and ground truth cell orders accumulated across the entire collection of test set notebooks.\n* Let $𝑆$ be the number of swaps of adjacent entries needed to sort the predicted cell order into the ground truth cell order. \n* In the worst case, a predicted order for a notebook with $n$ cells will need $12n(n-1)$ swaps to sort.\n* We sum the number of swaps from your predicted cell order across the entire collection of test set notebooks, and similarly with the worst-case number of swaps. \n* Finally we compute the Kendall tau correlation as:\n\n$$\nK = 1 - 4 \\frac{\\sum_i S_{i}}{\\sum_i n_i(n_i - 1)}\n$$\n\n* You can find a Python implementation in this notebook – Competition Metric - Kendall Tau Correlation \n\n
\n \n---\n\n
\n\n
SUBMISSION FILE INFORMATION\n\nFor each id in the test set (representing a notebook), you must predict cell_order, the correct ordering of its cells in terms of the cell ids. \n\n
\n\nThe file should contain a header and have the following format:\n\n```\nid,cell_order\n0009d135ece78d,ddfd239c c6cd22db 1372ae9b ...\n0010483c12ba9b,54c7cab3 fe66203e 7844d5f8 ...\n0010a919d60e4f,aafc3d23 80e077ec b190ebb4 ...\n0028856e09c5b7,012c9d02 d22526d1 3ae7ece3 ...\netc.\n```\n\n
IS THIS A CODE COMPETITION?\n\nYES\n\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n1.3 DATASET OVERVIEW
\n\n---\n\n
GENERAL INFORMATION\n\nThe dataset for this competition comprises about 160,000 Jupyter notebooks published by the Kaggle community. \n* Jupyter notebooks are the tool of choice for many data scientists for their ability to tell a narrative with both code and natural language.\n* These two types of discourse are contained within cells of the notebook, and we refer to these cells as either code cells or markdown cells (markdown being the text formatting language used by Jupyter).\n\n
\n\n---\n\n
\n\nYour task is to predict the correct ordering of the cells in a given notebook whose markdown cells have been shuffled.\n\n
\n\n---\n\n
\n\nThe notebooks in this dataset have been selected and processed to ensure their suitability for the competition task. All notebooks:\n1. Have been published publicly on Kaggle under the Apache 2.0 open source license.\n2. Represent the most recently published version of the notebook.\n3. Contain at least one code cell and at least one markdown cell.\n4. Have code written in the Python language.\n5. Have had empty cells removed.\n\n
\n\n---\n\n
\n\nThis competition is broken into two stages\n* The first-stage test set contains notebooks from an approximately 90-day historical window of time.\n* The second-stage test set will contain a similar number of notebooks, collected from a future 90-day window of time. \n * This is necessary to prevent models from looking up the order of existing public notebooks. \n * The selection criteria for second-stage notebooks will be monitored for competition fairness purposes. \n * For example, it will exclude competition participants' own notebooks.\n\n
FILE INFORMATION\n\n\n**`train/`** \n- A folder comprising about 140,000 JSON files with the filenames corresponding to the id field in the csv files. \n- Each file contains the code and markdown cells of a Kaggle notebook. \n - The code cells are in their original (correct) order. \n - The markdown cells have been shuffled and placed after the code cells.\n\n
\n\n**`train_orders.csv`** \n- Gives the correct order of the cells for each notebook in the train/ folder.\n- **Columns**\n * **`id`**\n * The notebook in file {id}.json\n * **`cell_order`**\n * A space delimited list of the correct cell ordering given in terms of the order in {id}.json.\n \n
\n\n**`train_ancestors.csv`** \n- On Kaggle, a user may \"fork\" (that is, copy) the notebook of another user to create their own version. This file contains the forking history of notebooks in the training set. Note: There is no corresponding file for the test set.\n- **Columns**\n * **`ancestor_id`**\n * Identifies sets of notebooks that have a common origin or \"ancestor\". \n * As no notebook in the test set has an ancestor in the training set, you may find this field to be of use as a grouping factor when constructing validation splits.\n * **`parent_id`**\n * Indicates that some version of the notebook id was forked from some version of the notebook **`parent_id`**. \n * The notebook **`parent_id`** may or may not be present in the training data. \n * The parent may be missing because someone had forked a private notebook of their own, for instance.\n\n
\n\n**`test/`** \n- A few example notebooks from the test set. \n- The actual test set comprises about 20,000 notebooks in a format similar to the training set notebooks. \n- No notebook in the test set has an ancestor in the training set.\n\n
\n\n**`sample_submission.csv`**\n- A sample submission file in the correct format\n\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n2 SETUP ⤒
\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n2.1 ACCELERATOR DETECTION
\n\n---\n\nIn order to use **`TPU`**, we use **`TPUClusterResolver`** for the initialization which is necessary to connect to the remote cluster and initialize cloud TPUs. Let's go over two important points\n\n1. When using TPU on Kaggle, you don't need to specify arguments for **`TPUClusterResolver`**\n2. However, on **G**oogle **C**ompute **E**ngine (**GCE**), you will need to do the following:\n\n
\n\n```python\n# The name you gave to the TPU to use\nTPU_WORKER = 'my-tpu-name'\n\n# or you can also specify the grpc path directly\n# TPU_WORKER = 'grpc://xxx.xxx.xxx.xxx:8470'\n\n# The zone you chose when you created the TPU to use on GCP.\nZONE = 'us-east1-b'\n\n# The name of the GCP project where you created the TPU to use on GCP.\nPROJECT = 'my-tpu-project'\n\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=TPU_WORKER, zone=ZONE, project=PROJECT)\n```\n\n\n 🛑 WARNING:
- Although the Tensorflow documentation says it is the project name that should be provided for the argument `project`, it is actually the Project ID, that you should provide. This can be found on the GCP project dashboard page.
\n
\n\n","metadata":{}},{"cell_type":"code","source":"print(f\"\\n... ACCELERATOR SETUP STARTING ...\\n\")\n\n# Detect hardware, return appropriate distribution strategy\ntry:\n # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n TPU = tf.distribute.cluster_resolver.TPUClusterResolver() \nexcept ValueError:\n TPU = None\n\nif TPU:\n print(f\"\\n... RUNNING ON TPU - {TPU.master()}...\")\n tf.config.experimental_connect_to_cluster(TPU)\n tf.tpu.experimental.initialize_tpu_system(TPU)\n strategy = tf.distribute.experimental.TPUStrategy(TPU)\nelse:\n print(f\"\\n... RUNNING ON CPU/GPU ...\")\n # Yield the default distribution strategy in Tensorflow\n # --> Works on CPU and single GPU.\n strategy = tf.distribute.get_strategy() \n\n# What Is a Replica?\n# --> A single Cloud TPU device consists of FOUR chips, each of which has TWO TPU cores. \n# --> Therefore, for efficient utilization of Cloud TPU, a program should make use of each of the EIGHT (4x2) cores. \n# --> Each replica is essentially a copy of the training graph that is run on each core and \n# trains a mini-batch containing 1/8th of the overall batch size\nN_REPLICAS = strategy.num_replicas_in_sync\n \nprint(f\"... # OF REPLICAS: {N_REPLICAS} ...\\n\")\n\nprint(f\"\\n... ACCELERATOR SETUP COMPLTED ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:03:21.685381Z","iopub.execute_input":"2022-05-25T01:03:21.685634Z","iopub.status.idle":"2022-05-25T01:03:21.700064Z","shell.execute_reply.started":"2022-05-25T01:03:21.685598Z","shell.execute_reply":"2022-05-25T01:03:21.698870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n2.2 COMPETITION DATA ACCESS
\n\n---\n\nTPUs read data must be read directly from **G**oogle **C**loud **S**torage **(GCS)**. Kaggle provides a utility library – **`KaggleDatasets`** – which has a utility function **`.get_gcs_path`** that will allow us to access the location of our input datasets within **GCS**.
\n\n\n 📌 TIPS:
- If you have multiple datasets attached to the notebook, you should pass the name of a specific dataset to the `get_gcs_path()` function. In our case, the name of the dataset is the name of the directory the dataset is mounted within.
\n
","metadata":{}},{"cell_type":"code","source":"print(\"\\n... DATA ACCESS SETUP STARTED ...\\n\")\n\nif TPU:\n # Google Cloud Dataset path to training and validation images\n DATA_DIR = KaggleDatasets().get_gcs_path('AI4Code')\n save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\n load_locally = tf.saved_model.LoadOptions(experimental_io_device='/job:localhost')\nelse:\n # Local path to training and validation images\n DATA_DIR = \"/kaggle/input/AI4Code\"\n load_locally = save_locally = None\n\nprint(f\"\\n... DATA DIRECTORY PATH IS:\\n\\t--> {DATA_DIR}\")\n\nprint(f\"\\n... IMMEDIATE CONTENTS OF DATA DIRECTORY IS:\")\nfor file in tf.io.gfile.glob(os.path.join(DATA_DIR, \"*\")): print(f\"\\t--> {file}\")\n\nprint(\"\\n\\n... DATA ACCESS SETUP COMPLETED ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:03:21.701381Z","iopub.execute_input":"2022-05-25T01:03:21.701633Z","iopub.status.idle":"2022-05-25T01:03:21.714803Z","shell.execute_reply.started":"2022-05-25T01:03:21.701599Z","shell.execute_reply":"2022-05-25T01:03:21.714017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n2.3 LEVERAGING XLA OPTIMIZATIONS
\n\n---\n\n\n**XLA** (Accelerated Linear Algebra) is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes. **The results are improvements in speed and memory usage**.\n\n
\n\nWhen a TensorFlow program is run, all of the operations are executed individually by the TensorFlow executor. Each TensorFlow operation has a precompiled GPU/TPU kernel implementation that the executor dispatches to.\n\nXLA provides us with an alternative mode of running models: it compiles the TensorFlow graph into a sequence of computation kernels generated specifically for the given model. Because these kernels are unique to the model, they can exploit model-specific information for optimization.
\n\n\n 🛑 WARNING:
- XLA can not currently compile functions where dimensions are not inferrable: that is, if it's not possible to infer the dimensions of all tensors without running the entire computation
\n
\n\n\n 📌 NOTE:
- XLA compilation is only applied to code that is compiled into a graph (in TF2 that's only a code inside tf.function).
- The jit_compile API has must-compile semantics, i.e. either the entire function is compiled with XLA, or an errors.InvalidArgumentError exception is thrown)\n
\n\n","metadata":{}},{"cell_type":"code","source":"print(f\"\\n... XLA OPTIMIZATIONS STARTING ...\\n\")\n\nprint(f\"\\n... CONFIGURE JIT (JUST IN TIME) COMPILATION ...\\n\")\n# enable XLA optmizations (10% speedup when using @tf.function calls)\ntf.config.optimizer.set_jit(True)\n\nprint(f\"\\n... XLA OPTIMIZATIONS COMPLETED ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:03:21.716701Z","iopub.execute_input":"2022-05-25T01:03:21.716928Z","iopub.status.idle":"2022-05-25T01:03:21.722698Z","shell.execute_reply.started":"2022-05-25T01:03:21.716897Z","shell.execute_reply":"2022-05-25T01:03:21.721844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n2.4 BASIC DATA DEFINITIONS & INITIALIZATIONS
\n\n---\n","metadata":{}},{"cell_type":"code","source":"print(\"\\n... BASIC DATA SETUP STARTING ...\\n\\n\")\n\n# Get json directory paths\nTRAIN_JSON_DIR = os.path.join(DATA_DIR, \"train\")\nTEST_JSON_DIR = os.path.join(DATA_DIR, \"test\")\n\n# Get all json file paths\nTRAIN_JSON_PATHS = glob(os.path.join(TRAIN_JSON_DIR, \"*.json\"), recursive=True)\nTEST_JSON_PATHS = glob(os.path.join(TEST_JSON_DIR, \"*.json\"), recursive=True)\n\n# Get number of train and test files\nN_TRAIN = len(TRAIN_JSON_PATHS)\nN_TEST = len(TEST_JSON_PATHS)\n\n# Get CSV filepaths\nTRAIN_ANCESTORS_CSV = os.path.join(DATA_DIR, \"train_ancestors.csv\")\nTRAIN_ORDERS_CSV = os.path.join(DATA_DIR, \"train_orders.csv\")\nSS_CSV = os.path.join(DATA_DIR, \"sample_submission.csv\")\n\n# Convert CSV into dataframe\ntrain_ancestors_df = pd.read_csv(TRAIN_ANCESTORS_CSV)\ntrain_orders_df = pd.read_csv(TRAIN_ORDERS_CSV)\nss_df = pd.read_csv(SS_CSV)\n\nprint(\"\\n... TRAIN ANCESTORS DATAFRAME... \\n\")\ndisplay(train_ancestors_df)\n\nprint(\"\\n... TRAIN ORDERS DATAFRAME... \\n\")\ndisplay(train_orders_df)\n\nprint(\"\\n\\n\\n... ORIGINAL SUBMISSION DATAFRAME... \\n\")\ndisplay(ss_df)\n\n# For debugging purposes when the test set hasn't been substituted we will know\nDEBUG=len(ss_df)==4\n\n\nprint(\"\\n... BASIC DATA SETUP FINISHED ...\\n\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:03:21.724365Z","iopub.execute_input":"2022-05-25T01:03:21.724683Z","iopub.status.idle":"2022-05-25T01:03:25.950194Z","shell.execute_reply.started":"2022-05-25T01:03:21.724647Z","shell.execute_reply":"2022-05-25T01:03:25.949394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n2.5 UPDATE DATAFRAME WITH NOTEBOOK DATA
\n\n---\n\nThis takes a while. In the future (and during debugging) we will load these dataframes directly from file. I will create a dataset shortly containing this information for easy loading.","metadata":{}},{"cell_type":"code","source":"FIRST_RUN = not os.path.isfile(\"/kaggle/input/ai4code-train-dataframe/train.csv\")\n\ndef load_json_to_df(all_fpaths, do_parallel=True):\n def __fpath_to_df(fpath):\n tmp_df = pd.read_json(fpath, dtype={'cell_type': 'category', 'source': 'str'}).reset_index().rename({\"index\":\"cell_id\"}, axis=1)\n tmp_df[\"id\"] = fpath.rsplit(\".\", 1)[0].rsplit(\"/\", 1)[-1]\n return tmp_df\n if do_parallel:\n all_example_dfs = Parallel()(delayed(__fpath_to_df)(fpath) for fpath in tqdm(all_fpaths, total=len(all_fpaths)))\n else:\n all_example_dfs = [__fpath_to_df(fpath) for fpath in tqdm(all_fpaths, total=len(all_fpaths))]\n return pd.concat(all_example_dfs).reset_index(drop=True)\n\nif FIRST_RUN:\n print(\"\\n... CREATING TRAIN AND TEST DATAFRAMES (20-30 MINUTES) ...\\n\")\n train_df = load_json_to_df(TRAIN_JSON_PATHS, do_parallel=False)\n train_df = train_df[[\"id\", \"cell_id\", \"cell_type\", \"source\"]]\n train_df.to_csv(\"train.csv\", index=False, encoding='utf-8', quoting=csv.QUOTE_NONNUMERIC)\nelse:\n print(\"\\n... LOADING TRAIN DATAFRAME AND CREATING TEST DATAFRAME (1-3 MINUTES) ...\\n\")\n train_df = pd.read_csv(\"/kaggle/input/ai4code-train-dataframe/train.csv\", keep_default_na=False)\n \ntest_df = load_json_to_df(TEST_JSON_PATHS)\ntest_df = test_df[[\"id\", \"cell_id\", \"cell_type\", \"source\"]]\n\nprint(\"\\n... ALL TRAIN EXAMPLES AS A DATAFRAME ...\\n\\n\")\ndisplay(train_df)\n\nprint(\"\\n\\n\\n\\n... ALL TEST EXAMPLES AS A DATAFRAME ...\\n\\n\")\ndisplay(train_df)\n\nprint(\"\\n\\n\\n\\n... VIEW THE ROWS THAT WERE PREVIOUSLY CAUSING PROBLEMS THAT CONTAIN NAN LIKE STRINGS ...\\n\\n\")\nnan_weirdos = [2076836, 2915099, 3416950, 4260446]\nfor row_idx in nan_weirdos:\n _row = train_df.iloc[row_idx]\n with open([x for x in TRAIN_JSON_PATHS if _row[\"id\"] in x][0]) as json_file:\n data = json.load(json_file)\n print(f\"ROW INDEX: {row_idx}\\nSOURCE IN OUR DATAFRAME : \", _row[\"source\"])\n print(\"SOURCE DIRECTLY FROM JSON FIL E: \", data[\"source\"][_row[\"cell_id\"]], \"\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:17:55.449396Z","iopub.execute_input":"2022-05-25T01:17:55.450046Z","iopub.status.idle":"2022-05-25T01:18:26.616644Z","shell.execute_reply.started":"2022-05-25T01:17:55.450007Z","shell.execute_reply":"2022-05-25T01:18:26.615894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n\n 3 HELPER FUNCTION & CLASSES ⤒\n
\n\n---","metadata":{}},{"cell_type":"code","source":"def flatten_l_o_l(nested_list):\n \"\"\" Flatten a nested list of lists \"\"\"\n return [item for sublist in nested_list for item in sublist]\n\ndef display_markdown(markdown_str):\n \"\"\" Wrapper function to display markdown as output of code cell \"\"\"\n display(Markdown(markdown_str))\n\ndef display_code(code_str):\n \"\"\" Wrapper function to display markdown as output of code cell \"\"\"\n display(Code(code_str))\n \ndef get_ex_order(ex_id, orders_df=train_orders_df):\n return orders_df[orders_df[\"id\"]==ex_id].cell_order.values[0].split()\n \ndef display_notebook(ex_id=None, df=train_df, show_ordered=True, render_markdown=True, order_df=train_orders_df):\n \"\"\" Function to allow for visualization of a complete notebook \"\"\"\n \n # Get random ex_id if not provided\n if ex_id is None: ex_id = df[\"id\"].sample(1).values[0]\n print(f\"\\n\\n\\n\\n... INVESTIGATING AND VISUALIZING EXAMPLE {ex_id} –– CELLS WILL BE {'ORDERED' if show_ordered else 'UNORDERED'} ...\\n\\n\\n\\n\")\n \n # Get unordered subset of dataframe\n u_sub_df = df[df[\"id\"]==ex_id].reset_index(drop=True)\n \n # Get unordered subset of dataframe\n if show_ordered:\n cell_id_sorter = {c_id:i for i, c_id in enumerate(get_ex_order(ex_id))}\n u_sub_df[\"sorter\"] = u_sub_df.cell_id.map(cell_id_sorter)\n o_sub_df = u_sub_df.sort_values(by=\"sorter\").reset_index(drop=True).drop(columns=[\"sorter\"])\n \n for i, (_, row) in enumerate(u_sub_df.iterrows() if not show_ordered else o_sub_df.iterrows()):\n print(\"\\n\\n\")\n display_markdown(\"---\")\n print()\n display_markdown(f\"{'----- '+'CELL '+str(i+1)+' OF TYPE '+str(row.cell_type.upper())+' -----':^120}\")\n display_markdown(\"---\")\n print()\n if render_markdown:\n display_markdown(row[\"source\"]) if row[\"cell_type\"]==\"markdown\" else display_code(row[\"source\"])\n else:\n display_code(row[\"source\"])\n print()\n display_markdown(\"---\")","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:18:26.618428Z","iopub.execute_input":"2022-05-25T01:18:26.619015Z","iopub.status.idle":"2022-05-25T01:18:26.631631Z","shell.execute_reply.started":"2022-05-25T01:18:26.618964Z","shell.execute_reply":"2022-05-25T01:18:26.630871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n\n 4 DATASET EXPLORATION ⤒\n
\n\n
\n\n---\n\n
\n\nFor each notebook (example) in our dataset, we have:\n* some number (n) of **code** rows and some number (m) of **markdown** rows. \n* Each notebook has all the code cells given first with the markdown cells following. \n* The code cells are in the correct relative order, while the markdown cells are shuffled.\n\n
\n\n---\n\n\nWe will now investigate properties that might inform on relationships allowing us to predict the correct relative order of the markdown cells.","metadata":{}},{"cell_type":"markdown","source":"
\n\n4.0 ADD SOME INFORMATION TO THE DATAFRAME(S)
\n\n---\n\nWe also create a smaller dataframe called the meta dataframe that will house only general notebook information\n* Number of total cells\n* Number of code cells\n* Number of markdown cells\n* Fraction of code cells\n* Fraction of markdown cells\n* Position of cell in notebook (ground truth if known)\n","metadata":{}},{"cell_type":"code","source":"# There's definitely a faster way to do this with grouping then applying then ungrouping\ndef add_style_specific_counts(df=train_df):\n id_w_style_to_count = df.groupby(df[\"id\"]+\"_\"+df[\"cell_type\"])[\"source\"].count().reset_index()\\\n .groupby(\"index\").first()[\"source\"].to_dict()\n df[\"n_code_cells\"] = (df[\"id\"]+\"_code\").progress_apply(lambda x: id_w_style_to_count.get(x, 0))\n df[\"n_markdown_cells\"] = (df[\"id\"]+\"_markdown\").progress_apply(lambda x: id_w_style_to_count.get(x, 0))\n return df\n\ndef add_position_information(df=train_df, orders_df=train_orders_df):\n all_cell_ids_in_order = orders_df.cell_order.apply(lambda x: x.split()).to_list()\n all_cell_pos = flatten_l_o_l([range(len(sub_cell_ids)) for sub_cell_ids in all_cell_ids_in_order])\n all_cell_ids_in_order = flatten_l_o_l(all_cell_ids_in_order)\n cell_id_2_pos = {c_id:pos for c_id,pos in zip(all_cell_ids_in_order, all_cell_pos)}\n df.insert(4, \"cell_pos\", df[\"cell_id\"].map(cell_id_2_pos))\n return df\n\ntrain_df = add_position_information(train_df)\ntrain_df[\"n_total_cells\"] = train_df.groupby(\"id\")[\"source\"].transform(\"count\")\ntrain_df[\"relative_position\"] = (train_df[\"cell_pos\"]+1)/train_df[\"n_total_cells\"]\ntrain_df = add_style_specific_counts(train_df)\ntrain_df[\"code_fraction\"] = train_df[\"n_code_cells\"]/train_df[\"n_total_cells\"]\ntrain_df[\"markdown_fraction\"] = train_df[\"n_markdown_cells\"]/train_df[\"n_total_cells\"]\n\ntest_df[\"n_total_cells\"] = test_df.groupby(\"id\")[\"source\"].transform(\"count\")\ntest_df = add_style_specific_counts(test_df)\ntest_df[\"code_fraction\"] = test_df[\"n_code_cells\"]/test_df[\"n_total_cells\"]\ntest_df[\"markdown_fraction\"] = test_df[\"n_markdown_cells\"]/test_df[\"n_total_cells\"]\n\ntrain_meta_df = train_df.drop_duplicates(\"id\").reset_index(drop=True)[[\"id\", \"n_total_cells\", \"n_code_cells\", \"n_markdown_cells\", \"code_fraction\", \"markdown_fraction\"]]\ntest_meta_df = test_df.drop_duplicates(\"id\").reset_index(drop=True)[[\"id\", \"n_total_cells\", \"n_code_cells\", \"n_markdown_cells\", \"code_fraction\", \"markdown_fraction\"]]\n\ndisplay(train_df.head())\ndisplay(test_df.head())","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:18:26.634234Z","iopub.execute_input":"2022-05-25T01:18:26.634659Z","iopub.status.idle":"2022-05-25T01:19:18.828769Z","shell.execute_reply.started":"2022-05-25T01:18:26.634627Z","shell.execute_reply":"2022-05-25T01:19:18.828044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n4.1 LOOK AT AN EXAMPLE PRIOR TO DETAILED INVESTIGATION
\n\n---\n\n","metadata":{}},{"cell_type":"code","source":"demo_id = train_meta_df[(train_meta_df.n_code_cells==5)&(train_meta_df.n_markdown_cells==5)].id.unique()[0]\ndisplay_notebook(ex_id=demo_id)","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:22:19.422668Z","iopub.execute_input":"2022-05-25T01:22:19.422934Z","iopub.status.idle":"2022-05-25T01:22:20.500337Z","shell.execute_reply.started":"2022-05-25T01:22:19.422905Z","shell.execute_reply":"2022-05-25T01:22:20.499522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n4.1 INVESTIGATE NOTEBOOK DISTRIBUTION METADATA
\n\n---\n\nTBD\n\n
OBSERVATIONS\n\n* The lengths of the notebooks as described by total cell count, markdown cell count, and code cell count, all skew heavily towards shorter rather than longer. \n * n_total_cells has a mean of only **45.75** while the maximum value is **over 1000**\n * Similar patterns are seen within n_code_cells and n_markdown_cells.
\n* There are certain heuristic rules that always exist:\n * There are always at least 2 cells per notebook\n * There are always at least 1 code cell per notebook\n * There are always at least 1 markdown cell per notebook
\n* Some observations can be gleaned contrasting notebook and markdown cells\n * Notebook cells are more than twice as common as Markdown cells (67% vs. 33%)\n * i.e. The average notebook is comprised of ~30.2 notebook cells and ~15.5 markdown cells (based on the average total notebook length of ~45.7 cells.\n * Some notebook are almost entirely code cells (a fraction of 99.7% of cells)\n * Some notebook are almost entirely markdown cells (a fraction of 98.8% of cells)
","metadata":{}},{"cell_type":"code","source":"print(\"\\n... TRANING DATA BASIC INFORMATION ...\\n\")\ndisplay(train_meta_df.describe().T)\n\nprint(\"\\n\\n\\n... PLOTS TO SHOW TOTAL AND STYLE SPECIFIC NOTEBOOK LENGTH CELL COUNTS ...\\n\")\n\nfig = px.histogram(train_meta_df, [\"n_total_cells\"], title=\"Number of Total Cells Per Notebook Log Y-Axis\", nbins=200, marginal=\"violin\", log_y=True)\nfig.update_layout(showlegend=False)\nfig.show()\n\nfig = px.histogram(train_meta_df, [\"n_code_cells\", \"n_markdown_cells\"], barmode=\"overlay\", title=\"Number of Markdown v. Code Cells Per Notebook Log Y-Axis\", nbins=200, marginal=\"violin\", log_y=True)\nfig.update_layout(showlegend=False)\nfig.show()\n\nfig = px.histogram(train_meta_df, [\"code_fraction\", \"markdown_fraction\"], barmode=\"overlay\", title=\"Distribution of Fractional Composition of Markdown v. Code Cells Per Notebook Log Y-Axis\", nbins=200, marginal=\"violin\", log_y=True)\nfig.update_layout(showlegend=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:22:52.455455Z","iopub.execute_input":"2022-05-25T01:22:52.455747Z","iopub.status.idle":"2022-05-25T01:22:58.129950Z","shell.execute_reply.started":"2022-05-25T01:22:52.455718Z","shell.execute_reply":"2022-05-25T01:22:58.129144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n4.2 INVESTIGATE LENGTHS OF INDIVIDUAL CELL BLOCKS
\n\n---\n\nTBD\n\n
OBSERVATIONS\n\n* TBD","metadata":{}},{"cell_type":"code","source":"train_df[\"n_cell_chars\"] = train_df[\"source\"].progress_apply(len)\ntrain_df.describe().T","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:22:58.131615Z","iopub.execute_input":"2022-05-25T01:22:58.132040Z","iopub.status.idle":"2022-05-25T01:23:12.479889Z","shell.execute_reply.started":"2022-05-25T01:22:58.132005Z","shell.execute_reply":"2022-05-25T01:23:12.479195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n4.3 FIX THE BASE64 IMAGE BLOCKS
\n\n---\n\nTBD\n\n
OBSERVATIONS\n\n* TBD","metadata":{}},{"cell_type":"code","source":"def remove_long_useless_strs(src):\n delim_pairs = [(\";base64,\", '\\\\\"'),\n (\";base64,\", '\\)'),\n (\"weight = b\\'\", \"'\"),\n (\"PARAM = b'\", \"'\"),]\n \n for delim_1, delim_2 in delim_pairs:\n src = re.sub(f'{delim_1}.*?{delim_2}','(replaced)', src, flags=re.DOTALL)\n return src\n\ntrain_df[\"source\"] = train_df[\"source\"].progress_apply(remove_long_useless_strs)\ntrain_df[\"n_cell_chars\"] = train_df[\"source\"].progress_apply(len)\ntrain_df.describe().T","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:23:12.481624Z","iopub.execute_input":"2022-05-25T01:23:12.481927Z","iopub.status.idle":"2022-05-25T01:24:39.563084Z","shell.execute_reply.started":"2022-05-25T01:23:12.481888Z","shell.execute_reply":"2022-05-25T01:24:39.562227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n5 MODELLING ⤒
\n\n---\n\nSimilar to the host provided example notebook, we will leverage TFIDF and XGBRanker. However, we will approach it in a slightly different way and will add in many additional features.\n\n**We will also leverage RAPIDS to accelerate things!**","metadata":{}},{"cell_type":"markdown","source":"
\n\n5.1 CREATE TRAIN/VAL SPLITS
\n\n---\n\nThe **`df_ancestors.csv`** file identifies groups of notebooks derived from a common origin, that is, notebooks belonging to the same forking tree.\n\nTo prevent leakage, the validation cannot have any notebooks with an ancestor in the training set (or viceaversa). We therefore form a validation split using **`ancestor_id`** as a grouping factor.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GroupShuffleSplit\n\n# Apply merge\ntrain_df = pd.merge(train_df, train_ancestors_df[[\"id\", \"ancestor_id\"]], on=\"id\", how=\"left\")\n\nVAL_FRAC = 0.1\nFEAT_COLS = ['id', 'cell_id', 'cell_type', 'source', 'n_total_cells',\n 'n_code_cells', 'n_markdown_cells', 'code_fraction', \n 'markdown_fraction', 'n_cell_chars',]\nLABEL_COLS = ['cell_pos',] # or relative position\nGROUP_COLS = [\"ancestor_id\"]\ng_splitter = GroupShuffleSplit(n_splits=1, test_size=VAL_FRAC, random_state=0)\n\n# Split, keeping notebooks with a common origin (ancestor_id) together\ntrain_ids, val_ids = next(g_splitter.split(train_df[FEAT_COLS], train_df[LABEL_COLS], groups=train_df[GROUP_COLS]))\nids_train, ids_valid = ids[ids_train], ids[ids_valid]\n\nval_df = train_df.iloc[val_ids].reset_index(drop=True)\ntrain_df = train_df.iloc[train_ids].reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:35:56.476257Z","iopub.execute_input":"2022-05-25T01:35:56.476729Z","iopub.status.idle":"2022-05-25T01:35:58.355546Z","shell.execute_reply.started":"2022-05-25T01:35:56.476691Z","shell.execute_reply":"2022-05-25T01:35:58.354634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n5.2 CONVERT TO CUDF TO FREE MEMORY AND PREPARE FOR MODELLING
\n\n---\n","metadata":{}},{"cell_type":"code","source":"val_df = cudf.from_pandas(val_df)\ntrain_df = cudf.from_pandas(train_df)\ngc.collect(); gc.collect();\n\ntrain_df","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:39:34.099644Z","iopub.execute_input":"2022-05-25T01:39:34.099927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}![](\"https://storage.googleapis.com/kaggle-competitions/kaggle/35887/logos/header.png?t=2022-05-09-22-33-02\")
![](\"https://storage.googleapis.com/kaggle-media/Images/notebook_cell_examples.png\")