{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<div style=\"padding:20px;color:white;margin:0;font-size:24px;text-align:center;display:fill;border-radius:5px;background-color:#7b6b59;overflow:hidden\">Table of Contents</div>\n\n<a id=\"toc\"></a>\n- [1. Imports](#1)\n- [2. Load Dataset](#2)","metadata":{}},{"cell_type":"markdown","source":"<a id=\"1\"></a>\n# <div style=\"padding:20px;color:white;margin:0;font-size:24px;text-align:center;display:fill;border-radius:5px;background-color:#7b6b59;overflow:hidden\">Imports</div>","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport glob\nfrom tqdm import tqdm","metadata":{"execution":{"iopub.status.busy":"2023-02-12T09:53:37.999859Z","iopub.execute_input":"2023-02-12T09:53:38.000203Z","iopub.status.idle":"2023-02-12T09:53:38.005010Z","shell.execute_reply.started":"2023-02-12T09:53:38.000176Z","shell.execute_reply":"2023-02-12T09:53:38.003863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DATA_PATH = \"/kaggle/input/AI4Code/\"","metadata":{"execution":{"iopub.status.busy":"2023-02-12T09:53:38.625732Z","iopub.execute_input":"2023-02-12T09:53:38.626059Z","iopub.status.idle":"2023-02-12T09:53:38.629903Z","shell.execute_reply.started":"2023-02-12T09:53:38.626034Z","shell.execute_reply":"2023-02-12T09:53:38.629072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"2\"></a>\n# <div style=\"padding:20px;color:white;margin:0;font-size:24px;text-align:center;display:fill;border-radius:5px;background-color:#7b6b59;overflow:hidden\">Load Dataset</div>","metadata":{}},{"cell_type":"code","source":"def read_json_files(mode:str):\n    \"\"\"\n    \"\"\"\n    \n    # Get the list of all json files containing the notebooks' cells\n    notebook_files = sorted(list(glob.glob(f\"{DATA_PATH}/{mode}/*.json\"))) \n    \n    # Each notebook is represented by a dataframe and we keep a list of notebook dataframes\n    notebooks_df_list = []\n    for path in tqdm(notebook_files, desc=\"Loading the Notebooks\"):\n        notebooks_df_list.append(\n            pd.read_json(path, dtype={'cell_type': 'category', 'source': 'str'}).assign(\n                id=path.split('/')[-1].split('.')[0]).rename_axis('cell_id')\n        )\n        \n    \n    entire_df = pd.concat(notebooks_df_list).reset_index(drop=False)\n\n    return entire_df\n    ","metadata":{"execution":{"iopub.status.busy":"2023-02-12T09:07:29.199897Z","iopub.execute_input":"2023-02-12T09:07:29.200227Z","iopub.status.idle":"2023-02-12T09:07:29.206511Z","shell.execute_reply.started":"2023-02-12T09:07:29.200200Z","shell.execute_reply":"2023-02-12T09:07:29.205326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#train_df = read_json_files(\"train\")","metadata":{"execution":{"iopub.status.busy":"2023-02-12T09:07:29.728768Z","iopub.execute_input":"2023-02-12T09:07:29.729130Z","iopub.status.idle":"2023-02-12T09:44:17.533643Z","shell.execute_reply.started":"2023-02-12T09:07:29.729098Z","shell.execute_reply":"2023-02-12T09:44:17.532507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#train_df.head(10)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**WORK IN PROGRESS**","metadata":{}},{"cell_type":"markdown","source":"# Problem formulation \n\nThe stated goal of this competition is somewhat interesting. The first line states that the goal is to understand the relationship between the code and comment cells. The second line challenges us to reconstruct the order of the markdown cells and this is purely how the winner is measured (the first goal does not matter as far as evaluations concernedthe markdowns need to be ordered AND put in the right slots between the code…so the first goal does matter indirectly). We need to rank the markdown cells - identify which comes after the other & put them in he right slot between the appropriate code cells (which are already ordered correctly). The evaluation is based on Kendell Tau - and since folks have already shared some good links explaining the same, we needn’t go deep into the same. It is just an indicator between -1 to +1 on how well the predicted and the actual ranks match.\n\n## Learning to Rank Framework\n\nFor a moment, keeping the code part aside, this essentially boils down to ranking a series of texts in the right order. In that sense, it can be reframed as a ranking problem. \n\n- **Rank:** The correct numeric position of a cell we will call the rank of the cell. We can find the ranks of the cells within a notebook by referencing the true ordering of cell ids as given in `train_orders.csv`\n\nNow this provides a great relief as there is an extensive body of work under the **“Learning to Rank”** framework which can now be leveraged. Of course, language models also come into picture. Most of the solutions presented revolve around these 2 items - leveraging **language models** and leveraging **the Learning to Rank** framework. We now have a clear sense of direction and can dig into both these aspects and how they help us design a suitable model. \n\nBut before that - What could be the intent of this competition? This was an interesting question posted https://www.kaggle.com/competitions/AI4Code/discussion/325205 in one of the discussions and my take on it is that this could just be an extension of Google search to cater more efficiently to Code results.\n\nWe have a query (this is the equivalent of the markdown cell), we have content (this will be the code cell) and what we want is the code cell that is best matched to the query. This is an exciting area. Facebook is doing a lot of work in neural code search, **Microsoft** has **CodeBert** and is also neck-deep in the code search domain. Tying natural language to code has immense downstream possibilities.\n\n## Approaches\n\nComing to the potential approaches, we have the following:\n\n- Rank only the Markdowns without paying heed to the code cells. Use plain NLP for this combined with the LTR (Learning To Rank) frameworks. Surprisingly this seems to get OK results. When given a \"context\" in the form of code cells these work really well\n\n- Tie the Markdown to the code. **In this case we dont need to develop a model for ranking because the code cells are already ranked.** As long as we are able to match the code to markdown, we directly get the markdown order.\n\n- See how we can leverage both the above approaches.\n\n\n## A simple baseline Ranking model\n\nQuite simply, the goal of a ranking model is to sort data in an optimal and relevant order.\n\nWe'll look at how to:\n\n- Wrangle the competition data and create validation splits,\n- Represent the code cell orders with a feature,\n- Build a ranking model with XGBoost,\n- Evaluate predictions with a Python implementation of the competition metric, and,\n- Format predictions to make a successful submission.\n\nThe algorithm we'll be using for our baseline model uses the cell ranks as the target, so let's create a dataframe of the ranks for each notebook.\n\n\n1. https://towardsdatascience.com/learning-to-rank-a-complete-guide-to-ranking-using-machine-learning-4c9688d370d4","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}