{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Notes\n\nFirst at all, to understand the goal of the competition and available approaches, read the \"[My understanding of the domain - Taking a step back](https://www.kaggle.com/competitions/AI4Code/discussion/328905)\" thread by @Allohvk.\n\nNice [live twitch stream session](https://www.twitch.tv/videos/1482350967) by Rob Mulla and [this is the result](https://www.kaggle.com/code/robikscube/google-ai4code-data-to-parquet-twitch-stream-eda).\n\n### Metric\n\n* Wikipedia [Kendall rank correlation coefficient](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient)\n* Notebook [Competition Metric - Kendall Tau Correlation](https://www.kaggle.com/code/ryanholbrook/competition-metric-kendall-tau-correlation/notebook)\n\n### Observations\n\nFrom [EDA by Sanskar Hasija](https://www.kaggle.com/code/odins0n/ai4code-detailed-eda) (A good EDA to start):\n* A total of 139256 notebooks are provided train set.\n* Only 4 notebooks are provided test set.\n* A total of 146300 cells in train the train dataframe constructed which include two types of cell_type.\n* 1/3rd consist of Markdown Cells.\n\nFrom [Comprehensive EDA by ANDREAS PALMGREN](https://www.kaggle.com/code/andreaspalmgren/ai4code-comprehensive-eda)\n* Almost all markdown cells in English but there are cells in other languages (40 different languages).\n\nFrom this notebook (see \"Markdown cells EDA\" below)\n* markdown cells must be preprocessed\n* there are markdown cells repeated in the same notebook with different id\n\n### To Do\n\n* Think about \"ancestor\" meaning in this context\n* ~~open a thread in discussion about markdown cells repeated with different id~~. This [thread](https://www.kaggle.com/competitions/AI4Code/discussion/328324).\n* preprocess markdown cells:\n    * avoid img html tags\n    * null markdown cell after preprocess?\n","metadata":{}},{"cell_type":"markdown","source":"## Markdown cells EDA\n\nWe are going to compute a md5 hash in every markdown cell in order to find identical cells in notebooks.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom hashlib import md5\nfrom pathlib import Path\n\nDATA_DIR = \"/kaggle/input/AI4Code\"\n\npaths = Path(f'{DATA_DIR}/train/').glob(\"*.json\")\nx = [p for p in paths]\nr = [pd.read_json(i,dtype={'cell_type': 'category', 'source': 'str'}).assign(id=i.stem) for i in x]\nallcells = pd.concat(r)\nallcells['hash'] = allcells.source.str.encode('utf-8').apply(lambda h: md5(h).hexdigest())\nallcells['code_len'] = allcells.source.str.len()","metadata":{"execution":{"iopub.status.busy":"2022-05-31T18:08:04.094193Z","iopub.execute_input":"2022-05-31T18:08:04.094639Z","iopub.status.idle":"2022-05-31T18:08:24.584967Z","shell.execute_reply.started":"2022-05-31T18:08:04.094606Z","shell.execute_reply":"2022-05-31T18:08:24.583847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"allcells.sample(10)","metadata":{"execution":{"iopub.status.busy":"2022-05-31T18:08:24.586492Z","iopub.execute_input":"2022-05-31T18:08:24.586965Z","iopub.status.idle":"2022-05-31T18:08:24.63444Z","shell.execute_reply.started":"2022-05-31T18:08:24.586927Z","shell.execute_reply":"2022-05-31T18:08:24.633424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Top identical markdown cells ","metadata":{}},{"cell_type":"code","source":"allcells.query('cell_type ==\"markdown\"').groupby('hash').filter(lambda x: len(x) > 1).groupby('hash').count().nlargest(10, 'id')","metadata":{"execution":{"iopub.status.busy":"2022-05-31T18:08:24.635976Z","iopub.execute_input":"2022-05-31T18:08:24.636547Z","iopub.status.idle":"2022-05-31T18:08:25.327949Z","shell.execute_reply.started":"2022-05-31T18:08:24.636501Z","shell.execute_reply":"2022-05-31T18:08:25.326989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Top largest identical markdown cells","metadata":{}},{"cell_type":"code","source":"allcells.query('cell_type ==\"markdown\"').groupby('hash').filter(lambda x: len(x) > 1).nlargest(20, 'code_len').sort_values(by=['code_len','hash'],ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-31T18:08:25.329881Z","iopub.execute_input":"2022-05-31T18:08:25.330237Z","iopub.status.idle":"2022-05-31T18:08:25.981721Z","shell.execute_reply.started":"2022-05-31T18:08:25.330205Z","shell.execute_reply":"2022-05-31T18:08:25.980355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Top identical markdown cells in the same notebook","metadata":{}},{"cell_type":"code","source":"allcells.query('cell_type ==\"markdown\"').groupby(['hash','id']).filter(lambda x: len(x) > 1).groupby(['hash','id']).count().nlargest(10, 'source')","metadata":{"execution":{"iopub.status.busy":"2022-05-31T18:08:25.983208Z","iopub.execute_input":"2022-05-31T18:08:25.983587Z","iopub.status.idle":"2022-05-31T18:08:26.969919Z","shell.execute_reply.started":"2022-05-31T18:08:25.983535Z","shell.execute_reply":"2022-05-31T18:08:26.968861Z"},"trusted":true},"execution_count":null,"outputs":[]}]}