{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# AI4Code Languages distribution\n\nCompetition data consists of ~160k kaggle notebooks. Kaggle community is international and alothough most notebooks are commented in English, there are plenty kernels with all kinds of languages. In this kernel I demonstrate the distribution of different languages in trainset and I hope it can help you better choose your models.\n\nMoreover, kaggle kernels can be written in two programming languages: Python and R. Distribution of programming languages in trainset is also adressed in this kernel.","metadata":{}},{"cell_type":"code","source":"import json\nimport os\nimport re\n\nimport fasttext\nimport pandas as pd\nimport plotly.express as px\nimport pygments.lexers\nfrom tqdm.auto import tqdm\n\n\ntqdm.pandas()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-05-16T12:08:50.513058Z","iopub.execute_input":"2022-05-16T12:08:50.514069Z","iopub.status.idle":"2022-05-16T12:08:52.396406Z","shell.execute_reply.started":"2022-05-16T12:08:50.513926Z","shell.execute_reply":"2022-05-16T12:08:52.395084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lang_codes = pd.read_csv(\"../input/wikipedia-language-iso639/lang.csv\", index_col=0).squeeze()\ntrain_orders = pd.read_csv(\"../input/AI4Code/train_orders.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T12:08:52.400070Z","iopub.execute_input":"2022-05-16T12:08:52.400490Z","iopub.status.idle":"2022-05-16T12:08:54.148385Z","shell.execute_reply.started":"2022-05-16T12:08:52.400440Z","shell.execute_reply":"2022-05-16T12:08:54.147230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Natural Languages distribution","metadata":{}},{"cell_type":"code","source":"nl_detector = fasttext.load_model(\"../input/fasttext-language-identification/lid.176.bin\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T12:08:57.139822Z","iopub.execute_input":"2022-05-16T12:08:57.140191Z","iopub.status.idle":"2022-05-16T12:08:58.544164Z","shell.execute_reply.started":"2022-05-16T12:08:57.140152Z","shell.execute_reply":"2022-05-16T12:08:58.542939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def detect_nl(nb_id, detector):\n    with open(f\"../input/AI4Code/train/{nb_id}.json\", 'r') as f_in:\n        notebook = json.load(f_in)\n    md_cells = [cell_id for cell_id in notebook['cell_type'] if notebook['cell_type'][cell_id] == 'markdown']\n    langs = []\n    for cell_id in md_cells:\n        cell_content = notebook['source'][cell_id].replace('\\n', ' ')\n        langs.append(detector.predict(cell_content, k=1)[0][0][len(\"__label__\"):])\n    return max(set(langs), key=langs.count)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T12:08:58.546187Z","iopub.execute_input":"2022-05-16T12:08:58.546448Z","iopub.status.idle":"2022-05-16T12:08:58.559315Z","shell.execute_reply.started":"2022-05-16T12:08:58.546418Z","shell.execute_reply":"2022-05-16T12:08:58.558173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_nl = train_orders['id'].progress_apply(detect_nl, detector=nl_detector)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T12:08:58.560537Z","iopub.execute_input":"2022-05-16T12:08:58.561185Z","iopub.status.idle":"2022-05-16T12:21:51.402183Z","shell.execute_reply.started":"2022-05-16T12:08:58.561141Z","shell.execute_reply":"2022-05-16T12:21:51.401021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lang_counts = train_nl.value_counts(dropna=False).reset_index()\nlang_counts['language'] = lang_counts['index'].map(lang_codes).fillna(lang_counts['index'])\nlang_counts = lang_counts.rename(columns={\"id\": \"count\"})\n\nfig = px.pie(lang_counts, values='count', names='language', title=\"Natural Languages distribution in Train data\")\nfig.update_traces(textposition='inside', textinfo='percent+label')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T12:21:51.404932Z","iopub.execute_input":"2022-05-16T12:21:51.405867Z","iopub.status.idle":"2022-05-16T12:21:53.029693Z","shell.execute_reply.started":"2022-05-16T12:21:51.405814Z","shell.execute_reply":"2022-05-16T12:21:53.028392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"93% kernels are commented in English, but 7% are not, and that may be significant if you're fighting for thousandths on the leaderboard!\nThe plot is interactive and you can turn off English if you want to look more closely at other languages. But be aware that percentage shown on graph is always relative to the shown (\"enabled\") elements.","metadata":{}},{"cell_type":"markdown","source":"# Programming Languages distribution","metadata":{}},{"cell_type":"markdown","source":"I tried several methods for detecting programming languages: guesslang was too slow, Pygments was highly inaccurate, so I ended up with simple regex checking format of imports. Of course there are some notebooks that do not import anything at all, but there aren't many of them. I'd be happy to learn about a better method.","metadata":{}},{"cell_type":"code","source":"def detect_pl(nb_id):\n    with open(f\"../input/AI4Code/train/{nb_id}.json\", 'r') as f_in:\n        notebook = json.load(f_in)\n    src_cells = [cell_id for cell_id in notebook['cell_type'] if notebook['cell_type'][cell_id] == 'code']\n    py_regex = r\"\\bimport \\w+\"\n    r_regex = r\"\\blibrary\\(\\w+\"\n    langs = []\n    for cell_id in src_cells:\n        cell_content = notebook['source'][cell_id]\n        if re.search(py_regex, cell_content):\n            langs.append(\"Python\")\n        if re.search(r_regex, cell_content):\n            langs.append(\"R\")\n    if len(langs) > 0:\n        return max(set(langs), key=langs.count)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T12:21:53.031554Z","iopub.execute_input":"2022-05-16T12:21:53.031845Z","iopub.status.idle":"2022-05-16T12:21:53.040555Z","shell.execute_reply.started":"2022-05-16T12:21:53.031812Z","shell.execute_reply":"2022-05-16T12:21:53.039801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_pl = train_orders['id'].progress_apply(detect_pl)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T12:21:53.041549Z","iopub.execute_input":"2022-05-16T12:21:53.041788Z","iopub.status.idle":"2022-05-16T12:27:19.698838Z","shell.execute_reply.started":"2022-05-16T12:21:53.041758Z","shell.execute_reply":"2022-05-16T12:27:19.697802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pl_counts = train_pl.value_counts(dropna=False).reset_index()\npl_counts = pl_counts.rename(columns={\"id\": \"count\", \"index\": \"language\"})\n\nfig = px.pie(pl_counts, values='count', names='language', title=\"Programming Language distribution in Train data\")\nfig.update_traces(textposition='inside', textinfo='percent+label')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T12:27:19.700684Z","iopub.execute_input":"2022-05-16T12:27:19.700911Z","iopub.status.idle":"2022-05-16T12:27:19.782383Z","shell.execute_reply.started":"2022-05-16T12:27:19.700884Z","shell.execute_reply":"2022-05-16T12:27:19.781715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Only 62 kernels from trainset are written in R (or my regex is not good enough). Nulls on the plot correspond either to simple educational kernels without imports (mostly in python) or to kernels which contain only bash cells, executing some py-scripts.","metadata":{}}]}