{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook demonstrate how to use transformers model to detect language for markdown cells in train data of this competition: https://www.kaggle.com/competitions/AI4Code/data. This dataset has multiple languages in notebooks, so understanding about language in each notebook may help you for better tokenize and model. As an example:\n\n* we have japanese, hindi and chinese notebooks in this dataset, and these languages can't be tokenized by symbols(space, colon, semicolons,...). \n* These words can't be splited by spaces, so if you want to deal with these languages, you need to use some specific tokenize models for it. \n* Or you should remove these notebooks when training model alphabet language notebooks, because incorrected tokens in these notebooks can add more noise to our model.\n* ...\n\nIn any cases, it is important for detect our markdown language. So i create this notebook as an example. Please don't run this notebook in kaggle kernel, it tooks 2 days in my computer with GPU, so it will never complete in a kaggle kernel. If you want to use directly detected language result from this notebook, you can use this dataset:\n* https://www.kaggle.com/datasets/astrung/google-ai4code-train-markdown-language\n\n**Please upvote both this notebook and dataset if it is helpful for you**\n\nAnd this is a short summary about languages in train dataset:\n\n    en         0.758186(%)\n    unknown    0.159912(%)\n    pt         0.015380(%)\n    tr         0.010126(%)\n    ru         0.009829(%)\n    it         0.009750(%)\n    ja         0.007455(%)\n    es         0.006610(%)\n    zh         0.004808(%)\n    fr         0.003614(%)\n    hi         0.003609(%)\n    ur         0.003102(%)\n    nl         0.001805(%)\n    vi         0.001731(%)\n    de         0.001352(%)\n    sw         0.001291(%)\n    pl         0.000670(%)\n    th         0.000428(%)\n    bg         0.000165(%)\n    ar         0.000123(%)\n    el         0.000054(%)\n\nYou can get language label from this model: https://huggingface.co/papluca/xlm-roberta-base-language-detection. I use this model for predict language in this notebooks. However, you can try with other models, too.\n\nPlease note, some of languages in this train data does not have corresponding labels in this langugage model. As a example, some of notebooks are Indonesian languages, but since we don't have this label in model, we get some annoying classes with very low confidence(I choose `prob <= 0.8` as threshold) -> I assign these cases to \"Unknown\". So \"Unknown\" label means detections model predicted labels with very low confidence, and it may be new language(not 20 languages of our model)","metadata":{}},{"cell_type":"code","source":"import os\nimport ast\nimport pandas as pd\nfrom transformers import pipeline, set_seed\nfrom bs4 import BeautifulSoup\nfrom markdown import markdown","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"arr = os.listdir('../input/AI4Code/train')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dfs = []\nfor file in arr:\n#     print(file)\n    df = pd.read_json(os.path.join('train', file), orient='column')\n    df['file'] = os.path.join('train', file)\n    df = df[df['cell_type'] != 'code'].copy()\n    dfs.append(df)\nlen(dfs)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"generator = pipeline('text-classification', model='papluca/xlm-roberta-base-language-detection')\n# for df in dfs:\n#     df['source2'] = df['source'].apply(lambda x: x[:513])\n#     df['detected_language_result'] = generator(df['source2'].values.tolist())\n#     df['language_label'] = df['detected_language_result'].apply(lambda x: x['label'])\n#     df['language_prob'] = df['detected_language_result'].apply(lambda x: x['score'])\n#     df = df.drop(columns=['detected_language_result', 'source2'])\n# #     print(df.head())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dfs[20000]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = pd.concat(dfs)\ndf_train","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def convert_markdown(text):\n    html = markdown(text)\n    text = ' '.join(BeautifulSoup(html).findAll(text=True)).replace('\\n', ' ')\n    return text[:513]\ndf_train['source2'] = df_train['source'].apply(lambda x: convert_markdown(x))\n# generator(df_train['source2'].values[:1000].tolist())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['detected_language_result'] = None\ndf_train['language_label'] = None\ndf_train['language_prob'] = None","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def infer_lang(df):\n    df = df.copy()\n    df['detected_language_result'] = generator(df['source2'].values.tolist())\n    df['language_label'] = df['detected_language_result'].apply(lambda x: x['label'])\n    df['language_prob'] = df['detected_language_result'].apply(lambda x: x['score'])\n    return df\n#     df = df.drop(columns=['detected_language_result', 'source2'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_result = []\nfor start in range(0, len(df_train), 3000):\n    df_result.append(infer_lang(df_train[start:start+3000]))\n    print(start)\n#     if start > 10:\n#         break","metadata":{"scrolled":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(df_result)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final = pd.concat(df_result)\ndf_final","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_final = df_final.drop(columns=['source2', 'detected_language_result'])\ndf_final.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final.language_label.value_counts()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'pt'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'it'].sort_values('language_prob').tail(30)","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'tr'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ur doens't look good. Let check carefully","metadata":{}},{"cell_type":"code","source":"df_final[df_final.language_label == 'ur'].sort_values('language_prob').tail(400).head(30)","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'ja']","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'th'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'hi'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'ru']","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# sw doens't look good. Let check carefully ","metadata":{}},{"cell_type":"code","source":"df_final[df_final.language_label == 'sw'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'es'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'zh'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'nl'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'fr'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'vi'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'de'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'pl'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'bg'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'el'].sort_values('language_prob')","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final[df_final.language_label == 'ar'].sort_values('language_prob')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final.language_label.value_counts()/len(df_final)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_norm = df_final.copy()\ndf_norm.loc[df_norm['language_prob'] < 0.8, 'language_label'] = 'unknown'\ndf_norm.loc[df_norm['language_prob'] < 0.8, 'language_prob'] = 0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_norm.language_label.value_counts()/len(df_final)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_norm.to_csv('df_language.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}