{
  "id": 327626,
  "title": "Dealing with multi languages in notebooks",
  "url": "/competitions/AI4Code/discussion/327626",
  "author_name": "",
  "post_date": "2022-05-28T08:23:06.869767900Z",
  "votes": 14,
  "comment_count": 5,
  "views": 0,
  "content": "<p>We can see so many languages in markdown cells:</p>\n<ul>\n<li>Most of them are English (75%)</li>\n<li>Some of them are Chinese</li>\n<li>Some of them are Hindi</li>\n<li>Some of them are Japanese</li>\n<li>….</li>\n</ul>\n<p>In fact, Hindi, Chinese, Japanese are not alphabet language, so you can't split into tokens just by spaces, colon, semicolons, … So how can we deal with these language ? I don't think we should use specific tokenized model for each language, because it costs too much time.</p>\n<p>Beside, we only have English in test set so should we drop these language when training model for avoiding noise of incorrect token ?</p>\n<p>Please share with me ideals about dealing with these language.</p>\n<p>This is my short summary about detected language. I have create <a href=\"https://www.kaggle.com/code/astrung/language-detection-transformer\" target=\"_blank\">a notebook for detect language</a>, using <a href=\"https://huggingface.co/papluca/xlm-roberta-base-language-detection\" target=\"_blank\">this model</a> with 20 languages. But some languages are unseen languages with this model (Indonesian, …), so I use unknown label for these languages. As you can see, most of notebook are English. </p>\n<pre><code>en         0.758186(%)\nunknown    0.159912(%)\npt         0.015380(%)\ntr         0.010126(%)\nru         0.009829(%)\nit         0.009750(%)\nja         0.007455(%)\nes         0.006610(%)\nzh         0.004808(%)\nfr         0.003614(%)\nhi         0.003609(%)\nur         0.003102(%)\nnl         0.001805(%)\nvi         0.001731(%)\nde         0.001352(%)\nsw         0.001291(%)\npl         0.000670(%)\nth         0.000428(%)\nbg         0.000165(%)\nar         0.000123(%)\nel         0.000054(%)\n</code></pre>\n<p>If you want to use directly my result, you can use this dataset: <a href=\"https://www.kaggle.com/datasets/astrung/google-ai4code-train-markdown-language\" target=\"_blank\">https://www.kaggle.com/datasets/astrung/google-ai4code-train-markdown-language</a></p>\n<p><strong>Please upvote both this notebook and dataset if it is helpful for you</strong></p>",
  "messages": [
    {
      "id": "1803843",
      "postDate": "05/28/2022 08:23:06",
      "content": "<p>We can see so many languages in markdown cells:</p>\n<ul>\n<li>Most of them are English (75%)</li>\n<li>Some of them are Chinese</li>\n<li>Some of them are Hindi</li>\n<li>Some of them are Japanese</li>\n<li>….</li>\n</ul>\n<p>In fact, Hindi, Chinese, Japanese are not alphabet language, so you can't split into tokens just by spaces, colon, semicolons, … So how can we deal with these language ? I don't think we should use specific tokenized model for each language, because it costs too much time.</p>\n<p>Beside, we only have English in test set so should we drop these language when training model for avoiding noise of incorrect token ?</p>\n<p>Please share with me ideals about dealing with these language.</p>\n<p>This is my short summary about detected language. I have create <a href=\"https://www.kaggle.com/code/astrung/language-detection-transformer\" target=\"_blank\">a notebook for detect language</a>, using <a href=\"https://huggingface.co/papluca/xlm-roberta-base-language-detection\" target=\"_blank\">this model</a> with 20 languages. But some languages are unseen languages with this model (Indonesian, …), so I use unknown label for these languages. As you can see, most of notebook are English. </p>\n<pre><code>en         0.758186(%)\nunknown    0.159912(%)\npt         0.015380(%)\ntr         0.010126(%)\nru         0.009829(%)\nit         0.009750(%)\nja         0.007455(%)\nes         0.006610(%)\nzh         0.004808(%)\nfr         0.003614(%)\nhi         0.003609(%)\nur         0.003102(%)\nnl         0.001805(%)\nvi         0.001731(%)\nde         0.001352(%)\nsw         0.001291(%)\npl         0.000670(%)\nth         0.000428(%)\nbg         0.000165(%)\nar         0.000123(%)\nel         0.000054(%)\n</code></pre>\n<p>If you want to use directly my result, you can use this dataset: <a href=\"https://www.kaggle.com/datasets/astrung/google-ai4code-train-markdown-language\" target=\"_blank\">https://www.kaggle.com/datasets/astrung/google-ai4code-train-markdown-language</a></p>\n<p><strong>Please upvote both this notebook and dataset if it is helpful for you</strong></p>",
      "rawMarkdown": "We can see so many languages in markdown cells:\n* Most of them are English (75%)\n* Some of them are Chinese\n* Some of them are Hindi\n* Some of them are Japanese\n* ....\n\nIn fact, Hindi, Chinese, Japanese are not alphabet language, so you can't split into tokens just by spaces, colon, semicolons, ... So how can we deal with these language ? I don't think we should use specific tokenized model for each language, because it costs too much time.\n\nBeside, we only have English in test set so should we drop these language when training model for avoiding noise of incorrect token ?\n\nPlease share with me ideals about dealing with these language.\n\nThis is my short summary about detected language. I have create [a notebook for detect language](https://www.kaggle.com/code/astrung/language-detection-transformer), using [this model](https://huggingface.co/papluca/xlm-roberta-base-language-detection) with 20 languages. But some languages are unseen languages with this model (Indonesian, ...), so I use unknown label for these languages. As you can see, most of notebook are English. \n\n\n    en         0.758186(%)\n    unknown    0.159912(%)\n    pt         0.015380(%)\n    tr         0.010126(%)\n    ru         0.009829(%)\n    it         0.009750(%)\n    ja         0.007455(%)\n    es         0.006610(%)\n    zh         0.004808(%)\n    fr         0.003614(%)\n    hi         0.003609(%)\n    ur         0.003102(%)\n    nl         0.001805(%)\n    vi         0.001731(%)\n    de         0.001352(%)\n    sw         0.001291(%)\n    pl         0.000670(%)\n    th         0.000428(%)\n    bg         0.000165(%)\n    ar         0.000123(%)\n    el         0.000054(%)\n\nIf you want to use directly my result, you can use this dataset: https://www.kaggle.com/datasets/astrung/google-ai4code-train-markdown-language\n\n**Please upvote both this notebook and dataset if it is helpful for you**",
      "votes": null
    },
    {
      "id": "1804178",
      "postDate": "05/28/2022 16:10:38",
      "content": "<p>Is it mentioned somewhere that we will only have English in the test data?  </p>",
      "rawMarkdown": "Is it mentioned somewhere that we will only have English in the test data?",
      "votes": null
    },
    {
      "id": "1804454",
      "postDate": "05/29/2022 02:19:52",
      "content": "<p><a href=\"https://www.kaggle.com/alireza9\" target=\"_blank\">@alireza9</a> <br>\nIf you check on test data (only 4 notebooks), you will see all notebooks are English<br>\nBut in this topic, someone have pointed out that test data will be replaced with multi languages in submission:<br>\n<a href=\"https://www.kaggle.com/competitions/AI4Code/discussion/324608\" target=\"_blank\">https://www.kaggle.com/competitions/AI4Code/discussion/324608</a></p>",
      "rawMarkdown": "alireza9 \nIf you check on test data (only 4 notebooks), you will see all notebooks are English\nBut in this topic, someone have pointed out that test data will be replaced with multi languages in submission:\nhttps://www.kaggle.com/competitions/AI4Code/discussion/324608",
      "votes": null
    },
    {
      "id": "1804508",
      "postDate": "05/29/2022 04:57:31",
      "content": "<p>Thanks for sharing! I know another option:</p>\n<pre><code>from langdetect import detect\n\nfor t in texts:\n    print(detect(t), t)\n</code></pre>",
      "rawMarkdown": "Thanks for sharing! I know another option:\n```\nfrom langdetect import detect\n\nfor t in texts:\n    print(detect(t), t)\n```",
      "votes": null
    },
    {
      "id": "1811583",
      "postDate": "06/04/2022 20:57:59",
      "content": "<p>i am using this to detect language </p>\n<p>`<br>\nfrom whatthelang import WhatTheLang<br>\nwtl = WhatTheLang()</p>\n<p>def detect_lang(x):<br>\n    if x.cell_type == 'code' :<br>\n        return('code')<br>\n    else :<br>\n        try :<br>\n            return(wtl.predict_lang(x.surce))<br>\n        except : <br>\n            return 'error'<br>\n`</p>\n<p>much faster</p>",
      "rawMarkdown": "i am using this to detect language \n\n`\nfrom whatthelang import WhatTheLang\nwtl = WhatTheLang()\n\ndef detect_lang(x):\n    if x.cell_type == 'code' :\n        return('code')\n    else :\n        try :\n            return(wtl.predict_lang(x.surce))\n        except : \n            return 'error'\n`\n\nmuch faster",
      "votes": null
    },
    {
      "id": "1812415",
      "postDate": "06/05/2022 19:11:22",
      "content": "<p>There is nothing mentioned about language in the test data in the description. So there may be a combination of different languages in different notebooks.</p>",
      "rawMarkdown": "There is nothing mentioned about language in the test data in the description. So there may be a combination of different languages in different notebooks.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1804178,
      "author_name": "alireza9",
      "author_url": "",
      "post_date": "05/28/2022 16:10:38",
      "content": "<p>Is it mentioned somewhere that we will only have English in the test data?  </p>",
      "votes": null,
      "replies": [
        {
          "id": 1804454,
          "author_name": "astrung",
          "author_url": "",
          "post_date": "05/29/2022 02:19:52",
          "content": "<p><a href=\"https://www.kaggle.com/alireza9\" target=\"_blank\">@alireza9</a> <br>\nIf you check on test data (only 4 notebooks), you will see all notebooks are English<br>\nBut in this topic, someone have pointed out that test data will be replaced with multi languages in submission:<br>\n<a href=\"https://www.kaggle.com/competitions/AI4Code/discussion/324608\" target=\"_blank\">https://www.kaggle.com/competitions/AI4Code/discussion/324608</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1804508,
      "author_name": "sapr3s",
      "author_url": "",
      "post_date": "05/29/2022 04:57:31",
      "content": "<p>Thanks for sharing! I know another option:</p>\n<pre><code>from langdetect import detect\n\nfor t in texts:\n    print(detect(t), t)\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1811583,
      "author_name": "servietsky",
      "author_url": "",
      "post_date": "06/04/2022 20:57:59",
      "content": "<p>i am using this to detect language </p>\n<p>`<br>\nfrom whatthelang import WhatTheLang<br>\nwtl = WhatTheLang()</p>\n<p>def detect_lang(x):<br>\n    if x.cell_type == 'code' :<br>\n        return('code')<br>\n    else :<br>\n        try :<br>\n            return(wtl.predict_lang(x.surce))<br>\n        except : <br>\n            return 'error'<br>\n`</p>\n<p>much faster</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1812415,
      "author_name": "ashrafr",
      "author_url": "",
      "post_date": "06/05/2022 19:11:22",
      "content": "<p>There is nothing mentioned about language in the test data in the description. So there may be a combination of different languages in different notebooks.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1803843": "We can see so many languages in markdown cells:\n* Most of them are English (75%)\n* Some of them are Chinese\n* Some of them are Hindi\n* Some of them are Japanese\n* ....\n\nIn fact, Hindi, Chinese, Japanese are not alphabet language, so you can't split into tokens just by spaces, colon, semicolons, ... So how can we deal with these language ? I don't think we should use specific tokenized model for each language, because it costs too much time.\n\nBeside, we only have English in test set so should we drop these language when training model for avoiding noise of incorrect token ?\n\nPlease share with me ideals about dealing with these language.\n\nThis is my short summary about detected language. I have create [a notebook for detect language](https://www.kaggle.com/code/astrung/language-detection-transformer), using [this model](https://huggingface.co/papluca/xlm-roberta-base-language-detection) with 20 languages. But some languages are unseen languages with this model (Indonesian, ...), so I use unknown label for these languages. As you can see, most of notebook are English. \n\n\n    en         0.758186(%)\n    unknown    0.159912(%)\n    pt         0.015380(%)\n    tr         0.010126(%)\n    ru         0.009829(%)\n    it         0.009750(%)\n    ja         0.007455(%)\n    es         0.006610(%)\n    zh         0.004808(%)\n    fr         0.003614(%)\n    hi         0.003609(%)\n    ur         0.003102(%)\n    nl         0.001805(%)\n    vi         0.001731(%)\n    de         0.001352(%)\n    sw         0.001291(%)\n    pl         0.000670(%)\n    th         0.000428(%)\n    bg         0.000165(%)\n    ar         0.000123(%)\n    el         0.000054(%)\n\nIf you want to use directly my result, you can use this dataset: https://www.kaggle.com/datasets/astrung/google-ai4code-train-markdown-language\n\n**Please upvote both this notebook and dataset if it is helpful for you**",
    "1804178": "Is it mentioned somewhere that we will only have English in the test data?",
    "1804454": "alireza9 \nIf you check on test data (only 4 notebooks), you will see all notebooks are English\nBut in this topic, someone have pointed out that test data will be replaced with multi languages in submission:\nhttps://www.kaggle.com/competitions/AI4Code/discussion/324608",
    "1804508": "Thanks for sharing! I know another option:\n```\nfrom langdetect import detect\n\nfor t in texts:\n    print(detect(t), t)\n```",
    "1811583": "i am using this to detect language \n\n`\nfrom whatthelang import WhatTheLang\nwtl = WhatTheLang()\n\ndef detect_lang(x):\n    if x.cell_type == 'code' :\n        return('code')\n    else :\n        try :\n            return(wtl.predict_lang(x.surce))\n        except : \n            return 'error'\n`\n\nmuch faster",
    "1812415": "There is nothing mentioned about language in the test data in the description. So there may be a combination of different languages in different notebooks."
  },
  "source": "meta"
}