![](\"https://i.imgur.com/duo35j4.png\")
While I was exploring the notebooks' source I've realised that there are a few that aren't written in English.
This got me wondering - how many notebooks within the train dataset are in another language and how does this impact the overall models.
So, this is what I've found:
\nI just used the basic LanguageDetector from the spacy library and then extracted the language of the first 200 chars within a notebook. I've put the 200 barrier to not overload the memory - this process took about 1hr and 30 mins.
test data too?id and language mapping can be found in AI4Code - Processed DataWhile I was exploring the notebooks' source I've realised that there are a few that aren't written in English.
This got me wondering - how many notebooks within the train dataset are in another language and how does this impact the overall models.
So, this is what I've found:
\nI just used the basic LanguageDetector from the spacy library and then extracted the language of the first 200 chars within a notebook. I've put the 200 barrier to not overload the memory - this process took about 1hr and 30 mins.
test data too?id and language mapping can be found in AI4Code - Processed Data\n\n### Some thoughts\n\n* *should we delete* training notebooks that are non-english or should we try to *incorporate* them?\n* is this *90-10 proportion the same* for the `test` data too?\n\n### Data and Full Analysis\n\n* **Dataset**: the full dataset with the notebook `id` and language mapping can be found in [AI4Code - Processed Data](https://www.kaggle.com/datasets/andradaolteanu/ai4code-processed-data)\n* **Notebook** with full analysis: https://www.kaggle.com/code/andradaolteanu/ai4code-language-detection-and-model-tuning/notebook",
"votes": null
},
{
"id": "1815469",
"postDate": "06/09/2022 04:23:09",
"content": "Excellent and interesting analysis @andradaolteanu .
\n A noob question here.. how about train the model on only English first and test on other languages? After that train on whole data? Is transfer learning effective on language models?
It will encourage bias over languages in your training and validation distribution.
", "rawMarkdown": "It will encourage bias over languages in your training and validation distribution.", "votes": null }, { "id": "1815756", "postDate": "06/09/2022 12:06:04", "content": "Would there be other languages in the test set too? In that case, can a pre-trained language_2_language model be used as step 1 to convert to english?
", "rawMarkdown": "Would there be other languages in the test set too? In that case, can a pre-trained language_2_language model be used as step 1 to convert to english?", "votes": null }, { "id": "1815911", "postDate": "06/09/2022 16:08:58", "content": "Great insights!
", "rawMarkdown": "Great insights!", "votes": null }, { "id": "1815913", "postDate": "06/09/2022 16:14:30", "content": "that's the thing, we don't know if the test set mantains the same proportions as the training set - so whatever we choose it needs to be robust
", "rawMarkdown": "that's the thing, we don't know if the test set mantains the same proportions as the training set - so whatever we choose it needs to be robust", "votes": null }, { "id": "1816223", "postDate": "06/10/2022 01:55:03", "content": "@andradaolteanu thanks for the discovery. I have the similar problem with one of the recent project which has combination of two language (english, native language). I have been thinking how the other language can impact the performance of Model or shouldn't be impacted at all. Any suggestions?
\nWill wait for your notebook. Thanks
Interesting and brilliant approach to see what language written over actual world. Thanks you so much to give us wonderful insights! π@andradaolteanu
", "rawMarkdown": "Interesting and brilliant approach to see what language written over actual world. Thanks you so much to give us wonderful insights! π@andradaolteanu", "votes": null }, { "id": "1818619", "postDate": "06/13/2022 01:23:11", "content": "Thanks for the info. I am new to this field.π
", "rawMarkdown": "Thanks for the info. I am new to this field.π", "votes": null }, { "id": "1820866", "postDate": "06/15/2022 04:22:38", "content": "If it rises to the top of the leaderboard in the future, other PLM models(multilingual model) can be considered. Thank you for the good analysis. @andradaolteanu ππ
", "rawMarkdown": "If it rises to the top of the leaderboard in the future, other PLM models(multilingual model) can be considered. Thank you for the good analysis. @andradaolteanu ππ", "votes": null }, { "id": "1821298", "postDate": "06/15/2022 12:04:27", "content": "Thanks for the info. I am new to this field.π
", "rawMarkdown": "Thanks for the info. I am new to this field.π", "votes": null }, { "id": "1829109", "postDate": "06/22/2022 10:17:45", "content": "Hmm. This was pretty much the first blocker I ran into right after loading the dataset. Thanks for sharing the insights.
", "rawMarkdown": "Hmm. This was pretty much the first blocker I ran into right after loading the dataset. Thanks for sharing the insights.", "votes": null }, { "id": "1843853", "postDate": "07/05/2022 07:00:15", "content": "It's very interesting that there is no any Chineseβ¦
", "rawMarkdown": "It's very interesting that there is no any Chinese...", "votes": null }, { "id": "1861183", "postDate": "07/18/2022 20:14:02", "content": "Thanks for this analysis!
", "rawMarkdown": "Thanks for this analysis!", "votes": null }, { "id": "1880224", "postDate": "08/01/2022 14:49:22", "content": "Thank you for the good analysis. I have curious if using only the top N languages frequently occurred in the training dataset is a helpful way to train the model.
", "rawMarkdown": "Thank you for the good analysis. I have curious if using only the top N languages frequently occurred in the training dataset is a helpful way to train the model.", "votes": null } ], "comments": [ { "id": 1815469, "author_name": "cid007", "author_url": "", "post_date": "06/09/2022 04:23:09", "content": "Excellent and interesting analysis @andradaolteanu .
\n A noob question here.. how about train the model on only English first and test on other languages? After that train on whole data? Is transfer learning effective on language models?
It will encourage bias over languages in your training and validation distribution.
", "votes": null, "replies": [] }, { "id": 1818619, "author_name": "cid007", "author_url": "", "post_date": "06/13/2022 01:23:11", "content": "Thanks for the info. I am new to this field.π
", "votes": null, "replies": [] } ] }, { "id": 1815756, "author_name": "virajkadam", "author_url": "", "post_date": "06/09/2022 12:06:04", "content": "Would there be other languages in the test set too? In that case, can a pre-trained language_2_language model be used as step 1 to convert to english?
", "votes": null, "replies": [ { "id": 1815913, "author_name": "andradaolteanu", "author_url": "", "post_date": "06/09/2022 16:14:30", "content": "that's the thing, we don't know if the test set mantains the same proportions as the training set - so whatever we choose it needs to be robust
", "votes": null, "replies": [] } ] }, { "id": 1815911, "author_name": "zacchaeus", "author_url": "", "post_date": "06/09/2022 16:08:58", "content": "Great insights!
", "votes": null, "replies": [] }, { "id": 1816223, "author_name": "nknarendra7", "author_url": "", "post_date": "06/10/2022 01:55:03", "content": "@andradaolteanu thanks for the discovery. I have the similar problem with one of the recent project which has combination of two language (english, native language). I have been thinking how the other language can impact the performance of Model or shouldn't be impacted at all. Any suggestions?
\nWill wait for your notebook. Thanks
Interesting and brilliant approach to see what language written over actual world. Thanks you so much to give us wonderful insights! π@andradaolteanu
", "votes": null, "replies": [] }, { "id": 1820866, "author_name": "heuingttiang", "author_url": "", "post_date": "06/15/2022 04:22:38", "content": "If it rises to the top of the leaderboard in the future, other PLM models(multilingual model) can be considered. Thank you for the good analysis. @andradaolteanu ππ
", "votes": null, "replies": [] }, { "id": 1821298, "author_name": "paijon", "author_url": "", "post_date": "06/15/2022 12:04:27", "content": "Thanks for the info. I am new to this field.π
", "votes": null, "replies": [] }, { "id": 1829109, "author_name": "aniketbiprojit", "author_url": "", "post_date": "06/22/2022 10:17:45", "content": "Hmm. This was pretty much the first blocker I ran into right after loading the dataset. Thanks for sharing the insights.
", "votes": null, "replies": [] }, { "id": 1843853, "author_name": "conan1024", "author_url": "", "post_date": "07/05/2022 07:00:15", "content": "It's very interesting that there is no any Chineseβ¦
", "votes": null, "replies": [] }, { "id": 1861183, "author_name": "artsolo", "author_url": "", "post_date": "07/18/2022 20:14:02", "content": "Thanks for this analysis!
", "votes": null, "replies": [] }, { "id": 1880224, "author_name": "nyoungwoo", "author_url": "", "post_date": "08/01/2022 14:49:22", "content": "Thank you for the good analysis. I have curious if using only the top N languages frequently occurred in the training dataset is a helpful way to train the model.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1815141": "While I was exploring the notebooks' `source` I've realised that there are a **few that aren't written in English**.\n\nThis got me wondering - how many notebooks within the `train` dataset are in another language and how does this impact the overall models.\n\nSo, this is what I've found:\n\n### The methodology\n\nI just used the basic `LanguageDetector` from the `spacy` library and then extracted the language of the first 200 chars within a notebook. I've put the `200` barrier to not overload the memory - this process took about 1hr and 30 mins.\n\n### Key takeaways\n\n* There are 44 total distinct languages present in the notebooks\n* *90% of the notebooks are in English* and only 10% of the rest are written in other languages\n* The full list of all non-English languages is as follows: ['Portuguese', 'Russian', 'Turkish', 'Japanese', 'Italian', 'Korean', 'Spanish, Castilian', 'German', 'French', 'Indonesian', 'Catalan, Valencian', 'Vietnamese', 'Tagalog', 'Romanian,\\xa0Moldavian, Moldovan', 'Norwegian', 'Danish', 'Estonian', 'Afrikaans', 'Dutch,\\xa0Flemish', 'Somali', 'Polish', 'Swedish', 'Welsh', 'Thai', 'Finnish', 'Ukrainian', 'Bulgarian', 'Persian', 'Croatian', 'Hungarian', 'Swahili', 'Slovenian', 'Lithuanian', 'Czech', 'Arabic', 'Albanian', 'Greek, Modern (1453β)', 'Latvian', 'Slovak', 'Macedonian', 'Hebrew', 'Bengali', 'Hindi']\n\n\n\n### Some thoughts\n\n* *should we delete* training notebooks that are non-english or should we try to *incorporate* them?\n* is this *90-10 proportion the same* for the `test` data too?\n\n### Data and Full Analysis\n\n* **Dataset**: the full dataset with the notebook `id` and language mapping can be found in [AI4Code - Processed Data](https://www.kaggle.com/datasets/andradaolteanu/ai4code-processed-data)\n* **Notebook** with full analysis: https://www.kaggle.com/code/andradaolteanu/ai4code-language-detection-and-model-tuning/notebook",
"1815469": "Excellent and interesting analysis @andradaolteanu .\n A noob question here.. how about train the model on only English first and test on other languages? After that train on whole data? Is transfer learning effective on language models?",
"1815752": "It will encourage bias over languages in your training and validation distribution.",
"1815756": "Would there be other languages in the test set too? In that case, can a pre-trained language_2_language model be used as step 1 to convert to english?",
"1815911": "Great insights!",
"1815913": "that's the thing, we don't know if the test set mantains the same proportions as the training set - so whatever we choose it needs to be robust",
"1816223": "andradaolteanu thanks for the discovery. I have the similar problem with one of the recent project which has combination of two language (english, native language). I have been thinking how the other language can impact the performance of Model or shouldn't be impacted at all. Any suggestions?\nWill wait for your notebook. Thanks",
"1818615": "Interesting and brilliant approach to see what language written over actual world. Thanks you so much to give us wonderful insights! π@andradaolteanu",
"1818619": "Thanks for the info. I am new to this field.π",
"1820866": "If it rises to the top of the leaderboard in the future, other PLM models(multilingual model) can be considered. Thank you for the good analysis. @andradaolteanu ππ",
"1821298": "Thanks for the info. I am new to this field.π",
"1829109": "Hmm. This was pretty much the first blocker I ran into right after loading the dataset. Thanks for sharing the insights.",
"1843853": "It's very interesting that there is no any Chinese...",
"1861183": "Thanks for this analysis!",
"1880224": "Thank you for the good analysis. I have curious if using only the top N languages frequently occurred in the training dataset is a helpful way to train the model."
},
"source": "meta"
}