Thank you kaggle and host for holding such an exciting competition. We have learned a lot.
\nHere is what I tried, it was really tough…
\ndef devanagari_number_to_arabic_number(\n answer_text: str, context: str, only_whole_numeric: bool\n) -> str:\n if only_whole_numeric and not answer_text.strip().isnumeric():\n return answer_text\n\n devanagari_to_arabic = {\n \"०\": \"0\",\n \"१\": \"1\",\n \"२\": \"2\",\n \"३\": \"3\",\n \"४\": \"4\",\n \"५\": \"5\",\n \"६\": \"6\",\n \"७\": \"7\",\n \"८\": \"8\",\n \"९\": \"9\",\n }\n\n new_answer_text = answer_text\n for devanagari_num, arabic_num in devanagari_to_arabic.items():\n new_answer_text = new_answer_text.replace(devanagari_num, arabic_num)\n if context.find(new_answer_text) > 0:\n return new_answer_text\n return answer_text\nOne interesting finding is that if the question is same or similar, the answer_text tend to be same. I tried to fill the predicted text of submission file using question similarity (calculated jaccard and levenshtein distance) with train.csv. I could only achieve 0.004, but there exist same questions and answer_texts in test data with train data. notebook
\nI hope I will achieve a solo gold in the next one ;)
\n", "comments": [ { "id": 1584790, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2021-11-16T19:00:14.873000", "content": "\n\nGroupKFold using first 20 characters of contexts as a key
\n
\n I noticed that there exist some duplicated contexts in training data. Some of them are not completely same contexts, but almost the same contexts with minor change. So I used GroupKFold using first 20 characters of contexts as a key.
Good find!
", "votes": 1, "replies": [ { "id": 1585022, "author_name": "guchio3", "author_url": "", "post_date": "2021-11-17T01:59:46.967000", "content": "Thank you~ :D
", "votes": 0, "replies": [] } ] }, { "id": 1584478, "author_name": "Ayush Singh", "author_url": "", "post_date": "2021-11-16T15:01:56.130000", "content": "Congrats @guchio3 Great solution
", "votes": 1, "replies": [ { "id": 1585021, "author_name": "guchio3", "author_url": "", "post_date": "2021-11-17T01:59:35.137000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 1584059, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-11-16T08:34:22.720000", "content": "Congrats @guchio3 ! Have you validated (models, weighs for model ensemble) on the original training data alone?
", "votes": 1, "replies": [ { "id": 1584068, "author_name": "guchio3", "author_url": "", "post_date": "2021-11-16T08:41:56.737000", "content": "Hi, thank you for your commment!
\nYes, I basically used original dataset alone for validation, but I used original one with external datasets (xquad & mlqa) for validation only when I created pseudo labels.
", "votes": 1, "replies": [] }, { "id": 1584071, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-11-16T08:45:29.270000", "content": "Thank you! That's actually what we did too. BTW I didn't catch that you used pseudo labels, did they work?
", "votes": 1, "replies": [] }, { "id": 1584085, "author_name": "guchio3", "author_url": "", "post_date": "2021-11-16T09:04:37.480000", "content": "Ah, actually my pseudo label is not for test dataset, but for validation dataset. (so it just re-labels the training data to decrease label (=answer_start) noise.)
\nIt didn't change the validation score a lot, but I think it should contribute to variation for the ensemble.
Thank you kaggle and host for holding such an exciting competition. We have learned a lot.
\nHere is what I tried, it was really tough…
\ndef devanagari_number_to_arabic_number(\n answer_text: str, context: str, only_whole_numeric: bool\n) -> str:\n if only_whole_numeric and not answer_text.strip().isnumeric():\n return answer_text\n\n devanagari_to_arabic = {\n \"०\": \"0\",\n \"१\": \"1\",\n \"२\": \"2\",\n \"३\": \"3\",\n \"४\": \"4\",\n \"५\": \"5\",\n \"६\": \"6\",\n \"७\": \"7\",\n \"८\": \"8\",\n \"९\": \"9\",\n }\n\n new_answer_text = answer_text\n for devanagari_num, arabic_num in devanagari_to_arabic.items():\n new_answer_text = new_answer_text.replace(devanagari_num, arabic_num)\n if context.find(new_answer_text) > 0:\n return new_answer_text\n return answer_text\nOne interesting finding is that if the question is same or similar, the answer_text tend to be same. I tried to fill the predicted text of submission file using question similarity (calculated jaccard and levenshtein distance) with train.csv. I could only achieve 0.004, but there exist same questions and answer_texts in test data with train data. notebook
\nI hope I will achieve a solo gold in the next one ;)
\n", "rawMarkdown": "Thank you kaggle and host for holding such an exciting competition. We have learned a lot.\n\nHere is what I tried, it was really tough...\n\n### key points of final submission\n - **GroupKFold using first 20 characters of contexts as a key**\n - I noticed that there exist some duplicated contexts in training data. Some of them are not completely same contexts, but almost the same contexts with minor change. So I used GroupKFold using first 20 characters of contexts as a key.\n - **Weighted ensemble of 7 all data train models (trained 2 epochs, weights are tuned by optuna)**\n 1. XLMRoBERTa\n 2. XLMRoBERTa using all the \"answer_text\" in the context (many positive labels)\n 3. XLMRoBERTa, filtering the positive labels of samples which have 11 or more \"ansewr_text\" in the context. I tried this to de-noise the labels because labels of samples which have more \"answer_text\" should tend to be wrong. I also filtered text batches which contain \"answer_text\" but do not hold start/end position because if this answer_text is the right one, this case should be noise, too.\n 4. XLMRoBERTa with negative under sampling. pos:neg = 1 : 1\n 5. XLMRoBERTa with negative under sampling. pos:neg = 1 : 5\n 6. XLMRoBERTa with train pseudo labeling. I tried this because pseudo labels can make the label more accurate. The first \"answer_text\" should not be right one. In addition, there exist cases where the models cannot answer the question because of text batch. If the information about question and \"answer_text\" are not in the same text batch, I think models should not answer the question correctly. Pseudo labeling should filter these cases, too.\n 7. XLMRoBERTa with train pseudo labeling, and trained only using the text batches which have \"answer_text\".\n - **[cumax function](https://github.com/guchio3/kaggle-chaii-2021/blob/c697d485c2192bab0b60abfbc521741728b8618d/src/model/chaii_model.py#L192-L203)**\n - **number text postprocessing**\n - if the predicted texts contains only Devanagari numbers, I changed it arabic ones.\n```\ndef devanagari_number_to_arabic_number(\n answer_text: str, context: str, only_whole_numeric: bool\n) -> str:\n if only_whole_numeric and not answer_text.strip().isnumeric():\n return answer_text\n\n devanagari_to_arabic = {\n \"०\": \"0\",\n \"१\": \"1\",\n \"२\": \"2\",\n \"३\": \"3\",\n \"४\": \"4\",\n \"५\": \"5\",\n \"६\": \"6\",\n \"७\": \"7\",\n \"८\": \"8\",\n \"९\": \"9\",\n }\n\n new_answer_text = answer_text\n for devanagari_num, arabic_num in devanagari_to_arabic.items():\n new_answer_text = new_answer_text.replace(devanagari_num, arabic_num)\n if context.find(new_answer_text) > 0:\n return new_answer_text\n return answer_text\n```\n\n### tried, but not used\n - **[char level ensembling](https://github.com/guchio3/kaggle-chaii-2021/blob/2d974b88c2689b1d09047947d2c8d19838320d9e/src/prediction/prediction_result.py#L50-L209)**\n - I've implemented char level ensembling logic to ensemble diverse (ex. muril) models, but I didn't use it because it decreased public LB score so much. This worked for private LB, but I couldn't adopt it anyway... \n - **2 stage modeling**\n - I thought that the models don't know which batched texts are good. So I tired 2 stage modeling. In 1st stage, I calculate the probability of holding the \"answer_text\" for each batched text. In 2nd stage, I tried 2 things. ① multiply predicted probability to logits of each batched text. ② filter top N batched text, and calculate logits using the model trained only using batched texts which have \"answer_text\". The local AUC of 1st stage was 0.964, but both approaches did not work so well.\n\nOne interesting finding is that if the question is same or similar, the answer_text tend to be same. I tried to fill the predicted text of submission file using question similarity (calculated jaccard and levenshtein distance) with train.csv. I could only achieve 0.004, but there exist same questions and answer_texts in test data with train data. [notebook](https://www.kaggle.com/guchio3/fill-sub-by-train/notebook)\n\nI hope I will achieve a solo gold in the next one ;)\n\n**[code](https://github.com/guchio3/kaggle-chaii-2021)**\n**[experiment results](https://wandb.ai/guchio3/kaggle-chaii-2021/table?workspace=user-guchio3)**", "votes": 12 }, { "id": 1584790, "postDate": "2021-11-16T19:00:14.873Z", "content": "\n\nGroupKFold using first 20 characters of contexts as a key
\n
\n I noticed that there exist some duplicated contexts in training data. Some of them are not completely same contexts, but almost the same contexts with minor change. So I used GroupKFold using first 20 characters of contexts as a key.
Good find!
", "rawMarkdown": "> GroupKFold using first 20 characters of contexts as a key\nI noticed that there exist some duplicated contexts in training data. Some of them are not completely same contexts, but almost the same contexts with minor change. So I used GroupKFold using first 20 characters of contexts as a key.\n\nGood find!", "votes": 1, "replies": [ { "id": 1585022, "postDate": "2021-11-17T01:59:46.967Z", "content": "Thank you~ :D
", "rawMarkdown": "Thank you~ :D" } ] }, { "id": 1584478, "postDate": "2021-11-16T15:01:56.130Z", "content": "Congrats @guchio3 Great solution
", "rawMarkdown": "Congrats @guchio3 Great solution", "votes": 1, "replies": [ { "id": 1585021, "postDate": "2021-11-17T01:59:35.137Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 1584059, "postDate": "2021-11-16T08:34:22.720Z", "content": "Congrats @guchio3 ! Have you validated (models, weighs for model ensemble) on the original training data alone?
", "rawMarkdown": "Congrats @guchio3 ! Have you validated (models, weighs for model ensemble) on the original training data alone?", "votes": 1, "replies": [ { "id": 1584068, "postDate": "2021-11-16T08:41:56.737Z", "content": "Hi, thank you for your commment!
\nYes, I basically used original dataset alone for validation, but I used original one with external datasets (xquad & mlqa) for validation only when I created pseudo labels.
", "rawMarkdown": "Hi, thank you for your commment!\n\nYes, I basically used original dataset alone for validation, but I used original one with external datasets (xquad & mlqa) for validation only when I created pseudo labels.", "votes": 1 }, { "id": 1584071, "postDate": "2021-11-16T08:45:29.270Z", "content": "Thank you! That's actually what we did too. BTW I didn't catch that you used pseudo labels, did they work?
", "rawMarkdown": "Thank you! That's actually what we did too. BTW I didn't catch that you used pseudo labels, did they work?", "votes": 1 }, { "id": 1584085, "postDate": "2021-11-16T09:04:37.480Z", "content": "Ah, actually my pseudo label is not for test dataset, but for validation dataset. (so it just re-labels the training data to decrease label (=answer_start) noise.)
\nIt didn't change the validation score a lot, but I think it should contribute to variation for the ensemble.