{ "id": 287938, "title": "24th place solution & attempt – only with XLM-RoBERTa🍵", "content": "

Thank you to Kaggle and host for the opportunity to have a fun competition.

\n

To be honest, I did not understand how to ensemble models with different tokenizers like XLM-Roberta and Muril. Therefore, I used only XLM-Roberta to create the models, and I think this is the reason why my score did not improve enough. Now I am impressed with other people's solutions and how they ensemble like this.

\n
    \n
  1. At the beginning of the competition, I first worked on data augmentation. I tried to create Hindi and Tamil version of SQUAD using googletranslate, as many people had done by now. However, instead of increasing my score, it decreased significantly. I threw it away immediately.
    2. \n
  2. I tried to make separate Hindi and Tamil models. This also lowered my score significantly. I took this fact against me and guessed that adding data from other similar languages would increase the score.
    3. \n
  3. I decided to add TyDi-QA's Bengali and Telugu to the study. I think my score went up reasonably well. (CV and LB were both unstable, so it's hard to show exactly how much they improved.)
    4. \n
  4. I cleaned the context of the training data. First, I removed useless line breaks and spaces, but it had no effect on the score. Next, I automatically removed strings that looked like tags in the Wikipedia source code using regular expressions. This also did not cause any significant change in the score.
    5. \n
  5. I created multiple models and ensembled them by changing the max_length (the unit of context division) in various ways, such as 320, 384, and 448. This was inspired by my previous experience in Melanoma competition where we were able to increase our score by creating models with different image sizes. As a result, my score increased significantly.
    6. \n
  6. When I changed the doc_stride (the length of the overlap when splitting the context) in various ways, I noticed that the score fluctuated greatly. In my experiments with CV, I got relatively high scores in the range of doc_stride=150~190 for max_length=512. In the end, I chose doc_stride=176,180, but the scores were very unstable and it was difficult to decide on one doc_stride. In a way, I thought this competition had a \"lottery by doc_stride\" aspect to it (which it didn't, actually).
    7. \n
\n

I wanted to find out what the teams \"tkm kh\" and \"zzy + Zacchaeus\", who dominated the top positions from the beginning of this competition, were aware of, and I wanted to find out too. I learned a lot.
\n@thedrcat, congratulations on your win! I can learn a lot from your solution.
\n@tkm2261, @kanbehmw, @zzy990106, @zacchaeus, and all the other participants deserve my respect🍵

", "comments": [], "comment_count": 0, "messages": [ { "id": 1583703, "postDate": "2021-11-16T02:35:31.970Z", "content": "

Thank you to Kaggle and host for the opportunity to have a fun competition.

\n

To be honest, I did not understand how to ensemble models with different tokenizers like XLM-Roberta and Muril. Therefore, I used only XLM-Roberta to create the models, and I think this is the reason why my score did not improve enough. Now I am impressed with other people's solutions and how they ensemble like this.

\n
    \n
  1. At the beginning of the competition, I first worked on data augmentation. I tried to create Hindi and Tamil version of SQUAD using googletranslate, as many people had done by now. However, instead of increasing my score, it decreased significantly. I threw it away immediately.
    2. \n
  2. I tried to make separate Hindi and Tamil models. This also lowered my score significantly. I took this fact against me and guessed that adding data from other similar languages would increase the score.
    3. \n
  3. I decided to add TyDi-QA's Bengali and Telugu to the study. I think my score went up reasonably well. (CV and LB were both unstable, so it's hard to show exactly how much they improved.)
    4. \n
  4. I cleaned the context of the training data. First, I removed useless line breaks and spaces, but it had no effect on the score. Next, I automatically removed strings that looked like tags in the Wikipedia source code using regular expressions. This also did not cause any significant change in the score.
    5. \n
  5. I created multiple models and ensembled them by changing the max_length (the unit of context division) in various ways, such as 320, 384, and 448. This was inspired by my previous experience in Melanoma competition where we were able to increase our score by creating models with different image sizes. As a result, my score increased significantly.
    6. \n
  6. When I changed the doc_stride (the length of the overlap when splitting the context) in various ways, I noticed that the score fluctuated greatly. In my experiments with CV, I got relatively high scores in the range of doc_stride=150~190 for max_length=512. In the end, I chose doc_stride=176,180, but the scores were very unstable and it was difficult to decide on one doc_stride. In a way, I thought this competition had a \"lottery by doc_stride\" aspect to it (which it didn't, actually).
    7. \n
\n

I wanted to find out what the teams \"tkm kh\" and \"zzy + Zacchaeus\", who dominated the top positions from the beginning of this competition, were aware of, and I wanted to find out too. I learned a lot.
\n@thedrcat, congratulations on your win! I can learn a lot from your solution.
\n@tkm2261, @kanbehmw, @zzy990106, @zacchaeus, and all the other participants deserve my respect🍵

", "rawMarkdown": "Thank you to Kaggle and host for the opportunity to have a fun competition.\n\nTo be honest, I did not understand how to ensemble models with different tokenizers like XLM-Roberta and Muril. Therefore, I used only XLM-Roberta to create the models, and I think this is the reason why my score did not improve enough. Now I am impressed with other people's solutions and how they ensemble like this.\n\n1. At the beginning of the competition, I first worked on data augmentation. I tried to create Hindi and Tamil version of SQUAD using googletranslate, as many people had done by now. However, instead of increasing my score, it decreased significantly. I threw it away immediately.\n2. I tried to make separate Hindi and Tamil models. This also lowered my score significantly. **I took this fact against me and guessed that adding data from other similar languages would increase the score.**\n3. **I decided to add TyDi-QA's Bengali and Telugu to the study. I think my score went up reasonably well. (CV and LB were both unstable, so it's hard to show exactly how much they improved.)**\n4. I cleaned the context of the training data. First, I removed useless line breaks and spaces, but it had no effect on the score. Next, I automatically removed strings that looked like tags in the Wikipedia source code using regular expressions. This also did not cause any significant change in the score.\n5. **I created multiple models and ensembled them by changing the max_length (the unit of context division) in various ways, such as 320, 384, and 448. This was inspired by my previous experience in Melanoma competition where we were able to increase our score by creating models with different image sizes. As a result, my score increased significantly.**\n6. When I changed the doc_stride (the length of the overlap when splitting the context) in various ways, I noticed that the score fluctuated greatly. In my experiments with CV, I got relatively high scores in the range of doc_stride=150~190 for max_length=512. In the end, I chose doc_stride=176,180, but the scores were very unstable and it was difficult to decide on one doc_stride. In a way, I thought this competition had a \"lottery by doc_stride\" aspect to it (which it didn't, actually).\n\nI wanted to find out what the teams \"tkm kh\" and \"zzy + Zacchaeus\", who dominated the top positions from the beginning of this competition, were aware of, and I wanted to find out too. I learned a lot.\n@thedrcat, congratulations on your win! I can learn a lot from your solution.\n@tkm2261, @kanbehmw, @zzy990106, @zacchaeus, and all the other participants deserve my respect🍵", "votes": 11 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:22:38Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7253416, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/24th-place-solution-attempt-only-with-xlm-roberta" }