# 18th place solution: label variation

Thank you kaggle and host for holding such an exciting competition. We have learned a lot.
Here is what I tried, it was really tough…
key points of final submission
GroupKFold using first 20 characters of contexts as a key
I noticed that there exist some duplicated contexts in training data. Some of them are not completely same contexts, but almost the same contexts with minor change. So I used GroupKFold using first 20 characters of contexts as a key.
Weighted ensemble of 7 all data train models (trained 2 epochs, weights are tuned by optuna)
XLMRoBERTa
XLMRoBERTa using all the "answer_text" in the context (many positive labels)
XLMRoBERTa, filtering the positive labels of samples which have 11 or more "ansewr_text" in the context. I tried this to de-noise the labels because labels of samples which have more "answer_text" should tend to be wrong. I also filtered text batches which contain "answer_text" but do not hold start/end position because if this answer_text is the right one, this case should be noise, too.
XLMRoBERTa with negative under sampling. pos:neg = 1 : 1
XLMRoBERTa with negative under sampling. pos:neg = 1 : 5
XLMRoBERTa with train pseudo labeling. I tried this because pseudo labels can make the label more accurate. The first "answer_text" should not be right one. In addition, there exist cases where the models cannot answer the question because of text batch. If the information about question and "answer_text" are not in the same text batch, I think models should not answer the question correctly. Pseudo labeling should filter these cases, too.
XLMRoBERTa with train pseudo labeling, and trained only using the text batches which have "answer_text".
[cumax function](https://github.com/guchio3/kaggle-chaii-2021/blob/c697d485c2192bab0b60abfbc521741728b8618d/src/model/chaii_model.py#L192-L203)
number text postprocessing
if the predicted texts contains only Devanagari numbers, I changed it arabic ones.
def devanagari_number_to_arabic_number(
    answer_text: str, context: str, only_whole_numeric: bool
) -> str:
    if only_whole_numeric and not answer_text.strip().isnumeric():
        return answer_text

    devanagari_to_arabic = {
        "०": "0",
        "१": "1",
        "२": "2",
        "३": "3",
        "४": "4",
        "५": "5",
        "६": "6",
        "७": "7",
        "८": "8",
        "९": "9",
    }

    new_answer_text = answer_text
    for devanagari_num, arabic_num in devanagari_to_arabic.items():
        new_answer_text = new_answer_text.replace(devanagari_num, arabic_num)
    if context.find(new_answer_text) > 0:
        return new_answer_text
    return answer_text
tried, but not used
[char level ensembling](https://github.com/guchio3/kaggle-chaii-2021/blob/2d974b88c2689b1d09047947d2c8d19838320d9e/src/prediction/prediction_result.py#L50-L209)
I've implemented char level ensembling logic to ensemble diverse (ex. muril) models, but I didn't use it because it decreased public LB score so much. This worked for private LB, but I couldn't adopt it anyway…
2 stage modeling
I thought that the models don't know which batched texts are good. So I tired 2 stage modeling. In 1st stage, I calculate the probability of holding the "answer_text" for each batched text. In 2nd stage, I tried 2 things. ① multiply predicted probability to logits of each batched text. ② filter top N batched text, and calculate logits using the model trained only using batched texts which have "answer_text". The local AUC of 1st stage was 0.964, but both approaches did not work so well.
One interesting finding is that if the question is same or similar, the answer_text tend to be same. I tried to fill the predicted text of submission file using question similarity (calculated jaccard and levenshtein distance) with train.csv. I could only achieve 0.004, but there exist same questions and answer_texts in test data with train data.
[notebook](https://www.kaggle.com/guchio3/fill-sub-by-train/notebook)
I hope I will achieve a solo gold in the next one ;)
[code](https://github.com/guchio3/kaggle-chaii-2021)
[experiment results](https://wandb.ai/guchio3/kaggle-chaii-2021/table?workspace=user-guchio3)