# 18th place solution (hypotheses on the datasets)

Thanks to Google and Kaggle for hosting this exciting competition. Also, congratulations to all teams!
I'm really happy to see many great solutions in discussion threads right after the end of the competition. It made me want to share my own solution and insights with others.
Hypotheses
Before describing modeling and post-processing, let me start by explaining about my hypotheses on this competition's datasets.
Meaning of Target Values
We see that each label has a strange set of discrete target values such as {3/9, 4/9, 1/2, 5/9, 6/9, 7/9, 5/6, 8/9, 9/9}). Although it has been already discussed in some threads like
[https://www.kaggle.com/c/google-quest-challenge/discussion/120772](https://www.kaggle.com/c/google-quest-challenge/discussion/120772)
, I have a different hypothesis on how these target values are generated. My hypothesis mainly consists of the following points: 
1. Each target value is obtained from three raters. 
2. Each rater chooses one of multiple answer values when annotating some label value. Each label has its own set of answer values. For example, question_well_written has {1, 2, 3}, question_type_choice has {0, 1}, and answer_satisfaction has {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}. I assume answer values 1, 2, and 3 for question_well_written corresponds to "No", "Maybe", "Yes", respectively.
3. Each target value corresponds to (the sum of three annotator's answer values) / (the maximum possible sum of answer values). For example, if three annotators give answer values 2, 2, and 3, respectively for question_well_written, its target value becomes (2 + 2 + 3) / (3 + 3 + 3) = 7 / 9. 
4. We are able to have target values {3/9, 4/9, 5/9, 6/9, 7/9, 8/9, 9/9} for question_well_written with 1., 2., and 3. However, it doesn't cover some of actual values 1/2 and 5/6. I guess these values are obtained by cases where there are only two raters for some target values.
Although 4. makes me less confident on my hypothesis, I still believe having three raters for each target is more realistic in comparison with having more than 10 raters for each target.
Train/Test split
We can obtain the following information:
- There are multiple question-answer pairs sharing common questions in the training dataset. 
- There are no such pairs in the test dataset (both public and private). 
- The training and test datasets don't share any common question.
This list makes me assume the following dataset split:
- Training: (questions with frequency = 1) + (questions with frequency > 1). 
- Test (public):  questions with frequency = 1
- Test (private): questions with frequency = 1. 
- These three datasets don't share common questions.
I started using only questions with frequency = 1 for validation based on this assumption. It changed my single model's CV score from 0.420 to 0.382, which is much closer to my score in test datasets.
Modeling
Loss function
I used binary cross entropy for each target after applying rank-based min-max scaling. Min-max scaling is same with the one mentioned in 3rd place solution (
[https://www.kaggle.com/c/google-quest-challenge/discussion/129927](https://www.kaggle.com/c/google-quest-challenge/discussion/129927)
).  The use of min-max scaling is based on my hypothesis on target values.  For example, I assume 3/9 for question_well_written means very strong "No".
In addition, I replaced each target value with its rank before min-max scaling. It boosts models' performance on labels having skewed target value distributions. The following figure is a distribution of answer_well_written's target values. In this case, separating questions with target values 8 / 9 and 1 has more impact on Spearman's rho in comparison with separating questions with target values 0.5 and 8 / 9. The use of rank-based min-max scaling convert a set of target values from  {3/9, 4/9, 1/2, 5/9, 6/9, 7/9, 5/6, 8/9, 9/9} to {0, 0.001, 0.004, 0.008, 0.042, 0.137, 0.233, 0.482, 1.000} and encourages models to detect differences between questions that result in larger impacts on Spearman's rho. I got +0.01 by rank-based min-max scaling in CV.
![answer_well_written](assets/f676f64167ca08bd62165754.png)
Architecture
My solution is based on two architectures.
Model 1
(Public LB = 0.392, CV = 0.383)
It consists of a single BERT (or XLNet) encoder. Input comprises of question_title (no truncation), question_body (head-tail truncation), and answer (head-trail truncation). The figure below depicts the overview of its structure. Although all labels share the same encoder, each label has its own independent classification head. Features for each label are collected from three sections with three pooling methods, max-pooling, avg-pooling, and attention-pooling. Then, collected features are passed to a dropout and a single linear layer.
![model1](assets/7eab71464fbaedac91d55d00.jpg)
Model 2
(Public LB = N/A, CV = 0.390)
My second architecture consists of two encoders. It's inspired by
[https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer](https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer)
. The figure below depicts the overview of its structure. The first encoder takes question_title (no truncation) and question_body (head-tail truncation) as its input, and the second one takes question_title (no truncation) and answer (head-tail truncation) as its input.  Its head classifier's structure and a features extraction method is almost same with Model 1.
![model2](assets/ca954f5be9dfe88e2f6ad235.jpg)
Ensemble:
Model 1 (BERT) * 0.2 + Model 1 (XLNet) * 0.2 + Model 2 (BERT) * 0.3 + Model 2 (XLNet) * 0.3
CV: 0.404
Training settings
Model 1
BERT: batch size = 12, optimizer = AdamW, lr = 5e-5
XLNet: batch size = 10, optimizer = AdamW, lr = 4.5e-5
Model 2
BERT: batch size = 12 (6 * 2), optimizer = AdamW, lr = 5e-5
XLNet: batch size = 12 (4 * 3), optimizer = AdamW, lr = 4.5e-5
Post Processing
I tried to find a list of segments that should have same values for each label by using a greedy algorithm based on OOF predictions. 
1. First, this algorithm tries to find a value x1 such that we can maximize Spearman's rho on OOF predictions by replacing prediction values in [0, x1) with x1 / 2.
2. Then, it tries to a value x2 (>= x1)  such that we can Spearman's rho by replacing prediction values in [x1, x2) with (x1 + x2) / 2
3. Keep this procedure while it successfully finds such xi.
I checked score improvement on all labels by 3-fold validation, and I decided to use this post processing on labels that I could see score improvement larger than 5e-4. This post processing increased both of CV and LB scores by around 0.03.
External data for predicting question_type_spelling
If we open an URL for some stackexchange page (
[https://english.stackexchange.com/questions/522357/when-did-spelling-ic-words-ick-start-stop-being-popular](https://english.stackexchange.com/questions/522357/when-did-spelling-ic-words-ick-start-stop-being-popular)
), we can see there are some tags at the bottom of question bodies. There are tags that are closely related to spelling (e.g., "orthography" and "suffixes"), and we can retrieve tags for almost all questions in the competition's datasets.
I collected top-200 tags that most frequently appear with "orthography" and chose 70 from them by removing irrelevant ones like "american-english".  I made a very simple classifier that just returns (host is "english.stackexchange.com" or "ell.stackexchange.com) & (question has one of 70 tags). This classifier gives me high Spearman's rho (0.5-0.7) for question_type_spelling in the training dataset and the public test dataset. However, its score looks worse than scores obtained by a standard use of model prediction and post-processing. I'm so sad...