# 41-th Place Solution

DATA PREPROCESSING
NER
: we pretrained NER-model (bert-base) for code blocks replacement with special token [CODE]
Reg.exp.
: we cleaned up LaTeX snippets and replaced them with [MATH] token
Crop strategy
: from both side (we cut the middle part, if sequence was longer than 450 tokens)
Max. seq. lengths
: 450
MODEL PARAMETERS
Model architecture
: roberta-large
Pretrained
: Yes (from transformers library)
Model design
: siamese bi-encoder (dot-product of question-answer pair -> sigmoid)
Pooling strategy
: [CLS] token
TRAINING PARAMETERS
Optimizer
: AdamW (from transformers library)
Max grad. norm
: 5
Epochs
: 5
Batch size
: 8 (1 * 8 accum. steps)
LR
: 2.5e-05
LR-scheduler
: linear
Warmup steps
: 50
Seed
: 228 :)
ENSEMBLE PARAMETERS
Blend of
: 7-fold models
Fold strategy
: GroupKFold
Blend strategy
: probabilities averaging
POST-PROCESSING
For each target:
1. Sort values (ascending)
2. Initialize an anchor value (the smallest one)
3. Iterate over each value and calculate the difference between it and anchor
4. If difference > TARGET-EPSILON, then collapse all values between anchor and current value into one bin
5. Update anchor (set current value)
6. Repeat from 3 until end
Finally, normalize bins to (0, 1) range
TARGET-EPSILONs (for each) target estimated via out of fold predictions (from 7 folds)
ANOTHER TRICKS
SWA
: implemented, but did not have time to submit
Categorical features
: was concatenated with bi-encoder output. No improvements
Additional data
: few additional data was added to fill up some flawed targets. Did not help
Combine crop strategies
: different crop strategies (from left, rigth, both and middle parts) were combined as a TT augmentations. Did not help
Different pooling strategies
: average pooling, average pooling of N last layers. [CLS] was the best one
Freezing
differential, full, only n layers, etc. No freezing was the best
Reproducibility
All code to reproduce our experiments could be found here:
[https://github.com/alexeykarnachev/kaggle_google_qa_labeling](https://github.com/alexeykarnachev/kaggle_google_qa_labeling)