# [ABBA McCandless] - 2nd place solution overview 

A summary of our solution - 0.945/0.932 (Public/Private)
In short
: Trust your CV
I used 4 folds stratified by image size and quality factor with 20k holdout (5K Cover + 15K corresponding positive images). My team has a different split w/o folds but also with a holdout. We used the ranked average of second-level stacking (CatBoost and XGBoost) to blend individual models into ensembles and merge their predictions.
Models
I used only B6/B7, and my teammates had B4/B5/B6/B7/MixNet plus SRNet and some hand-crafted features from "classical" steganalysis.
I trained my models on RGB input (using cv2.imread) and then fine-tuned on non-rounded RGB images (by manually decoding DCT->YCbCr -> RGB omitting rounding step). Lastly, I replaced Swish activation to Mish, which improved every model's score in my ensemble by +0.00044 on average.
Augmentations
Like probably everyone, I've used D4 augmentations with coarse dropout for the training and D4 augmentations during the inference.
Losses
I trained two heads for my models to predict binary (cover/stego) and multi-class (4 classes) outcomes. Based on early experiments, this showed to work well and speed up the convergence a little. I've played with different loss functions, including weighted BCE/CE, focal loss, and directly optimizing RoC AUC. But plain BCE/CE proved to work the best.
Ensembling
For ensembling I had tested many ideas, but the final solution was to train XGBoost classifier using predictions of my B6/B7 models. Second level stacking was done using holdout predictions. The feature matrix was 20000x380 made from binary and multi-class outputs, and all intermediate logits from TTA). I ran 5-fold cross-validation using groupkfold split on image_id to find the best hyperparameters for XGboost, and that was my ensemble.
My team made a similar approach to their model zoo but trained CatBoost stacker. We selected our ensembles for the final blend based on individual CV scores and Spearman correlation. The lower correlation factor - the higher LB score on the public we had.
To come to this solution, it took almost two months. There have been dozens of unsuccessful training runs and experiments on this way to gold. To name a few:
* Training models on DCT input (Didn't even pass 0.92)
* Training ResNet and DenseNet family (Barely reached 0.935 LB)
* Training model to always have Cover + Stego image pair in batch 
* Direct Roc AUC optimization
* Metric learning with ArcFace
* Training with extra data (iStego 100K)
* B0-B7, Heavier models performed better
* Training stacking model on embeddings
* Training on TPU. In general, it worked, but Kaggle TPU machine is limited in its CPU/RAM, so the training speed was deficient.