# EDIT) Private 1089th Place Approach (BIG DROP from public 7th) EDIT) We shared our approach with concern about shake-up before private leaderboard was revealed. Unfortunately, we experienced way bigger shake-up than we expected in private LB. Since we have no objective evidence about our bad private score, we cannot specify which part was wrong in our submissions. So please take care while reading our approach. +) Congratulations for winners and those who did overcome shake-up. ================ Our Sharing Before Private LB =============== 0. Intro It was great experience for us to participate in this competition. So we want to say thank you to organizers of this challenge and others helped us a lot. [@pudae81](/pudae81) Your great introductory seminar about kaggle and code competition helped us a lot in joining this competition. We'll keep our fingers crossed for your journey to new career. 🤞 [@humananalog](/humananalog) We couldn't make it this far without great jobs shared by you. We really appreciates your sharing of awesome training and inference kernels. And that's why we share our humble approach to community even if we're not sure whether it's overfitted to public LB or not. 1. Overview of Our Approach ![Method Overview](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure1.png) original image: [https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure1.png](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure1.png) Figure above shows our approach. We used MTCNN as face detector and attention-based models as prediction model. 2. Face Detector : MTCNN We used MTCNN to detect face in video frame. We resized frame to 1280x720 before we pass it to MTCNN. And parameters passed to MTCNN is - thresholds of [0.6, 0.7. 0.95] - min_face_size of (720/30) 3. Model Architecture ![Model Architecture](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure2.png) original image: [https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure2.png](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure2.png) Model architecture we used is shown above. We first extract feature vectors from each face crop, then apply self-attention based model to sequence of feature vectors to aggregate information. ImageNet pretrained efficient-b4 was used as our face crop feature extractor. We froze first 4 blocks due to our GPU memory capability. We have two similar models to get prediction of video from sequence of sampled face features. 3.2.1. Model 1 ![Model 1 figure](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure3.png) original image: [https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure3.png](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure3.png) Our first model uses self-attention between all sampled faces. "self-attention module" in figure contains KQV projection, self-attention, and residual connection. Then fully-connected layer follows to get prediction for each face crop. Finally, we take max prediction value for video prediction values. The best score of this model in LB was 0.24827 without any TTA. 3.2.2. Model 2 ![Model 2 figure](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure4.png) original image: [https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure4.png](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/figure4.png) As figure shows, our second model takes self-attention between faces in the same frame first, then takes self-attention between all sampled faces. After that, the procedure is same with Model 1. The best score of this model in LB was 0.25178 without any TTA. 4. Training Method 4.1. Face Crops Preparation sample 100 frames uniformly for each videos. use MTCNN to detect face in frame. 2x extra margin added to face crop box(so the size is 9 times bigger in area sense) for further augmentation at train time face aligned with five face landmarks. 4.2. Train/Valid Split One of our teammates clustered all real videos so that no actor appears in the videos from different clusters at the same time. After that, he splitted some clusters as validation set so that validation set has videos of different resolutions. With this cluster based real video split, their corresponding fake videos are splitted to the group of their original videos. Every train epoch, we used all real videos and sampled fake videos randomly to balance the number of real and fake samples. 4.3. Augmentation at train time Sample 1 to 20 frames from previsously sampled 100 frames and load corresponding face crops. Videowise (i.e. applied to all face crops samely) DFDC preview dataset paper based quality augmentation [](https://arxiv.org/abs/1910.08854) [](https://arxiv.org/abs/1910.08854) [https://arxiv.org/abs/1910.08854](https://arxiv.org/abs/1910.08854) Various image transforms such as rotate, equalize, solarize, posterize etc... Random horizontal flip Imagewise Random Crop Randomness occures in margin and face-center translation Resize Interpolation we use three different resize interpolation : bilinear, bicubic, nearest 4.4. Optimizer We used same optimizer settings for both model. - SGD with momentum 0.9 - Learning Rate Schedule - 5 epochs of linear warm-up from 0 to 0.001 - Cosine annealing from 0.001 to 0 through 35 epochs (total 40 epochs) 4.5. Further Regularization weight decay: 0.0001 dropout layer right before final fully connected layer: p=0.4 label smoothing epsilon: 0.01 4.6. Validation losses of model 1 while training ![Model 1 loss](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/model1.png) original image: [https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/model1.png](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/model1.png) 4.7. Validation losses of model 2 while training ![Model 2 loss](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/model2.png) original image: [https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/model2.png](https://raw.githubusercontent.com/jinhanpark/Deepfake-Detection-Challenge/master/assets/model2.png) 5. Inference Strategy With our inference strategy below, we got 0.23086 in LB 5.1. Sample Faces Sample 20 frames per video uniformly. Detect and crop at most 3 faces from each frame with threshold stated above. 5.2. Ensemble We averaged predictions from two models 5.3. Test Time Augmentation for each model For each face detected, make 8 different crops of different margin and offset - 2 of 8 : two different margins of 1.1 and 1.15 without offsets. - 2 of 8 : two different y-axis offset of 0.15, -0.15 with margin of 1.15 - 4 of 8 : horizontal flips of above 4 crops 6. Outro Thank you for reading. All questions are welcome.