# 10th place solution 

Top 10th Place Solution for the CIDAUT AI Fake Scene Classification 2024 Competition
1.
Context
Business context
:
[Competition Overview](https://www.kaggle.com/competitions/cidaut-ai-fake-scene-classification-2024/overview)
Data context
:
[Competition Data](https://www.kaggle.com/competitions/cidaut-ai-fake-scene-classification-2024/data)
In this competition, the task was to classify images into two categories:
editada
(i.e., “fake” or retouched)
real
(i.e., an authentic, unedited image)
2.
Overview of the Approach
2.1 Models and algorithms used
To address this problem, we trained and compared
three image classification models
from the
torchvision.models
library:
Vision Transformer (ViT)
Model based on the paper
["An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale"](https://arxiv.org/abs/2010.11929)
.
Public score around
0.9263
.
EfficientNet V2
A variant of EfficientNet described in
[EfficientNetV2: Smaller Models and Faster Training](https://arxiv.org/abs/2104.00298)
.
Best public score of
0.9476
(this placed us in the Top 10 on the leaderboard).
RegNet
A family of models introduced in the paper
["Designing Network Design Spaces"](https://arxiv.org/abs/2003.13678)
.
Public score of
0.9300
(though I suspect with proper tuning it could surpass EfficientNet).
We imported the official pretrained weights from
torchvision
, then replaced the last layer (
head
or
classifier
) to output 2 classes (one for the
editada
class and one for the
real
class).
2.2 Data preprocessing, feature engineering, and/or feature selection
Image size
:
We standardized all images to
224x224
(or
256x256
with a
CenterCrop(224)
).
Augmentations
:
RandomHorizontalFlip
,
RandomResizedCrop(224)
, etc. to enrich the dataset and reduce overfitting.
The mean and std for normalization are the usual ImageNet values (
[0.485, 0.456, 0.406]
/
[0.229, 0.224, 0.225]
).
No complex feature engineering
(the features are automatically learned by the CNNs/Transformers).
No manual feature elimination
(we used all channels and layers provided by the pretrained weights).
2.3 Validation strategy
To evaluate our models, we primarily used a
train/validation split
:
In our final notebooks
:
A simple split (e.g., 95% / 5% or 85% / 15% depending on the iteration).
We tracked the
AUC
on the validation set and the Kaggle public leaderboard.
Before making official submissions
(to test more strategies, e.g., a “three-way split”):
70% / 15% / 15%, which gave us a mini local test set.
This helped us verify overfitting risks.
With EfficientNetV2 and RegNet, we repeatedly reached a train AUROC of 1.0, demonstrating that the models were powerful enough to achieve very strong results.
In the end, the best final validation approach mirroring the final LB was a 95% / 5% split, despite some risk of overfitting.
3.
Details of the Submission
3.1 Important/impactful strategies/decisions
Comparing three model families
:
We knew that ViT (Vision Transformer), EfficientNet, and RegNet are all competitive for image classification.
The idea was to explore all three in parallel to find a balance among convergence speed, overfitting, and final score.
Learning rate and Scheduler
:
We fixed the
learning rate
at
5e-5
and applied a
StepLR(gamma=0.7)
to reduce the LR every epoch.
This played a crucial role in preventing overfitting (especially for RegNet).
Data augmentation
:
Horizontal flip, random crop
Choice of batch size
:
batch_size=8
achieved the best results for each model.
EfficientNet V2
:
Clearly the top-performing model on this dataset, reaching
0.9476
on the leaderboard.
Lighter than RegNet, converges quickly, and seems to generalize better.
3.2 Creative or non-obvious strategies/decisions
Small test-time augmentation (TTA)
:
In a few runs, we averaged predictions from the horizontally flipped image + the original.
Very slight improvement on the LB, so we left it out in the final version to keep the code shorter.
Comparison of a “three-way split” locally
:
With a mini local test set to estimate generalization.
This confirmed that RegNet (with more epochs) easily overfits (train AUC=1.0) while EfficientNet V2 was better at generalizing.
3.3 What didn’t work well
Aggressive augmentations
(e.g., extreme rotations, strong color jitter):
Paradoxically, this worsened the AUC. The dataset seemed too “straightforward” to benefit from heavy augmentations.
Increasing input size to 384x384
:
No major improvement in our runs (and more expensive training).
Stacking more final layers
or heavier
fine-tuning
of intermediate layers**:
We mostly froze the pretrained
backbones
in the first blocks and “unfroze” the last portion.
For ViT, fully fine-tuning didn’t significantly improve the final score.
4.
Personal Reflections
Below are some sample plots of training and validation curves for each model tested:
![figure 1](https://www.kaggleusercontent.com/kf/218507867/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..X8A_sbwng6grYb4Ff3xORw.jSu0TAkCoVwuhrW_adjEBYdB5fpsA_GiNgcA4Y8tfJ5XiQjsWIPYScsypAHxFequwg10jYFioa0-GXLqr0QCUWtAFdaD3phCl7FGiYPFqkSEvve7FYS_Q3muwGYjCsoICawjqAM7-0O_U_RiVGyOz450e_t7F-OrurcwfGpa5lV-UaEDwlch5kcV1BoTdDdTvx7z6-fljtH8X_MxIAuBP_QnzA_7TQiriy2A7YTzviNuUaDkmvJAxe25ucOjVhtduuobbl4Dugxhr1JuLfJQMqJxnoOJKYwT-KfDQTmz__ZKA-WhEZOXGPHU83yadGOA4bhGei4ga6VhknVUH--ZFHatlTDukyEKBYf4YEJ5skp8woFS1txmyylW0rv5-V51nDS-dQfjrw2_WHi377bea5tkzG7lm6ry8KhjJrBK7gH8--dhM0RPajfHhJAZtb2B0l3TOyzt6XFSsprarGQi8XvJ-UEheXAxZfhmiCNIA9FHHT21RgyYKBwHpQC-rvm6aDF1bob8KrfTnw0y2m77QaHUUCZdo_5pa-a8YDnAb14qIbU2Ux8E00UN_pHylMzlc15JXssiJCjhdlU4_pkkRpaex-_IiYpK3r5XWGLBTXt_E0dAAP_40bv8sLYMyhb0g1qLLjllsB02LnHZMc-aEhhvL_QTknsWXOGgsHd3zo1ZrkzLLZ4246nyVW6HHzoC.w16zKoj8Bo80Ong-gFK15w/__results___files/__results___35_0.png)
![figure 2](https://www.kaggleusercontent.com/kf/218507867/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..X8A_sbwng6grYb4Ff3xORw.jSu0TAkCoVwuhrW_adjEBYdB5fpsA_GiNgcA4Y8tfJ5XiQjsWIPYScsypAHxFequwg10jYFioa0-GXLqr0QCUWtAFdaD3phCl7FGiYPFqkSEvve7FYS_Q3muwGYjCsoICawjqAM7-0O_U_RiVGyOz450e_t7F-OrurcwfGpa5lV-UaEDwlch5kcV1BoTdDdTvx7z6-fljtH8X_MxIAuBP_QnzA_7TQiriy2A7YTzviNuUaDkmvJAxe25ucOjVhtduuobbl4Dugxhr1JuLfJQMqJxnoOJKYwT-KfDQTmz__ZKA-WhEZOXGPHU83yadGOA4bhGei4ga6VhknVUH--ZFHatlTDukyEKBYf4YEJ5skp8woFS1txmyylW0rv5-V51nDS-dQfjrw2_WHi377bea5tkzG7lm6ry8KhjJrBK7gH8--dhM0RPajfHhJAZtb2B0l3TOyzt6XFSsprarGQi8XvJ-UEheXAxZfhmiCNIA9FHHT21RgyYKBwHpQC-rvm6aDF1bob8KrfTnw0y2m77QaHUUCZdo_5pa-a8YDnAb14qIbU2Ux8E00UN_pHylMzlc15JXssiJCjhdlU4_pkkRpaex-_IiYpK3r5XWGLBTXt_E0dAAP_40bv8sLYMyhb0g1qLLjllsB02LnHZMc-aEhhvL_QTknsWXOGgsHd3zo1ZrkzLLZ4246nyVW6HHzoC.w16zKoj8Bo80Ong-gFK15w/__results___files/__results___35_1.png)
Figures 1 and 2
show
EfficientNet
results:
We notice decent generalization but some fluctuations in validation accuracy and loss from epoch to epoch.
The best public leaderboard score reached about 0.9476.
![figure 3](https://www.kaggleusercontent.com/kf/218364683/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..W4ne8TOi7psjitxhBUJb-w.ztPc5SZr3OT3ZlQ08JLJzzYsEYmQlcvx6c9jwHVSsVS7Z-D1ueftooQ_kqOTlWTrFJE43EHL008Vpnef-XLBrEudqCInmHkY3eTtnTqImmImBloiQyGa8N7MgjT_ufcFwQuWBQN9bdo9_gC6R-Zjp4glHNNbhv_NuHtZXuHEaFqqq_pkk6vaA-xBY8EuBmF_XII_BjelUtD4acwM9fmpbIA3AxWG11qzam5yt22gkDLEB930WG-1Z18qfrJiMhGthfainoNxXfxRVFdvnrf5YNr6xOtjHIdeF_lGK4e8ZTahrL4mRWGdqnpUPvP4I_7XOfytNlpfWL-DB9SzXB822EupaUML_jGh-CkyKnrYYvDbmh57ptQx3dwmFjFKEKuEJbB7ez1Gzh6D_ysYy0LtGwdOdwfpLvW286F7Gt35vL_ATsAOIZ79Qn7cOf0Qbnmo7Q8XP7vvpE7fy7DywVF8LnpkYdqMIuTO9q6pJg1Mi8nZEwkgKkH2EVvIeVHD6p-i-irZmv7Bugcm7xvSSFtEiYG_8HgeBJEusc-vpO4cwxkjYkWG29lCOqjRK285AZ1FhyyXx-LqXqXXlipmZWvWN9BPFQccQh5bfiUNz064JbowLvGGVjGN06cTFvzHKzvT2cSv-idHH5sXRF1kl0wWuw.kCV5oHHuLVRrABAsxe3PFg/__results___files/__results___11_0.png)
![figure 4](https://www.kaggleusercontent.com/kf/218364683/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..W4ne8TOi7psjitxhBUJb-w.ztPc5SZr3OT3ZlQ08JLJzzYsEYmQlcvx6c9jwHVSsVS7Z-D1ueftooQ_kqOTlWTrFJE43EHL008Vpnef-XLBrEudqCInmHkY3eTtnTqImmImBloiQyGa8N7MgjT_ufcFwQuWBQN9bdo9_gC6R-Zjp4glHNNbhv_NuHtZXuHEaFqqq_pkk6vaA-xBY8EuBmF_XII_BjelUtD4acwM9fmpbIA3AxWG11qzam5yt22gkDLEB930WG-1Z18qfrJiMhGthfainoNxXfxRVFdvnrf5YNr6xOtjHIdeF_lGK4e8ZTahrL4mRWGdqnpUPvP4I_7XOfytNlpfWL-DB9SzXB822EupaUML_jGh-CkyKnrYYvDbmh57ptQx3dwmFjFKEKuEJbB7ez1Gzh6D_ysYy0LtGwdOdwfpLvW286F7Gt35vL_ATsAOIZ79Qn7cOf0Qbnmo7Q8XP7vvpE7fy7DywVF8LnpkYdqMIuTO9q6pJg1Mi8nZEwkgKkH2EVvIeVHD6p-i-irZmv7Bugcm7xvSSFtEiYG_8HgeBJEusc-vpO4cwxkjYkWG29lCOqjRK285AZ1FhyyXx-LqXqXXlipmZWvWN9BPFQccQh5bfiUNz064JbowLvGGVjGN06cTFvzHKzvT2cSv-idHH5sXRF1kl0wWuw.kCV5oHHuLVRrABAsxe3PFg/__results___files/__results___11_1.png)
Figures 3 and 4
show
RegNet
results:
These curves look quite “clean”, suggesting more stable learning behavior.
Although we achieved around 0.9300 on the public LB with limited time, I firmly believe that with an appropriate number of epochs, RegNet can surpass the 0.95 mark and potentially outperform EfficientNet.
Finally, I joined the competition only 9 days before its end, so I didn’t really have time to make plenty of submissions. I also chose to
respect
the Kaggle guideline:
“You cannot sign up to Kaggle from multiple accounts and therefore you cannot enter or submit from multiple accounts.”
…unlike certain others in the Top 10 🥸. Nonetheless, had I been able to iterate a bit more on RegNet (e.g., tuning hyperparameters, adjusting the epoch count), I’m convinced it would have achieved even better performance like +0.95.
5.
A Challenge with the Leaderboard
One key issue with this competition was the absence of a
split between the public and private leaderboard
, making
brute force
submissions a highly effective strategy. By learning from the leaderboard itself, participants could iteratively fine-tune predictions rather than genuinely optimizing the model.
This is apparent in many Top 10 solutions. For instance,
bestwater
(ranked 1st) openly admits to this approach in their post (
[link here](https://www.kaggle.com/competitions/cidaut-ai-fake-scene-classification-2024/discussion/557589#3101028)
). Additionally, it's almost certain that
solosid69
(top 2, 4, 5, 6, 7) used the same technique—leveraging
five different accounts
😶‍🌫️ to maximize submissions and brute force his way up the leaderboard.
It’s worth noting that
bestwater's honesty
in admitting this method is commendable, as it underscores the bigger problem: the
competition design
itself. With
no private leaderboard
, brute-forcing overshadowed more rigorous model optimization efforts.
Even with thorough hyperparameter tuning, the results presented here are likely close to the best achievable
without brute-forcing
. Nevertheless, I
enjoyed
this competition a lot; it was a fun and rewarding experience that allowed me to
discover new models
and gain deeper insights into their performance on real-world data.
6.
Sources
Reference Papers
[Vision Transformer (ViT)](https://arxiv.org/abs/2010.11929)
[EfficientNet V2](https://arxiv.org/abs/2104.00298)
[RegNet (Designing Network Design Spaces)](https://arxiv.org/abs/2003.13678)
Kaggle Notebooks
[Notebook showcasing ViT, EfficientNet, and RegNet](https://www.kaggle.com/code/quentinducoulombier/top-10-vit-efficientnet-regnet-cidaut-ai)
←
Most important notebook
[RegNet (3-way split) Notebook](https://www.kaggle.com/code/quentinducoulombier/cidaut-regnet)
[EfficientNet V2 (3-way split) Notebook](https://www.kaggle.com/code/quentinducoulombier/cidaut-efficientnetv2)
[ViT (3-way split) Notebook](https://www.kaggle.com/code/quentinducoulombier/cidaut-vit)