{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":87323,"databundleVersionId":9934427,"sourceType":"competition"}],"dockerImageVersionId":30839,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Using Pretrained Models for Fake vs Real Image Classification\n\n**Our Approach:**\n- We attained a top 5% ranking without resorting to training CNNs or engaging in extensive hyperparameter tuning.\n- Using state-of-the-art pretrained models from Huggingface to compute embeddings for the fake - real images, atop which we trained a simple classification layer.\n- Fair play and reproducibility – no multiple-account tricks (allegedly unlike some top-10 solutions 🤭🤭🤭).\n\n## Table of Contents\n1. [Analyze/Prepare the Data](#data)\n2. [Compute Image Embeddings](#embeddings)\n3. [Investigate the Embeddings](#investigate)\n4. [Train the Classifier](#classifier)\n5. [Create Submissions](#submissions)\n6. [Conclusion](#conclusion)","metadata":{}},{"cell_type":"code","source":"import os\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nimport numpy as np\nfrom PIL import Image\nimport pandas as pd\nfrom tqdm import tqdm\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn.manifold import TSNE\nfrom sklearn.decomposition import PCA\nfrom sklearn.metrics import roc_auc_score\nfrom IPython.display import clear_output\nfrom transformers import CLIPProcessor, CLIPModel\nfrom transformers import AutoImageProcessor, AutoModel\nfrom transformers import AutoFeatureExtractor, RegNetModel\nfrom transformers import AutoImageProcessor, AutoModelForImageClassification\nfrom transformers import BeitImageProcessor, BeitForImageClassification","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:14:52.133717Z","iopub.execute_input":"2025-02-13T21:14:52.133965Z","iopub.status.idle":"2025-02-13T21:15:17.665904Z","shell.execute_reply.started":"2025-02-13T21:14:52.133942Z","shell.execute_reply":"2025-02-13T21:15:17.665118Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Set seed for reproducibility\nrandom_seed = 42\ntorch.manual_seed(random_seed)\nnp.random.seed(random_seed)\n\n# Path to the data\nbase_dir = \"/kaggle/input/cidaut-ai-fake-scene-classification-2024\"\nbase_dir_output = \"/kaggle/working\"\nlabels_path = os.path.join(base_dir, \"train.csv\")\ntrain_path = os.path.join(base_dir, \"Train\")\ntest_path = os.path.join(base_dir, \"Test\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:15:17.666806Z","iopub.execute_input":"2025-02-13T21:15:17.667316Z","iopub.status.idle":"2025-02-13T21:15:17.677688Z","shell.execute_reply.started":"2025-02-13T21:15:17.667293Z","shell.execute_reply":"2025-02-13T21:15:17.676886Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(f\"Using device: {device}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:15:17.678527Z","iopub.execute_input":"2025-02-13T21:15:17.678798Z","iopub.status.idle":"2025-02-13T21:15:17.742294Z","shell.execute_reply.started":"2025-02-13T21:15:17.678774Z","shell.execute_reply":"2025-02-13T21:15:17.741152Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Analyze and Prepare the Data <a class=\"anchor\" id=\"data\"></a>\n- First we analyze the data to understand to see how the images are distributed.\n- Then we create datasets for training and validation.\n\n## Get Distribution of the Data","metadata":{}},{"cell_type":"code","source":"labels_df = pd.read_csv(labels_path)\n\nfig, ax = plt.subplots(1, 1, figsize=(4, 2))\nsns.countplot(x=\"label\", data=labels_df, ax=ax)\nax.set_title(\"Label Distribution\")\nax.set_xlabel(\"Label\")\nax.set_ylabel(\"Counts\")\nplt.tight_layout()\nplt.show()\n\ncounts = labels_df[\"label\"].value_counts()\nnormalized = labels_df[\"label\"].value_counts(normalize=True)\nresult_df = pd.DataFrame({\n    \"Count\": counts,\n    \"Percentage\": normalized\n})\nresult_df[\"Percentage\"] = result_df[\"Percentage\"].apply(lambda x: f\"{x:.2%}\")\nprint(result_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:15:17.743250Z","iopub.execute_input":"2025-02-13T21:15:17.743550Z","iopub.status.idle":"2025-02-13T21:15:18.038887Z","shell.execute_reply.started":"2025-02-13T21:15:17.743524Z","shell.execute_reply":"2025-02-13T21:15:18.038047Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Create Train and Validation Data\n\n- We have more real than fake images in the training set. We use this information to use the same class imbalance in our training and validation sets.","metadata":{}},{"cell_type":"code","source":"real_df = labels_df[labels_df['label'] == 'real'].copy()\nfake_df = labels_df[labels_df['label'] == 'editada'].copy()\n\nnum_real = len(real_df)\nnum_fake = len(fake_df)\nnum_train_real = int(0.8 * num_real)\nnum_train_fake = int(0.8 * num_fake)\n\n# Shuffle the DataFrames\nreal_df = real_df.sample(frac=1, random_state=random_seed)\nfake_df = fake_df.sample(frac=1, random_state=random_seed)\n\n# Split into train and validation\nreal_train = real_df.iloc[:num_train_real]\nreal_val = real_df.iloc[num_train_real:]\nfake_train = fake_df.iloc[:num_train_fake]\nfake_val = fake_df.iloc[num_train_fake:]\n\n# Combine and sort by image name\ntrain_df = pd.concat([real_train, fake_train])\nval_df = pd.concat([real_val, fake_val])\ntrain_df = train_df.sort_values('image').reset_index(drop=True)\nval_df = val_df.sort_values('image').reset_index(drop=True)\n\ntrain_df.to_csv(os.path.join(base_dir_output, \"test_split.csv\"), index=False)\nval_df.to_csv(os.path.join(base_dir_output, \"val_split.csv\"), index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:15:18.039727Z","iopub.execute_input":"2025-02-13T21:15:18.040042Z","iopub.status.idle":"2025-02-13T21:15:18.062017Z","shell.execute_reply.started":"2025-02-13T21:15:18.040018Z","shell.execute_reply":"2025-02-13T21:15:18.060918Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:15:18.065410Z","iopub.execute_input":"2025-02-13T21:15:18.065716Z","iopub.status.idle":"2025-02-13T21:15:18.097161Z","shell.execute_reply.started":"2025-02-13T21:15:18.065689Z","shell.execute_reply":"2025-02-13T21:15:18.096082Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def load_images_paths(df, path):\n    \"\"\"\n    Load all images from a dataframe and return a list of paths to the images.\n    \"\"\"\n    if path == \"train\":\n        file_path = train_path\n    elif path == \"test\":\n        file_path = test_path\n    else:\n        raise ValueError(f\"Invalid path: {path}\")\n    \n    image_paths = [os.path.join(file_path, filename) for filename in df[\"image\"]]\n    return image_paths\n\nall_train_images_paths = load_images_paths(labels_df, \"train\")\nall_test_images_paths = load_images_paths(pd.read_csv(os.path.join(base_dir, \"sample_submission.csv\")), \"test\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:15:18.101635Z","iopub.execute_input":"2025-02-13T21:15:18.101881Z","iopub.status.idle":"2025-02-13T21:15:18.116176Z","shell.execute_reply.started":"2025-02-13T21:15:18.101859Z","shell.execute_reply":"2025-02-13T21:15:18.115275Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Compute Image Embeddings <a class=\"anchor\" id=\"embeddings\"></a>\n- We use different pretrained model architectures and different versions to compute embeddings for the images.\n- We compute embeddings for the training and validation images and save them.\n- Afterwards we delete the models to free up (CUDA) memory.","metadata":{}},{"cell_type":"code","source":"os.makedirs(os.path.join(base_dir_output, \"embeddings\"), exist_ok=True)\n\ndef compute_embeddings_batched(model_type, model, processor, image_paths, save_path, batch_size=2):\n    \"\"\"\n    Extract embeddings from a list of image paths.\n    \"\"\"\n    all_embeddings = []\n    all_img_names = []\n    for i in tqdm(range(0, len(image_paths), batch_size)):\n        img_paths = image_paths[i:i+batch_size]\n        img_names = [os.path.basename(img_path) for img_path in img_paths]\n        all_img_names.extend(img_names)\n        images = [Image.open(image_path).convert(\"RGB\") for image_path in img_paths]\n        inputs = processor(images=images, return_tensors=\"pt\")\n        inputs = {k: v.to(device) for k, v in inputs.items()}\n        with torch.no_grad():\n            if model_type == \"clip\":\n                embeddings = model.get_image_features(**inputs)\n            elif model_type == \"regnety\":\n                outputs = model(**inputs)\n                embeddings = outputs.pooler_output\n                embeddings = embeddings[:, :, 0, 0]\n            elif model_type == \"dino\":\n                outputs = model(**inputs)\n                embeddings = outputs.pooler_output\n            elif model_type == \"swin\" or model_type == \"beit\":\n                outputs = model(**inputs)\n                embeddings = outputs.logits\n            else:\n                raise ValueError(f\"Model type {model_type} not supported.\")\n        all_embeddings.append(embeddings.cpu().numpy())\n    all_embeddings = np.concatenate(all_embeddings, axis=0)\n    df = pd.DataFrame(all_embeddings)\n    df[\"image\"] = all_img_names\n    df.to_csv(save_path, index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:15:18.117158Z","iopub.execute_input":"2025-02-13T21:15:18.117438Z","iopub.status.idle":"2025-02-13T21:15:18.126173Z","shell.execute_reply.started":"2025-02-13T21:15:18.117415Z","shell.execute_reply":"2025-02-13T21:15:18.124964Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Compute CLIP Embeddings\n### CLIP ViT Large","metadata":{}},{"cell_type":"code","source":"# Load the CLIP model and processor from HuggingFace\nmodel_clip_vit_large = CLIPModel.from_pretrained(\"openai/clip-vit-large-patch14\").to(device)\nprocessor_clip_vit_large = CLIPProcessor.from_pretrained(\"openai/clip-vit-large-patch14\")\n\n# Extract features from the images and save them\ncompute_embeddings_batched(\"clip\", model_clip_vit_large, processor_clip_vit_large, all_train_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_train.csv\"), 8)\ncompute_embeddings_batched(\"clip\", model_clip_vit_large, processor_clip_vit_large, all_test_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_test.csv\"), 8)\n\n# Clear memory\ndel model_clip_vit_large, processor_clip_vit_large\ntorch.cuda.empty_cache()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:15:18.127194Z","iopub.execute_input":"2025-02-13T21:15:18.127427Z","iopub.status.idle":"2025-02-13T21:16:26.498334Z","shell.execute_reply.started":"2025-02-13T21:15:18.127406Z","shell.execute_reply":"2025-02-13T21:16:26.497596Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### CLIP ViT Large 336","metadata":{}},{"cell_type":"code","source":"model_clip_vit_large_336 = CLIPModel.from_pretrained(\"openai/clip-vit-large-patch14-336\").to(device)\nprocessor_clip_vit_large_336 = CLIPProcessor.from_pretrained(\"openai/clip-vit-large-patch14-336\")\n\ncompute_embeddings_batched(\"clip\", model_clip_vit_large_336, processor_clip_vit_large_336, all_train_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_336_train.csv\"), 8)\ncompute_embeddings_batched(\"clip\", model_clip_vit_large_336, processor_clip_vit_large_336, all_test_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_336_test.csv\"), 8)\n\ndel model_clip_vit_large_336, processor_clip_vit_large_336\ntorch.cuda.empty_cache()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:16:26.499207Z","iopub.execute_input":"2025-02-13T21:16:26.499436Z","iopub.status.idle":"2025-02-13T21:17:58.793951Z","shell.execute_reply.started":"2025-02-13T21:16:26.499417Z","shell.execute_reply":"2025-02-13T21:17:58.792838Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### CLIP ViT Large LAION-2B","metadata":{}},{"cell_type":"code","source":"model_clip_vit_large_laion = CLIPModel.from_pretrained(\"laion/CLIP-ViT-g-14-laion2B-s12B-b42K\").to(device)\nprocessor_clip_vit_large_laion = CLIPProcessor.from_pretrained(\"laion/CLIP-ViT-g-14-laion2B-s12B-b42K\")\n\ncompute_embeddings_batched(\"clip\", model_clip_vit_large_laion, processor_clip_vit_large_laion, all_train_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_laion_train.csv\"), 8)\ncompute_embeddings_batched(\"clip\", model_clip_vit_large_laion, processor_clip_vit_large_laion, all_test_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_laion_test.csv\"), 8)\n\ndel model_clip_vit_large_laion, processor_clip_vit_large_laion\ntorch.cuda.empty_cache()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:17:58.795719Z","iopub.execute_input":"2025-02-13T21:17:58.796144Z","iopub.status.idle":"2025-02-13T21:20:12.403562Z","shell.execute_reply.started":"2025-02-13T21:17:58.796101Z","shell.execute_reply":"2025-02-13T21:20:12.402565Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Compute RegNet-Y Embeddings","metadata":{}},{"cell_type":"code","source":"model_regnety_1280 = RegNetModel.from_pretrained(\"facebook/regnet-y-1280-seer\").to(device)\nprocessor_regnety_1280 = AutoFeatureExtractor.from_pretrained(\"facebook/regnet-y-1280-seer\")\n\ncompute_embeddings_batched(\"regnety\", model_regnety_1280, processor_regnety_1280, all_train_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_regnety_1280_train.csv\"), 8)\ncompute_embeddings_batched(\"regnety\", model_regnety_1280, processor_regnety_1280, all_test_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_regnety_1280_test.csv\"), 8)\n\ndel model_regnety_1280, processor_regnety_1280\ntorch.cuda.empty_cache()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:20:12.404973Z","iopub.execute_input":"2025-02-13T21:20:12.405303Z","iopub.status.idle":"2025-02-13T21:23:57.308682Z","shell.execute_reply.started":"2025-02-13T21:20:12.405278Z","shell.execute_reply":"2025-02-13T21:23:57.307826Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Compute Swin Transformer Embeddings\n### Swin Transformer 192","metadata":{}},{"cell_type":"code","source":"model_swinv2_192 = AutoModelForImageClassification.from_pretrained(\"microsoft/swinv2-large-patch4-window12to16-192to256-22kto1k-ft\").to(device)\nprocessor_swinv2_192 = AutoImageProcessor.from_pretrained(\"microsoft/swinv2-large-patch4-window12to16-192to256-22kto1k-ft\")\n\ncompute_embeddings_batched(\"swin\", model_swinv2_192, processor_swinv2_192, all_train_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_swinv2_192_train.csv\"), 8)\ncompute_embeddings_batched(\"swin\", model_swinv2_192, processor_swinv2_192, all_test_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_swinv2_192_test.csv\"), 8)\n\ndel model_swinv2_192, processor_swinv2_192\ntorch.cuda.empty_cache()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:23:57.309897Z","iopub.execute_input":"2025-02-13T21:23:57.310170Z","iopub.status.idle":"2025-02-13T21:24:57.777726Z","shell.execute_reply.started":"2025-02-13T21:23:57.310148Z","shell.execute_reply":"2025-02-13T21:24:57.776813Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Swin Transformer 384","metadata":{}},{"cell_type":"code","source":"model_swinv2_384 = AutoModelForImageClassification.from_pretrained(\"microsoft/swinv2-large-patch4-window12to24-192to384-22kto1k-ft\").to(device)\nprocessor_swinv2_384 = AutoImageProcessor.from_pretrained(\"microsoft/swinv2-large-patch4-window12to24-192to384-22kto1k-ft\")\n\ncompute_embeddings_batched(\"swin\", model_swinv2_384, processor_swinv2_384, all_train_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_swinv2_384_train.csv\"), 8)\ncompute_embeddings_batched(\"swin\", model_swinv2_384, processor_swinv2_384, all_test_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_swinv2_384_test.csv\"), 8)\n\ndel model_swinv2_384, processor_swinv2_384\ntorch.cuda.empty_cache()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:24:57.779039Z","iopub.execute_input":"2025-02-13T21:24:57.779444Z","iopub.status.idle":"2025-02-13T21:26:33.768953Z","shell.execute_reply.started":"2025-02-13T21:24:57.779386Z","shell.execute_reply":"2025-02-13T21:26:33.768072Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Compute BEiT Embeddings\n### BEiT 224","metadata":{}},{"cell_type":"code","source":"model_beit_large_224 = BeitForImageClassification.from_pretrained(\"microsoft/beit-large-patch16-224\").to(device)\nprocessor_beit_large_224 = BeitImageProcessor.from_pretrained(\"microsoft/beit-large-patch16-224\")\n\ncompute_embeddings_batched(\"beit\", model_beit_large_224, processor_beit_large_224, all_train_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_beit_large_224_train.csv\"), 8)\ncompute_embeddings_batched(\"beit\", model_beit_large_224, processor_beit_large_224, all_test_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_beit_large_224_test.csv\"), 8)\n\ndel model_beit_large_224, processor_beit_large_224\ntorch.cuda.empty_cache()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:26:33.770214Z","iopub.execute_input":"2025-02-13T21:26:33.770528Z","iopub.status.idle":"2025-02-13T21:27:35.838640Z","shell.execute_reply.started":"2025-02-13T21:26:33.770501Z","shell.execute_reply":"2025-02-13T21:27:35.837509Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### BEiT 512","metadata":{}},{"cell_type":"code","source":"model_beit_large_512 = BeitForImageClassification.from_pretrained(\"microsoft/beit-large-patch16-512\").to(device)\nprocessor_beit_large_512 = BeitImageProcessor.from_pretrained(\"microsoft/beit-large-patch16-512\")\n\ncompute_embeddings_batched(\"beit\", model_beit_large_512, processor_beit_large_512, all_train_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_beit_large_512_train.csv\"), 8)\ncompute_embeddings_batched(\"beit\", model_beit_large_512, processor_beit_large_512, all_test_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_beit_large_512_test.csv\"), 8)\n\ndel model_beit_large_512, processor_beit_large_512\ntorch.cuda.empty_cache()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:27:35.840338Z","iopub.execute_input":"2025-02-13T21:27:35.840674Z","iopub.status.idle":"2025-02-13T21:31:02.745046Z","shell.execute_reply.started":"2025-02-13T21:27:35.840644Z","shell.execute_reply":"2025-02-13T21:31:02.744094Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Extract DINOv2 Embeddings","metadata":{}},{"cell_type":"code","source":"model_dino_giant = AutoModel.from_pretrained('facebook/dinov2-giant').to(device)\nprocessor_dino_giant = AutoImageProcessor.from_pretrained('facebook/dinov2-giant')\n\ncompute_embeddings_batched(\"dino\", model_dino_giant, processor_dino_giant, all_train_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_dino_giant_train.csv\"), 8)\ncompute_embeddings_batched(\"dino\", model_dino_giant, processor_dino_giant, all_test_images_paths, os.path.join(base_dir_output, \"embeddings\", \"embeddings_dino_giant_test.csv\"), 8)\n\ndel model_dino_giant, processor_dino_giant\ntorch.cuda.empty_cache()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:31:02.746163Z","iopub.execute_input":"2025-02-13T21:31:02.746422Z","iopub.status.idle":"2025-02-13T21:33:31.393085Z","shell.execute_reply.started":"2025-02-13T21:31:02.746388Z","shell.execute_reply":"2025-02-13T21:33:31.392072Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Investigate the Embeddings <a class=\"anchor\" id=\"investigate\"></a>\n- We investigate the embeddings to see how well they separate the fake and real images.\n- We use PCA and t-SNE to visualize the embeddings.","metadata":{}},{"cell_type":"code","source":"def create_train_val_embeddings_df(embeddings_path, train_df, val_df):\n    \"\"\"\n    Create a DataFrame with the embeddings for the training and validation images.\n    \"\"\"\n    all_embeddings = pd.read_csv(embeddings_path)\n    train_embeddings = all_embeddings[all_embeddings[\"image\"].isin(train_df[\"image\"])]\n    val_embeddings = all_embeddings[all_embeddings[\"image\"].isin(val_df[\"image\"])]\n    return train_embeddings, val_embeddings\n\ndef get_gt_labels_for_img_names(img_names):\n    \"\"\"\n    Get the ground truth labels for a list of image names.\n    img_names: List of image names.\n    \"\"\"\n    labels = []\n    for img_name in img_names:\n        label = labels_df[labels_df[\"image\"] == img_name][\"label\"].values[0]\n        labels.append(label)\n    # convert \"real\" to 1 and \"editada\" to 0\n    labels = [1 if label == \"real\" else 0 for label in labels]\n    return labels","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:33:31.394417Z","iopub.execute_input":"2025-02-13T21:33:31.394784Z","iopub.status.idle":"2025-02-13T21:33:31.400778Z","shell.execute_reply.started":"2025-02-13T21:33:31.394750Z","shell.execute_reply":"2025-02-13T21:33:31.399961Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Load the embeddings in case the notebook is restarted\ntrain_embeddings_clip_vit_large, val_embeddings_clip_vit_large = create_train_val_embeddings_df(os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_train.csv\"), train_df, val_df)\ntrain_embeddings_clip_vit_large_336, val_embeddings_clip_vit_large_336 = create_train_val_embeddings_df(os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_336_train.csv\"), train_df, val_df)\ntrain_embeddings_clip_vit_large_laion, val_embeddings_clip_vit_large_laion = create_train_val_embeddings_df(os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_laion_train.csv\"), train_df, val_df)\ntrain_embeddings_regnety_1280, val_embeddings_regnety_1280 = create_train_val_embeddings_df(os.path.join(base_dir_output, \"embeddings\", \"embeddings_regnety_1280_train.csv\"), train_df, val_df)\ntrain_embeddings_swinv2_192, val_embeddings_swinv2_192 = create_train_val_embeddings_df(os.path.join(base_dir_output, \"embeddings\", \"embeddings_swinv2_192_train.csv\"), train_df, val_df)\ntrain_embeddings_swinv2_384, val_embeddings_swinv2_384 = create_train_val_embeddings_df(os.path.join(base_dir_output, \"embeddings\", \"embeddings_swinv2_384_train.csv\"), train_df, val_df)\ntrain_embeddings_beit_large_224, val_embeddings_beit_large_224 = create_train_val_embeddings_df(os.path.join(base_dir_output, \"embeddings\", \"embeddings_beit_large_224_train.csv\"), train_df, val_df)\ntrain_embeddings_beit_large_512, val_embeddings_beit_large_512 = create_train_val_embeddings_df(os.path.join(base_dir_output, \"embeddings\", \"embeddings_beit_large_512_train.csv\"), train_df, val_df)\ntrain_embeddings_dino_giant, val_embeddings_dino_giant = create_train_val_embeddings_df(os.path.join(base_dir_output, \"embeddings\", \"embeddings_dino_giant_train.csv\"), train_df, val_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:33:31.404172Z","iopub.execute_input":"2025-02-13T21:33:31.404504Z","iopub.status.idle":"2025-02-13T21:33:33.955047Z","shell.execute_reply.started":"2025-02-13T21:33:31.404464Z","shell.execute_reply":"2025-02-13T21:33:33.954249Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(f\"Train embeddings CLIP ViT Large shape: {train_embeddings_clip_vit_large.shape}\")\nprint(f\"Validation embeddings CLIP ViT Large shape: {val_embeddings_clip_vit_large.shape}\")\nprint(f\"Train embeddings CLIP ViT Large 336 shape: {train_embeddings_clip_vit_large_336.shape}\")\nprint(f\"Validation embeddings CLIP ViT Large 336 shape: {val_embeddings_clip_vit_large_336.shape}\")\nprint(f\"Train embeddings CLIP ViT Large Laion shape: {train_embeddings_clip_vit_large_laion.shape}\")\nprint(f\"Validation embeddings CLIP ViT Large Laion shape: {val_embeddings_clip_vit_large_laion.shape}\")\nprint(f\"Train embeddings RegNetY 1280 shape: {train_embeddings_regnety_1280.shape}\")\nprint(f\"Validation embeddings RegNetY 1280 shape: {val_embeddings_regnety_1280.shape}\")\nprint(f\"Train embeddings SwinV2 192 shape: {train_embeddings_swinv2_192.shape}\")\nprint(f\"Validation embeddings SwinV2 192 shape: {val_embeddings_swinv2_192.shape}\")\nprint(f\"Train embeddings SwinV2 384 shape: {train_embeddings_swinv2_384.shape}\")\nprint(f\"Validation embeddings SwinV2 384 shape: {val_embeddings_swinv2_384.shape}\")\nprint(f\"Train embeddings BEiT Large 224 shape: {train_embeddings_beit_large_224.shape}\")\nprint(f\"Validation embeddings BEiT Large 224 shape: {val_embeddings_beit_large_224.shape}\")\nprint(f\"Train embeddings BEiT Large 512 shape: {train_embeddings_beit_large_512.shape}\")\nprint(f\"Validation embeddings BEiT Large 512 shape: {val_embeddings_beit_large_512.shape}\")\nprint(f\"Train embeddings DINOv2 Giant shape: {train_embeddings_dino_giant.shape}\")\nprint(f\"Validation embeddings DINOv2 Giant shape: {val_embeddings_dino_giant.shape}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:33:33.956715Z","iopub.execute_input":"2025-02-13T21:33:33.957092Z","iopub.status.idle":"2025-02-13T21:33:33.967944Z","shell.execute_reply.started":"2025-02-13T21:33:33.957057Z","shell.execute_reply":"2025-02-13T21:33:33.967023Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## PCA Visualization","metadata":{}},{"cell_type":"code","source":"def plot_pca_on_ax(ax, df, title, n_components=2):\n    embeddings = df.drop(columns=[\"image\"]).values\n    img_names = df[\"image\"].values\n    labels = get_gt_labels_for_img_names(img_names)\n\n    # Perform PCA on feature data\n    pca = PCA(n_components=n_components)\n    pca_results = pca.fit_transform(embeddings)\n    \n    ax.scatter(pca_results[:, 0], pca_results[:, 1], c=labels, cmap=\"coolwarm\", alpha=0.7)\n    ax.grid(True, linestyle=\"--\", alpha=0.6, linewidth=1, color=\"gray\", zorder=-1)\n    ax.set_title(title)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:33:33.968754Z","iopub.execute_input":"2025-02-13T21:33:33.969104Z","iopub.status.idle":"2025-02-13T21:33:33.990991Z","shell.execute_reply.started":"2025-02-13T21:33:33.969078Z","shell.execute_reply":"2025-02-13T21:33:33.990106Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, axs = plt.subplots(3, 3, figsize=(8, 8))\nplot_pca_on_ax(axs[0, 0], pd.concat([train_embeddings_clip_vit_large, val_embeddings_clip_vit_large], ignore_index=True), \"CLIP Large\")\nplot_pca_on_ax(axs[0, 1], pd.concat([train_embeddings_clip_vit_large_336, val_embeddings_clip_vit_large_336], ignore_index=True), \"CLIP Large 336\")\nplot_pca_on_ax(axs[0, 2], pd.concat([train_embeddings_clip_vit_large_laion, val_embeddings_clip_vit_large_laion], ignore_index=True), \"CLIP Large Laion\")\nplot_pca_on_ax(axs[1, 0], pd.concat([train_embeddings_regnety_1280, val_embeddings_regnety_1280], ignore_index=True), \"RegNetY 1280\")\nplot_pca_on_ax(axs[1, 1], pd.concat([train_embeddings_swinv2_192, val_embeddings_swinv2_192], ignore_index=True), \"SwinV2 192\")\nplot_pca_on_ax(axs[1, 2], pd.concat([train_embeddings_swinv2_384, val_embeddings_swinv2_384], ignore_index=True), \"SwinV2 384\")\nplot_pca_on_ax(axs[2, 0], pd.concat([train_embeddings_beit_large_224, val_embeddings_beit_large_224], ignore_index=True), \"BEiT Large 224\")\nplot_pca_on_ax(axs[2, 1], pd.concat([train_embeddings_beit_large_512, val_embeddings_beit_large_512], ignore_index=True), \"BEiT Large 512\")\nplot_pca_on_ax(axs[2, 2], pd.concat([train_embeddings_dino_giant, val_embeddings_dino_giant], ignore_index=True), \"DINOv2 Giant\")\n\nplt.suptitle(\"PCA (n=2) on Image Embeddings\", fontsize=16)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:33:33.992115Z","iopub.execute_input":"2025-02-13T21:33:33.992473Z","iopub.status.idle":"2025-02-13T21:33:38.735729Z","shell.execute_reply.started":"2025-02-13T21:33:33.992439Z","shell.execute_reply":"2025-02-13T21:33:38.734634Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## t-SNE Visualization","metadata":{}},{"cell_type":"code","source":"def plot_tsne_on_ax(ax, df, title, n_components=2, perplexity=30, learning_rate=\"auto\", max_iter=1_000, random_state=42):\n    embeddings = df.drop(columns=[\"image\"]).values\n    img_names = df[\"image\"].values\n    labels = get_gt_labels_for_img_names(img_names)\n\n    tsne = TSNE(n_components=n_components, perplexity=perplexity, \n                learning_rate=learning_rate, n_iter=max_iter, random_state=random_state)\n    tsne_results = tsne.fit_transform(embeddings)\n\n    scatter = ax.scatter(tsne_results[:, 0], tsne_results[:, 1],\n                         c=labels, cmap=\"coolwarm\", alpha=0.7)\n\n    ax.set_title(title)\n    ax.grid(True, linestyle=\"--\", alpha=0.6, linewidth=1, color=\"gray\", zorder=-1)\n    ax.set_axisbelow(True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:33:38.736687Z","iopub.execute_input":"2025-02-13T21:33:38.736965Z","iopub.status.idle":"2025-02-13T21:33:38.743425Z","shell.execute_reply.started":"2025-02-13T21:33:38.736940Z","shell.execute_reply":"2025-02-13T21:33:38.742470Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, axs = plt.subplots(3, 3, figsize=(8, 8))\nplot_tsne_on_ax(axs[0, 0], pd.concat([train_embeddings_clip_vit_large, val_embeddings_clip_vit_large], ignore_index=True), \"CLIP Large\")\nplot_tsne_on_ax(axs[0, 1], pd.concat([train_embeddings_clip_vit_large_336, val_embeddings_clip_vit_large_336], ignore_index=True), \"CLIP Large 336\")\nplot_tsne_on_ax(axs[0, 2], pd.concat([train_embeddings_clip_vit_large_laion, val_embeddings_clip_vit_large_laion], ignore_index=True), \"CLIP Large Laion\")\nplot_tsne_on_ax(axs[1, 0], pd.concat([train_embeddings_regnety_1280, val_embeddings_regnety_1280], ignore_index=True), \"RegNetY 1280\")\nplot_tsne_on_ax(axs[1, 1], pd.concat([train_embeddings_swinv2_192, val_embeddings_swinv2_192], ignore_index=True), \"SwinV2 192\")\nplot_tsne_on_ax(axs[1, 2], pd.concat([train_embeddings_swinv2_384, val_embeddings_swinv2_384], ignore_index=True), \"SwinV2 384\")\nplot_tsne_on_ax(axs[2, 0], pd.concat([train_embeddings_beit_large_224, val_embeddings_beit_large_224], ignore_index=True), \"BEiT Large 224\")\nplot_tsne_on_ax(axs[2, 1], pd.concat([train_embeddings_beit_large_512, val_embeddings_beit_large_512], ignore_index=True), \"BEiT Large 512\")\nplot_tsne_on_ax(axs[2, 2], pd.concat([train_embeddings_dino_giant, val_embeddings_dino_giant], ignore_index=True), \"DINOv2 Giant\")\n\nplt.suptitle(\"t-SNE (n=2) on Image Embeddings\", fontsize=16)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:33:38.744789Z","iopub.execute_input":"2025-02-13T21:33:38.745331Z","iopub.status.idle":"2025-02-13T21:34:16.780810Z","shell.execute_reply.started":"2025-02-13T21:33:38.745293Z","shell.execute_reply":"2025-02-13T21:34:16.779665Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Conclusion\n- CLIP Large, CLIP Large 336, and RegNet-Y embeddings seem to separate the fake and real images well even in the 2D space.\n- All other embeddings show no clear separation between the fake and real images in the 2D space.","metadata":{}},{"cell_type":"markdown","source":"# Train the Classifier <a class=\"anchor\" id=\"classifier\"></a>\n- We train a simple classifier on the embeddings to classify the images as fake or real.\n- We load the previously computed embeddings because we don't need to recompute them every time we restart the kernel.","metadata":{}},{"cell_type":"code","source":"class LinearClassifier(nn.Module):\n    def __init__(self, input_dim):\n        super(LinearClassifier, self).__init__()\n        self.fc = nn.Linear(input_dim, 1)\n\n    def forward(self, x):\n        return self.fc(x)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:34:16.782194Z","iopub.execute_input":"2025-02-13T21:34:16.782526Z","iopub.status.idle":"2025-02-13T21:34:16.787286Z","shell.execute_reply.started":"2025-02-13T21:34:16.782499Z","shell.execute_reply":"2025-02-13T21:34:16.786243Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def train_linear_classifier(train_df, val_df=None, epochs=5000, lr=0.01):\n    embeddings_train = train_df.drop(columns=[\"image\"]).values\n    embeddings_train = torch.tensor(embeddings_train).float().to(device)\n    img_names_train = train_df[\"image\"].values\n    labels_train = get_gt_labels_for_img_names(img_names_train)\n    labels_train = torch.tensor(labels_train).float().to(device)\n\n    if val_df is not None:\n        embeddings_val = val_df.drop(columns=[\"image\"]).values\n        embeddings_val = torch.tensor(embeddings_val).float().to(device)\n        img_names_val = val_df[\"image\"].values\n        labels_val = get_gt_labels_for_img_names(img_names_val)\n\n    # Define the linear probe model\n    model = LinearClassifier(embeddings_train.shape[1]).to(device)\n    criterion = nn.BCEWithLogitsLoss()\n    optimizer = optim.Adam(model.parameters(), lr=lr)\n\n    train_losses = []\n    val_losses = []\n    train_aucs = []\n    val_aucs = []\n\n    # Training loop\n    for epoch in range(1, epochs+1):\n        # Shuffle the data\n        perm = torch.randperm(embeddings_train.size(0))  # Generate a random \n        features_shuffled = embeddings_train[perm]\n        labels_shuffled = labels_train[perm]\n\n        model.train()\n        optimizer.zero_grad()\n\n        outputs = model(features_shuffled)\n        loss = criterion(outputs.squeeze(), labels_shuffled.float())\n        loss.backward()\n        train_losses.append(loss.item())\n        optimizer.step()\n\n        auc = roc_auc_score(labels_shuffled.cpu().numpy(), torch.sigmoid(outputs).cpu().detach().numpy().flatten())\n        train_aucs.append(auc)\n\n        if val_df is not None:\n            model.eval()\n            with torch.no_grad():\n                outputs_val = model(embeddings_val)\n                predictions_val = torch.sigmoid(outputs_val).cpu().numpy().flatten()\n                auc = roc_auc_score(labels_val, predictions_val)\n                val_aucs.append(auc)\n                val_loss = criterion(outputs_val.squeeze(), torch.tensor(labels_val).float().to(device))\n                val_losses.append(val_loss.item())\n\n        if epoch % 100 == 0:\n            clear_output(wait=True)\n            fig, ax = plt.subplots(1, 2, figsize=(8, 3))\n            ax[0].set_title(\"Losses\") \n            ax[0].plot(train_losses, label=\"Train Loss\")\n            if val_df is not None:\n                ax[0].plot(val_losses, label=\"Validation Loss\")\n            ax[0].set_xlabel(\"Epoch\")\n            ax[0].set_ylabel(\"Loss\")\n            ax[0].set_yscale(\"log\")\n            ax[0].grid(True, linestyle=\"--\")\n            ax[0].legend()\n\n            ax[1].set_title(\"Area Under ROC Curve\")\n            ax[1].plot(train_aucs, label=\"Train AUC\")\n            if val_df is not None:\n                ax[1].plot(val_aucs, label=\"Validation AUC\")\n            ax[1].set_xlabel(\"Epoch\")\n            ax[1].set_ylabel(\"AUC\")\n            ax[1].grid(True, linestyle=\"--\")\n            ax[1].legend()\n\n            fig.suptitle(f\"Training Progress Epoch: {epoch}/{epochs}\")\n            fig.tight_layout()\n            plt.show()\n\n    return model","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:34:16.788704Z","iopub.execute_input":"2025-02-13T21:34:16.789032Z","iopub.status.idle":"2025-02-13T21:34:16.809107Z","shell.execute_reply.started":"2025-02-13T21:34:16.789003Z","shell.execute_reply":"2025-02-13T21:34:16.808076Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trained_linear_classifier = train_linear_classifier(train_embeddings_clip_vit_large, val_embeddings_clip_vit_large)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:34:16.810194Z","iopub.execute_input":"2025-02-13T21:34:16.810536Z","iopub.status.idle":"2025-02-13T21:35:24.768932Z","shell.execute_reply.started":"2025-02-13T21:34:16.810508Z","shell.execute_reply":"2025-02-13T21:35:24.767868Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trained_linear_classifier = train_linear_classifier(train_embeddings_regnety_1280, val_embeddings_regnety_1280)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:35:24.769965Z","iopub.execute_input":"2025-02-13T21:35:24.770343Z","iopub.status.idle":"2025-02-13T21:36:33.173082Z","shell.execute_reply.started":"2025-02-13T21:35:24.770308Z","shell.execute_reply":"2025-02-13T21:36:33.171922Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trained_linear_classifier = train_linear_classifier(train_embeddings_swinv2_192, val_embeddings_swinv2_192)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:36:33.174230Z","iopub.execute_input":"2025-02-13T21:36:33.174616Z","iopub.status.idle":"2025-02-13T21:37:40.954079Z","shell.execute_reply.started":"2025-02-13T21:36:33.174576Z","shell.execute_reply":"2025-02-13T21:37:40.953017Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trained_linear_classifier = train_linear_classifier(train_embeddings_beit_large_224, val_embeddings_beit_large_224)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:37:40.955114Z","iopub.execute_input":"2025-02-13T21:37:40.955456Z","iopub.status.idle":"2025-02-13T21:38:45.750050Z","shell.execute_reply.started":"2025-02-13T21:37:40.955422Z","shell.execute_reply":"2025-02-13T21:38:45.748688Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trained_linear_classifier = train_linear_classifier(train_embeddings_dino_giant, val_embeddings_dino_giant)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:38:45.751041Z","iopub.execute_input":"2025-02-13T21:38:45.751353Z","iopub.status.idle":"2025-02-13T21:39:53.458761Z","shell.execute_reply.started":"2025-02-13T21:38:45.751328Z","shell.execute_reply":"2025-02-13T21:39:53.457876Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Create Submissions <a class=\"anchor\" id=\"submissions\"></a>\n- We create submissions using different embeddings.\n- Therefore train the classifier on the training and validation data and then predict the test data.","metadata":{}},{"cell_type":"code","source":"def create_submission_file(linear_model, embeddings_path, save_path):\n    all_embeddings = pd.read_csv(embeddings_path)\n    embeddings = all_embeddings.drop(columns=[\"image\"]).values\n    img_names = all_embeddings[\"image\"].values\n\n    linear_model.eval()\n    with torch.no_grad():\n        outputs = linear_model(torch.tensor(embeddings).float().to(device))\n        predictions = torch.sigmoid(outputs).cpu().numpy().flatten()\n    \n    submission_df = pd.DataFrame({\n        \"image\": img_names,\n        \"label\": predictions\n    })\n    submission_df.to_csv(save_path, index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:39:53.459733Z","iopub.execute_input":"2025-02-13T21:39:53.460094Z","iopub.status.idle":"2025-02-13T21:39:53.465940Z","shell.execute_reply.started":"2025-02-13T21:39:53.460066Z","shell.execute_reply":"2025-02-13T21:39:53.464845Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"linear_classifier_clip_vit_large = train_linear_classifier(pd.concat([train_embeddings_clip_vit_large, val_embeddings_clip_vit_large], ignore_index=True))\nlinear_classifier_clip_vit_large_336 = train_linear_classifier(pd.concat([train_embeddings_clip_vit_large_336, val_embeddings_clip_vit_large_336], ignore_index=True))\nlinear_classifier_clip_vit_large_laion = train_linear_classifier(pd.concat([train_embeddings_clip_vit_large_laion, val_embeddings_clip_vit_large_laion], ignore_index=True))\nlinear_classifier_regnety_1280 = train_linear_classifier(pd.concat([train_embeddings_regnety_1280, val_embeddings_regnety_1280], ignore_index=True))\nlinear_classifier_swinv2_192 = train_linear_classifier(pd.concat([train_embeddings_swinv2_192, val_embeddings_swinv2_192], ignore_index=True))\nlinear_classifier_swinv2_384 = train_linear_classifier(pd.concat([train_embeddings_swinv2_384, val_embeddings_swinv2_384], ignore_index=True))\nlinear_classifier_beit_large_224 = train_linear_classifier(pd.concat([train_embeddings_beit_large_224, val_embeddings_beit_large_224], ignore_index=True))\nlinear_classifier_beit_large_512 = train_linear_classifier(pd.concat([train_embeddings_beit_large_512, val_embeddings_beit_large_512], ignore_index=True))\nlinear_classifier_dino_giant = train_linear_classifier(pd.concat([train_embeddings_dino_giant, val_embeddings_dino_giant], ignore_index=True))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:39:53.466938Z","iopub.execute_input":"2025-02-13T21:39:53.467306Z","iopub.status.idle":"2025-02-13T21:48:04.540072Z","shell.execute_reply.started":"2025-02-13T21:39:53.467271Z","shell.execute_reply":"2025-02-13T21:48:04.539059Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"os.makedirs(os.path.join(base_dir_output, \"submissions\"), exist_ok=True)\ncreate_submission_file(linear_classifier_clip_vit_large, os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_test.csv\"), os.path.join(base_dir_output, \"submissions\", \"submission_clip_vit_large.csv\"))\ncreate_submission_file(linear_classifier_clip_vit_large_336, os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_336_test.csv\"), os.path.join(base_dir_output, \"submissions\", \"submission_clip_vit_large_336.csv\"))\ncreate_submission_file(linear_classifier_clip_vit_large_laion, os.path.join(base_dir_output, \"embeddings\", \"embeddings_clip_vit_large_laion_test.csv\"), os.path.join(base_dir_output, \"submissions\", \"submission_clip_vit_large_laion.csv\"))\ncreate_submission_file(linear_classifier_regnety_1280, os.path.join(base_dir_output, \"embeddings\", \"embeddings_regnety_1280_test.csv\"), os.path.join(base_dir_output, \"submissions\", \"submission_regnety_1280.csv\"))\ncreate_submission_file(linear_classifier_swinv2_192, os.path.join(base_dir_output, \"embeddings\", \"embeddings_swinv2_192_test.csv\"), os.path.join(base_dir_output, \"submissions\", \"submission_swinv2_192.csv\"))\ncreate_submission_file(linear_classifier_swinv2_384, os.path.join(base_dir_output, \"embeddings\", \"embeddings_swinv2_384_test.csv\"), os.path.join(base_dir_output, \"submissions\", \"submission_swinv2_384.csv\"))\ncreate_submission_file(linear_classifier_beit_large_224, os.path.join(base_dir_output, \"embeddings\", \"embeddings_beit_large_224_test.csv\"), os.path.join(base_dir_output, \"submissions\", \"submission_beit_large_224.csv\"))\ncreate_submission_file(linear_classifier_beit_large_512, os.path.join(base_dir_output, \"embeddings\", \"embeddings_beit_large_512_test.csv\"), os.path.join(base_dir_output, \"submissions\", \"submission_beit_large_512.csv\"))\ncreate_submission_file(linear_classifier_dino_giant, os.path.join(base_dir_output, \"embeddings\", \"embeddings_dino_giant_test.csv\"), os.path.join(base_dir_output, \"submissions\", \"submission_dino_giant.csv\"))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T21:48:04.541168Z","iopub.execute_input":"2025-02-13T21:48:04.541531Z","iopub.status.idle":"2025-02-13T21:48:05.372444Z","shell.execute_reply.started":"2025-02-13T21:48:04.541496Z","shell.execute_reply":"2025-02-13T21:48:05.371420Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Results\n\n| Backbone | Score |\n|----------|-----|\n| CLIP ViT Large LAION-2B | 0.9392 |\n| CLIP ViT Large | 0.9355 |\n| CLIP ViT Large 336 | 0.9307 |\n| RegNet-Y | 0.9238 |\n| Swin Transformer 384 | 0.9186 |\n| Swin Transformer 192 | 0.9168 |\n| DINOv2 | 0.8996 |\n| BEiT 512 | 0.8972 |\n| BEiT 224 | 0.8558 |\n\n- Note: My best result was 0.9403 using CLIP ViT Large embeddings and training the classifier for 5000 epochs with a learning rate of 0.01. This differs from the result in the table because initially did not set a random seed for notebook.","metadata":{}},{"cell_type":"markdown","source":"# Conclusion <a class=\"anchor\" id=\"conclusion\"></a>\n- We used pretrained models to compute embeddings for the images and trained a simple classifier on top of them.\n- We achieved a top 5% ranking without training CNNs or extensive hyperparameter tuning.\n- We showed that CLIP Large, CLIP Large 336, and RegNet-Y embeddings separate the fake and real images well in the 2D space, they also perform well in the classification task.\n- The best result was achieved using CLIP ViT Large LAION-2B embeddings even though they did not separate the fake and real images well in the 2D space.\n- Our best result was 0.9403 using CLIP ViT Large embeddings and training the classifier for 5000 epochs with a learning rate of 0.01, we could not reproduce the exact same result, but the result using CLIP ViT Large LAION-2B embeddings comes close.\n \n## Future Work\n- Use data augmentation to improve for better generalization and generation of additional data.\n- Use more advanced classifiers like SVMs or ensemble methods.\n- ...","metadata":{}}]}