## Metric

Submissions are evaluated according to the mean Precision @ 5 metric, where we introduce a small modification to avoid penalizing queries with fewer than 5 expected index images. In detail, the metric is computed as follows:

$$mP@5 = \frac{1}{Q} \sum_{q=1}^{Q} \frac{1}{min(n\_q, 5)} \sum\_{j=1}^{min(n\_q, 5)} rel\_q(j)$$

where:

- \\(Q\\) is the number of query images
- \\(n\_q\\) is the number of index images containing an object in common with the query image \\(q\\). Note that \\(n_q \gt 0\\).
- \\(rel_q(j)\\) denotes the relevance of prediciton \\(j\\) for the \\(q\\)-th query: it’s 1 if the \\(j\\)-th prediction is correct, and 0 otherwise

## Submission File

Unlike a traditional Kaggle code competition where notebooks are rerun top-to-bottom on a private test set, in this competition you will be submitting a model file. The model must take an image as an input, and return a float vector (i.e., the image embedding) at the output. The embedding dimensionality should be no greater than **64**. Your model must be packaged into a `submission.zip` file, and compatible with either TensorFlow 2.6.4 or Pytorch 1.11.0. In most cases, scoring is expected to take a few hours to complete.

The `submission.zip` should contain one (and only one) of the following:

- files and directories in the Tensorflow's [SavedModel format](https://www.tensorflow.org/tutorials/keras/save_and_load#savedmodel_format).
- file in PyTorch’s [TorchScript format](https://pytorch.org/docs/stable/jit.html), named “saved_model.pt”.

To emphasize once again: the embedding produced by the model must be at most 64D (if not, the submission will error out).

## Submission process

Kaggle will use the submitted model to:

 1. Extract embeddings for the private test and index image sets.
 2. Create a kNN (k = 5) lookup for each test sample, using the Euclidean distance between test and index embeddings.
 3. Score the quality of the lookups using the competition metric.

## Baseline kernels

To submit your model, all you need is a kernel that produces a `submission.zip` file containing the model. Here are examples:

**Tensorflow.** See [this kernel](https://www.kaggle.com/code/francischen1991/tf-baseline-v2-submission) for the minimal baseline model and [this kernel](https://www.kaggle.com/andrefaraujo/tf-baseline-submission) for the more complicated baseline model.

**PyTorch.** See [this kernel](https://www.kaggle.com/andrefaraujo/pytorch-baseline-submission).

## Code example to produce model

For **Tensorflow**, we provide a minimal **Tensorflow** baseline model based on pretrained Inception-v3 model with a linear layer to project the output feature to 64D.  In the end, the model is saved in the SavedModel format: 

```
import tensorflow as tf

image = tf.keras.layers.Input([None, None, 3], dtype=tf.uint8)
output = tf.cast(image, tf.float32)
output = tf.keras.layers.Lambda(lambda x: tf.image.resize(x, [224, 224]), name='resize')(output)
output = tf.keras.applications.inception_v3.preprocess_input(output)
output = tf.keras.applications.inception_v3.InceptionV3(weights='imagenet', input_shape=[None, None, 3],
                                                        include_top=False, pooling='avg')(output)
output = tf.keras.layers.Dense(64, name='embedding')(output)
output = tf.keras.layers.Lambda(lambda x: tf.math.l2_normalize(x, axis=-1), name='embedding_norm')(output)
model = tf.keras.Model(inputs=[image], outputs=[output])
model.summary()

tf.saved_model.save(model, 'inceptionv3/')
```
For more complicated examples, please check the example given in [this repository](https://github.com/google-research/google-research/tree/master/universal_embedding_challenge). Specifically, [this script](https://github.com/google-research/google-research/blob/master/universal_embedding_challenge/export_saved_model.py) shows an example of model exporting, for a model that is trained with [this script](https://github.com/google-research/google-research/blob/master/universal_embedding_challenge/train.py).

For **PyTorch**, see a similar example below that uses pretrained Inception-v3 model and a linear projection layer, and exports the model in the TorchScript format:

```
import torch
import torch.nn as nn
from torchvision import models
from torchvision import transforms

class MyModel(nn.Module):
  def __init__(self):
    super().__init__()
    inception_model = models.inception_v3(pretrained=True)
    inception_model.fc = nn.Linear(2048, 64)
    self.feature_extractor = inception_model
            
  def forward(self, x):
    x = transforms.functional.resize(x,size=[224, 224])
    x = x/255.0
    x = transforms.functional.normalize(x, 
                                            mean=[0.485, 0.456, 0.406], 
                                            std=[0.229, 0.224, 0.225])
    return self.feature_extractor(x).logits
    
model = MyModel()
model.eval()
saved_model = torch.jit.script(model)
saved_model.save('saved_model.pt')
```

## Feature extraction code

When your model is submitted, we run it in the competition images in order to extract embeddings. Please find below code snippets that show how the model is used in detail. These could be helpful for participants to make sure that their models are extracting embeddings correctly.

**Tensorflow**:

```
import numpy as np
from PIL import Image
import tensorflow as tf

# Model loading.
model = tf.saved_model.load(saved_model_path)
embedding_fn = model.signatures["serving_default"]

# Load image and extract its embedding.
image_tensor = tf.convert_to_tensor(
        np.array(Image.open(image_path).convert("RGB"))
    )
expanded_tensor = tf.expand_dims(image_tensor, axis=0)
embedding = embedding_fn(expanded_tensor)["embedding_norm"]
```

**PyTorch**:

```
from PIL import Image
import torch
from torchvision import transforms

# Model loading.
model = torch.jit.load(saved_model_path)
model.eval()
embedding_fn = model

# Load image and extract its embedding.
input_image = Image.open(image_path).convert("RGB")
convert_to_tensor = transforms.Compose([transforms.PILToTensor()])
input_tensor = convert_to_tensor(input_image)
input_batch = input_tensor.unsqueeze(0)
with torch.no_grad():
  embedding = torch.flatten(embedding_fn(input_batch)[0]).cpu().data.numpy()
```