Submissions are evaluated by computing the weighted mean of three metrics: mean [Average Precision](https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Average_precision) (AP) on relationships detection, [Recall@N](https://medium.com/@m_n_malaeb/recall-and-precision-at-k-for-recommender-systems-618483226c54) (where N=50), mean [Average Precision](https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Average_precision) on phrase detection (mean in mean Average Precision is taken over per-relationship APs).
See more details about the metric on [Open Images Challenge website](https://storage.googleapis.com/openimages/web/vrd_detection_metric.html).

The weights applied to each of the 3 metrics are `[0.4, 0.2, 0.4]`. 

Kaggle's production code in C# can be viewed [here](https://gist.github.com/wendykan/3ef30f74355f7da5ea9dfabcee4c13da). The metric is also implemented as a part of [Tensorflow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection). See [this Tutorial](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md) on running the evaluation in Python.

## Kernel Submissions
You can make submissions directly from Kaggle Kernels. By adding your teammates as collaborators on a kernel, you can share and edit code privately with them.

## Submission File
For each image in the test set, you must predict a list of boxes describing objects in the image. Each box is described as 

      ImageId, PredictionString 
      ImageId, {Confidence Label1 XMin1 YMin1 XMax1 YMax1 Label2 XMin2 YMin2 XMax2 YMax2 RelationLabel}, {...}