This competition is closed for submissions. Participants' selected code submissions were re-run by the host on a privately-held test set and the private leaderboard results have been finalized. Late submissions will not be opened, due to an inability to replicate the unique design of this competition.
\n\n\nThis code competition's training set is not available directly on Kaggle, as its size is prohibitively large to train in Kaggle. Instead, it's strongly recommended that you train offline and load the externally trained model as an external dataset into Kaggle Notebooks to perform inference on the Test Set. Review Getting Started for more detailed information.
\n\nThe full training set is just over 470 GB. We've made it available as one giant file, as well as 50 smaller files, each ~10 GB in size. You must accept the competition's rules to gain access to any of the links below.
\n\nFull dataset:
\n\n| all.zip (471.84 GB)\n |
Dataset split into smaller chunks:
\n\nA small sample training set has been made available below.
\nYou'll also need the sample_submission.csv, as it indicates the IDs you'll be making predictions for.
The data is comprised of .mp4 files, split into compressed sets of ~10GB apiece. A metadata.json accompanies each set of .mp4 files, and contains filename, label (REAL/FAKE), original and split columns, listed below under Columns.
You will be predicting whether or not a particular video is a deepfake. A deepfake could be either a face or voice swap (or both). In the training data, this is denoted by the string "REAL" or "FAKE" in the label column. In your submission, you will predict the probability that the video is a fake.
metadata.json with labels. the full set of training videos is available through the links provided above.To understand the datasets available for this competition, review the Getting Started information.
\nfilename - the filename of the videolabel - whether the video is REAL or FAKEoriginal - in the case that a train set video is FAKE, the original video is listed heresplit - this is always equal to "train".![](\"https://storage.googleapis.com/kaggle-media/competitions/deepfakes/111419_Deepfake_Kaggle_GraphicDescription_Male_350x350.jpg\")
\n\nAWS, Facebook, Microsoft, the Partnership on AI’s Media Integrity Steering Committee, and academics have come together to build the Deepfake Detection Challenge (DFDC). The goal of the challenge is to spur researchers around the world to build innovative new technologies that can help detect deepfakes and manipulated media.\n\nChallenge participants must submit their code into a black box environment for testing. Participants will have the option to make their submission open or closed when accepting the prize. Open proposals will be eligible for challenge prizes as long as they abide by the open source licensing terms. Closed proposals will be proprietary and not be eligible to accept the prizes. Regardless of which track is chosen, all submissions will be evaluated in the same way. Results will be shown on the leaderboard. \n\nThe PAI Steering Committee has emphasized the need to ensure that all technical efforts incorporate attention to how the resulting code and products based on it can be made as accessible and useful as possible to key frontline defenders of information quality such as journalists and civic leaders around the world. The DFDC results will be a contribution to this effort and building a robust response to the emergent threat deepfakes pose globally. ",
"isPublished": false,
"hasIsPublished": false,
"postTitle": "",
"hasPostTitle": false,
"mimeType": "",
"hasMimeType": false
},
{
"name": "Timeline",
"content": "* **March 3, 2020** - Entry deadline. You must accept the competition rules before this date in order to compete.\n\n* **March 3, 2020** - Team Merger deadline. This is the last day participants may join or merge teams.\n\n* **March 31, 2020** - Final submission deadline.\n\n* **June 12, 2020** - Private leaderboard revealed.\n\nNote: Because the private test set is held outside of Kaggle's platform, the private leaderboard will not be revealed immediately. We have estimated the scoring timeframe, but this is subject to revision.\n\nAll deadlines are at 11:59:00 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.",
"isPublished": false,
"hasIsPublished": false,
"postTitle": "",
"hasPostTitle": false,
"mimeType": "",
"hasMimeType": false
},
{
"name": "Prizes",
"content": "- **First Prize**: $500,000 USD\n- **Second Prize**: $300,000 USD\n- **Third Prize**: $100,000 USD\n- **Fourth Prize**: $60,000 USD\n- **Fifth Prize**: $40,000 USD \n",
"isPublished": false,
"hasIsPublished": false,
"postTitle": "",
"hasPostTitle": false,
"mimeType": "",
"hasMimeType": false
},
{
"name": "Code Requirements",
"content": "\n### This is a Code Competition\n\nSubmissions to this competition must be made through Notebooks. In order for the \"Submit to Competition\" button to be active after a commit, the following conditions must be met:\n\n- CPU Notebook <= 9 hours run-time\n- GPU Notebook <= 9 hours run-time on Kaggle's [P100 GPUs](https://www.kaggle.com/product-feedback/83643)\n- No internet access enabled\n- External data is allowed up to 1 GB in size. External data must be freely & publicly available, including pre-trained models. \n- No using other Kaggle notebooks or utility scripts as inputs to your submission notebook. External data also cannot be externally-referenced (i.e. via BigQuery, Github, external URL). Instead, load external models or datasets directly as an external data source.\n- No custom packages enabled in your submission notebook\n- Submission file must be named \"submission.csv\"\n\nPlease see the [Code Competition FAQ](https://www.kaggle.com/docs/competitions#kernels-only-FAQ) and [Getting Started](https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started) for specifics on this competition's unique design.",
"isPublished": false,
"hasIsPublished": false,
"postTitle": "",
"hasPostTitle": false,
"mimeType": "",
"hasMimeType": false
},
{
"name": "Getting Started",
"content": "### **Datasets**: \nThere are 4 groups of datasets associated with this competition.\n\n1. **Training Set: This dataset, containing labels for the target, is available for download outside of Kaggle for competitors to build their models.** It is broken up into 50 files, for ease of access and download. Due to its large size, it must be accessed through a GCS bucket which is only made available to participants after accepting the competition’s rules. Please read the rules fully before accessing the dataset, as they contain important details about the dataset’s permitted use. It is expected and encouraged that you train your models outside of Kaggle’s notebooks environment and submit to Kaggle by uploading the trained model as an external data source.\n2. **Public Validation Set**: When you commit your Kaggle notebook, the submission file output that is generated will be based on the small set of 400 videos/ids contained within this Public Validation Set. This is available on the Kaggle Data page as `test_videos.zip`\n3. **Public Test Set: This dataset is completely withheld and is what Kaggle’s platform computes the public leaderboard against.** When you “Submit to Competition” from the “Output” file of a committed notebook that contains the competition’s dataset, your code will be re-run in the background against this Public Test Set. When the re-run is complete, the score will be posted to the public leaderboard. If the re-run fails, you will see an error reflected in your “My Submissions” page. Unfortunately, we are unable to surface any details about your error, so as to prevent error-probing. You are limited to 2 submissions per day, including submissions which error.\n4. **Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard.** It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes. After the competition deadline, Kaggle transfers your 2 final selected submissions’ code to the host. They will re-run your code against this private dataset and return prediction submissions back to Kaggle for computing your final private leaderboard scores.\n\n### **Workflow:** \nThis is a walk-through of how to make a submission to this code competition. The workflow is unique to this competition, so please read carefully.\n\n1. **Accept the competition’s rules (Kaggle)**. Read these carefully before accepting, as there are important commitments you are making to the host about the use of their data.\n2. **Access & download the Training Set (Google Cloud Storage Bucket)**. This is hosted outside of Kaggle. The link to this dataset is made available only after accepting the competition’s rules. See details under the “Datasets” section above.\n3. **Train your models (Your choice - Local Machine, VM)**. You should be training your models outside of Kaggle using local or cloud resources. [AWS is offering promotional credits](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122254) for those needing additional computational resources to approach this challenge.\n4. **Use Kaggle Notebooks (Notebook or Script) to create your submission (Kaggle)**. This is a code competition, so submissions must be made through Kaggle. You can upload any pre-trained or locally-trained models as data sources to your submission notebook. Your submission is subject to the [Code Requirements](https://www.kaggle.com/c/deepfake-detection-challenge/overview/code-requirements) mandated by this competition.\n5. **Commit your Notebook/Script (Kaggle)**. Click commit on the notebook to run your code against the Public Validation Set. If successful, the notebook will generate a `submission.csv` file as an “Output”\n6. **“Submit to Competition” from the “Output” file created (Kaggle)**. This will make your official submission to the competition. The “Submit to Competition” button will only be available if your notebook meets the requirements specified in [Code Requirements](https://www.kaggle.com/c/deepfake-detection-challenge/overview/code-requirements). Your code will be re-run against the Public Test Set in the background. If successful, it will generate a score on the public leaderboard. You can review your submissions’ status on the “My Submissions” page. Note that this can take longer than the prior commit step due to the size of Public Test Set being larger than the Public Validation Set.\n7. **Select 2 final submissions by the Submission Deadline (Kaggle)**. On your “My Submissions” page, your team leader can select the checkbox under “Use for final score” next to the 2 submissions to be re-run against the Private Test set.\n8. **Wait and see!** After the submission deadline, Kaggle will transfer your 2 selected submissions’ code to the host. They will re-run your code against their Private Test Set and return prediction submissions back to Kaggle for computing your final private leaderboard scores. For the private test set evaluation, the time constraint for your code’s run time will scale with that private test set's size. Review the [Timeline Page](https://www.kaggle.com/c/deepfake-detection-challenge/overview/timeline) for details of the expected timing for the private leaderboard review.\n\n###**Tips for successful submissions**:\n\n- **Train your models outside of Kaggle.** The training set is intentionally not available as a competition data source directly through Kaggle’s notebooks. It is not practical to train a model on Kaggle within the time constraints imposed.\n- **Your code must output a submission.csv that predicts on any set of `test_videos.zip` that it is fed as an input.** We’ve designed the Public Validation/Test Sets to aid in ensuring your code is robust enough for the Private Test Set. The Public Validation Set gives an example of what you can expect your model should handle while working in an interactive notebook session. The Public Test Set allows you to test your code’s performance on a withheld test set. \n- **Ensure your code can be re-run on any unseen test set.** Do not hard-code your submissions based on the public validation set. This will result in failure when your code runs against the public test set. You will not benefit from probing the Public Validation/Test Sets, because your code must perform honestly on the host’s fully Private Test Set.\n- **You are limited to 9 hours of GPU compute time.** This constraint is also imposed on the Public Test Set re-run. You should anticipate the Public Test set to be 10 times the size of the Public Validation Set and budget accordingly. The 9 hour compute constraint will be applied to the submission in Kaggle, when evaluated against the Public Test Set. So if you are able to make a successful submission that posts a score to the public leaderboard without an error, then it has met the 9 hour constraint on the Public Test Set. Review the full set of [Code Requirements](https://www.kaggle.com/c/deepfake-detection-challenge/overview/code-requirements).\n- **You are still limited to Kaggle’s [global weekly per user GPU time constraints](https://www.kaggle.com/general/108481)**.\n- **Don’t make the mistake of creating multiple accounts or private sharing.** Any user who submits from multiple accounts will be removed at the end of the competition, without exception. This includes entire teams where only 1 member has created a multiple account. It also includes teams who are found to have shared privately with other teams or colluded with other teams to make any submissions. It also includes teams who engaged in private sharing prior to forming as a team. We will complete these removals at the end of the competition. If you have knowledge of any teams who have committed rules violations, please submit a claim to [Kaggle Compliance](https://www.kaggle.com/compliance) with your specific evidence.\n- **Resolving Submission Errors**: Below is a quick rundown of the some errors that users may receive on submission. Note that we are not in a position to share more granular details of any precise exceptions/errors, as doing so would make the competition vulnerable to error-probing by savvy participants. \n - **Notebook Timeout**: Your submission notebook exceeded the allowed runtime cap for the competition. Review the competition's [Code Requirements](https://www.kaggle.com/c/deepfake-detection-challenge/overview/code-requirements) for details, and note that the privately held rerun test set could be orders of magnitude larger than the publicly shared validation set.\n - **Submission Scoring Error**: Your notebook generated a `submission.csv` file with incorrect format. Some examples causing this are having the wrong number of rows or columns, empty values, an incorrect datatype for a value, or invalid submission values from what is expected for the privately rerun test set.\n - **Notebook Threw Exception**: While rerunning your code on the public test set (privately held), your notebook hit an error. \n - **Submission CSV Not Found**: Your Notebook did not output a `submission.csv` file. Some examples causing this are if your Notebook times out, or if it runs into an error. There is a chance this will be the error returned instead of \"Notebook Timeout\" depending on error sequencing.\n - **No `Submit to Competition` button on my notebook's output**: This indicates that you have enabled something that is prohibited for the submissions on that competition. Review the competition's [Code Requirements](https://www.kaggle.com/c/deepfake-detection-challenge/overview/code-requirements) for details\n - **Notebook Exceeded Allowed Compute**: This indicates you have violated a [Code Requirement](https://www.kaggle.com/c/deepfake-detection-challenge/overview/code-requirements) constraint during the rerun.\n - **Kaggle Error**: A rare system error. Please try resubmitting to resolve the error and contact Kaggle Support if it persists.\n\n\n\n**See the Deepfake Detection Challenge website for answers to more [Frequently Asked Questions](https://deepfakedetectionchallenge.ai/faqs).**",
"isPublished": false,
"hasIsPublished": false,
"postTitle": "",
"hasPostTitle": false,
"mimeType": "",
"hasMimeType": false
},
{
"name": "Evaluation",
"content": "Submissions are scored on log loss:\n\n\n$$\n\\textrm{LogLoss} = - \\frac{1}{n} \\sum_{i=1}^n \\left[ y_i \\log(\\hat{y}_i) + (1 - y_i) \\log(1 - \\hat{y}_i)\\right],\n$$\n
\n\nwhere\n\n- n is the number of videos being predicted\n- \\\\( \\hat{y}_i \\\\) is the predicted probability of the video being FAKE\n- \\\\( y_i \\\\) is 1 if the video is FAKE, 0 if REAL\n- \\\\( log() \\\\) is the natural (base e) logarithm\n\nA smaller log loss is better. The use of the logarithm provides extreme punishments for being both confident and wrong. In the worst possible case, a prediction that something is true when it is actually false will add infinite to your error score. In order to prevent this, predictions are bounded away from the extremes by a small value.\n\n## Submission File\nFor each `filename` in the test set, you must predict a probability for the `label` variable. The file should contain a header and have the following format:\n\n filename,label\n 10000.mp4,0\n 10001.mp4,0.5\n 10002.mp4,1\n etc.\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }