## Full dataset is now available from Zenodo If you are interested in the **FSDKaggle2019** dataset used for this competition, please download it from **Zenodo**. The Zenodo version contains all the dataset files, **including the full test set and labels** as well as as updated csv files with metadata and licenses. **Please download FSDKaggle2019 at** [**https://doi.org/10.5281/zenodo.3612637**](https://doi.org/10.5281/zenodo.3612637) ## Citation If you use the FSDKaggle2019 dataset or baseline code, please cite our DCASE 2019 paper: > Eduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis, Xavier Serra. *Audio tagging with noisy labels and minimal supervision*. In Proceedings of DCASE2019 Workshop, NYC, US (2019). URL: https://arxiv.org/abs/1906.02975 You can also consider citing our ISMIR 2017 paper, which describes how we gathered the manual annotations included in FSDKaggle2019. > Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry > Bogdanov, Andrés Ferraro, Sergio Oramas, Alastair Porter, and Xavier > Serra. *Freesound datasets: a platform for the creation of open audio > datasets*. In Proceedings of the 18th International Society for Music > Information Retrieval Conference (ISMIR 2017), pp 486-493. Suzhou, > China, 2017. ---- ---- ## Detected corrupted files in the curated train set The following 5 audio files in the curated train set have a wrong label, due to a bug in the file renaming process: `f76181c4.wav, 77b925c2.wav, 6a1f682a.wav, c7db12aa.wav, 7752cc8a.wav` The audio file `1d44b0bd.wav` in the curated train set was found to be corrupted (contains no signal) due to an error in format conversion. ---- ---- # Task Motivation Current machine learning techniques require large and varied datasets in order to provide good performance and generalization. However, manually labelling a dataset is time-consuming, which limits its size. Websites like Freesound or Flickr host large volumes of user-contributed audio and metadata, and labels can be inferred automatically from the metadata and/or making predictions with pre-trained models. Nevertheless, these automatically inferred labels might include a substantial level of label noise. The main research question addressed in this competition is how to adequately exploit a small amount of reliable, manually-labeled data, and a larger quantity of noisy web audio data in a multi-label audio tagging task with a large vocabulary setting. In addition, since the data comes from different sources, the task encourages domain adaptation approaches to deal with a potential domain mismatch. # Audio Dataset The dataset used in this challenge is called **FSDKaggle2019**, and it employs audio clips from the following sources: - Freesound Dataset ( [FSD](https://annotator.freesound.org/fsd/) ): a dataset being collected at the [MTG-UPF](https://www.upf.edu/web/mtg) based on [Freesound](https://freesound.org/) content organized with the [AudioSet Ontology](https://research.google.com/audioset////////ontology/index.html) - The soundtracks of a pool of Flickr videos taken from the [Yahoo Flickr Creative Commons 100M dataset (YFCC)](http://code.flickr.net/2014/10/15/the-ins-and-outs-of-the-yahoo-flickr-100-million-creative-commons-dataset/) The audio data is labeled using a vocabulary of 80 labels from Google’s [AudioSet Ontology](https://research.google.com/audioset////////ontology/index.html) [1], covering diverse topics: Guitar and other Musical instruments, Percussion, Water, Digestive, Respiratory sounds, Human voice, Human locomotion, Hands, Human group actions, Insect, Domestic animals, Glass, Liquid, Motor vehicle (road), Mechanisms, Doors, and a variety of Domestic sounds. The full list of categories can be inspected in the `sample_submission.csv` at the the bottom of this page. ### Ground Truth Labels The ground truth labels are provided at the clip-level, and express the presence of a sound category in the audio clip, hence can be considered *weak* labels or tags. Audio clips have variable lengths (roughly from 0.3 to 30s, see more details below). The audio content from [FSD](https://annotator.freesound.org/fsd/) has been manually labeled by humans following a data labeling process using the [Freesound Annotator](https://annotator.freesound.org/) platform. Most labels have inter-annotator agreement but not all of them. More details about the data labeling process and the [Freesound Annotator](https://annotator.freesound.org/) can be found in [2]. The [YFCC](http://code.flickr.net/2014/10/15/the-ins-and-outs-of-the-yahoo-flickr-100-million-creative-commons-dataset/) soundtracks were labeled using automated heuristics applied to the audio content and metadata of the original Flickr clips. Hence, a substantial amount of label noise can be expected. The label noise can vary widely in amount and type depending on the category, including in- and out-of-vocabulary noises. More information about some of the types of label noise that can be encountered is available in [3]. ### Format and License All clips are provided as uncompressed PCM 16 bit, 44.1 kHz, mono audio files. All clips used in this competition are released under Creative Commons (CC) licenses, some of them requiring attribution to their original authors and some forbidding further commercial reuse. In order to be able to comply with the CC licenses terms, a full list of audio clips with their associated licenses and a reference to the original content (in Freesound or Flickr) will be published at the end of the competition. Until then, the provided audio files can only be used for the sole purpose of participating in the competition. # Train set The **train set** is meant to be for system development. The idea is to limit the supervision provided (i.e., the manually-labeled data), thus promoting approaches to deal with label noise. The train set is composed of **two subsets** as follows: ### Curated subset The curated subset is a small set of manually-labeled data from [FSD](https://annotator.freesound.org/fsd/) . - Number of clips/class: 75 except in a few cases (where there are less) - Total number of clips: 4970 - Avge number of labels/clip: 1.2 - Total duration: 10.5 hours The duration of the audio clips ranges from 0.3 to 30s due to the diversity of the sound categories and the preferences of Freesound users when recording/uploading sounds. It can happen that a few of these audio clips present additional acoustic material beyond the provided ground truth label(s). ### Noisy subset The noisy subset is a larger set of noisy web audio data from Flickr videos taken from the [YFCC](http://code.flickr.net/2014/10/15/the-ins-and-outs-of-the-yahoo-flickr-100-million-creative-commons-dataset/) dataset [5]. - Number of clips/class: 300 - Total number of clips: 19815 - Avge number of labels/clip: 1.2 - Total duration: ~80 hours The duration of the audio clips ranges from 1s to 15s, with the vast majority lasting 15s. Considering the numbers above, per-class data distribution available for training is, for most of the classes, 300 clips from the noisy subset and 75 clips from the curated subset, which means 80% noisy - 20% curated at the clip level (not at the audio duration level, considering the variable-length clips). # Test set The **test set** is used for system evaluation and consists of manually-labeled data from [FSD](https://annotator.freesound.org/fsd/) . Since most of the train data come from [YFCC](http://code.flickr.net/2014/10/15/the-ins-and-outs-of-the-yahoo-flickr-100-million-creative-commons-dataset/) , some acoustic domain mismatch between the train and test set can be expected. All the acoustic material present in the test set is labeled, except human error, considering the vocabulary of 80 classes used in the competition. The test set is split into two subsets, for the **public** and **private** leaderboards. In this competition, the submission is to be made through Kaggle Kernels. Only the test subset corresponding to the *public* leaderboard is provided (without ground truth). Submissions must be made with inference models running in Kaggle Kernels. However, participants can decide to train also in the Kaggle Kernels or offline (see [Kernels Requirements](https://www.kaggle.com/c/freesound-audio-tagging-2019/overview/kernels-requirements) for details). This is a kernels-only competition with two stages. The first stage comprehends the submission period until the deadline on June 10th. After the deadline, in the second stage, Kaggle will rerun your selected kernels on an unseen test set. **The second-stage test set is approximately three times the size of the first.** You should plan your kernel's memory, disk, and runtime footprint accordingly. ## Files * **train_curated.csv** - ground truth labels for the **curated** subset of the training audio files (see Data Fields below) * **train_noisy.csv** - ground truth labels for the **noisy** subset of the training audio files (see Data Fields below) * **sample_submission.csv** - a sample submission file in the correct format, including the correct sorting of the sound categories; it contains the list of audio files found in the **test.zip** folder (corresponding to the public leaderboard) * **train_curated.zip** - a folder containing the audio (.wav) training files of the **curated** subset * **train_noisy.zip** - a folder containing the audio (.wav) training files of the **noisy** subset * **test.zip** - a folder containing the audio (.wav) test files for the public leaderboard ## Columns Each row of the **train_curated.csv** and **train_noisy.csv** files contains the following information: - **fname**: the audio file name, eg, `0006ae4e.wav` - **labels**: the audio classification label(s) (ground truth). Note that the number of labels per clip can be one, eg, `Bark` or more, eg, `"Walk_and_footsteps,Slam"`. # References [1] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. "Audio set: An ontology and human-labeled dataset for audio events." In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2017. [ [PDF](https://ai.google/research/pubs/pub45857) ] [2] Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. "Freesound Datasets: A Platform for the Creation of Open Audio Datasets." In Proceedings of the International Conference on Music Information Retrieval, 2017. [ [PDF](https://repositori.upf.edu/bitstream/handle/10230/33299/fonseca_ismir17_freesound.pdf) ] [3] Eduardo Fonseca, Manoj Plakal, Daniel P. W. Ellis, Frederic Font, Xavier Favory, and Xavier Serra. "Learning Sound Event Classifiers from Web Audio with Noisy Labels." In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2019. [ [PDF](https://arxiv.org/abs/1901.01189) ] [4] Frederic Font, Gerard Roma, and Xavier Serra. "Freesound technical demo." Proceedings of the 21st ACM international conference on Multimedia, 2013. [https://freesound.org](https://freesound.org/) [5] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li, YFCC100M: The New Data in Multimedia Research, Commun. ACM, 59(2):64–73, January 2016