# Overview
CI-AVSR is a competition about recognizing Cantonese In-car Commands with multimodal data, including audio and video. The CI-AVSR dataset consists of 4,984 samples (8.3 hours of video and audio) of 200 unique in-car commands recorded by 30 native Cantonese speakers. In addition, each sample is augmented by 10 different in-car ambient noises that are commonly heard. The goal of this competition is to predict the ASR system trained on **clean** and/or **in-domain noises** data, and test the system on a combined test set covering **clean**, **in-domain noises**, and **out-of-domain noises** data

More details can be found in this paper: http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.731.pdf.
If you use this dataset, please consider to cite:
```
@InProceedings{dai-EtAl:2022:LREC,
  author    = {Dai, Wenliang  and  Cahyawijaya, Samuel  and  Yu, Tiezheng  and  Barezi, Elham J.  and  Xu, Peng  and  YIU, Cheuk Tung  and  Frieske, Rita  and  Lovenia, Holy  and  Winata, Genta  and  Chen, Qifeng  and  Ma, Xiaojuan  and  Shi, Bertram  and  Fung, Pascale},
  title     = {CI-AVSR: A Cantonese Audio-Visual Speech Datasetfor In-car Command Recognition},
  booktitle      = {Proceedings of the Language Resources and Evaluation Conference},
  month          = {June},
  year           = {2022},
  address        = {Marseille, France},
  publisher      = {European Language Resources Association},
  pages     = {6786--6793},
  url       = {https://aclanthology.org/2022.lrec-1.731}
}
```

In addition to the proposed challenge above, we also introduce an additional challenge to recognize in-car commands covering three scenarios (subtasks):

### Subtask-1: Clean Speech Cantonese Multimodal ASR

In subtask 1, we simulate an ideal situation for an ASR system where clean video and audio data is provided to participants. The clean multimodal data consists of the clean audio data without any background noise and the video of the front view of the speaker. Participants are invited to build a multimodal Cantonese ASR model using only the **train-clean** and {valid-clean} splits as the training and validation set, respectively. We as organizers will evaluate their trained models on the **test-clean** split. The output of the ASR models is graphemes of the corresponding speech command whose vocabulary consists of 284 written Cantonese characters and 26 lower-cased English alphabets. We will provide the complete vocabulary along with the dataset.

### Subtask-2: In-domain Noisy Speech Cantonese Multimodal ASR
In subtask 2, we move closer to a real scenario of the in-car multimodal ASR setting, where the given speech command might be intervened by some noises such as the sounds of the engine combustion, horn, and other ambient noises. In this subtask, we assume that all the ambient noises are known such that additional training data with similar characteristic noises can be generated. Specifically, in subtask 2, we focus on evaluating the model's performance on in-domain (i.e. seen in the training set) data with five different noises, including windscreen wiper, horn, alarm, engine ignition, and door noises. Participants are allowed to train their models on the **train-clean** and **train-noisy (in-domain)** sets, and validate their models on both the **valid-clean** and **valid-noisy (in-domain)** sets. We will score their results on the **test-noisy (in-domain)**.

### Subtask-3: Out-of-domain Noisy Speech Cantonese Multimodal ASR
In subtask 3, we move even further to the real scenario of the in-car multimodal ASR setting, where we assume that the characteristics of noises are unknown and only appear during the inference time. More specifically, we focus on evaluating the generalization ability of models to see whether they can still perform well on out-of-domain (i.e. unseen in the training data) noisy data, which includes five various noises (background music, rain, hail, people talking, road ambiance). Participants are allowed to train their models on the **train-clean** and **train-noisy (in-domain)** sets, and validate their models on the **valid-clean** and **valid-noisy (in-domain)** sets. We will evaluate their models on the **test-noisy (out-of-domain)** set.  

If you are interested in the challenge, you can get the data split on our Github repo at https://github.com/HLTCHKUST/CI-AVSR, and consider citing our shared task proposal as follow:
```
@article{Dai2022KaggleCC,
  title={Kaggle Competition: Cantonese Audio-Visual Speech Recognition for In-car Commands},
  author={Wenliang Dai and Samuel Cahyawijaya and Tiezheng Yu and Elham J. Barezi and Pascale Fung},
  journal={ArXiv},
  year={2022},
  volume={abs/2207.02663}
}
```