{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Kaggle 2023 AI Report Essay - Category: Other (anything that does not fall into any other category):- Audio Data\n\nBy: Aderemi O.\n\nKaggle AI Essay: Audio Data Category.\n\nThe human voice as the ultimate instrument not only alludes to its ability to create sounds that capture thoughts and feelings instantaneously but to its utility that predates literacy. Long before poems were typed or written, they were meant to be read aloud as they were a way to record and memorize history - like the epic of Gilgamesh whose narration includes encounters with kings, gods and creatures not to forget human relationships, love, loss, loneliness, friendships, aspiration, ambition and fear of demise. Storytellers with the auditory nature of poems, memorized events and facts, so that in a group, they became historians.\n\nThe emergence of other sounds - with or without the accompaniment of the human voice - have also had broad applications in uplifting, delighting and enriching the human condition. For example Fredrich Nietzsche reiterated 'life without music will be a mistake' a sentiment many a few agree with. And some form of audio have been applied in sound therapy to assist with mental well being - for not all who are roaming are sane.\n\nHowever, despite the existence of a broad range of audio/sounds generated by nature, humans have used machines/computers to mimic the task of generating audio or sounds, like it has mimicked human activities in other domains.\n\nTurin's test from Alan Turin's scholarly paper 'Computing machinery and Intelligence' measures a machines ability to exhibit intelligent behaviour equivalent to or indistinguishable from, that of a human. And there have been quite some feat embarked upon in the audio domain:\n\n1965 - Ray Kurzweil: a piano piece created by a computer that is capable of pattern recognition in various compositions - the computer then uses these patterns to create new melodies.\n\n1970 - techniques for music composition, audio compression and audio restoration; MIDI(musical instrument digital interface), allows computers communicate with musical instruments.\n\n1977 - Standford researchers, Music1, a computer program that can generate original melodies.\n\n1978 - Digital audio workstation (DAW) generate and manipulate sound - Fairlight CMI.\n\n1980 - Max Matthew develops Music V system, a computer program that generates original melodies and harmonies.\n\n1990 - MP3 format audio compression without sacrificing quality.\n\n2001 - Francois Pat her - Continuator - machine learning software that improves along with musicians in real time.\n\n2016 - WaveNet, a deep learning model developed to generate realistic audio of human voices.\n\n2020 - Nvidia's WaveGlow: a neural network that can generate high quality audio from text.\n\nThe audio domain though have seen developments in recent years that have pushed the envelop of discovery. It is my attempt to classify AI activity in audio category from dataset given on Arxiv into speech recognition, music production/composition, realistic speech generation and audio compression.\n\nSPEECH RECOGNITION\n\nDifferent models have been applied to deliver speech recognition algorithms based on task being addressed. For example, to reduce data required to obtain state-of-the-art (SOTA) results, Wav2Vec models have been introduced, though normally applied as bidirectional encoder representation from transformer models in NLP - the speech community adopting some of their development methodologies.\n\nSome cases require Speech Emotion Recognition (SER) to obtain emotional intelligence and understand contextual meaning of speech. Hence, researches have explored implication of acoustic context and phone boundaries on local markers for SER using attention-based approach.\n\nResearch for multi-talker speech recognition (MT-ASR) has been shown to improve ASR performance on speech containing overlapping utterances from more than one speaker. The MT-ASR model is formed by combining well trained foundational model with a multi-talker mask model in a cascaded RNN-T encoder configuration.\n\nTwo state-of-the-art ASR systems (Wav2Vec2.0 and Whisper) were used to develop voicebots geared toward assisting kids acquire a foreign language.\n\nWhile traditional reference based metrics for evaluating ASR systems require costly ground-truth transcripts, a paper introduced NoRefER, a novel referenceless quality metric for ASR systems. NoRefER overcomes the shortcomings of referenced based metrics by fine-tuning a multi-lingual language model for pair-wise ranking ASR hypothesis using constructive learning with Siamese network architecture.\n\nThere have been improvement of dysarthric speech recognition using multi-stage augmented approach. Despite advancement in ASR, the SOTA ASR systems struggle to deal with impaired speech even with high resource languages. For some langauges the challenge gets amplified with added complexities  in  collecting data for dysarthric speakers. One paper aimed to improve the performance of dysarthric automatic speech recognition. A signal-based approach to generating dysarthric speech from healthy speech by modifying the speed and tempo, was first employed. Followed by Parallel Wave Generative (PWG) adversarial model that is trained on an English dysarthric dataset to isolate language-independent dysarthric speech patterns and further augment the signal adjusted speech samples. Next a fine-tuning and text correction strategies for conformer at different dysarthric speech levels. The fine-tuned confirmed achieved 18% word error rate (WER) and 17.2% character error rate (CER) on synthetically generated dysarthric speech from common voice speech dataset. This shows significant WER improvement of 81.8% compared to the baseline model trained solely on healthy data. Further validation on real English dysarthric speech shows a WER improvement of 124% compared to baseline trained on only healthy English LJSpeech dataset.\n\nAlso, increasing fairness and robustness of end-to-end speech recognition so equality and equity for all groups is maintained and for cases where subgroups have been neglected, it could be restored. When automatic speech recognition does not perform well for all subgroups in a populace, a robustness issue ensues. A paper presented a privacy preserving approach to improve fairness and robustness of end-to-end ASR without using metadata, zip codes or even speaker or utterance embeddings directly in training.Utterance level embeddings using speaker model trained on public dstaset, are extracted,which when used in an unsupervised manner creates acoustic clusters. Cluster IDs instead of speaker utterance embeddings are used as extra features during model training, which shows improvements for all demographic groups and in particular for different accents.\n\nMUSIC PRODUCTION/COMPOSITION\n\nMaking soundtracks using AI is beginning to make in-roads. The MusicGen model examines the task of conditional music generation. MusicGen comprise of single-stage transfer language models (LM) together with efficient token interleaving patterns, which eliminates the need for cascading several models. It was demonstrated how MusicGen generates high quality samples, while being conditioned on textual description or melodic features, allowing better controls over generated output.\n\nOne project presented a deep learning approach to generate monophonic melodies based on input beats, allowing interested individuals create their own music compositions. Three effective methods - LSTM with full Attention, LSTM with local Attention and Transformer with Relative Position Representation - are proposed for this novel task, providing great variation, harmony and structure in the generated music. The project allows anyone to compose their music by tapping their keyboards or 'recoloring' beat sequences from existing works.\n\nRegarded as an interesting and challenging area of AI music research field, due to difficulty of learning correlations between lyric and melody, Lyric-to-Melody generation network, ConL2M, is able to generate realistic melodies from lyric in user desired music style. The model generation consist of three novelties: 1. model dependencies of music attributes cross multiple sequences; inter branch memory fusion (Memofu) is proposed to enable information flow between branch stacked LSTM architecture; 2. Reference Style Embeddings (RSE) is proposed to improve quality of generation as well as control the musical style of generated melodies; 3. Sequence Level Statistical Loss ( Seqloss) is proposed to help the model learn sequence level features of melodies given lyrics.\n\nReconstruction and reconceptualizing a piece using new instrumentation styles as a practice of music re-arrangement is an important task of automatic music generation. Existing methods model source to target piece mapping using supervised learning. The paper examines music re-arrangement practice using self-supervised learning, with mapping styles as conditions controlled flexibly. The paper presented QnA, a query-based algorithm for multi-track music re-arrangement under encoder-decoder framework. QnA learns content representation from the mixture and function representation from each individual track, while the latter queries the former in order to re-arrange a new piece. The model highlights four(4) scenarios: 1. Re-instrumentation; 2. Piano cover generation; 3. Orchestration; 4. Voice separation.\n\nThere was one paper that investigated transfer learning approach to improve the performance of a deep neural networked-based model for automatic music generation on out-of-distribution (OOD) music genres. It had been established that these music generation models perform poorly on OOD music genres. As an example of OOD music genre, Iranian folk music was chosen. It was observed the transfer learning approach adapted efficiently MusicVAE - a large generative music model - to the sample OOD folk music dataset, lending credence to the potential of generating under-represented music genres in the future.\n\nThe MusicLM model sought to generate high fidelity music from text descriptions. The model sees conditional music generation as hierarchical sequence-to-sequence modelling task. State-of-the-art generative modelling for many domains is often dominated by Transformer-based autoregressive models or U-Net-based diffusion models. MusicLM shares similarities with auto regressive generative models acting on discrete tokens. MusicLM was evaluated by comparison with two recent baselines for generating music from descriptive text - Mubert and Riffusion. It achieved high quality audio output and long-term coherence, while remaining true to the text conditioning.\n\nAUDIO COMPRESSION\n\nLanguage models have been successfully used to model natural signals such as images, speech and music. A key component of these models is a high quality neural compression model that can compress high dimensional discrete tokens. One paper introduces a high-fidelity universal neural audio compression algorithm that achieves ~90x compression of 44KHz audio into tokens at just 8Kbps bandwidth.\n\nAlso, Implicit Neural Representations (INR) as a method for representing diverse data modalities was known. But as a method of audio compression was unexplored until now. Thus the study investigated the use of INRs in audio compression by introducing Siamese SIREN, a novel approach, based on the SIREN architecture. The Siamese SIREN achieved superior audio reconstruction fidelity while utilizing fewer network parameters compared to earlier INR architectures.\n\nREALISTIC SPEECH RECOGNITION\n\nLipVoicer generates natural sounding speech from sound less video, using text modality. Advances with current methods cannot produce high-quality speech with high levels of intelligibility for challenging and realistic dataset as LSR3. LipVoicer introduces a novel method that produces high quality speech for in-the-wild and rich datasets. The spoken text is predicted, given a silent video, using a pre-trained lip-reading network. The silent video is then conditioned by a  diffusion model and the extracted text is used by a classifier-guidance mechanism with a pre-trained ASR serving as the classifier. For in-the-wild datasets with hundreds of unique speakers in their test set and an unrestricted vocabulary, LipVoicer outperforms multiple lip-to-speech baselines on LSR2 and LSR3. The introduction of text modality component positively impacts the intelligibility of the produced speech, its clarity while listening and is empirically reflected in the substantial reduction of the WER metric.\n\nAda-TTA's novel task was generate low resource text-to-talking avatar. First, a generic zero-shot multi-speaker TTS model that disentangles the text content, timbre and prosody. Second, it embraces recent advances in neural rendering to achieve realistic audio-driven talking face video generation. The challenge has been difficulty to mimic out-of-domain audio for a traditional multi-speaker text-to-speech system and the constraint of limited data to render high-fidelity lip-synchronized talking avatars. The method demonstrated by Adaptive Text-to-Talking Avatar (Ada-TTA) synthesized realistic identity preserving audio-visual synchronized talking avatar videos.\n\n** **\n\nWith services (Eleven Labs, Murf, Superfone, VoiceAI, Resemble,playHt) available to clone or generate realistic human voices or audio using deep neural learning models, many are amazed. Some have asked: how does one determine what is true/real? Or if attention and intimacy can be feigned, how does one guard against knowing, liking and trusting a nefarious character?\n\nWell, deception and manipulation have always showed up in human dealings with each other: Hamlet by Shakespeare and Odyssey by Homer contain records; Its just now it can be done by machines, assisted by humans, like a puppeteer.\n\nWhile epistemology offers four areas how we arrive at what is true [i. Correspondence theory of truth - we believe as true what corresponds to our observable reality; ii. Coherence theory of truth - we believe as true what is logical and coherent; iii. Consensus theory of truth - we believe as true what most people believe; iv. Pragmatic theory of truth - we believe as true what is beneficial to us.], we might have to apply Eli Pariser's suggestion from his book 'The Filter Bubble'of being aware of the problem and using multiple sources to corroborate what constitutes our observable and perceptible reality - for when we know what's happening around us, can we know what's happening to us.\n\nAnd even when emotion can be manufactured or replicated by feigning or prompts, like the podcast Song Exploder have shown, data or data points are not the plural or synonyms of anecdotes.\n\nWith concerns of AI usurping faculties and aptitudes, humans still wield the power of steering its direction in terms of its pursuits of advancement and preserving its race.\n\nDiscoveries continue to proliferate. With Soapbox Labs voice technology for kids,Mozilla's common voice project to facilitate speech enabled applications, Cornell's connected experience (CX) Labs experimenting with podcast format, Google's AIKitchen, Amazon's Alexa program, we look on to see how AI in audio unfolds as we endeavor to expand audio's utility with technology's help.\n\n\n\n\n\n\n\n\n\n\n\n\n\n** **\n\nBibliography \n\n\nStuff You Should Know Podcast - How Poetry Works\n\nSong Exploder Podcast\n\nThe Filter Bubble by Eli Pariser\n\nEconTalk Podcast by Russ Roberts\n\nCommon Voice Project by Mozilla\n\nGoogle.com\n\n** **\n**** Speech Recognition\n** **\n\n•Towards Improving the Performance of Pre-trained Speech models for Low-Resource Languages through Lateral Inhibition\n\nAuthors: Andrei-Marius Avrum,\nRãzwan-Alexandru Smãdu, Vasile Pâis, Dumitru-Clementin Cercel,Radu Ion,Dan Tufis\n\n•Empirical Interpretation of the Relationship Between Speech Acoustic Context and Emotion Recognition\n\nAuthors: Anna Ollerenshaw, Md Asif Jalal\n\n•Cascaded Encoders for Fine-tuning ASR Models on Overlapped Speech\n\nAuthors: Richard Rose, Oscar Chang, Olivier Siohan\n\n•Automatic Speech Recognition of Non-Native Child Speech for Language Learning Applications\n\nAuthors: Simone Wills,Yu Bai, Cristain Tejedor-Garcia, Catia Cucchiarini, Helmer Strik\n\n•NoRefER: A Referenceless Quality Metric for Authomatic Speech Recognition via Semi-Supervised Language Model Fine-tuning With Contrastive Learning\n\nAuthors: Kamar Ali Yuksel, Thiago Ferreira, Golara Javadi, Mohamed El-Badrashiny,Ahmet Gunduz\n\n•Enhancing Prediction of Emotional Experience in Movies Using Deep Neural Networks: The Significance of Audio And Language\n\nAuthors: Sogand Mehrpour Mohammadi, Meysam Gouran Orimi, Hamidreza Rabiee\n\n•Unlocking Foundation Models for Privacy - Enhancing Speech Understanding: An Early Study In Low Resource Speech Training Leveraging Label-Guided Synthetic Speech Content\n\nAuthors: Tiantian Feng, Digbalay Bose, Xuan Shi, Shrikanth Narayanan\n\n•Intelligible Lip-to-Speech Synthesis With Speech Units\n\nAuthors: Jeongsoo Choi, Minsu Kim, Yong Manro\n\n•Dysarthric Speech Recognition using Adversarial And Signal Based Augmentation\n\nAuthors: Massa Baali, Ibrahim Almakky, Shady Shehata, Fakhri Karray\n\n•Improving Fairness and Robustness in End-to-End Speech Recognition through Unsupervised Clustering\n\nAuthors: Irina-Elena Veliche, Pascale Fung\n\n** **\n**** Music Production/Composition\n** **\n\n•Simple and Controllable Music Generation\n\nAuthors: Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexander Defossez\n\n•Controllable Lyrics-to-Melody Generation\n\nAuthors: Zhe Zhang, Yi Yu, Atsuhiro Takasu\n\n•Everybody Compose: Deep Beats To Music\n\nAuthors: Conghao Shen, Violet Z. Yao, Yixin Liu\n\n•QnA: Query-based Representation Learning for Multi-Track Symbolic Music Re-arrangement\n\nAuthors: Jingwei Zhao, Gus Xia, Ye Wang\n\n•Transfer Learning for Under-represented Music Generation\n\nAuthors: Anahita Doosi, Matthew Guzdial\n\n•MusicLM: Generating Music from Text\n\nAuthors: Andrea Agostinelli, Timo I. Denk, Jesse Engel, Mario Verzetti,Antoine Caillon, Quingquing Huang, Aren Jansen, Zalan Borsos,Adam Roberts,Marco Tagliasacchi,Matt Sharifi,Neil Zeghidour,Christian Frank\n\n** **\n**** Realistic Speech Generation\n** **\n\n\n•Ada-TTA:Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis\n\nAuthors: Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Chen Zhang,Xiang Yin,Zejun Ma,Zhou Zhao\n\n•LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading\n\nAuthors:Yochai Yemini,Aviv Shamsian,Lior Bracha,Sharon Gannot,Ethan Fetaya\n\n** **\n**** Audio Compression\n** **\n\n•Siamese SIREN:Audio Compression with Implicit Neural Representations\n\nAuthors:Luca A. Lazendöfer, Roger Wattenhofer\n\n•High-Fidelity Audio Compression with Improved RVQGAN\n\nAuthors: Rithesh Kumar, Prem Setharaman, Alejandro Luebs, Ishaan Kumar, Kundan Kumar\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}}]}