{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nAs neurologist Oliver Sacks once remarked, music appears to be “an essential part of the human condition” [1]. It then comes as no surprise that music — much like text or image — has not been overlooked as generative artificial intelligence (AI) advances. Music is commonly defined as a sequence of pitches or rhythms, or both, typically in a definite pattern [2]. But generating music is a challenging task [3]. First, it necessitates managing different attributes such as long-range structure, overlapping sounds, and emotional expression. Second, it leaves little room for mistakes given that the human ear readily catches harmonic errors [4, 5]. Third, in order to benefit creators, music generation models should allow precise control over details such as key, genre, and more. \n\nThe latest advancements in self-supervised audio representation learning and generative modeling provide a solid foundation to build models that rise to this challenge and generate compelling music. This essay considers the research associated with text-conditioned non-symbolic music generation (e.g., 70s jazz with piano motifs). Our primary focus is on research conducted from 2021 up to the current date. However, it is noteworthy that — to the best of our knowledge — all of the most relevant advancements were concentrated in the year 2023.\n","metadata":{}},{"cell_type":"markdown","source":"# Essay Structure\n\nFollowing the *Introduction*, the essay explores *The State of Text-Conditioned Music Generation in 2023*. It then transitions into a *Discussion* that examines relevant evaluation metrics and contemplates the broader societal and industrial impacts. The essay wraps up with a *Conclusion*, offering a well-rounded view of the subject.","metadata":{}},{"cell_type":"markdown","source":"# The State of Text-Conditioned Music Generation in 2023\n\nMusic has existed from the earliest periods of human civilization and has piqued the interest of computer scientists for nearly a century. Starting from the first substantial computer-generated music using Markov chains in 1956 [6], the field progressed to adopt more data-hungry approaches like MusicVAE in 2018 [7]. Still, a single system that captures various properties of music remained absent until Jukebox in 2020 [8]. Jukebox generated songs from rather diverse genres of music like jazz, rock, and hip-hop. It captured melody, rhythm, long-range structure, and timbres for a variety of instruments and allowed for the generation of lyrics/singing as well. \n\nBuilding on the achievement of Jukebox, the field of generative modeling has increasingly adopted a two-stage learning paradigm [9]. The paradigm begins with representation learning in the initial stage, where a model is trained to compress the representation of input data. The following stage involves constructing a generative model on top of the compressed, low-dimensional input space [10, 11, 12]. This is believed to be more efficient than learning on the raw input data, which can be tangled and computationally expensive representation to work on [11]. \n\nDrawing upon the two-stage learning paradigm, several valuable advancements have surfaced, all concentrated in the year 2023. The following discussion will navigate through five research efforts that employ this paradigm in their music generation models, opening with MusicLM published on the 26th of January 2023 and concluding with MusicGEN shared on the 8th of June 2023.\n\n## MusicLM\n\nMusicLM [13], a Transformer-based model, formulates a music generation task as hierarchical sequence-to-sequence modeling, as one would do with a language modeling task [14]. This approach can generate text-conditioned high-fidelity music that remains consistent over several minutes.\n\nMusicLM consists of three representation learning models and two decoder-only Transformers for hierarchical sequence-to-sequence modeling. These are w2v-BERT tokens for temporal coherence [15], MuLan tokens for text conditioning [16], and SoundStream tokens for high-fidelity music generation [17]. At the first stage — also described as semantic modeling — a decoder-only Transformer learns to map MuLan tokens to semantic tokens. Later, in the second stage — referred to as acoustic modeling — another decoder-only Transformer learns to predict SoundStream tokens from given MuLan and semantic tokens.\n \nMost notably, MuLan, a contrastive model between music and text modalities, can interpret music and text signals as one. For example, the model would return a similar output if the input is a \"90s guitar riff\" in either text or music form. Authors take advantage of MuLan's ability to interpret multiple modalities as one and scale training datasets without strictly requiring text and music pairs.\n\nModels in the MusicLM pipeline were trained with a massive 280 thousand hours of music data.\n\n## Moûsai\n\nMoûsai [9], a latent diffusion approach, generates several minutes of coherent, high-quality music. Unlike earlier works in the music generation literature, Moûsai stands out for its ability to synthesize high-fidelity music in almost real-time using a single consumer-grade hardware. \n\nMoûsai initially employs a pretrained large language model that encodes the meaning of text descriptions. Next, the first stage diffusion model generates a compressed representation of the audio while being conditioned on the encoded text. Finally, the second stage diffusion decoder utilizes the reduced representation from the first stage and creates the resulting music in accordance with the text description.\n\nThe training dataset of Moûsai is considerably smaller when compared to other works in the literature. The dataset includes 2.5 thousand hours of music data encompassing diverse genres, artists, and instruments. \n\n## Noise2Music\n\nNoise2Music [18], a diffusion-based music generation approach, creates thirty seconds of high-quality music segments. Uniquely, the authors experiment with two different input audio representations: (1) low-fidelity raw waveform and (2) log-mel spectrogram. Authors report that while raw waveform delivers better objective and subjective evaluation scores, the log-mel spectrogram approach achieves acceptable quality with fewer costs, bringing more scalability options.\n\nNoise2Music presents two pipelines for proposed input representations: waveform and spectrogram models. The shared first step is encoding the meaning of text description using a large language model. Then, the appropriate diffusion model compresses the input representation while also conditioning with the encoded text. Finally, a cascaded diffusion model for raw waveform or a vocoder model for log-mel spectrogram representation consumes the outputs from the preceding stage and generates the waveform music.\n\nNoise2Music used a large-scale dataset of 340 thousand hours of paired music and text. The massive dataset was built using a clever data mining technique. First, for a single music clip, a large language model generates multiple rich music descriptions in text form using music tags. Then, a contrastive music and audio model (MuLan) calculates the similarity score between the music clip and generated text descriptions and chooses the text description with the highest similarity score as the text pair for the music clip.\n\n## MeLoDy\n\nMeLoDy [19], a diffusion model for text-to-music generation, offers competitive results while keeping generation time faster than in real-time, unlike its predecessors. \n\nMeLoDy consists of five different models working with each other. First, representation learning models, MuLan [16], Wav2Vec2-Conformer [20, 21], and audio VAE [19] capture different properties of music. Later, two generative models, are concerned with either semantic (language model) or acoustic modeling (a novel dual-path diffusion model). Semantic modeling refers to the meaning and structure of the music while acoustic modeling refers to the perceived audio quality.\n\nMeLoDy was trained on a tremendous in-house music dataset of 257 thousand hours. The dataset was filtered to keep only non-vocal music data since the authors didn't intend to generate music with vocals. Finally, to counter the scarce availability of music description datasets, the authors utilized ChatGPT to create music descriptions from short tags.\n\n## MusicGEN\n\nMusicGEN [3], a music language model working with text and discretized music representations, delivers a simple and efficient method to generate high-quality music from text descriptions.\n\nThe MusicGEN pipeline first compresses raw waveform music into a series of integers — described as discrete music representations — using a variant of a work called EnCodec [22]. Later, a music language model jointly learns discrete music representations and their accompanying text condition. \n\nThe authors train three different scales of MusicGEN: small, medium, and large. Authors report that scores improve as the model size increases. However, scaling the size of the model comes with the expense of slow generation time and the requirement for adequate hardware. \n\nThe team used a modest dataset size of 20 thousand hours of high-quality, licensed music data. The dataset contains textual descriptions of music tracks and tags such as artist, genre, and others.\n","metadata":{}},{"cell_type":"markdown","source":"# Discussion\n\n## Evaluation\n\nEvaluation is a crucial component of large-scale generative model development. In essence, we evaluate to gain an insight into the quality of the outputs. A good evaluation considers multiple properties and helps researchers understand what improves and what impairs the outcome.\n\nWithin the Kaggle community, evaluation is highly valued. In a standard competition, the host defines an evaluation metric to measure the success of every competitor's solution. Naturally, the competitors make decisions based on the quantitative metric and iterate accordingly to create the final solution. \n\nFor example, we are developing a model that outputs yes if there's a dog in an image and outputs no otherwise. The number of correctly classified images would determine the accuracy metric. Specifically, if the model can correctly guess eight out of ten inputted images, it would achieve 80% accuracy. Here, accuracy is an objective evaluation metric; the metric that has quantifiable evidence. On the other hand, subjective evaluation helps estimate performance based on unquantifiable evidence. For example, preferences for generated music samples.\n\nImportantly, there is no perfect evaluation metric for music in the existing literature. This is largely because music is layered with a number of properties (e.g., timbre, rhythm, and melody) and is further complicated by the subjective nature of music perception [9].\n\n### Objective Evaluation\n\nFréchet Audio Distance (FAD) — originally meant to measure the output quality of music enhancement models — is widely used in recent text-to-music research [23]. Notably, FAD correlates better with human perception than some other objective evaluation metrics for audio quality. Figure 1 below shows the comparison of FAD scores between recent text-to-music models. A lower FAD score indicates better audio quality.\n\nKindly note; as not all models are made publicly available, our figures are constructed using the data shared by the authors. Consequently, not all models are represented in each figure.\n\n\n
\n \n
\n Figure 1: FAD Score Comparison\n
\n
\n\nMuLan is a contrastive model between the music and text modalities. Therefore, it can calculate a similarity score between music and text. The Mulan Similarity score helps to quantify the alignment between text condition and generated music. Figure 2 displays the MuLan Similarity score comparison between the models reviewed as well as Riffusion — a fine-tuned diffusion-based model for music generation — and Mubert, a commercially available product for music generation [24, 25]. Higher MuLan Similarity indicates better alignment between the conditioning text and generated music.\n\n
\n \n
\n Figure 2: MuLan Similarity Score Comparison\n
\n
\n\nBesides the most popular FAD and MuLan Similarity, other objective metrics exist in recent literature [13]. The reason for using multiple objective metrics is to evaluate various properties of generated music (e.g., vocal quality and acoustic characteristics).\n","metadata":{}},{"cell_type":"markdown","source":"### Subjective Evaluation\n\nSubjective evaluation metrics can vary from one research to another [13, 18]. During a typical human listening test, generated samples and the actual ground truth music are played to listeners. The listeners are then asked to rate their preference (strong, weak, or neutral) on a 5-point Likert scale. Finally, the total number of “wins” each model achieved is counted to determine the evaluation outcome. Figure 3 below shows the number of wins for different text-to-music models. A higher number indicates better subjective evaluation outcome.\n\n
\n \n
\n Figure 3: Number of Wins\n
","metadata":{}},{"cell_type":"markdown","source":"## Broader Impact\n\n### Risks and Responsible Model Development Practices\n\nLarge-scale generative models come with their share of ethical hurdles. Such models learn to mimic patterns, and in the process, they can adopt the biases present in the training data. These biases can be hard to detect and, in the context of music generation, can give rise to offensive language, either through learned associations or by chance. As a consequence, the models might produce mock accents or other linguistic harms in vocals [18].\n\n\nAlso, the data that the generative models are trained on is reflective of the unequal recording and digitization of samples from global musical cultures. In other words, the limited corpus of samples contributes to an unbalanced training corpus [18, 19]. Even more so, the categorization and labeling of music can oversimplify genres, particularly when the categories are assigned and labels are created with no input from the communities they represent [18]. This brings up questions about the appropriateness of AI-generated music for underrepresented cultures as well as concerns about cultural appropriation [13]. It is important to remember that appreciating the fine details in musical traditions needs a trained ear and cultural insight. What sounds “authentic” to an outsider might not resonate the same within the community itself [18].\n\nTo enhance generative models, metrics such as originality and generation diversity should be considered. That is, AI for music generation should mimic the intended style but should not copy them exactly [26]. In accordance with responsible model development practices, the experts behind MusicLM conducted a substantial study of memorization, finding that few examples are exact while 1 in 100 is an approximate match [13]. The Noise2Music team built duplication checks in the current pipeline of producing and releasing samples, while MusicGEN authors hope that the model simplification can help broaden the applications to more diverse datasets [18, 3]. \n\n### Creative Community and Realistic Usage Efforts\n\nAll examined research shares a common goal — developing tools that assist in creative music endeavors. Some underscore the value of collaboration with musicians and stakeholders to foster meaningful co-creation [18], while others strive to make creative expression accessible to everyone with fast models that can operate on a single consumer GPU [9]. More intricate controls, such as MusicGEN’s melody conditioning or MeLoDy’s interactive creation, can strengthen the potential of these co-creation tools further [3, 19]. Still, it is worth noting that generative models can provide unfair competition for artists. Therefore, promoting legally protected data and open research will contribute to a more fair and equal process [3]. \n\nThe potential of text-conditioned music generation is evident in the reviewed work. Still, the path to improvement is multifold: enhancing model interpretability, improving text-audio alignment, and reducing training and inference costs, to name a few. One obvious potential for quality improvement lies in scaling up data and model size [9]. Another less obvious opportunity is in expanding these models to complete or modify music samples, as was previously achieved with images [27]. To fuel future research, authors share datasets like MuLaMCap and MusicCaps and invite more work in this niche of generative artificial intelligence [18, 13].\n\n","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\n\nThe year 2023 marked a significant leap in text-conditioned, non-symbolic music generation. Advanced models such as MusicLM, Moûsai, Noise2Music, MeLoDy, and MusicGEN have come to demonstrate the potential of AI in generating high-fidelity music. Despite their individual strengths and different methodologies, all models share the common goal of advancing the field and pushing the boundaries of music generation using AI. These advances spotlight the critical role of evaluation within generative model development. Crafting an ideal music evaluation metric remains a challenge, given the multifaceted nature of music and the inherent subjectivity of its perception. Despite this, objective tools like FAD and MuLan Similarity — alongside subjective human listening tests — offer valuable performance metrics. \n\nIn the compelling world of AI music generation, we must not overlook the ethical implications surrounding large-scale generative models. Elusive biases and unbalanced representation in training data can raise concerns such as offensive content generation and cultural misrepresentation. To address these concerns, researchers implement responsible development practices and advocate for originality and diversity in AI-generated music. The models reviewed in this essay are developed not only with technical proficiency but also with an understanding that these tools are meant to assist humans in creative endeavors. As we look to the future, it is important to remember that music is not merely an integral part of the human condition — echoing the words of Oliver Sacks from the beginning of the essay — but also that the creation of music is deeply rooted in the human condition.\n\nThe symphony between human- and AI-created music continues. Its rhythm is ours to define.","metadata":{}},{"cell_type":"markdown","source":"# Works cited\n\n[1] Why the Brain Loves Music, Dr. Oliver Sacks, Columbia University, (Oct. 01, 2018). Accessed: Jul. 02, 2023. [Online Video]. Available: https://www.youtube.com/watch?v=KWTRQhF70k0\n\n[2] https://www.copyright.gov/prereg/music.html (accessed Jun. 26, 2023).\n\n[3] J. Copet et al., ‘Simple and Controllable Music Generation’. arXiv, Jun. 08, 2023. doi: 10.48550/arXiv.2306.05284.\n\n[4] S. V. Norman-Haignere, N. Kanwisher, J. H. McDermott, and B. R. Conway, ‘Divergence in the functional organization of human and macaque auditory cortex revealed by fMRI responses to harmonic tones’, Nat Neurosci, vol. 22, no. 7, pp. 1057–1060, Jul. 2019, doi: 10.1038/s41593-019-0410-7.\n\n[5] E. Fedorenko, J. H. McDermott, S. Norman-Haignere, and N. Kanwisher, ‘Sensitivity to musical structure in the human brain’, J Neurophysiol, vol. 108, no. 12, pp. 3289–3300, Dec. 2012, doi: 10.1152/jn.00209.2012.\n\n[6] L. A. Hiller and L. M. Isaacson, Experimental Music: Composition with an Electronic Computer. New York: McGraw-Hill, 1959.\n\n[7] A. Roberts, J. Engel, C. Raffel, C. Hawthorne, and D. Eck, ‘A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music’. arXiv, Nov. 11, 2019. doi: 10.48550/arXiv.1803.05428.\n\n[8] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, ‘Jukebox: A Generative Model for Music’. arXiv, Apr. 30, 2020. doi: 10.48550/arXiv.2005.00341.\n\n[9] F. Schneider, Z. Jin, and B. Schölkopf, ‘Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion’. arXiv, Jan. 30, 2023. doi: 10.48550/arXiv.2301.11757.\n\n[10] D. Yang et al., ‘Diffsound: Discrete Diffusion Model for Text-to-sound Generation’. arXiv, Apr. 28, 2023. doi: 10.48550/arXiv.2207.09983.\n\n[11] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, ‘High-Resolution Image Synthesis with Latent Diffusion Models’. arXiv, Apr. 13, 2022. doi: 10.48550/arXiv.2112.10752.\n\n[12] A. Ramesh et al., ‘Zero-Shot Text-to-Image Generation’. arXiv, Feb. 26, 2021. doi: 10.48550/arXiv.2102.12092.\n\n[13] A. Agostinelli et al., ‘MusicLM: Generating Music From Text’. arXiv, Jan. 26, 2023. doi: 10.48550/arXiv.2301.11325.\n\n[14] Z. Borsos et al., ‘AudioLM: a Language Modeling Approach to Audio Generation’. arXiv, Sep. 07, 2022. doi: 10.48550/arXiv.2209.03143.\n\n[15] Y.-A. Chung et al., ‘W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training’. arXiv, Sep. 13, 2021. doi: 10.48550/arXiv.2108.06209.\n\n[16] Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y. Li, and D. P. W. Ellis, ‘MuLan: A Joint Embedding of Music Audio and Natural Language’. arXiv, Aug. 25, 2022. doi: 10.48550/arXiv.2208.12415. \n\n[17] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, ‘SoundStream: An End-to-End Neural Audio Codec’. arXiv, Jul. 07, 2021. doi: 10.48550/arXiv.2107.03312.\n\n[18] Q. Huang et al., ‘Noise2Music: Text-conditioned Music Generation with Diffusion Models’. arXiv, Mar. 06, 2023. doi: 10.48550/arXiv.2302.03917.\n\n[19] M. W. Y. Lam et al., ‘Efficient Neural Music Generation’. arXiv, May 25, 2023. doi: 10.48550/arXiv.2305.15719.\n\n[20] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020.\n\n[21] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ‘wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations’, in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020, pp. 12449–12460. Accessed: Jul. 04, 2023. [Online]. Available: https://proceedings.neurips.cc/paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html\n\n[22] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, ‘High Fidelity Neural Audio Compression’. arXiv, Oct. 24, 2022. doi: 10.48550/arXiv.2210.13438.\n\n[23] K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, ‘Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms’. arXiv, Jan. 17, 2019. doi: 10.48550/arXiv.1812.08466.\n\n[24] ‘Riffusion’. https://www.riffusion.com/about (accessed Jul. 02, 2023).\n\n[25] ‘Mubert-Text-to-Music’. https://github.com/MubertAI/Mubert-Text-to-Music (accessed Jul. 02, 2023).\n\n[26] Z. Yin, F. Reuben, S. Stepney, and T. Collins, ‘Measuring When a Music Generation Algorithm Copies Too Much: The Originality Report, Cardinality Score, and Symbolic Fingerprinting by Geometric Hashing’, SN COMPUT. SCI., vol. 3, no. 5, p. 340, Jun. 2022, doi: 10.1007/s42979-022-01220-y.\n\n[27] C. Saharia et al., ‘Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding’. arXiv, May 23, 2022. doi: 10.48550/arXiv.2205.11487.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nsubmission=pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsubmission.head()\n\nsubmission.loc[0]['value']='Generative AI'\nsubmission.loc[1]['value']='https://www.kaggle.com/code/realsleim/the-state-of-text-conditioned-music-generation/'\nsubmission.loc[2]['value']='https://www.kaggle.com/code/amosii/artificial-intelligence-a-digital-renaissance'\nsubmission.loc[3]['value']='https://www.kaggle.com/code/mahadevan39/the-generative-leap/notebook'\nsubmission.loc[4]['value']='https://www.kaggle.com/code/pandoraalias/unleashing-creativity-on-gen-ai'\nsubmission.head()\n\nsubmission.to_csv('submission.csv',index=False)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-07-04T22:35:22.106746Z","iopub.execute_input":"2023-07-04T22:35:22.107266Z","iopub.status.idle":"2023-07-04T22:35:22.150368Z","shell.execute_reply.started":"2023-07-04T22:35:22.107227Z","shell.execute_reply":"2023-07-04T22:35:22.149443Z"},"trusted":true},"execution_count":null,"outputs":[]}]}