{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n\n\n\n

Automated Programming with Generative AI: Promises, pitfalls and the road ahead

\n

How has generative AI impacted the future of programming?

\n \n\n","metadata":{}},{"cell_type":"markdown","source":"# Section 1: Introduction\n\nThe dream of automated programming has captured the imagination of computer scientists for ages [1]. Recent advances in generative AI and code-generating large language models (LLMs) such as GPT-4 [2], Codex [3], AlphaCode [4] have finally made it possible to achieve this long-standing goal. Generative AI assistants powered by these LLMs such as Copilot (the AI pair programmer) [5], Code Whisperer [6], Ghostwriter [7] are being used on a daily basis by programmers for code completion. Further, programmers are increasingly relying on these AI assistants for code refactoring, code comprehension, debugging and other auxiliary programming tasks. The rise in popularity of generative AI assistants can be attributed to their interaction model and their stellar performance: they need to be prompted only by natural language description of the task and are able to output functions with hundreds of lines of code within seconds. This makes them easy to integrate in the workflow of programmers with a substantial increase in productivity [8]. \n\nThis essay tells the story of how we got to this stage of revolutionary technological advancement: Section 2 provides a timeline of the most noteworthy breakthroughs in generative AI followed by an overview of the underlying technological advancements in Section 3. With the increased adoption of generative AI assistants, several research areas are being rapidly explored: *How to prompt the AI assistant to communicate the desired intent of the user for a myriad of tasks?* (Section 4). *How to mitigate security and correctness challenges of using code generated by AI assistants?* (Section 5). \n This essay discusses the state of the art work done in these areas and how advances in generative AI have impacted the future of programming. Read on to know more!\n\n \"main\"\n
Created using Microsoft Bing Creater for the prompt: An AI assistant helping a programmer with a difficult programming task
","metadata":{}},{"cell_type":"markdown","source":"# Section 2: Timeline of revolutionary breakthroughs in Generative AI:\n\nWe present a chronological timeline of the most important innovations powered by Generative AI in the domains of text, audio, video, images, code/programming and design. \n\n- **Google Translate in 2016 (Text)** - Google Translate currently supports over 103 languages and hundreds of millions of users, helping people make connections everyday [9]. Neural machine translation models were integrated into Google Translate which significantly improved the quality of it's translations.\n\n- **DeepArt and Prisma in 2016 (Image)** - These applications convert photos to an artistic version by applying modifications to the images using a style transfer network, showing the creative potential of generative AI in the domain of image and art generation [10, 11].\n\n- **AlphaGo by DeepMind in 2016 (Game-Play)** - This was the first AI system to defeat a world champion at the game of Go. It utilized a combination of Monte Carlo tree search and deep neural networks, including both supervised learning and reinforcement learning techniques [12].\n\n- **WaveNet by OpenAI in 2016 (Audio)** - Wavenet sent ripples of excitment in the music and audio generation domain in 2016 when it was able to generate speech that mimics human voice and compose strikingly beautiful piano pieces [13].\n\n- **ThisPersonDoesNotExist in 2018 (Image)** - This is a generator of realistic images of people who do not actually exist, illustrating the powerful potential of Generative Adversarial Networks (GANs). This application really showed how effective GANs are at mimicing the distribution of real-world data [14].\n\n- **MuseNet by OpenAI in 2019 (Audio)** - MuseNet performs music generation and is based on the transformer model so is very effective; it can generate four-minute musical compositions with upto ten different musical instruments and can blend different musical genres [15].\n\n- **OpenAI's GPT-2 and GPT-3 in 2020 (Text/Code)** - These large language models surpassed all previous models with their unprecedented ability to generate coherent text for a wide variety of textual and also programming tasks. GPT-3 can write poetry, have philosophical discussions and chat with humans in a convincing manner. This was a major breakthrough that pushed the boundaries of generative AI in natural language processing.\n\n- **OpenAI's Codex and Github Copilot in 2021 (Code)** - Copilot powered by OpenAI's Codex model was trained on a large amount of code from public repositories on Github. It can generate programs based on natural language text and surrounding code context and is integrated into popular IDEs like Visual Studio Code, Intellij and NeoVim. \n\n- **Dall-E and Dall-E 2 in 2021 (Image)** - These AI systems generate really complex, realistic images and art from a natural language description! Check out this image created using Dall-E 2 [16].\n\n\"self-consistent\" \n\n- **Stable Diffusion in 2022 (Image)** - This generative model uses the concept of diffusion to generate high-quality realistic images [17]. Diffusion essentially uses noise to train the model by adding noise to the original data then reversing the process. These diffusion models are known to perform better than state-of-the-art GANs.\n\n- **ChatGPT (2022) and GPT-4 in 2023 (Text/Code)** - GPT-4 is the most powerful large language model to date and it surpasses all previous models in terms of it's ability to perform complex reasoning tasks [18]. GPT-4 has also outperformed ChatGPT by scoring high marks in difficult competitive examinations.","metadata":{}},{"cell_type":"markdown","source":"# Section 3: Technological advancements in Generative AI\n\nThere were several ground-breaking discoveries that eventually led to the creation of AI assistants like Github Copilot and GPT-4, in this section we will look at the AI models that deserve special mention.\n\n### **Recurrent Neural Networks**\n\nAn RNN model takes in a sequence (of tokens) and a previous hidden state (i.e. context) as input, and produces an updated hidden state which is propagated ahead in the learning stage. Neural sequence-based code generating models proposed in [19] employed RNNs to capture rich dependencies by representing the code context as a distributed vector, this was an improvement over using fixed-length context like in n-gram models [20].\n\n### **Long Short-term Memory RNN**\n\nLong short-term memory (LSTM) were an improved version of the RNN architecture that captured long-range dependencies more precisely than RNNs. LSTM uses gated hidden cells to offer a finer control over which part of the context needs to be carried forward and how much of it needs to be forgotten [21]. LSTMs comprise of three logistic sigmoid gates that are learnable and control the information passed to the next hidden layer: input gate, forget gate and output gate. The forget gate decides how much information is retained, the input gate is used to control which new information is stored in the memory and the output gate produces the output based on the current code token and the previous hidden state.\n\n### **Variational Autoencoders (VAEs)**\n\nFast-forward a few years, Variational Autoencoders (VAEs) were proposed with an encoder-decoder architecture and are used to produce new data that closely resembles the training data [22, 23]. They do this by generating a distribution over the latent space from which data points can be sampled and reconstructed. The encoder takes in the input data and outputs parameters of a proposed latent space distribution. A point from this distribution is then sampled and passed through the decoder to generate a reconstruction of the original input. The VAE is trained by minimizing the difference between the original input and its reconstruction (reconstruction loss).\n\n\"GAN\"\n
Face images generated with a variational autoencoder (source: Github)
\n\n### **Generative Adversarial Networks (GANs)**\n\nGenerative Adversarial Networks (GANs) were first introduced in 2014 and are composed primarily of two components: a generator and a discriminator, which are trained simultaneously and compete against each other, hence the term \"adversarial\". The generator takes random noise as input and attempts to generate data that is intended to come from the same distribution as the training data [24]. The generator's goal is to fool the discriminator into thinking that the data it generated is real, whereas the discriminator aims to classify data as real (training data) or fake (by the generator). \n\n \"GAN\"\n\nThe training process involves both the generator and discriminator being trained simultaneously in a two-player game, where the discriminator is trying to maximize its ability to classify real and fake data correctly, and the generator is trying to minimize the probability that the discriminator correctly classifies its output as fake. Eventually the generator is able to produce increasingly realistic data as training progresses. GANs have shown remarkable results especially for realistic image and art generation. \n\n### **Transformers (BERT, GPT, T5)**\n\nThe most revolutionary technological advancement that eventually led to the creation of powerful large language models like BERT (Bidirectional Encoder Representations) [25], T5 (Text-to-Text Transformer) [26] and GPT-3 was the transformer architecture proposed in 2017. \n\nThe key insight of the transformer architecture is the concept of *attention* which allows the model to focus on different parts of the input when generating each element of the output sequence. \nThe model used a stack of encoders and decoders, where the encoder receives a list of input embeddings and the decoder outpputs the translated embeddings. The encoder's inputs go through a *self-attention layer* that helps the encoder attend to other words in the input sequence as it encodes a specific word. The decoder has an attention layer in addition to a self-attention layer and feed-forward network that helps it focus on relevant parts of the input sentence. \n\n \"Transformer\"\n\nTransformers have significantly improved on a wide range of natural language processing and code-related tasks. Transformer models such as BERT, GPT, and T5 achieve state-of-the-art results on a wide range of tasks like language/code translation, question answering, text summarization and comprehension. In addition to NLP tasks, GPT-4 shines on numerous code related tasks: code translation [27], code comprehension [28] and even repairing it's own code [29].","metadata":{}},{"cell_type":"markdown","source":"# Section 4: Prompting generative AI models - make your wish their command\n\nThe field of machine learning has slowly been moving away from fine-tuning models for downstream tasks, instead prompt engineering and tuning has emerged as a promimising new direction. Prompting techniques employ a form of *in-context learning* to train the model and often include adding relevant contextual information like examples or a table schema, generating mutations of the prompt, adding explanations and instructions and summarizing complicated prompts for programming contest-style questions.\n\n### **Augmenting the prompt with relevant context** \nLarge language models like CodeT5 and Codex [30] have been trained on almost all publicly available code from Stackoverflow and Github. However, library APIs keep growing and updating continuously, and it is impossible to retrain these large language models periodically because of the substantial amount of compute required. This makes it difficult for these models to generalize to unseen libraries and functions that did not originally appear in their training data. In order to circumvent this problem, approaches like DocPrompting [31] have been proposed which retrieve the relevant code documentation given natural language intent and then generate code using both the original query and retrieved documentation. \n\n \"DocPrompting\"\n\nAs shown in the above figure, the retriever retrieves the top-k relevant documentation strings from a pool based on a textual similarity score between the original natural language query and the candidate documentation strings. The generator (which takes the form a language model) then generates code conditioned on both the retrieved documentation and the query. They achieve stellar performance on shell scripting and Python code generation. \n\nInstead of using documentation to provide additional context like in DocPrompting, Cedar [32] augments the model’s prompt with demonstrations of similar code usage examples from a demonstration pool. Their demonstration retriever uses a sparse retrieval model to rank and retrieve top-k code demonstrations based on relevance to the query similar to DocPrompting. The prompt then consists of the original query combined with code demonstrations, which considerably improves performance.\n\"Cedar\"\n\n### **Step-by-step reasoning using chain-of-thought prompting**\nOne of the most effective techniques in prompting large language models is chain-of-thought prompting that relies on the idea of generating a series of intermediate reasoning steps to significantly improve the ability of language models to perform complex reasoning tasks [33]. The key insight of this work is that by providing step-by-step reasoning in the few-shot examples, we are essentially nudging the model to learn to think in this step-by-step manner and reason better. Chain-of-thought prompting has shown to significantly improve performance of language models like Palm-540B on a range of arithmetic, commonsense and symbolic reasoning tasks and has been one of the most fundamental advancements in prompting language models.\n\n\"cot\"\n\nRecently, several works have emerged inspired by chain-of-thought prompting that attempt to adapt it to the domain of programming tasks. The challenge observed by the community was that even though LLMs are capable of successfully reasoning about the problem through intermediate steps, they often make logical and arithmetic errors even with the correct problem reasoning since they have not been trained specifically to solve and execute programs. To mitigate this, Program-aided language models (or PAL) first use the LLM to generate the intermediate reasoning but offloads the logical computation to a Python interpreter [34]. This work uses the best of both worlds by asking the LLMs to do what they are good at: breaking down the problem into runnable steps and then uses an interpreter to solve and compute the results of these intermediate steps.\n\n\"cot\"\n\nAnother similar work proposes ‘Program of thought’ prompting which also uses the LLM to generate the reasoning steps and a solver to solve them [35]. Both these techniques achieve state-of the art performance on several math, financial and numerical reasoning benchmarks. A popular variant of chain-of-thought style prompting techniques is (instead of using an external interpreter), you can train models to use a *scratchpad* to generate program execution traces line-by-line [36]. In this method, instead of attempting to directly produce the output, the model keeps track of it's intermediate results in the prompt itself which provides the relevant context for the next computation step. This means the model is less likely to make mistakes since it has a \"memory\" of the current computation; using a scratchpad is shown to improve performance drastically for multi-step problems.\n\n\"scratchpad\"\n\n### **Divide and Conquer Prompting**\n\nAlthough chain of thought performs exceptionally well, it's performance starts to degrade when there are a large number of reasoning steps required for a particular task. To tackle this issue, several divide and conquer based prompting approaches have been proposed that first decompose the problem into multiple steps and then each of the sub-problems are solved sequentially. *Least-to-most prompting* is an instantiation of this framework where they solve these subproblems in a specific order, where the prompt for each subproblem is appended with the answers obtained from previously solved subproblems [37]. A follow-up work to least-to-most prompting is the *decomposed prompting* technique where they also break down a complex problem into subproblems which are then handled by a library of sub-task specific LLMs each having their own few-shot prompts [38].\n\n\"least-to-most\"\n\n\n### **Refining the output of large language model using feedback**\nJust like humans, it is likely not possible for the language models to get the correct output for a programming task in the first attempt itself. Thus, an interactive scenario where the model and the human (or any external oracle) engage in a conversation and the human iteratively provides feedback and hints to the model is more likely to eventually generate the correct response. In the figure below, the model almost gets the correct Python program in step (d) but needs a small hint from the user to fix it’s error [39].\n\n\"feedback\"\n\nAnother state of the art technique which has been recently proposed is self-refinement: instead of using an external oracle (human) to provide feedback on the model’s output, can we use the model itself to get feedback on it’s own output and refine the previous outputs in the subsequent steps? [40, 41]\n\n\"Self-refine!\"\n\nIn the example below, we can see that the model generated an initial output (in blue) which was then evaluated by the model itself to generate the feedback (in red). It then used it’s own feedback to correct it’s previous output. What is amazing about this approach is that there is no supervised training data required: the model itself is the generator, feedback provider and refiner! This approach opens new directions for exploring ways in which language models can be used as oracles and evaluators for a variety of domains.\n\n \"self-refine\"\n\nThis self-refinement technique has also been used for self-debugging [42] where they elicit feedback from the language model by asking the model to generate an explanation for it’s own code. This code explanation along with the execution results are sent as prompt to the model to perform debugging.\n\n### **Prompting to generate multiple outputs**\n\nIn addition to decomposing the problem and providing better context, this type of prompting techniques ask the model to fully explore several different reasoning pathways and come up with multiple ways of thinking. [43] presented a novel decoding strategy called *self-consistency* which involves sampling a diverse set of reasoning options (instead of greedy decoding) and then marginalizing out the reasoning paths and aggregate by choosing the most consistent option amongst all available.\n\n\"self-consistent\"\n\nA similar idea is proposed in DIVERSE [44] where they generate different prompts to explore different reasoning paths for the same task; then use a verifier to filter out the incorrect reasoning paths and finally aggregate on one answer based on voting and the scores provided by the verifier. The effectiveness of these techniques illustrates that sampling several options from the LLM can help solve complex tasks since it explicitly requires the LLM to explore unique ways of reasoning.\n \n \"self-consistent\"","metadata":{}},{"cell_type":"markdown","source":"# Section 5: Correctness and security guarantees for generative AI \n\nSince large language models have been trained on code from Github which contains lots of bugs, the generated code is also bound to be buggy. In fact, an evaluation of code generated by Github Copilot revealed that on programming competition problems Copilot was able to generate correct solutions for Java on 57% of the tasks [45] so there is definitely room for improvement. \n\nSome techniques that have been proposed include using the LLM itself to generate test cases and then picking the best code solution by executing the generated code on the generated test cases. They expect that a solution which passes more test cases and has more similar solutions is more correct for the given problem, which seems reasonable [46].\n\n\"codet\"\n\nAnother similar technique has been proposed where they also use an LLM to independently sample candidate programs and candidate specifications based on the natural language description. They then run the programs on the generated input-output testcases and the result of this execution is fed to another trained model that predicts which program is correct and which testcases best certify the program's correctness [47].\n\n\"verify\"\n\nWhile these methods do not completely guarantee correctness of the program, they are a step in the right direction towards ensuring that the generated code is safe. Such guarantees will also help in improving the trust of users in content created by generative AI, especially in safety-critical domains.","metadata":{}},{"cell_type":"markdown","source":"# Section 6: The road ahead with Generative AI\n\n\nGenerative AI represents one of the most promising advancements in artificial intelligence today. Generative models have shown remarkable results on a myriad of difficult tasks from creating realistic images, composing music, generating summaries of text and even passing examinations difficult for humans. The potential applications for generative AI are far-reaching. For instance, it is already being used in content creation for entertainment, personalized advertisements, language translation, automating scientific discovery and so much more.\n\nHowever, while the potential is huge, so are the challenges and ethical implications. Stringent care needs to be taken to avoid spreading of fake information generated by the model, monitor the impact on education and economic aspects affecting people's jobs. As we continue to push the boundaries of what generative AI can do, it is essential that we also put considerable thought and effort into managing these risks. Generative AI is an exciting, rapidly advancing field with immense potential. As the technology continues to develop, it will require ongoing research, thoughtful policy, and broad societal engagement. After all, with great power comes great responsibility.","metadata":{}},{"cell_type":"markdown","source":"\n# Bibliography\n[1] [Synthesis: Dreams -> Programs](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1702636)
\n[2] [OpenAI GPT-4](https://openai.com/gpt-4)
\n[3] [OpenAI Codex](https://openai.com/blog/openai-codex)
\n[4] [AlphaCode DeepMind](https://alphacode.deepmind.com)
\n[5] [Github Copilot](https://github.com/features/copilot)
\n[6] [Amazon CodeWhisperer](https://aws.amazon.com/codewhisperer/)
\n[7] [GhostWriter](https://replit.com/site/ghostwriter)
\n[8] [Productivity assessment of neural code completion](https://dl.acm.org/doi/abs/10.1145/3520312.3534864)
\n[9] [Google Translate](https://blog.google/products/translate/ten-years-of-google-translate/)
\n[10] [DeepArt](https://creativitywith.ai/deepartio/)
\n[11] [Prisma](https://prisma-ai.com)
\n[12] [DeepMind AlphaGo](https://www.deepmind.com/research/highlighted-research/alphago)
\n[13] [OpenAI WaveNet](https://www.deepmind.com/blog/wavenet-a-generative-model-for-raw-audio)
\n[14] [Thispersondoesnotexist by Nvidia](https://thispersondoesnotexist.com)
\n[15] [OpenAI MuseNet](https://openai.com/research/musenet)
\n[16] [OpenAI Dall-E](https://openai.com/dall-e-2)
\n[17] [Stable Diffusion](https://stablediffusionweb.com)
\n[18] [ChatGPT](https://openai.com/blog/chatgpt)
\n[19] [Code completion with statistical language models](https://dl.acm.org/doi/pdf/10.1145/2594291.2594321)
\n[20] [On the naturalness of software](https://dl.acm.org/doi/pdf/10.1145/2902362)
\n[21] [Long Short-Term Memory](https://dl.acm.org/doi/10.1162/neco.1997.9.8.1735)
\n[22] [Auto-Encoding Variational Bayes](https://arxiv.org/pdf/1312.6114.pdf)
\n[23] [Stochastic Backpropagation and Approximate Inference in Deep Generative Models](https://arxiv.org/pdf/1401.4082.pdf)
\n[24] [Generative Adversarial Nets](https://arxiv.org/pdf/1406.2661.pdf)
\n[25] [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/pdf/1810.04805.pdf)
\n[26] [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://arxiv.org/abs/2109.00859)
\n[27] [AI Code Translator](https://vercel.com/templates/next.js/ai-code-translator)
\n[28] [Using GPT-4 to understand code](https://www.lesswrong.com/posts/ptY6X3BdW4kgqpZFo/using-gpt-4-to-understand-code)
\n[29] [Demystifying GPT Self-Repair for Code Generation](https://arxiv.org/pdf/2306.09896.pdf)
\n[30] [Evaluating Large Language Models Trained on Code](https://arxiv.org/abs/2107.03374)
\n[31] [DocPrompting: Generating Code by retrieving the docs](https://arxiv.org/pdf/2207.05987.pdf)
\n[32] [Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning](https://people.ece.ubc.ca/amesbah/resources/papers/cedar-icse23.pdf)
\n[33] [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf)
\n[34] [PAL: Program-aided language models](https://arxiv.org/pdf/2211.10435.pdf)
\n[35] [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://arxiv.org/abs/2211.12588)
\n[36] [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://arxiv.org/pdf/2112.00114.pdf)
\n[37] [Least-to-most prompting enables complex reasoning in large language models](https://arxiv.org/pdf/2205.10625.pdf)
\n[38] [Decomposed Prompting: A modular approach for solving complex tasks](https://arxiv.org/pdf/2210.02406.pdf)
\n[39] [Program synthesis with large language models](https://arxiv.org/pdf/2108.07732.pdf)
\n[40] [Self-Refine: A New MileStone in the AI Era](https://kargarisaac.medium.com/self-refine-a-new-milestone-in-the-ai-era-ac51568873da)
\n[41] [Self-Refine: Iterative Refinement with Self-Feedback](https://arxiv.org/pdf/2303.17651.pdf)
\n[42] [Teaching Large Language Models to Self-Debug](https://arxiv.org/pdf/2304.05128.pdf)
\n[43] [Self-consistency improves chain of thought reasoning in language models](https://arxiv.org/pdf/2203.11171.pdf)
\n[44] [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://arxiv.org/pdf/2206.02336.pdf)
\n[45] [An empirical evaluation of GitHub copilot's code suggestions](https://dl.acm.org/doi/abs/10.1145/3524842.3528470)
\n[46] [CodeT: Code generation with generated tests](https://arxiv.org/pdf/2207.10397.pdf)
\n[47] [I Speak, You Verify: Toward Trustworthy Neural Program Synthesis](https://arxiv.org/pdf/2210.00848.pdf) ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\nsubmission_df = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsubmission_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-14T22:35:49.520690Z","iopub.execute_input":"2023-07-14T22:35:49.521142Z","iopub.status.idle":"2023-07-14T22:35:49.547232Z","shell.execute_reply.started":"2023-07-14T22:35:49.521107Z","shell.execute_reply":"2023-07-14T22:35:49.544961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.loc[0]['value']='Generative AI for Programming'\nsubmission_df.loc[1]['value']='https://www.kaggle.com/code/shraddhabarke/programming-with-generative-ai'\nsubmission_df.loc[2]['value']='https://www.kaggle.com/code/jayitabhattacharyya/building-llms-from-scratch-generative-ai-report/comments#2343601'\nsubmission_df.loc[3]['value']='https://www.kaggle.com/code/jainam213/the-hitchhiker-s-guide-to-generative-ai/comments#2344032'\nsubmission_df.loc[4]['value']='https://www.kaggle.com/code/bhargav6031/advances-in-reinforcement-learning/comments#2344698'\nsubmission_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-14T22:35:49.552653Z","iopub.execute_input":"2023-07-14T22:35:49.553629Z","iopub.status.idle":"2023-07-14T22:35:49.569610Z","shell.execute_reply.started":"2023-07-14T22:35:49.553579Z","shell.execute_reply":"2023-07-14T22:35:49.568448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the submission to a CSV file\nsubmission_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-07-14T22:35:49.573032Z","iopub.execute_input":"2023-07-14T22:35:49.573956Z","iopub.status.idle":"2023-07-14T22:35:49.585066Z","shell.execute_reply.started":"2023-07-14T22:35:49.573915Z","shell.execute_reply":"2023-07-14T22:35:49.583814Z"},"trusted":true},"execution_count":null,"outputs":[]}]}