{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50947,"databundleVersionId":5741538,"sourceType":"competition"},{"sourceId":6145782,"sourceType":"datasetVersion","datasetId":3524381}],"dockerImageVersionId":30513,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# USING ARTIFICIAL INTELLIGENCE TECHNIQUES IN REQUIREMENT ENGINEERING STAGE OF SDLC FOR SOFTWARE INTELLIGENCE ","metadata":{}},{"cell_type":"markdown","source":"### *ABSTRACT*","metadata":{}},{"cell_type":"markdown","source":"Artificial Intelligence in general and Natural language Processing(NLP) in particular has made unprecedented progress recently in many areas of life, automating and enabling a lot of activities such as speech recognition, language translations, search engines(including text and image searches), text-generations,etc. Software engineering and Software Development Life Cycle(SDLC) is also not left out. Indeed, one of the most critical starting points of SDLC is the requirement engineering stage which traditionally has been dominated by business analysts. Unfortunately these business analysts have always done the job not just in a monotonous  way but  also in an error prone,  tedious,time-consuming, resource-intensive and generally inefficient  manner leading to poorly crafted works with lots of requirement creep and sometimes technical debts . This work looks at how this crucial initial stage could not just be automated but also improved using the state of the art and latest techniques in Artificial Intelligence and NLP.For the time being, this part of the work focuses mainly on introduction, background, problem definition, short literature review and review of related terms such as SDLC and NLP.\n","metadata":{}},{"cell_type":"markdown","source":"**Keywords:** NLP, Artificial Intelligence(AI), Software,Software Engineering, Requirement Gathering, SDLC.\n","metadata":{}},{"cell_type":"markdown","source":"### Introduction","metadata":{}},{"cell_type":"markdown","source":"Artificial Intelligence is leading many innovations in our World currently (Zhang et al, 2023), and it is also making lots of in-roads into the software development life cycle, the SDLC(Wangoo,2018). This section sets the background of this work, Problem Statement, Overview of SDLC with greater emphasis on Requirement Gathering stage, Natural Language Processing(NLP) and finally reviews the available literature-the details of the latter will come in subsequent works.\n","metadata":{}},{"cell_type":"markdown","source":"### Background:","metadata":{}},{"cell_type":"markdown","source":"Artificial Intelligence(AI) is currently being seen at the forefront of driving the 5th generation of industrial revolution. As of the time of writing, some top business executives are [calling](https://www.bostonglobe.com/2023/03/29/business/mit-scientists-tech-leaders-call-pause-artificial-intelligence-research/#:~:text=Tesla%20founder%20Elon%20Musk%2C%20Apple%20cofounder%20Steve%20Wozniak%2C,OpenAI%E2%80%99s%20ChatGPT%20and%20its%20more%20advanced%20version%2C%20GPT-4.) for a temporary halt in development of AI, there are also [debates](https://medium.com/swlh/will-ai-steal-your-job-or-will-it-make-you-irreplaceable-c2bd5ac42a2) currently going on regarding many jobs that are in the process of being displaced by it. Jobs such as content writers, editors, teachers, and even programmers are listed as being on the line. While these debates are still on-going with arguments for and against it, there is no doubt that AI is causing lots of disruptions in many industries. Many have also pointed to its risks both generally and in the military in particular. A case in point, A top [Google AI chief](https://www.theguardian.com/technology/2023/may/02/geoffrey-hinton-godfather-of-ai-quits-google-warns-dangers-of-machine-learning)(Geoffrey Hinton) recently resigned so that he could freely criticize the apparent risks associated with AI. Elons Musk himself-the owner of Twitter and Tesla-also pledged to build the so-called [“TruthGPT”](https://www.geeksforgeeks.org/what-is-truthgpt/)to compete with Microsoft-backed OpenAI’s GPT, Google’s Bard, etc.   All these serve to buttress further that AI is causing lots of upheaval and automating alot of domains traditionally done by human beings.\n\nIn the realm of software Engineering, AI is also not left out, ranging from Microsoft owned Github’s Copilot, ChatGPT’s code generation using Generative Pre-Trained Model(GPT), StackOverflow recently [reported](https://stackoverflow.blog/2023/06/07/self-healing-code-is-the-future-of-software-development/?utm_source=iterable&utm_medium=email&utm_campaign=the-overflow-newsletter) experiments currently being carried out in many areas  and also in Google such as using AI for code reviews, introducing plugins that could lead to the so called ‘self-healing codes’-that is the ability of codes already committed to the version system,being automatically corrected  and deployed to production by AI after being reviewed once the build process failed in the CI/CD pipeline.\n\nFor developers, it is not a secret that software engineering and development is really hard and complex (Yang and Sahraoui. 2022). On one hand, partly due to misalignment and the knowledge gap between business analysts or domain experts, who have expertise in their business domain, and software engineers who are masters of technicality, on the other hand. Thus, to simplify things, it has traditionally followed a cycle known as Software Development Life Cycle (SDLC), which outlines steps to software development such as Requirement Gathering, Requirement Specifications, Analysis, Design, Implementation, Testing (Ali, 2017) and Integration and finally Documentation, Training and Support, Deployment and Maintenance (Sharma, 2017). Requirement Engineering is obviously at the fore-front (Abdelnabi et al, 2021), with business analysts,product or project managers, and other key stakeholders teaming up to elicit these. Unfortunately, these processes have always been tedious and error prone, thus leading to codes constantly being re-worked and changed and driving up developmental costs and times. This means that there is an absolute need for software intelligence using AI and at least automation of some of these processes for efficiency.. With this background the  next section gives broader and more formal problem definition and statement which will be concretized after detailed literature review in Chapter 2. \n","metadata":{}},{"cell_type":"markdown","source":"### The Problem statement ","metadata":{}},{"cell_type":"markdown","source":"Requirement Engineering is the first stage of the SDLC. As one of the most crucial stages(Abdelnabi et al, 2021). It sets the proper background and foundation for all other stages of SDLC. It facilitates the proper understanding of the whole project regarding its scope and core functionality and the manner in which it is carried out can predict the success of any software project( Karunagaran,2020) and most failures of software projects are traceable to ineffective approach of this stage[(Sonbol et al,2022)](https://arxiv.org/abs/2206.00421). It is considered a bridge between the design and implementation phase of [SDLC(Al-Fedaghi,2021)](https://arxiv.org/abs/2107.13633) . It is expected to clearly and concisely define the functional and nonfunctional requirements of the proposed software system.\nThe human analysts and other stakeholders play a great role in the requirement elicitation process .These elicitation methods can involve interviews, prototyping,questionnaires, use cases, etc(Karunagaran, 2020). Generated documents are expressed in human languages. Which the  business or domain analyst  is expected to transform(Abdelnabi et al, 2021)  after analysis in order to provide a specification called Software Requirement Specification(SRS)\n\n\n","metadata":{}},{"cell_type":"markdown","source":"Sadly, there are inherent  challenges associated with using complex human languages in which the original documents were generated. These include issues such as ambiguity, incompleteness and inconsistencies(Maatuk and Abdelnabi,2021). Moreover, there are often trivial and insignificant details included in the documents, apart from the fact that it could be too long(for modern agile developers) and the elicitation process could be very tiresome,tedious and error prone.\n","metadata":{}},{"cell_type":"markdown","source":"Some of the above challenges have been addressed to a large extent using tools and AI techniques such as NLP(Hossain et al 2023). However most of them involve typical  rule based heuristics, lexical[(Sonbol et al, 2022)](https://arxiv.org/abs/2206.00421)  and semi-automatic approaches, information extraction, morphological analysis, conceptual model[(Hossain et al 2023)](https://arxiv.org/abs/2206.00421) and other related NLP techniques. \n","metadata":{}},{"cell_type":"markdown","source":"The above approaches have substantially yielded great results and improvements in solving these problems. But there still exists a gap as regards using the latest innovations in NLP such as using advanced Embeddings and in particular using Transformer and Generative AI to better solve these inherent difficulties found in natural language and requirement engineering process.","metadata":{}},{"cell_type":"markdown","source":"The afore-mentioned problems are summarized(Ammar et al 2012) and listed thus:\n1. Problem of Incompleteness(often leading to requirement creep)\n2. Problem of Ambiguity\n3. Problem of Imprecise and Vagueness of Requirements\n4. Difficulty in manually classifying Functional and Nonfunctional Requirements from requirement texts.\n5. Problem of Volatility of Requirements\n","metadata":{}},{"cell_type":"markdown","source":"### General Research Objective","metadata":{}},{"cell_type":"markdown","source":"It is important to note that most indigenous software companies are facing these challenges and in order to help them solve the problems. I will now list the general objectives(these will be concretized after detailed literature review) of this work: \n* To use the latest innovation  in Natural Language Processing(NLP) to elicit, build complete,well validated and efficient Requirement documentations from stakeholders\n* To use the latest approach NLP to build an efficient and accurate Requirement Engineering classifier from natural language user requirements that can classify functional(FR) and non-functional requirements(NFR)?\n* To detect and remove ambiguous and conflicting requirements and of course prioritize user requirements?\n* To produce either a Class Diagram(or any other UML artifact) or Software Requirement Specification (SRS) or both\n\n\n","metadata":{}},{"cell_type":"markdown","source":"It suffices to state here that due to time and resource constraint, it will not be possible to meet all these objectives at once, however I believe that this work will try to address at least one or two of the above objectives.                                    \nThe next section briefly reviews SDLC. ","metadata":{}},{"cell_type":"markdown","source":"### Overview of Software Development Life Cycle(SDLC)","metadata":{}},{"cell_type":"markdown","source":"**Software Development Life Cycle(SDLC)** is a software engineering process that  is broken down into many phases: Requirement Gathering, Requirement Specifications, Analysis, Design, Implementation, Testing and Integration and finally Documentation,Training and Support, Deployment and Maintenance.Though Agile software methodologies with its user stories and its philosophy that software requirement is never fully understood from inception-  is currently being used by many in the industries - however a lot of computer scientists, software developers and even agile enthusiast still embrace some aspect  and stages of SDLC.  A brief descriptions of each stage is describe below:\n","metadata":{}},{"cell_type":"markdown","source":"![SDLC](https://th.bing.com/th/id/R.ac81a0b8bf441116529d43cfc1a48c8c?rik=JFxl1AC7Pys7wg&pid=ImgRaw&r=0)","metadata":{}},{"cell_type":"markdown","source":"**Image Credit:** This is image was taken from [ArtOfTesting](https://artoftesting.com/software-development-life-cycle-sdlc)","metadata":{}},{"cell_type":"markdown","source":"#### 1. Requirement Engineering:","metadata":{}},{"cell_type":"markdown","source":"As the most important part of designing any software system (Hossain et al, 2023), this involves requirement elicitation, requirement gathering, management and validation. According to Pamela Z.(1997), “Requirements engineering is the branch of software engineering concerned with the *real-world goals* for, functions of, and constraints on software systems. It is also concerned with the relationship of these factors to *precise specifications* of software behavior, and to their *evolution over time and across software families*”((LaPlante, 2018. p. 25). Hence it aims to focus on what the customer wants in the application and not how (Computer Science Department-University of Cape Town, 2010). The above school further outlines some of these questions:\n**What exactly does the customer really want?**\n**What is it that the user needs to use the system?**\n**What impacts would application software make on  the users when developed?**\n\tUsually before the above questions can be answered, a number of steps are followed in Requirement engineering and these include: \nPreparation/Inception: Here the system’s scope and nature is clearly specified\nRequirement Elicitation: This involves initial gathering of the system requirements.\nElaboration: This improves the above number two steps to make it clearer.\nNegotiation: Priorities are defined here and any conflicts are resolved leading to the  key requirements being compiled.\nSpecification: This is where the Software Requirement Specification-which is the final result of the whole requirement engineering-is produced.\nValidation: This acts as a quality assurance(QA) checker ensuring there are no ambiguities,inconsistent and it is not inadequate and nothing essential is missing-that is that is complete. \nManagement: This ensures that any changes during the rest of SDLC is properly managed.\n\t  Often the above steps are a collaborative effort involving staff, customers, business analysts, project or product managers and other stakeholders. There are usually meetings, group discussions, brainstorming,questionnaires, etc to generate the requirements or (user stories; sometimes there are quick prototypes in modern agile environment)\n","metadata":{}},{"cell_type":"markdown","source":"#### 2. Analysis: ","metadata":{}},{"cell_type":"markdown","source":"This naturally follows from requirement engineering and forms an indispensable part of SDLC. And traditionally there is a business analyst that has the job of going through the analysis of the requirements documents to do proper elications,disambiguation of terms,prioritizing requirements and  requirements validation, separating functional and non-functional requirements,  and trying to manage it as well. This step usually ends with the popular Software Requirement Specifications(SRS).","metadata":{}},{"cell_type":"markdown","source":"At the end of this stage, the functional requirements of the system should be very clear as demonstreated in this diagram:","metadata":{}},{"cell_type":"markdown","source":"![ATM Core Functionality Diagram](https://www.conceptdraw.com/How-To-Guide/picture/ATM-system.png)","metadata":{}},{"cell_type":"markdown","source":"**Credit: The above image is taken from [ConceptDraw](https://www.conceptdraw.com/examples/uml-atm)**","metadata":{}},{"cell_type":"markdown","source":"#### 3. Design:","metadata":{}},{"cell_type":"markdown","source":"According to (Sharma and Pandy,2015), Design Phase is supposed to produce a model that should exude quality, determination, and product. This phase operates on 4 important levels: Abstraction, Information Hiding, also known as Encapsulation, Design Patterns, and Modularity. I recommend the trendy design patterns book, *Design Patterns Elements of Reusable Object-Oriented Software*, by the popular Gang of Four, for a reader interested in exploring more this aspect of SDLC. Also at this stage, an important principle of Object-Oriented Programming Design (OOP) called SOLID (The Single Responsibility, Open-Closed, Liskov-Substitution, Interface Segregation and Dependency Inversion) principles are considered and implemented here. SOLID promises understandable, readable, clean, testable, scalable, extensible, and modularized codes with reusability at its core. Again, the interested reader is advised to read more by its pioneer Robert C. Martin, though it was later enhanced by Michael Feathers with the acronym SOLID, in his 2000 paper titled *Design Principles and Design Patterns*. Other artifacts such as UML diagrams and prototypes, are produced at the end of this stage. Other important design considerations, at the architectural level, include: Caching, database designs(replication, caching, indexing, sharding), overall system scalability, resiliency, recovery and failure handling, security and access control, data storage and retrieval, integrity, consistency and availability, data privacy, backup and recovery, cross cutting concerns(logging and monitoring), system maintenance and updates, user experience(UX) and usability, usage patterns and traffic(for web applications), etc. Finally, Reactive systems have gained popularity since 2013-2014, led by (Boner et al, 2014), in their [Reactive Manifesto](https://www.reactivemanifesto.org/) published in 2014. They aimed to propose a system that is Reactive (Responsive, Resilient, Elastic and Message Driven), loosely coupled, flexible and scalable.","metadata":{}},{"cell_type":"markdown","source":"![Sample ATM UMLDiagram from this phase](http://www.obfuscata.com/wp-content/uploads/2017/11/UML-diagram-2.png)","metadata":{}},{"cell_type":"markdown","source":"**Credit: The above image is taken from [Obfuscata](http://www.obfuscata.com/wp-content/uploads/2017/11/UML-diagram-2.png)**","metadata":{}},{"cell_type":"markdown","source":"#### 4. Implementation: ","metadata":{}},{"cell_type":"markdown","source":"Ordinarily, the process of writing codes begins in this step. In latest techniques such as OOP and Aspect Oriented Programming (AOP), classes, interfaces, abstract and concrete classes, aspect join points and join cuts for application logs and other cross-cutting concerns are created and implemented here. Often, test cases are usually written as required by the latest Test-Driven Development (TDD) techniques.","metadata":{}},{"cell_type":"markdown","source":"#### Testing:","metadata":{}},{"cell_type":"markdown","source":"At this point in the cycle, diverse levels of tests are conducted starting from unit tests, if not already begun during and before the implementation stage, as required by TDD culture. Integration tests, which often include end-to-end tests(e2e) and UI testing, occur here as well. Finally, another important acceptability test, also called User Acceptance Test (UAT), takes place at this level. It ensures that the stakeholders and users accept the core functionality of the software as specified in SRS. Others include Regression Testing, Load Testing, Stress, Security, UI and Fuzzy Testing.\n","metadata":{}},{"cell_type":"markdown","source":"#### Documentation,Training/Support: ","metadata":{}},{"cell_type":"markdown","source":"Although not always included in official SDLC cycles; nevertheless, at this level, the user is not just educated but a written documentation explaining how the system works and that tries to resolve common issues that the user may face are documented. There is often a quick *Getting Started* manual, components are also explained. Any further face-to-face training is done at this point, and lastly, it also includes on-going support for potential issues the user may encounter when using the software.","metadata":{}},{"cell_type":"markdown","source":"#### Deployment:\n","metadata":{}},{"cell_type":"markdown","source":"In this penultimate stage, the developed application or system is made available for use. Usually there are two or three copies or versions: development copies in a *development* environment, for developers to test their works through their CI/CD pipelines, and *production* environment, where the tested codes that have passed quality assurance (QA) tests are made available to users. These two separate environments, moreover, ensure that customers continue to use the system while it has been maintained or upgraded. Sometimes there is a *sandboxed or staging* environment where developed features are tested before moving them to production.","metadata":{}},{"cell_type":"markdown","source":"#### Maintenance: ","metadata":{}},{"cell_type":"markdown","source":"Agile enthusiasts would always insist that the full scope of software requirements is never totally known beforehand. This stage constantly checks for not just new and unforeseen bugs, but also extra and new features, the so called versions management is done here leading to such terms like patch, major and minor version releases, long term support(LST) as is the case of NodeJS, Oracle Java, etc.                                   In general there is overall system monitoring of performance,user experience and security to see new ways to improve the system. Typically  MLOps(Machine Learning Operations) is used here these days.","metadata":{}},{"cell_type":"markdown","source":"![Components of MLOps](https://cms.databricks.com/sites/default/files/inline-images/mlops-components.png)","metadata":{}},{"cell_type":"markdown","source":"**Credit: [DataBricks](https://www.databricks.com/glossary/mlops)**","metadata":{}},{"cell_type":"markdown","source":"#### Quick Review of SDLC Process Models","metadata":{}},{"cell_type":"markdown","source":"Formerly, SDLC consists of cycles such as Planning, Defining, Designing, Building, Testing and Deployment (Ali, 2017). It is, nevertheless, made up of distinct process models. In each of these different process models, the above cycles are executed and implemented in diverse ways. These separate ways could be sequential, iterative, incremental, prototypical, etc. The models include Waterfall, Iterative, Big Bang, Agile, Rapid Application Development (RAD), V-Model, Spiral and finally Prototype Model. \nThe Waterfall model will be the default and assumed model of this work\n","metadata":{}},{"cell_type":"markdown","source":"### Review of Natural Language Processing(NLP)","metadata":{}},{"cell_type":"markdown","source":"Natural Language Processing(NLP) is one of the hottest fields in AI, it focuses on building machines(models) that are capable of manipulating human languages as well as humanly generated data. It aims to bridge the gap between human language and machines for effective communication. It originated from computational linguistics, though it is an engineering field. The aim of NLP is to disambiguate human language for easier understanding and processing by machines.NLP is broadly divided into 2 fields: Natural Language Understanding(NLU) which seeks to understand the semantics of human languages and Natural Language Generation which focuses on machine’s text generation ability.","metadata":{}},{"cell_type":"markdown","source":"**Popular NLP Current Applications And Tasks:** \nCurrently, NLP is being used in many day-to-day applications such as Apple Siri, Amazon Alexa, Google’s search engine and latest innovation in Generative Pre-Trained Model (GPT) and other Large Language Models (LLMs) such as [Bert](https://doi.org/10.48550/arXiv.1810.04805),[Gemini](https://deepmind.google/technologies/gemini/#introduction), [LLama2](https://ai.meta.com/llama/), among others. The tasks that can be performed by this AI field include, but is not limited to: Sentiment Analysis, Machine Translation, Named Entity Recognition (NER), Spam Detection, Grammatical Syntax Correction, Summarization, Topic Modelling, Text Information Retrieval, Text and Content Generation (Conversational Chatbots, Autocomplete, etc.),Code Generation, among others. There are also multimodal capabilites which accept and produce not only texts, but also images, audios, etc. Since this is a review, the interested reader is advised to check popular NLP courses by Andrew Ng on Coursera or any other material to learn more about each of the above tasks.","metadata":{}},{"cell_type":"markdown","source":"\n#### NLP Processing Techniques\n Almost all NLP tasks follow these techniques:\n","metadata":{}},{"cell_type":"markdown","source":"#### Data Preprocessing:                                                                   \nJust like every machine learning tasks, this prepares the text data in a format(often numeric) understandable by the model , also to improve performance. But for NLP, these include:","metadata":{}},{"cell_type":"markdown","source":"##### Tokenization: ","metadata":{}},{"cell_type":"markdown","source":"This breaks a text into sentences or a sentence into words. Example: “I am reading an essay”. The result would be after tokenization “I”, “am”, “reading”, “an”, “essay”.","metadata":{}},{"cell_type":"markdown","source":"**Short Demo**","metadata":{}},{"cell_type":"markdown","source":"****Import Spacy NLP Library****","metadata":{}},{"cell_type":"code","source":"import spacy","metadata":{"execution":{"iopub.status.busy":"2023-07-05T21:23:40.863420Z","iopub.execute_input":"2023-07-05T21:23:40.864411Z","iopub.status.idle":"2023-07-05T21:23:56.596218Z","shell.execute_reply.started":"2023-07-05T21:23:40.864367Z","shell.execute_reply":"2023-07-05T21:23:56.595261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Define a short web scraper function using BeautifulSoup\nimport requests\nimport re;\nfrom bs4 import BeautifulSoup\n\ndef webScrapper(url,html_tag):\n  reqObject = requests.get(url).text\n  beautifier = BeautifulSoup(reqObject);\n  tag_contents = beautifier.findAll(html_tag);\n\n  textLists = [sentence.text for sentence in tag_contents];\n  finalContents = \" \".join(textLists);\n  textsWithNewLinesRemoved = re.findall(\"\\w.*\",finalContents);\n  article = \" \".join(textsWithNewLinesRemoved);\n  return article;","metadata":{"execution":{"iopub.status.busy":"2023-07-05T21:20:44.581440Z","iopub.execute_input":"2023-07-05T21:20:44.582042Z","iopub.status.idle":"2023-07-05T21:20:45.042480Z","shell.execute_reply.started":"2023-07-05T21:20:44.582005Z","shell.execute_reply":"2023-07-05T21:20:45.041010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"url = \"https://edition.cnn.com/2023/06/16/africa/darfur-sudan-wagner-conflict-cmd-intl/index.html\";\narticle = webScrapper(url,\"p\")\narticle\nsentencesLists = article.split(\".\");\nprint(\"lent==\",len(sentencesLists))","metadata":{"execution":{"iopub.status.busy":"2023-07-05T21:21:57.365102Z","iopub.execute_input":"2023-07-05T21:21:57.365587Z","iopub.status.idle":"2023-07-05T21:21:57.590360Z","shell.execute_reply.started":"2023-07-05T21:21:57.365547Z","shell.execute_reply":"2023-07-05T21:21:57.589022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Initialize Spacy NLP pipeline","metadata":{}},{"cell_type":"code","source":"nlp = spacy.load(\"en_core_web_sm\");\ndoc = nlp(article)","metadata":{"execution":{"iopub.status.busy":"2023-07-05T21:27:32.646762Z","iopub.execute_input":"2023-07-05T21:27:32.647205Z","iopub.status.idle":"2023-07-05T21:27:34.628224Z","shell.execute_reply.started":"2023-07-05T21:27:32.647172Z","shell.execute_reply":"2023-07-05T21:27:34.626979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Lemmatization and Stemming: \n","metadata":{}},{"cell_type":"markdown","source":"This entails converting a word to its root and base form. Example: “came” would be “come”, “reading” would be “read”.","metadata":{}},{"cell_type":"markdown","source":"**Code Example:**","metadata":{}},{"cell_type":"code","source":"#import pandas\nimport pandas as pd;\nwords_lemma_dict = {\"Token\":\"Lemmatized Version\"};\ncounter = 0;\nindices = [];\nfor token in doc:\n  words_lemma_dict[token]=token.lemma_;\n  indices.append(counter);\n  counter+=1;\npd.DataFrame(pd.Series(words_lemma_dict)).head(7)","metadata":{"execution":{"iopub.status.busy":"2023-07-05T21:30:20.385867Z","iopub.execute_input":"2023-07-05T21:30:20.386279Z","iopub.status.idle":"2023-07-05T21:30:20.415495Z","shell.execute_reply.started":"2023-07-05T21:30:20.386247Z","shell.execute_reply":"2023-07-05T21:30:20.413899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Stop Words Removal:","metadata":{}},{"cell_type":"markdown","source":"This removes words(“a”, “an”,”the”,etc) that don’t provide much information about the text. Again in the above example “I”, “am” and probably “a” would be removed, leaving out only “reading, thesis”.\n","metadata":{}},{"cell_type":"markdown","source":"**Code Example:**","metadata":{}},{"cell_type":"code","source":"stoppedWordsAndPunctRemoved = [token for token in doc if not token.is_stop and not token.is_punct]\nstoppedWordsAndPunctRemoved[:6]","metadata":{"execution":{"iopub.status.busy":"2023-07-05T21:30:50.612293Z","iopub.execute_input":"2023-07-05T21:30:50.612714Z","iopub.status.idle":"2023-07-05T21:30:50.624733Z","shell.execute_reply.started":"2023-07-05T21:30:50.612678Z","shell.execute_reply":"2023-07-05T21:30:50.623293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Feature Extraction:","metadata":{}},{"cell_type":"markdown","source":"Once more, just like every ML technique, the texts are represented as features, usually vectors, or numbers, to be used by the modelling algorithm. Traditionally, feature extraction techniques have been Bag-of-Words, Term-Frequency, Inverse Document Frequency (TF-IDF), but due to inherent weaknesses, such as non-contextual, unintelligible, unordered representations with high dimensionality or the so-called cause of dimensionality, found in them, latest techniques involving word embeddings are now extensively used. Examples of such techniques include Word2Vec (word to vector), GloVe (Global Vector), Facebook, Meta’s, Fast-Text, and even higher still there are now Tranformers introduced in a paper titled [*Attention Is All You Need*](https://doi.org/10.48550/arXiv.1706.03762) by Vaswani et al (2017)","metadata":{}},{"cell_type":"markdown","source":"![TF-IDF](https://wordpress.deeplearning.ai/wp-content/uploads/2022/10/06.-Tokenizers_TF-IDF_Captioned-1-1024x577.png)","metadata":{}},{"cell_type":"markdown","source":"**Credit: The above image is taken from DeepLearning.AI**","metadata":{}},{"cell_type":"markdown","source":"**Code Example: Bag of word implementation using the above scraped article**","metadata":{}},{"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer\nvectorizer = CountVectorizer(stop_words='english')\nX = vectorizer.fit_transform(sentencesLists)\ndf_my_bow = pd.DataFrame(X.toarray(),columns=vectorizer.vocabulary_)\ndf_my_bow.head(7)","metadata":{"execution":{"iopub.status.busy":"2023-07-05T21:36:42.582065Z","iopub.execute_input":"2023-07-05T21:36:42.582539Z","iopub.status.idle":"2023-07-05T21:36:42.616780Z","shell.execute_reply.started":"2023-07-05T21:36:42.582486Z","shell.execute_reply":"2023-07-05T21:36:42.615318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Modeling:","metadata":{}},{"cell_type":"markdown","source":"After the above immediate technique has been done, the numerically represented features are fed into traditional NLP’s algorithms like Decision trees, Naïve (based on Bayes’ theorem) Bayes, Logistic Regressions, Hidden Markov Models, and Deep learning algorithms such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Autoencoders and latest *attention models*, called *Transformers*.","metadata":{}},{"cell_type":"markdown","source":"![Transformer](https://wordpress.deeplearning.ai/wp-content/uploads/2022/10/13-Transformer_CAPTIONED-1024x577.png)","metadata":{}},{"cell_type":"markdown","source":"**Credit:This image is taken from Deeplearning.AI**","metadata":{}},{"cell_type":"markdown","source":"## Short Literature Review of Related Works ","metadata":{}},{"cell_type":"markdown","source":"Here I will provide a short Literature review of related academic works. Much more detailed, extensive and refined review would be seen in Chapter 2. \nThese reviews would be divided into these sub-sections: The earliest works to automate requirement engineering with NLP/AI , Lingustic and Grammer Rules, GUI/Diagrammatic Tools Domain Ontology/Pattern Matching/Heuristic Rules-Based Techniques, and ML Models/Modern techniques.\n","metadata":{}},{"cell_type":"markdown","source":"### Earliest Works:","metadata":{}},{"cell_type":"markdown","source":"The earliest works in this field to automate requirement gathering and (analysis) stage of SDLC started more than four decades ago(in the 1980s) using such strategies like search and schema based  reasoning,the aim was to create intelligent tools, the assumption being that the first two stages of SDLC(Analysis and Design)were highly knowledge intensive and hence needed AI-powered tools which where basically rule-based at that time, the second technique was analysis of Natural Languages(premised on the realization that most of the available requirements texts were basically human languages) some systems such as  Harman(2000) used this technique. \nAbbott (1983) suggested a linguistic based model for analysis of English language expressed-software requirements;the aim was to the data types of basic programming language and their operations(methods and functions). Booch(1986) improved on this approach by introducing an Object-Oriented Design technique where nouns in the requirement texts would be mapped to objects and their classes while operations and actions would be mapped to verbs . \nSaeki et al (1987) proposed a way to incrementally build software libraries from formal  OOP specifications. The system would analyze informal requirements one sentence at a time. Here nouns and verbs would be extracted from requirement texts but unfortunately the system was unable to distinguish the key terms for SRS, this defect required the intervention of human analyst .Orlowska and Dunn(1990) proposed an interpreter for Natural language called NIAM. Mezian F.(1994) designed a system for the identification of data types and built a system capable of translating required texts to a VDM language used for formal specifications. It is important to note that this was able to detect ambiguities in the requirement documents. Mich and Garingo, Mich(1994) and Mich(1996)  proposed a natural language based prototype software called NL-OOPs that is capable of generating OO analysis models from the specifications of natural language. A number of researchers Fuchs et al.(1998),Cyre(1995) ,etc proposed and designed a language which is a controlled subset of our human natural languages to write SRS. \n \n","metadata":{}},{"cell_type":"markdown","source":"### Linguistic and Grammar Rules:","metadata":{}},{"cell_type":"markdown","source":"These are based on pure analysis and parsing of grammar and linguistic rules such as subject, verb and objects , leveraging on their relationships. Systems like an object-based linguistic interactor and translator called LOLITA(Mich ,1996) after analyzing the sentence, consider nouns as objects and end up producing OOPs from natural language specifications. According to Harmain and Gaizauskas(2003)  authors Moreno  & López(2000) also developed and defined a framework for the creation of the required system’s Object Oriented(OO) models with the aid of linguistic instrument techniques. However authors Kuchta and Padhiyar(2018)  proposed a methodology different from domain ontology and traditional statistical approaches due to inherent weaknesses. Rather they proposed a Grammatical analysis tool powered by OpenNLP and  WordNet. The procedures involve splitting(with the help of OPENLP) the texts into sentences,followed by Tokenization, Part of Speech(POS) tagging,Disambiguation of POS,Phrase Detection  and finally Noun phrases. The above steps are combined into an NLP pipeline.Then concepts are detected with the help of WordNet. Finally hypernym, synsets and hyponyms  are used to detect relationships between words. Their contribution helped to solve the problem of ambiguous concepts. Their next plan of action as of the time of writing was to generate class diagrams from the concepts that have been disambiguated.\n\n","metadata":{}},{"cell_type":"markdown","source":"### GUI/Diagrammatic Aided Tools:","metadata":{}},{"cell_type":"markdown","source":"In this category, there is usually a GUI that enables or aids the analysts in requirements elicitations and management. Authors such as Harman and Gaizauskas(2003) developed a Class-Model Builder which builds class diagrams specified in UML from natural language requirement texts. According to Mohd and Rodina(2010) a web-enabled platform called Circle was developed by authors Ambriola and Gervasi for the selection, elicitation and validation of software requirements.It is capable of information extraction and is capable of measuring consistency of extracted information.             Harmain and Gaizauskas(2003) built a system that has two versions, one can be used with the help of an analyst and the other is automated. The developed system is a GUI based tool called Class Builder that facilitates the work of business analysts by producing a UML class diagram. The steps involved: Obtaining functional requirements from requirement texts, It then uses NLP techniques\n(Tokenization,POS tagging,morphological analysis,parsing,etc) for syntactic and semantic analysis of the texts, the third step leverages the result of the second stage to produce objects,attributes and their corresponding relationships.Finally a static UML diagram is created which can be further improved by the analysts using graphical tool.                   This approach was evaluated on sample requirement texts from different textbooks calculating both precision and recall. The recall was 73% and precision was 66%. Though it was a good starting point for producing OOP conceptual models and class diagrams, however the limitations include: Small corpus was used, the grammar rules of their parser need improvement, the discourse module can also be improved. Finally it didn’t touch on the dynamic part of the system and the evaluation technique was incomplete.\n\n","metadata":{}},{"cell_type":"markdown","source":"### Domain Ontology/Pattern Matching and Heuristic Rules-Based Techniques:","metadata":{}},{"cell_type":"markdown","source":"Here certain patterns and keywords, entities, relationships and co references are extracted from the requirement  texts, this includes using heuristic rules. According to Gamage(2023) recent studies in these areas use Part of Speech(POS) tagging, domain centered databases, domain ontology and entities.  According to Zhou and Zhou(2004) as reported by Ahmad(2010) proposed a technique that uses Domain Ontology and NLP, this is based on the fact that the main classes are usually interrelated via different types of relationships such as one-to-one, many-to-one, etc. The OO classes are discovered via Part of Speech tagging, grammar parsing  and linguistic patterns, finally refinement is done using domain ontology.   \n\nResearchers  Al-Qaoud and Ahmad (2010)\n proposed a UI based RACE tool which is a desktop application that assists analysts and software engineers  in analyzing and managing requirement texts, with the aim of discovering key concepts and their relationships. The use case for RACE involves:Creation of Software Requirement texts; carrying out requirement analysis(noun, verb and noun phrase analysis) using apache java-based OpenNLP and WordNet developed at Princeton University, it applied NLP POS tagger to detect noun, noun phrases, used domain ontology for refinement of the identified concepts. The system was also capable of identifying four relationships such as IS-A(Generalization), Association Composition(aggregation) and Dependency OOP relationships.  The system codes were written in C# and evaluated on library system requirement texts and was able to identify the key concepts and modeled and generated the UML class diagram. However there was no indication that it was tested extensively and heuristic  rules are not usually very efficient and performant in real life. As of the time the work was reported, it could not identify one-to-one,one-to-many, many-to-many relationships and was restricted to windows operating system only. \n\n\n\nHerchi and Abdessalem(2012) implemented a similar approach as the afore-mentioned Al-Qaoud and Ahmad (2010)  RACE system . But in addition they introduced  XML and used a text-processing java programming based GATE open source framework built by University of Sheffield. They in addition used linguistic heuristic rules and just as the previous authors they used domain ontology for refining the identification of concepts.Their solution was in addition able to identify relationships among OOP objects. The solution performed better(recall:83%, precision 93%) in comparison with CM-Builder using case studies published in OO analysis books. However, as in the case with heuristics rules,it didn’t cover extensive rules. Though it solved the target problem(class diagram generation).It didn’t however conver the dynamic parts of UML.\n\n","metadata":{}},{"cell_type":"markdown","source":"### Machine Learning and Modern Techniques","metadata":{}},{"cell_type":"markdown","source":"ML models are typically trained to extract relationships and entities from NL texts through algorithmic training to recognize patterns in the context of the texts. According to Gamage(2023),these models are good in identifying links and relationships that exist among components. Such popular ML algorithms include SVM(Support Vector Machine), Tree based algorithm such as Decision Tree. They make use of initially defined keywords(pertaining to a particular domain) stored in  the database to a particular domain with the aid of domain ontology. Also according to the same author Gamage(2023),  reported that these authors Maatuk and Abdelnabi(2021)  made use of NLP in combination of heuristic rules to extract UML artifacts with aid of Stanford coreNLP library. This framework aided in achieving dependency parsing, extraction of information as well as tokenization.   Logistics, Perceptron classifiers were also used.In other works, Naive Bayes classifier was used by Arachi(2022) for the attributes and key terms needed to generate UML components. In other areas advanced algorithms such as CNN and GAN(Generative Adversarial Network) were used in generating the diagram of the system architecture.  As regards [Yang and  Sahraoui (2022)](https://arxiv.org/abs/2210.14441), they adopted NLP, ML,Model Driven Engineering(MDE) and Pattern Matching Techniques, their approach involves generating UML class diagrams from Software requirement specifications(SRS) of requirement texts. It’s remarkable that they are among the first to officially use a dataset departing from sole use of only heuristics, pattern matching and rule based which have dominated the works of earlier authors. Their steps include: First, they create a dataset of class diagrams from the NL specifications by breaking it into sentences. Secondly using ML classifiers they label  each sentence either as OOP class or relationship type, Thirdly using grammar/NL patterns each sentence is mapped into a fragment of UML. Lastly the whole fragments are combined together into one  big UML diagram with the aid of an algorithm that is capable of composing the whole fragments. Their study which is novel in some ways compared to previous works are useful for benchmarking and further studies though the performance on evaluation is poor-in terms of recall and precision. The approach was evaluated on a dataset containing 62 class diagrams consisting of 624 fragments. Perhaps the smallness of the datasets contributed to poor performance.\n\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"**It is important to emphasis that there are many other detailed literature reviewed, but due to time constraints and in order not to unduly enlongate this write up, I will stop here. Similarly, the work also has methodology, experimentation, discussions and conclusion. But for the same reason of the need to make it a short blog/write up, they were not included. However, the interested reader could see part one here in Research gate titled [USING ARTIFICIAL INTELLIGENCE TECHNIQUES IN THE REQUIREMENT ENGINEERING STAGE OF TRADITIONAL SDLC PROCESS](https://www.researchgate.net/publication/376950755_USING_ARTIFICIAL_INTELLIGENCE_TECHNIQUES_IN_THE_REQUIREMENT_ENGINEERING_STAGE_OF_TRADITIONAL_SDLC_PROCESS?channel=doi&linkId=658ede0a0bb2c7472b1edd42&showFulltext=true)** ","metadata":{}},{"cell_type":"markdown","source":"# CONCLUSION","metadata":{}},{"cell_type":"markdown","source":"In this short write-up or blog of a bigger project, I have basically laid the foundations for the following fuller work... In particular, I have shown that AI is causing lots of disruptive changes in various domains and software industry is not left out. The motivations of the study has been established and the problem arising from manual but error prone process of requirement engineering as done in many software companies. The SDLC cycles were briefly reviewed with emphasis on the Requirement Gathering stage. NLP which is the primary AI tool of this work was reviewed and this section ended with review a few number of related literature and works. Further and more detailed part 1 of the whole project could be see in this research gate paper titled [USING ARTIFICIAL INTELLIGENCE TECHNIQUES IN THE REQUIREMENT ENGINEERING STAGE OF TRADITIONAL SDLC PROCESS](https://www.researchgate.net/publication/376950755_USING_ARTIFICIAL_INTELLIGENCE_TECHNIQUES_IN_THE_REQUIREMENT_ENGINEERING_STAGE_OF_TRADITIONAL_SDLC_PROCESS?channel=doi&linkId=658ede0a0bb2c7472b1edd42&showFulltext=true) \n","metadata":{}},{"cell_type":"markdown","source":"# REFERENCES","metadata":{}},{"cell_type":"markdown","source":"Saeki, M. et al. (1989). Software development process from natural language specification. In the 11th International Conference on Software Engineering. IEEE, 64ś73. Available on https://doi.org/10.1145/74587.74594\n\nZave, P. (1997). Classification of research efforts in requirements engineering. ACM Computing Surveys, 29(4): 315–321 \n\nHarmain, H. M., & Gaizauskas, R. (2003). CM-Builder: A natural language-based CASE tool for object-oriented analysis. Automated Software Engineering Journal, 10(2), 157–181, United Arab Emirates University, UAE and University of Sheffield, UK.\n\nKumar, D. D. and Babar, M. A. (2009). An automated tool for generating UML models from natural language requirements. In Proceedings of the 2009 24th IEEE/ACM international conference on Automated Software Engineering. IEEE Computer Society, 680ś682. Available from https://doi.org/10.1109/ASE.2009.48\n\nDepartment of Computer Science. Software Engineering (2010), University of Cape Town, \nAl-Qaoud, M.I. and Ahmad, R. (2010). Class Diagram Extraction from Textual Requirements Using Natural Language Processing (NLP) Techniques.  In 2010 Second International Conference on Computer Research and Development. IEEE, 200ś204. Available from https://doi.org/10.1109/ICCRD.2010.71\n\nAmmar H., Abdelmoez W., Hamdi M.S. (2012) Software Engineering Using Artificial Intelligence Techniques: Current State and Open Problems. West Virginia University, Ahmed Bin Mohammed Military College, Arab Academy of Science, Technology and Maritime Transport.\n\nHerchi H., Abdessalem W.B. (2012). From user requirements to UML class diagram, Department of computer science, High Institute of Management of Tunis, Tunisia\nhttps://arxiv.org/ftp/arxiv/papers/1211/1211.0713.pdf\n\nShreta, S., Santo P. (2015). Integrating AI Techniques In SDLC-Design Phase Perspective, Jagannath University and Ministry of Communications, Govt of India, India.\n\nAli, K. (2017). A Study of Software Development Life Cycle Process Models. MS Computer Science Department of Computer Science, Lahore Leads University Main Campus, Kalma Chouk Lahore, Pakistan. International Journal of Advanced Research in Computer Science, Volume 8, No. 1, Jan-Feb 2017.  ISSN No. 0976-5697. http://www.ijarcs.info/\n\nSharma, M.K. (2017). A study of SDLC to develop well engineered software. Ph.D. Research Scholar (GNA University) Head, Sr. Assistant Professor Post Graduate Department of Computer Science Jagdish Chandra D.A.V. College, Dasuya, Punjab, India. International Journal of Advanced Research in Computer Science, Volume 8, No. 3, March – April 2017.  ISSN No. 0976-5697. http://www.ijarcs.info/\n\nKuchta, J. and Padhiyar, P. (2018). Extracting concepts from the software requirements specification using natural language processing. Proceedings - 2018 11th International Conference on Human System Interaction, HSI 2018, 443–448. Available from https://doi.org/10.1109/HSI.2018.8431221\n[Accessed 4 July 2023]\n\nWangoo, D. P. (2018). Artificial Intelligence Techniques in Software Engineering for Automated Software Reuse and Design. Ph.D. Research Scholar, Department of CSE & IT Amity School of Engineering and Technology Amity University, Noida, Uttar Pradesh, India Proceedings - 2018 4th International Conference on Computing Communication and Automation (ICCCA) [Accessed 15th April, 2023] at 14:56:35 UTC from IEEE Xplore\n\nLaplante P. A. (2018). Requirements Engineering for Software and Systems Pages 25, CRC Press Taylor & Francis Group, NW, Suite, Roca Raton, Florida, USA.\n\nElallaoui, M. et al (2018). Automatic Transformation of User Stories into UML Use Case Diagrams using NLP Techniques, University Ibn Tofail, Kenitra; Mohammed V University in Rabat, Morocco. \n\nKarunagaran, A. (2020). How Artificial Intelligence Can Transform Software Gathering Process? Faculty of Science and Technology Bournemouth University Bournemouth, United Kingdom\n\nAl-Fedaghi, S. (2021) Beyond SDLC: Process Modeling and Documentation Using Thinging Machines. Computer Engineering Department Kuwait University, Kuwait. IJCSNS International Journal of Computer Science and Network Security, VOL.21 No.7, July 2021\nhttps://arxiv.org/abs/2107.13633  https://doi.org/10.22937/IJCSNS.2021.21.7.23\n\nAbdelnabi, E.A, et al. (2021). Generating UML Use Case and Activity Diagrams Using NLP Techniques and Heuristics Rules, Faculty of Information Technology, Benghazi University, Libya.\n\nNasiri, S. et al (2021). From User Stories to UML Diagrams Driven by Ontological and Production Mode. LMMI Laboratory of ENSAM, Moulay Ismail University ISIC Research Team of ESTM Meknes, Morocco. International Journal of Advanced Computer Science and Applications (IJACSA), Vol. 12, No. 6, 2021. Available from http://dx.doi.org/10.14569/IJACSA.2021.0120637\n\nSonbol R., et al (2022). The Use of NLP-Based Text Representation Techniques to Support Requirement Engineering Tasks: A Systematic Mapping Review\nDepartment of Informatics, Higher Institute for Applied Sciences and Technology (HIAST), Faculty of Information and Communication Technology, Arab International University (AIU), Damascus, Syria. IEEE Last Accessed at  https://doi.org/10.1109/ACCESS.2022.3182372. https://arxiv.org/abs/2206.00421\n\nYang S. and Sahraoui H. (2022). Towards Automatically Extracting UML Class Diagrams from Natural Language Specifications. Universite De Montreal, Montreal, Canada. In ACM/IEEE 25th International Conference on Model Driven Engineering Languages and Systems (MODELS ’22 Companion), October 23–28, 2022, Montreal, QC, Canada. ACM, New York, NY, USA, 8 pages https://arxiv.org/abs/2210.14441  https://doi.org/10.1145/3550356.3561592.\n\nBudake, R. et al. (2023). Challenges And Future Of AI-Based Requirement Analysis: A Literature Review. Department of Computer Science K.H. College, Gargoti-416209, and Shivaji University, Maharashtra, India 2CSIBER, Kolhapur-416004, Maharashtra, India.\n\nGamage, Y. (2023). Automated Software Architecture Diagram Generator using Natural Language Processing. A Dissertation by Mr Yasitha Lalanga Gamage Submitted in partial fulfillment of the requirements for the BSc (Hons) in Computer Science degree at the University of Westminster, UK. May, 2023 http://dx.doi.org/10.13140/RG.2.2.31866.26563\n\nZhang, C. et al (2023). Generative Image AI Using Design Sketches as input: Opportunities and Challenges. In Creativity and Cognition (C&C ’23), June 19–21, 2023, Virtual Event, USA. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3591196.3596820\n\nHossain, B.A et al (2023). Natural Language Based Conceptual Modelling Frameworks: State of the Art and Future Opportunities ACM Computing Survey Available from https://doi.org/10.1145/3596597\n\nSharma, R. et al (2014) Machine Learning for Constituency Test of Coordinating Conjunctions in Requirements Specifications.  RAISE 2014: Proceedings of the 3rd International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering June 2014 Pages 25–31 \nhttp://dx.doi.org/10.1145/2593801.2593806\n\nVaswani, A. et al (2017). Attention Is All You Need. Google Brain and Google Research Team. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA https://doi.org/10.48550/arXiv.1706.03762","metadata":{}},{"cell_type":"markdown","source":"**Submission Preparation**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\npd.options.display.max_colwidth = None\nsubmission = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-reports/submission.csv\")\n","metadata":{"execution":{"iopub.status.busy":"2023-07-16T22:09:46.167562Z","iopub.execute_input":"2023-07-16T22:09:46.167958Z","iopub.status.idle":"2023-07-16T22:09:46.179122Z","shell.execute_reply.started":"2023-07-16T22:09:46.167935Z","shell.execute_reply":"2023-07-16T22:09:46.178240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv(\"submission.csv\", index=False)\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-16T22:09:52.077275Z","iopub.execute_input":"2023-07-16T22:09:52.077790Z","iopub.status.idle":"2023-07-16T22:09:52.087457Z","shell.execute_reply.started":"2023-07-16T22:09:52.077757Z","shell.execute_reply":"2023-07-16T22:09:52.086643Z"},"trusted":true},"execution_count":null,"outputs":[]}]}