{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.display import HTML\nimport time\n\nhandle = display(HTML(\"\"\"👌\"\"\"), display_id='html_marquee1')\ntime.sleep(2)\nhandle = display(HTML(\"\"\"~ “If you spend more on coffee than on cybersecurity, you will be hacked.” – Richard Clarke\"\"\"), display_id='html_marquee1', update=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-17T09:37:48.194322Z","iopub.execute_input":"2023-07-17T09:37:48.194801Z","iopub.status.idle":"2023-07-17T09:37:50.251636Z","shell.execute_reply.started":"2023-07-17T09:37:48.194766Z","shell.execute_reply":"2023-07-17T09:37:50.250137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Swords and Shields: State of Cybersecurity AI in 2022-2023
\n\n
\n By George Vyshnya (Georgii Vyshnia),
\n CTO @ SBC Group and Aistra\n
\n\n#
Executive Summary
\n \nThe **big war in Europe** that exploded on Feb 24, 2022, with massive invasion of Russian troops on the sovereign non-occupied territories of Ukraine, highlighted the critical importance of cybersecurity, along with its military manifestations in the form of cyberwarfare operations. The context to such a raise of interest is related to the fact that the kinetical invasion of Russian army was accompanied by massive cyber-offensive of Russian state (military) hacker teams as well as covered pro-Russian cybercriminal groups (controlled by Russian intelligence services) on the critical infrastructure systems in Ukraine (see *Appendix C* below for more detailed information on cyber-clashes between Russian cyber-detachments and Ukrainian cyber-defense units, strengthened and empowered by the support from the top-end professionals from Western countries-allies of Ukraine, in the initial weeks of the Russian invasion to non-occupied territories of Ukraine).\n\nIn parallel, at the same time (end of Feb 2022), there has been observed the massive cyber-offensive and cyber-espionage campaigns launched by Chinese state cyber-units. They targeted Ukraine, Russia, Belarus, Poland, and several other European countries (see *Appendix D* below for more details).\nThe successes of Ukraine and its allies to push back on the attackers from both China and Russia as of Feb 2022 set the stage and context for accelerated interest in cybersecurity AI in the industry (see *Appendix C* below for the specific discussion on the role of cybersecurity AI in the mentioned cyber-clashes between Ukraine and Russia). Such a trend continues to go up as of 2023.\n\nThe sound interest in cybersecurity AI also collided with effects that started to manifest prior to Feb 24, 2022.\n\n1. **Rapid digitalization spurred by the Covid-19 pandemic** has resulted in more cybercrime. Malware-as-a-service became a booming business for cyber criminals.\n2. **Wide spread of AI-facilitated attacks on the large groups and political nations** has been observed on the large scale (personalized and targeted AI-facilitated disinformation and ad campaigns during the US President elections in 2016 and Brexit rally in UK; sophisticated AI-driven ransomware attacks, like the famous Petya/NotPetya attack on Ukraine on Jun 27, 2017 – Jun 28, 2017; massive advent of DeepFake-based scams and cybercrimes etc.)\n\nAll the factors above caused the massive contributions and progress of the leading AI labs in business and academia on vibrant topics to improve the state of cybersecurity AI in 2022-2023. Moreover, the venture capitalists and angel investors in the AI industry, along with the industry experts, started to pay more attention to the promising cybersecurity AI products, cloud-based SaaS, and startups (see *Appendix E* below for more information). It is going to be the additional booster to the positive changes in the cybersecurity AI industry.\n\nThe aim of this report is to provide a comprehensive overview of the state-of-the-art with cybersecurity AI, along with the recent development in the field in 2022-2023.\n\nThe report is structured as follows:\n- *Chapter 1* discusses the **major risks associated with the wide adoption of AI in cybersecurity**.\n- *Chapter 2* contains the comprehensive review of the recent innovations in **cyber-defense AI** (wide use of Reinforcement Learning, novel AI methods of malware and intrusion detection, advent of AI-facilitated authentication systems, and addressing the problem of labeled data for AI model training in cybersecurity).\n- *Chapter 3* charters the landscape of the **cyber-attacking AI systems**.\n- *Chapter 4* is focused on by far the most challenging issue in the modern cybersecurity AI – that is, **Adversary ML attacks**.\n- *Chapter 5* outlines the **impact of Generative AI technologies on cybersecurity**.\n- *Chapter 6* outlines the status of **AI cybersecurity of IIoT, smart grids, and autonomous vehicles**.\n- *Chapter 7* articulates the dangers of **Computational Law and automated decision-making in cybersecurity**.\n- *Chapter 8* covers extensive topics related to **eXplainable AI (XAI) in cybersecurity**.\n- *Chapter 9* discusses the challenges and threats to **arms control in AI systems** used in cyberwarfare and cyberweapon scenarios.\n- *Chapter 10* stresses the importance of the **innovative education curricula** to ensure both the regular users and new generations of cybersecurity experts are ready to cope with the novel AI-driven cybersecurity threats.\n- *Chapter 11* concludes on the **status of risk mitigations** with respect to using AI in cybersecurity. It also contains futuristic pieces to outline both the **possible optimistic and pessimistic forecasts** on the future of AI in cybersecurity.\n- *Appendixes* contain the supplementary materials (supplementary notes on DeepFakes as well as the references to the relevant publications inclusive).\n\n
📌 Note: This essay notebook is submmitted under 'Other' category since it discusses the cross-topic matters applicable to a niche industry (that is, AI in cybersecurity).
\n \n#
Chapter 1. Risks of AI in Cybersecurity
\n \nAs mentioned in multiple sources cited across this document, AI has already proved its efficiency in the intellectual Intrusion Detection Systems (IDS), spam filtering, and malware detection.\n\nAmplification of the use of the novel AI algorithms and technologies in cybersecurity within and beyond the existing use cases can have multiple benefits. For instance, it will lead to improving threat detection, response, and prevention. However, it also poses some risks and challenges that need to be addressed. As mentioned in multiple sources [1, 2, 3], some of the dangers of AI in cybersecurity are\n\n- **Lack of transparency and explainability:** AI systems can be difficult to understand and interpret, making it challenging to understand how and why decisions are being made. This could lead to mistrust, errors, or legal issues.\n- **Overreliance on AI:** AI systems are not infallible and may have limitations or flaws. Relying too much on AI could result in complacency, reduced human skills, or missed threats.\n- **Bias and discrimination:** AI systems may reflect or amplify the biases of their data, algorithms, or developers. This could result in unfair or inaccurate outcomes for certain groups or individuals.\n- **Vulnerability to attacks:** AI systems may be targeted by malicious actors who want to compromise, manipulate, or sabotage them (via Adversary ML attacks). This could result in data breaches, misinformation, or damage to the system or its users.\n- **Lack of human oversight:** AI systems may operate autonomously or with minimal human intervention. This could result in ethical, legal, or social issues if the system does not align with human values, norms, or regulations.\n- **Privacy concerns:** AI systems may collect and process vast amounts of data, some of which may be sensitive or personal. This could result in data mishandling, either through intentional breaches or accidental leaks.\n- **Misuse of artificial intelligence:** AI systems may be used for malicious or harmful purposes by adversaries who want to exploit the vulnerabilities in a target system. This could be utilized in cyberattacks, cyberespionage, sabotage, or cyberwarfare. The potential to abuse AI goes hand in hand with its potential to make autonomous decisions.\n- **Assessment difficulties**: difficulty in assessing effectiveness and ensuring correct behavior in terms of threat coverage and in terms of rehearsal vs. actual attack conditions\n \n#
Chapter 2. Novel AI Inventions in Cyber-Defense
\n\nExplosive development of modern AI/DL technologies as well as research on the novel AI algorithms has disruptive effect on the improved cybersecurity in its cyber-defense part.\n\nAI labs both in business and academia contributed significant innovation into AI-facilitated cybersecurity systems in 2022-2023. In the sections below, we are going to review\n- Reinforcement learning (RL) and its impact on revolutionizing the modern cyber-defense\n- Other cyber-defense AI innovations\n\n##
Reinforcement Learning in Cybersecurity AI
\n\nThe real-world cybersecurity threads impose complex challenges on the professional cyberwarriors in the meantime.\n- Threat actors exploit weaknesses in the network and endpoint security in a very coordinated manner to set the stage for complex intrusions and attacks.\n- Such attacks could lead to the entire networks or/and multiple critical nodes within to be knocked down.\n- The novel IIoT systems have been proven to be extremely vulnerable to such coordinated cyber-offensives.\n\nDL/ML-based systems have been good companions in threat detection and protection up to the moment, mostly in the supervised learning setup. However, they cannot be on a par with the speed of plotting the new threats. Regular re-training the ML/DL models for such systems takes some time. It often happens the newly trained models become immediately obsolete as they get behind the real-world vulnerability landscape.\n\nTherefore, reinforcement learning (RL) techniques have manifested enormous potency in developing AI-based solutions for cyber-defense.\n\nRL is the field of ML that deals with sequential decision-making involving an agent which learns the desired action policy (behavior) incrementally while interacting with the environment. RL explores the different states of an environment using an explore-exploit mechanism/policy and hence the RL agent does not require complete knowledge or control of the environment.\n\nDifferent techniques and algorithms under deep reinforcement learning (DRL) have shown great promise in applications ranging from games to industrial processes, where it is claimed to augment systems with general AI capabilities [45]. These algorithms have recently also been used in cybersecurity, especially in threat detection and endpoint protection, where these demonstrate state-of-the-art results. Unlike supervised machines and pure deep learning, deep reinforcement learning is used in more diverse ways and is empowering many innovative applications in the threat defense landscape.\n\n
\n
\n
\n
\n \"DRL\n

Deep reinforcement learning model. Image courtesy to BD Tech Talks.

\n
\n
\n
\n
\n\nThe comprehensive review of the successful DRL implementations has been conveyed by [45]. They indicate that\n- DRL streamlines many unique applications in both intrusion and malware defense scenarios.\n- It enables the applications ranging from the ones that critically assist the development or selection of machine and deep learning algorithms to directly bolstering defenses against advanced Adversarial ML attacks (see more on Adversarial ML attacks in the chapter *“Cyberattacks on AI Systems: Security for AI vs. AI Security”* below).\n- DRL is also used in network anomaly detection as a replacement for a supervised learner, but the more empowering usage of DRL has been to form strategies for defense against modern and advanced Adversarial ML attacks.\n- DRL has also manifested great promise in designing capable defenses against advanced metamorphic malware.\n- DRL itself is a trending research topic in AI and is continually being enhanced with more powerful and efficient algorithms and techniques.\n\n*Deep PackGen* is one of the most recent DRL innovations [46]. It is aimed at improving defenses of the modern network intrusion detection systems (NIDS) from Adversarial ML attacks. *Deep PackGen* employs DRL to generate adversarial packets and aims to overcome the limitations of approaches in the literature [45]. By taking raw malicious network packets as inputs as well as systematically making perturbations on them, *Deep PackGen* camouflages them as benign packets while still maintaining their functionality. In the experiments, using publicly available data, it revealed that\n- Deep PackGen achieved an average adversarial success rate of 66.4% against various ML models and across different attack types.\n- More than 45% of the successful adversarial samples were out-of-distribution packets that evaded the decision boundaries of the classifiers.\n\nThe knowledge gained from the study [46] on the adversary's ability to make specific evasive perturbations to different types of malicious packets can help defenders enhance the robustness of their NIDS against evolving Adversarial ML attacks.\n\nCybersecurity solutions have shown promising performance when detecting ransomware samples that use fixed algorithms and encryption rates. However, due to the current explosion of AI, sooner than later, ransomware (and malware in general) will incorporate AI techniques to adapt its encryption behavior intelligently and dynamically to be undetected. It might ultimately result in ineffective and obsolete cybersecurity solutions. To prove it, *RansomAI*, a RL-based framework that can be integrated into existing ransomware samples to adapt their encryption behavior and stay stealthy while encrypting files, has been proposed in [63]. *RansomAI* presents an agent that learns the best encryption algorithm, rate, and duration that minimizes its detection (using a reward mechanism and a fingerprinting intelligent detection system) while maximizing its damage function. The proposed framework was validated in a ransomware, Ransomware-PoC, that infected a Raspberry Pi 4, acting as a crowdsensor. A pool of experiments with Deep Q-Learning and Isolation Forest (deployed on the agent and detection system, respectively) has demonstrated that *RansomAI* evades the detection of Ransomware-PoC affecting the Raspberry Pi 4 in a few minutes with >90% accuracy.\n\n
🔅 Note: RansomAI is in essence the ‘ethical’ Adversary ML system to be used by the cybersecurity experts to find the appropriate cyber-defenses against the real-world attacks. However, the fact of building such a system with such a high evasion ratio raises the alert on the possible replication of the same approach by the notorious cybercriminals and state-backed cyberwarfare operatives. It must be carefully watched by the cybersecurity experts to plot the timely and effective countermeasures.
\n\nAt the same time, there are certain drawbacks/challenges to adopting RL and DRL solutions in the real-world cybersecurity area.\n\nAs indicated in [48], the technical promises and advances put RL practitioners in a position to design systems that have never existed before and lack prior documentation in law and policy. Public agencies could intervene in complex dynamics that were previously too opaque to deliberate about, and long-held policy ambitions would finally be made tractable.\n\nThere is a great potential to enact RL in the domains of energy infrastructure, social media recommender systems, and transportation [48].\n\nThe advent of RL and DRL systems introduces new forms of risk that exacerbate the harm already generated by standard machine learning tools. The researchers [48] present a new typology of risks arising from RL design choices, falling under four categories:\n- scoping the horizon,\n- defining rewards,\n- pruning information, and\n- training multiple agents.\n\nTherefore, more strict policies toward the regulation of use of DRL systems should be elaborated. As mentioned in [48],\n
\n\"Rather than allowing RL systems to unilaterally reshape human domains, policymakers need new mechanisms for the rule of reason, foreseeability, and interoperability that match the risks these systems pose\".\n
\n\n
🔅 Note: The original definition of rule of reason indicates it to be a legal doctrine used to interpret the Sherman Antitrust Act, one of the cornerstones of United States antitrust law. In the context of [48], Thomas Krendl Gilbert and Co refer to the legal aspects of setting the balance between the advantages (PROS) and threats (CONS) of utilizing RL systems.
\n\n##
Other Innovations in Cybersecurity AI
\n\n**Malware detection.** As mentioned by Mohammed Khatoon in [49],\n\n
\n \"The escalating frequency and intricacy of cyber attacks have prompted the need to detect malware to maintain computer system security. However, conventional signature-based techniques have certain limitations when it comes to detecting evolving and complex threats. In recent years, machine learning (ML) has emerged as a potential solution to detect malware more effectively. ML algorithms possess the ability to analyze voluminous datasets and identify intricate patterns that are not easily discernible by humans.\"\n
\n\nSuch a state of the art in malware detection stimulates the novel approaches in the field to address the limitations.\n\nGood review of the prominent ML/DL-based malware detection algorithms provided in [49]. Furthermore, it discusses future directions in ML-based malware detection such as the integration of multiple ML algorithms and the application of eXplainable AI (XAI) techniques to enhance the interpretability of ML-based detection systems (see more detailed discussion about XAI in cybersecurity in the separate chapter *‘Explainable AI in Cybersecurity’* below).\n\nThe findings of [49] highlight the potential of ML-based techniques in improving the speed and accuracy of malware detection, thus contributing to enhancing cybersecurity.\n\n
\n
\n
\n
\n \"DRL\n

Malware Visualization and Automatic Classification: MalImg dataset. Image courtesy to Nataraj L., Karthikeyan S., Jacob G., Manjunath B. S., Malware Images: Visualization and Automatic Classification.

\n
\n
\n
\n
\n\nThe research on leveraging the novel DL architectures toward malware detection problems within the **Malware Visualization and Automatic Classification** paradigm (converting malware binaries into grayscale images and then passing them through neural networks for classification) has justified that utilizing the novel *CoAtNet* DL architecture [55] can help to improve the accuracy of malware detection [56, 57] compared to the classical CNN-based approaches invented since 2011.\n\nAdditional spin to improved *Malware Visualization and Automatic Classification* paradigm has been done by [58] as researchers have shown that augmenting supervised learning with self-supervised learning can improve performance. They used *Data2Vec*, which was proposed as a modality agnostic self-supervised framework to train neural networks. As a result, *BinImg2Vec* has been presented [58]. It is a framework of training malware binary image classifiers that incorporates both self-supervised learning and supervised learning to produce a model to consistently outperform ones trained only via supervised learning. The researchers were able to achieve a 4% improvement in classification performance and a 0.5% reduction in performance variance over multiple runs. They also demonstrated how their framework produces embeddings that can be well clustered, facilitating model explanability (see more details about eXplainable AI in the chapter *‘Explainable AI in Cybersecurity’* below).\n\n**Detection of attacks.** Another area of cybersecurity AI where tangible results were achieved in 2022-2023 is detection of attacks.\n\nThe novel algorithm tackling the multi-node multi-class classification problem in the setup to early detect cyber-attacks on distributed cyber-physical systems (CPS) has been proposed in [59].\n\nThe Human Immune System can inspire cybersecurity professionals to design an Artificial Immune System (AIS) based Intrusion Detection System (IDS). These biologically inspired algorithms using Self/Nonself and Danger Theory can directly augment IDS designs and implementations. In [60], the researchers included an examination into the elements of design necessary for building an AIS-IDS framework as well as presented an architecture to create such systems.\n\n**AI-assisted authentication.** AI-assisted authentication has become one of the essential cybersecurity research directions recently. It is one of the core technologies to facilitate wide range of authentication scenarios including facial recognition to access buildings, keystroke dynamics to unlock smartphones etc. The state of the art, taxonomy, and future roadmap for AI-assisted authentication has been reviewed in [61].\n\n**Acquiring quality labeled data for ML model training.** The current state of the art in cybersecurity uses ML methods to fight multiple cyberthreats in many use cases (malware detection, network intrusion detection, detecting email anomalies etc.). However, for ML models to be adequate, there is always a problem of how to obtain/generate the training set of labeled instances.\n\nIn the domain of cybersecurity, where security and privacy concerns are the top priorities, high-quality labeled data is difficult to acquire [53, 54]. There are typically many sensors/sniffers creating noisy datasets that are difficult to preprocess and label. Without proper labeling, it is impossible to train supervised ML models as well as detect anomalous activities accurately.\n\nIn addition, training and deploying a ML model in the domain of cybersecurity is hampered by the notorious “imbalanced data” problem [53]. Since malicious activities tend to be rare or covert, it is difficult to construct a set of labeled instances that can be used to train a ML model. Training models with unbalanced sets of instances can generate a high number of false alarms. As a result, human analysts may need to work through many alerts each day to determine which of the flagged instances are true anomalies.\n\nTo address the above-mentioned data labelling challenges, the researchers in [53] proposed to apply Active Learning (AL) techniques to labelling the data in cybersecurity-centric ML problems.\nActive learning (AL) is an interactive learning method intended to improve data labeling and model training pipelines. The AL process usually requires only a small set of labeled data to train an initial learner model, which can then start learning from human feedback. The learner model queries human analysts with the cases that generate the greatest model uncertainty, with the goal of improving prediction/classification performances. In practice, AL may work better with cases that are on, or slightly removed from decision boundaries, depending on factors such as the distributional properties of the instance data.\n\nAL has been widely used for data labeling and has been found to be effective in reducing labeling costs. It can support ML data preparation and help ML models overcome difficult challenges. However, while AL has been tested and applied in a variety of non-expert labeling tasks. The research per [53] demonstrates AL can be successfully implemented in the data labelling tasks where sufficient subject-matter expertise is required (as exemplified by cybersecurity-related data labelling scenarios).\n\nOne more dimension to the availability of better datasets for novel research in cybersecurity is the generous gestures of big companies and academia labs whenever they contribute to the open-source domain by publishing good-quality labeled datasets. One of the recent contributions of this sort was Avast-CTU Public CAPE Dataset [54]. The dataset has been collected in cooperation between Avast Software and Czech Technical University - AI Center (AIC). They made such a sample dataset available to support designing new ML methods for malware detection, especially for automatic detection of generic malicious behavior.\n \n#
Chapter 3. AI in Cyber-Attacks
\n\nUntil recent time, the basic AI use cases in cyber-attacks are basically [3] as follows:\n- It can be used to create malware that can evade detection by antivirus software.\n- It can also be used to create fake social media profiles and spread misinformation on social media platforms.\n- AI is used by the military and intelligence communities to identify specific objects in a photo or video (as a part of OSINT drills).\n\nHowever, with the recent progress in AI algorithms and models (especially with respect to proliferation of various Generative AI technologies), there will be new venues in AI-facilitated cyberattacks:\n- Sophisticated AI-aided phishing attacks and social engineering intrusions (see *‘Generative AI Impact on Cybersecurity’* chapter as well as *Appendix F (‘DeepFakes as a Cybersecurity Threat to Humans’)* below for more details).\n- Personalized disinformation and false news spread campaigns (this topic would not be covered by this essay for brevity reasons).\n- Amplified Adversary ML attacks on AI systems (see *‘Cyberattacks on AI Systems: Security for AI vs. AI Security’* chapter below for more details).\n- Uniting all the above-mentioned in ‘weaponized’ cyberwarfare use cases (see *‘Cybersecurity, Cyberwarfare, and Arms Control for AI’* chapter below for more details).\n- OSINT tools and techniques will be boosted by ingestion/utilizing Generative AI capabilities (see *‘Generative AI Impact on Cybersecurity’* chapter below for more details).\n \n#
Chapter 4. Cyber-Attacks on AI Systems: Security for AI vs. AI Security
\n\nNot too long ago, AI security embraced research and practice of how AI can empower cybersecurity, that is, AI for cyber-defense and cyber-attacks. Since Ian Goodfellow and his team popularized adversarial attacks on ML systems, security for AI became an important concern and part of AI security as well.\nAI technologies are vulnerable to an extensive set of manipulations that can trigger errors, infer private data from training datasets, degrade performance, or disclose model parameters. Researchers have demonstrated major vulnerabilities in numerous AI models, including many that have been deployed in public-facing contexts [23, 24]. As Andrew Moore testified before the U.S. Senate Committee on Armed Services in May 2022, defending AI systems from adversarial attacks is “absolutely the place where the battle’s being fought at the moment” [25].\n\nHowever, AI vulnerabilities may not map straightforwardly onto the traditional definition of a patch-to-fix cybersecurity vulnerability. The differences between AI vulnerabilities and more standard patch-to-fix vulnerabilities have generated ambiguity regarding the status of AI vulnerabilities and AI attacks.\nIt is essential to understand the threats to ML products as well as avoid common pitfalls in AI product development. Ebenezer R. H. P. Isaac and Jim Reno prepared a nice AI Product Security guide for developers, designers, managers, and researchers of AI products [19] to address it.\n\nAI-specific attacks appear in the ML attack surface [19]. These attacks are also collectively known as Adversarial ML attacks.\n- **Poisoning:** modifying a benign training dataset.\n- **Evasion:** give a malicious input to get an unexpected output.\n- **Oracle:** stealing information by probing the model.\n- **Adversarial reprogramming:** use the model for a task it is not intended to do.\n\nThe traditional attack surface exploits vulnerabilities that can be found in any software product, regardless of its specific purpose or application.\n\nTo get a bullet-proof AI product from the security standpoint, the authors of [19] recommend the AI product development teams to follow the best practices below:\n- Understand the implications of AI security and address those gaps in a ML pipeline.\n- A minimal security training programme for non-security professionals in the team.\n- Discuss security requirements and responsibilities as early as possible, preferably before the start of the project engagement.\n- Shift-left security and security by design.\n- Assess the privacy impact of your data before committing it to a model.\n- Be wary of AI-specific attacks like poisoning, evasion, and model/data access.\n- Periodic security audits and testing against latest security vulnerabilities.\n \nAt the same time, Adversarial ML as a cyberthreat to AI system develops with the quick pace now. The fundamental report on cybersecurity [21] outlined the most tangible findings/inventions with respect to Adversarial ML.\n- There is proposed a Generative Adversarial Networks-based approach, that takes original samples and produces adversarial examples to defeat ML malware detectors (classifiers).\n- There is a significant pool of research on the application of Adversarial ML to intrusion and malware detection. All the researchers concluded that adversarial attacks could retrograde the performance of the intrusion and malware classifiers.\n- All the intrusion and malware classifiers studied show similar results on normal data, but most of them got effected on the manipulated data (except neural networks and random forests).\n\nAnother fundamental report on Adversarial ML and cybersecurity [22] embraces two major narratives.\n- It provides a high-level discussion of AI vulnerabilities, including the ways in which they are disanalogous to other types of vulnerabilities, and the current state of affairs regarding information sharing and legal oversight of AI vulnerabilities.\n- It attempts to articulate broad recommendations for addressing the challenges imposed by Adversary ML and the known AI cybersecurity ‘holes’ (going far beyond the AI product security recommendations/best practices conveyed by [19]).\n\n
🔅 Note: Although the discussion itself as well as the recommendations articulated in [22] are sometimes USA-centric, most of them could be applicable to other regions and nations as well.
\n\nThe recommendations [22], categorized under four high-level topics, are as follows:\n- **1. Topic:** Extending Traditional Cybersecurity for AI Vulnerabilities\n- **1.1. Recommendation:** Organizations building or deploying AI models should use a risk management framework that addresses security throughout the AI system life cycle.\n- **1.2. Recommendation:** Adversarial machine learning researchers, cybersecurity practitioners, and AI organizations should actively experiment with extending existing cybersecurity processes to cover AI vulnerabilities.\n- **1.3. Recommendation:** Researchers and practitioners in the field of adversarial machine learning should consult with those addressing AI bias and robustness, as well as other communities with relevant expertise.\n- **2. Topic:** Improving Information Sharing and Organizational Security Mindsets\n- **2.1. Recommendation:** Organizations that deploy AI systems should pursue information sharing arrangements to foster an understanding of the threat.\n- **2.2. Recommendation:** AI deployers should emphasize building a culture of security that is embedded in AI development at every stage of the product life cycle.\n- **2.3. Recommendation:** Developers and deployers of high-risk AI systems must prioritize transparency.\n- **3. Topic:** Clarifying the Legal Status of AI Vulnerabilities\n- **3.1. Recommendation:** U.S. government agencies with authority over cybersecurity should clarify how AI-based security concerns fit into their regulatory structure.\n- **3.2. Recommendation:** There is no need at this time to amend anti-hacking laws to specifically address attacking AI systems.\n- **4. Topic:** Supporting Effective Research to Improve AI Security\n- **4.1. Recommendation:** Adversarial machine learning researchers and cybersecurity practitioners should seek to collaborate more closely than they have in the past.\n- **4.2. Recommendation:** Public efforts to promote AI research should more heavily emphasize AI security, including through funding open-source tooling that can promote more secure AI development.\n- **4.3. Recommendation:** Government policymakers should move beyond standards-writing toward providing test beds or enabling audits for assessing the security of AI models.\n\nA novel approach to protect smart grid systems from Adversarial ML attacks facilitated by GAN-based attacking agents is proposed in [26].\n\nAs mentioned in [2], the upcoming massive leverage of AI to the Adversarial ML (cyber-attacks on the AI systems) is going to change the cybersecurity landscape. The most tangible threats in this respect are forecasted as follows:\n1. Adversaries will use AI to observe cybersecurity defensive decisions and use Deep Learning (DL) network for automatic adversarial attacks against ML defense systems [2].\n2. AI will be used to generate poisoning attacks, targeted at defense AI systems, and poisoning the training data, resulting in inaccurate cyber-defense outputs based on the polluted learning data [2].\n3. Since most of the training data for AI defense algorithms are based on public and open access records on data breaches, adversaries will use the same data, or conduct training data stealing to learn how defense algorithms operate and design AI that can surpass the defense engines [2].\n4. Adversarial AI will create false positive and false negative misclassifications to disguise an actual attack [2].\n5. At the same time, the mathematical modeling and simulations build a solid prediction that AI may have a more limited impact on phishing than some people fear, at least for organizations that are already being targeted [4].\n6. Adversarial AI systems will have a dangerous potential for vulnerability discovery if testing systems can find new vulnerabilities for longer rather than just operate faster [4].\n\nThis in turn will raise the bar on the defense side, with rise of artificial intelligence for IT operations (*AIOps*) through multi-layered tech platforms for continuous integration and deployment (CI/CD) as well as automating and enhancing IT operations via ML analytics [2]. It will also cause the next-generation cybersecurity solutions based on AI algorithms to apply multi-view / multi-modal analytics, using multiple data sources in DL- based setup e.g., deep structured semantic models, followed by multi source approach, and multi-task learning strategies. Such an integrated approach can produce improved risk management and a better understanding of cyber risk maturity and security posture [2].\n\nAs mentioned in [45, 46], applying various deep reinforcement learning (DRL) techniques in the real-world intrusion detection and endpoint defense system will contribute to bolstering defenses against advanced Adversarial ML attacks.\n \n#
Chapter 5. Generative AI Impact on Cybersecurity
\n\nGenerative Artificial Intelligence (*GenAI*) has emerged as a powerful technology capable of autonomously producing highly realistic content in various domains, such as text, images, audio, and videos. In less than 9 years from the first contributions to GenAI field, it has become one of the predominant areas of the AI industry progress [34]. In [32], the industry experts compare the impact of GenAI to the impact of nuclear research.\n\nWith its potential for positive applications in creative arts, content generation, virtual assistants, and data synthesis, *GenAI* has garnered significant attention and adoption. However, the increasing adoption of GenAI raises concerns about its potential misuse in cybersecurity. Ease of applying GenAI tools and utilities for crafting convincing phishing emails, generating disinformation through deepfake videos, and spreading misinformation via authentic-looking social media posts, poses a new set of challenges and risks in the realm of cybersecurity.\n\nTo combat the threats posed by *GenAI*, researchers [27] propose leveraging the *Cyber Kill Chain (CKC)* to understand the lifecycle of cyberattacks, as a foundational model for cyber defense. They aim to provide a comprehensive analysis of the risk areas introduced by the offensive use of GenAI techniques in each phase of the *CKC* framework. They also analyze the strategies employed by threat actors as well as examine their utilization throughout different phases of the *CKC*, highlighting the implications for cyber defense. Additionally, GenAI-enabled defense strategies are proposed that are both attack-aware and adaptive. These strategies encompass various techniques such as detection, deception, and adversarial training, among others, aiming to effectively mitigate the risks posed by GenAI-induced cyber threats.\n\nPre-trained Large Language Models (*LLMs*) are an integral part of the modern GenAI landscape. They have resulted in breakthrough performances in complex AI tasks. Major AI companies with extensive computation capacities and budgets can develop and train these large models with billions and millions of parameters from scratch. Third parties, researchers, and practitioners are increasingly adopting these pre-trained models and fine-tuning them on their private data to accomplish their downstream AI tasks. However, it has been shown that an Adversary ML can extract/reconstruct the exact training samples from these LLMs, which can lead to revealing personally identifiable information. The issue has raised deep concerns about the privacy of LLMs. While the short-term reaction of the large business organizations was issuing rigorous prohibition notes to their employees as for feeding sensitive/confidential/private information to the cloud LLM-backed products like *ChatGPT* or *Google Bard* [47], the leading AI labs in both business and academia initiated a massive research effort recently to find the mitigations for the privacy-driven concerns in the long run.\n\nIn [33], the framework of differential privacy (*DP*) has been proposed. *DP* provides a rigorous approach that allows adding noise in the process of training or fine-tuning *LLMs* such that extracting the training data becomes infeasible (i.e., with a cryptographically small success probability).\nRecent advances in natural language generation (*NLG*) modeling combine dramatic improvements in text quality with unparalleled ease-of-use of *NLG*-backed tools. It brings to the table extensive cybersecurity threats to users in the text-related scenarios (phishing, tool-aided social engineering intrusions etc.) [36].\n\nSuch threats have been extensively assessed in [37]. Their exploration of threat models, when viewed alongside the survey on applied detection techniques, suggests that current domain-specific defenses are not adequate to defend against the vast majority of upcoming threat models. They further conclude that the field of machine-generated text detection has a multitude of open problems that urgently need attention to provide suitable defenses against widely available *NLG* model-backed tools. Existing detection methodologies often do not reflect realistic settings of class imbalance (in labeled text samples used in training of the models vs. real world) or unknown model architectures, nor do they incorporate sufficient transparency and fairness methods to ensure that such detection systems will not themselves cause harm [37].\n\nIn the profound report on *ChatGPT*, the most spoken *LLM*-based chatbot assistant technology launched, recently, disruption and challenges associated with it are discussed [28]. A special section is dedicated to cybersecurity threats originated by it. Hackers are leveraging *ChatGPT*-themed traps to propagate malware throughout Facebook, Instagram, and WhatsApp as demand for generative AI chatbots rises among people across the globe [29]. Meta, Facebook's parent company, released a research paper on the proliferation of malware disguised as *ChatGPT* across its various platforms. The company's security researchers have discovered 10 types of malware employing *ChatGPT* that have attacked customers' devices since March 2023 [28].\n\nResearchers in [30] make a point on *ChatGPT* to endanger the cybersecurity industry. At the same time, another group of researchers [31] proposed a swift and elegant way of deploying *ChatGPT*-based tools to protect email users from scam and phishing.\n\n*LLMs* create new avenues for efficient boosting the current *OSINT* methodologies and toolsets [51].\nAs demonstrated in [39], *LLMs* can unlock incredible potential in the field of the computer agent assistants to the human cybersecurity experts involved in detecting and predicting possible cyberattacks.\n\n*LLMs* such as *OpenAI Codex* are increasingly being used as AI-based coding assistants for programmers. It is critical to understand whether the code generated by such coding assistance ingests or creates the security vulnerabilities to the software applications being built. One of the recent case studies [38] aimed at assessing security risks of the *LLM*-generated programming code in C. Their results indicate that the security impact in this setting (low-level C with pointer and array manipulations) is small: AI-assisted users produce critical security bugs at a rate no greater than 10% more than the control, indicating the use of LLMs does not introduce new security risks.\n\nTo summarize the impact of *GenAI/LLM/NLG* models on the cybersecurity field, it is going to be similar to a double-edged sword, with both favorable and unfavorable consequences to cyber-defense anticipated. Preventing widespread harms from *LLMs/NLG* models will require coordinated effort across technical and social domains, necessitating alignment between AI researchers, cybersecurity professionals, and non-technical experts [37].\n\n
🔅 Note: The threats imposed by using DeepFakes in phishing attacks and disinformation campaigns, along with the possible defense strategies to cope with it, are beyound the scope of this document. In the peer review with Eric Saund, it was noted by him that \"... the topics of disinformation, misinformation, and information warfare are adjacent to technical cybersecurity, but adequate treatment requires discussion of the structure of a target society's information ecosystem and values around free speech. This is a major undertaking and far beyond the scope of what is expected from this essay\".
\n \n#
Chapter 6. AI Cybersecurity of IIoT, Smart Grids, and Autonomous Vehicles
\n\nWith the increased applications of AI in smart cities design, the role of AI is increasingly dominating in smart grids, Industrial Internet of Things (IIoT), intelligent transport systems, and autonomous vehicles.\n\n
\n
\n
\n
\n \"DRL\n

Advancing IIoT cybersecurity with AI. Image courtesy to Innovation News Network.

\n
\n
\n
\n
\n\nHowever, the increased connectivity makes these systems vulnerable to various cyber-attacks that can have catastrophic effects. Such attacks on the endpoint autonomous systems are already on the rise and are expected to become more commonplace in future.\n\nThus, there is a need to strengthen cybersecurity in this field.\n\nIn [41], the major automotive cyber-attacks over the past decade are discussed. The state-of-the-art solutions that leverage artificial intelligence (AI) are presented there as well. The researchers propose a roadmap towards building secure autonomous vehicles. They also highlight key open challenges that need to be addressed:\n- Data protection and privacy.\n- Tamper-proof AI: AI algorithms have shown superior performance in IDS and ADAS subsystems for autonomous vehicles; however, these algorithms are vulnerable to carefully crafted Adversarial ML attacks.\n- Securing automotive integrated circuit supply chains.\n\nSeveral novel AI-based frameworks to build reliable IIoS systems still lack the appropriate cybersecurity evaluation. As mentioned in [42], the cybersecurity aspects of such IIoT systems need more advanced research.\n\nCybersecurity risks, legislation, and challenges toward using AI in medical devices are reviewed in [43]. The researchers discuss both the technical risks of AI in medical devices and the EU legal frameworks to ensure the cybersecurity of such devices (MDR, NIS Directive, Cybersecurity Act, GDPR, the AI Act proposal and the NIS 2 Directive proposal). They indicated the potential pitfalls/limitations in two proposed legislations (the AI Act proposal and the NIS 2 Directive proposal) to affect the cybersecurity of the medical devices, if not addressed them promptly.\n\nTrustworthy AI framework for proactive detection and risk explanation of cyber-attacks in smart grids has been proposed in [44]. As a part of the effort, they performed the experiment with a state-of-the-art dataset to validate such a framework. The experiment established the proposed framework as a trustworthy AI by fulfilling the capabilities of reliability, fairness, explainability, transparency, reproducibility, and accountability.\n\nOverall, the AI cybersecurity risks in IIoT, smart grids, autonomous vehicles and other components of the interconnected smart city infrastructure are vibrant and not properly addressed by the current technical and social frameworks. The ability to catch up on it will be one of the key success factors to broader implementation of the smart city innovations.\n \n#
Chapter 7. The Dangers of Computational Law and Cybersecurity
\n\n[Gottfried Leibniz](https://en.wikipedia.org/wiki/Gottfried_Wilhelm_Leibniz) worked on many things. However, a theme that recurred throughout his career was the idea of turning the human law into an exercise in computation. As we all know, the Old Gottfried failed in this area. Nevertheless, three centuies lates, the humans gave it another try, with a bunch of systems to facilitate **Computational Law** implemented.\n\n
\n
\n
\n
\n \"DRL\n

Computational Law Metaphor. Image courtesy to Michael Genesereth/CodeX.

\n
\n
\n
\n
\n\nAs the [definition](https://en.wikipedia.org/wiki/Computational_law) states, \n
\n Computational Law is the branch of legal informatics concerned with the automation of legal reasoning. What distinguishes Computational Law systems from other instances of legal technology is their autonomy, i.e. the ability to answer legal questions without additional input from human legal experts.\n
\n\nWhile there are many possible applications of Computational Law, the primary focus of work in the field today is compliance management, i.e. the development and deployment of computer systems capable of assessing, facilitating, or enforcing compliance with rules and regulations. \n\nSome systems of this sort already exist, and they often employ AI capabilities. As an example, you can refer to the software packages to automate and file American income tax returns (*TurboTax* by Intuit, *H&R Block Tax Software* by H&R Block, and *TaxAct* by TaxAct Holdings, Inc.).\n\nAt the same time, the wide adoption of such systems exposes the end users to the severe potential risks.\n\n- Computational Law must seriously consider that not only does it face the same risks as other types of software and computer systems, but that failures within it may cause financial or physical damage, as well as injustice. \n- The consequences of Computational Legal systems failing are greater than if they were merely software and hardware.\n\nIn [6], the research group\n- Discusses how cybersecurity can address the above-mentioned flaws and weaknesses.\n- Goes through known issues in Computational Law as well as discusses them at various levels, from design to the physical realm.\n- Indicates the critical importance of adversarial problems related to the modern ML/DL systems (of which data poisoning is one of the classic examples).\n- Makes certain considerations regarding the Computational Law and bounding implications for the current and future legislations.\n\nThis team of researchers also presented their recommendations which are necessary for Computational Law to function globally, and which follow ideas in safety and security engineering [6]\n- Accommodation of threats.\n- Adequate use of AI systems.\n- Humans remaining in the center of AI system deployment.\n\nFinally, there is a hope in EU's proposed AI Act, which makes an important attempt at taking the specific problems which Computational Law bring into the legal sphere [6]. If it flies well, similar legal experience can be leveraged in other countries and regions across the globe.\n\nIn the US, similar concerns have been raised. As mentioned in the fundamental cybersecurity report [22], it is one of the critical tasks to clarify the legal status of AI vulnerabilities now. The researchers state the adversary cyber-attacks on AI systems should be handled within the legal framework established by the Federal Computer Fraud and Abuse Act.\n\n#
Chapter 8. Explainable AI in Cybersecurity
\n\nAs mentioned above, lack of transparency and explainability is one of the key risks associated with the widespread adoption of AI systems in highly sensitive areas like cybersecurity (as well as in medicine, autonomous vehicle driving, smart cities etc.). On many occasions, it is very challenging to understand the decision and bias to control and trust systems' unexpected or seemingly unpredictable outputs. That’s why the research in the eXplainable AI (XAI) area has recently become mainstream as of 2023.\n\nIn [7, 8, 9], the comprehensive review of the state of the art in XAI for cybersecurity in network systems has been provided. While researchers in [8, 9] are mostly focused on the literature review and publicly available information on the outcomes of the targeted XAI research projects, the group of the contributors to [7] explored the various approaches and XAI technologies (SHAP, LIME, DALEX, GBDT, xAI-GAN, RelEx, NAM etc.) that have been proposed to address such an important problem. The review follows a systematic classification of network-driven cybersecurity threats and issues (in particular, they mapped the known XAI techniques and applications to the classes of the cybersecurity threats as well as highlighted what criteria of an ideal XAI system are met by each of them).\n\nThe extensive review of the progress and outcomes of the targeted research projects of XAI across various types of cyberattacks and relevant applications (FeatureCloud, SPATIAL, Blue Hexagon-Hexnet, and LEMNA) has been put together in [9].\n\nThe group of authors of [7] formulates the future directions for the research in the cybersecurity XAI\n- Mitigating the risk of machine interpretation in cybersecurity\n- Establishing transparency of XAI methods in cybersecurity\n- Uncertainty handling for XAI in cybersecurity\n- Building secure and efficient algorithms for distributed XAI training\n- Identifying cybersecurity-specific XAI evaluation metrics\n\nWhen it comes to identifying the most useable known XAI approaches, it is highlighted that SHAP and LIME techniques are the most used, perhaps because they have been implemented in open-source frameworks for some time [8]. LEMNA appears to be a promising technique, developed with cybersecurity use cases in mind [8].\n\nIn [10], a novel approach to identifying relevant features of the input data has been proposed, based on the inspirations from the methods from the energy landscapes field, developed in the physical sciences. By identifying conserved weights within groups of minima of the loss landscapes, it is possible to identify the drivers of model decision making. Analogues to this idea exist in the molecular sciences, where coordinate invariants or order parameters are employed to identify critical features of a molecule. However, no such approach existed for machine learning loss landscapes before the contribution in [10]. The authors demonstrated the applicability of energy landscape methods to machine learning models by examples, both synthetic and from the real world.\n\nAs indicated in [12], designing appropriately fast and accurate XAI is still challenging, especially in numerical applications. Therefore, they propose a universal XAI model named Transparency Relying Upon Statistical Theory (TRUST), which is model-agnostic, high-performing, and suitable for numerical applications [12]. Simply put, TRUST XAI models the statistical behavior of the AI's outputs in an AI-based system. Factor analysis is then used to transform the input features into a new set of latent variables. The mutual information is used to rank these variables and pick only the most influential ones on the AI's outputs and call them \"representatives\" of the classes. At the end, multi-modal Gaussian distributions are used to determine the likelihood of any new sample belonging to each class. The effectiveness of TRUST has been demonstrated in a case study on cybersecurity of the industrial Internet of things (IIoT) using three different cybersecurity datasets (IIoT is a prominent application that deals with numerical data). The results show that TRUST XAI provides explanations for new random samples with an average success rate of 98%. Compared with LIME, a popular XAI model, TRUST is shown to be superior in the context of performance, speed, and the method of explainability [12].\n\nAs mentioned in [7], it is quite popular (while also productive) to build XAI systems/frameworks for every class of cybersecurity threats/cyberattacks as well as for every major type of cybersecurity applications. The recent developments in 2022-2023 brought new inventions of this sort into the public domain. The good review of the recent achievements and approaches to build explainable Intrusion Detection Systems (X-IDS) has been conveyed by [11]. A secure and trusted platform for XAI for Smart City applications has been discussed and defined in [13].\n\nThe framework to XAI for Graph Neural Networks (GNNs) to enhance trust management by exploring combining symbolic and sub-symbolic methods in cybersecurity (with domain knowledge incorporated) has been proposed in [62]. It is an essential contribution since ML on graph-structured data (GNNs, in particular) has recently received deepened interest in the context of intrusion detection in the cybersecurity domain. At the same time, similar to other connectionist models, GNNs generally lacked transparency in their decision making. With the contributions in [62], such an explainability gap has been filled, with road blockers to wider adoption of GNNs in building the cutting-edge intrusion detection systems (IDS) dismissed.\n\nThe LLM and Generative AI disruption of 2023 has affected the field of XAI in cybersecurity, like many other verticals and AI use cases. *ChatIDS*, the novel approach to explain alerts Intrusion Detection Systems (IDS) to non-experts by using large language models has been prototyped and described in [14]. The feasibility of *ChatIDS* has been evaluated by using *ChatGPT*. It has been demonstrated that *ChatIDS* has the potential to increase network security by proposing meaningful security measures in an intuitive language from IDS alerts. Nevertheless, some potential issues in areas such as trust, privacy, ethics, etc. remain vibrant. They need to be resolved, before *ChatIDS* might be put into practice.\n\nAs for the current challenges, open issues, and limitations to XAI in cybersecurity, they are chartered [9] as follows:\n- Security attacks on XAI-integrated cybersecurity frameworks\n- Managing balance between security and usability of XAI-integrated cybersecurity systems\n- Legal compliance\n- Performance of XAI algorithms\n- Privacy and Data protection issues\n \n
🔅 Note: Most of the contributions and new concepts in XAI described in the paragraphs above are applicable to AI systems beyond the cybersecurity field as well.
\n \n#
Chapter 9. Cybersecurity, Cyberwarfare, and Arms Control for AI
\n\nDue to AI systems’ ability to facilitate automated decision-making, it is now on the mainstream to ‘weaponize’ AI use in several cyberwarfare and kinetic warfare scenarios. Therefore, cybersecurity of AI systems becomes a matter of human safety to the major extent.\n\nSome of the experts and activists have already voiced concerns about the dangers of AI-enabled weapons systems. This raises the question of how feasible it will be to control military use of AI. Megan Lamberth and Paul Scharre [35] look at a number of characteristics that make AI difficult to control. They highlight three major reasons for AI in the warfare scenarios to be difficult to control:\n- It is a general-purpose technology.\n- It is an emerging technology.\n- Verifying the compliance of any AI-related agreement will pose unique challenges.\n\nThe researchers [35] also lay out some concrete steps that could be taken today to increase the likelihood that future AI arms control regimes will be successful. However, they indicate the geopolitical vector to be predominant at establishing arm control over AI. It can only be possible if one of the two scenarios gets implemented:\n- **Consensus.** All geopolitical ‘poles of power’ (in the currently developed tensions between (1) USA and its allies across the globe, and (2) China and its satellites) come to an agreement to follow the same AI ethics [50] and AI risk management frameworks (see Appendix B below) as well as equally pose limitations on the ‘weaponizing’ their AI systems.\n- **Keeping U.S. companies dominant in global IT.** It will ensure that the U.S. government retains the ability to control access to compute resources in the future; in particular, the U.S. strategy needs to keep Chinese firms dependent on compute resources that use U.S. technology.\n \n#
Chapter 10. Cybersecurity Education in the Age of AI
\n\nWhen I published my discussion topic on *Generative AI timeline* [34], one of the discussion comments highlighted the disappointment from the college/university education not to provide a comprehensive training on the recent *GenAI* inventions of the last nine years. It displays yet one more proof of a deep gap between business and academia in the current AI industry. Such a gap has a solid exposure on the regular human users in terms of their *‘illiteracy’* on the broad range of cybersecurity threats. In turn, modern AI-driven cyberthreats like DeepFakes or personalized disinformation campaigns hit the human brains directly, in an uncontrolled manner. Personal ‘cyber-hygiene’, along with the educated personal responses to the cyberthreats surfaced, becomes the critical part of coping with the cybercrime and cyberwarfare attacks across the national borders.\n\nAnother educational problem is that the aspiring cybersecurity professionals may not possess enough knowledge and skills to react to novel AI-originated threats, especially ones driven by Adversary ML technologies.\n\nThe good attempt to liquidate such a gap has been committed by the researchers in [40]. They developed the curriculum of the brand-new course, **“AI security threats against pervasive robotic systems\"**.\n\nThe topics of this course include:\n
\n
    \n
  1. Introduction, examples of attacks, and motivation.
    2. \n
  2. Robotic AI attack surfaces and penetration testing.
    3. \n
  3. Attack patterns and security strategies for input sensors.
    4. \n
  4. Training attacks and associated security strategies.
    5. \n
  5. Inference attacks and associated security strategies.
    6. \n
  6. Actuator attacks and associated security strategies.
    7. \n
  7. Ethics of AI, robotics, and cybersecurity.
    8. \n
\n
\n\n
🔅 Notes: \n \n
\n \n#
Chapter 11. Future of AI in Cybersecurity
\n\nIn the sections of this chapter, we are going to share the assessment of the cybersecurity AI risk mitigation as of the end of Q2 2023. \n\nWe will further expand the risk mitigation discussion with drawing two forecast scenarios as follows\n- Optimistic scenario forecast\n- Pessimistic scenario forecast\n\nAt this point, the implementation of any of the scenarios above is determined. The joint cross-disciplinary efforts of technical mavens in ML/DL/AI, policy makers, legislative practitioners, social psychologists, and other social-area experts can make a difference.\n\n##
Assessment of Risks of Utilizing AI in Cybersecurity
\n \nThe table below summarizes the overall assessment of the identified risks in cybersecurity AI:\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Risk DescriptionStatusComments
Lack of transparency and explainability: AI systems can be difficult to understand and interpret, making it challenging to understand how and why decisions are being made. This could lead to mistrust, errors, or legal issues.✅ Mitigation: on track

✔ Contingency: not required		Significant breakthroughs and achievements in eXplainable AI (XAI) for cybersecurity are going to fix the root causes to trigger such a risk. Therefore, the good mitigation of this risk is currently being implemented.
Overreliance on AI: AI systems are not infallible and may have limitations or flaws. Relying too much on AI could result in complacency, reduced human skills, or missed threats.✅ Mitigation: on track

✔ Contingency: not required		The implementation of the holistic approach to AI in cybersecurity (where AI systems are basically advisories to the human experts in the field) becomes the main avenue.
Bias and discrimination: AI systems may reflect or amplify the biases of their data, algorithms, or developers. This could result in unfair or inaccurate outcomes for certain groups or individuals🚩 Mitigation: poor

🚩 Contingency: poor		There are research-and-development efforts in progress to fix the bias of existing and newly trained AI systems in cybersecurity. However, they are yet to bring the satisfactory results.
Vulnerability to attacks: AI systems may be targeted by malicious actors who want to compromise, manipulate, or sabotage them (via Adversary ML attacks). This could result in data breaches, misinformation, or damage to the system or its users.🚩 Mitigation: poor

🔅 Contingency: not applicable		The problem with Adversary ML attacks does not have a reliable short-term resolution now. There are several research-and-development efforts in progress that bring the promise of improving the situation in the middle-to-long run.
Lack of human oversight: AI systems may operate autonomously or with minimal human intervention. This could result in ethical, legal, or social issues if the system does not align with human values, norms, or regulations.✅ Mitigation: on track

✔ Contingency: not required		The cumulative efforts with establishing appropriate safety, security, legislation, and ethics frameworks to control the modern AI systems are on the right track.

The inclination to keep the holistic approach to using AI in cybersecurity (with less space for autonomous decisions and more advisory role to the human experts) is going to mitigate this risk in the middle run.
Privacy concerns: AI systems may collect and process vast amounts of data, some of which may be sensitive or personal. This could result in data mishandling, either through intentional breaches or accidental leaks.🚩 Mitigation: poor

🚩 Contingency: poor		There are promising research-and-development efforts in progress to establish data privacy-centric AI system training frameworks in cybersecurity. However, they are in the initial stages of their development now. We can expect some benefits from them in the middle-to-long run only.
Misuse of artificial intelligence: AI systems may be used for malicious or harmful purposes by adversaries who want to exploit the vulnerabilities in a target system. This could be utilized in cyberattacks, cyberespionage, sabotage, or cyberwarfare. The potential to abuse AI goes hand in hand with its potential to make autonomous decisions.⚠ Mitigation: in progress, less then sufficient

🚩 Contingency: poor		Appropriate rigorous legislations are developed to be applied to cybercrimes, in general, and particularly AI-related cybercrimes. However, there is a lack of precedence/methodology of applying them in the systematic way now.

💥 On top of that, what is called the misuse of AI in the civic context is most likely going to remain a valid authorized (yet strictly controlled) use case in cyberwarfare, military, and state-backed cybersecurity operations.
Assessment difficulties: difficulty in assessing effectiveness and ensuring correct behavior in terms of threat coverage and in terms of rehearsal vs. actual attack conditions⚠ Mitigation: in progress, less then sufficient

🚩 Contingency: poor		Significant breakthroughs and achievements in eXplainable AI (XAI) for cybersecurity bring the promise to properly assess the threat coverage (based on the threts descriptions, presumably). Assessment of rehearsal vs. actual attack conditions has not been tackled as of the moment.
\n\n
💥 Warning: As we can see, there are some of the severe risks not mitigated or otherwise addressed still. It highlights the respective cross-disciplinary effort should be urgently put in place to find the appropriate resolutions.
\n \n##
Optimistic Scenario Forecast
\n \nThe optimistic scenario assumes to see the joint cross-disciplinary efforts of technical mavens in ML/DL/AI, policy makers, legislative practitioners, social psychologists, and other social-area experts to solidly establish sustainable cybersecurity AI development in the upcoming years, with the profound AI risk mitigation / contingency strategies identified and implemented on the global scale.\n \nSuch an optimistic picture will be in fact the mosaic of the multitude of individual cybersecurity AI achievements and breakthroughs. Some of such fundamental game changers are listed below:\n \n1. The new wave of research and prototyping will lead to building a pre-emptive AI-driven solution to defend from the adversary ML/AI risks and threats. In particular, [2] has already proposed the high-level architecture and the conceptual framework for such a preventative AI-driven cyber-defense system, with real-time intelligence, and dynamic risk analytics to be the integral parts of it.\n2. We will observe the major progress with implementing the holistic approach to AI in cybersecurity (aka Hybrid Augmented Intelligence) where AI-powered cognitive technologies are an essential part of the operational process in which the human element guides the process and plays a pivotal role [3, 21]. Such a human-AI collaboration can significantly increase the performance of a system over a system that only utilizes either AI processes or human cybersecurity experts.\n3. The amazing experiment in the quantitative modeling the impact of AI technological advances predicted certain AI-backed gains in the cyber-defense area [4]. They prove that there is much more opportunity for automated or accelerated patch deployment (to curate/fix the cyberthreats) than there is for automated patch development.\n4. The resurgence of AI has industry leaders counting the days until quantum computers go mainstream. There’s been considerable progress on quantum computing as of Q2 2023. The combination of AI technologies and quantum computing platforms brings the promise of delivering an exponential advantage for certain classes of problems in cybersecurity [5]. At the same time, it is forecasted to see it posing the most unexpected threats to cybersecurity [5].\n5. XAI in cybersecurity (reviewed in more details in the chapter ‘Explainable AI in Cybersecurity’ above) would become the must-have part of the ML pipelines in the cybersecurity AI systems [21].\n6. The major global, regional, and national-level stakeholders will come to an agreement as for the legal (see chapter ‘The Dangers of Computational Law and Cybersecurity’ above) and AI ethical frameworks [50] to ensure sustainable cybersecurity AI development without compromising/degrading the cybersecurity domain.\n7. AI risk management frameworks (see Appendix B) as well AI product security/safety guidelines will be embedded into DNA of the new AI systems from the very beginning of their development.\n8. Arms control over AI in the military use cases (cyberwarfare inclusive) would be properly established (as discussed in *‘Cybersecurity, Cyberwarfare, and Arms Control for AI’* chapter above).\n9. We will see more synergy between the leading AI labs in business and academia in the research and development efforts to address the critical cybersecurity AI risks in the next 2-3 years.\n \n##
Pessimistic Scenario Forecast
\n\nThe negative scenario forecast is very simple yet sharp. It is as simple as the statement, **“Cybersecurity will fail due to the wide adoption of AI”**. In *“weaponized” cyberwarfare world*, it will necessarily lead not only to cybercrime soaring across the globe but to more severe cyberwar clashes between the geopolitical opponents.\n\nThere are some arguments in support of it, as mentioned in [2]. The paragraphs below are actually quoting it,\n\n
🏴 Pessimistic arguments by Petar Radanliev, David De Roure, Carsten Maple, and Uchenna Ani:
\n\"1. New cybersecurity solutions based on AI algorithms are developed as isolated systems in industry and in academia. Academic researchers use old datasets to develop very specialized algorithmic solutions, which perform with great precision in testing environments, but fail in operational environments, because the threats have evolved since the data was collected. With industry, the reason for failure is the complete opposite. Organizations have vast amounts of data but use old algorithms, which are not trained to detect new and emerging cyberthreats.

\n2. Since academia is leading the research on AI solutions for cybersecurity, the performance analysis is conducted mostly on single data sets. These solutions would have been much stronger if the AI solution performance were tested on multiple data sets simultaneously, which can only happen in industry settings (because of data availability). Adversarial ML/AI algorithms are not bound by ethical considerations and data privacy regulations. Since these data privacy regulations are only applicable to the training data in defense algorithms, and if alternatives are not found very soon, the defense algorithms will likely lose to adversarial algorithms – that are trained for practical applications.

\n3. Defense algorithms that are designed as very specialized solutions, are almost never deployed and used for other purposes and lack transparency. This lack of model sharing and lack of interoperability results with a lot of work and effort invested in a defense algorithm that will probably never be adapted and reused for a different purpose. The key for continuous improvement is in sharing information and details on the type of algorithm, this will enable existing algorithms to be adapted and quickly and timely developed into new and relevant algorithms that can be deployed in a different (across) environment. Since adversarial ML algorithms are commonly shared (for a price), it is likely that adversarial algorithms will have the upper hand in this area of interoperability and applications across different environments.

\n4. The adoption of AI solutions for cybersecurity is still very costly, and almost unaffordable for small and medium sized organizations. These solutions are highly specialized and used to perform specific functions at speed and at scale. Hence, even if the cost were reduced (e.g., direct subsidies), the solutions might not be that relevant to small and medium sized companies. This would create advantages for adversaries, who could target these companies at scale – with AI driven cyberattacks.\"
","metadata":{}},{"cell_type":"markdown","source":"#
Appendix A. Cybersecurity Glossary and Fundamentals.
\n\nIf you like to jump on the key concepts and terms in cybersecurity (beyond the AI use cases in the area), below are some good resources and references to review.\n\n- One of the initial parts of [3] (namely, Chapter 2) contains the good refresher on Cybersecurity concepts, cyberattacks and cyberthreats.\n- The initial chapter of [8] (Introduction) provides the comprehensive overview of cyber risk concepts, along with the fundamentals of cybersecurity.","metadata":{}},{"cell_type":"markdown","source":"#
Appendix B. Cybersecurity AI Risk Management Frameworks.
\n\nThere are both industry- and government-driven frameworks and best practices that can help organizations to assess and mitigate the risks of AI in security [15, 16 and similar]. However, such frameworks are fresh and high-level as of this moment. Considerable efforts will be required to tailor their recommendations to the practical AI system use cases in real-world organizations.","metadata":{}},{"cell_type":"markdown","source":"#
Appendix C. Cyberwarfare frontiers of the war in Ukraine
\n\n
\n
\n
\n
\n \"...\"\n

Russia as a global cyber-offensive threat. Image courtesy to Microsoft.

\n
\n
\n
\n
\n\nRussian cyber-attacks against Ukraine have persisted ever since Russia's illegal annexation of Crimea in 2014, intensifying just before the 2022 invasion.\n\nDuring the prelude to the 2022 Russian invasion of Ukraine and the 2022 Russian invasion of Ukraine, multiple cyberattacks against Ukraine were recorded, as well as some attacks on Russia. The first major cyberattack took place on 14 January 2022, and took down more than a dozen of Ukraine's government websites. According to Ukrainian officials, around 70 government websites, including the Ministry of Foreign Affairs, the Cabinet of Ministers, and the Security and Defense Council, were attacked. Most of the sites were restored within hours of the attack. On 15 February 2022, another cyberattack took down multiple government and bank services. Ukraine's energy, media, financial, business, and non-profit sectors have suffered as of that time as well.\n\n
\n
\n
\n
\n \"...\"\n

Timeline of Russia's state-backed phishing campaigns in 2021-2022. Image courtesy to Google.

\n
\n
\n
\n
\n\nOn 24 February 2022, Russia launched a full-scale invasion of Ukraine. Western intelligence officials believed that this would be accompanied by a major cyberattack against Ukrainian infrastructure, but this threat did not materialize due to coordinated efforts of the several international government and cybersecurity business organizations (cyber-units of the intelligence agencies of USA, UK, and several EU nations, Microsoft, ESET, Cisco Tales etc.) that prevented the malicious activities of Russian state hackers as well as pro-Russian cybercriminal groups at that time.\n\nSince Feb 24, 2022, limited Russian cyber-attacks have undermined the distribution of medicines, food, and relief supplies. Their impact has ranged from preventing access to basic services to data theft and disinformation, including through deep fake technology. Other malicious cyber-activity involves sending of phishing emails, distributed denial-of-service attacks, and use of data-wiper malware, backdoors, surveillance software and information stealers.\n\nOrganizations and governments around the world have not been indifferent to the hybrid risks thus posed. EU-, US- and NATO-led initiatives have been carried out with the aim of neutralizing cyber-threats and protecting essential infrastructure. As part of these initiatives, the EU has activated its Cyber Rapid Response Teams (a project under Permanent Structured Cooperation (PESCO) in security and defense policy), to support Ukraine's cyber-defense.\n\nNon-government and private players have supported Ukraine through various cyber-resilience activities. Since the beginning of the invasion, a significant number of counterattacks have been launched by independent hackers, affecting the Russian state, security, banking, and media systems. The European Parliament has called for stepping up cybersecurity assistance to Ukraine and for making full use of the EU's cyber-sanctions regimes against individuals, entities, and bodies responsible for or involved in the various cyber-attacks targeting Ukraine.\n\nYou can refer to the resources below to see more details on Russian cyber-offensive as well as the defenses and counterattacks of Ukrainian cyber-units and their international allies.\n\n
🧰 Information: If you like to delve more into the timeline of Russian cyber-offensive on Ukraine in 2022-2023, along with the successes of Ukraine and its allies at defending/counter-attacking the cyber-agressor, you can review the resources below:\n \n
\n\n
🔅 Note: If you like to review the record tracks on cyberwarfare activities of Russia prior to 2022, whether related to the attacks on Ukraine or other nations, you can start your research by navigating to the links below\n \n
\n\n
🔅 Note 2: Russia demonstrated the actual integration of the pro-Russian cybercriminal groups across the globe in its state-backed cyberwarfare agenda. The article per The Servers Of The Largest Russian Darknet Market Hydra Were Confiscated In Germany. 543 Bitcoins Worth € 23 Million Were Also Seized provides a great example on how the police operation in Germany disrupted the pro-Russian hackers right after the Russsian invasion to Ukraine started.\n
","metadata":{}},{"cell_type":"markdown","source":"There is also a documentary film (Mar 9, 2023) about people behind **Cisco Talos’** efforts to protect Ukrainian critical infrastructure. In particular, it discusses the personal impact on the teams involved.","metadata":{}},{"cell_type":"code","source":"from IPython.display import IFrame, HTML \n# you only change video ID : 'TNzDMOg_zsw' , and width and height as you want\n\n\ndisplay(HTML(''))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-17T09:37:50.254826Z","iopub.execute_input":"2023-07-17T09:37:50.255791Z","iopub.status.idle":"2023-07-17T09:37:50.269448Z","shell.execute_reply.started":"2023-07-17T09:37:50.255732Z","shell.execute_reply":"2023-07-17T09:37:50.268006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"AI technologies are being actively used in the current Russia-Ukraine cyberwarfare. More information is provided in the articles referred below:\n\n
💥 References:\n \n \n
","metadata":{}},{"cell_type":"markdown","source":"#
Appendix D. Cyberwarfare operations of China
\n\n
\n
\n
\n
\n \"...\"\n

China's cyber threats. Image courtesy to Getty Images, via BBC.

\n
\n
\n
\n
\n\nAs noted in [Mystery of alleged Chinese hack on eve of Ukraine invasion](https://www.bbc.com/news/technology-60983346), some intelligence sources both in Western countries and Ukraine are quoted to tell that\n\n
\n \n
\n\nBased on the assessment above, it can be concluded the goal of Chinese attackers looked to have been espionage (stealing secrets) rather than the kind of a sabotage operations coordinated with Russia. Russia was accused of carrying out such sabotage operations just before the invasion, and when it started on Feb 24, 2023 (see *Apendix C* above). It can be also suspected Chinese actors had some insider information about the timeline of the full-scale invasion of Russia to play their own game in parallel to the sabotagious attacks from Russia.\n\nThe additional information about cyber-offensive traffic originated by Chinese actors in Feb 2022 could be grasped from the articles below\n\n
💥 References:\n \n

\n\nOn a generic note, the above-mentioned operations in Feb 2022 are well suited into cyberwarfare strategy of the government of China, as mentioned in CHINA’S CYBERATTACK STRATEGY EXPLAINED.
\n\nMilitary OSINT to learn more about the American military has recently become one of the strategic areas of China's cyberwarfare operations, as displayed in the report. As manifested by this report, AI/ML-based systems and tools are actively used by China’s military and defense industries to extract intelligence from various international information sources.\n\nAdditionally, some Western researches envision military and security threats from the boosted AI research in China. As indicated in [64],\n
\n \"The China-US AI research gap has continued to widen, with Chinese institutions producing 4.5 times as many papers than American institutions since 2010, and significantly more than the US, India, UK, and Germany combined. Moreover, China is significantly leading in areas with implications for security and geopolitics, such as surveillance, autonomy, scene understanding, and object detection.\"\n
\n\nMore details about the entire landscape and context of Chinese cyber-offensive operations, along with utilizing ML/AI technologies in the cyberwarfare operations, are available to review in the articles and reports below\n\n
💥 References:\n \n
","metadata":{}},{"cell_type":"markdown","source":"#
Appendix E. AI Industry Shift to Cybersecurity: Forbes AI50 List Insights
\n\n[Forbes AI50 List](https://www.forbes.com/lists/ai50/?sh=58b016cb290f) represents the most tangible AI companies in Western world (USA and its alies). Forbes experts that collate it select the most trending companies there. Sum of the investments in the AI companies is one of the attributes in the AI50 List to reflect the importance (and success, to an extent) of the mission of the companies listed.\n\nAs such, it can be a good proxy to assess the trends in the AI industry that are recognized by both the AI experts and VC/Angel investors.\n\n
🔅 Note: There is no any non-AI companies mentioned in this list. All listed companies contribute to building AI technologies in one of the industries (categories) defined by Forbes experts.\n
\n\nOne of the key focuses of Forbes AI50 List 2023 is the AI in cybersecurity. The respective charts and tables below are going to prove it with the data-driven insights.\n\nFirst of all, we can see Cybersecurity AI to become one of the major industry focuses in 2023 by reviewing the summary chart on AI50 List company counts by industry:","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\nimport seaborn as sns\n\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport plotly.express as px\nimport plotly.offline\n\ndef funding_to_numeric(financial_string):\n res = 0\n x = financial_string.replace('$', '')\n tokens = x.split(' ')\n num_val = float(tokens[0])\n multiplier = 0\n if tokens[1] == 'M':\n multiplier = 1000000\n else:\n multiplier = 1000000000\n res = int(num_val * multiplier)\n return res\n\ndef is_generative_ai(mission):\n res = 'No'\n \n generative_ai_technologies = [\n 'Chatbot application', \n 'Video and podcast editing',\n 'Open-source AI library',\n 'Photo editing',\n 'Copyrighting software', \n 'AI image generator',\n 'AI model training tools', \n 'AI model developer',\n 'Voice chatbots',\n 'Image and video editing',\n 'Synthetic video creation',\n 'Presentation creation software',\n 'Personalized search engine', \n ]\n \n if mission in generative_ai_technologies:\n res = 'Yes'\n \n return res\n\ndef fix_industry_2022_typos(industry):\n res = industry\n if res == 'Pharmacutical':\n res = 'Pharmaceutical'\n return res\n\ndef set_industry(mission_2023):\n res = 'N/A'\n \n media_entertainment = [ \n 'Video and podcast editing', \n 'Photo editing', \n 'Personalized search engine', \n 'Image and video editing',\n 'Synthetic video creation',\n 'Presentation creation software', ]\n \n consumer_tech = ['Voice analysis software']\n \n education = ['Space simulation software',]\n \n env_energy = ['Forestry satellite data analysis',]\n \n retail = ['Cashierless retail checkout',]\n \n customer_service = ['Chatbot application', 'Voice chatbots',]\n \n emp_support = ['Automated IT support', 'Digital assistant for lawyers', \n 'Legal contract management', 'Recruiting software',]\n \n agriculture = ['Weeding tractors for farming', ]\n \n cybersecurity_missions = ['Defense intelligence software',\n 'Defense software',\n 'Email cyberattack detection',\n 'Autonomous defense software',\n 'Cyberattack detection',\n ]\n \n sales_marketing_missions = ['Market intelligence search',\n 'Sales software',\n 'Copyrighting software',\n ]\n \n ds_missions = ['Data issue detection', \n 'Data storage and analytics', 'AI model developer', 'Data labeling software', \n ]\n \n ai_infrastructure = ['AI model training tools',\n 'Open-source AI library',\n 'Developer tools for AI',\n 'Data labeling provider', \n 'Internal workplace search',\n 'AI image generator',\n 'Personalized search engine', \n ]\n \n pharma = ['Clinical trial forecasting', \n 'Drug discovery', 'Drug discovery and diagnostics',]\n \n healthcare = ['Disease detection', 'Patient risk detection',]\n \n construction = ['Construction robots']\n \n transportation = ['Autonomous trucking technology', ]\n \n if mission_2023 in cybersecurity_missions:\n res = 'Cybersecurity'\n elif mission_2023 in sales_marketing_missions:\n res = 'Sales and Marketing'\n elif mission_2023 in ds_missions:\n res = 'Data Science'\n elif mission_2023 in ai_infrastructure:\n res = 'AI Infrastructure'\n elif mission_2023 in pharma:\n res = 'Pharmaceutical'\n elif mission_2023 in healthcare:\n res = 'Healthcare'\n elif mission_2023 in construction:\n res = 'Construction'\n elif mission_2023 in transportation:\n res = 'Transportation and Logistics'\n elif mission_2023 in agriculture:\n res = 'Agriculture'\n elif mission_2023 in emp_support:\n res = 'Employee Support'\n elif mission_2023 in customer_service:\n res = 'Customer Service'\n elif mission_2023 in retail:\n res = 'Retail'\n elif mission_2023 in env_energy:\n res = 'Environment and Energy'\n elif mission_2023 in education:\n res = 'Education'\n elif mission_2023 in consumer_tech:\n res = 'Consumer Technology'\n elif mission_2023 in media_entertainment:\n res = 'Media and Entertainment'\n return res","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-17T09:37:50.271822Z","iopub.execute_input":"2023-07-17T09:37:50.272713Z","iopub.status.idle":"2023-07-17T09:37:53.757103Z","shell.execute_reply.started":"2023-07-17T09:37:50.272659Z","shell.execute_reply":"2023-07-17T09:37:53.754933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run_locally = False\nif run_locally:\n ai50_2022_dataset_path = './AI50-2022.csv'\n ai50_2023_dataset_path = './AI50-2023.csv'\nelse:\n ai50_2022_dataset_path = '../input/forbes-ai50-2022/AI50 2022.csv'\n ai50_2023_dataset_path = '../input/forbes-ai50-2023/AI50-2023.csv'\n \ndf_2022 = pd.read_csv(ai50_2022_dataset_path)\n\ndf_2023 = pd.read_csv(ai50_2023_dataset_path)\n\ndf_2022['INDUSTRY'] = df_2022['INDUSTRY'].apply(lambda x: fix_industry_2022_typos(x))\n\ndf_2023['INDUSTRY'] = df_2023['MISSION'].apply(lambda x: set_industry(x))\ndf_2023['GENERATIVE_AI'] = df_2023['MISSION'].apply(lambda x: is_generative_ai(x))\ndf_2023['FUNDING'] = df_2023['FUNDING'].apply(lambda x: funding_to_numeric(x))\ndf_2022['FUNDING '] = df_2022['FUNDING '].apply(lambda x: funding_to_numeric(x))\ndf_2022.rename(columns = {'FUNDING ':'FUNDING'}, inplace = True)\n\ndfg2023 = df_2023['INDUSTRY'].value_counts().reset_index()\ndfg2023.columns = ['INDUSTRY', 'COMPANY_COUNT_2023']\n\ndfg2023[['COMPANY_COUNT_2023']] = dfg2023[['COMPANY_COUNT_2023']].applymap(np.int64)\n\ndfg2022 = df_2022['INDUSTRY'].value_counts().reset_index()\ndfg2022.columns = ['INDUSTRY', 'COMPANY_COUNT_2022']\n\nouter_merged = pd.merge(\n dfg2023, dfg2022, how=\"outer\", on=[\"INDUSTRY\"]\n)\n\nouter_merged['COMPANY_COUNT_2023'] = outer_merged['COMPANY_COUNT_2023'].fillna(0)\nouter_merged[['COMPANY_COUNT_2023']] = outer_merged[['COMPANY_COUNT_2023']].applymap(np.int64)\n\nfig = go.Figure()\n\nfig.add_bar(x=outer_merged['INDUSTRY'],y=outer_merged['COMPANY_COUNT_2022'], hovertext=\"2022\", name=\"2022\", marker_color=\"orange\")\nfig.add_bar(x=outer_merged['INDUSTRY'],y=outer_merged['COMPANY_COUNT_2023'], hovertext=\"2023\", name=\"2023\", marker_color=\"green\")\n\n# Update visual layout\nfig.update_layout(\n showlegend=True,\n width=800,\n height=400,\n autosize=False,\n margin=dict(t=30, b=0, l=5, r=5),\n template=\"plotly_white\",\n yaxis_title=\"Number of Companies\",\n title=dict(text='AI50 Industry Focus Changes in 2022-2023', font=dict(size=20), yref='paper')\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-17T09:37:53.760530Z","iopub.execute_input":"2023-07-17T09:37:53.761116Z","iopub.status.idle":"2023-07-17T09:37:56.039834Z","shell.execute_reply.started":"2023-07-17T09:37:53.761072Z","shell.execute_reply":"2023-07-17T09:37:56.038054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that\n\n- *Data Science and AI Infrastructure* industries remain the top priorities with AI innovations (and the VC capital invested into the respective companies from such industries); moreover, *Data Science* industry got more companies listed in AI50 in 2023 vs. 2022\n- *Cybersecurity, Sales and Marketing, Media and Entertainment, and Employment Suppport* industries got wider representation in AI50 2023 vs. 2022\n- *Pharmaceutial, Healthcare, and Customer Service* industries got less companies listed in 2023 vs. 2022\n- *Biotech and Financial Services* totally disappeared from the AI50 list in 2023\n\nLet's review the individual Cybersecurity companies highlighted by Forbes experts to list in AI50 2023","metadata":{}},{"cell_type":"code","source":"df_2023_cyber = df_2023[df_2023['INDUSTRY'] == 'Cybersecurity']\ndf_2023_cyber","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-17T09:37:56.042055Z","iopub.execute_input":"2023-07-17T09:37:56.042519Z","iopub.status.idle":"2023-07-17T09:37:56.101755Z","shell.execute_reply.started":"2023-07-17T09:37:56.042481Z","shell.execute_reply":"2023-07-17T09:37:56.100089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Additional proof to AI in cybersecurity to become one of the mainstream areas in AI industry can be obtained from reviewing *10* companies from Forbse AI50 2022 list to participate in 2023 list as well. We are going to see how their funding was changed in between the time of filing both lists.","metadata":{}},{"cell_type":"code","source":"df_2023_subset = df_2023[['NAME', 'INDUSTRY', 'FUNDING']]\ndf_2022_subset = df_2022[['NAME', 'FUNDING']]\ndf_2023_subset.rename(columns = {'FUNDING':'FUNDING_AS_OF_APR_2023'}, inplace = True)\ndf_2022_subset.rename(columns = {'FUNDING':'FUNDING_AS_OF_APR_2022'}, inplace = True)\n\n# calculate the fundings changes during the last year\ndf = df_2023_subset.merge(df_2022_subset, on='NAME')\ndf['FUNDING_CHANGES'] = df['FUNDING_AS_OF_APR_2023'] - df['FUNDING_AS_OF_APR_2022']\ndf","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-17T09:37:56.104361Z","iopub.execute_input":"2023-07-17T09:37:56.105698Z","iopub.status.idle":"2023-07-17T09:37:56.148749Z","shell.execute_reply.started":"2023-07-17T09:37:56.105457Z","shell.execute_reply":"2023-07-17T09:37:56.147299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the most remarkable recent (2022-2023) capital raise happened to **Abnoral Security (Cybersecurity)**. It raised **USD 210 M** recently.\n\n
💥 Note: You can refer to a separate notebook per Forbes AI50 Insights to review the entire set of insights drawn from the data in Forbes AI50 List. \n \n
","metadata":{}},{"cell_type":"markdown","source":"#
Appendix F. DeepFakes as a Cybersecurity Threat to Humans
\n\nDeepfakes (computationally-created entities – images, videos, audio modalities etc.) got widespread as Generative AI technologies have been proliferated in the last few years. Not only do they falsely represent reality, but they also impose a threat to many areas of systems and societies. The recent events of the war in Ukraine demonstrated the readiness of DeepFakes to be used as a cyberweapon as well as an instrument to plot the IPSO with the audiences in the enemy states.\n\nAs such, DeepFake issues remain on the radars of scientists and AI industry professionals as a topic of interest to various aspects of cybersecurity and cyber-safety.\n\nAutomatic detection of deepfakes has seen many new ML techniques, however, human detection capabilities have been far less explored. In turn, human’s ability to identify and properly react to DeepFakes is crucial in the real-world scenarios where plotting the influence on the individuals, social groups or even entire nation populations is the major goal DeepFake launches.\n\nSince AI-generated fake materials can propagate through many uncontrolled routes, everyone can potentially be targeted by DeepFake-driven scams and attacks [52]. The industry experts notice that the changes in citizen behavior ((I) critical thinking; (II) ability to differentiate DeepFake media from real audio, video, and image materials) may become the only effective defense against it.\nIn one of the recent research projects [17], human ability to detect Deepfake images of human faces has been scrutinized. The study aimed to assess human capability to identify image deepfakes of human faces (*StyleGAN2: FFHQ*) from non-deepfake images (*FFHQ*), and to assess the effectiveness of simple interventions intended to improve detection accuracy. Using an online survey, 280 participants were randomly allocated to one of four groups: a control group, and 3 assistance interventions. Each participant was shown a sequence of 20 images randomly selected from a pool of 50 deepfake and 50 real images of human faces. Participants were asked if each image was AI-generated or not, to report their confidence, and to describe the reasoning behind each response. Overall detection accuracy was only just above chance and none of the interventions significantly improved this. Participants' confidence in their answers was high and unrelated to accuracy. Assessing the results on a per-image basis reveals participants consistently found certain images harder to label correctly but reported similarly high confidence regardless of the image. Thus, although participant accuracy was 62% overall, this accuracy across images ranged quite evenly between 85% and 30%, with an accuracy of below 50% for one in every five images.\n\nAnother research [18] was focused on the human perception of audio Deepfakes. In their experiment, 472 unique human users competed against a state-of-the-art AI deepfake detection algorithm for 14912 total rounds of the game to distinguish between real and fake audio samples. Their key findings were as follows:\n- Humans and AI deepfake detection algorithms share similar strengths and weaknesses, both struggling to detect certain types of attacks.\n- This contrasts with the superhuman performance of AI in many application areas such as object detection or face recognition.\n\nConcerning human success factors, the researchers found that\n- IT professionals have no advantage over non-professionals.\n- Native speakers have an advantage over non-native speakers.\n- Older participants tend to be more susceptible to audio DeepFakes than younger ones.\n\nBoth research groups [17, 18] claim modern humans can suffer from the misperception of DeepFake materials in certain scenarios. There is a need for an urgent call to action to address such a cybersecurity threat since cruel criminals can exploit it to achieve their malicious goals. Moreover, such exploits can be easily “weaponized” if DeepFakes deployed as a part of cyberwarfare operations.","metadata":{}},{"cell_type":"markdown","source":"#
Appendix G. References
\n \n1. Kerem Gülen, *AI and Ethics: Balancing progress and protection* – Jan 16, 2023 - https://dataconomy.com/2023/01/16/artificial-intelligence-security-issues/\n2. Petar Radanliev, David De Roure, Carsten Maple, Uchenna Ani, *Super forecasting the technological singularity risks from artificial intelligence* - https://arxiv.org/ftp/arxiv/papers/2301/2301.10028.pdf\n3. Abhilash Chakraborty, Anupam Biswas, Ajoy Kumar Khan, *Artificial Intelligence for Cybersecurity: Threats, Attacks and Mitigation* - https://arxiv.org/ftp/arxiv/papers/2209/2209.13454.pdf\n4. Andrew J Lohn, Krystal Alex Jackson, *Will AI Make Cyber Swords or Shields: A few mathematical models of technological progress* - https://arxiv.org/pdf/2207.13825.pdf\n5. Md Jobair Hossain Faruk, Sharaban Tahora, Masrura Tasnim, Hossain Shahriar, Nazmus Sakib, *A Review of Quantum Cybersecurity: Threats, Risks and Opportunities* - https://arxiv.org/ftp/arxiv/papers/2207/2207.03534.pdf\n6. Kaspar Rosager Ludvigsen, Shishir Nagaraja, Angela Daly, *The Dangers of Computational Law and Cybersecurity; Perspectives from Engineering and the AI Act* - https://arxiv.org/pdf/2207.00295.pdf\n7. Gaith Rjoub, Jamal Bentahar, Omar Abdel Wahab, Rabeb Mizouni, Alyssa Song, Robin Cohen, Hadi Otrok, Azzam Mourad, *A Survey on Explainable Artificial Intelligence for Cybersecurity* - https://arxiv.org/pdf/2303.12942.pdf\n8. Carlos Mendes, Tatiane Nogueira Rios, *Explainable Artificial Intelligence and Cybersecurity: A Systematic Literature Review* - https://arxiv.org/pdf/2303.01259.pdf\n9. Gautam Srivastava, Rutvij H Jhaveri, Sweta Bhattacharya, Sharnil Pandya, Rajeswari, Praveen Kumar Reddy Maddikunta, Gokul Yenduri, Jon G. Hall, Mamoun Alazab, Thippa Reddy Gadekallu , *XAI for Cybersecurity: State of the Art, Challenges, Open Issues and Future Directions* - https://arxiv.org/pdf/2206.03585.pdf\n10. Maximilian P Niroomand, David J Wales, *Physics-Inspired Interpretability Of Machine Learning Models* - https://arxiv.org/pdf/2304.02381.pdf\n11. Subash Neupane, Jesse Ables, William Anderson, Sudip Mittal, Shahram Rahimi, Ioana Banicescu, Maria Seale, *Explainable Intrusion Detection Systems (X-IDS): A Survey of Current Methods, Challenges, and Opportunities* - https://arxiv.org/pdf/2207.06236.pdf\n12. Maede Zolanvari, Zebo Yang, Khaled Khan, Raj Jain, Nader Meskin, *TRUST XAI: Model-Agnostic Explanations for AI With a Case Study on IIoT Security* - https://arxiv.org/pdf/2205.01232.pdf\n13. M. Humayn Kabir, Khondokar Fida Hasan, Mohammad Kamrul Hasan, Keyvan Ansari, *Explainable Artificial Intelligence for Smart City Application: A Secure and Trusted Platform* - https://arxiv.org/ftp/arxiv/papers/2111/2111.00601.pdf\n14. Victor Jüttner, Martin Grimmer, Erik Buchmann, *ChatIDS: Explainable Cybersecurity Using Generative AI* - https://arxiv.org/pdf/2306.14504.pdf\n15. National Institute of Standards and Technology (NIST) - AI Risk Management Framework – Jan 26, 2023 - https://www.nist.gov/itl/ai-risk-management-framework\n16. Microsoft Corporation, Best practices for AI security risk management - https://www.microsoft.com/en-us/security/blog/2021/12/09/best-practices-for-ai-security-risk-management/\n17. Sergi D. Bray, Prof. Shane D. Johnson, Dr. Bennett Kleinberg, *Testing Human Ability To Detect “Deepfake” Images of Human Faces* - https://arxiv.org/ftp/arxiv/papers/2212/2212.05056.pdf\n18. Nicolas M. Müller, Karla Pizzi, Jennifer Williams, *Human Perception of Audio Deepfakes* - https://doi.org/10.48550/arXiv.2107.09667\n19. Ebenezer R. H. P. Isaac, Jim Reno, *AI Product Security: A Primer for Developers* - https://arxiv.org/pdf/2304.11087.pdf\n20. Hyrum Anderson, 2021. *The Practical Divide between Adversarial ML Research and Security Practice: A Red Team Perspective*.- Retrieved Jan 24, 2023 from https://www.usenix.org/conference/enigma2021/presentation/anderson\n21. Shahid Alam, *Cybersecurity: Past, Present and Future* - https://arxiv.org/ftp/arxiv/papers/2207/2207.01227.pdf\n22. Micah Musser, Andrew Lohn, James X. Dempsey, Jonathan Spring, Ram Shankar Siva Kumar, Brenda Leong, Christina Liaghati, Cindy Martinez, Crystal D. Grant, Daniel Rohrer, Heather Frase, Jonathan Elliott, John Bansemer, Mikel Rodriguez, Mitt Regan, Rumman Chowdhury, Stefan Hermanek, *Adversarial Machine Learning and Cybersecurity: Risks, Challenges, and Legal Implications* - https://arxiv.org/ftp/arxiv/papers/2305/2305.14553.pdf\n23. *“Case Studies,” MITRE*, accessed July 2, 2023. - https://atlas.mitre.org/studies/\n24. The *AI Incident Database* also lists several incidents that are tagged with the label “Intent:Deliberate or expected”; see AI Incident Database, accessed Jul 2, 2023. - https://incidentdatabase.ai/apps/discover/?classifications=CSET%3AIntent%3ADeliberate%20or%20expected&display=details&is_incident_report=true&page=1&sortBy=relevance\n25. Hearing, *“To receive testimony on artificial intelligence applications to operations in cyberspace,”* before the United States Senate Committee on Armed Services, 117th Congress (2022) (statement of Andrew Moore, vice president and general manager of Cloud AI and Industry Solutions, Google Cloud), 1:46:45. - https://www.armed-services.senate.gov/hearings/to-receive-testimony-on-artificialintelligence-applications-to-operations-in-cyberspace\n26. J.D. Ndibwile, *Artificial Intelligence-Based Smart Grid Vulnerabilities and Potential Solutions for Fake-Normal Attacks: A Short Review* - https://arxiv.org/ftp/arxiv/papers/2202/2202.07050.pdf\n27. Subash Neupane, Ivan A. Fernandez, Sudip Mittal, Shahram Rahimi, *Impacts and Risk of Generative AI Technology on Cyber Defense* - https://arxiv.org/pdf/2306.13033.pdf\n28. Sukhpal Singh Gill, Rupinder Kaur, *ChatGPT: Vision and Challenges* - https://arxiv.org/ftp/arxiv/papers/2305/2305.15323.pdf\n29. Carly Page, *Hackers are increasingly using ChatGPT lures to spread malware on Facebook* – May 3, 2023 - https://techcrunch.com/2023/05/03/malware-chatgpt-lures-facebook/\n30. Dash, Bibhu, and Pawankumar Sharma, *Are ChatGPT and Deepfake Algorithms Endangering the Cybersecurity Industry? A Review.* - International Journal of Engineering and Applied Sciences 10, no. 1 (2023).\n31. Enrico Cambiaso, Luca Caviglione, *Scamming the Scammers: Using ChatGPT to Reply Mails for Wasting Time and Resources* - https://arxiv.org/pdf/2303.13521.pdf\n32. *Pause Giant AI Experiments: An Open Letter* – Mar 22, 2023 - https://futureoflife.org/open-letter/pause-giant-ai-experiments/\n33. Rouzbeh Behnia, Mohamamdreza Ebrahimi, Jason Pacheco, Balaji Padmanabhan, *Privately Fine-Tuning Large Language Models with Differential Privacy* - https://arxiv.org/pdf/2210.15042.pdf\n34. Georgii Vyshnia, *Generative AI Timeline* – Jun 30, 2023 - https://www.kaggle.com/discussions/general/420167\n35. Megan Lamberth, Paul Scharre, *Arms Control for Artificial Intelligence* - https://tnsr.org/2023/05/arms-control-for-artificial-intelligence/, http://dx.doi.org/10.26153/tsw/46142\n36. Danny Palmer, *ChatGPT and more: What AI chatbots mean for the future of cybersecurity* – Feb 14, 2023 - https://www.zdnet.com/article/chatgpt-and-more-what-ai-chatbots-mean-for-the-future-of-cybersecurity/\n37. Evan Crothers, Nathalie Japkowicz, Herna Viktor, *Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods* - https://arxiv.org/pdf/2210.07321.pdf\n38. Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, Brendan Dolan-Gavitt, *Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants* - https://arxiv.org/pdf/2208.09727.pdf\n39. Reza Fayyazi, Shanchieh Jay Yang, *On the Uses of Large Language Models to Interpret Ambiguous Cyberattack Descriptions* - https://arxiv.org/pdf/2306.14062.pdf\n40. Sudip Mittal, Jingdao Chen, *AI Security Threats against Pervasive Robotic Systems: A Course for Next Generation Cybersecurity Workforce* - https://arxiv.org/pdf/2302.07953.pdf\n41. Vipin Kumar Kukkala, Sooryaa Vignesh Thiruloga, Sudeep Pasricha, *Roadmap for Cybersecurity in Autonomous Vehicles* - https://arxiv.org/ftp/arxiv/papers/2201/2201.10349.pdf\n42. Shih-Chun Lin, Chia-Hung Lin, Wei-Chi Chen, *Zero-Touch Network on Industrial IoT: An End-to-End Machine Learning Approach* - https://arxiv.org/pdf/2204.12605.pdf\n43. Elisabetta Biasin, Erik Kamenjasevic, Kaspar Rosager Ludvigsen, *Cybersecurity of AI medical devices: risks, legislation, and challenges* - https://arxiv.org/ftp/arxiv/papers/2303/2303.03140.pdf\n44. Md. Shirajum Munir, Sachin Shetty, Danda B. Rawat, *Trustworthy Artificial Intelligence Framework for Proactive Detection and Risk Explanation of Cyber Attacks in Smart Grid* - https://arxiv.org/pdf/2306.07993.pdf\n45. Mohit Sewak, Sanjay K. Sahay, Hemant Rathore, *Deep Reinforcement Learning for Cybersecurity Threat Detection and Protection: A Review* - https://arxiv.org/pdf/2206.02733.pdf\n46. Soumyadeep Hore, Jalal Ghadermazi, Diwas Paudel, Ankit Shah, Tapas K. Das, Nathaniel D. Bastian, *Deep PackGen: A Deep Reinforcement Learning Framework for Adversarial Network Packet Generation* - https://arxiv.org/pdf/2305.11039.pdf\n47. Robert Lemos, *Employees Are Feeding Sensitive Biz Data to ChatGPT, Raising Security Fears* – Mar 7, 2023 - https://www.darkreading.com/risk/employees-feeding-sensitive-business-data-chatgpt-raising-security-fears\n48. Thomas Krendl Gilbert, Sarah Dean, Tom Zick, Nathan Lambert, *Choices, Risks, and Reward Reports: Charting Public Policy for Reinforcement Learning Systems* - https://arxiv.org/ftp/arxiv/papers/2202/2202.05716.pdf\n49. Khatoon Mohammed, *Harnessing the Speed and Accuracy of Machine Learning to Advance Cybersecurity* - https://arxiv.org/ftp/arxiv/papers/2302/2302.12415.pdf\n50. Georgii Vyshnia, *AI Ethics Frameworks: Possible impact on the future of the industry* – Jun 21, 2023 - https://www.kaggle.com/discussions/general/418559\n51. Henk van Ess, *#ChatGPT: 4. Unlocking Geolocation with Large Language Models: A Workflow by Henk van Ess* – May 9, 2023 - https://www.digitaldigging.org/p/4-chatgpt-unlock-geolocation-data\n52. Henk van Ess, *Dark side of AI: 2. Voice Cloning for Crime*. – May 23, 2023 - https://www.digitaldigging.org/p/dark-side-of-ai-2-voice-cloning-for\n53. Mu-Huan Chung, Lu Wang, Sharon Li, Yuhong Yang, Calvin Giang, Khilan Jerath, Abhay Raman, David Lie, Mark Chignell, *Implementing Active Learning in Cybersecurity: Detecting Anomalies in Redacted Emails* - https://arxiv.org/pdf/2303.00870.pdf\n54. Branislav Bosansky, Dominik Kouba, Ondrej Manhal, Thorsten Sick, Viliam Lisy, Jakub Kroustek, Petr Somol, *Avast-CTU Public CAPE Dataset* - https://arxiv.org/pdf/2209.03188.pdf\n55. Zihang Dai, Hanxiao Liu, Quoc V. Le, Mingxing Tan (Google Research,Brain Team), *CoAtNet: Marrying Convolution and Attention for All Data Sizes* - https://arxiv.org/pdf/2106.04803.pdf\n56. Georgii Vyshnia, *MalImg CoatNet Model* – Apr 2023 - https://www.kaggle.com/code/gvyshnya/malimg-coatnet-model\n57. Georgii Vyshnia, Denys Frolov – *MalImg Malware Detection experiments* – Last updated Mar 25, 2023 - https://github.com/gvyshnya/malimg\n58. Joon Sern Lee, Kai Keng Tay, Zong Fu Chua, *BinImg2Vec: Augmenting Malware Binary Image Classification with Data2Vec* - https://arxiv.org/pdf/2209.00782.pdf\n59. Junyi Liu, Yifu Tang, Haimeng Zhao, Xieheng Wang, Fangyu Li, Jingyi Zhang, *CPS Attack Detection under Limited Local Information in Cyber Security: A Multi-node Multi-class Classification Ensemble Approach* - https://arxiv.org/pdf/2209.00170.pdf\n60. William Anderson, Kaneesha Moore, Jesse Ables, Sudip Mittal, Shahram Rahimi, Ioana Banicescu, Maria Seale, *Designing an Artificial Immune System inspired Intrusion Detection System* - https://arxiv.org/pdf/2208.07801.pdf\n61. Guangyi Zhu, Yasir Al-Qaraghuli, *AI-Assisted Authentication: State of the Art, Taxonomy and Future Roadmap* - https://arxiv.org/pdf/2204.12492.pdf\n62. Anna Himmelhuber, Dominik Dold, Stephan Grimm, Sonja Zillner, Thomas Runkler, *Detection, Explanation and Filtering of Cyber Attacks Combining Symbolic and Sub-Symbolic Methods* - https://arxiv.org/pdf/2212.13991.pdf\n63. Jan von der Assen, Alberto Huertas Celdrán, Janik Luechinger, Pedro Miguel Sánchez Sánchez, Gérôme Bovet, Gregorio Martínez Pérez, Burkhard Stiller, *RansomAI: AI-powered Ransomware for Stealthy Encryption* - https://arxiv.org/pdf/2306.15559.pdf\n64. Nathan Benaich, Ian Hogarth, *State of AI Report 2022* – Extracted Jul 4, 2023 - https://www.stateof.ai/\n\n","metadata":{}}]}