{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50947,"databundleVersionId":5741538,"sourceType":"competition"}],"dockerImageVersionId":30513,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-07-04T15:38:09.086699Z","iopub.execute_input":"2023-07-04T15:38:09.087570Z","iopub.status.idle":"2023-07-04T15:38:09.128189Z","shell.execute_reply.started":"2023-07-04T15:38:09.087522Z","shell.execute_reply":"2023-07-04T15:38:09.127055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**ADDENDUM 2024-01-13**\n\nFollowing the 6th monthly update of the webapp AI Ethics Primer - search within the bibliograph, released a new dataset on kaggle.\n\n[A companion to \"A primer on AI ethics via arXiv\"](https://www.kaggle.com/datasets/robertolofaro/a-companion-to-a-primer-on-ai-ethics-via-arxiv)\n\nThe dataset associated with the webapp will be updated on a monthly basis (extraction on the 11th, release within the following week-end).\n\nAs of today, it contains 134 papers for 2849 pages\n\nMore information about the evolution of the dataset and associated future publications will be added to the dataset, while this notebook will keep being a reference to the initiall essay submission in July 2023\n\nShould you be interested to know more, [contact me here on Kaggle or on Linkedin](https://kaggle.com/robertolofaro)","metadata":{}},{"cell_type":"markdown","source":"**ADDENDUM 2023-08-13**\n\nAs announced within the text essay, the webapp allowing to search within the content of the bibliography (i.e. each paper) has been updated, using papers on AI and ethics available on arXiv.\n\nThe update covers papers updated or released between 2023-07-03 and 2023-08-11.\n\nThe essay released on 2023-07-04 covered a selection of 10 papers out of over 600 found on arXiv, selected for ***accessibility*** to non-experts, and reference for interested parties, as of 2023-07-03.\n\nNotably, those influencing or deciding to the allocation resources for AI projects and activities.\n\nThis update:\n* extracted the 58 papers updated or released since 2023-07-03\n* out of those 58, reviewed a couple of dozen relevant to both my research/publishing activities and the original aim of this essay (a \"primer\")\n* out of that couple of dozen, considered only those updating on the previously selected and the \"accessibility\" criterion exposed above.\n\nThis added 12 further papers, now with a total of 23 (essay, original 10 selected, plus the additional 12).\n\n[The webapp is available here](https://robertolofaro.com/aiethicsprimer)\n\nWill update this comment on a monthly basis, after selecting on the 11th day of the month the updates, the following week-end.\n\n\n, and showing in which section the selected search key appears (later multiple search keys will be allowed)","metadata":{}},{"cell_type":"markdown","source":"# Section 1: Introduction and concept\n\n## a visual summmary\n\n\n\nThis essay is quite long- but its aim is to provide a reasoned access to material on AI ethics from arXiv and other sources- with a focus on the evolution 2020-2023.\n\nThe rationale? To misquote two giants... when a technology looks like magic, it is too important to be left to philosophers and technologists.\n\nOr: the ethical dimension and considerations should be accessible to anybody who can be affected (including users of AI), and anybody who is involved in influencing the evolution of ethical standards, including policy makers.\n\nBeside this tag cloud, will add soon a bibliography browsing and search tool.\n\nRead the Q&A below in this section for more information.\n\n\n## the purpose of this essay\n\n\n### ethics within the AI context: a context definition from the AI Index 2022 Annual Report\n\nAs reported within the [The AI Index 2022 Annual Report](https://arxiv.org/abs/2205.03468): \n*The rise of AI ethics everywhere: Research on fairness and transparency in AI has exploded since 2014, with a fivefold increase in related publications at ethics-related conferences. Algorithmic fairness and bias has shifted from being primarily an academic pursuit to becoming firmly embedded as a mainstream research topic with wide-ranging implications. Researchers with industry affiliations contributed 71% more publications year over year at ethics-focused conferences in recent years.*\n\n**AI Ethics** is one of the categories for [2023 Kaggle AI Report](https://www.kaggle.com/competitions/2023-kaggle-ai-report/) \"to describe what the community has learned over the past 2 years of working and experimenting\".\n\nThis essay is focused on AI ethics from both insider (industry and academia) and social/normative (e.g. legislators) perspective.\n\n\n### audience and structure of this essay\n\nThe intended audience is anybody with interest in AI ethics, not necessarily just within the AI community.\n\nThis extends to:\n- business practictioners interested in integrating AI within their products or services\n- potential stakeholders that could affect or be affected by choices involving AI trends\n- AI or ICT community members without a background in ethics, philosophy, political science.\n\nThis essay aims to summarize the history, current status, and trends of AI ethics, mainly through papers available on arXiv.\n\nIn order to allow also non-experts to both understand the context, recent trends, and potential evolution, this short essay includes visualizations of historical patterns and trend within STEM, as represented by the papers published on arXiv, and additional material from other sources.\n\nWhile as of 2023-07-02 a search on arXiv resulted in over 600 papers, it is a subject in constant evolution, and some papers or versions thereof were added and other withdrawn within the few days (2020-06-30 to 2026-07-02) that I spent searching the online archive.\n\nThe papers and other documents referenced are both linked within the text, and listed in Appendix A.\n\nAs a selection criteria, considering the diverse audience identified, and the multinational audience of Kaggle, after selecting and revising few dozen out of those 600 papers, plus other reference documentation, restricted the bibliography used and shared in this essay to those papers that fulfilled the following accessibility criterias:\n- did not require specific domain skills (be it AI or social sciences) to be understood\n- were not focused on a single country, and preferably contained results from multi-country surveys/studies\n- contained an extensive bibliography, should the readers be interested to further their understanding.\n\nTo keep this essay short yet informative, instead of sharing just my commentary, whenever feasible will quote from documentation contained in the bibliography, to have a logical train of thought but also, if possible, nudge the audience toward reading one of the documents that suggested in the bibliography.\n\nTherefore, while here and there hinted at and summarized my position on AI ethics, I let the papers within the bibliography to deliver the information that could give to readers an overview of history, current status, trends, and ongoing research.\n\nMy own position is actually more on the democratization of access to technology, not just AI, and its ethical consequences, and [shared it within this long \"parallel\" essay\"](https://robertolofaro.com/index.php?page=513).\n\nWhy splitting the two? To keep this one as short as possible and as a service to the community, along with the evolution of the bibliography.\n\n\n## information sources and date of reference\n\nThe data for this notebook were extracted on July 2nd from the [arXiv website](https://arxiv.org), and this notebook is attached to the dataset for [2023 Kaggle AI Report](https://www.kaggle.com/competitions/2023-kaggle-ai-report/data), a dataset that is described to be updated on a weekly basis.\n\nAs data sources, have been used both the [JSON file provided by Cornell on Kaggle](https://www.kaggle.com/datasets/Cornell-University/arxiv), as well as selected papers directly on ArXiv.\n\nWithin Appendix A are listed all the books, papers, and documents used, as well as courses, should some of the readers be interested in learning more about the subject.\n\nWithin Appendix B is provided the information that is contained within the charts, as well as a list of all the charts in this essay.\n\nThis is the tag cloud of the content (not just the abstracts) of the bibliography\n\n\nsee Appendix A for an explanation; you can search which papers and chapters within papers, including this essay, are relevant to a specific term [at this address](https://robertolofaro.com/searchkaggleaiethics.php). \n\nWhile this essay will stay \"as is\" (to keep the reference point 2020-2023), the tag cloud online (and associated bibliography) will be updated on a monthly basis, on the 11th of each month (starting 2023-08-11), released the following week-end, to mirror the evolution of the AI ethics field.\n\n\n## about the charts representing information from the arXiv website\n\nThe information within the JSON dataset was up to 2023-06-23.\n\nThe information shown in the charts has been extracted on 2023-07-02 via queries on arXiv website.\n\n\n## structure of the essay\n\nTo keep this notebook short, each section in this notebook contains data visualization selected to highlight specific elements of the analysis, as in this table:\n\n| Section | Purpose | Key visualizations |\n| --- | --- | --- |\n| Section 1 | Introduction and concept (this section) | descriptive |\n| Section 2 | History and prospective | charts and tables |\n| Section 3 | Current trends 2020-2023 | charts and tables |\n| Section 4 | Considerations and conclusions | tables |\n| Appendix A | Bibliographical references | tables |\n| Appendix B | Data and charts from arXiv | tables |\n\n\n## Summary Q&A about this essay\n\n- Q: why the choice of static charts?\n- A: the dataset from arXiv will be updated on a weekly basis, hence to give a reference point static information was needed\n\n- Q: is possible to have also the data used to produce the charts?\n- A: all the data is available within the Appendix B\n\n- Q: which tools were used to produce the material within the notebook?\n- A: open source: Jupyter Notebooks, DBGate (to convert the JSON to SQLite), and DBBrowser (as queries were less computationally intensive directly on the SQLite database than using Jupyter Notebook directly on the JSON file); the hardware is an offline Chromebook 8GB RAM 128GB space, within the Linux environment; CSV files were extracted from the SQLite database and converted in charts using the desktop versions of Excel and Powerpoint \n\n- Q: would it be possible to have the same notebook, but to run it dynamically against the dataset updated weekly?\n- A: a Jupyter Notebook and other tools to allow navigate the data dynamically (but with different visualizations, and extending beyond the theme AI ethics) will be released by the end of summer 2023, and the Jupyter Notebook version will be shared on Kaggle; note: the JSON dataset references the update_date, while all the charts have been, by choice, based on the \"first announced\" date that is available as a search field on arXiv under the \"advanced\" options\n\nShould you have remarks, comments, suggestions, you are welcomed to comment on this notebook, or contact me on either [Kaggle](https://kaggle.com/robertolofaro) or [Linkedin](https://linkedin.com/in/robertolofaro)\n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T15:38:21.542630Z","iopub.execute_input":"2023-07-04T15:38:21.543129Z","iopub.status.idle":"2023-07-04T15:38:21.674387Z","shell.execute_reply.started":"2023-07-04T15:38:21.543095Z","shell.execute_reply":"2023-07-04T15:38:21.672166Z"}}},{"cell_type":"markdown","source":"# Section 2: History and prospective\n\n\n\n\n## what is AI ethics and what does imply: a definition from the UN\n\nWithin [A Framework for Ethical AI at the United Nations\n](https://arxiv.org/abs/2104.12547) ethics are defined as *\"standards of right and wrong, acceptable and not acceptable, in a certain community. It includes an individual's rights, obligations, virtues, the benefits ot society, and a sense of fairness. Ethics will clearly differ for various cultures or groups. What is acceptable in one country may not be in another. They also change over time.\"* (page 11) and *\"For the UN, the challenge is hardest as we need to serve all people. However, we do have a basis to work from: The UN Charter, the Universal Declartioon of Human rights (UDHR) and a number of internationl treaties give us a basis for AI principles. In fact, some principles may not need to be reinvented, just applied in the context of AI.\"* (page 12)\n\n\n## why AI ethics is more complex than traditional ethics\n\nAs you can see from the chart above, the number of papers on AI and ethics increased significantly from 2016, probably associated to high media visibility due to the [AlphaGo](https://en.wikipedia.org/wiki/AlphaGo) and other media-hyped AI-related events, as reported also from other researchers (and will be presented in later sections of this essay).\n\nA key difference of AI ethics vs. traditional ethics is a consequence of the second part of the definition from the Ethical AI framework: AI is currently accessed via smartphones or computers, and therefore by definition has a global reach.\n\nWhenever introducing a technology or process/social change, usually the social and cultural context defines also what is permissible or accepted or relevant- and this had already various impacts in the past e.g. on naming products to avoid cultural misunderstandings in specific markets.\n\nWith AI along with pervasive computing and continuous data-access via e.g. smartphones, the ethical dimension gets a multicultural communication side that is not necessarily evident, if adopting just a technological perspective.\n\nThis was shown by recent discussions about bias in datasets used to train models (e.g. selecting people from a specific location or ethnic group to train on facial recognition, and then trying to apply it in locations with a different ethnic mix), that resulted then in bias in decision making.\n\nA further element is the speed: traditional ethics based on human decision-making leaves time to consider impacts of decisions, or even to amend them after a first feed-back from a restricted group, while a data-intensive, automated decision-making could generate further decisions faster than any human oversight or audit trying to remedy a potentially questionable initial decision.\n\nAlso, with current data-distribution technologies, and the current legal framework, there is little or no incentive to track down each use, and therefore to propagate also amendments to data or decisions taken by an algorithm.\n\nIn a chain of automated decisions, in a distributed environment, a wrong choice by the first decision-point might be \"inherited\" by further decision points at face value, and then, in turn, this might \"infect\" further decision-points downstream.\n\nThe negatively affected party would contact the last link within the chain, but would have limited interest and recourse to access the chain upstream, unless there were an \"automatic propagation of amendments\" right.\n\nThe European Union data privacy regulation tried to define a legal framework also to cover this algorithmic derivative data distribution, focusing mainly on the downstream side, but it is still a work in progress in terms of implementation, as there is e.g. no mandate to have a blockchain on data that would enforce the need to track the source and evolution of any data dissemination activity.\n\nHence, already in 2016 (also if it was enforced from 2018), the European Union data privacy regulation GDPR had specific sections in its preamble focused on restricting and making transparent the use of automated decision making- but more as a general conceptual framework (the \"privacy by default\" and \"privacy by design\"), than a specific set of measures to be implemented.\n\n\n## the cultural dimension of AI: a 2019 survey in eight countries\n\nWhile in the next section will discuss more details on the evolution 2020-2023, a paper published on arXiv contains an interesting survey done in July 2019 in eight countries, [Exciting, Useful, Worrying, Futuristic: Public Perception of Artificial Intelligence in 8 Countries](https://arxiv.org/abs/2001.00081).\n\nThe countries were Australia, Canaada, USA, South Korea, France, Brazil, India, Nigeria, and the structure of the study is described as *\"We surveyed public opinion of artificial intelligence with 10,005 respondents spanning eight countries across six continents, focusing on issues such as expected impacts of AI, sentiment towards AI, and variation in response by country.\"*\n\nThe study authors classified the sentiment across few groups:\n\n| sentiment group | % | people | characterization |\n| --- | --- | --- | --- | \n| Exciting | 18.9% | 1888 | positive feelings about AI | \n| Useful | 12.2% | 1223 | AI will be helpful and assist humans in completing tasks | \n| Worrying | 22.7% | 2269 | wide range of negative emotional responses, predominantly various forms of concern and fear | \n| Futuristic | 24.4% | 2439 | not necessarily positive or negative toward AI, but rather included for any mention of the futuristic nature of AI |\n| None | 34.9% | 3489 | responses that received only codes labeled 'other' | \n\nThe total exceeds 100%, as more than one answer was possible, and this chart represents the distribution across the countries:\n","metadata":{}},{"cell_type":"markdown","source":"\n\n\n## defining the dimensions of AI ethics: a 2018-2021 systematic literature review\n\nA study that carried out a systematic literature review, identifying 27 primary studies released between 2018 and 2021:\n\n| year | studies reviewed |\n| --- | --- |\n| 2018 | 2 |\n| 2019 | 4 |\n| 2020 | 19 |\n| 2021 | 2 |\n\nThe study, [Ethics of AI: A Systematic Literature Review of Principles and Challenges](https://arxiv.org/abs/2109.07906) resulted in identifying 22 principles and 15 challenging factors.\n\nAs stated within the conclusions: *\"most of the studies focus on four major principles i.e., transparency privacy, accountability and fairness\"*.\n\nThe paper describes each one of the principles,and, if you read it, you can see that the principles are overlapping with those presented in other studies, but with a different mix- a sign that our consensus on what is AI and what is AI ethics is still evolving.\n\nMore interesting, within the same paper, the challenges identified across the studies reviewed (in order of frequency):\n- Lack of ethical knowledge\n- Value principles\n- Lack of technical understanding\n- Extra constraints\n- Conflict in practice\n- Interpret principles differently\n- Highly general\n- Lack of Cross-cultural Cooperation\n- Lack of Guidance and Adoption\n- Machine distortion\n- Cases of ethical dilemmas\n- Pluralism of ethical methods\n- Business interest\n- No Legal framework\n- Lacking monitoring bodies.\n\nAs will be discussed in the following sections of this essay, also directing toward other papers on arXiv, this calls for a multidisciplinary approach in AI projects since the \"business case\" approval phase, and an initiative discussed within the conclusions had some success on that side- so much, that it is planning to extend the approach also to non-AI projects.\n\n\n## from perception and philosophy of AI ethics, to data: a database review on AI incidents\n\nSurveys and literature support identifying two key elements for an ethical AI framework (perception and developing consensus between experts), but the definition of a regulatory framework requires also visibility on the actual incidence of issues that should be solved or pre-empted by laws, regulations, etc.\n\nAs a new regulation or law might involve significant costs to those required to comply with it, before issuing either legislators usually consider the overall impact and cost/benefits.\n\nIt is the same approach used in risk management and personal insurance: would you take an insurance covering a risk that you identify as marginal or with extremely low probability? Whatever its cost, probably you would focus on other risks.\n\nAn interesting study analyzed the [AI Ethics Issues in Real World: Evidence from AI Incident Database](https://arxiv.org/abs/2206.07635).\n\nIn summary, it covered 150 cases between 2010 and 2021, and *\"the number of AI ethics incidents gradually increased from 2010 to 2016, and reached a peak in 2016. The trend may be related to the fast development of AI during that time period: Google’s AlphaGo beated world Go champion Lee Sedol; Microsoft’s AI device outperformed humans in language understanding; AI made significant breakthroughs in the medical field; Tesla’s self-driving vehicle sent patients to hospitals. After 2016, the number of AI ethics incidents declined until 2019, and then increased again, reaching a maximum in 2020. The trend is consistent with the heated discussion of AI ethics issues in society in recent years, which reflects people’s sensitivity to the risk of AI ethics issues\"*.\n\nThe aim was \"to build a taxonomy of AI ethics issues in real world\", and used the AI Incident Database, studying 150 incidents from 2010 to 2021, with a peak in 2016.\n\n\"150 AI ethics incidents were analyzed. A sharp increase of AI ethics incidents was seen after 2010, with most of them happening in the US, China, and the UK. \"\n\nFrom the document, these are the most common:\n\n| Application area | Frequency |\n| --- | --- |\n| intelligent service robots | 31 |\n| language/visioon models | 27 |\n| autonomous driving | 17 |\n| intelligent recommendatioon | 14 |\n| identity authentication | 14 |\n| AI supervision | 14 |\n\nThe ethics issues found were:\n\n| issue | frequency |\n| --- | --- |\n| inappropriate use (bad performance) | 48 | \n| racial discriminatioon | 38 | \n| physical safety | 32 | \n| unfair algorithm (evaluation) | 22 | \n| gender discrimination | 19 | \n| privacy | 12 | \n| unethical use (illegal use) | 11 | \n| mental health | 4 | \n\nThe top ethical principle was transparency, followed by justice and fairness, and then non-maleficence.\n\nTherefore, no surprise that 2016 was the year when, also in arXiv, as shown in the chart above, there was a sharp increase in papers concerning both AI and ethics.\n\nAnd this implies also a more focused normative approach, an attempt to regulate the unknown, that is not limited to the GDPR focusing on principles instead of specific cases.\n\n\n\n## the evolution of the normative framework of AI ethics\n\nIn the late 1990s, with the available of access to the Internet also for commercial uses, new services pushed the OECD to propose a framework for e-government, as had proposed a framework for the activities of multinational companies, and in 2019 [Recommendation of the Council on Artificial Intelligence](https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449) was added.\n\nWhenever there is a normative intervention, usually we have already shifted from an \"emerging\" phenomenon to an actual trend: a single event or incident might represent a case worth studying, but a collection of incidents representing a pattern, in a market economy, generally becomes the focus of new rules or at least a professional code of conduct.\n\nLooking just at key developments on that side since 2019:\n\n| when | source | document (and link) |\n| --- | --- | --- |\n| April 2019 | European Union - High-level Expert Group On Artificial Intelligence | [Ethics Guidelines For Trustworthy AI](https://ec.europa.eu/newsroom/dae/document.cfm?doc_id=60419) |\n| May 2019 | OECD - Compendium of Legal Instruments of the OECD | [Recommendation of the Council on Artificial Intelligence](https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449) |\n| 2022 | UNESCO | [Recommendation of the Council on Artificial Intelligence](https://unesdoc.unesco.org/ark:/48223/pf0000381137) |\n| June 2023 | European Parliament | [EU AI Act](https://www.europarl.europa.eu/news/en/headlines/society/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence?&at_campaign=20226-Digital&at_medium=Google_Ads&at_platform=Search&at_creation=RSA&at_goal=TR_G&at_advertiser=Webcomm&at_audience=ai%20eu&at_topic=Artificial_intelligence_Act&at_location=IT&gclid=Cj0KCQjwtO-kBhDIARIsAL6Lord2Lw-Jx1O7HtEVr_oVYiu8sLtN0YxuOjryfE-XNmljGFz2dD6h4h8aArLAEALw_wcB) |\n\nThe June 2023 AI Act is still a work-in-progress, but actually is not just from the European Parliament and European Commission, but also the European Central Bank ([see here the amendments](https://www.europarl.europa.eu/doceo/document/TA-9-2023-0236_EN.html)).\n\nAs I wrote previously, it was actually a journey that started from the GDPR in 2016, which was based on European experience since the 1990s on data privacy across all the Member States of the European Union (until 2020, also including UK).\n\nThe uniqueness of the experience of the European Union is that its Member States have still de facto a long legal history and culture to contribute, and therefore can act as a laboratory for multinational regulations.\n\nWhile the 1990s data privacy in European Union required a country-by-country adoption, the experience resulted in a GDPR regulation, i.e. immediately active across all Member States, with national watchdogs actually becoming a national component of the overall European Union data privacy framework.\n\nThis avoided the patchwork of laws and regulations that previously affected e.g. businesses trying to work across multiple EU Member States.\n\nAs stated above, the GDPR already included in 2016 reference to ethical, specifically transparency, inclusion, and traceability (in AI context, explainability) of decisions made by algorithms.\n\nHaving been enforced since 2018, the new EU AI Act has the added benefit of five years of application and monitoring across a market with more than 500 million citizens.\n\nGDPR does not apply just to the European Union- applies to any processing done by any organization involving European Union citizens' data- hence, its impact on the data privacy regulation worldwide, as many jurisdictions decided to converge on those principles also to simplify business choices.\n\nThe impacts of the EU AI Act already started before it is even implemented across the EU, as organizations worldwide are studying how to pre-empt compliance (by altering systems and processes before it is required in their own jurisdiction), and anyway its content will be revised while starting new AI projects, or evolving existing AI products and services.\n\nThe next section will discuss the information derived from a research on the arXiv website on publications concerning AI ethics.","metadata":{}},{"cell_type":"markdown","source":"# Section 3: Current trends 2020-2023\n\n\n## looking again at arXiv contents\n\nLooking again at the arXiv archive, this is the evolution for papers concerning ethics:\n\n\n\nThis chart shows that ethics has been a sustained theme since 2020, with hundreds of papers each year.\n\nIf we shift to the topic of this essay, AI Ethics, this is the result:\n\n.\n\nAs you can see, between 2020 and 2021 there was a sharp increase.\n\nOf course 2020 was the peak COVID-19 pandemic year, and this generated significant increases in the use of remote working, while companies further expanded the use of electronic means for activities that used to be carried out by humans, such as technical support.\n\nI do not know in other countries, but in Italy almost all the chat interaction boxes (and many phone contact centers) became increasingly automated since 2020, and only recently were added options to talk with human operators, often only after spending time with a chatbot.\n\nI did browse the list and abstracts of the 600+ papers on arXiv concerning AI ethics (with various keys to cover spelling variants etc), and selected few dozens to review.\n\nThe bibliography includes only those that deemed useful to share, see the introduction for the selection criterias, to simplify the work of those interesting in knowing more about the subject.\n\nAlso, most of the elements within the bibliography are relatively short (except few reference works, generally all the papers discussed in this article have less than 20 pages- I did not include others significantly longer, and those requiring specific domain knowledge, e.g. in AI or ethics; will discuss some of them in future writings).\n\nIt was interesting to see how many papers were announced or published in 2020, and how many focused on various application themes.\n\nThe number of papers stabilized to the new higher level in 2022, and in 2023 (look just at the first six months) there was a further increase: in six months, almost as many as in all of 2022 were announced.\n\nI used the announce date as point of reference for my research, instead of the update_date contained in the JSON file: this allowed to identify when the subject \"emerged\" as of interest to the authors (some of the papers had multiple updates- e.g papers announced in 2019, released in 2020, and then last updated in 2023).\n\nGenerally, when studying other development of \"emerging\" concepts, they appear as terms of reference within papers, then appear in abstracts, and finally, when become trendy, appear in titles (generally the time from content to abstracts is longer than the time from abstracts to titles).\n\nLooking at ethics and AI ethics (the legenda contains the search keys used) within arXiv, this is the appearance in abstracts 2020-2023:\n\n\n\nAnd this is the equivalent in titles:\n\n\n\nThe key element by looking at the abstracts and titles of papers concerned with AI ethics is a gradual shift toward the identification of a shared concept and taxonomy of AI ethics.\n\nMore recent papers also increased the consideration of risks and risks mitigation.\n\n\n## shifting to the actual implementation trends: going glocal\n\n\nGlocal is a concept that was quite common decades ago, but now and then surfaces again.\n\nThe concept is to think globally, act locally, i.e. integrate with the local context, while retaining a degree of harmonization that makes global operation feasible, without ignoring differences are critical to the local success.\n\nCould be something as simple as knowing the communication styles to implement with your audience, or as complex as integrating with local compliance frameworks.\n\nBeside deciding what is AI ethics, and what are its impacts and risks, and the best policies to adopt, papers on the subject published in recent years have added as a theme cultural differences.\n\nAs described in the previous sections of this essay, surveys showed quantitatively differences in expectations about AI, which could actually affect product and service design, as well as communication.\n\nAnother interesting study is: [The Different Faces of AI Ethics Across the World: A Principle-Implementation Gap Analysis](https://arxiv.org/abs/2206.03225).\n\nThe scope of the research? In the authors' own words *\"we provide a contextual and holistic evaluation of current ethical AI principles and their implementations by analyzing a set of 100 docuemnts and e websites containing 100 AI principles from 29 countries in 6 continents and 121 implementations of these principles from 14 countries in 6 continents.\"*\n\nThe most cited principle? Transparency, while most countries are covered by privacy, transparency, fairness, security, safety, responsibility, accountability, explainability, well-being, human rights, and inclusiveness.\n\nThe already quoted [A Framework for Ethical AI at the United Nations](https://arxiv.org/abs/2104.12547) obviously can benefit from an organizational perspective that is multilateral and global by definition, but it is still more moral guidance, than implementation, albeit eventually might guide toward a global treaty, as it happened before.\n\nAnd, actually, having access (meaning: humans with relevant expertise, not just documents) to all the history of treaty definition, and all the diplomatic organizational memory, across an array of issues since at least after WWII, could help in having a multidisciplinary, pre-emptive approach to regulation.\n\nA June 2022 paper went a step further: looking at what is currently available and being done to implement guidelines fro AI ethics: [Worldwide AI Ethics: a review of 200 guidelines and recommendations for AI governance](https://arxiv.org/abs/2206.11922).\n\nThe sheer number, 200 (those that they selected) is already an indication of an issue: we have a technology that has global reach, whose ethics are global by definition, and yet we have hundreds of different guidelines.\n\nIf there were a shared framework that is considered foundational, most of those could refer to the shared framework, and then add specific additional elements.\n\nThe authors themselves consider those 200 a \"sample\"- *\"several organizations have produced documents intended to standardize, in the normative sense, and promote guidance to our recent and rapid AI development. However, the full spectrum of ideas presented in these documents has not yet been analyzed, except for a few\nmeta-analyses and critical reviews of the field\"*.\n\nRecent years saw an increase in fines charged to multinational companies operating within the European Union (but also fines from other jurisdictions) all related to data-processing, and the sheer volume of data implies that quite often some AI elements are involved.\n\nFor any organization that cannot restrict itself to a single country, it makes therefore sense to support harmonization efforts.\n\nWhile it is to be expected lobbying to \"harmonize to the bottom\" (i.e. toward rules that have less impacts or constrain less future choices), the opening salvo of the EU AI Act might do for AI ethics what GDPR did for data privacy: raising the bar of compliance.\n\nA side-effect might to increase the cost of access to a market- if compliance adds significant layers of organizational complexity, this could imply lowering the number of potential actors.\n\nHence, both researchers from the business side and academia should keep a dialogue with legislative decision-makers, to avoid stifling innovation, while at the same time providing safeguards that, as shown in the above mentioned surveys, are of concern to the public at different levels in different countries.\n\n","metadata":{}},{"cell_type":"markdown","source":"# Section 4: Considerations and conclusions\n\nThe key element about AI ethics?\n\nTo paraphrase somebody else, AI ethics is too important to be left to experts in either AI or ethics.\n\nAs described in previous sections, papers on arXiv on the subject of AI ethics over the last few years show an increased interest in identifying a new \"normal\" that could be applied worldwide, as AI applications and services have a global scope by nature.\n\nWhile arXiv has a significant value to the STEM community, an extended study of AI ethics should also include other non-STEM sources: something that is outside the scope of this essay.\n\nAnyway, arXiv papers related to AI ethics already include non-STEM authors or themes but discussing STEM themes, and probably this multidisciplinary dialogue will further extend.\n\nFurther developing the AI ethics field would require, as with any other field, what a paper with the title [Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society](https://arxiv.org/abs/2001.04335) summarized as: *\"moving towards a more nuanced conversation about research\npriorities can help establish new opportunities for collaboration, aid the development of more consistent and coherent research agendas, and enable identification of previously neglected research areas.\"*\n\nIn current discourse on media, often AI ethics is described as ex-post, an add-on to products and services, as decades ago happened with quality: you added quality.\n\nWhile more technical activities could benefit from vertical specialization, AI ethics is characteristically multidisciplinary- as assessing its impacts does require including those representing domains that might be impacts.\n\nInteresting therefore an approach tried to get already in the funding (\"business case\" phase, in business) a systemic view of ethics, as discussed in this paper [ESR: Ethics and Society Review of Artificial Intelligence Research](https://arxiv.org/abs/2106.11521).\n\nIf you do not have time to read the paper, it is worth at least to read its abstract: *\"Artificial intelligence (AI) research is routinely criticized for its real and potential impacts on society, and we lack adequate institutional responses to this criticism and to the responsibility that it reflects. AI research often falls outside the purview of existing feedback mechanisms such as the Institutional Review Board (IRB), which\nare designed to evaluate harms to human subjects rather than harms to human society. In response, we have developed the Ethics and Society Review board (ESR), a feedback panel that works with researchers to mitigate negative ethical and societal aspects of AI research. The ESR’s main insight is to serve as a requirement for\nfunding: researchers cannot receive grant funding from a major AI funding program at our university until the researchers complete the ESR process for the proposal. In this article, we describe the ESR as we have designed and run it over its first year across 41 proposals. We analyze aggregate ESR feedback on these proposals,\nfinding that the panel most commonly identifies issues of harms to minority groups, inclusion of diverse stakeholders in the research plan, dual use, and representation in data. Surveys and interviews of researchers who interacted with the ESR found that 58% felt that it had influenced the design of their research project, 100% are willing to continue submitting future projects to the ESR, and that they sought additional scaffolding for reasoning through ethics and society issues.\"*.\n\nThe results were so significant, that in the conclusions was stated *\"supporting early-stage feedback on societal and ethical dimensions of AI research. The rocess, which we termed Ethics and Society Board (ESR), inserts itself into the grant funding process as a lever for reflection. By drawing on a cross-campus iterdisciplinary panel, researchers have a chance to engage in this reflection early on in the lifecycle of their projects. Moving forward, we are fielding interst in expanding this program beyond AI research. Other areas, such as public interst terchnology, sustainability, bioengineering and data science face similar questions.\"*\n\nOr: being AI ethics something that is multidisciplinary, and whose impacts might be global, could actually improve decision-making processes across the whole spectrum of research activities, introducing approaches that are relevant to any data-centric activity.\n\nA first case of application worth watching (and the reason why also the European Central Bank was involved within the EU AI Act) is the financial sector.\n\nCurrently, older generations still associate the financial sector with buildings, banks, etc.\n\nIn reality, fintech and others that represent de facto a \"shadow banking system\" (they are not bank but do everything a bank would do), and impact on any industry.\n\nIn a data-centric society, for those who live in data-rich environments, eventually data will have a value, and will be traded as such, probably at a speed and with a frequency high enough to require the intervention of intelligent agents.\n\nTherefore, interesting [On the Current and Emerging Challenges of Developing Fair and Ethical AI Solutions in Financial Services](https://arxiv.org/abs/2111.01306).\n\nFrom the papers published over the last few years on arXiv, and the trend in regulations, I think that it looks really familiar what is being developed within the AI ethics domain.\n\nIt is something that was described by John Rawls in the early 1970, within his \"A Theory of Justice\", specifically the section on \"A Veil of Ignorance\", page 136 and following.\n\nThe concept? *\"The idea of the original position is to set up a fair procedure so that any principles agreed to will be just.\"*\n\nIf you are interested in the subject of AI ethics but are unwilling to read the papers listed in this essay, few hundred pages overall, I can suggest two (free) online courses on the them, both from SAP:\n- 2019: [Creating Trustworthy and Ethical Artificial IntelligenceIn cooperation with members of the European Union High-Level Expert Group on Artificial Intelligence](https://open.sap.com/courses/aie1-tl)\n- 2023: [AI Ethics at SAP](https://open.sap.com/courses/aie1/)\n\nThanks for reading so far.\n","metadata":{}},{"cell_type":"markdown","source":"# Appendix A: Bibliographical references","metadata":{}},{"cell_type":"markdown","source":"This is the tag cloud of the content (not just the abstracts) of the bibliography\n\n\nThis essay contains just 10 papers selected from those on arXiv as of 2023-07-02, few hundred pages.\n\nTo ease access to the bibliography (also because some papers will probably evolve), you can search which papers and chapters within papers, including this essay, are relevant to a specific term [at this address](https://robertolofaro.com/searchkaggleaiethics.php).\n\nThis is just a beta of the search facility.\n\nIt replicates and adapts something I added many years ago for my own books and other publications (see e.g. [the ECB speech tag cloud search, online since 2019](https://robertolofaro.com/ECBSpeech_index.php))\n\nWhile this essay will stay \"as is\" (to keep the reference point 2020-2023), the tag cloud online (and associated bibliography) will be updated on a monthly basis, on the 11th of each month (starting 2023-08-11), released the following week-end, to mirror the evolution of the AI ethics field.\n","metadata":{}},{"cell_type":"markdown","source":"\n## SECTION 1 INTRODUCTION AND CONCEPT\n\n01 [The AI Index 2022 Annual Report](https://arxiv.org/abs/2205.03468)\n\n02 [2023 Kaggle AI Report](https://www.kaggle.com/competitions/2023-kaggle-ai-report/) \n\n03 [arXiv website](https://arxiv.org)\n\n04 [2023 Kaggle AI Report](https://www.kaggle.com/competitions/2023-kaggle-ai-report/data)\n\n05 [JSON file provided by Cornell on Kaggle](https://www.kaggle.com/datasets/Cornell-University/arxiv)\n\n\n## SECTION 2 HISTORY AND PROSPECTIVE\n\n01 [A Framework for Ethical AI at the United Nations\n](https://arxiv.org/abs/2104.12547)\n\n02 [AlphaGo](https://en.wikipedia.org/wiki/AlphaGo)\n\n03 [Exciting, Useful, Worrying, Futuristic: Public Perception of Artificial Intelligence in 8 Countries](https://arxiv.org/abs/2001.00081)\n\n04 [Ethics of AI: A Systematic Literature Review of Principles and Challenges](https://arxiv.org/abs/2109.07906)\n\n05 [AI Ethics Issues in Real World: Evidence from AI Incident Database](https://arxiv.org/abs/2206.07635)\n\n06 [Recommendation of the Council on Artificial Intelligence](https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449)\n\n07 [Ethics Guidelines For Trustworthy AI](https://ec.europa.eu/newsroom/dae/document.cfm?doc_id=60419) \n\n08 [Recommendation of the Council on Artificial Intelligence](https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449)\n\n09 [Recommendation of the Council on Artificial Intelligence](https://unesdoc.unesco.org/ark:/48223/pf0000381137)\n\n10 [EU AI Act](https://www.europarl.europa.eu/news/en/headlines/society/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence?&at_campaign=20226-Digital&at_medium=Google_Ads&at_platform=Search&at_creation=RSA&at_goal=TR_G&at_advertiser=Webcomm&at_audience=ai%20eu&at_topic=Artificial_intelligence_Act&at_location=IT&gclid=Cj0KCQjwtO-kBhDIARIsAL6Lord2Lw-Jx1O7HtEVr_oVYiu8sLtN0YxuOjryfE-XNmljGFz2dD6h4h8aArLAEALw_wcB)\n\n11 ([see here the amendments](https://www.europarl.europa.eu/doceo/document/TA-9-2023-0236_EN.html)\n\n\n## SECTION 3 CURRENT TRENDS 2020-2023\n\n01 [The Different Faces of AI Ethics Across the World: A Principle-Implementation Gap Analysis](https://arxiv.org/abs/2206.03225)\n\n02 [A Framework for Ethical AI at the United Nations](https://arxiv.org/abs/2104.12547) \n\n03 [Worldwide AI Ethics: a review of 200 guidelines and recommendations for AI governance](https://arxiv.org/abs/2206.11922)\n\n\n## SECTION 4 CONSIDERATIONS AND CONCLUSIONS\n\n01 [Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society](https://arxiv.org/abs/2001.04335)\n\n02 [ESR: Ethics and Society Review of Artificial Intelligence Research](https://arxiv.org/abs/2106.11521)\n\n03 [On the Current and Emerging Challenges of Developing Fair and Ethical AI Solutions in Financial Services](https://arxiv.org/abs/2111.01306)\n\n04 [Creating Trustworthy and Ethical Artificial IntelligenceIn cooperation with members of the European Union High-Level Expert Group on Artificial Intelligence](https://open.sap.com/courses/aie1-tl)\n\n05 [AI Ethics at SAP](https://open.sap.com/courses/aie1/)","metadata":{}},{"cell_type":"markdown","source":"# Appendix B: Data and charts from arXiv","metadata":{}},{"cell_type":"markdown","source":"\n## Data from arXiv queries:\n\n\n\n\n## Data about the sentiment groups:\n\nThe data were extracted from this paper: [Exciting, Useful, Worrying, Futuristic: Public Perception of Artificial Intelligence in 8 Countries](https://arxiv.org/abs/2001.00081) - see Section 2 for a discussion.\n\n\n| Country | Australia | Canada | USA | South Korea | France | Brazil | India | Nigeria |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Exciting | 17% | 14% | 15% | 6% | 10% | 23% | 36% | 25% |\n| Useful | 9% | 9% | 7% | 19% | 11% | 14% | 18% | 11% |\n| Worrying | 31% | 33% | 30% | 14% | 31% | 21% | 9% | 11% |\n| Futuristic | 22% | 21% | 19% | 38% | 20% | 34% | 24% | 19% |\n| None | 38% | 39% | 42% | 31% | 39% | 25% | 27% | 41% |\n\n\"Regarding ethical resources, our findings reinforce the importance of designing and developing AI responsibly to benefit society and minimize potential harms, and of sharing information about those efforts. \"\n\n\n## List of charts (already within the text)\n\n### SECTION 1 INTRODUCTION AND CONCEPT\n\n01 \n\n### SECTION 2 HISTORY AND PROSPECTIVE\n\n01 \n\n02 \n\n### SECTION 3 CURRENT TRENDS 2020-2023\n\n01 \n\n02 \n\n03 \n\n04 \n\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}