---
title: 'Topic Salience in the Academic Literature'
author: 'Dylan Waste'
date: '`r Sys.Date()`'
output:
  html_document:
    number_sections: true
    toc: true
---

# Introduction & Research Question
While Artificial Intelligence (AI) and Machine Learning (ML) development has almost occurred for the past century, recent public strides in the field (e.g., Google Translate, AlexNet, ChatGPT, etc.) have brought unprecedented attention to this nascent discipline. Even though a significant amount of this buzz has been purely speculative, AI and ML systems seem to get qualitatively more powerful year after year. Clearly, AI will be able to play a transformational role in our society. In particular, the broad applicability of ML to research problems in fields ranging from biomedical engineering to digital humanities means that developments in the area can be finetuned for various purposes. Regardless of where AI will be implemented, we humans are likely to see some downstream effects of these technologies: access to expert-system doctors, having a loan application denied by decision trees, etc. Therefore, this project attempts to deconstruct some important subtopics of AI Ethics to investigate their salience in the greater literature, questioning how researchers are considering the social implications of their work.

# Discussion of Ethics, Safety, & Accessibility
In light of the transformative social implications posed by Artificial Intelligence (AI) and Machine Learning (ML) systems, it becomes imperative to examine their ethical, safety, and accessibility dimensions. Discussion surrounding these critical aspects emphasizes the need for robust governance frameworks, academic research, and ethical considerations that extend beyond mere safety and governance. Ensuring the safety of emerging technologies encompasses privacy, security, and fairness, necessitating robust governance frameworks consisting of regulations and standards to mitigate risks and promote responsible use (Zuiderwijk, et al., 2021). It is crucial to approach these frameworks with critical perspectives, questioning assumptions, biases, unintended consequences, and power dynamics that may arise. 

Peter Suber’s book Open Access describes that, “Access gaps are worse at other affluent institutions, and worse still in the developing worlds. In 2008, Harvard subscribed to 98,900 serials and Yale to 73,900. The best-funded research library in India, at the Indian Institute of Science, subscribed to 10,600” (Suber, 2012). This disparity in access to academic literature highlights the significant difference in accessibility between institutions in the Global North and the Global South. 

Academic literature plays a vital role in exploring the evolving landscape of ethics, safety, and accessibility in emerging technologies. Scholars actively investigate the ethical dimensions of these technologies, contributing valuable insights into their societal impact (Viana-Lora & Nel-lo-Andreu, 2021). Ethical principles in research and development, rigorous ethical review processes, interdisciplinary collaborations, and open dialogue are promoted within the academic community to ensure that emerging technologies are developed and deployed with careful consideration of their ethical and social implications. Open access initiatives ensure that research findings are widely available, fostering transparency, knowledge dissemination, and public engagement, allowing for scrutiny, discussion, and integration into the wider social discourse. 

While access to knowledge and emerging technologies is essential for global AI accessibility, these programs do not go far enough in ensuring that all sorts of groups can leverage the transformative potential of AI. Despite the limitations of open-access initiatives, open-source technologies shatter development barriers and allow anyone to alter a given technology. As defined by IBM “Open source software is software developed and maintained via open collaboration, and made available, typically at no cost, for anyone to use, examine, alter and redistribute however they like. This contrasts with proprietary or closed source software applications—e.g. Microsoft Word, Adobe Illustrator—which are sold to end users by the creator or copyright holder, and cannot be edited, enhanced or redistributed except as specified by the copyright holder.” 

Open source software and frameworks as described by IBM, facilitate transparency, collaboration, and community involvement in technology development. By providing access to underlying code, they foster innovation, peer review, and collective scrutiny, allowing for the development of more ethical and accessible solutions (Botelho, 2021). This inclusive approach, combined with the design of technologies with accessibility in mind, ensures inclusivity, avoids discrimination, and promotes equal opportunities. Additionally, explainability is crucial to empower users to understand and question the decisions made by automated systems, reducing the risks of bias and potential harm. 

By integrating robust governance frameworks, critical perspectives, academic research, open access, and open source initiatives, emerging technologies can be developed and utilized responsibly and ethically. This comprehensive approach ensures the protection of individuals well-being, promotes inclusivity, and fosters transparency and accountability in the development and use of technology (Kooli, 2023). It is through these collective efforts that society can harness the potential benefits of AI and ML while addressing the ethical challenges and ensuring a more equitable and accessible technological landscape.

In addition to the discussions surrounding ethics, safety, and accessibility in emerging technologies, quantitative research projects can provide further insights into the trends and salience of topics in academic literature. 

# Data Collection and Preprocessing
These data were collected from the arXiv dataset from Kaggle. arXiv is an open access academic publishing platform that allows researchers to host their articles that have not necessarily gone through peer review. For measuring developments in AI and ML, in particular, the arXiv data set is an ideal corpus because the lightning-fast speed of development means that cutting-edge models/techniques become redundant quickly and therefore looking at only peer-reviewed sources washes out a lot of this dynamism.

Despite the analytical potential of this dataset of 2.3 research paper abstracts, there is a lot of depth missing when deciding to exclusively analyze these documents. If computational and technical constraints were not a concern, this project would have attempted to leverage actual text information from inside each document in arXiv. Still though, abstracts are supposed to describe the key concepts and steps taken in a given research project, so we can expect that these are largely congruent with the actual content of the papers.

Initially, I created an analysis of arXiv data for a final project for an R course I took in the Fall at university, so I decided to stick with the language for this project–this proved to be quite challenging. Primarily stemming from memory constraints, performing operations in R on the entire 2,292,057 documents arXiv corpus caused me to download the .json document in Python as a .CSV before uploading the document to this RMarkdown file. This allowed me to read the corpus in as a data.table object and perform much quicker operations

After collecting the appropriate data and uploading it as a data.table item, a set of typical text cleaning procedures were performed on the abstract: stopword/artifact removal, making all characters lowercase, etc. In particular, the removal of artifacts such as \n was critical because the abstracts in the .json file are not already cleaned so they frequently contain random characters that can seriously impact this analysis. Performing these operations enabled the most accurate next analysis, so this portion of the preprocessing pipeline is extremely important.

Even though there is already a good idea of what terms are relevant to this study, looking at the corpus with a preconceived notion of importance can lead to a biased analysis. After taking a critical perspective on the impact of these preconceptions, it became incredibly evident that some sort of exploratory data analysis of the context surrounding our terms of interest was necessary. After the arXiv was properly preprocessed, different TF-IDF was taken to select other relevant words in each subtopic  (e.g., bias, explainability, open source,  etc.).


```{r include = FALSE}
library(jsonlite)
library(tidyverse)
library(tidytext)
library(RColorBrewer) # metapackage of all tidyverse packages
```

```{r include = FALSE}
list.files(path = "../input")
```

```{r include = FALSE}
# Initialize an empty list
df <- data.table::fread("../input/arxiv-fullsetcsv/arxiv-metadata-oai-snapshot.csv")
```

```{r echo = FALSE}
message <- paste("Total Paper Count:", nrow(df))
print(message)
```

```{r include = FALSE}
# Convert the update_date column to a date object
df$update_date <- as.Date(df$update_date)

# Extract the year and remove it from the update_date column
df <- df %>%
  mutate(year = year(update_date), 
         month = month(update_date), 
         month_year = paste(month(update_date), year(update_date), sep = "-"))
```

```{r echo = FALSE}
# Okabe-Ito palette
okabe_ito <- c("#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7")

# Manually specify the colors for each term
my_colors <- c("AI" = okabe_ito[1], 
               "CL" = okabe_ito[2], 
               "ML" = okabe_ito[3], 
               "Computer Vision" = okabe_ito[4], 
               "Robotics" = okabe_ito[5], 
               "Multi-agent Systems" = okabe_ito[6])

search_terms <- function(df) {
  df <- df %>% 
    mutate(contains_ai = str_detect(categories, "cs.AI"), 
           contains_cl = str_detect(categories, "cs.CL"),
           contains_ml = str_detect(categories, "cs.LG"),
           contains_computer_vision = str_detect(categories, "cs.CV"),
           contains_robotics = str_detect(categories, "cs.RO"),
           contains_multiagent_systems = str_detect(categories, "cs.MA")) %>% 
    group_by(year) %>% 
    summarise(total = n(),
              mentions_ai = sum(contains_ai) / total * 100, 
              mentions_cl = sum(contains_cl) / total * 100,
              mentions_ml = sum(contains_ml) / total * 100,
              mentions_computervision = sum(contains_computer_vision) / total * 100,
              mentions_robotics = sum(contains_robotics) / total * 100,
              mentions_multiagent_systems = sum(contains_multiagent_systems) / total * 100) %>% 
    gather(key = "term", value = "percentage", -year, -total) %>%
    mutate(term = recode(term, 
                         mentions_ai = "AI", 
                         mentions_cl = "CL",
                         mentions_ml = "ML",
                         mentions_computervision = "Computer Vision",
                         mentions_robotics = "Robotics",
                         mentions_multiagent_systems = "Multi-agent Systems"))
  
  ggplot(df, aes(x = year, y = percentage, color = term)) +
    geom_line() +
    labs(x = "Year", y = "Percentage", 
         title = "Yearly percentage of Artificial Intelligence Papers on arXiv",
         color = "Term") +
    scale_color_manual(values = my_colors) +
    theme_minimal() +
    theme(axis.title.x = element_text(hjust = 0.5),
          axis.title.y = element_text(hjust = 0.5))
}

# Replace df with your actual data frame
search_terms(df)
```

```{r include = FALSE}
df <- df %>%
  mutate(os_present = ifelse(str_detect(abstract, "open source"), 1, 0),
         bias_present = ifelse(str_detect(abstract, "bias"), 1, 0),
         ethics_present = ifelse(str_detect(abstract, "ethics"), 1, 0),
         safety_present = ifelse(str_detect(abstract, "safety"), 1, 0),
         explanibility_present = ifelse(str_detect(abstract, "explainability"), 1, 0),
         class = case_when(
           str_detect(categories, "cs.AI") ~ "cs.AI",
           str_detect(categories, "cs.CL") ~ "cs.CL",
           str_detect(categories, "cs.LG") ~ "cs.LG",
           str_detect(categories, "cs.CV") ~ "cs.CV",
           str_detect(categories, "cs.RO") ~ "cs.RO",
           str_detect(categories, "cs.MA") ~ "cs.MA",
           TRUE ~ "Other"))

df <- df %>%
  filter(class != "Other")
```

```{r echo = FALSE}
message <- paste("AI-related Paper Count: ", nrow(df))
print(message)
```

```{r include = FALSE}
category_words <- df %>%
  unnest_tokens(word, abstract) %>%
  filter(!str_detect(word, "^(\\\\[A-Z,a-z])"), word != "") %>%
  anti_join(stop_words) %>%
  count(class, word, sort = TRUE)

total_categories <- category_words %>%
  group_by(class) %>% 
  summarize(total = sum(n))

category_words <- left_join(category_words, total_categories)

category_words <- category_words %>%
  bind_tf_idf(word, class, n)
```

```{r echo = FALSE}
category_words %>%
  filter(class == "cs.CL") %>%
  select(-total) %>%
  arrange(desc(tf_idf)) %>%
  slice_head(n = 15)

category_words %>%
  filter(class == "cs.RO") %>%
  select(-total) %>%
  arrange(desc(tf_idf)) %>%
  slice_head(n = 15)
```

```{r include = FALSE}
my_colors <- c("Open Source" = okabe_ito[1], 
               "Bias" = okabe_ito[2], 
               "Ethics" = okabe_ito[3], 
               "Safety" = okabe_ito[4], 
               "Neural Networks" = okabe_ito[5], 
               "Big Data" = okabe_ito[6], 
               "Transformer" = okabe_ito[7])
```

```{r echo = FALSE}
search_terms <- function(df) {
  df <- df %>% 
    mutate(contains_opensource = str_detect(abstract, "open source|opensource"), 
           contains_bias = str_detect(abstract, "bias|Bias"), 
           contains_ethics = str_detect(abstract, "ethics|Ethics"), 
           contains_safety = str_detect(abstract, "safety|Safety"),
           contains_nn = str_detect(abstract, "neural networks|neuralnetworks|NN"),
           contains_big_data = str_detect(abstract, "big data|BigData|bigdata"),
           contains_transformer = str_detect(abstract, "transformer|Transformer")) %>% 
    group_by(year) %>% 
    summarise(total = n(),
              mentions_opensource = sum(contains_opensource) / total * 100, 
              mentions_ethics = sum(contains_ethics) / total * 100,
              mentions_safety = sum(contains_safety) / total * 100,
              mentions_bias = sum(contains_bias) / total * 100,
              mentions_nn = sum(contains_nn) / total * 100,
              mentions_big_data = sum(contains_big_data) / total * 100,
              mentions_transformer = sum(contains_transformer) / total * 100) %>% 
    gather(key = "term", value = "percentage", -year, -total) %>% 
    mutate(term = recode(term, 
                         mentions_opensource = "Open Source", 
                         mentions_bias = "Bias",
                         mentions_ethics = "Ethics",
                         mentions_safety = "Safety",
                         mentions_nn = "Neural Networks",
                         mentions_big_data = "Big Data",
                         mentions_transformer = "Transformer"))
  
  ggplot(df, aes(x = year, y = percentage, color = term)) +
    geom_line() +
    labs(x = "Year", y = "Percentage", 
         title = "Yearly percentage of abstracts mentioning terms of interest",
         color = "Term") +
    scale_color_manual(values = my_colors) +
    theme_minimal() +
    theme(axis.title.x = element_text(hjust = 0.5),
          axis.title.y = element_text(hjust = 0.5))
}

# Replace df with your actual data frame
search_terms(df)
```

# Discussion on the Larger Social Implications of Academic and Scientific Literature 
The increasing trends in academic and scientific literature in the field of Artificial Intelligence (AI) and Machine Learning (ML) have a significant impact on shaping and influencing broader social adoption and discussion. The social impact of research is an emerging topic, with a growing number of publications in the past decade as funders prioritize social criteria. (Wang et al., 2022), However, there is a lack of systematic review across research fields. (Viana-Lora & Nel-lo-Andreu, 2021). Many studies attempt to empirically assess research impact, but the absence of standardized evaluation methods makes it complex. Therefore, adopting a mix of quantitative and qualitative approaches can ensure a more holistic assessment of social impact.


Here are a few key ways in which scientific and academic literature in AI and ML have impacted and influenced larger social trends: 

Policy Development: Policymakers rely on scientific literature to understand the impact and effectiveness of different AI and ML approaches, leading to evidence-based policies and regulations that influence societal practices. In the context of India, policymakers have embraced the potential of AI and ML for inclusive development and economic growth. The country has taken significant steps to leverage digital technologies, such as AI, to drive inclusive development through widespread internet access and affordable data. (Elias, 2023) For example, India has implemented large-scale digital ID and payment systems like Aadhaar and UPI, which have facilitated efficient government services, digitization of documents, and platforms for COVID-19 vaccination. These initiatives demonstrate the commitment to harnessing AI for social empowerment and improving accessibility to government services. Additionally, India's focus on "AI for All" reflects the intention to utilize AI tools in sectors like healthcare, agriculture, identity verification, and crowd management, both in the public and private sectors. These initiatives align with India's vision of becoming an economic superpower while prioritizing social inclusion and empowerment. (Elias, 2023)  Policymakers in India, supported by scientific literature, are actively driving the adoption and regulation of AI and ML technologies to shape societal practices and achieve their goals of inclusive development and economic growth.

Education: Moreover, scholarly publications contribute to public awareness and education. They disseminate research outcomes to a wider audience, educating and informing the public about critical AI-related issues such as ethics, privacy, and social implications. As people become aware of new knowledge, it can influence their perceptions, beliefs, and behaviors, driving social trends related to AI adoption and use. When this knowledge reaches a broader audience, including policymakers, professionals, and the general public, it stimulates discussions and informs decision-making processes that shape social trends. 

New AI-powered chatbots like ChatGPT are being tested as personalized tutors, providing customized explanations, practice, and feedback to students. (Singer, 2023) This innovation has the potential to supplement human teachers, offering one-on-one academic support at scale, tailored to the unique needs of each student. By leveraging AI technology, these chatbot tutors can democratize education, making personalized learning accessible to everyone, regardless of their geographical location, socioeconomic background, or resource limitations. This development opens up new possibilities for open access to technology and education, leveling the playing field and ensuring that every individual has an equal opportunity to learn and thrive. It is through scholarly publications and discussions like this that the potential of AI in education for open access becomes evident, highlighting the positive impact it can have on society by making education available to all.

Accessibility: Scientific and academic literature in the field of Artificial Intelligence (AI) and Machine Learning (ML) has played a transformative role in enhancing accessibility for individuals with disabilities, as highlighted by various studies and initiatives. New technologies utilizing AI have emerged to address the specific needs of students with disabilities in educational settings, paving the way for inclusive learning experiences. For autistic students, robots provide social modeling and feedback, aiding in communication skill development and fostering motivation. AI reading tutors offer real-time guidance to dyslexic students, supporting literacy improvement. AI-enabled devices audibly describe visual content, enabling blind students to access educational materials, while captioning and sign language avatars facilitate real-time translation for deaf students. These advancements aim to increase engagement, motivation, and accessibility for students with disabilities, empowering them to thrive academically. (Tugend, 2022)  

As researchers continue to explore customizable AI tutors and virtual peers, the potential for personalized academic support expands further. However, careful testing and thoughtful design, accompanied by teacher oversight, are essential to ensure the intended benefits are realized without unintended consequences. (Botelho, 2021) The integration of AI in special education holds significant promise for positively impacting learning outcomes and fostering a more inclusive educational environment.

Healthcare: Scientific and academic literature in the field of AI and ML has had a profound impact on social trends, particularly in the realm of global health. The literature highlights the enormous potential of AI to enhance health outcomes worldwide. However, it also emphasizes the importance of responsible development to prevent further health inequities. Many AI tools are developed and tested using data from high-income populations, leading to an implementation gap when deployed in low-resource settings. (Hosny, 2019) 

To address this, the literature calls for diverse global participation in building, validating, and governing these technologies. Collaboration and data sharing across nations are essential for creating equitable AI tools that work effectively across different populations. Evaluating AI systems for their performance across diverse groups and optimizing them for local contexts before widespread deployment is crucial. By considering inclusivity, the literature guides the creation of socially responsible AI technologies that can expand access to quality healthcare globally while avoiding unintended biases that could harm marginalized communities.

It is important to note that the impact of scientific and academic literature on social trends is not immediate or linear. It occurs through a complex interplay of factors, including the dissemination of knowledge, public understanding, policy changes, and cultural shifts (Wang, Jiachen, et al., 2022). Nonetheless, the cumulative effect of research findings can have a profound influence on societal adoption and discussion, shaping the way we think, act, and progress as a society.

# Conclusion
In all, this project lays the groundwork for a more nuanced understanding of the salience of ethics, accessibility, and their related topics in AI research. While we can see that these topics have become more frequently mentioned in papers related to AI over the past decade, these core concepts are still referenced in abstracts at quite a low rate. Despite the potential transformational impact that ethically-aligned and human-centered AI systems can have on our society at large (highlighted in the discussion section of this paper), a significant amount of research is failing to promote these concepts and ideas. As AI publications continue to make up a greater share of the research article output,  it is imperative that more researchers consider the social implications of their work.

# References 
Botelho, Fernando H. “Accessibility to Digital Technology: Virtual Barriers, Real Opportunities.” Assistive Technology, vol. 33, no. sup1, 2021, pp. 27–34, https://doi.org/10.1080/10400435.2021.1945705. 

Elias, Jibu. “AI for All: How India Is Carving Its Own Path in the Global Ai Race.” YouTube, 30 Jan. 2023, www.youtube.com/watch?v=smWW-FOlpl4. 

Hosny A, Aerts HJWL. Artificial intelligence for global health. Science. 2019 Nov 22;366(6468):955-956. doi: 10.1126/science.aay5189. PMID: 31753987; PMCID: PMC7790340.

Kooli, Chokri. “Chatbots in Education and Research: A Critical Examination of Ethical Implications and Solutions.” Sustainability, vol. 15, no. 7, 2023, p. 5614, https://doi.org/10.3390/su15075614.  

Singer, Natasha. “New A.I. Chatbot Tutors Could Upend Student Learning.” The New York Times, 8 June 2023, www.nytimes.com/2023/06/08/business/khan-ai-gpt-tutoring-bot.html. 

Suber, Peter. Open Access. The MIT Press, 2012. 

Tugend, Alina. “How Robots Can Assist Students with Disabilities.” The New York Times, 29 Mar. 2022, www.nytimes.com/2022/03/29/technology/ai-robots-students-disabilities.html. 

Viana-Lora, Alba, and Marta Gemma Nel-lo-Andreu. “Approaching the Social Impact of Research through a Literature Review.” International Journal of Qualitative Methods, vol. 20, 2021, p. 160940692110521, https://doi.org/10.1177/16094069211052189. 

Wang, Jiachen, et al. “When Research Topic Trend Prediction Meets Fact-Based Annotations.” Data Science and Engineering, vol. 7, no. 4, 2022, pp. 316–327, https://doi.org/10.1007/s41019-022-00197-1. 

Zuiderwijk, Anneke, et al. “Implications of the Use of Artificial Intelligence in Public Governance: A Systematic Literature Review and a Research Agenda.” Government Information Quarterly, vol. 38, no. 3, 2021, p. 101577, https://doi.org/10.1016/j.giq.2021.101577. 

# Generative AI Use Disclaimer
This report used GPT-4 and Claude 2.0 for coding assistance, expanding the writing of draft notes, and grammatical checks. However, all content, ideas, and citations were collected and developed without this assistance. These tools were leveraged for task acceleration, but not a source of original contributions to the content of this review.

```{r}
# Create the dataframe
sub <- tibble(
  type = c("essay_category", "essay_url", "feedback1_url", "feedback2_url", "feedback3_url"),
  value = c("Kaggle Competitions", "https://www.kaggle.com/code/dylanwaste/topic-salience-in-the-academic-literature", "https://www.kaggle.com/code/mistylight/mini-giants-small-language-models/comments#2345993", "https://www.kaggle.com/code/sohamjangra/ai-ethics-balancing-progress-and-responsibility/comments#2346010", "https://www.kaggle.com/code/luckypen/actionable-ai-ethics/comments#2346005"))

## Write to CSV
write_csv(sub, "submission.csv")

head(sub)

```