","metadata":{}},{"cell_type":"markdown","source":"This notebook demonstrates techniques of web scpaping and basic EDA.\n\nFeatures of this notebook:\n- Web Scrapping with BeautifulSoup;\n- Use cache of downloaded web pages;\n- Some measures against blocking as a bot;\n- Basic EDA.\n\n# Table of contents\n\n\n\n- [1 | Scrap the Web](#1)\n\n- [2 | Basic EDA](#2)\n\n- [3 | Conclusion](#3)\n\n\n\n\n\n# 1 | Scrap the Web\n\n\n\"Ethics of web scraping: It's important to note that web scraping can be unethical if it's used to collect private or sensitive data without the owner's consent. Always check the website's terms of use and privacy policy before scraping.\" [How to Get Started with Web Scraping: Tips and Tools for Beginners](https://www.kaggle.com/discussions/getting-started/414210)\n\nWebsite: [Preprints.org](https://www.preprints.org).","metadata":{}},{"cell_type":"code","source":"#import necessary libraries\n\nimport numpy as np\nimport pandas as pd\nfrom bs4 import BeautifulSoup\nimport requests\nfrom datetime import date\nimport time\nimport random\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2023-06-03T06:27:42.066785Z","iopub.execute_input":"2023-06-03T06:27:42.067346Z","iopub.status.idle":"2023-06-03T06:27:42.306646Z","shell.execute_reply.started":"2023-06-03T06:27:42.067316Z","shell.execute_reply":"2023-06-03T06:27:42.305784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# URL to scrap pageы with list of papers\nurl = 'https://www.preprints.org/subject/browse/all?page_num='","metadata":{"execution":{"iopub.status.busy":"2023-06-03T06:27:42.307984Z","iopub.execute_input":"2023-06-03T06:27:42.308700Z","iopub.status.idle":"2023-06-03T06:27:42.313681Z","shell.execute_reply.started":"2023-06-03T06:27:42.308657Z","shell.execute_reply":"2023-06-03T06:27:42.312650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use cache to store loaded pages locally (very useful for debugging)\n# Debugging may be required in case of changes in web page structure\n\n# Cache dataset can be created\n# cache_df = pd.DataFrame({\n# 'page':[],\n# 'soup':[]\n# })\n\n# Lets load the cache from file instead loading them\n# If you need a fresh data, create an empty cache dataset before the parcing\ncache_df = pd.read_csv('/kaggle/working/preprint_cache.csv')\n\ncache_df","metadata":{"execution":{"iopub.status.busy":"2023-06-03T06:27:42.315129Z","iopub.execute_input":"2023-06-03T06:27:42.315413Z","iopub.status.idle":"2023-06-03T06:27:48.928114Z","shell.execute_reply.started":"2023-06-03T06:27:42.315388Z","shell.execute_reply":"2023-06-03T06:27:48.926806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Parse the page n and put rows into papers_df dataset\n\ndef parse_page(n):\n\n soup = None\n\n if(len(cache_df[cache_df['page'] == n]) > 0):\n soup = cache_df[cache_df['page'] == n]['soup'].iloc[0]\n soup = BeautifulSoup(soup,\"lxml\")\n else:\n # Delay before next request to prevent possible blocking as a bot\n time.sleep(random.randint(500, 1000) / 200.0)\n\n # Some stuff against possible blocking as a bot\n # Taken from: https://stackoverflow.com/questions/43440397/requests-using-beautiful-soup-gets-blocked\n headers = requests.utils.default_headers()\n headers.update({'User-Agent':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'})\n link = requests.get(url + str(n), headers = headers)\n soup = BeautifulSoup(link.content,\"lxml\")\n cache_df.loc[len(cache_df.index)] = np.array([n, soup], dtype = object)\n\n # Start parsing\n \n # If number of pages is udefined, determine it\n global url_pages_total \n \n if(url_pages_total == 100000000):\n ultag = soup.find('ul', {'class':'pagination'})\n uls = ultag.find_all('li')\n url_pages_total = int(uls[4].string)\n\n\n # Parse next elements\n notes = soup.find_all('div', {'class':'search-content-box margin-serach-wrapper-left'})\n\n i = 0\n for note in notes:\n\n # Determine DOI\n doi = None # DOI may be missing\n\n doitag = note.find('span', {'class': 'content-box-header-element-5'})\n\n if doitag is not None:\n doitag = doitag.find('a')\n doi = doitag.string.strip()[4:]\n\n divs = note.find_all('div')\n\n atag = divs[2].find('a', href = True)\n\n title = atag.contents[0]\n\n # For full reference uncomment first line and comment second one line\n # href = 'https://www.preprints.org' + atag['href']\n href = atag['href'] \n\n abstract = note.find('div', {'class': 'abstract-content'}).contents\n abstract = repr(abstract)\n\n spans = note.find_all('span', {'class': 'search-content-header-label'})\n\n authors = []\n subject = []\n keywords = []\n\n authors_flag = False\n subject_flag = False\n keywords_flag = False\n\n sktags = note.find_all('a')\n for sktag in sktags:\n\n authors_flag = (sktag.has_attr('class')) and (sktag['class'][0] == 'author-selector')\n # subject_flag = (sktag.has_attr('href')) and ('/search?search_subject_area' in sktag['href'])\n subject_flag = (sktag.has_attr('href')) and ('/search' in sktag['href'])\n keywords_flag = (sktag.has_attr('href')) and ('keywords' in sktag['href']) and (not ('mailto' in sktag['href']))\n\n if (authors_flag):\n authors.append(sktag.string.strip())\n continue\n\n if (keywords_flag):\n ss = sktag.string\n if (ss is not None):\n keywords.append(ss.strip())\n continue \n \n if (subject_flag):\n sbcts = sktag.string.strip().split(',')\n for sbct in sbcts:\n subject.append(sbct.strip())\n continue\n \n continue # Just flag completion of the loop (for sktag in sktags)\n\n ss1 = repr(spans)\n ind1 = ss1.find('Online:')\n if (ind1 >= 0):\n ss2 = ss1[ind1 + 8:]\n ind2 = ss2.find('')\n \n online = ss2[:ind2]\n \n # Append to dataframe\n papers_df.loc[len(papers_df.index)] = np.array([\n doi,\n title,\n authors,\n subject,\n keywords,\n online,\n href,\n abstract\n ], dtype = object)\n \n return # The end of routine (def parse_page(n))","metadata":{"execution":{"iopub.status.busy":"2023-06-03T09:27:07.132098Z","iopub.execute_input":"2023-06-03T09:27:07.132508Z","iopub.status.idle":"2023-06-03T09:27:07.155858Z","shell.execute_reply.started":"2023-06-03T09:27:07.132478Z","shell.execute_reply":"2023-06-03T09:27:07.154590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Parce the pages\n# Notes: New insertion can occur during the scrapping. Copies of rows are possible.\n\n# Total number of pages\n# Will be corrected as soon as the 1st page will be scrapped\nurl_pages_total = 100000000\n\n# Create a Dataset for the papers\npapers_df = pd.DataFrame({\n 'doi':[],\n 'title':[],\n 'authors':[],\n 'subject':[],\n 'keywords':[],\n 'online':[],\n 'href':[],\n 'abstract':[]\n })\n\n\n# Set url to None to be sure the code will not request the Internet (silence mode)\n# Comment this line to download fresh data\nurl = None\n\n# Start from the page 1\nn = 1\n\nwhile (True):\n parse_page(n)\n print ('Page:', n, 'of', url_pages_total, 'parsed', end=\"\\r\")\n n += 1\n if (n > url_pages_total):\n break\n\n# Take a look at the papers\npapers_df","metadata":{"execution":{"iopub.status.busy":"2023-06-03T09:27:17.440277Z","iopub.execute_input":"2023-06-03T09:27:17.440701Z","iopub.status.idle":"2023-06-03T09:36:29.890679Z","shell.execute_reply.started":"2023-06-03T09:27:17.440668Z","shell.execute_reply":"2023-06-03T09:36:29.889564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the cache if need\n# cache_df.to_csv('/kaggle/working/preprint_cache.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2023-06-03T06:36:45.364219Z","iopub.execute_input":"2023-06-03T06:36:45.364490Z","iopub.status.idle":"2023-06-03T06:36:45.369024Z","shell.execute_reply.started":"2023-06-03T06:36:45.364467Z","shell.execute_reply":"2023-06-03T06:36:45.368251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# 2 | Basic EDA\n","metadata":{}},{"cell_type":"code","source":"# Get a basic info (some doi absent)\npapers_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-06-03T09:37:50.636622Z","iopub.execute_input":"2023-06-03T09:37:50.637862Z","iopub.status.idle":"2023-06-03T09:37:50.774778Z","shell.execute_reply.started":"2023-06-03T09:37:50.637819Z","shell.execute_reply":"2023-06-03T09:37:50.773627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Number of papers without doi\nlen(papers_df[pd.isnull(papers_df.doi)])","metadata":{"execution":{"iopub.status.busy":"2023-06-03T09:37:54.968436Z","iopub.execute_input":"2023-06-03T09:37:54.968845Z","iopub.status.idle":"2023-06-03T09:37:54.983684Z","shell.execute_reply.started":"2023-06-03T09:37:54.968812Z","shell.execute_reply":"2023-06-03T09:37:54.982959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Separate publication year into pubyear column\ndef get_year(dtstr):\n strs = dtstr.split(' ')\n return int(strs[2]) \n\npapers_df['pubyear'] = papers_df['online'].apply(lambda x: get_year(x))\npapers_df['pubyear']","metadata":{"execution":{"iopub.status.busy":"2023-06-03T09:37:58.067918Z","iopub.execute_input":"2023-06-03T09:37:58.068848Z","iopub.status.idle":"2023-06-03T09:37:58.144698Z","shell.execute_reply.started":"2023-06-03T09:37:58.068803Z","shell.execute_reply":"2023-06-03T09:37:58.143622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Number of publications per year\nvc = papers_df.pubyear.value_counts()\nplt.figure(figsize = (12,4))\nplt.bar(vc.index, vc.values)\nplt.title(\"Publications per year\")","metadata":{"execution":{"iopub.status.busy":"2023-06-03T09:38:00.732694Z","iopub.execute_input":"2023-06-03T09:38:00.733180Z","iopub.status.idle":"2023-06-03T09:38:01.066632Z","shell.execute_reply.started":"2023-06-03T09:38:00.733148Z","shell.execute_reply":"2023-06-03T09:38:01.064312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Collect Subject and Keywords for periods before 2021 and 2021 and later\nsubjects1 = set()\nsubjects2 = set()\nkeywords1 = set()\nkeywords2 = set()\n\nfor index, row in papers_df.iterrows():\n kwds = row['keywords']\n sbjcts = row['subject']\n if(row['pubyear'] < 2021):\n for sb in sbjcts:\n subjects1.add(sb)\n for kw in kwds:\n keywords1.add(kw)\n else:\n for sb in sbjcts:\n subjects2.add(sb)\n for kw in kwds:\n keywords2.add(kw)\n\n# Length of Subject and Keywords sets\nprint(len(subjects1))\nprint(len(subjects2))\nprint(len(keywords1))\nprint(len(keywords2))","metadata":{"execution":{"iopub.status.busy":"2023-06-03T09:38:06.459532Z","iopub.execute_input":"2023-06-03T09:38:06.459948Z","iopub.status.idle":"2023-06-03T09:38:09.529443Z","shell.execute_reply.started":"2023-06-03T09:38:06.459915Z","shell.execute_reply":"2023-06-03T09:38:09.528179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# New subjects (meems not related to AI)\nsubjects2 - subjects1","metadata":{"execution":{"iopub.status.busy":"2023-06-03T09:38:14.637559Z","iopub.execute_input":"2023-06-03T09:38:14.638015Z","iopub.status.idle":"2023-06-03T09:38:14.645568Z","shell.execute_reply.started":"2023-06-03T09:38:14.637978Z","shell.execute_reply":"2023-06-03T09:38:14.644194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# New keywords\n# The list is too large to view, thut the next line is commented\n# keywords2 - keywords1\n\n# Number of new keywords\nlen(keywords2 - keywords1)","metadata":{"execution":{"iopub.status.busy":"2023-06-03T09:38:17.016497Z","iopub.execute_input":"2023-06-03T09:38:17.017288Z","iopub.status.idle":"2023-06-03T09:38:17.034361Z","shell.execute_reply.started":"2023-06-03T09:38:17.017247Z","shell.execute_reply":"2023-06-03T09:38:17.033254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get number of papers with keyword in abstract\n# For period before 2021 and 2021 and later\ndef get_pn_for_keyword(kw):\n i1 = 0\n i2 = 0\n kwl = kw.lower()\n\n for index, row in papers_df.iterrows():\n abstr = repr(row['abstract']).lower()\n if (kwl in abstr):\n if(row['pubyear'] < 2021):\n i1 += 1\n else:\n i2 += 1\n return [i1, i2, i2 - i1]","metadata":{"execution":{"iopub.status.busy":"2023-06-03T09:54:20.020420Z","iopub.execute_input":"2023-06-03T09:54:20.020857Z","iopub.status.idle":"2023-06-03T09:54:20.028086Z","shell.execute_reply.started":"2023-06-03T09:54:20.020826Z","shell.execute_reply":"2023-06-03T09:54:20.026995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Analyses how many papers were published for each keyword\n\nAIKeywords = [\n 'Artificial intelligence',\n 'Big data',\n 'Chatbot',\n 'ChatGPT',\n 'Cognitive computing',\n 'Computer vision',\n 'Computational learning theory',\n 'Data mining',\n 'Data science',\n 'Deep learning',\n 'Machine intelligence',\n 'Machine learning',\n 'Machine translation',\n 'Artificial neutral network',\n 'Natural language generation',\n 'Natural language processing',\n 'Natural language understanding',\n 'Overfitting',\n 'Underfitting',\n 'Pattern recognition',\n 'Predictive analytics',\n 'Python',\n 'Reinforcement learning',\n 'Supervised learning',\n 'Unsupervised learning',\n 'Semantic annotation',\n 'Turing test'\n]\n\n# Create dataset for store results\npapers_by_kw = pd.DataFrame({'keyword':[], 'before2021':[],'2021andlater':[]}) \n\nn = 1\n\nfor kw in AIKeywords:\n pps = get_pn_for_keyword(kw)\n \n papers_by_kw.loc[len(papers_by_kw.index)] = np.array([\n kw,\n pps[0],\n pps[1]\n ], dtype = object)\n \n print ('Keyword:', n, 'of', len(AIKeywords), 'processed', end=\"\\r\")\n n += 1\n\npapers_by_kw","metadata":{"execution":{"iopub.status.busy":"2023-06-03T10:32:58.112845Z","iopub.execute_input":"2023-06-03T10:32:58.113931Z","iopub.status.idle":"2023-06-03T10:34:22.968634Z","shell.execute_reply.started":"2023-06-03T10:32:58.113888Z","shell.execute_reply":"2023-06-03T10:34:22.967816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# See published papers as a bar chart\n\nplt.figure(figsize = (12,4))\ndf = pd.DataFrame(\n {\n 'before 2021':list(papers_by_kw['before2021']),\n '2021 and after':list(papers_by_kw['2021andlater'])\n },\n index = AIKeywords)\n\nax = df.plot.barh()\nplt.ylabel('Keyword')\nplt.xlabel('Papers Published')\nplt.title('AI Related Papers')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-03T10:37:45.472679Z","iopub.execute_input":"2023-06-03T10:37:45.473674Z","iopub.status.idle":"2023-06-03T10:37:46.052699Z","shell.execute_reply.started":"2023-06-03T10:37:45.473635Z","shell.execute_reply":"2023-06-03T10:37:46.051667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# 3 | Conclusion\n\n\n\n\n\nThis notebook demonstrated a simple web scrapping of articles that can be used in further analyses by different deep methods.\n\n\nEDA shows a generally significant increase of AI related papers for 2021 and later in comparison with early published ones.\n\n\nAcknowledement: Creating of this notebook was inspired by the [How To Start Learning Web Scraping From Basic] (https://www.kaggle.com/discussions/getting-started/413850) post.\n\n","metadata":{}}]}