Coleridge Initiative - Show US the Data
Discover how data is used for the public good
Please dont forget to upvote the notebook
","metadata":{"papermill":{"duration":0.071487,"end_time":"2021-06-16T14:11:44.699809","exception":false,"start_time":"2021-06-16T14:11:44.628322","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## In this notebook basically we have to predict text for some strings by using nlp techniques\n*** \n>The objective of the competition is to identify the mention of datasets within scientific publications.\n\nThis competition challenges data scientists to show how publicly funded data are used to serve science and society. Evidence through data is critical if government is to address the many threats facing society, including; pandemics, climate change, Alzheimer’s disease, child hunger, increasing food production, maintaining biodiversity, and addressing many other challenges. Yet much of the information about data necessary to inform evidence and science is locked inside publications.\n\nCan natural language processing find the hidden-in-plain-sight data citations? Can machine learning find the link between the words used in research articles and the data referenced in the article?\n\nNow is the time for data scientists to help restore trust in data and evidence. In the United States, federal agencies are now mandated to show how their data are being used. The new Foundations of Evidence-based Policymaking Act requires agencies to modernize their data management. New Presidential Executive Orders are pushing government agencies to make evidence-based decisions based on the best available data and science. And the government is working to respond in an open and transparent way.\n\nThis competition will build just such an open and transparent approach. The results will show how public data are being used in science and help the government make wiser, more transparent public investments. It will help move researchers and governments from using ad-hoc methods to automated ways of finding out what datasets are being used to solve problems, what measures are being generated, and which researchers are the experts. Previous competitions have shown that it is possible to develop algorithms to automate the search and discovery of references to data. Now, we want the Kaggle community to develop the best approaches to identify critical datasets used in scientific publications.\n\nIn this competition, you'll use natural language processing (NLP) to automate the discovery of how scientific data are referenced in publications. Utilizing the full text of scientific publications from numerous research areas gathered from CHORUS publisher members and other sources, you'll identify data sets that the publications' authors used in their work.\n\nIt achieves this goal by working with the agencies to create value for the taxpayer from the careful use of data by building new technologies to enable secure access to and sharing of confidential microdata and by training agency staff to acquire modern data skills.\n\nIf successful, you'll help support evidence in government data. Automated NLP approaches will enable government agencies and researchers to quickly find the information they need. The approach will be used to develop data usage scorecards to better enable agencies to show how their data are used and bring down a critical barrier to the access and use of public data.","metadata":{"papermill":{"duration":0.07102,"end_time":"2021-06-16T14:11:44.843574","exception":false,"start_time":"2021-06-16T14:11:44.772554","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#Table of Content
\n* [1. Importing Modules & Libraries](#1)\n* [2. Data Exploration](#2)\n* [3. Data Vizualization](#3)\n* [4. Data Cleaning](#4)\n* [5. Baseline Model & Submission](#5)","metadata":{"papermill":{"duration":0.070983,"end_time":"2021-06-16T14:11:44.985416","exception":false,"start_time":"2021-06-16T14:11:44.914433","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#1. Importing Modules & Libraries
","metadata":{"papermill":{"duration":0.072463,"end_time":"2021-06-16T14:11:45.130487","exception":false,"start_time":"2021-06-16T14:11:45.058024","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)\n\nimport os\nimport re\nimport json\nimport time\nimport glob\nfrom collections import defaultdict\nfrom textblob import TextBlob\nfrom functools import partial\n\nimport pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport seaborn as sn\nimport plotly.express as px\nimport plotly.graph_objects as go\n\nimport nltk\nimport urllib.request\nfrom PIL import Image\nimport spacy\nnlp = spacy.load('en_core_web_lg', disable=['parser', 'ner'])\nnlp.max_length = 4000000\nfrom nltk.probability import FreqDist\nfrom wordcloud import WordCloud, STOPWORDS\n\nfrom tqdm.autonotebook import tqdm\nimport string\n\n%matplotlib inline\n\nos.listdir('/kaggle/input/coleridgeinitiative-show-us-the-data/')","metadata":{"papermill":{"duration":12.411484,"end_time":"2021-06-16T14:11:57.613052","exception":false,"start_time":"2021-06-16T14:11:45.201568","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:32:51.612530Z","iopub.execute_input":"2021-06-22T22:32:51.613244Z","iopub.status.idle":"2021-06-22T22:33:04.555769Z","shell.execute_reply.started":"2021-06-22T22:32:51.613114Z","shell.execute_reply":"2021-06-22T22:33:04.554746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are provided with 4 main pieces of data:\n\n* `train.csv:` The CSV file containing all the metadata of the publications, such as their title and the dataset they utilize.\n* `train:` The directory containing the actual publications that are referenced in train.csvin JSON format.\n* `test:` The directory containing the actual publications that will be used for testing purposes (thus, with no ground truth CSV file available).\n* `sample_submission.csv:` The CSV file containing all the publications IDs in the test set, for which we'll have to populate the prediction column.","metadata":{"papermill":{"duration":0.07102,"end_time":"2021-06-16T14:11:57.756794","exception":false,"start_time":"2021-06-16T14:11:57.685774","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#2. Data Exploration
","metadata":{"papermill":{"duration":0.07142,"end_time":"2021-06-16T14:11:57.899933","exception":false,"start_time":"2021-06-16T14:11:57.828513","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Reading csv files and train & test file paths","metadata":{"papermill":{"duration":0.07168,"end_time":"2021-06-16T14:11:58.042771","exception":false,"start_time":"2021-06-16T14:11:57.971091","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\nsample_sub = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')\ntrain_files_path = '../input/coleridgeinitiative-show-us-the-data/train'\ntest_files_path = '../input/coleridgeinitiative-show-us-the-data/test'","metadata":{"papermill":{"duration":0.21146,"end_time":"2021-06-16T14:11:58.325389","exception":false,"start_time":"2021-06-16T14:11:58.113929","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:33:04.557389Z","iopub.execute_input":"2021-06-22T22:33:04.557777Z","iopub.status.idle":"2021-06-22T22:33:04.696676Z","shell.execute_reply.started":"2021-06-22T22:33:04.557744Z","shell.execute_reply":"2021-06-22T22:33:04.695756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##Data Description
\ntrain.csv - labels and metadata for the training set train/test directory - the full text of the training/test set's publications in JSON format, broken into sections with section titles\n\n* `id` - publication id - note that there are multiple rows for some training documents, indicating multiple mentioned datasets.\n* `pub_title` - title of the publication (a small number of publications have the same title).\n* `dataset_title` - the title of the dataset that is mentioned within the publication.\n* `dataset_label` - a portion of the text that indicates the dataset.\n* `cleaned_label` - the dataset_label, as passed through the clean_text function from the Evaluation page.\n\nsample_submission.csv - a sample submission file in the correct format.\n* `Id` - publication id.\n* `PredictionString` - To be filled with equivalent of cleaned_label of train data.","metadata":{"papermill":{"duration":0.082355,"end_time":"2021-06-16T14:11:58.491606","exception":false,"start_time":"2021-06-16T14:11:58.409251","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Knowledge Bank : Literal Matching","metadata":{"papermill":{"duration":0.076401,"end_time":"2021-06-16T14:11:58.646442","exception":false,"start_time":"2021-06-16T14:11:58.570041","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Number of Unique values in each column","metadata":{"papermill":{"duration":0.070676,"end_time":"2021-06-16T14:11:58.788222","exception":false,"start_time":"2021-06-16T14:11:58.717546","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"📝Great! we don't have any null values.
","metadata":{"papermill":{"duration":0.07086,"end_time":"2021-06-16T14:11:58.930994","exception":false,"start_time":"2021-06-16T14:11:58.860134","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Observations\n\n- 1) There are duplicate id's meaning that there are some pulications that are using mutiple datasets. That's why that id is repeating.\n- 2) Same is the case with pub_title. A single publication is using mutiple datasets.\n- 3) There is NO one to one mapping of id and pub_title. Meaning that there are cases when two different publications (from two different authors) have same title. Well, interesting!!!\n- 4) There 45 dataset titles but 130 dataet labels. Meaning that there are some datasets that has multiple labels. We'll look into how these two are related.","metadata":{"papermill":{"duration":0.073109,"end_time":"2021-06-16T14:11:59.075357","exception":false,"start_time":"2021-06-16T14:11:59.002248","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Examining the section titles in each json file. Some json files have different titles, few are common in some","metadata":{"papermill":{"duration":0.070606,"end_time":"2021-06-16T14:11:59.217243","exception":false,"start_time":"2021-06-16T14:11:59.146637","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from nltk.tokenize import word_tokenize\nfrom nltk.tag import pos_tag","metadata":{"papermill":{"duration":0.077887,"end_time":"2021-06-16T14:11:59.36639","exception":false,"start_time":"2021-06-16T14:11:59.288503","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:33:04.698815Z","iopub.execute_input":"2021-06-22T22:33:04.699118Z","iopub.status.idle":"2021-06-22T22:33:04.703894Z","shell.execute_reply.started":"2021-06-22T22:33:04.699090Z","shell.execute_reply":"2021-06-22T22:33:04.702780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# No. of different dataset_label according to dataset_title","metadata":{"papermill":{"duration":0.070094,"end_time":"2021-06-16T14:11:59.50716","exception":false,"start_time":"2021-06-16T14:11:59.437066","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Duplicate ID,pub_title but different datasets","metadata":{"papermill":{"duration":0.071126,"end_time":"2021-06-16T14:11:59.649282","exception":false,"start_time":"2021-06-16T14:11:59.578156","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"📝 We have duplicate ids but we are not going to remove them from data because one research paper may consist of more than one datasets. Anyhow duplicate ids with respective datasets are shown below
","metadata":{"papermill":{"duration":0.070436,"end_time":"2021-06-16T14:11:59.791013","exception":false,"start_time":"2021-06-16T14:11:59.720577","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Single publication may have multiple datasets","metadata":{"papermill":{"duration":0.070143,"end_time":"2021-06-16T14:11:59.932472","exception":false,"start_time":"2021-06-16T14:11:59.862329","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#3. Data Vizualization
","metadata":{"papermill":{"duration":0.071959,"end_time":"2021-06-16T14:12:00.075565","exception":false,"start_time":"2021-06-16T14:12:00.003606","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Distribution of Id, pub_title in training data. How frequently each value occurs","metadata":{"papermill":{"duration":0.07159,"end_time":"2021-06-16T14:12:00.219477","exception":false,"start_time":"2021-06-16T14:12:00.147887","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Distribution of pub_title in data (first 20)","metadata":{"papermill":{"duration":0.07127,"end_time":"2021-06-16T14:12:00.362525","exception":false,"start_time":"2021-06-16T14:12:00.291255","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Grouping dataset of title and label","metadata":{"papermill":{"duration":0.071853,"end_time":"2021-06-16T14:12:01.080724","exception":false,"start_time":"2021-06-16T14:12:01.008871","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"📝Merge all text from json into train.csv with colmn name text
","metadata":{"papermill":{"duration":0.070949,"end_time":"2021-06-16T14:12:01.223756","exception":false,"start_time":"2021-06-16T14:12:01.152807","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def read_append_return(filename, train_files_path=train_files_path, output='text'):\n \"\"\"\n Function to read json file and then return the text data from them and append to the dataframe\n \"\"\"\n json_path = os.path.join(train_files_path, (filename+'.json'))\n headings = []\n contents = []\n combined = []\n with open(json_path, 'r') as f:\n json_decode = json.load(f)\n for data in json_decode:\n headings.append(data.get('section_title'))\n contents.append(data.get('text'))\n combined.append(data.get('section_title'))\n combined.append(data.get('text'))\n \n all_headings = ' '.join(headings)\n all_contents = ' '.join(contents)\n all_data = '. '.join(combined)\n \n if output == 'text':\n return all_contents\n elif output == 'head':\n return all_headings\n else:\n return all_data","metadata":{"papermill":{"duration":0.08252,"end_time":"2021-06-16T14:12:01.378232","exception":false,"start_time":"2021-06-16T14:12:01.295712","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:33:04.706071Z","iopub.execute_input":"2021-06-22T22:33:04.706515Z","iopub.status.idle":"2021-06-22T22:33:04.717100Z","shell.execute_reply.started":"2021-06-22T22:33:04.706470Z","shell.execute_reply":"2021-06-22T22:33:04.716040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝tqdm is used to show any code running with a progress bar
","metadata":{"papermill":{"duration":0.071068,"end_time":"2021-06-16T14:12:01.520744","exception":false,"start_time":"2021-06-16T14:12:01.449676","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\ntqdm.pandas() \ntrain_df['text'] = train_df['Id'].progress_apply(read_append_return)","metadata":{"papermill":{"duration":79.648128,"end_time":"2021-06-16T14:13:21.240154","exception":false,"start_time":"2021-06-16T14:12:01.592026","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:33:04.718084Z","iopub.execute_input":"2021-06-22T22:33:04.718356Z","iopub.status.idle":"2021-06-22T22:34:15.388707Z","shell.execute_reply.started":"2021-06-22T22:33:04.718328Z","shell.execute_reply":"2021-06-22T22:34:15.387797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝We have our text appended in our train dataframe
","metadata":{"papermill":{"duration":0.071213,"end_time":"2021-06-16T14:13:21.383761","exception":false,"start_time":"2021-06-16T14:13:21.312548","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df.head()","metadata":{"papermill":{"duration":0.094384,"end_time":"2021-06-16T14:13:21.549923","exception":false,"start_time":"2021-06-16T14:13:21.455539","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:34:15.389879Z","iopub.execute_input":"2021-06-22T22:34:15.390156Z","iopub.status.idle":"2021-06-22T22:34:15.412061Z","shell.execute_reply.started":"2021-06-22T22:34:15.390120Z","shell.execute_reply":"2021-06-22T22:34:15.410919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas()\nsample_sub['text'] = sample_sub['Id'].progress_apply(partial(read_append_return, train_files_path=test_files_path))","metadata":{"papermill":{"duration":0.141652,"end_time":"2021-06-16T14:13:21.764429","exception":false,"start_time":"2021-06-16T14:13:21.622777","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:34:15.414303Z","iopub.execute_input":"2021-06-22T22:34:15.414865Z","iopub.status.idle":"2021-06-22T22:34:15.474510Z","shell.execute_reply.started":"2021-06-22T22:34:15.414828Z","shell.execute_reply":"2021-06-22T22:34:15.473490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝Also, we have the text of for the sample_submission file
","metadata":{"papermill":{"duration":0.07353,"end_time":"2021-06-16T14:13:21.910863","exception":false,"start_time":"2021-06-16T14:13:21.837333","status":"completed"},"tags":[]}},{"cell_type":"code","source":"sample_sub.head()","metadata":{"papermill":{"duration":0.090419,"end_time":"2021-06-16T14:13:22.075713","exception":false,"start_time":"2021-06-16T14:13:21.985294","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:34:15.477943Z","iopub.execute_input":"2021-06-22T22:34:15.478374Z","iopub.status.idle":"2021-06-22T22:34:15.494907Z","shell.execute_reply.started":"2021-06-22T22:34:15.478330Z","shell.execute_reply":"2021-06-22T22:34:15.493687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub['text']","metadata":{"papermill":{"duration":0.084817,"end_time":"2021-06-16T14:13:22.234311","exception":false,"start_time":"2021-06-16T14:13:22.149494","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:34:15.497216Z","iopub.execute_input":"2021-06-22T22:34:15.497640Z","iopub.status.idle":"2021-06-22T22:34:15.510008Z","shell.execute_reply.started":"2021-06-22T22:34:15.497582Z","shell.execute_reply":"2021-06-22T22:34:15.509273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#4. Data Cleaning
","metadata":{"papermill":{"duration":0.073503,"end_time":"2021-06-16T14:13:22.38231","exception":false,"start_time":"2021-06-16T14:13:22.308807","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def text_cleaning(text):\n '''\n Converts all text to lower case, Removes special charecters, emojis and multiple spaces\n text - Sentence that needs to be cleaned\n '''\n text = ''.join([k for k in text if k not in string.punctuation])\n text = re.sub('r[^\\w\\s]', ' ', str(text).lower()).strip()\n lem = nltk.stem.wordnet.WordNetLemmatizer()\n text = lem.lemmatize(text)\n# text = re.sub('[^A-Za-z0-9]+', ' ', str(text).lower()).strip()\n# text = re.sub(\"/'+/g\", ' ', text)\n \n return text","metadata":{"papermill":{"duration":0.083576,"end_time":"2021-06-16T14:13:22.539585","exception":false,"start_time":"2021-06-16T14:13:22.456009","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:34:15.510970Z","iopub.execute_input":"2021-06-22T22:34:15.511425Z","iopub.status.idle":"2021-06-22T22:34:15.519962Z","shell.execute_reply.started":"2021-06-22T22:34:15.511393Z","shell.execute_reply":"2021-06-22T22:34:15.518987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas()\ntrain_df['text'] = train_df['text'].progress_apply(text_cleaning)","metadata":{"papermill":{"duration":144.81972,"end_time":"2021-06-16T14:15:47.433465","exception":false,"start_time":"2021-06-16T14:13:22.613745","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:34:15.521277Z","iopub.execute_input":"2021-06-22T22:34:15.521581Z","iopub.status.idle":"2021-06-22T22:36:40.589975Z","shell.execute_reply.started":"2021-06-22T22:34:15.521554Z","shell.execute_reply":"2021-06-22T22:36:40.588999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝We have our data cleaned!
","metadata":{"papermill":{"duration":0.074975,"end_time":"2021-06-16T14:15:47.583404","exception":false,"start_time":"2021-06-16T14:15:47.508429","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df.head()","metadata":{"papermill":{"duration":0.102526,"end_time":"2021-06-16T14:15:47.761406","exception":false,"start_time":"2021-06-16T14:15:47.65888","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:40.591876Z","iopub.execute_input":"2021-06-22T22:36:40.592291Z","iopub.status.idle":"2021-06-22T22:36:40.608459Z","shell.execute_reply.started":"2021-06-22T22:36:40.592246Z","shell.execute_reply":"2021-06-22T22:36:40.607511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"text = train_df['text']\nprint(text)","metadata":{"papermill":{"duration":0.091689,"end_time":"2021-06-16T14:15:47.944692","exception":false,"start_time":"2021-06-16T14:15:47.853003","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:40.609691Z","iopub.execute_input":"2021-06-22T22:36:40.609966Z","iopub.status.idle":"2021-06-22T22:36:40.617745Z","shell.execute_reply.started":"2021-06-22T22:36:40.609939Z","shell.execute_reply":"2021-06-22T22:36:40.617039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝Remove stop words from cleaned_label of train.csv
","metadata":{"papermill":{"duration":0.076637,"end_time":"2021-06-16T14:15:48.100256","exception":false,"start_time":"2021-06-16T14:15:48.023619","status":"completed"},"tags":[]}},{"cell_type":"code","source":"words =list(train_df['cleaned_label'].values)\nstopwords=['ourselves', 'hers','the','of','and','in', 'between', 'yourself', 'but', 'again','of', 'there', 'about', 'once', 'during', 'out', 'very', 'having', 'with', 'they', 'own', 'an', 'be', 'some', 'for', 'do', 'its', 'yours', 'such', 'into', 'of', 'most', 'itself', 'other', 'off', 'is', 's', 'am', 'or', 'who', 'as', 'from', 'him', 'each', 'the', 'themselves', 'until', 'below', 'are', 'we', 'these', 'your', 'his', 'through', 'don', 'nor', 'me', 'were', 'her', 'more', 'himself', 'this', 'down', 'should', 'our', 'their', 'while', 'above', 'both', 'up', 'to', 'ours', 'had', 'she', 'all', 'no', 'when', 'at', 'any', 'before', 'them', 'same', 'and', 'been', 'have', 'in', 'will', 'on', 'does', 'yourselves', 'then', 'that', 'because', 'what', 'over', 'why', 'so', 'can', 'did', 'not', 'now', 'under', 'he', 'you', 'herself', 'has', 'just', 'where', 'too', 'only', 'myself', 'which', 'those', 'i', 'after', 'few', 'whom', 't', 'being', 'if', 'theirs', 'my', 'against', 'a', 'by', 'doing', 'it', 'how', 'further', 'was', 'here', 'than']\nsplit_words=[]\nfor word in words:\n lo_w=[]\n list_of_words=str(word).split()\n for w in list_of_words:\n if w not in stopwords:\n lo_w.append(w)\n split_words.append(lo_w)\nallwords = []\nfor wordlist in split_words:\n allwords += wordlist","metadata":{"papermill":{"duration":0.2277,"end_time":"2021-06-16T14:15:48.404217","exception":false,"start_time":"2021-06-16T14:15:48.176517","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:40.618872Z","iopub.execute_input":"2021-06-22T22:36:40.619354Z","iopub.status.idle":"2021-06-22T22:36:40.774100Z","shell.execute_reply.started":"2021-06-22T22:36:40.619315Z","shell.execute_reply":"2021-06-22T22:36:40.773380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝100 Most common words (cleaned_label) - WordCloud
","metadata":{"papermill":{"duration":0.076035,"end_time":"2021-06-16T14:15:48.5566","exception":false,"start_time":"2021-06-16T14:15:48.480565","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"📝 25 Most Common Words in cleaned_label
","metadata":{"papermill":{"duration":0.075933,"end_time":"2021-06-16T14:15:48.709656","exception":false,"start_time":"2021-06-16T14:15:48.633723","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"📝Remove all special characters, punctuation and spaces from string\n
","metadata":{"papermill":{"duration":0.077909,"end_time":"2021-06-16T14:15:48.86518","exception":false,"start_time":"2021-06-16T14:15:48.787271","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"*Command to do a regex substitution*","metadata":{"papermill":{"duration":0.076885,"end_time":"2021-06-16T14:15:49.01838","exception":false,"start_time":"2021-06-16T14:15:48.941495","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def clean_text(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","metadata":{"papermill":{"duration":0.084924,"end_time":"2021-06-16T14:15:49.179635","exception":false,"start_time":"2021-06-16T14:15:49.094711","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:40.775256Z","iopub.execute_input":"2021-06-22T22:36:40.775714Z","iopub.status.idle":"2021-06-22T22:36:40.779181Z","shell.execute_reply.started":"2021-06-22T22:36:40.775684Z","shell.execute_reply":"2021-06-22T22:36:40.778515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#5. Baseline model and Submission
","metadata":{"papermill":{"duration":0.075559,"end_time":"2021-06-16T14:15:49.330857","exception":false,"start_time":"2021-06-16T14:15:49.255298","status":"completed"},"tags":[]}},{"cell_type":"code","source":"temp_1 = [x.lower() for x in train_df['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train_df['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train_df['cleaned_label'].unique()]\n\nexisting_labels = set(temp_1 + temp_2 + temp_3)\nid_list = []\nlables_list = []\nfor index, row in tqdm(sample_sub.iterrows()):\n sample_text = row['text']\n row_id = row['Id']\n temp_df = train_df[train_df['text'] == text_cleaning(sample_text)]\n cleaned_labels = temp_df['cleaned_label'].to_list()\n for known_label in existing_labels:\n if known_label in sample_text.lower():\n cleaned_labels.append(clean_text(known_label))\n cleaned_labels = [clean_text(x) for x in cleaned_labels]\n cleaned_labels = set(cleaned_labels)\n lables_list.append('|'.join(cleaned_labels))\n id_list.append(row_id)","metadata":{"papermill":{"duration":0.437865,"end_time":"2021-06-16T14:15:49.84447","exception":false,"start_time":"2021-06-16T14:15:49.406605","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:40.780332Z","iopub.execute_input":"2021-06-22T22:36:40.780806Z","iopub.status.idle":"2021-06-22T22:36:41.155023Z","shell.execute_reply.started":"2021-06-22T22:36:40.780776Z","shell.execute_reply":"2021-06-22T22:36:41.154059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission1 = pd.DataFrame()\nsubmission1['Id'] = id_list\nsubmission1['PredictionString'] = lables_list","metadata":{"papermill":{"duration":0.087057,"end_time":"2021-06-16T14:15:50.008621","exception":false,"start_time":"2021-06-16T14:15:49.921564","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:41.156556Z","iopub.execute_input":"2021-06-22T22:36:41.157094Z","iopub.status.idle":"2021-06-22T22:36:41.164771Z","shell.execute_reply.started":"2021-06-22T22:36:41.157052Z","shell.execute_reply":"2021-06-22T22:36:41.163908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission1.head()","metadata":{"papermill":{"duration":0.089898,"end_time":"2021-06-16T14:15:50.177522","exception":false,"start_time":"2021-06-16T14:15:50.087624","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:41.165834Z","iopub.execute_input":"2021-06-22T22:36:41.166206Z","iopub.status.idle":"2021-06-22T22:36:41.178251Z","shell.execute_reply.started":"2021-06-22T22:36:41.166176Z","shell.execute_reply":"2021-06-22T22:36:41.177453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# # # from this","metadata":{"papermill":{"duration":0.075851,"end_time":"2021-06-16T14:15:50.330056","exception":false,"start_time":"2021-06-16T14:15:50.254205","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from spacy import displacy\nfrom collections import Counter\nimport en_core_web_lg\nnlp = en_core_web_lg.load()","metadata":{"papermill":{"duration":6.785295,"end_time":"2021-06-16T14:15:57.191431","exception":false,"start_time":"2021-06-16T14:15:50.406136","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:41.179657Z","iopub.execute_input":"2021-06-22T22:36:41.180181Z","iopub.status.idle":"2021-06-22T22:36:49.112051Z","shell.execute_reply.started":"2021-06-22T22:36:41.180135Z","shell.execute_reply":"2021-06-22T22:36:49.110833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2 = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')","metadata":{"papermill":{"duration":0.091336,"end_time":"2021-06-16T14:15:57.360887","exception":false,"start_time":"2021-06-16T14:15:57.269551","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.113712Z","iopub.execute_input":"2021-06-22T22:36:49.114135Z","iopub.status.idle":"2021-06-22T22:36:49.124258Z","shell.execute_reply.started":"2021-06-22T22:36:49.114093Z","shell.execute_reply":"2021-06-22T22:36:49.123330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas()\nsample_sub2['text'] = sample_sub2['Id'].progress_apply(partial(read_append_return, train_files_path=test_files_path))","metadata":{"papermill":{"duration":0.13782,"end_time":"2021-06-16T14:15:57.575626","exception":false,"start_time":"2021-06-16T14:15:57.437806","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.125591Z","iopub.execute_input":"2021-06-22T22:36:49.125889Z","iopub.status.idle":"2021-06-22T22:36:49.181108Z","shell.execute_reply.started":"2021-06-22T22:36:49.125862Z","shell.execute_reply":"2021-06-22T22:36:49.180216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2['text'] = sample_sub2['text'].str.replace('[^\\w\\s]','')\nsample_sub2.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:49.182829Z","iopub.execute_input":"2021-06-22T22:36:49.183246Z","iopub.status.idle":"2021-06-22T22:36:49.213468Z","shell.execute_reply.started":"2021-06-22T22:36:49.183202Z","shell.execute_reply":"2021-06-22T22:36:49.212492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#del train_df\n# del sample_sub","metadata":{"papermill":{"duration":0.140234,"end_time":"2021-06-16T14:15:57.970392","exception":false,"start_time":"2021-06-16T14:15:57.830158","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.218560Z","iopub.execute_input":"2021-06-22T22:36:49.218932Z","iopub.status.idle":"2021-06-22T22:36:49.223122Z","shell.execute_reply.started":"2021-06-22T22:36:49.218900Z","shell.execute_reply":"2021-06-22T22:36:49.222198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2.head()","metadata":{"papermill":{"duration":0.094106,"end_time":"2021-06-16T14:15:58.146294","exception":false,"start_time":"2021-06-16T14:15:58.052188","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.225221Z","iopub.execute_input":"2021-06-22T22:36:49.225530Z","iopub.status.idle":"2021-06-22T22:36:49.241127Z","shell.execute_reply.started":"2021-06-22T22:36:49.225487Z","shell.execute_reply":"2021-06-22T22:36:49.240413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\ngc.collect()","metadata":{"papermill":{"duration":0.270269,"end_time":"2021-06-16T14:15:58.495758","exception":false,"start_time":"2021-06-16T14:15:58.225489","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.242315Z","iopub.execute_input":"2021-06-22T22:36:49.242652Z","iopub.status.idle":"2021-06-22T22:36:49.466335Z","shell.execute_reply.started":"2021-06-22T22:36:49.242613Z","shell.execute_reply":"2021-06-22T22:36:49.465372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datalist = pd.read_csv('../input/para-data/datalist.csv')","metadata":{"papermill":{"duration":0.101299,"end_time":"2021-06-16T14:15:58.675928","exception":false,"start_time":"2021-06-16T14:15:58.574629","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.467793Z","iopub.execute_input":"2021-06-22T22:36:49.468128Z","iopub.status.idle":"2021-06-22T22:36:49.485149Z","shell.execute_reply.started":"2021-06-22T22:36:49.468092Z","shell.execute_reply":"2021-06-22T22:36:49.483919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datalist.head()","metadata":{"papermill":{"duration":0.11767,"end_time":"2021-06-16T14:15:58.878224","exception":false,"start_time":"2021-06-16T14:15:58.760554","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.486502Z","iopub.execute_input":"2021-06-22T22:36:49.486795Z","iopub.status.idle":"2021-06-22T22:36:49.495159Z","shell.execute_reply.started":"2021-06-22T22:36:49.486766Z","shell.execute_reply":"2021-06-22T22:36:49.494246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datasetLabels= datalist['Data']\nprint(datasetLabels)","metadata":{"papermill":{"duration":0.088763,"end_time":"2021-06-16T14:15:59.051597","exception":false,"start_time":"2021-06-16T14:15:58.962834","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.496858Z","iopub.execute_input":"2021-06-22T22:36:49.497535Z","iopub.status.idle":"2021-06-22T22:36:49.511206Z","shell.execute_reply.started":"2021-06-22T22:36:49.497489Z","shell.execute_reply":"2021-06-22T22:36:49.510433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"a = [(x,i) for x, y in zip(sample_sub2['Id'],sample_sub2['text']) for i in datasetLabels if i in y]\nprint (a)","metadata":{"papermill":{"duration":0.093314,"end_time":"2021-06-16T14:16:03.269571","exception":false,"start_time":"2021-06-16T14:16:03.176257","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.512326Z","iopub.execute_input":"2021-06-22T22:36:49.512623Z","iopub.status.idle":"2021-06-22T22:36:49.527168Z","shell.execute_reply.started":"2021-06-22T22:36:49.512595Z","shell.execute_reply":"2021-06-22T22:36:49.526153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2 = pd.DataFrame(a, columns=['Id', 'PredictionString'])\nsample_sub2 = sample_sub2.groupby('Id').agg({'PredictionString': '|'.join}).reset_index()\nsample_sub2.head()","metadata":{"papermill":{"duration":0.111874,"end_time":"2021-06-16T14:16:03.467728","exception":false,"start_time":"2021-06-16T14:16:03.355854","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.528470Z","iopub.execute_input":"2021-06-22T22:36:49.528760Z","iopub.status.idle":"2021-06-22T22:36:49.555096Z","shell.execute_reply.started":"2021-06-22T22:36:49.528732Z","shell.execute_reply":"2021-06-22T22:36:49.554227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2['PredictionString'][3]","metadata":{"papermill":{"duration":0.09499,"end_time":"2021-06-16T14:16:03.649095","exception":false,"start_time":"2021-06-16T14:16:03.554105","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.556266Z","iopub.execute_input":"2021-06-22T22:36:49.556602Z","iopub.status.idle":"2021-06-22T22:36:49.562106Z","shell.execute_reply.started":"2021-06-22T22:36:49.556571Z","shell.execute_reply":"2021-06-22T22:36:49.561056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2['PredictionString'] = [x.lower() for x in sample_sub2['PredictionString'].unique()]","metadata":{"papermill":{"duration":0.093088,"end_time":"2021-06-16T14:16:03.82765","exception":false,"start_time":"2021-06-16T14:16:03.734562","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.563385Z","iopub.execute_input":"2021-06-22T22:36:49.563690Z","iopub.status.idle":"2021-06-22T22:36:49.573038Z","shell.execute_reply.started":"2021-06-22T22:36:49.563660Z","shell.execute_reply":"2021-06-22T22:36:49.572235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#my_list = [\"cohort study [lbc1936; n\",\"lothian birth cohort study [lbc1936\",\"Information Resource Incorporated (IRI\",\"on@3\", \"two#\", \"thre%e\"]\nremovetable = str.maketrans('', '', ',@#%([')\nsample_sub2['PredictionString'] = [s.translate(removetable) for s in sample_sub2['PredictionString']]\nprint(sample_sub2['PredictionString'])","metadata":{"papermill":{"duration":0.097129,"end_time":"2021-06-16T14:16:04.009195","exception":false,"start_time":"2021-06-16T14:16:03.912066","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.574572Z","iopub.execute_input":"2021-06-22T22:36:49.574959Z","iopub.status.idle":"2021-06-22T22:36:49.590945Z","shell.execute_reply.started":"2021-06-22T22:36:49.574918Z","shell.execute_reply":"2021-06-22T22:36:49.589903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# stop_words = [\"the \"\n# ]\n\n# pat = '|'.join(r\"\\b{}\\b\".format(x) for x in stop_words)\n# sample_sub2['PredictionString'] = sample_sub2['PredictionString'].str.replace(pat, '')\n\n\n# #top_words = set(stop_words)\n# #f = lambda x: ' '.join(w for w in x.split() if not w in stop_words)\n# #newdata['Verbatim2'] = newdata['Verbatim'].apply(f)\n\n# print (sample_sub2['PredictionString'])","metadata":{"papermill":{"duration":0.09454,"end_time":"2021-06-16T14:16:04.189484","exception":false,"start_time":"2021-06-16T14:16:04.094944","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.592458Z","iopub.execute_input":"2021-06-22T22:36:49.592863Z","iopub.status.idle":"2021-06-22T22:36:49.600393Z","shell.execute_reply.started":"2021-06-22T22:36:49.592820Z","shell.execute_reply":"2021-06-22T22:36:49.599508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission2 = pd.DataFrame()\nsubmission2['Id'] = sample_sub2['Id']\nsubmission2['PredictionString'] = sample_sub2['PredictionString']\nsubmission2.head()","metadata":{"papermill":{"duration":0.098245,"end_time":"2021-06-16T14:16:04.371733","exception":false,"start_time":"2021-06-16T14:16:04.273488","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:49.601716Z","iopub.execute_input":"2021-06-22T22:36:49.602114Z","iopub.status.idle":"2021-06-22T22:36:49.619867Z","shell.execute_reply.started":"2021-06-22T22:36:49.602073Z","shell.execute_reply":"2021-06-22T22:36:49.618800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# govt dataset","metadata":{}},{"cell_type":"code","source":"def read_json_pub(filename, train_files_path=train_files_path, output='text'):\n json_path = os.path.join(train_files_path, (filename+'.json'))\n headings = []\n contents = []\n combined = []\n with open(json_path, 'r') as f:\n json_decode = json.load(f)\n for data in json_decode:\n headings.append(data.get('section_title'))\n contents.append(data.get('text'))\n combined.append(data.get('section_title'))\n combined.append(data.get('text'))\n \n all_headings = ' '.join(headings)\n all_contents = ' '.join(contents)\n all_data = '. '.join(combined)\n \n if output == 'text':\n return all_contents\n elif output == 'head':\n return all_headings\n else:\n return all_data","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:49.621387Z","iopub.execute_input":"2021-06-22T22:36:49.621799Z","iopub.status.idle":"2021-06-22T22:36:49.630001Z","shell.execute_reply.started":"2021-06-22T22:36:49.621756Z","shell.execute_reply":"2021-06-22T22:36:49.629047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def text_cleaning2(text):\n '''\n Converts all text to lower case, Removes special charecters, emojis and multiple spaces\n text - Sentence that needs to be cleaned\n '''\n text = re.sub('[^A-Za-z0-9]+', ' ', str(text).lower()).strip()\n text = re.sub(' +', ' ', text)\n emoji_pattern = re.compile(\"[\"\n u\"\\U0001F600-\\U0001F64F\" # emoticons\n u\"\\U0001F300-\\U0001F5FF\" # symbols & pictographs\n u\"\\U0001F680-\\U0001F6FF\" # transport & map symbols\n u\"\\U0001F1E0-\\U0001F1FF\" # flags (iOS)\n \"]+\", flags=re.UNICODE)\n text = emoji_pattern.sub(r'', text)\n return text","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:49.631053Z","iopub.execute_input":"2021-06-22T22:36:49.631411Z","iopub.status.idle":"2021-06-22T22:36:49.640857Z","shell.execute_reply.started":"2021-06-22T22:36:49.631380Z","shell.execute_reply":"2021-06-22T22:36:49.640124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def clean_text2(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:49.641819Z","iopub.execute_input":"2021-06-22T22:36:49.642196Z","iopub.status.idle":"2021-06-22T22:36:49.659534Z","shell.execute_reply.started":"2021-06-22T22:36:49.642167Z","shell.execute_reply":"2021-06-22T22:36:49.658519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# random sample for testing\ntrain_sample=train_df.sample(n = 10)\n\ntrain_sample","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:49.660799Z","iopub.execute_input":"2021-06-22T22:36:49.661104Z","iopub.status.idle":"2021-06-22T22:36:49.683916Z","shell.execute_reply.started":"2021-06-22T22:36:49.661077Z","shell.execute_reply":"2021-06-22T22:36:49.683197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"govt_df = pd.read_csv('../input/govtdatasets/data_set_800.csv')","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:49.685060Z","iopub.execute_input":"2021-06-22T22:36:49.685325Z","iopub.status.idle":"2021-06-22T22:36:49.703156Z","shell.execute_reply.started":"2021-06-22T22:36:49.685298Z","shell.execute_reply":"2021-06-22T22:36:49.702295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"govt_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:49.704275Z","iopub.execute_input":"2021-06-22T22:36:49.704566Z","iopub.status.idle":"2021-06-22T22:36:49.712631Z","shell.execute_reply.started":"2021-06-22T22:36:49.704537Z","shell.execute_reply":"2021-06-22T22:36:49.711862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_time = time.time()\n\n\n#### remove >.5 jaccard matches from predicitons\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n\n#############################\n#path=train_data_path\npath=test_files_path\n\n#for training use train_sample\n\n#for submission use sample_sub\n\n#############\n\ncolumn_names = [\"Id\", \"PredictionString\"]\n\nsubmission3 = pd.DataFrame(columns = column_names)\n\nto_append=[]\nfor index, row in sample_sub.iterrows():\n to_append=[row['Id'],'']\n large_string = str(read_append_return(row['Id'],path))\n clean_string=text_cleaning2(large_string)\n for index, row2 in govt_df.iterrows():\n query_string = str(row2['title'])\n if query_string in clean_string:\n if to_append[1]!='' and clean_text2(query_string) not in to_append[1]:\n to_append[1]=to_append[1]+'|'+clean_text(query_string)\n if to_append[1]=='':\n to_append[1]=clean_text2(query_string)\n \n ###### remove similar jaccard\n #got_label=to_append[1].split('|')\n #filtered=[]\n #filtered_labels = ''\n #for label in sorted(got_label, key=len):\n #label = clean_text(label)\n #if len(filtered) == 0 or all(jaccard_similarity(label, got_label) < 1.0 for got_label in filtered):\n #filtered.append(label)\n #if filtered_labels!='':\n #filtered_labels=filtered_labels+'|'+label\n #if filtered_labels=='':\n #filtered_labels=label\n #to_append[1] = filtered_labels \n #print ('################')\n #print (to_append)\n #print (large_string)\n #print ('################')\n ###### remove similar jaccard\n govt_df_length = len(submission3)\n submission3.loc[govt_df_length] = to_append\n# submission3.to_csv('submission3.csv', index = False)\nprint(\"--- %s seconds ---\" % (time.time() - start_time))\nsubmission3","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:49.714362Z","iopub.execute_input":"2021-06-22T22:36:49.714792Z","iopub.status.idle":"2021-06-22T22:36:51.001363Z","shell.execute_reply.started":"2021-06-22T22:36:49.714748Z","shell.execute_reply":"2021-06-22T22:36:51.000507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission3['PredictionString'][0]","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:51.002389Z","iopub.execute_input":"2021-06-22T22:36:51.002676Z","iopub.status.idle":"2021-06-22T22:36:51.008520Z","shell.execute_reply.started":"2021-06-22T22:36:51.002648Z","shell.execute_reply":"2021-06-22T22:36:51.007544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission3 = pd.DataFrame()\n# submission3['Id'] = govt_data['Id']\n# submission3['PredictionString'] = govt_data['PredictionString']\n# submission3.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:51.010053Z","iopub.execute_input":"2021-06-22T22:36:51.010395Z","iopub.status.idle":"2021-06-22T22:36:51.020264Z","shell.execute_reply.started":"2021-06-22T22:36:51.010328Z","shell.execute_reply":"2021-06-22T22:36:51.019538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝Merging the two DataTables into one Submission File
","metadata":{"papermill":{"duration":0.084157,"end_time":"2021-06-16T14:16:04.54057","exception":false,"start_time":"2021-06-16T14:16:04.456413","status":"completed"},"tags":[]}},{"cell_type":"code","source":"##submission = pd.DataFrame(data= {'Id' : sample_sub['Id'], 'PredictionString' : ['','','','']})","metadata":{"papermill":{"duration":0.091457,"end_time":"2021-06-16T14:16:04.716641","exception":false,"start_time":"2021-06-16T14:16:04.625184","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.021848Z","iopub.execute_input":"2021-06-22T22:36:51.022112Z","iopub.status.idle":"2021-06-22T22:36:51.029903Z","shell.execute_reply.started":"2021-06-22T22:36:51.022086Z","shell.execute_reply":"2021-06-22T22:36:51.028948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##submission.head()","metadata":{"papermill":{"duration":0.091267,"end_time":"2021-06-16T14:16:04.8934","exception":false,"start_time":"2021-06-16T14:16:04.802133","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.031294Z","iopub.execute_input":"2021-06-22T22:36:51.031691Z","iopub.status.idle":"2021-06-22T22:36:51.038582Z","shell.execute_reply.started":"2021-06-22T22:36:51.031661Z","shell.execute_reply":"2021-06-22T22:36:51.037714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"merged = pd.concat([submission1, submission2,submission3], sort=True).drop_duplicates()\nsubmission = merged.dropna().groupby(['Id'], as_index=False).agg({'PredictionString' : '|'.join}).reset_index(drop=True)\nprint(submission)","metadata":{"papermill":{"duration":0.115711,"end_time":"2021-06-16T14:16:05.093586","exception":false,"start_time":"2021-06-16T14:16:04.977875","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.040160Z","iopub.execute_input":"2021-06-22T22:36:51.040703Z","iopub.status.idle":"2021-06-22T22:36:51.076871Z","shell.execute_reply.started":"2021-06-22T22:36:51.040661Z","shell.execute_reply":"2021-06-22T22:36:51.075852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['PredictionString'][0]","metadata":{"execution":{"iopub.status.busy":"2021-06-22T22:36:51.078193Z","iopub.execute_input":"2021-06-22T22:36:51.078785Z","iopub.status.idle":"2021-06-22T22:36:51.085033Z","shell.execute_reply.started":"2021-06-22T22:36:51.078740Z","shell.execute_reply":"2021-06-22T22:36:51.084100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(pd.concat(submission.melt(id_vars='Id').dropna() for df in [submission1,submission2])\n .groupby(['Id','variable'])['value'].apply(lambda x: '|'.join(x.unique()))\n .unstack()\n)","metadata":{"papermill":{"duration":0.113506,"end_time":"2021-06-16T14:16:05.294166","exception":false,"start_time":"2021-06-16T14:16:05.18066","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.086354Z","iopub.execute_input":"2021-06-22T22:36:51.086942Z","iopub.status.idle":"2021-06-22T22:36:51.117660Z","shell.execute_reply.started":"2021-06-22T22:36:51.086897Z","shell.execute_reply":"2021-06-22T22:36:51.116748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# remove Duplicates from dataframe","metadata":{"papermill":{"duration":0.085319,"end_time":"2021-06-16T14:16:05.464978","exception":false,"start_time":"2021-06-16T14:16:05.379659","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def remove_dup(strng):\n '''\n Input a string and split them \n '''\n return '|'.join(list(dict.fromkeys(strng.split('|'))))","metadata":{"papermill":{"duration":0.092575,"end_time":"2021-06-16T14:16:05.643636","exception":false,"start_time":"2021-06-16T14:16:05.551061","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.118972Z","iopub.execute_input":"2021-06-22T22:36:51.119288Z","iopub.status.idle":"2021-06-22T22:36:51.123140Z","shell.execute_reply.started":"2021-06-22T22:36:51.119258Z","shell.execute_reply":"2021-06-22T22:36:51.122512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# my_dict = {'Tags':[\"Museum, Art Museum, Shopping, Museum\",'Drink, Drink','Shop','Visit'],'Country':['USA','USA','USA', 'USA']}\n# df = pd.DataFrame(my_dict)\nsubmission['PredictionString'] = submission['PredictionString'].apply(lambda x: remove_dup(x))\nsubmission","metadata":{"papermill":{"duration":0.098577,"end_time":"2021-06-16T14:16:05.827942","exception":false,"start_time":"2021-06-16T14:16:05.729365","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.124273Z","iopub.execute_input":"2021-06-22T22:36:51.124750Z","iopub.status.idle":"2021-06-22T22:36:51.141253Z","shell.execute_reply.started":"2021-06-22T22:36:51.124716Z","shell.execute_reply":"2021-06-22T22:36:51.140299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['PredictionString'][3]","metadata":{"papermill":{"duration":0.094565,"end_time":"2021-06-16T14:16:06.00795","exception":false,"start_time":"2021-06-16T14:16:05.913385","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.142696Z","iopub.execute_input":"2021-06-22T22:36:51.143012Z","iopub.status.idle":"2021-06-22T22:36:51.153954Z","shell.execute_reply.started":"2021-06-22T22:36:51.142983Z","shell.execute_reply":"2021-06-22T22:36:51.152962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Sorting in a alphabetical Order**","metadata":{"papermill":{"duration":0.087186,"end_time":"2021-06-16T14:16:06.182376","exception":false,"start_time":"2021-06-16T14:16:06.09519","status":"completed"},"tags":[]}},{"cell_type":"code","source":"submission['PredictionString'] = submission['PredictionString'].apply(lambda x: '|'.join(sorted(x.split('|'))))\nprint(submission)","metadata":{"papermill":{"duration":0.09746,"end_time":"2021-06-16T14:16:06.366726","exception":false,"start_time":"2021-06-16T14:16:06.269266","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.155648Z","iopub.execute_input":"2021-06-22T22:36:51.156086Z","iopub.status.idle":"2021-06-22T22:36:51.166993Z","shell.execute_reply.started":"2021-06-22T22:36:51.156042Z","shell.execute_reply":"2021-06-22T22:36:51.166034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['PredictionString'][0]","metadata":{"papermill":{"duration":0.094812,"end_time":"2021-06-16T14:16:06.548554","exception":false,"start_time":"2021-06-16T14:16:06.453742","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.168208Z","iopub.execute_input":"2021-06-22T22:36:51.168519Z","iopub.status.idle":"2021-06-22T22:36:51.175026Z","shell.execute_reply.started":"2021-06-22T22:36:51.168486Z","shell.execute_reply":"2021-06-22T22:36:51.174236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{"papermill":{"duration":0.099874,"end_time":"2021-06-16T14:16:06.736394","exception":false,"start_time":"2021-06-16T14:16:06.63652","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.176129Z","iopub.execute_input":"2021-06-22T22:36:51.176602Z","iopub.status.idle":"2021-06-22T22:36:51.191143Z","shell.execute_reply.started":"2021-06-22T22:36:51.176571Z","shell.execute_reply":"2021-06-22T22:36:51.190166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{"papermill":{"duration":0.100202,"end_time":"2021-06-16T14:16:06.924659","exception":false,"start_time":"2021-06-16T14:16:06.824457","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.192377Z","iopub.execute_input":"2021-06-22T22:36:51.192660Z","iopub.status.idle":"2021-06-22T22:36:51.204656Z","shell.execute_reply.started":"2021-06-22T22:36:51.192633Z","shell.execute_reply":"2021-06-22T22:36:51.203567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#test = pd.concat([train_df,train_df],axis=0,ignore_index=True)","metadata":{"papermill":{"duration":0.095839,"end_time":"2021-06-16T14:16:07.109154","exception":false,"start_time":"2021-06-16T14:16:07.013315","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-06-22T22:36:51.205869Z","iopub.execute_input":"2021-06-22T22:36:51.206162Z","iopub.status.idle":"2021-06-22T22:36:51.213366Z","shell.execute_reply.started":"2021-06-22T22:36:51.206131Z","shell.execute_reply":"2021-06-22T22:36:51.212636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hope you like this kernel.\n Please don't forget to upvote and leave your valuable comment. \n Thank you!
","metadata":{"papermill":{"duration":0.08749,"end_time":"2021-06-16T14:16:07.284428","exception":false,"start_time":"2021-06-16T14:16:07.196938","status":"completed"},"tags":[]}}]}