A Novel Data Extraction Framework Using Natural Language Processing (DEFNLP) Techniques
\nDiscover how data is used for the public good
","metadata":{"papermill":{"duration":0.072756,"end_time":"2021-06-16T14:11:44.556097","exception":false,"start_time":"2021-06-16T14:11:44.483341","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#Please dont forget to upvote the notebook
","metadata":{"papermill":{"duration":0.071487,"end_time":"2021-06-16T14:11:44.699809","exception":false,"start_time":"2021-06-16T14:11:44.628322","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## In this notebook basically we have to predict text for some strings by using nlp techniques\n*** \n>The objective of the competition is to identify the mention of datasets within scientific publications.\n\nThis competition challenges data scientists to show how publicly funded data are used to serve science and society. Evidence through data is critical if government is to address the many threats facing society, including; pandemics, climate change, Alzheimer’s disease, child hunger, increasing food production, maintaining biodiversity, and addressing many other challenges. Yet much of the information about data necessary to inform evidence and science is locked inside publications.\n\nCan natural language processing find the hidden-in-plain-sight data citations? Can machine learning find the link between the words used in research articles and the data referenced in the article?\n\nNow is the time for data scientists to help restore trust in data and evidence. In the United States, federal agencies are now mandated to show how their data are being used. The new Foundations of Evidence-based Policymaking Act requires agencies to modernize their data management. New Presidential Executive Orders are pushing government agencies to make evidence-based decisions based on the best available data and science. And the government is working to respond in an open and transparent way.\n\nThis competition will build just such an open and transparent approach. The results will show how public data are being used in science and help the government make wiser, more transparent public investments. It will help move researchers and governments from using ad-hoc methods to automated ways of finding out what datasets are being used to solve problems, what measures are being generated, and which researchers are the experts. Previous competitions have shown that it is possible to develop algorithms to automate the search and discovery of references to data. Now, we want the Kaggle community to develop the best approaches to identify critical datasets used in scientific publications.\n\nIn this competition, you'll use natural language processing (NLP) to automate the discovery of how scientific data are referenced in publications. Utilizing the full text of scientific publications from numerous research areas gathered from CHORUS publisher members and other sources, you'll identify data sets that the publications' authors used in their work.\n\nIt achieves this goal by working with the agencies to create value for the taxpayer from the careful use of data by building new technologies to enable secure access to and sharing of confidential microdata and by training agency staff to acquire modern data skills.\n\nIf successful, you'll help support evidence in government data. Automated NLP approaches will enable government agencies and researchers to quickly find the information they need. The approach will be used to develop data usage scorecards to better enable agencies to show how their data are used and bring down a critical barrier to the access and use of public data.","metadata":{"papermill":{"duration":0.07102,"end_time":"2021-06-16T14:11:44.843574","exception":false,"start_time":"2021-06-16T14:11:44.772554","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#Table of Content
\n* [1. Importing Modules & Libraries](#1)\n* [2. Data Exploration](#2)\n* [3. Data Vizualization](#3)\n* [4. Data Cleaning](#4)\n* [5. Baseline Model & Submission](#5)","metadata":{"papermill":{"duration":0.070983,"end_time":"2021-06-16T14:11:44.985416","exception":false,"start_time":"2021-06-16T14:11:44.914433","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#1. Importing Modules & Libraries
","metadata":{"papermill":{"duration":0.072463,"end_time":"2021-06-16T14:11:45.130487","exception":false,"start_time":"2021-06-16T14:11:45.058024","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)\n\nimport os\nimport re\nimport json\nimport time\nimport glob\nfrom collections import defaultdict\nfrom textblob import TextBlob\nfrom functools import partial\n\nimport pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport seaborn as sn\nimport plotly.express as px\nimport plotly.graph_objects as go\n\nimport nltk\nimport urllib.request\nfrom PIL import Image\nimport spacy\nnlp = spacy.load('en_core_web_lg', disable=['parser', 'ner'])\nnlp.max_length = 4000000\nfrom nltk.probability import FreqDist\nfrom wordcloud import WordCloud, STOPWORDS\n\nfrom tqdm.autonotebook import tqdm\nimport string\n\n%matplotlib inline\n\nos.listdir('/kaggle/input/coleridgeinitiative-show-us-the-data/')","metadata":{"papermill":{"duration":12.411484,"end_time":"2021-06-16T14:11:57.613052","exception":false,"start_time":"2021-06-16T14:11:45.201568","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:49:01.747207Z","iopub.execute_input":"2021-10-10T20:49:01.747639Z","iopub.status.idle":"2021-10-10T20:49:15.158082Z","shell.execute_reply.started":"2021-10-10T20:49:01.747549Z","shell.execute_reply":"2021-10-10T20:49:15.156855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are provided with 4 main pieces of data:\n\n* `train.csv:` The CSV file containing all the metadata of the publications, such as their title and the dataset they utilize.\n* `train:` The directory containing the actual publications that are referenced in train.csvin JSON format.\n* `test:` The directory containing the actual publications that will be used for testing purposes (thus, with no ground truth CSV file available).\n* `sample_submission.csv:` The CSV file containing all the publications IDs in the test set, for which we'll have to populate the prediction column.","metadata":{"papermill":{"duration":0.07102,"end_time":"2021-06-16T14:11:57.756794","exception":false,"start_time":"2021-06-16T14:11:57.685774","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#2. Data Exploration
","metadata":{"papermill":{"duration":0.07142,"end_time":"2021-06-16T14:11:57.899933","exception":false,"start_time":"2021-06-16T14:11:57.828513","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Reading csv files and train & test file paths","metadata":{"papermill":{"duration":0.07168,"end_time":"2021-06-16T14:11:58.042771","exception":false,"start_time":"2021-06-16T14:11:57.971091","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\nsample_sub = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')\ntrain_files_path = '../input/coleridgeinitiative-show-us-the-data/train'\ntest_files_path = '../input/coleridgeinitiative-show-us-the-data/test'","metadata":{"papermill":{"duration":0.21146,"end_time":"2021-06-16T14:11:58.325389","exception":false,"start_time":"2021-06-16T14:11:58.113929","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:49:15.159504Z","iopub.execute_input":"2021-10-10T20:49:15.159795Z","iopub.status.idle":"2021-10-10T20:49:15.33278Z","shell.execute_reply.started":"2021-10-10T20:49:15.159767Z","shell.execute_reply":"2021-10-10T20:49:15.331723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##Data Description
\ntrain.csv - labels and metadata for the training set train/test directory - the full text of the training/test set's publications in JSON format, broken into sections with section titles\n\n* `id` - publication id - note that there are multiple rows for some training documents, indicating multiple mentioned datasets.\n* `pub_title` - title of the publication (a small number of publications have the same title).\n* `dataset_title` - the title of the dataset that is mentioned within the publication.\n* `dataset_label` - a portion of the text that indicates the dataset.\n* `cleaned_label` - the dataset_label, as passed through the clean_text function from the Evaluation page.\n\nsample_submission.csv - a sample submission file in the correct format.\n* `Id` - publication id.\n* `PredictionString` - To be filled with equivalent of cleaned_label of train data.","metadata":{"papermill":{"duration":0.082355,"end_time":"2021-06-16T14:11:58.491606","exception":false,"start_time":"2021-06-16T14:11:58.409251","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:49:15.335424Z","iopub.execute_input":"2021-10-10T20:49:15.3358Z","iopub.status.idle":"2021-10-10T20:49:15.361064Z","shell.execute_reply.started":"2021-10-10T20:49:15.335769Z","shell.execute_reply":"2021-10-10T20:49:15.359961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝Great! we don't have any null values.
","metadata":{"papermill":{"duration":0.07086,"end_time":"2021-06-16T14:11:58.930994","exception":false,"start_time":"2021-06-16T14:11:58.860134","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Observations\n\n- 1) There are duplicate id's meaning that there are some pulications that are using mutiple datasets. That's why that id is repeating.\n- 2) Same is the case with pub_title. A single publication is using mutiple datasets.\n- 3) There is NO one to one mapping of id and pub_title. Meaning that there are cases when two different publications (from two different authors) have same title. Well, interesting!!!\n- 4) There 45 dataset titles but 130 dataet labels. Meaning that there are some datasets that has multiple labels. We'll look into how these two are related.","metadata":{"papermill":{"duration":0.073109,"end_time":"2021-06-16T14:11:59.075357","exception":false,"start_time":"2021-06-16T14:11:59.002248","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"📝 We have duplicate ids but we are not going to remove them from data because one research paper may consist of more than one datasets. Anyhow duplicate ids with respective datasets are shown below
","metadata":{"papermill":{"duration":0.070436,"end_time":"2021-06-16T14:11:59.791013","exception":false,"start_time":"2021-06-16T14:11:59.720577","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#3. Data Vizualization
","metadata":{"papermill":{"duration":0.071959,"end_time":"2021-06-16T14:12:00.075565","exception":false,"start_time":"2021-06-16T14:12:00.003606","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"📝Merge all text from json into train.csv with colmn name text
","metadata":{"papermill":{"duration":0.070949,"end_time":"2021-06-16T14:12:01.223756","exception":false,"start_time":"2021-06-16T14:12:01.152807","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def read_append_return(filename, train_files_path=train_files_path, output='text'):\n \"\"\"\n Function to read json file and then return the text data from them and append to the dataframe\n \"\"\"\n json_path = os.path.join(train_files_path, (filename+'.json'))\n headings = []\n contents = []\n combined = []\n with open(json_path, 'r') as f:\n json_decode = json.load(f)\n for data in json_decode:\n headings.append(data.get('section_title'))\n contents.append(data.get('text'))\n combined.append(data.get('section_title'))\n combined.append(data.get('text'))\n \n all_headings = ' '.join(headings)\n all_contents = ' '.join(contents)\n all_data = '. '.join(combined)\n \n if output == 'text':\n return all_contents\n elif output == 'head':\n return all_headings\n else:\n return all_data","metadata":{"papermill":{"duration":0.08252,"end_time":"2021-06-16T14:12:01.378232","exception":false,"start_time":"2021-06-16T14:12:01.295712","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:49:15.362837Z","iopub.execute_input":"2021-10-10T20:49:15.3632Z","iopub.status.idle":"2021-10-10T20:49:15.374181Z","shell.execute_reply.started":"2021-10-10T20:49:15.363166Z","shell.execute_reply":"2021-10-10T20:49:15.37297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝tqdm is used to show any code running with a progress bar
","metadata":{"papermill":{"duration":0.071068,"end_time":"2021-06-16T14:12:01.520744","exception":false,"start_time":"2021-06-16T14:12:01.449676","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\ntqdm.pandas() \ntrain_df['text'] = train_df['Id'].progress_apply(read_append_return)","metadata":{"papermill":{"duration":79.648128,"end_time":"2021-06-16T14:13:21.240154","exception":false,"start_time":"2021-06-16T14:12:01.592026","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:49:15.375842Z","iopub.execute_input":"2021-10-10T20:49:15.376271Z","iopub.status.idle":"2021-10-10T20:50:37.923251Z","shell.execute_reply.started":"2021-10-10T20:49:15.376232Z","shell.execute_reply":"2021-10-10T20:50:37.922269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝We have our text appended in our train dataframe
","metadata":{"papermill":{"duration":0.071213,"end_time":"2021-06-16T14:13:21.383761","exception":false,"start_time":"2021-06-16T14:13:21.312548","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df.head()","metadata":{"papermill":{"duration":0.094384,"end_time":"2021-06-16T14:13:21.549923","exception":false,"start_time":"2021-06-16T14:13:21.455539","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:50:37.924706Z","iopub.execute_input":"2021-10-10T20:50:37.925035Z","iopub.status.idle":"2021-10-10T20:50:37.939799Z","shell.execute_reply.started":"2021-10-10T20:50:37.925001Z","shell.execute_reply":"2021-10-10T20:50:37.938783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas()\nsample_sub['text'] = sample_sub['Id'].progress_apply(partial(read_append_return, train_files_path=test_files_path))","metadata":{"papermill":{"duration":0.141652,"end_time":"2021-06-16T14:13:21.764429","exception":false,"start_time":"2021-06-16T14:13:21.622777","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:50:37.941132Z","iopub.execute_input":"2021-10-10T20:50:37.941438Z","iopub.status.idle":"2021-10-10T20:50:38.022106Z","shell.execute_reply.started":"2021-10-10T20:50:37.941407Z","shell.execute_reply":"2021-10-10T20:50:38.020856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝Also, we have the text of for the sample_submission file
","metadata":{"papermill":{"duration":0.07353,"end_time":"2021-06-16T14:13:21.910863","exception":false,"start_time":"2021-06-16T14:13:21.837333","status":"completed"},"tags":[]}},{"cell_type":"code","source":"sample_sub.head()","metadata":{"papermill":{"duration":0.090419,"end_time":"2021-06-16T14:13:22.075713","exception":false,"start_time":"2021-06-16T14:13:21.985294","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:50:38.025517Z","iopub.execute_input":"2021-10-10T20:50:38.025987Z","iopub.status.idle":"2021-10-10T20:50:38.041989Z","shell.execute_reply.started":"2021-10-10T20:50:38.025937Z","shell.execute_reply":"2021-10-10T20:50:38.040886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub['text']","metadata":{"papermill":{"duration":0.084817,"end_time":"2021-06-16T14:13:22.234311","exception":false,"start_time":"2021-06-16T14:13:22.149494","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:50:38.044547Z","iopub.execute_input":"2021-10-10T20:50:38.045002Z","iopub.status.idle":"2021-10-10T20:50:38.056584Z","shell.execute_reply.started":"2021-10-10T20:50:38.044953Z","shell.execute_reply":"2021-10-10T20:50:38.055469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#4. Data Cleaning
","metadata":{"papermill":{"duration":0.073503,"end_time":"2021-06-16T14:13:22.38231","exception":false,"start_time":"2021-06-16T14:13:22.308807","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def text_cleaning(text):\n '''\n Converts all text to lower case, Removes special charecters, emojis and multiple spaces\n text - Sentence that needs to be cleaned\n '''\n text = ''.join([k for k in text if k not in string.punctuation])\n text = re.sub('r[^\\w\\s]', ' ', str(text).lower()).strip()\n lem = nltk.stem.wordnet.WordNetLemmatizer()\n text = lem.lemmatize(text)\n# text = re.sub('[^A-Za-z0-9]+', ' ', str(text).lower()).strip()\n# text = re.sub(\"/'+/g\", ' ', text)\n \n return text","metadata":{"papermill":{"duration":0.083576,"end_time":"2021-06-16T14:13:22.539585","exception":false,"start_time":"2021-06-16T14:13:22.456009","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:50:38.058191Z","iopub.execute_input":"2021-10-10T20:50:38.058596Z","iopub.status.idle":"2021-10-10T20:50:38.068374Z","shell.execute_reply.started":"2021-10-10T20:50:38.058552Z","shell.execute_reply":"2021-10-10T20:50:38.067475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas()\ntrain_df['text'] = train_df['text'].progress_apply(text_cleaning)","metadata":{"papermill":{"duration":144.81972,"end_time":"2021-06-16T14:15:47.433465","exception":false,"start_time":"2021-06-16T14:13:22.613745","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:50:38.06967Z","iopub.execute_input":"2021-10-10T20:50:38.069978Z","iopub.status.idle":"2021-10-10T20:53:20.612391Z","shell.execute_reply.started":"2021-10-10T20:50:38.069947Z","shell.execute_reply":"2021-10-10T20:53:20.611198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝We have our data cleaned!
","metadata":{"papermill":{"duration":0.074975,"end_time":"2021-06-16T14:15:47.583404","exception":false,"start_time":"2021-06-16T14:15:47.508429","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df.head()","metadata":{"papermill":{"duration":0.102526,"end_time":"2021-06-16T14:15:47.761406","exception":false,"start_time":"2021-06-16T14:15:47.65888","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:20.614304Z","iopub.execute_input":"2021-10-10T20:53:20.614796Z","iopub.status.idle":"2021-10-10T20:53:20.630608Z","shell.execute_reply.started":"2021-10-10T20:53:20.6147Z","shell.execute_reply":"2021-10-10T20:53:20.62947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"text = train_df['text']\nprint(text)","metadata":{"papermill":{"duration":0.091689,"end_time":"2021-06-16T14:15:47.944692","exception":false,"start_time":"2021-06-16T14:15:47.853003","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:20.631679Z","iopub.execute_input":"2021-10-10T20:53:20.632003Z","iopub.status.idle":"2021-10-10T20:53:20.647094Z","shell.execute_reply.started":"2021-10-10T20:53:20.631972Z","shell.execute_reply":"2021-10-10T20:53:20.646016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝Remove stop words from cleaned_label of train.csv
","metadata":{"papermill":{"duration":0.076637,"end_time":"2021-06-16T14:15:48.100256","exception":false,"start_time":"2021-06-16T14:15:48.023619","status":"completed"},"tags":[]}},{"cell_type":"code","source":"words =list(train_df['cleaned_label'].values)\nstopwords=['ourselves', 'hers','the','of','and','in', 'between', 'yourself', 'but', 'again','of', 'there', 'about', 'once', 'during', 'out', 'very', 'having', 'with', 'they', 'own', 'an', 'be', 'some', 'for', 'do', 'its', 'yours', 'such', 'into', 'of', 'most', 'itself', 'other', 'off', 'is', 's', 'am', 'or', 'who', 'as', 'from', 'him', 'each', 'the', 'themselves', 'until', 'below', 'are', 'we', 'these', 'your', 'his', 'through', 'don', 'nor', 'me', 'were', 'her', 'more', 'himself', 'this', 'down', 'should', 'our', 'their', 'while', 'above', 'both', 'up', 'to', 'ours', 'had', 'she', 'all', 'no', 'when', 'at', 'any', 'before', 'them', 'same', 'and', 'been', 'have', 'in', 'will', 'on', 'does', 'yourselves', 'then', 'that', 'because', 'what', 'over', 'why', 'so', 'can', 'did', 'not', 'now', 'under', 'he', 'you', 'herself', 'has', 'just', 'where', 'too', 'only', 'myself', 'which', 'those', 'i', 'after', 'few', 'whom', 't', 'being', 'if', 'theirs', 'my', 'against', 'a', 'by', 'doing', 'it', 'how', 'further', 'was', 'here', 'than']\nsplit_words=[]\nfor word in words:\n lo_w=[]\n list_of_words=str(word).split()\n for w in list_of_words:\n if w not in stopwords:\n lo_w.append(w)\n split_words.append(lo_w)\nallwords = []\nfor wordlist in split_words:\n allwords += wordlist","metadata":{"papermill":{"duration":0.2277,"end_time":"2021-06-16T14:15:48.404217","exception":false,"start_time":"2021-06-16T14:15:48.176517","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:20.648832Z","iopub.execute_input":"2021-10-10T20:53:20.649461Z","iopub.status.idle":"2021-10-10T20:53:20.842847Z","shell.execute_reply.started":"2021-10-10T20:53:20.649412Z","shell.execute_reply":"2021-10-10T20:53:20.842021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Command to do a regex substitution*","metadata":{"papermill":{"duration":0.076885,"end_time":"2021-06-16T14:15:49.01838","exception":false,"start_time":"2021-06-16T14:15:48.941495","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def clean_text(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","metadata":{"papermill":{"duration":0.084924,"end_time":"2021-06-16T14:15:49.179635","exception":false,"start_time":"2021-06-16T14:15:49.094711","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:20.84417Z","iopub.execute_input":"2021-10-10T20:53:20.844847Z","iopub.status.idle":"2021-10-10T20:53:20.850085Z","shell.execute_reply.started":"2021-10-10T20:53:20.844801Z","shell.execute_reply":"2021-10-10T20:53:20.84895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#5. Baseline model and Submission
","metadata":{"papermill":{"duration":0.075559,"end_time":"2021-06-16T14:15:49.330857","exception":false,"start_time":"2021-06-16T14:15:49.255298","status":"completed"},"tags":[]}},{"cell_type":"code","source":"temp_1 = [x.lower() for x in train_df['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train_df['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train_df['cleaned_label'].unique()]\n\nexisting_labels = set(temp_1 + temp_2 + temp_3)\nid_list = []\nlables_list = []\nfor index, row in tqdm(sample_sub.iterrows()):\n sample_text = row['text']\n row_id = row['Id']\n temp_df = train_df[train_df['text'] == text_cleaning(sample_text)]\n cleaned_labels = temp_df['cleaned_label'].to_list()\n for known_label in existing_labels:\n if known_label in sample_text.lower():\n cleaned_labels.append(clean_text(known_label))\n cleaned_labels = [clean_text(x) for x in cleaned_labels]\n cleaned_labels = set(cleaned_labels)\n lables_list.append('|'.join(cleaned_labels))\n id_list.append(row_id)","metadata":{"papermill":{"duration":0.437865,"end_time":"2021-06-16T14:15:49.84447","exception":false,"start_time":"2021-06-16T14:15:49.406605","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:20.85163Z","iopub.execute_input":"2021-10-10T20:53:20.852297Z","iopub.status.idle":"2021-10-10T20:53:21.260494Z","shell.execute_reply.started":"2021-10-10T20:53:20.852218Z","shell.execute_reply":"2021-10-10T20:53:21.259471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission1 = pd.DataFrame()\nsubmission1['Id'] = id_list\nsubmission1['PredictionString'] = lables_list","metadata":{"papermill":{"duration":0.087057,"end_time":"2021-06-16T14:15:50.008621","exception":false,"start_time":"2021-06-16T14:15:49.921564","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:21.261742Z","iopub.execute_input":"2021-10-10T20:53:21.262048Z","iopub.status.idle":"2021-10-10T20:53:21.268966Z","shell.execute_reply.started":"2021-10-10T20:53:21.262019Z","shell.execute_reply":"2021-10-10T20:53:21.26791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission1.head()","metadata":{"papermill":{"duration":0.089898,"end_time":"2021-06-16T14:15:50.177522","exception":false,"start_time":"2021-06-16T14:15:50.087624","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:21.2703Z","iopub.execute_input":"2021-10-10T20:53:21.270593Z","iopub.status.idle":"2021-10-10T20:53:21.286178Z","shell.execute_reply.started":"2021-10-10T20:53:21.270566Z","shell.execute_reply":"2021-10-10T20:53:21.285103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from spacy import displacy\nfrom collections import Counter\nimport en_core_web_lg\nnlp = en_core_web_lg.load()","metadata":{"papermill":{"duration":6.785295,"end_time":"2021-06-16T14:15:57.191431","exception":false,"start_time":"2021-06-16T14:15:50.406136","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:21.287581Z","iopub.execute_input":"2021-10-10T20:53:21.288074Z","iopub.status.idle":"2021-10-10T20:53:28.713123Z","shell.execute_reply.started":"2021-10-10T20:53:21.288033Z","shell.execute_reply":"2021-10-10T20:53:28.712141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2 = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')","metadata":{"papermill":{"duration":0.091336,"end_time":"2021-06-16T14:15:57.360887","exception":false,"start_time":"2021-06-16T14:15:57.269551","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:28.717866Z","iopub.execute_input":"2021-10-10T20:53:28.718206Z","iopub.status.idle":"2021-10-10T20:53:28.72951Z","shell.execute_reply.started":"2021-10-10T20:53:28.718176Z","shell.execute_reply":"2021-10-10T20:53:28.728544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas()\nsample_sub2['text'] = sample_sub2['Id'].progress_apply(partial(read_append_return, train_files_path=test_files_path))","metadata":{"papermill":{"duration":0.13782,"end_time":"2021-06-16T14:15:57.575626","exception":false,"start_time":"2021-06-16T14:15:57.437806","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:28.731122Z","iopub.execute_input":"2021-10-10T20:53:28.731634Z","iopub.status.idle":"2021-10-10T20:53:28.800913Z","shell.execute_reply.started":"2021-10-10T20:53:28.731582Z","shell.execute_reply":"2021-10-10T20:53:28.799831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2['text'] = sample_sub2['text'].str.replace('[^\\w\\s]','')\nsample_sub2.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:28.80216Z","iopub.execute_input":"2021-10-10T20:53:28.802459Z","iopub.status.idle":"2021-10-10T20:53:28.833233Z","shell.execute_reply.started":"2021-10-10T20:53:28.802428Z","shell.execute_reply":"2021-10-10T20:53:28.832153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#del train_df\n# del sample_sub","metadata":{"papermill":{"duration":0.140234,"end_time":"2021-06-16T14:15:57.970392","exception":false,"start_time":"2021-06-16T14:15:57.830158","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:28.838221Z","iopub.execute_input":"2021-10-10T20:53:28.838531Z","iopub.status.idle":"2021-10-10T20:53:28.841865Z","shell.execute_reply.started":"2021-10-10T20:53:28.838502Z","shell.execute_reply":"2021-10-10T20:53:28.840794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2.head()","metadata":{"papermill":{"duration":0.094106,"end_time":"2021-06-16T14:15:58.146294","exception":false,"start_time":"2021-06-16T14:15:58.052188","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:28.852108Z","iopub.execute_input":"2021-10-10T20:53:28.852551Z","iopub.status.idle":"2021-10-10T20:53:28.870934Z","shell.execute_reply.started":"2021-10-10T20:53:28.852502Z","shell.execute_reply":"2021-10-10T20:53:28.869753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\ngc.collect()","metadata":{"papermill":{"duration":0.270269,"end_time":"2021-06-16T14:15:58.495758","exception":false,"start_time":"2021-06-16T14:15:58.225489","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:28.872493Z","iopub.execute_input":"2021-10-10T20:53:28.872846Z","iopub.status.idle":"2021-10-10T20:53:29.066523Z","shell.execute_reply.started":"2021-10-10T20:53:28.872813Z","shell.execute_reply":"2021-10-10T20:53:29.065135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datalist = pd.read_csv('../input/para-data/datalist.csv')","metadata":{"papermill":{"duration":0.101299,"end_time":"2021-06-16T14:15:58.675928","exception":false,"start_time":"2021-06-16T14:15:58.574629","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:29.068468Z","iopub.execute_input":"2021-10-10T20:53:29.06905Z","iopub.status.idle":"2021-10-10T20:53:29.089999Z","shell.execute_reply.started":"2021-10-10T20:53:29.069Z","shell.execute_reply":"2021-10-10T20:53:29.089047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datalist.head()","metadata":{"papermill":{"duration":0.11767,"end_time":"2021-06-16T14:15:58.878224","exception":false,"start_time":"2021-06-16T14:15:58.760554","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:29.094646Z","iopub.execute_input":"2021-10-10T20:53:29.095008Z","iopub.status.idle":"2021-10-10T20:53:29.103773Z","shell.execute_reply.started":"2021-10-10T20:53:29.094974Z","shell.execute_reply":"2021-10-10T20:53:29.102773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datasetLabels= datalist['Data']\nprint(datasetLabels)","metadata":{"papermill":{"duration":0.088763,"end_time":"2021-06-16T14:15:59.051597","exception":false,"start_time":"2021-06-16T14:15:58.962834","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:29.105205Z","iopub.execute_input":"2021-10-10T20:53:29.105512Z","iopub.status.idle":"2021-10-10T20:53:29.118373Z","shell.execute_reply.started":"2021-10-10T20:53:29.105484Z","shell.execute_reply":"2021-10-10T20:53:29.117313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"a = [(x,i) for x, y in zip(sample_sub2['Id'],sample_sub2['text']) for i in datasetLabels if i in y]\nprint (a)","metadata":{"papermill":{"duration":0.093314,"end_time":"2021-06-16T14:16:03.269571","exception":false,"start_time":"2021-06-16T14:16:03.176257","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:29.119859Z","iopub.execute_input":"2021-10-10T20:53:29.120313Z","iopub.status.idle":"2021-10-10T20:53:29.134156Z","shell.execute_reply.started":"2021-10-10T20:53:29.120268Z","shell.execute_reply":"2021-10-10T20:53:29.132971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2 = pd.DataFrame(a, columns=['Id', 'PredictionString'])\nsample_sub2 = sample_sub2.groupby('Id').agg({'PredictionString': '|'.join}).reset_index()\nsample_sub2.head()","metadata":{"papermill":{"duration":0.111874,"end_time":"2021-06-16T14:16:03.467728","exception":false,"start_time":"2021-06-16T14:16:03.355854","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:29.135661Z","iopub.execute_input":"2021-10-10T20:53:29.136071Z","iopub.status.idle":"2021-10-10T20:53:29.163939Z","shell.execute_reply.started":"2021-10-10T20:53:29.136037Z","shell.execute_reply":"2021-10-10T20:53:29.162804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub2['PredictionString'] = [x.lower() for x in sample_sub2['PredictionString'].unique()]","metadata":{"papermill":{"duration":0.093088,"end_time":"2021-06-16T14:16:03.82765","exception":false,"start_time":"2021-06-16T14:16:03.734562","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:29.165537Z","iopub.execute_input":"2021-10-10T20:53:29.165973Z","iopub.status.idle":"2021-10-10T20:53:29.172192Z","shell.execute_reply.started":"2021-10-10T20:53:29.165931Z","shell.execute_reply":"2021-10-10T20:53:29.171048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#my_list = [\"cohort study [lbc1936; n\",\"lothian birth cohort study [lbc1936\",\"Information Resource Incorporated (IRI\",\"on@3\", \"two#\", \"thre%e\"]\nremovetable = str.maketrans('', '', ',@#%([')\nsample_sub2['PredictionString'] = [s.translate(removetable) for s in sample_sub2['PredictionString']]\nprint(sample_sub2['PredictionString'])","metadata":{"papermill":{"duration":0.097129,"end_time":"2021-06-16T14:16:04.009195","exception":false,"start_time":"2021-06-16T14:16:03.912066","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:29.173818Z","iopub.execute_input":"2021-10-10T20:53:29.174502Z","iopub.status.idle":"2021-10-10T20:53:29.191912Z","shell.execute_reply.started":"2021-10-10T20:53:29.174454Z","shell.execute_reply":"2021-10-10T20:53:29.190665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# stop_words = [\"the \"\n# ]\n\n# pat = '|'.join(r\"\\b{}\\b\".format(x) for x in stop_words)\n# sample_sub2['PredictionString'] = sample_sub2['PredictionString'].str.replace(pat, '')\n\n\n# #top_words = set(stop_words)\n# #f = lambda x: ' '.join(w for w in x.split() if not w in stop_words)\n# #newdata['Verbatim2'] = newdata['Verbatim'].apply(f)\n\n# print (sample_sub2['PredictionString'])","metadata":{"papermill":{"duration":0.09454,"end_time":"2021-06-16T14:16:04.189484","exception":false,"start_time":"2021-06-16T14:16:04.094944","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:29.193361Z","iopub.execute_input":"2021-10-10T20:53:29.193772Z","iopub.status.idle":"2021-10-10T20:53:29.200312Z","shell.execute_reply.started":"2021-10-10T20:53:29.19374Z","shell.execute_reply":"2021-10-10T20:53:29.199113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission2 = pd.DataFrame()\nsubmission2['Id'] = sample_sub2['Id']\nsubmission2['PredictionString'] = sample_sub2['PredictionString']\nsubmission2.head()","metadata":{"papermill":{"duration":0.098245,"end_time":"2021-06-16T14:16:04.371733","exception":false,"start_time":"2021-06-16T14:16:04.273488","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:29.201815Z","iopub.execute_input":"2021-10-10T20:53:29.202155Z","iopub.status.idle":"2021-10-10T20:53:29.220486Z","shell.execute_reply.started":"2021-10-10T20:53:29.202121Z","shell.execute_reply":"2021-10-10T20:53:29.219206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# govt dataset","metadata":{}},{"cell_type":"code","source":"def read_json_pub(filename, train_files_path=train_files_path, output='text'):\n json_path = os.path.join(train_files_path, (filename+'.json'))\n headings = []\n contents = []\n combined = []\n with open(json_path, 'r') as f:\n json_decode = json.load(f)\n for data in json_decode:\n headings.append(data.get('section_title'))\n contents.append(data.get('text'))\n combined.append(data.get('section_title'))\n combined.append(data.get('text'))\n \n all_headings = ' '.join(headings)\n all_contents = ' '.join(contents)\n all_data = '. '.join(combined)\n \n if output == 'text':\n return all_contents\n elif output == 'head':\n return all_headings\n else:\n return all_data","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:29.2223Z","iopub.execute_input":"2021-10-10T20:53:29.222731Z","iopub.status.idle":"2021-10-10T20:53:29.233819Z","shell.execute_reply.started":"2021-10-10T20:53:29.222683Z","shell.execute_reply":"2021-10-10T20:53:29.232743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def text_cleaning2(text):\n '''\n Converts all text to lower case, Removes special charecters, emojis and multiple spaces\n text - Sentence that needs to be cleaned\n '''\n text = re.sub('[^A-Za-z0-9]+', ' ', str(text).lower()).strip()\n text = re.sub(' +', ' ', text)\n emoji_pattern = re.compile(\"[\"\n u\"\\U0001F600-\\U0001F64F\" # emoticons\n u\"\\U0001F300-\\U0001F5FF\" # symbols & pictographs\n u\"\\U0001F680-\\U0001F6FF\" # transport & map symbols\n u\"\\U0001F1E0-\\U0001F1FF\" # flags (iOS)\n \"]+\", flags=re.UNICODE)\n text = emoji_pattern.sub(r'', text)\n return text","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:29.235628Z","iopub.execute_input":"2021-10-10T20:53:29.236408Z","iopub.status.idle":"2021-10-10T20:53:29.247963Z","shell.execute_reply.started":"2021-10-10T20:53:29.236357Z","shell.execute_reply":"2021-10-10T20:53:29.246824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def clean_text2(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:29.249386Z","iopub.execute_input":"2021-10-10T20:53:29.250021Z","iopub.status.idle":"2021-10-10T20:53:29.262887Z","shell.execute_reply.started":"2021-10-10T20:53:29.249974Z","shell.execute_reply":"2021-10-10T20:53:29.262005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# random sample for testing\ntrain_sample=train_df.sample(n = 10)\n\ntrain_sample","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:29.264299Z","iopub.execute_input":"2021-10-10T20:53:29.26499Z","iopub.status.idle":"2021-10-10T20:53:29.291251Z","shell.execute_reply.started":"2021-10-10T20:53:29.264935Z","shell.execute_reply":"2021-10-10T20:53:29.290206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"govt_df = pd.read_csv('../input/govtdataset/data_set_800.csv')","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:29.292558Z","iopub.execute_input":"2021-10-10T20:53:29.292963Z","iopub.status.idle":"2021-10-10T20:53:29.318416Z","shell.execute_reply.started":"2021-10-10T20:53:29.292922Z","shell.execute_reply":"2021-10-10T20:53:29.317293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"govt_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:29.31978Z","iopub.execute_input":"2021-10-10T20:53:29.320131Z","iopub.status.idle":"2021-10-10T20:53:29.329379Z","shell.execute_reply.started":"2021-10-10T20:53:29.320096Z","shell.execute_reply":"2021-10-10T20:53:29.328162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_time = time.time()\n\n\n#### remove >.5 jaccard matches from predicitons\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n\n#############################\n#path=train_data_path\npath=test_files_path\n\n#for training use train_sample\n\n#for submission use sample_sub\n\n#############\n\ncolumn_names = [\"Id\", \"PredictionString\"]\n\nsubmission3 = pd.DataFrame(columns = column_names)\n\nto_append=[]\nfor index, row in sample_sub.iterrows():\n to_append=[row['Id'],'']\n large_string = str(read_append_return(row['Id'],path))\n clean_string=text_cleaning2(large_string)\n for index, row2 in govt_df.iterrows():\n query_string = str(row2['title'])\n if query_string in clean_string:\n if to_append[1]!='' and clean_text2(query_string) not in to_append[1]:\n to_append[1]=to_append[1]+'|'+clean_text(query_string)\n if to_append[1]=='':\n to_append[1]=clean_text2(query_string)\n \n ###### remove similar jaccard\n #got_label=to_append[1].split('|')\n #filtered=[]\n #filtered_labels = ''\n #for label in sorted(got_label, key=len):\n #label = clean_text(label)\n #if len(filtered) == 0 or all(jaccard_similarity(label, got_label) < 1.0 for got_label in filtered):\n #filtered.append(label)\n #if filtered_labels!='':\n #filtered_labels=filtered_labels+'|'+label\n #if filtered_labels=='':\n #filtered_labels=label\n #to_append[1] = filtered_labels \n #print ('################')\n #print (to_append)\n #print (large_string)\n #print ('################')\n ###### remove similar jaccard\n govt_df_length = len(submission3)\n submission3.loc[govt_df_length] = to_append\n# submission3.to_csv('submission3.csv', index = False)\nprint(\"--- %s seconds ---\" % (time.time() - start_time))\nsubmission3","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:29.330904Z","iopub.execute_input":"2021-10-10T20:53:29.331193Z","iopub.status.idle":"2021-10-10T20:53:30.778142Z","shell.execute_reply.started":"2021-10-10T20:53:29.331164Z","shell.execute_reply":"2021-10-10T20:53:30.777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission3 = pd.DataFrame()\n# submission3['Id'] = govt_data['Id']\n# submission3['PredictionString'] = govt_data['PredictionString']\n# submission3.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:30.779726Z","iopub.execute_input":"2021-10-10T20:53:30.780167Z","iopub.status.idle":"2021-10-10T20:53:30.78466Z","shell.execute_reply.started":"2021-10-10T20:53:30.780121Z","shell.execute_reply":"2021-10-10T20:53:30.783502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📝Merging DataTables into one Submission File
","metadata":{"papermill":{"duration":0.084157,"end_time":"2021-06-16T14:16:04.54057","exception":false,"start_time":"2021-06-16T14:16:04.456413","status":"completed"},"tags":[]}},{"cell_type":"code","source":"##submission = pd.DataFrame(data= {'Id' : sample_sub['Id'], 'PredictionString' : ['','','','']})","metadata":{"papermill":{"duration":0.091457,"end_time":"2021-06-16T14:16:04.716641","exception":false,"start_time":"2021-06-16T14:16:04.625184","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.786494Z","iopub.execute_input":"2021-10-10T20:53:30.786942Z","iopub.status.idle":"2021-10-10T20:53:30.799975Z","shell.execute_reply.started":"2021-10-10T20:53:30.786901Z","shell.execute_reply":"2021-10-10T20:53:30.799153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##submission.head()","metadata":{"papermill":{"duration":0.091267,"end_time":"2021-06-16T14:16:04.8934","exception":false,"start_time":"2021-06-16T14:16:04.802133","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.801362Z","iopub.execute_input":"2021-10-10T20:53:30.802Z","iopub.status.idle":"2021-10-10T20:53:30.811853Z","shell.execute_reply.started":"2021-10-10T20:53:30.801951Z","shell.execute_reply":"2021-10-10T20:53:30.81091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"merged = pd.concat([submission1, submission2,submission3], sort=True).drop_duplicates()\nsubmission = merged.dropna().groupby(['Id'], as_index=False).agg({'PredictionString' : '|'.join}).reset_index(drop=True)\nprint(submission)","metadata":{"papermill":{"duration":0.115711,"end_time":"2021-06-16T14:16:05.093586","exception":false,"start_time":"2021-06-16T14:16:04.977875","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.813206Z","iopub.execute_input":"2021-10-10T20:53:30.813849Z","iopub.status.idle":"2021-10-10T20:53:30.849378Z","shell.execute_reply.started":"2021-10-10T20:53:30.813804Z","shell.execute_reply":"2021-10-10T20:53:30.848324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(pd.concat(submission.melt(id_vars='Id').dropna() for df in [submission1,submission2])\n .groupby(['Id','variable'])['value'].apply(lambda x: '|'.join(x.unique()))\n .unstack()\n)","metadata":{"papermill":{"duration":0.113506,"end_time":"2021-06-16T14:16:05.294166","exception":false,"start_time":"2021-06-16T14:16:05.18066","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.850637Z","iopub.execute_input":"2021-10-10T20:53:30.850982Z","iopub.status.idle":"2021-10-10T20:53:30.881413Z","shell.execute_reply.started":"2021-10-10T20:53:30.85095Z","shell.execute_reply":"2021-10-10T20:53:30.880452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# remove Duplicates from dataframe","metadata":{"papermill":{"duration":0.085319,"end_time":"2021-06-16T14:16:05.464978","exception":false,"start_time":"2021-06-16T14:16:05.379659","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def remove_dup(strng):\n '''\n Input a string and split them \n '''\n return '|'.join(list(dict.fromkeys(strng.split('|'))))","metadata":{"papermill":{"duration":0.092575,"end_time":"2021-06-16T14:16:05.643636","exception":false,"start_time":"2021-06-16T14:16:05.551061","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.882886Z","iopub.execute_input":"2021-10-10T20:53:30.88318Z","iopub.status.idle":"2021-10-10T20:53:30.887912Z","shell.execute_reply.started":"2021-10-10T20:53:30.883151Z","shell.execute_reply":"2021-10-10T20:53:30.886768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# my_dict = {'Tags':[\"Museum, Art Museum, Shopping, Museum\",'Drink, Drink','Shop','Visit'],'Country':['USA','USA','USA', 'USA']}\n# df = pd.DataFrame(my_dict)\nsubmission['PredictionString'] = submission['PredictionString'].apply(lambda x: remove_dup(x))\nsubmission","metadata":{"papermill":{"duration":0.098577,"end_time":"2021-06-16T14:16:05.827942","exception":false,"start_time":"2021-06-16T14:16:05.729365","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.889392Z","iopub.execute_input":"2021-10-10T20:53:30.889886Z","iopub.status.idle":"2021-10-10T20:53:30.908902Z","shell.execute_reply.started":"2021-10-10T20:53:30.889812Z","shell.execute_reply":"2021-10-10T20:53:30.907363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Sorting in a alphabetical Order**","metadata":{"papermill":{"duration":0.087186,"end_time":"2021-06-16T14:16:06.182376","exception":false,"start_time":"2021-06-16T14:16:06.09519","status":"completed"},"tags":[]}},{"cell_type":"code","source":"submission['PredictionString'] = submission['PredictionString'].apply(lambda x: '|'.join(sorted(x.split('|'))))\nprint(submission)","metadata":{"papermill":{"duration":0.09746,"end_time":"2021-06-16T14:16:06.366726","exception":false,"start_time":"2021-06-16T14:16:06.269266","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.910372Z","iopub.execute_input":"2021-10-10T20:53:30.910725Z","iopub.status.idle":"2021-10-10T20:53:30.919621Z","shell.execute_reply.started":"2021-10-10T20:53:30.910646Z","shell.execute_reply":"2021-10-10T20:53:30.917927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['PredictionString'][1]","metadata":{"papermill":{"duration":0.094812,"end_time":"2021-06-16T14:16:06.548554","exception":false,"start_time":"2021-06-16T14:16:06.453742","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.921577Z","iopub.execute_input":"2021-10-10T20:53:30.922044Z","iopub.status.idle":"2021-10-10T20:53:30.934583Z","shell.execute_reply.started":"2021-10-10T20:53:30.921997Z","shell.execute_reply":"2021-10-10T20:53:30.933446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{"papermill":{"duration":0.099874,"end_time":"2021-06-16T14:16:06.736394","exception":false,"start_time":"2021-06-16T14:16:06.63652","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.935935Z","iopub.execute_input":"2021-10-10T20:53:30.936236Z","iopub.status.idle":"2021-10-10T20:53:30.952378Z","shell.execute_reply.started":"2021-10-10T20:53:30.936207Z","shell.execute_reply":"2021-10-10T20:53:30.951128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{"papermill":{"duration":0.100202,"end_time":"2021-06-16T14:16:06.924659","exception":false,"start_time":"2021-06-16T14:16:06.824457","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.953798Z","iopub.execute_input":"2021-10-10T20:53:30.954225Z","iopub.status.idle":"2021-10-10T20:53:30.968227Z","shell.execute_reply.started":"2021-10-10T20:53:30.954191Z","shell.execute_reply":"2021-10-10T20:53:30.967202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#test = pd.concat([train_df,train_df],axis=0,ignore_index=True)","metadata":{"papermill":{"duration":0.095839,"end_time":"2021-06-16T14:16:07.109154","exception":false,"start_time":"2021-06-16T14:16:07.013315","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-10-10T20:53:30.970541Z","iopub.execute_input":"2021-10-10T20:53:30.971035Z","iopub.status.idle":"2021-10-10T20:53:30.976524Z","shell.execute_reply.started":"2021-10-10T20:53:30.970984Z","shell.execute_reply":"2021-10-10T20:53:30.975032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PredictionString Analysis","metadata":{}},{"cell_type":"markdown","source":"# Baseline Modelling ","metadata":{}},{"cell_type":"code","source":"submission1['PredictionString'][0]","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:30.978359Z","iopub.execute_input":"2021-10-10T20:53:30.979059Z","iopub.status.idle":"2021-10-10T20:53:30.990866Z","shell.execute_reply.started":"2021-10-10T20:53:30.979006Z","shell.execute_reply":"2021-10-10T20:53:30.989742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fine Tunning Through Huggingface Models","metadata":{}},{"cell_type":"code","source":"submission2['PredictionString'][0]","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:30.992456Z","iopub.execute_input":"2021-10-10T20:53:30.993149Z","iopub.status.idle":"2021-10-10T20:53:31.002938Z","shell.execute_reply.started":"2021-10-10T20:53:30.993102Z","shell.execute_reply":"2021-10-10T20:53:31.001948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Govt Datasets","metadata":{}},{"cell_type":"code","source":"submission3['PredictionString'][0]","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:31.00414Z","iopub.execute_input":"2021-10-10T20:53:31.004412Z","iopub.status.idle":"2021-10-10T20:53:31.016958Z","shell.execute_reply.started":"2021-10-10T20:53:31.004385Z","shell.execute_reply":"2021-10-10T20:53:31.015756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PredictionString after Merging dataframes","metadata":{}},{"cell_type":"code","source":"submission['PredictionString'][0]","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:31.01839Z","iopub.execute_input":"2021-10-10T20:53:31.018942Z","iopub.status.idle":"2021-10-10T20:53:31.026689Z","shell.execute_reply.started":"2021-10-10T20:53:31.018899Z","shell.execute_reply":"2021-10-10T20:53:31.025959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.cluster import KMeans \nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.decomposition import PCA\nfrom sklearn.preprocessing import normalize\nfrom sklearn.metrics import pairwise_distances\nfrom nltk.tokenize import word_tokenize\nfrom nltk.stem.porter import PorterStemmer\nfrom nltk.corpus import stopwords\n\nfrom scipy.stats import multivariate_normal as mvn\nimport nltk\nimport os\nimport random\n\nplt.style.use('fivethirtyeight')\n","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:31.027915Z","iopub.execute_input":"2021-10-10T20:53:31.028397Z","iopub.status.idle":"2021-10-10T20:53:31.228405Z","shell.execute_reply.started":"2021-10-10T20:53:31.028356Z","shell.execute_reply":"2021-10-10T20:53:31.227282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer\ndata = submission['PredictionString']\n\n\ntf_idf_vectorizor = TfidfVectorizer(stop_words = 'english',#tokenizer = tokenize_and_stem,\n max_features = 20000)\ntf_idf = tf_idf_vectorizor.fit_transform(data)\ntf_idf_norm = normalize(tf_idf)\ntf_idf_array = tf_idf_norm.toarray()","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:31.23Z","iopub.execute_input":"2021-10-10T20:53:31.230326Z","iopub.status.idle":"2021-10-10T20:53:31.247536Z","shell.execute_reply.started":"2021-10-10T20:53:31.230292Z","shell.execute_reply":"2021-10-10T20:53:31.24643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Kmeans:\n \"\"\" K Means Clustering\n \n Parameters\n -----------\n k: int , number of clusters\n \n seed: int, will be randomly set if None\n \n max_iter: int, number of iterations to run algorithm, default: 200\n \n Attributes\n -----------\n centroids: array, k, number_features\n \n cluster_labels: label for each data point\n \n \"\"\"\n \n def __init__(self, k, seed = None, max_iter = 200):\n self.k = k\n self.seed = seed\n if self.seed is not None:\n np.random.seed(self.seed)\n self.max_iter = max_iter\n \n \n \n def initialise_centroids(self, data):\n \"\"\"Randomly Initialise Centroids\n \n Parameters\n ----------\n data: array or matrix, number_rows, number_features\n \n Returns\n --------\n centroids: array of k centroids chosen as random data points \n \"\"\"\n \n initial_centroids = np.random.permutation(data.shape[0])[:self.k]\n self.centroids = data[initial_centroids]\n\n return self.centroids\n \n \n def assign_clusters(self, data):\n \"\"\"Compute distance of data from clusters and assign data point\n to closest cluster.\n \n Parameters\n ----------\n data: array or matrix, number_rows, number_features\n \n Returns\n --------\n cluster_labels: index which minmises the distance of data to each\n cluster\n \n \"\"\"\n \n if data.ndim == 1:\n data = data.reshape(-1, 1)\n \n dist_to_centroid = pairwise_distances(data, self.centroids, metric = 'euclidean')\n self.cluster_labels = np.argmin(dist_to_centroid, axis = 1)\n \n return self.cluster_labels\n \n \n def update_centroids(self, data):\n \"\"\"Computes average of all data points in cluster and\n assigns new centroids as average of data points\n \n Parameters\n -----------\n data: array or matrix, number_rows, number_features\n \n Returns\n -----------\n centroids: array, k, number_features\n \"\"\"\n \n self.centroids = np.array([data[self.cluster_labels == i].mean(axis = 0) for i in range(self.k)])\n \n return self.centroids\n \n \n def convergence_calculation(self):\n \"\"\"\n Calculates \n \n \"\"\"\n pass\n \n def predict(self, data):\n \"\"\"Predict which cluster data point belongs to\n \n Parameters\n ----------\n data: array or matrix, number_rows, number_features\n \n Returns\n --------\n cluster_labels: index which minmises the distance of data to each\n cluster\n \"\"\"\n \n return self.assign_clusters(data)\n \n def fit_kmeans(self, data):\n \"\"\"\n This function contains the main loop to fit the algorithm\n Implements initialise centroids and update_centroids\n according to max_iter\n -----------------------\n \n Returns\n -------\n instance of kmeans class\n \n \"\"\"\n self.centroids = self.initialise_centroids(data)\n \n # Main kmeans loop\n for iter in range(self.max_iter):\n\n self.cluster_labels = self.assign_clusters(data)\n self.centroids = self.update_centroids(data) \n if iter % 100 == 0:\n print(\"Running Model Iteration %d \" %iter)\n print(\"Model finished running\")\n return self ","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:31.249158Z","iopub.execute_input":"2021-10-10T20:53:31.249894Z","iopub.status.idle":"2021-10-10T20:53:31.266347Z","shell.execute_reply.started":"2021-10-10T20:53:31.249832Z","shell.execute_reply":"2021-10-10T20:53:31.264913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.datasets import make_blobs\n# create blobs\ndata = make_blobs(n_samples=200, n_features=2, centers=3, cluster_std=1.6, random_state=50)\n# create np array for data points\npoints = data[0]\n# create scatter plot\nplt.scatter(data[0][:,0], data[0][:,1], c=data[1], cmap='viridis')\nplt.xlim(-15,15)\nplt.ylim(-15,15)\n\nX = data[0]\nX[2]","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:31.267637Z","iopub.execute_input":"2021-10-10T20:53:31.268069Z","iopub.status.idle":"2021-10-10T20:53:31.455555Z","shell.execute_reply.started":"2021-10-10T20:53:31.268033Z","shell.execute_reply":"2021-10-10T20:53:31.454535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_k = Kmeans(4, 1, 600)\ntemp_fitted = temp_k.fit_kmeans(X)\nnew_data = np.array([[1.066, -8.66],\n [1.87876, -6.516],\n [-1.59728965, 8.45369045],\n [1.87876, -6.516]])\ntemp_fitted.predict(new_data)","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:31.456885Z","iopub.execute_input":"2021-10-10T20:53:31.457156Z","iopub.status.idle":"2021-10-10T20:53:31.64728Z","shell.execute_reply.started":"2021-10-10T20:53:31.457129Z","shell.execute_reply":"2021-10-10T20:53:31.646257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sklearn_pca = PCA(n_components = 2)\nY_sklearn = sklearn_pca.fit_transform(tf_idf_array)\ntest_e = Kmeans(3, 1, 600)\n%time fitted = test_e.fit_kmeans(Y_sklearn)\npredicted_values = test_e.predict(Y_sklearn)\n\nplt.scatter(Y_sklearn[:, 0], Y_sklearn[:, 1], c=predicted_values, s=50, cmap='viridis')\n\ncenters = fitted.centroids\nplt.scatter(centers[:, 0], centers[:, 1],c='black', s=300, alpha=0.6);","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:31.648456Z","iopub.execute_input":"2021-10-10T20:53:31.648718Z","iopub.status.idle":"2021-10-10T20:53:32.028101Z","shell.execute_reply.started":"2021-10-10T20:53:31.648692Z","shell.execute_reply":"2021-10-10T20:53:32.026761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# SK learn Implementation","metadata":{}},{"cell_type":"code","source":"from sklearn.cluster import KMeans\nn_clusters = 3\nsklearn_pca = PCA(n_components = 2)\nY_sklearn = sklearn_pca.fit_transform(tf_idf_array)\nkmeans = KMeans(n_clusters= n_clusters, max_iter=600, algorithm = 'auto')\n%time fitted = kmeans.fit(Y_sklearn)\nprediction = kmeans.predict(Y_sklearn)\n\nplt.scatter(Y_sklearn[:, 0], Y_sklearn[:, 1],c=prediction ,s=50, cmap='viridis')\n\ncenters2 = fitted.cluster_centers_\nplt.scatter(centers2[:, 0], centers2[:, 1],c='black', s=300, alpha=0.6);","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:32.029634Z","iopub.execute_input":"2021-10-10T20:53:32.030036Z","iopub.status.idle":"2021-10-10T20:53:32.226186Z","shell.execute_reply.started":"2021-10-10T20:53:32.029995Z","shell.execute_reply":"2021-10-10T20:53:32.225017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Optimal Clusters","metadata":{}},{"cell_type":"code","source":"number_clusters = range(1, 5)\n\nkmeans = [KMeans(n_clusters=i, max_iter = 600) for i in number_clusters]\nkmeans\n\nscore = [kmeans[i].fit(Y_sklearn).score(Y_sklearn) for i in range(len(kmeans))]\nscore = [i*-1 for i in score]\n\nplt.plot(number_clusters, score)\nplt.xlabel('Number of Clusters')\nplt.ylabel('Score')\nplt.title('Elbow Method')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:32.227833Z","iopub.execute_input":"2021-10-10T20:53:32.228278Z","iopub.status.idle":"2021-10-10T20:53:32.645612Z","shell.execute_reply.started":"2021-10-10T20:53:32.228234Z","shell.execute_reply":"2021-10-10T20:53:32.644563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Extracting top features","metadata":{}},{"cell_type":"code","source":"def get_top_features_cluster(tf_idf_array, prediction, n_feats):\n labels = np.unique(prediction)\n dfs = []\n for label in labels:\n id_temp = np.where(prediction==label) # indices for each cluster\n x_means = np.mean(tf_idf_array[id_temp], axis = 0) # returns average score across cluster\n sorted_means = np.argsort(x_means)[::-1][:n_feats] # indices with top 20 scores\n features = tf_idf_vectorizor.get_feature_names()\n best_features = [(features[i], x_means[i]) for i in sorted_means]\n df = pd.DataFrame(best_features, columns = ['features', 'score'])\n dfs.append(df)\n return dfs\ndfs = get_top_features_cluster(tf_idf_array, prediction, 20)","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:32.65381Z","iopub.execute_input":"2021-10-10T20:53:32.654171Z","iopub.status.idle":"2021-10-10T20:53:32.666426Z","shell.execute_reply.started":"2021-10-10T20:53:32.654137Z","shell.execute_reply":"2021-10-10T20:53:32.665243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(8,6))\nsns.set(font_scale = 2)\nsns.barplot(x = 'score' , y = 'features', orient = 'h' , data = dfs[0][:15])","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:32.668347Z","iopub.execute_input":"2021-10-10T20:53:32.668795Z","iopub.status.idle":"2021-10-10T20:53:32.969109Z","shell.execute_reply.started":"2021-10-10T20:53:32.668745Z","shell.execute_reply":"2021-10-10T20:53:32.968023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8,6))\nsns.set(font_scale = 2)\nsns.barplot(x = 'score' , y = 'features', orient = 'h' , data = dfs[1][:15])","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:32.970638Z","iopub.execute_input":"2021-10-10T20:53:32.971244Z","iopub.status.idle":"2021-10-10T20:53:33.261447Z","shell.execute_reply.started":"2021-10-10T20:53:32.971196Z","shell.execute_reply":"2021-10-10T20:53:33.260355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8,6))\nsns.set(font_scale = 2)\nsns.barplot(x = 'score' , y = 'features', orient = 'h' , data = dfs[2][:15])","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:33.263214Z","iopub.execute_input":"2021-10-10T20:53:33.263657Z","iopub.status.idle":"2021-10-10T20:53:33.552715Z","shell.execute_reply.started":"2021-10-10T20:53:33.263607Z","shell.execute_reply":"2021-10-10T20:53:33.551423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i, df in enumerate(dfs):\n df.to_csv('df_'+str(i)+'.csv')","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:33.554451Z","iopub.execute_input":"2021-10-10T20:53:33.554928Z","iopub.status.idle":"2021-10-10T20:53:33.565292Z","shell.execute_reply.started":"2021-10-10T20:53:33.554861Z","shell.execute_reply":"2021-10-10T20:53:33.564179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_features(dfs):\n fig = plt.figure(figsize=(14,12))\n x = np.arange(len(dfs[0]))\n for i, df in enumerate(dfs):\n ax = fig.add_subplot(1, len(dfs), i+1)\n ax.set_title(\"Cluster: \"+ str(i), fontsize = 14)\n ax.spines[\"top\"].set_visible(False)\n ax.spines[\"right\"].set_visible(False)\n ax.set_frame_on(False)\n ax.get_xaxis().tick_bottom()\n ax.get_yaxis().tick_left()\n ax.ticklabel_format(axis='x', style='sci', scilimits=(-2,2))\n ax.barh(x, df.score, align='center', color='#40826d')\n yticks = ax.set_yticklabels(df.features)\n plt.show();\nplot_features(dfs)","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:53:33.566727Z","iopub.execute_input":"2021-10-10T20:53:33.56703Z","iopub.status.idle":"2021-10-10T20:53:34.100104Z","shell.execute_reply.started":"2021-10-10T20:53:33.567002Z","shell.execute_reply":"2021-10-10T20:53:34.098947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\nplt.figure(figsize=(100,50))\nplt.scatter(submission['PredictionString'],submission['Id'])\nplt.xlabel('Weight')\nplt.ylabel('Height')\nplt.title('Data Distribution')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-10T20:55:18.409659Z","iopub.execute_input":"2021-10-10T20:55:18.410538Z","iopub.status.idle":"2021-10-10T20:55:20.196941Z","shell.execute_reply.started":"2021-10-10T20:55:18.41049Z","shell.execute_reply":"2021-10-10T20:55:20.196014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **# References**","metadata":{}},{"cell_type":"markdown","source":"[https://www.kaggle.com/code/tayyabahussain/para-data](http://)\n\n[https://www.kaggle.com/code/tayyabahussain/coleridge-initiative-eda-modelling](http://)","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"Hope you like this kernel.\n Please don't forget to upvote and leave your valuable comment. \n Thank you!
😁","metadata":{}}]}