{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#

Coleridge Initiative - Show US the Data
Discover how data is used for the public good

","metadata":{}},{"cell_type":"markdown","source":"> 📑 Context : This competition challenges data scientists to show how publicly funded data are used to serve science and society. Evidence through data is critical if government is to address the many threats facing society, including; pandemics, climate change, Alzheimer’s disease, child hunger, increasing food production, maintaining biodiversity, and addressing many other challenges. Yet much of the information about data necessary to inform evidence and science is locked inside publications.\n\n> In this competition, you'll use natural language processing (NLP) to automate the discovery of how scientific data are referenced in publications. Utilizing the full text of scientific publications from numerous research areas gathered from CHORUS publisher members and other sources, you'll identify data sets that the publications' authors used in their work.\n\n> 📌 Goal : The objective of the competition is to identify the mention of datasets within scientific publications. ","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:34:57.610892Z","iopub.execute_input":"2021-07-07T13:34:57.611253Z","iopub.status.idle":"2021-07-07T13:34:57.618761Z","shell.execute_reply.started":"2021-07-07T13:34:57.611178Z","shell.execute_reply":"2021-07-07T13:34:57.617259Z"}}},{"cell_type":"markdown","source":"\n##

Table of Content

\n* [1. Importing necessary packages and libraries📚](#1)\n* [2. Loading the data ⌛](#2)\n* [3. Data Pre-Processing🔧](#4)\n* [4. Matching 📑](#4)\n* [5. Masked Language Modling 🤗](#5)\n","metadata":{}},{"cell_type":"markdown","source":"\n##

Importing necessary packages and libraries📚

","metadata":{}},{"cell_type":"markdown","source":"## Install packages :","metadata":{}},{"cell_type":"code","source":"!pip install datasets --no-index --find-links=file:///kaggle/input/coleridge-packages/packages/datasets\n!pip install ../input/coleridge-packages/seqeval-1.2.2-py3-none-any.whl\n!pip install ../input/coleridge-packages/tokenizers-0.10.1-cp37-cp37m-manylinux1_x86_64.whl\n!pip install ../input/coleridge-packages/transformers-4.5.0.dev0-py3-none-any.whl\n\n\nfrom IPython.display import clear_output\nclear_output()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-10-06T12:13:09.900622Z","iopub.execute_input":"2021-10-06T12:13:09.901004Z","iopub.status.idle":"2021-10-06T12:14:37.032480Z","shell.execute_reply.started":"2021-10-06T12:13:09.900925Z","shell.execute_reply":"2021-10-06T12:14:37.031566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Importing Libraries:","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport json\nimport os \nimport re\nimport string\nimport random\nimport statistics\nimport Levenshtein\nimport math\n\n\n# Data Visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set_style(\"whitegrid\")\n\nfrom wordcloud import WordCloud\nfrom collections import Counter\n\n#Text Color\nfrom termcolor import colored\n\n#NLP\nimport spacy\n\nfrom tqdm.auto import tqdm\n\nimport pathlib\nimport glob\n\n\nfrom datasets import load_dataset\nimport torch\nfrom transformers import AutoTokenizer, DataCollatorForLanguageModeling, \\\nAutoModelForMaskedLM, Trainer, TrainingArguments, pipeline\n\nfrom typing import List\nimport string\nfrom functools import partial\n#from statistics import *\nfrom statistics import mean, median, mode, stdev\nfrom sklearn.metrics import accuracy_score, classification_report\n\n\n\n\nimport warnings\nwarnings.filterwarnings('ignore')\n%matplotlib inline\n\nplt.rcParams['figure.figsize']=(8,6)","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:37.035753Z","iopub.execute_input":"2021-10-06T12:14:37.036030Z","iopub.status.idle":"2021-10-06T12:14:46.009521Z","shell.execute_reply.started":"2021-10-06T12:14:37.035993Z","shell.execute_reply":"2021-10-06T12:14:46.008764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def SeedEverything(seed: int):\n\n random.seed(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed_all(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\n print(f'Setted Pipeline SEED = {SEED}')\n\nSEED=2021\nSeedEverything(SEED)","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.010873Z","iopub.execute_input":"2021-10-06T12:14:46.011230Z","iopub.status.idle":"2021-10-06T12:14:46.022107Z","shell.execute_reply.started":"2021-10-06T12:14:46.011188Z","shell.execute_reply":"2021-10-06T12:14:46.021298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##

Loading the data ⌛

","metadata":{}},{"cell_type":"markdown","source":"* `train.csv` : Labels and metadata for the training set from scientific publications in the train folder ;\n* `train` - the full text of the training set's publications in JSON format, broken into sections with section titles\n* `test` - the full text of the test set's publications in JSON format, broken into sections with section titles\n* The `sample_subimission.csv` : a sample submission file in the correct format.","metadata":{}},{"cell_type":"markdown","source":"### 1 CSV files :","metadata":{}},{"cell_type":"code","source":"DATA_PATH = pathlib.Path('../input/coleridgeinitiative-show-us-the-data')","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.023628Z","iopub.execute_input":"2021-10-06T12:14:46.024137Z","iopub.status.idle":"2021-10-06T12:14:46.062463Z","shell.execute_reply.started":"2021-10-06T12:14:46.024097Z","shell.execute_reply":"2021-10-06T12:14:46.061477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Columns Description :**\n\n* `id `- publication id - note that there are multiple rows for some training documents, indicating multiple mentioned datasets\n* `pub_title` - title of the publication (a small number of publications have the same title)\n* `dataset_title` - the title of the dataset that is mentioned within the publication\n* `dataset_label` - a portion of the text that indicates the dataset\n* `cleaned_label` - the dataset_label, as passed through the clean_text function from the Evaluation page\n* `PredictionString`- To be filled with equivalent of cleaned_label of train data (just in sample submission).","metadata":{}},{"cell_type":"code","source":"MAX_SAMPLE = None\n\ntrain_path = '../input/coleridgeinitiative-show-us-the-data/train.csv'\npaper_train_folder = '../input/coleridgeinitiative-show-us-the-data/train/'\n\ntrain = pd.read_csv(train_path)\nprint (train.shape)\ntrain = train[:MAX_SAMPLE]\n#print (train.head())\nprint (\"=====================\")\n# Group by publication, training labels should have the same form as expected output.\ndf = train.groupby('Id').agg({\n 'pub_title': 'first',\n 'dataset_title': '|'.join,\n 'dataset_label': '|'.join,\n 'cleaned_label': '|'.join\n}).reset_index() \n#print (train.query(\"dataset_title.str.contains('|')\"))\n# print(train['dataset_title'][138])\n# print(train['dataset_label'][138])\n# print(train['cleaned_label'][138])\nprint (\"==================================\")\nprint('train size: ', len(df))\nprint (\"==================================\")\nprint (df.head())","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.066903Z","iopub.execute_input":"2021-10-06T12:14:46.067218Z","iopub.status.idle":"2021-10-06T12:14:46.544554Z","shell.execute_reply.started":"2021-10-06T12:14:46.067191Z","shell.execute_reply":"2021-10-06T12:14:46.543849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#reading train.csv\n#train=pd.read_csv(DATA_PATH /'train.csv')\nprint (df.shape)\ndf.sample(5)\n","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.548147Z","iopub.execute_input":"2021-10-06T12:14:46.548411Z","iopub.status.idle":"2021-10-06T12:14:46.567181Z","shell.execute_reply.started":"2021-10-06T12:14:46.548385Z","shell.execute_reply":"2021-10-06T12:14:46.566350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Let's take a look to the `Sample_submission.csv `:","metadata":{}},{"cell_type":"code","source":"output= df.tail(3000)\n#output=pd.read_csv(DATA_PATH /'sample_submission.csv') # output file\noutput.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.568790Z","iopub.execute_input":"2021-10-06T12:14:46.569157Z","iopub.status.idle":"2021-10-06T12:14:46.581257Z","shell.execute_reply.started":"2021-10-06T12:14:46.569120Z","shell.execute_reply":"2021-10-06T12:14:46.580210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2 Basic Analysis :","metadata":{}},{"cell_type":"markdown","source":"* Training set shape :","metadata":{}},{"cell_type":"code","source":"print('Dimension of the training Dataset : {}'.format(colored(train.shape,'blue')))","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.582704Z","iopub.execute_input":"2021-10-06T12:14:46.583336Z","iopub.status.idle":"2021-10-06T12:14:46.589470Z","shell.execute_reply.started":"2021-10-06T12:14:46.583297Z","shell.execute_reply":"2021-10-06T12:14:46.588533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Data Description :","metadata":{}},{"cell_type":"code","source":"output.info()","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.590978Z","iopub.execute_input":"2021-10-06T12:14:46.591584Z","iopub.status.idle":"2021-10-06T12:14:46.605891Z","shell.execute_reply.started":"2021-10-06T12:14:46.591537Z","shell.execute_reply":"2021-10-06T12:14:46.604759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output.isnull().sum().to_frame('NaN Values')","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.607268Z","iopub.execute_input":"2021-10-06T12:14:46.607600Z","iopub.status.idle":"2021-10-06T12:14:46.618298Z","shell.execute_reply.started":"2021-10-06T12:14:46.607566Z","shell.execute_reply":"2021-10-06T12:14:46.617376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We don't have any duplicated value on the train set","metadata":{}},{"cell_type":"code","source":"print('All rows :',colored(train.shape[0],'red'))\nfor column in train.columns:\n print(\"{} : {}\".format(column,colored(len(train[column].unique()),'blue')))","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.619762Z","iopub.execute_input":"2021-10-06T12:14:46.620339Z","iopub.status.idle":"2021-10-06T12:14:46.643858Z","shell.execute_reply.started":"2021-10-06T12:14:46.620303Z","shell.execute_reply":"2021-10-06T12:14:46.642720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* we have 14316 unique `Publication Id` that's mean there's some publication mentioning more than one data set.\n* for the `publication title` there's less unique values than `Publication Id` So there's diffirent publication with same title.\n* 45 `Dataset title` and 130 `Dataset Label` that's mean there some dataset with multiple labels","metadata":{}},{"cell_type":"markdown","source":"### 3 Reading JSON format :","metadata":{}},{"cell_type":"markdown","source":"The publications that we will use in train and test are provided in JSON format, broken up into sections with section titles.","metadata":{}},{"cell_type":"code","source":"#trainFilesPath =DATA_PATH /'train' #we will use this only, last 3000 files\n#testFilesPath = DATA_PATH /'test'\ntestFilesPath = DATA_PATH /'train'","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.645345Z","iopub.execute_input":"2021-10-06T12:14:46.645742Z","iopub.status.idle":"2021-10-06T12:14:46.649866Z","shell.execute_reply.started":"2021-10-06T12:14:46.645704Z","shell.execute_reply":"2021-10-06T12:14:46.648904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"def ReadJsonFiles(fileName, InputPath):\n \"\"\"\n This Function get the Publication text from Json file without Section titles\n \"\"\"\n \n JsonPATH = os.path.join(InputPath, (fileName+'.json'))\n \n publicationSections = []\n \n with open(JsonPATH, 'r') as file:\n json_decode = json.load(file)\n for data in json_decode:\n publicationSections.append(data.get('text'))\n \n publicationText = ' '.join(publicationSections)\n \n return publicationText","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.651498Z","iopub.execute_input":"2021-10-06T12:14:46.652128Z","iopub.status.idle":"2021-10-06T12:14:46.659423Z","shell.execute_reply.started":"2021-10-06T12:14:46.652091Z","shell.execute_reply":"2021-10-06T12:14:46.658427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's apply `ReadJsonFile` to train and Submission Set (Kaggle Test Set):","metadata":{}},{"cell_type":"code","source":"#Extract text from json file and plus its column to train and output csv file:\ntqdm.pandas()\n#train['publicationText']=train['Id'].progress_apply(lambda x:ReadJsonFiles(x,trainFilesPath))\n#output['publicationText']=output['Id'].progress_apply(lambda x:ReadJsonFiles(x,testFilesPath))\noutput['publicationText']=output['Id'].progress_apply(lambda x:ReadJsonFiles(x,testFilesPath))","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:14:46.661143Z","iopub.execute_input":"2021-10-06T12:14:46.661526Z","iopub.status.idle":"2021-10-06T12:15:04.704853Z","shell.execute_reply.started":"2021-10-06T12:14:46.661492Z","shell.execute_reply":"2021-10-06T12:15:04.703814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#train.sample(5)","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:04.706369Z","iopub.execute_input":"2021-10-06T12:15:04.706717Z","iopub.status.idle":"2021-10-06T12:15:04.710859Z","shell.execute_reply.started":"2021-10-06T12:15:04.706679Z","shell.execute_reply":"2021-10-06T12:15:04.709882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output.info","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:04.712869Z","iopub.execute_input":"2021-10-06T12:15:04.713383Z","iopub.status.idle":"2021-10-06T12:15:04.731663Z","shell.execute_reply.started":"2021-10-06T12:15:04.713345Z","shell.execute_reply":"2021-10-06T12:15:04.730572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##

Data Pre-Processing 🔧

","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:45:36.034275Z","iopub.execute_input":"2021-07-07T13:45:36.034683Z","iopub.status.idle":"2021-07-07T13:45:36.040544Z","shell.execute_reply.started":"2021-07-07T13:45:36.034596Z","shell.execute_reply":"2021-07-07T13:45:36.039327Z"}}},{"cell_type":"markdown","source":"Let's do some data cleaning\n\n`TextCleaning` function will help us to convert all text to lower case, remove special charecters, emojis and multiple spaces","metadata":{}},{"cell_type":"code","source":"def TextCleaning(text):\n \n \n text = ''.join([k for k in text if k not in string.punctuation])#Delete punctuation\n text = re.sub('[^A-Za-z0-9]+', ' ', str(text).lower()).strip() #Convert all text to lower case\n text = re.sub(' +', ' ', text)\n\n \n emoji_pattern = re.compile(\"[\"\n u\"\\U0001F600-\\U0001F64F\" # emoticons\n u\"\\U0001F300-\\U0001F5FF\" # symbols & pictographs\n u\"\\U0001F680-\\U0001F6FF\" # transport & map symbols\n u\"\\U0001F1E0-\\U0001F1FF\" # flags (iOS)\n \"]+\", flags=re.UNICODE)\n text = emoji_pattern.sub(r'', text)\n return text","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:04.733409Z","iopub.execute_input":"2021-10-06T12:15:04.733786Z","iopub.status.idle":"2021-10-06T12:15:04.740702Z","shell.execute_reply.started":"2021-10-06T12:15:04.733750Z","shell.execute_reply":"2021-10-06T12:15:04.739867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Example :\nTextCleaning('Hello World 😀')","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:04.742460Z","iopub.execute_input":"2021-10-06T12:15:04.742924Z","iopub.status.idle":"2021-10-06T12:15:04.751877Z","shell.execute_reply.started":"2021-10-06T12:15:04.742858Z","shell.execute_reply":"2021-10-06T12:15:04.750827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tqdm.pandas()\n#train['publicationText']=train['publicationText'].progress_apply(lambda x:TextCleaning(x))\noutput['publicationText']=output['publicationText'].progress_apply(lambda x: TextCleaning(x))","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:04.754066Z","iopub.execute_input":"2021-10-06T12:15:04.754421Z","iopub.status.idle":"2021-10-06T12:15:44.857834Z","shell.execute_reply.started":"2021-10-06T12:15:04.754386Z","shell.execute_reply":"2021-10-06T12:15:44.856992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:44.859271Z","iopub.execute_input":"2021-10-06T12:15:44.859826Z","iopub.status.idle":"2021-10-06T12:15:44.874090Z","shell.execute_reply.started":"2021-10-06T12:15:44.859787Z","shell.execute_reply":"2021-10-06T12:15:44.873204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"papers = {}\nfor paper_id in output['Id']:\n with open(f'{testFilesPath}/{paper_id}.json', 'r') as f:\n paper = json.load(f)\n papers[paper_id] = paper","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:44.875836Z","iopub.execute_input":"2021-10-06T12:15:44.876287Z","iopub.status.idle":"2021-10-06T12:15:46.517477Z","shell.execute_reply.started":"2021-10-06T12:15:44.876247Z","shell.execute_reply":"2021-10-06T12:15:46.516660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(papers)","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:46.519658Z","iopub.execute_input":"2021-10-06T12:15:46.519925Z","iopub.status.idle":"2021-10-06T12:15:46.529595Z","shell.execute_reply.started":"2021-10-06T12:15:46.519899Z","shell.execute_reply":"2021-10-06T12:15:46.528141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#papers","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:46.531342Z","iopub.execute_input":"2021-10-06T12:15:46.531719Z","iopub.status.idle":"2021-10-06T12:15:46.535924Z","shell.execute_reply.started":"2021-10-06T12:15:46.531682Z","shell.execute_reply":"2021-10-06T12:15:46.534962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##

Matching 📑

","metadata":{}},{"cell_type":"code","source":"","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##

Masked Modling Language 🤗

","metadata":{}},{"cell_type":"markdown","source":"## Load model and tokenizer","metadata":{}},{"cell_type":"code","source":"#PRETRAINED_PATH = '../input/coleridge-mlm-model/output-mlm/checkpoint-48000'\n#TOKENIZER_PATH = '../input/coleridge-mlm-model/model_tokenizer'\nPRETRAINED_PATH = '../input/dataset1/thesis-model/checkpoint-72000'\nTOKENIZER_PATH = '../input/dataset1/abdd-model_tokenizer'\n\n\nMAX_LENGTH = 64\nOVERLAP = 20\n\nPREDICT_BATCH = 32\n\nDATASET_SYMBOL = '$' # this symbol represents a dataset name\nNONDATA_SYMBOL = '#' # this symbol represents a non-dataset name","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:46.537497Z","iopub.execute_input":"2021-10-06T12:15:46.538128Z","iopub.status.idle":"2021-10-06T12:15:46.543643Z","shell.execute_reply.started":"2021-10-06T12:15:46.538090Z","shell.execute_reply":"2021-10-06T12:15:46.542701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH, use_fast=True)\nmodel = AutoModelForMaskedLM.from_pretrained(PRETRAINED_PATH)\n\nmlm = pipeline(\n 'fill-mask', \n model=model,\n tokenizer=tokenizer,\n device=0 if torch.cuda.is_available() else -1\n)","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:15:46.545488Z","iopub.execute_input":"2021-10-06T12:15:46.545860Z","iopub.status.idle":"2021-10-06T12:16:00.700892Z","shell.execute_reply.started":"2021-10-06T12:15:46.545822Z","shell.execute_reply":"2021-10-06T12:16:00.699987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def clean_text(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()\n\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n\ndef clean_paper_sentence(s):\n \"\"\"\n This function is essentially clean_text without lowercasing.\n \"\"\"\n s = re.sub('[^A-Za-z0-9]+', ' ', str(s)).strip()\n s = re.sub(' +', ' ', s)\n return s\n\ndef shorten_sentences(sentences):\n \"\"\"\n Sentences that have more than MAX_LENGTH words will be split\n into multiple sentences with overlappings.\n \"\"\"\n short_sentences = []\n for sentence in sentences:\n words = sentence.split()\n if len(words) > MAX_LENGTH:\n for p in range(0, len(words), MAX_LENGTH - OVERLAP):\n short_sentences.append(' '.join(words[p:p+MAX_LENGTH]))\n else:\n short_sentences.append(sentence)\n return short_sentences\n\nconnection_tokens = {'s', 'of', 'and', 'in', 'on', 'for', 'data', 'dataset'}\ndef find_mask_candidates(sentence):\n \"\"\"\n Extract masking candidates for Masked Dataset Modeling from a given $sentence.\n A candidate should be a continuous sequence of at least 2 words, \n each of these words either has the first letter in uppercase or is one of\n the connection words ($connection_tokens). Furthermore, the connection \n tokens are not allowed to appear at the beginning and the end of the\n sequence.\n \"\"\"\n def candidate_qualified(words):\n while len(words) and words[0].lower() in connection_tokens:\n words = words[1:]\n while len(words) and words[-1].lower() in connection_tokens:\n words = words[:-1]\n \n return len(words) >= 2\n \n candidates = []\n \n phrase_start, phrase_end = -1, -1\n for id in range(1, len(sentence)):\n word = sentence[id]\n if word[0].isupper() or word in connection_tokens:\n if phrase_start == -1:\n phrase_start = phrase_end = id\n else:\n phrase_end = id\n else:\n if phrase_start != -1:\n if candidate_qualified(sentence[phrase_start:phrase_end+1]):\n candidates.append((phrase_start, phrase_end))\n phrase_start = phrase_end = -1\n \n if phrase_start != -1:\n if candidate_qualified(sentence[phrase_start:phrase_end+1]):\n candidates.append((phrase_start, phrase_end))\n \n return candidates","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:16:00.702330Z","iopub.execute_input":"2021-10-06T12:16:00.702660Z","iopub.status.idle":"2021-10-06T12:16:00.718164Z","shell.execute_reply.started":"2021-10-06T12:16:00.702625Z","shell.execute_reply":"2021-10-06T12:16:00.716156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Transform :","metadata":{}},{"cell_type":"code","source":"mask = mlm.tokenizer.mask_token","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:16:00.719521Z","iopub.execute_input":"2021-10-06T12:16:00.719995Z","iopub.status.idle":"2021-10-06T12:16:00.738526Z","shell.execute_reply.started":"2021-10-06T12:16:00.719949Z","shell.execute_reply":"2021-10-06T12:16:00.737535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:16:00.739800Z","iopub.execute_input":"2021-10-06T12:16:00.740248Z","iopub.status.idle":"2021-10-06T12:16:00.761315Z","shell.execute_reply.started":"2021-10-06T12:16:00.740213Z","shell.execute_reply":"2021-10-06T12:16:00.760501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_test_data = []\n\n\nfor paper_id in output['Id']:\n #print (paper_id)\n #load paper\n paper = papers[paper_id]\n \n # extract sentences\n sentences = set([clean_paper_sentence(sentence) for section in paper \n for sentence in section['text'].split('.')\n ])\n sentences = shorten_sentences(sentences) # make sentences short\n sentences = [sentence for sentence in sentences if len(sentence) > 10] # only accept sentences with length > 10 chars\n sentences = [sentence for sentence in sentences if any(word in sentence.lower() for word in ['data', 'study'])]\n sentences = [sentence.split() for sentence in sentences] # sentence = list of words\n \n \n # mask\n test_data = []\n for sentence in sentences:\n for phrase_start, phrase_end in find_mask_candidates(sentence):\n dt_point = sentence[:phrase_start] + [mask] + sentence[phrase_end+1:]\n test_data.append((' '.join(dt_point), ' '.join(sentence[phrase_start:phrase_end+1]))) # (masked text, phrase)\n \n all_test_data.append(test_data)\n \n#print (len(all_test_data))\n#print(all_test_data[1])\n ","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:16:00.762512Z","iopub.execute_input":"2021-10-06T12:16:00.763049Z","iopub.status.idle":"2021-10-06T12:16:27.606982Z","shell.execute_reply.started":"2021-10-06T12:16:00.762998Z","shell.execute_reply":"2021-10-06T12:16:27.606209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Predict :","metadata":{}},{"cell_type":"code","source":"pred_labels = []\n\npbar = tqdm(total = len(all_test_data))\nfor test_data in all_test_data:\n \n pred_bag = set()\n \n if len(test_data):\n texts, phrases = list(zip(*test_data))\n #print (texts, \"+++++++++++++++++++++++++++++++++++++++++++++++\",phrases)\n #print (phrases, \"Phrases\")\n mlm_pred = []\n \n for p_id in range(0, len(texts), PREDICT_BATCH):\n print (p_id)\n batch_texts = texts[p_id:p_id+PREDICT_BATCH]\n \n batch_pred = mlm(list(batch_texts), targets=[f' {DATASET_SYMBOL}', f' {NONDATA_SYMBOL}'])\n #print (batch_pred) # important\n if len(batch_texts) == 1:\n batch_pred = [batch_pred]\n \n mlm_pred.extend(batch_pred)\n \n for (result1, result2), phrase in zip(mlm_pred, phrases):\n print (\"*************************************************\")\n print (result1['score'], result2['score'])\n print (result1['token_str'], result2['token_str'])\n print (phrase, \"Phrase\")\n print (\"*************************************************\")\n if (result1['score'] > result2['score']*1.5 and result1['token_str'] == DATASET_SYMBOL) or\\\n (result2['score'] > result1['score']*1.5 and result2['token_str'] == NONDATA_SYMBOL):\n pred_bag.add(clean_text(phrase))\n \n # filter labels by jaccard score \n filtered_labels = []\n #print (pred_bag)\n \n for label in sorted(pred_bag, key=len, reverse=True):\n if len(filtered_labels) == 0 or all(jaccard_similarity(label, got_label) < 0.75 for got_label in filtered_labels):\n filtered_labels.append(label)\n \n pred_labels.append('|'.join(filtered_labels))\n pbar.update(1)\n #print (pred_labels)\n #print (\"========================================================\")","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:16:27.608461Z","iopub.execute_input":"2021-10-06T12:16:27.608792Z","iopub.status.idle":"2021-10-06T12:25:01.546818Z","shell.execute_reply.started":"2021-10-06T12:16:27.608756Z","shell.execute_reply":"2021-10-06T12:25:01.545925Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_labels","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:25:01.550907Z","iopub.execute_input":"2021-10-06T12:25:01.551264Z","iopub.status.idle":"2021-10-06T12:25:01.599437Z","shell.execute_reply.started":"2021-10-06T12:25:01.551229Z","shell.execute_reply":"2021-10-06T12:25:01.598560Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_predictions=[]\n#for literal_match, mlm_pred in zip(literal_preds, pred_labels):\n #if literal_match!='' and mlm_pred not in literal_match:\n # final_predictions.append(literal_match +'|'+mlm_pred)\n #else:\n # if literal_match:\n # final_predictions.append(literal_match)\n # else :\n # final_predictions.append(mlm_pred)\n \n\n \n#final_predictions\n \n#output['PredictionString'] = final_predictions\noutput['PredictionString'] = pred_labels\noutput_file=output[['Id','cleaned_label','PredictionString']]\n#output_file.to_csv('OutputFile.csv', index=False)\n#print (output_file)","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:25:01.606883Z","iopub.execute_input":"2021-10-06T12:25:01.610166Z","iopub.status.idle":"2021-10-06T12:25:01.621940Z","shell.execute_reply.started":"2021-10-06T12:25:01.610126Z","shell.execute_reply":"2021-10-06T12:25:01.620817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#print (output.cleaned_label)\nprint(output)","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:25:01.626865Z","iopub.execute_input":"2021-10-06T12:25:01.628967Z","iopub.status.idle":"2021-10-06T12:25:01.647831Z","shell.execute_reply.started":"2021-10-06T12:25:01.628927Z","shell.execute_reply":"2021-10-06T12:25:01.647067Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## EVALUATION METRICES\n\n#1. Jaccard score\ndef jaccard(str1, str2): \n a = set(str1.lower()) \n b = set(str2.lower())\n c = a.intersection(b)\n return round(float(len(c)) / (len(a) + len(b) - len(c)), 2) \n\n\n\n#2. Levenshtein Distance\ndef levenshtein_distance(string1, string2):\n\n # the Levenshtein distance between string1 and string2\n l_dist = Levenshtein.distance(string1, string2)\n \n return l_dist\n\n\n#3. Cosine Similarity\n\nWORD = re.compile(r\"\\w+\")\n\n\ndef get_cosine(vec1, vec2):\n intersection = set(vec1.keys()) & set(vec2.keys())\n numerator = sum([vec1[x] * vec2[x] for x in intersection])\n\n sum1 = sum([vec1[x] ** 2 for x in list(vec1.keys())])\n sum2 = sum([vec2[x] ** 2 for x in list(vec2.keys())])\n denominator = math.sqrt(sum1) * math.sqrt(sum2)\n\n if not denominator:\n return 0.0\n else:\n return float(numerator) / denominator\n\n\ndef text_to_vector(text):\n words = WORD.findall(text)\n return Counter(words)\n\n\ndef cos_sim_driver_function(string1, string2):\n\n vector1 = text_to_vector(string1)\n vector2 = text_to_vector(string2)\n\n cosine = get_cosine(vector1, vector2)\n\n return round(cosine, 2)","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:25:01.651781Z","iopub.execute_input":"2021-10-06T12:25:01.654043Z","iopub.status.idle":"2021-10-06T12:25:01.669671Z","shell.execute_reply.started":"2021-10-06T12:25:01.653993Z","shell.execute_reply":"2021-10-06T12:25:01.668812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output_file['jaccard_score'] = output_file.apply(lambda x: jaccard(x['cleaned_label'], x['PredictionString']),axis=1)\noutput_file['levenshtein_distance'] = output_file.apply(lambda x: levenshtein_distance(x['cleaned_label'], x['PredictionString']),axis=1)\noutput_file['cosine_similarity'] = output_file.apply(lambda x: cos_sim_driver_function(x['cleaned_label'], x['PredictionString']),axis=1)\n \n ","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:25:01.673879Z","iopub.execute_input":"2021-10-06T12:25:01.676497Z","iopub.status.idle":"2021-10-06T12:25:01.978538Z","shell.execute_reply.started":"2021-10-06T12:25:01.676458Z","shell.execute_reply":"2021-10-06T12:25:01.977571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#output_file[['Id', 'pub_title', 'dataset_title', 'dataset_label', 'cleaned_label',\n # 'PredictionString', 'jaccard_score',\n # 'hamming_distance', 'levenshtein_distance', 'cosine_similarity']]","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:25:01.980731Z","iopub.execute_input":"2021-10-06T12:25:01.981092Z","iopub.status.idle":"2021-10-06T12:25:01.985812Z","shell.execute_reply.started":"2021-10-06T12:25:01.981057Z","shell.execute_reply":"2021-10-06T12:25:01.984070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Mean jaccard_score:\",statistics.mean(output_file['jaccard_score']))\nprint(\"Mean levenshtein_distance:\", statistics.mean(output_file['levenshtein_distance']))\nprint(\"Mean cosine_similarity:\", statistics.mean(output_file['cosine_similarity']))\n","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:25:01.987659Z","iopub.execute_input":"2021-10-06T12:25:01.988109Z","iopub.status.idle":"2021-10-06T12:25:02.006004Z","shell.execute_reply.started":"2021-10-06T12:25:01.988071Z","shell.execute_reply":"2021-10-06T12:25:02.005145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nprint(output_file[10:15])","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:25:02.007427Z","iopub.execute_input":"2021-10-06T12:25:02.008006Z","iopub.status.idle":"2021-10-06T12:25:02.019219Z","shell.execute_reply.started":"2021-10-06T12:25:02.007957Z","shell.execute_reply":"2021-10-06T12:25:02.017978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#sample_submission[\"PredictionString\"][1]\n\n#'common core of data|nces common core of data|trends in international mathematics and science study|schools and staffing survey|integrated postsecondary education data system|ipeds|progress in international reading literacy study'","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:25:02.020695Z","iopub.execute_input":"2021-10-06T12:25:02.021085Z","iopub.status.idle":"2021-10-06T12:25:02.025161Z","shell.execute_reply.started":"2021-10-06T12:25:02.021044Z","shell.execute_reply":"2021-10-06T12:25:02.023876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#y_eval= output_file['cleaned_label'].values\n#predict_y= output_file['PredictionString'].values\n#print(classification_report(y_eval, predict_y))","metadata":{"execution":{"iopub.status.busy":"2021-10-06T12:25:02.027154Z","iopub.execute_input":"2021-10-06T12:25:02.027431Z","iopub.status.idle":"2021-10-06T12:25:02.036070Z","shell.execute_reply.started":"2021-10-06T12:25:02.027390Z","shell.execute_reply":"2021-10-06T12:25:02.035279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}