{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\nThis competition challenges data scientists to show how publicly funded data are used to serve science and society. Evidence through data is critical if government is to address the many threats facing society, including; pandemics, climate change, Alzheimer’s disease, child hunger, increasing food production, maintaining biodiversity, and addressing many other challenges. Yet much of the information about data necessary to inform evidence and science is locked inside publications.\nCan natural language processing find the hidden-in-plain-sight data citations? Can machine learning find the link between the words used in research articles and the data referenced in the article?\n\nIn this notebook, I will try to adress this issue using NLP techniques.\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"Hi all,\n\nSpecial thanks to helper notebooks 🙏🏻:\n\n* [📝Coleridge Initiative-EDA📚& Baseline Model🎯\n](https://www.kaggle.com/prashansdixit/coleridge-initiative-eda-baseline-model)\n* [Twitter sentiment Extaction-Analysis,EDA and Model](https://www.kaggle.com/chienhsianghung/coleridge-initiative-naive-cv-local-validation)","metadata":{}},{"cell_type":"markdown","source":"# **1. First import libraries and modules**","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\nimport warnings\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)\nimport os\nimport re\nimport json\nimport glob\nimport pandas as pd\nimport numpy as np\nfrom collections import defaultdict\nfrom textblob import TextBlob\nfrom functools import partial\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport seaborn as sns\nfrom collections import Counter\nimport nltk\nimport spacy\nnlp = spacy.load('en_core_web_lg', disable=['parser', 'ner'])\nnlp.max_length = 4000000\nfrom nltk.probability import FreqDist\nfrom wordcloud import WordCloud, STOPWORDS\nfrom tqdm.autonotebook import tqdm\nimport string\ndef random_colours(number_of_colors):\n    colors = []\n    for i in range(number_of_colors):\n        colors.append(\"#\"+''.join([random.choice('0123456789ABCDEF') for j in range(6)]))\n    return colors\nos.listdir('/kaggle/input/coleridgeinitiative-show-us-the-data/')","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:30:44.133974Z","iopub.execute_input":"2021-08-11T22:30:44.134638Z","iopub.status.idle":"2021-08-11T22:30:56.153374Z","shell.execute_reply.started":"2021-08-11T22:30:44.134547Z","shell.execute_reply":"2021-08-11T22:30:56.152488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#  2. Explore Datasets","metadata":{}},{"cell_type":"markdown","source":"**Let's load the datasets**\n","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\ntrain_files_path = '../input/coleridgeinitiative-show-us-the-data/train'\ntest_files_path = '../input/coleridgeinitiative-show-us-the-data/test'\nsample_sub = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')\n","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:30:56.156616Z","iopub.execute_input":"2021-08-11T22:30:56.156875Z","iopub.status.idle":"2021-08-11T22:30:56.296412Z","shell.execute_reply.started":"2021-08-11T22:30:56.156849Z","shell.execute_reply":"2021-08-11T22:30:56.295416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at our data","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:30:56.298255Z","iopub.execute_input":"2021-08-11T22:30:56.298641Z","iopub.status.idle":"2021-08-11T22:30:56.321742Z","shell.execute_reply.started":"2021-08-11T22:30:56.298609Z","shell.execute_reply":"2021-08-11T22:30:56.320556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Data Description**\n\nid - publication id - note that there are multiple rows for some training documents, indicating multiple mentioned datasets.\n\npub_title - title of the publication (a small number of publications have the same title).\n\ndataset_title - the title of the dataset that is mentioned within the publication.\n\ndataset_label - a portion of the text that indicates the dataset.\n\ncleaned_label - the dataset_label, as passed through the clean_text function from the Evaluation page.","metadata":{}},{"cell_type":"code","source":"sample_sub.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:30:56.323319Z","iopub.execute_input":"2021-08-11T22:30:56.323619Z","iopub.status.idle":"2021-08-11T22:30:56.333376Z","shell.execute_reply.started":"2021-08-11T22:30:56.323585Z","shell.execute_reply":"2021-08-11T22:30:56.332238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Description:**\n\nId - publication id.\n\nPredictionString - To be filled with equivalent of cleaned_label of train data.","metadata":{}},{"cell_type":"code","source":"train.info()\n","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:30:56.334376Z","iopub.execute_input":"2021-08-11T22:30:56.334645Z","iopub.status.idle":"2021-08-11T22:30:56.357598Z","shell.execute_reply.started":"2021-08-11T22:30:56.334616Z","shell.execute_reply":"2021-08-11T22:30:56.356565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We don't have any nut values.\n\nLet's find unique values in each column.","metadata":{}},{"cell_type":"markdown","source":"**Contents**\n","metadata":{}},{"cell_type":"code","source":"def read_append_return(filename, train_files_path=train_files_path, output='text'):\n    #Function to read json file and then return the text data from them and append to the dataframe\n    json_path = os.path.join(train_files_path, (filename+'.json'))\n    headings = []\n    contents = []\n    combined = []\n    with open(json_path, 'r') as f:\n        json_decode = json.load(f)\n        for data in json_decode:\n            headings.append(data.get('section_title'))\n            contents.append(data.get('text'))\n            combined.append(data.get('section_title'))\n            combined.append(data.get('text'))\n    all_headings = ' '.join(headings)\n    all_contents = ' '.join(contents)\n    all_data = '. '.join(combined)\n    \n    if output == 'text':\n        return all_contents\n    elif output == 'head':\n        return all_headings\n    else:\n        return all_data","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:30:56.358701Z","iopub.execute_input":"2021-08-11T22:30:56.358987Z","iopub.status.idle":"2021-08-11T22:30:56.365651Z","shell.execute_reply.started":"2021-08-11T22:30:56.35893Z","shell.execute_reply":"2021-08-11T22:30:56.364861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas()   \n#tqdm is used to show any code running with a progress bar. \ntrain['text'] = train['Id'].progress_apply(read_append_return)\n","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:30:56.366609Z","iopub.execute_input":"2021-08-11T22:30:56.367003Z","iopub.status.idle":"2021-08-11T22:33:32.222682Z","shell.execute_reply.started":"2021-08-11T22:30:56.36697Z","shell.execute_reply":"2021-08-11T22:33:32.221684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:33:32.225807Z","iopub.execute_input":"2021-08-11T22:33:32.22637Z","iopub.status.idle":"2021-08-11T22:33:32.240336Z","shell.execute_reply.started":"2021-08-11T22:33:32.226324Z","shell.execute_reply":"2021-08-11T22:33:32.239566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we have we have our text appended in our train sample.","metadata":{}},{"cell_type":"markdown","source":"# 3. Data visualizaion","metadata":{}},{"cell_type":"code","source":"train.describe()","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:33:32.241604Z","iopub.execute_input":"2021-08-11T22:33:32.241966Z","iopub.status.idle":"2021-08-11T22:33:33.475092Z","shell.execute_reply.started":"2021-08-11T22:33:32.241939Z","shell.execute_reply":"2021-08-11T22:33:33.47414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = train.groupby('cleaned_label').count()['text'].reset_index().sort_values(by='text',ascending=False)\ntemp.style.background_gradient(cmap='Reds')","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:33:33.47625Z","iopub.execute_input":"2021-08-11T22:33:33.476511Z","iopub.status.idle":"2021-08-11T22:33:33.549994Z","shell.execute_reply.started":"2021-08-11T22:33:33.476484Z","shell.execute_reply":"2021-08-11T22:33:33.549339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Data Visualization","metadata":{}},{"cell_type":"markdown","source":"**Let's draw a Funnel-Chart for visualization**\n","metadata":{}},{"cell_type":"code","source":"fig = go.Figure(go.Funnelarea(\n    text =temp.cleaned_label,\n    values = temp.text,\n    title = {\"position\": \"top center\", \"text\": \"Funnel-Chart of cleaned_label Distribution\"}\n    ))\nfig.show()\n","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:33:33.551045Z","iopub.execute_input":"2021-08-11T22:33:33.551567Z","iopub.status.idle":"2021-08-11T22:33:33.629234Z","shell.execute_reply.started":"2021-08-11T22:33:33.551537Z","shell.execute_reply":"2021-08-11T22:33:33.628303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Data cleaning","metadata":{}},{"cell_type":"code","source":"def text_cleaning(text):\n    #Converts all text to lower case, Removes special charecters, emojis and multiple spaces text - Sentence that needs to be cleaned\n    text = ''.join([k for k in text if k not in string.punctuation])\n    text = re.sub('[^A-Za-z0-9]+', ' ', str(text).lower()).strip()    \n    return text","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:33:33.630386Z","iopub.execute_input":"2021-08-11T22:33:33.630691Z","iopub.status.idle":"2021-08-11T22:33:33.635668Z","shell.execute_reply.started":"2021-08-11T22:33:33.630664Z","shell.execute_reply":"2021-08-11T22:33:33.634493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas()\ntrain['text'] = train['text'].progress_apply(text_cleaning)","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:33:33.636868Z","iopub.execute_input":"2021-08-11T22:33:33.63711Z","iopub.status.idle":"2021-08-11T22:36:25.866847Z","shell.execute_reply.started":"2021-08-11T22:33:33.637085Z","shell.execute_reply":"2021-08-11T22:36:25.865998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Most common words**","metadata":{}},{"cell_type":"code","source":"train['temp_list'] = train['cleaned_label'].apply(lambda x:str(x).split())\ntop = Counter([item for sublist in train['temp_list'] for item in sublist])\ntemp = pd.DataFrame(top.most_common(20))\ntemp.columns = ['Common_words','count']\ntemp.style.background_gradient(cmap='Greens')","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:25.868239Z","iopub.execute_input":"2021-08-11T22:36:25.868533Z","iopub.status.idle":"2021-08-11T22:36:25.922654Z","shell.execute_reply.started":"2021-08-11T22:36:25.868492Z","shell.execute_reply":"2021-08-11T22:36:25.921585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from wordcloud import WordCloud, STOPWORDS\nstopwords = set(STOPWORDS)\n\ndef show_wordcloud(data, title = None):\n    wordcloud = WordCloud(\n        background_color='white',\n        stopwords=stopwords,\n        max_words=200,\n        max_font_size=40, \n        scale=3,\n        random_state=1 # chosen at random by flipping a coin; it was heads\n).generate(str(data))\n\n    fig = plt.figure(1, figsize=(15, 15))\n    plt.axis('off')\n    if title: \n        fig.suptitle(title, fontsize=20)\n        fig.subplots_adjust(top=2.3)\n\n    plt.imshow(wordcloud)\n    plt.show()\n\nshow_wordcloud(train['cleaned_label'])","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:25.925951Z","iopub.execute_input":"2021-08-11T22:36:25.926262Z","iopub.status.idle":"2021-08-11T22:36:26.328017Z","shell.execute_reply.started":"2021-08-11T22:36:25.926229Z","shell.execute_reply":"2021-08-11T22:36:26.327012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**common words:in cleaned_label**","metadata":{}},{"cell_type":"code","source":"fig = px.bar(temp, x=\"Common_words\", y=\"count\", title='Commmon Words in cleaned_label', orientation='v', \n             width=700, height=500,color='Common_words')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:26.329571Z","iopub.execute_input":"2021-08-11T22:36:26.330086Z","iopub.status.idle":"2021-08-11T22:36:27.650137Z","shell.execute_reply.started":"2021-08-11T22:36:26.330043Z","shell.execute_reply":"2021-08-11T22:36:27.649105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from wordcloud import WordCloud, STOPWORDS\nstopwords = set(STOPWORDS)\n\ndef show_wordcloud(data, title = None):\n    wordcloud = WordCloud(\n        background_color='white',\n        stopwords=stopwords,\n        max_words=200,\n        max_font_size=40, \n        scale=3,\n        random_state=1 # chosen at random by flipping a coin; it was heads\n).generate(str(data))\n\n    fig = plt.figure(1, figsize=(15, 15))\n    plt.axis('off')\n    if title: \n        fig.suptitle(title, fontsize=20)\n        fig.subplots_adjust(top=2.3)\n\n    plt.imshow(wordcloud)\n    plt.show()\n\nshow_wordcloud(train['text'])","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:27.651684Z","iopub.execute_input":"2021-08-11T22:36:27.652271Z","iopub.status.idle":"2021-08-11T22:36:28.113074Z","shell.execute_reply.started":"2021-08-11T22:36:27.652229Z","shell.execute_reply":"2021-08-11T22:36:28.11215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**common words: in text**\n","metadata":{}},{"cell_type":"code","source":"fig = px.bar(temp, x=\"Common_words\", y=\"count\", title='Commmon Words in text', orientation='v', \n             width=700, height=500,color='Common_words')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:28.114438Z","iopub.execute_input":"2021-08-11T22:36:28.115004Z","iopub.status.idle":"2021-08-11T22:36:28.233798Z","shell.execute_reply.started":"2021-08-11T22:36:28.114961Z","shell.execute_reply":"2021-08-11T22:36:28.232787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Baseline Model and Submission","metadata":{}},{"cell_type":"code","source":"def prepare_text(text, nlp=nlp):\n    '''\n    Returns the text after stop-word removal and lemmatization.\n    text - Sentence to be processed\n    nlp - Spacy NLP model\n    '''\n    doc = nlp(text)\n    lemma_list = [token.lemma_ for token in doc if not token.is_stop]\n    lemmatized_sentence = ' '.join(lemma_list)\n    \n    return lemmatized_sentence","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:28.235272Z","iopub.execute_input":"2021-08-11T22:36:28.235857Z","iopub.status.idle":"2021-08-11T22:36:28.241779Z","shell.execute_reply.started":"2021-08-11T22:36:28.235813Z","shell.execute_reply":"2021-08-11T22:36:28.241039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:28.243032Z","iopub.execute_input":"2021-08-11T22:36:28.243607Z","iopub.status.idle":"2021-08-11T22:36:28.260885Z","shell.execute_reply.started":"2021-08-11T22:36:28.243566Z","shell.execute_reply":"2021-08-11T22:36:28.259689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas()\nsample_sub['text'] = sample_sub['Id'].progress_apply(partial(read_append_return, train_files_path=test_files_path))","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:28.262297Z","iopub.execute_input":"2021-08-11T22:36:28.262698Z","iopub.status.idle":"2021-08-11T22:36:28.349423Z","shell.execute_reply.started":"2021-08-11T22:36:28.262666Z","shell.execute_reply":"2021-08-11T22:36:28.348388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_1 = [x.lower() for x in train['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train['cleaned_label'].unique()]\n\nexisting_labels = set(temp_1 + temp_2 + temp_3)\n\nprint(f'len(temp_1) = {len(temp_1)}')\nprint(f'len(temp_2) = {len(temp_2)}')\nprint(f'len(temp_3) = {len(temp_3)}')\nprint(f'len(existing_labels) = {len(existing_labels)}')\n\nid_list = []\nlables_list = []\nfor index, row in tqdm(sample_sub.iterrows()):\n    sample_text = row['text']\n    row_id = row['Id']\n    temp_df = train[train['text'] == text_cleaning(sample_text)]\n    cleaned_labels = temp_df['cleaned_label'].to_list()\n    \n    for known_label in existing_labels:\n        if known_label in sample_text.lower():\n            cleaned_labels.append(clean_text(known_label))\n            \n    cleaned_labels = [clean_text(x) for x in cleaned_labels]\n    cleaned_labels = set(cleaned_labels)\n    lables_list.append('|'.join(cleaned_labels))\n    id_list.append(row_id)","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:28.350664Z","iopub.execute_input":"2021-08-11T22:36:28.350933Z","iopub.status.idle":"2021-08-11T22:36:28.744896Z","shell.execute_reply.started":"2021-08-11T22:36:28.350904Z","shell.execute_reply":"2021-08-11T22:36:28.744026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame()\nsubmission['Id'] = id_list\nsubmission['PredictionString'] = lables_list","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:28.747663Z","iopub.execute_input":"2021-08-11T22:36:28.747951Z","iopub.status.idle":"2021-08-11T22:36:28.754439Z","shell.execute_reply.started":"2021-08-11T22:36:28.747923Z","shell.execute_reply":"2021-08-11T22:36:28.753076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:28.756102Z","iopub.execute_input":"2021-08-11T22:36:28.756531Z","iopub.status.idle":"2021-08-11T22:36:28.774096Z","shell.execute_reply.started":"2021-08-11T22:36:28.75649Z","shell.execute_reply":"2021-08-11T22:36:28.773218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)\n","metadata":{"execution":{"iopub.status.busy":"2021-08-11T22:36:28.775451Z","iopub.execute_input":"2021-08-11T22:36:28.775955Z","iopub.status.idle":"2021-08-11T22:36:28.789218Z","shell.execute_reply.started":"2021-08-11T22:36:28.775923Z","shell.execute_reply":"2021-08-11T22:36:28.788326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}