{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About Coleridge Initiative\n\nThe Coleridge Initiative is a not-for-profit organization originally established at New York University. It was set up in order to inform the decision-making of the Commission on Evidence-based Policymaking and has since worked with dozens of government agencies at the federal, state, and local levels to ensure that data are more effectively used for public decision-making.\n\nIt achieves this goal by working with the agencies to create value for the taxpayer from the careful use of data by building new technologies to enable secure access to and sharing of confidential microdata and by training agency staff to acquire modern data skills."},{"metadata":{},"cell_type":"markdown","source":"# Competition Challenge\n\nThe objective of the competition is to identify the mention of datasets within scientific publications. Your predictions will be short excerpts from the publications that appear to note a dataset. Predictions that more accurately match the precise words used to identify the dataset within the publication will score higher."},{"metadata":{},"cell_type":"markdown","source":"# Evaluation Metric\n\nSubmissions are evaluated on a Jaccard-based FBeta score between predicted texts and ground truth texts, with Beta = 0 (an F0 or precision score). Multiple predictions are delineated with a pipe (|) character in the submission file."},{"metadata":{},"cell_type":"markdown","source":"# Code Requirements\n\n- CPU Notebook <= 9 hours run-time\n- GPU Notebook <= 9 hours run-time\n- Internet access disabled\n- Freely & publicly available external data is allowed, including pre-trained models\n- Submission file must be named submission.csv"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%matplotlib inline\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport tqdm\nfrom tqdm.auto import tqdm as tqdmp\ntqdmp.pandas()\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom IPython.display import display\n\nfrom plotly.offline import iplot\n#to link plotly to pandas\nimport cufflinks as cf\ncf.go_offline()\ncf.set_config_file(offline = False, world_readable = True)\n\nimport plotly.express as px #Plotly express\n\nplt.rcParams['figure.figsize'] = (12, 8)\nplt.rcParams['axes.titlesize'] = 16\nplt.style.use('seaborn-whitegrid')\nsns.set_palette('Set3')\n\nimport json\nimport collections\n\nimport itertools\nimport collections\nfrom collections import Counter\n\nfrom nltk.corpus import stopwords\n\nimport re\nfrom wordcloud import WordCloud\n\nimport gc\n\nimport os\nprint(os.listdir('/kaggle/input/coleridgeinitiative-show-us-the-data/'))\n\nimport warnings\nwarnings.simplefilter('ignore')\n\nfrom time import time, strftime, gmtime\nstart = time()\nimport datetime\nprint(str(datetime.datetime.now()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"base_dir = '../input/coleridgeinitiative-show-us-the-data/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(base_dir + 'train.csv')\nprint(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(base_dir + 'sample_submission.csv')\nprint(sub.shape)\nsub","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Number of train json files: {os.listdir(base_dir + 'train/').__len__()}\")\nprint(f\"Number of test json files: {os.listdir(base_dir + 'test/').__len__()}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Merge json with DF"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_files = os.listdir(base_dir + 'train/')\nprint(train_files[:5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The json files are named by the 'Id' in the train dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['Id'].values[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_json = pd.read_json(base_dir + 'train/' + train['Id'].values[0] + '.json')\nprint(train_json.shape)\ntrain_json.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__The json file contains the publication text separated by section title, we will have to join these texts (separated by sections) into a single text.__"},{"metadata":{},"cell_type":"markdown","source":"The competition organizers have already cleaned the label with the below code and expects the submitted label to be cleaned the same way."},{"metadata":{"trusted":true},"cell_type":"code","source":"def clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_json(x):\n    df = pd.read_json(base_dir + 'train/' + x + '.json')\n    text = ' '.join([' '.join(each) for each in df.astype(str).values])\n    text = clean_text(text)\n    return text","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train['text'] = train['Id'].progress_apply(lambda x: extract_json(x))\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.to_csv('./train_publication.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.describe().T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Publication Title\n__Let's explore the titles provided in the train set__"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['clean_pub_title'] = train['pub_title'].apply(clean_text)\ntrain['pub_title_len'] = train['clean_pub_title'].apply(lambda x: len(str(x)))\ntrain['pub_title_word_len'] = train['clean_pub_title'].apply(lambda x: len(str(x).split()))\ntrain.head(2)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\"Mean pub title length: {np.mean(train['pub_title_len'])}\")\ntrain['pub_title_len'].iplot(kind = 'hist', \n                            bins = 100,\n                            xTitle = 'Pub Title Length',\n                            yTitle = 'Count',\n                            title = 'Pub Title Length Distribution'\n                            )","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\"Mean pub title word length: {np.mean(train['pub_title_word_len'])}\")\ntrain['pub_title_word_len'].iplot(kind = 'hist', \n                            bins = 100,\n                            xTitle = 'Pub Title Word Length',\n                            yTitle = 'Count',\n                            title = 'Pub Title Word Length Distribution'\n                            )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__Plot Pub Title Word Cloud__"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_wordcloud(data, col, text = None):\n    stop = stopwords.words('english')\n    all_words = [word for each in data[col] for word in each.split() if word not in stop if len(word) > 1]\n    word_freq = Counter(all_words)\n\n    wordcloud = WordCloud(width = 900,\n                          height = 500,\n                          max_words = 200,\n                          max_font_size = 100,\n                          relative_scaling = 0.5,\n                          background_color = \"rgba(255, 255, 255, 0)\", \n                          mode = \"RGBA\",\n                          normalize_plurals = True).generate_from_frequencies(word_freq)\n    plt.figure(figsize = (16, 12))\n    plt.imshow(wordcloud, interpolation = 'bilinear')\n    plt.title(text, fontsize = 16)\n    plt.axis(\"off\")\n    plt.show()\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_wordcloud(train, 'clean_pub_title', 'WordCloud of Train Pub_title')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_most_frequent(df, top, col, title = None):\n    stop = stopwords.words('english')\n    all_words = [word for each in train[col] for word in each.split() if word not in stop if len(word) > 1]\n    word_freq = Counter(all_words)\n    freq = {'words': [w for w, c in word_freq.most_common(top)], 'counts': [c for w, c in word_freq.most_common(top)]}\n    fig = px.bar(freq, \n                 x = 'words', \n                 y = 'counts', \n                 title = title\n                )\n    fig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_most_frequent(train, 20, 'clean_pub_title', 'Top 20 Frequent words used in Pub_title')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- __Looks like most of the publications related to medical domain__"},{"metadata":{},"cell_type":"markdown","source":"#  Cleaned Label"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp = train[['cleaned_label']].copy()\nprint(f\"There are {train['cleaned_label'].nunique()} unique cleaned labels provided in the train set\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp['cleaned_label_len'] = temp['cleaned_label'].apply(lambda x: len(str(x)))\nprint(f\"Mean cleaned_label length: {np.mean(temp['cleaned_label_len'])}\")\ntemp['cleaned_label_len'].iplot(kind = 'hist', \n                            xTitle = 'Cleaned Label Length',\n                            yTitle = 'Count',\n                            title = 'Cleaned Label Length Distribution'\n                            )","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp['cleaned_label_wordlen'] = temp['cleaned_label'].apply(lambda x: len(str(x).split()))\nprint(f\"Mean cleaned_label word length: {np.mean(temp['cleaned_label_wordlen'])}\")\ntemp['cleaned_label_wordlen'].iplot(kind = 'hist', \n                            xTitle = 'Cleaned Label Length',\n                            yTitle = 'Count',\n                            title = 'Cleaned Label Word Length Distribution'\n                            )\ndel temp\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_wordcloud(train, 'cleaned_label', 'WordCloud of Train Label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_most_frequent(train, 20, 'cleaned_label', 'Top 20 Frequent words used in Cleaned Label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['cleaned_label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset Title"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp = train[['dataset_title']].copy()\nprint(f\"There are {train['dataset_title'].nunique()} unique dataset_title provided in the train set\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"temp['dataset_title_len'] = temp['dataset_title'].apply(lambda x: len(clean_text(str(x))))\nprint(f\"Mean dataset_title length: {np.mean(temp['dataset_title_len'])}\")\ntemp['dataset_title_len'].iplot(kind = 'hist', \n                            xTitle = 'Dataset Title Length',\n                            yTitle = 'Count',\n                            title = 'Dataset Title Length Distribution'\n                            )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"temp['dataset_title_wordlen'] = temp['dataset_title'].apply(lambda x: len(clean_text(str(x)).split()))\nprint(f\"Mean dataset_title length: {np.mean(temp['dataset_title_wordlen'])}\")\ntemp['dataset_title_wordlen'].iplot(kind = 'hist', \n                            xTitle = 'Dataset Title Word Length',\n                            yTitle = 'Count',\n                            title = 'Dataset Title Word Length Distribution'\n                            )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_wordcloud(train, 'dataset_title', 'WordCloud of Train Dataset Title')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_most_frequent(train, 20, 'dataset_title', 'Top 20 Frequent words used in Dataset Title')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['dataset_title'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__Dataset Titles associated with Dataset Labels__"},{"metadata":{"trusted":true},"cell_type":"code","source":"labels, n_labels = [], []\ntitles = []\n\nfor i, title in enumerate(train['dataset_title'].unique()):\n    titles.append(title)\n    label = train['dataset_label'][train['dataset_title'] == title].unique()\n    labels.append(label)\n    n_labels.append(len(labels))\ntitles_labels = pd.DataFrame({'Dataset Title': titles, 'Dataset Label': labels, 'Num Labels': n_labels})\ntitles_labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Publicaiton Text"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train['text_len'] = train['text'].progress_apply(lambda x: len(x))\ntrain['text_wordlen'] = train['text'].progress_apply(lambda x: len(x.split()))\ntrain[['Id', 'text_len', 'text_wordlen']].head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\"Mean text length: {np.mean(train['text_len'])}\")\ntrain['text_len'].iplot(kind = 'hist', \n                            bins = 100,\n                            xTitle = 'Clean Text Length',\n                            yTitle = 'Count',\n                            title = 'Clean Text Length Distribution'\n                            )","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\"Mean text length: {np.mean(train['text_wordlen'])}\")\ntrain['text_wordlen'].iplot(kind = 'hist', \n                            bins = 100,\n                            xTitle = 'Clean Text Word Length',\n                            yTitle = 'Count',\n                            title = 'Clean Text Word Length Distribution'\n                            )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_wordcloud(train, 'text', 'WordCloud of Publication Text')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_most_frequent(train, 50, 'text', 'Top 50 Frequent words used in Cleaned Publication Text')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_test_json(x):\n    df = pd.read_json(base_dir + 'test/' + x + '.json')\n    text = ' '.join([' '.join(each) for each in df.astype(str).values])\n    text = clean_text(text)\n    return text","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['text'] = sub['Id'].progress_apply(lambda x: extract_test_json(x))\nsub.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv('./test_publication.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = []\n\ndatasets_titles = [x.lower() for x in set(train['dataset_title'].unique()).union(set(train['dataset_label'].unique()))]\n\nfor index in sub['Id']:\n    print(index)\n    pub_text = sub[sub['Id'] == index]['text'].str.cat(sep = '\\n').lower()\n    label = []\n    for d_title in datasets_titles:\n        if d_title in pub_text:\n            label.append(clean_text(d_title))\n            #print(label)\n    labels.append('|'.join(label))\n\nsub['PredictionString'] = labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['PredictionString'] = labels\nsub[['Id', 'PredictionString']].to_csv('./submission.csv', index = False)\n\nsub[['Id', 'PredictionString']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# WIP...."},{"metadata":{"trusted":true},"cell_type":"code","source":"finish = time()\nprint(strftime(\"%H:%M:%S\", gmtime(finish - start)))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}