{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Notebook Contents

\n\nIn this notebook I'll do some Exploratory Data Analysis, trying to update it with new sections in the next weeks.\n\nThe sections are:\n\n\n
\n

Notebook Contents

\n\n
","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\npd.options.display.max_columns = 50\npd.options.display.max_colwidth = 200\nimport os\n\n! pip install --index-url https://test.pypi.org/simple/ PyARMViz\nfrom dataclasses import dataclass\nfrom mlxtend.preprocessing import TransactionEncoder\nfrom mlxtend.frequent_patterns import apriori, association_rules\nimport colorama\nfrom colorama import Fore, Back, Style\nimport folium\nimport json\nimport geopandas as gpd\nfrom PyARMViz import PyARMViz\nfrom PyARMViz.Rule import generate_rule_from_dict\n\nimport re\nimport pyproj\nfrom pyproj import Proj, transform\n\nfrom shapely.ops import cascaded_union\nimport matplotlib\nimport matplotlib.pyplot as plt\nplt.rcParams.update({'figure.max_open_warning': 0})\nplt.style.use('fivethirtyeight')\nimport seaborn as sns # visualization\nimport warnings # Supress warnings \nwarnings.filterwarnings('ignore')\n\nimport plotly.graph_objs as go\nfrom PIL import Image\n\nfrom tqdm import tqdm\n\ny_ = Fore.YELLOW\nr_ = Fore.RED\ng_ = Fore.GREEN\nb_ = Fore.BLUE\nm_ = Fore.MAGENTA\nc_ = Fore.CYAN\nsr_ = Style.RESET_ALL\n\ndef get_df_basic_information(df, color, df_name): \n \n n_rows, n_columns = df.shape\n \n mb_size = round(df.memory_usage(deep=True).sum()/1000000., 3)\n \n print(\"\"\"{0}{1}\\n\n N rows: {2}\\tN columns: {3}\\n\n Memory Usage: {4} Mb\\n\\n\\n\"\"\".format(color, df_name,\n n_rows, n_columns, mb_size))\n\nroot_path = '/kaggle/input/coleridgeinitiative-show-us-the-data'\ntrain_path = os.path.join(root_path, 'train')\ntest_path = os.path.join(root_path, 'test')\n\n\ndef pretty_files_root(root_path, max_enum = 5):\n \n print(c_ + str(root_path))\n train_path = os.path.join(root_path, 'train')\n test_path = os.path.join(root_path, 'test')\n ident = 1\n \n print(\"\\t\"*ident, y_ + str(os.path.join(root_path, 'train.csv')))\n print(\"\\n\")\n print(\"\\t\"*ident, y_ + str(os.path.join(root_path, 'sample_submission.csv')))\n print(\"\\n\")\n print(\"\\t\"*ident, y_ + train_path)\n files_list = os.listdir(train_path)\n \n for i, file in enumerate(files_list):\n new_ident = ident+1\n if i < max_enum:\n print('\\t'*new_ident, b_ + str(file))\n \n print(\"\\n\")\n print(\"\\t\"*ident, y_ + test_path)\n files_list = os.listdir(test_path)\n \n for i, file in enumerate(files_list):\n new_ident = ident+1\n if i < max_enum:\n print('\\t'*new_ident, b_ + str(file))\n \n\nroot_path = \"/kaggle/input/coleridgeinitiative-show-us-the-data/\"\ntrain_path = os.path.join(root_path, 'train')\ntest_path = os.path.join(root_path, 'test')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

Files Structure

\n\nThe objective of the competition is to identify the mention of datasets within scientific publications. Your predictions will be short excerpts from the publications that appear to note a dataset. Predictions that more accurately match the precise words used to identify the dataset within the publication will score higher. Predictions should be cleaned using the clean_text function from the Evaluation page to ensure proper matching.\n\nPublications are provided in JSON format, broken up into sections with section titles.\n\nThe goal in this competition is not just to match known dataset strings but to generalize to datasets that have never been seen before using NLP and statistical techniques. Not all datasets have been identified in train, but you have been provided enough information to generalize.\n\nNote that the hidden test set has roughly ~8000 publications, many times the size of the public test set. Plan your compute time accordingly.\n\n**Files**\n\n**train** - the full text of the training set's publications in JSON format, broken into sections with section **titles**
\n**test** - the full text of the test set's publications in JSON format, broken into sections with section titles
\n**train.csv** - labels and metadata for the training set
\n**sample_submission.csv** - a sample submission file in the correct format
\n\n**Columns**\n\n*id* - publication id - note that there are multiple rows for some training documents, indicating multiple mentioned datasets
\n*pub_title* - title of the publication (a small number of publications have the same title)
\n*dataset_title* - the title of the dataset that is mentioned within the publication
\n*dataset_label* - a portion of the text that indicates the dataset
\n*cleaned_label* - the dataset_label, as passed through the clean_text function from the Evaluation page","metadata":{}},{"cell_type":"markdown","source":"\n\n
Visual explanation of our Data
","metadata":{}},{"cell_type":"markdown","source":"Our working directory tree (I limited train json to show just 5, there are many more of course)","metadata":{}},{"cell_type":"code","source":"pretty_files_root(root_path, 5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n","metadata":{}},{"cell_type":"markdown","source":"\n\n
Train.csv
","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(os.path.join(root_path, 'train.csv'))\nget_df_basic_information(train, y_, 'train.csv')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"{}Some sample rows from train.csv\".format(c_))\ndisplay(train.sample(3))\n\nassert train.Id.nunique() == len(os.listdir(train_path)), \"Number of Publications files does not coincide with number of ids\"\n\nprint(\"{}There are {} unique datasets in train.csv\".format(c_, train.cleaned_label.nunique()))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cmap_plot = plt.get_cmap('jet_r')\n\nexplo_df_1 = (train.groupby('Id')['cleaned_label'].nunique().reset_index()\n.cleaned_label.value_counts().rename(\"Number_of_Publications\").reset_index()\n.rename(columns = {'index': 'Datasets_in_Publication'}))\n\nprint(\"{}Number of Datasets per Publication\".format(c_))\nfig, (ax0, ax1) = plt.subplots(1, 2, figsize=(18, 6),gridspec_kw={'width_ratios': [2.5, 1]})\n\nexplo_df_1.set_index('Datasets_in_Publication')['Number_of_Publications'].plot(kind = 'bar', \n ax = ax0, \n title = 'Number of Datasets per Publication Distribution',\n width = 0.7)\n\nax0.set_xlim([0, 10])\nax0.set_ylabel('Number_of_Publications')\nfont_size=14\nbbox=[-0.2, 0, 1.2, 0.7]\nax1.axis('off')\nccolors = plt.cm.BuPu(np.full(len(explo_df_1.columns), 0.1))\nmpl_table = ax1.table(cellText = explo_df_1.values, bbox=bbox, colLabels=explo_df_1.columns, colColours=ccolors)\nmpl_table.auto_set_font_size(False)\nmpl_table.set_fontsize(font_size)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"{}Number of publications per Dataset\".format(c_))\n\nexplo_df_2 = (train.groupby('cleaned_label')['Id'].nunique().rename('Number_of_Publications')\n .reset_index()\n .sort_values('Number_of_Publications', ascending = False, ignore_index = True))\n\nfig, (ax0, ax1, ax2) = plt.subplots(1, 3, figsize=(30, 8),gridspec_kw={'width_ratios': [1.5, 0.6, 1]})\n\nax0.set_ylabel('Number_of_Datasets')\nax0.set_xlim(0, 800)\nax0.title.set_text('Number of Publications per Dataset distribution')\n\nsns.histplot(explo_df_2['Number_of_Publications'], color = 'red', ax = ax0)\n\nsns.violinplot(y = explo_df_2['Number_of_Publications'], ax=ax1, color = 'blue', orient = 'v', saturation = 0.4)\nax1.title.set_text('Violin Plot')\nax1.title.set_size(12)\n\nbbox=[-0.2, 0, 1.2, 0.9]\nax2.axis('off')\nax2.title.set_text('Top 10 datasets per number of Publications')\nax2.title.set_size(12)\n\ntop_datasets = explo_df_2.sort_values('Number_of_Publications', ascending = False).head(10).rename(columns = {'Number_of_Publications': 'Num_publications'})\n\nccolors = plt.cm.BuPu(np.full(len(top_datasets.columns), 0.1))\nmpl_table = ax2.table(cellText = top_datasets.values, bbox=bbox, colLabels=top_datasets.columns, colColours=ccolors)\nmpl_table.auto_set_font_size(False)\nmpl_table.auto_set_column_width(col=list(range(len(top_datasets.columns))))\nmpl_table.set_fontsize(14)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"{}Distribution of Number of publications per Dataset\".format(c_))\ndisplay(explo_df_2.Number_of_Publications.quantile(np.linspace(0.05, 1, 20)).to_frame().transpose().rename_axis('Quantile'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
Association Rules Mining
\n\nCheck [this](https://stackabuse.com/association-rule-mining-via-apriori-algorithm-in-python/) nice tutorial for reference. \nThere's nothing complex here, just looking for datasets appearing together in publications. \n\nI'll consider a Publication as a transaction where the items bought are the datasets. Hopefully finding patterns now may help anyone in improving their models later. \n\n\nGiven an item $A$ we define **Support of $A$**, $S(A)$ as:\n\n$S(A)$ = $\\frac{transactions \\ containing \\ A}{All \\ transactions}$;\n\nHere we can substitute *transactions* with *publications* and *item* with *dataset*. \n\nGiven $2$ items $A$ and $B$ we define **Confidence of $A$ -> $B$**, $C(A->B)$ as:\n\n$C(A->B)$ = $\\frac{transactions \\ containing \\ A \\ and \\ B}{transactions \\ containing \\ A}$;\n\nsee that it is not commutative (I may buy many eggs without ever buying bacon, but when I buy bacon I buy also eggs... Yeah, something like that).\n\nGiven $2$ items $A$ and $B$ we define **Lift of $A$ -> $B$**, $L(A->B)$ as:\n\n$L(A->B)$ = $\\frac{C(A->B)}{S(B)}$.\n\n","metadata":{}},{"cell_type":"code","source":"print(\"{}Market Basket Analysis: datasets occurring together\".format(c_))\ndef to_list(x):\n return [x]\n\npd.options.display.max_colwidth = 300\ntrain['cleaned_label_list'] = train['cleaned_label'].apply(to_list)\n\ntransactions = train.groupby('Id').cleaned_label_list.sum().rename('cleaned_label_list').reset_index()\nte = TransactionEncoder()\nte_ary = te.fit(transactions.cleaned_label_list).transform(transactions.cleaned_label_list)\n\ndf = pd.DataFrame(te_ary, columns=te.columns_)\n\nfrequent_itemsets = (apriori(df, min_support=0.001, use_colnames=True))\nrules = (association_rules(frequent_itemsets, metric='support', min_threshold=0.005))\nrules = rules.rename(columns = {'antecedents': 'dataset1', 'consequents': 'dataset2', \n 'antecedent support': 'dataset1_support', \n 'consequent support': 'dataset2_support'}).drop(['leverage', 'conviction'], axis = 1)\n\nrules['dataset1'] = rules['dataset1'].astype(str).str.replace(\"frozenset|\\{|\\}|\\(|\\)|\\'\", \"\")\nrules['dataset2'] = rules['dataset2'].astype(str).str.replace(\"frozenset|\\{|\\}|\\(|\\)|\\'\", \"\")\n\nrules = (rules.merge(explo_df_2.rename(columns = {'cleaned_label': 'dataset1', 'Number_of_Publications': 'count1'}),\n on = 'dataset1')\n .merge(explo_df_2.rename(columns = {'cleaned_label': 'dataset2', 'Number_of_Publications': 'count2'}),\n on = 'dataset2'))\n\nprint(\"{} Top 5 Confidence\".format(c_))\ndisplay(rules.sort_values('confidence', ascending = False, ignore_index = True).head(5))\nprint(\"{} Top 5 Support\".format(c_))\ndisplay(rules.sort_values('support', ascending = False, ignore_index = True).head(5))\nprint(\"{} Top 5 Lift\".format(c_))\ndisplay(rules.sort_values('lift', ascending = False, ignore_index = True).head(5))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see rules with a 1.0 confidence value which is pretty crucial: it means that when dataset1 occurs in a publication also dataset2 occurs, at least for our training data. ","metadata":{}},{"cell_type":"code","source":"table_confidence = (rules[['dataset1', 'dataset2', 'confidence']]\n .sort_values('confidence', ascending = False, ignore_index = True).head(10))\n\ntable_confidence['confidence'] = table_confidence['confidence'].round(3)\n\nbbox=[-0, 0, 1, 0.95]\n\nfig, ax = plt.subplots(1, 1, figsize = (13, 6)) \nax.axis('off')\nax.title.set_text('Top 10 dataset1->dataset2 rules per Confidence')\nax.title.set_size(12)\nccolors = plt.cm.BuPu(np.full(len(table_confidence.columns), 0.2))\n\nmpl_table = ax.table(cellText = table_confidence.values, bbox=bbox, colLabels=table_confidence.columns, colColours=ccolors,\n cellColours=[['w', 'w', 'r']]*3+[['w', 'w', 'w']]*7)\nmpl_table.auto_set_font_size(False)\nmpl_table.auto_set_column_width(col=list(range(len(table_confidence.columns))))\nmpl_table.set_fontsize(12)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hopefully in the next days I'll start with some proper text analysis","metadata":{}}]}