1. Understanding the Comptetion
\n\n\n\n**Background** - The Coleridge Initiative is a not-for-profit organization, originally established at New York University, that is working with governments to ensure that data are more effectively used for public decision-making."},{"metadata":{},"cell_type":"markdown","source":"### Competition Objective\n\nOne liner - We are required to build an algorithm that can find our what are the datasets that a publications uses.\n\nDescription - In this competition, we need to develop an algorithm to automate the discovery of how scientific data are referenced in publications. We have with us the full text of scientific publications from numerous research areas, we'll identify data sets that the publications' authors used in their work.\n\nWe have a labelled dataset (train set) that we'll use to develop our algorithm. The unlabelled dataset (test set) will be used for evaluation of the algorithm. Let's look into the data to understand the data and the comptetion better.\n\nThis type of automation will be very useful in showing what datasets are used in a particular type of publications or the reverse, what are the potential usages of a datset."},{"metadata":{},"cell_type":"markdown","source":"> It's also important to understand the evaluation process of this competition because it is little different. Look into this [Evaluation Process📢(Jaccard,FBeta)](https://www.kaggle.com/pashupatigupta/ci-how-score-is-calculated-jaccard-fbeta) notebook for a detailed explanation of evaluation process."},{"metadata":{},"cell_type":"markdown","source":"##2. Understanding the Data
\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport re\nimport json\nimport pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\nfrom wordcloud import WordCloud, STOPWORDS\nimport plotly.graph_objects as go\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"List of data file provided as input"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('../input/coleridgeinitiative-show-us-the-data')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have a train.csv file. We also have train and test folders. Let's look into the folders first then we'll look into files."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Files in train directory : \\n\")\nprint(os.listdir('../input/coleridgeinitiative-show-us-the-data/train')[:5])\nprint(\"\\nFiles in test directory : \\n\")\nprint(os.listdir('../input/coleridgeinitiative-show-us-the-data/test')[:5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have some json files in both the directories. Let's look into the files to find out what are they -"},{"metadata":{"trusted":true},"cell_type":"code","source":"with open('../input/coleridgeinitiative-show-us-the-data/train/f8b03c87-9d1a-4f20-b76b-cb6c69d447b2.json') as f:\n sample = json.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample[:2]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Well, these json files are full text version of publication. Let's what are the sections in a paper -"},{"metadata":{"trusted":true},"cell_type":"code","source":"for s in sample:\n print(s['section_title'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Woah! So we have each and every detail of a publication available in a json format. We can use these details to generate some features for model building. This is about the json files.\n\nNow let's look at train.csv file that I believe has the labels information."},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\ntrain.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Columns\n\n- id (publication id) - note that there are multiple rows for some training documents, indicating multiple mentioned datasets\n- pub_title - title of the publication (a small number of publications have the same title)\n- dataset_title - the title of the dataset that is mentioned within the publication\n- dataset_label - a portion of the text that indicates the dataset\n- cleaned_label - the dataset_label, as passed through the clean_text function from the Evaluation page\n\nSo, we have 'id', 'publication_title' and 'cleaned_label' columns. The id column is same as the json filenames. So using this id column we can have any information about the publication."},{"metadata":{},"cell_type":"markdown","source":"##3. EDA and Data Prepataion
"},{"metadata":{},"cell_type":"markdown","source":"### 3.1 EDA"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def basic_eda(df, row_limit=5, list_elements_limit=10):\n ### rows and columns\n print('Info : There are {} columns in the dataset'.format(df.shape[1]))\n print('Info : There are {} rows in the dataset'.format(df.shape[0]))\n \n print(\"==================================================\")\n \n ## data types\n print(\"\\nData type information of different columns\")\n dtypes_df = pd.DataFrame(df.dtypes).reset_index().rename(columns={0:'dtype', 'index':'column_name'})\n cat_df = dtypes_df[dtypes_df['dtype']=='object']\n num_df = dtypes_df[dtypes_df['dtype']!='object']\n print('Info : There are {} categorical columns'.format(len(cat_df)))\n print('Info : There are {} numerical columns'.format(len(dtypes_df)-len(cat_df)))\n \n if list_elements_limit >= len(cat_df):\n print(\"Categorical columns : \", list(cat_df['column_name']))\n else:\n print(\"Categorical columns : \", list(cat_df['column_name'])[:list_elements_limit])\n \n if list_elements_limit >= len(num_df):\n print(\"Numerical columns : \", list(num_df['column_name']))\n else:\n print(\"Numerical columns : \", list(num_df['column_name'])[:list_elements_limit])\n \n #dtypes_df['dtype'].value_counts().plot.bar()\n display(dtypes_df.head(row_limit))\n \n# print(\"==================================================\")\n# print(\"\\nDescription of numerical variables\")\n \n# #### Describibg numerical columns\n# desc_df_num = df[list(num_df['column_name'])].describe().T.reset_index().rename(columns={'index':'column_name'})\n# display(desc_df_num.head(row_limit))\n \n print(\"==================================================\")\n print(\"\\nDescription of categorical variables\")\n \n desc_df_cat = df[list(cat_df['column_name'])].describe().T.reset_index().rename(columns={'index':'column_name'})\n display(desc_df_cat.head(row_limit))\n \n return","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"basic_eda(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Observations\n\n- 1) There are duplicate id's meaning that there are some pulications that are using mutiple datasets. That's why that id is repeating.\n- 2) Same is the case with pub_title. A single publication is using mutiple datasets.\n- 3) There is NO one to one mapping of id and pub_title. Meaning that there are cases when two different publications (from two different authors) have same title. Well, interesting!!!\n- 4) There 45 dataset titles but 130 dataet labels. Meaning that there are some datasets that has multiple labels. We'll look into how these two are related."},{"metadata":{},"cell_type":"markdown","source":"### 3.1.1. Duplicate Id's and dataset labels"},{"metadata":{"trusted":true},"cell_type":"code","source":"id_df = train[train['Id'] == '170113f9-399c-489e-ab53-2faf5c64c5bc'].drop_duplicates('dataset_title')\nid_df[['Id', 'dataset_title']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Note -: As we can see this \"170113f9-399c-489e-ab53-2faf5c64c5bc\" Id is mentioning multiple datasets. So, for each id in test we'll need to predict all posible datasets used."},{"metadata":{},"cell_type":"markdown","source":"### 3.1.2. Duplicate pub_title and dataset label"},{"metadata":{"trusted":true},"cell_type":"code","source":"pub_df = train[train['pub_title'] == 'Science and Engineering Indicators 2008'].drop_duplicates('dataset_title')\npub_df[['pub_title', 'dataset_title']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Note - As we observed in the above artifact there are publication titles using multiple datasets."},{"metadata":{},"cell_type":"markdown","source":"### 3.1.3. Multiple publications having same title\n\nThere is NO one to one mapping of id and pub_title. Meaning that there are cases when two different publications (from two different authors) have same title."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Five such example (pub_title) where case 3.1.3 happens...\\n\")\ni=0\nfor pt in train['pub_title'].unique():\n pub_df = train[train['pub_title'] == pt].drop_duplicates('Id')\n if pub_df.shape[0] > 1:\n print(pt)\n i = i+1\n if i==5:\n break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pub_df = train[train['pub_title'] == 'Characteristics and Production Costs of U.S. Hog Farms, 2004'].drop_duplicates('Id')\npub_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.1.4. Dataset titles and labels\n\nA single dataset can have multiple labels."},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_titles = train['dataset_title'].unique()\ndup_title = []\ncount = []\ndup_list = []\nfor ut in unique_titles:\n title_df = train[train['dataset_title'] == ut]\n tdf = title_df[['Id', 'dataset_title', 'dataset_label']].drop_duplicates('dataset_label')\n if tdf.shape[0] > 1:\n #print(ut)\n dup_title.append(ut)\n count.append(tdf.shape[0])\n dup_list.append(list(tdf['dataset_label']))\n \ndup_df = pd.DataFrame({'dataset_title':dup_title, 'label_count':count, 'label_list':dup_list})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dup_df.set_index('dataset_title')['label_count'].sort_values(ascending=False).plot.barh(figsize=(12,18))\nplt.title(\"No of labels that a datset have\")\nplt.xlabel('labels_count')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.2 Data Preparation\n\nWe'll read the text of a publication from the json file and put it in the train dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_text(filename, test=False):\n if test:\n df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/test/{}.json'.format(filename))\n else:\n df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/train/{}.json'.format(filename))\n text = \" \".join(list(df['text']))\n return text","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['text'] = train['Id'].apply(get_text)\ntrain.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we have the text content of each publication let's do some wordcloud analysis."},{"metadata":{},"cell_type":"markdown","source":"### 3.2.1 WordCloud of publication titles"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"words_in_titles = list(train.pub_title.str.split(expand=True).stack())\n\nwordcloud = WordCloud(stopwords = STOPWORDS,\n background_color = \"white\",\n width = 3000,\n height = 2000\n ).generate(' '.join(words_in_titles))\nplt.figure(1, figsize = (18, 12))\nplt.imshow(wordcloud)\nplt.axis('off')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.2.2 WordCloud of most frequent words in the texts"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"text = ' '.join(train['text'].sample(frac=0.3))\nwordcloud = WordCloud(background_color='white', stopwords=STOPWORDS, width=2560, height=1440).generate(text)\n\nbarplot_dim = (15, 15)\nax = plt.subplots(figsize=barplot_dim, facecolor='w')\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.axis(\"off\")\nplt.tight_layout(pad=0)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# A text cleaning function\ndef clean_text(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##4. Baseline Model
"},{"metadata":{},"cell_type":"markdown","source":"### Hypothesis building \n\nInstead of directly jumping into models like BERT, XLNet, GPT-3, let's think simple here. In any publication the authors mentions the names of the datasets that are used in their work. So by simple string matching we can find out whether a dataset is mentioned in a publication or not.\n\nSo, instead of inferering from the publication (which datasets are used) we'll be finding out if a particular dataset is used in publication or not. For this we need a list of possible datasets and we can get it from the training set. BUT this isn't what the competition demands. This is just a baseline hypothesis."},{"metadata":{},"cell_type":"markdown","source":"### 4.1 Preparing test set"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_files = os.listdir('../input/coleridgeinitiative-show-us-the-data/test')\ntest_files","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.DataFrame({'Id':test_files})\ntest['Id'] = test['Id'].apply(lambda x : x.split('.')[0])\ntest['text'] = test['Id'].apply(get_text, test=True)\ntest","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 Let's check this hypothesis on training data (To check if it's even worth to use this)"},{"metadata":{"trusted":true},"cell_type":"code","source":"is_present = []\nfor exp in train.iterrows():\n if exp[1]['cleaned_label'] in clean_text(exp[1]['text']):\n is_present.append(1)\n else:\n is_present.append(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['present'] = is_present\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check the accuracy"},{"metadata":{"trusted":true},"cell_type":"code","source":"acc = (train['present'].sum() / len(train))*100\nprint(\"Accuracy on Traininig set : {}%\".format(acc))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Superb! This hypothesis gives 100% accuracy on training set.\n\n### 4.3 Making submission file"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')\nids = submission_df['Id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"datasets_titles = [x.lower() for x in set(train['dataset_title'].unique()).union(set(train['dataset_label'].unique()))]\n\nlabels = []\nfor index in submission_df['Id']:\n publication_text = test[test['Id'] == index].text.str.cat(sep='\\n').lower()\n #print(publication_text)\n label = []\n for dataset_title in datasets_titles:\n if dataset_title in publication_text:\n label.append(clean_text(dataset_title))\n labels.append('|'.join(label))\n\nsubmission_df['PredictionString'] = labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame()\nsubmission['Id'] = ids\nsubmission['PredictionString'] = labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### We can see that string matching gives 100% accuracy on train set. On submission as well it will probably give a good score. This model can definetely serve as a baseline.\n\n#### Note - Accuracy isn't the actual evaluation metric. The actual evaluation metric is Jaccard similatity base FBeta(0.5) score. I have prepared this [Notebook](https://www.kaggle.com/pashupatigupta/ci-how-score-is-calculated-jaccard-fbeta) that implements the evaluation metric and it also evaluates the baseline on actual metric.\n\n#### If you found it useful please consider appreciating it by an UPVOTE. Thanks!"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}