{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"INTRODUCTION:\n\nThe Coleridge Initiative is a not-for-profit that has been established to use data for social good. One way in which the organization does this is by furthering science through publicly available research.\n\nGoal:\nIn this competition, we need to develop an algorithm using natural language processing (NLP) to automate the discovery of how scientific data is referenced in publications. \nWe have to identify the data sets that publications authors used in their work. For this we have full text of scientific publications from numerous research areas. This model will eventually enable government agencies and researchers to quickly find the information they need. The approach will be used to develop data usage scorecards to better enable agencies to show how their data are used and bring down a critical barrier to the access and use of public data.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\npd.set_option('display.max_columns', None)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n    #for filename in filenames:\n        #print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\ntrain_files_path = '../input/coleridgeinitiative-show-us-the-data/train/'\ntest_files_path = '../input/coleridgeinitiative-show-us-the-data/test/'\n\nos.listdir('/kaggle/input/coleridgeinitiative-show-us-the-data/')\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\nWe are provided with 4 main pieces of data:\n\n1. train.csv: The CSV file containing all the metadata of the publications, such as their title and the dataset they utilize.\n2. train: The directory containing the actual publications that are referenced in train.csvin JSON format.\n3. test: The directory containing the actual publications that will be used for testing purposes (thus, with no ground truth CSV file available).\n4. sample_submission.csv: The CSV file containing all the publications IDs in the test set, for which we'll have to populate the prediction column.","metadata":{}},{"cell_type":"code","source":"#load libraries\nimport pandas as pd\nimport numpy as np\nimport json\n\n\n# Data Visualization\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nsns.set_style(\"darkgrid\")\n\nimport cv2\nfrom wordcloud import WordCloud, STOPWORDS\n\n\nimport nltk\nnltk.download(['punkt', 'wordnet'])\nnltk.download('stopwords')\n  \nfrom nltk.tokenize import word_tokenize\nfrom nltk.stem import WordNetLemmatizer\nimport re\nfrom nltk.corpus import stopwords\nfrom nltk.tokenize import RegexpTokenizer\nfrom sqlalchemy import create_engine\n\nfrom sklearn.pipeline import Pipeline, FeatureUnion\nfrom sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\nfrom sklearn.metrics import classification_report\nfrom sklearn.metrics import precision_recall_fscore_support\n\n\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.multioutput import MultiOutputClassifier\nfrom sklearn.tree import DecisionTreeClassifier\n\nfrom sklearn.model_selection import train_test_split, GridSearchCV\nfrom sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.columns","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#checking for missing values\ntrain.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The 'train' dataset is free of any missing values.","metadata":{}},{"cell_type":"code","source":"#number of rows and columns in train dataset\ntrain.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# finding unique values in each columns\nfor col in train.columns:\n    print(col + \":\" + str(len(train[col].unique())))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Inference\n1. The Training Dataset has 19,661 samples but only 14,316 unique IDs in the dataset. This means that some publications include a multitude of datasets.\n2. The pub_title unique count is also less than the Id unique counts. This points to the precense of several occurences of having 2 separate publications, each with a unique ID, but sharing the exact same title.\n3. Also, there are a total of 45 unique dataset_title and 130 unique dataset_label. It means that a single dataset could have multible labels throughout different publications.","metadata":{}},{"cell_type":"code","source":"#Publication titles with more than one unique Ids\ngroup_pub_title = train.drop_duplicates(\"Id\").groupby('pub_title').count()\ngroup_pub_title[group_pub_title['Id'] >1][['Id']].head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#lets take a look at 'pub_title'== 'A quantitative examination of lightning as a predictor of peak winds in tropical cyclones':\ntrain[train['pub_title'] == \"A quantitative examination of lightning as a predictor of peak winds in tropical cyclones\"]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Duplicate Id's and dataset labels:\nid_df = train[train['Id'] == '170113f9-399c-489e-ab53-2faf5c64c5bc'].drop_duplicates('dataset_title')\nid_df[['Id', 'dataset_title']]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Inference: As we can see this \"170113f9-399c-489e-ab53-2faf5c64c5bc\" Id is mentioning multiple datasets. So, for each id in test we'll need to predict all posible datasets used.","metadata":{}},{"cell_type":"code","source":"#there are 130 unique labels for 45 unique dataset, lets take a look at distribution of these labels in dataset:\ndataset_label_distribution = train.drop_duplicates('dataset_label').groupby('dataset_title').count()[['dataset_label']].sort_values(by = 'dataset_label', ascending = False).reset_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset_label_distribution.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#lets visualize the dataset-label distribution:\n\nplt.figure(figsize=(10, 7))\nsns.barplot(y='dataset_title', x='dataset_label', data=dataset_label_distribution.head(20))\nplt.title('Dataset-labels distribution', fontsize = 20)\nplt.yticks(fontsize = 8)\nplt.ylabel('dataset_title', fontsize = 12)\nplt.xlabel('unique_dateset_label_count', fontsize = 12);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Publictation titles EDA:\n","metadata":{}},{"cell_type":"code","source":"# lets take a look at 130 unique publication titles\npd.DataFrame(train['pub_title'].unique())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wordcloud\n\nWord Cloud is a data visualization technique used for representing text data in which the size of each word indicates its frequency or importance. Significant textual data points can be highlighted using a word cloud. Word clouds are widely used for analyzing data from social network websites.\n\nWe will write a simple and intuitive function plot_wordcloud that will help us plot wordclouds with ease.","metadata":{}},{"cell_type":"code","source":"def plot_wordcloud(column, title):\n    \n    \"\"\"\n    Function to Plot Wordcloud of given dataframe column.\n    \n    params: column(string): The Column of the DataFrame for plotting.\n            title(string) : The Title of the Wordcloud.\n    \"\"\"\n    # Define stopwords\n    stopwords = set(STOPWORDS) \n    \n    # Define the Wordcloud    \n    wordcloud = WordCloud(width = 800, \n                          height = 800,\n                          background_color ='black',\n                          min_font_size = 10,\n                          stopwords = stopwords).generate(' '.join(train[column])) \n\n    # Plot the WordCloud image                        \n    plt.figure(figsize = (8, 8), facecolor = None) \n    plt.imshow(wordcloud) \n    plt.axis(\"off\") \n    plt.tight_layout(pad = 0) \n    plt.title('Wordcloud: ' + title, fontsize = 20)\n\n    plt.show() ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Most frequent words in 'pub_title'\nplot_wordcloud(column = 'pub_title', title = 'Publication Title')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Text EDA:\nLets add a text column for each row corresponding to the full text : We will create a lambda function to get the text from the JSON file and append it to the new column in 'train' dataframe.","metadata":{}},{"cell_type":"code","source":"from tqdm.autonotebook import tqdm\n#tqdm is used to show any code running with a progress bar.","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#add text to 'train' columns:\ntqdm.pandas()\ntrain['text'] = train.progress_apply(lambda x : pd.read_json(train_files_path + x['Id'] + \".json\")['text'].str.cat(sep=' '), axis = 1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#WordCloud of most frequent words in the texts\ntext = ' '.join(train['text'].sample(frac=0.3))\nwordcloud = WordCloud(background_color='black', stopwords=STOPWORDS, width=800, height=800).generate(text)\n\nbarplot_dim = (8, 8)\nax = plt.subplots(figsize=barplot_dim, facecolor='w')\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.axis(\"off\")\nplt.tight_layout(pad=0)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create a function to Preprocess the data using Basic NLP Filters:","metadata":{}},{"cell_type":"code","source":"# A text cleaning function\ndef clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tqdm.pandas()\ntrain['text'] = train['text'].progress_apply(clean_text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#WordCloud of most frequent words in the texts after cleaning : \ntext = ' '.join(train['text'].sample(frac=0.3))\nwordcloud = WordCloud(background_color='black', stopwords=STOPWORDS, width=800, height=800).generate(text)\n\nbarplot_dim = (8, 8)\nax = plt.subplots(figsize=barplot_dim, facecolor='w')\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.axis(\"off\")\nplt.tight_layout(pad=0)\nplt.title('WorldCloud: text')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# credits:\n1. https://www.kaggle.com/pashupatigupta/starter-competition-data-eda-and-modelling\n2. https://www.kaggle.com/ishandutta/coleridge-complete-eda-in-one-notebook\n3. https://www.kaggle.com/anthokalel/coleridge-complete-eda\n4. https://www.kaggle.com/harshsharma511/start-to-end-easy-understanding-eda-model\n5. https://www.kaggle.com/ahmedewida/coleridge-model-using-gensim","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}