{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n","metadata":{}},{"cell_type":"markdown","source":"

Coleridge: Start To End

\n

In this notebook I will drive you through all the important steps required to understand and complete this competition.

","metadata":{}},{"cell_type":"markdown","source":"

First Understand What is Coleridge Initiative?

\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"

Competition Goal

\nIn this competition, we need to develop an algorithm using natural language processing (NLP) to automate the discovery of how scientific data is referenced in publications. We have to identify the data sets that publications authors used in their work. For this we have full text of scientific publications from numerous research areas.\nThis model will eventually enable government agencies and researchers to quickly find the information they need. The approach will be used to develop data usage scorecards to better enable agencies to show how their data are used and bring down a critical barrier to the access and use of public data.\n

\nIn short, we have to build a model to identify what are the datasets that a publication is using.\n\n","metadata":{}},{"cell_type":"markdown","source":"

Let's understand the DATA (EDA)

\n

\nFiles Provided\n","metadata":{}},{"cell_type":"code","source":"#Libraries\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport pandas as pd\nimport numpy as np\nimport os\nimport re\nimport json\nimport string\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom tqdm.autonotebook import tqdm\nfrom functools import partial\nfrom wordcloud import WordCloud, STOPWORDS\nimport nltk\nimport spacy\nnlp = spacy.load('en_core_web_lg', disable=['parser', 'ner'])\nnlp.max_length = 4000000\nfrom nltk.probability import FreqDist\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Train Data Exploration","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.columns","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Data Description

\n

\n","metadata":{}},{"cell_type":"code","source":"train.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we have no 'NULL' values in the train data","metadata":{}},{"cell_type":"code","source":"for col in train.columns:\n print(col + \":\" + str(len(train[col].unique())))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Inference

\n\n\n- The Training Dataset has 19,661 samples but only 14,316 unique IDs in the dataset. This means that some publications include a multitude of datasets. \n\n\n- The pub_title unique count is also less than the Id unique counts. This points to the precense of several occurences of having 2 separate publications, each with a unique ID, but sharing the exact same title.\n\n\n- Also, there are a total of 45 unique dataset_title and 130 unique dataset_label. It means that a single dataset could have multible labels throughout different publications.","metadata":{}},{"cell_type":"markdown","source":"Sample Submission","metadata":{}},{"cell_type":"code","source":"sample_sub = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')\nsample_sub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Data Description

\n

\n","metadata":{}},{"cell_type":"markdown","source":"

Data Processing

","metadata":{}},{"cell_type":"markdown","source":"Now we will create a function to get the text from the JSON file and append it to the new column in table","metadata":{}},{"cell_type":"code","source":"train_files_path = '../input/coleridgeinitiative-show-us-the-data/train'\ntest_files_path = '../input/coleridgeinitiative-show-us-the-data/test'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def json_to_text(filename, train_files_path=train_files_path, output='text'):\n json_path = os.path.join(train_files_path, (filename+'.json'))\n headings = []\n contents = []\n combined = []\n with open(json_path, 'r') as f:\n json_decode = json.load(f)\n for data in json_decode:\n headings.append(data.get('section_title'))\n contents.append(data.get('text'))\n combined.append(data.get('section_title'))\n combined.append(data.get('text'))\n \n all_headings = ' '.join(headings)\n all_contents = ' '.join(contents)\n all_data = '. '.join(combined)\n \n if output == 'text':\n return all_contents\n elif output == 'head':\n return all_headings\n else:\n return all_data","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tqdm.pandas()\ntrain['text'] = train['Id'].progress_apply(json_to_text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see the Train Data now","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now apply the function to submission Data","metadata":{}},{"cell_type":"code","source":"tqdm.pandas()\nsample_sub['text'] = sample_sub['Id'].progress_apply(partial(json_to_text, train_files_path=test_files_path))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create a function to Preprocess the data using Basic NLP Filters (all text to lower case, Removes special charecters, emojis and multiple spaces)","metadata":{}},{"cell_type":"code","source":"def text_cleaning(text):\n text = ''.join([k for k in text if k not in string.punctuation])\n text = re.sub('[^A-Za-z0-9]+', ' ', str(text).lower()).strip()\n text = re.sub(' +', ' ', text)\n emoji_pattern = re.compile(\"[\"\n u\"\\U0001F600-\\U0001F64F\" # emoticons\n u\"\\U0001F300-\\U0001F5FF\" # symbols & pictographs\n u\"\\U0001F680-\\U0001F6FF\" # transport & map symbols\n u\"\\U0001F1E0-\\U0001F1FF\" # flags (iOS)\n \"]+\", flags=re.UNICODE)\n text = emoji_pattern.sub(r'', text)\n return text","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tqdm.pandas()\ntrain['text'] = train['text'].progress_apply(text_cleaning)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

EDA with Visualization

","metadata":{}},{"cell_type":"code","source":"text = ' '.join(train['text'].sample(frac=0.3))\nwordcloud = WordCloud(background_color='white', stopwords=STOPWORDS, width=2000, height=1200).generate(text)\n\nbarplot_dim = (15, 15)\nax = plt.subplots(figsize=barplot_dim, facecolor='w')\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.title('Top 100 Most Common Words in Publications Text', fontsize=50)\nplt.axis(\"off\")\nplt.tight_layout(pad=0)\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Baseline Model

","metadata":{}},{"cell_type":"code","source":"def clean_text(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()\ntemp_1 = [x.lower() for x in train['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train['cleaned_label'].unique()]\n\nexisting_labels = set(temp_1 + temp_2 + temp_3)\nid_list = []\nlables_list = []\nfor index, row in tqdm(sample_sub.iterrows()):\n sample_text = row['text']\n row_id = row['Id']\n temp_df = train[train['text'] == text_cleaning(sample_text)]\n cleaned_labels = temp_df['cleaned_label'].to_list()\n for known_label in existing_labels:\n if known_label in sample_text.lower():\n cleaned_labels.append(clean_text(known_label))\n cleaned_labels = [clean_text(x) for x in cleaned_labels]\n cleaned_labels = set(cleaned_labels)\n lables_list.append('|'.join(cleaned_labels))\n id_list.append(row_id)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame()\nsubmission['Id'] = id_list\nsubmission['PredictionString'] = lables_list","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Currently working on BERT model for this competition for better results. Hopefully share with you soon. Thank You!!","metadata":{}},{"cell_type":"markdown","source":"

References

\n\n\nhttps://www.kaggle.com/ishandutta/coleridge-complete-eda-in-one-notebook\n\nhttps://www.kaggle.com/harshsharma511/starter-competition-data-eda-and-modelling\n\nhttps://www.kaggle.com/harshsharma511/starter-competition-data-eda-and-modelling\n\nhttps://www.kaggle.com/harshsharma511/coleridge-initiative-eda-baseline-model","metadata":{}}]}