{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n
\n

πŸ”₯⚑ Blazing Fast EDA - Coleridge + Baseline Model ⚑πŸ”₯

\n \n\n\n
\n
","metadata":{}},{"cell_type":"markdown","source":"
\n
\n

Table of Contents

\n
\n \n
\n
0. Reference Notebooks
\n
1. Importing the required libraries
\n
2. Modin makes all the difference
\n
3. Data Exploration and Acquisition
\n
4. Data Preprocessing and Cleaning
\n
5. Data Visualization
\n
6. Baseline Model
\n
7. Submission
\n
8. Conclusion
\n \n
\n
","metadata":{}},{"cell_type":"markdown","source":"
\n
\n

0. Reference Notebooks πŸ““

\n
\n \n
\n
1. πŸ“Coleridge Initiative-EDAπŸ“š & Baseline Model🎯
\n
\n
","metadata":{}},{"cell_type":"markdown","source":"\n
\n
\n

1. Importing the required libraries πŸ“—

\n
\n
","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)\nwarnings.filterwarnings('ignore', category=UserWarning)\n\nimport os\nimport re\nimport json\nimport glob\nimport subprocess\nimport sys\nfrom collections import defaultdict\nfrom textblob import TextBlob\nfrom functools import partial\n\nimport numpy as np # linear algebra\n\n# visualization libraries\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport seaborn as sns\n\n# NLP libraries\nimport nltk\nimport spacy\nnlp = spacy.load('en_core_web_lg', disable=['parser', 'ner'])\nnlp.max_length = 4000000\nfrom nltk.probability import FreqDist\nfrom wordcloud import WordCloud, STOPWORDS\nimport string\n\nfrom tqdm.autonotebook import tqdm","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
\n

2. Modin makes all the difference

\n
\n
\n
Modin is a data processing library similar to Pandas. However it runs on all the cores available on the machine, unlike Pandas which runs only on a single core. Modin uses Ray or Dask behind the scenes for parallel computation. Using Modin significantly improves the code execution time as compared to the reference notebook.
\n
","metadata":{}},{"cell_type":"code","source":"# Uncomment and install in your Kaggle working directory.\n!pip install ../input/pythonmodin/modin-0.9.1-py3-none-manylinux1_x86_64.whl","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import modin.pandas as pd","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"base_path = '/kaggle/input/coleridgeinitiative-show-us-the-data/'\n\n# reading csv files and train & test file paths\ntrain_df = pd.read_csv(base_path + 'train.csv')\nsample_sub_df = pd.read_csv(base_path + 'sample_submission.csv')\ntrain_files_path = base_path + 'train/'\ntest_files_path = base_path + 'test/'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
\n

3. Data Exploration and Acquisition πŸ”

\n
\n
","metadata":{}},{"cell_type":"code","source":"train_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n
\n
The dataset has no null values
\n
\n
","metadata":{}},{"cell_type":"code","source":"#finding unique values in each column\n[(f\"{col}:{len(train_df[col].unique())}\") for col in train_df.columns]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_text_from_json_file(filename, folder_path=train_files_path, select_only='all'):\n '''\n Function to get the text from the json file and return it. This will be used to append the returned text \n to the dataframe\n '''\n json_file_path = os.path.join(folder_path, filename+'.json')\n headings_list = []\n body_list = []\n combined_list = []\n \n with open(json_file_path, 'r') as fhandle:\n json_str = json.load(fhandle)\n \n for item in json_str:\n headings_list.append(item.get('section_title'))\n body_list.append(item.get('text'))\n combined_list.append(item.get('section_title'))\n combined_list.append(item.get('text'))\n \n all_headings = ' '.join(headings_list)\n all_body = ' '.join(body_list)\n all_combined = ' '.join(combined_list)\n \n if select_only == 'all':\n return all_combined\n elif select_only == 'heading':\n return all_headings\n elif select_only == 'body':\n return all_body","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain_df['text'] = train_df['Id'].apply(get_text_from_json_file)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nsample_sub_df['text'] = sample_sub_df['Id'].apply(partial(get_text_from_json_file, folder_path=test_files_path))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def preprocess_text(text):\n \n '''\n This function converts the input text to lower case, then removes punctuation, emojis and \n multiple spaces and finally performs lemmatization\n \n text - Sentence that needs to be cleaned\n '''\n \n text = ''.join([k for k in text if k not in string.punctuation])\n text = re.sub('r[^\\w\\s]', ' ', str(text).lower()).strip()\n lem = nltk.stem.wordnet.WordNetLemmatizer()\n text = lem.lemmatize(text)\n \n return text","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def clean_text(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower().strip())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
\n

4. Data Preprocessing and Cleaning 🧹

\n
\n
","metadata":{}},{"cell_type":"code","source":"%%time\ntrain_df['text'] = train_df['text'].apply(preprocess_text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \nsample_sub_df['text'] = sample_sub_df['text'].apply(preprocess_text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
\n

5. Data Visualization πŸ“Š

\n
\n
","metadata":{}},{"cell_type":"code","source":"words = list(train_df['cleaned_label'].values)\nstop_words = ['ourselves', 'hers','the','of','and','in', 'between', 'yourself', 'but', 'again','of', 'there', 'about', 'once', 'during', 'out', 'very', 'having', 'with', 'they', 'own', 'an', 'be', 'some', 'for', 'do', 'its', 'yours', 'such', 'into', 'of', 'most', 'itself', 'other', 'off', 'is', 's', 'am', 'or', 'who', 'as', 'from', 'him', 'each', 'the', 'themselves', 'until', 'below', 'are', 'we', 'these', 'your', 'his', 'through', 'don', 'nor', 'me', 'were', 'her', 'more', 'himself', 'this', 'down', 'should', 'our', 'their', 'while', 'above', 'both', 'up', 'to', 'ours', 'had', 'she', 'all', 'no', 'when', 'at', 'any', 'before', 'them', 'same', 'and', 'been', 'have', 'in', 'will', 'on', 'does', 'yourselves', 'then', 'that', 'because', 'what', 'over', 'why', 'so', 'can', 'did', 'not', 'now', 'under', 'he', 'you', 'herself', 'has', 'just', 'where', 'too', 'only', 'myself', 'which', 'those', 'i', 'after', 'few', 'whom', 't', 'being', 'if', 'theirs', 'my', 'against', 'a', 'by', 'doing', 'it', 'how', 'further', 'was', 'here', 'than']\nsplit_words = []\nfor word in words:\n lo_w = []\n list_of_words = str(word).split()\n for w in list_of_words:\n if w not in stop_words:\n lo_w.append(w)\n split_words.append(lo_w)\nall_words = []\nfor word_list in split_words:\n all_words += word_list","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mostcommon_small = FreqDist(all_words).most_common(30)\nx, y = zip(*mostcommon_small)\nplt.figure(figsize=(50,30))\nplt.margins(0.02)\nplt.bar(x, y)\nplt.xlabel('Words', fontsize=50)\nplt.ylabel('Frequency of Words', fontsize=50)\nplt.yticks(fontsize=40)\nplt.xticks(rotation=60, fontsize=40)\nplt.tight_layout(pad=0)\nplt.title('Freq of 30 Most Common Words in cleaned_label', fontsize=60)\nplt.show()\n\nmost_common_words = FreqDist(all_words).most_common(100)\nword_cloud = WordCloud(width=1600, height=800, background_color='white', stopwords=STOPWORDS).generate(str(most_common_words))\n\nfig = plt.figure(figsize=(30,10), facecolor='white')\nplt.imshow(word_cloud, interpolation=\"bilinear\")\nplt.axis('off')\nplt.title('Top 100 Most Common Words in cleaned_label', fontsize=50)\nplt.tight_layout(pad=0)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
\n

6. Baseline Model πŸ“ˆ

\n
\n
","metadata":{}},{"cell_type":"code","source":"temp_1 = [x.lower() for x in train_df['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train_df['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train_df['cleaned_label'].unique()]\n\nexisting_labels = set(temp_1 + temp_2 + temp_3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"id_list = []\npredictions_list = []\n\nfor index, row in tqdm(sample_sub_df.iterrows()):\n# print('\\n\\n\\n')\n# print(index, row, '-'*50)\n sample_text = row['text']\n \n row_id = row['Id']\n temp_df = train_df[train_df['Id'] == row['Id']]\n# print('PRINTING TEMP_DF')\n# print(temp_df.head())\n \n# print('SAMPLE TEXT')\n# print(sample_text)\n# print('TEMP_DF TEXT')\n# print(temp_df['text'][0])\n \n cleaned_labels = temp_df['cleaned_label'].to_list()\n for known_label in existing_labels:\n if known_label in sample_text.lower():\n cleaned_labels.append(clean_text(known_label))\n \n cleaned_labels = [clean_text(x) for x in cleaned_labels]\n cleaned_labels = set(cleaned_labels)\n predictions_list.append('|'.join(cleaned_labels))\n id_list.append(row_id)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
\n

7. Submission πŸ“œ

\n
\n
","metadata":{}},{"cell_type":"code","source":"submission = pd.DataFrame()\nsubmission['Id'] = id_list\nsubmission['PredictionString'] = predictions_list\nsubmission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('/kaggle/working/submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!cd /kaggle/working/ && ls","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
\n

8. Conclusion 🎁

\n
\n
\n
This notebook uses basic string matching to match the names of the datasets mentioned in the research papers. Further improvements can be made where the model generalizes and identifies the datasets not present in the given list.
And that is a work in progress πŸ˜ƒ
\n
","metadata":{}},{"cell_type":"markdown","source":"
\n
\n

\n Voila!!! You have reached the end of this kernel. Feel free to fork this kernel. If you find this notebook useful, then please upvote and leave your valuable comments. Thank you. πŸ™\n

\n
\n

Created by Kartik B Bhargav

\n\n
\n
","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}