Please upvote the notebook as well if you find it useful and forking the notebook🎯
","metadata":{}},{"cell_type":"markdown","source":"\n#Coleridge Initiative🖋📝 - EDA📚 & Baseline Model🎯
","metadata":{}},{"cell_type":"markdown","source":"#Coleridge Initiative - Show US the Data
Discover how data is used for the public good
Table of Content
\n* [1. Importing necessary modules and libraries📚](#1)\n* [2. Data Exploration🔍](#2)\n* [3. Data Cleaning🔧](#3)\n* [4. Data Vizualization🎨](#4)\n* [5. Baseline model and Submission📝](#5)","metadata":{}},{"cell_type":"markdown","source":"\n#1. Importing necessary modules and libraries📚
","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)\n\nimport os\nimport re\nimport json\nimport glob\nfrom collections import defaultdict\nfrom textblob import TextBlob\nfrom functools import partial\n\nimport pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport seaborn as sns\n\nimport nltk\nimport spacy\nnlp = spacy.load('en_core_web_lg', disable=['parser', 'ner'])\nnlp.max_length = 4000000\nfrom nltk.probability import FreqDist\nfrom wordcloud import WordCloud, STOPWORDS\n\nfrom tqdm.autonotebook import tqdm\nimport string\n\n%matplotlib inline\n\nos.listdir('/kaggle/input/coleridgeinitiative-show-us-the-data/')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are provided with 4 main pieces of data:\n\n* `train.csv:` The CSV file containing all the metadata of the publications, such as their title and the dataset they utilize.\n* `train:` The directory containing the actual publications that are referenced in train.csvin JSON format.\n* `test:` The directory containing the actual publications that will be used for testing purposes (thus, with no ground truth CSV file available).\n* `sample_submission.csv:` The CSV file containing all the publications IDs in the test set, for which we'll have to populate the prediction column.","metadata":{}},{"cell_type":"markdown","source":"\n#2. Data Exploration🔍
","metadata":{}},{"cell_type":"code","source":"# reading csv files and train & test file paths\ntrain_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\nsample_sub = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')\ntrain_files_path = '../input/coleridgeinitiative-show-us-the-data/train'\ntest_files_path = '../input/coleridgeinitiative-show-us-the-data/test'","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head(6)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##Data Description
\ntrain.csv - labels and metadata for the training set train/test directory - the full text of the training/test set's publications in JSON format, broken into sections with section titles\n\n* `id` - publication id - note that there are multiple rows for some training documents, indicating multiple mentioned datasets.\n* `pub_title` - title of the publication (a small number of publications have the same title).\n* `dataset_title` - the title of the dataset that is mentioned within the publication.\n* `dataset_label` - a portion of the text that indicates the dataset.\n* `cleaned_label` - the dataset_label, as passed through the clean_text function from the Evaluation page.\n\nsample_submission.csv - a sample submission file in the correct format.\n* `Id` - publication id.\n* `PredictionString` - To be filled with equivalent of cleaned_label of train data.","metadata":{}},{"cell_type":"code","source":"train_df.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great! we don't have any null values.","metadata":{}},{"cell_type":"code","source":"[print(f\"{col}:{len(train_df[col].unique())}\") for col in train_df.columns] #finding unique values in each column","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_append_return(filename, train_files_path=train_files_path, output='text'):\n \"\"\"\n Function to read json file and then return the text data from them and append to the dataframe\n \"\"\"\n json_path = os.path.join(train_files_path, (filename+'.json'))\n headings = []\n contents = []\n combined = []\n with open(json_path, 'r') as f:\n json_decode = json.load(f)\n for data in json_decode:\n headings.append(data.get('section_title'))\n contents.append(data.get('text'))\n combined.append(data.get('section_title'))\n combined.append(data.get('text'))\n \n all_headings = ' '.join(headings)\n all_contents = ' '.join(contents)\n all_data = '. '.join(combined)\n \n if output == 'text':\n return all_contents\n elif output == 'head':\n return all_headings\n else:\n return all_data","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas() #tqdm is used to show any code running with a progress bar. \ntrain_df['text'] = train_df['Id'].progress_apply(read_append_return)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have our text appended in our train dataframe.","metadata":{}},{"cell_type":"code","source":"%%time\ntqdm.pandas()\nsample_sub['text'] = sample_sub['Id'].progress_apply(partial(read_append_return, train_files_path=test_files_path))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Also, we have the text of for the sample_submission file","metadata":{}},{"cell_type":"markdown","source":"\n#3. Data Cleaning🔧
","metadata":{}},{"cell_type":"code","source":"def text_cleaning(text):\n '''\n Converts all text to lower case, Removes special charecters, emojis and multiple spaces\n text - Sentence that needs to be cleaned\n '''\n text = ''.join([k for k in text if k not in string.punctuation])\n text = re.sub('[^A-Za-z0-9]+', ' ', str(text).lower()).strip()\n# text = re.sub(\"/'+/g\", ' ', text)\n \n return text","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntqdm.pandas()\ntrain_df['text'] = train_df['text'].progress_apply(text_cleaning)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It takes time!🙄","metadata":{}},{"cell_type":"code","source":"# %%time \n# tqdm.pandas()\n# sample_sub['text'] = sample_sub['text'].progress_apply(text_cleaning)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have our data cleaned!","metadata":{}},{"cell_type":"markdown","source":"\n#4. Data Vizualization🎨
","metadata":{}},{"cell_type":"code","source":"words =list( train_df['cleaned_label'].values)\nstopwords=['ourselves', 'hers','the','of','and','in', 'between', 'yourself', 'but', 'again','of', 'there', 'about', 'once', 'during', 'out', 'very', 'having', 'with', 'they', 'own', 'an', 'be', 'some', 'for', 'do', 'its', 'yours', 'such', 'into', 'of', 'most', 'itself', 'other', 'off', 'is', 's', 'am', 'or', 'who', 'as', 'from', 'him', 'each', 'the', 'themselves', 'until', 'below', 'are', 'we', 'these', 'your', 'his', 'through', 'don', 'nor', 'me', 'were', 'her', 'more', 'himself', 'this', 'down', 'should', 'our', 'their', 'while', 'above', 'both', 'up', 'to', 'ours', 'had', 'she', 'all', 'no', 'when', 'at', 'any', 'before', 'them', 'same', 'and', 'been', 'have', 'in', 'will', 'on', 'does', 'yourselves', 'then', 'that', 'because', 'what', 'over', 'why', 'so', 'can', 'did', 'not', 'now', 'under', 'he', 'you', 'herself', 'has', 'just', 'where', 'too', 'only', 'myself', 'which', 'those', 'i', 'after', 'few', 'whom', 't', 'being', 'if', 'theirs', 'my', 'against', 'a', 'by', 'doing', 'it', 'how', 'further', 'was', 'here', 'than']\nsplit_words=[]\nfor word in words:\n lo_w=[]\n list_of_words=str(word).split()\n for w in list_of_words:\n if w not in stopwords:\n lo_w.append(w)\n split_words.append(lo_w)\nallwords = []\nfor wordlist in split_words:\n allwords += wordlist","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 100 Most common words (cleaned_label) - WordCloud","metadata":{}},{"cell_type":"code","source":"mostcommon = FreqDist(allwords).most_common(100)\nwordcloud = WordCloud(width=1600, height=800, background_color='white', stopwords=STOPWORDS).generate(str(mostcommon))\nfig = plt.figure(figsize=(30,10), facecolor='white')\nplt.imshow(wordcloud, interpolation=\"bilinear\")\nplt.axis('off')\nplt.title('Top 100 Most Common Words in cleaned_label', fontsize=50)\nplt.tight_layout(pad=0)\nplt.show()\n\nmostcommon_small = FreqDist(allwords).most_common(25)\nx, y = zip(*mostcommon_small)\nplt.figure(figsize=(50,30))\nplt.margins(0.02)\nplt.bar(x, y)\nplt.xlabel('Words', fontsize=50)\nplt.ylabel('Frequency of Words', fontsize=50)\nplt.yticks(fontsize=40)\nplt.xticks(rotation=60, fontsize=40)\nplt.tight_layout(pad=0)\nplt.title('Freq of 25 Most Common Words in cleaned_label', fontsize=60)\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def prepare_text(text, nlp=nlp):\n '''\n Returns the text after stop-word removal and lemmatization.\n text - Sentence to be processed\n nlp - Spacy NLP model\n '''\n doc = nlp(text)\n lemma_list = [token.lemma_ for token in doc if not token.is_stop]\n lemmatized_sentence = ' '.join(lemma_list)\n \n return lemmatized_sentence","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def clean_text(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n# tqdm.pandas()\n# train_df['text'] = train_df['text'].progress_apply(prepare_text)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#5. Baseline model and Submission📝
","metadata":{}},{"cell_type":"code","source":"temp_1 = [x.lower() for x in train_df['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train_df['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train_df['cleaned_label'].unique()]\n\nexisting_labels = set(temp_1 + temp_2 + temp_3)\nid_list = []\nlables_list = []\nfor index, row in tqdm(sample_sub.iterrows()):\n sample_text = row['text']\n row_id = row['Id']\n temp_df = train_df[train_df['text'] == text_cleaning(sample_text)]\n cleaned_labels = temp_df['cleaned_label'].to_list()\n for known_label in existing_labels:\n if known_label in sample_text.lower():\n cleaned_labels.append(clean_text(known_label))\n cleaned_labels = [clean_text(x) for x in cleaned_labels]\n cleaned_labels = set(cleaned_labels)\n lables_list.append('|'.join(cleaned_labels))\n id_list.append(row_id)","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame()\nsubmission['Id'] = id_list\nsubmission['PredictionString'] = lables_list","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# pd.set_option(\"display.max_rows\", None, \"display.max_columns\", None)\nsubmission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Upvote ........","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}