{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n đź“ś Coleridge Initiative
\n 🔍 Complete EDA
","metadata":{}},{"cell_type":"markdown","source":"\n* [1. Introduction](#section-one)\n* [2. Data Understanding](#section-two)\n* [3. EDA for text publications](#section-three)\n* [4. EDA for dataset titles](#section-four)\n* [5. EDA for publication titles](#section-five)\n","metadata":{}},{"cell_type":"markdown","source":"\n\n\n> 📑 Context : This competition challenges data scientists to show how publicly funded data are used to serve science and society. Evidence through data is critical if government is to address the many threats facing society, including; pandemics, climate change, Alzheimer’s disease, child hunger, increasing food production, maintaining biodiversity, and addressing many other challenges. Yet much of the information about data necessary to inform evidence and science is locked inside publications.\n\n> In this competition, you'll use natural language processing (NLP) to automate the discovery of how scientific data are referenced in publications. Utilizing the full text of scientific publications from numerous research areas gathered from CHORUS publisher members and other sources, you'll identify data sets that the publications' authors used in their work.\n\n> 📌 Goal : The objective of the competition is to identify the mention of datasets within scientific publications. \n\n> Challenges : \n* It is an unsupervised task, the test set can have other datasets than those who are present in the train folder.\n* Some dataset labels are in the same in the ground truth.\n\n#### Librairies 📚","metadata":{}},{"cell_type":"code","source":"!pip install textstat\n\n#Basics\nimport numpy as np\nimport pandas as pd\nimport glob\nimport seaborn as sn\nimport seaborn as sn\nimport matplotlib.pyplot as plt\nfrom collections import defaultdict\nimport warnings\nimport gc\n\n#NLP librairies\nfrom nltk.tokenize import RegexpTokenizer\nfrom nltk.stem import WordNetLemmatizer,PorterStemmer\nfrom nltk.corpus import stopwords\nimport re\nfrom wordcloud import WordCloud, STOPWORDS\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom textstat import flesch_reading_ease","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n* `train.csv` : Labels and metadata for the training set from scientific publications in the train folder ;\n* `train` - the full text of the training set's publications in JSON format, broken into sections with section titles\n* `test` - the full text of the test set's publications in JSON format, broken into sections with section titles\n* The `sample_subimission.csv` : a sample submission file in the correct format.","metadata":{}},{"cell_type":"code","source":"DIR_TRAIN = \"../input/coleridgeinitiative-show-us-the-data/train/\"\nDIR_TEST = \"../input/coleridgeinitiative-show-us-the-data/test/\"\n\nDIR_TRAIN_CSV = \"../input/coleridgeinitiative-show-us-the-data/train.csv\"\ntrain_csv = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\nwarnings.filterwarnings(\"ignore\")","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_csv.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 2.1 Data description
\n\nThe train_csv file contains five columns : \n\n`id` - note that there are multiple rows for some training documents, indicating multiple mentioned datasets ;\n\n`pub_title` - title of the publication (a small number of publications have the same title) ;\n\n`dataset_title` - the title of the dataset that is mentioned within the publication ;\n\n`dataset_label` - a portion of the text that indicates the dataset ;\n\n`cleaned_label` - the dataset_label, as passed through the clean_text function from the Evaluation page","metadata":{"trusted":true}},{"cell_type":"markdown","source":"I'm adding a `text` column for each row corresponding to the full text : ","metadata":{}},{"cell_type":"code","source":"train_csv['text'] = train_csv.apply(lambda x : pd.read_json(DIR_TRAIN + x['Id'] + \".json\")['text'].str.cat(sep=' '), axis = 1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_csv.describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 2.2 Id publications with multiple dataset titles
\n\n> **Information** : There are Id publications in which there are multiple mention of dataset titles. How much there are ?\n","metadata":{}},{"cell_type":"code","source":"group_pub_dataset_title = train_csv.groupby('Id').count()[['dataset_title']].sort_values(by = \"dataset_title\", ascending = False)\nid_multiple_dataset = group_pub_dataset_title[group_pub_dataset_title['dataset_title'] >1][['dataset_title']].reset_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsn.barplot(x = id_multiple_dataset['dataset_title'].iloc[:20],\n y = id_multiple_dataset['Id'].iloc[:20])\n\nplt.title(\"How much dataset titles by Id publications\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"\")\nplt.xlabel(\"Count\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For example, the publication Id \"\"84ed3c4c-f57b-440c-8062-b8dff66a8421\" is duplicated two times in the train_csv with different dataset titles : ","metadata":{}},{"cell_type":"code","source":"train_csv[train_csv.duplicated(subset=['Id'])]\ntrain_csv[train_csv['Id'] == \"84ed3c4c-f57b-440c-8062-b8dff66a8421\"]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 2.3 Publication titles with multiple dataset titles
\n\n> **Information** : There are publication titles in which there are multiple mention of dataset title. How many are there ?\n","metadata":{}},{"cell_type":"code","source":"group_pub_dataset_title = train_csv.groupby('pub_title').count()[['dataset_title']].sort_values(by = \"dataset_title\", ascending = False)\npub_title_multiple_dataset = group_pub_dataset_title[group_pub_dataset_title['dataset_title'] >1][['dataset_title']].reset_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsn.barplot(x = pub_title_multiple_dataset['dataset_title'].iloc[:20],\n y = pub_title_multiple_dataset['pub_title'].iloc[:20])\n\nplt.title(\"How much dataset titles by publication titles\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"\")\nplt.xlabel(\"Count\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 2.3 Publication title with different Id publications
\n\n> **Information** : Each publication title with different Id publications : it means the same publication title for two differents publications ! \n\nHere are the five first publication title which have two different Id publications. There are in total 45.","metadata":{}},{"cell_type":"code","source":"group_pub_title = train_csv.drop_duplicates(\"Id\").groupby('pub_title').count()\ngroup_pub_title[group_pub_title['Id'] >1][['Id']].head(5)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For example, for the publication title \"A quantitative examination of lightning as a predictor of peak winds in tropical cyclones\" : ","metadata":{}},{"cell_type":"code","source":"train_csv[train_csv['pub_title'] == \"A quantitative examination of lightning as a predictor of peak winds in tropical cyclones\"]","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 2.4 Dataset titles with different dataset labels
\n\n> **Information** : How many dataset labels are there by dataset title ?","metadata":{}},{"cell_type":"code","source":"dataset_title_multiple_label = train_csv.drop_duplicates('dataset_label').groupby('dataset_title').count()[['dataset_label']].sort_values(by = 'dataset_label', ascending = False).reset_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsn.barplot(y = dataset_title_multiple_label['dataset_title'].iloc[:20],\n x = dataset_title_multiple_label['dataset_label'].iloc[:20])\n\nplt.title(\"How much dataset labels by dataset title there are ?\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"\")\nplt.xlabel(\"Count\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n 3.1 Number of words in text publications
\n\n> **Information** : How many words are there in texts ?\n\nI took a sample of 1000 texts representative of the distribution of number of words by text because there are texts with more than 80,000 words so we don't see in the graphic the distribution around the mean. ","metadata":{}},{"cell_type":"code","source":"#train_csv['text_splitted'] = train_csv['text'].str.split()\n#train_csv['nb_words'] = train_csv['text_splitted'].apply(len)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsn.distplot(pd.Series(train_csv['text'].unique()).apply(len), kde=True)\n\nplt.title(\"Sample of the distribution of number of words by text\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Density\")\nplt.xlabel(\"Number of words\", fontsize=14)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In mean, each text has 5000 words. The distribution is skewed on the right : there are also many texts between 5000 and 8000 words.","metadata":{}},{"cell_type":"markdown","source":" 3.2 The mean of word length in text publications
\n\n> **Information** : What is the mean of word length in text ?","metadata":{}},{"cell_type":"code","source":"#train_csv['avg_length_word'] = train_csv['text_splitted'].apply(lambda x : [len(i) for i in x]).map(lambda x: np.mean(x))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\n\nsn.distplot(pd.Series(train_csv['text'].unique()).apply(lambda x : x.split()).apply(lambda x : [len(i) for i in x]).map(lambda x: np.mean(x)), kde=True)\n\nplt.title(\"Average word length in text\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Density\")\nplt.xlabel(\"Average word length\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Is there really words with length around 49 characters ?\nWhat are these words ?","metadata":{}},{"cell_type":"code","source":"long_word_length = pd.Series(train_csv['text'].unique()).apply(lambda x : x.split())\ndef get_long_length(row):\n for x in row:\n if len(x)>40:\n return x\nlong_word_length.apply(get_long_length).unique()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are just web adresses and one with many stars. ","metadata":{}},{"cell_type":"markdown","source":" 3.3 Mostwords in text publications
\n\n> **Information** : What are the mostwords in text publications ?","metadata":{}},{"cell_type":"code","source":"stopwords = stopwords.words('english')\n\ndef preprocess(sentence):\n sentence=str(sentence)\n sentence = sentence.lower()\n sentence=sentence.replace('{html}',\"\") \n cleanr = re.compile('<.*?>')\n cleantext = re.sub(cleanr, '', sentence)\n rem_url=re.sub(r'http\\S+', '',cleantext)\n rem_num = re.sub('[0-9]+', '', rem_url)\n tokenizer = RegexpTokenizer(r'\\w+')\n tokens = tokenizer.tokenize(rem_num) \n filtered_words = [w for w in tokens if len(w) > 2 if not w in stopwords]\n return \" \".join(filtered_words)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_csv['clean_text'] = train_csv['text'].map(lambda s:preprocess(s))\nmostwords_in_text=defaultdict(int)\ndef get_mostwords_in_text(row):\n for word in row.split():\n mostwords_in_text[word] += 1\npd.Series(train_csv['clean_text'].unique()).apply(get_mostwords_in_text)\nmostwords_in_text = dict(sorted(mostwords_in_text.items(), key=lambda x: x[1], reverse = True))\nmostwords_in_text = pd.DataFrame.from_dict(mostwords_in_text, orient = 'index').reset_index()\nmostwords_in_text.columns = ['mostword', 'count']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsn.barplot(x = mostwords_in_text['count'].iloc[:20], \n y = mostwords_in_text['mostword'].iloc[:20])\n\nplt.title(\"Mostwords in text\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Words\")\nplt.xlabel(\"Count\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 3.4 Ngrams (bigram and trigram) for text publications
\n\n> **Information** : Ngrams are simply contiguous sequences of n words. For example “riverbank”,” The three musketeers” etc.If the number of words is two, it is called bigram. For 3 words it is called a trigram and so on. Looking at most frequent n-grams can give us a better understanding of the context in which the word was used.","metadata":{}},{"cell_type":"code","source":"def get_top_ngram(corpus, n=None):\n vec = CountVectorizer(ngram_range=(n, n)).fit(corpus)\n bag_of_words = vec.transform(corpus)\n sum_words = bag_of_words.sum(axis=0) \n words_freq = [(word, sum_words[0, idx]) \n for word, idx in vec.vocabulary_.items()]\n words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\n return words_freq[:10]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bigram_text = get_top_ngram(train_csv['clean_text'].unique(), 2)\nbigram_in_text = pd.DataFrame.from_dict(dict(bigram_text), orient = 'index').reset_index()\nbigram_in_text.columns = ['bigram', 'count']\n\nplt.figure(figsize=(16, 6))\nsn.barplot(x = bigram_in_text['count'].iloc[:20], \n y = bigram_in_text['bigram'].iloc[:20])\n\nplt.title(\"Bigrams in text\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Bigrams\")\nplt.xlabel(\"Count\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\ntrigram_text = get_top_ngram(train_csv['clean_text'].unique(), 3)\ntrigram_in_text = pd.DataFrame.from_dict(dict(trigram_text), orient = 'index').reset_index()\ntrigram_in_text.columns = ['trigram', 'count']\n\nplt.figure(figsize=(16, 6))\nsn.barplot(x = trigram_in_text['count'].iloc[:20], \n y = trigram_in_text['trigram'].iloc[:20])\n\nplt.title(\"Trigrams in text\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Trigrams\")\nplt.xlabel(\"Count\", fontsize=14)\n\n\"\"\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 3.5 Wordclouds for text publications
\n\n> **Information** : Wordcloud is a great way to represent text data. The size and color of each word that appears in the wordcloud indicate it’s frequency or importance.","metadata":{}},{"cell_type":"code","source":"stopwords = set(STOPWORDS)\nwordcloud = WordCloud(background_color='black',\n stopwords=stopwords,\n max_words=100,\n max_font_size=30,\n scale=3,\n random_state=1)\n \nwordcloud=wordcloud.generate(str(train_csv['text'].unique()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(1, figsize=(12, 12))\nplt.axis('off')\nplt.imshow(wordcloud)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 3.6 Text complexity in text publications
\n\n> **Information** : How readable (difficult to read) the text is and what type of reader can fully understand it ? Do we need a college degree to understand the message or a first-grader can clearly see what the point is ?","metadata":{}},{"cell_type":"code","source":"train_csv['text_readable'] = train_csv['text'].apply(lambda x : flesch_reading_ease(x))\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsn.distplot(train_csv['text_readable'].iloc[5000:10000], kde=True)\n\nplt.title(\"How readable are text publications ? (based on 5000 text samples)\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Density\")\nplt.xlabel(\"Text readability score\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The mean of readability score is around 30. It means that people from college to high school can read the scientific publications !","metadata":{}},{"cell_type":"markdown","source":"\n 4.1 Number of words in dataset titles
\n\n> **Information** : How many words are there in dataset titles ?\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsn.distplot(pd.Series(train_csv['dataset_title'].unique()).apply(len), kde=True)\n\nplt.title(\"Distribution of number of words by dataset titles\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Density\")\nplt.xlabel(\"Number of words\", fontsize=14)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are in mean 5 words in dataset titles. ","metadata":{}},{"cell_type":"markdown","source":" 4.2 The mean of word length in dataset titles
\n\n> **Information** : What is the mean of word length in dataset titles ?","metadata":{}},{"cell_type":"code","source":"#train_csv['avg_length_word_title'] = train_csv['title_splitted'].apply(lambda x : [len(i) for i in x]).map(lambda x: np.mean(x))\nplt.figure(figsize=(16, 6))\nsn.distplot(pd.Series(train_csv['dataset_title'].unique()).apply(lambda x : x.split()).apply(lambda x : [len(i) for i in x]).map(lambda x: np.mean(x)), kde=True)\n\nplt.title(\"Average word length in dataset titles\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Density\")\nplt.xlabel(\"Average dataset title word length\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 4.3 Mostwords in dataset titles
\n\n> **Information** : What are the mostwords in dataset titles ?","metadata":{}},{"cell_type":"code","source":"train_csv['clean_dataset_title'] = train_csv['dataset_title'].map(lambda s:preprocess(s))\nmostwords_in_dataset_title=defaultdict(int)\ndef get_mostwords_in_text(row):\n for word in row.split():\n mostwords_in_dataset_title[word] += 1\npd.Series(train_csv['clean_dataset_title'].unique()).apply(get_mostwords_in_text)\nmostwords_in_dataset_title = dict(sorted(mostwords_in_dataset_title.items(), key=lambda x: x[1], reverse = True))\nmostwords_in_dataset_title = pd.DataFrame.from_dict(mostwords_in_dataset_title, orient = 'index').reset_index()\nmostwords_in_dataset_title.columns = ['mostword', 'count']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsn.barplot(x = mostwords_in_dataset_title['count'].iloc[:20], \n y = mostwords_in_dataset_title['mostword'].iloc[:20])\n\nplt.title(\"Mostwords in dataset titles\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Words\")\nplt.xlabel(\"Count\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 4.4 Ngrams (bigram and trigram) for dataset titles
\n\n> **Information** : Ngrams are simply contiguous sequences of n words. For example “riverbank”,” The three musketeers” etc.If the number of words is two, it is called bigram. For 3 words it is called a trigram and so on. Looking at most frequent n-grams can give us a better understanding of the context in which the word was used.","metadata":{}},{"cell_type":"code","source":"bigram_dataset_title = get_top_ngram(train_csv['dataset_title'].unique(), 2)\nbigram_dataset_title = pd.DataFrame.from_dict(dict(bigram_dataset_title), orient = 'index').reset_index()\nbigram_dataset_title.columns = ['bigram', 'count']\n\nplt.figure(figsize=(16, 6))\nsn.barplot(x = bigram_dataset_title['count'].iloc[:20], \n y = bigram_dataset_title['bigram'].iloc[:20])\n\nplt.title(\"Bigrams in dataset titles\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Bigrams\")\nplt.xlabel(\"Count\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trigram_dataset_title = get_top_ngram(train_csv['dataset_title'].unique(), 3)\ntrigram_dataset_title = pd.DataFrame.from_dict(dict(trigram_dataset_title), orient = 'index').reset_index()\ntrigram_dataset_title.columns = ['trigram', 'count']\n\nplt.figure(figsize=(16, 6))\nsn.barplot(x = trigram_dataset_title['count'].iloc[:20], \n y = trigram_dataset_title['trigram'].iloc[:20])\n\nplt.title(\"Trigrams in dataset_title\", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Trigrams\")\nplt.xlabel(\"Count\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 4.5 Wordclouds for dataset titles
\n\n> **Information** : Wordcloud is a great way to represent text data. The size and color of each word that appears in the wordcloud indicate it’s frequency or importance.","metadata":{}},{"cell_type":"code","source":"stopwords = set(STOPWORDS)\nwordcloud = WordCloud(background_color='black',\n stopwords=stopwords,\n max_words=100,\n max_font_size=30,\n scale=3,\n random_state=1)\n \nwordcloud=wordcloud.generate(str(train_csv['dataset_title'].unique()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(1, figsize=(12, 12))\nplt.axis('off')\nplt.imshow(wordcloud)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 4.6 Text complexity in dataset titles
\n\n> **Information** : How readable (difficult to read) the dataset title is and what type of reader can fully understand it ? Do we need a college degree to understand the message or a first-grader can clearly see what the point is ?","metadata":{}},{"cell_type":"code","source":"train_csv['dataset_title_readable'] = pd.Series(train_csv['dataset_title'].unique()).apply(lambda x : flesch_reading_ease(x))\nplt.figure(figsize=(16, 6))\nsn.distplot(train_csv['dataset_title_readable'], kde=True)\n\nplt.title(\"How readable are dataset titles ? \", fontsize=16)\nplt.xticks(fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Density\")\nplt.xlabel(\"Dataset title readability score\", fontsize=14)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Thank you for reading my notebook. I hope you enjoyed it.
\n\n\nTO BE CONTINUED...","metadata":{}},{"cell_type":"markdown","source":"**Credits :** \n\n* https://neptune.ai/blog/exploratory-data-analysis-natural-language-processing-tools (very good tutorial on NLP data analysis)","metadata":{}}]}