{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on May 14, 2023 by Marília Prata, mpwolke","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport json\n\n#Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-14T20:23:17.533463Z","iopub.execute_input":"2023-05-14T20:23:17.533855Z","iopub.status.idle":"2023-05-14T20:23:18.578228Z","shell.execute_reply.started":"2023-05-14T20:23:17.533827Z","shell.execute_reply":"2023-05-14T20:23:18.576861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://brainstation-23.com/wp-content/uploads/2022/06/Kaggle-Participation.jpg)Brain Station 23","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv\", delimiter=',', encoding='UTF-8')\npd.set_option('display.max_columns', None)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:23:24.621530Z","iopub.execute_input":"2023-05-14T20:23:24.621948Z","iopub.status.idle":"2023-05-14T20:23:25.019231Z","shell.execute_reply.started":"2023-05-14T20:23:24.621918Z","shell.execute_reply":"2023-05-14T20:23:25.018041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Missing Values","metadata":{}},{"cell_type":"code","source":"df.isnull().sum()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-14T19:50:35.968991Z","iopub.execute_input":"2023-05-14T19:50:35.969666Z","iopub.status.idle":"2023-05-14T19:50:35.988302Z","shell.execute_reply.started":"2023-05-14T19:50:35.969628Z","shell.execute_reply":"2023-05-14T19:50:35.987357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Alaa Sedeeq https://www.kaggle.com/alaasedeeq/commonlit-readability-eda/execution\n\n\n#Define a function to plot a bar plot easily\n\ndef bar_plot(df,x,x_title,y,title,colors=None,text=None):\n    fig = px.bar(x=x,\n                 y=y,\n                 text=text,\n                 labels={x: x_title.title()},          # replaces default labels by column name\n                 data_frame=df,\n                 color=colors,\n                 barmode='group',\n                 template=\"simple_white\",\n                 color_discrete_sequence=px.colors.qualitative.Prism)\n    \n    texts = [df[col].values for col in y]\n    for i, t in enumerate(texts):\n        fig.data[i].text = t\n        fig.data[i].textposition = 'inside'\n        \n    fig['layout'].title=title\n\n    for trace in fig.data:\n        trace.name = trace.name.replace('_',' ').title()\n\n    fig.update_yaxes(tickprefix=\"\", showgrid=True)\n\n    fig.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:23:36.278830Z","iopub.execute_input":"2023-05-14T20:23:36.279202Z","iopub.status.idle":"2023-05-14T20:23:36.288777Z","shell.execute_reply.started":"2023-05-14T20:23:36.279174Z","shell.execute_reply":"2023-05-14T20:23:36.287572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Alaa Sedeeq https://www.kaggle.com/alaasedeeq/commonlit-readability-eda/execution\n\n\n# lets define a function to plot a histogram plot easily\n\ndef hist_plot(df,x,title):\n    fig = px.histogram(x=df[x],\n                       color_discrete_sequence=colors,\n                       opacity=0.8)\n\n    fig['layout'].title=title\n    fig.update_yaxes(tickprefix=\"\", showgrid=True)\n    fig.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:23:41.898272Z","iopub.execute_input":"2023-05-14T20:23:41.898694Z","iopub.status.idle":"2023-05-14T20:23:41.906149Z","shell.execute_reply.started":"2023-05-14T20:23:41.898650Z","shell.execute_reply":"2023-05-14T20:23:41.904645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Word Frequency on Writeup's Titles","metadata":{}},{"cell_type":"code","source":"#By Alaa Sedeeq https://www.kaggle.com/alaasedeeq/commonlit-readability-eda/execution\n\nimport plotly.express as px\nimport plotly.graph_objs as go\nimport itertools\n\n#Code by Alaa Sedeeq https://www.kaggle.com/alaasedeeq/commonlit-readability-eda/comments#1292516\n\n#Find words spreading (each word frequency)\nfreq_d = pd.Series(' '.join(df['Title of Writeup']).split()).value_counts()\n#Plot the words distribution\nfig = px.line(freq_d,\n              title='Word frequency on Writeups Titles')\nfig.update_layout(showlegend=False) ","metadata":{"execution":{"iopub.status.busy":"2023-05-14T19:57:07.982001Z","iopub.execute_input":"2023-05-14T19:57:07.982717Z","iopub.status.idle":"2023-05-14T19:57:08.089848Z","shell.execute_reply.started":"2023-05-14T19:57:07.982675Z","shell.execute_reply":"2023-05-14T19:57:08.088885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Replace the Nans, otherwise you'll get that error below (Float found=> Nan) \n\nTypeError: sequence item 105: expected str instance, float found","metadata":{}},{"cell_type":"code","source":"# categorical features with missing values\ncategorical_nan = [feature for feature in df.columns if df[feature].isna().sum()>0 and df[feature].dtypes=='O']\nprint(categorical_nan)","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:10:16.643887Z","iopub.execute_input":"2023-05-14T20:10:16.644414Z","iopub.status.idle":"2023-05-14T20:10:16.657414Z","shell.execute_reply.started":"2023-05-14T20:10:16.644363Z","shell.execute_reply":"2023-05-14T20:10:16.656274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# replacing missing values in categorical features\nfor feature in categorical_nan:\n    df[feature] = df[feature].fillna('None')","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:10:44.525202Z","iopub.execute_input":"2023-05-14T20:10:44.525641Z","iopub.status.idle":"2023-05-14T20:10:44.534332Z","shell.execute_reply.started":"2023-05-14T20:10:44.525603Z","shell.execute_reply":"2023-05-14T20:10:44.533068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[categorical_nan].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:11:01.343416Z","iopub.execute_input":"2023-05-14T20:11:01.344213Z","iopub.status.idle":"2023-05-14T20:11:01.357441Z","shell.execute_reply.started":"2023-05-14T20:11:01.344173Z","shell.execute_reply":"2023-05-14T20:11:01.356304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Word frequency on Writeups","metadata":{}},{"cell_type":"code","source":"#By Alaa Sedeeq https://www.kaggle.com/alaasedeeq/commonlit-readability-eda/execution\n\nimport plotly.express as px\nimport plotly.graph_objs as go\nimport itertools\n\n#Code by Alaa Sedeeq https://www.kaggle.com/alaasedeeq/commonlit-readability-eda/comments#1292516\n\n#Find words spreading (each word frequency)\nfreq_d = pd.Series(' '.join(df['Writeup']).split()).value_counts()\n#Plot the words distribution\nfig = px.line(freq_d,\n              title='Word frequency on Writeups')\nfig.update_layout(showlegend=False) ","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:11:25.412278Z","iopub.execute_input":"2023-05-14T20:11:25.412742Z","iopub.status.idle":"2023-05-14T20:11:27.643428Z","shell.execute_reply.started":"2023-05-14T20:11:25.412707Z","shell.execute_reply":"2023-05-14T20:11:27.641893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prepared_as_text = [line for line in df['Title of Writeup']]\ntext_prepared_results = '/n'.join(prepared_as_text)\n\ntext= ' '.join(t for t in df['Title of Writeup'])\nwords_list= text.split()","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:24:44.199014Z","iopub.execute_input":"2023-05-14T20:24:44.199394Z","iopub.status.idle":"2023-05-14T20:24:44.208764Z","shell.execute_reply.started":"2023-05-14T20:24:44.199365Z","shell.execute_reply":"2023-05-14T20:24:44.207484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"word_freq= {}\n\nfor word in set(words_list):\n    word_freq[word]= words_list.count(word)\n    \n#sorting the dictionary \nword_freq = dict(sorted(word_freq.items(), reverse=True, key=lambda item: item[1]))","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:24:49.135294Z","iopub.execute_input":"2023-05-14T20:24:49.135739Z","iopub.status.idle":"2023-05-14T20:24:50.438065Z","shell.execute_reply.started":"2023-05-14T20:24:49.135705Z","shell.execute_reply":"2023-05-14T20:24:50.436776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import itertools\n\n#sort the data and put it in a data frame for the visualization\nword_freq_temp = dict(itertools.islice(word_freq.items(), 25))\nword_freq_df = pd.DataFrame(word_freq_temp.items(),columns=['word','count']).sort_values('count',ascending=False)","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:25:28.508650Z","iopub.execute_input":"2023-05-14T20:25:28.509065Z","iopub.status.idle":"2023-05-14T20:25:28.521973Z","shell.execute_reply.started":"2023-05-14T20:25:28.509032Z","shell.execute_reply":"2023-05-14T20:25:28.520764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Frequent words on Writeups Titles","metadata":{}},{"cell_type":"code","source":"import plotly.express as px\n\n#Code by Alaa Sedeeq https://www.kaggle.com/alaasedeeq/commonlit-readability-eda/comments#1292516\n\nbar_plot(word_freq_df.reset_index(),\n         'word',\n         'Words',\n         ['count'],\n         title='20 Frequent words on Writeups Titles')","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:27:11.833614Z","iopub.execute_input":"2023-05-14T20:27:11.834193Z","iopub.status.idle":"2023-05-14T20:27:11.930123Z","shell.execute_reply.started":"2023-05-14T20:27:11.834133Z","shell.execute_reply":"2023-05-14T20:27:11.928999Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import nltk \nimport string\nfrom wordcloud import WordCloud\nnltk.download('stopwords')\nfrom nltk.corpus import stopwords\nstop = stopwords.words('english')\nfrom nltk.stem import WordNetLemmatizer\nfrom textblob import TextBlob,Word\nfrom collections import Counter","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:28:27.961843Z","iopub.execute_input":"2023-05-14T20:28:27.962239Z","iopub.status.idle":"2023-05-14T20:28:29.064619Z","shell.execute_reply.started":"2023-05-14T20:28:27.962210Z","shell.execute_reply":"2023-05-14T20:28:29.063446Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Alaa Sedeeq https://www.kaggle.com/alaasedeeq/commonlit-readability-eda/comments#1292516\n#Bigrams\nfrom nltk.util import ngrams    \n\ndef get_n_grans_count(text, n_grams, min_freq):\n    output = {}\n    tokens = nltk.word_tokenize(text)\n\n    #Create the n_gram\n    if n_grams == 2:\n        gs = nltk.bigrams(tokens)\n        \n    elif n_grams == 3:\n        gs = nltk.trigrams(tokens)\n\n    else:\n        return 'Only 2_grams and 3_grams are supported'\n    #compute frequency distribution for all the bigrams in the text\n    fdist = nltk.FreqDist(gs)\n    for k,v in fdist.items():\n        if v > min_freq:\n            index = ' '.join(k)\n            output[index] = v\n    \n    return output","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:29:12.802257Z","iopub.execute_input":"2023-05-14T20:29:12.802887Z","iopub.status.idle":"2023-05-14T20:29:12.812975Z","shell.execute_reply.started":"2023-05-14T20:29:12.802851Z","shell.execute_reply":"2023-05-14T20:29:12.811951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Bigrams","metadata":{}},{"cell_type":"code","source":"two_grams = get_n_grans_count(text, n_grams=2, min_freq=10)\ntwo_grams_df = pd.DataFrame(data=two_grams.items())\ntwo_grams_df = two_grams_df.sort_values(by=1,ascending=False).rename(columns={0:'Two grams',1:'Count'})\ntwo_grams_df","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:29:31.981758Z","iopub.execute_input":"2023-05-14T20:29:31.982152Z","iopub.status.idle":"2023-05-14T20:29:32.261934Z","shell.execute_reply.started":"2023-05-14T20:29:31.982123Z","shell.execute_reply":"2023-05-14T20:29:32.260783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Frequent Bigram on Writeup Titles","metadata":{}},{"cell_type":"code","source":"bar_plot(two_grams_df.iloc[:20],\n         'Two grams',\n         'Two grams',\n         ['Count'],\n         title='Frequent bigram on Writeup Titles')","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:30:26.914805Z","iopub.execute_input":"2023-05-14T20:30:26.915247Z","iopub.status.idle":"2023-05-14T20:30:27.003851Z","shell.execute_reply.started":"2023-05-14T20:30:26.915212Z","shell.execute_reply":"2023-05-14T20:30:27.002421Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Trigrams","metadata":{}},{"cell_type":"code","source":"three_grams = get_n_grans_count(text, n_grams=3, min_freq=0)\nthree_grams_df = pd.DataFrame(data=three_grams.items())\nthree_grams_df = three_grams_df.sort_values(by=1,ascending=False).rename(columns={0:'Three grams',1:'Count'})\nthree_grams_df","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:31:30.831815Z","iopub.execute_input":"2023-05-14T20:31:30.832234Z","iopub.status.idle":"2023-05-14T20:31:31.112526Z","shell.execute_reply.started":"2023-05-14T20:31:30.832204Z","shell.execute_reply":"2023-05-14T20:31:31.111252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bar_plot(three_grams_df.iloc[:20],\n         'Three grams',\n         'Three grams',\n         ['Count'],\n         title='Frequent trigrams on Writeups Titles')","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:32:45.500780Z","iopub.execute_input":"2023-05-14T20:32:45.501300Z","iopub.status.idle":"2023-05-14T20:32:45.590662Z","shell.execute_reply.started":"2023-05-14T20:32:45.501240Z","shell.execute_reply":"2023-05-14T20:32:45.589124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Words length","metadata":{}},{"cell_type":"code","source":"words_length = {}\n\nfor word in set(words_list):\n    words_length[word] = len(word)\n    \nwords_length = dict(sorted(words_length.items(), reverse=True, key=lambda item: item[1]))\n#sort the data and put it in a data frame for the visualization\nword_length_temp = dict(itertools.islice(words_length.items(), 25))\nwords_length_df = pd.DataFrame(words_length.items(),columns=['Title of Writeup','count']).sort_values('count',ascending=False)","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:34:13.125440Z","iopub.execute_input":"2023-05-14T20:34:13.125929Z","iopub.status.idle":"2023-05-14T20:34:13.148036Z","shell.execute_reply.started":"2023-05-14T20:34:13.125895Z","shell.execute_reply":"2023-05-14T20:34:13.146615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Sentence level analysis\n\nSentence level analysis Text statistics include sentence length distribution, minimum, maximum, and average length. To check the sentence length distribution. Code and output are as follows:","metadata":{}},{"cell_type":"code","source":"df['sentence_len']= df['Title of Writeup'].str.len()\nprint('Max length     : {} \\nMin length     : {} \\nAverage Length : {}'.\\\n      format(max(df['sentence_len']),min(df['sentence_len']),df['sentence_len'].mean()))","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:35:49.141765Z","iopub.execute_input":"2023-05-14T20:35:49.142183Z","iopub.status.idle":"2023-05-14T20:35:49.156776Z","shell.execute_reply.started":"2023-05-14T20:35:49.142153Z","shell.execute_reply":"2023-05-14T20:35:49.155474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#The Longest Sentence on Writeups Titles","metadata":{}},{"cell_type":"code","source":"#the longest sentence we have\ndf[df['sentence_len']==max(df['sentence_len'])]['Title of Writeup'].values[0]","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:36:47.823125Z","iopub.execute_input":"2023-05-14T20:36:47.823583Z","iopub.status.idle":"2023-05-14T20:36:47.833130Z","shell.execute_reply.started":"2023-05-14T20:36:47.823527Z","shell.execute_reply":"2023-05-14T20:36:47.831808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#The Shortest sentence: 'models'","metadata":{}},{"cell_type":"code","source":"#the shortest sentence we have\ndf[df['sentence_len']==min(df['sentence_len'])]['Title of Writeup'].values[0]","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:37:48.639816Z","iopub.execute_input":"2023-05-14T20:37:48.640251Z","iopub.status.idle":"2023-05-14T20:37:48.650252Z","shell.execute_reply.started":"2023-05-14T20:37:48.640219Z","shell.execute_reply":"2023-05-14T20:37:48.649107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Sentences Lenght Distribution","metadata":{}},{"cell_type":"code","source":"colors = px.colors.qualitative.Prism\n\nhist_plot(df,\n          'sentence_len',\n          title='Sentences lenght distribution with spaces')","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:39:24.196892Z","iopub.execute_input":"2023-05-14T20:39:24.197926Z","iopub.status.idle":"2023-05-14T20:39:24.327953Z","shell.execute_reply.started":"2023-05-14T20:39:24.197887Z","shell.execute_reply":"2023-05-14T20:39:24.326597Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Competition's Titles","metadata":{}},{"cell_type":"code","source":"#By Rob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream\n\nfig, ax = plt.subplots(figsize=(4, 4))\ndf[\"Title of Competition\"].value_counts().head(10).sort_values(ascending=True).plot(\n    kind=\"barh\", color='g', ax=ax, title=\"Kaggle Title of Competitions\"\n)\nax.set_xlabel(\"Number of Training Examples\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-14T20:00:53.320963Z","iopub.execute_input":"2023-05-14T20:00:53.321390Z","iopub.status.idle":"2023-05-14T20:00:53.702595Z","shell.execute_reply.started":"2023-05-14T20:00:53.321358Z","shell.execute_reply":"2023-05-14T20:00:53.701410Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[\"Title of Writeup\"].value_counts()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-14T20:00:59.950627Z","iopub.execute_input":"2023-05-14T20:00:59.951729Z","iopub.status.idle":"2023-05-14T20:00:59.962053Z","shell.execute_reply.started":"2023-05-14T20:00:59.951694Z","shell.execute_reply":"2023-05-14T20:00:59.961229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Titles of Writeups: 1st place Solution \n\nWe have a difference between 1st place (lower case: 44 times) and 1st Place (upper case)","metadata":{}},{"cell_type":"code","source":"#By Rob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream\n\nfig, ax = plt.subplots(figsize=(4, 4))\ndf[\"Title of Writeup\"].value_counts().head(10).sort_values(ascending=True).plot(\n    kind=\"barh\", color='r', ax=ax, title=\"Kaggle Competitions Writeups Rock!\"\n)\nax.set_xlabel(\"Number of Training Examples\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-14T20:01:04.849058Z","iopub.execute_input":"2023-05-14T20:01:04.849623Z","iopub.status.idle":"2023-05-14T20:01:05.133703Z","shell.execute_reply.started":"2023-05-14T20:01:04.849573Z","shell.execute_reply":"2023-05-14T20:01:05.132484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Checking Writeup Titles  \n\nWe have Solution, Place,Medal, Silver and even Bronze. Where is the Gold? Not many, just for fews.","metadata":{}},{"cell_type":"code","source":"##Code by Taha07  https://www.kaggle.com/taha07/data-scientists-jobs-analysis-visualization/notebook\n\nfrom wordcloud import WordCloud\nfrom wordcloud import STOPWORDS\nstopwords = set(STOPWORDS)\nwordcloud = WordCloud(background_color = 'black',\n                      colormap ='Set2',\n                      height =2000,\n                      width = 2000\n                     ).generate(str(df[\"Title of Writeup\"]))\nplt.rcParams['figure.figsize'] = (12,12)\nplt.axis(\"off\")\nplt.imshow(wordcloud)\nplt.title(\"Title of Kaggle Competitions Writeups\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-14T20:01:10.442786Z","iopub.execute_input":"2023-05-14T20:01:10.443229Z","iopub.status.idle":"2023-05-14T20:01:13.646831Z","shell.execute_reply.started":"2023-05-14T20:01:10.443192Z","shell.execute_reply":"2023-05-14T20:01:13.645996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Thank, Organisers, Hosting, Surprised, Congratulation, Fun and Kaggle: Writeups ","metadata":{}},{"cell_type":"code","source":"##Code by Taha07  https://www.kaggle.com/taha07/data-scientists-jobs-analysis-visualization/notebook\n\nfrom wordcloud import WordCloud\nfrom wordcloud import STOPWORDS\nstopwords = set(STOPWORDS)\nwordcloud = WordCloud(background_color = 'lightblue',\n                      colormap ='Set3',\n                      height =2000,\n                      width = 2000\n                     ).generate(str(df[\"Writeup\"]))\nplt.rcParams['figure.figsize'] = (12,12)\nplt.axis(\"off\")\nplt.imshow(wordcloud)\nplt.title(\"Kaggle Competitions Writeups\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-14T20:01:35.159488Z","iopub.execute_input":"2023-05-14T20:01:35.159941Z","iopub.status.idle":"2023-05-14T20:01:38.643318Z","shell.execute_reply.started":"2023-05-14T20:01:35.159852Z","shell.execute_reply":"2023-05-14T20:01:38.642452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import glob\nimport spacy\nfrom spacy import displacy\nfrom pathlib import Path\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom wordcloud import WordCloud, STOPWORDS, ImageColorGenerator","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:01:45.897977Z","iopub.execute_input":"2023-05-14T20:01:45.898654Z","iopub.status.idle":"2023-05-14T20:01:57.718691Z","shell.execute_reply.started":"2023-05-14T20:01:45.898619Z","shell.execute_reply":"2023-05-14T20:01:57.717510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Abu Bakar https://www.kaggle.com/bakar31/writing-evaluation-noob-eda/notebook\n\ndef get_top_n_words(corpus, n=None, remove_stop_words=False, n_words=1): \n    \n    if remove_stop_words:\n        vec = CountVectorizer(stop_words = 'english', ngram_range=(n_words, n_words)).fit(corpus)\n    else:\n        vec = CountVectorizer(ngram_range=(n_words, n_words)).fit(corpus)\n        \n    bag_of_words = vec.transform(corpus)\n    sum_words = bag_of_words.sum(axis=0) \n    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\n    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\n    return words_freq[:n]","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:02:01.415001Z","iopub.execute_input":"2023-05-14T20:02:01.415926Z","iopub.status.idle":"2023-05-14T20:02:01.425790Z","shell.execute_reply.started":"2023-05-14T20:02:01.415878Z","shell.execute_reply":"2023-05-14T20:02:01.424382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -U spacy","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-14T20:02:07.347535Z","iopub.execute_input":"2023-05-14T20:02:07.347971Z","iopub.status.idle":"2023-05-14T20:02:21.691815Z","shell.execute_reply.started":"2023-05-14T20:02:07.347925Z","shell.execute_reply":"2023-05-14T20:02:21.690380Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!python -m spacy download en_core_web_sm","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-14T20:02:24.431569Z","iopub.execute_input":"2023-05-14T20:02:24.431961Z","iopub.status.idle":"2023-05-14T20:02:46.526855Z","shell.execute_reply.started":"2023-05-14T20:02:24.431928Z","shell.execute_reply":"2023-05-14T20:02:46.525691Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Working on: Solution sharing by Jiwei Liu 1st Place - 9 years ago \n\nCompetition: Tradeshift Text Classification\n\nhttps://www.kaggle.com/c/3984/discussion/10901\n\nhttps://www.kaggle.com/competitions/tradeshift-text-classification/overview\n\nOnly 3 Kaggle Notebooks!","metadata":{}},{"cell_type":"code","source":"import spacy\nfrom spacy.lang.ru.examples import sentences \n\nnlp = spacy.load(\"en_core_web_sm\")\ndoc = nlp(\"\"\"Hi, sorry to disappoint you that there is no magic but brute forcing and many many machine hours. All our work are based on Dmitry and tinrtgu's great benchmarks, and Tianqi Chen's great tool Xgboost. https://github.com/tqchen/xgboost\n\nMany many thanks! You are the true heroes!\n\nOur winning solution ensembles 14 two-stage xgb models and 7 online models. Our best single xgb model gets 0.0043835/0.0044595 for public and private LB. It is generated as follows:\n\n1) Use the second half training data as base and the first half training data as meta, instead of random split. (this is key!)\n\n2) we use four base classifiers: random forest for numerical features, SGDClassifier for sparse features, online logistic for all features and xgb for all features.\n\n3) For meta classifier, we use xgb with depth 18, 120 trees and 0.09 eta.\n\nThe xgb models could be memory intensive. We use a 8-core 32 GB memory server for most of our submissions. Thank my boss for the machine :P\n\nWe will make a formal description and code release after some cleaning up. Cheers!\"\"\")\nentities=[(i, i.label_, i.label) for i in doc.ents]\nentities\n\n#print(doc.text)\n#for token in doc:\n #   print(token.text, token.pos_, token.dep_)","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:02:56.099129Z","iopub.execute_input":"2023-05-14T20:02:56.099577Z","iopub.status.idle":"2023-05-14T20:02:57.767639Z","shell.execute_reply.started":"2023-05-14T20:02:56.099542Z","shell.execute_reply":"2023-05-14T20:02:57.766467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#We will make a formal description and code release after some cleaning up. Cheers!\n\nThat part of the Solution I understood.","metadata":{}},{"cell_type":"code","source":"displacy.render(doc, style = \"ent\",jupyter = True)","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:03:07.456377Z","iopub.execute_input":"2023-05-14T20:03:07.456839Z","iopub.status.idle":"2023-05-14T20:03:07.468063Z","shell.execute_reply.started":"2023-05-14T20:03:07.456805Z","shell.execute_reply":"2023-05-14T20:03:07.466868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Tokenization","metadata":{}},{"cell_type":"code","source":"import spacy\n\nnlp = spacy.load('en_core_web_sm')\ndoc = nlp(\"\"\"Hi, sorry to disappoint you that there is no magic but brute forcing and many many machine hours. All our work are based on Dmitry and tinrtgu's great benchmarks, and Tianqi Chen's great tool Xgboost. https://github.com/tqchen/xgboost\n\nMany many thanks! You are the true heroes!\n\nOur winning solution ensembles 14 two-stage xgb models and 7 online models. Our best single xgb model gets 0.0043835/0.0044595 for public and private LB. It is generated as follows:\n\n1) Use the second half training data as base and the first half training data as meta, instead of random split. (this is key!)\n\n2) we use four base classifiers: random forest for numerical features, SGDClassifier for sparse features, online logistic for all features and xgb for all features.\n\n3) For meta classifier, we use xgb with depth 18, 120 trees and 0.09 eta.\n\nThe xgb models could be memory intensive. We use a 8-core 32 GB memory server for most of our submissions. Thank my boss for the machine :P\n\nWe will make a formal description and code release after some cleaning up. Cheers!\"\"\")\nfor token in doc:\n    print(token.text)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-14T20:03:14.147269Z","iopub.execute_input":"2023-05-14T20:03:14.147678Z","iopub.status.idle":"2023-05-14T20:03:15.553863Z","shell.execute_reply.started":"2023-05-14T20:03:14.147647Z","shell.execute_reply":"2023-05-14T20:03:15.552568Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Part-Of-Speech (POS) Tagging","metadata":{}},{"cell_type":"code","source":"nlp = spacy.load('en_core_web_sm')\n\n# Create an nlp object\ndoc = nlp(\"\"\"Hi, sorry to disappoint you that there is no magic but brute forcing and many many machine hours. All our work are based on Dmitry and tinrtgu's great benchmarks, and Tianqi Chen's great tool Xgboost. https://github.com/tqchen/xgboost\n\nMany many thanks! You are the true heroes!\n\nOur winning solution ensembles 14 two-stage xgb models and 7 online models. Our best single xgb model gets 0.0043835/0.0044595 for public and private LB. It is generated as follows:\n\n1) Use the second half training data as base and the first half training data as meta, instead of random split. (this is key!)\n\n2) we use four base classifiers: random forest for numerical features, SGDClassifier for sparse features, online logistic for all features and xgb for all features.\n\n3) For meta classifier, we use xgb with depth 18, 120 trees and 0.09 eta.\n\nThe xgb models could be memory intensive. We use a 8-core 32 GB memory server for most of our submissions. Thank my boss for the machine :P\n\nWe will make a formal description and code release after some cleaning up. Cheers!\"\"\")\n \n# Iterate over the tokens\nfor token in doc:\n    # Print the token and its part-of-speech tag\n    print(token, token.tag_, token.pos_, spacy.explain(token.tag_))","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-14T20:03:32.408534Z","iopub.execute_input":"2023-05-14T20:03:32.408913Z","iopub.status.idle":"2023-05-14T20:03:33.810850Z","shell.execute_reply.started":"2023-05-14T20:03:32.408883Z","shell.execute_reply":"2023-05-14T20:03:33.809686Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#WordCloud - Spacy","metadata":{}},{"cell_type":"code","source":"from wordcloud import WordCloud, STOPWORDS\nimport matplotlib.pyplot as plt\nnlp = spacy.load(\"en_core_web_sm\")  # make sure to use larger model!\ntokens = nlp(\"\"\"Hi, sorry to disappoint you that there is no magic but brute forcing and many many machine hours. All our work are based on Dmitry and tinrtgu's great benchmarks, and Tianqi Chen's great tool Xgboost. https://github.com/tqchen/xgboost\n\nMany many thanks! You are the true heroes!\n\nOur winning solution ensembles 14 two-stage xgb models and 7 online models. Our best single xgb model gets 0.0043835/0.0044595 for public and private LB. It is generated as follows:\n\n1) Use the second half training data as base and the first half training data as meta, instead of random split. (this is key!)\n\n2) we use four base classifiers: random forest for numerical features, SGDClassifier for sparse features, online logistic for all features and xgb for all features.\n\n3) For meta classifier, we use xgb with depth 18, 120 trees and 0.09 eta.\n\nThe xgb models could be memory intensive. We use a 8-core 32 GB memory server for most of our submissions. Thank my boss for the machine :P\n\nWe will make a formal description and code release after some cleaning up. Cheers!\"\"\")\n\nnewText =''\nfor word in tokens:\n if word.pos_ in ['ADJ', 'NOUN']:\n  newText = \" \".join((newText, word.text.lower()))\n\nwordcloud = WordCloud(stopwords=STOPWORDS, colormap='Reds', background_color=\"blue\").generate(newText)\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.axis(\"off\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-14T20:03:42.922018Z","iopub.execute_input":"2023-05-14T20:03:42.922433Z","iopub.status.idle":"2023-05-14T20:03:44.683630Z","shell.execute_reply.started":"2023-05-14T20:03:42.922389Z","shell.execute_reply":"2023-05-14T20:03:44.682130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Post-Mortem. Posted Eleven years ago - Most Creative Title among Many, many Solutions\n\nPost-Mortem \n\nBy BarrenWuffet (9th Place on that \"Raising Money to Fund an Organizational Mission\" competition)\n\nUnfortunately, Barren \"last seen on Kaggle\" was 6 months ago. I hope to see you soon back.\n\n\"Anybody do or find anything interesting in this dataset? Or find any good tools for working with large data sets?\"\n\n\"I found it barely manageable given the size.  It also took me awhile to wrap by head around what was in each of the files. Also the database restrictions (1 and 2 but not 3, 1 and 2 and 3, 2 and 3 but not 1, etc) made a really difficult task much harder and considerably less enjoyable.  I ended up having to write 12 large files (and accompanying SQL code) for each model to compensate for this.\"\n\n\"I started out with a ~2% sample of the training data in R but even this was rough.  I tried using the ff package without much luck.  I ended up doing most of the data manipulation using SQL Server 2008 R2 Express and the SSMS which I found to be a bright spot in the whole process as it performed really well given the data size.  I especially appreciated the data import manager which helps with wide data sets.\"\n\n\"My best model ended up just being prior averages for prospectid, zip5, and packageid with linear regression.  I predicted donation amount (not amount2) and response rate using separate models.  I'd then do predictedGift^1.15 * predictedResponseRate for a final prediction.\"\n\n\"I tried to use some of the demographic data but had a hard time as I was using zip5 as the key to get state abbreviations as a factor, but some of the zip codes cross state lines which leads to duplicates and zip9, even when indexed, just took too long.\"\n\n\"I think the contest was a cool idea but would have been much better on just one of the 3 databases.  Without that restriction I would have had more time to explore the demographic and historical data.  The one thing I would have really liked to explore is people's giving before the training period especially as much of the mailings seemed to be political and the test data is sitting within 12 months of a presidential election.\"\n\nhttps://www.kaggle.com/c/2863/discussion/2712","metadata":{}},{"cell_type":"code","source":"mor = df[(df['Title of Writeup']=='Post-Mortem')].reset_index(drop=True)\nmor.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-14T20:03:53.084301Z","iopub.execute_input":"2023-05-14T20:03:53.084730Z","iopub.status.idle":"2023-05-14T20:03:53.105817Z","shell.execute_reply.started":"2023-05-14T20:03:53.084695Z","shell.execute_reply":"2023-05-14T20:03:53.104634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Closing Arguments:\n\nAfter Kaggle added Notebooks/Datasets on tier progression, more users started to share code and Data on Competitions.\n\nTherefore, it became easier to learn with each others. Cause only Solutions aren't enough for kagglers (mostly beginners) to understand how things really go. Just sharing Solutions works perfectly on a \"site for competitions\".\n\nSince Kaggle is now a Platform for Data Science and Machine Learning, the new approach seems more democratic. Once more users can reach upper tier positions and share much more material. Besides, the \"Status Quo\" changed a little bit.\n\nReaching higher tier positions on Competitions depends on the kaggler capacity, knowledge, skills and experience on competitions. \n\nGrowing on Notebooks progression depends on The Audience and algorithms approval. It's subjective.\n\nThough all of us have the capacity to discern and make our options about what we're going to support or just ignore.\n\nThe more of us grow, the stronger our community becomes.","metadata":{}},{"cell_type":"markdown","source":"#Waiting for new competitions memes thread\n\nTopic by Artgor, meme by Tunguz\n\n![](https://pbs.twimg.com/media/EPSnRIUXsAw2E2I?format=png&name=900x900)https://www.kaggle.com/general/127889","metadata":{}},{"cell_type":"markdown","source":"#Acknowledgements:\n\nMPWolke https://www.kaggle.com/code/mpwolke/pranks-tar-files-russian-spacy-mp3\n\nMPWolke https://www.kaggle.com/code/mpwolke/rock-pop-lyrics\n\nAlaa Sedeeq https://www.kaggle.com/alaasedeeq/commonlit-readability-eda/execution\n\n\nAbu Bakar https://www.kaggle.com/bakar31/writing-evaluation-noob-eda/notebook\n\nTaha07  https://www.kaggle.com/taha07/data-scientists-jobs-analysis-visualization/notebook\n\nRob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream","metadata":{}}]}