{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Stack Exchange - Tag Classifier","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> **Note:** Main focus of this kernel is not objective. Instead of focusing on getting as high accuracy score as possible, we will focus on real-world scenarios, ML/DS concepts","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 01 - Data - High Level Overview\n\nDataset contains content from **disparate(different) stack exchange sites**, containing a **mix of both technical and non-technical** questions.\n\n| Feature | Desc |\n| --- | --- |\n| Id | Unique identifier for each question |\n| Title | The question's title |\n| Body | The body of the question |\n| Tags | The tags associated with the question (all lowercase, should not contain tabs '\\t' or ampersands '&') |\n\n\n<br><br>\nThe questions are **randomized** and contains a mix of verbose **text sites as well as sites related to math and programming**. The number of **questions from each site may vary**, and **no filtering** has been performed on the questions (such as closed questions).\n\n**SIZE:** ~3GB\n\n\n| Source | Inference |\n| --- | --- |\n| \"disparate(different) stack exchange sites\" | Question are from different domains |\n|\"questions from each site may vary\"| Chance of underfitting/overfitting |\n| \"both technical and non-technical\" \"text sites as well as sites related to math and programming\" | Question text may be numbers/words/symbols |\n|\"questions are **randomized**\"| End to end solution is needed |\n| \"no filtering\" | -- |","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 02 - Objective / Problem Statement\n\nPredict **tags** based on input question's **Title** and **Body**\n\nProblem Type: `Classification`","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 03 - Real world use case constraints\n\n| Constraint | Required / Not Required | Comment |\n| --- | --- | --- |\n| High Precision | required | Impact on UX | \n| High Recall | required | Impact on UX | \n| Low Latency | not requred | No significant impact on UX |\n| High Interpretability | required | Impact on UX |\n\n\n<br><br><br>\nIf you forgot about precision and recall: \n\n> When a search engine returns 30 pages only 20 of which were relevant while failing to return 40 additional relevant pages, its precision is 20/30 = 2/3 while its recall is 20/60 = 1/3. So, in this case, precision is **\"how useful the search results are\"**, and recall is **\"how complete the results are\".** [wikipedia](https://en.wikipedia.org/wiki/Precision_and_recall)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 04 - Mapping Real World Problem to Machine Learning Problem\n\n1. Type of ML Problem\n2. Key Performance Indicatiors\n\n### Type of ML problem\n\n- Classification Problem (Not multi-class)\n- Multi-Label Classification [sci-kit docs](https://scikit-learn.org/stable/modules/multiclass.html)\n\n### Key Performance Indicatiors\n\nNot binary classification.\n\n- F1 Score\n- Micro F1 Score\n- Macro F1 Score\n- Hamming Loss\n\n### i. F1-Score\n\nWighted average of precision and recall. When all weights are 1 $\\implies$ Harmonic mean $\\implies \\frac{\\sum{w_i}}{\\frac{w_i}{x_i}} $\n\n$$F1 = \\frac{2PR}{P + R}$$\n\nwhere  $p=\\frac{tp}{tp+fp}$,  $r=\\frac{tp}{tp+fn}$\n\n> It combines power of both P and R\n\n### ii. Macro F1 Score \n\nSimple average of F1 Score of each class\n\n$$ F1_{macro} = \\frac{1}{n_{classes}} \\sum_{k=1}^{n_{classes}} \\text{F1}_{k} $$ \n\n> Doesn't take class imbalance into account\n\n\n### iii. Micro F1 Score\n\n$$ F1_{micro} = \\frac{2 P_{micro} R_{micro}} {P_{micro} + R_{micro}} $$\n\nwhere  $p_{micro}= \\sum_{k \\in C} \\frac{tp_{k}}{tp_{k}+fp_{k}}$ and $r_{micro}= \\sum_{k \\in C} \\frac{tp_{k}}{tp_{k}+fn_{k}}$ \n\n- We are calculating $P$ and $R$ from all classes and using it in $F1-Score$ formula\n\n- **Note:** F1-Score can be high even when $P$ and $R$ of minority class is very small **WHEN** $P$ and $R$ of majority class is very high \n\n> It is sort of weighted average so, takes class imbalance into account (both numerator and denominator of $P_{micro}$ $R_{micro}$ will incr/decr with imbalance)\n\n### iv. Hamming Loss\n\n$$ Hamming Loss(\\hat{y_{i}}, y_{i}) = \\frac{1}{N_{samples}} \\sum_{i=1}^{N_{samples}} \\frac{xor(\\hat{y_{i}}, y_{i})}{N_{labels}} $$\n\nwhere $\\hat{y_{i}}, y_{i}$ are encoded vectors vectors","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## A. EDA - Preliminary Analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport sys\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/facebook-recruiting-iii-keyword-extraction/Train.zip\")\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**FEATURES DESC**\n\n|Name|Desc|Data Type|\n| --- | --- | --- |\n| Id | Unique |Continous, Numerical |\n| Title | ascii | Mixed |\n| Body  | ascii | Mixed |\n| Tags | Target | Categorical, Multi-Label |\n\n**Number of samples:** 6,034,195 (6M) <bR>\n**Number of feats:** 4","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Data Cleaning - Removing Duplicates","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# get duplicates\ndf_dups = df[df.duplicated(['Title', 'Body', 'Tags'])]\nprint('Total Duplicates: ', len(df_dups))\nprint('ratio: ', len(df_dups)/len(df))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# remove duplicates\ndf = df.drop_duplicates(['Title', 'Body', 'Tags'])\nprint('After removing dups: ', len(df))\nprint('ratio: ', len(df)/6034194)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Analysis of Target Varible - Tags","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**a. distribution of number of tags per qn**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"x = df[\"Tags\"].apply(lambda x: type(x)==float)\nx[x==True]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# removing `Tags` which are float instead of str\ndf.drop([err_idx for err_idx in x[x==True].index], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df[\"num_of_tags\"] = df[\"Tags\"].apply(lambda x: len(x.split(\" \")))\ndf['num_of_tags'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.close()\n\nplt.bar(\n    df.num_of_tags.value_counts().index,\n    df.num_of_tags.value_counts()\n)\n\nplt.xlabel('Number of tags')\nplt.ylabel('Freq (x10^6)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Observation and Inference:**\n    \n- Maximum 5 tags per question averaging 3 tags per question","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**b. unique tags**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer\n\n# get unique tags w/ help of BoW. Tags are space separated\nvectorizer = CountVectorizer(tokenizer = lambda x: x.split())\n\n# fit_transform\n# - learn the vocabulary and store in `vectorizer`\n# - convert training data into feature vectors\n#    - converts each input (tag) into one hot encoded based on vocab\ntag_vecs = vectorizer.fit_transform(df['Tags'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# learnt vocabulary\nvocab = vectorizer.get_feature_names()\nprint(vocab[:5])\n\n# total vocabulary\nprint('Total vocabulary: ', len(vocab))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# one hot encoded training data\nprint('Num of samples: ', tag_vecs.shape[0])\nprint('Size of one hot encoded vec (each val represents a tag): ', tag_vecs.shape[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# distribution of unique tags\nfreq_of_tags = tag_vecs.sum(axis=0).getA1() # (1, vocab_size) -> (vocab_size) i.e flatten it\ntags = vocab\n\ntag_freq = zip(tags[:5], freq_of_tags[:5])\n\nfor tag, freq in tag_freq:\n    print(tag, ':', freq)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sorted_idxs = np.argsort(- freq_of_tags) # -1: descending\n\nsorted_freqs = freq_of_tags[sorted_idxs] \nsorted_tags  = np.array(tags)[sorted_idxs]\n\nfor tag, freq in zip(sorted_tags[:5], sorted_freqs[:5]):\n    print(tag, ':', freq)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# distribution of occurances\nplt.close()\n\nplt.plot(sorted_freqs)\n\nplt.title(\"Distribution of number of times tag appeared questions\\n\")\nplt.grid()\nplt.xlabel(\"Tag idx in vocabulary\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# zoom in first 1k\nplt.close()\n\nplt.plot(sorted_freqs[:1000])\n\nplt.title(\"Distribution of number of times tag appeared questions\\n\")\nplt.grid()\nplt.xlabel(\"Tag idx in vocabulary\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# zoom in first 200\nplt.close()\n\nplt.plot(sorted_freqs[:200])\n\nplt.title(\"Distribution of number of times tag appeared questions\\n\")\nplt.grid()\nplt.xlabel(\"Tag idx in vocabulary\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# zoom in first 100\nplt.close()\n\n# quantiles with 0.05 difference\nplt.scatter(x=list(range(0,100,5)), y=sorted_freqs[0:100:5], c='orange', label=\"quantiles with 0.05 intervals\")\n# quantiles with 0.25 difference\nplt.scatter(x=list(range(0,100,25)), y=sorted_freqs[0:100:25], c='m', label = \"quantiles with 0.25 intervals\")\n\nfor x,y in zip(list(range(0,100,25)), sorted_freqs[0:100:25]):\n    plt.annotate(s=\"({} , {})\".format(x,y), xy=(x,y), xytext=(x-0.05, y+500))\n    \n#for x,y in zip(list(range(0,100,5)), sorted_freqs[0:100:5]):\n#    plt.annotate(s=\"({} , {})\".format(x,y), xy=(x,y), xytext=(x-0.05, y+500))\n\nx=100\ny=sorted_freqs[100]\nplt.annotate(s=\"({} , {})\".format(x,y), xy=(x,y), xytext=(x-0.05, y+500))\n\n\nplt.plot(sorted_freqs[:100])\n\nplt.legend()\nplt.grid()\n\nplt.title(\"Distribution of top 100 tags\\n\")\nplt.xlabel(\"Tag idx in vocabulary\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()\n\n\n# ---------------------------------------------------------------------\n# PDF AND CDF\nplt.close()\nplt.figure(figsize=(10,10))\n\nplt.subplot(211)\ncounts, bin_edges = np.histogram(sorted_freqs, bins=100, \n                                 density = True)\npdf = counts/(sum(counts))\n#print(pdf);\n#print(bin_edges)\ncdf = np.cumsum(pdf)\n\nplt.title(\"CDF all tags\\n\")\nplt.xlabel(\"Freq of tag occurances\")\nplt.ylabel(\"Percent of Tags out of all tags\")\nplt.grid()\n\nplt.plot(bin_edges[1:], cdf)\n\n# -------------\nplt.subplot(212)\ncounts, bin_edges = np.histogram(sorted_freqs[:100], bins=100, \n                                 density = True)\npdf = counts/(sum(counts))\n#print(pdf);\n#print(bin_edges)\ncdf = np.cumsum(pdf)\n\n#plt.title(\"CDF top 100 tags\\n\")\nplt.xlabel(\"Freq of to 100 tag occurances\")\nplt.ylabel(\"Percent of Tags ut of 100 tags\")\nplt.grid()\n\nplt.plot(bin_edges[1:], cdf)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"| | **0BSERVATION** | **INFERENCE** |\n|---|---|---|\n|1| Top 100 tags appear atleast 13k times. Of which top 15 tags appear 100k times | Most of our tags i.e 420,6207 tags occur less than 13k(max value) and only 15 tags occur more than 100k times. (Huge difference). <br> We may easily overfit on top 15 tags(Analyze these tags) |\n|2| 98.8% of all tags' frequencies occur insignificantly  | Only 2.2% of tags occur more frequently <br> Highly imbalanced. Micro F1 might be good choice but if model predicts 2.2% with high precision and high recall, we wont be able to handle class imbalance |","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# visulaize all tags wrt their frequencies\nfrom wordcloud import WordCloud\n\n# input is (tag, fre) tuple\ntup = dict(zip(sorted_tags, sorted_freqs))\n\n#Initializing WordCloud using frequencies of tags.\nwordcloud = WordCloud(    background_color='black',\n                          width=1600,\n                          height=800,\n                    ).generate_from_frequencies(tup)\n\nfig = plt.figure(figsize=(30,20))\nplt.imshow(wordcloud)\nplt.axis('off')\nplt.tight_layout(pad=0)\nfig.savefig(\"tag.png\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Observation:**\n\n- Most of questions are from CS/IT domain\n- Common tags include\n    - Java\n    - C#\n    - PHP\n    - Javascript\n    - Android\n    - python\n    - JQuery\n    - ASP.NET\n\n\n**Inference:**\n\n- Most of frequent tags are *progamming languages* ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Analyze Top 20 Tags (Occur more than 50k times each)**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.close()\nplt.figure(figsize=(20, 5))\n\nplt.bar(sorted_tags[:20], sorted_freqs[:20])\n\nplt.xlabel('Top 20 Tags')\nplt.ylabel('Counts')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Most common tags are either\n    1. Programming langs: c#, java, php, js etc.\n    2. OS: android, iphone, ios, linux (windows exists in word cloud)\n\n- No significance shown for other non-tech domains\n- With time, popularity of programming langs may change","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}