{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Problem Statemtent\n\n* Suggest the tags based on the content that was there in the question posted on Stackoverflow.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Real World / Business Objectives and Constraints\n* Predict as many tags as possible with high precision and recall.\n* Incorrect tags could impact customer experience on StackOverflow.\n* No strict latency constraints.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Performance metric\n* Micro-Averaged F1-Score (Mean F Score) : The F1 score can be interpreted as a weighted average of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal. The formula for the F1 score is:\n\nF1 = 2 * (precision * recall) / (precision + recall)\n\n* In the multi-class and multi-label case, this is the weighted average of the F1 score of each class.\n\n'Micro f1 score':\nCalculate metrics globally by counting the total true positives, false negatives and false positives. This is a better metric when we have class imbalance.\n\n'Macro f1 score':\nCalculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.\n\n* Hamming loss : The Hamming loss is the fraction of labels that are incorrectly predicted.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Type of Machine Learning Problem\n* It is a multi-label classification problem\n* Multi-label Classification: Multilabel classification assigns to each sample a set of target labels. This can be thought as predicting properties of a data-point that are not mutually exclusive, such as topics that are relevant for a document. A question on Stackoverflow might be about any of C, Pointers, FileIO and/or memory-management at the same time or none of these.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Importing Required Libraries\nimport re\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nsns.set()\nfrom tqdm import tqdm\nfrom wordcloud import WordCloud, STOPWORDS\n\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom nltk.corpus import stopwords\nfrom nltk.tokenize import word_tokenize\nfrom nltk.stem.snowball import SnowballStemmer\n\nfrom sklearn.multiclass import OneVsRestClassifier\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn import metrics\nfrom sklearn.metrics import f1_score,precision_score,recall_score\nfrom sklearn import svm\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.naive_bayes import GaussianNB\nfrom datetime import datetime\nfrom skmultilearn.adapt import mlknn\nfrom skmultilearn.problem_transform import ClassifierChain\nfrom skmultilearn.problem_transform import BinaryRelevance\nfrom skmultilearn.problem_transform import LabelPowerset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reading Dataset\ntrain = pd.read_csv(\"/kaggle/input/facebook-recruiting-iii-keyword-extraction/Train.zip\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Commented code is Working but not model is not trained by kaggle due to computing power..!! \n* Move Down for Working Code.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train.drop_duplicates('Title', inplace = True)\n# train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train.dropna(inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train['Tag_Count'] = train['Tags'].apply(lambda x: len(str(x).split()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train['Tag_Count'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fig = plt.figure(figsize = [10, 7])\n# sns.countplot(train['Tag_Count'])\n# plt.title('Distribution of tag count')\n# plt.ylabel('Frequency')\n# plt.xlabel('Tag count')\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# cvect = CountVectorizer(tokenizer = lambda x: str(x).split())\n# X = cvect.fit_transform(train['Tags'])\n# X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# X.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# tag_names = cvect.get_feature_names()\n# print(len(tag_names), tag_names[:10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# tag_freq = X.sum(axis = 0)\n# type(tag_freq), tag_freq.A1.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(tag_freq)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# tag_freq_df = pd.Series(tag_freq.A1, index = tag_names)\n# tag_freq_df.sort_values(ascending = False, inplace = True)\n# tag_freq_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # plot distribution of tag frequency\n# fig = plt.figure(figsize=[10,7])\n# plt.plot(tag_freq_df.values,\n#          c = sns.xkcd_rgb['greenish cyan'])\n# plt.title('Tag frequency distribution')\n# plt.ylabel('Frequency')\n# plt.xlabel('Tag ID')\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # plot distribution of tag frequency\n# fig = plt.figure(figsize=[10,7])\n# plt.plot(tag_freq_df.iloc[:1000].values,\n#          c = sns.xkcd_rgb['greenish cyan'])\n# plt.title('Tag frequency distribution for first 1k values')\n# plt.ylabel('Frequency')\n# plt.xlabel('Tag ID')\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # plot distribution of tag frequency\n# fig = plt.figure(figsize=[10,7])\n# plt.plot(tag_freq_df.iloc[:500].values,\n#          c = sns.xkcd_rgb['greenish cyan'])\n# plt.title('Tag frequency distribution for first 500 values')\n# plt.ylabel('Frequency')\n# plt.xlabel('Tag ID')\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fig = plt.figure(figsize = [14,7])\n\n# plt.plot(tag_freq_df.iloc[:100].values,\n#          c = sns.xkcd_rgb['greenish cyan'])\n# plt.scatter(x = list(range(0,100,5)), y = tag_freq_df.iloc[0:100:5], c = 'orange', label = \"quantiles with 0.05 intervals\")\n\n# # quantiles with 0.25 difference\n# plt.scatter(x = list(range(0,100,25)), y = tag_freq_df.iloc[0:100:25], c = 'm', label = \"quantiles with 0.25 intervals\")\n\n# for x,y in zip(list(range(0,100,25)), tag_freq_df.iloc[0:100:25]):\n#     plt.annotate(s = \"({} , {})\".format(x,y), xy = (x,y))\n\n# plt.title('first 100 tags: Distribution of number of times tag appeared questions')\n# plt.grid()\n# plt.xlabel(\"Tag number\")\n# plt.ylabel(\"Number of times tag appeared\")\n# plt.legend()\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# tag_freq_df.index.name = 'Tag'\n# tag_freq_df = tag_freq_df.reset_index(name = 'Counts')\n# tag_freq_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Store tags greater than 10K in one list\n# lst_tags_gt_10k = tag_freq_df[tag_freq_df.Counts>10000].Tag\n\n# #Print the length of the list\n# print ('{} Tags are used more than 10000 times'.format(len(lst_tags_gt_10k)))\n\n# # Store tags greater than 100K in one list\n# lst_tags_gt_100k = tag_freq_df[tag_freq_df.Counts>100000].Tag\n\n# #Print the length of the list.\n# print ('{} Tags are used more than 100000 times'.format(len(lst_tags_gt_100k)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# data = dict(dict(zip(tag_freq_df.Tag, tag_freq_df.Counts)).items())\n# wordcloud = WordCloud(    background_color='black',\n#                           width=1600,\n#                           height=800,\n#                     ).generate_from_frequencies(data)\n\n# fig = plt.figure(figsize=(30,20))\n# plt.imshow(wordcloud)\n# plt.axis('off')\n# plt.tight_layout(pad=0)\n# fig.savefig(\"tag.png\")\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Plot top 30 tags\n# fig = plt.figure(figsize=[20,10])\n# sns.barplot(x = tag_freq_df.iloc[:50].Tag,\n#             y = tag_freq_df.iloc[:50].Counts,\n#            color = sns.xkcd_rgb['greenish cyan'])\n# plt.title('Frequency of top 50 Tags')\n# plt.xlabel('Tags')\n# plt.ylabel('Frequency')\n# plt.xticks(rotation=90)\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Preprocessing**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# stopwords_list = set(stopwords.words('english'))\n# stemmer = SnowballStemmer(\"english\")\n\n# def cleanhtml(data):\n#     data = data.lower()\n#     cleanr = re.compile('<.*?>')\n#     cleantext = re.sub(cleanr, ' ', str(data))\n#     return cleantext\n\n# def cleanpunc(sentence):\n#     pattern = re.compile('[^A-Za-z]+')\n#     sentence = re.sub(pattern, ' ', sentence)\n#     return sentence","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %%time\n\n# postWithCode = 0\n# lenPreProcessed = 0\n# lenPostProcessed = 0\n# questionProcessed = 0\n# final_data = []\n# for index, (row) in tqdm(enumerate(train[:1000000].itertuples())):\n#     is_code = 0\n    \n#     title = row.Title\n#     body = row.Body\n#     tags = row.Tags\n    \n#     if '<code>' in body:\n#         postWithCode += 1\n#         is_code = 1\n#     x = len(body) + len(title)\n#     lenPreProcessed += x\n    \n#     code = str(re.findall(r'<code>(.*?)</code>', body, flags = re.DOTALL))\n#     cleaned_body = re.sub('<code>(.*?)</code>', '', body, flags=re.MULTILINE|re.DOTALL) \n    \n#     CompleteText = str(title) + ' ' + str(cleaned_body)\n    \n#     CompleteText_ch = cleanhtml(CompleteText)\n#     CompleteText_cp = cleanpunc(CompleteText_ch)\n#     word_list = word_tokenize(CompleteText_cp)\n    \n#     text = ' '.join(str(stemmer.stem(word)) for word in word_list if word not in stopwords_list and (len(word) != 1 \\\n#                                                                                                      or word == 'c'))\n    \n#     lenPostProcessed += len(text)\n#     tup = (text, code, tags, x, len(text), is_code)    \n#     final_data.append(tup)\n\n#     questionProcessed += 1\n#     if questionProcessed%100000 == 0:\n#         print(\"number of questions completed = \",questionProcessed)\n\n# avglenPreProcessed = (lenPreProcessed*1.0)/questionProcessed\n# avglenPostProcessed = (lenPostProcessed*1.0)/questionProcessed\n\n# print( \"Avg. length of questions(Title+Body) before processing: %d\"%avglenPreProcessed)\n# print( \"Avg. length of questions(Title+Body) after processing: %d\"%avglenPostProcessed)\n# print (\"Percent of questions containing code: %d\"%((postWithCode*100.0)/questionProcessed))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df = pd.DataFrame(final_data, columns=[\"Text\", \"code\", \"tags\", \"lenPreprocessed\", \"lenPostprocessed\", \"iscode\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df.to_pickle(\"cleaned_data.pkl\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df1 = pd.read_pickle(\"/kaggle/input/cleaned-file-stp/cleaned_data.pkl\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df1.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df1.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df1.drop(['code', 'lenPostprocessed', 'lenPreprocessed', 'iscode'], axis = 1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Machine Learning Models**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Converting Tags For Multilabel Problem","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# vectorizer = CountVectorizer(tokenizer = lambda x: x.split(), binary ='true')\n# multilabel_y = vectorizer.fit_transform(df1['tags'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# def tags_to_choose(n):\n#     t = multilabel_y.sum(axis = 0).tolist()[0]\n#     sorted_tags_i = sorted(range(len(t)), key = lambda i: t[i], reverse = True)\n#     multilabel_yn = multilabel_y[:, sorted_tags_i[:n]]\n#     return multilabel_yn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# def question_explained_fn(n):\n#     multilabel_yn = tags_to_choose(n)\n#     x = multilabel_yn.sum(axis = 1)\n#     return (np.count_nonzero(x == 0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# question_explained = []\n# total_tags = multilabel_y.shape[1]\n# total_ques = df1.shape[0]\n# for i in range(500, total_tags, 100):\n#     question_explained.append(np.round(((total_ques - question_explained_fn(i))/ total_ques)*100, 3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fig, ax = plt.subplots()\n# ax.plot(question_explained)\n# xlabel = list(500 + np.array(range(-50,450,50))*50)\n# ax.set_xticklabels(xlabel)\n# plt.xlabel(\"Number of tags\")\n# plt.ylabel(\"percentage of Questions coverd partially\")\n# plt.grid()\n# plt.show()\n# # you can choose any number of tags based on your computing power, minimun is 50(it covers 90% of the tags)\n# print(\"with \",5500,\"tags we are covering \",question_explained[50],\"% of questions\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# multilabel_yx = tags_to_choose(5500)\n# print(\"number of questions that are not covered :\", question_explained_fn(5500),\"out of \", total_ques)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(\"Number of tags in sample :\", multilabel_y.shape[1])\n# print(\"number of tags taken :\", multilabel_yx.shape[1],\"(\",(multilabel_yx.shape[1]/multilabel_y.shape[1])*100,\"%)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We consider top 15% tags which covers 99% of the questions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Splitting Data into train and test\n# total_size = df1.shape[0]\n# train_size = int(0.80*total_size)\n\n# x_train = df1.head(train_size)\n# x_test = df1.tail(total_size - train_size)\n\n# y_train = multilabel_yx[0 : train_size,:]\n# y_test = multilabel_yx[train_size : total_size,:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(\"Number of data points in train data :\", y_train.shape)\n# print(\"Number of data points in test data :\", y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Featurizing Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# vectorizer = TfidfVectorizer(min_df = 0.00009, max_features = 200000, smooth_idf = True, norm = \"l2\", \\\n#                              tokenizer = lambda x: x.split(), sublinear_tf = False, ngram_range = (1,3))\n# x_train_multilabel = vectorizer.fit_transform(x_train['Text'])\n# x_test_multilabel = vectorizer.transform(x_test['Text'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(\"Dimensions of train data X:\",x_train_multilabel.shape, \"Y :\",y_train.shape)\n# print(\"Dimensions of test data X:\",x_test_multilabel.shape,\"Y:\",y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"SGD Classifier with OnevsRest Classifier","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# classifier = OneVsRestClassifier(SGDClassifier(loss='log', alpha=0.00001, penalty='l1'), n_jobs=-1)\n# classifier.fit(x_train_multilabel, y_train)\n# predictions = classifier.predict(x_test_multilabel)\n\n# print(\"accuracy :\",metrics.accuracy_score(y_test,predictions))\n# print(\"macro f1 score :\",metrics.f1_score(y_test, predictions, average = 'macro'))\n# print(\"micro f1 scoore :\",metrics.f1_score(y_test, predictions, average = 'micro'))\n# print(\"hamming loss :\",metrics.hamming_loss(y_test,predictions))\n# print(\"Precision recall report :\\n\",metrics.classification_report(y_test, predictions))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The Above Code has taken title with Body also and could not be trained due to limited resource.\n* Kaggle kernels doesn't have sufficient memory to train 5000 (tags to predict) models\n* If you guys have good machine you can try to train 5000 models on it and check their performance\n* you can also try other models like logistic regression, SVM etc.\n* i am going for first 100 tags.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Using Text as Title only...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.drop_duplicates(['Title'], inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['Tag_Count'] = train['Tags'].apply(lambda x : len(str(x).split()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.dropna(inplace = True)\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot distribution of tag count\nfig = plt.figure(figsize=[10,7])\nsns.countplot(train['Tag_Count'])\nplt.title('Distribution of tag count')\nplt.ylabel('Frequency')\nplt.xlabel('Tag count')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tag_vectorizer = CountVectorizer(tokenizer = lambda x : str(x).split())\ntag_mat = tag_vectorizer.fit_transform(train['Tags'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tag_mat.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tag_names = tag_vectorizer.get_feature_names()\nlen(tag_names)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tag_freq = tag_mat.sum(axis = 0)\ntag_freq.A1.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tag_freq_ser = pd.Series(tag_freq.A1, index = tag_names)\ntag_freq_ser.sort_values(ascending  = False, inplace = True)\ntag_freq_ser.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot distribution of tag frequency\nfig = plt.figure(figsize=[10,7])\nplt.plot(tag_freq_ser.values,\n         c=sns.xkcd_rgb['greenish cyan'])\nplt.title('Tag frequency distribution')\nplt.ylabel('Frequency')\nplt.xlabel('Tag ID')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot distribution of tag frequency\nfig = plt.figure(figsize=[10,7])\nplt.plot(tag_freq_ser[:1000].values,\n         c=sns.xkcd_rgb['greenish cyan'])\nplt.title('Tag frequency distribution of First 1000 tags')\nplt.ylabel('Frequency')\nplt.xlabel('Tag ID')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot distribution of tag frequency\nfig = plt.figure(figsize=[10,7])\nplt.plot(tag_freq_ser[:500].values,\n         c=sns.xkcd_rgb['greenish cyan'])\nplt.title('Tag frequency distribution of First 500 tags')\nplt.ylabel('Frequency')\nplt.xlabel('Tag ID')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot distribution of tag frequency for Top 100 Tags\nfig = plt.figure(figsize=[10,7])\nplt.plot(tag_freq_ser[:100].values,\n         c=sns.xkcd_rgb['greenish cyan'])\nplt.title('Tag frequency distribution of First 100 Tags')\nplt.ylabel('Frequency')\nplt.xlabel('Tag ID')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot distribution of tag frequency for Top 100 Tags\nfig = plt.figure(figsize = [14,7])\n\nplt.plot(tag_freq_ser.iloc[:100].values,\n         c = sns.xkcd_rgb['greenish cyan'])\nplt.scatter(x = list(range(0,100,5)), y = tag_freq_ser.iloc[0:100:5], c = 'orange', label = \"quantiles with 0.05 intervals\")\n\n# quantiles with 0.25 difference\nplt.scatter(x = list(range(0,100,25)), y = tag_freq_ser.iloc[0:100:25], c = 'm', label = \"quantiles with 0.25 intervals\")\n\nfor x,y in zip(list(range(0,100,25)), tag_freq_ser.iloc[0:100:25]):\n    plt.annotate(s = \"({} , {})\".format(x,y), xy = (x,y))\n\nplt.title('first 100 tags: Distribution of number of times tag appeared questions')\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot word count for tags\nwordcloud = WordCloud(background_color='black', height = 800 , width = 1600,\n                      max_words=200).generate_from_frequencies(tag_freq_ser)\nfig = plt.figure(figsize=[30,20])\nplt.title('WordCloud of Tags')\nplt.axis('off')\nplt.imshow(wordcloud)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot top 30 tags\nfig = plt.figure(figsize=[20,10])\nsns.barplot(x=tag_freq_ser.iloc[:50].index,\n            y=tag_freq_ser.iloc[:50].values,\n           color=sns.xkcd_rgb['greenish cyan'])\nplt.title('Frequency of top 50 Tags')\nplt.xlabel('Tags')\nplt.ylabel('Frequency')\nplt.xticks(rotation=90)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data preprocessing**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def clean_text(sentence):\n    pattern = re.compile(r'[^a-z]+')\n    sentence = sentence.lower()\n    sentence = pattern.sub(' ', sentence).strip()\n    \n    # Tokenize\n    word_list = word_tokenize(sentence)\n    stopwords_list = set(stopwords.words('english'))\n    text = ' '.join(str(stemmer.stem(word)) for word in word_list if word not in stopwords_list and (len(word) != 1 \\\n                                                                                                     or word == 'c'))    \n    return text\n\n# create tqdm for pandas\ntqdm.pandas()\n# clean text data\nstemmer = SnowballStemmer(\"english\")\ntrain['Title'] = train['Title'].progress_apply(lambda x: clean_text(str(x)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.drop(['Body'], axis = 1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train.to_pickle(\"cleaned_data-1.pkl\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_pickle(\"/kaggle/input/cleaned-file-stp-1/cleaned_data-1.pkl\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Machine Learning Models ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Reduce Number of Tags**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tag_vectorizer = CountVectorizer(tokenizer = lambda x : str(x).split(), binary = True)\nmultinomial_y = tag_vectorizer.fit_transform(train['Tags'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def tags_to_choose(n):\n    t = multinomial_y.sum(axis = 0).tolist()[0]\n    sorted_tags = sorted(range(len(t)), key = lambda i : t[i], reverse = True)\n    multinomial_yn = multinomial_y[: , sorted_tags[:n]]\n    return multinomial_yn\n\ndef questions_explained_fn(n):\n    multinomial_yn = tags_to_choose(n)\n    x = multinomial_yn.sum(axis = 1)\n    return (np.count_nonzero(x == 0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_explained = []\ntotal_ques = multinomial_y.shape[0]\ntotal_tags = multinomial_y.shape[1]\nfor i in range(500, total_tags, 100):\n    questions_explained.append(np.round(((total_ques - questions_explained_fn(i))/total_ques)*100,3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots()\nax.plot(questions_explained)\nxlabel = list(500 + np.array(range( -50, 450, 50)) * 50)\nax.set_xticklabels(xlabel)\nplt.xlabel(\"Number of tags\")\nplt.ylabel(\"percentage of Questions coverd partially\")\nplt.grid()\nplt.show()\n# you can choose any number of tags based on your computing power, minimun is 50(it covers 90% of the tags)\nprint(\"with \",5500,\"tags we are covering \",questions_explained[50],\"% of questions\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Taking First 100 top tags\nmultinomial_yx = tags_to_choose(100)\nprint(\"number of questions that are not covered :\", questions_explained_fn(5500),\"out of \", total_ques)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of tags in sample :\", multinomial_y.shape[1])\nprint(\"number of tags taken :\", multinomial_yx.shape[1],\"(\",(multinomial_yx.shape[1]/multinomial_y.shape[1])*100,\"%)\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print percent of question covered with number of tags\nprint('#Tags\\t%Ques')\nfor idx in range(500, 7500, 500):\n    print(idx, '\\t', questions_explained[int(idx/100)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Splitting train and test data\ntotal_size = train.shape[0]\ntrain_size = int(0.80*total_size)\n\nx_train = train.head(train_size)\nx_test = train.tail(total_size - train_size)\n\ny_train = multinomial_yx[0:train_size,:]\ny_test = multinomial_yx[train_size:total_size,:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of data points in train data :\", y_train.shape)\nprint(\"Number of data points in test data :\", y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Featurizing Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"start = datetime.now()\nvectorizer = TfidfVectorizer(min_df=0.00009, max_features=200000, smooth_idf=True, norm=\"l2\", \\\n                             tokenizer = lambda x: x.split(), sublinear_tf=False, ngram_range=(1,3))\nx_train_multilabel = vectorizer.fit_transform(x_train['Title'])\nx_test_multilabel = vectorizer.transform(x_test['Title'])\nprint(\"Time taken to run this cell :\", datetime.now() - start)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Dimensions of train data X:\",x_train_multilabel.shape, \"Y :\",y_train.shape)\nprint(\"Dimensions of test data X:\",x_test_multilabel.shape,\"Y:\",y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Model training**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"SGDClassifier one vs rest","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"classifier = OneVsRestClassifier(SGDClassifier(loss='log', alpha=0.00001, penalty='l1'), n_jobs=-1)\nclassifier.fit(x_train_multilabel, y_train)\npredictions = classifier.predict(x_test_multilabel)\n\nprint(\"accuracy :\",metrics.accuracy_score(y_test,predictions))\nprint(\"macro f1 score :\",metrics.f1_score(y_test, predictions, average = 'macro'))\nprint(\"micro f1 scoore :\",metrics.f1_score(y_test, predictions, average = 'micro'))\nprint(\"hamming loss :\",metrics.hamming_loss(y_test,predictions))\nprint(\"Precision recall report :\\n\",metrics.classification_report(y_test, predictions))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We had got pretty decent results only with 100 tags\n* if you have more computing power try with more number of tags.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}