{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Stack Overflow: Tag Prediction"},{"metadata":{},"cell_type":"markdown","source":"

1. Business Problem

"},{"metadata":{},"cell_type":"markdown","source":"

1.1 Description

"},{"metadata":{},"cell_type":"markdown","source":"

Description

\n

\nStack Overflow is the largest, most trusted online community for developers to learn, share their programming knowledge, and build their careers.
\n
\nStack Overflow is something which every programmer use one way or another. Each month, over 50 million developers come to Stack Overflow to learn, share their knowledge, and build their careers. It features questions and answers on a wide range of topics in computer programming. The website serves as a platform for users to ask and answer questions, and, through membership and active participation, to vote questions and answers up or down and edit questions and answers in a fashion similar to a wiki or Digg. As of April 2014 Stack Overflow has over 4,000,000 registered users, and it exceeded 10,000,000 questions in late August 2015. Based on the type of tags assigned to questions, the top eight most discussed topics on the site are: Java, JavaScript, C#, PHP, Android, jQuery, Python and HTML.
\n
\n

"},{"metadata":{},"cell_type":"markdown","source":"

Problem Statemtent

\nSuggest the tags based on the content that was there in the question posted on Stackoverflow."},{"metadata":{},"cell_type":"markdown","source":"

Source: https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/

\n"},{"metadata":{},"cell_type":"markdown","source":"

1.3 Real World / Business Objectives and Constraints

"},{"metadata":{},"cell_type":"markdown","source":"1. Predict as many tags as possible with high precision and recall.\n2. Incorrect tags could impact customer experience on StackOverflow.\n3. No strict latency constraints."},{"metadata":{},"cell_type":"markdown","source":"

2. Machine Learning problem

"},{"metadata":{},"cell_type":"markdown","source":"

2.1 Data

"},{"metadata":{},"cell_type":"markdown","source":"

2.1.1 Data Overview

"},{"metadata":{},"cell_type":"markdown","source":"All of the data is in 2 files: Train and Test.
\n
\nTrain.csv contains 4 columns: Id,Title,Body,Tags.
\nTest.csv contains the same columns but without the Tags, which you are to predict.
\nSize of Train.csv - 6.75GB
\nSize of Test.csv - 2GB
\nNumber of rows in Train.csv = 6034195
\n
\nThe questions are randomized and contains a mix of verbose text sites as well as sites related to math and programming. The number of questions from each site may vary, and no filtering has been performed on the questions (such as closed questions).
\n
\n"},{"metadata":{},"cell_type":"markdown","source":"__Data Field Explaination__\n\nDataset contains 6,034,195 rows. The columns in the table are:
\n
\nId - Unique identifier for each question
\nTitle - The question's title
\nBody - The body of the question
\nTags - The tags associated with the question in a space-seperated format (all lowercase, should not contain tabs '\\t' or ampersands '&')
\n
\n\n
"},{"metadata":{},"cell_type":"markdown","source":"

2.2 Mapping the real-world problem to a Machine Learning Problem

"},{"metadata":{},"cell_type":"markdown","source":"

2.2.1 Type of Machine Learning Problem

"},{"metadata":{},"cell_type":"markdown","source":"

It is a multi-label classification problem.
\nMulti-label Classification: Multilabel classification assigns to each sample a set of target labels. This can be thought as predicting properties of a data-point that are not mutually exclusive, such as topics that are relevant for a document. A question on Stackoverflow might be about any of C, Pointers, FileIO and/or memory-management at the same time or none of these. \n

"},{"metadata":{},"cell_type":"markdown","source":"

2.2.2 Performance metric

"},{"metadata":{},"cell_type":"markdown","source":"Micro-Averaged F1-Score (Mean F Score) : \nThe F1 score can be interpreted as a weighted average of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal. The formula for the F1 score is:\n\nF1 = 2 * (precision * recall) / (precision + recall)
\n\nIn the multi-class and multi-label case, this is the weighted average of the F1 score of each class.
\n\n'Micro f1 score':
\nCalculate metrics globally by counting the total true positives, false negatives and false positives. This is a better metric when we have class imbalance.\n
\n\n'Macro f1 score':
\nCalculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.\n\n Hamming loss : The Hamming loss is the fraction of labels that are incorrectly predicted.
"},{"metadata":{},"cell_type":"markdown","source":"

3. Exploratory Data Analysis

"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\nimport pandas as pd\nimport sqlite3\nimport csv\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\nfrom wordcloud import WordCloud\nimport re\nimport os\nfrom sqlalchemy import create_engine # database connection\nimport datetime as dt\nfrom nltk.corpus import stopwords\nfrom nltk.tokenize import word_tokenize\nfrom nltk.stem.snowball import SnowballStemmer\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.multiclass import OneVsRestClassifier\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn import metrics\nfrom sklearn.metrics import f1_score,precision_score,recall_score\nfrom sklearn import svm\nfrom sklearn.linear_model import LogisticRegression\nfrom skmultilearn.adapt import mlknn\nfrom skmultilearn.problem_transform import ClassifierChain\nfrom skmultilearn.problem_transform import BinaryRelevance\nfrom skmultilearn.problem_transform import LabelPowerset\nfrom sklearn.naive_bayes import GaussianNB\nfrom datetime import datetime","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

3.1 Data Loading and Cleaning

"},{"metadata":{},"cell_type":"markdown","source":"

3.1.1 Using Pandas with SQLite to Load the data

"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"#Creating db file from csv\n\nif not os.path.isfile('train.db'):\n start = datetime.now()\n disk_engine = create_engine('sqlite:///train.db')\n start = dt.datetime.now()\n chunksize = 180000\n j = 0\n index_start = 1\n for df in pd.read_csv('Train.csv', names=['Id', 'Title', 'Body', 'Tags'], chunksize=chunksize, iterator=True, encoding='utf-8', ):\n df.index += index_start\n j+=1\n print('{} rows'.format(j*chunksize))\n df.to_sql('data', disk_engine, if_exists='append')\n index_start = df.index[-1] + 1\n print(\"Time taken to run this cell :\", datetime.now() - start)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

3.1.2 Counting the number of rows

"},{"metadata":{"trusted":false},"cell_type":"code","source":"if os.path.isfile('train.db'):\n start = datetime.now()\n con = sqlite3.connect('train.db')\n num_rows = pd.read_sql_query(\"\"\"SELECT count(*) FROM data\"\"\", con)\n print(\"Number of rows in the database :\",\"\\n\",num_rows['count(*)'].values[0])\n con.close()\n print(\"Time taken to count the number of rows :\", datetime.now() - start)\nelse:\n print(\"Please download the train.db file from drive or run the above cell to genarate train.db file\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

3.1.3 Checking for duplicates

"},{"metadata":{"trusted":false},"cell_type":"code","source":"if os.path.isfile('train.db'):\n start = datetime.now()\n con = sqlite3.connect('train.db')\n df_no_dup = pd.read_sql_query('SELECT Title, Body, Tags, COUNT(*) as cnt_dup FROM data GROUP BY Title, Body, Tags', con)\n con.close()\n print(\"Time taken to run this cell :\", datetime.now() - start)\nelse:\n print(\"Please download the train.db file from drive or run the first to genarate train.db file\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"df_no_dup.head()\n# we can observe that there are duplicates","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(\"number of duplicate questions :\", num_rows['count(*)'].values[0]- df_no_dup.shape[0], \"(\",(1-((df_no_dup.shape[0])/(num_rows['count(*)'].values[0])))*100,\"% )\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# number of times each question appeared in our database\ndf_no_dup.cnt_dup.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"start = datetime.now()\ndf_no_dup[\"tag_count\"] = df_no_dup[\"Tags\"].apply(lambda text: len(text.split(\" \")))\n# adding a new feature number of tags per question\nprint(\"Time taken to run this cell :\", datetime.now() - start)\ndf_no_dup.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# distribution of number of tags per question\ndf_no_dup.tag_count.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"#Creating a new database with no duplicates\nif not os.path.isfile('train_no_dup.db'):\n disk_dup = create_engine(\"sqlite:///train_no_dup.db\")\n no_dup = pd.DataFrame(df_no_dup, columns=['Title', 'Body', 'Tags'])\n no_dup.to_sql('no_dup_train',disk_dup)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#This method seems more appropriate to work with this much data.\nif os.path.isfile('train_no_dup.db'):\n start = datetime.now()\n con = sqlite3.connect('train_no_dup.db')\n tag_data = pd.read_sql_query(\"\"\"SELECT Tags FROM no_dup_train\"\"\", con)\n con.close()\n\n # Let's now drop unwanted column.\n tag_data.drop(tag_data.index[0], inplace=True)\n #Printing first 5 columns from our data frame\n tag_data.head()\n print(\"Time taken to run this cell :\", datetime.now() - start)\nelse:\n print(\"Please download the train.db file from drive or run the above cells to genarate train.db file\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

3.2 Analysis of Tags

"},{"metadata":{},"cell_type":"markdown","source":"

3.2.1 Total number of unique tags

"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"# Importing & Initializing the \"CountVectorizer\" object, which is scikit-learn's bag of words tool.\n\n#by default 'split()' will tokenize each tag using space.\nvectorizer = CountVectorizer(tokenizer = lambda x: x.split())\n# fit_transform() does two functions: First, it fits the model\n# and learns the vocabulary; second, it transforms our training data\n# into feature vectors. The input to fit_transform should be a list of strings.\ntag_dtm = vectorizer.fit_transform(tag_data['Tags'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(\"Number of data points :\", tag_dtm.shape[0])\nprint(\"Number of unique tags :\", tag_dtm.shape[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#'get_feature_name()' gives us the vocabulary.\ntags = vectorizer.get_feature_names()\n#Lets look at the tags we have.\nprint(\"Some of the tags we have :\", tags[:10])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

3.2.3 Number of times a tag appeared

"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"#Lets now store the document term matrix in a dictionary.\nfreqs = tag_dtm.sum(axis=0).A1\nresult = dict(zip(tags, freqs))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#Saving this dictionary to csv files.\nif not os.path.isfile('tag_counts_dict_dtm.csv'):\n with open('tag_counts_dict_dtm.csv', 'w') as csv_file:\n writer = csv.writer(csv_file)\n for key, value in result.items():\n writer.writerow([key, value])\ntag_df = pd.read_csv(\"tag_counts_dict_dtm.csv\", names=['Tags', 'Counts'])\ntag_df.head()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"tag_df_sorted = tag_df.sort_values(['Counts'], ascending=False)\ntag_counts = tag_df_sorted['Counts'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.plot(tag_counts)\nplt.title(\"Distribution of number of times tag appeared questions\")\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.plot(tag_counts[0:10000])\nplt.title('first 10k tags: Distribution of number of times tag appeared questions')\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()\nprint(len(tag_counts[0:10000:25]), tag_counts[0:10000:25])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.plot(tag_counts[0:1000])\nplt.title('first 1k tags: Distribution of number of times tag appeared questions')\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()\nprint(len(tag_counts[0:1000:5]), tag_counts[0:1000:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.plot(tag_counts[0:500])\nplt.title('first 500 tags: Distribution of number of times tag appeared questions')\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()\nprint(len(tag_counts[0:500:5]), tag_counts[0:500:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.plot(tag_counts[0:100], c='b')\nplt.scatter(x=list(range(0,100,5)), y=tag_counts[0:100:5], c='orange', label=\"quantiles with 0.05 intervals\")\n# quantiles with 0.25 difference\nplt.scatter(x=list(range(0,100,25)), y=tag_counts[0:100:25], c='m', label = \"quantiles with 0.25 intervals\")\n\nfor x,y in zip(list(range(0,100,25)), tag_counts[0:100:25]):\n plt.annotate(s=\"({} , {})\".format(x,y), xy=(x,y), xytext=(x-0.05, y+500))\n\nplt.title('first 100 tags: Distribution of number of times tag appeared questions')\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.legend()\nplt.show()\nprint(len(tag_counts[0:100:5]), tag_counts[0:100:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Store tags greater than 10K in one list\nlst_tags_gt_10k = tag_df[tag_df.Counts>10000].Tags\n#Print the length of the list\nprint ('{} Tags are used more than 10000 times'.format(len(lst_tags_gt_10k)))\n# Store tags greater than 100K in one list\nlst_tags_gt_100k = tag_df[tag_df.Counts>100000].Tags\n#Print the length of the list.\nprint ('{} Tags are used more than 100000 times'.format(len(lst_tags_gt_100k)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observations:
\n1. There are total 153 tags which are used more than 10000 times.\n2. 14 tags are used more than 100000 times.\n3. Most frequent tag (i.e. c#) is used 331505 times.\n4. Since some tags occur much more frequenctly than others, Micro-averaged F1-score is the appropriate metric for this probelm."},{"metadata":{},"cell_type":"markdown","source":"

3.2.4 Tags Per Question

"},{"metadata":{"trusted":false},"cell_type":"code","source":"#Storing the count of tag in each question in list 'tag_count'\ntag_quest_count = tag_dtm.sum(axis=1).tolist()\n#Converting each value in the 'tag_quest_count' to integer.\ntag_quest_count=[int(j) for i in tag_quest_count for j in i]\nprint ('We have total {} datapoints.'.format(len(tag_quest_count)))\n\nprint(tag_quest_count[:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print( \"Maximum number of tags per question: %d\"%max(tag_quest_count))\nprint( \"Minimum number of tags per question: %d\"%min(tag_quest_count))\nprint( \"Avg. number of tags per question: %f\"% ((sum(tag_quest_count)*1.0)/len(tag_quest_count)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"sns.countplot(tag_quest_count, palette='gist_rainbow')\nplt.title(\"Number of tags in the questions \")\nplt.xlabel(\"Number of Tags\")\nplt.ylabel(\"Number of questions\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observations:
\n1. Maximum number of tags per question: 5\n2. Minimum number of tags per question: 1\n3. Avg. number of tags per question: 2.899\n4. Most of the questions are having 2 or 3 tags"},{"metadata":{},"cell_type":"markdown","source":"

3.2.5 Most Frequent Tags

"},{"metadata":{"scrolled":false,"trusted":false},"cell_type":"code","source":"# Ploting word cloud\nstart = datetime.now()\n\n# Lets first convert the 'result' dictionary to 'list of tuples'\ntup = dict(result.items())\n#Initializing WordCloud using frequencies of tags.\nwordcloud = WordCloud( background_color='black',\n width=1600,\n height=800,\n ).generate_from_frequencies(tup)\n\nfig = plt.figure(figsize=(30,20))\nplt.imshow(wordcloud)\nplt.axis('off')\nplt.tight_layout(pad=0)\nfig.savefig(\"tag.png\")\nplt.show()\nprint(\"Time taken to run this cell :\", datetime.now() - start)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observations:
\nA look at the word cloud shows that \"c#\", \"java\", \"php\", \"asp.net\", \"javascript\", \"c++\" are some of the most frequent tags."},{"metadata":{},"cell_type":"markdown","source":"

3.2.6 The top 20 tags

"},{"metadata":{"trusted":false},"cell_type":"code","source":"i=np.arange(30)\ntag_df_sorted.head(30).plot(kind='bar')\nplt.title('Frequency of top 20 tags')\nplt.xticks(i, tag_df_sorted['Tags'])\nplt.xlabel('Tags')\nplt.ylabel('Counts')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observations:
\n1. Majority of the most frequent tags are programming language.\n2. C# is the top most frequent programming language.\n3. Android, IOS, Linux and windows are among the top most frequent operating systems."},{"metadata":{},"cell_type":"markdown","source":"

3.3 Cleaning and preprocessing of Questions

"},{"metadata":{},"cell_type":"markdown","source":"

3.3.1 Preprocessing

"},{"metadata":{},"cell_type":"markdown","source":"
    \n
  1. Sample 1M data points
    2. \n
  2. Separate out code-snippets from Body
    3. \n
  3. Remove Spcial characters from Question title and description (not in code)
    4. \n
  4. Remove stop words (Except 'C')
    5. \n
  5. Remove HTML Tags
    6. \n
  6. Convert all the characters into small letters
    7. \n
  7. Use SnowballStemmer to stem the words
    8. \n
"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"def striphtml(data):\n cleanr = re.compile('<.*?>')\n cleantext = re.sub(cleanr, ' ', str(data))\n return cleantext\nstop_words = set(stopwords.words('english'))\nstemmer = SnowballStemmer(\"english\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def create_connection(db_file):\n \"\"\" create a database connection to the SQLite database\n specified by db_file\n :param db_file: database file\n :return: Connection object or None\n \"\"\"\n try:\n conn = sqlite3.connect(db_file)\n return conn\n except Error as e:\n print(e)\n \n return None\n\ndef create_table(conn, create_table_sql):\n \"\"\" create a table from the create_table_sql statement\n :param conn: Connection object\n :param create_table_sql: a CREATE TABLE statement\n :return:\n \"\"\"\n try:\n c = conn.cursor()\n c.execute(create_table_sql)\n except Error as e:\n print(e)\n \ndef checkTableExists(dbcon):\n cursr = dbcon.cursor()\n str = \"select name from sqlite_master where type='table'\"\n table_names = cursr.execute(str)\n print(\"Tables in the databse:\")\n tables =table_names.fetchall() \n print(tables[0][0])\n return(len(tables))\n\ndef create_database_table(database, query):\n conn = create_connection(database)\n if conn is not None:\n create_table(conn, query)\n checkTableExists(conn)\n else:\n print(\"Error! cannot create the database connection.\")\n conn.close()\n\nsql_create_table = \"\"\"CREATE TABLE IF NOT EXISTS QuestionsProcessed (question text NOT NULL, code text, tags text, words_pre integer, words_post integer, is_code integer);\"\"\"\ncreate_database_table(\"Processed.db\", sql_create_table)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"start = datetime.now()\nread_db = 'train_no_dup.db'\nwrite_db = 'Processed.db'\nif os.path.isfile(read_db):\n conn_r = create_connection(read_db)\n if conn_r is not None:\n reader =conn_r.cursor()\n reader.execute(\"SELECT Title, Body, Tags From no_dup_train ORDER BY RANDOM() LIMIT 1000000;\")\n\nif os.path.isfile(write_db):\n conn_w = create_connection(write_db)\n if conn_w is not None:\n tables = checkTableExists(conn_w)\n writer =conn_w.cursor()\n if tables != 0:\n writer.execute(\"DELETE FROM QuestionsProcessed WHERE 1\")\n print(\"Cleared All the rows\")\nprint(\"Time taken to run this cell :\", datetime.now() - start)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__ we create a new data base to store the sampled and preprocessed questions __"},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"start = datetime.now()\npreprocessed_data_list=[]\nreader.fetchone()\nquestions_with_code=0\nlen_pre=0\nlen_post=0\nquestions_proccesed = 0\nfor row in reader:\n\n is_code = 0\n\n title, question, tags = row[0], row[1], row[2]\n\n if '' in question:\n questions_with_code+=1\n is_code = 1\n x = len(question)+len(title)\n len_pre+=x\n\n code = str(re.findall(r'(.*?)', question, flags=re.DOTALL))\n\n question=re.sub('(.*?)', '', question, flags=re.MULTILINE|re.DOTALL)\n question=striphtml(question.encode('utf-8'))\n\n title=title.encode('utf-8')\n\n question=str(title)+\" \"+str(question)\n question=re.sub(r'[^A-Za-z]+',' ',question)\n words=word_tokenize(str(question.lower()))\n\n #Removing all single letter and and stopwords from question exceptt for the letter 'c'\n question=' '.join(str(stemmer.stem(j)) for j in words if j not in stop_words and (len(j)!=1 or j=='c'))\n\n len_post+=len(question)\n tup = (question,code,tags,x,len(question),is_code)\n questions_proccesed += 1\n writer.execute(\"insert into QuestionsProcessed(question,code,tags,words_pre,words_post,is_code) values (?,?,?,?,?,?)\",tup)\n if (questions_proccesed%100000==0):\n print(\"number of questions completed=\",questions_proccesed)\n\nno_dup_avg_len_pre=(len_pre*1.0)/questions_proccesed\nno_dup_avg_len_post=(len_post*1.0)/questions_proccesed\n\nprint( \"Avg. length of questions(Title+Body) before processing: %d\"%no_dup_avg_len_pre)\nprint( \"Avg. length of questions(Title+Body) after processing: %d\"%no_dup_avg_len_post)\nprint (\"Percent of questions containing code: %d\"%((questions_with_code*100.0)/questions_proccesed))\n\nprint(\"Time taken to run this cell :\", datetime.now() - start)","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"# close the connections, or else you will end up with locks\nconn_r.commit()\nconn_w.commit()\nconn_r.close()\nconn_w.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"if os.path.isfile(write_db):\n conn_r = create_connection(write_db)\n if conn_r is not None:\n reader =conn_r.cursor()\n reader.execute(\"SELECT question From QuestionsProcessed LIMIT 10\")\n print(\"Questions after preprocessed\")\n print('='*100)\n reader.fetchone()\n for row in reader:\n print(row)\n print('-'*100)\nconn_r.commit()\nconn_r.close()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"#Taking 1 Million entries to a dataframe.\nwrite_db = 'Processed.db'\nif os.path.isfile(write_db):\n conn_r = create_connection(write_db)\n if conn_r is not None:\n preprocessed_data = pd.read_sql_query(\"\"\"SELECT question, Tags FROM QuestionsProcessed\"\"\", conn_r)\nconn_r.commit()\nconn_r.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"preprocessed_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(\"number of data points in sample :\", preprocessed_data.shape[0])\nprint(\"number of dimensions :\", preprocessed_data.shape[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

4. Machine Learning Models

"},{"metadata":{},"cell_type":"markdown","source":"

4.1 Converting tags for multilabel problems

"},{"metadata":{},"cell_type":"markdown","source":"\n\n\n\n\n\n\n\n\n\n\n\n\n
Xy1y2y3y4
x10110
x11000
x10100
"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"# binary='true' will give a binary vectorizer\nvectorizer = CountVectorizer(tokenizer = lambda x: x.split(), binary='true')\nmultilabel_y = vectorizer.fit_transform(preprocessed_data['tags'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Sample the number of tags instead considering all of them (due to limitation of computing power)"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"def tags_to_choose(n):\n t = multilabel_y.sum(axis=0).tolist()[0]\n sorted_tags_i = sorted(range(len(t)), key=lambda i: t[i], reverse=True)\n multilabel_yn=multilabel_y[:,sorted_tags_i[:n]]\n return multilabel_yn\n\ndef questions_explained_fn(n):\n multilabel_yn = tags_to_choose(n)\n x= multilabel_yn.sum(axis=1)\n return (np.count_nonzero(x==0))","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"questions_explained = []\ntotal_tags=multilabel_y.shape[1]\ntotal_qs=preprocessed_data.shape[0]\nfor i in range(500, total_tags, 100):\n questions_explained.append(np.round(((total_qs-questions_explained_fn(i))/total_qs)*100,3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"fig, ax = plt.subplots()\nax.plot(questions_explained)\nxlabel = list(500+np.array(range(-50,450,50))*50)\nax.set_xticklabels(xlabel)\nplt.xlabel(\"Number of tags\")\nplt.ylabel(\"Number Questions coverd partially\")\nplt.grid()\nplt.show()\n# choose any number of tags based on the computing power, minimun is 50(it covers 90% of the tags)\nprint(\"with \",5500,\"tags we are covering \",questions_explained[50],\"% of questions\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"multilabel_yx = tags_to_choose(5500)\nprint(\"number of questions that are not covered :\", questions_explained_fn(5500),\"out of \", total_qs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(\"Number of tags in sample :\", multilabel_y.shape[1])\nprint(\"number of tags taken :\", multilabel_yx.shape[1],\"(\",(multilabel_yx.shape[1]/multilabel_y.shape[1])*100,\"%)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__ Considering top 15% tags which covers 99% of the questions __"},{"metadata":{},"cell_type":"markdown","source":"

4.2 Split the data into test and train (80:20)

"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"total_size=preprocessed_data.shape[0]\ntrain_size=int(0.80*total_size)\n\nx_train=preprocessed_data.head(train_size)\nx_test=preprocessed_data.tail(total_size - train_size)\n\ny_train = multilabel_yx[0:train_size,:]\ny_test = multilabel_yx[train_size:total_size,:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(\"Number of data points in train data :\", y_train.shape)\nprint(\"Number of data points in test data :\", y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

4.3 Featurizing data

"},{"metadata":{"trusted":false},"cell_type":"code","source":"start = datetime.now()\nvectorizer = TfidfVectorizer(min_df=0.00009, max_features=200000, smooth_idf=True, norm=\"l2\", \\\n tokenizer = lambda x: x.split(), sublinear_tf=False, ngram_range=(1,3))\nx_train_multilabel = vectorizer.fit_transform(x_train['question'])\nx_test_multilabel = vectorizer.transform(x_test['question'])\nprint(\"Time taken to run this cell :\", datetime.now() - start)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(\"Dimensions of train data X:\",x_train_multilabel.shape, \"Y :\",y_train.shape)\nprint(\"Dimensions of test data X:\",x_test_multilabel.shape,\"Y:\",y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

4.4 Applying Logistic Regression with OneVsRest Classifier

"},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"# This takes about 6-7 hours to run.\nclassifier = OneVsRestClassifier(SGDClassifier(loss='log', alpha=0.00001, penalty='l1'), n_jobs=-1)\nclassifier.fit(x_train_multilabel, y_train)\npredictions = classifier.predict(x_test_multilabel)\n\nprint(\"accuracy :\",metrics.accuracy_score(y_test,predictions))\nprint(\"macro f1 score :\",metrics.f1_score(y_test, predictions, average = 'macro'))\nprint(\"micro f1 scoore :\",metrics.f1_score(y_test, predictions, average = 'micro'))\nprint(\"hamming loss :\",metrics.hamming_loss(y_test,predictions))\nprint(\"Precision recall report :\\n\",metrics.classification_report(y_test, predictions))\n","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"from sklearn.externals import joblib\njoblib.dump(classifier, 'lr_with_equal_weight.pkl') ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

4.5 Modeling with less data points (0.5M data points) and more weight to title and 500 tags only.

"},{"metadata":{"trusted":false},"cell_type":"code","source":"sql_create_table = \"\"\"CREATE TABLE IF NOT EXISTS QuestionsProcessed (question text NOT NULL, code text, tags text, words_pre integer, words_post integer, is_code integer);\"\"\"\ncreate_database_table(\"Titlemoreweight.db\", sql_create_table)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"read_db = 'train_no_dup.db'\nwrite_db = 'Titlemoreweight.db'\ntrain_datasize = 400000\nif os.path.isfile(read_db):\n conn_r = create_connection(read_db)\n if conn_r is not None:\n reader =conn_r.cursor()\n # for selecting first 0.5M rows\n reader.execute(\"SELECT Title, Body, Tags From no_dup_train LIMIT 500001;\")\n # for selecting random points\n #reader.execute(\"SELECT Title, Body, Tags From no_dup_train ORDER BY RANDOM() LIMIT 500001;\")\n\nif os.path.isfile(write_db):\n conn_w = create_connection(write_db)\n if conn_w is not None:\n tables = checkTableExists(conn_w)\n writer =conn_w.cursor()\n if tables != 0:\n writer.execute(\"DELETE FROM QuestionsProcessed WHERE 1\")\n print(\"Cleared All the rows\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

4.5.1 Preprocessing of questions

"},{"metadata":{},"cell_type":"markdown","source":"
    \n
  1. Separate Code from Body
    2. \n
  2. Remove Spcial characters from Question title and description (not in code)
    3. \n
  3. Give more weightage to title : Add title three times to the question
    4. \n \n
  4. Remove stop words (Except 'C')
    5. \n
  5. Remove HTML Tags
    6. \n
  6. Convert all the characters into small letters
    7. \n
  7. Use SnowballStemmer to stem the words
    8. \n
"},{"metadata":{"trusted":false},"cell_type":"code","source":"start = datetime.now()\npreprocessed_data_list=[]\nreader.fetchone()\nquestions_with_code=0\nlen_pre=0\nlen_post=0\nquestions_proccesed = 0\nfor row in reader:\n \n is_code = 0\n \n title, question, tags = row[0], row[1], str(row[2])\n \n if '' in question:\n questions_with_code+=1\n is_code = 1\n x = len(question)+len(title)\n len_pre+=x\n \n code = str(re.findall(r'(.*?)', question, flags=re.DOTALL))\n \n question=re.sub('(.*?)', '', question, flags=re.MULTILINE|re.DOTALL)\n question=striphtml(question.encode('utf-8'))\n \n title=title.encode('utf-8')\n \n # adding title three time to the data to increase its weight\n # add tags string to the training data\n \n question=str(title)+\" \"+str(title)+\" \"+str(title)+\" \"+question\n \n# if questions_proccesed<=train_datasize:\n# question=str(title)+\" \"+str(title)+\" \"+str(title)+\" \"+question+\" \"+str(tags)\n# else:\n# question=str(title)+\" \"+str(title)+\" \"+str(title)+\" \"+question\n\n question=re.sub(r'[^A-Za-z0-9#+.\\-]+',' ',question)\n words=word_tokenize(str(question.lower()))\n \n #Removing all single letter and and stopwords from question exceptt for the letter 'c'\n question=' '.join(str(stemmer.stem(j)) for j in words if j not in stop_words and (len(j)!=1 or j=='c'))\n \n len_post+=len(question)\n tup = (question,code,tags,x,len(question),is_code)\n questions_proccesed += 1\n writer.execute(\"insert into QuestionsProcessed(question,code,tags,words_pre,words_post,is_code) values (?,?,?,?,?,?)\",tup)\n if (questions_proccesed%100000==0):\n print(\"number of questions completed=\",questions_proccesed)\n\nno_dup_avg_len_pre=(len_pre*1.0)/questions_proccesed\nno_dup_avg_len_post=(len_post*1.0)/questions_proccesed\n\nprint( \"Avg. length of questions(Title+Body) before processing: %d\"%no_dup_avg_len_pre)\nprint( \"Avg. length of questions(Title+Body) after processing: %d\"%no_dup_avg_len_post)\nprint (\"Percent of questions containing code: %d\"%((questions_with_code*100.0)/questions_proccesed))\n\nprint(\"Time taken to run this cell :\", datetime.now() - start)","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"# closing the connections\nconn_r.commit()\nconn_w.commit()\nconn_r.close()\nconn_w.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__ Sample quesitons after preprocessing of data __"},{"metadata":{"trusted":false},"cell_type":"code","source":"if os.path.isfile(write_db):\n conn_r = create_connection(write_db)\n if conn_r is not None:\n reader =conn_r.cursor()\n reader.execute(\"SELECT question From QuestionsProcessed LIMIT 10\")\n print(\"Questions after preprocessed\")\n print('='*100)\n reader.fetchone()\n for row in reader:\n print(row)\n print('-'*100)\nconn_r.commit()\nconn_r.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__ Saving Preprocessed data to a Database __"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"#Taking 0.5 Million entries to a dataframe.\nwrite_db = 'Titlemoreweight.db'\nif os.path.isfile(write_db):\n conn_r = create_connection(write_db)\n if conn_r is not None:\n preprocessed_data = pd.read_sql_query(\"\"\"SELECT question, Tags FROM QuestionsProcessed\"\"\", conn_r)\nconn_r.commit()\nconn_r.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"preprocessed_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(\"number of data points in sample :\", preprocessed_data.shape[0])\nprint(\"number of dimensions :\", preprocessed_data.shape[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__ Converting string Tags to multilable output variables __ "},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"vectorizer = CountVectorizer(tokenizer = lambda x: x.split(), binary='true')\nmultilabel_y = vectorizer.fit_transform(preprocessed_data['tags'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__ Selecting 500 Tags __"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"questions_explained = []\ntotal_tags=multilabel_y.shape[1]\ntotal_qs=preprocessed_data.shape[0]\nfor i in range(500, total_tags, 100):\n questions_explained.append(np.round(((total_qs-questions_explained_fn(i))/total_qs)*100,3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"fig, ax = plt.subplots()\nax.plot(questions_explained)\nxlabel = list(500+np.array(range(-50,450,50))*50)\nax.set_xticklabels(xlabel)\nplt.xlabel(\"Number of tags\")\nplt.ylabel(\"Number Questions coverd partially\")\nplt.grid()\nplt.show()\n# you can choose any number of tags based on your computing power, minimun is 500(it covers 90% of the tags)\nprint(\"with \",5500,\"tags we are covering \",questions_explained[50],\"% of questions\")\nprint(\"with \",500,\"tags we are covering \",questions_explained[0],\"% of questions\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# we will be taking 500 tags\nmultilabel_yx = tags_to_choose(500)\nprint(\"number of questions that are not covered :\", questions_explained_fn(500),\"out of \", total_qs)","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"x_train=preprocessed_data.head(train_datasize)\nx_test=preprocessed_data.tail(preprocessed_data.shape[0] - 400000)\n\ny_train = multilabel_yx[0:train_datasize,:]\ny_test = multilabel_yx[train_datasize:preprocessed_data.shape[0],:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(\"Number of data points in train data :\", y_train.shape)\nprint(\"Number of data points in test data :\", y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

4.5.2 Featurizing data with TfIdf vectorizer

"},{"metadata":{"trusted":false},"cell_type":"code","source":"start = datetime.now()\nvectorizer = TfidfVectorizer(min_df=0.00009, max_features=200000, smooth_idf=True, norm=\"l2\", \\\n tokenizer = lambda x: x.split(), sublinear_tf=False, ngram_range=(1,3))\nx_train_multilabel = vectorizer.fit_transform(x_train['question'])\nx_test_multilabel = vectorizer.transform(x_test['question'])\nprint(\"Time taken to run this cell :\", datetime.now() - start)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(\"Dimensions of train data X:\",x_train_multilabel.shape, \"Y :\",y_train.shape)\nprint(\"Dimensions of test data X:\",x_test_multilabel.shape,\"Y:\",y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

4.5.3 Applying Logistic Regression with OneVsRest Classifier

"},{"metadata":{"trusted":false},"cell_type":"code","source":"start = datetime.now()\nclassifier = OneVsRestClassifier(SGDClassifier(loss='log', alpha=0.00001, penalty='l1'), n_jobs=-1)\nclassifier.fit(x_train_multilabel, y_train)\npredictions = classifier.predict (x_test_multilabel)\n\n\nprint(\"Accuracy :\",metrics.accuracy_score(y_test, predictions))\nprint(\"Hamming loss \",metrics.hamming_loss(y_test,predictions))\n\n\nprecision = precision_score(y_test, predictions, average='micro')\nrecall = recall_score(y_test, predictions, average='micro')\nf1 = f1_score(y_test, predictions, average='micro')\n \nprint(\"Micro-average quality numbers\")\nprint(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\nprecision = precision_score(y_test, predictions, average='macro')\nrecall = recall_score(y_test, predictions, average='macro')\nf1 = f1_score(y_test, predictions, average='macro')\n \nprint(\"Macro-average quality numbers\")\nprint(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\nprint (metrics.classification_report(y_test, predictions))\nprint(\"Time taken to run this cell :\", datetime.now() - start)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"joblib.dump(classifier, 'lr_with_more_title_weight.pkl') ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"start = datetime.now()\nclassifier_2 = OneVsRestClassifier(LogisticRegression(penalty='l1'), n_jobs=-1)\nclassifier_2.fit(x_train_multilabel, y_train)\npredictions_2 = classifier_2.predict(x_test_multilabel)\nprint(\"Accuracy :\",metrics.accuracy_score(y_test, predictions_2))\nprint(\"Hamming loss \",metrics.hamming_loss(y_test,predictions_2))\n\n\nprecision = precision_score(y_test, predictions_2, average='micro')\nrecall = recall_score(y_test, predictions_2, average='micro')\nf1 = f1_score(y_test, predictions_2, average='micro')\n \nprint(\"Micro-average quality numbers\")\nprint(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\nprecision = precision_score(y_test, predictions_2, average='macro')\nrecall = recall_score(y_test, predictions_2, average='macro')\nf1 = f1_score(y_test, predictions_2, average='macro')\n \nprint(\"Macro-average quality numbers\")\nprint(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\nprint (metrics.classification_report(y_test, predictions_2))\nprint(\"Time taken to run this cell :\", datetime.now() - start)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.3"}},"nbformat":4,"nbformat_minor":1}