{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Facebook Recruiting III - Keyword Extraction\n- Hola amigos, this notebook covers my code for the **Facebook Recruiting III - Keyword Extraction** challenege, which can be found [here](https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction).","metadata":{}},{"cell_type":"markdown","source":"# Installing & Importing Packages","metadata":{}},{"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-09-22T05:26:23.828131Z","iopub.execute_input":"2021-09-22T05:26:23.828877Z","iopub.status.idle":"2021-09-22T05:26:23.867750Z","shell.execute_reply.started":"2021-09-22T05:26:23.828761Z","shell.execute_reply":"2021-09-22T05:26:23.866870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport joblib\nimport pandas as pd\nimport sqlite3\nimport csv\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\nimport re\nimport datetime as dt\n\nfrom IPython.display import Image\nfrom wordcloud import WordCloud\nfrom sqlalchemy import create_engine # Database connection\nfrom nltk.corpus import stopwords\nfrom nltk.tokenize import word_tokenize\nfrom nltk.stem.snowball import SnowballStemmer\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.multiclass import OneVsRestClassifier\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn import metrics\nfrom sklearn.metrics import f1_score,precision_score,recall_score\nfrom sklearn import svm\nfrom sklearn.linear_model import LogisticRegression\nfrom skmultilearn.adapt import mlknn\nfrom skmultilearn.problem_transform import ClassifierChain\nfrom skmultilearn.problem_transform import BinaryRelevance\nfrom skmultilearn.problem_transform import LabelPowerset\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.model_selection import GridSearchCV\nfrom datetime import datetime","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:26:23.869323Z","iopub.execute_input":"2021-09-22T05:26:23.870031Z","iopub.status.idle":"2021-09-22T05:26:26.066271Z","shell.execute_reply.started":"2021-09-22T05:26:23.869983Z","shell.execute_reply":"2021-09-22T05:26:26.065529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Stack Overflow: Tag Prediction","metadata":{}},{"cell_type":"code","source":"Image(filename=\"../input/d/elemento/facebook-recruiting-iii-keyword-extraction/title.jpg\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:26:26.067671Z","iopub.execute_input":"2021-09-22T05:26:26.068495Z","iopub.status.idle":"2021-09-22T05:26:26.109315Z","shell.execute_reply.started":"2021-09-22T05:26:26.068432Z","shell.execute_reply":"2021-09-22T05:26:26.108302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

1. Business Problem

\n\n

1.1 Description

\n

Description

\n

\nStack Overflow is the largest, most trusted online community for developers to learn, share their programming knowledge, and build their careers.
\n
\nStack Overflow is something which every programmer use one way or another. Each month, over 50 million developers come to Stack Overflow to learn, share their knowledge, and build their careers. It features questions and answers on a wide range of topics in computer programming. The website serves as a platform for users to ask and answer questions, and, through membership and active participation, to vote questions and answers up or down and edit questions and answers in a fashion similar to a wiki or Digg. As of April 2014 Stack Overflow has over 4,000,000 registered users, and it exceeded 10,000,000 questions in late August 2015. Based on the type of tags assigned to questions, the top eight most discussed topics on the site are: Java, JavaScript, C#, PHP, Android, jQuery, Python and HTML.
\n
\n

\n

Problem Statemtent

\nSuggest the tags based on the content that was there in the question posted on Stackoverflow.\n

Source: https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/

\n\n

1.2 Source / useful links

\nData Source : https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/data
\nYoutube : https://youtu.be/nNDqbUhtIRg
\nResearch paper : https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tagging-1.pdf
\nResearch paper : https://dl.acm.org/citation.cfm?id=2660970&dl=ACM&coll=DL\n\n

1.3 Real World / Business Objectives and Constraints

\n1. Predict as many tags as possible with high precision and recall.\n2. Incorrect tags could impact customer experience on StackOverflow.\n3. No strict latency constraints.","metadata":{}},{"cell_type":"markdown","source":"

2. Machine Learning problem

\n\n

2.1 Data

\n\n

2.1.1 Data Overview

\nRefer: https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/data\n
\nAll of the data is in 2 files: Train and Test.
\n
\nTrain.csv contains 4 columns: Id,Title,Body,Tags.
\nTest.csv contains the same columns but without the Tags, which you are to predict.
\nSize of Train.csv - 6.75GB
\nSize of Test.csv - 2GB
\nNumber of rows in Train.csv = 6034195
\n
\nThe questions are randomized and contains a mix of verbose text sites as well as sites related to math and programming. The number of questions from each site may vary, and no filtering has been performed on the questions (such as closed questions).
\n
\n__Data Field Explaination__\n\nDataset contains 6,034,195 rows. The columns in the table are:
\n
\nId - Unique identifier for each question
\nTitle - The question's title
\nBody - The body of the question
\nTags - The tags associated with the question in a space-seperated format (all lowercase, should not contain tabs '\\t' or ampersands '&')
\n
\n
\n\n

2.1.2 Example Data point

\n
\nTitle:  Implementing Boundary Value Analysis of Software Testing in a C++ program?\nBody : 
\n        #include<\n        iostream>\\n\n        #include<\n        stdlib.h>\\n\\n\n        using namespace std;\\n\\n\n        int main()\\n\n        {\\n\n                 int n,a[n],x,c,u[n],m[n],e[n][4];\\n         \n                 cout<<\"Enter the number of variables\";\\n         cin>>n;\\n\\n         \n                 cout<<\"Enter the Lower, and Upper Limits of the variables\";\\n         \n                 for(int y=1; y<n+1; y++)\\n         \n                 {\\n                 \n                    cin>>m[y];\\n                 \n                    cin>>u[y];\\n         \n                 }\\n         \n                 for(x=1; x<n+1; x++)\\n         \n                 {\\n                 \n                    a[x] = (m[x] + u[x])/2;\\n         \n                 }\\n         \n                 c=(n*4)-4;\\n         \n                 for(int a1=1; a1<n+1; a1++)\\n         \n                 {\\n\\n             \n                    e[a1][0] = m[a1];\\n             \n                    e[a1][1] = m[a1]+1;\\n             \n                    e[a1][2] = u[a1]-1;\\n             \n                    e[a1][3] = u[a1];\\n         \n                 }\\n         \n                 for(int i=1; i<n+1; i++)\\n         \n                 {\\n            \n                    for(int l=1; l<=i; l++)\\n            \n                    {\\n                 \n                        if(l!=1)\\n                 \n                        {\\n                    \n                            cout<<a[l]<<\"\\\\t\";\\n                 \n                        }\\n            \n                    }\\n            \n                    for(int j=0; j<4; j++)\\n            \n                    {\\n                \n                        cout<<e[i][j];\\n                \n                        for(int k=0; k<n-(i+1); k++)\\n                \n                        {\\n                    \n                            cout<<a[k]<<\"\\\\t\";\\n               \n                        }\\n                \n                        cout<<\"\\\\n\";\\n            \n                    }\\n        \n                 }    \\n\\n        \n                 system(\"PAUSE\");\\n        \n                 return 0;    \\n\n        }\\n\n        
\\n\\n\n        
The answer should come in the form of a table like
\\n\\n\n        
       \n        1            50              50\\n       \n        2            50              50\\n       \n        99           50              50\\n       \n        100          50              50\\n       \n        50           1               50\\n       \n        50           2               50\\n       \n        50           99              50\\n       \n        50           100             50\\n       \n        50           50              1\\n       \n        50           50              2\\n       \n        50           50              99\\n       \n        50           50              100\\n\n        
\\n\\n\n        
if the no of inputs is 3 and their ranges are\\n\n        1,100\\n\n        1,100\\n\n        1,100\\n\n        (could be varied too)
\\n\\n\n        
The output is not coming,can anyone correct the code or tell me what\\'s wrong?
\\n'\nTags : 'c++ c'\n
\n\n

2.2 Mapping the real-world problem to a Machine Learning Problem

\n\n

2.2.1 Type of Machine Learning Problem

\n

It is a multi-label classification problem
\nMulti-label Classification: Multilabel classification assigns to each sample a set of target labels. This can be thought of as predicting properties of a data-point that are not mutually exclusive, such as topics that are relevant for a document. A question on Stackoverflow might be about any of C, Pointers, FileIO and/or memory-management at the same time or none of these.
\n__Credit__: http://scikit-learn.org/stable/modules/multiclass.html\n

\n\n

2.2.2 Performance metric

\nMicro-Averaged F1-Score (Mean F Score) : \nThe F1 score can be interpreted as a weighted average of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal. The formula for the F1 score is:\n\nF1 = 2 * (precision * recall) / (precision + recall)
\n\nIn the multi-class and multi-label case, this is the weighted average of the F1 score of each class.
\n\n'Micro f1 score':
\nCalculate metrics globally by counting the total true positives, false negatives and false positives. This is a better metric when we have class imbalance.\n
\n\n'Macro f1 score':
\nCalculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.\n
\n\nhttps://www.kaggle.com/wiki/MeanFScore
\nhttp://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html
\n
\n Hamming loss : The Hamming loss is the fraction of labels that are incorrectly predicted.
\nhttps://www.kaggle.com/wiki/HammingLoss
","metadata":{}},{"cell_type":"markdown","source":"

3. Exploratory Data Analysis

\n

3.1 Data Loading and Cleaning

\n

3.1.1 Using Pandas with SQLite to Load the data

","metadata":{}},{"cell_type":"code","source":"# Creating db file from csv\n# Learn SQL: https://www.w3schools.com/sql/default.asp\n\nif not os.path.isfile('train.db'):\n start = datetime.now()\n disk_engine = create_engine('sqlite:///train.db')\n start = dt.datetime.now()\n chunksize = 180000\n j = 0\n index_start = 1\n for df in pd.read_csv(\n '../input/facebook-recruiting-iii-keyword-extraction/Train.zip', \n names=['Id', 'Title', 'Body', 'Tags'], chunksize=chunksize, \n iterator=True, encoding='utf-8'\n ):\n df.index += index_start\n j += 1\n print('{} rows'.format(j*chunksize))\n df.to_sql('data', disk_engine, if_exists = 'append')\n index_start = df.index[-1] + 1\n print(\"Time taken to run this cell: \", datetime.now() - start)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-09-22T05:26:26.111279Z","iopub.execute_input":"2021-09-22T05:26:26.111894Z","iopub.status.idle":"2021-09-22T05:33:23.583933Z","shell.execute_reply.started":"2021-09-22T05:26:26.111837Z","shell.execute_reply":"2021-09-22T05:33:23.583230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3.1.2 Counting the number of rows

","metadata":{}},{"cell_type":"code","source":"if os.path.isfile('train.db'):\n start = datetime.now()\n con = sqlite3.connect('train.db')\n num_rows = pd.read_sql_query(\"\"\"SELECT count(*) FROM data\"\"\", con)\n print(\"Number of rows in the database: \",\"\\n\",num_rows['count(*)'].values[0])\n con.close() # Always remember to close the database\n print(\"Time taken to count the number of rows: \", datetime.now() - start)\nelse:\n print(\"Please download the train.db file from drive or run the above cell to genarate train.db file\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:33:23.585780Z","iopub.execute_input":"2021-09-22T05:33:23.586699Z","iopub.status.idle":"2021-09-22T05:33:28.750693Z","shell.execute_reply.started":"2021-09-22T05:33:23.586652Z","shell.execute_reply":"2021-09-22T05:33:28.749713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3.1.3 Checking for duplicates

","metadata":{}},{"cell_type":"code","source":"# Learn SQl: https://www.w3schools.com/sql/default.asp\nif os.path.isfile('train.db'):\n start = datetime.now()\n con = sqlite3.connect('train.db')\n df_no_dup = pd.read_sql_query('SELECT Title, Body, Tags, COUNT(*) as cnt_dup FROM data GROUP BY Title, Body, Tags', con)\n con.close()\n print(\"Time taken to run this cell: \", datetime.now() - start)\nelse:\n print(\"Please download the train.db file from drive or run the first to genarate train.db file\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:33:28.751956Z","iopub.execute_input":"2021-09-22T05:33:28.752272Z","iopub.status.idle":"2021-09-22T05:37:07.233395Z","shell.execute_reply.started":"2021-09-22T05:33:28.752236Z","shell.execute_reply":"2021-09-22T05:37:07.230948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We can observe that there are duplicates\ndf_no_dup.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:37:07.235791Z","iopub.execute_input":"2021-09-22T05:37:07.236307Z","iopub.status.idle":"2021-09-22T05:37:07.256724Z","shell.execute_reply.started":"2021-09-22T05:37:07.236214Z","shell.execute_reply":"2021-09-22T05:37:07.255780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(np.sum(df_no_dup['cnt_dup'])) # Total Number of Questions\nprint(num_rows['count(*)'].values[0]) # Total Number of Questions\nprint(df_no_dup.shape[0]) # Total Number of Unique Questions","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:37:07.257914Z","iopub.execute_input":"2021-09-22T05:37:07.258477Z","iopub.status.idle":"2021-09-22T05:37:07.272366Z","shell.execute_reply.started":"2021-09-22T05:37:07.258408Z","shell.execute_reply":"2021-09-22T05:37:07.270785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Number of duplicate questions: \", num_rows['count(*)'].values[0] - df_no_dup.shape[0],\n \"(\",(1 - ((df_no_dup.shape[0])/(num_rows['count(*)'].values[0])))*100,\"% )\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:37:07.274616Z","iopub.execute_input":"2021-09-22T05:37:07.275609Z","iopub.status.idle":"2021-09-22T05:37:07.284468Z","shell.execute_reply.started":"2021-09-22T05:37:07.275559Z","shell.execute_reply":"2021-09-22T05:37:07.283383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Number of times each question appeared in our database\n# It means that there are 2656284 number of questions which only occur once, 1272336 number of \n# questions which only occur twice, and so on.\ndf_no_dup.cnt_dup.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:37:07.285499Z","iopub.execute_input":"2021-09-22T05:37:07.285739Z","iopub.status.idle":"2021-09-22T05:37:07.334289Z","shell.execute_reply.started":"2021-09-22T05:37:07.285710Z","shell.execute_reply":"2021-09-22T05:37:07.333582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_no_dup.info(show_counts = True)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:37:07.338592Z","iopub.execute_input":"2021-09-22T05:37:07.338999Z","iopub.status.idle":"2021-09-22T05:37:09.211122Z","shell.execute_reply.started":"2021-09-22T05:37:07.338953Z","shell.execute_reply":"2021-09-22T05:37:09.209925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start = datetime.now()\ndf_no_dup[\"tag_count\"] = df_no_dup[\"Tags\"].apply(\n lambda text: len(text.split(\" \")) if text != None else 0 \n)\n\n# Adding a new feature number of tags per question\nprint(\"Time taken to run this cell: \", datetime.now() - start)\ndf_no_dup.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:37:09.212475Z","iopub.execute_input":"2021-09-22T05:37:09.212766Z","iopub.status.idle":"2021-09-22T05:37:13.435308Z","shell.execute_reply.started":"2021-09-22T05:37:09.212734Z","shell.execute_reply":"2021-09-22T05:37:13.434488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Distribution of number of tags per question\ndf_no_dup.tag_count.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:37:13.436626Z","iopub.execute_input":"2021-09-22T05:37:13.437563Z","iopub.status.idle":"2021-09-22T05:37:13.469006Z","shell.execute_reply.started":"2021-09-22T05:37:13.437518Z","shell.execute_reply":"2021-09-22T05:37:13.468120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating a new database with no duplicates\nif not os.path.isfile('../input/d/elemento/facebook-recruiting-iii-keyword-extraction/train_no_dup.db'):\n disk_dup = create_engine(\"sqlite:///train_no_dup.db\")\n no_dup = pd.DataFrame(df_no_dup, columns=['Title', 'Body', 'Tags'])\n no_dup.to_sql('no_dup_train', disk_dup)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:37:13.470395Z","iopub.execute_input":"2021-09-22T05:37:13.471019Z","iopub.status.idle":"2021-09-22T05:37:13.488777Z","shell.execute_reply.started":"2021-09-22T05:37:13.470973Z","shell.execute_reply":"2021-09-22T05:37:13.487838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This method seems more appropriate to work with this much data.\n# Creating the connection with database file.\nif os.path.isfile('../input/d/elemento/facebook-recruiting-iii-keyword-extraction/train_no_dup.db'):\n start = datetime.now()\n con = sqlite3.connect('../input/d/elemento/facebook-recruiting-iii-keyword-extraction/train_no_dup.db')\n tag_data = pd.read_sql_query(\"\"\"SELECT Tags FROM no_dup_train\"\"\", con)\n # Always remember to close the database\n con.close()\n\n # Let's now drop unwanted column.\n tag_data.drop(tag_data.index[0], inplace = True)\n # Printing first 5 rows from our data frame\n tag_data.head()\n print(\"Time taken to run this cell: \", datetime.now() - start)\nelse:\n print(\"Please download the train.db file from drive or run the above cells to genarate train.db file\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:37:13.490419Z","iopub.execute_input":"2021-09-22T05:37:13.490959Z","iopub.status.idle":"2021-09-22T05:38:39.913952Z","shell.execute_reply.started":"2021-09-22T05:37:13.490916Z","shell.execute_reply":"2021-09-22T05:38:39.912871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3.2 Analysis of Tags

\n

3.2.1 Total number of unique tags

","metadata":{}},{"cell_type":"code","source":"# Importing & Initializing the \"CountVectorizer\" object, which is scikit-learn's bag of words tool.\n# By default 'split()' will tokenize each tag using space.\nvectorizer = CountVectorizer(tokenizer = lambda x: x.split())\n\n# fit_transform() does two functions\n# First, it fits the model & learns the vocabulary\n# Second, it transforms our training data into feature vectors. \n# The input to fit_transform should be a list of strings.\ntag_dtm = vectorizer.fit_transform(tag_data['Tags'])","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:38:39.914947Z","iopub.execute_input":"2021-09-22T05:38:39.915161Z","iopub.status.idle":"2021-09-22T05:39:08.061308Z","shell.execute_reply.started":"2021-09-22T05:38:39.915136Z","shell.execute_reply":"2021-09-22T05:39:08.059907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Number of data points :\", tag_dtm.shape[0])\nprint(\"Number of unique tags :\", tag_dtm.shape[1])","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:08.062700Z","iopub.execute_input":"2021-09-22T05:39:08.062941Z","iopub.status.idle":"2021-09-22T05:39:08.069671Z","shell.execute_reply.started":"2021-09-22T05:39:08.062915Z","shell.execute_reply":"2021-09-22T05:39:08.068020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 'get_feature_name()' gives us the vocabulary.\ntags = vectorizer.get_feature_names()\n# Let's look at the tags we have.\nprint(\"Some of the tags we have :\", tags[:10])","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:08.071161Z","iopub.execute_input":"2021-09-22T05:39:08.071568Z","iopub.status.idle":"2021-09-22T05:39:08.129611Z","shell.execute_reply.started":"2021-09-22T05:39:08.071535Z","shell.execute_reply":"2021-09-22T05:39:08.128701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3.2.3 Number of times a tag appeared

","metadata":{}},{"cell_type":"code","source":"# https://stackoverflow.com/questions/15115765/how-to-access-sparse-matrix-elements\n# Let's now store the document term matrix in a dictionary.\nfreqs = tag_dtm.sum(axis = 0).A1\nresult = dict(zip(tags, freqs))","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:08.130779Z","iopub.execute_input":"2021-09-22T05:39:08.131018Z","iopub.status.idle":"2021-09-22T05:39:08.212617Z","shell.execute_reply.started":"2021-09-22T05:39:08.130991Z","shell.execute_reply":"2021-09-22T05:39:08.211612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Saving this dictionary to csv files.\nif not os.path.isfile('../input/d/elemento/facebook-recruiting-iii-keyword-extraction/tag_counts_dict_dtm.csv'):\n with open('tag_counts_dict_dtm.csv', 'w') as csv_file:\n writer = csv.writer(csv_file)\n for key, value in result.items():\n writer.writerow([key, value])\ntag_df = pd.read_csv(\"../input/d/elemento/facebook-recruiting-iii-keyword-extraction/tag_counts_dict_dtm.csv\", names=['Tags', 'Counts'])\ntag_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:08.216225Z","iopub.execute_input":"2021-09-22T05:39:08.216930Z","iopub.status.idle":"2021-09-22T05:39:08.276252Z","shell.execute_reply.started":"2021-09-22T05:39:08.216889Z","shell.execute_reply":"2021-09-22T05:39:08.275434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tag_df_sorted = tag_df.sort_values(['Counts'], ascending=False)\ntag_counts = tag_df_sorted['Counts'].values","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:08.277710Z","iopub.execute_input":"2021-09-22T05:39:08.277942Z","iopub.status.idle":"2021-09-22T05:39:08.293115Z","shell.execute_reply.started":"2021-09-22T05:39:08.277915Z","shell.execute_reply":"2021-09-22T05:39:08.292059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(tag_counts)\nplt.title(\"Distribution of number of times tag appeared in questions\")\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:08.294515Z","iopub.execute_input":"2021-09-22T05:39:08.294760Z","iopub.status.idle":"2021-09-22T05:39:08.598966Z","shell.execute_reply.started":"2021-09-22T05:39:08.294732Z","shell.execute_reply":"2021-09-22T05:39:08.597929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(tag_counts[0:10000])\nplt.title('First 10k tags: Distribution of number of times tag appeared in questions')\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()\nprint(len(tag_counts[0:10000:25]), tag_counts[0:10000:25])","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-09-22T05:39:08.600798Z","iopub.execute_input":"2021-09-22T05:39:08.601098Z","iopub.status.idle":"2021-09-22T05:39:08.836766Z","shell.execute_reply.started":"2021-09-22T05:39:08.601066Z","shell.execute_reply":"2021-09-22T05:39:08.835752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(tag_counts[0:1000])\nplt.title('first 1k tags: Distribution of number of times tag appeared in questions')\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()\nprint(len(tag_counts[0:1000:5]), tag_counts[0:1000:5])","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-09-22T05:39:08.838607Z","iopub.execute_input":"2021-09-22T05:39:08.839013Z","iopub.status.idle":"2021-09-22T05:39:09.072730Z","shell.execute_reply.started":"2021-09-22T05:39:08.838972Z","shell.execute_reply":"2021-09-22T05:39:09.071790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(tag_counts[0:500])\nplt.title('first 500 tags: Distribution of number of times tag appeared in questions')\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.show()\nprint(len(tag_counts[0:500:5]), tag_counts[0:500:5])","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-09-22T05:39:09.073972Z","iopub.execute_input":"2021-09-22T05:39:09.074309Z","iopub.status.idle":"2021-09-22T05:39:09.318233Z","shell.execute_reply.started":"2021-09-22T05:39:09.074276Z","shell.execute_reply":"2021-09-22T05:39:09.317566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(tag_counts[0:100], c='b')\nplt.scatter(x=list(range(0,100,5)), y=tag_counts[0:100:5], c='orange', label=\"quantiles with 0.05 intervals\")\n# Quantiles with 0.25 difference\nplt.scatter(x=list(range(0,100,25)), y=tag_counts[0:100:25], c='m', label = \"quantiles with 0.25 intervals\")\n\nfor x,y in zip(list(range(0,100,25)), tag_counts[0:100:25]):\n plt.annotate(s=\"({} , {})\".format(x,y), xy=(x,y), xytext=(x-0.05, y+500))\n\nplt.title('First 100 tags: Distribution of number of times tag appeared questions')\nplt.grid()\nplt.xlabel(\"Tag number\")\nplt.ylabel(\"Number of times tag appeared\")\nplt.legend()\nplt.show()\nprint(len(tag_counts[0:100:5]), tag_counts[0:100:5])","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:09.319497Z","iopub.execute_input":"2021-09-22T05:39:09.319894Z","iopub.status.idle":"2021-09-22T05:39:09.656054Z","shell.execute_reply.started":"2021-09-22T05:39:09.319865Z","shell.execute_reply":"2021-09-22T05:39:09.655298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Store tags greater than 10K in one list\nlst_tags_gt_10k = tag_df[tag_df.Counts>10000].Tags\n#Print the length of the list\nprint ('{} Tags are used more than 10000 times'.format(len(lst_tags_gt_10k)))\n# Store tags greater than 100K in one list\nlst_tags_gt_100k = tag_df[tag_df.Counts>100000].Tags\n#Print the length of the list.\nprint ('{} Tags are used more than 100000 times'.format(len(lst_tags_gt_100k)))","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:09.657335Z","iopub.execute_input":"2021-09-22T05:39:09.657625Z","iopub.status.idle":"2021-09-22T05:39:09.668301Z","shell.execute_reply.started":"2021-09-22T05:39:09.657587Z","shell.execute_reply":"2021-09-22T05:39:09.666175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Observations:
\n1. There are total 153 tags which are used more than 10000 times.\n2. 14 tags are used more than 100000 times.\n3. Most frequent tag (i.e. c#) is used 331505 times.\n4. Since some tags occur much more frequenctly than others, Micro-averaged F1-score is the appropriate metric for this probelm.","metadata":{}},{"cell_type":"markdown","source":"

3.2.4 Tags Per Question

","metadata":{}},{"cell_type":"code","source":"# Storing the count of tag in each question in list 'tag_count'\ntag_quest_count = tag_dtm.sum(axis=1).tolist()\n# Converting list of lists into single list, we will get [[3], [4], [2], [2], [3]] and we are converting this to [3, 4, 2, 2, 3]\ntag_quest_count=[int(j) for i in tag_quest_count for j in i]\nprint ('We have total {} datapoints.'.format(len(tag_quest_count)))\n\nprint(tag_quest_count[:5])","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:09.670099Z","iopub.execute_input":"2021-09-22T05:39:09.670378Z","iopub.status.idle":"2021-09-22T05:39:14.543101Z","shell.execute_reply.started":"2021-09-22T05:39:09.670348Z","shell.execute_reply":"2021-09-22T05:39:14.542056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print( \"Maximum number of tags per question: %d\"%max(tag_quest_count))\nprint( \"Minimum number of tags per question: %d\"%min(tag_quest_count))\nprint( \"Avg. number of tags per question: %f\"% ((sum(tag_quest_count)*1.0)/len(tag_quest_count)))","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:14.570326Z","iopub.execute_input":"2021-09-22T05:39:14.570637Z","iopub.status.idle":"2021-09-22T05:39:14.738899Z","shell.execute_reply.started":"2021-09-22T05:39:14.570600Z","shell.execute_reply":"2021-09-22T05:39:14.738141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(tag_quest_count, palette='gist_rainbow')\nplt.title(\"Number of tags in the questions \")\nplt.xlabel(\"Number of Tags\")\nplt.ylabel(\"Number of questions\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:14.740070Z","iopub.execute_input":"2021-09-22T05:39:14.740300Z","iopub.status.idle":"2021-09-22T05:39:34.411985Z","shell.execute_reply.started":"2021-09-22T05:39:14.740274Z","shell.execute_reply":"2021-09-22T05:39:34.410270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Observations:
\n1. Maximum number of tags per question: 5\n2. Minimum number of tags per question: 1\n3. Avg. number of tags per question: 2.899\n4. Most of the questions are having 2 or 3 tags","metadata":{}},{"cell_type":"markdown","source":"

3.2.5 Most Frequent Tags

","metadata":{}},{"cell_type":"code","source":"# Plotting word cloud\nstart = datetime.now()\n\n# Lets first convert the 'result' dictionary to 'list of tuples'\ntup = dict(result.items())\n# Initializing WordCloud using frequencies of tags.\nwordcloud = WordCloud( \n background_color='black', width=1600, height=800,\n).generate_from_frequencies(tup)\n\nfig = plt.figure(figsize=(30,20))\nplt.imshow(wordcloud)\nplt.axis('off')\nplt.tight_layout(pad=0)\nfig.savefig(\"tag.png\")\nplt.show()\nprint(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:34.413865Z","iopub.execute_input":"2021-09-22T05:39:34.414244Z","iopub.status.idle":"2021-09-22T05:39:38.816238Z","shell.execute_reply.started":"2021-09-22T05:39:34.414198Z","shell.execute_reply":"2021-09-22T05:39:38.815156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Observations:
\nA look at the word cloud shows that \"c#\", \"java\", \"php\", \"asp.net\", \"javascript\", \"c++\" are some of the most frequent tags.","metadata":{}},{"cell_type":"markdown","source":"

3.2.6 The top 20 tags

","metadata":{}},{"cell_type":"code","source":"i = np.arange(20)\ntag_df_sorted.head(20).plot(kind='bar')\nplt.title('Frequency of top 20 tags')\nplt.xticks(i, tag_df_sorted.head(20)[\"Tags\"])\nplt.xlabel('Tags')\nplt.ylabel('Counts')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:38.817743Z","iopub.execute_input":"2021-09-22T05:39:38.818079Z","iopub.status.idle":"2021-09-22T05:39:39.275240Z","shell.execute_reply.started":"2021-09-22T05:39:38.818047Z","shell.execute_reply":"2021-09-22T05:39:39.274532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Observations:
\n1. Majority of the most frequent tags are programming language.\n2. C# is the top most frequent programming language.\n3. Android, IOS, Linux and windows are among the top most frequent operating systems.","metadata":{}},{"cell_type":"markdown","source":"

3.3 Cleaning and preprocessing of Questions

\n

3.3.1 Preprocessing

\n
    \n
  1. Sample 1M data points
    2. \n
  2. Separate out code-snippets from Body
    3. \n
  3. Remove special characters from Question title and description (not in code)
    4. \n
  4. Remove stop words (Except 'C')
    5. \n
  5. Remove HTML Tags
    6. \n
  6. Convert all the characters into small letters
    7. \n
  7. Use SnowballStemmer to stem the words
    8. \n
","metadata":{}},{"cell_type":"code","source":"def striphtml(data):\n cleanr = re.compile('<.*?>')\n cleantext = re.sub(cleanr, ' ', str(data))\n return cleantext\nstop_words = set(stopwords.words('english'))\nstemmer = SnowballStemmer(\"english\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:39.276583Z","iopub.execute_input":"2021-09-22T05:39:39.277840Z","iopub.status.idle":"2021-09-22T05:39:39.321769Z","shell.execute_reply.started":"2021-09-22T05:39:39.277785Z","shell.execute_reply":"2021-09-22T05:39:39.320417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# http://www.sqlitetutorial.net/sqlite-python/create-tables/\ndef create_connection(db_file):\n \"\"\" \n Create a database connection to the SQLite database specified by db_file\n :param db_file: database file\n :return: Connection object or None\n \"\"\"\n try:\n conn = sqlite3.connect(db_file)\n return conn\n except Error as e:\n print(e)\n \n return None\n\ndef create_table(conn, create_table_sql):\n \"\"\" \n Create a table from the create_table_sql statement\n :param conn: Connection object\n :param create_table_sql: a CREATE TABLE statement\n :return:\n \"\"\"\n try:\n c = conn.cursor()\n c.execute(create_table_sql)\n except Error as e:\n print(e)\n \ndef checkTableExists(dbcon):\n cursr = dbcon.cursor()\n str = \"select name from sqlite_master where type='table'\"\n table_names = cursr.execute(str)\n print(\"Tables in the databse:\")\n tables = table_names.fetchall() \n print(tables[0][0])\n return(len(tables))\n\ndef create_database_table(database, query):\n conn = create_connection(database)\n if conn is not None:\n create_table(conn, query)\n checkTableExists(conn)\n else:\n print(\"Error! cannot create the database connection.\")\n conn.close()\n\nsql_create_table = \"\"\"CREATE TABLE IF NOT EXISTS QuestionsProcessed (question text NOT NULL, code text, tags text, words_pre integer, words_post integer, is_code integer);\"\"\"\ncreate_database_table(\"Processed.db\", sql_create_table)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:39.323249Z","iopub.execute_input":"2021-09-22T05:39:39.323572Z","iopub.status.idle":"2021-09-22T05:39:39.367014Z","shell.execute_reply.started":"2021-09-22T05:39:39.323532Z","shell.execute_reply":"2021-09-22T05:39:39.365988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# http://www.sqlitetutorial.net/sqlite-delete/\n# https://stackoverflow.com/questions/2279706/select-random-row-from-a-sqlite-table\n# We create a new data base to store the sampled and preprocessed questions\n\nstart = datetime.now()\nread_db = '../input/d/elemento/facebook-recruiting-iii-keyword-extraction/train_no_dup.db'\nwrite_db = 'Processed.db'\nif os.path.isfile(read_db):\n conn_r = create_connection(read_db)\n if conn_r is not None:\n reader = conn_r.cursor()\n reader.execute(\"SELECT Title, Body, Tags From no_dup_train ORDER BY RANDOM() LIMIT 1000000;\")\n\nif os.path.isfile(write_db):\n conn_w = create_connection(write_db)\n if conn_w is not None:\n tables = checkTableExists(conn_w)\n writer = conn_w.cursor()\n if tables != 0:\n writer.execute(\"DELETE FROM QuestionsProcessed WHERE 1\")\n print(\"Cleared all the rows\")\n\nprint(\"Time taken to run this cell: \", datetime.now() - start)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T05:39:39.370358Z","iopub.execute_input":"2021-09-22T05:39:39.370715Z","iopub.status.idle":"2021-09-22T05:51:09.089951Z","shell.execute_reply.started":"2021-09-22T05:39:39.370666Z","shell.execute_reply":"2021-09-22T05:51:09.087419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# http://www.bernzilla.com/2008/05/13/selecting-a-random-row-from-an-sqlite-table/\n\nstart = datetime.now()\npreprocessed_data_list = []\nreader.fetchone()\nquestions_with_code=0\nlen_pre=0\nlen_post=0\nquestions_processed = 0\nfor row in reader:\n is_code = 0\n title, question, tags = row[0], row[1], row[2]\n\n if '' in question:\n questions_with_code += 1\n is_code = 1\n x = len(question) + len(title)\n len_pre += x\n\n code = str(re.findall(r'(.*?)', question, flags=re.DOTALL))\n question = re.sub('(.*?)', '', question, flags=re.MULTILINE|re.DOTALL)\n question = striphtml(question.encode('utf-8'))\n title = title.encode('utf-8')\n\n question = str(title) + \" \" + str(question)\n question = re.sub(r'[^A-Za-z]+',' ',question)\n words = word_tokenize(str(question.lower()))\n\n # Removing all single letter and stopwords from question except for the letter 'c'\n question=' '.join(\n str(stemmer.stem(j)) for j in words if j not in stop_words and (len(j) != 1 or j == 'c')\n )\n\n len_post += len(question)\n tup = (question, code, tags, x, len(question), is_code)\n questions_processed += 1\n writer.execute(\"insert into QuestionsProcessed(question,code,tags,words_pre,words_post,is_code) values (?,?,?,?,?,?)\",tup)\n if (questions_processed % 100000 == 0):\n print(\"Number of questions completed = \",questions_processed)\n\nno_dup_avg_len_pre = (len_pre*1.0) / questions_processed\nno_dup_avg_len_post = (len_post*1.0) / questions_processed\n\nprint(\"Avg length of questions(Title+Body) before processing: %d\"%no_dup_avg_len_pre)\nprint(\"Avg length of questions(Title+Body) after processing: %d\"%no_dup_avg_len_post)\nprint(\"Percent of questions containing code: %d\"%((questions_with_code*100.0)/questions_processed))\nprint(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-09-22T05:51:09.094958Z","iopub.execute_input":"2021-09-22T05:51:09.095662Z","iopub.status.idle":"2021-09-22T06:30:04.953120Z","shell.execute_reply.started":"2021-09-22T05:51:09.095606Z","shell.execute_reply":"2021-09-22T06:30:04.951892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Don't forget to close the connections, or else you will end up with locks\nconn_r.commit()\nconn_w.commit()\nconn_r.close()\nconn_w.close()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:04.955895Z","iopub.execute_input":"2021-09-22T06:30:04.956325Z","iopub.status.idle":"2021-09-22T06:30:05.029733Z","shell.execute_reply.started":"2021-09-22T06:30:04.956283Z","shell.execute_reply":"2021-09-22T06:30:05.028075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if os.path.isfile(write_db):\n conn_r = create_connection(write_db)\n if conn_r is not None:\n reader =conn_r.cursor()\n reader.execute(\"SELECT question From QuestionsProcessed LIMIT 10\")\n print(\"Questions after preprocessed\")\n print('='*100)\n reader.fetchone()\n for row in reader:\n print(row)\n print('-'*100)\nconn_r.commit()\nconn_r.close()","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-09-22T06:30:05.034600Z","iopub.execute_input":"2021-09-22T06:30:05.035083Z","iopub.status.idle":"2021-09-22T06:30:05.053792Z","shell.execute_reply.started":"2021-09-22T06:30:05.035044Z","shell.execute_reply":"2021-09-22T06:30:05.051962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Taking 1 Million entries to a dataframe.\nwrite_db = 'Processed.db'\nif os.path.isfile(write_db):\n conn_r = create_connection(write_db)\n if conn_r is not None:\n preprocessed_data = pd.read_sql_query(\"\"\"SELECT question, Tags FROM QuestionsProcessed\"\"\", conn_r)\nconn_r.commit()\nconn_r.close()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:05.060188Z","iopub.execute_input":"2021-09-22T06:30:05.061100Z","iopub.status.idle":"2021-09-22T06:30:08.857050Z","shell.execute_reply.started":"2021-09-22T06:30:05.061038Z","shell.execute_reply":"2021-09-22T06:30:08.856167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preprocessed_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:08.860356Z","iopub.execute_input":"2021-09-22T06:30:08.860704Z","iopub.status.idle":"2021-09-22T06:30:08.886189Z","shell.execute_reply.started":"2021-09-22T06:30:08.860642Z","shell.execute_reply":"2021-09-22T06:30:08.885118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Number of data points in sample: \", preprocessed_data.shape[0])\nprint(\"Number of dimensions: \", preprocessed_data.shape[1])","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:08.887803Z","iopub.execute_input":"2021-09-22T06:30:08.888255Z","iopub.status.idle":"2021-09-22T06:30:08.903884Z","shell.execute_reply.started":"2021-09-22T06:30:08.888210Z","shell.execute_reply":"2021-09-22T06:30:08.902967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

4. Machine Learning Models

\n

4.1 Converting tags for multilabel problems

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Xy1y2y3y4
x10110
x11000
x10100
","metadata":{}},{"cell_type":"code","source":"# binary='true' will give a binary vectorizer\nvectorizer = CountVectorizer(tokenizer = lambda x: x.split(), binary='true')\nmultilabel_y = vectorizer.fit_transform(preprocessed_data['tags'])","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:08.906251Z","iopub.execute_input":"2021-09-22T06:30:08.906727Z","iopub.status.idle":"2021-09-22T06:30:20.759265Z","shell.execute_reply.started":"2021-09-22T06:30:08.906683Z","shell.execute_reply":"2021-09-22T06:30:20.758094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def tags_to_choose(n):\n t = multilabel_y.sum(axis=0).tolist()[0]\n sorted_tags_i = sorted(range(len(t)), key=lambda i: t[i], reverse=True)\n multilabel_yn = multilabel_y[:,sorted_tags_i[:n]]\n return multilabel_yn\n\ndef questions_explained_fn(n):\n multilabel_yn = tags_to_choose(n)\n x = multilabel_yn.sum(axis=1)\n return (np.count_nonzero(x==0))","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:20.764142Z","iopub.execute_input":"2021-09-22T06:30:20.764646Z","iopub.status.idle":"2021-09-22T06:30:20.775103Z","shell.execute_reply.started":"2021-09-22T06:30:20.764600Z","shell.execute_reply":"2021-09-22T06:30:20.773954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"questions_explained = []\ntotal_tags = multilabel_y.shape[1]\ntotal_qs = preprocessed_data.shape[0]\nfor i in range(500, total_tags, 100):\n questions_explained.append(np.round(((total_qs-questions_explained_fn(i))/total_qs)*100,3))","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:20.777576Z","iopub.execute_input":"2021-09-22T06:30:20.778111Z","iopub.status.idle":"2021-09-22T06:30:55.905814Z","shell.execute_reply.started":"2021-09-22T06:30:20.778071Z","shell.execute_reply":"2021-09-22T06:30:55.904780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots()\nax.plot(questions_explained)\nxlabel = list(500 + np.array(range(-50,450,50))*50)\nax.set_xticklabels(xlabel)\nplt.xlabel(\"Number of tags\")\nplt.ylabel(\"Number Questions coverd partially\")\nplt.grid()\nplt.show()\n\n# You can choose any number of tags based on your computing power, minimun is 50(it covers 90% of the tags)\nprint(\"With \", 5500, \"tags we are covering \", questions_explained[50], \"% of questions\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:55.907616Z","iopub.execute_input":"2021-09-22T06:30:55.907952Z","iopub.status.idle":"2021-09-22T06:30:56.218815Z","shell.execute_reply.started":"2021-09-22T06:30:55.907894Z","shell.execute_reply":"2021-09-22T06:30:56.217894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"multilabel_yx = tags_to_choose(5500)\nprint(\"Number of questions that are not covered:\", questions_explained_fn(5500),\"out of\", total_qs)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:56.220339Z","iopub.execute_input":"2021-09-22T06:30:56.220637Z","iopub.status.idle":"2021-09-22T06:30:56.403257Z","shell.execute_reply.started":"2021-09-22T06:30:56.220607Z","shell.execute_reply":"2021-09-22T06:30:56.401985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We consider top 15% tags which covers 99% of the questions\nprint(\"Number of tags in sample:\", multilabel_y.shape[1])\nprint(\"Number of tags taken:\", multilabel_yx.shape[1],\"(\",(multilabel_yx.shape[1]/multilabel_y.shape[1])*100,\"%)\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:56.405331Z","iopub.execute_input":"2021-09-22T06:30:56.405646Z","iopub.status.idle":"2021-09-22T06:30:56.413606Z","shell.execute_reply.started":"2021-09-22T06:30:56.405608Z","shell.execute_reply":"2021-09-22T06:30:56.412640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

4.2 Split the data into test and train (80:20)

","metadata":{}},{"cell_type":"code","source":"total_size = preprocessed_data.shape[0]\ntrain_size = int(0.80*total_size)\n\nx_train = preprocessed_data.head(train_size)\nx_test = preprocessed_data.tail(total_size - train_size)\n\ny_train = multilabel_yx[0:train_size, : ]\ny_test = multilabel_yx[train_size:total_size, : ]","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:56.415670Z","iopub.execute_input":"2021-09-22T06:30:56.415920Z","iopub.status.idle":"2021-09-22T06:30:56.493382Z","shell.execute_reply.started":"2021-09-22T06:30:56.415893Z","shell.execute_reply":"2021-09-22T06:30:56.491922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Number of data points in train data:\", y_train.shape)\nprint(\"Number of data points in test data:\", y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:56.495570Z","iopub.execute_input":"2021-09-22T06:30:56.495923Z","iopub.status.idle":"2021-09-22T06:30:56.502600Z","shell.execute_reply.started":"2021-09-22T06:30:56.495871Z","shell.execute_reply":"2021-09-22T06:30:56.501500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

4.3 Featurizing data

","metadata":{}},{"cell_type":"code","source":"# Removing the train.db file from Kaggle Output folder, in order to free the occupied space\nos.remove(\"./train.db\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:56.504974Z","iopub.execute_input":"2021-09-22T06:30:56.505371Z","iopub.status.idle":"2021-09-22T06:30:56.810041Z","shell.execute_reply.started":"2021-09-22T06:30:56.505340Z","shell.execute_reply":"2021-09-22T06:30:56.808957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# In kaggle kernel, TF-IDF is not working with max_features = 200000 & ngram_range = (1, 3) \n# due to memory constraints. Hence, setting the max_features = 100000, ngram_range = (1, 2)\nstart = datetime.now()\nvectorizer = TfidfVectorizer(\n min_df = 0.00009, max_features = 100000, smooth_idf = True, norm = \"l2\",\n tokenizer = lambda x: x.split(), sublinear_tf = False, ngram_range = (1,2)\n)\nx_train_multilabel = vectorizer.fit_transform(x_train['question'])\nx_test_multilabel = vectorizer.transform(x_test['question'])\nprint(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:30:56.811626Z","iopub.execute_input":"2021-09-22T06:30:56.812047Z","iopub.status.idle":"2021-09-22T06:34:53.096656Z","shell.execute_reply.started":"2021-09-22T06:30:56.812004Z","shell.execute_reply":"2021-09-22T06:34:53.095495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Dimensions of train data X:\", x_train_multilabel.shape, \"Y :\",y_train.shape)\nprint(\"Dimensions of test data X:\", x_test_multilabel.shape, \"Y:\",y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:34:53.099289Z","iopub.execute_input":"2021-09-22T06:34:53.099595Z","iopub.status.idle":"2021-09-22T06:34:53.106460Z","shell.execute_reply.started":"2021-09-22T06:34:53.099563Z","shell.execute_reply":"2021-09-22T06:34:53.105435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# https://www.analyticsvidhya.com/blog/2017/08/introduction-to-multi-label-classification/\n# https://stats.stackexchange.com/questions/117796/scikit-multi-label-classification\n# classifier = LabelPowerset(GaussianNB())\n\"\"\"\nfrom skmultilearn.adapt import MLkNN\nclassifier = MLkNN(k=21)\n\n# train\nclassifier.fit(x_train_multilabel, y_train)\n\n# predict\npredictions = classifier.predict(x_test_multilabel)\nprint(accuracy_score(y_test,predictions))\nprint(metrics.f1_score(y_test, predictions, average = 'macro'))\nprint(metrics.f1_score(y_test, predictions, average = 'micro'))\nprint(metrics.hamming_loss(y_test,predictions))\n\n\"\"\"\n# We are getting memory error because the multilearn package \n# is trying to convert the data into dense matrix\n# ---------------------------------------------------------------------------\n# MemoryError Traceback (most recent call last)\n# in ()\n# ----> classifier.fit(x_train_multilabel, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:34:53.108241Z","iopub.execute_input":"2021-09-22T06:34:53.108599Z","iopub.status.idle":"2021-09-22T06:34:53.126783Z","shell.execute_reply.started":"2021-09-22T06:34:53.108556Z","shell.execute_reply":"2021-09-22T06:34:53.125582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

4.4 Applying Logistic Regression with OneVsRest Classifier

","metadata":{}},{"cell_type":"code","source":"# This will be taking so much time try not to run it, download the lr_with_equal_weight.pkl file and use to predict\n# This takes about 6-7 hours to run.\n# classifier = OneVsRestClassifier(SGDClassifier(loss='log', alpha=0.00001, penalty='l1'), n_jobs=-1)\n# classifier.fit(x_train_multilabel, y_train)\n# predictions = classifier.predict(x_test_multilabel)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:34:53.128904Z","iopub.execute_input":"2021-09-22T06:34:53.129228Z","iopub.status.idle":"2021-09-22T06:34:53.137401Z","shell.execute_reply.started":"2021-09-22T06:34:53.129195Z","shell.execute_reply":"2021-09-22T06:34:53.136606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This cell won't run due to the difference in the versions of Joblib, that are used to\n# store the model and to load the model.\n# classifier = joblib.load(\"../input/d/elemento/facebook-recruiting-iii-keyword-extraction/lr_with_equal_weight.pkl\")\n# predictions = classifier.predict(x_test_multilabel)\n# print(\"accuracy :\",metrics.accuracy_score(y_test,predictions))\n# print(\"macro f1 score :\",metrics.f1_score(y_test, predictions, average = 'macro'))\n# print(\"micro f1 scoore :\",metrics.f1_score(y_test, predictions, average = 'micro'))\n# print(\"hamming loss :\",metrics.hamming_loss(y_test,predictions))\n# print(\"Precision recall report :\\n\",metrics.classification_report(y_test, predictions))","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:34:53.139403Z","iopub.execute_input":"2021-09-22T06:34:53.140150Z","iopub.status.idle":"2021-09-22T06:34:53.152754Z","shell.execute_reply.started":"2021-09-22T06:34:53.140112Z","shell.execute_reply":"2021-09-22T06:34:53.151594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For saving the Model\n# from sklearn.externals import joblib\n# joblib.dump(classifier, 'lr_with_equal_weight.pkl') ","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:34:53.154703Z","iopub.execute_input":"2021-09-22T06:34:53.155116Z","iopub.status.idle":"2021-09-22T06:34:53.169272Z","shell.execute_reply.started":"2021-09-22T06:34:53.155075Z","shell.execute_reply":"2021-09-22T06:34:53.168580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

4.5 Modeling with less data points (0.5M data points) and more weight to title and 500 tags only.

","metadata":{}},{"cell_type":"code","source":"sql_create_table = \"\"\"CREATE TABLE IF NOT EXISTS QuestionsProcessed (question text NOT NULL, code text, tags text, words_pre integer, words_post integer, is_code integer);\"\"\"\ncreate_database_table(\"Titlemoreweight.db\", sql_create_table)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:34:53.170513Z","iopub.execute_input":"2021-09-22T06:34:53.171076Z","iopub.status.idle":"2021-09-22T06:34:53.204076Z","shell.execute_reply.started":"2021-09-22T06:34:53.171044Z","shell.execute_reply":"2021-09-22T06:34:53.203321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# http://www.sqlitetutorial.net/sqlite-delete/\n# https://stackoverflow.com/questions/2279706/select-random-row-from-a-sqlite-table\n\nread_db = '../input/d/elemento/facebook-recruiting-iii-keyword-extraction/train_no_dup.db'\nwrite_db = 'Titlemoreweight.db'\ntrain_datasize = 400000\nif os.path.isfile(read_db):\n conn_r = create_connection(read_db)\n if conn_r is not None:\n reader =conn_r.cursor()\n # For selecting first 0.5M rows\n reader.execute(\"SELECT Title, Body, Tags From no_dup_train LIMIT 500001;\")\n # For selecting random points\n # reader.execute(\"SELECT Title, Body, Tags From no_dup_train ORDER BY RANDOM() LIMIT 500001;\")\n\nif os.path.isfile(write_db):\n conn_w = create_connection(write_db)\n if conn_w is not None:\n tables = checkTableExists(conn_w)\n writer =conn_w.cursor()\n if tables != 0:\n writer.execute(\"DELETE FROM QuestionsProcessed WHERE 1\")\n print(\"Cleared All the rows\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:34:53.205465Z","iopub.execute_input":"2021-09-22T06:34:53.205689Z","iopub.status.idle":"2021-09-22T06:34:53.390328Z","shell.execute_reply.started":"2021-09-22T06:34:53.205664Z","shell.execute_reply":"2021-09-22T06:34:53.389415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

4.5.1 Preprocessing of questions

\n
    \n
  1. Separate Code from Body
    2. \n
  2. Remove Spcial characters from Question title and description (not in code)
    3. \n
  3. Give more weightage to title : Add title three times to the question
    4. \n
  4. Remove stop words (Except 'C')
    5. \n
  5. Remove HTML Tags
    6. \n
  6. Convert all the characters into small letters
    7. \n
  7. Use SnowballStemmer to stem the words
    8. \n
","metadata":{}},{"cell_type":"code","source":"# http://www.bernzilla.com/2008/05/13/selecting-a-random-row-from-an-sqlite-table/\nstart = datetime.now()\npreprocessed_data_list=[]\nreader.fetchone()\nquestions_with_code=0\nlen_pre=0\nlen_post=0\nquestions_proccesed = 0\nfor row in reader:\n \n is_code = 0\n \n title, question, tags = row[0], row[1], str(row[2])\n \n if '' in question:\n questions_with_code+=1\n is_code = 1\n x = len(question)+len(title)\n len_pre+=x\n \n code = str(re.findall(r'(.*?)', question, flags=re.DOTALL))\n \n question=re.sub('(.*?)', '', question, flags=re.MULTILINE|re.DOTALL)\n question=striphtml(question.encode('utf-8'))\n \n title=title.encode('utf-8')\n \n # Adding title three time to the data to increase its weight\n # Add tags string to the training data\n \n question = str(title) + \" \" + str(title) + \" \" + str(title) + \" \" + question\n \n# if questions_proccesed<=train_datasize:\n# question=str(title)+\" \"+str(title)+\" \"+str(title)+\" \"+question+\" \"+str(tags)\n# else:\n# question=str(title)+\" \"+str(title)+\" \"+str(title)+\" \"+question\n\n question=re.sub(r'[^A-Za-z0-9#+.\\-]+',' ',question)\n words=word_tokenize(str(question.lower()))\n \n # Removing all single letter and and stopwords from question exceptt for the letter 'c'\n question=' '.join(str(stemmer.stem(j)) for j in words if j not in stop_words and (len(j)!=1 or j=='c'))\n \n len_post+=len(question)\n tup = (question,code,tags,x,len(question),is_code)\n questions_proccesed += 1\n writer.execute(\"insert into QuestionsProcessed(question,code,tags,words_pre,words_post,is_code) values (?,?,?,?,?,?)\",tup)\n if (questions_proccesed%100000==0):\n print(\"number of questions completed=\",questions_proccesed)\n\nno_dup_avg_len_pre=(len_pre*1.0)/questions_proccesed\nno_dup_avg_len_post=(len_post*1.0)/questions_proccesed\n\nprint( \"Avg. length of questions(Title+Body) before processing: %d\"%no_dup_avg_len_pre)\nprint( \"Avg. length of questions(Title+Body) after processing: %d\"%no_dup_avg_len_post)\nprint (\"Percent of questions containing code: %d\"%((questions_with_code*100.0)/questions_proccesed))\n\nprint(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T06:34:53.393027Z","iopub.execute_input":"2021-09-22T06:34:53.394220Z","iopub.status.idle":"2021-09-22T07:02:05.819832Z","shell.execute_reply.started":"2021-09-22T06:34:53.394169Z","shell.execute_reply":"2021-09-22T07:02:05.818262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Never forget to close the conections or else we will end up with database locks\nconn_r.commit()\nconn_w.commit()\nconn_r.close()\nconn_w.close()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:05.822070Z","iopub.execute_input":"2021-09-22T07:02:05.822488Z","iopub.status.idle":"2021-09-22T07:02:05.862898Z","shell.execute_reply.started":"2021-09-22T07:02:05.822409Z","shell.execute_reply":"2021-09-22T07:02:05.862140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sample quesitons after preprocessing of data\nif os.path.isfile(write_db):\n conn_r = create_connection(write_db)\n if conn_r is not None:\n reader =conn_r.cursor()\n reader.execute(\"SELECT question From QuestionsProcessed LIMIT 10\")\n print(\"Questions after preprocessed\")\n print('='*100)\n reader.fetchone()\n for row in reader:\n print(row)\n print('-'*100)\nconn_r.commit()\nconn_r.close()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:05.867378Z","iopub.execute_input":"2021-09-22T07:02:05.868328Z","iopub.status.idle":"2021-09-22T07:02:05.888166Z","shell.execute_reply.started":"2021-09-22T07:02:05.868280Z","shell.execute_reply":"2021-09-22T07:02:05.886709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Saving Preprocessed data to a Database\n# Taking 0.5 Million entries to a dataframe.\nwrite_db = 'Titlemoreweight.db'\nif os.path.isfile(write_db):\n conn_r = create_connection(write_db)\n if conn_r is not None:\n preprocessed_data = pd.read_sql_query(\"\"\"SELECT question, Tags FROM QuestionsProcessed\"\"\", conn_r)\nconn_r.commit()\nconn_r.close()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:05.892186Z","iopub.execute_input":"2021-09-22T07:02:05.892646Z","iopub.status.idle":"2021-09-22T07:02:07.877218Z","shell.execute_reply.started":"2021-09-22T07:02:05.892610Z","shell.execute_reply":"2021-09-22T07:02:07.875990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preprocessed_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:07.881008Z","iopub.execute_input":"2021-09-22T07:02:07.881298Z","iopub.status.idle":"2021-09-22T07:02:07.897952Z","shell.execute_reply.started":"2021-09-22T07:02:07.881270Z","shell.execute_reply":"2021-09-22T07:02:07.896592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Number of data points in sample:\", preprocessed_data.shape[0])\nprint(\"Number of dimensions:\", preprocessed_data.shape[1])","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:07.900309Z","iopub.execute_input":"2021-09-22T07:02:07.900666Z","iopub.status.idle":"2021-09-22T07:02:07.921357Z","shell.execute_reply.started":"2021-09-22T07:02:07.900632Z","shell.execute_reply":"2021-09-22T07:02:07.919941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Converting string Tags to multilable output variables \nvectorizer = CountVectorizer(tokenizer = lambda x: x.split(), binary='true')\nmultilabel_y = vectorizer.fit_transform(preprocessed_data['tags'])","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:07.923908Z","iopub.execute_input":"2021-09-22T07:02:07.924303Z","iopub.status.idle":"2021-09-22T07:02:12.832745Z","shell.execute_reply.started":"2021-09-22T07:02:07.924268Z","shell.execute_reply":"2021-09-22T07:02:12.831808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Selecting 500 Tags\nquestions_explained = []\ntotal_tags = multilabel_y.shape[1]\ntotal_qs = preprocessed_data.shape[0]\nfor i in range(500, total_tags, 100):\n questions_explained.append(np.round(((total_qs-questions_explained_fn(i))/total_qs)*100,3))","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:12.835247Z","iopub.execute_input":"2021-09-22T07:02:12.835598Z","iopub.status.idle":"2021-09-22T07:02:30.224315Z","shell.execute_reply.started":"2021-09-22T07:02:12.835553Z","shell.execute_reply":"2021-09-22T07:02:30.223164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots()\nax.plot(questions_explained)\nxlabel = list(500+np.array(range(-50,450,50))*50)\nax.set_xticklabels(xlabel)\nplt.xlabel(\"Number of tags\")\nplt.ylabel(\"Number Questions coverd partially\")\nplt.grid()\nplt.show()\n# you can choose any number of tags based on your computing power, minimun is 500(it covers 90% of the tags)\nprint(\"With \",5500,\"tags we are covering \",questions_explained[50],\"% of questions\")\nprint(\"With \",500,\"tags we are covering \",questions_explained[0],\"% of questions\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:30.226181Z","iopub.execute_input":"2021-09-22T07:02:30.226484Z","iopub.status.idle":"2021-09-22T07:02:30.511153Z","shell.execute_reply.started":"2021-09-22T07:02:30.226437Z","shell.execute_reply":"2021-09-22T07:02:30.509571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We will be taking 500 tags\nmultilabel_yx = tags_to_choose(500)\nprint(\"Number of questions that are not covered :\", questions_explained_fn(500),\"out of \", total_qs)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:30.513068Z","iopub.execute_input":"2021-09-22T07:02:30.513320Z","iopub.status.idle":"2021-09-22T07:02:30.612330Z","shell.execute_reply.started":"2021-09-22T07:02:30.513293Z","shell.execute_reply":"2021-09-22T07:02:30.610946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train = preprocessed_data.head(train_datasize)\nx_test = preprocessed_data.tail(preprocessed_data.shape[0] - 400000)\n\ny_train = multilabel_yx[0:train_datasize,:]\ny_test = multilabel_yx[train_datasize:preprocessed_data.shape[0],:]","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:30.614231Z","iopub.execute_input":"2021-09-22T07:02:30.614546Z","iopub.status.idle":"2021-09-22T07:02:30.638224Z","shell.execute_reply.started":"2021-09-22T07:02:30.614516Z","shell.execute_reply":"2021-09-22T07:02:30.637034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Number of data points in train data :\", y_train.shape)\nprint(\"Number of data points in test data :\", y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:30.640361Z","iopub.execute_input":"2021-09-22T07:02:30.640622Z","iopub.status.idle":"2021-09-22T07:02:30.647722Z","shell.execute_reply.started":"2021-09-22T07:02:30.640595Z","shell.execute_reply":"2021-09-22T07:02:30.646644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

4.5.2 Featurizing data with TfIdf vectorizer

","metadata":{}},{"cell_type":"code","source":"# In kaggle kernel, TF-IDF is not working with max_features = 200000 & ngram_range = (1, 3) \n# due to memory constraints. Hence, setting the max_features = 100000, ngram_range = (1, 2)\nstart = datetime.now()\nvectorizer = TfidfVectorizer(\n min_df = 0.00009, max_features = 100000, smooth_idf = True, norm = \"l2\",\n tokenizer = lambda x: x.split(), sublinear_tf = False, ngram_range = (1,2)\n)\nx_train_multilabel = vectorizer.fit_transform(x_train['question'])\nx_test_multilabel = vectorizer.transform(x_test['question'])\nprint(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:02:30.649841Z","iopub.execute_input":"2021-09-22T07:02:30.650149Z","iopub.status.idle":"2021-09-22T07:05:00.182331Z","shell.execute_reply.started":"2021-09-22T07:02:30.650118Z","shell.execute_reply":"2021-09-22T07:05:00.180866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Dimensions of train data X:\",x_train_multilabel.shape, \"Y :\",y_train.shape)\nprint(\"Dimensions of test data X:\",x_test_multilabel.shape,\"Y:\",y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:05:00.184806Z","iopub.execute_input":"2021-09-22T07:05:00.185219Z","iopub.status.idle":"2021-09-22T07:05:00.194244Z","shell.execute_reply.started":"2021-09-22T07:05:00.185170Z","shell.execute_reply":"2021-09-22T07:05:00.192757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

4.5.3 Applying Logistic Regression with OneVsRest Classifier

","metadata":{}},{"cell_type":"code","source":"# start = datetime.now()\n# classifier = OneVsRestClassifier(SGDClassifier(loss='log', alpha=0.00001, penalty='l1'), n_jobs=-1)\n# classifier.fit(x_train_multilabel, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:05:00.196811Z","iopub.execute_input":"2021-09-22T07:05:00.197312Z","iopub.status.idle":"2021-09-22T07:05:00.214957Z","shell.execute_reply.started":"2021-09-22T07:05:00.197244Z","shell.execute_reply":"2021-09-22T07:05:00.213992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Saving the Classifier\n# joblib.dump(classifier, 'lr_with_more_title_weight.pkl') \n\n# Loading the Classifier\nclassifier = joblib.load('../input/d/elemento/facebook-recruiting-iii-keyword-extraction/lr_with_more_title_weight.pkl') ","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:05:00.217173Z","iopub.execute_input":"2021-09-22T07:05:00.217519Z","iopub.status.idle":"2021-09-22T07:05:03.817022Z","shell.execute_reply.started":"2021-09-22T07:05:00.217472Z","shell.execute_reply":"2021-09-22T07:05:03.815326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = classifier.predict(x_test_multilabel)\n\nprint(\"Accuracy :\",metrics.accuracy_score(y_test, predictions))\nprint(\"Hamming loss \",metrics.hamming_loss(y_test,predictions))\n\nprecision = precision_score(y_test, predictions, average='micro')\nrecall = recall_score(y_test, predictions, average='micro')\nf1 = f1_score(y_test, predictions, average='micro')\n \nprint(\"Micro-average quality numbers\")\nprint(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\nprecision = precision_score(y_test, predictions, average='macro')\nrecall = recall_score(y_test, predictions, average='macro')\nf1 = f1_score(y_test, predictions, average='macro')\n \nprint(\"Macro-average quality numbers\")\nprint(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\nprint (metrics.classification_report(y_test, predictions))\nprint(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-09-22T07:05:03.820148Z","iopub.execute_input":"2021-09-22T07:05:03.821153Z","iopub.status.idle":"2021-09-22T07:05:16.018114Z","shell.execute_reply.started":"2021-09-22T07:05:03.821107Z","shell.execute_reply":"2021-09-22T07:05:16.016468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# start = datetime.now()\n# classifier_2 = OneVsRestClassifier(LogisticRegression(penalty='l1', solver='liblinear'), n_jobs=-1)\n# classifier_2.fit(x_train_multilabel, y_train)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-09-22T07:05:16.020801Z","iopub.execute_input":"2021-09-22T07:05:16.021249Z","iopub.status.idle":"2021-09-22T07:05:16.027700Z","shell.execute_reply.started":"2021-09-22T07:05:16.021202Z","shell.execute_reply":"2021-09-22T07:05:16.026160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Saving the Classifier\n# joblib.dump(classifier_2, 'lr_with_more_title_weight_2.pkl') \n\n# Loading the Classifier\nclassifier_2 = joblib.load('../input/d/elemento/facebook-recruiting-iii-keyword-extraction/lr_with_more_title_weight_2.pkl') ","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:05:16.029564Z","iopub.execute_input":"2021-09-22T07:05:16.029937Z","iopub.status.idle":"2021-09-22T07:05:19.781042Z","shell.execute_reply.started":"2021-09-22T07:05:16.029885Z","shell.execute_reply":"2021-09-22T07:05:19.780149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions_2 = classifier_2.predict(x_test_multilabel)\nprint(\"Accuracy :\",metrics.accuracy_score(y_test, predictions_2))\nprint(\"Hamming loss \",metrics.hamming_loss(y_test,predictions_2))\n\nprecision = precision_score(y_test, predictions_2, average='micro')\nrecall = recall_score(y_test, predictions_2, average='micro')\nf1 = f1_score(y_test, predictions_2, average='micro')\n \nprint(\"Micro-average quality numbers\")\nprint(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\nprecision = precision_score(y_test, predictions_2, average='macro')\nrecall = recall_score(y_test, predictions_2, average='macro')\nf1 = f1_score(y_test, predictions_2, average='macro')\n \nprint(\"Macro-average quality numbers\")\nprint(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\nprint (metrics.classification_report(y_test, predictions_2))\nprint(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-09-22T07:05:19.784479Z","iopub.execute_input":"2021-09-22T07:05:19.785418Z","iopub.status.idle":"2021-09-22T07:05:32.775810Z","shell.execute_reply.started":"2021-09-22T07:05:19.785359Z","shell.execute_reply":"2021-09-22T07:05:32.774062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5. Assignments

\n
    \n
  1. Use bag of words upto 4 grams and compute the micro f1 score with Logistic regression(OvR)
    2. \n
  2. Perform hyperparam tuning on alpha (or lambda) for Logistic regression to improve the performance using GridSearch
    3. \n
  3. Try OneVsRestClassifier with Linear-SVM (SGDClassifier with loss-hinge)
    4. \n
","metadata":{}},{"cell_type":"markdown","source":"### 5.1) BoW upto 4 grams, Micro-F1 score with Logistic Regression (OvR)","metadata":{}},{"cell_type":"code","source":"# Removing the train.db file from Kaggle Output folder, in order to free the occupied space\n# os.remove(\"./Processed.db\")\n# os.remove(\"./Titlemoreweight.db\")","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:05:32.779596Z","iopub.execute_input":"2021-09-22T07:05:32.780028Z","iopub.status.idle":"2021-09-22T07:05:32.785177Z","shell.execute_reply.started":"2021-09-22T07:05:32.779978Z","shell.execute_reply":"2021-09-22T07:05:32.783795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Featuring Data with Bag of Words\n# Due to memory constraints, won't be able to run this\n# start = datetime.now()\n# vectorizer = CountVectorizer(\n# min_df = 0.00009, max_features = 50000,\n# tokenizer = lambda x: x.split(), ngram_range = (1,4)\n# )\n# x_train_multilabel = vectorizer.fit_transform(x_train['question'])\n# x_test_multilabel = vectorizer.transform(x_test['question'])\n# print(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:05:32.788889Z","iopub.execute_input":"2021-09-22T07:05:32.789499Z","iopub.status.idle":"2021-09-22T07:05:32.806392Z","shell.execute_reply.started":"2021-09-22T07:05:32.789418Z","shell.execute_reply":"2021-09-22T07:05:32.804975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# start = datetime.now()\n# classifier_3 = OneVsRestClassifier(LogisticRegression(penalty='l1', solver='liblinear'), n_jobs=-1)\n# classifier_3.fit(x_train_multilabel, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:05:32.809056Z","iopub.execute_input":"2021-09-22T07:05:32.809803Z","iopub.status.idle":"2021-09-22T07:05:32.826019Z","shell.execute_reply.started":"2021-09-22T07:05:32.809737Z","shell.execute_reply":"2021-09-22T07:05:32.824150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Saving the Classifier\n# joblib.dump(classifier_3, 'lr_with_more_title_weight_3.pkl') \n\n# Loading the Classifier\n# classifier_3 = joblib.load('../input/d/elemento/facebook-recruiting-iii-keyword-extraction/lr_with_more_title_weight_3.pkl') ","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:05:32.827851Z","iopub.execute_input":"2021-09-22T07:05:32.828798Z","iopub.status.idle":"2021-09-22T07:05:32.846593Z","shell.execute_reply.started":"2021-09-22T07:05:32.828736Z","shell.execute_reply":"2021-09-22T07:05:32.844950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# predictions = classifier_3.predict(x_test_multilabel)\n\n# print(\"Accuracy :\",metrics.accuracy_score(y_test, predictions))\n# print(\"Hamming loss \",metrics.hamming_loss(y_test,predictions))\n\n# precision = precision_score(y_test, predictions, average='micro')\n# recall = recall_score(y_test, predictions, average='micro')\n# f1 = f1_score(y_test, predictions, average='micro')\n \n# print(\"Micro-average quality numbers\")\n# print(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\n# precision = precision_score(y_test, predictions, average='macro')\n# recall = recall_score(y_test, predictions, average='macro')\n# f1 = f1_score(y_test, predictions, average='macro')\n \n# print(\"Macro-average quality numbers\")\n# print(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\n# print (metrics.classification_report(y_test, predictions))\n# print(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:05:32.865695Z","iopub.execute_input":"2021-09-22T07:05:32.866070Z","iopub.status.idle":"2021-09-22T07:05:32.873969Z","shell.execute_reply.started":"2021-09-22T07:05:32.866035Z","shell.execute_reply":"2021-09-22T07:05:32.872413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.2) Perform Hyper-parameter tuning on Alpha (or lambda)","metadata":{}},{"cell_type":"code","source":"# Due to memory constraints, won't be able to run this\n# Defining the list of parameters and accuracies\n# params = [0.5, 1, 2]\n# acc = []\n\n# for c in params:\n# clf = OneVsRestClassifier(LogisticRegression(penalty='l1', solver='liblinear', C = c), n_jobs=-1)\n# clf.fit(x_train_multilabel, y_train)\n# predictions = clf.predict(x_test_multilabel)\n# acc.append(metrics.accuracy_score(y_test, predictions))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Once the accuracies are obtained, we can simply find out the value of C, corresponding to which,\n# we get the highest accuracy, and use it to train the final model, which will be 'classifier_4'","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Saving the Classifier\n# joblib.dump(classifier_4, 'lr_with_more_title_weight_4.pkl') \n\n# Loading the Classifier\n# classifier_4 = joblib.load('../input/d/elemento/facebook-recruiting-iii-keyword-extraction/lr_with_more_title_weight_4.pkl') ","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:54:11.194706Z","iopub.status.idle":"2021-09-22T07:54:11.195315Z","shell.execute_reply.started":"2021-09-22T07:54:11.195019Z","shell.execute_reply":"2021-09-22T07:54:11.195056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# predictions = classifier_4.predict(x_test_multilabel)\n\n# print(\"Accuracy :\",metrics.accuracy_score(y_test, predictions))\n# print(\"Hamming loss \",metrics.hamming_loss(y_test,predictions))\n\n# precision = precision_score(y_test, predictions, average='micro')\n# recall = recall_score(y_test, predictions, average='micro')\n# f1 = f1_score(y_test, predictions, average='micro')\n \n# print(\"Micro-average quality numbers\")\n# print(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\n# precision = precision_score(y_test, predictions, average='macro')\n# recall = recall_score(y_test, predictions, average='macro')\n# f1 = f1_score(y_test, predictions, average='macro')\n \n# print(\"Macro-average quality numbers\")\n# print(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\n# print (metrics.classification_report(y_test, predictions))\n# print(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:54:11.198073Z","iopub.status.idle":"2021-09-22T07:54:11.198670Z","shell.execute_reply.started":"2021-09-22T07:54:11.198343Z","shell.execute_reply":"2021-09-22T07:54:11.198372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.3) Try OneVsRest Classifier with Linear SVM (Loss-Hinge)","metadata":{}},{"cell_type":"code","source":"# Due to memory constraints, won't be able to run this\n# start = datetime.now()\n# classifier_5 = OneVsRestClassifier(SGDClassifier(loss='hinge', alpha=0.00001, penalty='l1'), n_jobs=-1)\n# classifier_5.fit(x_train_multilabel, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-09-22T07:54:11.200750Z","iopub.status.idle":"2021-09-22T07:54:11.201382Z","shell.execute_reply.started":"2021-09-22T07:54:11.201087Z","shell.execute_reply":"2021-09-22T07:54:11.201115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Saving the Classifier\n# joblib.dump(classifier_5, 'lr_with_more_title_weight_5.pkl') \n\n# Loading the Classifier\n# classifier_5 = joblib.load('../input/d/elemento/facebook-recruiting-iii-keyword-extraction/lr_with_more_title_weight_5.pkl') ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# predictions = classifier_5.predict(x_test_multilabel)\n\n# print(\"Accuracy :\",metrics.accuracy_score(y_test, predictions))\n# print(\"Hamming loss \",metrics.hamming_loss(y_test,predictions))\n\n# precision = precision_score(y_test, predictions, average='micro')\n# recall = recall_score(y_test, predictions, average='micro')\n# f1 = f1_score(y_test, predictions, average='micro')\n \n# print(\"Micro-average quality numbers\")\n# print(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\n# precision = precision_score(y_test, predictions, average='macro')\n# recall = recall_score(y_test, predictions, average='macro')\n# f1 = f1_score(y_test, predictions, average='macro')\n \n# print(\"Macro-average quality numbers\")\n# print(\"Precision: {:.4f}, Recall: {:.4f}, F1-measure: {:.4f}\".format(precision, recall, f1))\n\n# print (metrics.classification_report(y_test, predictions))\n# print(\"Time taken to run this cell :\", datetime.now() - start)","metadata":{},"execution_count":null,"outputs":[]}]}