{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Developing ML Model for CTR Prediction\n\n## 1 . Introduction\nIn Internet marketing, click-through rate (CTR) is a metric that measures the number of clicks advertisers receive on their ads per number of impressions. In this tutorial, I’ll walk you through an example of predicting CTR. I will use the dataset from a Kaggle competition, [Click-Through Rate Prediction](https://www.kaggle.com/competitions/avazu-ctr-prediction), sponsored by Avazu.In this research analysis, Criteo Labs is sharing 10 days’ worth of Avazu data for us to develop models predicting ad click-through rate (CTR). Given a user and the page he is visiting. what is the probability that the user will click on a given ad? The goal of this analysis is to make explarotary data anlaysis of given dataset and finally develop a ML model for prediction.","metadata":{}},{"cell_type":"markdown","source":"## 2. Exploratory Data Analysis\n\nAvazu dataset comprises of over 40 millions of rows and 24 columns.The dataset has the following features:\n\n* id: ad identifier\n* click: 0 for non-click, 1 for click\n* hour: in the format of YYMMDDHH\n* C1: some anonymized categorical variable e.g. 1002\n* banner_pos: where a banner is located, 1 and 0\n* site_id: site identifier\n* site_domain: hashed site domain\n* site_category: hashed site category e.g. 28905ebd\n* app_id: mobile app identifier\n* app_domain\n* app_category\n* device_id: mobile device identifier\n* device_ip: IP address\n* device_model: hashed model e.g. iPhone 6, Samsung\n* device_type: hashed device type e.g. tablet, smartphone\n* device_conn_type: hashed type of connection e.g. Wi-Fi, 4G\n* C14-C21: some more anonymized categorical variables\n\nWe can group all the features in the data into the following categories:\n\n* Target feature : click\n* site features : site_id, site_domain, site_category\n* app feature: app_id, app_domain, app_category\n* device feature: device_id, device_ip, device_model, device_type, device_conn_type\n* anonymized categorical features: C14-C21\n\n__Anonymized and hashed values:__ these are categorical features and their values correspond to some real and meaningful values. They are presented this way for privacy concerns.","metadata":{}},{"cell_type":"markdown","source":"__Importing Necessary Libraries__","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n%matplotlib inline\nimport math\nfrom sklearn.preprocessing import LabelEncoder \nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.model_selection import cross_val_score\nfrom imblearn.over_sampling import RandomOverSampler\nfrom sklearn.ensemble import RandomForestClassifier\nimport statsmodels.api as sm\nfrom statsmodels.stats.outliers_influence import variance_inflation_factor\nfrom sklearn.feature_selection import RFE\nimport random\nimport gzip\nimport category_encoders as ce","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"__Loading Dataset__","metadata":{}},{"cell_type":"code","source":"import gzip\nimport shutil\n\n#Since the source files are compressed, the files are converted into .csv using the built-in Python function \"gzip\"\n\"\"\"with gzip.open(\"/kaggle/input/avazu-ctr-prediction/train.gz\", \"rb\") as f_in:\n    with open(\"/kaggle/input/avazu-ctr-prediction/train.csv\", \"wb\") as f_out:\n        shutil.copyfileobj(f_in,f_out)\nwith gzip.open(\"/kaggle/input/avazu-ctr-prediction/test.gz\", \"rb\") as f_in:\n    with open(\"/kaggle/input/avazu-ctr-prediction/test.csv\", \"wb\") as f_out:\n        shutil.copyfileobj(f_in,f_out)\n\n# In terms of decreasing computation complexity, \n# a small part of the total dataset will be sampled and analysis will be carried out    \ntrain_df=pd.read_csv(\"./train.csv\", nrows=50000)\ntest_df=pd.read_csv(\"./test.csv\", nrows=50000)\"\"\"\n\nn = 40428967  #total number of records in the clickstream data \nsample_size = 1000000\nskip_values = sorted(random.sample(range(1,n), n-sample_size))\nparse_date = lambda val : pd.datetime.strptime(val, '%y%m%d%H')\nwith gzip.open('/kaggle/input/avazu-ctr-prediction/train.gz') as f:\n    train_df = pd.read_csv(f, parse_dates = ['hour'], date_parser = parse_date, dtype=types_train, skiprows = skip_values)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# view the some data\ntrain_df.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"__Some Descriptive Statistics__","metadata":{}},{"cell_type":"code","source":"# printing and plotting relative frequencies of the click and not click number\n\nprint(train_df['click'].value_counts()/len(train_df))\n\nsns.countplot(x='click',data=train_df, palette='hls')\nplt.show();\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the statistics abouve it can be inferred that the overall click through rate is approx. 17%, and approx. 83% is not clicked. So data is imbalanced","metadata":{}},{"cell_type":"code","source":"# Pearson correlation table to find the relationship with output with all input features. \n\ncorr = train_df.corr()\nf, ax = plt.subplots(figsize=(10,10))\nsns.heatmap(corr, ax=ax, annot=True,linewidths=3,cmap='YlGn')\nplt.title(\"Pearson correlation of Features\", y=1.05, size=15)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"__Observations:__\n\nC14 and C17 categorical features highly corelated.\n\nDevice_type and C1 features have high corelation.\n\nSo we can drop one of highly correlated features for ease of computation.\nAlso we can remove some other non-informative features such as id, device id. etc...","metadata":{}},{"cell_type":"code","source":"#check if the data has any missing value\n\ntrain_df.isnull().sum().sort_values(ascending=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We dont have any missing values any of our columns. So we dont need to deal with missing data.","metadata":{}},{"cell_type":"markdown","source":"## Data Preparation","metadata":{}},{"cell_type":"code","source":"# drop non-informative columns\n\ntrain_df= train_df.drop(\"id\", axis=1)\ntrain_df= train_df.drop(\"device_id\", axis=1)\ntrain_df= train_df.drop(\"device_ip\", axis=1)\ntrain_df= train_df.drop(\"site_id\", axis=1)\ntrain_df= train_df.drop(\"hour\", axis=1)\ntrain_df= train_df.drop([\"C14\", \"C1\"], axis=1)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# split train data for feature and label dataframes\nX= train_df.drop(\"click\",axis=1)\ny=train_df[\"click\"]\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# encode categorical features with Target Encoding. \n#It is useful when teh cardinality of the features are high like our situation\n\ntarget_encoder = ce.TargetEncoder()\nX = target_encoder.fit_transform(X, y)\nX.head(2)\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# employing oversampling method for balancing the data\n\nrandomsample=  RandomOverSampler()\nx_new, y_new = randomsample.fit_resample(X, y)\n\nfrom collections import Counter\nprint('Original dataset shape  {}'.format(Counter(y)))\nprint('Resampled dataset shape {}'.format(Counter(y_new)))\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#split train data randomly for training and testing, \n#It is going to used a 70/30 split — 70% of the data is used for training and 30% for testing:\nX_train, X_test, y_train, y_test = train_test_split(x_new, y_new, stratify= y_new, test_size= 0.3, random_state= 42)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# MinMaxScaler used to reduce the fake impact of large valued features on prediction\nscaler = MinMaxScaler()\nX_train[X_train.columns] = scaler.fit_transform(X_train[X_train.columns])\nX_train.head(2)","metadata":{"scrolled":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Design and Evaluation","metadata":{}},{"cell_type":"code","source":"# Random forest model have been used to prediction model \n\nmodel = RandomForestClassifier(random_state=42)\nmodel.fit(X_train, y_train)\nfeature_importances = model.feature_importances_\nfeatures = X_train.columns\ndf = pd.DataFrame({'features': features, 'importance': feature_importances})\ndf.sort_values(by='importance', ascending = False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this function helps in evaluation the given model and provide accuracy and confusion matrix. \n\ndef model_eval(model, X_test, y_test):\n    y_pred = model.predict(X_test)\n    accuracy = metrics.accuracy_score(y_test, y_pred)\n    #print(accuracy)\n    cnf_matrix = metrics.confusion_matrix(y_test, y_pred)\n    #print(cnf_matrix)\n    #metrics.plot_confusion_matrix(model, X_test, y_test)\n    #plt.show()\n    return accuracy, cnf_matrix","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# experimentation\n\ndf = df[df.importance > 0.02]\nrf_cols = []\nfor col in list(X_train.columns):\n    if col in list(df.features):\n        rf_cols.append(col)\n        \nX_train = X_train[rf_cols]\nX_test = X_test[rf_cols]\n\nmodel_rf = RandomForestClassifier(random_state=42)\nmodel_rf.fit(X_train, y_train)\nprint(rf_cols)\n\nprint(len(rf_cols))\npredict_rf = model_rf.predict(X_train)\npredict_rf_test = model_rf.predict(X_test)\n\naccuracy, cnf_matrix = model_eval(model_rf, X_train, y_train)\nprint(\"Train results\")\nprint(\"accuracy\",accuracy)\nprint(\"cnf_matrix \\n\", cnf_matrix)\n\naccuracy, cnf_matrix = model_eval(model_rf, X_test, y_test)\nprint(\"Test results\")\nprint(\"accuracy\",accuracy)\nprint(\"cnf_matrix \\n\", cnf_matrix)\n\ndraw_roc(model_rf, X_test, y_test)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Result\nAlthough there was a success rate of 78% in the training set in the Random Forest model, which is the first trial, this rate remained at 50% in the test data. When the ROC curve is examined, our model cannot give test results at the desired level. Further research should be done to solve and improve this problem.","metadata":{}}]}