{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EMSI Tabular Competition\n\n    This is a notebook containing one of the solutions of the EMSI Tabular Competition.\nThe dataset that we are going to manipulate is about the state of machines. So we need to create a model that predicts the state of machines ( Turned on or Turned off).","metadata":{}},{"cell_type":"markdown","source":"#### First let's import the libraries that we are going to use","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport pandas as pd\nfrom tensorflow import keras\nfrom sklearn.metrics import accuracy_score\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.feature_selection import mutual_info_classif\nimport optuna\nfrom lightgbm import LGBMClassifier\nfrom sklearn import metrics\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression, LinearRegression\nfrom category_encoders import MEstimateEncoder","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploring the data","metadata":{}},{"cell_type":"markdown","source":"##### let's import the datasets that we are going to use \n    train : the dataset that we are going to train.\n    test : The dataset that we are required to predict.","metadata":{}},{"cell_type":"code","source":"\ntrain = pd.read_csv(\"train.csv\")\ntest = pd.read_csv(\"test.csv\")\ntrain.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After printing the 5 first columns. We can see that whe have an id column, 30 numerical ones and 1 categorical column and finally a Target column (the one we hate to predict)\nNow Let's try to know more information about the columns","metadata":{}},{"cell_type":"code","source":"train.info()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So as we can see, we have one categorical column and the dataset doesn't contain null values.\n\nAlso we have the column \"id\" we don't need the model to take it into consideration in his training. So we better use this column as the index of the dataset. ","metadata":{}},{"cell_type":"code","source":"train = train.set_index('id')\ntrain.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's see how many machines are  turned off and how many are turned on.","metadata":{}},{"cell_type":"code","source":"train['target'].value_counts().plot(kind='bar')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see thats we have a good dataset because there is no big difference between the true state and false one.","metadata":{}},{"cell_type":"markdown","source":"    Now lets try to see if there is some correlation between the columns","metadata":{}},{"cell_type":"code","source":"\n#Correlation table\ncorr = train.corr()\nplt.figure(figsize=(18,18))\nsns.heatmap(corr, linewidth=0.1, fmt='.2f', annot=True, annot_kws={'size': 8})","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The correlation heatmap shows that barely any of the features are correlated. The highest values are around 30-33%.","metadata":{}},{"cell_type":"markdown","source":"#### Now let's try to understand the categorical column","metadata":{}},{"cell_type":"code","source":"train.f_27.value_counts()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.f_27.value_counts().nunique()\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well the the column have 11 unique value and there are at most 15 frequencies.","metadata":{}},{"cell_type":"markdown","source":"Since the column have different letters. Let's try to split then : each letter on a separate column ( We have 10 letters)","metadata":{}},{"cell_type":"code","source":"#Create new features\nnew_features = [f'f_27{i}' for i in 'abcdefghij']\ntrain[new_features] = train.f_27.str.split('', expand=True).loc[:,1:10]\n\ntrain[new_features].head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's plot each column for a better understanding.\n\nWe are going to to split the dataset in half because we have a lot of data and it will use a lot of memory.","metadata":{}},{"cell_type":"code","source":"#Split numerical columns in half for easier plotting\nnum_cols = list(train.select_dtypes(exclude=['object']))\nmidpoint = len(num_cols) // 2\ncols_first_half = num_cols[:midpoint]\ncols_second_half = num_cols[midpoint:]\n\n#Plot the histogram for each feture on each axis\nfig, ax = plt.subplots(4, 4, figsize=(20,20))\n\n#First half\nfor ax, col in zip(ax.flatten(), cols_first_half):\n    sns.histplot(train[col], kde=True, ax=ax)\n\nfig.tight_layout()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#Plot the histogram for each feture on each axis\nfig, ax = plt.subplots(4, 4, figsize=(20,20))\n\n#First half\nfor ax, col in zip(ax.flatten(), cols_second_half):\n    sns.histplot(train[col], kde=True, ax=ax)\n\nfig.tight_layout()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see: \n\n    -  We have some normally distributed data ( showing that data near the mean are more frequent in occurrence than data far from the mean) \n\n    -  We also have a column that can be a boolean features ( f_29).\n\n    -  f_30 can be a multiclass feature.\n\n    -  Histograms with multiple peaks show that the data is multimodal ( meaning that  it has more than one value that occurs most frequently and this can occur if the data comes from multiple distinct groups or populations.) \n\n","metadata":{}},{"cell_type":"markdown","source":"Exemple of a multi model data : \n\nSuppose that you have a dataset that consists of the heights of a group of people. If the histogram of this data has two peaks, it could be because the group of people consists of both men and women, and the two peaks correspond to the different average heights of the two genders.","metadata":{}},{"cell_type":"markdown","source":"## Training the data","metadata":{}},{"cell_type":"markdown","source":"##### let's define our input and output data. And then split them into train and test data","metadata":{}},{"cell_type":"code","source":"\n\n\n# Separate target from predictors\ny = train.target\nX = train.drop(['target'], axis=1)\nX_train, X_valid, y_train, y_valid = train_test_split(X, y, train_size=0.8, test_size=0.2,random_state=0)\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have to encode now the categorical columns.\nWe will be using the MEstimaeEncoder","metadata":{}},{"cell_type":"code","source":"categorical_cols = [col for col in new_features if X[col].nunique() > 10]\nmestimate = MEstimateEncoder(categorical_cols, m=5).fit(X_train,y_train)\nX_train = mestimate.transform(X_train)\nX_test = mestimate.transform(X_valid)\n\nX_train.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After using different models : We found that the lightGBM classifier is the best one so we will be using it as our model.\n\nThe LightGBM Classifier is a gradient boosting framework that uses decision trees as the base model and is designed to be faster and more efficient than other boosting algorithms.\nIt uses a leaf wise tree growth which means it grows node by node ","metadata":{}},{"cell_type":"markdown","source":"![Alt text](download.png)","metadata":{}},{"cell_type":"markdown","source":"Now we will be using HyperParameters: \n\n We will create a function that will test different parameters for our model and then we will use the best params to fit our model.","metadata":{}},{"cell_type":"markdown","source":"We will use optuna : it is a framework can be used to find the best values for the hyperparameters of a model or algorithm.","metadata":{}},{"cell_type":"code","source":"#Tune hyperparameters\ndef objective(trial):\n    param_grid = dict(n_estimators=trial.suggest_int('n_estimators', 20, 1000, 10), \n                      learning_rate=trial.suggest_float('learning_rate', 0, 1), \n                      max_depth=trial.suggest_int('max_depth', 3, 12))\n    clf = LGBMClassifier(**param_grid)\n    scores = cross_val_score(clf, X_train, y_train, cv=5, scoring='roc_auc')\n    return scores.mean().round(5)\n\nstudy = optuna.create_study(direction='maximize')\n\n#Test different hyperparameters 20 times\nstudy.optimize(objective, n_trials=20, show_progress_bar=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Gest the best parameters\nbest_params = study.best_trial.params\nprint('Best parameters:', best_params)\nprint('Best score: {:.2f}%'.format(study.best_value*100))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see the best parameters are : {'n_estimators': 280, 'learning_rate': 0.29812797422436194, 'max_depth': 5} with a cross val score of 99.93%","metadata":{}},{"cell_type":"markdown","source":"## Prediction\n\n Now let's use these parameters to predict our X_test","metadata":{}},{"cell_type":"code","source":"#Create model with recommended parameters\n\n# clf = LGBMClassifier(**best_params).fit(X_train,y_train)\nclf = LGBMClassifier(n_estimators=280,learning_rate=0.29812797422436194, max_depth= 5).fit(X_train,y_train)\ny_pred = clf.predict_proba(X_test)[:,1]\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\npd.DataFrame(y_pred.round())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_score(y_valid,pd.Series(y_pred.round()))\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see we have a score of 81.70% accuracy and it is a good score compared with other models ( like the xgboost with 77% accuracy)","metadata":{}},{"cell_type":"markdown","source":"## Predcting the Test file","metadata":{}},{"cell_type":"markdown","source":"Well now we will predict our test.csv and put in a dataframe then convert the result to a csv file for submission ","metadata":{}},{"cell_type":"code","source":"\n\nid = test[\"id\"]\ntest = test.set_index('id')\nnew_features = [f'f_27{i}' for i in 'abcdefghij']\ntest[new_features] = test.f_27.str.split('', expand=True).loc[:,1:10]\n\ntest = mestimate.transform(test)\n\ntarget =  clf.predict_proba(test)[:,1]\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(target.round())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rr = pd.DataFrame(target.round(1),  columns=['target']).set_index(id)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(rr)\nrr.to_csv('prediction.csv')","metadata":{},"execution_count":null,"outputs":[]}]}