{"cells":[{"metadata":{},"cell_type":"markdown","source":"![Alt Text](https://media.giphy.com/media/LFiOdYoOlEKac/giphy.gif)"},{"metadata":{},"cell_type":"markdown","source":"# If you have any thoughts or ideas for improvements would I highly appreciate it.\n\n# and if you liked the notebook please give it an upvote "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport pandas as pd\nimport numpy as np\n\n# Datavisulizing \nimport matplotlib.pyplot as plt\nimport seaborn as sns\n# import seaborn as sn\nimport missingno as msno\n\n\nfrom sklearn.impute import SimpleImputer\nimport scipy\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import auc, roc_curve\nfrom sklearn.model_selection import StratifiedKFold, GridSearchCV\nfrom tqdm import tqdm_notebook\n\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.preprocessing import LabelEncoder, OneHotEncoder\nimport category_encoders as ce\n\nfrom sklearn import metrics\n\nimport lightgbm as lgb\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"raw_train = pd.read_csv('../input/cat-in-the-dat-ii/train.csv', index_col='id')\nraw_test = pd.read_csv('../input/cat-in-the-dat-ii/test.csv', index_col='id')\nraw_submission = pd.read_csv('../input/cat-in-the-dat-ii/sample_submission.csv', index_col='id')\nprint(raw_train.shape, raw_test.shape, raw_submission.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#  ***Agenda***\n## Datavisualization \n    *   Identify key features \n    *   Identify columns with missing values\n    *   Identify Trends, Relationships and Distribution in the dataset\n## Feature engineering\n    *   Target encoding \n    *   Count encoding \n    *   Catboost encoding \n    *   Label encoding  \n    *   f_classif \n## Model training \n    *   RandomForest \n    *   LogisticRegression\n    *   XGBoost \n    *   LightGBM \n"},{"metadata":{},"cell_type":"markdown","source":"## Datavisualization \n>     *   Identify key feature"},{"metadata":{"trusted":true},"cell_type":"code","source":"raw_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"raw_train.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Description of the content in each column\n### Binary features\n* zerosundones_0 = contain only zero (counted enteries 528377) and ones (counted enteries 53729)\n* zerosundones_1 = contain only zero (counted enteries 474018) and ones (counted enteries 107979)\n* zerosundones_2 = contain only zero (counted enteries 419845) and ones (counted enteries 162225)\n* FvsT = contain only F (counted enteries 366212) and T (counted enteries 215774)\n* NvsY = contain only N (counted enteries 312344) and Y (counted enteries 269609)\n\n### Nominal features\n* colors = contain  three colors Red, Blue, Green\n* trigonometry = contain Triangle, Polygon, Circle, Star, Trapezoid, Square\n* animals = contain Hamster, Axolotl, Dog, Lion, Snake, Cat\n* contries =  contain Russia, Canada, Finland, Costa Rica, Snake, India, China\n* instruments = contain Bassoon, Theremin, Theremin, Oboe, Piano, India, China\n* random_0 = contain what seems like randomized letter and number combinations \n* random_1 = contain what seems like randomized letter and number combinations \n* random_2 = contain what seems like randomized letter and number combinations \n* random_3 = contain what seems like randomized letter and number combinations \n* random_4 = contain what seems like randomized letter and number combinations \n\n### Ordinal features \n* oneTwoTree = contain the numbers 1.0(counted enteries 227917), 2.0(counted enteries 155997), 3.0(counted enteries 197798)\n* competetitions_levels = contain Contributor, Grandmaster, Novice, Expert, Master\n* temperature = contain Hot, Warm, Freezing, Lava Hot, Cold, boiling Hot\n* alpha_0 = contain the letters in the alphabet (counted enteries 582084) \n* alpha_1 = contain the letters in the alphabet (counted enteries 582070) \n* alpha_2 = contain the letters in the alphabet (counted enteries 582287) \n\n### Potentially cyclical\n* day = contain the seven days; 1.0 (enteried 84724), 2.0 (enteries 65495), 3.0 (enteries 113835), 4.0 (enteries 23663), 5.0 (enteries 110464), 6.0 (enteries 97432), 7.0 (enteries 86435) \n* month =contain the 12 months; 1.0 (enteries 52154), 2.0 (enteries 40700), 3.0 (enteries 70160), 4.0 (enteries 14614), 5.0 (enteries 68906), 6.0 (enteries 60478), 7.0 (enteries 53480), 8.0 (enteries 79245), 9.0 (enteries 20620), 10.0 (enteries 2150), 11.0 (enteries 51165), 12.0 (enteries 68340)\n\n\n* target = contain zeros (enteries 487677) and ones (enteries 112323)\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Rename columns for traningset \nstructuret_train = raw_train.copy();\n\nstructuret_train = structuret_train.rename(columns = {'bin_0':'zerosundones_0'});\nstructuret_train = structuret_train.rename(columns = {'bin_1':'zerosundones_1'});\nstructuret_train = structuret_train.rename(columns = {'bin_2':'zerosundones_2'});\nstructuret_train = structuret_train.rename(columns = {'bin_3':'FvsT'});\nstructuret_train = structuret_train.rename(columns = {'bin_4':'NvsY'});\n\nstructuret_train = structuret_train.rename(columns = {'nom_0': 'colors'});\nstructuret_train = structuret_train.rename(columns = {'nom_1': 'trigonometry'});\nstructuret_train = structuret_train.rename(columns = {'nom_2': 'animals'});\nstructuret_train = structuret_train.rename(columns = {'nom_3': 'contries'});\nstructuret_train = structuret_train.rename(columns = {'nom_4': 'instruments'});\nstructuret_train = structuret_train.rename(columns = {'nom_5': 'random_0'});\nstructuret_train = structuret_train.rename(columns = {'nom_6': 'random_1'});\nstructuret_train = structuret_train.rename(columns = {'nom_7': 'random_2'});\nstructuret_train = structuret_train.rename(columns = {'nom_8': 'random_3'});\nstructuret_train = structuret_train.rename(columns = {'nom_9': 'random_4'});\n\nstructuret_train = structuret_train.rename(columns = {'ord_0': 'oneTwoTree'});\nstructuret_train = structuret_train.rename(columns = {'ord_1': 'competetitions_levels'});\nstructuret_train = structuret_train.rename(columns = {'ord_2': 'temperature'});\nstructuret_train = structuret_train.rename(columns = {'ord_3': 'alpha_0'});\nstructuret_train = structuret_train.rename(columns = {'ord_4': 'alpha_1'});\nstructuret_train = structuret_train.rename(columns = {'ord_5': 'alpha_2'});","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Rename columns for testset \nstructuret_test = raw_test.copy();\n\nstructuret_test = structuret_test.rename(columns = {'bin_0':'zerosundones_0'});\nstructuret_test = structuret_test.rename(columns = {'bin_1':'zerosundones_1'});\nstructuret_test = structuret_test.rename(columns = {'bin_2':'zerosundones_2'});\nstructuret_test = structuret_test.rename(columns = {'bin_3':'FvsT'});\nstructuret_test = structuret_test.rename(columns = {'bin_4':'NvsY'});\n\nstructuret_test = structuret_test.rename(columns = {'nom_0': 'colors'});\nstructuret_test = structuret_test.rename(columns = {'nom_1': 'trigonometry'});\nstructuret_test = structuret_test.rename(columns = {'nom_2': 'animals'});\nstructuret_test = structuret_test.rename(columns = {'nom_3': 'contries'});\nstructuret_test = structuret_test.rename(columns = {'nom_4': 'instruments'});\nstructuret_test = structuret_test.rename(columns = {'nom_5': 'random_0'});\nstructuret_test = structuret_test.rename(columns = {'nom_6': 'random_1'});\nstructuret_test = structuret_test.rename(columns = {'nom_7': 'random_2'});\nstructuret_test = structuret_test.rename(columns = {'nom_8': 'random_3'});\nstructuret_test = structuret_test.rename(columns = {'nom_9': 'random_4'});\n\nstructuret_test = structuret_test.rename(columns = {'ord_0': 'oneTwoTree'});\nstructuret_test = structuret_test.rename(columns = {'ord_1': 'competetitions_levels'});\nstructuret_test = structuret_test.rename(columns = {'ord_2': 'temperature'});\nstructuret_test = structuret_test.rename(columns = {'ord_3': 'alpha_0'});\nstructuret_test = structuret_test.rename(columns = {'ord_4': 'alpha_1'});\nstructuret_test = structuret_test.rename(columns = {'ord_5': 'alpha_2'});","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":">     *   Identify columns with missing values"},{"metadata":{"trusted":true},"cell_type":"code","source":"msno.matrix(raw_train, figsize = (30,5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"investigate_structuret_train = pd.DataFrame({'columns':structuret_train.columns})\ninvestigate_structuret_train['data_type'] = structuret_train.dtypes.values\ninvestigate_structuret_train['missing_val'] = structuret_train.isnull().sum().values \ninvestigate_structuret_train['uniques'] = structuret_train.nunique().values\ninvestigate_structuret_train\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will fill missing value in the section below (feature engineering)"},{"metadata":{},"cell_type":"markdown","source":">     *   Identify Trends, Relationships and Distribution in the dataset"},{"metadata":{},"cell_type":"markdown","source":"### The target value distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"target_dist = structuret_train.target.value_counts()\n\nbarplot = plt.bar(target_dist.index, target_dist, color = 'darkred', alpha = 0.8)\nbarplot[0].set_color('darkblue')\n\nplt.xlabel('Target', fontsize = 18)\n\nplt.show()\nprint(\"percentage of the target: {}%\".format(structuret_train.target.sum() / len(structuret_train.target)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like target ratio is unbalanced."},{"metadata":{},"cell_type":"markdown","source":"### Heatmap of the trainings data"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,10))\ncols = raw_train.select_dtypes(exclude=['object']).columns\ndata = raw_train[cols].corr()\nsns.heatmap(data, \n            xticklabels=data.columns.values,\n            yticklabels=data.columns.values)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It doesnt seems like there is any significant in the heatmap"},{"metadata":{},"cell_type":"markdown","source":"## Feature engineering\n    * Defining functions inorder to validate the most effective encoding"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Diving the data into train and validation and selecting ther target value \ndef feature_target(DataFrame):\n    \n    DataFrame.fillna(DataFrame.median(), inplace = True)\n    \n    y = DataFrame.target \n    X = DataFrame.drop(['target'], axis = 1)\n    X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.20, random_state=0)\n    \n    return X_train, X_valid, y_train, y_valid ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def test_method(X_train, X_valid, y_train, y_valid):\n    \n    train = lgb.Dataset(X_train, label=y_train)\n    valid = lgb.Dataset(X_valid, label=y_valid)\n    \n    param = {'num_leaves': 64, 'objective': 'binary', \n             'metric': 'auc', 'seed': 7}\n    \n    model_bst = lgb.train(param, train, num_boost_round=1000, valid_sets=[valid], \n                    early_stopping_rounds=10, verbose_eval=False)\n\n    valid_pred = model_bst.predict(X_valid)\n    \n    mae = mean_absolute_error(y_valid, valid_pred)\n    \n    print(f\"Validation MAE score: {mae:.4f}\")\n    \n    return model_bst","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#creating a list with all the categorical data with relatively low cardinality\ncategorical_data = [cat for cat in structuret_train if\n                    structuret_train[cat].nunique() < 10 and \n                    structuret_train[cat].dtype == \"object\"]\n\ncategorical_data_test = [cat for cat in structuret_test if\n                    structuret_test[cat].nunique() < 10 and \n                    structuret_test[cat].dtype == \"object\"]\n\n# We want to find the numerical data\nnumerical_data = [num for num in structuret_train if structuret_train[num].dtype in ['int64', 'float64']]\nnumerical_data_test = [num for num in structuret_test if structuret_test[num].dtype in ['int64', 'float64']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# making a copy of the renamed columns from raw_train(the original trainings data)\nlabel_train = structuret_train.copy() \nonehot_train = structuret_train.copy()\ncount_train = structuret_train.copy()\ntarget_train = structuret_train.copy()\ncatboost_train = structuret_train.copy()\n\n# making a copy of the renamed columns from raw_test(the original test data)\nlabel_test = structuret_test.copy() \nonehot_test = structuret_test.copy()\ncount_test = structuret_test.copy()\ntarget_test = structuret_test.copy()\ncatboost_test = structuret_test.copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ***Label encoding***\n\nLabel encoding assigns each unique value to a different integer.\nWe would see that in the temperature feature: \"Hot\" (0) < \"boiling hot\" (1) < \"lava hot\" (2) < \"cold\" (3) ect. \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# converting all enteries to strings\nlabel_train[categorical_data] = label_train[categorical_data].astype(str)  \nlabel_test[categorical_data] = label_test[categorical_data].astype(str) \n    \n# Label encoding\ncat_features = categorical_data\nencoder = LabelEncoder()\n\nlabel_encoded_train = label_train[cat_features].apply(encoder.fit_transform)\nlabel_encoded_test = label_test[cat_features].apply(encoder.fit_transform)\n\nlabel_train = label_train[numerical_data].join(label_encoded_train)\nlabel_test = label_test[numerical_data_test].join(label_encoded_test)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_t, X_v, y_t, y_v = feature_target(label_train)\nbst_results_label = test_method(X_t, X_v, y_t, y_v)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***Count Encoding***\n\nCount encoding take the number of times a categorical values that has occurred and replace it with a number. \nThis happens example if the value \"grandmaster\" occured 10 times in the competetion_level feature, then \"grandmaster\" would be replaced with the number 10.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"count_train[categorical_data] = count_train[categorical_data].astype(str)\ncount_train[numerical_data] = count_train[numerical_data].astype(float)\ncount_test[categorical_data_test] = count_test[categorical_data_test].astype(str)\ncount_test[numerical_data_test] = count_test[numerical_data_test].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Count encoding \ncat_features = categorical_data\n\ncount_enc = ce.CountEncoder()\n\ncount_encoded_train = count_enc.fit_transform(count_train[cat_features])\ncount_encoded_test = count_enc.fit_transform(count_test[cat_features])\n\ncount_encoded_train = label_train.join(count_encoded_train.add_suffix(\"_count\"))\ncount_encoded_test = label_test.join(count_encoded_test.add_suffix(\"_count\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_t, X_v, y_t, y_v = feature_target(count_encoded_train)\nbst_results_count = test_method(X_t, X_v, y_t, y_v)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***Target encoding ***\n\nTarget encoding takes the average of the target value and replaces the categorical value for that value of feature \nWe see that in animals values where we would calculate the average value for all the rows with animals == \"Hamster\", animals == \"dogs\", animals == \"cats\" ect.   "},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_features = categorical_data\n\ntarget_enc = ce.TargetEncoder(cols=cat_features)\n\ntarget_encoded_train = target_enc.fit_transform(target_train[cat_features], target_train.target)\ntarget_encoded_test = target_enc.transform(target_test[cat_features])\n\ntarget_encoded_train = count_encoded_train.join(target_encoded_train.add_suffix(\"_target\"))\ntarget_encoded_test = count_encoded_test.join(target_encoded_test.add_suffix(\"_target\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_t, X_v, y_t, y_v = feature_target(target_encoded_train)\nbst_results_target = test_method(X_t, X_v, y_t, y_v)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***catboost encoding***\n\ncatboost encoding is similar to target encoding since it takes the target probablity for a value. But the major difference between them is that catboost takes the probability for each row and calculate it from the rows before it"},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_features = categorical_data\n\ncatboost_enc = ce.CatBoostEncoder(cols=cat_features)\n\ncatboost_encoded_train = catboost_enc.fit_transform(catboost_train[cat_features], catboost_train.target)\ncatboost_encoded_test = catboost_enc.transform(catboost_test[cat_features])\n\ncatboost_encoded_train = count_encoded_train.join(catboost_encoded_train.add_suffix(\"_catboost\"))\ncatboost_encoded_test = count_encoded_test.join(catboost_encoded_test.add_suffix(\"_catboost\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_t, X_v, y_t, y_v = feature_target(catboost_encoded_train)\nbst_results_catboost = test_method(X_t, X_v, y_t, y_v)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Catboost dosent seem to a positive effect on the result therefore we will continue with the target_encoded data "},{"metadata":{},"cell_type":"markdown","source":">     *   f_classif \n        Drop feature that is not needed "},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.feature_selection import SelectKBest, f_classif\n\n\nfeature_cols = target_encoded_train.columns.drop('target')\nX_t, X_v, y_t, y_v = feature_target(target_encoded_train)\n\n# Keeping 14 features\nselector = SelectKBest(f_classif, k=13)\n\nX_new = selector.fit_transform(X_t, y_t)\nX_new","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get back the features we want to kept\nselected_features = pd.DataFrame(selector.inverse_transform(X_new), \n                                 index=X_t.index, \n                                 columns=feature_cols)\nselected_features.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dropping columns that has zero values\nselected_columns = selected_features.columns[selected_features.var() != 0]\n\n# using valid for the selected features.\nX_v[selected_columns].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Testing the effect of dropping the feature that has no influence on the target value \ninvestigate = pd.DataFrame(target_encoded_train[selected_columns])\ninvestigate['target'] = target_encoded_train.target\n\ninvestigate_test = pd.DataFrame(target_encoded_test[selected_columns])\n\nX_t, X_v, y_t, y_v = feature_target(investigate)\nbst = test_method(X_t, X_v, y_t, y_v)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like the result got slightly worse, but an observation reveales that the number of columns we keep influence the result greatly \n\nUsing target_encoded data which has 34 columns (include; count and target encoding) is an impressive amount af data to modelling on, eventhough the lightgbm model gives us a mae on 0.2824. \n\nThe investigate only contain 14 columns (include a combination of count and target values)\n\nThe models below will be trained on investigate, but if we cant get the result under 0.2824\n\nwe will use the target_encoded data in the final submission "},{"metadata":{},"cell_type":"markdown","source":"## ***Model training***"},{"metadata":{},"cell_type":"markdown","source":"* RandomForest\n\nRandom forest consists of individual decision trees that operate as an ensemble. \nMoreover it consists of leafs which is the point at the bottom where we make a prediction and the tree's depth is a measure of how many splits it makes before coming to a prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"# optimizing model RandomForestRegressor\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.datasets import make_regression\n\nRFR_model = RandomForestRegressor(n_estimators = 100, random_state = 0)\n\nX_train_1, X_valid_1, y_train_1, y_valid_1 = feature_target(investigate)\n\nRFR_model.fit(X_train_1, y_train_1)  \npreds_1 = RFR_model.predict(X_valid_1) \nRFR_mae = mean_absolute_error(preds_1, y_valid_1) \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"RFR_mae","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* LogisticRegression \n\nLogistic regression can describe ones data and explain the relationship between dependent binary variable and one or more nominal and ordinal vaeiables \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\n\nX_train_2, X_valid_2, y_train_2, y_valid_2 = feature_target(investigate)\n                                                    \nLR_model = LogisticRegression(C=0.03, max_iter=300)\nLR_model.fit(X_train_2, y_train_2)\npreds_2 = LR_model.predict_proba(X_valid_2)[:, 1]\n\nLR_mae = mean_absolute_error(preds_2, y_valid_2) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"LR_mae","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* XGBoost \n\nGradient boosting a method that goes through cycles to iteratively add models into an ensemble.\n\nGradient boosting work by \n\nIt begins by initializing the ensemble with a single model, whose predictions can be pretty naive\n\n1. use the current ensemble to generate predictions for each observation in the dataset\n2. These predictions are used to calculate a loss function\n3. use the loss function to fit a new model that will be added to the ensemble\n4. we add the new model to ensemble, and ...\n5. ... repeat!"},{"metadata":{"trusted":true},"cell_type":"code","source":"import xgboost as xgb\n\nX_train_3, X_valid_3, y_train_3, y_valid_3 = feature_target(investigate)\n\nxgb_model = xgb.XGBClassifier(max_depth=20,n_estimators=2020,colsample_bytree=0.20,learning_rate=0.020,objective='binary:logistic', n_jobs=-1)\n\nxgb_model.fit(X_train_3, y_train_3,eval_set=[(X_valid_3,y_valid_3)],verbose=0,early_stopping_rounds=200\n) \n\n# preds_3 = xgb_model.predict(X_valid_3) \npreds_3 = xgb_model.predict_proba(X_valid_3)[:,1]\nxgb_mae = mean_absolute_error(preds_3, y_valid_3) \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"xgb_mae","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* LightGBM \n\nGradient boosting uses tree based learning to optimize ones model "},{"metadata":{"trusted":true},"cell_type":"code","source":"import lightgbm as lgb\n\nX_train_4, X_valid_4, y_train_4, y_valid_4 = feature_target(investigate)\n\ntrain = lgb.Dataset(X_train_4, label=y_train_4)\nvalid = lgb.Dataset(X_valid_4, label=y_valid_4)\n    \nparam = {'num_leaves': 64, 'objective': 'binary', \n             'metric': 'auc', 'seed': 7}\n    \nlgb_model = lgb.train(param, train, num_boost_round=1000, valid_sets=[valid], \n                    early_stopping_rounds=10, verbose_eval=False)\n\npreds_4 = lgb_model.predict(X_valid_4)\n    \nlgb_mae = mean_absolute_error(y_valid_4, preds_4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lgb_mae","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scores = [RFR_mae, LR_mae, xgb_mae, lgb_mae]\npd.DataFrame(np.array([scores]),\n                   columns=['RFR_mae', 'LR_mae', 'xgb_mae', 'lgb_mae'])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In conclusion; The best model is lgb_model"},{"metadata":{},"cell_type":"markdown","source":"## Submission "},{"metadata":{"trusted":true},"cell_type":"code","source":"test_pred_0 = lgb_model.predict(investigate_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/sample_submission.csv')\nsub_id = submission_df['id']\nsubmission = pd.DataFrame({'id':sub_id})\nsubmission['target'] = test_pred_0\nsubmission.to_csv(\"submission5.csv\",index = False)\nprint('Model ready for submission!')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}