{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# 🎆Encode like there's no tomorrow\n\n**This kernel will be a combination of a variety of encoding techniques, pipelines and models with the aim of achieving a better score**\nI have exported the encoded dataset which I made in this notebook, feel free to use it in your pipelines and dont forget to credit this notebook. 😉\nHope this notebook is as fun to read as it was for me to write."},{"metadata":{},"cell_type":"markdown","source":"# 1. Data\nI will be using a custom dataset which I created. This contains the contest data which is cleaned, preprocessed and encoded. The different types of encoding used are as follows:\n"},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\ntest = pd.read_csv(\"../input/cat-in-the-dat-ii/test.csv\")\ntrain = pd.read_csv(\"../input/cat-in-the-dat-ii/train.csv\")\ntest_encoded = pd.read_csv(\"../input/catindat2encoded/test_encoded.csv\")\ntrain_encoded = pd.read_csv(\"../input/catindat2encoded/train_encoded.csv\")\ntest_id = pd.read_csv(\"../input/cat-in-the-dat-ii/sample_submission.csv\")['id']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Empty data was filled with mode for respective columns"},{"metadata":{"trusted":true},"cell_type":"code","source":"target = train['target']\ntest.drop(['id'], axis =1 , inplace = True)\ntrain.drop(['id','target'], axis =1 , inplace = True)\ndata = pd.concat([train,test])\nfor i in data.columns:\n    data[i] = data[i].fillna(data[i].mode()[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Binary data to boolean values using the function below"},{"metadata":{"trusted":true},"cell_type":"code","source":"def bin_encoder(integer):\n    if integer == 0 or integer == 'N' or integer == 'F':\n        return False\n    elif integer ==1 or integer == 'Y' or integer== 'T' :\n        return True\nbin = ['bin_0','bin_1','bin_2','bin_3','bin_4']\nfor i in bin:\n    data[i] = data[i].apply(bin_encoder)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* nom_0 to nom_4 were Label Encoded\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nfor i in range(0,5):\n    l_encoder = LabelEncoder()\n    key = \"nom_\" + str(i)\n    data[key] = l_encoder.fit_transform(data[key].fillna(\"NULL\").astype(str).values) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* nom_5 to nom_9 were target encoded due their high cardinality value"},{"metadata":{"trusted":true},"cell_type":"code","source":"from category_encoders import TargetEncoder\nfor i in range(5,10):\n    target_encoder = TargetEncoder()\n    key = \"nom_\" + str(i)\n    train_te = target_encoder.fit_transform(train[key].fillna(\"NULL\").astype(str).values,target).values\n    test_te = target_encoder.transform(test[key].fillna(\"NULL\").astype(str).values).values\n    data[key] =np.concatenate((train_te,test_te), axis=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* ordinal variables 0 to 3 were mapped using dicts and assigned values corresponding to their interpreted values"},{"metadata":{"trusted":true},"cell_type":"code","source":"ordinal = [\n    [1.0, 2.0, 3.0],\n    ['Novice', 'Contributor', 'Expert', 'Master', 'Grandmaster'],\n    ['Freezing', 'Cold', 'Warm', 'Hot', 'Boiling Hot', 'Lava Hot']\n]\n\nfor i in range(1, 3):\n    ordinal_dict = {i : j for j, i in enumerate(ordinal[i])}\n    key = \"ord_\" + str(i)\n    data[key] = (data[key]).map(ordinal_dict)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The alphabetical ordinal variables are encoded using Ordinal Encoder"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import OrdinalEncoder\noencoder = OrdinalEncoder(dtype=np.int16)\nfor enc in [\"ord_3\",\"ord_4\",\"ord_5\"]:\n    data[enc] = oencoder.fit_transform(np.array(data[enc]).reshape(-1,1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Feature generation from cyclic variables**\n* After reading a few notebooks I thought we should generate a few more features to extract information from day and month"},{"metadata":{"trusted":true},"cell_type":"code","source":"data['month_sin'] = np.sin((data['month'] - 1) * (2.0 * np.pi / 12))\ndata['month_cos'] = np.cos((data['month'] - 1) * (2.0 * np.pi / 12))\n\ndata['day_sin'] = np.sin((data['day'] - 1) * (2.0 * np.pi / 7))\ndata['day_cos'] = np.cos((data['day'] - 1) * (2.0 * np.pi / 7))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Scaling data to get normalized values"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler\nscaler= MinMaxScaler()\ndata = scaler.fit_transform(data)\ndata = pd.DataFrame(data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Exporting our newly made dataset to csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.concat([data[:600000],target],axis =1)\ntrain.to_csv(\"train_encoded.csv\",index = False)\ntest = data[600000:]\ndata[600000:].to_csv(\"test_encoded.csv\",index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Borrowing a bit of code from my [previous notebook](https://www.kaggle.com/amoghjrules/intro-to-stacking-averaging-base-models)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm\nfrom sklearn.base import BaseEstimator, TransformerMixin, RegressorMixin, clone\nclass average_stacking(BaseEstimator, RegressorMixin, TransformerMixin):\n    def __init__(self,models):\n        self.models = models\n    def fit(self, x,y):\n        self.model_clones = [clone(x) for x in self.models]\n        \n        for model in tqdm(self.model_clones):\n            model.fit(x,y)\n        return self\n    def predict(self, x):\n        preds = np.column_stack([\n            model.predict(x) for model in tqdm(self.model_clones)\n        ])\n        return np.mean(preds, axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.ensemble import GradientBoostingRegressor, AdaBoostClassifier\nfrom sklearn.base import BaseEstimator, TransformerMixin, RegressorMixin, clone\nfrom sklearn import linear_model\nimport xgboost as xgb\nimport lightgbm as lgb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"glm = linear_model.LogisticRegression( random_state=1, solver='lbfgs', max_iter=2020, fit_intercept=True, penalty='none', verbose=0)\nmodel_lgb = lgb.LGBMRegressor(objective='regression',num_leaves=5,\n                              learning_rate=0.05, n_estimators=920,\n                              max_bin = 55, bagging_fraction = 0.8,\n                              bagging_freq = 5, feature_fraction = 0.2319,\n                              feature_fraction_seed=9, bagging_seed=9,\n                              min_data_in_leaf =6, min_sum_hessian_in_leaf = 11)\nGBoost = GradientBoostingRegressor(n_estimators=1500, learning_rate=0.05,\n                                   max_depth=4, max_features='sqrt',\n                                   min_samples_leaf=15, min_samples_split=10, \n                                   verbose = True,\n                                   loss='huber', random_state =5)\nmodel_ada = AdaBoostClassifier(n_estimators= 2200, learning_rate= 0.75)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"averaged_models = average_stacking(models = (model_lgb, glm, GBoost, model_ada))\ntrain_y = train_encoded['target']\ntrain_encoded.drop(['target'], axis =1 , inplace = True)\n# averaged_models.fit(train_encoded, train_y)\n# avg_pred = averaged_models.predict(test_encoded)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"# pd.DataFrame({'id': test_id, 'target': avg_preds}).to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <span style=\"color:red\">Upvote and share if this notebook helped you in any way</span> 😁"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}