{"cells":[{"metadata":{"_uuid":"5aed118c710654bd2b80e76c8edda85d33fd3ef3"},"cell_type":"markdown","source":"# Outline\n1. [**Data Investigation**](#Data-Investigation**)\n    1. [Count the Missing Values](#Count-the-Missing-Values)\n    1. [Datatypes wise Columns Counts](#Datatypes-wise-Columns-Counts)\n    1. [Investigate the target variable distribution](#Investigate-the-target-variable-distribution)\n    1. [Investigate Statistics of Data](Investigate-Statistics-of-Data)\n    1. [Correlation of dataset](#Correlation-of-dataset)\n2. [**Remove Varible That Contrain Same Class**](#Remove-Varible-That-Contrain-Same-Class)\n3. [**Categorical to Numeric Variable Conversion for Model Training(Label Encoding)**](#Categorical-to-Numeric-Variable-Conversion-for-Model-Training(Label-Encoding)\n4. [**Bayesian Optimization for Best Parameter Tuning**](#Bayesian-Optimization-for-Best-Parameter-Tuning)\n5. [**Parameter as per Understanding of Model**](#Parameter-as-per-Understanding-of-Model)\n6. [**Model Training With KFold Cross Validation**](#Model-Training-With-KFold-Cross-Validation)\n7. [**Best CV Score Return By Model**](#Best-CV-Score-Return-By-Model)\n8. [**Features Importance**](#Features-Importance)\n9. [**Final Submission**](#Final-Submission)"},{"metadata":{"trusted":true,"_uuid":"d8bd440033554a686633710ab3011e3d252f224b","_kg_hide-input":true},"cell_type":"code","source":"\nimport pandas as pd\nimport numpy as np\nimport time\nimport lightgbm as lgb\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.model_selection import KFold\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.ensemble import GradientBoostingRegressor\nfrom xgboost import XGBRegressor\n\n\nimport os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nsns.set(style=\"darkgrid\")\nplt.style.use(\"fivethirtyeight\")\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f653fdab31436c075580d8d47043c2ba8a16f23c"},"cell_type":"code","source":"def load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e6350d20260a6127c2ae2d0e5aeeb89c82e5ecad"},"cell_type":"code","source":"%%time\ntrain_df = load_df(nrows=600000)\ntrain_corr = train_df.copy()\ntest_df = load_df(\"../input/test.csv\", nrows=1000000)\n# train_df.to_csv(\"500000.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eab45dc45202bf05769fa83fd2c1cc5e21dfb4c0"},"cell_type":"code","source":"#","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8bd6de94778f15d895bc3049b617d0592903570"},"cell_type":"markdown","source":"# 1. Data Investigation"},{"metadata":{"_uuid":"9f7b557c801984a83302483dc3c7a10fe5e0a2aa"},"cell_type":"markdown","source":"## 1.1 Count Missing Values\n* We can see that so missing Values are to high.\n* Missing data are a **common occurrence and can have a significant effect** on the conclusions that can be drawn from the** data.**"},{"metadata":{"trusted":true,"_uuid":"17ff9186474d6622bd080543f48aad5980796432","_kg_hide-input":true},"cell_type":"code","source":"train_df.isnull().sum().plot(kind=\"bar\", figsize = (20,8))\nplt.xlabel(\"Count\")\nplt.ylabel(\"Column Name\")\nplt.title(\"Missing Value Count By Column for 500000 Rows\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"69a9aa292325c35d7eadf21bac58c02726286699"},"cell_type":"markdown","source":"## 1.2 Count the Datatypes Columnwise\n\n* We can see that most of columns datatype is **object**.\n* The type of each attribute is important. Strings may need to be converted to floating point values or integers to represent categorical or ordinal values. \n* we can get an idea of the types of attributes by peeking at the raw data. You can also list the data types used by the DataFrame to characterize each attribute using the dtypes property.\n"},{"metadata":{"trusted":true,"_uuid":"793df7b1832649be29fc866cd79015dca2f98081","_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(20,8))\nax = sns.countplot(x=train_df.dtypes, data=train_df)\nplt.xlabel(\"Data Types\")\nplt.ylabel(\"Counts\")\nplt.title(\"Column Count By Datatypes\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1423e86374fbc76e4b990e85252a15c77f97462"},"cell_type":"markdown","source":"## 1.3 Investigate the Target Columns"},{"metadata":{"trusted":true,"_uuid":"18273862ac71daa7d138c9cf5ac82fd15f78fbc2","_kg_hide-input":true},"cell_type":"code","source":"target = train_df['totals.transactionRevenue'].fillna(0).astype(float)\n\ntarget = target.apply(lambda x: np.log(x) if x > 0 else x)\n\n\ndel train_df['totals.transactionRevenue']\n# plt.figure(figsize = (15,10))\nsns.jointplot(target.values,target.index,kind=\"regg\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da0ba1bd4ee01c46589a04685bd49f51687038e5"},"cell_type":"markdown","source":"## 1.4 Investigate the Statistics of Data\n\nDescriptive statistics can give you great insight into the shape of each attribute. Often you can create more summaries than you have time to review. The describe() function on the Pandas DataFrame lists 8 statistical properties of each attribute. They are:\n* Count.\n* Mean.\n* Standard Deviation.\n*  Minimum Value.\n* 25th Percentile.\n* 50th Percentile (Median).\n* 75th Percentile.\n* Maximum Value.\n"},{"metadata":{"_uuid":"e39085db2758a203b199f6c5bf594cd5316c4c57"},"cell_type":"markdown","source":"### Categorical Variable Columns Statistics"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":true,"_uuid":"2462c81f5379b9dffc81ff36f0a07ff19d9fa70c"},"cell_type":"code","source":"train_df.select_dtypes('object').describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa65aee16c19f3497b6b2ff759256dd0b68b4d1e"},"cell_type":"markdown","source":"### Numeric Statistics"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"cc2d59415203097b76fbeb3c9442ba9dc1d18de4"},"cell_type":"code","source":"train_df.select_dtypes(exclude=('object')).describe().boxplot(figsize=(20,8))\ntrain_df.select_dtypes(exclude=('object')).describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"78d5eed56ac47353fc7c9e4fc7cfdd6335f03204"},"cell_type":"markdown","source":"# 1.5 Correlation of dataset\n\n* only for numeric variable"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"4b5b3e6b4c8e2ca651958cba01527c14d9e7817d"},"cell_type":"code","source":"plt.figure(figsize=(15,15))\nsns.heatmap(train_corr.corr(method=\"kendall\"), annot=True)\ntrain_corr.corr(method='kendall').style.format(\"{:.2}\").background_gradient(cmap=plt.get_cmap('coolwarm'), axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"886f137677e007539bfd6f9cd1b7ec39d4742386"},"cell_type":"markdown","source":"# 2. Remove Variable That Contain Same Class"},{"metadata":{"trusted":true,"_uuid":"7697c41982c741bc2168a27c6eeac880297d9c5e"},"cell_type":"code","source":"columns = [col for col in train_df.columns if train_df[col].nunique() > 1]\ntrain_df = train_df[columns]\ntest_df = test_df[columns]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"773eb8d626ba8c2791ff3f7f332da56b0927a832"},"cell_type":"markdown","source":"# 3. Categorical to Numeric Variable Conversion for Model Training(Label Encoding)"},{"metadata":{"trusted":true,"_uuid":"a9ba46bf632a2cc0f670d570f086a6d9ecd6a6a1"},"cell_type":"code","source":"trn_len = train_df.shape[0]\nmerged_df = pd.concat([train_df, test_df])\n\nfor col in merged_df.columns:\n    if col in ['fullVisitorId']: continue\n    if merged_df[col].dtypes == object or merged_df[col].dtypes == bool:\n        merged_df[col], indexer = pd.factorize(merged_df[col])\n\ntrain_df = merged_df[:trn_len]\ntest_df = merged_df[trn_len:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a0bf00b4f875ad3baaf037809c3d176268ec454c"},"cell_type":"code","source":"train_df['totals.transactionRevenue']=target\ntrain_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c105dd73bc3da8790e57733cb6e83288cf0e42fb"},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10ffc9a1f6b4944d04a6a1a25e21557a6b438f8b"},"cell_type":"code","source":"#train_df[\"fullVisitorId\"] = train_df.fullVisitorId.astype(float)\n#test_df[\"fullVisitorId\"] = test_df[\"fullVisitorId\"].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ce2026b0b91c2c5c0e1a8ea7ac3a26140a9c327b"},"cell_type":"code","source":"import xgboost as xgb\nfrom lightgbm import LGBMRegressor\nimport lightgbm as lgb\nfrom bayes_opt import BayesianOptimization\nfrom sklearn.metrics import mean_squared_error\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5f8280ab545a39260c19afebe709b8a4d76885dc"},"cell_type":"code","source":"train_df.dtypes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"61fddab3c94de9caf80aa5dca5cf3e503742a2c5"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(train_df.drop(['totals.transactionRevenue','fullVisitorId'], axis=1),\n                                                    train_df['totals.transactionRevenue'], test_size=0.25)\n#del(df)\ndtrain = xgb.DMatrix(X_train, label=y_train)\n#del(X_train)\ndtest = xgb.DMatrix(X_test)\n#del(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"055a1c76e5aec87e1c57889de5408c68843fe7b7"},"cell_type":"code","source":"# def xgb_evaluate(max_depth, gamma,min_child_weight,max_delta_step,subsample,colsample_bytree):\n#     params = {'eval_metric': 'rmse',\n#               'max_depth': int(max_depth),\n#               'subsample': subsample,\n#               'eta': 0.1,\n#               'gamma': gamma,\n#               'colsample_bytree': colsample_bytree,   \n#               'min_child_weight': min_child_weight ,\n#               'max_delta_step':max_delta_step\n#              }\n#     # Used around 1000 boosting rounds in the full model\n#     cv_result = xgb.cv(params, dtrain, num_boost_round=100, nfold=3)    \n    \n#     # Bayesian optimization only knows how to maximize, not minimize, so return the negative RMSE\n#     return -1.0 * cv_result['test-rmse-mean'].iloc[-1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4426d1507918dece309b92550990b682e3039818"},"cell_type":"code","source":"# xgb_bo = BayesianOptimization(xgb_evaluate, {\n#                                     'max_depth': (2, 12),\n#                                      'gamma': (0.001, 10.0),\n#                                      'min_child_weight': (0, 20),\n#                                      'max_delta_step': (0, 10),\n#                                      'subsample': (0.4, 1.0),\n#                                      'colsample_bytree' :(0.4, 1.0)})\n# # Use the expected improvement acquisition function to handle negative numbers\n# # Optimally needs quite a few more initiation points and number of iterations\n# xgb_bo.maximize(init_points=3, n_iter=5, acq='ei')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"61737281517e18933e8ebca1c56f35608eac4151"},"cell_type":"code","source":"#params = xgb_bo.res['max']['max_params']\n#print(params)\nparams = {'max_depth': 6.714941854933043, 'gamma': 1.3250360141843498, 'min_child_weight': 13.0958516960316, 'max_delta_step': 8.88492863796954, 'subsample': 0.9864199446951019, 'colsample_bytree': 0.8376539278239742}\n#params = {'max_depth': 12.0, 'gamma': 0.001, 'min_child_weight': 8.740952582296343, 'max_delta_step': 10.0, 'subsample': 0.4, 'colsample_bytree': 1.0}\nparams['max_depth'] = int(params['max_depth'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"caa5c6d81c0258a8145622a2f268c23490de5c36"},"cell_type":"code","source":"# Train a new model with the best parameters from the search\nmodel2 = xgb.train(params, dtrain, num_boost_round=250)\n\n# Predict on testing and training set\ny_pred = model2.predict(dtest)\ny_train_pred = model2.predict(dtrain)\n\n# Report testing and training RMSE\nprint(np.sqrt(mean_squared_error(y_test, y_pred)))\nprint(np.sqrt(mean_squared_error(y_train, y_train_pred)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"24d19a64848c3098f642332fad13508751c6dc1c"},"cell_type":"code","source":"submission = test_df[['fullVisitorId']].copy()\n#test = transform((test_df.drop('fullVisitorId', axis=1)))\ndtest = xgb.DMatrix(test_df.drop('fullVisitorId', axis=1))\npredictions = model2.predict(dtest)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"953f2174274b97900c6639141d1851a2ea982820"},"cell_type":"markdown","source":"# 9.Final Submission"},{"metadata":{"trusted":true,"_uuid":"4a79241308ec980d9136191fabb65f3a66c3e78e"},"cell_type":"code","source":"\nsubmission.loc[:, 'PredictedLogRevenue'] = predictions\nsubmission[\"PredictedLogRevenue\"] = submission[\"PredictedLogRevenue\"].apply(lambda x : 0.0 if x < 0 else x)\nsubmission[\"PredictedLogRevenue\"] = submission[\"PredictedLogRevenue\"].fillna(0.0)\ngrouped_test = submission[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum().reset_index()\ngrouped_test.to_csv('submit.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"67b3fb4e91f0071e359a623bbb5f7314e561404e"},"cell_type":"code","source":"grouped_test.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d8ad2dcb6c188749718b01ca075ca48d2a19d288"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}