{"cells":[{"metadata":{"_uuid":"722c5fc4dbf32cb031b91e2ecbd7e8ede5f139d2"},"cell_type":"markdown","source":"## How to Reduce Training Times\n\nThis notebook takes the code EDA and LightGBM training from the following notebook:\n- https://www.kaggle.com/yhn112/data-exploration-lightgbm-catboost-lb-3-760\n\nAfter using the above code we optimise the pandas dataframe and then rerun the prediction and see the change in training time.  \nThis work will show you that by using just a couple of simple functions, you can reduce your training time.   When dataframes become very large this may save you many hours \n\nFor further info check out https://www.kaggle.com/richarde/random-forest-with-50-reduction-in-training-time\n\nGo down to **DataFrame Optimisation** header to see the added work\n"},{"metadata":{"_uuid":"27147b15332ddffc145a9c9e1857dca7b668db7b"},"cell_type":"markdown","source":"### Simple Exploration"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Loading packages\nimport pandas as pd #Analysis \nimport matplotlib.pyplot as plt #Visulization\nimport seaborn as sns #Visulization\nimport numpy as np #Analysis \nfrom scipy.stats import norm #Analysis \nfrom sklearn.preprocessing import StandardScaler #Analysis \nfrom scipy import stats #Analysis \nimport warnings \nwarnings.filterwarnings('ignore')\n%matplotlib inline\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"647c5a50c48a8fc2a2c44f53356bc6d7419f303d"},"cell_type":"markdown","source":"- train.csv - the training set\n- test.csv - the test set\n- sample_submission.csv - a sample submission file in the correct format - contains all card_ids you are expected to predict for.\n- historical_transactions.csv - up to 3 months' worth of historical transactions for each card_id\n- merchants.csv - additional information about all merchants / merchant_ids in the dataset.\n- new_merchant_transactions.csv - two months' worth of data for each card_id containing ALL purchases that card_id made at merchant_ids that were not visited in the historical data."},{"metadata":{"trusted":true,"_uuid":"2f3e05a20db0fd849a870e5712af2d69b4d7fafe"},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\", parse_dates=[\"first_active_month\"])\nprint(\"shape of train : \",train.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7ac6d0d218fca07658d220db99ed198d2557e790"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f1c8c454b4b391ca2a4251e26cfda4e6f10f8a2"},"cell_type":"markdown","source":"- first_active_month : ''YYYY-MM', month of first purchase\n- feature_1,2,3 : Anonymized card categorical feature\n- target : Loyalty numerical score calculated 2 months after historical and evaluation period"},{"metadata":{"trusted":true,"_uuid":"fe102452329416e9902cffb336e9a04a307a3f5d"},"cell_type":"code","source":"test = pd.read_csv(\"../input/test.csv\", parse_dates=[\"first_active_month\"])\nprint(\"shape of test : \",test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ea0872d482f2db9b119a9417effe99610ac8830a"},"cell_type":"markdown","source":"First_active_month of Train and Test looks similiar"},{"metadata":{"_uuid":"7c2ea0ea4f05539fcb77212d42b00ab3017ab783"},"cell_type":"markdown","source":"`feature_1 - feature_3` has 0.58 but `target - feature` low correaltion value"},{"metadata":{"_uuid":"5dcb3806cbf073a20918dd8965071dd515054310"},"cell_type":"markdown","source":"feature_3 has 1 when feautre_1 high than 3"},{"metadata":{"_uuid":"665f42f65c363e593f04cee4b5ef6d4ab92a7e3b"},"cell_type":"markdown","source":"feature_2 has not 3 when feature_1 == 5\nbut what is target low than -30???"},{"metadata":{"_uuid":"1affc2621e0ce520718471541fce720693be6e02"},"cell_type":"markdown","source":"### Missing value"},{"metadata":{"trusted":true,"_uuid":"61c9063db006d0023706da2fd6a68bcbea3f88c5"},"cell_type":"code","source":"# checking missing data\ntotal = train.isnull().sum().sort_values(ascending = False)\npercent = (train.isnull().sum()/train.isnull().count()*100).sort_values(ascending = False)\nmissing_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"afb3ee6f143762755fad62310cb6eccd3558bec6"},"cell_type":"code","source":"# checking missing data\ntotal = test.isnull().sum().sort_values(ascending = False)\npercent = (test.isnull().sum()/test.isnull().count()*100).sort_values(ascending = False)\nmissing_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"001883df0e6d41154a96cbc146b95966726d92e8"},"cell_type":"code","source":"import datetime\n\nfor df in [train,test]:\n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['year'] = df['first_active_month'].dt.year\n    df['month'] = df['first_active_month'].dt.month\n    df['elapsed_time'] = (datetime.date(2018, 2, 1) - df['first_active_month'].dt.date).dt.days\n\ntarget = train['target']\ndel train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b1dc9c508229c4ca8ec0bc7cb49e7f3c6e46c247"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f16f59b07c85cdcd397c28ebf01f9b4cca5ec0c"},"cell_type":"markdown","source":"### Simple Exploration : historical_transactions"},{"metadata":{"trusted":true,"_uuid":"6d123379e6664ef9cdd31bdf8f67a148ba1fe97b"},"cell_type":"code","source":"ht = pd.read_csv(\"../input/historical_transactions.csv\")\nprint(\"shape of historical_transactions : \",ht.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21161a1f1031e00d7fd76e7572e7b25e9e1ff316"},"cell_type":"code","source":"ht.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aaccb032fb6dbc53e49250e3dd957998a8b92907"},"cell_type":"markdown","source":"- card_id\t: Card identifier\n- month_lag\t: month lag to reference date\n- purchase_date\t: Purchase date\n- authorized_flag\t: Y' if approved, 'N' if denied\n- category_3\t: anonymized category\n- installments\t: number of installments of purchase\n- category_1 : \tanonymized category\n- merchant_category_id\t: Merchant category identifier (anonymized )\n- subsector_id\t: Merchant category group identifier (anonymized )\n- merchant_id\t: Merchant identifier (anonymized)\n- purchase_amount\t: Normalized purchase amount\n- city_id\t: City identifier (anonymized )\n- state_id\t: State identifier (anonymized )\n- category_2 : anonymized category"},{"metadata":{"trusted":true,"_uuid":"2d468a8c461d18cfc8bb2d36308f0c4f453f3e4e"},"cell_type":"code","source":"ht['authorized_flag'] = ht['authorized_flag'].map({'Y':1, 'N':0})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"45dacfcc7a14e680520f4637847534b569b521be"},"cell_type":"code","source":"def aggregate_historical_transactions(history):\n    \n    history.loc[:, 'purchase_date'] = pd.DatetimeIndex(history['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'merchant_id': ['nunique'],\n        'city_id': ['nunique'],\n        'purchase_amount': ['sum', 'median', 'max', 'min', 'std'],\n        'installments': ['sum', 'median', 'max', 'min', 'std'],\n        'purchase_date': [np.ptp],\n        'month_lag': ['min', 'max']\n        }\n    agg_history = history.groupby(['card_id']).agg(agg_func)\n    agg_history.columns = ['hist_' + '_'.join(col).strip() \n                           for col in agg_history.columns.values]\n    agg_history.reset_index(inplace=True)\n    \n    df = (history.groupby('card_id')\n          .size()\n          .reset_index(name='hist_transactions_count'))\n    \n    agg_history = pd.merge(df, agg_history, on='card_id', how='left')\n    \n    return agg_history\n\nhistory = aggregate_historical_transactions(ht)\ndel ht\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dd2e038c44e568ce88a3dbbe2cab6fad03922527"},"cell_type":"code","source":"train = pd.merge(train, history, on='card_id', how='left')\ntest = pd.merge(test, history, on='card_id', how='left')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f741be784db9431bdc920153e24912c879bf690"},"cell_type":"markdown","source":"### Simple Exploration : merchants.csv"},{"metadata":{"trusted":true,"_uuid":"aeb29c9260afa31aad9c62ecfb3c262112267a71"},"cell_type":"code","source":"merchant = pd.read_csv(\"../input/merchants.csv\")\nprint(\"shape of merchant : \",merchant.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26fdb80ab34d243c96b8146da9e523a8e2d92383"},"cell_type":"code","source":"merchant.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f247ab68d5427c7a5a3d598273a68d79b87ac7a"},"cell_type":"markdown","source":"- merchant_id : Unique merchant identifier\n- merchant_group_id\t: Merchant group (anonymized )\n- merchant_category_id\t: Unique identifier for merchant category (anonymized )\n- subsector_id\t: Merchant category group (anonymized )\n- numerical_1\t: anonymized measure\n- numerical_2\t: anonymized measure\n- category_1\t: anonymized category\n- most_recent_sales_range\t: Range of revenue (monetary units) in last active month --> A > B > C > D > E\n- most_recent_purchases_range\t: Range of quantity of transactions in last active month --> A > B > C > D > E\n- avg_sales_lag3\t: Monthly average of revenue in last 3 months divided by revenue in last active month\n- avg_purchases_lag3\t: Monthly average of transactions in last 3 months divided by transactions in last active month\n- active_months_lag3\t: Quantity of active months within last 3 months\n- avg_sales_lag6\t: Monthly average of revenue in last 6 months divided by revenue in last active month\n- avg_purchases_lag6\t: Monthly average of transactions in last 6 months divided by transactions in last active month\n- active_months_lag6\t: Quantity of active months within last 6 months\n- avg_sales_lag12\t: Monthly average of revenue in last 12 months divided by revenue in last active month\n- avg_purchases_lag12\t: Monthly average of transactions in last 12 months divided by transactions in last active month\n- active_months_lag12\t: Quantity of active months within last 12 months\n- category_4\t: anonymized category\n- city_id\t: City identifier (anonymized )\n- state_id\t: State identifier (anonymized )\n- category_2\t: anonymized category"},{"metadata":{"trusted":true,"_uuid":"23f953e61328d419ab84bb29cac71a6483b9af31"},"cell_type":"code","source":"# checking missing data\ntotal = merchant.isnull().sum().sort_values(ascending = False)\npercent = (merchant.isnull().sum()/merchant.isnull().count()*100).sort_values(ascending = False)\nmissing_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2e42513b79fa0a9ee74667fc19dc39024a00d976"},"cell_type":"markdown","source":"### Simple Exploration : new_merchants.csv"},{"metadata":{"trusted":true,"_uuid":"2158daadfff2f35ca36e0d1c5b5cbe795dc411f2"},"cell_type":"code","source":"new_merchant = pd.read_csv(\"../input/new_merchant_transactions.csv\")\nprint(\"shape of new_merchant_transactions : \",new_merchant.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b046368f582985ea824ec3c1242a1e4e45f71e9a"},"cell_type":"code","source":"new_merchant.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"141a8598508a27db2d7acd354c5849c0d498f768"},"cell_type":"markdown","source":"- card_id\t: Card identifier\n- month_lag\t: month lag to reference date\n- purchase_date\t: Purchase date\n- authorized_flag\t: Y' if approved, 'N' if denied\n- category_3\t: anonymized category\n- installments\t: number of installments of purchase\n- category_1\t: anonymized category\n- merchant_category_id\t: Merchant category identifier (anonymized )\n- subsector_id\t: Merchant category group identifier (anonymized )\n- merchant_id\t: Merchant identifier (anonymized)\n- purchase_amount\t: Normalized purchase amount\n- city_id\t: City identifier (anonymized )\n- state_id\t: State identifier (anonymized )\n- category_2\t: anonymized category\n"},{"metadata":{"trusted":true,"_uuid":"14841d7b310526d610c9499b477d08d27aab01ca"},"cell_type":"code","source":"# checking missing data\ntotal = new_merchant.isnull().sum().sort_values(ascending = False)\npercent = (new_merchant.isnull().sum()/new_merchant.isnull().count()*100).sort_values(ascending = False)\nmissing_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d20375c23b9528dd3125127939ff559e7520eb2"},"cell_type":"code","source":"new_merchant['authorized_flag'] = new_merchant['authorized_flag'].map({'Y':1, 'N':0})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d226c0360094b87654c9aec5677f76164836c6cd"},"cell_type":"code","source":"def aggregate_new_transactions(new_trans):    \n    agg_func = {\n        'authorized_flag': ['sum', 'mean'],\n        'merchant_id': ['nunique'],\n        'city_id': ['nunique'],\n        'purchase_amount': ['sum', 'median', 'max', 'min', 'std'],\n        'installments': ['sum', 'median', 'max', 'min', 'std'],\n        'month_lag': ['min', 'max']\n        }\n    agg_new_trans = new_trans.groupby(['card_id']).agg(agg_func)\n    agg_new_trans.columns = ['new_' + '_'.join(col).strip() \n                           for col in agg_new_trans.columns.values]\n    agg_new_trans.reset_index(inplace=True)\n    \n    df = (new_trans.groupby('card_id')\n          .size()\n          .reset_index(name='new_transactions_count'))\n    \n    agg_new_trans = pd.merge(df, agg_new_trans, on='card_id', how='left')\n    \n    return agg_new_trans\n\nnew_trans = aggregate_new_transactions(new_merchant)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"31612aef5e2d693a899776a2dfcd0f4af8e38ea2"},"cell_type":"code","source":"train = pd.merge(train, new_trans, on='card_id', how='left')\ntest = pd.merge(test, new_trans, on='card_id', how='left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1519c5ae5c2cdf2758010c5f767e36a36ff5dd3a"},"cell_type":"code","source":"train.info(verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8cb85391a726126ae42d9c6508a031d2658df9fc"},"cell_type":"markdown","source":"### Make a Baseline model\nkernel : https://www.kaggle.com/youhanlee/hello-elo-ensemble-will-help-you"},{"metadata":{"trusted":true,"_uuid":"923477910006486e5dd0a8e395a5c361e1419ef2"},"cell_type":"code","source":"use_cols = [col for col in train.columns if col not in ['card_id', 'first_active_month']]\n\ntrain = train[use_cols]\ntest = test[use_cols]\n\nfeatures = list(train[use_cols].columns)\ncategorical_feats = [col for col in features if 'feature_' in col]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1a1f1e1aeccbcf3e0049fdb1f465e0670f6b91c2"},"cell_type":"code","source":"for col in categorical_feats:\n    print(col, 'have', train[col].value_counts().shape[0], 'categories.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ba4b71e85aa8063f9dbbe5e8a3e8f6bd498cfdc3"},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nfor col in categorical_feats:\n    print(col)\n    lbl = LabelEncoder()\n    lbl.fit(list(train[col].values.astype('str')) + list(test[col].values.astype('str')))\n    train[col] = lbl.transform(list(train[col].values.astype('str')))\n    test[col] = lbl.transform(list(test[col].values.astype('str')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ad4ed61eaef4f37206043da70067007a394c0aac"},"cell_type":"code","source":"df_all = pd.concat([train, test])\ndf_all = pd.get_dummies(df_all, columns=categorical_feats)\n\nlen_train = train.shape[0]\n\ntrain = df_all[:len_train]\ntest = df_all[len_train:]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dfb27b1e938583a3ccd6d474e20a9106456c7ca6"},"cell_type":"markdown","source":"## DataFrame Optimisation\n**Lets look at the dataframe**"},{"metadata":{"trusted":true,"_uuid":"72d9688fba3da6f2c62d4a9e938e8c536e97588a"},"cell_type":"code","source":"train.info(verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e0343905a3dea257990c5dc10a743faa4729d336"},"cell_type":"markdown","source":"**There are several datatypes, below we convert to smaller types**"},{"metadata":{"trusted":true,"_uuid":"97f3a42fcc2c253c943b96e70d7fe83fa47a4e98"},"cell_type":"code","source":"def mem_usage(pandas_obj):\n    usage_b = pandas_obj.memory_usage(deep=True).sum()\n    usage_mb = usage_b / 1024 ** 2 # convert bytes to megabytes\n    return \"{:03.2f} MB\".format(usage_mb)\n\ntrain_mem_int = train.select_dtypes(include=['int'])\nconverted_int = train_mem_int.apply(pd.to_numeric,downcast='unsigned')\n\nprint(\"Size of integer types before {}\".format(mem_usage(train_mem_int)))\nprint(\"Size of integer types after {}\".format(mem_usage(converted_int)))\n\ncompare_ints = pd.concat([train_mem_int.dtypes,converted_int.dtypes],axis=1)\ncompare_ints.columns = ['No. of types before','No. of types after']\nprint(compare_ints.apply(pd.Series.value_counts))\n\ntrain_col_int = list(train_mem_int.columns)\nprint(train_col_int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f2d94cb4a553e76e88c6fe7ac95c14063e6aaa51"},"cell_type":"code","source":"train_mem_float = train.select_dtypes(include=['float'])\nconverted_float = train_mem_float.apply(pd.to_numeric,downcast='float')\n\nprint(\"Size of float types before: {}\".format(mem_usage(train_mem_float)))\nprint(\"Size of float types after: {}\".format(mem_usage(converted_float)))\n\ncompare_floats = pd.concat([train_mem_float.dtypes,converted_float.dtypes],axis=1)\ncompare_floats.columns = ['No. of types before','No. of types after']\nprint(compare_floats.apply(pd.Series.value_counts))\n\nprint(\" \")\n\ntrain_col_float = list(train_mem_float.columns)\nprint(train_col_float)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca02c0f2ca0090bfd65caffb3b7ab1702d57175b"},"cell_type":"markdown","source":"**From above we have reduced the sizes of our int and float datatypes and reduced the size considerably.  We will run lightgbm with the original types then rerun with these downcast datatypes.**"},{"metadata":{"trusted":true,"_uuid":"44a765e67afd943349311fe4a079827178efe840"},"cell_type":"code","source":"from sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error\nimport lightgbm as lgb\nimport time\n\n\nlgb_params = {\"objective\" : \"regression\", \"metric\" : \"rmse\", \n               \"max_depth\": 11, \"min_child_samples\": 20, \n               \"reg_alpha\": 1, \"reg_lambda\": 1,\n               \"num_leaves\" : 128, \"learning_rate\" : 0.005, \n               \"subsample\" : 0.8, \"colsample_bytree\" : 0.8, \n               \"verbosity\": -1}\n\nFOLDs = KFold(n_splits=8, shuffle=True, random_state=1989)\n\noof_lgb = np.zeros(len(train))\npredictions_lgb = np.zeros(len(test))\n\nfeatures_lgb = list(train.columns)\nfeature_importance_df_lgb = pd.DataFrame()\n\nstart = time.time()\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(train)):\n    trn_data = lgb.Dataset(train.iloc[trn_idx], label=target.iloc[trn_idx])\n    val_data = lgb.Dataset(train.iloc[val_idx], label=target.iloc[val_idx])\n\n    print(\"LGB \" + str(fold_) + \"-\" * 50)\n    num_round = 1000\n    clf = lgb.train(lgb_params, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval = 400)\n    oof_lgb[val_idx] = clf.predict(train.iloc[val_idx], num_iteration=clf.best_iteration)\n    predictions_lgb += clf.predict(test, num_iteration=clf.best_iteration) / FOLDs.n_splits\n    \nprint(np.sqrt(mean_squared_error(oof_lgb, target)))\nend = time.time()\nprint('Time taken in predictions: {0: .2f}'.format(end - start))\n#time_first = end - start","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"877b6822095d078224d2f3fd1bffec54810cd352"},"cell_type":"markdown","source":"**Note the time taken above and final rmse score**\n\n**We have run lightgbm with the original datatypes.  Below we change to the optimised types**"},{"metadata":{"trusted":true,"_uuid":"54146eb64a4e04be054f082be74d7efdee131086"},"cell_type":"code","source":"#replace the floats with smaller datatypes\ncolumns_to_overwrite_float = train_col_float \ntrain.drop(labels=columns_to_overwrite_float, axis=\"columns\", inplace=True)\ntrain[columns_to_overwrite_float] = converted_float[columns_to_overwrite_float]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb43b9954401fb33dbe2fa147aaaae5397aa40de"},"cell_type":"code","source":"#replace the floats with smaller datatypes\ncolumns_to_overwrite_int = train_col_int \ntrain.drop(labels=columns_to_overwrite_int, axis=\"columns\", inplace=True)\ntrain[columns_to_overwrite_int] = converted_int[columns_to_overwrite_int]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a85040d73cd811a4bc75c67ae11d3729643ea599"},"cell_type":"code","source":"train.info(verbose=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b2813048039fe582a27185d8238a10ac76357e20"},"cell_type":"markdown","source":"**Have reduced size of dataframe from 62MB to 33MB**\n\n**Now rerun with optimised datatypes**"},{"metadata":{"trusted":true,"_uuid":"99cefe53d9cf2a3331b1403b8097c458394b6342"},"cell_type":"code","source":"FOLDs = KFold(n_splits=8, shuffle=True, random_state=1989)\n\noof_lgb = np.zeros(len(train))\npredictions_lgb = np.zeros(len(test))\n\nfeatures_lgb = list(train.columns)\nfeature_importance_df_lgb = pd.DataFrame()\n\nstart = time.time()\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(train)):\n    trn_data = lgb.Dataset(train.iloc[trn_idx], label=target.iloc[trn_idx])\n    val_data = lgb.Dataset(train.iloc[val_idx], label=target.iloc[val_idx])\n\n    print(\"LGB \" + str(fold_) + \"-\" * 50)\n    num_round = 1000\n    clf = lgb.train(lgb_params, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval = 400)\n    oof_lgb[val_idx] = clf.predict(train.iloc[val_idx], num_iteration=clf.best_iteration)\n    predictions_lgb += clf.predict(test, num_iteration=clf.best_iteration) / FOLDs.n_splits\n    \nprint(np.sqrt(mean_squared_error(oof_lgb, target)))\nend = time.time()\nprint('Time taken in predictions: {0: .2f}'.format(end - start))\n#time_first = end - start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"186db2e6927a6fedf9a4e31411e761bb4d62c187"},"cell_type":"markdown","source":"**In this case the time saving has been quite small but as the size of the dataframe increases, applying these few simple tricks could save considerable time**"},{"metadata":{"trusted":true,"_uuid":"f46a48bc7f248c38c5f0748b8bf297ce1ae76bdc"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}