{"cells":[{"metadata":{"trusted":true,"_uuid":"d76f920283c64786aab1d3d25c739a969a7a73eb"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a0ecc5b8b65b5e8849e4f07632593402f374a9a"},"cell_type":"markdown","source":"# Background\n- In this competition, many kaggler suffer from the discrepancy between CV and LB.\n- This could be caused by some features which have different distribution in both train and test.\n- I asked some methods to find those features. https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/67850\n- Very kind kaggler, \"kain\" recommended some strategies.\n- After searching and studying them, I want to share the adversarial validation.\n- I referred to various kernels. Thanks for sharing! If you got a help, please upvote them!\n- https://www.kaggle.com/konradb/adversarial-validation-and-other-scary-terms\n- https://www.kaggle.com/rspadim/adversarial-validation-porto-seguro\n- https://www.kaggle.com/ogrellier/adversarial-validation-and-lb-shakeup\n- I forked this helpful kernel, https://www.kaggle.com/prashantkikani/ensembling-fe-is-the-answer. Thanks!"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import mean_squared_error\nimport gc\nimport time\nfrom pandas.core.common import SettingWithCopyWarning\nimport warnings\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold, GridSearchCV\n\n# I don't like SettingWithCopyWarnings ...\nwarnings.simplefilter('error', SettingWithCopyWarning)\ngc.enable()\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8c6c0d5d8cf00737b8ae536b0d528b32fac9166"},"cell_type":"markdown","source":"# Read data"},{"metadata":{"trusted":true,"_uuid":"6fe04db087c3d25b8b4a5873ee6f104e427525f0"},"cell_type":"code","source":"train = pd.read_csv('../input/create-extracted-json-fields-dataset/extracted_fields_train.gz', \n                    dtype={'date': str, 'fullVisitorId': str, 'sessionId':str}, nrows=None)\ntest = pd.read_csv('../input/create-extracted-json-fields-dataset/extracted_fields_test.gz', \n                   dtype={'date': str, 'fullVisitorId': str, 'sessionId':str}, nrows=None)\ntrain.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fc9435342a2a8135815268f5e55b916f68a5d3a8"},"cell_type":"markdown","source":"# Feature engineering"},{"metadata":{"trusted":true,"_uuid":"b04391e12fdca0199ff22ab7febd9b0b53d6cbb3"},"cell_type":"code","source":"for df in [train, test]:\n    df['date'] = pd.to_datetime(df['visitStartTime'], unit='s')\n    df['sess_date_dow'] = df['date'].dt.dayofweek\n    df['sess_date_hours'] = df['date'].dt.hour\n    df['sess_date_dom'] = df['date'].dt.day\n\n# https://www.kaggle.com/prashantkikani/teach-lightgbm-to-sum-predictions-fe\ndef browser_mapping(x):\n    browsers = ['chrome','safari','firefox','internet explorer','edge','opera','coc coc','maxthon','iron']\n    if x in browsers:\n        return x.lower()\n    elif  ('android' in x) or ('samsung' in x) or ('mini' in x) or ('iphone' in x) or ('in-app' in x) or ('playstation' in x):\n        return 'mobile browser'\n    elif  ('mozilla' in x) or ('chrome' in x) or ('blackberry' in x) or ('nokia' in x) or ('browser' in x) or ('amazon' in x):\n        return 'mobile browser'\n    elif  ('lunascape' in x) or ('netscape' in x) or ('blackberry' in x) or ('konqueror' in x) or ('puffin' in x) or ('amazon' in x):\n        return 'mobile browser'\n    elif '(not set)' in x:\n        return x\n    else:\n        return 'others'\n    \n    \ndef adcontents_mapping(x):\n    if  ('google' in x):\n        return 'google'\n    elif  ('placement' in x) | ('placememnt' in x):\n        return 'placement'\n    elif '(not set)' in x or 'nan' in x:\n        return x\n    elif 'ad' in x:\n        return 'ad'\n    else:\n        return 'others'\n    \ndef source_mapping(x):\n    if  ('google' in x):\n        return 'google'\n    elif  ('youtube' in x):\n        return 'youtube'\n    elif '(not set)' in x or 'nan' in x:\n        return x\n    elif 'yahoo' in x:\n        return 'yahoo'\n    elif 'facebook' in x:\n        return 'facebook'\n    elif 'reddit' in x:\n        return 'reddit'\n    elif 'bing' in x:\n        return 'bing'\n    elif 'quora' in x:\n        return 'quora'\n    elif 'outlook' in x:\n        return 'outlook'\n    elif 'linkedin' in x:\n        return 'linkedin'\n    elif 'pinterest' in x:\n        return 'pinterest'\n    elif 'ask' in x:\n        return 'ask'\n    elif 'siliconvalley' in x:\n        return 'siliconvalley'\n    elif 'lunametrics' in x:\n        return 'lunametrics'\n    elif 'amazon' in x:\n        return 'amazon'\n    elif 'mysearch' in x:\n        return 'mysearch'\n    elif 'qiita' in x:\n        return 'qiita'\n    elif 'messenger' in x:\n        return 'messenger'\n    elif 'twitter' in x:\n        return 'twitter'\n    elif 't.co' in x:\n        return 't.co'\n    elif 'vk.com' in x:\n        return 'vk.com'\n    elif 'search' in x:\n        return 'search'\n    elif 'edu' in x:\n        return 'edu'\n    elif 'mail' in x:\n        return 'mail'\n    elif 'ad' in x:\n        return 'ad'\n    elif 'golang' in x:\n        return 'golang'\n    elif 'direct' in x:\n        return 'direct'\n    elif 'dealspotr' in x:\n        return 'dealspotr'\n    elif 'sashihara' in x:\n        return 'sashihara'\n    elif 'phandroid' in x:\n        return 'phandroid'\n    elif 'baidu' in x:\n        return 'baidu'\n    elif 'mdn' in x:\n        return 'mdn'\n    elif 'duckduckgo' in x:\n        return 'duckduckgo'\n    elif 'seroundtable' in x:\n        return 'seroundtable'\n    elif 'metrics' in x:\n        return 'metrics'\n    elif 'sogou' in x:\n        return 'sogou'\n    elif 'businessinsider' in x:\n        return 'businessinsider'\n    elif 'github' in x:\n        return 'github'\n    elif 'gophergala' in x:\n        return 'gophergala'\n    elif 'yandex' in x:\n        return 'yandex'\n    elif 'msn' in x:\n        return 'msn'\n    elif 'dfa' in x:\n        return 'dfa'\n    elif '(not set)' in x:\n        return '(not set)'\n    elif 'feedly' in x:\n        return 'feedly'\n    elif 'arstechnica' in x:\n        return 'arstechnica'\n    elif 'squishable' in x:\n        return 'squishable'\n    elif 'flipboard' in x:\n        return 'flipboard'\n    elif 't-online.de' in x:\n        return 't-online.de'\n    elif 'sm.cn' in x:\n        return 'sm.cn'\n    elif 'wow' in x:\n        return 'wow'\n    elif 'baidu' in x:\n        return 'baidu'\n    elif 'partners' in x:\n        return 'partners'\n    else:\n        return 'others'\n\ntrain['device.browser'] = train['device.browser'].map(lambda x:browser_mapping(str(x).lower())).astype('str')\ntrain['trafficSource.adContent'] = train['trafficSource.adContent'].map(lambda x:adcontents_mapping(str(x).lower())).astype('str')\ntrain['trafficSource.source'] = train['trafficSource.source'].map(lambda x:source_mapping(str(x).lower())).astype('str')\n\ntest['device.browser'] = test['device.browser'].map(lambda x:browser_mapping(str(x).lower())).astype('str')\ntest['trafficSource.adContent'] = test['trafficSource.adContent'].map(lambda x:adcontents_mapping(str(x).lower())).astype('str')\ntest['trafficSource.source'] = test['trafficSource.source'].map(lambda x:source_mapping(str(x).lower())).astype('str')\n\ndef process_device(data_df):\n    print(\"process device ...\")\n    data_df['source.country'] = data_df['trafficSource.source'] + '_' + data_df['geoNetwork.country']\n    data_df['campaign.medium'] = data_df['trafficSource.campaign'] + '_' + data_df['trafficSource.medium']\n    data_df['browser.category'] = data_df['device.browser'] + '_' + data_df['device.deviceCategory']\n    data_df['browser.os'] = data_df['device.browser'] + '_' + data_df['device.operatingSystem']\n    return data_df\n\ntrain = process_device(train)\ntest = process_device(test)\n\ndef custom(data):\n    print('custom..')\n    data['device_deviceCategory_channelGrouping'] = data['device.deviceCategory'] + \"_\" + data['channelGrouping']\n    data['channelGrouping_browser'] = data['device.browser'] + \"_\" + data['channelGrouping']\n    data['channelGrouping_OS'] = data['device.operatingSystem'] + \"_\" + data['channelGrouping']\n    \n    for i in ['geoNetwork.city', 'geoNetwork.continent', 'geoNetwork.country','geoNetwork.metro', 'geoNetwork.networkDomain', 'geoNetwork.region','geoNetwork.subContinent']:\n        for j in ['device.browser','device.deviceCategory', 'device.operatingSystem', 'trafficSource.source']:\n            data[i + \"_\" + j] = data[i] + \"_\" + data[j]\n    \n    data['content.source'] = data['trafficSource.adContent'] + \"_\" + data['source.country']\n    data['medium.source'] = data['trafficSource.medium'] + \"_\" + data['source.country']\n    return data\n\ntrain = custom(train)\ntest = custom(test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59e7c9bcf14d3f430489d15d22757b91dd9d5ad1"},"cell_type":"markdown","source":"# Prepare features"},{"metadata":{"trusted":true,"_uuid":"e09ff3124721ad70eab65b924093b5fd0c12f9c6"},"cell_type":"code","source":"excluded_features = [\n    'date', 'fullVisitorId', 'sessionId', 'totals.transactionRevenue', \n    'visitId', 'visitStartTime'\n]\n\ncategorical_features = [\n    _f for _f in train.columns\n    if (_f not in excluded_features) & (train[_f].dtype == 'object')\n]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d207a80600a44e758b6011f24c5808f9b164866"},"cell_type":"markdown","source":"# Factorize features"},{"metadata":{"trusted":true,"_uuid":"4f00eb6b6166cd5be25ebad22aa9dc91a8a3ded2"},"cell_type":"code","source":"for f in categorical_features:\n    train[f], indexer = pd.factorize(train[f])\n    test[f] = indexer.get_indexer(test[f])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3538fd5ff97b478839e83038f3c768b3f1489a51"},"cell_type":"markdown","source":"# Prepare target for adversarial validation"},{"metadata":{"trusted":true,"_uuid":"dbfab0ff4fa09c95c63b67bd325df0769ae7f131"},"cell_type":"code","source":"print(train.shape, test.shape)\n\ntrain['istrain'] = 1\ntest['istrain'] = 0\n\ndf = pd.concat([train, test])\n\ntarget = df['istrain']\ndf.drop('istrain', axis=1, inplace=True)\n\nuse_cols = [col for col in df.columns if col not in excluded_features]\n\ndf = df[use_cols]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0b590f55685d07fb7560c95fbac656da4ecf821"},"cell_type":"markdown","source":"# Check auc"},{"metadata":{"trusted":true,"_uuid":"13b64ae673575c6f6450e45caf3f3fdf89868bc4"},"cell_type":"code","source":"%%time\n\nparams = {\n    \"objective\" : \"binary\",\n    \"metric\" : \"auc\",\n    \"num_leaves\" : 64,\n    \"learning_rate\" : 0.01,\n    \"bagging_fraction\" : 0.7,\n    \"feature_fraction\" : 0.7,\n    \"bagging_frequency\" : 1,\n    \"bagging_seed\" : 1989,\n    \"verbosity\" : -1,\n    'lambda_l1':1,\n    'lambda_l2':1,\n    'max_depth': -1,\n    'min_data_in_leaf': 100,\n    \"seed\": 1989,\n}\n\nFOLDs = KFold(n_splits=5, shuffle=True, random_state=1989)\nfeatures = list(df.columns)\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(df)):\n    print('#'*30, '{} of 5 folds'.format(fold_ +1), '#'*30)\n    trn_data = lgb.Dataset(df.iloc[trn_idx].values, label = target.iloc[trn_idx].values)\n    val_data = lgb.Dataset(df.iloc[val_idx].values, label = target.iloc[val_idx].values)\n    \n    num_round = 2000\n    clf = lgb.train(params, trn_data, num_round, valid_sets = [trn_data, val_data], \n                    verbose_eval=200, early_stopping_rounds = 50)\n    \n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"split\"] = clf.feature_importance(importance_type='split')\n    fold_importance_df[\"gain\"] = clf.feature_importance(importance_type='gain')\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)    \n    del trn_data, val_data, clf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0ef9b121784612116d863c917e5661fd07174087"},"cell_type":"code","source":"feature_importance_df = feature_importance_df.groupby('feature').mean().sort_values('gain', ascending=False).reset_index()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"596c9fe6c4f189e9e0ee2cfe2a5b0be30b3ff3d6"},"cell_type":"markdown","source":"# Feature importance when distinguish train from test"},{"metadata":{"_uuid":"61d5b3d2973883c9c3b88e21cd0f6cbb709481fe"},"cell_type":"markdown","source":"## Sorted by gain"},{"metadata":{"trusted":true,"_uuid":"a576dc8ed21b32a2e56fc34c7c2f569f55770262"},"cell_type":"code","source":"plt.figure(figsize=(10, 30))\nsns.barplot(x='gain', y='feature', data=feature_importance_df)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"52379b3ddaf6f18a9a99ba9074553e9bd3036b7f"},"cell_type":"markdown","source":"## Sorted by split"},{"metadata":{"trusted":true,"_uuid":"fc10c0fe2e5982243eca631122f1ed82c4a025fa"},"cell_type":"code","source":"plt.figure(figsize=(10, 30))\nsns.barplot(x='split', y='feature', data=feature_importance_df.sort_values('split', ascending=False))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b55ffe425958f99a1a178bf64e786ef4752c6359"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"398bc1edd3a1b4b200facefeb10db6753b9069f8"},"cell_type":"markdown","source":"# User-level"},{"metadata":{"trusted":true,"_uuid":"64872ec9040a6e66a72099fdd8775e40318e6b94"},"cell_type":"code","source":"del df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"113059325c40ce6051bf87b096cd15fda96dc27f"},"cell_type":"code","source":"%%time\nnew_train = train[use_cols + ['fullVisitorId']].groupby('fullVisitorId').mean()\nnew_test = test[use_cols + ['fullVisitorId']].groupby('fullVisitorId').mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ddba3bac76641983feca27d2e63b2473ad5e65b8"},"cell_type":"code","source":"print(new_train.shape, new_test.shape)\n\nnew_train['istrain'] = 1\nnew_test['istrain'] = 0\n\ndf = pd.concat([new_train, new_test])\n\ntarget = df['istrain']\ndf.drop('istrain', axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"83b26ef1661d4eecbcd64e8d968d60d8876cd142"},"cell_type":"code","source":"print(df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"206a98e0efacf0b69aa507cdc46b1b50886d6b49"},"cell_type":"code","source":"%%time\n\nparams = {\n    \"objective\" : \"binary\",\n    \"metric\" : \"auc\",\n    \"num_leaves\" : 64,\n    \"learning_rate\" : 0.01,\n    \"bagging_fraction\" : 0.7,\n    \"feature_fraction\" : 0.7,\n    \"bagging_frequency\" : 1,\n    \"bagging_seed\" : 1989,\n    \"verbosity\" : -1,\n    'lambda_l1':1,\n    'lambda_l2':1,\n    'max_depth': -1,\n    'min_data_in_leaf': 100,\n    \"seed\": 1989,\n}\n\nFOLDs = KFold(n_splits=5, shuffle=True, random_state=1989)\nfeatures = list(df.columns)\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(FOLDs.split(df)):\n    print('#'*30, '{} of 5 folds'.format(fold_ +1), '#'*30)\n    trn_data = lgb.Dataset(df.iloc[trn_idx].values, label = target.iloc[trn_idx].values)\n    val_data = lgb.Dataset(df.iloc[val_idx].values, label = target.iloc[val_idx].values)\n    \n    num_round = 2000\n    clf = lgb.train(params, trn_data, num_round, valid_sets = [trn_data, val_data], \n                    verbose_eval=200, early_stopping_rounds = 50)\n    \n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"split\"] = clf.feature_importance(importance_type='split')\n    fold_importance_df[\"gain\"] = clf.feature_importance(importance_type='gain')\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)    \n    del trn_data, val_data, clf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6eeea840aca755a38972c0f4e8f54ecbf8557958"},"cell_type":"code","source":"feature_importance_df = feature_importance_df.groupby('feature').mean().sort_values('gain', ascending=False).reset_index()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"44a462d63afbb5a2fc6c732ed2e6456cf1c35fdd"},"cell_type":"markdown","source":"# Feature importance"},{"metadata":{"_uuid":"441ebcb697153f5caa70ff82b4171fd34e773681"},"cell_type":"markdown","source":"## Sorted by gain"},{"metadata":{"trusted":true,"_uuid":"da0d12c41516b793f32d34674cb2d5e29b5f5f20"},"cell_type":"code","source":"plt.figure(figsize=(10, 30))\nsns.barplot(x='gain', y='feature', data=feature_importance_df)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6fff4743a457cfe5c95b588705330a47ee195b7d"},"cell_type":"markdown","source":"## Sorted by split"},{"metadata":{"trusted":true,"_uuid":"17fbc74645a057e573ef6968886268a3f85c8904"},"cell_type":"code","source":"plt.figure(figsize=(10, 30))\nsns.barplot(x='split', y='feature', data=feature_importance_df.sort_values('split', ascending=False))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9da5b4df660b780013aec8bc0a04bb058b39c09"},"cell_type":"markdown","source":"# Conclusion\n- Unfortunately, train and test are different.\n- So, we need to consider the difference when make features!"},{"metadata":{"_uuid":"d092d43ad4732c520b9a74564da714af6c5131f9"},"cell_type":"markdown","source":"- If there is bug or mistake, please talk to me! And if you have any feedback, please talk to me. \n- Hope this kernel helps you."},{"metadata":{"trusted":true,"_uuid":"2c15bc23139da1813311fd5f857c2d2a2e3aaf30"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}