{"cells":[{"metadata":{"_uuid":"2b2104d7222ce5b9efac99c402297beef8bede5e"},"cell_type":"markdown","source":"Let's find if any variable displays a distribution change between the train and the test set! It is a common problem in real-world data sets and a very important step in the feature selection process.\n\nWe will do this by trying to predict to which set each variable belongs to (train or test)."},{"metadata":{"trusted":true,"_uuid":"f905cd5578bfd6f1fae7eaae2d9de32cfdc31106"},"cell_type":"code","source":"#Import libraries\nimport pandas as pd\nimport numpy as np\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import cross_val_score\n\nimport os\nimport json\nfrom pandas.io.json import json_normalize","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"#Define function to load data (Kudos to Julian for that)\ndef load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"print('Loading Train set...')\ntrain = load_df()\nprint('Loading Test set...')\ntest = load_df(\"../input/test.csv\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"94ffa6a532a80011271746b32fec84a7e37c864d"},"cell_type":"code","source":"#Impute missing values (mean for numeric, mode for categorical)\nprint('Imputing missing values...')\nfor i in train.columns:\n    if train[i].dtype == 'object':\n      train[i] = train[i].fillna(train[i].mode().iloc[0])\n    elif (train[i].dtype != 'object'):\n      train[i] = train[i].fillna(np.mean(train[i]))\n\n\nfor i in test.columns:\n    if test[i].dtype == 'object':\n      test[i] = test[i].fillna(test[i].mode().iloc[0])\n    elif (test[i].dtype != 'object'):\n      test[i] = test[i].fillna(np.mean(test[i]))\n    \n\nprint('Nulls in train set:', train.isnull().sum().sum())\nprint('Nulls in test set:', test.isnull().sum().sum())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8e2ea11ec3ec2205ee3b9daf43d6df61cd064e92"},"cell_type":"code","source":"## label encode categorical variables\nprint('Label Encoding categorical variables...')\nfor col in train.columns:\n    if train[col].dtype == 'object':\n      train[col] = train[col].astype('category')\n      train[col] = train[col].cat.codes\n\nfor col in test.columns:\n    if test[col].dtype == 'object':\n      test[col] = test[col].astype('category')\n      test[col] = test[col].cat.codes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a849d05723f2bd63005178691af27152307167be"},"cell_type":"code","source":"## Creating a dummy y label and drop the target variable\ntrain['set'] = 0\ntest['set'] = 1\ntrain = train.drop(['totals.transactionRevenue', 'trafficSource.campaignCode'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e12bb7ccf1d28f92eb21e3ef9a706d7fae90c0a2"},"cell_type":"code","source":"## Use a sample set from both train and test and concatenate into a single dataframe\ntrain_df = train.sample(10000, random_state=697)\ntest_df = test.sample(10000, random_state=466)\n\nall_data = train_df.append(test_df)\ny_label = all_data['set']\nall_data = all_data.drop('set',axis=1)\n\n#Make sure the new dataframe contains all the initial features\nprint('New dataframe shape:', all_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"355ffddbf06083a44f962584052b9f1669c9ec04"},"cell_type":"code","source":"## Find all the features with covariate shift. Print during the procedure and then save in array\nmodel = RandomForestClassifier(n_estimators = 50, max_depth = 5, min_samples_leaf = 5)\nfeat_to_drop = []\nfor col in all_data.columns:\n    score = cross_val_score(model,pd.DataFrame(all_data[col]),y_label,cv=4,scoring='roc_auc')\n    if np.mean(score) > 0.8:\n        feat_to_drop.append(col)\n    print(col,np.mean(score))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c2727fa7c8aa15333d5c1c2f7d7eb77f4ed9cca2"},"cell_type":"code","source":"#Print number of features with covariate shift\nprint('Number of features with covariate shift:', len(feat_to_drop))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cd81b77562f790fc27102b0d48c6634ca16a36d"},"cell_type":"markdown","source":"14 of our independent variables have exceeded the 0.8 AUC threshold that we set. \n\nAs far as I know there are two main approaches to tackle shift distribution: The first one is to exclude those features with covariate shift which have been deemed unimportant in our model and the second is to produce a set of weights for those features with covariate shift which 'need' to be included in the model. The first case is simple. The second one can be performed with a Density Ratio Estimation. I will try both and will extend this kernel at a later stage. \n\nSome initial thoughts: \n'Device Browser' seems to present the greatest value but this could be due to its very high cardinality. 'adContent ' also presents high predictability of the test set but is ranked low in my model so it could be deleted. 'trafficSource.keyword' is also high so in covariate shift but has a fair impact on my model so it should be taken care of. The same goes for 'trafficSource.source'. 'geoNetwork.city' is really important in my LightGBM model but also displays covariate shift so definitely some sort of action is needed here. 'totals.pageviews' and 'totals.hits' are marginally above and below threshold repsectively but are very import explanatory variables so they will probably need some attention.\n\nFeel free to come up with your own strategies!\n\n##Edit 1: \nI excluded \"device.browser\" from my model and it slightly improved my PL score. My local CV score actually worsened a bit (from 1.5986 it went to 1.5992) but PL went from 1.4419 to 1.4415. This is not my current best model, I will try excluding more variables with covariate shift and then apply it to my best performing model to check the performance.\n\n##Edit 2: \nExcluded \"trafficSource.keyword\" in addition to \"device.browser\" and I see some minor improvement again. Local CV down to 1.5990, PL down to 1.4409\n\n##Edit 3: \nExcluded 'trafficSource.adContent'. Results seem to have worsened slightly: Local CV increased to 1.5993, PL increased to 1.4413"},{"metadata":{"_uuid":"62ab6748230f5673389d8dc30bf266ba3d22dbf0"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"3b567571a5771f484e95496d5ff6099d9447f084"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}