{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Microsoft's Malware detection challenge\n-----------------------------------------------------------------------------\n\n**Team**: VSM\n\n**Team member**: \n1. Sai Manish Sasanapuri (IMT2018520)\n2. Shubhayu Das (IMT2018523)\n3. Veerendra S. Devaraddi (IMT2018529)\n\n**Notebook execution time estimate:** upto 30 minutes on Kaggle, upto 4GB RAM usage\n\n----------------------------------------\nThis contest was held on a subset of the original competition's dataset. Our approach involves analysing every feature in the dataset. This is followed by fixing missing values, handling class imabalance, etc. The final model of choice here was LightGBM, trained with several fold cross validation.\n\nThere might be a bug in the final prediction part of the notebook, which leads to erronous predictions. Please utilize the Python script supplied to get the predictions, using the ```CatMapping.sav``` that is provided and/or generated by this notebook.\n\nWe had a severe shortage of time, which prevented us from playing with the hyperparameters further. \n\n**NOTE**: There was an error in the EDA notebook, which cannot be reproduced here. You will have to see the linked notebooks 1 and 2 below. This error happened everywhere, due to a stray \"i += 1\", which messed everything up during the commit. We have left the original notebooks as is, on Kaggle. But they are 2 different notebooks.\n\nOur original notebooks were messy, so we made this one notebook with everything in it. Link to *this* notebook on Kaggle: [Team VSM's final notebook](https://www.kaggle.com/shubhayudas6/team-vsm-s-final-notebook)\n\n1. [Link to EDA notebook](https://www.kaggle.com/shubhayudas6/team-vsm-eda). This will be referred to several times during the EDA in this notebook. We have avoided printing huge things here, but the original notebook contains all of it. There was one error in this linked EDA notebook(as extra i += 1) which messed up a lot of things. *That has been corrected in this notebook.*\n2. [Link to Model training notebook](https://www.kaggle.com/shubhayudas6/team-vsm-lightgbm?scriptVersionId=48202901). This code has almost not been changed; just reorganized.\n3. [Link to XGBoost model notebook](https://www.kaggle.com/shubhayudas6/team-vsm-xgboost-gpu?scriptVersionId=48208097). Mind you that this model was trained on the wrong data from the original EDA notebook. The corrected EDA in *this* notebook might improve scores a lot.\n4. [Link to CatBoost model notebook](https://www.kaggle.com/shubhayudas6/malware-detection-catboost-gpu) Again, this used the wrong output from the EDA notebook. This model takes too much RAM and will most probably crash. We didn't spend too much time on this one. Maybe reducing the number of estimators will make it work right. \n\nAll these notebooks will be made public on December 3rd, by 12PM.\n\nAll the models were trained on Kaggle, so please make sure you have the correct/compatible versions of the libraries."},{"metadata":{},"cell_type":"markdown","source":"## Importing all the requisite libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":false},"cell_type":"code","source":"# To store the preprocessing steps\nimport gc\nimport pickle\n\n# Pretty printing library; not really used\nfrom pprint import pprint\n\n# General data manipulation libraries\nimport pandas as pd\nimport numpy as np\nimport scipy.sparse as scsparse\n\n# Plotting libraries\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Libraries for getting the data ready for training\nfrom sklearn.preprocessing import StandardScaler, OneHotEncoder\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\n\n# Oversampler to fix class imbalance\nfrom imblearn.over_sampling import RandomOverSampler\n\n# Different models that were tried out. \nfrom sklearn.naive_bayes import BernoulliNB\nfrom sklearn.svm import LinearSVC\nfrom sklearn.calibration import CalibratedClassifierCV\nfrom lightgbm import LGBMClassifier\n\n# Performance evaluation metrics\nfrom sklearn.metrics import roc_auc_score, plot_confusion_matrix\n\n# We want to see all the columns in the DataFrames\npd.set_option('max_columns', None)\n\n# Enable the garbage cleaner\ngc.enable()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define a few constants"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"# File names\nTRAIN_SET = '../input/Malware-detection-NS/train.csv'\nTEST_SET = '../input/Malware-detection-NS/test.csv'\n\n# Important column names\nID_COL = 'MachineIdentifier'\nOUTPUT_COL = 'HasDetections'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Analysis of the training dataset"},{"metadata":{},"cell_type":"markdown","source":"## Load in the dataset"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"scrolled":false},"cell_type":"code","source":"train = pd.read_csv(TRAIN_SET, low_memory=False)\ntest = pd.read_csv(TEST_SET, low_memory=False)\n\nprint(f\"Train dataset size: {train.shape}\\nTest dataset size: {test.shape}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Taking a peek into the dataset"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"train.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Take a look at the output class distribution\n\nIn our opinions, this is one of the most important things to check in any classification problem. This give us insights about how imbalances the dataset (and some features are). In this particular dataset, we did expect heavy class imbalance, because the number of times malware affects a computer is going to be lesser than the amount of telemetry Microsoft collects from Windows."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"print(f\"Output feature class distribution:\\n{train[OUTPUT_COL].value_counts()}\")\nsns.countplot(y=OUTPUT_COL, data=train)\nplt.title('Output class distribution')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, so 15% of the entries in the dataset are **positive**(i.e. these systems have Malware). Not the worst we have seen, but we will try to address this issue later. First the EDA!"},{"metadata":{},"cell_type":"markdown","source":"## Starting with the EDA\n--------------------------------------\n\nFair bit of warning: we went overboard with this, because we thought understanding the dataset is very important. This is about to get very messy. Perhaps we did too much analysis, which decreased our scores. But we are glad we understood the dataset properly.\n\nI am *not* going to put the lengthy description of each feature; that is available on the contest page [here](https://www.kaggle.com/c/Malware-detection-NS/data).\n\nOk, we will define a few variables first, for later use. They will help in handling with the test dataset."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# Necessary variables for feature engineering\ni = 0\nfeatureDropList = []\ncategoricalFixerMappings = {}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Missing data (feature-wise)\n\nLet's take a look at the count of 'NA' in each of the features. As per the description of the columns, NA does mean that the feature is not defined for a particular row. It just means that the data is missing."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"naCount = list(\n sorted(\n zip(train.columns, train.isna().sum().values),\n reverse=True, key=lambda x: x[1])\n )\n\nfor col, count in naCount:\n print(f\"{col} has {count} NA values\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wow, some of the top features have too much missing. We simply need to drop them. Can't impute them or anything.\n\nWe will set our threshold this way: if more than 20% of the overall data is missing, we kiss the feature goodbye."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# Taking a look at all the higher features, and how much they are missing for confirmation\nextremeMissing = dict(filter(lambda x: x[1] > int(0.2 * train.shape[0]), naCount))\n\nprint(f\"Dropping {len(extremeMissing)} feature, on grounds of too much missing data.\\nFeatures:\\n\")\nfor feature, count in extremeMissing.items():\n print(f\"Feature: {feature};\\t\\t NA count: {count}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's go ahead and drop them"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"featureDropList += list(extremeMissing.keys())\n\ntrain = train.drop(labels=list(extremeMissing.keys()), axis=1)\n\n# Verify the shape of the DataFrame\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Missing data (row-wise)\n\nEntries with too much missing data are no good either. Even if it is categorical data, it will introduce extra noise into the model. Any row entry will too many NA's will be dropped. How much is \"too much\"? Let's find out.\n\n**Note**: This is applicable for the training data only. We simply cannot do this for the test dataset.\n\nThe other thing is that, there are quite a few features(columns) with around 10% missing data, which will interfere with this step. However, as we will see, the counts are small enough for us to do this. Hopefully this is not a big mistake."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"missingPerRowCount = train.isnull().sum(axis=1)\n\n# Sorry, I couldn't figure out a better way to display this\nprint(\"Number of missing features vs Count of such entries\")\nmissingPerRowCount.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, this is flat outrageous. We are going to ditch any row with more than 7 features missing(10% of 74 features that we have as of now). Again, this is such a small number, that it will hardly matter(hopefully)."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"print(f\"Original shape of training dataset: {train.shape}\")\nprint(f\"Dropping {missingPerRowCount[missingPerRowCount >= 7].shape[0]} rows\")\ntrain = train.drop(index=missingPerRowCount[missingPerRowCount > 7].index, axis=0)\n\nprint(f\"Final shape of training dataset: {train.shape}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Drop the id row from the train dataset\n\nJust to be safe, we will drop the ```MachineIdentifier``` feature from the training dataset, because well...we don't want a 0.999 AUC score. Crazy right? \n\nWe won't do this for the test dataset yet, because we need that row for the final submission."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"train = train.drop(labels=[ID_COL], axis=1)\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Dealing with categorical data\n\nWhen we did this for the first time, we noticed that the features often have the wrong datatype associated with them. So what we are going to do is:\n1. Get a count of the number of unique values in *every* feature.\n2. Sort out features with binary values (0.0/1.0). There are many of these. These will be converted into np.int8\n3. Sort out numeric features(int/float) which are actually categorical. Several of these too.\n\nAll the analysis will be done first. Then we will make a function to fix all the problems. This function will help with the test dataset."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"categoryCount = []\nfor col in train.columns:\n categoryCount.append(len(train[col].unique()))\n \ncategoryCount = np.array(categoryCount)\n\n# This is crazy hard to read, sorry. Could probably make a dataframe for this, but can't be bothered\n# Sorted in order of the number of unique categories\nfor col, count in sorted(zip(train.columns, categoryCount), key=lambda x: x[1]):\n print(f\"Feature: {col} Type: {train[col].dtype} Unique values: {count} Missing values: {train[col].isna().sum()}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, so we can see the number of binary features. Some of the features with three unique values are also binary, with the third feature being 'NA'. Let me attack these features and check the unique categories on them."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# Uh, no stereotypical gender division implied here. Go LGBTQ!!\npossiblyBinary = train.columns.values[categoryCount < 4]\n\n# Get the count for each feature\nfor feature in possiblyBinary:\n print(train[feature].value_counts(), end='\\n\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Good, so apart from ```Processor```, ```Census_OSArchitecture``` and ```Census_DeviceFamily``` are non-binary. Good for them.\n\nBut here are some other interesting observations:\n1. ```Processor``` and ```Census_OSArchitecture``` seems to be repeated features. I will check this later and drop one of them.\n2. Some features have horrible *category* distribution. These features will only cause more pain down the line(or so we think). So in a separate step(after fixing datatypes), we will be dropping them.\n\nFor now, all of the binary features will be converted into 8 bit integers.(later after categorical feature analysis).\n\n---------------------\n\nUp next, I will take a look at the numeric columns and see how many of them are actually categorical. This will be done based on the name of the feature and if required, the unique values. In general, truly unique numeric features will have a more unique values.(more than categorical, \"more\" is relative)."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"numericalAnalysisCols = train.select_dtypes(exclude='object').columns\n\nnumericalAnalysisCols","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok here are the original numeric fields, according to the names:\n1. ```Census_ProcessorCoreCount```\n2. ```Census_PrimaryDiskTotalCapacity```\n3. ```Census_SystemVolumeTotalCapacity```\n4. ```Census_TotalPhysicalRAM```\n5. ```Census_InternalPrimaryDiagonalDisplaySizeInInches``` (float)\n6. ```Census_InternalPrimaryDisplayResolutionHorizontal``` (float)\n7. ```Census_InternalPrimaryDisplayResolutionVertical``` (float)\n8. ```Census_InternalBatteryNumberOfCharges```\n\n**This is it**. Seems sus that we found it out just by the names? Well thank Microsoft for naming their features properly. Also, in our original EDA, we brute-force printed all the unique values. Link to our original EDA notebook is available in the very first cell.\n\n--------------------------\n\nJust a minute....there might be more features which have horrible class balance. We will have to weed them out later.\n\n------\n\nAnyways, time to define the function that will correct all the datatypes. While we are at it, we need to fill in all the missing data for the numeric columns. **We will not use the mode to impute, because the majority class count will keep increasing.** Instead we just filled with 0, which is not a great choice. "},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"possiblyBinary","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def dtypeFixer(data, training=False):\n dataset = data.copy(deep=True)\n \n # 8 bit ints\n actuallyBinary = [\n 'IsBeta',\n 'IsSxsPassiveMode',\n 'HasTpm',\n 'IsProtected',\n 'AutoSampleOptIn',\n 'SMode',\n 'Firewall',\n 'Census_HasOpticalDiskDrive',\n 'Census_IsPortableOperatingSystem',\n 'Census_IsFlightsDisabled',\n 'Census_IsSecureBootEnabled',\n 'Census_IsVirtualDevice',\n 'Census_IsTouchEnabled',\n 'Census_IsPenCapable',\n 'Census_IsAlwaysOnAlwaysConnectedCapable',\n 'Wdft_IsGamer'\n ]\n \n if training:\n actuallyBinary.append(OUTPUT_COL)\n \n # 32 bit ints\n actuallyInteger = [\n 'Census_ProcessorCoreCount',\n 'Census_PrimaryDiskTotalCapacity',\n 'Census_SystemVolumeTotalCapacity',\n 'Census_TotalPhysicalRAM',\n 'Census_InternalBatteryNumberOfCharges'\n ]\n \n # 32 bit floats\n actuallyFloat = [\n 'Census_InternalPrimaryDiagonalDisplaySizeInInches',\n 'Census_InternalPrimaryDisplayResolutionHorizontal',\n 'Census_InternalPrimaryDisplayResolutionVertical'\n ]\n \n # Actually categorical\n actuallyCategorical = [\n 'ProductName',\n 'EngineVersion',\n 'AppVersion',\n 'AvSigVersion',\n 'RtpStateBitfield',\n 'AVProductStatesIdentifier', \n 'AVProductsInstalled',\n 'AVProductsEnabled',\n 'CountryIdentifier',\n 'CityIdentifier',\n 'GeoNameIdentifier', \n 'LocaleEnglishNameIdentifier',\n 'Platform',\n 'Processor',\n 'OsVer',\n 'OsBuild',\n 'OsSuite',\n 'OsPlatformSubRelease',\n 'OsBuildLab',\n 'SkuEdition',\n 'IeVerIdentifier',\n 'UacLuaenable',\n 'Census_MDC2FormFactor',\n 'Census_DeviceFamily',\n 'Census_OEMNameIdentifier',\n 'Census_OEMModelIdentifier',\n 'Census_ProcessorManufacturerIdentifier',\n 'Census_ProcessorModelIdentifier',\n 'Census_PrimaryDiskTypeName',\n 'Census_ChassisTypeName',\n 'Census_PowerPlatformRoleName',\n 'Census_OSVersion',\n 'Census_OSArchitecture',\n 'Census_OSBranch',\n 'Census_OSBuildNumber',\n 'Census_OSBuildRevision',\n 'Census_OSEdition',\n 'Census_OSSkuName',\n 'Census_OSInstallTypeName',\n 'Census_OSInstallLanguageIdentifier',\n 'Census_OSUILocaleIdentifier',\n 'Census_OSWUAutoUpdateOptionsName',\n 'Census_GenuineStateName',\n 'Census_ActivationChannel',\n 'Census_FlightRing',\n 'Census_FirmwareManufacturerIdentifier',\n 'Census_FirmwareVersionIdentifier',\n 'Wdft_RegionIdentifier'\n ]\n \n assert(len(actuallyBinary) + len(actuallyInteger) + len(actuallyFloat) + len(actuallyCategorical) == len(dataset.columns),\n \"Mismatch in number of features hard coded and the number of features encountered\")\n \n dataset[actuallyBinary] = dataset[actuallyBinary].fillna(value=0, axis='columns').astype(np.int8)\n dataset[actuallyInteger] = dataset[actuallyInteger].fillna(value=0, axis='columns').astype(np.int32)\n dataset[actuallyFloat] = dataset[actuallyFloat].fillna(value=0.0, axis='columns').astype(np.float32)\n dataset[actuallyCategorical] = dataset[actuallyCategorical].astype('category')\n \n return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Moment of truth, lets apply the big function on the training data. (and yes, I did hardcode in the features). Notice the decrease in the memory usage."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train = dtypeFixer(train, training=True)\n\ntrain.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Phew that took a long while to do."},{"metadata":{},"cell_type":"markdown","source":"### Dealing with redundant features\n\nWe saw two features having the same values. Let's try to check if there are any others. We did this by looking at a sample of the dataset. Note that 2 features might have the same categories, but different values in each of the features. Even in that case, we will be dropping one of the features(chosen randomly), because we don't want redundant data in the dataset."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train.sample(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that the following features seem redundant:\n1. ```Processor``` and ```Census_OSArchitecture```\n2. ```OsBuild``` and ```Census_OSBuildNumber```\n3. ```OsBuildLab``` is redundant with a lot of other features.\n4. ```Census_OSEdition``` and ```Census_OSSkuName```\n\nIn all cases, the latter will be dropped"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"redundantFeatures = [\n 'Census_OSArchitecture',\n 'Census_OSBuildNumber',\n 'Census_OSSkuName',\n 'OsBuildLab'\n]\n\nfeatureDropList = featureDropList + redundantFeatures\n\ntrain = train.drop(labels=redundantFeatures, axis=1)\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Dealing with bad category distribution\n\nAs we saw earlier, some categories have too many entries into 1/few categories and nothing at all in the others. The threshold for decision here is: minority categories should compose atleast 5-10% of the total dataset feature."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"badDistributionCategories = [\n 'IsBeta',\n 'Census_IsFlightsDisabled',\n 'SMode',\n 'Census_IsPortableOperatingSystem',\n 'AutoSampleOptIn',\n 'Census_IsVirtualDevice',\n 'HasTpm',\n 'AVProductsEnabled',\n 'OsSuite',\n 'RtpStateBitfield',\n 'UacLuaenable',\n 'ProductName',\n 'OsVer',\n 'Census_DeviceFamily',\n 'Census_FlightRing' # Unknown class seems to form second majority category\n]\n\nfor feature in badDistributionCategories:\n print(train[feature].value_counts(), end='\\n\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Good bye! Also, this is where we diverge from our original EDA notebook. Things that are done from now on are likely to be slighly different(2-3 features missing). Due to some mistake in the original code, there are a few mistakes in the features selected. We don't think the score will change because of it."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train = train.drop(labels=badDistributionCategories, axis=1)\nfeatureDropList += badDistributionCategories\n\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Analyze remaining features manually\n\nOk, now the original suffering starts. I will be looking at each and every of these features manually. I have already done this earlier, but there were mistakes earlier. Small changes only, but I will be fixing things here.(a few different features). This will (hopefully) not change the performance significantly.\n\nIf you see the EDA notebook from here on, you will notice upto 7 changes(I think). Kindly see this notebook only.\n\n**Note**: Since this code is actually taken from the original EDA notebook, I might load in the training data again, to prevent any confusions later on."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"cat_cols = train.select_dtypes(include='category').columns\ncat_cols = sorted(cat_cols, reverse=True, key=lambda col: train[col].isna().sum())\n\nprint(cat_cols, len(cat_cols))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Comparison with what we had done till here in our submission notebooks. Not a part of the current flow."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"old = {'OrganizationIdentifier', 'CityIdentifier', 'Wdft_RegionIdentifier', 'Census_FirmwareManufacturerIdentifier', 'Census_FirmwareVersionIdentifier', 'Census_OEMModelIdentifier', 'Census_OEMNameIdentifier', 'IeVerIdentifier', 'Census_OSInstallLanguageIdentifier', 'AVProductStatesIdentifier', 'Census_PrimaryDiskTypeName', 'Census_ProcessorModelIdentifier', 'Census_ChassisTypeName', 'GeoNameIdentifier', 'OsBuildLab', 'Census_PowerPlatformRoleName', 'ProductName', 'EngineVersion', 'AppVersion', 'AvSigVersion', 'CountryIdentifier', 'LocaleEnglishNameIdentifier', 'Platform', 'Processor', 'OsVer', 'OsBuild', 'OsPlatformSubRelease', 'SkuEdition', 'Census_MDC2FormFactor', 'Census_DeviceFamily', 'Census_OSVersion', 'Census_OSBranch', 'Census_OSBuildRevision', 'Census_OSEdition', 'Census_OSSkuName', 'Census_OSInstallTypeName', 'Census_OSUILocaleIdentifier', 'Census_OSWUAutoUpdateOptionsName', 'Census_GenuineStateName', 'Census_ActivationChannel', 'Census_FlightRing'}\nold - set(cat_cols), set(cat_cols) - old","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define a function to plot our distribution plots. Arguably, this is not a very good plotting method, but it works fine."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"def plotClassDistribution(feature, normalize=True):\n temp = train[[feature, OUTPUT_COL]] #.dropna()\n temp = pd.get_dummies(data=temp, columns=[OUTPUT_COL])\n temp = temp.groupby(feature).agg(np.count_nonzero).reset_index()\n \n cat_cols = [OUTPUT_COL+\"_\"+str(i) for i in range(0, 2)]\n if normalize:\n temp[cat_cols] = temp[cat_cols].div(temp[cat_cols].sum(axis=1), axis=0)\n else:\n print(temp)\n \n plt.figure(figsize=(10, 8))\n plt.plot(temp[cat_cols], temp[feature].astype('object'), 'o', markersize=10)\n plt.grid()\n plt.legend([\"No Malware\", \"Has Malware\"])\n plt.xlabel(OUTPUT_COL)\n plt.ylabel(feature)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**And so it begins from here.** \n\nFeature [1/36]"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"feature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This feature will be dropped on ground of having too many categories to work with."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"featureDropList.append(feature)\ntrain = train.drop(labels=[feature], axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Moving on to the next feature\n\nFeature[2/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"def fixFeature1(dataset):\n counts = train[feature].value_counts()\n sub10k = counts[counts < 10000].index\n \n dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n dataset[feature] = dataset[feature].apply(lambda x: 'sub10k' if x in sub10k else x)\n\n mapping = {\n feature: \n {\"na\": \"NA\",\n \"fixers\": [{\"sub10k\": sub10k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature1(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [3/36]"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"def fixFeature3(dataset):\n counts = train[feature].value_counts()\n sub1k = counts[counts < 1000].index\n \n dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub1k else x)\n \n mapping = {\n feature: \n {\"na\": \"NA\",\n \"fixers\": [{\"sub1k\": sub1k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature3(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On to next feature.\n\nFeature [4/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again, too many categories....drop this one too."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"featureDropList.append(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On to feature [5/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is no clean divide, so I will be removing this for now. If SVMs work, then I will try adding this one back in with a non-linear transform."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"featureDropList.append(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [6/36] next"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature6(dataset):\n counts = train[feature].value_counts()\n \n dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n \n sub1k = counts[counts <= 1000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'sub1k' if x in sub1k else x)\n \n sub5k = counts[(counts > 1000) & (counts < 5500)].index\n dataset[feature] = dataset[feature].apply(lambda x: 'range_1k_5-5k' if x in sub5k else x)\n \n mapping = {\n feature: \n {\"na\": \"NA\",\n \"fixers\": [{\"range_1k_5-5k\": sub5k}, {\"sub1k\": sub1k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature6(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [7/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature7(dataset):\n counts = train[feature].value_counts()\n \n dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n \n sub1k = counts[counts <= 1000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'sub1k' if x in sub1k else x)\n \n sub5k = counts[(counts > 1000) & (counts < 5500)].index\n dataset[feature] = dataset[feature].apply(lambda x: 'range_1k_5-5k' if x in sub5k else x)\n\n mapping = {\n feature: \n {\"na\": \"not_org\",\n \"fixers\": [{\"range_1k_5-5k\": sub5k}, {\"sub1k\": sub1k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature7(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [8/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature8(dataset):\n counts = train[feature].value_counts()\n \n dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n \n sub1k = counts[counts <= 1000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'sub1k' if x in sub1k else x)\n \n sub5k = counts[(counts > 1000) & (counts < 5000)].index\n dataset[feature] = dataset[feature].apply(lambda x: 'range_1k_5k' if x in sub5k else x)\n \n mapping = {\n feature: \n {\"na\": \"NA\",\n \"fixers\": [{\"range_1k_5-5k\": sub5k}, {\"sub1k\": sub1k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature8(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [9/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i} :{feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Didn't bother doing anything apart from fillna. During original analysis, this was done at 3AM, so didn't fix this one"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature9(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n mapping = {\n feature: \n {\"na\": \"NA\",\n \"fixers\": []\n }\n }\n\n return mapping, dataset\n\nmapping, train = fixFeature9(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [10/36]. **This is differnt from the original EDA notebook.**"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i} :{feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am going to collect the last 3 categories into ```3.0```"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature9a(dataset):\n counts = train[feature].value_counts()\n \n dataset[feature] = dataset[feature].cat.add_categories('NOTA').fillna('NOTA')\n \n sub1k = counts[counts <= 1000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'NOTA' if x in sub1k else x)\n \n mapping = {\n feature: \n {\"na\": \"NA\",\n \"fixers\": [{\"NOTA\": sub1k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature9a(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [11/36]. This one was handled slightly differently in the original notebook. No difference in final outcome."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i} :{feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Just fix the NAs"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature10(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n \n mapping = {\n feature: \n {\"na\": \"NA\",\n \"fixers\": []\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature10(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [12/36]. A third way through already."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i} :{feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again the problem with too many categories. Going to drop it."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"featureDropList.append(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [13/36]. This is different from the original notebook."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Will collect the lower count ones and missing values into one category."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature12a(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('NOTA').fillna('NOTA')\n \n sub1k = counts[counts <= 1000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'NOTA' if x in sub1k else x)\n \n mapping = {\n feature: \n {\"na\": \"NOTA\",\n \"fixers\": [{\"NOTA\": sub1k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature12a(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [14/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This feature can tell us something about the place of usage of the computer. But then..pizza box as a computer chassis? Nope. That's when you ditch the feature. Nah, the real reason is the category imbalance. Too many notebook computers these days. I'll fix it"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"featureDropList.append(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [15/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this feature, the data is well distributed over many categories. It would make a nice plot, but meh. I'll just bucket it"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature15(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('sub_3k').fillna('sub_3k')\n \n sub3k = counts[counts < 3000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'sub_3k' if x in sub3k else x)\n \n range8k = counts[(counts >= 3000) & (counts <= 8000)].index\n dataset[feature] = dataset[feature].apply(lambda x: 'range_3k_8k' if x in range8k else x)\n \n mapping = {\n feature: \n {\"na\": \"sub_3k\",\n \"fixers\": [{\"sub3k\": sub3k}, {\"range_3k_8k\": range8k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature15(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [16/36] By this point, the original notebook is actually completely wrong. There was an extra ```i+=1``` somewhere."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'll bucket everything under 10k count into a single bucket."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature16a(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('NOTA').fillna('NOTA')\n \n sub10k = counts[counts < 10000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'NOTA' if x in sub10k else x)\n \n mapping = {\n feature: \n {\"na\": \"NOTA\",\n \"fixers\": [{\"NOTA\": sub10k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature16a(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [17/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# print(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So for this, we took a 2-vs-all category sort of thing"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature16(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n \n sub20k = counts[counts < 20000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub20k else x)\n \n mapping = {\n feature: \n {\"na\": \"others\",\n \"fixers\": [{\"others\": sub20k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature16(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [18/36] halfway through."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature17(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('sub20k').fillna('sub20k')\n \n sub20k = counts[counts < 20000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'sub20k' if x in sub20k else x)\n \n range20_100k = counts[(counts >= 20000) & (counts < 100000)].index\n dataset[feature] = dataset[feature].apply(lambda x: 'range20_100k' if x in range20_100k else x)\n \n mapping = {\n feature: \n {\"na\": \"sub20k\",\n \"fixers\": [{\"sub20k\": sub20k}, {\"range20_100k\": range20_100k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature17(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [19/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(dict(train[feature].value_counts()))\n\n# Too many categories to plot. Don't uncomment this.\n# plotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Nope. Goodbye"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"featureDropList.append(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [20/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now the problem with this feature is that the data is uniformaly spread over many classes. This makes bucketing really cumbersome to do, with no guaratees of performance. On top of that, the plot looks nearly symmetric after scaling. This makes it useless, will drop this feature."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"featureDropList.append(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [21/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again, the problem with this feature is that the data is too granular. It is spread so well, that on bucketing, it will form a new majority class, which is flat wrong. I don't know about trees, but this will cause misclassification. I will drop this feature."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"featureDropList.append(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [22/36]. Got to see the raw values in this feature."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I will make this Windows 10 vs others. Stupid Windows, Linux is life"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature22(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('notW10').fillna('notW10')\n \n sub20k = counts[counts < 20000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'notW10' if x in sub20k else x)\n \n mapping = {\n feature: \n {\"na\": \"notW10\",\n \"fixers\": [{\"notW10\": sub20k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature22(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [23/36]. This is taking forever."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am very sorry ARM. You are awesome, but for now, you are a part of x86"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature23(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].fillna('x86')\n \n sub1k = counts[counts < 1000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'x86' if x in sub1k else x)\n \n mapping = {\n feature: \n {\"na\": \"x86\",\n \"fixers\": [{\"x86\": sub1k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature23(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [24/36]. two-thirds way through. I can do it."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature24(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n \n sub20k = counts[counts < 20000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub20k else x)\n \n mapping = {\n feature: \n {\"na\": \"others\",\n \"fixers\": [{\"others\": sub20k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature24(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [25/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distribution of prediction feature seems to be uniform across categories. It might not be worth it to have this, but I will keep it, regularisation can remove it. Its such a clean feature."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature25(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n \n sub25k = counts[counts < 25000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub25k else x)\n \n mapping = {\n feature: \n {\"na\": \"others\",\n \"fixers\": [{\"others\": sub25k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature25(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [26/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Same analysis as the previous feature. This one is more likely to be useless because the majority category has too many entries.\n\nI feel bad about merging the servers with the trashy regular versions, but what to do."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature26(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n \n sub25k = counts[counts < 25000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub25k else x)\n \n mapping = {\n feature: \n {\"na\": \"others\",\n \"fixers\": [{\"others\": sub25k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature26(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [27/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again....but this one goes. I can't afford to keep too many such features. While the form factor of the PC gives good indication of the type of OS and its usage, we have those features already. I am not even going to bother to check the correlation(can't also, categorical features)."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"featureDropList.append(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [28/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\n# Too many categories\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sorry about the plot, but what the hell."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature28(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('sub2k').fillna('sub2k')\n \n sub2k = counts[counts < 2000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'sub2k' if x in sub2k else x)\n \n sub5k = counts[(counts >= 2000) &(counts < 5000)].index\n dataset[feature] = dataset[feature].apply(lambda x: 'range_2_5k' if x in sub5k else x)\n \n sub10k = counts[(counts >= 5000) &(counts < 10000)].index\n dataset[feature] = dataset[feature].apply(lambda x: 'range_5_10k' if x in sub10k else x)\n \n mapping = {\n feature: \n {\"na\": \"others\",\n \"fixers\": [{\"sub2k\": sub2k}, {\"range_2_5k\": sub5k}, {\"range_5_10k\": sub10k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature28(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I don't mind these many categories.\n\nFeature [29/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I don't feel good about bucketing here, because the class distribution across categories isn't nice. However, if I don't bucket, I remove this feature. So let L1 regularisation decide that."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature29(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n \n sub25k = counts[counts < 25000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub25k else x)\n \n mapping = {\n feature: \n {\"na\": \"others\",\n \"fixers\": [{\"others\": sub25k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature29(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [30/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature30(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('sub2k').fillna('sub2k')\n \n sub5k = counts[counts < 5000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'sub5k' if x in sub5k else x)\n \n sub10k = counts[(counts >= 5000) &(counts < 10000)].index\n dataset[feature] = dataset[feature].apply(lambda x: 'range_5_10k' if x in sub10k else x)\n \n sub20k = counts[(counts >= 10000) &(counts < 20000)].index\n dataset[feature] = dataset[feature].apply(lambda x: 'range_10_20k' if x in sub20k else x)\n \n mapping = {\n feature: \n {\"na\": \"others\",\n \"fixers\": [{\"sub5k\": sub5k}, {\"range_5_10k\": sub10k}, {\"range_10_20k\": sub20k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature30(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [31/36] Almost done, last lap"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Gonna do 3-vs-all. Seems like an unhelpful feature."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature31(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n \n sub25k = counts[counts < 25000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub25k else x)\n \n mapping = {\n feature: \n {\"na\": \"others\",\n \"fixers\": [{\"others\": sub25k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature31(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [32/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature32(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].fillna('Other')\n \n sub5k = counts[counts < 5000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'Clean' if x in sub5k else x)\n \n mapping = {\n feature: \n {\"na\": \"Other\",\n \"fixers\": [{\"Clean\": sub5k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature32(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [33/36]"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Bucket and fix it, not even going to think about it."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature33(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('sub_10k').fillna('sub_10k')\n \n sub10k = counts[counts < 10000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'sub_10k' if x in sub10k else x)\n \n sub20k = counts[(counts >= 10000) & (counts < 20000)].index\n dataset[feature] = dataset[feature].apply(lambda x: 'range_10_20k' if x in sub20k else x)\n \n mapping = {\n feature: \n {\"na\": \"others\",\n \"fixers\": [{\"sub_10k\": sub10k}, {\"range_10_20k\": sub20k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature33(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [34/36]. Last 3, before another gate of hell opens(sorry)."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You know what, I am going to let this be. Just convert NA's to UNKNOWN"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"mapping = {\n feature: \n {\"na\": \"UNKNOWN\",\n \"fixers\": []\n }\n }\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [35/36]. Last 2."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am going to convert UNKNOWN into Offline, because you don't know about the state, because the machine is offline. Or the pirated version disables internet access for that particular software."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature35(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].fillna('OFFLINE')\n \n sub5k = counts[counts < 5000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'OFFLINE' if x in sub5k else x)\n \n mapping = {\n feature: \n {\"na\": \"OFFLINE\",\n \"fixers\": [{\"OFFLINE\": sub5k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature35(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature [36/36]. Last one!!"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n# pprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Group the categories with too few counts into 'others'"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def fixFeature36(dataset):\n counts = train[feature].value_counts()\n dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n \n sub30k = counts[counts < 30000].index\n dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub30k else x)\n \n mapping = {\n feature: \n {\"na\": \"others\",\n \"fixers\": [{\"others\": sub30k}]\n }\n }\n\n return mapping, dataset\n \nmapping, train = fixFeature36(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\ncategoricalFixerMappings.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Done with all preprocessing now\n\nI have one array one all the features to drop. And another dict of mappings to fix the categorical columns. This dict will be saved in a binary file called ```CatMapping.sav``` too. This will help with the Python script.\n\nSo let me print out all the features that I am dropping. Along with this, I will also print the conversion dict, which is going to be pretty long. So warning, huge cells with seeming nonsense in them up ahead."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"featureDropList, len(featureDropList)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"with open('CatMapping.sav', 'wb') as fp:\n pickle.dump(categoricalFixerMappings, fp, protocol=pickle.HIGHEST_PROTOCOL)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"categoricalFixerMappings","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"del test, train, categoricalFixerMappings\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparing the data for training\n\nI will reload the datasets so that there is no collisions between the EDA and this training. We actually had 2 separate notebooks for this, This is the second notebook mentioned in the very first cell."},{"metadata":{},"cell_type":"markdown","source":"### Reload the datasets and the mapping dict"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train = pd.read_csv('../input/Malware-detection-NS/train.csv', low_memory=False)\ntest = pd.read_csv('../input/Malware-detection-NS/test.csv', low_memory=False)\n\nwith open('CatMapping.sav', 'rb') as fp:\n CategoricalMapping = pickle.load(fp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I will drop the Id column from the training data for good measure."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train.drop(labels=[ID_COL], axis=1, inplace=True)\ntrain.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### First step is to fix the data types\n\nLet's use the function that I defined a hundred or so cells up."},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train = dtypeFixer(train, training=True)\ntest = dtypeFixer(test, training=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let me drop the unnecessary features now"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train = train.drop(labels=featureDropList, axis=1)\ntest = test.drop(labels=featureDropList, axis=1)\n\ntrain.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Fix all the categorical features"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def CategoricalFixer(data, verbose=False):\n dataset = data.copy(deep=True)\n \n for feature in CategoricalMapping:\n if feature in dataset.columns.values:\n \n if verbose:\n print(f\"Fixing {feature} feature now.\")\n dataset[feature] = dataset[feature].astype('category')\n\n NAreplacement = CategoricalMapping[feature][\"na\"]\n buckets = CategoricalMapping[feature][\"fixers\"]\n\n if NAreplacement not in dataset[feature].unique():\n dataset[feature] = dataset[feature].cat.add_categories(NAreplacement)\n\n dataset[feature] = dataset[feature].fillna(NAreplacement)\n\n for bucket in buckets:\n name = list(bucket.keys())[0]\n mapping = bucket[name]\n dataset[feature] = dataset[feature].apply(lambda x: name if x in mapping else x)\n \n dataset[feature] = dataset[feature].astype('object')\n\n categorical_cols = dataset.select_dtypes(include='object').columns\n for feature in categorical_cols:\n dataset[feature] = dataset[feature].apply(str)\n \n return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train = CategoricalFixer(train)\ntest = CategoricalFixer(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Take a look at the train data to verify if everything is fine"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Beautiful so far. In our original code, we had removed more features, due a mistake in the preprocessing notebook. The original code for this sub part of the notebook is exactly the same(just reorganized better). The original code is linked in, in the very first cell of the notebook.\n\n(Note the reduced memory usage here)"},{"metadata":{},"cell_type":"markdown","source":"### Encode the data and form datasets\n\nFirst encode all the categorical features. Meanwhile, normalize all the numerical fields. All these don't make a lot of sense for Trees, but we will show a few linear models too.\n\nThen split the data into train and local test sets.\n\nFor the linear models, we will not have bother with any validation data. The LGBM tree model will use stratified evaluation sets,"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"y = train[OUTPUT_COL].values\ntrain = train.drop(labels=[OUTPUT_COL], axis=1)\n\ncategoricalCols = train.select_dtypes(include='object').columns\nnumericalCols = train.select_dtypes(exclude='object').columns\n\n# Define the preprocessing models\nscaler = StandardScaler()\nOHE = OneHotEncoder(handle_unknown='ignore')\n\ntemp = scaler.fit_transform(train[numericalCols])\nsparseNumeric = scsparse.csr_matrix(temp)\n\nencodedTrain = OHE.fit_transform(train[categoricalCols])\n\nX = scsparse.hstack([sparseNumeric, encodedTrain])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Now form train and test splits"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=6, stratify=y)\n\nX_train.shape, X_test.shape, y_train.shape, y_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"del X, y\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Fix class imbalance\n\nI will randomly upsample the minority class to improve the class imbalance. Something like SMOTE would be more effective, but SMOTE can't work on one-hot encoded data. This will only be done on the training dataset, because the test data and the local test dataset must be left untouched."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"oversampler = RandomOverSampler(sampling_strategy=0.3, random_state=1)\n\nX_train, y_train = oversampler.fit_resample(X_train, y_train)\nX_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train a few models for benchmarking\n\nThese will train on the entire training data, with no cross validation. The ```CalibratedClassifierCV``` is to get a probability estimate out of the model.(They don't have a ```predict_proba``` method)."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"NBcore = BernoulliNB(alpha=1e-6)\ntempNBModel = CalibratedClassifierCV(NBcore, method='isotonic', cv=6)\n\ntempNBModel.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"SVCcore = LinearSVC(\n C=0.3,\n dual=False,\n class_weight=\"balanced\",\n penalty='l1',\n max_iter=1500, # With these few iterations, it will never converge\n random_state=66,\n verbose=2\n)\ntempSVCModel = CalibratedClassifierCV(SVCcore, method='isotonic', cv=4)\n\n# This takes a long while don't run if you don't have time\n# tempSVCModel.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"NBpreds = tempNBModel.predict_proba(X_test)[:, 1]\n# SVMpreds = tempSVCModel.predict_proba(X_test)[:, 1]\n\nprint(f\"Simple Bernoulli Naive Bayes model got AUC score: {roc_auc_score(y_test, NBpreds)}\")\n# print(f\"Linear SVM classifier model got AUC score: {roc_auc_score(y_test, SVMpreds)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Real model training\n\nThis is the final model that we will be submitting. It is a LightGBM's Gradient Boosted Classifier, trained with 8 fold cross validation.\n\nFirst I will split the data"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"splitter = StratifiedKFold(n_splits=8, shuffle=True, random_state=8)\nsplitter.get_n_splits(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now define, fit and train the model!!!"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"model = None\nfor train_index, val_index in splitter.split(X_train, y_train):\n X_train_split, X_val = X_train[train_index], X_train[val_index]\n y_train_split, y_val = y_train[train_index], y_train[val_index]\n \n model = LGBMClassifier(\n num_leaves=64,\n n_estimators=1600,\n learning_rate=0.07,\n objective='binary',\n class_weight={0: 1, 1:3},\n subsample_for_bin=20000,\n subsample=0.8,\n reg_alpha=0.2,\n reg_lambda=0.05,\n random_state=4,\n n_jobs=-1,\n silent=False\n )\n\n model.fit(X_train_split, y_train_split, eval_set=[(X_val, y_val)], eval_metric='auc', early_stopping_rounds=50)\n\nmodel","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Evaluate the model on a local test set"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"predictions = model.predict_proba(X_test)[:, 1]\nroc_auc_score(y_test, predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Get predictions on the actual test dataset\n\nThis is the real deal.\n\n1. First extract the ID column for the final submission\n2. Then scale and one hot encode the data for training.\n3. Then get the predictions!\n\n\nThere is some error here, which we were not able to fix in time. The Python script does not have this problem, so please use that. The code is almost exactly the same, minus this elusive bug."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"identifiers = test[ID_COL]\ntest = test.drop(labels=[ID_COL], axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Scale the data accordingly"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"sparseNumericTest = scsparse.csr_matrix(scaler.transform(train[numericalCols]))\nencodedTest = scsparse.csr_matrix(OHE.transform(train[categoricalCols]))\n\nfinalTest = scsparse.csr_matrix(scsparse.hstack([sparseNumericTest, encodedTest]))\nfinalTest","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get the predictions"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"testPredictions = model.predict_proba(finalTest)[:, 1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Write the predictions to the output submission file."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"output = pd.DataFrame([identifiers, testPredictions], [ID_COL, OUTPUT_COL]).transpose()\noutput.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}