{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom pprint import pprint","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Import the datasets and get stats"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"scrolled":false},"cell_type":"code","source":"train = pd.read_csv('../input/Malware-detection-NS/train.csv', low_memory=False)\ntest = pd.read_csv('../input/Malware-detection-NS/test.csv', low_memory=False)\n\nPredictionCol = 'HasDetections'\nIdentifierCol = 'MachineIdentifier'\n\ntrain.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's get our hands dirty"},{"metadata":{},"cell_type":"markdown","source":"Ughh....\n\n**MachineIdentifie**r - Individual machine ID\n\n**ProductName** - Defender state information e.g. win8defender\n\n**EngineVersion** - Defender state information e.g. 1.1.12603.0\n\n**AppVersion** - Defender state information e.g. 4.9.10586.0\n\n**AvSigVersion** - Defender state information e.g. 1.217.1014.0\n\n**IsBeta** - Defender state information e.g. false\n\n**DefaultBrowsersIdentifier** - ID for the machine's default browser\n\n**AVProductStatesIdentifier** - ID for the specific configuration of a user's antivirus software\n\n**HasTpm** - True if machine has tpm\n\n**CountryIdentifier** - ID for the country the machine is located in\n\n**CityIdentifier** - ID for the city the machine is located in\n\n**OrganizationIdentifier** - ID for the organization the machine belongs in, organization ID is mapped to both specific companies and broad industries\n\n**GeoNameIdentifier** - ID for the geographic region a machine is located in\n\n**LocaleEnglishNameIdentifier** - English name of Locale ID of the current user\n\n**Platform** - Calculates platform name (of OS related properties and processor property)\n\n**Processor** - This is the process architecture of the installed operating system\n\n**OsVer** - Version of the current operating system\n\n**OsBuild** - Build of the current operating system\n\n**OsSuite** - Product suite mask for the current operating system.\n\n**OsPlatformSubRelease** - Returns the OS Platform sub-release (Windows Vista, Windows 7, Windows 8, TH1, TH2)\n\n**OsBuildLab** - Build lab that generated the current OS. Example: 9600.17630.amd64fre.winblue_r7.150109-2022\n\n**SkuEdition** - The goal of this feature is to use the Product Type defined in the MSDN to map to a 'SKU-Edition' name that is useful in population reporting. The valid Product Type are defined in %sdxroot%\\data\\windowseditions.xml. This API has been used since Vista and Server 2008, so there are many Product Types that do not apply to Windows 10. The 'SKU-Edition' is a string value that is in one of three classes of results. The design must hand each class.\n\n**IsProtected** - This is a calculated field derived from the Spynet Report's AV Products field. Returns: a. TRUE if there is at least one active and up-to-date antivirus product running on this machine. b. FALSE if there is no active AV product on this machine, or if the AV is active, but is not receiving the latest updates. c. null if there are no Anti Virus Products in the report. Returns: Whether a machine is protected.\n\n**AutoSampleOptIn** - This is the SubmitSamplesConsent value passed in from the service, available on CAMP 9+\n\n**PuaMode** - Pua Enabled mode from the service\n\n**SMode** - This field is set to true when the device is known to be in 'S Mode', as in, Windows 10 S mode, where only Microsoft Store apps can be installed\n\n**SmartScreen** - This is the SmartScreen enabled string value from registry. This is obtained by checking in order, HKLM\\SOFTWARE\\Policies\\Microsoft\\Windows\\System\\SmartScreenEnabled and HKLM\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Explorer\\SmartScreenEnabled. If the value exists but is blank, the value \"ExistsNotSet\" is sent in telemetry.\n\n**Firewall** - This attribute is true (1) for Windows 8.1 and above if windows firewall is enabled, as reported by the service.\n\n**UacLuaenable** - This attribute reports whether or not the \"administrator in Admin Approval Mode\" user type is disabled or enabled in UAC. The value reported is obtained by reading the regkey HKLM\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Policies\\System\\EnableLUA.\n\n**Census_MDC2FormFactor** - A grouping based on a combination of Device Census level hardware characteristics. The logic used to define Form Factor is rooted in business and industry standards and aligns with how people think about their device. (Examples: Smartphone, Small Tablet, All in One, Convertible...)\n\nCensus_DeviceFamily - AKA DeviceClass. Indicates the type of device that an edition of the OS is intended for. Example values: Windows.Desktop, Windows.Mobile, and iOS.Phone\n\nCensus_ProcessorCoreCount - Number of logical cores in the processor\n\nCensus_ProcessorClass - A classification of processors into high/medium/low. Initially used for Pricing Level SKU. No longer maintained and updated\n\nCensus_PrimaryDiskTotalCapacity - Amount of disk space on primary disk of the machine in MB\n\nCensus_PrimaryDiskTypeName - Friendly name of Primary Disk Type - HDD or SSD\n\nCensus_SystemVolumeTotalCapacity - The size of the partition that the System volume is installed on in MB\n\nCensus_HasOpticalDiskDrive - True indicates that the machine has an optical disk drive (CD/DVD)\n\nCensus_TotalPhysicalRAM - Retrieves the physical RAM in MB\n\nCensus_ChassisTypeName - Retrieves a numeric representation of what type of chassis the machine has. A value of 0 means xx\n\nCensus_InternalPrimaryDiagonalDisplaySizeInInches - Retrieves the physical diagonal length in inches of the primary display\n\nCensus_InternalPrimaryDisplayResolutionHorizontal - Retrieves the number of pixels in the horizontal direction of the internal display.\n\nCensus_InternalPrimaryDisplayResolutionVertical - Retrieves the number of pixels in the vertical direction of the internal display\n\nCensus_PowerPlatformRoleName - Indicates the OEM preferred power management profile. This value helps identify the basic form factor of the device\n\n\nCensus_OSVersion - Numeric OS version Example - 10.0.10130.0\n\nCensus_OSArchitecture - Architecture on which the OS is based. Derived from OSVersionFull. Example - amd64\n\nCensus_OSBranch - Branch of the OS extracted from the OsVersionFull. Example - OsBranch = fbl_partner_eeap where OsVersion 6.4.9813.0.amd64fre.fbl_partner_eeap.140810-0005\n\nCensus_OSBuildNumber - OS Build number extracted from the OsVersionFull. Example - OsBuildNumber = 10512 or 10240\n\nCensus_OSBuildRevision - OS Build revision extracted from the OsVersionFull. Example - OsBuildRevision = 1000 or 16458\n\nCensus_OSEdition - Edition of the current OS. Sourced from HKLM\\Software\\Microsoft\\Windows NT\\CurrentVersion@EditionID in registry. Example: Enterprise\n\nCensus_OSSkuName - OS edition friendly name (currently Windows only)\n\nCensus_OSInstallTypeName - Friendly description of what install was used on the machine i.e. clean\n\nCensus_OSWUAutoUpdateOptionsName - Friendly name of the WindowsUpdate auto-update settings on the machine.\n\nCensus_IsPortableOperatingSystem - Indicates whether OS is booted up and running via Windows-To-Go on a USB stick.\n\nCensus_GenuineStateName - Friendly name of OSGenuineStateID. 0 = Genuine\n\nCensus_ActivationChannel - Retail license key or Volume license key for a machine.\n\nCensus_IsFlightsDisabled - Indicates if the machine is participating in flighting.\n\nCensus_FlightRing - The ring that the device user would like to receive flights for. This might be different from the ring of the OS which is currently installed if the user changes the ring after getting a flight from a different ring.\n\nCensus_IsSecureBootEnabled - Indicates if Secure Boot mode is enabled.\n\nCensus_IsVirtualDevice - Identifies a Virtual Machine (machine learning model)\n\nCensus_IsTouchEnabled - Is this a touch device ?\n\nCensus_IsPenCapable - Is the device capable of pen input ?\n\nCensus_IsAlwaysOnAlwaysConnectedCapable - Retreives information about whether the battery enables the device to be AlwaysOnAlwaysConnected .\n\nWdft_IsGamer - Indicates whether the device is a gamer device or not based on its hardware combination.\n\n------------------------------------------------------\n**Wdft_RegionIdentifier** - NA\n\n**RtpStateBitfield** - NA\n\n**IsSxsPassiveMode** - NA\n\nAVProductsInstalled - NA\n\nAVProductsEnabled - NA\n\nCensus_IsWIMBootEnabled - NA\n\nCensus_ThresholdOptIn - NA\n\nCensus_FirmwareManufacturerIdentifier - NA\n\nCensus_FirmwareVersionIdentifier - NA\n\nCensus_IsFlightingInternal - NA\n\nCensus_OSInstallLanguageIdentifier - NA\n\nCensus_OSUILocaleIdentifier - NA\n\nCensus_InternalBatteryType - NA\n\nCensus_InternalBatteryNumberOfCharges - NA\n\nCensus_ProcessorManufacturerIdentifier - NA\n\nCensus_ProcessorModelIdentifier - NA\n\nCensus_OEMNameIdentifier - NA\n\nCensus_OEMModelIdentifier - NA\n\nIeVerIdentifier - NA\n"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"DropColList = []\npd.set_option('max_columns', None)\n\ntrain.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"NA_count = list(sorted(zip(train.columns, train.isna().sum().values), reverse=True, key=lambda x: x[1]))\nfor col, count in NA_count:\n    if count > 10000:\n        print(f\"{col} has {count} NA values\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some features have too much missing. Goodbye. The threshold is 65% of the entries are missing information for the feature. This threshold was chosen, so that ```OrganizationIdentifier``` passes the cut. It might be useful later."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"list(filter(lambda x: x[1] > int(0.3 * train.shape[0]), NA_count))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No one cares about PuaMode apparently\n\nCensus_ProcessorClass: Not maintained anymore. So its okay to ditch\n\nMicrosoft wants to promote Edge, but no one cares about a ripoff. So bye-bye DefaultBrowsersIdentifier.\n\nDon't know what ```Census_IsFlightingInternal```, ```Census_ThresholdOptIn``` and ```Census_IsWIMBootEnabled``` are.\n\nBattery type and Malware? Nope ```Census_InternalBatteryType```.\n\n```SmartScreen``` has around 50% missing. There are too many categories to impute values to. So drop it too\n\n----------\n```OrganizationIdentifier``` also doesn't make the cut. However, it feels like an important feature for the entries that have it. We can isolate entire companies that had a malware attack that way. Needs further analysis."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"Identifiers = train[IdentifierCol]\n\nDropColList += [IdentifierCol, 'PuaMode',\n                     'Census_ProcessorClass',\n                     'DefaultBrowsersIdentifier',\n                     'Census_IsFlightingInternal',\n                     'Census_InternalBatteryType',\n                     'Census_ThresholdOptIn',\n                     'Census_IsWIMBootEnabled',\n                     'SmartScreen']\n\ntrain = train.drop(labels=[IdentifierCol, 'PuaMode',\n                     'Census_ProcessorClass',\n                     'DefaultBrowsersIdentifier',\n                     'Census_IsFlightingInternal',\n                     'Census_InternalBatteryType',\n                     'Census_ThresholdOptIn',\n                     'Census_IsWIMBootEnabled', \n                     'SmartScreen'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With that out of the way, let me see which numeric column is actually categorical. This will come from the number of unique values among them."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"categorical_cols = train.select_dtypes(include='object').columns\nnumeric_cols = train.select_dtypes(exclude='object').columns\n\nnumeric_cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"counts = []\nfor col in numeric_cols:\n    counts.append(len(train[col].unique()))\n\nfor col, count in sorted(zip(numeric_cols, counts), key=lambda x: x[1]):\n    print(f\"{col} has {count} unique values and {train[col].isna().sum()} missing values\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"for col, count in sorted(zip(numeric_cols, counts), key=lambda x: x[1]):\n    if count <= 20:\n        print(train[col].value_counts(),end='\\n\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```IsBeta```, ```Census_IsFlightsDisabled```, ```SMode```, ```Census_IsPortableOperatingSystem```, ```UacLuaenable```, ```HasTpm```, ```Census_IsVirtualDevice``` and ```AutoSampleOptIn``` are useless. They have very poor class distributions. Good-bye to you too. Decision criteria was minority class had less than 10k samples, as opposed to 500,000 in the other class(es). These will not help the model."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"DropColList += ['IsBeta',\n                   'Census_IsFlightsDisabled',\n                   'SMode',\n                   'Census_IsPortableOperatingSystem',\n                   'AutoSampleOptIn',\n                   'UacLuaenable',\n                    'Census_IsVirtualDevice',\n                   'HasTpm']\n\ntrain = train.drop(labels=['IsBeta',\n                   'Census_IsFlightsDisabled',\n                   'SMode',\n                   'Census_IsPortableOperatingSystem',\n                   'AutoSampleOptIn',\n                   'UacLuaenable',\n                    'Census_IsVirtualDevice',\n                   'HasTpm'], axis=1)\n\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"At this point, we have a near stable number of base features. If a particular row is missing too many faetures, then its better to remove the row. Let me test for that now."},{"metadata":{"trusted":true},"cell_type":"code","source":"missingPerRowCount = train.isnull().sum(axis=1)\n\nmissingPerRowCount.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, so any entry missing 8 or more features is flat out for now. I might go even stricter at some later point, depending on how much I need to impute."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Original shape: {train.shape}\")\nprint(f\"Dropping {missingPerRowCount[missingPerRowCount >= 8].shape[0]} rows\")\ntrain = train.drop(index=missingPerRowCount[missingPerRowCount > 8].index, axis=0)\n\nprint(f\"Final shape: {train.shape}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"Many of the features have bugger all number of entries in some **categories**. Yes they aren't numeric. Need to fix all that. Let me list them:\n\n1. ```Census_ProcessorManufacturerIdentifier``` (simply drop 2 categories)\n2. ```RtpStateBitfield``` (is this worth it?)\n3. ```AVProductsInstalled``` (Combine categories 3 to 6 into a single category)\n4. ```AVProductsEnabled``` (1.0 vs rest?)\n5. ```OsSuite``` (768 and 256 only)"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"def transforms(data, inplace=False):\n    if inplace:\n        dataset = data\n    else:\n        dataset = data.copy(deep=True)\n    \n    dataset['Census_ProcessorManufacturerIdentifier'] = dataset[dataset['Census_ProcessorManufacturerIdentifier'].isin([1., 5.])]\n    dataset[dataset['Census_ProcessorManufacturerIdentifier'].isin([5.])] = 0.\n    \n    dataset[~dataset['RtpStateBitfield'].isin([7.])] = 0.\n    dataset[dataset['RtpStateBitfield'].isin([7.])] = 1.\n    \n    dataset[~dataset['AVProductsEnabled'].isin([1.])] = 0.\n    \n    dataset[~dataset['AVProductsInstalled'].isin([1., 2.])] = 3.\n    \n    dataset[~dataset['OsSuite'].isin([256, 768])] = 256\n    dataset[dataset['OsSuite'].isin([256])] = 0\n    dataset[dataset['OsSuite'].isin([768])] = 1\n    \n    \n    features = ['Census_ProcessorManufacturerIdentifier', 'RtpStateBitfield', 'AVProductsEnabled', 'AVProductsInstalled', 'OsSuite']\n    dataset[features] = dataset[features].astype('category')\n    \n    return features, dataset\n    \n\nfeatures, data = transforms(train)\ntrain[features] = data[features]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From here on, I will selectively examine columns of different datatypes and see what they contain. Specifically for floats and ints."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"float_cols = train.select_dtypes(include='float64').columns\n\ntrain[float_cols].info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"train[float_cols].sample(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"for col in float_cols:\n    print(f\"{col} has: {len(train[col].unique())} unique values\")\n    if len(train[col].unique()) < 100:\n        print(train[col].value_counts(), end='\\n\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This sucks, but a lot lot lot needs to be analyzed and fiddled around with. Its 3:15A.M.; good night"},{"metadata":{},"cell_type":"markdown","source":"Let me group the float columns into: categorical, binary and actual floats\n\n#### Real floats\n1. Census_ProcessorCoreCount (which system has 112 cores? HPC?)\n2. Census_PrimaryDiskTotalCapacity\n3. Census_SystemVolumeTotalCapacity\n4. Census_TotalPhysicalRAM\n5. Census_InternalPrimaryDiagonalDisplaySizeInInches (might want to cluster and convert to categorical. Can tell device type)\n6. Census_InternalPrimaryDisplayResolutionHorizontal\n7. Census_InternalPrimaryDisplayResolutionVertical (6, 7: screen resolution got anything to do with malware? Combine and make categorical features for analysis)\n8. Census_InternalBatteryNumberOfCharges\n\n#### Binary with NaN\n1. IsProtected\n2. Firewall\n3. Census_IsAlwaysOnAlwaysConnectedCapable\n4. Wdft_IsGamer \n\n\n#### Categorical\n1. AVProductStatesIdentifier (6744 unique values)\n2. CityIdentifier (too many, 39k+ unique values)\n3. OrganizationIdentifier (loads missing, might have to see how to deal with it)\n4. GeoNameIdentifier (272 unique values)\n5. IeVerIdentifier (hate IE, drop it? JK, 189 unique values)\n6. Census_OEMNameIdentifier (1729 unique values)\n7. Census_OEMModelIdentifier (44k+ unique values. Will have to drop it, but will hold for analysis)\n8. Census_ProcessorModelIdentifier (2335 unique values. Seems useful)\n9. Census_OSInstallLanguageIdentifier\n10. Census_FirmwareManufacturerIdentifier\n11. Census_FirmwareVersionIdentifier (cluster and then reduce?)\n12. Wdft_RegionIdentifier\n\n\nStupid company enter their data right. Now repeat this nonsense for int64 columns(groan)."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"int_cols = train.select_dtypes(include='int64').columns\n\ntrain[int_cols].info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"train[int_cols].sample(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"for col in int_cols:\n    print(f\"{col} has: {len(train[col].unique())} unique values\")\n    if len(train[col].unique()) < 100:\n        print(train[col].value_counts(), end='\\n\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let me group the ints into categorical, binary and real ints\n\n#### Real integers\nnone\n\n#### Binary\n1. IsSxsPassiveMode\n2. Census_HasOpticalDiskDrive\n3. Census_IsSecureBootEnabled\n4. Census_IsTouchEnabled\n5. Census_IsPenCapable\n6. HasDetections\n\n#### Categorical\n1. CountryIdentifier\n2. LocaleEnglishNameIdentifier\n3. OsBuild\n4. Census_OSBuildNumber\n5. Census_OSBuildRevision\n6. Census_OSUILocaleIdentifier\n\nThere ... are no real integers. Utterly useless people."},{"metadata":{},"cell_type":"markdown","source":"Need to go through each of these and see how to reduce the total number of distinct categories.\n\nAtleast the object features are not actually integers, which is something to be grateful for. Typical Microsoft"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"def DtypeFixer(data, inplace=False): \n    if inplace:\n        dataset = data\n    else:\n        dataset = data.copy(deep=True)\n\n    # Converting these to integer is enough, I will use np.int16\n    actuallyBinary = ['IsProtected',\n                      'Firewall',\n                      'Wdft_IsGamer',\n                      'Census_IsAlwaysOnAlwaysConnectedCapable',\n                      \n                      'IsSxsPassiveMode',\n                    'Census_HasOpticalDiskDrive',\n                    'Census_IsSecureBootEnabled',\n                    'Census_IsSecureBootEnabled',\n                    'Census_IsPenCapable',\n                    'HasDetections',\n                    'IsProtected',\n                    'AVProductsEnabled',\n                    'Firewall',\n                    'Census_IsAlwaysOnAlwaysConnectedCapable',\n                    'Wdft_IsGamer',\n                    'Census_ProcessorManufacturerIdentifier',\n                    'RtpStateBitfield',\n                    'OsSuite'\n                    ]\n\n    \n    # These go to 64 bit float, if they aren't\n    actuallyFloat = ['Census_ProcessorCoreCount',\n                     'Census_PrimaryDiskTotalCapacity',\n                     'Census_SystemVolumeTotalCapacity',\n                     'Census_TotalPhysicalRAM',\n                     'Census_InternalPrimaryDiagonalDisplaySizeInInches',\n                     'Census_InternalBatteryNumberOfCharges'\n                    ]\n\n    # These go as categorical columns\n    actuallyCategorical = ['AVProductStatesIdentifier',\n                           'CityIdentifier',\n                           'OrganizationIdentifier',\n                           'GeoNameIdentifier',\n                           'IeVerIdentifier',\n                           'Census_OEMNameIdentifier',\n                           'Census_OEMModelIdentifier',\n                           'Census_ProcessorModelIdentifier',\n                           'Census_OSInstallLanguageIdentifier',\n                           'Census_FirmwareManufacturerIdentifier',\n                           'Census_FirmwareVersionIdentifier',\n                           'Wdft_RegionIdentifier',\n                           'CountryIdentifier',\n                           'LocaleEnglishNameIdentifier',\n                           'OsBuild',\n                           'Census_OSBuildNumber',\n                           'Census_OSBuildRevision',\n                           'Census_OSUILocaleIdentifier'\n                          ]\n    \n    objectFeatures = train.select_dtypes(include='object').columns\n\n    dataset[actuallyBinary] = dataset[actuallyBinary].fillna(method='bfill', axis=0)\n    dataset[actuallyBinary] = dataset[actuallyBinary].astype(np.int16)\n\n    dataset[actuallyFloat] = dataset[actuallyFloat].astype(float)\n    dataset[actuallyCategorical] = dataset[actuallyCategorical].astype('category')\n    dataset[objectFeatures] = dataset[objectFeatures].astype('category')\n\n    return dataset\n\ntrain = DtypeFixer(train)\n# train = dataset.copy(deep=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"cat_cols = train.select_dtypes(include='category').columns\nprint(len(cat_cols))\n\ntrain[cat_cols].sample(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"for col in cat_cols:\n    print(f\"{col} has: {len(train[col].unique())} unique values\")\n    if len(train[col].unique()) < 100:\n        print(train[col].value_counts(), end='\\n\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the categorical columns:\n\n1. ```ProductName``` can be removed safely? The other categories form an overwhelming minority which does not help anyone.\n2. ```EngineVersion``` needs heavy bucketing. Will decide, when I am going through each feature again. Can stay for now.\n3. ```AppVersion``` needs heavy bucketing. Will decide, when I am going through each feature again. Can stay for now.\n4. ```AvSigVersion``` can be removed safely? Too many categories, need to see.\n5. ```AVProductStatesIdentifier``` has too many categories. Withold till high depth analysis.\n6. ```AVProductsInstalled``` is a constant, **drop it.**\n7. ```AVProductsEnabled``` is a constant throughout too. **drop it**\n8. ```CountryIdentifier``` keep with bucketing?\n9. ```CityIdentifier``` either cluster somehow, or get out.\n10. ```OrganizationIdentifier``` might be valuable, in depth analysis needed.\n11. ```GeoNameIdentifier``` similar to 8, 9. Needs further analysis\n12. ```LocaleEnglishNameIdentifier``` possibly useless/can be combined with other features.\n13. ```Platform``` can be bucketed easily too. windows10 or not can be the decision boundary. Might as well remove it.\n14. ```Processor``` ARM can say bye-bye here I guess. They will be forced into x86 (a sin nowadays,but what the hell)\n15. ```OsVer``` either 10.0.0.0 or not. Better off removing it.\n16. ```OsBuild``` can shrink/combine with above.\n17. ```OsSuite``` has only 1 unique value, **drop it**\n18. ```OsPlatformSubRelease``` will stay, but will bucket all entries below 15k into one category\n19. ```OsBuildLab``` is a problem child. Will get special treatment, because everyone is special.\n20. ```SkuEdition``` will have a bucket for \"Industry\", which store all sub 5k categories\n21. ```IeVerIdentifier``` cluster maybe, depending on known outcomes.\n22. ```Census_MDC2FormFactor``` I hate this one, but if I keep it, I will be bucketing stuff into servers and tablets\n23. ```Census_DeviceFamily```, Lol, 3 \"Windows\".\n24. ```Census_OEMNameIdentifier``` might want to keep and cluster this one.\n25. ```Census_OEMModelIdentifier``` can somehow combine with 24? Otherwise drop\n26. ```Census_ProcessorManufacturerIdentifier``` has only 1 unique value, **drop it**\n27. ```Census_ProcessorModelIdentifier``` looks like very useful information. Need to filter it down.\n28. ```Census_PrimaryDiskTypeName``` finally some good \\<censored\\> feature. Still imbalanced of course.\n29. ```Census_ChassisTypeName``` I am so going to drop this one\n30. ```Census_PowerPlatformRoleName```: drop it? I think so\n31. ```Census_OSVersion``` problem child 2. No worries.\n32. ```Census_OSArchitecture``` redundant with 6, **drop it**\n33. ```Census_OSBranch``` bucket sub 10k categories\n34. ```Census_OSBuildNumber``` redundant with 16, **drop it**\n35. ```Census_OSBuildRevision``` definitely needs reduction, if not removal\n36. ```Census_OSEdition``` redundant with 9 and 19?\n37. ```Census_OSSkuName``` bucket, analyze with 9, 18\n38. ```Census_OSInstallTypeName``` merge the 2 \"Clean\"s\n39. ```Census_OSUILocaleIdentifier``` bucket the single occurances\n40. ```Census_OSWUAutoUpdateOptionsName``` might have to drop this one. Individual analysis needed\n41. ```Census_GenuineStateName``` merge unknown with offline? Need to see class distributions for each to decide\n42. ```Census_ActivationChannel``` redundant? \n43. ```Census_FlightRing``` might drop this one too. Category count imbalance.\n44. ```Census_OSInstallLanguageIdentifier``` not bad, will bucket sub 10k\n45. ```Census_FirmwareManufacturerIdentifier``` cluster or die out\n46. ```Census_FirmwareVersionIdentifier``` no choice but to drop I guess.\n47. ```Wdft_RegionIdentifier``` not bad...just remove one category and merge with something else\n\nOk, let's implement a function to first analyze each feature(with different calls). After analysis, actions will be taken.\n\nFor each categorical feature, first I will see the class distribution, then define a function that can fix the problem(s). This notebook is only going to be for EDA. I will copy these functions to another notebook, where there will be no analysis,  only code.\n\nBut first, remove useless stuff"},{"metadata":{"trusted":true},"cell_type":"code","source":"DropColList += ['AVProductsInstalled',\n                'AVProductsEnabled',\n                'OsSuite',\n                'Census_ProcessorManufacturerIdentifier',\n                'Census_OSArchitecture',\n                'Census_OSBuildNumber']\n\ntrain = train.drop(labels=['AVProductsInstalled',\n                'AVProductsEnabled',\n                'OsSuite',\n                'Census_ProcessorManufacturerIdentifier',\n                'Census_OSArchitecture',\n                'Census_OSBuildNumber'], axis=1)\n\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_cols = train.select_dtypes(include='category').columns\ncat_cols = sorted(cat_cols, reverse=True, key=lambda col: train[col].isna().sum())\n\nprint(cat_cols)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Individual feature EDA"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plotClassDistribution(feature, normalize=True):\n    temp = train[[feature, PredictionCol]] #.dropna()\n    temp = pd.get_dummies(data=temp, columns=[PredictionCol])\n    temp = temp.groupby(feature).agg(np.count_nonzero).reset_index()\n    \n    cat_cols = [PredictionCol+\"_\"+str(i) for i in range(0, 2)]\n    if normalize:\n        temp[cat_cols] = temp[cat_cols].div(temp[cat_cols].sum(axis=1), axis=0)\n    else:\n        print(temp)\n    \n    plt.figure(figsize=(10, 8))\n    plt.plot(temp[cat_cols], temp[feature].astype('object'), 'o', markersize=12)\n    plt.grid()\n    plt.legend([\"No Malware\", \"Has Malware\"])\n    plt.xlabel(PredictionCol)\n    plt.ylabel(feature)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Defining array of fixing functions and categorical values to drop"},{"metadata":{"trusted":true},"cell_type":"code","source":"CategoricalFixers = {}\nDropCatCols = []\ni = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature0(dataset):\n    counts = train[feature].value_counts()\n    sub5k = counts[counts < 5000].index\n\n    dataset[feature] = dataset[feature].cat.add_categories('not_org').fillna('not_org')\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub5k' if x in sub5k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"not_org\",\n                   \"fixers\": [{\"sub5k\": sub5k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature0(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some cities definitely have malware and others don't. For now, I will ignore all features with too many categories. 2:40AM is hardly the time to learn all these."},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)\n\ni += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'll make a new category for NA and all sub 10k categories will be grouped"},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature2(dataset):\n    counts = train[feature].value_counts()\n    sub10k = counts[counts < 10000].index\n    \n    dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub10k' if x in sub10k else x)\n\n    mapping = {\n                feature: \n                   {\"na\": \"NA\",\n                   \"fixers\": [{\"sub10k\": sub10k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature2(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature3(dataset):\n    counts = train[feature].value_counts()\n    sub1k = counts[counts < 1000].index\n    \n    dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub1k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"NA\",\n                   \"fixers\": [{\"sub1k\": sub1k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature3(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again too many categories, *sigh*"},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is no clean divide, so I will be removing this for now. If SVMs work, then I will try adding this one back in with a non-linear transform."},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature6(dataset):\n    counts = train[feature].value_counts()\n    \n    dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n    \n    sub1k = counts[counts <= 1000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub1k' if x in sub1k else x)\n    \n    sub5k = counts[(counts > 1000) & (counts < 5500)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range_1k_5-5k' if x in sub5k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"NA\",\n                   \"fixers\": [{\"range_1k_5-5k\": sub5k}, {\"sub1k\": sub1k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature6(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature7(dataset):\n    counts = train[feature].value_counts()\n    \n    dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n    \n    sub1k = counts[counts <= 1000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub1k' if x in sub1k else x)\n    \n    sub5k = counts[(counts > 1000) & (counts < 5500)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range_1k_5-5k' if x in sub5k else x)\n\n    mapping = {\n                feature: \n                   {\"na\": \"not_org\",\n                   \"fixers\": [{\"range_1k_5-5k\": sub5k}, {\"sub1k\": sub1k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature7(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature8(dataset):\n    counts = train[feature].value_counts()\n    \n    dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n    \n    sub1k = counts[counts <= 1000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub1k' if x in sub1k else x)\n    \n    sub5k = counts[(counts > 1000) & (counts < 5000)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range_1k_5k' if x in sub5k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"NA\",\n                   \"fixers\": [{\"range_1k_5-5k\": sub5k}, {\"sub1k\": sub1k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature8(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i} :{feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Don't bother doing anything apart from fillna"},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature9(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n    mapping = {\n                feature: \n                   {\"na\": \"NA\",\n                   \"fixers\": []\n                   }\n              }\n\n    return mapping, dataset\n\nmapping, train = fixFeature9(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i} :{feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature10(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('NA').fillna('NA')\n    \n    sub1k = counts[counts <= 1000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub1k' if x in sub1k else x)\n    \n    sub5k = counts[(counts > 1000) & (counts < 5000)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range_1k_5k' if x in sub5k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"not_org\",\n                   \"fixers\": [{\"range_1k_5k\": sub5k}, {\"sub1k\": sub1k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature10(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data is distributed near-uniformly over a lot of classes, so I don't think it would be of a lot of use. Ditch this one too."},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i} :{feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This feature can tell us something about the place of usage of the computer. But then..pizza box as a computer chassis? Nope. That's when you ditch the feature. Nah, the real reason is the category imbalance. Too many notebooks these days."},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There seem to be a few outliers, but I think we can manage with that."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature12(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('sub_3k').fillna('sub_3k')\n    \n    sub3k = counts[counts < 3000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub_3k' if x in sub3k else x)\n    \n    range8k = counts[(counts >= 3000) & (counts <= 8000)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range_3k_8k' if x in range8k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"sub_3k\",\n                   \"fixers\": [{\"sub3k\": sub3k}, {\"range_3k_8k\": range8k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature12(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"4AM, good night. About 43 more features to go."},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This column contains a lot of information, which is captured in other features anyways. So I am going to extract the part after the hyphen only. **later**. For now, I am dropping this feature on grounds of redundance."},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is probably the most sensible feature and therefore the most irritating. Production servers almost never get malware, which is valueable information. However, because of this, we have too few entries, which are swamped out by other categories. Grrrr.... I will have to group all the others into an other category now."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature13(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n    \n    sub20k = counts[counts < 20000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub20k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"others\": sub20k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature13(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am going to drop this feature, because an absolutely overwhelming majority of people have never heard of Linux and use Windows. The others are not even worth bucketing"},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The problem here is that there are too many classes with too few entries. This makes tend to zero in the plot, whereas that is not really the case. Either ways, there is a huge majority of entries in 2 different categories. Lets see. I am going to follow a 2 vs all approach here."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature16(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n    \n    sub20k = counts[counts < 20000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub20k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"others\": sub20k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature13(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am thinking of making 2 bucket: sub 20k and 20k-100k. There is again a category imbalance at play here. Sucks."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature16(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('sub20k').fillna('sub20k')\n    \n    sub20k = counts[counts < 20000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub20k' if x in sub20k else x)\n    \n    range20_100k = counts[(counts >= 20000) & (counts < 100000)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range20_100k' if x in range20_100k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"sub20k\",\n                   \"fixers\": [{\"sub20k\": sub20k}, {\"range20_100k\": range20_100k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature13(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts().sort_values(ascending=False).head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Nope. Goodbye"},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(dict(train[feature].value_counts()))\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data is too granular. If I try to bucket, then I will probably end up forming a bew majority class. I really wanted to use this one, but what to do."},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are outliers, which might be good. Anyways, bucket this guy"},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature21(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n    \n    sub1k = counts[counts < 5000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub1k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"others\": sub1k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train1 = fixFeature21(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train1[feature].isna().sum()} null values and {len(train1[feature].unique())} unique categories\")\ntrain1[feature].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Its no good, bucketing is forming a new majority category. Anything short of this bucketing and there will be too many categories to deal with. Will be removing this feature. I am mentally shuddering at what the VIF score will be for this already. I have sooo many features pending still."},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature22(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n    \n    sub10k = counts[counts < 10000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub10k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"others\": sub10k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature22(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\ntrain[feature].value_counts()\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am very sorry ARM. You are awesome."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature23(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].fillna('x86')\n    \n    sub1k = counts[counts < 1000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'x86' if x in sub1k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"x86\",\n                   \"fixers\": [{\"x86\": sub1k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature23(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You are dying Intel, don't get too happy."},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am going to remove this feature, because there are just too many entries into the majority class."},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature25(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n    \n    sub20k = counts[counts < 20000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub20k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"others\": sub20k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature25(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distribution of prediction feature seems to be uniform across categories. It might not be worth it to have this, but I will keep it, regularisation can remove it. Its such a clean feature."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature26(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n    \n    sub25k = counts[counts < 25000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub25k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"others\": sub25k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature26(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Same analysis as the previous feature. This one is more likely to be useless because the majority category has too many entries.\n\nI feel bad about merging the servers with the trashy regular versions, but what to do."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature27(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n    \n    sub25k = counts[counts < 25000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub25k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"others\": sub25k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature27(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again....but this one goes. I can't afford to keep too many such features. While the form factor of the PC gives good indication of the type of OS and its usage, we have those features already. I am not even going to bother to check the correlation(can't also, categorical features)."},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Bye,"},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sorry about the plot, but what the hell."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature30(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('sub2k').fillna('sub2k')\n    \n    sub2k = counts[counts < 2000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub2k' if x in sub2k else x)\n    \n    sub5k = counts[(counts >= 2000) &(counts < 5000)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range_2_5k' if x in sub5k else x)\n    \n    sub10k = counts[(counts >= 5000) &(counts < 10000)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range_5_10k' if x in sub10k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"sub2k\": sub2k}, {\"range_2_5k\": sub5k}, {\"range_5_10k\": sub10k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature30(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am really not sure about the above one, but 3 new buckets is a must. Let's see how it performs later maybe."},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I don't feel good about bucketing here, because the class distribution across categories isn't nice. However, if I don't bucket, I remove this feature. So let L1 regularisation decide that."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature31(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n    \n    sub25k = counts[counts < 25000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub25k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"others\": sub25k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature31(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature32(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('sub2k').fillna('sub2k')\n    \n    sub5k = counts[counts < 5000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub5k' if x in sub5k else x)\n    \n    sub10k = counts[(counts >= 5000) &(counts < 10000)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range_5_10k' if x in sub10k else x)\n    \n    sub20k = counts[(counts >= 10000) &(counts < 20000)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range_10_20k' if x in sub20k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"sub5k\": sub5k}, {\"range_5_10k\": sub10k}, {\"range_10_20k\": sub20k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature32(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No dice honestly.... I don't want to keep it at all. But the name sounds important because MS tends to mess up in build revisions too. The model can decide maybe? I am not letting it have any more categories, nope.\n\nI think 14 more *categorical* features. I am not even going to touch the other types. Remind me to make a DNN to take care of this later on."},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Gonna do 3-vs-all. Seems like an unhelpful feature."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature33(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n    \n    sub25k = counts[counts < 25000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub25k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"others\": sub25k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature33(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You know what? It is redundant with feature 33. Bye-bye"},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Yeah okay, just combine the two \"Clean\" installs"},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature35(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].fillna('Other')\n    \n    sub5k = counts[counts < 5000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'Clean' if x in sub5k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"Other\",\n                   \"fixers\": [{\"Clean\": sub5k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature35(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature36(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('sub_10k').fillna('sub_10k')\n    \n    sub10k = counts[counts < 10000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'sub_10k' if x in sub10k else x)\n    \n    sub20k = counts[(counts >= 10000) & (counts < 20000)].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'range_10_20k' if x in sub20k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"sub_10k\": sub10k}, {\"range_10_20k\": sub20k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature36(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You know what, I am going to let this be. Just convert NA's to UNKNOWN"},{"metadata":{"trusted":true},"cell_type":"code","source":"mapping = {\n            feature: \n               {\"na\": \"UNKNOWN\",\n               \"fixers\": []\n               }\n          }\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am going to convert UNKNOWN into Offline"},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature38(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].fillna('OFFLINE')\n    \n    sub5k = counts[counts < 5000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'OFFLINE' if x in sub5k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"OFFLINE\",\n                   \"fixers\": [{\"OFFLINE\": sub5k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature38(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fixFeature39(dataset):\n    counts = train[feature].value_counts()\n    dataset[feature] = dataset[feature].cat.add_categories('others').fillna('others')\n    \n    sub30k = counts[counts < 30000].index\n    dataset[feature] = dataset[feature].apply(lambda x: 'others' if x in sub30k else x)\n    \n    mapping = {\n                feature: \n                   {\"na\": \"others\",\n                   \"fixers\": [{\"others\": sub30k}]\n                   }\n              }\n\n    return mapping, dataset\n    \nmapping, train = fixFeature39(train.copy(deep=True))\nprint(f\"Feature: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\nprint(train[feature].value_counts())\n\nCategoricalFixers.update(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i += 1\nfeature = cat_cols[i]\nprint(f\"Feature {i}: {feature} has {train[feature].isna().sum()} null values and {len(train[feature].unique())} unique categories\")\npprint(train[feature].value_counts())\n\nplotClassDistribution(feature, False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DropCatCols.append(feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.drop(labels=DropCatCols, axis=1)\ntrain = train.reset_index()\n\nDropColList = DropColList + DropCatCols\nprint(len(DropColList))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train.select_dtypes(include='object').columns] = train.select_dtypes(include='object').astype('category')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DropColList","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CategoricalFixers","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now I will go replace my burnt brain with a 8085; then continue to develop the code to apply the above mapping to train and test sets."},{"metadata":{},"cell_type":"markdown","source":"Save the convertion code and test whether the number of features are okay."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pickle\n\nwith open('convertor.sav', 'wb') as fp:\n    pickle.dump(CategoricalFixers, fp, protocol=pickle.HIGHEST_PROTOCOL)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"assert(sorted(CategoricalFixers.keys()) == sorted(train.select_dtypes(include='category').columns.values))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}