{"cells":[{"metadata":{"_uuid":"e51a3e4d732473a3de0ecdb4db85de65f3c18f5e"},"cell_type":"markdown","source":"# Converting target values to fractions\n\nFollowing the insights from this awesome kernel \n[https://www.kaggle.com/raddar/towards-de-anonymizing-the-data-some-insights](http://) \n\nI wrote this small kernel which will automatically try to convert all target values from train data into rational fractions with a given precision.\n"},{"metadata":{"trusted":true,"_uuid":"071f8e240ae3ca2faadc34a4e80893333301746e"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom fractions import Fraction\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a3fc6f1b74d89dd04eb31f2850cbae1d01409d52"},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d9dc633e552e0425310258bd71ab5aa388f00f8"},"cell_type":"markdown","source":"As was explained in original above kernel, all target values in train data set were actually transformed using log2 function. \n\nThe original values can be obtained by applying 2^target backtransformation"},{"metadata":{"trusted":true,"_uuid":"363a73f22130b3e695e27fbff9bfa0a40c1c8522"},"cell_type":"code","source":"train.drop(['first_active_month','feature_1','feature_2','feature_3'],axis=1, inplace=True)\ntrain.set_index(['card_id'],inplace=True)\ntrain['target_original'] = 2**train['target']\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"90b1a2065152438f8626ed6ec8ab7b8c99180143"},"cell_type":"markdown","source":"As it turnes out some of the original target values in target_original column are really close to ratios like 2/5, 8/9 etc.\n\nThe function below will calculate the smallest fraction which comes close enough to original target value whith a given numerical precision"},{"metadata":{"trusted":true,"_uuid":"91b6a29c2348610a362b0110999e2c48eb1c6ceb"},"cell_type":"code","source":"def calcRatio(df, column, precision):\n    limit = 10\n    \n    frac = 'fraction'\n    dfrac = 'decimal'\n    diff = 'diff'\n    \n    df[frac] = df[column].apply(lambda x: Fraction(x).limit_denominator(limit))\n    df[dfrac] = df[frac]*1.\n    df[diff] = abs(df[column]-df[dfrac])\n    mask = df[diff] > precision\n    \n    while len(df[mask]) > 0:\n        limit = limit*10\n        df.loc[mask, frac] = df[mask][column].apply(lambda x: Fraction(x).limit_denominator(limit))\n        df[diff] = abs(df[column]-df[frac]*1.)\n        mask = df[diff] > precision \n\n    df[dfrac] = df[frac]*1.\n    df['numerator']=df[frac].apply(lambda x: x.numerator)\n    df['denominator']=df[frac].apply(lambda x: x.denominator)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be5d4d1f262470f69e3b0fc3045525c20795d9d4"},"cell_type":"markdown","source":"Now lets figure out fraction values which were used to calculate original target values."},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"d254f0bf741a63921b24c9b4a738760da044c065"},"cell_type":"code","source":"calcRatio(train, 'target_original', 10**(-6))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a0412530e5e6c291787d5fa8fb588294ea6f182"},"cell_type":"markdown","source":"As you can see below lots of original target values at the lower end can be matched pretty well by fractions with small denominator. I exclude the outliers where ratio = 0 and list some values just to get an impression how many target values can be matched pretty well by fractions with low denominator.\n\nColumn 'decimal' contains decimal value of determined fraction and 'diff' the difference to original target value"},{"metadata":{"_uuid":"f23d1d76313ccc143f503c0c67bd5fc4cac2f0de","trusted":true},"cell_type":"code","source":"train.sort_values(['numerator','denominator'], inplace=True)\ntrain[train.fraction!=0][2400:2450]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2435f756e36f0833e3581cc0c93b0405b009e49","trusted":true},"cell_type":"code","source":"train.sort_values(['denominator'], inplace=True)\ntrain[train.fraction!=0][2400:2450]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"69fc0b02a163e60b169ee76bd243eb1da8396074"},"cell_type":"code","source":"calcRatio(train, 'target_original', 10**(-3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"06555aa0e66ad4e43540fe6f75a5bc39c8f8b46e"},"cell_type":"code","source":"train.sort_values(['card_id'], inplace=True)\ntrain[(train.fraction!=0)&(train['diff']<(10**(-7)))]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5873f37d70cf11ee291a6e357bc4a8a8c782fea"},"cell_type":"markdown","source":"I hope this may lead to more insights and can be used for postprocessing."},{"metadata":{"_uuid":"130b9b70ea6ac711b32789930d3220a06565ab86"},"cell_type":"markdown","source":"There was a hypothesis that some of the denominator/numerator values may match a number of authorized transactions from new/historical data. Lets check it. I'll import these data and join with above fractionized target set"},{"metadata":{"trusted":true,"_uuid":"eae30dfa478d39f3eea7e0e8f3b3cb268ac3544d"},"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"797d55314ef8e0829d3db98e0403175a1cd2fdb7"},"cell_type":"code","source":"usecols = ['authorized_flag','card_id','installments','merchant_id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"20afac2372239222db2ad936280ed282cb1bb7b3"},"cell_type":"code","source":"hist_xfer = pd.read_csv('../input/historical_transactions.csv', usecols=usecols )\nhist_xfer['new']=0\nhist_xfer = reduce_mem_usage(hist_xfer, True)\nnew_xfer = pd.read_csv('../input/new_merchant_transactions.csv', usecols=usecols)\nnew_xfer['new']=1\nnew_xfer = reduce_mem_usage(new_xfer, True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f31e1ce3145f0db35cfb3d6e151ee9377707a830"},"cell_type":"code","source":"xfer = pd.concat([new_xfer,hist_xfer])\ndel new_xfer, hist_xfer\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8e8334bc57c97a7fd4ef2f703c8eec53ae70d035"},"cell_type":"code","source":"xfer.reset_index(drop=True)\nxfer['authorized_flag'] = xfer['authorized_flag'].map({'Y': 1, 'N': 0}).astype(np.int16)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5b68370eb6b2ad9e5bd9fb5db5a19c0d77dd1fd7"},"cell_type":"code","source":"xfer['auth_new']=xfer['authorized_flag']*xfer['new']\nxfer['auth_hist']=abs(xfer['authorized_flag']*(xfer['new']-1))\nxfer.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d8a04adb82bbe8e942660f1109cf8399b53372fa"},"cell_type":"code","source":"aggs= {'installments': ['sum'],\n       'authorized_flag': ['sum'],\n       'merchant_id': ['nunique', 'count'],\n       'auth_new': ['sum'],\n       'auth_hist': ['sum']\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9fa4a0b8a2e027a43e5cf6d7e6460ca8b0081386"},"cell_type":"code","source":"df = xfer.groupby('card_id').agg(aggs)\ndf.columns = df.columns.map('_'.join)\ndf['max'] = df[['installments_sum', 'merchant_id_count']].max(axis=1)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true,"_uuid":"6b895f3e57ca1e595606088f8162e1595c87dea0"},"cell_type":"code","source":"train = train.join(df, on='card_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e215c6ea6551bf84008e460b16800c6a1ec8cc44","scrolled":true},"cell_type":"code","source":"calcRatio(train, 'target_original', 10**(-6))\ntrain.sort_values(['denominator'], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cdabc6e8428e21d07e6d4516839a7f288bcafd74"},"cell_type":"markdown","source":"Lets check if there are some matches for denominator and authorized new transactions"},{"metadata":{"trusted":true,"_uuid":"2e36a39a678b3a96ceac974ad685a5cb80a10ec3"},"cell_type":"code","source":"display_col=['target','target_original','fraction','decimal','numerator','denominator','auth_new_sum','auth_hist_sum']\nmask = (train['denominator'] == train['auth_new_sum']) & (train['diff']<10**(-4)) & (train['fraction']!=0)\ntrain[mask][display_col].head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"be28b4854f0f6805efbcb95f8f9bd1b3c21612d7"},"cell_type":"code","source":"train[mask][display_col].tail(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"81a83055be41b06738bcdae3cccb3dfe5f3a34aa"},"cell_type":"code","source":"print(len(train[mask]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4616af5454079292d30a5a0017cf4efc467cb8e"},"cell_type":"markdown","source":"There are some matches, but mostly for denominator == 1 similar situation for numerator"},{"metadata":{"trusted":true,"_uuid":"45eb58839aac10f344f48a1b899087c2ea46406f"},"cell_type":"code","source":"mask = (train['numerator'] == train['auth_new_sum'])  &  (train['diff']<10**(-4)) &(train['fraction']!=0)\ntrain[mask][display_col].head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33096b870c05aa2fc91fa20cfacb75d17442ee91"},"cell_type":"code","source":"train[mask][display_col].tail(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"912680dacdd173927a5c97bab624b8dd110a34d8"},"cell_type":"code","source":"print(len(train[mask]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8c0a2372d1d2a9b388b8358a80269c0f78842f31"},"cell_type":"markdown","source":"But even if there would have been more matches: a fraction 1/3 could have been calculated by 10/33, 20/60. Looks like these matches of numerator and number of auth transactions happened just by coincidence"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}