{"cells":[{"metadata":{"_uuid":"e51a3e4d732473a3de0ecdb4db85de65f3c18f5e"},"cell_type":"markdown","source":"# Digging deeper into anonymization\n\nIt is clear that some features, and the target itself is somehow obfuscated. Well, guess what - I really like working with these kind of puzzles - lookup BNP Paribas competition in Kaggle :). And it was quite obvious for me that same tricks could be applied here as well (at least at some level). I am going to share what I found in this competition so far."},{"metadata":{"trusted":true,"_uuid":"071f8e240ae3ca2faadc34a4e80893333301746e"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\ntrain = pd.read_csv('../input/train.csv')\nmerchants = pd.read_csv('../input/merchants.csv')\nnew_merchant_transactions = pd.read_csv('../input/new_merchant_transactions.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9483a8b40e8914d6c9fb70dd47eada579892abda"},"cell_type":"markdown","source":"## ~~ purchase_amount ~~"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"For simplicity, I am going to ignore historical_transactions, but all the findings below transfers to this table as well.\n\nThe most interesting feature in transactions table is `purchase_amount`. And it should be natural to expect some integers in it's distribution, and the values to be positive - which is obviously not the case here!"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"29f58de559cf3723e19d92bcb37950d780504a1a"},"cell_type":"code","source":"new_merchant_transactions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ed30c8c3500a436ae1f6cac0715a00b84b3e2973"},"cell_type":"code","source":"new_merchant_transactions['purchase_amount_integer'] = new_merchant_transactions.purchase_amount.apply(lambda x: x == np.round(x))\nprint(new_merchant_transactions.groupby('purchase_amount_integer')['card_id'].count())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"663a27b7e3d99f31bca478c601cda90dfefe43c6"},"cell_type":"markdown","source":"So imagine yourself in the shoes of the person who had the task to anonymize this column. One of the most straightforward ways to do that is apply some kind of demeaning and scaling. And with that assumption in mind it is pretty straightforward to reverse engineer that! So after some optimization and manual checking you can find the mean and scale parameters, so that the values would make much more sense. \n\nThis is the magic formula that I was able to find:"},{"metadata":{"trusted":true,"_uuid":"a33b772df6bcb1b408bdb44d916fe8709a6a5d22"},"cell_type":"code","source":"new_merchant_transactions['purchase_amount_new'] = np.round(new_merchant_transactions['purchase_amount'] / 0.00150265118 + 497.06,8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"27442bf7c01e45a2780326f13a80bffa46747344"},"cell_type":"code","source":"new_merchant_transactions.purchase_amount_new.head(100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"382c35bc2ba7b214ddd4347d699fe591454c3aea"},"cell_type":"markdown","source":"Pretty awesome! it seems we can safely round up to 2 decimal points and get a nice looking `purchase_amount` values!"},{"metadata":{"trusted":true,"_uuid":"482e28bf83cff64c73c99955c756d513a17620b7"},"cell_type":"code","source":"new_merchant_transactions['purchase_amount_new'] = np.round(new_merchant_transactions['purchase_amount'] / 0.00150265118 + 497.06,2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"39f84fa0877690914bbdf75249e197d5cacdad5d"},"cell_type":"markdown","source":"Now check how many integers we got now..."},{"metadata":{"trusted":true,"_uuid":"04860cf68c1d865e003061c8daf056ca69d341f5"},"cell_type":"code","source":"new_merchant_transactions['purchase_amount_integer'] = new_merchant_transactions.purchase_amount_new.apply(lambda x: x == np.round(x))\nprint(new_merchant_transactions.groupby('purchase_amount_integer')['card_id'].count())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f8959092e4c6f5c130ce6550db7103a4299c076"},"cell_type":"markdown","source":"That is roughly 40% - which is too high to be it a coincidence - do you agree?:)\n\nWe can actually investigate it a little bit further to find more meanining to this column:"},{"metadata":{"trusted":true,"_uuid":"723734e0e1ee7b1045763e214dec6eda0956c370"},"cell_type":"code","source":"new_merchant_transactions.groupby('purchase_amount_new')['card_id'].count().reset_index(name='count').sort_values('count',ascending=False).head(100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9190f54d6d183e7fb72f2e1c0763e2f9cbcd0dab"},"cell_type":"markdown","source":"So we both got some nice looking integer values and x.99 ones which you would expect a price to have. So it seems the `purhcase_amount` could actually mean how much money was spent on each transaction! Now this reveals some interesting opportunities to make more meaningful features!"},{"metadata":{"_uuid":"e7a1395a85ba63a271c3d1419334efa9ebb9f354"},"cell_type":"markdown","source":"## ~~ numerical_1 & numerical_2 ~~"},{"metadata":{"trusted":true,"_uuid":"d70b8286a99fb736d505fc0a5071d6d919822cb9"},"cell_type":"code","source":"merchants.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"022b450a497dd0ae53a89f20f6be0b57b1de0b40"},"cell_type":"markdown","source":"Now as we know that `purchase_amount` was anonymized in lazy fashion, you could expect something similar in these features as well - and in fact it turns out to be true:"},{"metadata":{"trusted":true,"_uuid":"0ba2bba68f64f00b845925b0449e0fc02a69086d"},"cell_type":"code","source":"merchants['numerical_1'] = np.round(merchants['numerical_1'] / 0.009914905 + 5.79639, 0)\nmerchants['numerical_2'] = np.round(merchants['numerical_2'] / 0.009914905 + 5.79639, 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c5c1d1858f27de703b1964d715737971f3516cda"},"cell_type":"code","source":"merchants.groupby('numerical_1')['merchant_id'].count().head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c51ea5f8669b1facb19f5623ee2e50bef1771fa3"},"cell_type":"code","source":"merchants.groupby('numerical_2')['merchant_id'].count().head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9c609556e36e77a1fd0fadc99d92c953a3bd443"},"cell_type":"markdown","source":"If you were to investigate further, all values are integers! However, I still strugle to find the meaning of these columns - one of my hypothesis was that it could represent how many \"loyalty\" products were purchased of this merchant in last 3/6/12 months? This could be stong feature on aggregated level, I guess...\n\nOther features (avg sales and such) are still a mystery for me, but maybe they are not of that importance. Morever, as you could notice, so far I have exploited high numeric precision of float numbers - and sales features are rounded up - so it is practically impossible to reverse engineer something here."},{"metadata":{"_uuid":"bfc32e4dbdce69c7652941dbecaf71bc73ee4f95"},"cell_type":"markdown","source":"## ~~ target ~~"},{"metadata":{"_uuid":"1d9dc633e552e0425310258bd71ab5aa388f00f8"},"cell_type":"markdown","source":"Now this is probably the most interesting part! There were some hints in the forums about log10(2) being used in the outlier value of -33.xxxxx; \n\nBut one thing people missed out was that the same log10(2) is used for all the target values as well!"},{"metadata":{"trusted":true,"_uuid":"363a73f22130b3e695e27fbff9bfa0a40c1c8522"},"cell_type":"code","source":"train['target_new'] = 10**(train['target']*np.log10(2))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e82b53f7f310002e19b2278e60d5a69afc631f3a"},"cell_type":"code","source":"train.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c12a61d0e586d9bc58cb2ed19fd8da0a231805d6"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"030fdfb617716babed5aa86df22975c8ebd0530c"},"cell_type":"code","source":"train.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"be92835ff3e279248f3d14ab0eebf2835d43ee17"},"cell_type":"code","source":"import math\ntrain['target_new2'] = np.log10(train['target_new'])/np.log10(2)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c7bc670898b6478ba39b195e45b42510aad41aab"},"cell_type":"code","source":"train.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2df775576fcc45feb1a92fb0a831ba2a24001ada"},"cell_type":"code","source":"np.log10( 0.566331)/np.log10(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d89cdd57ba2cf839a092ef988776e3b750274c9d"},"cell_type":"code","source":"train[(train['target'] < -30)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7df50de9b195a98845d4477cf1eed196f971d1ca"},"cell_type":"code","source":"train['target_new2'] = math.pow(10,(train['target_new2']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bfdc40d009db2b4deb989fe044b04c226963c5b2"},"cell_type":"code","source":"train['target_new'].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"52acc53f9b1e834f52ed331c581cd13003161926"},"cell_type":"markdown","source":"You might say, so what? there are no integers or something like that... \n\nHowever, there are no negative values! AND, if you looked closely, it is actually ratios what we are dealing with!\n\nIf you tried putting some of these values into https://www.wolframalpha.com/ , it does a good job trying to explain how one could get this kind of float value.\n\nThere is a couple I was able to find reasonably fast:"},{"metadata":{"trusted":true,"_uuid":"5b7447f286909a44eac42016a3ece904649f8967"},"cell_type":"code","source":"print(train['target_new'][2], 29/18)\nprint(train['target_new'][29823], 973/300)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d7a6940567a1c0978ce1a72cca66d05c37dad51"},"cell_type":"markdown","source":"There are a lot \"clean\" looking divisions, and it is enough to conclude that the customer loyalty is in fact measured by some kind of ratio. I believe, this ratio represents some kind of aggregate transaction `purchase_amount` ratios in the future. But I have never personally worked with loyalty prediction; I think there are more people who would understand right away what these numbers mean."},{"metadata":{"_uuid":"4c83d096d671b460e46b5d664d3529d0e3d69596"},"cell_type":"markdown","source":"## tl;dr "},{"metadata":{"_uuid":"eeeef8f1fd94416974f0763d0bac070e179a39e5"},"cell_type":"markdown","source":"Anonymous values are demeaned, unscaled or log transformed. However, it is pretty easy to reverse that and try to guess the meaning of anonymous features.\n\nThe target column itself is a ratio of some kind - probably sum of transactions compared to historical... \n\nNeed more input on this one from you guys!"},{"metadata":{"_uuid":"83134de8317eda640447fcc7cb232e4f94cfca9e"},"cell_type":"markdown","source":"## Thank you for reading, happy hunting!"},{"metadata":{"_uuid":"b3c4b642d81ca07fc0349b05f6ff80880f56dd95"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"bd67a0463a476aa4e76ee885653a325bdea79b53"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"f23d1d76313ccc143f503c0c67bd5fc4cac2f0de"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"d2435f756e36f0833e3581cc0c93b0405b009e49"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"e5873f37d70cf11ee291a6e357bc4a8a8c782fea"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}