{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"Raddar astonished us with his insights on purchase amount and other features in his seminal kernel: https://www.kaggle.com/raddar/towards-de-anonymizing-the-data-some-insights .  I decided to have a further look to see if one could have found it without resorting to optimization procedures and manual tweaking.  Here is what I got.  Hope it will please some. \n\nDisclaimer: this is only providing some methodology tips. It does not reuslt in insights that Raddar did not disclose.  I prefer to be clear on that given we are in the last week period."},{"metadata":{"_uuid":"2a5391fc49ffefea5c54f2d851e14c077a8a53c8"},"cell_type":"markdown","source":"We will focus on purchase amount. Let's load all data we have."},{"metadata":{"trusted":true,"_uuid":"73a491909dd54fcf0dfa3ed5b65956a910992ca3"},"cell_type":"code","source":"historical_transactions = pd.read_csv('../input/historical_transactions.csv', usecols=['purchase_amount'])\nnew_transactions = pd.read_csv('../input/new_merchant_transactions.csv', usecols=['purchase_amount'])\ndata = pd.concat((historical_transactions, new_transactions))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"518c035b0ae3f74c76448cd11ce92044c8a82455"},"cell_type":"markdown","source":"This data has been normalized as Raddar claimed.  Indeed, its mean is close to 0:"},{"metadata":{"trusted":true,"_uuid":"e8d35174a707051f584787b977857173a8958232"},"cell_type":"code","source":"data.purchase_amount.mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c74647ebfc160dad6f526aab87b19ecc388b1245"},"cell_type":"markdown","source":"Purchase amounts should be non negative.  My preferred Brazil expert, namely my team mate Giba, told me there is no minimum on purchase amount on cards in Brazil, hence we can start with a minimum at 0:"},{"metadata":{"trusted":true,"_uuid":"225d9c3fdff5c457ccf61c112e0cecaa5b536e03"},"cell_type":"code","source":"data['new_amount'] = (data.purchase_amount - data.purchase_amount.min())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f8761ffb743450f29ea8c10a7d7fee4d949095c9"},"cell_type":"markdown","source":"Let's look at the different amounts we have in the data."},{"metadata":{"trusted":true,"_uuid":"b5c20d5e93b16aabe019f444829f9d1fdb7f8b11"},"cell_type":"code","source":"s = data.groupby('new_amount').new_amount.first().to_frame().reset_index(drop=True)\ns.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3400641d623a03c44838c6bedb422a5a9b0211ab"},"cell_type":"markdown","source":"Let's now compute the successive differences between these amounts.  These should be multiples of cents if purchase real unit is a monetary unit like the Real."},{"metadata":{"trusted":true,"_uuid":"701149a1dddd3d9db0cfdc7fc057cf017b474e1d"},"cell_type":"code","source":"s['delta'] = s.new_amount.diff(1)\ns.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c09509fdf71c04ef36efb1fafc1cb5b9f24848b"},"cell_type":"markdown","source":"We see that the deltas seem constant.  For how long are they constant?  One way to find out is to look at when delta is higher:"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"00f444e368bedf7779498ad27c8f4336a57e96e8"},"cell_type":"code","source":"s[s.delta > 2e-5].head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92824242c74c3546c9ad150c4ec45bf84b6a6b39"},"cell_type":"markdown","source":"Allright, deltas are small until index 5263.  Let's focus on that part.  We also get rid of the first row as its delta was undefined."},{"metadata":{"trusted":true,"_uuid":"8c51e5463e08dd3f52afc395140effda41aea355"},"cell_type":"code","source":"s = s[1:52623]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb72490f6752d3a3daf56c36dcd8d55a34ca75fe"},"cell_type":"code","source":"s.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b07baf040ac1a161931309e99219b51f3e7f204"},"cell_type":"code","source":"s.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"453f4ed7d7d54018585595d3741e6cdb544e076c"},"cell_type":"code","source":"s.delta.mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"85d392c7851bc75234c5598310f382cdf91877d5"},"cell_type":"markdown","source":"This is the smallest posisble difference between two amounts in our data.  If this is one cent, then we can retrieve the orignal purchase amounts by a simple division:"},{"metadata":{"trusted":true,"_uuid":"e77393f576da09aa445f68758340bb2384c33034"},"cell_type":"code","source":"data['new_amount'] = data.new_amount / (100 * s.delta.mean())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff1bad8d7cd10e2e2c492a03050a699ce92c19db"},"cell_type":"markdown","source":"Let's look at most frequent values."},{"metadata":{"trusted":true,"_uuid":"f321ac37a7d5ccbe8950a502b5044f2231dc0fad"},"cell_type":"code","source":"data.new_amount.value_counts().head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"13434150815f75e0a148edc278da73ac0946ef38"},"cell_type":"markdown","source":"Pretty close to integers, aren't they?"},{"metadata":{"_uuid":"f3904a2143f1361ad1c7f584a724f571344a2cb9"},"cell_type":"markdown","source":"If these numbers represent money, then they should be rounded to 2 decimals."},{"metadata":{"trusted":true,"_uuid":"f27e54f7a2a1b6f62a407f3a6d063cc14d58ba95"},"cell_type":"code","source":"data['two_decimal_amount'] = np.round(data.new_amount, 2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d58e4429ac6ad3b4b3bac89898a10c550626f9e7"},"cell_type":"markdown","source":"How much did we round?"},{"metadata":{"trusted":true,"_uuid":"a90a195274646e0dca14f86e9948550adca672d4"},"cell_type":"code","source":"np.abs(data.two_decimal_amount - data.new_amount).mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a063890e110e4eaef4cfb0ed26c789ae14fb01c"},"cell_type":"markdown","source":"Not much... :)"},{"metadata":{"_uuid":"20717d853d9890f23f7e8e055d64b51be06cae03"},"cell_type":"markdown","source":"How many integers do we get?"},{"metadata":{"trusted":true,"_uuid":"074e8a60bfffba31a4fb3d47eea6d68c5d3de2b6"},"cell_type":"code","source":"(data.two_decimal_amount == np.round(data.two_decimal_amount)).mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0baec2d625aff040bf11258b907d70137651f9ad"},"cell_type":"markdown","source":"Nearly 40%."},{"metadata":{"_uuid":"3dd96061fa4643ea4acab6f8c28f22419ab23116"},"cell_type":"markdown","source":"Is this better than Raddar's kernel?  He computed this for new transactions only.  Let's see what we get."},{"metadata":{"trusted":true,"_uuid":"fed4827c92cbae8181733b1909a30212828b418c"},"cell_type":"code","source":"tmp = data[-new_transactions.shape[0]:]\n(tmp.two_decimal_amount == np.round(tmp.two_decimal_amount)).sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"911a9c719c653c55591582a6237c3ad71a73a089"},"cell_type":"markdown","source":"This is exactly the same number that Raddar had in his kernel."},{"metadata":{"_uuid":"ea28610add19e5955260e139717b5ac3748ca94b"},"cell_type":"markdown","source":"The only added value of mine, if any, is that we derived these in a quite straightforward way.  Hope you enjoyed it."},{"metadata":{"trusted":true,"_uuid":"e42988309a5533769279bb946ff3f09d16eeae89"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}