Hello and welcome to another kaggle challenge!

Today's task is a bit different. We can all train some machine learning model now. Also get very high scores. Question is, does a higher score on a metric ensure the quality of a model in real life? What happens when your model makes a bad prediction and you have to suffer for it? 

Consider that you've just been appointed as a data scientist at Bank X. The bank sees a lot of credit card transactions every day, most of them are legitimate but as the rules of nature dictate, some of them are unfortunatley fraudalent. The bank did manage to build an automated pipeline using machine learning to detect fradualent transactions, which has very good performance in terms of metrics but in practice it's resulting in customers getting frustrated and leaving the bank. But ...... the metrics are high, what happened? 

Turns out, if you maximise for hard metrics (accuracy, f1, precision etc.) on a test dataset, it's not a good indicator of how your model performs on real world messy, chaotic data. In your banks case, the model was rejecting and blocking legitimate transactions alongside fraud cases.

The bank now wants you, the talented data scientist to analyse their models and figure out what is wrong with them, and whether it is possible to tweak the model not lose more customers. Because, lost customers means lost money. When fraud happens, you're only losing the fraudalent amount but when a customer leaves, you lose both future transactions and customer trust, which are more expensive. 

We've prepared a starter notebook for you which takes care of most of the cost calculation for the bank. All you have to do is, experiment with different models and thresholds and find an optimal value which results in the maximum cost reduction from this whole process for the bank.