{"cells":[
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# Exploring Accuracy\n\nI didn't see many people experimenting with the accuracy so I was curious if I could find anything from it. My goal was to find some sort of relationship between accuracy and the variation in x and y. The purpose of this was so that I could eventually make a way to artificially populate the data set with 'ghosts' of the same data point in a localized radius (obtained from this relationship between accuracy and distance). Then use either KNN/KDTree or something similar to [ZFTurbo's method](https://www.kaggle.com/zfturbo/facebook-v-predicting-check-ins/mad-scripts-battle/code). "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport scipy\n\n%matplotlib inline"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "train_data = pd.read_csv('../input/train.csv')"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "train_data.describe()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "train_data.head()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "p = plt.hist(acc_df.mean_accuracy, bins=np.arange(min(acc_df.mean_accuracy), max(acc_df.mean_accuracy) + 1, 1))\nplt.xlabel('Mean Accuracy')\nplt.ylabel('Count')\nplt.title('Counts of mean accuracy for places')\n\nplt.show()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "p = plt.hist(acc_df.std_x, bins=np.arange(min(acc_df.std_x), max(acc_df.std_x) + .01, .01))\nplt.xlabel('Std(y)')\nplt.ylabel('Count')\nplt.title('Counts of std(x) for places')\n\nplt.show()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "p = plt.hist(acc_df.std_y, bins=np.arange(min(acc_df.std_y), max(acc_df.std_y) + .001, .001))\nplt.xlabel('Std(y)')\nplt.ylabel('Count')\nplt.title('Counts of std(y) for places')\n\nplt.show()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "p = plt.scatter(acc_df.mean_accuracy,acc_df.std_x)\nplt.xlabel('Mean Accuracy')\nplt.ylabel('Std(x)')\nplt.title('std(x) vs Mean accuracy for each place')\n\nplt.show()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Some outliers in this graph but for the most part this is about what was expected, higher accuracy generally leads to lower deviation in distance and the opposite for lower accuracy. The variation in this data is larger than what would be ideal, which could cause some distance approximations to be very inaccurate but it is a start."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "p = plt.scatter(acc_df.mean_accuracy, acc_df.std_y)\nplt.xlabel('Mean Accuracy')\nplt.ylabel('Std(y)')\nplt.title('std(y) vs mean accuracy for each place')\n\nplt.show()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "This looks better an more promising but that is also to be expected since there is less variation among y data"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "[Reference for below](http://stackoverflow.com/questions/3938042/fitting-exponential-decay-with-no-initial-guessing)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "#Exponential Decay function\ndef func(x, a, b, c):\n    return a*np.exp(-b*(x+c))\n\npopt, pcov = scipy.optimize.curve_fit(func, acc_df.mean_accuracy, acc_df.std_x)\npopt"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "p = plt.scatter(acc_df.mean_accuracy,acc_df.std_x)\nl = plt.plot([i for i in range(0,1000)], [func(i, 1.09251753e+00,   4.33988866e-03,   2.86223906e+01) for i in range(0,1000)], 'r', linewidth=1)\nplt.xlabel('Mean Accuracy')\nplt.ylabel('Std(x)')\nplt.title('std(x) vs Mean accuracy for each place (exp decay)')\n\nplt.show()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "#csch = 1/sinh and \ndef func2(x,a,b,c):\n    return a*1/(np.sinh(b*x)) + c\n\npopt, pcov = scipy.optimize.curve_fit(func2, acc_df.mean_accuracy, acc_df.std_x)\npopt"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "p = plt.scatter(acc_df.mean_accuracy,acc_df.std_x)\nl = plt.plot([i for i in range(0,1000)], [func2(i, 108.75643807,   40.73096679,    0.67921053) for i in range(0,1000)], 'r', linewidth=1)\nplt.xlabel('Mean Accuracy')\nplt.ylabel('Std(x)')\nplt.title('std(x) vs Mean accuracy for each place (csch)')\n\nplt.show()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "#1/x\ndef func4(x,a,b,c):\n    return a*(1/(b*(x+c)))\n\npopt, pcov = scipy.optimize.curve_fit(func4, acc_df.mean_accuracy, acc_df.std_x)\npopt"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "p = plt.scatter(acc_df.mean_accuracy,acc_df.std_x)\nl = plt.plot([i for i in range(0,1000)], [func4(i, 97.59074952,   0.8234309 ,  95.34280786) for i in range(0,1000)], 'r', linewidth=1)\nplt.xlabel('Mean Accuracy')\nplt.ylabel('Std(x)')\nplt.title('std(x) vs Mean accuracy for each place (1/x)')\n\nplt.show()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "This wasn't what I was hoping to find though the predictions would be conservative for the accuracies with larger std(x) which I would prefer rather than overpredicting the distance. Perhaps someone better versed in numpy and/or statistics can come up with a better model."
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "I wanted to make some attempt on the y data:"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "popt, pcov = scipy.optimize.curve_fit(func, acc_df.mean_accuracy, acc_df.std_y)\npopt"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "p = plt.scatter(acc_df.mean_accuracy,acc_df.std_y)\nl = plt.plot([i for i in range(0,1000)], [func(i, -0.01277189,  1.4337799 ,  2.71943575) for i in range(0,1000)], 'r', linewidth=1)\nplt.xlabel('Mean Accuracy')\nplt.ylabel('Std(y)')\nplt.title('std(y) vs Mean accuracy for each place (exp decay)')\n\nplt.show()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "popt, pcov = scipy.optimize.curve_fit(func4, acc_df.mean_accuracy, acc_df.std_y)\npopt"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "p = plt.scatter(acc_df.mean_accuracy,acc_df.std_y)\nl = plt.plot([i for i in range(0,1000)], [func(i, 221.24964964,   12.02131292,  804.60004683) for i in range(0,1000)], 'r', linewidth=1)\nplt.xlabel('Mean Accuracy')\nplt.ylabel('Std(y)')\nplt.title('std(y) vs Mean accuracy for each place (1/x)')\n\nplt.show()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Not what I was hoping for either but I may use some of these equations for now to get started on generating artificial data points. "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": ""
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}