{"cells":[
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# Summary\n\nWe're given the prompt from Facebook: \n> The goal of this competition is to predict which place a person would like to check in to. \nFor the purposes of this competition, Facebook created an artificial world consisting of more than 100,000 places located in a 10 km by 10 km square. \nFor a given set of coordinates, your task is to return a ranked list of the most likely places. \nData was fabricated to resemble location signals coming from mobile devices, giving you a flavor of what it takes to work with real data complicated by inaccurate and noisy values. Inconsistent and erroneous location data can disrupt experience for services like Facebook Check In.\n\nRather than ask, “How do I model this artificial world?” let’s ask a different question, “How were these data generated?”\nTreating this as an analysis problem, rather than a modeling problem,\ncould lead to deeper insights than tuning model hyperparameters.\nPlus, it's a fun challenge. "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": ""
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": ""
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport scipy.stats as stats    \nimport scipy.special as sps\n%matplotlib inline"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "# Read in the data using the first column as the index\ndf = pd.read_csv('../input/train.csv', index_col=0)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df.head()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df.describe()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# Tour of the variables\nLet's pause and see what we're dealing with. \n\n## row_id\nThe primary key for our data. Nothing to see here...\n\n## x, y\nWe're told this is a 10 km x 10 km square and, indeed, x and y run between 0 and 10 so it’s probably fair to assume the units of x and y are kilometers. \nThe precision of x and y is 0.0001 km which is 10 cm (that’s 4 in in Menlo Park).\n\n## accuracy\nThis is interesting. `accuracy` varies between 1-1033 with an average of 82.8. \nIt's given as an integer which is odd. \n\n## time\nAgain an integer. Based on previous analyses, we suspect its units of are minutes. \n\n## place_id\n`place_id` ranges between $10^9$ and $10^{10}$. \nThere are $9\\times10^9$ possible unique values but only $2.9\\times10^7$ were used, 0.3% of the availability. \nIt's a odd choice given $9.99\\times10^{32}/2^{32} = 2.3$ because it just overflows the SQL `INT` data type. "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "\n\n\n\n\n\n\n"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "nb_total = df.place_id.count()\nnb_unique = df.place_id.drop_duplicates().count()\n\nprint('Number place_ids: {}'.format(nb_total))\nprint('Unique place_ids: {}'.format(nb_unique))\nprint(\"Average number of duplicates: %.1f\" % (nb_total/nb_unique))"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "There are an average of 269 check ins per place. "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "from pandas.tools.plotting import scatter_matrix\ndf_sample = df[df.place_id == 4823777529]\nscatter_matrix(df_sample.drop('place_id', axis=1), diagonal='kde', figsize=(11,11))"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "`x` and `y` are look gaussian but `accuracy` and `time` are something more complicated. Again we see how all these variables are uncorrelated and the scatter in `x` is much greater than in `y`."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "shape, scale = 1.34, 199.38 # mean and dispersion\n\ns = df.place_id.value_counts().values\n\n#Display the histogram of the samples, along with the probability density function:\nax = df.place_id.value_counts().plot.kde()\nax.set_xlim(0, 2000)\n\ncount, bins, ignored = plt.hist(s, 100, normed=True)\n#y = bins**(shape-1)*(np.exp(-bins/scale) /\n#                      (sps.gamma(shape)*scale**shape))\n#y = stats.gamma.pdf(bins, a=.6, loc=.999, scale=2.0549)\n#rv = stats.maxwell(loc=-249.6547, scale=336.860199)\nrv = stats.frechet_r(1.1, loc=0.89, scale=280)\ny = rv.pdf(bins)\nax.plot(bins, y, linewidth=2, color='r')"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "You can swap out distributions fairly easily but I haven't found one that fits yet. \nLet me know if you figure it out!"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": ""
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}