{"cells":[
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output.\n\nimport matplotlib.pyplot as plt\n\ndf_train = pd.read_csv('../input/train.csv')\ndf_test = pd.read_csv('../input/test.csv')"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# Overview\n## Training data"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "print('# of training items: %d' % df_train.shape[0])\nprint('Columns: %s' % df_train.columns.values)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "print('Range of x: %d -> %d' % (df_train['x'].min(), df_train['x'].max()))\nprint('Range of y: %d -> %d' % (df_train['y'].min(), df_train['y'].max()))\nprint('Range of accuracy: %d -> %d' % (df_train['accuracy'].min(), df_train['accuracy'].max()))\nprint('Range of place_id: %d -> %d' % (df_train['place_id'].min(), df_train['place_id'].max()))\nprint('Range of time: %d -> %d' % (df_train['time'].min(), df_train['time'].max()))"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train.head(3)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "## Testing data"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "print('# of testing items: %d' % df_test.shape[0])\nprint('Columns: %s' % df_test.columns.values)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "print('Range of x: %d -> %d' % (df_test['x'].min(), df_test['x'].max()))\nprint('Range of y: %d -> %d' % (df_test['y'].min(), df_test['y'].max()))\nprint('Range of accuracy: %d -> %d' % (df_test['accuracy'].min(), df_test['accuracy'].max()))\nprint('Range of time: %d -> %d' % (df_test['time'].min(), df_test['time'].max()))"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_test.head(3)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# Analyze place area"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "print('# of places: %d' % df_train['place_id'].unique().size)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "# Create a DataFrame to store box area for each place\ndef create_df_agg(df):\n    gr_place = df.groupby(['place_id'])\n    df_train_agg = gr_place.agg({'x': [np.min, np.max], 'y': [np.min, np.max], 'time': np.ma.count})\n    df_train_agg.insert(2, 'height', df_train_agg['y']['amax'] - df_train_agg['y']['amin'])\n    df_train_agg.insert(5, 'width', df_train_agg['x']['amax'] - df_train_agg['x']['amin'])\n    return df_train_agg"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train_agg = create_df_agg(df_train)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train_agg.head(5)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "* Basically, normal spots like restaurants, shops should span around several hundreds of meters. \n* For big sites like theme parks may span several kms, but hope that not so many.\n* Here, I found that there are some places span several kms. \n* Let's discover the means & variance of width & height of the areas to explore how often big sites appear in data."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train_agg['width'].describe()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train_agg['height'].describe()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "* The mean of width says that there are many big sites which span several kms.\n* Let's discover distributions of check-ins for some specific places"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "### Distribution of check-ins for some specific places (Alpha reflects accuracy)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "def draw_place(df, place_sets):\n    plt.figure(figsize=(12, 6))\n    for place_set in place_sets:\n        plt.subplot(place_set[1])\n        plt.title(place_set[2])\n        check_ins = df[df['place_id'] == place_set[0]]\n        # Use alpha to reflect accuracy\n        rgba_colors = np.zeros((check_ins.shape[0],4))\n        rgba_colors[:, 0] = 1.\n        rgba_colors[:, 3] = 1/check_ins['accuracy']\n        plt.scatter(check_ins['x'], check_ins['y'], color=rgba_colors)\n    plt.show()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "draw_place(df_train, [[1000474694, 221, 'Area=(0.3084, 0.0470), 77 check-ins'], \n                       [1000017288, 222, 'Area=(0.0524, 0.6727), 95 check-ins'],\n                       [1000616752, 223, 'Area=(6.8033, 0.0928), 642 check-ins'],\n                       [1000705331, 224, 'Area=(9.8780, 0.6727), 962 check-ins']\n                      ])"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "* For big sites (3 & 4), most of their accurate check-ins focus on the center of the place. And the around check-ins are not so accurate.\n* So, the wide are of big sites may be due to the noise.\n* Hope that removing some noisy check-ins and may reduce the width of the areas."
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# Analyze accuracy"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train['accuracy'].value_counts().head(5)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "* Remember that `accuracy` ranges from 1 -> 1033"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plt.hist(df_train['accuracy'], bins=100, facecolor='red')\nplt.show()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "* It seems that the more smaller value, the more accurate.\n* Most of check-ins' accuracy range less that 100."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "# Remove noisy check-ins\ndf_train1 = df_train[df_train['accuracy'] <= 100]"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train_agg1 = create_df_agg(df_train1)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train_agg1['width'].describe()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train_agg['height'].describe()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "* Unfortunately, remove noisy check-ins does not reduce the width of areas so much (removing more noisy check-ins may reduce the width but we cannot remove so many).\n* But, if we look at distribution of check-ins, we can specify more compact areas for the places by removing the check-ins sparsely placed around the centers.\n* Even we cannot calculate the area more accurately and the areas are still so wide, what if we use the center point of each place??? Using center point of places may help us to apply KNN algo."
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# 'Time' analysis"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plt.hist(df_train['time'], bins=218)\nplt.title('Histogram of time values')\nplt.xlabel('Time values')\nplt.ylabel('Frequency')\nplt.show()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "* Need to explore more about time."
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}