{"cells":[{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"import os\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom mpl_toolkits.mplot3d import Axes3D\nfrom bokeh.plotting import figure, show, output_notebook\n\n%matplotlib inline\nplt.rcParams['figure.figsize'] = (15.0, 15.0) # set default size of plots\noutput_notebook()"},{"cell_type":"markdown","metadata":{},"source":"## Data Read"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":""},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"train_dir = \"../input\"\ntrain_file = \"train.csv\"\n\nfbcheckin_train_tbl = pd.read_csv(os.path.join(train_dir, train_file))"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"# Few statistics\nfbcheckin_train_stats_df = fbcheckin_train_tbl.describe()\nfbcheckin_train_stats_df"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"num_train = len(fbcheckin_train_tbl)\nprint(\"Train samples: {}\".format(num_train))\nprint(\"Unique places: {}\".format(fbcheckin_train_tbl.place_id.unique().size))\nprint(\"Avg samples per places: {}\".format(num_train/float(fbcheckin_train_tbl.place_id.unique().size)))"},{"cell_type":"markdown","metadata":{},"source":"## Data Visualization"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"# Sort by place_id\nfbcheckin_train_tbl = fbcheckin_train_tbl.sort_values(by=\"place_id\")"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"# Take few samples for the visualization\nsample_fbcheckin_train_tbl = fbcheckin_train_tbl[:10000].copy()"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"ax = sample_fbcheckin_train_tbl.plot(kind='hexbin', x='x', y='y', C='place_id', colormap='RdYlGn')\nax.set_xlabel(\"GPS-X\")\nax.set_ylabel(\"GPS-Y\")\nax.set_title(\"Topology of a few places users checked-in based on their last GPS co-ordinates\")"},{"cell_type":"markdown","metadata":{},"source":"## Some analysis"},{"cell_type":"markdown","metadata":{},"source":"This is interesting. In this plot, each color represents a unique business place. Individual hexagon represents last known GPS co-ordinates of users who checked into these places. As we can see from the statistics above, variance of the GPS-X and GPS-Y over all samples is nearly similar, std=2.857601e+00 for X and std=2.887505e+00 for Y. However, if we look at the distribution of user's GPS co-ordinates associated with place_id, we see that these co-ordinates are more scattered over X than Y. For some places they span the whole X range! We can infer that either the dataset (more precisely what we sampled) consist of places which are located very close to each other or the large variance is due to inaccurate GPS locations; we should check accuracies of all these points."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"ax = sample_fbcheckin_train_tbl.plot(kind='hexbin', x='x', y='y', C='accuracy')\nax.set_xlabel(\"GPS-X\")\nax.set_ylabel(\"GPS-Y\")\nax.set_title(\"Accuracy of the GPS locations\")"},{"cell_type":"markdown","metadata":{},"source":"From this plot, we can observe that many of the locations have low to medium accuracies. Our data should speak the same. Let's validate this. "},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"acc_min, acc_max = fbcheckin_train_tbl[\"accuracy\"].min(), fbcheckin_train_tbl[\"accuracy\"].max()\nprint(\"Locations with accuracy above average: {}%\".format(\n        sum(sample_fbcheckin_train_tbl[\"accuracy\"] > (acc_max-acc_min)/2.0)*100/float(sample_fbcheckin_train_tbl.shape[0])))"},{"cell_type":"markdown","metadata":{},"source":"Only 1.51% of the locations have acuracy above average. Let's take one of the business locations and try to visualize accuracy spread of the user's locations who visited it. "},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"place_id = sample_fbcheckin_train_tbl.place_id.unique()[7]\ndf_place = fbcheckin_train_tbl[fbcheckin_train_tbl[\"place_id\"]==place_id]\n\nfig, ax = plt.subplots()\ncax = plt.scatter(df_place[\"x\"], df_place[\"y\"], c=df_place[\"accuracy\"], s=150.0, cmap=plt.cm.Reds)\ncbar = fig.colorbar(cax, ticks=[df_place[\"accuracy\"].min(), \n                        (df_place[\"accuracy\"].max()+df_place[\"accuracy\"].min())/2, df_place[\"accuracy\"].max()])"},{"cell_type":"markdown","metadata":{},"source":"At first, it might seem that locations are spread out in y. However, if we look carefully we see that the Y axis scale is very small compared to the X axis. We can find the same thing from their statistics: "},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"print(\"X min:{}, max:{}, var:{}\".format(df_place[\"x\"].min(), df_place[\"x\"].max(), df_place[\"x\"].var()))\nprint(\"Y min:{}, max:{}, var:{}\".format(df_place[\"y\"].min(), df_place[\"y\"].max(), df_place[\"y\"].var()))"},{"cell_type":"markdown","metadata":{},"source":"Clearly locations are more spread out in X, and as found earlier, sometimes they even span the whole X range. Based on these findings, we can guess that the people are mostly coming from left or right side of this mini palces map or the roads are planned in such a way.\n\nNow, to find out the actual location of the place the easiest approach would be to take the mean value. To get more precise estimate we should take the weighted average of the locations with weights being the accuracy of the location. This way we give more importance to locations reported with high accuracy that the lower ones. Let's check the difference. "},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"place_id = sample_fbcheckin_train_tbl.place_id.unique()[7]\ndf_place = fbcheckin_train_tbl[fbcheckin_train_tbl[\"place_id\"]==place_id]\n\nx_wt = df_place[\"accuracy\"]*df_place[\"x\"]\nx_wt_mean = x_wt.sum()/float(sum(df_place[\"accuracy\"]))\n\ny_wt = df_place[\"accuracy\"]*df_place[\"y\"]\ny_wt_mean = y_wt.sum()/float(sum(df_place[\"accuracy\"]))\n\nfig, ax = plt.subplots()\ncax = plt.scatter(df_place[\"x\"], df_place[\"y\"], c=df_place[\"accuracy\"], s=150.0, cmap=plt.cm.Reds)\ncbar = fig.colorbar(cax, ticks=[df_place[\"accuracy\"].min(), \n                        (df_place[\"accuracy\"].max()+df_place[\"accuracy\"].min())/2, df_place[\"accuracy\"].max()])\nplt.plot(x_wt_mean, y_wt_mean, \"x\", c=\"red\", markersize=40)\nplt.plot(df_place[\"x\"].mean(), df_place[\"y\"].mean(), \"x\", c=\"green\", markersize=40)"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"# bokeh plot: x, y, accuracy\ncolors = [\n    \"#%02x%02x%02x\" % (int((place % (2**24)) >> 16 & 0x0000FF), \n                           int((place % (2**24)) >> 8 & 0x0000FF), \n                           int((place % (2**24)) & 0x0000FF)) for place in sample_fbcheckin_train_tbl[\"place_id\"]\n]\n\nacc_min, acc_max  = fbcheckin_train_tbl[\"accuracy\"].min(), fbcheckin_train_tbl[\"accuracy\"].max()\ncircle_rad_max = 1.5\n\nradii = [\n    \n     circle_rad_max*acc/(acc_max-acc_min) for acc in sample_fbcheckin_train_tbl[\"accuracy\"]\n]\n\np = figure(title = \"Places sample distribution over (x,y)\")\np.xaxis.axis_label = 'x'\np.yaxis.axis_label = 'y'\n\np.circle(sample_fbcheckin_train_tbl[\"x\"], sample_fbcheckin_train_tbl[\"y\"],\n         radius=radii,fill_color=colors, fill_alpha=0.2, size=10)\n\nshow(p)"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"fig = plt.figure()\nax = fig.add_subplot(111, projection='3d')\nax.scatter(sample_fbcheckin_train_tbl[\"x\"], sample_fbcheckin_train_tbl[\"y\"],\n           sample_fbcheckin_train_tbl[\"accuracy\"], c=sample_fbcheckin_train_tbl[\"place_id\"])\n\nax.set_xlabel('x')\nax.set_ylabel('y')\nax.set_zlabel('Accuracy')\nplt.title(\"Places sample distribution over (x,y,accuracy)\")\nplt.show()"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":0}