{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"e1157b68-9f70-4444-8e7e-c40f8b0d9703"},"source":"# Kaggle - Facebook recruiting"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"68d71158-4d9b-4e90-aa8f-2d8ebe58c963"},"outputs":[],"source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib.pyplot as plt\nfrom scipy.stats import gaussian_kde\nimport time\nimport seaborn as sns \n%matplotlib inline"},{"cell_type":"markdown","metadata":{"_cell_guid":"e529adba-e97b-4401-b33a-bd700b8541bd"},"source":"### Part 1 - Loading data\nThis will include sampling the data to 1M rows in case we want to do anything computationally tricky."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"a56cd44a-12a9-4879-ab54-b1f0968436b6"},"outputs":[],"source":"df_train = pd.read_csv(\"../input/train.csv\")\ndf_test = pd.read_csv(\"../input/test.csv\")\ndf_train.head()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"da8e8de4-f831-45e0-ba4b-004344a63827"},"outputs":[],"source":"# Sample them for quicker visualisations\ndf_train_sample = df_train.sample(n=1000000)\ndf_test_sample = df_test.sample(n=1000000)"},{"cell_type":"markdown","metadata":{"_cell_guid":"5ff2bb4b-38bb-4d74-b74c-746cdaf3c795"},"source":"### Part 2 - Quick visualisations\nLet's start with some basic histograms, showing the distribution of accuracy and time."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"de568c36-93fd-4d8f-a93c-8b0a36edfa34"},"outputs":[],"source":"counts1, bins1 = np.histogram(df_train[\"accuracy\"], bins=50)\nbinsc1 = bins1[:-1] + np.diff(bins1)/2.\n\ncounts2, bins2 = np.histogram(df_test[\"accuracy\"], bins=50)\nbinsc2 = bins2[:-1] + np.diff(bins2)/2.\n\nplt.figure(0, figsize=(14,4))\n\nplt.subplot(121)\nplt.bar(binsc1, counts1/(counts1.sum()*1.0), width=np.diff(bins1)[0])\nplt.grid(True)\nplt.xlabel(\"Accuracy\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Train\")\n\nplt.subplot(122)\nplt.bar(binsc2, counts2/(counts2.sum()*1.0), width=np.diff(bins2)[0])\nplt.grid(True)\nplt.xlabel(\"Accuracy\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Test\")\n\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"243766f4-4d76-4013-897b-424199bf65da"},"source":"Accuracy has some interesting structure, but is relatively consistent across train/test.\n\nCheck time distributions:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"19bbf055-1dd6-4371-84b0-d1c2f075f929"},"outputs":[],"source":"current_palette = sns.color_palette()\n\ncounts1, bins1 = np.histogram(df_train[\"time\"], bins=50)\nbinsc1 = bins1[:-1] + np.diff(bins1)/2.\n\ncounts2, bins2 = np.histogram(df_test[\"time\"], bins=50)\nbinsc2 = bins2[:-1] + np.diff(bins2)/2.\n\nplt.figure(1, figsize=(12,3))\n\nplt.subplot(121)\nplt.bar(binsc1, counts1/(counts1.sum()*1.0), width=np.diff(bins1)[0], color=current_palette[0])\nplt.grid(True)\nplt.xlabel(\"Time\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Train\")\n\nplt.subplot(122)\nplt.bar(binsc2, counts2/(counts2.sum()*1.0), width=np.diff(bins2)[0], color=current_palette[1])\nplt.grid(True)\nplt.xlabel(\"Time\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Test\")\n\nplt.show()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"badec1b6-dc34-4916-9b69-10ee29561e3f"},"outputs":[],"source":"plt.figure(2, figsize=(12,3))\nplt.bar(binsc1, counts1/(counts1.sum()*1.0), width=np.diff(bins1)[0], color=current_palette[0], label=\"Train\")\nplt.bar(binsc2, counts2/(counts2.sum()*1.0), width=np.diff(bins2)[0], color=current_palette[1], label=\"Test\")\nplt.grid(True)\nplt.xlabel(\"Time\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Test\")\nplt.legend()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"4c2a63b4-100f-4e93-b219-ee19374a7dfa"},"source":"The two dips of time in training set are curious, if looking at counts per unit time they might need to be normalised.\n\nAnother thing we can look at is how frequently different locations appear."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"025f10cd-aca8-429f-8002-e885f443a95b"},"outputs":[],"source":"# Check how how frequently different locations appear\ndf_placecounts = df_train[\"place_id\"].value_counts()\n\ncounts, bins = np.histogram(df_placecounts.values, bins=50)\nbinsc = bins[:-1] + np.diff(bins)/2.\n\nplt.figure(3, figsize=(12,6))\nplt.bar(binsc, counts/(counts.sum()*1.0), width=np.diff(bins)[0])\nplt.grid(True)\nplt.xlabel(\"Number of place occurances\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Train\")\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"297732a1-cf03-46ff-ab5c-3d11dbbad5e3"},"source":"OK, so most places appear around 100 times.\n\nLet's see if accuracy changes with \"time\" at all:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"e71f0cbd-d09b-4579-a95a-e475dfaf97d6"},"outputs":[],"source":"# Check if accuracy of signal corresponds with time\nplt.figure(4, figsize=(12,10))\n\nplt.subplot(211)\nplt.scatter(df_train_sample[\"time\"], df_train_sample[\"accuracy\"], s=1, c='k', lw=0, alpha=0.1)\nplt.xlabel(\"Time\")\nplt.ylabel(\"Accuracy\")\nplt.xlim(df_train_sample[\"time\"].min(), df_train_sample[\"time\"].max())\nplt.ylim(df_train_sample[\"accuracy\"].min(), df_train_sample[\"accuracy\"].max())\nplt.title(\"Train\")\n\nplt.subplot(212)\nplt.scatter(df_test_sample[\"time\"], df_test_sample[\"accuracy\"], s=1, c='k', lw=0, alpha=0.1)\nplt.xlabel(\"Time\")\nplt.ylabel(\"Accuracy\")\nplt.xlim(df_test_sample[\"time\"].min(), df_test_sample[\"time\"].max())\nplt.ylim(df_test_sample[\"accuracy\"].min(), df_test_sample[\"accuracy\"].max())\nplt.title(\"Test\")\n\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"7e169489-a592-423a-b9fa-d1715a8b2d50"},"source":"Not really - but we can see the two time dips in the training plot, and this emphases that accuracy is somewhat perfentially banded.\n\nWhat about if the accuracy varies with location?"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"05e2e8be-2c79-4909-b39a-8ec1f6777703"},"outputs":[],"source":"# Does the accuracy vary with location?  Check within 100x100m spots\ndf_train_sample[\"xround\"] = df_train_sample[\"x\"].round(decimals=1)\ndf_train_sample[\"yround\"] = df_train_sample[\"y\"].round(decimals=1)\ndf_groupxy = df_train_sample.groupby([\"xround\", \"yround\"]).agg({\"accuracy\":[np.mean, np.std]})\ndf_groupxy.head()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"bddc3898-4d2d-43b3-acd1-19f7f8af2b78"},"outputs":[],"source":"idx = np.asarray(list(df_groupxy.index.values))\nplt.figure(5, figsize=(14,6))\n\nplt.subplot(121)\nplt.scatter(idx[:,0], idx[:,1], s=20, c=df_groupxy[\"accuracy\", \"mean\"], marker='s', lw=0, cmap=plt.cm.viridis)\nplt.colorbar().set_label(\"Mean accuracy\")\nplt.grid(True)\nplt.xlabel(\"X\")\nplt.ylabel(\"Y\")\nplt.xlim(0,10)\nplt.ylim(0,10)\n\nplt.subplot(122)\nplt.scatter(idx[:,0], idx[:,1], s=20, c=df_groupxy[\"accuracy\", \"std\"], marker='s', lw=0, cmap=plt.cm.viridis)\nplt.colorbar().set_label(\"Std accuracy\")\nplt.grid(True)\nplt.xlabel(\"X\")\nplt.ylabel(\"Y\")\nplt.xlim(0,10)\nplt.ylim(0,10)\n\nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"9df1c66e-152b-443c-87c4-a9b37cb51877"},"source":"No major structure here.\n### Part 3 - Exploring places and times\n\nFor the next parts, I've created a list of the top places (by check in counts), and chosen the top 20 to investigate further."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"a2436028-77f6-4f38-950e-b5caa7516025"},"outputs":[],"source":"# Get a list of the top 20 places for future reference\ndf_topplaces = df_placecounts.iloc[0:20]\nl_topplaces = list(df_topplaces.index)\nprint(l_topplaces)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"934fcf67-1a9a-4959-801a-1f2966627a30"},"outputs":[],"source":"# Check if any of the top places have time correlated visits\nplt.figure(6, figsize=(14,10))\nfor i in range(len(l_topplaces)):\n    place = l_topplaces[i]\n\n    df_place = df_train[df_train[\"place_id\"]==place]\n\n    counts, bins = np.histogram(df_place[\"time\"], bins=50, range=[df_train[\"time\"].min(), df_train[\"time\"].max()])\n    binsc = bins[:-1] + np.diff(bins)/2.\n    \n    plt.subplot(5,4,i+1)\n    plt.bar(binsc, counts/(counts.sum()*1.0), width=np.diff(bins)[0])\n    plt.xlim(df_train[\"time\"].min(), df_train[\"time\"].max())\n    plt.grid(True)\n    plt.xlabel(\"Time\")\n    plt.ylabel(\"Fraction\")\n    plt.gca().get_xaxis().set_ticks([])\n    plt.title(\"pid: \" + str(place))\n    \nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"8bcf3af8-c9ff-41f3-b58b-0b53ce5777cc"},"source":"Well, some places are visited at certain time periods for sure, but can't do much else until we disentangle time.\n\nThe time interval in train goes from 1 - 800,000, presumably we can modulo these times to view cyclic nature (hours of days, days of week, etc).\n\nThe best guess is seconds or minutes, which would equate to the training spanning 9 or 555 days respectively."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"53ec1d79-92dd-4db4-af52-c28a83d098cc"},"outputs":[],"source":"# Try to infer time\nplt.figure(7, figsize=(14,10))\nfor i in range(len(l_topplaces)):\n    place = l_topplaces[i]\n\n    df_place = df_train[df_train[\"place_id\"]==place]\n\n    # Try % 3600*24 to see daily trend assuming it's in seconds\n    # Try %   60*24 if minutes\n    counts, bins = np.histogram(df_place[\"time\"]%(60*24), bins=50)\n    binsc = bins[:-1] + np.diff(bins)/2.\n    \n    plt.subplot(5,4,i+1)\n    plt.bar(binsc, counts/(counts.sum()*1.0), width=np.diff(bins)[0])\n    plt.grid(True)\n    plt.xlabel(\"Time\")\n    plt.ylabel(\"Fraction\")\n    plt.gca().get_xaxis().set_ticks([])\n    plt.title(\"pid: \" + str(place))\n    \nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"0ef8fd6a-7c20-40a6-a848-3821369bcda4"},"source":"Minutes looks pretty promising. This means we have ~555 days in train and ~140 in test.\n\nFrom this, we can look at day of week to identify trends (weekends), day (to find longer term seasonality).\n\nThe next step is to add some columns representing our new time. Getting this exactly right (within the minute, so that \"hours\" are defined by clock hours, which probably correlate better with place visits) will probably be crucial later on, but as a first it doesn't matter if we're out by a bit."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"fec5c8e9-7842-4aff-a3df-7e2984b8d225"},"outputs":[],"source":"# Add some columns to make calculations easier\ndf_train[\"hour\"] = (df_train[\"time\"]%(60*24))/60.\ndf_train[\"dayofweek\"] = np.ceil((df_train[\"time\"]%(60*24*7))/(60.*24))\ndf_train[\"dayofyear\"] = np.ceil((df_train[\"time\"]%(60*24*365))/(60.*24))\ndf_train.head()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"79e649bb-b21c-4e3d-bf8d-7bcaa18e5a03"},"outputs":[],"source":"df_train_sample[\"hour\"] = (df_train_sample[\"time\"]%(60*24))/60.\ndf_train_sample[\"dayofweek\"] = np.ceil((df_train_sample[\"time\"]%(60*24*7))/(60.*24))\ndf_train_sample[\"dayofyear\"] = np.ceil((df_train_sample[\"time\"]%(60*24*365))/(60.*24))"},{"cell_type":"markdown","metadata":{"_cell_guid":"0ac26dd5-f472-43a2-939b-67da1399edbc"},"source":"Look at the aggregate number of visits per weekday for the top 20 locations, this should show weekends, hopefully."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"196724da-0393-42e1-a196-edb561067555"},"outputs":[],"source":"# Check the top 20 locations again for any weekly trends\nplt.figure(8, figsize=(14,10))\nfor i in range(20):\n    place = l_topplaces[i]\n    df_place = df_train[df_train[\"place_id\"]==place]\n\n    # Group by weekday\n    df_groupday = df_place.groupby(\"dayofweek\").agg(\"count\")\n\n    plt.subplot(5,4,i+1)\n    plt.bar(df_groupday.index.values-0.5, df_groupday[\"time\"], width=1)\n    plt.grid(True)\n    plt.xlabel(\"Day\")\n    plt.ylabel(\"Count\")\n    plt.title(\"pid: \" + str(place))\n    \nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"22863645-afdf-436d-ab93-ec9c40ff7730"},"source":"Some appear to have weekend-like behvaiour... what about looking at day of year."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"946229b6-3e71-4101-b107-a008229bb945"},"outputs":[],"source":"plt.figure(9, figsize=(14,10))\nfor i in range(20):\n    place = l_topplaces[i]\n    df_place = df_train[df_train[\"place_id\"]==place]\n\n    # Add some colums\n    df_place = df_place[df_place[\"time\"]<(60*24*365)] # Restrict to 1 year so the counts don't double up\n    df_groupday = df_place.groupby(\"dayofyear\").agg(\"count\")\n\n    plt.subplot(5,4,i+1)\n    plt.bar(df_groupday.index.values-0.5, df_groupday[\"time\"], width=1)\n    plt.grid(True)\n    plt.xlabel(\"Day of year\")\n    plt.ylabel(\"Count\")\n    plt.xlim(0,365)\n    plt.title(\"pid: \" + str(place))\n    \nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"24c1ec33-91cf-4998-8047-151ca552ac05"},"source":"Somewhat weird trends, looks like some of the top places only opened up business part way through.\n\nThe next interesting thing is to look at the distribution of (x,y) points for a given location:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"563bbb91-a365-4d0f-97e3-c9771cf90439"},"outputs":[],"source":"# Check the 2d distribution of (x,y) for the top 20 places\nplt.figure(10, figsize=(14,16))\ncmapm = plt.cm.viridis\ncmapm.set_bad(\"0.5\",1.)\n\nfor i in range(len(l_topplaces)):\n    place = l_topplaces[i]\n    df_place = df_train[df_train[\"place_id\"]==place]\n    counts, binsX, binsY = np.histogram2d(df_place[\"x\"], df_place[\"y\"], bins=100)\n    extent = [binsX.min(),binsX.max(),binsY.min(),binsY.max()]\n\n    plt.subplot(5,4,i+1)\n    plt.imshow(np.log10(counts.T),\n               interpolation='none',\n               origin='lower',\n               extent=extent,\n               aspect=\"auto\",\n               cmap=cmapm)\n    plt.grid(True, c='0.6', lw=0.5)\n    plt.xlabel(\"X\")\n    plt.ylabel(\"Y\")\n    plt.title(\"pid: \" + str(place))\n\nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"67ed9b74-c125-48b3-bce3-0d3fa6a79c5b"},"source":"\nThe distributions are different for different locations, but many span a huge x-range relative to the y-range (maybe roads are aligned this way?)\n\nLet's re-visit the accuracy to see if it changes we get further away from the presumed centroid location:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"16e58c03-b8fe-443a-95f3-23e8d14f4753"},"outputs":[],"source":"# See if the accuracy varies with distance from centroid point\nplt.figure(11, figsize=(14,16))\n\nfor i in range(len(l_topplaces)):\n    plt.subplot(5,4,i+1)\n    plt.gca().set_axis_bgcolor(\"0.5\")\n    place = l_topplaces[i]\n    df_place = df_train[df_train[\"place_id\"]==place]\n    plt.scatter(df_place[\"x\"], df_place[\"y\"], s=10, c=df_place[\"accuracy\"], lw=0, cmap=plt.cm.viridis)\n    plt.grid(True, c='0.6', lw=0.5)\n    plt.xlabel(\"X\")\n    plt.ylabel(\"Y\")\n    plt.title(\"pid: \" + str(place))\n    \nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"1e88996d-4e00-4e4c-af7d-3fd5ba047b08"},"source":"Nope, not really. What about time variance per location?"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c1585b62-fe16-4fa7-8235-2474f9e2d46f"},"outputs":[],"source":"# See if the time varies with distance from centroid point\nplt.figure(12, figsize=(14,16))\n\nfor i in range(len(l_topplaces)):\n    plt.subplot(5,4,i+1)\n    plt.gca().set_axis_bgcolor(\"0.5\")\n    place = l_topplaces[i]\n    df_place = df_train[df_train[\"place_id\"]==place]\n    plt.scatter(df_place[\"x\"], df_place[\"y\"], s=10, c=df_place[\"hour\"], lw=0, cmap=plt.cm.viridis)\n    plt.grid(True, c='0.6', lw=0.5)\n    plt.xlabel(\"X\")\n    plt.ylabel(\"Y\")\n    plt.title(\"pid: \" + str(place))\n    \nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"8993400d-7056-48ff-95e0-34db9e38119b"},"source":"This certainly shows different places are preferntially visited at different hours. This will be useful for predictions, since for a given \"hour\" the list of probably places will be reduced.\n\nLet's pick an arbitrary place, and see if it's shape is discernible over the background noise."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"e08490df-d66b-41ae-82db-8bf31ca62903"},"outputs":[],"source":"# Pick a place, and see if it's shape profile stands out against background noise (i.e., every other point)\ni = 11\nplace = l_topplaces[i]\ndf_place = df_train[df_train[\"place_id\"]==place]\nxmin, xmax = df_place[\"x\"].min(), df_place[\"x\"].max()\nymin, ymax = df_place[\"y\"].min(), df_place[\"y\"].max()\ndf_noise = df_train[(df_train[\"x\"]>xmin) &\n                    (df_train[\"x\"]<xmax) &\n                    (df_train[\"y\"]>ymin) &\n                    (df_train[\"y\"]<ymax)]\n\nplt.figure(13, figsize=(8,4))\nplt.subplot(121)\nplt.gca().set_axis_bgcolor(\"0.5\")\nplt.scatter(df_noise[\"x\"], df_noise[\"y\"], s=10, c='k', lw=0, alpha=0.005)\nplt.xlabel(\"X\")\nplt.ylabel(\"Y\")\nplt.title(\"pid: \" + str(place))\nplt.xlim(xmin,xmax)\nplt.ylim(ymin,ymax)\nplt.grid(True, c='0.6', lw=0.5)\n\nplt.subplot(122)\nplt.gca().set_axis_bgcolor(\"0.5\")\nplt.scatter(df_noise[\"x\"], df_noise[\"y\"], s=10, c='k', lw=0, alpha=0.005)\nplt.scatter(df_place[\"x\"], df_place[\"y\"], s=10, c=current_palette[5], lw=0, alpha=0.5)\nplt.xlabel(\"X\")\nplt.ylabel(\"Y\")\nplt.title(\"pid: \" + str(place))\nplt.xlim(xmin,xmax)\nplt.ylim(ymin,ymax)\nplt.grid(True, c='0.6', lw=0.5)\n\nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"f6ee0bc3-4046-4db7-8717-aaa65830bb05"},"source":"...not really. OK, let's revisit this x-axis stretching business by visualising these top 20 locations on a map."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"b5e25f56-8fc3-4945-86f8-de95a0af8c56"},"outputs":[],"source":"# Go back to the x-axis stretching, and visualise some location checkins on a map\nplt.figure(14, figsize=(12,12))\n\nfor i in range(20):\n    place = l_topplaces[i]\n    df_place = df_train[df_train[\"place_id\"]==place]\n    plt.scatter(df_place[\"x\"], df_place[\"y\"], s=3, alpha=0.5, c=plt.cm.viridis(int(i*(255/20.))), lw=0)\n    \nplt.grid(True)\nplt.xlabel(\"X\")\nplt.ylabel(\"Y\")\nplt.tight_layout()\nplt.xlim(0,10)\nplt.ylim(0,10)\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"ddac5094-980b-41da-944f-36e470da2dce"},"source":"This highlights the variance, looking suspiciously like streets. We can look at the standard deviations to illustrate this:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8c73f501-d80e-4c29-85f3-54e03405c531"},"outputs":[],"source":"# Check the stdev of x/y for each place\ndf_groupplace = df_train.groupby(\"place_id\").agg({\"time\":\"count\", \"x\":\"std\", \"y\":\"std\"})\ndf_groupplace.sort_values(by=\"time\", inplace=True, ascending=False)\ndf_groupplace.head()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"1ece4664-021e-4a3a-9f94-0a895f2f9f6a"},"outputs":[],"source":"# Density plot\ngkde_stddevx = gaussian_kde(df_groupplace[\"x\"][~df_groupplace[\"x\"].isnull()].values)\ngkde_stddevy = gaussian_kde(df_groupplace[\"y\"][~df_groupplace[\"y\"].isnull()].values)\n\n# Compute the functions\nrangeX = np.linspace(0, 3, 100)\nx_density = gkde_stddevx(rangeX)\ny_density = gkde_stddevy(rangeX)\n\nplt.figure(15, figsize=(12,6))\nplt.subplot(111)\nplt.plot(rangeX, x_density, c=current_palette[0], ls=\"-\", alpha=0.75)\nplt.plot(rangeX, y_density, c=current_palette[1], ls=\"-\", alpha=0.75)\nplt.gca().fill_between(rangeX, 0, x_density, facecolor=current_palette[0], alpha=0.2)\nplt.gca().fill_between(rangeX, 0, y_density, facecolor=current_palette[1], alpha=0.2)\nplt.ylabel(\"Density\")\nplt.xlabel(\"Std dev\")\nplt.plot([], [], c=current_palette[0], alpha=0.2, linewidth=10, label=\"stddev x\")\nplt.plot([], [], c=current_palette[1], alpha=0.2, linewidth=10, label=\"stddev y\")\nplt.legend()\nplt.grid(True)"},{"cell_type":"markdown","metadata":{"_cell_guid":"87b6b5da-8416-4e6d-85b0-b3d579202208"},"source":"That's pretty striking and will be useful later on."},{"cell_type":"markdown","metadata":{"_cell_guid":"f2b726a2-c128-41ab-8bc1-6bf04d64b30e"},"source":"Back to time, what if the accuracy varies as a function of hour of day?"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"db813bc5-2ba1-4d24-b31a-b4a311572799"},"outputs":[],"source":"# With the new found time features, we can re-check how accuracy varies with it:\nplt.figure(19, figsize=(12,6))\nplt.scatter(df_train_sample[\"hour\"], df_train_sample[\"accuracy\"], s=1, c='k', lw=0, alpha=0.05)\nplt.xlabel(\"Hour\")\nplt.ylabel(\"Accuracy\")\nplt.xlim(df_train_sample[\"hour\"].min(), df_train_sample[\"hour\"].max())\nplt.ylim(df_train_sample[\"accuracy\"].min(), df_train_sample[\"accuracy\"].max())\nplt.title(\"Train\")\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"350a07bd-315f-4070-99a9-ed47a04b8249"},"source":"Pretty consistent...\n\n### Part 4 - Predictions pre-processing\n\nOne method of determining which place a location might check in to is by looking at the density of points previously used to checkin.  You could check each point against each density map, and take the maximum."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"55e85b92-3903-4818-9641-33b7ae4d82c1"},"outputs":[],"source":"# Try some KDEs, if we can define the density where check-ins are likely, maybe we can assign points based on this\n# They will also be time variant\ni = 11\nplace = l_topplaces[i]\ndf_place = df_train[df_train[\"place_id\"]==place]\nxmin, xmax = df_place[\"x\"].min(), df_place[\"x\"].max()\nymin, ymax = df_place[\"y\"].min(), df_place[\"y\"].max()\n\n# Calculate the KDE\nres = 200 # resolution\ngkde_place = gaussian_kde(np.asarray((df_place[\"x\"], df_place[\"y\"])))\nx_flat = np.linspace(xmin, xmax, res)\ny_flat = np.linspace(ymin, ymax, res)\nx, y = np.meshgrid(x_flat,y_flat)\ngrid_coords = np.append(x.reshape(-1,1),y.reshape(-1,1),axis=1)\nz = gkde_place(grid_coords.T)\nz = z.reshape(res,res)\n\n# Plot\nextent = [xmin,xmax,ymin,ymax]\nplt.figure(20, figsize=(12,6))\n\n# KDE only\nplt.subplot(121)\nplt.imshow(z[::-1,:],\n           extent=extent,\n           aspect=\"auto\",\n           cmap=plt.cm.viridis,\n           interpolation=\"bilinear\")\nplt.grid(False)\nplt.xlabel(\"X\")\nplt.ylabel(\"Y\")\nplt.title(\"pid: \" + str(place))\nplt.xlim(xmin,xmax)\nplt.ylim(ymin,ymax)\n\n# Overplot the points\nplt.subplot(122)\nplt.imshow(z[::-1,:],\n           extent=extent,\n           aspect=\"auto\",\n           cmap=plt.cm.viridis,\n           interpolation=\"bilinear\")\nplt.colorbar().set_label(\"density\")\nplt.scatter(df_place[\"x\"], df_place[\"y\"], s=10, c='k', lw=0, alpha=0.5)\nplt.grid(False)\nplt.xlabel(\"X\")\nplt.ylabel(\"Y\")\nplt.title(\"pid: \" + str(place))\nplt.xlim(xmin,xmax)\nplt.ylim(ymin,ymax)\n\nplt.tight_layout()\nplt.show()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3f799d52-a318-4e74-89fb-62ec1bb5c70a"},"outputs":[],"source":"# Try some more\npids = [0,8,9,10,11,14] # A few places\nkdes = []\nplt.figure(21, figsize=(14,5))\nfor i in range(len(pids)):\n    place = l_topplaces[pids[i]]\n    df_place = df_train[df_train[\"place_id\"]==place]\n    xmin, xmax = df_place[\"x\"].min(), df_place[\"x\"].max()\n    ymin, ymax = df_place[\"y\"].min(), df_place[\"y\"].max()\n\n    # Calculate the KDE\n    res = 50 # resolution\n    gkde_place = gaussian_kde(np.asarray((df_place[\"x\"], df_place[\"y\"])))\n    kdes.append(gkde_place) # Keep these KDEs for later\n    x_flat = np.linspace(xmin, xmax, res)\n    y_flat = np.linspace(ymin, ymax, res)\n    x, y = np.meshgrid(x_flat,y_flat)\n    grid_coords = np.append(x.reshape(-1,1),y.reshape(-1,1),axis=1)\n    z = gkde_place(grid_coords.T)\n    z = z.reshape(res,res)\n\n    # Plot\n    extent = [xmin,xmax,ymin,ymax]\n    \n    # KDE only\n    plt.subplot(2,6,i+1)\n    plt.imshow(z[::-1,:],\n               extent=extent,\n               aspect=\"auto\",\n               cmap=plt.cm.viridis,\n               interpolation=\"bilinear\")\n    plt.grid(False)\n    plt.xlabel(\"X\")\n    plt.ylabel(\"Y\")\n    plt.title(\"pid: \" + str(place))\n    plt.xlim(xmin,xmax)\n    plt.ylim(ymin,ymax)\n\n    # Overplot the points\n    plt.subplot(2,6,i+7)\n    plt.imshow(z[::-1,:],\n               extent=extent,\n               aspect=\"auto\",\n               cmap=plt.cm.viridis,\n               interpolation=\"bilinear\")\n    plt.scatter(df_place[\"x\"], df_place[\"y\"], s=5, c='k', lw=0, alpha=0.5)\n    plt.grid(False)\n    plt.xlabel(\"X\")\n    plt.ylabel(\"Y\")\n    plt.title(\"pid: \" + str(place))\n    plt.xlim(xmin,xmax)\n    plt.ylim(ymin,ymax)\n\nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"b3f261d7-cb75-4ab5-8f19-d2cfb4e1c9a7"},"source":"### Next steps?\n\n- Could try using KDEs of each place to get a metric for each (x,y) location indicating how close to the place it is\n- Play with time offset to try and get more precise definition of hours\n- Use time to eliminate certain places from being possible at given hours\n- Find out what accuracy does\n"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}