{"cells":[
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "The popularity of place ids is changing with time. The script below will calculate the popularity of places in the test data. Lets start with the basic time calculations below:"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n### read in train data\ntrain = pd.read_csv('../input/train.csv')\ntrain['hour'] = (train['time']//60)%24+1 # 1 to 24 hours\ntrain['weekday'] = (train['time']//1440)%7+1 # 1 to 7 weekdays\ntrain['month'] = (train['time']//43800)%12+1 # rough estimate, month = 30.4 days \ntrain['year'] = (train['time']//525600)+1 # year 1 and 2\n"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Next we will calculate median for train x, y data for each place id"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "\nplaces = train.groupby('place_id')['x','y'].agg(np.median) # median is better than mean for this data\nplaces.columns = ['xc', 'yc']  # the median values xc, yc are the centers of the places\nplaces['train_count']=0 # count of train data per place id in a small window around xc, yc\nplaces['test_count']=0 # count of test data per place id in a small window around xc, yc\n\nprint(places.head())"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "limit train data to similar timeframe as test data (from year 1). When we did the calculations for xc,yc in the previous step, we used the full data set which gave more resolution for finding xc,yc"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "\n# limit train data to similar timeframe as test data\ntrain = train.query('month>=7 and month<=12 and year==1')\n\n"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "read in the test data"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "\n### read in test data\ntest = pd.read_csv('../input/test.csv')\ntest['hour'] = (test['time']//60)%24+1 # 1 to 24 hours\ntest['weekday'] = (test['time']//1440)%7+1 # 1 to 7 weekdays\ntest['month'] = (test['time']//43800)%12+1 # rough estimate, month = 30.4 days\ntest['year'] = (test['time']//525600)+1 # year 2\n"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Calculate counts within a small window around xc,yc for both train and test data. It takes about 3 hours to go through all place ids. I have shown 5 placeids which have biggest differences between counts "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "\ntr = np.asarray(train[['x','y']])\nte = np.asarray(test[['x','y']])\n\nprint(len(tr))\nprint(len(te))\n\n## ratio for scaling counts based on data sizes. It will be close to 1\ntrain_test_ratio = len(train)/float(len(test))\nprint(train_test_ratio)\n"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "\nwindowsize = 0.01\n#for i,placeid in enumerate(places.index):  # use this to iterate over all placeids. Takes 3 hrs\nfor i,placeid in enumerate([6305130230, 3938632601, 5705309641, 5676118839, 4501921430]):  \n    xmax=places.xc[placeid]+windowsize\n    xmin=places.xc[placeid]-windowsize\n    ymax=places.yc[placeid]+windowsize\n    ymin=places.yc[placeid]-windowsize\n    places = places.set_value(placeid, 'train_count', ( (xmin < tr[:,0]) & (tr[:,0] < xmax) & (ymin < tr[:,1]) & (tr[:,1] < ymax)  ).sum()  )   \n    places = places.set_value(placeid, 'test_count', ( (xmin < te[:,0]) & (te[:,0] < xmax) & (ymin < te[:,1]) & (te[:,1] < ymax)  ).sum() * train_test_ratio  )   \n"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "The count columns can be scaled and used for prediction in the model. Let's plot the results to see if the calculations are correct. The red dot in the plots is the xc,yc center for the place\n"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "placeid = 6305130230\nxmin= places.loc[placeid,'xc']-0.3\nxmax=places.loc[placeid,'xc']+0.3\nymin=places.loc[placeid,'yc']-0.3\nymax=places.loc[placeid,'yc']+0.3\n      \ntrain_subset = train.query('( @xmin<= x <= @xmax) and ( @ymin<=y<= @ymax)')\ntest_subset = test.query('( @xmin<= x <= @xmax) and ( @ymin<=y<= @ymax)')\n    \nfig = plt.figure(1, figsize=(10,4))\n\nax1 = plt.subplot(1,2,1)\nax1.scatter(train_subset['x'], train_subset['y'], s=1, c='g', marker=\"s\", label='first', edgecolors='none')\nax1.scatter(places['xc'][placeid], places['yc'][placeid], s=25, c='r', marker=\"s\", label='first', edgecolors='none')\nax1.set(xlim=(xmin, xmax))\nax1.set(ylim=(ymin, ymax))\nplt.title(\"train: place_id \" + str(placeid)) \nplt.xlabel('x')\nplt.ylabel('y')\n    \nax2 = plt.subplot(1,2,2)\nax2.scatter(test_subset['x'], test_subset['y'], s=1, c='b', marker=\"s\", label='first', edgecolors='none')\nax2.scatter(places['xc'][placeid], places['yc'][placeid], s=25, c='r', marker=\"s\", label='first', edgecolors='none')\nax2.set(xlim=(xmin, xmax))\nax2.set(ylim=(ymin, ymax))\nplt.title(\"test: place_id \" + str(placeid)) \nplt.xlabel('x')\nplt.ylabel('y')\n    \nplt.show()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Let's plot a few more place ids"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "fig, axs = plt.subplots(4,2, figsize=(10, 20))\naxs = axs.ravel()\n\nfor i,placeid in enumerate([3938632601, 5705309641, 5676118839, 4501921430]):     \n    xmin= places.loc[placeid,'xc']-0.3\n    xmax=places.loc[placeid,'xc']+0.3\n    ymin=places.loc[placeid,'yc']-0.3\n    ymax=places.loc[placeid,'yc']+0.3\n     \n    train_subset = train.query('( @xmin<= x <= @xmax) and ( @ymin<=y<= @ymax)')\n    test_subset = test.query('( @xmin<= x <= @xmax) and ( @ymin<=y<= @ymax)')\n    \n    axs[i*2].scatter(train_subset['x'], train_subset['y'], s=1, c='g', marker=\"s\", label='first', edgecolors='none')\n    axs[i*2].scatter(places.loc[placeid,'xc'], places.loc[placeid,'yc'], s=25, c='r', marker=\"s\", label='first', edgecolors='none')\n    axs[i*2].set(xlim=(xmin, xmax))\n    axs[i*2].set(ylim=(ymin, ymax))\n    axs[i*2].set_title(\"train: place_id \" + str(placeid)) \n    axs[i*2].set_xlabel('x')\n    axs[i*2].set_ylabel('y')\n    \n    axs[i*2+1].scatter(test_subset['x'], test_subset['y'], s=1, c='b', marker=\"s\", label='first', edgecolors='none')\n    axs[i*2+1].scatter(places.loc[placeid,'xc'], places.loc[placeid,'yc'], s=25, c='r', marker=\"s\", label='first', edgecolors='none')\n    axs[i*2+1].set(xlim=(xmin, xmax))\n    axs[i*2+1].set(ylim=(ymin, ymax))\n    axs[i*2+1].set_title(\"test: place_id \" + str(placeid)) \n    axs[i*2+1].set_xlabel('x')\n    axs[i*2+1].set_ylabel('y')\n\nplt.show()"
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}