{"cells":[
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# The Time\nIn this script I show that data are for 699 days with time step of 1 minute.\nI compare it with my own simulated data.\n## Ranges\nFirst, let see what are the range of the data."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nplt.rcParams['figure.figsize'] = (16,5)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "train = pd.read_csv('../input/train.csv', index_col='row_id')\ntrain.describe()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "- x and y are in the range of [0,10]. \n- accuracy is in the range of [1, 1033].\n- time is in the range of [1, 786239]"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "test = pd.read_csv('../input/test.csv', index_col='row_id')\ntest['place_id'] = -1\n#test.head()\ntest.describe()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "The same thing is for the test set, except:\n- time is in the range of [786242, 1006589]\nThe test and train are time splited."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df = pd.concat([train, test])\nidx_test = (df.place_id == -1)\nprint(df.head())\nprint(df.tail())\ndf.describe()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# FFT\nNow we caclulate number of total checkins in a 60 time step interval. \nThen we calculate its fast fourier transform.\nThe x axe shows the number of events in the whole period. "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "checkins, bins = np.histogram(df.time, bins=range(0,df.time.max()+60,60))\nfft = np.fft.fft(checkins)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plt.xlim(10,3000)\nplt.ylim(10**3,10**8)\nfor x in [100, 200, 300, 400]:\n    plt.axvline(x,color='red', ls='--')\nfor x in [699,699*2, 699*3]:\n    plt.axvline(x,color='green', ls='--')\nfor x in [499, 599, 799, 899]:\n    plt.axvline(x,color='orange', ls='--')\nfor x in [1298]:\n    plt.axvline(x,color='black', ls='--')    \nplt.loglog(np.sqrt(fft * fft.conj()).real)\nplt.xlabel('Number of events')"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "The first peak at the lef (Dashed red line) shows 100 events. \nThe other redlines are its harmonics. \nInterestingly, we do not have any harmonic at 300 and 400.\nIf we have 100 weeks, then we should have 700 days. \nIf we zoom in, \nwe see that there is a peak at 699 and its harmonics at 1398 and 2097 (green lines).\nOrange lines are modulations of the higher frequency event (days) with a lower frequency event (weeks), \nat 499, 599, 799, 899.\n\n## Simulation\nNow lets simulate. We assume a business that is open 8 hours a day, 5 days per week.\nThis business has 1 hour of lunch break. We produce the data for about 100 weeks (699 days)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "rng = pd.date_range('1/3/2013',periods=(100*7-1)*24,freq='H')"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "checkin_sim = pd.DataFrame(index=rng)\ncheckin_sim['open'] = 0\ncheckin_sim['dayofweek'] = rng.dayofweek\ncheckin_sim['month'] = rng.month\ncheckin_sim['day'] = rng.day\ncheckin_sim['hour'] = rng.hour"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "checkin_sim.ix[(checkin_sim.hour>8) & (checkin_sim.hour<17),'open']=1 # working hours\ncheckin_sim.ix[(checkin_sim.hour==12),'open']=0 # lunch break\ncheckin_sim.ix[checkin_sim.dayofweek>4,'open']=0 # weekends\ncheckin_sim.ix[(checkin_sim.month == 8) & (checkin_sim.day<15),'open']=0 # summer holidays. \ncheckin_sim.ix[(checkin_sim.month == 1) & (checkin_sim.day<7),'open']=0 # new years.\ncheckin_sim.ix[(checkin_sim.month == 12) & (checkin_sim.day>24),'open']=0 # x-mass"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "fft = np.fft.fft(checkin_sim.open)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plt.xlim(10,3000)\nplt.ylim(0.01,10**4)\nfor x in [100, 200, 300, 400]:\n    plt.axvline(x,color='red', ls='--')\nfor x in [699,699*2, 699*3]:\n    plt.axvline(x,color='green', ls='--')\nfor x in [499, 599, 799, 899, 1298]:\n    plt.axvline(x,color='orange', ls='--')\nfor x in [1298]:\n    plt.axvline(x,color='black', ls='--')    \nplt.loglog(np.sqrt(fft * fft.conj()).real)\nplt.xlabel('Number of events')"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "It is similar, isn't it?\n\nMy conclusion about time:\n\n1. We have data for 699 days. \n2. The time step is one minute.\n"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df.time.max()/(699*24*60)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Almost there!\n\nHere we discovered the unit of time. \nThis remind me of [Oil drop experiment](https://en.wikipedia.org/wiki/Oil_drop_experiment), \nin which [Robert Andrews Millikan](https://en.wikipedia.org/wiki/Robert_Andrews_Millikan)\ndiscovered the unit charge of electricity, the charge of an electron. \n"
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}