{"cells":[
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "## Detect the unit of variable 'time'\n### Nan Wang\n#### May 27, 2016\n\nFor competition **Facebook V: Predicting Check Ins**  \n[https://www.kaggle.com/c/facebook-v-predicting-check-ins](https://www.kaggle.com/c/facebook-v-predicting-check-ins)\n\nThe activities of merchants usually peaks at certain hours in a given day, or in certain days in a given week. So the density of check-ins over time will show periodicity to some extent. Detection of this periodicity is the key to determine the unit of the variable 'time'.\nThe basic idea is using the autocorrelation to infer the unit of the variable 'time'. It is quick and simple. "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "suppressMessages(library(readr))\nsuppressMessages(library(dplyr))\n\ntrain_set = read_csv('../input/train.csv')\n#dim(train_set)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "train_set = train_set %>% \n    arrange(place_id,time) %>% group_by(place_id) %>%\n    mutate(time_1st = first(time),time_diff=time-time_1st)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Since different merchant may have different peak hours, we calculate the time difference between each time point and the first. By doing this, the peak hours of different merchants are aligned for a better detection of periodicity."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "range(train_set$time)\n\nmax(train_set$time)/3600/24  #if the unit is second\nmax(train_set$time)/60/24    #if the unit is minute\nmax(train_set$time)/24/365   #if the unit is hour"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Now it seems that the possible choices for the unit are 'second' or 'minute'. If the unit is hour, the data spans almost 90 years, which is impossible. Now we can estimate the density of check-ins as:"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "time_den = density(train_set$time_diff,bw = 60,na.rm = T,n=13104)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "n=13104 because"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "max(train_set$time)/60"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "By setting the bandwidth to be 60, our density estimate is either one observation per minute (if the unit is second), or one observation per hour (if the unit is minute)."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plot(time_den)\nplot(time_den,xlim=c(0,1000))"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "There are too many observations with small time differeces, so"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "time_den = density(train_set$time_diff[train_set$time_diff>1e3],bw = 60,na.rm = T,n=13104)\nplot(time_den)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Now we can estimate the autocorrelation given different time lags. If the unit is second, we should observe peaks at multiples of **(60*24)=1440**, (not 3600*24, since we set the bandwidth in the density estimation to be 60)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "acf(time_den$y,lag.max = 60*25)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "which is not the case. If the unit is minute, we should observe peaks at multiples of **24**"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "acf(time_den$y,lag.max = 70)\nabline(v=c(24,48),col='red')"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "which is what we observed.\n\n### Conclusion\nThe unit of the 'time' variable is minute."
 }
],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"}}, "nbformat": 4, "nbformat_minor": 0}