{"cells":[
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "# This R environment comes with all of CRAN preinstalled, as well as many other helpful packages\n# The environment is defined by the kaggle/rstats docker image: https://github.com/kaggle/docker-rstats\n# For example, here's several helpful packages to load in \n\nlibrary(ggplot2) # Data visualization\nlibrary(readr) # CSV file I/O, e.g. the read_csv function\nsuppressPackageStartupMessages(library(dplyr, quietly=TRUE))\nsuppressPackageStartupMessages(library(data.table, quietly=TRUE))\nsuppressPackageStartupMessages(library(bit64, quietly=TRUE))\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nsystem(\"ls ../input\")\ndf_train <- fread(\"../input/train.csv\", integer64 = \"character\", showProgress = FALSE)\n# Any results you write to the current directory are saved as output."
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "The city is a 10 Km^2 by  10km ^2 grid. We first try to  check the units of x and y coordinates.\nThe could be latitide, longitude, meters, feet, or something else."
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Lets check the first few rows of the training data"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "head(df_train)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "x = 0.7941, y=9.0809 does not look like a latitude, longitude combination. Moreover its highly unlikely\nthat the same city will have multiple values of latitude/logitude.\nLets examine the min and max values of x and y"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "min(df_train$x)\nmax(df_train$x)\nmin(df_train$y)\nmax(df_train$y)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "So the minimum values of both x and y are 0, maximum values for both is 10. This fits perfectly in our\n10 x 10 grid. This means both x and y are in kilometers."
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Lets check accuracy. Is it in meters, ft, kilometers or any other unit or just a number?"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Lets check max value of accuracy first."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "max(df_train$accuracy)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Its unlikely that 10 x 10 km-squared city will have max values of accuracy set to 1033 km.\nIt is more likely meters or ft"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "What about the min value of accuracy?"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "min(df_train$accuracy)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "This confirms accuracy is in meters or ft. Android API gives location accuracy in meters. \nWe will use that for the time being. \nLets examine the data set more"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df0 <- df_train %>% group_by(place_id) %>% summarise(x,y,accuracy,time,total=n())  %>% arrange(-total)\ndf0_acc <- df0[df0$accuracy ==1,]\ndf0_acc1 <- df0[df0$accuracy == 1033,]"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "dim(df0)\ndim(df0_acc)\ndim(df0_acc1)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "So there are 220273 rows with accuracy of 1 meter and just 6 rows with large accuracy of 1033m. \nLets examine some more. "
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Lets draw a simple boxplot of x and y coordinates of any 1 place-id"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "head(df0)\n"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "head(unique(df0$place_id),10)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df1 <- df_train %>% group_by(place_id) %>% summarise(x,y,accuracy,time)\ndf2 <- df1[df1$place_id == 8772469670,]\ndf2 <-df2 %>% group_by(time) %>% arrange(time)\n\ndf3 <- df1[df1$place_id == 1623394281,]\ndf3 <-df3 %>% group_by(time) %>% arrange(time)\ndf3$row_id <- c(1:dim(df3)[1])"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "dim(df2)\nhead(df2)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "par(mfrow=c(2,1))\nboxplot(df2$x)\nboxplot(df2$y)\n"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "par(mfrow=c(2,1))\nboxplot(df3$x)\nboxplot(df3$y)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "This shows there are more outliers in x than y. So y measurements are more accurate."
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Lets examine more locations."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df4 <- df1[df1$place_id == 1308450003,]\ndf4 <-df4 %>% group_by(time) %>% arrange(time)\ndf4$row_id <- c(1:dim(df4)[1])\n\ndf5 <- df1[df1$place_id == 4823777529,]\ndf5 <-df4 %>% group_by(time) %>% arrange(time)\ndf5$row_id <- c(1:dim(df5)[1])\n\ndf6 <- df1[df1$place_id == 9586338177,]\ndf6 <-df6 %>% group_by(time) %>% arrange(time)\ndf6$row_id <- c(1:dim(df6)[1])\n\ndf7 <- df1[df1$place_id == 9129780742,]\ndf7 <- df7 %>% group_by(time) %>% arrange(time)\ndf7$row_id <- c(1:dim(df7)[1])"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Lets examine time. The units could be hours, minutes, seconds or something else."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_time <- df_train %>% group_by(time) %>% summarise(total=n()) %>% arrange(-time)\nhead(df_time)\n"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_time <- df_train %>% group_by(time) %>% summarise(total=n()) %>% arrange(-total)\nhead(df_time)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "We see that multiple check-ins happended at the same time. So it is unlikely that time is in seconds.\nIt could be minutes or hours. Lets make a simple plot of row_id and time for few places"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "par(mfrow=c(2,3))\nplot(df2$row_id, df2$time)\nplot(df3$row_id, df3$time)\nplot(df4$row_id, df4$time)\nplot(df5$row_id, df5$time)\nplot(df6$row_id, df6$time)\nplot(df7$row_id, df7$time)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "We see that row_id and time vary almost linearly if we ignore outliers. And it also shows that time has \na start time of 0. \nLets examine min and max time"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "min(df_time$time)\nmax(df_time$time)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "the min time is 1. It could be 1 min or 1 hour after start time.\nthe max time is 786239. \nIf it is hours it means 786239/24 = 32759.96 days or 89.75 years. This is highly unlikely.\nSo the the unit of time is minutes.\n\nHence the first check-in was 1 minute after epoch."
 }
],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"}}, "nbformat": 4, "nbformat_minor": 0}