---
title: "Cracking Accuracy"
author: "Nigel Henry"
date: "June 20, 2016"
---

Many have cracked the code of `time`, by `accuracy` still seems a mystery.
For example, [kennmyers](https://www.kaggle.com/kennmyers/facebook-v-predicting-check-ins/exploring-accuracy/comments) showed us that although there is some relationship between `sd(x)` or `sd(y)` vs. `mean(accuracy)` summarised by place, this is much weaker than expected.

Here are some thoughts building upon some **really good** work by [BeyondBeneath](https://www.kaggle.com/beyondbeneath/facebook-v-predicting-check-ins/testing)

As some have seen, 
1. there are three peaks in accuracy, and
2. accuracy "increases with time."

Let's see that again:

```{r, message=FALSE, warning=FALSE}
library(dplyr)
library(ggplot2)
library(gridExtra)
library(reshape2)
train <- read.csv("../input/train.csv")

week = 60 * 24 * 7
train$week <- floor(train$time / week)
weekstats <- train %>% group_by(week) %>%
  summarize(
    acc_mean = mean(accuracy), checkins = length(row_id)
  )

  acc_byWeek <- ggplot(weekstats, aes(x = week, y = acc_mean)) + geom_line(size=2, col=1) + labs(y="Mean Accuracy")
  acc_dens <- ggplot(train, aes(x = accuracy)) + geom_density()
  grid.arrange(acc_byWeek, acc_dens, ncol=2)
```

But what are these peaks in mean `accuracy` either in time or in the distribution?  Some suggested that they are related.  Well, probably not...

Let's look at how the accuracy by week compares to the number of check in's per week.
We'll normalize each variable to show it on the same axis:
```{r}
weekstats <- weekstats %>% scale %>% as.data.frame %>% melt(id="week")
ggplot(weekstats, aes(x = week, y = value, color=variable)) + geom_line(size=2) + scale_color_discrete(breaks=c("checkins","acc_mean"), labels=c("Number of Checkins", "Mean Accuracy")) + ylab("normalized value")
```

We see that the periods of low checkins, the average accuracy of check-ins actually increases. Hmm... what can be going on?

Some have suggested that accuracy probably has to do with how the device is estimating location.  Specifically - GPS is more likely to work indoors than outdoors, etc.
I think this is very reasonable, and fits with what we're seeing.
**Perhaps this is some kind of college town, and in the summer and winter breaks there are a lot less check ins, and the check ins that do happen are more likely to be outdoors and therefore have higher accuracy.**
This would also mesh with that BeyondBeneath showed that the *distribution* of accuracy is the thing that's changing over time.

But first... Let's just check to make sure `accuracy` does have any real meaning on location at all...
```{r}
# calculate the centroid for each place
places <- train %>% group_by(place_id) %>%
  summarize(
    x_mean = mean(x), y_mean = mean(y), n = length(row_id),
    x_sd = sd(x), y_sd = sd(y)
    )
places <- places %>% sample_n(2000, weight = 1*(n > 40)) %>% arrange(place_id)

# plot each point in the training set, based on difference from its place_id's centroid
train <- train %>% filter(place_id %in% places$place_id) %>%
            mutate(x_diff = x - places$x_mean[match(place_id, places$place_id)],
                y_diff = y - places$y_mean[match(place_id, places$place_id)])
ggplot(train, aes(x=x_diff, y=y_diff)) + geom_point(aes(size=accuracy), pch=1)
```

Once you average together locations, we can see that the higher accuracy check ins are more likely to be closer to the centroid.
Or can you see it?  Let's try again:
```{r}
diffFromCenterX <- ggplot(train, aes(y = x_diff, x = accuracy)) + geom_point() + ylab("difference between x and mean of x (of its corresponding place)")
diffFromCenterX <- ggplot(train, aes(y = x_diff, x = accuracy)) + geom_point() + ylab("difference between y and mean of y (of its corresponding place)")
  grid.arrange(diffFromCenterX, diffFromCenterX, ncol=2)
```

Indeed, this was shown by [Wenbo](https://www.kaggle.com/cwenboshi/facebook-v-predicting-check-ins/exploring-accuracy) and [MarekKowalski](https://www.kaggle.com/marekkowalski/facebook-v-predicting-check-ins/accuracy).


OK, so back to what these spikes in check in accuracy actually mean... BeyondBelief found there were 5 peaks in accuracy, three low ones (5.0, 10.0 and 30.0), one high (65.0), and one very high (165.0), I'm guessing that these peaks are check-ins located by cell tower, wifi scan, cellphone location + wifi scan, GPS, and fixed IP address.  With the variations being the strength to the wifi, GPS, etc.

If this is true, how would we know, and how does this help us anyway?  Well it should be true that certain locations should be more likely located by a certain mode, and therefore vice versa.  We should be able to find groups of locations, e.g. outdoor vs. indoor, that have a similar characteristic distribution of the accuracy check-ins.  Let's look:
```{r}
# create a data frame with each place in its own row, and the distribution of accuracy in the columns

acc_max <- max(train$accuracy)
acc_density <- tapply(train$accuracy, train$place_id,
                      function(dat) { density(dat, n = 100, from=0, to=250, bw=5)$y } )
densities <- matrix(nrow = nrow(acc_density), ncol = 100)
for (place in 1:nrow(acc_density))
  densities[place,] <- acc_density[[place]]

# now find clusters of places based on how the accuracy of checkins at each place is distributed
set.seed(1)
kms <- kmeans(densities, centers = 4)
acc_distributions <- t(kms$centers) %>% as.data.frame() %>%
  mutate(acc_bins = seq(1, 250, length.out = 100)) %>%
  melt(id="acc_bins")
ggplot(acc_distributions, aes(x = acc_bins, y = value, color = variable)) + geom_line() + facet_wrap(~variable) +
  labs(x="accuracy", y="density", color="k-means cluster")
```

We do in fact see groups of places defined by having peaks in accuracy exactly where we would expect them.  Of course, we have "forced" these clusters into being, they are only "real" if each clusters is a significant size and the clustering explains the variation.

```{r}
kms$size
kms$betweenss / kms$totss
```

Not bad.