This section analizes the **data**, providing a **descriptive** review. The principal document is the **train.csv** file that contains the sample to be studied and for which predictions are requested.

```{r results='hide',message=FALSE,warning=FALSE}
#library(ggplot2) # Data visualization
#library(readr) # CSV file I/O, e.g. the read_csv function
library(data.table)
library(dplyr)
library(reshape2)

#Get Data
pathToFiles <- '../input/'
numberOfRowsToRead <- -1
set.seed(55446)
dt.train <- as.data.table(read.table(paste0(pathToFiles, "train.csv"),  header=T, sep=",",nrows=numberOfRowsToRead))
```

### 1. Train.csv Description
+ File type: Text,comma separated, with header.
+ File size: 554Mb Compressed. ~1Gb Uncompressed.
+ Records: ~29 Million.
+ Headers: "row_id","x","y","accuracy","time","place_id"

#### 1.1. Initial Data Set Analyis

Displaying a summary of the file contents:
```{r}
summary(dt.train)
```

A few interesting points from the summary:

+ "x" and "y" have a range between 1 and 10
+ Time starts by 1, suggesting a *t0+n, n>=1, n={1,2,3...}* format. It has not evident cyclic time-based pattern.
+ Accuracy has a range from 1 to ~1,000. 
+ Accuracy mean is pretty different from the median, suggesting nonormality. However, its overall effect on the data sample seems to be lower than 1-2%.

Sample size Reduction:
```{r}
sample.size.proportion <- 0.05*nrow(dt.train)

dt.subset <- dt.train[
  sample(seq(1,nrow(dt.train),1)
         ,sample.size.proportion)
  ,]
 
#Reallocating memory
rm(dt.train)

#And comparing it to the full-size data set. 
summary(dt.subset)
```

+ "x" and "y" combinations shape is square. Will the points around the perimeter have a particular behaviour?
```{r,echo=FALSE}
plot(x=dt.subset$x,y=dt.subset$y,xlab = "X coordinate",ylab = "y coordinate")
```

#### 1.2. Accuracy  

And regarding the distribution of Accuracy:
```{r,echo=FALSE}
#Histogram
hist(dt.subset$accuracy,xlab = "Accuracy", main="Accuracy Distribution")

#Boxplot
bxplt <- boxplot(dt.subset$accuracy,main="Accuracy Values Boxplot")
```
```{r}
numerator <- nrow(dt.subset[
  (
    (dt.subset$accuracy>bxplt$stats[5]) |
    (dt.subset$accuracy<bxplt$stats[1])
  )
  ])


denominator <- nrow(dt.subset[
  (
    (dt.subset$accuracy<=bxplt$stats[5]) &&
      (dt.subset$accuracy>=bxplt$stats[1])
  )
  ])

#Which proportion of data is an outlier? 
numerator/denominator
```
Over 15% of the sample Accuracy are extreme values. An Exponential or Poisson distribution may be behind it. This data could be normalized through logarithmic transformation, reducing the outlier rate up to 3 percent. Whether it will be worth it or not, I would not guess at this moment.

#### 1.2. Relationships

There are three generic obvious relationships:

1. *X* and *Y* coordinates.
2. *Coordinates* and number of operations with a certain *place_id* per *time* unit.
3. *Accuracy* and number of operations per *time* unit.

Other thay may exist:

4. *Accuracy* and *coordinates*
5. *Accuracy* and *place_id*
6. *Time* and operations per *place_id*
7. Extreme *coordinates* and *accuracy* and/or operations per *place_id*

There is no apparent overall seasonability within the data set. The first approach consists on considering each time unit as the reference unit. Wider groupings may be considered later.

Now we summarize the data to group it and explore correlations. Although accuracy and frequencies have different measurement units, they won't be scaled by now in order to offer a view of them closer to their actual values.
```{r,echo=FALSE}
dt.subset$time60 <- round(dt.subset$time/60+1,0)
```

##### 1.2.1. Accuracy Relationships

+ Accuracy effect upon the number of operations per time unit is not linearly significant, except in two specific periods, in which it seems to affect negatively the operations rate by ~10-15%.
+ Accuracy effect on operations rate (number of operations per time unit) is not only generally  almost null, but inverse to its magnitude. The greater the accuracy value, the lower the number of operations per time unit.

Let's study how is *accuracy* related to number of operations and time:
```{r, echo=FALSE}
dt.subset.accuracy.summary <- summarise (group_by(dt.subset,as.factor(time60)),median(as.numeric(accuracy)),length(accuracy))
colnames(dt.subset.accuracy.summary) <- c("time","accuracyAvg","operationsNumber")
dt.subset.accuracy.summary <-dt.subset.accuracy.summary[order(time)]

hist(dt.subset.accuracy.summary$accuracyAvg)

plot(x=dt.subset.accuracy.summary$time,y=dt.subset.accuracy.summary$accuracyAvg)
```
```{r}
median(dt.subset.accuracy.summary$accuracyAvg)
```

In general terms, number of operations and accuracy is not linearly related. That fact changes at certain accuracy levels, that I think, they should be studied separately.
```{r}
#Linear variance explanation.
mod.lm <- lm(accuracyAvg ~ operationsNumber, data = dt.subset.accuracy.summary)
summary(mod.lm)
#Spearman correlation coefficient
cor(dt.subset.accuracy.summary$accuracyAvg,
    dt.subset.accuracy.summary$operationsNumber
    ,method = "spearman")
```

##### 1.2.3. Operations and Accuracy

The number of operations compared to accuracy over time suggests the existence of an accuracy range that seems to provide very stable number of operations, and the existence of other levels that generate an important change in the number of operations. This is the reason why probably outliers should be treated apart:

![Accuracy vs Operations](https://raw.githubusercontent.com/gonzsanc/Master-in-Data-Science/master/Other/Kaggle_imgs/AccuracyOverOperationsLabel.jpg)