# Exploring the CERN LHCb Data

```{r results="asis"}
suppressPackageStartupMessages(library(readr))
train <- read_csv("../input/training.csv")
# setting a random seed for reproducibility
set.seed(1)
# randomly reordering the train set since it's currently ordered
train <- train[sample.int(nrow(train)),]
test  <- read_csv("../input/test.csv")
cat("There are", nrow(train), " rows and", ncol(train), "in the train set.")
cat("There are", nrow(test),  " rows and", ncol(test),  "in the test set.")
```

Let's see what columns are in the train set

```{r}
names(train)
cat("The test data has all the columns the training set has, except for", paste(setdiff(names(train), names(test)), collapse=","))
```

We'll also go ahead and look at the first samples in the training set

```{r results="asis"}
suppressPackageStartupMessages(library(xtable))
head.table <- function(table) {
  html <- print(xtable(head(table, n=10)), type="html", print.results=FALSE)
  cat(paste0("<div style='overflow:auto; border-width: 2'>", html, "</div>"))
}
head.table(train)
```

```{r}
suppressPackageStartupMessages(library(ggvis))
train %>% ggvis(~mass) %>% layer_histograms(fill:="#20beff")
```

It looks like all the columns are numeric. Let's see what variables are correlated with each other. We'll use [Spearman's rank correlation coefficient](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) since we've not normalized the features.

```{r fig.width=10, fig.height=10}
library(corrplot)
corrplot(cor(train, method="spearman"), method="square", order="hclust", hclust.method="complete")
```

Here we, see that we have several groups of highly correlated features. Many of these aren't surprising. For example, the variables relating to track isolation tend to be highly correlated.

Mass is uncorrelated with the signal variable (plus all the other inputs). This may be due to the large peak in the mass distribution, but this may be helpful in having target predictions that are uncorrelated with mass.

```{r fig.width=8, fig.height=10}
suppressPackageStartupMessages(library(randomForest))
rf <- randomForest(train[,2:47], factor(train$signal), ntree=100, imp=TRUE, sampsize=10000)
imp <- importance(rf, type=1)
featureImportance <- data.frame(Feature=row.names(imp), Importance=imp[,1])

suppressPackageStartupMessages(library(ggplot2))
ggplot(featureImportance, aes(x=reorder(Feature, Importance), y=Importance)) +
  geom_bar(stat="identity", fill="#53cfff") +
  coord_flip() + 
  theme_light(base_size=20) +
  xlab("Importance") +
  ylab("") + 
  ggtitle("Random Forest Feature Importance\n") +
  theme(plot.title=element_text(size=18))
```

Let's plot two of the most important features.

```{r}
ggplot(train, aes(x=p0_track_Chi2Dof, y=SPDhits, color=factor(signal))) +
  geom_point()
```