{"cells":[{"metadata":{},"cell_type":"markdown","source":"<p style=\"font-size:13pt;\"> This kernel shows you how to find the best combination of three methods using a visual approach.</p>\n\nIn machine learning, mixtures of different models are often used to reduce variance. This raises the question of the best mix. Several points make the answer difficult.\n\n1. The mixture can only be determined in the training data set but not in the test data set.\n\n2. Finding the best mix for several models is problematic because of the dimensionality, since a maximum of three dimensions can be represented - perhaps four or more dimensions due to different colours and shapes.\n\n![https://media0.giphy.com/media/m7yOA6RLIho8o/giphy.gif?cid=ecf05e47ca526e89aeeab8dd7f51516936a2c39d081bdae9&rid=giphy.gif](https://media0.giphy.com/media/m7yOA6RLIho8o/giphy.gif?cid=ecf05e47ca526e89aeeab8dd7f51516936a2c39d081bdae9&rid=giphy.gif)\n\nIn this kernel I will show you how to find a good mix with three different methods using the cat2 competition dataset.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Firstly, let´s read and adjust the test dataset:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"Cat_Train <- read.csv(\"../input/cat-in-the-dat-ii/train.csv\")\n\np = NA\nfor(j in 2:25){\n  o <- ifelse(is.na(Cat_Train[, j]) | Cat_Train[, j] ==\"\", -1, Cat_Train[, j])\n  p = cbind(p, o)\n}\n\nrm(o)\n\np <- as.data.frame(p[ ,-1])\n\ncolnames(p) <- colnames(Cat_Train)[2:25]\n\nfor (i in c(1:24)) {\n  p[, i] <- as.factor(p[, i])\n}\n\n\nset.seed(12345)\nzs <- runif(nrow(p), min = 0, max = 1)\nTRAIN <- p[zs <= 0.8,]\nTEST <- p[zs > 0.8,]\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using h2o to chose the models from:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"library(h2o)\nh2o.init()\n\nh2oTr <- as.h2o(TRAIN)\nh2oTe <- as.h2o(TEST)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Choice is: \n\n1. Logit\n2. Naive Bayes Classifier,\n3. and Neural Nets ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Logit:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"LOGIT <- \n    h2o.glm(\n        y = \"target\",\n        training_frame = h2oTr,\n        validation_frame = h2oTe,\n        link = \"logit\",\n        family = \"binomial\",\n        alpha = 1,\n        lambda = 0.000006\n        )\nplot(h2o.performance(model = LOGIT, valid = T))\nh2o.auc(LOGIT, valid = T)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Naive Bayes Classifier:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"NBC <-\n  h2o.naiveBayes(\n        y = \"target\",\n        training_frame = h2oTr,\n        validation_frame = h2oTe,\n        laplace = 4\n        )\nplot(h2o.performance(model = NBC, valid = T))\nh2o.auc(NBC, valid = T)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Neural Net:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"NN <- \n    h2o.deeplearning(y = \"target\",\n        training_frame = h2oTr,\n        validation_frame = h2oTe,\n        hidden = c(128, 64, 32),\n        epochs = 8,\n        l2 = .001,\n        activation = \"RectifierWithDropout\",                  \n        hidden_dropout_ratios = c(.15,.1,.05),\n        mini_batch_size = 10000   \n        )\n\nplot(h2o.performance(model = NN, valid = T))\nh2o.auc(NN, valid = T)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Prediction:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"PredLOGIT <- predict(LOGIT, newdata = h2oTe)\nPredNBC<- predict(NBC, newdata = h2oTe)\nPredNN <- predict(NN, newdata = h2oTe)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Transforming the h2o data.frame back to data.frame","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"PredLOGIT <- as.data.frame(PredLOGIT)\nPredLOGIT$p1 <- rank(PredLOGIT$p1)/length(unique(PredLOGIT$p1))\nPredNBC <- as.data.frame(PredNBC)\nPredNBC$p1 <- rank(PredNBC$p1)/length(unique(PredNBC$p1))\nPredNN <- as.data.frame(PredNN)\nPredNN$p1 <- rank(PredNN$p1)/length(unique(PredNN$p1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Loading packages for calculation of auc and for parallel loops as well.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"library(pROC)\nlibrary(doParallel)\nlibrary(parallel)\nNumOfCores <- detectCores() \nregisterDoParallel(NumOfCores)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using a parallel loop to check for the best mix.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"H=c(0:100*0.01)\nA = rep(NA, times = length(H))\nAA = NA\n\nfor(h in H) {\n  B=c(0:(100-h*100)*.01)\n  k = 1\n      \n  A = foreach::foreach(b = B, .combine = rbind, .inorder = T, .multicombine = F) %dopar% {\n      \n      BagPred <-\n          PredLOGIT$p1 * h + \n          PredNBC$p1 * b + \n          PredNN$p1 * (1 - b - h)\n      \n      auc(TEST$target, BagPred)\n  }\n  A = as.data.frame(A)\n\n  if(length(H)-nrow(A) != 0){\n      NAs = as.data.frame(rep(NA, times = (length(H)-nrow(A))))\n      colnames(NAs) = colnames(A)\n      A = rbind(A, NAs)\n  }\n  AA = rbind(AA, t(A))\n\n  A = rep(NA, times = length(H))\n\n}\n\nAA <- AA[-1,]\ncolnames(AA) <- seq(from = 0, to = 1, by = .01)\nrownames(AA) <- seq(from = 0, to = 1, by = .01)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To visualize, we use a heatmap:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"library(pheatmap)\npheatmap::pheatmap(AA, cluster_rows = F, cluster_cols = F, width = 20, height = 20, fontsize = 9)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Blended AUC:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"noquote(paste(\"Blended AUC: \", max(AA, na.rm = T), sep = \"\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The more red the area, the better the mixture.\n\n> First row is without Logit.\n>\n> First column is without Naive Bayes.\n>\n> Last column is without the Neural Net.\n\nThe mixed models auc is higher compared to all single ones. This is because the weighting decreases the variance of the prediction.\n\nThe best mix is not 30% each. It´s around 60-80% Logit, 15-20% Naive Bayes and the residual to the Neural Net.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now let´s check for performance in LB:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"Cat_Test <- read.csv(\"../input/cat-in-the-dat-ii/test.csv\")\n\nq = NA\nfor(j in 2:24){\n  o <- ifelse(is.na(Cat_Test[, j]) | Cat_Test[, j] ==\"\", -1, Cat_Test[, j])\n  q = cbind(q, o)\n}\n\nrm(o)\n\nq <- as.data.frame(q[ ,-1])\n\ncolnames(q) <- colnames(Cat_Test)[2:24]\n\nfor (i in c(1:23)) {\n  q[, i] <- as.factor(q[, i])\n}\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"p_h2o <- as.h2o(p)\nq_h2o <- as.h2o(q)\n\nLOGIT <- h2o.glm(\n        y = \"target\",\n        training_frame = p_h2o,\n        link = \"logit\",\n        family = \"binomial\",\n        alpha = 1,\n        lambda = 0.000006\n    )\n\nNBC <-\n  h2o.naiveBayes(\n        y = \"target\",\n        training_frame = p_h2o,\n        laplace = 4\n    )\n\nNN <- h2o.deeplearning(y = \"target\",\n        training_frame = p_h2o,\n        hidden = c(128, 64, 32),\n        epochs = 8,\n        l2 = .001,\n        activation = \"RectifierWithDropout\",                  \n        hidden_dropout_ratios = c(.15,.1,.05),\n        mini_batch_size = 10000    \n    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Prediction:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"PredLOGIT <- predict(LOGIT, newdata = q_h2o)\nPredLOGIT <- as.data.frame(PredLOGIT)\nPredNBC<- predict(NBC, newdata = q_h2o)\nPredNBC <- as.data.frame(PredNBC)\nPredNN <- predict(NN, newdata = q_h2o)\nPredNN <- as.data.frame(PredNN)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Submission:","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"samp_sub <- read.csv(\"../input/cat-in-the-dat-ii/sample_submission.csv\")\nsamp_sub$target <- rank(PredLOGIT$p1)/length(unique(PredLOGIT$p1)) * .80 + \n                   rank(PredNBC$p1)/length(unique(PredNBC$p1)) * .18 +\n                   rank(PredNN$p1)/length(unique(PredNN$p1)) * .02\n\nwrite.csv(x = samp_sub, file = \"submission.csv\", row.names = F)\nhead(samp_sub)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As can be seen, the LB score is close to validation score.","execution_count":null}],"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.6.0","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":4}