{"cells":[{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# set up environment for project\n\nlibrary(tidyverse)\nlibrary(caret)\nlibrary(plotly)\nlibrary(matrixStats)\nlibrary(grDevices)\nlibrary(gridExtra)\nlibrary(keras)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **1. Introduction**"},{"metadata":{},"cell_type":"markdown","source":"## **1.1 Background**\n\nIn general, \n\n**The MNIST database** (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems. The database is also widely used for training and testing in the field of machine learning.\n\nIn this project, I experiment different machine learning model and compare their performance."},{"metadata":{},"cell_type":"markdown","source":"## **1.2 Data preparation**"},{"metadata":{"trusted":true},"cell_type":"code","source":"dtrain <- read.csv(\"../input/train.csv\")\ndtest <- read.csv(\"../input/test.csv\")\nsample_sub <- read.csv(\"../input/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x <- dtrain[,-1]\ny <- dtrain[,1]\ny <- factor(y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x2 <- matrix(as.numeric(unlist(x)), nrow = nrow(x))\nx_test <- matrix(as.numeric(unlist(dtest)), nrow = nrow(dtest))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **2. Data exploratory analysis**"},{"metadata":{"trusted":true},"cell_type":"code","source":"dim(x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create a function to visualize the matrix to visible number."},{"metadata":{"trusted":true},"cell_type":"code","source":"matrix.rotate <- function(img) { \n  img <- matrix(img, nrow=28L, byrow=T)\n  t(apply(img, 2, rev))\n}\nmnist.image <- function(img){\n  l = length(img)\n  rc = round(sqrt(l),0)\n  par(mfcol = c(rc, rc), mar=c(0.1,0.1,1,0.1))\n  for (i in img){\n    image(matrix.rotate(x2[i,]),\n          col = gray.colors(33), \n          axes = F, \n          main = paste(y[i]))\n  }\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mnist.image(seq(1,25,1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reduce the number of pixels by function remove near zero value from caret package."},{"metadata":{"trusted":true},"cell_type":"code","source":"nzv <- nearZeroVar(x2)\npar(mfrow = c(1,1))\nimage(matrix(1:784 %in% nzv, 28, 28),\n      col = gray.colors(33))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#col_index <- setdiff(1:ncol(x2), nzv)\n#length(col_index)\n\n#x3 <- x[col_index]\n#x_test_3 <- dtest[col_index]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Preprocessing by BoxCox."},{"metadata":{"trusted":true},"cell_type":"code","source":"#prex3 <- preProcess(x3, method = \"BoxCox\")\n#x3 <- predict(prex3, x3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **3. Modeling**\n## **3.1 KNN & SVM by using caret package**"},{"metadata":{"trusted":true},"cell_type":"code","source":"#control = trainControl(method = \"cv\",\n#                      number = 5,\n #                     p = 0.8)\n#set.seed(123)\n#fx_svm <- train(x3,y,\n#         method = \"svmRadial\",\n#         trControl = control)\n\n#set.seed(2019)\n#fx_knn <- train(x3,y,\n#               method = \"knn\",\n#               trControl = control)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#plot(fx_svm)\n#plot(fx_knn)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#yhat_svm <- predict(fx_svm, x_test_3, type = \"raw\")\n#yhat_knn <- predict(fx_knn, x_test_3, type = \"raw\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a dataframe with our svm & knn\n#submission_svm <- data.frame(ImageId = sample_sub$ImageId, Label = yhat_svm)\n#submission_knn <- data.frame(ImageId = sample_sub$ImageId, Label = yhat_knn)\n\n# save our file\n#write_csv(submission_svm, 'submission_svm.csv')\n#write_csv(submission_knn, 'submission_knn.csv')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This model score 0.96942"},{"metadata":{},"cell_type":"markdown","source":"## **3.2 Neural Network using Keras**"},{"metadata":{"trusted":true},"cell_type":"code","source":"#index <- createDataPartition(dtrain$label, times = 1, p = 0.8, list = F)\ntrain_set <- dtrain\ntest_set <- dtest","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create the data for CNN model\ndf_x <- train_set[,-1]\ndf_y <- train_set[,1]\ndf_y <- to_categorical(df_y)\n\n# create testing data\ndf_test_x <- test_set\n\n# normalize to scale from 0 to 1\ndf_x <- df_x/255\ndf_test_x <- df_test_x/255\n\n# conver to matrix\ndf_x <- matrix(as.numeric(unlist(df_x)), nrow = nrow(df_x))\ndf_test_x <- matrix(as.numeric(unlist(df_test_x)), nrow = nrow(df_test_x))\n\n# reshape to convert each row to matrix 28 x 28\ndim(df_x) <- c(nrow(df_x), 28, 28, 1)\ndim(df_test_x) <- c(nrow(df_test_x), 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#build simple model\nmodel <- keras_model_sequential()\nmodel %>%\n\n# add first conv layer with filter F32-C3 x2  \n  layer_conv_2d(filters = 32, kernel_size = c(3,3),\n                activation = \"relu\", input_shape = c(28,28,1)) %>%\n  layer_batch_normalization() %>%\n  \n  layer_conv_2d(filters = 32, kernel_size = c(3,3),\n                activation = \"relu\") %>%\n  layer_batch_normalization() %>%\n  \n  layer_conv_2d(filters = 32, kernel_size = c(5,5), strides = c(2), \n                padding = \"Same\",\n                activation = \"relu\") %>%\n  layer_batch_normalization() %>%\n    layer_dropout(rate = 0.4) %>%\n\n# add second conv layer with filter F64-C3 x2 + F64-C5 \n  layer_conv_2d(filters = 64, kernel_size = c(3,3),\n                activation = \"relu\") %>%\n  layer_batch_normalization() %>%\n  \n  layer_conv_2d(filters = 64, kernel_size = c(3,3),\n                activation = \"relu\") %>%\n  layer_batch_normalization() %>%\n  \n  layer_conv_2d(filters = 64, kernel_size = c(5,5), strides = c(2), \n                padding = \"Same\",\n                activation = \"relu\") %>%\n  layer_batch_normalization() %>%\n    layer_dropout(rate = 0.4) %>%  \n  \n# adding dense layer  \n  \n  layer_flatten() %>%\n    layer_dense(units = 128, activation = 'relu', input_shape = c(784)) %>% \n  layer_batch_normalization() %>%\n  layer_dropout(rate = 0.4) %>% \n\n  layer_dense(units = 10, activation = 'softmax')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#complie model\nmodel %>% compile(loss = \"categorical_crossentropy\",\n                   optimizer = \"adam\",\n                    metrics = c(\"accuracy\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen <- image_data_generator(\n    featurewise_center = F,\n    samplewise_center=F,\n    featurewise_std_normalization = F,\n    samplewise_std_normalization=F,\n    zca_whitening=F,\n    horizontal_flip = F,\n    vertical_flip = F,\n    width_shift_range = 0.2,\n    height_shift_range = 0.2,\n    zoom_range = 0.12,\n    rotation_range = 0.1,\n    shear_range = 0.2\n)\n\n\ndatagen %>% fit_image_data_generator(df_x)\n\n# create model using data with new generated data\n\nhistory <- model %>%\n            fit_generator(flow_images_from_data(df_x, df_y, datagen, batch_size = 64),\n                          steps_per_epoch = nrow(df_x)/64, epochs = 30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"yhat_cnn <- model %>% predict_classes(df_test_x, batch_size=64)\nconfusionMatrix(factor(yhat_cnn), factor(test_set[,1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a dataframe with CNN model\nmy_submission <- data.frame(ImageId = sample_sub$ImageId, Label = yhat_cnn)\n\n# save our file\nwrite_csv(my_submission, 'submission_cnn.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Model 2 - "},{"metadata":{"trusted":true},"cell_type":"code","source":"#build simple model\nmodel2 <- keras_model_sequential()\nmodel2 %>%\n\n# add first conv layer with filter F32-C3 x2  \n  layer_conv_2d(filters = 32, kernel_size = c(3,3),\n                activation = \"relu\", input_shape = c(28,28,1)) %>%\n  layer_batch_normalization() %>%\n  \n  layer_conv_2d(filters = 32, kernel_size = c(3,3),\n                activation = \"relu\") %>%\n  layer_batch_normalization() %>%\n  \n  layer_conv_2d(filters = 32, kernel_size = c(5,5), strides = c(2), \n                padding = \"Same\",\n                activation = \"relu\") %>%\n  layer_batch_normalization() %>%\n    layer_dropout(rate = 0.4) %>%\n\n# add second conv layer with filter F64-C3 x2 + F64-C5 \n  layer_conv_2d(filters = 64, kernel_size = c(3,3),\n                activation = \"relu\") %>%\n  layer_batch_normalization() %>%\n  \n  layer_conv_2d(filters = 64, kernel_size = c(3,3),\n                activation = \"relu\") %>%\n  layer_batch_normalization() %>%\n  \n  layer_conv_2d(filters = 64, kernel_size = c(5,5), strides = c(2), \n                padding = \"Same\",\n                activation = \"relu\") %>%\n  layer_batch_normalization() %>%\n    layer_dropout(rate = 0.4) %>%  \n  \n# adding dense layer  \n  \n  layer_flatten() %>%\n    layer_dense(units = 512, activation = 'relu', input_shape = c(784)) %>% \n  layer_batch_normalization() %>%\n  layer_dropout(rate = 0.4) %>% \n    \n    layer_dense(units = 256, activation = 'relu') %>% \n  layer_batch_normalization() %>%\n  layer_dropout(rate = 0.4) %>% \n    \n    layer_dense(units = 128, activation = 'relu') %>% \n  layer_batch_normalization() %>%\n  layer_dropout(rate = 0.4) %>% \n    \n  layer_dense(units = 10, activation = 'softmax')\n\n#complie model\nmodel2 %>% compile(loss = \"categorical_crossentropy\",\n                   optimizer = \"adam\",\n                    metrics = c(\"accuracy\"))\n\n# fit model\nmodel2 %>% fit(df_x, df_y,\n              epochs = 30, batch_size = 128,\n              validation_split = 0.2)\n\n# calculate accuracy\nyhat_cnn_2 <- model2 %>% predict_classes(df_test_x, batch_size=64)\n\n# create a dataframe with CNN model\nmy_submission_2 <- data.frame(ImageId = sample_sub$ImageId, Label = yhat_cnn_2)\n\n# save our file\nwrite_csv(my_submission_2, 'submission_cnn_2.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This cnn model score 0.9918"},{"metadata":{},"cell_type":"markdown","source":"This kernel is continued..."}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}