{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"機器學習以目標值可以分為兩大類回歸與分類\n1. 回歸: 例如房價預測, 預測值與實際值會盡量縮小誤差值\n2. 分類: 例如圖片分類, 判斷出圖片是貓? 狗? 或是其他動物\n\nmnist 屬於分類問題, kaggle 提供數萬張手寫數字圖片, 28x28解析度, 分別是0~9不同共10種不同數字, 接下來會使用機器學習與深度學習進行圖片分類預測!\n\n機器學習(machine learning)與深度學習(deep learning)兩者差異在於, 機器學習指的是單層或是少數層的模型, 深度學習則是一層一層架構出來, model 看起來就是層層堆積!"},{"metadata":{},"cell_type":"markdown","source":"## 資料準備\n訓練集與測試集資料共42000張與28000張資料"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(r\"../input/digit-recognizer/train.csv\")\ntest_df = pd.read_csv(r\"../input/digit-recognizer/test.csv\")\nprint(\"train_df shape:\", train_df.shape)\nprint(\"test_df shape:\", test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"訓練集當中, 第一欄為label, 代表此筆資料的數字, 其餘 pixel0 ~ pixel783 是28*28解析度的圖片"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"test data 只有 28x28 的影像資料"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"觀察 train data label 是否有缺值, 或是資料不平均?"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(train_df.iloc[:,0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"把資料視覺化, 觀看 mnsit 圖像 "},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,6))\nfor i in range(40):\n    plt.subplot(4,10,i+1)\n    plt.axis(\"off\")\n    plt.title(\"label \" + str(train_df.iloc[i,0]))\n    plt.imshow(train_df.iloc[i,1:].values.reshape(28,28), cmap=\"gray\")\nplt.tight_layout()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"進行資料縮放, sklearn 提供 StandardScaler, MinMaxScaler, MaxAbsScaler 常用三種, 通常資料可能會遇到不同項尺度, 如房價預測中價錢, 距離, 面積大小就非同一尺度, 就需要進行尺度縮放, 才不會讓尺度不一致導致模型訓練出來失真!\n* StandardScaler : 數據化成均值=0, 標準差=1\n* MinMaxScaler：縮放至 [0,1] 範圍\n* MaxAbsScaler：縮放至 [-1,1] 範圍 "},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = train_df.iloc[:,1:]\ny_train = train_df.iloc[:,0]\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(train_df.iloc[:,1:], train_df.iloc[:,0], test_size=0.2)\nX_train_org = X_train\nX_test_org = X_test\nstandardScaler = StandardScaler()\nstandardScaler.fit(X_train)\nX_train = standardScaler.transform(X_train)\nX_test = standardScaler.transform(X_test)\n\nprint(X_train.shape)\nprint(X_test.shape)\nprint(y_train.shape)\nprint(y_test.shape)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"使用機器學習(machine learning)進行分類, 未調整參數情況下, 約可達到95%還有進步空間!"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\nrandomForestClassifier = RandomForestClassifier()\nrandomForestClassifier.fit(X_train, y_train)\nrandomForestClassifier.score(X_test, y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.tree import DecisionTreeClassifier\ndecisionTreeClassifier = DecisionTreeClassifier()\ndecisionTreeClassifier.fit(X_train, y_train)\nprint(decisionTreeClassifier.score(X_test, y_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train_org = np.asarray(X_train_org).reshape(-1,28,28)\nX_test_org = np.asarray(X_test_org).reshape(-1,28,28)\nprint(X_train_org.shape)\nprint(X_test_org.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 深度學習\n開始進入深度學習領域, 進行神經網路(Neural Networks)模型建立, 目前 kaggle 最普遍也最易上手API應該算 keras, 接下來會以 keras 進行模型建立\n\n需先將 label 進行 one-hot encoder, Ex: 1-->000000001, 9-->100000000,避免最後計算誤差會因為數字大小而有差異!"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils import to_categorical\ny_train = to_categorical(y_train, 10)\ny_test = to_categorical(y_test, 10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"進行 model 建立, Sequenctial 建立模型, 將每層可再加入不同layer, 最後一層使用 softmax, 以下範例建立神經網路不到10行即可建立, 訓練後基本上都可達到95%以上\n* Dense: 最基本單層 layer, 設定多少神經元, 設定激勵函數, 只是在第一層要記得設定輸入大小(input_shape)\n* Dropout: 防止過擬和發生, 強制停用一定比例神經元, 強迫其他神經元進行學習\n* Optimizer: 最基本是使用梯度下降, 其他則是由梯度下降延伸出來(Ex:adam)\n* activation: 採用 relu, 將數值轉成[0,x], 小於0就直接等於0\n\n![](https://www.researchgate.net/profile/Paulo_Campos_Souza/publication/331097835/figure/fig2/AS:726236208840705@1550159613046/Artificial-neural-network-of-multiple-layers-and-outputs-31.ppm)"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.layers import Dense\nfrom keras.layers import Dropout\nfrom keras.models import Sequential\nfrom keras.optimizers import Adam\nfrom keras.losses import categorical_crossentropy\n\nmodel = Sequential()\nmodel.add(Dense(250, activation=\"relu\" ,input_shape=(784,)))\nmodel.add(Dropout(0.25))\nmodel.add(Dense(100, activation=\"relu\"))\nmodel.add(Dropout(0.25))\nmodel.add(Dense(10, activation=\"softmax\"))\nmodel.compile(optimizer=\"Adam\", loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"將 train test 帶入模型進行訓練!\n* batch_size: 每次帶入多少張圖片進行訓練, 如果有GPU情況之下即可進行平行運算\n* epochs: 整個訓練集重複訓練幾次. (Ex:epochs=3, 整個 X_train 從頭到尾重複訓練3次)\n* validation_data: 訓練完之後, 進行 test 資料驗證"},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(X_train, y_train, batch_size=50 ,epochs=4, validation_data=(X_test, y_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.reshape(-1,28,28,1)\nX_test = X_test.reshape(-1,28,28,1)\nprint(X_train.shape)\nprint(X_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 卷積神經網路\n卷積神經網路運用於圖片上可得到較高準確性, 每層 Convloution 可將特徵值提取, MaxPooling 將資料縮減只留下有用資訊(最大值),最後在經過 Flatten 拉成一維數據,再建立一般神經網路!\n\n![](https://miro.medium.com/max/1872/1*SGPGG7oeSvVlV5sOSQ2iZw.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.layers import Convolution2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Flatten\n\nmodelCNN = Sequential()\nmodelCNN.add(Convolution2D(32, (3,3), strides=(1,1), padding=\"same\", input_shape=(28, 28, 1), activation=\"relu\" ))\nmodelCNN.add(Convolution2D(32, (3,3), strides=(1,1), padding=\"same\", activation=\"relu\" ))\nmodelCNN.add(MaxPooling2D(pool_size=(2,2), padding=\"same\"))\nmodelCNN.add(Dropout(0.25))\n\nmodelCNN.add(Convolution2D(64, (3,3), strides=(1,1), padding=\"same\", activation=\"relu\" ))\nmodelCNN.add(Convolution2D(64, (3,3), strides=(1,1), padding=\"same\", activation=\"relu\" ))\nmodelCNN.add(MaxPooling2D(pool_size=(2,2), padding=\"same\"))\nmodelCNN.add(Dropout(0.25))\n\nmodelCNN.add(Flatten())\nmodelCNN.add(Dense(128, activation=\"relu\"))\nmodelCNN.add(Dropout(0.5))\nmodelCNN.add(Dense(10, activation=\"softmax\"))\nmodelCNN.compile(optimizer=\"adam\", loss=categorical_crossentropy, metrics=[\"accuracy\"])\nmodelCNN.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"model 可視化, 顯示每層 input 與 output, 最後一層 output 數量10, 對應 0~9 十個數字"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils import plot_model\nplot_model(modelCNN, show_shapes=True, show_layer_names=False, to_file='model.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"raw","source":"訓練模型, 訓練完了之後也可將模型另存, 下次使用時再次讀取"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import load_model\n\nhistoryCNN = modelCNN.fit(X_train, y_train, batch_size=30, epochs=5, validation_data=(X_test, y_test))\n#modelCNN.save('modelCNN.h5')   # HDF5 file, you have to pip3 install h5py if don't have it\n#modelCNN = load_model('modelCNN.h5')\n#modelCNN.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"進行 model predict, 同時將每個 epochs accuracy 可視化!"},{"metadata":{"trusted":true},"cell_type":"code","source":"result = modelCNN.predict_classes(X_test, batch_size=30)\nprint(\"Key:\", historyCNN.history.keys())\nplt.plot(historyCNN.history[\"val_accuracy\"], \"ro-\", label=\"val_accuracy\")\nplt.plot(historyCNN.history[\"accuracy\"], \"bo-\", label=\"train_accuracy\")\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"使用 confusion matrix 進行 predict 與 label 結果比對!"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nconfusion = confusion_matrix(np.argmax(y_test,axis=1), result)\nplt.figure(figsize=(12,8))\nsns.heatmap(confusion, annot=True, fmt=\"d\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.crosstab(np.argmax(y_test,axis=1), result, rownames=[\"label\"], colnames=[\"predict\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"將把判斷錯誤的圖片顯示出來, 有些影像圖片連人類都很難進行判斷"},{"metadata":{"trusted":true},"cell_type":"code","source":"errorAddress = ~np.equal(np.argmax(y_test,axis=1), result)\nprint(\"Error number =\", errorAddress.sum())\nplt.figure(figsize=(12,4))\nfor i in range(20):\n    plt.subplot(2, 10, i+1)\n    plt.axis(\"off\")\n    plt.title(\"correct:\" + str(np.argmax(y_test[errorAddress][i]))+ \"\\n predict:\"+ str(result[errorAddress][i]))\n    plt.imshow(X_test_org[errorAddress][i].reshape((28,28)), cmap=\"gray\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"如果訓練資料真的很少的話, 還可使用 ImageDataGenerator 進行訓練集的擴充, 可將資料進行微調(Ex: 水平翻轉, 選轉, 位移等). 使模型有多資料進行訓練\nReduceLROnPlateau 適用於當 accuracy 不再進步時, 就自動停下來"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\n\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', \n                                            patience=3, \n                                            verbose=1, \n                                            factor=0.5, \n                                            min_lr=0.00001)\n\ndatagen = ImageDataGenerator(\n    featurewise_center=True,\n    featurewise_std_normalization=True,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    horizontal_flip=True)\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"單張轉換"},{"metadata":{"trusted":true},"cell_type":"code","source":"transformParameters = {\"theta\":30}\nplt.subplot(2,2,1)\nplt.imshow(datagen.apply_transform(X_train_org[1].reshape(28,28,1), {\"theta\":30}).reshape(28,28), cmap=\"gray\")\n\nplt.subplot(2,2,2)\nplt.imshow(datagen.apply_transform(X_train_org[1].reshape(28,28,1), {\"flip_horizontal\":True}).reshape(28,28), cmap=\"gray\")\n\nplt.subplot(2,2,3)\nplt.imshow(datagen.apply_transform(X_train_org[1].reshape(28,28,1), {\"flip_vertical\":True}).reshape(28,28), cmap=\"gray\")\n\nplt.subplot(2,2,4)\nplt.imshow(datagen.apply_transform(X_train_org[1].reshape(28,28,1), {\"shear\":0.9}).reshape(28,28), cmap=\"gray\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"fit_generator 與 fit 差異在於, fit_generator 中使用具有 Generator 功能, 進行對於 datagen 迭代, 每次具有 batch_size 數目的圖片產生, 所以每個 step 會是總數/ batch_size"},{"metadata":{"trusted":true},"cell_type":"code","source":"modelCNN.fit_generator(datagen.flow(X_train, y_train, batch_size=32), steps_per_epoch=len(X_train)/32, \n                       validation_data=(X_test, y_test), epochs=30, callbacks=[learning_rate_reduction])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"經過 datagen 後所預測結果! 以此 model 進行測試資料集答案提交!"},{"metadata":{"trusted":true},"cell_type":"code","source":"result = modelCNN.predict_classes(X_test, batch_size=30)\nerrorAddress = ~np.equal(np.argmax(y_test,axis=1), result)\nprint(\"Error number =\", errorAddress.sum())\nplt.figure(figsize=(16,8))\nfor i in range(40):\n    plt.subplot(4, 10, i+1)\n    plt.axis(\"off\")\n    plt.title(\"correct:\" + str(np.argmax(y_test[errorAddress][i]))+ \"\\n predict:\"+ str(result[errorAddress][i]))\n    plt.imshow(X_test_org[errorAddress][i].reshape((28,28)), cmap=\"gray\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#modelCNN.fit_generator(datagen.flow(X_train, y_train, batch_size=32), steps_per_epoch=len(X_train)/32, validation_data=(X_test, y_test), epochs=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submit_test_df = standardScaler.transform(test_df)\nsubmit_test_df = submit_test_df.reshape(-1,28,28,1)\nresult = modelCNN.predict_classes(submit_test_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"最後上傳提交至 kaggle 即可完成(準確率大約 0.987)"},{"metadata":{"trusted":true},"cell_type":"code","source":"answer = pd.DataFrame()\nanswer[\"ImageId\"] = np.arange(1,len(result)+1,1)\nanswer[\"Label\"] = result\nanswer.to_csv(\"sumbit.csv\")\nanswer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}