{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-03-08T09:54:27.213613Z","iopub.execute_input":"2023-03-08T09:54:27.215369Z","iopub.status.idle":"2023-03-08T09:54:27.259878Z","shell.execute_reply.started":"2023-03-08T09:54:27.215317Z","shell.execute_reply":"2023-03-08T09:54:27.258769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This notebook explores an analytic perspective on the structure of a basic machine learning algorithm. As such, it then intends to assess which factors primarily affect model accuracy. The primary details of this notebook is then provided below as follows:\n\nDataset Utilized: MNIST and Eye Gaze Data Sets\nApproach: Basic Machine Learning Models, Convolutional Layers, Application of Various Activation Functions, Implementation of Various Optimizers, Comparison with Pre-trained Models, Data Pre-treatment\n\nFrom this, this experiment is then presented to explore the 10 following hypotheses.\n\nOn Data Pre-Processing\n\n1) The number of samples included in the training data set directly correlate to the accuracy of the model; that is, more samples indicate a higher validation accuracy.\n\n2) The level of variety involved in the training and testing data sets provide a direct relationship between the nature of the data as well as that of the model accuracy\n\n3) It is hypothesized that normalizing the data allows for a more uniform and basic analysis of data; hence, in  ontrast to an unnormalize set when subjected to the same model, it is expected that the former will yield a higher validation accuracy during testing\n\n4) In image processing, treating or converting the dataset into grayscale would yield for a more accurate validation accuracy when subjected within the same model\n\n5) Lastly, the implementation of data cleaning and filtering methods will allow for the generation of a more varying data set; and hence, will yield an improved validation accuracy rating in contrast to immediately subjecting the data into the model for training and testing.\n\nOn Model Structure\n1) The number of layers included in a model directly correlate with its accuracy; that is, the more layers the model has, the more accurate the result will be.\n\n2) The number of units within the convolutional layer directly correlates with its accuracy; hence, the higher the units implemented, a greater and more plausible accuracy level will be recorded\n\n3) The number of training epochs are dirrectly correlated with the accuracy of the model; therefore, increasing the number of epochs will also yield higher validation accuracies from the model.\n\n4) It is hypothesized that machine learning models are, in thier current state, specific rather than general; hence, inconsistencies will be observed when introduced or applied in a different dataset.\n\n5) \n","metadata":{}},{"cell_type":"markdown","source":"I. On Data Pre-Processing","metadata":{}},{"cell_type":"code","source":"\"\"\"\nThis section will present the creation of a basic machine learning model for the purpose of classifying images primarily based on the MNIST data set. As such, this set of codes will then involve\nstructures aimed at exploring, testing, and analyzing the results acquired based on the five (5) hypothese given under this subject.\n\"\"\"","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:54:27.261352Z","iopub.execute_input":"2023-03-08T09:54:27.261777Z","iopub.status.idle":"2023-03-08T09:54:27.270282Z","shell.execute_reply.started":"2023-03-08T09:54:27.26174Z","shell.execute_reply":"2023-03-08T09:54:27.269154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#A The number of samples included in the training data set directly correlate to the accuracy of the model; that is, more samples indicate a higher validation accuracy.\n\n# Import the Necessary Libraries and Data Set\n\nimport tensorflow as tf\nimport tensorflow.keras as keras\nimport matplotlib.pyplot as plt\n\n#train_df= pd.read_csv('/kaggle/input/fashionmnist/fashion-mnist_train.csv')\n#test_df = pd.read_csv('/kaggle/input/fashionmnist/fashion-mnist_test.csv')\n\nfashion_mnist = tf.keras.datasets.fashion_mnist\n(train_images, train_labels), (valid_images, valid_labels) = fashion_mnist.load_data()\n\nnumber_of_classes = train_labels.max() + 1\nnumber_of_classes\n\n# Reshaping the Data\ntrain_images = train_images.reshape(60000, 784)\nvalid_images = valid_images.reshape(10000, 784)\n\n#Data Normalization for Hypothesis 1.3\n#train_images = train_images.astype('float32') / 255\n#valid_images = valid_images.astype('float32') / 255\n\nimport tensorflow.keras as keras\nnum_classes = 10\n\ntrain_labels = keras.utils.to_categorical(train_labels, num_classes)\nvalid_labels = keras.utils.to_categorical(valid_labels, num_classes)\n\nfrom tensorflow.keras.models import Sequential\n\nmodel = Sequential()\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.layers import Flatten\nfrom tensorflow.keras.models import Sequential\n\nmodel.add(Dense(units=1024, activation='relu', input_shape=(784,)))\nmodel.add(Dense(units = 10, activation='softmax'))\n\n# This serves as the addition of more layers into the model for hypothesis 2.1\n#model.add(Dense(units = 1024, activation='relu'))\n#model.add(Dense(units = 1024, activation='relu'))\n#model.add(Dense(units = 1024, activation='relu'))\n\n#This serves as the final layer for classification; hence, having 10 units and an activation function of softmax","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:54:27.272026Z","iopub.execute_input":"2023-03-08T09:54:27.27343Z","iopub.status.idle":"2023-03-08T09:54:41.710821Z","shell.execute_reply.started":"2023-03-08T09:54:27.273377Z","shell.execute_reply":"2023-03-08T09:54:41.709697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#After constructing the base model, we utilize the set of codes below in order to be provided with a visualization of such for us to clarify and confirm our proposed design\n\nmodel.summary()\n\nkeras.utils.plot_model(model, show_shapes=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:54:41.715036Z","iopub.execute_input":"2023-03-08T09:54:41.715438Z","iopub.status.idle":"2023-03-08T09:54:41.983207Z","shell.execute_reply.started":"2023-03-08T09:54:41.715408Z","shell.execute_reply":"2023-03-08T09:54:41.982055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Once done, we then compile the model to be later on subject to fitting or training.\nmodel.compile(loss='categorical_crossentropy', metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:54:41.985482Z","iopub.execute_input":"2023-03-08T09:54:41.985875Z","iopub.status.idle":"2023-03-08T09:54:42.009698Z","shell.execute_reply.started":"2023-03-08T09:54:41.985833Z","shell.execute_reply":"2023-03-08T09:54:42.008758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#This set of codes allow for the training of the model. Some elements that may be evaluated under this section would be that of the number of epochs will be tackled under hypothesis 2.3\n\nhistory = model.fit(\n    train_images,\n    train_labels,\n    epochs=5,\n    verbose=True,\n    validation_data=(valid_images, valid_labels)\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:54:42.011251Z","iopub.execute_input":"2023-03-08T09:54:42.011633Z","iopub.status.idle":"2023-03-08T09:55:19.814815Z","shell.execute_reply.started":"2023-03-08T09:54:42.011596Z","shell.execute_reply":"2023-03-08T09:55:19.81344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The results acquired here will then serve as the primary benchmark of the proponents for the performance of the model. With the various variations in the hypothesis listed in sections 1 and 2, the primary goals will then be to observe how such individual manipulations are able to impact this value.","metadata":{}},{"cell_type":"code","source":"#Generally, one may opt to enter the syntax \"model.predict(train_images[0:10])\" for assessing the performance of the model;\n#however, it will provide one that may not be properly interpreted by a human being; hence, for a better manner of visualization\n#and presentation, the set of codes are then presented below.\n\nimport numpy as np\n\ndata_idx = 59998 # The question number to study with. Feel free to change up to 59999.\n\ntrain_images = train_images.reshape(60000, 28,28)\nvalid_images = valid_images.reshape(10000, 28, 28)\n \nplt.figure()\nplt.imshow(train_images[data_idx], cmap='gray')\nplt.colorbar()\nplt.grid(False)\nplt.show()\n\nx_values = range(number_of_classes)\n\ntrain_images = train_images.reshape(60000, 784)\nvalid_images = valid_images.reshape(10000, 784)\n\npred = model.predict(train_images[data_idx:data_idx+1]).flatten()\n\nplt.figure()\nplt.bar(x_values, pred)\nplt.xticks(range(10))\nplt.show()\n\nprint(\"predicted: \", np.argmax(pred))\nprint(\"correct answer:\", train_labels[data_idx])","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:56:17.75726Z","iopub.execute_input":"2023-03-08T09:56:17.758333Z","iopub.status.idle":"2023-03-08T09:56:18.798589Z","shell.execute_reply.started":"2023-03-08T09:56:17.758292Z","shell.execute_reply":"2023-03-08T09:56:18.797329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#This section then introduces the creation of a second model for hypothesis 1.1 or A.\ntrain_df2= pd.read_csv('/kaggle/input/fashionmnist/fashion-mnist_train.csv')\ntest_df2 = pd.read_csv('/kaggle/input/fashionmnist/fashion-mnist_test.csv')\n\n","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:55:20.373034Z","iopub.status.idle":"2023-03-08T09:55:20.37356Z","shell.execute_reply.started":"2023-03-08T09:55:20.373298Z","shell.execute_reply":"2023-03-08T09:55:20.373324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#B The level of variety involved in the training and testing data sets provide a direct relationship between the nature of the data as well as that of the model accuracy","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:55:20.376151Z","iopub.status.idle":"2023-03-08T09:55:20.37665Z","shell.execute_reply.started":"2023-03-08T09:55:20.376379Z","shell.execute_reply":"2023-03-08T09:55:20.376415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#C It is hypothesized that normalizing the data allows for a more uniform and basic analysis of data; hence, in  ontrast to an unnormalize set when subjected to the same model, it is expected that the former will yield a higher validation accuracy during testing\n\n# Import the Necessary Libraries and Data Set\n\nimport tensorflow as tf\nimport tensorflow.keras as keras\nimport matplotlib.pyplot as plt\n\n#train_df= pd.read_csv('/kaggle/input/fashionmnist/fashion-mnist_train.csv')\n#test_df = pd.read_csv('/kaggle/input/fashionmnist/fashion-mnist_test.csv')\n\nfashion_mnist13 = tf.keras.datasets.fashion_mnist\n(train_images13, train_labels13), (valid_images13, valid_labels13) = fashion_mnist13.load_data()\n\nnumber_of_classes13 = train_labels13.max() + 1\nnumber_of_classes13\n\n# Reshaping the Data\ntrain_images13 = train_images13.reshape(60000, 784)\nvalid_images13 = valid_images13.reshape(10000, 784)\n\n#Data Normalization for Hypothesis 1.3\ntrain_images13 = train_images13.astype('float32') / 255\nvalid_images13 = valid_images13.astype('float32') / 255\n\nimport tensorflow.keras as keras\nnum_classes13 = 10\n\ntrain_labels13 = keras.utils.to_categorical(train_labels13, num_classes13)\nvalid_labels13 = keras.utils.to_categorical(valid_labels13, num_classes13)\n\nfrom tensorflow.keras.models import Sequential\n\nmodel13 = Sequential()\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.layers import Flatten\nfrom tensorflow.keras.models import Sequential\n\nmodel13.add(Dense(units=1024, activation='relu', input_shape=(784,)))\nmodel13.add(Dense(units = 10, activation='softmax'))\n\n# This serves as the addition of more layers into the model for hypothesis 2.1\n#model.add(Dense(units = 1024, activation='relu'))\n#model.add(Dense(units = 1024, activation='relu'))\n#model.add(Dense(units = 1024, activation='relu'))\n\n#This serves as the final layer for classification; hence, having 10 units and an activation function of softmax","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:55:20.379761Z","iopub.status.idle":"2023-03-08T09:55:20.381075Z","shell.execute_reply.started":"2023-03-08T09:55:20.380761Z","shell.execute_reply":"2023-03-08T09:55:20.380789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#After constructing the base model, we utilize the set of codes below in order to be provided with a visualization of such for us to clarify and confirm our proposed design\n\nmodel13.summary()\n\nkeras.utils.plot_model(model13, show_shapes=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Once done, we then compile the model to be later on subject to fitting or training.\nmodel13.compile(loss='categorical_crossentropy', metrics=['accuracy'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#This set of codes allow for the training of the model. Some elements that may be evaluated under this section would be that of the number of epochs will be tackled under hypothesis 2.3\n\nhistory = model13.fit(\n    train_images,\n    train_labels,\n    epochs=5,\n    verbose=True,\n    validation_data=(valid_images, valid_labels)\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\n\ndata_idx = 59998 # The question number to study with. Feel free to change up to 59999.\n\ntrain_images = train_images.reshape(60000, 28,28)\nvalid_images = valid_images.reshape(10000, 28, 28)\n \nplt.figure()\nplt.imshow(train_images[data_idx], cmap='gray')\nplt.colorbar()\nplt.grid(False)\nplt.show()\n\nx_values = range(number_of_classes)\n\ntrain_images = train_images.reshape(60000, 784)\nvalid_images = valid_images.reshape(10000, 784)\n\npred = model.predict(train_images[data_idx:data_idx+1]).flatten()\n\nplt.figure()\nplt.bar(x_values, pred)\nplt.xticks(range(10))\nplt.show()\n\nprint(\"predicted: \", np.argmax(pred))\nprint(\"correct answer:\", train_labels[data_idx])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#D In image processing, treating or converting the dataset into grayscale would yield for a more accurate validation accuracy when subjected within the same model","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:55:20.38249Z","iopub.status.idle":"2023-03-08T09:55:20.383387Z","shell.execute_reply.started":"2023-03-08T09:55:20.383085Z","shell.execute_reply":"2023-03-08T09:55:20.383126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#E Lastly, the implementation of data cleaning and filtering methods will allow for the generation of a more varying data set; and hence, will yield an improved validation accuracy rating in contrast to immediately subjecting the data into the model for training and testing.","metadata":{"execution":{"iopub.status.busy":"2023-03-08T09:55:20.385093Z","iopub.status.idle":"2023-03-08T09:55:20.38575Z","shell.execute_reply.started":"2023-03-08T09:55:20.38547Z","shell.execute_reply":"2023-03-08T09:55:20.385498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"II. On Model Structure","metadata":{}},{"cell_type":"code","source":"# A. The number of layers included in a model directly correlate with its accuracy; \n# that is, the more layers the model has, the more accurate the result will be","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\nimport tensorflow.keras as keras\nimport matplotlib.pyplot as plt\n\n#train_df= pd.read_csv('/kaggle/input/fashionmnist/fashion-mnist_train.csv')\n#test_df = pd.read_csv('/kaggle/input/fashionmnist/fashion-mnist_test.csv')\n\nfashion_mnist21 = tf.keras.datasets.fashion_mnist\n(train_images21, train_labels21), (valid_images21, valid_labels21) = fashion_mnist21.load_data()\n\nnumber_of_classes21 = train_labels21.max() + 1\nnumber_of_classes21\n\n# Reshaping the Data\ntrain_images21 = train_images21.reshape(60000, 784)\nvalid_images21 = valid_images21.reshape(10000, 784)\n\n#Data Normalization for Hypothesis 1.3\n#train_images21 = train_images21.astype('float32') / 255\n#valid_images21 = valid_images21.astype('float32') / 255\n\nimport tensorflow.keras as keras\nnum_classes21 = 10\n\ntrain_labels21 = keras.utils.to_categorical(train_labels13, num_classes13)\nvalid_labels21 = keras.utils.to_categorical(valid_labels13, num_classes13)\n\nfrom tensorflow.keras.models import Sequential\n\nmodel21 = Sequential()\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.layers import Flatten\nfrom tensorflow.keras.models import Sequential\n\nmodel21.add(Dense(units=1024, activation='relu', input_shape=(784,)))\nmodel21.add(Dense(units = 1024, activation='relu'))\nmodel21.add(Dense(units = 10, activation='softmax'))\n\nmodel211 = Sequential()\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.layers import Flatten\nfrom tensorflow.keras.models import Sequential\n\nmodel212.add(Dense(units=1024, activation='relu', input_shape=(784,)))\nmodel212.add(Dense(units = 1024, activation='relu'))\nmodel212.add(Dense(units = 1024, activation='relu'))\nmodel212.add(Dense(units = 10, activation='softmax'))\n\nmodel213.add(Dense(units=1024, activation='relu', input_shape=(784,)))\nmodel213.add(Dense(units = 1024, activation='relu'))\nmodel213.add(Dense(units = 1024, activation='relu'))\nmodel213.add(Dense(units = 1024, activation='relu'))\nmodel213.add(Dense(units = 10, activation='softmax'))\n# This serves as the addition of more layers into the model for hypothesis 2.1\n#model.add(Dense(units = 1024, activation='relu'))\n#model.add(Dense(units = 1024, activation='relu'))\n#model.add(Dense(units = 1024, activation='relu'))\n\n#This serves as the final layer for classification; hence, having 10 units and an activation function of softmax","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# B. The number of units within the convolutional layer directly correlates with its accuracy; \n# hence, the higher the units implemented, a greater and more plausible accuracy level will be recorded","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# C. The number of training epochs are dirrectly correlated with the accuracy of the model; \n# therefore, increasing the number of epochs will also yield higher validation accuracies from the model.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# D. It is hypothesized that machine learning models are, in thier current state, specific rather than general; \n# hence, inconsistencies will be observed when introduced or applied in a different dataset.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"E.","metadata":{},"execution_count":null,"outputs":[]}]}