{"cells":[{"cell_type":"markdown","metadata":{},"source":"## Biological and Artificial Neurons \nBiological neurons communicate using electrical impulses and neurotransmitters, forming complex, interconnected networks that process information.They enable complex thought and learning.\n\nArtificial neurons, inspired by biological ones, use binary inputs and outputs to perform logical computations like AND, OR, and NOT operations.Ttey provide a simplified mathematical model for computational tasks."},{"cell_type":"markdown","metadata":{},"source":"### The Perceptron \nThe Perceptron is a simple artificial neural network introduced by Frank Rosenblatt in 1957. It consists of threshold logic units (TLUs), where each unit computes a weighted sum of its inputs and applies a step function to determine the output:\n$$\nh_{W,b}(X) = \\phi (XW + b)\n$$\n- X: Input features. \n- W: Weights matrix. \n- b: Bias vector. \n- ϕ: Activation function. \n\nThe Perceptron learning rule updates weights based on errors, inspired by Hebbian learning. The perceptron learning rule is:\n$$ w_{i,j}^{(t+1)} = w_{i,j}^{(t)} + \\eta (y_j - \\hat{y}_j)x_i $$\nHowever, it is limited to linearly separable problems, leading to the development of Multilayer Perceptrons (MLPs) for solving complex tasks like XOR classification.\n\nPerceptrons do not output probabilities, lack regularization, stop training early after zero errors, and may generalize worse than logistic regression or SVM, but they can train faster."},{"cell_type":"markdown","metadata":{},"source":"### Multilayer Perceptron (MLP) and Backpropagation \n\n**A Multilayer Perceptron (MLP)** is a type of artificial neural network composed of:\n- Input Layer: Takes the input features.\n- Hidden Layers: Contain neurons that process inputs using activation functions.\n- Output Layer: Produces the final prediction.\n\n**Backpropagation Algorithm**\nBackpropagation is the process of updating weights using gradient descent:\n1. Compute the weighted sum of inputs and apply an activation function.THEN Pass results to the next layer until reaching the output.\n2. Compare predicted output with actual values using a loss function.\n3. Compute the gradient of the loss with respect to each weight.THEN Adjust weights using gradient descent.\n\n**Common activation functions**:\n- **Sigmoid**: $\\sigma(z) = \\frac{1}{1 + e^{-z}}$ \n- **Tanh**: $\\tanh(z) = 2\\sigma(2z) - 1$ \n- **ReLU**: $ReLU(z) = \\max(0, z)$ \n\n "},{"cell_type":"markdown","metadata":{},"source":"### Regression and Classification MLPs \n\n**Number of Output Neurons** \n- Regression: One neuron for a single value or multiple neurons for multi-output. \n- Classification: One neuron for binary classification, multiple neurons for multiclass classification.\n\n**Activation Function in Output Layer** \n- Regression: None , or ReLU/Sigmoid/Tanh if bounded output is needed. \n- Classification: Sigmoid for binary classification, Softmax for multiclass classification.\n\n**Loss Function** \n- Regression: MSE or Huber. \n- Classification: log loss.\n\n**Output Interpretation** \n- Regression: Direct numerical values. \n- Classification: Probabilities (0-1 for sigmoid, class probabilities summing to 1 for softmax).\n"},{"cell_type":"markdown","metadata":{},"source":"## Implementing MLPs with Keras "},{"cell_type":"code","execution_count":1,"metadata":{"execution":{"iopub.execute_input":"2025-09-21T01:21:10.530289Z","iopub.status.busy":"2025-09-21T01:21:10.530080Z","iopub.status.idle":"2025-09-21T01:21:31.766320Z","shell.execute_reply":"2025-09-21T01:21:31.765507Z","shell.execute_reply.started":"2025-09-21T01:21:10.530270Z"},"trusted":true},"outputs":[],"source":"import tensorflow as tf"},{"cell_type":"code","execution_count":2,"metadata":{"execution":{"iopub.execute_input":"2025-09-21T01:30:49.648265Z","iopub.status.busy":"2025-09-21T01:30:49.647659Z","iopub.status.idle":"2025-09-21T01:30:50.301289Z","shell.execute_reply":"2025-09-21T01:30:50.300483Z","shell.execute_reply.started":"2025-09-21T01:30:49.648241Z"},"trusted":true},"outputs":[],"source":"# load the dataset\nmnist = tf.keras.datasets.mnist.load_data()"},{"cell_type":"code","execution_count":3,"metadata":{"execution":{"iopub.execute_input":"2025-09-21T01:22:15.387323Z","iopub.status.busy":"2025-09-21T01:22:15.387062Z","iopub.status.idle":"2025-09-21T01:22:15.391322Z","shell.execute_reply":"2025-09-21T01:22:15.390726Z","shell.execute_reply.started":"2025-09-21T01:22:15.387303Z"},"trusted":true},"outputs":[],"source":"# split dataset into train ,test sets\n(X_train_full, y_train_full), (X_test, y_test) = mnist\nX_train, y_train = X_train_full[:-5000], y_train_full[:-5000] # actual training\nX_valid, y_valid = X_train_full[-5000:], y_train_full[-5000:] # validation"},{"cell_type":"code","execution_count":4,"metadata":{"execution":{"iopub.execute_input":"2025-09-21T01:22:17.260413Z","iopub.status.busy":"2025-09-21T01:22:17.259734Z","iopub.status.idle":"2025-09-21T01:22:17.425654Z","shell.execute_reply":"2025-09-21T01:22:17.424876Z","shell.execute_reply.started":"2025-09-21T01:22:17.260387Z"},"trusted":true},"outputs":[],"source":"# scale pixel values between 0 and 1\nX_train, X_valid, X_test = X_train / 255.0, X_valid / 255., X_test / 255.0"},{"cell_type":"code","execution_count":5,"metadata":{"execution":{"iopub.execute_input":"2025-09-21T01:22:21.516262Z","iopub.status.busy":"2025-09-21T01:22:21.515985Z","iopub.status.idle":"2025-09-21T01:22:25.023194Z","shell.execute_reply":"2025-09-21T01:22:25.022573Z","shell.execute_reply.started":"2025-09-21T01:22:21.516240Z"},"trusted":true},"outputs":[{"name":"stderr","output_type":"stream","text":["c:\\Users\\DELL\\miniconda3\\envs\\workshop_env\\lib\\site-packages\\keras\\src\\layers\\reshaping\\flatten.py:37: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.\n"," super().__init__(**kwargs)\n"]}],"source":"tf.random.set_seed(42) # set random seed to make the results reproducible\n\n# define a neural network with three layers\nmodel = tf.keras.Sequential([ \ntf.keras.layers.Flatten(input_shape=[28, 28]), \ntf.keras.layers.Dense(300, activation=\"relu\"), \ntf.keras.layers.Dense(100, activation=\"relu\"), \ntf.keras.layers.Dense(10, activation=\"softmax\") \n ])"},{"cell_type":"code","execution_count":6,"metadata":{"execution":{"iopub.execute_input":"2025-09-21T01:22:26.498847Z","iopub.status.busy":"2025-09-21T01:22:26.498219Z","iopub.status.idle":"2025-09-21T01:22:26.513206Z","shell.execute_reply":"2025-09-21T01:22:26.512590Z","shell.execute_reply.started":"2025-09-21T01:22:26.498823Z"},"trusted":true},"outputs":[{"data":{"text/html":["Model: \"sequential\" \n","┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n","┃ Layer (type) ┃ Output Shape ┃ Param # ┃\n","┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n","│ flatten (Flatten ) │ (None , 784 ) │ 0 │\n","├─────────────────────────────────┼────────────────────────┼───────────────┤\n","│ dense (Dense ) │ (None , 300 ) │ 235,500 │\n","├─────────────────────────────────┼────────────────────────┼───────────────┤\n","│ dense_1 (Dense ) │ (None , 100 ) │ 30,100 │\n","├─────────────────────────────────┼────────────────────────┼───────────────┤\n","│ dense_2 (Dense ) │ (None , 10 ) │ 1,010 │\n","└─────────────────────────────────┴────────────────────────┴───────────────┘\n"," \n"],"text/plain":["┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n","┃\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0m┃\n","┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n","│ flatten (\u001b[38;5;33mFlatten\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m784\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n","├─────────────────────────────────┼────────────────────────┼───────────────┤\n","│ dense (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m300\u001b[0m) │ \u001b[38;5;34m235,500\u001b[0m │\n","├─────────────────────────────────┼────────────────────────┼───────────────┤\n","│ dense_1 (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m) │ \u001b[38;5;34m30,100\u001b[0m │\n","├─────────────────────────────────┼────────────────────────┼───────────────┤\n","│ dense_2 (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m10\u001b[0m) │ \u001b[38;5;34m1,010\u001b[0m │\n","└─────────────────────────────────┴────────────────────────┴───────────────┘\n"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":[" Total params: 266,610 (1.02 MB)\n"," Trainable params: 266,610 (1.02 MB)\n"," Non-trainable params: 0 (0.00 B)\n",""]},"metadata":{},"output_type":"display_data"}],"source":"# analyze training performance\nimport matplotlib.pyplot as plt\nimport pandas as pd \npd.DataFrame(history.history).plot(figsize=(8, 5), xlim=[0, 9], ylim=[0, 1], grid=True ,style=[\"r--\", \"r--.\", \"b-\", \"b-*\"])\nplt.show()"},{"cell_type":"code","execution_count":10,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["\u001b[1m313/313\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 7ms/step - accuracy: 0.9667 - loss: 0.1074\n","Test accuracy: 0.97\n"]}],"source":"loss , acc = model.evaluate(X_test, y_test)\nprint(f\"Test accuracy: {acc:.2f}\")"},{"cell_type":"code","execution_count":14,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 100ms/step\n","[[0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n"," [0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n"," [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]\n"]}],"source":"X_new = X_test[:3]\ny_proba = model.predict(X_new)\nprint (y_proba.round()) "},{"cell_type":"code","execution_count":12,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["[7 2 1]\n"]}],"source":"import numpy as np\ny_pred = y_proba.argmax(axis=-1)\nprint(y_pred)"},{"cell_type":"code","execution_count":null,"metadata":{"trusted":true},"outputs":[],"source":""}],"metadata":{"kaggle":{"accelerator":"none","dataSources":[{"databundleVersionId":861823,"sourceId":3004,"sourceType":"competition"}],"dockerImageVersionId":31153,"isGpuEnabled":false,"isInternetEnabled":true,"language":"python","sourceType":"notebook"},"kernelspec":{"display_name":"workshop_env","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.15"}},"nbformat":4,"nbformat_minor":4}