![\"image-1\"](\"https://i.ibb.co/gV6Td9Z/image-1.png\")
\n\n\n\n\nBe Tryborg :/
","metadata":{}},{"cell_type":"markdown","source":"First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0.
","metadata":{}},{"cell_type":"code","source":"# Python ≥3.5 is required\nimport sys\nassert sys.version_info >= (3, 5)\n\n# Scikit-Learn ≥0.20 is required\nimport sklearn\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport gc\nfrom tensorflow.keras.utils import to_categorical\n\nassert sklearn.__version__ >= \"0.20\"\n\ntry:\n # %tensorflow_version only exists in Colab.\n %tensorflow_version 2.x\nexcept Exception:\n pass\n\n# TensorFlow ≥2.0 is required\nimport tensorflow as tf\nfrom tensorflow import keras\nassert tf.__version__ >= \"2.0\"\n\n%load_ext tensorboard\n\n# Common imports\nimport numpy as np\nimport os\n\n# to make this notebook's output stable across runs\nnp.random.seed(42)\n\n# To plot pretty figures\n%matplotlib inline\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nmpl.rc('axes', labelsize=14)\nmpl.rc('xtick', labelsize=12)\nmpl.rc('ytick', labelsize=12)\n\n# Where to save the figures\nPROJECT_ROOT_DIR = \".\"\nCHAPTER_ID = \"deep\"\nIMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\nos.makedirs(IMAGES_PATH, exist_ok=True)\n\ndef save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n print(\"Saving figure\", fig_id)\n if tight_layout:\n plt.tight_layout()\n plt.savefig(path, format=fig_extension, dpi=resolution)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-02T06:14:19.025514Z","iopub.execute_input":"2021-12-02T06:14:19.026316Z","iopub.status.idle":"2021-12-02T06:14:24.563143Z","shell.execute_reply.started":"2021-12-02T06:14:19.026228Z","shell.execute_reply":"2021-12-02T06:14:24.562375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Leaky ReLU
","metadata":{}},{"cell_type":"code","source":"def leaky_relu(z, alpha=0.01):\n return np.maximum(alpha*z, z)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:14:25.62901Z","iopub.execute_input":"2021-12-02T06:14:25.62949Z","iopub.status.idle":"2021-12-02T06:14:25.640903Z","shell.execute_reply.started":"2021-12-02T06:14:25.629434Z","shell.execute_reply":"2021-12-02T06:14:25.639761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(z, leaky_relu(z, 0.05), \"b-\", linewidth=2)\nplt.plot([-5, 5], [0, 0], 'k-')\nplt.plot([0, 0], [-0.5, 4.2], 'k-')\nplt.grid(True)\nprops = dict(facecolor='black', shrink=0.1)\nplt.annotate('Leak', xytext=(-3.5, 0.5), xy=(-5, -0.2), arrowprops=props, fontsize=14, ha=\"center\")\nplt.title(\"Leaky ReLU activation function\", fontsize=14)\nplt.axis([-5, 5, -0.5, 4.2])\n\nsave_fig(\"leaky_relu_plot\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-02T06:14:25.642355Z","iopub.execute_input":"2021-12-02T06:14:25.643859Z","iopub.status.idle":"2021-12-02T06:14:26.507041Z","shell.execute_reply.started":"2021-12-02T06:14:25.643809Z","shell.execute_reply":"2021-12-02T06:14:26.504722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"[m for m in dir(keras.activations) if not m.startswith(\"_\")]","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:14:26.508357Z","iopub.execute_input":"2021-12-02T06:14:26.50877Z","iopub.status.idle":"2021-12-02T06:14:26.52298Z","shell.execute_reply.started":"2021-12-02T06:14:26.508733Z","shell.execute_reply":"2021-12-02T06:14:26.522003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"[m for m in dir(keras.layers) if \"relu\" in m.lower()]","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:14:26.529088Z","iopub.execute_input":"2021-12-02T06:14:26.529508Z","iopub.status.idle":"2021-12-02T06:14:26.541225Z","shell.execute_reply.started":"2021-12-02T06:14:26.529457Z","shell.execute_reply":"2021-12-02T06:14:26.540374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's train a neural network on tps november using the Leaky ReLU:
","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/tabular-playground-series-dec-2021/train.csv')\nsubmission = pd.read_csv('../input/tabular-playground-series-dec-2021/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:20:11.45346Z","iopub.execute_input":"2021-12-02T06:20:11.453734Z","iopub.status.idle":"2021-12-02T06:20:27.924514Z","shell.execute_reply.started":"2021-12-02T06:20:11.453704Z","shell.execute_reply":"2021-12-02T06:20:27.923777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#reduce memory\ndef reduce_mem_usage(df, verbose=True):\n numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n start_mem = df.memory_usage().sum() / 1024**2 \n for col in df.columns:\n col_type = df[col].dtypes\n if col_type in numerics:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64) \n end_mem = df.memory_usage().sum() / 1024**2\n if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n return df","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:20:27.926351Z","iopub.execute_input":"2021-12-02T06:20:27.926612Z","iopub.status.idle":"2021-12-02T06:20:27.94067Z","shell.execute_reply.started":"2021-12-02T06:20:27.926578Z","shell.execute_reply":"2021-12-02T06:20:27.938194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = reduce_mem_usage(train)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:20:27.941909Z","iopub.execute_input":"2021-12-02T06:20:27.942173Z","iopub.status.idle":"2021-12-02T06:20:49.598285Z","shell.execute_reply.started":"2021-12-02T06:20:27.942137Z","shell.execute_reply":"2021-12-02T06:20:49.597502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y = train['Cover_Type']\n\ntrain = train.drop(3403875) #droping single row of label 5\ny = y.drop(3403875)\n\ntrain = train.head(3000000) #just for fast computing remove these 2 lines to use full dataset\ny = y.head(3000000)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:20:49.600125Z","iopub.execute_input":"2021-12-02T06:20:49.600829Z","iopub.status.idle":"2021-12-02T06:20:50.458592Z","shell.execute_reply.started":"2021-12-02T06:20:49.600782Z","shell.execute_reply":"2021-12-02T06:20:50.457862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler, OrdinalEncoder\n\n#no preprocessing done will do in next version...\nstd = StandardScaler()\nencoder = OrdinalEncoder()\n\ny = pd.DataFrame(to_categorical(encoder.fit_transform(y.values.reshape(-1,1))))\n\ntrain.drop(['Id','Cover_Type','Soil_Type7', 'Soil_Type15'], axis=1, inplace=True)\n\nX_train, X_test , y_train, y_test = train_test_split(train,y, stratify=y, test_size=0.2, random_state=42, shuffle=True)\nX_train = pd.DataFrame(std.fit_transform(X_train), columns = X_train.columns)\nX_test = pd.DataFrame(std.transform(X_test), columns = X_test.columns)\n\ndel train, y, std, encoder\n\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train.shape, X_test.shape, y_train.shape, y_test.shape","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:23:46.928031Z","iopub.execute_input":"2021-12-02T06:23:46.928298Z","iopub.status.idle":"2021-12-02T06:23:46.935559Z","shell.execute_reply.started":"2021-12-02T06:23:46.928263Z","shell.execute_reply":"2021-12-02T06:23:46.934799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras import Input, layers, Model\nfrom keras.layers import Dense\n\ndef get_model():\n #this function is used to get the keras functional model\n \n #setting up reuseability\n keras.backend.clear_session()\n tf.keras.backend.clear_session()\n tf.random.set_seed(42)\n np.random.seed(42)\n \n input_ = Input(shape=(X_train.shape[1])) #input shape where (batch_size, column numbers) we only specify column number here\n \n x = Dense(300,activation= 'LeakyReLU', kernel_initializer=\"he_normal\")(input_) \n x = Dense(100,activation= 'LeakyReLU', kernel_initializer=\"he_normal\")(x)\n \n output_ = Dense(y_train.shape[1], activation='softmax')(x)\n \n model = Model(inputs=[input_], outputs =[output_])\n \n return model\n","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:23:54.045493Z","iopub.execute_input":"2021-12-02T06:23:54.046046Z","iopub.status.idle":"2021-12-02T06:23:54.052693Z","shell.execute_reply.started":"2021-12-02T06:23:54.046004Z","shell.execute_reply":"2021-12-02T06:23:54.052005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n del model\nexcept:\n model = get_model()\n\nmodel.compile(loss=\"categorical_crossentropy\",\n optimizer=keras.optimizers.Adam(learning_rate=0.001),\n metrics=[\"acc\",]);\n\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:24:00.904615Z","iopub.execute_input":"2021-12-02T06:24:00.905485Z","iopub.status.idle":"2021-12-02T06:24:01.151141Z","shell.execute_reply.started":"2021-12-02T06:24:00.905435Z","shell.execute_reply":"2021-12-02T06:24:01.150458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train, y_train, epochs=10,batch_size=2024,\n validation_data=(X_test, y_test))","metadata":{"scrolled":true,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2021-12-02T06:24:08.22531Z","iopub.execute_input":"2021-12-02T06:24:08.225749Z","iopub.status.idle":"2021-12-02T06:25:34.039504Z","shell.execute_reply.started":"2021-12-02T06:24:08.225715Z","shell.execute_reply":"2021-12-02T06:25:34.038658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's try PReLU:","metadata":{}},{"cell_type":"code","source":"def get_model():\n #this function is used to get the keras functional model\n \n #setting up reuseability\n keras.backend.clear_session()\n tf.keras.backend.clear_session()\n tf.random.set_seed(42)\n np.random.seed(42)\n \n input_ = Input(shape=(X_train.shape[1])) #input shape where (batch_size, column numbers) we only specify column number here\n \n x = Dense(300,activation= 'PReLU', kernel_initializer=\"he_normal\")(input_) \n x = Dense(100,activation= 'PReLU', kernel_initializer=\"he_normal\")(x)\n \n output_ = Dense(y_train.shape[1], activation='softmax')(x)\n \n model = Model(inputs=[input_], outputs =[output_])\n \n return model","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:26:23.681227Z","iopub.execute_input":"2021-12-02T06:26:23.681499Z","iopub.status.idle":"2021-12-02T06:26:23.687691Z","shell.execute_reply.started":"2021-12-02T06:26:23.681458Z","shell.execute_reply":"2021-12-02T06:26:23.687005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = get_model()\n\nmodel.compile(loss=\"categorical_crossentropy\",\n optimizer=keras.optimizers.Adam(learning_rate=0.001),\n metrics=[\"acc\",]);","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:26:31.555179Z","iopub.execute_input":"2021-12-02T06:26:31.555679Z","iopub.status.idle":"2021-12-02T06:26:31.603749Z","shell.execute_reply.started":"2021-12-02T06:26:31.555615Z","shell.execute_reply":"2021-12-02T06:26:31.603051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train, y_train, epochs=10,batch_size=2024,\n validation_data=(X_test, y_test))\n\ndel history, model\n\ngc.collect()","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2021-12-02T06:26:32.498285Z","iopub.execute_input":"2021-12-02T06:26:32.498754Z","iopub.status.idle":"2021-12-02T06:27:58.155775Z","shell.execute_reply.started":"2021-12-02T06:26:32.498713Z","shell.execute_reply":"2021-12-02T06:27:58.15498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
ELU
","metadata":{}},{"cell_type":"code","source":"def elu(z, alpha=1):\n return np.where(z < 0, alpha * (np.exp(z) - 1), z)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:28:01.357805Z","iopub.execute_input":"2021-12-02T06:28:01.358276Z","iopub.status.idle":"2021-12-02T06:28:01.362718Z","shell.execute_reply.started":"2021-12-02T06:28:01.358237Z","shell.execute_reply":"2021-12-02T06:28:01.361788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(z, elu(z), \"b-\", linewidth=2)\nplt.plot([-5, 5], [0, 0], 'k-')\nplt.plot([-5, 5], [-1, -1], 'k--')\nplt.plot([0, 0], [-2.2, 3.2], 'k-')\nplt.grid(True)\nplt.title(r\"ELU activation function ($\\alpha=1$)\", fontsize=14)\nplt.axis([-5, 5, -2.2, 3.2])\n\nsave_fig(\"elu_plot\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-02T06:28:01.599309Z","iopub.execute_input":"2021-12-02T06:28:01.599744Z","iopub.status.idle":"2021-12-02T06:28:02.284291Z","shell.execute_reply.started":"2021-12-02T06:28:01.599706Z","shell.execute_reply":"2021-12-02T06:28:02.283503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Implementing ELU in TensorFlow is trivial, just specify the activation function when building each layer:","metadata":{}},{"cell_type":"code","source":"keras.layers.Dense(10, activation=\"elu\")","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:28:10.058876Z","iopub.execute_input":"2021-12-02T06:28:10.059666Z","iopub.status.idle":"2021-12-02T06:28:10.06627Z","shell.execute_reply.started":"2021-12-02T06:28:10.059603Z","shell.execute_reply":"2021-12-02T06:28:10.065593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
SELU
","metadata":{}},{"cell_type":"markdown","source":"\n\nThis activation function was proposed in this great paper by Günter Klambauer, Thomas Unterthiner and Andreas Mayr, published in June 2017. During training, a neural network composed exclusively of a stack of dense layers using the SELU activation function and LeCun initialization will self-normalize: the output of each layer will tend to preserve the same mean and variance during training, which solves the vanishing/exploding gradients problem. As a result, this activation function outperforms the other activation functions very significantly for such neural nets, so you should really try it out. Unfortunately, the self-normalizing property of the SELU activation function is easily broken: you cannot use ℓ1 or ℓ2 regularization, regular dropout, max-norm, skip connections or other non-sequential topologies (so recurrent neural networks won't self-normalize). However, in practice it works quite well with sequential CNNs. If you break self-normalization, SELU will not necessarily outperform other activation functions.
","metadata":{}},{"cell_type":"code","source":"from scipy.special import erfc\n\n# alpha and scale to self normalize with mean 0 and standard deviation 1\n# (see equation 14 in the paper):\nalpha_0_1 = -np.sqrt(2 / np.pi) / (erfc(1/np.sqrt(2)) * np.exp(1/2) - 1)\nscale_0_1 = (1 - erfc(1 / np.sqrt(2)) * np.sqrt(np.e)) * np.sqrt(2 * np.pi) * (2 * erfc(np.sqrt(2))*np.e**2 + np.pi*erfc(1/np.sqrt(2))**2*np.e - 2*(2+np.pi)*erfc(1/np.sqrt(2))*np.sqrt(np.e)+np.pi+2)**(-1/2)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:28:17.028765Z","iopub.execute_input":"2021-12-02T06:28:17.029591Z","iopub.status.idle":"2021-12-02T06:28:17.039886Z","shell.execute_reply.started":"2021-12-02T06:28:17.02955Z","shell.execute_reply":"2021-12-02T06:28:17.038744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def selu(z, scale=scale_0_1, alpha=alpha_0_1):\n return scale * elu(z, alpha)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:28:17.261502Z","iopub.execute_input":"2021-12-02T06:28:17.262003Z","iopub.status.idle":"2021-12-02T06:28:17.266306Z","shell.execute_reply.started":"2021-12-02T06:28:17.261954Z","shell.execute_reply":"2021-12-02T06:28:17.265039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(z, selu(z), \"b-\", linewidth=2)\nplt.plot([-5, 5], [0, 0], 'k-')\nplt.plot([-5, 5], [-1.758, -1.758], 'k--')\nplt.plot([0, 0], [-2.2, 3.2], 'k-')\nplt.grid(True)\nplt.title(\"SELU activation function\", fontsize=14)\nplt.axis([-5, 5, -2.2, 3.2])\n\nsave_fig(\"selu_plot\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-02T06:28:18.266956Z","iopub.execute_input":"2021-12-02T06:28:18.267686Z","iopub.status.idle":"2021-12-02T06:28:18.744Z","shell.execute_reply.started":"2021-12-02T06:28:18.267607Z","shell.execute_reply":"2021-12-02T06:28:18.743045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\nBy default, the SELU hyperparameters (scale and alpha) are tuned in such a way that the mean output of each neuron remains close to 0, and the standard deviation remains close to 1 (assuming the inputs are standardized with mean 0 and standard deviation 1 too). Using this activation function, even a 1,000 layer deep neural network preserves roughly mean 0 and standard deviation 1 across all layers, avoiding the exploding/vanishing gradients problem:\n
","metadata":{}},{"cell_type":"code","source":"np.random.seed(42)\nZ = np.random.normal(size=(500, 100)) # standardized inputs\nfor layer in range(1000):\n W = np.random.normal(size=(100, 100), scale=np.sqrt(1 / 100)) # LeCun initialization\n Z = selu(np.dot(Z, W))\n means = np.mean(Z, axis=0).mean()\n stds = np.std(Z, axis=0).mean()\n if layer % 100 == 0:\n print(\"Layer {}: mean {:.2f}, std deviation {:.2f}\".format(layer, means, stds))","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:28:28.151061Z","iopub.execute_input":"2021-12-02T06:28:28.151325Z","iopub.status.idle":"2021-12-02T06:28:30.716909Z","shell.execute_reply.started":"2021-12-02T06:28:28.151295Z","shell.execute_reply":"2021-12-02T06:28:30.715963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using SELU is easy:","metadata":{}},{"cell_type":"code","source":"keras.layers.Dense(10, activation=\"selu\",\n kernel_initializer=\"lecun_normal\")","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:28:33.618209Z","iopub.execute_input":"2021-12-02T06:28:33.618696Z","iopub.status.idle":"2021-12-02T06:28:33.626308Z","shell.execute_reply.started":"2021-12-02T06:28:33.618657Z","shell.execute_reply":"2021-12-02T06:28:33.625474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Let's create a neural net for Fashion MNIST with 100 hidden layers, using the SELU activation function:","metadata":{}},{"cell_type":"code","source":"np.random.seed(42)\ntf.random.set_seed(42)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:28:37.082353Z","iopub.execute_input":"2021-12-02T06:28:37.082916Z","iopub.status.idle":"2021-12-02T06:28:37.08907Z","shell.execute_reply.started":"2021-12-02T06:28:37.082876Z","shell.execute_reply":"2021-12-02T06:28:37.088128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = keras.models.Sequential()\nmodel.add(keras.layers.Flatten(input_shape=[X_train.shape[1],]))\nmodel.add(keras.layers.Dense(300, activation=\"selu\",\n kernel_initializer=\"lecun_normal\"))\nfor layer in range(99):\n model.add(keras.layers.Dense(100, activation=\"selu\",\n kernel_initializer=\"lecun_normal\"))\nmodel.add(keras.layers.Dense(y_train.shape[1], activation=\"softmax\"))","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:29:02.936325Z","iopub.execute_input":"2021-12-02T06:29:02.936987Z","iopub.status.idle":"2021-12-02T06:29:03.754226Z","shell.execute_reply.started":"2021-12-02T06:29:02.936946Z","shell.execute_reply":"2021-12-02T06:29:03.753477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(loss=\"categorical_crossentropy\",\n optimizer=keras.optimizers.Adam(learning_rate=0.001),\n metrics=[\"acc\",]);","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:29:03.755835Z","iopub.execute_input":"2021-12-02T06:29:03.756059Z","iopub.status.idle":"2021-12-02T06:29:03.7665Z","shell.execute_reply.started":"2021-12-02T06:29:03.756027Z","shell.execute_reply":"2021-12-02T06:29:03.765657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Now let's train it. Do not forget to scale the inputs to mean 0 and standard deviation 1:","metadata":{}},{"cell_type":"code","source":"_means = X_train.values.mean(axis=0, keepdims=True)\n_stds = X_train.values.std(axis=0, keepdims=True)\nX_train_scaled = (X_train - _means) / _stds\nX_test_scaled = (X_test - _means) / _stds","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:29:09.631144Z","iopub.execute_input":"2021-12-02T06:29:09.631691Z","iopub.status.idle":"2021-12-02T06:29:11.352683Z","shell.execute_reply.started":"2021-12-02T06:29:09.631648Z","shell.execute_reply":"2021-12-02T06:29:11.351914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train_scaled, y_train, epochs=5,batch_size=1024,\n validation_data=(X_test_scaled, y_test))\n\ndel model, history\ngc.collect()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-02T06:29:11.354133Z","iopub.execute_input":"2021-12-02T06:29:11.354383Z","iopub.status.idle":"2021-12-02T06:33:10.175131Z","shell.execute_reply.started":"2021-12-02T06:29:11.354348Z","shell.execute_reply":"2021-12-02T06:33:10.174387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Now look at what happens if we try to use the ReLU activation function instead:","metadata":{}},{"cell_type":"code","source":"np.random.seed(42)\ntf.random.set_seed(42)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:35:58.388127Z","iopub.execute_input":"2021-12-02T06:35:58.388446Z","iopub.status.idle":"2021-12-02T06:35:58.423036Z","shell.execute_reply.started":"2021-12-02T06:35:58.388412Z","shell.execute_reply":"2021-12-02T06:35:58.421822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = keras.models.Sequential()\nmodel.add(keras.layers.Flatten(input_shape=[X_train.shape[1],]))\nmodel.add(keras.layers.Dense(300, activation=\"relu\", kernel_initializer=\"he_normal\"))\nfor layer in range(99):\n model.add(keras.layers.Dense(100, activation=\"relu\", kernel_initializer=\"he_normal\"))\nmodel.add(keras.layers.Dense(y_train.shape[1], activation=\"softmax\"))","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:35:58.594013Z","iopub.execute_input":"2021-12-02T06:35:58.594548Z","iopub.status.idle":"2021-12-02T06:35:59.449079Z","shell.execute_reply.started":"2021-12-02T06:35:58.59451Z","shell.execute_reply":"2021-12-02T06:35:59.448328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(loss=\"categorical_crossentropy\",\n optimizer=keras.optimizers.Adam(learning_rate=0.001),\n metrics=[\"acc\",]);","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:35:59.45055Z","iopub.execute_input":"2021-12-02T06:35:59.450809Z","iopub.status.idle":"2021-12-02T06:35:59.46146Z","shell.execute_reply.started":"2021-12-02T06:35:59.450776Z","shell.execute_reply":"2021-12-02T06:35:59.460807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train_scaled, y_train, epochs=5,batch_size=1024,\n validation_data=(X_test_scaled, y_test))\n\ndel model, history\ngc.collect()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-02T06:35:59.4628Z","iopub.execute_input":"2021-12-02T06:35:59.463067Z","iopub.status.idle":"2021-12-02T06:40:28.115192Z","shell.execute_reply.started":"2021-12-02T06:35:59.463034Z","shell.execute_reply":"2021-12-02T06:40:28.114428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Not great at all, we suffered from the vanishing/exploding gradients problem.","metadata":{}},{"cell_type":"markdown","source":"
Sometimes applying BN before the activation function works better (there's a debate on this topic). Moreover, the layer before a BatchNormalization layer does not need to have bias terms, since the BatchNormalization layer some as well, it would be a waste of parameters, so you can set use_bias=False when creating those layers:","metadata":{}},{"cell_type":"code","source":"model = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.BatchNormalization(),\n keras.layers.Dense(300, use_bias=False),\n keras.layers.BatchNormalization(),\n keras.layers.Activation(\"relu\"),\n keras.layers.Dense(100, use_bias=False),\n keras.layers.BatchNormalization(),\n keras.layers.Activation(\"relu\"),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:43:54.375258Z","iopub.execute_input":"2021-12-02T06:43:54.375691Z","iopub.status.idle":"2021-12-02T06:43:54.440137Z","shell.execute_reply.started":"2021-12-02T06:43:54.37565Z","shell.execute_reply":"2021-12-02T06:43:54.439483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(loss=\"categorical_crossentropy\",\n optimizer=keras.optimizers.Adam(learning_rate=0.001),\n metrics=[\"acc\",]);","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:43:54.44134Z","iopub.execute_input":"2021-12-02T06:43:54.441576Z","iopub.status.idle":"2021-12-02T06:43:54.449547Z","shell.execute_reply.started":"2021-12-02T06:43:54.441551Z","shell.execute_reply":"2021-12-02T06:43:54.448886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train, y_train, epochs=10,batch_size=1024,\n validation_data=(X_test, y_test))\n\ndel model, history\ngc.collect()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-02T06:43:54.450789Z","iopub.execute_input":"2021-12-02T06:43:54.451206Z","iopub.status.idle":"2021-12-02T06:46:23.978964Z","shell.execute_reply.started":"2021-12-02T06:43:54.451169Z","shell.execute_reply":"2021-12-02T06:46:23.978273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
All Keras optimizers accept clipnorm or clipvalue arguments:\n \n Gradient clipping is a technique to prevent exploding gradients in very deep networks, usually in recurrent neural networks. ... With gradient clipping, pre-determined gradient threshold be introduced, and then gradients norms that exceed this threshold are scaled down to match the norm.","metadata":{}},{"cell_type":"code","source":"optimizer = keras.optimizers.SGD(clipvalue=1.0)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:46:23.980135Z","iopub.execute_input":"2021-12-02T06:46:23.98046Z","iopub.status.idle":"2021-12-02T06:46:23.985154Z","shell.execute_reply.started":"2021-12-02T06:46:23.980421Z","shell.execute_reply":"2021-12-02T06:46:23.984379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"optimizer = keras.optimizers.SGD(clipnorm=1.0)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:46:23.986218Z","iopub.execute_input":"2021-12-02T06:46:23.987021Z","iopub.status.idle":"2021-12-02T06:46:23.997756Z","shell.execute_reply.started":"2021-12-02T06:46:23.98698Z","shell.execute_reply":"2021-12-02T06:46:23.996977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\nLet's split the fashion MNIST training set in two:\n
Note that model_B_on_A and model_A actually share layers now, so when we train one, it will update both models. If we want to avoid that, we need to build model_B_on_A on top of a clone of model_A:","metadata":{}},{"cell_type":"code","source":"model_A_clone = keras.models.clone_model(model_A)\nmodel_A_clone.set_weights(model_A.get_weights())\nmodel_B_on_A = keras.models.Sequential(model_A_clone.layers[:-1])\nmodel_B_on_A.add(keras.layers.Dense(1, activation=\"sigmoid\"))","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:50.593938Z","iopub.execute_input":"2021-12-02T06:47:50.59417Z","iopub.status.idle":"2021-12-02T06:47:50.678593Z","shell.execute_reply.started":"2021-12-02T06:47:50.594137Z","shell.execute_reply":"2021-12-02T06:47:50.677967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for layer in model_B_on_A.layers[:-1]:\n layer.trainable = False\n\nmodel_B_on_A.compile(loss=\"categorical_crossentropy\",\n optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n metrics=[\"accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:50.679826Z","iopub.execute_input":"2021-12-02T06:47:50.680072Z","iopub.status.idle":"2021-12-02T06:47:50.689405Z","shell.execute_reply.started":"2021-12-02T06:47:50.680038Z","shell.execute_reply":"2021-12-02T06:47:50.688676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model_B_on_A.fit(X_train_B, y_train_B, epochs=4,\n validation_data=(X_valid_B, y_valid_B))\n\nfor layer in model_B_on_A.layers[:-1]:\n layer.trainable = True\n\nmodel_B_on_A.compile(loss=\"categorical_crossentropy\",\n optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n metrics=[\"accuracy\"])\nhistory = model_B_on_A.fit(X_train_B, y_train_B, epochs=16,\n validation_data=(X_valid_B, y_valid_B))","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-02T06:47:50.69054Z","iopub.execute_input":"2021-12-02T06:47:50.691172Z","iopub.status.idle":"2021-12-02T06:47:55.1278Z","shell.execute_reply.started":"2021-12-02T06:47:50.691131Z","shell.execute_reply":"2021-12-02T06:47:55.126994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
So, what's the final verdict?","metadata":{}},{"cell_type":"code","source":"model_B.evaluate(X_test_B, y_test_B)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:55.130875Z","iopub.execute_input":"2021-12-02T06:47:55.131385Z","iopub.status.idle":"2021-12-02T06:47:55.302306Z","shell.execute_reply.started":"2021-12-02T06:47:55.131352Z","shell.execute_reply":"2021-12-02T06:47:55.301668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_B_on_A.evaluate(X_test_B, y_test_B)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:55.303535Z","iopub.execute_input":"2021-12-02T06:47:55.303875Z","iopub.status.idle":"2021-12-02T06:47:55.494141Z","shell.execute_reply.started":"2021-12-02T06:47:55.303838Z","shell.execute_reply":"2021-12-02T06:47:55.493451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Great! We got quite a bit of transfer: the error rate dropped by a factor of 4.9!","metadata":{}},{"cell_type":"code","source":"(100 - 97.05) / (100 - 99.40)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:55.495375Z","iopub.execute_input":"2021-12-02T06:47:55.495622Z","iopub.status.idle":"2021-12-02T06:47:55.50066Z","shell.execute_reply.started":"2021-12-02T06:47:55.495589Z","shell.execute_reply":"2021-12-02T06:47:55.499908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Momentum optimization
","metadata":{}},{"cell_type":"code","source":"optimizer = keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:55.502064Z","iopub.execute_input":"2021-12-02T06:47:55.502486Z","iopub.status.idle":"2021-12-02T06:47:55.510652Z","shell.execute_reply.started":"2021-12-02T06:47:55.502448Z","shell.execute_reply":"2021-12-02T06:47:55.509933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nesterov Accelerated Gradient","metadata":{}},{"cell_type":"code","source":"optimizer = keras.optimizers.SGD(learning_rate=0.001, momentum=0.9, nesterov=True)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:55.512238Z","iopub.execute_input":"2021-12-02T06:47:55.512721Z","iopub.status.idle":"2021-12-02T06:47:55.522466Z","shell.execute_reply.started":"2021-12-02T06:47:55.512685Z","shell.execute_reply":"2021-12-02T06:47:55.521652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" AdaGrad","metadata":{}},{"cell_type":"code","source":"optimizer = keras.optimizers.Adagrad(learning_rate=0.001)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:55.523721Z","iopub.execute_input":"2021-12-02T06:47:55.524038Z","iopub.status.idle":"2021-12-02T06:47:55.531196Z","shell.execute_reply.started":"2021-12-02T06:47:55.524004Z","shell.execute_reply":"2021-12-02T06:47:55.530439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" RMSProp","metadata":{}},{"cell_type":"code","source":"optimizer = keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:55.532463Z","iopub.execute_input":"2021-12-02T06:47:55.532768Z","iopub.status.idle":"2021-12-02T06:47:55.540102Z","shell.execute_reply.started":"2021-12-02T06:47:55.532731Z","shell.execute_reply":"2021-12-02T06:47:55.539322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Adam Optimization","metadata":{}},{"cell_type":"code","source":"optimizer = keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:55.543124Z","iopub.execute_input":"2021-12-02T06:47:55.543308Z","iopub.status.idle":"2021-12-02T06:47:55.549519Z","shell.execute_reply.started":"2021-12-02T06:47:55.543286Z","shell.execute_reply":"2021-12-02T06:47:55.548734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Adamax Optimization","metadata":{}},{"cell_type":"code","source":"optimizer = keras.optimizers.Adamax(learning_rate=0.001, beta_1=0.9, beta_2=0.999)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:55.550867Z","iopub.execute_input":"2021-12-02T06:47:55.551132Z","iopub.status.idle":"2021-12-02T06:47:55.558058Z","shell.execute_reply.started":"2021-12-02T06:47:55.551098Z","shell.execute_reply":"2021-12-02T06:47:55.557298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Nadam Optimization","metadata":{}},{"cell_type":"code","source":"optimizer = keras.optimizers.Nadam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:47:55.55917Z","iopub.execute_input":"2021-12-02T06:47:55.559719Z","iopub.status.idle":"2021-12-02T06:47:55.566715Z","shell.execute_reply.started":"2021-12-02T06:47:55.559609Z","shell.execute_reply":"2021-12-02T06:47:55.565913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Learning Rate Scheduling","metadata":{}},{"cell_type":"markdown","source":" Power Scheduling","metadata":{}},{"cell_type":"markdown","source":"```lr = lr0 / (1 + steps / s)**c```\n* Keras uses `c=1` and `s = 1 / decay`","metadata":{}},{"cell_type":"code","source":"optimizer = keras.optimizers.SGD(learning_rate=0.01, decay=1e-4)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:48:48.242942Z","iopub.execute_input":"2021-12-02T06:48:48.243722Z","iopub.status.idle":"2021-12-02T06:48:48.248663Z","shell.execute_reply.started":"2021-12-02T06:48:48.243661Z","shell.execute_reply":"2021-12-02T06:48:48.247667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])\nmodel.compile(loss=\"categorical_crossentropy\",\n optimizer=optimizer,\n metrics=[\"acc\",]);","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:48:48.483579Z","iopub.execute_input":"2021-12-02T06:48:48.48423Z","iopub.status.idle":"2021-12-02T06:48:48.520107Z","shell.execute_reply.started":"2021-12-02T06:48:48.484191Z","shell.execute_reply":"2021-12-02T06:48:48.519468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_epochs = 20\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,batch_size=1024,\n validation_data=(X_test_scaled, y_test))\n\ndel model\ngc.collect()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-02T06:48:51.952002Z","iopub.execute_input":"2021-12-02T06:48:51.95295Z","iopub.status.idle":"2021-12-02T06:52:00.106285Z","shell.execute_reply.started":"2021-12-02T06:48:51.952898Z","shell.execute_reply":"2021-12-02T06:52:00.105584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math\n\nlearning_rate = 0.01\ndecay = 1e-4\nbatch_size = 32\nn_steps_per_epoch = math.ceil(len(X_train) / batch_size)\nepochs = np.arange(n_epochs)\nlrs = learning_rate / (1 + decay * epochs * n_steps_per_epoch)\n\nplt.plot(epochs, lrs, \"o-\")\nplt.axis([0, n_epochs - 1, 0, 0.01])\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Learning Rate\")\nplt.title(\"Power Scheduling\", fontsize=14)\nplt.grid(True)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-02T06:52:00.107875Z","iopub.execute_input":"2021-12-02T06:52:00.10813Z","iopub.status.idle":"2021-12-02T06:52:00.305093Z","shell.execute_reply.started":"2021-12-02T06:52:00.108096Z","shell.execute_reply":"2021-12-02T06:52:00.304446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Exponential Scheduling","metadata":{}},{"cell_type":"markdown","source":"```lr = lr0 * 0.1**(epoch / s)```","metadata":{}},{"cell_type":"code","source":"def exponential_decay_fn(epoch):\n return 0.01 * 0.1**(epoch / 20)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:52:00.306174Z","iopub.execute_input":"2021-12-02T06:52:00.306417Z","iopub.status.idle":"2021-12-02T06:52:00.310589Z","shell.execute_reply.started":"2021-12-02T06:52:00.306383Z","shell.execute_reply":"2021-12-02T06:52:00.30988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def exponential_decay(lr0, s):\n def exponential_decay_fn(epoch):\n return lr0 * 0.1**(epoch / s)\n return exponential_decay_fn\n\nexponential_decay_fn = exponential_decay(lr0=0.01, s=20)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:52:00.312467Z","iopub.execute_input":"2021-12-02T06:52:00.312911Z","iopub.status.idle":"2021-12-02T06:52:00.32071Z","shell.execute_reply.started":"2021-12-02T06:52:00.312873Z","shell.execute_reply":"2021-12-02T06:52:00.319914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])\n\nmodel.compile(loss=\"categorical_crossentropy\",\n optimizer=\"nadam\",\n metrics=[\"acc\",]);\nn_epochs = 20","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:52:00.322419Z","iopub.execute_input":"2021-12-02T06:52:00.323026Z","iopub.status.idle":"2021-12-02T06:52:00.359139Z","shell.execute_reply.started":"2021-12-02T06:52:00.322983Z","shell.execute_reply":"2021-12-02T06:52:00.358502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lr_scheduler = keras.callbacks.LearningRateScheduler(exponential_decay_fn)\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,batch_size=1024,\n validation_data=(X_test_scaled, y_test),\n callbacks=[lr_scheduler])\n\ndel model\ngc.collect()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-02T06:52:00.360207Z","iopub.execute_input":"2021-12-02T06:52:00.360538Z","iopub.status.idle":"2021-12-02T06:56:00.051524Z","shell.execute_reply.started":"2021-12-02T06:52:00.360467Z","shell.execute_reply":"2021-12-02T06:56:00.050804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history.epoch, history.history[\"lr\"], \"o-\")\nplt.axis([0, n_epochs - 1, 0, 0.011])\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Learning Rate\")\nplt.title(\"Exponential Scheduling\", fontsize=14)\nplt.grid(True)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-02T06:56:00.052764Z","iopub.execute_input":"2021-12-02T06:56:00.053096Z","iopub.status.idle":"2021-12-02T06:56:00.252127Z","shell.execute_reply.started":"2021-12-02T06:56:00.053056Z","shell.execute_reply":"2021-12-02T06:56:00.251445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" The schedule function can take the current learning rate as a second argument:","metadata":{}},{"cell_type":"code","source":"def exponential_decay_fn(epoch, lr):\n return lr * 0.1**(1 / 20)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:56:00.253444Z","iopub.execute_input":"2021-12-02T06:56:00.253902Z","iopub.status.idle":"2021-12-02T06:56:00.257526Z","shell.execute_reply.started":"2021-12-02T06:56:00.253865Z","shell.execute_reply":"2021-12-02T06:56:00.256876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" If you want to update the learning rate at each iteration rather than at each epoch, you must write your own callback class:","metadata":{}},{"cell_type":"code","source":"K = keras.backend\n\nclass ExponentialDecay(keras.callbacks.Callback):\n def __init__(self, s=40000):\n super().__init__()\n self.s = s\n\n def on_batch_begin(self, batch, logs=None):\n # Note: the `batch` argument is reset at each epoch\n lr = K.get_value(self.model.optimizer.learning_rate)\n K.set_value(self.model.optimizer.learning_rate, lr * 0.1**(1 / s))\n\n def on_epoch_end(self, epoch, logs=None):\n logs = logs or {}\n logs['lr'] = K.get_value(self.model.optimizer.learning_rate)\n\nmodel = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])\nlr0 = 0.01\noptimizer = keras.optimizers.Nadam(learning_rate=lr0)\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=optimizer, metrics=[\"acc\",])\nn_epochs = 25\n\ns = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\nexp_decay = ExponentialDecay(s)\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,batch_size=1024,\n validation_data=(X_test_scaled, y_test),\n callbacks=[exp_decay])\n\ndel model\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T06:56:00.258809Z","iopub.execute_input":"2021-12-02T06:56:00.259273Z","iopub.status.idle":"2021-12-02T07:02:26.20932Z","shell.execute_reply.started":"2021-12-02T06:56:00.259237Z","shell.execute_reply":"2021-12-02T07:02:26.208565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_steps = n_epochs * len(X_train) // 32\nsteps = np.arange(n_steps)\nlrs = lr0 * 0.1**(steps / s)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T07:02:26.212498Z","iopub.execute_input":"2021-12-02T07:02:26.212716Z","iopub.status.idle":"2021-12-02T07:02:26.371018Z","shell.execute_reply.started":"2021-12-02T07:02:26.21269Z","shell.execute_reply":"2021-12-02T07:02:26.370168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(steps, lrs, \"-\", linewidth=2)\nplt.axis([0, n_steps - 1, 0, lr0 * 1.1])\nplt.xlabel(\"Batch\")\nplt.ylabel(\"Learning Rate\")\nplt.title(\"Exponential Scheduling (per batch)\", fontsize=14)\nplt.grid(True)\nplt.show()","metadata":{"scrolled":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-02T07:02:26.372392Z","iopub.execute_input":"2021-12-02T07:02:26.373181Z","iopub.status.idle":"2021-12-02T07:02:26.794267Z","shell.execute_reply.started":"2021-12-02T07:02:26.373131Z","shell.execute_reply":"2021-12-02T07:02:26.793622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Piecewise Constant Scheduling","metadata":{}},{"cell_type":"code","source":"def piecewise_constant_fn(epoch):\n if epoch < 5:\n return 0.01\n elif epoch < 15:\n return 0.005\n else:\n return 0.001","metadata":{"execution":{"iopub.status.busy":"2021-12-02T07:02:26.79543Z","iopub.execute_input":"2021-12-02T07:02:26.796799Z","iopub.status.idle":"2021-12-02T07:02:26.801508Z","shell.execute_reply.started":"2021-12-02T07:02:26.796756Z","shell.execute_reply":"2021-12-02T07:02:26.800816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def piecewise_constant(boundaries, values):\n boundaries = np.array([0] + boundaries)\n values = np.array(values)\n def piecewise_constant_fn(epoch):\n return values[np.argmax(boundaries > epoch) - 1]\n return piecewise_constant_fn\n\npiecewise_constant_fn = piecewise_constant([5, 15], [0.01, 0.005, 0.001])","metadata":{"execution":{"iopub.status.busy":"2021-12-02T07:02:26.802772Z","iopub.execute_input":"2021-12-02T07:02:26.803052Z","iopub.status.idle":"2021-12-02T07:02:26.811028Z","shell.execute_reply.started":"2021-12-02T07:02:26.80301Z","shell.execute_reply":"2021-12-02T07:02:26.810286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lr_scheduler = keras.callbacks.LearningRateScheduler(piecewise_constant_fn)\n\nmodel = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"acc\",])\nn_epochs = 25\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,batch_size=1024,\n validation_data=(X_test_scaled, y_test),\n callbacks=[lr_scheduler])\n\ndel model\ngc.collect()","metadata":{"_kg_hide-output":true,"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-02T07:02:26.812367Z","iopub.execute_input":"2021-12-02T07:02:26.81271Z","iopub.status.idle":"2021-12-02T07:07:52.876353Z","shell.execute_reply.started":"2021-12-02T07:02:26.812672Z","shell.execute_reply":"2021-12-02T07:07:52.875571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history.epoch, [piecewise_constant_fn(epoch) for epoch in history.epoch], \"o-\")\nplt.axis([0, n_epochs - 1, 0, 0.011])\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Learning Rate\")\nplt.title(\"Piecewise Constant Scheduling\", fontsize=14)\nplt.grid(True)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-02T07:07:52.877871Z","iopub.execute_input":"2021-12-02T07:07:52.878136Z","iopub.status.idle":"2021-12-02T07:07:53.078236Z","shell.execute_reply.started":"2021-12-02T07:07:52.878101Z","shell.execute_reply":"2021-12-02T07:07:53.0775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Performance Scheduling","metadata":{}},{"cell_type":"code","source":"tf.random.set_seed(42)\nnp.random.seed(42)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T07:07:53.079341Z","iopub.execute_input":"2021-12-02T07:07:53.079592Z","iopub.status.idle":"2021-12-02T07:07:53.098857Z","shell.execute_reply.started":"2021-12-02T07:07:53.079557Z","shell.execute_reply":"2021-12-02T07:07:53.098078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lr_scheduler = keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)\n\nmodel = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])\noptimizer = keras.optimizers.SGD(learning_rate=0.02, momentum=0.9)\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=optimizer, metrics=[\"acc\",])\nn_epochs = 25\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n validation_data=(X_test_scaled, y_test),\n callbacks=[lr_scheduler])\n\ndel model\ngc.collect()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-02T07:07:53.100223Z","iopub.execute_input":"2021-12-02T07:07:53.101009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history.epoch, history.history[\"lr\"], \"bo-\")\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Learning Rate\", color='b')\nplt.tick_params('y', colors='b')\nplt.gca().set_xlim(0, n_epochs - 1)\nplt.grid(True)\n\nax2 = plt.gca().twinx()\nax2.plot(history.epoch, history.history[\"val_loss\"], \"r^-\")\nax2.set_ylabel('Validation Loss', color='r')\nax2.tick_params('y', colors='r')\n\nplt.title(\"Reduce LR on Plateau\", fontsize=14)\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" tf.keras schedulers","metadata":{}},{"cell_type":"code","source":"model = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])\ns = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\nlearning_rate = keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1)\noptimizer = keras.optimizers.SGD(learning_rate)\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=optimizer, metrics=[\"AUC\"])\nn_epochs = 25\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,batch_size=1024,\n validation_data=(X_test_scaled, y_test))\ndel model\ngc.collect()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" For piecewise constant scheduling, try this:","metadata":{}},{"cell_type":"code","source":"learning_rate = keras.optimizers.schedules.PiecewiseConstantDecay(\n boundaries=[5. * n_steps_per_epoch, 15. * n_steps_per_epoch],\n values=[0.01, 0.005, 0.001])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 1Cycle scheduling","metadata":{}},{"cell_type":"code","source":"K = keras.backend\n\nclass ExponentialLearningRate(keras.callbacks.Callback):\n def __init__(self, factor):\n self.factor = factor\n self.rates = []\n self.losses = []\n def on_batch_end(self, batch, logs):\n self.rates.append(K.get_value(self.model.optimizer.learning_rate))\n self.losses.append(logs[\"loss\"])\n K.set_value(self.model.optimizer.learning_rate, self.model.optimizer.learning_rate * self.factor)\n\ndef find_learning_rate(model, X, y, epochs=1, batch_size=32, min_rate=10**-5, max_rate=10):\n init_weights = model.get_weights()\n iterations = math.ceil(len(X) / batch_size) * epochs\n factor = np.exp(np.log(max_rate / min_rate) / iterations)\n init_lr = K.get_value(model.optimizer.learning_rate)\n K.set_value(model.optimizer.learning_rate, min_rate)\n exp_lr = ExponentialLearningRate(factor)\n history = model.fit(X, y, epochs=epochs, batch_size=batch_size,\n callbacks=[exp_lr])\n K.set_value(model.optimizer.learning_rate, init_lr)\n model.set_weights(init_weights)\n return exp_lr.rates, exp_lr.losses\n\ndef plot_lr_vs_loss(rates, losses):\n plt.plot(rates, losses)\n plt.gca().set_xscale('log')\n plt.hlines(min(losses), min(rates), max(rates))\n plt.axis([min(rates), max(rates), min(losses), (losses[0] + min(losses)) / 2])\n plt.xlabel(\"Learning rate\")\n plt.ylabel(\"Loss\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Warning**: In the `on_batch_end()` method, `logs[\"loss\"]` used to contain the batch loss, but in TensorFlow 2.2.0 it was replaced with the mean loss (since the start of the epoch). This explains why the graph below is much smoother than in the book (if you are using TF 2.2 or above). It also means that there is a lag between the moment the batch loss starts exploding and the moment the explosion becomes clear in the graph. So you should choose a slightly smaller learning rate than you would have chosen with the \"noisy\" graph. Alternatively, you can tweak the `ExponentialLearningRate` callback above so it computes the batch loss (based on the current mean loss and the previous mean loss):\n\n```python\nclass ExponentialLearningRate(keras.callbacks.Callback):\n def __init__(self, factor):\n self.factor = factor\n self.rates = []\n self.losses = []\n def on_epoch_begin(self, epoch, logs=None):\n self.prev_loss = 0\n def on_batch_end(self, batch, logs=None):\n batch_loss = logs[\"loss\"] * (batch + 1) - self.prev_loss * batch\n self.prev_loss = logs[\"loss\"]\n self.rates.append(K.get_value(self.model.optimizer.learning_rate))\n self.losses.append(batch_loss)\n K.set_value(self.model.optimizer.learning_rate, self.model.optimizer.learning_rate * self.factor)\n```","metadata":{}},{"cell_type":"code","source":"tf.random.set_seed(42)\nnp.random.seed(42)\n\nmodel = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])\nmodel.compile(loss=\"categorical_crossentropy\",\n optimizer=keras.optimizers.SGD(learning_rate=1e-3),\n metrics=[\"acc\",])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_size = 1024\nrates, losses = find_learning_rate(model, X_train_scaled, y_train, epochs=1, batch_size=batch_size)\nplot_lr_vs_loss(rates, losses)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class OneCycleScheduler(keras.callbacks.Callback):\n def __init__(self, iterations, max_rate, start_rate=None,\n last_iterations=None, last_rate=None):\n self.iterations = iterations\n self.max_rate = max_rate\n self.start_rate = start_rate or max_rate / 10\n self.last_iterations = last_iterations or iterations // 10 + 1\n self.half_iteration = (iterations - self.last_iterations) // 2\n self.last_rate = last_rate or self.start_rate / 1000\n self.iteration = 0\n def _interpolate(self, iter1, iter2, rate1, rate2):\n return ((rate2 - rate1) * (self.iteration - iter1)\n / (iter2 - iter1) + rate1)\n def on_batch_begin(self, batch, logs):\n if self.iteration < self.half_iteration:\n rate = self._interpolate(0, self.half_iteration, self.start_rate, self.max_rate)\n elif self.iteration < 2 * self.half_iteration:\n rate = self._interpolate(self.half_iteration, 2 * self.half_iteration,\n self.max_rate, self.start_rate)\n else:\n rate = self._interpolate(2 * self.half_iteration, self.iterations,\n self.start_rate, self.last_rate)\n self.iteration += 1\n K.set_value(self.model.optimizer.learning_rate, rate)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_epochs = 25\nonecycle = OneCycleScheduler(math.ceil(len(X_train) / batch_size) * n_epochs, max_rate=0.05)\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs, batch_size=batch_size,\n validation_data=(X_test_scaled, y_test),\n callbacks=[onecycle])\n\ndel model\ngc.collect()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" $\\ell_1$ and $\\ell_2$ regularization","metadata":{}},{"cell_type":"code","source":"layer = keras.layers.Dense(100, activation=\"elu\",\n kernel_initializer=\"he_normal\",\n kernel_regularizer=keras.regularizers.l2(0.01))\n# or l1(0.1) for ℓ1 regularization with a factor of 0.1\n# or l1_l2(0.1, 0.01) for both ℓ1 and ℓ2 regularization, with factors 0.1 and 0.01 respectively","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.Dense(300, activation=\"elu\",\n kernel_initializer=\"he_normal\",\n kernel_regularizer=keras.regularizers.l2(0.01)),\n keras.layers.Dense(100, activation=\"elu\",\n kernel_initializer=\"he_normal\",\n kernel_regularizer=keras.regularizers.l2(0.01)),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\",\n kernel_regularizer=keras.regularizers.l2(0.01))\n])\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"acc\",])\nn_epochs = 2\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,batch_size=1024,\n validation_data=(X_test_scaled, y_test))\n\ndel model\ngc.collect()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from functools import partial\n\nRegularizedDense = partial(keras.layers.Dense,\n activation=\"elu\",\n kernel_initializer=\"he_normal\",\n kernel_regularizer=keras.regularizers.l2(0.01))\n\nmodel = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n RegularizedDense(300),\n RegularizedDense(100),\n RegularizedDense(y_train.shape[1], activation=\"softmax\")\n])\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"acc\",])\nn_epochs = 2\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,batch_size=1024,\n validation_data=(X_test_scaled, y_test))\n\ndel model\ngc.collect()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Dropout","metadata":{}},{"cell_type":"code","source":"model = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.Dropout(rate=0.2),\n keras.layers.Dense(300, activation=\"elu\", kernel_initializer=\"he_normal\"),\n keras.layers.Dropout(rate=0.2),\n keras.layers.Dense(100, activation=\"elu\", kernel_initializer=\"he_normal\"),\n keras.layers.Dropout(rate=0.2),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"acc\",])\nn_epochs = 2\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,batch_size=1024,\n validation_data=(X_test_scaled, y_test))\n\ndel model\ngc.collect()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Alpha Dropout","metadata":{}},{"cell_type":"code","source":"tf.random.set_seed(42)\nnp.random.seed(42)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n keras.layers.AlphaDropout(rate=0.2),\n keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.AlphaDropout(rate=0.2),\n keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n keras.layers.AlphaDropout(rate=0.2),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])\noptimizer = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9, nesterov=True)\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=optimizer, metrics=[\"acc\",])\nn_epochs = 20\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,batch_size=1024,\n validation_data=(X_test_scaled, y_test))\n\ngc.collect()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.evaluate(X_test_scaled, y_test)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.evaluate(X_train_scaled, y_train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train_scaled, y_train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" MC Dropout","metadata":{}},{"cell_type":"code","source":"tf.random.set_seed(42)\nnp.random.seed(42)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_probas = np.stack([model(X_test_scaled.values, training=True)\n for sample in range(100)])\ny_proba = y_probas.mean(axis=0)\ny_std = y_probas.std(axis=0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.round(model.predict(X_test_scaled[:1]), 2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.round(y_probas[:, :1], 2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.round(y_proba[:1], 2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_std = y_probas.std(axis=0)\nnp.round(y_std[:1], 2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = np.argmax(y_proba, axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class MCDropout(keras.layers.Dropout):\n def call(self, inputs):\n return super().call(inputs, training=True)\n\nclass MCAlphaDropout(keras.layers.AlphaDropout):\n def call(self, inputs):\n return super().call(inputs, training=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.random.set_seed(42)\nnp.random.seed(42)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mc_model = keras.models.Sequential([\n MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout) else layer\n for layer in model.layers\n])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mc_model.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"optimizer = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9, nesterov=True)\nmc_model.compile(loss=\"categorical_crossentropy\", optimizer=optimizer, metrics=[\"acc\",])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mc_model.set_weights(model.get_weights())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can use the model with MC Dropout:","metadata":{}},{"cell_type":"code","source":"np.round(np.mean([mc_model.predict(X_test_scaled[:1]) for sample in range(100)], axis=0), 2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Max norm","metadata":{}},{"cell_type":"code","source":"layer = keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n kernel_constraint=keras.constraints.max_norm(1.))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MaxNormDense = partial(keras.layers.Dense,\n activation=\"selu\", kernel_initializer=\"lecun_normal\",\n kernel_constraint=keras.constraints.max_norm(1.))\n\nmodel = keras.models.Sequential([\n keras.layers.Flatten(input_shape=[X_train.shape[1],]),\n MaxNormDense(300),\n MaxNormDense(100),\n keras.layers.Dense(y_train.shape[1], activation=\"softmax\")\n])\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"acc\",])\nn_epochs = 2\nhistory = model.fit(X_train_scaled, y_train, epochs=n_epochs,batch_size=1024,\n validation_data=(X_test_scaled, y_test))\n\ndel model\ngc.collect()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This notebook is forked from **Hands on machine learning with sklearn, tensorflow and keras book** github repo please check out the original author's work.([link](https://github.com/ageron/handson-ml2))\n\nThank you.\n\n\n","metadata":{}}]}Avoiding Overfitting Through Regularization
","metadata":{}},{"cell_type":"markdown","source":"