{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"markdown","source":"## INTRODUCTION\n\nAs you may know, Kaggle kernels have a **limit for how much time they can keep running**.   \nSometimes, it's enough time to train a descent model, but sometimes it isn't. \n\nIf you have a big model, using very big data, it's possible that Kaggle's kernels available time is not enough for you. (Even if you're using a GPU)\n\nFor that reason, this kernel brings you a simple way to continuously train a Keras model for the time you want.   \nThis allows you to commit and run your kernel without worrying whether it will be interrupted for exceeding execution time or not. \n\n## Creating a custom callback\n\nHere, we are going to create a custom callback, called `TimerCallback`, in order to manage training time and eventually interrupt it.\n\nA `Callback` in Keras is an object that you pass either to the `model.fit` or `model.fit_generator` methods, in order to execute additional commands between batches, epochs, etc.   \nThere are several kinds of ready-to-use callbacks, such as to save the best model in each epoch, to interrupt training when some metric or loss reaches a condition, to show graphs, etc., and there is also the possibility of creating custom ones.\n\nOurs will interrupt training shortly before our time limit.\n\n\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"collapsed":true},"cell_type":"code","source":"import time \n\n#let's also import the abstract base class for our callback\nfrom keras.callbacks import Callback\n\n#defining the callback\nclass TimerCallback(Callback):\n    \n    def __init__(self, maxExecutionTime, byBatch = False, on_interrupt=None):\n        \n# Arguments:\n#     maxExecutionTime (number): Time in minutes. The model will keep training \n#                                until shortly before this limit\n#                                (If you need safety, provide a time with a certain tolerance)\n\n#     byBatch (boolean)     : If True, will try to interrupt training at the end of each batch\n#                             If False, will try to interrupt the model at the end of each epoch    \n#                            (use `byBatch = True` only if each epoch is going to take hours)          \n\n#     on_interrupt (method)          : called when training is interrupted\n#         signature: func(model,elapsedTime), where...\n#               model: the model being trained\n#               elapsedTime: the time passed since the beginning until interruption   \n\n        \n        self.maxExecutionTime = maxExecutionTime * 60\n        self.on_interrupt = on_interrupt\n        \n        #the same handler is used for checking each batch or each epoch\n        if byBatch == True:\n            #on_batch_end is called by keras every time a batch finishes\n            self.on_batch_end = self.on_end_handler\n        else:\n            #on_epoch_end is called by keras every time an epoch finishes\n            self.on_epoch_end = self.on_end_handler\n    \n    \n    #Keras will call this when training begins\n    def on_train_begin(self, logs):\n        self.startTime = time.time()\n        self.longestTime = 0            #time taken by the longest epoch or batch\n        self.lastTime = self.startTime  #time when the last trained epoch or batch was finished\n    \n    \n    #this is our custom handler that will be used in place of the keras methods:\n        #`on_batch_end(batch,logs)` or `on_epoch_end(epoch,logs)`\n    def on_end_handler(self, index, logs):\n        \n        currentTime      = time.time()                           \n        self.elapsedTime = currentTime - self.startTime    #total time taken until now\n        thisTime         = currentTime - self.lastTime     #time taken for the current epoch\n                                                               #or batch to finish\n        \n        self.lastTime = currentTime\n        \n        #verifications will be made based on the longest epoch or batch\n        if thisTime > self.longestTime:\n            self.longestTime = thisTime\n        \n        \n        #if the (assumed) time taken by the next epoch or batch is greater than the\n            #remaining time, stop training\n        remainingTime = self.maxExecutionTime - self.elapsedTime\n        if remainingTime < self.longestTime:\n            \n            self.model.stop_training = True  #this tells Keras to not continue training\n            print(\"\\n\\nTimerCallback: Finishing model training before it takes too much time. (Elapsed time: \" + str(self.elapsedTime/60.) + \" minutes )\\n\\n\")\n            \n            #if we have passed the `on_interrupt` callback, call it here\n            if self.on_interrupt is not None:\n                self.on_interrupt(self.model, self.elapsedTime)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac7e6a7e4a0c3d6ab5b78dffa4c5240f43d1b0b7"},"cell_type":"markdown","source":"## Using callbacks\n\nUsing callbacks in Keras is very simple, you just pass a list of them to the fit method. Suppose we want to stop training before reaching 350 minutes (5:50 hours):\n\n    timerCallback = TimerCallback(350)\n    model.fit(x_train, y_train, ..... , callbacks = [timerCallback, someOtherCallback])\n    \nYou can explore the `on_interrupt` method to save the model or its weights at interruption:\n\n    timerCallback = TimerCallback(350, \n                    on_interrupt = lambda model, elapsed: model.save_weights('my_weights.h5'))\n    \nOr you can use a `ModelCheckpoint` or a `LambdaCallback` if you want more customization:\n\n    from keras.callbacks import ModelCheckpoint\n    model.fit(x_train, y_train, ...., callbacks = [timerCallback, \n                                                   ModelCheckpoint('my_weights.h5')])\n\nSee more on callbacks here:  https://keras.io/callbacks/\n    \n## EXAMPLE\n\nNow, let's take a toy model for digit classification and interrupt it's training in five minutes, for instance.\n\nSince this Kernel is not about models, how to make good networks, etc., let's not take too much time explaining the details of data loading and model creation.   \nI'm sure there are lots of tutorial kernels on this :)\n\n### Loading and checking:"},{"metadata":{"trusted":true,"_uuid":"0bd02ed66ed37ebf3d5a7eecc459996439acecbd","collapsed":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom keras.utils import to_categorical\n\n%matplotlib inline\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\n\n#loading data - as for a demonstration of the callback,\n#we won't worry about validation data in this kernel (but it would work the same way)\ntrainData = pd.read_csv('../input/train.csv')\ny_train = to_categorical(np.array(trainData['label']))                   \nx_train = np.array(trainData[list(trainData)[1:]]).reshape((-1,28,28,1)) \n    #labels as one-hot encoded vectors (ex: label 2 will become [0,0,1,0,0,0,0,0,0,0])\n    #x shaped as images with one channel\n    \n    \n#quick check\ndef quickCheck(x, y, predicted=None):\n    #plotting images\n    fig,ax = plt.subplots(nrows=1,ncols=10, figsize=(10,2))\n    for i in range(10):\n        ax[i].imshow(x[i].reshape((28,28)))\n    plt.show()\n    \n    #printing labels\n    y = np.argmax(y, axis=1) #converting from one-hot to numerical labels\n    print(\"  \" + \"      \".join([str(i) for i in y[:10]]) + \" <- labels\")\n    \n    #printing predicted if passed\n    if predicted is not None:\n        predicted = np.argmax(predicted, axis=1)\n        print(\"  \" + \"      \".join([str(i) for i in predicted[:10]]) + \" <- predicted labels\")\n\nquickCheck(x_train,y_train)\n\n\n\n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1b33b1a91254ff7cb7007859a75afe0eaaef056"},"cell_type":"markdown","source":"### Creating a model"},{"metadata":{"trusted":true,"_uuid":"58bf8e8e74b4cfbbd2ac26da49b06298b084063e","collapsed":true},"cell_type":"code","source":"from keras.layers import Input, Conv2D, Dense, Flatten, MaxPooling2D\nfrom keras.models import Model\n\n#a simple convolutional model (not worried about it's capabilities)\ndef createModel():\n    inputImage = Input((28,28,1))\n    output = Conv2D(10, 3, activation='tanh')(inputImage)\n    output = Conv2D(20, 3, activation='tanh')(output)\n    output = MaxPooling2D((4,4))(output)\n    output = Conv2D(10, 3, activation='tanh')(output)\n    output = Flatten()(output)\n    output = Dense(10, activation='sigmoid')(output)\n\n    model = Model(inputImage, output)\n    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n\n    model.summary()\n    return model\n\nmodel = createModel()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dad500b642e3b7ff25109f31cd30d89e3e011178"},"cell_type":"markdown","source":"### Training the model for 5 minutes at most \n\n#### For your Kaggle kernel, you could try 350 minutes, as it has a 360 minutes limit\n\n.\n\nOf course that usually you should enable the GPU for your kernel to run many times faster. This callback technique is absolutely not necessary for such a simple model with a simple dataset, but might come in handy when you're working on really big models and data :)\n"},{"metadata":{"trusted":true,"_uuid":"791fcd77e1d8084338238a9c66369e746ffa1420","collapsed":true},"cell_type":"code","source":"model.fit(x_train, y_train, epochs = 1000000000, callbacks=[TimerCallback(5)])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"53f11165eb8c7e8a732740fe962c01e7f8c2b5e6"},"cell_type":"markdown","source":"### Saving the model's weights as outputs (you can then download these weights later or use them as inputs to other kernels)\n\nYou could also explore `model.save(file)` and `model = load_model(file)`, if you prefer. (Some models might have serialization problems, because of this I always prefer `save_weights()` )"},{"metadata":{"trusted":true,"_uuid":"6c228e2e7ce541c78dc31a2fe4488ad702433cbc","collapsed":true},"cell_type":"code","source":"#a function compatible with the on_interrupt handler\ndef saveWeights(model, elapsed):\n    model.save_weights(\"model_weights.h5\")\n\n#fitting with the callback\ncallbacks = [TimerCallback(5, on_interrupt=saveWeights)]\nmodel.fit(x_train,y_train, epochs = 100000000000, callbacks=callbacks)\n\n\n#check that the weights were saved:\nimport os\nos.listdir(\".\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dbced22f54753cb0292f774dcecdb35a6643db99"},"cell_type":"markdown","source":"### Using the saved weights in a new model and checking predictions"},{"metadata":{"trusted":true,"_uuid":"af1eb88b27b4b17e2b0bbb7ba588c9ec26bc798f","collapsed":true},"cell_type":"code","source":"#although it uses the same creator function, it's a different model from the previous one\ndel(model)\nmodel2 = createModel()\n\n#load weights - this only works if the model has the same layer types and the same parameters\nmodel2.load_weights('model_weights.h5') #\n\n#evaluate model2\nprint(\"\\n\\nEvaluating model 2:\")\nloss, acc = model2.evaluate(x_train, y_train)\nprint('model 2 loss: ' + str(loss))\nprint('model 2 acc:  ' + str(acc))\n\n#predicting and checking\npredicts = model2.predict(x_train[25:35])\nquickCheck(x_train[25:35],y_train[25:35], predicts)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}