{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## TFModelQuantizer\n\n\n\n\nA Model Graph Quantization package for precision based optimized inference and scalability. This package is based on the [Nvidia TensorRT (TRT) library](https://github.com/NVIDIA/TensorRT) for faster inference time on large scale models. The quantization is either FP32,FP16 or INT8 with calibration which speeds up model inference to great extents. This package is compatible with Tensorflow Models and can optimize Tensorflow frozen graphs to the specified precision using just 3 lines of code. The package uses maximum engine capacity provided by TensorRT and enough cached memory to avoid any overloading (out of memory errors) .TensorRT uses tensorflow allocators for its memory management and hence all of tensorflow memory configurations also apply to TensorRT. The library can be found [here](https://pypi.org/project/TFModelQuantizer/)\n\n","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:51:59.109725Z","iopub.execute_input":"2021-05-26T16:51:59.110056Z","iopub.status.idle":"2021-05-26T16:51:59.125324Z","shell.execute_reply.started":"2021-05-26T16:51:59.110023Z","shell.execute_reply":"2021-05-26T16:51:59.124121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TFModelQuantizer\n\nTFModelQuantizer is a Python package which optimizes inference runtime by quantizing frozen model graphs. This package is compatible with Tensorflow models and uses Nvidia TensorRT calibration for optimization of Inference Time. Traditional models like Resnet,Inception,etc are sufficiently large models which takes a huge amount of time during inference in a unquantized manner. The package converts a presaved model graph (tf.Graph) into either of the following precisions:\n\n- FP32: Floating point 32 variant \n- FP16: Floating point 16 variant\n- INT8: Integer 8 variant (with additional calibration)\n\nBy converting model weights to a quantized manner, there is a less overhead on the memory buffer (Host GPU or device CPU memory). This functions by first freezing the model and then determines the conversion of the precision based on user's input. The precision conversion is done by the GraphConverter API of Tensorflow which quantizes the node weights. \nSegmentation of subgraphs in the frozen graph is kept as default to 3, and the maximum_cached_engines for host device inference is kept to 100 as an upper limit. However for large models like Resnet variants , these should not cross the engine size limit. The engine refers to the TensorRT backend. The systematic overview of model conversion is provided in the image:\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"## Package Specifications\n\nThe TFModelQuantizer uses the TensorRT (TRT) Inference for faster performance. The package can be found in [Github](https://github.com/abhilash1910/TFModelQuantizer/) and can be installed as follows:\n\n```!pip install TFModelQuantizer```\n\nIn the next step, we load a model from the Keras.applications module. For instance , Resnet 50 has been chosen for this purpose. Since the package uses TRT dynamic runtime for inference, we will only be performing inference on this. The beabuty of dynamic runtime during model conversion is that there is less memory overhead. Previously during conversion , static mode was used. Only subgraph segmentation happens before runtime , and the optimization part of the subgraph happens during the runtime. \nWe first load the Resnet 50 model and save it:\n\n```\nmodel = ResNet50(weights='imagenet')\nmodel_dir = 'savedmodels/resnet50_saved_model'\nmodel = ResNet50(include_top=True, weights='imagenet')\nmodel.save(model_dir)\n```\n\nNow we call the TFModelQuantizer class from the package. Then we specify the quantization precision (either of FP32,FP16 or INT8 with calibration). This takes as input the saved model from the directory and the precision and converts the model weights into the new precision and freezes the graph. Essentially this means it converts the model weights to a \".pb\" format (protobuffer).This is done by the following lines:\n\n```\nmodel_quantizer_fp16=TFModelQuantizer.TFModelQuantizer(model_dir,\"FP16\")\nmodel_q_16=model_quantizer_fp16.quantize(model_dir+'_FP16')\n```\n\nFor instance in the example here, we are converting to FP16, which will provide a significant boost of speed in runtime.\nTo check the runtime of the inference of the elephant image, we can do:\n\n```\nstart_time=time.time()\npreds = model_q_16.predict(x)\n# decode the results into a list of tuples (class, description, probability)\n# (one such list for each sample in the batch)\nprint('Predicted:', decode_predictions(preds, top=3)[0])\nprint(\"Inference Time on FP16 Quantized Model\",time.time()-start_time)\n```\nAnd thus we get the outputs (inference) and the time which is significantly smaller than the unquantized model. When iterated over optimized TF datasets like TFrecords , this package can provide significant speedupds, with just 3 lines of code. \n\n\nHere we are using the Elephant Image (taken from Google) as our input datapoint.\n\n","metadata":{}},{"cell_type":"code","source":"!pip install TFModelQuantizer","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:52:05.850755Z","iopub.execute_input":"2021-05-26T16:52:05.851112Z","iopub.status.idle":"2021-05-26T16:52:15.158723Z","shell.execute_reply.started":"2021-05-26T16:52:05.851053Z","shell.execute_reply":"2021-05-26T16:52:15.157698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.applications.resnet50 import ResNet50\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.applications.resnet50 import preprocess_input, decode_predictions\nimport numpy as np\nfrom TFModelQuantizer import TFModelQuantizer\nimport time\n\n#Load and save the Resnet Model\nmodel = ResNet50(weights='imagenet')\nmodel_dir = 'savedmodels/resnet50_saved_model'\nmodel = ResNet50(include_top=True, weights='imagenet')\nmodel.save(model_dir)\n\n#Convert the Model Graph to TRT TensorRT Runtime Inference FP16\nstart_time=time.time()\nmodel_quantizer_fp16=TFModelQuantizer.TFModelQuantizer(model_dir,\"FP16\")\nmodel_q_16=model_quantizer_fp16.quantize(model_dir+'_FP16')\ndiff_fp16=time.time()-start_time\nprint(diff_fp16)\n\n#Preprocessing the input for Inference\nimg_path = '../input/elephant-image/elephant.jpg'\nimg = image.load_img(img_path, target_size=(224, 224))\nx = image.img_to_array(img)\nx = np.expand_dims(x, axis=0)\nx = preprocess_input(x)\n\n","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:52:15.161751Z","iopub.execute_input":"2021-05-26T16:52:15.162094Z","iopub.status.idle":"2021-05-26T16:53:27.098743Z","shell.execute_reply.started":"2021-05-26T16:52:15.162049Z","shell.execute_reply":"2021-05-26T16:53:27.097794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_path = '../input/elephant-image/elephant.jpg'","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:53:27.100618Z","iopub.execute_input":"2021-05-26T16:53:27.101128Z","iopub.status.idle":"2021-05-26T16:53:27.104960Z","shell.execute_reply.started":"2021-05-26T16:53:27.101086Z","shell.execute_reply":"2021-05-26T16:53:27.103987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Comparison of the Runtime Performance\n\nNow we see the runtime from the standard unquantized Resnet 50 model. It takes around 3.9 seconds on this task.","metadata":{}},{"cell_type":"code","source":"start_time=time.time()\npreds = model.predict(x)\n# decode the results into a list of tuples (class, description, probability)\n# (one such list for each sample in the batch)\nprint('Predicted:', decode_predictions(preds, top=3)[0])\nprint(\"Inference Time on Unquantized Model\",time.time()-start_time)\n","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:53:27.106701Z","iopub.execute_input":"2021-05-26T16:53:27.107246Z","iopub.status.idle":"2021-05-26T16:53:32.057871Z","shell.execute_reply.started":"2021-05-26T16:53:27.107206Z","shell.execute_reply":"2021-05-26T16:53:32.056708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Floating Point 16 (FP16) Quantization\n\nThe runtime of the FP16 TRT quantized Model is 2.26 seconds . Hence a significant improvement from the traditional model.","metadata":{}},{"cell_type":"code","source":"start_time=time.time()\npreds = model_q_16.predict(x)\n# decode the results into a list of tuples (class, description, probability)\n# (one such list for each sample in the batch)\nprint('Predicted:', decode_predictions(preds, top=3)[0])\nprint(\"Inference Time on FP16 Quantized Model\",time.time()-start_time)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:53:32.059546Z","iopub.execute_input":"2021-05-26T16:53:32.059930Z","iopub.status.idle":"2021-05-26T16:53:34.252047Z","shell.execute_reply.started":"2021-05-26T16:53:32.059888Z","shell.execute_reply":"2021-05-26T16:53:34.251189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Floating Point 32 (FP32) Quantization\n\nThis should take a little more time during the inference mode than the FP16 counterpart, but should be less than the unquantized model. For this , we first convert the saved model to FP32 format and then we run the inference. \nWe see that the runtime is 2.47 seconds on FP32.","metadata":{}},{"cell_type":"code","source":"#Convert the Model Graph to TRT TensorRT Runtime Inference FP32\nstart_time=time.time()\nmodel_quantizer_fp32=TFModelQuantizer.TFModelQuantizer(model_dir,\"FP32\")\nmodel_q_32=model_quantizer_fp32.quantize(model_dir+'_FP32')\ndiff_fp32=time.time()-start_time\nprint(diff_fp32)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:53:34.253513Z","iopub.execute_input":"2021-05-26T16:53:34.253862Z","iopub.status.idle":"2021-05-26T16:54:08.740193Z","shell.execute_reply.started":"2021-05-26T16:53:34.253824Z","shell.execute_reply":"2021-05-26T16:54:08.738419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_time=time.time()\npreds = model_q_32.predict(x)\n# decode the results into a list of tuples (class, description, probability)\n# (one such list for each sample in the batch)\nprint('Predicted:', decode_predictions(preds, top=3)[0])\nprint(\"Inference Time on FP32 Quantized Model\",time.time()-start_time)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:54:08.745121Z","iopub.execute_input":"2021-05-26T16:54:08.745415Z","iopub.status.idle":"2021-05-26T16:54:11.445405Z","shell.execute_reply.started":"2021-05-26T16:54:08.745384Z","shell.execute_reply":"2021-05-26T16:54:11.444544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Integer 8 (INT8) Quantization\n\nNow we will be seeing the comparison, between INT8 and the other 3 variants. INT8 is a special form and requires sophisticated calibration for conversion. For this we have used the default TensorRT calibration techniques for INT8, and as a requirement we have to attach a calibration input for the quantizer to convert the model weights. Hence here we can see that a dummy variable has been initialized which is np.ones having the same dimensions that of the input , and the dtype as 'float32'. It is important to mention the dtype in this case as it will be used in the calibration step. We provide the input dummy variable to the TFModelQuantizer (the additional 'dummy_one_batch') for this INT8. Ideally INT8 quantization is the fastest method for inference and this can be shown with the runtime of 1.95 seconds.","metadata":{}},{"cell_type":"code","source":"dummy_one_batch = np.ones((32, 224, 224, 3),dtype='float32')\n#Convert the Model Graph to TRT TensorRT Runtime Inference INT8 with calibration\nstart_time=time.time()\nmodel_quantizer_int8=TFModelQuantizer.TFModelQuantizer(model_dir,\"INT8\",dummy_one_batch)\nmodel_q_8=model_quantizer_int8.quantize(model_dir+'_INT8')\ndiff_int8=time.time()-start_time\nprint(diff_int8)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:54:11.449942Z","iopub.execute_input":"2021-05-26T16:54:11.451987Z","iopub.status.idle":"2021-05-26T16:54:50.106284Z","shell.execute_reply.started":"2021-05-26T16:54:11.451948Z","shell.execute_reply":"2021-05-26T16:54:50.105298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_time=time.time()\npreds = model_q_8.predict(x)\n# decode the results into a list of tuples (class, description, probability)\n# (one such list for each sample in the batch)\nprint('Predicted:', decode_predictions(preds, top=3)[0])\nprint(\"Inference Time on INT8 Quantized Model\",time.time()-start_time)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:54:50.109698Z","iopub.execute_input":"2021-05-26T16:54:50.109976Z","iopub.status.idle":"2021-05-26T16:54:52.367820Z","shell.execute_reply.started":"2021-05-26T16:54:50.109948Z","shell.execute_reply":"2021-05-26T16:54:52.366893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## To summarize\n\nThe table provided below shows the performance of the quantized models during inference. As mentioned earlier, INT8 has the fatest inference time followed by FP16 and FP32. However this is just for a single image on a moderate architecture. The scale up is significant for extremely large models of the Inception variant with enormous data.","metadata":{}},{"cell_type":"code","source":"from plotly import figure_factory as FF\nfrom plotly.offline import iplot, init_notebook_mode\n\nquantization_table_data = [['Precision', 'Inference Time(seconds)'],\n ['Unquantized',3.9002022743225098],\n ['FP32', 2.471080780029297],\n ['FP16',2.2663280963897705], \n ['INT8', 1.9520716667175293],\n ]\nfigure = FF.create_table(quantization_table_data, height_constant=60)\niplot(figure)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:41:27.651556Z","iopub.execute_input":"2021-05-26T16:41:27.651966Z","iopub.status.idle":"2021-05-26T16:41:32.802141Z","shell.execute_reply.started":"2021-05-26T16:41:27.651883Z","shell.execute_reply":"2021-05-26T16:41:32.801266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n","metadata":{}},{"cell_type":"markdown","source":"## Runtime Performance on Inception V3\n\nHere we will be analysing the performance statistics on InceptionV3 network. We are using the base pretrained model for inference only. No modifications or subsequent additions of certain MLPs have been done. We will be analysing for FP32,FP16 and INT8(calibrated) performance.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications.inception_v3 import InceptionV3\nfrom tensorflow.keras.layers import Input\n\nbase_model = InceptionV3(weights='imagenet', include_top=False)\nmodel_dir = 'savedmodels/inceptionv3_saved_model'\nmodel.save(model_dir)\n\n#Convert the Model Graph to TRT TensorRT Runtime Inference FP16\nstart_time=time.time()\nmodel_quantizer_fp16=TFModelQuantizer.TFModelQuantizer(model_dir,\"FP16\")\nmodel_q_16=model_quantizer_fp16.quantize(model_dir+'_FP16')\ndiff_fp16=time.time()-start_time\nprint(diff_fp16)\n\n#Preprocessing the input for Inference\nimg_path = '../input/elephant-image/elephant.jpg'\nimg = image.load_img(img_path, target_size=(224, 224))\nx = image.img_to_array(img)\nx = np.expand_dims(x, axis=0)\nx = preprocess_input(x)\n\nstart_time=time.time()\npreds = model_q_16.predict(x)\n# decode the results into a list of tuples (class, description, probability)\n# (one such list for each sample in the batch)\nprint('Predicted:', decode_predictions(preds, top=3)[0])\nprint(\"Inference Time on FP16 Quantized Model\",time.time()-start_time)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:56:01.852474Z","iopub.execute_input":"2021-05-26T16:56:01.852911Z","iopub.status.idle":"2021-05-26T16:57:06.241667Z","shell.execute_reply.started":"2021-05-26T16:56:01.852872Z","shell.execute_reply":"2021-05-26T16:57:06.240724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Convert the Model Graph to TRT TensorRT Runtime Inference FP32\nstart_time=time.time()\nmodel_quantizer_fp32=TFModelQuantizer.TFModelQuantizer(model_dir,\"FP32\")\nmodel_q_32=model_quantizer_fp32.quantize(model_dir+'_FP32')\ndiff_fp32=time.time()-start_time\nprint(diff_fp32)\n\nstart_time=time.time()\npreds = model_q_32.predict(x)\n# decode the results into a list of tuples (class, description, probability)\n# (one such list for each sample in the batch)\nprint('Predicted:', decode_predictions(preds, top=3)[0])\nprint(\"Inference Time on FP32 Quantized Model\",time.time()-start_time)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:57:06.243153Z","iopub.execute_input":"2021-05-26T16:57:06.243650Z","iopub.status.idle":"2021-05-26T16:57:43.166477Z","shell.execute_reply.started":"2021-05-26T16:57:06.243609Z","shell.execute_reply":"2021-05-26T16:57:43.165483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dummy_one_batch = np.ones((32, 224, 224, 3),dtype='float32')\n#Convert the Model Graph to TRT TensorRT Runtime Inference INT8 with calibration\nstart_time=time.time()\nmodel_quantizer_int8=TFModelQuantizer.TFModelQuantizer(model_dir,\"INT8\",dummy_one_batch)\nmodel_q_8=model_quantizer_int8.quantize(model_dir+'_INT8')\ndiff_int8=time.time()-start_time\nprint(diff_int8)\n\nstart_time=time.time()\npreds = model_q_8.predict(x)\n# decode the results into a list of tuples (class, description, probability)\n# (one such list for each sample in the batch)\nprint('Predicted:', decode_predictions(preds, top=3)[0])\nprint(\"Inference Time on INT8 Quantized Model\",time.time()-start_time)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:58:46.493665Z","iopub.execute_input":"2021-05-26T16:58:46.494004Z","iopub.status.idle":"2021-05-26T16:59:26.403902Z","shell.execute_reply.started":"2021-05-26T16:58:46.493974Z","shell.execute_reply":"2021-05-26T16:59:26.402935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from plotly import figure_factory as FF\nfrom plotly.offline import iplot, init_notebook_mode\n\nquantization_table_data = [['Precision', 'Inference Time(seconds)'],\n ['FP32', 2.479130506515503],\n ['FP16',2.3265860080718994], \n ['INT8', 2.331540822982788],\n ]\nfigure = FF.create_table(quantization_table_data, height_constant=60)\niplot(figure)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:59:40.329943Z","iopub.execute_input":"2021-05-26T16:59:40.330301Z","iopub.status.idle":"2021-05-26T16:59:40.374497Z","shell.execute_reply.started":"2021-05-26T16:59:40.330267Z","shell.execute_reply":"2021-05-26T16:59:40.373612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}}]}