{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"                               ** Attention mechanism in keras **\n\n\nThis notbook tries to explain what often-quoted Attention mechanism is. we will discuss how a simple Attention model can be implemented in Keras. The purpose of this demo is to show how a simple Attention layer can be implemented in keras. The NLP field adressed here is sentiment analysis. The dataset is publically availble  [here](https://archive.ics.uci.edu/ml/datasets/Sentiment+Labelled+Sentences)\n\n\n\n![image.png](attachment:7f841086-f0f0-494e-aa5b-29dab9f4905c.png)\n\n# What is Attention?\nWhen we think about the English word “Attention”, we know that it means directing your focus at something and taking greater notice. The Attention mechanism in Deep Learning is based off this concept of directing your focus, and it pays greater attention to certain factors when processing the data.\n\nIn broad terms, Attention is one component of a network’s architecture, and is in charge of managing and quantifying the interdependence:\n\nBetween the input and output elements (General Attention)\nWithin the input elements (Self-Attention)\nLet me give you an example of how Attention works in a translation task. Say we have the sentence “How was your day”, which we would like to translate to the French version - “Comment se passe ta journée”. What the Attention component of the network will do for each word in the output sentence is map the important and relevant words from the input sentence and assign higher weights to these words, enhancing the accuracy of the output prediction.\n\n\n","metadata":{},"attachments":{"7f841086-f0f0-494e-aa5b-29dab9f4905c.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAtUAAACGCAYAAAASNnYbAAAgAElEQVR4Ae19O64dR7DkWwPXIZsbISGTG6BDc3DpPJsAHQ7oXV+GHEEOFyCPlgACYwjQAuhoCz2oT1RlRlX19/xvCCBOd9cvMyoiM7tP36P/mfSfEBACQkAICAEhIASEgBAQAocQ+J9DozVYCAgBISAEhIAQEAJCQAgIgUlFtUggBISAEBACQkAICAEhIAQOIqCi+iCAGi4EhIAQEAJCQAgIASEgBFRUiwNCQAgIASEgBISAEBACQuAgAiqqDwKo4UJACAgBISAEhIAQEAJCQEW1OCAEhIAQEAJCQAgIASEgBA4ioKL6IIAaLgSEgBAQAkJACAgBISAEVFSLA0JACAgBISAEhIAQEAJC4CACKqoPAqjhQkAICAEhIASEgBAQAkJARbU4IASEgBAQAkJACAgBISAEDiKwoqj+Of3+4e306vXb6dWHb9NPXvDnt+ldaPvyg1vu7/zv5+Rn8Kf59zx9n6bp+5dz+Zpwfvdng/D94XgJiyPvnqbfBdcJ0V7Pwf06sGvY461u/Jg+v347ff5767jaf+jDLXArxqIUc6rF6WhoN3d8See3sGd78L5Xu/f4qjFC4AUgsFxUR9GjyOwUMWg/UFR//9KZ95TgBxu32jdIaudLaBsKjIsE4uNFy+Yt3OLXlr6bDTntgPNx5rR2TtN6Du73ya5hj7f6cpyf1oeffz7Vhwa3wC0bf8gea/dW1B6pv8OBMLplP69v93Ht3DK+sk0IXBOBxaI6JpvwROhLeorbPEmNwWz/09sYYF6fsajea59NamaHXEA0148fbigwLpJArhB4L+LX8Z3aOsP5OLPVkqX+6zm43ye7hj1eso3bT8vPmyuqrbsPqgvr4p7j/Rzcs9rpxlzf7tNq53TIaCYhcP8ILBTVKem9eh2+hkxCbF4BKUXrt/qaCBXJMYiY1ynwlS0KdrxqEQp2XEMRj9dKcN32tfD7NVCkZ5vL2sGP5BNssHO449mieuzrBDzimrDDzVxPbN8Pz9PnD2+netMC7PEtAb4Ktj7V+fv+h6VoHnpi73Atr/f4MSOs3JpuXmuj/4o+jvnyHL+6T3sJH+wYXMuv25T9q9cTzvk8Fx2fvzzV13asPXEvgaOZo+5EPvJ+g3voNvQ3c+VzeDUo24p9dPgWm6yvFp+0/rsvz+mVqjgX9h0m5tetOvxya5W9DOPm/YJ/6AfbcQ6fUhxIvdM+Wh2QnYgXGY/KoexjfM3JHpt58xjGf7L7mDlU54UXAVuzx5kb5TWheJ5sTT78mNy8ATeM+dO8Dlb2DuvgEz58q5x22Af4x3vmOGXHlfhjuZL88nZb3GEL3omyYy3PYLv/dLYYf8P1wklrI4bP8D91mbfDrpvivvHJYVcf3jiuB1uxZz87r+iZtmiPm9NwBf6Uz4Rn4b/BJHRxNlhcZvBwY8ju4oPhnc2JwY6qzWDBGNfEkV6c9T61+inO60AICIEdCMwX1Qg+OZik4EdBCH1i4R0syEJHkMmJMIkXgq5Bk+csQccGMDcHCq1qh5+D1oB9dr41QOXAGN6jtv/5tbIt8DX7XgLVYI40X7KzBMnsI84jDsVm6ht9qv7HoqDY4JNKb56RfdG3smbax9LXgoCEUta09tnjKRcs1dZZ/Db4VRJQqB94j/N5sj34UddnrKxbc1jFtq6/8NEkvLiXdU2PK+HjfE5ttXjN56M9sfyyx/Tu/5xf1n8U0Ws4mPaxFmrxfIRPx8e0hsdiFmPSVuw7eKc62GJ9ePfhyZ2jWLf74tae5ZNHDJi9Khyb37Nkd45/Yc8KZkm3sDsV+rmfw8/qmzTq+nlsnV7YhUbPaV7Ykva6xuxmeORexTzZDv4v2OF4m7EruSSdlxjk/LM4IAbkNd2cufgtOBNm1Nf6FveqaI9soXGWS8n/ER4zdjPv4hp1HscdflWrh03hZF5zhIF1WsdCQAgcQmC2qE4iromzCRZhaQ4ESOgQdA4M8W6/BKhqcwrYCMD17r8E0trVPxnA/FzEm/7xsGMfd+meU9BEHxc8ORmFMSVwhREUiDFJbApPr6rf6ItEZrviRqW0UQD1fXNiylinPbTr1N7BlzJnsQnJk5JPHVb8cmPR3tiW7EHfWfyasZg0Gpe+CQGHbF97HIdY29MxCik7Ix+Pserso91r5grZ43yObcA4WVD3wWMVWt1YXsc4UOfIF806Y7/MBPGwXb/2SDi6fbRctz6btTG+2mfXaI+d7ucwpiIb64TP6G/mSfx7jb/xNxVpPaxhsY1j4I/1JU5s+WRXiqt5XoZLdp/sMc8V20x8tVPbcWSPtdsex3WdPkY8swuFY7sPqc3i4dbgoeHc2hqnM7FtlgvE795cdr24To1lzi6Hkd+v0A97Hm3FPse5PSfscnO6CXNCC2ma4HPGew4P1rS12x6HSfnczhvbxvvrsIEmit8eH+uzjoWAEDiGwExRnYJNfWoGkdOvgERx16/lgjkpGCGQYZ761bgtPKP4TXHpx2bnYjAJ41MQcWM66ztIltpdZ3NiA5i5PBesYHv5ujB/je2CL+bqzB/mLn2Lz8Hvp+mdfTUk+lSTS7mxya8EhL62iPR2YVxnX/L49FX5XOCdaWv8SuvArzn8miSCvev5ZTGwxxFfti+dl31B4YG9MJ99rGh83lfwcbao4CTq9tVoItrksQpmWbyibSUxGqP59Y5iH/a6ajJhUK/bWZriytnqOWjtSnMEjPK8btycj9bfeYxb39PYUixZRyIfQqwIc/Y+U2frg5t/kU9usVhUg9+xpayfcXd7lvwsdhNW/noumsgea3fSTI2LfrzBHpzocp9sQgzPdrv1rOs4jj5kW8M1ay/55zVo9z9PRnPFtWF7+MYBHCNtuDVhf/QVHEjze31XfNz+wS/Mg/XL2snu4gu3kw9sm8PTYmWPGcdwbuedxdXHjTiV/UPcmRtS47oOhYAQ2IHAuKiOAq9BxwcQk5TRzwTrFAhNHxiGvubdMO6LoFeSA4JnCWg5YJTzlBBsoY7l4ifWNPa59tGJDWCmjwuICLpImmEMjs2Y7mG0y2JkE4w9DqPpnMayTXxu149tJlmOkgmejtt9qPOQPbXBJ9R43fdl2+aKGe7rzi0G9jiu2RYJxcTYFzd85Wr3oGKVfOhjQckuzET2tHabAsSt7LEKTW7sgJPoN95Lt0ias8tTu749DuP9ubMrNs8/nawW2Hna49UYzxYGYd6n6ffyhDrg+DT9Ht5VNXHA+jDHwzVasPO2xY/d7zE3U+zLfe1ez/Ep78vnv33xmDho16070B7ZfUitFg+LUzt2gf/R9rEdzdwzfi/rysRTrBvmM3vePqnuetS9GG1dEzutD2Gmuf2zbfa4M87xCv51LaW4EUxQUT1ASpeFwGkRGBbVKcCbJ6d53eZ6FHcovhHQfJGL/iVZxoBTi5rUjrH1iVrpj8IC708263GRjXME8myPDaxrMOTAmMdwEoj2lwKFEma21fpSl06JrAT83DcVRtSWMStFU+xbMbPBPgXw+qS6ay+wIB/TXnjc+rZzkLZBnBJ0XINsxfoc7Df45RIVjXNFULet2lP3w/qQrkY8sq1+n3Nf7Dvh6GwDf4vPhE8uDu2+l31uxvb4lX0hG+xeznLAAuAK53kOxjmL5gkPN09YINnd+uixmMXYzYFCrsYR50bGjd+lDueWzxYXt3aXM6O1Mk746j/7XnRNxb/dF7dmcCDsYY9TZI+1OwxLc3Ks9tj6PWC0WM92v1pdNKOJe57/C3a4sYRl9Lvi3uUcdEUY4Saw5iVY3dNQXQO9wmcXZ6zn7MYe5NhJbR4PmtfabY+DAXzu5p3HtWs7uEWctD7rWAgIgWMIDIpqBLdO8RGFbl4Bwbn7RQcUZsk4BH087bZFQwoc+Yn4lx8lQdjkh4SA8fhlENsnBVw8Wfd217ZwPflmx3YhdAGs9pgPVgiEsIMTXZ0nHaUAn/x6nsKvRxRs4voVF78u9icnA+xB+Bryw7fpu3sqUftiHfvHl35v+rgVm8j8iqvhQ+xj/fIJy/vBybzaGvdnzq/Ylu21x2b9sscWS/MtCbkTNs/8gk3A3vPY+WvbmCtsD9anpAY+1wIsrW/xZrycXnCjmR0Z7+W8XxUHWh92B159+eGKjGTX+l//6PtI66GQKV+ne/yd7/nXcsoeVyfSUba9tPMeYS0USYVrz9N33r/ZIiT7YH+xpewzNiY8xUdM6GssccG0OXvr/gV/+pwwYwsWXod1D0oHd+D4DVwYJzcinzhbEQOtPfN21HXztwlGW7UtxJE0j9/THHuaPePYYgwve532xOrN9FqMB0O9LeER2zt2sw98zvNmXrZxpOVItNXwErhG38O8ps1joDMhIAS2IDAoqrdMob5CQAgIgZeKQHtj8FKROInfKvBOAqMmEQJC4DoIqKi+Du5aVQgIgYdAQEX1kW2MT1DLU/H8RL6cH5lZY4WAEBACl0dARfXlMdeKQkAIPAwCKqqPbSW9GqLXEI7BqdFCQAhcFQEV1VeFX4sLASEgBISAEBACQkAIPAICKqofYRflgxAQAkJACAgBISAEhMBVEVBRfVX4tbgQEAJCQAgIASEgBITAIyDgiur//d//nfRPGIgD4oA4IA6IA+KAOCAOvGQO/Prrr5vr/Kao3jyDBggBISAEhIAQEAJCQAgIgQdC4JdfftnsjYrqzZBpgBAQAkJACAgBISAEhMAjI3DSojpMpn/C4N45YL++uXdf7s1+Ya/4cW7O3ivH7tXuc++n5t8eM8Sl7ZjN8Yzx3HrTMHxSHRb9979J/4TBXXMgvA+G/wKn9d/lEGDsFU8UT0/NAebY5dh9bCW2+9S4aL6XozXm0jFmavRRPFVUq2i+66J5KXkcFYhCzH4EGPulvVL7yykETrXXzLH9bL3sSLb7VHhonpenIebSZZn8eKsdxVNFtYpqFdWPFxduwiMOTkr4Ly/hn3vPmWM3QfwVRrDd58ZJ8z+u9phLK+inLjMIHMVTRbWKahXVMwJT034EODgpsT9uYr/W3jLH9rP1siPZ7mvhp3XvX5PMpcsy+fFWO4qnimoV1SqqHy8u3IRHHJyUwO8/gd/aHjLHboL4K4xgu28NV9lzP1plLq2gn7rMIHAUTxXVKqpVVM8ITE37EeDgpER9P4n6XvaKObafrZcdyXbfC96y8/Y0zFy6LJMfb7WjeKqoVlGtovrx4sJNeMTBSQn59hLyve8Jc+wmiL/CCLb73vdB9l9P28ylFfRTlxkEjuKpolpFtYrqGYGpaT8CHJyUeK+XeB8Ve+bYfrZediTb/aj7I7/Or3nm0mWZ/HirHcVzZ1H9Y/r4+u30iv+9/zZ9v3CR+scnb8fHv05M4n++TW/IT7cG2j/9eJjiNGH6NH39ZweWN4bHaoH8/dzyuez78/T98WLH2T1i7IcJFpx5/Tz9YeLHIR6aeUbrnnv+0bru+l+GdxxDgAtfX+GbW+OB+zPHZkntNP40/f7T9/7551OJAe/+5MZv0zvEgw/fJmqdvn+peejz337e3hnbvbxfJueKDw+Ta5f3fTkHM5d6fMO1xNM2n0Xuf/mBbvEz9PVc/jF9Dhqgfm7QA5xswbPn7rGi2hXREL1PjPOk+TF9dHMsE8jO9/23FARTkftz+vo+BLYtxeDC+jnhvfntZxEx1izXbjDx/fFpCwYt5luLjYBJudG4MTx2CSQm3zbw9AQ0fy0FIR+Y5kc8Uitjb7XrjsGZ12+noqv/pmkrD92cKwrJc8+/bA9i1iBmApcTFlFHY8OyT208OecY5thQPz9DUVwL6VRAG43bdnscJ/w5/f7h7ZQKbXucV7Pxwh4PjZkmtnsRI3vzRTefi2NXaEFzXJa3p8SbuTRDu3LzxzeNbVEdeF71EucM3P7yPH1+bXQzt9idtm3Bs+fiCYvqafq3U4SOyZOL8ANF9bGkuLR+bl9KaGdIfGPMloV/DJPl+dk23GSUovrGAvgugaxMjD1B+WsqqoHH7P+hFRqKTwLrDeG5uXzu+Vkr7flli+rr+7s9vrSY+Tl26TuQkgrnWFSYJ9DhKV0pPKjvFOID9a1P71LRvXQjzXYv+Ym9+/jJPkjyWCzNofbHxIu5hJjb+4xPqj88uRvMKIfwLY19Ah04bzg+TeB156ayt9AdX9uCZ8/N0xbV/7WFKIouvCqSnkQhmeArMyTS0fW+GFKgmX86Xfu8nV6VAn7FOmtvEFAQvH8yr4n4J099DJJPHCzn8ck4YE18HZkL/+E67ikHsA5z4cbiefponvLDJrz+sW7et9OrYAdsMzcjPN4W4Nxmn1Sizfbfmhh2CaRbVOevvjLmJWnGvvZrstTv3Z//Lz7dAu9L/54KH/QaYz/cO3CG+DziYeGD0yg0/Tx9zd9gRew//chPvFOssfwq8/82fgUj9clxqsSPsXYaHwe+/fsf7EUMfFu/7cGNKcZCS+X8W/5Wro190Ax4B3+3Xg9+bMaHvyUcxp2pxgnsucXW7WuOecCEPplja6XUK6JtURGLDxQZHA/cORcZfN63iO1ueOP8RJz+Nn0vHPCvG2KvUHRj38f8rQ/BwJUYv/O64EvRmrNnfk/mfdHYU+PDXOozLl0Fr1NxXV9j4ifV4bzcVIah8cYyPaFm7cytd49tW/Ds+XeeohoBMgdHCBPCT8WaCRRZsKkdBWnb3pARAaYXmG1SiO8G5/mQoEpB2X8PfHVQKTZku3GOdWYxQOLqfe09wGEwvw+ipnCmBNXFmJKh26e83mj+Bieyj9sPzb0jsO8SiEuaMaKYr38RYMxXY6Y/glYSm55UI+iselL9CcVi4q/jyn/TxFzy34zVIjXFm3qeuIvzqo00f9Uez+/XH8QP0o6LUaSFehOLmAOboHUqOHg8zvH1P84Ra/L5SKven1pU9eNzG5t4vD8nX2bjTu5LeQJ2+30lTCgGbNZ3LA7SjYy90Q26tUWEKzKCvu1TO1NgTOUJHpg+xa/Y7Vy1pR6x3Y435KPHg3B2ubNyOczn94f4S1xh7vP5rH1sr87L66KXwI25VFnWHtX85HOT4/uUOGz14dvDWJP/2mXu+soWPHuOnreozuKCQNMdMZJaFjkCaxEigkbv6VAvwOZ5UFgj4XSK5hRkkMBG66c1YDMSzlAcnNg664axmM9jwIHP+tfHAfOM7PKBtDO/K/L7GLg54F/At9mr6lexB/1joocPwNz6R0+rOnMPMS9cofk613cJxBTJUTQuiSYZcRKOwSpy0L5v5gNXT4CPfI2xH+6n5QyO33+bvsY/Qka86HDNFW7gWu3veNwUGR1tYO3I3VYbW+JH1Xy1p70Gmxf0QUVzfaJINsL+gVYZD+wHYspSbEK/pHVam7TXrOXiDvxun7TDprWfzLHVesrFNQoH1rMrIq5cVDOWfh9SDOQ+7Q0c+E5cy/uCp9UljtN+rt0P9VvOSafGaIsGalE9TfE1pvx+tOP7FPKWzWPh2xdfRLNeVuvuDjpuwbPnznmKaiSBItgkZC/8Niin9vwawaA4HRLSJJT0xCPPX4rtXKSXJ0vt+m5ul7BnhIJ14TPbPYsBAp1PvHM4eAxbu7i9zEU4OIyooF2eowbmJsA7PBYwLoUO9iZ81rndfuwM8rsEwkV1PLc25mN8PRyURUk6iU1FNYLOuifV6Sttz9mqjYZrTqMo1Cp/+jyu83G7L0QOxo/C7bpe4LP3obXZcd5pydyEjmJNWdNydYwHnoBCc4wHnzvb2TbSZxpr7UjH/DQaxVz43FPQ7dJ3JqUtMOxxaHbn3XiAooNf9+BzKMB/st1u3x2WnRiKnFJ40MsjC/zF/uV86PbWrd/mmLGt6nsNbJhLnmn+zPE6f8sSXntyRXXIZfabmVH+c4W3X+eez7bg2fPztEW1S3Kt0H2Q5mCxdL5CsG59no/Hr2w3gasrGASn0s/P631ewiTY6Mfz+VLwW1rP+8BrJYx4jjom9w8Feva3scfhsVA0uODdzl3X5b1bf75LIJxEO0+qvZhyIv0Q/ogIyTb0UFENnLYU1eW9/HgjWIvShmtO7y3XmMdL52XdyO2+Nionl9pRQFf7w9jkA661Ntf5O0W001YvVlhdtHpa8n+p3eM/7z/P5fxyukdMDEV3vQGY62/bduk7ktIXv66oyEV1eYWD9R/igyk6QqFS+nZeB4EG7CfbbX1yxyig6aFIuhmpeLV4z++P30vwct+NjbOX9lZtVpPnOWYuWZ7xsS+q8TDoafr8pf6hYtBC5fPodabHzW1b8GR8w/kJi+o2QSShZ6EiIZQnxbl/eUqKJJACBUTfe+UgCdX3D9cwBk88fKDhIMPrdwjvknZqxxrliQv8mi2qRxggoSDJhjW8X1iv4MDr5XPYk/qb+ciHeUySj64PjUfhgfXwtAuYo31UdDv7FuaG72XuHQF7l0C4qM6JsgaaFFBwHhNyTrI+aD1u4OkFE77G2A8TLHPaaNn9RCbxBfElcXEUf6oWHK/tU93BDaLvz/GDzzvxo/GLx7Q2O4x4PJ8jViCGEj7QIrTqtGf8j/rC3CU+t7GJ9ejxQf9c6JEtvu8Ah5EfM7pnjjEHy3ksjM1X2FHj5ty22+M4gS3A7XGe3cYLe1wWbw/Ybrfvxt+EW1vs8nWPbyeOE1ewl4kbeT/MtwVoPxJ7Rz7peidWmD3fig9zqWVbveLzU7oe81e4aYvfvAZ+G13EB0P2fH6u2nq/R1vw7Hl5rKjmu2cExUKQKtb49d6n9Ff2ECqEW776y4EYd+Ef4zuV9W68JZufv8xT1kegx9eQfq5mfTOurFWSDeagAIf2QVFdi+Q0/iNh0AuGKFSHOGBN4F/WNk+3QhuuO1yt/ZzcesG4/kFTssfMG/EyexD2H7ZhbVcgZQzs/6CHbCs2m3HgS9mT3j4Nru0SSDcxpgK5YJBf/UgByQYdX3DHIPbaPsnqyfAxrzH2w/3rcKbqphbF/hcznqav+Vc7jhXVTxN+MSHsLQpQ2IriBVqs/3OavnYwrnzCt55Wyy+A+LjUjIWWMBfOqVCK42b0hCI7+hLnMNoNBdVCbEK8tHr0+Ni9auOGHVf3F3HVYJB94L0ouBitM8dmlRR1jfXsL/akUaW46Ok1Ftp5rHlKjfWg84At3tNGW++T7e75VjEy2MB37HPmQjePmBunlr+4oUs+vfn0HH+9Cpj39rpv42kLRK2xHU/mUo9vuNYrqvGNaiqqQ/6q37baB0aYo3xGPdncV1ru+mALnj1HdxbV2zdeYhFm1+DAUYH0RKNr6xBg7K+x/1rzseMOc2wdM6/fi+0WTx+bp+fcX+bS9dl93xYcxVNFNe789XnRnwE6Z5Cxcx8VyH2Hh+taz9jbfdGxiohTcIA5dl3Gr1+d7T4FFprjZWqKubSeherZQ+AoniqqVUw/ZDGNBHNUID3R6do6BBh77Ik+X2byP8e+M8fWMfP6vdjuc2CjOV+GzphL12f3fVtwFE8V1SqqVVTfdwy4Wes5OCnJv4wkf8l9Zo7drBjIMLb7kphprcfSIXOJqKbTjQgcxVNFtYpqFdUbRafu6xDg4KRk/ljJ/Bb2kzm2jpnX78V23wKWsuE+9clcuj6779uCo3iqqFZRraL6vmPAzVrPwUlJ+z6T9i3vG3PsZsVAhrHdt4yxbLtt3TKXiGo63YjAUTxVVKuoVlG9UXTqvg4BDk5KzrednO9xf5hj65h5/V5s9z1iL5tvQ8/Mpeuz+74tOIqnimoV1Sqq7zsG3Kz1HJyUhG8jCT/SPjDHblYMZBjb/Uh7Il8uq3PmElFNpxsROIqnimoV1SqqN4pO3dchwMFJyfayyfYl4M0cW8fM6/diu1/CXsnH8+ifuXR9dt+3BUfxHBbVv/766xQm1z9hcO8cgMTF6ctzWdhfHvN71+tW+++VY/dq99b9Uf/zxwBx6bQYA89ffvkFh6s/h0X16hnUUQgIASEgBISAEBACQkAIPBACKqofaDPlihAQAkJACAgBISAEhMB1EDhpUR0m0z9hcO8cCK984L979+Xe7Bf2ih/n5uy9cuxe7T73fmr+7TFDXNqO2RzPGE/UD2s/h69/hEX1hwXn+cMC4Xo5XMP7fPgvcFr/XQ4Bxl68vxzvXwrWzLHLsfvYSmz3S9kv+Xn6GMBcOsZMjT6Kp4pq/frHQ988HRWIQsx+BBh7JdTTJ9SXjilzbD9bLzuS7X7p+yj/98cG5tJlmfx4qx3FU0W1imoV1Y8XF27CIw5OSpz7E6ew62PHHLsJ4q8wgu3W/vb3V7gs48JcWkE/dZlB4CieKqpVVKuonhGYmvYjwMFJCXI5QQqjbRgxx/az9bIj2W7t+7Z9F14VL+bSZZn8eKsdxVNFtYpqFdWPFxduwiMOTkqENREKi9NgwRy7CeKvMILtFh9Ow4eXiCNzaQX91GUGgaN4qqhWUa2iekZgatqPAAenl5jw5PN5iyXm2H62XnYk2y2enJcnj4wvc+myTH681Y7iqaJaRbWK6seLCzfhEQenR05s8u06RRFz7CaIv8IItlv8uQ5/HgF35tIK+qnLDAJH8VRRraJaRfWMwNS0HwEOTo+QwOTDbRU/zLH9bL3sSLZbvLotXt3TfjCXLsvkx1vtKJ4Hi+qf09f3b6dXr99Or95/m76fqkD959v0Jsz56Ucs+L7/9hTX+PjXNYVnfA22GfuiAMnmexLlyNZjuP+YPp6aFzv4tUUg37+EfX2evlOc+Pnn0/Tqyw93NfT9/Le99GP6HPylfrbHSztm7Ec8+xfaef08/WH2+I9PYT+epq//nEf3555/6K/x8d+/nlMs4Xgy6HPKGOj9vw29rsLMYMMcG2tspM+f0+8fWMvT1Oj757fp3eu307s/f46X2NDCdg/9LtrIOSfwZI4rBpvhnBv7JJ7Q+mfU5ansdvNknb357efDPURiLs3SMPM4cqiXr2z7h29Tl+2xD+fJpKMyb29uMizl28Qrn0unKeZccD1+8np2Mlq7ycFZ+yaN5tIAACAASURBVIvzpDk34WnNyMfHimon+EHyC31ycZxI/mP6uFSAY95TFtWNHVsSdSfhwEb4gnPn65Y1Tt83FMVHkvDWotqv18FsYzB3QXHn2C0Cgcg5cbZFdRDx0/S7jTh/P0+vvjxPnztFeUd3L+ISYz/cT2jn9dvJJj1f9J1eH+eef+hv4TJu1P3NBI9zRc0gvvzxieLvinjn/T+NXhs7iq+n37+AE3NsLCwkVtLt1CuqW32HGPDuy/P0blRojBfutrDdvOflHNqw+45ryD1rMF7Bh7Imzed5kvYRnDySX0brja77/LKRTyqqp2lKGqgFbDqv+S5pIZ3bY0PhUnRzkRvmYm2ZcXwY8iVypT3O/UIurnbxYH8e83YppNmndIOMh12+r58HZ6xNXF/7eaioLkXXp/S0xSbEKAyIvwSEfYG7rLP3SXVjxzZBpgAyn/jK07bi67Y1RoFk7/XDmFFgXbLj0ust2YP2LQKJgvvwNL2j4NAU1SGwuOSKxDwIRGvV+GD9GHvsSfMJfcYnCbU47CXzZuxGntrx557frtU/XlNUI2Y+Tx/jt4JtHGr8AJ6bYhHW2f+NY2PHgb3p49XGVObYWEI52X54ml51tWtGNvpGwZDmqEWJGbPxkO0e+jvYy03xdjDHcE3at/6+HufL2vVDv03+kv1xHRXVXYa6IjMWzKYwDsWu0QqeHn/+YgpizBrHcqGNxvbTrdvc2CKftuPWXPFzk2ZX2MnaXLOm7XOgqLYJoSewfK08wv+/0//BqyImeRax5MI8vvJBQaD0+S2/FhLGm7v00o6i2wmI7chJCWtk+8Z33D3f2uBeiur3T+nVlTgvJcBsV/mKxCS94oPF4T9gjK/easGRAhL5BkxG67jrfi4Ezo+f0qs24Qap2ARcCbOyB25evLbTwY3HG/8Lfnk/ytwhQOb5m5u2XvCka1sEAjHGTw4m5U44fTXl7qKNUGPgMWOt2F7aMWM/TKIDXoCTeP2j4aPjBbTyPH3Nr4vha/I0T9KQ5VCZ/7fxKxh2rOPkf+A3il2vp+LrwLd/G22/7X+rZHxs/DfFBmLKm9/+Sq9dQUfxlZq+rcX/+HoN+nybvsbXbhJeNi76/r7QgW3Vjvw1u4sNhBFjg/i1QfPMsbHGkFzTZ9Vvm8DjU2n7mocpLhAjxuusa2G7C18ofpW4aGNl6JOxq3zO++f2PeSp0XXoJecWnt/YwfsOW9N1m+NoLbufsOP9mF9DWx2HkF8qBuCc5arX19P0NWu84tXJ4cZn+HgPn8yldQxMvRyf+Ykxnf/8+0d6HYSux5mMRpbX54dPfB40aor75QlND+g8X+IbBT43I3F4BM8wx/6iGgExi7ErPOpTRGPEVoKxFTWNa/qYRBNIj/YiKmpHAMI72nyexlPAh8DIlqHI0A/vheIcfuVziJptxnmx8b9p8kGrJr307jqd03qYb4SJnxtr+a/f/Rw5CGPvCGPf1wRz9Cf7fKE8P7fvuy0YbhFIDTBemPykOvSzT6t8+5GAAFk/xidjv6idT9/y32gkLXJMaTjmOFiLhMT5ep40h/Oq8zR/5TzP79fPeoOeUSTMvVvKnMcYaKIU1rYw8fx2NjTzpb6ujym4aixBsVN9D3vhx3Ef4FVt8/3buMvtrNvUjvny/MDC7eX6G2nm2Fg5RtOuKGiLatZ3OC9FuLmBHq+13MJ2L2sj/X1R6ee4QFi6NlN8Fu7S3ub+yE1ljZwDm33tXqd81HCb+TXoDz6QD6zNpRzu+8PfqnX28Z7PmUvL7Ms9IpdNLuPCeMR1p580V8yB5YYu3KjNPbXua85rLN/s5Tltvh35F/N36G8fajU+LOfn3Xhmw3YX1Z60gyBIwpgrqkvx10kKzVqUnJp2DtBkR9N/LqjQ2KH4mn4cNHKyzLbx3XVjUw5c/o7bPKFnH0v/tA7P1wTGPB64N+3NzUoNTL0/HuP1eK/bdls8zM89xJx87vXbIpAoSjyRNoGjLZptwAgBwt9VuyQ8igAv4Dpj39ufeM1qB8fliVYtBBsOOQ2AQ7U/c3rp3CfqVr9pPIrCtp39S/ZWe0K7vwabMafVRDjmNfr92S/vR2+etI4fx2vB1voE3fdfbuf+KLL5pqcXTxjL0TlzbCwrU1RP9j1LTvChn9U3n3P/8YpzLWz3yL92LzNHoJNSKJt9RnGDNu7L5yiAUdBSXG32Mbe765RPoj/uGvOrz+XCefKBtc/nwMnfQBtduViRMSQ/h3tw4/2YS3O8q21JD3jXOF4/UFTH3GmKWT6v64ajVkMuZ8bca3Iqn/vJmjO39v0U1R1BQKhWmLgGcTdJog3Mkdg0rhEQgkB+Kty0s4AG86GwLZ/FTis6Dga2zRzTGiU4AA+05ydbbDOfBxxS0MLXXd6OXn8bFLi9zIVglT/xdMIFyBxEeA4kxYLX65pwm7601735rb1zc7t+GwPcloATxYiiOgs/BB1XVAeRmuAxRcH7u+qEj03Mje5fxAXGfriP0EbWn+dqLUobjjmdtzGJObd07jWb9UZ6qQWg12PPN14v9PE+tDa7eZx/KdYAG9wMh/7NOoSn96vGLD+u4w+t7/uzL60dsNXGi3CMmHMKzTPHxsLyRfUUk214SkcJnvTdPoHLWrcxYLzosIXtdvtuY1yzl3n/8nVfRGZseczovOG2KUKNDbzvsDVdT2M8r7ONjj/ML+Y+zvs+8Pw4Z26lb2d4rcFDP+MjfLrHT+bSkHSloVNQh7aYy0ze4nOMH11He/iM+kqFsddQmD9prjyZbs7tRHGy+As9sX/WLfa9+wTbrG3tiLPaNl4mn2/H00+070k1RNqIMgScmgRx99h8DYlCs0kyPmBgHARUE4kXTdPuxNx+/dX0XxCXDR5D0QGTUpjP28g28HmbCP18SEglQZEPPN8oMMKfXjvPgb7hM/UP+z0Kqt7eubnsvL25uX3L+RaB+KK6BobPX+pP6oUAUYNBeuJlz5O8KIF7zb2YM8Z+uG+sHZzH+FLjScMhp3Mk5VoYMKeXzn288vxtbV9qR9FZ7Q9zJB9wrbXZrlM11rlpK3EGWsScbbxrY0mKsx6P1h/G2/eHL/XGmtv53PrGx6lvjSfcPjpnjo3F1WoyJvsPz9Nn85N6Xt9UcGPyFYkZXUefbPfIP8/JnB8LjzL20As4sfWccgfb0t9H4kvWYs3RtZBN16g/PRhr/CQfmIt87m3u6MrFioqjH3ef15lLI86l60kHbc5CvqOiunfzuLqoNnORUe7JdOfJte8+0KHvVM+cfcFf89Q7andsV5hkG551WRztKqpBaC7o2utZSBA7hLSzqC5/KMQCoXME6Gof2bEg2FZoHBBM4oIvNCcnMo9Nnm/Nk14qWgsG9CS4rAesOch1MaqJuBc4YfNsUIT/vB7bx/jk87RHjG8OijR33c/1wW+LQJqiOsSZ8BvVobiLT7CDuI1A408U2XPIyn69XK+9tCPGvtVV3kfmhika3E16l8N48tkmUuZ0/xzfBC0VicxRPu9wsvGLx7Q2V4xyX/yNRil82utJp1XLTSxgLea5PB6YF0Vya5uPB2hHf+Bn7OjuF9oHWGzUPHNsrLG2qJ7wbdRrvFdK+h4m4JTku4XJ2ADXwnbXfSceNRwK7YQd+mTs0r5WXpf+yA3Iw+AWxpd2b4PnSWrzXOjYxGuwzdwOG0Y+cH5B/2wz2+PPK1f35JDh3hRNerwu3Z+55IjmThJv3SsfnfbE6xmOu6I1TMB9l9ahp+I0X7rZrb+RnXLwuBD2ebtd27bbY+e6OVmPpxlkDncU1SAogqMhFIiOwOieaKb+IHsoVEKxhnN3h4t5WDD4ZYxQ5LgAAJvyk/LOX/qWQIMn6VgjP21fFptdIz85sjZgvnKNAl8JImnsm0/P8VdCsG4XhxxI0lcdz9PH+Ff59UlcCZbZh1pwmyAX2rAfbr6aDEMQ2BQ4sR6Ccgwu2d+yHvtvbkQwvmBF9sZ24ycl5y1Ba4tA+oJLyTgFonBcxc3iN7rKX6X1C27X74FPGPvhvjXasXywccZqkP+iH22VN8zp0Tl+8SboDHqErTVu8FPUDr97SRa+dTnf2ox18U2Uj3Mp1sKmEjPtGllT6JNuSvq2ejzQB79mkmOp+x/v5D5Zn/iVlTk7ih/Z/9KXY8ZOzTPHxnLqFdVI7iiqvb778SCtsJTox3akFra77DtzyO4tOBQ+XeyseTTmivd5DxH3S3y3ewrupXxUcgSv78bmvtEOq8vEy1X5qNiE9atekQP7PhjuYQ7CZqxdjhWw9zE+mUtD7sWbRLuH+dg+jbZ97HU7KRXBqSkXs+DoaKyZJ+or9+fXOJK+YOtSHl1aO+fwuFbN38YUd7gaTzeqnuwoqh+DiMMg1gkq6nu/e35UIFUqOtqKAGMvHd2vjm5175hjWzl6rf5s963iK7tuX7PMpWtx+lHWPYqnimoV0Q/3v221ieCoQB4lUFzDD8be7ouObz9Z38MeMceuwfM9a7Ld94C1bLxNzTKX9vBRYyoCR/FUUa2iWkV11ZOOTogABycl5dtMyve8L8yxE9L3rFOx3fe8B7L9urpmLp2VuC9g8qN4qqhWUa2i+gUEimu4yMFJyfe6yfcR8WeOXYPne9Zkux9xb+TTZfTOXNrDR42pCBzFU0W1imoV1VVPOjohAhyclGQvk2RfEs7MsRPS96xTsd0vac/k62njAHPprMR9AZMfxVNFtYpqFdUvIFBcw0UOTkqmp02mwvP4b8peQxdhTWlDWjiVfplL1+L0o6x7FE8V1SqqVVQ/SjS4MT84OJ0qiWgeFSTgAHPsxiQwNIfthj/6FLe3coC5NCSdGlYhcBRPFdUqqlVUr5KaOm1FgIPT1mSh/iowljjAHNvK0Wv1Z7uX/FS7tDDiAHPpWpx+lHWP4qmiWkW1iupHiQY35gcHp1FS0HUVDHs5wBy7MQkMzWG79/qvcdIOc2lIOjWsQuAonsOi+tdff43vfYUF9E8Y3DMHoCRx+vI8FvaXx/yetbrH9nvl2L3avWePNOa8cUBcOi2+wPOXX37B4erPYVG9egZ1FAJCQAgIASEgBISAEBACD4SAiuoH2ky5IgSEgBAQAkJACAgBIXAdBE5aVIfJ9E8Y3DsHwisf+O/efbk3+4W94se5OXuvHLtXu8+9n5p/e8wQl7ZjNsczxhP1w9rP4esfYVH9EYT+COLeORDe5cN/gdP673IIMPb3ziXZf3vxkDl2OXYfW4ntFrduj1v3sifMpWPM1OijeKqo1q9/PPTN01GBKMTsR4Cxv5ckJTvvp8Bhju1n62VHst3i3P1w7tb2irl0WSY/3mpH8VRRraJaRfXjxYWb8IiD060lI9lz/4UMc+wmiL/CCLZbXLx/Ll5rD5lLK+inLjMIHMVTRbWKahXVMwJT034EODhdK+lo3cctWJhj+9l62ZFstzj6uBw9994yly7L5Mdb7SieKqpVVKuofry4cBMecXA6d3LR/C+vMGGO3QTxVxjBdou7L4+7p9pz5tIK+qnLDAJH8VRRraJaRfWMwNS0HwEOTqdKIppHBQg4wBzbz9bLjmS74Y8+xe2tHGAuXZbJj7faUTxVVKuoVlH9eHHhJjzi4LQ1Wai/CowlDjDHboL4K4xgu5f8VLu0MOIAc2kF/dRlBoGjeKqoVlGtonpGYGrajwAHp1FS0HUVDHs5wBzbz9bLjmS79/qvcdIOc+myTH681Y7iebCo/jl9ff92evX67fTq/bfp+80UqD+mjxezCRg8T380/qMtYxRsMv/e/PZzf0H7z7fpTZjr04/9czT2Pl6A2iOQ71/8Pn3+2wSOv5+nV6+fp+/m0ikP49pffuQpf06/f8i2fPkrHr/78+fplvv5bXr3+mn6/YRTWuMY+2EB8FfANPvp+Jx1HNuyvk7I+++/PcV1P/51at5D90/T13/83FhzUfsn9HOIe9E/7K28Pz0mHodlm9b1Z45Z/vnjH9PnwKOiLbQmjTmNT9MUdOiuRa28nU6lP7Z7iAd48Nrnlz8+hb1q+TWcp+z1OlzPPf8aO6GVEBu8XsBXi0mNFbbvaj+As4s/Y6xgm9dJW3esXn/j/lj8mEtgdvcz87jEW9aDbf/wbRqlBp+naKVeXol5s8aXwF2bd1zOZZto+nOfbsKzY8yxohpEjEnvfAK3BFp33JJ7adwfn/ba3xN4R4zAaqVol+z999TzHRD1oq1XnHubQHIRa4NJDjIlwV6yqHZrJduOJnUXDHvBrxMk9l5i7Ic8sUW1vTm311FUnJD3/cTY0e4O/mJum+CD/6uT7An9HOIe/UL8qvEPtvuC4TS4zNuyfQ3m2JiruaimZD5NvaI6XPNJ/+efT9O7L8/TOxsbxosttrDdQ1zAAyosV/NoB3c38XTn/EN/y3zzObzx38aKkmPBbVt8DzgGnMvYQb9sX18jrc2Nnf/9mD7aGFf8nV9vDi/m0ph8SQMll03pvOYUm2PsMc2IArlb/KZxXDQH/bQ3tGne2FZ0xTbR2hc4XY9n35hDRXUh1qf0pIkTyBwRbqmtJf4Wgq8U7kbRLuJz6vlOIO5Fm6+wxiaBuCK2CsaJftCn9j7hUVirBJvTzOuK6tNMOZyFsR/ywyZE8/QtxZen6U38NmxFYrwCv4Y+dfXZJt1t47fEpQN9u7YfmO+M+8IcG5IRBcSHJ9JUKgJqoTFNU7jZdLoLiT4U2VyUjFdbamG7F3lAD66O5azlvTz3/EN/C1fmtYLaAzUHzuMTWBStW3i8pe9/04T1/I3nvM3/hoI67CPsK74u78ccXsylJe7ZdpcP+CFLN/8EvTxN78I3qL2iOo55ar4BDevU4t1akL4VsnM5m3zXi5wdwTMYeKCotsXkmCxJnPWxvyUhiImvIiAQEIjb7djQZ9ze2sN9sdbourObRWCLgE/f8iswC0l/JFo7VxCcu1POfsSAGjA0a9B8sLdghHaMdT60+ADzR/vcIpBVYuaiOp5XfrvA4dr8k6+4FvbGJPBiQzO2fXLg5rABzo2twSzeHGDN0J+DaC48oMdaaOS1w5M6jF/xCgxjP+RW1sCbT8/xlaakzRxf3j9PH21RDV4XncxopBMjoHsbP5Jmqia+xq/W054WPcUEaNd6mj5+mnt9BH2NZqH1NbaTn1zkIG5Z+xADXGGREzf6Y28tDrwv6DvXh8dc65w5Ns66KIj5SVhbVAedNDrOGi36HC+0qoXtHuIHHkB3mTvMBz7HHiZ+1Fz9Nb/yFHnw6Uf+5iRx3e53me+30atZ+NYlxz6bX2Dzp+dURNq8ZQtJ9CPfSvGJ6+ZGu+CEsREPxIpvU9Ju/tYFcaW8Zgkcss1Fh9NUv/lFPg996rc3ZV3Sk9VfsdtgUXD8h9YucyNOZJscVjUmzb1ay1xaRcDcyfE55g3zaiOfh/vNrA03riwYtPU8fW/ySquxMiTPWR8eQae2x2WPj+AZLN1fVDtSQ2BEwkxqFnZ89ziPh5Ar+dJdmw8K0/QvCYTb/Xgio7OjtdWP5fY8VxFgf25X8NrAgWPCK4qUMOj7lJMyjzfnGAcs//2vBpkoRsKuJ34OGo9yvl4gbdHalbILNCEAmGJ5qQ3Fc+iH4/z+JpK4C1ZuPm+fe3qev8JOc8zZRE8FXPDz809xbfiW2uq75PncFvIdsBj7IacKP7+lJBy1lnT25rcf/qbV8D7Ml7Q7rxHognUO3aT4lHXNya6bIEOMQv+3k0+s9alTWq+2+3PSKPlVk3z6m4l524ED4m+2DTErzz3Cwe4LbDz10zS7ximPmWMdGuZLJll3dFVvINv3qYMmoc90I2oKj/GCsy1s9xCTwgsUe2mPmQ987rldCzqfi/GuMtrBH/AJ7fUBFrju1+vzbTYnFr/wN0F5jqI3Pq+6SljZdrO+yfV9DHysgCaK3kpRa+fntSseuEl1n8UH4Ahc2zkTjt6m+mCt7d/jCXNplni2McZ/87cDlJdargcNJe67PJXnRMG99LAmYFX0BHuiJtONjH3XGs2X/NyNZzZyd1HtCdsWvXHzM8FxV+wIAVGFu1FDwtSHEk4sTO01BIFMRhSu5bNPRticBACibyR+KQDwR4ZLtmRBwl8kumJrxQ7C9IEr3cHiWsEQ8+Fu3s0Lm+bvtstc1pYHO14vECoqRyp2Cdl3ioVueYKbkrj9Wqv0zgHEJnK0uWDl1rL22WOM7H96m2aKaldgh7nsGvY4rePs7C89MfZDvhlNlSRTkiO43E88qf8KjTQxpibGpC2OGbQuimgzD+JJo01oCPHP3CS0hUZel3UMfWdd+yJmyXbENMKsG2trwQB/alKvbcO9g69X+mSODag4TflbGGiu8jdxG9dTP1s010Iizc39xyvOtbDdQ3wtD3D8/pt/IltuLmtOw14mboLLtZ35tHReik7LZaOFNJ745nKS51Kyr9oT/PfXWI9+fHlwFIrgEivsE2c8hR/ZlDGBD8DW2MyY2D3y+MK21mY/x6h9EL9Waoq5NMe72tbJTyHfmIc9XFQHzUAnVT95xpA/MJZzCRfvdB7nwliXd6q1lzzah2e1cGdRDZFmwobNBylB0kgI9EukiUWjaU+EM22zd4mYK6zZktMSvjxFwlpIbnl+T3QkIAg8z40kVz5TeysmaxfE1fkEPka0BbP8dKyZG2OKDfXJQR0L/MxeBOyLz2ivT8w8Vh1bV4r5HubZIpAmUFSd1KNOoYuboXfhfc1SVIchOXBh/+yT3VxYY2w3WHXWqk+ja4CrxoWjlPQxL9vkfLTBz61V50nrXa6oRmKt71GTvqAJ6AjnwLj7B11VA7ao9XrjmLKwbikC5nRl5oSdsBvfJsFebqdzjlld2w0Gaf8R0xDj+jhAx7wGrt/y53p9Jy1CZ6lgCBqiIjlooiT49HU3tOQ+TR+vv3VnbPcQ4y4PsI+8v/Xc84O43CnCee/53OfU+RxZclPheptj2vlRVENPRjuDfAQf37wPcRe+Z1/fP6Vfx0INABwbjYyKbrbH+4C1/Q11a7P3s20vWBm7ytPzgd/MFebSMgM7BXUYxDnAnpMuXB7hX8uxeWVgTB1PuuzZMZjjXJe34+kt2VdUDwkaxA5yexJa8rSkMSKNQoQw7M/02WttkPBE8+T1xEaCqXb6dj/Wz1uL1erDki0ZB2BmAg0Lk8/t2mgr+GK+EDRyAV1t8tgn/8LeUOG9UrTWjns73iQQG0SsTuz10XHob9vs+HAcA02/ELZPk2uw4flsYWuPaSG2gc7d/Db42eM4pV3DHqf13DxkAk4Z+yF3LH/B65BkYkIkfaHd6AjzNhpx/OYYw0mTdU/rdm7ksZ5PrD3tdd6/Zj8Wzn2MWrLd2wB80meLg2+fG3t7bcwxcK/9bJN31N2H5+nzh6rLcK1+NZ14XwpxTNpoBQ3rP9nu4R6MeBGLsFEOY34wl5dyYNte8nfUHWuFeME2Ox2mvkk71f7gv7+2sEaY0z44QvFcbhjyjQfixJJNnXbWnN2jvvZbm/0cbXtvzpLjO7jZ/jhmLs2zMOmgctz0jrw239KE3JFvHlOOws2c+YztuUg3Nwa4AW20k5cL+WP4gIhylrHwIofb8GxN2lVUg1BcxPF1nJekk0UQz20iDeTJpMaco7Gj9tQfIvXkRVEZ14V4TPHvx3JA8XP5O3Yr7IWCFetC5CWI4OlzXuc17tRHgbB/Zz0v3jxXCTzsEwXFlWKGqG/5c5tAUhJFIElyoSBkBR+PB+8dN4k3zJP6pmRufgPUBC9XrNq16GsxnqOMm7MpP1Eor6Q4G6lw7sxjA3FZr40p5QpjP+SJiwVVB0nrpAOnI2orCTVrxM27FGNYE6O5KcYUvQ40lG1IScbECPiRNYkYVV69QHuOFz4ewjbEioWYtYAD9sXHkIE/NxYbmGOFfM1B0rFP8lnvr1FUh3PoGTfCpsgoc5JWyvX1B2w39qD5JB6EdnDBFl+4lnIt8wPnlX+81/3z+ofzfv4FvnVsXvaL9cfnHT5inVDMmbxqi+1Se5Rvhvr5E/VHxXR+fcYj+deO8bjmfSh5eLQv2Kd2vgbH/6bmNbsxCzPf7TemrrPltT12neLJbPx3eYUfDuG86izOVb75WbKxteXUV1ibW+ffUVSDCEguhuwgeSGNDQDp7gZFcSSHSzgkDBc80tgqkLQmiI27otrOZMzn+U7qY/4JwNIfdhtxJjHgjgwkz766/vgDEurDyQdjrPiL0DM27tcPwlrAGnYYzHk+PseTtXL3aO1jfMwest13fr5HIFHkBTck3CytTqGb+Pc0/f43/c9UYl/sHe7M0zx+DQowCHqdtZrCFnZyUIrXZ2wK/Tn40esqtQBpA+xsUM1QMfa9hGDjAGJDNxHhW5aG5zMaCdydiTGIHykOsCYwb0c3EdvnCb+kUOJITyuwN4xx2qfYiF85QezEuDLGxrD+2rMxawYH7IvH/T5iAnNsnAB7RTUSPDQe+tQieo7j6cld7Ttet9/CdmMPms+GB2FfwAWTD8q1EG+YHy2Xea9H5/iFmxDjoE/YOORb1+YOn9APMaxw3fgIPfS0ZfKj02CZ1+IT5gQOOSbbuTGm/GJJwrH9H7olP3zsgG8cQ/jmw2s+2Uw2mYd9ZZ+tnR0cmEt9xuEmseYj1E3uIVLMCblPySntjHPaaP9QseoMa9bcEubOhTR4gPzXLnuRK6vxHFizo6gGgfSJ4KLP2+XCUYEMdKPLKxBg7B9NJymxcuK+XS08Gv7BH+bYClreRBe2+xH3Rj5dJhYwl26C4HdsxFE8VVR37vwUDC4TDC6B81GB3HFsuLrpjP0l9vt8a9QnSngqlp7U2SfZj6Ob8+F4WoyYY1cn/UoD2O57wVt2npa/p8CTubSSguo2QOAoniqqVVRPpxD2rc5xVCAD3enyCgQY+1vlyGq78BUxvqZ0X9XeXrJd7dcdx0Dm2Apa3kQXtvsl7JV8PE+MYC7dBMHv2IijeKqovuOEoiC1HKSOCuSOV6LzsQAAAn5JREFUY8PVTWfsxddlvgqjbRgxx65O+pUGsN3a9237LrwqXsyllRRUtwECR/FUUa2iWk+qB+LS5WMIcHBSIqyJUFicBgvm2DHGXm402y0+nIYPLxFH5tLlWPyYKx3FU0W1imoV1Y8ZG67uFQenl5jw5PN5iyXm2NVJv9IAtls8OS9PHhlf5tJKCqrbAIGjeKqoVlGtonogLl0+hgAHp0dObPLtOkURc+wYYy83mu0Wf67Dn0fAnbl0ORY/5kpH8VRRraJaRfVjxoare8XB6RESmHy4reKHOXZ10q80gO0Wr26LV/e0H8yllRRUtwECR/FUUa2iWkX1QFy6fAwBDk73lKhk630UOcyxY4y93Gi2W3y7D77d4j4xly7H4sdc6Siew6L6119/jT+sHxbQP2FwzxyA9MXpy/NY2F8e83vW6h7b75Vj92r3nj3SmPPGAXHptPgCz19++QWHqz+HRfXqGdRRCAgBISAEhIAQEAJCQAg8EAIqqh9oM+WKEBACQkAICAEhIASEwHUQOFxUh6/HwyT6JwzEAXFAHBAHxAFxQBwQB14yB7aW8+71j62D1V8ICAEhIASEgBAQAkJACAiBaVJRLRYIASEgBISAEBACQkAICIGDCKioPgighgsBISAEhIAQEAJCQAgIARXV4oAQEAJCQAgIASEgBISAEDiIgIrqgwBquBAQAkJACAgBISAEhIAQUFEtDggBISAEhIAQEAJCQAgIgYMIqKg+CKCGCwEhIASEgBAQAkJACAgBFdXigBAQAkJACAgBISAEhIAQOIiAiuqDAGq4EBACQkAICAEhIASEgBBQUS0OCAEhIASEgBAQAkJACAiBgwj8fyOV4UGr/g8QAAAAAElFTkSuQmCC"}}},{"cell_type":"code","source":"import os,re\nimport seaborn as sns\nfrom tqdm import tqdm\nimport pandas as pd\nimport numpy as np\nimport tensorflow as tf\nprint('tensorflow version:',tf.__version__)\nfrom tensorflow.keras.preprocessing.text import Tokenizer \nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\nfrom tensorflow.keras.layers import LSTM, InputLayer, Dense, Embedding, Input\nfrom tensorflow.keras import Model\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pylab as plt\n","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:22.151373Z","iopub.execute_input":"2021-07-05T09:41:22.152221Z","iopub.status.idle":"2021-07-05T09:41:22.161833Z","shell.execute_reply.started":"2021-07-05T09:41:22.152161Z","shell.execute_reply":"2021-07-05T09:41:22.160928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#import datasets\nimdb=pd.read_csv('../input/sentiment-labelled-sentences-data-set/sentiment labelled sentences/amazon_cells_labelled.txt', sep=\"\t\", )\nimdb.columns=['text','label']\nprint(imdb.shape)\namazon=pd.read_csv('../input/sentiment-labelled-sentences-data-set/sentiment labelled sentences/imdb_labelled.txt', sep=\"\t\", header = None)\namazon.columns=['text','label']\nprint(amazon.shape)\nyelp=pd.read_csv('../input/sentiment-labelled-sentences-data-set/sentiment labelled sentences/yelp_labelled.txt', sep=\"\t\", header = None)\nyelp.columns=['text','label']\nprint(yelp.shape)\n#concatenation \ndata=pd.concat([imdb,amazon,yelp])","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:22.163549Z","iopub.execute_input":"2021-07-05T09:41:22.164247Z","iopub.status.idle":"2021-07-05T09:41:22.194637Z","shell.execute_reply.started":"2021-07-05T09:41:22.164202Z","shell.execute_reply":"2021-07-05T09:41:22.193641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.head(5)","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:22.196956Z","iopub.execute_input":"2021-07-05T09:41:22.197635Z","iopub.status.idle":"2021-07-05T09:41:22.208752Z","shell.execute_reply.started":"2021-07-05T09:41:22.197589Z","shell.execute_reply":"2021-07-05T09:41:22.207582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(data.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:22.210612Z","iopub.execute_input":"2021-07-05T09:41:22.211065Z","iopub.status.idle":"2021-07-05T09:41:22.220036Z","shell.execute_reply.started":"2021-07-05T09:41:22.211023Z","shell.execute_reply":"2021-07-05T09:41:22.21859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#cleaning text\ndef clean_text(dataset):\n    \n    for i in range(dataset.shape[0]):\n        sentence=dataset.iloc[i,0]\n        sentence=re.sub('[^A-Za-z0-9]+', ' ', str(sentence).lower()).strip()\n        dataset.iloc[i,0]=sentence\n        #print(sentence)\n        \n    return data\n\ncorpus=clean_text(data)","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:22.221965Z","iopub.execute_input":"2021-07-05T09:41:22.222822Z","iopub.status.idle":"2021-07-05T09:41:23.136968Z","shell.execute_reply.started":"2021-07-05T09:41:22.222759Z","shell.execute_reply":"2021-07-05T09:41:23.136042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(corpus.shape[0])","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:23.138246Z","iopub.execute_input":"2021-07-05T09:41:23.138534Z","iopub.status.idle":"2021-07-05T09:41:23.143566Z","shell.execute_reply.started":"2021-07-05T09:41:23.138506Z","shell.execute_reply":"2021-07-05T09:41:23.142503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corpus.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:23.145133Z","iopub.execute_input":"2021-07-05T09:41:23.145559Z","iopub.status.idle":"2021-07-05T09:41:23.162916Z","shell.execute_reply.started":"2021-07-05T09:41:23.145518Z","shell.execute_reply":"2021-07-05T09:41:23.161925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Balanced dataset\nsns.countplot(x=\"label\", data=corpus)\ncorpus['label'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:23.164266Z","iopub.execute_input":"2021-07-05T09:41:23.164688Z","iopub.status.idle":"2021-07-05T09:41:23.292044Z","shell.execute_reply.started":"2021-07-05T09:41:23.164659Z","shell.execute_reply":"2021-07-05T09:41:23.290973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokenizer=Tokenizer(num_words=1000,filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n')\ntokenizer.fit_on_texts(corpus['text'])\ntext_matrix=tokenizer.texts_to_sequences(corpus['text'])","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:23.294848Z","iopub.execute_input":"2021-07-05T09:41:23.295271Z","iopub.status.idle":"2021-07-05T09:41:23.626659Z","shell.execute_reply.started":"2021-07-05T09:41:23.295228Z","shell.execute_reply":"2021-07-05T09:41:23.625728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len_mat=[]\nfor i in range(len(text_matrix)):\n    len_mat.append(len(text_matrix[i]))\ntext_pad = pad_sequences(text_matrix, maxlen=32, padding='post')    ","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:23.628323Z","iopub.execute_input":"2021-07-05T09:41:23.628714Z","iopub.status.idle":"2021-07-05T09:41:23.673129Z","shell.execute_reply.started":"2021-07-05T09:41:23.628671Z","shell.execute_reply":"2021-07-05T09:41:23.672187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Model1 without attention\nvocab_lenght=3000\n\"\"\"\nmodel1=tf.keras.Sequential([\n    InputLayer(input_shape=(text_pad.shape[1],)),\n    Embedding(input_dim=vocab_lenght+1, output_dim=32, embeddings_initializer='uniform'),\n    LSTM(100,dropout=0.3,recurrent_dropout=0.2, ),\n    Dense(1,activation='sigmoid')    \n]\n)\n\"\"\"\ninputs1=Input(shape=(text_pad.shape[1],))\nx1=Embedding(input_dim=vocab_lenght+1,output_dim=32,\\\n             input_length=text_pad.shape[1],embeddings_regularizer=tf.keras.regularizers.l2(.001))(inputs1)\nx1=LSTM(100,dropout=0.3,recurrent_dropout=0.2)(x1)\noutputs1=Dense(1,activation='sigmoid')(x1)\nmodel1=Model(inputs1,outputs1)\n\nmodel1.summary()\n","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:23.674546Z","iopub.execute_input":"2021-07-05T09:41:23.674973Z","iopub.status.idle":"2021-07-05T09:41:23.822408Z","shell.execute_reply.started":"2021-07-05T09:41:23.674931Z","shell.execute_reply":"2021-07-05T09:41:23.821338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model1.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'] )","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:41:23.823644Z","iopub.execute_input":"2021-07-05T09:41:23.823974Z","iopub.status.idle":"2021-07-05T09:41:23.836392Z","shell.execute_reply.started":"2021-07-05T09:41:23.823943Z","shell.execute_reply":"2021-07-05T09:41:23.83539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Training model 1 without Attention\nx_train=np.array(text_pad)\ny_train=np.array(corpus['label'])\nhistory=model1.fit(x_train,y_train,batch_size=64,epochs=10,verbose=1,shuffle=True,validation_split=0.2).history","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:42:35.241941Z","iopub.execute_input":"2021-07-05T09:42:35.242302Z","iopub.status.idle":"2021-07-05T09:42:52.758137Z","shell.execute_reply.started":"2021-07-05T09:42:35.242274Z","shell.execute_reply":"2021-07-05T09:42:52.757094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# About 77% of accuracy has been observed with basic LSTM and without Attention. Lets now implement an Attetion layer and add it to the LSMT-Based model. ","metadata":{}},{"cell_type":"code","source":"plt.figure()\nplt.ylabel(\"Loss (training and validation)\")\nplt.xlabel(\"Training Steps\")\nplt.ylim([0,2])\nplt.plot(history[\"loss\"])\nplt.plot(history[\"val_loss\"])\n\nplt.figure()\nplt.ylabel(\"Accuracy (training and validation)\")\nplt.xlabel(\"Training Steps\")\nplt.ylim([0,1])\nplt.plot(history[\"accuracy\"])\nplt.plot(history[\"val_accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:43:01.591467Z","iopub.execute_input":"2021-07-05T09:43:01.591853Z","iopub.status.idle":"2021-07-05T09:43:01.881176Z","shell.execute_reply.started":"2021-07-05T09:43:01.591819Z","shell.execute_reply":"2021-07-05T09:43:01.880254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras.layers import Layer\nimport keras.backend as K\nclass attention(Layer):\n    \n    \"\"\"This class implents the attention mechanism layer: We will define a class named Attention as a derived class of the Layer class. We need to define four functions as per the Keras custom layer generation rule. \n    These are build(),call (), compute_output_shape() and get_config().\"\"\"\n    def __init__(self,**kwargs):\n        super(attention,self).__init__(**kwargs)\n        \n        \n    \"\"\"Inside build (), we will define our weights and biases, i.e., Wa and B . \n    If the previous LSTM layer’s output shape is (None, 32, 100)\n    then our output weight should be (100, 1) and bias should be (100, 1) dimensional.\"\"\"\n    def build(self,input_shape):\n        self.w=self.add_weight(name=\"att_weight\",shape=(input_shape[-1],1),initializer=\"normal\")\n        self.b=self.add_weight(name=\"att_bias\",shape=(input_shape[1],1),initializer=\"zeros\")\n        super(attention,self).build(input_shape)\n        \n        \n    \"\"\"Inside call (), we will write the main logic of Attention. We simply must create a Multi-Layer Perceptron (MLP). \n    Therefore, we will take the dot product of weights and inputs followed by the addition of bias terms. \n    After that, we apply a ‘tanh’ followed by a softmax layer. This softmax gives the alignment scores. \n    Its dimension will be the number of hidden states in the LSTM, i.e., 32 in this case. \n    Taking its dot product along with the hidden states will provide the context vector:\"\"\"    \n    def call(self,x):\n        et=K.squeeze(k.tanh(K.dot(x,self.w)+self.b), axis=-1)\n        at=K.softmax(et)\n        at=K.expand_dims(at,axis=-1)\n        output=x*at\n        return k.sum(output,axis=1)\n    \n    def compute_output_shape(self,input_shape):\n        return (input_shape[0],input_shape[-1])\n    \"The get_config() method collects the input shape and other information about the model.\"\"\"\n    def get_config(self):\n        return super(attention,self).get_config()        ","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:53:05.873916Z","iopub.execute_input":"2021-07-05T09:53:05.874468Z","iopub.status.idle":"2021-07-05T09:53:05.885704Z","shell.execute_reply.started":"2021-07-05T09:53:05.874419Z","shell.execute_reply":"2021-07-05T09:53:05.884649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#The new model with attention mechanism\ninputs=Input(shape=(text_pad.shape[1],))\nx=Embedding(input_dim=vocab_lenght+1,output_dim=32,\\\n             input_length=text_pad.shape[1],embeddings_regularizer=tf.keras.regularizers.l2(.001))(inputs)\nx1=LSTM(100,return_sequences=True,dropout=0.3,recurrent_dropout=0.2)(x)\natte_layer=attention()(x1)\noutputs=Dense(1,activation='sigmoid',trainable=True)(atte_layer)\nmodel=Model(inputs,outputs)\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:56:25.291749Z","iopub.execute_input":"2021-07-05T09:56:25.292342Z","iopub.status.idle":"2021-07-05T09:56:25.452025Z","shell.execute_reply.started":"2021-07-05T09:56:25.292308Z","shell.execute_reply":"2021-07-05T09:56:25.451104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']  )\nhistory1=model.fit(x_train,y_train,epochs=10, validation_split=0.2,verbose=1,batch_size=64,shuffle=True).history","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:58:57.671824Z","iopub.execute_input":"2021-07-05T09:58:57.672416Z","iopub.status.idle":"2021-07-05T09:59:19.087554Z","shell.execute_reply.started":"2021-07-05T09:58:57.672367Z","shell.execute_reply":"2021-07-05T09:59:19.08669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure()\nplt.ylabel(\"Loss (training and validation)\")\nplt.xlabel(\"Training Steps\")\nplt.ylim([0,2])\nplt.plot(history1[\"loss\"])\nplt.plot(history1[\"val_loss\"])\n\nplt.figure()\nplt.ylabel(\"Accuracy (training and validation)\")\nplt.xlabel(\"Training Steps\")\nplt.ylim([0,1])\nplt.plot(history1[\"accuracy\"])\nplt.plot(history1[\"val_accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2021-07-05T09:59:50.554653Z","iopub.execute_input":"2021-07-05T09:59:50.555043Z","iopub.status.idle":"2021-07-05T09:59:50.827388Z","shell.execute_reply.started":"2021-07-05T09:59:50.555012Z","shell.execute_reply":"2021-07-05T09:59:50.826664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Accuracy is now better about 0.8145\n# Please pay more attention to the attention mechanism !!!\n# Enjoy !!!","metadata":{}}]}