{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30776,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\n\nimport keras\nfrom keras import ops\nfrom keras import layers\nimport keras_cv\n\nimport math\n\nfrom IPython.display import display, HTML","metadata":{"execution":{"iopub.status.busy":"2024-10-05T15:05:42.460122Z","iopub.execute_input":"2024-10-05T15:05:42.460868Z","iopub.status.idle":"2024-10-05T15:05:42.466447Z","shell.execute_reply.started":"2024-10-05T15:05:42.460823Z","shell.execute_reply":"2024-10-05T15:05:42.465535Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Introduction","metadata":{}},{"cell_type":"markdown","source":"### 1.1 SimCLR: What for?","metadata":{}},{"cell_type":"markdown","source":"This notebook is based on the paper [SimCLR: a Simple Framework for Contrastive Learning of Visual Representations](https://arxiv.org/abs/2002.05709).\nThis framework allows to learn useful representations in a self-supervised manner. \n\nSupervised learning for vision tasks is the mainstream approach. Labeling large amount of images is cost-expensive and time-consuming. Transfer learning is generally a good solution for mitigating that issue. Contrastive learning may provide an alternative solution.\n\nHow to use it?\n- Pretrain the encoder to learn useful representations without labeling images, using SimCLR.   \n- Then fine-tune the model (the pretrained encoder and head on top of it) for a specific task in a supervised way. A small amount of labelled images is necessary for this fine-tuning.\n\nAnd if you do it right (that is to say you manage to select the best hyperparameters which is not an easy task as there are a lot), when fine-tuned on image classification datasets, SimCLR performs on par with or better than a strong supervised baseline.","metadata":{}},{"cell_type":"markdown","source":"### 1.2 SimCLR: How does it work?","metadata":{}},{"cell_type":"markdown","source":"The central idea behind the paper is quite simple, allow the model to learn if a pair of images were derived from the same or different initial image:\n* SimCLR generates 2 different views (Xi and Xj) for each image (X) by applying random transformations. \n* Different views from the same image are referred as `positive pairs` \n* Views from different images are referred as `negative pairs`.\n* An encoder network (f) and a projection head (g) are trained to maximize agreement using a contrastive loss.\n* Projections (Zi and Zj) of `positive pairs` tend to be close whereas projections of `negative pairs` tend to get distant.","metadata":{}},{"cell_type":"markdown","source":"![my_schema.png](attachment:b5badde7-da44-4108-aba9-4337e0501707.png)","metadata":{},"attachments":{"b5badde7-da44-4108-aba9-4337e0501707.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAABEAAAAG5CAYAAACOWkE5AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAEnQAABJ0Ad5mH3gAAHMUSURBVHhe7d0JnFxlne//36nqJUt30t3pBAKEJYusQhIQEFAEhAB6x/G6BRmvLI5KrjjeGS84Ki91cEscFy5KREZF/zqA4HYFhGEUhVxHWRKWBBCyACFkTzrpdHqtqv/5nq4nVE5O77We+rzDoU+tp+rU2Z7veZ7neBmfAQAAAAAAxFgi+xcAAAAAACC2CEAAAAAAAEDsEYAAAAAAAIDYIwABAAAAAACxRwACAAAAAABijwAEAAAAAADEHgEIAAAAAACIPQIQAAAAAAAQewQgAAAAAAAg9ghAAAAAAABA7BGAAAAAAACA2CMAAQAAAAAAsUcAAgAAAAAAYo8ABAAAAAAAxB4BCAAAAAAAiD0CEAAAAAAAEHsEIAAAAAAAIPYIQAAAAAAAQOwRgAAAAAAAgNgjAAEAAAAAALFHAAIAAAAAAGKPAAQAAAAAAMQeAQgAAAAAAIg9AhAAAAAAABB7BCAAAAAAACD2CEAAAAAAAEDsEYAAAAAAZWDJkiXmed5+w8KFC+3JJ5/MPmN09B56r7G+TxS9p/ucAFDuCEAAAACAMvD73/8+O/aaO+64w8455xzbvn179p7ScWHHhRdemL0HACoLAQgAABXkM5/5zL4zw254wxveYMuWLcs+A1FaWlqCAagEixcvtkwmY2vWrAlu79y505599tlgfDRuv/324P1OOumk7D2j097enh17jd5T761pAEC5IwABAKCCPP7449mx1zz22GP2pje9ydauXZu9Jz5c1f2RBDwKOvQaoNK9+uqr2bF+ronMzTffbLNnz94X6qlmhltXNOgxPcdxj7n7VJtEYapbV8LPz30/PUfP1Tqo7Yzcf//9wWP6PLpf47m1Qob6PO75er37DHq+q+Wi1+tx93o9BgD5QAACAEAFyj1DPGvWrOC+cGEpDtra2rJjw6ez5WE7duwIBqASXHvttUHB3wUOp5xyip111lnBuHz0ox8N1n0t6wo+1URGTWUcPabnuFoZbj3atWtX8Pe6666zL3/5y/vWFff8e++9Nwgh3vWud+17Pz1Hz33qqaeC20MZzucRhSj6nu4z6Pl33XVXMK779finP/3pYFDICwD5QAACAEDM6IysmsXknj11tUPcGWQVRNwZVp2ddQUTdzZXZ3zd4zobO9QZ49wzuXpMBSnRdHPPBOc218md1qJFi4Jxd7ZZNH0VgkQFQT0u+qy53y/387vniMb1Hm4897GhzlC7767vocc0rmnqdaL54T6zeyyONXBQWs3NzXbVVVfZfffdl72n34IFC2zbtm3B8P3vfz8IEd73vvcFtxWMKjSQW2+9NfibS8vu0qVLg/dWMKHnf/e73w0e+/GPf2x/+ctfgvsVrOqv3lOf4cwzz7SHH344eJ6mr9ddc801we1cI/k8mm7u47/85S+Dv87kyZPtyiuvtNWrV2fvAYAx8jc6AACgQvgFj4x23+HBL6AEj/sFlsjHTznllODxxYsXRz6uQa/1CzgH3K/79P7h+zXcc889+16jaej9/YJP8Ff8QtQBr/ELXgNOyw1PPPFE5HeVgeaBX9g64D49V9xt0bT1GXKf54bbbrsteM5A09B3Ezcf3XfVd9f3AcbCLXdu/Qlzy13u4+41ucufW7fc8p/7voOtd2551njUZwi/rww0rcE+j8Y1OOHHtf5rnXLPc9s3ABgraoAAAFDhVD3++uuvD8Z19lV0RtXfzwdnYHUmV1XIXe0FcWdn3ePywAMPBH9F9/mFkODxY489dtAzxrlOOOGEoDaGzgyr9oSe7xdq9p0J1nR1djh3WnpfvwAUPEfPFdX80Flvd1uP6/Vy2WWXBZ8t9/uJOol0zxGNh8+cy0jOUOvMt55zzz33BLdzq/U7+oyPPvrofk0UgGKZOXNm8Pfb3/72vj40/v3f/z3429TUFPzNdcghhwR/c9dnN2jdPfzww4PHv/e97wW1mlxtJ1dzS1yNDDe9XCP9PFHUsarWKa3n2r5p+5M7fQAYLQIQAAAq0OLFi4OCuQoHCjfUpl9cJ6lqs6+mGa2trUEhJ2z+/Pk2ZcqUYPjwhz8c3Of6BxA1+1AhRI+7q08oNFDYoPdVe35HBX8FCJrO2972tuC1KqysXLkyeFxhhj6HXhcVIJx66qnBe2ha5557bvbegU2aNGlfHwkDfb/BuHn0sY99LJimXHTRRcHfsPe///3Bcy6++OLsPf1ULV8Bir7P3LlzgyYzNIFBKWgZFS2Lbj1TYCBaxsMUUCi0y12f3aD19vzzz98Xjuhxvad7Pxee6DE93wWuuUb6eaJoG6LQxTWBk8bGxuwYAIweAQgAABVKBfNvfvObwbgKGDrb6s6+uhoguUPuJTCXL18ePF+DzvSK2ttHGeqMsXzpS1+yF154IagZoud88Ytf3HcmObcGiBs+8pGPBI8N1yuvvBJ8VoUMCll0Blrvo2kpBIrizl6H5eMMtea9vrumr1oiKqi5DhyB0XLL30DrYhSFh6qhlLseaFz3DVQr6ac//Wmw3GqdDtOy/eCDD+57Pz1H2xO9l9YdV1tK3DqeazSfx3Hf/4ILLgjWL9dJqgLfsV7CFwAC/sEDAACoEAuy7etz2+fn3ufa0ocHPUf0nKjH/ULOfv1yuOc7bhrhQc//7ne/u6//D79wFNyvvgS2bds2YF8bEjUt9/nc9wv3PaLPmHs7d9D7yaycfkfce7vbMtA80uDew31fd1vcc0TfT4M+p/7qfs0HoBxp/QwvzwBQjagBAgBABYk6Q/zxj388+HvnnXdGnn2NouYbfiE/GFc195/85Cf7akZEGeyM8fHHHx/81dlaNb3Re//zP//zvjPJuj1S7vuppog+n+ivPqNqmbjPoe8Zfn/VPnGPR32nsZyhdu+ry3yq6ZG+83/8x38E8+bd73538BhQTtQExV1GlmYkAKqdpxQkOw4AAGJOl6tVoX3x4sWRl7AEEC+6tLSakajpipqqAUA1owYIAAAAEFM7duwI+ssh/AAAAhAAAKrSSDpZBAAAiAOawAAAAAAAgNijBggAAAAAAIg9AhAAAAAAABB7BCAAAAAAACD2CEAAAAAAAEDsEYAAAAAAAIDYIwABAKAC3XjjjTZv3jzzPI+hRMP8+fOD3wEYCOtp6QfWUwC5uAwuAAAVSAf1K1asyN5CqcydO5ffAQNiPS0PCqGWL1+evQWgmhGAAABQgXRmE+WBQykMhPW0fLCeAhCawAAAAAAAgNijBggAABUofGaZ3XnxMO8xXCwrpcO8BxCFGiAAAAAAACD2CEAAAAAAAEDsEYAAAAAAAIDYIwABAAAAAACxRwACAAAAAABijwAEAAAAAADEHgEIAAAAAACIPQIQAAAAAAAQewQgAAAAAAAg9ghAAAAAAABA7BGAAAAAAACA2CMAAQAAAAAAsUcAAgAAAAAAYo8ABAAAAAAAxB4BCAAAAAAAiD0CEAAAAAAAEHsEIAAAAAAAIPYIQAAAAAAAQOwRgAAAAAAAgNgjAAEAAAAAALFHAAIAAAAAAGKPAAQAAAAAAMQeAQgAAAAAAIg9AhAAAAAAABB7BCAAAAAAACD2CEAAAAAAAEDsEYAAAAAAAIDYIwABAAAAAACxRwACAAAAAABijwAEAAAAAADEHgEIAAAAAACIPQIQAAAAAAAQewQgAAAAAAAg9ghAAAAAAABA7BGAAAAAAACA2CMAAQAAAAAAsUcAAgAAAAAAYo8ABAAAAAAAxB4BCAAAAAAAiD0CEAAAAAAAEHsEIAAAAAAAIPYIQAAAAAAAQOwRgAAAAAAAgNgjAAEAAAWzbNky8zzP1q5dm72nX/h+jS9ZsiQYBwAAKAQCEAAAAAAAEHsEIAAAoOQymYxdc8012VsAAAD5RwACAABKbvbs2TSBAQAABUUAAgAACm7WrFlBPx9ueNOb3pR9BAAAoDgIQAAAQMGtWbMmaObihocffjj7CIBykRtS5g633357MGg83KExAFQSAhAAAAAA+4WUGhYvXhzcv3DhwuAvAFQ6AhAAAAAA+1FNj2uvvdZuu+224LZCEIUiM2fODG4DQCUiAAEAAACwn0WLFtmCBQv21f5YtmwZTWAAVDwCEAAAAAD7qL+P+++/32666absPQAQD15GddkAAEBF0ZnYXOzOi4d5j+Gq1GVFn1v9f1xzzTXZe/prgOjqTerQuBKawbCeAohCDRAAAAAAATV90WWrc8MPAIgLAhAAAAAAQS2PpUuX2q233pq9BwDihQAEAAAAgF122WV21VVX2VlnnZW9BwDihQAEAAAAqHK6uov691ANEPWfkTssWbIk+ywAqGx0ggoAQAWig7/SYd5juFhWSod5DyAKNUAAAAAAAEDsEYAAAAAAAIDYIwABAAAAAACxRwACAAAAAABijwAEAAAAAADEHgEIAAAAAACIPQIQAAAAAAAQewQgAAAAAAAg9ghAAADAsNx44402b9488zyvLIf58+cHnxEAACCKl/FlxwEAQIVQgT9XMXbnChhWrFiRvVWe5s6dW/DPWIp5j8rEslI6zHsAUQhAAACoQKU4uA9Ps1wVel5QsMJwsayUDvMeQBSawAAAAAAAgNijBggAABWoHGqAbL9uXnastKZcv3+Tl0LPC84sY7hYVkqHeQ8gCjVAAAAAAABA7BGAAAAAAACA2CMAAQAAAAAAsUcAAgAAAMTAjTfeaPPmzQv6vyjHQZfS1mcEgFIhAAEAAABi4Ic//KE98cQT2VvlZ8WKFcFnBIBSIQABAAAAYkABQ7mrhM8IIL4IQAAAAAAAQOx5GS6KDQBAxVF7+lzF2J2Hp7n9unnZsdKacv3+Z5QLPS9KMe9RmYq9rBwwvT9kR0rMe0t2JKsU2yvWUwBCDRAAAAAAABB7BCAAAAAAACD2CEAAAAAAAEDsEYAAAAAAAIDYIwABAAAAAACxRwACAAAAAABijwAEAAAAAADEHgEIAAAAAACIPQIQAAAAAAAQewQgAAAAAAAg9ghAAAAAAABA7BGAAAAAAACA2CMAAQAAAAAAsUcAAgAAAAAAYo8ABAAAAAAAxB4BCAAAAAAAiD0CEAAAAAAAEHsEIAAAAAAAIPYIQAAAAAAAQOwRgACoCrfffrt5nmdr167N3tPvwgsvtEWLFgXjs2fP3jcOAAAAIF4IQABUhYULF9qCBQv2CzgUitx///32yU9+MnsPAAAAgLgiAAFQNW666aYg8FDwIZdccoktXrzYZs6cGdxevXp18BwAAAAA8UMAAqBqKOi46qqrguBjyZIlNmvWLLvmmmuyj+7fHAYAAABAvBCAAKgqrobHtddea7feemswDgAAACD+CEAAVJXcTlBfeeWV7BgAAACAuCMAAVBV/vVf/zXoDFV9f6gpDAAAAIDqQAACoGosW7bMli5dGjSDUd8f6gOEPj8AAACA6kAAAqBqXHbZZUEnqO6qL1/84heDQETBCAAAAIB4IwABUBV01Zc1a9bsd5nbhQsXBs1hFIQAAAAAiDcv48uOAwCACuF5XnasXzF25+Fpbr9uXnastKZcvyI71q/Q86IU8x6VqdjLygHT+0N2pMS8t2RHskqxvWI9BSDUAAEAAAAAALFHAAIAAAAAAGKPAAQAAAAAAMQeAQgAAAAAAIg9AhAAAAAAABB7BCAAAAAAACD2CEAAAAAAAEDsEYAAAAAAAIDYIwABAAAAAACxRwACAAAAAABiz8v4suMAUHIrfv+0vbB8rSVrktl7Ktu7PvH27BiQX57nZcf6FWN3Hp7m9uvmZcdKa8r1K7Jj/Qo9L0ox71GZir2sHDC9P2RHSsx7S3YkqxTbK9ZTAEIAAqCsfP2jS+1nN//a6v1/cfBQ5tfZMSC/SnFwH54mAUg/DqUwkGIvKwdMjwBkH9ZTAEITGABlpSZRE4Qf9V5dLAYAAAAA5YEABAAAxMoTTzyRHQMAAHgNAQgAABjSH/5QJnXph+Gcc84hBAEAAAcgAAEAAJHa2trs85//vM2bNy8IFSqFPvc73/nO4C8AAIBDAAKgOqjzs2BImxca9j0GIAgNfvWrXwWhx1FHHWVf+MIXKrI2xYsvvhiEIAAAAA4BCIB4Uy/wiYR5EyaYNU2y9PRW6zlyqnXNnrZvsOZG88aPz74AqE5q4vK//tf/CkIPBQcKPSq9BoW+EyEIAABwCEAAxJKXyQS1OxJpf0OXSVq6car1HnyYdRx3tLWdfqxtPe8423bucbb9nOP8+w81m9CcfSVQXXKbuHzrW9+KXbMR1WbR9wIAACAAARA/GbOeQyZZ59EHWedpTZY6d7ydftGr9ta3rrELznzWLpz/jL3t9c/Yxf5wkT+c/NYXbeYZW8y8/tcC1eRv//Zvg+YiQ2lqasqOlb+5c+dmx/qpZsutt96avQUAAKqVl/Flx1ECN954o/3whz+0FStWZO8BRkdncC+//HK7+uqrs/dUphsW3WI/X3q31Xt12XtGJqj5YSnbcvGJtufo6VZzwk5rmd1mt8/5jc2o35Xt78MLgo7+Z3r2lNXaXx9vsB+9fbalUon+ZjN58kD6zuxYeWIbVBjFWB+90HI6lt25mruoBshgtT8UlKg2Ra7t183LjpXWlOv3X37XrVsX/Abh7/Pggw/aW97yluyt0cvnvEe8FXtZOWB6ZXLxJi+02hVjnWE9BRCFGiAlRsED+aLl6Ac/+EH2VnVSk5eu6Y22+YLj7OIL/2rXXfQL+9wJv7dPTX3EMukOW7Fnkn19w6n24+3H2q92H2nPdTZZps+zqalea2lM2PYFR1vbmUcG+Yh/pNT/pjHHNqgwKm19VI2JX/7yl9lb0T74wQ9mx8rfkUceGYQdYa5vEwClVUmX1QYQLwQgJUbBA/nEgX3S7OAJlj6z1U49eYO9e/6j9jeHP2fnTX7J9nal7a87GuzuDUfZQ5un26PbW+3lzgbr8l8zKdNnTY0Zsze2+CXByZaeUGuZWv+9QmeP4ohtUOFU0vqozzpUZ6HhZiXlTp83HIKoRoi+Z9z6OQEqjWqcEYIAKAUCEACVL5OxTH2ttb/h9Tb//C774cLv2mmHbrC63gb71e459qmXzrQvff7ddufHTrOaK1+1TR+psVUfOsTuuOsY++LOU62tr8WOm7LLbnn3LfbZS++2jZ883nYumG01k1vMq62rmtogqE7ucrGDhQIKE1SrotKoucs3v/nN7K1+XB4XcaN113VmXEmokQWgFAhAyozaJzIwjGSAz/PMS3iWaKmzhpaUzWjabnW1KevN1NjOznH2atsE27yh0drWjzfb6N+/3rPOl/3Httbbxt0TbXfPOOvzn9vUuMcmTe60mulJS0z1h8akWY1XFTVBnKhljGH4Q6VRGKAzseFOUD/xiU/s11fGO97xjuxY5dF3ueyyy7K3+unMs743UKkUeqhPHoUeunT1F77whYoLE/QdtB4SggAoJgIQAJVLBc5EjXkt063uiBY78WMrbc6Fr1hzepz9Z+eR9rWOk+yxf2uxjo+lzB5da4mNrwRhRqarwzJ7tlrX7Z22/SNJ+5+/WWBv+69L7b0Pfci++tfz7aoz77eL//4Ja76py2pOqLdUcrI/nWTQxwgQFyp86AxsVPihWhO5HYaqA9RKpr5uwp2fKgTR1WHKnToqViFXHToyjH2YP39+ME8rlVtuFXq4GhSD1d4qdy4EqeTvAKCyEIAAqFzJGqtpSFjTvB5rndtpxx62zQ6Z3G6KKbaum2DrljfZ7udrLPVyn2U6esx6e4OXZdJpf0hZakfKetembNvTnm1ekbRtK2ps51P+89fWWE9nre1pHWczTuy0E9/YZvXH1NreGZPMamv8g2g2nahsKmzoKjXhM68u/HAUguh2pfX/EUWdvIa/x7e+9a2yvzyuwhvOkOdPpXYY7pq4KCzQchunwEDfJeqqTQBQCFwGt8R0NiIXPwdGKm7L0Egug+uNb7SG2WYn/+BVa27ssQ80rLdkost66vbYN//3ubbst7Ospn2LpffsCWp+DMmfd964eksecaTtPKfONv9Nrf149q/tTS1r7YMPLrTfPjXLTvzOKkvu6rR0qtN/wdDNY8r9Mrhsg/KrmPNztNNy4Uf4krZqJqKwo6mpKXvPgcLTLNfL4A40L/TddeY8XNAa6eVxS/k7Iz+Kta3L17KiEGw0NSXK9TK42t6Ew0cFlPnsmLuY6ymAysFpTAAVK9XcaHXTJtoZE7fbKeO2W6fXZU9uaLXv3nO+rV090RJtOyzT0+MfBWVfMBAdEwXHRZ5lelOW3rrN6v+8yab8YIP94r/m2OKV59jcKZvtn077syX+R531XNhk3kEHm9cycEERKEcDhR9q4jJU+BEH+n5cHheVyF3VKC7rqGo2hZvWuZAHAAqJAARAxco0Nlp9U4O9YfxOO6l+h/UkOmzVpin2w9+91V58ucG8zh2WCZq9DJGA6GE3pFOW3rHN6p7YYlPu2mi/XjHHvrzmbDth2ib72Gl/tsz7xlnnW1os03yQZRom+x8ipdNKeheg7KnvgHD4oYKVCiNxDz8cV5DMpWBIIchIz66Xgs5iM4x8iAMtu2rKFRf6LlF988QlBKH/nsoaKr1/IAwfAQiAyuQfz7a9zrPtJySszzPrSNXZvW0n2ar1zTZl+V+ttm2P/5RRbuLUx4c/pC1p0/600Wb+/AW7e9lMu+mZ0+y0ujV2+jHP2/RLN1nzBV2WmTHDMg0TCUFQ9lTzI6rKeZzOKg+XCl25fZ2IuyIOUK5UQ0JBXZxo+6PtUK64hCD031NZ1PxKvxnir6ICkNmzZ9uiRYuyt/otW7YsSO3Wrl2bvQdA7GWzhl2vS9rO45P+lixjXelau6/9eHv6lWZreHqdJdv2mr9x6H/iaPkvn/zIVjvo3pft7r8cYbesnm+nNqyzU49bawcv3GLN53dZ+vDpZg0T/CcTgKB8qQNFwo/9RV0elyr4KFcK6IaqpRQOEirFQCFIJVylaTD57M8ExcFvVh0qKgDRwdvSpUuD0MPRwctVV11lM2fOzN4DoFpc+bqH7B9PuM/qkqngdl2iz5LJTH/Nj7GGH1lpL2EpS9ph9661g7692r7xHxfbyheOsasnPm0fPfERu/r6n9sJb95qmcZp5k88+yqgfOiKEV/4wheyt/odeeSRQfXzag0/HJ3tq9TL46J6uNpJUZeszl1+3/GOd2THKou2Q1EhiLZdCm8BIJ8qKgA566yzgrDDnbFZsmSJrVmzxm666abgNoDqoLoWGo6ZvNVOatlkCS9jmbRnfW11lt5TgBDCM6vfvNfqnm+3l56ebJtfbLS+rjqbUN9nJ5643ppmpKyvudHfovpPjGgKQxVYlIoKEOHCvAobCj8UgqByL4+L6uD6p4kKP9SMS8GBC0HCnYpWEheChENZhbdxWRej+qZhKP2A6lNRAYh88pOfDEIPhR/XXnut3XbbbdlHAFSbw5IJO7ImEQQgqc6E7bj7EGtf3uI/kv8dWsbzp9OZtjk/WmnP3ubZ2Q990L72zNk205KWnjHFNp94qKXqarLPfo0KUuoETX+BYtIyF6754cKPSq0qXwgDFbzUZ4pqgwCl4q7aFA7RXfjhaPnV7Upfr7UOqglC1LpIIAkgXyouAFFTl8WLFwfhx4IFC2zhwoXZRwBUGp3R0hUpVMVV1XtHGhLUmBcMwb+MP3T5Q09+mr5EymTM6+61xI6MJVbXWWJTrVk6YW889mV7/3lPWMP4bvOfkX2y2cv2wr6z7/pLVV4UiwoLCj9y+wtQoSKqyQdeC0HCuDwuSsWFH+GrNqkW9Oc+97nsrdcoFIkD1UyLWhej5gUAjEbFBSDiqgGuXr06+Aug/KkQkRt2NDc321FHHRUUMFRQ0+Mjrb6rDpB1tRYFIPp/sjdjib7CVmdMW5/V7k5Z8/OeNbzqTz+dsItOfs4++c4/2OQJXf7j/QFIp3XY2swzwbij76mDOKCQXB8WUeFHJVeRLzSdPQ8XvDQPtb3KnZdAoQ0Ufmj9VU2PcA2JuIlaF0XHC9TKAjBWFReA6Gov6gj14Ycf3tcUBkD5UVCpsEPNPxR26K8LO3QAEy5Q6MBupH0SZLK9gbh/xZLoTtm4rR3Wu6vX1qXG2c7eWuvtTez3CcbbRDvJOyN76zU6M89VJlAoChK1nhF+jE7U5XFdCAIUiwLMcPihUEDrcdzDD0frYlQIonWRWlkAxqLiAhBdBlcdoapDVPX/oaYwXAIXKD8KM5588sngQGWos6c6oPuHf/iH7K3hS2cy/pB+LQApSmdWntXs6bVJq9us+9Uue6S7wbank5ZMpLJ1P17TbFMj2zMrAFIgNNR8AUZC61pUbQUV6Ak/ho/L46KUovq7cDUiqiX8cAhBABRCRQUgt99+u91///1BR6ii/j9mzZoVhCIAyo/OVg2nVocOckbTeZs6P9UgmYRnHa1J625Ug5gC8jxLd3VZauNG69i+1zb2ttiPNp9o/7z2bNvRO87fqO4fwuh7KQQJzwdXoAr37A+MVrjmhyj8CBfmMbSovlJc0yKgUFRrkvBjf4PVyiIEATAaFRWAXHLJJUEHqOoI1dGOQqHIsmXLsvcAKBc6YNMVJ4YS1aHbcChw2No93jIZz9+aZay2tduSk1P+I8kgqCiYTNrSfb2WTqcs429GV7ZPswe2HWWd6VrzQgGIKPxQCBIOeXTwppogHMQhH6IukxmXjhFLgcvjopiirtqkfYeWw2oNP5yobZlCkKjQFwCGUlEBiK7VfM0112Rv9VNTGN2vvwDKiw5Mwgd0YaqaP9LaH4o2NHx2zVn20ecusO5UjU0c320XXPSozTt9vXlNU82rrStMkxiFH1ObrOucedZ60hR7z6QnbXp3h+3eOtHSqYFrn+gAVmfxwmeVOZOFQlBhIXzWFCPj1tlw4ZOOjJFvCj/CtYvcCYTh1KKsBlG12RT6av9JCAJgJCquDxAAlcH1czHUZetG0/eH076nztp211tnOmk9/uasadxea2jwx5oT5tV5BasFkqnxrK+h1rxxCZvo9Vptd9qsw5/WEHmLK1CF+2PQwZvmFb3bIx8IP/LHrbNAoUTV/HDhx0hPDsSdmqbRPw+AsSIAAZB3OqBT1dSh+rdQEBCuETEsyjX8Yfode23yzT12167D7XddB9t7x79kb33zRpu4tMfqT62zmmSDJZK1QwYTw6LaJP6Q8SfcMSlpL51cb7tn11lLbZ0d/livzbxjj9XuUYesQ9OBbVTTBB3EUb0ewzHQGU8VDkbbpAzRXB8MQL5pe6/wI3d9VvgR1QcN+kXNG0IQACNBAAIgb3QQp+rhqsobLqDpgGXdunXBwZ0zltofktzaaX3re2zV6kPtuZcOtr5Un7U2tNsbj9hs40/wrO3UJus9aJx5NXnoE2TCBEu0NtjUM/rs6DPa7KJjnrfZ07fZsz1TrH2nZ3WbO8xSw09adIY+qqCq+acACRiIW8/CFH5oucpdx5Af2n6p4AXki+tUNyr84KpNg4tqTqr5SQgCYDgIQADkhevQM6oGgwr6ri2z/sqoa3/kyLTvsPb1e+zB3823Pz4819b1mR0ybqv9U8tym/S2pK362BHWfnyLJW2ieZmMP6T9F+UOAwQWQW2PdPB897q+pmnmHTvDjruhzxZet8ZuPfcOe/fxT9mt7a+zF3bVWLJtp1naf88RBC3q8T+qUKWDYj0GhKmwpOUjqmkZ4UdhKWAKV78HRkP7y3AHnoQfIxPVREghiOYrAAzGP7YfqASAYvBChSV+DoxUOSxDrg1zuNbHQAd0KsC94x3viAxAblh0i/186d1W79Vl7xmEcor6pO2dNcXqG83mXtBpc47baqeducbW75pq7b0T7PmnJ9j6deNt5YZpltmctOZnFVL4r/Xnm7djl3k72y2j4CIIPfyhttYS06ZYd0uddU6ts/SMbktNzNj7Z623E6bvtAmnbLZtmUb7654ZtmvTBFu9/CDrunundT/a4b9pX/975HggfWd2bGADnbnS/Cl01Xu2QflV6Pmpmh8DNZMqxm8X/n7br5uXHSutKdevyI71K+S80Loa7q+nkP2usI7mR6nmY3i6uiKYlqHw/lL7ynwEbAd8zzLpWsoL7e7zMf81DzUvw52Iu/Wx1OsO625l4HeqPtQAATBq7mx0uBqv6MyMDvSizmbpwGSstT8C/j7L60lZwzMbreaJHbbsD6+3R/86016sqbU3TH3ZPjDjcTvpzVtswn9P27bzDrKtJx9q3a3TrUvD1EOsr36SZfr8zWDa39kFO7yMZbyk9U1stb3Tp9uOY4+w7W852LZcfJDNfccWu/CiZ2x+0wb/OWm7fe8x9uCmo2zTnw6xjlfr/Pfoyr7HyGleaF6FUaUXuVQriD5iSi8qlFQIzG+D4Yi6dKv2idQuGjmdZNH6GL5SjtZHmpICGEhea4Cs+P3T9sLytZZUe/sYeNcn3p4dKxxSR4xVqZYhnXFRrY+oqvg6+6JmL6Opjj+iGiCOvnJNwnoPnmqts3ps5qm7remsXqs/Mm1H1bXZlNq9tnp7q2XaktbwsjoyNUsmMrZiU4OtDC5fmw6yi3QmYQ31vXb6jG02sSlh41tqLT2t1zLj05ZOetbXl7Ataxps45pae/YvdZbYnTRvbb0ltrWZt3uvfozg4+QaTg0Qx3XkFhUm6SCvEM0b2AblV6Hmpw7mFTQOphi/Xfj7VWMNEAnPByeqb4KxYh3Nj1LNx4GWFSfftYcO+J4xrgHiqMN1NcEN7zvDir3usO5WBn6n6pPXAOTrH11qP7v511bv/4uDhzK/zo4VDisdxqoUy5BrZxs+2FABXcFH1BVOhmtUAYiTSZmNn2Q2+WDb81Gz3afW2L9M/7O9a+pq69ZsSmYsk/T/eGmr9frsZ23H2C93zfHnWcJSGc/60jU2tXa3fWDaw3ZYos8fPMv06WVp+5cNp9uDm4+wzctm2PgV22zaA6ssE1Si0/z23zz0OzgjCUBE8zSqSq/rPyXfl0VkG5RfhZifUeGH1rXw+leM3y78/QhA9ufOSOdzPWUdzY9SzceBlhUpRNOpA75nFQQgon2mQpDBFHvdYd2tDPxO1SevAciYCi5laKQFl9FgpcNYFXsZUkFMVb2jaimoDfNYD/zHvB1JJIKrvqSbPUvXZ6x1TotNnDHRNs2rs/SUlGWm9dh7pj9u57U+a5n0RKtRYJvpbw3oeWnb3DvefrD9GNvZ2WC7uyba5D/UWd3GpO18apN1buywvu568zpTlmzv9l8QvGxQo9mODBSCULgqf/menwOFH1oOwgf7xfjtwt+PAORA+n3CV7waC9bR/CjVfBxoWSlE+CEHfM8qCUBkqBCk2OsO625l4HeqPvQBAmBYVChXrQ8VyMLhh+usM9+1E0YlnbZMd495m7ot+VK3bXvGszVP1NtLy+vspcfq7cVH6uyFxybZquVTbcOTk6x9Zb3tWdU/dKyqC25venyivfLYeHvxL/W2zn/dmifrbM/z/g5xfa/VbumwZHuXv8fMTq8AVHBSnyDhqvSa7zq4C3fAiHhS0KhmZrm0bOQjaEThuAATCO8rHfX3EXUZdIyNtos6FgGAwVADZBDUAEElKMYypAK3rj6hdra5VBgba5OXsIJvR/z5k2huskRjg2ZeUGMk+Jt9zPr6LL1jp2V6e/2hr//+MRjrdmSgq34U7IoBbIPGJF/zM6qZmQs/XMfCpfjtwtOkBkg/rYvh9dQFw2PFOpofxZ6PWne1/Q73k6VlRTU/8lVDKOyA71lFNUCcgToQ14mFYobHrLuVgd+p+lADBMCgVONDBbGo8EOFsXyGHwWl/Znbp/WmLNPd5w+9lunqsXR20Ljus1Q6e2WY7PNLaKB5rANr/TaIH1XjHir8QHmJ+m1ccIzqo3VXTdeiOgkvZPiBfgof1WdWWFTTUgDVpzICECVxwZA2LzTsewxAXrmzVzqIyy2IiQ4uBrrEbdlSwB8MnqX3dlhq6xZLbfGHzZstvWlTMGg8tW2bpbu7LZNK9T+/DOiAOaq6tH4bXRoV8aGDcx2kh9c5LQOEH+UtqpNi1QqJqsGFeHN9ZUUh/CiOqO2ltqvavoZP6ACoLuUdgKhKkjo0nDDB32NMsvT0Vus5cqp1zZ62b7DmRvPGj8++AEA+uEJY1AGcaiPoQD983f2Ksi9UHWAoQwo6oqrTq48I1RZA5Rss/MhHcycUntbR8LYxqhkE4kvbakKv8uVCkPB2FkD1KMsAxPMLIKrdkUj7HzCTtHTjVOs9+DDrOO5oazv9WNt63nG27dzjbPs5x/n3H2o2oTn7SgBj5drOhquJuir4VN8tnYH6FFDhSr8ZKpfOSIabvUi++9hBYWnbGLWO6rel6n38qVliuONilB9tb9lnAtWr/AKQjFnPIZOs8+iDrPO0JkudO95Ov+hVe+tb19gFZz5rF85/xt72+mfsYn+4yB9OfuuLNvOMLf1V1cvzxC1QEVTwUrXdqDMjrmd1zkKXnmt+FA6hFFzpCjGc1ao8+s2i+tlR8EETp8qjGiBRIYi2rVS9j6+BLlmN8qRAcrBL5gKIr7IKQFTzI2F9tmvuDNt08Ym29cNHWed1rfaRLz9pn/3CH+xfPn6PffXyu+1f3/t/7Wvv+419deHddul1f7KzFz1jSS/lvwMJCDAaKoAN1Kmm2tHqYJ7LbpYP/RZRvdm7JhQUsiqHCz/CtQMUfqi2FSpTVCeM7rdG/AwUfkQFYSgdrZe53D4TQHUpmwBETV66pjfa5guOs4sv/Ktdd9Ev7HMn/N4+NfURy6Q7bMWeSfb1Dafaj7cfa7/afaQ919lkmT7PpqZ6raUxYdsXHG1tZx5pmaAmCEEIMFyu5kC4jboO3lQA00E8Z7HKjzvLHBWC6PcMF6hRflzwqHUwl8KPqE5vUVkUHodrzVHgih/19xFu9qJ9ppqMcuKgvGifGQ5BXLNfANXDy+TxYsc3LLrFfr70bqv36rL3DJ+X8axr3lRr+9vZdv1bf2HnH73SrLfBLF1rL3f22hPtB9vNG8+yWRO32UHjdtubJ2+2MyZttrR12XNbp9nH/u/fmW3qssYfPWuJ7pR5Kf9rjfGrPZC+MztWOFx7GmM1lmXItVdWQSyXO3jjqhOVQQdv4UK0fkOFV+GDPdHv7UIttkH5Ndz56cKPcPCoAvNw+9kpxW8Xnub268qjCvmU61dkx/oVel6MZN6r1kfU76xt7HCwjuZHIeajtrv6fXP3oeH9Z7F/vwOmt/+uoWS80K6oGMtx1LzXb6V9ZvgkwUjWyeFi3a0M/E7Vp/QBiD/5zLg663j9sXbaOS/aB65cZofW91qrP/xs9xz78/ZDbedNrbZ3Xa/tXrvV6idPsJracdbyvk6b+s5OW9S40loSHbamvc5e2tlsn/qPv7EJj+21af9vu6U62i3T060lOzuxkSEAQSUYzTKkA4CBLtPnqm4PpwCG8hFVyBL9lrlBln53/f7uQI9tUH4NZ34OFH7obLHOUA533SvFbxeeJgFIv6GmF1UrS+vgcPpVYh3Nj3zPR1ebZ7DwQ4r9+x0wPQKQfdw0BwpB8t30kHW3MvA7VZ/SN4HxFzov4Vmipc4aWlI2o2m71dWmrDdTYzs7x9mrbRNs84ZGa1s/3myjf/96zzpf9h/bWm8bd0+03T3jrM9/blPjHps0udNqpictMdUfGpNmNf4CHVqogWqnHb4OxqPCD1W7J/yoTPrdoq4WomDE/daq8aMhKihB8SiEGmv4gcqi35bL48ZHVPghKjxTc7L8aTsbtU66fSSAeCtdDRBNNllrXtNUq52WscOXtNnpMzba+w96zn7ZMcueyzTbuhsabNO94yy1pcsy3Smzvj7zEspsPEu21lnioHprv3yq9R1a6z+UseZxHfaO1y23zbsbbc2OqbbjX8y6lvvPzewxL9VrGW9keQ81QFAJRrIMqSDsagDkijprhcqkq4ZEXYZRv21uYcudfWYblF9DzU+tf+ED7NGGH6X47cLTpAZIv+FMT50TH3XUUdlbr4nq0DgX62h+5Gs+DhZ+RIXQxf79DpgeNUD2CU9T66ROCIV/y+HWzhoK625l4HeqPqWrAZKssZqGhDXN67HWuZ127GHb7JDJ7Zb2H9q6boKtW95ku5+vsdTLfZbp6DHr7Q1elkmn/SFlqR0p612bsm1Pe7Z5RdK2raixnU/5z19bYz2dtbandZzNOLHTTnxjm9UfU2t7Z0wyq63xF/Ky6fcVKBrt3FXw0hnH8I5eB946ACf8iAcFIDp4Cwufab7hhhuyYyiWqLOLWv+odVUduDxu5dPvFO7zQ1R7Mir8QHkbaJ3UsVJULVkA8VCyNMCrn2Djj6i3k776op32+bV2+bTVdua4V62zrsO23DbBXv7ENNvzJ7NM3x7/2RnFc/0vzMr09Vimo90O/eHzNvOrT9usrz1lTV99xe7936fag3eeaE/uONT+7po19q1f/tGmfL7Jnr/sJMs0NPhvU6sUxR9I91AddMCmnXlUtU4dsEVVA0Vl05mrqIO6XDqLSaGreLT+KYTMpfVO4QfrX/UY6PK4CkHC1MFmuKCN0tFvofAjvN3UflTBMyqTq4EXpuOmcOfiAOKhZAFIqrnR6qZNtDMmbrdTxm23Tq/LntzQat+953xbu3qiJdp2WKanx2z/3ONAyjGCLMOzTG/K0lu3Wf2fN9mUH2ywX/zXHFu88hybO2Wz/dNpf7bE/6izngubzDvoYPNaONuG+NPOe7BL3A73ahOoPCpoRdUEyUUtkOKICj+03hF+VCfVtgtXr1ehOjcE0TIzUMfGKD4XfhS600yUhvaXA9XOIgQB4qdkfYCkjj7aZpyQsu/8651WO67btlqH3ffYfPvane+1aX98wsa/uMkyXjL77BFQ7Q5fwjL2widOsu2vb7W7TrvDzjjoZfvb1e+y9mUTrfH7NWZ7d5m9vC54Zrh2iTPaPkC0o9RBC727oxgGWoZ0RkoF3PAZRJ3t0AFb1OVRER+uQDVYLQ8VwsPLR1y2QYsWLbKlS5fad7/7XfvIRz6SvbfwwuujQqhwvzua77p/rM3OSrH/CE+zXBV6XuRj3kcFHFomFIq5GntqWhHu0ycu62ixjfY307ob1WGtwg/9PkOdRCj2enrA9OgDZJ+hpqmwI1wbS7+vwpHB+ukZSLF/e4wOv1P1KU0NEH+5anudZ9tPSFifv8x1pOrs3raTbNX6Zpuy/K9W27bHf8ooP5r6+PCHtCVt2p822syfv2B3L5tpNz1zmp1Wt8ZOP+Z5m37pJmu+oMsyM2ZYpmGilvTsi/PD7SwHutIGUEha/nRgrYPmcOHWneUg/Ig3twwM1cQlvHyM1Rve8IbgQCI8lMLatWuDv7t27Qr+lkqhwg9UPtUACheqVMjOba5ILa3S0robFX7oBNdwwg9UFh0bhWv0aBlQKBKu/QOgchU/AMlmDbtel7Sdxyf9T5CxrnSt3dd+vD39SrM1PL3Okm17Fcf1P3G0/JdPfmSrHXTvy3b3X46wW1bPt1Mb1tmpx621gxdusebzuyx9+HSzhgn+k/MfgIg2lvkOQj7zmc8cULhQoePmm2/OPqO4lixZsu9zLFu2LHtvdXvyySeD+bFw4cLsPcU1WJMXhR8csMVfqfoOeOyxx7JjcOIWfozmLGixaRtYKYY6s1yK9Rj9NO+jwg9Xi5J9aTy5mj25tCwoBGF9BOKhNDVAfFe+7iH7xxPus7pkKrhdl+izZDLTX/PDLzzmQ9pLWMqSdti9a+2gb6+2b/zHxbbyhWPs6olP20dPfMSuvv7ndsKbt1qmcZr5E8++Kv/yGYQ8/vjj2bHXqNDx0Y9+dEQF7paWlqCQPlZf/epXrbm5ORj/9re/HfytZCOdLwqe9HwFQeUifNbfFbq0U0d1UAF73bp1tnPnzuBAvdgF7ocffjioQuoGvBZCVnrNjyuuuKKsAwYVTrW/rRRaLsKFLZQH1d6KCj84kRB/akIcbsau8EPbPkIQFMuNN94YLHMqZ5RqmD9/fvA54qboAYgOhTUcM3mrndSyyRKef4Cc9qyvrc7SewoQQvhl2frNe63u+XZ76enJtvnFRuvrqrMJ9X124onrrWlGyvqaG/054T8xdKCu6/WrcO8G3Q4PWjDdoHRYQ7izOyefQcjixYuDgsW2bduCNu5yxx132O233x6MD0UFo7G69957g/f5yle+YrNmzQqmv3379uyjlWmk8yWqev1JJ50U/DbD/S0KSdU5ucRt9dJBuoIvVbXXMqm/KmyV4iy+aohpZ6pabOqfQ+MKHHXbUe0pBblRj2l7k9vERuO5tc60vs2ePTt47MILLzxgW6Tbej8Xcuq5uTXn3OtcLbt816pT+DGcfqHK3dVXX23Lly8PlqdyHLS902esFKqtVUmBTbXQcVz4OI3wo7roxNFQnRUDhaRlsNRNr7RP1eeIm5LVADksmbAjaxJBAJLqTNiOuw+x9uUt/iP5P1uY8fzpdKZtzo9W2rO3eXb2Qx+0rz1zts20pKVnTLHNJx5qqbqa7LNfo5Q3d9CGLzxowXSDDmTcMBgXhORjIzplypSgg79Pf/rTwW13tkKFgdzCgg72XYFctx130K/CgQoernCgQbcHCzTuvvvu4O/5559vF1xwQTD+wAMPBH9F76v3UaFGXNMQTcNRzYncabpB3OvDBRsVegYqQOmx3O+t7+D6AnDNdVSwce891HwRvb+bvga9v76L3u/aa68NnqO/ekzTdwU993oZqvDmnp87P4aa/0NxBV+uMgFHQZjObGmHprCvUDu1N73pTfuW9XDtqC9/+ctB56Siz6DbWp+0rL/rXe8KgtTcx7SeaP1529vetl8TG41rOlq/9fpLLrnE1qxZEzx2//33H9Ac57rrrgvez4Wceq5qzum9Hb1Oz5F89h2i0CkO4Qfyy13phTPK5UW/S24/LKLwQ/tTwo/qon1k+ASSjuEJQVAMOlYrB+XyOfKpZAFIjXnBEPzL+EOXP/S8VgDNu0zGvO5eS+zIWGJ1nSU21ZqlE/bGY1+295/3hDWM7zb/Gdkn9yv0QclQQclIXHTRRcFf95l15iK3AKCDfRUQBipQP/vss0HBI7cGhG5/4xvfyN46kIID1fyYOXOmvf3tbw/uC1cXlfb29v3+umno9QoOcqcZJVywUaEnqgClgpAey/3e+g7ve9/7srf6qdCj9xG9r8KUwej93fRF7/+hD30oe2toQxXeHH2m3Pmhz37XXXcF4yMxbty4YKdNG2UMRstGKWoGqTadmsio9tqCBQuC+7Ts/+UvfwnWM21T9FePX3XVVdbY2LiviUBuzTf3Wq0jbn3Wuq7H9PpTTjkluE+03dM2Q9PWY3oPV3Puxz/+cfDXueeee4LHr7zyyuw9Y1NfX0/zMwyoXLfRWmdyT07kBoWOqy2lwe1HXQ2ucA2q8Ptp0ImFcEBaago+wrV4dRKBkwnVS799uON4Hb8TggCVq2QBiHZ+ulqLvxv0P4Rnyd6MJfoK21Y8bX1Wuztlzc971vCqP/10wi46+Tn75Dv/YJMndPmPFzCACdFBTyHb/epso1JqV1hQoUIUdOg+R+P33XefHXLIIXbbbbcFz80tHET1OSI6GFJB3YULF198cVC4UKE9t1A/mJdffjn4q+lqmirsiD53Lt2vz+U+k6ajQkru99qwYYN9//vfD8ZVGyb3eytsUEDiqGCkQpAGvZe+h84yR80X0XRdocl9Nr3nNddcExTIxBXMzjrrrOB2rqEKb7k0LT3H1ejRjnekurq67Ec/+lH2FrA/LcOqBaKmeGrGVwi5fYBoPcl16qmnBuuJaq+de+652XvNVq5cGfz98Ic/HISqevymm24KmpS58NC9V/i1jtqq6jH3ekfbPdG6rm2C9j8KQqNoWya5rx+L7u7u4Cw/EKZgTH31qFlFudUQ0jrjwngJH69oP+9qS4nb77uTMOEaVOH3E+1XFfqXSwgSFX7oWI3wA1GdFROCoNj8Q6qiDXFXsgDEPzTO/r//X7EkulM2bmuH9e7qtXWpcbazt9Z6exORn0A769xBByrhQdWC3KANpIbBgg3tTHVmXq9VISRffvvb3wZ/deAvkyZN2tcso7W1NTjQGMzkyZPtoYcestNOO23QwoHjzprqAEjP1+AObnKbwQzm8MMPD/6qhode787QHnbYYcH9zvvf//6gMHL88ccHt1WAUiFF9+kMkqjw5MIa95kG+t7vec97gvmkQe81FB3IKejRe46m74ThFt5EzZnE1egZLe2YVcANh0moTloeXOihQZdI1rJRTlXv3fbge9/7XlCY0hljnVVWOOmCTldQ0mN33nlnMK5tl6O+KcS93lHAK7k1QNzgmsAVkuY//TxgIDq7rFp7Wh51fFBuhW2tN9qP5Z7ccPv5cA3L4XDrnjvp8fvf/z74W0qqNavtYi4dr+l3KUWfSfnkvaU8hko3UAgyUL9/AMpXyQKQtL/zS2fSrwUg/u3C86xmT69NWt1m3a922SPdDbY9nbRkIhVZ90M7v9xBByXhQRtDN+ggxg1her0LPnTWR7fzQQf5qmrqzsS8+c1vDg5S1ORi9erVwUGGDvhzq4PncgUFtY9XAPHFL34xeI2rbRFFz1dNj4H827/9W3as36pVq4K/r7zySvDXUUgjOrgSFXJ+8pOfjPrMqwt/XA2Q3EFnkUfCzRfXTEd9duh9BgoU1B+M5Ba6nOEW3vJNn1WF3ahmSYg3BRvuwEy1PHSWyoUexZDbB4iG4VBfQi6g0DqjANM1dVOtEMkNdVUg0/Pf/e532xlnnBE8ru2SHtfrXfAo2jao1pVCWj2W+9ly++IpJBWwWBcxFFcrJKrafam4oCK3xuLXvva14O9om9FpP7hjx47srdLSuqltZW4grGO0qP4fUL20TESFIKo5lM8TmgAKr2QBiDo/1SCZhGcdrUnrbvQPSIN7CsQ/2E13dVlq40br2L7XNva22I82n2j/vPZs29E7zp8Z+Q9hFJIUIvjILQi42ho6SFH7W0cFiajCgLhCuf5eeumlwbi42hiD1QBxZ340vdyQQU07xJ0pcoGE3kvvqffO9X/+z/8J/r7wwgvB6xXYuOrno6GaIpJbK0VDbmekQxlovqggpvcK7/jcWWv3uKtqn2uowluhqfp9uEM3xJ9qHOh3d+FcMQwUtA5FQaCCTx1cuvfQ+qEwU81lXHMzt36Kzjzr+XqdnqOmdC5MVdjhQlwXMv70pz8NtlnuOaWgdZFaWRgOFby1fJdD53Pvfe97g7+qnSVqUqrjC62D4Rqbw+H2g+5Eysc//vHgbylofYxb+BE+TilHOjlTiVwIEj6W1wmG8FWDdBICQHkqWQCiwGFr93i/4Ov5nyJjta3dlpyc8h9JBkFFwWTSlu7rtXQ6ZRn/669sn2YPbDvKOtO15uUpANGOVGduFHzo4CWfwcfJJ5+cHXuNDvZ18K/28qLgQQf/7kBfBYpwNVXV9HCP6/mf/OQn9/VL4QoeA3FnMV3g4Kgg4qajM0XXX3/9vvdUwUUFGN1203UHPXPmzAlCCg0KcHL768ilzhDD3HxVIUeFIPUNMppCmHvv8HzR53Fnv0TfL/f9ddY6d75FfcahCm+DGc1yo+UtTAd49ENQPQY6SCu0Rx99dL9Q1A2i9VPjrm8d0bqh+1zTL9XUcu+hs8Nf+tKXgvtFz3W12jSodlZuzS6tq3qNm4beM/e9ta5pG+me4wZ9LnG3C021ccqp6RHKW7HX4Shaz7T/Uuih/bPr4HSsAYH2pdpnj+XEx1go/Aivj5rfOnar5JofV1xxRVkHDApoKrlJoJYRHduH1019JxeC6G+hjrlUm9hdmVAdC6upaFTt40qhbYq+S+4JXKDQPP+AL29HfDcsusV+vvRuq/fqsvdEyE6t/Usz7NAz+uyHc+6xXvPs5rbj7OV7G23dl5ss07nLMt1d+Q9CFH4cPMU6j51jJ89fbdd88tf2lacX2K+fO8HmXP+k1W3tskzONB9I9zdTGCm3Mx3OgYtW+lzFOAAvF9qAqzCiZi9qDqMmMqologMtFXQwPFHLUFRnbqIDDzp0qx6q/aED4YEK3NpGhR+rpm1QIYTXR61v4QNhBeQKqMaqmvcfpVboea+zxzfccEPwtxTrqJqGqSmbaHoKPVSbUydHXNM07b/d83QiQMGjTmToqkwK/V2/VxJ+v1II/2ZR2z/V/Mh3h7Ssp6VTyHnvmhmHKTyLWm9lrNNX2OHWv1zh9W20FEboONGtz/kW9f7uPp0YLEafXFHKdR098HNlR4ogNOnYbbdKVgOkfU+dte2ut8500nr8j9E0bq81NPhjzQnz6vy5Hp7zeZKp8ayvoda8cQmb6PVabXfarMOfVh5/V+1UhxN+VDudTVKbfPVXogMj10TGdWyK0VMtkKgzFG6HTdXM6jDU71wufQzEmQ6GwwUq/S5RASWqmwpMrrNi1UxQbcuoQlQpqLajqImp9ttxOFurQlcudWBfblfjQfnS8hMVZBdqvVU44MIPd8VANT1X+JEv7e3t2bHCiHp/1TDTdylV+FEKOgYoZvNkHKj4AYhyDX+Yfsdem3xzj92163D7XdfB9t7xL9lb37zRJi7tsfpT66wm2WCJZG1+ggmlVv6Q8SfcMSlpL51cb7tn11lLbZ0d/livzbxjj9XuUYesKKZwcxU1IVFzE7XVx9hp56wQJHyQpx2zDq7pFyTeXC2gwQ7EPvjBD2bHUEg6qxwOm/T70CkqxAVi6rC4mJ0Vj4TrSNhxTcsG4vq80qBq+uUod99YX18f2XwUGEy+avMNh9tf6Dg5t2mnan642h+qTaFw0q17OqHomqyJamjp/nvvvXdfMxp18q/X5dbSUi0uPabmNrpf45/5zGf2vV7P1233HrnvI/rrnqtBn2mo99fzRePaZuQ263HT0X16vqal2xr03rlXqKoECj+0vVftUI4DSqNkNUCSWzutb32PrVp9qD330sHWl+qz1oZ2e+MRm238CZ61ndpkvQeNM68mD32CTJhgidYGm3pGnx19RptddMzzNnv6Nnu2Z4q17/SsbnOHWYr4o9jU7je3zwBVp1Ub/aH6xcDwqamLQpCo9sw64OYsdDypcK2CVG74Ea4NpNvUACmeqKZnajNejoVdFJ7WTQUfrraH1tlCnDUeLdeflesTS1ztCDVTDV9ZzW1fovopk6j3K7XckwDd3d30k4VR0X60GH3GuO3D29/+9uBvmEIAbUtyr9KomtZquhauXaGa13pM1Cn/V77ylWB8MKr9peBCVJNDt917iN7nQx/6UDCuAFTPVZM5DeELMQxGTWFUy8xdcEFhiqajE6a7du0KQpTc99P3df0PVhqFH9ruaD8Q7kQXhVWyACTTvsPa1++xB3833/748Fxb12d2yLit9k8ty23S25K26mNHWPvxLZa0ieqoxB/S/otyhwECC93vP67nu9f1NU0z79gZdtwNfbbwujV267l32LuPf8pubX+dvbCrxpJtO83S/nuONWgBypQKX6reG6YDQG14y+nAG2Oj3zRc80OFE3Xsl3umSgdt4VAEhaN5rfUwl34jHfyw/lUfHfiqsFKuAZirlq4TE47OtOq+3D66XOfGroCljovdSQ036D2i3q/UopqmKZQEhkvbbi0zxTiLP1RHp9///veD4EBhgJrGaH1zFzQIF65Vi0TPUU1sUYigdfnhhx8Obqu2l16f26+Igk9tr/Q6PVfNcBRM6HluOxYOOnSBgiuvvDLYZgz1/o4Lk9w8/dnPfhb8Vbii7yj6Xnq9Pos+l6brap9UgvCxl+afliOCkOIpfieojnKK+qTtnTXF6hvN5l7QaXOO22qnnbnG1u+aau29E+z5pyfY+nXjbeWGaZbZnLTmZxVS+K9Vtacdu8zb2W4ZBRf6Chpqay0xbYp1t9RZ59Q6S8/ottTEjL1/1no7YfpOm3DKZtuWabS/7plhuzZNsNXLD7Kuu3da96Md/pv29b9HjtF2gjoSqr6VK48/B6rESJYhHeDpoDtMG2MVjsPNZVBZXPiRy4Uf7mDfLQMqjOtAg21Qfg01P90Zn1yjrUbNb1c6+Zj3A22Pc2n9DQdk/M6jE/Wbaf7rd8jlto35wnpaOoWc96MJP8YyfdfBsAr/uVdIc9zjChnc1c1csxPX6WjUc9w80mcLP1+i7hM1X7nzzjsPCD30PgojFFi4xxS4qIb3cN5fQY+uDqkwRwHHaaedFgQtGr/00kv31UIJU4gQrpk2XMNZTnK3w258uH9F46rBIvqs4e1OmJrk5dZSkzwuvkMKzZIxLbvlqGQ1QBRkeD0pa3hmo9U8scOW/eH19uhfZ9qLNbX2hqkv2wdmPG4nvXmLTfjvadt23kG29eRDrbt1unVpmHqI9dVPskyf//HT/g8S/CgZy3hJ65vYanunT7cdxx5h299ysG25+CCb+44tduFFz9j8pg3+c9J2+95j7MFNR9mmPx1iHa/W+e/RVdylCigRFbTUJCZcFV8bZiXPxTiLgcJQ54lR4Uf4qgausF2MKrs4kOZ71Jln/X6V4MYbbwy2FTpgLLdh/vz5weerFG57HD4bmEvPQeFENU1TQOnOaAMD0Xp79tlnF+3EkQvO1fTE9euhsEDjCiPUT498+9vf3ldb5N///d+Dv4NtY6K4Wl4D1TpRjS81c1FfHCoYh9cXBRFq4q771XRFnbcq6HAGe381g3edLH/jG98Iwg/VatH97ju6GiC5w2jDjyjqn0ODmuxp0P7FjWtwj2tfqEFBqgb9RhoUjGlwTc01qFmyAg0NQ4UfEg4/kF+lqwHiaOo1Ces9eKq1zuqxmafutqazeq3+yLQdVddmU2r32urtrZZpS1rDy/5C7j89mcjYik0NtnLrREun0v6CrxwkYQ31vXb6jG02sSlh41tqLT2t1zLj05ZOetbXl7Ataxps45pae/YvdZbYnTRvbb0ltrWZt3vvgVGXjxogqASjWYYUeGgjHbURVlOZSimMoZ92ruGdpQs/hgo62Abl13Dnpw6awgeNIz3zXIrfTiGDCu3lynX+XGj5mvdab8PBZS5tj3XgnIt1dHQG+s20Hmp9zKXt57p160ZccIzCNrZ0ijXvdSz161//Omi+kHvWP2ws01dYoBoc4RoXoivBnHHGGUFNiiiuxsdQNUDUj4ialDi575tba0MBiLtyY5jeR52WXnDBBUG46GqJaD1TX0DDeX910qp+SpzbbrvtgI5Uc4Vrp4xUeDkpV8XcdIRnSdy2W6WrAeJoBqfSVvvKq7brkR5b8f8dab9ZMcd+sOF4y3Ql7exxm+3vDltlHzjpSf/A8Cl79zufsPf+7eM2b+Fma/gfSWu8rN4mXjbOxn9wgrV8wLPT3vucv4I/ZZeevtz+7vCn7bLWp+2VvRPstvVH278/crr9+T+PsPpf7LCa/9xuybUbBgw/gDjTQZ1qAUT1eq+D7aGqZaN86CzDaMMPlI7Wv0rsFLWcww8p9/mXS4WmcLgRRg2QwlNoNlD/PMBwaD1VU1M129CyFK7llw+qAaFCvpqTuBBBtRF0W/1sKNAIX11R47rPhR0D0fuIali4fkPk8MMPz47tT2GEpuuohkbudBV+uFoimicKOlRDY7jvr4sk5H5HdxnuqO8YV2z7C6v0NUByJRLBVV/SzZ6l6zPWOqfFJs6YaJvm1Vl6Ssoy03rsPdMft/Nan7VMeqLVWL1Zpj/D8by0be4dbz/Yfozt7Gyw3V0TbfIf6qxuY9J2PrXJOjd2WF93vXmdKUu2d/cHL0OgBggqwViXoag+CUQHhSpE0y9IedIBus4chzvMGmn4wTYov0YyP6POPCsU0Znn4SjFbxeeZjkqxXwY6TT12ytozj1brHVXB72uKaJuq/DAOpofQ81HBZDh7alOEqhgOxb8fqVT6nnPb18Zwr9TFG2PHY0PdTv3rzqDFXdb2/2Bwm8dc6vmn47hDlx+siNFEJ4lcVt2yysAkX0fJ2OpQw6xvoOabdtJtZZuTlu6tdv+9tBV9qZpa22yV2MtukJuthJLwkvZxp4JduOrJ9jWvRNsR8d4m/T/aq1mS9LqXnjJkm27/Od6QROaA37VARCAoBLkYxmKOhh38t0h3FBW/P5pe2H5WkvqEtgx8K5PRF+ybiwGCz9G2pkt26D8Gun8HEunqKX47cLTvPSWVdmx0vnp3x+fHetXivkwkmm++OKLwfZWf3O5/nrUBFEHx9ruavvLOpofw5mP+l3CTUPHug/k9yudUs/78PRdUw6Ul/DvFG7+ljueD9r2qw+RXJqGgg/tA9z0Dlx+syNFEJp07LZb5ReADMb/qInmJks0NvT/MonEa7+QvkZfn6V37LRMb68/9PXfPwYEIKgE+VqGVKjWwV9UFXKdAYtqLlMIX//oUvvZzb+2etXwioGHMr/OjuWHfiedqQx3WKuaA1HNKobCNii/RjM/o848D2edK8VvF54mAUi/4U5T629U/0vh31uPa1AYwjqaH8OZj/p9VCsrN5waTbCci9+vdEo177X8aB8d7t9HNbryXZjG2BV7OckNQKKCD+fAz5UdKYLQpGO33aqMAGTfJ8xYoqHRvIkT/U/u3/R/nUz2F/K/iFkqZZnduy3T1+cPqf7njAEBCCpBvpehqA41Zbhnpceq4EFqkeVzO5Lv8EPYBuXXaOfnaDpFLcVvF54mAUi/4UxzoPV3qGYWrKP5Mdz5OFDTtKGu2DMQfr/SKea8d6GlOkONOpEk/PblqdjrqPYFCkD+4R/+Idj+D7RdOfBzZUeKIDTp2C27pe8EdTj0IwSDZ+m9HZbausVSW/xh82ZLb9oUDBpPbdtm6e5uy6TGHn4A1UoH4lEH49qxh8+MoXgGKjzprORoww+UD/2G4YMg/d4DHUij8qhZS3j91Vk/nf1D+dA2VeFjLu336BQVubRPdrW0dGykGrRax9lmYzgUqGrZGU2oirGrjAAklxKowQYAY6ZEOupslzszpp0+imew8EMH6oQflU/rWrjQpd+dQlc8qFZduGad1l+FzRwAlx/VvApfycMVdsvdjTfeGOyndfa4HAddSlufMQ5UY5bQAyOlbT7HbaVVeQEIgKLQwblCEP3NpUKZznRENZNB/rlCMDU/4k/NzNQRZi6dedb6hsql/l3CPf5rvVXgRfhRvqKughZVi6fc6HOXc4FcxxXh7Vwl0rqrfXB4GQFQ/ghAAAxIB+k6WInqh0BnPsIdfJW1fTXF0uaFhnKtQebCpnCNGxd+UHiKH511jjrzTOBYmVQQ1XZS67Kj9VYFQMLL8hcVMpd70zTts8tdJXzG4SAEASoTAQiAIelMZVQ7dRXKVNU29+C+7Hiev6VLmDdhgn+0MsnS01ut58ip1jV72r7BmhvNGz8++4LyoDP/mrfhA23XGS3hR3xFnXlWIbrczzxjf672TlT4ofUY5U+/F03TMBjtoyu9b7Sbb745aJ7EJXpRLSrrMrhFxlVgUAmKuQzpTHRUdfx8ngXJ13YkuDKUZcyzpFmixlJTp1tf80TrOrTJuqYmrXtywn+Onmc2ddlmq9uy2zI7Nva/OI9Gsx3RAZUOsMMHVSo05bvaPNug/MrX/FQhS73EhwvPuetZKX678DS5Cky/8DT1u2lbGQ4ww5e7HQ7W0fwYy3xU+BgOPYZ7ZbRi/37h6f3u2R3ZsdI679iW7Fi/clhPx0rHRFoucrfTgynXdXfJkiV27bXX2oIFC+y+++7L3ls9ynUbe+Dnyo4UQWjSZbvsjlZeAxCMXLmudKgcxV6GBiqciwrng122czjyEoD4s6D3kEZLNY4zr6nP6hpTdvphe21cQ8J6WiZYj/+3b0IieJ6GzAvt1vFin625J2WW9u8LbfjHYqQByEDzV80iCtFhItug/Mrn/IwKHFUdf926dcF4KX678DQJQPrlTlOFodFc7nYgrKP5Mdb5qN9U/bnkGk6gVezfLzw9ApDX5HOaWhbCzduGUq7rLgFIeW5jw5+rlMp12R0tmsAAGBGdfVb73agq3Cq4l7qXfNX8SFif7Zo7wzZdfKJt/fBR1nldq33ky0/aZ7/wB/uXj99jX738bvvX9/5f+9r7fmNfXXi3XXrdn+zsRc9Y0kv571C6jbzCDxV4w+GHKzjlO/xAedM6Fu4sUMtGVC0slI+o5kouwETlomkaHDX/jQo/1FRYx0dx2ldv377dFi1aZC0tLUGBXH8/85nPBPfLk08+aRdeeGHwmAbXjGag+4FyQAACYMRcVfyoM1/qJb9UBTR1aNo1vdE2X3CcXXzhX+26i35hnzvh9/apqY9YJt1hK/ZMsq9vONV+vP1Y+9XuI+25zibL9Hk2NdVrLY0J277gaGs780jLKHQvctrtwo/wAZXmsQ6qCD+qkwrO4VpVdIpavvS7hGsJqNBM+BEPUf0vlXunqMgvreM6zgnvqxWQ6QSQ1vfhNI2qFAoxli5dajt37gxu6++Xv/xlu+6664Lbqjly//3326c//elgeOyxxwa9H8MXDlxLRf3RxQ0BCIBR00F9uIM4UQEtqgPPwkuaHTzB0me22qknb7B3z3/U/ubw5+y8yS/Z3q60/XVHg9294Sh7aPN0e3R7q73c2WBd/msmZfqsqTFj9sYWf48z2dITai1T67+XV5zqh66pQ1T4Qc0PaB2LOvOM8uLOCudyhSHW4XjQ70inqNUrquaHlgmFH7lX79J6H4cr3dx7771BcDFr1ixbs2ZN0AzinnvuCR5TKJJr8uTJduWVV9rq1auz9/Qb6H4M7Yorrih5+KBlWSFv3BCAABgTnZ3Wjj58qUCFH9pwR1UP1kFEXsMRf6ecqa+19je83uaf32U/XPhdO+3QDVbX22C/2j3HPvXSmfalz7/b7vzYaVZz5au26SM1tupDh9gddx1jX9x5qrX1tdhxU3bZLe++xT576d228ZPH284Fs61mcot5tXUFrQ0yUCdqLvwAhEJ0edP2TGeFc7mCEb9bvNA0rTpFBZxat7WfDl+6XMrl7P1YrFy5Mvj74Q9/2GbOnBmMX3zxxcFfZ/HixXbKKacENT4UlKi5zGD3Y/iuvvpqW758eRA8lWrQ8b0+R9wQgAAYM3e2I2qHH+4XxBUUwoWFMVEb04RniZY6a2hJ2Yym7VZXm7LeTI3t7Bxnr7ZNsM0bGq1t/Xizjf796z3rfNl/bGu9bdw90Xb3jLM+/7lNjXts0uROq5metMRUf2hMmtV4BasJoqryhB8YDh1oR9W2QnkI1+By4UccCkE4kAq84UIvTdPiS8cwUeFHuOZH3Bx++OHB3+9973u2du3aYPz2228P/jY3Nwd/TzrpJHv00UeDYzsFHqoZsmzZsgHvB8oBAQiAvNDBgEKQwfoFcVWF9Vc1Q6KuJDMimYy/Fasxr2W61R3RYid+bKXNufAVa06Ps//sPNK+1nGSPfZvLdbxsZTZo2stsfGVIMzIdHVYZs9W67q907Z/JGn/8zcL7G3/dam996EP2Vf/er5ddeb9dvHfP2HNN3VZzQn1lkpO9qeTDPoYyZeBepBXfx+EH4gSdeYZ5SG8HmsdHusVsVDeogIubdPpFDVe9JsOVLsrbuu4+uxwnZZqOOSQQ4KgQ81fVItD911yySXBcz/1qU8Ff2fPnh3U7tBrncbGxgHvB8oBAQiAvNKBf1QBXmfHjjrqqP1CjzHXAknWWE1Dwprm9Vjr3E479rBtdsjk9uBKtlvXTbB1y5ts9/M1lnq5zzIdPWa9vcHLMum0P6QstSNlvWtTtu1pzzavSNq2FTW28yn/+WtrrKez1va0jrMZJ3baiW9ss/pjam3vjElmtTX+QcDYNp1R7YhF863UV9FBedPZRgrW5U0hcJzPCuM1dIoaX9o/67cM1+qJY/hxwgknZMf2p/47tIy/733vy95jQRCi5i3XXHNNcPuCCy4IaoWoqYs6SNVjqv0x0P1AOfAyauCDklGamoufAyNVrsuQDgCjOvbM5WqN5PYfcsOiW+znS++2eq8ue8/AvPGN1jDb7OQfvGrNjT32gYb1lkx0WU/dHvvm/z7Xlv12ltW0b7H0nj1BzY8h+fPOG1dvySOOtJ3n1Nnmv6m1H8/+tb2pZa198MGF9tunZtmJ31llyV2dlk51+i8YunnMA+k7s2P9dDCl4Cd3vmg+KPwoRaGJbVB+FWt+RnUyrAPVqMtT51P4+116y6rsWOn89O+Pz471K8YyHJ4PTiGar7GO5keh5qPrxDqX9mnr1q0Lxov9+4Wn97tnd2THSuu8Y1uyY/1KsZ4Od5raP+skRfiKTiMNP1h3KwO/U/WhBgiAghisXxBHBxnhA4yRSDU3Wt20iXbGxO12yrjt1ul12ZMbWu2795xva1dPtETbDsv09Ph7t+wLBqJ9XbC/8yzTm7L01m1W/+dNNuUHG+wX/zXHFq88x+ZO2Wz/dNqfLfE/6qznwibzDjrYvJaRdW4YVfOjlOEHKpfCjjA1LxtzszKMmgpFasKG6kKnqPEyWPih7S418IDKRwACoGB0Fmyos6E33HBDdmzkMo2NVt/UYG8Yv9NOqt9hPYkOW7Vpiv3wd2+1F19uMK9zh2WCZi9DJCB62A3plKV3bLO6J7bYlLs22q9XzLEvrznbTpi2yT522p8t875x1vmWFss0H2SZhsn+h0jpdIHeZVAu/MhF+IHR0rITpgN3Cl2FF64SLwp6VQiO+l0Qf1FN0+gUtfJoG6pmL+HwQ8cyQ53QAVA5CEAAFIzOgums9GBGXQsk47/2dZ5tPyFhfZ5ZR6rO7m07yVatb7Ypy/9qtW17/KeMchOnPj78IW1Jm/anjTbz5y/Y3ctm2k3PnGan1a2x04953qZfusmaL+iyzIwZlmmYOGgIUq09yKP4hrPOYfS0rYrqu0hX6CH8qG5aBqI6RUVlcOFHuBNb/aaq+RG+1D+AykUAAqAg3MGE/g5lxAeJ2axh1+uStvP4pL8ly1hXutbuaz/enn6l2RqeXmfJtr1moXadI+a/fPIjW+2ge1+2u/9yhN2yer6d2rDOTj1urR28cIs1n99l6cOnmzVM8J8cHYDou1VLD/IoDzqAH0vTMkRTnytan6O2aRSOICooE4RVnsHCDwVbrN9AvBCAACgIHVDo4GE4VUb13NEU2K583UP2jyfcZ3XJVHC7LtFnyWSmv+bHWMOPrLSXsJQl7bB719pB315t3/iPi23lC8fY1ROfto+e+Ihdff3P7YQ3b7VM4zTzJ559Vb9nMo8dUAWa8APFoIN5VcFHfriaNcMJdFG9tH1XgRmVY7Dwg5ofQDwRgAAoCB00qH8LtZtVj9o6KFQHgeowLuoM2Uj6AlFdCw3HTN5qJ7VssoSXsUzas762Okvv2T+EyAvPrH7zXqt7vt1eenqybX6x0fq66mxCfZ+deOJ6a5qRsr7mRn+L6j8x2xSm0zpsi20Ixh3CDxRSeL2iU9T8UAGJeYnhiuoUFeVJ67b6TRoo/Ig6VgFQ+QhAABSFCv3qC0MHFbomvC4RqINEXTZSB4yqXj7SAsZhyYQdWZMIApBUZ8J23H2ItS/Xpfaim6OMRcbzp9OZtjk/WmnP3ubZ2Q990L72zNk205KWnjHFNp94qKXqarLPNhtvE+1k7+zsrf7Cqb474QcKJXzm2RXcMXru7HD4ksPAYKI6RUV50fFG1OXEdTxC+AHEGwEIgJJQDREdJKqWiA42RnPd9RrzgiH4l/GHLn/oyU/Tl0j+Z/S6ey2xI2OJ1XWW2FRrlk7YG4992d5/3hPWML7b/Gdkn2zWaE1BDRh9V3qQR6HpwD181SUd3BOCjJ76/AifHabjYgxHVKeoQtO00tN2UTU/widdtA3V70b4AcQbAQiAsjHStrae+vnwEkEAov8nezOW6Mt/7Y9caeuz2t0pa37es4ZX/emnE3bRyc/ZJ9/5B5s8oct/fP8ARgfAqu1CO2IUg2pUhc880ynq6Kj/nvB80/o81KW9AUfhfhjNqUrLhcLh30DBJuEHUB0IQABULHV32v///n/FkuhO2bitHda7q9fWpcbZzt5a6+1NFPETAANT07LwmWc6RR0ZhR/hq1MpxKRqPEYiallx/U6g+Aaq+eFqo7JuA9WBAARAxUpnMv6Qfi0AGUUzmpHzrGZPr01a3Wbdr3bZI90Ntj2dtGQiFar7AZSGDuKjzmRy5nl4VEiKunQ1Z4eRL1oPaZpWfAo/FEDlUq05wg+guhCAAKhY6vxUg2QSnnW0Jq27UQ1iCsjzLN3VZamNG61j+17b2NtiP9p8ov3z2rNtR+84f6NKPRCUnmorqMCei05Rh+bOEOcWklQwiqpVA4wFTdOKj/ADgBCAAKhYChy2do+3TMbzt2YZq23ttuTklP9IMggqCiaTtnRfr6XTKcv4m9GV7dPsgW1HWWe61jwCEJQJdeinS0/ncu3fcSAVjtRUKFxIUgGJK3qgEGiaVjou/ABQfQhAAFQcRRsaPrvmLPvocxdYd6rGJo7vtgsuetTmnb7evKap5tXWFaZJjMKPqU3Wdc48az1pir1n0pM2vbvDdm+daOlUgWufACOkS0/TKerQXPihgCiXCklc9QX5FK5tQNO04iP8AKobAQiAitW+p87adtdbZzppPf7mrGncXmto8MeaE+bVeQWrBZKp8ayvoda8cQmb6PVabXfarMOfFpU/UIboFHVoUZe7pZCEQqBpWnGoI+MoqhXHeg1UNwIQAJVHuYY/TL9jr02+ucfu2nW4/a7rYHvv+JfsrW/eaBOX9lj9qXVWk2ywRLI2P8GEapP4Q8afcMekpL10cr3tnl1nLbV1dvhjvTbzjj1Wu0cdsgLlRWecw4Uu4cxzP4Uf4RoxUc2HgHzQshUugNM0Lb8UfoQ7MhaFwaoVB6C6EYCUGc/zGBhGNFSz5NZO61vfY6tWH2rPvXSw9aX6rLWh3d54xGYbf4Jnbac2We9B48yryUOfIBMmWKK1waae0WdHn9FmFx3zvM2evs2e7Zli7Ts9q9vcYZaq/PgjahljGP5QrtwlXHNx5rm/oBQ+U6zaMlzxBYWk2kU0TSsMdwnrcF8+QnM2AEIAAqBiZdp3WPv6Pfbg7+bbHx+ea+v6zA4Zt9X+qWW5TXpb0lZ97AhrP77FkjbRvEzGH9L+i3KHAQKLoLZHOni+e11f0zTzjp1hx93QZwuvW2O3nnuHvfv4p+zW9tfZC7tqLNm20yztv2cZF4JR3XTmWQWvXNV85lmFTS53i1KhaVr+ufADAAZDAFJiXFYP+TRv3rzsWHXQFViSnb3W/Pvnrea3G+xnPzzHfvXgyfZ4d4O9bfpz9p1j7rcLL11nB32223Z8aIZt/5uZlplzpGVe5w9HH2U2tdm8ZE7tEAUfNTWWOPRg6339Ebbr3Dm284MzbNuiGfa2f1pnn7jq/9nZk9dZV5dnX1l/pt361Hxbc8ds2/XcOMt07x04UCljbIMKpxzXR1W958xzf/ATPkvswg/VlgEKbaCwjaZpo6OmLYQfAIaDAKTErrjiiqortKIwVJDV2aOq4vn/9aSs4ZmNVvPEDlv2h9fbo3+daS/W1Nobpr5sH5jxuJ305i024b+nbdt5B9nWkw+17tbp1qVh6iHWVz/JMn3+ZjCtGh8KLzKW8ZLWN7HV9k6fbjuOPcK2v+Vg23LxQTb3HVvswouesflNG/znpO32vcfYg5uOsk1/OsQ6Xq3z36OrIgMQtkGFUc7r40BnnsNXQIkrFS5VyAxXkdd8US0ZoFgUtoX756Fp2sgp/IiqzQUAUbyMLzsOACV3w6Jb7OdL77Z6ry57zzBoK1aTsN6Dp1rrrB6beepuazqr1+qPTNtRdW02pXavrd7eapm2pDW8rI5MzZKJjK3Y1GArg8vXpoPsIp1JWEN9r50+Y5tNbErY+JZaS0/rtcz4tKWTnvX1JWzLmgbbuKbWnv1LnSV2J81bW2+JbW3m7d4b2fzlgfSd2TEgv8J9joxkd64Q4Kijjsre6qcCw4oVKwatARGe5qW3rMqOlc5P//747Fi/weaDK1yGmxmoZky4edBgxjLvR6qY04qzUs3H4Uw3qgCvmlrhcGQ4wtP73bM7smOldd6xLdmxfvma/wpvwzXYtC1ToBkOkoq97rDuVgZ+p+pDAAKgrIwqAHEyKbPxk8wmH2x7Pmq2+9Qa+5fpf7Z3TV1t3dq/JTOWSfp/vLTVen32s7Zj7Je75vg7u4SlMp71pWtsau1u+8C0h+2wRJ8/eJbp08vS9i8bTrcHNx9hm5fNsPErttm0B1YFTXD8ifqD/+ahHahDAIJCGetBm0KAc845J3urn2qGKAQZSHialRSAKPxQYSkfl7st5gFzeFrIj2Id/g53WVFhPbxsqhA/0o47w9OLawCi9TnqCk4u/FCAVMz1NEqpp4/h4XeqPgQgAMrKmAIQSSSCq76kmz1L12esdU6LTZwx0TbNq7P0lJRlpvXYe6Y/bue1PmuZ9ESrsXp/b9ffGtDz0ra5d7z9YPsxtrOzwXZ3TbTJf6izuo1J2/nUJuvc2GF93fXmdaYs2d7tvyB42aAIQFAo+ThoUwEifBWU8JlnFTTUPEbNQ8LTrKQAJOq7qnCp8GOk1eWLecAcnhbyo1iHv8NdVrSeKZAMN0XT1ZtG0jQrPL04BiCDhR/adrn5Vcz1NEqpp4/h4XeqPvQBAiBe0mnLdPeYt6nbki9127ZnPFvzRL29tLzOXnqs3l58pM5eeGySrVo+1TY8OcnaV9bbnlX9Q8equuD2pscn2iuPjbcX/1Jv6/zXrXmyzvY87+8Q1/da7ZYOS7Z3+XvM7PSACjZUp6iqJaI+Ym644YbgdqVS8BEOP1TbZTThR7GF+2vB2JVjv0eu8B6mmiF0ivoaV5MrHH64S33Tjw+AoVADBEBZGXMNkKH4m7xEc5MlGhsU+wc1RoK/2cesr8/SO3ZaprfXH/r67x8DaoCgUPJ11mqgM89qGqJChh5X4Wznzp0HTLMSaoDoO4Q7pHWFpdFe8aWYZwxvvPHGoEr/YE2TMHwKlNT589VXX529p7BGuqyMpmlarvD04lQDxIUf4aZCA63PxVxPo5R6+hgefqfqQw0QANVB+zO3T+tNWaa7zx96LdPVY+nsoHHdZ6l09sow2ecDMTbQmWfVmFCBQ/S3Eq8So88cvjSm+76jDT+KTQX15cuXBwflDGMfFCQUK/wYDdVgCHfIq+W42q8MM1D4oXBoLGEmgOpDAAKgOijgDwbP0ns7LLV1i6W2+MPmzZbetCkYNJ7ats3S3d2WSaX6nw9UAXcGdTDhK6eUOxUadSbdhTii8EO1KWhWgnI2VNO0ajNY+FFJYSaA8kAAAqD6ZFS7Y5ABqFIKCAbyxz/+MTtW/lyBKTf8kKiCJVCOooI6LdOVWBNrLLQOK8ik5geAfCEAAQCgyqm5i6rYhwODXJVSA8SFH+GCopoVjPSSokCpDNQ0TWFAtXSKqu+pDmvD67ILPwYLbAFgIAQgAABUsc9//vNBPxmDhR8y1OPlQt8lfLZY4YdqfwCVJKppmtbDaugPxDVhC4c96iOF8APAWBCAAABQxRSAxKkdfbifBJ0t/tznPpe9BVSW4XaKqmCk0vrpGYj7flHhh7ZVhB8AxoIABACAKqd+MdatWxcULuLUTISq8oiDoTpFVfChpiI33HBDcLuSDVTzQ9slwg8A+eBldE0wAABQUTxv/8sU5XN3rkKIClNDXXXi0ltWZcdK56d/f3x2bH8qKCn8CHckmQ+FnPeIl3wtK6rhoWBA62Yu1Q7ReqrHtcyHm6r97tkd2bHSOu/YluxYv6j54MKP8HfQd1QtrpGGH6VeT9lOVAZ+p+pDDRAAALAfhQa6CoVqhVRijRAVlLjcLeJEy3RUp6jqwNgFBuHgoJKoFks+ww8AGAgBCAAAiKR+QRQk6IxYJXUiqs/M5W4RN1GdosaBwo+oq1C5zosJPwDkEwEIAAAYUrgjxnIV1V8CECdxCgTUfGew8AMA8o0ABAAAxEalBDXASKm5S1RYUKn0faIuwa0mL4QfAAqFAAQAAAAoY7pcdVRYUKkUfnzhC1844Pso+NB3BYBCIQABAAAAyphCAXWCqn5A4iAqzFHfPdTgAlBoBCAAAABAmVPfNroyk4KQSrw600DcVZvi9J0AlC8CEAAAAKBCKAhRYLBixYqKDw0UfqjZC+EHgGIhAAEAAAAqzNy5c4MgRLVCKjFAoOYHgFIgAAEAAAAqlPoFUZCQyWSy95Q/F35wyWoAxUYAAgAAAKBoCD8AlAoBCAAAAICiqcbww/M8hjIcUH0IQAAAAAAAQOwRgAAAAABAHqmTWlSWefPmZccQZwQgAGJt0aJFNnv27Oyt16ja4+233x6M63E9DwAAIB+uuOIKCtQVRIHV5Zdfnr2FOCMAARBrN910k61Zs8aWLFmSvac/FJk1a5YtXLgwew8AAED+XH311bZ8+fLg6jwM5T+sWLEi+M0QfwQgAGLvtttus2uvvdbWrl1ry5Yts6VLl9qtt96afdRs9erVQVACAAAAIL4IQADEnmp6qMaHan588YtftKuuusrOOuus7KNmF1544X41RAAAAADEDwEIgKqgGh/3339/MFDbAwAAAKg+BCAAqsIrr7ySHbOgKQwAAACA6kIAAqAqfPazn7XFixfbggULuOILAAAAUIUIQADEnuvf45prrgmav6gZjLsELgAAAIDqQAACINbU3EVXgFHnpzJz5sygE9RLLrkkuA0AAACgOhCAAIg1NXdRsxddCcZxnaBy5RcAAACgehCAAIi1++67LxjCMplM0CQGAAAAQHUgAAEQX6tXm23alL3hu/XWyNu7du2y888/32zZsuwDAAAAAOKGAARAfI0bZ6a+PhR6fOtbZn/84363V37nO/bIrFn24yVLbJ4eu//+7AsBAAAAxA0BCID4Ouwws6VLzS66yOyll8x++MP9bp/w6KN26uOP25yPf7z/8euvz74QAAAAQNx4GTWEB4A4U42Pgw/O3vANdRuoAJ7nZcf6FWN3Hp7mpbesyo6Vzk///vjsWL9SzAcOpTCQYi8r4en97tkd2bHSOu/YluxYP9ZTAKVCDRAA8RcON4a6DQAAACB2CEAAAAAAAEDsEYAAAAAAAIDYIwABAAAAAACxRwACAAAAAABijwAEAAAAAADEHgEIAAAAAACIPQIQAAAAAAAQewQgAAAAAAAg9ghAAAAAAABA7BGAAAAAAACA2CMAAQAAAAAAsUcAAgAAAAAAYo8ABAAAAAAAxB4BCAAAAAAAiD0CEAAAAAAAEHsEIAAAAAAAIPYIQAAAAAAAQOwRgAAAAAAAgNgjAAEAAAAAALFHAAIAAAAAAGKPAAQAAAAAAMQeAQgAAAAAAIg9AhAAAAAAABB7BCAAAAAAACD2CEAAAAAAAEDsEYAAAAAAAIDY8zK+7DgAAKgQnudlx/oVY3cenmY5KsV84FAKAyn2shKe3u+e3ZEdK63zjm3JjvVjPQVQKtQAAQAAAAAAsUcAAgAAhmXu3LnZsfI0b9687BgAAMCBCEAAAMCwXHHFFWUbMiicufzyy7O3AAAADkQfIAAAVCDat5cO8x7DRR8g/egDBEC5oAYIAAAAAACIPQIQAAAAAAAQezSBAQCgAlG9u3SY9xiuUjeBKVfFWGdYTwFEoQYIAAAAAACIPQIQAAAAIAbK/VLVwuWqAZQSAQgAAAAQA+V8qWrhctUASo0+QAAAqEC0by8d5j2Gi2WldJj3AKJQAwQAAAAAAMQeAQgAACi42bNn25IlS7K3XpN7/7Jly4KztvoLAACQbwQgAAAAAAAg9ghAAABAWTjrrLOCdvr6CwAAkG8EIAAAoCysXbs2aAKjvwAAAPlGAAIAAIri2muvDQKO3GHNmjXZRwEAAAqLAAQAABTF4sWLgyYuucOsWbOyjwIAABQWAQgAAAAAAIg9AhAAAAAAABB7BCAAAAAAACD2CEAAAAAAAEDseRn1QAYAACqKrqCSKw67c13+9iMf+YjdcsstduSRR2bvLT9xnPcoDJaV0mHeA4hCDRAAAFA4r7xitmdP9obvP/9zwNs9PT325XPPtSNbW4Pbgeeey44AAACMDQEIAAAonK4us/e8pz/kuOsus69/PfL20//1X7b95put9Uc/2v/x73wn+0YAAABjQwACAAAKZ/Zss898xuz8881+/nOz3/wm8vbr//Ef7cxNm+yolSv3f/yb38y+EYBimO2vs2o+Eh6WLFkSPJ47DgCVhj5AAACoQCqE5Cr73bmCjWOOMaupGd3tMlJx8x4lE4dl5fbbb7dLLrnE1qxZYzNnzgy+0+LFi+2aa67JPqM8sZ4CiEIAAgBABeLgvnSY9xiuOCwrlRJ4hLGeAohCExgAAAAAB1i0aJHNmjVrv/BDTWRUKwQAKhEBCAAAAID9LFu2zJYuXWq33npr9h4AqHwEIAAAAAD2c9lll9lVV11lZ511VvYeAKh8BCAAAAAA9tFVXtTp6U033ZS9BwDigQAEAAAAQGDt2rV27bXX2m233Za9BwDigwAEAAAAQEAdny5YsMAWLlyYvQcA4oMABAAAAEDg/vvvDwZdRnbOnDnBXw0KRgCg0hGAAAAAADDbs8cymcy+4YXf/nbfeNAfyOrVduGFF2af7Nu0KTsCAJWBAAQAAACA2b/9m9ldd/WPf+tbZv/tvx1w+5LaWjvnnHP6b//qV/2PAUCF8DKKdAEAQEVRlfRc7M6Lh3mP4aq4ZaWvz+wDHzAbN86sqcnsa1/bd/v3y5fbgqeesv+cPt3OXrCg//FvfjP7wvLDegogCgEIAAAViIP70mHeY7gqcllRCPKTn5hddtnwbpcp1lMAUQhAAACoQBzclw7zHsPFslI6zHsAUegDBAAAAAAAxB4BCAAAAAAAiD0CEAAAAAAAEHsEIAAAAAAAIPYIQAAAAAAAQOwRgAAAAAAAgNgjAAEAAAAAALFHAAIAAAAAAGKPAAQAAAAAAMQeAQgAAAAAAIg9AhAAAAAAABB7BCAAAAAAACD2CEAAAAAAAEDsEYAAAAAAAIDYIwABAAAAAACxRwACAAAAAABijwAEAAAAAADEHgEIAAAAAACIPS/jy44DAIAK4XledgylxqEUBsJ6Wj5YTwEINUAAAAAAAEDsEYAAAFCB5s6dmx1DKc2bNy87BhyI9bQ8sJ4CcAhAAACoQFdccQUH9SWmwu3ll1+evQUciPW09FhPAeSiDxAAAAAAABB71AABAAAAAACxRwACAAAAAABijwAEAAAAAADEHgEIAAAAAACIPQIQAAAAAAAQewQgAAAAAAAg9ghAAAAAAABA7BGAAAAAAACA2CMAAQAAAAAAsUcAAgAAAAAAYo8ABAAAAAAAxB4BCAAAAAAAiD0CEAAAAAAAEHsEIAAAAAAAIPYIQAAAAAAAQOwRgAAAAAAAgNgjAAEAAAAAALFHAAIAAAAAAGLO7P8Ha+0tQoUifk8AAAAASUVORK5CYII="}}},{"cell_type":"markdown","source":"# 2. Preparation tasks","metadata":{}},{"cell_type":"markdown","source":"### 2.1 Define hyper-parameters","metadata":{}},{"cell_type":"code","source":"# Ratio of images used for model fine-tuning in a supervised way.\nratio_of_labeled_images = 0.02\n# Batch size for supervised learning.\nbatch_size_supervised = 32\n# Batch size for self-supervised learning.\nbatch_size_unsupervised = 128\n# Parameters of image transformation\nbrightness = 0.6\njitter = 0.2\ncrop_height = 17 \ncrop_width = 17\n# Temperature parameter of the contrastive loss function\ntemperature = 0.1\n# Number of dimensions of the latent space \ndim = 10","metadata":{"execution":{"iopub.status.busy":"2024-10-05T14:02:22.497881Z","iopub.execute_input":"2024-10-05T14:02:22.498316Z","iopub.status.idle":"2024-10-05T14:02:22.502767Z","shell.execute_reply.started":"2024-10-05T14:02:22.498278Z","shell.execute_reply":"2024-10-05T14:02:22.501858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.2 Prepare dataset","metadata":{}},{"cell_type":"code","source":"def prepare_dataset(X, y, batch_size=32):\n    \n    X = X.to_numpy() # x is an array of shape (784,)\n    X = X.reshape((X.shape[0],28,28,1)) # x is an array of shape (28,28,1)\n    X = np.repeat(X, 3, axis=3) # x is an array of shape (28,28,3). Each pixel has 3 times the same value (same value in each channel) \n    \n    if y is None:\n        dataset = tf.data.Dataset.from_tensor_slices(X)\n    else:\n        dataset = tf.data.Dataset.from_tensor_slices((X, y))\n    \n    return dataset.batch(batch_size).prefetch(tf.data.AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2024-10-05T14:02:23.727633Z","iopub.execute_input":"2024-10-05T14:02:23.728655Z","iopub.status.idle":"2024-10-05T14:02:23.734724Z","shell.execute_reply.started":"2024-10-05T14:02:23.728607Z","shell.execute_reply":"2024-10-05T14:02:23.733812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')    \n#y_train = df_train.pop('label')\ntarget = df_train.pop('label')\n\n# Dividing the pixel values by 255\ndf_train = df_train / 255.\n\n# BASELINE MODEL:\n# Preparing the labeled dataset for training the baseline model\n# Dataset for BASELINE model has 100% of the labelled images.\nX_train, X_val, y_train, y_val = train_test_split(\n    df_train, \n    target, \n    test_size=0.2, \n    random_state=42)\n\n# Training dataset - Labeled\nlabeled_train_baseline = prepare_dataset(\n    X_train,\n    y_train, \n    batch_size=32\n    )\n\n# Validation dataset - Labeled\nlabeled_val_baseline = prepare_dataset(\n    X_val,\n    y_val, \n    batch_size=32\n    )\n\n# SIMCLR :\n# Preparing the unlabeled dataset for training the encoder.\n# The encoder is trained with a large % of unlabelled images.\nlabeled_dataset_size = int(df_train.shape[0]*ratio_of_labeled_images)\n\nunlabeled_train = prepare_dataset(\n    df_train[labeled_dataset_size:], \n    np.zeros(df_train.shape[0]-labeled_dataset_size),\n    batch_size=128\n    )\n\n# SIMCLR:\n# Preparing the labeled dataset for fine-tuning the model.\n# Fine-tuning uses a very small % labelled images.\nX_train, X_val, y_train, y_val = train_test_split(\n    df_train[:labeled_dataset_size], \n    target[:labeled_dataset_size], \n    test_size=0.2, \n    random_state=42)\n\n# Training dataset - Labeled\nlabeled_train_simclr = prepare_dataset(\n    X_train,\n    y_train, \n    batch_size=32\n    )\n\n# Validation dataset - Labeled\nlabeled_val_simclr = prepare_dataset(\n    X_val,\n    y_val, \n    batch_size=32\n    )","metadata":{"execution":{"iopub.status.busy":"2024-10-05T15:55:09.484882Z","iopub.execute_input":"2024-10-05T15:55:09.485307Z","iopub.status.idle":"2024-10-05T15:55:17.175374Z","shell.execute_reply.started":"2024-10-05T15:55:09.485268Z","shell.execute_reply":"2024-10-05T15:55:17.174375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's display a few images from the `labeled_train_baseline` set with their labels:","metadata":{}},{"cell_type":"code","source":"fig=plt.figure(figsize=((5*2.2, 2*2.2))) #Width, height in inches\n\nfor images, labels in labeled_train_baseline.take(1):\n    for i in range(images.shape[0]):\n        fig.add_subplot(2, 5, i+1)\n        plt.imshow(images[i])\n        plt.title(f'label = {labels[i]}')\n        plt.axis('off')\n        if (i==9):\n            break\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-10-05T15:55:49.828314Z","iopub.execute_input":"2024-10-05T15:55:49.829256Z","iopub.status.idle":"2024-10-05T15:55:50.462445Z","shell.execute_reply.started":"2024-10-05T15:55:49.829210Z","shell.execute_reply":"2024-10-05T15:55:50.461483Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.3 Create Encoder","metadata":{}},{"cell_type":"markdown","source":"A neural network base encoder that extracts representation vectors from augmented data examples. SimCLR allows various choices of the network architecture without any constraints.","metadata":{}},{"cell_type":"code","source":"def get_encoder(inputs):\n    \n    inputs = keras.Input(shape=(28,28,3))\n    x = keras.layers.BatchNormalization()(inputs)\n    x = keras.layers.Conv2D(4, (3, 3), activation=\"relu\")(x)\n    x = keras.layers.MaxPooling2D(pool_size=(2, 2))(x)\n    x = keras.layers.BatchNormalization()(x)\n    x = keras.layers.Conv2D(16, (3, 3), activation=\"relu\")(x)\n    x = keras.layers.MaxPooling2D(pool_size=(2, 2))(x)\n    x = keras.layers.BatchNormalization()(x)\n    x = keras.layers.Flatten()(x)\n    representations = keras.layers.Dense(10, activation=\"relu\")(x)\n    \n    encoder = keras.Model(inputs, representations, name='encoder')\n    \n    return encoder","metadata":{"execution":{"iopub.status.busy":"2024-10-05T13:53:22.548335Z","iopub.execute_input":"2024-10-05T13:53:22.549281Z","iopub.status.idle":"2024-10-05T13:53:22.556977Z","shell.execute_reply.started":"2024-10-05T13:53:22.549237Z","shell.execute_reply":"2024-10-05T13:53:22.555941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.4 Create MLP","metadata":{}},{"cell_type":"markdown","source":"A small neural network projection head that maps representations to the space where contrastive loss is applied. In the paper, they used a MLP with one hidden layer as the found it beneficial to define the contrastive loss on Zi’s rather than Hi’s.","metadata":{}},{"cell_type":"code","source":"def get_mlp(inputs, activation=\"relu\"):\n    \n    inputs = keras.Input((10,))\n    x = keras.layers.BatchNormalization()(inputs)\n    projections = keras.layers.Dense(10, activation=activation)(x)\n    \n    mlp = keras.Model(inputs, projections, name='mlp')\n    \n    return mlp","metadata":{"execution":{"iopub.status.busy":"2024-10-05T15:59:51.557176Z","iopub.execute_input":"2024-10-05T15:59:51.557871Z","iopub.status.idle":"2024-10-05T15:59:51.563379Z","shell.execute_reply.started":"2024-10-05T15:59:51.557817Z","shell.execute_reply":"2024-10-05T15:59:51.562488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.5 Create data augmentation","metadata":{}},{"cell_type":"markdown","source":"A stochastic data augmentation module that transforms any given data example randomly resulting in two correlated views of the same image, which we consider as a positive pair.\n\nTwo simple augmentations are sequentially applied: `random cropping` followed by resize back to the original size and `random color distortions`. The implementation for the latter transformation is a copy from this excellent notebook [Semi-supervised image classification using contrastive pretraining with SimCLR\n](https://keras.io/examples/vision/semisupervised_simclr/#image-augmentations) by Andras Beres.","metadata":{}},{"cell_type":"code","source":"# Distorts the color distibutions of images\nclass RandomColorDistort(layers.Layer):\n    def __init__(self, brightness=0, jitter=0, **kwargs):\n        super().__init__(**kwargs)\n\n        self.seed_generator = keras.random.SeedGenerator(1337)\n        self.brightness = brightness\n        self.jitter = jitter\n\n    def get_config(self):\n        print(\"get_config()\")\n        config = super().get_config()\n        config.update({\"brightness\": self.brightness, \"jitter\": self.jitter})\n        return config\n\n    def call(self, images, training=True):\n        if training:\n            batch_size = ops.shape(images)[0]\n\n            # Same for all colors\n            brightness_scales = 1 + keras.random.uniform(\n                (batch_size, 1, 1, 1),\n                minval=-self.brightness,\n                maxval=self.brightness,\n                seed=self.seed_generator,\n            )\n            # Different for all colors\n            jitter_matrices = keras.random.uniform(\n                (batch_size, 1, 3, 3), \n                minval=-self.jitter, \n                maxval=self.jitter,\n                seed=self.seed_generator,\n            )\n\n            color_transforms = (\n                ops.tile(ops.expand_dims(ops.eye(3), axis=0), (batch_size, 1, 1, 1))\n                * brightness_scales\n                + jitter_matrices\n            )\n            images = ops.clip(ops.matmul(images, color_transforms), 0, 1)\n\n        return images","metadata":{"execution":{"iopub.status.busy":"2024-10-05T15:59:53.819783Z","iopub.execute_input":"2024-10-05T15:59:53.820693Z","iopub.status.idle":"2024-10-05T15:59:53.831360Z","shell.execute_reply.started":"2024-10-05T15:59:53.820648Z","shell.execute_reply":"2024-10-05T15:59:53.830389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_data_augmenter(inputs, crop_height=17, crop_width=17):\n    \n    inputs = keras.Input(shape=(28,28,3))\n    x1 = keras.layers.RandomCrop(crop_height, crop_width)(inputs)\n    x1 = keras.layers.Resizing(28,28)(x1)\n    x1 = RandomColorDistort(brightness=0.6, jitter=0.2)(x1)\n    \n    data_augmenter = keras.Model(inputs, x1, name=\"data_augmenter\")\n    \n    return data_augmenter","metadata":{"execution":{"iopub.status.busy":"2024-10-05T15:59:54.623941Z","iopub.execute_input":"2024-10-05T15:59:54.624825Z","iopub.status.idle":"2024-10-05T15:59:54.630448Z","shell.execute_reply.started":"2024-10-05T15:59:54.624782Z","shell.execute_reply":"2024-10-05T15:59:54.629620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can test the `data augmentation` module:","metadata":{"execution":{"iopub.status.busy":"2024-10-05T13:53:26.119372Z","iopub.execute_input":"2024-10-05T13:53:26.120052Z","iopub.status.idle":"2024-10-05T13:53:26.141522Z","shell.execute_reply.started":"2024-10-05T13:53:26.120010Z","shell.execute_reply":"2024-10-05T13:53:26.140700Z"}}},{"cell_type":"code","source":"# Prepare 10 images\nX = X_train[0:10].to_numpy() # x is an array of shape (784,)\nX = X.reshape((X.shape[0],28,28,1)) # x is an array of shape (28,28,1)\nX = np.repeat(X, 3, axis=3)\nX = ops.array(X)\n\n# Get augmented versions\ninputs = keras.Input(shape=(28,28,3))\ndata_augmenter = get_data_augmenter(inputs)\naugmented_images = data_augmenter(X)\n\n# Display \nfig=plt.figure(figsize=((5*2.2, 2*2.2)))\n\nfor i in range(augmented_images.numpy().shape[0]):\n    fig.add_subplot(2, 5, i+1)\n    plt.imshow(augmented_images[i])\n    plt.axis('off')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-10-05T15:42:07.339911Z","iopub.execute_input":"2024-10-05T15:42:07.340584Z","iopub.status.idle":"2024-10-05T15:42:07.751629Z","shell.execute_reply.started":"2024-10-05T15:42:07.340543Z","shell.execute_reply":"2024-10-05T15:42:07.749975Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.6 Create contrastive loss","metadata":{}},{"cell_type":"markdown","source":"The loss function for a positive pair (i,j) is defined with the following formula:\n\n![Loss formula.png](attachment:d3ca16ca-578c-4e56-bd85-cb06aaa15031.png)\n\nwhere:\n* N is the mini-batch size, 2N is therefore the number of augmented views.\n* (Z_i, Z_k), with k different from i and j, are the negative pairs. \n* sim(v, w) is the cosine similarity of vectors v and w.\n* tau denotes a temperature parameter.\n\nThe final loss is computed across all positive pairs, both (i; j) and (j; i), in a mini-batch.","metadata":{},"attachments":{"d3ca16ca-578c-4e56-bd85-cb06aaa15031.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAdMAAABVCAYAAAASYA7cAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAEnQAABJ0Ad5mH3gAADTySURBVHhe7Z0LXM33/8dfuf85jdqwbBGpX+bSCFstpsw1q/BruSuXKdfCClOY2n6cMPlNbMsml9UIU7NcMmLlRygKlSKXoihddT3v//d7+lan0zl1uhnzeT4eHzrvc873+/lezvv9+Xy+74sacYDBYDAYDEa9aSb8z2AwGAwGo54wY8pgMBgMRgNhxpTBYDAYjAbCjCmDwWAwGA2EGVMGg8FgMBoIM6YMBoPBYDQQZkwZDAaDwWggzJgyGAwGg9FAmDFlMBgMBqOBMGPKYDAYDEYDYcaUwWAwGIwGwowpg8FgMBgNhBlTBoPBYDAaCDOmDMbrREkKIrYtxbLg+3h1ykUVIjXCB47LgpH6OtS4yr+CbbM9EZyU3YBrVIL0467o63QcGYKk7mQjmrtX1gQnIJfVFqsVZkwZjNcFeoZony+xrdVMrLPQhpogfrkpRW60L+Zva4kF68ZA69XodMNo2x+OHkY4v3QTTqQWCsI6Qvdxevc5DBraCxqCqO68AUNHZ5ic94DHifuceWbUBDOmDMZrATdT+XML5kd8gvWz+kFUV6NEGbi8bQp6qg3ArIAEFAvipobS/4Tn/KuYsH4K+oiaC9J/OmpooTUCrsta4ivnXxBXUPdpId2/iN/ODMYE0y4NGzS10MZI11lo/dUa+MXlCkKGIpgxZTBeAyjjLDYtuITxy8ZBt2U91KskBZd+DUEiruJIRBJyBHGTQmn4c9N6nB3/OSbqthWErwvNoWE6Eys1f4Lr/pt1HLwUIOFUIM7YjoFJxxaCrP6oaZjAYWUHiF1/RXwxW+9VBjOmDMY/nnwkHNkFP+M5mDKgvSCrI817YZrvPuzw/gXHXIdBUxA3HYTihGB4+/XG8inv43UzpVLU3oX5bEuk/Wc3TqfXYZG1+DZO+z/E9E+NoNkoy+ItoWU+GbPTfOFz+tEr9Kz9xcKMKYPxTyc/Gge2xMJqwgcNeObYHCL9MZi3aBI+1GotyJqSLFw+sAdhVmNhqtVSkL1uqKFt3+GY1PEwdoTcgUSQ1gyh6Maf2JM6DuM/fFOQNQJte2HEJA38uOMU7qjWkdcOZkwZjH80hPxrZ+B/1xgjjDq/Ik5HHPk3Eer/CFYj+qHjK9PpJqBVD3w0XgdH9p7DbZWMWDaun/wD2ZOGoV/bxjxx7WDw0TDoHgnCmdsFgowhCzOmDEZjUvIYl/etwaRx0+DiuR4uk6wxyX0/LqcLXpml8QhYNA3jBnaBmpqatKkPXo+wzFLuzTScdjWpkKupD8Y0/3iUZobCVV+9Qt5TfBKxwV6YZTUe02dO5bbVE4PtvRAcn6VgCS4fCRfDEGMyCH06K3p+RihJj8S+lbMx08UDXl5ieLougrNYDJeRyxCczvWrNA57Jvaq3L/X5QrPztL4XRijLvRXzQxel58iNcIXrrOc4C5eD6dxH8HceT9ic7ntlKTgwq61cHRyh9hzMcbpD4G991mkllTvdWnCJQTG9MWQPp1qGQAUIv3yfqyZaY6e3Pn6zM0PFyo8YEuQfnotRrmHcSamqWjq/YugN2AgtMKjEP+Uv0dqIf8GQna3xNxxfRUsjRciNWwb7AdX3nuK2xh4R+cJ3ylHDa31BmCU1nVcjK9/sM0/GmIwGI1D0W066jyCtGx/oOs5JWUySQqFrjAjPbu9FPdcUibjKU6mICdjzooYk1vYU0HIkRJIM0SjydX/CqUVy3xekkW39zqSFm/9MJScAmIpp/zt4vt0yn0MiURjaM3ZVJL5Fve9O+Rvq0NaLqGUJYiqUBRHfpMtyD3sicz3SigrYgMZYx4FphQLsgJKC9tAZtz+dcWRVC6VUpxCYZ4WXL8G0aL1buS8M6qib5KnIeSkJSIDl9105OvVtPP6M2E/JfQ05AvueAaRU0hK1T5TEd3ztydoraTQrFJBpogSygjfSGYi6RiisulOIDe/43T2qJhsTVwo6GGB8PnG5sXsvzTOl0bClDwingkSZZRSVthaMjDxpqjCqme0oq96luS0dTcdDAykwIPf0jyD0eTie4AC+dflLTiSUmTvvXJKb5LvSC3S8Yjg7gaGPMyYMhiNQh7F+U4hEUaTOLKq0pMk+ZEVDMg+MLmK0ZCknaQVgzRINGILXcrijW8e3fZzojn+8Zw5qU5xpJh0OWWt7RpKz+R1Xd5FEhtrEPScKShF5tvFkSTWFVH/rVGcOq1OSawPmcKe/O/J7VGqOBfIGFMO3tBz+69mTLlXKYHzpIZEZOVLcUUynROMOaBDVr6xVY5Lcs+fbLnvVDf0ORQpHkbov5WiFHW6nJLr5DPUhGy+WE9isbis+fhxhuJX+tFlNIn05pJvTLnxbgJe1P6l512XZgTeEwTKSKdQFxMa6XuTM6tVkTw9RSusPehUSqUZlDwOIgcDMUVWvZg1cI8CZ+gSZgRSiiBhVMKWeRmMxiA7EnvFR5GrMwSmfd4QhGWode0H8/4PcODENTwRZDxqHc3gsmkpep3cjFXfX8TDqN1wvzoc6yb2RE0uN63eao928mufbd/D2BncvDHhZ3xzIBZFghjZT/AgozP6vqsJRVGaai1aoS1OYcem3fjzjkzGnWZvw2j2COiq10VFaMHE0gQ9ZUNv1JqjZWt+z4NgadqjynGptWgJ3pUp9dEzVF1UzMOTB5ngOo1ONYWWNuuJSb/+Dr8Nq7F8+fKy5jAJ5ro5+CtuIHYHi2Hfu72CZeKy7ED66tOxJ6meSRF46r1/AcldHJo9AOr2h2rO7NSxG/rqZiApLQs1LfRS2gUc2NMD04Z1l3t+J0HenRwM8FgI8wrnsVJkXj2D3z42QFeVo2c00K2vNriOQPpUglEFZkwZjAZDKLxxHntu5QIf6OKdVnLqs7km3u3bGbmnbuFelQgHPpZwPraLByDiC1t8vDQTDm5j0KWFUvVbA23RrXdfaCMT4aev42G5ci7MQyZnl5TRTG8svlzTB5Fb58K8R3s0Ux8Iq8XrsOPIXXSZYAlDUV1URFu83aGdEqWiiQ7qqmrtIuRlqpAgQK0NOnTugDblp4uycCNgNey+LcBcn9WYoK/MkDWHevePsWzHAnzStZUgqwf13r+AmgYMPl2IHfM+VMHJKhOpuYWVg51qcAOEi6fw27Bx+Li7vLd1M4iMrGFTxbBnIup0OHTf16l7mFNqLp7XZPxfU5gxZTAaTCkyUu7hLv/n7WP4YZMXvLxkWwCS+jpAvNoUXeVnWmqaMJq7GAu07yMxtyExB2po3VYEqWmIeYwMVWcOam9j6OqfcMb/G8wz0wNyL+PotrVwHD8EH83dj/h6ZN/5O6DcWzi00hEe90dim48DjGsM31FDG/2xmDf1Q2jVa+BSnbrtX0CtPd6znoWpH3ZBg1Mr0H38uf8iRv/bGNqqHFJ+AiKOFcCoe0eFKxaMusOMKYPRYJqhXQdNaPF/9raCY/mSn3yzUzQDKcD90BDctXaAxc0N+Hz9SaTVy35xs+P83LLl3T6doVmuIVu3g0YNyVkpNxFX7rTBANsV2HE6HlScjrhzv8FniTFSf14N1wO3VYxvbExaoZ2GSPi7NiQouBeCtVPX4+aQdfh+2XB0bVOu1ggluRl4ViAcATdzjN23Fs7uKzDTfD72xDdGerw67F9KKXJj98PZ3hluDmMxcs1Jhd7M1dGAlqi10plu3dIHEooSInEspif6da9HEg8tEf6vccYg/yiYMWUwGkwzqPc0xDBe/0cnIaVQkXIsxKMb8UivojgJxYmHsO70h9i4eRO2brfB4y2rsebwnYrQE9XJw+2rkbjPKV0T8754p1zZvfEW3tVMxF9JaQq3WRp3GFO/u4CKleAWb0Hf1BIOXj740a4Njpy6jsfCWy+OdnjrXW4E8Bd3Lms8Ebxh2oP5i/+CoXg7VlnoVck5TAV8GNJErD2bzr9C/rV9+CbVAp7r5uCTN4Kx7mhcPc6zLHXZv1SCkntBcPdtC+cfNmPNXHMkfbUZB2LzhfeVkJmK24810aNTeyWzyLqmDyzE3cvnEK79HvTeUWEGXUE2Um8/AtcRaLDpbDWYMWUwGgE17RFYusYKouhQ/BlTPaqQsq9gl9sx3KvwDOJkuVewwy0KFi5j0bVFW+hOWoGtk7OxY4UYgUnKFWzRkyzkydlryr6O3/3DAYMFWD+tb9lyL0/zN6E9UAf56VlQtsVivz8QniZnVlp0Rs/eb0O7W0dUdad6EbRGJ+3uQH4GsvKVzYtLkH5uC6ZabUOpqQEQE4rDAbvgLSyti90XwtrUDA4PJ8B+SCfu889x98JjjJrQF7jCzbx/64SJA7pVX17lY2H3fYdtv16WG/jIU9f982Qhcu859HEYyV3vUqQnJ3IDlTeh8UbNBlCSmYY7uVrQ76LkSkjTBz7BnAkDVUsfKLmL8wHngQG6eLcuiR0k2Ui7kwUd/S4vIJ3kK4jg1ctgMBqIJCeafO36k8hsDQUlZlWEREhy4inIfRF9HZ4uyCRUnPY/8nUwJe0qYSESKozyJhPuZ6khG6sqUB4aA9EYcj91vzI8pTzOVGEoRhZFikcQRvpSnIKQzbJtatAg1z+qxBZKssLJ02QibY2S6Z0QylI9NEaIC4UO2frfqbr/itCY6uE35durHmohoQKuX4awId+454JMFgk9j9tLdoOsycX3dwoLPUDfzjMlEb8t2aY3j/zicoTvlPOEwtxMlcSwlu+X/74puYU9EeTyNGT/5Tzl+mFMMPWh2JrCf/jY0dCVpCVaREFpij4o3DN9vqaIvKpXXhmSZH+aLELd40WzQslFy4DmBT0UBAxZXpgx5RWNn9Ncclq9gOy9L1OeIGfUgZKb5Dtau+LHWl2pMf52ih9TVIAH2RoZ0jCbaTRjmjVZ2LrT3shHZdcqI4w2WhpVKl6RbaXBKLlFfhMMKpWx7jCysdtCYRllSrTcmOquP0wRe93Ibt4q2rhxJc0wNiWb1bspQiaGsJJyZexQNWZUoIRTxMPsdtKpk9+Ti/0Sctsopo1ujmQ7w418Ix6W9bkkjvwXTiULIy2hb3rcsbmQP9fvkjh/WjjNgozKExeIjMhimiv33lOK83elaRaVxyoysqBpC/0prkh+e1pkZDGVFvrHVcbCShW3HtnJxeZKkSSR/5TJJL70tPK94icUc3QzOZjpcedtJDmID1NUmoLzId2uiAzczipMYiFJ+Z2cB2mRSFePTJXE5jZo/+XwMaqmGqTtFka5gkgxNQ+GiJ5x71uQsfiiyjq1JGor9RdZcv3PECSqUNsAh/FijOnzWPKzsyTnkIfcJblHgfO21iFQmFGVGjLRMP7RVBjTul73rLPkZmBIDkEPqhuml5ayGaSGQxA9brROCwMLWHAz7hwqTdxHDjuuKzaYKYE0rwl/X2WJPJQMFmQpvEpbTfTI2i++WiIGKXnh5NHHUno8TUsORW21IJG1HyUq7AjjBTwzzUXsz19hSaYVHMy1oEalKG7VjnmD1ZvW6NizJ7SFVwxGrbxhCKvZWgg49L+akwO8VGjCyMoaBgHHcD61sUqR5yPhSgRSR07A6L6luHroGgaZ9VDg1EMoTE5GKwOthoesKESCnDs3cRHv15J7mJB/PRT+Nz/B9OE6ChxcJMi+fAJ7tcbik/faCbImIv8mTvo/gM30j9GdedoopIlPC++t+Bs8V8VjpuNY6PGZUXKSENXyXbzNvMEYjBdEewyYtRyfR+zGwWuNEQ7yIlBD2wHT4fF5LDYdlMno1CDaobflHNgVn8OetetxUNsOU/QVVUrNRsz5Z/io31vC68YmlzPqkUjVeR/vdavBm5buI+S7AHTyWohPuyjKiZWByOAT6DrpYxjUp+C7yhQjJeQnbO60GCs+1VYh9Ob1pImNaRYu+/+IXzpbYYIxPwLjRlLRV1Bk1gc1hL4xGIxqEEqLi8DnYigtLJb+XxfUND/G0m97YdeG35BY/IpMT9U6wWzpSnywaxsCEmsJH1EJPlnDVPx0+ies/8oL39gaoI3wTiV8+MoJ/JQxFCO0G5AdqSYoDTfO34KGlRH0WyszTaXIPO8Hr6dz4Dmll8L0kmXpAw0w65MeTarIKTMcO72y4O75GfSb1Gi/2jStMc2+hmN7I6FtY44Bb3C7ogc49SvBeqgWG9289JQi934MLl+OwX2+fBbj7yPzHLzsbTB62jfSLEt3V8/A6M8c4HWuPH5RFVqg08gV2G18Cm67riH3lbGnI/DVbiMcctuPmBdxH0qSEbwzFZ85f6xamIlKlODJ5V+xbdd5pJZwv6trJ3AwrB+W2xopmVRwBj31JDZ45GKVz0z0rshXKEt5+sCRMG4qo89Tch8nNuxE9qpvMKc3H0jNUIYa/+BU+LuRKUXGcVf0HH0Gc0NDsMFcg9MJW+B21xJbpuvXmMibUQuphzCzy0T8JY7EreVGVZ/rUDaSTuyGePsZkOEg6HLq93x0SwyfvwSzRvasElTOQ7m3cWLXVmw/3xyDDSW4eCoTg+dZo/PFs7jf+z3QYS8cGLAL//tq6N8Qb8hodEpSEOGzGUH6S+E5SpVsOS8DhUiN2IX1QT2w1nMUOr1yI/FMXPZ2hO2XaTB37InHd9+E1dJFmKEsjWBBLHatOowODoswXml+38c47mSPkDG+2DyqqSYnuYjb9Q12d5iOVeMNqukOhhxSN6QmgS8HZETQWEJBj4tJknGG1tj/QDGyNR0Z9UNZKSxpvKEFGdj9TDEyMYpl8Y8mZOZ+omqdwqJ48rfvL1M2S0JFcb5kJSovl5VDcf5f0Qr/m8Sc4RkMBkM5TbfMW3gXV47HAUON0LtDKk5sPIK3l01SsmTxAsm/BC8TTSiuMK9CM9mEy/kv4xpZPhJ/cYfN5vb4YqUNeosqPbzURH0x/YsZaPHVAiz7JQFlvpESZF/Yj7U/PcHwiUPLnMO48W3LniawNCnEb+IDuJDdFvq2bkqeLTEYDAajnCYzppSWiMhooL/ZO8jeuw3HBy1WvuZOqTjuPBjq4/cgqU5ZtQtx79Ai6Ks74lCqilk22w7C8vAMfkZevxa+DEZ1ScH1osiOxO6vDyDz/SH4UFfeQ5Ezkv/6CBNMn+CXr3/hjCR/kp/jztULuIVWaN9OJoF2s3bo8Db3/VsXcPXOc0HIYDAYjJpoMmNamnYfkXgLb1/Zi53FU/HV+O7KY7bU1NF9+HzscB2OrnXqUQtoGozEsh0zYdzxdY61IeRFn8bPfD1NZUmom7dHpx6anJE8jpPRzzhBC7yh8WbZewqpPWdoOQpn8Kyxxhprr0njaSJjWorM1Ht4zP1foDMZbrP71fLwWgT9cXb1qOvXHKL3Pm3UuoSNA+98tRyaCk567e19OB2va52OUmSnP8J94VXNpCE5PY/7vzW6ffwppmgk4s+ou+AlUvIf4OaVx9Cwn4DhOqpVlFA4g2eNNdZYe00aTxMZ00Jkpj1BLsZioYOZUkNHudewb+VyuLvZw9x+n8qFiPnv7XH+HM5un8N8pAdCUwuFd14WmkNzlBcyFJz02lsUvh3VWdiOqsjU06yVDni7w/9J/2rWdRQWLRuG+//1gU9ECopzExG6fRu+6+iKA54W6PIyjU8YDAbjJaYJjGkhUkO9sX7v3VoSM2Tj2q6fkPpvN6ybZY43fv4RR2Mr5kfKKbmDw+4H0c7ZG1vWzMPYpA344sAtbm6mIv9IB6RmUO9jAiv+hF9/gDRFJ6PwMZKiuRmvhimG9OlQJsu+hj/uzkXMVScY3AzE1h3HkNzLFdEnvsRwrbrUOWQwGIzXm0Y2poTi+AB4njPCf7bNxiBBWgFl4caRo7icyWn70nu4cHMQJvQHrgTux29axhigpyi1lyyE/MhAHO4zCZ92bQNKT0bsYw101Win+oH8Qx2Q1DoNhcNaa4iunkVEnPyghDtvMWdxKFoDI9bOgHmnssX00gc3EKGlDZ1ugzFu1iIsX74IsywM0fGlWjKXo+QhwrwdYWU1FdPHDYTm4LnwDnuosMgzpZ7D916ecBrXkxsIdcf4PQmo4t+WfwOHvlmMcT3VoT5wEly+/qXs3mQwGIy6whmIpqE0nvysx5BHRGbZ6+KHFO7tRI6+0ZQjG2oqrWghIq0qdR1VQUK5YWtIG1bkE/t6FXQrrwNZrR5h0W066jyCNCw20tmKclwSKk45Qe5mBjTM+QglSeNJy5DWNdQYQUu276fAwMCKdjjkHEXdy3n5KoxI0inccx6tCLlXFl8reUYxvnNJD8ZCRSIFSBLIb8IwGmaqTTD2oshqNR8fUcgSe+4eqrkQFoPBYNRE0xlTTrU9jztIzrZzaPWGVTTPblVlfcQKyksi1VSIVxl5FOtjRdBeQ2G5L53abxqkdSVnkM0wPX6dWagd6UDisDThAxx8Pc1AL3IwMxbqaVrSsBGOJA68SmmyCRt4JPcoyNGobFvVmoh0LVZRQKx8sem/j9JEP5o4K5AeynZIWqJKQ4mh5HgaQksm/EBnDjqSBgzIXr7kVcElEo/gvlvwshwlg8F4FWlCY6oKQpYkE2+KKnxOiX7utEPVWSY/47DSJpFdIKUwPVh3pNmSrGmEq79cEeMSyrkXTWFB22nJMO78juCujYoV/JuWEkoLWkQiGJFj0D0Zg/iUwtyMOeOvqGgxX9B4M1n5XKeSvIskNtaQyfZUBl9XcmKdV0UYDAajKk3kzasiJcm4EpiCkbNHoG/JdRwK6wEzvTJPU0o9hqWD9WH+9Xlk8nMleXLuIeZiCUyGvIfOzOu0jkiQHeGLhT/0wopVNjDsKOts1Bwi7X4YMs4B4u++hPHJUETcfRmSNzRDh/c+xnSzf6HHW2X3SBnN0bI1H1hbhKIS+YwfhUiOjsO/9N9G87a9YTlnDPCbL3adS5dOv6UhXPE30XpAD6hLXzMYDEb9+HuNaYt/wXKNNYrP/oy1Xx6D9hcTZUr8lKLgWSr+9DqCi9WcQgiFCVdwPFUXQ9/T+psP4lWmBMUlikYqMojU0a7Ny5AQQw0te0zE9tP7sPzDtyozNhXdwcVjsRCNGA5jHVkjy5OB+Evt8EEv3nu5LfSsZ2GBwTV8v/e8UCQ7C7F/pWNI35oKNDMYDEbt/M12SAT96dtxevd/8NWWNbDVr0w3qKb1KbbHJyNssaIAmyKk3IhCtMYHGCjzHYaqNMMbxrPx37kJ2LB+Py6ny8fpliI36QS2fnUI72z9AuO71xYmw1eweL96GFFDmvpsBNyrrSR0PhIDvLE2/d/YumUG+sl7Wecl4mKiHgwE72U1zYGwmW2KzJ9+wP4rWdwEPQ0J0d3Qu5u8EWb8I+Er5mxbimXB94WViVcBvmKODxyXBQsDwNcBQn70DsxeE4ykhpTdk6apHVuPJDiyZCOau2fWBCfUWrbwpZ7UUUYUTqsZY5BGc9CTSOzftgfnUgtBuTEIPhgNs+XWGKwwdx6jVlq8i+Frd8PXkhDiNgXjJi2Gu9gLXp6u+NxqAubsjEfPlXvww6zaslfxvIWP7D+HpXRcYwynoGQUy4YT1dSK0xEXGY7QQF986zYXZrrC4Cj3D+w6lVQ1lKUKfDzzZjj6iLD1sBj2vauXqipNjsWFD3ujW8Ut0h4DpsyFvUYINu+9gKePbuB8l/dh0K7p56X5+Y1R3JpRb+gZon2+xLZWM7HOQvsVWYngBrXRvpi/rSUWrBsDrVej042AGtoa2sPD5BKWepxEam2rZ0qg+2HYHaCDoX01BUl9eAOGjs4wOe8BjxP3FYbgVcAptJeT0jt0eNUWOiUN8SilnEhvstTVoWFznWie5RRy8Q2vWk6M8TeTQ7d8Z5AGd0tBz5H8E+sZrlT8iCL3riQLXRFhxA6KlXEWqqSAUk55kJXddxReEQIkTzGlBH5BTiGPhNflPKEwN1OCyJa8vnUmS9+b3N3VdOTl5dEvv/zCawMKDg4WpIwXSzGlhbqTyeS9dFvh/VQLkqcU6T2ZdNGf7P3jqUgQNzWStJO0wmQO+d1+vUL/KpCkUZi7Fdn6xtSjBGQhJfvPIi2HIHrcCGaCLyHqbmJPvrdyBEl1Xl5jynj1KE6iQIeyUBsN25/oVr1r15ZQTszPZKdnSVuj5G9e3pB+Q/ZrZGuzPqMID2fyS5I1rLyn+DyFHr5lNVv5lb4RJI5sXD/eyMhIEovF0ubg4EDGxsZSI8q/Zsb070Hy9BS5GozhrvUzQVJHSq6Tj6lG2X29JISeCuImRfKYQl2HkrH4Ir2mplSKJOUoORpM5X7HdTwLpTfJd6QxN5hObaTQviJKCXImA7loAFmY7w6j8WihA0u3tXAyECEzwB2L/nsR2fVaoWkOUe9p2LTDEIHB11G5QMov7YoxdeEVdGwXjX3fboKXlxfXNsInRAM6nVoJn+PIT0DEuW54r5v88141tNT7BDNsDACtgRig17jP3Hv16oVx48bhs88+Q//+/TFhwgRYWFhI38vJyZH+z3iR5CPhyC74Gc/BlAHtBVkdad4L03z3YYf3LzjmOgwNWTRUDUJxQjC8/Xpj+ZT3UVteuH8yalpDMXt2Bv7jE4Z0lXUJd/5unYX/9WH41LixnAtbQst8Mman+cLn9CN+JF4dwagyGI1E2ZLaIP6OFlmQZ3h6/UeGRQkUuP4Xii3kt1BKOZc2k5l0RqmgWflRkvRjyRT6Xw/6wmYQiUQmNGP1dgpJkh/VSigv0otMJvtTcuMMWxVSPiPlYTPTv4m8cPLoY0gOQQ8aaYbyIsikCI9hpNFIS5SvNhIqjPImE9E08ktU9khHnhyK2mpJfTzCG3lWz2/XgkTWfpSo4NkQm5kyGpkW6Gi2BNvFVhDl/o4vZ23E7yn1rOrTsicmrJ6E91rxY8tmEA10xukcOQem8nZkOrpLP9YV5gu+xMZfL3Izwb+we70jRnWvXiy9rdEy/LXfFl2b0KlDfibKZqYvGkL+tTPwv2uMEUadXxGnI478mwj1fwSrEf3Q8ZXpdFOhhlYGxhivexZ7z9ypwSFRBu78nfR/jknDezXyrL4dDD4aBt0jQThzu0CQVcKMKaPxUdOEkYMbvrbUAW75YOn633Gvnh55rzLq6lVTQci//sdR8hiX963BpHHT4OK5Hi6TrDHJXSb0qjQeAYumYdzALhUhUOqD1yNMGkeehtOuJjKhUYMxzT8epZmhcNVXr5D3FJ9EbLAXZlmNx/SZU7lt9cRgey8Ex2dJlyiqko+Ei2GIMRmEPp0VVUomlKRHYt/K2Zjp4gEvLzE8XRfBWSyGy8hlCE7nC3LEYc/EXpX797pc4dFZGr8LY9SF/qqZwevyU6RG+MJ1lhPcxevhNO4jmDvvRywf3lGSggu71sLRyR1iz8UYpz8E9t5nFXqqliZcQmBMXwzpU9sSZSHSL+/Hmpnm6Mmdr8/c/HChohxlCdJPr8Uo9zBkC5LG5wXtv7UOBozqhPCLt/FUECmHH0Cdwm61TzGu2rJ+IVLDtsF+cOX9p7iNgXe0ogpmamitNwCjtK7jYnyGIJOBG9UzGE2AhIpu76XJGryO06unR96rzWu1zCsUWdCy/YGu55SUySQpFLrCjPTs9lKcrDNacTIFOfEpII3JLUzGnSclkGaIRpOr/5WqeaQlWXR7ryNp8ZoSQ8kpILayWIY0LeYYEonG0Jqzcs4mkjvkb6ujvIhGURz5TbYg97AnMt8roayIDWSMeRSYUp5JvIDSwjaQGbd/XXFk1fzixSkU5mnB9WsQLVrvRs47oyr6JnkaQk5aIjJw2U1Hvl5NO6+X57kuoachX3DHM4icQlKq9pmK6J6/PUFrJYVm1eRnXkIZ4RurP/bQnUBufsfp7FEx2Zq4UNBDVZdG68qL3P9zivO1Ieh4UkStObR5b/1hZLL1KhUKkjKE/upZktPW3XSQL+px8FuaZzCaXHwPVCn0ERgcqTxSROrYpFW9yAgHM6aMJuQ5JQcuIT3+RyayIe+oTDnF8c+mbsb0IQXNMyhTSKJFFJQmGKS/iZKordRfqiBF1H9rFKeKaiKPU3ZTSITR1Txm+dzHVgoKDEjDPgZpkGjEFrqUxW89j277OdEcJaEnxZFi0uX6o+0aSs/kbyIh7zL0nCkoRebbxZEk1lXe/5JYHzKFPfnfk9ujVGEukDGmHLyh5/ZfzZhKQ7DmSa+bfN7ncmMO6JCVb2yV4yqv/FTd0OdQpHgYof9WiqrppPMexkNNyOaL9dJ7S9p8/Dgj8Sv96DKaRHpzyTemCYtU1Gf/0kpi/Png7iuVDGM55edYdoCjhKxQctGqnqeb9+heYe0hhFqWIXkcRA4GYoqsZZNVuUeBM3QJMwIpRZCUw5Z5GU1IG3Qd74JtzsZA7gGscv4BF1i9UCWUoui5LsSROaAcb4zrWJZpQpK0B+OFpcTunhdQz6fPZeRfgfeo2diTVPtWmhsuxhUqRkrgVGQXlfBWVTnZkdgrPopcnSEw7fOGICxDrWs/mPd/gAMnruGJIONR62gGl01L0evkZqz6/iIeRu2G+9XhWDexJ1oKn1FEq7fao1qOjbbvYewMbt6Y8DO+ORCLirxZ2U/wIKMz+r6rCUWpXdRatEJbnMKOTbvx553symNs9jaMZo+Arnpd1KMWTCxN0LMiHSqHWnne6EGwNO1R5bjUWrQE72ee+ugZqi4o5uHJg0xwnUanmvLRNOuJSb/+Dr8Nq7F8+fKy5jAJ5ro5+CtuIHYHK05kIl1+Pe4KffXpKt0HSqnP/pvpYfqhk/Cz0oaGlRH0W9e8iF1JC3Ts1hO6eIS0zJqyopUgLewo9nzwKYb1bCPIeCTIu5ODAR4LYa5V7t1fisyrZ/DbxwboqugJgFI00K2vNpCUBnlVxowpo2lR64KRbhvhaaaN3D+/wZKNf9bBxZ3RrMd0HLrmBysYwmpgd6kCrh/5SAz8Drv6T4GFwvSQ9VWyhMIb57HnVi7wgS7ekTqLydBcE+/27YzcU7dwr0r6mObQMJ2P7eIBiPjCFh8vzYSD2xh0qVdh+rbo1rsvtJGJ8NPX8bD8/irMQyZnl5TRTG8svlzTB5Fb58K8R3s0Ux8Iq8XrsOPIXXSZYAlDUV3UY1u83aGdEoWqiQ7qqmrsIuRlcueyNtTaoEPnDmhTfrooCzcCVsPu2wLM9VmNCfqKDClPc6h3/xjLdizAJ11lQsnqSn33n5mIy2ElGNSvG/iM2XUjC7nPa3BBose4+PslDJtkiu5VLkQziIysYVPFuGci6nQ4dN/XqV+oU2ounsvpMWZMGU2OmoYxFn69EMYQQdS6TsPAV5rG8eblK9tEIwz66Ne9nnGSHJQaik0+HeCxZCg0FWq5+irZUmSk3MNd/s/bx/DDJj7uV7YFIKmvA8SrTdFVfqbFO6rNXYwF2veRmKuSn6YS1NC6rQjSXsc8Roaqix9qb2Po6p9wxv8bzDPTA3Iv4+i2tXAcPwQfzd2P+IJXY9RHubdwaKUjPO6PxDYfBxhXzL4UoYY2+mMxb+qH0KrXwKU6qu+fkBcbgUOZfTDCsEujGx+6fx77fzPEv4e8q2QgIQMfh36sAEbdOypctagPzJgymh7KwZ1Lkbhr6w7v5UMa5u4vScCe8d3LvO66f40LhU2o8CgDkV7T4BicIgjqRuN48/KVbcKRaTgYhtUSUKgIpeP8jp1ImTMbo7SULaLWV8k2Q7sOmtDi/+xtBcfyJT/5ZvehgutegPuhIbhr7QCLmxvw+fqTSKvX5eRmx/m5Zcu7fTpDs1w7tm4HDUV1MgQoNxFX7rTBANsV2HE6vixP9Lnf4LPEGKk/r4brgduqhWI0Kq3QTkPVRCISFNwLwdqp63FzyDp8v2w4urYpV+mEktwMPCsQjoCbOcbuWwtn9xWYaT4fe+JVmP3WSh32L+U5kmOjcL8iWUohUk9vwwq3ZbBf+CuSimu7+O0h+j9lJqsI9yNO4IzVWJgqvcfLIRQlROJYTM/6D1C1RPg/ufuZGVNGE8P9YE5swtztOvh+83T0ETVwHFjv5y51gVMET2IQvGkxpnxxAflF9XvO2ygz09IUxP55C1qjBkCPP9aSBzi9yQ1uLp9jYUACioWPKYcPFfCHx4VRWDepV/XnkQ1Wss2g3tMQw3jdGJ2EFIWDm0I8uhGP9CphIITixENYd/pDbNy8CVu32+DxltVYc/gOakwmrpA83L4aifvQgIl5X7xTfku88Rbe1UzEX0lpCrdZGncYU7+7gIqV4BZvQd/UEg5ePvjRrg2OnLqOhtQbqR/t8Na73AjgL+5c1ngiSpEbuwfzF/8FQ/F2rLLQq1KQggr4MKSJWHs2nX/F3QP78E2qBTzXzcEnbwRj3dG4epxnWeqyfwF6glv/uwEM7oXu6oTc6AD8nG0B9/kj0PniEZy5o+zxQikyU+9x1+JtdNJQsmoiScKpXTdhaz1QhcF6Ie5ePodw7feg905dB6jZSL39COjRCfI1VpgxZTQhhIL4g1jlFIPP9qyARZd6zqzkadBzl9ooQHzAKiwUB+FBq854R5DWh8aYmVJaPP4XJcLgPl2hzlc+2XUI2TbLMX94Z1zcdQ53+IG/7Gy9WuNmju8vxokTi/F+u2aCrDvG70ng5hWNo2TVtEdg6RoriKJD8WdM9ahCyr6CXW7HIFtRj3KvYIdbFCxcxqJri7bQnbQCWydnY8cKMQKTlFfYKXqShTw5e03Z1/G7fzhgsADrp/UtW+7laf4mtAfqID89SyYlZVWK/f5AeJrcEbfojJ6934Z2t46o6k71ImiNTtrdgfwMZOUrmxeXIP3cFky12oZSUwMgJhSHA3bBW1haF7svhLWpGRweToD9kE7c55/j7oXHGDWhL3CFm3n/1gkTB3RDtQcufCzsvu+w7dfLcgMfeeq6f4Gc2/jf8ScwHd0LrS7sxY7Y97HI6h08OrUP3z3jDKW6soF2MTLTHiFXpyu6VCw7yCKkD7w3ChNUSR8ouYvzAeeBAbp4V75sY21IspF2Jws6+l2qP2sVvHoZjEZHkvYnrTEzJYfAJLlwgoYgodywNaTdBEnqqyENh9ClGYH3BEHd2LlzJ61atYpu3rwpTXrPV49RDu9yb8Edk2xi/1LKCl1JWrAin6gECt/pTfv4cANpiIEu6TkepZQaowuK6XGIKxnaB9JDhZ/Lo9gd7uSX+FyaXtEYRuQSmi68x1MWklA9HKQ6kpxo8rXrTyKzNRSUmFUREiHJiacg90X0dUVaSQkVp/2PfB1MSbtKWIiQNo5TSRqysaoC5aExEI0h91P3K/tTHmeqMBQkiyLFIwgjfSlOQchm2TY1aJDrH1XiCiVZ4eRpMpG2Rsn0TghlqX4uhLhQ6JCt/52q+68IjakeflO+veohFhIq4PpliOrhHWVI6HncXrIbZE0uvr9TWOgB+naeKYn4bck2vXnkFydfJEKomKQwhrV8v/z3Tckt7Ikgl6e++y/fvgb1H2NN9t6RlbHCtcIXrTAi0bwgShMkValb+kBJsj9NFkFhrGitSENvDGhe0ENBUAkzpoymoSie/O1NyMzzHGWo/KNRBc4A+FjJKIQCSgn1JtfVS8luQQAl1qfEljLqYUx5w1lFqShpycnJwjfKUWRMBWOA/jTG0qHOcbrSslHG01SouKFMyapuTKUUP6aoAA+yNTKkYTbTaMY0a7Kwdae9kY/Kvp8RRhstOaVYfh5EtpUGo+QW+U0Q4mz5pjuMbOy2UFhGmVEtN6a66w9TxF43spu3ijZuXEkzjE3JZvVuilBYik8YjIgcFMYnlnDGe5jdTjp18ntysV9CbhvFtNHNkWxnuJFvxMOyPpfEkf/CqWRhpCX0TY87Nhfy5/pdEudPC6dZkFF54gKREVlMc+Xee0px/q40zaLyWEVGFjRtoT/FFclvT4uMLKbSQv+4ylhYqcLWIzu52FwpkiTynzKZxJeeVr5X/IRijm4mBzM97ryNJAfxYYpKU3A+pNsVkYHbWbnY1jIkKb+T8yAtEunqkamy2OJ6719IvKBhSx77fqKNdqakZ7mBQpWWUJSh4BKJDbVppLJyidI4Y/63o1pVIGkMtciSO4YMQaIqNQ90mDFlND6Sp3RJbE0G8plv6gqfpHzizqo1TSXJFGjH/Wilie1LKCdqN319OJHyUv4g10GTy25y2eBwlZoOWfvFV/+hNnBmWjcUGFMh24qGjQft2/0NNxvoT5aep1Ss45tDMT5TyESVEl5KlWwdjWkTUmFM69qXrLPkZvCqJbovG9w0bqL78lUOC2lZw9LEfeSw47pig8nd9/Ma+5pX+d1yrwsiyEOn/HpmUZSvb8XAqSq1JbqXUF7E19THxJuipAUxmhKW6J7xQilEyu//wfRf++Dbr/8N/YpAtDrCPx/0/S/Ojx0KA9lAeFWeu/BOSofv8ANFFdsdHJ6u99I5ENDjGzgX3gJDP7XF5Bmu2OH9b1z70gP7+Lyh+dHYte18mfOMwmem6ujjuB/hXwxGuypyvpU/M5V+GdmRp7An9WM4ThwAUdJ+OO6MwT8mtcYbhrCarYWAQ/9DKj90eiXQhJGVNQwCjuF8au0uZqqRj4QrEUgdOQGj+5bi6qFrGGTWQ0FYCKEwORmtDLSqP1NtCPnJiArNxUjLQehW8XPWxUc9OqF5xiUEXNWEbgdFz0Ozcf3kMdy0GY/hCuOjM3D52EloTTcTCmI0IdIE+g9gM/1juTjWMpgxZTQipciN9oXj56lw2rUcI2uMd6sB3sP0JxfYLC6GjamOzE3K/dATruB4KpB31BNukX3gMLkfRGqt0WP6HuTEb8a4Wt3iXxUI+bevITT3Q1h+IBs39y/06NISGRd+x9V3upY5YMkPHiQPEeI8GvaByeDG6jKDhvImO3hQVcm+qrTHgFnL8XnEbhy81hjhIC8CNbQdMB0en8di00GZjE4Noh16W86BXfE57Fm7Hge17TBFX1FNlWzEnH+Gj/q9JbxuHOhpCm7nmmD8B9pl912rXrB2+wQJ/pux4svT6LfoE3RRYAspJRTfbdaE14rRCt9HdjSCf3wLk8xrzpzVcIqREvITNndajBWfait2cuJ+XAxGIyChYulSqyk5BSXXc4mogNKiDpPYzqTsWVM1x5EGPHepD3/rMm8hJfvPIo2ROyhGWL4qc/IZSTZfLCMHZ38lS+hlidpN5PPEKkVwKDGzo9Vuy2iF/02ZggQvyzKvhAoiPEmHuyfq5TTC9V7qiDV5L91uzGfqTYzk8R/kbDiL/G43blVO5XC/4eRfacHKU/T0ZThNkjQKc7cgS59omXtSFu66Bi0hrSauS8xT5n8whXxi5J26KlHj/xHsKoNRfwpuYI/jNMxPHoFvF3wADVVXXIqf4cH9FDzkZmEXT/2BM4nlswddTPY/iX223StHgXQPh2Z9gomZbkg6PB3diy7A08AYPy2IxK3leojddRDZ42diSPsk7Jk4EjOOSPPyqIAOrP1OIFB+qTf1EGZ2cQEC/8TuCdqCsKm4j0MzHZG02B/LjbihRH0puIYdtu7IWLkLKz/UrD1MoEZKuFOwEEOS5nLn16hxl/1UJfMcvJZuxe/njgv3hh6G2ZjDYtF6LB/SsewzqsA/NvivMza0csL3nxtWiYl8eeFXenZi+oZWWP+9fcNjtGtDchdH3I5C02k+hnb8uzOVFSL1uCfsQgbjJ7GF4jST9BDB8z/D/mF7q+qJxqbkPo6vdkWI6X8gHtdV+e9AalIZjAaRRVFbrau7xzekiWaRf3LVIkqUG0Zu2lqVXn1SJ4aymSNfFWLlwsNKQkDqyd/tgFRnnnOzWQcydG2smcXL44DUKBQ/pHDvZbQy5CE3B3tVKKCU8O3kuDKkEZ2RXnYk9PyWHzmvDKQ4uRCpKjwNoSV9XCgkrSnvzhy65buKVgberDWUh81MGa8MdC8AU98/gKHndsOhdztOwD9bdcXckP/DkDfbor/zSkzSb8CsTkopNxnahqU/XkXJ01gc+f0yYGQB695vokXfOdi8fAhqyFDXABo+M6WMUKw0P4j+R7fAtqts1Yz68hLMTBmMVwRmTBmMlwLemJphol8iIFqEoKQtFWXYVIVyE/BXnAgfGjXcE7M02huD3l+Cq9zf3MyUGVMGoxaYMWUwGAwGo4Gw0BgGg8FgMBoIM6YMBoPBYDQQZkwZDAaDwWggzJgyGAwGg9FAmDFlMBgMBqOBMGPKYDAYDEYDYcaUwWAwGIwGAfw/H922qQnGSDIAAAAASUVORK5CYII="}}},{"cell_type":"code","source":"# Compute the contrastive loss for a mini-batch\n# projections_1 is (batch_size,proj_dim)\n# projections_2 is (batch_size,proj_dim)\n# temperature > 0\n\ndef contrastive_loss(projections_1, projections_2, temperature=1):\n    \n    projections_1 = ops.normalize(projections_1, axis=1)\n    projections_2 = ops.normalize(projections_2, axis=1)\n    projections = ops.concatenate([projections_1, projections_2], axis=0) \n    similarities = (\n        ops.matmul(projections, ops.transpose(projections)) / temperature\n    )\n    \n    # each element on the diagonal represents the self-similarity score of its feature vector \n    # we need to ignore such elements\n    # to do this, we will replace the elements on the diagonal by -inf \n    # so they will contribute to nothing, when applying a softmax\n    batch_size=ops.shape(projections_1)[0]\n    diag=ops.ones(2*batch_size)*-math.inf\n    similarities=tf.linalg.set_diag(similarities,diag)\n    \n    # we need to generate the contrastive labels\n    # each row of the similarities matrix has 2*batch_size elements\n    # the element representing the similarity for a positive pair is the:\n    # (batch_size+1)th element on the 1st row\n    # (batch_size+2)th element on the 2nd row\n    # ...\n    # (batch_size*2)th element on the (batch_size)th row\n    # 1st element on the (batch_size+1)th row\n    # ...\n    # (batch_size)th element on the last row\n    # the contrastive_labels is a matrix of the same shape as similarities\n    # with ones at the position corresponind to positive pairs in the similarities matrix\n    # and zeros elsewhere\n    contrastive_labels_1 = ops.arange(batch_size, 2*batch_size)\n    contrastive_labels_2 = ops.arange(0, batch_size)\n    contrastive_labels = ops.concatenate([contrastive_labels_1,contrastive_labels_2])\n    \n    # The trick here is that since each image is similar to exactly 1 other image in the input batch (positive pair) \n    # (if we ignore the similarity of a feature vector with itself), \n    # we can consider this to be a classification-like setting \n    # where the probability distribution of the similarity probability between images represents \n    # a classification task where one of them will be close to 1.0 and the rest will be close to 0.0.\n    pairs_loss = keras.losses.sparse_categorical_crossentropy(\n        contrastive_labels, similarities, from_logits=True\n    )\n    \n    # The final loss is computed across all positive pairs, both (i; j) and (j; i), in a mini-batch.   \n    pairs_loss_split = ops.split(pairs_loss, 2)\n     \n    return (pairs_loss_split[0]+pairs_loss_split[1])/2","metadata":{"execution":{"iopub.status.busy":"2024-10-05T13:53:29.388181Z","iopub.execute_input":"2024-10-05T13:53:29.388575Z","iopub.status.idle":"2024-10-05T13:53:29.398884Z","shell.execute_reply.started":"2024-10-05T13:53:29.388537Z","shell.execute_reply":"2024-10-05T13:53:29.397778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we have defined a function to calculate the `contrastive loss`, let's test it. For the sake of simplicity, we are going to run the test simulating a batch of 3 images and a latent space of 2-dimensions.","metadata":{}},{"cell_type":"code","source":"test_config = {\n    '1': {\n        'description': 'Identical representations for positive pairs',\n        'z1': [[0.1,0.2], [0.8,0.8], [-0.2,0.1]],\n        'z2': [[0.1,0.2], [0.8,0.8], [-0.2,0.1]],\n        'temperature': [0.1, 1, 10],\n    },\n    '2': {\n        'description': 'Close representations for positive pairs',\n        'z1': [[0.1,0.2], [0.8,0.8], [-0.2,0.1]],\n        'z2': [[0.12,0.22], [0.82, 0.78], [-0.18,0.12]],\n        'temperature': [0.1, 1, 10],\n    },\n    '3': {\n        'description': 'Distant representations for positive pairs',\n        'z1': [[0.1,0.2], [0.8,0.8], [-0.2,0.1]],\n        'z2': [[0.1,1.9], [-3.8,2.1], [1.2,1.1]],\n        'temperature': [0.1, 1, 10],\n    },\n    '4': {\n        'description': 'Identical representations for non-positive pairs',\n        'z1': [[0.1,0.2], [0.8,0.8], [-0.2,0.1]],\n        'z2': [[-0.2,0.1], [0.1,0.2], [0.8,0.8]],\n        'temperature': [0.1, 1, 10],\n    },\n}","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:02:47.649527Z","iopub.execute_input":"2024-10-05T16:02:47.650188Z","iopub.status.idle":"2024-10-05T16:02:47.659389Z","shell.execute_reply.started":"2024-10-05T16:02:47.650146Z","shell.execute_reply":"2024-10-05T16:02:47.658373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def test_contrastive_loss(config):\n    html = ''\n    th = '<th style=\"width:50%\">Test description</th>'\n    for i, config in enumerate(config.items()):\n        desc = config[1]['description']\n        td = f'<td>{desc}</td>'\n        for t in config[1]['temperature']:\n            if i == 0:\n                th = th + f'<th>Loss with temp = {t}</th>'\n            projections_1 = ops.convert_to_tensor(config[1]['z1'])\n            projections_2 = ops.convert_to_tensor(config[1]['z2'])\n            td = td + f'<td>{np.round(contrastive_loss(projections_1, projections_2, t), 1)}</td>'\n        tr = f'<tr>{td}</tr>'\n        html = html + tr\n    trh = f'<tr>{th}</tr>'\n    html = f'<table>{trh}{html}</table>'\n    display(HTML(html))\n    \ntest_contrastive_loss(test_config)","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:02:49.257279Z","iopub.execute_input":"2024-10-05T16:02:49.257696Z","iopub.status.idle":"2024-10-05T16:02:49.331361Z","shell.execute_reply.started":"2024-10-05T16:02:49.257654Z","shell.execute_reply":"2024-10-05T16:02:49.330459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.7 Create contrastive accuracy metric","metadata":{}},{"cell_type":"markdown","source":"To follow the progress of the learning of representations during the unsupervised pretraining, we need to create a custom metric. We'll use `contrastive accuracy` which is the ratio of cases in which the representation of an image is more similar to its differently augmented version's one, than to the representation of any other image in the current batch. This metric can be used for hyperparameter tuning even in the case when there are no labeled examples.","metadata":{}},{"cell_type":"code","source":"# Compute the contrastive accuracy for a mini-batch\n# projections_1 is (batch_size,proj_dim)\n# projections_2 is (batch_size,proj_dim)\n\ndef contrastive_accuracy(projections_1, projections_2):\n    \n    projections_1 = ops.normalize(projections_1, axis=1)\n    projections_2 = ops.normalize(projections_2, axis=1)\n    projections = ops.concatenate([projections_1, projections_2], axis=0) \n    similarities = (\n        ops.matmul(projections, ops.transpose(projections))\n    )\n    \n    # each element on the diagonal represents the self-similarity score of its feature vector \n    # we need to ignore such elements\n    # to do this, we will replace the elements on the diagonal by +inf \n    batch_size=ops.shape(projections_1)[0]\n    diag=ops.ones(2*batch_size)*-math.inf\n    similarities=tf.linalg.set_diag(similarities,diag)    \n    \n    # we need to generate the contrastive labels\n    contrastive_labels_1 = ops.arange(batch_size, 2*batch_size)\n    contrastive_labels_2 = ops.arange(0, batch_size)\n    contrastive_labels = ops.concatenate([contrastive_labels_1,contrastive_labels_2])\n    \n    # Find the index (per row) with the highest similarity \n    indexes = ops.argmax(similarities, axis=1)\n    \n    # Calculate the ratio of cases in which the representation of an image is more similar to its augmented version, \n    # than to the representation of any other image in the current batch\n    diff = indexes-contrastive_labels\n    acc = ops.shape(ops.where(diff==0))[1]/(2*batch_size)\n    \n    return acc","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:03:25.638648Z","iopub.execute_input":"2024-10-05T16:03:25.639154Z","iopub.status.idle":"2024-10-05T16:03:25.647599Z","shell.execute_reply.started":"2024-10-05T16:03:25.639113Z","shell.execute_reply":"2024-10-05T16:03:25.646670Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def test_contrastive_accuracy(config):\n    trh = '<tr><th>Test description</th><th>Accuracy</th></tr>'\n    tr = ''\n    for i, config in enumerate(config.items()):\n        desc = config[1]['description']\n        td = f'<td>{desc}</td>'\n        projections_1 = ops.convert_to_tensor(config[1]['z1'])\n        projections_2 = ops.convert_to_tensor(config[1]['z2'])\n        td = td + f'<td>{np.round(contrastive_accuracy(projections_1, projections_2), 4)}</td>'\n        tr = tr + f'<tr>{td}</tr>'\n    html = f'<table>{trh}{tr}</table>'\n    display(HTML(html))\n    \ntest_contrastive_accuracy(test_config)","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:03:27.068960Z","iopub.execute_input":"2024-10-05T16:03:27.069635Z","iopub.status.idle":"2024-10-05T16:03:27.104765Z","shell.execute_reply.started":"2024-10-05T16:03:27.069590Z","shell.execute_reply":"2024-10-05T16:03:27.103791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Baseline","metadata":{}},{"cell_type":"markdown","source":"### 3.1 Build the baseline model","metadata":{}},{"cell_type":"code","source":"inputs = keras.Input(shape=(28,28,3))\nencoder = get_encoder(inputs)\nrepresentations = encoder(inputs)\n\n#mlp = get_mlp(representations, \"softmax\")\nmlp = get_mlp(representations)\npredictions = mlp(representations)\n\nbaseline = keras.Model(inputs, predictions, name='baseline')\nbaseline.summary()","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:03:46.727409Z","iopub.execute_input":"2024-10-05T16:03:46.727797Z","iopub.status.idle":"2024-10-05T16:03:46.821023Z","shell.execute_reply.started":"2024-10-05T16:03:46.727759Z","shell.execute_reply":"2024-10-05T16:03:46.820151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.2 Train the baseline model with 100% of the training set!!","metadata":{}},{"cell_type":"code","source":"sparse_categorical_crossentropy = keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n\nbaseline.compile(\n    optimizer=tf.keras.optimizers.Adam(),\n    loss=sparse_categorical_crossentropy,\n    metrics=[\"accuracy\"],\n    )\n\nearly_stopping = tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\",patience=10)\n\nhistory = baseline.fit(x=labeled_train_baseline, validation_data=labeled_val_baseline, epochs=100, callbacks=[early_stopping])","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:04:23.707654Z","iopub.execute_input":"2024-10-05T16:04:23.708068Z","iopub.status.idle":"2024-10-05T16:05:12.948575Z","shell.execute_reply.started":"2024-10-05T16:04:23.708028Z","shell.execute_reply":"2024-10-05T16:05:12.947596Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.3 Evaluate the performance of the baseline model trained with 100% of the training set!!","metadata":{}},{"cell_type":"code","source":"# item : \"loss\" or \"accuracy\"\ndef plot_history(item):\n    plt.plot(history.history[item], label=item)\n    plt.plot(history.history[\"val_\" + item], label=\"val_\" + item)\n    plt.xlabel(\"Epochs\")\n    plt.ylabel(item)\n    plt.title(\"Train and Validation {} Over Epochs\".format(item), fontsize=14)\n    plt.legend()\n    plt.grid()\n    plt.show()\n\nplot_history(\"loss\")\nplot_history(\"accuracy\")","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:08:17.673002Z","iopub.execute_input":"2024-10-05T16:08:17.673414Z","iopub.status.idle":"2024-10-05T16:08:18.285670Z","shell.execute_reply.started":"2024-10-05T16:08:17.673376Z","shell.execute_reply":"2024-10-05T16:08:18.284680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Self-supervised learning with SimCLR","metadata":{}},{"cell_type":"code","source":"inputs = keras.Input(shape=(28,28,3), name=\"input images\")\n\ndata_augmenter = get_data_augmenter(inputs)\nx1 = data_augmenter(inputs)\nx2 = data_augmenter(inputs)\n\nencoder = get_encoder(inputs)\nh1 = encoder(x1)\nh2 = encoder(x2)\n\ninputs_mlp = keras.Input(shape=(10,), name=\"representations\")\nmlp = get_mlp(inputs_mlp)\nz1 = mlp(h1)\nz2 = mlp(h2)\n\nz = keras.layers.Lambda(lambda x: ops.concatenate([x[0], x[1]], axis=1), output_shape=(20,))([z1,z2])\n\nsimCLR = keras.Model(inputs, z, name='simCLR')\nsimCLR.summary(line_length=100)","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:08:28.807211Z","iopub.execute_input":"2024-10-05T16:08:28.807593Z","iopub.status.idle":"2024-10-05T16:08:28.971048Z","shell.execute_reply.started":"2024-10-05T16:08:28.807554Z","shell.execute_reply":"2024-10-05T16:08:28.970159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def c_loss(y_true, y_pred):\n    projections_1, projections_2 = ops.split(y_pred, 2, axis=-1)\n    return contrastive_loss(projections_1, projections_2, 0.1)","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:08:32.846479Z","iopub.execute_input":"2024-10-05T16:08:32.847321Z","iopub.status.idle":"2024-10-05T16:08:32.852249Z","shell.execute_reply.started":"2024-10-05T16:08:32.847254Z","shell.execute_reply":"2024-10-05T16:08:32.851127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def c_accuracy(y_true, y_pred):\n    projections_1, projections_2 = ops.split(y_pred, 2, axis=-1)\n    return contrastive_accuracy(projections_1, projections_2)","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:08:33.654803Z","iopub.execute_input":"2024-10-05T16:08:33.655689Z","iopub.status.idle":"2024-10-05T16:08:33.660271Z","shell.execute_reply.started":"2024-10-05T16:08:33.655649Z","shell.execute_reply":"2024-10-05T16:08:33.659202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"simCLR.compile(loss=c_loss, optimizer='Adam', metrics=[c_accuracy])\nhistory = simCLR.fit(x=unlabeled_train, epochs=100)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-10-05T16:09:08.954998Z","iopub.execute_input":"2024-10-05T16:09:08.955939Z","iopub.status.idle":"2024-10-05T16:15:48.307964Z","shell.execute_reply.started":"2024-10-05T16:09:08.955880Z","shell.execute_reply":"2024-10-05T16:15:48.306967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history.history[\"loss\"], label=\"c_loss\")\nplt.plot(history.history[\"c_accuracy\"], label=\"c_accuracy\")\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"c_loss and c_accuracy\")\nplt.title(\"Contrastive Loss and Accuracy Over Epochs\", fontsize=14)\nplt.legend()\nplt.grid()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:18:17.828617Z","iopub.execute_input":"2024-10-05T16:18:17.829495Z","iopub.status.idle":"2024-10-05T16:18:18.137105Z","shell.execute_reply.started":"2024-10-05T16:18:17.829453Z","shell.execute_reply":"2024-10-05T16:18:18.136198Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Supervised fine-tuning with only 2% of the training data!!","metadata":{}},{"cell_type":"code","source":"inputs = keras.Input(shape=(28,28,3))\nrepresentations = encoder(inputs)\npredictions = layers.Dense(10)(representations)\nbaseline = keras.Model(inputs, predictions, name='baseline')\nbaseline.summary()","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:19:10.338817Z","iopub.execute_input":"2024-10-05T16:19:10.339719Z","iopub.status.idle":"2024-10-05T16:19:10.374735Z","shell.execute_reply.started":"2024-10-05T16:19:10.339678Z","shell.execute_reply":"2024-10-05T16:19:10.373800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sparse_categorical_crossentropy = keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n\nbaseline.compile(\n    optimizer=tf.keras.optimizers.Adam(),\n    loss=sparse_categorical_crossentropy,\n    metrics=[\"accuracy\"],\n    )\n\nearly_stopping = tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\",patience=5)\n\nhistory = baseline.fit(x=labeled_train_simclr, validation_data=labeled_val_simclr, epochs=100, callbacks=[early_stopping])","metadata":{"execution":{"iopub.status.busy":"2024-10-05T16:20:08.293020Z","iopub.execute_input":"2024-10-05T16:20:08.293769Z","iopub.status.idle":"2024-10-05T16:20:15.981010Z","shell.execute_reply.started":"2024-10-05T16:20:08.293726Z","shell.execute_reply":"2024-10-05T16:20:15.980192Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# item : \"loss\" or \"accuracy\"\ndef plot_history(item):\n    plt.plot(history.history[item], label=item)\n    plt.plot(history.history[\"val_\" + item], label=\"val_\" + item)\n    plt.xlabel(\"Epochs\")\n    plt.ylabel(item)\n    plt.title(\"Train and Validation {} Over Epochs\".format(item), fontsize=14)\n    plt.legend()\n    plt.grid()\n    plt.show()\n\nplot_history(\"loss\")\nplot_history(\"accuracy\")","metadata":{"execution":{"iopub.status.busy":"2024-10-05T14:10:36.026031Z","iopub.execute_input":"2024-10-05T14:10:36.027167Z","iopub.status.idle":"2024-10-05T14:10:36.578823Z","shell.execute_reply.started":"2024-10-05T14:10:36.027123Z","shell.execute_reply":"2024-10-05T14:10:36.577876Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]}]}