{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**由于本地没有可用的gpu资源以及kaggle kernels gpu的时长限制，故借本次比赛的机会学习使用TPU进行训练与推理**\n\n该代码主要依据Araik Tamazian的notebook [FC Ensemble External Data (EffNet+DenseNet)](https://www.kaggle.com/atamazian/fc-ensemble-external-data-effnet-densenet) 进行改写\n\n同时也参考了以下内容：\n* [simple resnet baseline](https://www.kaggle.com/nekokiku/simple-resnet-baseline) by Neko Kiku\n* [del_duplicate_image](https://github.com/seefun/TorchUtils/blob/master/torch_utils/dataset/del_duplicate_image.py) by seefun\n* [Getting started with 100+ flowers on TPU](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu) by Martine Goerner\n* [TPU Flowers](https://www.kaggle.com/tusharkendre/tpu-flowers) by Tushar Kendre and Shreyaansh Gupta (random_blockout augmentation)\n* [TPU: ENet B7 + DenseNet](https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet) by Wojtek Rosa\n\n对以上作者的无私分享表示由衷的感谢！\n\n","metadata":{}},{"cell_type":"markdown","source":"本notebook可在kaggle kernels的TPU环境下实现一键跑通，并在private test上获得0.988左右的成绩。如果大家再选择不同的模型组合，训练策略以及数据增强方法进行训练与推理，最终对多个结果采取投票的方法进行融合，成绩可达0.989-0.990+。关于投票的代码在其他小伙伴分享的notebook中已有详细的实现，这里就不再赘述了。\n\n在本次比赛中并未使用太多的数据增强方法，也还没来得及尝试沐神的ResNeSt，所以还是有很大的提升空间的，欢迎大家进行魔改，也希望大家多多分享改进方案。\n\n**<span style=\"color:blue\">第一次分享notebook，大家有什么意见尽管提；如果对大家有帮助，也请各位不要吝啬vote呀，哈哈哈</span>**","metadata":{}},{"cell_type":"code","source":"!pip install -q efficientnet","metadata":{"papermill":{"duration":9.650132,"end_time":"2021-06-16T10:08:28.613337","exception":false,"start_time":"2021-06-16T10:08:18.963205","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T01:53:26.720299Z","iopub.execute_input":"2021-07-01T01:53:26.72072Z","iopub.status.idle":"2021-07-01T01:53:35.417475Z","shell.execute_reply.started":"2021-07-01T01:53:26.720621Z","shell.execute_reply":"2021-07-01T01:53:35.416229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math, re, os, random\nimport tensorflow as tf\nimport tensorflow_addons as tfa\nimport efficientnet.tfkeras as efn\nimport numpy as np\nimport pandas as pd\nimport gc\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\n\nprint(\"TF version \" + tf.__version__)","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":7.387615,"end_time":"2021-06-16T10:08:36.032484","exception":false,"start_time":"2021-06-16T10:08:28.644869","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T01:53:35.419662Z","iopub.execute_input":"2021-07-01T01:53:35.420133Z","iopub.status.idle":"2021-07-01T01:53:42.606364Z","shell.execute_reply.started":"2021-07-01T01:53:35.420074Z","shell.execute_reply":"2021-07-01T01:53:42.605349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TPU detection","metadata":{"papermill":{"duration":0.03119,"end_time":"2021-06-16T10:08:36.095963","exception":false,"start_time":"2021-06-16T10:08:36.064773","status":"completed"},"tags":[]}},{"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\n# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect() # TPU detection\n    strategy = tf.distribute.TPUStrategy(tpu)\nexcept ValueError:\n    strategy = tf.distribute.MirroredStrategy() # for GPU or multi-GPU machines\n\nprint(\"Number of accelerators: \", strategy.num_replicas_in_sync)","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","papermill":{"duration":5.875849,"end_time":"2021-06-16T10:08:42.003107","exception":false,"start_time":"2021-06-16T10:08:36.127258","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T01:53:42.608087Z","iopub.execute_input":"2021-07-01T01:53:42.608394Z","iopub.status.idle":"2021-07-01T01:53:48.835095Z","shell.execute_reply.started":"2021-07-01T01:53:42.608364Z","shell.execute_reply":"2021-07-01T01:53:48.833901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration","metadata":{"papermill":{"duration":0.031422,"end_time":"2021-06-16T10:08:42.066626","exception":false,"start_time":"2021-06-16T10:08:42.035204","status":"completed"},"tags":[]}},{"cell_type":"code","source":"IMAGE_SIZE = [224, 224]\nEPOCHS = 60\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync","metadata":{"papermill":{"duration":0.038715,"end_time":"2021-06-16T10:08:42.136782","exception":false,"start_time":"2021-06-16T10:08:42.098067","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T01:53:48.837033Z","iopub.execute_input":"2021-07-01T01:53:48.837453Z","iopub.status.idle":"2021-07-01T01:53:48.842358Z","shell.execute_reply.started":"2021-07-01T01:53:48.83741Z","shell.execute_reply":"2021-07-01T01:53:48.841052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data access and classes","metadata":{"papermill":{"duration":0.037933,"end_time":"2021-06-16T10:08:42.207575","exception":false,"start_time":"2021-06-16T10:08:42.169642","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"TPUs read data directly from Google Cloud Storage (GCS), so we need to copy the dataset to a GCS bucket co-located with the TPU. To do that, pass the name of a specific dataset to the get_gcs_path function. The name of the dataset is the name of the directory it is mounted in. \n\n这里特别说明一下，Google要求所有在Kaggle平台上使用TPU进行模型训练的用户须将数据放在GCS上，故该notebook的数据集已公开在kaggle datasets上。考虑到测试集与训练集可能包含相同的标注噪声，该数据集选用未经清洗的原始数据集。","metadata":{"papermill":{"duration":0.039909,"end_time":"2021-06-16T10:08:42.294457","exception":false,"start_time":"2021-06-16T10:08:42.254548","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train = pd.read_csv('../input/classify-leaves/train.csv')\nCLASSES = sorted(list(set(train['label'])))\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('leavestfrec')\n\nTraining_filenames, Validation_filenames = [], []\n\nfor i in range(5):\n    Training_filenames.append(tf.io.gfile.glob(GCS_DS_PATH + f'/train{i}.tfrec'))\n    Validation_filenames.append(tf.io.gfile.glob(GCS_DS_PATH + f'/val{i}.tfrec'))\n\n\nTEST_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/test.tfrec')\ntrain['label'].value_counts()","metadata":{"papermill":{"duration":9.605614,"end_time":"2021-06-16T10:08:51.932622","exception":false,"start_time":"2021-06-16T10:08:42.327008","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T01:53:48.843898Z","iopub.execute_input":"2021-07-01T01:53:48.844417Z","iopub.status.idle":"2021-07-01T01:53:49.966658Z","shell.execute_reply.started":"2021-07-01T01:53:48.844365Z","shell.execute_reply":"2021-07-01T01:53:49.965739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Random blockout augmentation","metadata":{"papermill":{"duration":0.032036,"end_time":"2021-06-16T10:08:52.22592","exception":false,"start_time":"2021-06-16T10:08:52.193884","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def random_erasing(img, sl=0.1, sh=0.2, rl=0.4, p=0.3):\n    h = tf.shape(img)[0]\n    w = tf.shape(img)[1]\n    c = tf.shape(img)[2]\n    origin_area = tf.cast(h*w, tf.float32)\n\n    e_size_l = tf.cast(tf.round(tf.sqrt(origin_area * sl * rl)), tf.int32)\n    e_size_h = tf.cast(tf.round(tf.sqrt(origin_area * sh / rl)), tf.int32)\n\n    e_height_h = tf.minimum(e_size_h, h)\n    e_width_h = tf.minimum(e_size_h, w)\n\n    erase_height = tf.random.uniform(shape=[], minval=e_size_l, maxval=e_height_h, dtype=tf.int32)\n    erase_width = tf.random.uniform(shape=[], minval=e_size_l, maxval=e_width_h, dtype=tf.int32)\n\n    erase_height = abs(erase_height-10)\n    erase_width = abs(erase_width-10)\n    \n    erase_area = tf.zeros(shape=[erase_height, erase_width, c])\n    erase_area = tf.cast(erase_area, tf.uint8)\n\n    pad_h = h - erase_height\n    pad_top = tf.random.uniform(shape=[], minval=0, maxval=pad_h, dtype=tf.int32)\n    pad_bottom = pad_h - pad_top\n\n    pad_w = w - erase_width\n    pad_left = tf.random.uniform(shape=[], minval=0, maxval=pad_w, dtype=tf.int32)\n    pad_right = pad_w - pad_left\n\n    erase_mask = tf.pad([erase_area], [[0,0],[pad_top, pad_bottom], [pad_left, pad_right], [0,0]], constant_values=1)\n    erase_mask = tf.squeeze(erase_mask, axis=0)\n    erased_img = tf.multiply(tf.cast(img,tf.float32), tf.cast(erase_mask, tf.float32))\n\n    return tf.cond(tf.random.uniform([], 0, 1) > p, lambda: tf.cast(img, img.dtype), lambda:  tf.cast(erased_img, img.dtype))","metadata":{"papermill":{"duration":0.049666,"end_time":"2021-06-16T10:08:52.307666","exception":false,"start_time":"2021-06-16T10:08:52.258","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T01:53:49.96815Z","iopub.execute_input":"2021-07-01T01:53:49.968537Z","iopub.status.idle":"2021-07-01T01:53:49.98723Z","shell.execute_reply.started":"2021-07-01T01:53:49.968496Z","shell.execute_reply":"2021-07-01T01:53:49.985915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset functions","metadata":{"papermill":{"duration":0.031955,"end_time":"2021-06-16T10:08:52.372043","exception":false,"start_time":"2021-06-16T10:08:52.340088","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef onehot(image,label):\n    return image,tf.one_hot(label, len(CLASSES))\n\ndef read_labeled_tfrecord(example):\n    feature  = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"label\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, feature)\n    image = decode_image(example['image'])\n    label = tf.cast(example['label'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    feature  = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict leaves classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, feature)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_brightness(image, 0.5)\n    image = random_erasing(image)\n    return image, label\n\ndef data_hflip(image, idnum):\n    image = tf.image.flip_left_right(image)\n    return image, idnum\n\ndef get_training_dataset(training_filenames, do_onehot=False):\n    dataset = load_dataset(training_filenames, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    if do_onehot:\n        dataset = dataset.map(onehot, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(validation_filenames, ordered=False, do_onehot=False):\n    dataset = load_dataset(validation_filenames, labeled=True, ordered=ordered)\n    if do_onehot:\n        dataset = dataset.map(onehot, num_parallel_calls=AUTO)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False, augmented=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.map(data_hflip, num_parallel_calls=AUTO)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\nNUM_TRAINING_IMAGES = 14683\nNUM_VALIDATION_IMAGES = 3670\nNUM_TEST_IMAGES = 8800\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nVALIDATION_STEPS = -(-NUM_VALIDATION_IMAGES // BATCH_SIZE) # The \"-(-//)\" trick rounds up instead of down :-)\nTEST_STEPS = -(-NUM_TEST_IMAGES // BATCH_SIZE)             # The \"-(-//)\" trick rounds up instead of down :-)\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","metadata":{"papermill":{"duration":0.059013,"end_time":"2021-06-16T10:08:52.465952","exception":false,"start_time":"2021-06-16T10:08:52.406939","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T01:53:49.989211Z","iopub.execute_input":"2021-07-01T01:53:49.989743Z","iopub.status.idle":"2021-07-01T01:53:50.013889Z","shell.execute_reply.started":"2021-07-01T01:53:49.989697Z","shell.execute_reply":"2021-07-01T01:53:50.012702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Models and training","metadata":{"papermill":{"duration":0.066578,"end_time":"2021-06-16T10:09:01.289497","exception":false,"start_time":"2021-06-16T10:09:01.222919","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Custom LR scheduler","metadata":{"papermill":{"duration":0.065247,"end_time":"2021-06-16T10:09:01.420345","exception":false,"start_time":"2021-06-16T10:09:01.355098","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Learning rate schedule for TPU, GPU and CPU.\n# Using an LR ramp up because fine-tuning a pre-trained model.\n# Starting with a high LR would break the pre-trained weights.\n\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 9\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .87\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=1)\n\nrng = [i for i in range(25 if EPOCHS<25 else EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","metadata":{"papermill":{"duration":0.228886,"end_time":"2021-06-16T10:09:01.714915","exception":false,"start_time":"2021-06-16T10:09:01.486029","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T01:53:50.016804Z","iopub.execute_input":"2021-07-01T01:53:50.017419Z","iopub.status.idle":"2021-07-01T01:53:50.227241Z","shell.execute_reply.started":"2021-07-01T01:53:50.017371Z","shell.execute_reply":"2021-07-01T01:53:50.226301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_model(base_model):\n    base_model.trainable = True\n    model = tf.keras.Sequential([\n        base_model,\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    model.compile(\n        optimizer='nadam',\n        loss = 'categorical_crossentropy',\n#         metrics=[tfa.metrics.F1Score(len(CLASSES), average='macro')],\n        metrics=['accuracy'],\n        # NEW on TPU in TensorFlow 24: sending multiple batches to the TPU at once saves communications\n        # overheads and allows the XLA compiler to unroll the loop on TPU and optimize hardware utilization.\n        steps_per_execution=16\n    )\n    return model","metadata":{"papermill":{"duration":0.076748,"end_time":"2021-06-16T10:09:01.858817","exception":false,"start_time":"2021-06-16T10:09:01.782069","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T01:53:50.228646Z","iopub.execute_input":"2021-07-01T01:53:50.228932Z","iopub.status.idle":"2021-07-01T01:53:50.234422Z","shell.execute_reply.started":"2021-07-01T01:53:50.228906Z","shell.execute_reply":"2021-07-01T01:53:50.233683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## EfficientNet B7","metadata":{"papermill":{"duration":0.065911,"end_time":"2021-06-16T10:09:01.991401","exception":false,"start_time":"2021-06-16T10:09:01.92549","status":"completed"},"tags":[]}},{"cell_type":"code","source":"for i in range(len(Training_filenames)):\n    \n    print(f'Start training the fold{i} !')\n    with strategy.scope():\n        enet = efn.EfficientNetB7(weights='noisy-student', \n                                include_top=False,\n                                pooling='avg',\n                                input_shape=(*IMAGE_SIZE, 3))\n        model1 = get_model(enet)\n    \n\n    chk_callback1 = tf.keras.callbacks.ModelCheckpoint(f'effnetb7_{i}_best.h5',\n                                                       save_weights_only=True,\n#                                                        monitor='val_f1_score',\n                                                       monitor='val_accuracy',\n                                                       mode='max',\n                                                       save_best_only=True,\n                                                       verbose=1)\n    \n    history1 = model1.fit(get_training_dataset(Training_filenames[i], do_onehot=True), \n                      steps_per_epoch=STEPS_PER_EPOCH, \n                      epochs=EPOCHS, \n                      validation_data=get_validation_dataset(Validation_filenames[i], do_onehot=True),\n                      validation_steps=VALIDATION_STEPS,\n                      callbacks=[lr_callback, chk_callback1],\n                      verbose=2)\n\n    del chk_callback1, history1, model1\n    gc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-07-01T01:53:50.235406Z","iopub.execute_input":"2021-07-01T01:53:50.235791Z","iopub.status.idle":"2021-07-01T02:58:16.467271Z","shell.execute_reply.started":"2021-07-01T01:53:50.235763Z","shell.execute_reply":"2021-07-01T02:58:16.455363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## DenseNet201","metadata":{"papermill":{"duration":0.147394,"end_time":"2021-06-16T10:35:31.461037","exception":false,"start_time":"2021-06-16T10:35:31.313643","status":"completed"},"tags":[]}},{"cell_type":"code","source":"for i in range(len(Training_filenames)):\n    \n    print(f'Start training the fold{i} !')\n    with strategy.scope():\n        densenet = tf.keras.applications.DenseNet201(weights='imagenet', \n                                                    include_top=False,\n                                                    pooling='avg',\n                                                    input_shape=(*IMAGE_SIZE, 3))\n        model2 = get_model(densenet)\n\n    chk_callback2 = tf.keras.callbacks.ModelCheckpoint(f'densenet201_{i}_best.h5',\n                                                       save_weights_only=True,\n#                                                        monitor='val_f1_score',\n                                                       monitor='val_accuracy',\n                                                       mode='max',\n                                                       save_best_only=True,\n                                                       verbose=1)\n    \n    history2 = model2.fit(get_training_dataset(Training_filenames[i], do_onehot=True), \n                      steps_per_epoch=STEPS_PER_EPOCH, \n                      epochs=EPOCHS, \n                      validation_data=get_validation_dataset(Validation_filenames[i], do_onehot=True),\n                      validation_steps=VALIDATION_STEPS,\n                      callbacks=[lr_callback, chk_callback2],\n                      verbose=2)\n\n    del chk_callback2, history2, model2\n    gc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-07-01T02:58:16.469251Z","iopub.status.idle":"2021-07-01T02:58:16.46995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predictions","metadata":{"papermill":{"duration":0.196742,"end_time":"2021-06-16T10:55:40.423786","exception":false,"start_time":"2021-06-16T10:55:40.227044","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def run_inference(modellist, model):\n    test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n    test_images_ds = test_ds.map(lambda image, idnum: image)\n    predslist = []\n    for modelpath in modellist:\n        model.load_weights(modelpath)\n        predslist.append(model.predict(test_images_ds,verbose=0, steps=TEST_STEPS))\n    \n    preds = np.sum(predslist, axis = 0)\n    del model\n    gc.collect()\n    \n    return preds","metadata":{"papermill":{"duration":0.203824,"end_time":"2021-06-16T10:55:40.823752","exception":false,"start_time":"2021-06-16T10:55:40.619928","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T02:58:16.471477Z","iopub.status.idle":"2021-07-01T02:58:16.472129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/classify-leaves/train.csv')\ntrain.head()\nleaves_labels = sorted(list(set(train['label'])))\nn_classes = len(leaves_labels)\nclass_to_num = dict(zip(leaves_labels, range(n_classes)))\nnum_to_class = {v : k for k, v in class_to_num.items()}","metadata":{"papermill":{"duration":0.247445,"end_time":"2021-06-16T10:55:41.699673","exception":false,"start_time":"2021-06-16T10:55:41.452228","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T02:58:16.473422Z","iopub.status.idle":"2021-07-01T02:58:16.474049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\nprint('Calculating predictions...')\nmodel1list = [f'./effnetb7_{i}_best.h5' for i in range(len(Training_filenames))]\nmodel2list = [f'./densenet201_{i}_best.h5' for i in range(len(Training_filenames))]\nmodel1 = get_model(enet)\nmodel2 = get_model(densenet)\nprobs1 = run_inference(model1list,model1)\nprobs2 = run_inference(model2list,model2)\nprobabilities = 0.5*probs1 + 0.5*probs2\npredictions = np.argmax(probabilities, axis=-1)\n\npreds = []\nfor i in predictions:\n    preds.append(num_to_class[i])\n\nprint('Generating submission file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, preds]), fmt=['%s', '%s'], delimiter=',', header='image,label', comments='')","metadata":{"papermill":{"duration":31.217285,"end_time":"2021-06-16T10:56:13.112407","exception":false,"start_time":"2021-06-16T10:55:41.895122","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-01T02:58:16.475276Z","iopub.status.idle":"2021-07-01T02:58:16.475929Z"},"trusted":true},"execution_count":null,"outputs":[]}]}