{"cells":[{"metadata":{"_uuid":"ae60ceed7e677b90d17e5b37947064749e2eaa7a","_cell_guid":"73b00826-57ae-4dd1-9f1d-a53854594568"},"cell_type":"markdown","source":"## Effective augmentation - mixup & cutout/random erasing\nThanks to [JihaoLiu](https://www.kaggle.com/jihaoliu), he introduced \"mixup\" in [his discussion post](https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/discussion/47730) at [TensorFlow Speech Recognition Challenge](https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/).\n\nAnd there's also similarly effective augmentation method cutout[2] or random erasing[3] introduced in [yu4u's github repository](https://github.com/yu4u/mixup-generator). These are the same basically.\n\nPaper:\n- [1] [Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz, \"mixup: Beyond Empirical Risk Minimization,\" in \narXiv:1710.09412, 2017 \\[cs.LG\\]](https://arxiv.org/abs/1710.09412)\n- [2] [T. DeVries and G. W. Taylor, \"Improved Regularization of Convolutional Neural Networks with Cutout,\" in arXiv:1708.04552, 2017.](https://arxiv.org/abs/1708.04552)\n- [3] [Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, \"Random Erasing Data Augmentation,\" in arXiv:1708.04896, 2017.](https://arxiv.org/abs/1708.04896)\n\nI confirmed to be useful for this acoustic scene classification task as well.\n\nHere's what's used in my code for mixup. Cutout/random erasing is also introduced at the bottom of this post.","outputs":[],"execution_count":null},{"metadata":{"collapsed":true,"_uuid":"9976e5f699ce4a37e16c26d30df2aa75dba58318","_cell_guid":"f4bc5b54-ff56-4525-b73c-459ff666cff8","trusted":false},"cell_type":"code","source":"import numpy as np\n\ndef mixup(data, one_hot_labels, alpha=1, debug=False):\n    np.random.seed(42)\n\n    batch_size = len(data)\n    weights = np.random.beta(alpha, alpha, batch_size)\n    index = np.random.permutation(batch_size)\n    x1, x2 = data, data[index]\n    x = np.array([x1[i] * weights [i] + x2[i] * (1 - weights[i]) for i in range(len(weights))])\n    y1 = np.array(one_hot_labels).astype(np.float)\n    y2 = np.array(np.array(one_hot_labels)[index]).astype(np.float)\n    y = np.array([y1[i] * weights[i] + y2[i] * (1 - weights[i]) for i in range(len(weights))])\n    if debug:\n        print('Mixup weights', weights)\n    return x, y","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fd9bbfe3a18a47351f915d536ce318036de1771a","_cell_guid":"6246a2c3-a794-4a13-b75c-a8d4afb58720"},"cell_type":"markdown","source":"### Example\nFirst, prepare data.","outputs":[],"execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":false,"collapsed":true},"cell_type":"code","source":"%matplotlib inline\nfrom matplotlib import pyplot as plt\nimport os\nimport numpy as np\nimport pandas as pd\nimport keras\n\ndatadir = \"../input/\"\n\n# Load data as is\nX_train_org = np.load(datadir+'X_train.npy')\nX_test = np.load(datadir+'X_test.npy')\ny_labels_train = pd.read_csv(datadir+'y_train.csv', sep=',')['scene_label'].tolist()\n\n# Make label list and converters\nlabels = sorted(list(set(y_labels_train)))\nlabel2int = {l:i for i, l in enumerate(labels)}\nint2label = {i:l for i, l in enumerate(labels)}\n\n# Map y_train to int labels\ny_train_org = keras.utils.to_categorical([label2int[l] for l in y_labels_train])\n\n# Train/Validation split --> X_train/y_train, X_valid/y_valid\nsplitlist = pd.read_csv(datadir+'crossvalidation_train.csv', sep=',')['set'].tolist()\nX_train = np.array([x for i, x in enumerate(X_train_org) if splitlist[i] == 'train'])\nX_valid = np.array([x for i, x in enumerate(X_train_org) if splitlist[i] == 'test'])\ny_train = np.array([y for i, y in enumerate(y_train_org) if splitlist[i] == 'train'])\ny_valid = np.array([y for i, y in enumerate(y_train_org) if splitlist[i] == 'test'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false,"collapsed":true},"cell_type":"code","source":"# Now augment by mixup --> X_train + mixup'ed X_train, y_train + mixup'ed y_train\n# ** This is slight deviation from original idea in paper,\n#    using mixup as preprocessing. **\ntmp_X, tmp_y = mixup(X_train, y_train, alpha=1)\nX_train, y_train = np.r_[X_train, tmp_X], np.r_[y_train, tmp_y]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd291714ac84cc5430ebc45ddf2bac8f2da7b1ec","_cell_guid":"4b018352-2451-4527-8fd3-4c870596dbed"},"cell_type":"markdown","source":"### Visualize mixup results\nExample above shows double the entire dataset by mixup, here let's use just small number of samples and  see how it is mixed-up. ","outputs":[],"execution_count":null},{"metadata":{"_uuid":"07ae34b4cf85d37392a5f2b111d1e3a4380a9495","_cell_guid":"6e519f41-c528-45fa-aa37-906cee4f3179","trusted":false,"collapsed":true},"cell_type":"code","source":"# Here we pick first five samples, and mix-up.\nfive_X, five_y = X_train[:5], y_train[:5]\nmixup_X, mixup_y = mixup(five_X, five_y, alpha=3, debug=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5629dfc99a5aa0acf84a8bf29b70fddf283f5315","_cell_guid":"8cb005eb-f64c-4b7c-b5cb-df856b55b7ac","trusted":false,"collapsed":true},"cell_type":"code","source":"# Visualize them.\ndef plot_dataset(XYs, titles):\n    for i, (x, y) in enumerate(XYs):\n        fig, (axL, axR) = plt.subplots(ncols=2, figsize=(10,4))\n        axL.pcolormesh(x)\n        axL.set_title('%s [%d]' % (titles[0], i))\n        axL.grid(True)\n        axR.pcolormesh(y)\n        axR.set_title('%s [%d]' % (titles[1], i))\n        axR.grid(True)\n        plt.show()\n\nplot_dataset(zip(five_X, mixup_X), ('Original data', 'Mixup result'))# Here we pick first two samples, and mix-up.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97c2e58059d39eb731d19fcc7f0767425da18f42","_cell_guid":"8516d2df-a249-4313-83f3-0c53af0dda03","trusted":false,"collapsed":true},"cell_type":"code","source":"print(mixup_y)","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"99a973c0d8f0655ec083ea277bcbf5a42bfa4588","_cell_guid":"e7112a5a-8d7e-48b0-83be-eda19cb2efa6"},"cell_type":"markdown","source":"You can see both X and label y are mixed (except for 0th due to mixed itself).\n\n### How it worked\n- Effective to prevent overfitting. Train/valid accuracy gets closer.\n- Improves 1-2% so far in my attempts.\n\n### Deviation from original paper[1]\nI introduced mixup as preprocessing to augment training samples.\nBut it is basically intended to use in every mini-batch.\n```python\nfor (x1, y1), (x2, y2) in zip(loader1, loader2):\n    lam = numpy.random.beta(alpha, alpha)\n    x = Variable(lam * x1 + (1. - lam) * x2)\n    y = Variable(lam * y1 + (1. - lam) * y2)\n    optimizer.zero_grad()\n    loss(net(x), y).backward()\n    optimizer.step()\n```\n(quoted from paper)\n\n### For Keras users, better implementation there\nYou can use the following with `ImageDataGenerator`, it's useful. It also contains cutout/random_erasing.\n\n- [github/yu4u/mixup-generator, An implementation of \"mixup: Beyond Empirical Risk Minimization\"](https://github.com/yu4u/mixup-generator)\n\nYou can use them as follows (quoted from it's sample)\n```python\ndatagen3 = ImageDataGenerator(\n    width_shift_range=0.1,\n    height_shift_range=0.1,\n    horizontal_flip=True,\n    preprocessing_function=get_random_eraser(v_l=0, v_h=255)) ## <==== Cutout/randome erasing part.\n\ngenerator3 = MixupGenerator(x_train, y_train, alpha=1.0, datagen=datagen3)() ## <=== Plus mixup.\n```\n\n_CAUTION:_ Be careful to feed `v_l` & `v_h` to get_randome_eraser correctly, or your data will be wrongly augmented. The default 0/255 doesn't work and will generate wrong image.\n\nHere's from my code.\n```python\ndatagen = ImageDataGenerator(\n    featurewise_center=True,  # set input mean to 0 over the dataset\n       :\n   preprocessing_function=get_random_eraser(v_l=np.min(X_train), v_h=np.max(X_train)) # Trainset's boundaries.\n)\nmygenerator = MixupGenerator(X_train, y_train, alpha=1.0, batch_size=batch_size, datagen=datagen)\n    :\nmodel.fit_generator(mygenerator(),\n                    steps_per_epoch=X_train.shape[0] // batch_size,\n                    epochs=epochs,\n                    validation_data=test_datagen.flow(X_valid, y_valid), callbacks=callbacks)\n```\n\n","outputs":[],"execution_count":null},{"metadata":{"collapsed":true,"_uuid":"1cb7ee205fae313e8c38534aa5e05daa16b6605e","_cell_guid":"42f3d66a-993e-40c2-9883-bad9a9926527","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}