{"cells":[{"metadata":{"_uuid":"5d52707f344f0dcfe5baa73532d64fc3456e06c6"},"cell_type":"markdown","source":"\n## Work through the first Lesson for Practical Deep Learning for Coders\nSlight update to revert back from current FastAI verson to version compatable with coursework. The notebook is forked from [William Hortons's Fast AI Lesson 1 Notebook](https://www.kaggle.com/hortonhearsafoo/fast-ai-lesson-1)"},{"metadata":{"_uuid":"acec03a5349f652365134989c150cbdf28ba2076","trusted":true},"cell_type":"code","source":"# Put these at the top of every notebook, to get automatic reloading and inline plotting\n%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"18646a2c1083f52ad3ea7ab38a59c77ab61e9af5"},"cell_type":"code","source":"%%capture\n!pip install fastai==0.7.0\n!pip install torchtext==0.2.3","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"23458afac17c6c5ab691b5bad3e1292082539bfc","trusted":true},"cell_type":"code","source":"# This file contains all the main external libs we'll use\nfrom fastai.imports import *","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14fa222ea053d567aaeb95f9786449c34bf87617","trusted":true},"cell_type":"code","source":"from fastai.transforms import *\nfrom fastai.conv_learner import *\nfrom fastai.model import *\nfrom fastai.dataset import *\nfrom fastai.sgdr import *\nfrom fastai.plots import *","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"188e90203f18be6de43fd149dbe94af0da40eb03"},"cell_type":"markdown","source":"`PATH` is the path to your data - if you use the recommended setup approaches from the lesson, you won't need to change this. `sz` is the size that the images will be resized to in order to ensure that the training runs quickly. We'll be talking about this parameter a lot during the course. Leave it at `224` for now."},{"metadata":{"_uuid":"276daac1c68af55ffc03e89d20c853bdf23e8522","trusted":true},"cell_type":"code","source":"PATH = \"../input/\"\nTMP_PATH = \"/tmp/tmp\"\nMODEL_PATH = \"/tmp/model/\"\nsz=224","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2a6564a6294ee094ba60038c5a94074318f18337"},"cell_type":"markdown","source":"Check that NVIDIA GPU is setup and available for use by PyTorch. Also check for CuDNN"},{"metadata":{"_uuid":"9ba310c51d959cab508081d01d87a3aaa2ab5085","trusted":true},"cell_type":"code","source":"torch.cuda.is_available()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aeffd45f60ecfcb90aa0cd37f09bbb89cbee8c01","trusted":true},"cell_type":"code","source":"torch.backends.cudnn.enabled","execution_count":null,"outputs":[]},{"metadata":{"heading_collapsed":true,"_uuid":"e7371d1918f4903352b82bb8f4b2ca55fa4386ca"},"cell_type":"markdown","source":"## First look at cat pictures\n* First reorganize the cat pictures from the Kaggle Datase\n    * The Kaggle Cats/Dogs set is organized differently than in the Fast AI course, the labels are embedded in the file names. \n* Take a look at the initial cat pictures"},{"metadata":{"hidden":true,"scrolled":true,"_uuid":"ee3a6d31ba52de6d1f294998d48b56073ee31604","trusted":true},"cell_type":"code","source":"os.listdir(PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"89fb04c770b2ef3fca6edf0263296d49da467cfb"},"cell_type":"code","source":"fnames = np.array([f'train/{f}' for f in sorted(os.listdir(f'{PATH}train'))])\nlabels = np.array([(0 if 'cat' in f else 1) for f in fnames])","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"_uuid":"09f87570c2cd82eb3eed603565240021a5a888fe","trusted":true},"cell_type":"code","source":"img = plt.imread(f'{PATH}{fnames[3]}')\nplt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a316bf2ce1d0e7985bb0d22df2d8cfcfc45c38f0"},"cell_type":"code","source":"img.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7116eb8a9a95e6c4d09868852ddc7e1c7637ace2"},"cell_type":"code","source":"img[:4,:4]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b776e883d6d870b8b997d6959a3f35cfe29c3ab1"},"cell_type":"markdown","source":"## Create a NN model in 3 lines\n* Utilize a pretrained model (resnet34)\n* First time running the model takes longer because model needs to be downloaded"},{"metadata":{"trusted":true,"_uuid":"df19d420f22aa346c36bec7ead65a3ae04bb864c"},"cell_type":"code","source":"arch = resnet34 #Set model archatecture\n#format data using FASTAI ImageClassifierData Class\ndata = ImageClassifierData.from_names_and_array(\n    path = PATH,\n    fnames = fnames, #Directory of all image file names\n    y = labels, #labels taken from filenames in previous cell\n    classes = ['dogs', 'cats'], #set labels\n    test_name = 'test', #test directory\n    tfms = tfms_from_model(arch, sz)\n)\nlearn = ConvLearner.pretrained(arch, data, precompute=True, tmp_name=TMP_PATH, models_name=MODEL_PATH)\nlearn.fit(0.01, 2) #Learning Rate set to 0.01, and n_Epochs are 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"96c790fe1f6047aadf651e93ce5195feb4e984ba"},"cell_type":"code","source":"learn.fit(0.01, 2)","execution_count":null,"outputs":[]},{"metadata":{"heading_collapsed":true,"_uuid":"500f86792c2bd9c9373d883eb301b74219a9b2a7"},"cell_type":"markdown","source":"## Our first model: quick start"},{"metadata":{"hidden":true,"_uuid":"98cddf2e4f8c750fb55756749af75acb62500c73"},"cell_type":"markdown","source":"We're going to use a <b>pre-trained</b> model, that is, a model created by some one else to solve a different problem. Instead of building a model from scratch to solve a similar problem, we'll use a model trained on ImageNet (1.2 million images and 1000 classes) as a starting point. The model is a Convolutional Neural Network (CNN), a type of Neural Network that builds state-of-the-art models for computer vision. We'll be learning all about CNNs during this course.\n\nWe will be using the <b>resnet34</b> model. resnet34 is a version of the model that won the 2015 ImageNet competition. Here is more info on [resnet models](https://github.com/KaimingHe/deep-residual-networks). We'll be studying them in depth later, but for now we'll focus on using them effectively.\n\nHere's how to train and evalulate a *dogs vs cats* model in 3 lines of code, and under 20 seconds:"},{"metadata":{"_uuid":"7aa60a8f5fecaf1e2054fbb53029fa881108ea24","trusted":true},"cell_type":"code","source":"# Uncomment the below if you need to reset your precomputed activations\n# shutil.rmtree(f'{PATH}tmp', ignore_errors=True)","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"scrolled":true,"_uuid":"34c2f123f94ac01f68f8ab36a2943486fbe35ab5","trusted":true},"cell_type":"code","source":"arch=resnet34\ndata = ImageClassifierData.from_names_and_array(\n    path=PATH, \n    fnames=fnames, \n    y=labels, \n    classes=['dogs', 'cats'], \n    test_name='test', \n    tfms=tfms_from_model(arch, sz)\n)\nlearn = ConvLearner.pretrained(arch, data, precompute=True, tmp_name=TMP_PATH, models_name=MODEL_PATH)\nlearn.fit(0.01, 2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef0642de78f7ec021545e8341d2f808c419915c9"},"cell_type":"markdown","source":"## Evaluate the model that has been created in three lines above\n* Take a look at some of the predictions\n* Exaine some of the attributres of the ImageClassifierData\n* See what kind of images exist that the model is uncertan about"},{"metadata":{"trusted":true,"_uuid":"2d49934e40e5bab6d96f03e00aad28b315b4f5a2"},"cell_type":"code","source":"data.classes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aecfd0ce81ffd57ccd4d8e574a5f9483ed872e11"},"cell_type":"code","source":"data.val_y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"136f540bc40fc51ea7da50253cfa4b3967c2e642"},"cell_type":"code","source":"log_preds = learn.predict()\nlog_preds.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9f2b4b93ccf7112f9a825efba2ca8c9083633eb5"},"cell_type":"code","source":"log_preds[:10] #take a look at the last ten log-predictions\n#first column represents log-probability of dogs, second cats`","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8463ef797b45f378a57e95679b0c753a2a1bd843"},"cell_type":"code","source":"preds = np.argmax(log_preds, axis=1) #which column is higher? (dogs or cats)\nprobs = np.exp(log_preds[:,1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a76a9ddb98c9f0abf9e21c2a963398a163862a26"},"cell_type":"code","source":"def rand_by_mask(mask): \n    return np.random.choice(np.where(mask)[0], min(len(preds), 4), replace = False)\n\ndef rand_by_correct(is_correct):\n    return rand_by_mask((preds== data.val_y)==is_correct)\n\ndef plots(ims, figsize = (12,6), rows = 1, titles = None):\n    f= plt.figure(figsize = figsize)\n    for i in range(len(ims)):\n        sp = f.add_subplot(rows, len(ims)//rows, i+1)\n        sp.axis('Off')\n        if titles is not None: sp.set_title(titles[i], fontsize = 16)\n        plt.imshow(ims[i])\n\ndef load_img_id(ds, idx): return np.array(PIL.Image.open(PATH+ds.fnames[idx]))\n\ndef plot_val_with_titles(idxs, title):\n    imgs = [load_img_id(data.val_ds, x) for x in idxs]\n    title_probs = [probs[x] for x in idxs]\n    print(title)\n    return plots(imgs, rows=1, titles = title_probs, figsize = (16,8)) if len(imgs)>0 else print(\"Not Found\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"290090548d044028bb27b7b64ebdce2d1af45279"},"cell_type":"code","source":"plot_val_with_titles(rand_by_correct(True), \"Correctly Classified\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10d95118b81dfe020dec9700822f4c37798c567a"},"cell_type":"code","source":"plot_val_with_titles(rand_by_correct(False), \"Incorrectly Classified\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"973fb3b2fa8ae73cd46b3b5b16f999a053e49388"},"cell_type":"code","source":"def most_by_mask(mask, mult):\n    idxs = np.where(mask)[0]\n    return idxs[np.argsort(mult * probs[idxs])[:4]]\n\ndef most_by_correct(y, is_correct):\n    mult = -1 if (y==1)==is_correct else 1\n    return most_by_mask(((preds == data.val_y)==is_correct) & (data.val_y == y), mult)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d758ebd3f64158fb99a090f1e34259a47cb66472"},"cell_type":"code","source":"plot_val_with_titles(most_by_correct(0, True), \"Most Cat Like Cats\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19954de06cdedf64c7ffa84716a3eaeeeaec7ffd"},"cell_type":"code","source":"plot_val_with_titles(most_by_correct(1, True), \"Most Dog Like Dogs\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"57ef07ddfebab1e941db98d1aeef4b5772e8e63f"},"cell_type":"code","source":"plot_val_with_titles(most_by_correct(0, False), \"Least Cat Like Cats\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0e54e447e892ce9176c24b16a8356c7205310012"},"cell_type":"code","source":"plot_val_with_titles(most_by_correct(1, False), \"Least Dog Like Dogs\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f6e8501f662273bdd1d504839f5740cecf7eec9f"},"cell_type":"code","source":"most_uncertain = np.argsort(np.abs(probs -0.5))[:4]\nplot_val_with_titles(most_uncertain, \"Most uncertain preds\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3af97a52dafcdb22b8f82c49624eb2ed41b87bd7"},"cell_type":"markdown","source":"## How to Choose a Learning Rate\nAs stated in the class the learning rate is generally the most important hyperparameter for NN, particularly within the Fast AI framework"},{"metadata":{"trusted":true,"_uuid":"5b6ec83bd52ea2012b393007b4a4d8de4a44dd2c"},"cell_type":"code","source":"learn = ConvLearner.pretrained(arch, data, precompute = True, tmp_name=TMP_PATH, models_name=MODEL_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"81f53c8ef1e1ad2f6a823ee82d020bf707d9d304"},"cell_type":"code","source":"lrf = learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ded3eaa213a85b720612d35c0ec80b7661dbe6d0"},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"18a8ef5d6a4bbe0aedea1e8de1286654d606ebe5"},"cell_type":"markdown","source":"Can just adjust this number to get good results (fast AI internalizes hyperparameter tuning)\n\n[Paper referencing Deep Network Learning Rates](https://arxiv.org/abs/1506.01186)\n\nRule of thumb: Find lowest point in learning schedule plot, and dial back a factor of ten.\n- For instance below, take low point 10e-1 and set 10e-2 as initial set learning rate\n"},{"metadata":{"trusted":true,"_uuid":"2dc07d546cabb7ed9dab9a329e561e80be06d14a"},"cell_type":"code","source":"learn.sched.plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"505e4110b52aae0be48b5daa8dae53a69a84eda3"},"cell_type":"markdown","source":"## Data Augmentation"},{"metadata":{"trusted":true,"_uuid":"e63e3ad9c39affcb30bcb99859fbfb1b13f00224"},"cell_type":"code","source":"tfms = tfms_from_model(resnet34, sz, aug_tfms=transforms_side_on, max_zoom = 1.1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2968080070f931a4c49adb4bf9e5b73d05a3317d"},"cell_type":"markdown","source":"transforms_side_on only slightly tweaks angle of image and only does a horizontal flip of the image\n\nNext refromat the ImageClassifierData using the transforms"},{"metadata":{"trusted":true,"_uuid":"b51ed09dcad0c0380abd4c4b31adb358bfe0eb05"},"cell_type":"code","source":"def get_augs():\n    data = ImageClassifierData.from_names_and_array(\n        path = PATH,\n        fnames = fnames, #Directory of all image file names\n        y = labels, #labels taken from filenames in previous cell\n        classes = ['dogs', 'cats'], #set labels\n        test_name = 'test', #test directory\n        tfms = tfms,\n        bs=2,\n        num_workers=1\n    )\n    x,_ = next(iter(data.aug_dl))\n    return data.trn_ds.denorm(x)[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"add85d74f87b9c72786ce6354290fb593db5be8c"},"cell_type":"code","source":"ims = np.stack([get_augs() for i in range(6)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"941112589ba110a6daacc46a9c8b62a6cfc33a19"},"cell_type":"code","source":"plots(ims, rows=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2f512070a1728b371e071670c261b2c6820d29a8"},"cell_type":"code","source":"data = ImageClassifierData.from_names_and_array(\n    path = PATH,\n    fnames = fnames, #Directory of all image file names\n    y = labels, #labels taken from filenames in previous cell\n    classes = ['dogs', 'cats'], #set labels\n    test_name = 'test', #test directory\n    tfms = tfms,\n) #This reformats the data with the trasforms\nlearn = ConvLearner.pretrained(arch, data, precompute=True, tmp_name = TMP_PATH, models_name = MODEL_PATH)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"db6a6e881b416c61105ff15c39762cbc0f5b6af3"},"cell_type":"markdown","source":"When we create a new classifier by setting pre-comptue = True to create a linear NN layer that sits on top of the Resnet34 precomputed neural network. Since we're only training a linear layer and not actually changing the activation function in the resent34  model the data augmentation doesn't help since it's not impacting any of the activations of the NN"},{"metadata":{"trusted":true,"_uuid":"bc2bd0df5d3aa4d73c9b56265ef0eb1b11ba6c30"},"cell_type":"code","source":"learn.fit(1e-2, 1) #Re train teh model with single epoch","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2b3df2fe1b683f121c4071f160a0786aaee89b2a"},"cell_type":"markdown","source":"Set Precompute to False to utilize data augmentation."},{"metadata":{"trusted":true,"_uuid":"c5b06cc66615ee4a0dca1990387d08c61bd79c55"},"cell_type":"code","source":"learn.precompute=False ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42690ca3f98ae6130b5d4542cb16d0ab216e9724"},"cell_type":"code","source":"learn.fit(1e-2, 3, cycle_len=1) #takes some time to run","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50aa1479984eecad78870045eefbb312c232ba58"},"cell_type":"markdown","source":"Accuracy isn't particularly getting better, but the amount of overfitting that occurs is reduced. \n\nAdditionally theres the addition of the cycle_length parameter. This uses **Stochastic Gradient Descent with Restarts** to tweak learning rate as we go thorugh itterations\n*     The reduction of learning rate reductino is called *learning rate annealing*\n*     A good function to do learning rate annealing is the cosine function. This allows for more refinement when getting close to the ideal solution"},{"metadata":{"trusted":true,"_uuid":"73712538d67e44811ab04125e61ae0bddcf454b4"},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5b58e9f86a197cd22394d4bdf47b8b884de00093"},"cell_type":"markdown","source":"## Differential Learning Rate"},{"metadata":{"trusted":true,"_uuid":"de145ed94a5964c1d889329071c6bb46d57aa72c"},"cell_type":"code","source":"learn.unfreeze()\nlr = np.array([1e-4, 1e-3, 1e-2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d14833d0e4aa1d8ea0ca27aac7e9f01f6657e471"},"cell_type":"code","source":"learn.fit(lr, 3, cycle_len = 1, cycle_mult = 2) #Re-fit model with differential learning rate. Some time is required to run the 6 epochs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f35222078c66b25d2a4da09dfb2d82a9fcb047bd"},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"24a7bdaffebc695f814391acda4183fd9cb0638d"},"cell_type":"code","source":"learn.save('224_all')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"da6b48d28ade09f72aa59bae007771ca38eddef5"},"cell_type":"code","source":"learn.load('224_all')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41cc2608765c2c410b63c47835a242341b75cba6"},"cell_type":"code","source":"log_preds,y = learn.TTA()\nprobs = np.mean(np.exp(log_preds),0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"845b1b5c31e4481c7db77d6144d9ae5b709617a9"},"cell_type":"code","source":"accuracy_np(probs, y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc5cc15fb786f3d0a41895922af6ea547f194df7"},"cell_type":"markdown","source":"## Review the easty steps to train a world class image classifier (Within the FastAI framework)\n1. Ena ble data augmentation, and precompute=True\n2. Use lr_find() to find the highest learning rate where loss is still clearly improving\n3. Train last lyer from precomputed activations for 1-2 epochs\n4. Train last layer with data augmentation (i.e. precompute = False) for 2-3 epochs with cycle_len = 1\n5. Unfreeze all layers\n6. Set earlier layers to 3x-10x lower learning rate than next higher layer\n7. Use lr_find() again\n8. Train full network with cycle_mult=2 until overfitting"},{"metadata":{"trusted":true,"_uuid":"54f8f22dbe546286d98be37f657b0e6513991741"},"cell_type":"markdown","source":"## Analyze Results"},{"metadata":{"trusted":true,"_uuid":"fada948f028df62d9f5dd885b63cb1525c2d03ae"},"cell_type":"code","source":"preds = np.argmax(probs, axis=1)\nprobs = probs[:,1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd8029658edff9dd6cbf5c096877dda95dd491e8"},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y, preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3b9def733405cedcb5f525bcdbca8e7b89e658cd"},"cell_type":"code","source":"plot_confusion_matrix(cm, data.classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d83606f8184d4d4a1520b47b6b53ac1dd00e98e5"},"cell_type":"code","source":"plot_val_with_titles(most_by_correct(0, False), \"Most incorrect cats\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7632294970837dbe9bbf6ecbc69a5ad28fb477d8"},"cell_type":"code","source":"plot_val_with_titles(most_by_correct(1, False), \"Most incorrect dogs\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f218a3141bf0820e891dfb22ee7b41ef55868814"},"cell_type":"markdown","source":"## Predict test results and submit "},{"metadata":{"trusted":true,"_uuid":"14f26612907f841649389499303383015ba6fe9a"},"cell_type":"code","source":"test_pred  = learn.predict(is_test=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"58c9efdd4b5f6f9032941aff9ed67a95b0dc560d"},"cell_type":"code","source":"test_pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"faa32badbc5cabfcbf7fd5f52fbeabc52d1ccc51"},"cell_type":"code","source":"pred = (np.argmax(test_pred, axis =1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee083504dd7bab5760d845dbfa5ab85d0d6ec1a0"},"cell_type":"code","source":"submission = pd.DataFrame({'id': os.listdir(f'{PATH}test'), 'label': pred})\nsubmission.to_csv('submission.csv')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"toc":{"colors":{"hover_highlight":"#DAA520","navigate_num":"#000000","navigate_text":"#333333","running_highlight":"#FF0000","selected_highlight":"#FFD700","sidebar_border":"#EEEEEE","wrapper_background":"#FFFFFF"},"moveMenuLeft":true,"nav_menu":{"height":"266px","width":"252px"},"navigate_menu":true,"number_sections":true,"sideBar":true,"threshold":4,"toc_cell":false,"toc_section_display":"block","toc_window_display":false,"widenNotebook":false}},"nbformat":4,"nbformat_minor":1}