{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","collapsed":true},"cell_type":"markdown","source":"<center>\n<h1> Danbury AI June 2018: Workshop Part 2</h1>\n<h2>The Titanic Disaster</h2>\n</center>\n\n**About this Notebook **\n\nThis notebook explores the process of using a neural network to produce a predictive model of the titanic dataset hosted on kaggle. There are many steps in this process that must not be overlooked or it will be more difficult to optimize the performance of your network. In particular, taking time to do EDA, feature engineering, and be considerate when preprocessing the data will stave of many potential modeling headaches later on. It is our aim to make you comfortable with the basics of each of these stages, so you can effectively use neural networks on your data. \n\nLearning Objectives:\n* Learn how to use neural networks on small datasets. \n* Become familiar with the data science process. \n* How to use a neural network on datasets with categorical and quantitative features. \n\n**Competition Overview  **\n\nThe sinking of the RMS Titanic is one of the most infamous shipwrecks in history.  On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships.\n\nOne of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for the passengers and crew. Although there was some element of luck involved in surviving the sinking, some groups of people were more likely to survive than others, such as women, children, and the upper-class.\n\nIn this challenge, we ask you to complete the analysis of what sorts of people were likely to survive. In particular, we ask you to apply the tools of machine learning to predict which passengers survived the tragedy.\n\n[Kaggle Competition Source](https://www.kaggle.com/c/titanic)\n\n**Key Challenges of Using a Neural Network**\n\nThe number of parameters a statistical model has determines the range of functions it can represent. We call this range the *capacity* of a model. Models with greater capacity have a tendency to overfit the training data. Neural Networks are the epitome of high-capacity models. High-capacity models generally don't do well on small datasets. Large datasets balance out a neural network's tendency to overfit. With small datasets, we must pay great attention to how we regularize the network in order to fight this overfitting behavior. While there are many ways of doing this, in this notebook we will be using the dropout method. \n\nThe Titanic data is a very small dataset of about 800 or so samples. This means our success is dependent upon proper regularization of the network. Lower capacity models like random forests or svms would do much better out of the box because they will not overfit as reaidly; however, if we tune the network well, we can generally beat their performance. \n\n\n# Library Imports"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# General Libraries \nimport numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport os\n\n# Scikit-Learn Libraries\nfrom sklearn.metrics import log_loss\nfrom sklearn.model_selection import StratifiedKFold,cross_val_score,train_test_split,learning_curve\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn import linear_model,svm,tree\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier\n\n# Keras Libraries \nfrom keras.models import Model\nfrom keras.layers import Input, Dense, Dropout\n\n# Interactive Widgets\nfrom ipywidgets import interact_manual\nfrom IPython.display import display\n\n# Print out the folders where our datasets live. \nprint(\"Datasets: {0}\".format(os.listdir(\"../input\")))","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"971af06e07d0ec7ed5ba2d86bdd002322b096f15"},"cell_type":"markdown","source":"## 1. Exploratory Data Analysis ( EDA )\nWe will begin by exploring the dataset. Getting a feel for the data helps us determine what features we want to use in our model and how we may process them in the ETL stage. In addition, we will compute some baselines based on our initial understanding of the data and use them later when we are evaluating our model's performance. "},{"metadata":{"_uuid":"f2d4ea65b45cbd15f8cd5139b7d9a5b2c7630f7e","trusted":true},"cell_type":"code","source":"# Load Test Data\ntestDF = pd.read_csv(\"../input/titanic/test.csv\")\n\n# Load Training Data\ntitanicDF = pd.read_csv(\"../input/titanic/train.csv\")\ntitanicDF.head(10)","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"c4e268196e44b404ccacb593e13d121ce533c5eb"},"cell_type":"markdown","source":"Let's visualize the data. "},{"metadata":{"_uuid":"bd3c153a8ff780a11bbe5f1f564efe6859f56e56","trusted":true},"cell_type":"code","source":"nRows = 3\nnCols = 3\n\nplt.figure(figsize=(6*3,6*3))\nplt.subplot(nRows,nCols,1)\n\ntitanicDF[\"Survived\"].value_counts().plot.pie(autopct=\"%.2f%%\")\n\nplt.subplot(nRows,nCols,2)\ntitanicDF[\"Sex\"].value_counts().plot.pie(autopct=\"%.2f%%\")\n\nplt.subplot(nRows,nCols,3)\ntitanicDF[\"Pclass\"].value_counts().plot.pie()\n\nplt.subplot(nRows,nCols,4)\ntitanicDF[\"SibSp\"].value_counts().plot.pie()\n\nplt.subplot(nRows,nCols,5)\ntitanicDF[\"Parch\"].value_counts().plot.pie()\n\nplt.subplot(nRows,nCols,6)\ntitanicDF[\"Embarked\"].value_counts().plot.pie()\n\nplt.subplot(nRows,nCols,7)\nplt.title(\"Age Histogram\")\ntitanicDF[\"Age\"].hist()\n\nplt.subplot(nRows,nCols,9)\nplt.title(\"Fare Histogram\")\ntitanicDF[\"Fare\"].hist()\n\nplt.show()","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"98f5dedcf6cd5cb2308977c72c49d741c608cd3c"},"cell_type":"markdown","source":"**Workshop Question:** What other visualizations would help us understand the dataset? "},{"metadata":{"_uuid":"a5b8870832d1cec9bd98f57f5494596567015fb4"},"cell_type":"markdown","source":"# 2. Feature Engineering\n\nWhen we use features from a dataset to produce new features it is called *feature engineering*. From our our understanding of the problem, children were given a high precedence on life boats, yet we do not have an is_child feature. How would we engineer this feature? Let's explore. \n\n**1.) How many passengers are children?** \n\nWe ask this question because we would like to determine which men were children, since we know being a woman or child has a better chance of being on the lifeboats and therefore has a greater chance of survival. The most obvious fields for deriving this feature are: Age, PArch, and SibSp. We will use age here. We must, however, make an assumption as to what age marks a child."},{"metadata":{"_uuid":"bf3074a563faeca37af60a7980601747d05da69a","trusted":true},"cell_type":"code","source":"# Plot\ndef plotChilds(ageThresh=10):\n    df = titanicDF[\"Age\"].dropna()\n    childFeat = df.map(lambda x: x<ageThresh)\n    \n    plt.figure(figsize=(12,12))\n    \n    plt.subplot(221)\n    plt.title(\"Size of Age Group\")\n    childFeat.value_counts().plot.pie(autopct=\"%.2f%%\")\n    \n    \n    newDF = titanicDF.assign(isChild=df.map(lambda x: x<ageThresh))\n    res = newDF.groupby(\"isChild\")[\"Survived\"]\n    adultC,childC = res.count()\n    adultS,childS = res.sum()\n    \n    plt.subplot(222)\n    plt.title(\"Age Group Survival\")\n    plt.pie([childC-childS,childS],labels=[\"Died\",\"Survived\"],autopct=\"%.2f%%\")\n    \n    plt.subplot(223)\n    plt.title(\"Age Group Gender Distribution\")\n    newDF.groupby(\"isChild\")[\"Sex\"].value_counts()[1].plot.pie(autopct=\"%.2f%%\")\n    \n    plt.subplot(224)\n    plt.title(\"Age Group Class Distribution\")\n    newDF.groupby(\"isChild\")[\"Pclass\"].value_counts()[1].plot.pie(autopct=\"%.2f%%\")\n    \n    plt.show()\n    display(newDF.head())\n    \ninteract_manual(plotChilds,ageThresh=(0,100))","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"23216e502495b80cae4d5e8561e61be175e0ad51"},"cell_type":"markdown","source":"**Workshop Questions:** How do we interpret the above visualizations? At what age would we consider a passenger a child? Should we use the presence of parents via the *Parch* feature as well? "},{"metadata":{"_uuid":"bb821c896d3f3e4f98714269cfa167f89733b4fa"},"cell_type":"markdown","source":"## 3. Preprocessing\nHere we do some simple preprocessing. Mainly we translate categorical variables to a [one-hot representation](https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/) and fill missing values with zeros. We do not use the cabin, ticket, or name features in this notebook in order to keep things simple."},{"metadata":{"_uuid":"ce3cfd8399e0e9088e276ab8c1b3b2a8f6af68d0","trusted":true},"cell_type":"code","source":"# Our prepprocessing function which is applied to every row of the target dataframe. \ndef preprocessRow(row):\n    # Process Categorical Variables - One-Hot-Encoding\n    sex      = [0,0]\n    embarked = [0,0,0]\n    pclass   = [0,0,0]\n    \n    if row[\"Sex\"] == \"male\":\n        sex = [0,1]\n    elif row[\"Sex\"] == \"female\":\n        sex = [1,0]\n    \n    if row[\"Embarked\"] == \"S\":\n        embarked = [0,0,1]\n    elif row[\"Embarked\"] == \"C\":\n        embarked = [0,1,0]\n    elif row[\"Embarked\"] == \"Q\":\n        embarked = [1,0,0]\n    \n    if row[\"Pclass\"] == 1:\n        pclass   = [0,0,1]\n    elif row[\"Pclass\"] == 2:\n        pclass   = [0,1,0]\n    elif row[\"Pclass\"] == 3:\n        pclass   = [1,0,0]\n    \n    return pclass+sex+[row[\"Age\"],row[\"SibSp\"],row[\"Parch\"],row[\"Fare\"]]+embarked\n\n# Labels for the feature columns. \nfeatureLabels = [\"3 Class\",\"2 Class\",\"1 Class\",\"Female\",\"Male\",\"Age\",\"SibSp\",\n                 \"Parch\",\"Fare\",\"Q Embarked\",\"C Embarked\",\"S Embarked\"]\n\n# Fill Missing Values\ntitanicDF = titanicDF.fillna(0).sample(frac=1)\n\n# Preprocess Data\ntitanicMat = np.stack(titanicDF.apply(preprocessRow,axis=1).values)\n\n# View what the training vectors look like. \ntmp = pd.DataFrame(titanicMat)\ntmp.columns = featureLabels\ntmp.head()","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"0c01e1163a3077b1605ddcf9c2d45227267e5540"},"cell_type":"markdown","source":"** Workshop Questios ** \n* What type of preprocessing may we want to do on ticket, name, and cabin to include them in our model? Would the length of the passengers name be helpful? \n* When we use features from a dataset to produce new features it is called *feature engineering*. What features may we engineer to make modeling easier? From our our understanding of the problem, children were given a high precedence on life boats, yet we do not have an is_child feature. How would we engineer this feature? \n\nIn order to plot our learning curves, which allow us to understand how well our model is performing, we need a training and validation set. We train on the training set and hold out the validation set. Since our model is not exposed to the validation data during training, we can use it to determine how well our model generalizes to new data; however, for a better measure of how our model will perform we must use a cross-validation method. "},{"metadata":{"_uuid":"5fec47d301edc40bb29f27e909fbd020b323d337","collapsed":true,"trusted":true},"cell_type":"code","source":"# Size of validation set. \nsplitSize = 0.2\n\ntitanic_X, titanic_y = [titanicMat, titanicDF[\"Survived\"].values]\ntitanic_train_x, titanic_validation_x, titanic_train_y , titanic_validation_y = train_test_split(titanic_X,titanic_y, test_size=splitSize)","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"25ac627374a41b8dfc9e77de12bca84b8ed19f68"},"cell_type":"markdown","source":"## 4. Naive Baselines\nBefore you begin modeling it is a good idea to hand write some baseline models based on your understanding of the problem and basic dataset statistics. A common baseline model in binary classification is to simply predict the the most represented class. In our case, this simply means: given any feature vector representing a passenger, we predict they die because that is the most common occurrence. This baseline is very helpful in the context of deep learning because if we catch our models performing similarly to our mean baseline, by having a similar loss, we know that no useful representation has been learned. "},{"metadata":{"_uuid":"d62cdfd3c09ffbab5d45248d423c31633567c9cc","trusted":true},"cell_type":"code","source":"def meanBaseline(df):\n    nRows = df[\"Survived\"].shape[0]\n    mean = df[\"Survived\"].sum()/nRows\n    return log_loss(df[\"Survived\"],np.full(nRows,mean))\n\nprint(\"Mean Baseline: {0}\".format(meanBaseline(titanicDF)))","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"c4fb864e6ec39e8e6de10d01103df8bc6fb2e289"},"cell_type":"markdown","source":"We also know from history that women and children were given precedence on the life boats. Being able to predict the latent  variable which represents a passenger getting on a lifeboat is the same as predicting survival -- to my knowledge, all that successfully boarded these boats survived. We use this knowledge to write a gender model: if a passenger is a female they survive, otherwise they die. "},{"metadata":{"_uuid":"c1f20bc14387789be174a5bec8b772acdf3372a1","trusted":true},"cell_type":"code","source":"def genderBaseline(df):\n    nRows = df[\"Survived\"].shape[0]\n    pred = df[\"Sex\"].map(lambda x: 1 if x == \"female\" else 0)\n    res = df[\"Survived\"]==pred\n    return res.sum()/res.count()\n\nprint(\"Gender Baseline: {0}\".format(genderBaseline(titanicDF)))","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"9a1ea4669ecedd73563b6e2e5741c92f0506bfa1"},"cell_type":"markdown","source":"**Workshop Problem: ** What other naive baselines can we derive from our understanding of the problem?  "},{"metadata":{"_uuid":"8c3a5b306f71858c141f8280f1ee1a542fb767d6","collapsed":true,"trusted":true},"cell_type":"code","source":"# Put your code here. ","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"a655a2d8c78a7b757618bd8e446f69757aa484ae"},"cell_type":"markdown","source":"## 5. Non-Neural Modeling\nIn this section we will train many scikit models in order to gain some perspective on how different models perform on the data. What is absent here is hyperparameter adjustment, so we must take these results with a grain of salt. The results here do not represent the optimal result we can get with these models -- we'd need to tune hyperparams for that."},{"metadata":{"_uuid":"bb4563cb3d60bf13f7c1fb6058c0cf335f79f75e","trusted":true},"cell_type":"code","source":"models = {\"Linear Regression\":linear_model.LinearRegression(),\"Logistic Regression\":linear_model.LogisticRegression(),\n         \"Ridge Regression\":linear_model.Ridge(),\"Lasso Regression\":linear_model.Lasso(),\n         \"Bayesian Ridge Regression\":linear_model.BayesianRidge(),\"Perceptron\":linear_model.Perceptron(max_iter=1000),\n         \"Support Vector Machine\":svm.SVC(gamma=\"auto\"),\"Gaussian Naive Bayes\":GaussianNB(),\"Decision Tree\":tree.DecisionTreeClassifier(),\n         \"Random Forest\":RandomForestClassifier(),\"AdaBoost\":AdaBoostClassifier(),\n         \"Gradient Boosting\":GradientBoostingClassifier()}\nkFolds = 40\n\ndef scoringFN(model,X,y):\n    pred = model.predict(X)\n    pred[pred <= 0.5] = 0\n    pred[pred > 0.5] = 1\n    return np.sum(y == pred)/y.shape[0]\n\nfor mod in models:\n    model = models[mod]\n    cross_val = cross_val_score(model, titanic_X, titanic_y,cv=kFolds,scoring= scoringFN).mean()\n    print(\"{0:30} {1} Fold Cross-Validation Accuracy: {2:7f}\".format(mod,kFolds,cross_val))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1eb074b97481ceabe60ec2d091bfe8aaec56cc3c"},"cell_type":"markdown","source":"From the results displayed above we can see that the gradient boosting method achieved the best results with the default hyperparameters. We will now adjust the parameters of the gradient boosting model and analyze the resulting learning curves."},{"metadata":{"_uuid":"9c4921a4ed27e5e17968bc619f76f7cc6b229ff0","collapsed":true,"trusted":false},"cell_type":"code","source":"# Source: http://scikit-learn.org/stable/auto_examples/model_selection/plot_learning_curve.html\ndef plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,\n                        n_jobs=1, train_sizes=np.linspace(.1, 1.0, 5)):\n    \n    plt.figure(figsize=(20,12))\n    plt.title(title,size=30)\n    if ylim is not None:\n        plt.ylim(*ylim)\n    plt.xlabel(\"Training examples\",size=15)\n    plt.ylabel(\"Score\",size=15)\n    train_sizes, train_scores, test_scores = learning_curve(\n        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)\n    train_scores_mean = np.mean(train_scores, axis=1)\n    train_scores_std = np.std(train_scores, axis=1)\n    test_scores_mean = np.mean(test_scores, axis=1)\n    test_scores_std = np.std(test_scores, axis=1)\n    plt.grid()\n\n    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,\n                     train_scores_mean + train_scores_std, alpha=0.1,\n                     color=\"r\")\n    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,\n                     test_scores_mean + test_scores_std, alpha=0.1, color=\"g\")\n    plt.plot(train_sizes, train_scores_mean, 'o-', color=\"r\",\n             label=\"Training score\")\n    plt.plot(train_sizes, test_scores_mean, 'o-', color=\"g\",\n             label=\"Cross-validation score\")\n\n    plt.legend(loc=\"best\")\n    return plt\n\n# Gradient Boosting Classifier Parameters (Partial List)\nlearning_rate     = 0.1\nn_estimators      = 100\nmax_depth         = 3\nmin_samples_split = 2\nsubsample         = 1\n\n# Define the gradient boosting classifier. \ngradBoost = GradientBoostingClassifier(learning_rate=learning_rate,n_estimators=n_estimators,\n                                       max_depth=max_depth,min_samples_split=min_samples_split,subsample=subsample)\n\n# Plot the learning curve of the gradient boosting classifier. \nplot_learning_curve(gradBoost,\"Gradient Boosting Learning Curve\",titanic_X,titanic_y,cv=5).show()\n\n# Fit model on the full dataset. \ngradBoost.fit(titanic_X,titanic_y)","execution_count":10,"outputs":[]},{"metadata":{"_uuid":"eab157f512ec8160cf8d1109faa904cd54704336"},"cell_type":"markdown","source":"## 6. Simple Feed-Forward Neural Network\nWe begin our neural network adventure by defining a very simple five layer feed-forward neural network. The architecture is as follows: \n\n1. Input Layer: 12 inputs.\n2. Hidden Layer: 20 sigmoid neurons. \n3. Hidden Layer: 20 sigmoid neurons. \n4. Hidden Layer: 20 sigmoid neurons. \n5. Output Layer: 1 sigmoid neuron. \n\nWe will be using the [adam optimizer](https://www.coursera.org/learn/deep-neural-network/lecture/w9VCZ/adam-optimization-algorithm) with its default keras parameters and the [binary crossentropy](https://en.wikipedia.org/wiki/Cross_entropy) loss function. While a good understanding of the optimizer and loss function is essential for deep learning research, for our purposes we will pass over these details. \n\nWe will also use the classification accuracy metric to evaluate how our network is performing: \n$$\\huge\\frac{1}{N}\\sum_{i=1}^N [\\hat{y}_i == r(y_i)] $$\n$$\n\\huge r(x) = \n\\left\\{\n\\begin{array}{ll}\n      0 & x\\leq 0.5 \\\\\n      1 & otherwise \\\\\n\\end{array} \n\\right. \n$$"},{"metadata":{"_uuid":"79dda595910cd2320cc64e931df0fb5efe263341","collapsed":true,"trusted":false},"cell_type":"code","source":"# Network Definition.\n# Here we use the keras functional api.\ninputs = Input(shape=(titanic_train_x.shape[1],),name=\"input\")\nx = Dense(20,activation=\"sigmoid\")(inputs)\nx = Dense(20,activation=\"sigmoid\")(x)\nx = Dense(20,activation=\"sigmoid\")(x)\nout = Dense(1,activation=\"sigmoid\", name=\"output\")(x)\n\n# Instantiate the network.\nsimpleModel = Model(inputs=inputs, outputs=out)\n\n# Compile the network. \nsimpleModel.compile(optimizer=\"adam\",loss=\"binary_crossentropy\", metrics=['acc'])\n\n# Pretty print the details of the network. \nsimpleModel.summary()","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"31b743d36deb968a74209612ac181b2394b57684"},"cell_type":"markdown","source":"** Workshop question: ** Why does the first dense layer have 260 parameters? Why does the second layer have 420 parameters? Explain why we are seeing this number of parameters. \n\nWe will now train the network on the training split and validate on the validation split."},{"metadata":{"_uuid":"c16fb0f27356e8063ce77f7274c2c83ac3039864","collapsed":true,"trusted":false},"cell_type":"code","source":"hist = simpleModel.fit(titanic_train_x, titanic_train_y,validation_data=(titanic_validation_x,titanic_validation_y), batch_size=30,epochs=30, verbose=1)","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"28253b8f2626fed4494d3e1aa54ac90778b7e3cc","collapsed":true,"trusted":false},"cell_type":"code","source":"def learningCurves(hist):\n    histAcc_train = hist.history['acc']\n    histLoss_train = hist.history['loss']\n    histAcc_validation = hist.history['val_acc']\n    histLoss_validation = hist.history['val_loss']\n    maxValAcc = np.max(histAcc_validation)\n    minValLoss = np.min(histLoss_validation)\n\n    plt.figure(figsize=(12,12))\n    epochs = len(histAcc_train)\n    plt.plot(range(epochs),np.full(epochs,meanBaseline(titanicDF)),label=\"Unbiased Estimator\", color=\"red\")\n\n    plt.plot(range(epochs),histLoss_train, label=\"Training Loss\", color=\"#acc6ef\")\n    plt.plot(range(epochs),histAcc_train, label=\"Training Accuracy\", color = \"#005ff9\" )\n\n    plt.plot(range(epochs),histLoss_validation, label=\"Validation Loss\", color=\"#a7e295\")\n    plt.plot(range(epochs),histAcc_validation, label=\"Validation Accuracy\",color=\"#3ddd0d\")\n\n    plt.scatter(np.argmax(histAcc_validation),maxValAcc,zorder=10,color=\"green\")\n    plt.scatter(np.argmin(histLoss_validation),minValLoss,zorder=10,color=\"green\")\n\n    plt.xlabel('Epochs',fontsize=14)\n    plt.title(\"Learning Curves\",fontsize=20)\n\n    plt.legend()\n    plt.show()\n\n    print(\"Max validation accuracy: {0}\".format(maxValAcc))\n    print(\"Minimum validation loss: {0}\".format(minValLoss))\n    \nlearningCurves(hist)","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"135bd757144eb562a2aea01738b9b6cf4c8da618"},"cell_type":"markdown","source":"** Workshop Questions**  \n* Do the learning curves indicate that we have a good model? What would make a good model? \n* Can you explain the difference between the loss and accuracy curves? Why are there periods where the loss is decreasing, but the accuracy is not? \n* Depending on the train/validation split you may see the validation loss decrease faster than the training loss. What would explain this behavior? "},{"metadata":{"_uuid":"e6fb019b86212deae0b1d50f0e3902e118f153d3"},"cell_type":"markdown","source":"## 7. Adjustable Feed-Forward Neural Network\nIn this section you will experiment with different neural network architectures and evaluate the resulting learning curves. \n\n**Workshop Activity** \n\nAdjust the hyperparameters below and evaluate how the impact the learning curves. "},{"metadata":{"_uuid":"e99285595aedcfc03b4a6695b0bbbec1800d068b","collapsed":true,"trusted":false},"cell_type":"code","source":"#### Training Hyperparameters ####\nbatch_size = 300\nepochs = 1000\n\n#### Model Hyperparameters  ####\nnLayers = 3\nlayerSize = 80\ndropoutPercent = 0.87# Regularization \n\n# Possible loss fuctions: https://keras.io/losses/\n# mean_squared_error, mean_absolute_error,mean_absolute_percentage_error,mean_squared_logarithmic_error\n# squared_hinge, hinge, categorical_hinge, logcosh, kullback_leibler_divergence, poisson, cosine_proximity\nlossFn = 'binary_crossentropy'\n\n# Possible optimizers: https://keras.io/optimizers/\n# SGD, RMSprop, Adagrad, Adadelta, Adamax, Nadam\noptimizer = 'adam'\n\n# Possible Activation Functions: https://keras.io/activations/\n# elu, selu, softplus, softsign, relu, tanh, hard_sigmoid, linear\n# Possible Advanced Activations: https://keras.io/layers/advanced-activations/\n# LeakyReLU, PReLU, ELU, ThresholdedReLU\nactivationFn = 'sigmoid'","execution_count":25,"outputs":[]},{"metadata":{"_uuid":"2a154d8ca92ef560a199d7ea16c668eec45dc2fe"},"cell_type":"markdown","source":"Here we have a more sophisticated way of constructing networks parameterized by architectural hyperparameters you set above. "},{"metadata":{"_uuid":"9877ada87fab6e9a80edf2c0d0c27b9eec1fa081","collapsed":true,"trusted":false},"cell_type":"code","source":"# Model Architecture \ndef makeModel(inputShape,nLayers,layerSize,dropoutPercent,lossFn,optimizer):\n    inputs = Input(shape=(inputShape,),name=\"input\")\n    x = None \n    \n    for layer in range(nLayers):\n        if x == None:\n            x = inputs\n\n        x = Dense(layerSize, activation=activationFn,name=\"fc\"+str(layer))(x)\n        x = Dropout(dropoutPercent,name=\"fc_dropout_\"+str(layer))(x)\n\n    out = Dense(1,activation=\"sigmoid\", name=\"output\")(x)\n\n    model = Model(inputs=inputs, outputs=out)\n    model.compile(optimizer=optimizer,\n                  loss=lossFn,\n                  metrics=['acc'])\n    \n    return model\n\nmodelMain = makeModel(titanic_train_x.shape[1],nLayers,layerSize,dropoutPercent,lossFn,optimizer)\nmodelMain.summary()","execution_count":26,"outputs":[]},{"metadata":{"_uuid":"4ea98b53ab27f61444ef57849fa73760dee27618","collapsed":true,"trusted":false},"cell_type":"code","source":"hist = modelMain.fit(titanic_train_x, titanic_train_y,validation_data=(titanic_validation_x,titanic_validation_y), batch_size=batch_size,epochs=epochs, verbose=0)\nlearningCurves(hist)","execution_count":27,"outputs":[]},{"metadata":{"_uuid":"ae800aeb0408a39023fc486dfe7b798a3cf44282"},"cell_type":"markdown","source":"** Workshop Questions**  \n* Do the learning curves indicate that we have a good model? What would make a good model? \n* Can you explain the difference between the loss and accuracy curves? Why are there periods where the loss is decreasing, but the accuracy is not? \n* Depending on the train/validation split you may see the validation loss decrease faster than the training loss. What would explain this behavior? "},{"metadata":{"_uuid":"81b845beff86d8a8f7eee53988b46c650dd5e5f9"},"cell_type":"markdown","source":"## 8. Cross-Validation\nIf our learning curve indicates healthy learning ( i.e. the training and validation metrics do not diverge ), we may want to use a cross-validation method in order to get a more accurate estimation of how well our model generalizes to new data. We will use a technique called called k-fold cross-validation"},{"metadata":{"_uuid":"4d9654e72daebf20773ccbc754341747a56c0d06","collapsed":true,"trusted":false},"cell_type":"code","source":" # Cross-Validation Parameter \nkFolds = 3\n\nkfold = StratifiedKFold(n_splits=kFolds, shuffle=True)\nmeans = []\nstds = []\nlossesLs = []\naccuracyLs = []\n\nrunningLoss = []\nrunningAccuracy = []\n\n# Train on k-folds of the data. \nfor train, test in kfold.split(titanic_X, titanic_y):\n    \n    # Create new instance of our model. \n    model = makeModel(titanic_X.shape[1],nLayers,layerSize,dropoutPercent,lossFn,optimizer)\n    \n    # Train the model on this kfold. \n    model.fit(titanic_X[train], titanic_y[train],batch_size=batch_size,epochs=epochs, verbose=0)\n\n    # Evaluate the model\n    loss,acc = model.evaluate(titanic_X[test], titanic_y[test], verbose=0)\n    \n    # Log Cross-Validation Data\n    lossesLs.append(loss)\n    accuracyLs.append(acc)\n    mean = np.mean(lossesLs)\n    std = np.std(lossesLs)\n    \n    accuracyMean = np.mean(accuracyLs)\n    accuracyStd = np.std(accuracyLs)\n    \n    runningLoss.append(mean)\n    runningAccuracy.append(accuracyMean)\n    \n    print(\"Loss: %.2f%% (+/- %.2f%%) | Accuracy: %.2f%% (+/- %.2f%%)\" % (mean*100,std,accuracyMean*100,accuracyStd))\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f291cf517871b190ffcf50cd2ad18c356f163dee"},"cell_type":"markdown","source":"**Workshop Questions** \n* Should we use more k-folds? How would you make this decision? \n* Does the cross-validated accuracy and loss differ from the validation/training metrics above? Can you explain why the are different or the same? "},{"metadata":{"_uuid":"74d7820e4593ec0f4376ac4f1d8dbe8fabbc3daa","collapsed":true},"cell_type":"markdown","source":"## 9. Application to Test Set\nNow that we have designed, trained, and evaluated our neural network, we will apply it to the training set and save the results to a csv file which can be uploaded to kaggle for scoring."},{"metadata":{"_uuid":"454a72c836fc24786c83c474183fbdb1b44bfa3a","collapsed":true,"trusted":false},"cell_type":"code","source":"# The thrshold function which assigns a class of 0 or 1 based on the sigmoid output of the network. \ndef thresholdFn(x):\n    if(x < 0.5):\n        return 0\n    else:\n        return 1\n\npred = modelMain.predict(np.stack(testDF.apply(preprocessRow,axis=1)))\n    \n# Save the predictions to a CSV file in the format suitable for the competition. \ndata_to_submit = pd.DataFrame.from_items([\n    ('PassengerId',testDF[\"PassengerId\"]),\n    ('Survived', pd.Series(np.hstack(pred)).map(thresholdFn))])\n\ndata_to_submit.to_csv('neuralNet.csv', index = False)","execution_count":23,"outputs":[]},{"metadata":{"_uuid":"fe82f39f84fa0a91827257575f9b545c1dd57dd6"},"cell_type":"markdown","source":"## 10. Next Steps\nThis notebook introduced a simple workflow of the stages involved in modeing categorical and categorical tabular data with neural networks. Here are some of the next steps you can take to make your analysis more robust:\n\n* Implement a hyperparameter search procedure. A common approach is grid search.\n* Add more features from the training set. In this notebook we ignore several columns like name and cabin. Figuring out how to include these features may help make better predictions. \n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}