{"cells":[ { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": "#!/usr/bin/env python3\n'''\nRandomforest - plot: # of trees - accuracy for various training set sizes\n - plot: training set size - accuracy\nModifications by: Noah Carter\nDate 4/25/16\nOriginal Author: Hideki Ikeda\nDate 7/11/15\nDesign: Uses an outer loop of training set size and an inner loop of forest size to capture the performance data\nfor each model; outputs the results;\nthen, outputs the graph of the highest forest size's performance for different training set sizes\n'''\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.cross_validation import cross_val_score\n\ndef main():\n '''\n Define: Reads the training data and allows for the initialization of a few commonly changed variables and parameters. Calls performTests(...)\n Design: axs is a container which keeps track of figures which will display once the program finishes.\n n_trees is the set of forest sizes. ADJUSTING n_trees TO SMALLER OR FEWER ELEMENTS WILL EXPEDITE THE PROGRAM \n '''\n # loading training data\n print('Loading training data')\n data = pd.read_csv('../input/train.csv')\n fig,axs=plt.subplots(1,2,figsize=(20,14))\n #n_trees = [10, 15, 20, 25, 30, 40]#, 50, 70, 100, 150]\n #I elected to use the below forest sizes (I couldn't use all sizes due to the 1200 second time cap)\n #n_trees=[10,50,100]\n #however, FOR DEMONSTRATION, THESE FOREST SIZES ARE USED\n n_trees=[10,20,25]\n performTests(data,42000,18,n_trees,False,True,axs)\ndef plotThisForestAccuracy(n_trees,scores,sc_array,std_array,showStd,axs):\n '''\n Define: Plots classificatory accuracy for the model which was built from by largest random forest as \n a function of training set size.\n Design: n_trees: the number of trees in the forest; \n scores: the scores as a list;sc_array: scores as a numpy array;\n std_array: standard deviations of scores as numpy array;showStd: whether or not graph should show lines for\n standard deviations; axs:the object containing the figures to show\n ''' \n pd.Series(scores, index=n_trees).plot(ax=axs[0])\n if (showStd):\n pd.Series(sc_array + std_array,index=n_trees).plot(style=\"b--\",ax=axs[0])\n pd.Series(sc_array - std_array,index=n_trees).plot(style=\"b--\",ax=axs[0])\n axs[0].set_xlabel(\"# of trees\")\n axs[0].set_ylabel(\"CV score\")\n axs[0].set_title(\"Forest Size vs. Average Accuracy for Varying Training Set Sizes\")\n ###plt.savefig('cv_trees.png')\ndef plotByTrainingSetSize(n_obsInTrainingSet,scores,sc_array,std_array,showStd,axs):\n '''\n Define: Plots classificatory accuracy for the model which was built from the largest random forest as \n a function of training set size.\n Design: n_obsInTrainingSet: the number of observations in the training set; \n scores: the scores as a list;sc_array: scores as a numpy array;\n std_array: standard deviations of scores as numpy array;showStd: whether or not graph should show lines for\n standard deviations; axs:the object containing the figures to show\n Document: Creates and modifies a plot. Conditionally plots the standard deviations. \n '''\n pd.Series(scores, index=n_obsInTrainingSet).plot(ax=axs[1])\n if (showStd):\n pd.Series(sc_array + std_array,index=n_obsInTrainingSet).plot(style=\"b--\",ax=axs[1])\n pd.Series(sc_array - std_array,index=n_obsInTrainingSet).plot(style=\"b--\",ax=axs[1])\n axs[1].set_xlabel(\"Size of Training Set\")\n axs[1].set_ylabel(\"CV score\")\n axs[1].set_title(\"Training Set Size vs. Average Accuracy for Largest Forest Size\")\n #######plt.savefig('cv_trees2.png')\n \ndef trainAndTest(n_trees,X_tr,y_tr,scores,scores_std):\n '''\n Define: Builds each tree in each forest, tests them, and reports the score data by appending to the parameters\n Design: n_trees: a list of the sizes of each forest to train; \n X_tr: a result of the \"values\" method on the data, contains the class for each observation in the training set;\n y_tr: a result of the \"values\" method on the data, contains the pixel fields for each observation in the training set;\n scores: a list which will hold the scores for each forest;\n scores_std: a list which will hold the stds for the scores of each forest\n '''\n for n_tree in n_trees:\n print(n_tree)\n\n recognizer = RandomForestClassifier(n_tree)\n score = cross_val_score(recognizer, X_tr, y_tr)\n scores.append(np.mean(score))\n scores_std.append(np.std(score))\ndef performTests(data,numberOfObservations,numberOfDivisions,n_trees,showStd1,showStd2,axs):\n '''\n Define: Manages the other functions and the general procedure\n Design: data: the numpy dataframe;\n numberOfObservations: number of observations to recognize in the data;\n numberOfDivisions: desired number of different training set sizes;\n n_trees: list of different forest sizes to build;\n showStd1: whether or not to show the lines for the standard deviations on the first graph;\n showStd2: whether or not to show the lines for the standard deviations on the second graph;\n axs: the object containing the figures to show;\n Document: Calculates the positions of the breaks for the different training set sizes.\n In a loop, gets the performance data for each training set for each of the forests.\n In this loop, also calls for the display and plotting of the performances\n Finally, prints and plots the performance of the largest forest and calls for the printing of slopes.\n '''\n sizeOfGap=int(numberOfObservations/numberOfDivisions)\n \n scoresListByTrainingSet=list()\n stdListByTrainingSet=list()\n n_obsInTrainingSet=list(range(sizeOfGap,numberOfObservations+1,sizeOfGap))\n #f1=plt.figure() #create a new figure\n #get the data and plot by number of trees\n for i in range(sizeOfGap,numberOfObservations+1,sizeOfGap):\n #note that the values in the training set are not selected randomly, but sequentially\n X_tr = data.values[:i, 1:].astype(float)\n y_tr = data.values[:i, 0]\n \n scores = list()\n scores_std = list()\n \n print('Start learning for training set of size ',i,'...')\n \n trainAndTest(n_trees,X_tr,y_tr,scores,scores_std)\n \n sc_array = np.array(scores)\n std_array = np.array(scores_std)\n print('Scores: ', sc_array)\n print('Stds : ', std_array)\n \n plotThisForestAccuracy(n_trees,scores,sc_array,std_array,showStd1,axs)\n \n scoresListByTrainingSet.append(scores[len(n_trees)-1]) #store the mean and std performance\n stdListByTrainingSet.append(scores_std[len(n_trees)-1]) #for the largest forest size on this\n #training set size\n #use the data to plot by training set size for the largest forest\n scoresByTrainingSet=np.array(scoresListByTrainingSet)\n stdByTrainingSet=np.array(stdListByTrainingSet)\n print(\"Scores of Highest Forest Size: \",scoresByTrainingSet)\n print(\"Standard deviations of Scores of Highest Forest Size: \",stdByTrainingSet)\n plotByTrainingSetSize(n_obsInTrainingSet,scoresListByTrainingSet,scoresByTrainingSet,stdByTrainingSet,showStd2,axs)\n \n printSlopes(range(sizeOfGap,numberOfObservations+1,sizeOfGap),scoresListByTrainingSet,axs)\ndef printSlopes(xList,yList,axs):\n '''\n Define: Calculates and prints slopes\n Design: If the coordinates are not equal in number, breaks. Otherwise, loops through the coordinates to\n calculate slopes. Prints the slopes.\n Document: xList: list of x coordinates, yList: list of y coordinates,\n axs: the object containing the figures to show\n '''\n slopeList=list()\n if (len(xList)!=len(yList)):\n print (\"Lists must be of same length.\")\n return\n for i in range(0,len(xList)):\n if(i+1