{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#  Disclaimer\n\nHi all, \n\nI am just learning how this works and this is the first competition that I have tried while learning some actual ML models\n(I've done others with just some linear regression models to practice those in the past).  So far I have only gone through \nthe intro course in Machine Learning and have learned Decision Trees and Random Forests.  I've also tried an XGBoost algorithm\nbut I don't know enough to make it better than the Random Forests yet.  Hopefull this notebook can give others ideas on how \nI've implemented a (suboptimal) solution.  Also, any feedback at all on how to improve my workflow or code would be greatly valued.\n\nThanks!  ","metadata":{}},{"cell_type":"markdown","source":"# Setup Libraries and Environment\n\nWe'll start with loading the different libraries that we are likely to use.\n\nIn this case, the following libraries are called:<br />\n1.)  numpy  -  This library will give us mathematical processes<br />\n2.)  pandas - This library will allow us to manage the data<br />\n3.)  sklearn - This library will provide us with the machine learning models<br /> \n4.)  seaborn -  This library will help with investigation of features and exploration of data<br />\n5.)  matplotlib.pyplot  -  This will give us the plotting capabilities from the matplot library\n\nFurthermore, we will check the availability of the data and report the file-paths for the data in this project.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\n\n#import sklearn as ...  #Which methods from the SK library will be determined later.\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.ensemble import RandomForestRegressor \nfrom sklearn.model_selection import RepeatedKFold\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.feature_selection import RFE, SelectFromModel\nfrom sklearn.model_selection import RandomizedSearchCV\nfrom sklearn.model_selection import GridSearchCV\n\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-06-11T05:22:30.089771Z","iopub.execute_input":"2022-06-11T05:22:30.090276Z","iopub.status.idle":"2022-06-11T05:22:30.111891Z","shell.execute_reply.started":"2022-06-11T05:22:30.090229Z","shell.execute_reply":"2022-06-11T05:22:30.110878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#  Read Datasets\n\nNext up, we will read in the datasets.","metadata":{}},{"cell_type":"code","source":"train_data_filepath = \"/kaggle/input/challenge-datascience-wind-turbine-cs/training_set.csv\" #store the file-path for the training data\ntest_data_filepath = \"/kaggle/input/challenge-datascience-wind-turbine-cs/test_set.csv\" #store the file-path for the test data\n\n# Uncomment when first running or to reload original data\ntraining_data = pd.read_csv(train_data_filepath)  #read the training data in and store as training_data\nprint(\"Training Data Read In\")\n\n################  Uncomment the following lines to report the first few rows of the data and shape of the data frame.\n#training_data.head() #print the first 5 rows of the training data\n#print(training_data.shape)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:22:33.403304Z","iopub.execute_input":"2022-06-11T05:22:33.40416Z","iopub.status.idle":"2022-06-11T05:22:42.503221Z","shell.execute_reply.started":"2022-06-11T05:22:33.404102Z","shell.execute_reply":"2022-06-11T05:22:42.501973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will also do the same for the test data","metadata":{}},{"cell_type":"code","source":"test_data = pd.read_csv(test_data_filepath)  #read the test data and store as test_data\nprint(\"Test Data Read In\")\n\n######################  Uncomment the following lines to report the first few rows of the data and shape of the data frame.\ntest_data.head()\n#test_data.shape","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:22:42.504838Z","iopub.execute_input":"2022-06-11T05:22:42.505224Z","iopub.status.idle":"2022-06-11T05:22:46.106593Z","shell.execute_reply.started":"2022-06-11T05:22:42.505191Z","shell.execute_reply":"2022-06-11T05:22:46.10558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When investigating the data, MAC_CODE is a categorical variable for Wind Turbines 1 through 4.\nWe will create a dummy variable in both the training and test datasets to deal with this.\n\nFor this one, we are going to use the one-hot-encoding which provides three columns where the values are 000, 100, 010, or 001\ndepending on which variable is true for the category.","metadata":{}},{"cell_type":"code","source":"#Working with the training data\ndummies = pd.get_dummies(training_data['MAC_CODE'], drop_first=True)  #Create a dummy variable.\ntraining_data = pd.concat([training_data, dummies], axis = 1)  #Concatonate the data into a single dataframe with the last three columns representing the dummy variable\n\n#Doing the same thing to the test data\ndummies = pd.get_dummies(test_data['MAC_CODE'], drop_first=True)  #Create the dummy variable\ntest_data = pd.concat([test_data, dummies], axis = 1)  #Concatonate the data into a single dataframe with the last three columns representing the dummy variable\n\nprint('Dummy Variables Created')","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:22:48.967498Z","iopub.execute_input":"2022-06-11T05:22:48.967875Z","iopub.status.idle":"2022-06-11T05:22:49.191978Z","shell.execute_reply.started":"2022-06-11T05:22:48.967844Z","shell.execute_reply":"2022-06-11T05:22:49.190945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we need to drop the MAC_CODE variable from both data sets so we can apply regression methods.","metadata":{}},{"cell_type":"code","source":"training_data = training_data.drop(['MAC_CODE'], axis=1)  #Removes the MAC_CODE variable from the dataframe\ntest_data = test_data.drop(['MAC_CODE'], axis=1)\n\nprint('Categorical Variable Dropped')","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:22:54.079221Z","iopub.execute_input":"2022-06-11T05:22:54.079607Z","iopub.status.idle":"2022-06-11T05:22:54.400691Z","shell.execute_reply.started":"2022-06-11T05:22:54.079578Z","shell.execute_reply":"2022-06-11T05:22:54.399635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Summary, Visualization, and Data Cleaning\n\nSince the data has now been read in, we can now investigate the training set.\n","metadata":{}},{"cell_type":"markdown","source":"## Step 1:  NA Removal/Imputation\nFind missing values in test_data and remove/impute those columns in test and train so that you won't use these variables.","metadata":{}},{"cell_type":"code","source":"test_data.isna().sum().sort_values(ascending=False).head(20)  #Identify Variables with missing values","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:22:57.999162Z","iopub.execute_input":"2022-06-11T05:22:57.999546Z","iopub.status.idle":"2022-06-11T05:22:58.047543Z","shell.execute_reply.started":"2022-06-11T05:22:57.999516Z","shell.execute_reply":"2022-06-11T05:22:58.046519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#I can't get the following line to work...  Any help?\n#NA_List = [col for col in test_data.columns if test_data[col].isna().any()]  #Create a list of all variables with NA in them  \n\nNA_List = ['Grid_voltage_std',    #An alternate method would be to do this by hand...\n           'Grid_voltage_max', \n           'Grid_voltage_min', \n           'Grid_voltage',\n           'Gearbox_inlet_temperature',\n           'Generator_converter_speed_max',\n           'Gearbox_inlet_temperature_std',\n           'Gearbox_inlet_temperature_max',\n           'Gearbox_inlet_temperature_min',\n           'Generator_converter_speed_std',\n           'Generator_converter_speed',\n           'Generator_converter_speed_min',\n           'Absolute_wind_direction_c',\n           'Nacelle_angle_c']\n\nprint('List of columns contaning NA created')","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:23:01.633255Z","iopub.execute_input":"2022-06-11T05:23:01.633619Z","iopub.status.idle":"2022-06-11T05:23:01.640394Z","shell.execute_reply.started":"2022-06-11T05:23:01.63359Z","shell.execute_reply":"2022-06-11T05:23:01.639094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### OPTION 1:  REMOVE Columns with NA's","metadata":{}},{"cell_type":"code","source":"#########  Remove columns in both the training and test datasets that could impact the results due to missing values.\ntraining_data = training_data.drop(NA_List, axis=1)\ntest_data = test_data.drop(NA_List, axis=1)\nprint('Removal Complete')","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:23:04.52753Z","iopub.execute_input":"2022-06-11T05:23:04.528281Z","iopub.status.idle":"2022-06-11T05:23:04.63935Z","shell.execute_reply.started":"2022-06-11T05:23:04.528236Z","shell.execute_reply":"2022-06-11T05:23:04.638106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### OPTION 2:  IMPUTE SIMPLE","metadata":{}},{"cell_type":"code","source":"#imp1 = SimpleImputer(missing_values=np.nan, strategy='mean')  #Set up the tool\n#imp2 = SimpleImputer(missing_values=np.nan, strategy='mean')\n\n#imp_training = imp1.fit(training_data[NA_List])  #Create an instance of this tool\n#imp_test = imp2.fit(test_data[NA_List])\n\n#training_data[NA_List] = imp_training.transform(training_data[NA_List])  #Apply the tool to the training data\n#test_data[NA_List] = imp_test.transform(test_data[NA_List])  #Note that we can't use the same tool because there are a different number of columns in the dataframes\n\n#type(training_data)  #Checking the type of the dataset that's left to make sure that it's still a panda's data frame and not an array.","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:16.715901Z","iopub.execute_input":"2022-06-11T02:23:16.716367Z","iopub.status.idle":"2022-06-11T02:23:16.721461Z","shell.execute_reply.started":"2022-06-11T02:23:16.716321Z","shell.execute_reply":"2022-06-11T02:23:16.720378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Making sure the NA's are delt with","metadata":{}},{"cell_type":"markdown","source":"As long as there are no more NA's in the data, the following should produce a list of all 0's","metadata":{}},{"cell_type":"code","source":"test_data.isna().sum().sort_values(ascending=False).head(20) #Here, we are checking to see if the test_data has the NA's dealt with","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:23:09.166934Z","iopub.execute_input":"2022-06-11T05:23:09.167498Z","iopub.status.idle":"2022-06-11T05:23:09.207192Z","shell.execute_reply.started":"2022-06-11T05:23:09.167452Z","shell.execute_reply":"2022-06-11T05:23:09.206227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_data.isna().sum().sort_values(ascending=False).head(20) #Let's do the same for the training data","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:23:11.835926Z","iopub.execute_input":"2022-06-11T05:23:11.837119Z","iopub.status.idle":"2022-06-11T05:23:11.905531Z","shell.execute_reply.started":"2022-06-11T05:23:11.837063Z","shell.execute_reply":"2022-06-11T05:23:11.904391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 2:  Investigate the Response Variable","metadata":{}},{"cell_type":"markdown","source":"Let's check out our goal.","metadata":{}},{"cell_type":"code","source":"training_data.TARGET.describe() #Getting a picture of the summary statistics for this data.","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:23:15.597097Z","iopub.execute_input":"2022-06-11T05:23:15.59749Z","iopub.status.idle":"2022-06-11T05:23:15.639326Z","shell.execute_reply.started":"2022-06-11T05:23:15.597459Z","shell.execute_reply":"2022-06-11T05:23:15.63845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_data.TARGET.isna().sum() #Find out how many NA values are in the response variable (Make sure these aren't a problem)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:16.882977Z","iopub.execute_input":"2022-06-11T02:23:16.883306Z","iopub.status.idle":"2022-06-11T02:23:16.891605Z","shell.execute_reply.started":"2022-06-11T02:23:16.883276Z","shell.execute_reply":"2022-06-11T02:23:16.890452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(training_data.TARGET)  #Let's look at a histogram of this data","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:16.893117Z","iopub.execute_input":"2022-06-11T02:23:16.893794Z","iopub.status.idle":"2022-06-11T02:23:18.125859Z","shell.execute_reply.started":"2022-06-11T02:23:16.893722Z","shell.execute_reply":"2022-06-11T02:23:18.124782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That histogram does not look good.  Let's zoom in a bit.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nsns.histplot(training_data.TARGET, ax=ax)\nax.set_xlim([-1,200])\n","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:18.127475Z","iopub.execute_input":"2022-06-11T02:23:18.127873Z","iopub.status.idle":"2022-06-11T02:23:19.00568Z","shell.execute_reply.started":"2022-06-11T02:23:18.127836Z","shell.execute_reply":"2022-06-11T02:23:19.004275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Maybe a bit more?","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nsns.histplot(training_data.TARGET, ax=ax, binwidth = 0.25)\nax.set_xlim([-1,2])","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:19.007249Z","iopub.execute_input":"2022-06-11T02:23:19.007764Z","iopub.status.idle":"2022-06-11T02:23:35.647393Z","shell.execute_reply.started":"2022-06-11T02:23:19.007718Z","shell.execute_reply":"2022-06-11T02:23:35.64625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Step 3:  Attend to any special features of the response variable","metadata":{}},{"cell_type":"markdown","source":"#  Investigate potential Features","metadata":{}},{"cell_type":"code","source":"training_data.columns # List the columns available to see features.","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:23:20.419933Z","iopub.execute_input":"2022-06-11T05:23:20.420345Z","iopub.status.idle":"2022-06-11T05:23:20.4279Z","shell.execute_reply.started":"2022-06-11T05:23:20.420314Z","shell.execute_reply":"2022-06-11T05:23:20.426872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"hm...  I have yet to do the feature engineering course to find out how to improve my models for random forests.  I know that order preserving transformations do very little to help improve a model, so I'll have to learn this later.  ","metadata":{}},{"cell_type":"markdown","source":"# Set Up Cross Validation framework","metadata":{}},{"cell_type":"markdown","source":"##  Step 1.)  Split Data","metadata":{}},{"cell_type":"markdown","source":"Let's start by getting the training data split into an X matrix and y vector.","metadata":{}},{"cell_type":"code","source":"training_data['Rotor_speed'] = training_data['Rotor_speed'] - training_data['Rotor_speed'].mean()\ntraining_data['Rotor_speed_sq'] = training_data['Rotor_speed'].pow(2)\n\ntest_data['Rotor_speed'] = test_data['Rotor_speed'] - test_data['Rotor_speed'].mean()\ntest_data['Rotor_speed_sq'] = test_data['Rotor_speed'].pow(2)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:23:25.1Z","iopub.execute_input":"2022-06-11T05:23:25.100439Z","iopub.status.idle":"2022-06-11T05:23:25.117571Z","shell.execute_reply.started":"2022-06-11T05:23:25.100406Z","shell.execute_reply":"2022-06-11T05:23:25.116444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y = training_data['TARGET']\nFeatures = training_data.drop(['TARGET'], axis=1).columns.values.tolist()\nX = training_data[Features]","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:23:28.307709Z","iopub.execute_input":"2022-06-11T05:23:28.308137Z","iopub.status.idle":"2022-06-11T05:23:28.610572Z","shell.execute_reply.started":"2022-06-11T05:23:28.308104Z","shell.execute_reply":"2022-06-11T05:23:28.609554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X.Rotor_speed_sq.describe()\n#X.head()  #This will allow us to see the first few rows of X\n#training_data.Rotor_speed.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:23:39.114104Z","iopub.execute_input":"2022-06-11T05:23:39.115028Z","iopub.status.idle":"2022-06-11T05:23:39.142695Z","shell.execute_reply.started":"2022-06-11T05:23:39.114973Z","shell.execute_reply":"2022-06-11T05:23:39.141673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y.head()  #This will allow us to see the first few rows of Y","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:15:00.952317Z","iopub.execute_input":"2022-06-11T05:15:00.952734Z","iopub.status.idle":"2022-06-11T05:15:00.960711Z","shell.execute_reply.started":"2022-06-11T05:15:00.9527Z","shell.execute_reply":"2022-06-11T05:15:00.959593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train and Test Models","metadata":{}},{"cell_type":"markdown","source":"## Direct Random Forest Model","metadata":{}},{"cell_type":"code","source":"#n_estimators = [int(x) for x in np.linspace(start = 90, stop=120, num=10)]\n#max_features = ['auto', 'sqrt']\n#max_depth = [int(x) for x in np.linspace(10,100, num = 11)]\n#max_depth.append(None)\n#min_samples_split = [2,5,10]\n#min_samples_leaf = [1,2,4]\n#bootstrap = [True, False]\n\n#random_grid = {'n_estimators': n_estimators,\n#               'max_features': max_features,\n#               'max_depth': max_depth,\n#               'min_samples_split': min_samples_split,\n#               'min_samples_leaf': min_samples_leaf,\n#               'bootstrap': bootstrap}\n#print(random_grid)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:35.861241Z","iopub.execute_input":"2022-06-11T02:23:35.862001Z","iopub.status.idle":"2022-06-11T02:23:35.870593Z","shell.execute_reply.started":"2022-06-11T02:23:35.861947Z","shell.execute_reply":"2022-06-11T02:23:35.869543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#rf = RandomForestRegressor()\n#rf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 50, cv = 3, verbose = 2, random_state = 42, n_jobs = -1)\n\n#rf_random.fit(X,y)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:35.87198Z","iopub.execute_input":"2022-06-11T02:23:35.872428Z","iopub.status.idle":"2022-06-11T02:23:35.883943Z","shell.execute_reply.started":"2022-06-11T02:23:35.87238Z","shell.execute_reply":"2022-06-11T02:23:35.882974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#rf_random.best_params_","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:35.88533Z","iopub.execute_input":"2022-06-11T02:23:35.885756Z","iopub.status.idle":"2022-06-11T02:23:35.897686Z","shell.execute_reply.started":"2022-06-11T02:23:35.885721Z","shell.execute_reply":"2022-06-11T02:23:35.896579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We're just going to try a random forest model on this one.  I've done a grid search on the hyper-perameters for my best model, but I'll just leave the setup here.","metadata":{}},{"cell_type":"markdown","source":"### Quick Validation train/test split","metadata":{}},{"cell_type":"code","source":"#Quick validation\n\n#X_train, X_valid, y_train, y_valid = train_test_split(X,y, test_size = 0.25, random_state=42)\n\n\nmodel_RF = RandomForestRegressor(n_estimators = 100, max_features = 22, max_depth = 110, min_samples_split = 8, min_samples_leaf = 1, bootstrap = False, n_jobs = -1, random_state = 42)  #Create the instance of the tool \n#model_RF.fit(X_train,y_train)\n#y_est = model_RF.predict(X_valid) #run the model on the X matrix to produce a target result\n#mae = mean_absolute_error(y_valid, y_est)\n#print(\"MAE: %.3f\" %mae)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:23:59.101652Z","iopub.execute_input":"2022-06-11T05:23:59.102066Z","iopub.status.idle":"2022-06-11T05:30:35.81634Z","shell.execute_reply.started":"2022-06-11T05:23:59.102034Z","shell.execute_reply":"2022-06-11T05:30:35.815111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#y_estimated = model_RF.predict(test_data)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T04:22:13.804892Z","iopub.execute_input":"2022-06-11T04:22:13.805384Z","iopub.status.idle":"2022-06-11T04:22:17.287364Z","shell.execute_reply.started":"2022-06-11T04:22:13.805347Z","shell.execute_reply":"2022-06-11T04:22:17.286518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Full Cross validation","metadata":{}},{"cell_type":"code","source":"#Create a set of scores from the cross validation model and then report the summary of the results.  \n#This is very helpful for a grid search because you can store the results, produce boxplots and determine where\n#the improvements stop.  These are currently commented out in order to keep computation time low for the notebook.\n\n#cv = RepeatedKFold(n_splits = 5, n_repeats = 3, random_state = 42)  #Set up the cross validation methods\n#n_scores = cross_val_score(model_RF, X, y, scoring = 'neg_mean_absolute_error', cv = cv, n_jobs=-1, error_score='raise')\n#print('MAE: %.3f (%.3f)' % (np.mean(n_scores), np.std(n_scores)))","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:52.021355Z","iopub.execute_input":"2022-06-11T02:23:52.022496Z","iopub.status.idle":"2022-06-11T02:23:52.028225Z","shell.execute_reply.started":"2022-06-11T02:23:52.022443Z","shell.execute_reply":"2022-06-11T02:23:52.027191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Features = [ 'ID',\n#             'Generator_speed', \n#             'Rotor_speed', \n#             'Pitch_angle_std', \n#             'Pitch_angle', \n#             'Date_time', \n#             'Generator_speed_max', \n#             'Pitch_angle_max', \n#             'Generator_stator_temperature',\n#             'Generator_bearing_1_temperature_max',\n#             'Generator_bearing_1_temperature',\n#             'Generator_speed_std',\n#             'Rotor_speed_max',\n#             'Generator_bearing_1_temperature_min']\n#X = training_data[Features]","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:52.029692Z","iopub.execute_input":"2022-06-11T02:23:52.030339Z","iopub.status.idle":"2022-06-11T02:23:52.03986Z","shell.execute_reply.started":"2022-06-11T02:23:52.030301Z","shell.execute_reply":"2022-06-11T02:23:52.039067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Quick validation\n\n#X_train, X_valid, y_train, y_valid = train_test_split(X,y, test_size = 0.25, random_state=42)\n#\n#model_RF.fit(X_train,y_train)\n#y_est = model_RF.predict(X_valid) #run the model on the X matrix to produce a target result\n#mae = mean_absolute_error(y_valid, y_est)\n#print(\"MAE: %.3f\" %mae)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:52.040889Z","iopub.execute_input":"2022-06-11T02:23:52.041563Z","iopub.status.idle":"2022-06-11T02:23:52.054214Z","shell.execute_reply.started":"2022-06-11T02:23:52.041524Z","shell.execute_reply":"2022-06-11T02:23:52.05304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##  Run Models on Testing Set","metadata":{}},{"cell_type":"code","source":"#model_RF = RandomForestRegressor(n_estimators = 60, n_jobs = -1, random_state = 42)  #Create the instance of the tool ","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:52.058068Z","iopub.execute_input":"2022-06-11T02:23:52.058847Z","iopub.status.idle":"2022-06-11T02:23:52.071052Z","shell.execute_reply.started":"2022-06-11T02:23:52.058807Z","shell.execute_reply":"2022-06-11T02:23:52.070243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#test_data = test_data[Features]\n#test_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:52.072694Z","iopub.execute_input":"2022-06-11T02:23:52.073315Z","iopub.status.idle":"2022-06-11T02:23:52.084538Z","shell.execute_reply.started":"2022-06-11T02:23:52.073274Z","shell.execute_reply":"2022-06-11T02:23:52.083717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Once you have a model that is optimal, fit the model to all of the data.\nmodel_RF.fit(X,y)\ny_estimated = model_RF.predict(test_data)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:31:12.708193Z","iopub.execute_input":"2022-06-11T05:31:12.708597Z","iopub.status.idle":"2022-06-11T05:39:51.620573Z","shell.execute_reply.started":"2022-06-11T05:31:12.708567Z","shell.execute_reply":"2022-06-11T05:39:51.619518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#If you want to see which factors were most considered in the trees in the result in terms of factor imporatance\n#you can run these and then plot a and b in a line-plot (b is the frequency, a is the names.)\n\n#importances = model_RF.feature_importances_\n\n#final_df = pd.DataFrame({\"Features\": X.columns, \"Importances\": importances})\n#final_df.set_index(\"Importances\")\n\n#final_df = final_df.sort_values(\"Importances\")\n\n#plt.figure(figsize = (20,3))\n#plt.xticks(rotation = 45)\n#sns.barplot(x=\"Features\", y = \"Importances\", data = final_df)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:52.100707Z","iopub.execute_input":"2022-06-11T02:23:52.101306Z","iopub.status.idle":"2022-06-11T02:23:52.112491Z","shell.execute_reply.started":"2022-06-11T02:23:52.101267Z","shell.execute_reply":"2022-06-11T02:23:52.111296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#final_df.tail(16)","metadata":{"execution":{"iopub.status.busy":"2022-06-11T02:23:52.113992Z","iopub.execute_input":"2022-06-11T02:23:52.114381Z","iopub.status.idle":"2022-06-11T02:23:52.126016Z","shell.execute_reply.started":"2022-06-11T02:23:52.114345Z","shell.execute_reply":"2022-06-11T02:23:52.125025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Submission File","metadata":{}},{"cell_type":"code","source":"#Creating a submission file requires pulling the resulting predicted values.\nSubmission= pd.DataFrame({\"ID\": test_data['ID'], \"TARGET\": y_estimated}) #ID is the other required column for the submission.\nSubmission.to_csv('/kaggle/working/submission', index = False)  #Storing the csv of the submission file.","metadata":{"execution":{"iopub.status.busy":"2022-06-11T05:40:14.443828Z","iopub.execute_input":"2022-06-11T05:40:14.444265Z","iopub.status.idle":"2022-06-11T05:40:15.078246Z","shell.execute_reply.started":"2022-06-11T05:40:14.444231Z","shell.execute_reply":"2022-06-11T05:40:15.077334Z"},"trusted":true},"execution_count":null,"outputs":[]}]}