{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Airplane delay: Analysis and Prediction\nGiven historical data about flights in the country, including the delay information that was computed a-posteriori (so the ground truth is available), we want to build a model that can be used to predict how many minutes of delay a flight might experience in the future. This model should provide useful information for the airport to manage better its resources, to minimize the delays and their impact on the journey of its passengers. Alternatively, astute passengers could even use the model to choose the best time for flying, such as to avoid delays.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"### 1. Data Pre-Processing\n>    #### 1.1 Numerical FEATURES \n>    #### 1.2 Categorical FEATURES \n>    #### 1.3 Semi Categorical FEATURES\n\n### 2. Prediction Models \n\n### 3. Improvement AND Model Selection","metadata":{}},{"cell_type":"code","source":"import sys\nimport json\nimport warnings\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.dates as mdates\nimport seaborn as sns\nimport tensorflow as tf\nimport numpy as np\n\nimport sklearn as skl\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.linear_model import Ridge\nfrom sklearn.preprocessing import StandardScaler\n\nplt.rcParams['figure.figsize'] = (12,12)\nif not sys.warnoptions:\n    warnings.simplefilter(\"ignore\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Pre-Processing \n>    ### 1.1 Numerical Features  ","metadata":{}},{"cell_type":"markdown","source":"First, we import the training and testing data. We will be using pandas library to manipulate and display data.","metadata":{}},{"cell_type":"code","source":"## Import Data \nanalysis_airports = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/airports.csv\",engine='python')\nanalysis_airlines = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/airlines.csv\",engine='python')\n\ntrain_ft = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_features.csv\",engine='python')\ntrain_label = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_targets.csv\",engine='python')\n\ntest_ft = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/test_features.csv\",engine='python')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Concat Data And Rename \nall_data = pd.concat([train_ft,train_label['ARRIVAL_DELAY']],axis=1)\ntrain_data = all_data.drop(columns=['ARRIVAL_DELAY'],axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_data.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This overview gave us an idea about the data structure we will be using as well as the different fields of the data set. We notice that there are so many features in this data set. So in the first part we will choose the most useful features to keep.\n\nWe start by dropping the '__ID__','__YEAR__' column which is totally useless for the prediction.","metadata":{}},{"cell_type":"code","source":"# Describe Numerical Features\nnumerical_fts_describe = all_data.describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Delete Semi-Numerical/Semi-Categorical Features\nnumerical_fts_describe = numerical_fts_describe.drop(['ID','MONTH','DAY','DAY_OF_WEEK','YEAR','CANCELLED','DIVERTED','CANCELLATION_REASON'],axis=1)\nnumerical_fts = numerical_fts_describe.columns\nnumerical_fts_describe","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1. Let's look for missing data, and fill Nan numerical Data\n2. The range of values of each column varies widely. So normalizing the data is an important step in order to avoid any scaling problem and also to speed up the training process (gradient descent will converge fast)","metadata":{}},{"cell_type":"code","source":"## Missing Values - Analysis \nall_data[numerical_fts].isna().sum().to_frame()\n\n#'''\n#FLIGHT_NUMBER          0\n#SCHEDULED_DEPARTURE    0\n#DEPARTURE_TIME         0\n#DEPARTURE_DELAY        0\n#TAXI_OUT               0\n#WHEELS_OFF             0\n#SCHEDULED_TIME         0\n#AIR_TIME               0\n#DISTANCE               0\n#SCHEDULED_ARRIVAL      0\n#ARRIVAL_DELAY          0\n#dtype: int64\n#'''\n\n#NO MISSING VALUES ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def fill_nan(df,fts) : # Fill Missing values with the mean \n    df[fts] = df[fts].fillna(df.mean())\n    return df\n\ndef normalize(df,df_refer,fts) : # Normalizing Data \n    df[fts] = (df[fts] - df_refer[fts].mean())/df_refer[fts].std()\n    \ndef preparation_num(df,df_refer,fts) : \n    # preparation of numerical data \n    # df2 = is the dataframe reference, so while normalizing the test set \n    # we use the mean and variance of the \"all_data\" ~ Train set \n    df = fill_nan(df,fts)\n    normalize(df,df_refer,fts)\n    return df \n\nnum_data = all_data[numerical_fts]\nnum_data = preparation_num(num_data,num_data,numerical_fts)\nnum_data.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Correlation Matrix\n\nAs we can see we have so many features describing the data samples. Probably not all of these feature are importatnt to predict our target which is the delay arrival.\n\nSo now, we are going to keep only columns which are relevent to this problem. So let's have a quick look to the correlation between different features using the correlation matrix.","metadata":{}},{"cell_type":"code","source":"corr_num = num_data[:1000000].corr()\ncorr_num = np.abs(corr_num)\nmask = np.zeros_like(corr_num)\nmask[np.triu_indices_from(mask)] = True\nwith sns.axes_style(\"white\"):\n    f, ax = plt.subplots(figsize=(8, 8))\n    ax = sns.heatmap(corr_num, mask=mask, square=True, cmap=\"YlGnBu\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This heatmap gave us a great overview of our data as well as the relationship between different features.\n\nWe notice that there are many darkblue-colored squares: there are obvious correlations such as between __AIR_TIME__, __SCHEDULED_TIME__, __DISTANCE__ and __WHEELS_OFF__, __SCHEDULED_DEPARTURE__, __DEPARTURE_TIME__\n\nThese columns give almost the same information, to make sure of the relation between correlated features, let's look to the Scatter plot. ","metadata":{}},{"cell_type":"code","source":"sns.pairplot(num_data[:1000000])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_data.FLIGHT_NUMBER.value_counts().to_frame().head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"__FLIGHT_NUMBER__ is very random feature, which is normal, because it is __Flight Identifier (Not Sequntial)__ and we can see that it is not a continuos feature, it's a semi numerical but it takes too numerous values (~ 6100) so due to our material limitation, we cannot consider it as categorical. ==> Get rid of __FLIGHT_NUMBER__. \n\nAs viewed with __correlation Matrix__, __Scatter plot__ shows us that there's relation between some features as (__WHEELS_OFF ,DEPARTURE_TIME__), (__SCHEDULED_TIME, AIR_TIME, DISTANCE__)","metadata":{}},{"cell_type":"markdown","source":"### 1.2. CATEGORICAL FEATURES","metadata":{}},{"cell_type":"code","source":"## we don't need To include CANCELLED as categorical features (same value for all)\nall_data.CANCELLED.value_counts().to_frame()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## we don't need To include DIVERTED as categorical features (same value for all)\nall_data.DIVERTED.value_counts().to_frame()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical_fts = ['AIRLINE','ORIGIN_AIRPORT','DESTINATION_AIRPORT']\nprint(\"Number of AIRLINE values is {}\".format(len(all_data['AIRLINE'].unique())))\nprint(\"Number of ORIGIN_AIRPORT values is {}\".format(len(all_data['ORIGIN_AIRPORT'].unique())))\nprint(\"Number of DESTINATION_AIRPORT values is {}\".format(len(all_data['DESTINATION_AIRPORT'].unique())))\nall_data[categorical_fts].head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## MISSING VALUES \ndata_cat = all_data[categorical_fts]\ndata_cat.isnull().sum().to_frame()\n\n## NO MISSING VALUES","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Correlation Matrix ","metadata":{}},{"cell_type":"code","source":"cat_corr = data_cat.apply(lambda x: x.factorize()[0]).corr()\nmask = np.zeros_like(cat_corr)\nmask[np.triu_indices_from(mask)] = True\nwith sns.axes_style(\"white\"):\n    f, ax = plt.subplots(figsize=(8, 8))\n    ax = sns.heatmap(cat_corr, mask=mask, square=True, cmap=\"YlGnBu\")\n    \n# There's almost no correlation between selected categorical features","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The common idea with the categorical features is to use the One Hot Encoding, but here we have a problem, __DESTINATION_AIRPORT__, and __ORIGIN_AIRPORT__, have total distinct values of __1257__, which makes our data set very large. \n\n__Solution__ : Instead of working with airports, we can work with there Coordinates of airports given in AIRPORTS file, so :\n> __DESTINATION_AIRPORT__ will be prensented by numerical features __LATITUDE_dst__, __LONGITUDE_dst__\n> __ORIGIN_AIRPORT__ will be prensented by numerical features __LATITUDE_org__, __LONGITUDE_org__\n","metadata":{}},{"cell_type":"code","source":"def cata_to_num(data_src) :\n    df = data_src\n    df = pd.merge(df,analysis_airports[['IATA_CODE','LATITUDE','LONGITUDE']] ,how='left' , left_on='ORIGIN_AIRPORT', right_on='IATA_CODE')\n    df = df.rename(columns={'LATITUDE':'LATITUDE_org','LONGITUDE':'LONGITUDE_org'}).drop(['IATA_CODE'],axis=1)\n    df = pd.merge(df,analysis_airports[['IATA_CODE','LATITUDE','LONGITUDE']] ,how='left' ,left_on='DESTINATION_AIRPORT', right_on='IATA_CODE')\n    df = df.rename(columns={'LATITUDE':'LATITUDE_dst','LONGITUDE':'LONGITUDE_dst'}).drop(['IATA_CODE'],axis=1)\n    return df\n\ndata_cat = cata_to_num(data_cat)\ndata_cat.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1.3 SEMI NUMERICAL || SEMI CATEGORICAL FEATURES","metadata":{}},{"cell_type":"code","source":"min_delay = train_label.ARRIVAL_DELAY.min()\nmean_delay = train_label.ARRIVAL_DELAY.mean()\nmax_delay = train_label.ARRIVAL_DELAY.max() \n\nprint(\"The min arrival delay is {}\".format(min_delay))\nprint(\"The mean arrival delay is {}\".format(mean_delay))\nprint(\"The max arrival delay is {}\".format(max_delay))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Analyse Relation between day and delay \ndays_delay = all_data.groupby('DAY').ARRIVAL_DELAY.agg(['mean'])\ndays = all_data.DAY.unique()\n\nsns.barplot(np.sort(days),days_delay['mean'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Analyse Relation between month and delay\nmonth_delay = all_data.groupby('MONTH').ARRIVAL_DELAY.agg(['mean'])\nmonth = all_data.MONTH.unique()\nsns.barplot(np.sort(month),month_delay['mean'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#analysing relation between day_of_week and the delay\nday_w_delay = all_data.groupby('DAY_OF_WEEK').ARRIVAL_DELAY.agg(['mean'])\nday_w = all_data.DAY_OF_WEEK.unique()\nsns.barplot(day_w,day_w_delay['mean'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We conclude that the arrival delay depends on the month, and on the day, according to the plots, a large average delay is completed during the end of the year period, and in summer (which can be explained by the increase in numbers of trips in these periods), and a significant variation in the arrival delay compared to the day, and the increase of the delay is seen in week-end days.\n\nThe limitation of the material, forces us to use these features when the prediction model allows it. because the one hot encoding results in 50 addtive features. ","metadata":{}},{"cell_type":"code","source":"def processing(data_df,data_ref,num_fts,cat_fts) :\n    df_copy = data_df\n    df_copy = fill_nan(df_copy,num_fts)\n    normalize(df_copy,data_ref,num_fts)\n    return df_copy\n\nclass pre_processing() :\n    \n    def __init__(self,data_df,data_ref,num_fts,cat_fts) :\n        self.data_df = data_df\n        self.data_ref = data_ref\n        self.num_fts = num_fts\n        self.cat_fts = cat_fts \n    \n    def pre_process(self) :\n        return processing(self.data_df,\n                          self.data_ref,\n                          self.num_fts,\n                          self.cat_fts)\n    \n    def add_label(self,_data_df,_data_ref) :\n        _data_df = pd.concat([_data_df,_data_ref.ARRIVAL_DELAY],axis=1)\n        return _data_df\n        \n    def add_id(self,_data_df,_data_ref) :\n        _data_df = pd.concat([_data_ref.ID,_data_df],axis=1)\n        return _data_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_ft_selected = ['DEPARTURE_DELAY','SCHEDULED_DEPARTURE','SCHEDULED_ARRIVAL','SCHEDULED_TIME','DEPARTURE_TIME','TAXI_OUT','WHEELS_OFF','AIR_TIME','DISTANCE','LATITUDE_dst','LATITUDE_org','LONGITUDE_dst','LONGITUDE_org']\ncat_ft_selected = [] ## AIRLINE will be done separtly \nall_fts = num_ft_selected + cat_ft_selected","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data = cata_to_num(all_data)\ny_data = train_data['ARRIVAL_DELAY']\ntest_data = cata_to_num(test_ft)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_preprocess = pre_processing(train_data[all_fts],train_data[all_fts],num_ft_selected,cat_ft_selected)\nprint(\"pass create object\")\ntrain_df = train_preprocess.pre_process()\nprint(\"pass function pre_process\")\ndf_cat = pd.get_dummies(train_data['AIRLINE'], dummy_na=False, drop_first=True)\ntrain_df = pd.concat([train_df,df_cat],axis=1)\nprint(\"#####################################################################################\")\n\ntest_preprocess = pre_processing(test_data[all_fts],train_data[all_fts],num_ft_selected,cat_ft_selected)\nprint(\"pass create object\")\ntest_df = test_preprocess.pre_process()\nprint(\"pass function pre_process\")\ndf_cat_test = pd.get_dummies(test_data['AIRLINE'], dummy_na=False, drop_first=True)\ntest_df = pd.concat([test_df,df_cat_test],axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Models Preparation ","metadata":{}},{"cell_type":"code","source":"from sklearn import datasets, linear_model\nfrom sklearn.metrics import mean_squared_error, r2_score\nfrom sklearn.preprocessing import PolynomialFeatures\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.pipeline import Pipeline\nimport numpy as np\nfrom sklearn.linear_model import ElasticNet, Lasso,  BayesianRidge, LassoLarsIC\nfrom sklearn.ensemble import RandomForestRegressor,  GradientBoostingRegressor\nfrom sklearn.kernel_ridge import KernelRidge\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.base import BaseEstimator, TransformerMixin, RegressorMixin, clone\nfrom sklearn.model_selection import KFold, cross_val_score, train_test_split\nfrom sklearn.metrics import mean_squared_error\nimport xgboost as xgb\nimport lightgbm as lgb\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_df = train_df[:12000] ## Training Set Tuning \ny_train_df = y_data[:12000]\nX_test_df = test_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# First of all, you should split the data into a training set and a validation set\nX_train_df, X_val_df, y_train_df, y_val_df = train_test_split(X_train_df, y_train_df, random_state=1, test_size=0.2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class create_model() :\n    def __init__(self, model, batch_size=100) :\n        self.model = model\n        self.batch_size = batch_size \n    \n    def make_prediction(self,X) :\n        X_len= len(X)\n        range_predict= [(i*X_len)//self.batch_size for i in range (1, self.batch_size +1) ]\n        y_pred= self.model.predict(X[:range_predict[0]])\n        for i in range (1,len(range_predict)):\n            y_pred_i= self.model.predict(X[range_predict[i-1]:range_predict[i]])\n            y_pred = np.concatenate((y_pred, y_pred_i),axis=0)\n        return y_pred\n    \n    def mmse_score(self,X,Y) :\n        return np.sqrt(mean_squared_error(self.make_prediction(X), Y.values))\n    \n    def make_submission(self,X_test_df_,name) :\n        y_test_pred = self.make_prediction(X_test_df_)\n        submission_df = pd.DataFrame(data={'ID': test_ft.ID.values,'ARRIVAL_DELAY': y_test_pred.squeeze()})\n        submission_df.to_csv(\"/kaggle/working/submission{}.csv\".format(name), index=False)\n        return \"DONE\"\n\n\n# Define validation function\nn_folds = 5\n\ndef rmsle_cv(model,X_train,y_train):\n    kf = KFold(n_folds, shuffle=True, random_state=42).get_n_splits(X_train)\n    rmse = np.sqrt(-cross_val_score(model, X_train, y_train, scoring=\"neg_mean_squared_error\", cv = kf))\n    score = np.mean(rmse)\n    return(score)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### a. Polynomial Regression degree = 2","metadata":{}},{"cell_type":"code","source":"# Parametric Models \n# Polynomial : degree 2\nfrom sklearn.pipeline import make_pipeline\npolyReg = Pipeline([('poly', PolynomialFeatures(degree=2)),('linear', LinearRegression(fit_intercept=False))])\npolyReg.fit(X_train_df, y_train_df)\npolyRegModel = create_model(polyReg)\nvalMMSE = polyRegModel.mmse_score(X_val_df,y_val_df)\ntrainMMSE = polyRegModel.mmse_score(X_train_df,y_train_df)\n\n\nprint(\"TRAIN SET MSE  is {}\".format(trainMMSE))\nprint(\"Validation SET MSE is {}\".format(valMMSE))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crossVal = rmsle_cv(polyReg,X_train_df,y_train_df)\nprint(\"Cross Validation Score is {}\".format(crossVal))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### b. Gaussian Process Regressor ","metadata":{}},{"cell_type":"code","source":"## Gaussian Process \nfrom sklearn.gaussian_process import GaussianProcessRegressor\nfrom sklearn.gaussian_process.kernels import DotProduct, WhiteKernel\n\n\nkernel = DotProduct() + WhiteKernel()\nGPR = GaussianProcessRegressor(kernel=kernel,random_state=0).fit(X_train_df, y_train_df)\nGPR_model = create_model(GPR)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valMMSE = GPR_model.mmse_score(X_val_df,y_val_df)\ntrainMMSE = GPR_model.mmse_score(X_train_df,y_train_df)\n\nprint(\"TRAIN SET MSE  is {}\".format(trainMMSE))\nprint(\"Validation SET MSE is {}\".format(valMMSE))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crossVal = rmsle_cv(GPR,X_train_df[:10000],y_train_df[:10000])\nprint(\"Cross Validation Score is {}\".format(crossVal))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### c. Elastic NET ","metadata":{}},{"cell_type":"code","source":"## ElasticNet\nENet = make_pipeline(RobustScaler(), ElasticNet(alpha=0.0005, l1_ratio=.9, random_state=3))\nENet.fit(X_train_df, y_train_df)\nENetModel = create_model(ENet)\nvalMMSE = ENetModel.mmse_score(X_val_df,y_val_df)\ntrainMMSE = ENetModel.mmse_score(X_train_df,y_train_df)\n\nprint(\"TRAIN SET MSE  is {}\".format(trainMMSE))\nprint(\"Validation SET MSE is {}\".format(valMMSE))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crossVal = rmsle_cv(ENet,X_train_df,y_train_df)\nprint(\"Cross Validation Score is {}\".format(crossVal))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### d. Random Forest Regressor ","metadata":{}},{"cell_type":"code","source":"#Random Forest regressor\nfrom sklearn.ensemble import RandomForestRegressor\nforest_reg = RandomForestRegressor()\nforest_reg.fit(X_train_df,y_train_df)\nFRModel = create_model(forest_reg)\nvalMMSE = FRModel.mmse_score(X_val_df,y_val_df)\ntrainMMSE = FRModel.mmse_score(X_train_df,y_train_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"TRAIN SET MSE  is {}\".format(trainMMSE))\nprint(\"Validation SET MSE is {}\".format(valMMSE))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crossVal = rmsle_cv(forest_reg,X_train_df,y_train_df)\nprint(\"Cross Validation Score is {}\".format(crossVal))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### e. KERNEL Ridge Regressor  (Polynomial Kernel)","metadata":{}},{"cell_type":"code","source":"# Kernel Ridge \nKRR = KernelRidge(alpha=0.6, kernel='polynomial', degree=2, coef0=2.5)\nKRR.fit(X_train_df, y_train_df)\nKRRModel = create_model(KRR)\nvalMMSE = KRRModel.mmse_score(X_val_df,y_val_df)\ntrainMMSE = KRRModel.mmse_score(X_train_df, y_train_df)\n\nprint(\"TRAIN SET MSE  is {}\".format(trainMMSE))\nprint(\"Validation SET MSE is {}\".format(valMMSE))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crossVal = rmsle_cv(KRR,X_train_df[:10000], y_train_df[:10000])\nprint(\"Cross Validation Score is {}\".format(crossVal))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Model Selection \nGood Models based on Cross Validation : ElasticNet , Kernel Ridge Regressor\nRandom Forest seems to predict well sometimes, and give us good mse score but it almost overfits. ","metadata":{}},{"cell_type":"markdown","source":"##### First Idea of Improvement : \nAveraged base models, use the out-of-folds predictions of these base models to train an average model.\nThe procedure, for the training part, may be described as follows:\n\n1. Train several base models on train Dataset \n2. Predict using Cross Validation Prediction\n3. Use those predictions as inputs of another prediction model (output is y_train)","metadata":{}},{"cell_type":"markdown","source":"### 3.1. Average Model ENet and Kernel Ridge Regressor","metadata":{}},{"cell_type":"code","source":"## Average Model\n\nfrom sklearn.model_selection import cross_val_predict\n\nENet = make_pipeline(RobustScaler(), ElasticNet(alpha=0.0005, l1_ratio=.9, random_state=3))\nKRR = KernelRidge(alpha=0.6, kernel='polynomial', degree=3, coef0=2.5)\n\nENetModel = create_model(ENet)\nKRRModel = create_model(KRR)\n\nmodels_selected = [\"ENet\",\"KRR\"] ## Models with High Cross Validation Score Selected \n\n## Predicted Using Cross Validation Prediction\ny_pred_ENet = cross_val_predict(ENet,X_train_df,y_train_df,cv=5)\ny_pred_KRR = cross_val_predict(KRR,X_train_df,y_train_df,cv=5)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Make last predictions as inputs, and try to fit a model to find a mapping between \n# those predictions and y_train (AVERAGE MODEL)\n# We are going to use Lasso Model with robust scaler\n\nlasso = Lasso(alpha =0.0005, random_state=1)\nlasso.fit(np.array([y_pred_ENet,y_pred_KRR]).T, y_train_df)\n\nLassoModel = create_model(lasso)\ntrainMMSE = LassoModel.mmse_score(np.array([y_pred_ENet,y_pred_KRR]).T,y_train_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Train Mse Based on AVERAGE Model is {}\".format(trainMMSE))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred_ENet_val = ENet.fit(X_train_df,y_train_df).predict(X_val_df)   \nKRR.fit(X_train_df,y_train_df)\ny_pred_KRR_val = KRR.predict(X_val_df)\n\nvalMMSE = LassoModel.mmse_score(np.array([y_pred_ENet_val,y_pred_KRR_val]).T,y_val_df)\nprint(\"Validation Mse Based on AVERAGE Model is {}\".format(valMMSE))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Validation Mse Based on AVERAGE Model is {}\".format(valMMSE))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cross_Val = rmsle_cv(lasso,np.array([y_pred_ENet,y_pred_KRR]).T,y_train_df)\nprint(\"Cross Validation Score of Average Model is {}\".format(cross_Val))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_df_large = train_df[:200000] ## Training Set Tuning \ny_train_df_large = y_data[:200000]\nX_test_df = test_df\n\nX_train_df_large, X_val_df_large, y_train_df_, y_val_df_ = train_test_split(X_train_df_large, y_train_df_large, random_state=1, test_size=0.2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.2. Elastic Net with Polynomial Features","metadata":{}},{"cell_type":"code","source":"# Elastic Net with Polynomial Features \n\nENet = make_pipeline(RobustScaler(), ElasticNet(alpha=0.0005, l1_ratio=.9, random_state=3))\npoly = PolynomialFeatures(degree=2).fit_transform(X_train_df_large)\nENet.fit(poly, y_train_df_)\nENetModel = create_model(ENet)\nvalMMSE = ENetModel.mmse_score(PolynomialFeatures(degree=2).fit_transform(X_val_df_large),y_val_df_)\ntrainMMSE = ENetModel.mmse_score(poly, y_train_df_)\n\nprint(\"TRAIN SET MSE  is {}\".format(trainMMSE))\nprint(\"Validation SET MSE is {}\".format(valMMSE))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crossVal = rmsle_cv(ENet,X_train_df[:100000], y_train_df[:100000])\nprint(crossVal)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### SELECTED MODEL : Average model of Enet and KRR VS Polynomial ENet. \nThe Enet with polynomial features gives the best result in terms of training, validation and test RMSE. However to be used this model needs a training dataset of 200000 samples whereas the kernel Ridge Regressor offers similar results trained on 10000 samples. To conclude we choose the ENet with polynomial features of degree 2 for this challenge as we have a large amount of data and because the RMSE result is the best but Kernel Ridge Regressor would be a better model in the case of a smaller dataset.\n","metadata":{}},{"cell_type":"markdown","source":"### DEEP NEURAL NETWORK \n- We implemented this modele using __Tensorflow Addons__\n\n- For this model we used the __NovoGrad Optimizer__ which uses gradient descent method with __layer-wise gradient normalization__ and __decoupled weight decay__.\n\n- Add a __droup out__ to face the overfitting \n\n- Use __exponential decrease cyclical learning rate__ scheduler \n\n- We train the model using __leaky ReLU__ activation \n\n- __Batchnormalisation__ to make loss function smoother\n\n- All these addons allow us to train a decent DNN with minimal hyperparameter tuning especially for learning rate.\n\n- ___N.B : It takes a lot of time to run the DNN Code, if you want to test, please uncoment the code___","metadata":{}},{"cell_type":"code","source":"# Deep Neural Network \n#! pip install optuna\n#! pip install tensorflow_addons\n#import tensorflow \n#import tensorflow.keras as keras\n#from tensorflow.keras.models import Sequential\n#from tensorflow.keras.layers import Dense\n#import optuna\n#import tensorflow_addons as tfa\n#from tensorflow.keras.layers import Dropout , BatchNormalization \n#from tensorflow.nn import leaky_relu\n#from tensorflow.keras import backend as K\n#from tensorflow_addons.optimizers.novograd import NovoGrad","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#model = Sequential()\n#n_cols = X_train_df.shape[1]\n#print(\"N_cols=\",n_cols)\n\n#def rmse(y_true, y_pred):\n#        return K.sqrt(K.mean(K.square(y_pred - y_true))) \n#def LR(x):\n#    return leaky_relu(x, alpha=0.1)\n\n#model.add(Dense(300, activation=LR, input_shape= (n_cols,)))\n#model.add(BatchNormalization())\n#model.add(Dropout(0.15))\n#model.add(Dense(300, activation=LR))\n#model.add(BatchNormalization())\n#model.add(Dropout(0.1))\n#model.add(Dense(320, activation=LR))\n#model.add(BatchNormalization())\n#model.add(Dropout(0.1))\n#model.add(Dense(350, activation=LR))\n#model.add(BatchNormalization())\n#model.add(Dense(200, activation=LR))\n#model.add(BatchNormalization())\n#model.add(Dense(200, activation=LR))\n#model.add(BatchNormalization())\n#model.add(Dense(1))\n\n#lr_schedule = tfa.optimizers.ExponentialCyclicalLearningRate(\n#            initial_learning_rate=1e-4,\n#            maximal_learning_rate=1e-2,\n#            step_size=2000,\n#            scale_mode=\"cycle\",\n#            gamma=0.96,\n#            name=\"MyCyclicScheduler\")\n\n# Optimizer\n\n#opt = NovoGrad(learning_rate=lr_schedule)\n#opt = tfa.optimizers.Lookahead(opt) ##WRAP OPTIMIZER WITH LookAhead layer \n##Early stopping\n#es =tensorflow.keras.callbacks.EarlyStopping(monitor='val_loss',restore_best_weights=True, #patience=200)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#model.compile(optimizer=opt, loss=rmse)\n# train model\n#history = model.fit(X_train_df, y_train_df, verbose=0, batch_size=40,validation_split=0.2, #epochs=1000, callbacks=[es])\n \n# list all data in history\n#print(history.history.keys())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_prediction(X, model):\n    y_pred = model.predict(X)\n    return y_pred","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#y_val_pred = make_prediction(X_val_df, model)\n#val_rmse = np.sqrt(mean_squared_error(y_val_pred, y_val_df.values))\n#print(\"VAL RMSE: {:.5f}\".format(float(val_rmse)))\n#y_train_pred = make_prediction(X_train_df, model)\n#train_rmse = np.sqrt(mean_squared_error(y_train_pred, y_train_df.values))\n#print(\"Train RMSE: {:.5f}\".format(float(train_rmse)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### VAL RMSE: 7.51688\n\n#### Train RMSE: 7.82663","metadata":{}},{"cell_type":"markdown","source":"## Conclusion \n\nThus, with correct feature selection and less complex models chosen by us, we find that ENet with Polynomial Features degree 2 and Kernel Ridge Regressor predicts the air delay better considering the RMSE and Cross validation score. Further, if given some more time, we would have tried to optimize the hyperparameters of the models, optimise the RAM utilisation and would have tried a few more stacked average models to improve the performance of the model.\n\nWe tried different kinds of models, deep neural network one, non parametric ones and simple linear regression to finally choose the ENet with polynomial features and we submit the scores of both this model and the KRR model.","metadata":{}}]}