{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Flight Delays Prediction","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"**Table of Contents:**\n\n1. Introduction\n2. Data Recuperation and Analysis\n3. Data Cleaning and Feature Selection\n4. Data Preprocessing\n5. Model Training and Selection:\n\n    a] Model Selection : Ridge Regression, ElasticNet, Lightgbm\n    \n    b] Model Optimization\n6. Testing \n7. Conclusion","metadata":{}},{"cell_type":"markdown","source":"# 1. Introduction","metadata":{}},{"cell_type":"markdown","source":"Nowadays, flight delays are very common because a lot of variables contribute to the time necessary for planes to depart and arrive in airports.\nIn this challenge, we want to develop a model that predicts flight arrival delays in minutes for future flights. This is a regression problem, and we will be using pretrained regressors from Machine Learning libraries like SciKit Learn and lightgbm.","metadata":{}},{"cell_type":"markdown","source":"# 2. Data Recuperation and Analysis","metadata":{}},{"cell_type":"markdown","source":"### Data Recuperation","metadata":{}},{"cell_type":"markdown","source":"Let's start with some imports.","metadata":{}},{"cell_type":"code","source":"! pip install scikit-learn==0.21\n! pip install hypopt","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nimport json\nimport warnings\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.dates as mdates\nimport seaborn as sns\n\nimport numpy as np\n\n\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.linear_model import Ridge\nfrom sklearn.preprocessing import StandardScaler\nfrom hypopt import GridSearch\nfrom matplotlib import rc \n\nif not sys.warnoptions:\n    warnings.simplefilter(\"ignore\")\n\nrc('figure', **{'dpi': 200})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we can load the training data.","metadata":{}},{"cell_type":"code","source":"X_dtype = {\n    'ID'                   : int,\n    'YEAR'                 : int,  \n    'MONTH'                : int,  \n    'DAY'                  : int,  \n    'DAY_OF_WEEK'          : int,  \n    'AIRLINE'              : str, \n    'FLIGHT_NUMBER'        : str,  \n    'TAIL_NUMBER'          : str, \n    'ORIGIN_AIRPORT'       : str, \n    'DESTINATION_AIRPORT'  : str, \n    'SCHEDULED_DEPARTURE'  : str,  \n    'DEPARTURE_TIME'       : str, \n    'DEPARTURE_DELAY'      : float,\n    'TAXI_OUT'             : str, \n    'WHEELS_OFF'           : str,\n    'SCHEDULED_TIME'       : float,\n    'AIR_TIME'             : float,\n    'DISTANCE'             : int,\n    'SCHEDULED_ARRIVAL'    : str,\n    'DIVERTED'             : int,  \n    'CANCELLED'            : int,  \n    'CANCELLATION_REASON'  : str\n}\n\ny_dtype = {\n    'ID'                   : int,\n    \"ARRIVAL_DELAY\"        : float\n}\n\nX_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_features.csv\", dtype=X_dtype)\ny_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_targets.csv\", dtype=y_dtype)\nX_test_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/test_features.csv\", dtype=X_dtype)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A glimpse of the data can be seen in the following table.","metadata":{}},{"cell_type":"code","source":"X_train_df=pd.merge(X_train_df,y_train_df, on='ID')\nX_train_df.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can have some information about the flights.","metadata":{}},{"cell_type":"code","source":"X_train_df.describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We also have information about both the Airports ans Airlines.","metadata":{}},{"cell_type":"code","source":"airports = pd.read_csv('/kaggle/input/eurecom-aml-2021-challenge-1/data/airports.csv',\n                       usecols=['IATA_CODE', 'AIRPORT', 'LATITUDE', 'LONGITUDE']).rename({'AIRPORT': 'NAME'}, axis='columns')\nairlines = pd.read_csv('/kaggle/input/eurecom-aml-2021-challenge-1/data/airlines.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"airlines.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"airports.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Analysis","metadata":{}},{"cell_type":"markdown","source":"For this part, we're going to add certain columns to our database, so that we can explore the data.\n\nIt is important to ask the right questions to learn more about the correlation between features and to show how important are some features.\nAt first we'll add the column 'DATE' that contains a datetime value in the format yy-mm-dd and the column 'DELAYED' that is a boolean variable showing if the flight is delayed or not.","metadata":{}},{"cell_type":"code","source":"X_analysis=X_train_df.copy()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert date-time features from str to datetime format\ndef parse_hhmm(x):\n    try: return pd.datetime.strptime(x, '%H%M')#.time()\n    except: return pd.NaT\n\nX_analysis.DEPARTURE_TIME = X_analysis.DEPARTURE_TIME.apply(parse_hhmm)\nX_analysis.SCHEDULED_ARRIVAL = X_analysis.SCHEDULED_ARRIVAL.apply(parse_hhmm)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df=X_analysis\ndf['DELAYED'] = (df.ARRIVAL_DELAY > 0).astype(str)\ndf['DATE'] = pd.to_datetime(df[['YEAR', 'MONTH', 'DAY']])\ndf.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's start with the affect of time-related features.","metadata":{}},{"cell_type":"markdown","source":"## A. Impact of Time-related features","metadata":{}},{"cell_type":"markdown","source":"Effect of Date","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nfig.set_size_inches(10, 5, forward=True)\nsns.histplot(df, x='DATE', hue='DELAYED', bins=48, multiple='stack', palette=['C0', 'C3'], ax=ax)\nax.xaxis.set_major_formatter(mdates.DateFormatter(\"%d/%m\"))\nax.set_title('Seasonal effect on flight delay?')\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now the affect of the day of week","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nfig.set_size_inches(5, 3, forward=True)\nsns.histplot(df, x='DAY_OF_WEEK', hue='DELAYED', bins=48, multiple='stack', palette=['C0', 'C3'], ax=ax)\nax.set_title('Day of week effect on flight delay?')\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And finally the effect of Departure time.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nsns.histplot(df, x='DEPARTURE_TIME', hue='DELAYED', bins=48, multiple='stack', palette=['C0', 'C3'], ax=ax)\nax.xaxis.set_major_formatter(mdates.DateFormatter(\"%H:%M\"))\nax.set_title('Effect of Departure time on arrival Delay ')\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***Conclusion*** : As we can see, Datetime features contribute immensely to Arrival Delays. For the first figure, we can see that during a peak of flights ( around Christmas and in Summer), the percentage of flights that are delayed over all the flights is clearly higher than the rest of the year. Equivalently, it seems that from 21H:00, more than half the flights get delayed.","metadata":{}},{"cell_type":"markdown","source":"## B. Impact of Logistical Features:","metadata":{}},{"cell_type":"markdown","source":"Impact of Airport","metadata":{}},{"cell_type":"code","source":"#_____________________________________________\n# Function that define how delays are grouped\ndelay_type = lambda x:((0,1)[x > 5],2)[x > 45]\ndf['DELAY_LEVEL'] = df['DEPARTURE_DELAY'].apply(delay_type)\ndf_top=df.loc[(df['ORIGIN_AIRPORT']=='ATL') | (df['ORIGIN_AIRPORT']=='ORD') | (df['ORIGIN_AIRPORT']=='DFW') | (df['ORIGIN_AIRPORT']=='DEN') | (df['ORIGIN_AIRPORT']=='LAX')]\n#____________________________________________________\nfig = plt.figure(1, figsize=(10,7))\nax = sns.countplot(y=\"ORIGIN_AIRPORT\", hue='DELAY_LEVEL', data=df_top)\n#____________________________________________________________________________________\n# We replace the abbreviations by the full names of the companies and set the labels\nlabels = [item.get_text() for item in ax.get_yticklabels()]\nax.set_yticklabels(labels)\nplt.setp(ax.get_xticklabels(), fontsize=12, weight = 'normal', rotation = 0);\nplt.setp(ax.get_yticklabels(), fontsize=12, weight = 'bold', rotation = 0);\nax.yaxis.label.set_visible(False)\nplt.xlabel('Flight count', fontsize=16, weight = 'bold', labelpad=10)\n#________________\n# Set the legend\nL = plt.legend()\nL.get_texts()[0].set_text('on time (t < 5 min)')\nL.get_texts()[1].set_text('small delay (5 < t < 45 min)')\nL.get_texts()[2].set_text('large delay (t > 45 min)')\nplt.title('Top 5 busiest airports and their arrival delay count ')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Impact of distance","metadata":{}},{"cell_type":"code","source":"query = df[(df.TAIL_NUMBER=='N594HA')].sort_values(by=['DISTANCE'],ascending=True).drop_duplicates(subset=['DISTANCE'])\nplt.plot(list(query['DISTANCE']),list(query['ARRIVAL_DELAY']))\nplt.title('Impact of distance on Arrival Delay')\nplt.xlabel('Distance', fontsize=16, labelpad=10)\nplt.ylabel('Arrival Delay', fontsize=16, labelpad=10)\n\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Impact of Airline","metadata":{}},{"cell_type":"code","source":"#_____________________________________________\n# Function that define how delays are grouped\ndelay_type = lambda x:((0,1)[x > 5],2)[x > 45]\ndf['DELAY_LEVEL'] = df['DEPARTURE_DELAY'].apply(delay_type)\n#____________________________________________________\nfig = plt.figure(1, figsize=(10,7))\nax = sns.countplot(y=\"AIRLINE\", hue='DELAY_LEVEL', data=df)\n#____________________________________________________________________________________\n# We replace the abbreviations by the full names of the companies and set the labels\nlabels = [item.get_text() for item in ax.get_yticklabels()]\nax.set_yticklabels(labels)\nplt.setp(ax.get_xticklabels(), fontsize=12, weight = 'normal', rotation = 0);\nplt.setp(ax.get_yticklabels(), fontsize=12, weight = 'bold', rotation = 0);\nax.yaxis.label.set_visible(False)\nplt.xlabel('Flight count', fontsize=16, weight = 'bold', labelpad=10)\n#________________\n# Set the legend\nL = plt.legend()\nplt.title('Airline Impact on Arrival Delay')\nL.get_texts()[0].set_text('on time (t < 5 min)')\nL.get_texts()[1].set_text('small delay (5 < t < 45 min)')\nL.get_texts()[2].set_text('large delay (t > 45 min)')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Data cleaning and Feature Selection","metadata":{}},{"cell_type":"markdown","source":"Now we will check how complete our Dataset is and if any data is missing.","metadata":{}},{"cell_type":"code","source":"X_train_df.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that apart from the Cancellation_Reason where the value NaN does have a meaning, none of the feautres' values are missing.","metadata":{}},{"cell_type":"markdown","source":"## Data Cleaning","metadata":{}},{"cell_type":"code","source":"#Here are the airports that sould be in the database:\nprint(\"The listed airports are : \",list(airports['IATA_CODE'].unique()))\n\n#Here are the airports in the dataset\nl=list(X_train_df['ORIGIN_AIRPORT'].unique())\nprint(\"The airports that are in our dataset are: \", l)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We noticed then that many of the mentionned airports in the training Dataset do not match a fully known airport in the airports.csv file. These airports have unkown code names. In order to figure out how to deal with these samples, we calculated the percentage of airports with missing refenreces:","metadata":{}},{"cell_type":"code","source":"#percentage of airports with unkown code names\n\ns=0\nfor i in X_train_df['ORIGIN_AIRPORT']:\n    if i.isnumeric():\n        s+=1\nprint (round(s*100/X_train_df.shape[0],4),' %')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We found that the samples with missing data constitute around 8% of the data. We judged it’s better to get rid of it:","metadata":{}},{"cell_type":"code","source":"#dropping the rows with unknown airports\ndef removing_airports(X_train_df):\n    L=[]\n    H=list(X_train_df['ORIGIN_AIRPORT'])\n    for i in range(len(H)):\n        if H[i].isnumeric():\n           L.append(i)\n    X_train_df=X_train_df.drop(L)\n    return(X_train_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For now, we have only been working with Origin_Airport feature. We wanted to check if there is any remaining problem with the Destination_Airports after dropping the unknown Origin_Airports:","metadata":{}},{"cell_type":"code","source":"\n#we wanted to check that once we remove unknown origin_airports there no remaining unknown destination_airport because they match together\n#(This cell should be executed after preprocessing )\ns=0\nfor i in X_train_df['DESTINATION_AIRPORT']:\n    if i.isnumeric():\n        s+=1\nprint (s*100/X_train_df.shape[0],' %')\n        ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We noticed that whenever the departure airport is missing, the corresponding arrival airport is missing too and vice versa. So no further cleaning is needed.","metadata":{}},{"cell_type":"markdown","source":"# labeling non categorcial features","metadata":{}},{"cell_type":"markdown","source":"In order to label all the existing Airports in the training set, the airports dataset and the testing set, we have to merge all the Airport IATA_CODES from the 3 datasets.","metadata":{}},{"cell_type":"code","source":"r=X_train_df['ORIGIN_AIRPORT'].append(X_test_df['ORIGIN_AIRPORT'],ignore_index=True)\nr=r.append(airports['IATA_CODE'],ignore_index=True)\nr=r.append(X_train_df['DESTINATION_AIRPORT'],ignore_index=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#labeling airports, airlines and planes\n\nfrom sklearn.preprocessing import LabelEncoder\n\ndef encoding_str(X_train_df,airports,airlines):\n    enc = LabelEncoder()\n    enc.fit(r)\n    X_train_df['ORIGIN_AIRPORT_ENCODED'] = enc.transform(X_train_df['ORIGIN_AIRPORT'])\n    X_train_df['DESTINATION_AIRPORT_ENCODED'] = enc.transform(X_train_df['DESTINATION_AIRPORT'])\n\n\n    enc2 = LabelEncoder()\n    enc2.fit(X_train_df['TAIL_NUMBER'])\n    X_train_df['TAIL_NUMBER_ENCODED'] = enc2.transform(X_train_df['TAIL_NUMBER'])\n\n    enc3 = LabelEncoder()\n    enc3.fit(X_train_df['AIRLINE'])\n    X_train_df['AIRLINE_ENCODED'] = enc3.transform(X_train_df['AIRLINE'])\n\n    return (X_train_df)\n#we drope te initial names as we will train with encoded features\n\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#we will encode the timing from the hhmm format to the corresponding minutes counting from midnight\n    \ndef time_encoding(x):\n    return int(x[0:2])*60 + int(x[2:4])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Feature Selection","metadata":{}},{"cell_type":"markdown","source":"We will start with an \"obvious\" feature selection : no training needed, no performance classification. For now, we will only use reasoning to get rid of the unnecessary features.","metadata":{}},{"cell_type":"code","source":"#list of features to drop\nto_drop=[]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#drop the features that have been encoded and thus replaced by their label\nto_drop+=['ORIGIN_AIRPORT','DESTINATION_AIRPORT','TAIL_NUMBER','AIRLINE']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set(X_train_df['DIVERTED'].tolist())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set(X_train_df['CANCELLED'].tolist())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set(X_train_df['YEAR'].tolist())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set(X_train_df['CANCELLATION_REASON'].tolist())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, these 4 features have only one value and thus are at no use for our delay prediction. So we decided to drop them.","metadata":{}},{"cell_type":"code","source":"to_drop+=['CANCELLATION_REASON','YEAR','CANCELLED','DIVERTED']\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"More advanced feature selection needed to go through the Correlation Matrix in order to better understand the relations between the features. This step justifies why the labeling proceeded the feature selection (altough we might label features that might be dropped) : the non categorical features needed to be labelled in order to show up on the correlation matrix.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nheatmap = sns.heatmap(X_train_df.corr(), vmin=-1, vmax=1, annot=True)\nheatmap.set_title('Correlation Heatmap', fontdict={'fontsize':18}, pad=12);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We noticed further that some others features are simply a combination of other features. Such features don’t help decreasing the entropy of the model as they don’t reveal any new information. ","metadata":{}},{"cell_type":"code","source":"#Dropping features that can found out by coombining the remaining features\n\nto_drop+=['WHEELS_OFF']#, 'SCHEDULED_DEPARTURE', 'SCHEDULED_ARRIVAL']\n\n#We judged that flight_nuber needed to be dropped as it has nothing to do with the delay prediction\nto_drop+=['FLIGHT_NUMBER']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\nSo the final list of features to drop is the following:","metadata":{}},{"cell_type":"code","source":"to_drop","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Data Preprocessing","metadata":{}},{"cell_type":"markdown","source":"Now,it's time to train ! But first, we need to make sure that the data is preprocessed.","metadata":{}},{"cell_type":"code","source":"def preprocess_data(X_train_df):\n    \"\"\"Preprocess data.\n\n    Parameters\n    ----------\n    df: pandas DataFrame.\n        The input data.\n    feature_names: list of strings.\n        The names of selected features.\n    imputer: sklearn.impute.SimpleImputer\n        The imputation transformer for completing missing values.\n    scaler: sklearn.preprocessing.StandardScaler.\n        The scaler used to normalize the features.\n\n    Returns\n    -------\n    X: numpy array.\n        The preprocessed data.\n        \n    \"\"\"\n     \n\n    X_train_df=encoding_str(X_train_df,airports,airlines)\n    X_train_df['DEPARTURE_TIME']=X_train_df['DEPARTURE_TIME'].apply(time_encoding)\n\n    #converting TAXI_OUT feature to int\n    X_train_df['TAXI_OUT']=X_train_df['TAXI_OUT'].apply(int)\n    \n    #dropping the unnecessary features\n    X_train_df=X_train_df.drop(to_drop,axis=1)\n\n\n    return X_train_df\n#X_train_df=removing_airports(X_train_df)\nX_train_df=preprocess_data(X_train_df)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_df=X_train_df['ARRIVAL_DELAY']\nX_train_df=X_train_df.drop(['ID','ARRIVAL_DELAY'],axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n#split\nxtrain,xtest,ytrain,ytest=train_test_split(X_train_df,y_train_df,test_size=0.20,random_state=20)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Selection And Validation","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import KFold, cross_val_score\nimport lightgbm as lgb\nfrom sklearn.kernel_ridge import KernelRidge\nfrom sklearn.linear_model import ElasticNet, Lasso,  BayesianRidge, LassoLarsIC","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Validation function\nn_folds = 5\ndef rmsle_cv(model):\n    kf = KFold(n_folds, shuffle=True, random_state=42).get_n_splits(xtrain.values)\n    rmse= np.sqrt(-cross_val_score(model, xtrain.values, ytrain, scoring=\"neg_mean_squared_error\", cv = kf))\n    return(rmse)\nmodels=[]\nscores=[]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rdg = Ridge()\nscore = rmsle_cv(rdg)\nmodels.append('Ridge')\nscores.append(score)\nprint(\"Ridge score: {:.4f} \\n\" .format(score.mean()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_lgb = lgb.LGBMRegressor(objective='regression',num_leaves=5,\n                              learning_rate=0.05, n_estimators=720,\n                              max_bin = 55, bagging_fraction = 0.8,\n                              bagging_freq = 5, feature_fraction = 0.2319,\n                              feature_fraction_seed=9, bagging_seed=9,\n                              min_data_in_leaf =6, min_sum_hessian_in_leaf = 11)\nscore = rmsle_cv(model_lgb)\nmodels.append('LightGBMRegressor')\nscores.append(score)\nprint(\"LGBM score: {:.4f} \\n\" .format(score.mean()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import RobustScaler\nENet =make_pipeline(RobustScaler(), ElasticNet(alpha=0.0005, l1_ratio=.9, random_state=3))\nscore = rmsle_cv(ENet)\nmodels.append('ENet')\nscores.append(score)\nprint(\"ElasticNet score: {:.4f} \\n\".format(score.mean()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"NB: We sometimes get an error when running the ENetRegressor, but the average score we get is around 5.5.","metadata":{}},{"cell_type":"code","source":"#In case we get an error.\n#models.append('ENet')\n#scores.append(np.array[5.5029])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sc=[]\nfor i in scores:   \n    sc.append(sum(i)/len(i))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sc","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_models=pd.DataFrame({'Model':models,'Score' :sc},index=[1,2,3])\ndf_models","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# B] Optimization","metadata":{}},{"cell_type":"markdown","source":"In this section, we optimize the parameter alpha of the Ridge Regressor.","metadata":{}},{"cell_type":"code","source":"param_grid = [\n  {'alpha': list(np.linspace(0,1,num=10))},\n]\n\nopt = GridSearch(model=Ridge(tol=0.01), param_grid=param_grid)\nopt.fit(xtrain, ytrain, xtest, ytest, scoring='neg_mean_squared_error')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(opt.best_params)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"alphas = [0,0.01,0.1,1, 10]\nRMSE = []\nfor a in alphas:\n    ridge = Ridge(alpha=a, fit_intercept=False)\n    ridge.fit(xtrain, ytrain)\n    RMSE.append(np.sqrt(mean_squared_error(ridge.predict(xtest), ytest.values)))\n\nax = plt.gca()\nax.plot(alphas, RMSE)\nax.set_xscale('log')\nax.set_xlim(ax.get_xlim()[::-1])  # reverse axis\nplt.xlabel('alpha')\nplt.ylabel('RMSE')\nplt.title('RMSE as a function of the regularization')\nplt.axis('tight')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Testing Model ","metadata":{}},{"cell_type":"code","source":"X_test_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/test_features.csv\", dtype=X_dtype)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test_df=preprocess_data(X_test_df)\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"col1=X_test_df['ID']\nX_test_df=X_test_df.drop('ID',axis=1)\nX_test_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rdg.fit(xtrain,ytrain)\ny_test_pred=rdg.predict(X_test_df)\ny_test_pred","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test_df['ID']=col1\nsubmission_df = pd.DataFrame(data={'ID': X_test_df.ID.values,\n                                   'ARRIVAL_DELAY': y_test_pred.squeeze()})\n\n# Save the predictions into a csv file\n# Notice that this file should be saved under the directory `/kaggle/working` \n# so that you can download it later\nsubmission_df.to_csv(\"/kaggle/working/submission.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! head -5 \"/kaggle/working/submission.csv\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}}]}