{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# I) Implementation of data","metadata":{}},{"cell_type":"markdown","source":"**General installations**","metadata":{}},{"cell_type":"code","source":"! pip install scikit-learn==0.21\n! pip install hypopt","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**General imports**","metadata":{}},{"cell_type":"code","source":"import sys\nimport json\nimport warnings\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.dates as mdates\nimport seaborn as sns\nimport tensorflow as tf\nimport numpy as np\nimport tensorflow_probability as tfp\n\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.linear_model import Ridge\nfrom sklearn.preprocessing import StandardScaler\nfrom hypopt import GridSearch\nfrom matplotlib import rc \nfrom sklearn.preprocessing import LabelEncoder\n\nif not sys.warnoptions:\n    warnings.simplefilter(\"ignore\")\n\nrc('figure', **{'dpi': 200})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Load the training data with the adapted format**","metadata":{}},{"cell_type":"code","source":"X_dtype = {\n    'ID'                   : int,\n    'YEAR'                 : int,  \n    'MONTH'                : int,  \n    'DAY'                  : int,  \n    'DAY_OF_WEEK'          : int,  \n    'AIRLINE'              : str, \n    'FLIGHT_NUMBER'        : str,  \n    'TAIL_NUMBER'          : str, \n    'ORIGIN_AIRPORT'       : str, \n    'DESTINATION_AIRPORT'  : str, \n    'SCHEDULED_DEPARTURE'  : str,  \n    'DEPARTURE_TIME'       : str, \n    'DEPARTURE_DELAY'      : float,\n    'TAXI_OUT'             : str, \n    'WHEELS_OFF'           : str,\n    'SCHEDULED_TIME'       : float,\n    'AIR_TIME'             : float,\n    'DISTANCE'             : int,\n    'SCHEDULED_ARRIVAL'    : str,\n    'DIVERTED'             : int,  \n    'CANCELLED'            : int,  \n    'CANCELLATION_REASON'  : str\n}\n\ny_dtype = {\n    'ID'                   : int,\n    \"ARRIVAL_DELAY\"        : float\n}\n\nX_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_features.csv\", dtype=X_dtype)\ny_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_targets.csv\", dtype=y_dtype)\nX_test_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/test_features.csv\", dtype=X_dtype)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Additional data are available about the airports and the airlines**","metadata":{}},{"cell_type":"code","source":"airports = pd.read_csv('/kaggle/input/eurecom-aml-2021-challenge-1/data/airports.csv',usecols=['IATA_CODE', 'AIRPORT', 'LATITUDE', 'LONGITUDE']).rename({'AIRPORT': 'NAME'}, axis='columns')\nairlines = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/airlines.csv\",usecols=['IATA_CODE', 'AIRLINE']).rename({'AIRLINE': 'NAME'}, axis='columns')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Now, we can merge feature dataframe and target dataframe for data exploration**","metadata":{}},{"cell_type":"code","source":"df = pd.merge(X_train_df, y_train_df, on='ID')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We can creat  new columns used for data exploration**","metadata":{}},{"cell_type":"code","source":"df['DELAYED'] = (df.ARRIVAL_DELAY > 0).astype(str)\ndf['DATE'] = pd.to_datetime(df[['YEAR', 'MONTH', 'DAY']])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"****","metadata":{}},{"cell_type":"markdown","source":"# II) Data analysis","metadata":{}},{"cell_type":"markdown","source":"### 1) Prior distribution","metadata":{}},{"cell_type":"markdown","source":"**Before starting, an interesting point can be the distribution of the delays. Thanks to this distribution, we can have a prior belief of what the delays are likely to be.**","metadata":{}},{"cell_type":"code","source":"df[\"ARRIVAL_DELAY\"].hist(bins = 300, figsize=(10, 7))\nplt.xlabel('DELAY')\nplt.ylabel(\"NUMBER OF FLIGHTS\");","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Thanks to the SSP course (Statistical Signal Processing), we can conclude that the arrival delay follows a prior distrubution which looks like a shifted Rayleigh distribution. This distribution is also often used to modelise the delay of transmition in a network.**\n\n**https://fr.wikipedia.org/wiki/Loi_de_Rayleigh**","metadata":{}},{"cell_type":"markdown","source":"### 2) Data exploration","metadata":{}},{"cell_type":"markdown","source":"**Intuitively, we can have an opinion about the influence of a parameter on the arrival delay. Some features seem to be highly correlated to the arrival delay whereas some others seems to have no impact on the delay. We can explore this assumption that can be useful for the model selection.**","metadata":{}},{"cell_type":"markdown","source":"****","metadata":{}},{"cell_type":"markdown","source":"**For example, a parameter such as the airline can have an influence. Some airlines are more ponctual than others and we can expect that the arrival delay is linked to the airline.**\n\n**Let's verify this statement.**","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nsns.histplot(df, x='AIRLINE', hue='DELAYED', multiple='stack', palette=['C0', 'C3'], ax=ax)\nax.set_title('Overview of the flights')\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**This plot is not really usable. Nevertheless, we can compare the percentage of flights delayed and the mean of the delay for every airline.**","metadata":{}},{"cell_type":"code","source":"df1 = df.copy(deep=True)\n\ndf3 = df1.groupby(['AIRLINE']).count()\n\ndf5 = df1.groupby(['AIRLINE']).ARRIVAL_DELAY.mean()\n\ndf2 = df1[(df1.DELAYED==\"True\")].groupby(['AIRLINE']).count()\ndf2['PERCENT_DELAYED']= df2['ID']/df3['ID']\ndf2['MEAN_DELAY']=df5\ndf4 = df2.sort_values(by = 'PERCENT_DELAYED', ascending=False)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df4.plot(y=\"PERCENT_DELAYED\", kind = 'bar')\ndf4.plot(y=\"MEAN_DELAY\", kind = 'line')\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Now, we can deduce that the percentage of flight delayed and the average delay can vary greatly depending on the airline. Concenquently, these features are highly correlated to the delay of the flight.**","metadata":{}},{"cell_type":"markdown","source":"*****","metadata":{}},{"cell_type":"markdown","source":"**On the other hand, the influence of some parameters are unknown a priori. Let's take the distance for exemple. We don't really know if this parameter is correlated to the arrival delay, so let's have a closer look.**","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nsns.histplot(df, x='DISTANCE', hue='DELAYED', multiple='stack', palette=['C0', 'C3'], ax=ax, binwidth=250)\nax.set_title('Flights delay per distance')\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Again, what's really interesting is to compare the percentage of flights delayed by distance.**","metadata":{}},{"cell_type":"code","source":"df6 = df.copy(deep=True)\n\ndf8 = df6.groupby(['DISTANCE']).count()\n\ndf9 = df6.groupby(['DISTANCE']).ARRIVAL_DELAY.mean()\n\ndf7 = df6[(df6.DELAYED==\"True\")].groupby(['DISTANCE']).count()\ndf7['PERCENT_DELAYED']= df7['ID']/df8['ID']\ndf7['MEAN_DELAY']=df9","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df7.plot(y='PERCENT_DELAYED', kind='bar')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Not relevant at all, we better split at specific distances :**\n- **0-500km     --> short distance**\n- **500-1500km  --> average distance**\n- **+ 1500km    --> long distance**","metadata":{}},{"cell_type":"code","source":"#1 determine position to which split\n\ni_1=0\nwhile df7.iloc[i_1].name < 500 :\n    i_1+=1\n    \ni_2=i_1\nwhile df7.iloc[i_2].name < 1500 :\n    i_2+=1\n\n#2 Same thing than before\n    \ndf_1 = df7.iloc[:i_1]\ndf_2 = df7.iloc[i_1:i_2]\ndf_3 = df7.iloc[i_2:]\n\np_all = []\np_all.append((df_1['PERCENT_DELAYED'].sum())/i_1)\np_all.append((df_2['PERCENT_DELAYED'].sum())/(i_2-i_1))\np_all.append((df_3['PERCENT_DELAYED'].sum())/(1344-i_2))\n\nx_dist = []\nx_dist.append(\"0-500km\")\nx_dist.append(\"500-1500km\")\nx_dist.append(\">1500km\")\n\n#3 Plot by range\n\nplt.bar(x_dist, p_all)\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**In this case, the distance seems to have no incidence at all on the probability of delay. This feature is maybe not so useful in the prediction of the delay.**\n","metadata":{}},{"cell_type":"markdown","source":"****","metadata":{}},{"cell_type":"markdown","source":"**This short analysis leads to the conclution that some features seem to be relevant in the prediction of the delay whereas some others are much less important. This conclusion can be usefull for our model selection. Indeed, the feature selection is a labourious task and it's not scalable to do it by hand. Fortunately, models like tree regression do this selection automaticaly. This model could be suitable to take into account the difference in the importance of the features.**","metadata":{}},{"cell_type":"markdown","source":"****\n","metadata":{}},{"cell_type":"markdown","source":"# III) Data Pre-processing","metadata":{}},{"cell_type":"markdown","source":"## 1) Pre-processing step by step","metadata":{}},{"cell_type":"markdown","source":"**We can easily understand that some feature are not relevant at all. Our algorithm to predict the delay has nothing to do with the flight number FLIGHT_NUMBER, the identifier ID and the year YEAR. Consequently, these features can be dropped.**","metadata":{}},{"cell_type":"code","source":"X_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_features.csv\", dtype=X_dtype)\ny_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_targets.csv\", dtype=y_dtype)\nX_test_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/test_features.csv\", dtype=X_dtype)\n\nX_train_df = X_train_df.drop([\"ID\", \"YEAR\", \"FLIGHT_NUMBER\"], axis = 1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"****","metadata":{}},{"cell_type":"markdown","source":"**To be as realistic as possible, we put ourself in the shoes of a potential passenger who wants  to predict the delay of his flight BEFORE the flight. Of course, this scenario implies a strong assumption: some features provided wont be used. Indeed features like AIRTIME and DIVERTED are useful information, nevertheless, this information is available once the flight has already begun.**","metadata":{}},{"cell_type":"code","source":"feature_names = ['MONTH', 'DAY',  'DAY_OF_WEEK', 'AIRLINE', 'TAIL_NUMBER', 'ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', 'SCHEDULED_DEPARTURE',\n                'DEPARTURE_DELAY', 'WHEELS_OFF', 'SCHEDULED_TIME',  'DISTANCE', 'SCHEDULED_ARRIVAL', 'CANCELLED', 'CANCELLATION_REASON']\ntarget_name = ['ARRIVAL_DELAY']\n\nX_train_df = X_train_df[feature_names]\ny_train_df = y_train_df[target_name]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"****","metadata":{}},{"cell_type":"markdown","source":"**Now, we have to modify the time data to use it. Currently, the time data has a string format.To use it, another format is required.**","metadata":{}},{"cell_type":"code","source":"def parse_hhmm(x):\n    try: return pd.datetime.strptime(x, '%H%M')#.time()\n    except: return pd.NaT\n\nX_train_df.SCHEDULED_DEPARTURE = X_train_df.SCHEDULED_DEPARTURE.apply(parse_hhmm)\nX_train_df.WHEELS_OFF = X_train_df.WHEELS_OFF.apply(parse_hhmm)\nX_train_df.SCHEDULED_ARRIVAL = X_train_df.SCHEDULED_ARRIVAL.apply(parse_hhmm)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def minutes_since_midnight(dt):\n    return dt.hour * 60 + dt.minute\n\nX_train_df.SCHEDULED_DEPARTURE = X_train_df.SCHEDULED_DEPARTURE.apply(minutes_since_midnight)\nX_train_df.WHEELS_OFF = X_train_df.WHEELS_OFF.apply(minutes_since_midnight)\nX_train_df.SCHEDULED_ARRIVAL = X_train_df.SCHEDULED_ARRIVAL.apply(minutes_since_midnight)\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_df.describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Now, we can standardize the features by removing the mean and scaling to unit variance. Of course, this standardization is only possible for numerical format not string format.**","metadata":{}},{"cell_type":"code","source":"feature_names_stand = ['MONTH', 'DAY',  'DAY_OF_WEEK', 'SCHEDULED_DEPARTURE', 'DEPARTURE_DELAY', 'WHEELS_OFF', \n                       'SCHEDULED_TIME',  'DISTANCE', 'SCHEDULED_ARRIVAL']\ntarget_name_stand = ['ARRIVAL_DELAY']\n\n\n## Standardize X\nX_train_stand = X_train_df[feature_names_stand]\nX_scaler = StandardScaler().fit(X_train_stand.values)\nX_train_stand = X_scaler.transform(X_train_stand.values)\n    #replace the columns values\nX_train_df[feature_names_stand] = X_train_stand\n\n\n\n## Standardize y\ny_train_stand = y_train_df[target_name_stand]\ny_scaler = StandardScaler().fit(y_train_stand.values)\ny_train_stand = y_scaler.transform(y_train_stand.values)\n    #replace the columns values\ny_train_df[target_name_stand] = y_train_stand\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Finally, we have to transform string values to numerical ones.**\n\n**We use a LabelEncoder that associates one number for each string value.**","metadata":{}},{"cell_type":"code","source":"\n\n#Create instance of labelEncoder\nle = LabelEncoder()\n\n#Assign the numerical values and replace the string ones by them\n\nX_train_df['AIRLINE'] = le.fit_transform(X_train_df['AIRLINE'])\nX_train_df['TAIL_NUMBER'] = le.fit_transform(X_train_df['TAIL_NUMBER'])\nX_train_df['ORIGIN_AIRPORT'] = le.fit_transform(X_train_df['ORIGIN_AIRPORT'])\nX_train_df['DESTINATION_AIRPORT'] = le.fit_transform(X_train_df['DESTINATION_AIRPORT'])\nX_train_df['CANCELLATION_REASON'] = le.fit_transform(X_train_df['CANCELLATION_REASON'])\n\n\nX_train_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_df.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Fill the missing values\nX_train_df = X_train_df.fillna({\"SCHEDULED_ARRIVAL\": 0})\nX_train_df = X_train_df.fillna({\"WHEELS_OFF\": 0})\nX_train_df.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Split the train into train and validation\nX_train_df, X_val_df, y_train_df, y_val_df = train_test_split(X_train_df, y_train_df, random_state=1, test_size=0.2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2) Pipeline","metadata":{}},{"cell_type":"markdown","source":"**This part is just like the previous part but implemented in a function to pack the pre-processing.**","metadata":{}},{"cell_type":"code","source":"X_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_features.csv\", dtype=X_dtype)\ny_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_targets.csv\", dtype=y_dtype)\nX_test_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/test_features.csv\", dtype=X_dtype)\n\ndef parse_hhmm(x):\n    try: return pd.datetime.strptime(x, '%H%M')#.time()\n    except: return pd.NaT\ndef minutes_since_midnight(dt):\n    return dt.hour * 60 + dt.minute","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def preprocess_input(df, scaler):\n    \"\"\"\n    Parameters\n    ----------\n    df: pandas DataFrame.\n        The input data.\n    scaler: sklearn.preprocessing.StandardScaler.\n        The scaler used to normalize the features.\n\n    Returns\n    -------\n    X: numpy array.\n        The preprocessed data.\n    \"\"\"\n    feature_names = ['MONTH', 'DAY',  'DAY_OF_WEEK', 'AIRLINE', 'TAIL_NUMBER', 'ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', 'SCHEDULED_DEPARTURE',\n                'DEPARTURE_DELAY', 'WHEELS_OFF', 'SCHEDULED_TIME',  'DISTANCE', 'SCHEDULED_ARRIVAL', 'CANCELLED', 'CANCELLATION_REASON']\n\n    df = df[feature_names]\n    \n    df.SCHEDULED_DEPARTURE = df.SCHEDULED_DEPARTURE.apply(parse_hhmm)\n    df.SCHEDULED_ARRIVAL = df.SCHEDULED_ARRIVAL.apply(parse_hhmm)\n    df.WHEELS_OFF = df.WHEELS_OFF.apply(parse_hhmm)\n    \n    df.SCHEDULED_DEPARTURE = df.SCHEDULED_DEPARTURE.apply(minutes_since_midnight)\n    df.SCHEDULED_ARRIVAL = df.SCHEDULED_ARRIVAL.apply(minutes_since_midnight)\n    df.WHEELS_OFF = df.WHEELS_OFF.apply(minutes_since_midnight)\n    \n    \n    feature_names_stand = ['MONTH', 'DAY',  'DAY_OF_WEEK', 'SCHEDULED_DEPARTURE', 'DEPARTURE_DELAY', 'WHEELS_OFF', \n                       'SCHEDULED_TIME',  'DISTANCE', 'SCHEDULED_ARRIVAL']\n    ## Standardize X\n    df_stand = df[feature_names_stand]\n    df_scaler = scaler.fit(df_stand.values)\n    df_stand = df_scaler.transform(df_stand.values)\n        #replace the columns values\n    df[feature_names_stand] = df_stand\n    \n    #Create instance of labelEncoder\n    le = LabelEncoder()\n\n    #Assign the numerical values and replace the string ones by them\n\n    df['AIRLINE'] = le.fit_transform(df['AIRLINE'])\n    df['TAIL_NUMBER'] = le.fit_transform(df['TAIL_NUMBER'])\n    df['ORIGIN_AIRPORT'] = le.fit_transform(df['ORIGIN_AIRPORT'])\n    df['DESTINATION_AIRPORT'] = le.fit_transform(df['DESTINATION_AIRPORT'])\n    df['CANCELLATION_REASON'] = le.fit_transform(df['CANCELLATION_REASON'])\n\n    #Fill the missing values\n    df = df.fillna({\"SCHEDULED_ARRIVAL\": 0})\n    df = df.fillna({\"WHEELS_OFF\": 0})\n    \n    return df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def preprocess_output(df, scaler):\n    \"\"\"\n    Parameters\n    ----------\n    df: pandas DataFrame.\n        The output data.\n    scaler: sklearn.preprocessing.StandardScaler.\n        The scaler used to normalize the features.\n\n    Returns\n    -------\n    X: numpy array.\n        The preprocessed data.\n    \"\"\"\n    target_name_stand = ['ARRIVAL_DELAY']\n    df = df[target_name_stand]\n\n    ## Standardize y\n    df_stand = df[target_name_stand]\n    df_scaler = scaler.fit(df_stand.values)\n    df_stand = df_scaler.transform(df_stand.values)\n        #replace the columns values\n    df[target_name_stand] = df_stand\n    \n    return df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**## This step can last several minutes. ##**","metadata":{}},{"cell_type":"code","source":"X_scaler = StandardScaler()\nX_train = preprocess_input(X_train_df, X_scaler)\n\ny_scaler = StandardScaler()\ny_train = preprocess_output(y_train_df, y_scaler)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Split the train into train and validation\nX_train_df, X_val_df, y_train_df, y_val_df = train_test_split(X_train, y_train, random_state=1, test_size=0.2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"****","metadata":{}},{"cell_type":"markdown","source":"# IV) Model selection","metadata":{}},{"cell_type":"markdown","source":"**Our model selection will depend on the size of the dataset and its complexity.**","metadata":{}},{"cell_type":"code","source":"X_train_df.info","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We have 42512060 elements divided into 10 columns. This is a huge dataset, however we work on laptops having only CPUs (no GPUs). Therefore:**\n- **No CNN (no GPU)**\n- **No NN (too large dataset)**\n- **Regression --> but feature selction before doing so**\n- **Decision tree (+ different intervals of delay) --> best idea because feature selection done by the tree.\n    https://scikit-learn.org/stable/auto_examples/tree/plot_tree_regression.html#sphx-glr-auto-examples-tree-plot-tree-regression-py --> decision trees can be used to make predictions too !**","metadata":{}},{"cell_type":"markdown","source":"**We begin with a tree regression.**","metadata":{}},{"cell_type":"code","source":"from sklearn.tree import DecisionTreeRegressor\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.metrics import accuracy_score\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_prediction(X, model, scaler):\n    \"\"\"Makes predictions given a preprocessed dataset.\n\n    Parameters\n    ----------\n    X: numpy array.\n        The input data, which already is pre-processed.\n    model: an hypopt or sklearn model.\n        The trained model used for making predictions.\n    scaler: sklearn.preprocessing.StandardScaler.\n        The scaler used to normalize the targets.\n\n    Returns\n    -------\n    y_pred: numpy array.\n        The unnormalized predictions.\n    \"\"\"\n    y_pred = scaler.inverse_transform(model.predict(X))\n    return y_pred","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Now, we can try to predict the validation test thanks to a model trained on the training set. We can evaluate the accuracy of the prediction thanks to the RMSE.**","metadata":{}},{"cell_type":"code","source":"model = DecisionTreeRegressor(max_depth=8)\nmodel.fit(X_train_df, y_train_df)\n\ny_pred_val = make_prediction(X_val_df, model, y_scaler)\ny_pred_train = make_prediction(X_train_df, model, y_scaler)\n\ny_val = y_scaler.inverse_transform(y_val_df['ARRIVAL_DELAY'])\ny_train = y_scaler.inverse_transform(y_train_df['ARRIVAL_DELAY'])\n\n\nRMSE_val = np.sqrt(mean_squared_error(y_pred_val, y_val))\nRMSE_train = np.sqrt(mean_squared_error(y_pred_train, y_train))\n\nprint(\"RMSE training = \"+ str(RMSE_train))\nprint(\"RMSE validation = \"+ str(RMSE_val))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Let's check with different tree depth.**","metadata":{}},{"cell_type":"code","source":"y_val = y_scaler.inverse_transform(y_val_df['ARRIVAL_DELAY'])\ny_train = y_scaler.inverse_transform(y_train_df['ARRIVAL_DELAY'])\n\nrmse_val = []\nrmse_train = []\ndepth = []\n\nfor i in range (1,10) :\n    model = DecisionTreeRegressor(max_depth=i)\n    model.fit(X_train_df, y_train_df)\n\n    y_pred_val = make_prediction(X_val_df, model, y_scaler)\n    y_pred_train = make_prediction(X_train_df, model, y_scaler)\n\n    RMSE_val = np.sqrt(mean_squared_error(y_pred_val, y_val))\n    RMSE_train = np.sqrt(mean_squared_error(y_pred_train, y_train))\n\n    rmse_val.append(RMSE_val)\n    rmse_train.append(RMSE_train)\n    depth.append(i)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.subplot(2,1,1)\nplt.plot(depth, rmse_train)\nplt.title('RMSE training function of depth')\nplt.xlabel('depth')\nplt.ylabel('RMSE training')\n\nplt.subplot(2,1,2)\nplt.plot(depth, rmse_val)\nplt.title('RMSE validation function of depth')\nplt.xlabel('depth')\nplt.ylabel('RMSE validation')\n\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The RMSE for the training set and the validation set are nearly the same. This means that the model generalizes well. Nevertheless, the RMSE are rather large. This seems that the model works poorly.**","metadata":{}},{"cell_type":"markdown","source":"**It can be relevant to use another model.**","metadata":{}},{"cell_type":"markdown","source":"**Test random forest**","metadata":{}},{"cell_type":"code","source":"model = RandomForestRegressor(max_depth = 10)\nmodel.fit(X_train_df, y_train_df)\n\ny_pred_val = make_prediction(X_val_df, model, y_scaler)\ny_pred_train = make_prediction(X_train_df, model, y_scaler)\n\ny_val = y_scaler.inverse_transform(y_val_df['ARRIVAL_DELAY'])\ny_train = y_scaler.inverse_transform(y_train_df['ARRIVAL_DELAY'])\n\n\nRMSE_val = np.sqrt(mean_squared_error(y_pred_val, y_val))\nRMSE_train = np.sqrt(mean_squared_error(y_pred_train, y_train))\n\nprint(\"RMSE training = \"+ str(RMSE_train))\nprint(\"RMSE validation = \"+ str(RMSE_val))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.feature_importances_","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols = ['MONTH', 'DAY',  'DAY_OF_WEEK', 'AIRLINE', 'TAIL_NUMBER', 'ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', 'SCHEDULED_DEPARTURE',\n                'DEPARTURE_DELAY', 'WHEELS_OFF', 'SCHEDULED_TIME',  'DISTANCE', 'SCHEDULED_ARRIVAL', 'CANCELLED', 'CANCELLATION_REASON']\nX_train_df = pd.DataFrame(X_train_df, columns=cols)\n\nsorted_idx = model.feature_importances_.argsort()\nplt.barh(X_train_df.columns[sorted_idx], model.feature_importances_[sorted_idx])\nplt.xlabel(\"Random Forest Feature Importance\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Thanks to the Random Forest Regressor model , we can plot the importance of every feature. In this case, the predominant feature is the DEPARTURE_DELAY by far. It's impressive to notice how the other features are under-used.**","metadata":{}},{"cell_type":"markdown","source":"# V) Model Evaluation","metadata":{}},{"cell_type":"code","source":"#Pre-process the test dataset\nX_test = preprocess_input(X_test_df, X_scaler)\n\n#Predict the delays\ny_pred_test = make_prediction(X_test, model, y_scaler)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a dataframe containing the predictions\nsubmission_df = pd.DataFrame(data={'ID': X_test_df.ID.values,\n                                   'ARRIVAL_DELAY': y_pred_test.squeeze()})\n\n# Save the predictions into a csv file\n# Notice that this file should be saved under the directory `/kaggle/working` \n# so that you can download it later\nsubmission_df.to_csv(\"/kaggle/working/submission.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the submission file\n! head -5 \"/kaggle/working/submission.csv\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","metadata":{},"execution_count":null,"outputs":[]}]}