{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Challenge 1\n# Airplane delay: Analysis and Prediction\n\n## Context\n\nEvery day, in US, there are thousands of flights departures and arrivals (prior to COVID): unfortunately, as you may have noticed yourself, flight delays are not a rare event!!\n\nIn this notebook, we play the role of a data scientist working in the travel industry, specifically on air transportation of passengers. We want to explore the data collected by the Department of Transportation (DoT) to understand passengers' behavior, as well as the properties of all flights, across several airline companies.\n\nNow, given historical data about flights in the country, including the delay information that was computed a-posteriori (so the ground truth is available), we want to build a model that can be used to predict how many minutes of delay a flight might experience in the future. This model should provide useful information for the airport to manage better its resources, to minimize the delays and their impact on the journey of its passengers. Alternatively, astute passengers could even use the model to choose the best time for flying, such as to avoid delays.","metadata":{}},{"cell_type":"markdown","source":"Let's install some python packages used in this notebook","metadata":{}},{"cell_type":"code","source":"! pip install scikit-learn==0.21\n! pip install hypopt","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nimport json\nimport warnings\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.dates as mdates\nimport seaborn as sns\nimport tensorflow as tf\nimport numpy as np\nimport tensorflow_probability as tfp\n\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.linear_model import Ridge\nfrom sklearn.preprocessing import StandardScaler\nfrom hypopt import GridSearch\nfrom matplotlib import rc \n\nif not sys.warnoptions:\n    warnings.simplefilter(\"ignore\")\n\nrc('figure', **{'dpi': 200})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the GPU device name.\ndevice_name = tf.test.gpu_device_name()\n\n# The device name should look like the following:\nif device_name == '/device:GPU:0':\n    print('Found GPU at: {}'.format(device_name))\nelse:\n    raise SystemError('GPU device not found')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\n\n# If there's a GPU available...\nif torch.cuda.is_available():    \n\n    # Tell PyTorch to use the GPU.    \n    device = torch.device(\"cuda\")\n\n    print('There are %d GPU(s) available.' % torch.cuda.device_count())\n\n    print('We will use the GPU:', torch.cuda.get_device_name(0))\n\n# If not...\nelse:\n    print('No GPU available, using the CPU instead.')\n    device = torch.device(\"cpu\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, you can load the training data.","metadata":{}},{"cell_type":"code","source":"X_dtype = {\n    'ID'                   : int,\n    'YEAR'                 : int,  \n    'MONTH'                : int,  \n    'DAY'                  : int,  \n    'DAY_OF_WEEK'          : int,  \n    'AIRLINE'              : str, \n    'FLIGHT_NUMBER'        : int,  \n    'TAIL_NUMBER'          : str, \n    'ORIGIN_AIRPORT'       : str, \n    'DESTINATION_AIRPORT'  : str, \n    'SCHEDULED_DEPARTURE'  : str,  \n    'DEPARTURE_TIME'       : str, \n    'DEPARTURE_DELAY'      : float,\n    'TAXI_OUT'             : int, \n    'WHEELS_OFF'           : str,\n    'SCHEDULED_TIME'       : float,\n    'AIR_TIME'             : float,\n    'DISTANCE'             : int,\n    'SCHEDULED_ARRIVAL'    : str,\n    'DIVERTED'             : int,  \n    'CANCELLED'            : int,  \n    'CANCELLATION_REASON'  : str\n}\n\ny_dtype = {\n    'ID'                   : int,\n    \"ARRIVAL_DELAY\"        : float\n}\n\nX_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_features.csv\", dtype=X_dtype)\ny_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_targets.csv\", dtype=y_dtype)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You may also use additional information about airports and airlines, which can be found in the files `/kaggle/input/eurecom-aml-2021-challenge-1/data/airports.csv` and `/kaggle/input/eurecom-aml-2021-challenge-1/data/airlines.csv`, respectively.","metadata":{}},{"cell_type":"code","source":"# Convert date-time features from str to datetime format\ndef parse_hhmm(x):\n    try: return pd.datetime.strptime(x, '%H%M')#.time()\n    except: return pd.NaT\n\nX_train_df.DEPARTURE_TIME = X_train_df.DEPARTURE_TIME.apply(parse_hhmm)\nX_train_df.SCHEDULED_ARRIVAL = X_train_df.SCHEDULED_ARRIVAL.apply(parse_hhmm)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Merge feature dataframe and target dataframe for data exploration\ndf = pd.merge(X_train_df, y_train_df, on='ID')\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create new columns used for data exploration\ndf['DELAYED'] = (df.ARRIVAL_DELAY > 0).astype(str)\ndf['DATE'] = pd.to_datetime(df[['YEAR', 'MONTH', 'DAY']])\n\nairports = pd.read_csv('/kaggle/input/eurecom-aml-2021-challenge-1/data/airports.csv',\n                    usecols=['IATA_CODE', 'LATITUDE', 'LONGITUDE']).rename({'AIRPORT': 'NAME'}, axis='columns')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data analysis\nIn this challenge, you are free (and encouraged) to explore in depth the data you have.\nNext, you have examples of simple queries on our data, to perform exploration and compute statistics\n\n**NOTE:** finding the right question to ask is difficult! Don't be afraid to complement the questions below, with your own questions that, in your opinion, are valuable ways to inspect data. This can give you extra points!\n\n**NOTE 2:** the presentation quality is critical in any business-oriented data analysis. Take time to create informative plots, rather than endless tables!\n\n- Basic queries:\n  - How many unique origin airports?\n  - How many unique destination airports?\n  - How many carriers?\n  - How many flights that have a scheduled departure time later than 18h00?\n\n\n- Statistics on flight volume: this kind of statistics are helpful to reason about delays. Indeed, it is plausible to assume that \"*the more flights in an airport, the higher the probability of delay*\".\n  - How many flights in each month of the year?\n  - Is there any relationship between the number of flights and the days of week?\n  - How many flights in different days of months and in different hours of days?\n  - Which are the top 20 busiest airports (this depends on inbound and outbound traffic)?\n  - Which are the top 20 busiest carriers?\n\n\n- Statistics on the fraction of delayed flights\n  - What is the percentage of delayed flights (over total flights) for different hours of the day?\n  - Which hours of the day are characterized by the longest flight delay?\n  - What are the fluctuation of the percentage of delayed flights over different time granularities?\n  - What is the percentage of delayed flights which depart from one of the top 20 busiest airports?\n  - What is the percentage of delayed flights which belong to one of the top 20 busiest carriers?","metadata":{}},{"cell_type":"markdown","source":"Below is a very simple example of data exploration.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nsns.histplot(df, x='DEPARTURE_TIME', hue='DELAYED', bins=48, multiple='stack', palette=['C0', 'C3'], ax=ax)\nax.xaxis.set_major_formatter(mdates.DateFormatter(\"%H:%M\"))\nax.set_title('Overview of the flights')\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots()\nsns.histplot(df, x='DATE', hue='DELAYED', bins=48, multiple='stack', palette=['C0', 'C3'], ax=ax)\nax.xaxis.set_major_formatter(mdates.DateFormatter(\"%d/%m\"))\nax.set_title('Seasonal effect on flight delay?')\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_df1 = df.groupby('ORIGIN_AIRPORT').agg({'ARRIVAL_DELAY':'count'}).rename(columns={'ARRIVAL_DELAY': 'COUNT'}).rename_axis('AIRPORT')\n_df2 = df.groupby('DESTINATION_AIRPORT').agg({'ARRIVAL_DELAY':'count'}).rename(columns={'ARRIVAL_DELAY': 'COUNT'}).rename_axis('AIRPORT')\ntop_airports = _df1.join(_df2, rsuffix='_ORIGIN', lsuffix='_DEST').sum(axis=1).sort_values(ascending=False).index[:15]\ntop_airports","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['DEPARTURE_DATE'] = pd.to_datetime(df.DATE.dropna().dt.date.astype(str) + ' ' + df.DEPARTURE_TIME.dropna().dt.time.astype(str))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"query = df[(df.TAIL_NUMBER=='N407AS') & (df.DEPARTURE_DATE > '2015-07-05') & (df.DEPARTURE_DATE < '2015-07-31')]\nquery = df[(df.TAIL_NUMBER=='N594HA')]\nsns.lineplot(data=query, x='DEPARTURE_DATE', y='ARRIVAL_DELAY', markers=['o'],)\nplt.xticks(rotation=45)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import statsmodels.graphics.tsaplots\nquery = df[(df.TAIL_NUMBER=='N594HA')].sort_values('DEPARTURE_TIME').ARRIVAL_DELAY.dropna()\nstatsmodels.graphics.tsaplots.plot_acf(query.values.squeeze(), )\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.groupby('TAIL_NUMBER')['ARRIVAL_DELAY'].count().sort_values(ascending=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.sort_values('DEPARTURE_TIME').groupby('TAIL_NUMBER')['ARRIVAL_DELAY'].apply(pd.Series.autocorr, lag=1).dropna().sort_values(ascending=False)[10:]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"query = df[df.TAIL_NUMBER == 'N594HA']\n# query\nquery = pd.merge(query, airports.add_prefix(\"ORIGIN_\"), left_on='ORIGIN_AIRPORT', right_on='ORIGIN_IATA_CODE', how='left')\nquery = pd.merge(query, airports.add_prefix(\"DESTINATION_\"), left_on='DESTINATION_AIRPORT', right_on='DESTINATION_IATA_CODE', how='left')\nquery = query.groupby(['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', \n               'ORIGIN_LATITUDE', 'ORIGIN_LONGITUDE', \n               'DESTINATION_LATITUDE', 'DESTINATION_LONGITUDE'])['ARRIVAL_DELAY'].mean().reset_index().sort_values('ARRIVAL_DELAY')\nquery","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport matplotlib\nfrom mpl_toolkits.basemap import Basemap\n\nnorm = matplotlib.colors.Normalize(vmin=0.0, vmax=100.0)\ncmap = matplotlib.cm.get_cmap('RdYlGn_r')\n\nfig, ax = plt.subplots(figsize=[6, 3], dpi=300)\n\nm = Basemap(projection='cyl',\n            llcrnrlat=10, urcrnrlat=50,\n            llcrnrlon=-165, urcrnrlon=-110,)\nm.shadedrelief()\nm.drawcountries()\n\n\nfor i, flight in query.iterrows():\n    x, y = m([flight.ORIGIN_LONGITUDE, flight.DESTINATION_LONGITUDE], \n               [flight.ORIGIN_LATITUDE, flight.DESTINATION_LATITUDE])\n\n    ax.annotate('', xy=(x[0], y[0]),  xycoords='data',\n                xytext=(x[1], y[1]), textcoords='data',\n                arrowprops=dict(arrowstyle=\"->\", \n                                color=cmap(norm(flight.ARRIVAL_DELAY)), lw=1, \n                                connectionstyle=\"angle3,angleA=0,angleB=-135\"))\n\nfig.colorbar(matplotlib.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax)\nax.set_title('Flight paths and delays of N594HA')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"query = df[df.TAIL_NUMBER == 'N594HA']\nquery = pd.merge(query, airports.add_prefix(\"ORIGIN_\"), left_on='ORIGIN_AIRPORT', right_on='ORIGIN_IATA_CODE', how='left')\nquery = pd.merge(query, airports.add_prefix(\"DESTINATION_\"), left_on='DESTINATION_AIRPORT', right_on='DESTINATION_IATA_CODE', how='left')\nquery = query.groupby(['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', \n               'ORIGIN_LATITUDE', 'ORIGIN_LONGITUDE', \n               'DESTINATION_LATITUDE', 'DESTINATION_LONGITUDE'])['ARRIVAL_DELAY'].mean().reset_index().sort_values('ARRIVAL_DELAY')\nquery = pd.merge(query, query, left_on=['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT'], right_on=['DESTINATION_AIRPORT', 'ORIGIN_AIRPORT'], how='inner')\nquery = query[['ORIGIN_AIRPORT_x', 'DESTINATION_AIRPORT_x', 'ORIGIN_AIRPORT_y', 'DESTINATION_AIRPORT_y', 'ORIGIN_LATITUDE_x', 'ORIGIN_LONGITUDE_x',\n               'DESTINATION_LATITUDE_x', 'DESTINATION_LONGITUDE_x', 'ARRIVAL_DELAY_x', 'ARRIVAL_DELAY_y']]\n\nquery['DELAY'] = query.ARRIVAL_DELAY_x + query.ARRIVAL_DELAY_y\nquery = query.sort_values('DELAY').iloc[::2]\n\nquery[['ORIGIN_AIRPORT_x', 'DESTINATION_AIRPORT_x', 'ORIGIN_AIRPORT_y', 'DESTINATION_AIRPORT_y', 'DELAY']]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"norm = matplotlib.colors.Normalize(vmin=0.0, vmax=120.0)\ncmap = matplotlib.cm.get_cmap('RdYlGn_r')\n\nfig, ax = plt.subplots(figsize=[6, 3], dpi=300)\n\nm = Basemap(projection='cyl',\n            llcrnrlat=15, urcrnrlat=50,\n            llcrnrlon=-165, urcrnrlon=-110,)\nm.shadedrelief()\nm.drawcountries()\n\n\nfor i, flight in query.iterrows():\n    x, y = m([flight.ORIGIN_LONGITUDE_x, flight.DESTINATION_LONGITUDE_x], \n               [flight.ORIGIN_LATITUDE_x, flight.DESTINATION_LATITUDE_x])\n    ax.plot(x, y, 'ok', ms=1)\n    ax.annotate('', xy=(x[0], y[0]),  xycoords='data',\n                xytext=(x[1], y[1]), textcoords='data',\n                arrowprops=dict(arrowstyle=\"<->\", \n                                color=cmap(norm(flight.DELAY)), lw=1, ))\n\nfig.colorbar(matplotlib.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax)\nax.set_title('Connection route and delays of N594HA')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Adding necessary features according to data analysis","metadata":{}},{"cell_type":"markdown","source":"The major challenge with this dataset was feature encoding. Usually we use ordinal features, which are easy to implement. But this dataset contained a lot of categorical values, and most of them were not ordinal. We had to find strategies to implement them in the most efficient way. We first looked at 1oK encoding, but given the cardinality of the features, we quickly gave up this strategy. You will find below some tries we made to encode our variables.","metadata":{}},{"cell_type":"code","source":"# Sum is a new features taking into account the flight whole delay\n\ndf['SUM']= df['DEPARTURE_DELAY']+ df['TAXI_OUT']+df['AIR_TIME']\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We tried a strategy to encode the flight number ,the tail number, ...\n# Compute mean delay for all the flights with the same departure hour, tail number and flight number\n# Leads to overfitting -> not use in the final version\n\ndf['HOUR'] = df['DEPARTURE_TIME'].dt.hour\ndf_mean = df[['HOUR', 'FLIGHT_NUMBER', 'TAIL_NUMBER', 'ARRIVAL_DELAY']].groupby(\n    ['HOUR', 'FLIGHT_NUMBER', 'TAIL_NUMBER'], as_index=False).mean()\n\ndf_mean.rename(columns={'ARRIVAL_DELAY':'MEAN_ARRIVAL'}, inplace=True)\n\ndf = df.merge(df_mean, on=['HOUR', 'FLIGHT_NUMBER', 'TAIL_NUMBER'], how='left')\n\ndf_mean_2 = df[['FLIGHT_NUMBER', 'TAIL_NUMBER', 'ARRIVAL_DELAY']].groupby(['FLIGHT_NUMBER', 'TAIL_NUMBER'],\n                                                                          as_index=False).mean()\ndf_mean_2.rename(columns={'ARRIVAL_DELAY':'MEAN_ARRIVAL_2'}, inplace=True)\ndf = df.merge(df_mean_2, on=['FLIGHT_NUMBER', 'TAIL_NUMBER'], how='left')\ndf['MEAN_ARRIVAL'].fillna(df['MEAN_ARRIVAL_2'], inplace=True)\ndf.drop('MEAN_ARRIVAL_2', axis=1, inplace=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# To encode the airports we tried to use the additional dataset. But encoding them with latitude and\n# didn't improve our model predictions\n# Fill missing values from airports with online found data\n\nairports = pd.read_csv('/kaggle/input/eurecom-aml-2021-challenge-1/data/airports.csv',\n                    usecols=['IATA_CODE', 'LATITUDE', 'LONGITUDE'])\n\nairports.loc[airports['IATA_CODE']=='ECP', 'LATITUDE'] = 30\nairports.loc[airports['IATA_CODE']=='ECP', 'LONGITUDE'] = -85\nairports.loc[airports['IATA_CODE']=='PBG', 'LATITUDE'] = 44\nairports.loc[airports['IATA_CODE']=='PBG', 'LONGITUDE'] = -73\nairports.loc[airports['IATA_CODE']=='UST', 'LATITUDE'] = 29\nairports.loc[airports['IATA_CODE']=='UST', 'LONGITUDE'] = -81\n\ndf_airports = df[['FLIGHT_NUMBER', 'ORIGIN_AIRPORT', 'DESTINATION_AIRPORT']].drop_duplicates()\n\ndf_airports = df_airports.merge(airports, how='left', left_on=['ORIGIN_AIRPORT'], \n              right_on=['IATA_CODE']).drop(['IATA_CODE'], axis=1)\ndf_airports.rename(columns={'LATITUDE':'ORIGIN_LATITUDE', 'LONGITUDE':'ORIGIN_LONGITUDE'}, inplace=True)\n\ndf_airports = df_airports.merge(airports, how='left', left_on=['DESTINATION_AIRPORT'], \n              right_on=['IATA_CODE']).drop(['IATA_CODE'], axis=1)\ndf_airports.rename(columns={'LATITUDE':'DESTINATION_LATITUDE', 'LONGITUDE':'DESTINATION_LONGITUDE'}, \n                   inplace=True)\n\ndf_airports.dropna(inplace=True)\ndf_airports.drop_duplicates(inplace=True)\n\ndf = df.merge(df_airports, on=['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', 'FLIGHT_NUMBER'], how='left')\n\ndf_airports = df_airports[['FLIGHT_NUMBER', 'DESTINATION_LATITUDE', 'DESTINATION_LONGITUDE',\n                          'ORIGIN_LATITUDE', 'ORIGIN_LONGITUDE']].drop_duplicates()\n\ndf_airports = df_airports.groupby('FLIGHT_NUMBER').agg({'DESTINATION_LATITUDE' : lambda x:x.value_counts().index[0],\n                                                       'DESTINATION_LONGITUDE' : lambda x:x.value_counts().index[0],\n                                                        'ORIGIN_LATITUDE' : lambda x:x.value_counts().index[0],\n                                                       'ORIGIN_LONGITUDE' : lambda x:x.value_counts().index[0]})\n\ndf_airports.rename(columns={'DESTINATION_LATITUDE' : 'DEST_LAT', 'DESTINATION_LONGITUDE' : 'DEST_LON',\n                           'ORIGIN_LATITUDE': 'ORG_LAT', 'ORIGIN_LONGITUDE': 'ORG_LON'}, inplace=True)\n\ndf = df.merge(df_airports, on=['FLIGHT_NUMBER'], how='left')\n\ndf['ORIGIN_LATITUDE'].fillna(df['ORG_LAT'], inplace=True)\ndf.drop('ORG_LAT', axis=1, inplace=True)\ndf['ORIGIN_LONGITUDE'].fillna(df['ORG_LON'], inplace=True)\ndf.drop('ORG_LON', axis=1, inplace=True)\n\ndf['DESTINATION_LATITUDE'].fillna(df['DEST_LAT'], inplace=True)\ndf.drop('DEST_LAT', axis=1, inplace=True)\ndf['DESTINATION_LONGITUDE'].fillna(df['DEST_LON'], inplace=True)\ndf.drop('DEST_LON', axis=1, inplace=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The small cardinality of the feature Airline, allowed us to use dummies encoding\ndf = pd.get_dummies(df, columns=['AIRLINE'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert the arrival time and departure time into minutes since the midnight\ndef minutes_since_midnight(dt):\n    return dt.hour * 60 + dt.minute\n\ndf.SCHEDULED_ARRIVAL = df.SCHEDULED_ARRIVAL.apply(minutes_since_midnight)\ndf.DEPARTURE_TIME = df.DEPARTURE_TIME.apply(minutes_since_midnight)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features = ['ARRIVAL_DELAY', 'SUM','DAY_OF_WEEK', 'SCHEDULED_TIME', 'SCHEDULED_ARRIVAL', 'MEAN_ARRIVAL',\n           'DESTINATION_LATITUDE', 'DESTINATION_LONGITUDE', 'ORIGIN_LATITUDE', 'ORIGIN_LONGITUDE']\nsns.heatmap(df[features].corr())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Separating features and target \n\nX_train_df = df.drop('ARRIVAL_DELAY', axis=1)\n\ny_train_df = df[['ID', 'ARRIVAL_DELAY']]\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Pre-processing\n\nThe previous step should give you a better understanding of which pre-processing is required for the data.\nThis may include:\n\n- Normalising and standardising the given data;\n- Removing outliers;\n- Carrying out feature selection, possibly using metrics derived from information theory;\n- Handling missing information in the dataset;\n- Augmenting the dataset with external information;\n- Combining existing features.","metadata":{}},{"cell_type":"markdown","source":"Below is a very basic example of pre-processing steps.","metadata":{}},{"cell_type":"code","source":"# First of all, you should split the data into a training set and a validation set\n\nX_train_df, X_val_df, y_train_df, y_val_df = train_test_split(\n    X_train_df, y_train_df, random_state=1, test_size=0.2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Selection of features as follows\nfeature_names = ['DAY_OF_WEEK', 'SCHEDULED_ARRIVAL', 'SUM', 'SCHEDULED_TIME', 'DIVERTED']\ntarget_name = ['ARRIVAL_DELAY']\n\n\nX_train_df = X_train_df[feature_names]\ny_train_df = y_train_df[target_name]\n\nX_val_df = X_val_df[feature_names]\ny_val_df = y_val_df[target_name]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Filling missing values by the mean along each column.\n# These statistics should be estimated by using the training set.\nimputer = SimpleImputer(missing_values=np.nan, strategy='mean')\nimputer.fit(X_train_df)\n\nX_train = imputer.transform(X_train_df)\nX_val = imputer.transform(X_val_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Standardize the features by removing the mean and scaling to unit variance\n# Similarly to the preivous step, the statistics used for standardization\n# should be computed across the training set only\nX_scaler = StandardScaler()\nX_scaler.fit(X_train)\n\nX_train = X_scaler.transform(X_train)\nX_val = X_scaler.transform(X_val)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We should also standardize the targets.\ny_scaler = StandardScaler()\ny_scaler.fit(y_train_df)\n\ny_train = y_scaler.transform(y_train_df)\ny_val = y_scaler.transform(y_val_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can define a wrapper function that performs the pipeline of pre-processing data. You should modify this function according to your pipeline.","metadata":{}},{"cell_type":"code","source":"def preprocess_data(df, feature_names, imputer, scaler):\n    \"\"\"Preprocess data.\n\n    Parameters\n    ----------\n    df: pandas DataFrame.\n        The input data.\n    feature_names: list of strings.\n        The names of selected features.\n    imputer: sklearn.impute.SimpleImputer\n        The imputation transformer for completing missing values.\n    scaler: sklearn.preprocessing.StandardScaler.\n        The scaler used to normalize the features.\n\n    Returns\n    -------\n    X: numpy array.\n        The preprocessed data.\n    \"\"\"\n    # Select features\n    \n    \n    X_df = df.copy()\n    X_df['SUM']= X_df['DEPARTURE_DELAY']+X_df['TAXI_OUT']+X_df['AIR_TIME']\n    \n    X_df = X_df[feature_names]\n    \n    # Pre-process datetime features\n    \n    X_df.SCHEDULED_ARRIVAL = X_df.SCHEDULED_ARRIVAL.apply(parse_hhmm)\n    X_df.SCHEDULED_ARRIVAL = X_df.SCHEDULED_ARRIVAL.apply(minutes_since_midnight)\n    \n    # Impute missing values\n    X = imputer.transform(X_df)\n    \n    # Normalize features\n    X = scaler.transform(X)\n\n    return X","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can define a wrapper function used to make predictions given a pre-processed dataset. You can freely modify this function according to your approach.","metadata":{}},{"cell_type":"code","source":"def make_prediction(X, model, scaler):\n    \"\"\"Makes predictions given a preprocessed dataset.\n\n    Parameters\n    ----------\n    X: numpy array.\n        The input data, which already is pre-processed.\n    model: an hypopt or sklearn model.\n        The trained model used for making predictions.\n    scaler: sklearn.preprocessing.StandardScaler.\n        The scaler used to normalize the targets.\n\n    Returns\n    -------\n    y_pred: numpy array.\n        The unnormalized predictions.\n    \"\"\"\n    y_pred = scaler.inverse_transform(model.predict(X))\n    return y_pred","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After training the model, we can evaluate the performance of the model on the training and validation sets in terms of root mean squared error (RMSE).","metadata":{}},{"cell_type":"markdown","source":"## Model Selection","metadata":{}},{"cell_type":"markdown","source":"We tried several regression models. The best one was Random Forest.","metadata":{}},{"cell_type":"code","source":"#RandomForest\n\nfrom sklearn.ensemble import RandomForestRegressor\nregr = RandomForestRegressor()\nregr.fit(X_train, y_train)\n\ny_train_pred = make_prediction(X_train, regr, y_scaler)\ntrain_rmse = np.sqrt(mean_squared_error(y_train_pred, y_train_df.values))\nprint(\"Training RMSE: {:.5f}\".format(float(train_rmse)))\n\ny_val_pred = make_prediction(X_val, regr, y_scaler)\nval_rmse = np.sqrt(mean_squared_error(y_val_pred, y_val_df.values))\nprint(\"Validation RMSE: {:.5f}\".format(float(val_rmse)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#ExtraTrees\n\nfrom sklearn.ensemble import ExtraTreesRegressor\nregExtra = ExtraTreesRegressor().fit(X_train, y_train)\n\ny_train_pred = make_prediction(X_train, regExtra, y_scaler)\ntrain_rmse = np.sqrt(mean_squared_error(y_train_pred, y_train_df.values))\nprint(\"Training RMSE: {:.5f}\".format(float(train_rmse)))\n\ny_val_pred = make_prediction(X_val, regExtra, y_scaler)\nval_rmse = np.sqrt(mean_squared_error(y_val_pred, y_val_df.values))\nprint(\"Validation RMSE: {:.5f}\".format(float(val_rmse)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#SVR\n\nfrom sklearn.svm import SVR\n\nmodel = SVR(kernel='rbf', gamma=\"auto\")\nmodel.fit(X_train, y_train)\n\ny_train_pred = make_prediction(X_train, model, y_scaler)\ntrain_rmse = np.sqrt(mean_squared_error(y_train_pred, y_train_df.values))\nprint(\"Training RMSE: {:.5f}\".format(float(train_rmse)))\n\ny_val_pred = make_prediction(X_val, model, y_scaler)\nval_rmse = np.sqrt(mean_squared_error(y_val_pred, y_val_df.values))\nprint(\"Validation RMSE: {:.5f}\".format(float(val_rmse)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#XGB\n\nimport xgboost as xgb\nxgbr = xgb.XGBRegressor(verbosity=0) \nxgbr.fit(X_train, y_train) \n\ny_train_pred = make_prediction(X_train, xgbr, y_scaler)\ntrain_rmse = np.sqrt(mean_squared_error(y_train_pred, y_train_df.values))\nprint(\"Training RMSE: {:.5f}\".format(float(train_rmse)))\n\ny_val_pred = make_prediction(X_val, xgbr, y_scaler)\nval_rmse = np.sqrt(mean_squared_error(y_val_pred, y_val_df.values))\nprint(\"Validation RMSE: {:.5f}\".format(float(val_rmse)))\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Evaluation\n\nSome form of pre-evaluation will inevitably be required in the preceding sections in order to both select an appropriate model and configure its parameters appropriately.\nIn this final section, you may evaluate other aspects of the model such as:\n\n- Assessing the running time of your model;\n- Determining whether some aspects can be parallelised;\n- Training the model with smaller subsets of the data.\n- etc.\n\nThe goal of this challenge is to construct a model for predicting arrival delays\n\n\nYour submission is a CSV file containing your <b>final model's predictions on the given test data</b>. \n    This file should contain a header and have the following format:\n    \n    ```\n    ID,ARRIVAL_DELAY\n    1561528,30.497905409756292\n    845015,32.75533601062071\n    717459,34.758212570279404\n    451496,32.73011833829676\n    etc.\n    ```\n\nA leaderboard for this challenge will be ranked using the root mean squared error between the predicted values and the observed arrival delays. However, you can use other metrics for regression tasks in your presentation notebook to evaluate many aspects of your model.","metadata":{}},{"cell_type":"markdown","source":"Below is an example of creating a submission file.","metadata":{}},{"cell_type":"code","source":"# Load the test data\nX_test_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/test_features.csv\", dtype=X_dtype)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Preprocessing data\nX_test = preprocess_data(X_test_df, feature_names, imputer, X_scaler)\n\n# Make predictions\ny_test_pred = make_prediction(X_test, regr, y_scaler)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a dataframe containing the predictions\nsubmission_df = pd.DataFrame(data={'ID': X_test_df.ID.values,\n                                   'ARRIVAL_DELAY': y_test_pred.squeeze()})\n\n# Save the predictions into a csv file\n# Notice that this file should be saved under the directory `/kaggle/working` \n# so that you can download it later\nsubmission_df.to_csv(\"/kaggle/working/submission.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the submission file\n! head -5 \"/kaggle/working/submission.csv\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}