{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Challenge 1\n# Airplane delay: Analysis and Prediction\n\n## Context\n\nEvery day, in US, there are thousands of flights departures and arrivals (prior to COVID): unfortunately, as you may have noticed yourself, flight delays are not a rare event!!\n\nIn this notebook, we play the role of a data scientist working in the travel industry, specifically on air transportation of passengers. We want to explore the data collected by the Department of Transportation (DoT) to understand passengers' behavior, as well as the properties of all flights, across several airline companies.\n\nNow, given historical data about flights in the country, including the delay information that was computed a-posteriori (so the ground truth is available), we want to build a model that can be used to predict how many minutes of delay a flight might experience in the future. This model should provide useful information for the airport to manage better its resources, to minimize the delays and their impact on the journey of its passengers. Alternatively, astute passengers could even use the model to choose the best time for flying, such as to avoid delays.","metadata":{}},{"cell_type":"markdown","source":"Let's install some python packages used in this notebook","metadata":{}},{"cell_type":"code","source":"! pip install scikit-learn==0.21\n! pip install hypopt","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nimport json\nimport warnings\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.dates as mdates\nimport seaborn as sns\nimport tensorflow as tf\nimport numpy as np\nimport tensorflow_probability as tfp\n\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.linear_model import Ridge\nfrom sklearn.preprocessing import StandardScaler\nfrom hypopt import GridSearch\nfrom matplotlib import rc \n\nif not sys.warnoptions:\n    warnings.simplefilter(\"ignore\")\n\nrc('figure', **{'dpi': 200})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the GPU device name.\ndevice_name = tf.test.gpu_device_name()\n\n# The device name should look like the following:\nif device_name == '/device:GPU:0':\n    print('Found GPU at: {}'.format(device_name))\nelse:\n    raise SystemError('GPU device not found')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\n\n# If there's a GPU available...\nif torch.cuda.is_available():    \n\n    # Tell PyTorch to use the GPU.    \n    device = torch.device(\"cuda\")\n\n    print('There are %d GPU(s) available.' % torch.cuda.device_count())\n\n    print('We will use the GPU:', torch.cuda.get_device_name(0))\n\n# If not...\nelse:\n    print('No GPU available, using the CPU instead.')\n    device = torch.device(\"cpu\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, you can load the training data.","metadata":{}},{"cell_type":"code","source":"X_dtype = {\n    'ID'                   : int,\n    'YEAR'                 : int,  \n    'MONTH'                : int,  \n    'DAY'                  : int,  \n    'DAY_OF_WEEK'          : int,  \n    'AIRLINE'              : str, \n    'FLIGHT_NUMBER'        : str,  \n    'TAIL_NUMBER'          : str, \n    'ORIGIN_AIRPORT'       : str, \n    'DESTINATION_AIRPORT'  : str, \n    'SCHEDULED_DEPARTURE'  : str,  \n    'DEPARTURE_TIME'       : str, \n    'DEPARTURE_DELAY'      : float,\n    'TAXI_OUT'             : int, \n    'WHEELS_OFF'           : str,\n    'SCHEDULED_TIME'       : float,\n    'AIR_TIME'             : float,\n    'DISTANCE'             : int,\n    'SCHEDULED_ARRIVAL'    : str,\n    'DIVERTED'             : int,  \n    'CANCELLED'            : int,  \n    'CANCELLATION_REASON'  : str\n}\n\ny_dtype = {\n    'ID'                   : int,\n    \"ARRIVAL_DELAY\"        : float\n}\n\nX_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_features.csv\", dtype=X_dtype)\ny_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_targets.csv\", dtype=y_dtype)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You may also use additional information about airports and airlines, which can be found in the files `/kaggle/input/eurecom-aml-2021-challenge-1/data/airports.csv` and `/kaggle/input/eurecom-aml-2021-challenge-1/data/airlines.csv`, respectively.","metadata":{}},{"cell_type":"code","source":"# Convert date-time features from str to datetime format\ndef parse_hhmm(x):\n    try: return pd.datetime.strptime(x, '%H%M')#.time()\n    except: return pd.NaT","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Merge feature dataframe and target dataframe for data exploration\ndf = pd.merge(X_train_df, y_train_df, on='ID')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create new columns used for data exploration : mean arrival delay\n\ndf['DEPARTURE_TIME'] = df['DEPARTURE_TIME'].apply(parse_hhmm)\ndf['HOUR'] = df['DEPARTURE_TIME'].dt.hour\ndf_mean = df[['HOUR', 'FLIGHT_NUMBER', 'TAIL_NUMBER', 'ARRIVAL_DELAY']].groupby(\n    ['HOUR', 'FLIGHT_NUMBER', 'TAIL_NUMBER'], as_index=False).mean()\n\ndf_mean.rename(columns={'ARRIVAL_DELAY':'MEAN_ARRIVAL'}, inplace=True)\n\ndf = df.merge(df_mean, on=['HOUR', 'FLIGHT_NUMBER', 'TAIL_NUMBER'], how='left')\n\ndf_mean_2 = df[['FLIGHT_NUMBER', 'TAIL_NUMBER', 'ARRIVAL_DELAY']].groupby(['FLIGHT_NUMBER', 'TAIL_NUMBER'],\n                                                                          as_index=False).mean()\ndf_mean_2.rename(columns={'ARRIVAL_DELAY':'MEAN_ARRIVAL_2'}, inplace=True)\ndf = df.merge(df_mean_2, on=['FLIGHT_NUMBER', 'TAIL_NUMBER'], how='left')\ndf['MEAN_ARRIVAL'].fillna(df['MEAN_ARRIVAL_2'], inplace=True)\ndf.drop('MEAN_ARRIVAL_2', axis=1, inplace=True)\n\n\n# Load the test data \nX_test_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/test_features.csv\", dtype=X_dtype)\nX_test_df['DEPARTURE_TIME'] = X_test_df['DEPARTURE_TIME'].apply(parse_hhmm)\nX_test_df['HOUR'] = X_test_df['DEPARTURE_TIME'].dt.hour\nX_test_df = X_test_df.merge(df_mean, on=['HOUR', 'FLIGHT_NUMBER', 'TAIL_NUMBER'], how='left')\n\ndel df_mean\n\nX_test_df = X_test_df.merge(df_mean_2, on=['FLIGHT_NUMBER', 'TAIL_NUMBER'], how='left')\nX_test_df['MEAN_ARRIVAL'].fillna(X_test_df['MEAN_ARRIVAL_2'], inplace=True)\nX_test_df.drop('MEAN_ARRIVAL_2', axis=1, inplace=True)\n\ndf_mean_2 = df_mean_2[['FLIGHT_NUMBER', 'MEAN_ARRIVAL_2']].groupby(['FLIGHT_NUMBER'], as_index=False).mean()\n\n\nX_test_df = X_test_df.merge(df_mean_2, on=['FLIGHT_NUMBER'], how='left')\nX_test_df['MEAN_ARRIVAL'].fillna(X_test_df['MEAN_ARRIVAL_2'], inplace=True)\nX_test_df.drop('MEAN_ARRIVAL_2', axis=1, inplace=True)\n\nprint('Missing values on train dataframe:', df.isna().sum())\nprint('Missing values on test dataframe:', X_test_df.isna().sum())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\ndel df_mean_2\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create new columns used for data exploration : origin and arrival airport info\n\nairports = pd.read_csv('/kaggle/input/eurecom-aml-2021-challenge-1/data/airports.csv',\n                    usecols=['IATA_CODE', 'LATITUDE', 'LONGITUDE']).rename({'AIRPORT': 'NAME'}, axis='columns')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fill missing data on airports with web information\nairports.loc[airports['IATA_CODE']=='ECP', 'LATITUDE'] = 30\nairports.loc[airports['IATA_CODE']=='ECP', 'LONGITUDE'] = -85\nairports.loc[airports['IATA_CODE']=='PBG', 'LATITUDE'] = 44\nairports.loc[airports['IATA_CODE']=='PBG', 'LONGITUDE'] = -73\nairports.loc[airports['IATA_CODE']=='UST', 'LATITUDE'] = 29\nairports.loc[airports['IATA_CODE']=='UST', 'LONGITUDE'] = -81\n\ndf_airports = df[['FLIGHT_NUMBER', 'ORIGIN_AIRPORT', 'DESTINATION_AIRPORT']].drop_duplicates()\n\ndf_airports = df_airports.merge(airports, how='left', left_on=['ORIGIN_AIRPORT'],\n              right_on=['IATA_CODE']).drop(['IATA_CODE','NAME'], axis=1)\ndf_airports.rename(columns={'LATITUDE':'ORIGIN_LATITUDE', 'LONGITUDE':'ORIGIN_LONGITUDE'}, inplace=True)\n\ndf_airports = df_airports.merge(airports, how='left', left_on=['DESTINATION_AIRPORT'],\n              right_on=['IATA_CODE']).drop(['IATA_CODE','NAME'], axis=1)\ndf_airports.rename(columns={'LATITUDE':'DESTINATION_LATITUDE', 'LONGITUDE':'DESTINATION_LONGITUDE'}, inplace=True)\n\ndel airports\ngc.collect()\n\ndf_airports.dropna(inplace=True)\ndf_airports.drop_duplicates(inplace=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_airports","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.merge(df_airports, on=['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT'], how='left')\nX_test_df = X_test_df.merge(df_airports, on=['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT'], how='left')\n\ndf_airports = df_airports[['FLIGHT_NUMBER', 'DESTINATION_LATITUDE', 'DESTINATION_LONGITUDE',\n                          'ORIGIN_LATITUDE', 'ORIGIN_LONGITUDE']].drop_duplicates()\n\ndf_airports = df_airports.group_by('FLIGHT_NUMBER').agg({'DESTINATION_LATITUDE' : lambda x:x.value_counts().index[0],\n                                                       'DESTINATION_LONGITUDE' : lambda x:x.value_counts().index[0],\n                                                        'ORIGIN_LATITUDE' : lambda x:x.value_counts().index[0],\n                                                       'ORIGIN_LONGITUDE' : lambda x:x.value_counts().index[0]})\n\ndf_airports.rename(columns={'DESTINATION_LATITUDE' : 'DEST_LAT', 'DESTINATION_LONGITUDE' : 'DEST_LONG',\n                           'ORIGIN_LATITUDE': 'ORG_LAT', 'ORIGIN_LONGITUDE': 'ORG_LON'}, inplace=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.merge(df_airports, on=['FLIGHT_NUMBER'], how='left')\nX_test_df = X_test_df.merge(df_airports, on=['FLIGHT_NUMBER'], how='left')\n\ndf['ORIGIN_LATITUDE'].fillna(df['ORG_LAT'], inplace=True)\ndf.drop('ORG_LAT', axis=1, inplace=True)\ndf['ORIGIN_LONGITUDE'].fillna(df['ORG_LON'], inplace=True)\ndf.drop('ORG_LON', axis=1, inplace=True)\n\ndf['DESTINATION_LATITUDE'].fillna(df['DEST_LAT'], inplace=True)\ndf.drop('DEST_LAT', axis=1, inplace=True)\ndf['DESTINATION_LONGITUDE'].fillna(df['DEST_LON'], inplace=True)\ndf.drop('DEST_LON', axis=1, inplace=True)\n\nX_test_df['ORIGIN_LATITUDE'].fillna(X_test_df['ORG_LAT'], inplace=True)\nX_test_df.drop('ORG_LAT', axis=1, inplace=True)\nX_test_df['ORIGIN_LONGITUDE'].fillna(X_test_df['ORG_LON'], inplace=True)\nX_test_df.drop('ORG_LON', axis=1, inplace=True)\n\nX_test_df['DESTINATION_LATITUDE'].fillna(X_test_df['DEST_LAT'], inplace=True)\nX_test_df.drop('DEST_LAT', axis=1, inplace=True)\nX_test_df['DESTINATION_LONGITUDE'].fillna(X_test_df['DEST_LON'], inplace=True)\nX_test_df.drop('DEST_LON', axis=1, inplace=True)\n\n\nprint('Missing values on train dataframe:', df.isna().sum())\nprint('Missing values on test dataframe:', X_test_df.isna().sum())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data analysis\nIn this challenge, you are free (and encouraged) to explore in depth the data you have.\nNext, you have examples of simple queries on our data, to perform exploration and compute statistics\n\n**NOTE:** finding the right question to ask is difficult! Don't be afraid to complement the questions below, with your own questions that, in your opinion, are valuable ways to inspect data. This can give you extra points!\n\n**NOTE 2:** the presentation quality is critical in any business-oriented data analysis. Take time to create informative plots, rather than endless tables!\n\n- Basic queries:\n  - How many unique origin airports?\n  - How many unique destination airports?\n  - How many carriers?\n  - How many flights that have a scheduled departure time later than 18h00?\n\n\n- Statistics on flight volume: this kind of statistics are helpful to reason about delays. Indeed, it is plausible to assume that \"*the more flights in an airport, the higher the probability of delay*\".\n  - How many flights in each month of the year?\n  - Is there any relationship between the number of flights and the days of week?\n  - How many flights in different days of months and in different hours of days?\n  - Which are the top 20 busiest airports (this depends on inbound and outbound traffic)?\n  - Which are the top 20 busiest carriers?\n\n\n- Statistics on the fraction of delayed flights\n  - What is the percentage of delayed flights (over total flights) for different hours of the day?\n  - Which hours of the day are characterized by the longest flight delay?\n  - What are the fluctuation of the percentage of delayed flights over different time granularities?\n  - What is the percentage of delayed flights which depart from one of the top 20 busiest airports?\n  - What is the percentage of delayed flights which belong to one of the top 20 busiest carriers?","metadata":{}},{"cell_type":"markdown","source":"Below is a very simple example of data exploration.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots()\nsns.histplot(df, x='DEPARTURE_TIME', hue='DELAYED', bins=48, multiple='stack', palette=['C0', 'C3'], ax=ax)\nax.xaxis.set_major_formatter(mdates.DateFormatter(\"%H:%M\"))\nax.set_title('Overview of the flights')\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots()\nsns.histplot(df, x='DATE', hue='DELAYED', bins=48, multiple='stack', palette=['C0', 'C3'], ax=ax)\nax.xaxis.set_major_formatter(mdates.DateFormatter(\"%d/%m\"))\nax.set_title('Seasonal effect on flight delay?')\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_df1 = df.groupby('ORIGIN_AIRPORT').agg({'ARRIVAL_DELAY':'count'}).rename(columns={'ARRIVAL_DELAY': 'COUNT'}).rename_axis('AIRPORT')\n_df2 = df.groupby('DESTINATION_AIRPORT').agg({'ARRIVAL_DELAY':'count'}).rename(columns={'ARRIVAL_DELAY': 'COUNT'}).rename_axis('AIRPORT')\ntop_airports = _df1.join(_df2, rsuffix='_ORIGIN', lsuffix='_DEST').sum(axis=1).sort_values(ascending=False).index[:15]\ntop_airports","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['DEPARTURE_DATE'] = pd.to_datetime(df.DATE.dropna().dt.date.astype(str) + ' ' + df.DEPARTURE_TIME.dropna().dt.time.astype(str))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"query = df[(df.TAIL_NUMBER=='N407AS') & (df.DEPARTURE_DATE > '2015-07-05') & (df.DEPARTURE_DATE < '2015-07-31')]\nquery = df[(df.TAIL_NUMBER=='N594HA')]\nsns.lineplot(data=query, x='DEPARTURE_DATE', y='ARRIVAL_DELAY', markers=['o'],)\nplt.xticks(rotation=45)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import statsmodels.graphics.tsaplots\nquery = df[(df.TAIL_NUMBER=='N594HA')].sort_values('DEPARTURE_TIME').ARRIVAL_DELAY.dropna()\nstatsmodels.graphics.tsaplots.plot_acf(query.values.squeeze(), )\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.groupby('TAIL_NUMBER')['ARRIVAL_DELAY'].count().sort_values(ascending=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.sort_values('DEPARTURE_TIME').groupby('TAIL_NUMBER')['ARRIVAL_DELAY'].apply(pd.Series.autocorr, lag=1).dropna().sort_values(ascending=False)[10:]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"query = df[df.TAIL_NUMBER == 'N594HA']\n# query\nquery = pd.merge(query, airports.add_prefix(\"ORIGIN_\"), left_on='ORIGIN_AIRPORT', right_on='ORIGIN_IATA_CODE', how='left')\nquery = pd.merge(query, airports.add_prefix(\"DESTINATION_\"), left_on='DESTINATION_AIRPORT', right_on='DESTINATION_IATA_CODE', how='left')\nquery = query.groupby(['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', \n               'ORIGIN_LATITUDE', 'ORIGIN_LONGITUDE', \n               'DESTINATION_LATITUDE', 'DESTINATION_LONGITUDE'])['ARRIVAL_DELAY'].mean().reset_index().sort_values('ARRIVAL_DELAY')\nquery","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport matplotlib\nfrom mpl_toolkits.basemap import Basemap\n\nnorm = matplotlib.colors.Normalize(vmin=0.0, vmax=100.0)\ncmap = matplotlib.cm.get_cmap('RdYlGn_r')\n\nfig, ax = plt.subplots(figsize=[6, 3], dpi=300)\n\nm = Basemap(projection='cyl',\n            llcrnrlat=10, urcrnrlat=50,\n            llcrnrlon=-165, urcrnrlon=-110,)\nm.shadedrelief()\nm.drawcountries()\n\n\nfor i, flight in query.iterrows():\n    x, y = m([flight.ORIGIN_LONGITUDE, flight.DESTINATION_LONGITUDE], \n               [flight.ORIGIN_LATITUDE, flight.DESTINATION_LATITUDE])\n\n    ax.annotate('', xy=(x[0], y[0]),  xycoords='data',\n                xytext=(x[1], y[1]), textcoords='data',\n                arrowprops=dict(arrowstyle=\"->\", \n                                color=cmap(norm(flight.ARRIVAL_DELAY)), lw=1, \n                                connectionstyle=\"angle3,angleA=0,angleB=-135\"))\n\nfig.colorbar(matplotlib.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax)\nax.set_title('Flight paths and delays of N594HA')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"query = df[df.TAIL_NUMBER == 'N594HA']\nquery = pd.merge(query, airports.add_prefix(\"ORIGIN_\"), left_on='ORIGIN_AIRPORT', right_on='ORIGIN_IATA_CODE', how='left')\nquery = pd.merge(query, airports.add_prefix(\"DESTINATION_\"), left_on='DESTINATION_AIRPORT', right_on='DESTINATION_IATA_CODE', how='left')\nquery = query.groupby(['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', \n               'ORIGIN_LATITUDE', 'ORIGIN_LONGITUDE', \n               'DESTINATION_LATITUDE', 'DESTINATION_LONGITUDE'])['ARRIVAL_DELAY'].mean().reset_index().sort_values('ARRIVAL_DELAY')\nquery = pd.merge(query, query, left_on=['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT'], right_on=['DESTINATION_AIRPORT', 'ORIGIN_AIRPORT'], how='inner')\nquery = query[['ORIGIN_AIRPORT_x', 'DESTINATION_AIRPORT_x', 'ORIGIN_AIRPORT_y', 'DESTINATION_AIRPORT_y', 'ORIGIN_LATITUDE_x', 'ORIGIN_LONGITUDE_x',\n               'DESTINATION_LATITUDE_x', 'DESTINATION_LONGITUDE_x', 'ARRIVAL_DELAY_x', 'ARRIVAL_DELAY_y']]\n\nquery['DELAY'] = query.ARRIVAL_DELAY_x + query.ARRIVAL_DELAY_y\nquery = query.sort_values('DELAY').iloc[::2]\n\nquery[['ORIGIN_AIRPORT_x', 'DESTINATION_AIRPORT_x', 'ORIGIN_AIRPORT_y', 'DESTINATION_AIRPORT_y', 'DELAY']]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"norm = matplotlib.colors.Normalize(vmin=0.0, vmax=120.0)\ncmap = matplotlib.cm.get_cmap('RdYlGn_r')\n\nfig, ax = plt.subplots(figsize=[6, 3], dpi=300)\n\nm = Basemap(projection='cyl',\n            llcrnrlat=15, urcrnrlat=50,\n            llcrnrlon=-165, urcrnrlon=-110,)\nm.shadedrelief()\nm.drawcountries()\n\n\nfor i, flight in query.iterrows():\n    x, y = m([flight.ORIGIN_LONGITUDE_x, flight.DESTINATION_LONGITUDE_x], \n               [flight.ORIGIN_LATITUDE_x, flight.DESTINATION_LATITUDE_x])\n    ax.plot(x, y, 'ok', ms=1)\n    ax.annotate('', xy=(x[0], y[0]),  xycoords='data',\n                xytext=(x[1], y[1]), textcoords='data',\n                arrowprops=dict(arrowstyle=\"<->\", \n                                color=cmap(norm(flight.DELAY)), lw=1, ))\n\nfig.colorbar(matplotlib.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax)\nax.set_title('Connection route and delays of N594HA')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Pre-processing\n\nThe previous step should give you a better understanding of which pre-processing is required for the data.\nThis may include:\n\n- Normalising and standardising the given data;\n- Removing outliers;\n- Carrying out feature selection, possibly using metrics derived from information theory;\n- Handling missing information in the dataset;\n- Augmenting the dataset with external information;\n- Combining existing features.","metadata":{}},{"cell_type":"markdown","source":"Below is a very basic example of pre-processing steps.","metadata":{}},{"cell_type":"code","source":"# First of all, you should split the data into a training set and a validation set\nX_train_df = df.drop('ARRIVAL_DELAY')\ny_train_df = df(['ID', 'ARRIVAL_DELAY'])\n\nX_train_df, X_val_df, y_train_df, y_val_df = train_test_split(\n    X_train_df, y_train_df, random_state=1, test_size=0.2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert the arrival time and departure time into minutes since the midnight\ndef minutes_since_midnight(dt):\n    return dt.hour * 60 + dt.minute\n\nX_train_df.SCHEDULED_ARRIVAL = X_train_df.SCHEDULED_ARRIVAL.apply(minutes_since_midnight)\nX_train_df.DEPARTURE_TIME = X_train_df.DEPARTURE_TIME.apply(minutes_since_midnight)\n\nX_val_df.SCHEDULED_ARRIVAL = X_val_df.SCHEDULED_ARRIVAL.apply(minutes_since_midnight)\nX_val_df.DEPARTURE_TIME = X_val_df.DEPARTURE_TIME.apply(minutes_since_midnight)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Here we use 4 features as follows\nfeature_names = ['DAY_OF_WEEK', 'AIR_TIME', 'SCHEDULED_ARRIVAL', 'SUM', 'SCHEDULED_TIME', 'DIVERTED', 'MEAN_ARRIVAL']\ntarget_name = ['ARRIVAL_DELAY']\n\nX_train_df['SUM']= X_train_df['DEPARTURE_DELAY']+X_train_df['TAXI_OUT']\nX_val_df['SUM']= X_val_df['DEPARTURE_DELAY']+X_val_df['TAXI_OUT']\n\nX_train_df = X_train_df[feature_names]\ny_train_df = y_train_df[target_name]\n\nX_val_df = X_val_df[feature_names]\ny_val_df = y_val_df[target_name]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Filling missing values by the mean along each column.\n# These statistics should be estimated by using the training set.\nimputer = SimpleImputer(missing_values=np.nan, strategy='mean')\nimputer.fit(X_train_df)\n\nX_train = imputer.transform(X_train_df)\nX_val = imputer.transform(X_val_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Standardize the features by removing the mean and scaling to unit variance\n# Similarly to the preivous step, the statistics used for standardization\n# should be computed across the training set only\nX_scaler = StandardScaler()\nX_scaler.fit(X_train)\n\nX_train = X_scaler.transform(X_train)\nX_val = X_scaler.transform(X_val)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We should also standardize the targets.\ny_scaler = StandardScaler()\ny_scaler.fit(y_train_df)\n\ny_train = y_scaler.transform(y_train_df)\ny_val = y_scaler.transform(y_val_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can define a wrapper function that performs the pipeline of pre-processing data. You should modify this function according to your pipeline.","metadata":{}},{"cell_type":"code","source":"def preprocess_data(df, feature_names, imputer, scaler):\n    \"\"\"Preprocess data.\n\n    Parameters\n    ----------\n    df: pandas DataFrame.\n        The input data.\n    feature_names: list of strings.\n        The names of selected features.\n    imputer: sklearn.impute.SimpleImputer\n        The imputation transformer for completing missing values.\n    scaler: sklearn.preprocessing.StandardScaler.\n        The scaler used to normalize the features.\n\n    Returns\n    -------\n    X: numpy array.\n        The preprocessed data.\n    \"\"\"\n    # Select features\n    X_df = df\n    X_df['SUM']= X_df['DEPARTURE_DELAY']+X_df['TAXI_OUT']\n    \n    X_df = X_df[feature_names]\n    # Pre-process datetime features\n    #X_df.DEPARTURE_TIME = X_df.DEPARTURE_TIME.apply(parse_hhmm)\n    X_df.SCHEDULED_ARRIVAL = X_df.SCHEDULED_ARRIVAL.apply(parse_hhmm)\n    X_df.SCHEDULED_ARRIVAL = X_df.SCHEDULED_ARRIVAL.apply(minutes_since_midnight)\n    \n    #X_df.DEPARTURE_TIME = X_df.DEPARTURE_TIME.apply(minutes_since_midnight)\n\n    # Impute missing values\n    X = imputer.transform(X_df)\n    \n    # Normalize features\n    X = scaler.transform(X)\n\n    return X","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Selection\n\nPerhaps one of the most important segments of this challenge involves the selection of a model that can successfully handle the given data and yield sensible predictions.\nInstead of focusing exclusively on your final chosen model, it is also important to share your thought process in this notebook by additionally describing alternative candidate models.\nThere is a wealth of models to choose from, such as <i>decision trees</i>, <i>random forests</i>, <i>(Bayesian) neural networks</i>, <i>Gaussian processes</i>, <i>LASSO regression</i>, and so on.\nThere are several factors which may influence your decision:\n\n- What is the model's complexity?\n- Is the model interpretable?\n- Is the model capable of handling different data-types?\n- Does the model return uncertainty estimates along with predictions?\n","metadata":{}},{"cell_type":"markdown","source":"In this baseline solution, we use the Ridge regression model, which is a linear least-square model with L2 regularization on its parameters. There is a hyper-parameter that should be tuned which is the regularization strength $\\alpha$.\nIntuitively, this hyper-parameter controls the amount of shrinkage of the parameters of the model: the larger the value of $\\alpha$ the greater the amount of shrinkage.\n\nSection 3.4.1 of the book *The Elements of Statistical Learning: Data Mining, Inference, and Prediction* from Trevor Hastie et al. ([link](https://web.stanford.edu/~hastie/Papers/ESLII.pdf)) is a good reference for the Ridge regression model and other variants of the linear regression model.","metadata":{}},{"cell_type":"markdown","source":"## Parameter Optimisation\n\nIrrespective of your choice, it is highly likely that your model will have one or more parameters that require tuning.\nThere are several techniques for carrying out such a procedure, including cross-validation, Bayesian optimisation, and several others.\nAs before, an analysis into which parameter tuning technique best suits your model is expected before proceeding with the optimisation of your model.","metadata":{}},{"cell_type":"markdown","source":"The below cells demonstrate tuning the hyper-parameters $\\alpha$ of the Ridge regression model by using cross-validation.","metadata":{}},{"cell_type":"markdown","source":"We can define a wrapper function used to make predictions given a pre-processed dataset. You can freely modify this function according to your approach.","metadata":{}},{"cell_type":"code","source":"def make_prediction(X, model, scaler):\n    \"\"\"Makes predictions given a preprocessed dataset.\n\n    Parameters\n    ----------\n    X: numpy array.\n        The input data, which already is pre-processed.\n    model: an hypopt or sklearn model.\n        The trained model used for making predictions.\n    scaler: sklearn.preprocessing.StandardScaler.\n        The scaler used to normalize the targets.\n\n    Returns\n    -------\n    y_pred: numpy array.\n        The unnormalized predictions.\n    \"\"\"\n    y_pred = scaler.inverse_transform(model.predict(X))\n    return y_pred","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After training the model, we can evaluate the performance of the model on the training and validation sets in terms of root mean squared error (RMSE).","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\nregr = RandomForestRegressor()\nregr.fit(X_train, y_train)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred = make_prediction(X_train, regr, y_scaler)\ntrain_rmse = np.sqrt(mean_squared_error(y_train_pred, y_train_df.values))\nprint(\"Training RMSE: {:.5f}\".format(float(train_rmse)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_val_pred = make_prediction(X_val, regr, y_scaler)\nval_rmse = np.sqrt(mean_squared_error(y_val_pred, y_val_df.values))\nprint(\"Validation RMSE: {:.5f}\".format(float(val_rmse)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.ensemble import ExtraTreesRegressor\nregExtra = ExtraTreesRegressor().fit(X_train, y_train)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred = make_prediction(X_train, regExtra, y_scaler)\ntrain_rmse = np.sqrt(mean_squared_error(y_train_pred, y_train_df.values))\nprint(\"Training RMSE: {:.5f}\".format(float(train_rmse)))\ny_val_pred = make_prediction(X_val, regExtra, y_scaler)\nval_rmse = np.sqrt(mean_squared_error(y_val_pred, y_val_df.values))\nprint(\"Validation RMSE: {:.5f}\".format(float(val_rmse)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.svm import SVR\n\nmodel = SVR(kernel='rbf', gamma=\"auto\")\nmodel.fit(x_train, y_train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred = make_prediction(X_train, model, y_scaler)\ntrain_rmse = np.sqrt(mean_squared_error(y_train_pred, y_train_df.values))\nprint(\"Training RMSE: {:.5f}\".format(float(train_rmse)))\n\ny_val_pred = make_prediction(X_val, model, y_scaler)\nval_rmse = np.sqrt(mean_squared_error(y_val_pred, y_val_df.values))\nprint(\"Validation RMSE: {:.5f}\".format(float(val_rmse)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import xgboost as xgb\nxgbr = xgb.XGBRegressor(verbosity=0) \nxgbr.fit(X_train, y_train) \n\ny_train_pred = make_prediction(X_train, xgbr, y_scaler)\ntrain_rmse = np.sqrt(mean_squared_error(y_train_pred, y_train_df.values))\nprint(\"Training RMSE: {:.5f}\".format(float(train_rmse)))\ny_val_pred = make_prediction(X_val, xgbr, y_scaler)\nval_rmse = np.sqrt(mean_squared_error(y_val_pred, y_val_df.values))\nprint(\"Validation RMSE: {:.5f}\".format(float(val_rmse)))\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Evaluation\n\nSome form of pre-evaluation will inevitably be required in the preceding sections in order to both select an appropriate model and configure its parameters appropriately.\nIn this final section, you may evaluate other aspects of the model such as:\n\n- Assessing the running time of your model;\n- Determining whether some aspects can be parallelised;\n- Training the model with smaller subsets of the data.\n- etc.\n\nThe goal of this challenge is to construct a model for predicting arrival delays\n\n\nYour submission is a CSV file containing your <b>final model's predictions on the given test data</b>. \n    This file should contain a header and have the following format:\n    \n    ```\n    ID,ARRIVAL_DELAY\n    1561528,30.497905409756292\n    845015,32.75533601062071\n    717459,34.758212570279404\n    451496,32.73011833829676\n    etc.\n    ```\n\nA leaderboard for this challenge will be ranked using the root mean squared error between the predicted values and the observed arrival delays. However, you can use other metrics for regression tasks in your presentation notebook to evaluate many aspects of your model.","metadata":{}},{"cell_type":"markdown","source":"Below is an example of creating a submission file.","metadata":{}},{"cell_type":"code","source":"# Load the test data\nX_test_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/test_features.csv\", dtype=X_dtype)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Preprocessing data\nX_test = preprocess_data(X_test_df, feature_names, imputer, X_scaler)\n\n# Make predictions\ny_test_pred = make_prediction(X_test, regr, y_scaler)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a dataframe containing the predictions\nsubmission_df = pd.DataFrame(data={'ID': X_test_df.ID.values,\n                                   'ARRIVAL_DELAY': y_test_pred.squeeze()})\n\n# Save the predictions into a csv file\n# Notice that this file should be saved under the directory `/kaggle/working` \n# so that you can download it later\nsubmission_df.to_csv(\"/kaggle/working/submission.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the submission file\n! head -5 \"/kaggle/working/submission.csv\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}