{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Challenge 1 \n\n# Airplane delay: Analysis and Prediction\n\n## Context\n\nEvery day, in US, there are thousands of flights departures and arrivals (prior to COVID): unfortunately, as you may have noticed yourself, flight delays are not a rare event!!\n\nIn this notebook, we play the role of a data scientist working in the travel industry, specifically on air transportation of passengers. We want to explore the data collected by the Department of Transportation (DoT) to understand passengers' behavior, as well as the properties of all flights, across several airline companies.\n\nNow, given historical data about flights in the country, including the delay information that was computed a-posteriori (so the ground truth is available), we want to build a model that can be used to predict how many minutes of delay a flight might experience in the future. This model should provide useful information for the airport to manage better its resources, to minimize the delays and their impact on the journey of its passengers. Alternatively, astute passengers could even use the model to choose the best time for flying, such as to avoid delays.","metadata":{}},{"cell_type":"markdown","source":"Let's install some python packages used in this notebook","metadata":{}},{"cell_type":"code","source":"import sys\nimport json\nimport warnings\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.dates as mdates\nimport seaborn as sns\nimport tensorflow as tf\nimport numpy as np\nimport tensorflow_probability as tfp\n\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.model_selection import train_test_split, GridSearchCV\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.linear_model import SGDRegressor\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.pipeline import Pipeline\nfrom matplotlib import rc\n\nif not sys.warnoptions:\n    warnings.simplefilter(\"ignore\")\n\nrc('figure', **{'dpi': 200})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, you can load the training data.","metadata":{}},{"cell_type":"code","source":"X_dtype = {\n    'ID'                   : int,\n    'YEAR'                 : int,  \n    'MONTH'                : int,  \n    'DAY'                  : int,  \n    'DAY_OF_WEEK'          : int,  \n    'AIRLINE'              : str, \n    'FLIGHT_NUMBER'        : str,  \n    'TAIL_NUMBER'          : str, \n    'ORIGIN_AIRPORT'       : str, \n    'DESTINATION_AIRPORT'  : str, \n    'SCHEDULED_DEPARTURE'  : str,  \n    'DEPARTURE_TIME'       : str, \n    'DEPARTURE_DELAY'      : float,\n    'TAXI_OUT'             : str, \n    'WHEELS_OFF'           : str,\n    'SCHEDULED_TIME'       : float,\n    'AIR_TIME'             : float,\n    'DISTANCE'             : int,\n    'SCHEDULED_ARRIVAL'    : str,\n    'DIVERTED'             : int,  \n    'CANCELLED'            : int,  \n    'CANCELLATION_REASON'  : str\n}\n\ny_dtype = {\n    'ID'                   : int,\n    \"ARRIVAL_DELAY\"        : float\n}\n\nX_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_features.csv\", dtype=X_dtype)\ny_train_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/train_targets.csv\", dtype=y_dtype)\nX_test_df = pd.read_csv(\"/kaggle/input/eurecom-aml-2021-challenge-1/data/test_features.csv\", dtype=X_dtype)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def minutes_since_midnight(dt):\n    return dt.hour * 60 + dt.minute\n\ndef parse_hhmm(x):\n    if x == \"2400\":\n        x = \"0000\"\n    try: return pd.datetime.strptime(x, '%H%M')#.time()\n    except: \n        print(\"found Invalid datetime\", x)\n        return pd.NaT\n\ndef parse_times(dataframe):\n\n    dataframe.TAXI_OUT = dataframe.TAXI_OUT.astype(int)\n    dataframe.SCHEDULED_DEPARTURE = dataframe.SCHEDULED_DEPARTURE.apply(parse_hhmm)\n    dataframe.SCHEDULED_ARRIVAL = dataframe.SCHEDULED_ARRIVAL.apply(parse_hhmm)\n    dataframe.DEPARTURE_TIME = dataframe.DEPARTURE_TIME.apply(parse_hhmm)\n\n    dataframe.SCHEDULED_DEPARTURE = dataframe.SCHEDULED_DEPARTURE.apply(minutes_since_midnight)\n    dataframe.SCHEDULED_ARRIVAL = dataframe.SCHEDULED_ARRIVAL.apply(minutes_since_midnight)\n    dataframe.DEPARTURE_TIME = dataframe.DEPARTURE_TIME.apply(minutes_since_midnight)\n\n    pd.set_option('display.max_rows', 22)\n    dataframe.dtypes\n\nparse_times(X_train_df)\nparse_times(X_test_df)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You may also use additional information about airports and airlines, which can be found in the files `/kaggle/input/eurecom-aml-2021-challenge-1/data/airports.csv` and `/kaggle/input/eurecom-aml-2021-challenge-1/data/airlines.csv`, respectively.","metadata":{}},{"cell_type":"code","source":"airports = pd.read_csv('/kaggle/input/eurecom-aml-2021-challenge-1/data/airports.csv',\n                       usecols=['IATA_CODE', 'AIRPORT', 'LATITUDE', 'LONGITUDE']).rename({'AIRPORT': 'NAME'}, axis='columns')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The following cells have been converted to md, to avoid re-executing them \n## Data analysis\nIn this challenge, you are free (and encouraged) to explore in depth the data you have.\nNext, you have examples of simple queries on our data, to perform exploration and compute statistics\n\n**NOTE:** finding the right question to ask is difficult! Don't be afraid to complement the questions below, with your own questions that, in your opinion, are valuable ways to inspect data. This can give you extra points!\n\n**NOTE 2:** the presentation quality is critical in any business-oriented data analysis. Take time to create informative plots, rather than endless tables!\n\n- Basic queries:\n  - How many unique origin airports?\n  - How many unique destination airports?\n  - How many carriers?\n  - How many flights that have a scheduled departure time later than 18h00?\n\n\n- Statistics on flight volume: this kind of statistics are helpful to reason about delays. Indeed, it is plausible to assume that \"*the more flights in an airport, the higher the probability of delay*\".\n  - How many flights in each month of the year?\n  - Is there any relationship between the number of flights and the days of week?\n  - How many flights in different days of months and in different hours of days?\n  - Which are the top 20 busiest airports (this depends on inbound and outbound traffic)?\n  - Which are the top 20 busiest carriers?\n\n\n- Statistics on the fraction of delayed flights\n  - What is the percentage of delayed flights (over total flights) for different hours of the day?\n  - Which hours of the day are characterized by the longest flight delay?\n  - What are the fluctuation of the percentage of delayed flights over different time granularities?\n  - What is the percentage of delayed flights which depart from one of the top 20 busiest airports?\n  - What is the percentage of delayed flights which belong to one of the top 20 busiest carriers?","metadata":{}},{"cell_type":"markdown","source":"Below is a very simple example of data exploration.","metadata":{}},{"cell_type":"markdown","source":"fig, ax = plt.subplots()\nsns.histplot(df, x='DEPARTURE_TIME', hue='DELAYED', bins=48, multiple='stack', palette=['C0', 'C3'], ax=ax)\nax.xaxis.set_major_formatter(mdates.DateFormatter(\"%H:%M\"))\nax.set_title('Overview of the flights')\nplt.xticks(rotation=45)\nplt.show()","metadata":{}},{"cell_type":"markdown","source":"fig, ax = plt.subplots()\nsns.histplot(df, x='DATE', hue='DELAYED', bins=48, multiple='stack', palette=['C0', 'C3'], ax=ax)\nax.xaxis.set_major_formatter(mdates.DateFormatter(\"%d/%m\"))\nax.set_title('Seasonal effect on flight delay?')\nplt.xticks(rotation=45)\nplt.show()","metadata":{}},{"cell_type":"markdown","source":"_df1 = df.groupby('ORIGIN_AIRPORT').agg({'ARRIVAL_DELAY':'count'}).rename(columns={'ARRIVAL_DELAY': 'COUNT'}).rename_axis('AIRPORT')\n_df2 = df.groupby('DESTINATION_AIRPORT').agg({'ARRIVAL_DELAY':'count'}).rename(columns={'ARRIVAL_DELAY': 'COUNT'}).rename_axis('AIRPORT')\ntop_airports = _df1.join(_df2, rsuffix='_ORIGIN', lsuffix='_DEST').sum(axis=1).sort_values(ascending=False).index[:15]\ntop_airports","metadata":{}},{"cell_type":"markdown","source":"df['DEPARTURE_DATE'] = pd.to_datetime(df.DATE.dropna().dt.date.astype(str) + ' ' + df.DEPARTURE_TIME.dropna().dt.time.astype(str))","metadata":{}},{"cell_type":"markdown","source":"query = df[(df.TAIL_NUMBER=='N407AS') & (df.DEPARTURE_DATE > '2015-07-05') & (df.DEPARTURE_DATE < '2015-07-31')]\nquery = df[(df.TAIL_NUMBER=='N594HA')]\nsns.lineplot(data=query, x='DEPARTURE_DATE', y='ARRIVAL_DELAY', markers=['o'],)\nplt.xticks(rotation=45)","metadata":{}},{"cell_type":"markdown","source":"import statsmodels.graphics.tsaplots\nquery = df[(df.TAIL_NUMBER=='N594HA')].sort_values('DEPARTURE_TIME').ARRIVAL_DELAY.dropna()\nstatsmodels.graphics.tsaplots.plot_acf(query.values.squeeze(), )\nplt.show()","metadata":{}},{"cell_type":"markdown","source":"df.groupby('TAIL_NUMBER')['ARRIVAL_DELAY'].count().sort_values(ascending=False)","metadata":{}},{"cell_type":"markdown","source":"df.sort_values('DEPARTURE_TIME').groupby('TAIL_NUMBER')['ARRIVAL_DELAY'].apply(pd.Series.autocorr, lag=1).dropna().sort_values(ascending=False)[10:]","metadata":{}},{"cell_type":"markdown","source":"query = df[df.TAIL_NUMBER == 'N594HA']\n# query\nquery = pd.merge(query, airports.add_prefix(\"ORIGIN_\"), left_on='ORIGIN_AIRPORT', right_on='ORIGIN_IATA_CODE', how='left')\nquery = pd.merge(query, airports.add_prefix(\"DESTINATION_\"), left_on='DESTINATION_AIRPORT', right_on='DESTINATION_IATA_CODE', how='left')\nquery = query.groupby(['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', \n               'ORIGIN_LATITUDE', 'ORIGIN_LONGITUDE', \n               'DESTINATION_LATITUDE', 'DESTINATION_LONGITUDE'])['ARRIVAL_DELAY'].mean().reset_index().sort_values('ARRIVAL_DELAY')\nquery","metadata":{}},{"cell_type":"markdown","source":"import matplotlib.pyplot as plt\nimport matplotlib\nfrom mpl_toolkits.basemap import Basemap\n\nnorm = matplotlib.colors.Normalize(vmin=0.0, vmax=100.0)\ncmap = matplotlib.cm.get_cmap('RdYlGn_r')\n\nfig, ax = plt.subplots(figsize=[6, 3], dpi=300)\n\nm = Basemap(projection='cyl',\n            llcrnrlat=10, urcrnrlat=50,\n            llcrnrlon=-165, urcrnrlon=-110,)\nm.shadedrelief()\nm.drawcountries()\n\n\nfor i, flight in query.iterrows():\n    x, y = m([flight.ORIGIN_LONGITUDE, flight.DESTINATION_LONGITUDE], \n               [flight.ORIGIN_LATITUDE, flight.DESTINATION_LATITUDE])\n\n    ax.annotate('', xy=(x[0], y[0]),  xycoords='data',\n                xytext=(x[1], y[1]), textcoords='data',\n                arrowprops=dict(arrowstyle=\"->\", \n                                color=cmap(norm(flight.ARRIVAL_DELAY)), lw=1, \n                                connectionstyle=\"angle3,angleA=0,angleB=-135\"))\n\nfig.colorbar(matplotlib.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax)\nax.set_title('Flight paths and delays of N594HA')\nplt.show()","metadata":{}},{"cell_type":"markdown","source":"query = df[df.TAIL_NUMBER == 'N594HA']\nquery = pd.merge(query, airports.add_prefix(\"ORIGIN_\"), left_on='ORIGIN_AIRPORT', right_on='ORIGIN_IATA_CODE', how='left')\nquery = pd.merge(query, airports.add_prefix(\"DESTINATION_\"), left_on='DESTINATION_AIRPORT', right_on='DESTINATION_IATA_CODE', how='left')\nquery = query.groupby(['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', \n               'ORIGIN_LATITUDE', 'ORIGIN_LONGITUDE', \n               'DESTINATION_LATITUDE', 'DESTINATION_LONGITUDE'])['ARRIVAL_DELAY'].mean().reset_index().sort_values('ARRIVAL_DELAY')\nquery = pd.merge(query, query, left_on=['ORIGIN_AIRPORT', 'DESTINATION_AIRPORT'], right_on=['DESTINATION_AIRPORT', 'ORIGIN_AIRPORT'], how='inner')\nquery = query[['ORIGIN_AIRPORT_x', 'DESTINATION_AIRPORT_x', 'ORIGIN_AIRPORT_y', 'DESTINATION_AIRPORT_y', 'ORIGIN_LATITUDE_x', 'ORIGIN_LONGITUDE_x',\n               'DESTINATION_LATITUDE_x', 'DESTINATION_LONGITUDE_x', 'ARRIVAL_DELAY_x', 'ARRIVAL_DELAY_y']]\n\nquery['DELAY'] = query.ARRIVAL_DELAY_x + query.ARRIVAL_DELAY_y\nquery = query.sort_values('DELAY').iloc[::2]\n\nquery[['ORIGIN_AIRPORT_x', 'DESTINATION_AIRPORT_x', 'ORIGIN_AIRPORT_y', 'DESTINATION_AIRPORT_y', 'DELAY']]","metadata":{}},{"cell_type":"markdown","source":"norm = matplotlib.colors.Normalize(vmin=0.0, vmax=120.0)\ncmap = matplotlib.cm.get_cmap('RdYlGn_r')\n\nfig, ax = plt.subplots(figsize=[6, 3], dpi=300)\n\nm = Basemap(projection='cyl',\n            llcrnrlat=15, urcrnrlat=50,\n            llcrnrlon=-165, urcrnrlon=-110,)\nm.shadedrelief()\nm.drawcountries()\n\n\nfor i, flight in query.iterrows():\n    x, y = m([flight.ORIGIN_LONGITUDE_x, flight.DESTINATION_LONGITUDE_x], \n               [flight.ORIGIN_LATITUDE_x, flight.DESTINATION_LATITUDE_x])\n    ax.plot(x, y, 'ok', ms=1)\n    ax.annotate('', xy=(x[0], y[0]),  xycoords='data',\n                xytext=(x[1], y[1]), textcoords='data',\n                arrowprops=dict(arrowstyle=\"<->\", \n                                color=cmap(norm(flight.DELAY)), lw=1, ))\n\nfig.colorbar(matplotlib.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax)\nax.set_title('Connection route and delays of N594HA')\nplt.show()","metadata":{}},{"cell_type":"markdown","source":"## Data Pre-processing\n\nThe previous step should give you a better understanding of which pre-processing is required for the data.\nThis may include:\n\n- Normalising and standardising the given data;\n- Removing outliers;\n- Carrying out feature selection, possibly using metrics derived from information theory;\n- Handling missing information in the dataset;\n- Augmenting the dataset with external information;\n- Combining existing features.","metadata":{}},{"cell_type":"code","source":"def merge(X, y):\n    df = pd.merge(X, airports.rename({'LATITUDE':'LATITUDE_ORIGIN', 'LONGITUDE':'LONGITUDE_ORIGIN'},axis='columns', errors = 'raise'),how='left',left_on='ORIGIN_AIRPORT', right_on = 'IATA_CODE')\n    df = pd.merge(df, airports.rename({'LATITUDE':'LATITUDE_DESTINATION', 'LONGITUDE':'LONGITUDE_DESTINATION'}, axis='columns',errors = 'raise'),how='left',left_on='DESTINATION_AIRPORT', right_on = 'IATA_CODE')\n    if y is not None:\n        df = pd.merge(df, y, on='ID')\n    else:\n        df['ARRIVAL_DELAY'] = 0\n    return df\n\ndf = merge(X_train_df, y_train_df)\ndf_test = merge(X_test_df, None)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols_to_drop = ['ID', 'YEAR', 'TAIL_NUMBER', 'FLIGHT_NUMBER', 'WHEELS_OFF', 'DIVERTED',\n                'CANCELLED', 'CANCELLATION_REASON',\n                'IATA_CODE_x', 'NAME_x', 'IATA_CODE_y', 'NAME_y',\n                'ORIGIN_AIRPORT', 'DESTINATION_AIRPORT']\n\ndef drops(dataframe, cols_to_drop):\n    return dataframe.drop(cols_to_drop, axis = 1)\n\ndf = drops(df, cols_to_drop)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols_to_drop.remove('ID')\ndf_test = drops(df_test, cols_to_drop)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_rows', 50)\ndf.head(6)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical = ['AIRLINE', 'DAY_OF_WEEK', 'MONTH']\ndef make_categorical(features, dataframe):\n    df_cat = dataframe[features].astype('category')\n    dataframe = pd.concat([dataframe,pd.get_dummies(df_cat)],axis=1)\n    return dataframe.drop(categorical, axis = 1)\n\ndf = make_categorical(categorical, df)\ndf_test = make_categorical(categorical, df_test)\npd.set_option('display.max_rows', 100)\ndf_test.dtypes","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_name = 'ARRIVAL_DELAY'\ny = df[target_name]\nX = df.drop(target_name,axis = 1) \nX_test = df_test.drop(target_name,axis = 1) \nXtrain, Xval, ytrain, yval = train_test_split(\n    X, y, random_state=10, test_size=0.4)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"param_space0 = {'est__learning_rate': ['optimal', 'invscaling']}\nparam_space1 = {'est__loss': ['squared_loss', 'huber', 'epsilon_insensitive']}\nparam_space2 = {'est__penalty': ['l1', 'l2', 'elasticnet'], 'est__alpha': [0.001, 0.0001, 0.00001]}","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# we limit the max_iter in the param_search to keep time down\nmodel = SGDRegressor(tol=0.001, max_iter=250)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipe0 = Pipeline([('scl', StandardScaler()), \n                      ('imp', SimpleImputer()),\n                      ('est', model)])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gdcv0 = GridSearchCV(estimator=pipe0, param_grid=param_space0, scoring='neg_root_mean_squared_error', verbose=1, cv=3)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# gdcv0.fit(Xtrain, ytrain)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Best score: %0.3f\" % gdcv0.best_score_)\nprint(\"Best parameters set:\")\nbest_parameters = gdcv0.best_estimator_.get_params()\nfor param_name in sorted(param_space0.keys()):\n    print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = SGDRegressor(tol=0.001, learning_rate='optimal')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipe1 = Pipeline([('scl', StandardScaler()), \n                      ('imp', SimpleImputer()),\n                      ('est', model)])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gdcv1 = GridSearchCV(estimator=pipe1, param_grid=param_space1, scoring='neg_root_mean_squared_error', verbose=1, cv=3)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# gdcv1.fit(Xtrain, ytrain)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Best score: %0.3f\" % gdcv1.best_score_)\nprint(\"Best parameters set:\")\nbest_parameters = gdcv1.best_estimator_.get_params()\nfor param_name in sorted(param_space1.keys()):\n    print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = SGDRegressor(tol=0.001, learning_rate='optimal', loss='squared_loss')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipe2= Pipeline([('scl', StandardScaler()), \n                      ('imp', SimpleImputer()),\n                      ('est', model)])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gdcv2 = GridSearchCV(estimator=pipe2, param_grid=param_space2, scoring='neg_root_mean_squared_error', verbose=1, n_jobs=-1)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# gdcv2.fit(Xtrain, ytrain)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Best score: %0.3f\" % gdcv2.best_score_)\nprint(\"Best parameters set:\")\nbest_parameters = gdcv2.best_estimator_.get_params()\nfor param_name in sorted(param_space2.keys()):\n    print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_pipe = make_pipeline(StandardScaler(),\n                     SimpleImputer(),\n                     SGDRegressor(tol=0.001, alpha=0.001, learning_rate='optimal', loss='squared_loss', penalty='l1'),\n                     )","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_pipe.fit(Xtrain, ytrain)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred = (final_pipe.predict(Xtrain))\ny_val_pred =(final_pipe.predict(Xval))\ntrain_rmse = np.sqrt(mean_squared_error(y_train_pred, ytrain))\nprint(\"Training RMSE: {:.5f}\".format(float(train_rmse)))\nval_rmse = np.sqrt(mean_squared_error(y_val_pred, yval))\nprint(\"Validation RMSE: {:.5f}\".format(float(val_rmse)))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# It seems at the end, a lot of the features we added were close to useless. Let's opt for a much simpler model,\n# as the performance hit is barely - noticeable, but the speed gain, significant. \nfor f, c in zip(X_test.columns, final_pipe[-1].coef_):\n    print(f, c)\n    if abs(c) < 0.5 and f in X_test.columns:\n        X = X.drop(f, axis=1)\n        X_test = X_test.drop(f, axis=1)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fit on the whole dataset and make prediction on the test set\nfinal_pipe.fit(X, y)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test = df_test.drop(['ID', 'ARRIVAL_DELAY'], axis=1)\nX_test.shape","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_test_pred = final_pipe.predict(X_test)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df = pd.DataFrame(data={'ID': df_test.ID.values,\n                                   'ARRIVAL_DELAY': y_test_pred.squeeze()})","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.to_csv(f\"submission{time.time()}.csv\", index=False)","metadata":{},"execution_count":null,"outputs":[]}]}