{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**Table of content:**\n\n1. Introduction\n\n2. Importing libraries and Reading the Dataset\n\n3. Data Analysis & Data cleaning\n\n4. Models:\n    - Linear Regression\n    - Creativity\n    - Lasso Regression\n    - XGBoost\n    \n\n5. Hypertuning of hyperparameters\n\n6. Best model\n\n7. Conclusion","metadata":{}},{"cell_type":"markdown","source":"**1. Introduction**","metadata":{"_uuid":"1c34ff4d-91a7-4e69-91b6-ae38e5ff2acb","_cell_guid":"4fed299b-1f3a-4c08-ade8-2cc7b25dbc10","trusted":true}},{"cell_type":"markdown","source":"The dataset contain 111 features of meteorological data for point on earth at a defined time. The aim is to predict the temperature at 2 metres above the ground. There are more than one million measures from all around the world. In this challenge, we will go step by step to implement the best strategy.","metadata":{"_uuid":"07c9221d-b464-4a92-969c-13203e5b263e","_cell_guid":"f32e0cb5-867a-4618-b42d-a51eb76fa863","trusted":true}},{"cell_type":"markdown","source":"**2. Importing libraries and Reading the Dataset**","metadata":{"_uuid":"bc7b90ea-2a0a-4242-b984-4aa9b8748c7c","_cell_guid":"bed8f597-2850-4e3e-a7b1-9a98ca7431af","trusted":true}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np # linear algebra\nimport matplotlib.pyplot as plt \nimport seaborn as sns\nfrom tqdm.notebook import tqdm \nfrom matplotlib import rc\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.preprocessing import MinMaxScaler, StandardScaler, RobustScaler\nfrom sklearn.feature_selection import VarianceThreshold, chi2, SelectKBest\nimport xgboost as xg\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import linear_model\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import RandomizedSearchCV, GridSearchCV\nfrom sklearn.linear_model import Lasso","metadata":{"_uuid":"b4b44db0-f20a-4f71-9542-1da137a2037e","_cell_guid":"5618ac75-1cbc-4c35-99b8-ed90a58829e5","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:31:06.292173Z","iopub.execute_input":"2022-05-11T21:31:06.292509Z","iopub.status.idle":"2022-05-11T21:31:06.300948Z","shell.execute_reply.started":"2022-05-11T21:31:06.292477Z","shell.execute_reply":"2022-05-11T21:31:06.299662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv('../input/eurecom-aml-2022-challenge-1/public/train.csv', low_memory=True)","metadata":{"_uuid":"7f94d56e-926f-4c8a-b3f2-686771f0c79d","_cell_guid":"f59db3b1-2d80-4c81-83cf-0fecaa1ea712","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:31:07.822758Z","iopub.execute_input":"2022-05-11T21:31:07.823055Z","iopub.status.idle":"2022-05-11T21:32:19.224231Z","shell.execute_reply.started":"2022-05-11T21:31:07.823018Z","shell.execute_reply":"2022-05-11T21:32:19.222970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = pd.read_csv('../input/eurecom-aml-2022-challenge-1/public/test_feat.csv', low_memory=True)","metadata":{"_uuid":"02d7b349-5873-4767-957e-4d89d59a0742","_cell_guid":"5129876a-e9ec-455d-aee4-db89d84ea859","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:19.226350Z","iopub.execute_input":"2022-05-11T21:32:19.226679Z","iopub.status.idle":"2022-05-11T21:32:38.049816Z","shell.execute_reply.started":"2022-05-11T21:32:19.226623Z","shell.execute_reply":"2022-05-11T21:32:38.048843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**3. Data Analysis & Data cleaning**","metadata":{"_uuid":"673b4066-b9c1-4027-91fd-d01c94f8b70a","_cell_guid":"7ed64d5b-014c-472b-94cd-340971f7fb70","trusted":true}},{"cell_type":"markdown","source":"First, let's observe the data we have:","metadata":{"_uuid":"f4a2e310-2286-487f-95d5-4edaf57b048c","_cell_guid":"d1905fd7-2a20-4d37-8f93-f422a4ae841e","trusted":true}},{"cell_type":"code","source":"df.head()","metadata":{"_uuid":"d096cc46-ab9e-4d1b-bb51-0c5ace2351da","_cell_guid":"01fcfd92-f9fb-4dd3-a4b3-50a4b92d66dd","collapsed":false,"papermill":{"duration":0.101931,"end_time":"2022-05-02T09:50:30.871896","exception":false,"start_time":"2022-05-02T09:50:30.769965","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that we have 111 features (excluding the index) so let's observe the data to understand them, clean the dataset and choose the right features after!","metadata":{"_uuid":"9955ad84-59c9-4b15-8c65-f6938a812b92","_cell_guid":"1847913d-7ce3-4cef-89bf-13bd0f1c2456","trusted":true}},{"cell_type":"markdown","source":"First, let's observe a map of the world to see the difference between the train set and the test set.","metadata":{"_uuid":"f20171c9-0b84-4875-abc5-bbfd09174c0b","_cell_guid":"e3ef33ed-93eb-4c5a-a2db-b57875da8270","trusted":true}},{"cell_type":"code","source":"tr_coordinates = df[['fact_latitude','fact_longitude']].drop_duplicates()\nte_coordinates = df_test[['fact_latitude','fact_longitude']].drop_duplicates()\nmerged = te_coordinates.merge(tr_coordinates, how='left', indicator=True)\ntropic_coordinates = merged[merged['_merge']=='left_only']\ncommon_coordinates = merged[merged['_merge']=='both']","metadata":{"_uuid":"f8e381b1-b7b5-4e3d-8186-64d0f7bd2e40","_cell_guid":"74dd316e-45dc-4702-b732-6dafc631a049","collapsed":false,"papermill":{"duration":0.268813,"end_time":"2022-05-02T09:50:31.19666","exception":false,"start_time":"2022-05-02T09:50:30.927847","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, (ax0, ax1) = plt.subplots(1, 2, figsize=[10, 3], dpi=200, sharex=True, sharey=True)\nax0.scatter(tr_coordinates['fact_longitude'], tr_coordinates['fact_latitude'], \n            s=1, c='tab:green')\nax1.scatter(common_coordinates['fact_longitude'], common_coordinates['fact_latitude'], \n            s=1, c='tab:orange')\nax1.scatter(tropic_coordinates['fact_longitude'], tropic_coordinates['fact_latitude'], \n            s=1, c='tab:red')\nfig.suptitle('Distribution of train and test points in the dataset', y=1.01, fontsize=14)\nplt.show()\n\nprint(tr_coordinates.shape, common_coordinates.shape, tropic_coordinates.shape)","metadata":{"_uuid":"85bfdf51-5b92-4ec1-8d58-f8ef186850e9","_cell_guid":"5ae8a99e-d03b-46e4-b909-138fae8e8119","collapsed":false,"papermill":{"duration":0.634653,"end_time":"2022-05-02T09:50:31.88648","exception":false,"start_time":"2022-05-02T09:50:31.251827","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that we have a lot of temperatures to predict near the equator but few data is existing in the train set. It may be a problem and we will have to keep this information in mind.\nAlso, we can see some points in the train set, in isolated areas that are not present in the test set.","metadata":{"_uuid":"921175f6-5c5c-4171-918c-fa36f0c3981b","_cell_guid":"77fc6364-1380-43f9-909e-a9686984eb20","trusted":true}},{"cell_type":"markdown","source":"So, first, we start by deleting data from our training set that won't be used for testing (data west from the US and everything below South America).","metadata":{"_uuid":"584d7ad2-8196-4106-b779-04342a56851d","_cell_guid":"e8cfd3cc-5537-40b8-835b-f55fa0ff049b","papermill":{"duration":0.060174,"end_time":"2022-05-02T09:50:32.007004","exception":false,"start_time":"2022-05-02T09:50:31.94683","status":"completed"},"tags":[],"trusted":true}},{"cell_type":"code","source":"print(df.shape)\ndf = df.drop(df[(df['fact_latitude']<0) & (df['fact_longitude']<-100)]['index'])\ndf = df.drop(df[(df['fact_latitude']<30) & (df['fact_longitude']<-130) & (df['fact_latitude']>0)]['index'])\ndf = df.drop(df[(df['fact_latitude']<-60)]['index'])\n\nprint(df.shape)","metadata":{"_uuid":"69f2aaa2-5fc4-4da2-b027-19fc05bfdbff","_cell_guid":"7ecfe4a4-586f-4996-9110-c15914a64096","collapsed":false,"papermill":{"duration":3.463736,"end_time":"2022-05-02T09:50:35.532875","exception":false,"start_time":"2022-05-02T09:50:32.069139","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:38.051366Z","iopub.execute_input":"2022-05-11T21:32:38.051693Z","iopub.status.idle":"2022-05-11T21:32:41.318096Z","shell.execute_reply.started":"2022-05-11T21:32:38.051645Z","shell.execute_reply":"2022-05-11T21:32:41.316989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that we removed arround 5,000 useless data.","metadata":{"_uuid":"400faa7d-76aa-4857-8abd-cceb7a7308ed","_cell_guid":"563ddda4-7785-4b4b-9727-fcf70fc72a6e","trusted":true}},{"cell_type":"markdown","source":"We rename the columns longitude and latitude because their name has been switched.","metadata":{"_uuid":"7c37b08e-ad52-4ef0-bb49-f16e4e6aeeb9","_cell_guid":"2300c3ee-daaf-4e8c-ac3c-3da79310d187","papermill":{"duration":0.060217,"end_time":"2022-05-02T09:50:35.655155","exception":false,"start_time":"2022-05-02T09:50:35.594938","status":"completed"},"tags":[],"trusted":true}},{"cell_type":"code","source":"df.rename(columns={'fact_latitude': 'fact_longitude', 'fact_longitude': 'fact_latitude'}, inplace=True)\ndf_test.rename(columns={'fact_latitude': 'fact_longitude', 'fact_longitude': 'fact_latitude'}, inplace=True)","metadata":{"_uuid":"d45f8b55-fe33-4a13-9c1d-ce671ca83ef1","_cell_guid":"f50ffbe8-5ebd-401e-a61d-19ccb7c9cb65","collapsed":false,"papermill":{"duration":0.095302,"end_time":"2022-05-02T09:50:35.812447","exception":false,"start_time":"2022-05-02T09:50:35.717145","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:41.320829Z","iopub.execute_input":"2022-05-11T21:32:41.321342Z","iopub.status.idle":"2022-05-11T21:32:41.328105Z","shell.execute_reply.started":"2022-05-11T21:32:41.321295Z","shell.execute_reply":"2022-05-11T21:32:41.327020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's look at the columns that has NaN values.","metadata":{"_uuid":"a75533b5-3dcc-41bc-ac88-c661a0a05bf2","_cell_guid":"89e507bd-842e-4afb-a944-2c59e024c4e4","papermill":{"duration":0.064228,"end_time":"2022-05-02T09:50:35.938165","exception":false,"start_time":"2022-05-02T09:50:35.873937","status":"completed"},"tags":[],"trusted":true}},{"cell_type":"code","source":"nan_percentage = df.isna().sum()/len(df) *100\nnan_percentage = nan_percentage.sort_values(ascending=False)\n\n# Percent missing data by feature\nf, ax = plt.subplots(1, 1, sharex=True, figsize=(50,20))\nsns.barplot(ax=ax, x=nan_percentage.index, y=nan_percentage)\nax.set_xlabel('Features', fontsize=15)\nax.set_ylabel('Percent of missing values', fontsize=15)\nax.set_title('Percent missing data by feature - TRAINING DATA', fontsize=10)\nax.set_xticklabels(nan_percentage.sort_values(ascending=False).index, rotation=90)\n\nprint(\"Histograms of NaN values\")","metadata":{"_uuid":"67c9ef6b-e866-4e7c-a772-34a9611b3696","_cell_guid":"d7e17be1-24b6-42ac-bcfa-5d1bce499f1e","collapsed":false,"papermill":{"duration":0.618086,"end_time":"2022-05-02T09:50:36.619429","exception":false,"start_time":"2022-05-02T09:50:36.001343","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"The number of rows with Nan in the set to predict is:\", len(df_test[df_test.isnull().any(axis=1)]))","metadata":{"_uuid":"b7284e7f-340e-4537-8fb0-43ca69ad5ae5","_cell_guid":"03008778-e897-4f52-a536-72e50f1b28eb","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are no NaN values in the test so we do not have to look for this.","metadata":{"_uuid":"c8148b12-6855-4723-bbdb-898a6f25264a","_cell_guid":"2f6ea921-637e-4042-9539-935cbf4cead1","trusted":true}},{"cell_type":"code","source":"df_nan = df[df.isnull().any(axis=1)]\nnan_coordinates = df_nan[['fact_latitude','fact_longitude']].drop_duplicates()\nfig, ax0 = plt.subplots(1, 1, figsize=[20, 6], dpi=300, sharex=True, sharey=True)\nax0.scatter(nan_coordinates['fact_latitude'], nan_coordinates['fact_longitude'], \n            s=1, c='tab:green')\nfig.suptitle('Distribution of NaN in the train set worldwide', y=1.01, fontsize=14)\nplt.show()","metadata":{"_uuid":"a5c04222-a7df-43d0-bc55-bb3fd0bcb58d","_cell_guid":"de72f79d-c044-4b21-b7fd-17e7f2e0c31e","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the NaN values in our train set are well distributed regarding the original train set so we can remove them because there are few of them (arround 5%) and removing them does not change the distribution of our training set.","metadata":{"_uuid":"8f682641-bd1c-4f15-a089-2dcd7bf36d2c","_cell_guid":"d08d6dcb-3a24-4faa-b781-bbbc83a8ceb4","trusted":true}},{"cell_type":"code","source":"print(df.shape)\ndf_dif = df.dropna()\nprint(df_dif.shape)","metadata":{"_uuid":"d813bd9f-8c52-4156-a7ea-9f342bd1eb1b","_cell_guid":"6ed81f9c-c450-454f-9e62-0f50a0c27843","collapsed":false,"papermill":{"duration":1.863809,"end_time":"2022-05-02T09:50:39.531034","exception":false,"start_time":"2022-05-02T09:50:37.667225","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:41.329965Z","iopub.execute_input":"2022-05-11T21:32:41.330582Z","iopub.status.idle":"2022-05-11T21:32:42.832200Z","shell.execute_reply.started":"2022-05-11T21:32:41.330535Z","shell.execute_reply":"2022-05-11T21:32:42.831185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's print an image of our modified training set and see if we affected the difference between our new training set and the testing set.","metadata":{"_uuid":"09bcb17d-de53-40fa-9b7f-440c3b1f643a","_cell_guid":"6c27d340-a40a-46ec-97f0-1fb85295a7fb","trusted":true}},{"cell_type":"code","source":"df_coordinates = df_dif[['fact_latitude','fact_longitude']].drop_duplicates()\nfig, ax0 = plt.subplots(1, 1, figsize=[20, 6], dpi=300, sharex=True, sharey=True)\nax0.scatter(df_coordinates['fact_latitude'], df_coordinates['fact_longitude'], \n            s=1, c='tab:green')\nfig.suptitle('Distribution of the test set modified worldwide', y=1.01, fontsize=14)\nplt.show()","metadata":{"_uuid":"defb4ae9-2eac-4a29-a908-e9067d083782","_cell_guid":"fc74be47-d071-42d1-9fed-820325ef7a30","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The distribution looks more, as possible by the data available, to the testing set.","metadata":{"_uuid":"b1640db1-e68a-4fae-b2c4-53c407c2ae52","_cell_guid":"7c0d1dae-996d-4511-9588-647f992a291c","trusted":true}},{"cell_type":"code","source":"tr_coordinates = df[['fact_latitude','fact_longitude']].drop_duplicates()\nte_coordinates = df_test[['fact_latitude','fact_longitude']].drop_duplicates()\nmerged = te_coordinates.merge(tr_coordinates, how='left', indicator=True)\ntropic_coordinates = merged[merged['_merge']=='left_only']\ncommon_coordinates = merged[merged['_merge']=='both']\n\ntr_dif_coordinates = df_dif[['fact_latitude','fact_longitude']].drop_duplicates()\nnew_merged = te_coordinates.merge(tr_dif_coordinates, how='left', indicator=True)\ntropic_new_coordinates = new_merged[merged['_merge']=='left_only']\ncommon_new_coordinates = new_merged[merged['_merge']=='both']","metadata":{"_uuid":"0442cde3-f70b-42fd-b862-7d321839ceeb","_cell_guid":"2200adc5-fb45-4dae-bb7c-095b34f7ae98","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, (ax0, ax1) = plt.subplots(1, 2, figsize=[10, 3], dpi=200, sharex=True, sharey=True)\nax0.scatter(common_coordinates['fact_latitude'], common_coordinates['fact_longitude'], \n            s=1, c='tab:orange')\nax0.scatter(tropic_coordinates['fact_latitude'], tropic_coordinates['fact_longitude'], \n            s=1, c='tab:red')\nax1.scatter(common_new_coordinates['fact_latitude'], common_new_coordinates['fact_longitude'], \n            s=1, c='tab:orange')\nax1.scatter(tropic_new_coordinates['fact_latitude'], tropic_new_coordinates['fact_longitude'], \n            s=1, c='tab:red')\nfig.suptitle('Difference between the distribution of new train points and test points in the dataset before and after cleanning', y=1.01, fontsize=14)\nplt.show()\n\nprint(tr_dif_coordinates.shape, common_new_coordinates.shape, tropic_new_coordinates.shape)","metadata":{"_uuid":"bef717f9-efdc-4e31-906a-6a5271ac3e56","_cell_guid":"cc8ea612-93e6-49a7-9595-051819ed8db8","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that only 2,000 locations differ between our new testing set and the testing set and that it represents 28% of the testing set that is not present in the training set. So we really need our model to generalize and not to overfit the data of the testing set even though, 72% of the locations to predict are in the training set.","metadata":{"_uuid":"55d421f5-3628-4c9a-a116-523f88fd9f0e","_cell_guid":"fa9e53a3-b7d4-45ba-bfcb-a00b7b567850","trusted":true}},{"cell_type":"markdown","source":"Then, we decided to look at the repartion of the target to see if they were skewed or not and to compare it with normalized labels.","metadata":{"_uuid":"2f25d0d0-885f-4419-bf1c-b1983b0f8d9e","_cell_guid":"e9e22d80-f1c2-46de-a3d4-0eef0efa9fb3","trusted":true}},{"cell_type":"code","source":"array = (df_dif['fact_temperature'].values-df_dif['fact_temperature'].mean())/df_dif['fact_temperature'].std()\nfig, (ax1,ax2) = plt.subplots(figsize=(16, 4), ncols=2, dpi=100)\nsns.histplot(df_dif['fact_temperature'], ax=ax1);\nax1.set_title('Distribution of raw train target');\nsns.histplot(array, ax=ax2);\nax2.set_title('Distribution of train target normalized');","metadata":{"_uuid":"ef41fcee-1b13-442b-9501-4833dff397c5","_cell_guid":"ce0020b7-9dd8-4142-8784-aa1d71aa2cef","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that there are no real difference between the raw train target and the target normalized. Since normazling the labels is making the assumption that the training labels are normalized, we decided not to do this assumption since it does not give us real advantage regarding the distribution of the labels.","metadata":{"_uuid":"cd6e47ed-70cf-46cf-9363-7e3fddfcda91","_cell_guid":"5f214ae5-6bc3-4f2d-9f7b-cdfb78d920f9","trusted":true}},{"cell_type":"markdown","source":"We add the date information in classic format *YYYY-MM-DD HH:MM:SS* (and not UNIX) by converting them.","metadata":{"_uuid":"1fe373ff-f577-4c0f-828d-28f9c0897c2b","_cell_guid":"944006a0-67e9-4d05-b5ea-a45c2c2a19a0","papermill":{"duration":0.068949,"end_time":"2022-05-02T09:50:39.799712","exception":false,"start_time":"2022-05-02T09:50:39.730763","status":"completed"},"tags":[],"trusted":true}},{"cell_type":"code","source":"df_dif['date'] = pd.to_datetime(df_dif['fact_time'], unit='s')\ndf_dif['year'] = pd.DatetimeIndex(df_dif['date']).year\ndf_dif['month'] = pd.DatetimeIndex(df_dif['date']).month\ndf_dif['day'] = pd.DatetimeIndex(df_dif['date']).day\ndf_dif['hour'] = pd.DatetimeIndex(df_dif['date']).hour\ndf_dif['minute'] = pd.DatetimeIndex(df_dif['date']).minute","metadata":{"_uuid":"19bf39f5-dcd9-4b7c-9114-5f75b1d1d3be","_cell_guid":"98d7a685-7f47-400f-bcf0-02881c935725","collapsed":false,"papermill":{"duration":1.175676,"end_time":"2022-05-02T09:50:41.04165","exception":false,"start_time":"2022-05-02T09:50:39.865974","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:42.833872Z","iopub.execute_input":"2022-05-11T21:32:42.834416Z","iopub.status.idle":"2022-05-11T21:32:44.274837Z","shell.execute_reply.started":"2022-05-11T21:32:42.834370Z","shell.execute_reply":"2022-05-11T21:32:44.273832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test['date'] = pd.to_datetime(df_test['fact_time'], unit='s')\ndf_test['year'] = pd.DatetimeIndex(df_test['date']).year\ndf_test['month'] = pd.DatetimeIndex(df_test['date']).month\ndf_test['day'] = pd.DatetimeIndex(df_test['date']).day\ndf_test['hour'] = pd.DatetimeIndex(df_test['date']).hour\ndf_test['minute'] = pd.DatetimeIndex(df_test['date']).minute","metadata":{"_uuid":"3e0a9c2e-5b85-432c-8e8d-5b74e83b0f5c","_cell_guid":"9f99bab3-0424-44c4-97db-68e2a3ca2114","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:44.280386Z","iopub.execute_input":"2022-05-11T21:32:44.283142Z","iopub.status.idle":"2022-05-11T21:32:44.662789Z","shell.execute_reply.started":"2022-05-11T21:32:44.283086Z","shell.execute_reply":"2022-05-11T21:32:44.661827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, we wanted to look if there were issues in the training set regarding rows that have the same timestamp and localisation but not the same data inside.","metadata":{"_uuid":"fea14d7c-a89d-46d3-a737-685345a4f1d8","_cell_guid":"7e1f1ab2-159a-431b-9106-8a24d7ee338f","trusted":true}},{"cell_type":"code","source":"duplicateRowsDF = df_dif[df_dif.duplicated(['fact_longitude', 'fact_latitude', 'fact_time'],keep=False)]\nprint(\"The number of rows with the same localisation and timestamp but different data is:\", len(duplicateRowsDF))","metadata":{"_uuid":"b43d7ff6-7716-4239-90d1-ff74bf296750","_cell_guid":"f3a989e9-edb8-4f88-a199-4280fe37aade","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are a lot of those but we do not know what to do with those because which one to keep, could we average those? To know if we could just simply drop them, we decided to look at geographical distribution of those rows and to compare the temporal distribution of the train data when we drop those rows with the temporal distrubution of the test set.","metadata":{"_uuid":"0714e335-0f2e-4ed3-abbd-c66850983c66","_cell_guid":"9180ac29-4693-4063-b161-56b657f17ae4","trusted":true}},{"cell_type":"code","source":"duplicate_coordinates = duplicateRowsDF[['fact_latitude','fact_longitude']].drop_duplicates()\nfig, ax0 = plt.subplots(1, 1, figsize=[20, 6], dpi=300, sharex=True, sharey=True)\nax0.scatter(duplicate_coordinates['fact_latitude'], duplicate_coordinates['fact_longitude'], \n            s=1, c='tab:green')\nfig.suptitle('Distribution of duplicates in the test set worldwide', y=1.01, fontsize=14)\nplt.show()\n\nprint(duplicate_coordinates.shape)","metadata":{"_uuid":"0d9d6794-fd45-47fa-9d14-a06bf80c927b","_cell_guid":"93477fd2-882c-4b75-8238-f0d9574dd1f1","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It looks a lot to the plot we made before so we can be reconforted to the possibility to drop those rows.","metadata":{"_uuid":"118628a0-22ee-4121-93db-9f6b394348a0","_cell_guid":"61a9f6e6-71d4-4187-b4d8-d615d3cde313","trusted":true}},{"cell_type":"code","source":"df_dif = df_dif.drop_duplicates(subset=['fact_longitude', 'fact_latitude', 'fact_time'],keep=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-11T21:32:44.664160Z","iopub.execute_input":"2022-05-11T21:32:44.664507Z","iopub.status.idle":"2022-05-11T21:32:46.423751Z","shell.execute_reply.started":"2022-05-11T21:32:44.664463Z","shell.execute_reply":"2022-05-11T21:32:46.422565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_dif.groupby([df_dif[\"year\"], df_dif[\"month\"], df_dif[\"day\"]]).count().plot(kind=\"bar\",legend=False)","metadata":{"_uuid":"fbadeb11-d050-44c5-9531-cce64866013d","_cell_guid":"ea26b14e-ac13-4c1a-b89a-3a31ae2c9f33","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.groupby([df_test[\"year\"], df_test[\"month\"], df_test[\"day\"]]).count().plot(kind=\"bar\",legend=False)","metadata":{"_uuid":"23e879cc-3df8-459b-b5c1-128dc1b7ebfc","_cell_guid":"e9356684-327e-4969-bbdd-3671ca67122e","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed, it does not change the time distribution of our training set regarding to our new testing set so it is positive. We can drop those rows.","metadata":{"_uuid":"ef351d29-0870-4201-9bb8-8dd473eebab4","_cell_guid":"dabdeacd-794e-47cd-8237-9179d6e61134","trusted":true}},{"cell_type":"markdown","source":"We decided to add some columns by calculating for each place the monthly and daily average climate temperature.","metadata":{"_uuid":"cc18a79b-aa6c-45e8-8804-cd443ee4060a","_cell_guid":"0e570123-ac17-4a70-9c48-9d6f3e224f7b","trusted":true}},{"cell_type":"code","source":"df_dif['localisation_monthly_mean'] = df_dif.groupby([\"year\", \"month\", \"fact_longitude\",\"fact_longitude\"])['climate_temperature'].transform('mean')\ndf_dif['localisation_daily_mean'] = df_dif.groupby([\"year\", \"day\", \"fact_longitude\",\"fact_longitude\"])['climate_temperature'].transform('mean')\ndf_test['localisation_monthly_mean'] = df_test.groupby([\"year\", \"month\", \"fact_longitude\",\"fact_longitude\"])['climate_temperature'].transform('mean')\ndf_test['localisation_daily_mean'] = df_test.groupby([\"year\", \"day\", \"fact_longitude\",\"fact_longitude\"])['climate_temperature'].transform('mean')","metadata":{"_uuid":"e229b804-f263-43d2-a55e-db44e1cc2ffe","_cell_guid":"d8adebd5-d48b-4239-bcab-23e8cb19aaa4","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:46.425943Z","iopub.execute_input":"2022-05-11T21:32:46.426250Z","iopub.status.idle":"2022-05-11T21:32:47.432721Z","shell.execute_reply.started":"2022-05-11T21:32:46.426206Z","shell.execute_reply":"2022-05-11T21:32:47.431768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We plot the corelation matrix in order to identify important features and do features selection.","metadata":{"_uuid":"41023a0d-4a12-4c95-a24c-1e684a9d7018","_cell_guid":"fb3db8ca-6b3f-4a0b-8d5c-6271ff3cebce","trusted":true}},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(30,30 ))\ncorr_df=df.corr()\nsns.heatmap(corr_df, mask=np.zeros_like(corr_df, dtype=np.bool), cmap=sns.diverging_palette(220, 10, as_cmap=True),\n            square=True, ax=ax)","metadata":{"_uuid":"f691092d-1615-40c3-8ca0-b40239f33f4b","_cell_guid":"3273fb30-ebf8-438d-8339-5b96a7943531","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### Correlation with output variable\ncor_target = abs(corr_df[\"fact_temperature\"])\n#Selecting highly correlated features\nrelevant_features = cor_target[cor_target>0.5]\nrelevant_features.sort_values(ascending=False)","metadata":{"_uuid":"4de9aae5-8966-4dd0-b37a-99620dee7825","_cell_guid":"b5b8556b-8390-4754-9f5c-433f1ca09990","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**4. Models**","metadata":{"_uuid":"1743ebc9-10d5-4cd2-be76-76f39366beb7","_cell_guid":"9e37d1d9-b339-4614-8862-859003390f9b","trusted":true}},{"cell_type":"markdown","source":"**4.1) Linear Regression**","metadata":{"_uuid":"c16e7489-2b27-4a0a-b7f9-dafd6e4030f0","_cell_guid":"8cd63b78-38fd-4d90-8a9e-8ccdc1ae3a7a","trusted":true}},{"cell_type":"code","source":"#We save in col_selection the names of the most correlated columns we use to train our model, given by the correlation matrix computed earlier\n\ncol_selection = [\n'fact_latitude','fact_longitude','climate_temperature','climate_pressure','topography_bathymetry','sun_elevation','wrf_psfc','gfs_precipitable_water','cmc_0_1_0_0','cmc_0_1_11_0','cmc_0_1_65_0','cmc_0_1_66_0','cmc_0_1_67_0','cmc_0_1_68_0','cmc_0_1_7_0',\n    'gfs_temperature_95000','gfs_temperature_90000', 'gfs_temperature_85000','gfs_temperature_80000', 'gfs_temperature_75000','gfs_temperature_70000','gfs_temperature_65000','gfs_temperature_60000','gfs_temperature_55000','gfs_temperature_50000','gfs_temperature_45000','gfs_temperature_40000','gfs_temperature_35000','gfs_temperature_30000','gfs_temperature_25000',\n    'cmc_0_0_6_2','cmc_0_3_0_0', 'gfs_humidity', 'gfs_precipitable_water', 'wrf_t2_next', 'wrf_rh2', 'wrf_rain', 'wrf_snow', 'gfs_2m_dewpoint_next', 'gfs_2m_dewpoint_grad',\n    'cmc_0_2_2_10','cmc_0_2_3_10','cmc_0_6_1_0',\n    'fact_temperature'\n]","metadata":{"_uuid":"efc8cbf1-2243-417b-959e-d29aa567d9d1","_cell_guid":"d236a3d1-d17d-4813-b82f-3b4dbdfc83c9","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:47.436510Z","iopub.execute_input":"2022-05-11T21:32:47.437040Z","iopub.status.idle":"2022-05-11T21:32:47.445054Z","shell.execute_reply.started":"2022-05-11T21:32:47.436993Z","shell.execute_reply":"2022-05-11T21:32:47.443986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = df_dif[col_selection].iloc[:, :-1].values\ny = df_dif[col_selection].iloc[:, -1].values","metadata":{"_uuid":"508f30d2-1f8f-43e8-ba77-ab4912a588ff","_cell_guid":"0d255d87-76ee-4840-a9cd-d637d5c71e85","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:47.446821Z","iopub.execute_input":"2022-05-11T21:32:47.448165Z","iopub.status.idle":"2022-05-11T21:32:49.614966Z","shell.execute_reply.started":"2022-05-11T21:32:47.447915Z","shell.execute_reply":"2022-05-11T21:32:49.613923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Xmean, Xstd, ymean, ystd = X.mean(0), X.std(0), y.mean(), y.std()\nX = (X - Xmean) / Xstd\ny = (y - ymean) / ystd","metadata":{"_uuid":"2e533808-8068-4a18-b538-cdec6bd575d0","_cell_guid":"fcefb145-dce1-4a05-810d-541498071197","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:49.616457Z","iopub.execute_input":"2022-05-11T21:32:49.616761Z","iopub.status.idle":"2022-05-11T21:32:50.505484Z","shell.execute_reply.started":"2022-05-11T21:32:49.616718Z","shell.execute_reply":"2022-05-11T21:32:50.504477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Xtr, Xval, ytr, yval = train_test_split(X, y, random_state=1, test_size=10000)","metadata":{"_uuid":"31a7ceb4-68fc-4bd8-9abc-afd41a661475","_cell_guid":"0dccb2b3-a8db-494b-a204-4cf34c876225","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:50.507320Z","iopub.execute_input":"2022-05-11T21:32:50.507638Z","iopub.status.idle":"2022-05-11T21:32:51.521894Z","shell.execute_reply.started":"2022-05-11T21:32:50.507595Z","shell.execute_reply":"2022-05-11T21:32:51.520893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Defining a fonction to compute RMSE of our model\n\ndef compute_rmse(y, ypred, ystd=1.):\n    return np.mean((y - ypred)**2)**0.5 * ystd\n\nmodel = LinearRegression()\nmodel.fit(Xtr, ytr)\nypred_tr = model.predict(Xtr)\nypred_val = model.predict(Xval)\n\nprint(f'Train RMSE: {compute_rmse(ytr, ypred_tr, ystd):.3f}')\nprint(f'Valid RMSE: {compute_rmse(yval, ypred_val, ystd):.3f}')","metadata":{"_uuid":"c11c95d3-af4c-455b-b62a-4dbf528f561b","_cell_guid":"34998625-2baf-4db7-9076-68f1be4c50f2","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:51.523674Z","iopub.execute_input":"2022-05-11T21:32:51.524031Z","iopub.status.idle":"2022-05-11T21:32:57.023920Z","shell.execute_reply.started":"2022-05-11T21:32:51.523983Z","shell.execute_reply":"2022-05-11T21:32:57.022771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Xmean, Xstd, ymean, ystd = X.mean(0), X.std(0), y.mean(), y.std()\nXte = df_test[col_selection[:-1]].values\nXte = (Xte - Xmean) / Xstd","metadata":{"_uuid":"3a48eb9e-8328-46e4-9235-1e8a398482b8","_cell_guid":"c7f22c2b-7e3d-4d6a-bbae-960d54fe32f3","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Prediction using a basic Linear Regression model \nypred_te = model.predict(Xte) * ystd + ymean  # Remember to un-standardize the predictions","metadata":{"_uuid":"88f43609-18fd-48ac-8ee1-a0db42c558df","_cell_guid":"c778c27a-6c45-45c7-bf93-0ff9bd0fa5cf","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**4.2) Creativity**","metadata":{"_uuid":"8461e2b1-154a-4366-b37c-1e685227b7af","_cell_guid":"7c7e213e-f840-4504-baba-a025f7f0289b","trusted":true}},{"cell_type":"markdown","source":"We tried to split our dataset in 9 parts according to their longitude and latitude in order to ameliorate the linear regression.","metadata":{"_uuid":"e23b3376-84e1-48dd-8035-85c054d7ae6c","_cell_guid":"bb8c86af-52bf-4bd8-ab77-9b3758d472bf","trusted":true}},{"cell_type":"code","source":"df_north_left = df_dif[(df_dif['fact_longitude']> 20) & (df_dif['fact_latitude']< -25)]\ndf_north_middle = df_dif[(df_dif['fact_longitude']> 20) & (df_dif['fact_latitude']>=-25) & (df_dif['fact_latitude']< 50)]\ndf_north_right = df_dif[(df_dif['fact_longitude']> 20) & (df_dif['fact_latitude']>=50)]\ndf_middle_left = df_dif[(df_dif['fact_longitude']<= 20) & (df_dif['fact_longitude']>= -20) & (df_dif['fact_latitude']< -25)]\ndf_middle_middle = df_dif[(df_dif['fact_longitude']<= 20) & (df_dif['fact_longitude']>= -20) & (df_dif['fact_latitude']>=-25) & (df_dif['fact_latitude']< 50)]\ndf_middle_right = df_dif[(df_dif['fact_longitude']<= 20) & (df_dif['fact_longitude']>= -20) & (df_dif['fact_latitude']>=50)]\ndf_south_left = df_dif[(df_dif['fact_longitude']< -20) & (df_dif['fact_latitude']< -25)]\ndf_south_middle = df_dif[(df_dif['fact_longitude']< -20) & (df_dif['fact_latitude']>=-25) & (df_dif['fact_latitude']< 50)]\ndf_south_right = df_dif[(df_dif['fact_longitude']< -20) & (df_dif['fact_latitude']>=50)]\n\nprint('size dataset north left:', df_north_left.shape)\nprint('size dataset north middle:', df_north_middle.shape)\nprint('size dataset north right:', df_north_right.shape)\nprint('size dataset middle left:', df_middle_left.shape)\nprint('size dataset middle middle:', df_middle_middle.shape)\nprint('size dataset middle right:', df_middle_right.shape)\nprint('size dataset south left:', df_south_left.shape)\nprint('size dataset south south:', df_south_middle.shape)\nprint('size dataset south right:', df_south_right.shape)","metadata":{"_uuid":"1534c66d-8bd3-479d-8a12-8999e1bf5594","_cell_guid":"7f4d14aa-06f5-4173-ab66-80534af6cc64","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr_df_NL = df_north_left.corr(method='pearson')\ncol_selection = []\nfor i in range (len(corr_df_NL.columns)):\n    if abs(corr_df_NL[\"fact_temperature\"][i]) > 0.6:\n        col_selection.append(corr_df_NL.columns[i])\nNL = col_selection\n\nX_north_left = df_north_left[col_selection].dropna().iloc[:, :-1].values\ny_north_left = df_north_left[col_selection].dropna().iloc[:, -1].values\nprint(y_north_left)\nXmean_north_left, Xstd_north_left, ymean_north_left, ystd_north_left = X_north_left.mean(0), X_north_left.std(0), y_north_left.mean(), y_north_left.std()\nX_north_left=(X_north_left-X_north_left.mean(0))/X_north_left.std(0)\ny_north_left=(y_north_left-y_north_left.mean(0))/y_north_left.std(0)","metadata":{"_uuid":"615dd11a-3702-49c9-85af-8c3e00d1afc8","_cell_guid":"72220e45-fa44-481d-b118-0f7e8a91a9c2","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Xtr, Xval, ytr, yval = train_test_split(X_north_left, y_north_left, random_state=1, test_size=80000)\n\nmodel_north_left = LinearRegression()\nmodel_north_left.fit(Xtr, ytr)\nypred_tr = model_north_left.predict(Xtr)\nprint(ypred_tr*ystd_north_left+ymean_north_left)\nypred_val = model_north_left.predict(Xval)\nprint(ypred_val*ystd_north_left+ymean_north_left)\n\nprint(f'Train RMSE: {compute_rmse(ytr, ypred_tr, ystd_north_left):.3f}')\nprint(f'Valid RMSE: {compute_rmse(yval, ypred_val, ystd_north_left):.3f}')","metadata":{"_uuid":"fd271705-a3bc-41d8-9077-3d021156724c","_cell_guid":"c692691e-9df7-463b-9fe8-986ee2ed3a44","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We ran a linear regression on each of the ninth dataset (we did not keep the code synthetise). But result obtain were not enougt satisfactory so we came back to the entire dataset.","metadata":{"_uuid":"5a68dbaf-2507-436d-b546-d2e5a72c084f","_cell_guid":"02f2d025-c2b5-41f1-b1aa-90483f8c3e4a","trusted":true}},{"cell_type":"markdown","source":"**4.3) Lasso Regression**","metadata":{"_uuid":"d30df419-2720-42fe-ad45-52efc9609b27","_cell_guid":"14bb06b4-7146-4222-8f5b-3c6d8e010876","trusted":true}},{"cell_type":"code","source":"#We save in col_selection the names of the most correlated columns we use to train our model, given by the correlation matrix computed earlier\n\ncol_selection = [\n'climate_temperature', 'cmc_0_0_6_2', 'cmc_0_1_0_0', 'cmc_0_3_5_500', 'gfs_2m_dewpoint', 'gfs_precipitable_water', 'gfs_temperature_25000','gfs_temperature_30000','gfs_temperature_40000','gfs_temperature_45000',\n'gfs_temperature_50000', 'gfs_temperature_55000', 'gfs_temperature_60000','gfs_temperature_65000',\n'gfs_temperature_70000','gfs_temperature_75000','gfs_temperature_80000','gfs_temperature_85000',\n'gfs_temperature_90000','gfs_temperature_92500','gfs_temperature_95000','gfs_temperature_97500',\n'wrf_t2_next','wrf_t2_interpolated','gfs_2m_dewpoint_next'\n]\n\n#Normalization of our data set\nX = df_dif[col_selection].values\ny = df_dif[\"fact_temperature\"].values\n\nXmean, Xstd, ymean, ystd = X.mean(0), X.std(0), y.mean(), y.std()\nX = (X - Xmean) / Xstd\ny = (y - ymean) / ystd\n\n#Splitting data set into training and validation sets\n\nXtr, Xval, ytr, yval = train_test_split(X, y, random_state=1, test_size=10000)\n\n#Here we use Lasso regression, which is an enhanced linear regression model trained with L1 prior as regularizer\n#First we don't optimize alpha hyperparameter\n\nmodel = Lasso(alpha=1)\nmodel.fit(Xtr, ytr)\nypred_tr = model.predict(Xtr)\nypred_val = model.predict(Xval)\n\nprint(f'Train RMSE: {compute_rmse(ytr, ypred_tr, ystd):.3f}')\nprint(f'Valid RMSE: {compute_rmse(yval, ypred_val, ystd):.3f}')","metadata":{"_uuid":"5a938ba1-7da7-48ea-8ae2-9e59a1b5f2d5","_cell_guid":"f21856f7-4026-45ee-a0a2-7ca24a3af63c","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's try to optimize our parameters.","metadata":{"_uuid":"817f5109-d902-480a-ba4c-029cd6e41ec8","_cell_guid":"42710799-0605-428d-96c4-8590e66c4fa8","trusted":true}},{"cell_type":"code","source":"#Optimization of alpha hyperparameter\n\ndef get_cv_idx(n, test_size=0.2, n_splits=2):\n    train_idx, test_idx = [], []\n    for _ in range(n_splits):\n        idx = np.random.permutation(n)\n        train_size = int(n * (1 - test_size)) if isinstance(test_size, float) else n - test_size\n        train_idx.append(idx[:train_size])\n        test_idx.append(idx[train_size:])\n    return train_idx, test_idx\n\ntrain_idx, cv_idx = get_cv_idx(len(Xtr), test_size=10000, n_splits=10)\n\nparam_grid = {\n    \"alpha\": [0.001, 0.01, 0.1, 1, 10]\n}\n\nsearch = GridSearchCV(model, \n                      param_grid,\n                      n_jobs=-1, \n                      verbose=1,\n                      cv=zip(train_idx, cv_idx), \n                      scoring='neg_root_mean_squared_error').fit(Xtr, ytr)\n\n#Normalizing test data set\n\nXte = df_test[col_selection].values\nXte = (Xte - Xmean) / Xstd\n\n#Prediction using Lasso model with optimized alpha hyperparameter\n\nypred_te = search.predict(Xte) * ystd + ymean  # Remember to un-standardize the predictions","metadata":{"_uuid":"46e430fd-be92-41f8-8658-508f75d0701d","_cell_guid":"998d513b-815f-46b9-a6bd-0ee0a4d48d0b","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The result improved with the optimization: we obtain a RMSE close to 2.4, so sligthly better than linear regression.","metadata":{"_uuid":"5b0744ce-9ae1-444b-9f08-d9b8a8a94a66","_cell_guid":"9158f6d4-624f-46a3-bb0c-d4b9f9b4e65e","trusted":true}},{"cell_type":"markdown","source":"**4.4) XGBoost**","metadata":{"_uuid":"9df2e20b-6aeb-40d3-a379-9377615d256e","_cell_guid":"252927ee-e30d-4cae-a9b4-7182fb8711b2","trusted":true}},{"cell_type":"markdown","source":"Finally, research showed us that XGBoost could be a working model. XGBoost is an implementation of the gradient boosted trees algorithm. We tried to implement it to see if this model could improve our result.","metadata":{"_uuid":"e3a8dde4-1637-42fc-ac22-72a5666807c6","_cell_guid":"3007214e-ac17-4bf0-84b6-cdd7f070ab41","trusted":true}},{"cell_type":"markdown","source":"First, we choose only the columns we need. Therefore we drop the index and all the coluns that tell us the data is available.","metadata":{}},{"cell_type":"code","source":"Xtr, Xval, ytr, yval = train_test_split(df_dif.drop(['index','wrf_available','gfs_available','cmc_available','date','gfs_soil_temperature_available','fact_temperature'], axis=1), df_dif['fact_temperature'], test_size=10000)\nXte=df_test.drop(['index','wrf_available','gfs_available','cmc_available','date','gfs_soil_temperature_available'], axis=1).values","metadata":{"_uuid":"68a33b7d-489f-4d5a-aba8-538ddc7bca90","_cell_guid":"10a362bf-4528-4259-9e4e-517e402069b9","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:32:57.025884Z","iopub.execute_input":"2022-05-11T21:32:57.026587Z","iopub.status.idle":"2022-05-11T21:33:01.642405Z","shell.execute_reply.started":"2022-05-11T21:32:57.026531Z","shell.execute_reply":"2022-05-11T21:33:01.641335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then we normalize our features using RobustScalar. This normalization is interesting because it allows very high temperature (data present in our features like climate_temperature), not to be centered too much. It normalizes the data while keeping outliers from distance to the middle. Therefore, here we assume that our modified train dataset is composed of exact data (there are no errors anymore).","metadata":{}},{"cell_type":"code","source":"scaler_features = RobustScaler()\nXtr = scaler_features.fit_transform(Xtr)\nXval = scaler_features.transform(Xval)\nXte =scaler_features.transform(Xte)\nprint(Xtr)","metadata":{"_uuid":"ab4e3596-d3ed-4935-bfbd-8161b954f8b8","_cell_guid":"21cf10c2-7c52-4f72-b603-eb5aca3d9123","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:33:01.644349Z","iopub.execute_input":"2022-05-11T21:33:01.644706Z","iopub.status.idle":"2022-05-11T21:33:11.265354Z","shell.execute_reply.started":"2022-05-11T21:33:01.644648Z","shell.execute_reply":"2022-05-11T21:33:11.264299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, we compute the variance of each features and we drop the features that are not enough variating in the dataset using a threshold of 0.4. They would not bring real information to our model.","metadata":{}},{"cell_type":"code","source":"selector_features = VarianceThreshold(threshold=0.4)\nXtr = selector_features.fit_transform(Xtr)\nXval = selector_features.transform(Xval)\nXte = selector_features.transform(Xte)\nselector_features.get_support()","metadata":{"execution":{"iopub.status.busy":"2022-05-11T21:33:11.266852Z","iopub.execute_input":"2022-05-11T21:33:11.267721Z","iopub.status.idle":"2022-05-11T21:33:13.663249Z","shell.execute_reply.started":"2022-05-11T21:33:11.267665Z","shell.execute_reply":"2022-05-11T21:33:13.662168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To optimize the speed of the model fitting, we use the GPU acceleration offered by Kaggle.","metadata":{}},{"cell_type":"code","source":"model = xg.XGBRegressor(objective ='reg:squarederror', tree_method='gpu_hist',\n                  n_estimators = 1000, seed = 123,learning_rate=0.21,max_depth=7,sub_sample=0.5,colsample_bytree=0.6,colsample_bylevel=0.8999)","metadata":{"_uuid":"63bd761f-a2df-4c15-aa68-0883fd57e544","_cell_guid":"7002200d-54a9-4ce8-8bcd-7fd90d5e8252","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:33:13.665265Z","iopub.execute_input":"2022-05-11T21:33:13.665631Z","iopub.status.idle":"2022-05-11T21:33:13.671407Z","shell.execute_reply.started":"2022-05-11T21:33:13.665587Z","shell.execute_reply":"2022-05-11T21:33:13.670368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(Xtr, ytr)\n  \n# Predict the model\nypred_tr = model.predict(Xtr)\nypred_val = model.predict(Xval)\nypred_test = model.predict(Xte)","metadata":{"_uuid":"80d5220a-bdcc-41ee-91b3-07eec53b75d6","_cell_guid":"aa3a07d2-4c14-41c7-a267-16d35fd2e75b","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-05-11T21:33:13.673297Z","iopub.execute_input":"2022-05-11T21:33:13.673993Z","iopub.status.idle":"2022-05-11T21:35:01.721419Z","shell.execute_reply.started":"2022-05-11T21:33:13.673946Z","shell.execute_reply":"2022-05-11T21:35:01.720406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Train RMSE: {compute_rmse(ytr, ypred_tr):.3f}')\nprint(f'Valid RMSE: {compute_rmse(yval, ypred_val):.3f}')","metadata":{"execution":{"iopub.status.busy":"2022-05-11T21:35:57.683236Z","iopub.execute_input":"2022-05-11T21:35:57.683777Z","iopub.status.idle":"2022-05-11T21:35:57.740050Z","shell.execute_reply.started":"2022-05-11T21:35:57.683740Z","shell.execute_reply":"2022-05-11T21:35:57.738987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We obtain better results than before. We will try to tune hyperparameters on this model to get closer to the temperature.","metadata":{"_uuid":"57fd5994-4128-4e8e-b1ef-feb515f1df92","_cell_guid":"ad280c0b-22cc-4caa-840a-c0a813ab5467","trusted":true}},{"cell_type":"markdown","source":"**5. Hypertuning of hyperparameters**","metadata":{"_uuid":"b64caa7c-01ef-4260-bd7b-acedf190a8b4","_cell_guid":"6b840fea-40a8-4fca-bc30-4f6193da0626","trusted":true}},{"cell_type":"markdown","source":"To find out the best hyperparameters for XGBoost, we apply the two most common algorithms: GridSearch and RandomSearch. We searched over every combination of specified parameter values.","metadata":{"_uuid":"ab7aa7d5-536e-4a51-8577-eae5db1d566c","_cell_guid":"f8882bae-3711-4fed-8ee8-1063a8af3a7d","trusted":true}},{"cell_type":"code","source":"params = { \n           'learning_rate': [0.01, 0.1, 0.18, 0.2, 0.22, 0.24, 0.3],\n           'n_estimators': [100, 200, 400,600,800,1000,1500,2000,2500,3000],\n            'max_depth':[4,5,6,7,8,9,10],\n            'subsample': np.arange(0.5, 1.0, 0.1),\n           'colsample_bytree': np.arange(0.4, 1.0, 0.1),\n           'colsample_bylevel': np.arange(0.4, 1.0, 0.1)\n}\nsearch = RandomizedSearchCV(estimator=model,\n                         param_distributions=params,\n                         scoring='neg_mean_squared_error',\n                         n_iter=5,\n                         verbose=1)\nsearch.fit(Xtr, ytr)","metadata":{"_uuid":"a642a4e6-9468-4c6e-8577-d9edbf38fac6","_cell_guid":"caee9378-3ed4-4b1b-a9a6-405d99966563","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Best parameters set found on cv set:\")\nprint(search.best_params_)\nprint()\nprint(\"Error on the trainning set\")\nypred_tr = search.predict(Xtr)\nprint(f'Train RMSE: {compute_rmse(ytr, ypred_tr):.3f}')\nprint()\nprint(\"Error on the validation set\")\nypred_val = search.predict(Xval)\nprint(f'Valid RMSE: {compute_rmse(yval, ypred_val):.3f}')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#ypred_te=search.predict(Xte)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df = pd.DataFrame(data={'index': df_test['index'].values,\n                                   'fact_temperature': ypred_te.squeeze()})\n\n# Save the predictions into a csv file\n# Notice that this file should be saved under the directory `/kaggle/working` \n# so that you can download it later\nsubmission_df.to_csv(\"/kaggle/working/submission.csv\", index=False)","metadata":{"_uuid":"0f1cfa4e-6bce-4bca-9d9a-59f41a49c936","_cell_guid":"ef32951a-823c-4846-a88f-b53396d96187","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**6. Best model**","metadata":{}},{"cell_type":"markdown","source":"The best model we obtain after hyperparameters optimization and some manual tests we made has those hyperparameters:\n* variance threshold of 2.5,\n* objective ='reg:squarederror', \n* tree_method='gpu_hist',\n* n_estimators = 10000,\n* seed = 123,\n* learning_rate=0.21,\n* max_depth=7,\n* sub_sample=0.5,\n* colsample_bytree=0.6,\n* colsample_bylevel=0.8999999","metadata":{}},{"cell_type":"code","source":"Xtr, Xval, ytr, yval = train_test_split(df_dif.drop(['index','wrf_available','gfs_available','cmc_available','date','gfs_soil_temperature_available','fact_temperature'], axis=1), df_dif['fact_temperature'], test_size=10000)\nXte=df_test.drop(['index','wrf_available','gfs_available','cmc_available','date','gfs_soil_temperature_available'], axis=1).values","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scaler_features = RobustScaler()\nXtr = scaler_features.fit_transform(Xtr)\nXval = scaler_features.transform(Xval)\nXte =scaler_features.transform(Xte)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"selector_features = VarianceThreshold(threshold=0.25)\nXtr = selector_features.fit_transform(Xtr)\nXval = selector_features.transform(Xval)\nXte = selector_features.transform(Xte)\nselector_features.get_support()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_model = xg.XGBRegressor(objective ='reg:squarederror', tree_method='gpu_hist',\n                  n_estimators = 10000, seed = 123,learning_rate=0.21,max_depth=7,sub_sample=0.5,colsample_bytree=0.6,colsample_bylevel=0.8999999)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_model.fit(Xtr, ytr)\n  \n# Predict the model\nypred_tr = best_model.predict(Xtr)\nypred_val = best_model.predict(Xval)\nypred_test = best_model.predict(Xte)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Error on the trainning set\")\nypred_tr = model.predict(Xtr)\nprint(f'Train RMSE: {compute_rmse(ytr, ypred_tr):.3f}')\nprint()\nprint(\"Error on the validation set\")\nypred_val = model.predict(Xval)\nprint(f'Valid RMSE: {compute_rmse(yval, ypred_val):.3f}')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ypred_te = model.predict(Xte)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df = pd.DataFrame(data={'index': df_test['index'].values,\n                                   'fact_temperature': ypred_te.squeeze()})\n\nsubmission_df.to_csv(\"/kaggle/working/submission.csv\", index=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**7. Conclusion**","metadata":{"_uuid":"0bbe2a97-8718-4971-a78d-5a4740d58de4","_cell_guid":"b02fc621-672c-4898-b8ad-bcfd7ef52253","trusted":true}},{"cell_type":"markdown","source":"- To conclude, this challenge showed us that domain knowledge is important for every machine learning project and that it is necessary to understand well manipulated data. \n- Therefore, our group decided to go for a data-driven approach, which had more sense for us than a model-driven approach. Finding out what is consistent, understanding different patterns in the dataset helps us to have a better understanding of how the data was generated and how to deal with those data such that a chosen model understands it. \n- Concerning the choice of models, we started by implementing a basic linear regression and lasso regression as we can have a great control of the given parameters and then, we decided to move towards to XGBoost, which has revealed to give the best results in terms of prediction.","metadata":{"_uuid":"c03ad113-94f5-4284-b695-0313f9ef7de5","_cell_guid":"1cdc3470-749a-47fb-bf25-8b1e90874127","trusted":true}}]}