{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":8533339,"sourceType":"datasetVersion","datasetId":5048722}],"dockerImageVersionId":30698,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.preprocessing import LabelEncoder\n\nfrom sklearn.model_selection import train_test_split\nfrom datetime import timedelta\nimport gc\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-05-28T09:07:30.274197Z","iopub.execute_input":"2024-05-28T09:07:30.274580Z","iopub.status.idle":"2024-05-28T09:07:31.624154Z","shell.execute_reply.started":"2024-05-28T09:07:30.274550Z","shell.execute_reply":"2024-05-28T09:07:31.622929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Introduction**\n\n The objective of this project is to develop a model to accurately forecast demand for grocery retailers. This will help reduce food waste and ensure product availability, addressing a critical issue in retail operations.\n","metadata":{"execution":{"iopub.status.busy":"2024-05-20T12:35:43.864191Z","iopub.execute_input":"2024-05-20T12:35:43.864622Z","iopub.status.idle":"2024-05-20T12:35:43.871069Z","shell.execute_reply.started":"2024-05-20T12:35:43.864591Z","shell.execute_reply":"2024-05-20T12:35:43.869286Z"}}},{"cell_type":"markdown","source":"# **Business Understanding**\n\n Sales is the primary parameter for measuring the success of any business, and Corporation Favorita is no exception. The business has the data of its sales from 2013 to 2017 across its stores located in different cities in Ecuador.\n\nDuring this time period, there have been holiday events, increases in oil price, and an earthquake in Ecuador. The grocery store management will like to know the impact of these occurrences on its sales. The management also wants to know the sales performance of its stores across different store types, clusters, cities and states.\n\nFurthermore, the management will like to know if there is a trend in the sales data, as well as the occurrence of seasonality to help identify the buying patterns of customers and prepare adequately for periods in the future where peak sales usually occur. The impact of promotion on sales is another key factor the business wants to derive insights on.\n\nThis project is aimed at providing these business insights by performing regression analysis using the available time-stamped historical data. In addition, and an optimal regression model will be built to predict future sales. This prediction will ensure strategic decision-making in the future.\n\nBy delving into the project, our goal is to support the management team of Corporation Favorita in extracting meaningful insights from their vast data, optimize operations and ultimately achieve sales growth, and help drive data-informed decision-making.","metadata":{}},{"cell_type":"markdown","source":"# **Data Exploration and Preprocessing**\n\nThe purpose of this step is to explore the different datasets provided by Kaggle and if there\n ","metadata":{}},{"cell_type":"code","source":"#let's load all the data\ntrain_data = pd.read_csv('/kaggle/input/grocery-data/train.csv' )\ntest_data = pd.read_csv('/kaggle/input/grocery-data/test.csv' )\nholidays_data = pd.read_csv('/kaggle/input/grocery-data/holidays_events.csv' )\noil_data = pd.read_csv('/kaggle/input/grocery-data/oil.csv' )\nstores_data = pd.read_csv('/kaggle/input/grocery-data/stores.csv' )\ntransactions_data = pd.read_csv('/kaggle/input/grocery-data/transactions.csv' )\n","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:07:35.040603Z","iopub.execute_input":"2024-05-28T09:07:35.041153Z","iopub.status.idle":"2024-05-28T09:07:38.532935Z","shell.execute_reply.started":"2024-05-28T09:07:35.041122Z","shell.execute_reply":"2024-05-28T09:07:38.531631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initial Exploration\ndef initial_exploration(df, name):\n    print(f\"Dataset: {name}\")\n    print(\"Shape:\", df.shape)\n    print(\"Columns:\", df.columns)\n    print(\"First few rows:\\n\", df.head())\n    print(\"Summary statistics:\\n\", df.describe())\n    print(\"Missing values:\\n\", df.isnull().sum())\n    print(\"\\n\")\n\n","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.236682Z","iopub.status.idle":"2024-05-27T23:29:44.237039Z","shell.execute_reply.started":"2024-05-27T23:29:44.236866Z","shell.execute_reply":"2024-05-27T23:29:44.236881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Handling Missing Values and Data Cleaning\n","metadata":{}},{"cell_type":"markdown","source":"**HOLIDAYS Data**","metadata":{}},{"cell_type":"code","source":"holidays_data.head()\nholidays_data.isna().sum()\nholidays_data = holidays_data.rename(columns={'type': 'holiday_type'})","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:07:38.981644Z","iopub.execute_input":"2024-05-28T09:07:38.982354Z","iopub.status.idle":"2024-05-28T09:07:38.996722Z","shell.execute_reply.started":"2024-05-28T09:07:38.982310Z","shell.execute_reply":"2024-05-28T09:07:38.995589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"holidays_data['holiday_type'].unique()","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:07:40.085374Z","iopub.execute_input":"2024-05-28T09:07:40.086078Z","iopub.status.idle":"2024-05-28T09:07:40.096604Z","shell.execute_reply.started":"2024-05-28T09:07:40.086032Z","shell.execute_reply":"2024-05-28T09:07:40.095618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def adjust_transferred_holidays(holidays_df):\n    for idx, row in holidays_df[holidays_df['holiday_type'] == 'Transfer'].iterrows():\n        corresponding_row = holidays_df[(holidays_df['description'] == row['description']) & \n                                        (holidays_df['holiday_type'] != 'Transfer')]\n        if not corresponding_row.empty:\n            holidays_df.at[idx, 'date'] = corresponding_row.iloc[0]['date']\n    \n    holidays_df.drop_duplicates(inplace=True)\n    \n    return holidays_df\n\nholidays_data = adjust_transferred_holidays(holidays_data)\n","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:07:41.849139Z","iopub.execute_input":"2024-05-28T09:07:41.849760Z","iopub.status.idle":"2024-05-28T09:07:41.873843Z","shell.execute_reply.started":"2024-05-28T09:07:41.849727Z","shell.execute_reply":"2024-05-28T09:07:41.872450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Merge Datasets","metadata":{}},{"cell_type":"code","source":"df1 = pd.merge(train_data, transactions_data, on=['date', 'store_nbr'], how='left')\ndf2 = pd.merge(df1, holidays_data, on='date', how='left')\ndf3 = pd.merge(df2, oil_data, on='date', how='left')\nmerged_data = pd.merge(df3, stores_data, on='store_nbr', how='left')\n\nmerged_data.head()\nmerged_data.duplicated().sum()","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:15:10.785747Z","iopub.execute_input":"2024-05-28T09:15:10.786782Z","iopub.status.idle":"2024-05-28T09:15:18.593872Z","shell.execute_reply.started":"2024-05-28T09:15:10.786742Z","shell.execute_reply":"2024-05-28T09:15:18.592766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"merged_copy = merged_data.copy()\nmerged_copy.drop(columns=['id'], inplace=True)","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:07:54.418845Z","iopub.execute_input":"2024-05-28T09:07:54.419200Z","iopub.status.idle":"2024-05-28T09:07:56.439273Z","shell.execute_reply.started":"2024-05-28T09:07:54.419172Z","shell.execute_reply":"2024-05-28T09:07:56.438219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Handling Remaining Missing Values","metadata":{}},{"cell_type":"code","source":"merged_copy.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:07:58.306767Z","iopub.execute_input":"2024-05-28T09:07:58.307144Z","iopub.status.idle":"2024-05-28T09:07:59.511370Z","shell.execute_reply.started":"2024-05-28T09:07:58.307117Z","shell.execute_reply":"2024-05-28T09:07:59.509355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize the trend of oil prices before filling the missing values in the 'oil_price' column\nmerged_copy['dcoilwtico'].plot()","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:08:02.115545Z","iopub.execute_input":"2024-05-28T09:08:02.115923Z","iopub.status.idle":"2024-05-28T09:08:02.862821Z","shell.execute_reply.started":"2024-05-28T09:08:02.115894Z","shell.execute_reply":"2024-05-28T09:08:02.861728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fill the missing values in the transactions column with 0, assuming that there were no transactions on those days\nmerged_copy['transactions'] = merged_copy['transactions'].fillna(0)\n\n# Fill the missing values in Holiday_type, locale, locale_name, description and transferred columns with 'No holiday'\ncol = ['holiday_type', 'locale', 'locale_name', 'description', 'transferred']\nmerged_copy[col] = merged_copy[col].fillna('No holiday')\n\n# Fill the missing values in the 'oil_price' column using backward fill\nmerged_copy['dcoilwtico'] = merged_copy['dcoilwtico'].bfill()\n\n# Check for remaining missing values\nprint(merged_copy.isna().sum())","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:08:10.419624Z","iopub.execute_input":"2024-05-28T09:08:10.420046Z","iopub.status.idle":"2024-05-28T09:08:13.515801Z","shell.execute_reply.started":"2024-05-28T09:08:10.420010Z","shell.execute_reply":"2024-05-28T09:08:13.514656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering \n\nAdd new columns to capture additional information.","metadata":{}},{"cell_type":"code","source":"def map_is_holiday(holiday_type):\n    if holiday_type in ['No holiday', 'Work Day']:\n        return 0\n    else:\n        return 1\n\n# Add the 'holiday_status' column using the 'holiday_type' column\nmerged_copy['is_holiday'] = merged_copy['holiday_type'].apply(map_is_holiday)\nmerged_copy.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:08:23.121328Z","iopub.execute_input":"2024-05-28T09:08:23.121734Z","iopub.status.idle":"2024-05-28T09:08:24.535980Z","shell.execute_reply.started":"2024-05-28T09:08:23.121688Z","shell.execute_reply":"2024-05-28T09:08:24.534789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Let's create new feature from the date column\n\nmerged_copy['date'] = pd.to_datetime(merged_copy['date'])\nmerged_copy['day_of_week'] = merged_copy['date'].dt.dayofweek\nmerged_copy['is_weekend'] = merged_copy['day_of_week'].isin([5, 6]).astype(int)\nmerged_copy['year'] = merged_copy['date'].dt.year\n\n# Set the 'date' column as index\nmerged_copy = merged_copy.set_index('date')\nmerged_copy.head()\n\n#Let's get a preview for our dataset from now\nmerged_copy.sample(8, random_state = 42)","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:08:28.206088Z","iopub.execute_input":"2024-05-28T09:08:28.206840Z","iopub.status.idle":"2024-05-28T09:08:30.662168Z","shell.execute_reply.started":"2024-05-28T09:08:28.206800Z","shell.execute_reply":"2024-05-28T09:08:30.661013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#We have our final dataset.\ntrain_df = merged_copy.copy()\ntrain_df.info()","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:08:35.355965Z","iopub.execute_input":"2024-05-28T09:08:35.356341Z","iopub.status.idle":"2024-05-28T09:08:36.874040Z","shell.execute_reply.started":"2024-05-28T09:08:35.356312Z","shell.execute_reply":"2024-05-28T09:08:36.872799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Exploration and Visualization","metadata":{}},{"cell_type":"code","source":"# Sample the data for efficient visualization\nsample_size = 300000  # Sample size for visualization\nmerged_sample = merged_data.sample(n=sample_size, random_state=42)\n\n# Total Sales Over Time\nplt.figure(figsize=(15, 8))\nmerged_sample['date'] = pd.to_datetime(merged_sample['date'])\nmerged_sample.groupby('date')['sales'].sum().plot()\nplt.title('Total Sales Over Time')\nplt.xlabel('Date')\nplt.ylabel('Total Sales')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:08:39.833110Z","iopub.execute_input":"2024-05-28T09:08:39.833481Z","iopub.status.idle":"2024-05-28T09:08:40.816313Z","shell.execute_reply.started":"2024-05-28T09:08:39.833454Z","shell.execute_reply":"2024-05-28T09:08:40.814972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This  plot shows the total sales over time. The general upward trend suggests that sales have been increasing over the years. Noticeable spikes in sales occur around specific periods, likely corresponding to holidays or promotional events. The sharp dips could be due to stockouts or other disruptions. The outliers, such as the significant drop in April 2016, could be attributed to the 7.8 magnitude earthquake that struck Ecuador on April 16, 2016. This natural disaster would have caused a temporary but severe impact on sales.","metadata":{}},{"cell_type":"code","source":"# Apply log transformation to sales to handle skewness\nmerged_data['log_sales'] = np.log1p(merged_data['sales'])  # log1p is used to handle log(0)\n\n# Plot the distribution of log-transformed sales\nplt.figure(figsize=(12, 8))\nsns.histplot(merged_data['log_sales'], kde=True, bins=50)\nplt.title('Distribution of Log-Transformed Sales')\nplt.xlabel('Log of Sales')\nplt.ylabel('Frequency')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:08:52.106896Z","iopub.execute_input":"2024-05-28T09:08:52.107268Z","iopub.status.idle":"2024-05-28T09:09:06.951883Z","shell.execute_reply.started":"2024-05-28T09:08:52.107241Z","shell.execute_reply":"2024-05-28T09:09:06.950780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The histogram of log-transformed sales data helps to normalize the highly skewed sales distribution. Most sales are concentrated at lower values, with a long tail extending towards higher sales. ","metadata":{}},{"cell_type":"code","source":"# Create bins for 'onpromotion'\nbins = [0, 1, 2, 5, 10, 20, 50, 100, 200, train_data['onpromotion'].max()]\nlabels = ['0', '1', '2-4', '5-9', '10-19', '20-49', '50-99', '100-199', '200+']\n\n# Bin the 'onpromotion' values\nmerged_data['promotion_bins'] = pd.cut(merged_data['onpromotion'], bins=bins, labels=labels, right=False)\n\n# Group by the binned 'onpromotion' and calculate mean sales\npromotion_sales_binned = merged_data.groupby('promotion_bins')['sales'].mean().reset_index()\n\n# Plot the result\nplt.figure(figsize=(12, 8))\nsns.barplot(data=promotion_sales_binned, x='promotion_bins', y='sales', palette='viridis')\nplt.title('Impact of Promotions on Sales')\nplt.xlabel('On Promotion (Binned)')\nplt.ylabel('Average Sales')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:09:17.356656Z","iopub.execute_input":"2024-05-28T09:09:17.357093Z","iopub.status.idle":"2024-05-28T09:09:17.783766Z","shell.execute_reply.started":"2024-05-28T09:09:17.357061Z","shell.execute_reply":"2024-05-28T09:09:17.782542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This bar plot demonstrates that promotions significantly boost sales. Average sales increase as the number of promotional days increases. The data shows that items on promotion for more extended periods have much higher sales, confirming the positive impact of promotions on customer buying behavior.","metadata":{}},{"cell_type":"code","source":"# Total Sales by Store\nstore_sales = merged_sample.groupby('store_nbr')['sales'].sum().reset_index()\n\nplt.figure(figsize=(15, 8))\nsns.barplot(x='store_nbr', y='sales', data=store_sales)\nplt.title('Total Sales by Store')\nplt.xlabel('Store Number')\nplt.ylabel('Total Sales')\nplt.xticks(rotation=90)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:09:22.135425Z","iopub.execute_input":"2024-05-28T09:09:22.135825Z","iopub.status.idle":"2024-05-28T09:09:22.807423Z","shell.execute_reply.started":"2024-05-28T09:09:22.135795Z","shell.execute_reply":"2024-05-28T09:09:22.806360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The bar plot of total sales by store highlights the performance disparities among different stores. Some stores have significantly higher total sales than others. This variation can be attributed to factors such as store location, size, customer demographics, and store management. ","metadata":{}},{"cell_type":"code","source":"# Impact of Holidays on Sales\naverage_sales_holiday = merged_copy.groupby('is_holiday')['sales'].mean().reset_index()\n\nplt.figure(figsize=(10, 6))\nsns.barplot(x='is_holiday', y='sales', data=average_sales_holiday)\nplt.title('Impact of Holidays on Sales')\nplt.xlabel('Is Holiday')\nplt.ylabel('Average Sales')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:15:47.603095Z","iopub.execute_input":"2024-05-28T09:15:47.603599Z","iopub.status.idle":"2024-05-28T09:15:47.924468Z","shell.execute_reply.started":"2024-05-28T09:15:47.603559Z","shell.execute_reply":"2024-05-28T09:15:47.923433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This bar plot indicates that holidays have a positive impact on sales. Sales are higher during holiday periods compared to non-holiday periods. This pattern suggests increased consumer spending during festive times, likely due to holiday shopping and celebrations.","metadata":{}},{"cell_type":"code","source":"# Create bins for oil prices\noil_bins = pd.cut(merged_copy['dcoilwtico'], bins=10)\noil_sales = merged_copy.groupby(oil_bins)['sales'].mean().reset_index()\n\n# Plot the result\nplt.figure(figsize=(12, 8))\nsns.barplot(x=oil_sales['dcoilwtico'].astype(str), y=oil_sales['sales'], palette='viridis')\nplt.title('Impact of Oil Prices on Sales')\nplt.xlabel('Oil Prices (Binned)')\nplt.ylabel('Average Sales')\nplt.xticks(rotation=45)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:15:52.182185Z","iopub.execute_input":"2024-05-28T09:15:52.182558Z","iopub.status.idle":"2024-05-28T09:15:52.568841Z","shell.execute_reply.started":"2024-05-28T09:15:52.182530Z","shell.execute_reply":"2024-05-28T09:15:52.567727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is a small relationship between oil prices and sales, with higher oil prices leading to lower sales.","metadata":{}},{"cell_type":"code","source":"# Draw the correlation Matrix\nplt.figure(figsize=(12, 8))\ntrain_df_num = train_df.select_dtypes(include=['float64', 'int64'])\ncorr_matrix = train_df_num.corr()\nsns.heatmap(corr_matrix, annot=True, fmt=\".2f\", cmap='coolwarm')\nplt.title('Correlation Matrix')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-05-28T09:15:55.847132Z","iopub.execute_input":"2024-05-28T09:15:55.847529Z","iopub.status.idle":"2024-05-28T09:15:57.200871Z","shell.execute_reply.started":"2024-05-28T09:15:55.847497Z","shell.execute_reply":"2024-05-28T09:15:57.199781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Interpretation of the Correlation Matrix**\n\nThe correlation matrix revealed several insights. There was a moderate positive correlation between sales and promotions, indicating that promotions generally boost sales. A weaker positive correlation was found between sales and transactions, suggesting that higher transaction volumes are associated with increased sales. The slight negative correlation between sales and oil prices suggested that higher oil prices might slightly reduce sales. Contrary to expectations, the correlation between holidays and sales was minimal, which might be attributed to data quality issues or specific characteristics of the dataset.","metadata":{}},{"cell_type":"markdown","source":"### Data Exploration and Preprocessing - Summary\n\nIn this section, we undertook a comprehensive process to explore and preprocess the data in preparation for our analysis and modeling tasks. \n\nWe began by loading the various datasets (`train_data`, `test_data`, `holidays_data`, `oil_data`, `stores_data`, and `transactions_data`) using the `pd.read_csv()` function from the Pandas library. Initial exploration of these datasets was conducted to understand their structure, including the shape, columns, first few rows, summary statistics, and missing values.\n\nFor the holidays dataset, we renamed the `type` column to `holiday_type` to enhance clarity. We then addressed the 'transferred' holidays, adjusting their dates to reflect the actual observed holiday dates. This step ensured that holidays transferred to another date were correctly aligned in our dataset.\n\nIn the oil dataset, we observed some missing values in the `dcoilwtico` column. We used forward and backward filling methods to handle these gaps, ensuring no missing data points that could disrupt the analysis.\n\nNext, we merged all datasets into a single consolidated dataframe, `merged_data`, using the `pd.merge()` function. This merging step was crucial to combine relevant information from different sources into one unified dataset, facilitating more comprehensive analysis.\n\nHandling missing values was a critical part of the preprocessing. We filled missing transaction values with zeros, assuming no transactions occurred on those days. Missing values in the `holiday_type`, `locale`, `locale_name`, `description`, and `transferred` columns were filled with 'No holiday', signifying non-holiday periods. The `dcoilwtico` column's missing values were filled using backward fill.\n\nTo enrich our dataset, we created new features derived from the date column. These included `day_of_week` and `is_weekend`, providing insights into the weekly sales patterns. Additionally, we introduced an `is_holiday` column to mark holiday periods distinctly.\n\nFor better time series analysis, we set the `date` column as the index of our dataframe. This restructuring made it easier to perform date-based operations and visualizations.\n\nFinally, we generated a correlation matrix to investigate the relationships between various numerical features in the dataset. The matrix highlighted a moderate positive correlation between sales and promotions, a weaker correlation between sales and transactions, and an unexpectedly low correlation between holidays and sales.\n\n\n### Next Steps: Feature Engineering\n\nMoving forward, we will focus on feature engineering. This will involve creating new features and modifying existing ones to enhance the performance of our machine learning models. We plan to introduce additional date features, generate lag and rolling window features, create interaction features, encode categorical variables, and perform feature selection to identify the most significant features for our models.\n\nBy enriching the dataset through these steps, we aim to capture more complex patterns and relationships, ultimately improving the accuracy and robustness of our predictive models.\n","metadata":{}},{"cell_type":"code","source":"train_df.sample(10, random_state = 42)\ntrain_df.columns","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.268485Z","iopub.status.idle":"2024-05-27T23:29:44.269080Z","shell.execute_reply.started":"2024-05-27T23:29:44.268895Z","shell.execute_reply":"2024-05-27T23:29:44.268911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drop some columns and display the dataset\ncolumns_to_drop = ['transactions', 'holiday_type', 'locale', 'locale_name', 'description', 'transferred', 'dcoilwtico', 'city',\n                   'state', 'type', 'cluster', 'is_weekend', 'day_of_week', 'is_holiday']\ntrain_df = train_df.drop(columns_to_drop, axis=1)\n\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.270258Z","iopub.status.idle":"2024-05-27T23:29:44.270664Z","shell.execute_reply.started":"2024-05-27T23:29:44.270478Z","shell.execute_reply":"2024-05-27T23:29:44.270494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Data Splitting**\nlets split the dataset to training and testing sets using time-based splits based on the years.","metadata":{}},{"cell_type":"code","source":"train_df['year'].unique()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.271797Z","iopub.status.idle":"2024-05-27T23:29:44.272176Z","shell.execute_reply.started":"2024-05-27T23:29:44.271988Z","shell.execute_reply":"2024-05-27T23:29:44.272004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Establish the years for the training set and validation set\ntrain_years = [2013, 2014, 2015, 2016]\nval_year = [2017]\n\n#Obtain the training set and validation set\ntrain_set = train_df.loc[(train_df['year'].isin(train_years) & train_df['year'].isin(train_years))]\nval_set = train_df.loc[(train_df['year'].isin(val_year) & train_df['year'].isin(val_year))]\n\n#Print the shape of the training set and validation set\ntrain_set.shape, val_set.shape","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.273897Z","iopub.status.idle":"2024-05-27T23:29:44.274453Z","shell.execute_reply.started":"2024-05-27T23:29:44.274159Z","shell.execute_reply":"2024-05-27T23:29:44.274182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_set.info()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.275979Z","iopub.status.idle":"2024-05-27T23:29:44.276501Z","shell.execute_reply.started":"2024-05-27T23:29:44.276215Z","shell.execute_reply":"2024-05-27T23:29:44.276237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extract the original index column for later use\ntrain_index = train_set.index\nval_index = val_set.index\n\n# Obtain the categorical columns to encode and numerical columns to scale (I dropped the 'Year' column here)\ncat_columns_to_encode = ['family']\nnum_columns_to_scale = ['store_nbr', 'onpromotion', 'sales']\n\n# Create seperate DataFrames for categorical columns and numerical columns of training set\ntrain_set_cat_df = train_set[cat_columns_to_encode]\ntrain_set_num_df = train_set[num_columns_to_scale]\n\n# Create seperate DataFrames for categorical columns and numerical columns of validation set\nval_set_cat_df = val_set[cat_columns_to_encode]\nval_set_num_df = val_set[num_columns_to_scale]","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.281329Z","iopub.status.idle":"2024-05-27T23:29:44.281749Z","shell.execute_reply.started":"2024-05-27T23:29:44.281554Z","shell.execute_reply":"2024-05-27T23:29:44.281572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Feature Encoding**","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import OneHotEncoder\n# Create an encoder object using OneHotEncoder.\n# Set sparse=False for dense output and drop='first' to avoid multicollinearity\n\nencoder = OneHotEncoder(sparse_output=False, drop='first')","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.282827Z","iopub.status.idle":"2024-05-27T23:29:44.283215Z","shell.execute_reply.started":"2024-05-27T23:29:44.283029Z","shell.execute_reply":"2024-05-27T23:29:44.283045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use OneHotEncoder to encode the categorical columns of the training set\nencoder.fit(train_set_cat_df)\ntrain_set_cat_encoded = encoder.transform(train_set_cat_df).tolist()\ntrain_set_cat_encoded_df = pd.DataFrame(train_set_cat_encoded, columns=encoder.get_feature_names_out(), index=train_index)\n\n# View the encoded columns of the training set\ntrain_set_cat_encoded_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.285180Z","iopub.status.idle":"2024-05-27T23:29:44.285788Z","shell.execute_reply.started":"2024-05-27T23:29:44.285421Z","shell.execute_reply":"2024-05-27T23:29:44.285460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use OneHotEncoder to encode the categorical columns of the validation set\nencoder.fit(val_set_cat_df)\nval_set_cat_encoded = encoder.transform(val_set_cat_df).tolist()\nval_set_cat_encoded_df = pd.DataFrame(val_set_cat_encoded, columns=encoder.get_feature_names_out(), index=val_index)\n\n# View the encoded columns of the validation set\nval_set_cat_encoded_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.286805Z","iopub.status.idle":"2024-05-27T23:29:44.287352Z","shell.execute_reply.started":"2024-05-27T23:29:44.287065Z","shell.execute_reply":"2024-05-27T23:29:44.287090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Feature Scaling**\nMinMaxScaler will be used for feature scaling as it is suitable for time series data because it preserves the relative differences between data points and maintains the temporal information.","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler\n# Create a scaler object using MinMaxScaler\n\nscaler = MinMaxScaler()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.288310Z","iopub.status.idle":"2024-05-27T23:29:44.288855Z","shell.execute_reply.started":"2024-05-27T23:29:44.288587Z","shell.execute_reply":"2024-05-27T23:29:44.288619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use MinMaxScaler to scale the numerical columns of the training set\nscaler.fit(train_set_num_df)\ntrain_set_num_scaled = scaler.transform(train_set_num_df).tolist()\ntrain_set_num_scaled_df = pd.DataFrame(train_set_num_scaled, columns=scaler.get_feature_names_out(), index=train_index)\n\n# View the scaled columns of the training set\ntrain_set_num_scaled_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.289774Z","iopub.status.idle":"2024-05-27T23:29:44.290260Z","shell.execute_reply.started":"2024-05-27T23:29:44.290020Z","shell.execute_reply":"2024-05-27T23:29:44.290041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use MinMaxScaler to scale the numerical columns of the validation set\nscaler.fit(val_set_num_df)\nval_set_num_scaled = scaler.transform(val_set_num_df).tolist()\nval_set_num_scaled_df = pd.DataFrame(val_set_num_scaled, columns=scaler.get_feature_names_out(), index=val_index)\n\n# View the scaled columns of the validation set\nval_set_num_scaled_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.291503Z","iopub.status.idle":"2024-05-27T23:29:44.292096Z","shell.execute_reply.started":"2024-05-27T23:29:44.291755Z","shell.execute_reply":"2024-05-27T23:29:44.291778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Combine the encoded and scaled DataFrames to get the final training and validation sets\ntrain_final = pd.concat([train_set_cat_encoded_df, train_set_num_scaled_df], axis=1)\nval_final = pd.concat([val_set_cat_encoded_df, val_set_num_scaled_df], axis=1)\n\n# Print the shape of the final training and validation sets\ntrain_final.shape, val_final.shape","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.296438Z","iopub.status.idle":"2024-05-27T23:29:44.296805Z","shell.execute_reply.started":"2024-05-27T23:29:44.296630Z","shell.execute_reply":"2024-05-27T23:29:44.296645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the first five rows of the final training set\n\ntrain_final.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.298237Z","iopub.status.idle":"2024-05-27T23:29:44.298655Z","shell.execute_reply.started":"2024-05-27T23:29:44.298453Z","shell.execute_reply":"2024-05-27T23:29:44.298474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the first five rows of the final validation set\n\nval_final.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.300500Z","iopub.status.idle":"2024-05-27T23:29:44.300891Z","shell.execute_reply.started":"2024-05-27T23:29:44.300701Z","shell.execute_reply":"2024-05-27T23:29:44.300718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Separate the features (X) and the target variable (y) for training and validation sets\nX_train = train_final.drop('sales', axis=1)\ny_train = train_final['sales']\nX_val = val_final.drop('sales', axis=1)\ny_val = val_final['sales']\n\n# Replace the elements of y_val with their absolute value in order to eliminate the presence of negative values in the y_val array\ny_val = np.abs(y_val)\ny_val = y_val.reset_index(drop=True)\ny_val = y_val[y_val >= 0]\n\n# Verify the lengths of the features(X) and the target variable (y) of the final training and validation sets\nprint(\"Length of X_train:\", len(X_train))\nprint(\"Length of y_train:\", len(y_train))\nprint(\"Length of X_test:\", len(X_val))\nprint(\"Length of y_test:\", len(y_val))","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.302286Z","iopub.status.idle":"2024-05-27T23:29:44.302690Z","shell.execute_reply.started":"2024-05-27T23:29:44.302500Z","shell.execute_reply":"2024-05-27T23:29:44.302516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modelling\nWe are now ready to use different models\nthe following models will be trained and evaluated :\n1. Linear Regression\n\nThe following evaluation metrics will be used to evaluate the performance of the models.\n* Mean Absolute Error (MAE)\n* Mean Squared Error (MSE)\n* Root Mean Squared Error (RMSE)","metadata":{}},{"cell_type":"code","source":"# Create an empty DataFrame to store the results of the evaluation metrics of each model\nfrom sklearn.metrics import mean_squared_error, mean_absolute_error, mean_squared_log_error\nresults = pd.DataFrame(columns=['Model', 'MAE', 'MSE', 'RMSE', 'RMSLE', 'MSLE'])\nresults","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.306478Z","iopub.status.idle":"2024-05-27T23:29:44.306869Z","shell.execute_reply.started":"2024-05-27T23:29:44.306678Z","shell.execute_reply":"2024-05-27T23:29:44.306694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Model 1 Linear Regression**","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nlr_model=LinearRegression()\nlr_model.fit(X_train, y_train)\nlr_pred=lr_model.predict(X_val)\nlr_mae=mean_absolute_error(y_val, lr_pred)\nlr_mse=mean_squared_error(y_val, lr_pred)\nlr_rmse=np.sqrt(lr_mse)\ny_eval_abs = abs(y_val)\nlr_predictions_abs = abs(lr_pred)\n\n\ndef linear_regression_msle_with_non_negative_targets(y_val, lr_pred):\n    # Add a constant value to all target values to ensure they are non-negative\n    y_val_non_negative = y_val + np.abs(np.min(y_val)) + 1\n    \n    # Add the same constant value to the predicted values\n    linear_regression_pred_non_negative = lr_pred + np.abs(np.min(y_val)) + 1\n    \n    # Calculate the MSLE using the transformed targets\n    linear_regression_msle = mean_squared_log_error(y_val_non_negative, linear_regression_pred_non_negative)\n    return linear_regression_msle\n\n# Calculate msle using the custom function, then calculate rmsle\nlinear_regression_msle = linear_regression_msle_with_non_negative_targets(y_val, lr_pred)\nlinear_regression_rmsle = np.sqrt(linear_regression_msle)\n\n# Create a dictionary for the evaluation metrics of Linear Regression model\nlinear_regression_evaluation_metrics = pd.DataFrame({\n    'Model': ['Linear Regression'],\n    'MAE': [lr_mae],\n    'MSE': [lr_mse],\n    'RMSE': [lr_rmse],\n    'MSLE': [linear_regression_msle],\n    'RMSLE': [linear_regression_rmsle]\n})\n\n# Initialize the results DataFrame if it does not exist\nif 'results' not in globals():\n    results = pd.DataFrame()\n\n# Add the linear regression evaluation metrics to the results DataFrame\nresults = pd.concat([results, linear_regression_evaluation_metrics], ignore_index=True)\n\n# Print the results DataFrame\nprint(results)\n","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.308118Z","iopub.status.idle":"2024-05-27T23:29:44.308523Z","shell.execute_reply.started":"2024-05-27T23:29:44.308307Z","shell.execute_reply":"2024-05-27T23:29:44.308323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12, 6))\nplt.scatter(range(len(y_train)), y_train, color='blue', label='Training data')\nplt.scatter(range(len(y_val)), y_val, color='green', label='Validation data')\nplt.plot(range(len(y_val)), lr_pred, color='red', label='Predictions')\nplt.xlabel('Index')\nplt.ylabel('Target Value')\nplt.title('Training, Validation, and Prediction Data')\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.309561Z","iopub.status.idle":"2024-05-27T23:29:44.309938Z","shell.execute_reply.started":"2024-05-27T23:29:44.309748Z","shell.execute_reply":"2024-05-27T23:29:44.309764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# Plot the predicted vs actual values\nplt.scatter(y_val, lr_pred)\nplt.xlabel('Actual Values')\nplt.ylabel('Predicted Values')\nplt.title('Predicted vs. Actual Values')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.311637Z","iopub.status.idle":"2024-05-27T23:29:44.312021Z","shell.execute_reply.started":"2024-05-27T23:29:44.311831Z","shell.execute_reply":"2024-05-27T23:29:44.311848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate residuals\nresiduals = y_val - lr_pred\n\n# Plot residuals\nplt.scatter(y_val, residuals)\nplt.xlabel('Actual Values')\nplt.ylabel('Residuals')\nplt.title('Residual Plot')\nplt.axhline(0, color='red', linestyle='--')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.313246Z","iopub.status.idle":"2024-05-27T23:29:44.313648Z","shell.execute_reply.started":"2024-05-27T23:29:44.313453Z","shell.execute_reply":"2024-05-27T23:29:44.313475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot distribution of actual and predicted values\nplt.hist(y_val, bins=30, alpha=0.5, label='Actual Values')\nplt.hist(lr_pred, bins=30, alpha=0.5, label='Predicted Values')\nplt.xlabel('Value')\nplt.ylabel('Frequency')\nplt.legend()\nplt.title('Distribution of Actual and Predicted Values')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.314751Z","iopub.status.idle":"2024-05-27T23:29:44.315158Z","shell.execute_reply.started":"2024-05-27T23:29:44.314968Z","shell.execute_reply":"2024-05-27T23:29:44.314985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Example check for data leakage\n# Ensure no overlap between training and validation sets\nassert len(set(X_train.index).intersection(set(X_val.index))) == 0, \"Data leakage detected: overlapping indices between training and validation sets\"\n","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.316240Z","iopub.status.idle":"2024-05-27T23:29:44.316644Z","shell.execute_reply.started":"2024-05-27T23:29:44.316449Z","shell.execute_reply":"2024-05-27T23:29:44.316470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate range and mean of actual target values\ny_range = np.max(y_val) - np.min(y_val)\ny_mean = np.mean(y_val)\n\nprint(f\"Range of actual target values: {y_range}\")\nprint(f\"Mean of actual target values: {y_mean}\")\n\n# Compare with metrics\nprint(f\"MAE as percentage of range: {lr_mae / y_range * 100:.2f}%\")\nprint(f\"RMSE as percentage of range: {lr_rmse / y_range * 100:.2f}%\")\nprint(f\"MAE as percentage of mean: {lr_mae / y_mean * 100:.2f}%\")\nprint(f\"RMSE as percentage of mean: {lr_rmse / y_mean * 100:.2f}%\")\n","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.317737Z","iopub.status.idle":"2024-05-27T23:29:44.318141Z","shell.execute_reply.started":"2024-05-27T23:29:44.317950Z","shell.execute_reply":"2024-05-27T23:29:44.317966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The mean target value is very small, around 0.0125. target values are concentrated near zero.\nthe target values in the validation set vary between approximately 0 and 1.","metadata":{}},{"cell_type":"code","source":"# Calculate normalized MAE and RMSE\nnormalized_mae = lr_mae / y_mean\nnormalized_rmse = lr_rmse / y_mean\n\nprint(f\"Normalized MAE: {normalized_mae:.4f}\")\nprint(f\"Normalized RMSE: {normalized_rmse:.4f}\")\n","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.319149Z","iopub.status.idle":"2024-05-27T23:29:44.319566Z","shell.execute_reply.started":"2024-05-27T23:29:44.319343Z","shell.execute_reply":"2024-05-27T23:29:44.319359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add a constant to avoid log(0) or log of negative values\ny_train_transformed = np.log1p(y_train - np.min(y_train) + 1)\ny_val_transformed = np.log1p(y_val - np.min(y_val) + 1)\n\n# Fit the Linear Regression model on transformed target values\nlr_model_transformed = LinearRegression()\nlr_model_transformed.fit(X_train, y_train_transformed)\n\n# Predict on the validation data\nlr_pred_transformed = lr_model_transformed.predict(X_val)\n\n# Inverse the transformation for predictions\nlr_pred_original_scale = np.expm1(lr_pred_transformed) + np.min(y_val) - 1\n\n# Calculate evaluation metrics for transformed model\nlr_mae_transformed = mean_absolute_error(y_val, lr_pred_original_scale)\nlr_mse_transformed = mean_squared_error(y_val, lr_pred_original_scale)\nlr_rmse_transformed = np.sqrt(lr_mse_transformed)\n\n# Print the evaluation metrics for transformed model\nprint(f\"Transformed MAE: {lr_mae_transformed:.4f}\")\nprint(f\"Transformed RMSE: {lr_rmse_transformed:.4f}\")\n\n# Plot the predicted vs actual values for transformed model\nplt.figure(figsize=(12, 6))\nplt.scatter(y_val, lr_pred_original_scale, alpha=0.5)\nplt.xlabel('Actual Values')\nplt.ylabel('Predicted Values')\nplt.title('Predicted vs. Actual Values (Transformed Model)')\nplt.plot([y_val.min(), y_val.max()], [y_val.min(), y_val.max()], 'r--')  # Line of perfect prediction\nplt.show()\n\n# Plot residuals for transformed model\nresiduals_transformed = y_val - lr_pred_original_scale\nplt.figure(figsize=(12, 6))\nplt.scatter(y_val, residuals_transformed, alpha=0.5)\nplt.xlabel('Actual Values')\nplt.ylabel('Residuals')\nplt.title('Residual Plot (Transformed Model)')\nplt.axhline(0, color='red', linestyle='--')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.320791Z","iopub.status.idle":"2024-05-27T23:29:44.321190Z","shell.execute_reply.started":"2024-05-27T23:29:44.320985Z","shell.execute_reply":"2024-05-27T23:29:44.321001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Random Forest","metadata":{}},{"cell_type":"code","source":"# Separate the features (X) and the target variable (y) for training and validation sets\nX_train = train_final.drop('sales', axis=1)\ny_train = train_final['sales']\nX_val = val_final.drop('sales', axis=1)\ny_val = val_final['sales']\n\n# Replace the elements of y_val with their absolute value in order to eliminate the presence of negative values in the y_val array\ny_val = np.abs(y_val)\ny_val = y_val.reset_index(drop=True)\ny_val = y_val[y_val >= 0]\n\n# Verify the lengths of the features(X) and the target variable (y) of the final training and validation sets\nprint(\"Length of X_train:\", len(X_train))\nprint(\"Length of y_train:\", len(y_train))\nprint(\"Length of X_val:\", len(X_val))\nprint(\"Length of y_val:\", len(y_val))\n\nfrom sklearn.metrics import mean_squared_error, mean_absolute_error, mean_squared_log_error\nresults = pd.DataFrame(columns=['Model', 'MAE', 'MSE', 'RMSE', 'RMSLE', 'MSLE'])\nresults\n\nfrom sklearn.ensemble import RandomForestRegressor\n\n# Initialize the Random Forest Regressor\nrf_model = RandomForestRegressor(n_estimators=50, max_depth=10, random_state=42, n_jobs=-1)\n\n# Fit the model on the training data\nrf_model.fit(X_train, y_train)\n\n# Predict on the validation data\nrf_pred = rf_model.predict(X_val)\n\n# Calculate evaluation metrics\nrf_mae = mean_absolute_error(y_val, rf_pred)\nrf_mse = mean_squared_error(y_val, rf_pred)\nrf_rmse = np.sqrt(rf_mse)\n\n# Custom function to calculate MSLE and RMSLE for non-negative targets\ndef msle_with_non_negative_targets(y_true, y_pred):\n    y_true_non_negative = y_true + np.abs(np.min(y_true)) + 1\n    y_pred_non_negative = y_pred + np.abs(np.min(y_true)) + 1\n    msle = mean_squared_log_error(y_true_non_negative, y_pred_non_negative)\n    return msle\n\n# Calculate MSLE and RMSLE\nrf_msle = msle_with_non_negative_targets(y_val, rf_pred)\nrf_rmsle = np.sqrt(rf_msle)\n\n# Create a dictionary for the evaluation metrics of Random Forest model\nrf_evaluation_metrics = pd.DataFrame({\n    'Model': ['Random Forest'],\n    'MAE': [rf_mae],\n    'MSE': [rf_mse],\n    'RMSE': [rf_rmse],\n    'MSLE': [rf_msle],\n    'RMSLE': [rf_rmsle]\n})\n\n# Add the Random Forest evaluation metrics to the results DataFrame\nresults = pd.concat([results, rf_evaluation_metrics], ignore_index=True)\n\n# Print the results DataFrame\nprint(results)\n","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.322251Z","iopub.status.idle":"2024-05-27T23:29:44.322657Z","shell.execute_reply.started":"2024-05-27T23:29:44.322466Z","shell.execute_reply":"2024-05-27T23:29:44.322483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model fine tuning","metadata":{}},{"cell_type":"markdown","source":"**grid search**","metadata":{}},{"cell_type":"code","source":"\"\"\"\nfrom sklearn.model_selection import GridSearchCV\n\nparam_grid = [\n    {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},\n    {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, 4]},]\n\nforest_reg = RandomForestRegressor(random_state=42)\n \ngrid_search = GridSearchCV(forest_reg, param_grid, cv=5, scoring='neg_mean_squared_error', return_train_score=True)\ngrid_search.fit(X_train, y_train)\n\"\"\"","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.323743Z","iopub.status.idle":"2024-05-27T23:29:44.324124Z","shell.execute_reply.started":"2024-05-27T23:29:44.323936Z","shell.execute_reply":"2024-05-27T23:29:44.323952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#grid_search.best_params_","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.325534Z","iopub.status.idle":"2024-05-27T23:29:44.325924Z","shell.execute_reply.started":"2024-05-27T23:29:44.325728Z","shell.execute_reply":"2024-05-27T23:29:44.325744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#grid_search.best_estimator_","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.326892Z","iopub.status.idle":"2024-05-27T23:29:44.327281Z","shell.execute_reply.started":"2024-05-27T23:29:44.327093Z","shell.execute_reply":"2024-05-27T23:29:44.327109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction on test dataset ","metadata":{}},{"cell_type":"code","source":"test_data.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.328279Z","iopub.status.idle":"2024-05-27T23:29:44.328677Z","shell.execute_reply.started":"2024-05-27T23:29:44.328487Z","shell.execute_reply":"2024-05-27T23:29:44.328503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Rename test to test_df to preserve the originality of test for later use\ntest_df = test_data\n\n# Set the 'date' column of test_df as index\ntest_df = test_df.set_index('date')\n\n# Extract the index column  of test_df for later use\ntest_index = test_df.index\n\n# View the resulting test dataset\ntest_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.330022Z","iopub.status.idle":"2024-05-27T23:29:44.330441Z","shell.execute_reply.started":"2024-05-27T23:29:44.330215Z","shell.execute_reply":"2024-05-27T23:29:44.330231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Specify new numerical columns to scale as 'sales' column is absent in test dataset\ntest_num_columns_to_scale = ['store_nbr', 'onpromotion']\n\n# Create seperate DataFrames for categorical columns and numerical columns of test_df\ntest_linear_reg_df = test_df[cat_columns_to_encode]\ntest_num_df = test_df[test_num_columns_to_scale]","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.331685Z","iopub.status.idle":"2024-05-27T23:29:44.332047Z","shell.execute_reply.started":"2024-05-27T23:29:44.331870Z","shell.execute_reply":"2024-05-27T23:29:44.331885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use OneHotEncoder to encode the categorical columns of test dataset\nencoder.fit(test_linear_reg_df)\ntest_linear_reg_encoded = encoder.transform(test_linear_reg_df).tolist()\ntest_linear_reg_encoded_df = pd.DataFrame(test_linear_reg_encoded, columns=encoder.get_feature_names_out(), index=test_index)\n\n# View the encoded columns of test dataset\ntest_linear_reg_encoded_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.333156Z","iopub.status.idle":"2024-05-27T23:29:44.333517Z","shell.execute_reply.started":"2024-05-27T23:29:44.333324Z","shell.execute_reply":"2024-05-27T23:29:44.333339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use MinMaxScaler to scale the numerical columns of test dataset\nscaler.fit(test_num_df)\ntest_num_scaled = scaler.transform(test_num_df).tolist()\ntest_num_scaled_df = pd.DataFrame(test_num_scaled, columns=scaler.get_feature_names_out(), index=test_index)\n\n# View the scaled columns of test dataset\ntest_num_scaled_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.335579Z","iopub.status.idle":"2024-05-27T23:29:44.335949Z","shell.execute_reply.started":"2024-05-27T23:29:44.335762Z","shell.execute_reply":"2024-05-27T23:29:44.335777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Combine the encoded and scaled DataFrames to get the final test dataset\ntest_final = pd.concat([test_linear_reg_encoded_df, test_num_scaled_df], axis=1)\n\n# Print the shape of the final test dataset\ntest_final.shape","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.337637Z","iopub.status.idle":"2024-05-27T23:29:44.337996Z","shell.execute_reply.started":"2024-05-27T23:29:44.337813Z","shell.execute_reply":"2024-05-27T23:29:44.337827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_final.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.338975Z","iopub.status.idle":"2024-05-27T23:29:44.339312Z","shell.execute_reply.started":"2024-05-27T23:29:44.339140Z","shell.execute_reply":"2024-05-27T23:29:44.339155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# utiliser le meilleur model pour la cell suivante","metadata":{}},{"cell_type":"code","source":"# Make predictions on the final test dataset using the best model \ntest_pred = lr_model.predict(test_final)\ntest_data['sales'] = np.abs(test_pred)\ny_val_final = test_data['sales']\n# Print the test predictions\nprint('Predictions:')\ntest_pred\n\n# Calculate residuals\nresiduals = y_val_final - test_pred\n\n# Plot residuals\nplt.scatter(y_val_final, residuals)\nplt.xlabel('Actual Values')\nplt.ylabel('Residuals')\nplt.title('Residual Plot')\nplt.axhline(0, color='red', linestyle='--')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.340718Z","iopub.status.idle":"2024-05-27T23:29:44.341092Z","shell.execute_reply.started":"2024-05-27T23:29:44.340897Z","shell.execute_reply":"2024-05-27T23:29:44.340912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 6))\nplt.scatter(y_val_final, test_pred, alpha=0.5)\nplt.plot([y_val_final.min(), y_val_final.max()], [y_val_final.min(), y_val_final.max()], 'r--', lw=2)\nplt.xlabel('Actual Values')\nplt.ylabel('Predicted Values')\nplt.title('Actual vs Predicted Values')\nplt.grid(True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-05-27T23:29:44.342597Z","iopub.status.idle":"2024-05-27T23:29:44.342960Z","shell.execute_reply.started":"2024-05-27T23:29:44.342774Z","shell.execute_reply":"2024-05-27T23:29:44.342789Z"},"trusted":true},"execution_count":null,"outputs":[]}]}