![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13474157%2F12985b7771d35940abaa8a6db8e72484%2Fexcl_cfips.png?generation=1687123492757230&alt=media\")
Below is a brief description + thoughts on my forecasting solution.
\nPrivate LB Place: 122/3,546
\nPrivate LB Score: 3.9884
\nCode: https://www.kaggle.com/code/abandura/mbd-forecasting-w-mars-lms-r
Thanks to everyone who shared their insights and code!
\nMany counties exhibit level shifts in their historical microbusiness density, which reduces overall accuracy if simply included in the modeling process. I separated these from the rest of the counties by identifying any counties with a month-to-month change greater than 25%, and simply used last value predictions for them. Many other contestants took a similar approach. Looking at the size distribution of these counties (visualized in the histogram below) we can see that it excludes most of the smallest counties as well as a group of larger volume counties.
\n
Small counties are unpredictable, as a single domain registration can dramatically change the density. However, looking at the larger counties with high variance it's clear that there is still a strong signal in many of them that could be leveraged for better predictions than a simple last value. For example, county ID 24013 has only one level shift up early on in the data set, and data before that shift could be removed and then interpolated so that it could be included in the forecast model training/prediction process. County IDs 10003 and 32510 have two level shifts that creates a \"hump\", which could be corrected in a similar manner using interpolation. I had planned to implement an algorithm to correct shifts for these larger counties, but unfortunately ran out of time.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13474157%2F6b3c0541a03e4f405b1aa745b943d5b6%2Flarge_excl_cfips.png?generation=1687123510916081&alt=media\")
Like many others, I also made the \"percent change\" adjustment to microbusiness density before training so that loss metrics like RMSE would weight small counties higher in the same way that SMAPE does. I used that adjusted target for all training other than the linear models, because they can be flexibly trained to directly minimize SMAPE (as described below).
\nOf the traditional autoregressive time-series forecasting methods (ARIMA, Holt-Winters, Exponential Smoothing, etc.) that I tried, Exponential Smoothing (ETS) was the best. I ended up using ETS forecasts as features for the linear modeling by rolling through the data and generating ets forecasts for 1-6 month time horizons.
\nOther features I toyed with included moving-average terms, difference terms, month and year, and the average of all previous MB density values.
\nFor CV I used the last 6 months of data, as shown below, and trained separate models for each forecast horizon. I wanted to use as much data as possible for training, since we only had a few years of historical data spanning the pandemic.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13474157%2F6ae7279f1d78b31a125355f0c2d34cd8%2Fcv_strategy.png?generation=1687123527511586&alt=media\")
I tried other ML approaches like XGBoost, but found the most success with the simpler linear models.
\nAnother nice feature of linear models is you can directly fit them to minimize SMAPE by solving the optimization problem with the following objective:
\n\nfn par df r_cols y norm_col l2 l1 \n preds par df.N\n l2p \n l1p \n ind par \n i_col r_colsind\n preds preds parinddf geti_colmeangeti_colsdgeti_col\n l2p l2p l2parind\n l1p l1p l1parind\n \n err smapeadfgety fcpreds l1p l2p\n err\n\nI tested L1 and L2 regularization, but found the lowest CV accuracy with no regularization. Looking at the fitted weight magnitudes, it seems that MA terms are most important in the forecasts. This was my best-scoring final submission.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13474157%2F8b4a56ee77fff6fe87562ab6916726f5%2Flm_weights.png?generation=1687130521279924&alt=media\")
https://en.wikipedia.org/wiki/Multivariate_adaptive_regression_spline
\nBecause I had success with the LMs, I though that MARS would be a natural improvement since it is an algorithm for fitting piecewise linear models to automatically deal with non-linearities and incorporate variable interactions.
\nI did some hyperparameter tuning, but had the most success with the default parameters, and based on the CV accuracy I hoped it would outperform the LMs. However that wasn't the case with the private LB (I'm still not sure why, perhaps MARS was more sensitive to erroneous Dec data…)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13474157%2Fb41c10f7948ea74940de90bd4e8f357a%2Fsmape_comp.png?generation=1687127830752749&alt=media\")
Below is a brief description + thoughts on my forecasting solution.
\nPrivate LB Place: 122/3,546
\nPrivate LB Score: 3.9884
\nCode: https://www.kaggle.com/code/abandura/mbd-forecasting-w-mars-lms-r
Thanks to everyone who shared their insights and code!
\nMany counties exhibit level shifts in their historical microbusiness density, which reduces overall accuracy if simply included in the modeling process. I separated these from the rest of the counties by identifying any counties with a month-to-month change greater than 25%, and simply used last value predictions for them. Many other contestants took a similar approach. Looking at the size distribution of these counties (visualized in the histogram below) we can see that it excludes most of the smallest counties as well as a group of larger volume counties.
\n
Small counties are unpredictable, as a single domain registration can dramatically change the density. However, looking at the larger counties with high variance it's clear that there is still a strong signal in many of them that could be leveraged for better predictions than a simple last value. For example, county ID 24013 has only one level shift up early on in the data set, and data before that shift could be removed and then interpolated so that it could be included in the forecast model training/prediction process. County IDs 10003 and 32510 have two level shifts that creates a \"hump\", which could be corrected in a similar manner using interpolation. I had planned to implement an algorithm to correct shifts for these larger counties, but unfortunately ran out of time.
\n
Like many others, I also made the \"percent change\" adjustment to microbusiness density before training so that loss metrics like RMSE would weight small counties higher in the same way that SMAPE does. I used that adjusted target for all training other than the linear models, because they can be flexibly trained to directly minimize SMAPE (as described below).
\nOf the traditional autoregressive time-series forecasting methods (ARIMA, Holt-Winters, Exponential Smoothing, etc.) that I tried, Exponential Smoothing (ETS) was the best. I ended up using ETS forecasts as features for the linear modeling by rolling through the data and generating ets forecasts for 1-6 month time horizons.
\nOther features I toyed with included moving-average terms, difference terms, month and year, and the average of all previous MB density values.
\nFor CV I used the last 6 months of data, as shown below, and trained separate models for each forecast horizon. I wanted to use as much data as possible for training, since we only had a few years of historical data spanning the pandemic.
\n
I tried other ML approaches like XGBoost, but found the most success with the simpler linear models.
\nAnother nice feature of linear models is you can directly fit them to minimize SMAPE by solving the optimization problem with the following objective:
\n\nfn par df r_cols y norm_col l2 l1 \n preds par df.N\n l2p \n l1p \n ind par \n i_col r_colsind\n preds preds parinddf geti_colmeangeti_colsdgeti_col\n l2p l2p l2parind\n l1p l1p l1parind\n \n err smapeadfgety fcpreds l1p l2p\n err\n\nI tested L1 and L2 regularization, but found the lowest CV accuracy with no regularization. Looking at the fitted weight magnitudes, it seems that MA terms are most important in the forecasts. This was my best-scoring final submission.
\n
https://en.wikipedia.org/wiki/Multivariate_adaptive_regression_spline
\nBecause I had success with the LMs, I though that MARS would be a natural improvement since it is an algorithm for fitting piecewise linear models to automatically deal with non-linearities and incorporate variable interactions.
\nI did some hyperparameter tuning, but had the most success with the default parameters, and based on the CV accuracy I hoped it would outperform the LMs. However that wasn't the case with the private LB (I'm still not sure why, perhaps MARS was more sensitive to erroneous Dec data…)
\n