![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5552115%2Fc85777fd4f981dee2765e2b78b7f1236%2Fcfips48155.png?generation=1678978908088571&alt=media\")
Interesting problem, and it was also interesting to see the solutions came up with. I'm pretty new to these competitions. The biggest uncertainty to me is how much probing of the hidden data is allowed, especially in a case like this where it matters a lot for time series prediction.
\nI also though SMAPE was a weird choice of metric that didn't make sense in a business sense. A change in a small county would have a large influence in SMAPE, where if you are trying to capture revenue streams, wouldn't you care a lot more about a change in the number of active forecasts in Los Angeles? I can only imagine this would be useful for allocation of advertising dollars.
\nThe code: Godaddy Microbusiness Final
\nAs has been pointed out by many people, SMAPE is a relative metric. With the kind of forecast values we are looking at (~1.2-1.5 for 1 month, ~3.2 for months 3-5 together), we can look at the contribution of an individual CFIPS.
\n$$\\Delta_{SMAPE,i} = \\frac{200}{n_{counties}} \\frac{|F_i - A_i|}{F_i+A_i}$$
\nWhich, in the case of something going from zero to non-zero can be ~0.0638 per month difference. At the same time, a prediction for most of these models seems to be on the order of 0.5% change per month, which is far below the smallest change in active entries for many 25% of the counties.
\nSecond, there are many CFIPS where the data is terrible. The hosts acknowledged that they had a methodology change in Jan 2021, leading to a number of jumps, and I suspect there was another change after the first month.
\nThe one which always bothered me was CFIPS 56033, Sheridan Co, WY, where there are 2.36 microbusinesses/working age person. I can only guess there is some bug where if there is a misclassification they dump it in that CFIPS. I also wondered if there was some fraud happening during COVID times as people chased PPP loans given the large jumps. However, many of these issues would eventually revert, like the below:
\n
So the question I think that make or breaks this, if we identify a large jump, will it revert? In this I believe we are helped by the gap between December and the first forecast month, March. I also hope GoDaddy gets better in their data collection procedures, which would help. In then end, if they want to use this in business, it would probably not be that useful to have a large machine learning error correction model for data collection issues.
\nMy solution is a mix of public leaderboard probing for individual CFIPS changes, reversion for outlier CFIPS, and a forecast of the smooth changes.
\nGiven the equation above and as @petersorenson360 suggested, and that the leaderboard was active to 4 decimal places, you are sensitive to changes of less than 1% in the value an individual value. Given you had about 20 days from revealed test to final submission deadline, you could probe about 100 CFIPS for final values. It was actually easier if you made the value worse since the equation above would allow you to work out the exact number of active entries.
\nMany large jumps would the revert, which we tried to work out how long that might take. This will probably be the difference in the end of who wins and loses, and I think the couple month gap between the test and private data will help ensure they revert. This was by far the top influence.
\nAfter seeing GiBa's @titericz notebook, his data cleaning method reminded me of something used in futures algorithmic trading, called a continuous contract. Basically, you need to take a discontinuous series of prices and make it smooth. So for what I will call the continuous forecast, I used a data cleaning method where I looked for large jumps in the number of active entries, then did a shift to smooth them out. So the month where there was a large active jump became zero active jump. This smoothing method gave the best CV score of the number that I tried.
I then setup the CV environment. For the model used XGBoost, added extra indicators, and prevented peeking in the future:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5552115%2Ff1ba9755aeb2b45525fd98359dbf9cd2%2FFeatureImportances_1.png?generation=1687036241940575&alt=media\")
I found a population cut of about 5,000 was helpful. Rounding the model to an integer number of active helped a bit. A lot of these external indicators proved more helpful for the longer term forecasts, where there is some reversion to an average.
\nOther notes:
\nscale variable used seemed to work in forecasting the public leaderboard, but did terribly in my CV environment.Nice work! Given the data quality, smoothing was something I also spent quite some time trying to figure out at the beginning of this challenge. Especially given the large changepoint in the data on 2021-01-01 because of the methodology change in how GoDaddy computed the microbusiness density of counties. The smoothing method you described sounds rather similar to my method described here:
Essentially my strategy was to go from left to right and shift the data on the left anytime there was a changepoints (I computed changepoints via differencing and identifying points outside 4 standard deviations of them mean). I’de iteratively repeat this until no more changepoints were detected. I'de really like to hear more about the specifics of your smoothing algorithm and what your CV/LB were with and without smoothing. For me, my CV would increase with smoothing, but public LB score dropped.
", "votes": 1, "replies": [ { "id": 2185166, "author_name": "Daniel Phalen", "author_url": "", "post_date": "2023-03-16T21:14:28", "content": "You probably would have gotten a similar result based on your technique.
\nFor the cross validation, I know in some public notebooks there was an explicit violation of proper CV technique since the smoothing was done before the cross validation. I think to properly setup your environment given the challenge and the data quality, all data cleaning, etc should be a part of the model itself and all cross validation should be done in a walk forward mode since it is a time series challenge.
\nFor example, the GRU models used with Group K fold would in this case use the the group CV for early stopping and validation of the training phase, but to test the overall model one would need to use the walk forward method.
\nNot sure of your setup, so this may not affect you
", "votes": 0, "replies": [ { "id": 2185210, "author_name": "ryan", "author_url": "", "post_date": "2023-03-16T21:57:09.690000", "content": "Thanks for your reply. Now that you mentioned this, I realize I performed smoothing before computing my CV… Whoops!
\nI also played around with GRU's a litle bit, but did realize this look ahead bias that public notebook had (they were reporting CV scores with a SMAPE of ~1, which is obviously not representative of LB scores) and changed the code to perform a sliding window approach during training (which is your description of \"walking forward\" I presume).
\nBest of luck in the coming months!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2184811, "author_name": "tarick.morty", "author_url": "", "post_date": "2023-03-16T16:11:24.337000", "content": "Thanks for sharing. How was your CV result month over month with and without using the Jan predictions?
", "votes": 0, "replies": [ { "id": 2185170, "author_name": "Daniel Phalen", "author_url": "", "post_date": "2023-03-16T21:20:51.127000", "content": "For the Jan predictions, not sure I can answer that until the end of the competition. Much of that is just a hope that the interaction of those predictions with my model will not cause some bias that I could not detect.
\nGiven my CV setup, a last value model gave these results on the original training data:
\nTrain up to and including data from 2022-01-01 and evaluate forecasts from 2022-04-01 to 2022-06-01 = 3.1956774734784967
\nTrain up to and including data from 2022-02-01 and evaluate forecasts from 2022-05-01 to 2022-07-01 = 3.4152913611520037
\nTrain up to and including data from 2022-03-01 and evaluate forecasts from 2022-06-01 to 2022-08-01 = 3.3317878979931765
\nTrain up to and including data from 2022-04-01 and evaluate forecasts from 2022-07-01 to 2022-09-01 = 3.3842283789667764
\nTrain up to and including data from 2022-05-01 and evaluate forecasts from 2022-08-01 to 2022-10-01 = 3.7841915766675602
\nAverage SMAPE = 3.4222353376516024
And the final model I used:
\nTrain up to and including data from 2022-01-01 and evaluate forecasts from 2022-04-01 to 2022-06-01 = 2.962060956397758
\nTrain up to and including data from 2022-02-01 and evaluate forecasts from 2022-05-01 to 2022-07-01 = 3.097867393456829
\nTrain up to and including data from 2022-03-01 and evaluate forecasts from 2022-06-01 to 2022-08-01 = 3.0168450673149243
\nTrain up to and including data from 2022-04-01 and evaluate forecasts from 2022-07-01 to 2022-09-01 = 2.9881429431194393
\nTrain up to and including data from 2022-05-01 and evaluate forecasts from 2022-08-01 to 2022-10-01 = 3.3630555199321344
\nAverage SMAPE = 3.085594376044217
Very useful！Thanks a lot.
", "votes": 0, "replies": [] } ], "comment_count": 6, "messages": [ { "id": 2184759, "postDate": "2023-03-16T15:49:45.443Z", "content": "Interesting problem, and it was also interesting to see the solutions came up with. I'm pretty new to these competitions. The biggest uncertainty to me is how much probing of the hidden data is allowed, especially in a case like this where it matters a lot for time series prediction.
\nI also though SMAPE was a weird choice of metric that didn't make sense in a business sense. A change in a small county would have a large influence in SMAPE, where if you are trying to capture revenue streams, wouldn't you care a lot more about a change in the number of active forecasts in Los Angeles? I can only imagine this would be useful for allocation of advertising dollars.
\nThe code: Godaddy Microbusiness Final
\nAs has been pointed out by many people, SMAPE is a relative metric. With the kind of forecast values we are looking at (~1.2-1.5 for 1 month, ~3.2 for months 3-5 together), we can look at the contribution of an individual CFIPS.
\n$$\\Delta_{SMAPE,i} = \\frac{200}{n_{counties}} \\frac{|F_i - A_i|}{F_i+A_i}$$
\nWhich, in the case of something going from zero to non-zero can be ~0.0638 per month difference. At the same time, a prediction for most of these models seems to be on the order of 0.5% change per month, which is far below the smallest change in active entries for many 25% of the counties.
\nSecond, there are many CFIPS where the data is terrible. The hosts acknowledged that they had a methodology change in Jan 2021, leading to a number of jumps, and I suspect there was another change after the first month.
\nThe one which always bothered me was CFIPS 56033, Sheridan Co, WY, where there are 2.36 microbusinesses/working age person. I can only guess there is some bug where if there is a misclassification they dump it in that CFIPS. I also wondered if there was some fraud happening during COVID times as people chased PPP loans given the large jumps. However, many of these issues would eventually revert, like the below:
\n
So the question I think that make or breaks this, if we identify a large jump, will it revert? In this I believe we are helped by the gap between December and the first forecast month, March. I also hope GoDaddy gets better in their data collection procedures, which would help. In then end, if they want to use this in business, it would probably not be that useful to have a large machine learning error correction model for data collection issues.
\nMy solution is a mix of public leaderboard probing for individual CFIPS changes, reversion for outlier CFIPS, and a forecast of the smooth changes.
\nGiven the equation above and as @petersorenson360 suggested, and that the leaderboard was active to 4 decimal places, you are sensitive to changes of less than 1% in the value an individual value. Given you had about 20 days from revealed test to final submission deadline, you could probe about 100 CFIPS for final values. It was actually easier if you made the value worse since the equation above would allow you to work out the exact number of active entries.
\nMany large jumps would the revert, which we tried to work out how long that might take. This will probably be the difference in the end of who wins and loses, and I think the couple month gap between the test and private data will help ensure they revert. This was by far the top influence.
\nAfter seeing GiBa's @titericz notebook, his data cleaning method reminded me of something used in futures algorithmic trading, called a continuous contract. Basically, you need to take a discontinuous series of prices and make it smooth. So for what I will call the continuous forecast, I used a data cleaning method where I looked for large jumps in the number of active entries, then did a shift to smooth them out. So the month where there was a large active jump became zero active jump. This smoothing method gave the best CV score of the number that I tried.
I then setup the CV environment. For the model used XGBoost, added extra indicators, and prevented peeking in the future:
\n
I found a population cut of about 5,000 was helpful. Rounding the model to an integer number of active helped a bit. A lot of these external indicators proved more helpful for the longer term forecasts, where there is some reversion to an average.
\nOther notes:
\nscale variable used seemed to work in forecasting the public leaderboard, but did terribly in my CV environment.Nice work! Given the data quality, smoothing was something I also spent quite some time trying to figure out at the beginning of this challenge. Especially given the large changepoint in the data on 2021-01-01 because of the methodology change in how GoDaddy computed the microbusiness density of counties. The smoothing method you described sounds rather similar to my method described here:
Essentially my strategy was to go from left to right and shift the data on the left anytime there was a changepoints (I computed changepoints via differencing and identifying points outside 4 standard deviations of them mean). I’de iteratively repeat this until no more changepoints were detected. I'de really like to hear more about the specifics of your smoothing algorithm and what your CV/LB were with and without smoothing. For me, my CV would increase with smoothing, but public LB score dropped.
", "rawMarkdown": "Nice work! Given the data quality, smoothing was something I also spent quite some time trying to figure out at the beginning of this challenge. Especially given the large changepoint in the data on `2021-01-01` because of the methodology change in how GoDaddy computed the microbusiness density of counties. The smoothing method you described sounds rather similar to my method described here:\n - [Outliers, Changepoints, and Smoothing](https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/393364)\n\nEssentially my strategy was to go from left to right and shift the data on the left anytime there was a changepoints (I computed changepoints via differencing and identifying points outside 4 standard deviations of them mean). I’de iteratively repeat this until no more changepoints were detected. I'de really like to hear more about the specifics of your smoothing algorithm and what your CV/LB were with and without smoothing. For me, my CV would increase with smoothing, but public LB score dropped. \n", "votes": 1, "replies": [ { "id": 2185166, "postDate": "2023-03-16T21:14:28Z", "content": "You probably would have gotten a similar result based on your technique.
\nFor the cross validation, I know in some public notebooks there was an explicit violation of proper CV technique since the smoothing was done before the cross validation. I think to properly setup your environment given the challenge and the data quality, all data cleaning, etc should be a part of the model itself and all cross validation should be done in a walk forward mode since it is a time series challenge.
\nFor example, the GRU models used with Group K fold would in this case use the the group CV for early stopping and validation of the training phase, but to test the overall model one would need to use the walk forward method.
\nNot sure of your setup, so this may not affect you
", "rawMarkdown": "You probably would have gotten a similar result based on your technique. \n\nFor the cross validation, I know in some public notebooks there was an explicit violation of proper CV technique since the smoothing was done before the cross validation. I think to properly setup your environment given the challenge and the data quality, all data cleaning, etc should be a part of the model itself and all cross validation should be done in a walk forward mode since it is a time series challenge. \n\nFor example, the GRU models used with Group K fold would in this case use the the group CV for early stopping and validation of the training phase, but to test the overall model one would need to use the walk forward method.\n\nNot sure of your setup, so this may not affect you\n\n\n", "replies": [ { "id": 2185210, "postDate": "2023-03-16T21:57:09.690Z", "content": "Thanks for your reply. Now that you mentioned this, I realize I performed smoothing before computing my CV… Whoops!
\nI also played around with GRU's a litle bit, but did realize this look ahead bias that public notebook had (they were reporting CV scores with a SMAPE of ~1, which is obviously not representative of LB scores) and changed the code to perform a sliding window approach during training (which is your description of \"walking forward\" I presume).
\nBest of luck in the coming months!
", "rawMarkdown": "Thanks for your reply. Now that you mentioned this, I realize I performed smoothing before computing my CV... Whoops! \n\nI also played around with GRU's a litle bit, but did realize this look ahead bias that public notebook had (they were reporting CV scores with a SMAPE of ~1, which is obviously not representative of LB scores) and changed the code to perform a sliding window approach during training (which is your description of \"walking forward\" I presume). \n\nBest of luck in the coming months!", "votes": 1 } ] } ] }, { "id": 2184811, "postDate": "2023-03-16T16:11:24.337Z", "content": "Thanks for sharing. How was your CV result month over month with and without using the Jan predictions?
", "rawMarkdown": "Thanks for sharing. How was your CV result month over month with and without using the Jan predictions?", "replies": [ { "id": 2185170, "postDate": "2023-03-16T21:20:51.127Z", "content": "For the Jan predictions, not sure I can answer that until the end of the competition. Much of that is just a hope that the interaction of those predictions with my model will not cause some bias that I could not detect.
\nGiven my CV setup, a last value model gave these results on the original training data:
\nTrain up to and including data from 2022-01-01 and evaluate forecasts from 2022-04-01 to 2022-06-01 = 3.1956774734784967
\nTrain up to and including data from 2022-02-01 and evaluate forecasts from 2022-05-01 to 2022-07-01 = 3.4152913611520037
\nTrain up to and including data from 2022-03-01 and evaluate forecasts from 2022-06-01 to 2022-08-01 = 3.3317878979931765
\nTrain up to and including data from 2022-04-01 and evaluate forecasts from 2022-07-01 to 2022-09-01 = 3.3842283789667764
\nTrain up to and including data from 2022-05-01 and evaluate forecasts from 2022-08-01 to 2022-10-01 = 3.7841915766675602
\nAverage SMAPE = 3.4222353376516024
And the final model I used:
\nTrain up to and including data from 2022-01-01 and evaluate forecasts from 2022-04-01 to 2022-06-01 = 2.962060956397758
\nTrain up to and including data from 2022-02-01 and evaluate forecasts from 2022-05-01 to 2022-07-01 = 3.097867393456829
\nTrain up to and including data from 2022-03-01 and evaluate forecasts from 2022-06-01 to 2022-08-01 = 3.0168450673149243
\nTrain up to and including data from 2022-04-01 and evaluate forecasts from 2022-07-01 to 2022-09-01 = 2.9881429431194393
\nTrain up to and including data from 2022-05-01 and evaluate forecasts from 2022-08-01 to 2022-10-01 = 3.3630555199321344
\nAverage SMAPE = 3.085594376044217
Very useful！Thanks a lot.
", "rawMarkdown": "Very useful！Thanks a lot." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:28:10Z", "competition_id": 41881, "competition_slug": "godaddy-microbusiness-density-forecasting", "team_id": 9685195, "solution_url": "/competitions/godaddy-microbusiness-density-forecasting/writeups/daniel-phalen-2nd-place-solution-the-godaddy-micro" }