Hello everyone,
\nI may not have the best performing model or the most clever, but I thought I would share my strategy for people interested. Prior to this competition I had no time series experience and learned a ton during this competition, so thanks to everyone who made their notebooks public and shared their ideas. My strategy consists of two different strategies/models.
\nThings I tried that did not work: GRU's, linear regression models (I had to be too selective with counties to get good performance), and traditional autoregressive models like ARMA didn't work for me. If anyone got these to work well, I'de love to hear how you did it.
\nFirst model: My first model is just a “last value” model and is meant to be a hedge against my main model. The strategy was this: Overfit the leaderboard to leak back the best values to use for a “last value” model. I figure since most public notebooks work via predicting the first value of the public lb and just repeating that value, then I should be in a pretty decent place if I can outperform the top public notebooks in the public LB (although there are no guarantees this will outperform in private LB).
\nSecond model: My second model was a modified version of this notebook:
\nFor my modifications I did the following:
\nFirst I performed a custom preprocessing scheme that I built, followed by another preprocessing scheme from the popular Better XGB Baseline notebook. More information on my preprocessing scheme can be found in my notebook explaining it here: Outliers, Changepoints, and Smoothing
Next, rather than only predicting the first value, I modified the code to autoregressively predict each value in the testing set.
I incorporating the ideas in the following notebook: Use Discrete Nature of microbusiness_density
I removed the meta model and just did a threefold ensemble of an XGBoost, CatBoost, and LGBM model.
Tons of other \"micro-adjustments\" to preprocessing, feature engineering, training, and postprocessing.
Removed \"suspicious\" looking code. See the topic below.
On the Dangers of Public Notebooks:
\nIt seems that the five best scoring public notebooks (maybe more, I stopped looking after five) have a fatal flaw in their submissions resulting from the following code at the bottom of each notebook:
\nfor i, row in sub.iterrows():\n test.iat[i,1] = (\n 0.65*test.iat[i,1] +\n 0.35*row[\"microbusiness_density\"]\n )\ntest.to_csv('submission.csv', index=False)\ntest.head()\nThis block of code will essentially cause all the “private LB” values to be 0.35 times the last predicted public LB value. Therefore, if users who blindly copied the top public notebooks didn’t check for this, their models will likely not generalize to the private LB data. Whether this was done intentionally or not, I do not know. In total, these notebooks had 2545 copies and edits (likely from a lot of the same people copying). Below is a sample from these submissions.
\n| row_id | \nmicrobusiness_density | \n
|---|---|
| 1001_2022-11-01 | \n3.404743687834736 | \n
| 1001_2022-12-01 | \n3.422746298213241 | \n
| 1001_2023-01-01 | \n3.3372338989153425 | \n
| 1001_2023-02-01 | \n1.1679193483055044 | \n
| 1001_2023-03-01 | \n1.1679193483055044 | \n
| 1001_2023-04-01 | \n1.1679193483055044 | \n
| 1001_2023-05-01 | \n1.1679193483055044 | \n
| 1001_2023-06-01 | \n1.1679193483055044 | \n
Best of luck in the coming months!
\nRyan P.
", "comments": [ { "id": 2182224, "author_name": "Yue Sun", "author_url": "", "post_date": "2023-03-15T02:36:28.140000", "content": "Another danger to know is to ensure that the features used to predict January data are validly calculated values when predicting March, April, and May, instead of null values. Otherwise, wrong features may lead to wrong prediction.
", "votes": 10, "replies": [ { "id": 2182239, "author_name": "ryan", "author_url": "", "post_date": "2023-03-15T02:46:46.063000", "content": "Yes, this is a really good point I didn't bring up in my original post and something that I was careful to do. Once I predicted a new value, the main features I had to update for my notebook were the lag features. Since I used lagged active features too, this also required me to compute the active feature for each predicted time step from the predicted value and the 2020 population (later adjusted with the 2021 population) via the following code:
\nactive_mask = (raw[\"dcount\"] == (TS + 1))\n\nraw.loc[active_mask, \"active\"] = ((raw.loc[active_mask, \"pop2020\"] * raw.loc[active_mask, \"ypred\"]) / 100).astype(int)\nThen updating the lag features with a function I made: raw = update_lags(raw)
Yes, was really surprised when I saw what top public notebook predicts for private LB.
\nI thought about using overfitted Jan data as well, but decided not to do it, thought that all errors will multiply, instead I based my models on 100% true Dec last values, don't know whether it was good or bad decision though, private will tell
Thank you for sharing!! 👋
\nWhat about reusing the last value model many times, and generating several predictions. I think the public notebook's pipeline is really useful, will it cause some undetectable dangers?
Thanks for your comment!
\nI'm not 100% sure what you mean by reusing the model many times. If I understand correctly, \"reusing the last value model many times, and generating several predictions\" is indeed what I did and what I meant when I said \"autoregressively\" generate predictions. Like most models, this likely has it's upsides and downsides. From my analysis, there seems to be a left-skew in the distribution of my predicted values (thus my predictions will tend to be more positive than negative). As an alternative method, it seems other people like @joshuascj used models trained with \"gaps\" to predict the following months, find his explanation here.
", "votes": 2, "replies": [ { "id": 2182272, "author_name": "Zhengyang", "author_url": "", "post_date": "2023-03-15T03:11:42.093000", "content": "Thanks for the reply!
\nI have found that without many modifications, the model will generate negative predictions.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2230430, "author_name": "Jahid hossan", "author_url": "", "post_date": "2023-04-22T11:10:59.950000", "content": "Impressive! Thank you for sharing with us.
", "votes": 0, "replies": [] } ], "comment_count": 7, "messages": [ { "id": 2182158, "postDate": "2023-03-15T01:18:04.510Z", "content": "Hello everyone,
\nI may not have the best performing model or the most clever, but I thought I would share my strategy for people interested. Prior to this competition I had no time series experience and learned a ton during this competition, so thanks to everyone who made their notebooks public and shared their ideas. My strategy consists of two different strategies/models.
\nThings I tried that did not work: GRU's, linear regression models (I had to be too selective with counties to get good performance), and traditional autoregressive models like ARMA didn't work for me. If anyone got these to work well, I'de love to hear how you did it.
\nFirst model: My first model is just a “last value” model and is meant to be a hedge against my main model. The strategy was this: Overfit the leaderboard to leak back the best values to use for a “last value” model. I figure since most public notebooks work via predicting the first value of the public lb and just repeating that value, then I should be in a pretty decent place if I can outperform the top public notebooks in the public LB (although there are no guarantees this will outperform in private LB).
\nSecond model: My second model was a modified version of this notebook:
\nFor my modifications I did the following:
\nFirst I performed a custom preprocessing scheme that I built, followed by another preprocessing scheme from the popular Better XGB Baseline notebook. More information on my preprocessing scheme can be found in my notebook explaining it here: Outliers, Changepoints, and Smoothing
Next, rather than only predicting the first value, I modified the code to autoregressively predict each value in the testing set.
I incorporating the ideas in the following notebook: Use Discrete Nature of microbusiness_density
I removed the meta model and just did a threefold ensemble of an XGBoost, CatBoost, and LGBM model.
Tons of other \"micro-adjustments\" to preprocessing, feature engineering, training, and postprocessing.
Removed \"suspicious\" looking code. See the topic below.
On the Dangers of Public Notebooks:
\nIt seems that the five best scoring public notebooks (maybe more, I stopped looking after five) have a fatal flaw in their submissions resulting from the following code at the bottom of each notebook:
\nfor i, row in sub.iterrows():\n test.iat[i,1] = (\n 0.65*test.iat[i,1] +\n 0.35*row[\"microbusiness_density\"]\n )\ntest.to_csv('submission.csv', index=False)\ntest.head()\nThis block of code will essentially cause all the “private LB” values to be 0.35 times the last predicted public LB value. Therefore, if users who blindly copied the top public notebooks didn’t check for this, their models will likely not generalize to the private LB data. Whether this was done intentionally or not, I do not know. In total, these notebooks had 2545 copies and edits (likely from a lot of the same people copying). Below is a sample from these submissions.
\n| row_id | \nmicrobusiness_density | \n
|---|---|
| 1001_2022-11-01 | \n3.404743687834736 | \n
| 1001_2022-12-01 | \n3.422746298213241 | \n
| 1001_2023-01-01 | \n3.3372338989153425 | \n
| 1001_2023-02-01 | \n1.1679193483055044 | \n
| 1001_2023-03-01 | \n1.1679193483055044 | \n
| 1001_2023-04-01 | \n1.1679193483055044 | \n
| 1001_2023-05-01 | \n1.1679193483055044 | \n
| 1001_2023-06-01 | \n1.1679193483055044 | \n
Best of luck in the coming months!
\nRyan P.
", "rawMarkdown": "Hello everyone,\n\nI may not have the best performing model or the most clever, but I thought I would share my strategy for people interested. Prior to this competition I had no time series experience and learned a ton during this competition, so thanks to everyone who made their notebooks public and shared their ideas. My strategy consists of two different strategies/models. \n\nThings I tried that did not work: GRU's, linear regression models (I had to be too selective with counties to get good performance), and traditional autoregressive models like ARMA didn't work for me. If anyone got these to work well, I'de love to hear how you did it. \n\n**First model:** My first model is just a “last value” model and is meant to be a hedge against my main model. The strategy was this: Overfit the leaderboard to leak back the best values to use for a “last value” model. I figure since most public notebooks work via predicting the first value of the public lb and just repeating that value, then I should be in a pretty decent place if I can outperform the top public notebooks in the public LB (although there are no guarantees this will outperform in private LB).\n\n**Second model:** My second model was a modified version of this notebook:\n - [Micro-Business. Catboost](https://www.kaggle.com/code/vadimkamaev/micro-business-catboost). \n\nFor my modifications I did the following:\n\n1. First I performed a custom preprocessing scheme that I built, followed by another preprocessing scheme from the popular [Better XGB Baseline](https://www.kaggle.com/code/titericz/better-xgb-baseline) notebook. More information on my preprocessing scheme can be found in my notebook explaining it here: [Outliers, Changepoints, and Smoothing](https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/393364)\n\n2. Next, rather than only predicting the first value, I modified the code to autoregressively predict each value in the testing set. \n\n3. I incorporating the ideas in the following notebook: [Use Discrete Nature of microbusiness_density](https://www.kaggle.com/code/vitalykudelya/use-discrete-nature-of-microbusiness-density)\n\n4. I removed the meta model and just did a threefold ensemble of an XGBoost, CatBoost, and LGBM model. \n\n5. Tons of other \"micro-adjustments\" to preprocessing, feature engineering, training, and postprocessing.\n\n6. Removed \"suspicious\" looking code. See the topic below.\n\n**On the Dangers of Public Notebooks:**\n\nIt seems that the five best scoring public notebooks (maybe more, I stopped looking after five) have a fatal flaw in their submissions resulting from the following code at the bottom of each notebook:\n\n\n```\nfor i, row in sub.iterrows():\n test.iat[i,1] = (\n 0.65*test.iat[i,1] +\n 0.35*row[\"microbusiness_density\"]\n )\ntest.to_csv('submission.csv', index=False)\ntest.head()\n```\n\nThis block of code will essentially cause all the “private LB” values to be 0.35 times the last predicted public LB value. Therefore, if users who blindly copied the top public notebooks didn’t check for this, their models will likely not generalize to the private LB data. Whether this was done intentionally or not, I do not know. In total, these notebooks had 2545 copies and edits (likely from a lot of the same people copying). Below is a sample from these submissions.\n\n| row_id | microbusiness_density |\n| ----------------- | --------------- |\n| 1001_2022-11-01 | 3.404743687834736 |\n| 1001_2022-12-01 | 3.422746298213241 |\n| 1001_2023-01-01 | 3.3372338989153425 |\n| 1001_2023-02-01 | 1.1679193483055044 |\n| 1001_2023-03-01 | 1.1679193483055044 |\n| 1001_2023-04-01 | 1.1679193483055044 |\n| 1001_2023-05-01 | 1.1679193483055044 |\n| 1001_2023-06-01 | 1.1679193483055044 |\n\nBest of luck in the coming months!\n\nRyan P.", "votes": 17 }, { "id": 2182224, "postDate": "2023-03-15T02:36:28.140Z", "content": "Another danger to know is to ensure that the features used to predict January data are validly calculated values when predicting March, April, and May, instead of null values. Otherwise, wrong features may lead to wrong prediction.
", "rawMarkdown": "Another danger to know is to ensure that the features used to predict January data are validly calculated values when predicting March, April, and May, instead of null values. Otherwise, wrong features may lead to wrong prediction.", "votes": 10, "replies": [ { "id": 2182239, "postDate": "2023-03-15T02:46:46.063Z", "content": "Yes, this is a really good point I didn't bring up in my original post and something that I was careful to do. Once I predicted a new value, the main features I had to update for my notebook were the lag features. Since I used lagged active features too, this also required me to compute the active feature for each predicted time step from the predicted value and the 2020 population (later adjusted with the 2021 population) via the following code:
\nactive_mask = (raw[\"dcount\"] == (TS + 1))\n\nraw.loc[active_mask, \"active\"] = ((raw.loc[active_mask, \"pop2020\"] * raw.loc[active_mask, \"ypred\"]) / 100).astype(int)\nThen updating the lag features with a function I made: raw = update_lags(raw)
Yes, was really surprised when I saw what top public notebook predicts for private LB.
\nI thought about using overfitted Jan data as well, but decided not to do it, thought that all errors will multiply, instead I based my models on 100% true Dec last values, don't know whether it was good or bad decision though, private will tell
Thank you for sharing!! 👋
\nWhat about reusing the last value model many times, and generating several predictions. I think the public notebook's pipeline is really useful, will it cause some undetectable dangers?
Thanks for your comment!
\nI'm not 100% sure what you mean by reusing the model many times. If I understand correctly, \"reusing the last value model many times, and generating several predictions\" is indeed what I did and what I meant when I said \"autoregressively\" generate predictions. Like most models, this likely has it's upsides and downsides. From my analysis, there seems to be a left-skew in the distribution of my predicted values (thus my predictions will tend to be more positive than negative). As an alternative method, it seems other people like @joshuascj used models trained with \"gaps\" to predict the following months, find his explanation here.
", "rawMarkdown": "Thanks for your comment! \n\nI'm not 100% sure what you mean by reusing the model many times. If I understand correctly, \"reusing the last value model many times, and generating several predictions\" is indeed what I did and what I meant when I said \"autoregressively\" generate predictions. Like most models, this likely has it's upsides and downsides. From my analysis, there seems to be a left-skew in the distribution of my predicted values (thus my predictions will tend to be more positive than negative). As an alternative method, it seems other people like @joshuascj used models trained with \"gaps\" to predict the following months, find his explanation [here](https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/394822). ", "votes": 2, "replies": [ { "id": 2182272, "postDate": "2023-03-15T03:11:42.093Z", "content": "Thanks for the reply!
\nI have found that without many modifications, the model will generate negative predictions.
", "rawMarkdown": "Thanks for the reply! \n\nI have found that without many modifications, the model will generate negative predictions.", "votes": 2 } ] } ] }, { "id": 2230430, "postDate": "2023-04-22T11:10:59.950Z", "content": "Impressive! Thank you for sharing with us.
", "rawMarkdown": "Impressive! Thank you for sharing with us." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:27:27Z", "competition_id": 41881, "competition_slug": "godaddy-microbusiness-density-forecasting", "team_id": 9629009, "solution_url": "/competitions/godaddy-microbusiness-density-forecasting/writeups/ryan-my-strategy-the-dangers-of-public-notebooks" }