\n
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Mar-2023/may.png\")
New: I've uploaded the formal model submit file that was required to receive the prize distribution: https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/425000
\nCode here - https://www.kaggle.com/kaggleqrdl/first-place-code, see below for an explanation.
\nAcknowledgements
\nLet me first acknowledge Kaggle, the Venture Forward folks, and GoDaddy. Not just for creating Kaggle itself, hosting the contest and providing the microbiz data - incredible feats of faith, truly - but also for being patient with my fixation on transparency. I am an egalitarian thru and thru, and firmly believe transparency is what gets us there.
\nIntroduction
\nThe contest was to predict microbusiness density 3,4,5 months lookahead for mar/apr/may 2023 per each of the 3135 US counties in the dataset. Microbusiness density here is roughly equal to the # of GoDaddy internet domains registered / adult population in a county as measured by a 2 year lagging US census.
\nEg: if a total of 10 GoDaddy domains were registered in De Kalb, MO with 200 adult population according to the 2021 census in 2023 February, than microbusiness density for that county would increase by 10/200 or 0.05 in that month.
\nWe were given historical microbiz data for 2019 August -> 2022 December to train with, and a not-so-secret 'public LB' of data for January to validate against. Scores on the public LB were provided by Kaggle, and we were given 5 submissions per day.
\nMethod
\nTo win this, I took a baseline prediction for the january public leaderboard based on overfitting it, and used linear regression to project into mar/apr/may
, [, , ,] - month lookahead\n, [, , ,] - month lookahead\n, [, , , , ] - month lookahead\nI selected these features by cross validating on the last possible window (2,3,4 month respectively) without overlap.
\nInsights
\nThe key ideas that seemed to help:
*every regression model in sklearn, various arima libraries (a lot of time here, wasted I feel), hierarchical, xgb, catboost, lgbm, pytorch/dll. I tried both manually tuning the previous as well as various auto approaches like grid search/optuna. Lots of very exciting results, but eventually it occurred to me I was probably just overfitting.
\nIt's worth observing that my lead against the #2 and #3 spots increased significantly from the public LB versus the private LB. I believe this was due to superior predictions for apr/may as my march results were somewhat lackluster. My lead over the top 20 spots also increased, though it is less clear how much of the public LB influenced their models.
\nIdeas that didn't seem as important
\nI also tried a lot of different datasets (including all the ones I shared publicly) up to and including google keyword trends for 'godaddy'. :) The above worked best when combined with early stopping. There would be sparks of good CV scores when using other features, but adding them frequently felt like overfitting.
\nI briefly looked into using zone files and whois databases, but these were not openly available so of course I didn't use them.
\nCrawling/Scraping would have in theory been public data, but I had committed to myself not to use any data that I didn't share openly, so I figured it wouldn't be worth the effort required.
\nEverything I used I shared in the dataset I made publically available at the beginning of the comp.
\nI did a lot of initial work around per county models, work I eventually decided was wasted. There was barely enough data to validate the models and features above. There might be a way of segregate the counties, the question is - Would you have to segregate your training data as well? Would the increased complexity be worth what might potentially be only incremental improvements? Hard to say, especially since observing incremental improvements is mostly impossible when your CV scores are so volatile.
\nFuture directions
\nOne thing I spent time looking at was correlation between the counties. There is an argument to be made I think that different counties may follow similar trends at potential lag times. There might be more to do here, but it would need to be done globally rather than individually, and your models would require appropriate data sufficiency.
Another feature I didn't plumb carefully enough was 1/2/3/etc annual renewal on GoDaddy domains, an idea I got by chatting with some domain resellers recently on a reseller domain community forum. I suspect they would provide a well of ideas around this that could be leveraged more.
\nFinal thoughts
\nAs I mentioned above, and others have mentioned, the data was non stationary except for some global linear growth coefficients. It's also been said that the smaller counties added a lot of noise, but I believe it was the right choice to try to see if we could do something there. There is a paucity of stats and analysis around more rural areas and emergent issues are hard to detect because of it.
\nAnother way of looking at the data might be volatility instead of direction. The wild fluctuations in active domains could just be due to data errors but they could also be due to underlying social and economic factors. In the latter case, we need to consider: could the volatility be an opportunity, like the first sparks of a fire? Could these counties benefit from more targeted and opportune investment? If volatile counties have increased potential, what are the triggers for volatility? These are questions and experiments (signal + intervention) worth considering
\nIf the VF folks would like to contact me (heh), feel free to reach out. As you may have noticed, I could talk about this stuff all day long with folks who are equally interested.
\n..
\nThere was a bug in the team update that didn't let me change the LB pointer which I had originally pointed to the message below, so you're stuck with both. I originally pointed it here as I was more interested in what other folks did as I wasn't really that optimistic about my simplistic approach above. In retrospect, I probably should have trusted the process of elimination that I went through.
\nprevious thread title: What is your mean active change over december for mar/apr/may?
\nEdit - to make clear up front, you can share your results from your submission by just executing this code. It uses microbusiness density and not active, but that should be fine for comparison.
\ndfc = pd.read_csv()\nrt = pd.read_csv().set_index()\ndfc[] = dfc[].apply( x:x[-:])\ndfc[] = dfc[].apply( x:(x[:x.index()]))\ndfc = dfc.reset_index().set_index()\ndfc[] = dfc.apply( r:(r[] - rt.loc[, ])/rt.loc[, ], axis = )\ndisplay(dfc.groupby().mean()[])\nβ¦.
\nIt's going to be awhile before we get final results, and I think May results might play a large role in determining the eventual winners. While we wait, it might fun to share what we actually predicted. We could compare our actual results, which would be good and I've uploaded my two here - but for those who don't want to and perhaps simpler might just be sharing mean % change over active.
\nHere's the code I used. It requires having accurate values for december in your dataframe and also have calculated 'active' as one of your columns.
\nIf you don't want to use active, you can also share microbusiness density as well, but I found it less interesting because of the pop shift. There were some outliers in the pop shift that made it harder to ensure that nothing went awry, so I liked having active available. I've shared both below.
\nYou can calculate cfips/first day with this code:
\ndfc[] = dfc[].apply( x:x[-:])\ndfc[] = dfc[].apply( x:(x[:x.index()]))\nCalculate active_chg values with this code:
\ndfc = dfc.reset_index().set_index()\ndfc[] = dfc.apply( r:(r[] - dfc.loc[, ])/dfc.loc[, ], axis = )\ndfc.groupby().mean()[]\nThis is my seasonal model
\nfirst_day_of_month\n--01 - \n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nName: active_chg, dtype: float64\nThis is my long trend
\nfirst_day_of_month\n--01 -\n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nName: active_chg, dtype: float64\nUsing mbd code if you didn't have an active column:
\ndfc = dfc.reset_index().set_index()\ndfc[] = dfc.apply( r:(r[] - dfc.loc[, ])/dfc.loc[, ], axis = )\ndfc.groupby().mean()[]\nSeasonal sub
\nfirst_day_of_month\n--01 -\n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nName: mbd_chg, dtype: float64\nLong trend sub
\nfirst_day_of_month\n--01 -\n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nName: mbd_chg, dtype: float64\nNote for february I just used the january values. Sharing january isn't really that helpful of course, I just do it for fun here.
", "comments": [ { "id": 2195701, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-03-24T20:30:58.880000", "content": "Here are the 3 history sequences for \"Dec, Jan, Feb, Mar, Apr, May\" in the train data. After we adjust for census population changes. It appears that May decreases in 2022 and 2021. And May increases in 2020. What will May do in 2023??
\n
In 2020 it was short term big drop and then huge economic recovery from COVID, so imo it was not general trend, but anomaly.
\nI didn't apply and seasonality/trend , just used one value for all 3 month.
Values from two models:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3092030%2Ff61811dd61189ff9b1630d2ec850dc81%2Fphoto_2023-03-25_20-40-01.jpg?generation=1679740819109837&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3092030%2Fc4b467a28feb57e71c4232f04eb1f39c%2Fphoto_2023-03-25_20-40-09.jpg?generation=1679740830549104&alt=media\")
For folks looking for a description of my model, I posted it here - https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/394810#2182222
\nHowever, my experience was similar to most folks - simple linear models were best, perhaps informed a bit by economic/census features, and there wasn't much to write about. The CV studies I did showed that using last split CV was frequently superior (though more volatile for sure) to using more extensive validation.
", "votes": 3, "replies": [] }, { "id": 2186232, "author_name": "tarick.morty", "author_url": "", "post_date": "2023-03-17T16:55:39.503000", "content": "Hi @kaggleqrdl since you worked on the trend and seasonal model, any reason why you went for a far more aggressive growth? Do you feel that we will get to see such a significant uptick in the coming months? From your mbd_chg data above, it seems your month over month growth is a factor of ~1.0048x (long trend sub). According to Chris calculations here, the m-o-m growth observed is around ~1.0032x.
Well, I let it run a little wild because of the layoffs in the tech sector. I sort of felt those folks might try to start up microbiz. Historically the tech sector has seen full employment, but lately less so, so I think using historical values we might be seeing some underestimations.
\nIt's all guesses and fun though. I suspect we'll see a lot of new folk getting gold here
", "votes": 1, "replies": [ { "id": 2186693, "author_name": "Sireesh Limbu", "author_url": "", "post_date": "2023-03-18T03:00:17.070000", "content": "I also thought the same last minute haha inspired from @uspangaliev's post here
", "votes": 0, "replies": [] } ] } ] }, { "id": 2185985, "author_name": "Shiner", "author_url": "", "post_date": "2023-03-17T13:25:43.397000", "content": "Share my 2 submissions:
\nfirst_day_of_month\n2023-03-01 0.010246\n2023-04-01 0.012392\n2023-05-01 0.017704\nfirst_day_of_month\n2023-03-01 0.009916\n2023-04-01 0.011304\n2023-05-01 0.015852\nHi! Nice that you came up with this! Fun! Following are mine:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F639336%2F15b429261357604f5de79763db1e03c1%2FMBD_Change.png?generation=1679055169201228&alt=media\")
The top one is with some multipliers for Mar, Apr and May along with a small functionality for increasing/decreasing trend for the last 5 months. The bottom one is the plain models with LastValue Blacklisting.
", "votes": 1, "replies": [] }, { "id": 2185793, "author_name": "tarick.morty", "author_url": "", "post_date": "2023-03-17T09:41:32.537000", "content": "mbd_chg with respect to December:
first_day_of_month\n2022-11-01 -0.006355\n2022-12-01 0.000000\n2023-01-01 0.008735\n2023-02-01 0.007559\n2023-03-01 0.011393\n2023-04-01 0.013531\n2023-05-01 0.014565\n2023-06-01 -0.537519\nName: mbd_chg, dtype: float64\nUsing @cdeotte mbd from his last value submission and your code, I get this. Jan, Feb, and June predictions are from a single model while the others are from my blend. I did not tune January, February and June.
\nfirst_day_of_month\n--01 \n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \n-06-01 \nName: mbd_chg, dtype: float64\nWhen I evaluated my models I used the median and not the mean, as the mean is dominated by large counties, while score is dominated by the more numerous small counties.
\nMedian is:
\nfirst_day_of_month\n--01 \n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \n-06-01 \nName: mbd_chg, dtype: float64\nCongratulations and thanks for sharing your thoughts!!
", "votes": 1, "replies": [ { "id": 2184920, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-16T17:24:12.857000", "content": "Thanks Chen. I was hoping some folks might share their results, but so far everyone seems a bit shy. Ah well.
", "votes": 0, "replies": [ { "id": 2184928, "author_name": "CPMP", "author_url": "", "post_date": "2023-03-16T17:31:06.883000", "content": "Not shy, but it requires some coding for me for instance. And there is little incentive to do so. I will try to find time.
", "votes": 1, "replies": [] }, { "id": 2184942, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-16T17:41:43.140000", "content": "Actually, I did provide some code above for just using microbusiness density.
\nPerhaps I should make this more clear.
\nIf you run this, it should work.
\ndfc = pd.read_csv()\ndfc[] = dfc[].apply( x:x[-:])\ndfc[] = dfc[].apply( x:(x[:x.index()]))\ndfc = dfc.reset_index().set_index()\ndfc[] = dfc.apply( r:(r[] - dfc.loc[, ])/dfc.loc[, ], axis = )\ndisplay(dfc.groupby().mean()[])\nNote that we cannot compare different Kagglers' microbusiness density ratios because everyone can have different Dec 2022 MD predictions. For example do people use Dec 2022 as is from the train data? OR do they change it? Some people may scale it using census data. Some people may predict Dec 2022 from their model, etc etc.
\nOR if someone sets December 2022 to all zeros they will have NaN ratios!
first_day_of_month\n2022-11-01 NaN\n2022-12-01 NaN\n2023-01-01 NaN\n2023-02-01 NaN\n2023-03-01 NaN\n2023-04-01 NaN\n2023-05-01 NaN\n2023-06-01 NaN\n@kaggleqrdl What Dec 2022 predictions do you use?
", "votes": 1, "replies": [] }, { "id": 2184970, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-03-16T17:59:37.700000", "content": "The most consistent and easiest approach to compare different Kagglers' submissions is to download \"last value baseline\" submission.csv file from notebook here. Then you can provide code to merge these predictions to each Kagglers' final submission.csv and compute ratios.
", "votes": 2, "replies": [] }, { "id": 2184978, "author_name": "CPMP", "author_url": "", "post_date": "2023-03-16T18:05:59.240000", "content": "\n\nActually, I did provide some code above for just using microbusiness density.
\n
Sure, but your asked for active change. If i use mbd from revealed test, then the change is inflated by the population change between December and January. It means I have to use what chris says, an upadated mbd.
\nOr I have to compute the change using active count, which requires some coding.
", "votes": 1, "replies": [] }, { "id": 2184994, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-16T18:27:42.923000", "content": "Yes, it does require having accurate december values. The pop shift problem should be OK as we'll all have it for comparison.
", "votes": 0, "replies": [] }, { "id": 2185004, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-16T18:33:21.120000", "content": "You can use this, I'll add it above
\nrt = pd.read_csv().set_index()\ndfc[] = dfc.apply( r:(r[] - rt.loc[, ])/rt.loc[, ], axis = )\nHi! thanks for sharing your thoughts and good luck π
", "votes": 1, "replies": [ { "id": 2184922, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-16T17:25:01.563000", "content": "Thanks Luca, good luck to you as well.
", "votes": 0, "replies": [] } ] }, { "id": 2184075, "author_name": "Shiner", "author_url": "", "post_date": "2023-03-16T05:58:31.843000", "content": "An interesting question, I just checked your submission files. Why you omit the big changes of some counties? Any reason, I feel a little confusing, because it seems a little bit dangerous.
\nFor instance:
\nDo you use moving average? Or just assume they will drop back?
", "votes": 1, "replies": [ { "id": 2184076, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-16T05:59:38.317000", "content": "I'll answer if you share your averages ;p @judith007
", "votes": 0, "replies": [] } ] }, { "id": 2183896, "author_name": "KrisztiΓ‘n Boros", "author_url": "", "post_date": "2023-03-16T02:32:31.303000", "content": "Wow, congrats!
", "votes": 1, "replies": [ { "id": 2183912, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-16T03:05:08.847000", "content": "Just to be clear, the public leaderboard here is nonsense as it frequently is in most Kaggle contests. I enjoy overfitting them to show why that is the case. That said, there may be some advantage to leveraging january last values as I tried to do, but that remains to be seen.
\nThere will very likely be fairly massive shakeup (shake down) here and I fully expect to be among the latter group.
", "votes": 2, "replies": [] } ] }, { "id": 2185003, "author_name": "Nic Vicol", "author_url": "", "post_date": "2023-03-16T18:31:34.337000", "content": "Given that the naive model was hard to beat, I was conservative about growth and damped further horizons. These are my average rates for active with respect to the actuals as of 2022-12-01.
ββββββββββββββ¬βββββββββββ\nβ date β mean β\nββββββββββββββͺβββββββββββ‘\nβ 2023-01-01 β 0.003256 β\nβ 2023-02-01 β 0.005212 β\nβ 2023-03-01 β 0.006449 β\nβ 2023-04-01 β 0.007289 β\nβ 2023-05-01 β 0.00794 β\nβ 2023-06-01 β 0.008466 β\nββββββββββββββ΄βββββββββββ\nThanks Nic! Good luck on your results.
", "votes": 0, "replies": [] } ] }, { "id": 2185262, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2023-03-16T22:55:17.237000", "content": "My results:
\n![\"\"](\"https://i.imgur.com/b5e5Hgu.png\")
:( Not very kaggly, I think.
\nDid you execute this code? Getting the mbd chg over data from revealed test should be fairly straightforward.
\ndfc = pd.read_csv()\nrt = pd.read_csv().set_index()\ndfc[] = dfc[].apply( x:x[-:])\ndfc[] = dfc[].apply( x:(x[:x.index()]))\ndfc = dfc.reset_index().set_index()\ndfc[] = dfc.apply( r:(r[] - rt.loc[, ])/rt.loc[, ], axis = )\ndisplay(dfc.groupby().mean()[])\nI admit to being a bit surprised that most folks hadn't already done this sort of EDA. The contest will likely be won by whoever called the macro trend correct, which isn't always in a straight line.
", "votes": 1, "replies": [ { "id": 2185369, "author_name": "", "author_url": "", "post_date": "2023-03-17T02:22:04.457000", "content": "", "votes": 0, "replies": [] }, { "id": 2185377, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2023-03-17T02:26:04.647000", "content": "I was running
\ndfc = dfc.reset_index().set_index(\"row_id\")\ndfc['mbd_chg'] = dfc.apply(lambda r:(r['microbusiness_density'] - dfc.loc[f\"{r['cfips']}_2022-12-01\", 'microbusiness_density'])/dfc.loc[f\"{r['cfips']}_2022-12-01\", 'microbusiness_density'], axis = 1)\ndfc.groupby(\"first_day_of_month\").mean()['mbd_chg']\nwhich is why I got all Nans. When I run your code, I get this:
\nfirst_day_of_month
\n2022-11-01 -1.000000
\n2022-12-01 -1.000000
\n2023-01-01 0.010812
\n2023-02-01 0.013198
\n2023-03-01 0.016380
\n2023-04-01 0.019872
\n2023-05-01 0.022822
\n2023-06-01 -1.000000
\nName: mbd_chg, dtype: float64
Similar to my long trend prediction, though my may numbers might be a bit aggressive. My seasonal is very aggressive, but I figured why not in case things change dramatically. Also similar to the results I pulled from someone elses prediction.
", "votes": 0, "replies": [] }, { "id": 2186005, "author_name": "Sireesh Limbu", "author_url": "", "post_date": "2023-03-17T13:40:57.070000", "content": "Ah quite similar to mine as well! :)
", "votes": 1, "replies": [] }, { "id": 2186694, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-18T03:00:58.810000", "content": "\nCan you share them? Did you use fairly simple models?
", "votes": 0, "replies": [] }, { "id": 2186696, "author_name": "Sireesh Limbu", "author_url": "", "post_date": "2023-03-18T03:07:08.267000", "content": "I've already shared my mbd_change in this thread :)
\nI used a blend of 2 catboost models along with Last Value blacklisting I have explained briefly here
My Positive scenario submission
\n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nMy Negative scenario submission
\n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nAh. There's the famous May drop. Looks great! Good luck!
", "votes": 1, "replies": [ { "id": 2189167, "author_name": "CPMP", "author_url": "", "post_date": "2023-03-20T09:58:45.930000", "content": "I have a drop for May too, see my numbers. It surprises me. Why would the drop be good?
", "votes": 0, "replies": [] }, { "id": 2189319, "author_name": "Sireesh Limbu", "author_url": "", "post_date": "2023-03-20T11:59:42.643000", "content": "Hi again. I think a lot of people with good CV scores were getting this drop, I was also a bit worried, but in the end I just went with what I got (And I don't have this drop). Here is the mention of the future trends. I think it has to do with seasonality that your models could grasp on. I am not 100% sure though.
", "votes": 0, "replies": [] }, { "id": 2192241, "author_name": "Vitaly Kudelya", "author_url": "", "post_date": "2023-03-22T14:34:09.267000", "content": "As I remember the drop in May happened in 2022 and 2021, may be it somehow related to tax declaration in April in USA or any other seasonal reason (or just coincidence).
\nFor me it seems that in May 2023 we will see drop for more than 50% probability )
Do you have a last_value group and a model prediction group? the result above is the average? Or you predict all cfips?
", "votes": 0, "replies": [ { "id": 2185336, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-17T01:03:19.807000", "content": "I predict all cfips. Manual blacklisting didn't seem to improve my scores, but I also had features which were similar in purpose, such as whether there was a change over some threshold in the last month(s). Rounding on active of course helps as well.
", "votes": 0, "replies": [ { "id": 2185339, "author_name": "Hugo", "author_url": "", "post_date": "2023-03-17T01:12:20.653000", "content": "thanks.
\nFor the XGB model I built, if I only calculate the prediction part,
I use the adjusted mbd. i think it is equal to use active.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2184087, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-16T06:16:20.820000", "content": "I pulled this from another contestants submission that they shared publicly. I won't give any names, but I found it interesting
\n-03-01 \n-04-01 \n-05-01 \nNew: I've uploaded the formal model submit file that was required to receive the prize distribution: https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/425000
\nCode here - https://www.kaggle.com/kaggleqrdl/first-place-code, see below for an explanation.
\nAcknowledgements
\nLet me first acknowledge Kaggle, the Venture Forward folks, and GoDaddy. Not just for creating Kaggle itself, hosting the contest and providing the microbiz data - incredible feats of faith, truly - but also for being patient with my fixation on transparency. I am an egalitarian thru and thru, and firmly believe transparency is what gets us there.
\nIntroduction
\nThe contest was to predict microbusiness density 3,4,5 months lookahead for mar/apr/may 2023 per each of the 3135 US counties in the dataset. Microbusiness density here is roughly equal to the # of GoDaddy internet domains registered / adult population in a county as measured by a 2 year lagging US census.
\nEg: if a total of 10 GoDaddy domains were registered in De Kalb, MO with 200 adult population according to the 2021 census in 2023 February, than microbusiness density for that county would increase by 10/200 or 0.05 in that month.
\nWe were given historical microbiz data for 2019 August -> 2022 December to train with, and a not-so-secret 'public LB' of data for January to validate against. Scores on the public LB were provided by Kaggle, and we were given 5 submissions per day.
\nMethod
\nTo win this, I took a baseline prediction for the january public leaderboard based on overfitting it, and used linear regression to project into mar/apr/may
, [, , ,] - month lookahead\n, [, , ,] - month lookahead\n, [, , , , ] - month lookahead\nI selected these features by cross validating on the last possible window (2,3,4 month respectively) without overlap.
\nInsights
\nThe key ideas that seemed to help:
*every regression model in sklearn, various arima libraries (a lot of time here, wasted I feel), hierarchical, xgb, catboost, lgbm, pytorch/dll. I tried both manually tuning the previous as well as various auto approaches like grid search/optuna. Lots of very exciting results, but eventually it occurred to me I was probably just overfitting.
\nIt's worth observing that my lead against the #2 and #3 spots increased significantly from the public LB versus the private LB. I believe this was due to superior predictions for apr/may as my march results were somewhat lackluster. My lead over the top 20 spots also increased, though it is less clear how much of the public LB influenced their models.
\nIdeas that didn't seem as important
\nI also tried a lot of different datasets (including all the ones I shared publicly) up to and including google keyword trends for 'godaddy'. :) The above worked best when combined with early stopping. There would be sparks of good CV scores when using other features, but adding them frequently felt like overfitting.
\nI briefly looked into using zone files and whois databases, but these were not openly available so of course I didn't use them.
\nCrawling/Scraping would have in theory been public data, but I had committed to myself not to use any data that I didn't share openly, so I figured it wouldn't be worth the effort required.
\nEverything I used I shared in the dataset I made publically available at the beginning of the comp.
\nI did a lot of initial work around per county models, work I eventually decided was wasted. There was barely enough data to validate the models and features above. There might be a way of segregate the counties, the question is - Would you have to segregate your training data as well? Would the increased complexity be worth what might potentially be only incremental improvements? Hard to say, especially since observing incremental improvements is mostly impossible when your CV scores are so volatile.
\nFuture directions
\nOne thing I spent time looking at was correlation between the counties. There is an argument to be made I think that different counties may follow similar trends at potential lag times. There might be more to do here, but it would need to be done globally rather than individually, and your models would require appropriate data sufficiency.
Another feature I didn't plumb carefully enough was 1/2/3/etc annual renewal on GoDaddy domains, an idea I got by chatting with some domain resellers recently on a reseller domain community forum. I suspect they would provide a well of ideas around this that could be leveraged more.
\nFinal thoughts
\nAs I mentioned above, and others have mentioned, the data was non stationary except for some global linear growth coefficients. It's also been said that the smaller counties added a lot of noise, but I believe it was the right choice to try to see if we could do something there. There is a paucity of stats and analysis around more rural areas and emergent issues are hard to detect because of it.
\nAnother way of looking at the data might be volatility instead of direction. The wild fluctuations in active domains could just be due to data errors but they could also be due to underlying social and economic factors. In the latter case, we need to consider: could the volatility be an opportunity, like the first sparks of a fire? Could these counties benefit from more targeted and opportune investment? If volatile counties have increased potential, what are the triggers for volatility? These are questions and experiments (signal + intervention) worth considering
\nIf the VF folks would like to contact me (heh), feel free to reach out. As you may have noticed, I could talk about this stuff all day long with folks who are equally interested.
\n..
\nThere was a bug in the team update that didn't let me change the LB pointer which I had originally pointed to the message below, so you're stuck with both. I originally pointed it here as I was more interested in what other folks did as I wasn't really that optimistic about my simplistic approach above. In retrospect, I probably should have trusted the process of elimination that I went through.
\nprevious thread title: What is your mean active change over december for mar/apr/may?
\nEdit - to make clear up front, you can share your results from your submission by just executing this code. It uses microbusiness density and not active, but that should be fine for comparison.
\ndfc = pd.read_csv()\nrt = pd.read_csv().set_index()\ndfc[] = dfc[].apply( x:x[-:])\ndfc[] = dfc[].apply( x:(x[:x.index()]))\ndfc = dfc.reset_index().set_index()\ndfc[] = dfc.apply( r:(r[] - rt.loc[, ])/rt.loc[, ], axis = )\ndisplay(dfc.groupby().mean()[])\nβ¦.
\nIt's going to be awhile before we get final results, and I think May results might play a large role in determining the eventual winners. While we wait, it might fun to share what we actually predicted. We could compare our actual results, which would be good and I've uploaded my two here - but for those who don't want to and perhaps simpler might just be sharing mean % change over active.
\nHere's the code I used. It requires having accurate values for december in your dataframe and also have calculated 'active' as one of your columns.
\nIf you don't want to use active, you can also share microbusiness density as well, but I found it less interesting because of the pop shift. There were some outliers in the pop shift that made it harder to ensure that nothing went awry, so I liked having active available. I've shared both below.
\nYou can calculate cfips/first day with this code:
\ndfc[] = dfc[].apply( x:x[-:])\ndfc[] = dfc[].apply( x:(x[:x.index()]))\nCalculate active_chg values with this code:
\ndfc = dfc.reset_index().set_index()\ndfc[] = dfc.apply( r:(r[] - dfc.loc[, ])/dfc.loc[, ], axis = )\ndfc.groupby().mean()[]\nThis is my seasonal model
\nfirst_day_of_month\n--01 - \n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nName: active_chg, dtype: float64\nThis is my long trend
\nfirst_day_of_month\n--01 -\n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nName: active_chg, dtype: float64\nUsing mbd code if you didn't have an active column:
\ndfc = dfc.reset_index().set_index()\ndfc[] = dfc.apply( r:(r[] - dfc.loc[, ])/dfc.loc[, ], axis = )\ndfc.groupby().mean()[]\nSeasonal sub
\nfirst_day_of_month\n--01 -\n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nName: mbd_chg, dtype: float64\nLong trend sub
\nfirst_day_of_month\n--01 -\n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nName: mbd_chg, dtype: float64\nNote for february I just used the january values. Sharing january isn't really that helpful of course, I just do it for fun here.
", "rawMarkdown": "New: I've uploaded the formal model submit file that was required to receive the prize distribution: https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/425000\n\nCode here - https://www.kaggle.com/kaggleqrdl/first-place-code, see below for an explanation.\n\n**Acknowledgements**\n\nLet me first acknowledge Kaggle, the Venture Forward folks, and GoDaddy. Not just for creating Kaggle itself, hosting the contest and providing the microbiz data - incredible feats of faith, truly - but also for being patient with my fixation on transparency. I am an egalitarian thru and thru, and firmly believe transparency is what gets us there.\n\n**Introduction**\n\nThe contest was to predict microbusiness density 3,4,5 months lookahead for mar/apr/may 2023 per each of the 3135 US counties in the dataset. Microbusiness density here is roughly equal to the # of GoDaddy [internet domains registered](https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/374293) / adult population in a county as measured by a 2 year lagging US census. \n\nEg: if a total of 10 GoDaddy domains were registered in [De Kalb, MO](https://en.wikipedia.org/wiki/De_Kalb,_Missouri) with 200 adult population according to the 2021 census in 2023 February, than microbusiness density for that county would increase by 10/200 or 0.05 in that month.\n\nWe were given historical microbiz data for 2019 August -> 2022 December to train with, and a not-so-secret 'public LB' of data for January to validate against. Scores on the public LB were provided by Kaggle, and we were given 5 submissions per day.\n\n**Method**\nTo win this, I took a baseline prediction for the january public leaderboard based on overfitting it, and used linear regression to project into mar/apr/may\n\n```python\n'sklearn.linear_model.LinearRegression', ['pct_college_2021', 'l11_uemp', 'shiftf1_l1_active',] - 4 month lookahead\n'sklearn.linear_model.LinearRegression', ['shiftf1_l1_active_logit', 'pct_college_2019', 'l4_active','l10_lf'] - 3 month lookahead\n'sklearn.linear_model.LinearRegression', ['l8_uemp', 'l1_ur', 'l11_uemp', 'l10_ur', 'pct_bb_2017'] - 2 month lookahead\n```\n\n1. l# here means lag # of months, shift is the actual value, just l#_ is pct_change\n2. shiftf1_l1_active_logit is whether there was a change in active over previous month (l1_active != 1)\n2. pct_college_2021 / pct_bb_2017 is from the census data the hosters shared\n3. uemp / lf / ur are from the dataset I shared here - https://www.kaggle.com/datasets/kaggleqrdl/gd2022datasets\n\nI selected these features by cross validating on the last possible window (2,3,4 month respectively) without overlap. \n\n**Insights**\nThe key ideas that seemed to help:\n- LB overfitting / probing. I did this by using the top public LB notebook plus focusing on the 10 or so counties that had recent significant volatility in the dataset. This was done because in non stationary time series data, the last value is your most powerful and compelling signal. It's worth noting that due to the [dec22/jan23](https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/413974) snafu (we were given 60% incorrect data, as measured by SMAPE), this technique was severely impaired and my lead in the public LB went from ~0.2 to ~0.08 over the top 20. Overfitting the LB doesn't work very well when the LB has wrong data. \nMore explanation here on this approach - https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/413109 I also explained these ideas early on in the contest [here](https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/375244) and [here](https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/378002). The [notebook](https://www.kaggle.com/kaggleqrdl/first-place-code) has code used for SMAPE root solving and probing techniques\n- LinearRegression - I tried practically every(*) other model and techniques, even dl/nn. LR consistently resulted in more reliable CV scores. Overfitting is always a concern with time series data and it's so easy to tune other models and convince yourself you found something superior.\n- Early stopping on greedy feature selection. This was done mostly on an intuition basis, but after running 1000s of experiments I developed a good feel for this I believe. I noticed overfitting occurring when I did more than 4 or 5 features, plus the decreased rate of improvement was another signal that overfitting had occurred. Certain features as well seemed to have more signal than others (eg, L10/11 employment, bb census) so I would keep going until they made it into the list but not much more.\n- Last window CV for feature selection. My hypothesis is that as a non stationary time series the relevant predictive features change and it's best to use the latest CV window (last few months of 2022). This played out in the results I saw over the experiments I did, but I will say std deviation on the error term was quite high, even if the mean was superior to longer CV periods. The risk was that so little CV time made it a bit of a gamble.\n- Use as much original training data as possible - Generalizing through data sufficiency was a reoccurring theme in this comp, and any technique that didn't appreciate it was degraded respectively. I used everything up to the lag window as per above. I didn't modify any of the data (beyond removing 3 problematic counties), even left the break in 2021 alone. I tried doing all sorts of things to clean up that data, but they just seemed to degrade results. \n- rounding based on active/population of course was a must. I found clipping, in the end, didn't help.\n\n*every regression model in sklearn, various arima libraries (a lot of time here, wasted I feel), hierarchical, xgb, catboost, lgbm, pytorch/dll. I tried both manually tuning the previous as well as various auto approaches like grid search/optuna. Lots of very exciting results, but eventually it occurred to me I was probably just overfitting.\n\nIt's worth observing that my lead against the #2 and #3 spots increased significantly from the public LB versus the private LB. I believe this was due to superior predictions for apr/may as my march results were somewhat lackluster. My lead over the top 20 spots also increased, though it is less clear how much of the public LB influenced their models.\n\n**Ideas that didn't seem as important**\n\nI also tried a lot of different datasets (including all the ones I shared publicly) up to and including google keyword trends for 'godaddy'. :) The above worked best when combined with early stopping. There would be sparks of good CV scores when using other features, but adding them frequently felt like overfitting. \n\nI briefly looked into using zone files and whois databases, but these were not openly available so of course I didn't use them. \n\nCrawling/Scraping would have in theory been public data, but I had committed to myself not to use any data that I didn't share openly, so I figured it wouldn't be worth the effort required. \n\nEverything I used I shared in the [dataset]( https://www.kaggle.com/datasets/kaggleqrdl/gd2022datasets) I made publically available at the beginning of the comp.\n\nI did a lot of initial work around per county models, work I eventually decided was wasted. There was barely enough data to validate the models and features above. There might be a way of segregate the counties, the question is - Would you have to segregate your training data as well? Would the increased complexity be worth what might potentially be only incremental improvements? Hard to say, especially since observing incremental improvements is mostly impossible when your CV scores are so volatile. \n\n\n**Future directions**\nOne thing I spent time looking at was correlation between the counties. There is an argument to be made I think that different counties may follow similar trends at potential lag times. There might be more to do here, but it would need to be done globally rather than individually, and your models would require appropriate data sufficiency. \n\nAnother feature I didn't plumb carefully enough was 1/2/3/etc annual renewal on GoDaddy domains, an idea I got by chatting with some domain resellers recently on a reseller domain community forum. I suspect they would provide a well of ideas around this that could be leveraged more.\n\n**Final thoughts**\n\nAs I mentioned above, and others have mentioned, the data was non stationary except for some global linear growth coefficients. It's also been said that the smaller counties added a lot of noise, but I believe it was the right choice to try to see if we could do something there. There is a paucity of stats and analysis around more rural areas and emergent issues are hard to detect because of it.\n\nAnother way of looking at the data might be volatility instead of direction. The wild fluctuations in active domains could just be due to data errors but they could also be due to underlying social and economic factors. In the latter case, we need to consider: could the volatility be an opportunity, like the first sparks of a fire? Could these counties benefit from more targeted and opportune investment? If volatile counties have increased potential, what are the triggers for volatility? These are questions and experiments (signal + intervention) worth considering\n\nIf the VF folks would like to contact me (heh), feel free to reach out. As you may have noticed, I could talk about this stuff all day long with folks who are equally interested.\n\n..\n\nThere was a bug in the team update that didn't let me change the LB pointer which I had originally pointed to the message below, so you're stuck with both. I originally pointed it here as I was more interested in what other folks did as I wasn't really that optimistic about my simplistic approach above. In retrospect, I probably should have trusted the process of elimination that I went through.\n\n\nprevious thread title: **What is your mean active change over december for mar/apr/may?**\n\nEdit - to make clear up front, you can share your results from your submission by just executing this code. It uses microbusiness density and not active, but that should be fine for comparison.\n\n```python\ndfc = pd.read_csv(\"submission.csv\")\nrt = pd.read_csv(\"revealed_test.csv\").set_index(\"row_id\")\ndfc['first_day_of_month'] = dfc['row_id'].apply(lambda x:x[-10:])\ndfc['cfips'] = dfc['row_id'].apply(lambda x:int(x[:x.index(\"_\")]))\ndfc = dfc.reset_index().set_index(\"row_id\")\ndfc['mbd_chg'] = dfc.apply(lambda r:(r['microbusiness_density'] - rt.loc[f\"{r['cfips']}_2022-12-01\", 'microbusiness_density'])/rt.loc[f\"{r['cfips']}_2022-12-01\", 'microbusiness_density'], axis = 1)\ndisplay(dfc.groupby(\"first_day_of_month\").mean()['mbd_chg'])\n```\n\n....\n\nIt's going to be awhile before we get final results, and I think May results might play a large role in determining the eventual winners. While we wait, it might fun to share what we actually predicted. We could compare our actual results, which would be good and I've uploaded my two [here](https://www.kaggle.com/datasets/kaggleqrdl/gd2022-subs) - but for those who don't want to and perhaps simpler might just be sharing mean % change over active.\n\nHere's the code I used. It requires having accurate values for december in your dataframe and also have calculated 'active' as one of your columns. \n\nIf you don't want to use active, you can also share microbusiness density as well, but I found it less interesting because of the pop shift. There were some outliers in the pop shift that made it harder to ensure that nothing went awry, so I liked having active available. I've shared both below.\n\nYou can calculate cfips/first day with this code:\n\n```python\ndfc['first_day_of_month'] = dfc['row_id'].apply(lambda x:x[-10:])\ndfc['cfips'] = dfc['row_id'].apply(lambda x:int(x[:x.index(\"_\")]))\n```\n\nCalculate active_chg values with this code:\n\n```python\ndfc = dfc.reset_index().set_index(\"row_id\")\ndfc['active_chg'] = dfc.apply(lambda r:(r['active'] - dfc.loc[f\"{r['cfips']}_2022-12-01\", 'active'])/dfc.loc[f\"{r['cfips']}_2022-12-01\", 'active'], axis = 1)\ndfc.groupby(\"first_day_of_month\").mean()['active_chg']\n```\n\nThis is my seasonal model \n\n```python\nfirst_day_of_month\n2022-11-01 -0.00635502 \n2022-12-01 0.00000000\n2023-01-01 0.00328686\n2023-02-01 0.00328686\n2023-03-01 0.01505438\n2023-04-01 0.02265933\n2023-05-01 0.02476927\nName: active_chg, dtype: float64\n```\n\nThis is my long trend\n```python\nfirst_day_of_month\n2022-11-01 -0.00635502\n2022-12-01 0.00000000\n2023-01-01 0.00328686\n2023-02-01 0.00328686\n2023-03-01 0.00881651\n2023-04-01 0.01215006\n2023-05-01 0.01580625\nName: active_chg, dtype: float64\n```\n\nUsing mbd code if you didn't have an active column:\n\n```python\ndfc = dfc.reset_index().set_index(\"row_id\")\ndfc['mbd_chg'] = dfc.apply(lambda r:(r['microbusiness_density'] - dfc.loc[f\"{r['cfips']}_2022-12-01\", 'microbusiness_density'])/dfc.loc[f\"{r['cfips']}_2022-12-01\", 'microbusiness_density'], axis = 1)\ndfc.groupby(\"first_day_of_month\").mean()['mbd_chg']\n```\n\nSeasonal sub\n```python\nfirst_day_of_month\n2022-11-01 -0.00635502\n2022-12-01 0.00000000\n2023-01-01 0.01208731\n2023-02-01 0.01208731\n2023-03-01 0.02388208\n2023-04-01 0.03161783\n2023-05-01 0.03375395\nName: mbd_chg, dtype: float64\n```\n\nLong trend sub\n\n```python\nfirst_day_of_month\n2022-11-01 -0.00635502\n2022-12-01 0.00000000\n2023-01-01 0.01208731\n2023-02-01 0.01208731\n2023-03-01 0.01764007\n2023-04-01 0.02099663\n2023-05-01 0.02467693\nName: mbd_chg, dtype: float64\n```\n\nNote for february I just used the january values. Sharing january isn't really that helpful of course, I just do it for fun here.\n", "votes": 48 }, { "id": 2195701, "postDate": "2023-03-24T20:30:58.880Z", "content": "Here are the 3 history sequences for \"Dec, Jan, Feb, Mar, Apr, May\" in the train data. After we adjust for census population changes. It appears that May decreases in 2022 and 2021. And May increases in 2020. What will May do in 2023??
\n
In 2020 it was short term big drop and then huge economic recovery from COVID, so imo it was not general trend, but anomaly.
\nI didn't apply and seasonality/trend , just used one value for all 3 month.
Values from two models:
\n
For folks looking for a description of my model, I posted it here - https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/394810#2182222
\nHowever, my experience was similar to most folks - simple linear models were best, perhaps informed a bit by economic/census features, and there wasn't much to write about. The CV studies I did showed that using last split CV was frequently superior (though more volatile for sure) to using more extensive validation.
", "rawMarkdown": "For folks looking for a description of my model, I posted it here - https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/394810#2182222 \n\nHowever, my experience was similar to most folks - simple linear models were best, perhaps informed a bit by economic/census features, and there wasn't much to write about. The CV studies I did showed that using last split CV was frequently superior (though more volatile for sure) to using more extensive validation.", "votes": 3 }, { "id": 2186232, "postDate": "2023-03-17T16:55:39.503Z", "content": "Hi @kaggleqrdl since you worked on the trend and seasonal model, any reason why you went for a far more aggressive growth? Do you feel that we will get to see such a significant uptick in the coming months? From your mbd_chg data above, it seems your month over month growth is a factor of ~1.0048x (long trend sub). According to Chris calculations here, the m-o-m growth observed is around ~1.0032x.
Well, I let it run a little wild because of the layoffs in the tech sector. I sort of felt those folks might try to start up microbiz. Historically the tech sector has seen full employment, but lately less so, so I think using historical values we might be seeing some underestimations.
\nIt's all guesses and fun though. I suspect we'll see a lot of new folk getting gold here
", "rawMarkdown": "Well, I let it run a little wild because of the layoffs in the tech sector. I sort of felt those folks might try to start up microbiz. Historically the tech sector has seen full employment, but lately less so, so I think using historical values we might be seeing some underestimations.\n\nIt's all guesses and fun though. I suspect we'll see a lot of new folk getting gold here", "votes": 1, "replies": [ { "id": 2186693, "postDate": "2023-03-18T03:00:17.070Z", "content": "I also thought the same last minute haha inspired from @uspangaliev's post here
", "rawMarkdown": "I also thought the same last minute haha inspired from @uspangaliev's post [here](https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/392971)" } ] } ] }, { "id": 2185985, "postDate": "2023-03-17T13:25:43.397Z", "content": "Share my 2 submissions:
\nfirst_day_of_month\n2023-03-01 0.010246\n2023-04-01 0.012392\n2023-05-01 0.017704\nfirst_day_of_month\n2023-03-01 0.009916\n2023-04-01 0.011304\n2023-05-01 0.015852\nHi! Nice that you came up with this! Fun! Following are mine:
\n
The top one is with some multipliers for Mar, Apr and May along with a small functionality for increasing/decreasing trend for the last 5 months. The bottom one is the plain models with LastValue Blacklisting.
", "rawMarkdown": "Hi! Nice that you came up with this! Fun! Following are mine:\n\n\n\n\nThe top one is with some multipliers for Mar, Apr and May along with a small functionality for increasing/decreasing trend for the last 5 months. The bottom one is the plain models with LastValue Blacklisting.\n", "votes": 1 }, { "id": 2185793, "postDate": "2023-03-17T09:41:32.537Z", "content": "mbd_chg with respect to December:
first_day_of_month\n2022-11-01 -0.006355\n2022-12-01 0.000000\n2023-01-01 0.008735\n2023-02-01 0.007559\n2023-03-01 0.011393\n2023-04-01 0.013531\n2023-05-01 0.014565\n2023-06-01 -0.537519\nName: mbd_chg, dtype: float64\nUsing @cdeotte mbd from his last value submission and your code, I get this. Jan, Feb, and June predictions are from a single model while the others are from my blend. I did not tune January, February and June.
\nfirst_day_of_month\n--01 \n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \n-06-01 \nName: mbd_chg, dtype: float64\nWhen I evaluated my models I used the median and not the mean, as the mean is dominated by large counties, while score is dominated by the more numerous small counties.
\nMedian is:
\nfirst_day_of_month\n--01 \n--01 \n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \n-06-01 \nName: mbd_chg, dtype: float64\nCongratulations and thanks for sharing your thoughts!!
", "rawMarkdown": "Congratulations and thanks for sharing your thoughts!!", "votes": 1, "replies": [ { "id": 2184920, "postDate": "2023-03-16T17:24:12.857Z", "content": "Thanks Chen. I was hoping some folks might share their results, but so far everyone seems a bit shy. Ah well.
", "rawMarkdown": "Thanks Chen. I was hoping some folks might share their results, but so far everyone seems a bit shy. Ah well.", "replies": [ { "id": 2184928, "postDate": "2023-03-16T17:31:06.883Z", "content": "Not shy, but it requires some coding for me for instance. And there is little incentive to do so. I will try to find time.
", "rawMarkdown": "Not shy, but it requires some coding for me for instance. And there is little incentive to do so. I will try to find time.", "votes": 1 }, { "id": 2184942, "postDate": "2023-03-16T17:41:43.140Z", "content": "Actually, I did provide some code above for just using microbusiness density.
\nPerhaps I should make this more clear.
\nIf you run this, it should work.
\ndfc = pd.read_csv()\ndfc[] = dfc[].apply( x:x[-:])\ndfc[] = dfc[].apply( x:(x[:x.index()]))\ndfc = dfc.reset_index().set_index()\ndfc[] = dfc.apply( r:(r[] - dfc.loc[, ])/dfc.loc[, ], axis = )\ndisplay(dfc.groupby().mean()[])\nNote that we cannot compare different Kagglers' microbusiness density ratios because everyone can have different Dec 2022 MD predictions. For example do people use Dec 2022 as is from the train data? OR do they change it? Some people may scale it using census data. Some people may predict Dec 2022 from their model, etc etc.
\nOR if someone sets December 2022 to all zeros they will have NaN ratios!
first_day_of_month\n2022-11-01 NaN\n2022-12-01 NaN\n2023-01-01 NaN\n2023-02-01 NaN\n2023-03-01 NaN\n2023-04-01 NaN\n2023-05-01 NaN\n2023-06-01 NaN\n@kaggleqrdl What Dec 2022 predictions do you use?
", "rawMarkdown": "Note that we cannot compare different Kagglers' **microbusiness density** ratios because everyone can have different Dec 2022 MD predictions. For example do people use Dec 2022 **as is** from the train data? OR do they change it? Some people may scale it using census data. Some people may predict Dec 2022 from their model, etc etc.\n\nOR if someone sets December 2022 to all zeros they will have `NaN` ratios!\n\n first_day_of_month\n 2022-11-01 NaN\n 2022-12-01 NaN\n 2023-01-01 NaN\n 2023-02-01 NaN\n 2023-03-01 NaN\n 2023-04-01 NaN\n 2023-05-01 NaN\n 2023-06-01 NaN\n\n@kaggleqrdl What Dec 2022 predictions do you use?", "votes": 1 }, { "id": 2184970, "postDate": "2023-03-16T17:59:37.700Z", "content": "The most consistent and easiest approach to compare different Kagglers' submissions is to download \"last value baseline\" submission.csv file from notebook here. Then you can provide code to merge these predictions to each Kagglers' final submission.csv and compute ratios.
", "rawMarkdown": "The most consistent and easiest approach to compare different Kagglers' submissions is to download \"last value baseline\" submission.csv file from notebook [here][1]. Then you can provide code to merge these predictions to each Kagglers' final submission.csv and compute ratios.\n\n[1]: https://www.kaggle.com/code/cdeotte/new-last-value-baseline-1-4631", "votes": 2 }, { "id": 2184978, "postDate": "2023-03-16T18:05:59.240Z", "content": "\n\nActually, I did provide some code above for just using microbusiness density.
\n
Sure, but your asked for active change. If i use mbd from revealed test, then the change is inflated by the population change between December and January. It means I have to use what chris says, an upadated mbd.
\nOr I have to compute the change using active count, which requires some coding.
", "rawMarkdown": "> Actually, I did provide some code above for just using microbusiness density. \n\nSure, but your asked for active change. If i use mbd from revealed test, then the change is inflated by the population change between December and January. It means I have to use what chris says, an upadated mbd.\n\nOr I have to compute the change using active count, which requires some coding.", "votes": 1 }, { "id": 2184994, "postDate": "2023-03-16T18:27:42.923Z", "content": "Yes, it does require having accurate december values. The pop shift problem should be OK as we'll all have it for comparison.
", "rawMarkdown": "Yes, it does require having accurate december values. The pop shift problem should be OK as we'll all have it for comparison." }, { "id": 2185004, "postDate": "2023-03-16T18:33:21.120Z", "content": "You can use this, I'll add it above
\nrt = pd.read_csv().set_index()\ndfc[] = dfc.apply( r:(r[] - rt.loc[, ])/rt.loc[, ], axis = )\nHi! thanks for sharing your thoughts and good luck π
", "rawMarkdown": "Hi! thanks for sharing your thoughts and good luck π", "votes": 1, "replies": [ { "id": 2184922, "postDate": "2023-03-16T17:25:01.563Z", "content": "Thanks Luca, good luck to you as well.
", "rawMarkdown": "Thanks Luca, good luck to you as well." } ] }, { "id": 2184075, "postDate": "2023-03-16T05:58:31.843Z", "content": "An interesting question, I just checked your submission files. Why you omit the big changes of some counties? Any reason, I feel a little confusing, because it seems a little bit dangerous.
\nFor instance:
\nDo you use moving average? Or just assume they will drop back?
", "rawMarkdown": "An interesting question, I just checked your submission files. Why you omit the big changes of some counties? Any reason, I feel a little confusing, because it seems a little bit dangerous.\n\nFor instance: \n- train.loc[cfips==39127, time==2022-12-01]: microbusiness_density=2.856936\n- your_sub.loc[cfips==39127, time==[03,04,05]]: microbusiness_density=1.293628,1.297313,1.300999 γ\n\nDo you use moving average? Or just assume they will drop back?", "votes": 1, "replies": [ { "id": 2184076, "postDate": "2023-03-16T05:59:38.317Z", "content": "I'll answer if you share your averages ;p @judith007
", "rawMarkdown": "I'll answer if you share your averages ;p @judith007 " } ] }, { "id": 2183896, "postDate": "2023-03-16T02:32:31.303Z", "content": "Wow, congrats!
", "rawMarkdown": "Wow, congrats!", "votes": 1, "replies": [ { "id": 2183912, "postDate": "2023-03-16T03:05:08.847Z", "content": "Just to be clear, the public leaderboard here is nonsense as it frequently is in most Kaggle contests. I enjoy overfitting them to show why that is the case. That said, there may be some advantage to leveraging january last values as I tried to do, but that remains to be seen.
\nThere will very likely be fairly massive shakeup (shake down) here and I fully expect to be among the latter group.
", "rawMarkdown": "Just to be clear, the public leaderboard here is nonsense as it frequently is in most Kaggle contests. I enjoy overfitting them to show why that is the case. That said, there may be some advantage to leveraging january last values as I tried to do, but that remains to be seen. \n\nThere will very likely be fairly massive shakeup (shake down) here and I fully expect to be among the latter group.", "votes": 2 } ] }, { "id": 2185003, "postDate": "2023-03-16T18:31:34.337Z", "content": "Given that the naive model was hard to beat, I was conservative about growth and damped further horizons. These are my average rates for active with respect to the actuals as of 2022-12-01.
ββββββββββββββ¬βββββββββββ\nβ date β mean β\nββββββββββββββͺβββββββββββ‘\nβ 2023-01-01 β 0.003256 β\nβ 2023-02-01 β 0.005212 β\nβ 2023-03-01 β 0.006449 β\nβ 2023-04-01 β 0.007289 β\nβ 2023-05-01 β 0.00794 β\nβ 2023-06-01 β 0.008466 β\nββββββββββββββ΄βββββββββββ\nThanks Nic! Good luck on your results.
", "rawMarkdown": "Thanks Nic! Good luck on your results." } ] }, { "id": 2185262, "postDate": "2023-03-16T22:55:17.237Z", "content": "My results:
\n
:( Not very kaggly, I think.
\nDid you execute this code? Getting the mbd chg over data from revealed test should be fairly straightforward.
\ndfc = pd.read_csv()\nrt = pd.read_csv().set_index()\ndfc[] = dfc[].apply( x:x[-:])\ndfc[] = dfc[].apply( x:(x[:x.index()]))\ndfc = dfc.reset_index().set_index()\ndfc[] = dfc.apply( r:(r[] - rt.loc[, ])/rt.loc[, ], axis = )\ndisplay(dfc.groupby().mean()[])\nI admit to being a bit surprised that most folks hadn't already done this sort of EDA. The contest will likely be won by whoever called the macro trend correct, which isn't always in a straight line.
", "rawMarkdown": ":( Not very kaggly, I think.\n\nDid you execute this code? Getting the mbd chg over data from revealed test should be fairly straightforward. \n\n\n\n```python\ndfc = pd.read_csv(\"submission.csv\")\nrt = pd.read_csv(\"revealed_test.csv\").set_index(\"row_id\")\ndfc['first_day_of_month'] = dfc['row_id'].apply(lambda x:x[-10:])\ndfc['cfips'] = dfc['row_id'].apply(lambda x:int(x[:x.index(\"_\")]))\ndfc = dfc.reset_index().set_index(\"row_id\")\ndfc['mbd_chg'] = dfc.apply(lambda r:(r['microbusiness_density'] - rt.loc[f\"{r['cfips']}_2022-12-01\", 'microbusiness_density'])/rt.loc[f\"{r['cfips']}_2022-12-01\", 'microbusiness_density'], axis = 1)\ndisplay(dfc.groupby(\"first_day_of_month\").mean()['mbd_chg'])\n```\n\nI admit to being a bit surprised that most folks hadn't already done this sort of EDA. The contest will likely be won by whoever called the macro trend correct, which isn't always in a straight line.", "votes": 1, "replies": [ { "id": 2185369, "postDate": "2023-03-17T02:22:04.457Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2185377, "postDate": "2023-03-17T02:26:04.647Z", "content": "I was running
\ndfc = dfc.reset_index().set_index(\"row_id\")\ndfc['mbd_chg'] = dfc.apply(lambda r:(r['microbusiness_density'] - dfc.loc[f\"{r['cfips']}_2022-12-01\", 'microbusiness_density'])/dfc.loc[f\"{r['cfips']}_2022-12-01\", 'microbusiness_density'], axis = 1)\ndfc.groupby(\"first_day_of_month\").mean()['mbd_chg']\nwhich is why I got all Nans. When I run your code, I get this:
\nfirst_day_of_month
\n2022-11-01 -1.000000
\n2022-12-01 -1.000000
\n2023-01-01 0.010812
\n2023-02-01 0.013198
\n2023-03-01 0.016380
\n2023-04-01 0.019872
\n2023-05-01 0.022822
\n2023-06-01 -1.000000
\nName: mbd_chg, dtype: float64
Similar to my long trend prediction, though my may numbers might be a bit aggressive. My seasonal is very aggressive, but I figured why not in case things change dramatically. Also similar to the results I pulled from someone elses prediction.
", "rawMarkdown": "Similar to my long trend prediction, though my may numbers might be a bit aggressive. My seasonal is very aggressive, but I figured why not in case things change dramatically. Also similar to the results I pulled from someone elses prediction." }, { "id": 2186005, "postDate": "2023-03-17T13:40:57.070Z", "content": "Ah quite similar to mine as well! :)
", "rawMarkdown": "Ah quite similar to mine as well! :)", "votes": 1 }, { "id": 2186694, "postDate": "2023-03-18T03:00:58.810Z", "content": "\nCan you share them? Did you use fairly simple models?
", "rawMarkdown": "@sireesh\n\nCan you share them? Did you use fairly simple models?" }, { "id": 2186696, "postDate": "2023-03-18T03:07:08.267Z", "content": "I've already shared my mbd_change in this thread :)
\nI used a blend of 2 catboost models along with Last Value blacklisting I have explained briefly here
My Positive scenario submission
\n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nMy Negative scenario submission
\n-01-01 \n-02-01 \n-03-01 \n-04-01 \n-05-01 \nAh. There's the famous May drop. Looks great! Good luck!
", "rawMarkdown": "Ah. There's the famous May drop. Looks great! Good luck!", "votes": 1, "replies": [ { "id": 2189167, "postDate": "2023-03-20T09:58:45.930Z", "content": "I have a drop for May too, see my numbers. It surprises me. Why would the drop be good?
", "rawMarkdown": "I have a drop for May too, see my numbers. It surprises me. Why would the drop be good?" }, { "id": 2189319, "postDate": "2023-03-20T11:59:42.643Z", "content": "Hi again. I think a lot of people with good CV scores were getting this drop, I was also a bit worried, but in the end I just went with what I got (And I don't have this drop). Here is the mention of the future trends. I think it has to do with seasonality that your models could grasp on. I am not 100% sure though.
", "rawMarkdown": "Hi again. I think a lot of people with good CV scores were getting this drop, I was also a bit worried, but in the end I just went with what I got (And I don't have this drop). [Here](https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting/discussion/392417) is the mention of the future trends. I think it has to do with seasonality that your models could grasp on. I am not 100% sure though." }, { "id": 2192241, "postDate": "2023-03-22T14:34:09.267Z", "content": "As I remember the drop in May happened in 2022 and 2021, may be it somehow related to tax declaration in April in USA or any other seasonal reason (or just coincidence).
\nFor me it seems that in May 2023 we will see drop for more than 50% probability )
Do you have a last_value group and a model prediction group? the result above is the average? Or you predict all cfips?
", "rawMarkdown": "Do you have a last_value group and a model prediction group? the result above is the average? Or you predict all cfips?", "replies": [ { "id": 2185336, "postDate": "2023-03-17T01:03:19.807Z", "content": "I predict all cfips. Manual blacklisting didn't seem to improve my scores, but I also had features which were similar in purpose, such as whether there was a change over some threshold in the last month(s). Rounding on active of course helps as well.
", "rawMarkdown": "I predict all cfips. Manual blacklisting didn't seem to improve my scores, but I also had features which were similar in purpose, such as whether there was a change over some threshold in the last month(s). Rounding on active of course helps as well.", "replies": [ { "id": 2185339, "postDate": "2023-03-17T01:12:20.653Z", "content": "thanks.
\nFor the XGB model I built, if I only calculate the prediction part,
I use the adjusted mbd. i think it is equal to use active.
", "rawMarkdown": "thanks. \nFor the XGB model I built, if I only calculate the prediction part, \n- March: 1.01157\n- April: 1.01702\n- May: 1.02149\nBut if average over the prediction part(60%) and the last_value part(40%), then\n- March: ~1.007\n- April: ~1.009\n- May: ~1.011\n\nI use the adjusted mbd. i think it is equal to use active.", "votes": 1 } ] } ] }, { "id": 2184087, "postDate": "2023-03-16T06:16:20.820Z", "content": "I pulled this from another contestants submission that they shared publicly. I won't give any names, but I found it interesting
\n-03-01 \n-04-01 \n-05-01 \n