Scores in the leaderboard are (0,1) range. Seen a topic of scores within .01-.05 range. What range will top 10% land at private LB? I mean your best guess.
", "messages": [ { "id": "1628910", "postDate": "12/25/2021 11:55:43", "content": "Scores in the leaderboard are (0,1) range. Seen a topic of scores within .01-.05 range. What range will top 10% land at private LB? I mean your best guess.
", "rawMarkdown": "Scores in the leaderboard are (0,1) range. Seen a topic of scores within .01-.05 range. What range will top 10% land at private LB? I mean your best guess.", "votes": null }, { "id": "1628985", "postDate": "12/25/2021 14:13:48", "content": "I think 0.04-0.05 if it is possible without lookahead bias can be considered as good score.
", "rawMarkdown": "I think 0.04-0.05 if it is possible without lookahead bias can be considered as good score.", "votes": null }, { "id": "1630126", "postDate": "12/27/2021 01:33:34", "content": "I guess 0.06 , I can get a score 0.06 for some specific assets with lgbm+optuna, so maybe in the realworld 0.06 is achievable
", "rawMarkdown": "I guess 0.06 , I can get a score 0.06 for some specific assets with lgbm+optuna, so maybe in the realworld 0.06 is achievable", "votes": null }, { "id": "1632687", "postDate": "12/29/2021 23:44:39", "content": "TLDR - private LB score will depend on market environment - high crypto volatility COULD mean higher correlation + higher private LBs and vice versa.
\nIn terms of the private LB final scores - it heavily depends on what happens in the markets during the testing period. I've seen that in certain 3 month periods within the data set i can get a validation score >0.07, and the same model in other 3 month periods won't get above 0.03.
\nOne explanation for the wide variation in these scores i believe could be the volatility of the time period. E.g. when crypto has a volatile period (see May 10th - 25th 2021). The whole market is more correlated - everything goes down together and there is fear + overreactions + more signal in the noise and more opportunity. This may be one factor that would result in higher model scores, not because of the models themselves, but because of the market.
\nYou can only measure your model scores relative to a given time period, E.g if you can improve predictions by 0.01 on your chosen 3 month time period(s) then you have a good model.
", "rawMarkdown": "TLDR - private LB score will depend on market environment - high crypto volatility COULD mean higher correlation + higher private LBs and vice versa. \n\nIn terms of the private LB final scores - it heavily depends on what happens in the markets during the testing period. I've seen that in certain 3 month periods within the data set i can get a validation score >0.07, and the same model in other 3 month periods won't get above 0.03.\n\nOne explanation for the wide variation in these scores i believe could be the volatility of the time period. E.g. when crypto has a volatile period (see May 10th - 25th 2021). The whole market is more correlated - everything goes down together and there is fear + overreactions + more signal in the noise and more opportunity. This may be one factor that would result in higher model scores, not because of the models themselves, but because of the market.\n\nYou can only measure your model scores relative to a given time period, E.g if you can improve predictions by 0.01 on your chosen 3 month time period(s) then you have a good model.", "votes": null } ], "comments": [ { "id": 1628985, "author_name": "alankabisov", "author_url": "", "post_date": "12/25/2021 14:13:48", "content": "I think 0.04-0.05 if it is possible without lookahead bias can be considered as good score.
", "votes": null, "replies": [] }, { "id": 1630126, "author_name": "cymchenzelin", "author_url": "", "post_date": "12/27/2021 01:33:34", "content": "I guess 0.06 , I can get a score 0.06 for some specific assets with lgbm+optuna, so maybe in the realworld 0.06 is achievable
", "votes": null, "replies": [] }, { "id": 1632687, "author_name": "tomforbes", "author_url": "", "post_date": "12/29/2021 23:44:39", "content": "TLDR - private LB score will depend on market environment - high crypto volatility COULD mean higher correlation + higher private LBs and vice versa.
\nIn terms of the private LB final scores - it heavily depends on what happens in the markets during the testing period. I've seen that in certain 3 month periods within the data set i can get a validation score >0.07, and the same model in other 3 month periods won't get above 0.03.
\nOne explanation for the wide variation in these scores i believe could be the volatility of the time period. E.g. when crypto has a volatile period (see May 10th - 25th 2021). The whole market is more correlated - everything goes down together and there is fear + overreactions + more signal in the noise and more opportunity. This may be one factor that would result in higher model scores, not because of the models themselves, but because of the market.
\nYou can only measure your model scores relative to a given time period, E.g if you can improve predictions by 0.01 on your chosen 3 month time period(s) then you have a good model.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1628910": "Scores in the leaderboard are (0,1) range. Seen a topic of scores within .01-.05 range. What range will top 10% land at private LB? I mean your best guess.", "1628985": "I think 0.04-0.05 if it is possible without lookahead bias can be considered as good score.", "1630126": "I guess 0.06 , I can get a score 0.06 for some specific assets with lgbm+optuna, so maybe in the realworld 0.06 is achievable", "1632687": "TLDR - private LB score will depend on market environment - high crypto volatility COULD mean higher correlation + higher private LBs and vice versa. \n\nIn terms of the private LB final scores - it heavily depends on what happens in the markets during the testing period. I've seen that in certain 3 month periods within the data set i can get a validation score >0.07, and the same model in other 3 month periods won't get above 0.03.\n\nOne explanation for the wide variation in these scores i believe could be the volatility of the time period. E.g. when crypto has a volatile period (see May 10th - 25th 2021). The whole market is more correlated - everything goes down together and there is fear + overreactions + more signal in the noise and more opportunity. This may be one factor that would result in higher model scores, not because of the models themselves, but because of the market.\n\nYou can only measure your model scores relative to a given time period, E.g if you can improve predictions by 0.01 on your chosen 3 month time period(s) then you have a good model." }, "source": "meta" }