Hey guys,
\nDid any of you notice the fact that the score for the train set (I mean after you split it into train and test, I split it using train_test_split(x, y,test_size=0.2, shuffle = True, random_state = 8)) is very different from the one that they get when they submit to the leaderboard (at first I got 90% f1-score for X_test from the train, and 62% on the leaderboard, then I tried again and got 68% f1-score for X_test from the train, and 50% on the leaderboard))?
\nAnd why do think something like this keeps on happening?
Hey guys,
\nDid any of you notice the fact that the score for the train set (I mean after you split it into train and test, I split it using train_test_split(x, y,test_size=0.2, shuffle = True, random_state = 8)) is very different from the one that they get when they submit to the leaderboard (at first I got 90% f1-score for X_test from the train, and 62% on the leaderboard, then I tried again and got 68% f1-score for X_test from the train, and 50% on the leaderboard))?
\nAnd why do think something like this keeps on happening?
yes this is leaky design, sometimes you use current time or even future time to predict stuff in the past. we need some way to avoid this leakage, but without a time feature this might be difficult. Maybe we can start by splitting the rows with same sp500 feature in different folds.
", "rawMarkdown": "yes this is leaky design, sometimes you use current time or even future time to predict stuff in the past. we need some way to avoid this leakage, but without a time feature this might be difficult. Maybe we can start by splitting the rows with same sp500 feature in different folds.", "votes": null }, { "id": "2324532", "postDate": "06/30/2023 16:52:44", "content": "that's my first time dealing with a stock-related dataset so could you please elaborate more about why would split the rows with the same sp500 feature in different folds is useful in this case?
", "rawMarkdown": "that's my first time dealing with a stock-related dataset so could you please elaborate more about why would split the rows with the same sp500 feature in different folds is useful in this case?", "votes": null }, { "id": "2325342", "postDate": "07/01/2023 09:01:16", "content": "ok in short:
\nIn market data the signal to noise ratio is really low, one big mistake that usually translate to abnormally high metrics is time leakage, that is using data from the future to predict the past. This usually mean that for evaluation we need to split data so that all event from the testing set happen after the training set.
\nHowever here we don't have time so it is a bit difficult to do the split correctly, but i noticed the sp500 feature only take a small number of values so that it is probably constant for a given time. A good start would be to split each time in different fold, it is not ideal as it doesn't ensure monotonicity of time but it should be better than nothing.
", "rawMarkdown": "ok in short:\n\nIn market data the signal to noise ratio is really low, one big mistake that usually translate to abnormally high metrics is time leakage, that is using data from the future to predict the past. This usually mean that for evaluation we need to split data so that all event from the testing set happen after the training set.\n\nHowever here we don't have time so it is a bit difficult to do the split correctly, but i noticed the sp500 feature only take a small number of values so that it is probably constant for a given time. A good start would be to split each time in different fold, it is not ideal as it doesn't ensure monotonicity of time but it should be better than nothing.", "votes": null }, { "id": "2325510", "postDate": "07/01/2023 12:24:01", "content": "ok, I get it now, Thank you very much for the explanation :)
", "rawMarkdown": "ok, I get it now, Thank you very much for the explanation :)", "votes": null }, { "id": "2344394", "postDate": "07/14/2023 13:38:30", "content": "Hey @amlhassan ! Can you share how you achieved 30/20% precision and recall for minority class. What did you do, which models used? Cost-sensivity? Resampling? Can you kindly share your working process?
", "rawMarkdown": "Hey @amlhassan ! Can you share how you achieved 30/20% precision and recall for minority class. What did you do, which models used? Cost-sensivity? Resampling? Can you kindly share your working process?", "votes": null }, { "id": "2344901", "postDate": "07/14/2023 21:49:45", "content": "Sure, I used CatBoost and I didn't do anything special to get those results, just CatBoost!
", "rawMarkdown": "Sure, I used CatBoost and I didn't do anything special to get those results, just CatBoost!", "votes": null }, { "id": "2345260", "postDate": "07/15/2023 08:30:41", "content": "Wow. You just take data, loaded it all in CatBoost and got such results? Impressive
", "rawMarkdown": "Wow. You just take data, loaded it all in CatBoost and got such results? Impressive", "votes": null } ], "comments": [ { "id": 2324515, "author_name": "lucasmorin", "author_url": "", "post_date": "06/30/2023 16:39:56", "content": "yes this is leaky design, sometimes you use current time or even future time to predict stuff in the past. we need some way to avoid this leakage, but without a time feature this might be difficult. Maybe we can start by splitting the rows with same sp500 feature in different folds.
", "votes": null, "replies": [ { "id": 2324532, "author_name": "amlhassan", "author_url": "", "post_date": "06/30/2023 16:52:44", "content": "that's my first time dealing with a stock-related dataset so could you please elaborate more about why would split the rows with the same sp500 feature in different folds is useful in this case?
", "votes": null, "replies": [ { "id": 2325342, "author_name": "lucasmorin", "author_url": "", "post_date": "07/01/2023 09:01:16", "content": "ok in short:
\nIn market data the signal to noise ratio is really low, one big mistake that usually translate to abnormally high metrics is time leakage, that is using data from the future to predict the past. This usually mean that for evaluation we need to split data so that all event from the testing set happen after the training set.
\nHowever here we don't have time so it is a bit difficult to do the split correctly, but i noticed the sp500 feature only take a small number of values so that it is probably constant for a given time. A good start would be to split each time in different fold, it is not ideal as it doesn't ensure monotonicity of time but it should be better than nothing.
", "votes": null, "replies": [ { "id": 2325510, "author_name": "amlhassan", "author_url": "", "post_date": "07/01/2023 12:24:01", "content": "ok, I get it now, Thank you very much for the explanation :)
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2344394, "author_name": "", "author_url": "", "post_date": "07/14/2023 13:38:30", "content": "Hey @amlhassan ! Can you share how you achieved 30/20% precision and recall for minority class. What did you do, which models used? Cost-sensivity? Resampling? Can you kindly share your working process?
", "votes": null, "replies": [ { "id": 2344901, "author_name": "amlhassan", "author_url": "", "post_date": "07/14/2023 21:49:45", "content": "Sure, I used CatBoost and I didn't do anything special to get those results, just CatBoost!
", "votes": null, "replies": [ { "id": 2345260, "author_name": "", "author_url": "", "post_date": "07/15/2023 08:30:41", "content": "Wow. You just take data, loaded it all in CatBoost and got such results? Impressive
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2324451": "Hey guys,\n\nDid any of you notice the fact that the score for the train set (I mean after you split it into train and test, I split it using train_test_split(x, y,test_size=0.2, shuffle = True, random_state = 8)) is very different from the one that they get when they submit to the leaderboard (at first I got 90% f1-score for X_test from the train, and 62% on the leaderboard, then I tried again and got 68% f1-score for X_test from the train, and 50% on the leaderboard))?\nAnd why do think something like this keeps on happening?", "2324515": "yes this is leaky design, sometimes you use current time or even future time to predict stuff in the past. we need some way to avoid this leakage, but without a time feature this might be difficult. Maybe we can start by splitting the rows with same sp500 feature in different folds.", "2324532": "that's my first time dealing with a stock-related dataset so could you please elaborate more about why would split the rows with the same sp500 feature in different folds is useful in this case?", "2325342": "ok in short:\n\nIn market data the signal to noise ratio is really low, one big mistake that usually translate to abnormally high metrics is time leakage, that is using data from the future to predict the past. This usually mean that for evaluation we need to split data so that all event from the testing set happen after the training set.\n\nHowever here we don't have time so it is a bit difficult to do the split correctly, but i noticed the sp500 feature only take a small number of values so that it is probably constant for a given time. A good start would be to split each time in different fold, it is not ideal as it doesn't ensure monotonicity of time but it should be better than nothing.", "2325510": "ok, I get it now, Thank you very much for the explanation :)", "2344394": "Hey @amlhassan ! Can you share how you achieved 30/20% precision and recall for minority class. What did you do, which models used? Cost-sensivity? Resampling? Can you kindly share your working process?", "2344901": "Sure, I used CatBoost and I didn't do anything special to get those results, just CatBoost!", "2345260": "Wow. You just take data, loaded it all in CatBoost and got such results? Impressive" }, "source": "meta" }