{ "id": 296744, "title": "The way I think the problem should be handled and where exactly are models going wrong ", "url": "/competitions/g-research-crypto-forecasting/discussion/296744", "author_name": "Sayantan Mazumdar", "post_date": "2021-12-23T11:39:06.669000", "votes": 4, "comment_count": 9, "views": 0, "content": "

Hi,
\nJust a quick observation i found that many models really are learning pretty random , one thing which I have found about the target is that when price after 16 min decreases probably Target is going to be negative and when not positive if same then 0 Makes total sense if not see my kernel https://www.kaggle.com/swaralipibose/explaining-the-target-variable-clearly-with-proof. Our models does not seem to be able to answer the question Will the price decrease in next 16 min compared to the next minute absolutely makes sense , The model always says that it increases or decreases something like imblance but there is no imbalance in dataset . And that's a big issue , cuz this means that models cant really predict if the target will be less or greater than 0 . I even trained a classifier for this. Open,Close,High,Low are just too naive for this competition I think , we still don't know what exactly drives crypto price , except a few things like :-

\n\n

Each up and back down in crypto graphs , do follow a normal distribution so if the price has been increasing for a long time most likely it has reached the peak of the normal distribution and is about to decrease

\n

Also to observe that if all the crypto prices are rolled by the same driver they will show some common relation

\n

With this I am taking a leave ,
\nBye

", "messages": [ { "id": 1626974, "postDate": "2021-12-23T11:39:06.670Z", "content": "

Hi,
\nJust a quick observation i found that many models really are learning pretty random , one thing which I have found about the target is that when price after 16 min decreases probably Target is going to be negative and when not positive if same then 0 Makes total sense if not see my kernel https://www.kaggle.com/swaralipibose/explaining-the-target-variable-clearly-with-proof. Our models does not seem to be able to answer the question Will the price decrease in next 16 min compared to the next minute absolutely makes sense , The model always says that it increases or decreases something like imblance but there is no imbalance in dataset . And that's a big issue , cuz this means that models cant really predict if the target will be less or greater than 0 . I even trained a classifier for this. Open,Close,High,Low are just too naive for this competition I think , we still don't know what exactly drives crypto price , except a few things like :-

\n\n

Each up and back down in crypto graphs , do follow a normal distribution so if the price has been increasing for a long time most likely it has reached the peak of the normal distribution and is about to decrease

\n

Also to observe that if all the crypto prices are rolled by the same driver they will show some common relation

\n

With this I am taking a leave ,
\nBye

", "rawMarkdown": "Hi,\nJust a quick observation i found that many models really are learning pretty random , one thing which I have found about the target is that when price after 16 min decreases probably Target is going to be negative and when not positive if same then 0 Makes total sense if not see my kernel https://www.kaggle.com/swaralipibose/explaining-the-target-variable-clearly-with-proof. Our models does not seem to be able to answer the question ```Will the price decrease in next 16 min compared to the next minute``` absolutely makes sense , The model always says that it increases or decreases something like imblance but there is no imbalance in dataset . And that's a big issue , cuz this means that models cant really predict if the target will be less or greater than 0 . I even trained a classifier for this. Open,Close,High,Low are just too naive for this competition I think , we still don't know what exactly drives crypto price , except a few things like :-\n\n* Season\n* Month\n* Weekday\n* Weekend\n* twitter social media \nApart from this one more observation I found is that\n\nEach up and back down in crypto graphs , do follow a normal distribution so if the price has been increasing for a long time most likely it has reached the peak of the normal distribution and is about to decrease\n\nAlso to observe that if all the crypto prices are rolled by the same driver they will show some common relation\n\nWith this I am taking a leave ,\nBye", "votes": 4 }, { "id": 1627113, "postDate": "2021-12-23T14:15:20.407Z", "content": "

Yes predicting 15 mins ahead just from the previous minute is really insufficient. You need at least some lagged features. And even then you will get low performance. Just for the sake of comparison, take the JS competition, in which we had better data. The top scoring notebooks on the -almost- classification problem of predicting up or down, barely reached 0.54 AUC.

", "rawMarkdown": "Yes predicting 15 mins ahead just from the previous minute is really insufficient. You need at least some lagged features. And even then you will get low performance. Just for the sake of comparison, take the JS competition, in which we had better data. The top scoring notebooks on the -almost- classification problem of predicting up or down, barely reached 0.54 AUC. ", "votes": 1, "replies": [ { "id": 1627960, "postDate": "2021-12-24T11:16:50.223Z", "content": "

I agree it can mean that the models cant really learn anything if we had window of like 1 hour or 3 hr then I think it would have helped the model

", "rawMarkdown": "I agree it can mean that the models cant really learn anything if we had window of like 1 hour or 3 hr then I think it would have helped the model" }, { "id": 1627974, "postDate": "2021-12-24T11:41:14.447Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1629077, "postDate": "2021-12-25T16:52:08.293Z", "content": "

If the models were to predict random values I believe we would expect to see a final score of 0 down to about 5th-6th decimal as this is what you get when generating random values within reasonable limits (I picked 25h-75th percentiles of Target, both uniform and normal sample dists, real dist is of course not normal, feel free to test t or something and report back) and calculating weighted correlation. Most models get a score way above this (in my tests >100x better than random values). Furthermore, everyone works under the same constraints so absolute model performance is irrelevant anyways?

", "rawMarkdown": "If the models were to predict random values I believe we would expect to see a final score of 0 down to about 5th-6th decimal as this is what you get when generating random values within reasonable limits (I picked 25h-75th percentiles of Target, both uniform and normal sample dists, real dist is of course not normal, feel free to test t or something and report back) and calculating weighted correlation. Most models get a score way above this (in my tests >100x better than random values). Furthermore, everyone works under the same constraints so absolute model performance is irrelevant anyways?", "replies": [ { "id": 1629299, "postDate": "2021-12-26T01:12:13.513Z", "content": "

How do you exactly get The score will be 0 down to 5th-6th decimal This is not a classification problem statement that you can get it? Here especially the odds of that happening is even lower, why because the ratio is between the next 16 minutes and 1 minute is very low compact within a region because you can't just expect a very huge swing in 16 min. So take a two number here a maximum value in the target and b minimum value in the target

\n

Take a list with all value between that though that's impossible, maybe have a distance of dx like we do in calculus between them to approximate it out

\n

Call it by the name L

\n

Take a random number in the list L, what's the probability that number will be exactly equal to the target value expected its 1/len(L) which is somewhat like 1/(a-b) and if not exact prediction it's good if they are even approximately equal so expect the numerator to be way bigger ,here the denominator is the range of the target, now as I have said the target value is very compact within a region 1/range should not be that low either, and machine learning models are not even that random, they may pick those values which are repeated more something like imbalance , and use them more, adding it all up its indeed expected .

", "rawMarkdown": "How do you exactly get ```The score will be 0 down to 5th-6th decimal``` This is not a classification problem statement that you can get it? Here especially the odds of that happening is even lower, why because the ratio is between the next 16 minutes and 1 minute is very low compact within a region because you can't just expect a very huge swing in 16 min. So take a two number here ```a``` maximum value in the target and ```b``` minimum value in the target\n\nTake a list with all value between that though that's impossible, maybe have a distance of dx like we do in calculus between them to approximate it out\n\nCall it by the name L\n\nTake a random number in the list L, what's the probability that number will be exactly equal to the target value expected its 1/len(L) which is somewhat like 1/(a-b) and if not exact prediction it's good if they are even approximately equal so expect the numerator to be way bigger ,here the denominator is the range of the target, now as I have said the target value is very compact within a region 1/range should not be that low either, and machine learning models are not even that random, they may pick those values which are repeated more something like imbalance , and use them more, adding it all up its indeed expected ." }, { "id": 1630392, "postDate": "2021-12-27T08:35:56.313Z", "content": "

Not sure what you mean by This is not a classification problem statement that you can get it? but what I was attempting to do was to test whether as (I believe) you claim the models are not learning anything which I guess would be similar to predicting random values within a reasonable range. Here's the code I used

\n
#Analyse Target\nt_desc = crypto_df.Target.describe()\nmu = t_desc['mean']\nsigma = t_desc['std']\nc = 0\n\nfor i in range(100):\n    crypto_df['Pred'] = np.random.normal(mu, sigma, size=crypto_df.shape[0])\n    c += weighted_correlation(crypto_df['Target'], crypto_df['Pred'], crypto_df['Weights'])\n    print(i)\nprint(c/100)\n
\n

However, thinking about the above it is probably, as I believe you are saying in your reply, not the best way. If we want to be sure what kind of correlation score we could get out of our models based on pure luck, how about this: 1) Generate random values for all rows, 2) Run an OLS with X = random values and y = Target, 3) Look at summary stats, 4) Calculate weighted correlation?

\n

Furthermore regarding common values recurring and machine learning models predicting those, this is plausible as you say, however, I have run a few OLS regressions on the data and believe that there is at the very least one feature (that I done some calculations to get) that has a sizable coefficient which is highly significantly different from 0.

", "rawMarkdown": "Not sure what you mean by `This is not a classification problem statement that you can get it?` but what I was attempting to do was to test whether as (I believe) you claim the models are not learning anything which I guess would be similar to predicting random values within a reasonable range. Here's the code I used\n```\n#Analyse Target\nt_desc = crypto_df.Target.describe()\nmu = t_desc['mean']\nsigma = t_desc['std']\nc = 0\n\nfor i in range(100):\n crypto_df['Pred'] = np.random.normal(mu, sigma, size=crypto_df.shape[0])\n c += weighted_correlation(crypto_df['Target'], crypto_df['Pred'], crypto_df['Weights'])\n print(i)\nprint(c/100)\n```\n\nHowever, thinking about the above it is probably, as I believe you are saying in your reply, not the best way. If we want to be sure what kind of correlation score we could get out of our models based on pure luck, how about this: 1) Generate random values for all rows, 2) Run an OLS with X = random values and y = Target, 3) Look at summary stats, 4) Calculate weighted correlation?\n\nFurthermore regarding common values recurring and machine learning models predicting those, this is plausible as you say, however, I have run a few OLS regressions on the data and believe that there is at the very least one feature (that I done some calculations to get) that has a sizable coefficient which is highly significantly different from 0." }, { "id": 1630447, "postDate": "2021-12-27T10:30:02.667Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1630457, "postDate": "2021-12-27T10:40:11.357Z", "content": "

Did you do a correlation plot of that feat? I think it's correlated with OLHC price values

", "rawMarkdown": "Did you do a correlation plot of that feat? I think it's correlated with OLHC price values " }, { "id": 1630686, "postDate": "2021-12-27T15:37:46.823Z", "content": "

Correlation with that feat among OHLC values is at its highest 0.000263 on Close prices

", "rawMarkdown": "Correlation with that feat among OHLC values is at its highest 0.000263 on Close prices" } ] } ], "comments": [ { "id": 1627113, "author_name": "Lucas Morin", "author_url": "", "post_date": "2021-12-23T14:15:20.407000", "content": "

Yes predicting 15 mins ahead just from the previous minute is really insufficient. You need at least some lagged features. And even then you will get low performance. Just for the sake of comparison, take the JS competition, in which we had better data. The top scoring notebooks on the -almost- classification problem of predicting up or down, barely reached 0.54 AUC.

", "votes": 1, "replies": [ { "id": 1627960, "author_name": "Sayantan Mazumdar", "author_url": "", "post_date": "2021-12-24T11:16:50.223000", "content": "

I agree it can mean that the models cant really learn anything if we had window of like 1 hour or 3 hr then I think it would have helped the model

", "votes": 0, "replies": [] }, { "id": 1627974, "author_name": "", "author_url": "", "post_date": "2021-12-24T11:41:14.447000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1629077, "author_name": "theomon", "author_url": "", "post_date": "2021-12-25T16:52:08.293000", "content": "

If the models were to predict random values I believe we would expect to see a final score of 0 down to about 5th-6th decimal as this is what you get when generating random values within reasonable limits (I picked 25h-75th percentiles of Target, both uniform and normal sample dists, real dist is of course not normal, feel free to test t or something and report back) and calculating weighted correlation. Most models get a score way above this (in my tests >100x better than random values). Furthermore, everyone works under the same constraints so absolute model performance is irrelevant anyways?

", "votes": 0, "replies": [ { "id": 1629299, "author_name": "Sayantan Mazumdar", "author_url": "", "post_date": "2021-12-26T01:12:13.513000", "content": "

How do you exactly get The score will be 0 down to 5th-6th decimal This is not a classification problem statement that you can get it? Here especially the odds of that happening is even lower, why because the ratio is between the next 16 minutes and 1 minute is very low compact within a region because you can't just expect a very huge swing in 16 min. So take a two number here a maximum value in the target and b minimum value in the target

\n

Take a list with all value between that though that's impossible, maybe have a distance of dx like we do in calculus between them to approximate it out

\n

Call it by the name L

\n

Take a random number in the list L, what's the probability that number will be exactly equal to the target value expected its 1/len(L) which is somewhat like 1/(a-b) and if not exact prediction it's good if they are even approximately equal so expect the numerator to be way bigger ,here the denominator is the range of the target, now as I have said the target value is very compact within a region 1/range should not be that low either, and machine learning models are not even that random, they may pick those values which are repeated more something like imbalance , and use them more, adding it all up its indeed expected .

", "votes": 0, "replies": [] }, { "id": 1630392, "author_name": "theomon", "author_url": "", "post_date": "2021-12-27T08:35:56.313000", "content": "

Not sure what you mean by This is not a classification problem statement that you can get it? but what I was attempting to do was to test whether as (I believe) you claim the models are not learning anything which I guess would be similar to predicting random values within a reasonable range. Here's the code I used

\n
#Analyse Target\nt_desc = crypto_df.Target.describe()\nmu = t_desc['mean']\nsigma = t_desc['std']\nc = 0\n\nfor i in range(100):\n    crypto_df['Pred'] = np.random.normal(mu, sigma, size=crypto_df.shape[0])\n    c += weighted_correlation(crypto_df['Target'], crypto_df['Pred'], crypto_df['Weights'])\n    print(i)\nprint(c/100)\n
\n

However, thinking about the above it is probably, as I believe you are saying in your reply, not the best way. If we want to be sure what kind of correlation score we could get out of our models based on pure luck, how about this: 1) Generate random values for all rows, 2) Run an OLS with X = random values and y = Target, 3) Look at summary stats, 4) Calculate weighted correlation?

\n

Furthermore regarding common values recurring and machine learning models predicting those, this is plausible as you say, however, I have run a few OLS regressions on the data and believe that there is at the very least one feature (that I done some calculations to get) that has a sizable coefficient which is highly significantly different from 0.

", "votes": 0, "replies": [] }, { "id": 1630447, "author_name": "", "author_url": "", "post_date": "2021-12-27T10:30:02.667000", "content": "", "votes": 0, "replies": [] }, { "id": 1630457, "author_name": "Sayantan Mazumdar", "author_url": "", "post_date": "2021-12-27T10:40:11.357000", "content": "

Did you do a correlation plot of that feat? I think it's correlated with OLHC price values

", "votes": 0, "replies": [] }, { "id": 1630686, "author_name": "theomon", "author_url": "", "post_date": "2021-12-27T15:37:46.823000", "content": "

Correlation with that feat among OHLC values is at its highest 0.000263 on Close prices

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1626974": "Hi,\nJust a quick observation i found that many models really are learning pretty random , one thing which I have found about the target is that when price after 16 min decreases probably Target is going to be negative and when not positive if same then 0 Makes total sense if not see my kernel https://www.kaggle.com/swaralipibose/explaining-the-target-variable-clearly-with-proof. Our models does not seem to be able to answer the question ```Will the price decrease in next 16 min compared to the next minute``` absolutely makes sense , The model always says that it increases or decreases something like imblance but there is no imbalance in dataset . And that's a big issue , cuz this means that models cant really predict if the target will be less or greater than 0 . I even trained a classifier for this. Open,Close,High,Low are just too naive for this competition I think , we still don't know what exactly drives crypto price , except a few things like :-\n\n* Season\n* Month\n* Weekday\n* Weekend\n* twitter social media \nApart from this one more observation I found is that\n\nEach up and back down in crypto graphs , do follow a normal distribution so if the price has been increasing for a long time most likely it has reached the peak of the normal distribution and is about to decrease\n\nAlso to observe that if all the crypto prices are rolled by the same driver they will show some common relation\n\nWith this I am taking a leave ,\nBye", "1627113": "Yes predicting 15 mins ahead just from the previous minute is really insufficient. You need at least some lagged features. And even then you will get low performance. Just for the sake of comparison, take the JS competition, in which we had better data. The top scoring notebooks on the -almost- classification problem of predicting up or down, barely reached 0.54 AUC. ", "1629077": "If the models were to predict random values I believe we would expect to see a final score of 0 down to about 5th-6th decimal as this is what you get when generating random values within reasonable limits (I picked 25h-75th percentiles of Target, both uniform and normal sample dists, real dist is of course not normal, feel free to test t or something and report back) and calculating weighted correlation. Most models get a score way above this (in my tests >100x better than random values). Furthermore, everyone works under the same constraints so absolute model performance is irrelevant anyways?" } }