Just for fun, given that the leaderboard score in this competition is the Pearson correlation coefficient, I thought it would be interesting to try to estimate the p-value to go with the correlations.
\nLet us say we have one cryptocurrency, and our data is collected once every minute. Currently the Private leaderboard consists of say 60 days of data, leading to 60x24x60 = 86400 observations.
\nWith this number of observations in order to obtain a p-value below 0.05 one would need a Pearson correlation coefficient of 0.0067 or above (It is curious to note that at the moment this very neatly coincides with the scores of the teams currently in the medal zone; the last bronze has a Pearson correlation coefficient of 0.0064).
\nIf we now take into consideration that we have 14 cryptocurrencies, and assuming that they all have an equal weight, then we now have 60x24x60x14 = 1,209,600 observations. In this case to obtain a p-value below 0.05 one would need a Pearson correlation coefficient of 0.0018 or above (corresponding to the ≈700 range on the current leaderboard).
\nNote that a p-value of 0.05 indicates that there is only a 5% chance of observing a Pearson correlation coefficient of 0.0018 if our submission actually had no correlation with the leaderboard data whatsoever.
\nSample code snippet to calculate the p-value for a given leaderboard score:
\nimport numpy as np\nfrom scipy.stats import t\n\nLB_score = 0.0067\nn_days = 60\nn_obs = n_days * 24 * 60\ndf = n_obs-2\nt_score = (LB_score * np.sqrt(df)) / (np.sqrt(1-LB_score**2))\np_value = (1 - t.cdf(x=t_score, df=df))*2\nprint(p_value)\nResult: 0.0489
\nFor the upcoming updates (p-value < 0.05):
\n| #days | \n1 cryptocurrency | \n14 cryptocurrencies | \n
|---|---|---|
| 75 | \n0.0060 | \n0.0016 | \n
| 90 | \n0.0055 | \n0.0015 | \n
Conclusion: At the end of this competition, anyone with a Private LB score of 0.0015 or above has most probably managed to find some signal in this data, and anyone with a score above 0.0055 has done magnificently well!
\nAll the best,
\ncarl
(Note that the p-values also apply to negative Pearson correlation coefficients too).
", "messages": [ { "id": "1742835", "postDate": "04/02/2022 10:39:20", "content": "Just for fun, given that the leaderboard score in this competition is the Pearson correlation coefficient, I thought it would be interesting to try to estimate the p-value to go with the correlations.
\nLet us say we have one cryptocurrency, and our data is collected once every minute. Currently the Private leaderboard consists of say 60 days of data, leading to 60x24x60 = 86400 observations.
\nWith this number of observations in order to obtain a p-value below 0.05 one would need a Pearson correlation coefficient of 0.0067 or above (It is curious to note that at the moment this very neatly coincides with the scores of the teams currently in the medal zone; the last bronze has a Pearson correlation coefficient of 0.0064).
\nIf we now take into consideration that we have 14 cryptocurrencies, and assuming that they all have an equal weight, then we now have 60x24x60x14 = 1,209,600 observations. In this case to obtain a p-value below 0.05 one would need a Pearson correlation coefficient of 0.0018 or above (corresponding to the ≈700 range on the current leaderboard).
\nNote that a p-value of 0.05 indicates that there is only a 5% chance of observing a Pearson correlation coefficient of 0.0018 if our submission actually had no correlation with the leaderboard data whatsoever.
\nSample code snippet to calculate the p-value for a given leaderboard score:
\nimport numpy as np\nfrom scipy.stats import t\n\nLB_score = 0.0067\nn_days = 60\nn_obs = n_days * 24 * 60\ndf = n_obs-2\nt_score = (LB_score * np.sqrt(df)) / (np.sqrt(1-LB_score**2))\np_value = (1 - t.cdf(x=t_score, df=df))*2\nprint(p_value)\nResult: 0.0489
\nFor the upcoming updates (p-value < 0.05):
\n| #days | \n1 cryptocurrency | \n14 cryptocurrencies | \n
|---|---|---|
| 75 | \n0.0060 | \n0.0016 | \n
| 90 | \n0.0055 | \n0.0015 | \n
Conclusion: At the end of this competition, anyone with a Private LB score of 0.0015 or above has most probably managed to find some signal in this data, and anyone with a score above 0.0055 has done magnificently well!
\nAll the best,
\ncarl
(Note that the p-values also apply to negative Pearson correlation coefficients too).
", "rawMarkdown": "Just for fun, given that the leaderboard score in this competition is the Pearson correlation coefficient, I thought it would be interesting to try to estimate the [*p*-value](https://en.wikipedia.org/wiki/P-value) to go with the correlations.\n\nLet us say we have one cryptocurrency, and our data is collected once every minute. Currently the Private leaderboard consists of say 60 days of data, leading to 60x24x60 = 86400 observations. \n\nWith this number of observations in order to obtain a *p*-value **below 0.05** one would need a Pearson correlation coefficient of **0.0067 or above** (It is curious to note that at the moment this very neatly coincides with the scores of the teams currently in the medal zone; the last bronze has a Pearson correlation coefficient of 0.0064). \n\nIf we now take into consideration that we have 14 cryptocurrencies, and assuming that they all have an equal weight, then we now have 60x24x60x14 = 1,209,600 observations. In this case to obtain a *p*-value **below 0.05** one would need a Pearson correlation coefficient of **0.0018 or above** (corresponding to the ≈700 range on the current leaderboard). \n\nNote that a *p*-value of 0.05 indicates that there is only a 5% chance of observing a Pearson correlation coefficient of 0.0018 if our submission actually had no correlation with the leaderboard data whatsoever.\n\nSample code snippet to calculate the *p*-value for a given leaderboard score:\n\n```\nimport numpy as np\nfrom scipy.stats import t\n\nLB_score = 0.0067\nn_days = 60\nn_obs = n_days * 24 * 60\ndf = n_obs-2\nt_score = (LB_score * np.sqrt(df)) / (np.sqrt(1-LB_score**2))\np_value = (1 - t.cdf(x=t_score, df=df))*2\nprint(p_value)\n```\nResult: 0.0489\n\nFor the upcoming updates (*p*-value < 0.05):\n\n| #days | 1 cryptocurrency | 14 cryptocurrencies |\n| --- | --- |---|\n| 75 | 0.0060 | 0.0016 |\n| 90 | 0.0055 | 0.0015 |\n\nConclusion: At the end of this competition, anyone with a Private LB score of 0.0015 or above has most probably managed to find some signal in this data, and anyone with a score above 0.0055 has done magnificently well!\n\nAll the best,\ncarl\n\n(Note that the *p*-values also apply to negative Pearson correlation coefficients too).", "votes": null }, { "id": "1746677", "postDate": "04/06/2022 02:15:07", "content": "What does it mean to have a Public leaderboard score of 0.0079 and -0.0091 in the last update 6 days ago ?
\nCould you please elaborate more on the method above ?
\nThanks
Dear @ptrikp
\nBoth of your LB scores correspond to a p-value of basically 0. This would indicate that these correlations, both the positive and the negative value, represent submissions that have genuine Pearson correlations with the Private LB data, and are most definitely not a fortuitous result of submitting data that has no correlation with the hidden data.
\nNote that here by using Pearson we are assuming a bivariate normal distribution (i.e. forming a straight line on a Q–Q plot). Empirically one can see that, for example, stock returns have more of a Cauchy distribution than a normal distribution, with somewhat heavier tails than a normal distribution (and an undefined variance), but that should be ok(?). However, the Pearson correlation is most certainly not resilient to outliers, and in the case that there are outliers in either ones submission, or in the private data, perhaps the Spearman's correlation coefficient would be more appropriate.
\nAlso, for example, I am not sure how the scores are calculated by the organizers, but for example the stitching together of 14 series, each with very different \\( \\mu \\) and \\( \\sigma^2 \\), into one long series, thus creating two multimodal distributions, and feeding that into a Pearson correlation could produce some spurious results. But on the other hand I do not think one can simply calculate 14 Pearson correlation values (one for each cryptocurrency), and take the mean Pearson value either (just like in general one cannot take the mean of two percentage scores to obtain an overall percentage). We are told that submissions are evaluated on a weighted version of the Pearson correlation coefficient so…
\nIn view of this I would add the caveat emptor that the p-values that I mention in the topic are only published for fun, and I am assuming that all of the criteria for the correct use of the Pearson correlation have been met.
\nThat aside, for more on the method I used I would suggest the Wikipedia pages p-value, Student's t-test, and t-statistic, and perhaps also the notebook \"Hypothesis testing: The t-test, p-values and power\".
\nAll the best,
\ncarl
A related question is: Suppose all 2077 teams had been allowed to submit one random number generated solution alongside their two other submissions. What proportion of medals would be won by these randomly generated solutions?
", "rawMarkdown": "A related question is: Suppose all 2077 teams had been allowed to submit one random number generated solution alongside their two other submissions. What proportion of medals would be won by these randomly generated solutions?", "votes": null }, { "id": "1748362", "postDate": "04/07/2022 14:08:41", "content": "Dear @jbomitchell
\nWonderful to hear form you. I would suggest that the answer to your question is none: currently the LB score associated with the last medal is 0.0064. Plugging that into the above routine (using the 14 cryptocurrency number of observations) returns a p-value of \\( 2\\times10^{-12} \\). So it is extremely unlikely that a LB score of 0.0064 could be obtained by a submission that actually had no correlation with the leaderboard data whatsoever. (Additionally, in the proposed scenario, the RNG solution would have to beat the teams other two submissions first, before even making it onto the LB, let alone win a medal).
\nNow, I am going to venture a guess that your question has two parts? I see that you are currently in 515th position. May I be so bold as to ask whether one of your submissions was indeed such a 'random number generated' solution? If so I would be most concerned with the validity of the LB scores, not because of the magnificent work that people have created, but because, as somewhat outlined in my reply to the other question on this topic, I am not entirely convinced that the Pearson correlation is ideally suited to this type of data, and perhaps the Spearman would be a better choice.
\nAll the best,
\ncarl
@carlmcbrideellis - I did not deliberately submit random numbers, though my second submission (a Random Forest with a rather small number of trees) has a score very close to zero. My 'first' submission is the median target value for each coin, which as you noted is around the top 25% right now.
", "rawMarkdown": "carlmcbrideellis - I did not deliberately submit random numbers, though my second submission (a Random Forest with a rather small number of trees) has a score very close to zero. My 'first' submission is the median target value for each coin, which as you noted is around the top 25% right now.", "votes": null }, { "id": "1748426", "postDate": "04/07/2022 15:21:26", "content": "Dear @jbomitchell
\nI think a Random Forest with few trees would, in the limit of one tree, simply become a single decision tree, and should still be a reasonable estimator. If say one had a shallow Random Forest, in the limit of max_depth=1 each individual tree would have just two leaves, i.e. return just two possible target values. In either case I am fairly sure that you did not end up submitting a series of random numbers.
Given that such time series are somewhat akin to a Martingale sequence, in conjunction with looking at the correlograms of 14 cryptocurrencies using the very last day of data I myself decided to simply submit a naïve model, consisting of the average target value of each coin over an arbitrary time window, and not invest any more time in the competition (BTW: currently one such submission of mine has a LB score of -0.0041) .
\nIf I was paying more attention to the scoring metric I should not have done that, because the correlation coefficient for which either one of the series is constant is undefined (NaN), given that the variance of a constant is zero. The fact that I have a LB score at all implies that the 14 series are indeed stitched together into one long series.
\nAll the best,
\ncarl
Here is a simple example of a spurious correlation caused by the stitching together of two sets of predictions having different levels (it doesn't matter how big the level difference is, just that one exists, here I arbitrarily choose 0.1). Say we have one series of observations [0.01, 0.01, 0.01, 0.01], and my predictions [0, 0, 0, 0], and a second series of observations [0.11, 0.11, 0.11, 0.11] and my predictions [0.1, 0.1, 0.1, 0.1]. Individually both of these series on their own have undefined correlation coefficients. However, upon stitching them together
\nfrom scipy import stats\nstats.spearmanr([0,0,0,0,0.1,0.1,0.1,0.1],[0.01,0.01,0.01,0.01,0.11,0.11,0.11,0.11])\none obtains a correlation of 1 with a p-value of 0.
\nAn intuitive explanation for what is observed is that by having two differing means we have created a step function for the ground truth, which is followed by the predictions, hence the overall correlation.
\nUpdate I have taken the notebook down as there is something I am not sure of. All the best, carl
", "rawMarkdown": "Here is a simple example of a spurious correlation caused by the stitching together of two sets of predictions having different levels (it doesn't matter how big the level difference is, just that one exists, here I arbitrarily choose 0.1). Say we have one series of observations [0.01, 0.01, 0.01, 0.01], and my predictions [0, 0, 0, 0], and a second series of observations [0.11, 0.11, 0.11, 0.11] and my predictions [0.1, 0.1, 0.1, 0.1]. Individually both of these series on their own have undefined correlation coefficients. However, upon stitching them together \n\n```\nfrom scipy import stats\nstats.spearmanr([0,0,0,0,0.1,0.1,0.1,0.1],[0.01,0.01,0.01,0.01,0.11,0.11,0.11,0.11])\n```\n\none obtains a correlation of 1 with a *p-*value of 0.\n\n~~Just for fun I have also written a small notebook simulating the Pearson correlation coefficient and the corresponding *p*-value between two series of random numbers with differing means that have been stitched together: [\"Calculating combined Pearson correlation values\"](https://www.kaggle.com/code/carlmcbrideellis/calculating-combined-pearson-correlation-values).~~ An intuitive explanation for what is observed is that by having two differing means we have created a step function for the ground truth, which is followed by the predictions, hence the overall correlation.\n\n**Update** I have taken the notebook down as there is something I am not sure of. All the best, carl", "votes": null } ], "comments": [ { "id": 1746677, "author_name": "ptrikp", "author_url": "", "post_date": "04/06/2022 02:15:07", "content": "What does it mean to have a Public leaderboard score of 0.0079 and -0.0091 in the last update 6 days ago ?
\nCould you please elaborate more on the method above ?
\nThanks
Dear @ptrikp
\nBoth of your LB scores correspond to a p-value of basically 0. This would indicate that these correlations, both the positive and the negative value, represent submissions that have genuine Pearson correlations with the Private LB data, and are most definitely not a fortuitous result of submitting data that has no correlation with the hidden data.
\nNote that here by using Pearson we are assuming a bivariate normal distribution (i.e. forming a straight line on a Q–Q plot). Empirically one can see that, for example, stock returns have more of a Cauchy distribution than a normal distribution, with somewhat heavier tails than a normal distribution (and an undefined variance), but that should be ok(?). However, the Pearson correlation is most certainly not resilient to outliers, and in the case that there are outliers in either ones submission, or in the private data, perhaps the Spearman's correlation coefficient would be more appropriate.
\nAlso, for example, I am not sure how the scores are calculated by the organizers, but for example the stitching together of 14 series, each with very different \\( \\mu \\) and \\( \\sigma^2 \\), into one long series, thus creating two multimodal distributions, and feeding that into a Pearson correlation could produce some spurious results. But on the other hand I do not think one can simply calculate 14 Pearson correlation values (one for each cryptocurrency), and take the mean Pearson value either (just like in general one cannot take the mean of two percentage scores to obtain an overall percentage). We are told that submissions are evaluated on a weighted version of the Pearson correlation coefficient so…
\nIn view of this I would add the caveat emptor that the p-values that I mention in the topic are only published for fun, and I am assuming that all of the criteria for the correct use of the Pearson correlation have been met.
\nThat aside, for more on the method I used I would suggest the Wikipedia pages p-value, Student's t-test, and t-statistic, and perhaps also the notebook \"Hypothesis testing: The t-test, p-values and power\".
\nAll the best,
\ncarl
A related question is: Suppose all 2077 teams had been allowed to submit one random number generated solution alongside their two other submissions. What proportion of medals would be won by these randomly generated solutions?
", "votes": null, "replies": [ { "id": 1748362, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "04/07/2022 14:08:41", "content": "Dear @jbomitchell
\nWonderful to hear form you. I would suggest that the answer to your question is none: currently the LB score associated with the last medal is 0.0064. Plugging that into the above routine (using the 14 cryptocurrency number of observations) returns a p-value of \\( 2\\times10^{-12} \\). So it is extremely unlikely that a LB score of 0.0064 could be obtained by a submission that actually had no correlation with the leaderboard data whatsoever. (Additionally, in the proposed scenario, the RNG solution would have to beat the teams other two submissions first, before even making it onto the LB, let alone win a medal).
\nNow, I am going to venture a guess that your question has two parts? I see that you are currently in 515th position. May I be so bold as to ask whether one of your submissions was indeed such a 'random number generated' solution? If so I would be most concerned with the validity of the LB scores, not because of the magnificent work that people have created, but because, as somewhat outlined in my reply to the other question on this topic, I am not entirely convinced that the Pearson correlation is ideally suited to this type of data, and perhaps the Spearman would be a better choice.
\nAll the best,
\ncarl
@carlmcbrideellis - I did not deliberately submit random numbers, though my second submission (a Random Forest with a rather small number of trees) has a score very close to zero. My 'first' submission is the median target value for each coin, which as you noted is around the top 25% right now.
", "votes": null, "replies": [] }, { "id": 1748426, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "04/07/2022 15:21:26", "content": "Dear @jbomitchell
\nI think a Random Forest with few trees would, in the limit of one tree, simply become a single decision tree, and should still be a reasonable estimator. If say one had a shallow Random Forest, in the limit of max_depth=1 each individual tree would have just two leaves, i.e. return just two possible target values. In either case I am fairly sure that you did not end up submitting a series of random numbers.
Given that such time series are somewhat akin to a Martingale sequence, in conjunction with looking at the correlograms of 14 cryptocurrencies using the very last day of data I myself decided to simply submit a naïve model, consisting of the average target value of each coin over an arbitrary time window, and not invest any more time in the competition (BTW: currently one such submission of mine has a LB score of -0.0041) .
\nIf I was paying more attention to the scoring metric I should not have done that, because the correlation coefficient for which either one of the series is constant is undefined (NaN), given that the variance of a constant is zero. The fact that I have a LB score at all implies that the 14 series are indeed stitched together into one long series.
\nAll the best,
\ncarl
Here is a simple example of a spurious correlation caused by the stitching together of two sets of predictions having different levels (it doesn't matter how big the level difference is, just that one exists, here I arbitrarily choose 0.1). Say we have one series of observations [0.01, 0.01, 0.01, 0.01], and my predictions [0, 0, 0, 0], and a second series of observations [0.11, 0.11, 0.11, 0.11] and my predictions [0.1, 0.1, 0.1, 0.1]. Individually both of these series on their own have undefined correlation coefficients. However, upon stitching them together
\nfrom scipy import stats\nstats.spearmanr([0,0,0,0,0.1,0.1,0.1,0.1],[0.01,0.01,0.01,0.01,0.11,0.11,0.11,0.11])\none obtains a correlation of 1 with a p-value of 0.
\nAn intuitive explanation for what is observed is that by having two differing means we have created a step function for the ground truth, which is followed by the predictions, hence the overall correlation.
\nUpdate I have taken the notebook down as there is something I am not sure of. All the best, carl
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1742835": "Just for fun, given that the leaderboard score in this competition is the Pearson correlation coefficient, I thought it would be interesting to try to estimate the [*p*-value](https://en.wikipedia.org/wiki/P-value) to go with the correlations.\n\nLet us say we have one cryptocurrency, and our data is collected once every minute. Currently the Private leaderboard consists of say 60 days of data, leading to 60x24x60 = 86400 observations. \n\nWith this number of observations in order to obtain a *p*-value **below 0.05** one would need a Pearson correlation coefficient of **0.0067 or above** (It is curious to note that at the moment this very neatly coincides with the scores of the teams currently in the medal zone; the last bronze has a Pearson correlation coefficient of 0.0064). \n\nIf we now take into consideration that we have 14 cryptocurrencies, and assuming that they all have an equal weight, then we now have 60x24x60x14 = 1,209,600 observations. In this case to obtain a *p*-value **below 0.05** one would need a Pearson correlation coefficient of **0.0018 or above** (corresponding to the ≈700 range on the current leaderboard). \n\nNote that a *p*-value of 0.05 indicates that there is only a 5% chance of observing a Pearson correlation coefficient of 0.0018 if our submission actually had no correlation with the leaderboard data whatsoever.\n\nSample code snippet to calculate the *p*-value for a given leaderboard score:\n\n```\nimport numpy as np\nfrom scipy.stats import t\n\nLB_score = 0.0067\nn_days = 60\nn_obs = n_days * 24 * 60\ndf = n_obs-2\nt_score = (LB_score * np.sqrt(df)) / (np.sqrt(1-LB_score**2))\np_value = (1 - t.cdf(x=t_score, df=df))*2\nprint(p_value)\n```\nResult: 0.0489\n\nFor the upcoming updates (*p*-value < 0.05):\n\n| #days | 1 cryptocurrency | 14 cryptocurrencies |\n| --- | --- |---|\n| 75 | 0.0060 | 0.0016 |\n| 90 | 0.0055 | 0.0015 |\n\nConclusion: At the end of this competition, anyone with a Private LB score of 0.0015 or above has most probably managed to find some signal in this data, and anyone with a score above 0.0055 has done magnificently well!\n\nAll the best,\ncarl\n\n(Note that the *p*-values also apply to negative Pearson correlation coefficients too).", "1746677": "What does it mean to have a Public leaderboard score of 0.0079 and -0.0091 in the last update 6 days ago ?\nCould you please elaborate more on the method above ?\nThanks", "1746780": "Dear @ptrikp \n\nBoth of your LB scores correspond to a *p*-value of basically 0. This would indicate that these correlations, both the positive *and* the negative value, represent submissions that have genuine Pearson correlations with the Private LB data, and are most definitely not a fortuitous result of submitting data that has no correlation with the hidden data.\n\nNote that here by using Pearson we are assuming a bivariate normal distribution (*i.e.* forming a straight line on a [Q–Q plot](https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot)). Empirically one can see that, for example, [stock returns have more of a Cauchy distribution](https://www.kaggle.com/code/carlmcbrideellis/s-p-500-daily-returns-normal-and-cauchy-fits/) than a normal distribution, with somewhat heavier tails than a normal distribution (and an undefined variance), but that should be ok(?). However, the Pearson correlation is most certainly not resilient to outliers, and in the case that there are outliers in either ones submission, or in the private data, perhaps the [Spearman's correlation coefficient](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) would be more appropriate.\n\nAlso, for example, I am not sure how the scores are calculated by the organizers, but for example the stitching together of 14 series, each with very different \\\\( \\mu \\\\) and \\\\( \\sigma^2 \\\\), into one long series, thus creating two [multimodal distributions](https://en.wikipedia.org/wiki/Multimodal_distribution), and feeding that into a Pearson correlation could produce some spurious results. But on the other hand I do not think one can simply calculate 14 Pearson correlation values (one for each cryptocurrency), and take the mean Pearson value either (just like in general one cannot take the mean of two percentage scores to obtain an overall percentage). We are told that [submissions are evaluated on a ***weighted version*** of the Pearson correlation coefficient](https://www.kaggle.com/competitions/g-research-crypto-forecasting/overview/evaluation) so...\n\nIn view of this I would add the *caveat emptor* that the *p*-values that I mention in the topic are only published for fun, and I am assuming that all of the criteria for the correct use of the Pearson correlation have been met.\n\nThat aside, for more on the method I used I would suggest the Wikipedia pages [p-value](https://en.wikipedia.org/wiki/P-value), [Student's t-test](https://en.wikipedia.org/wiki/Student%27s_t-test), and [t-statistic](https://en.wikipedia.org/wiki/T-statistic), and perhaps also the notebook [\"Hypothesis testing: The t-test, p-values and power\"](https://www.kaggle.com/code/carlmcbrideellis/hypothesis-testing-the-t-test-p-values-and-power/).\n\nAll the best,\ncarl", "1748307": "A related question is: Suppose all 2077 teams had been allowed to submit one random number generated solution alongside their two other submissions. What proportion of medals would be won by these randomly generated solutions?", "1748362": "Dear @jbomitchell \n\nWonderful to hear form you. I would suggest that the answer to your question is none: currently the LB score associated with the last medal is 0.0064. Plugging that into the above routine (using the 14 cryptocurrency number of observations) returns a *p*-value of \\\\( 2\\times10^{-12} \\\\). So it is extremely unlikely that a LB score of 0.0064 could be obtained by a submission that actually had no correlation with the leaderboard data whatsoever. (Additionally, in the proposed scenario, the RNG solution would have to beat the teams other two submissions first, before even making it onto the LB, let alone win a medal). \n\nNow, I am going to venture a guess that your question has two parts? I see that you are currently in 515th position. May I be so bold as to ask whether one of your submissions was indeed such a 'random number generated' solution? If so I would be most concerned with the validity of the LB scores, not because of the magnificent work that people have created, but because, as somewhat outlined in my reply to the other question on this topic, I am not entirely convinced that the Pearson correlation is ideally suited to this type of data, and perhaps the Spearman would be a better choice.\n\nAll the best,\ncarl", "1748392": "carlmcbrideellis - I did not deliberately submit random numbers, though my second submission (a Random Forest with a rather small number of trees) has a score very close to zero. My 'first' submission is the median target value for each coin, which as you noted is around the top 25% right now.", "1748426": "Dear @jbomitchell \n\nI think a Random Forest with few trees would, in the limit of one tree, simply become a single decision tree, and should still be a reasonable estimator. If say one had a shallow Random Forest, in the limit of `max_depth=1` each individual tree would have just two leaves, *i.e.* return just two possible target values. In either case I am fairly sure that you did not end up submitting a series of random numbers. \n\nGiven that such time series are somewhat akin to a [Martingale sequence](https://en.wikipedia.org/wiki/Martingale_(probability_theory)), in conjunction with looking at the [correlograms of 14 cryptocurrencies using the very last day of data](https://www.kaggle.com/code/carlmcbrideellis/correlograms-of-14-cryptocurrencies-1-day) I myself decided to simply submit a naïve model, consisting of the average target value of each coin over an arbitrary time window, and not invest any more time in the competition (BTW: currently one such submission of mine has a LB score of **-**0.0041) .\n\nIf I was paying more attention to the scoring metric I should not have done that, because the correlation coefficient for which either one of the series is constant is undefined (NaN), given that the variance of a constant is zero. The fact that I have a LB score at all implies that the 14 series are indeed stitched together into one long series.\n\nAll the best,\ncarl", "1748913": "Here is a simple example of a spurious correlation caused by the stitching together of two sets of predictions having different levels (it doesn't matter how big the level difference is, just that one exists, here I arbitrarily choose 0.1). Say we have one series of observations [0.01, 0.01, 0.01, 0.01], and my predictions [0, 0, 0, 0], and a second series of observations [0.11, 0.11, 0.11, 0.11] and my predictions [0.1, 0.1, 0.1, 0.1]. Individually both of these series on their own have undefined correlation coefficients. However, upon stitching them together \n\n```\nfrom scipy import stats\nstats.spearmanr([0,0,0,0,0.1,0.1,0.1,0.1],[0.01,0.01,0.01,0.01,0.11,0.11,0.11,0.11])\n```\n\none obtains a correlation of 1 with a *p-*value of 0.\n\n~~Just for fun I have also written a small notebook simulating the Pearson correlation coefficient and the corresponding *p*-value between two series of random numbers with differing means that have been stitched together: [\"Calculating combined Pearson correlation values\"](https://www.kaggle.com/code/carlmcbrideellis/calculating-combined-pearson-correlation-values).~~ An intuitive explanation for what is observed is that by having two differing means we have created a step function for the ground truth, which is followed by the predictions, hence the overall correlation.\n\n**Update** I have taken the notebook down as there is something I am not sure of. All the best, carl" }, "source": "meta" }