In my local CV, I was getting a score standard deviation between folds of about .015, and that was for test folds that were three months long. For periods of 2 weeks, I expect my model's score to have a standard deviation of .015 * sqrt(6), or .037.
\nIn other words, if my model is any indication, then the first update was almost all error, and the second update wasn't much better. I was expecting this, and I think many others were as well. You can see that a lot of grandmasters even have negative scores at this point, and most of them aren't complaining.
\nIt gets better. Whoever is in first place on the semifinal update should not be very surprised if the final update lowers them a few places!
\nSay the semifinal 1st place has a score of .05. On the final update they could easily score 0 for those two weeks. So their final score would be about .05 * 5/6 + 0 * 1/6 = .042. That's 4th place by the current score distribution.
\n(A few people complained that correlations don't add. Standard deviations do, if you add in quadrature, or when averaging, AKA root mean square. When I do that, I get .045 instead of .042. All this is about how surprised we should be. We shouldn't be surprised if .05 falls to .045 in the last update, and .042 isn't much further.)
\nOn the other hand, this conclusion ignores that all our scores are correlated. If the semifinal 1st place scores zero in the last two weeks, then it'll probably have been a bad week for everyone. So 1st place may not move all the way to 4th place. But it's still true that 1st place could easily go from .05 to .042 in the last update.
\nOn the other-other hand, I think that on this second update, the scores over .04 were extra-extra-lucky. That's just my opinion based on the fact that the curve of scores is convex for the top few rather than concave as we ordinarily see. Assuming that .04 is a better estimate of the top score than .05, then the 1st place semifinalist could fall to to .04 * 5/6 + 0 * 1/6 =.033, which is like falling to 21st place in the current distribution of scores.
\nThis is all quite rough and speculative, but I think the general lesson stands: you should forget any nervousness that's coming from noisy low scores. It's just the nature of the competition. The organizers could have decided to run the competition longer, but who wants to wait 6-12 months to get their score?
\nThis competition is a bit of a lottery. Why? Because asset trading is that way. It's probably good that some of you found this out in a competition before losing real money.
", "messages": [ { "id": "1717397", "postDate": "03/09/2022 22:29:13", "content": "In my local CV, I was getting a score standard deviation between folds of about .015, and that was for test folds that were three months long. For periods of 2 weeks, I expect my model's score to have a standard deviation of .015 * sqrt(6), or .037.
\nIn other words, if my model is any indication, then the first update was almost all error, and the second update wasn't much better. I was expecting this, and I think many others were as well. You can see that a lot of grandmasters even have negative scores at this point, and most of them aren't complaining.
\nIt gets better. Whoever is in first place on the semifinal update should not be very surprised if the final update lowers them a few places!
\nSay the semifinal 1st place has a score of .05. On the final update they could easily score 0 for those two weeks. So their final score would be about .05 * 5/6 + 0 * 1/6 = .042. That's 4th place by the current score distribution.
\n(A few people complained that correlations don't add. Standard deviations do, if you add in quadrature, or when averaging, AKA root mean square. When I do that, I get .045 instead of .042. All this is about how surprised we should be. We shouldn't be surprised if .05 falls to .045 in the last update, and .042 isn't much further.)
\nOn the other hand, this conclusion ignores that all our scores are correlated. If the semifinal 1st place scores zero in the last two weeks, then it'll probably have been a bad week for everyone. So 1st place may not move all the way to 4th place. But it's still true that 1st place could easily go from .05 to .042 in the last update.
\nOn the other-other hand, I think that on this second update, the scores over .04 were extra-extra-lucky. That's just my opinion based on the fact that the curve of scores is convex for the top few rather than concave as we ordinarily see. Assuming that .04 is a better estimate of the top score than .05, then the 1st place semifinalist could fall to to .04 * 5/6 + 0 * 1/6 =.033, which is like falling to 21st place in the current distribution of scores.
\nThis is all quite rough and speculative, but I think the general lesson stands: you should forget any nervousness that's coming from noisy low scores. It's just the nature of the competition. The organizers could have decided to run the competition longer, but who wants to wait 6-12 months to get their score?
\nThis competition is a bit of a lottery. Why? Because asset trading is that way. It's probably good that some of you found this out in a competition before losing real money.
", "rawMarkdown": "In my local CV, I was getting a score standard deviation between folds of about .015, and that was for test folds that were three months long. For periods of 2 weeks, I expect my model's score to have a standard deviation of .015 * sqrt(6), or .037. \n\nIn other words, if my model is any indication, then the first update was almost all error, and the second update wasn't much better. I was expecting this, and I think many others were as well. You can see that a lot of grandmasters even have negative scores at this point, and most of them aren't complaining.\n\nIt gets better. Whoever is in first place on the semifinal update should not be very surprised if the final update lowers them a few places!\n\nSay the semifinal 1st place has a score of .05. On the final update they could easily score 0 for those two weeks. So their final score would be about .05 * 5/6 + 0 * 1/6 = .042. That's 4th place by the current score distribution. \n\n(A few people complained that correlations don't add. ~~They do,~~ Standard deviations do, if you add in quadrature, or when averaging, AKA root mean square. When I do that, I get .045 instead of .042. All this is about how surprised we should be. We shouldn't be surprised if .05 falls to .045 in the last update, and .042 isn't much further.)\n\nOn the other hand, this conclusion ignores that all our scores are correlated. If the semifinal 1st place scores zero in the last two weeks, then it'll probably have been a bad week for everyone. So 1st place may not move all the way to 4th place. But it's still true that 1st place could easily go from .05 to .042 in the last update.\n\nOn the other-other hand, I think that on this second update, the scores over .04 were extra-extra-lucky. That's just my opinion based on the fact that the curve of scores is convex for the top few rather than concave as we ordinarily see. Assuming that .04 is a better estimate of the top score than .05, then the 1st place semifinalist could fall to to .04 * 5/6 + 0 * 1/6 =.033, which is like falling to 21st place in the current distribution of scores.\n\nThis is all quite rough and speculative, but I think the general lesson stands: you should forget any nervousness that's coming from noisy low scores. It's just the nature of the competition. The organizers could have decided to run the competition longer, but who wants to wait 6-12 months to get their score? \n\nThis competition is a bit of a lottery. Why? Because asset trading is that way. It's probably good that some of you found this out in a competition before losing real money.", "votes": null }, { "id": "1717424", "postDate": "03/09/2022 23:32:34", "content": "One more thing. Does anyone have the final leaderboard scores for the first update? I thought I did, but I only have the incomplete update. I'd like to make some plots of how our scores are moving relative to each other's.
", "rawMarkdown": "One more thing. Does anyone have the final leaderboard scores for the first update? I thought I did, but I only have the incomplete update. I'd like to make some plots of how our scores are moving relative to each other's.", "votes": null }, { "id": "1717647", "postDate": "03/10/2022 05:18:28", "content": "Link for first update leaderboard.
\n\n\nSo their final score would be about .05 * 5/6 + 0 * 1/6 = .042
\n
I don't think you can simply average correlations, if you split a test set into two then calculate the correlation for each subset, the average may not be even close to the correlation of the set as a whole. we can do that with MSE for instance but not Pearson correlation. Correct me if you think i'm wrong.
", "rawMarkdown": "[Link](https://drive.google.com/file/d/1NOQ_D3bmbFALmrDgm-9yiNnehFq-yKZo/view?usp=sharing) for first update leaderboard.\n\n---\n\n\n> So their final score would be about .05 * 5/6 + 0 * 1/6 = .042\n\nI don't think you can simply average correlations, if you split a test set into two then calculate the correlation for each subset, the average may not be even close to the correlation of the set as a whole. we can do that with MSE for instance but not Pearson correlation. Correct me if you think i'm wrong.", "votes": null }, { "id": "1717824", "postDate": "03/10/2022 08:44:58", "content": "Any reason why the current leader board 1st position Tom Forbes is not on the first run?
", "rawMarkdown": "Any reason why the current leader board 1st position Tom Forbes is not on the first run?", "votes": null }, { "id": "1717829", "postDate": "03/10/2022 08:51:33", "content": "I had the same question, I don't have an answer, thought about asking Tom Forbes, there are others also not in the first leaderboard
", "rawMarkdown": "I had the same question, I don't have an answer, thought about asking Tom Forbes, there are others also not in the first leaderboard", "votes": null }, { "id": "1717838", "postDate": "03/10/2022 09:03:26", "content": "Looks like the rabbit hole goes even deeper! Agree with you the aggregation of correlation from week to week; although it might be a fair guess.
", "rawMarkdown": "Looks like the rabbit hole goes even deeper! Agree with you the aggregation of correlation from week to week; although it might be a fair guess.", "votes": null }, { "id": "1717929", "postDate": "03/10/2022 10:59:07", "content": "Ok,
\neven if we disregard the non additivity of correlation,
even if we disregard also the situation of many fellow Kagglers with errored submissions
\nthat are yes, a bit \"nervous\" about it although I find that word quite unfortunate here ( 1 month without answers and everyone here respectful and polite).
can you please explain what exactly are you doing here?
\n\nI expect my model's score to have a standard deviation of .015 * sqrt(6), or .037
\n
Are you trying to infer standard error of one sample population from standard deviation
\nof another population? Really? Is that what you are proposing?
I think it is because some teams have changed their names.
\nMerging by TeamId may solve the problem.
Yes, look at TeamID. Tom Forbes changed his team name to his actual name, I'm guessing because his solve is doing well.
", "rawMarkdown": "Yes, look at TeamID. Tom Forbes changed his team name to his actual name, I'm guessing because his solve is doing well.", "votes": null }, { "id": "1718176", "postDate": "03/10/2022 15:14:26", "content": "@Shady Mokh, thanks for the link.
\nI took a reasonable shortcut because the estimation is very rough anyway. For a correlation you would actually use root mean square, but the answer should be close. When I do it the right way, I get .045 instead of .042. Not a huge difference, when considering that the fall would be from .05
Standard deviations do add, if you add in quadrature. I took a reasonable shortcut, because this is a rough estimate anyway. When I do it right, I get .045 instead of .042.
\nMost of us don't have errored submissions. I think it's reasonable to ask for logs, or supplemental data. However, it may not be wise for the organizer to give that information. They don't have an obligation to do it, and it could be more trouble than it's worth. In any endeavor there's a small percentage of people who can't get over their poor performance and just want to cause trouble.
\nNow for the last part. I was talking about my score mean and standard deviation across folds. Forget the metric, just assume it's normally distributed. (And there's another, perhaps more controversial adjustment for simplicity: that the 12 week score is the average of 2 week scores.) As I'm sure you know, the standard error of the mean equals the standard deviation divided by the square root of the sample size. The error in my estimation of my score goes down with sample size. It was .015 when my sample size was 12 weeks. When it's 2 weeks it should be much larger. Remember, I'm talking about the error in my estimation of my score, not the score itself.
\nIf you want to redo this exactly by the book, go ahead, but I think you'll come to the same general conclusion: the first updates are very poor estimates of our long-term scores.
", "rawMarkdown": "~~Correlations~~ Standard deviations do add, if you add in quadrature. I took a reasonable shortcut, because this is a rough estimate anyway. When I do it right, I get .045 instead of .042. \n\nMost of us don't have errored submissions. I think it's reasonable to ask for logs, or supplemental data. However, it may not be wise for the organizer to give that information. They don't have an obligation to do it, and it could be more trouble than it's worth. In any endeavor there's a small percentage of people who can't get over their poor performance and just want to cause trouble.\n\nNow for the last part. I was talking about my score mean and standard deviation across folds. Forget the metric, just assume it's normally distributed. (And there's another, perhaps more controversial adjustment for simplicity: that the 12 week score is the average of 2 week scores.) As I'm sure you know, the standard error of the mean equals the standard deviation divided by the square root of the sample size. The error in my estimation of my score goes down with sample size. It was .015 when my sample size was 12 weeks. When it's 2 weeks it should be much larger. Remember, I'm talking about the error in my estimation of my score, not the score itself. \n\nIf you want to redo this exactly by the book, go ahead, but I think you'll come to the same general conclusion: the first updates are very poor estimates of our long-term scores.", "votes": null }, { "id": "1718211", "postDate": "03/10/2022 15:51:00", "content": "\n\nThe standard error of the mean equals the standard deviation divided by the square root of the sample size
\n
Sure, but that applies to a population. And means that that you can estimate the standard error of the mean of your 3 month oof scores dividing sd by the square root of number of folds. What has nothing to do with multiplying by 6, it just doesnt work \"backwards\". Population of 3 month scores is different to population of 2 week scores. That estimation is just wrong .
", "rawMarkdown": "> The standard error of the mean equals the standard deviation divided by the square root of the sample size\n\nSure, but that applies to a population. And means that that you can estimate the standard error of the mean of your 3 month oof scores **dividing** sd by the square root of number of folds. What has nothing to do with multiplying by 6, it just doesnt work \"backwards\". **Population of 3 month scores is different to population of 2 week scores**. That estimation is just wrong .", "votes": null }, { "id": "1718247", "postDate": "03/10/2022 16:41:34", "content": "The population I'm proposing is a list of short-term scores. If it's reasonable to assume they're normally distributed, and if the long term scores were the average of short term scores, then everything else would follow.
\nI didn't multiply by 6. I multiplied by the square root of 6.
\nIf you're measuring something that's normally distributed, and you're measuring in groups of 60, and you get a standard deviation of group averages of 1cm, then groups of 10 will have a standard deviation of group averages of about 2.5cm. The standard deviation goes down with larger groups, and up for smaller ones. It does work backwards.
\nThe thing that you should be critiquing is my modification where the 12 week score is the average of 2 week scores. That's the weak spot.
", "rawMarkdown": "The population I'm proposing is a list of short-term scores. If it's reasonable to assume they're normally distributed, and if the long term scores were the average of short term scores, then everything else would follow.\n\nI didn't multiply by 6. I multiplied by the square root of 6. \n\nIf you're measuring something that's normally distributed, and you're measuring in groups of 60, and you get a standard deviation *of group averages* of 1cm, then groups of 10 will have a standard deviation *of group averages* of about 2.5cm. The standard deviation goes down with larger groups, and up for smaller ones. It does work backwards.\n\nThe thing that you should be critiquing is my modification where the 12 week score is the average of 2 week scores. That's the weak spot.", "votes": null }, { "id": "1718402", "postDate": "03/10/2022 19:36:57", "content": "\n\nThe thing that you should be critiquing is my modification where the 12 week score is the average of 2 week scores. That's the weak spot.
\n
Yes, that's what I refer to with they are different populations. I think there's no point in further discussion as we are currently not discussing if the estimation is correct but if the estimation is not so incorrect that can be useful in spite of the assumptions made.
", "rawMarkdown": "> The thing that you should be critiquing is my modification where the 12 week score is the average of 2 week scores. That's the weak spot.\n\nYes, that's what I refer to with they are different populations. I think there's no point in further discussion as we are currently not discussing if the estimation is correct but if the estimation is not so incorrect that can be useful in spite of the assumptions made.", "votes": null }, { "id": "1718477", "postDate": "03/10/2022 21:45:08", "content": "Seems so 😂, last week TomForbes was; 0.0138 under Liquidated name, huge jump in score though!
", "rawMarkdown": "Seems so 😂, last week TomForbes was; 0.0138 under Liquidated name, huge jump in score though!", "votes": null }, { "id": "1718489", "postDate": "03/10/2022 22:18:32", "content": "Yes I changed my team name… I think the large jump was due to one of my features being a ~2 day lag which didn't have a running start, so some of the predictions in the first two days will be slightly off. I thought about coding one in my final sub using supplemental_train but it seemed risky and i found it only had a 0.0001 negative impact on a final 3 month period. However, It might have had a more significant effect on the correlation metric over lower timeframes like 2 weeks
", "rawMarkdown": "Yes I changed my team name... I think the large jump was due to one of my features being a ~2 day lag which didn't have a running start, so some of the predictions in the first two days will be slightly off. I thought about coding one in my final sub using supplemental_train but it seemed risky and i found it only had a 0.0001 negative impact on a final 3 month period. However, It might have had a more significant effect on the correlation metric over lower timeframes like 2 weeks", "votes": null }, { "id": "1719662", "postDate": "03/12/2022 01:02:33", "content": "@tomforbes: brace yourself, it's churn.
", "rawMarkdown": "tomforbes: brace yourself, it's churn.", "votes": null }, { "id": "1719770", "postDate": "03/12/2022 05:17:12", "content": "Is the jump really that large…Too surprising
", "rawMarkdown": "Is the jump really that large...Too surprising", "votes": null }, { "id": "1720150", "postDate": "03/12/2022 14:12:28", "content": "@nathanrm Have you considered the possibility that my solution is actually deserving of 1st place?
", "rawMarkdown": "nathanrm Have you considered the possibility that my solution is actually deserving of 1st place?", "votes": null }, { "id": "1720321", "postDate": "03/12/2022 17:17:52", "content": "Yes, it's a slim possibility at this point, nothing more.
", "rawMarkdown": "Yes, it's a slim possibility at this point, nothing more.", "votes": null }, { "id": "1749654", "postDate": "04/08/2022 20:16:08", "content": "\nRegarding the \"non additivity of correlation\"
\n\nUn saludo muy cordial,
\ncarl
Hi Carl,
\nthanks for that (and for so many cool hints you give frequently here). I made a small comment in the notebook.
\nNaive adding correlation was smaller issue here than the already raised up questions. On the other hand, about p-value analysis, I'm not sure how much correlation of product errors might affect its validity. (Specially if we are dealing with by-date data quality issues).
\nThis competition has been quite dissapointing to me, not because of results (I have already stated my opinion on their validity) but because it erodes my confidence in Kaggle competitions platform that up to this moment was very high.
\nI will almost certainly continue competing hoping this was just an outlier but my intention was, is, to stop adding to my posts in this competition as I already have questioned the validity of it all, my opinion is clear and it drains too much energy from me in this moment.
\nBut the above rule was overriden by the \"answer to @carlmcbrideellis cool comments\" rule :-)
\nUn saludo!
", "rawMarkdown": "Hi Carl, \n\nthanks for that (and for so many cool hints you give frequently here). I made a small comment in the notebook.\n\nNaive adding correlation was smaller issue here than the already raised up questions. On the other hand, about p-value analysis, I'm not sure how much correlation of product errors might affect its validity. (Specially if we are dealing with by-date data quality issues).\n\nThis competition has been quite dissapointing to me, not because of results (I have already stated my opinion on their validity) but because it erodes my confidence in Kaggle competitions platform that up to this moment was very high. \n\nI will almost certainly continue competing hoping this was just an outlier but my intention was, is, to stop adding to my posts in this competition as I already have questioned the validity of it all, my opinion is clear and it drains too much energy from me in this moment.\n\nBut the above rule was overriden by the \"answer to @carlmcbrideellis cool comments\" rule :-)\n\nUn saludo!", "votes": null }, { "id": "1750188", "postDate": "04/09/2022 12:08:53", "content": "Dear @miguelpm
\nI have taken the notebook down for the moment as there is something I am not quite sure of.
\nFor the sake of completeness I shall reproduce your comment on the notebook here, along with my reply:
\n\n\n\"I agree with @jbomitchell that correlation detected with stiched series is nor spurius but true, as the act of stiching two series with different mean imposes a very strong non random structure to the data.\n(Correlation is just another face of (standarized) linear regression, and linear regression would definitely extract the information created by that level-shift in the data.\nSo imo considering spurius that correlation as a starting point is not helping too much in this case\"
\n
Exactly, I totally agree with you both; when I say spurious I guess I really wish to say that I am (fairly sure) that it is incorrect to calculate an over-all Pearson correlation coefficient for a collection of samples that have been drawn from two different populations (which is what is being done when stitching) because stitching together two different populations would create the \"non random structure\" you refer to.
\nThe Pearson correlation coefficient is parametric and assumes that samples are all drawn from a bivariate normal distribution. Having two different populations (with two different means and variances) in the data breaks this assumption.
\nAs an analogy, say we had a dataset of car values with age of car, and a dataset house prices with age of house, it would be meaningless to combine these two datasets together and then calculate an overall age/price correlation coefficient.
\nUn saludo muy cordial,
\ncarl
One more thing. Does anyone have the final leaderboard scores for the first update? I thought I did, but I only have the incomplete update. I'd like to make some plots of how our scores are moving relative to each other's.
", "votes": null, "replies": [ { "id": 1717647, "author_name": "shadymokh", "author_url": "", "post_date": "03/10/2022 05:18:28", "content": "Link for first update leaderboard.
\n\n\nSo their final score would be about .05 * 5/6 + 0 * 1/6 = .042
\n
I don't think you can simply average correlations, if you split a test set into two then calculate the correlation for each subset, the average may not be even close to the correlation of the set as a whole. we can do that with MSE for instance but not Pearson correlation. Correct me if you think i'm wrong.
", "votes": null, "replies": [] }, { "id": 1717824, "author_name": "jonnydossantos", "author_url": "", "post_date": "03/10/2022 08:44:58", "content": "Any reason why the current leader board 1st position Tom Forbes is not on the first run?
", "votes": null, "replies": [] }, { "id": 1717829, "author_name": "shadymokh", "author_url": "", "post_date": "03/10/2022 08:51:33", "content": "I had the same question, I don't have an answer, thought about asking Tom Forbes, there are others also not in the first leaderboard
", "votes": null, "replies": [] }, { "id": 1717838, "author_name": "jonnydossantos", "author_url": "", "post_date": "03/10/2022 09:03:26", "content": "Looks like the rabbit hole goes even deeper! Agree with you the aggregation of correlation from week to week; although it might be a fair guess.
", "votes": null, "replies": [] }, { "id": 1718151, "author_name": "sugghi", "author_url": "", "post_date": "03/10/2022 14:54:30", "content": "I think it is because some teams have changed their names.
\nMerging by TeamId may solve the problem.
Yes, look at TeamID. Tom Forbes changed his team name to his actual name, I'm guessing because his solve is doing well.
", "votes": null, "replies": [] }, { "id": 1718176, "author_name": "nathanrm", "author_url": "", "post_date": "03/10/2022 15:14:26", "content": "@Shady Mokh, thanks for the link.
\nI took a reasonable shortcut because the estimation is very rough anyway. For a correlation you would actually use root mean square, but the answer should be close. When I do it the right way, I get .045 instead of .042. Not a huge difference, when considering that the fall would be from .05
Seems so 😂, last week TomForbes was; 0.0138 under Liquidated name, huge jump in score though!
", "votes": null, "replies": [] }, { "id": 1718489, "author_name": "tomforbes", "author_url": "", "post_date": "03/10/2022 22:18:32", "content": "Yes I changed my team name… I think the large jump was due to one of my features being a ~2 day lag which didn't have a running start, so some of the predictions in the first two days will be slightly off. I thought about coding one in my final sub using supplemental_train but it seemed risky and i found it only had a 0.0001 negative impact on a final 3 month period. However, It might have had a more significant effect on the correlation metric over lower timeframes like 2 weeks
", "votes": null, "replies": [] }, { "id": 1719662, "author_name": "nathanrm", "author_url": "", "post_date": "03/12/2022 01:02:33", "content": "@tomforbes: brace yourself, it's churn.
", "votes": null, "replies": [] }, { "id": 1719770, "author_name": "noi031", "author_url": "", "post_date": "03/12/2022 05:17:12", "content": "Is the jump really that large…Too surprising
", "votes": null, "replies": [] }, { "id": 1720150, "author_name": "tomforbes", "author_url": "", "post_date": "03/12/2022 14:12:28", "content": "@nathanrm Have you considered the possibility that my solution is actually deserving of 1st place?
", "votes": null, "replies": [] }, { "id": 1720321, "author_name": "nathanrm", "author_url": "", "post_date": "03/12/2022 17:17:52", "content": "Yes, it's a slim possibility at this point, nothing more.
", "votes": null, "replies": [] } ] }, { "id": 1717929, "author_name": "miguelpm", "author_url": "", "post_date": "03/10/2022 10:59:07", "content": "Ok,
\neven if we disregard the non additivity of correlation,
even if we disregard also the situation of many fellow Kagglers with errored submissions
\nthat are yes, a bit \"nervous\" about it although I find that word quite unfortunate here ( 1 month without answers and everyone here respectful and polite).
can you please explain what exactly are you doing here?
\n\nI expect my model's score to have a standard deviation of .015 * sqrt(6), or .037
\n
Are you trying to infer standard error of one sample population from standard deviation
\nof another population? Really? Is that what you are proposing?
Standard deviations do add, if you add in quadrature. I took a reasonable shortcut, because this is a rough estimate anyway. When I do it right, I get .045 instead of .042.
\nMost of us don't have errored submissions. I think it's reasonable to ask for logs, or supplemental data. However, it may not be wise for the organizer to give that information. They don't have an obligation to do it, and it could be more trouble than it's worth. In any endeavor there's a small percentage of people who can't get over their poor performance and just want to cause trouble.
\nNow for the last part. I was talking about my score mean and standard deviation across folds. Forget the metric, just assume it's normally distributed. (And there's another, perhaps more controversial adjustment for simplicity: that the 12 week score is the average of 2 week scores.) As I'm sure you know, the standard error of the mean equals the standard deviation divided by the square root of the sample size. The error in my estimation of my score goes down with sample size. It was .015 when my sample size was 12 weeks. When it's 2 weeks it should be much larger. Remember, I'm talking about the error in my estimation of my score, not the score itself.
\nIf you want to redo this exactly by the book, go ahead, but I think you'll come to the same general conclusion: the first updates are very poor estimates of our long-term scores.
", "votes": null, "replies": [] }, { "id": 1718211, "author_name": "miguelpm", "author_url": "", "post_date": "03/10/2022 15:51:00", "content": "\n\nThe standard error of the mean equals the standard deviation divided by the square root of the sample size
\n
Sure, but that applies to a population. And means that that you can estimate the standard error of the mean of your 3 month oof scores dividing sd by the square root of number of folds. What has nothing to do with multiplying by 6, it just doesnt work \"backwards\". Population of 3 month scores is different to population of 2 week scores. That estimation is just wrong .
", "votes": null, "replies": [] }, { "id": 1718247, "author_name": "nathanrm", "author_url": "", "post_date": "03/10/2022 16:41:34", "content": "The population I'm proposing is a list of short-term scores. If it's reasonable to assume they're normally distributed, and if the long term scores were the average of short term scores, then everything else would follow.
\nI didn't multiply by 6. I multiplied by the square root of 6.
\nIf you're measuring something that's normally distributed, and you're measuring in groups of 60, and you get a standard deviation of group averages of 1cm, then groups of 10 will have a standard deviation of group averages of about 2.5cm. The standard deviation goes down with larger groups, and up for smaller ones. It does work backwards.
\nThe thing that you should be critiquing is my modification where the 12 week score is the average of 2 week scores. That's the weak spot.
", "votes": null, "replies": [] }, { "id": 1718402, "author_name": "miguelpm", "author_url": "", "post_date": "03/10/2022 19:36:57", "content": "\n\nThe thing that you should be critiquing is my modification where the 12 week score is the average of 2 week scores. That's the weak spot.
\n
Yes, that's what I refer to with they are different populations. I think there's no point in further discussion as we are currently not discussing if the estimation is correct but if the estimation is not so incorrect that can be useful in spite of the assumptions made.
", "votes": null, "replies": [] }, { "id": 1749654, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "04/08/2022 20:16:08", "content": "\nRegarding the \"non additivity of correlation\"
\n\nUn saludo muy cordial,
\ncarl
Hi Carl,
\nthanks for that (and for so many cool hints you give frequently here). I made a small comment in the notebook.
\nNaive adding correlation was smaller issue here than the already raised up questions. On the other hand, about p-value analysis, I'm not sure how much correlation of product errors might affect its validity. (Specially if we are dealing with by-date data quality issues).
\nThis competition has been quite dissapointing to me, not because of results (I have already stated my opinion on their validity) but because it erodes my confidence in Kaggle competitions platform that up to this moment was very high.
\nI will almost certainly continue competing hoping this was just an outlier but my intention was, is, to stop adding to my posts in this competition as I already have questioned the validity of it all, my opinion is clear and it drains too much energy from me in this moment.
\nBut the above rule was overriden by the \"answer to @carlmcbrideellis cool comments\" rule :-)
\nUn saludo!
", "votes": null, "replies": [] }, { "id": 1750188, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "04/09/2022 12:08:53", "content": "Dear @miguelpm
\nI have taken the notebook down for the moment as there is something I am not quite sure of.
\nFor the sake of completeness I shall reproduce your comment on the notebook here, along with my reply:
\n\n\n\"I agree with @jbomitchell that correlation detected with stiched series is nor spurius but true, as the act of stiching two series with different mean imposes a very strong non random structure to the data.\n(Correlation is just another face of (standarized) linear regression, and linear regression would definitely extract the information created by that level-shift in the data.\nSo imo considering spurius that correlation as a starting point is not helping too much in this case\"
\n
Exactly, I totally agree with you both; when I say spurious I guess I really wish to say that I am (fairly sure) that it is incorrect to calculate an over-all Pearson correlation coefficient for a collection of samples that have been drawn from two different populations (which is what is being done when stitching) because stitching together two different populations would create the \"non random structure\" you refer to.
\nThe Pearson correlation coefficient is parametric and assumes that samples are all drawn from a bivariate normal distribution. Having two different populations (with two different means and variances) in the data breaks this assumption.
\nAs an analogy, say we had a dataset of car values with age of car, and a dataset house prices with age of house, it would be meaningless to combine these two datasets together and then calculate an overall age/price correlation coefficient.
\nUn saludo muy cordial,
\ncarl