When I started looking into validation this morning (the eternal cv vs lb conundrum), top score was ~ .36. Mere hours later, I am back online, perplexed by the behaviour of the model - and top of the LB is > .8. In the words of Ron Burgundy \"well that escalated quickly\"…
", "messages": [ { "id": 1571439, "postDate": "2021-11-04T21:54:00.033Z", "content": "When I started looking into validation this morning (the eternal cv vs lb conundrum), top score was ~ .36. Mere hours later, I am back online, perplexed by the behaviour of the model - and top of the LB is > .8. In the words of Ron Burgundy \"well that escalated quickly\"…
", "rawMarkdown": "When I started looking into validation this morning (the eternal cv vs lb conundrum), top score was ~ .36. Mere hours later, I am back online, perplexed by the behaviour of the model - and top of the LB is > .8. In the words of Ron Burgundy \"well that escalated quickly\"...\n\n", "votes": 11 }, { "id": 1571477, "postDate": "2021-11-04T22:39:04.460Z", "content": "They are widely overfitting. Organisers warned that LB wouldn't be meaningfull. If you want to know what a good corr looks like, check the numerai leaderboard (https://numer.ai/tournament).
", "rawMarkdown": "They are widely overfitting. Organisers warned that LB wouldn't be meaningfull. If you want to know what a good corr looks like, check the numerai leaderboard (https://numer.ai/tournament).", "votes": 7, "replies": [ { "id": 1571481, "postDate": "2021-11-04T22:55:38.733Z", "content": "That's what I was suspecting. I plugged a simple validation on the last 10pct of the data into the top scoring kernel, and I am seeing really weird stuff there: for starters, validation score is getting worse from the first iteration (which means early stopping kicks in as soon as enough rounds pass), while the correlation is order of magnitude lower than what we see on LB. I didn't use the weights, but I don't think using unweighted version explains such a difference.
", "rawMarkdown": "That's what I was suspecting. I plugged a simple validation on the last 10pct of the data into the top scoring kernel, and I am seeing really weird stuff there: for starters, validation score is getting worse from the first iteration (which means early stopping kicks in as soon as enough rounds pass), while the correlation is order of magnitude lower than what we see on LB. I didn't use the weights, but I don't think using unweighted version explains such a difference. ", "votes": 2 }, { "id": 1571507, "postDate": "2021-11-04T23:43:04.370Z", "content": "The corr being magnitude lower is expected (from numerai we can see that the top1 oscillate between 0 and 0.08, there are some difference in the competition but it still is a good benchmark). For the early stopping, it depends on what metric you use for stopping… For the moment, without a good objective, it's difficult to ensure that we are correctly learning.
", "rawMarkdown": "The corr being magnitude lower is expected (from numerai we can see that the top1 oscillate between 0 and 0.08, there are some difference in the competition but it still is a good benchmark). For the early stopping, it depends on what metric you use for stopping... For the moment, without a good objective, it's difficult to ensure that we are correctly learning. ", "votes": 3 }, { "id": 1571510, "postDate": "2021-11-04T23:56:19.440Z", "content": "I tried L1 and L2, same thing: the validation loss keeps creeping up, irrespective of the parameters. If you plot the distribution of targets (per asset) you can see a spike around zero - so we could probably use sth like two-sided tweedie, if that's a thing… Funny enough, a linear model is pretty close in terms of cv-lb gap, which I think might be a hint.
", "rawMarkdown": "I tried L1 and L2, same thing: the validation loss keeps creeping up, irrespective of the parameters. If you plot the distribution of targets (per asset) you can see a spike around zero - so we could probably use sth like two-sided tweedie, if that's a thing... Funny enough, a linear model is pretty close in terms of cv-lb gap, which I think might be a hint.", "votes": 1 }, { "id": 1571879, "postDate": "2021-11-05T08:45:50.533Z", "content": "Well, in terms of cv-lb consistency my best model is in this notebook: https://www.kaggle.com/lucasmorin/minimal-fast-baseline
", "rawMarkdown": "Well, in terms of cv-lb consistency my best model is in this notebook: https://www.kaggle.com/lucasmorin/minimal-fast-baseline", "votes": 2 }, { "id": 1571942, "postDate": "2021-11-05T09:54:48.140Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1627128, "postDate": "2021-12-23T14:27:54.303Z", "content": "From the Numerai leaderboard, I can infer that any Pearson r between 0.0445 and 0.0350 is a good score. This competition is really based on local validation and how to do it properly.
", "rawMarkdown": "From the Numerai leaderboard, I can infer that any Pearson r between 0.0445 and 0.0350 is a good score. This competition is really based on local validation and how to do it properly." } ], "comments": [ { "id": 1571477, "author_name": "Lucas Morin", "author_url": "", "post_date": "2021-11-04T22:39:04.460000", "content": "They are widely overfitting. Organisers warned that LB wouldn't be meaningfull. If you want to know what a good corr looks like, check the numerai leaderboard (https://numer.ai/tournament).
", "votes": 7, "replies": [ { "id": 1571481, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2021-11-04T22:55:38.733000", "content": "That's what I was suspecting. I plugged a simple validation on the last 10pct of the data into the top scoring kernel, and I am seeing really weird stuff there: for starters, validation score is getting worse from the first iteration (which means early stopping kicks in as soon as enough rounds pass), while the correlation is order of magnitude lower than what we see on LB. I didn't use the weights, but I don't think using unweighted version explains such a difference.
", "votes": 2, "replies": [] }, { "id": 1571507, "author_name": "Lucas Morin", "author_url": "", "post_date": "2021-11-04T23:43:04.370000", "content": "The corr being magnitude lower is expected (from numerai we can see that the top1 oscillate between 0 and 0.08, there are some difference in the competition but it still is a good benchmark). For the early stopping, it depends on what metric you use for stopping… For the moment, without a good objective, it's difficult to ensure that we are correctly learning.
", "votes": 3, "replies": [] }, { "id": 1571510, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2021-11-04T23:56:19.440000", "content": "I tried L1 and L2, same thing: the validation loss keeps creeping up, irrespective of the parameters. If you plot the distribution of targets (per asset) you can see a spike around zero - so we could probably use sth like two-sided tweedie, if that's a thing… Funny enough, a linear model is pretty close in terms of cv-lb gap, which I think might be a hint.
", "votes": 1, "replies": [] }, { "id": 1571879, "author_name": "Lucas Morin", "author_url": "", "post_date": "2021-11-05T08:45:50.533000", "content": "Well, in terms of cv-lb consistency my best model is in this notebook: https://www.kaggle.com/lucasmorin/minimal-fast-baseline
", "votes": 2, "replies": [] }, { "id": 1571942, "author_name": "", "author_url": "", "post_date": "2021-11-05T09:54:48.140000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1627128, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-12-23T14:27:54.303000", "content": "From the Numerai leaderboard, I can infer that any Pearson r between 0.0445 and 0.0350 is a good score. This competition is really based on local validation and how to do it properly.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1571439": "When I started looking into validation this morning (the eternal cv vs lb conundrum), top score was ~ .36. Mere hours later, I am back online, perplexed by the behaviour of the model - and top of the LB is > .8. In the words of Ron Burgundy \"well that escalated quickly\"...\n\n", "1571477": "They are widely overfitting. Organisers warned that LB wouldn't be meaningfull. If you want to know what a good corr looks like, check the numerai leaderboard (https://numer.ai/tournament).", "1627128": "From the Numerai leaderboard, I can infer that any Pearson r between 0.0445 and 0.0350 is a good score. This competition is really based on local validation and how to do it properly." } }