Hi, kagglers,\nIt is a nice experience to compete and learn in this competition together. What is your confidence on your submission score? What is your method to prevent overfitting?
\n\nFor me, I have a local validation set split by time and all my model and ensemble plans are test on validation first. But you never know until the competition ends.
", "messages": [ { "id": "125955", "postDate": "07/05/2016 00:43:14", "content": "Hi, kagglers,\nIt is a nice experience to compete and learn in this competition together. What is your confidence on your submission score? What is your method to prevent overfitting?
\n\nFor me, I have a local validation set split by time and all my model and ensemble plans are test on validation first. But you never know until the competition ends.
", "rawMarkdown": "Hi, kagglers,\r\nIt is a nice experience to compete and learn in this competition together. What is your confidence on your submission score? What is your method to prevent overfitting?\r\n\r\nFor me, I have a local validation set split by time and all my model and ensemble plans are test on validation first. But you never know until the competition ends.", "votes": null }, { "id": "125958", "postDate": "07/05/2016 01:18:34", "content": "I think the public lb is reliable up to about the 3rd or 4th digit. With 4 million on public and 4 million on private and a random split between public and private, the statistical error should be quite low, on the order of 0.1%. Of course there is always overfitting, but my submissions usually don't change at the level of the 3rd digit when submitting slightly different models.
\n\nI also think the public lb is much more reliable than local cv, because it more closely resembles the private lb.
", "rawMarkdown": "I think the public lb is reliable up to about the 3rd or 4th digit. With 4 million on public and 4 million on private and a random split between public and private, the statistical error should be quite low, on the order of 0.1%. Of course there is always overfitting, but my submissions usually don't change at the level of the 3rd digit when submitting slightly different models.\r\n\r\nI also think the public lb is much more reliable than local cv, because it more closely resembles the private lb.", "votes": null }, { "id": "125973", "postDate": "07/05/2016 06:19:37", "content": "I expect a different kind of shake up from the competitions I've seen in the past (admittedly, not a lot). I think the law of large numbers comes into play in this dataset, and the results against two random 50% splits should be very close to each other, and therefore not a lot of changes in the LB scores. That said, this is the first time I've seen a competition where you can get a very reliable estimation of your LB score by submitting a small fraction, say 10%, and scaling the result to estimate the full submission. So, I expect there are several people sitting on good submissions until the last day. As I'm writing this I see 13 people enter the top 100 with recent jumps of 100+ .
", "rawMarkdown": "I expect a different kind of shake up from the competitions I've seen in the past (admittedly, not a lot). I think the law of large numbers comes into play in this dataset, and the results against two random 50% splits should be very close to each other, and therefore not a lot of changes in the LB scores. That said, this is the first time I've seen a competition where you can get a very reliable estimation of your LB score by submitting a small fraction, say 10%, and scaling the result to estimate the full submission. So, I expect there are several people sitting on good submissions until the last day. As I'm writing this I see 13 people enter the top 100 with recent jumps of 100+ .", "votes": null }, { "id": "125974", "postDate": "07/05/2016 06:20:07", "content": "", "rawMarkdown": "", "votes": null }, { "id": "125978", "postDate": "07/05/2016 07:42:48", "content": "With a test set on the order of magnitude of 8 million observations, and a public/private split that was apparently done at random, the potential for significant shakeups is essentially zero. I cannot even imagine a strategy that would deliberately overfit the public leaderboard. Small shifts can happen due to changes in the final digits, as @jturkewitz writes above, but that is it.
", "rawMarkdown": "With a test set on the order of magnitude of 8 million observations, and a public/private split that was apparently done at random, the potential for significant shakeups is essentially zero. I cannot even imagine a strategy that would deliberately overfit the public leaderboard. Small shifts can happen due to changes in the final digits, as @jturkewitz writes above, but that is it.", "votes": null }, { "id": "126045", "postDate": "07/05/2016 19:39:40", "content": "I did my tests on 2.5% of the data. If I overfitted, and probably I did, I overfitted just on that portion of the data, the rest of 97.5% should average things out and bring my score down to the real model accuracy.
", "rawMarkdown": "I did my tests on 2.5% of the data. If I overfitted, and probably I did, I overfitted just on that portion of the data, the rest of 97.5% should average things out and bring my score down to the real model accuracy.", "votes": null }, { "id": "126059", "postDate": "07/05/2016 22:36:38", "content": "@Andre. Its not about deliberately overfitting. Several people have noted you could submit 10% of the entries and the resulting score is very close to 10% of the score you would have received for a full submission. In this contest that saves a lot of compute time AND for those stealthy gamer types, allows them to check their potential LB standing without tipping their hand. Unfortunately for me, multiplying my scores by 10x doesn't put me in contention, but I'll be very surprised if the current winning submission (0.62120) stands for the next 24 hours.
", "rawMarkdown": "Andre. Its not about deliberately overfitting. Several people have noted you could submit 10% of the entries and the resulting score is very close to 10% of the score you would have received for a full submission. In this contest that saves a lot of compute time AND for those stealthy gamer types, allows them to check their potential LB standing without tipping their hand. Unfortunately for me, multiplying my scores by 10x doesn't put me in contention, but I'll be very surprised if the current winning submission (0.62120) stands for the next 24 hours.", "votes": null }, { "id": "126060", "postDate": "07/05/2016 23:00:12", "content": "@Dave Hurst. Wow, this sounds scaring. It is like a lot of predators are sitting behind and will eat the top 10 at the end...
", "rawMarkdown": "Dave Hurst. Wow, this sounds scaring. It is like a lot of predators are sitting behind and will eat the top 10 at the end...", "votes": null }, { "id": "126062", "postDate": "07/05/2016 23:19:07", "content": "[quote=Li Li;126060]
\n\n@Dave Hurst. Wow, this sounds scaring. It is like a lot of predators are sitting behind and will eat the top 10 at the end...
\n\n[/quote]
\n\nI don't think this will happen. Most people don't really care if you know their score, so they would just submit to the lb. I wouldn't be surprised to see the top score beat, but it is more likely that someone in the top 50 beats it than someone outside the top 50. It actually is kind of remarkable that no positions in the top 6 have changed based on the public lb. It will be interesting to see what the top 3 are doing that no one else is, as that is a significant improvement over the field. One theory is that the place id probabilities are not independent of each other, though I haven't figured out a good way to test this.
", "rawMarkdown": "[quote=Li Li;126060]\r\n\r\n@Dave Hurst. Wow, this sounds scaring. It is like a lot of predators are sitting behind and will eat the top 10 at the end...\r\n\r\n[/quote]\r\n\r\nI don't think this will happen. Most people don't really care if you know their score, so they would just submit to the lb. I wouldn't be surprised to see the top score beat, but it is more likely that someone in the top 50 beats it than someone outside the top 50. It actually is kind of remarkable that no positions in the top 6 have changed based on the public lb. It will be interesting to see what the top 3 are doing that no one else is, as that is a significant improvement over the field. One theory is that the place id probabilities are not independent of each other, though I haven't figured out a good way to test this.", "votes": null }, { "id": "126070", "postDate": "07/05/2016 23:48:50", "content": "24 for hours to go, and we will know.
", "rawMarkdown": "24 for hours to go, and we will know.", "votes": null }, { "id": "126091", "postDate": "07/06/2016 04:07:10", "content": "The next 24 hours will be interesting, that's for sure! We are about to learn a lot about the mindset of fellow Kagglers! I never expected to top the leaderboard and am still convinced that some "predators" are going to eat us alive. All we can do for now is to keep running :)
\n\nI will be happy to share the details of my approach with the other competitors after the deadline. Like Jared I spent quite some time looking for "clusters" of place_ids but I was also unable to find obvious patterns.
\n\nOn another note, with this amount of data and the restriction of single person teams, the winning submission is likely to be significantly worse than the entitlement and it would be interesting to see the limits of the data set by ensembling the top predictions after the deadline.
", "rawMarkdown": "The next 24 hours will be interesting, that's for sure! We are about to learn a lot about the mindset of fellow Kagglers! I never expected to top the leaderboard and am still convinced that some \"predators\" are going to eat us alive. All we can do for now is to keep running :)\r\n\r\nI will be happy to share the details of my approach with the other competitors after the deadline. Like Jared I spent quite some time looking for \"clusters\" of place_ids but I was also unable to find obvious patterns.\r\n\r\nOn another note, with this amount of data and the restriction of single person teams, the winning submission is likely to be significantly worse than the entitlement and it would be interesting to see the limits of the data set by ensembling the top predictions after the deadline.", "votes": null }, { "id": "126108", "postDate": "07/06/2016 10:04:56", "content": "Hi Tom Van de Wiele, that would be nice. The Kaggle Overfitting Community would like to tweak the code of the top LB predictions and then ensemble them :) \nWhat do you say?
", "rawMarkdown": "Hi Tom Van de Wiele, that would be nice. The Kaggle Overfitting Community would like to tweak the code of the top LB predictions and then ensemble them :) \r\nWhat do you say?", "votes": null }, { "id": "126109", "postDate": "07/06/2016 10:09:27", "content": "Sure!
", "rawMarkdown": "Sure!", "votes": null }, { "id": "126170", "postDate": "07/07/2016 00:05:18", "content": "This has to be one of the lowest shakeups in kaggle history.
", "rawMarkdown": "This has to be one of the lowest shakeups in kaggle history.", "votes": null }, { "id": "126175", "postDate": "07/07/2016 00:12:21", "content": "Indeed. Fitting for a test set that was both substantial and benign.
", "rawMarkdown": "Indeed. Fitting for a test set that was both substantial and benign.", "votes": null }, { "id": "126270", "postDate": "07/07/2016 08:12:25", "content": "0.00276 beats "See Click Predict Fix" (0.004)
", "rawMarkdown": "0.00276 beats \"See Click Predict Fix\" (0.004)", "votes": null } ], "comments": [ { "id": 125958, "author_name": "jturkewitz", "author_url": "", "post_date": "07/05/2016 01:18:34", "content": "I think the public lb is reliable up to about the 3rd or 4th digit. With 4 million on public and 4 million on private and a random split between public and private, the statistical error should be quite low, on the order of 0.1%. Of course there is always overfitting, but my submissions usually don't change at the level of the 3rd digit when submitting slightly different models.
\n\nI also think the public lb is much more reliable than local cv, because it more closely resembles the private lb.
", "votes": null, "replies": [] }, { "id": 125973, "author_name": "datadave", "author_url": "", "post_date": "07/05/2016 06:19:37", "content": "I expect a different kind of shake up from the competitions I've seen in the past (admittedly, not a lot). I think the law of large numbers comes into play in this dataset, and the results against two random 50% splits should be very close to each other, and therefore not a lot of changes in the LB scores. That said, this is the first time I've seen a competition where you can get a very reliable estimation of your LB score by submitting a small fraction, say 10%, and scaling the result to estimate the full submission. So, I expect there are several people sitting on good submissions until the last day. As I'm writing this I see 13 people enter the top 100 with recent jumps of 100+ .
", "votes": null, "replies": [] }, { "id": 125974, "author_name": "datadave", "author_url": "", "post_date": "07/05/2016 06:20:07", "content": "", "votes": null, "replies": [] }, { "id": 125978, "author_name": "andrenaef", "author_url": "", "post_date": "07/05/2016 07:42:48", "content": "With a test set on the order of magnitude of 8 million observations, and a public/private split that was apparently done at random, the potential for significant shakeups is essentially zero. I cannot even imagine a strategy that would deliberately overfit the public leaderboard. Small shifts can happen due to changes in the final digits, as @jturkewitz writes above, but that is it.
", "votes": null, "replies": [] }, { "id": 126045, "author_name": "radustoicescu", "author_url": "", "post_date": "07/05/2016 19:39:40", "content": "I did my tests on 2.5% of the data. If I overfitted, and probably I did, I overfitted just on that portion of the data, the rest of 97.5% should average things out and bring my score down to the real model accuracy.
", "votes": null, "replies": [] }, { "id": 126059, "author_name": "datadave", "author_url": "", "post_date": "07/05/2016 22:36:38", "content": "@Andre. Its not about deliberately overfitting. Several people have noted you could submit 10% of the entries and the resulting score is very close to 10% of the score you would have received for a full submission. In this contest that saves a lot of compute time AND for those stealthy gamer types, allows them to check their potential LB standing without tipping their hand. Unfortunately for me, multiplying my scores by 10x doesn't put me in contention, but I'll be very surprised if the current winning submission (0.62120) stands for the next 24 hours.
", "votes": null, "replies": [] }, { "id": 126060, "author_name": "aikinogard", "author_url": "", "post_date": "07/05/2016 23:00:12", "content": "@Dave Hurst. Wow, this sounds scaring. It is like a lot of predators are sitting behind and will eat the top 10 at the end...
", "votes": null, "replies": [] }, { "id": 126062, "author_name": "jturkewitz", "author_url": "", "post_date": "07/05/2016 23:19:07", "content": "[quote=Li Li;126060]
\n\n@Dave Hurst. Wow, this sounds scaring. It is like a lot of predators are sitting behind and will eat the top 10 at the end...
\n\n[/quote]
\n\nI don't think this will happen. Most people don't really care if you know their score, so they would just submit to the lb. I wouldn't be surprised to see the top score beat, but it is more likely that someone in the top 50 beats it than someone outside the top 50. It actually is kind of remarkable that no positions in the top 6 have changed based on the public lb. It will be interesting to see what the top 3 are doing that no one else is, as that is a significant improvement over the field. One theory is that the place id probabilities are not independent of each other, though I haven't figured out a good way to test this.
", "votes": null, "replies": [] }, { "id": 126070, "author_name": "andrenaef", "author_url": "", "post_date": "07/05/2016 23:48:50", "content": "24 for hours to go, and we will know.
", "votes": null, "replies": [] }, { "id": 126091, "author_name": "tvdwiele", "author_url": "", "post_date": "07/06/2016 04:07:10", "content": "The next 24 hours will be interesting, that's for sure! We are about to learn a lot about the mindset of fellow Kagglers! I never expected to top the leaderboard and am still convinced that some "predators" are going to eat us alive. All we can do for now is to keep running :)
\n\nI will be happy to share the details of my approach with the other competitors after the deadline. Like Jared I spent quite some time looking for "clusters" of place_ids but I was also unable to find obvious patterns.
\n\nOn another note, with this amount of data and the restriction of single person teams, the winning submission is likely to be significantly worse than the entitlement and it would be interesting to see the limits of the data set by ensembling the top predictions after the deadline.
", "votes": null, "replies": [] }, { "id": 126108, "author_name": "drarfc", "author_url": "", "post_date": "07/06/2016 10:04:56", "content": "Hi Tom Van de Wiele, that would be nice. The Kaggle Overfitting Community would like to tweak the code of the top LB predictions and then ensemble them :) \nWhat do you say?
", "votes": null, "replies": [] }, { "id": 126109, "author_name": "tvdwiele", "author_url": "", "post_date": "07/06/2016 10:09:27", "content": "Sure!
", "votes": null, "replies": [] }, { "id": 126170, "author_name": "radustoicescu", "author_url": "", "post_date": "07/07/2016 00:05:18", "content": "This has to be one of the lowest shakeups in kaggle history.
", "votes": null, "replies": [] }, { "id": 126175, "author_name": "andrenaef", "author_url": "", "post_date": "07/07/2016 00:12:21", "content": "Indeed. Fitting for a test set that was both substantial and benign.
", "votes": null, "replies": [] }, { "id": 126270, "author_name": "glimmung", "author_url": "", "post_date": "07/07/2016 08:12:25", "content": "0.00276 beats "See Click Predict Fix" (0.004)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "125955": "Hi, kagglers,\r\nIt is a nice experience to compete and learn in this competition together. What is your confidence on your submission score? What is your method to prevent overfitting?\r\n\r\nFor me, I have a local validation set split by time and all my model and ensemble plans are test on validation first. But you never know until the competition ends.", "125958": "I think the public lb is reliable up to about the 3rd or 4th digit. With 4 million on public and 4 million on private and a random split between public and private, the statistical error should be quite low, on the order of 0.1%. Of course there is always overfitting, but my submissions usually don't change at the level of the 3rd digit when submitting slightly different models.\r\n\r\nI also think the public lb is much more reliable than local cv, because it more closely resembles the private lb.", "125973": "I expect a different kind of shake up from the competitions I've seen in the past (admittedly, not a lot). I think the law of large numbers comes into play in this dataset, and the results against two random 50% splits should be very close to each other, and therefore not a lot of changes in the LB scores. That said, this is the first time I've seen a competition where you can get a very reliable estimation of your LB score by submitting a small fraction, say 10%, and scaling the result to estimate the full submission. So, I expect there are several people sitting on good submissions until the last day. As I'm writing this I see 13 people enter the top 100 with recent jumps of 100+ .", "125974": "", "125978": "With a test set on the order of magnitude of 8 million observations, and a public/private split that was apparently done at random, the potential for significant shakeups is essentially zero. I cannot even imagine a strategy that would deliberately overfit the public leaderboard. Small shifts can happen due to changes in the final digits, as @jturkewitz writes above, but that is it.", "126045": "I did my tests on 2.5% of the data. If I overfitted, and probably I did, I overfitted just on that portion of the data, the rest of 97.5% should average things out and bring my score down to the real model accuracy.", "126059": "Andre. Its not about deliberately overfitting. Several people have noted you could submit 10% of the entries and the resulting score is very close to 10% of the score you would have received for a full submission. In this contest that saves a lot of compute time AND for those stealthy gamer types, allows them to check their potential LB standing without tipping their hand. Unfortunately for me, multiplying my scores by 10x doesn't put me in contention, but I'll be very surprised if the current winning submission (0.62120) stands for the next 24 hours.", "126060": "Dave Hurst. Wow, this sounds scaring. It is like a lot of predators are sitting behind and will eat the top 10 at the end...", "126062": "[quote=Li Li;126060]\r\n\r\n@Dave Hurst. Wow, this sounds scaring. It is like a lot of predators are sitting behind and will eat the top 10 at the end...\r\n\r\n[/quote]\r\n\r\nI don't think this will happen. Most people don't really care if you know their score, so they would just submit to the lb. I wouldn't be surprised to see the top score beat, but it is more likely that someone in the top 50 beats it than someone outside the top 50. It actually is kind of remarkable that no positions in the top 6 have changed based on the public lb. It will be interesting to see what the top 3 are doing that no one else is, as that is a significant improvement over the field. One theory is that the place id probabilities are not independent of each other, though I haven't figured out a good way to test this.", "126070": "24 for hours to go, and we will know.", "126091": "The next 24 hours will be interesting, that's for sure! We are about to learn a lot about the mindset of fellow Kagglers! I never expected to top the leaderboard and am still convinced that some \"predators\" are going to eat us alive. All we can do for now is to keep running :)\r\n\r\nI will be happy to share the details of my approach with the other competitors after the deadline. Like Jared I spent quite some time looking for \"clusters\" of place_ids but I was also unable to find obvious patterns.\r\n\r\nOn another note, with this amount of data and the restriction of single person teams, the winning submission is likely to be significantly worse than the entitlement and it would be interesting to see the limits of the data set by ensembling the top predictions after the deadline.", "126108": "Hi Tom Van de Wiele, that would be nice. The Kaggle Overfitting Community would like to tweak the code of the top LB predictions and then ensemble them :) \r\nWhat do you say?", "126109": "Sure!", "126170": "This has to be one of the lowest shakeups in kaggle history.", "126175": "Indeed. Fitting for a test set that was both substantial and benign.", "126270": "0.00276 beats \"See Click Predict Fix\" (0.004)" }, "source": "meta" }