![\"enter](\"http://i.imgur.com/pnb6SKV.png\")
Here is a new thread, so those over there can continue to search for their magic feature.
\n\n@GWillie\nI genuinely respect your determination (despite the downvotes) to defend your argument with such thorough posts (23 well written posts, facts aside),
\n\nI also respect the entire kaggle community, as this seems to be what kaggle was made for, not just competitions (predictive modeling or not), but the open exchange and communication about data science...
\n\nNow I invite you @GWillie to state your entire argument(s), as I (and I'm sure others as well) will attempt to address the question:
\n\nIs This Truly Predictive Modeling?
", "messages": [ { "id": 144382, "postDate": "2016-11-14T03:14:59.650Z", "content": "[quote=GWillie;144371]
\n\nThis in no way refutes the point I made about Kaggle being an overfitting forum and the fact that companies are paying (wasting) a lot of money thinking they are getting predictive models when in fact they are getting models that are overfit to existing data.\n[/quote]
\n\nDefinition of overfitting, the stroke area:
\n\n
I think you should get back to statistical modeling theory before writing back (like: what is overfitting and what is underfitting). Not only you seem to not know predictive modeling is a process and there's no true/false predictive modeling (provide us your research paper about it if you are willing to challenge it), that process does not rely on assumptions of truthness of information to be a process (VERY IMPORTANT), and if you know your theory about predictive modeling, the process is an endless loop.
\n\nHere is an example from one of my slides with IBM which proves your "false predictive modeling" does not exist (only "predictive modeling" is existing), as errors found on the way are corrected, and the loop goes on indefinitely until a satisfactory solution by all parties is found:
\n\n![\"enter](\"http://i.imgur.com/EZLBYzN.png\")
More or less there's a bias you have and you can't even explain it using theory, therefore no one can understand what you are trying to explain us. Our models are not stopping in the hands on Bosch, we are only a helping part of the process for them and they will have to find a way to exploit them. The story stops here for us at Kaggle, and we may never know what will happen to the models: they could be thrown out or even improved, but this is not an issue on our hands.
\n\nThat's how Kaggle works: you are just an element of crowdsourcing to help predictive modeling to be carried on for the company who paid Kaggle to get help. The wealth of information provided from crowdsourcing is something you cannot find using your own data scientists. Anything found, even a minor detail, can be useful from a design perspective.
\n\nI think you don't get that companies are not paying only for the models. Insights are actually more important as the valuation (non-monetary) kicks in - it is not only about "numbers". You should read what is asked in the winners' documentation. I also think enough time was wasted in a topic ("true predictive modeling, definition still not given?") that does not make any sense, as the theory you learnt does not seem assimilated correctly.
\n\nNow that your "true/false predictive modeling" potential definition was invalidated, are you going to say the theory (which, for the predictive modeling process, is based on pragmatic evidence throughout the years that Data Mining is existing) is wrong? If yes, do you have any empirical and significant proof? (hint: there's none - there's a reason it's called a "process" which loops over itself like a snake, thus becomes invulnerable to errors as they are theoretically auto-corrected)
\n\nExtra: If you invaildate "they are theoretically auto-corrected" using practical sources (because why not, humans are prone to errors in a process), you end up with a single theoretical "false modeling" which is obviously generalizing to all processes as a human must be introduced somewhere. Therefore, there's only one process existing, not the existance of two separate processes (True/False). This is what we call a "Railing" in any existing process in this world, and that's what is supposed to prevent such mistake (if they happen to be created by humans). If your overfitting conviction was true, the Private Leaderboard is supposed to be entirely scrambled, in absolute values and not in rank. Obviously, this is not the case, overfitting was nearly non-existant (if not at all for many of us).
\n\nHence, both in theory and in practice your True/False predictive modeling hypotheses (we don't even know what they are, but we don't need to know that) are proven wrong.
\n\n(off-topic: self-note for re-usage of this wall of text for next data science course to teach, copy&paste text)
\n\n(edit: removed IBM from print screen)
", "rawMarkdown": "[quote=GWillie;144371]\r\n\r\nThis in no way refutes the point I made about Kaggle being an overfitting forum and the fact that companies are paying (wasting) a lot of money thinking they are getting predictive models when in fact they are getting models that are overfit to existing data.\r\n[/quote]\r\n\r\nDefinition of overfitting, the stroke area:\r\n\r\n![enter image description here][1]\r\n\r\nI think you should get back to **statistical modeling theory** before writing back (like: **what is overfitting and what is underfitting**). Not only you seem to not know **predictive modeling is a process** and there's **no true/false predictive modeling** (provide us your research paper about it if you are willing to challenge it), that process **does not rely on assumptions of truthness of information to be a process (VERY IMPORTANT)**, and if you know your theory about predictive modeling, the process is an **endless loop**.\r\n\r\nHere is an example from one of my slides with IBM which proves your \"false predictive modeling\" does not exist (**only \"predictive modeling\" is existing**), as errors found on the way are corrected, and the loop goes on indefinitely until a satisfactory solution by all parties is found:\r\n\r\n![enter image description here][2]\r\n\r\nMore or less there's a bias you have and you can't even explain it using theory, therefore no one can understand what you are trying to explain us. Our models are not stopping in the hands on Bosch, we are only a helping part of the process for them and they will have to find a way to exploit them. The story stops here for us at Kaggle, and **we may never know what will happen to the models**: they could be thrown out or even improved, but this is not an issue on our hands.\r\n\r\nThat's how Kaggle works: you are just an **element of crowdsourcing to help predictive modeling to be carried on** for the company who paid Kaggle to get help. **The wealth of information provided from crowdsourcing is something you cannot find using your own data scientists.** Anything found, even a minor detail, can be useful from a design perspective.\r\n\r\nI think you don't get that companies are not paying only for the models. Insights are actually more important as the valuation (non-monetary) kicks in - it is not only about \"numbers\". You should read what is asked in the [winners' documentation][3]. I also think enough time was wasted in a topic (\"true predictive modeling, definition still not given?\") that does not make any sense, as the theory you learnt does not seem assimilated correctly.\r\n\r\nNow that your **\"true/false predictive modeling\" potential definition was invalidated**, are you going to say the theory (which, for the predictive modeling process, is based on pragmatic evidence throughout the years that Data Mining is existing) is wrong? If yes, **do you have any empirical and significant proof?** (hint: there's none - there's a reason it's called a **\"process\" which loops over itself** like a snake, thus becomes **invulnerable to errors as they are theoretically auto-corrected**)\r\n\r\nExtra: If you invaildate \"they are theoretically auto-corrected\" using practical sources (because why not, humans are prone to errors in a process), you end up with a single theoretical \"false modeling\" which is obviously generalizing to all processes as a human must be introduced somewhere. Therefore, there's **only one process existing**, not the existance of two separate processes (True/False). This is what we call a **\"Railing\"** in any existing process in this world, and that's what is supposed to prevent such mistake (if they happen to be created by humans). If your overfitting conviction was true, the Private Leaderboard is supposed to be entirely scrambled, in **absolute values** and not in rank. Obviously, this is not the case, overfitting was nearly non-existant (if not at all for many of us).\r\n\r\nHence, both in theory and in practice your True/False predictive modeling hypotheses (we don't even know what they are, but we don't need to know that) are proven wrong.\r\n\r\n(off-topic: self-note for re-usage of this wall of text for next data science course to teach, copy&paste text)\r\n\r\n(edit: removed IBM from print screen)\r\n\r\n [1]: http://i.imgur.com/pnb6SKV.png\r\n [2]: http://i.imgur.com/EZLBYzN.png\r\n [3]: https://www.kaggle.com/wiki/WinningModelDocumentationTemplate", "votes": 9 }, { "id": 144336, "postDate": "2016-11-13T20:00:47.820Z", "content": "[quote=GWillie;144308]
\n\nThat will give you a higher score and better fits on the available data (thus the overfitting).
\n\n[/quote]
\n\nLeaderboard Public vs Private stability check: check
\n\n![\"enter](\"http://i.imgur.com/owBMhKj.png\")
![\"enter](\"http://i.imgur.com/TqcVIwn.png\")
If we all were truly overfitting no one of us should appear there, cf Santander. The difference is negligible in absolute terms, post-bias correction. Even in Santander, it was not even severe (but in terms of ranks it was severe enough as everyone was very close vs 0.50 random).
\n\nIn any case, following your statement, if a satellite makes a perfect circle (r = fixed distance) around the Earth, it should mean your algorithm to move the rocket periodically is overfitting the circle and cannot generalize to new circles when you will provide another distance. Isn't that too much naive? Obviously, your would try to make your algorithm generalize to any distance you input, not to do perfect on r=10000 only and wreck havoc on r=10000-0.0000000000000000001 which does not make sense if we follow what you said.
\n\nBut yes, in this world humans are making the algorithms for the satellites so they can orbit around the earth without wrecking havoc if you change something by 1 inch. If your statement was true, expect no satellites existing as they would all be destroyed or flying somewhere else without any possible control.
", "rawMarkdown": "[quote=GWillie;144308]\r\n\r\nThat will give you a **higher score and better fits on the available data (thus the overfitting)**.\r\n\r\n[/quote]\r\n\r\n* Noisy data: check\r\n* Imbalanced data: check\r\n* 30% samples only on Public LB: check\r\n\r\nLeaderboard Public vs Private stability check: check\r\n\r\n![enter image description here][1]\r\n\r\n![enter image description here][2]\r\n\r\nIf we all were truly overfitting no one of us should appear there, cf Santander. The difference is negligible in absolute terms, post-bias correction. Even in Santander, it was not even severe (but in terms of ranks it was severe enough as everyone was very close vs 0.50 random).\r\n\r\nIn any case, following your statement, if a satellite makes a perfect circle (r = fixed distance) around the Earth, it should mean your algorithm to move the rocket periodically is overfitting the circle and cannot generalize to new circles when you will provide another distance. Isn't that too much naive? Obviously, your would try to make your algorithm generalize to any distance you input, not to do perfect on r=10000 only and wreck havoc on r=10000-0.0000000000000000001 which does not make sense if we follow what you said.\r\n\r\nBut yes, in this world humans are making the algorithms for the satellites so they can orbit around the earth without wrecking havoc if you change something by 1 inch. If your statement was true, expect no satellites existing as they would all be destroyed or flying somewhere else without any possible control.\r\n\r\n [1]: http://i.imgur.com/owBMhKj.png\r\n [2]: http://i.imgur.com/TqcVIwn.png", "votes": 10 }, { "id": 144581, "postDate": "2016-11-14T17:54:44.810Z", "content": "@GWillie, I'm completely puzzled by your arguments. You seem to claim that some US election predictions being incorrect therefore shows that machine learning is bad? I may not have an education in statistics, but I think that your sample size of 1 may not be representative of the entire population.
\n\nI also don't understand why you think we're overfitting. The private leaderboard had 700,000 unseen samples that we had to predict on, and the leaderboard shows that people did not overfit very much to the 300k samples that they could see the results on versus the private set.
\n\nSince the goal of a machine learning model is to fit to unseen data, and here we have proof that our models can fit to (a considerable amount of) unseen data, logically we can say that our machine learning is not overfitting. How could we have overfitted to the private test set?
\n\nAlso, an example of what you consider to be predictive modelling would be great, so we can actually understand your view. Right now it might as well be phoning up your local fortune teller and asking them who's going to win the election.
\n\nPS: You're being downvoted because you're not actually making any logical discussion, rather resorting to attacks on well-respected forum members and saying "your argument is wrong" without providing a counterview. If you want a logical discussion, then you can start by presenting your view (i.e. what is predictive modelling?)
", "rawMarkdown": "@GWillie, I'm completely puzzled by your arguments. You seem to claim that some US election predictions being incorrect therefore shows that machine learning is bad? I may not have an education in statistics, but I think that your sample size of 1 may not be representative of the entire population.\r\n\r\nI also don't understand why you think we're overfitting. The private leaderboard had 700,000 **unseen** samples that we had to predict on, and the leaderboard shows that people did not overfit very much to the 300k samples that they could see the results on versus the private set.\r\n\r\nSince the goal of a machine learning model is to fit to unseen data, and here we have proof that our models can fit to (a considerable amount of) unseen data, logically we can say that our machine learning is not overfitting. How could we have overfitted to the **private test set**?\r\n\r\nAlso, an example of what you consider to be predictive modelling would be great, so we can actually understand your view. Right now it might as well be phoning up your local fortune teller and asking them who's going to win the election.\r\n\r\nPS: You're being downvoted because you're not actually making any logical discussion, rather resorting to attacks on well-respected forum members and saying \"your argument is wrong\" without providing a counterview. If you want a logical discussion, then you can start by presenting your view (i.e. what is predictive modelling?)", "votes": 8 }, { "id": 138393, "postDate": "2016-10-08T14:45:28.170Z", "content": "Here is a new thread, so those over there can continue to search for their magic feature.
\n\n@GWillie\nI genuinely respect your determination (despite the downvotes) to defend your argument with such thorough posts (23 well written posts, facts aside),
\n\nI also respect the entire kaggle community, as this seems to be what kaggle was made for, not just competitions (predictive modeling or not), but the open exchange and communication about data science...
\n\nNow I invite you @GWillie to state your entire argument(s), as I (and I'm sure others as well) will attempt to address the question:
\n\nIs This Truly Predictive Modeling?
", "rawMarkdown": "Here is a new thread, so those [over there][1] can continue to search for their magic feature.\r\n\r\n[@GWillie][2]\r\nI genuinely respect your determination (despite the downvotes) to defend your argument with such thorough posts (23 well written posts, facts aside), \r\n\r\nI also respect the entire kaggle community, as this seems to be what kaggle was made for, not just competitions (predictive modeling or not), but **the open exchange and communication about data science...**\r\n\r\nNow I invite you @GWillie to state your entire argument(s), as I (and I'm sure others as well) will attempt to address the question:\r\n\r\n**Is This Truly Predictive Modeling?**\r\n\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/24065/the-magical-feature-from-lb-0-3-to-0-4\r\n [2]: https://www.kaggle.com/gwillie", "votes": 7 }, { "id": 144578, "postDate": "2016-11-14T17:44:18.040Z", "content": "I made a decision early on never to vote down Kaggle posts. Boy, have I been tempted ...
", "rawMarkdown": "I made a decision early on never to vote down Kaggle posts. Boy, have I been tempted ...\r\n", "votes": 5 }, { "id": 144575, "postDate": "2016-11-14T17:37:18.313Z", "content": "[quote=GWillie;144387]
\n\nWow, that was a whole lot of poor English and poor logic.
\n[/quote]
Attacking a non native speaker on her use of English is not really helpful. Let's switch to French to see how good you are at discussing in a foreign language...
\n\nMore generally, personal attacks aren't great in forums. Stick to ideas and facts, and please do not attack the person.
", "rawMarkdown": "[quote=GWillie;144387]\r\n\r\nWow, that was a whole lot of poor English and poor logic. \r\n[/quote]\r\n\r\nAttacking a non native speaker on her use of English is not really helpful. Let's switch to French to see how good you are at discussing in a foreign language...\r\n\r\nMore generally, personal attacks aren't great in forums. Stick to ideas and facts, and please do not attack the person.\r\n\r\n", "votes": 5 }, { "id": 144614, "postDate": "2016-11-14T21:06:10.130Z", "content": "Ok, @GWillie. I am willing to assume you are right.
\n\n1) What would you do to "fix" overfitting on Kaggle? Disable Public Leaderboard? Not provide test data at all (ask people to submit their fully documented models)?
\n\n2) How would you ensure quality and consistency of training data gathering and preparation by hosting companies to increase usefulness of the models and their ability to generalize to unseen (possibly even not yet collected) data?
\n\n3) How would you go about assembling and incentivizing the data science community around your perfect data science process (is KDN better in your view, by the way)?
\n\nI will dare to say that leaderboard shakeups are part of the thrill and advancing by 0.000001 on the leaderboard is what keeps people glued to the monitors for weeks towards the end of competition. Teams are formed, people learn (statistics and computer science), virtual coins are distributed, people have fun! You don't like this game? You don't have to play it! And, assuming your name is not Nate, go build a better predictive model for US elections! Beat @FiveThirtyEight and be right every time!! We all here will thank you and bow our heads in respect.
", "rawMarkdown": "Ok, @GWillie. I am willing to assume you are right. \r\n\r\n1) What would you do to \"fix\" overfitting on Kaggle? Disable Public Leaderboard? Not provide test data at all (ask people to submit their fully documented models)? \r\n\r\n2) How would you ensure quality and consistency of training data gathering and preparation by hosting companies to increase usefulness of the models and their ability to generalize to unseen (possibly even not yet collected) data?\r\n\r\n3) How would you go about assembling and incentivizing the data science community around your perfect data science process (is KDN better in your view, by the way)? \r\n\r\nI will dare to say that leaderboard shakeups are part of the thrill and advancing by 0.000001 on the leaderboard is what keeps people glued to the monitors for weeks towards the end of competition. Teams are formed, people learn (statistics and computer science), virtual coins are distributed, people have fun! You don't like this game? You don't have to play it! And, assuming your name is not Nate, go build a better predictive model for US elections! Beat @FiveThirtyEight and be right every time!! We all here will thank you and bow our heads in respect.", "votes": 6 }, { "id": 138405, "postDate": "2016-10-08T16:42:53.133Z", "content": "It is. This is a factory dataset, you want to evaluate if the product is defective or not. Bosh knows about defective lots so it is highly improbable they would make a mistake like leaking a defective product on the ID feature. (factory are highly correlated within its process so it is obvious feature engineering will bring HUGE improvements) EDIT:(wrong assumption... yet they shared the leak!)
\n\nI have been a kaggle member recently and all i can say is that this is the perfect place to learn. People share knowledge without asking anything in return and you can learn way faster that you could on your own, or in the university. (i speak for myself, and i thank for such great people on the community)
\n\nThe post you are referring is sad. All the rating... You win competitions if you go in with the right mindset, and because this area is very challenging you have to be able to learn and innovate your approach. You will never win a competition by slightly changing some parameters on a public kernel. So study, ask questions and people will help you.
\n\nFor instance, in the red hat competition some people even went public with the leak so everybody can learn what it looks like and how to search for it, in case you ever find one which wreaks you model in real life.
", "rawMarkdown": "It is. This is a factory dataset, you want to evaluate if the product is defective or not. Bosh knows about defective lots so it is highly improbable they would make a mistake like leaking a defective product on the ID feature. (factory are highly correlated within its process so it is obvious feature engineering will bring HUGE improvements) EDIT:(wrong assumption... yet they shared the leak!)\r\n\r\nI have been a kaggle member recently and all i can say is that this is the perfect place to learn. People share knowledge without asking anything in return and you can learn way faster that you could on your own, or in the university. (i speak for myself, and i thank for such great people on the community)\r\n\r\nThe post you are referring is sad. All the rating... You win competitions if you go in with the right mindset, and because this area is very challenging you have to be able to learn and innovate your approach. You will never win a competition by slightly changing some parameters on a public kernel. So study, ask questions and people will help you.\r\n\r\nFor instance, in the red hat competition some people even went public with the leak so everybody can learn what it looks like and how to search for it, in case you ever find one which wreaks you model in real life.", "votes": 3 }, { "id": 138407, "postDate": "2016-10-08T16:57:54.660Z", "content": "I'll bite. The "Bosch" seems to be "discriminate" between "failure" and "non-failure". However, if at time T we have a failure, and later T + DT, we still have failure and between that we have the "test" sample, it becomes interesting to discern between "predictive" and "discriminative" and "interpolation". I.e. the future tells the past?
", "rawMarkdown": "I'll bite. The \"Bosch\" seems to be \"discriminate\" between \"failure\" and \"non-failure\". However, if at time T we have a failure, and later T + DT, we still have failure and between that we have the \"test\" sample, it becomes interesting to discern between \"predictive\" and \"discriminative\" and \"interpolation\". I.e. the future tells the past?", "votes": 1 }, { "id": 144513, "postDate": "2016-11-14T13:28:21.737Z", "content": "GWilli: can you state your argument again? Specifically regarding overfitting in kaggle competition (validation on unseen data). thanks.
\n\nI personally think the first sentence of Laurae`s response is what u need to counter first.
\n\nPS: Is this a troll attempt? o_O
", "rawMarkdown": "GWilli: can you state your argument again? Specifically regarding overfitting in kaggle competition (validation on unseen data). thanks.\r\n\r\nI personally think the first sentence of Laurae`s response is what u need to counter first. \r\n\r\nPS: Is this a troll attempt? o_O", "votes": 2 }, { "id": 144396, "postDate": "2016-11-14T04:09:38.407Z", "content": "[quote=GWillie;144389]
\n\nOh, and I especially like the part about my hypotheses are thus proven wrong, even though you don't even understand what my hypotheses are. Please stop making a fool of yourself on here.
\n\n[/quote]
\n\nThese are really unkind words. People will be more receptive to your arguments if it were more polite and it helps looking more professional.
\n\nNow, back to the topic. If Kaggle competitions are not truely predictive , then could you give any examples of predictive modelling.
\n\nOn the flip side, i believe kaggle competitions do overfit the data to some extent. We strive to get the last 0.0000000001 score, where in real life the sampling error due to time or spatial constrains are larger than the final decimal point of the score. (my first lesson in physics on uncertainty of measurement) . However, an overfit model still has predictive power, just less than what it claims.
\n\nLastly, regarding the elections, the prediction was within error, where the error is in the fine print which nobody reads. Also there was a large sampling bias/error for some subgroups of people. The statistically insignificant sample size of 1 you gave is both invalid and easily explained.
", "rawMarkdown": "[quote=GWillie;144389]\r\n\r\nOh, and I especially like the part about my hypotheses are thus proven wrong, even though you don't even understand what my hypotheses are. Please stop making a fool of yourself on here.\r\n\r\n[/quote]\r\n\r\nThese are really unkind words. People will be more receptive to your arguments if it were more polite and it helps looking more professional. \r\n\r\nNow, back to the topic. If Kaggle competitions are not truely predictive , then could you give any examples of predictive modelling. \r\n\r\nOn the flip side, i believe kaggle competitions do overfit the data to some extent. We strive to get the last 0.0000000001 score, where in real life the sampling error due to time or spatial constrains are larger than the final decimal point of the score. (my first lesson in physics on uncertainty of measurement) . However, an overfit model still has predictive power, just less than what it claims. \r\n\r\nLastly, regarding the elections, the prediction was within error, where the error is in the fine print which nobody reads. Also there was a large sampling bias/error for some subgroups of people. The statistically insignificant sample size of 1 you gave is both invalid and easily explained. \r\n\r\n", "votes": 2 }, { "id": 144393, "postDate": "2016-11-14T03:56:29.473Z", "content": "I think your example of U.S. elections doesnt reflect how was the data gathered. The underlying problems are that the data itself did not satisfy iid conditions. For example, polling was not permitted on random telephone numbers, so in most cases the poll samples were made people attending talks, conventions or other political events. Moreover, some polls used a repeated sample. meaning the same people were asked over and over again if they changed their mind through time. There is also the question of representational weighting to try to account for different backgrounds and ethnicities which allowed for the data to be spoiled by the choices of the data collector. Hence what im trying to say, its that the US election polls are not a good example of how good/bad predictive modeling is since the data was poor in the first place.
", "rawMarkdown": "I think your example of U.S. elections doesn't reflect how was the data gathered. The underlying problems are that the data itself did not satisfy iid conditions. For example, polling was not permitted on random telephone numbers, so in most cases the poll samples were made people attending talks, conventions or other political events. Moreover, some polls used a repeated sample. meaning the same people were asked over and over again if they changed their mind through time. There is also the question of representational weighting to try to account for different backgrounds and ethnicities which allowed for the data to be spoiled by the choices of the data collector. Hence what im trying to say, its that the US election polls are not a good example of how good/bad predictive modeling is since the data was poor in the first place.", "votes": 2 }, { "id": 144406, "postDate": "2016-11-14T04:57:57.800Z", "content": "Sorry, but I don't see how someone comes and says several times in one post that he proved me wrong and yet doesn't even understand what it is he's proving wrong. In my opinion, that is making a fool of oneself and thus should be noted as such. I don't know how I can be "polite" when getting so undeservedly attacked like that.
\n\nAs far as the U.S. elections, data gathering techniques get more and more sophisticated with each election. So I don't think that is the problem. Also, modeling techniques get more and more sophisticated with each election. So my point is, with better and better data gathering techniques and better and better statistical/modeling techniques, we are seeing worse and worse predictions. Seems to me this is evidence that we are getting better and better at fitting models to observed data, but those models are getting less and less effective at predicting unobserved outcomes.
\n\nAnd no, election predictions were not "within error." Going into the election night, predictions showed Clinton's chances of winning at between 85 and 92%. Within hours that fell to 40%. This is not within some "error."
\n\nAs far as overfitting, it's not about that last 0.0000000001 score, it's the relatively large movements such as doubling scores within 20-30 submissions. And while some overfit models can have predictive power (even a stopped clock is right twice a day), my point is that this is not, in general, a good robust approach to predictive modeling. And the fact that many models and competitions which strive to find good predictive models end up going unused is evidence of this.
", "rawMarkdown": "Sorry, but I don't see how someone comes and says several times in one post that he proved me wrong and yet doesn't even understand what it is he's proving wrong. In my opinion, that is making a fool of oneself and thus should be noted as such. I don't know how I can be \"polite\" when getting so undeservedly attacked like that.\r\n\r\nAs far as the U.S. elections, data gathering techniques get more and more sophisticated with each election. So I don't think that is the problem. Also, modeling techniques get more and more sophisticated with each election. So my point is, with better and better data gathering techniques and better and better statistical/modeling techniques, we are seeing worse and worse predictions. Seems to me this is evidence that we are getting better and better at fitting models to observed data, but those models are getting less and less effective at predicting unobserved outcomes.\r\n\r\nAnd no, election predictions were not \"within error.\" Going into the election night, predictions showed Clinton's chances of winning at between 85 and 92%. Within hours that fell to 40%. This is not within some \"error.\"\r\n\r\nAs far as overfitting, it's not about that last 0.0000000001 score, it's the relatively large movements such as doubling scores within 20-30 submissions. And while some overfit models can have predictive power (even a stopped clock is right twice a day), my point is that this is not, in general, a good robust approach to predictive modeling. And the fact that many models and competitions which strive to find good predictive models end up going unused is evidence of this.", "votes": -10 }, { "id": 144371, "postDate": "2016-11-14T02:16:55.287Z", "content": "I don't follow your point. Yes, obviously you don't construct an algorithm that does not include the variable over which you would like to vary. This in no way refutes the point I made about Kaggle being an overfitting forum and the fact that companies are paying (wasting) a lot of money thinking they are getting predictive models when in fact they are getting models that are overfit to existing data. We see more and more evidence of this every day.
", "rawMarkdown": "I don't follow your point. Yes, obviously you don't construct an algorithm that does not include the variable over which you would like to vary. This in no way refutes the point I made about Kaggle being an overfitting forum and the fact that companies are paying (wasting) a lot of money thinking they are getting predictive models when in fact they are getting models that are overfit to existing data. We see more and more evidence of this every day.", "votes": -10 }, { "id": 144580, "postDate": "2016-11-14T17:54:00.517Z", "content": "Please, feel free to vote me down, I'm used to it by now. The fact that I get voted down with no real attempt at logically discussing what I'm saying is evidence to me that I'm on the right track.
", "rawMarkdown": "Please, feel free to vote me down, I'm used to it by now. The fact that I get voted down with no real attempt at logically discussing what I'm saying is evidence to me that I'm on the right track.", "votes": -6 }, { "id": 144577, "postDate": "2016-11-14T17:39:31.103Z", "content": "I would not attempt to display such a lack of knowledge in logic in a forum where I did not speak the language. That was quite the display of hubris on her part. I was simply pointing that out.
", "rawMarkdown": "I would not attempt to display such a lack of knowledge in logic in a forum where I did not speak the language. That was quite the display of hubris on her part. I was simply pointing that out.", "votes": -6 }, { "id": 144576, "postDate": "2016-11-14T17:37:34.830Z", "content": "Continuing to hail the advances of machine learning is not adequately addressing the needs with regard to predictive modeling.
", "rawMarkdown": "Continuing to hail the advances of machine learning is not adequately addressing the needs with regard to predictive modeling.", "votes": -6 }, { "id": 144574, "postDate": "2016-11-14T17:36:53.360Z", "content": "And it's more evidence that, as I said, the adoption of machine learning continues to advance while true predictive modeling continues to stagnate, or even decline as we saw in the recent Presidential election.
", "rawMarkdown": "And it's more evidence that, as I said, the adoption of machine learning continues to advance while true predictive modeling continues to stagnate, or even decline as we saw in the recent Presidential election.", "votes": -2 }, { "id": 144570, "postDate": "2016-11-14T17:15:55.767Z", "content": "As far as addressing the first sentence of Laurae's response, I'm not sure which of his/her two responses you are referring to. In the first response, the first sentence basically makes the argument that, if we weren't all doing predictive modeling here, there wouldn't be so many people on the leaderboard with such high scores. But that couldn't be further from the truth, and is exactly the point I'm trying to make. You can have many people using the same canned machine learning fitting algorithms, and thus getting similar scores, and you can watch them all increase their scores and climb up the leader board as they find ways to exploit and overfit the data during the course of their hundreds and hundreds of submissions. This in no way proves that they are all creating models that will be effective in making predictions. In fact, the tendency to overfit - that this whole endeavor rewards - will do exactly the opposite. As far as the public vs. private leaderboard issue, it seems that for most of the Kaggle competitions I've seen, high scores on the private leaderboard are often just an indication that the hold-out portion of the test sets were very similar to the non hold-out portions, thus rewarding overfitting to the training set and non hold-out portion of the test set.
\n\nThe first sentence of his/her second response is an assumption on his/her part that I don't know what overfitting is. I don't know that baseless assumptions like that are even worthy of a response. He/she does go on to lecture me about how predictive modeling is a process and shows me a corporate slide outlining their process. While predictive modeling is a process, it is, more importantly, a design. You can apply the same process over and over to poorly designed projects and achieve the same inadequate results over and over. And in fact, this approach of setting up a process and then churning things through that mill without adequate formulation and design is yet another reason why the field has stagnated. Regarding proper design, the way that these Kaggle competitions are set up often limits ones ability to adequately formulate and design a model that will have good predictive capability. The need to keep data anonymous opens the door for people to overfit to specific instances of data and severely limits ones ability to create the necessary mappings between continuous input variables and either categorical or continuous response variables.
", "rawMarkdown": "As far as addressing the first sentence of Laurae's response, I'm not sure which of his/her two responses you are referring to. In the first response, the first sentence basically makes the argument that, if we weren't all doing predictive modeling here, there wouldn't be so many people on the leaderboard with such high scores. But that couldn't be further from the truth, and is exactly the point I'm trying to make. You can have many people using the same canned machine learning fitting algorithms, and thus getting similar scores, and you can watch them all increase their scores and climb up the leader board as they find ways to exploit and overfit the data during the course of their hundreds and hundreds of submissions. This in no way proves that they are all creating models that will be effective in making predictions. In fact, the tendency to overfit - that this whole endeavor rewards - will do exactly the opposite. As far as the public vs. private leaderboard issue, it seems that for most of the Kaggle competitions I've seen, high scores on the private leaderboard are often just an indication that the hold-out portion of the test sets were very similar to the non hold-out portions, thus rewarding overfitting to the training set and non hold-out portion of the test set.\r\n\r\nThe first sentence of his/her second response is an assumption on his/her part that I don't know what overfitting is. I don't know that baseless assumptions like that are even worthy of a response. He/she does go on to lecture me about how predictive modeling is a process and shows me a corporate slide outlining their process. While predictive modeling is a process, it is, more importantly, a design. You can apply the same process over and over to poorly designed projects and achieve the same inadequate results over and over. And in fact, this approach of setting up a process and then churning things through that mill without adequate formulation and design is yet another reason why the field has stagnated. Regarding proper design, the way that these Kaggle competitions are set up often limits ones ability to adequately formulate and design a model that will have good predictive capability. The need to keep data anonymous opens the door for people to overfit to specific instances of data and severely limits ones ability to create the necessary mappings between continuous input variables and either categorical or continuous response variables.\r\n\r\n", "votes": -8 }, { "id": 144561, "postDate": "2016-11-14T16:53:35.650Z", "content": "Here is a quote from a big data/data driven modeller, "In the day and age of data driven insights political campaigners that added big data techniques to their tool belt will have undoubtedly a clear advantage over the competition because they will apply machine learning techniques, real time insight and predictive modeling to influence voters even better through micro targeting, statistical analysis and machine algorithms." This is exactly my point. Because they are applying the requisite statistical and machine learning algorithms, they are, by their own assumption, doing predictive modeling. This is also the presiding sentiment here on Kaggle. But we saw how those assumptions failed during the presidential election, just as we see one excuse after another here on Kaggle why the latest competition turned out to be not really predictive modeling. At some point, I would think that we need to ask ourselves what is wrong with the current state of assumptions and work towards improving the design of predictive modeling projects. Yet people are more resigned to attack any messenger (like me) that comes on and challenges their notion that they are, by assumption, doing true predictive modeling.
", "rawMarkdown": "Here is a quote from a big data/data driven modeller, \"In the day and age of data driven insights political campaigners that added big data techniques to their tool belt will have undoubtedly a clear advantage over the competition because they will apply machine learning techniques, real time insight and predictive modeling to influence voters even better through micro targeting, statistical analysis and machine algorithms.\" This is exactly my point. Because they are applying the requisite statistical and machine learning algorithms, they are, by their own assumption, doing predictive modeling. This is also the presiding sentiment here on Kaggle. But we saw how those assumptions failed during the presidential election, just as we see one excuse after another here on Kaggle why the latest competition turned out to be not really predictive modeling. At some point, I would think that we need to ask ourselves what is wrong with the current state of assumptions and work towards improving the design of predictive modeling projects. Yet people are more resigned to attack any messenger (like me) that comes on and challenges their notion that they are, by assumption, doing true predictive modeling.", "votes": -6 }, { "id": 144389, "postDate": "2016-11-14T03:39:32.220Z", "content": "Oh, and I especially like the part about my hypotheses are thus proven wrong, even though you don't even understand what my hypotheses are. Please stop making a fool of yourself on here.
", "rawMarkdown": "Oh, and I especially like the part about my hypotheses are thus proven wrong, even though you don't even understand what my hypotheses are. Please stop making a fool of yourself on here.", "votes": -11 }, { "id": 144387, "postDate": "2016-11-14T03:38:33.237Z", "content": "Wow, that was a whole lot of poor English and poor logic. First, you seem to make the assumption that Kaggle is true predictive modeling, in order to disprove my claim that that Kaggle is not true predictive modeling. Seriously? A freshman course in logic would help you a great deal. I know exactly that predictive modeling is a process, and that process is not allowed to be implemented in its true form by competitions like those presented on Kaggle. Your slide that includes all the corporate jargon does not come close to explaining why companies are wasting millions of dollars on the holy grail predictive models and coming up woefully short. My example of the catastrophic failure of the predictions in the U.S. elections are evidence supporting this claim. And as far as my "true/false predictive modeling potential definition" ... please show me where I attempted to define such a thing. You assert that I said things which I did not, and then you attempt to go about trying to prove these made-up arguments to be false. Again, a freshman course in logic would help you a great deal. Until then, you have failed to "invalidate" anything that I have said. And this type of refusal to acknowledge reality is precisely why the current state of predictive modeling is in such disarray.
", "rawMarkdown": "Wow, that was a whole lot of poor English and poor logic. First, you seem to make the assumption that Kaggle is true predictive modeling, in order to disprove my claim that that Kaggle is not true predictive modeling. Seriously? A freshman course in logic would help you a great deal. I know exactly that predictive modeling is a process, and that process is not allowed to be implemented in its true form by competitions like those presented on Kaggle. Your slide that includes all the corporate jargon does not come close to explaining why companies are wasting millions of dollars on the holy grail predictive models and coming up woefully short. My example of the catastrophic failure of the predictions in the U.S. elections are evidence supporting this claim. And as far as my \"true/false predictive modeling potential definition\" ... please show me where I attempted to define such a thing. You assert that I said things which I did not, and then you attempt to go about trying to prove these made-up arguments to be false. Again, a freshman course in logic would help you a great deal. Until then, you have failed to \"invalidate\" anything that I have said. And this type of refusal to acknowledge reality is precisely why the current state of predictive modeling is in such disarray.", "votes": -18 }, { "id": 144308, "postDate": "2016-11-13T17:52:35.173Z", "content": "I'm sure everyone was watching the U.S. election and saw how wrong the experts were on their "predictions." Welcome to the current state of predictive modeling. And yes, I will stand by my statements. Overfitting competitions like these are not doing anything to advance the state of true predictive modeling. The mindset demonstrated by Abriosi above is exactly what I'm talking about. The focus here is to win competitions. And because these competitions are billed as "predictive modeling," then people assume that if they do well in the competitions, they are doing well at "predictive modeling." So yes, you must do some work and change perhaps many parameters to win these competitions. That will give you a higher score and better fits on the available data (thus the overfitting). But what we're seeing over and over and over, both here and elsewhere, is that this focus on overfitting is not getting where we want to go in the field of true predictive modeling. We saw this on a very high level with the U.S. election predictions.
", "rawMarkdown": "I'm sure everyone was watching the U.S. election and saw how wrong the experts were on their \"predictions.\" Welcome to the current state of predictive modeling. And yes, I will stand by my statements. Overfitting competitions like these are not doing anything to advance the state of true predictive modeling. The mindset demonstrated by Abriosi above is exactly what I'm talking about. The focus here is to win competitions. And because these competitions are billed as \"predictive modeling,\" then people assume that if they do well in the competitions, they are doing well at \"predictive modeling.\" So yes, you must do some work and change perhaps many parameters to win these competitions. That will give you a higher score and better fits on the available data (thus the overfitting). But what we're seeing over and over and over, both here and elsewhere, is that this focus on overfitting is not getting where we want to go in the field of true predictive modeling. We saw this on a very high level with the U.S. election predictions.", "votes": -18 }, { "id": 144572, "postDate": "2016-11-14T17:18:48.983Z", "content": "Owl, thank you for providing more support for my claim. Kaggle is a great platform for machine learning. What we have to be careful about though is making the leap from there to the assumption that it is thereby necessarily a great platform for predictive modeling. This is exactly the point I am trying to make. As machine learning algorithms get more and more advanced and commonplace, we are not seeing the transition we thought to predictive modeling becoming more and more advanced and accepted. At some point, we need to really examine why that is.
", "rawMarkdown": "Owl, thank you for providing more support for my claim. Kaggle is a great platform for machine learning. What we have to be careful about though is making the leap from there to the assumption that it is thereby necessarily a great platform for predictive modeling. This is exactly the point I am trying to make. As machine learning algorithms get more and more advanced and commonplace, we are not seeing the transition we thought to predictive modeling becoming more and more advanced and accepted. At some point, we need to really examine why that is." }, { "id": 144416, "postDate": "2016-11-14T06:55:43.690Z", "content": "I disagree that the data gathering got better, if you look at recent laws, they made it harder for pollers to interview people and some states made it just illegal to call at home and various others. Besides, some polls did get it right and they are the ones that used better data collection techniques
", "rawMarkdown": "I disagree that the data gathering got better, if you look at recent laws, they made it harder for pollers to interview people and some states made it just illegal to call at home and various others. Besides, some polls did get it right and they are the ones that used better data collection techniques" }, { "id": 144410, "postDate": "2016-11-14T05:32:33.247Z", "content": "[quote=GWillie;144406]
\n\nAs far as the U.S. elections, data gathering techniques get more and more sophisticated with each election. So I don't think that is the problem. Also, modeling techniques get more and more sophisticated with each election. So my point is, with better and better data gathering techniques and better and better statistical/modeling techniques, we are seeing worse and worse predictions. Seems to me this is evidence that we are getting better and better at fitting models to observed data, but those models are getting less and less effective at predicting unobserved outcomes.
\n\n[/quote]
\n\nThis usually happens when you overfit the data, which is not adequately sampling the true population. Reports in retrospect say that people have neglected the silent voters, for various reasons.
\n\n[quote=GWillie;144406]
\n\nAnd no, election predictions were not "within error." Going into the election night, predictions showed Clinton's chances of winning at between 85 and 92%. Within hours that fell to 40%. This is not within some "error."
\n\n[/quote]
\n\nI was referring to the swing states, where Clinton was predicted to marginally win. The error here is larger than Clinton's marginal win, so both outcomes are probable. This was something I somewhat recalled ...
\n\nLastly, your toned down post makes forums a nicer place to read :)
", "rawMarkdown": "[quote=GWillie;144406]\r\n\r\nAs far as the U.S. elections, data gathering techniques get more and more sophisticated with each election. So I don't think that is the problem. Also, modeling techniques get more and more sophisticated with each election. So my point is, with better and better data gathering techniques and better and better statistical/modeling techniques, we are seeing worse and worse predictions. **Seems to me this is evidence that we are getting better and better at fitting models to observed data, but those models are getting less and less effective at predicting unobserved outcomes.**\r\n\r\n[/quote]\r\n\r\nThis usually happens when you overfit the data, which is not adequately sampling the true population. Reports in retrospect say that people have neglected the silent voters, for various reasons. \r\n\r\n\r\n[quote=GWillie;144406]\r\n\r\nAnd no, election predictions were not \"within error.\" Going into the election night, predictions showed Clinton's chances of winning at between 85 and 92%. Within hours that fell to 40%. This is not within some \"error.\"\r\n\r\n[/quote]\r\n\r\nI was referring to the swing states, where Clinton was predicted to marginally win. The error here is larger than Clinton's marginal win, so both outcomes are probable. This was something I somewhat recalled ...\r\n\r\nLastly, your toned down post makes forums a nicer place to read :)" }, { "id": 144573, "postDate": "2016-11-14T17:29:38.090Z", "rawMarkdown": "", "isDeleted": true }, { "id": 144567, "postDate": "2016-11-14T17:06:53.750Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 144382, "author_name": "Laurae", "author_url": "", "post_date": "2016-11-14T03:14:59.650000", "content": "[quote=GWillie;144371]
\n\nThis in no way refutes the point I made about Kaggle being an overfitting forum and the fact that companies are paying (wasting) a lot of money thinking they are getting predictive models when in fact they are getting models that are overfit to existing data.\n[/quote]
\n\nDefinition of overfitting, the stroke area:
\n\n
I think you should get back to statistical modeling theory before writing back (like: what is overfitting and what is underfitting). Not only you seem to not know predictive modeling is a process and there's no true/false predictive modeling (provide us your research paper about it if you are willing to challenge it), that process does not rely on assumptions of truthness of information to be a process (VERY IMPORTANT), and if you know your theory about predictive modeling, the process is an endless loop.
\n\nHere is an example from one of my slides with IBM which proves your "false predictive modeling" does not exist (only "predictive modeling" is existing), as errors found on the way are corrected, and the loop goes on indefinitely until a satisfactory solution by all parties is found:
\n\n
More or less there's a bias you have and you can't even explain it using theory, therefore no one can understand what you are trying to explain us. Our models are not stopping in the hands on Bosch, we are only a helping part of the process for them and they will have to find a way to exploit them. The story stops here for us at Kaggle, and we may never know what will happen to the models: they could be thrown out or even improved, but this is not an issue on our hands.
\n\nThat's how Kaggle works: you are just an element of crowdsourcing to help predictive modeling to be carried on for the company who paid Kaggle to get help. The wealth of information provided from crowdsourcing is something you cannot find using your own data scientists. Anything found, even a minor detail, can be useful from a design perspective.
\n\nI think you don't get that companies are not paying only for the models. Insights are actually more important as the valuation (non-monetary) kicks in - it is not only about "numbers". You should read what is asked in the winners' documentation. I also think enough time was wasted in a topic ("true predictive modeling, definition still not given?") that does not make any sense, as the theory you learnt does not seem assimilated correctly.
\n\nNow that your "true/false predictive modeling" potential definition was invalidated, are you going to say the theory (which, for the predictive modeling process, is based on pragmatic evidence throughout the years that Data Mining is existing) is wrong? If yes, do you have any empirical and significant proof? (hint: there's none - there's a reason it's called a "process" which loops over itself like a snake, thus becomes invulnerable to errors as they are theoretically auto-corrected)
\n\nExtra: If you invaildate "they are theoretically auto-corrected" using practical sources (because why not, humans are prone to errors in a process), you end up with a single theoretical "false modeling" which is obviously generalizing to all processes as a human must be introduced somewhere. Therefore, there's only one process existing, not the existance of two separate processes (True/False). This is what we call a "Railing" in any existing process in this world, and that's what is supposed to prevent such mistake (if they happen to be created by humans). If your overfitting conviction was true, the Private Leaderboard is supposed to be entirely scrambled, in absolute values and not in rank. Obviously, this is not the case, overfitting was nearly non-existant (if not at all for many of us).
\n\nHence, both in theory and in practice your True/False predictive modeling hypotheses (we don't even know what they are, but we don't need to know that) are proven wrong.
\n\n(off-topic: self-note for re-usage of this wall of text for next data science course to teach, copy&paste text)
\n\n(edit: removed IBM from print screen)
", "votes": 9, "replies": [] }, { "id": 144336, "author_name": "Laurae", "author_url": "", "post_date": "2016-11-13T20:00:47.820000", "content": "[quote=GWillie;144308]
\n\nThat will give you a higher score and better fits on the available data (thus the overfitting).
\n\n[/quote]
\n\nLeaderboard Public vs Private stability check: check
\n\n
If we all were truly overfitting no one of us should appear there, cf Santander. The difference is negligible in absolute terms, post-bias correction. Even in Santander, it was not even severe (but in terms of ranks it was severe enough as everyone was very close vs 0.50 random).
\n\nIn any case, following your statement, if a satellite makes a perfect circle (r = fixed distance) around the Earth, it should mean your algorithm to move the rocket periodically is overfitting the circle and cannot generalize to new circles when you will provide another distance. Isn't that too much naive? Obviously, your would try to make your algorithm generalize to any distance you input, not to do perfect on r=10000 only and wreck havoc on r=10000-0.0000000000000000001 which does not make sense if we follow what you said.
\n\nBut yes, in this world humans are making the algorithms for the satellites so they can orbit around the earth without wrecking havoc if you change something by 1 inch. If your statement was true, expect no satellites existing as they would all be destroyed or flying somewhere else without any possible control.
", "votes": 10, "replies": [] }, { "id": 144581, "author_name": "anokas", "author_url": "", "post_date": "2016-11-14T17:54:44.810000", "content": "@GWillie, I'm completely puzzled by your arguments. You seem to claim that some US election predictions being incorrect therefore shows that machine learning is bad? I may not have an education in statistics, but I think that your sample size of 1 may not be representative of the entire population.
\n\nI also don't understand why you think we're overfitting. The private leaderboard had 700,000 unseen samples that we had to predict on, and the leaderboard shows that people did not overfit very much to the 300k samples that they could see the results on versus the private set.
\n\nSince the goal of a machine learning model is to fit to unseen data, and here we have proof that our models can fit to (a considerable amount of) unseen data, logically we can say that our machine learning is not overfitting. How could we have overfitted to the private test set?
\n\nAlso, an example of what you consider to be predictive modelling would be great, so we can actually understand your view. Right now it might as well be phoning up your local fortune teller and asking them who's going to win the election.
\n\nPS: You're being downvoted because you're not actually making any logical discussion, rather resorting to attacks on well-respected forum members and saying "your argument is wrong" without providing a counterview. If you want a logical discussion, then you can start by presenting your view (i.e. what is predictive modelling?)
", "votes": 8, "replies": [] }, { "id": 144578, "author_name": "Tilii", "author_url": "", "post_date": "2016-11-14T17:44:18.040000", "content": "I made a decision early on never to vote down Kaggle posts. Boy, have I been tempted ...
", "votes": 5, "replies": [] }, { "id": 144575, "author_name": "CPMP", "author_url": "", "post_date": "2016-11-14T17:37:18.313000", "content": "[quote=GWillie;144387]
\n\nWow, that was a whole lot of poor English and poor logic.
\n[/quote]
Attacking a non native speaker on her use of English is not really helpful. Let's switch to French to see how good you are at discussing in a foreign language...
\n\nMore generally, personal attacks aren't great in forums. Stick to ideas and facts, and please do not attack the person.
", "votes": 5, "replies": [] }, { "id": 144614, "author_name": "dmi3kno", "author_url": "", "post_date": "2016-11-14T21:06:10.130000", "content": "Ok, @GWillie. I am willing to assume you are right.
\n\n1) What would you do to "fix" overfitting on Kaggle? Disable Public Leaderboard? Not provide test data at all (ask people to submit their fully documented models)?
\n\n2) How would you ensure quality and consistency of training data gathering and preparation by hosting companies to increase usefulness of the models and their ability to generalize to unseen (possibly even not yet collected) data?
\n\n3) How would you go about assembling and incentivizing the data science community around your perfect data science process (is KDN better in your view, by the way)?
\n\nI will dare to say that leaderboard shakeups are part of the thrill and advancing by 0.000001 on the leaderboard is what keeps people glued to the monitors for weeks towards the end of competition. Teams are formed, people learn (statistics and computer science), virtual coins are distributed, people have fun! You don't like this game? You don't have to play it! And, assuming your name is not Nate, go build a better predictive model for US elections! Beat @FiveThirtyEight and be right every time!! We all here will thank you and bow our heads in respect.
", "votes": 6, "replies": [] }, { "id": 138405, "author_name": "Gonçalo Abreu", "author_url": "", "post_date": "2016-10-08T16:42:53.133000", "content": "It is. This is a factory dataset, you want to evaluate if the product is defective or not. Bosh knows about defective lots so it is highly improbable they would make a mistake like leaking a defective product on the ID feature. (factory are highly correlated within its process so it is obvious feature engineering will bring HUGE improvements) EDIT:(wrong assumption... yet they shared the leak!)
\n\nI have been a kaggle member recently and all i can say is that this is the perfect place to learn. People share knowledge without asking anything in return and you can learn way faster that you could on your own, or in the university. (i speak for myself, and i thank for such great people on the community)
\n\nThe post you are referring is sad. All the rating... You win competitions if you go in with the right mindset, and because this area is very challenging you have to be able to learn and innovate your approach. You will never win a competition by slightly changing some parameters on a public kernel. So study, ask questions and people will help you.
\n\nFor instance, in the red hat competition some people even went public with the leak so everybody can learn what it looks like and how to search for it, in case you ever find one which wreaks you model in real life.
", "votes": 3, "replies": [] }, { "id": 138407, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2016-10-08T16:57:54.660000", "content": "I'll bite. The "Bosch" seems to be "discriminate" between "failure" and "non-failure". However, if at time T we have a failure, and later T + DT, we still have failure and between that we have the "test" sample, it becomes interesting to discern between "predictive" and "discriminative" and "interpolation". I.e. the future tells the past?
", "votes": 1, "replies": [] }, { "id": 144513, "author_name": "stgrmks", "author_url": "", "post_date": "2016-11-14T13:28:21.737000", "content": "GWilli: can you state your argument again? Specifically regarding overfitting in kaggle competition (validation on unseen data). thanks.
\n\nI personally think the first sentence of Laurae`s response is what u need to counter first.
\n\nPS: Is this a troll attempt? o_O
", "votes": 2, "replies": [] }, { "id": 144396, "author_name": "leexa", "author_url": "", "post_date": "2016-11-14T04:09:38.407000", "content": "[quote=GWillie;144389]
\n\nOh, and I especially like the part about my hypotheses are thus proven wrong, even though you don't even understand what my hypotheses are. Please stop making a fool of yourself on here.
\n\n[/quote]
\n\nThese are really unkind words. People will be more receptive to your arguments if it were more polite and it helps looking more professional.
\n\nNow, back to the topic. If Kaggle competitions are not truely predictive , then could you give any examples of predictive modelling.
\n\nOn the flip side, i believe kaggle competitions do overfit the data to some extent. We strive to get the last 0.0000000001 score, where in real life the sampling error due to time or spatial constrains are larger than the final decimal point of the score. (my first lesson in physics on uncertainty of measurement) . However, an overfit model still has predictive power, just less than what it claims.
\n\nLastly, regarding the elections, the prediction was within error, where the error is in the fine print which nobody reads. Also there was a large sampling bias/error for some subgroups of people. The statistically insignificant sample size of 1 you gave is both invalid and easily explained.
", "votes": 2, "replies": [] }, { "id": 144393, "author_name": "AndreaFGenovese", "author_url": "", "post_date": "2016-11-14T03:56:29.473000", "content": "I think your example of U.S. elections doesnt reflect how was the data gathered. The underlying problems are that the data itself did not satisfy iid conditions. For example, polling was not permitted on random telephone numbers, so in most cases the poll samples were made people attending talks, conventions or other political events. Moreover, some polls used a repeated sample. meaning the same people were asked over and over again if they changed their mind through time. There is also the question of representational weighting to try to account for different backgrounds and ethnicities which allowed for the data to be spoiled by the choices of the data collector. Hence what im trying to say, its that the US election polls are not a good example of how good/bad predictive modeling is since the data was poor in the first place.
", "votes": 2, "replies": [] }, { "id": 144406, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T04:57:57.800000", "content": "Sorry, but I don't see how someone comes and says several times in one post that he proved me wrong and yet doesn't even understand what it is he's proving wrong. In my opinion, that is making a fool of oneself and thus should be noted as such. I don't know how I can be "polite" when getting so undeservedly attacked like that.
\n\nAs far as the U.S. elections, data gathering techniques get more and more sophisticated with each election. So I don't think that is the problem. Also, modeling techniques get more and more sophisticated with each election. So my point is, with better and better data gathering techniques and better and better statistical/modeling techniques, we are seeing worse and worse predictions. Seems to me this is evidence that we are getting better and better at fitting models to observed data, but those models are getting less and less effective at predicting unobserved outcomes.
\n\nAnd no, election predictions were not "within error." Going into the election night, predictions showed Clinton's chances of winning at between 85 and 92%. Within hours that fell to 40%. This is not within some "error."
\n\nAs far as overfitting, it's not about that last 0.0000000001 score, it's the relatively large movements such as doubling scores within 20-30 submissions. And while some overfit models can have predictive power (even a stopped clock is right twice a day), my point is that this is not, in general, a good robust approach to predictive modeling. And the fact that many models and competitions which strive to find good predictive models end up going unused is evidence of this.
", "votes": -10, "replies": [] }, { "id": 144371, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T02:16:55.287000", "content": "I don't follow your point. Yes, obviously you don't construct an algorithm that does not include the variable over which you would like to vary. This in no way refutes the point I made about Kaggle being an overfitting forum and the fact that companies are paying (wasting) a lot of money thinking they are getting predictive models when in fact they are getting models that are overfit to existing data. We see more and more evidence of this every day.
", "votes": -10, "replies": [] }, { "id": 144580, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T17:54:00.517000", "content": "Please, feel free to vote me down, I'm used to it by now. The fact that I get voted down with no real attempt at logically discussing what I'm saying is evidence to me that I'm on the right track.
", "votes": -6, "replies": [] }, { "id": 144577, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T17:39:31.103000", "content": "I would not attempt to display such a lack of knowledge in logic in a forum where I did not speak the language. That was quite the display of hubris on her part. I was simply pointing that out.
", "votes": -6, "replies": [] }, { "id": 144576, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T17:37:34.830000", "content": "Continuing to hail the advances of machine learning is not adequately addressing the needs with regard to predictive modeling.
", "votes": -6, "replies": [] }, { "id": 144574, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T17:36:53.360000", "content": "And it's more evidence that, as I said, the adoption of machine learning continues to advance while true predictive modeling continues to stagnate, or even decline as we saw in the recent Presidential election.
", "votes": -2, "replies": [] }, { "id": 144570, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T17:15:55.767000", "content": "As far as addressing the first sentence of Laurae's response, I'm not sure which of his/her two responses you are referring to. In the first response, the first sentence basically makes the argument that, if we weren't all doing predictive modeling here, there wouldn't be so many people on the leaderboard with such high scores. But that couldn't be further from the truth, and is exactly the point I'm trying to make. You can have many people using the same canned machine learning fitting algorithms, and thus getting similar scores, and you can watch them all increase their scores and climb up the leader board as they find ways to exploit and overfit the data during the course of their hundreds and hundreds of submissions. This in no way proves that they are all creating models that will be effective in making predictions. In fact, the tendency to overfit - that this whole endeavor rewards - will do exactly the opposite. As far as the public vs. private leaderboard issue, it seems that for most of the Kaggle competitions I've seen, high scores on the private leaderboard are often just an indication that the hold-out portion of the test sets were very similar to the non hold-out portions, thus rewarding overfitting to the training set and non hold-out portion of the test set.
\n\nThe first sentence of his/her second response is an assumption on his/her part that I don't know what overfitting is. I don't know that baseless assumptions like that are even worthy of a response. He/she does go on to lecture me about how predictive modeling is a process and shows me a corporate slide outlining their process. While predictive modeling is a process, it is, more importantly, a design. You can apply the same process over and over to poorly designed projects and achieve the same inadequate results over and over. And in fact, this approach of setting up a process and then churning things through that mill without adequate formulation and design is yet another reason why the field has stagnated. Regarding proper design, the way that these Kaggle competitions are set up often limits ones ability to adequately formulate and design a model that will have good predictive capability. The need to keep data anonymous opens the door for people to overfit to specific instances of data and severely limits ones ability to create the necessary mappings between continuous input variables and either categorical or continuous response variables.
", "votes": -8, "replies": [] }, { "id": 144561, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T16:53:35.650000", "content": "Here is a quote from a big data/data driven modeller, "In the day and age of data driven insights political campaigners that added big data techniques to their tool belt will have undoubtedly a clear advantage over the competition because they will apply machine learning techniques, real time insight and predictive modeling to influence voters even better through micro targeting, statistical analysis and machine algorithms." This is exactly my point. Because they are applying the requisite statistical and machine learning algorithms, they are, by their own assumption, doing predictive modeling. This is also the presiding sentiment here on Kaggle. But we saw how those assumptions failed during the presidential election, just as we see one excuse after another here on Kaggle why the latest competition turned out to be not really predictive modeling. At some point, I would think that we need to ask ourselves what is wrong with the current state of assumptions and work towards improving the design of predictive modeling projects. Yet people are more resigned to attack any messenger (like me) that comes on and challenges their notion that they are, by assumption, doing true predictive modeling.
", "votes": -6, "replies": [] }, { "id": 144389, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T03:39:32.220000", "content": "Oh, and I especially like the part about my hypotheses are thus proven wrong, even though you don't even understand what my hypotheses are. Please stop making a fool of yourself on here.
", "votes": -11, "replies": [] }, { "id": 144387, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T03:38:33.237000", "content": "Wow, that was a whole lot of poor English and poor logic. First, you seem to make the assumption that Kaggle is true predictive modeling, in order to disprove my claim that that Kaggle is not true predictive modeling. Seriously? A freshman course in logic would help you a great deal. I know exactly that predictive modeling is a process, and that process is not allowed to be implemented in its true form by competitions like those presented on Kaggle. Your slide that includes all the corporate jargon does not come close to explaining why companies are wasting millions of dollars on the holy grail predictive models and coming up woefully short. My example of the catastrophic failure of the predictions in the U.S. elections are evidence supporting this claim. And as far as my "true/false predictive modeling potential definition" ... please show me where I attempted to define such a thing. You assert that I said things which I did not, and then you attempt to go about trying to prove these made-up arguments to be false. Again, a freshman course in logic would help you a great deal. Until then, you have failed to "invalidate" anything that I have said. And this type of refusal to acknowledge reality is precisely why the current state of predictive modeling is in such disarray.
", "votes": -18, "replies": [] }, { "id": 144308, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-13T17:52:35.173000", "content": "I'm sure everyone was watching the U.S. election and saw how wrong the experts were on their "predictions." Welcome to the current state of predictive modeling. And yes, I will stand by my statements. Overfitting competitions like these are not doing anything to advance the state of true predictive modeling. The mindset demonstrated by Abriosi above is exactly what I'm talking about. The focus here is to win competitions. And because these competitions are billed as "predictive modeling," then people assume that if they do well in the competitions, they are doing well at "predictive modeling." So yes, you must do some work and change perhaps many parameters to win these competitions. That will give you a higher score and better fits on the available data (thus the overfitting). But what we're seeing over and over and over, both here and elsewhere, is that this focus on overfitting is not getting where we want to go in the field of true predictive modeling. We saw this on a very high level with the U.S. election predictions.
", "votes": -18, "replies": [] }, { "id": 144572, "author_name": "GWillie", "author_url": "", "post_date": "2016-11-14T17:18:48.983000", "content": "Owl, thank you for providing more support for my claim. Kaggle is a great platform for machine learning. What we have to be careful about though is making the leap from there to the assumption that it is thereby necessarily a great platform for predictive modeling. This is exactly the point I am trying to make. As machine learning algorithms get more and more advanced and commonplace, we are not seeing the transition we thought to predictive modeling becoming more and more advanced and accepted. At some point, we need to really examine why that is.
", "votes": 0, "replies": [] }, { "id": 144416, "author_name": "AndreaFGenovese", "author_url": "", "post_date": "2016-11-14T06:55:43.690000", "content": "I disagree that the data gathering got better, if you look at recent laws, they made it harder for pollers to interview people and some states made it just illegal to call at home and various others. Besides, some polls did get it right and they are the ones that used better data collection techniques
", "votes": 0, "replies": [] }, { "id": 144410, "author_name": "leexa", "author_url": "", "post_date": "2016-11-14T05:32:33.247000", "content": "[quote=GWillie;144406]
\n\nAs far as the U.S. elections, data gathering techniques get more and more sophisticated with each election. So I don't think that is the problem. Also, modeling techniques get more and more sophisticated with each election. So my point is, with better and better data gathering techniques and better and better statistical/modeling techniques, we are seeing worse and worse predictions. Seems to me this is evidence that we are getting better and better at fitting models to observed data, but those models are getting less and less effective at predicting unobserved outcomes.
\n\n[/quote]
\n\nThis usually happens when you overfit the data, which is not adequately sampling the true population. Reports in retrospect say that people have neglected the silent voters, for various reasons.
\n\n[quote=GWillie;144406]
\n\nAnd no, election predictions were not "within error." Going into the election night, predictions showed Clinton's chances of winning at between 85 and 92%. Within hours that fell to 40%. This is not within some "error."
\n\n[/quote]
\n\nI was referring to the swing states, where Clinton was predicted to marginally win. The error here is larger than Clinton's marginal win, so both outcomes are probable. This was something I somewhat recalled ...
\n\nLastly, your toned down post makes forums a nicer place to read :)
", "votes": 0, "replies": [] }, { "id": 144573, "author_name": "", "author_url": "", "post_date": "2016-11-14T17:29:38.090000", "content": "", "votes": 0, "replies": [] }, { "id": 144567, "author_name": "", "author_url": "", "post_date": "2016-11-14T17:06:53.750000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "144382": "[quote=GWillie;144371]\r\n\r\nThis in no way refutes the point I made about Kaggle being an overfitting forum and the fact that companies are paying (wasting) a lot of money thinking they are getting predictive models when in fact they are getting models that are overfit to existing data.\r\n[/quote]\r\n\r\nDefinition of overfitting, the stroke area:\r\n\r\n![enter image description here][1]\r\n\r\nI think you should get back to **statistical modeling theory** before writing back (like: **what is overfitting and what is underfitting**). Not only you seem to not know **predictive modeling is a process** and there's **no true/false predictive modeling** (provide us your research paper about it if you are willing to challenge it), that process **does not rely on assumptions of truthness of information to be a process (VERY IMPORTANT)**, and if you know your theory about predictive modeling, the process is an **endless loop**.\r\n\r\nHere is an example from one of my slides with IBM which proves your \"false predictive modeling\" does not exist (**only \"predictive modeling\" is existing**), as errors found on the way are corrected, and the loop goes on indefinitely until a satisfactory solution by all parties is found:\r\n\r\n![enter image description here][2]\r\n\r\nMore or less there's a bias you have and you can't even explain it using theory, therefore no one can understand what you are trying to explain us. Our models are not stopping in the hands on Bosch, we are only a helping part of the process for them and they will have to find a way to exploit them. The story stops here for us at Kaggle, and **we may never know what will happen to the models**: they could be thrown out or even improved, but this is not an issue on our hands.\r\n\r\nThat's how Kaggle works: you are just an **element of crowdsourcing to help predictive modeling to be carried on** for the company who paid Kaggle to get help. **The wealth of information provided from crowdsourcing is something you cannot find using your own data scientists.** Anything found, even a minor detail, can be useful from a design perspective.\r\n\r\nI think you don't get that companies are not paying only for the models. Insights are actually more important as the valuation (non-monetary) kicks in - it is not only about \"numbers\". You should read what is asked in the [winners' documentation][3]. I also think enough time was wasted in a topic (\"true predictive modeling, definition still not given?\") that does not make any sense, as the theory you learnt does not seem assimilated correctly.\r\n\r\nNow that your **\"true/false predictive modeling\" potential definition was invalidated**, are you going to say the theory (which, for the predictive modeling process, is based on pragmatic evidence throughout the years that Data Mining is existing) is wrong? If yes, **do you have any empirical and significant proof?** (hint: there's none - there's a reason it's called a **\"process\" which loops over itself** like a snake, thus becomes **invulnerable to errors as they are theoretically auto-corrected**)\r\n\r\nExtra: If you invaildate \"they are theoretically auto-corrected\" using practical sources (because why not, humans are prone to errors in a process), you end up with a single theoretical \"false modeling\" which is obviously generalizing to all processes as a human must be introduced somewhere. Therefore, there's **only one process existing**, not the existance of two separate processes (True/False). This is what we call a **\"Railing\"** in any existing process in this world, and that's what is supposed to prevent such mistake (if they happen to be created by humans). If your overfitting conviction was true, the Private Leaderboard is supposed to be entirely scrambled, in **absolute values** and not in rank. Obviously, this is not the case, overfitting was nearly non-existant (if not at all for many of us).\r\n\r\nHence, both in theory and in practice your True/False predictive modeling hypotheses (we don't even know what they are, but we don't need to know that) are proven wrong.\r\n\r\n(off-topic: self-note for re-usage of this wall of text for next data science course to teach, copy&paste text)\r\n\r\n(edit: removed IBM from print screen)\r\n\r\n [1]: http://i.imgur.com/pnb6SKV.png\r\n [2]: http://i.imgur.com/EZLBYzN.png\r\n [3]: https://www.kaggle.com/wiki/WinningModelDocumentationTemplate", "144336": "[quote=GWillie;144308]\r\n\r\nThat will give you a **higher score and better fits on the available data (thus the overfitting)**.\r\n\r\n[/quote]\r\n\r\n* Noisy data: check\r\n* Imbalanced data: check\r\n* 30% samples only on Public LB: check\r\n\r\nLeaderboard Public vs Private stability check: check\r\n\r\n![enter image description here][1]\r\n\r\n![enter image description here][2]\r\n\r\nIf we all were truly overfitting no one of us should appear there, cf Santander. The difference is negligible in absolute terms, post-bias correction. Even in Santander, it was not even severe (but in terms of ranks it was severe enough as everyone was very close vs 0.50 random).\r\n\r\nIn any case, following your statement, if a satellite makes a perfect circle (r = fixed distance) around the Earth, it should mean your algorithm to move the rocket periodically is overfitting the circle and cannot generalize to new circles when you will provide another distance. Isn't that too much naive? Obviously, your would try to make your algorithm generalize to any distance you input, not to do perfect on r=10000 only and wreck havoc on r=10000-0.0000000000000000001 which does not make sense if we follow what you said.\r\n\r\nBut yes, in this world humans are making the algorithms for the satellites so they can orbit around the earth without wrecking havoc if you change something by 1 inch. If your statement was true, expect no satellites existing as they would all be destroyed or flying somewhere else without any possible control.\r\n\r\n [1]: http://i.imgur.com/owBMhKj.png\r\n [2]: http://i.imgur.com/TqcVIwn.png", "144581": "@GWillie, I'm completely puzzled by your arguments. You seem to claim that some US election predictions being incorrect therefore shows that machine learning is bad? I may not have an education in statistics, but I think that your sample size of 1 may not be representative of the entire population.\r\n\r\nI also don't understand why you think we're overfitting. The private leaderboard had 700,000 **unseen** samples that we had to predict on, and the leaderboard shows that people did not overfit very much to the 300k samples that they could see the results on versus the private set.\r\n\r\nSince the goal of a machine learning model is to fit to unseen data, and here we have proof that our models can fit to (a considerable amount of) unseen data, logically we can say that our machine learning is not overfitting. How could we have overfitted to the **private test set**?\r\n\r\nAlso, an example of what you consider to be predictive modelling would be great, so we can actually understand your view. Right now it might as well be phoning up your local fortune teller and asking them who's going to win the election.\r\n\r\nPS: You're being downvoted because you're not actually making any logical discussion, rather resorting to attacks on well-respected forum members and saying \"your argument is wrong\" without providing a counterview. If you want a logical discussion, then you can start by presenting your view (i.e. what is predictive modelling?)", "138393": "Here is a new thread, so those [over there][1] can continue to search for their magic feature.\r\n\r\n[@GWillie][2]\r\nI genuinely respect your determination (despite the downvotes) to defend your argument with such thorough posts (23 well written posts, facts aside), \r\n\r\nI also respect the entire kaggle community, as this seems to be what kaggle was made for, not just competitions (predictive modeling or not), but **the open exchange and communication about data science...**\r\n\r\nNow I invite you @GWillie to state your entire argument(s), as I (and I'm sure others as well) will attempt to address the question:\r\n\r\n**Is This Truly Predictive Modeling?**\r\n\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/24065/the-magical-feature-from-lb-0-3-to-0-4\r\n [2]: https://www.kaggle.com/gwillie", "144578": "I made a decision early on never to vote down Kaggle posts. Boy, have I been tempted ...\r\n", "144575": "[quote=GWillie;144387]\r\n\r\nWow, that was a whole lot of poor English and poor logic. \r\n[/quote]\r\n\r\nAttacking a non native speaker on her use of English is not really helpful. Let's switch to French to see how good you are at discussing in a foreign language...\r\n\r\nMore generally, personal attacks aren't great in forums. Stick to ideas and facts, and please do not attack the person.\r\n\r\n", "144614": "Ok, @GWillie. I am willing to assume you are right. \r\n\r\n1) What would you do to \"fix\" overfitting on Kaggle? Disable Public Leaderboard? Not provide test data at all (ask people to submit their fully documented models)? \r\n\r\n2) How would you ensure quality and consistency of training data gathering and preparation by hosting companies to increase usefulness of the models and their ability to generalize to unseen (possibly even not yet collected) data?\r\n\r\n3) How would you go about assembling and incentivizing the data science community around your perfect data science process (is KDN better in your view, by the way)? \r\n\r\nI will dare to say that leaderboard shakeups are part of the thrill and advancing by 0.000001 on the leaderboard is what keeps people glued to the monitors for weeks towards the end of competition. Teams are formed, people learn (statistics and computer science), virtual coins are distributed, people have fun! You don't like this game? You don't have to play it! And, assuming your name is not Nate, go build a better predictive model for US elections! Beat @FiveThirtyEight and be right every time!! We all here will thank you and bow our heads in respect.", "138405": "It is. This is a factory dataset, you want to evaluate if the product is defective or not. Bosh knows about defective lots so it is highly improbable they would make a mistake like leaking a defective product on the ID feature. (factory are highly correlated within its process so it is obvious feature engineering will bring HUGE improvements) EDIT:(wrong assumption... yet they shared the leak!)\r\n\r\nI have been a kaggle member recently and all i can say is that this is the perfect place to learn. People share knowledge without asking anything in return and you can learn way faster that you could on your own, or in the university. (i speak for myself, and i thank for such great people on the community)\r\n\r\nThe post you are referring is sad. All the rating... You win competitions if you go in with the right mindset, and because this area is very challenging you have to be able to learn and innovate your approach. You will never win a competition by slightly changing some parameters on a public kernel. So study, ask questions and people will help you.\r\n\r\nFor instance, in the red hat competition some people even went public with the leak so everybody can learn what it looks like and how to search for it, in case you ever find one which wreaks you model in real life.", "138407": "I'll bite. The \"Bosch\" seems to be \"discriminate\" between \"failure\" and \"non-failure\". However, if at time T we have a failure, and later T + DT, we still have failure and between that we have the \"test\" sample, it becomes interesting to discern between \"predictive\" and \"discriminative\" and \"interpolation\". I.e. the future tells the past?", "144513": "GWilli: can you state your argument again? Specifically regarding overfitting in kaggle competition (validation on unseen data). thanks.\r\n\r\nI personally think the first sentence of Laurae`s response is what u need to counter first. \r\n\r\nPS: Is this a troll attempt? o_O", "144396": "[quote=GWillie;144389]\r\n\r\nOh, and I especially like the part about my hypotheses are thus proven wrong, even though you don't even understand what my hypotheses are. Please stop making a fool of yourself on here.\r\n\r\n[/quote]\r\n\r\nThese are really unkind words. People will be more receptive to your arguments if it were more polite and it helps looking more professional. \r\n\r\nNow, back to the topic. If Kaggle competitions are not truely predictive , then could you give any examples of predictive modelling. \r\n\r\nOn the flip side, i believe kaggle competitions do overfit the data to some extent. We strive to get the last 0.0000000001 score, where in real life the sampling error due to time or spatial constrains are larger than the final decimal point of the score. (my first lesson in physics on uncertainty of measurement) . However, an overfit model still has predictive power, just less than what it claims. \r\n\r\nLastly, regarding the elections, the prediction was within error, where the error is in the fine print which nobody reads. Also there was a large sampling bias/error for some subgroups of people. The statistically insignificant sample size of 1 you gave is both invalid and easily explained. \r\n\r\n", "144393": "I think your example of U.S. elections doesn't reflect how was the data gathered. The underlying problems are that the data itself did not satisfy iid conditions. For example, polling was not permitted on random telephone numbers, so in most cases the poll samples were made people attending talks, conventions or other political events. Moreover, some polls used a repeated sample. meaning the same people were asked over and over again if they changed their mind through time. There is also the question of representational weighting to try to account for different backgrounds and ethnicities which allowed for the data to be spoiled by the choices of the data collector. Hence what im trying to say, its that the US election polls are not a good example of how good/bad predictive modeling is since the data was poor in the first place.", "144406": "Sorry, but I don't see how someone comes and says several times in one post that he proved me wrong and yet doesn't even understand what it is he's proving wrong. In my opinion, that is making a fool of oneself and thus should be noted as such. I don't know how I can be \"polite\" when getting so undeservedly attacked like that.\r\n\r\nAs far as the U.S. elections, data gathering techniques get more and more sophisticated with each election. So I don't think that is the problem. Also, modeling techniques get more and more sophisticated with each election. So my point is, with better and better data gathering techniques and better and better statistical/modeling techniques, we are seeing worse and worse predictions. Seems to me this is evidence that we are getting better and better at fitting models to observed data, but those models are getting less and less effective at predicting unobserved outcomes.\r\n\r\nAnd no, election predictions were not \"within error.\" Going into the election night, predictions showed Clinton's chances of winning at between 85 and 92%. Within hours that fell to 40%. This is not within some \"error.\"\r\n\r\nAs far as overfitting, it's not about that last 0.0000000001 score, it's the relatively large movements such as doubling scores within 20-30 submissions. And while some overfit models can have predictive power (even a stopped clock is right twice a day), my point is that this is not, in general, a good robust approach to predictive modeling. And the fact that many models and competitions which strive to find good predictive models end up going unused is evidence of this.", "144371": "I don't follow your point. Yes, obviously you don't construct an algorithm that does not include the variable over which you would like to vary. This in no way refutes the point I made about Kaggle being an overfitting forum and the fact that companies are paying (wasting) a lot of money thinking they are getting predictive models when in fact they are getting models that are overfit to existing data. We see more and more evidence of this every day.", "144580": "Please, feel free to vote me down, I'm used to it by now. The fact that I get voted down with no real attempt at logically discussing what I'm saying is evidence to me that I'm on the right track.", "144577": "I would not attempt to display such a lack of knowledge in logic in a forum where I did not speak the language. That was quite the display of hubris on her part. I was simply pointing that out.", "144576": "Continuing to hail the advances of machine learning is not adequately addressing the needs with regard to predictive modeling.", "144574": "And it's more evidence that, as I said, the adoption of machine learning continues to advance while true predictive modeling continues to stagnate, or even decline as we saw in the recent Presidential election.", "144570": "As far as addressing the first sentence of Laurae's response, I'm not sure which of his/her two responses you are referring to. In the first response, the first sentence basically makes the argument that, if we weren't all doing predictive modeling here, there wouldn't be so many people on the leaderboard with such high scores. But that couldn't be further from the truth, and is exactly the point I'm trying to make. You can have many people using the same canned machine learning fitting algorithms, and thus getting similar scores, and you can watch them all increase their scores and climb up the leader board as they find ways to exploit and overfit the data during the course of their hundreds and hundreds of submissions. This in no way proves that they are all creating models that will be effective in making predictions. In fact, the tendency to overfit - that this whole endeavor rewards - will do exactly the opposite. As far as the public vs. private leaderboard issue, it seems that for most of the Kaggle competitions I've seen, high scores on the private leaderboard are often just an indication that the hold-out portion of the test sets were very similar to the non hold-out portions, thus rewarding overfitting to the training set and non hold-out portion of the test set.\r\n\r\nThe first sentence of his/her second response is an assumption on his/her part that I don't know what overfitting is. I don't know that baseless assumptions like that are even worthy of a response. He/she does go on to lecture me about how predictive modeling is a process and shows me a corporate slide outlining their process. While predictive modeling is a process, it is, more importantly, a design. You can apply the same process over and over to poorly designed projects and achieve the same inadequate results over and over. And in fact, this approach of setting up a process and then churning things through that mill without adequate formulation and design is yet another reason why the field has stagnated. Regarding proper design, the way that these Kaggle competitions are set up often limits ones ability to adequately formulate and design a model that will have good predictive capability. The need to keep data anonymous opens the door for people to overfit to specific instances of data and severely limits ones ability to create the necessary mappings between continuous input variables and either categorical or continuous response variables.\r\n\r\n", "144561": "Here is a quote from a big data/data driven modeller, \"In the day and age of data driven insights political campaigners that added big data techniques to their tool belt will have undoubtedly a clear advantage over the competition because they will apply machine learning techniques, real time insight and predictive modeling to influence voters even better through micro targeting, statistical analysis and machine algorithms.\" This is exactly my point. Because they are applying the requisite statistical and machine learning algorithms, they are, by their own assumption, doing predictive modeling. This is also the presiding sentiment here on Kaggle. But we saw how those assumptions failed during the presidential election, just as we see one excuse after another here on Kaggle why the latest competition turned out to be not really predictive modeling. At some point, I would think that we need to ask ourselves what is wrong with the current state of assumptions and work towards improving the design of predictive modeling projects. Yet people are more resigned to attack any messenger (like me) that comes on and challenges their notion that they are, by assumption, doing true predictive modeling.", "144389": "Oh, and I especially like the part about my hypotheses are thus proven wrong, even though you don't even understand what my hypotheses are. Please stop making a fool of yourself on here.", "144387": "Wow, that was a whole lot of poor English and poor logic. First, you seem to make the assumption that Kaggle is true predictive modeling, in order to disprove my claim that that Kaggle is not true predictive modeling. Seriously? A freshman course in logic would help you a great deal. I know exactly that predictive modeling is a process, and that process is not allowed to be implemented in its true form by competitions like those presented on Kaggle. Your slide that includes all the corporate jargon does not come close to explaining why companies are wasting millions of dollars on the holy grail predictive models and coming up woefully short. My example of the catastrophic failure of the predictions in the U.S. elections are evidence supporting this claim. And as far as my \"true/false predictive modeling potential definition\" ... please show me where I attempted to define such a thing. You assert that I said things which I did not, and then you attempt to go about trying to prove these made-up arguments to be false. Again, a freshman course in logic would help you a great deal. Until then, you have failed to \"invalidate\" anything that I have said. And this type of refusal to acknowledge reality is precisely why the current state of predictive modeling is in such disarray.", "144308": "I'm sure everyone was watching the U.S. election and saw how wrong the experts were on their \"predictions.\" Welcome to the current state of predictive modeling. And yes, I will stand by my statements. Overfitting competitions like these are not doing anything to advance the state of true predictive modeling. The mindset demonstrated by Abriosi above is exactly what I'm talking about. The focus here is to win competitions. And because these competitions are billed as \"predictive modeling,\" then people assume that if they do well in the competitions, they are doing well at \"predictive modeling.\" So yes, you must do some work and change perhaps many parameters to win these competitions. That will give you a higher score and better fits on the available data (thus the overfitting). But what we're seeing over and over and over, both here and elsewhere, is that this focus on overfitting is not getting where we want to go in the field of true predictive modeling. We saw this on a very high level with the U.S. election predictions.", "144572": "Owl, thank you for providing more support for my claim. Kaggle is a great platform for machine learning. What we have to be careful about though is making the leap from there to the assumption that it is thereby necessarily a great platform for predictive modeling. This is exactly the point I am trying to make. As machine learning algorithms get more and more advanced and commonplace, we are not seeing the transition we thought to predictive modeling becoming more and more advanced and accepted. At some point, we need to really examine why that is.", "144416": "I disagree that the data gathering got better, if you look at recent laws, they made it harder for pollers to interview people and some states made it just illegal to call at home and various others. Besides, some polls did get it right and they are the ones that used better data collection techniques", "144410": "[quote=GWillie;144406]\r\n\r\nAs far as the U.S. elections, data gathering techniques get more and more sophisticated with each election. So I don't think that is the problem. Also, modeling techniques get more and more sophisticated with each election. So my point is, with better and better data gathering techniques and better and better statistical/modeling techniques, we are seeing worse and worse predictions. **Seems to me this is evidence that we are getting better and better at fitting models to observed data, but those models are getting less and less effective at predicting unobserved outcomes.**\r\n\r\n[/quote]\r\n\r\nThis usually happens when you overfit the data, which is not adequately sampling the true population. Reports in retrospect say that people have neglected the silent voters, for various reasons. \r\n\r\n\r\n[quote=GWillie;144406]\r\n\r\nAnd no, election predictions were not \"within error.\" Going into the election night, predictions showed Clinton's chances of winning at between 85 and 92%. Within hours that fell to 40%. This is not within some \"error.\"\r\n\r\n[/quote]\r\n\r\nI was referring to the swing states, where Clinton was predicted to marginally win. The error here is larger than Clinton's marginal win, so both outcomes are probable. This was something I somewhat recalled ...\r\n\r\nLastly, your toned down post makes forums a nicer place to read :)", "144573": "", "144567": "" } }