Hi, I've begun work on this and I'm just wondering why such a prestigious company, with probably very deep pockets, is offering such a low reward for what seems to me extremely hard work?
\n\nThere are many competitions on Kaggle that offer 15K +(plus potential job) to the top team for, say, 1/4 of the work of this contest. I get that folks will be competing for the prestige and some just for fun, but what I'm at a loss here, is what Bosch expects to get out of this? I mean the dataset is extremely large in terms of rows, and the number of features is roughly > 3K! Additionally, there is a date/time component which further adds to the complexity of the solution.
\n\nDon't get me wrong, I don't mind the challenge, but I'm pressed for time and imo the competition reward should correlate with the opportunity cost. Do others feel my concern on this, or is this problem easier than I think?
\n\nThanks and good luck!
", "messages": [ { "id": 137947, "postDate": "2016-10-06T07:42:53.437Z", "content": "The pricemoney seems to go steadily down since years. I think the reason is, that the euphoria about (crowd intelligence) machine learning died a little after some of the competitions that got a lot of attention, like the netflix competition. Most of the solutions of big competitions with big money turned out to be useless in practice, being it due to complexity (as in netflix) or overfitting to the competition setting, including leakage, leaderboard feedback or poor generalization to outside data for whatever reasons. Im pretty sure responsible people in the big companies nowadays know about this risk and the pricemoney is adjusted accordingly. With 30k Bosch has nothing to lose and can just see what happens.
", "rawMarkdown": "The pricemoney seems to go steadily down since years. I think the reason is, that the euphoria about (crowd intelligence) machine learning died a little after some of the competitions that got a lot of attention, like the netflix competition. Most of the solutions of big competitions with big money turned out to be useless in practice, being it due to complexity (as in netflix) or overfitting to the competition setting, including leakage, leaderboard feedback or poor generalization to outside data for whatever reasons. Im pretty sure responsible people in the big companies nowadays know about this risk and the pricemoney is adjusted accordingly. With 30k Bosch has nothing to lose and can just see what happens.\r\n", "votes": 5 }, { "id": 137918, "postDate": "2016-10-06T01:22:13.867Z", "content": "Maybe if they threw in a free dishwasher to sweeten the pot ...
", "rawMarkdown": "Maybe if they threw in a free dishwasher to sweeten the pot ...", "votes": 5 }, { "id": 138285, "postDate": "2016-10-07T22:32:16.373Z", "content": "[quote=aldente;138276]
\n\n(1) In real world you should be in favor of "false positive" than "false negative".
\n\n(2) In this competition, participants walk in different path than in business, for the sake of higher LB's MCC score.
\n\nAm I missing something?
\n\n[/quote]
\n\nI think you may be misunderstanding how MCC is calculated and for what purposes. MCC is a very legitimate score for imbalanced datasets such as this one.
\n\nLet's try with an example. Below is a line with real sequence of events, followed by a line that represents our prediction:
\n\n0000000100\n0000000001\nThe prediction has one 1 FN and 1 FP. Its accuracy is 80% and MCC is -0.111
\n\nNext:
\n\n1010101010\n1010101001\nThis prediction also has 1 FN and 1 FP and its accuracy is 80%, but its MCC is 0.6. This also happens to be a much more balanced dataset.
\n\nNext:
\n\n0000000100\n0000000000\nNow we have "fixed" 1 FP and the accuracy here is 90%, yet MCC will go up only marginally to 0.0 compared to the first example.
\n\nNext:
\n\n0000000100\n0000000101\nNow we have "fixed" 1 FN from the first example. Again the accuracy is 90% but MCC will jump to 0.666, because we have fixed the underrepresented class.
\n\nMCC will penalize the errors in a less abundant class much more, which I think is appropriate for Bosch's business purposes.
", "rawMarkdown": "[quote=aldente;138276]\r\n\r\n(1) In real world you should be in favor of \"false positive\" than \"false negative\". \r\n \r\n(2) In this competition, participants walk in different path than in business, for the sake of higher LB's MCC score. \r\n\r\nAm I missing something?\r\n\r\n[/quote]\r\n\r\nI think you may be misunderstanding how MCC is calculated and for what purposes. MCC is a very legitimate score for imbalanced datasets such as this one.\r\n\r\nLet's try with an example. Below is a line with real sequence of events, followed by a line that represents our prediction:\r\n\r\n 0000000100\r\n 0000000001\r\n\r\nThe prediction has one 1 FN and 1 FP. Its accuracy is 80% and MCC is -0.111\r\n\r\nNext:\r\n\r\n 1010101010\r\n 1010101001\r\n\r\nThis prediction also has 1 FN and 1 FP and its accuracy is 80%, but its MCC is 0.6. This also happens to be a much more balanced dataset.\r\n\r\nNext:\r\n\r\n 0000000100\r\n 0000000000\r\n\r\nNow we have \"fixed\" 1 FP and the accuracy here is 90%, yet MCC will go up only marginally to 0.0 compared to the first example.\r\n\r\nNext:\r\n\r\n 0000000100\r\n 0000000101\r\n\r\nNow we have \"fixed\" 1 FN from the first example. Again the accuracy is 90% but MCC will jump to 0.666, because we have fixed the underrepresented class.\r\n\r\nMCC will penalize the errors in a less abundant class much more, which I think is appropriate for Bosch's business purposes.", "votes": 4 }, { "id": 138310, "postDate": "2016-10-08T00:26:50.927Z", "content": "[quote=aldente;138292]
\n\nWait a minute..,, maybe because of the number of 1 is far less than 0? Or am I missing something again?
\n\n[/quote]
\n\nYes, it is because there are 171 times more 0s than 1s in the dataset. But beyond that, it may help you if you think about the numerator part of MCC:
\n\n(TP*TN) - (FP*FN)
\n\nSo when will this expression have maximal value? One obvious answer is when the (FP*FN) is small or zero. The not so obvious answer is when TP=TN (i.e., when classes are perfectly balanced), because that will maximize (TP*TN). If you have 100 data points with a perfect balance of two classes, TP*TN=2500. If you introduce even the slightest imbalance (49 vs 51), the product will be 2499, and it will keep decreasing as the imbalance grows. That is why MCC is one of better scores to use for imbalanced datasets.
", "rawMarkdown": "[quote=aldente;138292]\r\n\r\nWait a minute..,, maybe because of the number of 1 is far less than 0? Or am I missing something again? \r\n\r\n[/quote]\r\n\r\nYes, it is because there are 171 times more 0s than 1s in the dataset. But beyond that, it may help you if you think about the numerator part of MCC:\r\n\r\n(TP*TN) - (FP*FN)\r\n\r\nSo when will this expression have maximal value? One obvious answer is when the (FP*FN) is small or zero. The not so obvious answer is when TP=TN (i.e., when classes are perfectly balanced), because that will maximize (TP*TN). If you have 100 data points with a perfect balance of two classes, TP*TN=2500. If you introduce even the slightest imbalance (49 vs 51), the product will be 2499, and it will keep decreasing as the imbalance grows. That is why MCC is one of better scores to use for imbalanced datasets.\r\n", "votes": 1 }, { "id": 138307, "postDate": "2016-10-08T00:06:41.267Z", "content": "[quote=aldente;138306]
\n\n@Taffey Lewis, I'm not sure I understand it correctly. Is it because of changes in number of FP/TP will also change number of FN/TN?
\n\n[/quote]
\n\nYes.
", "rawMarkdown": "[quote=aldente;138306]\r\n\r\n@Taffey Lewis, I'm not sure I understand it correctly. Is it because of changes in number of FP/TP will also change number of FN/TN? \r\n\r\n[/quote]\r\n\r\nYes.", "votes": 1 }, { "id": 137971, "postDate": "2016-10-06T10:48:20.247Z", "content": "Expectations of Bosch: "In this competition, Bosch is challenging Kagglers to predict internal failures using thousands of measurements and tests made for each component along the assembly line. This would enable Bosch to bring quality products at lower costs to the end user."
\n\nMany ML methods enable to reduce the number of features needed to obtain good results. Each measurement costs something, so reduction of ~3K features (measurements) to 200-500 may give not only a better prediction of internal failures but also additional reduction of costs - exactly what they want or even a little bit more. If there is no leak, it is a really good competition.
", "rawMarkdown": "Expectations of Bosch: *\"In this competition, Bosch is challenging Kagglers to predict internal failures using thousands of measurements and tests made for each component along the assembly line. This would enable Bosch to bring quality products at lower costs to the end user.\"*\r\n\r\nMany ML methods enable to reduce the number of features needed to obtain good results. Each measurement costs something, so reduction of ~3K features (measurements) to 200-500 may give not only a better prediction of internal failures but also additional reduction of costs - exactly what they want or even a little bit more. If there is no leak, it is a really good competition.", "votes": 1 }, { "id": 137915, "postDate": "2016-10-06T01:10:28.133Z", "content": "Hi, I've begun work on this and I'm just wondering why such a prestigious company, with probably very deep pockets, is offering such a low reward for what seems to me extremely hard work?
\n\nThere are many competitions on Kaggle that offer 15K +(plus potential job) to the top team for, say, 1/4 of the work of this contest. I get that folks will be competing for the prestige and some just for fun, but what I'm at a loss here, is what Bosch expects to get out of this? I mean the dataset is extremely large in terms of rows, and the number of features is roughly > 3K! Additionally, there is a date/time component which further adds to the complexity of the solution.
\n\nDon't get me wrong, I don't mind the challenge, but I'm pressed for time and imo the competition reward should correlate with the opportunity cost. Do others feel my concern on this, or is this problem easier than I think?
\n\nThanks and good luck!
", "rawMarkdown": "Hi, I've begun work on this and I'm just wondering why such a prestigious company, with probably very deep pockets, is offering such a low reward for what seems to me extremely hard work? \r\n\r\nThere are many competitions on Kaggle that offer 15K +(plus potential job) to the top team for, say, 1/4 of the work of this contest. I get that folks will be competing for the prestige and some just for fun, but what I'm at a loss here, is what Bosch expects to get out of this? I mean the dataset is extremely large in terms of rows, and the number of features is roughly > 3K! Additionally, there is a date/time component which further adds to the complexity of the solution.\r\n\r\nDon't get me wrong, I don't mind the challenge, but I'm pressed for time and imo the competition reward should correlate with the opportunity cost. Do others feel my concern on this, or is this problem easier than I think? \r\n\r\nThanks and good luck!\r\n\r\n\r\n\r\n\r\n\r\n\r\n", "votes": 1 }, { "id": 138417, "postDate": "2016-10-08T19:21:43.447Z", "content": "The relationship between reward and the inputs of time and effort is highly non-linear and a bit random. But I assure you that in many endeavors, you can get quite far in the long run through continued, intense, and varied effort. The time horizon should not be a single contest, but your entire lifetime.
\n\nYou are competing against people with a negative or zero disutility of effort with respect to Kaggle and predictive modeling. Such people often pay to play - either directly via lost income or indirectly via time. Given such competition, it is difficult to win except in small data contests where the final placement is often mostly a lottery.
", "rawMarkdown": "The relationship between reward and the inputs of time and effort is highly non-linear and a bit random. But I assure you that in many endeavors, you can get quite far in the long run through continued, intense, and varied effort. The time horizon should not be a single contest, but your entire lifetime.\r\n\r\nYou are competing against people with a negative or zero disutility of effort with respect to Kaggle and predictive modeling. Such people often pay to play - either directly via lost income or indirectly via time. Given such competition, it is difficult to win except in small data contests where the final placement is often mostly a lottery. ", "votes": 2 }, { "id": 137984, "postDate": "2016-10-06T13:06:29.327Z", "content": "I totally get your point @wordsmith.
\n\nAnswering your last question: yes it gets easier by time since you get more experience tackling problems, sometimes you find the problem similar to that of previous competitions, you have more code to reuse, you become better at using the tools (R, Python, other frameworks/tools). Also some of the top Kagglers managed to secure strong computing infrastructure to speed up their work.
\n\nFinally i think we don't do that for the money (though it is a strong motivation), there are many other reasons such as:
\n\n1) Growing & Learning (specially from other Kagglers)
\n2) Challenge & technical satisfaction
\n3) Building data science profile
\n4) Creating value by solving difficult problems (ex: Healthcare competitions)
I believe that any of those points are worth the effort ...
\n\nGood Luck
", "rawMarkdown": "I totally get your point @wordsmith.\r\n\r\nAnswering your last question: yes it gets easier by time since you get more experience tackling problems, sometimes you find the problem similar to that of previous competitions, you have more code to reuse, you become better at using the tools (R, Python, other frameworks/tools). Also some of the top Kagglers managed to secure strong computing infrastructure to speed up their work.\r\n\r\nFinally i think we don't do that for the money (though it is a strong motivation), there are many other reasons such as:\r\n\r\n1) Growing & Learning (specially from other Kagglers)Properly because they are already very good at managing their production lines (by looking at the failure ratio) and this is just like a test project to see if ML can help them further. \nGood luck fellows :)
", "rawMarkdown": "Properly because they are already very good at managing their production lines (by looking at the failure ratio) and this is just like a test project to see if ML can help them further. \r\nGood luck fellows :)", "votes": 2 }, { "id": 138306, "postDate": "2016-10-07T23:54:42.567Z", "content": "@Taffey Lewis, I'm not sure I understand it correctly. Is it because of changes in number of FP/TP will also change number of FN/TN?
", "rawMarkdown": "@Taffey Lewis, I'm not sure I understand it correctly. Is it because of changes in number of FP/TP will also change number of FN/TN? " }, { "id": 138295, "postDate": "2016-10-07T23:23:57.437Z", "content": "Aldente, it's the same thing. I just used TP as my point of reference in the other thread.
", "rawMarkdown": "Aldente, it's the same thing. I just used TP as my point of reference in the other thread.\r\n" }, { "id": 138292, "postDate": "2016-10-07T23:10:24.827Z", "content": "Hi @Tilii, thank you so much, looks like that's what I was missing ;)
\n\nNow I'm confused, from what I read in other forum's threads in order to get better MCC, reducing FP (maximizing TP) is important. But from your example, reducing FN gives better MCC.
\n\nWait a minute..,, maybe because of the number of 1 is far less than 0? Or am I missing something again?
", "rawMarkdown": "Hi @Tilii, thank you so much, looks like that's what I was missing ;) \r\n\r\nNow I'm confused, from what I read in other forum's threads in order to get better MCC, reducing FP (maximizing TP) is important. But from your example, reducing FN gives better MCC. \r\n\r\nWait a minute..,, maybe because of the number of 1 is far less than 0? Or am I missing something again? \r\n" }, { "id": 138276, "postDate": "2016-10-07T21:40:07.373Z", "content": "Hi! I'm more concerned about an aspect of business value of the results of competition.
\n\nSorry but IMHO, there is missing link between business value (real world) and this competition, especially in the usage of MCC.
\n\n(1) In real world you should be in favor of "false positive" than "false negative".
\n\nIt's dangerous for businesses to predict FN, i.e. to predict "good/0" when actually it's "failure/1".
\n\nSamsung was recalling millions of new Galaxy Note 7 over some battery failure (explosion). \nToyota was recalling millions of Prius and Lexus over airbag failure. \nBusinesses prefer FP over FN.
\n\n(2) In this competition, participants walk in different path than in business, for the sake of higher LB's MCC score.
\n\nParticipants pay attention more to maximize TP hence to minimize FP than FN. \nWe try to suppress FP as less as possible, while FN have less impact to the MCC score.
\n\nIn contrast to the business' concerns, in this competition it's more dangerous to predict FP (i.e. to predict "failure/1" when actually it's "good/0").
\n\nAlthough indeed less FP = more TP, but both FP and FN should be important, and the FN should have more attention for safety of the products, an aspect of business value.
\n\nAm I missing something? CMIIW. Thank you :)
", "rawMarkdown": "Hi! I'm more concerned about an aspect of business value of the results of competition. \r\n\r\nSorry but IMHO, there is missing link between business value (real world) and this competition, especially in the usage of MCC. \r\n\r\n(1) In real world you should be in favor of \"false positive\" than \"false negative\". \r\n\r\n**It's dangerous for businesses to predict FN, i.e. to predict \"good/0\" when actually it's \"failure/1\".** \r\n\r\nSamsung was recalling millions of new Galaxy Note 7 over some battery failure (explosion). \r\nToyota was recalling millions of Prius and Lexus over airbag failure. \r\nBusinesses prefer FP over FN. \r\n \r\n(2) In this competition, participants walk in different path than in business, for the sake of higher LB's MCC score. \r\n\r\nParticipants pay attention more to maximize TP hence to minimize FP than FN. \r\nWe try to suppress FP as less as possible, while FN have less impact to the MCC score. \r\n\r\nIn contrast to the business' concerns, in this competition **it's more dangerous to predict FP (i.e. to predict \"failure/1\" when actually it's \"good/0\").** \r\n\r\nAlthough indeed less FP = more TP, but both FP and FN should be important, and the FN should have more attention for safety of the products, an aspect of business value. \r\n\r\nAm I missing something? CMIIW. Thank you :) \r\n" }, { "id": 138115, "postDate": "2016-10-07T08:56:52.760Z", "content": "Thanks for the informative responses. They were motivating and much appreciated. @Grzegorz your comment about their interest in looking at cutting back on measurements to save money makes a lot of sense.
\n\nAnd so, I'm guessing, then, that Bosch is looking for very, very, outside-the-box strategies that their teams have not come up with? I mean, is it a good assumption that they've run their data on the latest, leading-edge technology, using most, if not all, open/closed source machine learning software packages?
\n\nCheers
", "rawMarkdown": "Thanks for the informative responses. They were motivating and much appreciated. @Grzegorz your comment about their interest in looking at cutting back on measurements to save money makes a lot of sense. \r\n\r\nAnd so, I'm guessing, then, that Bosch is looking for very, very, outside-the-box strategies that their teams have not come up with? I mean, is it a good assumption that they've run their data on the latest, leading-edge technology, using most, if not all, open/closed source machine learning software packages?\r\n\r\nCheers " }, { "id": 137933, "postDate": "2016-10-06T03:56:03.147Z", "rawMarkdown": "", "isDeleted": true }, { "id": 138313, "postDate": "2016-10-08T00:34:22.903Z", "content": "Excellent, thank you ;)
", "rawMarkdown": "Excellent, thank you ;) " } ], "comments": [ { "id": 137947, "author_name": "Malte Nalenz", "author_url": "", "post_date": "2016-10-06T07:42:53.437000", "content": "The pricemoney seems to go steadily down since years. I think the reason is, that the euphoria about (crowd intelligence) machine learning died a little after some of the competitions that got a lot of attention, like the netflix competition. Most of the solutions of big competitions with big money turned out to be useless in practice, being it due to complexity (as in netflix) or overfitting to the competition setting, including leakage, leaderboard feedback or poor generalization to outside data for whatever reasons. Im pretty sure responsible people in the big companies nowadays know about this risk and the pricemoney is adjusted accordingly. With 30k Bosch has nothing to lose and can just see what happens.
", "votes": 5, "replies": [] }, { "id": 137918, "author_name": "BreakfastPirate", "author_url": "", "post_date": "2016-10-06T01:22:13.867000", "content": "Maybe if they threw in a free dishwasher to sweeten the pot ...
", "votes": 5, "replies": [] }, { "id": 138285, "author_name": "Tilii", "author_url": "", "post_date": "2016-10-07T22:32:16.373000", "content": "[quote=aldente;138276]
\n\n(1) In real world you should be in favor of "false positive" than "false negative".
\n\n(2) In this competition, participants walk in different path than in business, for the sake of higher LB's MCC score.
\n\nAm I missing something?
\n\n[/quote]
\n\nI think you may be misunderstanding how MCC is calculated and for what purposes. MCC is a very legitimate score for imbalanced datasets such as this one.
\n\nLet's try with an example. Below is a line with real sequence of events, followed by a line that represents our prediction:
\n\n0000000100\n0000000001\nThe prediction has one 1 FN and 1 FP. Its accuracy is 80% and MCC is -0.111
\n\nNext:
\n\n1010101010\n1010101001\nThis prediction also has 1 FN and 1 FP and its accuracy is 80%, but its MCC is 0.6. This also happens to be a much more balanced dataset.
\n\nNext:
\n\n0000000100\n0000000000\nNow we have "fixed" 1 FP and the accuracy here is 90%, yet MCC will go up only marginally to 0.0 compared to the first example.
\n\nNext:
\n\n0000000100\n0000000101\nNow we have "fixed" 1 FN from the first example. Again the accuracy is 90% but MCC will jump to 0.666, because we have fixed the underrepresented class.
\n\nMCC will penalize the errors in a less abundant class much more, which I think is appropriate for Bosch's business purposes.
", "votes": 4, "replies": [] }, { "id": 138310, "author_name": "Tilii", "author_url": "", "post_date": "2016-10-08T00:26:50.927000", "content": "[quote=aldente;138292]
\n\nWait a minute..,, maybe because of the number of 1 is far less than 0? Or am I missing something again?
\n\n[/quote]
\n\nYes, it is because there are 171 times more 0s than 1s in the dataset. But beyond that, it may help you if you think about the numerator part of MCC:
\n\n(TP*TN) - (FP*FN)
\n\nSo when will this expression have maximal value? One obvious answer is when the (FP*FN) is small or zero. The not so obvious answer is when TP=TN (i.e., when classes are perfectly balanced), because that will maximize (TP*TN). If you have 100 data points with a perfect balance of two classes, TP*TN=2500. If you introduce even the slightest imbalance (49 vs 51), the product will be 2499, and it will keep decreasing as the imbalance grows. That is why MCC is one of better scores to use for imbalanced datasets.
", "votes": 1, "replies": [] }, { "id": 138307, "author_name": "Taffey Lewis", "author_url": "", "post_date": "2016-10-08T00:06:41.267000", "content": "[quote=aldente;138306]
\n\n@Taffey Lewis, I'm not sure I understand it correctly. Is it because of changes in number of FP/TP will also change number of FN/TN?
\n\n[/quote]
\n\nYes.
", "votes": 1, "replies": [] }, { "id": 137971, "author_name": "Grzegorz Sionkowski", "author_url": "", "post_date": "2016-10-06T10:48:20.247000", "content": "Expectations of Bosch: "In this competition, Bosch is challenging Kagglers to predict internal failures using thousands of measurements and tests made for each component along the assembly line. This would enable Bosch to bring quality products at lower costs to the end user."
\n\nMany ML methods enable to reduce the number of features needed to obtain good results. Each measurement costs something, so reduction of ~3K features (measurements) to 200-500 may give not only a better prediction of internal failures but also additional reduction of costs - exactly what they want or even a little bit more. If there is no leak, it is a really good competition.
", "votes": 1, "replies": [] }, { "id": 138417, "author_name": "Mike Kim", "author_url": "", "post_date": "2016-10-08T19:21:43.447000", "content": "The relationship between reward and the inputs of time and effort is highly non-linear and a bit random. But I assure you that in many endeavors, you can get quite far in the long run through continued, intense, and varied effort. The time horizon should not be a single contest, but your entire lifetime.
\n\nYou are competing against people with a negative or zero disutility of effort with respect to Kaggle and predictive modeling. Such people often pay to play - either directly via lost income or indirectly via time. Given such competition, it is difficult to win except in small data contests where the final placement is often mostly a lottery.
", "votes": 2, "replies": [] }, { "id": 137984, "author_name": "karimo", "author_url": "", "post_date": "2016-10-06T13:06:29.327000", "content": "I totally get your point @wordsmith.
\n\nAnswering your last question: yes it gets easier by time since you get more experience tackling problems, sometimes you find the problem similar to that of previous competitions, you have more code to reuse, you become better at using the tools (R, Python, other frameworks/tools). Also some of the top Kagglers managed to secure strong computing infrastructure to speed up their work.
\n\nFinally i think we don't do that for the money (though it is a strong motivation), there are many other reasons such as:
\n\n1) Growing & Learning (specially from other Kagglers)
\n2) Challenge & technical satisfaction
\n3) Building data science profile
\n4) Creating value by solving difficult problems (ex: Healthcare competitions)
I believe that any of those points are worth the effort ...
\n\nGood Luck
", "votes": 2, "replies": [] }, { "id": 137929, "author_name": "", "author_url": "", "post_date": "2016-10-06T03:51:15.233000", "content": "Properly because they are already very good at managing their production lines (by looking at the failure ratio) and this is just like a test project to see if ML can help them further. \nGood luck fellows :)
", "votes": 2, "replies": [] }, { "id": 138306, "author_name": "aldente", "author_url": "", "post_date": "2016-10-07T23:54:42.567000", "content": "@Taffey Lewis, I'm not sure I understand it correctly. Is it because of changes in number of FP/TP will also change number of FN/TN?
", "votes": 0, "replies": [] }, { "id": 138295, "author_name": "Taffey Lewis", "author_url": "", "post_date": "2016-10-07T23:23:57.437000", "content": "Aldente, it's the same thing. I just used TP as my point of reference in the other thread.
", "votes": 0, "replies": [] }, { "id": 138292, "author_name": "aldente", "author_url": "", "post_date": "2016-10-07T23:10:24.827000", "content": "Hi @Tilii, thank you so much, looks like that's what I was missing ;)
\n\nNow I'm confused, from what I read in other forum's threads in order to get better MCC, reducing FP (maximizing TP) is important. But from your example, reducing FN gives better MCC.
\n\nWait a minute..,, maybe because of the number of 1 is far less than 0? Or am I missing something again?
", "votes": 0, "replies": [] }, { "id": 138276, "author_name": "aldente", "author_url": "", "post_date": "2016-10-07T21:40:07.373000", "content": "Hi! I'm more concerned about an aspect of business value of the results of competition.
\n\nSorry but IMHO, there is missing link between business value (real world) and this competition, especially in the usage of MCC.
\n\n(1) In real world you should be in favor of "false positive" than "false negative".
\n\nIt's dangerous for businesses to predict FN, i.e. to predict "good/0" when actually it's "failure/1".
\n\nSamsung was recalling millions of new Galaxy Note 7 over some battery failure (explosion). \nToyota was recalling millions of Prius and Lexus over airbag failure. \nBusinesses prefer FP over FN.
\n\n(2) In this competition, participants walk in different path than in business, for the sake of higher LB's MCC score.
\n\nParticipants pay attention more to maximize TP hence to minimize FP than FN. \nWe try to suppress FP as less as possible, while FN have less impact to the MCC score.
\n\nIn contrast to the business' concerns, in this competition it's more dangerous to predict FP (i.e. to predict "failure/1" when actually it's "good/0").
\n\nAlthough indeed less FP = more TP, but both FP and FN should be important, and the FN should have more attention for safety of the products, an aspect of business value.
\n\nAm I missing something? CMIIW. Thank you :)
", "votes": 0, "replies": [] }, { "id": 138115, "author_name": "happyholidays", "author_url": "", "post_date": "2016-10-07T08:56:52.760000", "content": "Thanks for the informative responses. They were motivating and much appreciated. @Grzegorz your comment about their interest in looking at cutting back on measurements to save money makes a lot of sense.
\n\nAnd so, I'm guessing, then, that Bosch is looking for very, very, outside-the-box strategies that their teams have not come up with? I mean, is it a good assumption that they've run their data on the latest, leading-edge technology, using most, if not all, open/closed source machine learning software packages?
\n\nCheers
", "votes": 0, "replies": [] }, { "id": 137933, "author_name": "", "author_url": "", "post_date": "2016-10-06T03:56:03.147000", "content": "", "votes": 0, "replies": [] }, { "id": 138313, "author_name": "aldente", "author_url": "", "post_date": "2016-10-08T00:34:22.903000", "content": "Excellent, thank you ;)
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "137947": "The pricemoney seems to go steadily down since years. I think the reason is, that the euphoria about (crowd intelligence) machine learning died a little after some of the competitions that got a lot of attention, like the netflix competition. Most of the solutions of big competitions with big money turned out to be useless in practice, being it due to complexity (as in netflix) or overfitting to the competition setting, including leakage, leaderboard feedback or poor generalization to outside data for whatever reasons. Im pretty sure responsible people in the big companies nowadays know about this risk and the pricemoney is adjusted accordingly. With 30k Bosch has nothing to lose and can just see what happens.\r\n", "137918": "Maybe if they threw in a free dishwasher to sweeten the pot ...", "138285": "[quote=aldente;138276]\r\n\r\n(1) In real world you should be in favor of \"false positive\" than \"false negative\". \r\n \r\n(2) In this competition, participants walk in different path than in business, for the sake of higher LB's MCC score. \r\n\r\nAm I missing something?\r\n\r\n[/quote]\r\n\r\nI think you may be misunderstanding how MCC is calculated and for what purposes. MCC is a very legitimate score for imbalanced datasets such as this one.\r\n\r\nLet's try with an example. Below is a line with real sequence of events, followed by a line that represents our prediction:\r\n\r\n 0000000100\r\n 0000000001\r\n\r\nThe prediction has one 1 FN and 1 FP. Its accuracy is 80% and MCC is -0.111\r\n\r\nNext:\r\n\r\n 1010101010\r\n 1010101001\r\n\r\nThis prediction also has 1 FN and 1 FP and its accuracy is 80%, but its MCC is 0.6. This also happens to be a much more balanced dataset.\r\n\r\nNext:\r\n\r\n 0000000100\r\n 0000000000\r\n\r\nNow we have \"fixed\" 1 FP and the accuracy here is 90%, yet MCC will go up only marginally to 0.0 compared to the first example.\r\n\r\nNext:\r\n\r\n 0000000100\r\n 0000000101\r\n\r\nNow we have \"fixed\" 1 FN from the first example. Again the accuracy is 90% but MCC will jump to 0.666, because we have fixed the underrepresented class.\r\n\r\nMCC will penalize the errors in a less abundant class much more, which I think is appropriate for Bosch's business purposes.", "138310": "[quote=aldente;138292]\r\n\r\nWait a minute..,, maybe because of the number of 1 is far less than 0? Or am I missing something again? \r\n\r\n[/quote]\r\n\r\nYes, it is because there are 171 times more 0s than 1s in the dataset. But beyond that, it may help you if you think about the numerator part of MCC:\r\n\r\n(TP*TN) - (FP*FN)\r\n\r\nSo when will this expression have maximal value? One obvious answer is when the (FP*FN) is small or zero. The not so obvious answer is when TP=TN (i.e., when classes are perfectly balanced), because that will maximize (TP*TN). If you have 100 data points with a perfect balance of two classes, TP*TN=2500. If you introduce even the slightest imbalance (49 vs 51), the product will be 2499, and it will keep decreasing as the imbalance grows. That is why MCC is one of better scores to use for imbalanced datasets.\r\n", "138307": "[quote=aldente;138306]\r\n\r\n@Taffey Lewis, I'm not sure I understand it correctly. Is it because of changes in number of FP/TP will also change number of FN/TN? \r\n\r\n[/quote]\r\n\r\nYes.", "137971": "Expectations of Bosch: *\"In this competition, Bosch is challenging Kagglers to predict internal failures using thousands of measurements and tests made for each component along the assembly line. This would enable Bosch to bring quality products at lower costs to the end user.\"*\r\n\r\nMany ML methods enable to reduce the number of features needed to obtain good results. Each measurement costs something, so reduction of ~3K features (measurements) to 200-500 may give not only a better prediction of internal failures but also additional reduction of costs - exactly what they want or even a little bit more. If there is no leak, it is a really good competition.", "137915": "Hi, I've begun work on this and I'm just wondering why such a prestigious company, with probably very deep pockets, is offering such a low reward for what seems to me extremely hard work? \r\n\r\nThere are many competitions on Kaggle that offer 15K +(plus potential job) to the top team for, say, 1/4 of the work of this contest. I get that folks will be competing for the prestige and some just for fun, but what I'm at a loss here, is what Bosch expects to get out of this? I mean the dataset is extremely large in terms of rows, and the number of features is roughly > 3K! Additionally, there is a date/time component which further adds to the complexity of the solution.\r\n\r\nDon't get me wrong, I don't mind the challenge, but I'm pressed for time and imo the competition reward should correlate with the opportunity cost. Do others feel my concern on this, or is this problem easier than I think? \r\n\r\nThanks and good luck!\r\n\r\n\r\n\r\n\r\n\r\n\r\n", "138417": "The relationship between reward and the inputs of time and effort is highly non-linear and a bit random. But I assure you that in many endeavors, you can get quite far in the long run through continued, intense, and varied effort. The time horizon should not be a single contest, but your entire lifetime.\r\n\r\nYou are competing against people with a negative or zero disutility of effort with respect to Kaggle and predictive modeling. Such people often pay to play - either directly via lost income or indirectly via time. Given such competition, it is difficult to win except in small data contests where the final placement is often mostly a lottery. ", "137984": "I totally get your point @wordsmith.\r\n\r\nAnswering your last question: yes it gets easier by time since you get more experience tackling problems, sometimes you find the problem similar to that of previous competitions, you have more code to reuse, you become better at using the tools (R, Python, other frameworks/tools). Also some of the top Kagglers managed to secure strong computing infrastructure to speed up their work.\r\n\r\nFinally i think we don't do that for the money (though it is a strong motivation), there are many other reasons such as:\r\n\r\n1) Growing & Learning (specially from other Kagglers)