Dear All,\nI have noticed that the event rate in the train set is around 0.56%, however, all my models are generating event in the range of 0.25-0.35%\nIs there an event imbalance or i am missing something ..
", "messages": [ { "id": "137493", "postDate": "10/02/2016 03:43:20", "content": "Dear All,\nI have noticed that the event rate in the train set is around 0.56%, however, all my models are generating event in the range of 0.25-0.35%\nIs there an event imbalance or i am missing something ..
", "rawMarkdown": "Dear All,\r\nI have noticed that the event rate in the train set is around 0.56%, however, all my models are generating event in the range of 0.25-0.35%\r\nIs there an event imbalance or i am missing something ..", "votes": null }, { "id": "137499", "postDate": "10/02/2016 07:30:06", "content": "MCC (and possibly AUC in this case) is/are very sensitive to false positives, so the best results (outside of the .4xx club) have relatively few positives.
", "rawMarkdown": "MCC (and possibly AUC in this case) is/are very sensitive to false positives, so the best results (outside of the .4xx club) have relatively few positives.", "votes": null }, { "id": "137500", "postDate": "10/02/2016 07:34:51", "content": "Looked at my best submission, I have only 0.21% rate:) this competition its all about false positives.
", "rawMarkdown": "Looked at my best submission, I have only 0.21% rate:) this competition its all about false positives.", "votes": null }, { "id": "137511", "postDate": "10/02/2016 10:20:32", "content": "So does this mean that even if the event rate in the test set is around 0.5%, we can maximize our MCC score just by predicting 0.20% of labels correctly, rather than trying to increase our recall in which case we will encounter a lot of FP.
", "rawMarkdown": "So does this mean that even if the event rate in the test set is around 0.5%, we can maximize our MCC score just by predicting 0.20% of labels correctly, rather than trying to increase our recall in which case we will encounter a lot of FP.", "votes": null }, { "id": "137512", "postDate": "10/02/2016 11:04:25", "content": "[quote=Mohd Shadab Alam;137511]
\n\nSo does this mean that even if the event rate in the test set is around 0.5%, we can maximize our MCC score just by predicting 0.20% of labels correctly, rather than trying to increase our recall in which case we will encounter a lot of FP.
\n\n[/quote]
\n\nMCC maximization = Phi coefficient on the confusion matrix maximization = R-squared maximization
\n\nAssuming you have something like this as your confusion matrix:
\n\nTP FP\nFN TN\nYour two objectives to maximize the MCC is to, assuming FN>0 and FP>0 and FN*FP large enough to supersede the linear MCC effect:
\n\nGetting balanced results is penalizing yourself as a balanced FP*FN will be greater than a non-balanced FP*FN (e.g 1,000*1,000 > 500*1,500): you have a quadratic effect on the MCC that will supersede the linear effects as we don't have a small sample to compute the MCC.
\n\nIn short: you are better having FP = 0 or FN = 0 with FP or FN = |some number| than having FP = FN = |some number / 2| if you want to maximize MCC. Hence, if you don't know how to optimize one, you can try optimizing the other gracefully.
\n\nWith simple maths, if you fix FP+FN = |some number|, then max(MCC) is reached when either FP=0 or FN=0. Therefore, it is all about the minimization of either of these numbers considering |some number| "fixed" (a more "convoluted" way of what raddar said).
", "rawMarkdown": "[quote=Mohd Shadab Alam;137511]\r\n\r\nSo does this mean that even if the event rate in the test set is around 0.5%, we can maximize our MCC score just by predicting 0.20% of labels correctly, rather than trying to increase our recall in which case we will encounter a lot of FP.\r\n\r\n\r\n[/quote]\r\n\r\nMCC maximization = Phi coefficient on the confusion matrix maximization = R-squared maximization\r\n\r\nAssuming you have something like this as your confusion matrix:\r\n\r\n TP FP\r\n FN TN\r\n\r\nYour two objectives to maximize the MCC is to, assuming FN>0 and FP>0 and FN*FP large enough to supersede the linear MCC effect:\r\n\r\n* max(TP*TN) (maximum is having all samples labeled correctly)\r\n* min(FP*FN) (minimum is 0, from there you play fully with the linear effect)\r\n\r\nGetting balanced results is penalizing yourself as a balanced FP*FN will be greater than a non-balanced FP*FN (e.g 1,000*1,000 > 500*1,500): you have a quadratic effect on the MCC that will supersede the linear effects as we don't have a small sample to compute the MCC.\r\n\r\nIn short: you are better having FP = 0 or FN = 0 with FP or FN = |some number| than having FP = FN = |some number / 2| if you want to maximize MCC. Hence, if you don't know how to optimize one, you can try optimizing the other gracefully.\r\n\r\nWith simple maths, if you fix FP+FN = |some number|, then max(MCC) is reached when either FP=0 or FN=0. Therefore, it is all about the minimization of either of these numbers considering |some number| \"fixed\" (a more \"convoluted\" way of what raddar said).", "votes": null } ], "comments": [ { "id": 137499, "author_name": "happycube", "author_url": "", "post_date": "10/02/2016 07:30:06", "content": "MCC (and possibly AUC in this case) is/are very sensitive to false positives, so the best results (outside of the .4xx club) have relatively few positives.
", "votes": null, "replies": [] }, { "id": 137500, "author_name": "raddar", "author_url": "", "post_date": "10/02/2016 07:34:51", "content": "Looked at my best submission, I have only 0.21% rate:) this competition its all about false positives.
", "votes": null, "replies": [] }, { "id": 137511, "author_name": "outliar", "author_url": "", "post_date": "10/02/2016 10:20:32", "content": "So does this mean that even if the event rate in the test set is around 0.5%, we can maximize our MCC score just by predicting 0.20% of labels correctly, rather than trying to increase our recall in which case we will encounter a lot of FP.
", "votes": null, "replies": [] }, { "id": 137512, "author_name": "laurae2", "author_url": "", "post_date": "10/02/2016 11:04:25", "content": "[quote=Mohd Shadab Alam;137511]
\n\nSo does this mean that even if the event rate in the test set is around 0.5%, we can maximize our MCC score just by predicting 0.20% of labels correctly, rather than trying to increase our recall in which case we will encounter a lot of FP.
\n\n[/quote]
\n\nMCC maximization = Phi coefficient on the confusion matrix maximization = R-squared maximization
\n\nAssuming you have something like this as your confusion matrix:
\n\nTP FP\nFN TN\nYour two objectives to maximize the MCC is to, assuming FN>0 and FP>0 and FN*FP large enough to supersede the linear MCC effect:
\n\nGetting balanced results is penalizing yourself as a balanced FP*FN will be greater than a non-balanced FP*FN (e.g 1,000*1,000 > 500*1,500): you have a quadratic effect on the MCC that will supersede the linear effects as we don't have a small sample to compute the MCC.
\n\nIn short: you are better having FP = 0 or FN = 0 with FP or FN = |some number| than having FP = FN = |some number / 2| if you want to maximize MCC. Hence, if you don't know how to optimize one, you can try optimizing the other gracefully.
\n\nWith simple maths, if you fix FP+FN = |some number|, then max(MCC) is reached when either FP=0 or FN=0. Therefore, it is all about the minimization of either of these numbers considering |some number| "fixed" (a more "convoluted" way of what raddar said).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "137493": "Dear All,\r\nI have noticed that the event rate in the train set is around 0.56%, however, all my models are generating event in the range of 0.25-0.35%\r\nIs there an event imbalance or i am missing something ..", "137499": "MCC (and possibly AUC in this case) is/are very sensitive to false positives, so the best results (outside of the .4xx club) have relatively few positives.", "137500": "Looked at my best submission, I have only 0.21% rate:) this competition its all about false positives.", "137511": "So does this mean that even if the event rate in the test set is around 0.5%, we can maximize our MCC score just by predicting 0.20% of labels correctly, rather than trying to increase our recall in which case we will encounter a lot of FP.", "137512": "[quote=Mohd Shadab Alam;137511]\r\n\r\nSo does this mean that even if the event rate in the test set is around 0.5%, we can maximize our MCC score just by predicting 0.20% of labels correctly, rather than trying to increase our recall in which case we will encounter a lot of FP.\r\n\r\n\r\n[/quote]\r\n\r\nMCC maximization = Phi coefficient on the confusion matrix maximization = R-squared maximization\r\n\r\nAssuming you have something like this as your confusion matrix:\r\n\r\n TP FP\r\n FN TN\r\n\r\nYour two objectives to maximize the MCC is to, assuming FN>0 and FP>0 and FN*FP large enough to supersede the linear MCC effect:\r\n\r\n* max(TP*TN) (maximum is having all samples labeled correctly)\r\n* min(FP*FN) (minimum is 0, from there you play fully with the linear effect)\r\n\r\nGetting balanced results is penalizing yourself as a balanced FP*FN will be greater than a non-balanced FP*FN (e.g 1,000*1,000 > 500*1,500): you have a quadratic effect on the MCC that will supersede the linear effects as we don't have a small sample to compute the MCC.\r\n\r\nIn short: you are better having FP = 0 or FN = 0 with FP or FN = |some number| than having FP = FN = |some number / 2| if you want to maximize MCC. Hence, if you don't know how to optimize one, you can try optimizing the other gracefully.\r\n\r\nWith simple maths, if you fix FP+FN = |some number|, then max(MCC) is reached when either FP=0 or FN=0. Therefore, it is all about the minimization of either of these numbers considering |some number| \"fixed\" (a more \"convoluted\" way of what raddar said)." }, "source": "meta" }