I've built a binary classifier from the outlier set in the training data. It predicts outliers with accuracy of 0.82. It is not high accuracy, but it might be OK for estimating an order of magnitude of possible outliers in the test data. It turns out that that this classifier predicts about 10x of outliers in the test set vs. the outliers in the train set. I am not sure if low accuracy of my simple classifier would account for difference of probability of an outlier in the training set (0.01) and in the test set (0.24). Or would it?
\n\nIf it is remotely correct and probability of an outlier in the test set is 24 times the probability in the training set, then it can explain poor performance of models that I build. I noticed that as my CV score goes down, the LB score stays put or goes backwards. By looking at the histogram of predicted values, I see that my models do not predict outliers at all. Perhaps I do not have right features or my models do not fit the outliers, but combination of that with the higher probability of outliers in the test set might explain the LB score behavior.
\n\nAny comments?
", "messages": [ { "id": 460501, "postDate": "2019-01-23T21:17:38.703Z", "content": "I've built a binary classifier from the outlier set in the training data. It predicts outliers with accuracy of 0.82. It is not high accuracy, but it might be OK for estimating an order of magnitude of possible outliers in the test data. It turns out that that this classifier predicts about 10x of outliers in the test set vs. the outliers in the train set. I am not sure if low accuracy of my simple classifier would account for difference of probability of an outlier in the training set (0.01) and in the test set (0.24). Or would it?
\n\nIf it is remotely correct and probability of an outlier in the test set is 24 times the probability in the training set, then it can explain poor performance of models that I build. I noticed that as my CV score goes down, the LB score stays put or goes backwards. By looking at the histogram of predicted values, I see that my models do not predict outliers at all. Perhaps I do not have right features or my models do not fit the outliers, but combination of that with the higher probability of outliers in the test set might explain the LB score behavior.
\n\nAny comments?
", "rawMarkdown": "I've built a binary classifier from the outlier set in the training data. It predicts outliers with accuracy of 0.82. It is not high accuracy, but it might be OK for estimating an order of magnitude of possible outliers in the test data. It turns out that that this classifier predicts about 10x of outliers in the test set vs. the outliers in the train set. I am not sure if low accuracy of my simple classifier would account for difference of probability of an outlier in the training set (0.01) and in the test set (0.24). Or would it?\n\nIf it is remotely correct and probability of an outlier in the test set is 24 times the probability in the training set, then it can explain poor performance of models that I build. I noticed that as my CV score goes down, the LB score stays put or goes backwards. By looking at the histogram of predicted values, I see that my models do not predict outliers at all. Perhaps I do not have right features or my models do not fit the outliers, but combination of that with the higher probability of outliers in the test set might explain the LB score behavior. \n\nAny comments?", "votes": 5 }, { "id": 460842, "postDate": "2019-01-24T14:23:11.987Z", "content": "Some stats and approximation gives us 1420 +/- 1% of outliers in the test set.
\n\nApproach:\nWithout outliers, the target has a normal distribution.\n0 target vs real target in train set gives us 3.87 RMSE (0.01093)\n0 target in test set gives us 3.952 RMSE
\n\nTo have the same RMSE as in test set we have to have more outliers in train set (0.011513)
\n\nlen(test)*0.011513 = possible number of outliers in the test set (in case public-private LB were divided by the number of outliers also)
", "rawMarkdown": "Some stats and approximation gives us 1420 +/- 1% of outliers in the test set.\n\nApproach:\nWithout outliers, the target has a normal distribution.\n0 target vs real target in train set gives us 3.87 RMSE (0.01093)\n0 target in test set gives us 3.952 RMSE\n\nTo have the same RMSE as in test set we have to have more outliers in train set (0.011513)\n\nlen(test)*0.011513 = possible number of outliers in the test set (in case public-private LB were divided by the number of outliers also)", "votes": 3, "replies": [ { "id": 461078, "postDate": "2019-01-25T06:58:35.137Z", "content": "Maybe you are right, your calculation base on one assumption that the distribution between train data and lb test data should be same. thank you sir.
", "rawMarkdown": "Maybe you are right, your calculation base on one assumption that the distribution between train data and lb test data should be same. thank you sir." } ] }, { "id": 460908, "postDate": "2019-01-24T17:04:44.653Z", "content": "@Konstantin Yakovlev\nMy assumption was along the same lines before I did my classification. I assumed that test samples and train samples were randomly drawn from the same population, and then you estimation would be correct. What I saw in classification made me ask my question on the first place. \nIf estimation based on my classification is remotely correct, then assumption about the same population distribution in the train and test sets may not hold. And that might open a big can of worms IMHO
", "rawMarkdown": "@Konstantin Yakovlev\nMy assumption was along the same lines before I did my classification. I assumed that test samples and train samples were randomly drawn from the same population, and then you estimation would be correct. What I saw in classification made me ask my question on the first place. \nIf estimation based on my classification is remotely correct, then assumption about the same population distribution in the train and test sets may not hold. And that might open a big can of worms IMHO ", "votes": 1, "replies": [ { "id": 461081, "postDate": "2019-01-25T07:03:42.750Z", "content": "Good analysis.
", "rawMarkdown": "Good analysis." } ] }, { "id": 460903, "postDate": "2019-01-24T16:49:44.253Z", "content": "@Frank Puk RE:outlier definition\nBy looking and histogram of the target, I noticed that there are two clusters of values separated by a large gap. The separation starts at -18 target value, so
\n\ntrn.target.loc[trn.target > -18].describe()
\n\ncount 199710.000000\nmean -0.030879\nstd 1.717793\nmin -17.608147\n25% -0.844513\n50% -0.007783\n75% 0.777331\nmax 17.965068\nName: target, dtype: float64\ntrn.target.loc[trn.target < -18].describe()
\n\ncount 2.207000e+03\nmean -3.321928e+01\nstd 1.193982e-12\nmin -3.321928e+01\n25% -3.321928e+01\n50% -3.321928e+01\n75% -3.321928e+01\nmax -3.321928e+01\nName: target, dtype: float64\n199710 of target values are more than -17.608147 \nNo values between -3.321928e+01 and -17.608147 \n 2.207 of target values are below -3.321928e+01
\n\nHence the values below -3.321928e+01 are outliers
", "rawMarkdown": "@Frank Puk RE:outlier definition\nBy looking and histogram of the target, I noticed that there are two clusters of values separated by a large gap. The separation starts at -18 target value, so\n\n**trn.target.loc[trn.target > -18].describe()**\n\n count 199710.000000\n mean -0.030879\n std 1.717793\n min -17.608147\n 25% -0.844513\n 50% -0.007783\n 75% 0.777331\n max 17.965068\n Name: target, dtype: float64\n\n**trn.target.loc[trn.target < -18].describe()**\n\n count 2.207000e+03\n mean -3.321928e+01\n std 1.193982e-12\n min -3.321928e+01\n 25% -3.321928e+01\n 50% -3.321928e+01\n 75% -3.321928e+01\n max -3.321928e+01\n Name: target, dtype: float64\n\n199710 of target values are more than -17.608147 \nNo values between -3.321928e+01 and -17.608147 \n 2.207 of target values are below -3.321928e+01\n\nHence the values below -3.321928e+01 are outliers \n", "votes": 1 }, { "id": 460576, "postDate": "2019-01-24T02:20:29.137Z", "content": "I am assuming that you identified observations with target less than -33 as outliers. Simply looking at the accuracy is misleading because the proportion of outliers is as low as 0.01 in the training set and one can predict all 0's and get an accuracy ~0.99 . I am also considering that outlier prediction may be of great importance in this competition as they contribute a lot within the RMSE.
", "rawMarkdown": "I am assuming that you identified observations with target less than -33 as outliers. Simply looking at the accuracy is misleading because the proportion of outliers is as low as 0.01 in the training set and one can predict all 0's and get an accuracy ~0.99 . I am also considering that outlier prediction may be of great importance in this competition as they contribute a lot within the RMSE.", "votes": 1, "replies": [ { "id": 461077, "postDate": "2019-01-25T06:57:50.150Z", "content": "You are right,guy, we should do more to fit the outliers. we can use accuracy and recall as the metric.
", "rawMarkdown": "You are right,guy, we should do more to fit the outliers. we can use accuracy and recall as the metric." } ] }, { "id": 461716, "postDate": "2019-01-26T20:13:55.320Z", "content": "@Clifford Lai
\n\nIt looks like my model FPR is higher than 0.1. I guess I should build a classification model with less of the FPR\nThank you for the input!
", "rawMarkdown": "@Clifford Lai\n\nIt looks like my model FPR is higher than 0.1. I guess I should build a classification model with less of the FPR\nThank you for the input!" }, { "id": 460906, "postDate": "2019-01-24T16:55:12.737Z", "content": "@Clifford Lai\nI trained my classifier on a balanced set by sampling the non-outlier population to the equal amount of observations to number of outliers
\n\nnegative = train.loc[train.outliers == 0]\npositive = train.loc[train.outliers == 1]\nbalanced = pd.concat([negative.sample(n=positive.shape[0]), positive])\nbalanced_shaffled = balanced.sample(frac=1)\ny = balanced_shaffled.outliers.copy()\nX = balanced_shaffled.loc[:,selected_features].copy() \nThe result is interesting. So you obtained an average accuracy 0.82 from this bunch of balanced models?
\n\nIf that is the case, let' assume
\n\nSince the accuracy Pr(Yhat=Y)=0.82, then a+(1-b)=2*0.82=1.64 with equal priors for Y from the balanced case of the training set.
\n\nIn the test set, let
\n\nThus, M=M1+M2, M1~binomial(n1,a), M2~binomial(n2,b). Using a normal approximation, we can obtain M~N(n1* a+n2* b, n1* a* (1-a)+n2* b* (1-b)) and hence we can compute Pr(M>=m).
\n\nFor example, the following R code computes Pr(M>= 13512), the probability of detecting 13512=nr* 10 outliers in the test set if I set the FPR=b=0.1.
\n\n```\nn <- 123623\nr <- 0.01093023 \nn1 <- round(n*r) #1351\nn2 <- n-n1 #122272\nb <- 0.1 # Set FPR\na <- 1.64-(1-b) #TPR\nm <- round(n*r*10) #13512\nmean_M <- n1*a+n2*b \nsd_M <- sqrt(n1*a*(1-a)+n2*b*(1-b))
\n\n1-pnorm(m,mean = mean_M,sd = sd_M) #Pr(M>= m)\n```
\n\nIt tells us that under the hypothesis that the test set has the same proportion of outliers as the training set, we will have the following likelihoods,
\n\nSo the FPR is really important. Could you check the FPR of your fitted model? If your FPR>0.1, then it is not a rare event at all to detect such a large number of outliers in the test set.
", "rawMarkdown": "The result is interesting. So you obtained an average accuracy 0.82 from this bunch of balanced models? \n\nIf that is the case, let' assume\n\n- __a__=Pr(Yhat=1|Y=1), True positive rate (TPR)\n- __b__ =Pr(Yhat=1|Y=0), False positive rate (FPR)\n\nSince the accuracy Pr(Yhat=Y)=0.82, then a+(1-b)=2*0.82=1.64 with equal priors for Y from the balanced case of the training set.\n\nIn the test set, let\n\n- __n__: the test set size, 123623.\n- __r__: the outliers proportion, which may be set as the same ratio in the training set, 0.01093023 \n- n1=nr: the number of outliers\n- n2=n(1-r): the number of non-outliers\n- M: the total number of outliers detected by the classifier\n- M1: the true positive numbers\n- M2: the false positive numbers\n\nThus, M=M1+M2, M1~binomial(n1,a), M2~binomial(n2,b). Using a normal approximation, we can obtain M~N(n1* a+n2* b, n1* a* (1-a)+n2* b* (1-b)) and hence we can compute Pr(M>=m). \n\nFor example, the following R code computes Pr(M>= 13512), the probability of detecting 13512=nr* 10 outliers in the test set if I set the FPR=b=0.1.\n\n```\nn <- 123623\nr <- 0.01093023 \nn1 <- round(n*r) #1351\nn2 <- n-n1 #122272\nb <- 0.1 # Set FPR\na <- 1.64-(1-b) #TPR\nm <- round(n*r*10) #13512\nmean_M <- n1*a+n2*b \nsd_M <- sqrt(n1*a*(1-a)+n2*b*(1-b)) \n\n1-pnorm(m,mean = mean_M,sd = sd_M) #Pr(M>= m)\n```\n\nIt tells us that under the hypothesis that the test set has the same proportion of outliers as the training set, we will have the following likelihoods,\n\n- Pr(M>= 13512)=0 when FPR=b=0.05\n- Pr(M>= 13512)=0.0036 when FPR=b=0.1\n- Pr(M>= 13512)=0.362 when FPR=b=0.102\n- Pr(M>= 13512)=0.999 when FPR=b=0.105\n- Pr(M>= 13512)=1.000 when FPR=b=0.11\n\nSo the FPR is really important. Could you check the FPR of your fitted model? If your FPR>0.1, then it is not a rare event at all to detect such a large number of outliers in the test set.\n\n" } ] }, { "id": 460772, "postDate": "2019-01-24T11:44:15.850Z", "content": "May I know how you define what an outlier here means? Anything outside 3SD range from your dataset? Or just -33 target values?
", "rawMarkdown": "May I know how you define what an outlier here means? Anything outside 3SD range from your dataset? Or just -33 target values?", "replies": [ { "id": 461545, "postDate": "2019-01-26T10:03:42.843Z", "content": "@pukkinming outliers are mostly referring to just the -33 when talking about targets in this competition :)
", "rawMarkdown": "@pukkinming outliers are mostly referring to just the -33 when talking about targets in this competition :)", "votes": 1 } ] }, { "id": 463322, "postDate": "2019-01-29T19:58:25.583Z", "rawMarkdown": "", "isDeleted": true }, { "id": 463307, "postDate": "2019-01-29T19:27:15.830Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 460842, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2019-01-24T14:23:11.987000", "content": "Some stats and approximation gives us 1420 +/- 1% of outliers in the test set.
\n\nApproach:\nWithout outliers, the target has a normal distribution.\n0 target vs real target in train set gives us 3.87 RMSE (0.01093)\n0 target in test set gives us 3.952 RMSE
\n\nTo have the same RMSE as in test set we have to have more outliers in train set (0.011513)
\n\nlen(test)*0.011513 = possible number of outliers in the test set (in case public-private LB were divided by the number of outliers also)
", "votes": 3, "replies": [ { "id": 461078, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2019-01-25T06:58:35.137000", "content": "Maybe you are right, your calculation base on one assumption that the distribution between train data and lb test data should be same. thank you sir.
", "votes": 0, "replies": [] } ] }, { "id": 460908, "author_name": "sergeman", "author_url": "", "post_date": "2019-01-24T17:04:44.653000", "content": "@Konstantin Yakovlev\nMy assumption was along the same lines before I did my classification. I assumed that test samples and train samples were randomly drawn from the same population, and then you estimation would be correct. What I saw in classification made me ask my question on the first place. \nIf estimation based on my classification is remotely correct, then assumption about the same population distribution in the train and test sets may not hold. And that might open a big can of worms IMHO
", "votes": 1, "replies": [ { "id": 461081, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2019-01-25T07:03:42.750000", "content": "Good analysis.
", "votes": 0, "replies": [] } ] }, { "id": 460903, "author_name": "sergeman", "author_url": "", "post_date": "2019-01-24T16:49:44.253000", "content": "@Frank Puk RE:outlier definition\nBy looking and histogram of the target, I noticed that there are two clusters of values separated by a large gap. The separation starts at -18 target value, so
\n\ntrn.target.loc[trn.target > -18].describe()
\n\ncount 199710.000000\nmean -0.030879\nstd 1.717793\nmin -17.608147\n25% -0.844513\n50% -0.007783\n75% 0.777331\nmax 17.965068\nName: target, dtype: float64\ntrn.target.loc[trn.target < -18].describe()
\n\ncount 2.207000e+03\nmean -3.321928e+01\nstd 1.193982e-12\nmin -3.321928e+01\n25% -3.321928e+01\n50% -3.321928e+01\n75% -3.321928e+01\nmax -3.321928e+01\nName: target, dtype: float64\n199710 of target values are more than -17.608147 \nNo values between -3.321928e+01 and -17.608147 \n 2.207 of target values are below -3.321928e+01
\n\nHence the values below -3.321928e+01 are outliers
", "votes": 1, "replies": [] }, { "id": 460576, "author_name": "Clifford Lai", "author_url": "", "post_date": "2019-01-24T02:20:29.137000", "content": "I am assuming that you identified observations with target less than -33 as outliers. Simply looking at the accuracy is misleading because the proportion of outliers is as low as 0.01 in the training set and one can predict all 0's and get an accuracy ~0.99 . I am also considering that outlier prediction may be of great importance in this competition as they contribute a lot within the RMSE.
", "votes": 1, "replies": [ { "id": 461077, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2019-01-25T06:57:50.150000", "content": "You are right,guy, we should do more to fit the outliers. we can use accuracy and recall as the metric.
", "votes": 0, "replies": [] } ] }, { "id": 461716, "author_name": "sergeman", "author_url": "", "post_date": "2019-01-26T20:13:55.320000", "content": "@Clifford Lai
\n\nIt looks like my model FPR is higher than 0.1. I guess I should build a classification model with less of the FPR\nThank you for the input!
", "votes": 0, "replies": [] }, { "id": 460906, "author_name": "sergeman", "author_url": "", "post_date": "2019-01-24T16:55:12.737000", "content": "@Clifford Lai\nI trained my classifier on a balanced set by sampling the non-outlier population to the equal amount of observations to number of outliers
\n\nnegative = train.loc[train.outliers == 0]\npositive = train.loc[train.outliers == 1]\nbalanced = pd.concat([negative.sample(n=positive.shape[0]), positive])\nbalanced_shaffled = balanced.sample(frac=1)\ny = balanced_shaffled.outliers.copy()\nX = balanced_shaffled.loc[:,selected_features].copy() \nThe result is interesting. So you obtained an average accuracy 0.82 from this bunch of balanced models?
\n\nIf that is the case, let' assume
\n\nSince the accuracy Pr(Yhat=Y)=0.82, then a+(1-b)=2*0.82=1.64 with equal priors for Y from the balanced case of the training set.
\n\nIn the test set, let
\n\nThus, M=M1+M2, M1~binomial(n1,a), M2~binomial(n2,b). Using a normal approximation, we can obtain M~N(n1* a+n2* b, n1* a* (1-a)+n2* b* (1-b)) and hence we can compute Pr(M>=m).
\n\nFor example, the following R code computes Pr(M>= 13512), the probability of detecting 13512=nr* 10 outliers in the test set if I set the FPR=b=0.1.
\n\n```\nn <- 123623\nr <- 0.01093023 \nn1 <- round(n*r) #1351\nn2 <- n-n1 #122272\nb <- 0.1 # Set FPR\na <- 1.64-(1-b) #TPR\nm <- round(n*r*10) #13512\nmean_M <- n1*a+n2*b \nsd_M <- sqrt(n1*a*(1-a)+n2*b*(1-b))
\n\n1-pnorm(m,mean = mean_M,sd = sd_M) #Pr(M>= m)\n```
\n\nIt tells us that under the hypothesis that the test set has the same proportion of outliers as the training set, we will have the following likelihoods,
\n\nSo the FPR is really important. Could you check the FPR of your fitted model? If your FPR>0.1, then it is not a rare event at all to detect such a large number of outliers in the test set.
", "votes": 0, "replies": [] } ] }, { "id": 460772, "author_name": "FP", "author_url": "", "post_date": "2019-01-24T11:44:15.850000", "content": "May I know how you define what an outlier here means? Anything outside 3SD range from your dataset? Or just -33 target values?
", "votes": 0, "replies": [ { "id": 461545, "author_name": "HyperSeedOptimized", "author_url": "", "post_date": "2019-01-26T10:03:42.843000", "content": "@pukkinming outliers are mostly referring to just the -33 when talking about targets in this competition :)
", "votes": 1, "replies": [] } ] }, { "id": 463322, "author_name": "", "author_url": "", "post_date": "2019-01-29T19:58:25.583000", "content": "", "votes": 0, "replies": [] }, { "id": 463307, "author_name": "", "author_url": "", "post_date": "2019-01-29T19:27:15.830000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "460501": "I've built a binary classifier from the outlier set in the training data. It predicts outliers with accuracy of 0.82. It is not high accuracy, but it might be OK for estimating an order of magnitude of possible outliers in the test data. It turns out that that this classifier predicts about 10x of outliers in the test set vs. the outliers in the train set. I am not sure if low accuracy of my simple classifier would account for difference of probability of an outlier in the training set (0.01) and in the test set (0.24). Or would it?\n\nIf it is remotely correct and probability of an outlier in the test set is 24 times the probability in the training set, then it can explain poor performance of models that I build. I noticed that as my CV score goes down, the LB score stays put or goes backwards. By looking at the histogram of predicted values, I see that my models do not predict outliers at all. Perhaps I do not have right features or my models do not fit the outliers, but combination of that with the higher probability of outliers in the test set might explain the LB score behavior. \n\nAny comments?", "460842": "Some stats and approximation gives us 1420 +/- 1% of outliers in the test set.\n\nApproach:\nWithout outliers, the target has a normal distribution.\n0 target vs real target in train set gives us 3.87 RMSE (0.01093)\n0 target in test set gives us 3.952 RMSE\n\nTo have the same RMSE as in test set we have to have more outliers in train set (0.011513)\n\nlen(test)*0.011513 = possible number of outliers in the test set (in case public-private LB were divided by the number of outliers also)", "460908": "@Konstantin Yakovlev\nMy assumption was along the same lines before I did my classification. I assumed that test samples and train samples were randomly drawn from the same population, and then you estimation would be correct. What I saw in classification made me ask my question on the first place. \nIf estimation based on my classification is remotely correct, then assumption about the same population distribution in the train and test sets may not hold. And that might open a big can of worms IMHO ", "460903": "@Frank Puk RE:outlier definition\nBy looking and histogram of the target, I noticed that there are two clusters of values separated by a large gap. The separation starts at -18 target value, so\n\n**trn.target.loc[trn.target > -18].describe()**\n\n count 199710.000000\n mean -0.030879\n std 1.717793\n min -17.608147\n 25% -0.844513\n 50% -0.007783\n 75% 0.777331\n max 17.965068\n Name: target, dtype: float64\n\n**trn.target.loc[trn.target < -18].describe()**\n\n count 2.207000e+03\n mean -3.321928e+01\n std 1.193982e-12\n min -3.321928e+01\n 25% -3.321928e+01\n 50% -3.321928e+01\n 75% -3.321928e+01\n max -3.321928e+01\n Name: target, dtype: float64\n\n199710 of target values are more than -17.608147 \nNo values between -3.321928e+01 and -17.608147 \n 2.207 of target values are below -3.321928e+01\n\nHence the values below -3.321928e+01 are outliers \n", "460576": "I am assuming that you identified observations with target less than -33 as outliers. Simply looking at the accuracy is misleading because the proportion of outliers is as low as 0.01 in the training set and one can predict all 0's and get an accuracy ~0.99 . I am also considering that outlier prediction may be of great importance in this competition as they contribute a lot within the RMSE.", "461716": "@Clifford Lai\n\nIt looks like my model FPR is higher than 0.1. I guess I should build a classification model with less of the FPR\nThank you for the input!", "460906": "@Clifford Lai\nI trained my classifier on a balanced set by sampling the non-outlier population to the equal amount of observations to number of outliers\n\n negative = train.loc[train.outliers == 0]\n positive = train.loc[train.outliers == 1]\n balanced = pd.concat([negative.sample(n=positive.shape[0]), positive])\n balanced_shaffled = balanced.sample(frac=1)\n y = balanced_shaffled.outliers.copy()\n X = balanced_shaffled.loc[:,selected_features].copy() ", "460772": "May I know how you define what an outlier here means? Anything outside 3SD range from your dataset? Or just -33 target values?", "463322": "", "463307": "" } }