Hi all,\n I see many top kagglers say trust cv, so is it possible that the competition organizer specially design the public lb test data (which are actually not random sampled from all test data), right? Any idea is appreciated. Thank you! \nUpdated:\nOur team single model result is about 3.672 by@jiazhuang@salamatand our cv is about 3.647. \nOur another single model result is about 3.675 by@KimposiPungmudong and cv is about 3.642.
\n\nBy just weighted average this two result, we get about 3.666 public lb, actually this result has 3.599 private lb.
\n\nBut with manual setting of top outliers, we just set top120, we can get 3.644 public lb. but 3.618 in private lb.\ntop20 =-33\ntop20-80 = -33 * prob\ntop80-120 = -33 * 0.15 + 0.85 * predicted value
", "messages": [ { "id": "479128", "postDate": "02/27/2019 01:27:55", "content": "Hi all,\n I see many top kagglers say trust cv, so is it possible that the competition organizer specially design the public lb test data (which are actually not random sampled from all test data), right? Any idea is appreciated. Thank you! \nUpdated:\nOur team single model result is about 3.672 by@jiazhuang@salamatand our cv is about 3.647. \nOur another single model result is about 3.675 by@KimposiPungmudong and cv is about 3.642.
\n\nBy just weighted average this two result, we get about 3.666 public lb, actually this result has 3.599 private lb.
\n\nBut with manual setting of top outliers, we just set top120, we can get 3.644 public lb. but 3.618 in private lb.\ntop20 =-33\ntop20-80 = -33 * prob\ntop80-120 = -33 * 0.15 + 0.85 * predicted value
", "rawMarkdown": "Hi all,\n I see many top kagglers say trust cv, so is it possible that the competition organizer specially design the public lb test data (which are actually not random sampled from all test data), right? Any idea is appreciated. Thank you! \nUpdated:\nOur team single model result is about 3.672 by@[jiazhuang][1]@[salamat][2]and our cv is about 3.647. \nOur another single model result is about 3.675 by@[KimposiPungmudong][3] and cv is about 3.642.\n\nBy just weighted average this two result, we get about 3.666 public lb, actually this result has 3.599 private lb.\n\nBut with manual setting of top outliers, we just set top120, we can get 3.644 public lb. but 3.618 in private lb.\ntop20 =-33\ntop20-80 = -33 * prob\ntop80-120 = -33 * 0.15 + 0.85 * predicted value\n [1]: https://www.kaggle.com/jiazhuang\n [2]: https://www.kaggle.com/salamatburj\n [3]: https://www.kaggle.com/palend", "votes": null }, { "id": "479154", "postDate": "02/27/2019 01:48:56", "content": "I also have this question. Public LB(30%) seems have more outliers than the rest, which makes the PLB misleading.
", "rawMarkdown": "I also have this question. Public LB(30%) seems have more outliers than the rest, which makes the PLB misleading.", "votes": null }, { "id": "479177", "postDate": "02/27/2019 02:13:06", "content": "Based on my experience of a previous competition, feature engineering will always be effective in LB and CV, so I spend a lot of time doing this，which improve CV form 3.649 to 3.644, but the LB dropped from 3.691 to 3.7. Now I know it really work which can improve final results form 3.616 to 3.612 in single model. However due to the poor performance on LB, it seriously affected my later research direction :(
", "rawMarkdown": "Based on my experience of a previous competition, feature engineering will always be effective in LB and CV, so I spend a lot of time doing this，which improve CV form 3.649 to 3.644, but the LB dropped from 3.691 to 3.7. Now I know it really work which can improve final results form 3.616 to 3.612 in single model. However due to the poor performance on LB, it seriously affected my later research direction :(", "votes": null }, { "id": "479179", "postDate": "02/27/2019 02:15:30", "content": "I tried many ideas but couldn't see LB improve a little bit, which made me extremely frustrated during the competition. The thinking that I couldn't beat public kernel(LB 3.691) made me almost give up. I guess feature engineering needs some patience.
", "rawMarkdown": "I tried many ideas but couldn't see LB improve a little bit, which made me extremely frustrated during the competition. The thinking that I couldn't beat public kernel(LB 3.691) made me almost give up. I guess feature engineering needs some patience.", "votes": null }, { "id": "479181", "postDate": "02/27/2019 02:17:56", "content": "Yes, so you mean lb 3.691 will get about 3.612 private lb? thank you\nour result public lb 3.666(just weight average two single model 3.671 and 3.676) get 3.599 private lb.
", "rawMarkdown": "Yes, so you mean lb 3.691 will get about 3.612 private lb? thank you\nour result public lb 3.666(just weight average two single model 3.671 and 3.676) get 3.599 private lb.", "votes": null }, { "id": "479182", "postDate": "02/27/2019 02:19:19", "content": "CV：6.405~6.410 LB : 3.682~3.683 privite socre : 3.608 ~ 3.610\nafter dealt with outliers:\nLB : 3.672 ~ 3.673 private score : 3.638 ~ 3.639
", "rawMarkdown": "CV：6.405~6.410 LB : 3.682~3.683 privite socre : 3.608 ~ 3.610\nafter dealt with outliers:\nLB : 3.672 ~ 3.673 private score : 3.638 ~ 3.639", "votes": null }, { "id": "479190", "postDate": "02/27/2019 02:24:40", "content": "Yes, at the final time of features engineering about(3.672 public lb), whatever features i add the lb will worse many.
", "rawMarkdown": "Yes, at the final time of features engineering about(3.672 public lb), whatever features i add the lb will worse many.", "votes": null }, { "id": "479249", "postDate": "02/27/2019 03:11:51", "content": "Yes FE is the key. Here is my post on feature selection techniques https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052
", "rawMarkdown": "Yes FE is the key. Here is my post on feature selection techniques https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052", "votes": null }, { "id": "479254", "postDate": "02/27/2019 03:13:34", "content": "thank you!
", "rawMarkdown": "thank you!", "votes": null }, { "id": "479280", "postDate": "02/27/2019 03:36:31", "content": "LB 3.691 get 3.616private lb. By doing feature engineering, I have more than one thousand features, lb 3.7 get 3.613private lb, a better feature selection can get 3.612 private lb. The low lb lead to I thought some features maybe too complicated . At that time I don't know it it work on private lb, the big decline in lb cheated me
", "rawMarkdown": "LB 3.691 get 3.616private lb. By doing feature engineering, I have more than one thousand features, lb 3.7 get 3.613private lb, a better feature selection can get 3.612 private lb. The low lb lead to I thought some features maybe too complicated . At that time I don't know it it work on private lb, the big decline in lb cheated me", "votes": null }, { "id": "482462", "postDate": "03/03/2019 02:48:07", "content": "Some of my thoughts about manual mark\nWhen the degree of dispersion of the target of the test is not greater than the dispersion of the target of the train, the manual mark does not bring any positive gain.The expected return of manual mark the sample with a two-category probability p can be approximated=p*(-33--33*p)^2+(1-p)*((-33*p)^2--33)^2)=(-33)^2*(1-p)*(p-1)<0，It is always less than 0.
", "rawMarkdown": "Some of my thoughts about manual mark\nWhen the degree of dispersion of the target of the test is not greater than the dispersion of the target of the train, the manual mark does not bring any positive gain.The expected return of manual mark the sample with a two-category probability p can be approximated=p*(-33--33*p)^2+(1-p)*((-33*p)^2--33)^2)=(-33)^2*(1-p)*(p-1)<0，It is always less than 0.", "votes": null }, { "id": "482610", "postDate": "03/03/2019 11:15:42", "content": "Thank you , the p mean the probability of outlier？ and can you have a explanation for your formula.
", "rawMarkdown": "Thank you , the p mean the probability of outlier？ and can you have a explanation for your formula.", "votes": null }, { "id": "482682", "postDate": "03/03/2019 14:08:16", "content": "Yes,the p mean the probability of outlier.\nThat formula is just a rough estimate.\nI made two assumptions to simplify the problem
\n\n1)The sample of outliers is predicted to be -33*p, If you use the ‘simple trick’ shared by 1st player, then this assumption is approximately true.The predicted value of the outlier sample will be close to -33*p
\n\n2)The target of the normal sample is 0,Although this assumption is not true, but we know that normal samples are symmetric with zero.So the expected loss predicted under the RMSE formula will only be greater the rough estimate.
\n\nSo I think that the expect loss of manual marking the sample to -33 will only be greater than what I rough estimate.
", "rawMarkdown": "Yes,the p mean the probability of outlier.\nThat formula is just a rough estimate.\nI made two assumptions to simplify the problem\n\n1)The sample of outliers is predicted to be -33*p, If you use the ‘simple trick’ shared by 1st player, then this assumption is approximately true.The predicted value of the outlier sample will be close to -33*p\n\n2)The target of the normal sample is 0,Although this assumption is not true, but we know that normal samples are symmetric with zero.So the expected loss predicted under the RMSE formula will only be greater the rough estimate.\n\nSo I think that the expect loss of manual marking the sample to -33 will only be greater than what I rough estimate.", "votes": null }, { "id": "483011", "postDate": "03/04/2019 01:44:03", "content": "Good explanation, i can understand p(-33--33p)^2, but what does '(1-p)((-33p)^2--33)^2)' mean? thank you!
", "rawMarkdown": "Good explanation, i can understand p(-33--33p)^2, but what does '(1-p)((-33p)^2--33)^2)' mean? thank you!", "votes": null }, { "id": "483133", "postDate": "03/04/2019 07:21:54", "content": "There is a typo, here should be(1-p)((-33p)^2-(-33)^2), \n ((-33p)^2-(-33)^2) means the loss when a normal sample is mislabeled as -33， (-33p - 0)^2 represent existing loss, (-33 - 0)^2 represent the loss when a normal sample's target is -33, The difference is the loss of the mislabeling. Multiply (1-p) is the expected loss\nOf course, this part of the estimate is based on my previous two assumptions, used in the case of larger p (such as p>0.3 or 0.4)
", "rawMarkdown": "There is a typo, here should be(1-p)((-33p)^2-(-33)^2), \n ((-33p)^2-(-33)^2) means the loss when a normal sample is mislabeled as -33， (-33p - 0)^2 represent existing loss, (-33 - 0)^2 represent the loss when a normal sample's target is -33, The difference is the loss of the mislabeling. Multiply (1-p) is the expected loss\nOf course, this part of the estimate is based on my previous two assumptions, used in the case of larger p (such as p>0.3 or 0.4)", "votes": null } ], "comments": [ { "id": 479154, "author_name": "jiazhuang", "author_url": "", "post_date": "02/27/2019 01:48:56", "content": "I also have this question. Public LB(30%) seems have more outliers than the rest, which makes the PLB misleading.
", "votes": null, "replies": [] }, { "id": 479177, "author_name": "guang10", "author_url": "", "post_date": "02/27/2019 02:13:06", "content": "Based on my experience of a previous competition, feature engineering will always be effective in LB and CV, so I spend a lot of time doing this，which improve CV form 3.649 to 3.644, but the LB dropped from 3.691 to 3.7. Now I know it really work which can improve final results form 3.616 to 3.612 in single model. However due to the poor performance on LB, it seriously affected my later research direction :(
", "votes": null, "replies": [ { "id": 479181, "author_name": "codlife", "author_url": "", "post_date": "02/27/2019 02:17:56", "content": "Yes, so you mean lb 3.691 will get about 3.612 private lb? thank you\nour result public lb 3.666(just weight average two single model 3.671 and 3.676) get 3.599 private lb.
", "votes": null, "replies": [] }, { "id": 479280, "author_name": "guang10", "author_url": "", "post_date": "02/27/2019 03:36:31", "content": "LB 3.691 get 3.616private lb. By doing feature engineering, I have more than one thousand features, lb 3.7 get 3.613private lb, a better feature selection can get 3.612 private lb. The low lb lead to I thought some features maybe too complicated . At that time I don't know it it work on private lb, the big decline in lb cheated me
", "votes": null, "replies": [] } ] }, { "id": 479179, "author_name": "fangao", "author_url": "", "post_date": "02/27/2019 02:15:30", "content": "I tried many ideas but couldn't see LB improve a little bit, which made me extremely frustrated during the competition. The thinking that I couldn't beat public kernel(LB 3.691) made me almost give up. I guess feature engineering needs some patience.
", "votes": null, "replies": [ { "id": 479190, "author_name": "codlife", "author_url": "", "post_date": "02/27/2019 02:24:40", "content": "Yes, at the final time of features engineering about(3.672 public lb), whatever features i add the lb will worse many.
", "votes": null, "replies": [] } ] }, { "id": 479182, "author_name": "fangwenjun", "author_url": "", "post_date": "02/27/2019 02:19:19", "content": "CV：6.405~6.410 LB : 3.682~3.683 privite socre : 3.608 ~ 3.610\nafter dealt with outliers:\nLB : 3.672 ~ 3.673 private score : 3.638 ~ 3.639
", "votes": null, "replies": [] }, { "id": 479249, "author_name": "sheriytm", "author_url": "", "post_date": "02/27/2019 03:11:51", "content": "Yes FE is the key. Here is my post on feature selection techniques https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052
", "votes": null, "replies": [ { "id": 479254, "author_name": "codlife", "author_url": "", "post_date": "02/27/2019 03:13:34", "content": "thank you!
", "votes": null, "replies": [] } ] }, { "id": 482462, "author_name": "hkyhky", "author_url": "", "post_date": "03/03/2019 02:48:07", "content": "Some of my thoughts about manual mark\nWhen the degree of dispersion of the target of the test is not greater than the dispersion of the target of the train, the manual mark does not bring any positive gain.The expected return of manual mark the sample with a two-category probability p can be approximated=p*(-33--33*p)^2+(1-p)*((-33*p)^2--33)^2)=(-33)^2*(1-p)*(p-1)<0，It is always less than 0.
", "votes": null, "replies": [ { "id": 482610, "author_name": "codlife", "author_url": "", "post_date": "03/03/2019 11:15:42", "content": "Thank you , the p mean the probability of outlier？ and can you have a explanation for your formula.
", "votes": null, "replies": [] }, { "id": 482682, "author_name": "hkyhky", "author_url": "", "post_date": "03/03/2019 14:08:16", "content": "Yes,the p mean the probability of outlier.\nThat formula is just a rough estimate.\nI made two assumptions to simplify the problem
\n\n1)The sample of outliers is predicted to be -33*p, If you use the ‘simple trick’ shared by 1st player, then this assumption is approximately true.The predicted value of the outlier sample will be close to -33*p
\n\n2)The target of the normal sample is 0,Although this assumption is not true, but we know that normal samples are symmetric with zero.So the expected loss predicted under the RMSE formula will only be greater the rough estimate.
\n\nSo I think that the expect loss of manual marking the sample to -33 will only be greater than what I rough estimate.
", "votes": null, "replies": [] }, { "id": 483011, "author_name": "codlife", "author_url": "", "post_date": "03/04/2019 01:44:03", "content": "Good explanation, i can understand p(-33--33p)^2, but what does '(1-p)((-33p)^2--33)^2)' mean? thank you!
", "votes": null, "replies": [] }, { "id": 483133, "author_name": "hkyhky", "author_url": "", "post_date": "03/04/2019 07:21:54", "content": "There is a typo, here should be(1-p)((-33p)^2-(-33)^2), \n ((-33p)^2-(-33)^2) means the loss when a normal sample is mislabeled as -33， (-33p - 0)^2 represent existing loss, (-33 - 0)^2 represent the loss when a normal sample's target is -33, The difference is the loss of the mislabeling. Multiply (1-p) is the expected loss\nOf course, this part of the estimate is based on my previous two assumptions, used in the case of larger p (such as p>0.3 or 0.4)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "479128": "Hi all,\n I see many top kagglers say trust cv, so is it possible that the competition organizer specially design the public lb test data (which are actually not random sampled from all test data), right? Any idea is appreciated. Thank you! \nUpdated:\nOur team single model result is about 3.672 by@[jiazhuang][1]@[salamat][2]and our cv is about 3.647. \nOur another single model result is about 3.675 by@[KimposiPungmudong][3] and cv is about 3.642.\n\nBy just weighted average this two result, we get about 3.666 public lb, actually this result has 3.599 private lb.\n\nBut with manual setting of top outliers, we just set top120, we can get 3.644 public lb. but 3.618 in private lb.\ntop20 =-33\ntop20-80 = -33 * prob\ntop80-120 = -33 * 0.15 + 0.85 * predicted value\n [1]: https://www.kaggle.com/jiazhuang\n [2]: https://www.kaggle.com/salamatburj\n [3]: https://www.kaggle.com/palend", "479154": "I also have this question. Public LB(30%) seems have more outliers than the rest, which makes the PLB misleading.", "479177": "Based on my experience of a previous competition, feature engineering will always be effective in LB and CV, so I spend a lot of time doing this，which improve CV form 3.649 to 3.644, but the LB dropped from 3.691 to 3.7. Now I know it really work which can improve final results form 3.616 to 3.612 in single model. However due to the poor performance on LB, it seriously affected my later research direction :(", "479179": "I tried many ideas but couldn't see LB improve a little bit, which made me extremely frustrated during the competition. The thinking that I couldn't beat public kernel(LB 3.691) made me almost give up. I guess feature engineering needs some patience.", "479181": "Yes, so you mean lb 3.691 will get about 3.612 private lb? thank you\nour result public lb 3.666(just weight average two single model 3.671 and 3.676) get 3.599 private lb.", "479182": "CV：6.405~6.410 LB : 3.682~3.683 privite socre : 3.608 ~ 3.610\nafter dealt with outliers:\nLB : 3.672 ~ 3.673 private score : 3.638 ~ 3.639", "479190": "Yes, at the final time of features engineering about(3.672 public lb), whatever features i add the lb will worse many.", "479249": "Yes FE is the key. Here is my post on feature selection techniques https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052", "479254": "thank you!", "479280": "LB 3.691 get 3.616private lb. By doing feature engineering, I have more than one thousand features, lb 3.7 get 3.613private lb, a better feature selection can get 3.612 private lb. The low lb lead to I thought some features maybe too complicated . At that time I don't know it it work on private lb, the big decline in lb cheated me", "482462": "Some of my thoughts about manual mark\nWhen the degree of dispersion of the target of the test is not greater than the dispersion of the target of the train, the manual mark does not bring any positive gain.The expected return of manual mark the sample with a two-category probability p can be approximated=p*(-33--33*p)^2+(1-p)*((-33*p)^2--33)^2)=(-33)^2*(1-p)*(p-1)<0，It is always less than 0.", "482610": "Thank you , the p mean the probability of outlier？ and can you have a explanation for your formula.", "482682": "Yes,the p mean the probability of outlier.\nThat formula is just a rough estimate.\nI made two assumptions to simplify the problem\n\n1)The sample of outliers is predicted to be -33*p, If you use the ‘simple trick’ shared by 1st player, then this assumption is approximately true.The predicted value of the outlier sample will be close to -33*p\n\n2)The target of the normal sample is 0,Although this assumption is not true, but we know that normal samples are symmetric with zero.So the expected loss predicted under the RMSE formula will only be greater the rough estimate.\n\nSo I think that the expect loss of manual marking the sample to -33 will only be greater than what I rough estimate.", "483011": "Good explanation, i can understand p(-33--33p)^2, but what does '(1-p)((-33p)^2--33)^2)' mean? thank you!", "483133": "There is a typo, here should be(1-p)((-33p)^2-(-33)^2), \n ((-33p)^2-(-33)^2) means the loss when a normal sample is mislabeled as -33， (-33p - 0)^2 represent existing loss, (-33 - 0)^2 represent the loss when a normal sample's target is -33, The difference is the loss of the mislabeling. Multiply (1-p) is the expected loss\nOf course, this part of the estimate is based on my previous two assumptions, used in the case of larger p (such as p>0.3 or 0.4)" }, "source": "meta" }