{
"id": 95818,
"title": "Got -0.685 LB from OOF Log",
"url": "/competitions/champs-scalar-coupling/discussion/95818",
"author_name": "",
"post_date": "2019-06-15T12:59:38.510450Z",
"votes": 4,
"comment_count": 8,
"views": 0,
"content": "Many kernels are using the OOF feature generated by prediction FC feature of scalar_coupling_contributions file. I also used this ideia in my kernel and realized that actually increases the precision of the model. But, in addition, extracts another feature by taking the log of OOF Feature generated and a new feature was very interesting, increasing my last LB score from -0.55 to -0.685. Click here (Kernel LB -0.685)to make the kernel for anyone who wants to take a look. I wish to thank for all thooses kernels and works that made my kernel possible, especially the artgor outstanding kernels, link in reference secion.\n![\"Feature](\"https://www.kaggleusercontent.com/kf/15740540/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..XiRgUseuu1qjpioBStxJgA.w8g9Az6i4mEvacp0GjW1XjDzy0vnR4wHg5c9CnpOf09klBMkJ2vxsZ8fnJ7RYTjo5oMjifTzrSNwBKrUS53XxRulDY3DH1NlDIYJTiwfrMyrnwVheFV-xa0cUcxTYlFekk5_NUTgn_0eU_yVFBLvPxwACy2NiOWot43s4_iitVk.JWnTSD-8NU9srOUQQl9AkA/__results___files/__results___30_1.png\")
\n\nI've used few estimators, so it's possible to increase that score even more. This is not my Final Kernel, still improving and search of new Features.
",
"messages": [
{
"id": "553308",
"postDate": "06/15/2019 12:59:38",
"content": "Many kernels are using the OOF feature generated by prediction FC feature of scalar_coupling_contributions file. I also used this ideia in my kernel and realized that actually increases the precision of the model. But, in addition, extracts another feature by taking the log of OOF Feature generated and a new feature was very interesting, increasing my last LB score from -0.55 to -0.685. Click here (Kernel LB -0.685)to make the kernel for anyone who wants to take a look. I wish to thank for all thooses kernels and works that made my kernel possible, especially the artgor outstanding kernels, link in reference secion.\n
\n\nI've used few estimators, so it's possible to increase that score even more. This is not my Final Kernel, still improving and search of new Features.
",
"rawMarkdown": "Many kernels are using the OOF feature generated by prediction FC feature of `scalar_coupling_contributions` file. I also used this ideia in my kernel and realized that actually increases the precision of the model. But, in addition, extracts another feature by taking the log of OOF Feature generated and a new feature was very interesting, increasing my last LB score from -0.55 to -0.685. Click here ([Kernel LB -0.685](https://www.kaggle.com/adrianoavelar/molecule-pred-feature-eng-lb-0-55))to make the kernel for anyone who wants to take a look. I wish to thank for all thooses kernels and works that made my kernel possible, especially the artgor outstanding kernels, link in reference secion.\n\n\nI've used few estimators, so it's possible to increase that score even more. This is not my Final Kernel, still improving and search of new Features.",
"votes": null
},
{
"id": "553998",
"postDate": "06/16/2019 18:42:29",
"content": "Interesting! Have you tried predicting other features as well and use it as feature for the final model? I might be going over that in the upcoming days as well:)
",
"rawMarkdown": "Interesting! Have you tried predicting other features as well and use it as feature for the final model? I might be going over that in the upcoming days as well:)",
"votes": null
},
{
"id": "554001",
"postDate": "06/16/2019 18:49:29",
"content": "Any hypothesis on why taking the log of the features is helping the model a lot
",
"rawMarkdown": "Any hypothesis on why taking the log of the features is helping the model a lot",
"votes": null
},
{
"id": "554002",
"postDate": "06/16/2019 18:54:45",
"content": "Took this straight from stackexchange:
\n\nStatistically, if your variables are right-skew (that is, they have a long tail at the high end) then a measure such as correlation or regression can be influenced a lot by one or a few cases at the high end on one or both variables (outliers, leverage points, influential points). Taking the log can help this by reducing or eliminating skew.
",
"rawMarkdown": "Took this straight from stackexchange: \n\nStatistically, if your variables are right-skew (that is, they have a long tail at the high end) then a measure such as correlation or regression can be influenced a lot by one or a few cases at the high end on one or both variables (outliers, leverage points, influential points). Taking the log can help this by reducing or eliminating skew.",
"votes": null
},
{
"id": "554009",
"postDate": "06/16/2019 19:32:44",
"content": "Thanks a lot for the explanation
",
"rawMarkdown": "Thanks a lot for the explanation",
"votes": null
},
{
"id": "554609",
"postDate": "06/17/2019 18:21:17",
"content": "It is not all the features that we can use for prediction. FC is a good resource to use because it is highly correlated with scalar_coupling_constant. What I'm trying now is to find the angle between atoms because, according to Wikipedia:
\n\nhttps://en.wikipedia.org/wiki/J-coupling
\n\n\"J-coupling contains information about relative bond distances and angles\".
",
"rawMarkdown": "It is not all the features that we can use for prediction. FC is a good resource to use because it is highly correlated with scalar_coupling_constant. What I'm trying now is to find the angle between atoms because, according to Wikipedia:\n\nhttps://en.wikipedia.org/wiki/J-coupling\n\n\"J-coupling contains information about relative bond distances and angles\".",
"votes": null
},
{
"id": "556613",
"postDate": "06/20/2019 13:06:06",
"content": "Thanks for this thread and the kernel. It opened up a new world of making use of intermediate data for me as a Kaggle beginner. I ignored those additional files.
\n\nTo my understanding and as you know, taking the log of OOF features, or taking the log of any features, should not have any non-trivial effect on a gradient boost decision tree model as lightGBM used in the kernel, since each of the tree splits is made by thresholding one feature and thus any monotonic transformation of that features should not have any effect on the split for only the feature rank among samples matters.
\n\nIn your case, since you are using oof_fc and log_oof_fc together in your model, I speculate the model just chose to use any one of the two by numerical reasons or some other unimportant criteria. It would be interesting, if you have not done so, to test just using one of oof_fc and log_oof_fc and compare the results. I guess in that case the result will be similar and we will see a new importance of the combined current oof_fc and log_oof_fc importance.
\n\nThanks again for the interesting thread and kernel.
",
"rawMarkdown": "Thanks for this thread and the kernel. It **opened up a new world** of making use of intermediate data for me as a Kaggle beginner. I ignored those additional files.\n\nTo my understanding and as you know, taking the log of OOF features, or taking the log of any features, should not have any non-trivial effect on a gradient boost decision tree model as lightGBM used in the kernel, since each of the tree splits is made by thresholding one feature and thus any monotonic transformation of that features should not have any effect on the split for only the feature rank among samples matters. \n\nIn your case, since you are using oof_fc and log_oof_fc together in your model, I speculate the model just chose to use any one of the two by numerical reasons or some other unimportant criteria. It would be interesting, if you have not done so, to test just using one of oof_fc and log_oof_fc and compare the results. I guess in that case the result will be similar and we will see a new importance of the combined current oof_fc and log_oof_fc importance. \n\nThanks again for the interesting thread and kernel.",
"votes": null
},
{
"id": "556618",
"postDate": "06/20/2019 13:11:52",
"content": "Thanks for the explanation.
\n\nBut does skew of the feature matter for a gradient boost tree? We are not directly using the feature values as in neural networks by multiplying some coefficient onto it, but only using the relative order of this feature between samples. As long as the feature rank among all samples does not change, any transformation should not have any effect on the gradient boost tree, right?
",
"rawMarkdown": "Thanks for the explanation. \n\nBut does skew of the feature matter for a gradient boost tree? We are not directly using the feature values as in neural networks by multiplying some coefficient onto it, but only using the relative order of this feature between samples. As long as the feature rank among all samples does not change, any transformation should not have any effect on the gradient boost tree, right?",
"votes": null
},
{
"id": "557822",
"postDate": "06/21/2019 13:52:46",
"content": "Hi Buffalo, very insteresting ideas you put. I'll test oof_fc vs oof_fc_log.
",
"rawMarkdown": "Hi Buffalo, very insteresting ideas you put. I'll test oof_fc vs oof_fc_log.",
"votes": null
}
],
"comments": [
{
"id": 553998,
"author_name": "mkonings1",
"author_url": "",
"post_date": "06/16/2019 18:42:29",
"content": "Interesting! Have you tried predicting other features as well and use it as feature for the final model? I might be going over that in the upcoming days as well:)
",
"votes": null,
"replies": [
{
"id": 554609,
"author_name": "adrianoavelar",
"author_url": "",
"post_date": "06/17/2019 18:21:17",
"content": "It is not all the features that we can use for prediction. FC is a good resource to use because it is highly correlated with scalar_coupling_constant. What I'm trying now is to find the angle between atoms because, according to Wikipedia:
\n\nhttps://en.wikipedia.org/wiki/J-coupling
\n\n\"J-coupling contains information about relative bond distances and angles\".
",
"votes": null,
"replies": []
}
]
},
{
"id": 554001,
"author_name": "venkat555",
"author_url": "",
"post_date": "06/16/2019 18:49:29",
"content": "Any hypothesis on why taking the log of the features is helping the model a lot
",
"votes": null,
"replies": [
{
"id": 554002,
"author_name": "mkonings1",
"author_url": "",
"post_date": "06/16/2019 18:54:45",
"content": "Took this straight from stackexchange:
\n\nStatistically, if your variables are right-skew (that is, they have a long tail at the high end) then a measure such as correlation or regression can be influenced a lot by one or a few cases at the high end on one or both variables (outliers, leverage points, influential points). Taking the log can help this by reducing or eliminating skew.
",
"votes": null,
"replies": []
},
{
"id": 554009,
"author_name": "venkat555",
"author_url": "",
"post_date": "06/16/2019 19:32:44",
"content": "Thanks a lot for the explanation
",
"votes": null,
"replies": []
},
{
"id": 556618,
"author_name": "barnwellguy",
"author_url": "",
"post_date": "06/20/2019 13:11:52",
"content": "Thanks for the explanation.
\n\nBut does skew of the feature matter for a gradient boost tree? We are not directly using the feature values as in neural networks by multiplying some coefficient onto it, but only using the relative order of this feature between samples. As long as the feature rank among all samples does not change, any transformation should not have any effect on the gradient boost tree, right?
",
"votes": null,
"replies": []
}
]
},
{
"id": 556613,
"author_name": "barnwellguy",
"author_url": "",
"post_date": "06/20/2019 13:06:06",
"content": "Thanks for this thread and the kernel. It opened up a new world of making use of intermediate data for me as a Kaggle beginner. I ignored those additional files.
\n\nTo my understanding and as you know, taking the log of OOF features, or taking the log of any features, should not have any non-trivial effect on a gradient boost decision tree model as lightGBM used in the kernel, since each of the tree splits is made by thresholding one feature and thus any monotonic transformation of that features should not have any effect on the split for only the feature rank among samples matters.
\n\nIn your case, since you are using oof_fc and log_oof_fc together in your model, I speculate the model just chose to use any one of the two by numerical reasons or some other unimportant criteria. It would be interesting, if you have not done so, to test just using one of oof_fc and log_oof_fc and compare the results. I guess in that case the result will be similar and we will see a new importance of the combined current oof_fc and log_oof_fc importance.
\n\nThanks again for the interesting thread and kernel.
",
"votes": null,
"replies": [
{
"id": 557822,
"author_name": "adrianoavelar",
"author_url": "",
"post_date": "06/21/2019 13:52:46",
"content": "Hi Buffalo, very insteresting ideas you put. I'll test oof_fc vs oof_fc_log.
",
"votes": null,
"replies": []
}
]
}
],
"raw_markdown_by_id": {
"553308": "Many kernels are using the OOF feature generated by prediction FC feature of `scalar_coupling_contributions` file. I also used this ideia in my kernel and realized that actually increases the precision of the model. But, in addition, extracts another feature by taking the log of OOF Feature generated and a new feature was very interesting, increasing my last LB score from -0.55 to -0.685. Click here ([Kernel LB -0.685](https://www.kaggle.com/adrianoavelar/molecule-pred-feature-eng-lb-0-55))to make the kernel for anyone who wants to take a look. I wish to thank for all thooses kernels and works that made my kernel possible, especially the artgor outstanding kernels, link in reference secion.\n\n\nI've used few estimators, so it's possible to increase that score even more. This is not my Final Kernel, still improving and search of new Features.",
"553998": "Interesting! Have you tried predicting other features as well and use it as feature for the final model? I might be going over that in the upcoming days as well:)",
"554001": "Any hypothesis on why taking the log of the features is helping the model a lot",
"554002": "Took this straight from stackexchange: \n\nStatistically, if your variables are right-skew (that is, they have a long tail at the high end) then a measure such as correlation or regression can be influenced a lot by one or a few cases at the high end on one or both variables (outliers, leverage points, influential points). Taking the log can help this by reducing or eliminating skew.",
"554009": "Thanks a lot for the explanation",
"554609": "It is not all the features that we can use for prediction. FC is a good resource to use because it is highly correlated with scalar_coupling_constant. What I'm trying now is to find the angle between atoms because, according to Wikipedia:\n\nhttps://en.wikipedia.org/wiki/J-coupling\n\n\"J-coupling contains information about relative bond distances and angles\".",
"556613": "Thanks for this thread and the kernel. It **opened up a new world** of making use of intermediate data for me as a Kaggle beginner. I ignored those additional files.\n\nTo my understanding and as you know, taking the log of OOF features, or taking the log of any features, should not have any non-trivial effect on a gradient boost decision tree model as lightGBM used in the kernel, since each of the tree splits is made by thresholding one feature and thus any monotonic transformation of that features should not have any effect on the split for only the feature rank among samples matters. \n\nIn your case, since you are using oof_fc and log_oof_fc together in your model, I speculate the model just chose to use any one of the two by numerical reasons or some other unimportant criteria. It would be interesting, if you have not done so, to test just using one of oof_fc and log_oof_fc and compare the results. I guess in that case the result will be similar and we will see a new importance of the combined current oof_fc and log_oof_fc importance. \n\nThanks again for the interesting thread and kernel.",
"556618": "Thanks for the explanation. \n\nBut does skew of the feature matter for a gradient boost tree? We are not directly using the feature values as in neural networks by multiplying some coefficient onto it, but only using the relative order of this feature between samples. As long as the feature rank among all samples does not change, any transformation should not have any effect on the gradient boost tree, right?",
"557822": "Hi Buffalo, very insteresting ideas you put. I'll test oof_fc vs oof_fc_log."
},
"source": "meta"
}