{ "id": 98444, "title": "1JHC best CV score?", "url": "/competitions/champs-scalar-coupling/discussion/98444", "author_name": "", "post_date": "2019-07-03T20:10:17.806734300Z", "votes": 31, "comment_count": 83, "views": 0, "content": "

Out of curiosity, what is your best 1JHC MAE CV score?\nSo far, I have managed to get it to around 0.96 on 3 folds without hp optimization.

\n\n

UPDATE 7-7-2019: Thanks to Giba's great kernel and some hp optimization (manual so far), I have managed to decrease the MAE to around 0.82 on 3 folds.

\n\n

UPDATE 15-7-2019: More features, MAE around 0.68 on 5 folds.

", "messages": [ { "id": "567660", "postDate": "07/03/2019 20:10:17", "content": "

Out of curiosity, what is your best 1JHC MAE CV score?\nSo far, I have managed to get it to around 0.96 on 3 folds without hp optimization.

\n\n

UPDATE 7-7-2019: Thanks to Giba's great kernel and some hp optimization (manual so far), I have managed to decrease the MAE to around 0.82 on 3 folds.

\n\n

UPDATE 15-7-2019: More features, MAE around 0.68 on 5 folds.

", "rawMarkdown": "Out of curiosity, what is your best 1JHC MAE CV score?\nSo far, I have managed to get it to around **0.96 on 3 folds** without hp optimization. \n\n**UPDATE** 7-7-2019: Thanks to Giba's great [kernel](https://www.kaggle.com/titericz/giba-r-data-table-simple-features-1-17-lb?scriptVersionId=16835193) and some hp optimization (manual so far), I have managed to decrease the MAE to around **0.82 on 3 folds**. \n\n**UPDATE** 15-7-2019: More features, MAE around **0.68 on 5 folds**.", "votes": null }, { "id": "567678", "postDate": "07/03/2019 20:39:17", "content": "

Currently for 1JHC I have 1.022 MAE, which corresponds to -1.116 LB score.

", "rawMarkdown": "Currently for 1JHC I have 1.022 MAE, which corresponds to -1.116 LB score.", "votes": null }, { "id": "567688", "postDate": "07/03/2019 20:48:14", "content": "

CV for every type\nType Log MAE\n1JHC -0.49\n2JHH -1.83\n1JHN -1.19\n2JHN -1.85\n2JHC -1.38\n3JHH -1.82\n3JHC -1.31\n3JHN -2.1\nCV -1.49625\nLB -.1.6

", "rawMarkdown": "CV for every type\nType\tLog MAE\n1JHC\t-0.49\n2JHH\t-1.83\n1JHN\t-1.19\n2JHN\t-1.85\n2JHC\t-1.38\n3JHH\t-1.82\n3JHC\t-1.31\n3JHN\t-2.1\nCV\t-1.49625\nLB -.1.6", "votes": null }, { "id": "567971", "postDate": "07/04/2019 07:47:14", "content": "

1JHC CV MAE = 0.85 (5 fold)

", "rawMarkdown": "1JHC CV MAE = 0.85 (5 fold)", "votes": null }, { "id": "568271", "postDate": "07/04/2019 16:10:08", "content": "

My 1JHC MAE: 0.89

", "rawMarkdown": "My 1JHC MAE: 0.89", "votes": null }, { "id": "568284", "postDate": "07/04/2019 16:35:32", "content": "

1JHC 0,8 MAE \nI ‘m not a chemist but i don’t understand why 1JHC is the hardest? The atoms are just close. At the beginning, I thought i would be the easier! 😂

", "rawMarkdown": "1JHC 0,8 MAE \nI ‘m not a chemist but i don’t understand why 1JHC is the hardest? The atoms are just close. At the beginning, I thought i would be the easier! 😂", "votes": null }, { "id": "568352", "postDate": "07/04/2019 18:54:30", "content": "

For 1JHC my MAE is 0.62.\nI have spent the last three days creating new features which for some reason improved CV score a lot (to -1.635), but LB score got actually worse (to -1.095). Now, who knows whether to keep them or not :D

", "rawMarkdown": "For 1JHC my MAE is 0.62.\nI have spent the last three days creating new features which for some reason improved CV score a lot (to -1.635), but LB score got actually worse (to -1.095). Now, who knows whether to keep them or not :D", "votes": null }, { "id": "568396", "postDate": "07/04/2019 21:02:16", "content": "

The major factors for one-bond
\nC-H couplings are substituent effects, hybridization, and ring strain.

\n\n

Other coupling types might be more affected by geometric properties, which are easier to generate features from.

", "rawMarkdown": "The major factors for one-bond \nC-H couplings are substituent effects, hybridization, and ring strain.\n\nOther coupling types might be more affected by geometric properties, which are easier to generate features from.", "votes": null }, { "id": "568416", "postDate": "07/04/2019 22:19:27", "content": "

| Fold | LGB | GNN |\n|------------|------|------|\n| Group | 0.67 | 1.12 |\n| Stratified | 0.62 | 0.96 |

\n\n

My stratified fold is based off of a preliminary k-means clustering using some topological features.

\n\n

Update as of Jul 20: after some serious efforts in generating new features (reading papers in QM, etc). My best model's 1JHC's CV MAE is now 0.39 using GroupKFold.

", "rawMarkdown": "| Fold | LGB | GNN |\n|------------|------|------|\n| Group | 0.67 | 1.12 |\n| Stratified | 0.62 | 0.96 |\n\nMy stratified fold is based off of a preliminary k-means clustering using some topological features.\n\nUpdate as of Jul 20: after some serious efforts in generating new features (reading papers in QM, etc). My best model's 1JHC's CV MAE is now 0.39 using GroupKFold.", "votes": null }, { "id": "568427", "postDate": "07/04/2019 23:09:11", "content": "

LGB CV: -1.5733 / LB: -1.637 :\n| type | log | mae |\n| --- | --- | --- |\n| 1JHC | -0.5620 | 0.5700 |\n| 2JHH | -2.0817 | 0.1247 |\n| 1JHN | -1.3176 | 0.2677 |\n| 2JHN | -1.9128 | 0.1476 |\n| 2JHC | -1.3932 | 0.2482 |\n| 3JHH | -1.8386 | 0.1590 |\n| 3JHC | -1.3066 | 0.2707 |\n| 3JHN | -2.1737 | 0.1137 |

", "rawMarkdown": "LGB CV: -1.5733 / LB: -1.637 :\n| type | log | mae |\n| --- | --- | --- |\n| 1JHC | -0.5620 | 0.5700 |\n| 2JHH | -2.0817 | 0.1247 |\n| 1JHN | -1.3176 | 0.2677 |\n| 2JHN | -1.9128 | 0.1476 |\n| 2JHC | -1.3932 | 0.2482 |\n| 3JHH | -1.8386 | 0.1590 |\n| 3JHC | -1.3066 | 0.2707 |\n| 3JHN | -2.1737 | 0.1137 |", "votes": null }, { "id": "568536", "postDate": "07/05/2019 04:39:14", "content": "

how many features?

", "rawMarkdown": "how many features?", "votes": null }, { "id": "568546", "postDate": "07/05/2019 04:58:04", "content": "

LGB: CV: -1.494 / LB: -1.430

\n\n

type | log | mae\n1JHC | -0.4713 | 0.6242\n1JHN | -1.0188 | 0.3610\n2JHC | -1.4327 | 0.2387\n2JHH | -2.0086 | 0.1342\n2JHN | -1.6206 | 0.1978\n3JHC | -1.3490 | 0.2595\n3JHH | -1.9725 | 0.1391\n3JHN | -2.0821 | 0.1247

", "rawMarkdown": "LGB: CV: -1.494 / LB: -1.430\n\ntype | log\t | mae\n1JHC | -0.4713 | 0.6242\n1JHN | -1.0188 | 0.3610\n2JHC | -1.4327 | 0.2387\n2JHH | -2.0086 | 0.1342\n2JHN | -1.6206 | 0.1978\n3JHC | -1.3490 | 0.2595\n3JHH | -1.9725 | 0.1391\n3JHN | -2.0821 | 0.1247", "votes": null }, { "id": "568761", "postDate": "07/05/2019 12:04:14", "content": "

index | CV mean | std_err\n---- | ---- | ----\n1jhn-lgbm | -0.67 | 0.0044\n3jhn-lgbm | -1.8 | 0.0035\n2jhn-lgbm | -1.4 | 0.0039\n3jhc-lgbm | -0.47 | 0.0013\n2jhc-lgbm | -0.54 | 0.0013\n1jhc-lgbm | 0.14 | 0.0025\n3jhh-lgbm | -1.2 | 0.0021\n2jhh-lgbm | -1.2 | 0.0022

\n\n

Function to convert pandas to markdown: https://stackoverflow.com/a/39334770/10743932

", "rawMarkdown": "index | CV mean | std_err\n---- | ---- | ----\n1jhn-lgbm | -0.67 | 0.0044\n3jhn-lgbm | -1.8 | 0.0035\n2jhn-lgbm | -1.4 | 0.0039\n3jhc-lgbm | -0.47 | 0.0013\n2jhc-lgbm | -0.54 | 0.0013\n1jhc-lgbm | 0.14 | 0.0025\n3jhh-lgbm | -1.2 | 0.0021\n2jhh-lgbm | -1.2 | 0.0022\n\nFunction to convert pandas to markdown: https://stackoverflow.com/a/39334770/10743932", "votes": null }, { "id": "569110", "postDate": "07/06/2019 04:52:57", "content": "

| type | cv |\n| --- | --- |\n| 1JHC | -0.10803 |\n| 2JHH | -1.55402|\n| 1JHN | -0.55320 |\n| 2JHN | -0.96341 |\n| 2JHC | -0.85063 |\n| 3JHH | -1.43886 |\n| 3JHC | -0.72023 |\n| 3JHN | -1.48864 |\n|OOF CV | -0.95963 |\n|Public LB|-1.510|

", "rawMarkdown": "| type | cv |\n| --- | --- |\n| 1JHC | -0.10803 |\n| 2JHH | -1.55402|\n| 1JHN | -0.55320 |\n| 2JHN | -0.96341 |\n| 2JHC | -0.85063 |\n| 3JHH | -1.43886 |\n| 3JHC | -0.72023 |\n| 3JHN | -1.48864 |\n|OOF CV | -0.95963 |\n|Public LB|-1.510|", "votes": null }, { "id": "569113", "postDate": "07/06/2019 05:02:47", "content": "

It looks like there is a difference between the out of fold CV and the average CV across types.

", "rawMarkdown": "It looks like there is a difference between the out of fold CV and the average CV across types.", "votes": null }, { "id": "570002", "postDate": "07/07/2019 16:47:15", "content": "

The other scores so far (3 folds GroupKFold):

\n\n

|type|MAE|LMAE|\n|-----|-----|----|\n|1JHC| 0.829 |-0.1875|\n|2JHH|0.1976|-1.622|\n|1JHN|0.5079|-0.6775|\n|2JHN|0.2631|-1.335|\n|2JHC|0.3849|-0.9547|\n|3JHH|0.2331|-1.4568|\n|3JHC|0.4533|-0.7912\n|3JHN|0.1899|-1.661

\n\n

-1.215 LB <---> -1.0856 CV (0.4867 std)

", "rawMarkdown": "The other scores so far (3 folds GroupKFold):\n\n|type|MAE|LMAE|\n|-----|-----|----|\n|1JHC| 0.829 |-0.1875|\n|2JHH|0.1976|-1.622|\n|1JHN|0.5079|-0.6775|\n|2JHN|0.2631|-1.335|\n|2JHC|0.3849|-0.9547|\n|3JHH|0.2331|-1.4568|\n|3JHC|0.4533|-0.7912\n|3JHN|0.1899|-1.661\n\n\n-1.215 LB <---> -1.0856 CV (0.4867 std)", "votes": null }, { "id": "570043", "postDate": "07/07/2019 17:52:06", "content": "

~60

", "rawMarkdown": "~60", "votes": null }, { "id": "570057", "postDate": "07/07/2019 18:11:35", "content": "

Minor suggestion (since I had the same issue): if you want the table to be correctly formatted, you need to add a | at the end of each row. :)

", "rawMarkdown": "Minor suggestion (since I had the same issue): if you want the table to be correctly formatted, you need to add a `|` at the end of each row. :)", "votes": null }, { "id": "570059", "postDate": "07/07/2019 18:12:41", "content": "

I guess you did some features selection? I am working on this and will update the score if I find something meaningful.

", "rawMarkdown": "I guess you did some features selection? I am working on this and will update the score if I find something meaningful.", "votes": null }, { "id": "570128", "postDate": "07/07/2019 20:36:12", "content": "

On damnable ton of features. Somehow my LB is lower than CV, that's very strange.\ntype | log | mae |\n1JHC | -0.825 | 0.573 |\n1JHN | -1.650 | 0.296 |\n2JHC | -1.273 | 0.308 |\n2JHH | -2.138 | 0.151 |\n2JHN | -1.992 | 0.193 |\n3JHC | -1.841 | 0.433 |\n3JHH | -1.816 | 0.212 |\n3JHN | -2.32 | 0.157 |

", "rawMarkdown": "On damnable ton of features. Somehow my LB is lower than CV, that's very strange.\ntype | log | mae |\n1JHC | -0.825 | 0.573 |\n1JHN | -1.650 | 0.296 |\n2JHC | -1.273 | 0.308 |\n2JHH | -2.138 | 0.151 |\n2JHN | -1.992 | 0.193 |\n3JHC | -1.841 | 0.433 |\n3JHH | -1.816 | 0.212 |\n3JHN | -2.32 | 0.157 |", "votes": null }, { "id": "570133", "postDate": "07/07/2019 20:59:10", "content": "

that is exactly how many? :)

", "rawMarkdown": "that is exactly how many? :)", "votes": null }, { "id": "570209", "postDate": "07/08/2019 02:19:40", "content": "

On the other hand, many of the hydrogen change the angle constantly. Actually a lot of the relative positioning of the atoms change along the time. What you observe experimentally is the average coupling constant for all the possible conformations of the molecule. I don't really know how the conformers of this dataset was chosen, but up to I know, some of them are actually quite 'exotic' in the nature. I've found many examples in this dataset, where you obtain very different coupling constants where in practice you must observe exactly the same one, because the atoms are magnetically equivalent. I don't know if this could affect more thr 1JCH coupling constants

", "rawMarkdown": "On the other hand, many of the hydrogen change the angle constantly. Actually a lot of the relative positioning of the atoms change along the time. What you observe experimentally is the average coupling constant for all the possible conformations of the molecule. I don't really know how the conformers of this dataset was chosen, but up to I know, some of them are actually quite 'exotic' in the nature. I've found many examples in this dataset, where you obtain very different coupling constants where in practice you must observe exactly the same one, because the atoms are magnetically equivalent. I don't know if this could affect more thr 1JCH coupling constants", "votes": null }, { "id": "570271", "postDate": "07/08/2019 04:29:29", "content": "

Here's mine so far using LGBM with default parameters (as others have mentioned, there are probably significant gains from using different parameters each of the xJHy types, but haven't tested this yet):\n| type | LMAE | MAE |\n| --- | --- | --- |\n| 1JHC | -0.6615 | 0.5161 |\n| 1JHN | -1.2136 | 0.2971 |\n| 2JHH | -1.9525 | 0.1429 |\n| 2JHC | -1.4936 | 0.2246 |\n| 2JHN | -1.7773 | 0.1691 |\n| 3JHH | -1.9407 | 0.1436 |\n| 3JHC | -1.419 | 0.2420 |\n| 3JHN | -2.0771| 0.1253 |

\n\n

Final CV: -1.5669, Public LB: -1.712

\n\n

I've just got back into this competition after a couple of weeks holiday though so got a bit of catching up to do!

", "rawMarkdown": "Here's mine so far using LGBM with default parameters (as others have mentioned, there are probably significant gains from using different parameters each of the xJHy types, but haven't tested this yet):\n| type | LMAE | MAE |\n| --- | --- | --- |\n| 1JHC | -0.6615 | 0.5161 |\n| 1JHN | -1.2136 | 0.2971 |\n| 2JHH | -1.9525 | 0.1429 |\n| 2JHC | -1.4936 | 0.2246 |\n| 2JHN | -1.7773 | 0.1691 |\n| 3JHH | -1.9407 | 0.1436 |\n| 3JHC | -1.419 | 0.2420 |\n| 3JHN | -2.0771| 0.1253 |\n\nFinal CV: -1.5669, Public LB: -1.712\n\nI've just got back into this competition after a couple of weeks holiday though so got a bit of catching up to do!", "votes": null }, { "id": "570340", "postDate": "07/08/2019 06:50:44", "content": "

~500

", "rawMarkdown": "~500", "votes": null }, { "id": "570741", "postDate": "07/08/2019 18:14:34", "content": "

Your two numbers are not consistent ;)

", "rawMarkdown": "Your two numbers are not consistent ;)", "votes": null }, { "id": "570758", "postDate": "07/08/2019 18:32:26", "content": "

Why not consistent? Note, that my CV score is MAE, not Log MAE (because in original post MAE was given). Maybe that makes them to look not consistent.

\n\n

Or you mean they are not consistent with my current score? Then I admit, I spoiled consistency by improving my model a bit :D

", "rawMarkdown": "Why not consistent? Note, that my CV score is MAE, not Log MAE (because in original post MAE was given). Maybe that makes them to look not consistent.\n\nOr you mean they are not consistent with my current score? Then I admit, I spoiled consistency by improving my model a bit :D", "votes": null }, { "id": "570768", "postDate": "07/08/2019 18:41:27", "content": "

I think there's misunderstanding. 1.0 MAE is for 1 type, and LB is overall. Maybe @cpmpml thougth about MAE and logMAE for 1 type

", "rawMarkdown": "I think there's misunderstanding. 1.0 MAE is for 1 type, and LB is overall. Maybe @cpmpml thougth about MAE and logMAE for 1 type", "votes": null }, { "id": "571285", "postDate": "07/09/2019 12:58:37", "content": "

Something seems off there - log(0.573) is not -0.825.

", "rawMarkdown": "Something seems off there - log(0.573) is not -0.825.", "votes": null }, { "id": "571390", "postDate": "07/09/2019 14:54:27", "content": "

@chriskorn .573 is val MAE. Train MAE is less

", "rawMarkdown": "chriskorn .573 is val MAE. Train MAE is less", "votes": null }, { "id": "571397", "postDate": "07/09/2019 14:58:19", "content": "

Ah I see - sorry, that hadn't been clear to me before.

", "rawMarkdown": "Ah I see - sorry, that hadn't been clear to me before.", "votes": null }, { "id": "571404", "postDate": "07/09/2019 15:06:36", "content": "

Tx Stanislav, I thought the two numbers were for the same data.

", "rawMarkdown": "Tx Stanislav, I thought the two numbers were for the same data.", "votes": null }, { "id": "571430", "postDate": "07/09/2019 15:27:18", "content": "

Ma pleasure, Mr. Uncle!

", "rawMarkdown": "Ma pleasure, Mr. Uncle!", "votes": null }, { "id": "571473", "postDate": "07/09/2019 16:17:02", "content": "

1JHC consistent weakest score regards of my model. Not time yet, but pretty soon its on my plan schedule for exclusive use of my time :)

\n\n

I

", "rawMarkdown": "1JHC consistent weakest score regards of my model. Not time yet, but pretty soon its on my plan schedule for exclusive use of my time :)\n\nI", "votes": null }, { "id": "572137", "postDate": "07/10/2019 14:21:36", "content": "

This is quit good score without tunning.\nMay I ask how many features you used?

", "rawMarkdown": "This is quit good score without tunning.\nMay I ask how many features you used?", "votes": null }, { "id": "572387", "postDate": "07/10/2019 21:19:16", "content": "

Hey guys what is your training score compared to your validation score? i have training score like 0.02 but validation at like 0.22

", "rawMarkdown": "Hey guys what is your training score compared to your validation score? i have training score like 0.02 but validation at like 0.22", "votes": null }, { "id": "572392", "postDate": "07/10/2019 21:26:22", "content": "

Same situation, but my val score is worse than yours for 1jhc. I racked my brain trying to deal with this type, but no luck so far. \nMind to drop some knowledge and give a hint? ;)

", "rawMarkdown": "Same situation, but my val score is worse than yours for 1jhc. I racked my brain trying to deal with this type, but no luck so far. \nMind to drop some knowledge and give a hint? ;)", "votes": null }, { "id": "572393", "postDate": "07/10/2019 21:29:17", "content": "

Yes, same question as well. :)

", "rawMarkdown": "Yes, same question as well. :)", "votes": null }, { "id": "572395", "postDate": "07/10/2019 21:37:46", "content": "

actually, i am only dealing with 3jhh at the moment. currently at 0.22 mae with 3jhh

", "rawMarkdown": "actually, i am only dealing with 3jhh at the moment. currently at 0.22 mae with 3jhh", "votes": null }, { "id": "572525", "postDate": "07/11/2019 03:18:02", "content": "

I have 790 in total at the moment but many of these are useless features which I haven't removed yet, and some of them are not applicable for different types

", "rawMarkdown": "I have 790 in total at the moment but many of these are useless features which I haven't removed yet, and some of them are not applicable for different types", "votes": null }, { "id": "572622", "postDate": "07/11/2019 07:25:43", "content": "

Oh I thought you were talking about 1jhc.\nTho for 3jhh I have similar results: train 0.03 and val 0.2

", "rawMarkdown": "Oh I thought you were talking about 1jhc.\nTho for 3jhh I have similar results: train 0.03 and val 0.2", "votes": null }, { "id": "573533", "postDate": "07/12/2019 11:34:50", "content": "

My current submission is based on 1JHC 0.89. Without using Giba's feature, I'm able to reduce it to (valid_1's l1: 0.615775). Now I'm convinced that we could reduce it to around 0.5.

", "rawMarkdown": "My current submission is based on 1JHC 0.89. Without using Giba's feature, I'm able to reduce it to (valid_1's l1: 0.615775). Now I'm convinced that we could reduce it to around 0.5.", "votes": null }, { "id": "578551", "postDate": "07/17/2019 21:01:33", "content": "

this happens when you optimize hyper parameters against the train set. I've got bitten by the same beast. To overcome this you should keep one fold out of your cv as a pre-test set to check if hyper optimization has not overfit the train set...

", "rawMarkdown": "this happens when you optimize hyper parameters against the train set. I've got bitten by the same beast. To overcome this you should keep one fold out of your cv as a pre-test set to check if hyper optimization has not overfit the train set...", "votes": null }, { "id": "578573", "postDate": "07/17/2019 21:58:01", "content": "

0.587 MAE for 1JHC - LGBM with no parameter tuning. Around 200 features.\nUpdate 7/23 - 0.486 for 1JHC. \nUpdate 728 - 0.5133 for 1JHC, but now -1.689 LB with some re-design of feature selection and CV metrics.

", "rawMarkdown": "0.587 MAE for 1JHC - LGBM with no parameter tuning. Around 200 features.\nUpdate 7/23 - 0.486 for 1JHC. \nUpdate 728 - 0.5133 for 1JHC, but now -1.689 LB with some re-design of feature selection and CV metrics.", "votes": null }, { "id": "580776", "postDate": "07/20/2019 18:35:34", "content": "

@anjum48 How do you run you program with 790 features? I generated 500+ features but the program always crashs because of memory overflow (64g memory). Thanks

", "rawMarkdown": "anjum48 How do you run you program with 790 features? I generated 500+ features but the program always crashs because of memory overflow (64g memory). Thanks", "votes": null }, { "id": "580805", "postDate": "07/20/2019 19:41:53", "content": "

Maybe he has a bigger memory :D Have you tried reducing the memory with the script? You don't need float64 for everything. Also e. g. molecule names take less memory in DataFrame if you store them as ints.

", "rawMarkdown": "Maybe he has a bigger memory :D Have you tried reducing the memory with the script? You don't need float64 for everything. Also e. g. molecule names take less memory in DataFrame if you store them as ints.", "votes": null }, { "id": "580937", "postDate": "07/21/2019 05:23:59", "content": "

@waylongo At the moment my memory usage is quite a bit below 64GB. As @somnambwl mentioned if you use the appropriate data types you can get some major memory savings. Also, see if you can find ways of manipulating your data without needing to copy it, for example, a lot of the sklearn.preprocessing methods have a copy option which can prevent a memory spike when you call those methods (there are similar efficiencies to be found in pandas and numpy using in-place operations).

\n\n

My old laptop only had 4GB of RAM. Although it annoyed me at the time, it really taught me how to be careful around memory usage and I learned things I wouldn't have learnt if I had a more powerful machine.

", "rawMarkdown": "waylongo At the moment my memory usage is quite a bit below 64GB. As @somnambwl mentioned if you use the appropriate data types you can get some major memory savings. Also, see if you can find ways of manipulating your data without needing to copy it, for example, a lot of the `sklearn.preprocessing` methods have a `copy` option which can prevent a memory spike when you call those methods (there are similar efficiencies to be found in `pandas` and `numpy` using in-place operations).\n\nMy old laptop only had 4GB of RAM. Although it annoyed me at the time, it really taught me how to be careful around memory usage and I learned things I wouldn't have learnt if I had a more powerful machine.", "votes": null }, { "id": "580984", "postDate": "07/21/2019 07:49:07", "content": "

Hello, everyone. Currently, my validation score(using 20% of train) is -1.55 while my LB score is -1.65.

\n\n

Looking through this thread, most people have much lower error(thus better score) for LB compared to CV. Does anyone have any idea why this is the case?

", "rawMarkdown": "Hello, everyone. Currently, my validation score(using 20% of train) is -1.55 while my LB score is -1.65.\n\nLooking through this thread, most people have much lower error(thus better score) for LB compared to CV. Does anyone have any idea why this is the case?", "votes": null }, { "id": "580987", "postDate": "07/21/2019 07:58:44", "content": "

It might be that the public part of the dataset has more (in proportion) easier bond types to predict (2JHH, 3JHN, and 3JHH for example). Thus, you should trust your local CV.

", "rawMarkdown": "It might be that the public part of the dataset has **more** (in proportion) **easier bond types** to predict (2JHH, 3JHN, and 3JHH for example). Thus, you should trust your local CV.", "votes": null }, { "id": "580991", "postDate": "07/21/2019 08:04:30", "content": "

The explanation must be different because the score does not depend on proportion of each type....

", "rawMarkdown": "The explanation must be different because the score does not depend on proportion of each type....", "votes": null }, { "id": "581003", "postDate": "07/21/2019 08:29:30", "content": "

Good point indeed. The difference could be explained if some bond types aren't there at all in the public part. I guess that this claim could be verified by sending 8 submissions with constant values for one of the types each time and checking if this gives the same result each time.

", "rawMarkdown": "Good point indeed. The difference could be explained if some bond types aren't there at all in the public part. I guess that this claim could be verified by sending 8 submissions with constant values for one of the types each time and checking if this gives the same result each time.", "votes": null }, { "id": "581056", "postDate": "07/21/2019 10:32:58", "content": "

When you use, as an example, 5 fold CV, then the results on the OOF is based on a single model for each validation set, but the LB result is based on the average of 5 models (in case you average the models and do not rerun for the whole training set). This could explain why the LB result is better.

", "rawMarkdown": "When you use, as an example, 5 fold CV, then the results on the OOF is based on a single model for each validation set, but the LB result is based on the average of 5 models (in case you average the models and do not rerun for the whole training set). This could explain why the LB result is better.", "votes": null }, { "id": "581854", "postDate": "07/22/2019 13:24:03", "content": "

@helgi, you are probably right, with one fold only I have a LB of -2.001 for a val score of -2.017, I don't see any LB improvement.

", "rawMarkdown": "helgi, you are probably right, with one fold only I have a LB of -2.001 for a val score of -2.017, I don't see any LB improvement.", "votes": null }, { "id": "581857", "postDate": "07/22/2019 13:26:00", "content": "

one fold, lgb, validation score:

\n\n

1JHC -0.777\n1JHN -1.233\n2JHC -1.934\n2JHH -2.454\n2JHN -2.442\n3JHC -1.992\n3JHH -2.613\n3JHN -2.689

\n\n

total -2.017\nLB -2.001

\n\n

It seems averaging several folds explains why LB is better than CV for many.

\n\n

Despite my team name now (lgb, what else) I don't think lgb is competitive here, will move to graph NNs now.

", "rawMarkdown": "one fold, lgb, validation score:\n\n1JHC -0.777\n1JHN -1.233\n2JHC -1.934\n2JHH -2.454\n2JHN -2.442\n3JHC -1.992\n3JHH -2.613\n3JHN -2.689\n\ntotal -2.017\nLB -2.001\n\nIt seems averaging several folds explains why LB is better than CV for many.\n\nDespite my team name now (lgb, what else) I don't think lgb is competitive here, will move to graph NNs now.", "votes": null }, { "id": "581897", "postDate": "07/22/2019 14:09:13", "content": "

I started with lgb because the feature engineering made me understand some of the data and what matters.

", "rawMarkdown": "I started with lgb because the feature engineering made me understand some of the data and what matters.", "votes": null }, { "id": "581965", "postDate": "07/22/2019 15:34:48", "content": "

Amazing scores by 1 fold CV !\nMaybe there’re strong features I couldn’t find yet...\nAre you using any external dataset (ex. QM7/9)?

", "rawMarkdown": "Amazing scores by 1 fold CV !\nMaybe there’re strong features I couldn’t find yet...\nAre you using any external dataset (ex. QM7/9)?", "votes": null }, { "id": "581974", "postDate": "07/22/2019 15:47:04", "content": "

No external data.

", "rawMarkdown": "No external data.", "votes": null }, { "id": "581977", "postDate": "07/22/2019 15:51:46", "content": "

OK, thank you

", "rawMarkdown": "OK, thank you", "votes": null }, { "id": "581988", "postDate": "07/22/2019 16:11:30", "content": "

Thanks. Are you using lgb too?

", "rawMarkdown": "Thanks. Are you using lgb too?", "votes": null }, { "id": "581998", "postDate": "07/22/2019 16:24:22", "content": "

Yes, we’re using lgb.

", "rawMarkdown": "Yes, we’re using lgb.", "votes": null }, { "id": "582010", "postDate": "07/22/2019 16:49:49", "content": "

Wow that's one of a mighty first submission... Good job!\nHow are you doing this?! :D

", "rawMarkdown": "Wow that's one of a mighty first submission... Good job!\nHow are you doing this?! :D", "votes": null }, { "id": "582059", "postDate": "07/22/2019 17:54:48", "content": "
\n

How are you doing this?

\n
\n\n

Read all the forum, and crafted my features. I didn't reuse any of the public kernel features, but I may have recoded some of them without knowing. Then lots of experiments...

", "rawMarkdown": "> How are you doing this?\n\nRead all the forum, and crafted my features. I didn't reuse any of the public kernel features, but I may have recoded some of them without knowing. Then lots of experiments...", "votes": null }, { "id": "582061", "postDate": "07/22/2019 17:56:07", "content": "
\n

Wow that's one of a mighty first submission… Good job!

\n
\n\n

Thanks, but others have a much better 1st sub...

", "rawMarkdown": "> Wow that's one of a mighty first submission… Good job!\n\nThanks, but others have a much better 1st sub...", "votes": null }, { "id": "582158", "postDate": "07/22/2019 20:17:45", "content": "

It was kinda rhetorical question :) Of course you won't reveal the tricks for now. \nDid a ton of experiments myself, but nothing worked so far. Guess my gut is just not as skilled as yours...

\n\n
\n

others have a much better 1st sub

\n
\n\n

Trying to figure out how to become this \"others\" for somw time already

", "rawMarkdown": "It was kinda rhetorical question :) Of course you won't reveal the tricks for now. \nDid a ton of experiments myself, but nothing worked so far. Guess my gut is just not as skilled as yours...\n> others have a much better 1st sub\n\nTrying to figure out how to become this \"others\" for somw time already", "votes": null }, { "id": "582163", "postDate": "07/22/2019 20:31:58", "content": "

That's awesome. 👍

", "rawMarkdown": "That's awesome. 👍", "votes": null }, { "id": "583026", "postDate": "07/23/2019 23:05:35", "content": "

an update from me: i have 1JHC mae cv of 0.498577 using lightgbm. Using neural network, i have like 1.20 CV which is pretty bad in comparison. hoping to improve my NN performance so that it will work well in ensemble

\n\n

Edit: July 24 2019: new CV is 0.487761

", "rawMarkdown": "an update from me: i have 1JHC mae cv of 0.498577 using lightgbm. Using neural network, i have like 1.20 CV which is pretty bad in comparison. hoping to improve my NN performance so that it will work well in ensemble\n\nEdit: July 24 2019: new CV is 0.487761", "votes": null }, { "id": "584561", "postDate": "07/26/2019 06:55:50", "content": "

My current 5 folds CV, LGBM:\n1JHC -0.539175\n1JHN -1.131305\n2JHC -1.445589\n2JHH -1.974381\n2JHN -1.795792\n3JHC -1.481445\n3JHH -1.899323\n3JHN -2.111573\nmean -1.5473228144612587\nLB=-1.842

", "rawMarkdown": "My current 5 folds CV, LGBM:\n1JHC -0.539175\n1JHN -1.131305\n2JHC -1.445589\n2JHH -1.974381\n2JHN -1.795792\n3JHC -1.481445\n3JHH -1.899323\n3JHN -2.111573\nmean -1.5473228144612587\nLB=-1.842", "votes": null }, { "id": "584665", "postDate": "07/26/2019 09:25:24", "content": "

type | Log MAE |\n1JHC | -0.514 |\n2JHH | -1.845 |\n1JHN | -1.043 |\n2JHN | -1.750 |\n2JHC | -1.389 |\n3JHH | -1.814 |\n3JHC | -1.2818 |\n3JHN | -2.058 |

\n\n

CV 1.46, LB 1.53

", "rawMarkdown": "type | Log MAE |\n1JHC | -0.514 |\n2JHH | -1.845 |\n1JHN | -1.043 |\n2JHN | -1.750 |\n2JHC | -1.389 |\n3JHH | -1.814 |\n3JHC | -1.2818 |\n3JHN | -2.058 |\n\nCV 1.46, LB 1.53", "votes": null }, { "id": "592943", "postDate": "08/06/2019 02:10:34", "content": "

5fold GNN

\n\n

1JHC -1.2735724333156546\n1JHN -1.880088909076347\n2JHC -2.2438082211992447\n2JHH -3.148106336806997\n2JHN -2.934080757996177\n3JHC -2.012966179536739\n3JHH -2.941703155183916\n3JHN -3.089508672452637

\n\n

CV -2.440\nLB -2.639

", "rawMarkdown": "5fold GNN\n\n1JHC -1.2735724333156546\n1JHN -1.880088909076347\n2JHC -2.2438082211992447\n2JHH -3.148106336806997\n2JHN -2.934080757996177\n3JHC -2.012966179536739\n3JHH -2.941703155183916\n3JHN -3.089508672452637\n\nCV -2.440\nLB -2.639", "votes": null }, { "id": "592951", "postDate": "08/06/2019 02:52:46", "content": "

what? how?\n\"\"

", "rawMarkdown": "what? how?\n![](https://media.giphy.com/media/3o7btPCcdNniyf0ArS/giphy.gif)", "votes": null }, { "id": "592998", "postDate": "08/06/2019 05:07:42", "content": "

We should team as we have some types with way better score (and some with way worse) ;)

\n\n

Oops, we can't because of team size.

", "rawMarkdown": "We should team as we have some types with way better score (and some with way worse) ;)\n\nOops, we can't because of team size.", "votes": null }, { "id": "593033", "postDate": "08/06/2019 05:43:43", "content": "

We don't have domain knowledge so I think top rankers are doing very different solution from us ,we expected to team up.\n@cpmpml Hope to team with you one day,this time good luck to you.

", "rawMarkdown": "We don't have domain knowledge so I think top rankers are doing very different solution from us ,we expected to team up.\n@cpmpml Hope to team with you one day,this time good luck to you.", "votes": null }, { "id": "593048", "postDate": "08/06/2019 06:02:36", "content": "

@senkin13 Sure, I was thinking about this since Talking Data...

\n\n

We don't have domain knowledge either :(

", "rawMarkdown": "senkin13 Sure, I was thinking about this since Talking Data...\n\nWe don't have domain knowledge either :(", "votes": null }, { "id": "593076", "postDate": "08/06/2019 06:57:08", "content": "

@cpmpml Yes me too ,I hope no shakedown like TalkingData 😄

", "rawMarkdown": "cpmpml Yes me too ,I hope no shakedown like TalkingData 😄", "votes": null }, { "id": "593373", "postDate": "08/06/2019 14:06:54", "content": "

@senkin13 Thank you for your kind information. If you don't mind, may I ask did you train all molecules at once, or you pick only molecules with the associated type you want to separately train?

", "rawMarkdown": "senkin13 Thank you for your kind information. If you don't mind, may I ask did you train all molecules at once, or you pick only molecules with the associated type you want to separately train?", "votes": null }, { "id": "593683", "postDate": "08/07/2019 01:21:25", "content": "

We use both,good luck.

", "rawMarkdown": "We use both,good luck.", "votes": null }, { "id": "595645", "postDate": "08/09/2019 13:56:12", "content": "

5fold lgb

\n\n

1JHC -0.67306731\n1JHN -1.24935432\n2JHC -1.60763475\n2JHH -2.25585111\n2JHN -2.05962611\n3JHC -1.58876169\n3JHH -2.0570262\n3JHN -2.30307408

\n\n

CV -1.724299\nLB -1.981

\n\n

all features are downloaded from public kernels , I didn't created any other features because the kernels have realized all my ideas

", "rawMarkdown": "5fold lgb\n\n1JHC -0.67306731\n1JHN -1.24935432\n2JHC -1.60763475\n2JHH -2.25585111\n2JHN -2.05962611\n3JHC -1.58876169\n3JHH -2.0570262\n3JHN -2.30307408\n\nCV -1.724299\nLB -1.981\n\nall features are downloaded from public kernels , I didn't created any other features because the kernels have realized all my ideas", "votes": null }, { "id": "595711", "postDate": "08/09/2019 15:11:47", "content": "

What are the top features for 2jhc and 3jhc?

", "rawMarkdown": "What are the top features for 2jhc and 3jhc?", "votes": null }, { "id": "596452", "postDate": "08/10/2019 16:48:16", "content": "

Impressive, are you using features derived from QM7/9?

", "rawMarkdown": "Impressive, are you using features derived from QM7/9?", "votes": null }, { "id": "596560", "postDate": "08/10/2019 21:06:57", "content": "

Essentially speaking, charge weighted distance features are QM7 features.

", "rawMarkdown": "Essentially speaking, charge weighted distance features are QM7 features.", "votes": null }, { "id": "596571", "postDate": "08/10/2019 21:43:18", "content": "

Is QM7 allowed? I don't think so.

", "rawMarkdown": "Is QM7 allowed? I don't think so.", "votes": null }, { "id": "596573", "postDate": "08/10/2019 22:01:20", "content": "

You can't map from QM7 to the original trainset anyway, so it doesn't matter.

", "rawMarkdown": "You can't map from QM7 to the original trainset anyway, so it doesn't matter.", "votes": null }, { "id": "596674", "postDate": "08/11/2019 05:16:28", "content": "

I had tested QM9 from the kernel for my local model. As other people have said in the forum, only a very little improvement in local cv . (<0.1)

\n\n

update: I use a tiny model to test the QM9 importance in my current model. with QM9 local cv:1.365, without QM9 local cv:1.336

", "rawMarkdown": "I had tested QM9 from the kernel for my local model. As other people have said in the forum, only a very little improvement in local cv . (<0.1)\n\nupdate: I use a tiny model to test the QM9 importance in my current model. with QM9 local cv:1.365, without QM9 local cv:1.336", "votes": null }, { "id": "602104", "postDate": "08/18/2019 15:40:13", "content": "

type |ln_MAE |\n1JHC |-0.528|\n1JHN |-1.08|\n2JHC |-1.40|\n2JHH |-2.02|\n2JHN |-1.93|\n3JHC |-1.21|\n3JHH |-1.84|\n3JHN |-2.13|

\n\n

CV ln_MAE -1.52\nLB ln_MAE -1.669\nI'm training separate models for each coupling type. I'm using 100 features for each, selected by lightgbm importance from a 600+ feature pool.

", "rawMarkdown": "type |ln_MAE |\n1JHC |-0.528|\n1JHN |-1.08|\n2JHC |-1.40|\n2JHH |-2.02|\n2JHN |-1.93|\n3JHC |-1.21|\n3JHH |-1.84|\n3JHN |-2.13|\n\nCV ln_MAE -1.52\nLB ln_MAE -1.669\nI'm training separate models for each coupling type. I'm using 100 features for each, selected by lightgbm importance from a 600+ feature pool.", "votes": null }, { "id": "606990", "postDate": "08/24/2019 12:13:43", "content": "

LGB\n1JHC -0.704\n1JHN -1.285\n2JHC -1.455\n2JHH -2.180\n2JHN -1.976\n3JHC -1.415\n3JHH -2.011\n3JHN -2.187

\n\n

LB = -1.801

", "rawMarkdown": "LGB\n1JHC -0.704\n1JHN -1.285\n2JHC -1.455\n2JHH -2.180\n2JHN -1.976\n3JHC -1.415\n3JHH -2.011\n3JHN -2.187\n\nLB = -1.801", "votes": null } ], "comments": [ { "id": 567678, "author_name": "alijs1", "author_url": "", "post_date": "07/03/2019 20:39:17", "content": "

Currently for 1JHC I have 1.022 MAE, which corresponds to -1.116 LB score.

", "votes": null, "replies": [ { "id": 570741, "author_name": "cpmpml", "author_url": "", "post_date": "07/08/2019 18:14:34", "content": "

Your two numbers are not consistent ;)

", "votes": null, "replies": [] }, { "id": 570758, "author_name": "alijs1", "author_url": "", "post_date": "07/08/2019 18:32:26", "content": "

Why not consistent? Note, that my CV score is MAE, not Log MAE (because in original post MAE was given). Maybe that makes them to look not consistent.

\n\n

Or you mean they are not consistent with my current score? Then I admit, I spoiled consistency by improving my model a bit :D

", "votes": null, "replies": [] }, { "id": 570768, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/08/2019 18:41:27", "content": "

I think there's misunderstanding. 1.0 MAE is for 1 type, and LB is overall. Maybe @cpmpml thougth about MAE and logMAE for 1 type

", "votes": null, "replies": [] }, { "id": 571404, "author_name": "cpmpml", "author_url": "", "post_date": "07/09/2019 15:06:36", "content": "

Tx Stanislav, I thought the two numbers were for the same data.

", "votes": null, "replies": [] }, { "id": 571430, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/09/2019 15:27:18", "content": "

Ma pleasure, Mr. Uncle!

", "votes": null, "replies": [] } ] }, { "id": 567688, "author_name": "sahil711", "author_url": "", "post_date": "07/03/2019 20:48:14", "content": "

CV for every type\nType Log MAE\n1JHC -0.49\n2JHH -1.83\n1JHN -1.19\n2JHN -1.85\n2JHC -1.38\n3JHH -1.82\n3JHC -1.31\n3JHN -2.1\nCV -1.49625\nLB -.1.6

", "votes": null, "replies": [] }, { "id": 567971, "author_name": "kainsama", "author_url": "", "post_date": "07/04/2019 07:47:14", "content": "

1JHC CV MAE = 0.85 (5 fold)

", "votes": null, "replies": [] }, { "id": 568271, "author_name": "allvor", "author_url": "", "post_date": "07/04/2019 16:10:08", "content": "

My 1JHC MAE: 0.89

", "votes": null, "replies": [] }, { "id": 568284, "author_name": "jmtest", "author_url": "", "post_date": "07/04/2019 16:35:32", "content": "

1JHC 0,8 MAE \nI ‘m not a chemist but i don’t understand why 1JHC is the hardest? The atoms are just close. At the beginning, I thought i would be the easier! 😂

", "votes": null, "replies": [ { "id": 568396, "author_name": "phucco", "author_url": "", "post_date": "07/04/2019 21:02:16", "content": "

The major factors for one-bond
\nC-H couplings are substituent effects, hybridization, and ring strain.

\n\n

Other coupling types might be more affected by geometric properties, which are easier to generate features from.

", "votes": null, "replies": [] }, { "id": 570209, "author_name": "andcastillo", "author_url": "", "post_date": "07/08/2019 02:19:40", "content": "

On the other hand, many of the hydrogen change the angle constantly. Actually a lot of the relative positioning of the atoms change along the time. What you observe experimentally is the average coupling constant for all the possible conformations of the molecule. I don't really know how the conformers of this dataset was chosen, but up to I know, some of them are actually quite 'exotic' in the nature. I've found many examples in this dataset, where you obtain very different coupling constants where in practice you must observe exactly the same one, because the atoms are magnetically equivalent. I don't know if this could affect more thr 1JCH coupling constants

", "votes": null, "replies": [] } ] }, { "id": 568352, "author_name": "somnambwl", "author_url": "", "post_date": "07/04/2019 18:54:30", "content": "

For 1JHC my MAE is 0.62.\nI have spent the last three days creating new features which for some reason improved CV score a lot (to -1.635), but LB score got actually worse (to -1.095). Now, who knows whether to keep them or not :D

", "votes": null, "replies": [ { "id": 578551, "author_name": "asauve", "author_url": "", "post_date": "07/17/2019 21:01:33", "content": "

this happens when you optimize hyper parameters against the train set. I've got bitten by the same beast. To overcome this you should keep one fold out of your cv as a pre-test set to check if hyper optimization has not overfit the train set...

", "votes": null, "replies": [] } ] }, { "id": 568416, "author_name": "scaomath", "author_url": "", "post_date": "07/04/2019 22:19:27", "content": "

| Fold | LGB | GNN |\n|------------|------|------|\n| Group | 0.67 | 1.12 |\n| Stratified | 0.62 | 0.96 |

\n\n

My stratified fold is based off of a preliminary k-means clustering using some topological features.

\n\n

Update as of Jul 20: after some serious efforts in generating new features (reading papers in QM, etc). My best model's 1JHC's CV MAE is now 0.39 using GroupKFold.

", "votes": null, "replies": [] }, { "id": 568427, "author_name": "titericz", "author_url": "", "post_date": "07/04/2019 23:09:11", "content": "

LGB CV: -1.5733 / LB: -1.637 :\n| type | log | mae |\n| --- | --- | --- |\n| 1JHC | -0.5620 | 0.5700 |\n| 2JHH | -2.0817 | 0.1247 |\n| 1JHN | -1.3176 | 0.2677 |\n| 2JHN | -1.9128 | 0.1476 |\n| 2JHC | -1.3932 | 0.2482 |\n| 3JHH | -1.8386 | 0.1590 |\n| 3JHC | -1.3066 | 0.2707 |\n| 3JHN | -2.1737 | 0.1137 |

", "votes": null, "replies": [ { "id": 568536, "author_name": "hadxu123", "author_url": "", "post_date": "07/05/2019 04:39:14", "content": "

how many features?

", "votes": null, "replies": [] }, { "id": 570043, "author_name": "titericz", "author_url": "", "post_date": "07/07/2019 17:52:06", "content": "

~60

", "votes": null, "replies": [] }, { "id": 570059, "author_name": "yassinealouini", "author_url": "", "post_date": "07/07/2019 18:12:41", "content": "

I guess you did some features selection? I am working on this and will update the score if I find something meaningful.

", "votes": null, "replies": [] } ] }, { "id": 568546, "author_name": "azacharia", "author_url": "", "post_date": "07/05/2019 04:58:04", "content": "

LGB: CV: -1.494 / LB: -1.430

\n\n

type | log | mae\n1JHC | -0.4713 | 0.6242\n1JHN | -1.0188 | 0.3610\n2JHC | -1.4327 | 0.2387\n2JHH | -2.0086 | 0.1342\n2JHN | -1.6206 | 0.1978\n3JHC | -1.3490 | 0.2595\n3JHH | -1.9725 | 0.1391\n3JHN | -2.0821 | 0.1247

", "votes": null, "replies": [ { "id": 570057, "author_name": "yassinealouini", "author_url": "", "post_date": "07/07/2019 18:11:35", "content": "

Minor suggestion (since I had the same issue): if you want the table to be correctly formatted, you need to add a | at the end of each row. :)

", "votes": null, "replies": [] } ] }, { "id": 568761, "author_name": "marcogorelli", "author_url": "", "post_date": "07/05/2019 12:04:14", "content": "

index | CV mean | std_err\n---- | ---- | ----\n1jhn-lgbm | -0.67 | 0.0044\n3jhn-lgbm | -1.8 | 0.0035\n2jhn-lgbm | -1.4 | 0.0039\n3jhc-lgbm | -0.47 | 0.0013\n2jhc-lgbm | -0.54 | 0.0013\n1jhc-lgbm | 0.14 | 0.0025\n3jhh-lgbm | -1.2 | 0.0021\n2jhh-lgbm | -1.2 | 0.0022

\n\n

Function to convert pandas to markdown: https://stackoverflow.com/a/39334770/10743932

", "votes": null, "replies": [] }, { "id": 569110, "author_name": "kingychiu", "author_url": "", "post_date": "07/06/2019 04:52:57", "content": "

| type | cv |\n| --- | --- |\n| 1JHC | -0.10803 |\n| 2JHH | -1.55402|\n| 1JHN | -0.55320 |\n| 2JHN | -0.96341 |\n| 2JHC | -0.85063 |\n| 3JHH | -1.43886 |\n| 3JHC | -0.72023 |\n| 3JHN | -1.48864 |\n|OOF CV | -0.95963 |\n|Public LB|-1.510|

", "votes": null, "replies": [ { "id": 569113, "author_name": "kingychiu", "author_url": "", "post_date": "07/06/2019 05:02:47", "content": "

It looks like there is a difference between the out of fold CV and the average CV across types.

", "votes": null, "replies": [] } ] }, { "id": 570002, "author_name": "yassinealouini", "author_url": "", "post_date": "07/07/2019 16:47:15", "content": "

The other scores so far (3 folds GroupKFold):

\n\n

|type|MAE|LMAE|\n|-----|-----|----|\n|1JHC| 0.829 |-0.1875|\n|2JHH|0.1976|-1.622|\n|1JHN|0.5079|-0.6775|\n|2JHN|0.2631|-1.335|\n|2JHC|0.3849|-0.9547|\n|3JHH|0.2331|-1.4568|\n|3JHC|0.4533|-0.7912\n|3JHN|0.1899|-1.661

\n\n

-1.215 LB <---> -1.0856 CV (0.4867 std)

", "votes": null, "replies": [] }, { "id": 570128, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/07/2019 20:36:12", "content": "

On damnable ton of features. Somehow my LB is lower than CV, that's very strange.\ntype | log | mae |\n1JHC | -0.825 | 0.573 |\n1JHN | -1.650 | 0.296 |\n2JHC | -1.273 | 0.308 |\n2JHH | -2.138 | 0.151 |\n2JHN | -1.992 | 0.193 |\n3JHC | -1.841 | 0.433 |\n3JHH | -1.816 | 0.212 |\n3JHN | -2.32 | 0.157 |

", "votes": null, "replies": [ { "id": 570133, "author_name": "snovik1975", "author_url": "", "post_date": "07/07/2019 20:59:10", "content": "

that is exactly how many? :)

", "votes": null, "replies": [] }, { "id": 570340, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/08/2019 06:50:44", "content": "

~500

", "votes": null, "replies": [] }, { "id": 571285, "author_name": "chriskorn", "author_url": "", "post_date": "07/09/2019 12:58:37", "content": "

Something seems off there - log(0.573) is not -0.825.

", "votes": null, "replies": [] }, { "id": 571390, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/09/2019 14:54:27", "content": "

@chriskorn .573 is val MAE. Train MAE is less

", "votes": null, "replies": [] }, { "id": 571397, "author_name": "chriskorn", "author_url": "", "post_date": "07/09/2019 14:58:19", "content": "

Ah I see - sorry, that hadn't been clear to me before.

", "votes": null, "replies": [] } ] }, { "id": 570271, "author_name": "anjum48", "author_url": "", "post_date": "07/08/2019 04:29:29", "content": "

Here's mine so far using LGBM with default parameters (as others have mentioned, there are probably significant gains from using different parameters each of the xJHy types, but haven't tested this yet):\n| type | LMAE | MAE |\n| --- | --- | --- |\n| 1JHC | -0.6615 | 0.5161 |\n| 1JHN | -1.2136 | 0.2971 |\n| 2JHH | -1.9525 | 0.1429 |\n| 2JHC | -1.4936 | 0.2246 |\n| 2JHN | -1.7773 | 0.1691 |\n| 3JHH | -1.9407 | 0.1436 |\n| 3JHC | -1.419 | 0.2420 |\n| 3JHN | -2.0771| 0.1253 |

\n\n

Final CV: -1.5669, Public LB: -1.712

\n\n

I've just got back into this competition after a couple of weeks holiday though so got a bit of catching up to do!

", "votes": null, "replies": [ { "id": 572137, "author_name": "littlebeandog", "author_url": "", "post_date": "07/10/2019 14:21:36", "content": "

This is quit good score without tunning.\nMay I ask how many features you used?

", "votes": null, "replies": [] }, { "id": 572393, "author_name": "yassinealouini", "author_url": "", "post_date": "07/10/2019 21:29:17", "content": "

Yes, same question as well. :)

", "votes": null, "replies": [] }, { "id": 572525, "author_name": "anjum48", "author_url": "", "post_date": "07/11/2019 03:18:02", "content": "

I have 790 in total at the moment but many of these are useless features which I haven't removed yet, and some of them are not applicable for different types

", "votes": null, "replies": [] }, { "id": 580776, "author_name": "waylongo", "author_url": "", "post_date": "07/20/2019 18:35:34", "content": "

@anjum48 How do you run you program with 790 features? I generated 500+ features but the program always crashs because of memory overflow (64g memory). Thanks

", "votes": null, "replies": [] }, { "id": 580805, "author_name": "somnambwl", "author_url": "", "post_date": "07/20/2019 19:41:53", "content": "

Maybe he has a bigger memory :D Have you tried reducing the memory with the script? You don't need float64 for everything. Also e. g. molecule names take less memory in DataFrame if you store them as ints.

", "votes": null, "replies": [] }, { "id": 580937, "author_name": "anjum48", "author_url": "", "post_date": "07/21/2019 05:23:59", "content": "

@waylongo At the moment my memory usage is quite a bit below 64GB. As @somnambwl mentioned if you use the appropriate data types you can get some major memory savings. Also, see if you can find ways of manipulating your data without needing to copy it, for example, a lot of the sklearn.preprocessing methods have a copy option which can prevent a memory spike when you call those methods (there are similar efficiencies to be found in pandas and numpy using in-place operations).

\n\n

My old laptop only had 4GB of RAM. Although it annoyed me at the time, it really taught me how to be careful around memory usage and I learned things I wouldn't have learnt if I had a more powerful machine.

", "votes": null, "replies": [] } ] }, { "id": 571473, "author_name": "pcjimmmy", "author_url": "", "post_date": "07/09/2019 16:17:02", "content": "

1JHC consistent weakest score regards of my model. Not time yet, but pretty soon its on my plan schedule for exclusive use of my time :)

\n\n

I

", "votes": null, "replies": [] }, { "id": 572387, "author_name": "returnofsputnik", "author_url": "", "post_date": "07/10/2019 21:19:16", "content": "

Hey guys what is your training score compared to your validation score? i have training score like 0.02 but validation at like 0.22

", "votes": null, "replies": [ { "id": 572392, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/10/2019 21:26:22", "content": "

Same situation, but my val score is worse than yours for 1jhc. I racked my brain trying to deal with this type, but no luck so far. \nMind to drop some knowledge and give a hint? ;)

", "votes": null, "replies": [] }, { "id": 572395, "author_name": "returnofsputnik", "author_url": "", "post_date": "07/10/2019 21:37:46", "content": "

actually, i am only dealing with 3jhh at the moment. currently at 0.22 mae with 3jhh

", "votes": null, "replies": [] }, { "id": 572622, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/11/2019 07:25:43", "content": "

Oh I thought you were talking about 1jhc.\nTho for 3jhh I have similar results: train 0.03 and val 0.2

", "votes": null, "replies": [] } ] }, { "id": 573533, "author_name": "xincuimath", "author_url": "", "post_date": "07/12/2019 11:34:50", "content": "

My current submission is based on 1JHC 0.89. Without using Giba's feature, I'm able to reduce it to (valid_1's l1: 0.615775). Now I'm convinced that we could reduce it to around 0.5.

", "votes": null, "replies": [] }, { "id": 578573, "author_name": "wjholst", "author_url": "", "post_date": "07/17/2019 21:58:01", "content": "

0.587 MAE for 1JHC - LGBM with no parameter tuning. Around 200 features.\nUpdate 7/23 - 0.486 for 1JHC. \nUpdate 728 - 0.5133 for 1JHC, but now -1.689 LB with some re-design of feature selection and CV metrics.

", "votes": null, "replies": [] }, { "id": 580984, "author_name": "ykskks", "author_url": "", "post_date": "07/21/2019 07:49:07", "content": "

Hello, everyone. Currently, my validation score(using 20% of train) is -1.55 while my LB score is -1.65.

\n\n

Looking through this thread, most people have much lower error(thus better score) for LB compared to CV. Does anyone have any idea why this is the case?

", "votes": null, "replies": [ { "id": 580987, "author_name": "yassinealouini", "author_url": "", "post_date": "07/21/2019 07:58:44", "content": "

It might be that the public part of the dataset has more (in proportion) easier bond types to predict (2JHH, 3JHN, and 3JHH for example). Thus, you should trust your local CV.

", "votes": null, "replies": [] }, { "id": 580991, "author_name": "cpmpml", "author_url": "", "post_date": "07/21/2019 08:04:30", "content": "

The explanation must be different because the score does not depend on proportion of each type....

", "votes": null, "replies": [] }, { "id": 581003, "author_name": "yassinealouini", "author_url": "", "post_date": "07/21/2019 08:29:30", "content": "

Good point indeed. The difference could be explained if some bond types aren't there at all in the public part. I guess that this claim could be verified by sending 8 submissions with constant values for one of the types each time and checking if this gives the same result each time.

", "votes": null, "replies": [] }, { "id": 581056, "author_name": "helgith", "author_url": "", "post_date": "07/21/2019 10:32:58", "content": "

When you use, as an example, 5 fold CV, then the results on the OOF is based on a single model for each validation set, but the LB result is based on the average of 5 models (in case you average the models and do not rerun for the whole training set). This could explain why the LB result is better.

", "votes": null, "replies": [] }, { "id": 581854, "author_name": "cpmpml", "author_url": "", "post_date": "07/22/2019 13:24:03", "content": "

@helgi, you are probably right, with one fold only I have a LB of -2.001 for a val score of -2.017, I don't see any LB improvement.

", "votes": null, "replies": [] } ] }, { "id": 581857, "author_name": "cpmpml", "author_url": "", "post_date": "07/22/2019 13:26:00", "content": "

one fold, lgb, validation score:

\n\n

1JHC -0.777\n1JHN -1.233\n2JHC -1.934\n2JHH -2.454\n2JHN -2.442\n3JHC -1.992\n3JHH -2.613\n3JHN -2.689

\n\n

total -2.017\nLB -2.001

\n\n

It seems averaging several folds explains why LB is better than CV for many.

\n\n

Despite my team name now (lgb, what else) I don't think lgb is competitive here, will move to graph NNs now.

", "votes": null, "replies": [ { "id": 581897, "author_name": "cpmpml", "author_url": "", "post_date": "07/22/2019 14:09:13", "content": "

I started with lgb because the feature engineering made me understand some of the data and what matters.

", "votes": null, "replies": [] }, { "id": 581965, "author_name": "yoichi7yamakawa", "author_url": "", "post_date": "07/22/2019 15:34:48", "content": "

Amazing scores by 1 fold CV !\nMaybe there’re strong features I couldn’t find yet...\nAre you using any external dataset (ex. QM7/9)?

", "votes": null, "replies": [] }, { "id": 581974, "author_name": "cpmpml", "author_url": "", "post_date": "07/22/2019 15:47:04", "content": "

No external data.

", "votes": null, "replies": [] }, { "id": 581977, "author_name": "yoichi7yamakawa", "author_url": "", "post_date": "07/22/2019 15:51:46", "content": "

OK, thank you

", "votes": null, "replies": [] }, { "id": 581988, "author_name": "cpmpml", "author_url": "", "post_date": "07/22/2019 16:11:30", "content": "

Thanks. Are you using lgb too?

", "votes": null, "replies": [] }, { "id": 581998, "author_name": "yoichi7yamakawa", "author_url": "", "post_date": "07/22/2019 16:24:22", "content": "

Yes, we’re using lgb.

", "votes": null, "replies": [] }, { "id": 582010, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/22/2019 16:49:49", "content": "

Wow that's one of a mighty first submission... Good job!\nHow are you doing this?! :D

", "votes": null, "replies": [] }, { "id": 582059, "author_name": "cpmpml", "author_url": "", "post_date": "07/22/2019 17:54:48", "content": "
\n

How are you doing this?

\n
\n\n

Read all the forum, and crafted my features. I didn't reuse any of the public kernel features, but I may have recoded some of them without knowing. Then lots of experiments...

", "votes": null, "replies": [] }, { "id": 582061, "author_name": "cpmpml", "author_url": "", "post_date": "07/22/2019 17:56:07", "content": "
\n

Wow that's one of a mighty first submission… Good job!

\n
\n\n

Thanks, but others have a much better 1st sub...

", "votes": null, "replies": [] }, { "id": 582158, "author_name": "stanislavblinov", "author_url": "", "post_date": "07/22/2019 20:17:45", "content": "

It was kinda rhetorical question :) Of course you won't reveal the tricks for now. \nDid a ton of experiments myself, but nothing worked so far. Guess my gut is just not as skilled as yours...

\n\n
\n

others have a much better 1st sub

\n
\n\n

Trying to figure out how to become this \"others\" for somw time already

", "votes": null, "replies": [] }, { "id": 582163, "author_name": "yassinealouini", "author_url": "", "post_date": "07/22/2019 20:31:58", "content": "

That's awesome. 👍

", "votes": null, "replies": [] } ] }, { "id": 583026, "author_name": "returnofsputnik", "author_url": "", "post_date": "07/23/2019 23:05:35", "content": "

an update from me: i have 1JHC mae cv of 0.498577 using lightgbm. Using neural network, i have like 1.20 CV which is pretty bad in comparison. hoping to improve my NN performance so that it will work well in ensemble

\n\n

Edit: July 24 2019: new CV is 0.487761

", "votes": null, "replies": [] }, { "id": 584561, "author_name": "sergeyzlobin", "author_url": "", "post_date": "07/26/2019 06:55:50", "content": "

My current 5 folds CV, LGBM:\n1JHC -0.539175\n1JHN -1.131305\n2JHC -1.445589\n2JHH -1.974381\n2JHN -1.795792\n3JHC -1.481445\n3JHH -1.899323\n3JHN -2.111573\nmean -1.5473228144612587\nLB=-1.842

", "votes": null, "replies": [] }, { "id": 584665, "author_name": "ihorshylo", "author_url": "", "post_date": "07/26/2019 09:25:24", "content": "

type | Log MAE |\n1JHC | -0.514 |\n2JHH | -1.845 |\n1JHN | -1.043 |\n2JHN | -1.750 |\n2JHC | -1.389 |\n3JHH | -1.814 |\n3JHC | -1.2818 |\n3JHN | -2.058 |

\n\n

CV 1.46, LB 1.53

", "votes": null, "replies": [] }, { "id": 592943, "author_name": "senkin13", "author_url": "", "post_date": "08/06/2019 02:10:34", "content": "

5fold GNN

\n\n

1JHC -1.2735724333156546\n1JHN -1.880088909076347\n2JHC -2.2438082211992447\n2JHH -3.148106336806997\n2JHN -2.934080757996177\n3JHC -2.012966179536739\n3JHH -2.941703155183916\n3JHN -3.089508672452637

\n\n

CV -2.440\nLB -2.639

", "votes": null, "replies": [ { "id": 592951, "author_name": "robikscube", "author_url": "", "post_date": "08/06/2019 02:52:46", "content": "

what? how?\n\"\"

", "votes": null, "replies": [] }, { "id": 592998, "author_name": "cpmpml", "author_url": "", "post_date": "08/06/2019 05:07:42", "content": "

We should team as we have some types with way better score (and some with way worse) ;)

\n\n

Oops, we can't because of team size.

", "votes": null, "replies": [] }, { "id": 593033, "author_name": "senkin13", "author_url": "", "post_date": "08/06/2019 05:43:43", "content": "

We don't have domain knowledge so I think top rankers are doing very different solution from us ,we expected to team up.\n@cpmpml Hope to team with you one day,this time good luck to you.

", "votes": null, "replies": [] }, { "id": 593048, "author_name": "cpmpml", "author_url": "", "post_date": "08/06/2019 06:02:36", "content": "

@senkin13 Sure, I was thinking about this since Talking Data...

\n\n

We don't have domain knowledge either :(

", "votes": null, "replies": [] }, { "id": 593076, "author_name": "senkin13", "author_url": "", "post_date": "08/06/2019 06:57:08", "content": "

@cpmpml Yes me too ,I hope no shakedown like TalkingData 😄

", "votes": null, "replies": [] }, { "id": 593373, "author_name": "khahuras", "author_url": "", "post_date": "08/06/2019 14:06:54", "content": "

@senkin13 Thank you for your kind information. If you don't mind, may I ask did you train all molecules at once, or you pick only molecules with the associated type you want to separately train?

", "votes": null, "replies": [] }, { "id": 593683, "author_name": "senkin13", "author_url": "", "post_date": "08/07/2019 01:21:25", "content": "

We use both,good luck.

", "votes": null, "replies": [] } ] }, { "id": 595645, "author_name": "h4211819", "author_url": "", "post_date": "08/09/2019 13:56:12", "content": "

5fold lgb

\n\n

1JHC -0.67306731\n1JHN -1.24935432\n2JHC -1.60763475\n2JHH -2.25585111\n2JHN -2.05962611\n3JHC -1.58876169\n3JHH -2.0570262\n3JHN -2.30307408

\n\n

CV -1.724299\nLB -1.981

\n\n

all features are downloaded from public kernels , I didn't created any other features because the kernels have realized all my ideas

", "votes": null, "replies": [ { "id": 595711, "author_name": "returnofsputnik", "author_url": "", "post_date": "08/09/2019 15:11:47", "content": "

What are the top features for 2jhc and 3jhc?

", "votes": null, "replies": [] }, { "id": 596452, "author_name": "yoichi7yamakawa", "author_url": "", "post_date": "08/10/2019 16:48:16", "content": "

Impressive, are you using features derived from QM7/9?

", "votes": null, "replies": [] }, { "id": 596560, "author_name": "scaomath", "author_url": "", "post_date": "08/10/2019 21:06:57", "content": "

Essentially speaking, charge weighted distance features are QM7 features.

", "votes": null, "replies": [] }, { "id": 596571, "author_name": "cpmpml", "author_url": "", "post_date": "08/10/2019 21:43:18", "content": "

Is QM7 allowed? I don't think so.

", "votes": null, "replies": [] }, { "id": 596573, "author_name": "returnofsputnik", "author_url": "", "post_date": "08/10/2019 22:01:20", "content": "

You can't map from QM7 to the original trainset anyway, so it doesn't matter.

", "votes": null, "replies": [] }, { "id": 596674, "author_name": "h4211819", "author_url": "", "post_date": "08/11/2019 05:16:28", "content": "

I had tested QM9 from the kernel for my local model. As other people have said in the forum, only a very little improvement in local cv . (<0.1)

\n\n

update: I use a tiny model to test the QM9 importance in my current model. with QM9 local cv:1.365, without QM9 local cv:1.336

", "votes": null, "replies": [] } ] }, { "id": 602104, "author_name": "andyatkinson", "author_url": "", "post_date": "08/18/2019 15:40:13", "content": "

type |ln_MAE |\n1JHC |-0.528|\n1JHN |-1.08|\n2JHC |-1.40|\n2JHH |-2.02|\n2JHN |-1.93|\n3JHC |-1.21|\n3JHH |-1.84|\n3JHN |-2.13|

\n\n

CV ln_MAE -1.52\nLB ln_MAE -1.669\nI'm training separate models for each coupling type. I'm using 100 features for each, selected by lightgbm importance from a 600+ feature pool.

", "votes": null, "replies": [] }, { "id": 606990, "author_name": "vadymsg", "author_url": "", "post_date": "08/24/2019 12:13:43", "content": "

LGB\n1JHC -0.704\n1JHN -1.285\n2JHC -1.455\n2JHH -2.180\n2JHN -1.976\n3JHC -1.415\n3JHH -2.011\n3JHN -2.187

\n\n

LB = -1.801

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "567660": "Out of curiosity, what is your best 1JHC MAE CV score?\nSo far, I have managed to get it to around **0.96 on 3 folds** without hp optimization. \n\n**UPDATE** 7-7-2019: Thanks to Giba's great [kernel](https://www.kaggle.com/titericz/giba-r-data-table-simple-features-1-17-lb?scriptVersionId=16835193) and some hp optimization (manual so far), I have managed to decrease the MAE to around **0.82 on 3 folds**. \n\n**UPDATE** 15-7-2019: More features, MAE around **0.68 on 5 folds**.", "567678": "Currently for 1JHC I have 1.022 MAE, which corresponds to -1.116 LB score.", "567688": "CV for every type\nType\tLog MAE\n1JHC\t-0.49\n2JHH\t-1.83\n1JHN\t-1.19\n2JHN\t-1.85\n2JHC\t-1.38\n3JHH\t-1.82\n3JHC\t-1.31\n3JHN\t-2.1\nCV\t-1.49625\nLB -.1.6", "567971": "1JHC CV MAE = 0.85 (5 fold)", "568271": "My 1JHC MAE: 0.89", "568284": "1JHC 0,8 MAE \nI ‘m not a chemist but i don’t understand why 1JHC is the hardest? The atoms are just close. At the beginning, I thought i would be the easier! 😂", "568352": "For 1JHC my MAE is 0.62.\nI have spent the last three days creating new features which for some reason improved CV score a lot (to -1.635), but LB score got actually worse (to -1.095). Now, who knows whether to keep them or not :D", "568396": "The major factors for one-bond \nC-H couplings are substituent effects, hybridization, and ring strain.\n\nOther coupling types might be more affected by geometric properties, which are easier to generate features from.", "568416": "| Fold | LGB | GNN |\n|------------|------|------|\n| Group | 0.67 | 1.12 |\n| Stratified | 0.62 | 0.96 |\n\nMy stratified fold is based off of a preliminary k-means clustering using some topological features.\n\nUpdate as of Jul 20: after some serious efforts in generating new features (reading papers in QM, etc). My best model's 1JHC's CV MAE is now 0.39 using GroupKFold.", "568427": "LGB CV: -1.5733 / LB: -1.637 :\n| type | log | mae |\n| --- | --- | --- |\n| 1JHC | -0.5620 | 0.5700 |\n| 2JHH | -2.0817 | 0.1247 |\n| 1JHN | -1.3176 | 0.2677 |\n| 2JHN | -1.9128 | 0.1476 |\n| 2JHC | -1.3932 | 0.2482 |\n| 3JHH | -1.8386 | 0.1590 |\n| 3JHC | -1.3066 | 0.2707 |\n| 3JHN | -2.1737 | 0.1137 |", "568536": "how many features?", "568546": "LGB: CV: -1.494 / LB: -1.430\n\ntype | log\t | mae\n1JHC | -0.4713 | 0.6242\n1JHN | -1.0188 | 0.3610\n2JHC | -1.4327 | 0.2387\n2JHH | -2.0086 | 0.1342\n2JHN | -1.6206 | 0.1978\n3JHC | -1.3490 | 0.2595\n3JHH | -1.9725 | 0.1391\n3JHN | -2.0821 | 0.1247", "568761": "index | CV mean | std_err\n---- | ---- | ----\n1jhn-lgbm | -0.67 | 0.0044\n3jhn-lgbm | -1.8 | 0.0035\n2jhn-lgbm | -1.4 | 0.0039\n3jhc-lgbm | -0.47 | 0.0013\n2jhc-lgbm | -0.54 | 0.0013\n1jhc-lgbm | 0.14 | 0.0025\n3jhh-lgbm | -1.2 | 0.0021\n2jhh-lgbm | -1.2 | 0.0022\n\nFunction to convert pandas to markdown: https://stackoverflow.com/a/39334770/10743932", "569110": "| type | cv |\n| --- | --- |\n| 1JHC | -0.10803 |\n| 2JHH | -1.55402|\n| 1JHN | -0.55320 |\n| 2JHN | -0.96341 |\n| 2JHC | -0.85063 |\n| 3JHH | -1.43886 |\n| 3JHC | -0.72023 |\n| 3JHN | -1.48864 |\n|OOF CV | -0.95963 |\n|Public LB|-1.510|", "569113": "It looks like there is a difference between the out of fold CV and the average CV across types.", "570002": "The other scores so far (3 folds GroupKFold):\n\n|type|MAE|LMAE|\n|-----|-----|----|\n|1JHC| 0.829 |-0.1875|\n|2JHH|0.1976|-1.622|\n|1JHN|0.5079|-0.6775|\n|2JHN|0.2631|-1.335|\n|2JHC|0.3849|-0.9547|\n|3JHH|0.2331|-1.4568|\n|3JHC|0.4533|-0.7912\n|3JHN|0.1899|-1.661\n\n\n-1.215 LB <---> -1.0856 CV (0.4867 std)", "570043": "~60", "570057": "Minor suggestion (since I had the same issue): if you want the table to be correctly formatted, you need to add a `|` at the end of each row. :)", "570059": "I guess you did some features selection? I am working on this and will update the score if I find something meaningful.", "570128": "On damnable ton of features. Somehow my LB is lower than CV, that's very strange.\ntype | log | mae |\n1JHC | -0.825 | 0.573 |\n1JHN | -1.650 | 0.296 |\n2JHC | -1.273 | 0.308 |\n2JHH | -2.138 | 0.151 |\n2JHN | -1.992 | 0.193 |\n3JHC | -1.841 | 0.433 |\n3JHH | -1.816 | 0.212 |\n3JHN | -2.32 | 0.157 |", "570133": "that is exactly how many? :)", "570209": "On the other hand, many of the hydrogen change the angle constantly. Actually a lot of the relative positioning of the atoms change along the time. What you observe experimentally is the average coupling constant for all the possible conformations of the molecule. I don't really know how the conformers of this dataset was chosen, but up to I know, some of them are actually quite 'exotic' in the nature. I've found many examples in this dataset, where you obtain very different coupling constants where in practice you must observe exactly the same one, because the atoms are magnetically equivalent. I don't know if this could affect more thr 1JCH coupling constants", "570271": "Here's mine so far using LGBM with default parameters (as others have mentioned, there are probably significant gains from using different parameters each of the xJHy types, but haven't tested this yet):\n| type | LMAE | MAE |\n| --- | --- | --- |\n| 1JHC | -0.6615 | 0.5161 |\n| 1JHN | -1.2136 | 0.2971 |\n| 2JHH | -1.9525 | 0.1429 |\n| 2JHC | -1.4936 | 0.2246 |\n| 2JHN | -1.7773 | 0.1691 |\n| 3JHH | -1.9407 | 0.1436 |\n| 3JHC | -1.419 | 0.2420 |\n| 3JHN | -2.0771| 0.1253 |\n\nFinal CV: -1.5669, Public LB: -1.712\n\nI've just got back into this competition after a couple of weeks holiday though so got a bit of catching up to do!", "570340": "~500", "570741": "Your two numbers are not consistent ;)", "570758": "Why not consistent? Note, that my CV score is MAE, not Log MAE (because in original post MAE was given). Maybe that makes them to look not consistent.\n\nOr you mean they are not consistent with my current score? Then I admit, I spoiled consistency by improving my model a bit :D", "570768": "I think there's misunderstanding. 1.0 MAE is for 1 type, and LB is overall. Maybe @cpmpml thougth about MAE and logMAE for 1 type", "571285": "Something seems off there - log(0.573) is not -0.825.", "571390": "chriskorn .573 is val MAE. Train MAE is less", "571397": "Ah I see - sorry, that hadn't been clear to me before.", "571404": "Tx Stanislav, I thought the two numbers were for the same data.", "571430": "Ma pleasure, Mr. Uncle!", "571473": "1JHC consistent weakest score regards of my model. Not time yet, but pretty soon its on my plan schedule for exclusive use of my time :)\n\nI", "572137": "This is quit good score without tunning.\nMay I ask how many features you used?", "572387": "Hey guys what is your training score compared to your validation score? i have training score like 0.02 but validation at like 0.22", "572392": "Same situation, but my val score is worse than yours for 1jhc. I racked my brain trying to deal with this type, but no luck so far. \nMind to drop some knowledge and give a hint? ;)", "572393": "Yes, same question as well. :)", "572395": "actually, i am only dealing with 3jhh at the moment. currently at 0.22 mae with 3jhh", "572525": "I have 790 in total at the moment but many of these are useless features which I haven't removed yet, and some of them are not applicable for different types", "572622": "Oh I thought you were talking about 1jhc.\nTho for 3jhh I have similar results: train 0.03 and val 0.2", "573533": "My current submission is based on 1JHC 0.89. Without using Giba's feature, I'm able to reduce it to (valid_1's l1: 0.615775). Now I'm convinced that we could reduce it to around 0.5.", "578551": "this happens when you optimize hyper parameters against the train set. I've got bitten by the same beast. To overcome this you should keep one fold out of your cv as a pre-test set to check if hyper optimization has not overfit the train set...", "578573": "0.587 MAE for 1JHC - LGBM with no parameter tuning. Around 200 features.\nUpdate 7/23 - 0.486 for 1JHC. \nUpdate 728 - 0.5133 for 1JHC, but now -1.689 LB with some re-design of feature selection and CV metrics.", "580776": "anjum48 How do you run you program with 790 features? I generated 500+ features but the program always crashs because of memory overflow (64g memory). Thanks", "580805": "Maybe he has a bigger memory :D Have you tried reducing the memory with the script? You don't need float64 for everything. Also e. g. molecule names take less memory in DataFrame if you store them as ints.", "580937": "waylongo At the moment my memory usage is quite a bit below 64GB. As @somnambwl mentioned if you use the appropriate data types you can get some major memory savings. Also, see if you can find ways of manipulating your data without needing to copy it, for example, a lot of the `sklearn.preprocessing` methods have a `copy` option which can prevent a memory spike when you call those methods (there are similar efficiencies to be found in `pandas` and `numpy` using in-place operations).\n\nMy old laptop only had 4GB of RAM. Although it annoyed me at the time, it really taught me how to be careful around memory usage and I learned things I wouldn't have learnt if I had a more powerful machine.", "580984": "Hello, everyone. Currently, my validation score(using 20% of train) is -1.55 while my LB score is -1.65.\n\nLooking through this thread, most people have much lower error(thus better score) for LB compared to CV. Does anyone have any idea why this is the case?", "580987": "It might be that the public part of the dataset has **more** (in proportion) **easier bond types** to predict (2JHH, 3JHN, and 3JHH for example). Thus, you should trust your local CV.", "580991": "The explanation must be different because the score does not depend on proportion of each type....", "581003": "Good point indeed. The difference could be explained if some bond types aren't there at all in the public part. I guess that this claim could be verified by sending 8 submissions with constant values for one of the types each time and checking if this gives the same result each time.", "581056": "When you use, as an example, 5 fold CV, then the results on the OOF is based on a single model for each validation set, but the LB result is based on the average of 5 models (in case you average the models and do not rerun for the whole training set). This could explain why the LB result is better.", "581854": "helgi, you are probably right, with one fold only I have a LB of -2.001 for a val score of -2.017, I don't see any LB improvement.", "581857": "one fold, lgb, validation score:\n\n1JHC -0.777\n1JHN -1.233\n2JHC -1.934\n2JHH -2.454\n2JHN -2.442\n3JHC -1.992\n3JHH -2.613\n3JHN -2.689\n\ntotal -2.017\nLB -2.001\n\nIt seems averaging several folds explains why LB is better than CV for many.\n\nDespite my team name now (lgb, what else) I don't think lgb is competitive here, will move to graph NNs now.", "581897": "I started with lgb because the feature engineering made me understand some of the data and what matters.", "581965": "Amazing scores by 1 fold CV !\nMaybe there’re strong features I couldn’t find yet...\nAre you using any external dataset (ex. QM7/9)?", "581974": "No external data.", "581977": "OK, thank you", "581988": "Thanks. Are you using lgb too?", "581998": "Yes, we’re using lgb.", "582010": "Wow that's one of a mighty first submission... Good job!\nHow are you doing this?! :D", "582059": "> How are you doing this?\n\nRead all the forum, and crafted my features. I didn't reuse any of the public kernel features, but I may have recoded some of them without knowing. Then lots of experiments...", "582061": "> Wow that's one of a mighty first submission… Good job!\n\nThanks, but others have a much better 1st sub...", "582158": "It was kinda rhetorical question :) Of course you won't reveal the tricks for now. \nDid a ton of experiments myself, but nothing worked so far. Guess my gut is just not as skilled as yours...\n> others have a much better 1st sub\n\nTrying to figure out how to become this \"others\" for somw time already", "582163": "That's awesome. 👍", "583026": "an update from me: i have 1JHC mae cv of 0.498577 using lightgbm. Using neural network, i have like 1.20 CV which is pretty bad in comparison. hoping to improve my NN performance so that it will work well in ensemble\n\nEdit: July 24 2019: new CV is 0.487761", "584561": "My current 5 folds CV, LGBM:\n1JHC -0.539175\n1JHN -1.131305\n2JHC -1.445589\n2JHH -1.974381\n2JHN -1.795792\n3JHC -1.481445\n3JHH -1.899323\n3JHN -2.111573\nmean -1.5473228144612587\nLB=-1.842", "584665": "type | Log MAE |\n1JHC | -0.514 |\n2JHH | -1.845 |\n1JHN | -1.043 |\n2JHN | -1.750 |\n2JHC | -1.389 |\n3JHH | -1.814 |\n3JHC | -1.2818 |\n3JHN | -2.058 |\n\nCV 1.46, LB 1.53", "592943": "5fold GNN\n\n1JHC -1.2735724333156546\n1JHN -1.880088909076347\n2JHC -2.2438082211992447\n2JHH -3.148106336806997\n2JHN -2.934080757996177\n3JHC -2.012966179536739\n3JHH -2.941703155183916\n3JHN -3.089508672452637\n\nCV -2.440\nLB -2.639", "592951": "what? how?\n![](https://media.giphy.com/media/3o7btPCcdNniyf0ArS/giphy.gif)", "592998": "We should team as we have some types with way better score (and some with way worse) ;)\n\nOops, we can't because of team size.", "593033": "We don't have domain knowledge so I think top rankers are doing very different solution from us ,we expected to team up.\n@cpmpml Hope to team with you one day,this time good luck to you.", "593048": "senkin13 Sure, I was thinking about this since Talking Data...\n\nWe don't have domain knowledge either :(", "593076": "cpmpml Yes me too ,I hope no shakedown like TalkingData 😄", "593373": "senkin13 Thank you for your kind information. If you don't mind, may I ask did you train all molecules at once, or you pick only molecules with the associated type you want to separately train?", "593683": "We use both,good luck.", "595645": "5fold lgb\n\n1JHC -0.67306731\n1JHN -1.24935432\n2JHC -1.60763475\n2JHH -2.25585111\n2JHN -2.05962611\n3JHC -1.58876169\n3JHH -2.0570262\n3JHN -2.30307408\n\nCV -1.724299\nLB -1.981\n\nall features are downloaded from public kernels , I didn't created any other features because the kernels have realized all my ideas", "595711": "What are the top features for 2jhc and 3jhc?", "596452": "Impressive, are you using features derived from QM7/9?", "596560": "Essentially speaking, charge weighted distance features are QM7 features.", "596571": "Is QM7 allowed? I don't think so.", "596573": "You can't map from QM7 to the original trainset anyway, so it doesn't matter.", "596674": "I had tested QM9 from the kernel for my local model. As other people have said in the forum, only a very little improvement in local cv . (<0.1)\n\nupdate: I use a tiny model to test the QM9 importance in my current model. with QM9 local cv:1.365, without QM9 local cv:1.336", "602104": "type |ln_MAE |\n1JHC |-0.528|\n1JHN |-1.08|\n2JHC |-1.40|\n2JHH |-2.02|\n2JHN |-1.93|\n3JHC |-1.21|\n3JHH |-1.84|\n3JHN |-2.13|\n\nCV ln_MAE -1.52\nLB ln_MAE -1.669\nI'm training separate models for each coupling type. I'm using 100 features for each, selected by lightgbm importance from a 600+ feature pool.", "606990": "LGB\n1JHC -0.704\n1JHN -1.285\n2JHC -1.455\n2JHH -2.180\n2JHN -1.976\n3JHC -1.415\n3JHH -2.011\n3JHN -2.187\n\nLB = -1.801" }, "source": "meta" }