Hi Friends,
\n\nTill now I am using LGB and got CV -1.766 and LB -1.819 \nand I tried plain NN with the same features and got CV -1.83 and when I submit I got LB 1.185(plus value)\nI got good CV in NN but not able to get good LB. Is there any suggestions by experts? I want to know weather I did a mistake in NN usage or problem in features. Please suggest.
\n\nUpdate: Found the issue in preprocessing. After rectifying got -1.828 LB. Thanks for you suggestions.
", "messages": [ { "id": "606000", "postDate": "08/23/2019 04:31:25", "content": "Hi Friends,
\n\nTill now I am using LGB and got CV -1.766 and LB -1.819 \nand I tried plain NN with the same features and got CV -1.83 and when I submit I got LB 1.185(plus value)\nI got good CV in NN but not able to get good LB. Is there any suggestions by experts? I want to know weather I did a mistake in NN usage or problem in features. Please suggest.
\n\nUpdate: Found the issue in preprocessing. After rectifying got -1.828 LB. Thanks for you suggestions.
", "rawMarkdown": "Hi Friends,\n\nTill now I am using LGB and got CV -1.766 and LB -1.819 \nand I tried plain NN with the same features and got CV -1.83 and when I submit I got LB 1.185(plus value)\nI got good CV in NN but not able to get good LB. Is there any suggestions by experts? I want to know weather I did a mistake in NN usage or problem in features. Please suggest.\n\nUpdate: Found the issue in preprocessing. After rectifying got -1.828 LB. Thanks for you suggestions.", "votes": null }, { "id": "606024", "postDate": "08/23/2019 05:18:46", "content": "I would suspect an error in mapping NN output to submission file.
", "rawMarkdown": "I would suspect an error in mapping NN output to submission file.", "votes": null }, { "id": "606033", "postDate": "08/23/2019 05:34:11", "content": "Try to compare the two submission files (LGBM and NN) manually to see if all the outputs are bad. There might be a problem with only 1 coupling type.
", "rawMarkdown": "Try to compare the two submission files (LGBM and NN) manually to see if all the outputs are bad. There might be a problem with only 1 coupling type.", "votes": null }, { "id": "606064", "postDate": "08/23/2019 06:29:46", "content": "I have seen this too and I can say this is not an output mapping issue in my case. Some features that work well with lightgbm (or do they?) seem to make NN badly overfit.
", "rawMarkdown": "I have seen this too and I can say this is not an output mapping issue in my case. Some features that work well with lightgbm (or do they?) seem to make NN badly overfit.", "votes": null }, { "id": "606071", "postDate": "08/23/2019 06:44:47", "content": "Thank you @cpmpml. Will check.
", "rawMarkdown": "Thank you @cpmpml. Will check.", "votes": null }, { "id": "606074", "postDate": "08/23/2019 06:47:32", "content": "Thanks, I checked both submission files. and both are different(difference is high) and NN Mean also lower than LGB Mean.
", "rawMarkdown": "Thanks, I checked both submission files. and both are different(difference is high) and NN Mean also lower than LGB Mean.", "votes": null }, { "id": "606077", "postDate": "08/23/2019 06:48:14", "content": "Thanks @welcomeworld.
", "rawMarkdown": "Thanks @welcomeworld.", "votes": null }, { "id": "606135", "postDate": "08/23/2019 08:32:21", "content": "I had the problem looks like this one. But in my case it was just a mistake in test dataset preprocessing (I used the same features for LGB and NN but preprocessed it in different manners, so 2 feaures were mixed up). If you use pandas you can compare the names of the columns of inputs for model fitting and model prediction.
", "rawMarkdown": "I had the problem looks like this one. But in my case it was just a mistake in test dataset preprocessing (I used the same features for LGB and NN but preprocessed it in different manners, so 2 feaures were mixed up). If you use pandas you can compare the names of the columns of inputs for model fitting and model prediction.", "votes": null }, { "id": "606455", "postDate": "08/23/2019 15:42:05", "content": "It sounds like you preprocessed Xtrain but didn't preprocessor Xtest (or preprocessed it differently). Did you take your features from LGBM and standardize them? If you do, you cannot standardize Xtrain and Xtest separately. You must first concatenate them and standardize them together them split them back apart.
", "rawMarkdown": "It sounds like you preprocessed Xtrain but didn't preprocessor Xtest (or preprocessed it differently). Did you take your features from LGBM and standardize them? If you do, you cannot standardize Xtrain and Xtest separately. You must first concatenate them and standardize them together them split them back apart.", "votes": null }, { "id": "606648", "postDate": "08/23/2019 20:40:38", "content": "A simple sanity check can be:
\n\nThis is indeed a very common mistake.
", "rawMarkdown": "This is indeed a very common mistake.", "votes": null }, { "id": "606893", "postDate": "08/24/2019 09:06:51", "content": "It might be a good idea to check the dataset separation. When you divide validation data, you need to focus on molecular_name and divide by them rather than records in test.csv.
Check your NN submission file for any huge numbers (positive or negative). I was finding that for some molecules (e.g. water) that some features were not being activated and the NN would produce a handful of non-sensical values in both the test and OOF predictions
", "rawMarkdown": "Check your NN submission file for any huge numbers (positive or negative). I was finding that for some molecules (e.g. water) that some features were not being activated and the NN would produce a handful of non-sensical values in both the test and OOF predictions", "votes": null }, { "id": "607236", "postDate": "08/24/2019 21:29:55", "content": "Wouldn't this leak information of the test set to the train set?
", "rawMarkdown": "Wouldn't this leak information of the test set to the train set?", "votes": null }, { "id": "607904", "postDate": "08/26/2019 04:38:02", "content": "Thanks @cdeotte. I checked and found the issue in preprocessing.
", "rawMarkdown": "Thanks @cdeotte. I checked and found the issue in preprocessing.", "votes": null }, { "id": "608131", "postDate": "08/26/2019 11:38:39", "content": "@venkig Great, i'm glad you found the problem\n.
\n@jaimebadiola :
\n\n\nWouldn't this leak information of the test set to the train set?
\n
Leaking refers to transferring the test targets to train set. (That is obviously bad because then you would use the answers to predict the answers). Using information from test features isn't leaking. It is helpful and incorporated into most winning solutions at Kaggle. (Although people debate whether this is \"true data science\").
\n\nIf you wish to avoid using test features, then you standardize train and save the values of mean and std. Then let test = (test - train_mean) / train_std
\n\n\nUsing information from test features isn't leaking. It is helpful and incorporated into most winning solutions at Kaggle.
\n
This is one of the few things Kaggle teaches that is dangerous in real world machine learning. In real world you must build models that perform well on unseen data. The only way to check is to not use any test data information. In that case, one should fit a standard scaler on train data only, then apply it test data and future unseen data. But this is not relevant for Kaggle competitions, unfortunately.
", "rawMarkdown": "> Using information from test features isn't leaking. It is helpful and incorporated into most winning solutions at Kaggle.\n\nThis is one of the few things Kaggle teaches that is dangerous in real world machine learning. In real world you must build models that perform well on unseen data. The only way to check is to not use any test data information. In that case, one should fit a standard scaler on train data only, then apply it test data and future unseen data. But this is not relevant for Kaggle competitions, unfortunately.", "votes": null } ], "comments": [ { "id": 606024, "author_name": "cpmpml", "author_url": "", "post_date": "08/23/2019 05:18:46", "content": "I would suspect an error in mapping NN output to submission file.
", "votes": null, "replies": [ { "id": 606071, "author_name": "venkig", "author_url": "", "post_date": "08/23/2019 06:44:47", "content": "Thank you @cpmpml. Will check.
", "votes": null, "replies": [] } ] }, { "id": 606033, "author_name": "mankyfej", "author_url": "", "post_date": "08/23/2019 05:34:11", "content": "Try to compare the two submission files (LGBM and NN) manually to see if all the outputs are bad. There might be a problem with only 1 coupling type.
", "votes": null, "replies": [ { "id": 606074, "author_name": "venkig", "author_url": "", "post_date": "08/23/2019 06:47:32", "content": "Thanks, I checked both submission files. and both are different(difference is high) and NN Mean also lower than LGB Mean.
", "votes": null, "replies": [] } ] }, { "id": 606064, "author_name": "welcomeworld", "author_url": "", "post_date": "08/23/2019 06:29:46", "content": "I have seen this too and I can say this is not an output mapping issue in my case. Some features that work well with lightgbm (or do they?) seem to make NN badly overfit.
", "votes": null, "replies": [ { "id": 606077, "author_name": "venkig", "author_url": "", "post_date": "08/23/2019 06:48:14", "content": "Thanks @welcomeworld.
", "votes": null, "replies": [] } ] }, { "id": 606135, "author_name": "silantev", "author_url": "", "post_date": "08/23/2019 08:32:21", "content": "I had the problem looks like this one. But in my case it was just a mistake in test dataset preprocessing (I used the same features for LGB and NN but preprocessed it in different manners, so 2 feaures were mixed up). If you use pandas you can compare the names of the columns of inputs for model fitting and model prediction.
", "votes": null, "replies": [] }, { "id": 606455, "author_name": "cdeotte", "author_url": "", "post_date": "08/23/2019 15:42:05", "content": "It sounds like you preprocessed Xtrain but didn't preprocessor Xtest (or preprocessed it differently). Did you take your features from LGBM and standardize them? If you do, you cannot standardize Xtrain and Xtest separately. You must first concatenate them and standardize them together them split them back apart.
", "votes": null, "replies": [ { "id": 606812, "author_name": "cpmpml", "author_url": "", "post_date": "08/24/2019 05:52:05", "content": "This is indeed a very common mistake.
", "votes": null, "replies": [] }, { "id": 607236, "author_name": "jaimebadiola", "author_url": "", "post_date": "08/24/2019 21:29:55", "content": "Wouldn't this leak information of the test set to the train set?
", "votes": null, "replies": [] }, { "id": 607904, "author_name": "venkig", "author_url": "", "post_date": "08/26/2019 04:38:02", "content": "Thanks @cdeotte. I checked and found the issue in preprocessing.
", "votes": null, "replies": [] }, { "id": 608131, "author_name": "cdeotte", "author_url": "", "post_date": "08/26/2019 11:38:39", "content": "@venkig Great, i'm glad you found the problem\n.
\n@jaimebadiola :
\n\n\nWouldn't this leak information of the test set to the train set?
\n
Leaking refers to transferring the test targets to train set. (That is obviously bad because then you would use the answers to predict the answers). Using information from test features isn't leaking. It is helpful and incorporated into most winning solutions at Kaggle. (Although people debate whether this is \"true data science\").
\n\nIf you wish to avoid using test features, then you standardize train and save the values of mean and std. Then let test = (test - train_mean) / train_std
\n\n\nUsing information from test features isn't leaking. It is helpful and incorporated into most winning solutions at Kaggle.
\n
This is one of the few things Kaggle teaches that is dangerous in real world machine learning. In real world you must build models that perform well on unseen data. The only way to check is to not use any test data information. In that case, one should fit a standard scaler on train data only, then apply it test data and future unseen data. But this is not relevant for Kaggle competitions, unfortunately.
", "votes": null, "replies": [] } ] }, { "id": 606648, "author_name": "scaomath", "author_url": "", "post_date": "08/23/2019 20:40:38", "content": "A simple sanity check can be:
\n\nIt might be a good idea to check the dataset separation. When you divide validation data, you need to focus on molecular_name and divide by them rather than records in test.csv.
Check your NN submission file for any huge numbers (positive or negative). I was finding that for some molecules (e.g. water) that some features were not being activated and the NN would produce a handful of non-sensical values in both the test and OOF predictions
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "606000": "Hi Friends,\n\nTill now I am using LGB and got CV -1.766 and LB -1.819 \nand I tried plain NN with the same features and got CV -1.83 and when I submit I got LB 1.185(plus value)\nI got good CV in NN but not able to get good LB. Is there any suggestions by experts? I want to know weather I did a mistake in NN usage or problem in features. Please suggest.\n\nUpdate: Found the issue in preprocessing. After rectifying got -1.828 LB. Thanks for you suggestions.", "606024": "I would suspect an error in mapping NN output to submission file.", "606033": "Try to compare the two submission files (LGBM and NN) manually to see if all the outputs are bad. There might be a problem with only 1 coupling type.", "606064": "I have seen this too and I can say this is not an output mapping issue in my case. Some features that work well with lightgbm (or do they?) seem to make NN badly overfit.", "606071": "Thank you @cpmpml. Will check.", "606074": "Thanks, I checked both submission files. and both are different(difference is high) and NN Mean also lower than LGB Mean.", "606077": "Thanks @welcomeworld.", "606135": "I had the problem looks like this one. But in my case it was just a mistake in test dataset preprocessing (I used the same features for LGB and NN but preprocessed it in different manners, so 2 feaures were mixed up). If you use pandas you can compare the names of the columns of inputs for model fitting and model prediction.", "606455": "It sounds like you preprocessed Xtrain but didn't preprocessor Xtest (or preprocessed it differently). Did you take your features from LGBM and standardize them? If you do, you cannot standardize Xtrain and Xtest separately. You must first concatenate them and standardize them together them split them back apart.", "606648": "A simple sanity check can be:\n\n- compare the mean absolute difference between your new test predictions with your current best LB one.\n- compare the statistics of your test features with the train features.", "606812": "This is indeed a very common mistake.", "606893": "It might be a good idea to check the dataset separation. When you divide validation data, you need to focus on `molecular_name` and divide by them rather than records in `test.csv`.", "607059": "Check your NN submission file for any huge numbers (positive or negative). I was finding that for some molecules (e.g. water) that some features were not being activated and the NN would produce a handful of non-sensical values in both the test and OOF predictions", "607236": "Wouldn't this leak information of the test set to the train set?", "607904": "Thanks @cdeotte. I checked and found the issue in preprocessing.", "608131": "venkig Great, i'm glad you found the problem\n. \n@jaimebadiola :\n> Wouldn't this leak information of the test set to the train set?\n\nLeaking refers to transferring the test **targets** to train set. (That is obviously bad because then you would use the answers to predict the answers). Using information from test **features** isn't leaking. It is helpful and incorporated into most winning solutions at Kaggle. (Although people debate whether this is \"true data science\").\n\nIf you wish to avoid using test features, then you standardize train and save the values of mean and std. Then let `test = (test - train_mean) / train_std`", "608132": "> Using information from test features isn't leaking. It is helpful and incorporated into most winning solutions at Kaggle.\n\nThis is one of the few things Kaggle teaches that is dangerous in real world machine learning. In real world you must build models that perform well on unseen data. The only way to check is to not use any test data information. In that case, one should fit a standard scaler on train data only, then apply it test data and future unseen data. But this is not relevant for Kaggle competitions, unfortunately." }, "source": "meta" }