Dear Kagglers,\nI'm trying to implement neural network, 2 hidden layer, using Keras. But it seems to give too many False +ve when using Under sampling of majority class, and too many False -ve when using Over Sampling of minority class. And without sampling trick its simply predicting majority class.\nCan you give some advice?\nThank you
\n\nP.S. I know XGBoost is working fine, but I want to try Neural Net, as its a part of my final year project.
", "messages": [ { "id": "139018", "postDate": "10/12/2016 05:42:33", "content": "Dear Kagglers,\nI'm trying to implement neural network, 2 hidden layer, using Keras. But it seems to give too many False +ve when using Under sampling of majority class, and too many False -ve when using Over Sampling of minority class. And without sampling trick its simply predicting majority class.\nCan you give some advice?\nThank you
\n\nP.S. I know XGBoost is working fine, but I want to try Neural Net, as its a part of my final year project.
", "rawMarkdown": "Dear Kagglers,\r\nI'm trying to implement neural network, 2 hidden layer, using Keras. But it seems to give too many False +ve when using Under sampling of majority class, and too many False -ve when using Over Sampling of minority class. And without sampling trick its simply predicting majority class.\r\nCan you give some advice?\r\nThank you\r\n\r\nP.S. I know XGBoost is working fine, but I want to try Neural Net, as its a part of my final year project.", "votes": null }, { "id": "139021", "postDate": "10/12/2016 06:07:19", "content": "Hi, I would be very interested in some suggestions as well. Here's a link to the experiments I've tried so far: https://medium.com/@dsouza.amanda/learning-from-learning-curves-1a82c6f98f49#.vctunbm7n
\n\nDataset contains all of +ve samples, and undersampled -ve classes to make it a bit balanced. Stratified sampling is used on CV with 3 folds. Code in Python/Keras.
", "rawMarkdown": "Hi, I would be very interested in some suggestions as well. Here's a link to the experiments I've tried so far: https://medium.com/@dsouza.amanda/learning-from-learning-curves-1a82c6f98f49#.vctunbm7n\r\n\r\nDataset contains all of +ve samples, and undersampled -ve classes to make it a bit balanced. Stratified sampling is used on CV with 3 folds. Code in Python/Keras.", "votes": null }, { "id": "139022", "postDate": "10/12/2016 06:15:10", "content": "@Amanda Dsouza , are you maintaining the same ratio of majority and minority in the validation set as in train set?
", "rawMarkdown": "Amanda Dsouza , are you maintaining the same ratio of majority and minority in the validation set as in train set?", "votes": null }, { "id": "139024", "postDate": "10/12/2016 06:28:00", "content": "[quote=Amanda Dsouza;139021]
\n\nHi, I would be very interested in some suggestions as well. Here's a link to the experiments I've tried so far: https://medium.com/@dsouza.amanda/learning-from-learning-curves-1a82c6f98f49#.vctunbm7n
\n\nDataset contains all of +ve samples, and unsampled -ve classes to make it a bit balanced. Stratified sampling is used on CV with 3 folds. Code in Python/Keras.
\n\n[/quote]
\n\nMy first suggestion is to try the dropout layer between input and hidden, and maybe another one between hidden and output. Start small, say 0.2 and increase up to 0.8.
\n\nNext, try different optimizers and activation functions. Lastly, BatchNormalization might help even though it will slow things down.
", "rawMarkdown": "[quote=Amanda Dsouza;139021]\r\n\r\nHi, I would be very interested in some suggestions as well. Here's a link to the experiments I've tried so far: https://medium.com/@dsouza.amanda/learning-from-learning-curves-1a82c6f98f49#.vctunbm7n\r\n\r\nDataset contains all of +ve samples, and unsampled -ve classes to make it a bit balanced. Stratified sampling is used on CV with 3 folds. Code in Python/Keras.\r\n\r\n[/quote]\r\n\r\nMy first suggestion is to try the dropout layer between input and hidden, and maybe another one between hidden and output. Start small, say 0.2 and increase up to 0.8.\r\n\r\nNext, try different [__optimizers__](https://keras.io/optimizers/) and [__activation functions__](https://keras.io/layers/advanced-activations/). Lastly, [__BatchNormalization__](https://keras.io/layers/normalization/) might help even though it will slow things down.", "votes": null }, { "id": "139027", "postDate": "10/12/2016 06:39:16", "content": "In both training & validation sets, the split ratio is roughly the same 1:2 for +ve to -ve classes. It looks like Sklearn does an even split of both training & validation. However, I have specified class weights in my neural network model to give more importance to the +ve class.
", "rawMarkdown": "In both training & validation sets, the split ratio is roughly the same 1:2 for +ve to -ve classes. It looks like Sklearn does an even split of both training & validation. However, I have specified class weights in my neural network model to give more importance to the +ve class.", "votes": null }, { "id": "139029", "postDate": "10/12/2016 06:52:32", "content": "[quote=Amanda Dsouza;139027]
\n\nIn both training & validation sets, the split ratio is roughly the same 1:2 for +ve to -ve classes. It looks like Sklearn does an even split of both training & validation. However, I have specified class weights in my neural network model to give more importance to the +ve class.
\n\n[/quote]
\n\nAny ratio other than the original one, i.e. 0.005:99.5 for +ve to -ve is giving misleading results.
", "rawMarkdown": "[quote=Amanda Dsouza;139027]\r\n\r\nIn both training & validation sets, the split ratio is roughly the same 1:2 for +ve to -ve classes. It looks like Sklearn does an even split of both training & validation. However, I have specified class weights in my neural network model to give more importance to the +ve class.\r\n\r\n[/quote]\r\n\r\nAny ratio other than the original one, i.e. 0.005:99.5 for +ve to -ve is giving misleading results.", "votes": null }, { "id": "139071", "postDate": "10/12/2016 11:52:01", "content": "Hi Chetan Madhok ,
\n\nDo you adjust threshold to maximize mcc ?
\nOutput of NN would be probability and the way to adjust threshold is also important.
[quote=toshi_k;139071]
\n\nHi Chetan Madhok ,
\n\nDo you adjust threshold to maximize mcc ?
\nOutput of NN would be probability and the way to adjust threshold is also important.
[/quote]
\n\nI thought the same, maybe misleading results help in this case to maximize the MCC! \nAmanda, for your experiments did you use some kind of MCC-maximation? If not your results using more or less the raw data seem really good, maybe i should also give neural networks a try. :) For the PCA, i found the following way to perform well: Instead of transforming the data in the whole take each station and perform PCA in it. some of the meassurements are more or less the same inside the station. Using Xgboost this way works well!
", "rawMarkdown": "[quote=toshi_k;139071]\r\n\r\nHi Chetan Madhok ,\r\n\r\nDo you adjust threshold to maximize mcc ?[quote=Chetan Madhok;139029]
\n\n[/quote]
\n\nAny ratio other than the original one, i.e. 0.005:99.5 for +ve to -ve is giving misleading results.
\n\n[/quote]
\n\n@Chetan, I wonder why that is so. Any thoughts? What if we were to use different samples of negative class in the feature selection process.
", "rawMarkdown": "[quote=Chetan Madhok;139029]\r\n\r\n[/quote]\r\n\r\nAny ratio other than the original one, i.e. 0.005:99.5 for +ve to -ve is giving misleading results. \r\n\r\n[/quote]\r\n\r\n@Chetan, I wonder why that is so. Any thoughts? What if we were to use different samples of negative class in the feature selection process.", "votes": null }, { "id": "139273", "postDate": "10/13/2016 06:34:30", "content": "[quote=Malte Nalenz;139089]
\n\nI thought the same, maybe misleading results help in this case to maximize the MCC! \nAmanda, for your experiments did you use some kind of MCC-maximation? If not your results using more or less the raw data seem really good, maybe i should also give neural networks a try. :) For the PCA, i found the following way to perform well: Instead of transforming the data in the whole take each station and perform PCA in it. some of the meassurements are more or less the same inside the station. Using Xgboost this way works well!
\n\n[/quote]
\n\n[UPDATED] @Malte, I used MCC to directly evaluate the neural network. It is available in Keras as 'matthews_correlation'. https://github.com/fchollet/keras/blob/master/keras/metrics.py \nThanks for the tip on PCA. I will give it a try.
", "rawMarkdown": "[quote=Malte Nalenz;139089]\r\n\r\nI thought the same, maybe misleading results help in this case to maximize the MCC! \r\nAmanda, for your experiments did you use some kind of MCC-maximation? If not your results using more or less the raw data seem really good, maybe i should also give neural networks a try. :) For the PCA, i found the following way to perform well: Instead of transforming the data in the whole take each station and perform PCA in it. some of the meassurements are more or less the same inside the station. Using Xgboost this way works well!\r\n\r\n[/quote]\r\n\r\n[UPDATED] @Malte, I used MCC to directly evaluate the neural network. It is available in Keras as 'matthews_correlation'. https://github.com/fchollet/keras/blob/master/keras/metrics.py \r\nThanks for the tip on PCA. I will give it a try.", "votes": null }, { "id": "139274", "postDate": "10/13/2016 06:37:19", "content": "[quote=Tilii;139024]
\n\nMy first suggestion is to try the dropout layer between input and hidden, and maybe another one between hidden and output. Start small, say 0.2 and increase up to 0.8.
\n\nNext, try different optimizers and activation functions. Lastly, BatchNormalization might help even though it will slow things down.
\n\n[/quote]
\n\n@Tilii, thanks for the suggestions. What is the purpose of adding dropout between I/P and hidden? Would it be better/same/worse to reduce the feature set using a feature selection process?
", "rawMarkdown": "[quote=Tilii;139024]\r\n\r\nMy first suggestion is to try the dropout layer between input and hidden, and maybe another one between hidden and output. Start small, say 0.2 and increase up to 0.8.\r\n\r\nNext, try different [__optimizers__](https://keras.io/optimizers/) and [__activation functions__](https://keras.io/layers/advanced-activations/). Lastly, [__BatchNormalization__](https://keras.io/layers/normalization/) might help even though it will slow things down.\r\n\r\n[/quote]\r\n\r\n@Tilii, thanks for the suggestions. What is the purpose of adding dropout between I/P and hidden? Would it be better/same/worse to reduce the feature set using a feature selection process?", "votes": null }, { "id": "139275", "postDate": "10/13/2016 06:42:08", "content": "[quote=Amanda Dsouza;139274]
\n\n[quote=Tilii;139024]
\n\nMy first suggestion is to try the dropout layer between input and hidden, and maybe another one between hidden and output. Start small, say 0.2 and increase up to 0.8.
\n\nNext, try different optimizers and activation functions. Lastly, BatchNormalization might help even though it will slow things down.
\n\n[/quote]
\n\n@Tilii, thanks for the suggestions. What is the purpose of adding dropout between I/P and hidden? Would it be better/same/worse to reduce the feature set using a feature selection process?
\n\n[/quote]
\n\nDropouts will kill a few connections between the nodes of different layers. This helps to escape the Local minima and avoids over fitting.\nexample: a dropout(0.5) would kill 50% of connections between the two layers.
", "rawMarkdown": "[quote=Amanda Dsouza;139274]\r\n\r\n[quote=Tilii;139024]\r\n\r\nMy first suggestion is to try the dropout layer between input and hidden, and maybe another one between hidden and output. Start small, say 0.2 and increase up to 0.8.\r\n\r\nNext, try different [__optimizers__](https://keras.io/optimizers/) and [__activation functions__](https://keras.io/layers/advanced-activations/). Lastly, [__BatchNormalization__](https://keras.io/layers/normalization/) might help even though it will slow things down.\r\n\r\n[/quote]\r\n\r\n@Tilii, thanks for the suggestions. What is the purpose of adding dropout between I/P and hidden? Would it be better/same/worse to reduce the feature set using a feature selection process? \r\n\r\n[/quote]\r\n\r\nDropouts will kill a few connections between the nodes of different layers. This helps to escape the Local minima and avoids over fitting.\r\nexample: a dropout(0.5) would kill 50% of connections between the two layers.", "votes": null }, { "id": "139276", "postDate": "10/13/2016 06:46:56", "content": "[quote=Chetan Madhok;139275]
\n\nDropouts will kill a few connections between the nodes of different layers. This helps to escape the Local minima and avoids over fitting.\nexample: a dropout(0.5) would kill 50% of connections between the two layers.
\n\n[/quote]
\n\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.
", "rawMarkdown": "[quote=Chetan Madhok;139275]\r\n\r\nDropouts will kill a few connections between the nodes of different layers. This helps to escape the Local minima and avoids over fitting.\r\nexample: a dropout(0.5) would kill 50% of connections between the two layers.\r\n\r\n[/quote]\r\n\r\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.", "votes": null }, { "id": "139277", "postDate": "10/13/2016 06:48:48", "content": "[quote=Amanda Dsouza;139276]
\n\n[quote=Chetan Madhok;139275]
\n\nDropouts will kill a few connections between the nodes of different layers. This helps to escape the Local minima and avoids over fitting.\nexample: a dropout(0.5) would kill 50% of connections between the two layers.
\n\n[/quote]
\n\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.\n[/quote]
\n\nThen it would be smaller set of Random Features at each iterations, that would decrease the variance and hence more accurate results.
", "rawMarkdown": "[quote=Amanda Dsouza;139276]\r\n\r\n[quote=Chetan Madhok;139275]\r\n\r\nDropouts will kill a few connections between the nodes of different layers. This helps to escape the Local minima and avoids over fitting.\r\nexample: a dropout(0.5) would kill 50% of connections between the two layers.\r\n\r\n[/quote]\r\n\r\n\r\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.\r\n[/quote]\r\n\r\n\r\n\r\n Then it would be smaller set of Random Features at each iterations, that would decrease the variance and hence more accurate results.", "votes": null }, { "id": "139328", "postDate": "10/13/2016 13:34:07", "content": "Do you use MCC to directly train neural network ?
keras/metrics.py have matthews_correlation,
\nhttps://github.com/fchollet/keras/blob/master/keras/metrics.py
\nHowever, keras/objectives.py don't have matthews_correlation.
\nhttps://github.com/fchollet/keras/blob/master/keras/objectives.py
You use MCC just for evaluation, don't you ? Which objective do you use ?
I played with the neural network approach as well, I used tensorflow.\nI did quite advanced network configuration (including a per station pre-training).\nHowever I stopped the idea after several days of trials.
\n\nTo save you some time, I can also say that I tried to minimize the MCC as objective.\nBut tensorflow didn't like it at all... It took me a couple of hours to realize that the MCC is not differentiable, and therefore the gradient doesn't allow to identify which weights (neurones) to update.
\n\nI tried a few (close) mathematical approximation of the MCC,\nbut as I said, I didn't find anything thing converging reasonably well.\nBut if you do, I'd be super interested to hear about it
\n\nGood luck,\nLoic
", "rawMarkdown": "I played with the neural network approach as well, I used tensorflow.\r\nI did quite advanced network configuration (including a per station pre-training).\r\nHowever I stopped the idea after several days of trials.\r\n\r\nTo save you some time, I can also say that I tried to minimize the MCC as objective.\r\nBut tensorflow didn't like it at all... It took me a couple of hours to realize that the MCC is not differentiable, and therefore the gradient doesn't allow to identify which weights (neurones) to update.\r\n\r\nI tried a few (close) mathematical approximation of the MCC,\r\nbut as I said, I didn't find anything thing converging reasonably well.\r\nBut if you do, I'd be super interested to hear about it\r\n\r\nGood luck,\r\nLoic", "votes": null }, { "id": "139330", "postDate": "10/13/2016 13:54:32", "content": "yes, you are right. I use MCC to evaluate training & validation. Loss function is logloss. \nmodel.compile(loss='binary_crossentropy', optimizer=rmsprop, metrics=['matthews_correlation'])
", "rawMarkdown": "yes, you are right. I use MCC to evaluate training & validation. Loss function is logloss. \r\nmodel.compile(loss='binary_crossentropy', optimizer=rmsprop, metrics=['matthews_correlation'])", "votes": null }, { "id": "139332", "postDate": "10/13/2016 14:03:32", "content": "[quote=Amanda Dsouza;139330]
\n\nyes, you are right. I use MCC to evaluate training & validation. Loss function is logloss. \nmodel.compile(loss='binary_crossentropy', optimizer=rmsprop, metrics=['matthews_correlation'])
\n\n[/quote]
\n\nThank you, Amanda. I understand your approach with keras !
", "rawMarkdown": "[quote=Amanda Dsouza;139330]\r\n\r\nyes, you are right. I use MCC to evaluate training & validation. Loss function is logloss. \r\nmodel.compile(loss='binary_crossentropy', optimizer=rmsprop, metrics=['matthews_correlation'])\r\n\r\n[/quote]\r\n\r\nThank you, Amanda. I understand your approach with keras !", "votes": null }, { "id": "139371", "postDate": "10/13/2016 18:31:29", "content": "[quote=Amanda Dsouza;139276]
\n\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.
\n\n[/quote]
\n\nDropout applies only to the training phase, while validation is done with all units and all data. This is why dropout has the effect of reducing the complex co-adaptations on training data that lead to overfitting, while still retaining the useful data for validation.
\n\nTo put it differently: reducing the number of features presumably throws away useless data (although it may throw away useful data as well if not done properly). Dropout randomly "throws away" useful data - but not always the same subset - during training, which has the averaging effect and reduces overfitting.
", "rawMarkdown": "[quote=Amanda Dsouza;139276]\r\n\r\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.\r\n\r\n[/quote]\r\n\r\nDropout applies only to the training phase, while validation is done with all units and all data. This is why dropout has the effect of reducing the complex co-adaptations on training data that lead to overfitting, while still retaining the useful data for validation.\r\n\r\nTo put it differently: reducing the number of features presumably throws away useless data (although it may throw away useful data as well if not done properly). Dropout randomly \"throws away\" useful data - but not always the same subset - during training, which has the averaging effect and reduces overfitting.", "votes": null }, { "id": "139436", "postDate": "10/14/2016 05:56:01", "content": "[quote=Tilii;139371]
\n\n[quote=Amanda Dsouza;139276]
\n\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.
\n\n[/quote]
\n\nDropout applies only to the training phase, while validation is done with all units and all data. This is why dropout has the effect of reducing the complex co-adaptations on training data that lead to overfitting, while still retaining the useful data for validation.
\n\nTo put it differently: reducing the number of features presumably throws away useless data (although it may throw away useful data as well if not done properly). Dropout randomly "throws away" useful data - but not always the same subset - during training, which has the averaging effect and reduces overfitting.
\n\n[/quote]
\n\n@Tilii, @Chetan, thanks for the explanation. Do you know how this can be done in Keras? I get this exception when I try to add Dropout as a first layer.\nException: The first layer in a Sequential model must get an input_shape or batch_input_shape argument.
Also, as regards to Batch Normalization, would you still need to use it if you are using RMSProp for optimization or is it typically for SGD?
", "rawMarkdown": "[quote=Tilii;139371]\r\n\r\n[quote=Amanda Dsouza;139276]\r\n\r\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.\r\n\r\n[/quote]\r\n\r\nDropout applies only to the training phase, while validation is done with all units and all data. This is why dropout has the effect of reducing the complex co-adaptations on training data that lead to overfitting, while still retaining the useful data for validation.\r\n\r\nTo put it differently: reducing the number of features presumably throws away useless data (although it may throw away useful data as well if not done properly). Dropout randomly \"throws away\" useful data - but not always the same subset - during training, which has the averaging effect and reduces overfitting.\r\n\r\n\r\n\r\n[/quote]\r\n\r\n@Tilii, @Chetan, thanks for the explanation. Do you know how this can be done in Keras? I get this exception when I try to add Dropout as a first layer.\r\nException: The first layer in a Sequential model must get an `input_shape` or `batch_input_shape` argument.\r\n\r\nAlso, as regards to Batch Normalization, would you still need to use it if you are using RMSProp for optimization or is it typically for SGD?", "votes": null }, { "id": "139671", "postDate": "10/15/2016 18:32:22", "content": "[quote=Amanda Dsouza;139436]
\n\n@Tilii, @Chetan, thanks for the explanation. Do you know how this can be done in Keras? I get this exception when I try to add Dropout as a first layer.\nException: The first layer in a Sequential model must get an input_shape or batch_input_shape argument.
Also, as regards to Batch Normalization, would you still need to use it if you are using RMSProp for optimization or is it typically for SGD?
\n\n[/quote]
\n\nDropout can't be the first or the last entry in your model. See here for an example.
\n\nBatchNormalization can be used with any optimizer.
", "rawMarkdown": "[quote=Amanda Dsouza;139436]\r\n\r\n@Tilii, @Chetan, thanks for the explanation. Do you know how this can be done in Keras? I get this exception when I try to add Dropout as a first layer.\r\nException: The first layer in a Sequential model must get an `input_shape` or `batch_input_shape` argument.\r\n\r\nAlso, as regards to Batch Normalization, would you still need to use it if you are using RMSProp for optimization or is it typically for SGD?\r\n\r\n\r\n[/quote]\r\n\r\nDropout can't be the first or the last entry in your model. See [__here__](https://www.kaggle.com/fchollet/digit-recognizer/simple-deep-mlp-with-keras) for an example.\r\n\r\n[__BatchNormalization__](https://keras.io/layers/normalization/) can be used with any optimizer.", "votes": null }, { "id": "139733", "postDate": "10/16/2016 05:45:11", "content": "[quote=Amanda Dsouza;139436]\n@Tilii, @Chetan, thanks for the explanation. Do you know how this can be done in Keras? I get this exception when I try to add Dropout as a first layer.\nException: The first layer in a Sequential model must get an input_shape or batch_input_shape argument.
Also, as regards to Batch Normalization, would you still need to use it if you are using RMSProp for optimization or is it typically for SGD?
\n\n[/quote]
\n\nIt's easy to add dropouts. Just make sure you mention the input shape in the first layer.
\n\nmodel = Sequential()\nmodel.add(Dropout(0.2, input_shape=(60,)))\nmodel.add(Dense(60, init='normal', activation='relu'))\nmodel.add(Dropout(0.2))\nmodel.add(Dense(1, init='normal', activation='sigmoid'))\nHello，Chetan Madhok. I want to know what's your best score by Neural Net？Looking forward to your reply.
", "rawMarkdown": "Hello，Chetan Madhok. I want to know what's your best score by Neural Net？Looking forward to your reply.", "votes": null } ], "comments": [ { "id": 961668, "author_name": "jiangjinyu", "author_url": "", "post_date": "08/07/2020 11:34:45", "content": "Hello，Chetan Madhok. I want to know what's your best score by Neural Net？Looking forward to your reply.
", "votes": null, "replies": [] }, { "id": 139021, "author_name": "amandadsouza", "author_url": "", "post_date": "10/12/2016 06:07:19", "content": "Hi, I would be very interested in some suggestions as well. Here's a link to the experiments I've tried so far: https://medium.com/@dsouza.amanda/learning-from-learning-curves-1a82c6f98f49#.vctunbm7n
\n\nDataset contains all of +ve samples, and undersampled -ve classes to make it a bit balanced. Stratified sampling is used on CV with 3 folds. Code in Python/Keras.
", "votes": null, "replies": [] }, { "id": 139022, "author_name": "cmadhok94", "author_url": "", "post_date": "10/12/2016 06:15:10", "content": "@Amanda Dsouza , are you maintaining the same ratio of majority and minority in the validation set as in train set?
", "votes": null, "replies": [] }, { "id": 139024, "author_name": "tilii7", "author_url": "", "post_date": "10/12/2016 06:28:00", "content": "[quote=Amanda Dsouza;139021]
\n\nHi, I would be very interested in some suggestions as well. Here's a link to the experiments I've tried so far: https://medium.com/@dsouza.amanda/learning-from-learning-curves-1a82c6f98f49#.vctunbm7n
\n\nDataset contains all of +ve samples, and unsampled -ve classes to make it a bit balanced. Stratified sampling is used on CV with 3 folds. Code in Python/Keras.
\n\n[/quote]
\n\nMy first suggestion is to try the dropout layer between input and hidden, and maybe another one between hidden and output. Start small, say 0.2 and increase up to 0.8.
\n\nNext, try different optimizers and activation functions. Lastly, BatchNormalization might help even though it will slow things down.
", "votes": null, "replies": [] }, { "id": 139027, "author_name": "amandadsouza", "author_url": "", "post_date": "10/12/2016 06:39:16", "content": "In both training & validation sets, the split ratio is roughly the same 1:2 for +ve to -ve classes. It looks like Sklearn does an even split of both training & validation. However, I have specified class weights in my neural network model to give more importance to the +ve class.
", "votes": null, "replies": [] }, { "id": 139029, "author_name": "cmadhok94", "author_url": "", "post_date": "10/12/2016 06:52:32", "content": "[quote=Amanda Dsouza;139027]
\n\nIn both training & validation sets, the split ratio is roughly the same 1:2 for +ve to -ve classes. It looks like Sklearn does an even split of both training & validation. However, I have specified class weights in my neural network model to give more importance to the +ve class.
\n\n[/quote]
\n\nAny ratio other than the original one, i.e. 0.005:99.5 for +ve to -ve is giving misleading results.
", "votes": null, "replies": [] }, { "id": 139071, "author_name": "toshik", "author_url": "", "post_date": "10/12/2016 11:52:01", "content": "Hi Chetan Madhok ,
\n\nDo you adjust threshold to maximize mcc ?
\nOutput of NN would be probability and the way to adjust threshold is also important.
[quote=toshi_k;139071]
\n\nHi Chetan Madhok ,
\n\nDo you adjust threshold to maximize mcc ?
\nOutput of NN would be probability and the way to adjust threshold is also important.
[/quote]
\n\nI thought the same, maybe misleading results help in this case to maximize the MCC! \nAmanda, for your experiments did you use some kind of MCC-maximation? If not your results using more or less the raw data seem really good, maybe i should also give neural networks a try. :) For the PCA, i found the following way to perform well: Instead of transforming the data in the whole take each station and perform PCA in it. some of the meassurements are more or less the same inside the station. Using Xgboost this way works well!
", "votes": null, "replies": [] }, { "id": 139272, "author_name": "amandadsouza", "author_url": "", "post_date": "10/13/2016 06:31:55", "content": "[quote=Chetan Madhok;139029]
\n\n[/quote]
\n\nAny ratio other than the original one, i.e. 0.005:99.5 for +ve to -ve is giving misleading results.
\n\n[/quote]
\n\n@Chetan, I wonder why that is so. Any thoughts? What if we were to use different samples of negative class in the feature selection process.
", "votes": null, "replies": [] }, { "id": 139273, "author_name": "amandadsouza", "author_url": "", "post_date": "10/13/2016 06:34:30", "content": "[quote=Malte Nalenz;139089]
\n\nI thought the same, maybe misleading results help in this case to maximize the MCC! \nAmanda, for your experiments did you use some kind of MCC-maximation? If not your results using more or less the raw data seem really good, maybe i should also give neural networks a try. :) For the PCA, i found the following way to perform well: Instead of transforming the data in the whole take each station and perform PCA in it. some of the meassurements are more or less the same inside the station. Using Xgboost this way works well!
\n\n[/quote]
\n\n[UPDATED] @Malte, I used MCC to directly evaluate the neural network. It is available in Keras as 'matthews_correlation'. https://github.com/fchollet/keras/blob/master/keras/metrics.py \nThanks for the tip on PCA. I will give it a try.
", "votes": null, "replies": [] }, { "id": 139274, "author_name": "amandadsouza", "author_url": "", "post_date": "10/13/2016 06:37:19", "content": "[quote=Tilii;139024]
\n\nMy first suggestion is to try the dropout layer between input and hidden, and maybe another one between hidden and output. Start small, say 0.2 and increase up to 0.8.
\n\nNext, try different optimizers and activation functions. Lastly, BatchNormalization might help even though it will slow things down.
\n\n[/quote]
\n\n@Tilii, thanks for the suggestions. What is the purpose of adding dropout between I/P and hidden? Would it be better/same/worse to reduce the feature set using a feature selection process?
", "votes": null, "replies": [] }, { "id": 139275, "author_name": "cmadhok94", "author_url": "", "post_date": "10/13/2016 06:42:08", "content": "[quote=Amanda Dsouza;139274]
\n\n[quote=Tilii;139024]
\n\nMy first suggestion is to try the dropout layer between input and hidden, and maybe another one between hidden and output. Start small, say 0.2 and increase up to 0.8.
\n\nNext, try different optimizers and activation functions. Lastly, BatchNormalization might help even though it will slow things down.
\n\n[/quote]
\n\n@Tilii, thanks for the suggestions. What is the purpose of adding dropout between I/P and hidden? Would it be better/same/worse to reduce the feature set using a feature selection process?
\n\n[/quote]
\n\nDropouts will kill a few connections between the nodes of different layers. This helps to escape the Local minima and avoids over fitting.\nexample: a dropout(0.5) would kill 50% of connections between the two layers.
", "votes": null, "replies": [] }, { "id": 139276, "author_name": "amandadsouza", "author_url": "", "post_date": "10/13/2016 06:46:56", "content": "[quote=Chetan Madhok;139275]
\n\nDropouts will kill a few connections between the nodes of different layers. This helps to escape the Local minima and avoids over fitting.\nexample: a dropout(0.5) would kill 50% of connections between the two layers.
\n\n[/quote]
\n\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.
", "votes": null, "replies": [] }, { "id": 139277, "author_name": "cmadhok94", "author_url": "", "post_date": "10/13/2016 06:48:48", "content": "[quote=Amanda Dsouza;139276]
\n\n[quote=Chetan Madhok;139275]
\n\nDropouts will kill a few connections between the nodes of different layers. This helps to escape the Local minima and avoids over fitting.\nexample: a dropout(0.5) would kill 50% of connections between the two layers.
\n\n[/quote]
\n\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.\n[/quote]
\n\nThen it would be smaller set of Random Features at each iterations, that would decrease the variance and hence more accurate results.
", "votes": null, "replies": [] }, { "id": 139328, "author_name": "toshik", "author_url": "", "post_date": "10/13/2016 13:34:07", "content": "Do you use MCC to directly train neural network ?
keras/metrics.py have matthews_correlation,
\nhttps://github.com/fchollet/keras/blob/master/keras/metrics.py
\nHowever, keras/objectives.py don't have matthews_correlation.
\nhttps://github.com/fchollet/keras/blob/master/keras/objectives.py
You use MCC just for evaluation, don't you ? Which objective do you use ?
I played with the neural network approach as well, I used tensorflow.\nI did quite advanced network configuration (including a per station pre-training).\nHowever I stopped the idea after several days of trials.
\n\nTo save you some time, I can also say that I tried to minimize the MCC as objective.\nBut tensorflow didn't like it at all... It took me a couple of hours to realize that the MCC is not differentiable, and therefore the gradient doesn't allow to identify which weights (neurones) to update.
\n\nI tried a few (close) mathematical approximation of the MCC,\nbut as I said, I didn't find anything thing converging reasonably well.\nBut if you do, I'd be super interested to hear about it
\n\nGood luck,\nLoic
", "votes": null, "replies": [] }, { "id": 139330, "author_name": "amandadsouza", "author_url": "", "post_date": "10/13/2016 13:54:32", "content": "yes, you are right. I use MCC to evaluate training & validation. Loss function is logloss. \nmodel.compile(loss='binary_crossentropy', optimizer=rmsprop, metrics=['matthews_correlation'])
", "votes": null, "replies": [] }, { "id": 139332, "author_name": "toshik", "author_url": "", "post_date": "10/13/2016 14:03:32", "content": "[quote=Amanda Dsouza;139330]
\n\nyes, you are right. I use MCC to evaluate training & validation. Loss function is logloss. \nmodel.compile(loss='binary_crossentropy', optimizer=rmsprop, metrics=['matthews_correlation'])
\n\n[/quote]
\n\nThank you, Amanda. I understand your approach with keras !
", "votes": null, "replies": [] }, { "id": 139371, "author_name": "tilii7", "author_url": "", "post_date": "10/13/2016 18:31:29", "content": "[quote=Amanda Dsouza;139276]
\n\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.
\n\n[/quote]
\n\nDropout applies only to the training phase, while validation is done with all units and all data. This is why dropout has the effect of reducing the complex co-adaptations on training data that lead to overfitting, while still retaining the useful data for validation.
\n\nTo put it differently: reducing the number of features presumably throws away useless data (although it may throw away useful data as well if not done properly). Dropout randomly "throws away" useful data - but not always the same subset - during training, which has the averaging effect and reduces overfitting.
", "votes": null, "replies": [] }, { "id": 139436, "author_name": "amandadsouza", "author_url": "", "post_date": "10/14/2016 05:56:01", "content": "[quote=Tilii;139371]
\n\n[quote=Amanda Dsouza;139276]
\n\nYes, i know, but my question is more why to apply dropout between the input & hidden layer. It seems like that would be the same as using a smaller set of features.
\n\n[/quote]
\n\nDropout applies only to the training phase, while validation is done with all units and all data. This is why dropout has the effect of reducing the complex co-adaptations on training data that lead to overfitting, while still retaining the useful data for validation.
\n\nTo put it differently: reducing the number of features presumably throws away useless data (although it may throw away useful data as well if not done properly). Dropout randomly "throws away" useful data - but not always the same subset - during training, which has the averaging effect and reduces overfitting.
\n\n[/quote]
\n\n@Tilii, @Chetan, thanks for the explanation. Do you know how this can be done in Keras? I get this exception when I try to add Dropout as a first layer.\nException: The first layer in a Sequential model must get an input_shape or batch_input_shape argument.
Also, as regards to Batch Normalization, would you still need to use it if you are using RMSProp for optimization or is it typically for SGD?
", "votes": null, "replies": [] }, { "id": 139671, "author_name": "tilii7", "author_url": "", "post_date": "10/15/2016 18:32:22", "content": "[quote=Amanda Dsouza;139436]
\n\n@Tilii, @Chetan, thanks for the explanation. Do you know how this can be done in Keras? I get this exception when I try to add Dropout as a first layer.\nException: The first layer in a Sequential model must get an input_shape or batch_input_shape argument.
Also, as regards to Batch Normalization, would you still need to use it if you are using RMSProp for optimization or is it typically for SGD?
\n\n[/quote]
\n\nDropout can't be the first or the last entry in your model. See here for an example.
\n\nBatchNormalization can be used with any optimizer.
", "votes": null, "replies": [] }, { "id": 139733, "author_name": "cmadhok94", "author_url": "", "post_date": "10/16/2016 05:45:11", "content": "[quote=Amanda Dsouza;139436]\n@Tilii, @Chetan, thanks for the explanation. Do you know how this can be done in Keras? I get this exception when I try to add Dropout as a first layer.\nException: The first layer in a Sequential model must get an input_shape or batch_input_shape argument.
Also, as regards to Batch Normalization, would you still need to use it if you are using RMSProp for optimization or is it typically for SGD?
\n\n[/quote]
\n\nIt's easy to add dropouts. Just make sure you mention the input shape in the first layer.
\n\nmodel = Sequential()\nmodel.add(Dropout(0.2, input_shape=(60,)))\nmodel.add(Dense(60, init='normal', activation='relu'))\nmodel.add(Dropout(0.2))\nmodel.add(Dense(1, init='normal', activation='sigmoid'))\n