For past 10 days or so I've been trying to get GNN working. I took GNN code from @fnands's very very helpful kernel. https://www.kaggle.com/fnands/1-mpnn . I'm, however, using my own features. I'm using normalized version of the features (+one hot encoding for categorical ones) I developed for tree based Catboost model. With GNN I've reached -1.54 on my validation set. However, 5 fold group validation based Catboost model gives me -1.95 on my validation set and -2.0 on LB.
\n\nI'm struggling to speed up the iteration for my GNN. In tree based models, it is relatively simpler as\n1. Explicit feature importances are present.: With permutation importance (https://www.kaggle.com/speedwagon/permutation-importance) I'm trying to manage this for GNN.
\n\nFor run time improvement, one way could be to remove the redundant data. In @fnands implementation, I see that 95% of edges data is just zeros. It is because the edge data is multi dimensional. 2 dimensional representation of edge data is much more compact but less 'numpy parallelizable'. So no clear way here.
\n\nI use kaggle kernels for GPU . I know here there are many experienced people here. I'm very much interested in knowing how people improve upon their neural network models in a time efficient way (with meagre resources :D ). All suggestions are much appreciated !
\n\nThanks !!
", "messages": [ { "id": "604449", "postDate": "08/21/2019 11:57:45", "content": "For past 10 days or so I've been trying to get GNN working. I took GNN code from @fnands's very very helpful kernel. https://www.kaggle.com/fnands/1-mpnn . I'm, however, using my own features. I'm using normalized version of the features (+one hot encoding for categorical ones) I developed for tree based Catboost model. With GNN I've reached -1.54 on my validation set. However, 5 fold group validation based Catboost model gives me -1.95 on my validation set and -2.0 on LB.
\n\nI'm struggling to speed up the iteration for my GNN. In tree based models, it is relatively simpler as\n1. Explicit feature importances are present.: With permutation importance (https://www.kaggle.com/speedwagon/permutation-importance) I'm trying to manage this for GNN.
\n\nFor run time improvement, one way could be to remove the redundant data. In @fnands implementation, I see that 95% of edges data is just zeros. It is because the edge data is multi dimensional. 2 dimensional representation of edge data is much more compact but less 'numpy parallelizable'. So no clear way here.
\n\nI use kaggle kernels for GPU . I know here there are many experienced people here. I'm very much interested in knowing how people improve upon their neural network models in a time efficient way (with meagre resources :D ). All suggestions are much appreciated !
\n\nThanks !!
", "rawMarkdown": "For past 10 days or so I've been trying to get GNN working. I took GNN code from @fnands's very very helpful kernel. https://www.kaggle.com/fnands/1-mpnn . I'm, however, using my own features. I'm using normalized version of the features (+one hot encoding for categorical ones) I developed for tree based Catboost model. With GNN I've reached -1.54 on my validation set. However, 5 fold group validation based Catboost model gives me -1.95 on my validation set and -2.0 on LB.\n \nI'm struggling to speed up the iteration for my GNN. In tree based models, it is relatively simpler as\n1. **Explicit feature importances** are present.: With permutation importance (https://www.kaggle.com/speedwagon/permutation-importance) I'm trying to manage this for GNN. \n\n2. **Run time** is quite less for tree based models. Just 25 iterations of GNN takes about 8 hours ! \n\nFor run time improvement, one way could be to remove the redundant data. In @fnands implementation, I see that 95% of edges data is just zeros. It is because the edge data is multi dimensional. 2 dimensional representation of edge data is much more compact but less 'numpy parallelizable'. So no clear way here. \n\nI use kaggle kernels for GPU . I know here there are many experienced people here. I'm very much interested in knowing how people improve upon their neural network models in a time efficient way (with meagre resources :D ). All suggestions are much appreciated !\n\nThanks !!", "votes": null }, { "id": "604486", "postDate": "08/21/2019 12:51:47", "content": "Quick and dirty: train model - save weights - initialize saved weights and train further.\nIt's not really about time saving, but could help with kernel runtime limit.
", "rawMarkdown": "Quick and dirty: train model - save weights - initialize saved weights and train further.\nIt's not really about time saving, but could help with kernel runtime limit.", "votes": null }, { "id": "604508", "postDate": "08/21/2019 13:11:58", "content": "As an example: mpnn.save_weights(\"mymodel.h5\")\nand then: mpnn.load_weights(\"mymodel.h5\")
Yup. Thats what I'll do. Currently I'm trying to fit as many features as I can. Once the feature set is fixed, I'll be loading and saving weights to train it for longer period.
", "rawMarkdown": "Yup. Thats what I'll do. Currently I'm trying to fit as many features as I can. Once the feature set is fixed, I'll be loading and saving weights to train it for longer period.", "votes": null }, { "id": "608287", "postDate": "08/26/2019 15:39:47", "content": "What I see is that I just need to appropriately reduce the learning rate. AND give it time. Lots of time. -2.0 LB with GNN 😁 \nEagerly waiting to see how others have got < -2.5 with GNN.
", "rawMarkdown": "What I see is that I just need to appropriately reduce the learning rate. AND give it time. Lots of time. -2.0 LB with GNN 😁 \nEagerly waiting to see how others have got < -2.5 with GNN.", "votes": null }, { "id": "608308", "postDate": "08/26/2019 15:59:56", "content": "how long did you train to reach -2.0LB? I have one single model at this score, trained for ~16h.
", "rawMarkdown": "how long did you train to reach -2.0LB? I have one single model at this score, trained for ~16h.", "votes": null }, { "id": "608344", "postDate": "08/26/2019 16:43:19", "content": "\n\n\nEagerly waiting to see how others have got < -2.5 with GNN.
\n
How do you know they use GNN?
", "rawMarkdown": "> Eagerly waiting to see how others have got < -2.5 with GNN.\n\nHow do you know they use GNN?", "votes": null }, { "id": "608796", "postDate": "08/27/2019 07:11:12", "content": "\nhow long did you train to reach -2.0LB? I have one single model at this score, trained for ~16h.\n\n1. 5 fold CV helps a lot over a single model.\n2. 40 epoch each.
\nHow do you know they use GNN?\n \nI think you mean How do you know top-k people use GNN (k being < 10) 😃 . I don't. However, comments like\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/105088#604228 tells that there are some good kernels out there using GNNs.
I'm interested in learning how to efficiently train [G]NNs. If the sole factor differentiating my work with some other better work is feature engineering, I'm not interested. On the other hand, if something is related to preprocessing, learning rate, regularization, or even something better than try and see method of feature engineering, or any other aspect which can be transfer learnt to another problem, I'm very much interested in that.
Quick and dirty: train model - save weights - initialize saved weights and train further.\nIt's not really about time saving, but could help with kernel runtime limit.
", "votes": null, "replies": [ { "id": 604508, "author_name": "fnands", "author_url": "", "post_date": "08/21/2019 13:11:58", "content": "As an example: mpnn.save_weights(\"mymodel.h5\")\nand then: mpnn.load_weights(\"mymodel.h5\")
Yup. Thats what I'll do. Currently I'm trying to fit as many features as I can. Once the feature set is fixed, I'll be loading and saving weights to train it for longer period.
", "votes": null, "replies": [] } ] }, { "id": 608287, "author_name": "silence2", "author_url": "", "post_date": "08/26/2019 15:39:47", "content": "What I see is that I just need to appropriately reduce the learning rate. AND give it time. Lots of time. -2.0 LB with GNN 😁 \nEagerly waiting to see how others have got < -2.5 with GNN.
", "votes": null, "replies": [ { "id": 608308, "author_name": "optimo", "author_url": "", "post_date": "08/26/2019 15:59:56", "content": "how long did you train to reach -2.0LB? I have one single model at this score, trained for ~16h.
", "votes": null, "replies": [] }, { "id": 608344, "author_name": "cpmpml", "author_url": "", "post_date": "08/26/2019 16:43:19", "content": "\n\n\nEagerly waiting to see how others have got < -2.5 with GNN.
\n
How do you know they use GNN?
", "votes": null, "replies": [] }, { "id": 608796, "author_name": "silence2", "author_url": "", "post_date": "08/27/2019 07:11:12", "content": "\nhow long did you train to reach -2.0LB? I have one single model at this score, trained for ~16h.\n\n1. 5 fold CV helps a lot over a single model.\n2. 40 epoch each.
\nHow do you know they use GNN?\n \nI think you mean How do you know top-k people use GNN (k being < 10) 😃 . I don't. However, comments like\nhttps://www.kaggle.com/c/champs-scalar-coupling/discussion/105088#604228 tells that there are some good kernels out there using GNNs.
I'm interested in learning how to efficiently train [G]NNs. If the sole factor differentiating my work with some other better work is feature engineering, I'm not interested. On the other hand, if something is related to preprocessing, learning rate, regularization, or even something better than try and see method of feature engineering, or any other aspect which can be transfer learnt to another problem, I'm very much interested in that.