![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F314368%2Fcabe0b91e3333064b4d81ed5984ef351%2FNMPAlgo1.png?generation=1567064230681357&alt=media\")
So the main inspiration for our best solution came mostly from two papers: the Gilmer/Brain paper, as implemented in my public kernel, and then Neural Message Passing for Jet Physics..
\n\nFrom the latter paper I took the idea of adding an extra step of not only updating the node values at each step, but also updating the adjacency matrix at each step, as in this algorithm from the above mentioned paper:
Also, from this paper, of which @jaechang is a co-author, I added skip connections between layers, which allowed us to go deeper (although we saw no improvement above 7 layers).
\n\nThe message passing functions and update functions are different from what was done in my public kernel, and for those interested in the details, please have a look at this kernel (and pre-processing kernel )\nOf course our solution used 4/5 fold CV, and a bit of stacking, but the main workhorse of out model is described in the kernel. If you have any questions about the model please ask and I will clarify. I did it in TF2.0 as one of my goals for this Kaggle was to get familiar with the new version.
\n\nMy gut feeling is that this model has too many parameters and there is a simpler model out there that will do a better job. Any comments/suggestions/ideas are greatly appreciated.
\n\nOn a personal note, this has been the first Kaggle in which I have really competed and I had a great time! Thanks to everyone who made kernels, participated in discussions and who were just cool people in general. \nI learned a lot in this competition and hope to apply what I have learned in the future.
\n\nSpecifically, thanks to my teammates @kazanova , @rohanrao , @supreethmanyam and @ogrellier who were great and really helped to squeeze the most out of our solution. They were all great teammates and I can highly recommend working with them!
\n\nNow, I'm going to go read up on what the winners did... 🤓
", "messages": [ { "id": "611503", "postDate": "08/29/2019 09:48:02", "content": "So the main inspiration for our best solution came mostly from two papers: the Gilmer/Brain paper, as implemented in my public kernel, and then Neural Message Passing for Jet Physics..
\n\nFrom the latter paper I took the idea of adding an extra step of not only updating the node values at each step, but also updating the adjacency matrix at each step, as in this algorithm from the above mentioned paper:
Also, from this paper, of which @jaechang is a co-author, I added skip connections between layers, which allowed us to go deeper (although we saw no improvement above 7 layers).
\n\nThe message passing functions and update functions are different from what was done in my public kernel, and for those interested in the details, please have a look at this kernel (and pre-processing kernel )\nOf course our solution used 4/5 fold CV, and a bit of stacking, but the main workhorse of out model is described in the kernel. If you have any questions about the model please ask and I will clarify. I did it in TF2.0 as one of my goals for this Kaggle was to get familiar with the new version.
\n\nMy gut feeling is that this model has too many parameters and there is a simpler model out there that will do a better job. Any comments/suggestions/ideas are greatly appreciated.
\n\nOn a personal note, this has been the first Kaggle in which I have really competed and I had a great time! Thanks to everyone who made kernels, participated in discussions and who were just cool people in general. \nI learned a lot in this competition and hope to apply what I have learned in the future.
\n\nSpecifically, thanks to my teammates @kazanova , @rohanrao , @supreethmanyam and @ogrellier who were great and really helped to squeeze the most out of our solution. They were all great teammates and I can highly recommend working with them!
\n\nNow, I'm going to go read up on what the winners did... 🤓
", "rawMarkdown": "So the main inspiration for our best solution came mostly from two papers: the [Gilmer/Brain ](https://arxiv.org/abs/1704.01212) paper, as implemented in my [public kernel](https://www.kaggle.com/fnands/1-mpnn), and then [Neural Message Passing for Jet Physics.](https://dl4physicalsciences.github.io/files/nips_dlps_2017_29.pdf). \n\nFrom the latter paper I took the idea of adding an extra step of not only updating the node values at each step, but also updating the adjacency matrix at each step, as in this algorithm from the above mentioned paper: \n\nAlso, from [this paper](https://arxiv.org/abs/1805.10988), of which @jaechang is a co-author, I added skip connections between layers, which allowed us to go deeper (although we saw no improvement above 7 layers). \n\nThe message passing functions and update functions are different from what was done in my public kernel, and for those interested in the details, please have a look at [this kernel](https://www.kaggle.com/fnands/mpnn-skip-2) (and [pre-processing kernel](https://www.kaggle.com/fnands/internalmakegraphinput/output) )\nOf course our solution used 4/5 fold CV, and a bit of stacking, but the main workhorse of out model is described in the kernel. If you have any questions about the model please ask and I will clarify. I did it in TF2.0 as one of my goals for this Kaggle was to get familiar with the new version. \n\nMy gut feeling is that this model has too many parameters and there is a simpler model out there that will do a better job. Any comments/suggestions/ideas are greatly appreciated. \n\n\nOn a personal note, this has been the first Kaggle in which I have really competed and I had a great time! Thanks to everyone who made kernels, participated in discussions and who were just cool people in general. \nI learned a lot in this competition and hope to apply what I have learned in the future. \n\nSpecifically, thanks to my teammates @kazanova , @rohanrao , @supreethmanyam and @ogrellier who were great and really helped to squeeze the most out of our solution. They were all great teammates and I can highly recommend working with them!\n\nNow, I'm going to go read up on what the winners did... 🤓", "votes": null }, { "id": "611530", "postDate": "08/29/2019 10:04:56", "content": "Thank you su much, now with Heng baseline + your 2 notebooks I have a great MPNN base.
", "rawMarkdown": "Thank you su much, now with Heng baseline + your 2 notebooks I have a great MPNN base.", "votes": null }, { "id": "611544", "postDate": "08/29/2019 10:11:07", "content": "Congrats... \nGreat Work..\nVery Informative....!! Thanks for Sharing.....! @fnands
", "rawMarkdown": "Congrats... \nGreat Work..\nVery Informative....!! Thanks for Sharing.....! @fnands", "votes": null }, { "id": "611924", "postDate": "08/29/2019 14:02:58", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "611926", "postDate": "08/29/2019 14:03:20", "content": "Pleasure! 🙏
", "rawMarkdown": "Pleasure! 🙏", "votes": null }, { "id": "612290", "postDate": "08/29/2019 17:52:18", "content": "It's been a pleasure to team up with you @fnands ! You've got everything to make a great Kaggle Grandmaster !
", "rawMarkdown": "It's been a pleasure to team up with you @fnands ! You've got everything to make a great Kaggle Grandmaster !", "votes": null }, { "id": "612372", "postDate": "08/29/2019 18:42:53", "content": "Thanks for sharing, and congrats on th eend result. Progress in the last days was impressive, you could get to gold with one extra week I think.
", "rawMarkdown": "Thanks for sharing, and congrats on th eend result. Progress in the last days was impressive, you could get to gold with one extra week I think.", "votes": null }, { "id": "612397", "postDate": "08/29/2019 19:02:09", "content": "Congrats on your finish and great write up! I spent many days working with your MPNN code and learned a lot in the process.
", "rawMarkdown": "Congrats on your finish and great write up! I spent many days working with your MPNN code and learned a lot in the process.", "votes": null }, { "id": "612723", "postDate": "08/30/2019 00:50:23", "content": "Thanks and congrats @fnands , very informative .
", "rawMarkdown": "Thanks and congrats @fnands , very informative .", "votes": null }, { "id": "612725", "postDate": "08/30/2019 00:52:10", "content": "Congrats to your teammates as well @ogrellier @rohanrao @supreethmanyam @kazanova
", "rawMarkdown": "Congrats to your teammates as well @ogrellier @rohanrao @supreethmanyam @kazanova", "votes": null }, { "id": "612913", "postDate": "08/30/2019 04:53:10", "content": "Congrats !!
", "rawMarkdown": "Congrats !!", "votes": null }, { "id": "612972", "postDate": "08/30/2019 06:04:26", "content": "congratulations wonderful
", "rawMarkdown": "*congratulations wonderful*", "votes": null }, { "id": "613088", "postDate": "08/30/2019 07:59:14", "content": "Thank you, I'm glad it helped!
", "rawMarkdown": "Thank you, I'm glad it helped!", "votes": null }, { "id": "613091", "postDate": "08/30/2019 08:00:29", "content": "Thanks! Yeah there were many more things to try, but I am pretty happy with the top 1% result. \nCongrats with the gold!
", "rawMarkdown": "Thanks! Yeah there were many more things to try, but I am pretty happy with the top 1% result. \nCongrats with the gold!", "votes": null }, { "id": "613092", "postDate": "08/30/2019 08:01:10", "content": "🙏 🙏 🙏 thanks for all the help, was good working with you!
", "rawMarkdown": "🙏 🙏 🙏 thanks for all the help, was good working with you!", "votes": null }, { "id": "613093", "postDate": "08/30/2019 08:02:03", "content": "🙌
", "rawMarkdown": "🙌", "votes": null }, { "id": "613094", "postDate": "08/30/2019 08:02:16", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": null }, { "id": "613095", "postDate": "08/30/2019 08:02:27", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "613155", "postDate": "08/30/2019 09:07:07", "content": "I think I never see anyone with the 1st competition but can both contribute a very good working kernel which laid foundations for hundreds of others to use, and finish in top 20 like you. Congrats and thank you so much.
", "rawMarkdown": "I think I never see anyone with the 1st competition but can both contribute a very good working kernel which laid foundations for hundreds of others to use, and finish in top 20 like you. Congrats and thank you so much.", "votes": null }, { "id": "613495", "postDate": "08/30/2019 15:01:31", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": null } ], "comments": [ { "id": 611530, "author_name": "jesucristo", "author_url": "", "post_date": "08/29/2019 10:04:56", "content": "Thank you su much, now with Heng baseline + your 2 notebooks I have a great MPNN base.
", "votes": null, "replies": [ { "id": 611926, "author_name": "fnands", "author_url": "", "post_date": "08/29/2019 14:03:20", "content": "Pleasure! 🙏
", "votes": null, "replies": [] } ] }, { "id": 611544, "author_name": "veeralakrishna", "author_url": "", "post_date": "08/29/2019 10:11:07", "content": "Congrats... \nGreat Work..\nVery Informative....!! Thanks for Sharing.....! @fnands
", "votes": null, "replies": [ { "id": 611924, "author_name": "fnands", "author_url": "", "post_date": "08/29/2019 14:02:58", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 612290, "author_name": "ogrellier", "author_url": "", "post_date": "08/29/2019 17:52:18", "content": "It's been a pleasure to team up with you @fnands ! You've got everything to make a great Kaggle Grandmaster !
", "votes": null, "replies": [ { "id": 613092, "author_name": "fnands", "author_url": "", "post_date": "08/30/2019 08:01:10", "content": "🙏 🙏 🙏 thanks for all the help, was good working with you!
", "votes": null, "replies": [] } ] }, { "id": 612372, "author_name": "cpmpml", "author_url": "", "post_date": "08/29/2019 18:42:53", "content": "Thanks for sharing, and congrats on th eend result. Progress in the last days was impressive, you could get to gold with one extra week I think.
", "votes": null, "replies": [ { "id": 613091, "author_name": "fnands", "author_url": "", "post_date": "08/30/2019 08:00:29", "content": "Thanks! Yeah there were many more things to try, but I am pretty happy with the top 1% result. \nCongrats with the gold!
", "votes": null, "replies": [] } ] }, { "id": 612397, "author_name": "robikscube", "author_url": "", "post_date": "08/29/2019 19:02:09", "content": "Congrats on your finish and great write up! I spent many days working with your MPNN code and learned a lot in the process.
", "votes": null, "replies": [ { "id": 613088, "author_name": "fnands", "author_url": "", "post_date": "08/30/2019 07:59:14", "content": "Thank you, I'm glad it helped!
", "votes": null, "replies": [] } ] }, { "id": 612723, "author_name": "michelml", "author_url": "", "post_date": "08/30/2019 00:50:23", "content": "Thanks and congrats @fnands , very informative .
", "votes": null, "replies": [ { "id": 612725, "author_name": "michelml", "author_url": "", "post_date": "08/30/2019 00:52:10", "content": "Congrats to your teammates as well @ogrellier @rohanrao @supreethmanyam @kazanova
", "votes": null, "replies": [] }, { "id": 613093, "author_name": "fnands", "author_url": "", "post_date": "08/30/2019 08:02:03", "content": "🙌
", "votes": null, "replies": [] } ] }, { "id": 612913, "author_name": "silence2", "author_url": "", "post_date": "08/30/2019 04:53:10", "content": "Congrats !!
", "votes": null, "replies": [ { "id": 613094, "author_name": "fnands", "author_url": "", "post_date": "08/30/2019 08:02:16", "content": "Thanks!
", "votes": null, "replies": [] } ] }, { "id": 612972, "author_name": "nasere", "author_url": "", "post_date": "08/30/2019 06:04:26", "content": "congratulations wonderful
", "votes": null, "replies": [ { "id": 613095, "author_name": "fnands", "author_url": "", "post_date": "08/30/2019 08:02:27", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 613155, "author_name": "khahuras", "author_url": "", "post_date": "08/30/2019 09:07:07", "content": "I think I never see anyone with the 1st competition but can both contribute a very good working kernel which laid foundations for hundreds of others to use, and finish in top 20 like you. Congrats and thank you so much.
", "votes": null, "replies": [ { "id": 613495, "author_name": "fnands", "author_url": "", "post_date": "08/30/2019 15:01:31", "content": "Thanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "611503": "So the main inspiration for our best solution came mostly from two papers: the [Gilmer/Brain ](https://arxiv.org/abs/1704.01212) paper, as implemented in my [public kernel](https://www.kaggle.com/fnands/1-mpnn), and then [Neural Message Passing for Jet Physics.](https://dl4physicalsciences.github.io/files/nips_dlps_2017_29.pdf). \n\nFrom the latter paper I took the idea of adding an extra step of not only updating the node values at each step, but also updating the adjacency matrix at each step, as in this algorithm from the above mentioned paper: \n\nAlso, from [this paper](https://arxiv.org/abs/1805.10988), of which @jaechang is a co-author, I added skip connections between layers, which allowed us to go deeper (although we saw no improvement above 7 layers). \n\nThe message passing functions and update functions are different from what was done in my public kernel, and for those interested in the details, please have a look at [this kernel](https://www.kaggle.com/fnands/mpnn-skip-2) (and [pre-processing kernel](https://www.kaggle.com/fnands/internalmakegraphinput/output) )\nOf course our solution used 4/5 fold CV, and a bit of stacking, but the main workhorse of out model is described in the kernel. If you have any questions about the model please ask and I will clarify. I did it in TF2.0 as one of my goals for this Kaggle was to get familiar with the new version. \n\nMy gut feeling is that this model has too many parameters and there is a simpler model out there that will do a better job. Any comments/suggestions/ideas are greatly appreciated. \n\n\nOn a personal note, this has been the first Kaggle in which I have really competed and I had a great time! Thanks to everyone who made kernels, participated in discussions and who were just cool people in general. \nI learned a lot in this competition and hope to apply what I have learned in the future. \n\nSpecifically, thanks to my teammates @kazanova , @rohanrao , @supreethmanyam and @ogrellier who were great and really helped to squeeze the most out of our solution. They were all great teammates and I can highly recommend working with them!\n\nNow, I'm going to go read up on what the winners did... 🤓", "611530": "Thank you su much, now with Heng baseline + your 2 notebooks I have a great MPNN base.", "611544": "Congrats... \nGreat Work..\nVery Informative....!! Thanks for Sharing.....! @fnands", "611924": "Thank you!", "611926": "Pleasure! 🙏", "612290": "It's been a pleasure to team up with you @fnands ! You've got everything to make a great Kaggle Grandmaster !", "612372": "Thanks for sharing, and congrats on th eend result. Progress in the last days was impressive, you could get to gold with one extra week I think.", "612397": "Congrats on your finish and great write up! I spent many days working with your MPNN code and learned a lot in the process.", "612723": "Thanks and congrats @fnands , very informative .", "612725": "Congrats to your teammates as well @ogrellier @rohanrao @supreethmanyam @kazanova", "612913": "Congrats !!", "612972": "*congratulations wonderful*", "613088": "Thank you, I'm glad it helped!", "613091": "Thanks! Yeah there were many more things to try, but I am pretty happy with the top 1% result. \nCongrats with the gold!", "613092": "🙏 🙏 🙏 thanks for all the help, was good working with you!", "613093": "🙌", "613094": "Thanks!", "613095": "Thank you!", "613155": "I think I never see anyone with the 1st competition but can both contribute a very good working kernel which laid foundations for hundreds of others to use, and finish in top 20 like you. Congrats and thank you so much.", "613495": "Thanks!" }, "source": "existing_discussion", "complete": true, "competition_id": 14313, "competition_slug": "champs-scalar-coupling", "team_id": 3251919, "solution_url": "/competitions/champs-scalar-coupling/writeups/k-v-z-f-o-20th-place-solution-mpnn-part" }