Hi everyone,\nHere's a brief writeup of the method we used for the #1 entry. I was hoping to post this a bit sooner, and apologies for the delay.
\n\nWe have posted the code for our method, all of which is available under MIT license at:\nhttps://github.com/boschresearch/BCAI_kaggle_CHAMPS\nThe main code for the model is available in the src/ directory, though for those interested, the models/ directory contains slight variants on this code that were used in the ensemble (mainly earlier versions of the same architecture) so that you can recreate the predictions exactly.
First, a little bit of background on the team. This project was done at Bosch Research, specifically as collaboration between two groups, one at Bosch Corporate Research, and one at the Bosch Center for AI (BCAI). Our team consisted of both some ML experts and domain experts. To introduce our team:
\n\nJonathan Mailoa and Mordechai Kornbluth are both Research Engineers working out of the Boston lab of Bosch Research. They are domain experts on DFT and ML approach to molecular simulation, and have worked a great deal on molecular modeling, including lately some work with GNNs.
Myself (Zico Kolter, I'm a faculty member working in machine learning at CMU, but work in industry one day a week at BCAI in Pittsburgh), Devin Willmott (Research Scienst at BCAI), and Shaojie Bai (my student at CMU, but doing this while while interning at BCAI) were all coming to the competition from the ML side. I had actually done a bit of (pre-deep-learning, so ancient history) work in ML for molecular modeling, though we didn't end up using many of those methods.
Our overall approach is what I would call a kind of \"soft\" graph transformer. We wrote the model all from scratch for this work, instead of building upon any existing code. The model processes an entire molecule at once, simultaneously making a prediction for each of the scalar couplings in the molecule (we hadn't considered the per-atom approach that Quantum Uncertainty used, and frankly it sounds like that may be a pretty competitive approach, given that they did nearly as well with much less physical information).
\n\nUnlike a traditional graph model, though, we're really processing the data as more of a \"meta-graph\". In constrast to most graph methods for molecules, where atoms are nodes and bonds are edges, in our graph each atom, bond (both chemical bonds, and non-chemical bonds, i.e., just pairs of atoms are included in the model), and even triplets or quads, if desired, all becomes nodes for the graph transformer. This means that each molecule has on the order of ~500 nodes (depending on whether we include all the bonds or not, or whether we include triplets or quads, which only would be included for chemcial bonds). At each layer of the network, we maintain an embedding of for each node in the graph, of dimension d ~= 600-750 in most of our models.
\n\nFollowing the standard transformer architectures, at each layer of the network, we use self-attention layer that mixes the embeddings between the nodes. The \"standard\" scaled self-attention layer from the transformer paper would be something like (forgive the latex-esq notation formatted as code ... I'm entirely unprepared to describe model architectures without being able to write some form of equation):
\n\nZ' = W_1 Z softmax(Z^T W_2^T W_3 Z)
where W_1, W_2, and W_3 are weights of the layer. However, following the general practice of graph transformer architectures, we instead use a term
\n\nZ' = W_1 Z softmax(Z^T W_2^T W_3 Z - gamma*D)
where D is a distance matrix defined by the graph. For a \"hard\" graph transformer, this will work like the mask in normal self-attention layers, and be infinite for nodes that are not connected, and zero for nodes that are connected (and the gamma term would be fixed to one, say). In our \"soft\" version of the graph transformer, however, D was just the squared distance matrix between nodes in the graph, and gamma was a learnable parameters: as gamma went to zero, this would become a standard transformer with no notion of distance between objects, whereas as gamma went to infinity, it would become a hard graph transformer. To be even more precise, in the final architecture we used a multi-head version of this self-attention layers, as is also common in transformer models.
\n\nAs a final note, for this to work, we needed to define a distance measure between all the nodes in the graph. For e.g., atom-to-atom distances, we just used the actual distance between atoms, for atom-to-bond distances, we would use the the minimum distance from the atom to the two atoms in the bond, with similar extensions for triplets, quads, etc.
\n\nAfter the self-attention layer, we used the normal fully-connected and layer-norm layers standard to transformer architectures, and used models of depth ranging from 14-16 (depending on available memory). After the final embeddings, we had separate heads that would predict the final scalar coupling for the nodes that corresponded to fairs for which we needed the coupling value, using a simple two layer MLP for each type (or actually, for several sub-types of the bonds, which we'll mention below).
\n\nThe input representation (i.e., the first-layer embeddings for all nodes in the network), \nAs our input representation, for each type of node in the network, we would include a kind of hierarchical embedding, where were had different levels of specificity for the different atoms, bonds, etc.
\n\nAs an example, for each bond (again, really meaning just a pair of atoms ... I'm referring to pairs generally as bonds even if they are not chemical bonds in the molecule), we described it in terms of the two atoms belonging to the bond, but also in in terms of the number of bonds that each atom would have. Thus, each bond could be described by multiple given types at subtypes: first by just the type of atoms in the bond, then by the type and total number of bonds that each atom had, and then by a few additional properties such as the bond order, etc.
\n\nThis lead to substantially more coupling \"types\" than just the 8 that were used in the competition, and we actually had separate final layers for 33 different types of bonds, rather than the 8 in the competition (for instance, the 1JCH had very different properties depending on the number of bonds the C atom had), which definitely improved our predictions slightly.
\n\nIn addition to the \"discrete\" embedding, each node type would have associated with it one or two scalar constants that we would embed with a Fourier encoding, much like positional encoding in a standard sequential Tranformer model. For atoms, this consisted of the partial charge of the atom, as given by the OpenBabel library (*correction: original post said this was from RDKit, but RDKit was used for bond orders and conenctions, whereas OpenBabel was used for charges), just using some simple rules based upon graph structure; for bonds it was the distance between the two atoms; for triplets the angle between the center atom and the other others; and for quads the dihedral angle between the two planes formed by the center bond and the two other bonds (quads didn't end up helping too much for this particular task, though, so were left out most of our final models).
\n\nIn this end, we trained 13 models that we used for the final ensemble, which basically just corresponded to different iterations and versions of the same basic structure (at times we also included a few models based upon a more standard graph neural network approach from the PyTorch Geometric library, though they weren't included in the final ensemble). We timed about 4 final models to complete on the final day of competition, including the model which eventually got the best performance, which is why we managed to sneak into the top spot on the very last day. As I had mentioned in my last post, there really wasn't anything that much happening during the last ~4 days where we moved up the rankings, nor were we \"holding anything back\": our models simply kept improving each day, and we'd submit our best version of the ensemble, which kept bumping us up day by day.
\n\nOur best single model got about -3.08 on the public leaderboard, which makes me actually quite surprised, given that Quantum Uncertainty's best model was substantially better. But I think the fact that we predicted entire molecules at once actually may have increased the variance of predictions across all molecules, but therefore also seemingly made it work much better with the ensembling several different models. By taking a straight median across predictions from the best models, for instance, we could get to the ~-3.22 range, and with a slightly more involved blending scheme (using the median of all 13 models to determine which 9 models seemed best, then taking the mean of a few different medians of the different model predictions), we were able to achieve our score of -3.245 on the private leaderboard.
\n\nI want to thank the CHAMPS team for putting on an amazing competition. As many others have pointed out, the stability between the private/public leaderboards demonstrates an understanding of how to run a machine learning contest that sadly seems to be missing from many of the other contests I looked over on Kaggle previously.
\n\nI also again want to thank the Quantum Uncertainty team, who as I mentioned before, were our goalposts the entire competition. After reading their solution I'm coming away more convinced about Transformers as the architecture that's going to be dominant across many different domains, not just sequence models (despite the fact that I will never, ever, forgive the original paper for the monstrosity that is the \"query\", \"key\", and \"value\" terminology for self-attention layers ;-) ). I also think their per-atom transformer is an awesome idea, and something I wish we had thought of ... I think most likely it took us using a lot of domain knowledge and engineering to make back up the difference that their per-atom approach got. And while it's wild to me that a non-rotationally invariant model would do so well (since we only used distance as a feature at the bond level, our model is rotationally invariant), it's impossible to argue with results. Their model is excellent, and I think it actually goes to show there is substantial room for improvement still in the performance we can get on this task.
\n\nThanks again,\nZico, Devin, Shaojie, Jonathan, and Mordechai
", "messages": [ { "id": 612823, "postDate": "2019-08-30T03:02:09.963Z", "content": "Hi everyone,\nHere's a brief writeup of the method we used for the #1 entry. I was hoping to post this a bit sooner, and apologies for the delay.
\n\nWe have posted the code for our method, all of which is available under MIT license at:\nhttps://github.com/boschresearch/BCAI_kaggle_CHAMPS\nThe main code for the model is available in the src/ directory, though for those interested, the models/ directory contains slight variants on this code that were used in the ensemble (mainly earlier versions of the same architecture) so that you can recreate the predictions exactly.
First, a little bit of background on the team. This project was done at Bosch Research, specifically as collaboration between two groups, one at Bosch Corporate Research, and one at the Bosch Center for AI (BCAI). Our team consisted of both some ML experts and domain experts. To introduce our team:
\n\nJonathan Mailoa and Mordechai Kornbluth are both Research Engineers working out of the Boston lab of Bosch Research. They are domain experts on DFT and ML approach to molecular simulation, and have worked a great deal on molecular modeling, including lately some work with GNNs.
Myself (Zico Kolter, I'm a faculty member working in machine learning at CMU, but work in industry one day a week at BCAI in Pittsburgh), Devin Willmott (Research Scienst at BCAI), and Shaojie Bai (my student at CMU, but doing this while while interning at BCAI) were all coming to the competition from the ML side. I had actually done a bit of (pre-deep-learning, so ancient history) work in ML for molecular modeling, though we didn't end up using many of those methods.
Our overall approach is what I would call a kind of \"soft\" graph transformer. We wrote the model all from scratch for this work, instead of building upon any existing code. The model processes an entire molecule at once, simultaneously making a prediction for each of the scalar couplings in the molecule (we hadn't considered the per-atom approach that Quantum Uncertainty used, and frankly it sounds like that may be a pretty competitive approach, given that they did nearly as well with much less physical information).
\n\nUnlike a traditional graph model, though, we're really processing the data as more of a \"meta-graph\". In constrast to most graph methods for molecules, where atoms are nodes and bonds are edges, in our graph each atom, bond (both chemical bonds, and non-chemical bonds, i.e., just pairs of atoms are included in the model), and even triplets or quads, if desired, all becomes nodes for the graph transformer. This means that each molecule has on the order of ~500 nodes (depending on whether we include all the bonds or not, or whether we include triplets or quads, which only would be included for chemcial bonds). At each layer of the network, we maintain an embedding of for each node in the graph, of dimension d ~= 600-750 in most of our models.
\n\nFollowing the standard transformer architectures, at each layer of the network, we use self-attention layer that mixes the embeddings between the nodes. The \"standard\" scaled self-attention layer from the transformer paper would be something like (forgive the latex-esq notation formatted as code ... I'm entirely unprepared to describe model architectures without being able to write some form of equation):
\n\nZ' = W_1 Z softmax(Z^T W_2^T W_3 Z)
where W_1, W_2, and W_3 are weights of the layer. However, following the general practice of graph transformer architectures, we instead use a term
\n\nZ' = W_1 Z softmax(Z^T W_2^T W_3 Z - gamma*D)
where D is a distance matrix defined by the graph. For a \"hard\" graph transformer, this will work like the mask in normal self-attention layers, and be infinite for nodes that are not connected, and zero for nodes that are connected (and the gamma term would be fixed to one, say). In our \"soft\" version of the graph transformer, however, D was just the squared distance matrix between nodes in the graph, and gamma was a learnable parameters: as gamma went to zero, this would become a standard transformer with no notion of distance between objects, whereas as gamma went to infinity, it would become a hard graph transformer. To be even more precise, in the final architecture we used a multi-head version of this self-attention layers, as is also common in transformer models.
\n\nAs a final note, for this to work, we needed to define a distance measure between all the nodes in the graph. For e.g., atom-to-atom distances, we just used the actual distance between atoms, for atom-to-bond distances, we would use the the minimum distance from the atom to the two atoms in the bond, with similar extensions for triplets, quads, etc.
\n\nAfter the self-attention layer, we used the normal fully-connected and layer-norm layers standard to transformer architectures, and used models of depth ranging from 14-16 (depending on available memory). After the final embeddings, we had separate heads that would predict the final scalar coupling for the nodes that corresponded to fairs for which we needed the coupling value, using a simple two layer MLP for each type (or actually, for several sub-types of the bonds, which we'll mention below).
\n\nThe input representation (i.e., the first-layer embeddings for all nodes in the network), \nAs our input representation, for each type of node in the network, we would include a kind of hierarchical embedding, where were had different levels of specificity for the different atoms, bonds, etc.
\n\nAs an example, for each bond (again, really meaning just a pair of atoms ... I'm referring to pairs generally as bonds even if they are not chemical bonds in the molecule), we described it in terms of the two atoms belonging to the bond, but also in in terms of the number of bonds that each atom would have. Thus, each bond could be described by multiple given types at subtypes: first by just the type of atoms in the bond, then by the type and total number of bonds that each atom had, and then by a few additional properties such as the bond order, etc.
\n\nThis lead to substantially more coupling \"types\" than just the 8 that were used in the competition, and we actually had separate final layers for 33 different types of bonds, rather than the 8 in the competition (for instance, the 1JCH had very different properties depending on the number of bonds the C atom had), which definitely improved our predictions slightly.
\n\nIn addition to the \"discrete\" embedding, each node type would have associated with it one or two scalar constants that we would embed with a Fourier encoding, much like positional encoding in a standard sequential Tranformer model. For atoms, this consisted of the partial charge of the atom, as given by the OpenBabel library (*correction: original post said this was from RDKit, but RDKit was used for bond orders and conenctions, whereas OpenBabel was used for charges), just using some simple rules based upon graph structure; for bonds it was the distance between the two atoms; for triplets the angle between the center atom and the other others; and for quads the dihedral angle between the two planes formed by the center bond and the two other bonds (quads didn't end up helping too much for this particular task, though, so were left out most of our final models).
\n\nIn this end, we trained 13 models that we used for the final ensemble, which basically just corresponded to different iterations and versions of the same basic structure (at times we also included a few models based upon a more standard graph neural network approach from the PyTorch Geometric library, though they weren't included in the final ensemble). We timed about 4 final models to complete on the final day of competition, including the model which eventually got the best performance, which is why we managed to sneak into the top spot on the very last day. As I had mentioned in my last post, there really wasn't anything that much happening during the last ~4 days where we moved up the rankings, nor were we \"holding anything back\": our models simply kept improving each day, and we'd submit our best version of the ensemble, which kept bumping us up day by day.
\n\nOur best single model got about -3.08 on the public leaderboard, which makes me actually quite surprised, given that Quantum Uncertainty's best model was substantially better. But I think the fact that we predicted entire molecules at once actually may have increased the variance of predictions across all molecules, but therefore also seemingly made it work much better with the ensembling several different models. By taking a straight median across predictions from the best models, for instance, we could get to the ~-3.22 range, and with a slightly more involved blending scheme (using the median of all 13 models to determine which 9 models seemed best, then taking the mean of a few different medians of the different model predictions), we were able to achieve our score of -3.245 on the private leaderboard.
\n\nI want to thank the CHAMPS team for putting on an amazing competition. As many others have pointed out, the stability between the private/public leaderboards demonstrates an understanding of how to run a machine learning contest that sadly seems to be missing from many of the other contests I looked over on Kaggle previously.
\n\nI also again want to thank the Quantum Uncertainty team, who as I mentioned before, were our goalposts the entire competition. After reading their solution I'm coming away more convinced about Transformers as the architecture that's going to be dominant across many different domains, not just sequence models (despite the fact that I will never, ever, forgive the original paper for the monstrosity that is the \"query\", \"key\", and \"value\" terminology for self-attention layers ;-) ). I also think their per-atom transformer is an awesome idea, and something I wish we had thought of ... I think most likely it took us using a lot of domain knowledge and engineering to make back up the difference that their per-atom approach got. And while it's wild to me that a non-rotationally invariant model would do so well (since we only used distance as a feature at the bond level, our model is rotationally invariant), it's impossible to argue with results. Their model is excellent, and I think it actually goes to show there is substantial room for improvement still in the performance we can get on this task.
\n\nThanks again,\nZico, Devin, Shaojie, Jonathan, and Mordechai
", "rawMarkdown": "Hi everyone,\nHere's a brief writeup of the method we used for the #1 entry. I was hoping to post this a bit sooner, and apologies for the delay.\n\n##Update: 9/13/19\nWe have posted the code for our method, all of which is available under MIT license at:\nhttps://github.com/boschresearch/BCAI_kaggle_CHAMPS\nThe main code for the model is available in the `src/` directory, though for those interested, the `models/` directory contains slight variants on this code that were used in the ensemble (mainly earlier versions of the same architecture) so that you can recreate the predictions exactly.\n\n## Introduction\nFirst, a little bit of background on the team. This project was done at Bosch Research, specifically as collaboration between two groups, one at Bosch Corporate Research, and one at the Bosch Center for AI (BCAI). Our team consisted of both some ML experts and domain experts. To introduce our team:\n\n- Jonathan Mailoa and Mordechai Kornbluth are both Research Engineers working out of the Boston lab of Bosch Research. They are domain experts on DFT and ML approach to molecular simulation, and have worked a great deal on molecular modeling, including lately some work with GNNs.\n\n- Myself (Zico Kolter, I'm a faculty member working in machine learning at CMU, but work in industry one day a week at BCAI in Pittsburgh), Devin Willmott (Research Scienst at BCAI), and Shaojie Bai (my student at CMU, but doing this while while interning at BCAI) were all coming to the competition from the ML side. I had actually done a bit of (pre-deep-learning, so ancient history) work in ML for molecular modeling, though we didn't end up using many of those methods.\n\n\n##Overall architecture\nOur overall approach is what I would call a kind of \"soft\" graph transformer. We wrote the model all from scratch for this work, instead of building upon any existing code. The model processes an entire molecule at once, simultaneously making a prediction for each of the scalar couplings in the molecule (we hadn't considered the per-atom approach that Quantum Uncertainty used, and frankly it sounds like that may be a pretty competitive approach, given that they did nearly as well with much less physical information).\n\nUnlike a traditional graph model, though, we're really processing the data as more of a \"meta-graph\". In constrast to most graph methods for molecules, where atoms are nodes and bonds are edges, in our graph each atom, bond (both chemical bonds, and non-chemical bonds, i.e., just pairs of atoms are included in the model), and even triplets or quads, if desired, all becomes nodes for the graph transformer. This means that each molecule has on the order of ~500 nodes (depending on whether we include all the bonds or not, or whether we include triplets or quads, which only would be included for chemcial bonds). At each layer of the network, we maintain an embedding of for each node in the graph, of dimension d ~= 600-750 in most of our models.\n\nFollowing the standard transformer architectures, at each layer of the network, we use self-attention layer that mixes the embeddings between the nodes. The \"standard\" scaled self-attention layer from the transformer paper would be something like (forgive the latex-esq notation formatted as code ... I'm entirely unprepared to describe model architectures without being able to write some form of equation):\n\n`Z' = W_1 Z softmax(Z^T W_2^T W_3 Z)`\n\nwhere W_1, W_2, and W_3 are weights of the layer. However, following the general practice of graph transformer architectures, we instead use a term\n\n`Z' = W_1 Z softmax(Z^T W_2^T W_3 Z - gamma*D)`\n\nwhere D is a distance matrix defined by the graph. For a \"hard\" graph transformer, this will work like the mask in normal self-attention layers, and be infinite for nodes that are not connected, and zero for nodes that are connected (and the gamma term would be fixed to one, say). In our \"soft\" version of the graph transformer, however, D was just the squared distance matrix between nodes in the graph, and gamma was a learnable parameters: as gamma went to zero, this would become a standard transformer with no notion of distance between objects, whereas as gamma went to infinity, it would become a hard graph transformer. To be even more precise, in the final architecture we used a multi-head version of this self-attention layers, as is also common in transformer models.\n\nAs a final note, for this to work, we needed to define a distance measure between all the nodes in the graph. For e.g., atom-to-atom distances, we just used the actual distance between atoms, for atom-to-bond distances, we would use the the minimum distance from the atom to the two atoms in the bond, with similar extensions for triplets, quads, etc.\n\nAfter the self-attention layer, we used the normal fully-connected and layer-norm layers standard to transformer architectures, and used models of depth ranging from 14-16 (depending on available memory). After the final embeddings, we had separate heads that would predict the final scalar coupling for the nodes that corresponded to fairs for which we needed the coupling value, using a simple two layer MLP for each type (or actually, for several sub-types of the bonds, which we'll mention below).\n\n## Input features and embeddings\nThe input representation (i.e., the first-layer embeddings for all nodes in the network), \nAs our input representation, for each type of node in the network, we would include a kind of hierarchical embedding, where were had different levels of specificity for the different atoms, bonds, etc.\n\nAs an example, for each bond (again, really meaning just a pair of atoms ... I'm referring to pairs generally as bonds even if they are not chemical bonds in the molecule), we described it in terms of the two atoms belonging to the bond, but also in in terms of the number of bonds that each atom would have. Thus, each bond could be described by multiple given types at subtypes: first by just the type of atoms in the bond, then by the type and total number of bonds that each atom had, and then by a few additional properties such as the bond order, etc.\n\nThis lead to substantially more coupling \"types\" than just the 8 that were used in the competition, and we actually had separate final layers for 33 different types of bonds, rather than the 8 in the competition (for instance, the 1JCH had very different properties depending on the number of bonds the C atom had), which definitely improved our predictions slightly.\n\nIn addition to the \"discrete\" embedding, each node type would have associated with it one or two scalar constants that we would embed with a Fourier encoding, much like positional encoding in a standard sequential Tranformer model. For atoms, this consisted of the partial charge of the atom, as given by the OpenBabel library (*correction: original post said this was from RDKit, but RDKit was used for bond orders and conenctions, whereas OpenBabel was used for charges), just using some simple rules based upon graph structure; for bonds it was the distance between the two atoms; for triplets the angle between the center atom and the other others; and for quads the dihedral angle between the two planes formed by the center bond and the two other bonds (quads didn't end up helping too much for this particular task, though, so were left out most of our final models).\n\n## Ensembling\nIn this end, we trained 13 models that we used for the final ensemble, which basically just corresponded to different iterations and versions of the same basic structure (at times we also included a few models based upon a more standard graph neural network approach from the PyTorch Geometric library, though they weren't included in the final ensemble). We timed about 4 final models to complete on the final day of competition, including the model which eventually got the best performance, which is why we managed to sneak into the top spot on the very last day. As I had mentioned in my last post, there really wasn't anything that much happening during the last ~4 days where we moved up the rankings, nor were we \"holding anything back\": our models simply kept improving each day, and we'd submit our best version of the ensemble, which kept bumping us up day by day.\n\nOur best single model got about -3.08 on the public leaderboard, which makes me actually quite surprised, given that Quantum Uncertainty's best model was substantially better. But I think the fact that we predicted entire molecules at once actually may have increased the variance of predictions across all molecules, but therefore also seemingly made it work much better with the ensembling several different models. By taking a straight median across predictions from the best models, for instance, we could get to the ~-3.22 range, and with a slightly more involved blending scheme (using the median of all 13 models to determine which 9 models seemed best, then taking the mean of a few different medians of the different model predictions), we were able to achieve our score of -3.245 on the private leaderboard.\n\n## Other random notes\n- We used small amounts of dropout at each layer, as in standard transformer models, though found that it was best to use a very small amount of dropout.\n- At the very end of the competition, we did find that for our model a kind of cutout procedure (where we would randomly drop out two atoms from the network, plus all bonds, triplets, etc, that contained this atom), worked as a very effective regularizer.\n- We didn't use QM7/QM9 or in fact any of the extra data that was included in the competition besides the structures and train/test files (so just atoms and bonds).\n- We used RDKit/xyz2mol and a few other packages to parse the atomic structure to e.g., the bond and neighbor configuration feautres. Jonathan had a post about this earlier listing the packages we used.\n- I'm not even going to attempt to list all the things we tried that didn't work, but there was lots :-). Quad / dihedral angle information, for instance, actually seemed to _hurt_ generalization performance, as did including simple Coulomb forces at the bond level.\n- As you'd expect, we had a fair amount of compute resources for the work. Most of the models were trained on 4x RTX2080 Ti systems (we had 5 of these available through the month we were working on the project), with a handful also trained on six single V100s we got access to in the last week.\n\n## Final thoughts\nI want to thank the CHAMPS team for putting on an amazing competition. As many others have pointed out, the stability between the private/public leaderboards demonstrates an understanding of how to run a machine learning contest that sadly seems to be missing from many of the other contests I looked over on Kaggle previously.\n\nI also again want to thank the Quantum Uncertainty team, who as I mentioned before, were our goalposts the entire competition. After reading their solution I'm coming away more convinced about Transformers as the architecture that's going to be dominant across many different domains, not just sequence models (despite the fact that I will never, ever, forgive the original paper for the monstrosity that is the \"query\", \"key\", and \"value\" terminology for self-attention layers ;-) ). I also think their per-atom transformer is an awesome idea, and something I wish we had thought of ... I think most likely it took us using a _lot_ of domain knowledge and engineering to make back up the difference that their per-atom approach got. And while it's wild to me that a non-rotationally invariant model would do so well (since we only used distance as a feature at the bond level, our model is rotationally invariant), it's impossible to argue with results. Their model is excellent, and I think it actually goes to show there is substantial room for improvement still in the performance we can get on this task.\n\nThanks again,\nZico, Devin, Shaojie, Jonathan, and Mordechai", "votes": 147 }, { "id": 613111, "postDate": "2019-08-30T08:16:42.923Z", "content": "Great write up, and impressive that you built a transformer with priors extracted from domain knowledge.
\n\n\n\n\ndespite the fact that I will never, ever, forgive the original paper for the monstrosity that is the \"query\", \"key\", and \"value\" terminology for self-attention layers ;-)
\n
Exactly my thought! This is more a naming retrofit than anything else.
\n\nRe: rotational invariance, it was needed for us and helped with our first arch (pointnet-inspired), but didn't add much in the Atomic Transformer, I believe because the modules are already more or less canonically oriented. We added rotations during training and didn't up much.
\n\nIm surprised that cutout (I called it knockout b/c I didn't know this tech already has a name ;-) ) worked for you. Maybe we didn't train it right or missed hyperparams b/c this is one of the things I was sure it would work.
\n\nFinally, very cool (and simple, yet effective) ensembling strategy. In retrospect, this is one of our weakest spots.
\n\nCongratulations!
", "rawMarkdown": "Great write up, and impressive that you built a transformer with priors extracted from domain knowledge.\n\n> despite the fact that I will never, ever, forgive the original paper for the monstrosity that is the \"query\", \"key\", and \"value\" terminology for self-attention layers ;-) \n\nExactly my thought! This is more a naming retrofit than anything else.\n\nRe: rotational invariance, it was needed for us and helped with our first arch (pointnet-inspired), but didn't add much in the Atomic Transformer, I believe because the modules are already more or less canonically oriented. We added rotations during training and didn't up much.\n\nIm surprised that cutout (I called it knockout b/c I didn't know this tech already has a name ;-) ) worked for you. Maybe we didn't train it right or missed hyperparams b/c this is one of the things I was sure it would work.\n\nFinally, very cool (and simple, yet effective) ensembling strategy. In retrospect, this is one of our weakest spots.\n\nCongratulations!", "votes": 9 }, { "id": 612941, "postDate": "2019-08-30T05:29:53.463Z", "content": "Hi, Zico, first congrats on getting to the top. Just a heads-up that even though Kaggle doesn't have inline LaTeX support, equation environment $$ $$ is good:\n$$\n \\left(-{\\frac {\\hbar ^{2}}{2m}}\\nabla ^{2}+v(\\mathbf {r} )\\right)\\phi _{i}(\\mathbf {r} )=\\varepsilon _{i}\\phi _{i}(\\mathbf {r} )\n$$
Please enlighten us with more formulas!
", "rawMarkdown": "Hi, Zico, first congrats on getting to the top. Just a heads-up that even though Kaggle doesn't have inline LaTeX support, equation environment `$$ $$` is good:\n$$\n \\left(-{\\frac {\\hbar ^{2}}{2m}}\\nabla ^{2}+v(\\mathbf {r} )\\right)\\phi _{i}(\\mathbf {r} )=\\varepsilon _{i}\\phi _{i}(\\mathbf {r} )\n$$\n\nPlease enlighten us with more formulas!", "votes": 4 }, { "id": 630586, "postDate": "2019-09-20T12:33:56.880Z", "content": "Congratulations! Thanks for sharing. It's a pity that I miss this competition, Right now I am working on this problem, and I wonder if it is possible to get the test-set scalar coupling constant, so that I can test my model in a faster speed. Thanks a lot, congrats once again.
", "rawMarkdown": "Congratulations! Thanks for sharing. It's a pity that I miss this competition, Right now I am working on this problem, and I wonder if it is possible to get the test-set scalar coupling constant, so that I can test my model in a faster speed. Thanks a lot, congrats once again.", "votes": 1, "replies": [ { "id": 634894, "postDate": "2019-09-26T22:58:14.117Z", "content": "Do you have the option to do 'late submission' to check your answers against the test set answers? I think no one aside from the organizer has it.
", "rawMarkdown": "Do you have the option to do 'late submission' to check your answers against the test set answers? I think no one aside from the organizer has it.", "votes": 1 }, { "id": 635305, "postDate": "2019-09-27T11:02:17.983Z", "content": "If I understand your question then I am going to say YES. If you do a late submission you will get the results for both the public and private test sets
\n\nexample of late submission
\n\nPrivate Score, Public Score\n1.49953, 1.50156
\n\nor are you asking if you can download and have it for yourself?
", "rawMarkdown": "If I understand your question then I am going to say YES. If you do a late submission you will get the results for both the public and private test sets\n\nexample of late submission\n\n**Private Score, Public Score**\n1.49953, 1.50156\n\n\nor are you asking if you can download and have it for yourself?" } ] }, { "id": 625948, "postDate": "2019-09-13T16:06:08.637Z", "content": "@kelvinhe @manojprabhaakr @zhenlanwang and everyone else asking for details and source code -- the post was updated today with the github link.
", "rawMarkdown": "@kelvinhe @manojprabhaakr @zhenlanwang and everyone else asking for details and source code -- the post was updated today with the github link.", "votes": 1, "replies": [ { "id": 625955, "postDate": "2019-09-13T16:16:34.337Z", "content": "Thanks for sharing the code. Congrats once again.
", "rawMarkdown": "Thanks for sharing the code. Congrats once again. ", "votes": 1 }, { "id": 626005, "postDate": "2019-09-13T17:17:11.887Z", "content": "Thanks for sharing the code :D
", "rawMarkdown": "Thanks for sharing the code :D" }, { "id": 628267, "postDate": "2019-09-17T03:42:43.807Z", "content": "@mordechai1\nThanks for sharing. Have you ever tried the simplest feature such as only atom type and distance? Then, how much score is estimated for it?
", "rawMarkdown": "@mordechai1\nThanks for sharing. Have you ever tried the simplest feature such as only atom type and distance? Then, how much score is estimated for it?" } ] }, { "id": 613660, "postDate": "2019-08-30T17:56:33.480Z", "content": "Congrats on this amazing work. Could you please share the code for us to have a sneak peak on the exact solution?.
", "rawMarkdown": "Congrats on this amazing work. Could you please share the code for us to have a sneak peak on the exact solution?. ", "votes": 1 }, { "id": 613032, "postDate": "2019-08-30T07:05:17.613Z", "content": "Wow, very impressive solutions :) \nWe were really surprised about your performance. So fast step up!\nCongratulations!
", "rawMarkdown": "Wow, very impressive solutions :) \nWe were really surprised about your performance. So fast step up!\nCongratulations!", "votes": 1 }, { "id": 630592, "postDate": "2019-09-20T12:53:38.987Z", "content": "Hello, first of all congratulations! \nI am interested in the time your whole model would take to preprocess + predict for one molecule (let's say with 10 atoms) the scalar coupling constant between all atom pairs in the molecule.\nIn other words, it would be very interesting to know if your model can be useful for physicians (to replace their old and very heavy computational way to measure these constants).\nBecause as you said, with such amount of compute resources to build this model, I think the question is natural.
", "rawMarkdown": "Hello, first of all congratulations! \nI am interested in the time your whole model would take to preprocess + predict for one molecule (let's say with 10 atoms) the scalar coupling constant between all atom pairs in the molecule.\nIn other words, it would be very interesting to know if your model can be useful for physicians (to replace their old and very heavy computational way to measure these constants).\nBecause as you said, with such amount of compute resources to build this model, I think the question is natural.", "replies": [ { "id": 634891, "postDate": "2019-09-26T22:54:32.723Z", "content": "Thanks!\nTo pre-process the 140k molecules or so, it took ~1 hour or less. To do prediction on the test set (around 60k molecules), it took 10 minutes. So to process 1 molecule from scratch (geometry -> scalar coupling constants), it should take you around 0.03 - 0.04 s.
", "rawMarkdown": "Thanks!\nTo pre-process the 140k molecules or so, it took ~1 hour or less. To do prediction on the test set (around 60k molecules), it took 10 minutes. So to process 1 molecule from scratch (geometry -> scalar coupling constants), it should take you around 0.03 - 0.04 s.", "votes": 1 } ] }, { "id": 3242036, "postDate": "2025-07-05T13:25:12.707Z", "content": "great knowledge, Thank you or sharing this.
", "rawMarkdown": "great knowledge, Thank you or sharing this.\n" }, { "id": 2941036, "postDate": "2024-07-30T15:49:06.260Z", "content": "@zkolter and team, your hybrid approach to predicting molecular properties is impressive!
\nThe detailed explanation of your \"soft\" graph transformer model, including its innovative use of distance-based embeddings and multi-head self-attention, highlights the complexity and depth of your solution. The thorough breakdown of your methods and the transparent sharing of your code under the MIT license is incredibly valuable for the community.
Your approach of processing entire molecules as a \"meta-graph\" rather than individual nodes is particularly intriguing, and your success in the competition speaks to the effectiveness of this strategy. It's also great to see your acknowledgment of the Quantum Uncertainty team's work and your insights on future improvements. Thanks for sharing this comprehensive write-up and for contributing to the advancement of molecular modeling techniques.
", "rawMarkdown": "@zkolter and team, your hybrid approach to predicting molecular properties is impressive! \nThe detailed explanation of your \"soft\" graph transformer model, including its innovative use of distance-based embeddings and multi-head self-attention, highlights the complexity and depth of your solution. The thorough breakdown of your methods and the transparent sharing of your code under the MIT license is incredibly valuable for the community. \n\nYour approach of processing entire molecules as a \"meta-graph\" rather than individual nodes is particularly intriguing, and your success in the competition speaks to the effectiveness of this strategy. It's also great to see your acknowledgment of the Quantum Uncertainty team's work and your insights on future improvements. Thanks for sharing this comprehensive write-up and for contributing to the advancement of molecular modeling techniques." }, { "id": 631067, "postDate": "2019-09-21T09:22:59.053Z", "content": "Congrats!\nDid you train on AWS?
", "rawMarkdown": "Congrats!\nDid you train on AWS? ", "replies": [ { "id": 634893, "postDate": "2019-09-26T22:56:36.067Z", "content": "No, all the training were done on local workstation / HPC cluster GPUs
", "rawMarkdown": "No, all the training were done on local workstation / HPC cluster GPUs" } ] }, { "id": 617331, "postDate": "2019-09-04T02:36:47.117Z", "content": "Congratulations! The idea is very impressive.
", "rawMarkdown": "Congratulations! The idea is very impressive." }, { "id": 616482, "postDate": "2019-09-03T07:02:29.243Z", "content": "Congratulations @zkolter and Team. Thank you for the detailed writeup.
", "rawMarkdown": "Congratulations @zkolter and Team. Thank you for the detailed writeup." }, { "id": 616029, "postDate": "2019-09-02T16:01:17.793Z", "content": "Congrats Team, and thanks for sharing your solution approach.
", "rawMarkdown": "Congrats Team, and thanks for sharing your solution approach." }, { "id": 615173, "postDate": "2019-09-01T15:02:17.520Z", "content": "Congratulations. Could you please share the detailed features you use and which kind of models for training ?
", "rawMarkdown": "Congratulations. Could you please share the detailed features you use and which kind of models for training ? " }, { "id": 614671, "postDate": "2019-08-31T21:48:03.813Z", "content": "Great to see Kaggle connecting ML enthusiasts and domain experts π₯ π
", "rawMarkdown": "Great to see Kaggle connecting ML enthusiasts and domain experts π₯ π \n" }, { "id": 614493, "postDate": "2019-08-31T15:09:49.040Z", "content": "Amazing work and Congratulations to you and your team !
", "rawMarkdown": "Amazing work and Congratulations to you and your team !" }, { "id": 613626, "postDate": "2019-08-30T17:10:26.657Z", "content": "Congrats for the amazing solution! Thank you for sharing. @zkolter Could you share the source code with us?
", "rawMarkdown": "Congrats for the amazing solution! Thank you for sharing. @zkolter Could you share the source code with us?" }, { "id": 613584, "postDate": "2019-08-30T16:30:24.213Z", "content": "Thanks so much for the share. I understand there is a phenomenal amount of work and expertise behind a first position in this kind of competition - this is very humbling and inspiring at the same time. Congrats to you and your teammates @zkolter @jerrybai1995 @jpmailoa @mordechai1 @devster81
", "rawMarkdown": "Thanks so much for the share. I understand there is a phenomenal amount of work and expertise behind a first position in this kind of competition - this is very humbling and inspiring at the same time. Congrats to you and your teammates @zkolter @jerrybai1995 @jpmailoa @mordechai1 @devster81 " }, { "id": 613255, "postDate": "2019-08-30T11:04:37.770Z", "content": "Great job!
", "rawMarkdown": "Great job!" }, { "id": 613211, "postDate": "2019-08-30T09:59:46.303Z", "content": "Thanks, and congrats for the impressive LB progression.
", "rawMarkdown": "Thanks, and congrats for the impressive LB progression." }, { "id": 613162, "postDate": "2019-08-30T09:15:10.367Z", "content": "Very impressive. Congratulations!
", "rawMarkdown": "Very impressive. Congratulations!" }, { "id": 613099, "postDate": "2019-08-30T08:07:58.377Z", "content": "Very cool approach, and not really what I was expecting! Congratulations!
", "rawMarkdown": "Very cool approach, and not really what I was expecting! Congratulations!" }, { "id": 613050, "postDate": "2019-08-30T07:28:58.123Z", "content": "Great Job! Congratulations!
", "rawMarkdown": "Great Job! Congratulations!" }, { "id": 612954, "postDate": "2019-08-30T05:42:43.950Z", "content": "Congratulations
", "rawMarkdown": "Congratulations" }, { "id": 612923, "postDate": "2019-08-30T05:10:25.070Z", "content": "Congrats to your team and thank you for sharing your solution @zkolter .
", "rawMarkdown": "Congrats to your team and thank you for sharing your solution @zkolter ." }, { "id": 612837, "postDate": "2019-08-30T03:19:10.673Z", "content": "Congratulations! Amazing solutions! :-)
", "rawMarkdown": "Congratulations! Amazing solutions! :-)" }, { "id": 612832, "postDate": "2019-08-30T03:16:31.680Z", "content": "Great effort - a very mature approach to a difficult problem!
", "rawMarkdown": "Great effort - a very mature approach to a difficult problem!" }, { "id": 612827, "postDate": "2019-08-30T03:08:34.457Z", "content": "Huge congrats!it's a sweet victory of applied machine learning.
", "rawMarkdown": "Huge congrats!it's a sweet victory of applied machine learning." }, { "id": 619122, "postDate": "2019-09-05T21:01:49.487Z", "rawMarkdown": "", "isDeleted": true }, { "id": 613127, "postDate": "2019-08-30T08:35:44.203Z", "content": "Congratulations! Thank you for sharing.
", "rawMarkdown": "Congratulations! Thank you for sharing.", "votes": 2 }, { "id": 627689, "postDate": "2019-09-16T09:00:05.220Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" } ], "comments": [ { "id": 613111, "author_name": "AndrΓ©s Miguel Torrubia SΓ‘ez", "author_url": "", "post_date": "2019-08-30T08:16:42.923000", "content": "Great write up, and impressive that you built a transformer with priors extracted from domain knowledge.
\n\n\n\n\ndespite the fact that I will never, ever, forgive the original paper for the monstrosity that is the \"query\", \"key\", and \"value\" terminology for self-attention layers ;-)
\n
Exactly my thought! This is more a naming retrofit than anything else.
\n\nRe: rotational invariance, it was needed for us and helped with our first arch (pointnet-inspired), but didn't add much in the Atomic Transformer, I believe because the modules are already more or less canonically oriented. We added rotations during training and didn't up much.
\n\nIm surprised that cutout (I called it knockout b/c I didn't know this tech already has a name ;-) ) worked for you. Maybe we didn't train it right or missed hyperparams b/c this is one of the things I was sure it would work.
\n\nFinally, very cool (and simple, yet effective) ensembling strategy. In retrospect, this is one of our weakest spots.
\n\nCongratulations!
", "votes": 9, "replies": [] }, { "id": 612941, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2019-08-30T05:29:53.463000", "content": "Hi, Zico, first congrats on getting to the top. Just a heads-up that even though Kaggle doesn't have inline LaTeX support, equation environment $$ $$ is good:\n$$\n \\left(-{\\frac {\\hbar ^{2}}{2m}}\\nabla ^{2}+v(\\mathbf {r} )\\right)\\phi _{i}(\\mathbf {r} )=\\varepsilon _{i}\\phi _{i}(\\mathbf {r} )\n$$
Please enlighten us with more formulas!
", "votes": 4, "replies": [] }, { "id": 630586, "author_name": "Caiqing Jian", "author_url": "", "post_date": "2019-09-20T12:33:56.880000", "content": "Congratulations! Thanks for sharing. It's a pity that I miss this competition, Right now I am working on this problem, and I wonder if it is possible to get the test-set scalar coupling constant, so that I can test my model in a faster speed. Thanks a lot, congrats once again.
", "votes": 1, "replies": [ { "id": 634894, "author_name": "Jonathan Mailoa", "author_url": "", "post_date": "2019-09-26T22:58:14.117000", "content": "Do you have the option to do 'late submission' to check your answers against the test set answers? I think no one aside from the organizer has it.
", "votes": 1, "replies": [] }, { "id": 635305, "author_name": "rilesdg3", "author_url": "", "post_date": "2019-09-27T11:02:17.983000", "content": "If I understand your question then I am going to say YES. If you do a late submission you will get the results for both the public and private test sets
\n\nexample of late submission
\n\nPrivate Score, Public Score\n1.49953, 1.50156
\n\nor are you asking if you can download and have it for yourself?
", "votes": 0, "replies": [] } ] }, { "id": 625948, "author_name": "Mordechai", "author_url": "", "post_date": "2019-09-13T16:06:08.637000", "content": "@kelvinhe @manojprabhaakr @zhenlanwang and everyone else asking for details and source code -- the post was updated today with the github link.
", "votes": 1, "replies": [ { "id": 625955, "author_name": "Manoj Prabhakar", "author_url": "", "post_date": "2019-09-13T16:16:34.337000", "content": "Thanks for sharing the code. Congrats once again.
", "votes": 1, "replies": [] }, { "id": 626005, "author_name": "Zhenlan", "author_url": "", "post_date": "2019-09-13T17:17:11.887000", "content": "Thanks for sharing the code :D
", "votes": 0, "replies": [] }, { "id": 628267, "author_name": "AkiraSosa", "author_url": "", "post_date": "2019-09-17T03:42:43.807000", "content": "@mordechai1\nThanks for sharing. Have you ever tried the simplest feature such as only atom type and distance? Then, how much score is estimated for it?
", "votes": 0, "replies": [] } ] }, { "id": 613660, "author_name": "Manoj Prabhakar", "author_url": "", "post_date": "2019-08-30T17:56:33.480000", "content": "Congrats on this amazing work. Could you please share the code for us to have a sneak peak on the exact solution?.
", "votes": 1, "replies": [] }, { "id": 613032, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-08-30T07:05:17.613000", "content": "Wow, very impressive solutions :) \nWe were really surprised about your performance. So fast step up!\nCongratulations!
", "votes": 1, "replies": [] }, { "id": 630592, "author_name": "MaxR2D", "author_url": "", "post_date": "2019-09-20T12:53:38.987000", "content": "Hello, first of all congratulations! \nI am interested in the time your whole model would take to preprocess + predict for one molecule (let's say with 10 atoms) the scalar coupling constant between all atom pairs in the molecule.\nIn other words, it would be very interesting to know if your model can be useful for physicians (to replace their old and very heavy computational way to measure these constants).\nBecause as you said, with such amount of compute resources to build this model, I think the question is natural.
", "votes": 0, "replies": [ { "id": 634891, "author_name": "Jonathan Mailoa", "author_url": "", "post_date": "2019-09-26T22:54:32.723000", "content": "Thanks!\nTo pre-process the 140k molecules or so, it took ~1 hour or less. To do prediction on the test set (around 60k molecules), it took 10 minutes. So to process 1 molecule from scratch (geometry -> scalar coupling constants), it should take you around 0.03 - 0.04 s.
", "votes": 1, "replies": [] } ] }, { "id": 3242036, "author_name": "Abdul Wadood", "author_url": "", "post_date": "2025-07-05T13:25:12.707000", "content": "great knowledge, Thank you or sharing this.
", "votes": 0, "replies": [] }, { "id": 2941036, "author_name": "", "author_url": "", "post_date": "2024-07-30T15:49:06.260000", "content": "@zkolter and team, your hybrid approach to predicting molecular properties is impressive!
\nThe detailed explanation of your \"soft\" graph transformer model, including its innovative use of distance-based embeddings and multi-head self-attention, highlights the complexity and depth of your solution. The thorough breakdown of your methods and the transparent sharing of your code under the MIT license is incredibly valuable for the community.
Your approach of processing entire molecules as a \"meta-graph\" rather than individual nodes is particularly intriguing, and your success in the competition speaks to the effectiveness of this strategy. It's also great to see your acknowledgment of the Quantum Uncertainty team's work and your insights on future improvements. Thanks for sharing this comprehensive write-up and for contributing to the advancement of molecular modeling techniques.
", "votes": 0, "replies": [] }, { "id": 631067, "author_name": "fulowa", "author_url": "", "post_date": "2019-09-21T09:22:59.053000", "content": "Congrats!\nDid you train on AWS?
", "votes": 0, "replies": [ { "id": 634893, "author_name": "Jonathan Mailoa", "author_url": "", "post_date": "2019-09-26T22:56:36.067000", "content": "No, all the training were done on local workstation / HPC cluster GPUs
", "votes": 0, "replies": [] } ] }, { "id": 617331, "author_name": "Magichuang", "author_url": "", "post_date": "2019-09-04T02:36:47.117000", "content": "Congratulations! The idea is very impressive.
", "votes": 0, "replies": [] }, { "id": 616482, "author_name": "Deep Chatterjee", "author_url": "", "post_date": "2019-09-03T07:02:29.243000", "content": "Congratulations @zkolter and Team. Thank you for the detailed writeup.
", "votes": 0, "replies": [] }, { "id": 616029, "author_name": "Vijay Ram", "author_url": "", "post_date": "2019-09-02T16:01:17.793000", "content": "Congrats Team, and thanks for sharing your solution approach.
", "votes": 0, "replies": [] }, { "id": 615173, "author_name": "Kelvin", "author_url": "", "post_date": "2019-09-01T15:02:17.520000", "content": "Congratulations. Could you please share the detailed features you use and which kind of models for training ?
", "votes": 0, "replies": [] }, { "id": 614671, "author_name": "d_eremeev", "author_url": "", "post_date": "2019-08-31T21:48:03.813000", "content": "Great to see Kaggle connecting ML enthusiasts and domain experts π₯ π
", "votes": 0, "replies": [] }, { "id": 614493, "author_name": "Jack nick", "author_url": "", "post_date": "2019-08-31T15:09:49.040000", "content": "Amazing work and Congratulations to you and your team !
", "votes": 0, "replies": [] }, { "id": 613626, "author_name": "Zhenlan", "author_url": "", "post_date": "2019-08-30T17:10:26.657000", "content": "Congrats for the amazing solution! Thank you for sharing. @zkolter Could you share the source code with us?
", "votes": 0, "replies": [] }, { "id": 613584, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-30T16:30:24.213000", "content": "Thanks so much for the share. I understand there is a phenomenal amount of work and expertise behind a first position in this kind of competition - this is very humbling and inspiring at the same time. Congrats to you and your teammates @zkolter @jerrybai1995 @jpmailoa @mordechai1 @devster81
", "votes": 0, "replies": [] }, { "id": 613255, "author_name": "dan1234", "author_url": "", "post_date": "2019-08-30T11:04:37.770000", "content": "Great job!
", "votes": 0, "replies": [] }, { "id": 613211, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-30T09:59:46.303000", "content": "Thanks, and congrats for the impressive LB progression.
", "votes": 0, "replies": [] }, { "id": 613162, "author_name": "Stephen Foster", "author_url": "", "post_date": "2019-08-30T09:15:10.367000", "content": "Very impressive. Congratulations!
", "votes": 0, "replies": [] }, { "id": 613099, "author_name": "fnands", "author_url": "", "post_date": "2019-08-30T08:07:58.377000", "content": "Very cool approach, and not really what I was expecting! Congratulations!
", "votes": 0, "replies": [] }, { "id": 613050, "author_name": "Alexey Pustynnikov", "author_url": "", "post_date": "2019-08-30T07:28:58.123000", "content": "Great Job! Congratulations!
", "votes": 0, "replies": [] }, { "id": 612954, "author_name": "Tony Teng", "author_url": "", "post_date": "2019-08-30T05:42:43.950000", "content": "Congratulations
", "votes": 0, "replies": [] }, { "id": 612923, "author_name": "Joel Hanson", "author_url": "", "post_date": "2019-08-30T05:10:25.070000", "content": "Congrats to your team and thank you for sharing your solution @zkolter .
", "votes": 0, "replies": [] }, { "id": 612837, "author_name": "FGPC", "author_url": "", "post_date": "2019-08-30T03:19:10.673000", "content": "Congratulations! Amazing solutions! :-)
", "votes": 0, "replies": [] }, { "id": 612832, "author_name": "Bill Holst", "author_url": "", "post_date": "2019-08-30T03:16:31.680000", "content": "Great effort - a very mature approach to a difficult problem!
", "votes": 0, "replies": [] }, { "id": 612827, "author_name": "senkin13", "author_url": "", "post_date": "2019-08-30T03:08:34.457000", "content": "Huge congrats!it's a sweet victory of applied machine learning.
", "votes": 0, "replies": [] }, { "id": 619122, "author_name": "", "author_url": "", "post_date": "2019-09-05T21:01:49.487000", "content": "", "votes": 0, "replies": [] }, { "id": 613127, "author_name": "", "author_url": "", "post_date": "2019-08-30T08:35:44.203000", "content": "", "votes": 2, "replies": [] }, { "id": 627689, "author_name": "", "author_url": "", "post_date": "2019-09-16T09:00:05.220000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "612823": "Hi everyone,\nHere's a brief writeup of the method we used for the #1 entry. I was hoping to post this a bit sooner, and apologies for the delay.\n\n##Update: 9/13/19\nWe have posted the code for our method, all of which is available under MIT license at:\nhttps://github.com/boschresearch/BCAI_kaggle_CHAMPS\nThe main code for the model is available in the `src/` directory, though for those interested, the `models/` directory contains slight variants on this code that were used in the ensemble (mainly earlier versions of the same architecture) so that you can recreate the predictions exactly.\n\n## Introduction\nFirst, a little bit of background on the team. This project was done at Bosch Research, specifically as collaboration between two groups, one at Bosch Corporate Research, and one at the Bosch Center for AI (BCAI). Our team consisted of both some ML experts and domain experts. To introduce our team:\n\n- Jonathan Mailoa and Mordechai Kornbluth are both Research Engineers working out of the Boston lab of Bosch Research. They are domain experts on DFT and ML approach to molecular simulation, and have worked a great deal on molecular modeling, including lately some work with GNNs.\n\n- Myself (Zico Kolter, I'm a faculty member working in machine learning at CMU, but work in industry one day a week at BCAI in Pittsburgh), Devin Willmott (Research Scienst at BCAI), and Shaojie Bai (my student at CMU, but doing this while while interning at BCAI) were all coming to the competition from the ML side. I had actually done a bit of (pre-deep-learning, so ancient history) work in ML for molecular modeling, though we didn't end up using many of those methods.\n\n\n##Overall architecture\nOur overall approach is what I would call a kind of \"soft\" graph transformer. We wrote the model all from scratch for this work, instead of building upon any existing code. The model processes an entire molecule at once, simultaneously making a prediction for each of the scalar couplings in the molecule (we hadn't considered the per-atom approach that Quantum Uncertainty used, and frankly it sounds like that may be a pretty competitive approach, given that they did nearly as well with much less physical information).\n\nUnlike a traditional graph model, though, we're really processing the data as more of a \"meta-graph\". In constrast to most graph methods for molecules, where atoms are nodes and bonds are edges, in our graph each atom, bond (both chemical bonds, and non-chemical bonds, i.e., just pairs of atoms are included in the model), and even triplets or quads, if desired, all becomes nodes for the graph transformer. This means that each molecule has on the order of ~500 nodes (depending on whether we include all the bonds or not, or whether we include triplets or quads, which only would be included for chemcial bonds). At each layer of the network, we maintain an embedding of for each node in the graph, of dimension d ~= 600-750 in most of our models.\n\nFollowing the standard transformer architectures, at each layer of the network, we use self-attention layer that mixes the embeddings between the nodes. The \"standard\" scaled self-attention layer from the transformer paper would be something like (forgive the latex-esq notation formatted as code ... I'm entirely unprepared to describe model architectures without being able to write some form of equation):\n\n`Z' = W_1 Z softmax(Z^T W_2^T W_3 Z)`\n\nwhere W_1, W_2, and W_3 are weights of the layer. However, following the general practice of graph transformer architectures, we instead use a term\n\n`Z' = W_1 Z softmax(Z^T W_2^T W_3 Z - gamma*D)`\n\nwhere D is a distance matrix defined by the graph. For a \"hard\" graph transformer, this will work like the mask in normal self-attention layers, and be infinite for nodes that are not connected, and zero for nodes that are connected (and the gamma term would be fixed to one, say). In our \"soft\" version of the graph transformer, however, D was just the squared distance matrix between nodes in the graph, and gamma was a learnable parameters: as gamma went to zero, this would become a standard transformer with no notion of distance between objects, whereas as gamma went to infinity, it would become a hard graph transformer. To be even more precise, in the final architecture we used a multi-head version of this self-attention layers, as is also common in transformer models.\n\nAs a final note, for this to work, we needed to define a distance measure between all the nodes in the graph. For e.g., atom-to-atom distances, we just used the actual distance between atoms, for atom-to-bond distances, we would use the the minimum distance from the atom to the two atoms in the bond, with similar extensions for triplets, quads, etc.\n\nAfter the self-attention layer, we used the normal fully-connected and layer-norm layers standard to transformer architectures, and used models of depth ranging from 14-16 (depending on available memory). After the final embeddings, we had separate heads that would predict the final scalar coupling for the nodes that corresponded to fairs for which we needed the coupling value, using a simple two layer MLP for each type (or actually, for several sub-types of the bonds, which we'll mention below).\n\n## Input features and embeddings\nThe input representation (i.e., the first-layer embeddings for all nodes in the network), \nAs our input representation, for each type of node in the network, we would include a kind of hierarchical embedding, where were had different levels of specificity for the different atoms, bonds, etc.\n\nAs an example, for each bond (again, really meaning just a pair of atoms ... I'm referring to pairs generally as bonds even if they are not chemical bonds in the molecule), we described it in terms of the two atoms belonging to the bond, but also in in terms of the number of bonds that each atom would have. Thus, each bond could be described by multiple given types at subtypes: first by just the type of atoms in the bond, then by the type and total number of bonds that each atom had, and then by a few additional properties such as the bond order, etc.\n\nThis lead to substantially more coupling \"types\" than just the 8 that were used in the competition, and we actually had separate final layers for 33 different types of bonds, rather than the 8 in the competition (for instance, the 1JCH had very different properties depending on the number of bonds the C atom had), which definitely improved our predictions slightly.\n\nIn addition to the \"discrete\" embedding, each node type would have associated with it one or two scalar constants that we would embed with a Fourier encoding, much like positional encoding in a standard sequential Tranformer model. For atoms, this consisted of the partial charge of the atom, as given by the OpenBabel library (*correction: original post said this was from RDKit, but RDKit was used for bond orders and conenctions, whereas OpenBabel was used for charges), just using some simple rules based upon graph structure; for bonds it was the distance between the two atoms; for triplets the angle between the center atom and the other others; and for quads the dihedral angle between the two planes formed by the center bond and the two other bonds (quads didn't end up helping too much for this particular task, though, so were left out most of our final models).\n\n## Ensembling\nIn this end, we trained 13 models that we used for the final ensemble, which basically just corresponded to different iterations and versions of the same basic structure (at times we also included a few models based upon a more standard graph neural network approach from the PyTorch Geometric library, though they weren't included in the final ensemble). We timed about 4 final models to complete on the final day of competition, including the model which eventually got the best performance, which is why we managed to sneak into the top spot on the very last day. As I had mentioned in my last post, there really wasn't anything that much happening during the last ~4 days where we moved up the rankings, nor were we \"holding anything back\": our models simply kept improving each day, and we'd submit our best version of the ensemble, which kept bumping us up day by day.\n\nOur best single model got about -3.08 on the public leaderboard, which makes me actually quite surprised, given that Quantum Uncertainty's best model was substantially better. But I think the fact that we predicted entire molecules at once actually may have increased the variance of predictions across all molecules, but therefore also seemingly made it work much better with the ensembling several different models. By taking a straight median across predictions from the best models, for instance, we could get to the ~-3.22 range, and with a slightly more involved blending scheme (using the median of all 13 models to determine which 9 models seemed best, then taking the mean of a few different medians of the different model predictions), we were able to achieve our score of -3.245 on the private leaderboard.\n\n## Other random notes\n- We used small amounts of dropout at each layer, as in standard transformer models, though found that it was best to use a very small amount of dropout.\n- At the very end of the competition, we did find that for our model a kind of cutout procedure (where we would randomly drop out two atoms from the network, plus all bonds, triplets, etc, that contained this atom), worked as a very effective regularizer.\n- We didn't use QM7/QM9 or in fact any of the extra data that was included in the competition besides the structures and train/test files (so just atoms and bonds).\n- We used RDKit/xyz2mol and a few other packages to parse the atomic structure to e.g., the bond and neighbor configuration feautres. Jonathan had a post about this earlier listing the packages we used.\n- I'm not even going to attempt to list all the things we tried that didn't work, but there was lots :-). Quad / dihedral angle information, for instance, actually seemed to _hurt_ generalization performance, as did including simple Coulomb forces at the bond level.\n- As you'd expect, we had a fair amount of compute resources for the work. Most of the models were trained on 4x RTX2080 Ti systems (we had 5 of these available through the month we were working on the project), with a handful also trained on six single V100s we got access to in the last week.\n\n## Final thoughts\nI want to thank the CHAMPS team for putting on an amazing competition. As many others have pointed out, the stability between the private/public leaderboards demonstrates an understanding of how to run a machine learning contest that sadly seems to be missing from many of the other contests I looked over on Kaggle previously.\n\nI also again want to thank the Quantum Uncertainty team, who as I mentioned before, were our goalposts the entire competition. After reading their solution I'm coming away more convinced about Transformers as the architecture that's going to be dominant across many different domains, not just sequence models (despite the fact that I will never, ever, forgive the original paper for the monstrosity that is the \"query\", \"key\", and \"value\" terminology for self-attention layers ;-) ). I also think their per-atom transformer is an awesome idea, and something I wish we had thought of ... I think most likely it took us using a _lot_ of domain knowledge and engineering to make back up the difference that their per-atom approach got. And while it's wild to me that a non-rotationally invariant model would do so well (since we only used distance as a feature at the bond level, our model is rotationally invariant), it's impossible to argue with results. Their model is excellent, and I think it actually goes to show there is substantial room for improvement still in the performance we can get on this task.\n\nThanks again,\nZico, Devin, Shaojie, Jonathan, and Mordechai", "613111": "Great write up, and impressive that you built a transformer with priors extracted from domain knowledge.\n\n> despite the fact that I will never, ever, forgive the original paper for the monstrosity that is the \"query\", \"key\", and \"value\" terminology for self-attention layers ;-) \n\nExactly my thought! This is more a naming retrofit than anything else.\n\nRe: rotational invariance, it was needed for us and helped with our first arch (pointnet-inspired), but didn't add much in the Atomic Transformer, I believe because the modules are already more or less canonically oriented. We added rotations during training and didn't up much.\n\nIm surprised that cutout (I called it knockout b/c I didn't know this tech already has a name ;-) ) worked for you. Maybe we didn't train it right or missed hyperparams b/c this is one of the things I was sure it would work.\n\nFinally, very cool (and simple, yet effective) ensembling strategy. In retrospect, this is one of our weakest spots.\n\nCongratulations!", "612941": "Hi, Zico, first congrats on getting to the top. Just a heads-up that even though Kaggle doesn't have inline LaTeX support, equation environment `$$ $$` is good:\n$$\n \\left(-{\\frac {\\hbar ^{2}}{2m}}\\nabla ^{2}+v(\\mathbf {r} )\\right)\\phi _{i}(\\mathbf {r} )=\\varepsilon _{i}\\phi _{i}(\\mathbf {r} )\n$$\n\nPlease enlighten us with more formulas!", "630586": "Congratulations! Thanks for sharing. It's a pity that I miss this competition, Right now I am working on this problem, and I wonder if it is possible to get the test-set scalar coupling constant, so that I can test my model in a faster speed. Thanks a lot, congrats once again.", "625948": "@kelvinhe @manojprabhaakr @zhenlanwang and everyone else asking for details and source code -- the post was updated today with the github link.", "613660": "Congrats on this amazing work. Could you please share the code for us to have a sneak peak on the exact solution?. ", "613032": "Wow, very impressive solutions :) \nWe were really surprised about your performance. So fast step up!\nCongratulations!", "630592": "Hello, first of all congratulations! \nI am interested in the time your whole model would take to preprocess + predict for one molecule (let's say with 10 atoms) the scalar coupling constant between all atom pairs in the molecule.\nIn other words, it would be very interesting to know if your model can be useful for physicians (to replace their old and very heavy computational way to measure these constants).\nBecause as you said, with such amount of compute resources to build this model, I think the question is natural.", "3242036": "great knowledge, Thank you or sharing this.\n", "2941036": "@zkolter and team, your hybrid approach to predicting molecular properties is impressive! \nThe detailed explanation of your \"soft\" graph transformer model, including its innovative use of distance-based embeddings and multi-head self-attention, highlights the complexity and depth of your solution. The thorough breakdown of your methods and the transparent sharing of your code under the MIT license is incredibly valuable for the community. \n\nYour approach of processing entire molecules as a \"meta-graph\" rather than individual nodes is particularly intriguing, and your success in the competition speaks to the effectiveness of this strategy. It's also great to see your acknowledgment of the Quantum Uncertainty team's work and your insights on future improvements. Thanks for sharing this comprehensive write-up and for contributing to the advancement of molecular modeling techniques.", "631067": "Congrats!\nDid you train on AWS? ", "617331": "Congratulations! The idea is very impressive.", "616482": "Congratulations @zkolter and Team. Thank you for the detailed writeup.", "616029": "Congrats Team, and thanks for sharing your solution approach.", "615173": "Congratulations. Could you please share the detailed features you use and which kind of models for training ? ", "614671": "Great to see Kaggle connecting ML enthusiasts and domain experts π₯ π \n", "614493": "Amazing work and Congratulations to you and your team !", "613626": "Congrats for the amazing solution! Thank you for sharing. @zkolter Could you share the source code with us?", "613584": "Thanks so much for the share. I understand there is a phenomenal amount of work and expertise behind a first position in this kind of competition - this is very humbling and inspiring at the same time. Congrats to you and your teammates @zkolter @jerrybai1995 @jpmailoa @mordechai1 @devster81 ", "613255": "Great job!", "613211": "Thanks, and congrats for the impressive LB progression.", "613162": "Very impressive. Congratulations!", "613099": "Very cool approach, and not really what I was expecting! Congratulations!", "613050": "Great Job! Congratulations!", "612954": "Congratulations", "612923": "Congrats to your team and thank you for sharing your solution @zkolter .", "612837": "Congratulations! Amazing solutions! :-)", "612832": "Great effort - a very mature approach to a difficult problem!", "612827": "Huge congrats!it's a sweet victory of applied machine learning.", "619122": "", "613127": "Congratulations! Thank you for sharing.", "627689": "Thanks for sharing!" } }