I'm new to ML and programming in general. I've been trying to explore using 3D images based on the molecular geometry and a and placing 3D Gaussians centered on each atom to create a unique potential (image) and then using convolutional neural networks to learn the relationship between molecular geometry and coupling constant. But I've been running into trouble trying to efficiently load the data, and I'd appreciate any tips thoughts etc from more experienced programmers.
\n\nMy first thought was to make an image file for each molecule first. If I approximate the molecule as a 64x64x64 image of float32 values, then each molecule is roughly 1.05 MB. There are 130775 molecules in the training set, so that would require a little over a 130 GB to store. Then I could make a tensorflow pipeline to load the appropriate files for each minibatch, make some alterations, and then transfer to the GPU for training the network in batches. The hope was that this would be an efficient use of CPU and GPU, but it turns out I don't have 130 GB of storage atm without using one of my work computers which gets ethically dicey for a personal competition with a payout even though there's no way I'm winning a cash prize.
\n\nMy second thought was to generate a set of functions which the CPU could use to generate minibatches of images on the fly while the GPU was doing network training. So here's what I did:\n1. Wrote a program to generate an nx29x3 array of molecular coordinates, where n is the number of atoms in a training set (and 29 is the max number of atoms in any molecule)\n2. Used numpy.meshgrid to generate a 3D tensor for the X, Y and Z coordinates that was 64x64x64\n2. Given the array of coordinates, I looped over n, and for each n in a minibatch, I looped over all of the atoms in the molecule (i) as and made a gaussian centered at each location with an amplitude equal to its atomic number. I did this
\n\nR2=(X-x[i])**2 +(Y-y[i])**2 + (Z-z[i])**2\nV=atomic_number[i]*np.exp(-R2/4)
Where X, Y and Z were formed only once via a call to \nX, Y, Z = np.meshgrid(grid,grid,grid)\nBut that took a long time as well for each minibatch, meaning I was spending (significantly) more time building each dataset than actually propagating and back propagating the model.
To make things go faster, I tried using numba and decorating my functions with @njit\n That made things significantly faster, but still too slow.
\n\nTo put some numbers on it, building a 256 image minibatch of data took 56 second with the numba compiler, and if I made a 3D model with a ResNet 50 ish architecture, it took only 3s /epoch for a minibatch of 256 images .
\n\nDoes anyone have any clever ideas of how to build these 64x64x64x1 3D images on the fly with their many calls to numpy.exp fast enough that it isn't the bottleneck in this approach? Or would the best advice simply to be, get more storage space? Thanks for any suggestions or thoughts,\nJames Johns
", "messages": [ { "id": "600345", "postDate": "08/16/2019 02:17:40", "content": "I'm new to ML and programming in general. I've been trying to explore using 3D images based on the molecular geometry and a and placing 3D Gaussians centered on each atom to create a unique potential (image) and then using convolutional neural networks to learn the relationship between molecular geometry and coupling constant. But I've been running into trouble trying to efficiently load the data, and I'd appreciate any tips thoughts etc from more experienced programmers.
\n\nMy first thought was to make an image file for each molecule first. If I approximate the molecule as a 64x64x64 image of float32 values, then each molecule is roughly 1.05 MB. There are 130775 molecules in the training set, so that would require a little over a 130 GB to store. Then I could make a tensorflow pipeline to load the appropriate files for each minibatch, make some alterations, and then transfer to the GPU for training the network in batches. The hope was that this would be an efficient use of CPU and GPU, but it turns out I don't have 130 GB of storage atm without using one of my work computers which gets ethically dicey for a personal competition with a payout even though there's no way I'm winning a cash prize.
\n\nMy second thought was to generate a set of functions which the CPU could use to generate minibatches of images on the fly while the GPU was doing network training. So here's what I did:\n1. Wrote a program to generate an nx29x3 array of molecular coordinates, where n is the number of atoms in a training set (and 29 is the max number of atoms in any molecule)\n2. Used numpy.meshgrid to generate a 3D tensor for the X, Y and Z coordinates that was 64x64x64\n2. Given the array of coordinates, I looped over n, and for each n in a minibatch, I looped over all of the atoms in the molecule (i) as and made a gaussian centered at each location with an amplitude equal to its atomic number. I did this
\n\nR2=(X-x[i])**2 +(Y-y[i])**2 + (Z-z[i])**2\nV=atomic_number[i]*np.exp(-R2/4)
Where X, Y and Z were formed only once via a call to \nX, Y, Z = np.meshgrid(grid,grid,grid)\nBut that took a long time as well for each minibatch, meaning I was spending (significantly) more time building each dataset than actually propagating and back propagating the model.
To make things go faster, I tried using numba and decorating my functions with @njit\n That made things significantly faster, but still too slow.
\n\nTo put some numbers on it, building a 256 image minibatch of data took 56 second with the numba compiler, and if I made a 3D model with a ResNet 50 ish architecture, it took only 3s /epoch for a minibatch of 256 images .
\n\nDoes anyone have any clever ideas of how to build these 64x64x64x1 3D images on the fly with their many calls to numpy.exp fast enough that it isn't the bottleneck in this approach? Or would the best advice simply to be, get more storage space? Thanks for any suggestions or thoughts,\nJames Johns
", "rawMarkdown": "I'm new to ML and programming in general. I've been trying to explore using 3D images based on the molecular geometry and a and placing 3D Gaussians centered on each atom to create a unique potential (image) and then using convolutional neural networks to learn the relationship between molecular geometry and coupling constant. But I've been running into trouble trying to efficiently load the data, and I'd appreciate any tips thoughts etc from more experienced programmers. \n\n My first thought was to make an image file for each molecule first. If I approximate the molecule as a 64x64x64 image of float32 values, then each molecule is roughly 1.05 MB. There are 130775 molecules in the training set, so that would require a little over a 130 GB to store. Then I could make a tensorflow pipeline to load the appropriate files for each minibatch, make some alterations, and then transfer to the GPU for training the network in batches. The hope was that this would be an efficient use of CPU and GPU, but it turns out I don't have 130 GB of storage atm without using one of my work computers which gets ethically dicey for a personal competition with a payout even though there's no way I'm winning a cash prize.\n\nMy second thought was to generate a set of functions which the CPU could use to generate minibatches of images on the fly while the GPU was doing network training. So here's what I did:\n1. Wrote a program to generate an nx29x3 array of molecular coordinates, where n is the number of atoms in a training set (and 29 is the max number of atoms in any molecule)\n2. Used numpy.meshgrid to generate a 3D tensor for the X, Y and Z coordinates that was 64x64x64\n2. Given the array of coordinates, I looped over n, and for each n in a minibatch, I looped over all of the atoms in the molecule (i) as and made a gaussian centered at each location with an amplitude equal to its atomic number. I did this \n\n`R2=(X-x[i])**2 +(Y-y[i])**2 + (Z-z[i])**2`\n`V=atomic_number[i]*np.exp(-R2/4)`\n\nWhere X, Y and Z were formed only once via a call to \n`X, Y, Z = np.meshgrid(grid,grid,grid)`\nBut that took a long time as well for each minibatch, meaning I was spending (significantly) more time building each dataset than actually propagating and back propagating the model. \n\nTo make things go faster, I tried using numba and decorating my functions with @njit\n That made things significantly faster, but still too slow.\n\nTo put some numbers on it, building a 256 image minibatch of data took 56 second with the numba compiler, and if I made a 3D model with a ResNet 50 ish architecture, it took only 3s /epoch for a minibatch of 256 images .\n\nDoes anyone have any clever ideas of how to build these 64x64x64x1 3D images on the fly with their many calls to numpy.exp fast enough that it isn't the bottleneck in this approach? Or would the best advice simply to be, get more storage space? Thanks for any suggestions or thoughts,\nJames Johns", "votes": null }, { "id": "600743", "postDate": "08/16/2019 14:08:23", "content": "Hi James,
\n\nI would say this approach is.. ambitious. \nHow do you plan to read out the coupling values (i.e. inter-atomic values) with a 3D conv net? \nI would think that this approach is better suited to bulk values.
\n\nA lot of approaches to this problem so far encode the molecules as graphs, there by keeping all the geometric information, but in a much sparser format.
\n\nYou could still do the gaussians, i.e. add all the gausians and evaluate their values at each node (then adding it as a node feature). You can even try and learn the gaussian's parameter.
\n\nJust my 2c ;-)
", "rawMarkdown": "Hi James,\n\nI would say this approach is.. ambitious. \nHow do you plan to read out the coupling values (i.e. inter-atomic values) with a 3D conv net? \nI would think that this approach is better suited to bulk values. \n\nA lot of approaches to this problem so far encode the molecules as graphs, there by keeping all the geometric information, but in a much sparser format. \n\nYou could still do the gaussians, i.e. add all the gausians and evaluate their values at each node (then adding it as a node feature). You can even try and learn the gaussian's parameter. \n\nJust my 2c ;-)", "votes": null }, { "id": "600921", "postDate": "08/16/2019 18:39:15", "content": "Hi Fnand,\n Thanks for the comment. You're totally right about the ambition level. As a chemist, it just made sense as an extension of the Hohenberg Kohn theorems, that the energy is uniquely described by the potential energy. Several groups in chemistry have used that to try and fit molecular energies as a function of the potential energy (https://arxiv.org/ftp/arxiv/papers/1809/1809.02723.pdf which uses a U-shaped convnet based on the Hartree Fock density but also trains on the density) , and since the coupling constants are based on derivatives of the excited states (also determined by the density), it seems like this could be logically extended. So making a fake potential energy surface would let it learn a new set of rules to map from a potential energy to a coupling constant. I'm sure this prone to vast over fitting, and you're right, there is no unique information encoded by this approach other than the geometry (and what amounts to a physical law that is not important here).
\n\nMy other 2 reasons for trying scoping out this approach is that I haven't started looking into graph NN's yet, so I'm an ignoramus, and secondly to just curiosity to see what I could get as I'm finishing a Coursera course on convolutional neural networks now.
\n\nFinal two points, \n1. if you have a good tutorial or starting place, or even a favorite kernel here I'd love to start learning about how to approach it from a graph perspective (I was going to start with the references here [https://arxiv.org/pdf/1906.06256.pdf]).
\n\nAs an aside, I rewrote the code above in a little C++ with just iterating over the loops, and it took about 4.5 seconds to make a minibatch using gcc -O3 . Not sure if better algorithmic design could get closer to that number or not.
\n\nCheers,\nJames
", "rawMarkdown": "Hi Fnand,\n Thanks for the comment. You're totally right about the ambition level. As a chemist, it just made sense as an extension of the Hohenberg Kohn theorems, that the energy is uniquely described by the potential energy. Several groups in chemistry have used that to try and fit molecular energies as a function of the potential energy (https://arxiv.org/ftp/arxiv/papers/1809/1809.02723.pdf which uses a U-shaped convnet based on the Hartree Fock density but also trains on the density) , and since the coupling constants are based on derivatives of the excited states (also determined by the density), it seems like this could be logically extended. So making a fake potential energy surface would let it learn a new set of rules to map from a potential energy to a coupling constant. I'm sure this prone to vast over fitting, and you're right, there is no unique information encoded by this approach other than the geometry (and what amounts to a physical law that is not important here). \n\nMy other 2 reasons for trying scoping out this approach is that I haven't started looking into graph NN's yet, so I'm an ignoramus, and secondly to just curiosity to see what I could get as I'm finishing a Coursera course on convolutional neural networks now.\n\n\nFinal two points, \n1. if you have a good tutorial or starting place, or even a favorite kernel here I'd love to start learning about how to approach it from a graph perspective (I was going to start with the references here [https://arxiv.org/pdf/1906.06256.pdf]). \n\n2. I was using the sign of the \"potential\" to distinguish the two atoms involved in the coupling (ie positive features for the two atoms being coupled/predicted, negative for every other one). I had also adjusted the position of each molecule so that the heteroatom (for C and N cases) was located at the center of the box. A preliminary and inefficient attempt at this approach on the 1JHN set suggested that I get below a score of -1 on just that class (which should be the second hardest next to 1JHC)\n\nAs an aside, I rewrote the code above in a little C++ with just iterating over the loops, and it took about 4.5 seconds to make a minibatch using gcc -O3 . Not sure if better algorithmic design could get closer to that number or not.\n\nCheers,\nJames", "votes": null }, { "id": "601282", "postDate": "08/17/2019 10:40:20", "content": "Hi James,
\n\nInteresting! As an aside, I don't think I have heard the term Hartree-Fock since undergrad.
\n\nThe best papers I have found so far were: this one and this one
\n\nAs for implementations, see this post by Heng CherKeng and my own kernel
\n\nMaybe an interesting variation would be to add a gaussian at each node (atom), and then evaluate the field at each node (i.e. essentially add field strength as a node feature). As far as I understand your description that should be roughly what you had in mind?
\n\nIf you want to get even more wild you can encode the gaussian's parameters as node features and update the field after each message passing step (i.e. have the network learn the field). I have no idea whether that would help in any way, but it seems fun!
", "rawMarkdown": "Hi James, \n\nInteresting! As an aside, I don't think I have heard the term Hartree-Fock since undergrad. \n\nThe best papers I have found so far were: [this one](https://arxiv.org/pdf/1704.01212.pdf) and [this one](https://arxiv.org/pdf/1805.10988.pdf)\n\nAs for implementations, see [this post by Heng CherKeng](https://www.kaggle.com/c/champs-scalar-coupling/discussion/93972#latest-591759) and [my own kernel](https://www.kaggle.com/fnands/1-mpnn)\n\nMaybe an interesting variation would be to add a gaussian at each node (atom), and then evaluate the field at each node (i.e. essentially add field strength as a node feature). As far as I understand your description that should be roughly what you had in mind?\n\nIf you want to get even more wild you can encode the gaussian's parameters as node features and update the field after each message passing step (i.e. have the network learn the field). I have no idea whether that would help in any way, but it seems fun!", "votes": null } ], "comments": [ { "id": 600743, "author_name": "fnands", "author_url": "", "post_date": "08/16/2019 14:08:23", "content": "Hi James,
\n\nI would say this approach is.. ambitious. \nHow do you plan to read out the coupling values (i.e. inter-atomic values) with a 3D conv net? \nI would think that this approach is better suited to bulk values.
\n\nA lot of approaches to this problem so far encode the molecules as graphs, there by keeping all the geometric information, but in a much sparser format.
\n\nYou could still do the gaussians, i.e. add all the gausians and evaluate their values at each node (then adding it as a node feature). You can even try and learn the gaussian's parameter.
\n\nJust my 2c ;-)
", "votes": null, "replies": [ { "id": 600921, "author_name": "jjohnsumn", "author_url": "", "post_date": "08/16/2019 18:39:15", "content": "Hi Fnand,\n Thanks for the comment. You're totally right about the ambition level. As a chemist, it just made sense as an extension of the Hohenberg Kohn theorems, that the energy is uniquely described by the potential energy. Several groups in chemistry have used that to try and fit molecular energies as a function of the potential energy (https://arxiv.org/ftp/arxiv/papers/1809/1809.02723.pdf which uses a U-shaped convnet based on the Hartree Fock density but also trains on the density) , and since the coupling constants are based on derivatives of the excited states (also determined by the density), it seems like this could be logically extended. So making a fake potential energy surface would let it learn a new set of rules to map from a potential energy to a coupling constant. I'm sure this prone to vast over fitting, and you're right, there is no unique information encoded by this approach other than the geometry (and what amounts to a physical law that is not important here).
\n\nMy other 2 reasons for trying scoping out this approach is that I haven't started looking into graph NN's yet, so I'm an ignoramus, and secondly to just curiosity to see what I could get as I'm finishing a Coursera course on convolutional neural networks now.
\n\nFinal two points, \n1. if you have a good tutorial or starting place, or even a favorite kernel here I'd love to start learning about how to approach it from a graph perspective (I was going to start with the references here [https://arxiv.org/pdf/1906.06256.pdf]).
\n\nAs an aside, I rewrote the code above in a little C++ with just iterating over the loops, and it took about 4.5 seconds to make a minibatch using gcc -O3 . Not sure if better algorithmic design could get closer to that number or not.
\n\nCheers,\nJames
", "votes": null, "replies": [] }, { "id": 601282, "author_name": "fnands", "author_url": "", "post_date": "08/17/2019 10:40:20", "content": "Hi James,
\n\nInteresting! As an aside, I don't think I have heard the term Hartree-Fock since undergrad.
\n\nThe best papers I have found so far were: this one and this one
\n\nAs for implementations, see this post by Heng CherKeng and my own kernel
\n\nMaybe an interesting variation would be to add a gaussian at each node (atom), and then evaluate the field at each node (i.e. essentially add field strength as a node feature). As far as I understand your description that should be roughly what you had in mind?
\n\nIf you want to get even more wild you can encode the gaussian's parameters as node features and update the field after each message passing step (i.e. have the network learn the field). I have no idea whether that would help in any way, but it seems fun!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "600345": "I'm new to ML and programming in general. I've been trying to explore using 3D images based on the molecular geometry and a and placing 3D Gaussians centered on each atom to create a unique potential (image) and then using convolutional neural networks to learn the relationship between molecular geometry and coupling constant. But I've been running into trouble trying to efficiently load the data, and I'd appreciate any tips thoughts etc from more experienced programmers. \n\n My first thought was to make an image file for each molecule first. If I approximate the molecule as a 64x64x64 image of float32 values, then each molecule is roughly 1.05 MB. There are 130775 molecules in the training set, so that would require a little over a 130 GB to store. Then I could make a tensorflow pipeline to load the appropriate files for each minibatch, make some alterations, and then transfer to the GPU for training the network in batches. The hope was that this would be an efficient use of CPU and GPU, but it turns out I don't have 130 GB of storage atm without using one of my work computers which gets ethically dicey for a personal competition with a payout even though there's no way I'm winning a cash prize.\n\nMy second thought was to generate a set of functions which the CPU could use to generate minibatches of images on the fly while the GPU was doing network training. So here's what I did:\n1. Wrote a program to generate an nx29x3 array of molecular coordinates, where n is the number of atoms in a training set (and 29 is the max number of atoms in any molecule)\n2. Used numpy.meshgrid to generate a 3D tensor for the X, Y and Z coordinates that was 64x64x64\n2. Given the array of coordinates, I looped over n, and for each n in a minibatch, I looped over all of the atoms in the molecule (i) as and made a gaussian centered at each location with an amplitude equal to its atomic number. I did this \n\n`R2=(X-x[i])**2 +(Y-y[i])**2 + (Z-z[i])**2`\n`V=atomic_number[i]*np.exp(-R2/4)`\n\nWhere X, Y and Z were formed only once via a call to \n`X, Y, Z = np.meshgrid(grid,grid,grid)`\nBut that took a long time as well for each minibatch, meaning I was spending (significantly) more time building each dataset than actually propagating and back propagating the model. \n\nTo make things go faster, I tried using numba and decorating my functions with @njit\n That made things significantly faster, but still too slow.\n\nTo put some numbers on it, building a 256 image minibatch of data took 56 second with the numba compiler, and if I made a 3D model with a ResNet 50 ish architecture, it took only 3s /epoch for a minibatch of 256 images .\n\nDoes anyone have any clever ideas of how to build these 64x64x64x1 3D images on the fly with their many calls to numpy.exp fast enough that it isn't the bottleneck in this approach? Or would the best advice simply to be, get more storage space? Thanks for any suggestions or thoughts,\nJames Johns", "600743": "Hi James,\n\nI would say this approach is.. ambitious. \nHow do you plan to read out the coupling values (i.e. inter-atomic values) with a 3D conv net? \nI would think that this approach is better suited to bulk values. \n\nA lot of approaches to this problem so far encode the molecules as graphs, there by keeping all the geometric information, but in a much sparser format. \n\nYou could still do the gaussians, i.e. add all the gausians and evaluate their values at each node (then adding it as a node feature). You can even try and learn the gaussian's parameter. \n\nJust my 2c ;-)", "600921": "Hi Fnand,\n Thanks for the comment. You're totally right about the ambition level. As a chemist, it just made sense as an extension of the Hohenberg Kohn theorems, that the energy is uniquely described by the potential energy. Several groups in chemistry have used that to try and fit molecular energies as a function of the potential energy (https://arxiv.org/ftp/arxiv/papers/1809/1809.02723.pdf which uses a U-shaped convnet based on the Hartree Fock density but also trains on the density) , and since the coupling constants are based on derivatives of the excited states (also determined by the density), it seems like this could be logically extended. So making a fake potential energy surface would let it learn a new set of rules to map from a potential energy to a coupling constant. I'm sure this prone to vast over fitting, and you're right, there is no unique information encoded by this approach other than the geometry (and what amounts to a physical law that is not important here). \n\nMy other 2 reasons for trying scoping out this approach is that I haven't started looking into graph NN's yet, so I'm an ignoramus, and secondly to just curiosity to see what I could get as I'm finishing a Coursera course on convolutional neural networks now.\n\n\nFinal two points, \n1. if you have a good tutorial or starting place, or even a favorite kernel here I'd love to start learning about how to approach it from a graph perspective (I was going to start with the references here [https://arxiv.org/pdf/1906.06256.pdf]). \n\n2. I was using the sign of the \"potential\" to distinguish the two atoms involved in the coupling (ie positive features for the two atoms being coupled/predicted, negative for every other one). I had also adjusted the position of each molecule so that the heteroatom (for C and N cases) was located at the center of the box. A preliminary and inefficient attempt at this approach on the 1JHN set suggested that I get below a score of -1 on just that class (which should be the second hardest next to 1JHC)\n\nAs an aside, I rewrote the code above in a little C++ with just iterating over the loops, and it took about 4.5 seconds to make a minibatch using gcc -O3 . Not sure if better algorithmic design could get closer to that number or not.\n\nCheers,\nJames", "601282": "Hi James, \n\nInteresting! As an aside, I don't think I have heard the term Hartree-Fock since undergrad. \n\nThe best papers I have found so far were: [this one](https://arxiv.org/pdf/1704.01212.pdf) and [this one](https://arxiv.org/pdf/1805.10988.pdf)\n\nAs for implementations, see [this post by Heng CherKeng](https://www.kaggle.com/c/champs-scalar-coupling/discussion/93972#latest-591759) and [my own kernel](https://www.kaggle.com/fnands/1-mpnn)\n\nMaybe an interesting variation would be to add a gaussian at each node (atom), and then evaluate the field at each node (i.e. essentially add field strength as a node feature). As far as I understand your description that should be roughly what you had in mind?\n\nIf you want to get even more wild you can encode the gaussian's parameters as node features and update the field after each message passing step (i.e. have the network learn the field). I have no idea whether that would help in any way, but it seems fun!" }, "source": "meta" }