We want to thank Kaggle and CHAMPs for organizing such an awesome competition:
\n\nNeither my teammate @pavelgonchar nor I had any previous domain expertise and we made the decision early on that we would tackle this problem using a very pure deep learning way: letting the model build the features for us, not the other way around (b/c obviously we were at a disadvantage if we tried to become quantum experts in 1 month… hence our team name 🤖Quantum Uncertainty🤖 we didn't know if our yet-to-be-developed approach was going to work).
\n\nOur solution had two major parts: 1) the input representation and 2) deep learning architecture.
\n\nInput representation
\n\nThis is in our opinion the key part: we take a molecule and a source atom and move it so the source atom is @ (0,0,0). For each molecule we create N molecule siblings (N being as many source atoms are defined for that molecule), and each molecule sibling is translated so its source is at (0,0,0).
\n\nThe x (input) are three arrays of dimension 29 (maximum number of atoms):
1) x,y,z position of each atom, \n2) atom type index (C=0, H=1, etc…)\n3) j-coupling type index (1JHC=0,'2JHH=1,etc.)
Padding is done by placing -1 in atom type index and j-coupling type for molecules which have less than 29 atoms.
The y (ground truth) is just an array of dimension 29 containing j-couplings located at target atom indices.
Note that there is no graph information nor any other manually engineered features.
\n\nData augmentation
\n\nWe did two types of data augmentation:\n- Rotations: which worked and were useful in our first attempt model: pointnet-based, but proved worthless in the final models (atomic transformer).\n- J-coupling symmetry: as described in this discussion
\n\nFirst attempt: Pointnet-inspired architecture (got up to -2.28200 LB)
\n\nOur input representation is basically a point cloud: an unordered set of elements with absolute positions x,y,z and two attributes atom type and j-coupling type. We modified the Pointnet architecture to regress j-couplings. Training was a bit unstable and we tried many variations of the architecture, swapping FC layers by linear (fixed) projections (Hadamard), adding coulomb matrix as input, etc.
While this worked OK and got us to -2.28200 LB (ensembling a few models) we felt that this architecture was limited by the extreme pooling/bottleneck operation so we decided to explore other architectures: meet the Atomic Transformer.
\n\nFinal architecture: meet the Atomic Transformer
\n\nYou may know that the recent NLP revolution is mostly due to the transformer architecture described in the Attention is all you need paper. The vanilla transformer architecture uses a very clever technique to add positional encodings that are needed for position-dependent input, such as language.
\n\nOur input representation is a set, which means we can (and should) remove positional encoding. Prior to this competition we had no experience with transformers either but there's a section in Lex Fridman MIT podcast interviewing Orion Vinyals where he mentions the inherent position invariance of a barebone transformer encoder layer. This immediately triggered the idea of using transformer layers (encoders) stacked taking as an input x,y,z (normalized but otherwise as-is), and atom type and j-coupling type embeddings; just concatenated… nothing fancy.
The dimension of the embeddings was such that the total dimension of the input vectors was d_model (as normally reference in transformer literature). We started with 256 and got immediately great results surpassing our pointnet-inspired architecture so we followed this path.
We trained a total of 14 models, with varying dimensions from 512 to 2048 and layers from 6 to 24. Each model parameter size ranged from ~12M to ~100M (biggest model).
\n\nWe trained some models from scratch, others we fine-tuned. We also fine-tuned a few models on the troublesome j-couplings: reaching -2.12 CV on 1JHC on and -2.19 CV on 1JHN.
\n\nOur best score is an ensemble of 14 models achieving private LB of -3.22349, and our best single model achieved private LB of -3.16234, again just with x,y,z, atom type and j-coupling type inputs (no QM9, etc.).
What didn't work
\n\nMany things! We tried:
\n\nNaNs despite numerous attempts to fix it.Source code
\n\nWe will make source code available once we do clean up. It's a single jupyter notebook using FastAI. Be patient.
\n\nComputational resources
\n\nWe had more ideas than computational resources, even if our computational resources were not tiny:\n- 3 x 2080 Ti + 128 Gb RAM + 16c32t processor\n- 2 x 1080 Ti + 64 Gb RAM + 8c16t processor\n- Rented 8+ 2080 Ti + 64 Gb RAM + 16c32t processor (multiple machines rented as needed)
\n\nFinal thoughts
\n\nThis was our most fun and hardest competition so far:\n- Challenging problem\n- Most teams in top 5 had domain experts (although we went domainless as part of our strategy)\n- Hungry computational resources.
\n\nEven if we lost #1 position just a few hours before competition end we feel very excited we were able to achieve such [useful results for the organizers] (https://www.kaggle.com/c/champs-scalar-coupling/discussion/98375#latest-569312). In retrospect we believe a single model of the Atomic Transformer may achieve ever better results with further training.
\n\nBest - Pavel & Andres
\n\np.s. No graph NNs. We though graphs as manually engineered features that the model can infer by itself.
", "messages": [ { "id": 612141, "postDate": "2019-08-29T16:05:42.650Z", "content": "We want to thank Kaggle and CHAMPs for organizing such an awesome competition:
\n\nNeither my teammate @pavelgonchar nor I had any previous domain expertise and we made the decision early on that we would tackle this problem using a very pure deep learning way: letting the model build the features for us, not the other way around (b/c obviously we were at a disadvantage if we tried to become quantum experts in 1 month… hence our team name 🤖Quantum Uncertainty🤖 we didn't know if our yet-to-be-developed approach was going to work).
\n\nOur solution had two major parts: 1) the input representation and 2) deep learning architecture.
\n\nInput representation
\n\nThis is in our opinion the key part: we take a molecule and a source atom and move it so the source atom is @ (0,0,0). For each molecule we create N molecule siblings (N being as many source atoms are defined for that molecule), and each molecule sibling is translated so its source is at (0,0,0).
\n\nThe x (input) are three arrays of dimension 29 (maximum number of atoms):
1) x,y,z position of each atom, \n2) atom type index (C=0, H=1, etc…)\n3) j-coupling type index (1JHC=0,'2JHH=1,etc.)
Padding is done by placing -1 in atom type index and j-coupling type for molecules which have less than 29 atoms.
The y (ground truth) is just an array of dimension 29 containing j-couplings located at target atom indices.
Note that there is no graph information nor any other manually engineered features.
\n\nData augmentation
\n\nWe did two types of data augmentation:\n- Rotations: which worked and were useful in our first attempt model: pointnet-based, but proved worthless in the final models (atomic transformer).\n- J-coupling symmetry: as described in this discussion
\n\nFirst attempt: Pointnet-inspired architecture (got up to -2.28200 LB)
\n\nOur input representation is basically a point cloud: an unordered set of elements with absolute positions x,y,z and two attributes atom type and j-coupling type. We modified the Pointnet architecture to regress j-couplings. Training was a bit unstable and we tried many variations of the architecture, swapping FC layers by linear (fixed) projections (Hadamard), adding coulomb matrix as input, etc.
While this worked OK and got us to -2.28200 LB (ensembling a few models) we felt that this architecture was limited by the extreme pooling/bottleneck operation so we decided to explore other architectures: meet the Atomic Transformer.
\n\nFinal architecture: meet the Atomic Transformer
\n\nYou may know that the recent NLP revolution is mostly due to the transformer architecture described in the Attention is all you need paper. The vanilla transformer architecture uses a very clever technique to add positional encodings that are needed for position-dependent input, such as language.
\n\nOur input representation is a set, which means we can (and should) remove positional encoding. Prior to this competition we had no experience with transformers either but there's a section in Lex Fridman MIT podcast interviewing Orion Vinyals where he mentions the inherent position invariance of a barebone transformer encoder layer. This immediately triggered the idea of using transformer layers (encoders) stacked taking as an input x,y,z (normalized but otherwise as-is), and atom type and j-coupling type embeddings; just concatenated… nothing fancy.
The dimension of the embeddings was such that the total dimension of the input vectors was d_model (as normally reference in transformer literature). We started with 256 and got immediately great results surpassing our pointnet-inspired architecture so we followed this path.
We trained a total of 14 models, with varying dimensions from 512 to 2048 and layers from 6 to 24. Each model parameter size ranged from ~12M to ~100M (biggest model).
\n\nWe trained some models from scratch, others we fine-tuned. We also fine-tuned a few models on the troublesome j-couplings: reaching -2.12 CV on 1JHC on and -2.19 CV on 1JHN.
\n\nOur best score is an ensemble of 14 models achieving private LB of -3.22349, and our best single model achieved private LB of -3.16234, again just with x,y,z, atom type and j-coupling type inputs (no QM9, etc.).
What didn't work
\n\nMany things! We tried:
\n\nNaNs despite numerous attempts to fix it.Source code
\n\nWe will make source code available once we do clean up. It's a single jupyter notebook using FastAI. Be patient.
\n\nComputational resources
\n\nWe had more ideas than computational resources, even if our computational resources were not tiny:\n- 3 x 2080 Ti + 128 Gb RAM + 16c32t processor\n- 2 x 1080 Ti + 64 Gb RAM + 8c16t processor\n- Rented 8+ 2080 Ti + 64 Gb RAM + 16c32t processor (multiple machines rented as needed)
\n\nFinal thoughts
\n\nThis was our most fun and hardest competition so far:\n- Challenging problem\n- Most teams in top 5 had domain experts (although we went domainless as part of our strategy)\n- Hungry computational resources.
\n\nEven if we lost #1 position just a few hours before competition end we feel very excited we were able to achieve such [useful results for the organizers] (https://www.kaggle.com/c/champs-scalar-coupling/discussion/98375#latest-569312). In retrospect we believe a single model of the Atomic Transformer may achieve ever better results with further training.
\n\nBest - Pavel & Andres
\n\np.s. No graph NNs. We though graphs as manually engineered features that the model can infer by itself.
", "rawMarkdown": "We want to thank Kaggle and CHAMPs for organizing such an awesome competition: \n\n- No leakage.\n- Same distribution in train, private and public test (very stable CV vs. LB, public LB vs. private LB). This is very relevant in other competitions given so many participants when the 3 distributions are different some winners (not all, e.g. [CPMP approach to manually align test ~ train in the Microsoft malware comp](https://www.kaggle.com/c/microsoft-malware-prediction/discussion/84069#latest-499864)) are just lucky they fit private test distribution by chance; not the case here. \n- Inspiring and useful science problem hopefully used for good purposes as described in the context (new drugs, etc.).\n\nNeither my teammate @pavelgonchar nor I had any previous domain expertise and we made the decision early on that we would tackle this problem using a very pure deep learning way: letting the model build the features for us, not the other way around (b/c obviously we were at a disadvantage if we tried to become quantum experts in 1 month… hence our team name 🤖Quantum Uncertainty🤖 we didn't know if our yet-to-be-developed approach was going to work).\n\nOur solution had two major parts: 1) the input representation and 2) deep learning architecture.\n\n**Input representation**\n\nThis is in our opinion the key part: we take a molecule and a source atom and move it so the source atom is @ (0,0,0). For each molecule we create N molecule siblings (N being as many source atoms are defined for that molecule), and each molecule sibling is translated so its source is at (0,0,0). \n\nThe `x` (input) are three arrays of dimension 29 (maximum number of atoms): \n\n1) `x,y,z` position of each atom, \n2) `atom type` index (C=0, H=1, etc…)\n3) `j-coupling type` index (1JHC=0,'2JHH=1,etc.)\n\nPadding is done by placing -1 in `atom type` index and `j-coupling type` for molecules which have less than 29 atoms.\n\nThe `y` (ground truth) is just an array of dimension 29 containing j-couplings located at target atom indices.\n\nNote that there is no graph information nor any other manually engineered features.\n\n**Data augmentation**\n\nWe did two types of data augmentation:\n- Rotations: which worked and were useful in our first attempt model: pointnet-based, but proved worthless in the final models (atomic transformer).\n- J-coupling symmetry: as described [in this discussion](https://www.kaggle.com/c/champs-scalar-coupling/discussion/94706#latest-563148)\n\n**First attempt: Pointnet-inspired architecture (got up to -2.28200 LB)**\n\nOur input representation is basically a point cloud: an unordered set of elements with absolute positions `x,y,z` and two attributes `atom type` and `j-coupling type`. We modified the [Pointnet](https://arxiv.org/abs/1612.00593) architecture to regress j-couplings. Training was a bit unstable and we tried many variations of the architecture, swapping FC layers by linear (fixed) projections (Hadamard), adding coulomb matrix as input, etc. \n\nWhile this worked OK and got us to -2.28200 LB (ensembling a few models) we felt that this architecture was limited by the extreme pooling/bottleneck operation so we decided to explore other architectures: meet the Atomic Transformer.\n\n**Final architecture: meet the Atomic Transformer**\n\nYou may know that the recent NLP revolution is mostly due to the transformer architecture described in the [Attention is all you need paper](https://arxiv.org/abs/1706.03762). The vanilla transformer architecture uses a very clever technique to add positional encodings that are needed for position-dependent input, such as language. \n\nOur input representation is a set, which means we can (and should) remove positional encoding. Prior to this competition we had no experience with transformers either but there's a section in [Lex Fridman MIT podcast interviewing Orion Vinyals](https://www.youtube.com/watch?v=Kedt2or9xlo) where he mentions the inherent position invariance of a barebone transformer encoder layer. This immediately triggered the idea of using transformer layers (encoders) stacked taking as an input `x,y,z` (normalized but otherwise as-is), and `atom type` and `j-coupling type` embeddings; just concatenated… nothing fancy. \n\nThe dimension of the embeddings was such that the total dimension of the input vectors was `d_model` (as normally reference in transformer literature). We started with 256 and got immediately great results surpassing our pointnet-inspired architecture so we followed this path.\n\nWe trained a total of 14 models, with varying dimensions from 512 to 2048 and layers from 6 to 24. Each model parameter size ranged from ~12M to ~100M (biggest model).\n\nWe trained some models from scratch, others we fine-tuned. We also fine-tuned a few models on the troublesome j-couplings: reaching -2.12 CV on 1JHC on and -2.19 CV on 1JHN.\n\nOur best score is an ensemble of 14 models achieving private LB of -3.22349, and our best single model achieved private LB of -3.16234, again just with `x,y,z`, `atom type` and `j-coupling type` inputs (no QM9, etc.).\n\n**What didn't work**\n\nMany things! We tried:\n\n- Multi-task learning using contributions and other organization provided values.\n- Dropout: We tried multiple attempts to add dropout at various stages (embeddings, encoder layers, pre-decoder, etc.). None of them worked.\n- Knock-out: We added a variation in which as input we deleted 10% of the input atoms, the idea being that the model would build an internal representation of the missing atoms. Surprisingly this worked in that the model still converged nicely but failed to reduce train ~ val gap.\n- Rotations and TTA in Atomic Transformer: it didn't reduce train ~ val gap and didn't produce meaningful TTA gains.\n- Deep decoder: Our decoder is just a projection of ~ the model dimension to 1 (scalar coupling). We tried adding more expressive power to the decoder but this didn't help.\n- Fp16 training. This worked for models of dimensions 256 but as training evolved gave `NaN`s despite numerous attempts to fix it.\n\n**Source code**\n\nWe will make source code available once we do clean up. It's a single jupyter notebook using FastAI. Be patient.\n\n**Computational resources**\n\nWe had more ideas than computational resources, even if our computational resources were not tiny:\n- 3 x 2080 Ti + 128 Gb RAM + 16c32t processor\n- 2 x 1080 Ti + 64 Gb RAM + 8c16t processor\n- Rented 8+ 2080 Ti + 64 Gb RAM + 16c32t processor (multiple machines rented as needed)\n\n**Final thoughts**\n\nThis was our most fun and hardest competition so far:\n- Challenging problem\n- Most teams in top 5 had domain experts (although we went *domainless* as part of our strategy)\n- Hungry computational resources.\n\nEven if we lost #1 position just a few hours before competition end we feel very excited we were able to achieve such [useful results for the organizers] (https://www.kaggle.com/c/champs-scalar-coupling/discussion/98375#latest-569312). In retrospect we believe a single model of the Atomic Transformer may achieve ever better results with further training.\n\nBest - Pavel & Andres\n\np.s. No graph NNs. We though graphs as manually engineered features that the model can infer by itself.", "votes": 176 }, { "id": 679449, "postDate": "2019-11-22T19:36:03.223Z", "content": "Hi Andres, very inspired by your approach. Is the notebook available for us to study, could you point me to it?
", "rawMarkdown": "Hi Andres, very inspired by your approach. Is the notebook available for us to study, could you point me to it?", "votes": 3 }, { "id": 612258, "postDate": "2019-08-29T17:23:27.777Z", "content": "Very impressive how having machine learning expertise and no domain knowledge can get you a podium, it truely amazes me. Congrats @antorsae
", "rawMarkdown": "Very impressive how having machine learning expertise and no domain knowledge can get you a podium, it truely amazes me. Congrats @antorsae ", "votes": 3 }, { "id": 612153, "postDate": "2019-08-29T16:13:52.203Z", "content": "Thanks, very impressive solution. In a way my team mates approached the problem in a way similar to yours, by representing every atom position relative to the two atoms for which scaling is computed. But you way is a bit different as you keep the coordinates, while they worked with distances and angles. That using coordinates works that well is a bit of a surprise to me. If I were the organizers then i would have applied random rotations to the data... This said, again, congrats on the approach. I'll need to reread this to full grasp it.
\n\nI forgot the most important: congrats on the end result and the incredible lead during the competition.
", "rawMarkdown": "Thanks, very impressive solution. In a way my team mates approached the problem in a way similar to yours, by representing every atom position relative to the two atoms for which scaling is computed. But you way is a bit different as you keep the coordinates, while they worked with distances and angles. That using coordinates works that well is a bit of a surprise to me. If I were the organizers then i would have applied random rotations to the data... This said, again, congrats on the approach. I'll need to reread this to full grasp it. \n\nI forgot the most important: congrats on the end result and the incredible lead during the competition.", "votes": 3, "replies": [ { "id": 612189, "postDate": "2019-08-29T16:40:23.483Z", "content": "And thanks for citing my solution in Malware! ;)
", "rawMarkdown": "And thanks for citing my solution in Malware! ;)", "votes": 2 }, { "id": 612391, "postDate": "2019-08-29T18:59:13.183Z", "content": "Yeah! I liked the notion of alining distributions. In the IEEE fraud one I am trying to do the same automatically with deep learning instead of manual alignment. Not proven yet.
", "rawMarkdown": "Yeah! I liked the notion of alining distributions. In the IEEE fraud one I am trying to do the same automatically with deep learning instead of manual alignment. Not proven yet.", "votes": 1 }, { "id": 614292, "postDate": "2019-08-31T10:18:51.947Z", "content": "I tried to code it, and it led to a second dataset, but it performed worse than the fully manual one I guess my pretrained NN (in my brain) worked better ;)
", "rawMarkdown": "I tried to code it, and it led to a second dataset, but it performed worse than the fully manual one I guess my pretrained NN (in my brain) worked better ;)" } ] }, { "id": 612260, "postDate": "2019-08-29T17:24:17.397Z", "content": "I just want to say that your team's work is very impressive, and I certainly look forward to reading the implementation in more detail! For one, we also use transformer architecture, but it was a graph transformer. I think Zico will make a post about it sometime soon.
", "rawMarkdown": "I just want to say that your team's work is very impressive, and I certainly look forward to reading the implementation in more detail! For one, we also use transformer architecture, but it was a graph transformer. I think Zico will make a post about it sometime soon.", "votes": 4, "replies": [ { "id": 612435, "postDate": "2019-08-29T19:43:37.003Z", "content": "Thanks! You made us sweat a lot in the final hours. It was fun tough and we learned a few lessons in those final hours.
", "rawMarkdown": "Thanks! You made us sweat a lot in the final hours. It was fun tough and we learned a few lessons in those final hours.", "votes": 1 }, { "id": 612470, "postDate": "2019-08-29T20:05:57.970Z", "content": "On the day before the last day, we were very excited to finally try surpassing you guys! But we were 0.001 short... and couldn't do anything about it until the next day. =)
\n\nSince your transformer graph does not satisfy physics requirement of rotation/translation invariance by construction, can you comment on how much data augmentation you need to do to get good result, as well as the extra training resource / compute needed correspondingly? Your single model achieving -3.16 is really quite impressive considering that your architecture does not satisfy physics rules by construction!
", "rawMarkdown": "On the day before the last day, we were very excited to finally try surpassing you guys! But we were 0.001 short... and couldn't do anything about it until the next day. =)\n\nSince your transformer graph does not satisfy physics requirement of rotation/translation invariance by construction, can you comment on how much data augmentation you need to do to get good result, as well as the extra training resource / compute needed correspondingly? Your single model achieving -3.16 is really quite impressive considering that your architecture does not satisfy physics rules by construction!", "votes": 2 } ] }, { "id": 685920, "postDate": "2019-12-02T14:26:34.703Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 1 }, { "id": 662722, "postDate": "2019-10-31T22:53:02.790Z", "content": "Congratulations! Impressed that not used graph NNs but used transformer based architectures to deal with set, together with the position translation for target atom.
", "rawMarkdown": "Congratulations! Impressed that not used graph NNs but used transformer based architectures to deal with set, together with the position translation for target atom.", "votes": 1 }, { "id": 644660, "postDate": "2019-10-09T06:35:20.663Z", "content": "Hi,
\n\nThanks a lot for sharing your solution and code. I was reading your code and I have a question: In your code atom-transfomer-good-norm-pytorch, the result under the line learner.summary() shows two emdedding layers and two dropout layers at the end. Where do they come from? I can't find them in the forward method of your AtomTorchTransformer. I apologize if the question is too detailed and thanks in advance.
", "rawMarkdown": "Hi,\n\nThanks a lot for sharing your solution and code. I was reading your code and I have a question: In your code atom-transfomer-good-norm-pytorch, the result under the line learner.summary() shows two emdedding layers and two dropout layers at the end. Where do they come from? I can't find them in the forward method of your AtomTorchTransformer. I apologize if the question is too detailed and thanks in advance.", "votes": 1, "replies": [ { "id": 644889, "postDate": "2019-10-09T13:34:21.777Z", "content": "Check this code in the AtomTransformer constructor:
\n n_atom_embedding = d_model//2\n n_type_embedding = d_model - n_atom_embedding - 3 #- 1 - 1 -1 \n self.type_embedding = nn.Embedding(len(types)+1,n_type_embedding)\n self.atom_embedding = nn.Embedding(len(atoms)+1,n_atom_embedding)\n self.drop_type, self.drop_atom = nn.Dropout(embed_p), nn.Dropout(embed_p)\n
I see. I just want to clarify one thing then. In the learner.summary(), it only shows what layers are in the neural network regardless their order, am I right? Because the order of layers in learner.summary() seems to be different from the forward propagation order in the forward method. And those two embedding layers should show up at the beginning before going to transformer.
", "rawMarkdown": "I see. I just want to clarify one thing then. In the learner.summary(), it only shows what layers are in the neural network regardless their order, am I right? Because the order of layers in learner.summary() seems to be different from the forward propagation order in the forward method. And those two embedding layers should show up at the beginning before going to transformer." }, { "id": 645427, "postDate": "2019-10-10T05:53:59.577Z", "content": "Correct
", "rawMarkdown": "Correct" }, { "id": 646125, "postDate": "2019-10-10T22:09:14.377Z", "content": "Thank you very much!
", "rawMarkdown": "Thank you very much!" } ] }, { "id": 616487, "postDate": "2019-09-03T07:10:27.467Z", "content": "Congratulations @antorsae @pavelgonchar .Thank you for the approach and waiting for the notebook. The one thing that is inspirational for me is that though you don't have any domain expertise, you understand and solve the problem using DeepLearning techniques.
", "rawMarkdown": "Congratulations @antorsae @pavelgonchar .Thank you for the approach and waiting for the notebook. The one thing that is inspirational for me is that though you don't have any domain expertise, you understand and solve the problem using DeepLearning techniques.", "votes": 1 }, { "id": 615668, "postDate": "2019-09-02T08:18:17.627Z", "content": "Congratulations, good job
", "rawMarkdown": "Congratulations, good job", "votes": 1 }, { "id": 615365, "postDate": "2019-09-01T20:35:51.600Z", "content": "Really looking forward to your solution, especially as a fellow fastai community member. Your team really inspired us to keep striving in the final weeks of the competition. Sub -3 didn't feel achievable until we saw you pull it off.
", "rawMarkdown": "Really looking forward to your solution, especially as a fellow fastai community member. Your team really inspired us to keep striving in the final weeks of the competition. Sub -3 didn't feel achievable until we saw you pull it off.", "votes": 1 }, { "id": 613390, "postDate": "2019-08-30T13:15:31.037Z", "content": "Congratulations, you guys are inspirations.
", "rawMarkdown": "Congratulations, you guys are inspirations. ", "votes": 1 }, { "id": 613261, "postDate": "2019-08-30T11:11:49.883Z", "content": "Congrats!!
", "rawMarkdown": "Congrats!!", "votes": 1 }, { "id": 613184, "postDate": "2019-08-30T09:32:05.710Z", "content": "amazing work
", "rawMarkdown": "amazing work", "votes": 1 }, { "id": 612930, "postDate": "2019-08-30T05:14:15.730Z", "content": "Congrats @antorsae and thanks for sharing.
", "rawMarkdown": "Congrats @antorsae and thanks for sharing.", "votes": 1 }, { "id": 612842, "postDate": "2019-08-30T03:23:39.580Z", "content": "Congratulations! Amazing solutions! :-)
", "rawMarkdown": "Congratulations! Amazing solutions! :-)", "votes": 1 }, { "id": 612762, "postDate": "2019-08-30T01:43:47.027Z", "content": "Congratulations! I'm very excited to learn from your solutions.
", "rawMarkdown": "Congratulations! I'm very excited to learn from your solutions.", "votes": 1 }, { "id": 612517, "postDate": "2019-08-29T20:54:15.050Z", "content": "I'm just impressed by the way of using xyz coordinates. Need to read more bout Trnasformer Encoders.
\n\nThank you @antorsae for sharing the solution and congratulation for your creativity!
", "rawMarkdown": "I'm just impressed by the way of using xyz coordinates. Need to read more bout Trnasformer Encoders.\n\nThank you @antorsae for sharing the solution and congratulation for your creativity!", "votes": 1 }, { "id": 612406, "postDate": "2019-08-29T19:14:51.297Z", "content": "Thanks for sharing!\nWhen you say \"The y (ground truth) is just an array of dimension 29 containing j-couplings located at target atom indices.\"\nWhich atom index do you place the j-coupling values, as there are two atoms participating in a coupling?
", "rawMarkdown": "Thanks for sharing!\nWhen you say \"The y (ground truth) is just an array of dimension 29 containing j-couplings located at target atom indices.\"\nWhich atom index do you place the j-coupling values, as there are two atoms participating in a coupling?", "votes": 1, "replies": [ { "id": 612430, "postDate": "2019-08-29T19:39:04.393Z", "content": "Say there are N source/reference atoms (atom_index_0 in the csv file) for a given molecule. We build N sibling molecules, and for each the GT is the array of 29 atoms (max) where we place valid j-couplings at target (atom_index_1 in the csv) indices.
Oh, so each coupling gets its own datapoint? We have one datapoint per molecule, that confused me.
", "rawMarkdown": "Oh, so each coupling gets its own datapoint? We have one datapoint per molecule, that confused me." }, { "id": 612453, "postDate": "2019-08-29T19:53:36.653Z", "content": "Yes.
\n\nWe started building a Molecule Transformer which had all couplings in one shot (diff input representation) but stopped it bc we focussed on training the Atomic Transformer instead.
", "rawMarkdown": "Yes.\n\nWe started building a Molecule Transformer which had all couplings in one shot (diff input representation) but stopped it bc we focussed on training the Atomic Transformer instead." } ] }, { "id": 612252, "postDate": "2019-08-29T17:17:01.987Z", "content": "I am not sure that I understand your Input representation. Could you give a simply example, say for H2O? Thanks
", "rawMarkdown": "I am not sure that I understand your Input representation. Could you give a simply example, say for H2O? Thanks", "votes": 1 }, { "id": 612239, "postDate": "2019-08-29T17:10:27.217Z", "content": "Congrats!! Thank you for sharing. I look forward to seeing your code :D
", "rawMarkdown": "Congrats!! Thank you for sharing. I look forward to seeing your code :D", "votes": 1 }, { "id": 613298, "postDate": "2019-08-30T12:04:02.160Z", "content": "Congratulations! You are not using graph NN. Amazing ideas! I think it would be an excellent paper if you wrote it up your ideas.
", "rawMarkdown": "Congratulations! You are not using graph NN. Amazing ideas! I think it would be an excellent paper if you wrote it up your ideas. ", "votes": 2 }, { "id": 612569, "postDate": "2019-08-29T21:36:50.697Z", "content": "looking forward to the code! Thanks for sharing
", "rawMarkdown": "looking forward to the code! Thanks for sharing", "votes": 2 }, { "id": 612501, "postDate": "2019-08-29T20:36:44.797Z", "content": "Very impressive and the right way to go with AI discovery. Like the domainless approach! Congratulations!
", "rawMarkdown": "Very impressive and the right way to go with AI discovery. Like the domainless approach! Congratulations!", "votes": 2 }, { "id": 612474, "postDate": "2019-08-29T20:09:03.203Z", "content": "Congratulations! Looking forward to exploring your code, I'm very interested in solutions without domain knowledge.\nBtw, I just realized that we are form the same city lol
", "rawMarkdown": "Congratulations! Looking forward to exploring your code, I'm very interested in solutions without domain knowledge.\nBtw, I just realized that we are form the same city lol", "votes": 2 }, { "id": 612269, "postDate": "2019-08-29T17:29:57.993Z", "content": "Really impressed to see that you were able to push a pure Tranformer Encoder to such great performances. I tried using a straightforward Transformer model as well on a set of engineered features (no raw coordinates), but wasn't able to get results anywhere near as good as yours. Like you said, I think the key is your input representation creating the N sibling molecules.
\n\nCongrats on second place and the beautiful approach.
", "rawMarkdown": "Really impressed to see that you were able to push a pure Tranformer Encoder to such great performances. I tried using a straightforward Transformer model as well on a set of engineered features (no raw coordinates), but wasn't able to get results anywhere near as good as yours. Like you said, I think the key is your input representation creating the N sibling molecules. \n\nCongrats on second place and the beautiful approach.", "votes": 2 }, { "id": 612172, "postDate": "2019-08-29T16:27:23.473Z", "content": "Congrats.....\nGreat Work...\nThanks for Sharing...!! @antorsae
", "rawMarkdown": "Congrats.....\nGreat Work...\nThanks for Sharing...!! @antorsae " }, { "id": 612432, "postDate": "2019-08-29T19:40:41.273Z", "content": "Great work, looking forward to seeing the notebook
", "rawMarkdown": "Great work, looking forward to seeing the notebook", "votes": 1 }, { "id": 781190, "postDate": "2020-03-21T02:38:02.537Z", "content": "I look forward to seeing your code :D, now too
", "rawMarkdown": "I look forward to seeing your code :D, now too", "replies": [ { "id": 823848, "postDate": "2020-04-28T00:29:40.910Z", "content": "He published his code 8 months ago in github.
", "rawMarkdown": "He published his code 8 months ago in [github](https://github.com/antorsae/champs-scalar-coupling).", "votes": 1 } ] }, { "id": 718185, "postDate": "2020-01-14T06:28:11.320Z", "content": "Impressive!
", "rawMarkdown": "Impressive!" }, { "id": 612599, "postDate": "2019-08-29T22:17:53.037Z", "content": "Awesome. Thanks for sharing. Any estimate as to how much money you guys spent on this?
", "rawMarkdown": "Awesome. Thanks for sharing. Any estimate as to how much money you guys spent on this?", "replies": [ { "id": 614181, "postDate": "2019-08-31T07:12:21.793Z", "content": "In my case I bought a 3rd 2080 Ti which I will be using more. In renting I spent ~70$ in vast.ai
", "rawMarkdown": "In my case I bought a 3rd 2080 Ti which I will be using more. In renting I spent ~70$ in vast.ai", "votes": 1 } ] }, { "id": 618474, "postDate": "2019-09-05T08:37:25.403Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 618278, "postDate": "2019-09-05T04:23:00.593Z", "content": "Congratulations and thanks for sharing
", "rawMarkdown": "Congratulations and thanks for sharing", "votes": 1 }, { "id": 616456, "postDate": "2019-09-03T06:33:03.167Z", "content": "Congrats and thanks a lot for sharing.
", "rawMarkdown": "Congrats and thanks a lot for sharing.", "votes": 1 }, { "id": 612986, "postDate": "2019-08-30T06:25:48.727Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!", "votes": 1 }, { "id": 612786, "postDate": "2019-08-30T02:19:29.727Z", "content": "Congratulations! Thanks for sharing!
", "rawMarkdown": "Congratulations! Thanks for sharing!", "votes": 1 } ], "comments": [ { "id": 679449, "author_name": "whyvr", "author_url": "", "post_date": "2019-11-22T19:36:03.223000", "content": "Hi Andres, very inspired by your approach. Is the notebook available for us to study, could you point me to it?
", "votes": 3, "replies": [] }, { "id": 612258, "author_name": "Michel ML", "author_url": "", "post_date": "2019-08-29T17:23:27.777000", "content": "Very impressive how having machine learning expertise and no domain knowledge can get you a podium, it truely amazes me. Congrats @antorsae
", "votes": 3, "replies": [] }, { "id": 612153, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-29T16:13:52.203000", "content": "Thanks, very impressive solution. In a way my team mates approached the problem in a way similar to yours, by representing every atom position relative to the two atoms for which scaling is computed. But you way is a bit different as you keep the coordinates, while they worked with distances and angles. That using coordinates works that well is a bit of a surprise to me. If I were the organizers then i would have applied random rotations to the data... This said, again, congrats on the approach. I'll need to reread this to full grasp it.
\n\nI forgot the most important: congrats on the end result and the incredible lead during the competition.
", "votes": 3, "replies": [ { "id": 612189, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-29T16:40:23.483000", "content": "And thanks for citing my solution in Malware! ;)
", "votes": 2, "replies": [] }, { "id": 612391, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2019-08-29T18:59:13.183000", "content": "Yeah! I liked the notion of alining distributions. In the IEEE fraud one I am trying to do the same automatically with deep learning instead of manual alignment. Not proven yet.
", "votes": 1, "replies": [] }, { "id": 614292, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-31T10:18:51.947000", "content": "I tried to code it, and it led to a second dataset, but it performed worse than the fully manual one I guess my pretrained NN (in my brain) worked better ;)
", "votes": 0, "replies": [] } ] }, { "id": 612260, "author_name": "Jonathan Mailoa", "author_url": "", "post_date": "2019-08-29T17:24:17.397000", "content": "I just want to say that your team's work is very impressive, and I certainly look forward to reading the implementation in more detail! For one, we also use transformer architecture, but it was a graph transformer. I think Zico will make a post about it sometime soon.
", "votes": 4, "replies": [ { "id": 612435, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2019-08-29T19:43:37.003000", "content": "Thanks! You made us sweat a lot in the final hours. It was fun tough and we learned a few lessons in those final hours.
", "votes": 1, "replies": [] }, { "id": 612470, "author_name": "Jonathan Mailoa", "author_url": "", "post_date": "2019-08-29T20:05:57.970000", "content": "On the day before the last day, we were very excited to finally try surpassing you guys! But we were 0.001 short... and couldn't do anything about it until the next day. =)
\n\nSince your transformer graph does not satisfy physics requirement of rotation/translation invariance by construction, can you comment on how much data augmentation you need to do to get good result, as well as the extra training resource / compute needed correspondingly? Your single model achieving -3.16 is really quite impressive considering that your architecture does not satisfy physics rules by construction!
", "votes": 2, "replies": [] } ] }, { "id": 685920, "author_name": "Sean McConnell", "author_url": "", "post_date": "2019-12-02T14:26:34.703000", "content": "Congrats!
", "votes": 1, "replies": [] }, { "id": 662722, "author_name": "corochann", "author_url": "", "post_date": "2019-10-31T22:53:02.790000", "content": "Congratulations! Impressed that not used graph NNs but used transformer based architectures to deal with set, together with the position translation for target atom.
", "votes": 1, "replies": [] }, { "id": 644660, "author_name": "Dondon2019", "author_url": "", "post_date": "2019-10-09T06:35:20.663000", "content": "Hi,
\n\nThanks a lot for sharing your solution and code. I was reading your code and I have a question: In your code atom-transfomer-good-norm-pytorch, the result under the line learner.summary() shows two emdedding layers and two dropout layers at the end. Where do they come from? I can't find them in the forward method of your AtomTorchTransformer. I apologize if the question is too detailed and thanks in advance.
", "votes": 1, "replies": [ { "id": 644889, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2019-10-09T13:34:21.777000", "content": "Check this code in the AtomTransformer constructor:
\n n_atom_embedding = d_model//2\n n_type_embedding = d_model - n_atom_embedding - 3 #- 1 - 1 -1 \n self.type_embedding = nn.Embedding(len(types)+1,n_type_embedding)\n self.atom_embedding = nn.Embedding(len(atoms)+1,n_atom_embedding)\n self.drop_type, self.drop_atom = nn.Dropout(embed_p), nn.Dropout(embed_p)\n
I see. I just want to clarify one thing then. In the learner.summary(), it only shows what layers are in the neural network regardless their order, am I right? Because the order of layers in learner.summary() seems to be different from the forward propagation order in the forward method. And those two embedding layers should show up at the beginning before going to transformer.
", "votes": 0, "replies": [] }, { "id": 645427, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2019-10-10T05:53:59.577000", "content": "Correct
", "votes": 0, "replies": [] }, { "id": 646125, "author_name": "Dondon2019", "author_url": "", "post_date": "2019-10-10T22:09:14.377000", "content": "Thank you very much!
", "votes": 0, "replies": [] } ] }, { "id": 616487, "author_name": "Deep Chatterjee", "author_url": "", "post_date": "2019-09-03T07:10:27.467000", "content": "Congratulations @antorsae @pavelgonchar .Thank you for the approach and waiting for the notebook. The one thing that is inspirational for me is that though you don't have any domain expertise, you understand and solve the problem using DeepLearning techniques.
", "votes": 1, "replies": [] }, { "id": 615668, "author_name": "Sems Kurtoglu", "author_url": "", "post_date": "2019-09-02T08:18:17.627000", "content": "Congratulations, good job
", "votes": 1, "replies": [] }, { "id": 615365, "author_name": "Even", "author_url": "", "post_date": "2019-09-01T20:35:51.600000", "content": "Really looking forward to your solution, especially as a fellow fastai community member. Your team really inspired us to keep striving in the final weeks of the competition. Sub -3 didn't feel achievable until we saw you pull it off.
", "votes": 1, "replies": [] }, { "id": 613390, "author_name": "Pankaj Joshi", "author_url": "", "post_date": "2019-08-30T13:15:31.037000", "content": "Congratulations, you guys are inspirations.
", "votes": 1, "replies": [] }, { "id": 613261, "author_name": "Murat Sarıbaş", "author_url": "", "post_date": "2019-08-30T11:11:49.883000", "content": "Congrats!!
", "votes": 1, "replies": [] }, { "id": 613184, "author_name": "Xiao-Chuan", "author_url": "", "post_date": "2019-08-30T09:32:05.710000", "content": "amazing work
", "votes": 1, "replies": [] }, { "id": 612930, "author_name": "Joel Hanson", "author_url": "", "post_date": "2019-08-30T05:14:15.730000", "content": "Congrats @antorsae and thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 612842, "author_name": "FGPC", "author_url": "", "post_date": "2019-08-30T03:23:39.580000", "content": "Congratulations! Amazing solutions! :-)
", "votes": 1, "replies": [] }, { "id": 612762, "author_name": "Zhuang Jia", "author_url": "", "post_date": "2019-08-30T01:43:47.027000", "content": "Congratulations! I'm very excited to learn from your solutions.
", "votes": 1, "replies": [] }, { "id": 612517, "author_name": "Alexandre Sauvé", "author_url": "", "post_date": "2019-08-29T20:54:15.050000", "content": "I'm just impressed by the way of using xyz coordinates. Need to read more bout Trnasformer Encoders.
\n\nThank you @antorsae for sharing the solution and congratulation for your creativity!
", "votes": 1, "replies": [] }, { "id": 612406, "author_name": "Goran Rakocevic", "author_url": "", "post_date": "2019-08-29T19:14:51.297000", "content": "Thanks for sharing!\nWhen you say \"The y (ground truth) is just an array of dimension 29 containing j-couplings located at target atom indices.\"\nWhich atom index do you place the j-coupling values, as there are two atoms participating in a coupling?
", "votes": 1, "replies": [ { "id": 612430, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2019-08-29T19:39:04.393000", "content": "Say there are N source/reference atoms (atom_index_0 in the csv file) for a given molecule. We build N sibling molecules, and for each the GT is the array of 29 atoms (max) where we place valid j-couplings at target (atom_index_1 in the csv) indices.
Oh, so each coupling gets its own datapoint? We have one datapoint per molecule, that confused me.
", "votes": 0, "replies": [] }, { "id": 612453, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2019-08-29T19:53:36.653000", "content": "Yes.
\n\nWe started building a Molecule Transformer which had all couplings in one shot (diff input representation) but stopped it bc we focussed on training the Atomic Transformer instead.
", "votes": 0, "replies": [] } ] }, { "id": 612252, "author_name": "Zhenlan", "author_url": "", "post_date": "2019-08-29T17:17:01.987000", "content": "I am not sure that I understand your Input representation. Could you give a simply example, say for H2O? Thanks
", "votes": 1, "replies": [] }, { "id": 612239, "author_name": "Zhenlan", "author_url": "", "post_date": "2019-08-29T17:10:27.217000", "content": "Congrats!! Thank you for sharing. I look forward to seeing your code :D
", "votes": 1, "replies": [] }, { "id": 613298, "author_name": "XiaokangWang", "author_url": "", "post_date": "2019-08-30T12:04:02.160000", "content": "Congratulations! You are not using graph NN. Amazing ideas! I think it would be an excellent paper if you wrote it up your ideas.
", "votes": 2, "replies": [] }, { "id": 612569, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2019-08-29T21:36:50.697000", "content": "looking forward to the code! Thanks for sharing
", "votes": 2, "replies": [] }, { "id": 612501, "author_name": "Patrick Chan", "author_url": "", "post_date": "2019-08-29T20:36:44.797000", "content": "Very impressive and the right way to go with AI discovery. Like the domainless approach! Congratulations!
", "votes": 2, "replies": [] }, { "id": 612474, "author_name": "guiferviz", "author_url": "", "post_date": "2019-08-29T20:09:03.203000", "content": "Congratulations! Looking forward to exploring your code, I'm very interested in solutions without domain knowledge.\nBtw, I just realized that we are form the same city lol
", "votes": 2, "replies": [] }, { "id": 612269, "author_name": "Robin N", "author_url": "", "post_date": "2019-08-29T17:29:57.993000", "content": "Really impressed to see that you were able to push a pure Tranformer Encoder to such great performances. I tried using a straightforward Transformer model as well on a set of engineered features (no raw coordinates), but wasn't able to get results anywhere near as good as yours. Like you said, I think the key is your input representation creating the N sibling molecules.
\n\nCongrats on second place and the beautiful approach.
", "votes": 2, "replies": [] }, { "id": 612172, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-08-29T16:27:23.473000", "content": "Congrats.....\nGreat Work...\nThanks for Sharing...!! @antorsae
", "votes": 0, "replies": [] }, { "id": 612432, "author_name": "JM", "author_url": "", "post_date": "2019-08-29T19:40:41.273000", "content": "Great work, looking forward to seeing the notebook
", "votes": 1, "replies": [] }, { "id": 781190, "author_name": "yuquan.li", "author_url": "", "post_date": "2020-03-21T02:38:02.537000", "content": "I look forward to seeing your code :D, now too
", "votes": 0, "replies": [ { "id": 823848, "author_name": "XiaokangWang", "author_url": "", "post_date": "2020-04-28T00:29:40.910000", "content": "He published his code 8 months ago in github.
", "votes": 1, "replies": [] } ] }, { "id": 718185, "author_name": "Alberto Rekena", "author_url": "", "post_date": "2020-01-14T06:28:11.320000", "content": "Impressive!
", "votes": 0, "replies": [] }, { "id": 612599, "author_name": "", "author_url": "", "post_date": "2019-08-29T22:17:53.037000", "content": "", "votes": 0, "replies": [ { "id": 614181, "author_name": "", "author_url": "", "post_date": "2019-08-31T07:12:21.793000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 618474, "author_name": "", "author_url": "", "post_date": "2019-09-05T08:37:25.403000", "content": "", "votes": 1, "replies": [] }, { "id": 618278, "author_name": "", "author_url": "", "post_date": "2019-09-05T04:23:00.593000", "content": "", "votes": 1, "replies": [] }, { "id": 616456, "author_name": "", "author_url": "", "post_date": "2019-09-03T06:33:03.167000", "content": "", "votes": 1, "replies": [] }, { "id": 612986, "author_name": "", "author_url": "", "post_date": "2019-08-30T06:25:48.727000", "content": "", "votes": 1, "replies": [] }, { "id": 612786, "author_name": "", "author_url": "", "post_date": "2019-08-30T02:19:29.727000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "612141": "We want to thank Kaggle and CHAMPs for organizing such an awesome competition: \n\n- No leakage.\n- Same distribution in train, private and public test (very stable CV vs. LB, public LB vs. private LB). This is very relevant in other competitions given so many participants when the 3 distributions are different some winners (not all, e.g. [CPMP approach to manually align test ~ train in the Microsoft malware comp](https://www.kaggle.com/c/microsoft-malware-prediction/discussion/84069#latest-499864)) are just lucky they fit private test distribution by chance; not the case here. \n- Inspiring and useful science problem hopefully used for good purposes as described in the context (new drugs, etc.).\n\nNeither my teammate @pavelgonchar nor I had any previous domain expertise and we made the decision early on that we would tackle this problem using a very pure deep learning way: letting the model build the features for us, not the other way around (b/c obviously we were at a disadvantage if we tried to become quantum experts in 1 month… hence our team name 🤖Quantum Uncertainty🤖 we didn't know if our yet-to-be-developed approach was going to work).\n\nOur solution had two major parts: 1) the input representation and 2) deep learning architecture.\n\n**Input representation**\n\nThis is in our opinion the key part: we take a molecule and a source atom and move it so the source atom is @ (0,0,0). For each molecule we create N molecule siblings (N being as many source atoms are defined for that molecule), and each molecule sibling is translated so its source is at (0,0,0). \n\nThe `x` (input) are three arrays of dimension 29 (maximum number of atoms): \n\n1) `x,y,z` position of each atom, \n2) `atom type` index (C=0, H=1, etc…)\n3) `j-coupling type` index (1JHC=0,'2JHH=1,etc.)\n\nPadding is done by placing -1 in `atom type` index and `j-coupling type` for molecules which have less than 29 atoms.\n\nThe `y` (ground truth) is just an array of dimension 29 containing j-couplings located at target atom indices.\n\nNote that there is no graph information nor any other manually engineered features.\n\n**Data augmentation**\n\nWe did two types of data augmentation:\n- Rotations: which worked and were useful in our first attempt model: pointnet-based, but proved worthless in the final models (atomic transformer).\n- J-coupling symmetry: as described [in this discussion](https://www.kaggle.com/c/champs-scalar-coupling/discussion/94706#latest-563148)\n\n**First attempt: Pointnet-inspired architecture (got up to -2.28200 LB)**\n\nOur input representation is basically a point cloud: an unordered set of elements with absolute positions `x,y,z` and two attributes `atom type` and `j-coupling type`. We modified the [Pointnet](https://arxiv.org/abs/1612.00593) architecture to regress j-couplings. Training was a bit unstable and we tried many variations of the architecture, swapping FC layers by linear (fixed) projections (Hadamard), adding coulomb matrix as input, etc. \n\nWhile this worked OK and got us to -2.28200 LB (ensembling a few models) we felt that this architecture was limited by the extreme pooling/bottleneck operation so we decided to explore other architectures: meet the Atomic Transformer.\n\n**Final architecture: meet the Atomic Transformer**\n\nYou may know that the recent NLP revolution is mostly due to the transformer architecture described in the [Attention is all you need paper](https://arxiv.org/abs/1706.03762). The vanilla transformer architecture uses a very clever technique to add positional encodings that are needed for position-dependent input, such as language. \n\nOur input representation is a set, which means we can (and should) remove positional encoding. Prior to this competition we had no experience with transformers either but there's a section in [Lex Fridman MIT podcast interviewing Orion Vinyals](https://www.youtube.com/watch?v=Kedt2or9xlo) where he mentions the inherent position invariance of a barebone transformer encoder layer. This immediately triggered the idea of using transformer layers (encoders) stacked taking as an input `x,y,z` (normalized but otherwise as-is), and `atom type` and `j-coupling type` embeddings; just concatenated… nothing fancy. \n\nThe dimension of the embeddings was such that the total dimension of the input vectors was `d_model` (as normally reference in transformer literature). We started with 256 and got immediately great results surpassing our pointnet-inspired architecture so we followed this path.\n\nWe trained a total of 14 models, with varying dimensions from 512 to 2048 and layers from 6 to 24. Each model parameter size ranged from ~12M to ~100M (biggest model).\n\nWe trained some models from scratch, others we fine-tuned. We also fine-tuned a few models on the troublesome j-couplings: reaching -2.12 CV on 1JHC on and -2.19 CV on 1JHN.\n\nOur best score is an ensemble of 14 models achieving private LB of -3.22349, and our best single model achieved private LB of -3.16234, again just with `x,y,z`, `atom type` and `j-coupling type` inputs (no QM9, etc.).\n\n**What didn't work**\n\nMany things! We tried:\n\n- Multi-task learning using contributions and other organization provided values.\n- Dropout: We tried multiple attempts to add dropout at various stages (embeddings, encoder layers, pre-decoder, etc.). None of them worked.\n- Knock-out: We added a variation in which as input we deleted 10% of the input atoms, the idea being that the model would build an internal representation of the missing atoms. Surprisingly this worked in that the model still converged nicely but failed to reduce train ~ val gap.\n- Rotations and TTA in Atomic Transformer: it didn't reduce train ~ val gap and didn't produce meaningful TTA gains.\n- Deep decoder: Our decoder is just a projection of ~ the model dimension to 1 (scalar coupling). We tried adding more expressive power to the decoder but this didn't help.\n- Fp16 training. This worked for models of dimensions 256 but as training evolved gave `NaN`s despite numerous attempts to fix it.\n\n**Source code**\n\nWe will make source code available once we do clean up. It's a single jupyter notebook using FastAI. Be patient.\n\n**Computational resources**\n\nWe had more ideas than computational resources, even if our computational resources were not tiny:\n- 3 x 2080 Ti + 128 Gb RAM + 16c32t processor\n- 2 x 1080 Ti + 64 Gb RAM + 8c16t processor\n- Rented 8+ 2080 Ti + 64 Gb RAM + 16c32t processor (multiple machines rented as needed)\n\n**Final thoughts**\n\nThis was our most fun and hardest competition so far:\n- Challenging problem\n- Most teams in top 5 had domain experts (although we went *domainless* as part of our strategy)\n- Hungry computational resources.\n\nEven if we lost #1 position just a few hours before competition end we feel very excited we were able to achieve such [useful results for the organizers] (https://www.kaggle.com/c/champs-scalar-coupling/discussion/98375#latest-569312). In retrospect we believe a single model of the Atomic Transformer may achieve ever better results with further training.\n\nBest - Pavel & Andres\n\np.s. No graph NNs. We though graphs as manually engineered features that the model can infer by itself.", "679449": "Hi Andres, very inspired by your approach. Is the notebook available for us to study, could you point me to it?", "612258": "Very impressive how having machine learning expertise and no domain knowledge can get you a podium, it truely amazes me. Congrats @antorsae ", "612153": "Thanks, very impressive solution. In a way my team mates approached the problem in a way similar to yours, by representing every atom position relative to the two atoms for which scaling is computed. But you way is a bit different as you keep the coordinates, while they worked with distances and angles. That using coordinates works that well is a bit of a surprise to me. If I were the organizers then i would have applied random rotations to the data... This said, again, congrats on the approach. I'll need to reread this to full grasp it. \n\nI forgot the most important: congrats on the end result and the incredible lead during the competition.", "612260": "I just want to say that your team's work is very impressive, and I certainly look forward to reading the implementation in more detail! For one, we also use transformer architecture, but it was a graph transformer. I think Zico will make a post about it sometime soon.", "685920": "Congrats!", "662722": "Congratulations! Impressed that not used graph NNs but used transformer based architectures to deal with set, together with the position translation for target atom.", "644660": "Hi,\n\nThanks a lot for sharing your solution and code. I was reading your code and I have a question: In your code atom-transfomer-good-norm-pytorch, the result under the line learner.summary() shows two emdedding layers and two dropout layers at the end. Where do they come from? I can't find them in the forward method of your AtomTorchTransformer. I apologize if the question is too detailed and thanks in advance.", "616487": "Congratulations @antorsae @pavelgonchar .Thank you for the approach and waiting for the notebook. The one thing that is inspirational for me is that though you don't have any domain expertise, you understand and solve the problem using DeepLearning techniques.", "615668": "Congratulations, good job", "615365": "Really looking forward to your solution, especially as a fellow fastai community member. Your team really inspired us to keep striving in the final weeks of the competition. Sub -3 didn't feel achievable until we saw you pull it off.", "613390": "Congratulations, you guys are inspirations. ", "613261": "Congrats!!", "613184": "amazing work", "612930": "Congrats @antorsae and thanks for sharing.", "612842": "Congratulations! Amazing solutions! :-)", "612762": "Congratulations! I'm very excited to learn from your solutions.", "612517": "I'm just impressed by the way of using xyz coordinates. Need to read more bout Trnasformer Encoders.\n\nThank you @antorsae for sharing the solution and congratulation for your creativity!", "612406": "Thanks for sharing!\nWhen you say \"The y (ground truth) is just an array of dimension 29 containing j-couplings located at target atom indices.\"\nWhich atom index do you place the j-coupling values, as there are two atoms participating in a coupling?", "612252": "I am not sure that I understand your Input representation. Could you give a simply example, say for H2O? Thanks", "612239": "Congrats!! Thank you for sharing. I look forward to seeing your code :D", "613298": "Congratulations! You are not using graph NN. Amazing ideas! I think it would be an excellent paper if you wrote it up your ideas. ", "612569": "looking forward to the code! Thanks for sharing", "612501": "Very impressive and the right way to go with AI discovery. Like the domainless approach! Congratulations!", "612474": "Congratulations! Looking forward to exploring your code, I'm very interested in solutions without domain knowledge.\nBtw, I just realized that we are form the same city lol", "612269": "Really impressed to see that you were able to push a pure Tranformer Encoder to such great performances. I tried using a straightforward Transformer model as well on a set of engineered features (no raw coordinates), but wasn't able to get results anywhere near as good as yours. Like you said, I think the key is your input representation creating the N sibling molecules. \n\nCongrats on second place and the beautiful approach.", "612172": "Congrats.....\nGreat Work...\nThanks for Sharing...!! @antorsae ", "612432": "Great work, looking forward to seeing the notebook", "781190": "I look forward to seeing your code :D, now too", "718185": "Impressive!", "612599": "Awesome. Thanks for sharing. Any estimate as to how much money you guys spent on this?", "618474": "", "618278": "Congratulations and thanks for sharing", "616456": "Congrats and thanks a lot for sharing.", "612986": "Thank you for sharing!", "612786": "Congratulations! Thanks for sharing!" }, "complete": true, "source": "existing_discussion", "competition_id": 14313, "competition_slug": "champs-scalar-coupling", "team_id": 3251377, "solution_url": "/competitions/champs-scalar-coupling/writeups/quantum-uncertainty-2-solution-quantum-uncertainty" }