![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3356694%2Fba61a244d36f37fbe8b0ddf5e05f94e5%2Fimplementation.png?generation=1567077987502246&alt=media\")
For each connection we would like to find features that describe the influence other connections have on said connection. To do this we use a neural network that looks at pairs of connections. Its input are simply the connection types and all six Euclidean distances. We do this for all connections with shared weights, resulting in a tensor of size (1, channels, size, size-1). We efficiently implement the network using 1x1 2D-convolutions to process an entire molecule at once. Once we have calculated these features, we sum them up (see picture), concatenate with distances and connection types of single connections and use another neural network, implemented as 1D-convolutions with a kernel size of one to get the scc predictions. It is probably easier to understand with this picture:
\n\n
Our solution is invariant to\n- the order of connections fed to the network\n- translations and orthorgonal transformations
\n\nand
\n\nSumming and weight sharing enforce additivity of pair features. If there are two connections in the vicinity of a connection, that have the same physical influence (e.g. symmetric connections), their pair features should be the same! At least we hope so!
\n\nDetails of training:\n- L2 loss\n- Adam\n- no regularization/no dropout/no batchnorm\n- learning rate initially 1e-4, reduced to 1e-6 in multiple steps\n- 100 to 200 epochs
\n\nConnection types are one-hot encoded, distances are standardized and scc are standardized per type. We used up to 1024 hidden neurons in up to 5 layers and ReLU activations in both networks. Training took ~2 weeks on colab (mostly on a single K80) and gave us a score of -2.126.
\n\nWe believe that this approach has a fundamental flaw. We only look at pairs and not at higher order interactions. What can we do? Looking at triplets is too expensive.
\n\nSimilarly to how we build a matrix of pairs from single connections, we use the pair features (after summing) of the previously explained network to build another matrix! Pairs of pairs. We also concatenate the original features to the matrix (types and distances). We again use 1x1 2D-convolutions and sum it up. The final NN is given both the summed features of the pairs and pairs of pairs network. Training was done similarly to our initial approach and resulted in a score of -2.473.
\n\nThe cool thing about this architecture is that you can repeat the process of building matrices over and over again, increasing the effective field of view. We did it one more time. Sadly we ran out of time in the end. 2 weeks of training on kaggle kernels were insufficient.
\n\nOur best submission is a blend of submissions (-2.633).
\n\nWe did very little hyper parameter and architecture search and are quite surprised that it worked quite well. There are many improvements to be made.
", "messages": [ { "id": "611698", "postDate": "08/29/2019 11:45:10", "content": "For each connection we would like to find features that describe the influence other connections have on said connection. To do this we use a neural network that looks at pairs of connections. Its input are simply the connection types and all six Euclidean distances. We do this for all connections with shared weights, resulting in a tensor of size (1, channels, size, size-1). We efficiently implement the network using 1x1 2D-convolutions to process an entire molecule at once. Once we have calculated these features, we sum them up (see picture), concatenate with distances and connection types of single connections and use another neural network, implemented as 1D-convolutions with a kernel size of one to get the scc predictions. It is probably easier to understand with this picture:
\n\n
Our solution is invariant to\n- the order of connections fed to the network\n- translations and orthorgonal transformations
\n\nand
\n\nSumming and weight sharing enforce additivity of pair features. If there are two connections in the vicinity of a connection, that have the same physical influence (e.g. symmetric connections), their pair features should be the same! At least we hope so!
\n\nDetails of training:\n- L2 loss\n- Adam\n- no regularization/no dropout/no batchnorm\n- learning rate initially 1e-4, reduced to 1e-6 in multiple steps\n- 100 to 200 epochs
\n\nConnection types are one-hot encoded, distances are standardized and scc are standardized per type. We used up to 1024 hidden neurons in up to 5 layers and ReLU activations in both networks. Training took ~2 weeks on colab (mostly on a single K80) and gave us a score of -2.126.
\n\nWe believe that this approach has a fundamental flaw. We only look at pairs and not at higher order interactions. What can we do? Looking at triplets is too expensive.
\n\nSimilarly to how we build a matrix of pairs from single connections, we use the pair features (after summing) of the previously explained network to build another matrix! Pairs of pairs. We also concatenate the original features to the matrix (types and distances). We again use 1x1 2D-convolutions and sum it up. The final NN is given both the summed features of the pairs and pairs of pairs network. Training was done similarly to our initial approach and resulted in a score of -2.473.
\n\nThe cool thing about this architecture is that you can repeat the process of building matrices over and over again, increasing the effective field of view. We did it one more time. Sadly we ran out of time in the end. 2 weeks of training on kaggle kernels were insufficient.
\n\nOur best submission is a blend of submissions (-2.633).
\n\nWe did very little hyper parameter and architecture search and are quite surprised that it worked quite well. There are many improvements to be made.
", "rawMarkdown": "# Hey everyone,\n\n## this was our initial approach:\n\nFor each connection we would like to find features that describe the influence other connections have on said connection. To do this we use a neural network that looks at pairs of connections. Its input are simply the connection types and all six Euclidean distances. We do this for all connections with shared weights, resulting in a tensor of size (1, channels, size, size-1). We efficiently implement the network using **1x1 2D-convolutions** to process an entire molecule at once. Once we have calculated these features, we sum them up (see picture), concatenate with distances and connection types of single connections and use another neural network, implemented as **1D-convolutions** with a kernel size of one to get the scc predictions. It is probably easier to understand with this picture:\n\n\n\nOur solution is invariant to\n- the order of connections fed to the network\n- translations and orthorgonal transformations\n\nand\n\n- it does not require any padding\n- works with differently sized molecules.\n\nSumming and weight sharing enforce additivity of pair features. If there are two connections in the vicinity of a connection, that have the same physical influence (e.g. symmetric connections), their pair features should be the same! At least we hope so!\n\nDetails of training:\n- L2 loss\n- Adam\n- no regularization/no dropout/no batchnorm\n- learning rate initially 1e-4, reduced to 1e-6 in multiple steps\n- 100 to 200 epochs\n\nConnection types are one-hot encoded, distances are standardized and scc are standardized per type. We used up to 1024 hidden neurons in up to 5 layers and ReLU activations in both networks. Training took ~2 weeks on colab (mostly on a single K80) and gave us a score of **-2.126**. \n\nWe believe that this approach has a fundamental flaw. We only look at pairs and not at higher order interactions. What can we do? Looking at triplets is too expensive.\n\n## Improved approach:\n\nSimilarly to how we build a matrix of pairs from single connections, we use the pair features (after summing) of the previously explained network to build another matrix! Pairs of pairs. We also concatenate the original features to the matrix (types and distances). We again use **1x1 2D-convolutions** and sum it up. The final NN is given both the summed features of the pairs and pairs of pairs network. Training was done similarly to our initial approach and resulted in a score of **-2.473**.\n\nThe cool thing about this architecture is that you can repeat the process of building matrices over and over again, increasing the effective field of view. We did it one more time. Sadly we ran out of time in the end. 2 weeks of training on kaggle kernels were insufficient.\n\nOur best submission is a blend of submissions (**-2.633**).\n\nWe did very little hyper parameter and architecture search and are quite surprised that it worked quite well. There are many improvements to be made.", "votes": null }, { "id": "611804", "postDate": "08/29/2019 12:54:21", "content": "congrats @fluebbe and Thanks for sharing.
", "rawMarkdown": "congrats @fluebbe and Thanks for sharing.", "votes": null }, { "id": "611809", "postDate": "08/29/2019 12:58:27", "content": "2 weeks training yikes, well done!
", "rawMarkdown": "2 weeks training yikes, well done!", "votes": null }, { "id": "612056", "postDate": "08/29/2019 15:26:38", "content": "Thanks for sharing, and congrats on the result.
", "rawMarkdown": "Thanks for sharing, and congrats on the result.", "votes": null }, { "id": "612274", "postDate": "08/29/2019 17:35:06", "content": "Thanks for sharing! How do you make this kind of NN representation images @fluebbe ?
", "rawMarkdown": "Thanks for sharing! How do you make this kind of NN representation images @fluebbe ?", "votes": null } ], "comments": [ { "id": 611804, "author_name": "joelhanson", "author_url": "", "post_date": "08/29/2019 12:54:21", "content": "congrats @fluebbe and Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 611809, "author_name": "julianmukaj", "author_url": "", "post_date": "08/29/2019 12:58:27", "content": "2 weeks training yikes, well done!
", "votes": null, "replies": [] }, { "id": 612056, "author_name": "cpmpml", "author_url": "", "post_date": "08/29/2019 15:26:38", "content": "Thanks for sharing, and congrats on the result.
", "votes": null, "replies": [] }, { "id": 612274, "author_name": "michelml", "author_url": "", "post_date": "08/29/2019 17:35:06", "content": "Thanks for sharing! How do you make this kind of NN representation images @fluebbe ?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "611698": "# Hey everyone,\n\n## this was our initial approach:\n\nFor each connection we would like to find features that describe the influence other connections have on said connection. To do this we use a neural network that looks at pairs of connections. Its input are simply the connection types and all six Euclidean distances. We do this for all connections with shared weights, resulting in a tensor of size (1, channels, size, size-1). We efficiently implement the network using **1x1 2D-convolutions** to process an entire molecule at once. Once we have calculated these features, we sum them up (see picture), concatenate with distances and connection types of single connections and use another neural network, implemented as **1D-convolutions** with a kernel size of one to get the scc predictions. It is probably easier to understand with this picture:\n\n\n\nOur solution is invariant to\n- the order of connections fed to the network\n- translations and orthorgonal transformations\n\nand\n\n- it does not require any padding\n- works with differently sized molecules.\n\nSumming and weight sharing enforce additivity of pair features. If there are two connections in the vicinity of a connection, that have the same physical influence (e.g. symmetric connections), their pair features should be the same! At least we hope so!\n\nDetails of training:\n- L2 loss\n- Adam\n- no regularization/no dropout/no batchnorm\n- learning rate initially 1e-4, reduced to 1e-6 in multiple steps\n- 100 to 200 epochs\n\nConnection types are one-hot encoded, distances are standardized and scc are standardized per type. We used up to 1024 hidden neurons in up to 5 layers and ReLU activations in both networks. Training took ~2 weeks on colab (mostly on a single K80) and gave us a score of **-2.126**. \n\nWe believe that this approach has a fundamental flaw. We only look at pairs and not at higher order interactions. What can we do? Looking at triplets is too expensive.\n\n## Improved approach:\n\nSimilarly to how we build a matrix of pairs from single connections, we use the pair features (after summing) of the previously explained network to build another matrix! Pairs of pairs. We also concatenate the original features to the matrix (types and distances). We again use **1x1 2D-convolutions** and sum it up. The final NN is given both the summed features of the pairs and pairs of pairs network. Training was done similarly to our initial approach and resulted in a score of **-2.473**.\n\nThe cool thing about this architecture is that you can repeat the process of building matrices over and over again, increasing the effective field of view. We did it one more time. Sadly we ran out of time in the end. 2 weeks of training on kaggle kernels were insufficient.\n\nOur best submission is a blend of submissions (**-2.633**).\n\nWe did very little hyper parameter and architecture search and are quite surprised that it worked quite well. There are many improvements to be made.", "611804": "congrats @fluebbe and Thanks for sharing.", "611809": "2 weeks training yikes, well done!", "612056": "Thanks for sharing, and congrats on the result.", "612274": "Thanks for sharing! How do you make this kind of NN representation images @fluebbe ?" }, "source": "meta" }