{
"id": 106310,
"title": "62nd Place Solution.",
"url": "/competitions/champs-scalar-coupling/discussion/106310",
"author_name": "Ashesh.",
"post_date": "2019-08-29T03:30:42.548000",
"votes": 24,
"comment_count": 0,
"views": 0,
"content": "Code Specification
\n\nI'll try to briefly summarize different components of my solution.
\n\nModel Structure
\n\nI created 3 models. ( 4th one was a public kernel )\n1. Catboost (LB: -1.95)\n2. LightGBM (LB: -1.87)\n3. GNN (LB: -2.05)
\n\nFeatures for Catboost and LightGBM
\n\nBond detection\nInitially I inferred bonds(Code). However, I later used openbabel for bond detection.
\n\nNeighbor detection\nI defined neighbors in 5 ways:\na. Using the edge data, I was able to define 1st neighbor atoms, 2nd neighbor atoms and so on. I was also able to figure out intermediate atoms in path from atom_index_0 to atom_index_1. (Code)\nb. Using distance between atoms and standard bond length, neighbors were inferred which were close enough to be considered to be bonded (Code)\nc. Based on atom_index. For example, neighbors of atom_index 5 will be 3,4,6,7. (Code)\nd. k-nearest atoms were considered as neighbors. Code\ne. I divided the region around center of the bond by creating right circular cones with different base angles. Cone's axis was the bond. It gave me neighbors which were in different angular neighborhood ([0°-60°], (60°-120°) and so on). Code
\n\nNeighbor based features\nI created aggregate of Electronegativity, Valency, Lone pairs, bond angles,mass and distance for neighbors generated from above 5 ways.
\n\nAngle based features\na. Dihedral angle and some other angles were also computed for 2,3 bond apart atom pairs. (3JHH,3JHN,3JHC,2JHH,2JHN,2JHC)\nb. Aggregate of angles for neighbors.
\n\nEstimation of partial charge\nUsing electronegativity information and co-ordinates of atoms, I computed electronegativity vectors normalized by distance between the atoms. Doing a vector sum of these electronegativity vectors over bond neighbors yielded me an estimate of partial charge on each atom. Code. Later in the competition, I started to use sum of electronegativity difference normalized by distance as a measure of partial charge (no vector addition).
\n\nPotential based features\nUsed yukawa,coulomb potential using the estimated charge. Code.
\n\nOther features\nI used some of the atom features generate by openbabel. Using edge data, I was able to also infer cycle length and other related features (Code). I also used bond energies as feature between bonds of the molecule(Code). Carbon hybridization was used as a feature. For 3 bond distant atom pairs which had 2 sp2 hybridized carbon atoms as intermediates, Cis/Trans configuration was used as a feature (Code). I also tried to estimate how much pi bonds have electron 'donor' tendency. (Code)
\n\nFeatures for GNN
\n\nI had to recompute the edge features as they were to be computed between each possible atom pairs in a molecule. I ended up coding up a numpy heavy implementation of the top features again which I had computed for tree based models. Earlier features were computed mostly using pandas.
\n\nJourney
\n\nI started this competition after completing a playground competition. In the beginning I started working on it without looking at discussions and public kernels . Such arrogance :D . With time, however, improvements started saturating. Then I skimmed through the discussions ;) . \n![\"\"](\"https://i.chzbgr.com/full/6493143296/hA923C6DD/\")
\nAgain, after sometime, progress plateaued and with all humility I started understanding different ideas present in the public domain. https://www.kaggle.com/c/champs-scalar-coupling/discussion/96436#latest-565548 and https://www.kaggle.com/c/champs-scalar-coupling/discussion/93928#latest-588773 were the among the most helpful to me.\nWhen you know that something will most probably work, it is much easier to code. ;)\nAt some point, I saw a comment stating that one should use mse as the loss instead of mae. Initially, I had tried both of them but had not got much difference. Now however, I got a 0.2 jump !! At this point I was around -1.5 LB.\nThen I got wind of GNN from https://www.kaggle.com/fnands/1-mpnn .It was a nice experience reading up the GNN paper and understanding the code of the kernel. This collaborative nature of kaggle is just amazing. One can learn so much if only one is willing to look around and work. In my case, memory was the main issue with GNN. My features were not fitting into the memory. I changes all features to float16 and selected top few features . Then I started getting somewhere. 5 fold CV helped a lot to remove the jitter from the prediction. I did about 40-50 epochs in each fold with hand curated learning rate scheduling. On average, starting from 1e-3, I reduced LR every 8 iterations by a factor of .5 .
\n\nLooking back, I loved this journey and will definitely participate in more kaggle competitions.
\n\nTechnical Takeaways from the competition
\n\n\n- I learnt to use numpy much better and learnt to avoid pandas groupby,stack,unstack operations as much as possible. I was able to reduce my feature generation time about 10-50 times in different cases. When you think about it, you will be surprised how much can be done using matrix multiplications. Distances, angles, aggregated features for neighbors and much more. It started when I looked at this amazing kernel https://www.kaggle.com/cpmpml/ultra-fast-distance-matrix-computation
\n- I've a few redundant features due to which unnecessary time was spent in their computation, codebase got quite big and I was running into memory issues. In future, I'll try to add one feature at a time
\n- By looking at the brief summary of top solutions, I see that there was much less feature engineering for neural network based solutions. In future, I'll start on NN based models quite early into the competition.
\n- It is worth the time to do KFold CV. Results improve significantly.
\n
\n\nIt was a good competition. The experience taught me so many things and I'm greatful for kaggle and the competition organizers for this. I'll update it with notebooks soon.
",
"messages": [
{
"id": 610960,
"postDate": "2019-08-29T03:30:42.550Z",
"content": "Code Specification
\n\nI'll try to briefly summarize different components of my solution.
\n\nModel Structure
\n\nI created 3 models. ( 4th one was a public kernel )\n1. Catboost (LB: -1.95)\n2. LightGBM (LB: -1.87)\n3. GNN (LB: -2.05)
\n\nFeatures for Catboost and LightGBM
\n\nBond detection\nInitially I inferred bonds(Code). However, I later used openbabel for bond detection.
\n\nNeighbor detection\nI defined neighbors in 5 ways:\na. Using the edge data, I was able to define 1st neighbor atoms, 2nd neighbor atoms and so on. I was also able to figure out intermediate atoms in path from atom_index_0 to atom_index_1. (Code)\nb. Using distance between atoms and standard bond length, neighbors were inferred which were close enough to be considered to be bonded (Code)\nc. Based on atom_index. For example, neighbors of atom_index 5 will be 3,4,6,7. (Code)\nd. k-nearest atoms were considered as neighbors. Code\ne. I divided the region around center of the bond by creating right circular cones with different base angles. Cone's axis was the bond. It gave me neighbors which were in different angular neighborhood ([0°-60°], (60°-120°) and so on). Code
\n\nNeighbor based features\nI created aggregate of Electronegativity, Valency, Lone pairs, bond angles,mass and distance for neighbors generated from above 5 ways.
\n\nAngle based features\na. Dihedral angle and some other angles were also computed for 2,3 bond apart atom pairs. (3JHH,3JHN,3JHC,2JHH,2JHN,2JHC)\nb. Aggregate of angles for neighbors.
\n\nEstimation of partial charge\nUsing electronegativity information and co-ordinates of atoms, I computed electronegativity vectors normalized by distance between the atoms. Doing a vector sum of these electronegativity vectors over bond neighbors yielded me an estimate of partial charge on each atom. Code. Later in the competition, I started to use sum of electronegativity difference normalized by distance as a measure of partial charge (no vector addition).
\n\nPotential based features\nUsed yukawa,coulomb potential using the estimated charge. Code.
\n\nOther features\nI used some of the atom features generate by openbabel. Using edge data, I was able to also infer cycle length and other related features (Code). I also used bond energies as feature between bonds of the molecule(Code). Carbon hybridization was used as a feature. For 3 bond distant atom pairs which had 2 sp2 hybridized carbon atoms as intermediates, Cis/Trans configuration was used as a feature (Code). I also tried to estimate how much pi bonds have electron 'donor' tendency. (Code)
\n\nFeatures for GNN
\n\nI had to recompute the edge features as they were to be computed between each possible atom pairs in a molecule. I ended up coding up a numpy heavy implementation of the top features again which I had computed for tree based models. Earlier features were computed mostly using pandas.
\n\nJourney
\n\nI started this competition after completing a playground competition. In the beginning I started working on it without looking at discussions and public kernels . Such arrogance :D . With time, however, improvements started saturating. Then I skimmed through the discussions ;) . \n\nAgain, after sometime, progress plateaued and with all humility I started understanding different ideas present in the public domain. https://www.kaggle.com/c/champs-scalar-coupling/discussion/96436#latest-565548 and https://www.kaggle.com/c/champs-scalar-coupling/discussion/93928#latest-588773 were the among the most helpful to me.\nWhen you know that something will most probably work, it is much easier to code. ;)\nAt some point, I saw a comment stating that one should use mse as the loss instead of mae. Initially, I had tried both of them but had not got much difference. Now however, I got a 0.2 jump !! At this point I was around -1.5 LB.\nThen I got wind of GNN from https://www.kaggle.com/fnands/1-mpnn .It was a nice experience reading up the GNN paper and understanding the code of the kernel. This collaborative nature of kaggle is just amazing. One can learn so much if only one is willing to look around and work. In my case, memory was the main issue with GNN. My features were not fitting into the memory. I changes all features to float16 and selected top few features . Then I started getting somewhere. 5 fold CV helped a lot to remove the jitter from the prediction. I did about 40-50 epochs in each fold with hand curated learning rate scheduling. On average, starting from 1e-3, I reduced LR every 8 iterations by a factor of .5 .
\n\nLooking back, I loved this journey and will definitely participate in more kaggle competitions.
\n\nTechnical Takeaways from the competition
\n\n\n- I learnt to use numpy much better and learnt to avoid pandas groupby,stack,unstack operations as much as possible. I was able to reduce my feature generation time about 10-50 times in different cases. When you think about it, you will be surprised how much can be done using matrix multiplications. Distances, angles, aggregated features for neighbors and much more. It started when I looked at this amazing kernel https://www.kaggle.com/cpmpml/ultra-fast-distance-matrix-computation
\n- I've a few redundant features due to which unnecessary time was spent in their computation, codebase got quite big and I was running into memory issues. In future, I'll try to add one feature at a time
\n- By looking at the brief summary of top solutions, I see that there was much less feature engineering for neural network based solutions. In future, I'll start on NN based models quite early into the competition.
\n- It is worth the time to do KFold CV. Results improve significantly.
\n
\n\nIt was a good competition. The experience taught me so many things and I'm greatful for kaggle and the competition organizers for this. I'll update it with notebooks soon.
",
"rawMarkdown": "# Code Specification\nI'll try to briefly summarize different components of my solution.\n\n## Model Structure\nI created 3 models. ( 4th one was a [public kernel ](https://www.kaggle.com/ioohooi/chemistry-of-best-models-1-839 ) )\n1. Catboost (LB: -1.95)\n2. LightGBM (LB: -1.87)\n3. GNN (LB: -2.05)\n\n###Features for Catboost and LightGBM\n\n`Bond detection`\nInitially I inferred bonds([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/find_edges.py)). However, I later used [openbabel](http://openbabel.org/wiki/Main_Page) for bond detection. \n\n`Neighbor detection`\nI defined neighbors in 5 ways:\na. Using the **edge data**, I was able to define 1st neighbor atoms, 2nd neighbor atoms and so on. I was also able to figure out intermediate atoms in path from atom_index_0 to atom_index_1. ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/compute_intermediate_atoms.py#L121))\nb. Using distance between atoms and standard bond length, neighbors were inferred which were close enough to be considered to be bonded ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/neighbor_features_distance.py#L104))\nc. Based on atom_index. For example, neighbors of atom_index 5 will be 3,4,6,7. ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/neighbor_features_atom_index.py))\nd. k-nearest atoms were considered as neighbors. [Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/edge_features.py#L182)\ne. I divided the region around center of the bond by creating right circular cones with different base angles. Cone's axis was the bond. It gave me neighbors which were in **different angular neighborhood** ([0°-60°], (60°-120°) and so on). [Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/conical_segmented_features.py)\n\n`Neighbor based features`\nI created aggregate of Electronegativity, Valency, Lone pairs, bond angles,mass and distance for neighbors generated from above 5 ways.\n\n`Angle based features`\na. Dihedral angle and some other angles were also computed for 2,3 bond apart atom pairs. (3JHH,3JHN,3JHC,2JHH,2JHN,2JHC)\nb. Aggregate of angles for neighbors. \n\n`Estimation of partial charge`\nUsing electronegativity information and co-ordinates of atoms, I computed electronegativity vectors normalized by distance between the atoms. Doing a vector sum of these electronegativity vectors over bond neighbors yielded me an estimate of partial charge on each atom. [Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/edge_features.py#L95). Later in the competition, I started to use sum of electronegativity difference normalized by distance as a measure of partial charge (no vector addition).\n\n`Potential based features`\nUsed yukawa,coulomb potential using the estimated charge. [Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/atom_potentials.py#L14).\n\n`Other features`\nI used some of the atom features generate by openbabel. Using edge data, I was able to also infer cycle length and other related features ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/cycle_features.py#L23)). I also used bond energies as feature between bonds of the molecule([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/bond_features.py#L31)). Carbon hybridization was used as a feature. For 3 bond distant atom pairs which had 2 sp2 hybridized carbon atoms as intermediates, Cis/Trans configuration was used as a feature ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/intermediate_atom_features.py#L144)). I also tried to estimate how much pi bonds have electron 'donor' tendency. ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/pi_donor.py))\n\n\n###Features for GNN\nI had to recompute the edge features as they were to be computed between each possible atom pairs in a molecule. I ended up coding up a numpy heavy implementation of the top features again which I had computed for tree based models. Earlier features were computed mostly using pandas.\n\n# Journey\nI started this competition after completing a [playground competition](https://www.kaggle.com/c/competitive-data-science-predict-future-sales). In the beginning I started working on it without looking at discussions and public kernels . Such arrogance :D . With time, however, improvements started saturating. Then I skimmed through the discussions ;) . \n\nAgain, after sometime, progress plateaued and with all humility I started understanding different ideas present in the public domain. https://www.kaggle.com/c/champs-scalar-coupling/discussion/96436#latest-565548 and https://www.kaggle.com/c/champs-scalar-coupling/discussion/93928#latest-588773 were the among the most helpful to me.\n`When you know that something will most probably work, it is much easier to code. ;)`\nAt some point, I saw a comment stating that one should use mse as the loss instead of mae. Initially, I had tried both of them but had not got much difference. Now however, I got a 0.2 jump !! At this point I was around -1.5 LB.\nThen I got wind of GNN from https://www.kaggle.com/fnands/1-mpnn .It was a nice experience reading up the GNN paper and understanding the code of the kernel. This collaborative nature of kaggle is just amazing. One can learn so much if only one is willing to look around and work. In my case, memory was the main issue with GNN. My features were not fitting into the memory. I changes all features to float16 and selected top few features . Then I started getting somewhere. 5 fold CV helped a lot to remove the jitter from the prediction. I did about 40-50 epochs in each fold with hand curated learning rate scheduling. On average, starting from 1e-3, I reduced LR every 8 iterations by a factor of .5 . \n\nLooking back, I loved this journey and will definitely participate in more kaggle competitions.\n\n#Technical Takeaways from the competition\n1. I learnt to use **numpy** much better and learnt to avoid pandas **groupby,stack,unstack** operations as much as possible. I was able to reduce my feature generation time about 10-50 times in different cases. When you think about it, you will be surprised how much can be done using matrix multiplications. [Distances](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/gnn_common_utils_molecule_properties.py#L75), [angles](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/numpy_angle_computation.py), aggregated features for neighbors and much more. It started when I looked at this amazing kernel https://www.kaggle.com/cpmpml/ultra-fast-distance-matrix-computation \n2. I've a few redundant features due to which unnecessary time was spent in their computation, codebase got quite big and I was running into memory issues. In future, I'll try to **add one feature at a time**\n3. By looking at the brief summary of top solutions, I see that there was **much less feature engineering** for neural network based solutions. In future, I'll start on NN based models quite early into the competition.\n4. It is worth the time to do KFold CV. Results improve significantly.\n\nIt was a good competition. The experience taught me so many things and I'm greatful for kaggle and the competition organizers for this. I'll update it with notebooks soon.\n",
"votes": 23
}
],
"comments": [],
"raw_markdown_by_id": {
"610960": "# Code Specification\nI'll try to briefly summarize different components of my solution.\n\n## Model Structure\nI created 3 models. ( 4th one was a [public kernel ](https://www.kaggle.com/ioohooi/chemistry-of-best-models-1-839 ) )\n1. Catboost (LB: -1.95)\n2. LightGBM (LB: -1.87)\n3. GNN (LB: -2.05)\n\n###Features for Catboost and LightGBM\n\n`Bond detection`\nInitially I inferred bonds([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/find_edges.py)). However, I later used [openbabel](http://openbabel.org/wiki/Main_Page) for bond detection. \n\n`Neighbor detection`\nI defined neighbors in 5 ways:\na. Using the **edge data**, I was able to define 1st neighbor atoms, 2nd neighbor atoms and so on. I was also able to figure out intermediate atoms in path from atom_index_0 to atom_index_1. ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/compute_intermediate_atoms.py#L121))\nb. Using distance between atoms and standard bond length, neighbors were inferred which were close enough to be considered to be bonded ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/neighbor_features_distance.py#L104))\nc. Based on atom_index. For example, neighbors of atom_index 5 will be 3,4,6,7. ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/neighbor_features_atom_index.py))\nd. k-nearest atoms were considered as neighbors. [Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/edge_features.py#L182)\ne. I divided the region around center of the bond by creating right circular cones with different base angles. Cone's axis was the bond. It gave me neighbors which were in **different angular neighborhood** ([0°-60°], (60°-120°) and so on). [Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/conical_segmented_features.py)\n\n`Neighbor based features`\nI created aggregate of Electronegativity, Valency, Lone pairs, bond angles,mass and distance for neighbors generated from above 5 ways.\n\n`Angle based features`\na. Dihedral angle and some other angles were also computed for 2,3 bond apart atom pairs. (3JHH,3JHN,3JHC,2JHH,2JHN,2JHC)\nb. Aggregate of angles for neighbors. \n\n`Estimation of partial charge`\nUsing electronegativity information and co-ordinates of atoms, I computed electronegativity vectors normalized by distance between the atoms. Doing a vector sum of these electronegativity vectors over bond neighbors yielded me an estimate of partial charge on each atom. [Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/edge_features.py#L95). Later in the competition, I started to use sum of electronegativity difference normalized by distance as a measure of partial charge (no vector addition).\n\n`Potential based features`\nUsed yukawa,coulomb potential using the estimated charge. [Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/atom_potentials.py#L14).\n\n`Other features`\nI used some of the atom features generate by openbabel. Using edge data, I was able to also infer cycle length and other related features ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/cycle_features.py#L23)). I also used bond energies as feature between bonds of the molecule([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/bond_features.py#L31)). Carbon hybridization was used as a feature. For 3 bond distant atom pairs which had 2 sp2 hybridized carbon atoms as intermediates, Cis/Trans configuration was used as a feature ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/intermediate_atom_features.py#L144)). I also tried to estimate how much pi bonds have electron 'donor' tendency. ([Code](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/pi_donor.py))\n\n\n###Features for GNN\nI had to recompute the edge features as they were to be computed between each possible atom pairs in a molecule. I ended up coding up a numpy heavy implementation of the top features again which I had computed for tree based models. Earlier features were computed mostly using pandas.\n\n# Journey\nI started this competition after completing a [playground competition](https://www.kaggle.com/c/competitive-data-science-predict-future-sales). In the beginning I started working on it without looking at discussions and public kernels . Such arrogance :D . With time, however, improvements started saturating. Then I skimmed through the discussions ;) . \n\nAgain, after sometime, progress plateaued and with all humility I started understanding different ideas present in the public domain. https://www.kaggle.com/c/champs-scalar-coupling/discussion/96436#latest-565548 and https://www.kaggle.com/c/champs-scalar-coupling/discussion/93928#latest-588773 were the among the most helpful to me.\n`When you know that something will most probably work, it is much easier to code. ;)`\nAt some point, I saw a comment stating that one should use mse as the loss instead of mae. Initially, I had tried both of them but had not got much difference. Now however, I got a 0.2 jump !! At this point I was around -1.5 LB.\nThen I got wind of GNN from https://www.kaggle.com/fnands/1-mpnn .It was a nice experience reading up the GNN paper and understanding the code of the kernel. This collaborative nature of kaggle is just amazing. One can learn so much if only one is willing to look around and work. In my case, memory was the main issue with GNN. My features were not fitting into the memory. I changes all features to float16 and selected top few features . Then I started getting somewhere. 5 fold CV helped a lot to remove the jitter from the prediction. I did about 40-50 epochs in each fold with hand curated learning rate scheduling. On average, starting from 1e-3, I reduced LR every 8 iterations by a factor of .5 . \n\nLooking back, I loved this journey and will definitely participate in more kaggle competitions.\n\n#Technical Takeaways from the competition\n1. I learnt to use **numpy** much better and learnt to avoid pandas **groupby,stack,unstack** operations as much as possible. I was able to reduce my feature generation time about 10-50 times in different cases. When you think about it, you will be surprised how much can be done using matrix multiplications. [Distances](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/gnn_common_utils_molecule_properties.py#L75), [angles](https://github.com/ashesh-0/kaggle_competitions/blob/master/predicting_molecular_properties/numpy_angle_computation.py), aggregated features for neighbors and much more. It started when I looked at this amazing kernel https://www.kaggle.com/cpmpml/ultra-fast-distance-matrix-computation \n2. I've a few redundant features due to which unnecessary time was spent in their computation, codebase got quite big and I was running into memory issues. In future, I'll try to **add one feature at a time**\n3. By looking at the brief summary of top solutions, I see that there was **much less feature engineering** for neural network based solutions. In future, I'll start on NN based models quite early into the competition.\n4. It is worth the time to do KFold CV. Results improve significantly.\n\nIt was a good competition. The experience taught me so many things and I'm greatful for kaggle and the competition organizers for this. I'll update it with notebooks soon.\n"
}
}