Big thanks to all participants for their solutions, insights, and comments. This competition has been a great experience and an opportunity to learn new things.
\nI used this competition as a platform for testing some recently developed Graph Neural Networks. Because of that, I wasn't considering training any classical ML models like RF, SVM or GB trees. Neither was I planning on exploring pre-training or multi-task training methods -- even though I think the final solution would've benefit from them.
\nMy solution is an ensemble of three models: M1(42M params), M2(4.5M params), and M3(65M params). Models 1-3 are different variants of the same architecture. Below is an outline for the M3 setup:
\n1) I used only atom, bond, and graph features as input:
\n + Atom features: atomic symbol, formal charge, stereo
\n + Bond: bond type, direction, stereo
\n + Spectral-based positional encoding for atoms and bonds
\n + Features are embedded using a linear layer
\n2) 7 layers of custom Principal Neighbourhood Aggregation (PNA) convolutions
\n3) Dropout p=20% in the readout layer; batch normalization
\n4) AdamW optimizer, lr=1e-4; ReduceLROnPlateau scheduler; batch size depended on the model's size
\n5) Loss function: weighted BCE; class weights computed dynamically per batch
For inference, the checkpoints were selected with an early stopping criteria. The validation set was a random sample that was set aside from the training set (5% of the training data was used for that purpose). The final prediction was a weighted average over the M1-M3 predictions. The weights in the ensemble were optimized on the calibration set (the validation set was used for calibration) -- with the kappa coefficient as the target.
", "comments": [ { "id": 2157837, "author_name": "Bernhard Rohde", "author_url": "", "post_date": "2023-02-24T11:16:05.740000", "content": "Have you considered connecting your solution description to the Leaderboard entry?
\n(The somewhat hidden link for your 'team' is on this page: https://www.kaggle.com/competitions/euos-slas/team)
Thx for the pointer
", "votes": 0, "replies": [] } ] }, { "id": 2157849, "author_name": "Mr. Maniac", "author_url": "", "post_date": "2023-02-24T11:41:55.223000", "content": "Interesting to see that our two approaches (you GNNs, me classic ML/mostly GBMs) led to almost the same performance on the private leaderboard. I wanted to explore deep learning based models as well but due to time and (mostly) hardware limitations I could not. Would be interesting so see if ensembling our models would have led to improved predictions or not.
", "votes": 0, "replies": [] }, { "id": 2157180, "author_name": "Beardy Polonium", "author_url": "", "post_date": "2023-02-23T18:59:14.447000", "content": "The implementation is based on open source libraries like torch and pytorch-geometric.
Have you considered connecting your solution description to the Leaderboard entry?
\n(The somewhat hidden link for your 'team' is on this page: https://www.kaggle.com/competitions/euos-slas/team)
Thx for the pointer
", "rawMarkdown": "Thx for the pointer" } ] }, { "id": 2157178, "postDate": "2023-02-23T18:55:11.720Z", "content": "Big thanks to all participants for their solutions, insights, and comments. This competition has been a great experience and an opportunity to learn new things.
\nI used this competition as a platform for testing some recently developed Graph Neural Networks. Because of that, I wasn't considering training any classical ML models like RF, SVM or GB trees. Neither was I planning on exploring pre-training or multi-task training methods -- even though I think the final solution would've benefit from them.
\nMy solution is an ensemble of three models: M1(42M params), M2(4.5M params), and M3(65M params). Models 1-3 are different variants of the same architecture. Below is an outline for the M3 setup:
\n1) I used only atom, bond, and graph features as input:
\n + Atom features: atomic symbol, formal charge, stereo
\n + Bond: bond type, direction, stereo
\n + Spectral-based positional encoding for atoms and bonds
\n + Features are embedded using a linear layer
\n2) 7 layers of custom Principal Neighbourhood Aggregation (PNA) convolutions
\n3) Dropout p=20% in the readout layer; batch normalization
\n4) AdamW optimizer, lr=1e-4; ReduceLROnPlateau scheduler; batch size depended on the model's size
\n5) Loss function: weighted BCE; class weights computed dynamically per batch
For inference, the checkpoints were selected with an early stopping criteria. The validation set was a random sample that was set aside from the training set (5% of the training data was used for that purpose). The final prediction was a weighted average over the M1-M3 predictions. The weights in the ensemble were optimized on the calibration set (the validation set was used for calibration) -- with the kappa coefficient as the target.
", "rawMarkdown": "Big thanks to all participants for their solutions, insights, and comments. This competition has been a great experience and an opportunity to learn new things. \n\nI used this competition as a platform for testing some recently developed Graph Neural Networks. Because of that, I wasn't considering training any classical ML models like RF, SVM or GB trees. Neither was I planning on exploring pre-training or multi-task training methods -- even though I think the final solution would've benefit from them.\n\nMy solution is an ensemble of three models: M1(42M params), M2(4.5M params), and M3(65M params). Models 1-3 are different variants of the same architecture. Below is an outline for the M3 setup:\n\n1) I used only atom, bond, and graph features as input:\n + Atom features: atomic symbol, formal charge, stereo\n + Bond: bond type, direction, stereo\n + Spectral-based positional encoding for atoms and bonds\n + Features are embedded using a linear layer\n2) 7 layers of custom Principal Neighbourhood Aggregation (PNA) convolutions\n3) Dropout p=20% in the readout layer; batch normalization\n4) AdamW optimizer, lr=1e-4; ReduceLROnPlateau scheduler; batch size depended on the model's size\n5) Loss function: weighted BCE; class weights computed dynamically per batch\n\nFor inference, the checkpoints were selected with an early stopping criteria. The validation set was a random sample that was set aside from the training set (5% of the training data was used for that purpose). The final prediction was a weighted average over the M1-M3 predictions. The weights in the ensemble were optimized on the calibration set (the validation set was used for calibration) -- with the kappa coefficient as the target.", "votes": 2 }, { "id": 2157849, "postDate": "2023-02-24T11:41:55.223Z", "content": "Interesting to see that our two approaches (you GNNs, me classic ML/mostly GBMs) led to almost the same performance on the private leaderboard. I wanted to explore deep learning based models as well but due to time and (mostly) hardware limitations I could not. Would be interesting so see if ensembling our models would have led to improved predictions or not.
", "rawMarkdown": "Interesting to see that our two approaches (you GNNs, me classic ML/mostly GBMs) led to almost the same performance on the private leaderboard. I wanted to explore deep learning based models as well but due to time and (mostly) hardware limitations I could not. Would be interesting so see if ensembling our models would have led to improved predictions or not.\n" }, { "id": 2157180, "postDate": "2023-02-23T18:59:14.447Z", "content": "The implementation is based on open source libraries like torch and pytorch-geometric.