First of all, tons of thank you to the Kaggle community, the Kaggle team and competition hosts for the event. I am just starting out in the field (doing my bachelors now) and so if I made any mistake here or you have any suggestions, please let me know.\nI took part in the competition to learn and be comfortable with Pytorch with a hands on approach. I did lots of tensor manipulation. I did not have much expectation(it is intimidating to see so many data science experts, also I joined the competition late). And then I got a medal... weird.
\n\nModel details\nToo much talk, real deal is as follows:\nI trained one model responsible for calculating J-coupling(referred as JC next) of all types. The model produced 8 outputs, one each for a type, of which 7 of them were in don't care condition for any input. I directly optimized the evaluation metric, using it as loss.\nLike many teams I used an attention based NN model (Fast.ai's multi-head-attention of text module was an very important piece), to generate features and concatenated it with other engineered features like from 'distance is all you need' kernel. Wikipedia helped me generate few more features. Overall, it is just a variant of BERT.\nAtoms of the molecule whose JC was to be found were fed to the model one by one as input steps, zero padded 30 input steps. I added a classification token(well, actually a series of numbers that represented the learned token) as the first input step (time step sounded weird and atom step weirder) about which I did not know what would be suitable so I trained a small NN based on simple features like input type. All other input step consisted of features that was concerned with a particular atom. Each time step had a binary feature which told the model if the atom for which JC was to be computed. Features were collected from the attention network by concatenating output of first input step (say special input step, due to lack of better name, I think I need to be better with names) maximum and average of its output along all input steps with outputs from steps of atoms whose JC was to be calculated, all total 5X length of encoded output of any input step). These with hand engineered features (after going through few residual layers) were passed through another residual layer and then to a feed forward layer to generate 8 outputs, which was then trained with Adam optimizer using cyclic learning rate.
\n\nTraining details\nAt first I trained a models on the Kaggle kernel itself, and the best I was able to reach was -2.015 LB score in those 9 hours, batchsize of 1024 was used.\nI trained the final model on GCP (thank you for those free credits) for around 30 hours with a batchsize of 2048 on a V100 GPU, gradients were noisy with smaller batches. I did not do oversampling, the loss function took care of it. But, I did scale the embedding gradients based on their frequencies. I could not train it any more, credits were gone, and the competition was ending too. But 30 hours of training gave lb score of -2.607 from a single model.\nLater, I fine tuned the model for first three types(time was up) on Kaggle kernel by accordingly adjusting the weights for the respective type of JC. It gave me boost of around -0.015, not that it would have changed my final rank๐.
\n\nI learned a lot, read research papers, blogs, watched Jeremy Howard's lecture videos. \nIt was fun.
\n\nAgain, thanks to everyone.
\n\nAlso, if you have any suggestions, or any thing at all, leave a remark.
\n\nPS: wish me luck for my upcoming mid-semester examinations, I will need it.
", "messages": [ { "id": "620715", "postDate": "09/07/2019 21:57:14", "content": "First of all, tons of thank you to the Kaggle community, the Kaggle team and competition hosts for the event. I am just starting out in the field (doing my bachelors now) and so if I made any mistake here or you have any suggestions, please let me know.\nI took part in the competition to learn and be comfortable with Pytorch with a hands on approach. I did lots of tensor manipulation. I did not have much expectation(it is intimidating to see so many data science experts, also I joined the competition late). And then I got a medal... weird.
\n\nModel details\nToo much talk, real deal is as follows:\nI trained one model responsible for calculating J-coupling(referred as JC next) of all types. The model produced 8 outputs, one each for a type, of which 7 of them were in don't care condition for any input. I directly optimized the evaluation metric, using it as loss.\nLike many teams I used an attention based NN model (Fast.ai's multi-head-attention of text module was an very important piece), to generate features and concatenated it with other engineered features like from 'distance is all you need' kernel. Wikipedia helped me generate few more features. Overall, it is just a variant of BERT.\nAtoms of the molecule whose JC was to be found were fed to the model one by one as input steps, zero padded 30 input steps. I added a classification token(well, actually a series of numbers that represented the learned token) as the first input step (time step sounded weird and atom step weirder) about which I did not know what would be suitable so I trained a small NN based on simple features like input type. All other input step consisted of features that was concerned with a particular atom. Each time step had a binary feature which told the model if the atom for which JC was to be computed. Features were collected from the attention network by concatenating output of first input step (say special input step, due to lack of better name, I think I need to be better with names) maximum and average of its output along all input steps with outputs from steps of atoms whose JC was to be calculated, all total 5X length of encoded output of any input step). These with hand engineered features (after going through few residual layers) were passed through another residual layer and then to a feed forward layer to generate 8 outputs, which was then trained with Adam optimizer using cyclic learning rate.
\n\nTraining details\nAt first I trained a models on the Kaggle kernel itself, and the best I was able to reach was -2.015 LB score in those 9 hours, batchsize of 1024 was used.\nI trained the final model on GCP (thank you for those free credits) for around 30 hours with a batchsize of 2048 on a V100 GPU, gradients were noisy with smaller batches. I did not do oversampling, the loss function took care of it. But, I did scale the embedding gradients based on their frequencies. I could not train it any more, credits were gone, and the competition was ending too. But 30 hours of training gave lb score of -2.607 from a single model.\nLater, I fine tuned the model for first three types(time was up) on Kaggle kernel by accordingly adjusting the weights for the respective type of JC. It gave me boost of around -0.015, not that it would have changed my final rank๐.
\n\nI learned a lot, read research papers, blogs, watched Jeremy Howard's lecture videos. \nIt was fun.
\n\nAgain, thanks to everyone.
\n\nAlso, if you have any suggestions, or any thing at all, leave a remark.
\n\nPS: wish me luck for my upcoming mid-semester examinations, I will need it.
", "rawMarkdown": "First of all, **tons of thank you** to the Kaggle community, the Kaggle team and competition hosts for the event. I am just starting out in the field (doing my bachelors now) and so if I made any mistake here or you have any suggestions, please let me know.\nI took part in the competition to learn and be comfortable with Pytorch with a hands on approach. I did lots of tensor manipulation. I did not have much expectation(it is intimidating to see so many data science experts, also I joined the competition late). And then I got a medal... weird.\n\n**Model details**\nToo much talk, real deal is as follows:\nI trained **one model responsible for calculating J-coupling**(referred as JC next) **of all types**. The model **produced 8 outputs, one each for a type, of which 7 of them were in don't care condition for any input**. I directly optimized the evaluation metric, using it as loss.\nLike many teams I used an **attention based NN model** (Fast.ai's multi-head-attention of text module was an very important piece), to generate features and concatenated it with other engineered features like from 'distance is all you need' kernel. Wikipedia helped me generate few more features. Overall, it is just a **variant of BERT.**\nAtoms of the molecule whose JC was to be found were fed to the model one by one as input steps, zero padded 30 input steps. I added a classification token(well, actually a series of numbers that represented the learned token) as the first input step (time step sounded weird and atom step weirder) about which I did not know what would be suitable so I trained a small NN based on simple features like input type. All other input step consisted of features that was concerned with a particular atom. Each time step had a binary feature which told the model if the atom for which JC was to be computed. Features were collected from the attention network by concatenating output of first input step (say special input step, due to lack of better name, I think I need to be better with names) maximum and average of its output along all input steps with outputs from steps of atoms whose JC was to be calculated, all total 5X length of encoded output of any input step). These with hand engineered features (after going through few residual layers) were passed through another residual layer and then to a feed forward layer to generate 8 outputs, which was then trained with Adam optimizer using cyclic learning rate.\n\n**Training details**\nAt first I trained a models on the Kaggle kernel itself, and the best I was able to reach was -2.015 LB score in those 9 hours, batchsize of 1024 was used.\nI trained the final model on GCP (thank you for those free credits) for around **30 hours with a batchsize of 2048 on a V100 GPU**, gradients were noisy with smaller batches. I did not do oversampling, the loss function took care of it. But, I did scale the embedding gradients based on their frequencies. I could not train it any more, credits were gone, and the competition was ending too. But 30 hours of training gave lb score of **-2.607 from a single model.**\nLater, I fine tuned the model for first three types(time was up) on Kaggle kernel by accordingly adjusting the weights for the respective type of JC. It gave me boost of around -0.015, not that it would have changed my final rank๐. \n\nI learned a lot, read research papers, blogs, watched Jeremy Howard's lecture videos. \nIt was fun.\n\n**Again, thanks to everyone.**\n\nAlso, if you have any suggestions, or any thing at all, leave a remark.\n\nPS: wish me luck for my upcoming mid-semester examinations, I will need it.", "votes": null }, { "id": "620920", "postDate": "09/08/2019 04:15:33", "content": "Congratulations\nGreat work....
\n\nWish U Good Luck..
\n\nAnd Thanks for sharing your approach and insights.!! @greenahn
", "rawMarkdown": "Congratulations\nGreat work....\n\nWish U Good Luck..\n\nAnd Thanks for sharing your approach and insights.!! @greenahn", "votes": null }, { "id": "621275", "postDate": "09/08/2019 11:52:25", "content": "Thank you for your kind words.๐
", "rawMarkdown": "Thank you for your kind words.๐", "votes": null }, { "id": "621335", "postDate": "09/08/2019 12:47:19", "content": "Congrats Anupam...
", "rawMarkdown": "Congrats Anupam...", "votes": null }, { "id": "621714", "postDate": "09/08/2019 21:11:02", "content": "Thank you.
", "rawMarkdown": "Thank you.", "votes": null } ], "comments": [ { "id": 620920, "author_name": "veeralakrishna", "author_url": "", "post_date": "09/08/2019 04:15:33", "content": "Congratulations\nGreat work....
\n\nWish U Good Luck..
\n\nAnd Thanks for sharing your approach and insights.!! @greenahn
", "votes": null, "replies": [ { "id": 621275, "author_name": "greenahn", "author_url": "", "post_date": "09/08/2019 11:52:25", "content": "Thank you for your kind words.๐
", "votes": null, "replies": [] } ] }, { "id": 621335, "author_name": "patraism", "author_url": "", "post_date": "09/08/2019 12:47:19", "content": "Congrats Anupam...
", "votes": null, "replies": [ { "id": 621714, "author_name": "greenahn", "author_url": "", "post_date": "09/08/2019 21:11:02", "content": "Thank you.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "620715": "First of all, **tons of thank you** to the Kaggle community, the Kaggle team and competition hosts for the event. I am just starting out in the field (doing my bachelors now) and so if I made any mistake here or you have any suggestions, please let me know.\nI took part in the competition to learn and be comfortable with Pytorch with a hands on approach. I did lots of tensor manipulation. I did not have much expectation(it is intimidating to see so many data science experts, also I joined the competition late). And then I got a medal... weird.\n\n**Model details**\nToo much talk, real deal is as follows:\nI trained **one model responsible for calculating J-coupling**(referred as JC next) **of all types**. The model **produced 8 outputs, one each for a type, of which 7 of them were in don't care condition for any input**. I directly optimized the evaluation metric, using it as loss.\nLike many teams I used an **attention based NN model** (Fast.ai's multi-head-attention of text module was an very important piece), to generate features and concatenated it with other engineered features like from 'distance is all you need' kernel. Wikipedia helped me generate few more features. Overall, it is just a **variant of BERT.**\nAtoms of the molecule whose JC was to be found were fed to the model one by one as input steps, zero padded 30 input steps. I added a classification token(well, actually a series of numbers that represented the learned token) as the first input step (time step sounded weird and atom step weirder) about which I did not know what would be suitable so I trained a small NN based on simple features like input type. All other input step consisted of features that was concerned with a particular atom. Each time step had a binary feature which told the model if the atom for which JC was to be computed. Features were collected from the attention network by concatenating output of first input step (say special input step, due to lack of better name, I think I need to be better with names) maximum and average of its output along all input steps with outputs from steps of atoms whose JC was to be calculated, all total 5X length of encoded output of any input step). These with hand engineered features (after going through few residual layers) were passed through another residual layer and then to a feed forward layer to generate 8 outputs, which was then trained with Adam optimizer using cyclic learning rate.\n\n**Training details**\nAt first I trained a models on the Kaggle kernel itself, and the best I was able to reach was -2.015 LB score in those 9 hours, batchsize of 1024 was used.\nI trained the final model on GCP (thank you for those free credits) for around **30 hours with a batchsize of 2048 on a V100 GPU**, gradients were noisy with smaller batches. I did not do oversampling, the loss function took care of it. But, I did scale the embedding gradients based on their frequencies. I could not train it any more, credits were gone, and the competition was ending too. But 30 hours of training gave lb score of **-2.607 from a single model.**\nLater, I fine tuned the model for first three types(time was up) on Kaggle kernel by accordingly adjusting the weights for the respective type of JC. It gave me boost of around -0.015, not that it would have changed my final rank๐. \n\nI learned a lot, read research papers, blogs, watched Jeremy Howard's lecture videos. \nIt was fun.\n\n**Again, thanks to everyone.**\n\nAlso, if you have any suggestions, or any thing at all, leave a remark.\n\nPS: wish me luck for my upcoming mid-semester examinations, I will need it.", "620920": "Congratulations\nGreat work....\n\nWish U Good Luck..\n\nAnd Thanks for sharing your approach and insights.!! @greenahn", "621275": "Thank you for your kind words.๐", "621335": "Congrats Anupam...", "621714": "Thank you." }, "source": "meta" }