In other competitions we participants have a very clear view of whether the top models would be useful in a real setting (e.g segmenting cars and error is ~<2px per image) but others like this one one wonders where this is an still scientific work-in-progress and if at the end of the day you'd go back to costly quantum simulations to sweep corpus of target molecules.
\n\nSo direct question: what thresholds of LMAE would be practical for scientific usage and in what contexts?
", "messages": [ { "id": "567318", "postDate": "07/03/2019 10:43:53", "content": "In other competitions we participants have a very clear view of whether the top models would be useful in a real setting (e.g segmenting cars and error is ~<2px per image) but others like this one one wonders where this is an still scientific work-in-progress and if at the end of the day you'd go back to costly quantum simulations to sweep corpus of target molecules.
\n\nSo direct question: what thresholds of LMAE would be practical for scientific usage and in what contexts?
", "rawMarkdown": "In other competitions we participants have a very clear view of whether the top models would be useful in a real setting (e.g segmenting cars and error is ~<2px per image) but others like this one one wonders where this is an still scientific work-in-progress and if at the end of the day you'd go back to costly quantum simulations to sweep corpus of target molecules.\n\nSo direct question: what thresholds of LMAE would be practical for scientific usage and in what contexts?", "votes": null }, { "id": "567357", "postDate": "07/03/2019 11:47:21", "content": "When atomization energy is predicted, it is 1 kcal/mol or 0.043 eV or 0.0016 Ha, so called chemical accuracy, but in this case, I have no idea.
", "rawMarkdown": "When atomization energy is predicted, it is 1 kcal/mol or 0.043 eV or 0.0016 Ha, so called chemical accuracy, but in this case, I have no idea.", "votes": null }, { "id": "567420", "postDate": "07/03/2019 13:48:22", "content": "Different people have different uses, and I don't know what the organisers have in mind... But for me, I would use the near-instant computation of J-coupling to help assign my spectra. For this purpose then even 5% error would be ok.
\n\nFor more advanced purposes (such as giving a spectrum to the machine and asking it to predict the molecule based on J-coupling and chemical shift) the accuracy might have to be more like 1%.
", "rawMarkdown": "Different people have different uses, and I don't know what the organisers have in mind... But for me, I would use the near-instant computation of J-coupling to help assign my spectra. For this purpose then even 5% error would be ok.\n\nFor more advanced purposes (such as giving a spectrum to the machine and asking it to predict the molecule based on J-coupling and chemical shift) the accuracy might have to be more like 1%.", "votes": null }, { "id": "567644", "postDate": "07/03/2019 19:30:39", "content": "Do you know how does translates into LMAE for each j-coupling type?
", "rawMarkdown": "Do you know how does translates into LMAE for each j-coupling type?", "votes": null }, { "id": "567736", "postDate": "07/04/2019 00:27:21", "content": "That's a really good question, have been wondering that myself. Would really appreciate it if the organizers could shed some light on it.
", "rawMarkdown": "That's a really good question, have been wondering that myself. Would really appreciate it if the organizers could shed some light on it.", "votes": null }, { "id": "567738", "postDate": "07/04/2019 00:29:53", "content": "", "rawMarkdown": "", "votes": null }, { "id": "569312", "postDate": "07/06/2019 12:40:14", "content": "This is quite difficult to say, especially since the score is averaged over different coupling types, but I can try to give some insight. We've mostly worked with 1JHC couplings so far, and an MAE of 1 Hz for these are quite good, and we are currently able to achieve that accuracy training on 10K samples (extrapolating to 1M points would give us an accuracy of 0.3 Hz MAE).\nThere's a set amount of error in the level of theory that one uses to generate the training data, so once the error in your machine learning predictions becomes less that the error of the theory, you don't gain much by making improvements. However if you have a very accurate machine learning algorithm, you can start to use a more accurate but much more computationally expensive method and decrease the size of the training set.\nThat being set, before the competition launched I would have been slightly disappointing if the best solution ended up being less than -2 (based on rough accuracy estimates), and I would have been very excited to see solutions better than -3, so so far you're all doing an amazing job!
", "rawMarkdown": "This is quite difficult to say, especially since the score is averaged over different coupling types, but I can try to give some insight. We've mostly worked with 1JHC couplings so far, and an MAE of 1 Hz for these are quite good, and we are currently able to achieve that accuracy training on 10K samples (extrapolating to 1M points would give us an accuracy of 0.3 Hz MAE).\nThere's a set amount of error in the level of theory that one uses to generate the training data, so once the error in your machine learning predictions becomes less that the error of the theory, you don't gain much by making improvements. However if you have a very accurate machine learning algorithm, you can start to use a more accurate but much more computationally expensive method and decrease the size of the training set.\nThat being set, before the competition launched I would have been slightly disappointing if the best solution ended up being less than -2 (based on rough accuracy estimates), and I would have been very excited to see solutions better than -3, so so far you're all doing an amazing job!", "votes": null } ], "comments": [ { "id": 567357, "author_name": "somnambwl", "author_url": "", "post_date": "07/03/2019 11:47:21", "content": "When atomization energy is predicted, it is 1 kcal/mol or 0.043 eV or 0.0016 Ha, so called chemical accuracy, but in this case, I have no idea.
", "votes": null, "replies": [] }, { "id": 567420, "author_name": "phucco", "author_url": "", "post_date": "07/03/2019 13:48:22", "content": "Different people have different uses, and I don't know what the organisers have in mind... But for me, I would use the near-instant computation of J-coupling to help assign my spectra. For this purpose then even 5% error would be ok.
\n\nFor more advanced purposes (such as giving a spectrum to the machine and asking it to predict the molecule based on J-coupling and chemical shift) the accuracy might have to be more like 1%.
", "votes": null, "replies": [ { "id": 567644, "author_name": "antorsae", "author_url": "", "post_date": "07/03/2019 19:30:39", "content": "Do you know how does translates into LMAE for each j-coupling type?
", "votes": null, "replies": [] } ] }, { "id": 567736, "author_name": "tunguz", "author_url": "", "post_date": "07/04/2019 00:27:21", "content": "That's a really good question, have been wondering that myself. Would really appreciate it if the organizers could shed some light on it.
", "votes": null, "replies": [] }, { "id": 567738, "author_name": "tunguz", "author_url": "", "post_date": "07/04/2019 00:29:53", "content": "", "votes": null, "replies": [] }, { "id": 569312, "author_name": "larsbratholm", "author_url": "", "post_date": "07/06/2019 12:40:14", "content": "This is quite difficult to say, especially since the score is averaged over different coupling types, but I can try to give some insight. We've mostly worked with 1JHC couplings so far, and an MAE of 1 Hz for these are quite good, and we are currently able to achieve that accuracy training on 10K samples (extrapolating to 1M points would give us an accuracy of 0.3 Hz MAE).\nThere's a set amount of error in the level of theory that one uses to generate the training data, so once the error in your machine learning predictions becomes less that the error of the theory, you don't gain much by making improvements. However if you have a very accurate machine learning algorithm, you can start to use a more accurate but much more computationally expensive method and decrease the size of the training set.\nThat being set, before the competition launched I would have been slightly disappointing if the best solution ended up being less than -2 (based on rough accuracy estimates), and I would have been very excited to see solutions better than -3, so so far you're all doing an amazing job!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "567318": "In other competitions we participants have a very clear view of whether the top models would be useful in a real setting (e.g segmenting cars and error is ~<2px per image) but others like this one one wonders where this is an still scientific work-in-progress and if at the end of the day you'd go back to costly quantum simulations to sweep corpus of target molecules.\n\nSo direct question: what thresholds of LMAE would be practical for scientific usage and in what contexts?", "567357": "When atomization energy is predicted, it is 1 kcal/mol or 0.043 eV or 0.0016 Ha, so called chemical accuracy, but in this case, I have no idea.", "567420": "Different people have different uses, and I don't know what the organisers have in mind... But for me, I would use the near-instant computation of J-coupling to help assign my spectra. For this purpose then even 5% error would be ok.\n\nFor more advanced purposes (such as giving a spectrum to the machine and asking it to predict the molecule based on J-coupling and chemical shift) the accuracy might have to be more like 1%.", "567644": "Do you know how does translates into LMAE for each j-coupling type?", "567736": "That's a really good question, have been wondering that myself. Would really appreciate it if the organizers could shed some light on it.", "567738": "", "569312": "This is quite difficult to say, especially since the score is averaged over different coupling types, but I can try to give some insight. We've mostly worked with 1JHC couplings so far, and an MAE of 1 Hz for these are quite good, and we are currently able to achieve that accuracy training on 10K samples (extrapolating to 1M points would give us an accuracy of 0.3 Hz MAE).\nThere's a set amount of error in the level of theory that one uses to generate the training data, so once the error in your machine learning predictions becomes less that the error of the theory, you don't gain much by making improvements. However if you have a very accurate machine learning algorithm, you can start to use a more accurate but much more computationally expensive method and decrease the size of the training set.\nThat being set, before the competition launched I would have been slightly disappointing if the best solution ended up being less than -2 (based on rough accuracy estimates), and I would have been very excited to see solutions better than -3, so so far you're all doing an amazing job!" }, "source": "meta" }