{ "id": 93786, "title": "Additional Data usage", "url": "/competitions/champs-scalar-coupling/discussion/93786", "author_name": "", "post_date": "2019-05-30T02:22:29.377820300Z", "votes": 11, "comment_count": 8, "views": 0, "content": "

Creating this thread to discuss the usage of Additional data. Since the test data molecules do not have the additional data, it is important to know how we can make use of the Additional data to engineer new features

", "messages": [ { "id": "539382", "postDate": "05/30/2019 02:22:29", "content": "

Creating this thread to discuss the usage of Additional data. Since the test data molecules do not have the additional data, it is important to know how we can make use of the Additional data to engineer new features

", "rawMarkdown": "Creating this thread to discuss the usage of Additional data. Since the test data molecules do not have the additional data, it is important to know how we can make use of the Additional data to engineer new features", "votes": null }, { "id": "539397", "postDate": "05/30/2019 02:55:30", "content": "

I was just about to post about this as well. Here's an overview of what data is available in train vs test.

\n\n

ONLY TRAIN:\n- Mulliken Charges - per atom\n- Magnetic Shielding - per atom\n- Potential Energy - per molecule \n- Dipole Moment - per molecule\n- Scalar Coupling Contribution - per atom-atom interaction (Note: the sum of these values is the target value, of course we don't get these for test)

\n\n

BOTH TRAIN AND TEST:\n- Position - per atom (found in structure files)\n- Type - per atom-atom interaction (found in train/test csv)

\n\n

It seems we should not use the data that is only available to us in the training set unless we can somehow extract it from the structure. I'm still trying to figure out the meaning of the data we were given, so I'm not sure of how practical it would be to do that.

", "rawMarkdown": "I was just about to post about this as well. Here's an overview of what data is available in train vs test.\n\nONLY TRAIN:\n- Mulliken Charges - per atom\n- Magnetic Shielding - per atom\n- Potential Energy - per molecule \n- Dipole Moment - per molecule\n- Scalar Coupling Contribution - per atom-atom interaction (Note: the sum of these values is the target value, of course we don't get these for test)\n\nBOTH TRAIN AND TEST:\n- Position - per atom (found in structure files)\n- Type - per atom-atom interaction (found in train/test csv)\n\nIt seems we should not use the data that is only available to us in the training set unless we can somehow extract it from the structure. I'm still trying to figure out the meaning of the data we were given, so I'm not sure of how practical it would be to do that.", "votes": null }, { "id": "539417", "postDate": "05/30/2019 03:22:57", "content": "

This is some interesting analysis. I'm curious to seeing how potential energy and dipole moment could have contributed to the training. Definitely going to take a look into seeing if we're expected to extract it from the structure.

", "rawMarkdown": "This is some interesting analysis. I'm curious to seeing how potential energy and dipole moment could have contributed to the training. Definitely going to take a look into seeing if we're expected to extract it from the structure.", "votes": null }, { "id": "539458", "postDate": "05/30/2019 04:07:35", "content": "

Since we essentially only have access to the structure data for both the train and test set, here are some features that we can extract from the structure:\n- Number of protons/electrons - per atom \n( H - 1, C - 6, N - 7, O - 8 )

\n\n\n\n

I'm going to look into mulliken charge and magnetic shielding tensor and see if there's any way to get an approximation of those as well.

", "rawMarkdown": "Since we essentially only have access to the structure data for both the train and test set, here are some features that we can extract from the structure:\n- Number of protons/electrons - per atom \n( H - 1, C - 6, N - 7, O - 8 )\n\n- Electronegativity(EN) - per atom\n( H - 2.2, C - 2.55, N - 3.04, O - 3.44 )\nHigh EN atoms hold electrons closer than low EN atoms. If a high EN atom has a bond with a low EN atom, then the shared electrons spends more time near the high EN atom. This causes polarity, where the high EN atom is slightly negatively charged and the low EN atom is slightly positively charged. It may be that we can extract the dipole moment term from this.\n\n- Atomic radius - per atom \n( H - 0.53A, C - 0.7A, N - 0.65A, O - 0.6A ) - (A = Angstroms)\n\n- Distance between each atom - per atom-atom pair\nIf this distance is less than the sum of the atomic radii of the two atoms it implies a bond. The closer they are the stronger the bond. Stronger bonds have less potential energy. It may be that we can extract the potential energy term from this.\n\nI'm going to look into mulliken charge and magnetic shielding tensor and see if there's any way to get an approximation of those as well.", "votes": null }, { "id": "540936", "postDate": "06/01/2019 11:57:45", "content": "

Something to keep in mind re: external data: There are only 5 distinct atoms in this dataset. Additional variables is not likely to add that much at this level

", "rawMarkdown": "Something to keep in mind re: external data: There are only 5 distinct atoms in this dataset. Additional variables is not likely to add that much at this level", "votes": null }, { "id": "541049", "postDate": "06/01/2019 16:05:01", "content": "

I disagree with this sentiment. Those 5 atoms are largely responsible for all biological processes. We're dealing with relatively small molecules, but even those can form pretty complex molecules.

", "rawMarkdown": "I disagree with this sentiment. Those 5 atoms are largely responsible for all biological processes. We're dealing with relatively small molecules, but even those can form pretty complex molecules.", "votes": null }, { "id": "542106", "postDate": "06/03/2019 13:15:18", "content": "

Aaron, Do you have any breakthrough or a hint about this issue?\nI can't figure why we need the additional data, such as (mulliken charges, magnetic shielding, etc.) if it is only available for the training data? \nIf this question was already answered elsewhere, please link it, because I couldn't find any info on it, despite several threads with similar questions...

", "rawMarkdown": "Aaron, Do you have any breakthrough or a hint about this issue?\nI can't figure why we need the additional data, such as (mulliken charges, magnetic shielding, etc.) if it is only available for the training data? \nIf this question was already answered elsewhere, please link it, because I couldn't find any info on it, despite several threads with similar questions...", "votes": null }, { "id": "542179", "postDate": "06/03/2019 14:43:38", "content": "

The biggest problem is that we don´t have the same information in the train and test sets. Maybe we need to obtain the aditional information from other pretrained network and use it in the prediction model. What do you thing about this?

", "rawMarkdown": "The biggest problem is that we don´t have the same information in the train and test sets. Maybe we need to obtain the aditional information from other pretrained network and use it in the prediction model. What do you thing about this?", "votes": null }, { "id": "544438", "postDate": "06/05/2019 14:07:58", "content": "

This is in the context of adding external data to \"enrich\" the atoms (e.g. I initially tried adding properties from the periodic table of elements) . \nWith just 5 variables, adding more variables to \"generalize\" doesn't make sense

", "rawMarkdown": "This is in the context of adding external data to \"enrich\" the atoms (e.g. I initially tried adding properties from the periodic table of elements) . \nWith just 5 variables, adding more variables to \"generalize\" doesn't make sense", "votes": null } ], "comments": [ { "id": 539397, "author_name": "aekoch95", "author_url": "", "post_date": "05/30/2019 02:55:30", "content": "

I was just about to post about this as well. Here's an overview of what data is available in train vs test.

\n\n

ONLY TRAIN:\n- Mulliken Charges - per atom\n- Magnetic Shielding - per atom\n- Potential Energy - per molecule \n- Dipole Moment - per molecule\n- Scalar Coupling Contribution - per atom-atom interaction (Note: the sum of these values is the target value, of course we don't get these for test)

\n\n

BOTH TRAIN AND TEST:\n- Position - per atom (found in structure files)\n- Type - per atom-atom interaction (found in train/test csv)

\n\n

It seems we should not use the data that is only available to us in the training set unless we can somehow extract it from the structure. I'm still trying to figure out the meaning of the data we were given, so I'm not sure of how practical it would be to do that.

", "votes": null, "replies": [ { "id": 539417, "author_name": "anishagnihotri", "author_url": "", "post_date": "05/30/2019 03:22:57", "content": "

This is some interesting analysis. I'm curious to seeing how potential energy and dipole moment could have contributed to the training. Definitely going to take a look into seeing if we're expected to extract it from the structure.

", "votes": null, "replies": [] }, { "id": 542106, "author_name": "timsonrisa", "author_url": "", "post_date": "06/03/2019 13:15:18", "content": "

Aaron, Do you have any breakthrough or a hint about this issue?\nI can't figure why we need the additional data, such as (mulliken charges, magnetic shielding, etc.) if it is only available for the training data? \nIf this question was already answered elsewhere, please link it, because I couldn't find any info on it, despite several threads with similar questions...

", "votes": null, "replies": [] } ] }, { "id": 539458, "author_name": "aekoch95", "author_url": "", "post_date": "05/30/2019 04:07:35", "content": "

Since we essentially only have access to the structure data for both the train and test set, here are some features that we can extract from the structure:\n- Number of protons/electrons - per atom \n( H - 1, C - 6, N - 7, O - 8 )

\n\n\n\n

I'm going to look into mulliken charge and magnetic shielding tensor and see if there's any way to get an approximation of those as well.

", "votes": null, "replies": [] }, { "id": 540936, "author_name": "danofer", "author_url": "", "post_date": "06/01/2019 11:57:45", "content": "

Something to keep in mind re: external data: There are only 5 distinct atoms in this dataset. Additional variables is not likely to add that much at this level

", "votes": null, "replies": [ { "id": 541049, "author_name": "aekoch95", "author_url": "", "post_date": "06/01/2019 16:05:01", "content": "

I disagree with this sentiment. Those 5 atoms are largely responsible for all biological processes. We're dealing with relatively small molecules, but even those can form pretty complex molecules.

", "votes": null, "replies": [] }, { "id": 544438, "author_name": "danofer", "author_url": "", "post_date": "06/05/2019 14:07:58", "content": "

This is in the context of adding external data to \"enrich\" the atoms (e.g. I initially tried adding properties from the periodic table of elements) . \nWith just 5 variables, adding more variables to \"generalize\" doesn't make sense

", "votes": null, "replies": [] } ] }, { "id": 542179, "author_name": "khiwila", "author_url": "", "post_date": "06/03/2019 14:43:38", "content": "

The biggest problem is that we don´t have the same information in the train and test sets. Maybe we need to obtain the aditional information from other pretrained network and use it in the prediction model. What do you thing about this?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "539382": "Creating this thread to discuss the usage of Additional data. Since the test data molecules do not have the additional data, it is important to know how we can make use of the Additional data to engineer new features", "539397": "I was just about to post about this as well. Here's an overview of what data is available in train vs test.\n\nONLY TRAIN:\n- Mulliken Charges - per atom\n- Magnetic Shielding - per atom\n- Potential Energy - per molecule \n- Dipole Moment - per molecule\n- Scalar Coupling Contribution - per atom-atom interaction (Note: the sum of these values is the target value, of course we don't get these for test)\n\nBOTH TRAIN AND TEST:\n- Position - per atom (found in structure files)\n- Type - per atom-atom interaction (found in train/test csv)\n\nIt seems we should not use the data that is only available to us in the training set unless we can somehow extract it from the structure. I'm still trying to figure out the meaning of the data we were given, so I'm not sure of how practical it would be to do that.", "539417": "This is some interesting analysis. I'm curious to seeing how potential energy and dipole moment could have contributed to the training. Definitely going to take a look into seeing if we're expected to extract it from the structure.", "539458": "Since we essentially only have access to the structure data for both the train and test set, here are some features that we can extract from the structure:\n- Number of protons/electrons - per atom \n( H - 1, C - 6, N - 7, O - 8 )\n\n- Electronegativity(EN) - per atom\n( H - 2.2, C - 2.55, N - 3.04, O - 3.44 )\nHigh EN atoms hold electrons closer than low EN atoms. If a high EN atom has a bond with a low EN atom, then the shared electrons spends more time near the high EN atom. This causes polarity, where the high EN atom is slightly negatively charged and the low EN atom is slightly positively charged. It may be that we can extract the dipole moment term from this.\n\n- Atomic radius - per atom \n( H - 0.53A, C - 0.7A, N - 0.65A, O - 0.6A ) - (A = Angstroms)\n\n- Distance between each atom - per atom-atom pair\nIf this distance is less than the sum of the atomic radii of the two atoms it implies a bond. The closer they are the stronger the bond. Stronger bonds have less potential energy. It may be that we can extract the potential energy term from this.\n\nI'm going to look into mulliken charge and magnetic shielding tensor and see if there's any way to get an approximation of those as well.", "540936": "Something to keep in mind re: external data: There are only 5 distinct atoms in this dataset. Additional variables is not likely to add that much at this level", "541049": "I disagree with this sentiment. Those 5 atoms are largely responsible for all biological processes. We're dealing with relatively small molecules, but even those can form pretty complex molecules.", "542106": "Aaron, Do you have any breakthrough or a hint about this issue?\nI can't figure why we need the additional data, such as (mulliken charges, magnetic shielding, etc.) if it is only available for the training data? \nIf this question was already answered elsewhere, please link it, because I couldn't find any info on it, despite several threads with similar questions...", "542179": "The biggest problem is that we don´t have the same information in the train and test sets. Maybe we need to obtain the aditional information from other pretrained network and use it in the prediction model. What do you thing about this?", "544438": "This is in the context of adding external data to \"enrich\" the atoms (e.g. I initially tried adding properties from the periodic table of elements) . \nWith just 5 variables, adding more variables to \"generalize\" doesn't make sense" }, "source": "meta" }