If a minute data of a specific asset is missing, it creates NaN log_returns for times t-1 and t-16, because their log_return is calculated using data at time t).
\nAnd this triggers many zero betas because beta is based on a 3750period average of log_returns, so over-2-day timestamps (3750/60/24 days, to be exact) receive a ZERO beta (for that specific asset) too.
As was noticed way back, each and every first minute of a month (i.e.: 00:00 of 2021/10/01) is missing.
Here is the deal: if a timestamp is completely missing (such as the described 'month's first minute case'), in current 'host' methodology (including the latest supplemantal_train data), calculations 'go on', so beta is not affected. It is like, you almost shift the whole dataset for that missing timestamp, and behave as everything is fine.
\nWhat is the concrete affect?: if you have a consistent methodology (so if missing data means missing log_returns, which I think is the right way to do), your score for the affected 2 days drop to half.
\nHere are the options:
\nhost makes sure there is no missing timestamp, and got rid of this weird 'first-minute-missing' issue.
host makes sure whatever is done, it is done consistently: so if data is missing, you dont make up targets for a scenario (if the timestamp is completely missing) and not do that for other scenarios (timestamp is there but some asset data is missing).
host leaves it as it is, (but why?)
I am continuing hallucinating: so there is no such issue, everything is fine and consistent, host is very responsive and all is well etc.
If a minute data of a specific asset is missing, it creates NaN log_returns for times t-1 and t-16, because their log_return is calculated using data at time t).
\nAnd this triggers many zero betas because beta is based on a 3750period average of log_returns, so over-2-day timestamps (3750/60/24 days, to be exact) receive a ZERO beta (for that specific asset) too.
As was noticed way back, each and every first minute of a month (i.e.: 00:00 of 2021/10/01) is missing.
Here is the deal: if a timestamp is completely missing (such as the described 'month's first minute case'), in current 'host' methodology (including the latest supplemantal_train data), calculations 'go on', so beta is not affected. It is like, you almost shift the whole dataset for that missing timestamp, and behave as everything is fine.
\nWhat is the concrete affect?: if you have a consistent methodology (so if missing data means missing log_returns, which I think is the right way to do), your score for the affected 2 days drop to half.
\nHere are the options:
\nhost makes sure there is no missing timestamp, and got rid of this weird 'first-minute-missing' issue.
host makes sure whatever is done, it is done consistently: so if data is missing, you dont make up targets for a scenario (if the timestamp is completely missing) and not do that for other scenarios (timestamp is there but some asset data is missing).
host leaves it as it is, (but why?)
I am continuing hallucinating: so there is no such issue, everything is fine and consistent, host is very responsive and all is well etc.
I'm not too sure about 2, but I do agree that that nans seem to be triggering odd artificial zero-betas….
\nIn case it helps, this note book was very useful for me in understanding the precise methodology for target creation:
\nhttps://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota
You are not hallucinating but I feel like it's too late to change the target now.
", "rawMarkdown": "You are not hallucinating but I feel like it's too late to change the target now.", "votes": null }, { "id": "1652976", "postDate": "01/17/2022 06:30:06", "content": "As you are saying, it might be too late to change, although versions of this issue have been brought up early in the competition. Those early complaints did not bring up the issue with its concrete affect but the host could have acted.
\nThe bigger issue is that things are not being confirmed, so are we 100% sure what is expected of us, if a timestamp data is completely missing beyond the first minutes of each month?
", "rawMarkdown": "As you are saying, it might be too late to change, although versions of this issue have been brought up early in the competition. Those early complaints did not bring up the issue with its concrete affect but the host could have acted. \n\nThe bigger issue is that things are not being confirmed, so are we 100% sure what is expected of us, if a timestamp data is completely missing beyond the first minutes of each month?", "votes": null }, { "id": "1653318", "postDate": "01/17/2022 12:36:46", "content": "It's going to be the 3 option.
\nWhy? Because it is the easiest option for the host and Kaggle. By now, we know they are not going to change anything.
", "rawMarkdown": "It's going to be the 3 option.\n\nWhy? Because it is the easiest option for the host and Kaggle. By now, we know they are not going to change anything.", "votes": null }, { "id": "1656819", "postDate": "01/19/2022 16:46:29", "content": "We prefer to avoid modifying the data after the competition has started. You can expect the same data processing for the test dataset and training dataset.
", "rawMarkdown": "We prefer to avoid modifying the data after the competition has started. You can expect the same data processing for the test dataset and training dataset.", "votes": null } ], "comments": [ { "id": 1652221, "author_name": "maxcorr", "author_url": "", "post_date": "01/16/2022 12:35:47", "content": "I'm not too sure about 2, but I do agree that that nans seem to be triggering odd artificial zero-betas….
\nIn case it helps, this note book was very useful for me in understanding the precise methodology for target creation:
\nhttps://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota
You are not hallucinating but I feel like it's too late to change the target now.
", "votes": null, "replies": [ { "id": 1652976, "author_name": "abdulkadirguner", "author_url": "", "post_date": "01/17/2022 06:30:06", "content": "As you are saying, it might be too late to change, although versions of this issue have been brought up early in the competition. Those early complaints did not bring up the issue with its concrete affect but the host could have acted.
\nThe bigger issue is that things are not being confirmed, so are we 100% sure what is expected of us, if a timestamp data is completely missing beyond the first minutes of each month?
", "votes": null, "replies": [] } ] }, { "id": 1653318, "author_name": "agenlu", "author_url": "", "post_date": "01/17/2022 12:36:46", "content": "It's going to be the 3 option.
\nWhy? Because it is the easiest option for the host and Kaggle. By now, we know they are not going to change anything.
", "votes": null, "replies": [] }, { "id": 1656819, "author_name": "cstein06", "author_url": "", "post_date": "01/19/2022 16:46:29", "content": "We prefer to avoid modifying the data after the competition has started. You can expect the same data processing for the test dataset and training dataset.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1651795": "1. If a minute data of a specific asset is missing, it creates NaN log_returns for times t-1 and t-16, because their log_return is calculated using data at time t).\nAnd this triggers many zero betas because beta is based on a 3750period average of log_returns, so over-2-day timestamps (3750/60/24 days, to be exact) **receive a ZERO beta** (for that specific asset) too.\n\n2. **As was noticed way back**, each and every first minute of a month (i.e.: 00:00 of 2021/10/01) is missing.\n\nHere is the deal: if a timestamp is completely missing (such as the described 'month's first minute case'), in current 'host' methodology (including the latest supplemantal_train data), calculations 'go on', **so beta is not affected**. It is like, you almost shift the whole dataset for that missing timestamp, and behave as everything is fine. \n\nWhat is the concrete affect?: if you have a consistent methodology (so if missing data means missing log_returns, which I think is the right way to do), **your score for the affected 2 days drop to half.** \n\n\nHere are the options:\n\n1. host makes sure there is no missing timestamp, and got rid of this weird 'first-minute-missing' issue. \n\n2. host makes sure whatever is done, it is done consistently: so if data is missing, you dont make up targets for a scenario (if the timestamp is completely missing) and not do that for other scenarios (timestamp is there but some asset data is missing).\n\n3. host leaves it as it is, (but why?)\n\n4. I am continuing hallucinating: so there is no such issue, everything is fine and consistent, host is very responsive and all is well etc.", "1652221": "I'm not too sure about 2, but I do agree that that nans seem to be triggering odd artificial zero-betas.... \n\nIn case it helps, this note book was very useful for me in understanding the precise methodology for target creation:\n[https://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota](url)", "1652516": "You are not hallucinating but I feel like it's too late to change the target now.", "1652976": "As you are saying, it might be too late to change, although versions of this issue have been brought up early in the competition. Those early complaints did not bring up the issue with its concrete affect but the host could have acted. \n\nThe bigger issue is that things are not being confirmed, so are we 100% sure what is expected of us, if a timestamp data is completely missing beyond the first minutes of each month?", "1653318": "It's going to be the 3 option.\n\nWhy? Because it is the easiest option for the host and Kaggle. By now, we know they are not going to change anything.", "1656819": "We prefer to avoid modifying the data after the competition has started. You can expect the same data processing for the test dataset and training dataset." }, "source": "meta" }