After investigating a bit what could be wrong with my sub it seems that I still have significant reconstruction error on the target. I can't find where my mistake is. Would someone mind to share his online construction of the target ?
", "messages": [ { "id": "1696994", "postDate": "02/19/2022 09:43:12", "content": "After investigating a bit what could be wrong with my sub it seems that I still have significant reconstruction error on the target. I can't find where my mistake is. Would someone mind to share his online construction of the target ?
", "rawMarkdown": "After investigating a bit what could be wrong with my sub it seems that I still have significant reconstruction error on the target. I can't find where my mistake is. Would someone mind to share his online construction of the target ?", "votes": null }, { "id": "1697154", "postDate": "02/19/2022 12:08:54", "content": "@lucasmorin, can't give a full reprex easily, but mine uses this:
\nclass RunningTargets:\n def __init__(self,\n weights,\n maxlen=7*24*60\n ):\n self.weights = weights\n self.maxlen = maxlen\n self.batches_seen = 0\n\n # indexed in real time\n self.timestamp_deque = deque(list())\n\n # indexed 17 timestamps in past\n self.target_timestamp_deque = deque(list())\n\n # target features\n self.close_arr = np.empty(shape=(maxlen, 14))\n self.returns_arr = np.empty(shape=(maxlen, 14))\n self.market_arr = np.empty(shape=(maxlen,))\n self.targets_arr = np.empty(shape=(maxlen, 14))\n\n def append_close(self, timestamp, close_arr):\n self.batches_seen += 1\n self.timestamp_deque.append(timestamp)\n self.close_arr = np.append(self.close_arr, np.expand_dims(close_arr, axis=0), axis=0)\n self.close_arr = np.delete(self.close_arr, 0, axis=0)\n\n def append_returns(self):\n if self.batches_seen < 17:\n print(f\"Returns ready in {17 - self.batches_seen}...\")\n pass\n else:\n ts = self.timestamp_deque[-17]\n self.target_timestamp_deque.append(ts)\n returns_arr = (self.close_arr[-1]/self.close_arr[-16]) - 1\n self.returns_arr = np.append(self.returns_arr, np.expand_dims(returns_arr, axis=0), axis=0)\n self.returns_arr = np.delete(self.returns_arr, 0, axis=0)\n market = np.average(np.nan_to_num(returns_arr), weights=self.weights)\n self.market_arr = np.append(self.market_arr, market)\n self.market_arr = np.delete(self.market_arr, 0, axis=0)\n\n def append_targets(self, winlen=3750):\n if self.batches_seen < 17:\n pass\n else:\n if self.batches_seen < winlen:\n bet = 0\n if self.batches_seen == winlen - 1:\n print(\"Beta is now ready!\")\n else:\n num = np.mean((self.returns_arr[-winlen:].T * self.market_arr[-winlen:]), axis=1)\n den = np.mean(self.market_arr[-winlen:] ** 2, axis=0)\n bet = np.nan_to_num(num.T / den, nan=0., posinf=0., neginf=0.)\n targets = self.returns_arr[-1] - bet * self.market_arr[-1]\n self.targets_arr = np.append(self.targets_arr, np.expand_dims(targets, axis=0), axis=0)\n self.targets_arr = np.delete(self.targets_arr, 0, axis=0)\nweights and close_arr are both np arrays of shape (14,), indexed by asset id.It is warmed up for the ~ 3800 timestamps from the end of supplemental_train.csv using my local_api so that the Targets are accurate in time for the beginning of the private LB.
Thanks. I finally took the time to try it. It seems to be able to build target with nearly 0 error (1e-17) except for some very rare time ids. I'll try it in an only learning framework.
", "rawMarkdown": "Thanks. I finally took the time to try it. It seems to be able to build target with nearly 0 error (1e-17) except for some very rare time ids. I'll try it in an only learning framework.", "votes": null }, { "id": "1714436", "postDate": "03/07/2022 02:12:49", "content": "No good in doing so since the hosts seem to use a different target calculation method compared to targets in train data.
", "rawMarkdown": "No good in doing so since the hosts seem to use a different target calculation method compared to targets in train data.", "votes": null }, { "id": "1714766", "postDate": "03/07/2022 09:54:45", "content": "@noi031 Can you clarify why you think this is the case? If the target calculation method is different then there's no point in this competition at all.
", "rawMarkdown": "noi031 Can you clarify why you think this is the case? If the target calculation method is different then there's no point in this competition at all.", "votes": null }, { "id": "1718630", "postDate": "03/11/2022 03:01:18", "content": "I found that setting beta=0 created a noisy target as it would flip between \"average log return\" and \"average relative log return\" anytime you had a single NaN in your lookback window. As the data quality gets better towards the end of the training set this happens less often, and never for some coins.
\nFor the on-line reconstructed target I think it makes more sense to forward-fill the data (and thus never have beta=0). It has a higher error when compared to the target in the training set, but it gives you a less arbitrary signal for training. I think its a good bet that the data quality in the test set will match the end of the training set, or even be better, because the historical quality issues look like growing pains in the their tech stack.
", "rawMarkdown": "I found that setting beta=0 created a noisy target as it would flip between \"average log return\" and \"average relative log return\" anytime you had a single NaN in your lookback window. As the data quality gets better towards the end of the training set this happens less often, and never for some coins. \n\nFor the on-line reconstructed target I think it makes more sense to forward-fill the data (and thus never have beta=0). It has a higher error when compared to the target in the training set, but it gives you a less arbitrary signal for training. I think its a good bet that the data quality in the test set will match the end of the training set, or even be better, because the historical quality issues look like growing pains in the their tech stack.", "votes": null } ], "comments": [ { "id": 1697154, "author_name": "jagofc", "author_url": "", "post_date": "02/19/2022 12:08:54", "content": "@lucasmorin, can't give a full reprex easily, but mine uses this:
\nclass RunningTargets:\n def __init__(self,\n weights,\n maxlen=7*24*60\n ):\n self.weights = weights\n self.maxlen = maxlen\n self.batches_seen = 0\n\n # indexed in real time\n self.timestamp_deque = deque(list())\n\n # indexed 17 timestamps in past\n self.target_timestamp_deque = deque(list())\n\n # target features\n self.close_arr = np.empty(shape=(maxlen, 14))\n self.returns_arr = np.empty(shape=(maxlen, 14))\n self.market_arr = np.empty(shape=(maxlen,))\n self.targets_arr = np.empty(shape=(maxlen, 14))\n\n def append_close(self, timestamp, close_arr):\n self.batches_seen += 1\n self.timestamp_deque.append(timestamp)\n self.close_arr = np.append(self.close_arr, np.expand_dims(close_arr, axis=0), axis=0)\n self.close_arr = np.delete(self.close_arr, 0, axis=0)\n\n def append_returns(self):\n if self.batches_seen < 17:\n print(f\"Returns ready in {17 - self.batches_seen}...\")\n pass\n else:\n ts = self.timestamp_deque[-17]\n self.target_timestamp_deque.append(ts)\n returns_arr = (self.close_arr[-1]/self.close_arr[-16]) - 1\n self.returns_arr = np.append(self.returns_arr, np.expand_dims(returns_arr, axis=0), axis=0)\n self.returns_arr = np.delete(self.returns_arr, 0, axis=0)\n market = np.average(np.nan_to_num(returns_arr), weights=self.weights)\n self.market_arr = np.append(self.market_arr, market)\n self.market_arr = np.delete(self.market_arr, 0, axis=0)\n\n def append_targets(self, winlen=3750):\n if self.batches_seen < 17:\n pass\n else:\n if self.batches_seen < winlen:\n bet = 0\n if self.batches_seen == winlen - 1:\n print(\"Beta is now ready!\")\n else:\n num = np.mean((self.returns_arr[-winlen:].T * self.market_arr[-winlen:]), axis=1)\n den = np.mean(self.market_arr[-winlen:] ** 2, axis=0)\n bet = np.nan_to_num(num.T / den, nan=0., posinf=0., neginf=0.)\n targets = self.returns_arr[-1] - bet * self.market_arr[-1]\n self.targets_arr = np.append(self.targets_arr, np.expand_dims(targets, axis=0), axis=0)\n self.targets_arr = np.delete(self.targets_arr, 0, axis=0)\nweights and close_arr are both np arrays of shape (14,), indexed by asset id.It is warmed up for the ~ 3800 timestamps from the end of supplemental_train.csv using my local_api so that the Targets are accurate in time for the beginning of the private LB.
Thanks. I finally took the time to try it. It seems to be able to build target with nearly 0 error (1e-17) except for some very rare time ids. I'll try it in an only learning framework.
", "votes": null, "replies": [] }, { "id": 1714436, "author_name": "noi031", "author_url": "", "post_date": "03/07/2022 02:12:49", "content": "No good in doing so since the hosts seem to use a different target calculation method compared to targets in train data.
", "votes": null, "replies": [] }, { "id": 1714766, "author_name": "jagofc", "author_url": "", "post_date": "03/07/2022 09:54:45", "content": "@noi031 Can you clarify why you think this is the case? If the target calculation method is different then there's no point in this competition at all.
", "votes": null, "replies": [] } ] }, { "id": 1718630, "author_name": "chaseos", "author_url": "", "post_date": "03/11/2022 03:01:18", "content": "I found that setting beta=0 created a noisy target as it would flip between \"average log return\" and \"average relative log return\" anytime you had a single NaN in your lookback window. As the data quality gets better towards the end of the training set this happens less often, and never for some coins.
\nFor the on-line reconstructed target I think it makes more sense to forward-fill the data (and thus never have beta=0). It has a higher error when compared to the target in the training set, but it gives you a less arbitrary signal for training. I think its a good bet that the data quality in the test set will match the end of the training set, or even be better, because the historical quality issues look like growing pains in the their tech stack.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1696994": "After investigating a bit what could be wrong with my sub it seems that I still have significant reconstruction error on the target. I can't find where my mistake is. Would someone mind to share his online construction of the target ?", "1697154": "lucasmorin, can't give a full reprex easily, but mine uses this:\n\n```\nclass RunningTargets:\n def __init__(self,\n weights,\n maxlen=7*24*60\n ):\n self.weights = weights\n self.maxlen = maxlen\n self.batches_seen = 0\n \n # indexed in real time\n self.timestamp_deque = deque(list())\n \n # indexed 17 timestamps in past\n self.target_timestamp_deque = deque(list())\n \n # target features\n self.close_arr = np.empty(shape=(maxlen, 14))\n self.returns_arr = np.empty(shape=(maxlen, 14))\n self.market_arr = np.empty(shape=(maxlen,))\n self.targets_arr = np.empty(shape=(maxlen, 14))\n \n def append_close(self, timestamp, close_arr):\n self.batches_seen += 1\n self.timestamp_deque.append(timestamp)\n self.close_arr = np.append(self.close_arr, np.expand_dims(close_arr, axis=0), axis=0)\n self.close_arr = np.delete(self.close_arr, 0, axis=0)\n \n def append_returns(self):\n if self.batches_seen < 17:\n print(f\"Returns ready in {17 - self.batches_seen}...\")\n pass\n else:\n ts = self.timestamp_deque[-17]\n self.target_timestamp_deque.append(ts)\n returns_arr = (self.close_arr[-1]/self.close_arr[-16]) - 1\n self.returns_arr = np.append(self.returns_arr, np.expand_dims(returns_arr, axis=0), axis=0)\n self.returns_arr = np.delete(self.returns_arr, 0, axis=0)\n market = np.average(np.nan_to_num(returns_arr), weights=self.weights)\n self.market_arr = np.append(self.market_arr, market)\n self.market_arr = np.delete(self.market_arr, 0, axis=0)\n \n def append_targets(self, winlen=3750):\n if self.batches_seen < 17:\n pass\n else:\n if self.batches_seen < winlen:\n bet = 0\n if self.batches_seen == winlen - 1:\n print(\"Beta is now ready!\")\n else:\n num = np.mean((self.returns_arr[-winlen:].T * self.market_arr[-winlen:]), axis=1)\n den = np.mean(self.market_arr[-winlen:] ** 2, axis=0)\n bet = np.nan_to_num(num.T / den, nan=0., posinf=0., neginf=0.)\n targets = self.returns_arr[-1] - bet * self.market_arr[-1]\n self.targets_arr = np.append(self.targets_arr, np.expand_dims(targets, axis=0), axis=0)\n self.targets_arr = np.delete(self.targets_arr, 0, axis=0)\n```\n\n+ `weights` and `close_arr` are both np arrays of shape (14,), indexed by asset id.\n\nIt is warmed up for the ~ 3800 timestamps from the end of supplemental_train.csv using my `local_api` so that the Targets are accurate in time for the beginning of the private LB.", "1714038": "Thanks. I finally took the time to try it. It seems to be able to build target with nearly 0 error (1e-17) except for some very rare time ids. I'll try it in an only learning framework.", "1714436": "No good in doing so since the hosts seem to use a different target calculation method compared to targets in train data.", "1714766": "noi031 Can you clarify why you think this is the case? If the target calculation method is different then there's no point in this competition at all.", "1718630": "I found that setting beta=0 created a noisy target as it would flip between \"average log return\" and \"average relative log return\" anytime you had a single NaN in your lookback window. As the data quality gets better towards the end of the training set this happens less often, and never for some coins. \n\nFor the on-line reconstructed target I think it makes more sense to forward-fill the data (and thus never have beta=0). It has a higher error when compared to the target in the training set, but it gives you a less arbitrary signal for training. I think its a good bet that the data quality in the test set will match the end of the training set, or even be better, because the historical quality issues look like growing pains in the their tech stack." }, "source": "meta" }