Hi, All
\r\nweight: The weight of the row for evaluation purposes. This is calculated as the square root of the time since the last trade and then scaled so the mean is 1.
\r\nThat means weight data should NOT be in algo that predicts the data or be part of parameters of models we build ?
\r\nWe shouldn't use Weight in any way ,right ?
", "messages": [ { "id": "8311", "postDate": "01/28/2012 17:03:56", "content": "Hi, All
\r\nweight: The weight of the row for evaluation purposes. This is calculated as the square root of the time since the last trade and then scaled so the mean is 1.
\r\nThat means weight data should NOT be in algo that predicts the data or be part of parameters of models we build ?
\r\nWe shouldn't use Weight in any way ,right ?
", "rawMarkdown": "", "votes": null }, { "id": "8314", "postDate": "01/28/2012 17:59:06", "content": "Oddly enough the "weight" column is included as a feature in the random forest benchmark. I was wondering the same thing - why?
", "rawMarkdown": "", "votes": null }, { "id": "8317", "postDate": "01/28/2012 19:25:17", "content": "Let me take a shot at explaining this: In applying any given model, one would always know both the current time of day and the time of the last trade, so delta t from the last trade is always known. We as competitors may not know the actual value or values\r\n of the normalizing factor used in calculating the weight variable, but the challenge sponsor surely does. Any model dependence on the weight column is thus essentially just a proxy for sqrt(delta t), which is always known (and is a very reasonable model input).
\r\n[quote=Momchil Georgiev;8314]
\r\nOddly enough the "weight" column is included as a feature in the random forest benchmark. I was wondering the same thing - why?
\r\n[/quote]
\r\nNo particular reason. Naturally the RF benchmark isn't a very sophisticated benchmark - it was provided simply for an example of reading the data in, training a basic model, computing predictions, and saving those to an example submission.
", "rawMarkdown": "", "votes": null }, { "id": "8326", "postDate": "01/28/2012 23:03:49", "content": "Bruce is complete correct. Weight is actually a constant * sqrt(receivedtimediff1+1) so that even contemporaneous trades have some weight. You can figure out the constant rather easily, but its just there to normalize weight so the average weight\r\n is 1 and should hold no meaning for the competition.
", "rawMarkdown": "", "votes": null }, { "id": "9711", "postDate": "03/24/2012 00:38:26", "content": "OK I am trying to understand and still don't. The relationship Dan pointed out holds: constant * sqrt(receivedtimediff1+1). But receivedtimediff1 is the time between the trade last1 and the trade last2, it's not between current and last1. Sure, illiquid\r\n bonds trade less overall so "receivedtimediff0" and receivedtimediff1 must correlate pretty well, but still, shouldn't we be scored with a weighting based on receivedtimediff0?
", "rawMarkdown": "", "votes": null }, { "id": "9715", "postDate": "03/24/2012 01:18:53", "content": "The definition of received_time_diff_last1, as supplied on the Data tab of the challenge main page, seems pretty clear. Are you sure you didn't just misread it?
", "rawMarkdown": "", "votes": null }, { "id": "9727", "postDate": "03/24/2012 21:44:31", "content": "teaserebotier, the definition is "receivedtimedifflast{1-10}: The time difference between the trade and that of the previous {1-10}."\r\n
\r\n
I guess "previous" is ambiguous. It can refer to the time dimension or the dimension (left-to-right) along the array. From a trader's perspective, "previous trade" is the trade happening just before (time dimension)...
", "rawMarkdown": "", "votes": null }, { "id": "10280", "postDate": "04/22/2012 22:22:03", "content": "I guess "previous" is ambiguous. It can refer to the time dimension or the dimension (left-to-right) along the array. From a trader's perspective, "previous trade" is the trade happening just before (time dimension)...
\r\n>> received_time_diff_last{1-10}: The time difference between the trade and that of the previous {1-10}.
\r\nThat did read to me like received_time_diff_last1 is the time difference between the trade (last1) and the previous one (last2),\r\n but i guess for an array-oriented person ;-) its the difference between that trade and the one that chronogically... follows :)
\r\n", "rawMarkdown": "", "votes": null }, { "id": "10285", "postDate": "04/23/2012 00:20:33", "content": "
Hi Comp Admin,
\r\nwhy weight the fit in this way, ie why make the weight a function of time difference. This gives more weight to trades which are well separated in time and I see no reason for that.\r\n
", "rawMarkdown": "", "votes": null }, { "id": "10289", "postDate": "04/23/2012 01:20:58", "content": "[quote=teaserebotier;10280]
\r\nI guess "previous" is ambiguous...
\r\n[/quote]
\r\nMaybe the term "the trade" is what's throwing you off. As I understand it it, each record provides data pertaining to a single trade, which occurs at "the trade_price". The ten quantities trade_price_last_1 through trade_price_last10 thus correspond to trades\r\n that occurred at earlier times than "the trade". That's whay received_time_diff_last2 is always greater than received_time_diff_last1, for example. Received_time_diff2 is larger because the trade_price_last2 prior trade occurred a longer time in the past than\r\n the trade_price_last1 trade.
", "rawMarkdown": "", "votes": null }, { "id": "10293", "postDate": "04/23/2012 07:28:39", "content": "Ahh indeed, I didnt look at the data well enough... times are cumulative from trade0 to trade10. And I thought high of my English, cough, cough... Still, for those of us foreigners, a formula is clearer than a thousand words!
", "rawMarkdown": "", "votes": null }, { "id": "10300", "postDate": "04/23/2012 13:39:05", "content": "All of the ____lastN values are the values of the trade N before the current trade in the row. However, the received_time_diffN is the difference in time between the current trade and the trade N previously.
\r\nAs for why we chose this metric, its a combination of many things. I'll provide 2:
\r\n1. HIghly traded bonds dominate the trade examples (for obvous reasons). We don't want these bonds to dominate the results since we are interested in all bonds.
\r\n2. Trade prices after a long period of inactivity have higher variance and thus represent more knowledge. We want to reward this.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 8314, "author_name": "sirguessalot", "author_url": "", "post_date": "01/28/2012 17:59:06", "content": "Oddly enough the "weight" column is included as a feature in the random forest benchmark. I was wondering the same thing - why?
", "votes": null, "replies": [] }, { "id": 8317, "author_name": "bcragin", "author_url": "", "post_date": "01/28/2012 19:25:17", "content": "Let me take a shot at explaining this: In applying any given model, one would always know both the current time of day and the time of the last trade, so delta t from the last trade is always known. We as competitors may not know the actual value or values\r\n of the normalizing factor used in calculating the weight variable, but the challenge sponsor surely does. Any model dependence on the weight column is thus essentially just a proxy for sqrt(delta t), which is always known (and is a very reasonable model input).
\r\n[quote=Momchil Georgiev;8314]
\r\nOddly enough the "weight" column is included as a feature in the random forest benchmark. I was wondering the same thing - why?
\r\n[/quote]
\r\nNo particular reason. Naturally the RF benchmark isn't a very sophisticated benchmark - it was provided simply for an example of reading the data in, training a basic model, computing predictions, and saving those to an example submission.
", "votes": null, "replies": [] }, { "id": 8326, "author_name": "danglaser", "author_url": "", "post_date": "01/28/2012 23:03:49", "content": "Bruce is complete correct. Weight is actually a constant * sqrt(receivedtimediff1+1) so that even contemporaneous trades have some weight. You can figure out the constant rather easily, but its just there to normalize weight so the average weight\r\n is 1 and should hold no meaning for the competition.
", "votes": null, "replies": [] }, { "id": 9711, "author_name": "teaserebotier", "author_url": "", "post_date": "03/24/2012 00:38:26", "content": "OK I am trying to understand and still don't. The relationship Dan pointed out holds: constant * sqrt(receivedtimediff1+1). But receivedtimediff1 is the time between the trade last1 and the trade last2, it's not between current and last1. Sure, illiquid\r\n bonds trade less overall so "receivedtimediff0" and receivedtimediff1 must correlate pretty well, but still, shouldn't we be scored with a weighting based on receivedtimediff0?
", "votes": null, "replies": [] }, { "id": 9715, "author_name": "bcragin", "author_url": "", "post_date": "03/24/2012 01:18:53", "content": "The definition of received_time_diff_last1, as supplied on the Data tab of the challenge main page, seems pretty clear. Are you sure you didn't just misread it?
", "votes": null, "replies": [] }, { "id": 9727, "author_name": "yogurt", "author_url": "", "post_date": "03/24/2012 21:44:31", "content": "teaserebotier, the definition is "receivedtimedifflast{1-10}: The time difference between the trade and that of the previous {1-10}."\r\n
\r\n
I guess "previous" is ambiguous. It can refer to the time dimension or the dimension (left-to-right) along the array. From a trader's perspective, "previous trade" is the trade happening just before (time dimension)...
", "votes": null, "replies": [] }, { "id": 10280, "author_name": "teaserebotier", "author_url": "", "post_date": "04/22/2012 22:22:03", "content": "I guess "previous" is ambiguous. It can refer to the time dimension or the dimension (left-to-right) along the array. From a trader's perspective, "previous trade" is the trade happening just before (time dimension)...
\r\n>> received_time_diff_last{1-10}: The time difference between the trade and that of the previous {1-10}.
\r\nThat did read to me like received_time_diff_last1 is the time difference between the trade (last1) and the previous one (last2),\r\n but i guess for an array-oriented person ;-) its the difference between that trade and the one that chronogically... follows :)
\r\n", "votes": null, "replies": [] }, { "id": 10285, "author_name": "stellar", "author_url": "", "post_date": "04/23/2012 00:20:33", "content": "
Hi Comp Admin,
\r\nwhy weight the fit in this way, ie why make the weight a function of time difference. This gives more weight to trades which are well separated in time and I see no reason for that.\r\n
", "votes": null, "replies": [] }, { "id": 10289, "author_name": "bcragin", "author_url": "", "post_date": "04/23/2012 01:20:58", "content": "[quote=teaserebotier;10280]
\r\nI guess "previous" is ambiguous...
\r\n[/quote]
\r\nMaybe the term "the trade" is what's throwing you off. As I understand it it, each record provides data pertaining to a single trade, which occurs at "the trade_price". The ten quantities trade_price_last_1 through trade_price_last10 thus correspond to trades\r\n that occurred at earlier times than "the trade". That's whay received_time_diff_last2 is always greater than received_time_diff_last1, for example. Received_time_diff2 is larger because the trade_price_last2 prior trade occurred a longer time in the past than\r\n the trade_price_last1 trade.
", "votes": null, "replies": [] }, { "id": 10293, "author_name": "teaserebotier", "author_url": "", "post_date": "04/23/2012 07:28:39", "content": "Ahh indeed, I didnt look at the data well enough... times are cumulative from trade0 to trade10. And I thought high of my English, cough, cough... Still, for those of us foreigners, a formula is clearer than a thousand words!
", "votes": null, "replies": [] }, { "id": 10300, "author_name": "danglaser", "author_url": "", "post_date": "04/23/2012 13:39:05", "content": "All of the ____lastN values are the values of the trade N before the current trade in the row. However, the received_time_diffN is the difference in time between the current trade and the trade N previously.
\r\nAs for why we chose this metric, its a combination of many things. I'll provide 2:
\r\n1. HIghly traded bonds dominate the trade examples (for obvous reasons). We don't want these bonds to dominate the results since we are interested in all bonds.
\r\n2. Trade prices after a long period of inactivity have higher variance and thus represent more knowledge. We want to reward this.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "8311": "", "8314": "", "8317": "", "8318": "", "8326": "", "9711": "", "9715": "", "9727": "", "10279": "", "10280": "", "10285": "", "10289": "", "10293": "", "10300": "" }, "source": "meta" }