Congratulations to the winners! And, thanks to the sponsors and Kaggle for a terrific contest. There was clear differentiation from the rest of the pack as well as between the winners.
\r\nI'd love to know what insights, methods worked for people, and what didn't.
", "messages": [ { "id": 10719, "postDate": "2012-05-01T13:50:09.550Z", "content": "Kaggle competitions sometimes remind me cooking competitions. It very important to find that secret ingredient (and every cook has its own secret ingredient) , but then you need to find an original way of using it(slice and dice? boil? bake? use raw?) .\r\n
\r\nOur initial progress was based on RF model with minor data preprocessing. Actually, we had RF with private score of ~0.727 submitted on Feb 29. For little while we were stuck, making only small progress by using postprocessing and creating RF ensembles. Then\r\n Bruce found what we thought was the "secret" ingredient - GBM. Now we can see that it was not so secret :). We made a good run using GBMs only , however our best submission was ensemble of RFs and GBMs.
@Vivek: I did have the same experience of overfitting RF to training data. That's why I turned to linear regression. I agree with Halla, so there may be some normalization.
\r\nI also used time weighted VWAP, but I found std not that helpful.
Sergey, Bruce - seems like you had a very well 'cooked' RF model indeed. :-) Thanks for sharing, and congratulations again.\r\n
\r\n@desertnaut Since you asked, I had a miserable time tuning random forests. I think I got misled by the same problem that Cole and others mentioned in the other thread. After a point, the RF held-out scores diverged greatly from the test set scores, making\r\n life difficult. Would anyone have any insight into why this might have been the case? Perhaps, it was due to my silly mistake of not working with log transformed returns? GBMs were better, in that, at least the held out scores matched up with the test set\r\n scores (perhaps, they were tolerant of non-transformed prices?). Additional predictors like vwap, time weighted vwap and std dev of prices were quite useful.
\r\nI spent some time working in yield space but didn't get anywhere with that. Did anyone try anything with yields? Did anyone try anything special based on the time_to_maturity of the bonds, or apply any domain-specific tricks?
", "votes": 1 }, { "id": 10717, "postDate": "2012-05-01T12:48:55.397Z", "content": "I ended up using a combination of locally weighted non-linear regression, random forests and gradient boosting. I only used variables up to curvebasedprice_last4. I don't think that the winners used any secret trick. I'm guessing they just created\r\n an ensemble of different methods which is very effective but also very time consuming.
", "votes": 1 }, { "id": 10708, "postDate": "2012-05-01T07:37:01.033Z", "content": "Grats to the winners!
\r\nI had a very simple modeling tech.
\r\nGenerated some stats from the curve_based_prices_lastx and trade_price_lastxs (sd, average, median, linear extrapolation) and from some other features like present value of a bond with a given maturity and coupon with 10% reference rates. I only modeled\r\n the trading price of a bond, where the received_time_difference_last1>300, because the trade_price_last1 was a sufficient predictor on average for those with rtdl1<300. (That may have been a mistake.)
\r\nTried some regression techniques: linear regression, PACE regression, Regression trees (bagged), neural nets, local linear regression. PACE was great overall, neural nets were good where bonds were callable (handled the the callable non callable bonds separately).\r\n I clustered the dataset with the training set and reached the highest accuracy by voteing together 69 different models (took the median of the distinct models, it was better than the average :))
\r\nLoved this competition! Looking forward to finance related comps!
", "votes": 1 }, { "id": 10680, "postDate": "2012-05-01T01:04:19.333Z", "content": "Congratulations to the winners! And, thanks to the sponsors and Kaggle for a terrific contest. There was clear differentiation from the rest of the pack as well as between the winners.
\r\nI'd love to know what insights, methods worked for people, and what didn't.
", "votes": 1 }, { "id": 10744, "postDate": "2012-05-02T17:18:23.780Z", "content": "Vivek Sharma wrote:
\r\n@desertnaut Since you asked, I had a miserable time tuning random forests. I think I got misled by the same problem that Cole and others mentioned in the other thread. After a point, the RF held-out scores diverged greatly from the test set scores, making\r\n life difficult. Would anyone have any insight into why this might have been the case?
\r\nI had the same problem and asked in a different thread. Turns out that the test and train sets were made using different bonds, so the proper witholding loop is to withold all the trades from each of a select number of bonds. If you were witholding a random\r\n set of trades, the other trades from the same bonds would allow your model more info than it has on the test set.
", "votes": 2 }, { "id": 10733, "postDate": "2012-05-02T03:48:38.400Z", "content": "[quote=Vivek Sharma;10726]
\r\n@desertnaut Since you asked, I had a miserable time tuning random forests. I think I got misled by the same problem that Cole and others mentioned in the other thread. After a point, the RF held-out scores diverged greatly from the test set scores, making\r\n life difficult. Would anyone have any insight into why this might have been the case?
\r\n[/quote]
\r\nI ran into the same problem towards the end of the contest. At least in my case, there is a simple explanation. After getting the score on the held-out set, I went back and tweaked the parameters to make the score better. Essentially, I was overfitting to\r\n the held-out set. As the test set had completely different bonds, clearly the score on the test set had to be worse with this overfitted model.
\r\nHad I cross validated using the test set, tweaked the parameters to make the test score better and then tried the model on the held-out set, I would have gotten a worse score on the held-out set. Haven't actually tried this out, but one would expect this\r\n to be true in general.
", "votes": 2 }, { "id": 10728, "postDate": "2012-05-01T18:53:40.017Z", "content": "Halla, interesting point. It seems that training RFs by using sampled subsets (every 12th row) would eliminate this problem. I wonder if you and others who did well with RFs, did such sampling.
" }, { "id": 10722, "postDate": "2012-05-01T16:15:33.377Z", "content": "[quote=Neil Thomas;10702]
\r\nI don't see how you would be using data from the future. Each trade in a row still occurs before the response that you have to predict. But I would agree that the host probably did not intend the data to be used this way, if that is what you meant.
\r\n[/quote]
\r\nThanks.
\r\nA piece of my thinking is: at least two drawbacks of allowing use of test data.
\r\n1) the training and test are different bonds. If use test data to train, the model may fit to test bonds.
\r\n2) backtesting usually use some period of data for training and another period for test.
\r\nIn contrast, in this contest, assuming there are [t:t+9] for predicting t+10, [t+1:t+10] for predicting t+11.
\r\nThen by shifting, you have [t:t+8] for predicting t+9, [t+1:t+9] for predicting t+10. There exists overlapping between training and test.
\r\nI understand that this has been avoided since the modification of data.
\r\nBut there may exist [t:t+9]->t+10 (a), [t+6:t+15]->t+16, and then [t:t+8]->t+9, [t+6:t+14]->t+15 (b) for training.
\r\nNote that (b) is future data of (a). It cannot be used for real backtesting.
\r\nWelcome any correction.
" }, { "id": 10720, "postDate": "2012-05-01T14:47:52.560Z", "content": "Well said, Sergey. Congratulations to all the competitors and to Kaggle and Benchmark for an extremely well-run and well-formulated competition. Oh, and by the way, if any of you are lucky enough to get the chance to work with Sergey, take it! I've learned\r\n a lot and my scores improved considerably as a result of our collaboration.
" }, { "id": 10718, "postDate": "2012-05-01T12:57:44.907Z", "content": "Hi all, and congrats to the winners!
\r\nWell, it was a great experience, and our good final ranking came rather as a surprize: all three members of Toms' Friends we have started studying data mining only since last October, without any relevant prior experience whatsoever. None of us had any experience\r\n with R either (last time I had coded was back in 2000, and it was in Matlab). Despite of all these, we sat down and did some serious brainstorming, followed by extensive experimentation using R (no question we ended up with 100 submissions!).
\r\nWe used random forest for a crude initial forward feature selection, and when we (thought we had) found our feature set we proceeded to modelling with gradient boosting. This gave very competitive results early on, so we proceeded with detailed parameter\r\n tuning and averaging some of our best models' outputs. Pretty much that was all - no clustering, no test set usage, no outlier detection, only some handling of the missing values. We tried different approaches to variables modelling (averages and differentials),\r\n but it proved that nothing could surpass the non-transformed variables input. We tried to remove from the training set some price value ranges that were not present in the test set, but again this gave inferior results...
\r\nWe felt sorry to see Wayne (Zhang) not entering the final top ten, but hey Wayne, no worries, the future lies ahead...
\r\nVivek, any insight from your part??
\r\nivo,
\r\nDuring these last weeks of the contest, we felt that you and us we were running almost side by side... For us, rookies here, it was a hell of a race, and we would like to thank you for this (we would have sent you a hello message, if this was possible through\r\n kaggle...). Sincerely hope to see you around - and BTW, you have a HELL of a profile photo!! :-)
\r\n\r\n
Many thanx to my team mates, PepFriday & tinariwen. This was our first time here, but we are just starting, and hopefully we will stick around...
\r\n\r\n
des
" }, { "id": 10709, "postDate": "2012-05-01T07:50:13.180Z", "content": "Hey Neil,
\r\nI had been scalded by the other trading contest so I actually started with the shifted test--didn't get far actually but I gave it up before puttin my entire alg together. I made a few early subs with shifttest or shifttest+train, to no avail, then focused\r\n on train; in the spirit of the random forest I tried splitting the train set and optimize different models based on trade type, previous trade type for most trades, and cluster (based on variables); that meant having a lot of data to avoid bad conditioning,\r\n so I focused on the test set. Then I got mired in my own bugs and the fact that the metrics and param optimization, being done at the trade level, wasn't reflecting the real structure of the test set selection (algorithmic trading challenge, on liquidity shock\r\n recovery, where the test set was way different from the trading set), but that's another story. By the way, I still see unexplanaible differences, now between the 'public' and 'private' scores. Waiting for the silt to clear up on that one ;-)
" }, { "id": 10707, "postDate": "2012-05-01T07:25:17.383Z", "content": "[quote=Neil Thomas;10699]
\r\nJust shift the trades to the left by one, delete the last trade and bingo, you have a new training set (tradepricelast1 in the test set is now the response variable). ... Did anyone else try this?
\r\n[/quote]
\r\nI tried. I merely had time to implement a base model. When I used it to predict the most recent trade price using the previous nine trades in each row of this new training set, the errors were about 0.543±0.001 (with one third of the the test set held out\r\n at random). Unfortunately, the leaderboard score ended up with a pathetic 1.03. In hindsight, this is reasonable because the base model --- some variant of linear regression --- is not weighted, so that data weights are not taken into account in the regression\r\n process.
" }, { "id": 10703, "postDate": "2012-05-01T05:47:44.190Z", "content": "[quote=Wayne Zhang;10701]
\r\nI noticed that there're large errors for some curve prices. Also large differences between prediction targets and the last prices.
\r\nI have no idea for correcting such outliers, because I used linear regression, and most data are of small errors, which means the regression model fited to the small error samples.
\r\nNeil, do you have any good idea for dealing with these outliers?
\r\n[/quote]
\r\nFor the predictor variables, I clipped the values so that they fall within a few standard deviations from the mean. But I don't know how one would handle outliers in the responses.
" }, { "id": 10702, "postDate": "2012-05-01T05:43:22.247Z", "content": "[quote=Wayne Zhang;10700]
\r\nI did not use test data to construct a new training set, because I think it is impossible in reality.
\r\nYou cannot use future prices to train a model for today's prices.
[/quote]
\r\nI don't see how you would be using data from the future. Each trade in a row still occurs before the response that you have to predict. But I would agree that the host probably did not intend the data to be used this way, if that is what you meant.
" }, { "id": 10701, "postDate": "2012-05-01T05:31:11.873Z", "content": "[quote=Neil Thomas;10699]
\r\nMy approach was to predict the error in the curve based price and then use the predicted error to compute the trade prices. The error in previous curve based prices make excellent predictors for this. In a last minute attempt, I modeled the difference between\r\n the trade price and the previous trade price. The results weren't as good as that of the first approach, but were different enough to contribute to the overall blend and let me break into the top ten.
\r\n[/quote]
\r\nI noticed that there're large errors for some curve prices. Also large differences between prediction targets and the last prices.
\r\nI have no idea for correcting such outliers, because I used linear regression, and most data are of small errors, which means the regression model fited to the small error samples.
\r\nNeil, do you have any good idea for dealing with these outliers?
" }, { "id": 10700, "postDate": "2012-05-01T05:27:40.947Z", "content": "I did not use test data to construct a new training set, because I think it is impossible in reality.
\r\nYou cannot use future prices to train a model for today's prices.
I kept looking for that bottle of secret sauce, but never found it. Looking at the final scores, I am even more convinced that there is a short cut somewhere that most of us entirely missed.
\r\nMaybe others noticed this too: it is possible to use the test data for training. Just shift the trades to the left by one, delete the last trade and bingo, you have a new training set (tradepricelast1 in the test set is now the response variable).\r\n Looks like this is fair game as the rules don't restrict the usage of test data. I tried to take advantage of this, but couldn't work it into the blend in a meaningful way. Did anyone else try this? As the test data has a different set of bonds, one would\r\n think that there is some edge to be gained there. The test set contains information that the training set just cannot provide.
\r\nMy approach was to predict the error in the curve based price and then use the predicted error to compute the trade prices. The error in previous curve based prices make excellent predictors for this. In a last minute attempt, I modeled the difference between\r\n the trade price and the previous trade price. The results weren't as good as that of the first approach, but were different enough to contribute to the overall blend and let me break into the top ten.
" }, { "id": 10681, "postDate": "2012-05-01T01:21:29.890Z", "content": "Congratulations!
\r\nI used linear regression, with 2nd order cross-form and log.
" }, { "id": 10727, "postDate": "2012-05-01T18:42:30.460Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 10686, "postDate": "2012-05-01T02:07:09.250Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 10719, "author_name": "Sergey Yurgenson", "author_url": "", "post_date": "2012-05-01T13:50:09.550000", "content": "Kaggle competitions sometimes remind me cooking competitions. It very important to find that secret ingredient (and every cook has its own secret ingredient) , but then you need to find an original way of using it(slice and dice? boil? bake? use raw?) .\r\n
\r\nOur initial progress was based on RF model with minor data preprocessing. Actually, we had RF with private score of ~0.727 submitted on Feb 29. For little while we were stuck, making only small progress by using postprocessing and creating RF ensembles. Then\r\n Bruce found what we thought was the "secret" ingredient - GBM. Now we can see that it was not so secret :). We made a good run using GBMs only , however our best submission was ensemble of RFs and GBMs.
@Vivek: I did have the same experience of overfitting RF to training data. That's why I turned to linear regression. I agree with Halla, so there may be some normalization.
\r\nI also used time weighted VWAP, but I found std not that helpful.
Sergey, Bruce - seems like you had a very well 'cooked' RF model indeed. :-) Thanks for sharing, and congratulations again.\r\n
\r\n@desertnaut Since you asked, I had a miserable time tuning random forests. I think I got misled by the same problem that Cole and others mentioned in the other thread. After a point, the RF held-out scores diverged greatly from the test set scores, making\r\n life difficult. Would anyone have any insight into why this might have been the case? Perhaps, it was due to my silly mistake of not working with log transformed returns? GBMs were better, in that, at least the held out scores matched up with the test set\r\n scores (perhaps, they were tolerant of non-transformed prices?). Additional predictors like vwap, time weighted vwap and std dev of prices were quite useful.
\r\nI spent some time working in yield space but didn't get anywhere with that. Did anyone try anything with yields? Did anyone try anything special based on the time_to_maturity of the bonds, or apply any domain-specific tricks?
", "votes": 1, "replies": [] }, { "id": 10717, "author_name": "Glen Koundry", "author_url": "", "post_date": "2012-05-01T12:48:55.397000", "content": "I ended up using a combination of locally weighted non-linear regression, random forests and gradient boosting. I only used variables up to curvebasedprice_last4. I don't think that the winners used any secret trick. I'm guessing they just created\r\n an ensemble of different methods which is very effective but also very time consuming.
", "votes": 1, "replies": [] }, { "id": 10708, "author_name": "ivo", "author_url": "", "post_date": "2012-05-01T07:37:01.033000", "content": "Grats to the winners!
\r\nI had a very simple modeling tech.
\r\nGenerated some stats from the curve_based_prices_lastx and trade_price_lastxs (sd, average, median, linear extrapolation) and from some other features like present value of a bond with a given maturity and coupon with 10% reference rates. I only modeled\r\n the trading price of a bond, where the received_time_difference_last1>300, because the trade_price_last1 was a sufficient predictor on average for those with rtdl1<300. (That may have been a mistake.)
\r\nTried some regression techniques: linear regression, PACE regression, Regression trees (bagged), neural nets, local linear regression. PACE was great overall, neural nets were good where bonds were callable (handled the the callable non callable bonds separately).\r\n I clustered the dataset with the training set and reached the highest accuracy by voteing together 69 different models (took the median of the distinct models, it was better than the average :))
\r\nLoved this competition! Looking forward to finance related comps!
", "votes": 1, "replies": [] }, { "id": 10744, "author_name": "teaserebotier", "author_url": "", "post_date": "2012-05-02T17:18:23.780000", "content": "Vivek Sharma wrote:
\r\n@desertnaut Since you asked, I had a miserable time tuning random forests. I think I got misled by the same problem that Cole and others mentioned in the other thread. After a point, the RF held-out scores diverged greatly from the test set scores, making\r\n life difficult. Would anyone have any insight into why this might have been the case?
\r\nI had the same problem and asked in a different thread. Turns out that the test and train sets were made using different bonds, so the proper witholding loop is to withold all the trades from each of a select number of bonds. If you were witholding a random\r\n set of trades, the other trades from the same bonds would allow your model more info than it has on the test set.
", "votes": 2, "replies": [] }, { "id": 10733, "author_name": "Anil Thomas", "author_url": "", "post_date": "2012-05-02T03:48:38.400000", "content": "[quote=Vivek Sharma;10726]
\r\n@desertnaut Since you asked, I had a miserable time tuning random forests. I think I got misled by the same problem that Cole and others mentioned in the other thread. After a point, the RF held-out scores diverged greatly from the test set scores, making\r\n life difficult. Would anyone have any insight into why this might have been the case?
\r\n[/quote]
\r\nI ran into the same problem towards the end of the contest. At least in my case, there is a simple explanation. After getting the score on the held-out set, I went back and tweaked the parameters to make the score better. Essentially, I was overfitting to\r\n the held-out set. As the test set had completely different bonds, clearly the score on the test set had to be worse with this overfitted model.
\r\nHad I cross validated using the test set, tweaked the parameters to make the test score better and then tried the model on the held-out set, I would have gotten a worse score on the held-out set. Haven't actually tried this out, but one would expect this\r\n to be true in general.
", "votes": 2, "replies": [] }, { "id": 10728, "author_name": "Vivek Sharma", "author_url": "", "post_date": "2012-05-01T18:53:40.017000", "content": "Halla, interesting point. It seems that training RFs by using sampled subsets (every 12th row) would eliminate this problem. I wonder if you and others who did well with RFs, did such sampling.
", "votes": 0, "replies": [] }, { "id": 10722, "author_name": "Wayne Zhang", "author_url": "", "post_date": "2012-05-01T16:15:33.377000", "content": "[quote=Neil Thomas;10702]
\r\nI don't see how you would be using data from the future. Each trade in a row still occurs before the response that you have to predict. But I would agree that the host probably did not intend the data to be used this way, if that is what you meant.
\r\n[/quote]
\r\nThanks.
\r\nA piece of my thinking is: at least two drawbacks of allowing use of test data.
\r\n1) the training and test are different bonds. If use test data to train, the model may fit to test bonds.
\r\n2) backtesting usually use some period of data for training and another period for test.
\r\nIn contrast, in this contest, assuming there are [t:t+9] for predicting t+10, [t+1:t+10] for predicting t+11.
\r\nThen by shifting, you have [t:t+8] for predicting t+9, [t+1:t+9] for predicting t+10. There exists overlapping between training and test.
\r\nI understand that this has been avoided since the modification of data.
\r\nBut there may exist [t:t+9]->t+10 (a), [t+6:t+15]->t+16, and then [t:t+8]->t+9, [t+6:t+14]->t+15 (b) for training.
\r\nNote that (b) is future data of (a). It cannot be used for real backtesting.
\r\nWelcome any correction.
", "votes": 0, "replies": [] }, { "id": 10720, "author_name": "Bruce Cragin", "author_url": "", "post_date": "2012-05-01T14:47:52.560000", "content": "Well said, Sergey. Congratulations to all the competitors and to Kaggle and Benchmark for an extremely well-run and well-formulated competition. Oh, and by the way, if any of you are lucky enough to get the chance to work with Sergey, take it! I've learned\r\n a lot and my scores improved considerably as a result of our collaboration.
", "votes": 0, "replies": [] }, { "id": 10718, "author_name": "desertnaut", "author_url": "", "post_date": "2012-05-01T12:57:44.907000", "content": "Hi all, and congrats to the winners!
\r\nWell, it was a great experience, and our good final ranking came rather as a surprize: all three members of Toms' Friends we have started studying data mining only since last October, without any relevant prior experience whatsoever. None of us had any experience\r\n with R either (last time I had coded was back in 2000, and it was in Matlab). Despite of all these, we sat down and did some serious brainstorming, followed by extensive experimentation using R (no question we ended up with 100 submissions!).
\r\nWe used random forest for a crude initial forward feature selection, and when we (thought we had) found our feature set we proceeded to modelling with gradient boosting. This gave very competitive results early on, so we proceeded with detailed parameter\r\n tuning and averaging some of our best models' outputs. Pretty much that was all - no clustering, no test set usage, no outlier detection, only some handling of the missing values. We tried different approaches to variables modelling (averages and differentials),\r\n but it proved that nothing could surpass the non-transformed variables input. We tried to remove from the training set some price value ranges that were not present in the test set, but again this gave inferior results...
\r\nWe felt sorry to see Wayne (Zhang) not entering the final top ten, but hey Wayne, no worries, the future lies ahead...
\r\nVivek, any insight from your part??
\r\nivo,
\r\nDuring these last weeks of the contest, we felt that you and us we were running almost side by side... For us, rookies here, it was a hell of a race, and we would like to thank you for this (we would have sent you a hello message, if this was possible through\r\n kaggle...). Sincerely hope to see you around - and BTW, you have a HELL of a profile photo!! :-)
\r\n\r\n
Many thanx to my team mates, PepFriday & tinariwen. This was our first time here, but we are just starting, and hopefully we will stick around...
\r\n\r\n
des
", "votes": 0, "replies": [] }, { "id": 10709, "author_name": "teaserebotier", "author_url": "", "post_date": "2012-05-01T07:50:13.180000", "content": "Hey Neil,
\r\nI had been scalded by the other trading contest so I actually started with the shifted test--didn't get far actually but I gave it up before puttin my entire alg together. I made a few early subs with shifttest or shifttest+train, to no avail, then focused\r\n on train; in the spirit of the random forest I tried splitting the train set and optimize different models based on trade type, previous trade type for most trades, and cluster (based on variables); that meant having a lot of data to avoid bad conditioning,\r\n so I focused on the test set. Then I got mired in my own bugs and the fact that the metrics and param optimization, being done at the trade level, wasn't reflecting the real structure of the test set selection (algorithmic trading challenge, on liquidity shock\r\n recovery, where the test set was way different from the trading set), but that's another story. By the way, I still see unexplanaible differences, now between the 'public' and 'private' scores. Waiting for the silt to clear up on that one ;-)
", "votes": 0, "replies": [] }, { "id": 10707, "author_name": "The suffocated", "author_url": "", "post_date": "2012-05-01T07:25:17.383000", "content": "[quote=Neil Thomas;10699]
\r\nJust shift the trades to the left by one, delete the last trade and bingo, you have a new training set (tradepricelast1 in the test set is now the response variable). ... Did anyone else try this?
\r\n[/quote]
\r\nI tried. I merely had time to implement a base model. When I used it to predict the most recent trade price using the previous nine trades in each row of this new training set, the errors were about 0.543±0.001 (with one third of the the test set held out\r\n at random). Unfortunately, the leaderboard score ended up with a pathetic 1.03. In hindsight, this is reasonable because the base model --- some variant of linear regression --- is not weighted, so that data weights are not taken into account in the regression\r\n process.
", "votes": 0, "replies": [] }, { "id": 10703, "author_name": "Anil Thomas", "author_url": "", "post_date": "2012-05-01T05:47:44.190000", "content": "[quote=Wayne Zhang;10701]
\r\nI noticed that there're large errors for some curve prices. Also large differences between prediction targets and the last prices.
\r\nI have no idea for correcting such outliers, because I used linear regression, and most data are of small errors, which means the regression model fited to the small error samples.
\r\nNeil, do you have any good idea for dealing with these outliers?
\r\n[/quote]
\r\nFor the predictor variables, I clipped the values so that they fall within a few standard deviations from the mean. But I don't know how one would handle outliers in the responses.
", "votes": 0, "replies": [] }, { "id": 10702, "author_name": "Anil Thomas", "author_url": "", "post_date": "2012-05-01T05:43:22.247000", "content": "[quote=Wayne Zhang;10700]
\r\nI did not use test data to construct a new training set, because I think it is impossible in reality.
\r\nYou cannot use future prices to train a model for today's prices.
[/quote]
\r\nI don't see how you would be using data from the future. Each trade in a row still occurs before the response that you have to predict. But I would agree that the host probably did not intend the data to be used this way, if that is what you meant.
", "votes": 0, "replies": [] }, { "id": 10701, "author_name": "Wayne Zhang", "author_url": "", "post_date": "2012-05-01T05:31:11.873000", "content": "[quote=Neil Thomas;10699]
\r\nMy approach was to predict the error in the curve based price and then use the predicted error to compute the trade prices. The error in previous curve based prices make excellent predictors for this. In a last minute attempt, I modeled the difference between\r\n the trade price and the previous trade price. The results weren't as good as that of the first approach, but were different enough to contribute to the overall blend and let me break into the top ten.
\r\n[/quote]
\r\nI noticed that there're large errors for some curve prices. Also large differences between prediction targets and the last prices.
\r\nI have no idea for correcting such outliers, because I used linear regression, and most data are of small errors, which means the regression model fited to the small error samples.
\r\nNeil, do you have any good idea for dealing with these outliers?
", "votes": 0, "replies": [] }, { "id": 10700, "author_name": "Wayne Zhang", "author_url": "", "post_date": "2012-05-01T05:27:40.947000", "content": "I did not use test data to construct a new training set, because I think it is impossible in reality.
\r\nYou cannot use future prices to train a model for today's prices.
I kept looking for that bottle of secret sauce, but never found it. Looking at the final scores, I am even more convinced that there is a short cut somewhere that most of us entirely missed.
\r\nMaybe others noticed this too: it is possible to use the test data for training. Just shift the trades to the left by one, delete the last trade and bingo, you have a new training set (tradepricelast1 in the test set is now the response variable).\r\n Looks like this is fair game as the rules don't restrict the usage of test data. I tried to take advantage of this, but couldn't work it into the blend in a meaningful way. Did anyone else try this? As the test data has a different set of bonds, one would\r\n think that there is some edge to be gained there. The test set contains information that the training set just cannot provide.
\r\nMy approach was to predict the error in the curve based price and then use the predicted error to compute the trade prices. The error in previous curve based prices make excellent predictors for this. In a last minute attempt, I modeled the difference between\r\n the trade price and the previous trade price. The results weren't as good as that of the first approach, but were different enough to contribute to the overall blend and let me break into the top ten.
", "votes": 0, "replies": [] }, { "id": 10681, "author_name": "Wayne Zhang", "author_url": "", "post_date": "2012-05-01T01:21:29.890000", "content": "Congratulations!
\r\nI used linear regression, with 2nd order cross-form and log.
", "votes": 0, "replies": [] }, { "id": 10727, "author_name": "", "author_url": "", "post_date": "2012-05-01T18:42:30.460000", "content": "", "votes": 2, "replies": [] }, { "id": 10686, "author_name": "", "author_url": "", "post_date": "2012-05-01T02:07:09.250000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "10719": "", "10730": "", "10726": "", "10717": "", "10708": "", "10680": "", "10744": "", "10733": "", "10728": "", "10722": "", "10720": "", "10718": "", "10709": "", "10707": "", "10703": "", "10702": "", "10701": "", "10700": "", "10699": "", "10681": "", "10727": "", "10686": "" } }