Congratulations to the top winners! It has been a fun competition. Question for kaggle (this is my first competition, so bear with me if it has been answered before). Would Kaggle publish the top winning algo/code ? And it would be nice if there is a way\r\n for other competitors willing to share their algo/code with other interested participants.The final swing in the results was interesting! I didn't expect that. May be you should hold another betting competition for each of these to predict the winner :)
", "messages": [ { "id": "7771", "postDate": "01/09/2012 00:24:28", "content": "Congratulations to the top winners! It has been a fun competition. Question for kaggle (this is my first competition, so bear with me if it has been answered before). Would Kaggle publish the top winning algo/code ? And it would be nice if there is a way\r\n for other competitors willing to share their algo/code with other interested participants.The final swing in the results was interesting! I didn't expect that. May be you should hold another betting competition for each of these to predict the winner :)
", "rawMarkdown": "", "votes": null }, { "id": "7772", "postDate": "01/09/2012 01:13:57", "content": "Congratulations to Ildenfons with convincing first place and other top teams with good results.\r\n
\r\n@karmic - actually, change in teams position after final scoring was not as big as in some other competitions (it was known for a while that model by Xiaoshi Lu was overfitted to public test set).\r\n
\r\nI expect some details of top models to be known soon. I can say that simple correctly executed linear regression model could put you in the top ten. (However it was not our final model)
", "rawMarkdown": "", "votes": null }, { "id": "7774", "postDate": "01/09/2012 01:57:06", "content": "Congratulations to all top teams as well. I can vouch for Sergey's assertion that linear regression can place in the top 10, however I'm not sure what 'correctly executed' means wrt this data.
\r\nI am very interested to learn the extent to which others addressed the obvious differences in the test and training sets. In particular, it appears there was a regime shift after day 2 in the training data, with prices becoming much 'noisier' (I can quantify\r\n if interested). Scores for models from subsets of day 1 and day 2 data statistically similar to the testing set are close to the testing scores.
\r\nI am also interested if others developed distinct models for shocks near the opening (I had separate models for t<60 & t>60).
", "rawMarkdown": "", "votes": null }, { "id": "7776", "postDate": "01/09/2012 02:59:03", "content": "Cole/Sergey - I am definitely interested in knowing more on the details of linear reg. model. I must have missed the forest looking for tree. I know Niel also spoke about linear reg. model that gave him pretty good results. I was also surprised this competition\r\n had less turn out than others.
", "rawMarkdown": "", "votes": null }, { "id": "7778", "postDate": "01/09/2012 03:25:17", "content": "This was an interesting contest! Many thanks to the organizers & other competitors, and congratulations to IIdefons. Before discussing models, I thought I'd start a discussion about the data itself & how it generally impacted peoples' modeling approaches.\r\n So here are some observations of my own, in no particular order:
\r\nObservations
\r\n1. Bids/asks from T=1...T=47 seemed to provide little predictive value. My variable-selection algorithms dropped them. In the forums, I noticed others mentioned that they also saw little value in using these prices.
\r\n2. The error contribution right at the market open (at 8AM) was extremely large. For one model, I found 12% of squared error for the entire trading DAY occured in the first MINUTE of trading. I trained a seperate model for the open (the naive benchmark worked\r\n better than a regression at the open, for example) and got about a 0.0050 improvement, best case.
\r\n3. I didn't see price "resiliency" that the organizers discussed. Some of the examples the organizers posted showed stock prices bouncing back to pre-liquidity-event levels; we did not see this on average. Looking at the trade data in aggregate (via time\r\n averages, and various PCAs), we saw that for buys, the ask price jumped up immediately due to the liquidity event, and the bid price jumped up one time period later, and then both the bids & asks rose very slowly. The opposite happened for sells.
\r\n4. For some of our models, we found that training a separate model for each stock _underperformed_ training a general model for all stocks. So a per-stock model was not necessarily a big winner, as we first suspected.
\r\n5. Prediction accuracy varied across time. Using a holdout set & one of our models, I found that the error rose as you got farther from the liquidity-event trade. The RMSE was about 0.4 at T=52, rising to over 1.6 at T=100. RMSE rose roughly with sqrt(t),\r\n which, to me, implied some random-walk behavior away from the known prices at T=51.
\r\n6. The "liquidity event" trades did not seem to impact prices very much. Roughly 99.7% of the time, the VWAP was exactly equal to the best bid or ask at T=50. If there was a huge trade that ate through multiple levels of bid or ask prices, I would expect\r\n the VWAP to be different than the inside bid/ask immediately after the trade. It might have been somewhat more interesting if the trading data had some more large, market-moving trades.
\r\nSuggesetions for Improving the Contest
\r\nThere were a few things that I thought could be changed to improve this contest; others have mentioned these, but I'll reiterate them:
\r\n7. The sampling methods used to create the testing & training were different, and from my perspective, it would have been easier if they were sampled same way. The proportions of each security in testing vs training differed, of course. Also, the testing\r\n set was in random order, so why not also randomize the training set? One could correct for these testing vs training set differences by using different, per-stock weights for each row of data, or creating per-stock models. But this seemed like extra work\r\n that could have been avoided with uniform sampling. In the end, it took time away from focusing on the main goal of predicting the price behavior of the stocks.
\r\n8. The average prices for stocks in the dataset varied by a couple order of magnitudes, and when this was combined with the RMSE metric, this meant that high-price stocks (which contributed most to RMSE) dominated. For example, stock 75 -- with the highest\r\n price -- gave 36% of all squared error for one of our models. If the price data we were given was normalized (say, by dividing all prices by their VWAP), then perhaps the resulting models would be more generalizable across all stocks, regardless of price..
\r\nEverything considered, I thought this was an interesting contest in a "hot" area in finance. I look forward to reading about what others found & did to create their models!
", "rawMarkdown": "", "votes": null }, { "id": "7779", "postDate": "01/09/2012 03:26:28", "content": "Congratulations to Ildefons. Having grappled with this dataset over weeks, I can attest that an RMSE under 0.77 is a tremendous achievement.
\r\nAs a side note, Kaggle allowing submissions past the deadline is a great service for the contestants. I know I will be playing with this data a bit longer.
", "rawMarkdown": "", "votes": null }, { "id": "7785", "postDate": "01/09/2012 05:31:23", "content": "Congratulations to the winners !
\r\nMy best submission is slightly better than what I picked. Did you guys select your best submission?
\r\nKaggle public leaderboard gives you feedback to further investigate some techniques rather than others, but the public leaderboard of this competition is somewhat special, I spent too much time investigating the wrong techniques
", "rawMarkdown": "", "votes": null }, { "id": "7786", "postDate": "01/09/2012 05:38:35", "content": "Additional/expanded observations
\r\nA histogram of the liquidity shock times for the initial and final testing data had a sharp peak at t<60 s, and then very flat from ~ 6 minutes through end of day.
\r\nPartially because of this, and also due to other similarities, for most of the competition I trained with the initial testing set (last 50k rows in training). It looks like most of the initial testing data, all of day 1 & 2 data, and all of the final testing\r\n data follow similar dynamics, while data from day 3 on looks different.
\r\nTowards the end of the contest I switched to training with subsets of day 1 & 2 data sampled to match testing distributions. This resulted in better predictions, but in the end I think I may have spent too little time on the (t>60) models. It has occurred\r\n to me that the ability to quickly identify this regime change could be useful. Wrt my working on this data, this may only be the beginning:)
\r\nCareful attention to t<60 models resulted in a 1.4% overall final score improvement vs naive constant, so perhaps I did something more useful here.
\r\nAgain, thanks to the organizers and fellow competitors.
", "rawMarkdown": "", "votes": null }, { "id": "7788", "postDate": "01/09/2012 07:15:19", "content": "@Ali
\r\nI am surprised by the high correlation in public and private scores. I had done some modeling that led me to think the disparity would be much worse. Because of this, late in the competition, I tried very hard not to make too much of the public score, and\r\n to evaluate models based only on (out of sample) training data results. In the end the model I would have picked as doing best came in 2nd of my models, and the model I would have picked as 2nd came in first. But then the public leaderboard scores would have\r\n led you to the same conclusion.
\r\nIt may be that the winning model was not selected?
\r\nThe topic of public vs private leaderboard results and how to evaluate as a competitor is worthy of investigation.
\r\n\r\n
", "rawMarkdown": "", "votes": null }, { "id": "7794", "postDate": "01/09/2012 07:32:41", "content": "
This is an awesome way to wake up ! :-)
\r\nCongratulations to everyone that competed and thank you very much to the CMCRC team for setting up this competition and the support.
", "rawMarkdown": "", "votes": null }, { "id": "7799", "postDate": "01/09/2012 09:57:54", "content": "Congrats IIdefons Margrans!
", "rawMarkdown": "", "votes": null }, { "id": "7802", "postDate": "01/09/2012 14:57:23", "content": "Now, I think, will be a good time to learn what would be the score of internal model by Capital Markets. (preferably trained on the same training data set (or subset of it))
\r\n\r\n
I am wondering if anybody managed to use Neural Network successfully. In all our attempts NN did not performer better than Linear Regression. (At the end our model was combination of LR and Random Forest)
", "rawMarkdown": "", "votes": null }, { "id": "7804", "postDate": "01/09/2012 15:20:03", "content": "Congratulations to the winner!
", "rawMarkdown": "", "votes": null }, { "id": "7806", "postDate": "01/09/2012 15:34:45", "content": "[quote=Sergey Yurgenson;7802]
\r\nNow, I think, will be a good time to learn what would be the score of internal model by Capital Markets. (preferably trained on the same training data set (or subset of it))
\r\nI am wondering if anybody managed to use Neural Network successfully. In all our attempts NN did not performer better than Linear Regression. (At the end our model was combination of LR and Random Forest)
\r\n[/quote]
\r\nAgreed, and if it's really a 0.4 we want to see the code to check for signs of black magic :)
\r\nRe NN: I have yet to apply a NN with any real success in any facet of my work, etiher research or on Kaggle. I think they are just one of those methods that require a high level of expertise to set up properly. Not that they can't perform well (and they\r\n seem to be coming back into fashion in academia), but they aren't for the casual tinkerer in the same way that other methods are.
", "rawMarkdown": "", "votes": null }, { "id": "7807", "postDate": "01/09/2012 15:46:00", "content": "[quote=Christopher Hefele;7778]
\r\n2. The error contribution right at the market open (at 8AM) was extremely large. For one model, I found 12% of squared error for the entire trading DAY occured in the first MINUTE of trading. I trained a seperate model for the open (the naive benchmark worked\r\n better than a regression at the open, for example) and got about a 0.0050 improvement, best case.
\r\n[/quote]
\r\nThree biggest outliers (ranked by impact on simple linear regression model) from all data (train+public+private) those correspond to the out of market conditions (before market openning) unfortunately were included in the private test set. In a case of improper\r\n handling of the conditions these 3 rows may make a great impact on the private score (~0.07-0.15 depending on model).
\r\nThis is fragments with row_id's 758422,759056 and 769050 (I did not checked the id's, in a case of doubts that they are right, ask here, I will check). If someone with big difference between private/public scores (>= 0.07) interested he can look on the predictions\r\n of these rows, correct them (by filling by the bid50,ask50 values for example) and repost their predictions to check the difference.
\r\nBTW. Just curious has someone confident predictions on far horizons? Probably no. If it is interesting it is possible to fill horizons 26..50 (bid76/ask76..bid100/ask100) by values of the previous prediction (bid75/ask75) and check the score difference.
", "rawMarkdown": "", "votes": null }, { "id": "7812", "postDate": "01/09/2012 17:48:41", "content": "Thank you, CMCRC, for creating such an engaging and interesting competition. I would also like to thank all of the competitors for creating such a competitive atmosphere. Congratulations to Ildefons; you did an excellent job.
\r\n
\r\nI ended up not selecting my best score, but my private leaderboard score seems to have improved throughout the contest. I have just started learning statistics and R, and I picked up quite a lot throughout this competition. I hope to have the chance to compete\r\n against all of you in the future.
\r\n
\r\nThe data that we were provided with for this competition was very interesting in many respects. I would like to discuss Christopher and Cole's observations before getting into some of my own points.
\r\n
\r\n1. I noticed that although most of the predictive value was concentrated after t=45, after correcting t=0 to t=45 for outliers(most notably errors right after market open, which I will get into later), there was some predictive value to be had in these values. \r\n Predictors based on data from t=45 to t=50 also had some unfortunate nonlinear tendencies, which were minimized when using longer time horizons. Observations from t=0 to t=45 were also very useful for creating predictors based on volatility.
\r\n
\r\n2. Much of the daily error was concentrated right at market open. There were secondary areas of high error and volatility at 10:30, 13:30, and 15:00. There was a theory posited on these forums that these secondary areas were a result of other markets opening,\r\n which created arbitrage opportunities. The arbitrage opportunities may have unfolded in a predictable fashion, but I did not have time to solve this issue.
\r\n
\r\n3. I also found that a "per stock" model underperformed a model trained on the entire data set. Models trained on specific subsets of time(because there was a definite time of day effect) also underperformed models trained on the entire set.
\r\n
\r\n4. I found that this competition, because of its design, came down to two distinct predictions problems. Competitors had to predict the bid-ask spread as it recovered from t=51 to t=100, but also the trend of bid and ask prices from t=51 to t=100. As one\r\n may expect, the trend was much harder to predict than the spread. The fact that the trend could be predicted at all is interesting, but the predictive abilities of the models I tried were not extremely strong, partially because it was difficult to test my\r\n trend models(testing on out of sample training data was problematic, because the training data has similar trend characteristics throughout, whereas the test data, sampled in a different way, had drastically different trends). The spread, by contrast, had\r\n much more uniform characteristics across both the training and the testing sets.
\r\n
\r\n5. There were several dozen rows in the training and testing sets that were possibly erroneous. These observations all occured at market open, and in them, the bid-ask spread was huge(up to 600 for one stock!). These large spreads affected linear models\r\n unless they were corrected. Their spreads from t=51 to t=100 did not recover in the same fashion as the typical spreads, and a significant portion of the error came from these rows. A full list of the rows that i suspect were erroneous in the testing set\r\n is attached to this post. Note that I used an automated methodology to select these rows, so not all of them may be erroneous. While there were approximately 200 of these rows in the testing set, there were only about 250 in the training set, which made\r\n them extremely hard to predict. The final outcome was doubtless influenced heavily by these rows, as alegro has pointed out. I noticed swings of up to .008 on the public and private leaderboards by varying the predictions on these rows alone. It appears\r\n that the 70% of the test set that was held out for the private leaderboard score contained significant outliers(as evinced by the higher private RMSE scores vs public, although I could be incorrect). Correcting for these outliers(or perhaps even one or two\r\n observations) was a primary goal for many competitors, I am sure.
\r\n
\r\n6. I agree that the different sampling methods for the training and the testing sets affected the outcome of the competition heavily. The testing set was biased towards observations from the beginning of the day. The training set contained a large proportion\r\n of data from the beginning of the day, but also had a lot of observations from the end of the day, leading to a somewhat U-shaped time vs amount of observations graph. Attempting to correct for these variations did not aid my model, however.
\r\n
\r\n7. The algorithm that was used by the CMCRC data provider to clean the data prior to it being delivered to us was also relevant to the competition. I used time series outlier filtration, which uses a process found in a few papers that involves moving averages\r\n and a distance of 3 standard deviations to filter large outlying values from tick data. I found very few observations outside of this threshold. Additionally, all the ticks were in order, and none, aside from a few at the beginning of the day, could be unequivocally\r\n called "bad." Knowing how the data was removed and filtered prior to is being delivered to us might have impacted the accuracy of our predictors.
\r\n
\r\n8. Volume information would have been a huge boost to the predictive capability of my model, and I am sure those of many others. It would have aided in establishing how deep the limit book was at any given time. I would suggest that future competitions focused\r\n on predicting liquidity shocks address the outlier problem, shorten the prediction time horizon to 25, and give volume and timestamp information for every trade.
\r\n
\r\n9. The data exhibited significant heteroskedasticity, but I had little luck solving the issue with weighted models, clustering, or "per-stock" models. Feature selection helped to mitigate the issue by minimizing predictors that showed significant heteroskedasticity,\r\n but I was never able to solve the issue to my satisfaction. Did anyone manage to do so?
Again, thank you to CMCRC and all the competitors for creating such an engaging opportunity. I am very interested in finance and algorithmic trading, although my background is not necessarily in the field, and this was a good way to model tick data, which\r\n is usually very hard to obtain.
", "rawMarkdown": "", "votes": null }, { "id": "7813", "postDate": "01/09/2012 17:54:16", "content": "[quote=Capital Markets CRC;7780] Regarding (8) high priced stocks do have a disproportionate effect on RMSE. Again there is somewhat of a need to compromise. Suppose we normalize by dividing high stock prices by some factor. This will depress pvalue.\r\n Or if we leave pvalue unchanged this will distort the relationship between p_value and price. Once again we acknowledge that were we to run this again we would be able to improve implementation in this area. [/quote]
\r\nAgreed, and I acknowledge framing a competition involves a lot of difficult compromises. Perhaps another way to address this issue in future competitions might be to change the evaluation metric instead of the data -- for example, use RMSLE (root-mean-square\r\n of the difference between the logs of the prices), or the RMS of (predicted_price/actual_price) -1.
", "rawMarkdown": "", "votes": null }, { "id": "7814", "postDate": "01/09/2012 18:23:49", "content": "Anyone that had success with linear models want to elaborate on them? I had very little success with them (in R glm, glmnet, lm) and am curious where I went wrong.
", "rawMarkdown": "", "votes": null }, { "id": "7815", "postDate": "01/09/2012 18:28:06", "content": "[quote=Christopher Hefele;7813]
\r\n[quote=Capital Markets CRC;7780] Regarding (8) high priced stocks do have a disproportionate effect on RMSE. Again there is somewhat of a need to compromise. Suppose we normalize by dividing high stock prices by some factor. This will depress pvalue.\r\n Or if we leave pvalue unchanged this will distort the relationship between p_value and price. Once again we acknowledge that were we to run this again we would be able to improve implementation in this area. [/quote]
\r\nAgreed, and I acknowledge framing a competition involves a lot of difficult compromises. Perhaps another way to address this issue in future competitions might be to change the evaluation metric instead of the data -- for example, use RMSLE (root-mean-square\r\n of the difference between the logs of the prices), or the RMS of (predicted_price/actual_price) -1.
\r\n[/quote]
\r\nIdeally the metric would reflect the potential monetary benefit derived from the use of the algorithm. Likely a trader would, all things being equal, trade relatively more low priced shares vs high priced shares, so a weighted metric is appropriate. Christopher's\r\n metrics accomplish this.
", "rawMarkdown": "", "votes": null }, { "id": "7816", "postDate": "01/09/2012 18:35:42", "content": "[quote=alegro;7807]
\r\nThree biggest outliers (ranked by impact on simple linear regression model) from all data (train+public+private) those correspond to the out of market conditions (before market openning) unfortunately were included in the private test set. In a case of improper\r\n handling of the conditions these 3 rows may make a great impact on the private score (~0.07-0.15 depending on model).
\r\n\r\n
[/quote]
\r\nAlegro, would you please clarify how you identified these 3 test set rows as outliers? Was it just that they had similar properties (e.g. very early trading time of day) to outliers found in the training data? Thanks!
", "rawMarkdown": "", "votes": null }, { "id": "8019", "postDate": "01/09/2012 19:05:34", "content": "Ok. Here is the secret recipe for Linear Regression meal:
\r\ngo to your friendly neighbor datastore and choose couple fresh pieces of data (day1 and last 50k)
\r\ncut out bones and extra fat (leave only columns 5 170 206 207)
\r\ncook separately "seller initiated" transactions and "buyer initiated" transactions using your favorite linear regression function (do it separately for each askN and bidN to be predicted)
\r\nUse 200 created LRs to calculate required predictions and nicely plate them into submission file.
\r\nServe hot, because you do not want to miss 0.77590 public score and 0.77956 private score
\r\n:)
", "rawMarkdown": "", "votes": null }, { "id": "8020", "postDate": "01/09/2012 19:47:20", "content": "Hey Sergey.....sounds quite tasty :)
\r\nIs there a reason that you picked those columns for your regression model? Was this by trial and error?
", "rawMarkdown": "", "votes": null }, { "id": "8021", "postDate": "01/09/2012 20:06:13", "content": "[quote=Sergey Yurgenson;8019]
\r\nOk. Here is the secret recipe for Linear Regression meal:
\r\ngo to your friendly neighbor datastore and choose couple fresh pieces of data (day1 and last 50k)
\r\ncut out bones and extra fat (leave only columns 5 170 206 207)
\r\ncook separately "seller initiated" transactions and "buyer initiated" transactions using your favorite linear regression function (do it separately for each askN and bidN to be predicted)
\r\nUse 200 created LRs to calculate required predictions and nicely plate them into submission file.
\r\nServe hot, because you do not want to miss 0.77590 public score and 0.77956 private score
\r\n:)
\r\n[/quote]
\r\nMy best result was with a similar algorithm wrt separate buy/sell & bid/ask at each post shock time 51-100. But my model was more complicated: incorporating more predictors and designed to predict a windowed mean price rather than a price at a particular\r\n time point. Curious, how did you identify your predictors (bid41)? Your training data subset?
", "rawMarkdown": "", "votes": null }, { "id": "8023", "postDate": "01/09/2012 20:17:41", "content": "[quote=Bruce Cragin;7816]Alegro, would you please clarify how you identified these 3 test set rows as outliers? Was it just that they had similar properties (e.g. very early trading time of day) to outliers found in the training data? Thanks![/quote]These\r\n rows are outliers in histogram of per row RMSD's of a model responses (predictions) against naive model (bid50/ask50). Manual investigation shows that they have unusual pattern (with values) of predictors that does not represented in the train data. Changing\r\n predictions for these rows to the naive model values does not change public score (private score changed from 0.85444 to 0.77965).
\r\nThe model used in the experiment above is not complex linear model. Average of two runs of this model with different parameters wins the secondary milestone.
", "rawMarkdown": "", "votes": null }, { "id": "8024", "postDate": "01/09/2012 20:29:48", "content": "Interesting approach! Thanks for sharing these details.
", "rawMarkdown": "", "votes": null }, { "id": "8025", "postDate": "01/09/2012 20:38:49", "content": "[quote=Cole Harris;8021]
\r\n[quote=Sergey Yurgenson;8019]
\r\nOk. Here is the secret recipe for Linear Regression meal:
\r\ngo to your friendly neighbor datastore and choose couple fresh pieces of data (day1 and last 50k)
\r\ncut out bones and extra fat (leave only columns 5 170 206 207)
\r\ncook separately "seller initiated" transactions and "buyer initiated" transactions using your favorite linear regression function (do it separately for each askN and bidN to be predicted)
\r\nUse 200 created LRs to calculate required predictions and nicely plate them into submission file.
\r\nServe hot, because you do not want to miss 0.77590 public score and 0.77956 private score
\r\n:)
\r\n[/quote]
\r\nMy best result was with a similar algorithm wrt separate buy/sell & bid/ask at each post shock time 51-100. But my model was more complicated: incorporating more predictors and designed to predict a windowed mean price rather than a price at a particular\r\n time point. Curious, how did you identify your predictors (bid41)? Your training data subset?
\r\n[/quote]
\r\n\r\n
Our initial assumption was that last 50k was the best approximation to the test data sets. (At the end of competition part of our submissions was trained on "random" subsets designed to emulate test dataset). After some analysis we decided that day1 was\r\n "closer" to the last 50k and test set than other days. Thus we used day1+50k as training dataset for many submissions. For simple crossvalidation one can train models on day1 and validate them on last 50k. I do not remember testing any models without columns\r\n 206, 207, choice of other predictors was result of stepwise regression and crossvalidation.
\r\nI have to point out that we did not choose that model for our final submission.
", "rawMarkdown": "", "votes": null }, { "id": "8028", "postDate": "01/09/2012 20:56:55", "content": "Just to add a note to what Sergey said, the random subsets he mentioned (each of which had the same stock and buyer/seller-initiated counts as the test set) were in fact each drawn from the full training file. In the end, those models were quite competitive\r\n with the Day 1+last 50k-trained ones.
", "rawMarkdown": "", "votes": null }, { "id": "8032", "postDate": "01/09/2012 21:03:22", "content": "Attached are histograms of the number of total (bid and ask) price changes for various subsets of the training data.
\r\nThis result led me to think (1) something happened after day 2, (2) the final testing data was sampled at a time similar to days 1 & 2, and (3) the initial testing set was sampled primarily at a time similar to days 1 & 2 as well.
\r\nMatching this statistic between training and test produced training scores much more in line with leaderboard scores.
", "rawMarkdown": "", "votes": null }, { "id": "8033", "postDate": "01/09/2012 22:00:35", "content": "Cole, your two peaks in number of changes correspond closely to a quality I called easy or hard to predict (based on rmsd of a backward model prediction of the Day1 - Day6 data using the last 50000 as the training set). The plots below show, for each stock,\r\n the distribution of row_id's of a selection of the hardest (highest rmsd) and easiest to predict cases. The data points congregate in such a way that you can easily identify the various days. It's clear that for most if not all stocks Day 1 and perhaps to\r\n a slightly lesser extent Day 2 have many more of the easy to predict cases, in agreement with your result. I tried using only easy training data, using only hard training data etc, but did not find a combination that seemed to be especially effective, given\r\n that the distribution of easy vs. hard in the test set is fixed. But it's quite possible this was a missed opportunity...
", "rawMarkdown": "", "votes": null }, { "id": "8044", "postDate": "01/10/2012 16:17:12", "content": "[quote=Bruce Cragin;8033]
\r\nI tried using only easy training data, using only hard training data etc, but did not find a combination that seemed to be especially effective, given that the distribution of easy vs. hard in the test set is fixed. But it's quite possible this was a missed\r\n opportunity...
\r\n[/quote]
\r\nWrt the contest metric, I also think we missed an opportunity. On a more practical level, it seems very useful to have a means of determining a confidence level in your predictions.
", "rawMarkdown": "", "votes": null }, { "id": "8047", "postDate": "01/10/2012 19:52:59", "content": "Given the discussion above, I wish I had paid more attention to individual rows that were outliers! I just did a quick experiement to see how much a few rows might dominate RMSE. I used the naive predictor to make some predictions on the last 50K lines\r\n of the training dataset, and then I plotted the cumulative squared error across rows.
\r\nThe resulting plot is attached. It shows that errors are pretty concentrated, as we knew. But here are some numbers to back up that observation: 11% of all squared error was contributed by the worst 10 rows (out of 50K rows), 30% was contributed by\r\n the worst 100 rows, and 60% was contributed by the worst 1000. So it seems that improved predictions on just a few key rows could improve one's RMSE quite a bit.
\r\nNext, instead of using price errors (e.g. Price1-Price2), I tried using "log-errors" --- actually, log(Price1) - log(Price2) --- to see if that would be a better error metric to use with RMS. It was somwwhat better: 3% of all squared "log-error" was\r\n contributed by the worst 10 rows, 11% was contributed by the worse 100, and 35% was contributed by the worst 1000.
\r\nComparing this 'log error' result to the 'regular' errors, one can see that about 30% of all error was caused by the 1000 rows when 'log-errors' were used, but that same ~30% percent of all error was caused by only 100 rows with 'regular' error. So 'log-errors'\r\n seem 10x less concentrated, at least in this toy example. Nevertheless, some subset of rows still dominate, regardless of what metric is used.
", "rawMarkdown": "", "votes": null }, { "id": "8070", "postDate": "01/12/2012 06:10:29", "content": "Lots of good information on this thread... still haven't digested all of it. Kudos to the clever folks who were able to pinpoint the outliers down to the row numbers. I tried to handle the outliers in a more generic fashion, which resulted in some improvement,\r\n but obviously not enough to win.
\r\nIt was clear from the start that the training data does not represent the test data. I tried to make a subset of the training set by following the same steps that the organizers must have followed to make the test set. While the test set did not contain\r\n any time window overlaps, there were several overlaps in the training set. After filtering out the rows with time overlaps, there were about 150K rows left. Out of these, I picked 50K random rows to train on. The original intent of this second filtering was\r\n to speed up the training process, but the pared down 50K turned out to be as good as the whole 150K, as per prediction accuracy. In the end, I don't think eliminating the overlaps helped much. The results might have been pretty much the same with a randomly\r\n picked subset.
\r\nI see references to day 1, day 2 etc. in the posts. How does one identify data from the same day? Just pick clumps of rows for a security that has the same number of trades on the previous day? Is the assumption that all securities were sampled on each day?
\r\nMy initial impression was that the prediction accuracy would inversely correlate with volatility of the prices during the first 50 events. I still think this is true (easy to check, just haven't gotten to it), but wasn't able to capitalize on this. Categorizing\r\n the rows based on variance of the bid and ask prices and then training and predicting each category separately did not seem to help. Also tried categorizing based on other properties, such as the mean spread, jump in spread at event 50, variance of the spread,\r\n ratio of spread to price, security ID etc. - none other than categorizing based on initiator type seemed to help.
\r\nThe algorithm that worked best for me was linear regression. The spreads at event 49 and 50 and the VWAP turned out to be the most useful predictors. After several tweaks to the code, I was able to extract some predictive power out of many other columns\r\n in the data set, including the prices at event 47 and 48, trade volume, count of previous day's trades, sum of previous day's trade values and number of trades vs. quotes.
\r\nThe best private leaderboard scores for my models are given below. Note that the scores show generic accuracy of the model as I am not doing any per-row massaging of the results.
\r\n| Linear Regression | \r\n0.7781 | \r\n
| kNN | \r\n0.7848 | \r\n
| SVM | \r\n0.7956 | \r\n
| Random Forest | \r\n0.7974 | \r\n
| k-means | \r\n0.7982 | \r\n
| Blended | \r\n0.7752 | \r\n
\r\n
Training time varied greatly from model to model - from around 5 seconds for k-means to a few hours for SVM (on a laptop with a 2GHz Intel CPU). The linear regression model with the most predictive power took under two minutes to train.
\r\nFor SVM, I used an RBF kernel with the same predictors that was published by the organizers. The results were mediocre and the performance was abysmal. Hopefully, Tony will divulge more details of his model and I will know what went wrong with mine.
\r\nThe linear regression model runs out of gas by the 45th event. Replacing all subsequent predictions with the prediction for event 45 returns the same score. Maybe the blended model retains its predictive power longer - I haven't checked.
\r\nI have a long list of ideas that I wanted to try, but never got to. At the same time, I don't think any of those ideas will result in a score anywhere close to Ildefons'. Maybe it's time to move on to something else. Hm... what is this CHALEARN thingy over\r\n there...?
", "rawMarkdown": "", "votes": null }, { "id": "8071", "postDate": "01/12/2012 07:06:24", "content": "[quote=Neil Thomas;8070]
\r\nI see references to day 1, day 2 etc. in the posts. How does one identify data from the same day? Just pick clumps of rows for a security that has the same number of trades on the previous day? Is the assumption that all securities were sampled on each day?
\r\n[/quote]
\r\nThe data prior to the initial testing set appears to be ordered by day, then by stock, then time of day. Although I didn't check, I am pretty certain that all stocks are sampled multiple times each day. There appears to be six days of data. The organizers\r\n could have done some randomization, but I would guess not as some statistics change dramatically after 'day 2'. Curious - how well did your public scores correlate?
", "rawMarkdown": "", "votes": null }, { "id": "8072", "postDate": "01/12/2012 07:39:20", "content": "[quote=Cole Harris;8071]
\r\nCurious - how well did your public scores correlate?
\r\n[/quote]
\r\nHere are the corresponding public scores:
\r\n| Model | \r\nPublic score | \r\nPrivate score | \r\n
| Linear Regression | \r\n0.7703 | \r\n0.7781 | \r\n
| kNN | \r\n0.7790 | \r\n0.7848 | \r\n
| SVM | \r\n0.7902 | \r\n0.7956 | \r\n
| Random Forest | \r\n0.7899 | \r\n0.7974 | \r\n
| k-means | \r\n0.7811 | \r\n0.7982 | \r\n
| Blended | \r\n0.7651 | \r\n0.7752 | \r\n
On the number of days topic. In looking at the data description it describes the ptcount and pvalue as the prior days trade count and value. I used a SQL query like this:\r\n
\r\nselect distinct securityid , pvalue , ptcount
\r\nFROM [kaggle].[dbo].[training]
\r\nwhere securityid = 1
and it returns 37 lines. Am I wrong to assume that this means the data came from 37 different days? or are you just referring to the last 50k lines or the testing data?
", "rawMarkdown": "", "votes": null }, { "id": "8074", "postDate": "01/12/2012 18:08:36", "content": "[quote=BarrenWuffet;8073]
\r\nOn the number of days topic. In looking at the data description it describes the ptcount and pvalue as the prior days trade count and value. I used a SQL query like this:
\r\nselect distinct securityid , pvalue , ptcount
\r\nFROM [kaggle].[dbo].[training]
\r\nwhere securityid = 1
and it returns 37 lines. Am I wrong to assume that this means the data came from 37 different days? or are you just referring to the last 50k lines or the testing data?
\r\n[/quote]
\r\nTry to do the same excluding last 50k lines. Last 50k lines were initial test set that was later incorporated into training set.
\r\n", "rawMarkdown": "", "votes": null }, { "id": "8075", "postDate": "01/12/2012 18:09:20", "content": "
[quote=Neil Thomas;8072]
\r\n[quote=Cole Harris;8071]
\r\nCurious - how well did your public scores correlate?
\r\n[/quote]
\r\nHere are the corresponding public scores:
\r\n| Model | \r\nPublic score | \r\nPrivate score | \r\n
| Linear Regression | \r\n0.7703 | \r\n0.7781 | \r\n
| kNN | \r\n0.7790 | \r\n0.7848 | \r\n
| SVM | \r\n0.7902 | \r\n0.7956 | \r\n
| Random Forest | \r\n0.7899 | \r\n0.7974 | \r\n
| k-means | \r\n0.7811 | \r\n0.7982 | \r\n
| Blended | \r\n0.7651 | \r\n0.7752 | \r\n
[/quote]
\r\n\r\n
How did you do blending?
", "rawMarkdown": "", "votes": null }, { "id": "8076", "postDate": "01/12/2012 19:01:42", "content": "Thanks to all for sharing interesting details. I don't think I was able to take advantage of the idiosyncrasies of the data or the similarities between subets of training data and the test set (to the extent that others might have). My third rank was due\r\n to a random forest model that scored 0.77400 on the private and 0.76479 on the public test set. The model was trained on all of the training data (except for the last 50K which I used as the cross validation set and didn't add back to the training set). I\r\n didn't have much luck with my linear regression models - with my best model scoring 0.785 on the test set. It might have been because all my models were on log returns even though the competition metric was RMSE. Simple weighted average with the linear regression\r\n model improved my score marginally to 0.773.
", "rawMarkdown": "", "votes": null }, { "id": "8083", "postDate": "01/13/2012 02:39:08", "content": "[quote=Sergey Yurgenson;8075]
\r\nHow did you do blending?
\r\n[/quote]
\r\nI had the individual models make predictions on held-out data. Linear regression was used to determine optimal weights for each model and these weights were used to blend the submissions. I believe the official term is "stacking".
", "rawMarkdown": "", "votes": null }, { "id": "8084", "postDate": "01/13/2012 03:00:06", "content": "[quote=vsh;8076]
\r\nMy third rank was due to a random forest model that scored 0.77400 on the private and 0.76479 on the public test set.
\r\n[/quote]
\r\nThat's a very good score for an individual model. How many trees did you use? My random forest model with 200 trees was slow to run, so I had it predict the prices at events 2, 10 and 50 and then followed up with a linear interpolation. Asking it to predict\r\n the price at event 50 was probably a tall order. It might have been better to predict up to event 20 or so and then set all subsequent prices to the price at that event. Did you check to see how far the model could predict?
", "rawMarkdown": "", "votes": null }, { "id": "8086", "postDate": "01/13/2012 05:15:05", "content": "Neil, I used 200 trees too. Although, with a lower sample size: 100K and larger nodesize: 100 than the defaults which reduced training time (and also increased predictability). For a single bid/ask it took 10 minutes to train on a single CPU core. I used\r\n the largest compute instance on Amazon EC2 (with 16 cores) to train the random forest models for every bid/ask - this took less than 2 hours in total. I also tried with 300 trees but it didn't make much difference.
\r\nI checked the predictability at different intervals: 55,65,75,85 and 95 and the random forests were better than my linear models at all those points. However, note that my linear regression models didn't score as well as yours. I trained using log(returns)\r\n instead of price so that different securities could be compared against each other - did you use similar transformations in your random forests? I think in general random forests should do better than linear regression almost always.
\r\nSince I was using normalized prices, I also noticed (too close to the deadline) that individual security models and models on price (as opposed to log(returns)) combined well with my random forest model. I wasn't able to take full advantage though.
\r\n- Vivek Sharma
", "rawMarkdown": "", "votes": null }, { "id": "8087", "postDate": "01/13/2012 06:08:59", "content": "Just curious, has anyone evaluated the potential profitability of application of their models?
", "rawMarkdown": "", "votes": null }, { "id": "8094", "postDate": "01/13/2012 17:25:53", "content": "Cole, I have been thinking about this question and my own opinion is somewhat pessimistic. Let's put aside the (significant) practical challenges necessary to do HFT based on tick data and assume we can instantly enter/exit positions. Full disclosure:\r\n I am not a finance person and, as a grad student, have not seen finances since I found that $5 bill on the street the other week :)
\r\nWe aren't the market maker, so we don't get the privilege of seeing the liqiudity shock coming, nor did this contest assess to our ability to predict when the shocks are coming. That means the soonest we can react is at the t=51 time point, after the bid\r\n and ask have already gapped. We know from the naive baseline that the steady-state do-nothing model is "on average" (a loose, some would say wrong, interpretation of the RMSE) 86 pence away from the real bid/ask reaction, while the best contest models knocked\r\n about 10 pence off that. In my (possibly incorrect) interpretation of the situation, this is sort of an arbitrage window of about 10 pence when averaged over many trades.
\r\nIs that enough? I suppose a more thorough analysis that controls for the share price is necessary to really say. However, if we add back in real-world constraints and assume that other market participants have access to the same information (e.g. there\r\n was a large market sell of X shares T milliseconds ago), you have to assume that they are at least clever enough to run the linear regression that negates 9/10 of your forecasting advantage.
", "rawMarkdown": "", "votes": null }, { "id": "8095", "postDate": "01/13/2012 18:36:02", "content": "A lot of the HFT stuff is based on hardware (collocation, dedicated fiber, burned chips, etc) and entity structuring (as broker/dealer in order to be market maker and collect rebates from ECNs (which from my limited knowledge is where most money is made\r\n as opposed to correctly picking direction)). That being said, there are plenty of places that would talk with you about implementation of your ideas on their hardware/communication platform such as JUMP, Tidal or any of the firms mentioned in the 'Trading'\r\n section of eFinancialCareers.com website.
", "rawMarkdown": "", "votes": null }, { "id": "8118", "postDate": "01/16/2012 10:48:31", "content": "On the question of profitability it would be helpful if someone would tell us what are the typical financial arrangements between an exchange and a HFT organisation. I understand the exchange pays the HFT organisation for keeping the market "ticking over".\r\n They certainly could not afford to pay the brokerage of a retail trader.
", "rawMarkdown": "", "votes": null }, { "id": "8120", "postDate": "01/16/2012 15:29:55", "content": "[quote=Capital Markets CRC;8117]
\r\nWilliam, if we remove the shackles of specifically looking at liquidity shocks, do you believe that the techniques discussed and developed in this competition would be useful for finding market anomalies and inefficiencies?
\r\n[/quote]
\r\nI think the liquidty shock actually frames the problem nicely. You have to set your time origin somewhere, and I think picking a larget market order is one way of isolating a timeframe where you expect\r\nsomething to happen. If we were instead given unregistered, raw tick data, we would have to modify the models to handle the many times where the bid/ask are not moving (or "random walking", or "market open mayhem", etc.). This modifciation could\r\n be as simple as a feature given to a decision tree (e.g. has there been a recent market order), but I suspect it would require more deliberate intervention. I attribute any success from the models developed in this competition to the fact that the market\r\n reacts systematically to a buy vs. sell order.
\r\nAn interesting offshoot of this observation: what would happen instead if you gave us t51...t100 bids/asks and asked us to classify whether the trade was buyer/seller initiatied? I suspect the accuracy would be very high, but the more interesting thing\r\n to look at would be the anomolous trades which we don't classify correctly. If they share common traits (a big "if"), then you can really get into the question of exploiting inefficiencies.
", "rawMarkdown": "", "votes": null }, { "id": "8127", "postDate": "01/16/2012 21:17:04", "content": "I would look on it not as a problem of finding market anomalies and inefficiencies but as a problem of pedicuring anomalies and inefficiencies. I would expect inefficiencies to be very short-lived and thus providing good time reference point. One can select\r\n one specific type of inefficiency and create dataset containing market data before inefficiency happened (obviously, with some data records when inefficiency did not happened). Then task will be to create classification model to predict future inefficiency\r\n using past market data (close analog - Credit competition)
", "rawMarkdown": "", "votes": null }, { "id": "8131", "postDate": "01/17/2012 05:10:45", "content": "@William
\r\n'The application of a model' is not trivial, even ignoring the technical issues. Turning a model prediction (derived from all liquidity events) into a trade signal may not be the best approach, which is why I asked the question. I don't think any of my contest\r\n "buy" models ever predicted a future bid beating the vwap (or initial ask), so they would break even by never trading:)
\r\nHowever it is much easier to directly address the question (with some assumptions) if I produced the liquidity shock, can I predict if I can get out at a profit in short order? Or, if I detect a liquidity shock, can I exploit the subsequent move in prices\r\n for some an identifiable subset? I find potential here.
", "rawMarkdown": "", "votes": null }, { "id": "8170", "postDate": "01/19/2012 15:46:08", "content": "Will Ildefons algorithm be revealed and explained? Thank you.
", "rawMarkdown": "", "votes": null }, { "id": "52311", "postDate": "08/22/2014 14:53:14", "content": "", "rawMarkdown": "", "votes": null }, { "id": "81324", "postDate": "06/09/2015 06:50:12", "content": "Hi Guys, I am a beginner and i am trying to learn how you guys got the solution. If anybody could please post the solution here, it would be helpful for me to understand how you guys did it. I use RStudio. Id the code you have written is in R, i would highly appreciate that. Tks
", "rawMarkdown": "", "votes": null }, { "id": "126003", "postDate": "07/05/2016 13:04:13", "content": "congrats , wish a future.
", "rawMarkdown": "congrats , wish a future.", "votes": null }, { "id": "241939", "postDate": "11/10/2017 07:20:22", "content": "what is \"price changes\" ?
\nI have checked for tick to tick price change (bid[n]-bid[n-1]) , but I am not getting a plot similar to what you have shared.
Congratulations to Ildenfons with convincing first place and other top teams with good results.\r\n
\r\n@karmic - actually, change in teams position after final scoring was not as big as in some other competitions (it was known for a while that model by Xiaoshi Lu was overfitted to public test set).\r\n
\r\nI expect some details of top models to be known soon. I can say that simple correctly executed linear regression model could put you in the top ten. (However it was not our final model)
", "votes": null, "replies": [] }, { "id": 7774, "author_name": "dejavu", "author_url": "", "post_date": "01/09/2012 01:57:06", "content": "Congratulations to all top teams as well. I can vouch for Sergey's assertion that linear regression can place in the top 10, however I'm not sure what 'correctly executed' means wrt this data.
\r\nI am very interested to learn the extent to which others addressed the obvious differences in the test and training sets. In particular, it appears there was a regime shift after day 2 in the training data, with prices becoming much 'noisier' (I can quantify\r\n if interested). Scores for models from subsets of day 1 and day 2 data statistically similar to the testing set are close to the testing scores.
\r\nI am also interested if others developed distinct models for shocks near the opening (I had separate models for t<60 & t>60).
", "votes": null, "replies": [] }, { "id": 7776, "author_name": "karmicmenace", "author_url": "", "post_date": "01/09/2012 02:59:03", "content": "Cole/Sergey - I am definitely interested in knowing more on the details of linear reg. model. I must have missed the forest looking for tree. I know Niel also spoke about linear reg. model that gave him pretty good results. I was also surprised this competition\r\n had less turn out than others.
", "votes": null, "replies": [] }, { "id": 7778, "author_name": "chefele", "author_url": "", "post_date": "01/09/2012 03:25:17", "content": "This was an interesting contest! Many thanks to the organizers & other competitors, and congratulations to IIdefons. Before discussing models, I thought I'd start a discussion about the data itself & how it generally impacted peoples' modeling approaches.\r\n So here are some observations of my own, in no particular order:
\r\nObservations
\r\n1. Bids/asks from T=1...T=47 seemed to provide little predictive value. My variable-selection algorithms dropped them. In the forums, I noticed others mentioned that they also saw little value in using these prices.
\r\n2. The error contribution right at the market open (at 8AM) was extremely large. For one model, I found 12% of squared error for the entire trading DAY occured in the first MINUTE of trading. I trained a seperate model for the open (the naive benchmark worked\r\n better than a regression at the open, for example) and got about a 0.0050 improvement, best case.
\r\n3. I didn't see price "resiliency" that the organizers discussed. Some of the examples the organizers posted showed stock prices bouncing back to pre-liquidity-event levels; we did not see this on average. Looking at the trade data in aggregate (via time\r\n averages, and various PCAs), we saw that for buys, the ask price jumped up immediately due to the liquidity event, and the bid price jumped up one time period later, and then both the bids & asks rose very slowly. The opposite happened for sells.
\r\n4. For some of our models, we found that training a separate model for each stock _underperformed_ training a general model for all stocks. So a per-stock model was not necessarily a big winner, as we first suspected.
\r\n5. Prediction accuracy varied across time. Using a holdout set & one of our models, I found that the error rose as you got farther from the liquidity-event trade. The RMSE was about 0.4 at T=52, rising to over 1.6 at T=100. RMSE rose roughly with sqrt(t),\r\n which, to me, implied some random-walk behavior away from the known prices at T=51.
\r\n6. The "liquidity event" trades did not seem to impact prices very much. Roughly 99.7% of the time, the VWAP was exactly equal to the best bid or ask at T=50. If there was a huge trade that ate through multiple levels of bid or ask prices, I would expect\r\n the VWAP to be different than the inside bid/ask immediately after the trade. It might have been somewhat more interesting if the trading data had some more large, market-moving trades.
\r\nSuggesetions for Improving the Contest
\r\nThere were a few things that I thought could be changed to improve this contest; others have mentioned these, but I'll reiterate them:
\r\n7. The sampling methods used to create the testing & training were different, and from my perspective, it would have been easier if they were sampled same way. The proportions of each security in testing vs training differed, of course. Also, the testing\r\n set was in random order, so why not also randomize the training set? One could correct for these testing vs training set differences by using different, per-stock weights for each row of data, or creating per-stock models. But this seemed like extra work\r\n that could have been avoided with uniform sampling. In the end, it took time away from focusing on the main goal of predicting the price behavior of the stocks.
\r\n8. The average prices for stocks in the dataset varied by a couple order of magnitudes, and when this was combined with the RMSE metric, this meant that high-price stocks (which contributed most to RMSE) dominated. For example, stock 75 -- with the highest\r\n price -- gave 36% of all squared error for one of our models. If the price data we were given was normalized (say, by dividing all prices by their VWAP), then perhaps the resulting models would be more generalizable across all stocks, regardless of price..
\r\nEverything considered, I thought this was an interesting contest in a "hot" area in finance. I look forward to reading about what others found & did to create their models!
", "votes": null, "replies": [] }, { "id": 7779, "author_name": "anlthms", "author_url": "", "post_date": "01/09/2012 03:26:28", "content": "Congratulations to Ildefons. Having grappled with this dataset over weeks, I can attest that an RMSE under 0.77 is a tremendous achievement.
\r\nAs a side note, Kaggle allowing submissions past the deadline is a great service for the contestants. I know I will be playing with this data a bit longer.
", "votes": null, "replies": [] }, { "id": 7785, "author_name": "ahassaine", "author_url": "", "post_date": "01/09/2012 05:31:23", "content": "Congratulations to the winners !
\r\nMy best submission is slightly better than what I picked. Did you guys select your best submission?
\r\nKaggle public leaderboard gives you feedback to further investigate some techniques rather than others, but the public leaderboard of this competition is somewhat special, I spent too much time investigating the wrong techniques
", "votes": null, "replies": [] }, { "id": 7786, "author_name": "dejavu", "author_url": "", "post_date": "01/09/2012 05:38:35", "content": "Additional/expanded observations
\r\nA histogram of the liquidity shock times for the initial and final testing data had a sharp peak at t<60 s, and then very flat from ~ 6 minutes through end of day.
\r\nPartially because of this, and also due to other similarities, for most of the competition I trained with the initial testing set (last 50k rows in training). It looks like most of the initial testing data, all of day 1 & 2 data, and all of the final testing\r\n data follow similar dynamics, while data from day 3 on looks different.
\r\nTowards the end of the contest I switched to training with subsets of day 1 & 2 data sampled to match testing distributions. This resulted in better predictions, but in the end I think I may have spent too little time on the (t>60) models. It has occurred\r\n to me that the ability to quickly identify this regime change could be useful. Wrt my working on this data, this may only be the beginning:)
\r\nCareful attention to t<60 models resulted in a 1.4% overall final score improvement vs naive constant, so perhaps I did something more useful here.
\r\nAgain, thanks to the organizers and fellow competitors.
", "votes": null, "replies": [] }, { "id": 7788, "author_name": "dejavu", "author_url": "", "post_date": "01/09/2012 07:15:19", "content": "@Ali
\r\nI am surprised by the high correlation in public and private scores. I had done some modeling that led me to think the disparity would be much worse. Because of this, late in the competition, I tried very hard not to make too much of the public score, and\r\n to evaluate models based only on (out of sample) training data results. In the end the model I would have picked as doing best came in 2nd of my models, and the model I would have picked as 2nd came in first. But then the public leaderboard scores would have\r\n led you to the same conclusion.
\r\nIt may be that the winning model was not selected?
\r\nThe topic of public vs private leaderboard results and how to evaluate as a competitor is worthy of investigation.
\r\n\r\n
", "votes": null, "replies": [] }, { "id": 7794, "author_name": "ildefons", "author_url": "", "post_date": "01/09/2012 07:32:41", "content": "
This is an awesome way to wake up ! :-)
\r\nCongratulations to everyone that competed and thank you very much to the CMCRC team for setting up this competition and the support.
", "votes": null, "replies": [] }, { "id": 7799, "author_name": "zkc12311", "author_url": "", "post_date": "01/09/2012 09:57:54", "content": "Congrats IIdefons Margrans!
", "votes": null, "replies": [] }, { "id": 7802, "author_name": "ccccat", "author_url": "", "post_date": "01/09/2012 14:57:23", "content": "Now, I think, will be a good time to learn what would be the score of internal model by Capital Markets. (preferably trained on the same training data set (or subset of it))
\r\n\r\n
I am wondering if anybody managed to use Neural Network successfully. In all our attempts NN did not performer better than Linear Regression. (At the end our model was combination of LR and Random Forest)
", "votes": null, "replies": [] }, { "id": 7804, "author_name": "alegro", "author_url": "", "post_date": "01/09/2012 15:20:03", "content": "Congratulations to the winner!
", "votes": null, "replies": [] }, { "id": 7806, "author_name": "wcukierski", "author_url": "", "post_date": "01/09/2012 15:34:45", "content": "[quote=Sergey Yurgenson;7802]
\r\nNow, I think, will be a good time to learn what would be the score of internal model by Capital Markets. (preferably trained on the same training data set (or subset of it))
\r\nI am wondering if anybody managed to use Neural Network successfully. In all our attempts NN did not performer better than Linear Regression. (At the end our model was combination of LR and Random Forest)
\r\n[/quote]
\r\nAgreed, and if it's really a 0.4 we want to see the code to check for signs of black magic :)
\r\nRe NN: I have yet to apply a NN with any real success in any facet of my work, etiher research or on Kaggle. I think they are just one of those methods that require a high level of expertise to set up properly. Not that they can't perform well (and they\r\n seem to be coming back into fashion in academia), but they aren't for the casual tinkerer in the same way that other methods are.
", "votes": null, "replies": [] }, { "id": 7807, "author_name": "alegro", "author_url": "", "post_date": "01/09/2012 15:46:00", "content": "[quote=Christopher Hefele;7778]
\r\n2. The error contribution right at the market open (at 8AM) was extremely large. For one model, I found 12% of squared error for the entire trading DAY occured in the first MINUTE of trading. I trained a seperate model for the open (the naive benchmark worked\r\n better than a regression at the open, for example) and got about a 0.0050 improvement, best case.
\r\n[/quote]
\r\nThree biggest outliers (ranked by impact on simple linear regression model) from all data (train+public+private) those correspond to the out of market conditions (before market openning) unfortunately were included in the private test set. In a case of improper\r\n handling of the conditions these 3 rows may make a great impact on the private score (~0.07-0.15 depending on model).
\r\nThis is fragments with row_id's 758422,759056 and 769050 (I did not checked the id's, in a case of doubts that they are right, ask here, I will check). If someone with big difference between private/public scores (>= 0.07) interested he can look on the predictions\r\n of these rows, correct them (by filling by the bid50,ask50 values for example) and repost their predictions to check the difference.
\r\nBTW. Just curious has someone confident predictions on far horizons? Probably no. If it is interesting it is possible to fill horizons 26..50 (bid76/ask76..bid100/ask100) by values of the previous prediction (bid75/ask75) and check the score difference.
", "votes": null, "replies": [] }, { "id": 7812, "author_name": "vikpar", "author_url": "", "post_date": "01/09/2012 17:48:41", "content": "Thank you, CMCRC, for creating such an engaging and interesting competition. I would also like to thank all of the competitors for creating such a competitive atmosphere. Congratulations to Ildefons; you did an excellent job.
\r\n
\r\nI ended up not selecting my best score, but my private leaderboard score seems to have improved throughout the contest. I have just started learning statistics and R, and I picked up quite a lot throughout this competition. I hope to have the chance to compete\r\n against all of you in the future.
\r\n
\r\nThe data that we were provided with for this competition was very interesting in many respects. I would like to discuss Christopher and Cole's observations before getting into some of my own points.
\r\n
\r\n1. I noticed that although most of the predictive value was concentrated after t=45, after correcting t=0 to t=45 for outliers(most notably errors right after market open, which I will get into later), there was some predictive value to be had in these values. \r\n Predictors based on data from t=45 to t=50 also had some unfortunate nonlinear tendencies, which were minimized when using longer time horizons. Observations from t=0 to t=45 were also very useful for creating predictors based on volatility.
\r\n
\r\n2. Much of the daily error was concentrated right at market open. There were secondary areas of high error and volatility at 10:30, 13:30, and 15:00. There was a theory posited on these forums that these secondary areas were a result of other markets opening,\r\n which created arbitrage opportunities. The arbitrage opportunities may have unfolded in a predictable fashion, but I did not have time to solve this issue.
\r\n
\r\n3. I also found that a "per stock" model underperformed a model trained on the entire data set. Models trained on specific subsets of time(because there was a definite time of day effect) also underperformed models trained on the entire set.
\r\n
\r\n4. I found that this competition, because of its design, came down to two distinct predictions problems. Competitors had to predict the bid-ask spread as it recovered from t=51 to t=100, but also the trend of bid and ask prices from t=51 to t=100. As one\r\n may expect, the trend was much harder to predict than the spread. The fact that the trend could be predicted at all is interesting, but the predictive abilities of the models I tried were not extremely strong, partially because it was difficult to test my\r\n trend models(testing on out of sample training data was problematic, because the training data has similar trend characteristics throughout, whereas the test data, sampled in a different way, had drastically different trends). The spread, by contrast, had\r\n much more uniform characteristics across both the training and the testing sets.
\r\n
\r\n5. There were several dozen rows in the training and testing sets that were possibly erroneous. These observations all occured at market open, and in them, the bid-ask spread was huge(up to 600 for one stock!). These large spreads affected linear models\r\n unless they were corrected. Their spreads from t=51 to t=100 did not recover in the same fashion as the typical spreads, and a significant portion of the error came from these rows. A full list of the rows that i suspect were erroneous in the testing set\r\n is attached to this post. Note that I used an automated methodology to select these rows, so not all of them may be erroneous. While there were approximately 200 of these rows in the testing set, there were only about 250 in the training set, which made\r\n them extremely hard to predict. The final outcome was doubtless influenced heavily by these rows, as alegro has pointed out. I noticed swings of up to .008 on the public and private leaderboards by varying the predictions on these rows alone. It appears\r\n that the 70% of the test set that was held out for the private leaderboard score contained significant outliers(as evinced by the higher private RMSE scores vs public, although I could be incorrect). Correcting for these outliers(or perhaps even one or two\r\n observations) was a primary goal for many competitors, I am sure.
\r\n
\r\n6. I agree that the different sampling methods for the training and the testing sets affected the outcome of the competition heavily. The testing set was biased towards observations from the beginning of the day. The training set contained a large proportion\r\n of data from the beginning of the day, but also had a lot of observations from the end of the day, leading to a somewhat U-shaped time vs amount of observations graph. Attempting to correct for these variations did not aid my model, however.
\r\n
\r\n7. The algorithm that was used by the CMCRC data provider to clean the data prior to it being delivered to us was also relevant to the competition. I used time series outlier filtration, which uses a process found in a few papers that involves moving averages\r\n and a distance of 3 standard deviations to filter large outlying values from tick data. I found very few observations outside of this threshold. Additionally, all the ticks were in order, and none, aside from a few at the beginning of the day, could be unequivocally\r\n called "bad." Knowing how the data was removed and filtered prior to is being delivered to us might have impacted the accuracy of our predictors.
\r\n
\r\n8. Volume information would have been a huge boost to the predictive capability of my model, and I am sure those of many others. It would have aided in establishing how deep the limit book was at any given time. I would suggest that future competitions focused\r\n on predicting liquidity shocks address the outlier problem, shorten the prediction time horizon to 25, and give volume and timestamp information for every trade.
\r\n
\r\n9. The data exhibited significant heteroskedasticity, but I had little luck solving the issue with weighted models, clustering, or "per-stock" models. Feature selection helped to mitigate the issue by minimizing predictors that showed significant heteroskedasticity,\r\n but I was never able to solve the issue to my satisfaction. Did anyone manage to do so?
Again, thank you to CMCRC and all the competitors for creating such an engaging opportunity. I am very interested in finance and algorithmic trading, although my background is not necessarily in the field, and this was a good way to model tick data, which\r\n is usually very hard to obtain.
", "votes": null, "replies": [] }, { "id": 7813, "author_name": "chefele", "author_url": "", "post_date": "01/09/2012 17:54:16", "content": "[quote=Capital Markets CRC;7780] Regarding (8) high priced stocks do have a disproportionate effect on RMSE. Again there is somewhat of a need to compromise. Suppose we normalize by dividing high stock prices by some factor. This will depress pvalue.\r\n Or if we leave pvalue unchanged this will distort the relationship between p_value and price. Once again we acknowledge that were we to run this again we would be able to improve implementation in this area. [/quote]
\r\nAgreed, and I acknowledge framing a competition involves a lot of difficult compromises. Perhaps another way to address this issue in future competitions might be to change the evaluation metric instead of the data -- for example, use RMSLE (root-mean-square\r\n of the difference between the logs of the prices), or the RMS of (predicted_price/actual_price) -1.
", "votes": null, "replies": [] }, { "id": 7814, "author_name": "barrenwuffet", "author_url": "", "post_date": "01/09/2012 18:23:49", "content": "Anyone that had success with linear models want to elaborate on them? I had very little success with them (in R glm, glmnet, lm) and am curious where I went wrong.
", "votes": null, "replies": [] }, { "id": 7815, "author_name": "dejavu", "author_url": "", "post_date": "01/09/2012 18:28:06", "content": "[quote=Christopher Hefele;7813]
\r\n[quote=Capital Markets CRC;7780] Regarding (8) high priced stocks do have a disproportionate effect on RMSE. Again there is somewhat of a need to compromise. Suppose we normalize by dividing high stock prices by some factor. This will depress pvalue.\r\n Or if we leave pvalue unchanged this will distort the relationship between p_value and price. Once again we acknowledge that were we to run this again we would be able to improve implementation in this area. [/quote]
\r\nAgreed, and I acknowledge framing a competition involves a lot of difficult compromises. Perhaps another way to address this issue in future competitions might be to change the evaluation metric instead of the data -- for example, use RMSLE (root-mean-square\r\n of the difference between the logs of the prices), or the RMS of (predicted_price/actual_price) -1.
\r\n[/quote]
\r\nIdeally the metric would reflect the potential monetary benefit derived from the use of the algorithm. Likely a trader would, all things being equal, trade relatively more low priced shares vs high priced shares, so a weighted metric is appropriate. Christopher's\r\n metrics accomplish this.
", "votes": null, "replies": [] }, { "id": 7816, "author_name": "bcragin", "author_url": "", "post_date": "01/09/2012 18:35:42", "content": "[quote=alegro;7807]
\r\nThree biggest outliers (ranked by impact on simple linear regression model) from all data (train+public+private) those correspond to the out of market conditions (before market openning) unfortunately were included in the private test set. In a case of improper\r\n handling of the conditions these 3 rows may make a great impact on the private score (~0.07-0.15 depending on model).
\r\n\r\n
[/quote]
\r\nAlegro, would you please clarify how you identified these 3 test set rows as outliers? Was it just that they had similar properties (e.g. very early trading time of day) to outliers found in the training data? Thanks!
", "votes": null, "replies": [] }, { "id": 8019, "author_name": "ccccat", "author_url": "", "post_date": "01/09/2012 19:05:34", "content": "Ok. Here is the secret recipe for Linear Regression meal:
\r\ngo to your friendly neighbor datastore and choose couple fresh pieces of data (day1 and last 50k)
\r\ncut out bones and extra fat (leave only columns 5 170 206 207)
\r\ncook separately "seller initiated" transactions and "buyer initiated" transactions using your favorite linear regression function (do it separately for each askN and bidN to be predicted)
\r\nUse 200 created LRs to calculate required predictions and nicely plate them into submission file.
\r\nServe hot, because you do not want to miss 0.77590 public score and 0.77956 private score
\r\n:)
", "votes": null, "replies": [] }, { "id": 8020, "author_name": "leazar", "author_url": "", "post_date": "01/09/2012 19:47:20", "content": "Hey Sergey.....sounds quite tasty :)
\r\nIs there a reason that you picked those columns for your regression model? Was this by trial and error?
", "votes": null, "replies": [] }, { "id": 8021, "author_name": "dejavu", "author_url": "", "post_date": "01/09/2012 20:06:13", "content": "[quote=Sergey Yurgenson;8019]
\r\nOk. Here is the secret recipe for Linear Regression meal:
\r\ngo to your friendly neighbor datastore and choose couple fresh pieces of data (day1 and last 50k)
\r\ncut out bones and extra fat (leave only columns 5 170 206 207)
\r\ncook separately "seller initiated" transactions and "buyer initiated" transactions using your favorite linear regression function (do it separately for each askN and bidN to be predicted)
\r\nUse 200 created LRs to calculate required predictions and nicely plate them into submission file.
\r\nServe hot, because you do not want to miss 0.77590 public score and 0.77956 private score
\r\n:)
\r\n[/quote]
\r\nMy best result was with a similar algorithm wrt separate buy/sell & bid/ask at each post shock time 51-100. But my model was more complicated: incorporating more predictors and designed to predict a windowed mean price rather than a price at a particular\r\n time point. Curious, how did you identify your predictors (bid41)? Your training data subset?
", "votes": null, "replies": [] }, { "id": 8023, "author_name": "alegro", "author_url": "", "post_date": "01/09/2012 20:17:41", "content": "[quote=Bruce Cragin;7816]Alegro, would you please clarify how you identified these 3 test set rows as outliers? Was it just that they had similar properties (e.g. very early trading time of day) to outliers found in the training data? Thanks![/quote]These\r\n rows are outliers in histogram of per row RMSD's of a model responses (predictions) against naive model (bid50/ask50). Manual investigation shows that they have unusual pattern (with values) of predictors that does not represented in the train data. Changing\r\n predictions for these rows to the naive model values does not change public score (private score changed from 0.85444 to 0.77965).
\r\nThe model used in the experiment above is not complex linear model. Average of two runs of this model with different parameters wins the secondary milestone.
", "votes": null, "replies": [] }, { "id": 8024, "author_name": "bcragin", "author_url": "", "post_date": "01/09/2012 20:29:48", "content": "Interesting approach! Thanks for sharing these details.
", "votes": null, "replies": [] }, { "id": 8025, "author_name": "ccccat", "author_url": "", "post_date": "01/09/2012 20:38:49", "content": "[quote=Cole Harris;8021]
\r\n[quote=Sergey Yurgenson;8019]
\r\nOk. Here is the secret recipe for Linear Regression meal:
\r\ngo to your friendly neighbor datastore and choose couple fresh pieces of data (day1 and last 50k)
\r\ncut out bones and extra fat (leave only columns 5 170 206 207)
\r\ncook separately "seller initiated" transactions and "buyer initiated" transactions using your favorite linear regression function (do it separately for each askN and bidN to be predicted)
\r\nUse 200 created LRs to calculate required predictions and nicely plate them into submission file.
\r\nServe hot, because you do not want to miss 0.77590 public score and 0.77956 private score
\r\n:)
\r\n[/quote]
\r\nMy best result was with a similar algorithm wrt separate buy/sell & bid/ask at each post shock time 51-100. But my model was more complicated: incorporating more predictors and designed to predict a windowed mean price rather than a price at a particular\r\n time point. Curious, how did you identify your predictors (bid41)? Your training data subset?
\r\n[/quote]
\r\n\r\n
Our initial assumption was that last 50k was the best approximation to the test data sets. (At the end of competition part of our submissions was trained on "random" subsets designed to emulate test dataset). After some analysis we decided that day1 was\r\n "closer" to the last 50k and test set than other days. Thus we used day1+50k as training dataset for many submissions. For simple crossvalidation one can train models on day1 and validate them on last 50k. I do not remember testing any models without columns\r\n 206, 207, choice of other predictors was result of stepwise regression and crossvalidation.
\r\nI have to point out that we did not choose that model for our final submission.
", "votes": null, "replies": [] }, { "id": 8028, "author_name": "bcragin", "author_url": "", "post_date": "01/09/2012 20:56:55", "content": "Just to add a note to what Sergey said, the random subsets he mentioned (each of which had the same stock and buyer/seller-initiated counts as the test set) were in fact each drawn from the full training file. In the end, those models were quite competitive\r\n with the Day 1+last 50k-trained ones.
", "votes": null, "replies": [] }, { "id": 8032, "author_name": "dejavu", "author_url": "", "post_date": "01/09/2012 21:03:22", "content": "Attached are histograms of the number of total (bid and ask) price changes for various subsets of the training data.
\r\nThis result led me to think (1) something happened after day 2, (2) the final testing data was sampled at a time similar to days 1 & 2, and (3) the initial testing set was sampled primarily at a time similar to days 1 & 2 as well.
\r\nMatching this statistic between training and test produced training scores much more in line with leaderboard scores.
", "votes": null, "replies": [ { "id": 241939, "author_name": "precitionlady", "author_url": "", "post_date": "11/10/2017 07:20:22", "content": "what is \"price changes\" ?
\nI have checked for tick to tick price change (bid[n]-bid[n-1]) , but I am not getting a plot similar to what you have shared.
Cole, your two peaks in number of changes correspond closely to a quality I called easy or hard to predict (based on rmsd of a backward model prediction of the Day1 - Day6 data using the last 50000 as the training set). The plots below show, for each stock,\r\n the distribution of row_id's of a selection of the hardest (highest rmsd) and easiest to predict cases. The data points congregate in such a way that you can easily identify the various days. It's clear that for most if not all stocks Day 1 and perhaps to\r\n a slightly lesser extent Day 2 have many more of the easy to predict cases, in agreement with your result. I tried using only easy training data, using only hard training data etc, but did not find a combination that seemed to be especially effective, given\r\n that the distribution of easy vs. hard in the test set is fixed. But it's quite possible this was a missed opportunity...
", "votes": null, "replies": [] }, { "id": 8044, "author_name": "dejavu", "author_url": "", "post_date": "01/10/2012 16:17:12", "content": "[quote=Bruce Cragin;8033]
\r\nI tried using only easy training data, using only hard training data etc, but did not find a combination that seemed to be especially effective, given that the distribution of easy vs. hard in the test set is fixed. But it's quite possible this was a missed\r\n opportunity...
\r\n[/quote]
\r\nWrt the contest metric, I also think we missed an opportunity. On a more practical level, it seems very useful to have a means of determining a confidence level in your predictions.
", "votes": null, "replies": [] }, { "id": 8047, "author_name": "chefele", "author_url": "", "post_date": "01/10/2012 19:52:59", "content": "Given the discussion above, I wish I had paid more attention to individual rows that were outliers! I just did a quick experiement to see how much a few rows might dominate RMSE. I used the naive predictor to make some predictions on the last 50K lines\r\n of the training dataset, and then I plotted the cumulative squared error across rows.
\r\nThe resulting plot is attached. It shows that errors are pretty concentrated, as we knew. But here are some numbers to back up that observation: 11% of all squared error was contributed by the worst 10 rows (out of 50K rows), 30% was contributed by\r\n the worst 100 rows, and 60% was contributed by the worst 1000. So it seems that improved predictions on just a few key rows could improve one's RMSE quite a bit.
\r\nNext, instead of using price errors (e.g. Price1-Price2), I tried using "log-errors" --- actually, log(Price1) - log(Price2) --- to see if that would be a better error metric to use with RMS. It was somwwhat better: 3% of all squared "log-error" was\r\n contributed by the worst 10 rows, 11% was contributed by the worse 100, and 35% was contributed by the worst 1000.
\r\nComparing this 'log error' result to the 'regular' errors, one can see that about 30% of all error was caused by the 1000 rows when 'log-errors' were used, but that same ~30% percent of all error was caused by only 100 rows with 'regular' error. So 'log-errors'\r\n seem 10x less concentrated, at least in this toy example. Nevertheless, some subset of rows still dominate, regardless of what metric is used.
", "votes": null, "replies": [] }, { "id": 8070, "author_name": "anlthms", "author_url": "", "post_date": "01/12/2012 06:10:29", "content": "Lots of good information on this thread... still haven't digested all of it. Kudos to the clever folks who were able to pinpoint the outliers down to the row numbers. I tried to handle the outliers in a more generic fashion, which resulted in some improvement,\r\n but obviously not enough to win.
\r\nIt was clear from the start that the training data does not represent the test data. I tried to make a subset of the training set by following the same steps that the organizers must have followed to make the test set. While the test set did not contain\r\n any time window overlaps, there were several overlaps in the training set. After filtering out the rows with time overlaps, there were about 150K rows left. Out of these, I picked 50K random rows to train on. The original intent of this second filtering was\r\n to speed up the training process, but the pared down 50K turned out to be as good as the whole 150K, as per prediction accuracy. In the end, I don't think eliminating the overlaps helped much. The results might have been pretty much the same with a randomly\r\n picked subset.
\r\nI see references to day 1, day 2 etc. in the posts. How does one identify data from the same day? Just pick clumps of rows for a security that has the same number of trades on the previous day? Is the assumption that all securities were sampled on each day?
\r\nMy initial impression was that the prediction accuracy would inversely correlate with volatility of the prices during the first 50 events. I still think this is true (easy to check, just haven't gotten to it), but wasn't able to capitalize on this. Categorizing\r\n the rows based on variance of the bid and ask prices and then training and predicting each category separately did not seem to help. Also tried categorizing based on other properties, such as the mean spread, jump in spread at event 50, variance of the spread,\r\n ratio of spread to price, security ID etc. - none other than categorizing based on initiator type seemed to help.
\r\nThe algorithm that worked best for me was linear regression. The spreads at event 49 and 50 and the VWAP turned out to be the most useful predictors. After several tweaks to the code, I was able to extract some predictive power out of many other columns\r\n in the data set, including the prices at event 47 and 48, trade volume, count of previous day's trades, sum of previous day's trade values and number of trades vs. quotes.
\r\nThe best private leaderboard scores for my models are given below. Note that the scores show generic accuracy of the model as I am not doing any per-row massaging of the results.
\r\n| Linear Regression | \r\n0.7781 | \r\n
| kNN | \r\n0.7848 | \r\n
| SVM | \r\n0.7956 | \r\n
| Random Forest | \r\n0.7974 | \r\n
| k-means | \r\n0.7982 | \r\n
| Blended | \r\n0.7752 | \r\n
\r\n
Training time varied greatly from model to model - from around 5 seconds for k-means to a few hours for SVM (on a laptop with a 2GHz Intel CPU). The linear regression model with the most predictive power took under two minutes to train.
\r\nFor SVM, I used an RBF kernel with the same predictors that was published by the organizers. The results were mediocre and the performance was abysmal. Hopefully, Tony will divulge more details of his model and I will know what went wrong with mine.
\r\nThe linear regression model runs out of gas by the 45th event. Replacing all subsequent predictions with the prediction for event 45 returns the same score. Maybe the blended model retains its predictive power longer - I haven't checked.
\r\nI have a long list of ideas that I wanted to try, but never got to. At the same time, I don't think any of those ideas will result in a score anywhere close to Ildefons'. Maybe it's time to move on to something else. Hm... what is this CHALEARN thingy over\r\n there...?
", "votes": null, "replies": [] }, { "id": 8071, "author_name": "dejavu", "author_url": "", "post_date": "01/12/2012 07:06:24", "content": "[quote=Neil Thomas;8070]
\r\nI see references to day 1, day 2 etc. in the posts. How does one identify data from the same day? Just pick clumps of rows for a security that has the same number of trades on the previous day? Is the assumption that all securities were sampled on each day?
\r\n[/quote]
\r\nThe data prior to the initial testing set appears to be ordered by day, then by stock, then time of day. Although I didn't check, I am pretty certain that all stocks are sampled multiple times each day. There appears to be six days of data. The organizers\r\n could have done some randomization, but I would guess not as some statistics change dramatically after 'day 2'. Curious - how well did your public scores correlate?
", "votes": null, "replies": [] }, { "id": 8072, "author_name": "anlthms", "author_url": "", "post_date": "01/12/2012 07:39:20", "content": "[quote=Cole Harris;8071]
\r\nCurious - how well did your public scores correlate?
\r\n[/quote]
\r\nHere are the corresponding public scores:
\r\n| Model | \r\nPublic score | \r\nPrivate score | \r\n
| Linear Regression | \r\n0.7703 | \r\n0.7781 | \r\n
| kNN | \r\n0.7790 | \r\n0.7848 | \r\n
| SVM | \r\n0.7902 | \r\n0.7956 | \r\n
| Random Forest | \r\n0.7899 | \r\n0.7974 | \r\n
| k-means | \r\n0.7811 | \r\n0.7982 | \r\n
| Blended | \r\n0.7651 | \r\n0.7752 | \r\n
On the number of days topic. In looking at the data description it describes the ptcount and pvalue as the prior days trade count and value. I used a SQL query like this:\r\n
\r\nselect distinct securityid , pvalue , ptcount
\r\nFROM [kaggle].[dbo].[training]
\r\nwhere securityid = 1
and it returns 37 lines. Am I wrong to assume that this means the data came from 37 different days? or are you just referring to the last 50k lines or the testing data?
", "votes": null, "replies": [] }, { "id": 8074, "author_name": "ccccat", "author_url": "", "post_date": "01/12/2012 18:08:36", "content": "[quote=BarrenWuffet;8073]
\r\nOn the number of days topic. In looking at the data description it describes the ptcount and pvalue as the prior days trade count and value. I used a SQL query like this:
\r\nselect distinct securityid , pvalue , ptcount
\r\nFROM [kaggle].[dbo].[training]
\r\nwhere securityid = 1
and it returns 37 lines. Am I wrong to assume that this means the data came from 37 different days? or are you just referring to the last 50k lines or the testing data?
\r\n[/quote]
\r\nTry to do the same excluding last 50k lines. Last 50k lines were initial test set that was later incorporated into training set.
\r\n", "votes": null, "replies": [] }, { "id": 8075, "author_name": "ccccat", "author_url": "", "post_date": "01/12/2012 18:09:20", "content": "
[quote=Neil Thomas;8072]
\r\n[quote=Cole Harris;8071]
\r\nCurious - how well did your public scores correlate?
\r\n[/quote]
\r\nHere are the corresponding public scores:
\r\n| Model | \r\nPublic score | \r\nPrivate score | \r\n
| Linear Regression | \r\n0.7703 | \r\n0.7781 | \r\n
| kNN | \r\n0.7790 | \r\n0.7848 | \r\n
| SVM | \r\n0.7902 | \r\n0.7956 | \r\n
| Random Forest | \r\n0.7899 | \r\n0.7974 | \r\n
| k-means | \r\n0.7811 | \r\n0.7982 | \r\n
| Blended | \r\n0.7651 | \r\n0.7752 | \r\n
[/quote]
\r\n\r\n
How did you do blending?
", "votes": null, "replies": [] }, { "id": 8076, "author_name": "viveksha", "author_url": "", "post_date": "01/12/2012 19:01:42", "content": "Thanks to all for sharing interesting details. I don't think I was able to take advantage of the idiosyncrasies of the data or the similarities between subets of training data and the test set (to the extent that others might have). My third rank was due\r\n to a random forest model that scored 0.77400 on the private and 0.76479 on the public test set. The model was trained on all of the training data (except for the last 50K which I used as the cross validation set and didn't add back to the training set). I\r\n didn't have much luck with my linear regression models - with my best model scoring 0.785 on the test set. It might have been because all my models were on log returns even though the competition metric was RMSE. Simple weighted average with the linear regression\r\n model improved my score marginally to 0.773.
", "votes": null, "replies": [] }, { "id": 8083, "author_name": "anlthms", "author_url": "", "post_date": "01/13/2012 02:39:08", "content": "[quote=Sergey Yurgenson;8075]
\r\nHow did you do blending?
\r\n[/quote]
\r\nI had the individual models make predictions on held-out data. Linear regression was used to determine optimal weights for each model and these weights were used to blend the submissions. I believe the official term is "stacking".
", "votes": null, "replies": [] }, { "id": 8084, "author_name": "anlthms", "author_url": "", "post_date": "01/13/2012 03:00:06", "content": "[quote=vsh;8076]
\r\nMy third rank was due to a random forest model that scored 0.77400 on the private and 0.76479 on the public test set.
\r\n[/quote]
\r\nThat's a very good score for an individual model. How many trees did you use? My random forest model with 200 trees was slow to run, so I had it predict the prices at events 2, 10 and 50 and then followed up with a linear interpolation. Asking it to predict\r\n the price at event 50 was probably a tall order. It might have been better to predict up to event 20 or so and then set all subsequent prices to the price at that event. Did you check to see how far the model could predict?
", "votes": null, "replies": [] }, { "id": 8086, "author_name": "viveksha", "author_url": "", "post_date": "01/13/2012 05:15:05", "content": "Neil, I used 200 trees too. Although, with a lower sample size: 100K and larger nodesize: 100 than the defaults which reduced training time (and also increased predictability). For a single bid/ask it took 10 minutes to train on a single CPU core. I used\r\n the largest compute instance on Amazon EC2 (with 16 cores) to train the random forest models for every bid/ask - this took less than 2 hours in total. I also tried with 300 trees but it didn't make much difference.
\r\nI checked the predictability at different intervals: 55,65,75,85 and 95 and the random forests were better than my linear models at all those points. However, note that my linear regression models didn't score as well as yours. I trained using log(returns)\r\n instead of price so that different securities could be compared against each other - did you use similar transformations in your random forests? I think in general random forests should do better than linear regression almost always.
\r\nSince I was using normalized prices, I also noticed (too close to the deadline) that individual security models and models on price (as opposed to log(returns)) combined well with my random forest model. I wasn't able to take full advantage though.
\r\n- Vivek Sharma
", "votes": null, "replies": [] }, { "id": 8087, "author_name": "dejavu", "author_url": "", "post_date": "01/13/2012 06:08:59", "content": "Just curious, has anyone evaluated the potential profitability of application of their models?
", "votes": null, "replies": [] }, { "id": 8094, "author_name": "wcukierski", "author_url": "", "post_date": "01/13/2012 17:25:53", "content": "Cole, I have been thinking about this question and my own opinion is somewhat pessimistic. Let's put aside the (significant) practical challenges necessary to do HFT based on tick data and assume we can instantly enter/exit positions. Full disclosure:\r\n I am not a finance person and, as a grad student, have not seen finances since I found that $5 bill on the street the other week :)
\r\nWe aren't the market maker, so we don't get the privilege of seeing the liqiudity shock coming, nor did this contest assess to our ability to predict when the shocks are coming. That means the soonest we can react is at the t=51 time point, after the bid\r\n and ask have already gapped. We know from the naive baseline that the steady-state do-nothing model is "on average" (a loose, some would say wrong, interpretation of the RMSE) 86 pence away from the real bid/ask reaction, while the best contest models knocked\r\n about 10 pence off that. In my (possibly incorrect) interpretation of the situation, this is sort of an arbitrage window of about 10 pence when averaged over many trades.
\r\nIs that enough? I suppose a more thorough analysis that controls for the share price is necessary to really say. However, if we add back in real-world constraints and assume that other market participants have access to the same information (e.g. there\r\n was a large market sell of X shares T milliseconds ago), you have to assume that they are at least clever enough to run the linear regression that negates 9/10 of your forecasting advantage.
", "votes": null, "replies": [] }, { "id": 8095, "author_name": "barrenwuffet", "author_url": "", "post_date": "01/13/2012 18:36:02", "content": "A lot of the HFT stuff is based on hardware (collocation, dedicated fiber, burned chips, etc) and entity structuring (as broker/dealer in order to be market maker and collect rebates from ECNs (which from my limited knowledge is where most money is made\r\n as opposed to correctly picking direction)). That being said, there are plenty of places that would talk with you about implementation of your ideas on their hardware/communication platform such as JUMP, Tidal or any of the firms mentioned in the 'Trading'\r\n section of eFinancialCareers.com website.
", "votes": null, "replies": [] }, { "id": 8118, "author_name": "jc3626439", "author_url": "", "post_date": "01/16/2012 10:48:31", "content": "On the question of profitability it would be helpful if someone would tell us what are the typical financial arrangements between an exchange and a HFT organisation. I understand the exchange pays the HFT organisation for keeping the market "ticking over".\r\n They certainly could not afford to pay the brokerage of a retail trader.
", "votes": null, "replies": [] }, { "id": 8120, "author_name": "wcukierski", "author_url": "", "post_date": "01/16/2012 15:29:55", "content": "[quote=Capital Markets CRC;8117]
\r\nWilliam, if we remove the shackles of specifically looking at liquidity shocks, do you believe that the techniques discussed and developed in this competition would be useful for finding market anomalies and inefficiencies?
\r\n[/quote]
\r\nI think the liquidty shock actually frames the problem nicely. You have to set your time origin somewhere, and I think picking a larget market order is one way of isolating a timeframe where you expect\r\nsomething to happen. If we were instead given unregistered, raw tick data, we would have to modify the models to handle the many times where the bid/ask are not moving (or "random walking", or "market open mayhem", etc.). This modifciation could\r\n be as simple as a feature given to a decision tree (e.g. has there been a recent market order), but I suspect it would require more deliberate intervention. I attribute any success from the models developed in this competition to the fact that the market\r\n reacts systematically to a buy vs. sell order.
\r\nAn interesting offshoot of this observation: what would happen instead if you gave us t51...t100 bids/asks and asked us to classify whether the trade was buyer/seller initiatied? I suspect the accuracy would be very high, but the more interesting thing\r\n to look at would be the anomolous trades which we don't classify correctly. If they share common traits (a big "if"), then you can really get into the question of exploiting inefficiencies.
", "votes": null, "replies": [] }, { "id": 8127, "author_name": "ccccat", "author_url": "", "post_date": "01/16/2012 21:17:04", "content": "I would look on it not as a problem of finding market anomalies and inefficiencies but as a problem of pedicuring anomalies and inefficiencies. I would expect inefficiencies to be very short-lived and thus providing good time reference point. One can select\r\n one specific type of inefficiency and create dataset containing market data before inefficiency happened (obviously, with some data records when inefficiency did not happened). Then task will be to create classification model to predict future inefficiency\r\n using past market data (close analog - Credit competition)
", "votes": null, "replies": [] }, { "id": 8131, "author_name": "dejavu", "author_url": "", "post_date": "01/17/2012 05:10:45", "content": "@William
\r\n'The application of a model' is not trivial, even ignoring the technical issues. Turning a model prediction (derived from all liquidity events) into a trade signal may not be the best approach, which is why I asked the question. I don't think any of my contest\r\n "buy" models ever predicted a future bid beating the vwap (or initial ask), so they would break even by never trading:)
\r\nHowever it is much easier to directly address the question (with some assumptions) if I produced the liquidity shock, can I predict if I can get out at a profit in short order? Or, if I detect a liquidity shock, can I exploit the subsequent move in prices\r\n for some an identifiable subset? I find potential here.
", "votes": null, "replies": [] }, { "id": 8170, "author_name": "freshview", "author_url": "", "post_date": "01/19/2012 15:46:08", "content": "Will Ildefons algorithm be revealed and explained? Thank you.
", "votes": null, "replies": [] }, { "id": 52311, "author_name": "mlf26209010", "author_url": "", "post_date": "08/22/2014 14:53:14", "content": "", "votes": null, "replies": [] }, { "id": 81324, "author_name": "varunnagarajan", "author_url": "", "post_date": "06/09/2015 06:50:12", "content": "Hi Guys, I am a beginner and i am trying to learn how you guys got the solution. If anybody could please post the solution here, it would be helpful for me to understand how you guys did it. I use RStudio. Id the code you have written is in R, i would highly appreciate that. Tks
", "votes": null, "replies": [] }, { "id": 126003, "author_name": "zhaoqijie7", "author_url": "", "post_date": "07/05/2016 13:04:13", "content": "congrats , wish a future.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "7771": "", "7772": "", "7774": "", "7776": "", "7778": "", "7779": "", "7785": "", "7786": "", "7788": "", "7794": "", "7799": "", "7802": "", "7804": "", "7806": "", "7807": "", "7812": "", "7813": "", "7814": "", "7815": "", "7816": "", "8019": "", "8020": "", "8021": "", "8023": "", "8024": "", "8025": "", "8028": "", "8032": "", "8033": "", "8044": "", "8047": "", "8070": "", "8071": "", "8072": "", "8073": "", "8074": "", "8075": "", "8076": "", "8083": "", "8084": "", "8086": "", "8087": "", "8094": "", "8095": "", "8118": "", "8120": "", "8127": "", "8131": "", "8170": "", "52311": "", "81324": "", "126003": "congrats , wish a future.", "241939": "what is \"price changes\" ? \nI have checked for tick to tick price change (bid[n]-bid[n-1]) , but I am not getting a plot similar to what you have shared." }, "source": "meta" }