You say RMSEs are computed for bid and ask separately but you don't explain how you combine them afterwards. And then you say: "The winning model will be the one with the lowest cumulative RMSE across the entire prediction set." Cumulative means there\r\n is a sum going on, but that's clearly not what you're computing, so I assume you mean "the lowest average RMSE across the prediction set". So can we just get a formula of how you compute it?
\r\nTo make things precise, let B be the matrix of actual bids and Bpred matrix of predicted bids, we define A and Apred similarly. We have N observations so all matrices are dimensions N by 50.
\r\nThe evaluation mentions the RMSE will be computed separately for the bid and ask, so I assume that for observation i, RMSE_i=0.5\\sqrt{1/50*(\\sum_{j=1}^50 (B_{i,j}-Bpred_{i,j})^2)}+0.5\\sqrt{1/50*(\\sum_{j=1}^50 (A_{i,j}-Apred_{i,j})^2)} (in latex notation).
\r\nThen do we take the average over the all observations with RMSE=1/N\\sum_{i=1}^N RMSE_i?
\r\nOr is it that the RMSE is computed at each time slice for bid and asks separately, with something like:
\r\nRMSE_j=0.5\\sqrt{1/N*(\\sum_{i=1}^N (B_{i,j}-Bpred_{i,j})^2)}+0.5\\sqrt{1/N*(\\sum_{i=1}^N (A_{i,j}-Apred_{i,j})^2)}
\r\nand RMSE=1/50\\sum_{j=1}^50 RMSE_j
\r\nThey won't be the same due to convexity of the square root.
", "messages": [ { "id": "6639", "postDate": "11/20/2011 19:56:58", "content": "You say RMSEs are computed for bid and ask separately but you don't explain how you combine them afterwards. And then you say: "The winning model will be the one with the lowest cumulative RMSE across the entire prediction set." Cumulative means there\r\n is a sum going on, but that's clearly not what you're computing, so I assume you mean "the lowest average RMSE across the prediction set". So can we just get a formula of how you compute it?
\r\nTo make things precise, let B be the matrix of actual bids and Bpred matrix of predicted bids, we define A and Apred similarly. We have N observations so all matrices are dimensions N by 50.
\r\nThe evaluation mentions the RMSE will be computed separately for the bid and ask, so I assume that for observation i, RMSE_i=0.5\\sqrt{1/50*(\\sum_{j=1}^50 (B_{i,j}-Bpred_{i,j})^2)}+0.5\\sqrt{1/50*(\\sum_{j=1}^50 (A_{i,j}-Apred_{i,j})^2)} (in latex notation).
\r\nThen do we take the average over the all observations with RMSE=1/N\\sum_{i=1}^N RMSE_i?
\r\nOr is it that the RMSE is computed at each time slice for bid and asks separately, with something like:
\r\nRMSE_j=0.5\\sqrt{1/N*(\\sum_{i=1}^N (B_{i,j}-Bpred_{i,j})^2)}+0.5\\sqrt{1/N*(\\sum_{i=1}^N (A_{i,j}-Apred_{i,j})^2)}
\r\nand RMSE=1/50\\sum_{j=1}^50 RMSE_j
\r\nThey won't be the same due to convexity of the square root.
", "rawMarkdown": "", "votes": null }, { "id": "6695", "postDate": "11/22/2011 16:17:14", "content": "[quote=Capital Markets CRC;6691]
\r\nHi thrasibule, RMSE is computed by Kaggle according to the following methodology
\r\nEach cell (i.e. a bid or ask price) is treated as a unique value. Then, we take the average of (solution-prediction)^2 and then finally take the square root of that.
\r\n[/quote]
\r\nIt's what I thought - it would not make sense to treat bid and ask values differently.
", "rawMarkdown": "", "votes": null }, { "id": "6704", "postDate": "11/22/2011 18:50:06", "content": "Alright, so with N stocks, let C is an N by 100 matrix (columns being bid51, ask51, ... bid100, ask100). Let Cpred be our prediction. Then you compute RMSE as follows:
\r\n(1) \\sqrt{1/(100*N)\\sum_{i,j} (C_{i,j}-Cpred_{i,j})^2}
\r\nor is it:
\r\n(2) 1/N\\sum_i\\sqrt{1/100\\sum_{j} (C_{i,j}-Cpred_{i,j})^2}
\r\nGiven a model, the optimum for (2) won't be the same at all as the optimum for (1), so I'd like to know exactly which is which.
\r\nIf I compute (1) on 50000 observations drawn at random from the training dataset using the naive estimator, I get RMSE=1.41, and using (2), I get RMSE=0.83. The RMSE you report on the test dataset for the naive estimator is 1.1, so there is something strange\r\n going on here.
", "rawMarkdown": "", "votes": null }, { "id": "6725", "postDate": "11/22/2011 23:52:37", "content": "Thanks, it's very clear now.
\r\nThis python code should compute the RMSE using the naive estimator on the entire training data:
\r\nimport math\r\n
fh = open("training.csv","r")
i=0
r=0
for line in fh:
if i==0:
headers = line.split(",")
else:
data = line.split(",")
naive_bid = float(data[headers.index("bid49")])
naive_ask = float(data[headers.index("ask49")])
for j in range(headers.index("bid51"),len(headers),2):
r+=(float(data[j])-naive_bid)**2
for j in range(headers.index("ask51"),len(headers),2):
r+=(float(data[j])-naive_ask)**2
i+=1
fh.close()
print "RMSE: {0}".format(math.sqrt(r/(100*i)))
I get 1.45, whereas you get 1.1 for the testing data. It's possible that the testing data is quite different than the training data, but still a bit odd.
", "rawMarkdown": "", "votes": null }, { "id": "6736", "postDate": "11/23/2011 03:41:50", "content": "I have reason to doubt whether your clarification is accurate, although I'm not quite sure.
\r\nIf the explanation and the actual scoring mechanism is inconsistent, will the explanation change or will the scoring mechanism reimplemented following the explanation?
\r\nAnyway, I hope the explanation is actually correct and it's me who are wrong.
", "rawMarkdown": "", "votes": null }, { "id": "6739", "postDate": "11/23/2011 04:37:37", "content": "The reason is similar to thrasibule's response, the error reflected in the leaderboard is too small under this mechanism compared to the expected normal range of the errors. If the scoring mechanism is exact the same as your clarification, then the testing\r\n data must be quite different with training data, OR the split of testing data(30% public - 70% private) is not done pure randomly, otherwise I can't explain the scores in the current leaderboard. Thx.
", "rawMarkdown": "", "votes": null }, { "id": "6779", "postDate": "11/25/2011 16:23:05", "content": "[quote=Capital Markets CRC;6741]
\r\nThe testing data does score differently from training. Training data is continuously sampled. Testing data has deliberate time gaps so that data in one row does not inadvertently reveal the solution to another. Because of this difference we would recommend\r\n making predictions at more granular level for example per stock or clustering by trade count.
\r\n[/quote]
\r\nSame inconsistency exists with scoring on last 50k training rows. They must be consistent with the test set sampling way because they come from the test rows of the first version of the data set.
\r\nUp to 20% of error (in naive approach on last 50k training rows) comes from security with security_id = 75. This security has big price values and is clear outlier in the set of 102 securities. It is hard to believe that presence of this huge outlier will\r\n help in selecting best approach to predict the "stock market's short-term response following large trades" (with stated error function).
\r\nHow you will select milestone winners?
", "rawMarkdown": "", "votes": null }, { "id": "6794", "postDate": "11/26/2011 08:30:24", "content": "> the entire training set was sampled in the same way
\r\nMy assumption about last 50k rows was based on your answers in other thread:
\r\n"Yes, the last testing dataset has simply been concatenated to the original training dataset.
\r\nYes, the current testing set was sampled from 'fresh' data in the same way as the last."
Did you changed the dataset a second time after this?
\r\n> The nature of the data means that outliers and anomalies will occur.
\r\n> In the end we're looking for an optimal model not a perfect model.
In case when the scoring error has value ~1.27 in a testing set (last 50k rows) with the security 75 (~500 rows) and value ~1.02 without this one your selection of the best approach will highly depend on quality of prediction of these 500 rows (will be\r\n quote random). While all errors per security form (rougly) sample from Gaussian distribution this one security makes error that stay at distance (rougly) of 5 standard deviations from the mean. This behaviour is not anomaly and defined by big price/spread/volatility\r\n values of this security in comparison with the remaining ones and quadric scoring function. Just add to that that same naive approach scored with error ~0.85 on the leaderboard and optimality will go to a second plane and lady Luck to the first. :)
\r\n> The milestone winner will be the contestant on top of the leaderboard as of the cutoff dates.
\r\nWhat are cutoff dates with times and timezone?
", "rawMarkdown": "", "votes": null }, { "id": "6827", "postDate": "11/28/2011 05:49:22", "content": "I obeserved the same thing, ~1.2 - 1.5
\r\nthe lead borad score is rediculously high compared to my train/ train test set score
\r\nI have no idea where the big diffrence comes from
\r\nI even doubt whether the score is really calculated the way we think it is
", "rawMarkdown": "", "votes": null }, { "id": "6908", "postDate": "12/02/2011 01:28:02", "content": "I agree with thrasibule, it would be nice if Kaggle can reveal how the score is actually calculated.\r\n
\r\nI applied the naive method to the testing set and I got a score of about 0.85xx; and then I applied the EXACTLY THE SAME METHOD to 50k rows randomly sampled from the training set, the range of RMSE is 1.24 ~ 1.34. (I hv tested my code for 1000 times, not\r\n a single case beats 0.85)
\r\nThis is really odd. please explain.
", "rawMarkdown": "", "votes": null }, { "id": "6912", "postDate": "12/02/2011 04:10:37", "content": "I feel very sad that nobody takes our comment serious and check why the score on the leader board is inconsistent with training.
\r\nI was in the other competeion of kaggle before and they made mistakes. And only at the very end, they found and admitted it is their mistake.
\r\nI feel this probabliby is just some normalization effect. Since the score on the leader board generally correlates with my own validatation test score.
\r\nAslo, if you submit all zeros, you could see you will get a score roughly 700. This is impossible for the test set so it probably is a clear demonstration that something is wrong. Even though it may not serious.
\r\nI may be wrong, but it is better to get clear.
", "rawMarkdown": "", "votes": null }, { "id": "6918", "postDate": "12/02/2011 09:43:36", "content": "[quote=woshialex;6912]
\r\nAslo, if you submit all zeros, you could see you will get a score roughly 700.
\r\n[/quote]
\r\nI did this couple minutes ago (with all values = 1e-6) and got score 1430.79
", "rawMarkdown": "", "votes": null }, { "id": "6922", "postDate": "12/02/2011 17:11:40", "content": "Sorry about that, I submitted a score with 780 because I made something wrong and I thought they are just equivalent to zero. Actually I have big values in that data file. So I was wrong. Thanks for verification.
", "rawMarkdown": "", "votes": null }, { "id": "7242", "postDate": "12/16/2011 18:05:50", "content": "Something about the dataset strikes me as odd. From the earlier posts, it appears that the last 50K lines in the training set should make a good cross validation set as it was sampled using the same method that was used for the testing set. However, this\r\n cross validation set seems to have substantially different characteristics from the testing set. For starters, the naive benchmark of predicting the prices at events 51 to 100 as the price at event 50 results in an RMSE of 1.2695. The RMSE for the same benchmark\r\n is much lower on the testing set. Can someone confirm this?
\r\nMoreover, the training set seems more similar to the aforementioned cross validation set than the testing set. After making a few improvements to my prediction algorithm, I was able to confirm the accuracy gain by testing against the cross validation set.\r\n However, the RMSE on the testing set worsened.
\r\nThe upshot of all this is that I cannot gauge the effect of a code tweak other than by making a submission to Kaggle. The key to this competition may lie in two things:
\r\n1) Coming up with a cross validation set that can act as a reliable proxy for the test set.
\r\n2) Filtering the training set so that it has the same properties as the test set.
[quote=Capital Markets CRC;6914]
\r\nThe original testing dataset followed the same sampling method. However, we soon discovered that it would be possible to stitch together overlapping event windows to find solutions without developing a model.
\r\nFor this reason a fresh testing dataset was created, which included a filter to ensure no overlapping events. An unintended consequence of applying this procedure is a reduction in the incidence of large stocks in the testing dataset.
\r\nSince the market response is expected to be different for large stocks versus small stocks, we believe this is the most likely explanation for the difference in RMSEs between the two datasets.
\r\n[/quote]
\r\nFair enough, that could explain the difference in RMSE between the training and testing sets. However, it doesn't explain the difference between the old and new testing sets. Weren't both the testing sets sampled the same way? If that's the case, why do\r\n they score so differently?
\r\nWhy don't you spend 5 minutes and do this experiment... Take your new testing set and set all predictions to the corresponding prices at event #50. Compute the RMSE of your predictions since you know the actual answers. Submit your predictions to Kaggle\r\n and see if the system returns a score that is reasonable.
", "rawMarkdown": "", "votes": null }, { "id": "7356", "postDate": "12/20/2011 02:32:26", "content": "Hello Capital Markets CRC,
\r\nPlease see my post above. Were you able to verify Kaggle's scoring system for this competition? If you are not planning to, for whatever reason, let us know that as well. If this competition is a waste of everyone's time, I would like to know sooner than\r\n later.
", "rawMarkdown": "", "votes": null }, { "id": "7398", "postDate": "12/21/2011 06:14:25", "content": "[quote=Capital Markets CRC;7390]
\r\nHi Neil, the scoring system has been verified. That along with other issues are discussed here
\r\n
\r\nhttp://www.kaggle.com/c/AlgorithmicTradingChallenge/forums/t/1178/kaggle-please-check-your-scoring-system
[/quote]
\r\nHi Capital Markets CRC,
\r\nThanks for taking the trouble to verify the scoring system. This restores some amount of faith in this competition. You chose not to answer the question about the difference between the old and new test sets. The benchmark score for the new test set is 0.85\r\n while for the old one it is 1.27. I guess it's up to the contestants to solve this mystery.
\r\nI will continue to pull my hair out...
", "rawMarkdown": "", "votes": null }, { "id": "7808", "postDate": "01/09/2012 16:21:08", "content": "[quote=Capital Markets CRC;7400]From the chart we can see that the expected volatility between Jul 2011 and Aug 2011 has almost tripled from ~15% to ~45%. This has implications for the size of a liquidity shock and I suspect a naive benchmark would score\r\n differently for these two periods.
\r\n[/quote]
The only question is why we have not see this increased volatility in the first parts of the fragments (bid1/ask1...bid50/ask50)?
", "rawMarkdown": "", "votes": null }, { "id": "32669", "postDate": "10/25/2013 10:01:34", "content": "I like the hair wigs because they can help me to change the hair style frequently and there are a lot of cheap hair wigs for sale and the cost is even much than have my hair dyed or curled so I choose the hair wigs and I just get a Short Hair Wigs and suitable for me a lot.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 6695, "author_name": "sirguessalot", "author_url": "", "post_date": "11/22/2011 16:17:14", "content": "[quote=Capital Markets CRC;6691]
\r\nHi thrasibule, RMSE is computed by Kaggle according to the following methodology
\r\nEach cell (i.e. a bid or ask price) is treated as a unique value. Then, we take the average of (solution-prediction)^2 and then finally take the square root of that.
\r\n[/quote]
\r\nIt's what I thought - it would not make sense to treat bid and ask values differently.
", "votes": null, "replies": [] }, { "id": 6704, "author_name": "thrasibule", "author_url": "", "post_date": "11/22/2011 18:50:06", "content": "Alright, so with N stocks, let C is an N by 100 matrix (columns being bid51, ask51, ... bid100, ask100). Let Cpred be our prediction. Then you compute RMSE as follows:
\r\n(1) \\sqrt{1/(100*N)\\sum_{i,j} (C_{i,j}-Cpred_{i,j})^2}
\r\nor is it:
\r\n(2) 1/N\\sum_i\\sqrt{1/100\\sum_{j} (C_{i,j}-Cpred_{i,j})^2}
\r\nGiven a model, the optimum for (2) won't be the same at all as the optimum for (1), so I'd like to know exactly which is which.
\r\nIf I compute (1) on 50000 observations drawn at random from the training dataset using the naive estimator, I get RMSE=1.41, and using (2), I get RMSE=0.83. The RMSE you report on the test dataset for the naive estimator is 1.1, so there is something strange\r\n going on here.
", "votes": null, "replies": [] }, { "id": 6725, "author_name": "thrasibule", "author_url": "", "post_date": "11/22/2011 23:52:37", "content": "Thanks, it's very clear now.
\r\nThis python code should compute the RMSE using the naive estimator on the entire training data:
\r\nimport math\r\n
fh = open("training.csv","r")
i=0
r=0
for line in fh:
if i==0:
headers = line.split(",")
else:
data = line.split(",")
naive_bid = float(data[headers.index("bid49")])
naive_ask = float(data[headers.index("ask49")])
for j in range(headers.index("bid51"),len(headers),2):
r+=(float(data[j])-naive_bid)**2
for j in range(headers.index("ask51"),len(headers),2):
r+=(float(data[j])-naive_ask)**2
i+=1
fh.close()
print "RMSE: {0}".format(math.sqrt(r/(100*i)))
I get 1.45, whereas you get 1.1 for the testing data. It's possible that the testing data is quite different than the training data, but still a bit odd.
", "votes": null, "replies": [] }, { "id": 6736, "author_name": "stevejackson", "author_url": "", "post_date": "11/23/2011 03:41:50", "content": "I have reason to doubt whether your clarification is accurate, although I'm not quite sure.
\r\nIf the explanation and the actual scoring mechanism is inconsistent, will the explanation change or will the scoring mechanism reimplemented following the explanation?
\r\nAnyway, I hope the explanation is actually correct and it's me who are wrong.
", "votes": null, "replies": [] }, { "id": 6739, "author_name": "stevejackson", "author_url": "", "post_date": "11/23/2011 04:37:37", "content": "The reason is similar to thrasibule's response, the error reflected in the leaderboard is too small under this mechanism compared to the expected normal range of the errors. If the scoring mechanism is exact the same as your clarification, then the testing\r\n data must be quite different with training data, OR the split of testing data(30% public - 70% private) is not done pure randomly, otherwise I can't explain the scores in the current leaderboard. Thx.
", "votes": null, "replies": [] }, { "id": 6779, "author_name": "alegro", "author_url": "", "post_date": "11/25/2011 16:23:05", "content": "[quote=Capital Markets CRC;6741]
\r\nThe testing data does score differently from training. Training data is continuously sampled. Testing data has deliberate time gaps so that data in one row does not inadvertently reveal the solution to another. Because of this difference we would recommend\r\n making predictions at more granular level for example per stock or clustering by trade count.
\r\n[/quote]
\r\nSame inconsistency exists with scoring on last 50k training rows. They must be consistent with the test set sampling way because they come from the test rows of the first version of the data set.
\r\nUp to 20% of error (in naive approach on last 50k training rows) comes from security with security_id = 75. This security has big price values and is clear outlier in the set of 102 securities. It is hard to believe that presence of this huge outlier will\r\n help in selecting best approach to predict the "stock market's short-term response following large trades" (with stated error function).
\r\nHow you will select milestone winners?
", "votes": null, "replies": [] }, { "id": 6794, "author_name": "alegro", "author_url": "", "post_date": "11/26/2011 08:30:24", "content": "> the entire training set was sampled in the same way
\r\nMy assumption about last 50k rows was based on your answers in other thread:
\r\n"Yes, the last testing dataset has simply been concatenated to the original training dataset.
\r\nYes, the current testing set was sampled from 'fresh' data in the same way as the last."
Did you changed the dataset a second time after this?
\r\n> The nature of the data means that outliers and anomalies will occur.
\r\n> In the end we're looking for an optimal model not a perfect model.
In case when the scoring error has value ~1.27 in a testing set (last 50k rows) with the security 75 (~500 rows) and value ~1.02 without this one your selection of the best approach will highly depend on quality of prediction of these 500 rows (will be\r\n quote random). While all errors per security form (rougly) sample from Gaussian distribution this one security makes error that stay at distance (rougly) of 5 standard deviations from the mean. This behaviour is not anomaly and defined by big price/spread/volatility\r\n values of this security in comparison with the remaining ones and quadric scoring function. Just add to that that same naive approach scored with error ~0.85 on the leaderboard and optimality will go to a second plane and lady Luck to the first. :)
\r\n> The milestone winner will be the contestant on top of the leaderboard as of the cutoff dates.
\r\nWhat are cutoff dates with times and timezone?
", "votes": null, "replies": [] }, { "id": 6827, "author_name": "woshialex", "author_url": "", "post_date": "11/28/2011 05:49:22", "content": "I obeserved the same thing, ~1.2 - 1.5
\r\nthe lead borad score is rediculously high compared to my train/ train test set score
\r\nI have no idea where the big diffrence comes from
\r\nI even doubt whether the score is really calculated the way we think it is
", "votes": null, "replies": [] }, { "id": 6908, "author_name": "yzhao0527", "author_url": "", "post_date": "12/02/2011 01:28:02", "content": "I agree with thrasibule, it would be nice if Kaggle can reveal how the score is actually calculated.\r\n
\r\nI applied the naive method to the testing set and I got a score of about 0.85xx; and then I applied the EXACTLY THE SAME METHOD to 50k rows randomly sampled from the training set, the range of RMSE is 1.24 ~ 1.34. (I hv tested my code for 1000 times, not\r\n a single case beats 0.85)
\r\nThis is really odd. please explain.
", "votes": null, "replies": [] }, { "id": 6912, "author_name": "woshialex", "author_url": "", "post_date": "12/02/2011 04:10:37", "content": "I feel very sad that nobody takes our comment serious and check why the score on the leader board is inconsistent with training.
\r\nI was in the other competeion of kaggle before and they made mistakes. And only at the very end, they found and admitted it is their mistake.
\r\nI feel this probabliby is just some normalization effect. Since the score on the leader board generally correlates with my own validatation test score.
\r\nAslo, if you submit all zeros, you could see you will get a score roughly 700. This is impossible for the test set so it probably is a clear demonstration that something is wrong. Even though it may not serious.
\r\nI may be wrong, but it is better to get clear.
", "votes": null, "replies": [] }, { "id": 6918, "author_name": "alegro", "author_url": "", "post_date": "12/02/2011 09:43:36", "content": "[quote=woshialex;6912]
\r\nAslo, if you submit all zeros, you could see you will get a score roughly 700.
\r\n[/quote]
\r\nI did this couple minutes ago (with all values = 1e-6) and got score 1430.79
", "votes": null, "replies": [] }, { "id": 6922, "author_name": "woshialex", "author_url": "", "post_date": "12/02/2011 17:11:40", "content": "Sorry about that, I submitted a score with 780 because I made something wrong and I thought they are just equivalent to zero. Actually I have big values in that data file. So I was wrong. Thanks for verification.
", "votes": null, "replies": [] }, { "id": 7242, "author_name": "anlthms", "author_url": "", "post_date": "12/16/2011 18:05:50", "content": "Something about the dataset strikes me as odd. From the earlier posts, it appears that the last 50K lines in the training set should make a good cross validation set as it was sampled using the same method that was used for the testing set. However, this\r\n cross validation set seems to have substantially different characteristics from the testing set. For starters, the naive benchmark of predicting the prices at events 51 to 100 as the price at event 50 results in an RMSE of 1.2695. The RMSE for the same benchmark\r\n is much lower on the testing set. Can someone confirm this?
\r\nMoreover, the training set seems more similar to the aforementioned cross validation set than the testing set. After making a few improvements to my prediction algorithm, I was able to confirm the accuracy gain by testing against the cross validation set.\r\n However, the RMSE on the testing set worsened.
\r\nThe upshot of all this is that I cannot gauge the effect of a code tweak other than by making a submission to Kaggle. The key to this competition may lie in two things:
\r\n1) Coming up with a cross validation set that can act as a reliable proxy for the test set.
\r\n2) Filtering the training set so that it has the same properties as the test set.
[quote=Capital Markets CRC;6914]
\r\nThe original testing dataset followed the same sampling method. However, we soon discovered that it would be possible to stitch together overlapping event windows to find solutions without developing a model.
\r\nFor this reason a fresh testing dataset was created, which included a filter to ensure no overlapping events. An unintended consequence of applying this procedure is a reduction in the incidence of large stocks in the testing dataset.
\r\nSince the market response is expected to be different for large stocks versus small stocks, we believe this is the most likely explanation for the difference in RMSEs between the two datasets.
\r\n[/quote]
\r\nFair enough, that could explain the difference in RMSE between the training and testing sets. However, it doesn't explain the difference between the old and new testing sets. Weren't both the testing sets sampled the same way? If that's the case, why do\r\n they score so differently?
\r\nWhy don't you spend 5 minutes and do this experiment... Take your new testing set and set all predictions to the corresponding prices at event #50. Compute the RMSE of your predictions since you know the actual answers. Submit your predictions to Kaggle\r\n and see if the system returns a score that is reasonable.
", "votes": null, "replies": [] }, { "id": 7356, "author_name": "anlthms", "author_url": "", "post_date": "12/20/2011 02:32:26", "content": "Hello Capital Markets CRC,
\r\nPlease see my post above. Were you able to verify Kaggle's scoring system for this competition? If you are not planning to, for whatever reason, let us know that as well. If this competition is a waste of everyone's time, I would like to know sooner than\r\n later.
", "votes": null, "replies": [] }, { "id": 7398, "author_name": "anlthms", "author_url": "", "post_date": "12/21/2011 06:14:25", "content": "[quote=Capital Markets CRC;7390]
\r\nHi Neil, the scoring system has been verified. That along with other issues are discussed here
\r\n
\r\nhttp://www.kaggle.com/c/AlgorithmicTradingChallenge/forums/t/1178/kaggle-please-check-your-scoring-system
[/quote]
\r\nHi Capital Markets CRC,
\r\nThanks for taking the trouble to verify the scoring system. This restores some amount of faith in this competition. You chose not to answer the question about the difference between the old and new test sets. The benchmark score for the new test set is 0.85\r\n while for the old one it is 1.27. I guess it's up to the contestants to solve this mystery.
\r\nI will continue to pull my hair out...
", "votes": null, "replies": [] }, { "id": 7808, "author_name": "alegro", "author_url": "", "post_date": "01/09/2012 16:21:08", "content": "[quote=Capital Markets CRC;7400]From the chart we can see that the expected volatility between Jul 2011 and Aug 2011 has almost tripled from ~15% to ~45%. This has implications for the size of a liquidity shock and I suspect a naive benchmark would score\r\n differently for these two periods.
\r\n[/quote]
The only question is why we have not see this increased volatility in the first parts of the fragments (bid1/ask1...bid50/ask50)?
", "votes": null, "replies": [] }, { "id": 32669, "author_name": "lolum02kicks", "author_url": "", "post_date": "10/25/2013 10:01:34", "content": "I like the hair wigs because they can help me to change the hair style frequently and there are a lot of cheap hair wigs for sale and the cost is even much than have my hair dyed or curled so I choose the hair wigs and I just get a Short Hair Wigs and suitable for me a lot.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "6639": "", "6695": "", "6704": "", "6725": "", "6736": "", "6739": "", "6779": "", "6794": "", "6827": "", "6908": "", "6912": "", "6918": "", "6922": "", "7242": "", "7315": "", "7356": "", "7398": "", "7808": "", "32669": "" }, "source": "meta" }