{ "id": 1111, "title": "Milestone Entries and Reviews", "url": "/competitions/AlgorithmicTradingChallenge/discussion/1111", "author_name": "", "post_date": "2011-11-30T19:49:38.700Z", "votes": null, "comment_count": 42, "views": 30951, "content": "

Are milestone entry winners required to provide an academic paper like the HHP?

", "messages": [ { "id": "6878", "postDate": "11/30/2011 19:49:38", "content": "

Are milestone entry winners required to provide an academic paper like the HHP?

", "rawMarkdown": "", "votes": null }, { "id": "6890", "postDate": "12/01/2011 04:52:03", "content": "

I don't think so, HHP rules specify that but for this competition there are no such rules.

", "rawMarkdown": "", "votes": null }, { "id": "6896", "postDate": "12/01/2011 10:29:36", "content": "

Is the milestone prize awarded based on the public data (i.e. the 30% of the test data used for the public leaderboard) or on the private data (i.e. the other 70%)?\r\n

", "rawMarkdown": "", "votes": null }, { "id": "6900", "postDate": "12/01/2011 14:22:10", "content": "

[quote=Alec Stephenson;6896]

\r\n

Is the milestone prize awarded based on the public data (i.e. the 30% of the test data used for the public leaderboard) or on the private data (i.e. the other 70%)?

\r\n

[/quote]

\r\n

There was answer on related question in the "RMSE clarification" thread: "The milestone winner will be the contestant on top of the leaderboard as of the cutoff dates."

", "rawMarkdown": "", "votes": null }, { "id": "6916", "postDate": "12/02/2011 08:05:11", "content": "

In that case, great job Xiaoshi.
\r\nI'm going to have to step-up a bit to compete with all these clever young people!

", "rawMarkdown": "", "votes": null }, { "id": "7451", "postDate": "12/23/2011 06:39:47", "content": "

Congratulations, Alec!

\r\n

Quote from "Futurama" for the admin
\r\nHigh Priest: Your words guide us.
\r\nPriests: [chanting] We're dumb.

", "rawMarkdown": "", "votes": null }, { "id": "7452", "postDate": "12/23/2011 06:46:08", "content": "

I wonder what was the public score and rank of Alec on November 30

", "rawMarkdown": "", "votes": null }, { "id": "7453", "postDate": "12/23/2011 07:56:16", "content": "

[quote=Ali Hassaïne;7452]

\r\n

I wonder what was the public score and rank of Alec on November 30

\r\n

[/quote]

\r\n

Public score was the same as it is today because his most recent submission was on November 24. Rank was something like 5-10th (leader score was 0.76X) if I remember correctly

\r\n

Is everyone else overfitting (tuning their parameters with leaderboard scores etc.) or is it just variance with different datasets and methods?

", "rawMarkdown": "", "votes": null }, { "id": "7454", "postDate": "12/23/2011 08:31:24", "content": "

[quote=Ali Hassaïne;7452]

\r\n

I wonder what was the public score and rank of Alec on November 30

\r\n

[/quote]

\r\n

Here is a nice feature of the leaderboard, you can wind back time,

\r\n

http://www.kaggle.com/c/AlgorithmicTradingChallenge/Leaderboard?asOf=2011-11-30

", "rawMarkdown": "", "votes": null }, { "id": "7455", "postDate": "12/23/2011 10:46:21", "content": "

Competition Admin,

\r\n

In RMSE clarification thread, You stated "The milestone winner will be the contestant on top of the leaderboard as of the cutoff dates." I don't think such statement means another set of data from the leaderboard is used to determine the milestone prize,\r\n I even didn't bother to select the best 5 of my submissions which have overall good performance in the "Submissions" page, how can the rules changed after so many days passed?

", "rawMarkdown": "", "votes": null }, { "id": "7457", "postDate": "12/23/2011 13:53:05", "content": "

Hey Xiaoshi, I believe they were implicitly referring to the private leaderboard, though I do agree the language was not clear and the admins should have spoken up long ago, when people congratulated you in the forum.   Kaggle milestones have been judged\r\n on the private set in the past. For instance, in the HHP prize:
\r\n
\r\n"The milestone prizes will be ranked by the private leaderboard score (i.e. the score on the remaining 70% of the data) so it is possible that the private score ranking will be different than the public score ranking."

\r\n

Putting aside any personal factors,  it is in everyone's best interest that \r\nall prizes are judged based on the private set.  There are too many ways to game/tune/overfit on the public set (not saying you are doing this, just that in general this is why the private set exists... to see whose model is really the best).  Again though,\r\n such rules should be made explicit up front so that this confusion doesn't have to happen.

", "rawMarkdown": "", "votes": null }, { "id": "7458", "postDate": "12/23/2011 15:40:36", "content": "

Sorry but when saying "leaderboard" before the competition ends I see no reason to deem it as the private one since the one has no meaning before it becomes visual to all. In HHP prize it 's clearly stated it's private leaderboard so no other understanding\r\n can be made, actually if you check HHP rules, you can find quite the opposite thing about the understanding of "the leaderboard" without saying public or private:

\r\n

"The Data Sets are:

\r\n
- the "Training and Validation Data Set", which is to be used by Entrants to develop the algorithms that generate their Entries and evaluate the efficacy of their algorithms;\n- the "Feedback Data Set" which will be used to calculate standings on the Leaderboard (described in Rule 11 below); and\n- the "Scoring Data Set" which will be used to determine the winners of the Milestone Prizes and Grand Prize.\n
\r\n

"

\r\n

"If an Entrant does not designate five Grand Prize Entries by the Grand Prize Deadline, his/her/its five Entries with the lowest prediction score on the Leaderboard will be automatically designated for judging."

\r\n

"The Leaderboard scores will be determined using the Feedback Data Set and are for informational purposes only and will not be used to determine prize winners, except as described in Rule 10 above."

\r\n

You see, I see no reason to deem "leaderboard" under normal context before competition ends as the private one, the admin can also see what alegro and Alec Stephenson's understanding about his words from this exact thread. If there's one leaderboard they\r\n are referring to "implicitly" I find no clue anywhere to deem it as the private one rather than the public one. if they find our natural understanding is definitely not what he wants to say, why not make it clear ASAP rather than digging out the thread after\r\n so many days, after two milestone prize submission deadline have all passed?

\r\n

Don't misunderstand me, I am not saying using public one to determine milestone prize winner has any advantage than using private one, but when a rule is set there we must follow it, I see no reason to change it in such a way and after the deadline we've\r\n any chance to do anything. Do you think the current status is fair to one not picking 5 submissions based on overall performance? Do you think breaking rules after deadline without any notification or clue can be found by competitors is even better than using\r\n imperfect rules? I'd like they give a serious response about this issue

", "rawMarkdown": "", "votes": null }, { "id": "7459", "postDate": "12/23/2011 16:32:28", "content": "

Question: "How you will select milestone winners?"
\r\nAnswer: "The milestone winner will be the contestant on top of the leaderboard as of the cutoff dates."

\r\n

About what "implicit referring" you have talk? Question was about selection procedure. Are you see any word about this? Are you see "the leaderboard" word-combination? What reason you have to think that the definite article was used to point on unknown private\r\n leadeboard but not to public leaderboard that we have seen at time of the answer? In what you beleived at the congratulations time, why do not asked about clarification?

\r\n

It is apparent that judging on private data is better. But why you think ("Putting aside any personal factors") that accepting rule changing at any time will produce less confusion in a future than responsibility for given promises? May be mistakenly given,\r\n but they cost someone time and not small (looking on amount of submissions). In this case there was enough time to make clarification before the second milestone. I am sorry, but it looks like as full irresponsibility or intentional deception. How often in\r\n your environment someone orders a job, looks at the execution process and decides to not pay because his order was wrong?

\r\n

By the way there is upfront rule. From the Kaggle's "Terms & Conditions":

\r\n

"7.3.any leader board appearing in connection with a Competition is indicative only and makes no representations and creates no entitlements in relation to any Award;"

", "rawMarkdown": "", "votes": null }, { "id": "7461", "postDate": "12/23/2011 17:22:01", "content": "

Alegro, "implicit" means implied without stating it outright.  I was just giving one plausible explanation based on how the process has worked in other competitions and how (in the future) it ought to work.

\r\n

I am in no way affiliated with, representing, defending, or in any way justfying CRC's actions.  I'm bowing out of the conversation from here out; this is a matter for the competion host and Kaggle to resolve.  I don't want to put words in anybody's mouth\r\n or create any problems.

", "rawMarkdown": "", "votes": null }, { "id": "7463", "postDate": "12/23/2011 17:43:53", "content": "

Although I will stay out of this debate because I, like everyone else in this competition, have a strong vested interested in the outcome, something Xiaoshi said sparked my curiosity.  Are the scores on the private leaderboard that may be used to award the\r\n milestone prizes computed based on all of our entries, or on our selection(or lack thereof) of 5 "best" predictions?  It's not a major concern, but I ask this because I, like Xiaoshi, did not select 5 best entries prior to the milestone deadline.  Thanks.

\r\n

 

\r\n

P.S.   I also think that William should be banned from competing until he tells us what major revelation he had yesterday ;)  And I want it to be a good story, too, featuring falling apples and/or beams of light.

", "rawMarkdown": "", "votes": null }, { "id": "7464", "postDate": "12/23/2011 17:52:57", "content": "

Vikp,

\r\n

I don't know whether they select 5 or sth., since I think only the score on the leaderboard(the public leaderboard) matters for the first 2 milestone prizes. I even not submit some of my versions which are pretty good locallcy but derived(modified) from\r\n bad-score submissions since I thought they are only meaningful in stage 3 of this competition. I don't think there is anyone here think the milestone prizes are based on private data before the post is updated today.

", "rawMarkdown": "", "votes": null }, { "id": "7465", "postDate": "12/23/2011 18:16:06", "content": "

[quote=VikP;7463]

\r\n

And I want it to be a good story, too, featuring falling apples and/or beams of light.

\r\n

[/quote]

\r\n

Twas 3 nights before Christmas, when all through the house
\r\nNot a creature was stirring, not even a spouse.
\r\nThe regressions were sent to the cpu with care,
\r\nIn hopes that  the bid-ask spread would soon be there...

", "rawMarkdown": "", "votes": null }, { "id": "7466", "postDate": "12/23/2011 18:28:53", "content": "

When I first read about the milestone price, I just assumed they meant the private leaderboard. But after reading this thread, I agree it can be read as the public one too.

\r\n

But, why did it take the competition admin so long to announce the winner ? Especially when they monitored this forum and posted on other threads and people have publicly congratualated Xiaoshi ? I thought he had quietly got his cheque already !

", "rawMarkdown": "", "votes": null }, { "id": "7468", "postDate": "12/23/2011 20:20:20", "content": "

[quote=B Yang;7466]

\r\n

But, why did it take the competition admin so long to announce the winner ? Especially when they monitored this forum and posted on other threads and people have publicly congratualated Xiaoshi ? I thought he had quietly got his cheque already !

\r\n

[/quote]

\r\n

That confuses me too. Apparently, machinery was not set up for milestone prize based on private leaderboard (long lag).  And no timely clarification was made. That may mean anything. Unfortunately it may be indication  that it was not the original plan and\r\n rules were changed in the middle of the competition.  Some clarifications from Kaggle would be nice.

", "rawMarkdown": "", "votes": null }, { "id": "7469", "postDate": "12/23/2011 21:08:56", "content": "

Let Capital Markets CRC cut a check to both Xiaoshi and Alec. Problem solved.

", "rawMarkdown": "", "votes": null }, { "id": "7470", "postDate": "12/23/2011 21:10:23", "content": "

William, "how (in the future) it ought to work" - In my opinion revealing hold out scores (by any way) of/for a part of contestants during a contest is not right decision. This contradicts to principle of equal competition conditions for all.

", "rawMarkdown": "", "votes": null }, { "id": "7472", "postDate": "12/23/2011 21:47:39", "content": "

Put yourself in Xiaoshi's shoes - you see your name at the top of the board when the milestone deadline comes and then weeks later it's announced that someone else based on a secret ranking has won the milestone prize.

\r\n

It does not pass the smell test whether it was implicit in the rules or not. If it was the original intent then it should have been made abundantly clear. Also, the private leaderboard should have been revealed for all to see.

\r\n

I like Neil's suggestion to cut a check to both Xiaoshi and Alec.

", "rawMarkdown": "", "votes": null }, { "id": "7477", "postDate": "12/23/2011 22:59:47", "content": "

[quote=Capital Markets CRC;7476]

\r\n

Regarding the posts surrounding the award of the November 30 prize we are not ignoring the proverbial elephant in the room we will have a response soon.

\r\n

[/quote]

\r\n

Actually the elephant is in both rooms. :)

", "rawMarkdown": "", "votes": null }, { "id": "7479", "postDate": "12/23/2011 23:04:54", "content": "

[quote=B Yang;7477]

\r\n

[quote=Capital Markets CRC;7476]

\r\n

Regarding the posts surrounding the award of the November 30 prize we are not ignoring the proverbial elephant in the room we will have a response soon.

\r\n

[/quote]

\r\n

Actually the elephant is in both rooms. :)

\r\n

[/quote]

\r\n*grabs popcorn*", "rawMarkdown": "", "votes": null }, { "id": "7480", "postDate": "12/23/2011 23:08:45", "content": "

It's not an issue for a single milestone prize, it's for both.

\r\n

Another thing:
\r\nThe current way makes the info unsymmetric. If alegro and Alec. knows their submissions get first in 70%-private data used finally we should also get the concrete placement. Please don't tell me it's even not that 70% of data is used to perform your milestone\r\n testing(Am I got too sensitive after this accident, but is there anything certain at all here?

\r\n

Edit:

\r\n

Alec. congrats me and admin annouce him as first milestone winner, then alegro says sth. strongly for me and then he is annouced as the second winner. So guys be hurry to send all your best bless to me this night, god will bless you the final $8000 winner.

", "rawMarkdown": "", "votes": null }, { "id": "7485", "postDate": "12/23/2011 23:48:37", "content": "

[quote=Capital Markets CRC;7482]

\r\n

[quote=Xiaoshi Lu;7480]

\r\n

The current way makes the info unsymmetric.

\r\n

[/quote]

\r\n

In the interests of full transparency and removal of information asymmetry both leaderboards will be published for public inspection

\r\n

[/quote]

\r\n

I've just made available the \r\nMilestone 1 and \r\nMilestone 2 private leaderboard ranks.

", "rawMarkdown": "", "votes": null }, { "id": "7487", "postDate": "12/24/2011 00:15:15", "content": "

Thanks Jeff.  Were the milestone leaderboards calculated on our 5 "best" predictions, as selected by Kaggle, or were all of our submissions taken into account?

", "rawMarkdown": "", "votes": null }, { "id": "7488", "postDate": "12/24/2011 00:32:43", "content": "

[quote=VikP;7487]

\r\n

Thanks Jeff.  Were the milestone leaderboards calculated on our 5 "best" predictions, as selected by Kaggle, or were all of our submissions taken into account?

\r\n

[/quote]

\r\n

If you had a submission(s) selected, they were used otherwise the remainder were picked on your public performance.

", "rawMarkdown": "", "votes": null }, { "id": "7489", "postDate": "12/24/2011 00:58:47", "content": "

@Xiaoshi: Happy to pass the prize on to you, just send me an email: I''m easy to find on the interwebs.

", "rawMarkdown": "", "votes": null }, { "id": "7490", "postDate": "12/24/2011 01:15:25", "content": "

Thx Alec. very moved to here that but I won't accept your offering anyway.

\r\n

When people around me asks me why you like to become a professional programming competitor(most on TopCoder rather than here), I say a big reason is that you earn based on the fact you are better without having to persuade anyone accepting your idea, and\r\n when you lose, you can always find the technology reason on yourself. So what I mostly care is the fair competition environment which makes me comfortable, this time most of my sorrow doesn't come from the lost of prize(yes still a bit) but from the way of\r\n the lost and the fact that rules can be trampled even in programming competition. I don't like the issue resolved in such a funny way(at least pretty funny from my point of view).

\r\n

[Edit: God will bless you on the final $8000 for you kindness(see my last post)]

", "rawMarkdown": "", "votes": null }, { "id": "7491", "postDate": "12/24/2011 01:54:44", "content": "

[quote=Jeff Moser;7488]

\r\n

[quote=VikP;7487]

\r\n

Thanks Jeff.  Were the milestone leaderboards calculated on our 5 "best" predictions, as selected by Kaggle, or were all of our submissions taken into account?

\r\n

[/quote]

\r\n

If you had a submission(s) selected, they were used otherwise the remainder were picked on your public performance.

\r\n

[/quote]

\r\n

Jeff,

\r\n

Were the 5 best submissions averaged to determine the standing or did you just pick the best one among those 5 submissions?

", "rawMarkdown": "", "votes": null }, { "id": "7492", "postDate": "12/24/2011 03:05:34", "content": "

[quote=Neil Thomas;7491]

\r\n

Were the 5 best submissions averaged to determine the standing or did you just pick the best one among those 5 submissions?

\r\n

[/quote]

\r\n

We picked the best

", "rawMarkdown": "", "votes": null }, { "id": "7493", "postDate": "12/24/2011 05:09:17", "content": "

@Xiaoshi, @Alec, @Competition Host

\r\n

Xiaoshi, with regard to your rejection of the Alec's suggestion, I want to ask you. Would you like to accept splitting of two milestone prizes across 3 participants (you, Alec and me) in equal amounts? Of course, in a case if it is acceptable for all parties\r\n (Xiaoshi, Alec, Competition Host). I will be happy if this will be done on regular basis (without money transfers between the members).

\r\n

Joke: You need to less compete at the TopCoder and collaboration will not look so funny for you.

\r\n

(for guys who not know - any kind of collaboration at time of a contest is strictly prohibited at the TopCoder)

", "rawMarkdown": "", "votes": null }, { "id": "7495", "postDate": "12/24/2011 09:50:38", "content": "

I won't accept a single cent from you two for sure.

\r\n

Frankly, following my thinking it should not be competitors who suffer any lost in this case, if they publish result as you two are winners but the rules are interpreted(at least naturally and deemed by most people especially after their misleading clarification\r\n before the first milestone prize deadline) I'm the winners then they should pay both of the amounts(totally 4000) since they must follow their publications and they must also follow their rules, it's that simple.

\r\n

"Must" is said from my understanding of a competition with formal rules, ok, it's also simple that they are organizers, they are the resource holder so they can do whatever they want, I can do nothing about it except reevaluating the competition value for\r\n organizers who deem their rules in that attitude. Not devoting time to people not worthy working for is the sole thing what a competitor can choose, they have thousands competitors and don't need to care about my feeling, anyway.

", "rawMarkdown": "", "votes": null }, { "id": "7528", "postDate": "12/28/2011 02:12:54", "content": "

You can't hold a competition which just let the best model wins, you must define how submissions are judged for prizes. We've shown all the reasons that why public board is expected to be used based on all the info before 11/30 and even 12/22 but you can't\r\n provide a single clue we can follow before that date to realize there is a private leaderboard for milestone prize. In that case for anyone strggule for this prize he will only care about public scores rather than whether their submissions are "best" under\r\n the explanation you give now! (In the similar way, there can also be competitors struggle for the final prize from the very beginning so they can ignore public scores since that's not what they care)
\r\n
\r\nThe fact is not "It's unclear that private leaderboard is used for milestone prize", but "It's quite clear(at least to most competitors, and especially after the clarification in the forum and first milestone's deadline) public leaderboard is used for milestone\r\n prize".
\r\n
\r\nAgain, when discussing whether a submission is "best" we must first make the judging criteria clear. If public leaderabord is the criteria I will have a basic judgement I can make the best via efforts. If private leaderboard is the criteria, then for this competition\r\n in my mind who is the "best" is just uncontroallable(I tell you why after the competition as I say in the forum) so I won't take much time play it at all. It's the rules and your clarificaiton let me me believe you use public leaderabord as the criteria for\r\n milestone prize, and you switch the definition after we can do anything. You said "1. Have the rules been changed mid competiton? No" but you can't provide any clue to let people believe what you said is true.
\r\n
\r\nPlease note, rules are not what in your mind, but what you write and recoganized by all the participants, compare what you write to what your explanation you are giving now to check whether rules are changed, not what you are explaining now and what was in\r\n your mind!
\r\n
\r\nlogic Shared by most competitors:
\r\nUnderstanding rules -> Realize how much can be expected to be earned -> Make efforts to earn based on the rules
\r\n
\r\nBut the fact:
\r\nYou want a best apple but writes as you want a best orange -> I am taking all the time to find the best orange -> You ask me to pick from my oranges which most likely to be best under apple's criteria and compare your best apples to check whether giving me\r\n a supplementary payment. What you should do is check all things you are receiving(no matter apples or oranges) under the orange criteria, since that's what you write, although not what in your mind.
\r\n
\r\nAbout your understanding of my quote:
\r\nMy big reason to compete is I don't need to debate with others, but what you do and explain just force me to write debate text again and again. I am not good at debate but I am sure your whole logic is just incorrect. When things are unreasonable, you can't\r\n make it reasonable to people deeply understanding the idea no matter how good you are at debate and explanation.
\r\n
\r\n

", "rawMarkdown": "", "votes": null }, { "id": "7529", "postDate": "12/28/2011 05:05:30", "content": "

My 2 cents. I had interpreted the milestone prize evaluation criteria the same as Xiaoshi. I thought it not optimal, but reasoned the organizers might want to award mid-contest prizes while not divulging any private results. I think it is important to realize\r\n that the objective of the contestants is to win. The  'best model' is by definition the one that wins according to the organizers' definition of winning irrespective of the appropriateness of that definition.

", "rawMarkdown": "", "votes": null }, { "id": "7533", "postDate": "12/28/2011 14:46:42", "content": "

Looks like this discussion is at the dead end. And it is at the dead end because two sides do not want to resolve the situation. They want to prove they are right and opponent is wrong. It is not constructive approach.\r\n
\r\nlet's try to figure out what sides want.
\r\n1. Both sides want to save faces by not admitting they are wrong or made a mistake.
\r\n2. XL wants prize money
\r\n3. XL, probably, wants something he can put in his "resume".
\r\n4. I really do not know if extra $2K is important to CM CRC. It may be important if to allocate this $2K it is necessary to go to higher management, that was skeptical about the competition from the start. In reality $2K should not be big money for Australian\r\n corporation.
\r\nBoth Alec and algro were nobly willing to split prize money. let's start from that idea and officially announce two prizes for each milestone:
\r\n-Best result on public leaderboard
\r\n- Best result on private leaderboard
\r\nlet's assume that CM CRC will make everything possible to make each prize =$1K. However let's ask XL to agree to that solution before prize amount is known.
\r\nCurrent stubborn behavior of both sides is not good for their reputation. Or maybe I am wrong and nowadays reputation means nothing and money is everything? Or maybe both sides think that compromise is bad for their reputation?

", "rawMarkdown": "", "votes": null }, { "id": "7544", "postDate": "12/28/2011 20:46:11", "content": "

I have no reason to complain if they give the extra 2000 since I get everything expected(resume means nothing for me), but from the current discussion I see almost no hope that can be done. For every logic they are saying I can point out why it's incorrect\r\n but for my logic they are ignoring and not give enough responses(they are saying the best should win again and again rather than focus on the meaning of "the leaderboard" before the end of any Kaggle competition, the rules text, the clarification text, and\r\n why they are very silent before the deadline of two milestone prizes in this post)

\r\n

About reputation: compromise is used when there's no clue to determine who is right, but their current explanation just can't even persuade most of the outliers here, in my eyes I'm just struggle for the profit I should get, that's not anything from bad\r\n reputation. About their reputation - I don't think their current explanation is anything good for their reputation but it seems they think defending bad decision is even better than following rules for faces.

", "rawMarkdown": "", "votes": null }, { "id": "7548", "postDate": "12/28/2011 22:43:21", "content": "
\r\n
    \r\n
      \r\n
    1. Under "Public Leaderboard" it says "This leaderboard is calculated on approximately 30% of the test data so the final standings may be different."\r\n
    2. If you go to any expired competition and click on "Results" they all point toward the "Private Leaderboard". In the entire history of Kaggle competition, to our knowledge only the private leaderboard has been used in evaluation.\r\n
    3. If we were to judge on the public leaderboard then the concept of a private leaderboard which nobody can see doesn't really make a lot of sense. From this one might infer that leaderboard refers to the private leaderboard.\r\n
    4. On the issue of different criteria for the milestone prize and the final prize, focusing on the definition and rules as you wish to do we do not see anything in the wording\r\n
    \r\n
\r\n

1. so the "final standing" may be different. - We all know the final testing is performed on private data, this sentence has nothing to do with milestone prize's testing.

\r\n

2. We are not discussing the final result, we are discussing milestone prize. Following your logic I can say for any competition which is not ended when you click "Leaderboard" there is only one leaderboard which is the public leaderboard, in my limited\r\n memory only HHR competiton gives milestone prize and let's see what's specified in that competition's rules about leaderboard and milestone prize:

\r\n\r\n

Please visit http://www.kaggle.com/c/AlgorithmicTradingChallenge and see the "Leaderboard" at the left side of the page, do you see I'm still on the top of the leaderboard? That explains what's "Leaderboard"(before a competition ends), not "public leaderboard"\r\n or "private leaderboard". And only after a competition ends, that exact leaderboard will become the private leaderboard, for any competition\r\nnot ended yet, the leaderboard always means public leaderboard if not specified elsewhere.

\r\n

 

\r\n

In point 2 you are reference expired competition to support your idea, why you reference an\r\nexpired competitions to a ongoing competition, why you don't check the page of any ongoing competitions? This is an ongoing competition.

\r\n

3. Quite the opposite. The sense of private leaderboard is just picking the best model, which you always care most. However that's always done after the competition ends, before it ends private leaderboard's info must be hidden otherwise competitos can deduce\r\n certain info from that leaderboard, that just adds another reason why private leaderboard should\r\nNOT be used before the ending of the competition. Check Cole Harris's post for the explanation of this opinion.

\r\n

4. One is "winner" the other is "leader". Win is calculated on private leaderabord for every Kaggle competition, but lead is defined by the positisons in the leaderboard, in point 2 I've addressed all the points why we say there is only one leaderboard before\r\n the competition ends without specified elsewhere.

\r\n

 

\r\n

5. Please also check the forum post at http://www.kaggle.com/forums/t/827/reliving-the-leaderboards-of-the-past ,  compare the usage of "leaderboard", "private leaderboard" and "public leaderboard" in  Jeff Moser's mouth. When saying a leaderboard\r\nas of the date before competition ends, it always means public leaderboard.

\r\n

6. If you find our understanding is different than yours, why keep silent for this post so long, after all people think I get the first milestone prize?

\r\n

7. If private leaderboard is used for milestone prize, you must specify which sumissions are used for the "private judging", is the most recent one? submission getting best score in public leaderabord, or the submissions chosen by competitors and if the\r\n chosen ones how many can be chosen? Such info are all defined in HHR competition clearly but can't be found anywhere for this competition, without those info nobody will realize extra judging will be performed for milestone prize. Have you see VikP's confusion\r\n in this same thread about this issue?

\r\n

===============================

\r\n

"who is the "best" is just uncontroallable"

\r\n

When I say "uncontroallable", I means you can't ensure(control) the best model will win this competition. Details given after the competition.

\r\n

 

\r\n
", "rawMarkdown": "", "votes": null }, { "id": "7551", "postDate": "12/28/2011 23:44:54", "content": "

Hi All,

\r\n

I wanted to take a moment to clarify the purposes of the public and private leaderboards.

\r\n

Kaggle provides public leaderboards for two primary reasons:

\r\n\r\n

These public leaderboards have no bearing on any prize money that is awarded, since this would encourage poorer models that overfit the data. This is true for milestone prizes as well: if we provide competitors with an incentive to maximize their public\r\n leaderboard score, then the public leaderboard will no longer accurately represent the generalization performance of the model.

\r\n

The private leaderboard functions to estimate the generalization abilities of competitor’s models and to select the best-performing models.

\r\n

Thus, all prizes are based solely on the private leaderboard. This has been the case for every Kaggle competition thus far, including the milestone prizes on the Heritage Health Prize.

\r\n

We sincerely apologize for the confusion regarding the milestone prizes in this competition, and will do our best to ensure that it does not happen in the future.

", "rawMarkdown": "", "votes": null }, { "id": "7553", "postDate": "12/29/2011 01:37:10", "content": "

Ben,

\r\n

1. We are not discussing whether private or public leaderboard is better for picking out best model, we are discussing following the current rules written in this competition(and all the interactions in the forum) which one SHOULD used?

\r\n

2. AFAIK HHP is the sole competition here besides this one which has milestone prize, check how the rules there specify about milestone prize and check here, how can a competitor realize for this one the milestone prize is given in the same way with so different\r\n description??

\r\n

3. What's the expected behavior is one thing, what's in the actual rules and followed by competitors is anothor thing, what will be your feeling if you work for an order and finally they tell you the order is described incorrectly and reject to pay you money?

\r\n

4. If it's Kaggle's mistake who doesn't write the rules as your client expect then Kaggle should buy the order themselves. As a competitor I don't quite care about whether the issue is at Kaggle or at the client, I care about my legal benefit.

\r\n

5. If you believe the mistake is at me rather than you two sides, then try to give me enough reasons in the simiar way as I list above, again, tell how a competitor can realize private leaderboard will be used for milestone prize in this competition before\r\n 12/22 based on all the rules, forum interactions or all other info available, but not what should be done in your mind and what's expected in the future.

\r\n

Thx.

", "rawMarkdown": "", "votes": null }, { "id": "7593", "postDate": "12/31/2011 02:50:43", "content": "

[Edit: Received the email]

", "rawMarkdown": "", "votes": null }, { "id": "8257", "postDate": "01/25/2012 11:54:13", "content": "

Finally, I've to say sth. about what happen from then on.

\r\n

They told me they didn't have final say on the money and asked me to write a document describing what happened and submit to them, then they can send it to the one who is responsible for this issue. I did it 3 weeks ago, then there's no response at all.\r\n I asked them several times since then and each time they said "We are busy" or sth. like this.

\r\n

Can I tell this behavior as deception on purpose??

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 6890, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/01/2011 04:52:03", "content": "

I don't think so, HHP rules specify that but for this competition there are no such rules.

", "votes": null, "replies": [] }, { "id": 6896, "author_name": "alecstephenson", "author_url": "", "post_date": "12/01/2011 10:29:36", "content": "

Is the milestone prize awarded based on the public data (i.e. the 30% of the test data used for the public leaderboard) or on the private data (i.e. the other 70%)?\r\n

", "votes": null, "replies": [] }, { "id": 6900, "author_name": "alegro", "author_url": "", "post_date": "12/01/2011 14:22:10", "content": "

[quote=Alec Stephenson;6896]

\r\n

Is the milestone prize awarded based on the public data (i.e. the 30% of the test data used for the public leaderboard) or on the private data (i.e. the other 70%)?

\r\n

[/quote]

\r\n

There was answer on related question in the "RMSE clarification" thread: "The milestone winner will be the contestant on top of the leaderboard as of the cutoff dates."

", "votes": null, "replies": [] }, { "id": 6916, "author_name": "alecstephenson", "author_url": "", "post_date": "12/02/2011 08:05:11", "content": "

In that case, great job Xiaoshi.
\r\nI'm going to have to step-up a bit to compete with all these clever young people!

", "votes": null, "replies": [] }, { "id": 7451, "author_name": "alegro", "author_url": "", "post_date": "12/23/2011 06:39:47", "content": "

Congratulations, Alec!

\r\n

Quote from "Futurama" for the admin
\r\nHigh Priest: Your words guide us.
\r\nPriests: [chanting] We're dumb.

", "votes": null, "replies": [] }, { "id": 7452, "author_name": "ahassaine", "author_url": "", "post_date": "12/23/2011 06:46:08", "content": "

I wonder what was the public score and rank of Alec on November 30

", "votes": null, "replies": [] }, { "id": 7453, "author_name": "herrahuu", "author_url": "", "post_date": "12/23/2011 07:56:16", "content": "

[quote=Ali Hassaïne;7452]

\r\n

I wonder what was the public score and rank of Alec on November 30

\r\n

[/quote]

\r\n

Public score was the same as it is today because his most recent submission was on November 24. Rank was something like 5-10th (leader score was 0.76X) if I remember correctly

\r\n

Is everyone else overfitting (tuning their parameters with leaderboard scores etc.) or is it just variance with different datasets and methods?

", "votes": null, "replies": [] }, { "id": 7454, "author_name": "salimali", "author_url": "", "post_date": "12/23/2011 08:31:24", "content": "

[quote=Ali Hassaïne;7452]

\r\n

I wonder what was the public score and rank of Alec on November 30

\r\n

[/quote]

\r\n

Here is a nice feature of the leaderboard, you can wind back time,

\r\n

http://www.kaggle.com/c/AlgorithmicTradingChallenge/Leaderboard?asOf=2011-11-30

", "votes": null, "replies": [] }, { "id": 7455, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/23/2011 10:46:21", "content": "

Competition Admin,

\r\n

In RMSE clarification thread, You stated "The milestone winner will be the contestant on top of the leaderboard as of the cutoff dates." I don't think such statement means another set of data from the leaderboard is used to determine the milestone prize,\r\n I even didn't bother to select the best 5 of my submissions which have overall good performance in the "Submissions" page, how can the rules changed after so many days passed?

", "votes": null, "replies": [] }, { "id": 7457, "author_name": "wcukierski", "author_url": "", "post_date": "12/23/2011 13:53:05", "content": "

Hey Xiaoshi, I believe they were implicitly referring to the private leaderboard, though I do agree the language was not clear and the admins should have spoken up long ago, when people congratulated you in the forum.   Kaggle milestones have been judged\r\n on the private set in the past. For instance, in the HHP prize:
\r\n
\r\n"The milestone prizes will be ranked by the private leaderboard score (i.e. the score on the remaining 70% of the data) so it is possible that the private score ranking will be different than the public score ranking."

\r\n

Putting aside any personal factors,  it is in everyone's best interest that \r\nall prizes are judged based on the private set.  There are too many ways to game/tune/overfit on the public set (not saying you are doing this, just that in general this is why the private set exists... to see whose model is really the best).  Again though,\r\n such rules should be made explicit up front so that this confusion doesn't have to happen.

", "votes": null, "replies": [] }, { "id": 7458, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/23/2011 15:40:36", "content": "

Sorry but when saying "leaderboard" before the competition ends I see no reason to deem it as the private one since the one has no meaning before it becomes visual to all. In HHP prize it 's clearly stated it's private leaderboard so no other understanding\r\n can be made, actually if you check HHP rules, you can find quite the opposite thing about the understanding of "the leaderboard" without saying public or private:

\r\n

"The Data Sets are:

\r\n
- the "Training and Validation Data Set", which is to be used by Entrants to develop the algorithms that generate their Entries and evaluate the efficacy of their algorithms;\n- the "Feedback Data Set" which will be used to calculate standings on the Leaderboard (described in Rule 11 below); and\n- the "Scoring Data Set" which will be used to determine the winners of the Milestone Prizes and Grand Prize.\n
\r\n

"

\r\n

"If an Entrant does not designate five Grand Prize Entries by the Grand Prize Deadline, his/her/its five Entries with the lowest prediction score on the Leaderboard will be automatically designated for judging."

\r\n

"The Leaderboard scores will be determined using the Feedback Data Set and are for informational purposes only and will not be used to determine prize winners, except as described in Rule 10 above."

\r\n

You see, I see no reason to deem "leaderboard" under normal context before competition ends as the private one, the admin can also see what alegro and Alec Stephenson's understanding about his words from this exact thread. If there's one leaderboard they\r\n are referring to "implicitly" I find no clue anywhere to deem it as the private one rather than the public one. if they find our natural understanding is definitely not what he wants to say, why not make it clear ASAP rather than digging out the thread after\r\n so many days, after two milestone prize submission deadline have all passed?

\r\n

Don't misunderstand me, I am not saying using public one to determine milestone prize winner has any advantage than using private one, but when a rule is set there we must follow it, I see no reason to change it in such a way and after the deadline we've\r\n any chance to do anything. Do you think the current status is fair to one not picking 5 submissions based on overall performance? Do you think breaking rules after deadline without any notification or clue can be found by competitors is even better than using\r\n imperfect rules? I'd like they give a serious response about this issue

", "votes": null, "replies": [] }, { "id": 7459, "author_name": "alegro", "author_url": "", "post_date": "12/23/2011 16:32:28", "content": "

Question: "How you will select milestone winners?"
\r\nAnswer: "The milestone winner will be the contestant on top of the leaderboard as of the cutoff dates."

\r\n

About what "implicit referring" you have talk? Question was about selection procedure. Are you see any word about this? Are you see "the leaderboard" word-combination? What reason you have to think that the definite article was used to point on unknown private\r\n leadeboard but not to public leaderboard that we have seen at time of the answer? In what you beleived at the congratulations time, why do not asked about clarification?

\r\n

It is apparent that judging on private data is better. But why you think ("Putting aside any personal factors") that accepting rule changing at any time will produce less confusion in a future than responsibility for given promises? May be mistakenly given,\r\n but they cost someone time and not small (looking on amount of submissions). In this case there was enough time to make clarification before the second milestone. I am sorry, but it looks like as full irresponsibility or intentional deception. How often in\r\n your environment someone orders a job, looks at the execution process and decides to not pay because his order was wrong?

\r\n

By the way there is upfront rule. From the Kaggle's "Terms & Conditions":

\r\n

"7.3.any leader board appearing in connection with a Competition is indicative only and makes no representations and creates no entitlements in relation to any Award;"

", "votes": null, "replies": [] }, { "id": 7461, "author_name": "wcukierski", "author_url": "", "post_date": "12/23/2011 17:22:01", "content": "

Alegro, "implicit" means implied without stating it outright.  I was just giving one plausible explanation based on how the process has worked in other competitions and how (in the future) it ought to work.

\r\n

I am in no way affiliated with, representing, defending, or in any way justfying CRC's actions.  I'm bowing out of the conversation from here out; this is a matter for the competion host and Kaggle to resolve.  I don't want to put words in anybody's mouth\r\n or create any problems.

", "votes": null, "replies": [] }, { "id": 7463, "author_name": "vikpar", "author_url": "", "post_date": "12/23/2011 17:43:53", "content": "

Although I will stay out of this debate because I, like everyone else in this competition, have a strong vested interested in the outcome, something Xiaoshi said sparked my curiosity.  Are the scores on the private leaderboard that may be used to award the\r\n milestone prizes computed based on all of our entries, or on our selection(or lack thereof) of 5 "best" predictions?  It's not a major concern, but I ask this because I, like Xiaoshi, did not select 5 best entries prior to the milestone deadline.  Thanks.

\r\n

 

\r\n

P.S.   I also think that William should be banned from competing until he tells us what major revelation he had yesterday ;)  And I want it to be a good story, too, featuring falling apples and/or beams of light.

", "votes": null, "replies": [] }, { "id": 7464, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/23/2011 17:52:57", "content": "

Vikp,

\r\n

I don't know whether they select 5 or sth., since I think only the score on the leaderboard(the public leaderboard) matters for the first 2 milestone prizes. I even not submit some of my versions which are pretty good locallcy but derived(modified) from\r\n bad-score submissions since I thought they are only meaningful in stage 3 of this competition. I don't think there is anyone here think the milestone prizes are based on private data before the post is updated today.

", "votes": null, "replies": [] }, { "id": 7465, "author_name": "wcukierski", "author_url": "", "post_date": "12/23/2011 18:16:06", "content": "

[quote=VikP;7463]

\r\n

And I want it to be a good story, too, featuring falling apples and/or beams of light.

\r\n

[/quote]

\r\n

Twas 3 nights before Christmas, when all through the house
\r\nNot a creature was stirring, not even a spouse.
\r\nThe regressions were sent to the cpu with care,
\r\nIn hopes that  the bid-ask spread would soon be there...

", "votes": null, "replies": [] }, { "id": 7466, "author_name": "byang1", "author_url": "", "post_date": "12/23/2011 18:28:53", "content": "

When I first read about the milestone price, I just assumed they meant the private leaderboard. But after reading this thread, I agree it can be read as the public one too.

\r\n

But, why did it take the competition admin so long to announce the winner ? Especially when they monitored this forum and posted on other threads and people have publicly congratualated Xiaoshi ? I thought he had quietly got his cheque already !

", "votes": null, "replies": [] }, { "id": 7468, "author_name": "ccccat", "author_url": "", "post_date": "12/23/2011 20:20:20", "content": "

[quote=B Yang;7466]

\r\n

But, why did it take the competition admin so long to announce the winner ? Especially when they monitored this forum and posted on other threads and people have publicly congratualated Xiaoshi ? I thought he had quietly got his cheque already !

\r\n

[/quote]

\r\n

That confuses me too. Apparently, machinery was not set up for milestone prize based on private leaderboard (long lag).  And no timely clarification was made. That may mean anything. Unfortunately it may be indication  that it was not the original plan and\r\n rules were changed in the middle of the competition.  Some clarifications from Kaggle would be nice.

", "votes": null, "replies": [] }, { "id": 7469, "author_name": "anlthms", "author_url": "", "post_date": "12/23/2011 21:08:56", "content": "

Let Capital Markets CRC cut a check to both Xiaoshi and Alec. Problem solved.

", "votes": null, "replies": [] }, { "id": 7470, "author_name": "alegro", "author_url": "", "post_date": "12/23/2011 21:10:23", "content": "

William, "how (in the future) it ought to work" - In my opinion revealing hold out scores (by any way) of/for a part of contestants during a contest is not right decision. This contradicts to principle of equal competition conditions for all.

", "votes": null, "replies": [] }, { "id": 7472, "author_name": "sirguessalot", "author_url": "", "post_date": "12/23/2011 21:47:39", "content": "

Put yourself in Xiaoshi's shoes - you see your name at the top of the board when the milestone deadline comes and then weeks later it's announced that someone else based on a secret ranking has won the milestone prize.

\r\n

It does not pass the smell test whether it was implicit in the rules or not. If it was the original intent then it should have been made abundantly clear. Also, the private leaderboard should have been revealed for all to see.

\r\n

I like Neil's suggestion to cut a check to both Xiaoshi and Alec.

", "votes": null, "replies": [] }, { "id": 7477, "author_name": "byang1", "author_url": "", "post_date": "12/23/2011 22:59:47", "content": "

[quote=Capital Markets CRC;7476]

\r\n

Regarding the posts surrounding the award of the November 30 prize we are not ignoring the proverbial elephant in the room we will have a response soon.

\r\n

[/quote]

\r\n

Actually the elephant is in both rooms. :)

", "votes": null, "replies": [] }, { "id": 7479, "author_name": "sirguessalot", "author_url": "", "post_date": "12/23/2011 23:04:54", "content": "

[quote=B Yang;7477]

\r\n

[quote=Capital Markets CRC;7476]

\r\n

Regarding the posts surrounding the award of the November 30 prize we are not ignoring the proverbial elephant in the room we will have a response soon.

\r\n

[/quote]

\r\n

Actually the elephant is in both rooms. :)

\r\n

[/quote]

\r\n*grabs popcorn*", "votes": null, "replies": [] }, { "id": 7480, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/23/2011 23:08:45", "content": "

It's not an issue for a single milestone prize, it's for both.

\r\n

Another thing:
\r\nThe current way makes the info unsymmetric. If alegro and Alec. knows their submissions get first in 70%-private data used finally we should also get the concrete placement. Please don't tell me it's even not that 70% of data is used to perform your milestone\r\n testing(Am I got too sensitive after this accident, but is there anything certain at all here?

\r\n

Edit:

\r\n

Alec. congrats me and admin annouce him as first milestone winner, then alegro says sth. strongly for me and then he is annouced as the second winner. So guys be hurry to send all your best bless to me this night, god will bless you the final $8000 winner.

", "votes": null, "replies": [] }, { "id": 7485, "author_name": "jeffmoser", "author_url": "", "post_date": "12/23/2011 23:48:37", "content": "

[quote=Capital Markets CRC;7482]

\r\n

[quote=Xiaoshi Lu;7480]

\r\n

The current way makes the info unsymmetric.

\r\n

[/quote]

\r\n

In the interests of full transparency and removal of information asymmetry both leaderboards will be published for public inspection

\r\n

[/quote]

\r\n

I've just made available the \r\nMilestone 1 and \r\nMilestone 2 private leaderboard ranks.

", "votes": null, "replies": [] }, { "id": 7487, "author_name": "vikpar", "author_url": "", "post_date": "12/24/2011 00:15:15", "content": "

Thanks Jeff.  Were the milestone leaderboards calculated on our 5 "best" predictions, as selected by Kaggle, or were all of our submissions taken into account?

", "votes": null, "replies": [] }, { "id": 7488, "author_name": "jeffmoser", "author_url": "", "post_date": "12/24/2011 00:32:43", "content": "

[quote=VikP;7487]

\r\n

Thanks Jeff.  Were the milestone leaderboards calculated on our 5 "best" predictions, as selected by Kaggle, or were all of our submissions taken into account?

\r\n

[/quote]

\r\n

If you had a submission(s) selected, they were used otherwise the remainder were picked on your public performance.

", "votes": null, "replies": [] }, { "id": 7489, "author_name": "alecstephenson", "author_url": "", "post_date": "12/24/2011 00:58:47", "content": "

@Xiaoshi: Happy to pass the prize on to you, just send me an email: I''m easy to find on the interwebs.

", "votes": null, "replies": [] }, { "id": 7490, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/24/2011 01:15:25", "content": "

Thx Alec. very moved to here that but I won't accept your offering anyway.

\r\n

When people around me asks me why you like to become a professional programming competitor(most on TopCoder rather than here), I say a big reason is that you earn based on the fact you are better without having to persuade anyone accepting your idea, and\r\n when you lose, you can always find the technology reason on yourself. So what I mostly care is the fair competition environment which makes me comfortable, this time most of my sorrow doesn't come from the lost of prize(yes still a bit) but from the way of\r\n the lost and the fact that rules can be trampled even in programming competition. I don't like the issue resolved in such a funny way(at least pretty funny from my point of view).

\r\n

[Edit: God will bless you on the final $8000 for you kindness(see my last post)]

", "votes": null, "replies": [] }, { "id": 7491, "author_name": "anlthms", "author_url": "", "post_date": "12/24/2011 01:54:44", "content": "

[quote=Jeff Moser;7488]

\r\n

[quote=VikP;7487]

\r\n

Thanks Jeff.  Were the milestone leaderboards calculated on our 5 "best" predictions, as selected by Kaggle, or were all of our submissions taken into account?

\r\n

[/quote]

\r\n

If you had a submission(s) selected, they were used otherwise the remainder were picked on your public performance.

\r\n

[/quote]

\r\n

Jeff,

\r\n

Were the 5 best submissions averaged to determine the standing or did you just pick the best one among those 5 submissions?

", "votes": null, "replies": [] }, { "id": 7492, "author_name": "jeffmoser", "author_url": "", "post_date": "12/24/2011 03:05:34", "content": "

[quote=Neil Thomas;7491]

\r\n

Were the 5 best submissions averaged to determine the standing or did you just pick the best one among those 5 submissions?

\r\n

[/quote]

\r\n

We picked the best

", "votes": null, "replies": [] }, { "id": 7493, "author_name": "alegro", "author_url": "", "post_date": "12/24/2011 05:09:17", "content": "

@Xiaoshi, @Alec, @Competition Host

\r\n

Xiaoshi, with regard to your rejection of the Alec's suggestion, I want to ask you. Would you like to accept splitting of two milestone prizes across 3 participants (you, Alec and me) in equal amounts? Of course, in a case if it is acceptable for all parties\r\n (Xiaoshi, Alec, Competition Host). I will be happy if this will be done on regular basis (without money transfers between the members).

\r\n

Joke: You need to less compete at the TopCoder and collaboration will not look so funny for you.

\r\n

(for guys who not know - any kind of collaboration at time of a contest is strictly prohibited at the TopCoder)

", "votes": null, "replies": [] }, { "id": 7495, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/24/2011 09:50:38", "content": "

I won't accept a single cent from you two for sure.

\r\n

Frankly, following my thinking it should not be competitors who suffer any lost in this case, if they publish result as you two are winners but the rules are interpreted(at least naturally and deemed by most people especially after their misleading clarification\r\n before the first milestone prize deadline) I'm the winners then they should pay both of the amounts(totally 4000) since they must follow their publications and they must also follow their rules, it's that simple.

\r\n

"Must" is said from my understanding of a competition with formal rules, ok, it's also simple that they are organizers, they are the resource holder so they can do whatever they want, I can do nothing about it except reevaluating the competition value for\r\n organizers who deem their rules in that attitude. Not devoting time to people not worthy working for is the sole thing what a competitor can choose, they have thousands competitors and don't need to care about my feeling, anyway.

", "votes": null, "replies": [] }, { "id": 7528, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/28/2011 02:12:54", "content": "

You can't hold a competition which just let the best model wins, you must define how submissions are judged for prizes. We've shown all the reasons that why public board is expected to be used based on all the info before 11/30 and even 12/22 but you can't\r\n provide a single clue we can follow before that date to realize there is a private leaderboard for milestone prize. In that case for anyone strggule for this prize he will only care about public scores rather than whether their submissions are "best" under\r\n the explanation you give now! (In the similar way, there can also be competitors struggle for the final prize from the very beginning so they can ignore public scores since that's not what they care)
\r\n
\r\nThe fact is not "It's unclear that private leaderboard is used for milestone prize", but "It's quite clear(at least to most competitors, and especially after the clarification in the forum and first milestone's deadline) public leaderboard is used for milestone\r\n prize".
\r\n
\r\nAgain, when discussing whether a submission is "best" we must first make the judging criteria clear. If public leaderabord is the criteria I will have a basic judgement I can make the best via efforts. If private leaderboard is the criteria, then for this competition\r\n in my mind who is the "best" is just uncontroallable(I tell you why after the competition as I say in the forum) so I won't take much time play it at all. It's the rules and your clarificaiton let me me believe you use public leaderabord as the criteria for\r\n milestone prize, and you switch the definition after we can do anything. You said "1. Have the rules been changed mid competiton? No" but you can't provide any clue to let people believe what you said is true.
\r\n
\r\nPlease note, rules are not what in your mind, but what you write and recoganized by all the participants, compare what you write to what your explanation you are giving now to check whether rules are changed, not what you are explaining now and what was in\r\n your mind!
\r\n
\r\nlogic Shared by most competitors:
\r\nUnderstanding rules -> Realize how much can be expected to be earned -> Make efforts to earn based on the rules
\r\n
\r\nBut the fact:
\r\nYou want a best apple but writes as you want a best orange -> I am taking all the time to find the best orange -> You ask me to pick from my oranges which most likely to be best under apple's criteria and compare your best apples to check whether giving me\r\n a supplementary payment. What you should do is check all things you are receiving(no matter apples or oranges) under the orange criteria, since that's what you write, although not what in your mind.
\r\n
\r\nAbout your understanding of my quote:
\r\nMy big reason to compete is I don't need to debate with others, but what you do and explain just force me to write debate text again and again. I am not good at debate but I am sure your whole logic is just incorrect. When things are unreasonable, you can't\r\n make it reasonable to people deeply understanding the idea no matter how good you are at debate and explanation.
\r\n
\r\n

", "votes": null, "replies": [] }, { "id": 7529, "author_name": "dejavu", "author_url": "", "post_date": "12/28/2011 05:05:30", "content": "

My 2 cents. I had interpreted the milestone prize evaluation criteria the same as Xiaoshi. I thought it not optimal, but reasoned the organizers might want to award mid-contest prizes while not divulging any private results. I think it is important to realize\r\n that the objective of the contestants is to win. The  'best model' is by definition the one that wins according to the organizers' definition of winning irrespective of the appropriateness of that definition.

", "votes": null, "replies": [] }, { "id": 7533, "author_name": "ccccat", "author_url": "", "post_date": "12/28/2011 14:46:42", "content": "

Looks like this discussion is at the dead end. And it is at the dead end because two sides do not want to resolve the situation. They want to prove they are right and opponent is wrong. It is not constructive approach.\r\n
\r\nlet's try to figure out what sides want.
\r\n1. Both sides want to save faces by not admitting they are wrong or made a mistake.
\r\n2. XL wants prize money
\r\n3. XL, probably, wants something he can put in his "resume".
\r\n4. I really do not know if extra $2K is important to CM CRC. It may be important if to allocate this $2K it is necessary to go to higher management, that was skeptical about the competition from the start. In reality $2K should not be big money for Australian\r\n corporation.
\r\nBoth Alec and algro were nobly willing to split prize money. let's start from that idea and officially announce two prizes for each milestone:
\r\n-Best result on public leaderboard
\r\n- Best result on private leaderboard
\r\nlet's assume that CM CRC will make everything possible to make each prize =$1K. However let's ask XL to agree to that solution before prize amount is known.
\r\nCurrent stubborn behavior of both sides is not good for their reputation. Or maybe I am wrong and nowadays reputation means nothing and money is everything? Or maybe both sides think that compromise is bad for their reputation?

", "votes": null, "replies": [] }, { "id": 7544, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/28/2011 20:46:11", "content": "

I have no reason to complain if they give the extra 2000 since I get everything expected(resume means nothing for me), but from the current discussion I see almost no hope that can be done. For every logic they are saying I can point out why it's incorrect\r\n but for my logic they are ignoring and not give enough responses(they are saying the best should win again and again rather than focus on the meaning of "the leaderboard" before the end of any Kaggle competition, the rules text, the clarification text, and\r\n why they are very silent before the deadline of two milestone prizes in this post)

\r\n

About reputation: compromise is used when there's no clue to determine who is right, but their current explanation just can't even persuade most of the outliers here, in my eyes I'm just struggle for the profit I should get, that's not anything from bad\r\n reputation. About their reputation - I don't think their current explanation is anything good for their reputation but it seems they think defending bad decision is even better than following rules for faces.

", "votes": null, "replies": [] }, { "id": 7548, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/28/2011 22:43:21", "content": "
\r\n
    \r\n
      \r\n
    1. Under "Public Leaderboard" it says "This leaderboard is calculated on approximately 30% of the test data so the final standings may be different."\r\n
    2. If you go to any expired competition and click on "Results" they all point toward the "Private Leaderboard". In the entire history of Kaggle competition, to our knowledge only the private leaderboard has been used in evaluation.\r\n
    3. If we were to judge on the public leaderboard then the concept of a private leaderboard which nobody can see doesn't really make a lot of sense. From this one might infer that leaderboard refers to the private leaderboard.\r\n
    4. On the issue of different criteria for the milestone prize and the final prize, focusing on the definition and rules as you wish to do we do not see anything in the wording\r\n
    \r\n
\r\n

1. so the "final standing" may be different. - We all know the final testing is performed on private data, this sentence has nothing to do with milestone prize's testing.

\r\n

2. We are not discussing the final result, we are discussing milestone prize. Following your logic I can say for any competition which is not ended when you click "Leaderboard" there is only one leaderboard which is the public leaderboard, in my limited\r\n memory only HHR competiton gives milestone prize and let's see what's specified in that competition's rules about leaderboard and milestone prize:

\r\n\r\n

Please visit http://www.kaggle.com/c/AlgorithmicTradingChallenge and see the "Leaderboard" at the left side of the page, do you see I'm still on the top of the leaderboard? That explains what's "Leaderboard"(before a competition ends), not "public leaderboard"\r\n or "private leaderboard". And only after a competition ends, that exact leaderboard will become the private leaderboard, for any competition\r\nnot ended yet, the leaderboard always means public leaderboard if not specified elsewhere.

\r\n

 

\r\n

In point 2 you are reference expired competition to support your idea, why you reference an\r\nexpired competitions to a ongoing competition, why you don't check the page of any ongoing competitions? This is an ongoing competition.

\r\n

3. Quite the opposite. The sense of private leaderboard is just picking the best model, which you always care most. However that's always done after the competition ends, before it ends private leaderboard's info must be hidden otherwise competitos can deduce\r\n certain info from that leaderboard, that just adds another reason why private leaderboard should\r\nNOT be used before the ending of the competition. Check Cole Harris's post for the explanation of this opinion.

\r\n

4. One is "winner" the other is "leader". Win is calculated on private leaderabord for every Kaggle competition, but lead is defined by the positisons in the leaderboard, in point 2 I've addressed all the points why we say there is only one leaderboard before\r\n the competition ends without specified elsewhere.

\r\n

 

\r\n

5. Please also check the forum post at http://www.kaggle.com/forums/t/827/reliving-the-leaderboards-of-the-past ,  compare the usage of "leaderboard", "private leaderboard" and "public leaderboard" in  Jeff Moser's mouth. When saying a leaderboard\r\nas of the date before competition ends, it always means public leaderboard.

\r\n

6. If you find our understanding is different than yours, why keep silent for this post so long, after all people think I get the first milestone prize?

\r\n

7. If private leaderboard is used for milestone prize, you must specify which sumissions are used for the "private judging", is the most recent one? submission getting best score in public leaderabord, or the submissions chosen by competitors and if the\r\n chosen ones how many can be chosen? Such info are all defined in HHR competition clearly but can't be found anywhere for this competition, without those info nobody will realize extra judging will be performed for milestone prize. Have you see VikP's confusion\r\n in this same thread about this issue?

\r\n

===============================

\r\n

"who is the "best" is just uncontroallable"

\r\n

When I say "uncontroallable", I means you can't ensure(control) the best model will win this competition. Details given after the competition.

\r\n

 

\r\n
", "votes": null, "replies": [] }, { "id": 7551, "author_name": "benhamner", "author_url": "", "post_date": "12/28/2011 23:44:54", "content": "

Hi All,

\r\n

I wanted to take a moment to clarify the purposes of the public and private leaderboards.

\r\n

Kaggle provides public leaderboards for two primary reasons:

\r\n\r\n

These public leaderboards have no bearing on any prize money that is awarded, since this would encourage poorer models that overfit the data. This is true for milestone prizes as well: if we provide competitors with an incentive to maximize their public\r\n leaderboard score, then the public leaderboard will no longer accurately represent the generalization performance of the model.

\r\n

The private leaderboard functions to estimate the generalization abilities of competitor’s models and to select the best-performing models.

\r\n

Thus, all prizes are based solely on the private leaderboard. This has been the case for every Kaggle competition thus far, including the milestone prizes on the Heritage Health Prize.

\r\n

We sincerely apologize for the confusion regarding the milestone prizes in this competition, and will do our best to ensure that it does not happen in the future.

", "votes": null, "replies": [] }, { "id": 7553, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/29/2011 01:37:10", "content": "

Ben,

\r\n

1. We are not discussing whether private or public leaderboard is better for picking out best model, we are discussing following the current rules written in this competition(and all the interactions in the forum) which one SHOULD used?

\r\n

2. AFAIK HHP is the sole competition here besides this one which has milestone prize, check how the rules there specify about milestone prize and check here, how can a competitor realize for this one the milestone prize is given in the same way with so different\r\n description??

\r\n

3. What's the expected behavior is one thing, what's in the actual rules and followed by competitors is anothor thing, what will be your feeling if you work for an order and finally they tell you the order is described incorrectly and reject to pay you money?

\r\n

4. If it's Kaggle's mistake who doesn't write the rules as your client expect then Kaggle should buy the order themselves. As a competitor I don't quite care about whether the issue is at Kaggle or at the client, I care about my legal benefit.

\r\n

5. If you believe the mistake is at me rather than you two sides, then try to give me enough reasons in the simiar way as I list above, again, tell how a competitor can realize private leaderboard will be used for milestone prize in this competition before\r\n 12/22 based on all the rules, forum interactions or all other info available, but not what should be done in your mind and what's expected in the future.

\r\n

Thx.

", "votes": null, "replies": [] }, { "id": 7593, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/31/2011 02:50:43", "content": "

[Edit: Received the email]

", "votes": null, "replies": [] }, { "id": 8257, "author_name": "xiaoshilu", "author_url": "", "post_date": "01/25/2012 11:54:13", "content": "

Finally, I've to say sth. about what happen from then on.

\r\n

They told me they didn't have final say on the money and asked me to write a document describing what happened and submit to them, then they can send it to the one who is responsible for this issue. I did it 3 weeks ago, then there's no response at all.\r\n I asked them several times since then and each time they said "We are busy" or sth. like this.

\r\n

Can I tell this behavior as deception on purpose??

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "6878": "", "6890": "", "6896": "", "6900": "", "6916": "", "7451": "", "7452": "", "7453": "", "7454": "", "7455": "", "7457": "", "7458": "", "7459": "", "7461": "", "7463": "", "7464": "", "7465": "", "7466": "", "7468": "", "7469": "", "7470": "", "7472": "", "7477": "", "7479": "", "7480": "", "7485": "", "7487": "", "7488": "", "7489": "", "7490": "", "7491": "", "7492": "", "7493": "", "7495": "", "7528": "", "7529": "", "7533": "", "7544": "", "7548": "", "7551": "", "7553": "", "7593": "", "8257": "" }, "source": "meta" }