The various re-starts might play a role, but I look at the leaderboard and the type of .000x progress being made and I cannot help but thinking that there are still TWO LONG months to go... it feels like a lot of time.
", "messages": [ { "id": "59695", "postDate": "12/06/2014 21:20:01", "content": "The various re-starts might play a role, but I look at the leaderboard and the type of .000x progress being made and I cannot help but thinking that there are still TWO LONG months to go... it feels like a lot of time.
", "rawMarkdown": "", "votes": null }, { "id": "59701", "postDate": "12/06/2014 22:57:23", "content": "Giulio, you are quite pessimism on this competition. :) The leaderboard scores are improving and the number of teams are growing as well. I do believe that there will be more brilliant ideas appearing. Especially the CTR prediction involves time-series, which is more challenging and fun.
", "rawMarkdown": "", "votes": null }, { "id": "59708", "postDate": "12/07/2014 02:51:17", "content": "Yeah, you're right. This competition and I haven't clicked yet :-)
\nI just cannot get over the fact that since my original doubts, 12 days ago, there has been something like 0.0008 improvement on the leaderboard. Now, Kaggle has always been about the pips, but generally that's the type of progress you see in the last 10 days of a competition, not with 2 FULL MONTHS to go...
", "rawMarkdown": "", "votes": null }, { "id": "59713", "postDate": "12/07/2014 04:14:46", "content": "I agree with Giulio.
\nI don't think the ranking or over all point will change too much. 0.389 is the limit. This will be a close race.
", "rawMarkdown": "", "votes": null }, { "id": "59714", "postDate": "12/07/2014 04:23:23", "content": "shakeup may appear at the end :)
", "rawMarkdown": "", "votes": null }, { "id": "59716", "postDate": "12/07/2014 04:40:52", "content": "[quote=superfan123;59713]
\nI agree with Giulio.
\nI don't think the ranking or over all point will change too much. 0.389 is the limit. This will be a close race.
\n[/quote]
\nNot sure what the limit is, but I'm pretty sure we'll get lower than .389. How much lower, that's where I have doubts...
", "rawMarkdown": "", "votes": null }, { "id": "59719", "postDate": "12/07/2014 04:50:07", "content": "Interesting thing is online learning algorithm outperforms the other methods. I am wondering it is due to data scalability or intrinsic characters that are only able to be extracted by adaptive learning?
", "rawMarkdown": "", "votes": null }, { "id": "59721", "postDate": "12/07/2014 05:46:09", "content": "[quote=simeng;59719]
\nInteresting thing is online learning algorithm outperforms the other methods. I am wondering it is due to data scalability or intrinsic characters that are only able to be extracted by adaptive learning?
\n[/quote]
\nI am not sure what you mean. There are no conceptual differences between batch learning and online learning, the difference lies in the implementation.
\nOn the topic of this thread: the reason why top participants' scores are so narrowly distributed and so slow to improve is because everybody is using FTRL-proximal, which was state of the art as of the beginning of the competition. So the problem lies not with the competition, but with the lack of imagination of the entrants.
\nSo now we wait for somebody to come up with a different and transformationally better approach, and distance himself from the current top entries. And believe me, chances are it will happen.
\nThis is why two more months are required: to give us the time to come up with something new. That's the entire *point*, you see? If the point just was for participants to brush up an implementation of the one approach that is already known to be the best, then sure, only two weeks would suffice. But then why would you even organize a competition to begin with?
\nSomething I will add: from experience, it is nearly impossible to win a competition with the approach that the majority of the top 50 entrants are using. It is quite intuitive: you can't do better if you're doing the same thing. For instance, take a competition where one extremely effective approach was widely known and (over) used: the Higgs competition and gradient boosting. Although the overwhelming majority of the top entrants were using the XGB lib, none of the 3 winning entries were based on it. The 4th entry only used it for a fraction of the model. There's a pattern there.
", "rawMarkdown": "", "votes": null }, { "id": "59731", "postDate": "12/07/2014 12:55:47", "content": "Well the competition and production are two different environment.
\nFTRL-proximal is very effective to take short term trend and update on the fly. That's very important for online production.
\nOn the other hand FM (the winning solution of last CTR competition) is more sophisticated and more accurate. Of cause it's more complex and need more tuning. No free lunch.
\nI expect FM based solution will win the competition again. But most commercial search engine will continue use FTRL-proximal based implementation.
\nBTW, anybody get lucky to play with FM model?
", "rawMarkdown": "", "votes": null }, { "id": "59739", "postDate": "12/07/2014 18:06:42", "content": "Great input, fchollet!
\n@Superfan123, I have similar thoughts as you, and that's exactly why I meant online and batch models. I guess I should keep trying as well as waiting with patience.
", "rawMarkdown": "", "votes": null }, { "id": "59741", "postDate": "12/07/2014 19:43:41", "content": "[quote=Giulio;59695]
\nThe various re-starts might play a role, but I look at the leaderboard and the type of .000x progress being made and I cannot help but thinking that there are still TWO LONG months to go... it feels like a lot of time.
\n[/quote]
\nI agree with Giulio. The criteo data are much richer than this one. There is even a lot of room to improve further for criteo's data after the completion. In this one, you can see many raws are the same.
", "rawMarkdown": "", "votes": null }, { "id": "59767", "postDate": "12/08/2014 07:59:33", "content": "[quote=Giulio;59695]
\nThe various re-starts might play a role, but I look at the leaderboard and the type of .000x progress being made and I cannot help but thinking that there are still TWO LONG months to go... it feels like a lot of time.
\n[/quote]
\nI completely agree. There isn't much room for improvement, we might as well end the competition right now. Should we like vote or smth for this?
", "rawMarkdown": "", "votes": null }, { "id": "59768", "postDate": "12/08/2014 08:09:22", "content": "[quote=Ivan Lobov;59767]
\n[quote=Giulio;59695]
\nThe various re-starts might play a role, but I look at the leaderboard and the type of .000x progress being made and I cannot help but thinking that there are still TWO LONG months to go... it feels like a lot of time.
\n[/quote]
\nI completely agree. There isn't much room for improvement, we might as well end the competition right now. Should we like vote or smth for this?
\n[/quote]
\ndoes this have anything to do with your being number 1 on the leaderboard right now?
\n;)
", "rawMarkdown": "", "votes": null }, { "id": "59775", "postDate": "12/08/2014 11:04:00", "content": "10 days ago, my internal scores were: 0.40xx on training set ; 0.42xx on validation set, no submission yet.
\nNow, 0.39xx on training set ; 0.40xx on validation set ; no submission yet
\nI expect to gain 0.02 before the end of this month , so let us continue wrangling and look at other competitions if you feel tired with this one :)
\n[quote=Ivan Lobov;59767]
\nI completely agree. There isn't much room for improvement, we might as well end the competition right now. Should we like vote or smth for this?
\n[/quote]
", "rawMarkdown": "", "votes": null }, { "id": "59778", "postDate": "12/08/2014 11:23:19", "content": "Yeah this is really long. That was my original impression when the competition has just started, restarted or rerestarted.
\nBUT I am in general against to modify the timeline (or rules) in the middle of the competition.
\n\nI haven't even started to work with the data yet because of the late deadline.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 59701, "author_name": "simengy82", "author_url": "", "post_date": "12/06/2014 22:57:23", "content": "Giulio, you are quite pessimism on this competition. :) The leaderboard scores are improving and the number of teams are growing as well. I do believe that there will be more brilliant ideas appearing. Especially the CTR prediction involves time-series, which is more challenging and fun.
", "votes": null, "replies": [] }, { "id": 59708, "author_name": "adjgiulio", "author_url": "", "post_date": "12/07/2014 02:51:17", "content": "Yeah, you're right. This competition and I haven't clicked yet :-)
\nI just cannot get over the fact that since my original doubts, 12 days ago, there has been something like 0.0008 improvement on the leaderboard. Now, Kaggle has always been about the pips, but generally that's the type of progress you see in the last 10 days of a competition, not with 2 FULL MONTHS to go...
", "votes": null, "replies": [] }, { "id": 59713, "author_name": "superfan123", "author_url": "", "post_date": "12/07/2014 04:14:46", "content": "I agree with Giulio.
\nI don't think the ranking or over all point will change too much. 0.389 is the limit. This will be a close race.
", "votes": null, "replies": [] }, { "id": 59714, "author_name": "herimanitra", "author_url": "", "post_date": "12/07/2014 04:23:23", "content": "shakeup may appear at the end :)
", "votes": null, "replies": [] }, { "id": 59716, "author_name": "adjgiulio", "author_url": "", "post_date": "12/07/2014 04:40:52", "content": "[quote=superfan123;59713]
\nI agree with Giulio.
\nI don't think the ranking or over all point will change too much. 0.389 is the limit. This will be a close race.
\n[/quote]
\nNot sure what the limit is, but I'm pretty sure we'll get lower than .389. How much lower, that's where I have doubts...
", "votes": null, "replies": [] }, { "id": 59719, "author_name": "simengy82", "author_url": "", "post_date": "12/07/2014 04:50:07", "content": "Interesting thing is online learning algorithm outperforms the other methods. I am wondering it is due to data scalability or intrinsic characters that are only able to be extracted by adaptive learning?
", "votes": null, "replies": [] }, { "id": 59721, "author_name": "fchollet", "author_url": "", "post_date": "12/07/2014 05:46:09", "content": "[quote=simeng;59719]
\nInteresting thing is online learning algorithm outperforms the other methods. I am wondering it is due to data scalability or intrinsic characters that are only able to be extracted by adaptive learning?
\n[/quote]
\nI am not sure what you mean. There are no conceptual differences between batch learning and online learning, the difference lies in the implementation.
\nOn the topic of this thread: the reason why top participants' scores are so narrowly distributed and so slow to improve is because everybody is using FTRL-proximal, which was state of the art as of the beginning of the competition. So the problem lies not with the competition, but with the lack of imagination of the entrants.
\nSo now we wait for somebody to come up with a different and transformationally better approach, and distance himself from the current top entries. And believe me, chances are it will happen.
\nThis is why two more months are required: to give us the time to come up with something new. That's the entire *point*, you see? If the point just was for participants to brush up an implementation of the one approach that is already known to be the best, then sure, only two weeks would suffice. But then why would you even organize a competition to begin with?
\nSomething I will add: from experience, it is nearly impossible to win a competition with the approach that the majority of the top 50 entrants are using. It is quite intuitive: you can't do better if you're doing the same thing. For instance, take a competition where one extremely effective approach was widely known and (over) used: the Higgs competition and gradient boosting. Although the overwhelming majority of the top entrants were using the XGB lib, none of the 3 winning entries were based on it. The 4th entry only used it for a fraction of the model. There's a pattern there.
", "votes": null, "replies": [] }, { "id": 59731, "author_name": "superfan123", "author_url": "", "post_date": "12/07/2014 12:55:47", "content": "Well the competition and production are two different environment.
\nFTRL-proximal is very effective to take short term trend and update on the fly. That's very important for online production.
\nOn the other hand FM (the winning solution of last CTR competition) is more sophisticated and more accurate. Of cause it's more complex and need more tuning. No free lunch.
\nI expect FM based solution will win the competition again. But most commercial search engine will continue use FTRL-proximal based implementation.
\nBTW, anybody get lucky to play with FM model?
", "votes": null, "replies": [] }, { "id": 59739, "author_name": "simengy82", "author_url": "", "post_date": "12/07/2014 18:06:42", "content": "Great input, fchollet!
\n@Superfan123, I have similar thoughts as you, and that's exactly why I meant online and batch models. I guess I should keep trying as well as waiting with patience.
", "votes": null, "replies": [] }, { "id": 59741, "author_name": "songgc", "author_url": "", "post_date": "12/07/2014 19:43:41", "content": "[quote=Giulio;59695]
\nThe various re-starts might play a role, but I look at the leaderboard and the type of .000x progress being made and I cannot help but thinking that there are still TWO LONG months to go... it feels like a lot of time.
\n[/quote]
\nI agree with Giulio. The criteo data are much richer than this one. There is even a lot of room to improve further for criteo's data after the completion. In this one, you can see many raws are the same.
", "votes": null, "replies": [] }, { "id": 59767, "author_name": "ivanlobov", "author_url": "", "post_date": "12/08/2014 07:59:33", "content": "[quote=Giulio;59695]
\nThe various re-starts might play a role, but I look at the leaderboard and the type of .000x progress being made and I cannot help but thinking that there are still TWO LONG months to go... it feels like a lot of time.
\n[/quote]
\nI completely agree. There isn't much room for improvement, we might as well end the competition right now. Should we like vote or smth for this?
", "votes": null, "replies": [] }, { "id": 59768, "author_name": "vvarkey", "author_url": "", "post_date": "12/08/2014 08:09:22", "content": "[quote=Ivan Lobov;59767]
\n[quote=Giulio;59695]
\nThe various re-starts might play a role, but I look at the leaderboard and the type of .000x progress being made and I cannot help but thinking that there are still TWO LONG months to go... it feels like a lot of time.
\n[/quote]
\nI completely agree. There isn't much room for improvement, we might as well end the competition right now. Should we like vote or smth for this?
\n[/quote]
\ndoes this have anything to do with your being number 1 on the leaderboard right now?
\n;)
", "votes": null, "replies": [] }, { "id": 59775, "author_name": "herimanitra", "author_url": "", "post_date": "12/08/2014 11:04:00", "content": "10 days ago, my internal scores were: 0.40xx on training set ; 0.42xx on validation set, no submission yet.
\nNow, 0.39xx on training set ; 0.40xx on validation set ; no submission yet
\nI expect to gain 0.02 before the end of this month , so let us continue wrangling and look at other competitions if you feel tired with this one :)
\n[quote=Ivan Lobov;59767]
\nI completely agree. There isn't much room for improvement, we might as well end the competition right now. Should we like vote or smth for this?
\n[/quote]
", "votes": null, "replies": [] }, { "id": 59778, "author_name": "gaborfodor", "author_url": "", "post_date": "12/08/2014 11:23:19", "content": "Yeah this is really long. That was my original impression when the competition has just started, restarted or rerestarted.
\nBUT I am in general against to modify the timeline (or rules) in the middle of the competition.
\n\nI haven't even started to work with the data yet because of the late deadline.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "59695": "", "59701": "", "59708": "", "59713": "", "59714": "", "59716": "", "59719": "", "59721": "", "59731": "", "59739": "", "59741": "", "59767": "", "59768": "", "59775": "", "59778": "" }, "source": "meta" }