![\"olympics.jpg\"](\"https://i.postimg.cc/NM48MMP0/olympics.jpg\")
As there was already a discussion about this on Twitter and others have expressed similar concerns, it might make sense to discuss this topic here on Kaggle - specifically as early as possible.
\nFirst of all, I find the idea of this competition very cool. A lot of great ideas get first developed and researched on Kaggle, and there is sometimes a bit of a lack of public perception of these. So collecting and publishing them sounds like a good thing to do.
\nHowever, reviewing these will always need to follow some sort of typical research peer process. This competition is following a similar direction, by having peer review (fellow kagglers) and expert assessment (program chairs, GMs).
\nHowever, one of my reasons why I started Kaggling was to exactly move away from this subjective process. I always felt particularly intrigued by the fully objective metric-based assessment of my submissions. There is no human intervention when it is decided who is on top or not.
\nNow with awarding medals and points to this competition, I feel concerned about the integrity of the competition tier. Top Kagglers have spent countless hours of their life to work on competitive objective metric-based competitions and being judged by other kagglers, who you are actually competing with on the global leaderboard, feels a bit unnerving to me.
\nI know that there are different opinions about this, so happy to hear what others think. By the way, I think Kaggle should start their own scientific journal where people can submit and publish their work.
", "messages": [ { "id": 2256541, "postDate": "2023-05-12T15:02:56.027Z", "content": "As there was already a discussion about this on Twitter and others have expressed similar concerns, it might make sense to discuss this topic here on Kaggle - specifically as early as possible.
\nFirst of all, I find the idea of this competition very cool. A lot of great ideas get first developed and researched on Kaggle, and there is sometimes a bit of a lack of public perception of these. So collecting and publishing them sounds like a good thing to do.
\nHowever, reviewing these will always need to follow some sort of typical research peer process. This competition is following a similar direction, by having peer review (fellow kagglers) and expert assessment (program chairs, GMs).
\nHowever, one of my reasons why I started Kaggling was to exactly move away from this subjective process. I always felt particularly intrigued by the fully objective metric-based assessment of my submissions. There is no human intervention when it is decided who is on top or not.
\nNow with awarding medals and points to this competition, I feel concerned about the integrity of the competition tier. Top Kagglers have spent countless hours of their life to work on competitive objective metric-based competitions and being judged by other kagglers, who you are actually competing with on the global leaderboard, feels a bit unnerving to me.
\nI know that there are different opinions about this, so happy to hear what others think. By the way, I think Kaggle should start their own scientific journal where people can submit and publish their work.
", "rawMarkdown": "As there was already a discussion about this on Twitter and others have expressed similar concerns, it might make sense to discuss this topic here on Kaggle - specifically as early as possible.\n\nFirst of all, I find the idea of this competition very cool. A lot of great ideas get first developed and researched on Kaggle, and there is sometimes a bit of a lack of public perception of these. So collecting and publishing them sounds like a good thing to do.\n\nHowever, reviewing these will always need to follow some sort of typical research peer process. This competition is following a similar direction, by having peer review (fellow kagglers) and expert assessment (program chairs, GMs). \n\nHowever, one of my reasons why I started Kaggling was to exactly move away from this subjective process. I always felt particularly intrigued by the fully objective metric-based assessment of my submissions. There is no human intervention when it is decided who is on top or not.\n\nNow with awarding medals and points to this competition, I feel concerned about the integrity of the competition tier. Top Kagglers have spent countless hours of their life to work on competitive objective metric-based competitions and being judged by other kagglers, who you are actually competing with on the global leaderboard, feels a bit unnerving to me.\n\nI know that there are different opinions about this, so happy to hear what others think. By the way, I think Kaggle should start their own scientific journal where people can submit and publish their work.", "votes": 72 }, { "id": 2257992, "postDate": "2023-05-13T19:07:42.467Z", "content": "The competitions tier is the only Kaggle tier that isn’t based on upvotes. I’d hate to see more qualitative scoring seep into that tier and make the competitions KGM title more questionable. A writing contest really isn’t very similar to other Kaggle competitions. Giving out notebook medals would make more sense than competitions medals since people are submitting notebooks to be judged. The suggestion of a separate “Analytics KGM” tier to recognize analytics competitions also makes sense to me.
", "rawMarkdown": "The competitions tier is the only Kaggle tier that isn’t based on upvotes. I’d hate to see more qualitative scoring seep into that tier and make the competitions KGM title more questionable. A writing contest really isn’t very similar to other Kaggle competitions. Giving out notebook medals would make more sense than competitions medals since people are submitting notebooks to be judged. The suggestion of a separate “Analytics KGM” tier to recognize analytics competitions also makes sense to me.", "votes": 20, "replies": [ { "id": 2258137, "postDate": "2023-05-13T23:25:39.907Z", "content": "Indeed, giving notebook medals makes sense. But Analytics Tier awarding competition medals based on human evaluation will only transfer the issue. I have never seen, for example, a sports medal given to an athlete (at the Olympics, for instance) based on a human evaluation (that's how the founder of Kaggle Anthony was inspired when he created Kaggle, I read an article about this: Kaggle is \"born\" on the idea to make data science like sports). The same is true for chess games and math Olympiads (medals are not awarded based on human evaluation). Therefore, the issue goes beyond mixing analytics medals with leaderboard medals. Without getting into specifics, if someone has tried to participate in both analytics competitions and the Leaderboard ML competitions on Kaggle (at least once), they will sense that the platform and the seriousness are not the same (mainly the evaluation).
\nAdditionally, I don't like to discuss the increased likelihood, potential, and risk of gaming the system with the Analytics competitions tier (people already sell medals in leaderboard competitions, and kagglers suffered from that…). Adding an analytics competition tier (awarding medals based on human evaluation) just shifts the issue(not resolving it).
Yes, the move from objective to human evaluation for competitions (which will need to incorporate peer evaluation eventually to scale) is my concern. It would be nice to have one tier that is based totally on objectively scored contests.
\nI remember the early days of Kaggle when \"Making Data Science a Sport\" was the slogan. The Kaggle philosophy has shifted over the years from being a place to compete to being a place to collaborate and compete on data science projects. When the website started, there were only competitions with no overall rankings or user levels. Then they introduced user rankings and an overall leaderboard based on competitions performance only. If we wanted to dig into Kaggle history, the introduction of notebook/kernel and discussion tiers was not without controversy:
\nhttps://www.kaggle.com/discussions/general/22208
\nhttps://www.kaggle.com/discussions/general/22248
\n\nI have never seen, for example, a sports medal given to an athlete (at the Olympics, for instance) based on a human evaluation….
\n
![\"olympics2.jpg\"](\"https://i.postimg.cc/rmKW8JWf/olympics2.jpg\")
![\"horse.jpg\"](\"https://i.postimg.cc/gjpZvPQT/horse.jpg\")
![\"golas.webp\"](\"https://i.postimg.cc/659WnQbX/golas.webp\")
(Just for the first and second photo): They had to demonstrate their abilities and put in a lot of effort to qualify for this stage in front of the jury(=judges). For the rest, they just count on the strength of their hoof (Horces) and strategies of the riders, the same for Footballers: their strength, intelligence and talent(And the Video Assistant Referee VAR is to decide, in the event of a violation of the Rules: you can compare this to what Kaggle applies on the Leaderboard at the end of competition to eliminate cheaters, etc.)
\nThen, Aside from the \"mind\" and scientific competitions that I mentioned (Chess competitions and math Olympiads) , you may also have a look at long-distance races, sprints, swimmers, tennis, etc (Only their talent: no huamn Evaluation).
@dynamic24 Thanks for the link :), and for bringing me back to, as some kagglers called it, \"the good old times\" of kaggle that I do not know :)
\nAnd yes, I 100% agree (objective evaluation, far from a human evaluation).
\nFor the other two tiers (notebooks and discussions), even though most people today are aware that titles created solely based on upvotes are not always credible (of course, there are exceptions), the goal of creating these tiers was good : it was to encourage sharing and to help others grow along with them. Many Kagglers would not have been able to share all of their expertise and be as generous to the Kaggle community without the existence of these two tiers. Also, everyone is aware that the competition tier is the most credible (and I hope it stays that way). additionally, when kagglers share a notebook or an idea =discussion (and win medals based on upvotes), they are not doing so (at least explicitly) in a competitive manner (it is not a competition, so it is OK…). But when it comes with competitions, things change…
Also, there isn't a sport where one group of competitors are judged by humans, and another by objective metrics, especially when they are all competing in the same category.
\nThis is a great competition - but I think it makes more sense to move it to a separate Analytics tier, as other Kagglers have suggested already.
", "rawMarkdown": "Also, there isn't a sport where one group of competitors are judged by humans, and another by objective metrics, especially when they are all competing in the same category.\n\nThis is a great competition - but I think it makes more sense to move it to a separate Analytics tier, as other Kagglers have suggested already.\n\n\n", "votes": 2 }, { "id": 2258961, "postDate": "2023-05-14T15:12:55.877Z", "content": "Dear Abir, I respect your views.
\n\n\n(Just for the first and second photo): They had to demonstrate their abilities and put in a lot of effort to qualify for this stage in front of the jury(=judges).
\n
But I thought the issue was having a subjective judge (jury). That is why I pulled those pictures, i.e to present a case that there are Olympic sports where medals are awarded based on human judgment, even the seemingly objective sport such as football (more goals that opponents give you wins. Simple. right?) the subjective judgment of the referee can swing the outcome of any game. VAR isn't perfect either. Why? VAR is operated by humans. There were some high profile VAR errors in the English Premier League in the current season which the referee's body had to apologize for.
\nI understand people’s skepticism about panel judged ranking points [the original post seems to be pivoted around it] diluting the competition. However, one could argue that, while the individual competitions' LBs are metric-based and hence are bias free, the ranking points algorithm is not necessarily objective. It's an algorithm derived by a panel of kaggle team (experts).
", "rawMarkdown": "Dear Abir, I respect your views.\n> (Just for the first and second photo): They had to demonstrate their abilities and put in a lot of effort to qualify for this stage in front of the **jury(=judges).**\n\nBut I thought the issue was having a subjective judge (jury). That is why I pulled those pictures, i.e to present a case that there are Olympic sports where medals are awarded based on human judgment, even the seemingly objective sport such as football (more goals that opponents give you wins. Simple. right?) the subjective judgment of the referee can swing the outcome of any game. VAR isn't perfect either. Why? VAR is operated by humans. There were some high profile VAR errors in the English Premier League in the current season which [the referee's body](https://www.theguardian.com/football/2022/sep/04/premier-league-request-review-of-controversial-var-decisions) had to apologize for.\n\nI understand people’s skepticism about panel judged ranking points [the original post seems to be pivoted around it] diluting the competition. However, one could argue that, while the individual competitions' LBs are metric-based and hence are bias free, the ranking points algorithm is not necessarily objective. It's an algorithm derived by a panel of kaggle team (experts). \n", "votes": 3 }, { "id": 2259163, "postDate": "2023-05-14T18:13:51.343Z", "content": "@desalegngeb first, let me thank you for your nice response.
\nOk, you are right. I just wanted to point out that they are fundamentally uncomparable. Even with that, look at this (they're already complaining…). Soon, the AI will assign Olympic medals :) .AI_assign_olympic_medals_and_beyond
\nKaggle should be ahead of them: here is the house of data science and somewhat of AI :) . your last sentences are logical, but I think it's acceptable (I'm referring to the algo of Kaggle and objectivity…).
@rai555 yes, it is right, but to me, in this tier or a different one, it's the human evaluation (that awards medals) that doesn't make sense (I explained it well in my comments…), and I see that adding a different analytics tier doesn't actually address the issue but it just transfers it(…). But let's stick with this plan to add a new tier. There are competitors, and medals that will be awarded( regardless of how they are given out :)), so, these are also competitions, even analytics ones. Therefore, the current tier should be renamed, perhaps as the Leaderboard (ML, DL, RL)Competition tier, and the new tier would be called the Analytics Competition tier (although the two will have leaderboards :) ). (X will be Leadarboard(ML, DL, RL) Competition Grandmaster, and Y will be Analytics Competition Grandmaster, for example?) Seems odd :) I'm not sure how logical or practical(…) it will be. (Of course, this is, if the main issue (human evaluation, which awards medals) does not count, no matter what is going on there :)…)
", "rawMarkdown": "@rai555 yes, it is right, but to me, in this tier or a different one, it's the human evaluation (that awards medals) that doesn't make sense (I explained it well in my comments...), and I see that adding a different analytics tier doesn't actually address the issue but it just transfers it(...). But let's stick with this plan to add a new tier. There are competitors, and medals that will be awarded( regardless of how they are given out :)), so, these are also competitions, even analytics ones. Therefore, the current tier should be renamed, perhaps as the **Leaderboard (ML, DL, RL)Competition tier**, and the new tier would be called the **Analytics Competition tier** (although the two will have leaderboards :) ). (X will be Leadarboard(ML, DL, RL) Competition Grandmaster, and Y will be Analytics Competition Grandmaster, for example?) Seems odd :) I'm not sure how logical or practical(...) it will be. (Of course, this is, if the main issue (human evaluation, which awards medals) does not count, no matter what is going on there :)...)", "votes": 1 } ] } ] }, { "id": 2258436, "postDate": "2023-05-14T07:53:59.813Z", "content": "I'm 100% with psi on this. Essay writing is great, but I don't think an \"upvote\" (or equivalent) competition should count in the same way as a metric-based competition.
", "rawMarkdown": "I'm 100% with psi on this. Essay writing is great, but I don't think an \"upvote\" (or equivalent) competition should count in the same way as a metric-based competition.", "votes": 15 }, { "id": 2260057, "postDate": "2023-05-15T12:48:22.433Z", "content": "Kaggle wants to give medals and points because this is the only way to maximize participation. But they are precisely so precious because there were never subjective and everybody knows what they stand for. Mixing standard competitions with \"upvote-based\" mechanics in the same ranking will deteriorate the Kaggle competition GM brand/points. Let's create a separate analytics competitions tier.
", "rawMarkdown": "Kaggle wants to give medals and points because this is the only way to maximize participation. But they are precisely so precious because there were never subjective and everybody knows what they stand for. Mixing standard competitions with \"upvote-based\" mechanics in the same ranking will deteriorate the Kaggle competition GM brand/points. Let's create a separate analytics competitions tier. ", "votes": 13, "replies": [ { "id": 2261414, "postDate": "2023-05-16T10:01:49.950Z", "content": "\n\nthe only way to maximize participation
\n
Absolutely understandable, and here is the other side: there are some other ds/dl competition platforms. All of them are less popular then Kaggle. rn K have a certain level of trustworthiness / respect with their approach to competitions. If that will drift into something else, towards other platforms, why wont competitors drift to them too?
", "rawMarkdown": ">the only way to maximize participation\n\nAbsolutely understandable, and here is the other side: there are ~~a lot of~~ some other ds/dl competition platforms. All of them are less popular then Kaggle. rn K have a certain level of trustworthiness / respect with their approach to competitions. If that will drift into something else, towards other platforms, why wont competitors drift to them too?", "votes": 2 } ] }, { "id": 2257911, "postDate": "2023-05-13T18:09:23.660Z", "content": "I'd suggest making a separate \"analytics\" tier if there is going to be enough competitions of this type in the future.
", "rawMarkdown": "I'd suggest making a separate \"analytics\" tier if there is going to be enough competitions of this type in the future.", "votes": 13, "replies": [ { "id": 2258739, "postDate": "2023-05-14T12:51:46.290Z", "content": "great suggestion! unfortunately doesn't seem to be on the horizon that there would be enough competitions. would be interesting to have them though!
", "rawMarkdown": "great suggestion! unfortunately doesn't seem to be on the horizon that there would be enough competitions. would be interesting to have them though!", "votes": 2, "replies": [ { "id": 2259060, "postDate": "2023-05-14T16:58:06.903Z", "content": "There were already a few of these before. Maybe kaggle wants more of these.
", "rawMarkdown": "There were already a few of these before. Maybe kaggle wants more of these.", "votes": 1 } ] }, { "id": 2259301, "postDate": "2023-05-14T20:43:03.897Z", "content": "I agree with separate tier , This competition is “2023 Kaggle AI Report”, I guess we could expect at least once a year similar competition
", "rawMarkdown": "I agree with separate tier , This competition is “2023 Kaggle AI Report”, I guess we could expect at least once a year similar competition \n", "votes": 3 } ] }, { "id": 2262056, "postDate": "2023-05-16T17:52:00.140Z", "content": "The has been an official update here
\nhttps://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/410802
", "rawMarkdown": "The has been an official update here\n\nhttps://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/410802", "votes": 5, "replies": [ { "id": 2262145, "postDate": "2023-05-16T18:52:58.530Z", "content": "So now you can boost your ranking by getting points, but can't become a competition GM since they don't award medals. If there is not gonna be so many analytics comps to create a separate tier, this sounds like a decent compromise.
", "rawMarkdown": "So now you can boost your ranking by getting points, but can't become a competition GM since they don't award medals. If there is not gonna be so many analytics comps to create a separate tier, this sounds like a decent compromise.", "replies": [ { "id": 2262187, "postDate": "2023-05-16T19:16:01.363Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2256681, "postDate": "2023-05-12T16:47:03.310Z", "content": "My opinion: The very nature of this competition is subjective, since evaluating an essay involves humans and lacks an objective mathematical evaluation (as every other kaggle competition), therefore representing a new challenge.
\nOn the one hand, Kagglers may feel that an obtained medal from this (analytics) competition is not worth the same as a traditional one. On the other hand, participants also want a prize/recognition for their effort, which also sounds completely understandable.
\nSome questions that come to my mind:
\nI believe that these competitions are (at least today) not as abundant as regular ones, so becoming a \"Kaggle Analytics grandmaster\" is out of the scope for now. I think that these competitions will make the community better and will eventually evolve through trial an error to a more objective one.
\nI wish to hear solutions from other people on how would they solve this subjectiveness issue or if they think no medals should be given at all.
", "rawMarkdown": "My opinion: The very nature of this competition is subjective, since evaluating an essay involves humans and lacks an objective mathematical evaluation (as every other kaggle competition), therefore representing a new challenge. \n\nOn the one hand, Kagglers *may* feel that an obtained medal from this (analytics) competition is not worth the same as a traditional one. On the other hand, participants also want a prize/recognition for their effort, which also sounds completely understandable.\n\nSome questions that come to my mind: \n- Should Kaggle create a different medal for this kind of competition or should not give medals at all? In the past, no medals were given for analytics competitions. In my opinion, giving medals might encourage participation which ultimately provide fruitful results and strive the community.\n- Is there an objective way to evaluate a subjective competition? So far, the best efforts seem to involve a panel of experts. I ignore if there might be a solution where an algorithm evaluates a written essay (similar to the Feedback prize competition) but super challenging in my opinion.\n\nI believe that these competitions are (at least today) not as abundant as regular ones, so becoming a \"Kaggle Analytics grandmaster\" is out of the scope for now. I think that these competitions will make the community better and will eventually evolve through trial an error to a more objective one.\n\nI wish to hear solutions from other people on how would they solve this subjectiveness issue or if they think no medals should be given at all.", "votes": 4, "replies": [ { "id": 2256692, "postDate": "2023-05-12T16:53:43.390Z", "content": "@philippsinger I just read your comment on not awarding points. I think that sounds very reasonable. Give medals, not points.
", "rawMarkdown": "@philippsinger I just read your comment on not awarding points. I think that sounds very reasonable. Give medals, not points.", "votes": 1 }, { "id": 2258447, "postDate": "2023-05-14T07:59:58.397Z", "content": "\n\nOn the other hand, participants also want a prize/recognition for their effort
\n
how about 10000$ x7 for each category?
", "rawMarkdown": ">On the other hand, participants also want a prize/recognition for their effort\n\nhow about 10000$ x7 for each category?", "votes": 1, "replies": [ { "id": 2260614, "postDate": "2023-05-15T18:50:56.133Z", "content": "@bakeryproducts people's motifs aren't solely money based, however I believe that having a separate medal category would be the best
", "rawMarkdown": "@bakeryproducts people's motifs aren't solely money based, however I believe that having a separate medal category would be the best", "votes": 1 } ] }, { "id": 2322503, "postDate": "2023-06-29T10:13:16.130Z", "content": "Just having a look at this competition, and considering the issues of subjectivity in Analytics competitions and medals, points, etc.
\nIn some of the past ones, e.g. for NFL, where there is an outside judging panel, a clear rubric and objectives for what analysis should provide to the hosts, it might be easier to justify medals or points as well as prizes.
With essays and peer reviews it might get a bit murky. Have not fully understood the evaluation process - \"an expert grading panel of seven Kaggle Grandmasters\" is not quite the same thing as academics or journal editors or reviewers. Grandmasters come in different forms here. But if there were different rankings for Analytics so a pathway for a Grandmaster in that area, then it might make more sense in future. As this seems an experimental trial it may take time to work out what will be best. But it would seem a positive for Kaggle and those with and developing analytics skills to have a way to be recognised and rewarded and incentive to participate for some. At least with plagiarism checks it is an improvement on the competitions copy and submit to medals pathway.
\nRegarding an objective way to evaluate - the goal stated is \"to centralize and summarize the rapid advancements in AI from the past two years\". With scholarly articles provided and requirements for citations, one presumes references or bibliography, perhaps a form of NLP could do some evaluation - maybe an area for a future competition or combined with an future analytics competition.
", "rawMarkdown": "Just having a look at this competition, and considering the issues of subjectivity in Analytics competitions and medals, points, etc.\nIn some of the past ones, e.g. for NFL, where there is an outside judging panel, a clear rubric and objectives for what analysis should provide to the hosts, it might be easier to justify medals or points as well as prizes. \n\nWith essays and peer reviews it might get a bit murky. Have not fully understood the evaluation process - \"an expert grading panel of seven Kaggle Grandmasters\" is not quite the same thing as academics or journal editors or reviewers. Grandmasters come in different forms here. But if there were different rankings for Analytics so a pathway for a Grandmaster in that area, then it might make more sense in future. As this seems an experimental trial it may take time to work out what will be best. But it would seem a positive for Kaggle and those with and developing analytics skills to have a way to be recognised and rewarded and incentive to participate for some. At least with plagiarism checks it is an improvement on the competitions copy and submit to medals pathway.\n\nRegarding an objective way to evaluate - the goal stated is \"to centralize and summarize the rapid advancements in AI from the past two years\". With scholarly articles provided and requirements for citations, one presumes references or bibliography, perhaps a form of NLP could do some evaluation - maybe an area for a future competition or combined with an future analytics competition. " } ] }, { "id": 2256740, "postDate": "2023-05-12T17:23:33.077Z", "content": "With all due respect to Kaggle's efforts to broaden its data science initiatives, I just wanted to add a comment here (particularly for Kaggle and its cool staff) regarding the method through which medals would be awarded based on \"human evaluation\". The process will only truly function and make sense in one circumstance: if Kaggle is successful in developing a \"ChatGPT4\" on the platform that can automatically and without human assistance evaluate the work of competitors and offer medals. And even then, catastrophes might result. I sincerely hope Kaggle stays loyal to its principles and values and what its \"lengthy\" history means to the data science community. The scientific journal is a good direction, as Philipp suggests here. Otherwise, whether human evaluation offers medals or \"ChatGPT evaluation\" offers medals (etc.), Kaggle runs the risk of producing future GrandDisasters rather than GrandMasters.
", "rawMarkdown": "With all due respect to Kaggle's efforts to broaden its data science initiatives, I just wanted to add a comment here (particularly for Kaggle and its cool staff) regarding the method through which medals would be awarded based on \"human evaluation\". The process will only truly function and make sense in one circumstance: if Kaggle is successful in developing a \"ChatGPT4\" on the platform that can automatically and without human assistance evaluate the work of competitors and offer medals. And even then, catastrophes might result. I sincerely hope Kaggle stays loyal to its principles and values and what its \"lengthy\" history means to the data science community. The scientific journal is a good direction, as Philipp suggests here. Otherwise, whether human evaluation offers medals or \"ChatGPT evaluation\" offers medals (etc.), Kaggle runs the risk of producing future **GrandDisasters** rather than **GrandMasters**.", "votes": 5 }, { "id": 2262935, "postDate": "2023-05-17T08:30:17.903Z", "content": "I support the viewpoint of keeping a \"pure\" competition tier, with only objective scoring (metric based) for all the competitions. There is always the question of double medals rewards if you get medal in the analytic competition and for a Notebook in that analytic competition that is your support material for the same submission.
", "rawMarkdown": "I support the viewpoint of keeping a \"pure\" competition tier, with only objective scoring (metric based) for all the competitions. There is always the question of double medals rewards if you get medal in the analytic competition and for a Notebook in that analytic competition that is your support material for the same submission.", "votes": 3 }, { "id": 2259456, "postDate": "2023-05-15T02:43:47.360Z", "content": "Kagglers are also experiencing inflation.😆
", "rawMarkdown": "Kagglers are also experiencing inflation.😆", "votes": 4 }, { "id": 2256729, "postDate": "2023-05-12T17:15:03.020Z", "content": "I honestly don't think the subjectivity aspect is a problem. Just about every college curriculum involves a combination of objective and subjective assessment. In fact, there is generally a greater emphasis on subjective assessment as you progress through the academic ranks. A competition like this is not that different from a Master's dissertation or a PhD thesis, but in a very condensed form. The assessment format is broadly the same.
\nNow, I have to say that I may be biased in this case. As a data analyst I am more at home with distilling and communicating information than I am with pure machine learning (although this is something I'm working on). I don't have the programming and data science skills and experience that top Kagglers have, so a competition like this is something that someone like me could potentially do relatively well in.
\nSo I suppose the question becomes: is this something that fits in with the general Kaggle format/ethos? Personally, I think it does but I can also understand how a change to a less objective scoring system with the same benefits could devalue user's rankings and medals, especially considering the work and effort that has gone into many people's Kaggle portfolio.
", "rawMarkdown": "I honestly don't think the subjectivity aspect is a problem. Just about every college curriculum involves a combination of objective and subjective assessment. In fact, there is generally a greater emphasis on subjective assessment as you progress through the academic ranks. A competition like this is not that different from a Master's dissertation or a PhD thesis, but in a very condensed form. The assessment format is broadly the same.\n\nNow, I have to say that I may be biased in this case. As a data analyst I am more at home with distilling and communicating information than I am with pure machine learning (although this is something I'm working on). I don't have the programming and data science skills and experience that top Kagglers have, so a competition like this is something that someone like me could potentially do relatively well in. \n\nSo I suppose the question becomes: is this something that fits in with the general Kaggle format/ethos? Personally, I think it does but I can also understand how a change to a less objective scoring system with the same benefits could devalue user's rankings and medals, especially considering the work and effort that has gone into many people's Kaggle portfolio.", "votes": 2, "replies": [ { "id": 2258745, "postDate": "2023-05-14T12:57:40.750Z", "content": "Great points! but it is the unhappiness with the subjectivity of a lot of evaluations in society that is the root cause of a lot of issues. Its always easier/ideal to have more objective metrics, especially for a data driven/data oriented platform. But with essays creating a fair, transperent and understandable algorithm for evaluation is quite tricky in its own way. One easy but impractical way to avoid this would be to have multiple isolated panel of experts reviewing the essays as that would provide a broader consensus. Ideally, it would be better to use the community's consensus on topics covered, quality of the content and so on all evaluated into a score but doing so wouldn't be simple and additionally may face cold-start issues on some quality works that just didn't get enough engagement in the first place.
\nit'd be interesting to see where this goes in the future, as there is a significant discussion to be had to provide rewards for analytics competitions without separating them (as then the competitions medal changes from an objective marker of one's capability to a more subjective one in its place).
", "rawMarkdown": "Great points! but it is the unhappiness with the subjectivity of a lot of evaluations in society that is the root cause of a lot of issues. Its always easier/ideal to have more objective metrics, especially for a data driven/data oriented platform. But with essays creating a fair, transperent and understandable algorithm for evaluation is quite tricky in its own way. One easy but impractical way to avoid this would be to have multiple isolated panel of experts reviewing the essays as that would provide a broader consensus. Ideally, it would be better to use the community's consensus on topics covered, quality of the content and so on all evaluated into a score but doing so wouldn't be simple and additionally may face cold-start issues on some quality works that just didn't get enough engagement in the first place.\n\nit'd be interesting to see where this goes in the future, as there is a significant discussion to be had to provide rewards for analytics competitions without separating them (as then the competitions medal changes from an objective marker of one's capability to a more subjective one in its place).", "votes": 1 } ] }, { "id": 2258732, "postDate": "2023-05-14T12:44:34.120Z", "content": "While I understand the perspective of wanting objectivity in competitive scenarios over subjectivity of written content submissions and general perspective in research (I've had my fair share of paper rejections wherein I didn't agree with some of the feedback provided).
\nthe core idea behind crowdsourcing essays and research (and potentially developing a more comprehensive combined essay based on top submissions) is rather excellent. Maybe as a first iteration it'd be great for the community to also suggest implementations that would work. Personally i'd say objective metrics isn't easy. It would be interesting if the competition was phrased differently, say ranking and justifying the top X number of developments in a particular field here in kaggle and in the AI space in general, and analysing general consensus based on upvotes as well as similarity of ranking in other essays.
\nthe general quality of what is written can also be assessed based on comments. It would have to be defined rather clearly though.
\nconsidering the potential impracticality and opaqueness of this solution, I like the ideas suggested of moving away from medals or points, but community engagement is dependent on points and medals as well, and more engagement does produce a better summary of the landscape here. tough one to crack but my thoughts are
\ninterested on peoples thoughts, and see where this goes in time as well
", "rawMarkdown": "While I understand the perspective of wanting objectivity in competitive scenarios over subjectivity of written content submissions and general perspective in research (I've had my fair share of paper rejections wherein I didn't agree with some of the feedback provided).\n\nthe core idea behind crowdsourcing essays and research (and potentially developing a more comprehensive combined essay based on top submissions) is rather excellent. Maybe as a first iteration it'd be great for the community to also suggest implementations that would work. Personally i'd say objective metrics isn't easy. It would be interesting if the competition was phrased differently, say ranking and justifying the top X number of developments in a particular field here in kaggle and in the AI space in general, and analysing general consensus based on upvotes as well as similarity of ranking in other essays. \n\nthe general quality of what is written can also be assessed based on comments. It would have to be defined rather clearly though.\n\nconsidering the potential impracticality and opaqueness of this solution, I like the ideas suggested of moving away from medals or points, but community engagement is dependent on points and medals as well, and more engagement does produce a better summary of the landscape here. tough one to crack but my thoughts are\n\n- would be an ideal opportunity to trial out algorithms and methodologies (maybe using past evaluations to create a competition for developing such algorithms) to objectively evaluate essays, and get relatively similar results to what multiple different sets of expert panels say.\n- for the time being, consider that avoiding points and medals to such competitions may distort the community based on the panel's decisions (award swag instead?)\n\ninterested on peoples thoughts, and see where this goes in time as well", "votes": 1 }, { "id": 2259008, "postDate": "2023-05-14T16:09:28.410Z", "content": "Totally agree, this is not ml or even cs comp
", "rawMarkdown": "Totally agree, this is not ml or even cs comp", "votes": -1, "replies": [ { "id": 2261192, "postDate": "2023-05-16T07:06:45.030Z", "content": "\n\nTotally agree, this is not ml or even cs comp
\n
You can challenge the subjective nature of the evaluation methods. But kaggle is a data science platform and DS is not exclusively a CS field. Regarding ML, since it is an open ended essay writing challenge, one can substantiate his/her essay thesis using a ML model result(s). So ML is not out of scope for this challenge.
", "rawMarkdown": "> Totally agree, this is **not ml** or even **cs** comp\n\nYou can challenge the subjective nature of the evaluation methods. But kaggle is a data science platform and DS is not exclusively a CS field. Regarding ML, since it is an open ended essay writing challenge, one can substantiate his/her essay thesis using a ML model result(s). So ML is not out of scope for this challenge. " } ] }, { "id": 2331669, "postDate": "2023-07-05T17:31:14.977Z", "content": "This is a nice competition
", "rawMarkdown": "This is a nice competition\n" }, { "id": 2262933, "postDate": "2023-05-17T08:29:52.370Z", "content": "I strongly support the viewpoint of keeping a \"pure\" competition tier, with only objective scoring (metric based) for all the competitions. There is always the question of double medals rewards if you get medal in the analytic competition and for a Notebook in that analytic competition that is your support material for the same submission.
", "rawMarkdown": "I strongly support the viewpoint of keeping a \"pure\" competition tier, with only objective scoring (metric based) for all the competitions. There is always the question of double medals rewards if you get medal in the analytic competition and for a Notebook in that analytic competition that is your support material for the same submission." }, { "id": 2258536, "postDate": "2023-05-14T09:57:56.217Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "comments": [ { "id": 2257992, "author_name": "dynamic24", "author_url": "", "post_date": "2023-05-13T19:07:42.467000", "content": "The competitions tier is the only Kaggle tier that isn’t based on upvotes. I’d hate to see more qualitative scoring seep into that tier and make the competitions KGM title more questionable. A writing contest really isn’t very similar to other Kaggle competitions. Giving out notebook medals would make more sense than competitions medals since people are submitting notebooks to be judged. The suggestion of a separate “Analytics KGM” tier to recognize analytics competitions also makes sense to me.
", "votes": 20, "replies": [ { "id": 2258137, "author_name": "", "author_url": "", "post_date": "2023-05-13T23:25:39.907000", "content": "Indeed, giving notebook medals makes sense. But Analytics Tier awarding competition medals based on human evaluation will only transfer the issue. I have never seen, for example, a sports medal given to an athlete (at the Olympics, for instance) based on a human evaluation (that's how the founder of Kaggle Anthony was inspired when he created Kaggle, I read an article about this: Kaggle is \"born\" on the idea to make data science like sports). The same is true for chess games and math Olympiads (medals are not awarded based on human evaluation). Therefore, the issue goes beyond mixing analytics medals with leaderboard medals. Without getting into specifics, if someone has tried to participate in both analytics competitions and the Leaderboard ML competitions on Kaggle (at least once), they will sense that the platform and the seriousness are not the same (mainly the evaluation).
\nAdditionally, I don't like to discuss the increased likelihood, potential, and risk of gaming the system with the Analytics competitions tier (people already sell medals in leaderboard competitions, and kagglers suffered from that…). Adding an analytics competition tier (awarding medals based on human evaluation) just shifts the issue(not resolving it).
Yes, the move from objective to human evaluation for competitions (which will need to incorporate peer evaluation eventually to scale) is my concern. It would be nice to have one tier that is based totally on objectively scored contests.
\nI remember the early days of Kaggle when \"Making Data Science a Sport\" was the slogan. The Kaggle philosophy has shifted over the years from being a place to compete to being a place to collaborate and compete on data science projects. When the website started, there were only competitions with no overall rankings or user levels. Then they introduced user rankings and an overall leaderboard based on competitions performance only. If we wanted to dig into Kaggle history, the introduction of notebook/kernel and discussion tiers was not without controversy:
\nhttps://www.kaggle.com/discussions/general/22208
\nhttps://www.kaggle.com/discussions/general/22248
\n\nI have never seen, for example, a sports medal given to an athlete (at the Olympics, for instance) based on a human evaluation….
\n
(Just for the first and second photo): They had to demonstrate their abilities and put in a lot of effort to qualify for this stage in front of the jury(=judges). For the rest, they just count on the strength of their hoof (Horces) and strategies of the riders, the same for Footballers: their strength, intelligence and talent(And the Video Assistant Referee VAR is to decide, in the event of a violation of the Rules: you can compare this to what Kaggle applies on the Leaderboard at the end of competition to eliminate cheaters, etc.)
\nThen, Aside from the \"mind\" and scientific competitions that I mentioned (Chess competitions and math Olympiads) , you may also have a look at long-distance races, sprints, swimmers, tennis, etc (Only their talent: no huamn Evaluation).
@dynamic24 Thanks for the link :), and for bringing me back to, as some kagglers called it, \"the good old times\" of kaggle that I do not know :)
\nAnd yes, I 100% agree (objective evaluation, far from a human evaluation).
\nFor the other two tiers (notebooks and discussions), even though most people today are aware that titles created solely based on upvotes are not always credible (of course, there are exceptions), the goal of creating these tiers was good : it was to encourage sharing and to help others grow along with them. Many Kagglers would not have been able to share all of their expertise and be as generous to the Kaggle community without the existence of these two tiers. Also, everyone is aware that the competition tier is the most credible (and I hope it stays that way). additionally, when kagglers share a notebook or an idea =discussion (and win medals based on upvotes), they are not doing so (at least explicitly) in a competitive manner (it is not a competition, so it is OK…). But when it comes with competitions, things change…
Also, there isn't a sport where one group of competitors are judged by humans, and another by objective metrics, especially when they are all competing in the same category.
\nThis is a great competition - but I think it makes more sense to move it to a separate Analytics tier, as other Kagglers have suggested already.
", "votes": 2, "replies": [] }, { "id": 2258961, "author_name": "A Legacy Grandmaster!", "author_url": "", "post_date": "2023-05-14T15:12:55.877000", "content": "Dear Abir, I respect your views.
\n\n\n(Just for the first and second photo): They had to demonstrate their abilities and put in a lot of effort to qualify for this stage in front of the jury(=judges).
\n
But I thought the issue was having a subjective judge (jury). That is why I pulled those pictures, i.e to present a case that there are Olympic sports where medals are awarded based on human judgment, even the seemingly objective sport such as football (more goals that opponents give you wins. Simple. right?) the subjective judgment of the referee can swing the outcome of any game. VAR isn't perfect either. Why? VAR is operated by humans. There were some high profile VAR errors in the English Premier League in the current season which the referee's body had to apologize for.
\nI understand people’s skepticism about panel judged ranking points [the original post seems to be pivoted around it] diluting the competition. However, one could argue that, while the individual competitions' LBs are metric-based and hence are bias free, the ranking points algorithm is not necessarily objective. It's an algorithm derived by a panel of kaggle team (experts).
", "votes": 3, "replies": [] }, { "id": 2259163, "author_name": "", "author_url": "", "post_date": "2023-05-14T18:13:51.343000", "content": "@desalegngeb first, let me thank you for your nice response.
\nOk, you are right. I just wanted to point out that they are fundamentally uncomparable. Even with that, look at this (they're already complaining…). Soon, the AI will assign Olympic medals :) .AI_assign_olympic_medals_and_beyond
\nKaggle should be ahead of them: here is the house of data science and somewhat of AI :) . your last sentences are logical, but I think it's acceptable (I'm referring to the algo of Kaggle and objectivity…).
@rai555 yes, it is right, but to me, in this tier or a different one, it's the human evaluation (that awards medals) that doesn't make sense (I explained it well in my comments…), and I see that adding a different analytics tier doesn't actually address the issue but it just transfers it(…). But let's stick with this plan to add a new tier. There are competitors, and medals that will be awarded( regardless of how they are given out :)), so, these are also competitions, even analytics ones. Therefore, the current tier should be renamed, perhaps as the Leaderboard (ML, DL, RL)Competition tier, and the new tier would be called the Analytics Competition tier (although the two will have leaderboards :) ). (X will be Leadarboard(ML, DL, RL) Competition Grandmaster, and Y will be Analytics Competition Grandmaster, for example?) Seems odd :) I'm not sure how logical or practical(…) it will be. (Of course, this is, if the main issue (human evaluation, which awards medals) does not count, no matter what is going on there :)…)
", "votes": 1, "replies": [] } ] } ] }, { "id": 2258436, "author_name": "Gleb", "author_url": "", "post_date": "2023-05-14T07:53:59.813000", "content": "I'm 100% with psi on this. Essay writing is great, but I don't think an \"upvote\" (or equivalent) competition should count in the same way as a metric-based competition.
", "votes": 15, "replies": [] }, { "id": 2260057, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2023-05-15T12:48:22.433000", "content": "Kaggle wants to give medals and points because this is the only way to maximize participation. But they are precisely so precious because there were never subjective and everybody knows what they stand for. Mixing standard competitions with \"upvote-based\" mechanics in the same ranking will deteriorate the Kaggle competition GM brand/points. Let's create a separate analytics competitions tier.
", "votes": 13, "replies": [ { "id": 2261414, "author_name": "Gleb", "author_url": "", "post_date": "2023-05-16T10:01:49.950000", "content": "\n\nthe only way to maximize participation
\n
Absolutely understandable, and here is the other side: there are some other ds/dl competition platforms. All of them are less popular then Kaggle. rn K have a certain level of trustworthiness / respect with their approach to competitions. If that will drift into something else, towards other platforms, why wont competitors drift to them too?
", "votes": 2, "replies": [] } ] }, { "id": 2257911, "author_name": "CroDoc", "author_url": "", "post_date": "2023-05-13T18:09:23.660000", "content": "I'd suggest making a separate \"analytics\" tier if there is going to be enough competitions of this type in the future.
", "votes": 13, "replies": [ { "id": 2258739, "author_name": "Rahul Namboodiri", "author_url": "", "post_date": "2023-05-14T12:51:46.290000", "content": "great suggestion! unfortunately doesn't seem to be on the horizon that there would be enough competitions. would be interesting to have them though!
", "votes": 2, "replies": [ { "id": 2259060, "author_name": "CroDoc", "author_url": "", "post_date": "2023-05-14T16:58:06.903000", "content": "There were already a few of these before. Maybe kaggle wants more of these.
", "votes": 1, "replies": [] } ] }, { "id": 2259301, "author_name": "RB", "author_url": "", "post_date": "2023-05-14T20:43:03.897000", "content": "I agree with separate tier , This competition is “2023 Kaggle AI Report”, I guess we could expect at least once a year similar competition
", "votes": 3, "replies": [] } ] }, { "id": 2262056, "author_name": "Dieter", "author_url": "", "post_date": "2023-05-16T17:52:00.140000", "content": "The has been an official update here
\nhttps://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/410802
", "votes": 5, "replies": [ { "id": 2262145, "author_name": "CroDoc", "author_url": "", "post_date": "2023-05-16T18:52:58.530000", "content": "So now you can boost your ranking by getting points, but can't become a competition GM since they don't award medals. If there is not gonna be so many analytics comps to create a separate tier, this sounds like a decent compromise.
", "votes": 0, "replies": [ { "id": 2262187, "author_name": "", "author_url": "", "post_date": "2023-05-16T19:16:01.363000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2256681, "author_name": "moth", "author_url": "", "post_date": "2023-05-12T16:47:03.310000", "content": "My opinion: The very nature of this competition is subjective, since evaluating an essay involves humans and lacks an objective mathematical evaluation (as every other kaggle competition), therefore representing a new challenge.
\nOn the one hand, Kagglers may feel that an obtained medal from this (analytics) competition is not worth the same as a traditional one. On the other hand, participants also want a prize/recognition for their effort, which also sounds completely understandable.
\nSome questions that come to my mind:
\nI believe that these competitions are (at least today) not as abundant as regular ones, so becoming a \"Kaggle Analytics grandmaster\" is out of the scope for now. I think that these competitions will make the community better and will eventually evolve through trial an error to a more objective one.
\nI wish to hear solutions from other people on how would they solve this subjectiveness issue or if they think no medals should be given at all.
", "votes": 4, "replies": [ { "id": 2256692, "author_name": "moth", "author_url": "", "post_date": "2023-05-12T16:53:43.390000", "content": "@philippsinger I just read your comment on not awarding points. I think that sounds very reasonable. Give medals, not points.
", "votes": 1, "replies": [] }, { "id": 2258447, "author_name": "Gleb", "author_url": "", "post_date": "2023-05-14T07:59:58.397000", "content": "\n\nOn the other hand, participants also want a prize/recognition for their effort
\n
how about 10000$ x7 for each category?
", "votes": 1, "replies": [ { "id": 2260614, "author_name": "moth", "author_url": "", "post_date": "2023-05-15T18:50:56.133000", "content": "@bakeryproducts people's motifs aren't solely money based, however I believe that having a separate medal category would be the best
", "votes": 1, "replies": [] } ] }, { "id": 2322503, "author_name": "something4kag", "author_url": "", "post_date": "2023-06-29T10:13:16.130000", "content": "Just having a look at this competition, and considering the issues of subjectivity in Analytics competitions and medals, points, etc.
\nIn some of the past ones, e.g. for NFL, where there is an outside judging panel, a clear rubric and objectives for what analysis should provide to the hosts, it might be easier to justify medals or points as well as prizes.
With essays and peer reviews it might get a bit murky. Have not fully understood the evaluation process - \"an expert grading panel of seven Kaggle Grandmasters\" is not quite the same thing as academics or journal editors or reviewers. Grandmasters come in different forms here. But if there were different rankings for Analytics so a pathway for a Grandmaster in that area, then it might make more sense in future. As this seems an experimental trial it may take time to work out what will be best. But it would seem a positive for Kaggle and those with and developing analytics skills to have a way to be recognised and rewarded and incentive to participate for some. At least with plagiarism checks it is an improvement on the competitions copy and submit to medals pathway.
\nRegarding an objective way to evaluate - the goal stated is \"to centralize and summarize the rapid advancements in AI from the past two years\". With scholarly articles provided and requirements for citations, one presumes references or bibliography, perhaps a form of NLP could do some evaluation - maybe an area for a future competition or combined with an future analytics competition.
", "votes": 0, "replies": [] } ] }, { "id": 2256740, "author_name": "", "author_url": "", "post_date": "2023-05-12T17:23:33.077000", "content": "With all due respect to Kaggle's efforts to broaden its data science initiatives, I just wanted to add a comment here (particularly for Kaggle and its cool staff) regarding the method through which medals would be awarded based on \"human evaluation\". The process will only truly function and make sense in one circumstance: if Kaggle is successful in developing a \"ChatGPT4\" on the platform that can automatically and without human assistance evaluate the work of competitors and offer medals. And even then, catastrophes might result. I sincerely hope Kaggle stays loyal to its principles and values and what its \"lengthy\" history means to the data science community. The scientific journal is a good direction, as Philipp suggests here. Otherwise, whether human evaluation offers medals or \"ChatGPT evaluation\" offers medals (etc.), Kaggle runs the risk of producing future GrandDisasters rather than GrandMasters.
", "votes": 5, "replies": [] }, { "id": 2262935, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2023-05-17T08:30:17.903000", "content": "I support the viewpoint of keeping a \"pure\" competition tier, with only objective scoring (metric based) for all the competitions. There is always the question of double medals rewards if you get medal in the analytic competition and for a Notebook in that analytic competition that is your support material for the same submission.
", "votes": 3, "replies": [] }, { "id": 2259456, "author_name": "hyd", "author_url": "", "post_date": "2023-05-15T02:43:47.360000", "content": "Kagglers are also experiencing inflation.😆
", "votes": 4, "replies": [] }, { "id": 2256729, "author_name": "Jared Savage", "author_url": "", "post_date": "2023-05-12T17:15:03.020000", "content": "I honestly don't think the subjectivity aspect is a problem. Just about every college curriculum involves a combination of objective and subjective assessment. In fact, there is generally a greater emphasis on subjective assessment as you progress through the academic ranks. A competition like this is not that different from a Master's dissertation or a PhD thesis, but in a very condensed form. The assessment format is broadly the same.
\nNow, I have to say that I may be biased in this case. As a data analyst I am more at home with distilling and communicating information than I am with pure machine learning (although this is something I'm working on). I don't have the programming and data science skills and experience that top Kagglers have, so a competition like this is something that someone like me could potentially do relatively well in.
\nSo I suppose the question becomes: is this something that fits in with the general Kaggle format/ethos? Personally, I think it does but I can also understand how a change to a less objective scoring system with the same benefits could devalue user's rankings and medals, especially considering the work and effort that has gone into many people's Kaggle portfolio.
", "votes": 2, "replies": [ { "id": 2258745, "author_name": "Rahul Namboodiri", "author_url": "", "post_date": "2023-05-14T12:57:40.750000", "content": "Great points! but it is the unhappiness with the subjectivity of a lot of evaluations in society that is the root cause of a lot of issues. Its always easier/ideal to have more objective metrics, especially for a data driven/data oriented platform. But with essays creating a fair, transperent and understandable algorithm for evaluation is quite tricky in its own way. One easy but impractical way to avoid this would be to have multiple isolated panel of experts reviewing the essays as that would provide a broader consensus. Ideally, it would be better to use the community's consensus on topics covered, quality of the content and so on all evaluated into a score but doing so wouldn't be simple and additionally may face cold-start issues on some quality works that just didn't get enough engagement in the first place.
\nit'd be interesting to see where this goes in the future, as there is a significant discussion to be had to provide rewards for analytics competitions without separating them (as then the competitions medal changes from an objective marker of one's capability to a more subjective one in its place).
", "votes": 1, "replies": [] } ] }, { "id": 2258732, "author_name": "Rahul Namboodiri", "author_url": "", "post_date": "2023-05-14T12:44:34.120000", "content": "While I understand the perspective of wanting objectivity in competitive scenarios over subjectivity of written content submissions and general perspective in research (I've had my fair share of paper rejections wherein I didn't agree with some of the feedback provided).
\nthe core idea behind crowdsourcing essays and research (and potentially developing a more comprehensive combined essay based on top submissions) is rather excellent. Maybe as a first iteration it'd be great for the community to also suggest implementations that would work. Personally i'd say objective metrics isn't easy. It would be interesting if the competition was phrased differently, say ranking and justifying the top X number of developments in a particular field here in kaggle and in the AI space in general, and analysing general consensus based on upvotes as well as similarity of ranking in other essays.
\nthe general quality of what is written can also be assessed based on comments. It would have to be defined rather clearly though.
\nconsidering the potential impracticality and opaqueness of this solution, I like the ideas suggested of moving away from medals or points, but community engagement is dependent on points and medals as well, and more engagement does produce a better summary of the landscape here. tough one to crack but my thoughts are
\ninterested on peoples thoughts, and see where this goes in time as well
", "votes": 1, "replies": [] }, { "id": 2259008, "author_name": "empty", "author_url": "", "post_date": "2023-05-14T16:09:28.410000", "content": "Totally agree, this is not ml or even cs comp
", "votes": -1, "replies": [ { "id": 2261192, "author_name": "A Legacy Grandmaster!", "author_url": "", "post_date": "2023-05-16T07:06:45.030000", "content": "\n\nTotally agree, this is not ml or even cs comp
\n
You can challenge the subjective nature of the evaluation methods. But kaggle is a data science platform and DS is not exclusively a CS field. Regarding ML, since it is an open ended essay writing challenge, one can substantiate his/her essay thesis using a ML model result(s). So ML is not out of scope for this challenge.
", "votes": 0, "replies": [] } ] }, { "id": 2331669, "author_name": "Canum05", "author_url": "", "post_date": "2023-07-05T17:31:14.977000", "content": "This is a nice competition
", "votes": 0, "replies": [] }, { "id": 2262933, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2023-05-17T08:29:52.370000", "content": "I strongly support the viewpoint of keeping a \"pure\" competition tier, with only objective scoring (metric based) for all the competitions. There is always the question of double medals rewards if you get medal in the analytic competition and for a Notebook in that analytic competition that is your support material for the same submission.
", "votes": 0, "replies": [] }, { "id": 2258536, "author_name": "", "author_url": "", "post_date": "2023-05-14T09:57:56.217000", "content": "", "votes": -2, "replies": [] } ], "raw_markdown_by_id": { "2256541": "As there was already a discussion about this on Twitter and others have expressed similar concerns, it might make sense to discuss this topic here on Kaggle - specifically as early as possible.\n\nFirst of all, I find the idea of this competition very cool. A lot of great ideas get first developed and researched on Kaggle, and there is sometimes a bit of a lack of public perception of these. So collecting and publishing them sounds like a good thing to do.\n\nHowever, reviewing these will always need to follow some sort of typical research peer process. This competition is following a similar direction, by having peer review (fellow kagglers) and expert assessment (program chairs, GMs). \n\nHowever, one of my reasons why I started Kaggling was to exactly move away from this subjective process. I always felt particularly intrigued by the fully objective metric-based assessment of my submissions. There is no human intervention when it is decided who is on top or not.\n\nNow with awarding medals and points to this competition, I feel concerned about the integrity of the competition tier. Top Kagglers have spent countless hours of their life to work on competitive objective metric-based competitions and being judged by other kagglers, who you are actually competing with on the global leaderboard, feels a bit unnerving to me.\n\nI know that there are different opinions about this, so happy to hear what others think. By the way, I think Kaggle should start their own scientific journal where people can submit and publish their work.", "2257992": "The competitions tier is the only Kaggle tier that isn’t based on upvotes. I’d hate to see more qualitative scoring seep into that tier and make the competitions KGM title more questionable. A writing contest really isn’t very similar to other Kaggle competitions. Giving out notebook medals would make more sense than competitions medals since people are submitting notebooks to be judged. The suggestion of a separate “Analytics KGM” tier to recognize analytics competitions also makes sense to me.", "2258436": "I'm 100% with psi on this. Essay writing is great, but I don't think an \"upvote\" (or equivalent) competition should count in the same way as a metric-based competition.", "2260057": "Kaggle wants to give medals and points because this is the only way to maximize participation. But they are precisely so precious because there were never subjective and everybody knows what they stand for. Mixing standard competitions with \"upvote-based\" mechanics in the same ranking will deteriorate the Kaggle competition GM brand/points. Let's create a separate analytics competitions tier. ", "2257911": "I'd suggest making a separate \"analytics\" tier if there is going to be enough competitions of this type in the future.", "2262056": "The has been an official update here\n\nhttps://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/410802", "2256681": "My opinion: The very nature of this competition is subjective, since evaluating an essay involves humans and lacks an objective mathematical evaluation (as every other kaggle competition), therefore representing a new challenge. \n\nOn the one hand, Kagglers *may* feel that an obtained medal from this (analytics) competition is not worth the same as a traditional one. On the other hand, participants also want a prize/recognition for their effort, which also sounds completely understandable.\n\nSome questions that come to my mind: \n- Should Kaggle create a different medal for this kind of competition or should not give medals at all? In the past, no medals were given for analytics competitions. In my opinion, giving medals might encourage participation which ultimately provide fruitful results and strive the community.\n- Is there an objective way to evaluate a subjective competition? So far, the best efforts seem to involve a panel of experts. I ignore if there might be a solution where an algorithm evaluates a written essay (similar to the Feedback prize competition) but super challenging in my opinion.\n\nI believe that these competitions are (at least today) not as abundant as regular ones, so becoming a \"Kaggle Analytics grandmaster\" is out of the scope for now. I think that these competitions will make the community better and will eventually evolve through trial an error to a more objective one.\n\nI wish to hear solutions from other people on how would they solve this subjectiveness issue or if they think no medals should be given at all.", "2256740": "With all due respect to Kaggle's efforts to broaden its data science initiatives, I just wanted to add a comment here (particularly for Kaggle and its cool staff) regarding the method through which medals would be awarded based on \"human evaluation\". The process will only truly function and make sense in one circumstance: if Kaggle is successful in developing a \"ChatGPT4\" on the platform that can automatically and without human assistance evaluate the work of competitors and offer medals. And even then, catastrophes might result. I sincerely hope Kaggle stays loyal to its principles and values and what its \"lengthy\" history means to the data science community. The scientific journal is a good direction, as Philipp suggests here. Otherwise, whether human evaluation offers medals or \"ChatGPT evaluation\" offers medals (etc.), Kaggle runs the risk of producing future **GrandDisasters** rather than **GrandMasters**.", "2262935": "I support the viewpoint of keeping a \"pure\" competition tier, with only objective scoring (metric based) for all the competitions. There is always the question of double medals rewards if you get medal in the analytic competition and for a Notebook in that analytic competition that is your support material for the same submission.", "2259456": "Kagglers are also experiencing inflation.😆", "2256729": "I honestly don't think the subjectivity aspect is a problem. Just about every college curriculum involves a combination of objective and subjective assessment. In fact, there is generally a greater emphasis on subjective assessment as you progress through the academic ranks. A competition like this is not that different from a Master's dissertation or a PhD thesis, but in a very condensed form. The assessment format is broadly the same.\n\nNow, I have to say that I may be biased in this case. As a data analyst I am more at home with distilling and communicating information than I am with pure machine learning (although this is something I'm working on). I don't have the programming and data science skills and experience that top Kagglers have, so a competition like this is something that someone like me could potentially do relatively well in. \n\nSo I suppose the question becomes: is this something that fits in with the general Kaggle format/ethos? Personally, I think it does but I can also understand how a change to a less objective scoring system with the same benefits could devalue user's rankings and medals, especially considering the work and effort that has gone into many people's Kaggle portfolio.", "2258732": "While I understand the perspective of wanting objectivity in competitive scenarios over subjectivity of written content submissions and general perspective in research (I've had my fair share of paper rejections wherein I didn't agree with some of the feedback provided).\n\nthe core idea behind crowdsourcing essays and research (and potentially developing a more comprehensive combined essay based on top submissions) is rather excellent. Maybe as a first iteration it'd be great for the community to also suggest implementations that would work. Personally i'd say objective metrics isn't easy. It would be interesting if the competition was phrased differently, say ranking and justifying the top X number of developments in a particular field here in kaggle and in the AI space in general, and analysing general consensus based on upvotes as well as similarity of ranking in other essays. \n\nthe general quality of what is written can also be assessed based on comments. It would have to be defined rather clearly though.\n\nconsidering the potential impracticality and opaqueness of this solution, I like the ideas suggested of moving away from medals or points, but community engagement is dependent on points and medals as well, and more engagement does produce a better summary of the landscape here. tough one to crack but my thoughts are\n\n- would be an ideal opportunity to trial out algorithms and methodologies (maybe using past evaluations to create a competition for developing such algorithms) to objectively evaluate essays, and get relatively similar results to what multiple different sets of expert panels say.\n- for the time being, consider that avoiding points and medals to such competitions may distort the community based on the panel's decisions (award swag instead?)\n\ninterested on peoples thoughts, and see where this goes in time as well", "2259008": "Totally agree, this is not ml or even cs comp", "2331669": "This is a nice competition\n", "2262933": "I strongly support the viewpoint of keeping a \"pure\" competition tier, with only objective scoring (metric based) for all the competitions. There is always the question of double medals rewards if you get medal in the analytic competition and for a Notebook in that analytic competition that is your support material for the same submission.", "2258536": "" } }