Hi Kagglers,
\nThanks to those who have weighed in on this competition counting towards competition rankings. We have heard strong demand from past analytics competition participants for a format that shows on profiles and counts towards progression. As a result, we had anticipated the reception to this format to be more unanimously embraced. Taking into account the valid pushback about abrupt impacts on progression, and based on feedback from our expert grading panel, we will be altering this competition to a format that offers points but not medals.
\nCompetitions have historically rewarded excellence in creating the most accurate model, yet this is often caveated with the fact that model accuracy is but one facet of a data scientist’s purview. If there is one thing that the recent rise of generative AI has made clear, it is that subjective and qualitative evaluations are now a critical part of data science. As a platform and community, it is important that we develop the capacity to work in subjective evaluation settings as well as we work in more traditional settings, where holdout accuracy is the key consideration.
\nThis competition is one experiment towards creating useful structures in this space. We believe the proposed task and grading process for this competition are set up to achieve rigorous participant rankings, reward our community for their contributions towards this AI Report, and result in a meaningful resource for the ML community.
\nThanks
", "messages": [ { "id": 2261845, "postDate": "2023-05-16T15:25:10.297Z", "content": "Hi Kagglers,
\nThanks to those who have weighed in on this competition counting towards competition rankings. We have heard strong demand from past analytics competition participants for a format that shows on profiles and counts towards progression. As a result, we had anticipated the reception to this format to be more unanimously embraced. Taking into account the valid pushback about abrupt impacts on progression, and based on feedback from our expert grading panel, we will be altering this competition to a format that offers points but not medals.
\nCompetitions have historically rewarded excellence in creating the most accurate model, yet this is often caveated with the fact that model accuracy is but one facet of a data scientist’s purview. If there is one thing that the recent rise of generative AI has made clear, it is that subjective and qualitative evaluations are now a critical part of data science. As a platform and community, it is important that we develop the capacity to work in subjective evaluation settings as well as we work in more traditional settings, where holdout accuracy is the key consideration.
\nThis competition is one experiment towards creating useful structures in this space. We believe the proposed task and grading process for this competition are set up to achieve rigorous participant rankings, reward our community for their contributions towards this AI Report, and result in a meaningful resource for the ML community.
\nThanks
", "rawMarkdown": "Hi Kagglers,\n\nThanks to those who have weighed in on this competition counting towards competition rankings. We have heard strong demand from past analytics competition participants for a format that shows on profiles and counts towards progression. As a result, we had anticipated the reception to this format to be more unanimously embraced. Taking into account the valid pushback about abrupt impacts on progression, and based on feedback from our expert grading panel, **we will be altering this competition to a format that offers points but not medals.**\n\nCompetitions have historically rewarded excellence in creating the most accurate model, yet this is often caveated with the fact that model accuracy is but one facet of a data scientist’s purview. If there is one thing that the recent rise of generative AI has made clear, it is that subjective and qualitative evaluations are now a critical part of data science. As a platform and community, it is important that we develop the capacity to work in subjective evaluation settings as well as we work in more traditional settings, where holdout accuracy is the key consideration.\n\nThis competition is one experiment towards creating useful structures in this space. We believe the proposed task and grading process for this competition are set up to achieve rigorous participant rankings, reward our community for their contributions towards this AI Report, and result in a meaningful resource for the ML community.\n\nThanks\n", "votes": 29 }, { "id": 2282633, "postDate": "2023-05-31T17:50:06.307Z", "content": "Please double-blind the review process! GMs are worried that their competitors will discriminate against them, but I believe the opposite will happen: It will be very tempting to think, \"This notebook was written by a top GM, so it must be good.\"
", "rawMarkdown": "Please double-blind the review process! GMs are worried that their competitors will discriminate against them, but I believe the opposite will happen: It will be very tempting to think, \"This notebook was written by a top GM, so it must be good.\"", "votes": 10 }, { "id": 2263057, "postDate": "2023-05-17T09:54:23.273Z", "content": "This still does not solve the issue that competitors decide on other competitors regarding points. From my point, the expert panel is not an objective authority. Imagine in any sport that your competitors decide on your scores.
", "rawMarkdown": "This still does not solve the issue that competitors decide on other competitors regarding points. From my point, the expert panel is not an objective authority. Imagine in any sport that your competitors decide on your scores.", "votes": 11, "replies": [ { "id": 2263559, "postDate": "2023-05-17T17:24:58.800Z", "content": "for instance, former athletes are picked to serve as Olympic judges for sports like gymnastics. By analogy, if \"retired\" Kaggle competition grandmasters will be (one day) members of the expert panel (who are currently not Kaggle competitors), we can also say that they may be former friends (or former teammates) of competitors who still compete on Kaggle and may make a subjective assessment. It is really impossible to guarantee 100% objectivity when it comes to human evaluation. unless Kaggle establishes a committee in the future (exactly like in the Olympics) and accepts the judges nominated by the federations(…) based on multiple conditions, etc(…), but I believe something like that can only happen in the Olympics :) But it's already a big step made by kaggle
", "rawMarkdown": "for instance, former athletes are picked to serve as Olympic judges for sports like gymnastics. By analogy, if \"retired\" Kaggle competition grandmasters will be (one day) members of the expert panel (who are currently not Kaggle competitors), we can also say that they may be former friends (or former teammates) of competitors who still compete on Kaggle and may make a subjective assessment. It is really impossible to guarantee 100% objectivity when it comes to human evaluation. unless Kaggle establishes a committee in the future (exactly like in the Olympics) and accepts the judges nominated by the federations(...) based on multiple conditions, etc(...), but I believe something like that can only happen in the Olympics :) But it's already a big step made by kaggle", "votes": 2 }, { "id": 2263831, "postDate": "2023-05-18T00:48:43.570Z", "content": "Imo it should be fine provided some safeguards (judges not participating, competitors being anonymised, transparent rating grid, re-evaluation if disagreement between judges…).
", "rawMarkdown": "Imo it should be fine provided some safeguards (judges not participating, competitors being anonymised, transparent rating grid, re-evaluation if disagreement between judges...).", "votes": 6, "replies": [ { "id": 2264193, "postDate": "2023-05-18T08:32:11.183Z", "content": "judges are not competitors in the contest, see here: https://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/400084#2256547
\n\"competitors\" is % the global leaderboard. Imagine a situation in which competitors x and y already hold titles in the competition tier (as well as how points earned in an analytical competition can change the overall rank and what a subjective evaluation can do for example…).
\nIt's a great idea to keep competitors' notebooks anonymous (but Kaggle should create a feature for that). For \"re-evaluation if disagreement between judges\": there is only 1 judge (a competition grandmaster) per category (making a total of 7 judges) + 3 peer reviews for each notebook. I hope the participants really make constructive peer reviews, anonymity will really help, otherwise, we will see peer reviews similar to the upvotes on the notebooks tier: sometimes it seems like a beauty contest= beautiful notebook (though a beauty contest is more than merely receiving \"congratulations\" for one's beauty).
The competition aims and encourages us to close the gap between submissions and the quality of work in actual peer-reviewed journals which have blind reviews. However, I do not think the reviews will be blind or complete anonymity will be maintained. To combat this, we can go in the opposite direction and make the judging process completely transparent, making sure everyone would then know how the participants are being scored against a given rubric. I suppose the answer to this would be expert review (having an additional guest judge that is not participating in the competition and preferably is not even on the platform) from a specific domain to work with the Kaggle team, except of course- in the topic where we're required to detail on the impact of Kaggle competitions. External reviews can definitely work really well in public competitions like this one, but there are also potential constraints like finding volunteers who can adequately represent a section and give enough time to grade an essay. I feel peer reviews are definitely beneficial in the sense that they provide the Kaggle team and the writer with a lot of information, like the perspective of the reader. Though those peer reviews would not directly impact any rankings, they would in the end be crucial in how the final articles are shaped.
", "rawMarkdown": "The competition aims and encourages us to close the gap between submissions and the quality of work in actual peer-reviewed journals which have blind reviews. However, I do not think the reviews will be blind or complete anonymity will be maintained. To combat this, we can go in the opposite direction and make the judging process completely transparent, making sure everyone would then know how the participants are being scored against a given rubric. I suppose the answer to this would be expert review (having an additional guest judge that is not participating in the competition and preferably is not even on the platform) from a specific domain to work with the Kaggle team, except of course- in the topic where we're required to detail on the impact of Kaggle competitions. External reviews can definitely work really well in public competitions like this one, but there are also potential constraints like finding volunteers who can adequately represent a section and give enough time to grade an essay. I feel peer reviews are definitely beneficial in the sense that they provide the Kaggle team and the writer with a lot of information, like the perspective of the reader. Though those peer reviews would not directly impact any rankings, they would in the end be crucial in how the final articles are shaped. ", "votes": 1 } ] } ] }, { "id": 2264068, "postDate": "2023-05-18T06:01:20.457Z", "content": "I would guess that only top Kaggle competitors pay attention to points, while medals provide a feeling of accomplishment and promote competition among a broad range of participants including first timers, intermittant Kagglers, those who finally earned their first silver, etc.
\nAs you look for alternative rewards for excellence in analytics competitions, you might consider awarding bonus medals in the Notebooks category. For example, the Kaggle progression system treats Notebooks medals at something like 4:1 ratio to Competitions medals, so winning essays might be awarded 4 gold Notebooks medals, and similar 4x silver and 4x bronze awards could be given for other highly rated essays.
\nI bet something like that would create a lot of interest without affecting those who are concerned about impacts on global rankings or Competitions status.
", "rawMarkdown": "I would guess that only top Kaggle competitors pay attention to points, while medals provide a feeling of accomplishment and promote competition among a broad range of participants including first timers, intermittant Kagglers, those who finally earned their first silver, etc.\n\nAs you look for alternative rewards for excellence in analytics competitions, you might consider awarding bonus medals in the Notebooks category. For example, the Kaggle progression system treats Notebooks medals at something like 4:1 ratio to Competitions medals, so winning essays might be awarded 4 gold Notebooks medals, and similar 4x silver and 4x bronze awards could be given for other highly rated essays. \n\nI bet something like that would create a lot of interest without affecting those who are concerned about impacts on global rankings or Competitions status.", "votes": 4, "replies": [ { "id": 2264175, "postDate": "2023-05-18T08:21:00.610Z", "content": "The suggestion for notebook medals as you envisioned it is a good one, but according to kaggle (and it is true, I see people complain about that…) people want to see a leaderboard and scores on analytical competitions, this is the idea behind it. however, it could be similar to a community competition (neither points nor medals), just a leaderboard and scores, even though some community competitions are similar to featured competitions.
", "rawMarkdown": "The suggestion for notebook medals as you envisioned it is a good one, but according to kaggle (and it is true, I see people complain about that...) people want to see a leaderboard and scores on analytical competitions, this is the idea behind it. however, it could be similar to a community competition (neither points nor medals), just a leaderboard and scores, even though some community competitions are similar to featured competitions.", "replies": [ { "id": 2264865, "postDate": "2023-05-18T19:36:03.657Z", "content": "I agree a leaderboard would add to these competitions and allow for competitors to submit drafts that improve over the course of the competition.
\nMy suggestion on that is to have an AI model score be the public LB that can provide immediate scoring and the human reviewer panel scores be the private LB that determines the final result.
", "rawMarkdown": "I agree a leaderboard would add to these competitions and allow for competitors to submit drafts that improve over the course of the competition. \n\nMy suggestion on that is to have an AI model score be the public LB that can provide immediate scoring and the human reviewer panel scores be the private LB that determines the final result. " } ] } ] }, { "id": 2308762, "postDate": "2023-06-19T07:37:12.483Z", "content": "It's great to see Kaggle actively adapting and evolving based on community feedback. The decision to alter the competition format to offer points without medals shows a willingness to find a balance that considers the needs and perspectives of participants. This update reflects Kaggle's commitment to fostering a diverse and inclusive data science community.
", "rawMarkdown": "It's great to see Kaggle actively adapting and evolving based on community feedback. The decision to alter the competition format to offer points without medals shows a willingness to find a balance that considers the needs and perspectives of participants. This update reflects Kaggle's commitment to fostering a diverse and inclusive data science community.", "votes": 1 }, { "id": 2270902, "postDate": "2023-05-23T13:40:25.767Z", "content": "Thanks for the update!
\nIn my view this issue had and still has two sides to it.
\nOpponents:
\nSupporters:
\nThe way I see it, the current update addresses only one of the four I listed, i.e., protects the competitions GM tier or any other tier due to the no medal rule leaving the rest unresolved. So here is my idea on how the other points could have been fairly addressed except the first one, a judging panel is unavoidable.
\n\n\nAward medals and points but consider each entry as a team of FIVE (the max possible competition team members allowed).
\n
This way:
\nSo at least for future analytics competition it could be something to ponder.
\nP.S. Reducing the ranking points was already raised by Psi.
", "rawMarkdown": "Thanks for the update!\n\nIn my view this issue had and still has two sides to it.\n\n**Opponents**:\n\n1. Having a **panel of judges** as opposed to metric based evaluation leading to a possible bias. \n2. The impact of the (possibly biased) **ranking point**s will have on the global competition ranking. \n3. fear of analytics competition (again due to the human evaluation method) offering an easy route to the competition GM tier and **inflating the GM tier**.\n\n**Supporters**:\n\n1. Winners of analytics competitions will finally have something to show (on their kaggle profile) for their effort.\n\nThe way I see it, the current update addresses only one of the four I listed, i.e., protects the competitions GM tier or any other tier due to the no medal rule leaving the rest unresolved. So here is my idea on how the other points could have been fairly addressed except the first one, a judging panel is unavoidable. \n\n> **Award medals and points but consider each entry as a team of FIVE (the max possible competition team members allowed).** \n\nThis way:\n\n- winner will get only **a fifth of the total points** thereby addressing the point inflation issue. \n- since it is considered as **a team medal** it effectively blocks-off a route to competition GM tier (the solo gold will have to be earned somewhere else in the conventional ML comps.)\n- but as @rturley commented first timers, intermittent Kagglers,etc. will have a chance of getting a medal if they end up winning. I believe this group of kagglers value medals more than points. Ranking points for unranked kaggler is as good as nothing. \n\nSo at least for future analytics competition it could be something to ponder. \n\nP.S. Reducing the ranking points was already raised by Psi. \n", "votes": 1 }, { "id": 2262052, "postDate": "2023-05-16T17:47:16.410Z", "content": "The Experiment is Hot! Community wouldl love to flame around 🔥
", "rawMarkdown": "The Experiment is Hot! Community wouldl love to flame around 🔥", "votes": -3 }, { "id": 2269779, "postDate": "2023-05-22T17:45:33.910Z", "content": "You're most welcome
", "rawMarkdown": "You're most welcome", "votes": -2 }, { "id": 2262435, "postDate": "2023-05-16T23:31:08.980Z", "content": "This is an interesting and quite carefully considered take on this. Agree on the subjective evaluation perspective of generative AI, but there maybe better and more carefully designed ways to do this. For example multiple panels of experts can be used from different parts of the world to reduce the potential for any bias in terms of decision making from one panel, as its a subset of a large community.
\nAdditionally, the community's own voice can be quantified based on the amount of upvotes, the level of engagement, the sentiment of the comments, as well as for example quantifying how many of the points an essay covered out of the total number of topics brought up by the community, weighted by the importance of each topic.
\nWhile this may seem complicated and would surely have loopholes in its own way, it would still be better to objectively quantify a score based on a metric that can be widely understood and accepted by the wider kaggle community, than the black box subjective evaluations of expert panels.the better we can evaluate subjective topics, the better we can get at understanding and quantifying subjective topics as well. maybe a hot-take, but it would be interesting to see crowd sourced developments in design of strategies that can evaluate subjective topics to a degree of significant detail
", "rawMarkdown": "This is an interesting and quite carefully considered take on this. Agree on the subjective evaluation perspective of generative AI, but there maybe better and more carefully designed ways to do this. For example multiple panels of experts can be used from different parts of the world to reduce the potential for any bias in terms of decision making from one panel, as its a subset of a large community. \n\nAdditionally, the community's own voice can be quantified based on the amount of upvotes, the level of engagement, the sentiment of the comments, as well as for example quantifying how many of the points an essay covered out of the total number of topics brought up by the community, weighted by the importance of each topic.\n\nWhile this may seem complicated and would surely have loopholes in its own way, it would still be better to objectively quantify a score based on a metric that can be widely understood and accepted by the wider kaggle community, than the black box subjective evaluations of expert panels.the better we can evaluate subjective topics, the better we can get at understanding and quantifying subjective topics as well. maybe a hot-take, but it would be interesting to see crowd sourced developments in design of strategies that can evaluate subjective topics to a degree of significant detail" }, { "id": 2281106, "postDate": "2023-05-30T15:20:54.043Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 2282633, "author_name": "Adrian Dip", "author_url": "", "post_date": "2023-05-31T17:50:06.307000", "content": "Please double-blind the review process! GMs are worried that their competitors will discriminate against them, but I believe the opposite will happen: It will be very tempting to think, \"This notebook was written by a top GM, so it must be good.\"
", "votes": 10, "replies": [] }, { "id": 2263057, "author_name": "Psi", "author_url": "", "post_date": "2023-05-17T09:54:23.273000", "content": "This still does not solve the issue that competitors decide on other competitors regarding points. From my point, the expert panel is not an objective authority. Imagine in any sport that your competitors decide on your scores.
", "votes": 11, "replies": [ { "id": 2263559, "author_name": "", "author_url": "", "post_date": "2023-05-17T17:24:58.800000", "content": "for instance, former athletes are picked to serve as Olympic judges for sports like gymnastics. By analogy, if \"retired\" Kaggle competition grandmasters will be (one day) members of the expert panel (who are currently not Kaggle competitors), we can also say that they may be former friends (or former teammates) of competitors who still compete on Kaggle and may make a subjective assessment. It is really impossible to guarantee 100% objectivity when it comes to human evaluation. unless Kaggle establishes a committee in the future (exactly like in the Olympics) and accepts the judges nominated by the federations(…) based on multiple conditions, etc(…), but I believe something like that can only happen in the Olympics :) But it's already a big step made by kaggle
", "votes": 2, "replies": [] }, { "id": 2263831, "author_name": "Lucas Morin", "author_url": "", "post_date": "2023-05-18T00:48:43.570000", "content": "Imo it should be fine provided some safeguards (judges not participating, competitors being anonymised, transparent rating grid, re-evaluation if disagreement between judges…).
", "votes": 6, "replies": [ { "id": 2264193, "author_name": "", "author_url": "", "post_date": "2023-05-18T08:32:11.183000", "content": "judges are not competitors in the contest, see here: https://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/400084#2256547
\n\"competitors\" is % the global leaderboard. Imagine a situation in which competitors x and y already hold titles in the competition tier (as well as how points earned in an analytical competition can change the overall rank and what a subjective evaluation can do for example…).
\nIt's a great idea to keep competitors' notebooks anonymous (but Kaggle should create a feature for that). For \"re-evaluation if disagreement between judges\": there is only 1 judge (a competition grandmaster) per category (making a total of 7 judges) + 3 peer reviews for each notebook. I hope the participants really make constructive peer reviews, anonymity will really help, otherwise, we will see peer reviews similar to the upvotes on the notebooks tier: sometimes it seems like a beauty contest= beautiful notebook (though a beauty contest is more than merely receiving \"congratulations\" for one's beauty).
The competition aims and encourages us to close the gap between submissions and the quality of work in actual peer-reviewed journals which have blind reviews. However, I do not think the reviews will be blind or complete anonymity will be maintained. To combat this, we can go in the opposite direction and make the judging process completely transparent, making sure everyone would then know how the participants are being scored against a given rubric. I suppose the answer to this would be expert review (having an additional guest judge that is not participating in the competition and preferably is not even on the platform) from a specific domain to work with the Kaggle team, except of course- in the topic where we're required to detail on the impact of Kaggle competitions. External reviews can definitely work really well in public competitions like this one, but there are also potential constraints like finding volunteers who can adequately represent a section and give enough time to grade an essay. I feel peer reviews are definitely beneficial in the sense that they provide the Kaggle team and the writer with a lot of information, like the perspective of the reader. Though those peer reviews would not directly impact any rankings, they would in the end be crucial in how the final articles are shaped.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2264068, "author_name": "RS Turley", "author_url": "", "post_date": "2023-05-18T06:01:20.457000", "content": "I would guess that only top Kaggle competitors pay attention to points, while medals provide a feeling of accomplishment and promote competition among a broad range of participants including first timers, intermittant Kagglers, those who finally earned their first silver, etc.
\nAs you look for alternative rewards for excellence in analytics competitions, you might consider awarding bonus medals in the Notebooks category. For example, the Kaggle progression system treats Notebooks medals at something like 4:1 ratio to Competitions medals, so winning essays might be awarded 4 gold Notebooks medals, and similar 4x silver and 4x bronze awards could be given for other highly rated essays.
\nI bet something like that would create a lot of interest without affecting those who are concerned about impacts on global rankings or Competitions status.
", "votes": 4, "replies": [ { "id": 2264175, "author_name": "", "author_url": "", "post_date": "2023-05-18T08:21:00.610000", "content": "The suggestion for notebook medals as you envisioned it is a good one, but according to kaggle (and it is true, I see people complain about that…) people want to see a leaderboard and scores on analytical competitions, this is the idea behind it. however, it could be similar to a community competition (neither points nor medals), just a leaderboard and scores, even though some community competitions are similar to featured competitions.
", "votes": 0, "replies": [ { "id": 2264865, "author_name": "RS Turley", "author_url": "", "post_date": "2023-05-18T19:36:03.657000", "content": "I agree a leaderboard would add to these competitions and allow for competitors to submit drafts that improve over the course of the competition.
\nMy suggestion on that is to have an AI model score be the public LB that can provide immediate scoring and the human reviewer panel scores be the private LB that determines the final result.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2308762, "author_name": "Pooja Chauhan", "author_url": "", "post_date": "2023-06-19T07:37:12.483000", "content": "It's great to see Kaggle actively adapting and evolving based on community feedback. The decision to alter the competition format to offer points without medals shows a willingness to find a balance that considers the needs and perspectives of participants. This update reflects Kaggle's commitment to fostering a diverse and inclusive data science community.
", "votes": 1, "replies": [] }, { "id": 2270902, "author_name": "A Legacy Grandmaster!", "author_url": "", "post_date": "2023-05-23T13:40:25.767000", "content": "Thanks for the update!
\nIn my view this issue had and still has two sides to it.
\nOpponents:
\nSupporters:
\nThe way I see it, the current update addresses only one of the four I listed, i.e., protects the competitions GM tier or any other tier due to the no medal rule leaving the rest unresolved. So here is my idea on how the other points could have been fairly addressed except the first one, a judging panel is unavoidable.
\n\n\nAward medals and points but consider each entry as a team of FIVE (the max possible competition team members allowed).
\n
This way:
\nSo at least for future analytics competition it could be something to ponder.
\nP.S. Reducing the ranking points was already raised by Psi.
", "votes": 1, "replies": [] }, { "id": 2262052, "author_name": "Ruslan Isaev", "author_url": "", "post_date": "2023-05-16T17:47:16.410000", "content": "The Experiment is Hot! Community wouldl love to flame around 🔥
", "votes": -3, "replies": [] }, { "id": 2269779, "author_name": "Abdullah.O.Usman", "author_url": "", "post_date": "2023-05-22T17:45:33.910000", "content": "You're most welcome
", "votes": -2, "replies": [] }, { "id": 2262435, "author_name": "Rahul Namboodiri", "author_url": "", "post_date": "2023-05-16T23:31:08.980000", "content": "This is an interesting and quite carefully considered take on this. Agree on the subjective evaluation perspective of generative AI, but there maybe better and more carefully designed ways to do this. For example multiple panels of experts can be used from different parts of the world to reduce the potential for any bias in terms of decision making from one panel, as its a subset of a large community.
\nAdditionally, the community's own voice can be quantified based on the amount of upvotes, the level of engagement, the sentiment of the comments, as well as for example quantifying how many of the points an essay covered out of the total number of topics brought up by the community, weighted by the importance of each topic.
\nWhile this may seem complicated and would surely have loopholes in its own way, it would still be better to objectively quantify a score based on a metric that can be widely understood and accepted by the wider kaggle community, than the black box subjective evaluations of expert panels.the better we can evaluate subjective topics, the better we can get at understanding and quantifying subjective topics as well. maybe a hot-take, but it would be interesting to see crowd sourced developments in design of strategies that can evaluate subjective topics to a degree of significant detail
", "votes": 0, "replies": [] }, { "id": 2281106, "author_name": "", "author_url": "", "post_date": "2023-05-30T15:20:54.043000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "2261845": "Hi Kagglers,\n\nThanks to those who have weighed in on this competition counting towards competition rankings. We have heard strong demand from past analytics competition participants for a format that shows on profiles and counts towards progression. As a result, we had anticipated the reception to this format to be more unanimously embraced. Taking into account the valid pushback about abrupt impacts on progression, and based on feedback from our expert grading panel, **we will be altering this competition to a format that offers points but not medals.**\n\nCompetitions have historically rewarded excellence in creating the most accurate model, yet this is often caveated with the fact that model accuracy is but one facet of a data scientist’s purview. If there is one thing that the recent rise of generative AI has made clear, it is that subjective and qualitative evaluations are now a critical part of data science. As a platform and community, it is important that we develop the capacity to work in subjective evaluation settings as well as we work in more traditional settings, where holdout accuracy is the key consideration.\n\nThis competition is one experiment towards creating useful structures in this space. We believe the proposed task and grading process for this competition are set up to achieve rigorous participant rankings, reward our community for their contributions towards this AI Report, and result in a meaningful resource for the ML community.\n\nThanks\n", "2282633": "Please double-blind the review process! GMs are worried that their competitors will discriminate against them, but I believe the opposite will happen: It will be very tempting to think, \"This notebook was written by a top GM, so it must be good.\"", "2263057": "This still does not solve the issue that competitors decide on other competitors regarding points. From my point, the expert panel is not an objective authority. Imagine in any sport that your competitors decide on your scores.", "2264068": "I would guess that only top Kaggle competitors pay attention to points, while medals provide a feeling of accomplishment and promote competition among a broad range of participants including first timers, intermittant Kagglers, those who finally earned their first silver, etc.\n\nAs you look for alternative rewards for excellence in analytics competitions, you might consider awarding bonus medals in the Notebooks category. For example, the Kaggle progression system treats Notebooks medals at something like 4:1 ratio to Competitions medals, so winning essays might be awarded 4 gold Notebooks medals, and similar 4x silver and 4x bronze awards could be given for other highly rated essays. \n\nI bet something like that would create a lot of interest without affecting those who are concerned about impacts on global rankings or Competitions status.", "2308762": "It's great to see Kaggle actively adapting and evolving based on community feedback. The decision to alter the competition format to offer points without medals shows a willingness to find a balance that considers the needs and perspectives of participants. This update reflects Kaggle's commitment to fostering a diverse and inclusive data science community.", "2270902": "Thanks for the update!\n\nIn my view this issue had and still has two sides to it.\n\n**Opponents**:\n\n1. Having a **panel of judges** as opposed to metric based evaluation leading to a possible bias. \n2. The impact of the (possibly biased) **ranking point**s will have on the global competition ranking. \n3. fear of analytics competition (again due to the human evaluation method) offering an easy route to the competition GM tier and **inflating the GM tier**.\n\n**Supporters**:\n\n1. Winners of analytics competitions will finally have something to show (on their kaggle profile) for their effort.\n\nThe way I see it, the current update addresses only one of the four I listed, i.e., protects the competitions GM tier or any other tier due to the no medal rule leaving the rest unresolved. So here is my idea on how the other points could have been fairly addressed except the first one, a judging panel is unavoidable. \n\n> **Award medals and points but consider each entry as a team of FIVE (the max possible competition team members allowed).** \n\nThis way:\n\n- winner will get only **a fifth of the total points** thereby addressing the point inflation issue. \n- since it is considered as **a team medal** it effectively blocks-off a route to competition GM tier (the solo gold will have to be earned somewhere else in the conventional ML comps.)\n- but as @rturley commented first timers, intermittent Kagglers,etc. will have a chance of getting a medal if they end up winning. I believe this group of kagglers value medals more than points. Ranking points for unranked kaggler is as good as nothing. \n\nSo at least for future analytics competition it could be something to ponder. \n\nP.S. Reducing the ranking points was already raised by Psi. \n", "2262052": "The Experiment is Hot! Community wouldl love to flame around 🔥", "2269779": "You're most welcome", "2262435": "This is an interesting and quite carefully considered take on this. Agree on the subjective evaluation perspective of generative AI, but there maybe better and more carefully designed ways to do this. For example multiple panels of experts can be used from different parts of the world to reduce the potential for any bias in terms of decision making from one panel, as its a subset of a large community. \n\nAdditionally, the community's own voice can be quantified based on the amount of upvotes, the level of engagement, the sentiment of the comments, as well as for example quantifying how many of the points an essay covered out of the total number of topics brought up by the community, weighted by the importance of each topic.\n\nWhile this may seem complicated and would surely have loopholes in its own way, it would still be better to objectively quantify a score based on a metric that can be widely understood and accepted by the wider kaggle community, than the black box subjective evaluations of expert panels.the better we can evaluate subjective topics, the better we can get at understanding and quantifying subjective topics as well. maybe a hot-take, but it would be interesting to see crowd sourced developments in design of strategies that can evaluate subjective topics to a degree of significant detail", "2281106": "" } }