Dear orgs,
\nhaving a bit more instructions on specific Notebooks to be graded would definitely be helpful.
\nSome problems:
\nHence, the questions:
\nIn hindsight, probably, submissions with a single URL to an essay need to have been obligatory before July 5th. Then, it would've been easy to publish a whole list of ~350 Notebooks that need to be reviewed.
", "messages": [ { "id": 2332807, "postDate": "2023-07-06T12:48:19.233Z", "content": "Dear orgs,
\nhaving a bit more instructions on specific Notebooks to be graded would definitely be helpful.
\nSome problems:
\nHence, the questions:
\nIn hindsight, probably, submissions with a single URL to an essay need to have been obligatory before July 5th. Then, it would've been easy to publish a whole list of ~350 Notebooks that need to be reviewed.
", "rawMarkdown": "Dear orgs, \n\nhaving a bit more instructions on specific Notebooks to be graded would definitely be helpful. \n\nSome problems:\n\n- Notebooks with essays are barely distinguishable from supplementary notebooks (e.g. [essay](https://www.kaggle.com/code/thedrcat/how-to-win-a-kaggle-competition) + [a supplement](https://www.kaggle.com/code/thedrcat/using-llms-to-extract-structured-data/notebook))\n- Apparently, some teams released >1 notebooks while the [submission instructions](https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/submission-instructions) clearly state that \"each team should write a single essay focusing on a single topic.\" (e.g. [essay1](https://www.kaggle.com/code/sophieb/a-journey-through-the-text-data-competitions) and [essay2](https://www.kaggle.com/code/sophieb/ai-report-text-data-kaggle-and-arxiv) from the same authors or as many as [16 notebooks](https://www.kaggle.com/mpwolke/code?userId=3012786&sortBy=dateCreated&tab=profile) released by a single individual)\n\nHence, the questions:\n - how do we know which notebooks to review?\n - how do we mitigate the risk that reviews are issued to notebooks that were not supposed to be reviewed?\n\nIn hindsight, probably, submissions with a single URL to an essay need to have been obligatory before July 5th. Then, it would've been easy to publish a whole list of ~350 Notebooks that need to be reviewed. ", "votes": 12 }, { "id": 2332924, "postDate": "2023-07-06T14:18:05.687Z", "content": "I think these are great questions. I am as well a bit confused about which notebooks are the final submissions.
", "rawMarkdown": "I think these are great questions. I am as well a bit confused about which notebooks are the final submissions.", "votes": 3 }, { "id": 2333395, "postDate": "2023-07-06T23:29:31.673Z", "content": "Hi Kashnitsky,
\nOpen only those that have that Score 1.
\nI think the intention is to read all (or as many as we could) that have Score 1 (=submitted).
\nBy the way, I'm the one that published 16 Notebooks cause I was exploring the dataset. At the beginning I hadn't understood that it was to write an essay : )
\nHowever, I submitted ONLY one. And I don't bother about votes. Just to learn.
\nNow, I'm reading the recently created, most part was written by beginners.
\nThey have a lot to teach us.
Well this is confusing. My submission is not showing up with a Score in front of it. What does that mean, and how can I get that to happen? The URL in my submission file correctly goes to the public notebook.
", "rawMarkdown": "Well this is confusing. My submission is not showing up with a Score in front of it. What does that mean, and how can I get that to happen? The URL in my submission file correctly goes to the public notebook.", "votes": 1, "replies": [ { "id": 2333413, "postDate": "2023-07-07T00:28:06.337Z", "content": "Thanks, @mpwolke, it's helpful. Not intuitive though :) the recall is not perfect as not all of the participants submitted through the notebook (I first did it manually). But still, this helps to filter out some trash.
\n@esaund is you create the submission file in the notebook itself, run the notebook, go to the Output tab, find the submission file there, and hit \"Submit\" – your notebook will get a score of 1.
", "rawMarkdown": "Thanks, @mpwolke, it's helpful. Not intuitive though :) the recall is not perfect as not all of the participants submitted through the notebook (I first did it manually). But still, this helps to filter out some trash. \n\n@esaund is you create the submission file in the notebook itself, run the notebook, go to the Output tab, find the submission file there, and hit \"Submit\" – your notebook will get a score of 1. ", "votes": 1 }, { "id": 2333415, "postDate": "2023-07-07T00:33:58.003Z", "content": "I read both of your Notebooks Eric. Indeed, both are Not showing any Score at all. Though you're on the LB.
\nBesides, it's written that you're the 59th to make a submission.
There should be a Score 1 even inside your Notebook.
", "rawMarkdown": "I read both of your Notebooks Eric. Indeed, both are Not showing any Score at all. Though you're on the LB.\nBesides, it's written that you're the 59th to make a submission.\n\nThere should be a Score 1 even inside your Notebook." }, { "id": 2333424, "postDate": "2023-07-07T00:51:51.890Z", "content": "Thanks @kashnitsky but the Output tab only shows an Output Data cell with something that can be copied and pasted to a clipboard. No Submit button.
", "rawMarkdown": "Thanks @kashnitsky but the Output tab only shows an Output Data cell with something that can be copied and pasted to a clipboard. No Submit button." }, { "id": 2333450, "postDate": "2023-07-07T01:44:50.997Z", "content": "Hi Eric,
\nYou only have a button on Output if you wrote something.to_csv and the URLs:
\nExamples:
\nsubmission.to_csv('submission.csv',index = False) or
\ndf.to_csv('submission.csv', index=False)
Otherwise, only the Output will be there with No submission button to make.
\nwe have still options to submit (AFTER YOU WROTE SUBMISSION TO CSV and the URLs):
\nNotebook, on the 3 Dots beside copy and Edit. Click on the dots, that will open menu:
\n\"submit to competition\" is the 4th line.
Inside the Notebook, on the right side below Models: another submit to competition.
\nDON'T FORGET to write SOMETHING LIKE THAT:
\ndf = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')
\ndf.iloc[0, 1] = 'Generative AI'
\ndf.iloc[1, 1] = 'https://www.kaggle.com/code/amitprasad/generative-ai-turbocharging-ai-hopes-and-fears'
\ndf.iloc[2, 1] = '0'
\ndf.iloc[3, 1] = '0'
\ndf.iloc[4, 1] = '0'
df.to_csv('submission.csv', index=False)
", "rawMarkdown": "Hi Eric,\n\nYou only have a button on Output if you wrote something.to_csv and the URLs:\n\nExamples:\nsubmission.to_csv('submission.csv',index = False) or\ndf.to_csv('submission.csv', index=False)\n\nOtherwise, only the Output will be there with No submission button to make.\n\nwe have still options to submit (AFTER YOU WROTE SUBMISSION TO CSV and the URLs): \n\nNotebook, on the 3 Dots beside copy and Edit. Click on the dots, that will open menu:\n\"submit to competition\" is the 4th line.\n\nInside the Notebook, on the right side below Models: another submit to competition.\n\nDON'T FORGET to write SOMETHING LIKE THAT:\n\n#First\n\ndf = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\n\n#Then your own URLs\n\ndf.iloc[0, 1] = 'Generative AI'\ndf.iloc[1, 1] = 'https://www.kaggle.com/code/amitprasad/generative-ai-turbocharging-ai-hopes-and-fears'\ndf.iloc[2, 1] = '0'\ndf.iloc[3, 1] = '0'\ndf.iloc[4, 1] = '0'\n\n#Finally\ndf.to_csv('submission.csv', index=False)\n" } ] }, { "id": 2333632, "postDate": "2023-07-07T05:01:02.970Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2333781, "postDate": "2023-07-07T06:58:47.883Z", "content": "Nothing indicates in the instructions of the submission that it should be submitted via notebook (as in the case of the code competition on Kaggle) I only submitted the updated csv file which contains the URL of my public notebook (as required). And even in the \"Pinned Q&A Thread\", speaking of submissions, @paultimothymooney Paul just talks about uploading the csv file, see here (in his response): https://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/400084#2255821
\nThis is:
\n\"…If you follow the eligibility guidelines and then upload a submission.csv file (where you have replaced these empty URLs with valid ones) …\"
I believe the issue will be resolved if the participant (or team) writes at the beginning of each notebook that has been the subject of the submission. Essay Topic : ……, this indicates that the notebook is the one that needs to be reviewed, and it also orients on the topic (I've seen cases where sometimes it's not clear exactly if the topic of the notebook, for eg, is about Kaggle competitions or the 7th category: Other…: \"everything\" is in the notebook…)
", "rawMarkdown": "I believe the issue will be resolved if the participant (or team) writes at the beginning of each notebook that has been the subject of the submission. Essay Topic : ......, this indicates that the notebook is the one that needs to be reviewed, and it also orients on the topic (I've seen cases where sometimes it's not clear exactly if the topic of the notebook, for eg, is about Kaggle competitions or the 7th category: Other...: \"everything\" is in the notebook...)", "votes": 1, "replies": [ { "id": 2333230, "postDate": "2023-07-06T18:33:54.263Z", "content": "Indeed, that’d somewhat help. Not a perfect solution though, but rather “what if everyone does X” :) some instruction from the org team will do better.
", "rawMarkdown": "Indeed, that’d somewhat help. Not a perfect solution though, but rather “what if everyone does X” :) some instruction from the org team will do better. ", "votes": 1 } ] }, { "id": 2332949, "postDate": "2023-07-06T14:45:44.110Z", "content": "Great questions! I agree its been really hard to figure out which notebooks are final essays and which ones are just noise. Earlier on I was seeing all these notebooks being submitted that clearly weren't essays so I decided to just stick to the rules without getting distracted with people just trying to get upvotes.
", "rawMarkdown": "Great questions! I agree its been really hard to figure out which notebooks are final essays and which ones are just noise. Earlier on I was seeing all these notebooks being submitted that clearly weren't essays so I decided to just stick to the rules without getting distracted with people just trying to get upvotes.", "votes": 1, "replies": [ { "id": 2333010, "postDate": "2023-07-06T15:31:37.203Z", "content": "Indeed. With a reservation that sticking to the rules might also mean providing feedback on non-essay notebooks.
", "rawMarkdown": "Indeed. With a reservation that sticking to the rules might also mean providing feedback on non-essay notebooks. ", "votes": 1 } ] }, { "id": 2333401, "postDate": "2023-07-07T00:00:49.700Z", "content": "I observe the notebooks are continuously updated by their authors. It would be nice if the notebook content is frozen during peer review feedback process. This will ensure consistency.
", "rawMarkdown": "I observe the notebooks are continuously updated by their authors. It would be nice if the notebook content is frozen during peer review feedback process. This will ensure consistency. " }, { "id": 2333243, "postDate": "2023-07-06T18:52:11.320Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2332924, "author_name": "CroDoc", "author_url": "", "post_date": "2023-07-06T14:18:05.687000", "content": "I think these are great questions. I am as well a bit confused about which notebooks are the final submissions.
", "votes": 3, "replies": [] }, { "id": 2333395, "author_name": "Marília Prata", "author_url": "", "post_date": "2023-07-06T23:29:31.673000", "content": "Hi Kashnitsky,
\nOpen only those that have that Score 1.
\nI think the intention is to read all (or as many as we could) that have Score 1 (=submitted).
\nBy the way, I'm the one that published 16 Notebooks cause I was exploring the dataset. At the beginning I hadn't understood that it was to write an essay : )
\nHowever, I submitted ONLY one. And I don't bother about votes. Just to learn.
\nNow, I'm reading the recently created, most part was written by beginners.
\nThey have a lot to teach us.
Well this is confusing. My submission is not showing up with a Score in front of it. What does that mean, and how can I get that to happen? The URL in my submission file correctly goes to the public notebook.
", "votes": 1, "replies": [ { "id": 2333413, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2023-07-07T00:28:06.337000", "content": "Thanks, @mpwolke, it's helpful. Not intuitive though :) the recall is not perfect as not all of the participants submitted through the notebook (I first did it manually). But still, this helps to filter out some trash.
\n@esaund is you create the submission file in the notebook itself, run the notebook, go to the Output tab, find the submission file there, and hit \"Submit\" – your notebook will get a score of 1.
", "votes": 1, "replies": [] }, { "id": 2333415, "author_name": "Marília Prata", "author_url": "", "post_date": "2023-07-07T00:33:58.003000", "content": "I read both of your Notebooks Eric. Indeed, both are Not showing any Score at all. Though you're on the LB.
\nBesides, it's written that you're the 59th to make a submission.
There should be a Score 1 even inside your Notebook.
", "votes": 0, "replies": [] }, { "id": 2333424, "author_name": "Eric Saund", "author_url": "", "post_date": "2023-07-07T00:51:51.890000", "content": "Thanks @kashnitsky but the Output tab only shows an Output Data cell with something that can be copied and pasted to a clipboard. No Submit button.
", "votes": 0, "replies": [] }, { "id": 2333450, "author_name": "Marília Prata", "author_url": "", "post_date": "2023-07-07T01:44:50.997000", "content": "Hi Eric,
\nYou only have a button on Output if you wrote something.to_csv and the URLs:
\nExamples:
\nsubmission.to_csv('submission.csv',index = False) or
\ndf.to_csv('submission.csv', index=False)
Otherwise, only the Output will be there with No submission button to make.
\nwe have still options to submit (AFTER YOU WROTE SUBMISSION TO CSV and the URLs):
\nNotebook, on the 3 Dots beside copy and Edit. Click on the dots, that will open menu:
\n\"submit to competition\" is the 4th line.
Inside the Notebook, on the right side below Models: another submit to competition.
\nDON'T FORGET to write SOMETHING LIKE THAT:
\ndf = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')
\ndf.iloc[0, 1] = 'Generative AI'
\ndf.iloc[1, 1] = 'https://www.kaggle.com/code/amitprasad/generative-ai-turbocharging-ai-hopes-and-fears'
\ndf.iloc[2, 1] = '0'
\ndf.iloc[3, 1] = '0'
\ndf.iloc[4, 1] = '0'
df.to_csv('submission.csv', index=False)
", "votes": 0, "replies": [] } ] }, { "id": 2333632, "author_name": "", "author_url": "", "post_date": "2023-07-07T05:01:02.970000", "content": "", "votes": 0, "replies": [] }, { "id": 2333781, "author_name": "", "author_url": "", "post_date": "2023-07-07T06:58:47.883000", "content": "Nothing indicates in the instructions of the submission that it should be submitted via notebook (as in the case of the code competition on Kaggle) I only submitted the updated csv file which contains the URL of my public notebook (as required). And even in the \"Pinned Q&A Thread\", speaking of submissions, @paultimothymooney Paul just talks about uploading the csv file, see here (in his response): https://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/400084#2255821
\nThis is:
\n\"…If you follow the eligibility guidelines and then upload a submission.csv file (where you have replaced these empty URLs with valid ones) …\"
I believe the issue will be resolved if the participant (or team) writes at the beginning of each notebook that has been the subject of the submission. Essay Topic : ……, this indicates that the notebook is the one that needs to be reviewed, and it also orients on the topic (I've seen cases where sometimes it's not clear exactly if the topic of the notebook, for eg, is about Kaggle competitions or the 7th category: Other…: \"everything\" is in the notebook…)
", "votes": 1, "replies": [ { "id": 2333230, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2023-07-06T18:33:54.263000", "content": "Indeed, that’d somewhat help. Not a perfect solution though, but rather “what if everyone does X” :) some instruction from the org team will do better.
", "votes": 1, "replies": [] } ] }, { "id": 2332949, "author_name": "Jonathan Bown", "author_url": "", "post_date": "2023-07-06T14:45:44.110000", "content": "Great questions! I agree its been really hard to figure out which notebooks are final essays and which ones are just noise. Earlier on I was seeing all these notebooks being submitted that clearly weren't essays so I decided to just stick to the rules without getting distracted with people just trying to get upvotes.
", "votes": 1, "replies": [ { "id": 2333010, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2023-07-06T15:31:37.203000", "content": "Indeed. With a reservation that sticking to the rules might also mean providing feedback on non-essay notebooks.
", "votes": 1, "replies": [] } ] }, { "id": 2333401, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2023-07-07T00:00:49.700000", "content": "I observe the notebooks are continuously updated by their authors. It would be nice if the notebook content is frozen during peer review feedback process. This will ensure consistency.
", "votes": 0, "replies": [] }, { "id": 2333243, "author_name": "", "author_url": "", "post_date": "2023-07-06T18:52:11.320000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2332807": "Dear orgs, \n\nhaving a bit more instructions on specific Notebooks to be graded would definitely be helpful. \n\nSome problems:\n\n- Notebooks with essays are barely distinguishable from supplementary notebooks (e.g. [essay](https://www.kaggle.com/code/thedrcat/how-to-win-a-kaggle-competition) + [a supplement](https://www.kaggle.com/code/thedrcat/using-llms-to-extract-structured-data/notebook))\n- Apparently, some teams released >1 notebooks while the [submission instructions](https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/submission-instructions) clearly state that \"each team should write a single essay focusing on a single topic.\" (e.g. [essay1](https://www.kaggle.com/code/sophieb/a-journey-through-the-text-data-competitions) and [essay2](https://www.kaggle.com/code/sophieb/ai-report-text-data-kaggle-and-arxiv) from the same authors or as many as [16 notebooks](https://www.kaggle.com/mpwolke/code?userId=3012786&sortBy=dateCreated&tab=profile) released by a single individual)\n\nHence, the questions:\n - how do we know which notebooks to review?\n - how do we mitigate the risk that reviews are issued to notebooks that were not supposed to be reviewed?\n\nIn hindsight, probably, submissions with a single URL to an essay need to have been obligatory before July 5th. Then, it would've been easy to publish a whole list of ~350 Notebooks that need to be reviewed. ", "2332924": "I think these are great questions. I am as well a bit confused about which notebooks are the final submissions.", "2333395": "Hi Kashnitsky,\n\n#Focus on Score 1 (submitted Notebooks). It's written on Code list below the title.\nOpen only those that have that Score 1.\n\nI think the intention is to read all (or as many as we could) that have Score 1 (=submitted).\n\nBy the way, I'm the one that published 16 Notebooks cause I was exploring the dataset. At the beginning I hadn't understood that it was to write an essay : )\n\nHowever, I submitted ONLY one. And I don't bother about votes. Just to learn.\n\nNow, I'm reading the recently created, most part was written by beginners. \nThey have a lot to teach us.", "2333215": "I believe the issue will be resolved if the participant (or team) writes at the beginning of each notebook that has been the subject of the submission. Essay Topic : ......, this indicates that the notebook is the one that needs to be reviewed, and it also orients on the topic (I've seen cases where sometimes it's not clear exactly if the topic of the notebook, for eg, is about Kaggle competitions or the 7th category: Other...: \"everything\" is in the notebook...)", "2332949": "Great questions! I agree its been really hard to figure out which notebooks are final essays and which ones are just noise. Earlier on I was seeing all these notebooks being submitted that clearly weren't essays so I decided to just stick to the rules without getting distracted with people just trying to get upvotes.", "2333401": "I observe the notebooks are continuously updated by their authors. It would be nice if the notebook content is frozen during peer review feedback process. This will ensure consistency. ", "2333243": "" } }