{ "id": 421777, "title": "Guidance and support for the peer review process.", "url": "/competitions/2023-kaggle-ai-report/discussion/421777", "author_name": "", "post_date": "2023-07-06T17:42:35.911648800Z", "votes": null, "comment_count": 2, "views": 0, "content": "

This summarizes and builds on issues raised in other discussion posts about the need for guidance and support for the peer review process.

\n
    \n

  1. Many notebooks do not indicate the essay category. Is there an easy and systematic way to find that?

    2. \n

  2. Some authors created more than one notebook. But only one can be named in the submission file. (For example, I wrote on two topics and stand by both essays. But on one of them, I indicated that the essay was not eligible for the competition.) How is the actual submission essay to be indicated in a consistent way for everyone to see?

    3. \n

  3. I was hoping that, after July 5, there would become available a listing of all competition-submitted notebooks per category, so we could easily filter for notebooks eligible for the competition on a per-category basis, while still having access to other notebooks that might still be worth reading.

    4. \n

  4. Please confirm: After providing peer commentary, we should change our submissions.csv file to include the URLs of the essays we commented on in the 'feedbackX_url' rows.

    5. \n

  5. Please confirm: If we find more than three essays to be interesting enough to provide feedback for, it is fine to comment on more than three. These would be additional rows in the submissions.csv file, named feedback4_url, feedback5_url, etc. Correct?

    6. \n

  6. Comments vary in quality. It is easy for someone to run an article through ChatGPT and get generic pablum, or otherwise provide rote feedback that adds no value. The closest thing to scoring comment quality is upvotes on comments. Is this the mechanism we should use? Number of comments percolates up to the Notebooks page, but not upvotes or any other indicators of comment quality.

    7. \n

  7. The Notebook page has filters for ordering the notebooks. These are tuned for competitions having objective scores on data predictions, but not on essays. Anyone can artificially promote their essay under the \"Recently run\" filter by re-running it. The \"Hotness\" filter is opaque but appears to favor essays that got attention when the competition first opened. It would be helpful to have an ordering that rank-orded essays by some other more neutral criteria, in need of quality comments, or even randomly.

    8. \n
\n
\n

In other words, some means would be appreciated to help figure out which essays we should focus on providing comments for.

\n

I guess one of us could take this on as a data preparation task:
\n -scraping the site
\n -extracting topic from the essay
\n -allowing authors to indicate topic and flag which essay is their official submission
\n -rating comments as contentful and useful or not
\n -??
\nBut Kaggle would be in the best positition to do this properly.

", "messages": [ { "id": "2333175", "postDate": "07/06/2023 17:42:35", "content": "

This summarizes and builds on issues raised in other discussion posts about the need for guidance and support for the peer review process.

\n
    \n

  1. Many notebooks do not indicate the essay category. Is there an easy and systematic way to find that?

    2. \n

  2. Some authors created more than one notebook. But only one can be named in the submission file. (For example, I wrote on two topics and stand by both essays. But on one of them, I indicated that the essay was not eligible for the competition.) How is the actual submission essay to be indicated in a consistent way for everyone to see?

    3. \n

  3. I was hoping that, after July 5, there would become available a listing of all competition-submitted notebooks per category, so we could easily filter for notebooks eligible for the competition on a per-category basis, while still having access to other notebooks that might still be worth reading.

    4. \n

  4. Please confirm: After providing peer commentary, we should change our submissions.csv file to include the URLs of the essays we commented on in the 'feedbackX_url' rows.

    5. \n

  5. Please confirm: If we find more than three essays to be interesting enough to provide feedback for, it is fine to comment on more than three. These would be additional rows in the submissions.csv file, named feedback4_url, feedback5_url, etc. Correct?

    6. \n

  6. Comments vary in quality. It is easy for someone to run an article through ChatGPT and get generic pablum, or otherwise provide rote feedback that adds no value. The closest thing to scoring comment quality is upvotes on comments. Is this the mechanism we should use? Number of comments percolates up to the Notebooks page, but not upvotes or any other indicators of comment quality.

    7. \n

  7. The Notebook page has filters for ordering the notebooks. These are tuned for competitions having objective scores on data predictions, but not on essays. Anyone can artificially promote their essay under the \"Recently run\" filter by re-running it. The \"Hotness\" filter is opaque but appears to favor essays that got attention when the competition first opened. It would be helpful to have an ordering that rank-orded essays by some other more neutral criteria, in need of quality comments, or even randomly.

    8. \n
\n
\n

In other words, some means would be appreciated to help figure out which essays we should focus on providing comments for.

\n

I guess one of us could take this on as a data preparation task:
\n -scraping the site
\n -extracting topic from the essay
\n -allowing authors to indicate topic and flag which essay is their official submission
\n -rating comments as contentful and useful or not
\n -??
\nBut Kaggle would be in the best positition to do this properly.

", "rawMarkdown": "This summarizes and builds on issues raised in other discussion posts about the need for guidance and support for the peer review process.\n\n1. Many notebooks do not indicate the essay category. Is there an easy and systematic way to find that?\n\n2. Some authors created more than one notebook. But only one can be named in the submission file. (For example, I wrote on two topics and stand by both essays. But on one of them, I indicated that the essay was not eligible for the competition.) How is the actual submission essay to be indicated in a consistent way for everyone to see?\n\n3. I was hoping that, after July 5, there would become available a listing of all competition-submitted notebooks *per category*, so we could easily filter for notebooks eligible for the competition on a per-category basis, while still having access to other notebooks that might still be worth reading.\n\n4. Please confirm: After providing peer commentary, we should change our submissions.csv file to include the URLs of the essays we commented on in the 'feedbackX_url' rows.\n\n5. Please confirm: If we find more than three essays to be interesting enough to provide feedback for, it is fine to comment on more than three. These would be additional rows in the submissions.csv file, named feedback4_url, feedback5_url, etc. Correct?\n\n6. Comments vary in quality. It is easy for someone to run an article through ChatGPT and get generic pablum, or otherwise provide rote feedback that adds no value. The closest thing to scoring comment quality is upvotes on comments. Is this the mechanism we should use? Number of comments percolates up to the Notebooks page, but not upvotes or any other indicators of comment quality. \n\n7. The Notebook page has filters for ordering the notebooks. These are tuned for competitions having objective scores on data predictions, but not on essays. Anyone can artificially promote their essay under the \"Recently run\" filter by re-running it. The \"Hotness\" filter is opaque but appears to favor essays that got attention when the competition first opened. It would be helpful to have an ordering that rank-orded essays by some other more neutral criteria, in need of quality comments, or even randomly.\n\n---\n\nIn other words, some means would be appreciated to help figure out which essays we should focus on providing comments for.\n\nI guess one of us could take this on as a data preparation task:\n -scraping the site\n -extracting topic from the essay\n -allowing authors to indicate topic and flag which essay is their official submission\n -rating comments as contentful and useful or not\n -??\nBut Kaggle would be in the best positition to do this properly.", "votes": null }, { "id": "2334568", "postDate": "07/07/2023 19:13:58", "content": "

See pinned Q&A thread for answers to common questions (and for better visibility by the Kaggle Team).

\n

We won't have a final list of the submissions and the categories until the July 16 deadline. For example, a large fraction of the leaderboard submitted to us only placeholder values and they have until July 16th to update those submission.csv files with the appropriate URLs. As such, it is too early to exclude any public notebooks from our search.

\n

My recommended method for finding essays to provide peer-feedback to would be as follows:

\n\n

Alternatively, here is a preliminary list of all of the public notebooks, sorted by category, such that you can quickly filter to see only your chosen category.

\n

To answer your specific questions:

\n

Q1: How to find the essay category?
\nA1: Notebook title, notebook body, or supplementary dataset

\n

Q2: How to determine if an notebook is associated with a submission?
\nA2: This will happen after the July 16 submission deadline

\n

Q3: Can I have a list of essays sorted by topic?
\nA3: Sure

\n

Q4: What are the submission instructions?
\nA4: See https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/submission-instructions. After you complete your essay you'll need to update the essay URL. Likewise, you'll need to update the peer feedback URL after you complete the peer feedback task.

\n

Q5: Does my submission.csv file need to resemble the template?
\nA5: Yes. For more detail see submission instructions and ./data/sample_submission.csv. If your submission.csv file does not resemble the template then your submission will fail and you will receive an error message.

\n

Q6: What is the guidance for providing high quality peer feedback?
\nA6: See https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/peer-feedback-instructions.

\n\n

Q7: How should I sort the notebooks in the code listing?
\nA7: I recommend sorting by \"recently created\" as most of the serious submissions were not published until very near to the deadline.

\n

Hopefully that helps!

", "rawMarkdown": "See [pinned Q&A thread for answers to common questions](https://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/400084) (and for better visibility by the Kaggle Team).\n\nWe won't have a final list of the submissions and the categories until the July 16 deadline. For example, a large fraction of the leaderboard submitted to us only placeholder values and they have until July 16th to update those submission.csv files with the appropriate URLs. As such, it is too early to exclude any public notebooks from our search.\n\nMy recommended method for finding essays to provide peer-feedback to would be as follows:\n- Navigate to the [code listing](https://www.kaggle.com/competitions/2023-kaggle-ai-report/code)\n- Sort by \"recently created\"\n- Find a notebook that has a title that seems aligned with the topic that you chose for your own essay\n- Verify that it does not yet already have 3x thoughtful comments providing peer feedback (this information can only be found on the ./code listing page)\n- If it already has >3 comments providing peer-feedback, keep scrolling until you find one that does not.\n- If all of the notebooks in your category already have >3 comments providing peer-feedback, expand your search to also include additional categories\n\n**Alternatively, [here is a preliminary list of all of the public notebooks, sorted by category](https://www.kaggle.com/datasets/paultimothymooney/list-of-public-notebooks-ai-report-competition), such that you can quickly filter to see only your chosen category.**\n\n\nTo answer your specific questions:\n\nQ1: How to find the essay category?\nA1: Notebook title, notebook body, or supplementary dataset\n\nQ2: How to determine if an notebook is associated with a submission?\nA2: This will happen after the July 16 submission deadline\n\nQ3: Can I have a list of essays sorted by topic?\nA3: [Sure](https://www.kaggle.com/datasets/paultimothymooney/list-of-public-notebooks-ai-report-competition)\n\nQ4: What are the submission instructions?\nA4: See https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/submission-instructions. After you complete your essay you'll need to update the essay URL. Likewise, you'll need to update the peer feedback URL after you complete the peer feedback task.\n\nQ5: Does my submission.csv file need to resemble the template?\nA5: Yes. For more detail see [submission instructions](https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/submission-instructions) and [./data/sample_submission.csv](https://www.kaggle.com/competitions/2023-kaggle-ai-report/data?select=sample_submission.csv). If your submission.csv file does not resemble the template then your submission will fail and you will receive an error message.\n\nQ6: What is the guidance for providing high quality peer feedback?\nA6: See https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/peer-feedback-instructions.\n - Provide an accurate summary of the notebook being reviewed\n - Assess whether appropriate context was set, and that all necessary\nreferences were cited\n - Evaluate the overall quality of the writing and the composition of the\nwork\n - Evaluate the significance of the work and interest to the ML community\n - Verify the correctness of the claims made within the essay\n - Provide feedback concerning potential flaws for any unsupported claims \n - Assess whether there exists content that might be considered inappropriate (e.g.\nplagiarism, academic misconduct, or incorrect statements) \n \nQ7: How should I sort the notebooks in the code listing?\nA7: I recommend sorting by \"recently created\" as most of the serious submissions were not published until very near to the deadline.\n\n\n\n\nHopefully that helps!", "votes": null }, { "id": "2365180", "postDate": "07/30/2023 03:58:16", "content": "

Thanks for the suggestion.

", "rawMarkdown": "Thanks for the suggestion.", "votes": null } ], "comments": [ { "id": 2334568, "author_name": "paultimothymooney", "author_url": "", "post_date": "07/07/2023 19:13:58", "content": "

See pinned Q&A thread for answers to common questions (and for better visibility by the Kaggle Team).

\n

We won't have a final list of the submissions and the categories until the July 16 deadline. For example, a large fraction of the leaderboard submitted to us only placeholder values and they have until July 16th to update those submission.csv files with the appropriate URLs. As such, it is too early to exclude any public notebooks from our search.

\n

My recommended method for finding essays to provide peer-feedback to would be as follows:

\n\n

Alternatively, here is a preliminary list of all of the public notebooks, sorted by category, such that you can quickly filter to see only your chosen category.

\n

To answer your specific questions:

\n

Q1: How to find the essay category?
\nA1: Notebook title, notebook body, or supplementary dataset

\n

Q2: How to determine if an notebook is associated with a submission?
\nA2: This will happen after the July 16 submission deadline

\n

Q3: Can I have a list of essays sorted by topic?
\nA3: Sure

\n

Q4: What are the submission instructions?
\nA4: See https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/submission-instructions. After you complete your essay you'll need to update the essay URL. Likewise, you'll need to update the peer feedback URL after you complete the peer feedback task.

\n

Q5: Does my submission.csv file need to resemble the template?
\nA5: Yes. For more detail see submission instructions and ./data/sample_submission.csv. If your submission.csv file does not resemble the template then your submission will fail and you will receive an error message.

\n

Q6: What is the guidance for providing high quality peer feedback?
\nA6: See https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/peer-feedback-instructions.

\n\n

Q7: How should I sort the notebooks in the code listing?
\nA7: I recommend sorting by \"recently created\" as most of the serious submissions were not published until very near to the deadline.

\n

Hopefully that helps!

", "votes": null, "replies": [] }, { "id": 2365180, "author_name": "yangxu21", "author_url": "", "post_date": "07/30/2023 03:58:16", "content": "

Thanks for the suggestion.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2333175": "This summarizes and builds on issues raised in other discussion posts about the need for guidance and support for the peer review process.\n\n1. Many notebooks do not indicate the essay category. Is there an easy and systematic way to find that?\n\n2. Some authors created more than one notebook. But only one can be named in the submission file. (For example, I wrote on two topics and stand by both essays. But on one of them, I indicated that the essay was not eligible for the competition.) How is the actual submission essay to be indicated in a consistent way for everyone to see?\n\n3. I was hoping that, after July 5, there would become available a listing of all competition-submitted notebooks *per category*, so we could easily filter for notebooks eligible for the competition on a per-category basis, while still having access to other notebooks that might still be worth reading.\n\n4. Please confirm: After providing peer commentary, we should change our submissions.csv file to include the URLs of the essays we commented on in the 'feedbackX_url' rows.\n\n5. Please confirm: If we find more than three essays to be interesting enough to provide feedback for, it is fine to comment on more than three. These would be additional rows in the submissions.csv file, named feedback4_url, feedback5_url, etc. Correct?\n\n6. Comments vary in quality. It is easy for someone to run an article through ChatGPT and get generic pablum, or otherwise provide rote feedback that adds no value. The closest thing to scoring comment quality is upvotes on comments. Is this the mechanism we should use? Number of comments percolates up to the Notebooks page, but not upvotes or any other indicators of comment quality. \n\n7. The Notebook page has filters for ordering the notebooks. These are tuned for competitions having objective scores on data predictions, but not on essays. Anyone can artificially promote their essay under the \"Recently run\" filter by re-running it. The \"Hotness\" filter is opaque but appears to favor essays that got attention when the competition first opened. It would be helpful to have an ordering that rank-orded essays by some other more neutral criteria, in need of quality comments, or even randomly.\n\n---\n\nIn other words, some means would be appreciated to help figure out which essays we should focus on providing comments for.\n\nI guess one of us could take this on as a data preparation task:\n -scraping the site\n -extracting topic from the essay\n -allowing authors to indicate topic and flag which essay is their official submission\n -rating comments as contentful and useful or not\n -??\nBut Kaggle would be in the best positition to do this properly.", "2334568": "See [pinned Q&A thread for answers to common questions](https://www.kaggle.com/competitions/2023-kaggle-ai-report/discussion/400084) (and for better visibility by the Kaggle Team).\n\nWe won't have a final list of the submissions and the categories until the July 16 deadline. For example, a large fraction of the leaderboard submitted to us only placeholder values and they have until July 16th to update those submission.csv files with the appropriate URLs. As such, it is too early to exclude any public notebooks from our search.\n\nMy recommended method for finding essays to provide peer-feedback to would be as follows:\n- Navigate to the [code listing](https://www.kaggle.com/competitions/2023-kaggle-ai-report/code)\n- Sort by \"recently created\"\n- Find a notebook that has a title that seems aligned with the topic that you chose for your own essay\n- Verify that it does not yet already have 3x thoughtful comments providing peer feedback (this information can only be found on the ./code listing page)\n- If it already has >3 comments providing peer-feedback, keep scrolling until you find one that does not.\n- If all of the notebooks in your category already have >3 comments providing peer-feedback, expand your search to also include additional categories\n\n**Alternatively, [here is a preliminary list of all of the public notebooks, sorted by category](https://www.kaggle.com/datasets/paultimothymooney/list-of-public-notebooks-ai-report-competition), such that you can quickly filter to see only your chosen category.**\n\n\nTo answer your specific questions:\n\nQ1: How to find the essay category?\nA1: Notebook title, notebook body, or supplementary dataset\n\nQ2: How to determine if an notebook is associated with a submission?\nA2: This will happen after the July 16 submission deadline\n\nQ3: Can I have a list of essays sorted by topic?\nA3: [Sure](https://www.kaggle.com/datasets/paultimothymooney/list-of-public-notebooks-ai-report-competition)\n\nQ4: What are the submission instructions?\nA4: See https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/submission-instructions. After you complete your essay you'll need to update the essay URL. Likewise, you'll need to update the peer feedback URL after you complete the peer feedback task.\n\nQ5: Does my submission.csv file need to resemble the template?\nA5: Yes. For more detail see [submission instructions](https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/submission-instructions) and [./data/sample_submission.csv](https://www.kaggle.com/competitions/2023-kaggle-ai-report/data?select=sample_submission.csv). If your submission.csv file does not resemble the template then your submission will fail and you will receive an error message.\n\nQ6: What is the guidance for providing high quality peer feedback?\nA6: See https://www.kaggle.com/competitions/2023-kaggle-ai-report/overview/peer-feedback-instructions.\n - Provide an accurate summary of the notebook being reviewed\n - Assess whether appropriate context was set, and that all necessary\nreferences were cited\n - Evaluate the overall quality of the writing and the composition of the\nwork\n - Evaluate the significance of the work and interest to the ML community\n - Verify the correctness of the claims made within the essay\n - Provide feedback concerning potential flaws for any unsupported claims \n - Assess whether there exists content that might be considered inappropriate (e.g.\nplagiarism, academic misconduct, or incorrect statements) \n \nQ7: How should I sort the notebooks in the code listing?\nA7: I recommend sorting by \"recently created\" as most of the serious submissions were not published until very near to the deadline.\n\n\n\n\nHopefully that helps!", "2365180": "Thanks for the suggestion." }, "source": "meta" }