{
  "id": 315609,
  "title": "ID of Test Files",
  "url": "/competitions/denoising-shabby/discussion/315609",
  "author_name": "",
  "post_date": "2022-03-29T04:20:56.993277700Z",
  "votes": 1,
  "comment_count": 10,
  "views": 0,
  "content": "<p>How to find out the ID of the test images that are required in the submission file?</p>",
  "messages": [
    {
      "id": "1738145",
      "postDate": "03/29/2022 04:20:56",
      "content": "<p>How to find out the ID of the test images that are required in the submission file?</p>",
      "rawMarkdown": "How to find out the ID of the test images that are required in the submission file?",
      "votes": null
    },
    {
      "id": "1739113",
      "postDate": "03/29/2022 19:00:39",
      "content": "<p>Images are assigned an ID in alphanumeric order. You can use the <code>sorted()</code> built-in method on the list of file names to get the correct ordering, and then each image is assigned its index + 1 as an ID (the image at index 0 has ID 1, and so on).</p>\n<p>The Data page has been updated with \"mapping.txt\" (also linked here), which contains the same information.</p>",
      "rawMarkdown": "Images are assigned an ID in alphanumeric order. You can use the `sorted()` built-in method on the list of file names to get the correct ordering, and then each image is assigned its index + 1 as an ID (the image at index 0 has ID 1, and so on).\n\nThe Data page has been updated with \"mapping.txt\" (also linked here), which contains the same information.",
      "votes": null
    },
    {
      "id": "1739459",
      "postDate": "03/30/2022 04:07:48",
      "content": "<p>Did Exactly that but got the following error with my submission:</p>\n<p>ERROR: Unable to find 2200850 required key values in the 'id' column<br>\nERROR: Unable to find the required key value '1_0_408' in the 'id' column<br>\nERROR: Unable to find the required key value '1_0_409' in the 'id' column<br>\nERROR: Unable to find the required key value '1_0_410' in the 'id' column<br>\nERROR: Unable to find the required key value '1_0_411' in the 'id' column<br>\nERROR: Unable to find the required key value '1_1_408' in the 'id' column<br>\nERROR: Unable to find the required key value '1_1_409' in the 'id' column<br>\nERROR: Unable to find the required key value '1_1_410' in the 'id' column<br>\nERROR: Unable to find the required key value '1_1_411' in the 'id' column<br>\nERROR: Unable to find the required key value '1_2_408' in the 'id' column</p>\n<p>even though the dimensions of the first image in test set are (314, 408)</p>",
      "rawMarkdown": "Did Exactly that but got the following error with my submission:\n\nERROR: Unable to find 2200850 required key values in the 'id' column\nERROR: Unable to find the required key value '1_0_408' in the 'id' column\nERROR: Unable to find the required key value '1_0_409' in the 'id' column\nERROR: Unable to find the required key value '1_0_410' in the 'id' column\nERROR: Unable to find the required key value '1_0_411' in the 'id' column\nERROR: Unable to find the required key value '1_1_408' in the 'id' column\nERROR: Unable to find the required key value '1_1_409' in the 'id' column\nERROR: Unable to find the required key value '1_1_410' in the 'id' column\nERROR: Unable to find the required key value '1_1_411' in the 'id' column\nERROR: Unable to find the required key value '1_2_408' in the 'id' column\n\neven though the dimensions of the first image in test set are (314, 408)",
      "votes": null
    },
    {
      "id": "1739467",
      "postDate": "03/30/2022 04:33:43",
      "content": "<p>Sample Code: </p>\n<pre><code>submission = []\nfor idx, file in tqdm(enumerate(sorted(os.listdir(t_dir)))):\n    img = np.array(Image.open(f\"{t_dir}/{file}\").convert('L')) / 255.0\n    for i in range(len(img)):\n        for j in range(len(img[0])):\n            submission.append((f\"{idx+1}_{i}_{j}\", img[i][j]))   \npd.DataFrame(submission, columns=['id', 'value']).to_csv(\"submission.csv\", index=False)\n</code></pre>",
      "rawMarkdown": "Sample Code: \n\n```\nsubmission = []\nfor idx, file in tqdm(enumerate(sorted(os.listdir(t_dir)))):\n    img = np.array(Image.open(f\"{t_dir}/{file}\").convert('L')) / 255.0\n    for i in range(len(img)):\n        for j in range(len(img[0])):\n            submission.append((f\"{idx+1}_{i}_{j}\", img[i][j]))   \npd.DataFrame(submission, columns=['id', 'value']).to_csv(\"submission.csv\", index=False)\n```",
      "votes": null
    },
    {
      "id": "1740397",
      "postDate": "03/30/2022 21:08:19",
      "content": "<p>The code used to generate the evaluation file is here <a href=\"https://github.com/sparkfish/shabby-pages/blob/dev/build/gen_evaluation.py\" target=\"_blank\">https://github.com/sparkfish/shabby-pages/blob/dev/build/gen_evaluation.py</a></p>",
      "rawMarkdown": "The code used to generate the evaluation file is here https://github.com/sparkfish/shabby-pages/blob/dev/build/gen_evaluation.py",
      "votes": null
    },
    {
      "id": "1740399",
      "postDate": "03/30/2022 21:09:55",
      "content": "<p>I just uploaded a new version of the dataset, please try with the new files.</p>\n<p>It's possible that PIL images and cv2 images are indexed in the opposite order, so please try re-train with the new data and adapt the code above for your own submission and try again.</p>",
      "rawMarkdown": "I just uploaded a new version of the dataset, please try with the new files.\n\nIt's possible that PIL images and cv2 images are indexed in the opposite order, so please try re-train with the new data and adapt the code above for your own submission and try again.",
      "votes": null
    },
    {
      "id": "1740583",
      "postDate": "03/31/2022 04:34:56",
      "content": "<p>Used the script you mentioned<br>\nNow getting this error while submitting</p>\n<p><code>Evaluation Exception: Missing submission schema mapping</code></p>",
      "rawMarkdown": "Used the script you mentioned\nNow getting this error while submitting\n\n\n```Evaluation Exception: Missing submission schema mapping```",
      "votes": null
    },
    {
      "id": "1740801",
      "postDate": "03/31/2022 09:38:40",
      "content": "<p>I am also getting the exception while submitting <br>\n<code>Evaluation Exception: Missing submission schema mapping</code></p>\n<p>Can you please look into it <a href=\"https://www.kaggle.com/augraphy\" target=\"_blank\">@augraphy</a> </p>",
      "rawMarkdown": "I am also getting the exception while submitting \n`Evaluation Exception: Missing submission schema mapping`\n\nCan you please look into it @augraphy",
      "votes": null
    },
    {
      "id": "1741306",
      "postDate": "03/31/2022 17:49:34",
      "content": "<p>This should be fixed now, can you try again?</p>",
      "rawMarkdown": "This should be fixed now, can you try again?",
      "votes": null
    },
    {
      "id": "1741315",
      "postDate": "03/31/2022 18:06:43",
      "content": "<p>Still same error</p>\n<p>In case you want to  see my code:</p>\n<pre><code>with open(\"evaluation.csv\", \"w\") as outfile:\n    outfile.write(\"id,value\\n\")\n    for idx, file in tqdm(enumerate(sorted(os.listdir(t_dir)))):\n        img = cv2.cvtColor(cv2.imread(f\"{t_dir}/{file}\"), cv2.COLOR_BGR2GRAY) / 255.0\n        y,x = img.shape\n        for j in range(y):\n            for i in range(x):\n                outfile.write(f\"{idx+1}_{j}_{i},{img[j][i]}\\n\")\n</code></pre>",
      "rawMarkdown": "Still same error\n\nIn case you want to  see my code:\n\n```\nwith open(\"evaluation.csv\", \"w\") as outfile:\n    outfile.write(\"id,value\\n\")\n    for idx, file in tqdm(enumerate(sorted(os.listdir(t_dir)))):\n        img = cv2.cvtColor(cv2.imread(f\"{t_dir}/{file}\"), cv2.COLOR_BGR2GRAY) / 255.0\n        y,x = img.shape\n        for j in range(y):\n            for i in range(x):\n                outfile.write(f\"{idx+1}_{j}_{i},{img[j][i]}\\n\")\n```",
      "votes": null
    },
    {
      "id": "1742412",
      "postDate": "04/01/2022 20:54:44",
      "content": "<p>Can you upload the submission file to Google Drive etc. and <a target=\"_blank\">email us</a> a link? </p>",
      "rawMarkdown": "Can you upload the submission file to Google Drive etc. and [email us](mailto:opensource@sparkfish.com) a link?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1739113,
      "author_name": "augraphy",
      "author_url": "",
      "post_date": "03/29/2022 19:00:39",
      "content": "<p>Images are assigned an ID in alphanumeric order. You can use the <code>sorted()</code> built-in method on the list of file names to get the correct ordering, and then each image is assigned its index + 1 as an ID (the image at index 0 has ID 1, and so on).</p>\n<p>The Data page has been updated with \"mapping.txt\" (also linked here), which contains the same information.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1739459,
          "author_name": "ritvik1909",
          "author_url": "",
          "post_date": "03/30/2022 04:07:48",
          "content": "<p>Did Exactly that but got the following error with my submission:</p>\n<p>ERROR: Unable to find 2200850 required key values in the 'id' column<br>\nERROR: Unable to find the required key value '1_0_408' in the 'id' column<br>\nERROR: Unable to find the required key value '1_0_409' in the 'id' column<br>\nERROR: Unable to find the required key value '1_0_410' in the 'id' column<br>\nERROR: Unable to find the required key value '1_0_411' in the 'id' column<br>\nERROR: Unable to find the required key value '1_1_408' in the 'id' column<br>\nERROR: Unable to find the required key value '1_1_409' in the 'id' column<br>\nERROR: Unable to find the required key value '1_1_410' in the 'id' column<br>\nERROR: Unable to find the required key value '1_1_411' in the 'id' column<br>\nERROR: Unable to find the required key value '1_2_408' in the 'id' column</p>\n<p>even though the dimensions of the first image in test set are (314, 408)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1739467,
          "author_name": "ritvik1909",
          "author_url": "",
          "post_date": "03/30/2022 04:33:43",
          "content": "<p>Sample Code: </p>\n<pre><code>submission = []\nfor idx, file in tqdm(enumerate(sorted(os.listdir(t_dir)))):\n    img = np.array(Image.open(f\"{t_dir}/{file}\").convert('L')) / 255.0\n    for i in range(len(img)):\n        for j in range(len(img[0])):\n            submission.append((f\"{idx+1}_{i}_{j}\", img[i][j]))   \npd.DataFrame(submission, columns=['id', 'value']).to_csv(\"submission.csv\", index=False)\n</code></pre>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1740397,
          "author_name": "augraphy",
          "author_url": "",
          "post_date": "03/30/2022 21:08:19",
          "content": "<p>The code used to generate the evaluation file is here <a href=\"https://github.com/sparkfish/shabby-pages/blob/dev/build/gen_evaluation.py\" target=\"_blank\">https://github.com/sparkfish/shabby-pages/blob/dev/build/gen_evaluation.py</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1740399,
          "author_name": "augraphy",
          "author_url": "",
          "post_date": "03/30/2022 21:09:55",
          "content": "<p>I just uploaded a new version of the dataset, please try with the new files.</p>\n<p>It's possible that PIL images and cv2 images are indexed in the opposite order, so please try re-train with the new data and adapt the code above for your own submission and try again.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1740583,
          "author_name": "ritvik1909",
          "author_url": "",
          "post_date": "03/31/2022 04:34:56",
          "content": "<p>Used the script you mentioned<br>\nNow getting this error while submitting</p>\n<p><code>Evaluation Exception: Missing submission schema mapping</code></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1741306,
          "author_name": "augraphy",
          "author_url": "",
          "post_date": "03/31/2022 17:49:34",
          "content": "<p>This should be fixed now, can you try again?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1741315,
          "author_name": "ritvik1909",
          "author_url": "",
          "post_date": "03/31/2022 18:06:43",
          "content": "<p>Still same error</p>\n<p>In case you want to  see my code:</p>\n<pre><code>with open(\"evaluation.csv\", \"w\") as outfile:\n    outfile.write(\"id,value\\n\")\n    for idx, file in tqdm(enumerate(sorted(os.listdir(t_dir)))):\n        img = cv2.cvtColor(cv2.imread(f\"{t_dir}/{file}\"), cv2.COLOR_BGR2GRAY) / 255.0\n        y,x = img.shape\n        for j in range(y):\n            for i in range(x):\n                outfile.write(f\"{idx+1}_{j}_{i},{img[j][i]}\\n\")\n</code></pre>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1742412,
          "author_name": "augraphy",
          "author_url": "",
          "post_date": "04/01/2022 20:54:44",
          "content": "<p>Can you upload the submission file to Google Drive etc. and <a target=\"_blank\">email us</a> a link? </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1740801,
      "author_name": "haresh12",
      "author_url": "",
      "post_date": "03/31/2022 09:38:40",
      "content": "<p>I am also getting the exception while submitting <br>\n<code>Evaluation Exception: Missing submission schema mapping</code></p>\n<p>Can you please look into it <a href=\"https://www.kaggle.com/augraphy\" target=\"_blank\">@augraphy</a> </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1738145": "How to find out the ID of the test images that are required in the submission file?",
    "1739113": "Images are assigned an ID in alphanumeric order. You can use the `sorted()` built-in method on the list of file names to get the correct ordering, and then each image is assigned its index + 1 as an ID (the image at index 0 has ID 1, and so on).\n\nThe Data page has been updated with \"mapping.txt\" (also linked here), which contains the same information.",
    "1739459": "Did Exactly that but got the following error with my submission:\n\nERROR: Unable to find 2200850 required key values in the 'id' column\nERROR: Unable to find the required key value '1_0_408' in the 'id' column\nERROR: Unable to find the required key value '1_0_409' in the 'id' column\nERROR: Unable to find the required key value '1_0_410' in the 'id' column\nERROR: Unable to find the required key value '1_0_411' in the 'id' column\nERROR: Unable to find the required key value '1_1_408' in the 'id' column\nERROR: Unable to find the required key value '1_1_409' in the 'id' column\nERROR: Unable to find the required key value '1_1_410' in the 'id' column\nERROR: Unable to find the required key value '1_1_411' in the 'id' column\nERROR: Unable to find the required key value '1_2_408' in the 'id' column\n\neven though the dimensions of the first image in test set are (314, 408)",
    "1739467": "Sample Code: \n\n```\nsubmission = []\nfor idx, file in tqdm(enumerate(sorted(os.listdir(t_dir)))):\n    img = np.array(Image.open(f\"{t_dir}/{file}\").convert('L')) / 255.0\n    for i in range(len(img)):\n        for j in range(len(img[0])):\n            submission.append((f\"{idx+1}_{i}_{j}\", img[i][j]))   \npd.DataFrame(submission, columns=['id', 'value']).to_csv(\"submission.csv\", index=False)\n```",
    "1740397": "The code used to generate the evaluation file is here https://github.com/sparkfish/shabby-pages/blob/dev/build/gen_evaluation.py",
    "1740399": "I just uploaded a new version of the dataset, please try with the new files.\n\nIt's possible that PIL images and cv2 images are indexed in the opposite order, so please try re-train with the new data and adapt the code above for your own submission and try again.",
    "1740583": "Used the script you mentioned\nNow getting this error while submitting\n\n\n```Evaluation Exception: Missing submission schema mapping```",
    "1740801": "I am also getting the exception while submitting \n`Evaluation Exception: Missing submission schema mapping`\n\nCan you please look into it @augraphy",
    "1741306": "This should be fixed now, can you try again?",
    "1741315": "Still same error\n\nIn case you want to  see my code:\n\n```\nwith open(\"evaluation.csv\", \"w\") as outfile:\n    outfile.write(\"id,value\\n\")\n    for idx, file in tqdm(enumerate(sorted(os.listdir(t_dir)))):\n        img = cv2.cvtColor(cv2.imread(f\"{t_dir}/{file}\"), cv2.COLOR_BGR2GRAY) / 255.0\n        y,x = img.shape\n        for j in range(y):\n            for i in range(x):\n                outfile.write(f\"{idx+1}_{j}_{i},{img[j][i]}\\n\")\n```",
    "1742412": "Can you upload the submission file to Google Drive etc. and [email us](mailto:opensource@sparkfish.com) a link?"
  },
  "source": "meta"
}