{
  "id": 664195,
  "title": "Data Quality Issues (publications.csv)",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/664195",
  "author_name": "",
  "post_date": "2025-12-23T02:45:35.432570600Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<ul>\n<li>It appears that a large proportion of the samples are duplicated</li>\n</ul>\n<pre><code>publications = pd.read_csv('data/publications.csv')\npublications.shape\n# (216602, 4)\ncount = publications[['pdf_name', 'page', 'page_text']].drop_duplicates().shape[0]\ncount\n# 107865\n</code></pre>\n<ul>\n<li>Page numbers do not appear to be continuous within many PDFs</li>\n</ul>\n<pre><code>publications = publications.drop_duplicates(\n    subset=['pdf_name', 'page', 'page_text'],\n    keep='first'\n)\n\npublications['page'] = pd.to_numeric(publications['page'], errors='coerce')\npublications = publications.sort_values(['pdf_name', 'page'])\n\npublications['page_diff'] = (\n    publications\n    .groupby('pdf_name')['page']\n    .diff()\n)\n\nabnormal = publications[\n    (publications['page_diff'].notna()) &amp;\n    (publications['page_diff'] != 1)\n]\n\nbad_pdfs = abnormal['pdf_name'].unique()\n\nlen(bad_pdfs), bad_pdfs[:10]\n# (235,\n#  array(['AAA 1-2, 1908-9.pdf', 'AAS 11 45-111 Omura-KamanK2001 (2002).pdf',\n#         'AC, 1963.pdf', 'ACC, 1970.pdf', 'AKT 1, 1990.pdf', 'AKT 11a.pdf',\n#         'AKT 2, 1995.pdf', 'AKT 3, 1995.pdf', 'AKT 5, 2008.pdf',\n#         'AKT 5.pdf'], dtype=object))\n</code></pre>\n<p>It seems that at least 235 PDFs contain non-continuous page numbers.</p>\n<p>I would like to ask the organizers whether the duplicated entries are unintentional, and whether it is necessary to provide complete page coverage for each publication.</p>",
  "messages": [
    {
      "id": "3380775",
      "postDate": "12/23/2025 02:45:35",
      "content": "<ul>\n<li>It appears that a large proportion of the samples are duplicated</li>\n</ul>\n<pre><code>publications = pd.read_csv('data/publications.csv')\npublications.shape\n# (216602, 4)\ncount = publications[['pdf_name', 'page', 'page_text']].drop_duplicates().shape[0]\ncount\n# 107865\n</code></pre>\n<ul>\n<li>Page numbers do not appear to be continuous within many PDFs</li>\n</ul>\n<pre><code>publications = publications.drop_duplicates(\n    subset=['pdf_name', 'page', 'page_text'],\n    keep='first'\n)\n\npublications['page'] = pd.to_numeric(publications['page'], errors='coerce')\npublications = publications.sort_values(['pdf_name', 'page'])\n\npublications['page_diff'] = (\n    publications\n    .groupby('pdf_name')['page']\n    .diff()\n)\n\nabnormal = publications[\n    (publications['page_diff'].notna()) &amp;\n    (publications['page_diff'] != 1)\n]\n\nbad_pdfs = abnormal['pdf_name'].unique()\n\nlen(bad_pdfs), bad_pdfs[:10]\n# (235,\n#  array(['AAA 1-2, 1908-9.pdf', 'AAS 11 45-111 Omura-KamanK2001 (2002).pdf',\n#         'AC, 1963.pdf', 'ACC, 1970.pdf', 'AKT 1, 1990.pdf', 'AKT 11a.pdf',\n#         'AKT 2, 1995.pdf', 'AKT 3, 1995.pdf', 'AKT 5, 2008.pdf',\n#         'AKT 5.pdf'], dtype=object))\n</code></pre>\n<p>It seems that at least 235 PDFs contain non-continuous page numbers.</p>\n<p>I would like to ask the organizers whether the duplicated entries are unintentional, and whether it is necessary to provide complete page coverage for each publication.</p>",
      "rawMarkdown": "It appears that a large proportion of the samples are duplicated\n\n```python\npublications = pd.read_csv('data/publications.csv')\npublications.shape\n# (216602, 4)\ncount = publications[['pdf_name', 'page', 'page_text']].drop_duplicates().shape[0]\ncount\n# 107865\n```\n\n- Page numbers do not appear to be continuous within many PDFs\n\n```python\npublications = publications.drop_duplicates(\n    subset=['pdf_name', 'page', 'page_text'],\n    keep='first'\n)\n\npublications['page'] = pd.to_numeric(publications['page'], errors='coerce')\npublications = publications.sort_values(['pdf_name', 'page'])\n\npublications['page_diff'] = (\n    publications\n    .groupby('pdf_name')['page']\n    .diff()\n)\n\nabnormal = publications[\n    (publications['page_diff'].notna()) &\n    (publications['page_diff'] != 1)\n]\n\nbad_pdfs = abnormal['pdf_name'].unique()\n\nlen(bad_pdfs), bad_pdfs[:10]\n# (235,\n#  array(['AAA 1-2, 1908-9.pdf', 'AAS 11 45-111 Omura-KamanK2001 (2002).pdf',\n#         'AC, 1963.pdf', 'ACC, 1970.pdf', 'AKT 1, 1990.pdf', 'AKT 11a.pdf',\n#         'AKT 2, 1995.pdf', 'AKT 3, 1995.pdf', 'AKT 5, 2008.pdf',\n#         'AKT 5.pdf'], dtype=object))\n```\n\nIt seems that at least 235 PDFs contain non-continuous page numbers.\n\n\nI would like to ask the organizers whether the duplicated entries are unintentional, and whether it is necessary to provide complete page coverage for each publication.",
      "votes": null
    },
    {
      "id": "3381006",
      "postDate": "12/23/2025 15:08:15",
      "content": "<p>Yes, duplicated entries are unintentional, good catch!</p>",
      "rawMarkdown": "Yes, duplicated entries are unintentional, good catch!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3381006,
      "author_name": "deeppast",
      "author_url": "",
      "post_date": "12/23/2025 15:08:15",
      "content": "<p>Yes, duplicated entries are unintentional, good catch!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3380775": "It appears that a large proportion of the samples are duplicated\n\n```python\npublications = pd.read_csv('data/publications.csv')\npublications.shape\n# (216602, 4)\ncount = publications[['pdf_name', 'page', 'page_text']].drop_duplicates().shape[0]\ncount\n# 107865\n```\n\n- Page numbers do not appear to be continuous within many PDFs\n\n```python\npublications = publications.drop_duplicates(\n    subset=['pdf_name', 'page', 'page_text'],\n    keep='first'\n)\n\npublications['page'] = pd.to_numeric(publications['page'], errors='coerce')\npublications = publications.sort_values(['pdf_name', 'page'])\n\npublications['page_diff'] = (\n    publications\n    .groupby('pdf_name')['page']\n    .diff()\n)\n\nabnormal = publications[\n    (publications['page_diff'].notna()) &\n    (publications['page_diff'] != 1)\n]\n\nbad_pdfs = abnormal['pdf_name'].unique()\n\nlen(bad_pdfs), bad_pdfs[:10]\n# (235,\n#  array(['AAA 1-2, 1908-9.pdf', 'AAS 11 45-111 Omura-KamanK2001 (2002).pdf',\n#         'AC, 1963.pdf', 'ACC, 1970.pdf', 'AKT 1, 1990.pdf', 'AKT 11a.pdf',\n#         'AKT 2, 1995.pdf', 'AKT 3, 1995.pdf', 'AKT 5, 2008.pdf',\n#         'AKT 5.pdf'], dtype=object))\n```\n\nIt seems that at least 235 PDFs contain non-continuous page numbers.\n\n\nI would like to ask the organizers whether the duplicated entries are unintentional, and whether it is necessary to provide complete page coverage for each publication.",
    "3381006": "Yes, duplicated entries are unintentional, good catch!"
  },
  "source": "meta"
}