I wanted to try out LLM models to retrieve transliterations and translation from the massive 952 PDF OCRs when it got to me that some of the documents may not be publicly available.
\nSo, to be double sure, are the OCRs (publications.csv) from the datasets in the Data section available under CC BY-SA 4.0?
", "messages": [ { "id": "3382540", "postDate": "12/27/2025 19:25:24", "content": "\nI wanted to try out LLM models to retrieve transliterations and translation from the massive 952 PDF OCRs when it got to me that some of the documents may not be publicly available.
\nSo, to be double sure, are the OCRs (publications.csv) from the datasets in the Data section available under CC BY-SA 4.0?
", "rawMarkdown": "deeppast @ryanholbrook \n\nI wanted to try out LLM models to retrieve transliterations and translation from the massive 952 PDF OCRs when it got to me that some of the documents may not be publicly available.\n\nSo, to be double sure, are the OCRs (publications.csv) from the datasets in the Data section available under CC BY-SA 4.0?", "votes": null }, { "id": "3382894", "postDate": "12/28/2025 19:05:43", "content": "Thank you for your thoughtful question. We do not have the rights to put a blanket CC on the publications which we ran through an OCR and post-processing workflow. The outcome of this workflow, while somewhat different from the PDF, was not \"transformative\" and therefore not a strong case for \"fair use\". That said, the majority of these works are \"out of print\" and some are \"out of copyright\", and most cannot be found in digital formats or online, but are only in special collections of certain universities. We provide these resources out of necessity because of the lack of a database for translations (in various languages). And we included the bibliography (with dates for copyright restrictions) in order to adhere to academic standards of citation. It would be advisable to keep the use of this data for ML and not public display, and for any models as outcomes to cite the bibliography we provided for continuity.
", "rawMarkdown": "Thank you for your thoughtful question. We do not have the rights to put a blanket CC on the publications which we ran through an OCR and post-processing workflow. The outcome of this workflow, while somewhat different from the PDF, was not \"transformative\" and therefore not a strong case for \"fair use\". That said, the majority of these works are \"out of print\" and some are \"out of copyright\", and most cannot be found in digital formats or online, but are only in special collections of certain universities. We provide these resources out of necessity because of the lack of a database for translations (in various languages). And we included the bibliography (with dates for copyright restrictions) in order to adhere to academic standards of citation. It would be advisable to keep the use of this data for ML and not public display, and for any models as outcomes to cite the bibliography we provided for continuity.", "votes": null }, { "id": "3382941", "postDate": "12/28/2025 21:32:13", "content": "Thank you. I'll take it that to be fair we should either limit the external/cloud LLM processing to the articles wirth clearly expired copyright or limit oursleves to the locally ran models.
", "rawMarkdown": "Thank you. I'll take it that to be fair we should either limit the external/cloud LLM processing to the articles wirth clearly expired copyright or limit oursleves to the locally ran models.", "votes": null } ], "comments": [ { "id": 3382894, "author_name": "deeppast", "author_url": "", "post_date": "12/28/2025 19:05:43", "content": "Thank you for your thoughtful question. We do not have the rights to put a blanket CC on the publications which we ran through an OCR and post-processing workflow. The outcome of this workflow, while somewhat different from the PDF, was not \"transformative\" and therefore not a strong case for \"fair use\". That said, the majority of these works are \"out of print\" and some are \"out of copyright\", and most cannot be found in digital formats or online, but are only in special collections of certain universities. We provide these resources out of necessity because of the lack of a database for translations (in various languages). And we included the bibliography (with dates for copyright restrictions) in order to adhere to academic standards of citation. It would be advisable to keep the use of this data for ML and not public display, and for any models as outcomes to cite the bibliography we provided for continuity.
", "votes": null, "replies": [ { "id": 3382941, "author_name": "angantyr", "author_url": "", "post_date": "12/28/2025 21:32:13", "content": "Thank you. I'll take it that to be fair we should either limit the external/cloud LLM processing to the articles wirth clearly expired copyright or limit oursleves to the locally ran models.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3382540": "deeppast @ryanholbrook \n\nI wanted to try out LLM models to retrieve transliterations and translation from the massive 952 PDF OCRs when it got to me that some of the documents may not be publicly available.\n\nSo, to be double sure, are the OCRs (publications.csv) from the datasets in the Data section available under CC BY-SA 4.0?", "3382894": "Thank you for your thoughtful question. We do not have the rights to put a blanket CC on the publications which we ran through an OCR and post-processing workflow. The outcome of this workflow, while somewhat different from the PDF, was not \"transformative\" and therefore not a strong case for \"fair use\". That said, the majority of these works are \"out of print\" and some are \"out of copyright\", and most cannot be found in digital formats or online, but are only in special collections of certain universities. We provide these resources out of necessity because of the lack of a database for translations (in various languages). And we included the bibliography (with dates for copyright restrictions) in order to adhere to academic standards of citation. It would be advisable to keep the use of this data for ML and not public display, and for any models as outcomes to cite the bibliography we provided for continuity.", "3382941": "Thank you. I'll take it that to be fair we should either limit the external/cloud LLM processing to the articles wirth clearly expired copyright or limit oursleves to the locally ran models." }, "source": "meta" }