{ "id": 234576, "title": "some publications does not look like publications", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/234576", "author_name": "", "post_date": "2021-04-25T02:36:15.030546900Z", "votes": 10, "comment_count": 3, "views": 0, "content": "

Some publications are very weird and does not look like scientific publications.
\nfor example : c12a3ad2-17bc-4b8b-8779-77ba2680ae96 contains 1576310 tokens which is a lot, maybe those are books.

\n

Looking at the head of the json file, we have this :

\n

\"AN OFFICIAL JOURNAL OF THE AMERICAN COLLEGE OF RHEUMATOLOGY \\n \\n Arthritis & Rheumatism \\n \\n VOLUME 65, NUMBER 10 (SUPPLEMENT) \\n \\n OCTOBER 2013 \\n \\n ABSTRACT SUPPLEMENT 2013 ANNUAL MEETING \\n October 25\\xad30, 2013 San Diego, CA \\n \\n \\x0cGet On-Demand Annual \\n Meeting Education with SessionSelect! \\n \\n ABTOTENNUDSE! E \\n Z&ZzZZZZZZZZ \\n \\n SessionSelect is a comprehensive digital library of recorded presentations from the annual meeting. Within 24 hours of virtually every live session, you can view fully-synchronized presentations with speaker slides and audio--just as if you were actually in the room. Take advantage of the most convenient way to experience the education from the annual meeting at your own pace. With SessionSelect you can: \\n t4 UBZVQUPEBUFPOUIFNPTUSFMFWBOUJOGPSNBUJPOBOESFTFBSDIQFSUBJOJOHUPSIFVNBUJDEJTFBTFT t\" QQMZTFTTJPOJOGPSNBUJPOUPZPVSQSBDUJDFUPFOIBODFZPVSLOPXMFEHF \\n DPNQFUFODF \\n and performance t7JFXDPOUFOUPOZPVSNPCJMFEFWJDFPSEPXOMPBE.1öMFTGPSMJTUFOJOHUPBVEJPPOUIFHP \\n SessionSelect Lounge \\n /P/FFEUP.JTTB4FTTJPO8IJMF:PVSF0OTJUF \\n Got some down time between sessions? Relax while you replay sessions--at your convenience--on dedicated computer terminals. \\n Location: Hall D Lobby Hours: .POEBZ \\n 0DUPCFS …\"

", "messages": [ { "id": "1283523", "postDate": "04/25/2021 02:36:15", "content": "

Some publications are very weird and does not look like scientific publications.
\nfor example : c12a3ad2-17bc-4b8b-8779-77ba2680ae96 contains 1576310 tokens which is a lot, maybe those are books.

\n

Looking at the head of the json file, we have this :

\n

\"AN OFFICIAL JOURNAL OF THE AMERICAN COLLEGE OF RHEUMATOLOGY \\n \\n Arthritis & Rheumatism \\n \\n VOLUME 65, NUMBER 10 (SUPPLEMENT) \\n \\n OCTOBER 2013 \\n \\n ABSTRACT SUPPLEMENT 2013 ANNUAL MEETING \\n October 25\\xad30, 2013 San Diego, CA \\n \\n \\x0cGet On-Demand Annual \\n Meeting Education with SessionSelect! \\n \\n ABTOTENNUDSE! E \\n Z&ZzZZZZZZZZ \\n \\n SessionSelect is a comprehensive digital library of recorded presentations from the annual meeting. Within 24 hours of virtually every live session, you can view fully-synchronized presentations with speaker slides and audio--just as if you were actually in the room. Take advantage of the most convenient way to experience the education from the annual meeting at your own pace. With SessionSelect you can: \\n t4 UBZVQUPEBUFPOUIFNPTUSFMFWBOUJOGPSNBUJPOBOESFTFBSDIQFSUBJOJOHUPSIFVNBUJDEJTFBTFT t\" QQMZTFTTJPOJOGPSNBUJPOUPZPVSQSBDUJDFUPFOIBODFZPVSLOPXMFEHF \\n DPNQFUFODF \\n and performance t7JFXDPOUFOUPOZPVSNPCJMFEFWJDFPSEPXOMPBE.1öMFTGPSMJTUFOJOHUPBVEJPPOUIFHP \\n SessionSelect Lounge \\n /P/FFEUP.JTTB4FTTJPO8IJMF:PVSF0OTJUF \\n Got some down time between sessions? Relax while you replay sessions--at your convenience--on dedicated computer terminals. \\n Location: Hall D Lobby Hours: .POEBZ \\n 0DUPCFS …\"

", "rawMarkdown": "Some publications are very weird and does not look like scientific publications.\nfor example : c12a3ad2-17bc-4b8b-8779-77ba2680ae96 contains 1576310 tokens which is a lot, maybe those are books.\n\nLooking at the head of the json file, we have this :\n\n\"AN OFFICIAL JOURNAL OF THE AMERICAN COLLEGE OF RHEUMATOLOGY \\n \\n Arthritis & Rheumatism \\n \\n VOLUME 65, NUMBER 10 (SUPPLEMENT) \\n \\n OCTOBER 2013 \\n \\n ABSTRACT SUPPLEMENT 2013 ANNUAL MEETING \\n October 25\\xad30, 2013 San Diego, CA \\n \\n \\x0cGet On-Demand Annual \\n Meeting Education with SessionSelect! \\n \\n ABTOTENNUDSE! E \\n Z&ZzZZZZZZZZ \\n \\n SessionSelect is a comprehensive digital library of recorded presentations from the annual meeting. Within 24 hours of virtually every live session, you can view fully-synchronized presentations with speaker slides and audio--just as if you were actually in the room. Take advantage of the most convenient way to experience the education from the annual meeting at your own pace. With SessionSelect you can: \\n t4 UBZVQUPEBUFPOUIFNPTUSFMFWBOUJOGPSNBUJPOBOESFTFBSDIQFSUBJOJOHUPSIFVNBUJDEJTFBTFT t\" QQMZTFTTJPOJOGPSNBUJPOUPZPVSQSBDUJDFUPFOIBODFZPVSLOPXMFEHF \\n DPNQFUFODF \\n and performance t7JFXDPOUFOUPOZPVSNPCJMFEFWJDFPSEPXOMPBE.1öMFTGPSMJTUFOJOHUPBVEJPPOUIFHP \\n SessionSelect Lounge \\n /P/FFEUP.JTTB4FTTJPO8IJMF:PVSF0OTJUF \\n Got some down time between sessions? Relax while you replay sessions--at your convenience--on dedicated computer terminals. \\n Location: Hall D Lobby Hours: .POEBZ \\n 0DUPCFS ...\"", "votes": null }, { "id": "1283735", "postDate": "04/25/2021 07:59:29", "content": "

Dear Professor,

\n

You are correct, this is a book containing abstracts of all of the posters, lectures, and presentations given at the 2013 annual meeting of the American College of Rheumatology in San Diego, California. It is 1331 pages long, which would explain the large number of tokens.

", "rawMarkdown": "Dear Professor,\n\nYou are correct, this is a book containing abstracts of all of the posters, lectures, and presentations given at the 2013 annual meeting of the American College of Rheumatology in San Diego, California. It is 1331 pages long, which would explain the large number of tokens.", "votes": null }, { "id": "1286323", "postDate": "04/27/2021 20:15:44", "content": "

I think the data is not properly cleaned and present increasing computations.

", "rawMarkdown": "I think the data is not properly cleaned and present increasing computations.", "votes": null }, { "id": "1296644", "postDate": "05/07/2021 11:54:03", "content": "

Hi @mchahhou - what did you decide to do with this meeting abstracts? I'm considering leaving it out as it a) breaks my Spacy sentencizer and b) is not representative of \"normal\" papers. c) lots of the text is nonsense or missing whitespace

\n

Did you take a different approach?

", "rawMarkdown": "Hi @mchahhou - what did you decide to do with this meeting abstracts? I'm considering leaving it out as it a) breaks my Spacy sentencizer and b) is not representative of \"normal\" papers. c) lots of the text is nonsense or missing whitespace\n\nDid you take a different approach?", "votes": null } ], "comments": [ { "id": 1283735, "author_name": "amysillman", "author_url": "", "post_date": "04/25/2021 07:59:29", "content": "

Dear Professor,

\n

You are correct, this is a book containing abstracts of all of the posters, lectures, and presentations given at the 2013 annual meeting of the American College of Rheumatology in San Diego, California. It is 1331 pages long, which would explain the large number of tokens.

", "votes": null, "replies": [] }, { "id": 1286323, "author_name": "darknez", "author_url": "", "post_date": "04/27/2021 20:15:44", "content": "

I think the data is not properly cleaned and present increasing computations.

", "votes": null, "replies": [] }, { "id": 1296644, "author_name": "cascadenite", "author_url": "", "post_date": "05/07/2021 11:54:03", "content": "

Hi @mchahhou - what did you decide to do with this meeting abstracts? I'm considering leaving it out as it a) breaks my Spacy sentencizer and b) is not representative of \"normal\" papers. c) lots of the text is nonsense or missing whitespace

\n

Did you take a different approach?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1283523": "Some publications are very weird and does not look like scientific publications.\nfor example : c12a3ad2-17bc-4b8b-8779-77ba2680ae96 contains 1576310 tokens which is a lot, maybe those are books.\n\nLooking at the head of the json file, we have this :\n\n\"AN OFFICIAL JOURNAL OF THE AMERICAN COLLEGE OF RHEUMATOLOGY \\n \\n Arthritis & Rheumatism \\n \\n VOLUME 65, NUMBER 10 (SUPPLEMENT) \\n \\n OCTOBER 2013 \\n \\n ABSTRACT SUPPLEMENT 2013 ANNUAL MEETING \\n October 25\\xad30, 2013 San Diego, CA \\n \\n \\x0cGet On-Demand Annual \\n Meeting Education with SessionSelect! \\n \\n ABTOTENNUDSE! E \\n Z&ZzZZZZZZZZ \\n \\n SessionSelect is a comprehensive digital library of recorded presentations from the annual meeting. Within 24 hours of virtually every live session, you can view fully-synchronized presentations with speaker slides and audio--just as if you were actually in the room. Take advantage of the most convenient way to experience the education from the annual meeting at your own pace. With SessionSelect you can: \\n t4 UBZVQUPEBUFPOUIFNPTUSFMFWBOUJOGPSNBUJPOBOESFTFBSDIQFSUBJOJOHUPSIFVNBUJDEJTFBTFT t\" QQMZTFTTJPOJOGPSNBUJPOUPZPVSQSBDUJDFUPFOIBODFZPVSLOPXMFEHF \\n DPNQFUFODF \\n and performance t7JFXDPOUFOUPOZPVSNPCJMFEFWJDFPSEPXOMPBE.1öMFTGPSMJTUFOJOHUPBVEJPPOUIFHP \\n SessionSelect Lounge \\n /P/FFEUP.JTTB4FTTJPO8IJMF:PVSF0OTJUF \\n Got some down time between sessions? Relax while you replay sessions--at your convenience--on dedicated computer terminals. \\n Location: Hall D Lobby Hours: .POEBZ \\n 0DUPCFS ...\"", "1283735": "Dear Professor,\n\nYou are correct, this is a book containing abstracts of all of the posters, lectures, and presentations given at the 2013 annual meeting of the American College of Rheumatology in San Diego, California. It is 1331 pages long, which would explain the large number of tokens.", "1286323": "I think the data is not properly cleaned and present increasing computations.", "1296644": "Hi @mchahhou - what did you decide to do with this meeting abstracts? I'm considering leaving it out as it a) breaks my Spacy sentencizer and b) is not representative of \"normal\" papers. c) lots of the text is nonsense or missing whitespace\n\nDid you take a different approach?" }, "source": "meta" }