{ "id": 2144, "title": "Reading enormous JSON files", "url": "/competitions/cprod1/discussion/2144", "author_name": "", "post_date": "2012-07-07T00:20:50.287Z", "votes": null, "comment_count": 8, "views": 6998, "content": "

Does anyone have any tips for working with JSON files of this size? Are you doing some homebrew trickery to delimit them before reading? I can't imagine that it's possible to load the training-anotated-text.json file into memory, yet, because of the structure\r\n of JSON files, you can't deal with it in a stream like you can a large CSV.

\r\n

I've tried loading just the products file with rjson and ~10 minutes later and to RAM left, I'm still waiting.

", "messages": [ { "id": "12180", "postDate": "07/07/2012 00:20:50", "content": "

Does anyone have any tips for working with JSON files of this size? Are you doing some homebrew trickery to delimit them before reading? I can't imagine that it's possible to load the training-anotated-text.json file into memory, yet, because of the structure\r\n of JSON files, you can't deal with it in a stream like you can a large CSV.

\r\n

I've tried loading just the products file with rjson and ~10 minutes later and to RAM left, I'm still waiting.

", "rawMarkdown": "", "votes": null }, { "id": "12189", "postDate": "07/07/2012 06:21:19", "content": "

I'd start by reading some part of the file as a stream and then dumping into a "sample" file, just to figure out what's going on. While I see  your point, I can't believe there's no way around loading the whole thing into memory - I mean, 0.7gb zipped? 

", "rawMarkdown": "", "votes": null }, { "id": "12190", "postDate": "07/07/2012 07:02:42", "content": "

Yeah, I know there has to be a way. But it seems like it would especially be easier if the top-level element was an array rather than an object.

\r\n

As far as the data structure is concerned, it defines it as a monolithic whole, and splitting out the parts seems like a trick task to me. Not just as easy as pattern matching to delimit array elements.

", "rawMarkdown": "", "votes": null }, { "id": "12219", "postDate": "07/09/2012 11:22:25", "content": "

I don't know what language you're working in, but in python you can read it line by line by using\r\n
\r\nopen('file','r').readline()
\r\nor
\r\nopen('file','r').readlines('Amount of memory, not number of lines')

", "rawMarkdown": "", "votes": null }, { "id": "12220", "postDate": "07/09/2012 11:40:54", "content": "

FWIW (i am not very clear on the relationship between files, so this refers just to text processing): the initial and closing curly brackets (in the huge files) are there for no good reason at all, so 

\r\n

1. get rid of those + "textItem"

\r\n

2. parse each succesive block into a separate line

\r\n

 

\r\n

I've got a Perl script running now that tries to achieve that, if it works i'll post it here. 

\r\n

 

\r\n

In general, due to the size, the whole thing will be a pain to process...

", "rawMarkdown": "", "votes": null }, { "id": "12221", "postDate": "07/09/2012 11:46:07", "content": "
\r\n

open('file','r').readline()

\r\n
\r\n

Will read is a text file. That's not a problem, if it were lines of text I could operate on virtually unlimited sized files in a stream. The challenge is parsing it as a JSON object.

\r\n

Konrad, that's very helpful.

", "rawMarkdown": "", "votes": null }, { "id": "12222", "postDate": "07/09/2012 11:46:07", "content": "
\r\n

open('file','r').readline()

\r\n
\r\n

Will read is a text file. That's not a problem, if it were lines of text I could operate on virtually unlimited sized files in a stream. The challenge is parsing it as a JSON object.

\r\n

Konrad, that's very helpful.

", "rawMarkdown": "", "votes": null }, { "id": "12229", "postDate": "07/09/2012 16:25:41", "content": "

Ok, here it is. Since I'm not exactly what you'd define as the world's best Perl programmer, a few words of explanation:

\r\n

 

\r\n\r\n
This is very much work in progress, so any comments / improvements / cleaned up versions of the code are welcome.
", "rawMarkdown": "", "votes": null }, { "id": "12375", "postDate": "07/14/2012 01:13:53", "content": "

Hi,

\r\n

I think many of you might know this, but since nobody mentioned it, here's a crude way to do it in Python, using the json module in the standard library

\r\n

 

\r\n

import json

\r\n

filename = 'training-annotated-text.json'

\r\n

f = open(filename, 'r')

\r\n

json_obj = json.load(f) 

\r\n

# json_obj is a dict containing a key 'TextItem'

\r\n

json_data = json_obj['TextItem']

\r\n

# json_data is another dict having the reviews in form of Python lists. These lists are values of the dict.

\r\n

# The lists all contain every word of the review as a string. They have to be concatenated to read the whole review as one string.

\r\n

 

\r\n

Hope this helps.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 12189, "author_name": "konradb", "author_url": "", "post_date": "07/07/2012 06:21:19", "content": "

I'd start by reading some part of the file as a stream and then dumping into a "sample" file, just to figure out what's going on. While I see  your point, I can't believe there's no way around loading the whole thing into memory - I mean, 0.7gb zipped? 

", "votes": null, "replies": [] }, { "id": 12190, "author_name": "zacstewart", "author_url": "", "post_date": "07/07/2012 07:02:42", "content": "

Yeah, I know there has to be a way. But it seems like it would especially be easier if the top-level element was an array rather than an object.

\r\n

As far as the data structure is concerned, it defines it as a monolithic whole, and splitting out the parts seems like a trick task to me. Not just as easy as pattern matching to delimit array elements.

", "votes": null, "replies": [] }, { "id": 12219, "author_name": "nmca48984", "author_url": "", "post_date": "07/09/2012 11:22:25", "content": "

I don't know what language you're working in, but in python you can read it line by line by using\r\n
\r\nopen('file','r').readline()
\r\nor
\r\nopen('file','r').readlines('Amount of memory, not number of lines')

", "votes": null, "replies": [] }, { "id": 12220, "author_name": "konradb", "author_url": "", "post_date": "07/09/2012 11:40:54", "content": "

FWIW (i am not very clear on the relationship between files, so this refers just to text processing): the initial and closing curly brackets (in the huge files) are there for no good reason at all, so 

\r\n

1. get rid of those + "textItem"

\r\n

2. parse each succesive block into a separate line

\r\n

 

\r\n

I've got a Perl script running now that tries to achieve that, if it works i'll post it here. 

\r\n

 

\r\n

In general, due to the size, the whole thing will be a pain to process...

", "votes": null, "replies": [] }, { "id": 12221, "author_name": "zacstewart", "author_url": "", "post_date": "07/09/2012 11:46:07", "content": "
\r\n

open('file','r').readline()

\r\n
\r\n

Will read is a text file. That's not a problem, if it were lines of text I could operate on virtually unlimited sized files in a stream. The challenge is parsing it as a JSON object.

\r\n

Konrad, that's very helpful.

", "votes": null, "replies": [] }, { "id": 12222, "author_name": "zacstewart", "author_url": "", "post_date": "07/09/2012 11:46:07", "content": "
\r\n

open('file','r').readline()

\r\n
\r\n

Will read is a text file. That's not a problem, if it were lines of text I could operate on virtually unlimited sized files in a stream. The challenge is parsing it as a JSON object.

\r\n

Konrad, that's very helpful.

", "votes": null, "replies": [] }, { "id": 12229, "author_name": "konradb", "author_url": "", "post_date": "07/09/2012 16:25:41", "content": "

Ok, here it is. Since I'm not exactly what you'd define as the world's best Perl programmer, a few words of explanation:

\r\n

 

\r\n\r\n
This is very much work in progress, so any comments / improvements / cleaned up versions of the code are welcome.
", "votes": null, "replies": [] }, { "id": 12375, "author_name": "jaidevd", "author_url": "", "post_date": "07/14/2012 01:13:53", "content": "

Hi,

\r\n

I think many of you might know this, but since nobody mentioned it, here's a crude way to do it in Python, using the json module in the standard library

\r\n

 

\r\n

import json

\r\n

filename = 'training-annotated-text.json'

\r\n

f = open(filename, 'r')

\r\n

json_obj = json.load(f) 

\r\n

# json_obj is a dict containing a key 'TextItem'

\r\n

json_data = json_obj['TextItem']

\r\n

# json_data is another dict having the reviews in form of Python lists. These lists are values of the dict.

\r\n

# The lists all contain every word of the review as a string. They have to be concatenated to read the whole review as one string.

\r\n

 

\r\n

Hope this helps.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "12180": "", "12189": "", "12190": "", "12219": "", "12220": "", "12221": "", "12222": "", "12229": "", "12375": "" }, "source": "meta" }