When I try to parse attrs using Python's json module, I get ValueError with some examples, for example with those:
{"Тип объявления":"Продам", "Количество комнат":"2", "Вид объекта":"Вторичка", "Этаж":"3", "Этаж":"Не первый и не последний", "Этажей в доме":"< 5", "Этажей в доме":"4", "Тип дома":"Кирпичный", "Адрес":"Мира ул 28"}
{"Тип объявления":"Сдам", "Срок аренды":"На длительный срок", "Комнат в квартире":"4", "Этаж":"Не первый и не последний", "Этаж":"3", "Этажей в доме":"5-8", "Этажей в доме":"5", "Тип дома":"Кирпичный", "Адрес":"Асаткина ул 31"}
{"Тип объявления":"Продам", "Количество комнат":"1", "Вид объекта":"Вторичка", "Этаж":"Не первый и не последний", "Этаж":"11", "Этажей в доме":"13-16", "Этажей в доме":"13", "Тип дома":"Кирпичный", "Адрес":"Нижняя Дуброва ул 17"}
However when I copy and paste, it works (at least with some):
\n
In [43]: json.loads('{"Тип объявления":"Продам", "Количество комнат":"1", "Вид объекта":"Вторичка", "Этаж":"Не первый и не последний", "Этаж":"11", "Этажей в доме":"13-16", "Этажей в доме":"13", "Тип дома":"Кирпичный", "Адрес":"Нижняя Дуброва ул17"}')
Out[43]:
{u'\\u0410\\u0434\\u0440\\u0435\\u0441': u'\\u041d\\u0438\\u0436\\u043d\\u044f\\u044f \\u0414\\u0443\\u0431\\u0440\\u043e\\u0432\\u0430 \\u0443\\u043b17',
u'\\u0412\\u0438\\u0434 \\u043e\\u0431\\u044a\\u0435\\u043a\\u0442\\u0430': u'\\u0412\\u0442\\u043e\\u0440\\u0438\\u0447\\u043a\\u0430',
u'\\u041a\\u043e\\u043b\\u0438\\u0447\\u0435\\u0441\\u0442\\u0432\\u043e \\u043a\\u043e\\u043c\\u043d\\u0430\\u0442': u'1',
u'\\u0422\\u0438\\u043f \\u0434\\u043e\\u043c\\u0430': u'\\u041a\\u0438\\u0440\\u043f\\u0438\\u0447\\u043d\\u044b\\u0439',
u'\\u0422\\u0438\\u043f \\u043e\\u0431\\u044a\\u044f\\u0432\\u043b\\u0435\\u043d\\u0438\\u044f': u'\\u041f\\u0440\\u043e\\u0434\\u0430\\u043c',
u'\\u042d\\u0442\\u0430\\u0436': u'11',
u'\\u042d\\u0442\\u0430\\u0436\\u0435\\u0439 \\u0432 \\u0434\\u043e\\u043c\\u0435': u'13'}
Any ideas?
", "messages": [ { "id": "50148", "postDate": "07/08/2014 19:05:14", "content": "When I try to parse attrs using Python's json module, I get ValueError with some examples, for example with those:
{"Тип объявления":"Продам", "Количество комнат":"2", "Вид объекта":"Вторичка", "Этаж":"3", "Этаж":"Не первый и не последний", "Этажей в доме":"< 5", "Этажей в доме":"4", "Тип дома":"Кирпичный", "Адрес":"Мира ул 28"}
{"Тип объявления":"Сдам", "Срок аренды":"На длительный срок", "Комнат в квартире":"4", "Этаж":"Не первый и не последний", "Этаж":"3", "Этажей в доме":"5-8", "Этажей в доме":"5", "Тип дома":"Кирпичный", "Адрес":"Асаткина ул 31"}
{"Тип объявления":"Продам", "Количество комнат":"1", "Вид объекта":"Вторичка", "Этаж":"Не первый и не последний", "Этаж":"11", "Этажей в доме":"13-16", "Этажей в доме":"13", "Тип дома":"Кирпичный", "Адрес":"Нижняя Дуброва ул 17"}
However when I copy and paste, it works (at least with some):
\n
In [43]: json.loads('{"Тип объявления":"Продам", "Количество комнат":"1", "Вид объекта":"Вторичка", "Этаж":"Не первый и не последний", "Этаж":"11", "Этажей в доме":"13-16", "Этажей в доме":"13", "Тип дома":"Кирпичный", "Адрес":"Нижняя Дуброва ул17"}')
Out[43]:
{u'\\u0410\\u0434\\u0440\\u0435\\u0441': u'\\u041d\\u0438\\u0436\\u043d\\u044f\\u044f \\u0414\\u0443\\u0431\\u0440\\u043e\\u0432\\u0430 \\u0443\\u043b17',
u'\\u0412\\u0438\\u0434 \\u043e\\u0431\\u044a\\u0435\\u043a\\u0442\\u0430': u'\\u0412\\u0442\\u043e\\u0440\\u0438\\u0447\\u043a\\u0430',
u'\\u041a\\u043e\\u043b\\u0438\\u0447\\u0435\\u0441\\u0442\\u0432\\u043e \\u043a\\u043e\\u043c\\u043d\\u0430\\u0442': u'1',
u'\\u0422\\u0438\\u043f \\u0434\\u043e\\u043c\\u0430': u'\\u041a\\u0438\\u0440\\u043f\\u0438\\u0447\\u043d\\u044b\\u0439',
u'\\u0422\\u0438\\u043f \\u043e\\u0431\\u044a\\u044f\\u0432\\u043b\\u0435\\u043d\\u0438\\u044f': u'\\u041f\\u0440\\u043e\\u0434\\u0430\\u043c',
u'\\u042d\\u0442\\u0430\\u0436': u'11',
u'\\u042d\\u0442\\u0430\\u0436\\u0435\\u0439 \\u0432 \\u0434\\u043e\\u043c\\u0435': u'13'}
Any ideas?
", "rawMarkdown": "", "votes": null }, { "id": "51075", "postDate": "07/23/2014 00:49:14", "content": "How about try `eval` them into python dictionaries directly?
", "rawMarkdown": "", "votes": null }, { "id": "51081", "postDate": "07/23/2014 05:21:22", "content": "Tried that to start with.
EDIT: Turns out that a combination of json.loads() and eval() works. Here's a demo:
https://github.com/zygmuntz/kaggle-avito
Handling Cyrillic letters in Python is quite messy. I wish all modern programming languages were Cyrillic-based. Then everyone by now would ask on forums how to make the language tackle this weird Latin alphabet.
\nTry putting this at the beginning of your script (before all imports):
\n# -*- coding: utf-8 -*-
\nDunno if this works in iPython notebooks though.
", "rawMarkdown": "", "votes": null }, { "id": "51111", "postDate": "07/23/2014 23:56:17", "content": "For people using pandas, `attrs` can be converted to a column of dictionaries by:
\nimport pandas as pd
train = pd.read_csv('avito_train.tsv', sep='\\t')
train['attrs'] = train['attrs'].fillna('{}').astype(str).apply(lambda x:eval(x.replace('/"', '\\'').replace("'}", '"}')))
Hello, there is another way to convert attrs to dictionary using json:
\njson.loads(re.sub('/\\"(?!(,\\s"|}))','\\\\"',item["attrs"]).replace("\\t"," ").replace("\\n"," ")) if len(item["attrs"])>0 else {}
\nThis is tested and works well.
", "rawMarkdown": "", "votes": null }, { "id": "52498", "postDate": "08/27/2014 10:15:41", "content": "Perfect thanks
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 51075, "author_name": "apophenia", "author_url": "", "post_date": "07/23/2014 00:49:14", "content": "How about try `eval` them into python dictionaries directly?
", "votes": null, "replies": [] }, { "id": 51081, "author_name": "zygmunt", "author_url": "", "post_date": "07/23/2014 05:21:22", "content": "Tried that to start with.
EDIT: Turns out that a combination of json.loads() and eval() works. Here's a demo:
https://github.com/zygmuntz/kaggle-avito
Handling Cyrillic letters in Python is quite messy. I wish all modern programming languages were Cyrillic-based. Then everyone by now would ask on forums how to make the language tackle this weird Latin alphabet.
\nTry putting this at the beginning of your script (before all imports):
\n# -*- coding: utf-8 -*-
\nDunno if this works in iPython notebooks though.
", "votes": null, "replies": [] }, { "id": 51111, "author_name": "apophenia", "author_url": "", "post_date": "07/23/2014 23:56:17", "content": "For people using pandas, `attrs` can be converted to a column of dictionaries by:
\nimport pandas as pd
train = pd.read_csv('avito_train.tsv', sep='\\t')
train['attrs'] = train['attrs'].fillna('{}').astype(str).apply(lambda x:eval(x.replace('/"', '\\'').replace("'}", '"}')))
Hello, there is another way to convert attrs to dictionary using json:
\njson.loads(re.sub('/\\"(?!(,\\s"|}))','\\\\"',item["attrs"]).replace("\\t"," ").replace("\\n"," ")) if len(item["attrs"])>0 else {}
\nThis is tested and works well.
", "votes": null, "replies": [] }, { "id": 52498, "author_name": "delijati80", "author_url": "", "post_date": "08/27/2014 10:15:41", "content": "Perfect thanks
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "50148": "", "51075": "", "51081": "", "51083": "", "51111": "", "51117": "", "52498": "" }, "source": "meta" }