{ "id": 9830, "title": "More Russian Encoding Issues with R on Windows", "url": "/competitions/avito-prohibited-content/discussion/9830", "author_name": "", "post_date": "2014-07-21T13:07:11.147Z", "votes": null, "comment_count": 2, "views": 1675, "content": "

I have asked the following question on StackOverflow, but no one has been able to provide any insight.  I wanted to post it here as well since others may have come across this issue:

\n

http://stackoverflow.com/questions/24830574/output-not-being-encoded-consistently

\n

If I write head(output) my text is not encoded properly (as shown above) whereas if I simply write output$Title[0:3] it will display the text correctly like so:

\n

> output$Title[0:3]
[[1]]
[1] "Renault Logan, 2005"[[2]]
[1] "Складское помещение, 345 м²"[[3]]
[1] "Су-шеф"

\n

However, if I write:

\n

> head(output)
Id Title IsProhibited
1 10000074 Renault Logan, 2005 0
2 10000124 Ñêëàäñêîå ïîìåùåíèå, 345 ì<U+00B2> 0
3 10000175 Ñó-øåô 0
4 10000196 3-ê êâàðòèðà, 64 ì<U+00B2>, 3/5 ýò. 0
5 10000387 Samsung galaxy S4 mini GT-I9190 (÷¸ðíûé) 0
6 10000395 Êàðòèíà Êðûì. Ïîñåëîê Àðîìàò (õîëñò, ìàñëî) 0

\n

Or if I try to write to a CSV (using write.table) the output is not encoded properly (as shown above).

\n

Here is a sample of my data for a reproducible example:

\n

# create test data
test <- structure(list(Id = c(10000074L, 10000124L, 10000175L, 10000196L,
10000387L, 10000395L), Title = c("Zeit 9-25 кг новые автокресла", "2-к квартира, 55 м², 1 эт.",
"Достойная работа", "ВАЗ 2106, 1994", "Водитель с личным а/м Газель",
"Комната 45 м² в 1-к, 3/14 эт."), IsProhibited = structure(c(1L,
1L, 1L, 1L, 1L, 1L), .Label = c("0", "1"), class = "factor")), .Names = c("Id",
"Title", "IsProhibited"), row.names = c(NA, 6L), class = "data.frame")

\n

# outputs correctly as Russian characters
output$Title[0:6]

\n

# but, why do the same rows output incorrectly
# when using head() or write.table()?
head(test)

\n

How can I get my data to be output correctly?

\n

Note: I have set my encoding like so:

\n

Sys.setlocale("LC_CTYPE", "russian") # set locale for Russian encoding...

", "messages": [ { "id": "51008", "postDate": "07/21/2014 13:07:11", "content": "

I have asked the following question on StackOverflow, but no one has been able to provide any insight.  I wanted to post it here as well since others may have come across this issue:

\n

http://stackoverflow.com/questions/24830574/output-not-being-encoded-consistently

\n

If I write head(output) my text is not encoded properly (as shown above) whereas if I simply write output$Title[0:3] it will display the text correctly like so:

\n

> output$Title[0:3]
[[1]]
[1] "Renault Logan, 2005"[[2]]
[1] "Складское помещение, 345 м²"[[3]]
[1] "Су-шеф"

\n

However, if I write:

\n

> head(output)
Id Title IsProhibited
1 10000074 Renault Logan, 2005 0
2 10000124 Ñêëàäñêîå ïîìåùåíèå, 345 ì<U+00B2> 0
3 10000175 Ñó-øåô 0
4 10000196 3-ê êâàðòèðà, 64 ì<U+00B2>, 3/5 ýò. 0
5 10000387 Samsung galaxy S4 mini GT-I9190 (÷¸ðíûé) 0
6 10000395 Êàðòèíà Êðûì. Ïîñåëîê Àðîìàò (õîëñò, ìàñëî) 0

\n

Or if I try to write to a CSV (using write.table) the output is not encoded properly (as shown above).

\n

Here is a sample of my data for a reproducible example:

\n

# create test data
test <- structure(list(Id = c(10000074L, 10000124L, 10000175L, 10000196L,
10000387L, 10000395L), Title = c("Zeit 9-25 кг новые автокресла", "2-к квартира, 55 м², 1 эт.",
"Достойная работа", "ВАЗ 2106, 1994", "Водитель с личным а/м Газель",
"Комната 45 м² в 1-к, 3/14 эт."), IsProhibited = structure(c(1L,
1L, 1L, 1L, 1L, 1L), .Label = c("0", "1"), class = "factor")), .Names = c("Id",
"Title", "IsProhibited"), row.names = c(NA, 6L), class = "data.frame")

\n

# outputs correctly as Russian characters
output$Title[0:6]

\n

# but, why do the same rows output incorrectly
# when using head() or write.table()?
head(test)

\n

How can I get my data to be output correctly?

\n

Note: I have set my encoding like so:

\n

Sys.setlocale("LC_CTYPE", "russian") # set locale for Russian encoding...

", "rawMarkdown": "", "votes": null }, { "id": "51014", "postDate": "07/21/2014 17:00:07", "content": "

It may be related to Windows operating system (or Windows version of R). Namely, I also managed to display data correctly. However only if I limited loading of data to first few hundred records (after then comes some unicode that my R version cannot handle). Anyways with correctly displaying data d , if I write edit(d) then I see that R is not using UTF8 font. It displays cyrillic characters incorrectly.

\n

However, in my experiments I found out that even using incorrectly displaying data the things are working. Naturally, it is impossible to read text - that is why I have limited only to category and subcategory text columns.

\n

If possible try R under Linux or Macintosh, I have been told that unicode support / fonts are there working much easier. However, I have no personal experience with that.

", "rawMarkdown": "", "votes": null }, { "id": "51015", "postDate": "07/21/2014 17:14:35", "content": "

Thanks for your comments.  It has been recommended that I use Linux + R.  But, that's not really an option for me right now.  It is unfortunate to be held up by such a simple thing as text encoding.  For now,  I will just struggle through unless someone has a better option.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 51014, "author_name": "sfin2015", "author_url": "", "post_date": "07/21/2014 17:00:07", "content": "

It may be related to Windows operating system (or Windows version of R). Namely, I also managed to display data correctly. However only if I limited loading of data to first few hundred records (after then comes some unicode that my R version cannot handle). Anyways with correctly displaying data d , if I write edit(d) then I see that R is not using UTF8 font. It displays cyrillic characters incorrectly.

\n

However, in my experiments I found out that even using incorrectly displaying data the things are working. Naturally, it is impossible to read text - that is why I have limited only to category and subcategory text columns.

\n

If possible try R under Linux or Macintosh, I have been told that unicode support / fonts are there working much easier. However, I have no personal experience with that.

", "votes": null, "replies": [] }, { "id": 51015, "author_name": "dominic1", "author_url": "", "post_date": "07/21/2014 17:14:35", "content": "

Thanks for your comments.  It has been recommended that I use Linux + R.  But, that's not really an option for me right now.  It is unfortunate to be held up by such a simple thing as text encoding.  For now,  I will just struggle through unless someone has a better option.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "51008": "", "51014": "", "51015": "" }, "source": "meta" }