{ "id": 9800, "title": "Reading Russian UTF8 data in Windows with R", "url": "/competitions/avito-prohibited-content/discussion/9800", "author_name": "", "post_date": "2014-07-17T15:05:40.097Z", "votes": null, "comment_count": 1, "views": 4397, "content": "

Hi,

\n

I am using R 3.0.2 64-bit on Windows 7 and had first a bit problems reading in the Russian data using it (I have been told UTF R read.csv etc works much better/easier at Linux/Macintosh).

\n

I noticed that easiest way to read Russian UTF8 to R is to change locale first to Russian and then just do read.csv with UTF-8 encoding (without locale changing it was always failing, as my normal locale in Windows is Finnish). This can be done as:

\n

# Read first 10 rows
Sys.setlocale('LC_ALL', 'russian');
d2 <- read.csv("avito_test.tsv", encoding="UTF-8", sep="\\t", header=TRUE, nrows=10)

\n

Update:
- Above has issue that reading of data breaks (when going to say nrows=1000). I thought changing UTF-8 to UTF8 helped but it did not, it reads all in but data in cyrillic font is totally broken.

\n

So any help how to get data loaded in at R is appreciated :)

", "messages": [ { "id": "50785", "postDate": "07/17/2014 15:05:40", "content": "

Hi,

\n

I am using R 3.0.2 64-bit on Windows 7 and had first a bit problems reading in the Russian data using it (I have been told UTF R read.csv etc works much better/easier at Linux/Macintosh).

\n

I noticed that easiest way to read Russian UTF8 to R is to change locale first to Russian and then just do read.csv with UTF-8 encoding (without locale changing it was always failing, as my normal locale in Windows is Finnish). This can be done as:

\n

# Read first 10 rows
Sys.setlocale('LC_ALL', 'russian');
d2 <- read.csv("avito_test.tsv", encoding="UTF-8", sep="\\t", header=TRUE, nrows=10)

\n

Update:
- Above has issue that reading of data breaks (when going to say nrows=1000). I thought changing UTF-8 to UTF8 helped but it did not, it reads all in but data in cyrillic font is totally broken.

\n

So any help how to get data loaded in at R is appreciated :)

", "rawMarkdown": "", "votes": null }, { "id": "50857", "postDate": "07/18/2014 15:39:16", "content": "

Solved above issue - loading with locale set to Russian only without encoding specification seems to be working. However, cyrillic characters are just displayed totally wrong - atleast on my machine.

\n

Below is R sample code if someone has any use for it. It does not score well, but is just simple example. It does not use title, keywords in modelling just category, subcategory, price and urls_cnt.

\n

# Load in data
setwd("H:\\\\kaggle\\\\avito_prob_Content")
Sys.setlocale('LC_ALL', 'russian');
dTrain <- read.csv("avito_train.tsv", sep="\\t", header=TRUE,  nrows=1562937)

# Pre-process (replace NA values by zeroes)
dTrain$price[is.na(dTrain$price)] <- 0
dTrain$is_blocked[is.na(dTrain$is_blocked)] <- 0
dTrain$urls_cnt[is.na(dTrain$urls_cnt)] <- 0

# Create new fields: log price, zero price indicator and
# merged category field
dTrain$logPrice <- log(1+dTrain$price)
dTrain$zeroPrice <- as.integer(dTrain$price == 0)
dTrain$mergedCat <- paste(dTrain$category,dTrain$subcategory, sep="/")

# compute E[is_blocked|category,subcategory] to lookup "table" 
ts1 <- tapply(dTrain$is_blocked, dTrain$mergedCat, mean)
categories = as.numeric(ts1)
names(categories) = rownames(ts1)

# Prepare training data
dSample <- data.frame(is_blocked=dTrain$is_blocked)
dSample$mergedCat <- dTrain$mergedCat
dSample$logPrice <- dTrain$logPrice
dSample$urls_cnt <- dTrain$urls_cnt
dSample$mergedCatScore <- categories[dSample$mergedCat]
dSample$Z1 <- dSample$logPrice * dSample$mergedCatScore

# Train logistic regression model
glm.out1 = glm(is_blocked ~ mergedCatScore + logPrice + urls_cnt + Z1, family=binomial(logit), data=dSample)

# Evaluate model (just visually)
outputData <- data.frame(is_blocked=dSample$is_blocked, estimated=fitted(glm.out1), mergedCat=dSample$mergedCatScore)
sortedData <- outputData[order(-outputData$estimated
),]
cs1 <- cumsum(sortedData$is_blocked)
plot(cs1,col="#880088",cex=0.1,xlim=c(0,500000)) 

# Load test data in
dTest <- read.csv("avito_test.tsv", sep="\\t", header=TRUE)

# Pre-process test data, similarly as dTrain
dTest$price[is.na(dTest$price)] <- 0
dTest$urls_cnt[is.na(dTest$urls_cnt)] <- 0
dTest$logPrice <- log(1+dTest$price)
dTest$zeroPrice <- as.integer(dTest$price == 0)
dTest$mergedCat <- paste(dTest$category,dTest$subcategory, sep="/")

# Save R objects (they are faster to load and compressed) which
# are not used in this example but in case you want to use later.
save(dTest, file="dTest.dat")
save(dTrain, file="dTrain.dat")

# Check for covariate shift, just via numbers / eyes :)
# (if there is such then that is indication that weighted logistic
# regression should be applied)
summary(dTrain$urls_cnt)
summary(dTest$urls_cnt)summary(dTrain$logPrice)
summary(dTest$logPrice)
# -> tail of dTest$urls_cnt is longer than dTrain$urls_cnt, but
# otherwise no clear covariate shift observed

# Prepare test data
dSample <- data.frame(mergedCat=dTest$mergedCat)
dSample$logPrice <- dTest$logPrice
dSample$urls_cnt <- dTest$urls_cnt
dSample$mergedCatScore <- categories[dSample$mergedCat]
dSample$Z1 <- dTest$logPrice * dSample$mergedCatScore

# Do predictions
output <- predict(glm.out1,type="response",dSample)
outputData <- data.frame(id=dTest$itemid, estimated=output)
sortedData <- outputData[order(-outputData$estimated
),]
sortedData$estimated <- NULL

# write output to file
write.csv(sortedData,"submission1.csv",row.names=FALSE)

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 50857, "author_name": "sfin2015", "author_url": "", "post_date": "07/18/2014 15:39:16", "content": "

Solved above issue - loading with locale set to Russian only without encoding specification seems to be working. However, cyrillic characters are just displayed totally wrong - atleast on my machine.

\n

Below is R sample code if someone has any use for it. It does not score well, but is just simple example. It does not use title, keywords in modelling just category, subcategory, price and urls_cnt.

\n

# Load in data
setwd("H:\\\\kaggle\\\\avito_prob_Content")
Sys.setlocale('LC_ALL', 'russian');
dTrain <- read.csv("avito_train.tsv", sep="\\t", header=TRUE,  nrows=1562937)

# Pre-process (replace NA values by zeroes)
dTrain$price[is.na(dTrain$price)] <- 0
dTrain$is_blocked[is.na(dTrain$is_blocked)] <- 0
dTrain$urls_cnt[is.na(dTrain$urls_cnt)] <- 0

# Create new fields: log price, zero price indicator and
# merged category field
dTrain$logPrice <- log(1+dTrain$price)
dTrain$zeroPrice <- as.integer(dTrain$price == 0)
dTrain$mergedCat <- paste(dTrain$category,dTrain$subcategory, sep="/")

# compute E[is_blocked|category,subcategory] to lookup "table" 
ts1 <- tapply(dTrain$is_blocked, dTrain$mergedCat, mean)
categories = as.numeric(ts1)
names(categories) = rownames(ts1)

# Prepare training data
dSample <- data.frame(is_blocked=dTrain$is_blocked)
dSample$mergedCat <- dTrain$mergedCat
dSample$logPrice <- dTrain$logPrice
dSample$urls_cnt <- dTrain$urls_cnt
dSample$mergedCatScore <- categories[dSample$mergedCat]
dSample$Z1 <- dSample$logPrice * dSample$mergedCatScore

# Train logistic regression model
glm.out1 = glm(is_blocked ~ mergedCatScore + logPrice + urls_cnt + Z1, family=binomial(logit), data=dSample)

# Evaluate model (just visually)
outputData <- data.frame(is_blocked=dSample$is_blocked, estimated=fitted(glm.out1), mergedCat=dSample$mergedCatScore)
sortedData <- outputData[order(-outputData$estimated
),]
cs1 <- cumsum(sortedData$is_blocked)
plot(cs1,col="#880088",cex=0.1,xlim=c(0,500000)) 

# Load test data in
dTest <- read.csv("avito_test.tsv", sep="\\t", header=TRUE)

# Pre-process test data, similarly as dTrain
dTest$price[is.na(dTest$price)] <- 0
dTest$urls_cnt[is.na(dTest$urls_cnt)] <- 0
dTest$logPrice <- log(1+dTest$price)
dTest$zeroPrice <- as.integer(dTest$price == 0)
dTest$mergedCat <- paste(dTest$category,dTest$subcategory, sep="/")

# Save R objects (they are faster to load and compressed) which
# are not used in this example but in case you want to use later.
save(dTest, file="dTest.dat")
save(dTrain, file="dTrain.dat")

# Check for covariate shift, just via numbers / eyes :)
# (if there is such then that is indication that weighted logistic
# regression should be applied)
summary(dTrain$urls_cnt)
summary(dTest$urls_cnt)summary(dTrain$logPrice)
summary(dTest$logPrice)
# -> tail of dTest$urls_cnt is longer than dTrain$urls_cnt, but
# otherwise no clear covariate shift observed

# Prepare test data
dSample <- data.frame(mergedCat=dTest$mergedCat)
dSample$logPrice <- dTest$logPrice
dSample$urls_cnt <- dTest$urls_cnt
dSample$mergedCatScore <- categories[dSample$mergedCat]
dSample$Z1 <- dTest$logPrice * dSample$mergedCatScore

# Do predictions
output <- predict(glm.out1,type="response",dSample)
outputData <- data.frame(id=dTest$itemid, estimated=output)
sortedData <- outputData[order(-outputData$estimated
),]
sortedData$estimated <- NULL

# write output to file
write.csv(sortedData,"submission1.csv",row.names=FALSE)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "50785": "", "50857": "" }, "source": "meta" }