{ "id": 7136, "title": "Is there any example code for this competition?", "url": "/competitions/galaxy-zoo-the-galaxy-challenge/discussion/7136", "author_name": "", "post_date": "2014-02-20T08:58:01.297Z", "votes": null, "comment_count": 1, "views": 2159, "content": "

Hi, all,

\n

I 'd rather to know that is there any example code for introduction to this competition? Becuase I'm new to this galaxy recognition domain.

\n

 

\n

Thanks~

\n

Milton

", "messages": [ { "id": "39127", "postDate": "02/20/2014 08:58:01", "content": "

Hi, all,

\n

I 'd rather to know that is there any example code for introduction to this competition? Becuase I'm new to this galaxy recognition domain.

\n

 

\n

Thanks~

\n

Milton

", "rawMarkdown": "", "votes": null }, { "id": "40039", "postDate": "03/09/2014 11:54:15", "content": "

Hi,
Sorry for late response. Anyways, below is sample code I used to extract features from training data set. Similarly it needs to be run for test data set. 

Remarks:
- run time can be 3-6 days (separately for both training and test) depending from your hardware.
- images are not properly pre-processed (like histogram equalized). However I have checked that marginal distributions of features are roughly same for training and test data sets.
- Fourier transform is not proper (should be 2D and not 1D)
- I am converting RGB data to gray using formula: 0.21*red + 0.72*green + 0.07*blue - thus color information is lost (feel free to add colour channel features too)
- I my experiements I am fitting regression tree per class, this example just has sample code for Output1.1 class at the end. So you need to do either 37 separate trees (to output all classes), or you could do multivariate regression tree (which predicts all 37 simultaneously) - I am not sure how well it would perform.
- This sample fits only one regression tree (thus no ensemble). I have also tested linear regression, logistic regression, support vector regression and random forest. However I have found sgbm = stochastic gradient boosting to work best so far.

Required files:
- training_solutions_rev1.csv and training data jpg files must be in folder where R is being run (you can set working directory using setwd("c:\\\\temp") - if you have all files in c:\\temp.

\n

Required R packages (these you must install, running R as admin, using command install.packages and if I recall correctly EBImage required a bit more complicated install): 
-fractaldim
-biOps
-EBImage
-rpart

\n


Sample R Code:

\n

# Include needed libraries
library(fractaldim)
library(biOps)
library(EBImage)
library(rpart)

\n

# Set random number generator seed (so that results can be replicated later)
set.seed(1234)

\n

# get training solutions (to figure out jpeg filenames and also to get Y values = ClassX.Y)
trSolutions <- read.csv("training_solutions_rev1.csv", header=TRUE)

\n

# Basic features
distinctValues <- NULL
meanv <- NULL
logmean <- NULL
medianv <- NULL
sd <- NULL
qr <- NULL
logsd <- NULL
q25 <- NULL
q75 <- NULL

\n

# Fractal features
fd1 <- NULL
fd2 <- NULL
fd3 <- NULL

\n

# Fourier features
fftConcentration <- NULL
fftReal <- NULL
fftImaginary <- NULL

\n

# Gray tail mass distribution features
linFitAlpha <- NULL
linFitBeta <- NULL

\n

# Blob features
blobs <- NULL
meanMass <- NULL
medianMass <- NULL
sdMass <- NULL
maxMass <- NULL
countSD1Mass <- NULL
countSD2Mass <- NULL
countSD3Mass <- NULL
countSD4Mass <- NULL

\n

#
# Loop over all training data (= images)
#
for(imgi in 1:nrow(trSolutions)){

\n

# Load JPEG file
print(sprintf("Processing %s (%d/%d)", trSolutions$GalaxyID[imgi], imgi, nrow(trSolutions)))
flush.console()
x <- readJpeg(sprintf("%s.jpg", trSolutions$GalaxyID[imgi]))

\n

#
# Extract features from images
#

\n

# Calculate number of blobs
y = thresh(x, 10, 10, 0.10)
y = opening(y, makeBrush(5, shape="disc"))
z = bwlabel(y)
blobs[imgi] <- max(z)

\n

# Calculate blob segments
blobMassDistribution <- table(z)
segment <- rownames(blobMassDistribution)
mass <- as.numeric(blobMassDistribution)
massTable <- data.frame(segment,mass)

\n

# Remove background blob
massTableFiltered <- massTable[ massTable$mass < max(massTable$mass),]

\n

# Calculate blob mass distribution features (simple statistics)
meanMass[imgi] <- mean(massTableFiltered$mass)
medianMass[imgi] <- median(massTableFiltered$mass)
sdMass[imgi] <- sd(massTableFiltered$mass)
maxMass[imgi] <- max(massTableFiltered$mass)

\n

# Calculate number of "big blobs", defined by 1,2,3 and 4 standard mass deviations from mean mass
countSD1Mass[imgi] <- sum(massTableFiltered$mass >= meanMass[imgi] + 1*sdMass[imgi])
countSD2Mass[imgi] <- sum(massTableFiltered$mass >= meanMass[imgi] + 2*sdMass[imgi])
countSD3Mass[imgi] <- sum(massTableFiltered$mass >= meanMass[imgi] + 3*sdMass[imgi])
countSD4Mass[imgi] <- sum(massTableFiltered$mass >= meanMass[imgi] + 4*sdMass[imgi])

\n

dims <- dim(x)
plot(x)
img <- data.frame(x)

\n

xsize <- min (dim(img))
ysize <- xsize

\n

# get R,G,B channels data
red <- (img[, 1:ysize])
green <- (img[, (ysize+1):(ysize*2)])
blue <- (img[, (ysize*2+1):(ysize*3)])

\n

# convert to gray levels
gray <- as.integer(unlist(0.21 * red + 0.72 * green + 0.07 * blue))

\n


plevels <- table(gray)
distinctValues[imgi] <- length(rownames(plevels))
levelValues <- as.integer( rownames(plevels) )

\n

densityValues <- as.numeric (plevels)
cumulativeValues <- cumsum(densityValues)
cumulativeProbabilityValues <- cumulativeValues / max(cumulativeValues)

\n

standardizedLevelValues <- (levelValues - min(levelValues)) / (max(levelValues)-min(levelValues))
massd <- data.frame(rowIndex=1:length(levelValues),standardizedLevelValues,cumulativeProbabilityValues)

\n

# Left truncate by 10% (to remove "darkness from background" and focus to more interesting part of mass distribution)
dropIndex <- as.integer(length(levelValues) * 0.10)
massd <- massd[massd$rowIndex > dropIndex,]

\n

massd$standardizedLevelValues <- massd$standardizedLevelValues - min(massd$standardizedLevelValues)
massd$standardizedLevelValues <- sqrt(massd$standardizedLevelValues)
linfit <- lm(cumulativeProbabilityValues~standardizedLevelValues, data=massd)
linFitAlpha[imgi] <- as.numeric(linfit$coefficients[1])
linFitBeta[imgi] <- as.numeric(linfit$coefficients[2])

\n

# Calculate basic features
meanv[imgi] <- mean(gray)
logmean[imgi] <- mean(log(1+gray))
medianv[imgi] <- median(gray)
sd[imgi] <- sd(gray)
logsd[imgi] <- sd(log(1+gray))
q25[imgi] <- quantile(gray, 0.25)
q75[imgi] <- quantile(gray, 0.75)
qr[imgi] <- q75[imgi] - q25[imgi]

\n

# Calculate Fourier transform features
fourierTrans <- fft(gray)
fftConcentration[imgi] <- mean( abs(fourierTrans) ) # the more peaked around 0,0 at complex space the more periodical
fftReal[imgi] <- mean( abs(Re(fourierTrans)) )
fftImaginary[imgi] <- mean( abs(Im(fourierTrans)) )

\n

# Calculate fractal dimension (3 diff. estimates)
dim(gray) <- c(xsize, ysize)
d1 <- fd.estim.isotropic(gray, p.index = 1, direction='hv', plot.loglog = FALSE, plot.allpoints = FALSE)
fd1[imgi] <- d1$fd

\n

d2 <- fd.estim.squareincr(gray, p.index = 1, plot.loglog = FALSE, plot.allpoints = FALSE)
fd2[imgi] <- d2$fd

\n

d3 <- fd.estim.filter1(gray, p.index = 1, plot.loglog = FALSE, plot.allpoints = FALSE)
fd3[imgi] <- d3$fd
}

\n

#
# Attach training data and gathered features
#
outputData <- trSolutions
outputData$distinctValues <- distinctValues
outputData$meanv <- meanv
outputData$logmean <- logmean
outputData$medianv <- medianv
outputData$sd <- sd
outputData$qr <- qr
outputData$logsd <- logsd
outputData$q25 <- q25
outputData$q75 <- q75
outputData$fd1 <- fd1
outputData$fd2 <- fd2
outputData$fd3 <- fd3
outputData$fftConcentration <-fftConcentration
outputData$fftReal <-fftReal
outputData$fftImaginary <- fftImaginary
outputData$linFitAlpha <- linFitAlpha
outputData$linFitBeta <- linFitBeta
outputData$meanMass <- meanMass
outputData$medianMass <- medianMass
outputData$sdMass <- sdMass
outputData$maxMass <- maxMass
outputData$countSD1Mass <- countSD1Mass
outputData$countSD2Mass <- countSD2Mass
outputData$countSD3Mass <- countSD3Mass
outputData$countSD4Mass <- countSD4Mass
outputData$blobs <- blobs
outputData$GalaxyID <- NULL

\n


# Save outputs
write.csv(outputData, "outputData.csv")

\n

# Fit sample regression tree to predict Class1.1 (you need to do similarly for others classes).
# and feel free to add variable selection etc
fit <- rpart(Class1.1 ~ meanMass+medianMass+countSD1Mass+countSD2Mass+countSD3Mass+countSD4Mass+blobs+linFitAlpha+linFitBeta
+fftConcentration+fftReal+fftImaginary+meanv+sd+logsd+medianv+fd1+fd2+fd3+distinctValues+maxMass+sdMass, data=outputData)
# Calculate mean squared error for tree prediction and for unconditional mean prediction
mseTree <- mean( (predict(fit) - outputData$Class1.1)^2 )
mseMean <- mean( (mean(outputData$Class1.1) - outputData$Class1.1)^2 )

\n

# If improvement is clearly < 1 then fitting regression tree was worth it, provided it did not over-fit
# to data which you need to check separately, (= result is better than unconditional mean estimate)
improvement <- mseTree/mseMean

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 40039, "author_name": "sfin2015", "author_url": "", "post_date": "03/09/2014 11:54:15", "content": "

Hi,
Sorry for late response. Anyways, below is sample code I used to extract features from training data set. Similarly it needs to be run for test data set. 

Remarks:
- run time can be 3-6 days (separately for both training and test) depending from your hardware.
- images are not properly pre-processed (like histogram equalized). However I have checked that marginal distributions of features are roughly same for training and test data sets.
- Fourier transform is not proper (should be 2D and not 1D)
- I am converting RGB data to gray using formula: 0.21*red + 0.72*green + 0.07*blue - thus color information is lost (feel free to add colour channel features too)
- I my experiements I am fitting regression tree per class, this example just has sample code for Output1.1 class at the end. So you need to do either 37 separate trees (to output all classes), or you could do multivariate regression tree (which predicts all 37 simultaneously) - I am not sure how well it would perform.
- This sample fits only one regression tree (thus no ensemble). I have also tested linear regression, logistic regression, support vector regression and random forest. However I have found sgbm = stochastic gradient boosting to work best so far.

Required files:
- training_solutions_rev1.csv and training data jpg files must be in folder where R is being run (you can set working directory using setwd("c:\\\\temp") - if you have all files in c:\\temp.

\n

Required R packages (these you must install, running R as admin, using command install.packages and if I recall correctly EBImage required a bit more complicated install): 
-fractaldim
-biOps
-EBImage
-rpart

\n


Sample R Code:

\n

# Include needed libraries
library(fractaldim)
library(biOps)
library(EBImage)
library(rpart)

\n

# Set random number generator seed (so that results can be replicated later)
set.seed(1234)

\n

# get training solutions (to figure out jpeg filenames and also to get Y values = ClassX.Y)
trSolutions <- read.csv("training_solutions_rev1.csv", header=TRUE)

\n

# Basic features
distinctValues <- NULL
meanv <- NULL
logmean <- NULL
medianv <- NULL
sd <- NULL
qr <- NULL
logsd <- NULL
q25 <- NULL
q75 <- NULL

\n

# Fractal features
fd1 <- NULL
fd2 <- NULL
fd3 <- NULL

\n

# Fourier features
fftConcentration <- NULL
fftReal <- NULL
fftImaginary <- NULL

\n

# Gray tail mass distribution features
linFitAlpha <- NULL
linFitBeta <- NULL

\n

# Blob features
blobs <- NULL
meanMass <- NULL
medianMass <- NULL
sdMass <- NULL
maxMass <- NULL
countSD1Mass <- NULL
countSD2Mass <- NULL
countSD3Mass <- NULL
countSD4Mass <- NULL

\n

#
# Loop over all training data (= images)
#
for(imgi in 1:nrow(trSolutions)){

\n

# Load JPEG file
print(sprintf("Processing %s (%d/%d)", trSolutions$GalaxyID[imgi], imgi, nrow(trSolutions)))
flush.console()
x <- readJpeg(sprintf("%s.jpg", trSolutions$GalaxyID[imgi]))

\n

#
# Extract features from images
#

\n

# Calculate number of blobs
y = thresh(x, 10, 10, 0.10)
y = opening(y, makeBrush(5, shape="disc"))
z = bwlabel(y)
blobs[imgi] <- max(z)

\n

# Calculate blob segments
blobMassDistribution <- table(z)
segment <- rownames(blobMassDistribution)
mass <- as.numeric(blobMassDistribution)
massTable <- data.frame(segment,mass)

\n

# Remove background blob
massTableFiltered <- massTable[ massTable$mass < max(massTable$mass),]

\n

# Calculate blob mass distribution features (simple statistics)
meanMass[imgi] <- mean(massTableFiltered$mass)
medianMass[imgi] <- median(massTableFiltered$mass)
sdMass[imgi] <- sd(massTableFiltered$mass)
maxMass[imgi] <- max(massTableFiltered$mass)

\n

# Calculate number of "big blobs", defined by 1,2,3 and 4 standard mass deviations from mean mass
countSD1Mass[imgi] <- sum(massTableFiltered$mass >= meanMass[imgi] + 1*sdMass[imgi])
countSD2Mass[imgi] <- sum(massTableFiltered$mass >= meanMass[imgi] + 2*sdMass[imgi])
countSD3Mass[imgi] <- sum(massTableFiltered$mass >= meanMass[imgi] + 3*sdMass[imgi])
countSD4Mass[imgi] <- sum(massTableFiltered$mass >= meanMass[imgi] + 4*sdMass[imgi])

\n

dims <- dim(x)
plot(x)
img <- data.frame(x)

\n

xsize <- min (dim(img))
ysize <- xsize

\n

# get R,G,B channels data
red <- (img[, 1:ysize])
green <- (img[, (ysize+1):(ysize*2)])
blue <- (img[, (ysize*2+1):(ysize*3)])

\n

# convert to gray levels
gray <- as.integer(unlist(0.21 * red + 0.72 * green + 0.07 * blue))

\n


plevels <- table(gray)
distinctValues[imgi] <- length(rownames(plevels))
levelValues <- as.integer( rownames(plevels) )

\n

densityValues <- as.numeric (plevels)
cumulativeValues <- cumsum(densityValues)
cumulativeProbabilityValues <- cumulativeValues / max(cumulativeValues)

\n

standardizedLevelValues <- (levelValues - min(levelValues)) / (max(levelValues)-min(levelValues))
massd <- data.frame(rowIndex=1:length(levelValues),standardizedLevelValues,cumulativeProbabilityValues)

\n

# Left truncate by 10% (to remove "darkness from background" and focus to more interesting part of mass distribution)
dropIndex <- as.integer(length(levelValues) * 0.10)
massd <- massd[massd$rowIndex > dropIndex,]

\n

massd$standardizedLevelValues <- massd$standardizedLevelValues - min(massd$standardizedLevelValues)
massd$standardizedLevelValues <- sqrt(massd$standardizedLevelValues)
linfit <- lm(cumulativeProbabilityValues~standardizedLevelValues, data=massd)
linFitAlpha[imgi] <- as.numeric(linfit$coefficients[1])
linFitBeta[imgi] <- as.numeric(linfit$coefficients[2])

\n

# Calculate basic features
meanv[imgi] <- mean(gray)
logmean[imgi] <- mean(log(1+gray))
medianv[imgi] <- median(gray)
sd[imgi] <- sd(gray)
logsd[imgi] <- sd(log(1+gray))
q25[imgi] <- quantile(gray, 0.25)
q75[imgi] <- quantile(gray, 0.75)
qr[imgi] <- q75[imgi] - q25[imgi]

\n

# Calculate Fourier transform features
fourierTrans <- fft(gray)
fftConcentration[imgi] <- mean( abs(fourierTrans) ) # the more peaked around 0,0 at complex space the more periodical
fftReal[imgi] <- mean( abs(Re(fourierTrans)) )
fftImaginary[imgi] <- mean( abs(Im(fourierTrans)) )

\n

# Calculate fractal dimension (3 diff. estimates)
dim(gray) <- c(xsize, ysize)
d1 <- fd.estim.isotropic(gray, p.index = 1, direction='hv', plot.loglog = FALSE, plot.allpoints = FALSE)
fd1[imgi] <- d1$fd

\n

d2 <- fd.estim.squareincr(gray, p.index = 1, plot.loglog = FALSE, plot.allpoints = FALSE)
fd2[imgi] <- d2$fd

\n

d3 <- fd.estim.filter1(gray, p.index = 1, plot.loglog = FALSE, plot.allpoints = FALSE)
fd3[imgi] <- d3$fd
}

\n

#
# Attach training data and gathered features
#
outputData <- trSolutions
outputData$distinctValues <- distinctValues
outputData$meanv <- meanv
outputData$logmean <- logmean
outputData$medianv <- medianv
outputData$sd <- sd
outputData$qr <- qr
outputData$logsd <- logsd
outputData$q25 <- q25
outputData$q75 <- q75
outputData$fd1 <- fd1
outputData$fd2 <- fd2
outputData$fd3 <- fd3
outputData$fftConcentration <-fftConcentration
outputData$fftReal <-fftReal
outputData$fftImaginary <- fftImaginary
outputData$linFitAlpha <- linFitAlpha
outputData$linFitBeta <- linFitBeta
outputData$meanMass <- meanMass
outputData$medianMass <- medianMass
outputData$sdMass <- sdMass
outputData$maxMass <- maxMass
outputData$countSD1Mass <- countSD1Mass
outputData$countSD2Mass <- countSD2Mass
outputData$countSD3Mass <- countSD3Mass
outputData$countSD4Mass <- countSD4Mass
outputData$blobs <- blobs
outputData$GalaxyID <- NULL

\n


# Save outputs
write.csv(outputData, "outputData.csv")

\n

# Fit sample regression tree to predict Class1.1 (you need to do similarly for others classes).
# and feel free to add variable selection etc
fit <- rpart(Class1.1 ~ meanMass+medianMass+countSD1Mass+countSD2Mass+countSD3Mass+countSD4Mass+blobs+linFitAlpha+linFitBeta
+fftConcentration+fftReal+fftImaginary+meanv+sd+logsd+medianv+fd1+fd2+fd3+distinctValues+maxMass+sdMass, data=outputData)
# Calculate mean squared error for tree prediction and for unconditional mean prediction
mseTree <- mean( (predict(fit) - outputData$Class1.1)^2 )
mseMean <- mean( (mean(outputData$Class1.1) - outputData$Class1.1)^2 )

\n

# If improvement is clearly < 1 then fitting regression tree was worth it, provided it did not over-fit
# to data which you need to check separately, (= result is better than unconditional mean estimate)
improvement <- mseTree/mseMean

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "39127": "", "40039": "" }, "source": "meta" }