![\"enter](\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4721/TSNE_features_numeric.png\")
The attachment on this post shows in two dimensions how two features' missing value patterns are close to each other depending depending on all the other features, using t-SNE (train numeric features only, not including test). There are many mini-clusters of features. Some are overlapping, as their missing value patterns are identical.
\n\nSum of squared error of the 2-dimensional reproduction: 0.099456712
\n\nInitial matrix: Pearson's correlation coefficient = Phi coefficient = MCC of missing values (969x969 grid)
\n\nWarning: very large picture (4212x3312, be ready to zoom!). In pink color lies the label we are trying to predict: supposed 0 as NA and 1 as present. The closest one to the label was obviously L3_S32_F3850, found here. But as it is mainly univariate, we can't judge whether it is the best or not to use. However, a quick look at the picture shows features might be clustered together (on their patterns).
\n\n
I added the correlation matrix if you need to reproduce the result (or if you are looking to tweak). Use the following if in Windows to get exactly the same result for the T-SNE visual:
\n\nlibrary(data.table)\nlibrary(Rtsne)\nlibrary(ggplot2)\nlibrary(ggrepel)\ncor_out <- as.matrix(fread("cor_train_numeric.csv", header = TRUE, sep = ","))\ngc(verbose = FALSE)\nset.seed(78)\ntsne_model <- Rtsne(data.frame(cor_out),\n dims = 2,\n #initial_dims = 50,\n initial_dims = ncol(cor_out),\n perplexity = 322, #floor((ncol(cor_out)-1)/3)\n theta = 0.00,\n check_duplicates = FALSE,\n pca = FALSE,\n max_iter = 1350,\n verbose = TRUE,\n is_distance = FALSE)\ncorMatrix_out <- as.data.frame(tsne_model$Y)\n cor_kmeans <- kmeans(corMatrix_out, centers = 5, iter.max = 10, nstart = 3)\n corMatrix_outclust <- as.factor(c(cor_kmeans$cluster[1:968], 6))\ncorMatrix_names <- colnames(cor_out)\nggplot(corMatrix_out, aes(x = V1, y = V2, color = corMatrix_outclust, shape = corMatrix_outclust)) + geom_point(size = 2.5) + geom_rug() + stat_ellipse(type = "norm") + ggtitle("T-SNE of Features") + xlab("X") + ylab("Y") + labs(color = "Cluster", shape = "Cluster") + geom_text_repel(aes(x = V1, y = V2, label = corMatrix_names), size = 2.8)\nTo replicate the correlation matrix, use the propagate library with a small grid size (like 194 => 15 matrices), else you will literally explode R memory. It uses the ff package to store data in a file, and then you would just have to pick the correlation matrix out of created file variable.
\n\nScript for reproducing the correlation matrix (need 16GB or more RAM depending on your R implementation, along with half an hour):
\n\nlibrary(propagate)\nfor (i in 1:969) {\n cat("\\rStep ", i, "\\n", sep = "")\n train_data[!is.na(train_data[, i]), i] <- 1\n train_data[is.na(train_data[, i]), i] <- 0\n if (i %% 10) {gc(verbose = FALSE)}\n}\ngc(verbose = FALSE)\n\ncor_table <- bigcor(train_data, fun = "cor", size = 194, verbose = TRUE)\nThe attachment on this post shows in two dimensions how two features' missing value patterns are close to each other depending depending on all the other features, using t-SNE (train numeric features only, not including test). There are many mini-clusters of features. Some are overlapping, as their missing value patterns are identical.
\n\nSum of squared error of the 2-dimensional reproduction: 0.099456712
\n\nInitial matrix: Pearson's correlation coefficient = Phi coefficient = MCC of missing values (969x969 grid)
\n\nWarning: very large picture (4212x3312, be ready to zoom!). In pink color lies the label we are trying to predict: supposed 0 as NA and 1 as present. The closest one to the label was obviously L3_S32_F3850, found here. But as it is mainly univariate, we can't judge whether it is the best or not to use. However, a quick look at the picture shows features might be clustered together (on their patterns).
\n\n
I added the correlation matrix if you need to reproduce the result (or if you are looking to tweak). Use the following if in Windows to get exactly the same result for the T-SNE visual:
\n\nlibrary(data.table)\nlibrary(Rtsne)\nlibrary(ggplot2)\nlibrary(ggrepel)\ncor_out <- as.matrix(fread("cor_train_numeric.csv", header = TRUE, sep = ","))\ngc(verbose = FALSE)\nset.seed(78)\ntsne_model <- Rtsne(data.frame(cor_out),\n dims = 2,\n #initial_dims = 50,\n initial_dims = ncol(cor_out),\n perplexity = 322, #floor((ncol(cor_out)-1)/3)\n theta = 0.00,\n check_duplicates = FALSE,\n pca = FALSE,\n max_iter = 1350,\n verbose = TRUE,\n is_distance = FALSE)\ncorMatrix_out <- as.data.frame(tsne_model$Y)\n cor_kmeans <- kmeans(corMatrix_out, centers = 5, iter.max = 10, nstart = 3)\n corMatrix_outclust <- as.factor(c(cor_kmeans$cluster[1:968], 6))\ncorMatrix_names <- colnames(cor_out)\nggplot(corMatrix_out, aes(x = V1, y = V2, color = corMatrix_outclust, shape = corMatrix_outclust)) + geom_point(size = 2.5) + geom_rug() + stat_ellipse(type = "norm") + ggtitle("T-SNE of Features") + xlab("X") + ylab("Y") + labs(color = "Cluster", shape = "Cluster") + geom_text_repel(aes(x = V1, y = V2, label = corMatrix_names), size = 2.8)\nTo replicate the correlation matrix, use the propagate library with a small grid size (like 194 => 15 matrices), else you will literally explode R memory. It uses the ff package to store data in a file, and then you would just have to pick the correlation matrix out of created file variable.
\n\nScript for reproducing the correlation matrix (need 16GB or more RAM depending on your R implementation, along with half an hour):
\n\nlibrary(propagate)\nfor (i in 1:969) {\n cat("\\rStep ", i, "\\n", sep = "")\n train_data[!is.na(train_data[, i]), i] <- 1\n train_data[is.na(train_data[, i]), i] <- 0\n if (i %% 10) {gc(verbose = FALSE)}\n}\ngc(verbose = FALSE)\n\ncor_table <- bigcor(train_data, fun = "cor", size = 194, verbose = TRUE)\n@Laurae ... very nice work!!.... I have few questions :
\n\n1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.\n2)How do you normally choose theta and perplexity in each case.\n3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.
\n\nThank you .
", "rawMarkdown": "Laurae ... very nice work!!.... I have few questions :\r\n\r\n1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.\r\n2)How do you normally choose theta and perplexity in each case.\r\n3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)\r\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.\r\n\r\nThank you .", "votes": null }, { "id": "132601", "postDate": "08/27/2016 10:07:54", "content": "[quote=Alpha;132591]
\n\n@Laurae ... very nice work!!.... I have few questions :
\n\n1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.\n2)How do you normally choose theta and perplexity in each case.\n3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.
\n\nThank you .
\n\n[/quote]
\n\n\n\n\n1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.
\n
If you meant what is the difference between those two:
\n\nIt is the reason why pre-processing/cleaning is more important to take care about than the machine learning model you use: you will spend more time doing an appropriate pre-processing/cleaning than applying a model (in a business scenario). A 1 day/month computing task was reduced to 1 hour or so, along with optimization for the target metric.
\n\n\n\n\n2)How do you normally choose theta and perplexity in each case.
\n
Theta = 0 for exact t-SNE, ]0, 1] for Barnes-Hut t-SNE - it is up to you to choose because the computational complexity changes, but also the approximation can have a negative/beneficial result on gradient descent
\n\nPerplexity = floor((ncol(my_data)-1)/3) to minimize the distance matrix reproduction error (maximum accepted perplexity threshold by t-SNE)
\n\n\n\n\n3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)
\n
Similar plot not at all. The plot I used was showing features, not observations. As there are features that are identical when using MCC on patterns, then there are many overlapping points. In case of observations, most of them are not unique and you will end up with ~1M points on a scatterplot. But then, you can color them by class, which is its typical use. In addition, you may do intra-class clusters. However, on 1M points it would take literally forever to perform a t-SNE. You will also potentially overplot, which will require a smart use of scatterplots.
\n\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.
\n\nWhat cluster? (your question is too large or I did not understand the question - still trying to give an appropriate answer in the mean time though)
\n\nThere are many embedding techniques (dimensionality reduction), ranging from FA, PCA, ICA, MFA..., to SVD, NNMF, t-SNE, largeVis, Autoencoders... All depends on your knowledge on the data (such as presence of missing values, matrix input size, output features, sparsity, computational availabilities...). Labels have mostly nothing to do with embedding, the embedding quality can only be assessed using a supervised machine learning model to test its appropriateness (and the loss of information against the original training set).
\n\nIf the target is a visualization, a technique itself is not alone: you'd rather know what is a Promax, Varimax, etc. when doing a Factor Analysis for instance to not discard a technique "because the visualization was bad". Same for choosing the right seed: typically 100 to 1000 tries to minimize the reproduction error / get better output is enough, and I doubt someone will want to run t-SNE 100 times on 2 million observations for a "up to 99.99%+ reproduction error gain" (which is largely possible on sparse matrices when the dimensionality increases).
", "rawMarkdown": "[quote=Alpha;132591]\r\n\r\n@Laurae ... very nice work!!.... I have few questions :\r\n\r\n1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.\r\n2)How do you normally choose theta and perplexity in each case.\r\n3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)\r\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.\r\n\r\nThank you .\r\n\r\n[/quote]\r\n\r\n> 1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.\r\n\r\nIf you meant what is the difference between those two:\r\n\r\n* One is computable in less than one hour (from loading data to computing t-SNE) while the other would take forever (or never end)\r\n* We don't need 1M features just to look how original features (turned into observations) are spatially together in a 2D plot\r\n* We can compute directly using the MCC matrix which is the metric we are optimizing for initially (MCC = Phi coefficient on contingency table = Pearson correlation on binary values)\r\n\r\nIt is the reason why pre-processing/cleaning is more important to take care about than the machine learning model you use: you will spend more time doing an appropriate pre-processing/cleaning than applying a model (in a business scenario). A 1 day/month computing task was reduced to 1 hour or so, along with optimization for the target metric.\r\n\r\n> 2)How do you normally choose theta and perplexity in each case.\r\n\r\nTheta = 0 for exact t-SNE, ]0, 1] for Barnes-Hut t-SNE - it is up to you to choose because the computational complexity changes, but also the approximation can have a negative/beneficial result on gradient descent\r\n\r\nPerplexity = floor((ncol(my_data)-1)/3) to minimize the distance matrix reproduction error (maximum accepted perplexity threshold by t-SNE)\r\n\r\n> 3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)\r\n\r\nSimilar plot not at all. The plot I used was showing features, not observations. As there are features that are identical when using MCC on patterns, then there are many overlapping points. In case of observations, most of them are not unique and you will end up with ~1M points on a scatterplot. But then, you can color them by class, which is its typical use. In addition, you may do intra-class clusters. However, on 1M points it would take literally forever to perform a t-SNE. You will also potentially overplot, which will require a smart use of scatterplots.\r\n\r\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.\r\n\r\nWhat cluster? (your question is too large or I did not understand the question - still trying to give an appropriate answer in the mean time though)\r\n\r\nThere are many embedding techniques (dimensionality reduction), ranging from FA, PCA, ICA, MFA..., to SVD, NNMF, t-SNE, largeVis, Autoencoders... All depends on your knowledge on the data (such as presence of missing values, matrix input size, output features, sparsity, computational availabilities...). Labels have mostly nothing to do with embedding, the embedding quality can only be assessed using a supervised machine learning model to test its appropriateness (and the loss of information against the original training set).\r\n\r\nIf the target is a visualization, a technique itself is not alone: you'd rather know what is a Promax, Varimax, etc. when doing a Factor Analysis for instance to not discard a technique \"because the visualization was bad\". Same for choosing the right seed: typically 100 to 1000 tries to minimize the reproduction error / get better output is enough, and I doubt someone will want to run t-SNE 100 times on 2 million observations for a \"up to 99.99%+ reproduction error gain\" (which is largely possible on sparse matrices when the dimensionality increases).", "votes": null }, { "id": "132603", "postDate": "08/27/2016 11:18:20", "content": "@Laurae Thank you for your detailed reply.Please forgive me but I have some few confusions/questions:
\n\n1)If I understood correctly,tsne tries to embed the multi dimensional data points into lower dimensions(say 2 or 3 D) in such a way that relative distance between data points remains same (atleast tries to error the error between relative distance of data points in higher dimensions and relative distance of data points in lower dimensions).Is is something to related to MCC matrix which you talked about.
\n\n2) If I understood correctly about when to use which one\n a)we can visualize features on 2D plot if we perform Rtsne on correlation matrix ;
\n\n(b)we can visualize observations on 2D plot if we perform Rtsne on actual data ;\n3)This is related to my last post question : "Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class."
\n\nPlease consider the pasted image.Observations from class are merged inside other class obervation. I want to visualize this ( well separated by label colours ) using Rtsne.For this kind of data what kind of embedding you would recommend.\nI'm motivated by the Rtsne on MNIST digit classification data which is indeed high dimensional and Rtsne could well separate the classes.
\n\nPlease correct me wherever I'm wrong .
\n\nThank you once again for your help !!
", "rawMarkdown": "Laurae Thank you for your detailed reply.Please forgive me but I have some few confusions/questions:\r\n\r\n\r\n1)If I understood correctly,tsne tries to embed the multi dimensional data points into lower dimensions(say 2 or 3 D) in such a way that relative distance between data points remains same (atleast tries to error the error between relative distance of data points in higher dimensions and relative distance of data points in lower dimensions).Is is something to related to MCC matrix which you talked about.\r\n\r\n2) If I understood correctly about when to use which one\r\n a)we can visualize features on 2D plot if we perform Rtsne on correlation matrix ; \r\n\r\n (b)we can visualize observations on 2D plot if we perform Rtsne on actual data ;\r\n\r\n3)This is related to my last post question : \"Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.\"\r\n\r\nPlease consider the pasted image.Observations from class are merged inside other class obervation. I want to visualize this ( well separated by label colours ) using Rtsne.For this kind of data what kind of embedding you would recommend.\r\nI'm motivated by the Rtsne on MNIST digit classification data which is indeed high dimensional and Rtsne could well separate the classes.\r\n\r\n\r\nPlease correct me wherever I'm wrong .\r\n\r\nThank you once again for your help !!", "votes": null }, { "id": "132605", "postDate": "08/27/2016 11:52:26", "content": "\n\n\n1) Is is something to related to MCC matrix which you talked about.
\n
t-SNE has nothing related to MCC directly. It uses the Kullback-Leibler divergence as the metric to optimize. The input used is the Pearson's correlation matrix on binary missing values, which is the same as Phi coefficient (which is the same as MCC).
\n\n\n\n\n2) If I understood correctly about when to use which one
\n\n\na)we can visualize features on 2D plot if we perform Rtsne on correlation matrix ; \n
Yes but that's not the only way to visualize features. You can also transpose the training set to invert observations/features' roles.
\n\n\n\n\n\n(b)we can visualize observations on 2D plot if we perform Rtsne on actual data ;\n
Yes.
\n\n\n\n\n3)This is related to my last post question : "Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class."
\n \nPlease consider the pasted image.Observations from class are merged inside other class obervation. I want to visualize this ( well separated by label colours ) using Rtsne.For this kind of data what kind of embedding you would recommend.
\n
Without knowing what method was used nor what the colors are referring to, it is impossible to provide an appropriate answer. You need to remember that most embeddings (Factor Analysis, Multidimensional Scaling, t-SNE, largeVis...) are reliant on random initialization and an optimizer. One result on a randomized method is not enough to find whether the method is inappropriate. One can only make more tries to try getting an appropriate visualization, or change method if it is still not appropriate enough.
\n\nAssuming you have a 70,000x784 matrix to deal with:
\n\nIf the objective is visualization, and the computation available time was supposedly infinite:
\n\nIf the objective is visualization, and the computation available time is slightly limited (like 1 day):
\n\nIf the objective is visualization, and the computation available time is very limited (like 1 hour):
\n\n@Laurae colours are class labels.Points in the plot are coloured by class labels(binary).I'm trying to use Rtsne on it and I want to know would that be appropriate for such inseparable data .Or are there any other similar technique which could be used to see separate clusters of classes on 2D
", "rawMarkdown": "Laurae colours are class labels.Points in the plot are coloured by class labels(binary).I'm trying to use Rtsne on it and I want to know would that be appropriate for such inseparable data .Or are there any other similar technique which could be used to see separate clusters of classes on 2D", "votes": null }, { "id": "142377", "postDate": "11/01/2016 21:35:48", "content": "@Laurae, what's a good resource for learning about t-SNE?
", "rawMarkdown": "Laurae, what's a good resource for learning about t-SNE?", "votes": null }, { "id": "142385", "postDate": "11/01/2016 22:54:24", "content": "@HueyKwik -> I've recently found this. Excellent resource for playing with and developing some intuition about how it works.
", "rawMarkdown": "HueyKwik -> I've recently found [this][1]. Excellent resource for playing with and developing some intuition about how it works.\r\n\r\n\r\n [1]: http://distill.pub/2016/misread-tsne/", "votes": null }, { "id": "142395", "postDate": "11/02/2016 00:04:23", "content": "[quote=Alchemist;142385]
\n\n@HueyKwik -> I've recently found this. Excellent resource for playing with and developing some intuition about how it works.
\n\n[/quote]
\n\nThis site really helps understand some of the odd behavior of t-SNE. Thanks for sharing!
", "rawMarkdown": "[quote=Alchemist;142385]\r\n\r\n@HueyKwik -> I've recently found [this][1]. Excellent resource for playing with and developing some intuition about how it works.\r\n\r\n\r\n [1]: http://distill.pub/2016/misread-tsne/\r\n\r\n[/quote]\r\n\r\nThis site really helps understand some of the odd behavior of t-SNE. Thanks for sharing!", "votes": null } ], "comments": [ { "id": 132591, "author_name": "arpangupta87", "author_url": "", "post_date": "08/27/2016 06:16:57", "content": "@Laurae ... very nice work!!.... I have few questions :
\n\n1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.\n2)How do you normally choose theta and perplexity in each case.\n3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.
\n\nThank you .
", "votes": null, "replies": [] }, { "id": 132601, "author_name": "laurae2", "author_url": "", "post_date": "08/27/2016 10:07:54", "content": "[quote=Alpha;132591]
\n\n@Laurae ... very nice work!!.... I have few questions :
\n\n1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.\n2)How do you normally choose theta and perplexity in each case.\n3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.
\n\nThank you .
\n\n[/quote]
\n\n\n\n\n1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.
\n
If you meant what is the difference between those two:
\n\nIt is the reason why pre-processing/cleaning is more important to take care about than the machine learning model you use: you will spend more time doing an appropriate pre-processing/cleaning than applying a model (in a business scenario). A 1 day/month computing task was reduced to 1 hour or so, along with optimization for the target metric.
\n\n\n\n\n2)How do you normally choose theta and perplexity in each case.
\n
Theta = 0 for exact t-SNE, ]0, 1] for Barnes-Hut t-SNE - it is up to you to choose because the computational complexity changes, but also the approximation can have a negative/beneficial result on gradient descent
\n\nPerplexity = floor((ncol(my_data)-1)/3) to minimize the distance matrix reproduction error (maximum accepted perplexity threshold by t-SNE)
\n\n\n\n\n3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)
\n
Similar plot not at all. The plot I used was showing features, not observations. As there are features that are identical when using MCC on patterns, then there are many overlapping points. In case of observations, most of them are not unique and you will end up with ~1M points on a scatterplot. But then, you can color them by class, which is its typical use. In addition, you may do intra-class clusters. However, on 1M points it would take literally forever to perform a t-SNE. You will also potentially overplot, which will require a smart use of scatterplots.
\n\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.
\n\nWhat cluster? (your question is too large or I did not understand the question - still trying to give an appropriate answer in the mean time though)
\n\nThere are many embedding techniques (dimensionality reduction), ranging from FA, PCA, ICA, MFA..., to SVD, NNMF, t-SNE, largeVis, Autoencoders... All depends on your knowledge on the data (such as presence of missing values, matrix input size, output features, sparsity, computational availabilities...). Labels have mostly nothing to do with embedding, the embedding quality can only be assessed using a supervised machine learning model to test its appropriateness (and the loss of information against the original training set).
\n\nIf the target is a visualization, a technique itself is not alone: you'd rather know what is a Promax, Varimax, etc. when doing a Factor Analysis for instance to not discard a technique "because the visualization was bad". Same for choosing the right seed: typically 100 to 1000 tries to minimize the reproduction error / get better output is enough, and I doubt someone will want to run t-SNE 100 times on 2 million observations for a "up to 99.99%+ reproduction error gain" (which is largely possible on sparse matrices when the dimensionality increases).
", "votes": null, "replies": [] }, { "id": 132603, "author_name": "arpangupta87", "author_url": "", "post_date": "08/27/2016 11:18:20", "content": "@Laurae Thank you for your detailed reply.Please forgive me but I have some few confusions/questions:
\n\n1)If I understood correctly,tsne tries to embed the multi dimensional data points into lower dimensions(say 2 or 3 D) in such a way that relative distance between data points remains same (atleast tries to error the error between relative distance of data points in higher dimensions and relative distance of data points in lower dimensions).Is is something to related to MCC matrix which you talked about.
\n\n2) If I understood correctly about when to use which one\n a)we can visualize features on 2D plot if we perform Rtsne on correlation matrix ;
\n\n(b)we can visualize observations on 2D plot if we perform Rtsne on actual data ;\n3)This is related to my last post question : "Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class."
\n\nPlease consider the pasted image.Observations from class are merged inside other class obervation. I want to visualize this ( well separated by label colours ) using Rtsne.For this kind of data what kind of embedding you would recommend.\nI'm motivated by the Rtsne on MNIST digit classification data which is indeed high dimensional and Rtsne could well separate the classes.
\n\nPlease correct me wherever I'm wrong .
\n\nThank you once again for your help !!
", "votes": null, "replies": [] }, { "id": 132605, "author_name": "laurae2", "author_url": "", "post_date": "08/27/2016 11:52:26", "content": "\n\n\n1) Is is something to related to MCC matrix which you talked about.
\n
t-SNE has nothing related to MCC directly. It uses the Kullback-Leibler divergence as the metric to optimize. The input used is the Pearson's correlation matrix on binary missing values, which is the same as Phi coefficient (which is the same as MCC).
\n\n\n\n\n2) If I understood correctly about when to use which one
\n\n\na)we can visualize features on 2D plot if we perform Rtsne on correlation matrix ; \n
Yes but that's not the only way to visualize features. You can also transpose the training set to invert observations/features' roles.
\n\n\n\n\n\n(b)we can visualize observations on 2D plot if we perform Rtsne on actual data ;\n
Yes.
\n\n\n\n\n3)This is related to my last post question : "Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class."
\n \nPlease consider the pasted image.Observations from class are merged inside other class obervation. I want to visualize this ( well separated by label colours ) using Rtsne.For this kind of data what kind of embedding you would recommend.
\n
Without knowing what method was used nor what the colors are referring to, it is impossible to provide an appropriate answer. You need to remember that most embeddings (Factor Analysis, Multidimensional Scaling, t-SNE, largeVis...) are reliant on random initialization and an optimizer. One result on a randomized method is not enough to find whether the method is inappropriate. One can only make more tries to try getting an appropriate visualization, or change method if it is still not appropriate enough.
\n\nAssuming you have a 70,000x784 matrix to deal with:
\n\nIf the objective is visualization, and the computation available time was supposedly infinite:
\n\nIf the objective is visualization, and the computation available time is slightly limited (like 1 day):
\n\nIf the objective is visualization, and the computation available time is very limited (like 1 hour):
\n\n@Laurae colours are class labels.Points in the plot are coloured by class labels(binary).I'm trying to use Rtsne on it and I want to know would that be appropriate for such inseparable data .Or are there any other similar technique which could be used to see separate clusters of classes on 2D
", "votes": null, "replies": [] }, { "id": 142377, "author_name": "", "author_url": "", "post_date": "11/01/2016 21:35:48", "content": "@Laurae, what's a good resource for learning about t-SNE?
", "votes": null, "replies": [] }, { "id": 142385, "author_name": "alchemist", "author_url": "", "post_date": "11/01/2016 22:54:24", "content": "@HueyKwik -> I've recently found this. Excellent resource for playing with and developing some intuition about how it works.
", "votes": null, "replies": [] }, { "id": 142395, "author_name": "jeffhebert", "author_url": "", "post_date": "11/02/2016 00:04:23", "content": "[quote=Alchemist;142385]
\n\n@HueyKwik -> I've recently found this. Excellent resource for playing with and developing some intuition about how it works.
\n\n[/quote]
\n\nThis site really helps understand some of the odd behavior of t-SNE. Thanks for sharing!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "132294": "The attachment on this post shows in two dimensions how two features' missing value patterns are close to each other depending depending on all the other features, using t-SNE (train numeric features only, not including test). There are many mini-clusters of features. Some are overlapping, as their missing value patterns are identical.\r\n\r\nSum of squared error of the 2-dimensional reproduction: 0.099456712\r\n\r\nInitial matrix: Pearson's correlation coefficient = Phi coefficient = MCC of missing values (969x969 grid)\r\n\r\nWarning: very large [picture][2] (4212x3312, be ready to zoom!). In pink color lies the label we are trying to predict: supposed 0 as NA and 1 as present. The closest one to the label was obviously L3_S32_F3850, found [here][1]. But as it is mainly univariate, we can't judge whether it is the best or not to use. However, a quick look at the picture shows features might be clustered together (on their patterns).\r\n\r\n![enter image description here][3]\r\n\r\nI added the correlation matrix if you need to reproduce the result (or if you are looking to tweak). Use the following if in Windows to get exactly the same result for the T-SNE visual:\r\n\r\n library(data.table)\r\n library(Rtsne)\r\n library(ggplot2)\r\n library(ggrepel)\r\n cor_out <- as.matrix(fread(\"cor_train_numeric.csv\", header = TRUE, sep = \",\"))\r\n gc(verbose = FALSE)\r\n set.seed(78)\r\n tsne_model <- Rtsne(data.frame(cor_out),\r\n dims = 2,\r\n #initial_dims = 50,\r\n initial_dims = ncol(cor_out),\r\n perplexity = 322, #floor((ncol(cor_out)-1)/3)\r\n theta = 0.00,\r\n check_duplicates = FALSE,\r\n pca = FALSE,\r\n max_iter = 1350,\r\n verbose = TRUE,\r\n is_distance = FALSE)\r\n corMatrix_out <- as.data.frame(tsne_model$Y)\r\n cor_kmeans <- kmeans(corMatrix_out, centers = 5, iter.max = 10, nstart = 3)\r\n corMatrix_outclust <- as.factor(c(cor_kmeans$cluster[1:968], 6))\r\n corMatrix_names <- colnames(cor_out)\r\n ggplot(corMatrix_out, aes(x = V1, y = V2, color = corMatrix_outclust, shape = corMatrix_outclust)) + geom_point(size = 2.5) + geom_rug() + stat_ellipse(type = \"norm\") + ggtitle(\"T-SNE of Features\") + xlab(\"X\") + ylab(\"Y\") + labs(color = \"Cluster\", shape = \"Cluster\") + geom_text_repel(aes(x = V1, y = V2, label = corMatrix_names), size = 2.8)\r\n\r\nTo replicate the correlation matrix, use the *propagate* library with a small grid size (like 194 => 15 matrices), else you will literally explode R memory. It uses the ff package to store data in a file, and then you would just have to pick the correlation matrix out of created file variable.\r\n\r\nScript for reproducing the correlation matrix (need 16GB or more RAM depending on your R implementation, along with half an hour):\r\n\r\n library(propagate)\r\n for (i in 1:969) {\r\n cat(\"\\rStep \", i, \"\\n\", sep = \"\")\r\n train_data[!is.na(train_data[, i]), i] <- 1\r\n train_data[is.na(train_data[, i]), i] <- 0\r\n if (i %% 10) {gc(verbose = FALSE)}\r\n }\r\n gc(verbose = FALSE)\r\n \r\n cor_table <- bigcor(train_data, fun = \"cor\", size = 194, verbose = TRUE)\r\n\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/22909/expeditive-exploration-models-on-data/132183#post132183\r\n [2]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4721/TSNE_features_numeric.png\r\n [3]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4721/TSNE_features_numeric.png", "132591": "Laurae ... very nice work!!.... I have few questions :\r\n\r\n1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.\r\n2)How do you normally choose theta and perplexity in each case.\r\n3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)\r\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.\r\n\r\nThank you .", "132601": "[quote=Alpha;132591]\r\n\r\n@Laurae ... very nice work!!.... I have few questions :\r\n\r\n1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.\r\n2)How do you normally choose theta and perplexity in each case.\r\n3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)\r\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.\r\n\r\nThank you .\r\n\r\n[/quote]\r\n\r\n> 1) Difference between Rtsne done on correlation matrix and Rtsne done on actual data.\r\n\r\nIf you meant what is the difference between those two:\r\n\r\n* One is computable in less than one hour (from loading data to computing t-SNE) while the other would take forever (or never end)\r\n* We don't need 1M features just to look how original features (turned into observations) are spatially together in a 2D plot\r\n* We can compute directly using the MCC matrix which is the metric we are optimizing for initially (MCC = Phi coefficient on contingency table = Pearson correlation on binary values)\r\n\r\nIt is the reason why pre-processing/cleaning is more important to take care about than the machine learning model you use: you will spend more time doing an appropriate pre-processing/cleaning than applying a model (in a business scenario). A 1 day/month computing task was reduced to 1 hour or so, along with optimization for the target metric.\r\n\r\n> 2)How do you normally choose theta and perplexity in each case.\r\n\r\nTheta = 0 for exact t-SNE, ]0, 1] for Barnes-Hut t-SNE - it is up to you to choose because the computational complexity changes, but also the approximation can have a negative/beneficial result on gradient descent\r\n\r\nPerplexity = floor((ncol(my_data)-1)/3) to minimize the distance matrix reproduction error (maximum accepted perplexity threshold by t-SNE)\r\n\r\n> 3) If we would have taken actual data ,performed Rtsne on it and then plotted two dimentions spit by Rtsne and coloured by class(target) variable [performing No k-means] ,then was it possible to get similar plot showing clusters of classes(target)\r\n\r\nSimilar plot not at all. The plot I used was showing features, not observations. As there are features that are identical when using MCC on patterns, then there are many overlapping points. In case of observations, most of them are not unique and you will end up with ~1M points on a scatterplot. But then, you can color them by class, which is its typical use. In addition, you may do intra-class clusters. However, on 1M points it would take literally forever to perform a t-SNE. You will also potentially overplot, which will require a smart use of scatterplots.\r\n\r\n4) Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.\r\n\r\nWhat cluster? (your question is too large or I did not understand the question - still trying to give an appropriate answer in the mean time though)\r\n\r\nThere are many embedding techniques (dimensionality reduction), ranging from FA, PCA, ICA, MFA..., to SVD, NNMF, t-SNE, largeVis, Autoencoders... All depends on your knowledge on the data (such as presence of missing values, matrix input size, output features, sparsity, computational availabilities...). Labels have mostly nothing to do with embedding, the embedding quality can only be assessed using a supervised machine learning model to test its appropriateness (and the loss of information against the original training set).\r\n\r\nIf the target is a visualization, a technique itself is not alone: you'd rather know what is a Promax, Varimax, etc. when doing a Factor Analysis for instance to not discard a technique \"because the visualization was bad\". Same for choosing the right seed: typically 100 to 1000 tries to minimize the reproduction error / get better output is enough, and I doubt someone will want to run t-SNE 100 times on 2 million observations for a \"up to 99.99%+ reproduction error gain\" (which is largely possible on sparse matrices when the dimensionality increases).", "132603": "Laurae Thank you for your detailed reply.Please forgive me but I have some few confusions/questions:\r\n\r\n\r\n1)If I understood correctly,tsne tries to embed the multi dimensional data points into lower dimensions(say 2 or 3 D) in such a way that relative distance between data points remains same (atleast tries to error the error between relative distance of data points in higher dimensions and relative distance of data points in lower dimensions).Is is something to related to MCC matrix which you talked about.\r\n\r\n2) If I understood correctly about when to use which one\r\n a)we can visualize features on 2D plot if we perform Rtsne on correlation matrix ; \r\n\r\n (b)we can visualize observations on 2D plot if we perform Rtsne on actual data ;\r\n\r\n3)This is related to my last post question : \"Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.\"\r\n\r\nPlease consider the pasted image.Observations from class are merged inside other class obervation. I want to visualize this ( well separated by label colours ) using Rtsne.For this kind of data what kind of embedding you would recommend.\r\nI'm motivated by the Rtsne on MNIST digit classification data which is indeed high dimensional and Rtsne could well separate the classes.\r\n\r\n\r\nPlease correct me wherever I'm wrong .\r\n\r\nThank you once again for your help !!", "132605": "> 1) Is is something to related to MCC matrix which you talked about.\r\n\r\nt-SNE has nothing related to MCC directly. It uses the Kullback-Leibler divergence as the metric to optimize. The input used is the Pearson's correlation matrix on binary missing values, which is the same as Phi coefficient (which is the same as MCC).\r\n\r\n> 2) If I understood correctly about when to use which one\r\n\r\n> a)we can visualize features on 2D plot if we perform Rtsne on correlation matrix ; \r\n\r\nYes but that's not the only way to visualize features. You can also transpose the training set to invert observations/features' roles.\r\n\r\n> (b)we can visualize observations on 2D plot if we perform Rtsne on actual data ;\r\n\r\nYes.\r\n\r\n> 3)This is related to my last post question : \"Which 2/3-D embedding would you recommend for data where points of one class is spread through out cluster of other class.\"\r\n\r\n> Please consider the pasted image.Observations from class are merged inside other class obervation. I want to visualize this ( well separated by label colours ) using Rtsne.For this kind of data what kind of embedding you would recommend.\r\n\r\nWithout knowing what method was used nor what the colors are referring to, it is impossible to provide an appropriate answer. You need to remember that most embeddings (Factor Analysis, Multidimensional Scaling, t-SNE, largeVis...) are reliant on random initialization and an optimizer. One result on a randomized method is not enough to find whether the method is inappropriate. One can only make more tries to try getting an appropriate visualization, or change method if it is still not appropriate enough.\r\n\r\nAssuming you have a 70,000x784 matrix to deal with:\r\n\r\nIf the objective is visualization, and the computation available time was supposedly infinite:\r\n\r\n* All possible methods with lot of different seeds\r\n\r\nIf the objective is visualization, and the computation available time is slightly limited (like 1 day):\r\n\r\n* Truncated PCA if it is fast enough\r\n* Truncated ICA if it is fast enough\r\n* 10 to 20 tries on Barnes-Hut t-SNE approximation using different hyperparameters with the same seed with 2 dimension output, with fixed theta depending on how fast your computer singlethreading is\r\n* 10 to 20 tries on largeVis using different hyperparameters with the same seed and single-threaded with 2 dimension output\r\n\r\nIf the objective is visualization, and the computation available time is very limited (like 1 hour):\r\n\r\n* largeVis 1 try with 2 dimension output and \"good\" hyperparameters, as the original paper of largeVis described", "132606": "Laurae colours are class labels.Points in the plot are coloured by class labels(binary).I'm trying to use Rtsne on it and I want to know would that be appropriate for such inseparable data .Or are there any other similar technique which could be used to see separate clusters of classes on 2D", "142377": "Laurae, what's a good resource for learning about t-SNE?", "142385": "HueyKwik -> I've recently found [this][1]. Excellent resource for playing with and developing some intuition about how it works.\r\n\r\n\r\n [1]: http://distill.pub/2016/misread-tsne/", "142395": "[quote=Alchemist;142385]\r\n\r\n@HueyKwik -> I've recently found [this][1]. Excellent resource for playing with and developing some intuition about how it works.\r\n\r\n\r\n [1]: http://distill.pub/2016/misread-tsne/\r\n\r\n[/quote]\r\n\r\nThis site really helps understand some of the odd behavior of t-SNE. Thanks for sharing!" }, "source": "meta" }