{
  "id": 21109,
  "title": "How is KDE applied here?",
  "url": "/competitions/facebook-v-predicting-check-ins/discussion/21109",
  "author_name": "",
  "post_date": "2016-05-20T20:17:12.710Z",
  "votes": null,
  "comment_count": 5,
  "views": 850,
  "content": "<p>Hi all, </p>\n\n<p>A couple guys on the forum are applying KDE (Kernel density estimation) to the x,y coordinates.</p>\n\n<p>I'm not a stats guy, so I read about KDE on wiki - seems like it's used for estimating density function for random variables.</p>\n\n<p>Could a stats guy perhaps briefly educate me a little bit in terms of how to apply KDE here? Are x, y coordinates the random variables? What do we do after I use KDE and figured out the density functions? Or what do we use KDE for?</p>\n\n<p>Also what is the most popular python package for doing KDE?</p>\n\n<p>Thanks in advance.</p>",
  "messages": [
    {
      "id": "120835",
      "postDate": "05/20/2016 20:17:12",
      "content": "<p>Hi all, </p>\n\n<p>A couple guys on the forum are applying KDE (Kernel density estimation) to the x,y coordinates.</p>\n\n<p>I'm not a stats guy, so I read about KDE on wiki - seems like it's used for estimating density function for random variables.</p>\n\n<p>Could a stats guy perhaps briefly educate me a little bit in terms of how to apply KDE here? Are x, y coordinates the random variables? What do we do after I use KDE and figured out the density functions? Or what do we use KDE for?</p>\n\n<p>Also what is the most popular python package for doing KDE?</p>\n\n<p>Thanks in advance.</p>",
      "rawMarkdown": "Hi all, \r\n\r\nA couple guys on the forum are applying KDE (Kernel density estimation) to the x,y coordinates.\r\n\r\nI'm not a stats guy, so I read about KDE on wiki - seems like it's used for estimating density function for random variables.\r\n\r\nCould a stats guy perhaps briefly educate me a little bit in terms of how to apply KDE here? Are x, y coordinates the random variables? What do we do after I use KDE and figured out the density functions? Or what do we use KDE for?\r\n\r\nAlso what is the most popular python package for doing KDE?\r\n\r\nThanks in advance.",
      "votes": null
    },
    {
      "id": "120850",
      "postDate": "05/20/2016 23:40:06",
      "content": "<p>The x and y are observations of a random variable X and Y. </p>\n\n<p>Random variables have a probability density function (PDF) associated with it. What KDE is doing is estimating this PDF using the observations of X,Y.</p>\n\n<p>Does this make more sense? Not sure about the package, still looking.</p>",
      "rawMarkdown": "The x and y are observations of a random variable X and Y. \r\n\r\nRandom variables have a probability density function (PDF) associated with it. What KDE is doing is estimating this PDF using the observations of X,Y.\r\n\r\nDoes this make more sense? Not sure about the package, still looking.",
      "votes": null
    },
    {
      "id": "120851",
      "postDate": "05/20/2016 23:50:10",
      "content": "<p>So, it's just a density plot, but possibly in &gt;1 dimension. Not familiar with a density plot? Think histogram, but with a smoothed line instead of binning and bars.</p>\n\n<p>As above, the line looks like your run of the mill probability density function. In R, try something like </p>\n\n<p><code>x &lt;- rnorm(1E3); par(mfrow=c(2,1)); hist(x); plot(density(x))</code></p>\n\n<p>A 2D is just like a &quot;heat map&quot;. It's the multivariate extension of your standard density plot.</p>\n\n<p>As you may have guessed, I use R, so I can't help with Python package much (well, not more than searching myself!).</p>",
      "rawMarkdown": "So, it's just a density plot, but possibly in >1 dimension. Not familiar with a density plot? Think histogram, but with a smoothed line instead of binning and bars.\r\n\r\nAs above, the line looks like your run of the mill probability density function. In R, try something like \r\n\r\n`x <- rnorm(1E3); par(mfrow=c(2,1)); hist(x); plot(density(x))`\r\n\r\nA 2D is just like a \"heat map\". It's the multivariate extension of your standard density plot.\r\n\r\nAs you may have guessed, I use R, so I can't help with Python package much (well, not more than searching myself!).",
      "votes": null
    },
    {
      "id": "120913",
      "postDate": "05/21/2016 16:45:29",
      "content": "<p>Thanks a lot @zafarali and @RyanBatt, appreciate it.</p>\n\n<p>in terms of python packages, I found this but haven't tried myself yet -\n<a href=\"https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/\">https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/</a></p>\n\n<p>Another question is, let's say with KDE we find out that the PDF for x or y coordinates follow some parametric distribution, such as mill or binomial or others - How does this information help us in terms of modeling?</p>",
      "rawMarkdown": "Thanks a lot @zafarali and @RyanBatt, appreciate it.\r\n\r\nin terms of python packages, I found this but haven't tried myself yet -\r\nhttps://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/\r\n\r\nAnother question is, let's say with KDE we find out that the PDF for x or y coordinates follow some parametric distribution, such as mill or binomial or others - How does this information help us in terms of modeling?",
      "votes": null
    },
    {
      "id": "120918",
      "postDate": "05/21/2016 17:37:02",
      "content": "<p>@jasoos Like a histogram, it won't tell you which distribution your data were drawn from ... necessarily. But, you can look at the kde/ hist and say &quot;o yeah, that looks like a normal&quot;, e.g.</p>\n\n<p>Really though, you don't care which distribution it is. What you want is what it gives you ... the density!</p>\n\n<p>Say you need to guess the ID of an animal. If I show you a density plot representing the distribution of sizes of individual animals in that species, and it has most of its mass between 90 and 100 ft long (yeah, I used ft :-/), you can rule out most life forms on earth right away (extant species, anyway). So you might then think &quot;hmm, blue whale is a decent guess&quot;. On the other hand, if I tell you an organism is 5&#956;m long, you can rule out several phyla, but might be left guessing among bacteria.</p>\n\n<p>(Maybe the bio reference is useless, but it's what I do so, that's my effort at making the statistic relatable)</p>\n\n<p>So it's useful because it can tell us where the places tend to be located in x and y. </p>",
      "rawMarkdown": "jasoos Like a histogram, it won't tell you which distribution your data were drawn from ... necessarily. But, you can look at the kde/ hist and say \"o yeah, that looks like a normal\", e.g.\r\n\r\nReally though, you don't care which distribution it is. What you want is what it gives you ... the density!\r\n\r\nSay you need to guess the ID of an animal. If I show you a density plot representing the distribution of sizes of individual animals in that species, and it has most of its mass between 90 and 100 ft long (yeah, I used ft :-/), you can rule out most life forms on earth right away (extant species, anyway). So you might then think \"hmm, blue whale is a decent guess\". On the other hand, if I tell you an organism is 5μm long, you can rule out several phyla, but might be left guessing among bacteria.\r\n\r\n(Maybe the bio reference is useless, but it's what I do so, that's my effort at making the statistic relatable)\r\n\r\nSo it's useful because it can tell us where the places tend to be located in x and y.",
      "votes": null
    },
    {
      "id": "120926",
      "postDate": "05/21/2016 18:37:21",
      "content": "<p>Thanks @RyanBatt - excellent and intuitive explanation!</p>",
      "rawMarkdown": "Thanks @RyanBatt - excellent and intuitive explanation!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 120850,
      "author_name": "zafarali",
      "author_url": "",
      "post_date": "05/20/2016 23:40:06",
      "content": "<p>The x and y are observations of a random variable X and Y. </p>\n\n<p>Random variables have a probability density function (PDF) associated with it. What KDE is doing is estimating this PDF using the observations of X,Y.</p>\n\n<p>Does this make more sense? Not sure about the package, still looking.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120851,
      "author_name": "zeraes",
      "author_url": "",
      "post_date": "05/20/2016 23:50:10",
      "content": "<p>So, it's just a density plot, but possibly in &gt;1 dimension. Not familiar with a density plot? Think histogram, but with a smoothed line instead of binning and bars.</p>\n\n<p>As above, the line looks like your run of the mill probability density function. In R, try something like </p>\n\n<p><code>x &lt;- rnorm(1E3); par(mfrow=c(2,1)); hist(x); plot(density(x))</code></p>\n\n<p>A 2D is just like a &quot;heat map&quot;. It's the multivariate extension of your standard density plot.</p>\n\n<p>As you may have guessed, I use R, so I can't help with Python package much (well, not more than searching myself!).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120913,
      "author_name": "howardxie",
      "author_url": "",
      "post_date": "05/21/2016 16:45:29",
      "content": "<p>Thanks a lot @zafarali and @RyanBatt, appreciate it.</p>\n\n<p>in terms of python packages, I found this but haven't tried myself yet -\n<a href=\"https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/\">https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/</a></p>\n\n<p>Another question is, let's say with KDE we find out that the PDF for x or y coordinates follow some parametric distribution, such as mill or binomial or others - How does this information help us in terms of modeling?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120918,
      "author_name": "zeraes",
      "author_url": "",
      "post_date": "05/21/2016 17:37:02",
      "content": "<p>@jasoos Like a histogram, it won't tell you which distribution your data were drawn from ... necessarily. But, you can look at the kde/ hist and say &quot;o yeah, that looks like a normal&quot;, e.g.</p>\n\n<p>Really though, you don't care which distribution it is. What you want is what it gives you ... the density!</p>\n\n<p>Say you need to guess the ID of an animal. If I show you a density plot representing the distribution of sizes of individual animals in that species, and it has most of its mass between 90 and 100 ft long (yeah, I used ft :-/), you can rule out most life forms on earth right away (extant species, anyway). So you might then think &quot;hmm, blue whale is a decent guess&quot;. On the other hand, if I tell you an organism is 5&#956;m long, you can rule out several phyla, but might be left guessing among bacteria.</p>\n\n<p>(Maybe the bio reference is useless, but it's what I do so, that's my effort at making the statistic relatable)</p>\n\n<p>So it's useful because it can tell us where the places tend to be located in x and y. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120926,
      "author_name": "howardxie",
      "author_url": "",
      "post_date": "05/21/2016 18:37:21",
      "content": "<p>Thanks @RyanBatt - excellent and intuitive explanation!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "120835": "Hi all, \r\n\r\nA couple guys on the forum are applying KDE (Kernel density estimation) to the x,y coordinates.\r\n\r\nI'm not a stats guy, so I read about KDE on wiki - seems like it's used for estimating density function for random variables.\r\n\r\nCould a stats guy perhaps briefly educate me a little bit in terms of how to apply KDE here? Are x, y coordinates the random variables? What do we do after I use KDE and figured out the density functions? Or what do we use KDE for?\r\n\r\nAlso what is the most popular python package for doing KDE?\r\n\r\nThanks in advance.",
    "120850": "The x and y are observations of a random variable X and Y. \r\n\r\nRandom variables have a probability density function (PDF) associated with it. What KDE is doing is estimating this PDF using the observations of X,Y.\r\n\r\nDoes this make more sense? Not sure about the package, still looking.",
    "120851": "So, it's just a density plot, but possibly in >1 dimension. Not familiar with a density plot? Think histogram, but with a smoothed line instead of binning and bars.\r\n\r\nAs above, the line looks like your run of the mill probability density function. In R, try something like \r\n\r\n`x <- rnorm(1E3); par(mfrow=c(2,1)); hist(x); plot(density(x))`\r\n\r\nA 2D is just like a \"heat map\". It's the multivariate extension of your standard density plot.\r\n\r\nAs you may have guessed, I use R, so I can't help with Python package much (well, not more than searching myself!).",
    "120913": "Thanks a lot @zafarali and @RyanBatt, appreciate it.\r\n\r\nin terms of python packages, I found this but haven't tried myself yet -\r\nhttps://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/\r\n\r\nAnother question is, let's say with KDE we find out that the PDF for x or y coordinates follow some parametric distribution, such as mill or binomial or others - How does this information help us in terms of modeling?",
    "120918": "jasoos Like a histogram, it won't tell you which distribution your data were drawn from ... necessarily. But, you can look at the kde/ hist and say \"o yeah, that looks like a normal\", e.g.\r\n\r\nReally though, you don't care which distribution it is. What you want is what it gives you ... the density!\r\n\r\nSay you need to guess the ID of an animal. If I show you a density plot representing the distribution of sizes of individual animals in that species, and it has most of its mass between 90 and 100 ft long (yeah, I used ft :-/), you can rule out most life forms on earth right away (extant species, anyway). So you might then think \"hmm, blue whale is a decent guess\". On the other hand, if I tell you an organism is 5μm long, you can rule out several phyla, but might be left guessing among bacteria.\r\n\r\n(Maybe the bio reference is useless, but it's what I do so, that's my effort at making the statistic relatable)\r\n\r\nSo it's useful because it can tell us where the places tend to be located in x and y.",
    "120926": "Thanks @RyanBatt - excellent and intuitive explanation!"
  },
  "source": "meta"
}