6

When I graph a density plot in R, and all the numbers are slightly greater than 0, I get essentially a vertical line at x = 0. But when all the numbers are exactly equal to 0, I get some sort of bell curve. Why is that? It seems counterintuitive.

The command I used to plot the curves was

for (i in 0:60) {
    cur_data = subset(data, time == i)
    p <- ggplot(cur_data, aes(x=error)) +
                geom_density() +
                theme_bw() +
                xlab(paste("Error distribution (minute ", i, ")", sep="")) +
                xlim(0, 1)
    ggsave(...)
}

At i = 60, cur_data should be entirely populated by the values 0.0.

Density plot, entries slightly greater than 0

Density plot, entries all 0

(Originally posted on Stack Overflow; was told to post here.)

gung - Reinstate Monica
  • 145,122
Jessica
  • 2,091
  • 1
    Hint: how is the density curve computed? What math is R doing to create that plot? – Sycorax Nov 10 '14 at 21:27
  • 1
    I have no idea. If I were to guess, R first finds the cdf, and then takes the (approximate) derivative at each point to find the pdf. Except in this case the derivative is infinity, so you'd think it would be a vertical line. – Jessica Nov 10 '14 at 21:34
  • 1
    Actually since there is a finite number of points, the cdf is discrete and the derivative would actually be infinity at every point. So to approximate the derivative at any given point I'd take the interval (x, x + dt) for every point x and compute the change in y over that interval. But in this case it would be a spike at 0 because this approximate derivative would always be 0 at any point other than x = 0. So I guess that brings me back to where I was before. – Jessica Nov 10 '14 at 21:36
  • 1
    If you're getting all-zero data, then you don't even have a continuous density (whatever the rest of the distribution does that you didn't observe, a lot of the probability is exactly at zero), so it doesn't necessarily make sense to attempt to compute a derivative. Certainly if you have no data anywhere else (even if the population distribution were continuous elsewhere), the exercise on such a sample is utterly futile. You're entirely reliant on assumptions about its behavior where you have no data ... and your estimate should reflect your assumptions, not the specifics of the kernel. – Glen_b Nov 10 '14 at 21:42
  • 1
    Wait, if the data is all zero, shouldn't the cdf be a step function? So if we were to approximate the derivative at 0 by finding $\Delta y/\Delta x$ in some small interval around 0, wouldn't it just be $1 / (\Delta x)$? – Jessica Nov 10 '14 at 21:54

1 Answers1

4

You should explain what the intuition is that you have that the behavior runs counter to - it would make it easier to focus the explanation to address that.

A kernel density estimate is the convolution of the sample probability function ($n$ point masses of size $\frac{1}{n}$) and the kernel function (itself, by default, a normal density).

The result in the default case is a mixture of normal (Gaussian) densities, each with center at the data values, each with standard deviation $h$ (the bandwidth of the kernel), and weight $\frac{1}{n}$.

When all the data are coincident, the resulting mixture density is a sum of $n$ weighted densities, all with the same mean and standard deviation ... which is just the kernel itself, centered at that data value.

The difference in behavior you see might relate to the trim argument in ggplot2::stat_density. When the range of values is exactly zero, my guess is that it's setting trim to FALSE (or at least something other than TRUE), but when it's even a little larger than 0 it's at the default (TRUE). You'd need to look into the source to double check, but that would be my guess. If that's what's happening, you should be able to modify that behavior.

Glen_b
  • 282,281
  • Thanks, I didn't know that stuff about the kernel density estimate. I thought it was a weird result because the numbers are only changing very slightly, so you'd think that would produce a very small change in the density function. (Actually I still think it's kind of a weird result because presumably the weighted sum of Gaussian densities is a continuous function of their means, so if the means change slightly, why should the density change so much?) – Jessica Nov 10 '14 at 21:48
  • 1
    Ah. Sorry, I see what you're concerned about. That may be related to the trim argument in ggplot2::stat_density. When the range is exactly zero, my guess is that it's setting trim to FALSE (or at least something other than TRUE), but when it's even a little larger than 0 it's setting it to TRUE (the default). You'd need to look into the source to double check, but that would be my guess. You should be able to modify that behavior. – Glen_b Nov 10 '14 at 22:09