Maximum Likelihood Estimator of rate parameter of the exponential distribution (MLE)

Question

I want to find the maximum likelihood estimator of the "rate parameter theta of the Exponential Distribution".

So i followed the following commands in R:

 x=rexp(500,rate=2)
 f <- function(x,theta){
        sum(-dexp(x,rate=theta,log=T))
 }
 optimize(f=f,x=x,interval=c(0,5))

I do not understand the logic - why do we consider maximum=FALSE(default) in the optimize function?

Why haven't we used maximum=TRUE to find the MLE?

Answer 1

The term inside your definition of f :- sum(-dexp(x,rate=theta,log=T)) is NOT the likelihood, but something else.

What is it that is being calculated?

When you consider what it is that is being optimized there, you will also understand why you're minimizing that function in order to maximize the likelihood.

To quote your own algebra, here's the likelihood:

$\cal{L}(\theta)=\prod_{i=1}^{n}\theta e^{-\theta x_i}=\theta^n e^{-\theta \sum_{i=1}^{n}x_i}$

dexp with log=TRUE doesn't return the density. Here's what the help says: log, log.p logical; if TRUE, probabilities p are given as log(p). ... that is when you say log=TRUE you get the log of the density.

The likelihood at $\theta$ will be the product of the densities, taken at each data point.

The log-likelihood is the sum of the log-densities, over the data points, evaluated at a given $\theta$.

That is, sum(dexp(x,rate=theta,log=T)) would be the log-likelihood function. We'd want to maximize that.

But we have sum(-dexp(x,rate=theta,log=T)) (don't ask me why they didn't write the obviously equivalent but presumably faster -sum(dexp(x,rate=theta,log=T))).

That is, the program is minimizing the negative log-likelihood, which is equivalent to maximizing the log-likelihood. Here's the result on calling f on theta values between 1 and 3:

By contrast, this is what the likelihood function looks like:

sum(dexp(x,rate=theta,log=T)) is calculating $θ^ne^{−θ∑^n_{i=1}x_i}$?

It's calculating the log of that quantity.

But here I see I have the minus sign in every program related to MLE in my lecture sheet.

Minimizing rather than maximizing is a convention. There's no particular need for it.

The R documentation is saying that in optim function par Initial values for the parameters to be optimized over. How can I select the initial values?

Would you please tell me how can I relate this program
fexp = function(theta, x){ prod(dexp(x,rate=(1/theta))) }
res3<-optimize(f=fexp,interval=c(0,50), maximum=T, x=x)
res3

with my above program that I have posted in the question?

Why here is the prod function being called?

Because the likelihood is a product.

And why here we have mentioned maximum=T?

Because it's computing the likelihood, which we want to maximize.

Edit: I notice another issue with the above code: it says rate = 1/theta. That implies that the theta there is not the rate parameter of your earlier mathematics and code, but is in fact a scale parameter. Watch out for that! Another thing to watch out for is that likelihood calculations often have underflow problems (and sometimes, overflow problems).