1

I have a log-likelihood function with four parameters, out of which the variability of the likelihood with two parameters is extremely smooth. It suggests that the derivative is small for large ranges of these two parameters. Although I know there is a maximum, I have never achieved it through any optimization (local or global) I tried. If I use my own gradient functions, it takes a lot of time to converge, and when it converges, it converges to a wrong value. Are there some techniques by which I can handle such smooth log-likelihoods?

I am writing the likelihood with just these two parameters in the following equation.

$$ \log(p(Z|\Theta)) = -\sum_{i = 1}^N \left[ \log(\pi (F(f_i, \Theta))) + \frac{\sum_{l=1}^L Z_l(f_i)}{F(f_i, \Theta)} \right], $$

where $Z$ are the measurements and $F$ is a model of the expectation of the measurements that is a function of the parameters.

$$F(f_i, \Theta) = M\left[1+ \sum_{q = 1}^{N-1}\left(1 - \frac{q}{N}\right)\left[\exp(-j2\pi f_iq)G(q, \Theta) + \exp(j2\pi f_iq)G(-q, \Theta)\right]\right]$$

The function $G(q, \Theta)$ is difficult.

$G(q, \eta, \Lambda) = \exp(jC_1q) \int_{0}^{\infty} x^\eta \exp(-\Lambda x) \exp(-jC_2\exp(-c_3x)) dx, $ where $\Theta = [\eta, \Lambda]$. I also have used approximations of this integral.

EDIT

Based on the comment of @Cryo, I wanted to make it clear that, for now, I do not have a solid idea about the "acceptable" ranges of these parameters. So, the solutions I usually get are almost equally likely (because these solutions are on the plateau of the cost function). When I try reconstructing with $F(\hat{\Theta})$, I usually see they are a really good fit with the measurements $Z$.

Then, my correct question would be about how to make the optimization faster. If I consider all four parameters in my model, the gradient descent quasi-Newton-type algorithms take a lot of time to converge to a solution due to this plateau with these two parameters. This is frustrating because I want to make some Monte Carlo simulations on parameter sweeps. One such Monte Carlo run takes weeks to compute.

EDIT

I tried plotting the log-likelihood with the two problematic parameters by fixing the other two. The true values are $\eta = 2$, and $\Lambda = 3.6$. I can see there is clearly a maximum, but I have put the color scheme in a way that it is easier to see. You can see the other picture with a broader colorbar and notice how smooth the cost is.
enter image description here

enter image description here

CfourPiO
  • 275
  • 2
    If you only have two parameters, $\eta,,\Lambda$, why not plot log-likelihood for the different values they can take. With likelihood being smooth I guess you may be implying that there is a large range of values for these parameters under which the observations are almost equally likely. Perhaps you need some extra information, or extra prior to make the problem more constrained? If not, perhaps you need to be more specific about what 'converging to the wrong value' is and isn't? If my confidence interval is 0...100, true Max Likelihood is at 50 and I converge to 45, is this good enough? – Cryo Jan 05 '24 at 07:28
  • @Cryo Thanks for the suggestion. I need to know what ranges of these values are "acceptable " regarding my application. I still have a very vague idea about it. The "wrong" estimates are on the plateau but not exactly at the maxima. For example, if I take the estimated values and reconstruct $F(\hat{\Theta})$, to my eyes, they look like a very good fit with the observations $Z$ (as you correctly pointed out, these solutions are almost equally likely). Then, for now, my correct question would be how to converge faster because this optimization takes a lot of time due to the plateau I think. – CfourPiO Jan 05 '24 at 11:30
  • 1
    It's hard to give specific advice because there's not enough information given to duplicate the issues you're seeing. @Cryo 's suggestion of a contour plot would seem to be essential to look for areas where an iterative procedure might have trouble finding the maximum and where lack of numerical precision might be an issue. Using software that allow control over the numerical precision might be helpful (i.e., for complicated log likelihoods, the standard double precision might not be good enough). – JimB Jan 13 '24 at 20:20
  • @JimB I agree. I fixed the first two parameters and I tried plotting the log-likelihood. I figured that each time the log-likelihood function was computed, it was taking a lot of time. I have fixed that issue and I am able to see the log-likelihood with the two problematic parameters. I am editing the question with it. I think it will be probably informative for the community to comment on it by looking at the function. – CfourPiO Jan 13 '24 at 20:38
  • 1
    This looks essentially like the question addressed at https://stats.stackexchange.com/questions/7308. – whuber Jan 13 '24 at 20:54
  • 1
    @whuber Thank you. It gives me a nice perspective. I will study the log-likelihood for different true parameter combinations to see if there is a pattern. Further, I will try to do a Monte Carlo run with one setting to see if my estimation bias is acceptable. – CfourPiO Jan 15 '24 at 10:24

0 Answers0