24

I would like to know how to transform negative values to log(), since I have heteroskedastic data. I've read that log(x+1) solves the problem but this doesn't work with my data and I continue getting NaNs as result.

For e.g. I get this warning message (I didn't put my complete database because I think one of my negative values is enough to show the problem):

> log(-1.27+1)
[1] NaN
Warning message:
In log(-1.27 + 1) : NaNs produced
>

UPDATE:

Here is an histogram of my data. I'm working with palaeontological time series of chemical measurements. If the difference between (for e.g.) variables like Ca and Zn is too big, I need some type of data standardization, which is why I'm testing the log() function.

enter image description here

This is my raw data

mkt
  • 18,245
  • 11
  • 73
  • 172
Darwin PC
  • 541
  • 2
    The logarithm is only defined for positive numbers, and is usually used as a statistical transformation on positive data so that a model will preserve this positiveness. The log(x+1) transformation will is only defined for x > -1, as then x + 1 is positive. It'd be good to know your reason for wanting to log transform your data. – Matthew Drury Jun 04 '15 at 04:58
  • It is because Into my database I have some extreme values, and when I plot a histogram almost all data bars are grouped at left. I would like to have a better data distribution to plot it in a heatmap. – Darwin PC Jun 04 '15 at 05:06
  • What variable are you measuring? It's presumably not a count or frequency, but perhaps it is temperature? In which case converting to Kelvin will ensure it is non negative. Or is it difference data (e.g. Stock price change from previous day)? – tristan Jun 04 '15 at 06:02
  • 4
    Tell us more about the data, including the range, mean, frequencies of negative, zero and positive values. It could be that a generalized linear model with log link makes most sense for the data so long as it is reasonable to think that the mean response is positive. It could be that you should not be transforming at all. – Nick Cox Jun 04 '15 at 06:34
  • 1
    In practice getting the right functional form for the relationship is much more important than getting a convenient error structure. Heteroscedasticity can be dealt with by changing the estimation procedure, not just by transformation. Also cube roots can cope with positive, zero and negative numbers. – Nick Cox Jun 04 '15 at 06:39
  • For clarity I updated my question with a histogram of my data. – Darwin PC Jun 04 '15 at 06:56
  • 10
    Thanks for adding details. For such data 0 has a meaning (equality!) that should be respected, indeed preserved. For that and other reasons I would use cube roots. In practice, you will need some variation on sign(x) * (abs(x))^(1/3), the details depending on software syntax. For more on cube roots see e.g. http://www.stata-journal.com/sjpdf.html?articlenum=st0223 (see esp. pp.152-3).We used cube roots to help visualization of a response variable that can be positive and negative in http://www.nature.com/nature/journal/v500/n7464/full/nature12382.html?WT.ec_id=NATURE-20130829 – Nick Cox Jun 04 '15 at 07:54
  • Note that "Kg" should strictly be "kg" in your graph. – Nick Cox Jun 04 '15 at 07:54
  • 1
    Your use of R is secondary here. The same issues arise regardless of the software being used and software-specific questions would be off-topic here in any case. I edited the title accordingly. – Nick Cox Jun 04 '15 at 07:58
  • 9
    Why aren't you transforming the original variables instead of the differences? – whuber Jun 04 '15 at 08:23
  • 2
    @whuber has a penetrating comment as usual. It could be that $\log (x / y) = \log x - \log y$ makes more sense, i.e. the skewness in the original variables would be much improved and $x = y$ maps simply to $\log x - \log y = 0$. Clearly $x$ and $y$ must still be positive. – Nick Cox Jun 04 '15 at 08:33
  • 1
    This question merits a better title. It seems its essence is in how to transform data to remove heteroskedasticity, not just how to take a logarithm of a negative number (which does not make sense, at least in simple mathematics). – Richard Hardy Jun 04 '15 at 14:12
  • Your raw data consists a variable indicating year and 138 variables with anonymous names var001-var138 (itself poor data management practice, unless you are being coy on purpose). I think to get us interested, and to keep the discussion focused, to need at an absolute minimum to nominate one of two variables which concern you. – Nick Cox Jun 05 '15 at 07:29
  • My data have chemical and climate variables, and I put intentionally anonymous names. At the moment I want to prepare this data before to do PCA and plot this with heatmap2( ) . That is why first I'm trying to test the logarithmic transformation. – Darwin PC Jun 05 '15 at 07:56
  • 2
    You need to motivate people, who do this for free and only because they find it interesting! If you tell me that the graph above is based on var042 minus var065, or whatever, then I and everybody else have something to discuss. Otherwise I can't see what you are expecting or asking us to do beyond what we have suggested. Nor are you responding to any of the specific suggestions already made. It's already established that log() is pointless unless for positive arguments. – Nick Cox Jun 05 '15 at 09:34
  • 1
    Unfortunately, these are not the raw data: they are the differences. The raw data would be the original chemical measurements. Your analysis will be more successful starting with those. – whuber Jun 05 '15 at 15:53
  • Dear @NickCox thanks for your suggestions, apparently my problem was simple maths, I just changed the 1 from my initial formula log(x+1) to another constant (the minimum value) as explained Jeromy Anglim here, it is log(x+C). Now the logarithmic transformation works great with my data, of course it is transforming the negative to positive first. Regarding to transform data to remove heteroskedasticity I have to formulate a new question. – Darwin PC Jun 06 '15 at 01:47
  • 5
    You solved the mathematical problem. @whuber's suggestion or cube roots would still, I think, be easier to work with, especially if the constant is purely empirical or varies between variables. A good rule for choice of transformations is only to use transformations that would work for similar data you can imagine. Thus $\log(x + 4)$ "works" for $x > -4$ but would fail if your next batch was bounded by $-5$.. – Nick Cox Jun 06 '15 at 07:00
  • 1
    The fact that $\log(x)$ needs to be $\log(x+c)$ and it is very difficult to (1) solve for $c$ and (2) account for uncertainty in $c$ in the final analysis makes this approach very questionable. I'd use a transformation-invariant method such as a semiparametric ordinal model for continuous data: https://hbiostat.org/rmsc/cony.html – Frank Harrell Sep 23 '22 at 12:13

2 Answers2

15

Since logarithm is only defined for positive numbers, you can't take the logarithm of negative values. However, if you are aiming at obtaining a better distribution for your data, you can apply the following transformation.

Suppose you have skewed negative data:

x <- rlnorm(n = 1e2, meanlog = 0, sdlog = 1)
x <- x - 5
plot(density(x))

then you can apply a first transformation to make your data lie in $(-1,1)$:

z <- (x - min(x)) / (max(x) - min(x)) * 2 - 1
z <- z[-min(z)]
z <- z[-max(z)]
min(z); max(z)

and finally apply the inverse hyperbolic tangent:

t <- atanh(z)
plot(density(t))

Now, your data look approximately normally distributed. This is also called Fisher transformation.

stochazesthai
  • 5,106
  • 13
    You solved the immediate mathematical problem. But I don't think most likely consumers of statistical results would find it easy to think about $\text{atanh}[(x - \min(x)) / (\max(x) - \min(x))]$ as a response scale and in modelling you would need to think what error structure makes sense.The scale would be sensitive to the empirical minimum and maximum. – Nick Cox Jun 04 '15 at 06:38
  • 3
    @NickCox You are absolutely right. Maybe if the OP add more details about his problem, we could figure out an alternative solution! – stochazesthai Jun 04 '15 at 06:43
  • The inner argument in my first comment is not what is being transformed, but the spirit of my comment is I think unaffected. – Nick Cox Jun 04 '15 at 08:35
  • Dear @stochazesthai thanks for your detailed explanation, but I can't apply your code to my data. I updated my question with a link of my raw data at the end. – Darwin PC Jun 05 '15 at 06:18
  • 1
    The statements z <- z[-max(z)] and z <- z[-min(z)] inappropriately shrink z down to a single value. Also the general function atanh(((x - min(x)) / (max(x) - min(x)))) produces Inf for the minimum and maximum values of x. – Max Ghenis Aug 07 '18 at 19:39
  • 4
    Despite the large number of upvotes, this seems to me a poor solution, as (1) it is going to be hard, even impossible, to compare results with other datasets with different minimum and maximum (2) the transformation therefore does not respect sign of the original, which is both possible and preferable (3) atanh() is not widely familiar (or intuitive, as people are wont to say) Use sign(y) log(1 + y) or the cube root or asinh(). Note that Fisher transformation is a respected, respectable solution if the distribution is already bounded by [-1, 1], as is true of sample correlations. – Nick Cox Sep 27 '22 at 09:57
10

This has been covered in detail in the comments, but there still isn't an answer stating this. So for the benefit of future readers:

Please DON'T fiddle with your negative values (especially differences!) so that you can apply a log transformation. Strategies such as adding constants introduce bias; this can work out alright if done with care but can also lead to completely incorrect and uninterpretable results. See this thread for more detail about this: Interpreting log-log regression with log(1+x) as independent variable.

Instead, use a transformation that handles negative values naturally. The cube root transformation is an obvious candidate here, as it is simple, easier to interpret than complex options such as rescaled inverse hyperbolic tangents, and has the advantage that the interesting and special case of 0 is preserved.

As Nick Cox mentions, you will need to implement this carefully yourself because just raising your values to the power of 1/3 will not work in R. Instead, you'll need something like sign(x) * (abs(x))^(1/3).

More generally, it's best to think carefully about the nature of the data and the goal of the analysis while deciding on a transformation, and not letting canned procedures guide how you handle the data.

mkt
  • 18,245
  • 11
  • 73
  • 172
  • 1
    this is a perfect solution, thank you. I see people add a +1 ALL.the.time and its been working for me so far until it didn't. I was running a linear model with a covariate and wanted to visual the data and this was the only transformation that worked under all conditions including ones with big outliers. – Ahdee Sep 22 '22 at 22:22