117

I need to calculate the number of non-NaN elements in a numpy ndarray matrix. How would one efficiently do this in Python? Here is my simple code for achieving this: 

import numpy as np

def numberOfNonNans(data):
    count = 0
    for i in data:
        if not np.isnan(i):
            count += 1
    return count

Is there a built-in function for this in numpy? Efficiency is important because I'm doing Big Data analysis.

Thnx for any help!

Travis-Dudeson
  • 59
  • 1
  • 12
jjepsuomi
  • 3,971
  • 7
  • 41
  • 70

4 Answers4

212
np.count_nonzero(~np.isnan(data))

~ inverts the boolean matrix returned from np.isnan.

np.count_nonzero counts values that is not 0\false. .sum should give the same result. But maybe more clearly to use count_nonzero

Testing speed: 

In [23]: data = np.random.random((10000,10000))

In [24]: data[[np.random.random_integers(0,10000, 100)],:][:, [np.random.random_integers(0,99, 100)]] = np.nan

In [25]: %timeit data.size - np.count_nonzero(np.isnan(data))
1 loops, best of 3: 309 ms per loop

In [26]: %timeit np.count_nonzero(~np.isnan(data))
1 loops, best of 3: 345 ms per loop

In [27]: %timeit data.size - np.isnan(data).sum()
1 loops, best of 3: 339 ms per loop

data.size - np.count_nonzero(np.isnan(data)) seems to barely be the fastest here. other data might give different relative speed results.

M4rtini
  • 12,568
  • 3
  • 32
  • 41
18

Quick-to-write alterantive

Even though is not the fastest choice, if performance is not an issue you can use:

sum(~np.isnan(data)).

Performance:

In [7]: %timeit data.size - np.count_nonzero(np.isnan(data))
10 loops, best of 3: 67.5 ms per loop

In [8]: %timeit sum(~np.isnan(data))
10 loops, best of 3: 154 ms per loop

In [9]: %timeit np.sum(~np.isnan(data))
10 loops, best of 3: 140 ms per loop
Community
  • 1
  • 1
G M
  • 17,694
  • 10
  • 75
  • 78
  • This answer provides the sum which is not the same as counting the number of elements ... You should use `len` instead. – BenT Mar 28 '20 at 15:37
  • 4
    @BenT the sum of a bool array elements that meet a certain condition is the same providing the len of a subset array with the elements that meet a certain condition. Can you please clarify where this is wrong? – G M Mar 30 '20 at 09:26
  • 3
    My mistake I forgot a Boolean got return. – BenT Mar 30 '20 at 13:55
5

To determine if the array is sparse, it may help to get a proportion of nan values

np.isnan(ndarr).sum() / ndarr.size

If that proportion exceeds a threshold, then use a sparse array, e.g. - https://sparse.pydata.org/en/latest/

Darren Weber
  • 1,362
  • 17
  • 20
  • This is very helpful and interesting but was this answer intended for a different question..? :) – jtlz2 Mar 08 '22 at 19:09
4

An alternative, but a bit slower alternative is to do it over indexing.

np.isnan(data)[np.isnan(data) == False].size

In [30]: %timeit np.isnan(data)[np.isnan(data) == False].size
1 loops, best of 3: 498 ms per loop

The double use of np.isnan(data) and the == operator might be a bit overkill and so I posted the answer only for completeness.

Manuel
  • 2,011
  • 4
  • 17
  • 33