The difference between comparison to np.nan and isnull()

Question

I supposed that

data[data.agefm.isnull()]

and

data[data.agefm == numpy.nan]

are equivalent. But no, the first truly returns rows where agefm is NaN, but the second returns an empty DataFrame. I thank that omitted values are always equal to np.nan, but it seems wrong.

agefm column has float64 type:

(Pdb) data.agefm.describe()
count    2079.000000
mean       20.686388
std         5.002383
min        10.000000
25%        17.000000
50%        20.000000
75%        23.000000
max        46.000000
Name: agefm, dtype: float64

Could you explain me please, what does data[data.agefm == np.nan] mean exactly?

They are not the same: http://stackoverflow.com/questions/20320022/why-in-numpy-nan-nan-is-false-while-nan-in-nan-is-true — ayhan, Dec 27 '16 at 09:40
@Divakar So, if I should, is this a gap in *pandas* or I do a conceptual mistake? — sergzach, Dec 27 '16 at 09:44
Not really familiar with pandas isnull method, but we use np.isnan to detect NaNs with NumPy arrays. — Divakar, Dec 27 '16 at 09:45

piRSquared · Accepted Answer · 2016-12-27T10:11:48.137

37

np.nan is not comparable to np.nan... directly.

np.nan == np.nan

False

While

np.isnan(np.nan)

True

Could also do

pd.isnull(np.nan)

True

examples
Filters nothing because nothing is equal to np.nan

s = pd.Series([1., np.nan, 2.])
s[s != np.nan]

0    1.0
1    NaN
2    2.0
dtype: float64

Filters out the null

s = pd.Series([1., np.nan, 2.])
s[s.notnull()]

0    1.0
2    2.0
dtype: float64

Use odd comparison behavior to get what we want anyway. If np.nan != np.nan is True then

s = pd.Series([1., np.nan, 2.])
s[s == s]

0    1.0
2    2.0
dtype: float64

Just dropna

s = pd.Series([1., np.nan, 2.])
s.dropna()

0    1.0
2    2.0
dtype: float64

edited Dec 27 '16 at 10:11

answered Dec 27 '16 at 09:48

piRSquared

265,629
48
427
571

1

Are `np.isnan` and `pd.isnull` equivalent? – sergzach Dec 27 '16 at 09:52
@sergzach no, but close. I believe `pd.isnull` checks a few more things for null. – piRSquared Dec 27 '16 at 09:53
4

@sergzach Yes for `NaN`, not for `None`. You cannot use `np.isnan` for object arrays. – ayhan Dec 27 '16 at 09:53
1

Your example is very good but may be not to use `s[s == s]` in practice? It seems ambiguous. – sergzach Dec 27 '16 at 10:09

The difference between comparison to np.nan and isnull()

1 Answers1

Linked