strange behavior: apparently a numpy nan in a pandas dataframe showing up as dtype('float64')

Question

I am working on a dataframe that has many custom functions, libraries, and calculations. After doing some critical calcs, I noticed some errors in calculations that should have returned a float:

To inspect one of the calculations, I do the following.

dFA.loc['20101120']['variable x']

which returns (in small caps)

nan

Then, to confirm that this thing is what looks like a weird (small caps) numpy.nan (True or False) I do:

dFA.loc['20101120']['variable x'] == np.nan

Which returns:

False

Then I do:

dFA.loc['20101120']['variable x'].dtype

Which returns:

dtype('float64')

Also:

dFA.loc['20101120']['variable x'] > 1000
False

Also:

dFA.loc['20101120']['variable x'] < 1000
False

score 3 · Accepted Answer · answered Jan 08 '16 at 00:14

3

dFA.loc['20101120']['variable x'] == np.nan

Oops. NaN is never equal to NaN.

np.isnan(dFA.loc['20101120']['variable x'])

answered Jan 08 '16 at 00:14

Ignacio Vazquez-Abrams

740,318
145
1,296
1,325

What do you mean NaN is never equal to NaN? Why is this nan showing up as float64? – Luis Miguel Jan 08 '16 at 00:20
2

I mean exactly what I said; performing an equality comparison between two NaNs will always result in false. NaNs are just floats with specific bit sequences. – Ignacio Vazquez-Abrams Jan 08 '16 at 00:21

score 2 · Answer 2 · edited May 23 '17 at 12:15

2

All comparisons with np.nan evaluate to False by definition.

>>> np.nan == np.nan
False
>>> np.nan <= 1
False
>>> np.nan > 1
False

np.nan is a float:

>>> np.nan.__class__
<type 'float'>

... just a very special one.

edited May 23 '17 at 12:15

Community

1
1

answered Jan 08 '16 at 00:20

timgeb

73,231
20
109
138

strange behavior: apparently a numpy nan in a pandas dataframe showing up as dtype('float64')

2 Answers2