How do I get this math to calculate correctly

Question

I get this result:

8.01 - 8.0 #=> 0.009999999999999787

I want my answer to be 0.01. What can I do to fix this? How can I get this to happen? I guess this is float math. Is this due to floating point precision?

Rationals are good for this: `(8.01r - 8r).to_f` – steenslag Jan 30 '18 at 09:37 — steenslag, Jan 30 '18 at 09:37

score 5 · Accepted Answer · answered Jan 30 '18 at 02:51

5

You can use Ruby's arbitrary-precision decimal data type, BigDecimal:

BigDecimal.new("8.01") - BigDecimal.new("8") # => 0.1e-1

answered Jan 30 '18 at 02:51

Abe Voelker

27,824
14
78
97

How do I convert scientific notation back to a number? – Jwan622 Jan 30 '18 at 06:01
1

@Jwan622 you can use `to_s('F')` to convert the value to a string using "floating point notation". – Stefan Jan 30 '18 at 07:53
Thanks a lot man! – Jwan622 Jan 30 '18 at 15:58

How do I get this math to calculate correctly

1 Answers1