Skip to main content
Announcements
Qlik Introduces a New Era of Visualization! READ ALL ABOUT IT
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Weird ROUND()

I want to round number:

=num(round(39965.666666666999, 0.1), '#,###.###############')


My expectation is to get 39965.7, but the reality is 39,965.700000000004000

Is it bug or feature?

OC: Win Server 2008 EE

QV: 9.00.7440.8 SR3 x64

Does anybody have any idea how to get the proper result for the round?

11 Replies
johnw
Champion III
Champion III

Calculate your calendar DateHour the same way, and they'll match.

In reality:

1 * 3 / 3 = 1
1 / 3 * 3 = 1

In a computer:

1 * 3 / 3 = 1.00000000
1 / 3 * 3 = 0.99999999

That's exactly what you're doing. You're calculating the "same" value in several different ways, and thinking that somehow they're going to match. They're not going to match. You're using a computer. Computers have limited precision regardless of what base they're using, be that binary or decimal or hexadecimal or whatever. Any fraction that can only be represented as a repeating decimal in your chosen base is not going to have an exact representation in a computer.

To solve your problem, you have two options that I can think of. First, you can stick with integer arithmetic for 100% reliability. You say you can't do that, but you can. Seconds are integers. That's what your source data was in. You can preserve that, and only calculate things like timestamps on the fly using the 100% accurate seconds. That's very likely overkill for your situation, though. If your problem is simply that you have multiple storage values for the same fundamental time, you just have to calculate all your times the same way. If you're going to have a Timestamp field, and a DateHour field, and another DateHour field in a calendar, calculate them the same way, and the "incorrect" values will all match, and will therefore not cause you this problem, or any other practical problems that I've thought of.

See attached.

Not applicable
Author

It works.

Thank you very much