Why does the answer to this:
5.15-2.55-2.6
not come out to 0?
try it in your console and see what you get?
I'm asking for a friend.
-John
5.15-2.55-2.6
Moderators: Susan Smith, admin, Gabriel
Re: 5.15-2.55-2.6
This is caused by the difference between decimal fraction and binary fraction values. IE-the binary values of the decimal numbers you picked don't EXACTLY equal the original decimal value. This is only a problem with some fractions. Whole numbers are exactly equal between decimal and binary. This is also why sometimes an apparent zero value will print as -0.00 or .00CR.
Gordon jump in if I have gotten it wrong.
Gordon jump in if I have gotten it wrong.
Re: 5.15-2.55-2.6
Very interesting.
I do note that if its printed in any kind of formatted way, or if its tested in an if statement, then it still works correctly.
Gabriel
I do note that if its printed in any kind of formatted way, or if its tested in an if statement, then it still works correctly.
Gabriel
Re: 5.15-2.55-2.6
Yes John. You stated it correctly. All programming languages that work with floating point numbers have the same problem. By using scientific notation internally the binary fractions used in floating point math do not precisely match decimal fractions. Floating point arithmetic necessarily and specifically accepts the notion that results will be infinitesimally imprecise.
For that reason when adding a long column of numbers it is a good idea to round the result every so often. This will erase any accumulating low level inaccuracy.
In BR the number of decimal places where rounding occurs during output formatting and comparisons, defaults to 6 decimal places. However the RD setting can be changed. See: http://brwiki2.brulescorp.com/index.php?title=RD
For that reason when adding a long column of numbers it is a good idea to round the result every so often. This will erase any accumulating low level inaccuracy.
In BR the number of decimal places where rounding occurs during output formatting and comparisons, defaults to 6 decimal places. However the RD setting can be changed. See: http://brwiki2.brulescorp.com/index.php?title=RD
Re: 5.15-2.55-2.6
You can better see the results using formated output:
Returns: 0.000000000000000444
You can see that there are 15 0's after the decimal place.
Loop 1 million times:
Returns: 0.0000000004440892098500630000
Not surprisingly, moves the decimal 6 places
Add the Round Command (to 15 digits):
Returns 0
Code: Select all
pr using "form n 30.28":5.15-2.55-2.6
You can see that there are 15 0's after the decimal place.
Loop 1 million times:
Code: Select all
00001 PRINT Newpage
00010 FOR Loop=1 TO 1000000
00020 LET A+=(5.15-2.55-2.6)
00030 NEXT Loop
00040 PRINT USING "FORM N 30.28": A
Not surprisingly, moves the decimal 6 places
Add the Round Command (to 15 digits):
Code: Select all
00001 PRINT Newpage
00010 FOR Loop=1 TO 1000000
00020 LET A+=Round(5.15-2.55-2.6,15)
00030 NEXT Loop
00040 PRINT USING "FORM N 30.28": A