Issue with math: float calculation, ToStr()
Try this: ? 1.1* 9 console prints 9.9 But try this: ? str(1.1 * 9) console prints 9.900001 I assume this is some sort of floating point rounding error having to do with binary math and how c...
Forum Discussion
"Komag" wrote:
Try this:
? 1.1* 9
console prints 9.9
But try this:
? str(1.1 * 9)
console prints 9.900001
A couple of options:
x=1.1 * 9
' 1) use Fix to try to force 1 digit of precision. may or may not work?
? Str(Fix(x * 10)/10)
' 2) use printf-style conversion
? x.ToStr("%3.1f")
' Note that Str prefixes a blank on non-negative numbers, the above ToStr does not.
' if you want that, use x.ToStr("% 3.1f")
