Issue with math: float calculation, ToStr()
Try this: ? 1.1* 9 console prints 9.9 But try this: ? str(1.1 * 9) console prints 9.900001 I assume this is some sort of floating point rounding error having to do with binary math and how c...
Forum Discussion
For some reason I had the impression that x.ToStr() was only for integers, not floats. Because that actually prints correctly:
Thanks for the suggestions. I like the idea of always doing all math with integers, but at this point I don't want to go back and rework everything, so I'm going to go with "laundering" all my floats before converting to strings, with:
And then I'll use float.ToStr() for double measure (unless there are Roku models or firmware on which that won't work?)
? str(1.1*9)
9.900001
? (1.1*9).tostr()
9.9
Thanks for the suggestions. I like the idea of always doing all math with integers, but at this point I don't want to go back and rework everything, so I'm going to go with "laundering" all my floats before converting to strings, with:
FUNCTION washFloat(dirtyFloat)
Int10 = Cint(dirtyFloat *10)
cleanFloat = Int10 / 10
RETURN cleanFloat
END FUNCTION
And then I'll use float.ToStr() for double measure (unless there are Roku models or firmware on which that won't work?)
