Roku Developer Program

Developers and content creators—a complete solution for growing an audience directly.
cancel
Showing results for 
Search instead for 
Did you mean: 
Komag
Level 9

Issue with math: float calculation, ToStr()

Try this:
? 1.1* 9
console prints 9.9

But try this:
? str(1.1 * 9)
console prints 9.900001

I assume this is some sort of floating point rounding error having to do with binary math and how computers work, etc etc. But it's a problem for me, because in my game when the player is holding 9 spider abdomens in his or her inventory, on the list the weight is showing up at 9.900001, when everything else is just one decimal point, not six!

Any way to force a float to only be one decimal point accurate?

Is my only recourse to always check the final string and count characters after the "." ?
0 Kudos
7 Replies
Komag
Level 9

Re: Issue with math: float calculation, ToStr()

I just implemented this to brute force solve it:


FUNCTION checkStrFloat(Str AS STRING) AS STRING ' trig by getItemDscr(1), c2Inventory(1) ' Search for a . in the string, then check if the string is longer than one character past the . and if so, trim off the rest
StrLen = Str.Len()
FOR i = 1 TO StrLen
chr = Str.Mid(i -1, 1) ' String.Mid() is 0 based, not 1 based, so need to -1
IF chr = "." ' Ex: "9.900001", i will be 2
IF StrLen > i +1 ' If there is more than just one character after the decimal point
StrNEW = Left(Str, i+1) ' Left() is 1 based ' Ex: i +1 will be 3, thus first three characters of "9.900001" will be "9.9", one character past the "."
? "checkStrFloat() found long float Str, orig " Str ", shortened " StrNew
RETURN StrNEW
END IF
END IF
END FOR
RETURN Str
END FUNCTION
0 Kudos
belltown
Level 7

Re: Issue with math: float calculation, ToStr()

Due to the way floating point numbers are implemented, your number could be "9.899999", in which case your function will return "9.8" when it should return "9.9".

I'd do something like this (assuming your numbers are stored internally as Doubles, but you want to print them to one significant decimal place):


function convert(f as double) as string
 a = CInt(f * 10)
 n = a \ 10
 m = abs(a) mod 10
 return n.toStr() + "." + m.toStr()
end function
https://github.com/belltown/
0 Kudos
Roku Employee
Roku Employee

Re: Issue with math: float calculation, ToStr()

"Komag" wrote:
Try this:
? 1.1* 9
console prints 9.9

But try this:
? str(1.1 * 9)
console prints 9.900001


A couple of options:

x=1.1 * 9

' 1) use Fix to try to force 1 digit of precision.  may or may not work?
? Str(Fix(x * 10)/10)

' 2) use printf-style conversion
? x.ToStr("%3.1f")

' Note that Str prefixes a blank on non-negative numbers, the above ToStr does not.
' if you want that, use x.ToStr("% 3.1f")
0 Kudos
renojim
Level 8

Re: Issue with math: float calculation, ToStr()

My advice is to only use integer math unless there's a compelling reason not to (which there never is). If you only want one decimal point of precision then multiply everything by ten (9 spiders at 11 tenths per abdomen is 99 tenths). There's any number of ways to display 99 tenths as 9.9.

-JT
0 Kudos
squirreltown
Level 8

Re: Issue with math: float calculation, ToStr()

"renojim" wrote:
My advice is to only use integer math unless there's a compelling reason not to (which there never is).  

Animation might be an exception.
Kinetics Screensavers
0 Kudos
Komag
Level 9

Re: Issue with math: float calculation, ToStr()

For some reason I had the impression that x.ToStr() was only for integers, not floats. Because that actually prints correctly:
? str(1.1*9)
 9.900001
? (1.1*9).tostr()
9.9

Thanks for the suggestions. I like the idea of always doing all math with integers, but at this point I don't want to go back and rework everything, so I'm going to go with "laundering" all my floats before converting to strings, with:



FUNCTION washFloat(dirtyFloat)
Int10 = Cint(dirtyFloat *10)
cleanFloat = Int10 / 10
RETURN cleanFloat
END FUNCTION

And then I'll use float.ToStr() for double measure (unless there are Roku models or firmware on which that won't work?)
0 Kudos
Roku Employee
Roku Employee

Re: Issue with math: float calculation, ToStr()

"Komag" wrote:

And then I'll use float.ToStr() for double measure (unless there are Roku models or firmware on which that won't work?)


Default ToStr() on intrinsic types (numeric, Boolean, and string) goes back to firmware 7.0.
ToStr and string Format with printf format specifiers goes back to firmware 7.6.
0 Kudos