Issue with math: float calculation, ToStr()
Try this: ? 1.1* 9 console prints 9.9 But try this: ? str(1.1 * 9) console prints 9.900001 I assume this is some sort of floating point rounding error having to do with binary math and how c...
Forum Discussion
I just implemented this to brute force solve it:
FUNCTION checkStrFloat(Str AS STRING) AS STRING ' trig by getItemDscr(1), c2Inventory(1) ' Search for a . in the string, then check if the string is longer than one character past the . and if so, trim off the rest
StrLen = Str.Len()
FOR i = 1 TO StrLen
chr = Str.Mid(i -1, 1) ' String.Mid() is 0 based, not 1 based, so need to -1
IF chr = "." ' Ex: "9.900001", i will be 2
IF StrLen > i +1 ' If there is more than just one character after the decimal point
StrNEW = Left(Str, i+1) ' Left() is 1 based ' Ex: i +1 will be 3, thus first three characters of "9.900001" will be "9.9", one character past the "."
? "checkStrFloat() found long float Str, orig " Str ", shortened " StrNew
RETURN StrNEW
END IF
END IF
END FOR
RETURN Str
END FUNCTION
