Roku Developer Program

Developers and content creators—a complete solution for growing an audience directly.
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
Level 10

Re: color convert

"Komag" wrote:
Is the reason those last two weren't properly "fixed" have to do with perfectly accurate binary representation being impossible for some decimals?

Mmmm, no.
Close but no cigar (in my best judgement, after carefully re-reading your guess).
Hint: meditate over the int() and fix() signatures (i.e. argument and return types)
0 Kudos
Highlighted
Level 9

Re: color convert

Well, don't int and fix take in a float, not a double float, so the extra precision is wasted?
0 Kudos
Highlighted
Level 10

Re: color convert

"Komag" wrote:
Well, don't int and fix take in a float, not a double float, so the extra precision is wasted?

Yes - the second time it was int() and fix() to be blamed - because of them taking a Float argument, when feeding them a Double it will get down-cast to whatever a Float can hold. And a Float has only 24 bits for the mantissa/significant (to store significant digits). Now 24 bits is exactly how many we needed after the 8-bit shift right - but the conversion Double->Float has also rounded up the number (which generally is the right thing to do), so the precision is 1 bit less (or is it 0.5 bit?), which here was enough to screw up the result

A possible fix for int/fix functions is to make them take Double argument. Or an unspecified type argument and then choose what to do internally.
0 Kudos
Highlighted
Level 9

Re: color convert

Maybe you could convert the double to a string, then truncate anything after a ".", then convert back
0 Kudos
Highlighted
Level 10

Re: color convert

"Komag" wrote:
Maybe you could convert the double to a string, then truncate anything after a ".", then convert back

Convert Double using what, Str(value as Float) as String ? :mrgreen:

I did find solution for the particular case, as seen in the code above. My side-note point was to highlight how delicate it is doing binary shifts with floating points... akin to how hedgehogs copulate*

(*) very, very carefully indeed.
0 Kudos