
squirreltown
Roku Guru
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2015
04:55 PM
Re: Str(value as Float) as String sometimes inaccurate!
Rek-thank you, must have been a squirrel !! when I was reading that part of the docs.
Mark - ignoring the "about" in your statement, does that mean that 1000000.5 is the same as 1000000?
So it's 7 total digits, regardless of which side of the decimal?
Mark - ignoring the "about" in your statement, does that mean that 1000000.5 is the same as 1000000?
So it's 7 total digits, regardless of which side of the decimal?
Kinetics Screensavers
belltown
Roku Guru
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2015
05:04 PM
Re: Str(value as Float) as String sometimes inaccurate!
"squirreltown" wrote:
Rek-thank you, must have been a squirrel !! when I was reading that part of the docs.
Mark - ignoring the "about" in your statement, does that mean that 1000000.5 is the same as 1000000?
So it's 7 total digits, regardless of which side of the decimal?
That's generally how floating-point numbers work; it doesn't matter where the decimal point is.
And "about" means "about". If you want an explanation (and a large squirrel-headache) for the underlying reason why it's "about" and not "exactly", then read these:
http://en.wikipedia.org/wiki/Floating_point
http://en.wikipedia.org/wiki/Single-precision_floating-point_format
- « Previous
-
- 1
- 2
- Next »