belltown
Roku Guru
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2012
01:15 PM
Another JSON Parser [JSONDecoder]
I've written a new JSON parser. Here it is: https://github.com/belltown/Roku/blob/master/JSONDecoder/source/JSONDecoder.brs
There are already a couple of other JSON parsers out there: one by hoffmcs, http://forums.roku.com/viewtopic.php?f=34&t=30688&p=186934#p186934, and rdJSONParser in librokudev, see http://forums.roku.com/viewtopic.php?f=34&t=32208&p=200787 and https://github.com/rokudev/librokudev/blob/master/source/librokudev/source/rokudev_src/rdJSON.brs.
The hoffmcs parser is very slow, although does a fairly good job of parsing most JSON, although strings containing any UNICODE characters are treated as invalid. The librokudev parser is faster than the hoffmcs parser for smaller files, but still rather slow for larger files, and does not parse all valid JSON; for example, it fails to parse JSON text containing names that have characters such as "$", "." and "-", and it won't parse backslash-escaped characters in strings (except for the " character), and UNICODE characters are not parsed.
The new JSON parser should parse any valid JSON and is faster than the other parsers for larger JSON responses. It will also convert UNICODE characters that have ASCII equivalents.
The parser has no external dependencies. Simply copy the JSONDecoder.brs file into your source directory and use as follows:
A single JSONDecoder object may be used for multiple calls to decodeJSON ().
Unicode characters less than \u0080 (128) are converted to ASCII.
Unicode characters of \u0080 (128) and higher are converted to "." characters.
Characters with ASCII values of 128 and higher are converted to "." characters.
The JSON 'null' value is converted to 'invalid'.
Numbers are converted to integers unless they are too large, or contain a fraction or exponent, in which case a float is used.
For performance reasons, all valid JSON strings should be handled correctly, but not all invalid JSON will be detected.
If the input string is not valid JSON, the code should not crash, although it is not defined what will be returned.
Rudimentary error diagnostics are provided by calling errorString () if decodeJSON () returns Invalid.
I ran some tests on my Roku 1 to compare execution times for the different parsers using the Twitter API, changing the value of rpp to return different numbers of results:
http://search.twitter.com/search.json?q=roku&rpp=1&include_entities=true&result_type=mixed
rpp=1 => 2,304 bytes. JSONDecoder: 0.3 secs, rdJSONParser: 0.05 secs, hoffmcs parser: 1.0 secs
rpp=10 => 11,331 bytes. JSONDecoder: 1.7 secs, rdJSONParser: 0.4 secs, hoffmcs parser: 7.1 secs
rpp=50 => 47,000 bytes. JSONDecoder: 7.5 secs, rdJSONParser: 4.9 secs, hoffmcs parser: 57.8 secs
rpp=78 => 80,002 bytes. JSONDecoder: 12.5 secs, rdJSONParser: 12.6 secs, hoffmcs parser: 140.0 secs
rpp=100 => 101,195 bytes. JSONDecoder: 15.9 secs, rdJSONParser: 19.8 secs, hoffmcs parser: 212.3 secs
Obviously, XML responses are parsed more efficiently, since Roku has native support for XML. However, not all APIs provide XML output. For smaller JSON queries (less than 75K bytes), rdJSONParser is faster although may not necessarily parse all JSON responses. For larger queries, JSONDecoder seems to work faster and should parse all valid JSON responses. The hoffmcs parser seems slower in all cases.
Send a PM to belltown if you find any problems.
There are already a couple of other JSON parsers out there: one by hoffmcs, http://forums.roku.com/viewtopic.php?f=34&t=30688&p=186934#p186934, and rdJSONParser in librokudev, see http://forums.roku.com/viewtopic.php?f=34&t=32208&p=200787 and https://github.com/rokudev/librokudev/blob/master/source/librokudev/source/rokudev_src/rdJSON.brs.
The hoffmcs parser is very slow, although does a fairly good job of parsing most JSON, although strings containing any UNICODE characters are treated as invalid. The librokudev parser is faster than the hoffmcs parser for smaller files, but still rather slow for larger files, and does not parse all valid JSON; for example, it fails to parse JSON text containing names that have characters such as "$", "." and "-", and it won't parse backslash-escaped characters in strings (except for the " character), and UNICODE characters are not parsed.
The new JSON parser should parse any valid JSON and is faster than the other parsers for larger JSON responses. It will also convert UNICODE characters that have ASCII equivalents.
The parser has no external dependencies. Simply copy the JSONDecoder.brs file into your source directory and use as follows:
decoderObject = JSONDecoder ()
json = decoderObject.decodeJSON ("<json formatted string>")
if json <> Invalid
' Handle the json object/array ...
else
print decoderObject.errorString ()
endif
A single JSONDecoder object may be used for multiple calls to decodeJSON ().
Unicode characters less than \u0080 (128) are converted to ASCII.
Unicode characters of \u0080 (128) and higher are converted to "." characters.
Characters with ASCII values of 128 and higher are converted to "." characters.
The JSON 'null' value is converted to 'invalid'.
Numbers are converted to integers unless they are too large, or contain a fraction or exponent, in which case a float is used.
For performance reasons, all valid JSON strings should be handled correctly, but not all invalid JSON will be detected.
If the input string is not valid JSON, the code should not crash, although it is not defined what will be returned.
Rudimentary error diagnostics are provided by calling errorString () if decodeJSON () returns Invalid.
I ran some tests on my Roku 1 to compare execution times for the different parsers using the Twitter API, changing the value of rpp to return different numbers of results:
http://search.twitter.com/search.json?q=roku&rpp=1&include_entities=true&result_type=mixed
rpp=1 => 2,304 bytes. JSONDecoder: 0.3 secs, rdJSONParser: 0.05 secs, hoffmcs parser: 1.0 secs
rpp=10 => 11,331 bytes. JSONDecoder: 1.7 secs, rdJSONParser: 0.4 secs, hoffmcs parser: 7.1 secs
rpp=50 => 47,000 bytes. JSONDecoder: 7.5 secs, rdJSONParser: 4.9 secs, hoffmcs parser: 57.8 secs
rpp=78 => 80,002 bytes. JSONDecoder: 12.5 secs, rdJSONParser: 12.6 secs, hoffmcs parser: 140.0 secs
rpp=100 => 101,195 bytes. JSONDecoder: 15.9 secs, rdJSONParser: 19.8 secs, hoffmcs parser: 212.3 secs
Obviously, XML responses are parsed more efficiently, since Roku has native support for XML. However, not all APIs provide XML output. For smaller JSON queries (less than 75K bytes), rdJSONParser is faster although may not necessarily parse all JSON responses. For larger queries, JSONDecoder seems to work faster and should parse all valid JSON responses. The hoffmcs parser seems slower in all cases.
Send a PM to belltown if you find any problems.
7 REPLIES 7
Trevor
Visitor
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2012
02:24 PM
Re: Another JSON Parser [JSONDecoder]
Very nice, thanks for sharing!
*** Trevor Anderson - bloggingwordpress.com - moviemavericks.com ***
kbenson
Visitor
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-22-2012
05:56 PM
Re: Another JSON Parser [JSONDecoder]
"belltown" wrote:
I ran some tests on my Roku 1 to compare execution times for the different parsers using the Twitter API, changing the value of rpp to return different numbers of results:
http://search.twitter.com/search.json?q=roku&rpp=1&include_entities=true&result_type=mixed
rpp=1 => 2,304 bytes. JSONDecoder: 0.3 secs, rdJSONParser: 0.05 secs, hoffmcs parser: 1.0 secs
rpp=10 => 11,331 bytes. JSONDecoder: 1.7 secs, rdJSONParser: 0.4 secs, hoffmcs parser: 7.1 secs
rpp=50 => 47,000 bytes. JSONDecoder: 7.5 secs, rdJSONParser: 4.9 secs, hoffmcs parser: 57.8 secs
rpp=78 => 80,002 bytes. JSONDecoder: 12.5 secs, rdJSONParser: 12.6 secs, hoffmcs parser: 140.0 secs
rpp=100 => 101,195 bytes. JSONDecoder: 15.9 secs, rdJSONParser: 19.8 secs, hoffmcs parser: 212.3 secs
Obviously, XML responses are parsed more efficiently, since Roku has native support for XML. However, not all APIs provide XML output. For smaller JSON queries (less than 75K bytes), rdJSONParser is faster although may not necessarily parse all JSON responses. For larger queries, JSONDecoder seems to work faster and should parse all valid JSON responses. The hoffmcs parser seems slower in all cases.
Wow, great!
Considering how it's more correct in data it can parse and that it's almost as fast for most small inputs (and faster for large ones), I'd be happy to swap rdJSONParser out for your implementation if you're amenable. The whole purpose of librokudev is to be a community library of commonly needed routines, but that only works if people contribute. 🙂
-- GandK Labs
Check out Reversi! in the channel store!
Check out Reversi! in the channel store!
belltown
Roku Guru
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-23-2012
01:10 AM
Re: Another JSON Parser [JSONDecoder]
"kbenson" wrote:
Wow, great!
Considering how it's more correct in data it can parse and that it's almost as fast for most small inputs (and faster for large ones), I'd be happy to swap rdJSONParser out for your implementation if you're amenable. The whole purpose of librokudev is to be a community library of commonly needed routines, but that only works if people contribute. 🙂
You're welcome to add my code to librokudev, and make any changes to it that you see fit. Let me know if there's anything I need to do to make that happen. Since rdJSONParser has a different interface from my code and is faster for smaller inputs, even though it doesn't handle all JSON data, you might want to keep both versions around just in case someone has a preference for using one parser rather than the other.
kbenson
Visitor
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-23-2012
10:39 AM
Re: Another JSON Parser [JSONDecoder]
"belltown" wrote:
You're welcome to add my code to librokudev, and make any changes to it that you see fit. Let me know if there's anything I need to do to make that happen. Since rdJSONParser has a different interface from my code and is faster for smaller inputs, even though it doesn't handle all JSON data, you might want to keep both versions around just in case someone has a preference for using one parser rather than the other.
Yeah, it will probably live on in some manner for backwards compatibility, but the run-time difference is small enough on most inputs that I think most people that would want to use it would be much happier with something less likely to blow up from an odd character.
Thanks!
-- GandK Labs
Check out Reversi! in the channel store!
Check out Reversi! in the channel store!
agmark
Visitor
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-28-2012
05:44 PM
Re: Another JSON Parser [JSONDecoder]
Thank you for your work on this belltown. JSON is used a lot and the Roku box needs all the help it can get in dealing with it. Maybe someday we'll get a native json component, but until then... :roll:
gonzotek
Visitor
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-12-2012
12:50 PM
Re: Another JSON Parser [JSONDecoder]
Roku firmware 4.8 (and beyond) now includes a JSON parser: http://sdkdocs.roku.com/display/RokuSDK ... ctSincev48
It does not have support for encoding of Brightscript objects into json strings, just parsing from json strings to brightscript objects, so the json builder function found in libRokuDev may still be of utility to some people.
It does not have support for encoding of Brightscript objects into json strings, just parsing from json strings to brightscript objects, so the json builder function found in libRokuDev may still be of utility to some people.
Remoku.tv - A free web app for Roku Remote Control!
Want to control your Roku from nearly any phone, computer or tablet? Get started at http://help.remoku.tv
by Apps4TV - Applications for television and beyond: http://www.apps4tv.com
Want to control your Roku from nearly any phone, computer or tablet? Get started at http://help.remoku.tv
by Apps4TV - Applications for television and beyond: http://www.apps4tv.com
TheEndless
Channel Surfer
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-12-2012
01:09 PM
Re: Another JSON Parser [JSONDecoder]
"gonzotek" wrote:
Roku firmware 4.8 (and beyond) now includes a JSON parser: http://sdkdocs.roku.com/display/RokuSDK ... ctSincev48
It does not have support for encoding of Brightscript objects into json strings, just parsing from json strings to brightscript objects, so the json builder function found in libRokuDev may still be of utility to some people.
Probably worth noting that it's also only available in 4.8+, so you'd still need another library for JSON support on the Roku 1s (and logic to prevent a syntax error/crash on 3.x).
My Channels: http://roku.permanence.com - Twitter: @TheEndlessDev
Instant Watch Browser (NetflixIWB), Aquarium Screensaver (AQUARIUM), Clever Clocks Screensaver (CLEVERCLOCKS), iTunes Podcasts (ITPC), My Channels (MYCHANNELS)
Instant Watch Browser (NetflixIWB), Aquarium Screensaver (AQUARIUM), Clever Clocks Screensaver (CLEVERCLOCKS), iTunes Podcasts (ITPC), My Channels (MYCHANNELS)