Roku Developer Program

Developers and content creators—a complete solution for growing an audience directly.
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
TheEndless
Level 9

SimpleJSON BrightScript Library

First of all, thanks to hoffmcs for the original JSON Parser! viewtopic.php?f=34&t=30688&p=186934#p186934

I was originally going to use his library for one of my channels, but found the performance was a bit slow for my liking, so I started looking into either optimizing it, or coming up with a better way to do it. Realizing that the BrightScript array and associative array syntax is very similar to the JSON syntax, that's the route I started down first, and ultimately what I stuck with. My resulting SimpleJSON functions have proven to be as much as 100 times faster than hoffmcs's full blown parser, depending on how long the JSON string is. They're certainly not ideal in every situation, but they've worked for me with everything I've thrown at them so far...

(Handy tip: I've found the SimpleJSONBuilder function to be a very convenient way of serializing "objects" for stuffing into the registry.)

Consider the following JSON string:
{
"Books": [
{
"Title": "Book 1",
"Author": "Author 1",
"Page Count": 25
},
{
"Title": "Book 2",
"Author": null,
"Page Count": 15
}
]
}

In this example, there are three things that prevent us from using it directly as a BrightScript array...
  1. The key names are quoted

  2. Some key names have spaces in them

  3. "null" is not a valid BrightScript value


Using a little creative regex, I was able to address #1 and #2 by removing the quotes and spaces from the JSON string keys (NOTE: this results in a "PageCount" key instead of a "Page Count" key in the resulting associative array). #3 is addressed simply by declaring a variable named "null" and setting it to invalid:
null = invalid

With all of that accomplished, we can now directly Eval the resulting string, and we get a BrightScript associative array!

First, and example of how to use these functions...
' Parse JSON string
jsonString = urlTransfer.GetToString( "http://my.json.url/blah.json" )
brightScriptAArray = SimpleJSONParser( jsonString )

' Build JSON string
brightScriptAArray = {
Books: [
{ Title: "Book 1", Author: "Author 1", PageCount: 25 },
{ Title: "Book 2", Author: "Author 2", PageCount: 15 }
]
}
jsonString = SimpleJSONBuilder( brightScriptAArray )


And now the source...
Function SimpleJSONParser( jsonString As String ) As Object
' setup "null" variable
null = invalid

regex = CreateObject( "roRegex", Chr(34) + "([a-zA-Z0-9_\-\s]*)" + Chr(34) + "\:", "i" )
regexSpace = CreateObject( "roRegex", "[\s]", "i" )
regexQuote = CreateObject( "roRegex", "\\" + Chr(34), "i" )

' Replace escaped quotes
jsonString = regexQuote.ReplaceAll( jsonString, Chr(34) + " + Chr(34) + " + Chr(34) )

jsonMatches = regex.Match( jsonString )
iLoop = 0
While jsonMatches.Count() > 1
' strip spaces from key
key = regexSpace.ReplaceAll( jsonMatches[ 1 ], "" )
jsonString = regex.Replace( jsonString, key + ":" )
jsonMatches = regex.Match( jsonString )

' break out if we're stuck in a loop
iLoop = iLoop + 1
If iLoop > 5000 Then
Exit While
End If
End While

jsonObject = invalid
' Eval the BrightScript formatted JSON string
Eval( "jsonObject = " + jsonString )
Return jsonObject
End Function


Function SimpleJSONBuilder( jsonArray As Object ) As String
Return SimpleJSONAssociativeArray( jsonArray )
End Function


Function SimpleJSONAssociativeArray( jsonArray As Object ) As String
jsonString = "{"

For Each key in jsonArray
jsonString = jsonString + Chr(34) + key + Chr(34) + ":"
value = jsonArray[ key ]
If Type( value ) = "roString" Then
jsonString = jsonString + Chr(34) + value + Chr(34)
Else If Type( value ) = "roInt" Or Type( value ) = "roFloat" Then
jsonString = jsonString + value.ToStr()
Else If Type( value ) = "roBoolean" Then
jsonString = jsonString + IIf( value, "true", "false" )
Else If Type( value ) = "roArray" Then
jsonString = jsonString + SimpleJSONArray( value )
Else If Type( value ) = "roAssociativeArray" Then
jsonString = jsonString + SimpleJSONBuilder( value )
End If
jsonString = jsonString + ","
Next
If Right( jsonString, 1 ) = "," Then
jsonString = Left( jsonString, Len( jsonString ) - 1 )
End If

jsonString = jsonString + "}"
Return jsonString
End Function


Function SimpleJSONArray( jsonArray As Object ) As String
jsonString = "["

For Each value in jsonArray
If Type( value ) = "roString" Then
jsonString = jsonString + Chr(34) + value + Chr(34)
Else If Type( value ) = "roInt" Or Type( value ) = "roFloat" Then
jsonString = jsonString + value.ToStr()
Else If Type( value ) = "roBoolean" Then
jsonString = jsonString + IIf( value, "true", "false" )
Else If Type( value ) = "roArray" Then
jsonString = jsonString + SimpleJSONArray( value )
Else If Type( value ) = "roAssociativeArray" Then
jsonString = jsonString + SimpleJSONAssociativeArray( value )
End If
jsonString = jsonString + ","
Next
If Right( jsonString, 1 ) = "," Then
jsonString = Left( jsonString, Len( jsonString ) - 1 )
End If

jsonString = jsonString + "]"
Return jsonString
End Function

Function IIf( Condition, Result1, Result2 )
If Condition Then
Return Result1
Else
Return Result2
End If
End Function

I hope you find it useful. All questions, comments, and criticisms are welcome! Thanks...
My Channels: http://roku.permanence.com - Twitter: @TheEndlessDev
Instant Watch Browser (NetflixIWB), Aquarium Screensaver (AQUARIUM), Clever Clocks Screensaver (CLEVERCLOCKS), iTunes Podcasts (ITPC), My Channels (MYCHANNELS)
0 Kudos
9 REPLIES 9\
kbenson
Level 7

Re: SimpleJSON BrightScript Library

This results in nearly 8x speedup of JSON parsing when keys don't have spaces. On my reference JSON, based on yours bet extended slightly and made much larger, the original simpleJSONParser takes 4705 ms to parse it 10 times. This new version takes 871 ms. And that's with a penalty because of a space in the name of one of the keys (more on that below).


Function simpleJSONParser( jsonString As String ) As Object
q = chr(34)

beforeKey = "[,{]"
keyFiller = "[^:]*?"
keyNospace = "[-_\w\d]+"
valueStart = "[" +q+ "\d\[{]|true|false|null"
reReplaceKeySpaces = "("+beforeKey+")\s*"+q+"("+keyFiller+")("+keyNospace+")\s+("+keyNospace+")\s*"+q+"\s*:\s*(" + valueStart + ")"

regexKeyUnquote = CreateObject( "roRegex", q + "([a-zA-Z0-9_\-\s]*)" + q + "\:", "i" )
regexKeyUnspace = CreateObject( "roRegex", reReplaceKeySpaces, "i" )
regexQuote = CreateObject( "roRegex", "\\" + q, "i" )

' setup "null" variable
null = invalid

' Replace escaped quotes
jsonString = regexQuote.ReplaceAll( jsonString, q + " + q + " + q )

while regexKeyUnspace.isMatch( jsonString )
jsonString = regexKeyUnspace.ReplaceAll( jsonString, "\1\2\3\4: \5" )
end while

jsonString = regexKeyUnquote.ReplaceAll( jsonString, "\1:" )

jsonObject = invalid
' Eval the BrightScript formatted JSON string
Eval( "jsonObject = " + jsonString )
Return jsonObject
End Function


When keys have spaces, a performance penalty is imposed, but it's a fairly static penalty, and it's only imposed once for each set of spaces in a key, for the key with the most spaces. That can be optimized away somewhat at the cost of a slightly more complex regex, but it's pretty fast now. Basically it's just trading regex complexity for algorithm complexity, but since the regexes are all highly optimized C, that's a good trade.

I build the key space removing regex out of variables for readability. While the regex object supports the /x flag for readability, without line spanning strings it's less useful than it could be. The actual regex I'm using to remove a space from a quoted key is

([,{])\s*"([^:]*?)([-_\w\d]+)\s+([-_\w\d]+)\s*"\s*:\s*(["\d\[{]|true|false|null)

This can be optimized slightly to remove more than one space at a time by utilizing empty match vars, but only as much as we add in. We can replace two spaces at a time, or three spaces at a time, etc, and still replace a single space, but the extra complexity on an already ugly regex just makes me cringe. Plus, I'm tired. Smiley Happy

If you don't mind, I'll add the JSON lib to librokudev. Instead of "simple", the prefix would be "rd" to match the library naming structure. I would of course to include comments crediting the original source (meaning both you and hoffmcs).

P.S. This started out as me thinking that caching the regex objects as globals would be a simple optimization, but PCRE seems to already cache regex objects requested so the second request doesn't incur a regex compile penalty, which is nice.
-- GandK Labs
Check out Reversi! in the channel store!
0 Kudos
TheEndless
Level 9

Re: SimpleJSON BrightScript Library

I know just enough regex to confuse the hell out of myself every time I look back at an expression I've written, so if you say your's is more efficient, you won't get an argument from me! Smiley Tongue
Your function, however, doesn't work with the actual JSON strings I'm working with. I have some keys with multiple (non-contiguous) spaces, which your function doesn't seem to be taking into account.

Here's an example of one of the JSON strings that fails to parse with your code:
{"params":["",["serverstatus","0","999"]],"method":"slim.request","id":1,"result":{"other player count":0,"info total albums":"1658","player count":0,"version":"7.5.1","uuid":"f8c06881-02a6-4e2e-bae3-4ca9e08fe66d","sn player count":0,"info total artists":"1711","info total songs":"20179","lastscan":"1284783999","info total genres":"69"}}
My Channels: http://roku.permanence.com - Twitter: @TheEndlessDev
Instant Watch Browser (NetflixIWB), Aquarium Screensaver (AQUARIUM), Clever Clocks Screensaver (CLEVERCLOCKS), iTunes Podcasts (ITPC), My Channels (MYCHANNELS)
0 Kudos
kbenson
Level 7

Re: SimpleJSON BrightScript Library

Whoops, one of the last things I changed caused a regression. This works with the sample you pasted. I was accidentally removing the quotes at the same time I was removing spaces, so the next replace of a space in a key didn't work, since it wasn't quoted.


Function simpleJSONParser( jsonString As String ) As Object
q = chr(34)

beforeKey = "[,{]"
keyFiller = "[^:]*?"
keyNospace = "[-_\w\d]+"
valueStart = "[" +q+ "\d\[{]|true|false|null"
reReplaceKeySpaces = "("+beforeKey+")\s*"+q+"("+keyFiller+")("+keyNospace+")\s+("+keyNospace+")\s*"+q+"\s*:\s*(" + valueStart + ")"

regexKeyUnquote = CreateObject( "roRegex", q + "([a-zA-Z0-9_\-\s]*)" + q + "\:", "i" )
regexKeyUnspace = CreateObject( "roRegex", reReplaceKeySpaces, "i" )
regexQuote = CreateObject( "roRegex", "\\" + q, "i" )

' setup "null" variable
null = invalid

' Replace escaped quotes
jsonString = regexQuote.ReplaceAll( jsonString, q + " + q + " + q )

while regexKeyUnspace.isMatch( jsonString )
jsonString = regexKeyUnspace.ReplaceAll( jsonString, "\1"+q+"\2\3\4"+q+": \5" )
end while

jsonString = regexKeyUnquote.ReplaceAll( jsonString, "\1:" )

jsonObject = invalid
' Eval the BrightScript formatted JSON string
Eval( "jsonObject = " + jsonString )
Return jsonObject
End Function
-- GandK Labs
Check out Reversi! in the channel store!
0 Kudos
TheEndless
Level 9

Re: SimpleJSON BrightScript Library

I just tested this out with some of my larger, more complex JSON strings, and I'm seeing the same performance bump. Nice! Thanks for the optimization.

As for adding it to your library, go for it. I posted it here for everyone to use. The more consolidation of "snippets", the better!
My Channels: http://roku.permanence.com - Twitter: @TheEndlessDev
Instant Watch Browser (NetflixIWB), Aquarium Screensaver (AQUARIUM), Clever Clocks Screensaver (CLEVERCLOCKS), iTunes Podcasts (ITPC), My Channels (MYCHANNELS)
0 Kudos
kbenson
Level 7

Re: SimpleJSON BrightScript Library

"TheEndless" wrote:
I just tested this out with some of my larger, more complex JSON strings, and I'm seeing the same performance bump. Nice! Thanks for the optimization.


Heh, no problem. As I stated I really just meant to try a quick caching optimization, but one thing led to another...


As for adding it to your library, go for it. I posted it here for everyone to use. The more consolidation of "snippets", the better!


Thanks! Hopefully we'll have a version out tomorrow night.
-- GandK Labs
Check out Reversi! in the channel store!
0 Kudos
hoffmcs
Level 7

Re: SimpleJSON BrightScript Library

Nice job guys. Very clever method for decoding JSON.
0 Kudos
kbenson
Level 7

Re: SimpleJSON BrightScript Library

It occurred to me the other day that while quick, this isn't the most compliant JSON parser. Specifically, it's not reversible, since space removal in keys is destructive WRT the original JSON.

Spaces are allowed in associative array keys, but not in the definition syntax we are using. Yet variable["key with spaces"] = "something" is perfectly valid BrightScript. I think I'll be modifying the implementation I've come up with to correctly handle that.

This brings up an interesting question though, which is why isn't it valid to do the following:
[code]
aa = {
"foo": "bar"
"one two": 3
}


I think it would be a real benefit if BrightScript could parse this. First of all, JSON can be parsed directly (except for null, but TheEndless has a good solution for that). Second, it's theoretically much easier to implement than many requests that have been made (a fair number by myself, I admit) for enhancements to BrightScript. Currently this syntax causes a syntax error, so enabling it has no chance of introducing unintended side-effects because of how people are currently using the language.

Any chance of getting an official Roku standpoint on this? They have stated in the past they are amenable to extending BrightScript (e.g. adding closures, which would be WONDERFUL), but I haven't heard anything about whether they are still interested in extending the core language at all.
-- GandK Labs
Check out Reversi! in the channel store!
0 Kudos
TheEndless
Level 9

Re: SimpleJSON BrightScript Library

"kbenson" wrote:
It occurred to me the other day that while quick, this isn't the most compliant JSON parser. Specifically, it's not reversible, since space removal in keys is destructive WRT the original JSON.

Right. This is precisely why I called it "SimpleJSON" and why I said it wasn't ideal for every situation. hoffmcs's library properly handles spaces in the key name, but it parses the JSON one character at a time, which isn't ideal, performance-wise.
My Channels: http://roku.permanence.com - Twitter: @TheEndlessDev
Instant Watch Browser (NetflixIWB), Aquarium Screensaver (AQUARIUM), Clever Clocks Screensaver (CLEVERCLOCKS), iTunes Podcasts (ITPC), My Channels (MYCHANNELS)
0 Kudos
kbenson
Level 7

Re: SimpleJSON BrightScript Library

"TheEndless" wrote:
"kbenson" wrote:
It occurred to me the other day that while quick, this isn't the most compliant JSON parser. Specifically, it's not reversible, since space removal in keys is destructive WRT the original JSON.

Right. This is precisely why I called it "SimpleJSON" and why I said it wasn't ideal for every situation. hoffmcs's library properly handles spaces in the key name, but it parses the JSON one character at a time, which isn't ideal, performance-wise.


Yeah, I'm thinking the nice middle-ground will be to generate a unique string that we can replace the space characters with, and then go through afterwards and replace those with spaces after the fact. That way we can still use eval for speed, but return a more correct parse. If the speed difference is non-negligible, I'll probably make it a toggle.
-- GandK Labs
Check out Reversi! in the channel store!
0 Kudos