Roku Developer Program

[TheEndless]

First of all, thanks to hoffmcs for the original JSON Parser! viewtopic.php?f=34&t=30688&p=186934#p186934

I was originally going to use his library for one of my channels, but found the performance was a bit slow for my liking, so I started looking into either optimizing it, or coming up with a better way to do it. Realizing that the BrightScript array and associative array syntax is very similar to the JSON syntax, that's the route I started down first, and ultimately what I stuck with. My resulting SimpleJSON functions have proven to be as much as 100 times faster than hoffmcs's full blown parser, depending on how long the JSON string is. They're certainly not ideal in every situation, but they've worked for me with everything I've thrown at them so far...

(Handy tip: I've found the SimpleJSONBuilder function to be a very convenient way of serializing "objects" for stuffing into the registry.)

Consider the following JSON string:

{
    "Books": [
        {
            "Title": "Book 1",
            "Author": "Author 1",
            "Page Count": 25
        },
        {
            "Title": "Book 2",
            "Author": null,
            "Page Count": 15
        }
    ]
}

In this example, there are three things that prevent us from using it directly as a BrightScript array...

1. The key names are quoted


2. Some key names have spaces in them


3. "null" is not a valid BrightScript value

Using a little creative regex, I was able to address #1 and #2 by removing the quotes and spaces from the JSON string keys (NOTE: this results in a "PageCount" key instead of a "Page Count" key in the resulting associative array). #3 is addressed simply by declaring a variable named "null" and setting it to invalid:

null = invalid

With all of that accomplished, we can now directly Eval the resulting string, and we get a BrightScript associative array!

First, and example of how to use these functions...

' Parse JSON string
jsonString = urlTransfer.GetToString( "http://my.json.url/blah.json" )
brightScriptAArray = SimpleJSONParser( jsonString )

' Build JSON string
brightScriptAArray = {
    Books: [ 
        { Title: "Book 1", Author: "Author 1", PageCount: 25 },
        { Title: "Book 2", Author: "Author 2", PageCount: 15 }
    ]
}
jsonString = SimpleJSONBuilder( brightScriptAArray )

And now the source...

Function SimpleJSONParser( jsonString As String ) As Object
    ' setup "null" variable
    null = invalid
    
    regex = CreateObject( "roRegex", Chr(34) + "([a-zA-Z0-9_\-\s]*)" + Chr(34) + "\:", "i" )
    regexSpace = CreateObject( "roRegex", "[\s]", "i" )
    regexQuote = CreateObject( "roRegex", "\\" + Chr(34), "i" )

    ' Replace escaped quotes
    jsonString = regexQuote.ReplaceAll( jsonString, Chr(34) + " + Chr(34) + " + Chr(34) )
    
    jsonMatches = regex.Match( jsonString )
    iLoop = 0
    While jsonMatches.Count() > 1
        ' strip spaces from key
        key = regexSpace.ReplaceAll( jsonMatches[ 1 ], "" )
        jsonString = regex.Replace( jsonString, key + ":" )
        jsonMatches = regex.Match( jsonString )

        ' break out if we're stuck in a loop
        iLoop = iLoop + 1
        If iLoop > 5000 Then
            Exit While
        End If
    End While

    jsonObject = invalid
    ' Eval the BrightScript formatted JSON string
    Eval( "jsonObject = " + jsonString )
    Return jsonObject
End Function


Function SimpleJSONBuilder( jsonArray As Object ) As String
    Return SimpleJSONAssociativeArray( jsonArray )
End Function


Function SimpleJSONAssociativeArray( jsonArray As Object ) As String
    jsonString = "{"
    
    For Each key in jsonArray
        jsonString = jsonString + Chr(34) + key + Chr(34) + ":"
        value = jsonArray[ key ]
        If Type( value ) = "roString" Then
            jsonString = jsonString + Chr(34) + value + Chr(34)
        Else If Type( value ) = "roInt" Or Type( value ) = "roFloat" Then
            jsonString = jsonString + value.ToStr()
        Else If Type( value ) = "roBoolean" Then
            jsonString = jsonString + IIf( value, "true", "false" )
        Else If Type( value ) = "roArray" Then
            jsonString = jsonString + SimpleJSONArray( value )
        Else If Type( value ) = "roAssociativeArray" Then
            jsonString = jsonString + SimpleJSONBuilder( value )
        End If
        jsonString = jsonString + ","
    Next
    If Right( jsonString, 1 ) = "," Then
        jsonString = Left( jsonString, Len( jsonString ) - 1 )
    End If
    
    jsonString = jsonString + "}"
    Return jsonString
End Function


Function SimpleJSONArray( jsonArray As Object ) As String
    jsonString = "["
    
    For Each value in jsonArray
        If Type( value ) = "roString" Then
            jsonString = jsonString + Chr(34) + value + Chr(34)
        Else If Type( value ) = "roInt" Or Type( value ) = "roFloat" Then
            jsonString = jsonString + value.ToStr()
        Else If Type( value ) = "roBoolean" Then
            jsonString = jsonString + IIf( value, "true", "false" )
        Else If Type( value ) = "roArray" Then
            jsonString = jsonString + SimpleJSONArray( value )
        Else If Type( value ) = "roAssociativeArray" Then
            jsonString = jsonString + SimpleJSONAssociativeArray( value )
        End If
        jsonString = jsonString + ","
    Next
    If Right( jsonString, 1 ) = "," Then
        jsonString = Left( jsonString, Len( jsonString ) - 1 )
    End If
    
    jsonString = jsonString + "]"
    Return jsonString
End Function

Function IIf( Condition, Result1, Result2 )
    If Condition Then
        Return Result1
    Else
        Return Result2
    End If
End Function

I hope you find it useful. All questions, comments, and criticisms are welcome! Thanks...

[kbenson]

This results in nearly 8x speedup of JSON parsing when keys don't have spaces. On my reference JSON, based on yours bet extended slightly and made much larger, the original simpleJSONParser takes 4705 ms to parse it 10 times. This new version takes 871 ms. And that's with a penalty because of a space in the name of one of the keys (more on that below).

Function simpleJSONParser( jsonString As String ) As Object
        q = chr(34)

        beforeKey  = "[,{]"
        keyFiller  = "[^:]*?"
        keyNospace = "[-_\w\d]+"
        valueStart = "[" +q+ "\d\[{]|true|false|null"
        reReplaceKeySpaces = "("+beforeKey+")\s*"+q+"("+keyFiller+")("+keyNospace+")\s+("+keyNospace+")\s*"+q+"\s*:\s*(" + valueStart + ")"

        regexKeyUnquote = CreateObject( "roRegex", q + "([a-zA-Z0-9_\-\s]*)" + q + "\:", "i" )
        regexKeyUnspace = CreateObject( "roRegex", reReplaceKeySpaces, "i" )
        regexQuote = CreateObject( "roRegex", "\\" + q, "i" )

        ' setup "null" variable
        null = invalid

        ' Replace escaped quotes
        jsonString = regexQuote.ReplaceAll( jsonString, q + " + q + " + q )

        while regexKeyUnspace.isMatch( jsonString )
                jsonString = regexKeyUnspace.ReplaceAll( jsonString, "\1\2\3\4: \5" )
        end while

        jsonString = regexKeyUnquote.ReplaceAll( jsonString, "\1:" )

        jsonObject = invalid
        ' Eval the BrightScript formatted JSON string
        Eval( "jsonObject = " + jsonString )
        Return jsonObject
End Function

When keys have spaces, a performance penalty is imposed, but it's a fairly static penalty, and it's only imposed once for each set of spaces in a key, for the key with the most spaces. That can be optimized away somewhat at the cost of a slightly more complex regex, but it's pretty fast now. Basically it's just trading regex complexity for algorithm complexity, but since the regexes are all highly optimized C, that's a good trade.

I build the key space removing regex out of variables for readability. While the regex object supports the /x flag for readability, without line spanning strings it's less useful than it could be. The actual regex I'm using to remove a space from a quoted key is

([,{])\s*"([^:]*?)([-_\w\d]+)\s+([-_\w\d]+)\s*"\s*:\s*(["\d\[{]|true|false|null)

This can be optimized slightly to remove more than one space at a time by utilizing empty match vars, but only as much as we add in. We can replace two spaces at a time, or three spaces at a time, etc, and still replace a single space, but the extra complexity on an already ugly regex just makes me cringe. Plus, I'm tired. 🙂

If you don't mind, I'll add the JSON lib to librokudev. Instead of "simple", the prefix would be "rd" to match the library naming structure. I would of course to include comments crediting the original source (meaning both you and hoffmcs).

P.S. This started out as me thinking that caching the regex objects as globals would be a simple optimization, but PCRE seems to already cache regex objects requested so the second request doesn't incur a regex compile penalty, which is nice.

[TheEndless]

I know just enough regex to confuse the hell out of myself every time I look back at an expression I've written, so if you say your's is more efficient, you won't get an argument from me! 😛
Your function, however, doesn't work with the actual JSON strings I'm working with. I have some keys with multiple (non-contiguous) spaces, which your function doesn't seem to be taking into account.

Here's an example of one of the JSON strings that fails to parse with your code:

{"params":["",["serverstatus","0","999"]],"method":"slim.request","id":1,"result":{"other player count":0,"info total albums":"1658","player count":0,"version":"7.5.1","uuid":"f8c06881-02a6-4e2e-bae3-4ca9e08fe66d","sn player count":0,"info total artists":"1711","info total songs":"20179","lastscan":"1284783999","info total genres":"69"}}

[kbenson]

Whoops, one of the last things I changed caused a regression. This works with the sample you pasted. I was accidentally removing the quotes at the same time I was removing spaces, so the next replace of a space in a key didn't work, since it wasn't quoted.

Function simpleJSONParser( jsonString As String ) As Object
        q = chr(34)

        beforeKey  = "[,{]"
        keyFiller  = "[^:]*?"
        keyNospace = "[-_\w\d]+"
        valueStart = "[" +q+ "\d\[{]|true|false|null"
        reReplaceKeySpaces = "("+beforeKey+")\s*"+q+"("+keyFiller+")("+keyNospace+")\s+("+keyNospace+")\s*"+q+"\s*:\s*(" + valueStart + ")"

        regexKeyUnquote = CreateObject( "roRegex", q + "([a-zA-Z0-9_\-\s]*)" + q + "\:", "i" )
        regexKeyUnspace = CreateObject( "roRegex", reReplaceKeySpaces, "i" )
        regexQuote = CreateObject( "roRegex", "\\" + q, "i" )

        ' setup "null" variable
        null = invalid

        ' Replace escaped quotes
        jsonString = regexQuote.ReplaceAll( jsonString, q + " + q + " + q )

        while regexKeyUnspace.isMatch( jsonString )
                jsonString = regexKeyUnspace.ReplaceAll( jsonString, "\1"+q+"\2\3\4"+q+": \5" )
        end while

        jsonString = regexKeyUnquote.ReplaceAll( jsonString, "\1:" )

        jsonObject = invalid
        ' Eval the BrightScript formatted JSON string
        Eval( "jsonObject = " + jsonString )
        Return jsonObject
End Function

[TheEndless]

I just tested this out with some of my larger, more complex JSON strings, and I'm seeing the same performance bump. Nice! Thanks for the optimization.

As for adding it to your library, go for it. I posted it here for everyone to use. The more consolidation of "snippets", the better!

[kbenson]

"TheEndless" wrote:
I just tested this out with some of my larger, more complex JSON strings, and I'm seeing the same performance bump. Nice! Thanks for the optimization.

Heh, no problem. As I stated I really just meant to try a quick caching optimization, but one thing led to another...

As for adding it to your library, go for it. I posted it here for everyone to use. The more consolidation of "snippets", the better!

Thanks! Hopefully we'll have a version out tomorrow night.

[hoffmcs]

Nice job guys. Very clever method for decoding JSON.

[kbenson]

It occurred to me the other day that while quick, this isn't the most compliant JSON parser. Specifically, it's not reversible, since space removal in keys is destructive WRT the original JSON.

Spaces are allowed in associative array keys, but not in the definition syntax we are using. Yet variable["key with spaces"] = "something" is perfectly valid BrightScript. I think I'll be modifying the implementation I've come up with to correctly handle that.

This brings up an interesting question though, which is why isn't it valid to do the following:
[code]
aa = {
"foo": "bar"
"one two": 3
}

I think it would be a real benefit if BrightScript could parse this.  First of all, JSON can be parsed directly (except for null, but TheEndless has a good solution for that).  Second, it's theoretically much easier to implement than many requests that have been made (a fair number by myself, I admit) for enhancements to BrightScript.  Currently this syntax causes a syntax error, so enabling it has no chance of introducing unintended side-effects because of how people are currently using the language.

Any chance of getting an official Roku standpoint on this?  They have stated in the past they are amenable to extending BrightScript (e.g. adding closures, which would be WONDERFUL), but I haven't heard anything about whether they are still interested in extending the core language at all.

[TheEndless]

"kbenson" wrote:
It occurred to me the other day that while quick, this isn't the most compliant JSON parser. Specifically, it's not reversible, since space removal in keys is destructive WRT the original JSON.

Right. This is precisely why I called it "SimpleJSON" and why I said it wasn't ideal for every situation. hoffmcs's library properly handles spaces in the key name, but it parses the JSON one character at a time, which isn't ideal, performance-wise.

[kbenson]

"TheEndless" wrote:

"kbenson" wrote:
It occurred to me the other day that while quick, this isn't the most compliant JSON parser. Specifically, it's not reversible, since space removal in keys is destructive WRT the original JSON.

Right. This is precisely why I called it "SimpleJSON" and why I said it wasn't ideal for every situation. hoffmcs's library properly handles spaces in the key name, but it parses the JSON one character at a time, which isn't ideal, performance-wise.

Yeah, I'm thinking the nice middle-ground will be to generate a unique string that we can replace the space characters with, and then go through afterwards and replace those with spaces after the fact. That way we can still use eval for speed, but return a more correct parse. If the speed difference is non-negligible, I'll probably make it a toggle.