roString.tokenize() quirks? [explained]
I eyed today to use tokenize() string method but was surprised by its behavior:
But even with one separator char things don't go as planned - here is example of trying to parse CSV:
Since this function was practically undocumented till a month or so ago, can we fix these issues? Say to behave akin to str.split(). Any forum developers that would be hurt by change of behavior?
PS. Alternatively, can you allow passing roRegex as the parameter? That will make the method more powerful (i swear i will use it then!) and the regex library is already included. That will address both issues, since one can choose to pass CreateObject("roRegex", " with ", "i") for the first case; for the seconds, can use pattern "," vs ",+" ("[,;:\t]" etc), as needed
BrightScript Debugger> ? ("Nothing to Say with DJ Charlie").tokenize(" with ")
No
ng
o
Say
DJ
C
arl
e
This was expected to use " with " as separator and split only in two pieces, "Nothing to Say" and "DJ Charlie". Instead it seems to use the parameter passed as set of separator characters. (Undocumented)But even with one separator char things don't go as planned - here is example of trying to parse CSV:
Here the empty middle-initial part (since Mr.Schmoe unlike John Q. Public has no middle name) got lost. Seems to drop the empty tokens (Undocumented)
BrightScript Debugger> ? ("Joe,,Schmoe,2012-12-12").tokenize(",")
Joe
Schmoe
2012-12-12
Since this function was practically undocumented till a month or so ago, can we fix these issues? Say to behave akin to str.split(). Any forum developers that would be hurt by change of behavior?
PS. Alternatively, can you allow passing roRegex as the parameter? That will make the method more powerful (i swear i will use it then!) and the regex library is already included. That will address both issues, since one can choose to pass CreateObject("roRegex", " with ", "i") for the first case; for the seconds, can use pattern "," vs ",+" ("[,;:\t]" etc), as needed
