"jbrave" wrote:
As far as benchmarking, I would imagine part 1 of the problem would be to get an accurate measurement of the time.mark() command, so one could calculate the overhead of the test itself, then wrap each statement to be tested like:
time1.mark()
statement to be tested.
time2.mark()
result = time2 - time1
starting with simple things like integer assignment and moving up from there?
I have a feeling that most the built in parts of BrightScript will perform moderately well. I don't know enough about benchmarking a language to tell what's worth doing and what isn't, but that just means that before starting anything like this in earnest some research would definitely help.
It's worth noting that BrightScript's implementation suggests that they tried to sidestep most of the language performance optimization issues by using components written in C/C++ for performance (and ease of integrating shared libraries, I assume). For the most part this works wonderfully. Manually implementing any included component with BrightScript alone (if given the capabilities) would lead to a VERY slow display (graphics code generally has a lot of loops to twiddle all or a portion of the bits in the display buffer).
In some tests, BrightScript isn't slow at all. For example, it takes ~600-700 ms to assign a simple division to a variable 10k times, compared to ~16 ms in Perl on a 2.2 Core Duo system. Considering the difference in processors, processor extensions, language age and optimizations, I think ~40x reduction in speed for running something on the Roku is actually quite good, for the time being at least. Then again, returning that simple division from a function and assigning it to the variable the same number of times incurs only ~10% penalty in Perl, but a 5000% penalty in BrightScript. This is most likely due to Perl optimizations (such as inlining the function), but highlights that BrightScript probably has many avenues for optimization that haven't been followed, and nor have they been needed yet.
In truth, I haven't really encountered many spots where BrightScript itself was the bottleneck. The only one I can think of was when implementing CRC checks (see librokudev), and that was mitigated by just using the pre-cached CRC table and referring to it. Another spot that may become more painful in the future is doing operations on roByteArray objects. As it is now, copies of portions of a byte array could do with an interface method, as manually looping over larger byte arrays DOES get to be time consuming.
In the end I think it's probably better to spend our time writing code that actually accomplishes a task, rather than benchmarking for its own sake. I think Roku will be a lot more responsive to speeding up an operation in BrightScript when we can show that it's directly impacting something we are trying to accomplish. Until then, their resources are probably stretched too thin to do general optimization in BrightScript unless it's obvious and speeds up the whole language at once.
-- GandK Labs
Check out Reversi! in the channel store!
