I've now had a few hours to hack at this, and for anyone interested here is what I've accomplished so far as well as where I think it can be taken.
I have coded up a set of "classes" in a traditional XUnit style with a test case class containing assertion methods as well as functionality for running, collecting results of, and reporting these results (currently printing to the debug console). I accomplished the error handling by having all of the assert methods call a single fail() member routine which stores an error message as an attribute and then performs i=1/0 to raise a runtime error. The runner executes the test fixture using eval(). If an error is seen via the get last error function first the attribute is check in which case the error was a failed assertion and is recorded as a 'test failure', if the attribute is not set than a runtime error in the test itself has been encountered so this is recorded as a 'test error'.
The major drawbacks of this approach is the lack of syntactical support for object oriented programming and the inability to do automated test aggregation due to lack of meta programming features. To work around the OOP issues I used a scheme where object allocation and initialization segmented into two different procedures. This allows a "class" to be subclassed by having the subclass call the initializer of its parent in its own initializer and then setting which ever attributes and methods are necessary. This allows for function overriding. The big downside of this approach is that the subclass still needs to 'attach' all it's own methods to the object. In the scope of a single class that will not be subclassed this isn't much of an issue, but it gets to be a pain very quick to have to implement an initializer where the test methods are specified twice not to mention how easy it is to forget to 'attach' a test method which would then not be executed. I don't think this is the right way to go in the long run unless formal syntax for OOP is added to BrightScript. I toyed with the idea of writing a pre-processor so that you could annotate methods as being members of a class and the necessary intializer could then be generated, but I don't think I want to put that much work to implement functionality that could likely be added to the core language in the future anyway. However, If it turns out that syntactic support for OOP is a highly desired feature by users and there is a firm confirmation that it will not be added in the future I will reconsider this position.
The alternative approach I'm thinking about currently is to use a more procedural approach (from the stand point of the test author) which I think would alleviate some of the difficult with writing tests in the OOP setup I described above, but would result in some implementations issues I haven't yet figured how to resolve. If tests are written in a purely procedural manner than test discovery would prove to be much simpler. Using some of the file-system functions a harness could look for files with some convention (for instance, all files whose names begin with 'test'), and than read and do basic parsing of the file to find all declared routines with a convention (again, say all subroutines whose names start with 'test') which will result in a list of methods that should be executed as tests. This would be much more complicated with the OOP approach as the lack of OOP syntax makes it difficult to tell 1) what method is the constructor for the 'class' and 2) what 'class' a particular method belongs to. This could probably be accomplished with a larger set of conventions such as having one test case "class" per file and assuming all "constructor" functions start with "new", but this seems unnecessarily restrictive to me. Though maybe I'm being too picky. I'd appreciate other people's opinion as to whether it is too restrictive.
With the procedural approach there are issues on the implementation side. For instance, how to discriminate between assertion failures and test errors. With the OOP approach I can store an error message on the object, but with a purely procedural approach where would/could I stick this? Seems like I'd be able to state that a test failed, but not for what reason. Additionally, due to lack of being to get some time of 'stack trace object' instead of just the line number where the error occured it would prove difficult to determine what assert actually failed. With the error message in the OOP approach I can atleast print the values that failed the assert. You could only tell that test X failed (as apposed to errored) so a lot of diligence would be required to ensure that there was only a single assert in each test. Which is generally a good practice, but quite an impedement if a requirement.
So this is where I am currently at. A lot of answers, but I've made some decent progress. Though I wouldn't be surprised if the end result looked nothing what I have so far. Any comments/questions/suggestions would be greatly appreciated.
-Mark
