It's currently unsupported, because we don't have a whole lot of experience with encodings of 5.1 audio.
However, it should work in most cases and we are very interested in following the this thread about your experiences with 5.1 audio encoding.
One thing to be aware of when testing this out, is that there can be only one audio stream in encodings the Roku box recognizes. Please see this earlier post about removing extra audio streams without re-encoding your content:
http://forums.roku.com/viewtopic.php?t=25489--Kevin