What's the audio codec you're using? Roku never has supported AVI and I sincerely doubt they ever will. It's meant to be more of an Internet streaming device than something for streaming your personal collection. If that's your main purpose for getting one then you may be disappointed. Look into using Plex which will transcode on-the-fly into a supported format, but requires a computer to be always running.
I've asked multiple times - what audio codec are you using? Get MediaInfo and analyze what's inside your MKV. I and many others use MKVs all the time, so there's a problem with yours. If you're trying to play multichannel AAC (other than stereo) then it won't work.
AC-3 should be fine. There's also the issue of whatever your Roku is connected to. It has to support the format as well, but I think everything should support AC-3. Try playing with the audio settings on your Roku: Settings->Audio.