Brunson,
Here's what I'm doing to make a stream that works on the Roku (and doesn't start rebuffering after an hour). I'm pulling from a single VOB file ripped from a DVD.
ffmpeg -i INPUT.vob -vn -acodec copy INPUT.ac3
BeSweet.exe -core( -input "INPUT.ac3" -output "INPUT.wav" -2ch ) -azid( -n1 -c normal -L -3db --maximize ) -profile( ~~~~~ Default Profile ~~~~~ )
del INPUT.ac3
ffmpeg -i INPUT.vob -i INPUT.wav -map 0:0 -map 1:0 -acodec aac -ab 160k -ac 2 -vcodec libx264 -vpre normal -crf 21 -threads 0 -s 720x480 -t 1:39:39 -aspect 4:3 -r 29.97 OUTPUT.mp4
del INPUT.wav
The rebuffering issue appears to be tied to something that happens during the transcoding of the ac3 stream into aac. The only thing that I've found that works reliably to overcome the rebuffering issue is to move the audio to an interim format (wav in this case).
The first step splits the ac3 stream from the VOB file into it's own file. The second step (using BeSweet) does a downmix from 5.1 to a stereo wav. I then use the original VOB file as the source for the video and the newly created wav as the source for the audio. I'm also doing cleanup along the way (deleting the ac3 and wav files when they're no longer needed). You'll need to adjust the size (-s), duration (-t), aspect ratio (-aspect) and framerate (-r) values to match those of your source video. I've been shaving a second or so off the end of the videos too. I thought this was necessary at one time, but I'm not sure it actually is. I haven't been willing to spend the encoding time to test and verify yet.
If anybody tries this, please post back and let me know if it works for you.