For everyone that's having trouble with videos rebuffering here's what I've found to work:
There seems to be an issue with transcoding directly from a 5.1 ac3 stream into aac. What I've done to overcome the issue is decoded the ac3 into a wav file (and doing a downmix along the way), then I use that wav file as the source for the audio during the conversion. I also trim 1-2 seconds off the end of the video. Doing these two things results in videos that will play successfully all the way through. I have not done any tinkering with the moov atoms, although I probably should have played with that.
I'm using ffmpeg and BeSweet to do this for me. Here are the commands I run:
c:\ffmpeg\bin\ffmpeg -i INPUT.vob -vn -acodec copy INPUT.ac3
"C:\Program Files\BeSweet\BeSweet.exe" -core( -input "INPUT.ac3" -output "INPUT.wav" -2ch ) -azid( -c light --maximize )
del INPUT.ac3
c:\ffmpeg\bin\ffmpeg -i INPUT.vob -i INPUT.wav -map 0:0 -map 1:0 -acodec aac -ab 160k -ac 2 -vcodec libx264 -vpre normal -crf 21 -threads 0 -s 720x404 -t 1:30:00 -aspect 16:9 -r 23.976 OUTPUT.mp4
del INPUT.wav
The first ffmpeg call splits the 5.1 ac3 track out into it's own file. Then I call BeSweet to do a downmix and spit out a stereo wav file. I then delete the ac3 file (it's no longer needed). I then use ffmpeg to do the encode using the video stream from the vob file and the audio from the wav file. I specify the size (-s), the duration (-t), stripping of a couple of seconds, the aspect ratio (-aspect) and the frame rate (-r). Finally, delete the wav file. I have done several videos this way and they are all working successfully. If anyone tries this, I'd love to hear if it was successfull or not.
I'm starting to wonder now if it's actually an issue with how the video and audio streams are being packaged in an mp4 (or m4v). If my suspicions are correct about the issue actually revolving around differing lengths in the audio and video streams (the audio streams seem to be longer than the video streams when you demux them, which is why shortening the duration helps), then it may indicate an issue with how ffmpeg is muxing the streams. I'm going to try doing it with mp4box at some point, but I haven't gotten there yet.