I recently read somewhere it was the audio volume "leveling" function has a bug that causes this. I had that enabled, and since shutting it off (about 2 weeks ago) I haven't had the issue again. Give it a try!
I was able to fix this in my ultra by going into the audio settings and selecting a specific option instead of "Auto Detect." My Roku is plugged into a receiver that supports dolby digital+, so I selected that option and the choppiness went away - no restart or waiting required. I suspect there may be an issue with the negotiation in some cases.