I posted this question on reddit, and got an answer. I'm sharing it here in case anyone else come across sync issues:
"I'm guessing the issue is that the Roku is waiting for a key frame in order to start playing the video. IIRC, Roku recommends that every segment begins with a key frame (sometimes called i-frames). If the first segment is missing one till 2 seconds in it may explain the issue. I'd check your encoder settings and see if you can figure out how to make that change."
I looked at my encoder settings (Sorenson Squeeze 10), and indeed, my keyframes are set to every 2 seconds. This seems like too much of a coincidence to be wrong.
I'm currently investigating how to force a keyframe on the first frame of the file.
Hope this helps others who are searching.
