I have a 2.2Mbps DSL connection at home I use to test 1.6Mbps 720p streams from Wowza (which play just fine on another Roku XD at the office). I can see the roku pulling each file in sequence, so it's getting them all, but it takes about 5-6 minutes to fill the "retrieving" bar here on about 90% of the loads. The other 10% jump in as I'd expect, within just a few seconds.
Here's tcpdump grepped onto the GETs, it started playing this time at segment 96. It started pulling them down at segment 53, so that's 10s*(96-53)= 430s to start actual playback.