Roku Developer Program

[wmd1947]

Hi there,

I am investigating bandwidth usage for a live stream backed by a CDN, and it seems Roku devices are responsible for over 60% of all usage, despite only being used by 10% of users. This looks like some massive bug, and causing us to consider banning Roku devices entirely.

Would it be possible for some advice to be provided on this?

We are seeing Roku devices request the same MPEG segment up to 10 times, and rarely just once. In the usual case, each MPEG segment is requested 5 times. It looks like there might be some crazy aggressive retry/timeout mechanism that is malfunctioning.

Aside from costing us money, this is likely to be impacting all users.

Are there any M3U8 tags we can use to control retries? Can Roku please comment on what can be causing this?

Example impacted stream URL: https://live.livetvstream.co.uk/LS-63503-4/index.m3u8

Thanks,

[RokuWimM]

Hi,

Thank you for reporting this issue and providing a test stream. We're trying to reproduce this issue and would like to know the firmware version of the devices that create this extreme network load. Would you be able to provide the user agent of an HTTP GET request that's issued multiple times.

Thanks,
Wim.

[wmd1947]

Thanks for taking a look at this. Below is output of a simple stats script, fed a CDN log file. I have removed non-Roku agents; each section usually contains top 20 entries.

(Google Cloud Storage URLs because I can't attach stuff here. Shout and I'll make them more accessible if necessary)

https://storage.googleapis.com/download/storage/v1/b/roku-forums-crazy-device-logs/o/summary.txt?gen...

Below it is a sample of all requests made by one IP address, showing the constant retries for a single segment

https://storage.googleapis.com/download/storage/v1/b/roku-forums-crazy-device-logs/o/url-by-frequenc...

Raw request log for that same agent:

https://storage.googleapis.com/download/storage/v1/b/roku-forums-crazy-device-logs/o/single-agent-lo...

[wmd1947]

It occurred to me that the devices might be racing duplicate requests and cancelling the slowest, in order to e.g. cope with bad wifi. Unfortunately we (and most people) are billed by request at the application layer than actual transfers at the network layer.

[RokuWimM]

I suspect each of these requests fetch a different range of the segment. The total amount of bytes downloaded should not be higher.

Does this CDN have issues with range requests? Is the bandwidth consumption for Roku devices higher, or is only the number of HTTP (partial) GET requests higher?

[vespa59]

@wmd1947 OP, did you ever figure this out? I'm seeing similar behavior during a particularly busy event and am looking for clues. 

[wmd1947]

I verified the requests were indeed "partial content" responses, and the Roku boxes were passing a Range: header.  I did not verify whether the sum of those partial responses is equivalent to a normal client, but according to our CDN reports, they most certainly are not.

This behaviour is unfortunately non-optional -- it's baked into the requirements spec for Roku-compatible HLS feeds.

We purchased a Roku device to test future configuration changes against, but I'm not hopeful of finding a solution.

If you are stuck dealing with per-request CDN pricing, consider moving to a CDN that does not feature such pricing. It's a lame solution. Thankfully Roku does not account for most of our traffic. If it did, probably we would consider more aggressive solutions, but for now the pain is manageable.

Really it would be so much better if the devices did not issue these requests by default. Other players (such as Video.js) only use partial content requests during seeking. It seems nonsensical to request partial content for an otherwise linear, live stream where the user will almost certainly consume a full MPEG chunk

Sorry I can't be of more assistance. If you find out anything please feel comment on the thread - would really love a solution to it.

[vespa59]

Will do, and thanks so much for the quick response!