The problem is with CABAC reference frames. The specification says there can be up to 16. Roku Media Player will play videos at 720p or less with 16 frames, but not 1080p with 16 frames.
This is very definitively a problem with the way Roku Media Player works. It's possible that I just didn't run into this before, even though I have a 1080p video that has 16 frames that worked before and doesn't work now.
If Roku would properly document what works and what doesn't, this wouldn't have been a problem.
Now that I've done Roku's debugging for them, the least they could do is properly publish the specifications for what RMP does and does not accept. I still think it's a fault on Roku's part that compression settings that are legal and works at 720p and below don't work at higher resolutions, and it's still Roku's fault for not providing proper documentation.
I was thinking of doing more testing to find out just what else is wrong, but I'm not going to put any more effort into this. There is a simple work-around that seems to address the problem, so unless I run into another problem I'm done with this topic.
And if anyone cares, I tried quite a number of other DLNA servers before going with Windows Media Player, including Plex and Kodi and at least one other I don't remember, and I didn't like them. I don't like the way they snoop on the user, keeping records on everything you do. They also had other faults, like not working if fast forward and other problems I don't remember. Yes, at least one of them would transcode videos on the fly, but that meant you couldn't move backwards to repeat something you missed, and it put a big load on the serving computers. Say what you want about Microsoft (and after 50 years working with computers I could say a lot), Windows Media Player actually works quite well as a DLNA server, it puts only a trivial load on the computer running it, and it's one of Microsoft's better efforts: which is probably why they discontinued it.