Other than the Streaming Stick models, most of the available set-top Roku players will respond to IR commands.
To eliminate the set-top devices (like your Ultra and Premiere) from receiving the IR signal from the remote, try blocking the IR receiver on the device. (piece of tape or cardboard or other obstacle) on the unit window itself, or try placing it behind the TV or turning it around.
Just a note on the gaming remote, (I have ones that came with my Ultra 4660 models with the A/B buttons), and the Voice/pairing remotes in general. The Voice Remotes are interchangeable between devices. (ie. if you wanted the gaming remote to control another Roku device other than the Ultra, you would just pair it to the other Roku device using the pairing button located under the battery cover. It would then cease controlling your Ultra through RF/wifi and begin controlling the other Roku device). I have found though that some of my different Voice remotes are still capable of controlling nearby devices through IR while others do not. (comes in handy sometimes if I misplace a remote and don't want to pair another just to change a channel or something, but not so much when it performs an unintended action on a nearby device).
While the gaming remote is no longer sold by Roku, you can still find them on Ebay. (just search for roku gaming remote rc03). I have actually replaced mine with the new rechargeable Voice Remotes Pro (versions 1 and 2) since the gaming remotes did not come with a Mute button. https://www.roku.com/products/accessories . Version 1 still has physical headphone jack while Version 2 is more for Bluetooth/using through Mobile App for headphone usage.
In order to find out your Roku device model numbers, just go to Settings/System/About. To see which remote model you have paired to your device, you can go to Settings/Remotes&Devices/Remotes/About.
-----
The Roku documention on their remotes used to be better but they revamped it a couple of years ago so lacks some of the detail it had previously. This is where to start, though. (look at the side articles on the right side of screen for addition info). https://support.roku.com/article/360011621473
