You have to use your TV's remote. If it keeps going back to the Ultra's input make sure "1-touch play" isn't checked under Settings->System->Control other devices.
Roku Community Streaming Expert
Help others find this answer and click "Accept as Solution."
If you appreciate my answer, maybe give me a Kudo.
I am not a Roku employee.