I would guess that your remote is faulty or there's interference by something like sunlight hitting the front of your device. Try using the mobile app as a remote and if the problem goes away you'll know there's a problem with your remote.
Roku Mobile App - Free for iOS® or Android™ | Roku
Roku Community Streaming Expert
Help others find this answer and click "Accept as Solution."
If you appreciate my answer, maybe give me a Kudo.
I am not a Roku employee.