Not sure what model/firmware you have, or even what form of casting you are doing (casting, screen mirroring) in what app/process, but next time it fails, try disabling/enabling WiFi on your phone instead of rebooting the Roku (its a lot quicker and easier than the Roku reboot generally speaking).
I have a roku express. I’m not sure of the firmware level. Disabling and enabling the WiFi on my phone doesn’t help. I am trying to send a video from my YouTube app to the roku. My YouTube app is up to date.