There may be an issue updating the remote's firmware: Check the current firmware version of the remote (Settings/Remotes & devices/Remotes/<remote name>/About (most WiFi Direct remotes should be on firmware version 2.01.19.108)
Otherwise, it sounds like there may be an issue with the WiFi Direct ("Device connect") or general network stack configuration.
Try this:
1) On 9.3.x, reset your network settings (Settings/System/Advanced system settings/Network connection reset/Reset connection)
2) After restarting, upgrade to 9.4.x (Settings/System/System Update/Check now)
3) Verify "Device connect" is enabled: (Settings/system/Advanced system settings/Device connect=enabled)
4) If the voice remote still doesnt work on 9.4, reset network settings as per 1)
5) If none of that works, and the remote still doesnt work/9.4 upgrade wont hold, there is the option of a factory reset
6) Barring 5), message @RokuDanny-R (provide him with your account email/model #/device ID/serial #/location and a brief description of the issue - trying to upgrade to firmware 9.4 to take advantage of AirPlay2, but you lose access to your voice remote and it keeps reverting to 9.3)