I don't know quite how it would work with a message dialog open, but I have called port.GetMessage() in a temporary while loop to constantly clear through anything coming in. For all I know it's a bad idea though, not an expert.
Roku does not recommend that 2D components and built-in BrightScript components co-exist. In situations like that I'll invalidate my roScreen AND its associated roMessagePort before showing the built-in component, then re-create them after closing the built-in component. Invalidating the roScreen's port ensures that none of the key press events that occurred while the built-in component was displayed will be received by the roScreen's port. If you do invalidate the roScreen at any point, make sure there is an underlying facade layer (e.g. an roImageCanvas), otherwise the channel will close when the roScreen is closed. This does mean, however, that you won't be able to get the roMessageDialog overlaid on top of the roScreen, although that's not really a supported feature anyway; if it works today, it may not work tomorrow. Another option is to build your own 2D message dialog using the same roScreen and roMessagePort, then you can do whatever you want with the screen display and key press events.