I am unable to find any documentation about the back button press handling for these dialogs. I have checked in the onKeyEvent observer function for a back button key press, and never get the key press.
I am assuming this is default behavior, but is there a setting that changes the default behavior? I'm not seeing it in the docs.
I don't know if you found anymore information, but I just ran into the same issue and came across this old discussion: https://community.roku.com/t5/Roku-Developer-Program/Back-key-not-closing-RSG-Dialogs/td-p/454568
Basically a dialog isn't a dialog unless it's assigned to the scene's dialog parameter. I think it's a bug for the back key to get silently eaten the way it is, but clearly Roku has no intention of changing the undocumented behavior or said documentation.
-JT
The solution that I arrived at was creating a class of dialog that extends the dialog of the type you're needing, creating it with a boolean field and intercepting the onKeyEvent in that class. Then in the dialog that is a child of that class needs to set an observer on that field and hide and show itself accordingly. It's a ridiculous way of doing it, but it works, and that's Roku in a nutshell. It's also why I love Roku.