In your onKeyEvent that controls movement to get to your rowlist you need something like below
if NOT m.buttongroup.hasFocus()
Use this to tell when the buttongroup has focus and in your rowlist movement do not allow the focus back to the rowlist until the press of up/down and the buttongroup isn't focused. When the buttongroup has focus do not allow up/down keypresses to have an effect in your rowlist onKeyEvent.
I assume you use an observer to watch the buttongroup button press. In the call the observer makes you can set focus back on the rowlist after a selection has been made in the buttongroup. Also use the if m.buttongroup.hasFocus() and key = "back" to set focus back on the rowlist and act like cancel (aka no choice made).