One way would be to uninstall the app and then reinstall, but I know that's a pain. I'm not on my home network right now, so I can't get to any of the settings on my phone app. It's trying to find my players when they aren't here.
Removing the app from your iOS device and re-adding it is the only way I've found to clear out devices.
As a bit of information, the device keeps not only the devices tied to your account, but any device you connect to. For example, if you're at a friend's house, and you connect to his Roku, it will continue to show in your app (though grayed out) when you return home.
When you install the app on your iOS device, it will show the Roku devices on your current network. When you log in, it will show any device on your account that's not on your network (say, a device you own that's at another location).
There appears to be no press-hold or swipe left option to remove grayed out devices.
Your best bet might be to offer feedback through the App Store.
I'll bring this up with people in charge of the app. It's a great suggestion!