It depends how you attach the cover art. RMP does display the cover art if you add it as a tag within the MP3 file. I did that to all of mine, and the cover art shows up fine on RMP. It is a bit of a project, but it is a lot better having the cover art displayed on the television while the music plays.
The software that I used to append the cover art is called MP3tag. It is free. Google it any you will find the software and many reviews on the software easily.
This will work regardless of whether you are using a USB stick in the Roku as storage for the music, or with a DLNA server such as the one included with Windows, or Media Monkey.