I'm guessing you're asking for an official Roku response, but here's my 2 cents...
I have no idea if there is a limit, but I've had easily close to 100 layers or more without issue. As for performance, I've only noticed an appreciable difference based on content, rather than quantity. The biggest drag being large images, scaled images, and transparent overlays. Using "Source" instead of the default of "Source_Over" wherever possible helps somewhat. Slicing your larger images (i.e. over half the screen height or width) into smaller chunks also helps a lot (this can be done using SourceRect rather than literally slicing your image into multiple files). Setting SetRequireAllImagesToDraw to false helps if you have a lot of images. And finally, I've also found that keeping your static layers separate from your dynamic layers is a good idea, so you're not redrawing content unnecessarily.
My Channels: http://roku.permanence.com - Twitter: @TheEndlessDev
Instant Watch Browser (NetflixIWB), Aquarium Screensaver (AQUARIUM), Clever Clocks Screensaver (CLEVERCLOCKS), iTunes Podcasts (ITPC), My Channels (MYCHANNELS)