Actually, it includes many free and open-source components:
http://support.roku.com/entries/246042- ... -resourcesBut destruk is right: the entire system is not 'open' in the sense that it would be easy to build a complete RokuOS yourself. I haven't looked into it deeply, but my understanding is that the core parts of what makes the system a 'Roku'(the Roku look and feel, apis, brightscript interpreter, etc., possibly even hardware drivers) are not available as open code or compiled binaries.
Remoku.tv - A free web app for Roku Remote Control!
Want to control your Roku from nearly any phone, computer or tablet? Get started at http://help.remoku.tv
by Apps4TV - Applications for television and beyond: http://www.apps4tv.com