First you need to post something from your debug logs,telnet port 8085 to your box and print the error or the console.use putty for windows or console for mac and linux.
Next you need to read the sdk documentation on what are the supported formats also use the search field on this forum to find similar problem to what you have.
Third and Final ,my motto is research on weekends and post questions on weekdays.You will get a faster answer on weekdays as most people have a life so do not expect much of an answer by posting a question on weekends.
Of course,me and my partner have a quarell thats why i am here on a weekend "I have no life"
Video Formats are here http://sdkdocs.roku.com/display/RokuSDKv48/Developer+Guide#DeveloperGuide-31SupportedVideoFormats paragraph 3.1
debugging instructions are herehttp://sdkdocs.roku.com/display/RokuSDKv48/Developer+Guide#DeveloperGuide-13WhatDoINeedtoGetStarted paragraph 8.1
Since I am old school,Might I suggest you print a copy of this documentation so you can highlight notes,it have good tips for development.
Channel: Goaheadmission
O/S : Linux
Home media: MYTHROKUPLAYER With Homerun