I have only before had this with Roku built in but recently bought a Roku Express for a tv I have. I'm used to my tvs Roku starting almost instantly but the hdmi plugged Express one takes a good 1 to 2 minutes before it will let the home page come up. I don't know if it matters but I unplug it between uses since I move my tv occasionally (it's small). Is this lag/wake up time normal?
Like @StreamerUser said, leave your Roku powered on. Use the included power supply, not the switched USB port on the TV.
Note that leaving the Roku powered means you'll want to use the HOME key to return to the home screen when you're done watching. If you are streaming, and you turn off the TV, Roku will keep streaming (unless it loses power, which a switched USB port will do).
DBDukes Roku Community Streaming Expert Note: I am not a Roku employee.
If this post solves your problem please help others find this answer and click "Accept as Solution."