Yes, follow what @andyross said. You do have personal information stored on the device, mostly login information for any channels that need it. And of course anyone that fires it up can see your Roku account email address in the Settings/About menu.
And you are correct that someone could simply plug it in and use it as it they were you. Depending on the channel, they might even be able to make media purchases. Of course, they wouldn't be able to get to those purchases on any other device unless they hack your account info from the Roku. Is that possible? I don't know, but most likely it's possible.
Dan Roku Stick (3600), Ultra (4640), Ultra (4670), Premiere (3920), Insignia 720p Roku TV, Sharp 4K Roku TV, Nvidia Shield, Windows 10 Pro x64 running Serviio and Plex on a wired Gigabit network.