Yes, follow what @andyross said. You do have personal information stored on the device, mostly login information for any channels that need it. And of course anyone that fires it up can see your Roku account email address in the Settings/About menu.
And you are correct that someone could simply plug it in and use it as it they were you. Depending on the channel, they might even be able to make media purchases. Of course, they wouldn't be able to get to those purchases on any other device unless they hack your account info from the Roku. Is that possible? I don't know, but most likely it's possible.
Dan Roku Stick (3600), Ultra (4640), Ultra (4670), Premiere (3920), Insignia 720p Roku TV, Sharp 4K Roku TV, Nvidia Shield, Windows 10 Pro x64 running Serviio and Plex on a wired Gigabit network.
I would only add that an area of special caution with hardware leaving your control might be any stored credit card information. From the main menu on my Roku interface, it can be found with the following selections: SETTINGS ==> PAYMENT METHOD and there is an option to UPDATE PAYMENT METHOD.
At this point, my system indicates one stored credit card by showing its last 4 digits. Despite this, I suppose it's possible that the full credit card information is actually stored elsewhere (on a Roku server?) with only a link to this data stored on the local Roku device. But I'm just guessing on this point.
In any case, I would think that all such credit card data would be among the items cleared by the FACTORY RESET option, as previously commented.