That is up to you. If you plan on keeping it as a backup, you can simply place it in a drawer.
If you are going to get rid of it, then "Unlink" it from your Account at https://my.roku.com/account and perform a "Factory Reset" of the device. (Settings/System/Advanced System Settings/Factory Reset.
I would probably setup the new Roku Express first on the same account as your previous device so all your Channels/apps and Subscriptions will be linked to the old account.
Just for information, can have multiple multiple Roku devices on same account. (no need to get rid of one to use another.)
Roku Community Streaming Expert Just another Roku user... I am not a Roku employee. Insignia RokuTV, Ultra 4660, Premiere+ 3921, Express 4k+ 3941, Streambar 9102