Unfortunately I lost my remote and been using the app remote. But I accidentally factory reset my Roku now I can't be able set it up. Since it is not on wifi anymore I can't connect to my Roku anymore.
Unfortunately, no.
New Roku devices, and Factory Reset Roku devices require a physical Roku remote (or Roku compatible) to proceed through setup and activation. The necessary remotes can be found at Best Buy, Target, Walmart, etc. if you need one immediately. (about $20 for a Voice Remote).
There are also some cheaper IR (infrared) universal streaming remotes compatible with Roku that you can use to get through setup. (unless you have a Streaming Stick model - then you need a Voice Remote).
If you have a Roku with an Ethernet port, connect to that if you can and it should find and connect to your local network without problem the next time you power the Roku up. Once it's connected to your net you should be able to use the Roku remote app on your phone again. Otherwise, you're going to need a compatible physical remote.