First, the media server function built into Windows is terrible, as it can't play media that isn't supported by Roku (it has no issues with supported media). It has no useful metadata retrieval and poor library management.
AvsGunnar has suggested several good media server products, and all are free to use (might offer some additional functions for a cost). I personally use Serviio, but since that requires using the Roku Media Player (there is no Serviio app for Roku), and RMP has so many issues that haven't been addressed for close to 5 years, I would suggest any that were mentioned.
But the main problem is RMP. As I mentioned it hasn't been updated for close to 5 years, and even then the majority of the last updates were for music, not video, playback. Roku devices only support a handful of containers (MKV, MP4, MOV, TS) and even fewer video and audio codecs. If your media is unsupported, you need a media server that can transcode your media into a version that Roku can play. I have Plex, Emby and Jellyfin all running on a test server on my network and can recommend any of them for general use. My only caveat for any of them is that Jellyfin and Plex do not support playlist (PLS) files, and Emby has a few limitations with them. I personally use playlists extensively, which is why Serviio is still my primary media server.
Disclaimer: I am the author of the Roku profiles (and several TV profiles as well) used in Serviio, and a moderator on their forum. But I am not a paid employee of Serviio. I am happy to assist anyone that has questions about how to use a media server on their home network, even if it's not Serviio.
