Don't think there's a limit in the software. I use portable 5TB drives and they have been fine. I haven't tested my 10TB backup drives, but think it's a non-issue (assuming it's a drive file system that it supports)
NTFS for me works, and I have been fine with 5TB, 10TB, and 14TB drives (mainly Western Digital Elements / EasyStore externals). Try a different USB cable if have one handy (it sounds like it shouldn't work, but it did with me after a tip from another forum member). Original non-working cable then worked after switching back (I had other similar cables from other drives for spare to test). Otherwise specifying exact models that have problems could possibly inform on what any future firmware updates / fixes there might need to be.
I have a 6TB and a 8TB external hard drives that connect just fine with my 2019 Roku Ultra but will not connect to my 2020 Roku Ultra but anything that's 2TB or less will connect to the 2020 Roku Ultra, what do you think the problem is?
Roku Media Player worked with my 10 TB WD-Easystore but not with 18 TB WD-Elements. It recognized the disk and files, but does not play. Click on 'play' a single file, it either flash back to the file name, or forever loading the file at 13%......
This issue is present on all of my Ultra devices (2018/2019/2020, I have three), it seems to be Roku Media Player that broke USB playback via USB drives. It will either decide to play or not. It was fine prior to the switch to the new Media Player, and has been buggy all throughout 2021 and still now. The issue you are experiencing is more recent within past couple months. Completely broke the way I normally use the boxes, which is to plug in a HD and play files. Now they just won't play. Roku Plex app works fine (direct play with no transcoding) with the same files, but requires you to run your own Plex server on a comp of yours and run it through the app. Media Player app is broken.