From your description, it's hard to be sure if this is just the common result of running a device from the days of 480i on a 4K display or there is more going on. I did see a light review of the converter on youtube, and that guy was unimpressed saying the colors made it hard to see, so maybe a different converter would be better. You could also try your TV's picture controls – hopefully they are separate for each input.