This was the solution, you can try different settings on the 4k to choose the best one but it seems the auto settings are the one causing issue
My TV does not allow 4K, but it was defaulting to 1080p. I changed it to 720p and it appears to have done the trick.
