The Chromecast/Roku PQ could be an apples-to-oranges comparison. The reason is that there's most likely different streams from different content delivery networks dedicated to each service. In other words, the Roku Amazon channel might be hitting a different content delivery network and stream than the Chromecast, and there may be issues between you and the delivery point of the content delivery network.
I don't have any issues with Amazon Prime video at my home on my Roku. In addition to what jeffrok suggested, you could try setting a different DNS server in your router, like OpenDNS or Google's DNS, vs. whatever your ISP's DNS is. You may end up with a different (and better) path to the Amazon Roku stream this way. Worth a shot - takes but a couple of minutes and a router reboot.
OK - I'll try tweaking router DNS - I use OpenDNS at the moment - hopefully that wouldn't make any difference as it will mess up everything in the house - I use OpenDNS as a first line protection against various sites.