That was a great suggestion, malloys. Windows 7 had the same problem (perhaps because it was running inside Parallels on my Mac), but my Android tablet was able to telnet successfully to the Roku and run genkey!
I am still mystified as to why it doesn't work from the Mac, when port 8085 works fine. Port 8080 is used for a lot of web stuff (since it's similar to port 80 which is http) so it seems like a poor choice for Roku to listen on, given various other things competing for that port, which presumably is/was the problem, although I never found anything blocking 8080 or listening on it or otherwise interfering.
Harumph. I still think that genkey should be a web service in the Developer section, not somehow associated with a particular piece of Roku hardware. Put that in the Suggestion box.
In any case, I seem to now have the required generated key and packaged application, so I'm making some progresss again, finally. Thanks to all who responded!!