This has been bugging me for a few years, and solutions have only recently crept up on Google, so to further spread the word, here is the solution
Disable IPv6 in FireFox (via about:config -> network.dns.disableipv6) or use http://127.0.0.1/ to access via IPv4 directly.
Then it's just as fast as IE on localhost. Whee!
More details here.