This series of articles has drawn out the cargo cultists who insist that HTTP benchmarks must be run over the network, or that a real application should be tested, as if adding more variables makes a benchmark more useful. What would adding NIC overhead and network latency into the mix prove when the goal is to test a HTTP server, or a dynamic content stack? Zed Shaw addresses this in an excellent article on confounding, but the fundamental point is that a useful benchmark must isolate the variable being tested.
Critics may claim that for many web applications the database is the bottleneck so there is little value in testing anything else. If so that’s easily addressed with in-memory caching and alternative data stores, and then what becomes the bottleneck? Probably the HTTP server or dynamic content stack. Modern hardware and databases are very efficient and performance problems tend to be caused by poor choices in hardware or software rather than database or network overhead.
However in the interest of comprehensiveness here is a plot showing the previous test of dynamic content when the client and server are on separate machines, communicating over a gigabit VLAN. Please note that no TCP tuning was done other than the original tuning to increase the number of ephemeral ports. Both machines are identical hardware from SoftLayer (with whom my only affiliation is that of a satisfied customer).
The absolutely difference between localhost and running over the VLAN is relatively uninteresting given that it is dependent on the switching hardware, NIC hardware & driver, kernel, TCP tuning parameters, etc. Latency increases as expected, but requests/sec appear to be converging.
