What is the value of saturating a hardware link with traffic? Should you consider "rolling your own" with open source software? Robustness testing vs. load testing? How do you value intelligent network protocol impairments?
This report compares capabilities of Maxwell Pro to hardware link saturators and open source network emulator software.
Network emulators delay, duplicate, drop, re-order, apply jitter, and fragment packets. These operations on packets are called “impairments”. Most network emulators introduce impairments based on statistical distributions, such as normal, fixed, and Gaussian. Most network emulators apply the impairments based on groups of packets or packet “flows”. Simple criteria define these groups or flows, such as IP address, protocol type. Beyond these basics, network emulators generally fall into two categories: (1) hardware link saturators, introducing impairments at line speed and (2) open source network emulation software. The purpose of a network emulator is to discover and correct problems prior to deployment while emulating the full range of suboptimal conditions that occur on production networks affecting network devices.
Link saturators introduce impairments at line speed and fully saturate high capacity links. This is a form of load testing -- determining whether or not the device under test can handle the load. In production networks, these situations occur during a DoS (Denial of Service) attack and/or as a result of lack of planning or provisioning of adequate network bandwidth. Adding network impairments (packet drop, duplicate, etc.) in this situation is somewhat superfluous since a fully saturated network will already be showing signs of degraded service. Many of these products advertise the ability to corrupt or repair Ethernet CRC errors. However, for several years, it is not possible to acquire a system or card without built-in CRC. Thus testing for Ethernet CRC errors is still important to the creators of Ethernet network chips (Intel and Broadcom), but rarely of use or of interest to users or developers of applications and devices operating above the physical layers.
Open source network emulator software must be installed on a hardware platform and "tuned". Often the results produced by these hybrids are only approximations because the integrator has underestimated the requirements to tune and test the performance of the open source software on the particular hardware platform. In some cases, the approximation will be "good enough", in other cases, the lack of precision and reliability of the performance will render the results meaningless.
A modern network emulator should truly emulate all network conditions, such as multiple flows of traffic through the device, statistically significant, individual and combinatorial, network impairments, protocol impairments and bandwidth limited (small pipe) situations. While every situation is different, the Q/A manager must consider the return on investment. While the hardware saturator-style network emulator provides load testing, Maxwell provides robustness testing. Robustness testing is typically more critical than load testing. Robust network devices will be able to withstand a wide variety of malevolent packets either accidentally or deliberately created. If datagrams with zero length IP header options cause a device to freeze up (a common bug in early TCP/IP implementations due to an obvious, but wrong, choice of algorithm for option processing) and the device crashes, then it really does not matter if the datagrams arrived on a 100 Mbps or 10 Gig link. What mattered is the product failure.