Network Emulator Comparison: Maxwell vs. "Brand X"

Capability Maxwell Pro Brand X
Impairments at wire speed X
Operates at Layer 2 for easy setup
Network Impairments:
Delay (latency), duplicate, drop/lose, re-order, jitter, fragment, burst errors ✓(1)
Bandwidth limiting (emulate slow link)
Concurrent multiple flows
Introduce different impairments on concurrent multiple flows
Protocol Impairments (“smart” impairments at layer 3 or 4) X
User plug-ins:
Keep track of and respond to protocol states X
Program your own impairment in C or C++ X
Customize for particular situation, e.g. legacy network X
Protocol Impairments — TCP/IP:
Hundreds of TCP/IP robustness and vulnerability tests X
IPv6 and IPv4 fragmentation impairments X
Illegal IP options X
Replace SYN/ACK when FIN is expected X
Insert random delays into ACK packets X
Protocol Impairments -- Session Initiation Protocol (SIP):
Add empty CR at end of message and adjust Content-Length X
Add blanks before CRLFs in request/status line without adjusting ContentLength X
Add trailing dot to DNS names found in request/status line and message header X
Insert invalid domain name (with several extra octets in From: header line X
TIA-921 and ITU G.1050 Emulation ✓(2)
Packet capture and decode uses open standards
Unlimited number of protocols captured and decoded Limit to just over 200
Time-varying network conditions ✓(3)
Command line interface Limited to TCL
Graphical user interface Browser interface
Remote control and operation via VNC, SSH, Telnet See note (4)
Remotely operate via a program written in:
Java X
Perl X
Python X
Tcl
Define scenarios and save in profile
Single / multi user operation Single user/multiple flows Multi-user
Single / multi user operation Single user/multiple flows Multi-user
Act as one end of the network conversation X
Manipulate contents of headers and payload:
Randomly corrupt a packet
Intelligently corrupt a packet (for exercising higher level protocols, it is preferable to do this intelligently, not randomly) X
Create a packet and selectively insert into the flow X
Dynamically create and change distinct flows of packets X
Inspect packet headers to select packets for modification ✓(5)
Inspect packet data to select packets for modification ✓(5)
Rewrite packet headers X
Rewrite packet data X
The content of that packet X
The content of predecessor packets X
The content of packets in other flows X
The mathematical model of packet flows X
The position in the protocol handshake X

NOTES:

(1) Brand “X” lists many impairments, however, there is no way for the user to create his/her own impairments. Brand “X” offers “decimate” for example to drop every tenth packet, but what if you wanted to drop the 3rd, 7th, and 15th packets? With Maxwell the user can define any impairment he wants, but with Brand “X” it would require custom development work ($$).

(2) IWL supports the complete set of scenarios for the TIA-921 and ITU G.1050 standard that defines the full range of conditions that are likely to occur in modern IP networks. Our customers advise us that we are the only supplier to implement all the tests/conditions/scenarios defined by the standard. Brand “X” claims to support the standard; inquire if there are any limitations to this support.

(3) Brand “X” information suggests that impairments can be applied over time applying a counter or trigger. It is not clear if they can apply the impairments over a statistical distribution, for example:

(4) Brand “X” information discusses integration with other test tools, however, no information is publicly available about how this is done. Thus, it is not clear if remote access is supported via popular protocols such as VNC, SSH, telnet.

(5) Brand “X” information states that they can filter on any field anywhere in the packet. However, once they do that and match a filter, then what do they do with the packet or the field? Can they re-write the field, or is it just limited to an impairment on a packet?

(6) Brand “X” supports many different types of network interface cards (e.g. OC-3, T1). These interfaces can be very expensive and quickly increase the price of the network emulator. IWL takes a different approach by recommending a network topology and configuration so that standard, off-the-shelf, network switches, routers, etc. can be used to configure the lab environment.

Categories of Network Emulators

Network emulators delay, duplicate, drop, re-order, apply jitter, and fragment packets. These operations on packets are called “impairments”. Most network emulators introduce impairments based on statistical distributions, such as normal, fixed, and Gaussian. Most network emulators apply the impairments based on groups of packets or packet “flows”. Simple criteria define these groups or flows, such as IP address, protocol type. Beyond these basics, network emulators generally fall into two categories: (1) hardware link saturators, introducing impairments at line speed and (2) open source network emulation software. The purpose of a network emulator is to discover and correct problems prior to deployment while emulating the full range of suboptimal conditions that occur on production networks affecting network devices.

Hardware Link Saturators

Link saturators introduce impairments at line speed and fully saturate high capacity links. This is a form of load testing -- determining whether or not the device under test can handle the load. In production networks, these situations occur during a DoS (Denial of Service) attack and/or as a result of lack of planning or provisioning of adequate network bandwidth. Adding network impairments (packet drop, duplicate, etc.) in this situation is somewhat superfluous since a fully saturated network will already be showing signs of degraded service. Many of these products advertise the ability to corrupt or repair Ethernet CRC errors. However, for several years, it is not possible to acquire a system or card without built-in CRC. Thus testing for Ethernet CRC errors is still important to the creators of Ethernet network chips (Intel and Broadcom), but rarely of use or of interest to users or developers of applications and devices operating above the physical layers.

Open source network emulator software

Open source network emulator software must be installed on a hardware platform and “tuned”. Often the results produced by these hybrids are only approximations because the integrator has underestimated the requirements to tune and test the performance of the open source software on the particular hardware platform. In some cases, the approximation will be “good enough”, in other cases, the lack of precision and reliability of the performance will render the results meaningless.

Value of Maxwell

A modern network emulator should truly emulate all network conditions, such as multiple flows of traffic through the device, statistically significant, individual and combinatorial, network impairments, protocol impairments and bandwidth limited (small pipe) situations. While every situation is different, the Q/A manager must consider the return on investment. While the hardware saturator-style network emulator provides load testing, Maxwell provides robustness testing. Robustness testing is typically more critical than load testing. Robust network devices will be able to withstand a wide variety of malevolent packets either accidentally or deliberately created. If datagrams with zero length IP header options cause a device to freeze up (a common bug in early TCP/IP implementations due to an obvious, but wrong, choice of algorithm for option processing) and the device crashes, then it really does not matter if the datagrams arrived on a 100 Mbps or 10 Gig link. What mattered is the product failure.

*Other hardware configurations available. Contact info@iwl.com (1) Controlled by external software that can vary operating parameters roughly ten times per second. (2) Based on Netem burst model; requires consultation with IWL Engineer.

Contact Us


© 2018 InterWorking Labs, Inc. dba IWL. ALL RIGHTS RESERVED.
Web: https://iwl.com
Phone: +1.831.460.7010
Email: info@iwl.com