The Advantages and Disadvantages of Open-Source Tools for Protocol Testing
Understanding Open-Source Tools and Protocol Testing
What are Open-Source Tools?
Open-source tools are software applications with freely available source code, allowing users to view, modify, and distribute open-source code.
The open-source tools used for network protocol testing applications fall into four categories: protocol analyzers, packet crafting tools, test framework tools, and traffic generation tools (to put a heavy traffic load on a network connection).
Protocol Analyzers allow the user to capture and record a flow of packets (network traffic), then analyze a captured traffic file to understand the sequence of events in a protocol operation.
Packet Crafting Tools allow the user to craft or forge a pathological packet that can be inserted into the packet flow to determine if the receiving device behaves appropriately.
Test framework tools allow the creation and organization of tests and the recording of test results. These tools have configuration options and provide libraries for reusing code.
Traffic generation tools allow users to bombard the device under test with a traffic flow, saturating the link. The traffic flow is specified to include a specific mix of network protocols.
Open-source protocol testing tools are used by:
Software developers and testers to ensure their software operates correctly over various network protocols.
Network engineers and system administrators troubleshoot network issues, test the performance and security of the network, find vulnerabilities, and ensure that the systems are communicating correctly.
Service providers ensure that their services are reliable, have high performance, and can support an increased number of simultaneous connections. (Examples: Verizon, AWS, PayPal, SalesForce, Comcast …)
IoT Developers ensure the correct operation and compatibility of Internet of Things (IoT) devices that communicate using a wide range of protocols.
Manufacturers of networking hardware (like routers, switches, etc.) to validate that their hardware works correctly with various protocols.
What is Protocol Testing?
Network protocol testing is the process of finding bugs, errors, or vulnerabilities in a particular network protocol implementation. A written specification represents the network protocol. The tester must verify that the performance of the protocol is consistent with the specification.
Without network protocol testing, various devices on the Internet:
(1) may not be able to function as specified. For example, something as simple as the TCP handshake may fail, or fragmentation may be unsuccessful.
(2) they may not be able to exchange packets with each other; they may not be able to interoperate. This is especially critical on the Internet, where devices from different manufacturers must interoperate.
(3) may need help to handle a heavy traffic load properly. The receiving device may hang or crash under the load rather than responding according to the specification (typically to “fail gracefully”).
(4) may report incorrect values or incorrect states. For example, a device may note that it has received no errors when the error counters have rolled over (like a speedometer rolling over to zero).
With proper network protocol testing, devices implementing the protocol will operate successfully in various conditions in production environments and production applications.
How Open-Source Tools and Protocol Testing Work Together
Open-source tools contribute to network protocol testing by filling in some of the difficult portions of the overall testing endeavor. Users may then customize these tools for their specific network protocol testing needs.
The most popular open-source network protocol testing tool is Wireshark. Wireshark captures a flow of packets and then decodes these packets, and displays the decoded results in a color-coded format so that the user may identify a particular protocol by a color.
Wireshark helps users answer questions about the network or individual device behavior. For example, a large number of TCP retransmissions could indicate a problem with a network link. Or, unusual traffic patterns between a specific source and destination could indicate a security issue.
In addition to Wireshark, even more, open-source packet analyzers are listed here.
The second category of open-source testing tools is packet crafting tools. Scapy is a command line tool with a powerful interactive packet manipulation library in Python. Scapy can forge or decode packets of a vast number of protocols, send them on the wire, capture them, and match requests and replies. It supports packet injection, custom packet formats, and scripting. PcapPlusPlus, Pypacker, and Ostinato are also freely available open-source packages that provide for packet crafting. These tools allow the user to craft a special packet that can be inserted into the packet flow to determine if the receiving device behaves appropriately.
The third category of open-source test tools is test automation frameworks. A test automation framework provides:
A graphical user interface and a command line interface for executing a series of tests
A method for saving and storing test results in multiple formats
An API for integrating other test tools, such as Wireshark
A set of libraries that perform standard routines for common test scenarios
A mechanism for identifying the device under test.
The majority of open-source test automation frameworks are intended for testing websites – the expected load on the website, virtual users of the website, or other performance tests of the website. These test automation frameworks are not suitable for network protocol testing.
A framework that is suitable for network protocol testing is Robot Framework. Robot Framework is an open-source test automation framework used by Nokia Networks (among others). Robot Framework is fully extendable with Python and Java programming languages. It has many shared test libraries. It is used for the IWL MQTT Test Suite.
The fourth category of open-source test tools are traffic generators. These software products execute on specific hardware configurations described by the open-source software team. These products generate a heavy traffic load on the link carrying traffic to the device under test. One example is Open Traffic Generator (OTG). Another popular choice is from Cisco Systems, called Cisco TRex. TRex is an open-source, low-cost, stateful, and stateless traffic generator fuelled by DPDK. It generates L3-7 traffic and provides one tool capabilities offered by commercial tools.
DPDK is the Data Plane Development Kit with libraries to accelerate packet processing workloads running on various CPU architectures. (DPDK is open-source).
TRex can scale up to 200 GB/sec with one server.
Advantages of Using Open-Source Tools for Protocol Testing
The advantages of using these open-source tools for protocol testing are:
No purchasing approval process is required. The open-source tools are freely available.
There is potential community support, provided the culture of the community is helpful and proactive.
It is likely cost-effective if the tool provides the specific functionality required.
If there is no commercial alternative, and the decision is between using open source and creating your tool or solution, starting with the open-source tool could save a person months of labor.
Free to Download and Use
Open-source tools are popular because users can freely download the code, compile it, and use it to perform the task. These users do not need to go through the approval process to purchase a commercial product – which is tedious, time-consuming, and may not be successful in their organization. Open-source tools can be modified and shared freely with others in the organization.
Community Support
Community Support. If there are enough users of the open-source code, a community will form. Participants in the community can provide advice on the best ways to utilize open-source code. They may develop enhancements and extensions to the code that are shared with the community, thus extending the capabilities of the code for more uses.
Finally, there is an element of discovery where multiple community members find they have a common problem or have discovered a serious bug. This allows the developer community to determine priorities.
Cost Effective
Cost-effective utilization. Assuming that the open-source tool provides the required functionality, the user may begin using it and quickly move forward on the user’s project.
Lack of Commercial Product Alternatives
No commercial alternative. If there is no commercial alternative, and the decision is between using open source and creating your tool or solution, starting with the open-source tool could save months or years of labor.
Disadvantages of Using Open-Source Tools for Protocol Testing
The disadvantages of using these tools for protocol testing are:
No calibration
Possibility of “abandonware.”
Dependencies
Software license restrictions
Lack of Calibration
Test and measurement tools must be calibrated so that the result is accurate when installed on the hardware. For example, if the user specifies a 50% reduction in bandwidth in a network emulation tool, but the tool only limits 5% of the bandwidth. The tool needs to be appropriately calibrated and will produce incorrect results. Most users do not run through a series of calibration tests when they install the open-source tool onto hardware simply because they are unaware of the requirement for this step.
Possibility of “abandonware.”
Often open-source tools are developed as part of a senior thesis or a graduate Ph.D. program. The resulting code serves as an example of the student’s abilities. Once that student has published and gained all there is to gain from the project, it is often abandoned. Occasionally, it is popular enough that a few users will voluntarily support the project, add new capabilities, and fix bugs. Alternatively, a commercial enterprise might “pick up” the project and turn it into a commercial product. In that case, the users were possibly better off starting with a commercial product rather than an open-source one, mainly if there is a significant time gap between the abandoning of the project and taking up the commercial support.
Dependencies on other packages
Open-source test tools often have dependencies on other open-source tools. If the other open-source tools are no longer available or have become deprecated, then building the desired open-source tool will become impossible. An excellent example is the forced migration of Python 2 to Python 3. The libraries, tools, and repositories for Python 2 were removed from the Internet. Even though it was possible to continue using Python 2, as a practical matter, without the repositories and supporting infrastructure, it was not possible to compile Python 2 code.
Software license restrictions
There are several different kinds of open-source software licenses. Some of these licenses require that you submit changes, bug fixes, or enhancements to the open-source code to the supplier of that code. This is done so that the maximum number of users benefit from improvements to the open-source code. However, a commercial organization pays its software developers for an enhancement that provides a competitive advantage. In this case, the organization wants to avoid giving the enhanced open-source code to the community. In many organizations, open-source tools or code is banned for this reason.
Take Advantage of the Right Tools for Your Protocol Testing
Open-source tools can add value to network-protocol testing. The four key areas are (1) packet capture and analysis tools, (2) packet crafting tools, (3) automated test frameworks, and (4) traffic generation tools. Price is removed as a barrier to users freely downloading and utilizing the open-source tools. A proactive community with an open, helping culture can provide advice and support. At the same time, one must be mindful of issues with tool calibration, the possibility of the project (meaning the open-source tool) being abandoned, and community support fading away. Dependencies on other tools mean that one is always at the mercy of the whims of the leaders for the particular open-source project. Finally, software license restrictions can preclude a commercial organization from considering open-source code.
An old joke in the software industry is, “How was it possible that God created the world in only seven days?”
The answer is that God did not have an installed base. Freed from the requirements of consistency, backward compatibility, meaningful documentation, and technical support, open-source developers make significant contributions, but the commercial organization is often left picking up the pieces.