Limited Market, Highly Skilled Developers
Many quality assurance directors and vice presidents of engineering are faced with the decision to make or buy protocol test suites. These managers know they need to thoroughly test their new product implementation if they plan to release a high quality product, on time, and under budget. Nevertheless, first time managers are often startled by the price of complete turnkey protocol test suite products. Protocol test suites are developed for a limited market with a specific purpose; such products are not sold in high volume and require extremely skilled development engineers to produce.
Just Use Open Source
Many managers believe they can avoid the protocol testing problem if they incorporate a widely deployed TCP network stack. Unfortunately, this does not assure robust performance; widely deployed, open source, Linux TCP/IP stacks have failures! The Maxwell Pro TCP/IP Test Suite, for example, found failures not only in conformance, but actually multiple crash bugs in open source TCP/IP stacks. In one case, the TCP/IP stack was already embedded in a released product with millions of units shipped.
Just Use Free Attack Tools
Another ill-founded belief is that using freely available software tools that perform a few well known attacks (vulnerability testing) against a TCP/IP stack will be adequate "proof" of testing. However, these tools do not attempt TCP/IP conformance testing and only perform a small subset of the possible attacks.
Thus, the question to consider is what would it take to create your own TCP/IP Test Suite?
How to Make Your Own TCP/IP Test Suite
You will need:
- Three man-years of development from an experienced software engineer with a speciality in network protocols. (Cost estimate: $300,000)
- Investment in training and learning lower network layer development tools specific to the target testing platform and the desired high level user interface tools. (Cost estimate: $50,000)
- Access to IETF experts for consultation on RFC ambiguities (Cost estimate: unknown)
- Two man years of an experienced software test engineer with a speciality in network protocols. (Cost estimate: $160,000)
- Access to at least four TCP/IP stacks from a wide range of suppliers to cross-check and verify the test grading and expectations. (Cost estimate: unknown)
- Part-time software engineer for ongoing maintenance (e.g. bug fixes, integration with other test frameworks, code modifications for special purposes)(Cost estimate: $50,000 annually and on-going)
Best Case Total Cost: $510,000 Best Case Total Time: Two Years
These estimates assume $100,000 burdened overhead for one engineer per year.
If multiple developers are employed simultaneously, and managed effectively, the time to complete could be one year.
Risk versus Reward
The cost and time estimates presented above assume a perfect environment. With a "buy" decision, you have the product in your hands, ready to go, right away. With a "make" decision, there's always a risk that trouble will arise during the "making". For example, what if the key developer working on the protocol test suite leaves your company in the beginning or middle of the project? What if the design and architecture of the protocol test suite are flawed and your developer needs to begin anew? What if the developer does not effectively transition the project to the software maintenance engineer? Or worse, what if protocol test suite is unmaintainable? Or the on-going maintenance requires a very large training component?
These considerations cause many companies to adopt the policy that is informally expressed as "if you tie, you buy". This means that if the development costs are greater or the same as the cost to purchase an off-the-shelf product, then the decision should be to purchase.
More information on make versus buy decisions:
Higaki, Wesley H., "Applying an Improved Economic Model to Software Buy-vs-Build Decisions", HP Journal, August 1995.
Balakrishnan, Jaydeep, and Chun Hung Cheng. "The Theory of Constraints and the Make-or-Buy Decision: An Update and Review." Journal of Supply Chain Management: A Global Review of Purchasing & Supply 41, no. 1 (2005): 40–47.
Gardiner, Stanley C., and John H. Blackstone, Jr. "The 'Theory of Constraints' and the Make-or-Buy Decision." International Journal of Purchasing & Materials Management 27, no. 3 (1991): 38–43.