Limited Market, Highly Skilled Developers
Creating a network protocol test suite for a specific network protocol requires a very specialized software architect. The individual must be thoroughly knowledgeable about the protocol which typically requires protocol implementation experience. In addition, the individual must have a thorough understanding of how to test network protocol implementations in general, and this particular protocol specifically. This typically requires experience using products with the protocol in real-world environments. Finally, the individual must be able to translate this knowledge into a test suite architecture and implementation that is usable and maintainable by the test engineers.
Recruiting, hiring, motivating and retaining this kind of individual will be a challenge. If you have that individual on your staff, do you believe that assigning him to this type of project is the best utilization of his skills leveraging the maximum value for your company and its products? The answer is probably "NO"! You are probably better off assigning that person to work on the core products that make money for your company.
What are your alternatives?
Just Use Open Source
Many managers believe they can avoid the protocol testing problem if they incorporate a widely deployed SNMP engine in their products. However, widely deployed, open source, SNMP engines have failures! It will not be acceptable to your clients to tell them that the failure is not with your product, but with the open source SNMP engine; the clients will see it as your problem.
Just Use Free Attack Tools
Another ill-founded belief is that using freely available software tools that perform a few well known attacks (vulnerability testing) against an SNMP engine will be adequate "proof" of testing. However, these tools do not attempt SNMP conformance testing and only perform a small subset of the possible attacks.
Thus, the question to consider is what would it take to create your own SNMP Test Suite?
How to Make Your Own SNMP Test Suite
You will need:
- Five man-years of development from an experienced software engineer with a speciality in network protocols. (Cost estimate: $500,000)
- Investment in training and learning lower network layer development tools specific to the target testing platform and the desired high level user interface tools. (Cost estimate: $50,000)
- Access to IETF experts for consultation on RFC ambiguities (Cost estimate: priceless)
- Two man years of an experienced software test engineer with a speciality in network protocols. (Cost estimate: $160,000)
- Access to at least three SNMP engines from a wide range of suppliers to cross-check and verify the test grading and expectations. (Cost estimate: unknown)
- Part-time software engineer for ongoing maintenance (e.g. bug fixes, integration with other test frameworks, code modifications for special purposes)(Cost estimate: $50,000 annually and on-going)
Best Case Total Cost: $760,000
Best Case Total Time: Two Years
These estimates assume $100,000 burdened overhead for one engineer per year.
If multiple developers are employed simultaneously, and managed effectively, the time to complete could be shortened.
Risk versus Reward
The cost and time estimates presented above assume a perfect environment.
With a "buy" decision, you have the product in your hands, ready to go, right away.
With a "make" decision, there's always a risk that trouble will arise during the "making". For example, what if the key developer working on the protocol test suite leaves your company in the beginning or middle of the project? What if the design and architecture of the protocol test suite are flawed and your developer needs to begin anew? What if the developer does not effectively transition the project to the software maintenance engineer? Or worse, what if protocol test suite is unmaintainable? Or the on-going maintenance requires a very large training component?
These considerations cause many companies to adopt the policy that is informally expressed as "if you tie, you buy". This means that if the development costs are greater or the same as the cost to purchase an off-the-shelf product, then the decision should be to purchase.
More information on make versus buy decisions:
Higaki, Wesley H., "Applying an Improved Economic Model to Software Buy-vs-Build Decisions", HP Journal, August 1995.
Balakrishnan, Jaydeep, and Chun Hung Cheng. "The Theory of Constraints and the Make-or-Buy Decision: An Update and Review." Journal of Supply Chain Management: A Global Review of Purchasing & Supply 41, no. 1 (2005): 40–47.
Gardiner, Stanley C., and John H. Blackstone, Jr. "The 'Theory of Constraints' and the Make-or-Buy Decision." International Journal of Purchasing & Materials Management 27, no. 3 (1991): 38–43.