POSIX Standard Results Codes

Using POSIX standard result codes in SNMP testing

Buyers of network and data communication products are placing greater emphasis on quality than ever before. Networks are too complex; the costs of tracking down, finding and fixing bugs in heterogeneous, dynamic networks is simply too high.

Inevitably, your Quality Assurance department must develop a quality management system. The most pervasive (and popular) of these systems is ISO 9001, a set of quality process guidelines from the International Standards Organization (ISO). ISO's purpose is to facilitate international trade by providing a single set of standards that people everywhere would recognize and respect. The current guidelines are ISO 9001:2000.

Maybe your organization plans to apply for the Malcolm Baldridge National Quality Award. Or maybe your vice president decided it was to time to do an evaluation of the Quality Assurance Department to determine if the benefits are worth the investment. Regardless of the reason for reviewing your quality procedures, the cornerstone to all of them is clearly documented and demonstrated process and quality improvement. Quality standards are process standards not product standards.

“ISO 10007:1995, Quality management - Guidelines for configuration management. Gives you guidelines to ensure that a complex product continues to function when components are changed individually."

Guidelines you must follow

Essentially this means that the organization must keep excellent records. The records must identify the quality results of a set of tests run against each release. These generally should be date and time stamped, recording the revision level of the test suite as well as the revision level of the product, its PROMs, and its internal OS.

The organization must demonstrate comparative test results over subsequent releases -- regression testing. This is the selective retesting of a system that has been modified to ensure that any bugs have been fixed and that no other previously-working functions have failed as a result of the fixes and that newly added features have not created problems with previous versions of the software.

This testing is initiated after a programmer has attempted to fix a recognized problem or has added source code to a program that may have inadvertently introduced errors. Newly-modified code must still comply with its specified requirements and unmodified code must not be affected by the maintenance activity.

Thus to meet the ISO requirement, test results from successive product releases must be compared to previous test results, and differences noted, recorded, and addressed.

How does SilverCreek, the SNMP Test Suite, help meet this guideline?

SNMP testers will be happy to know that SilverCreek uses the POSIX Standard Result Codes Defined in IEEE Std 1003.3 1991. POSIX is the "Portable Operating System Interface" and was developed by the IEEE. These result codes are organized by a result code number and description as follows:

0-PASS
1-FAIL
2-UNRESOLVED
3-NOTINUSE
4-UNSUPPORTED
5-UNTESTED
6-UNINITIATED
7-NORESULT

The POSIX standard also allows developers to add their own results given their particular testing environment. So for example, SilverCreek has defined ABORT so that on certain kinds of failure conditions no further machine resources are consumed.

Buyers of network and data communications equipment want test results that are deterministic and comply to an industry standard, such as IEEE Std 1003.3 1991.

In addition, to get ISO 9001 certification and many related quality certifications and approvals, one has to maintain records of test results by date, product release number, test case and result. Many of the ISO documents recommend using a database. Of course, SilverCreek supports ODBC compliant databases. SilverCreek also supports the Journal file that uses the POSIX standard codes and stores the results in a delimited text file. This file may be imported into a database, a spreadsheet, or even an XML style sheet, so that managers and testers can study results and maintain excellent reporting standards.

If you would like to read and study more on this, a good resource is www.opengroup.org and http://tetworks.opengroup.org

Also, for background on "TET", the test environment toolkit, see http://tetworks.opengroup.org/Products/tet.htm

LEARN MORE

POSIX Standard Results Codes

Using POSIX standard result codes in SNMP testing

Guidelines you must follow

How does SilverCreek, the SNMP Test Suite, help meet this guideline?

InterWorking Labs