Subject: PWG> Test Plan for RFC 1759

From: Chris Wellens <[email-masked]>
Date: Sun, 29 Dec 1996 14:35:49 -0800 (PST)

Following is the Test Plan for interoperability,
comprised of two documents:  the first part is updated
from a draft distributed at the New York meeting, and
the second part is from Jay Martin at Underscore
(unedited).

			Test Plan 
		for Products Implementing the
		   Printer MIB, RFC 1759 
		   (DRAFT 1.2, 12.29.96)

The primary goal of standardization is to achieve 
interoperability of systems developed by independent sources.  
The IETF standardization process as defined by RFC 2026 
"The Internet Standards Process -- Revision 3" 
(http://ds.internic.net/rfc/rfc2026.txt) requires 
interoperability testing (see section 4.1.2) before that 
specification can be advanced from Proposed Standard
to Draft Standard on the three level IETF standards
track.  (Note that this requirement differentiates the 
IETF standards process from many other organizations  
that define standards.)

A lack of interoperability can be due to a poorly specified  
standard and/or to low quality implementations of the  
standard. Both of these causes must be corrected before  
interoperability can be achieved. However, the IETF  
standardization process only requires that non-interoperable 
due to poor specification be corrected and leaves  
"correction" of low quality implementations to "the market."

Implementation of an SNMP MIB requires implementation of a 
network transport protocol, such as UDP, and implementation 
of the SNMP protocol.  The SNMP protocol requires two 
entities, one acting in a "manager" role and the other 
acting in the "agent" role. An SNMP MIB is implemented in 
both an agent and a manager.  Demonstrating interoperability  
requires that both agents and managers be implemented.
Unlike other products implementing SNMP (or other IETF 
protocols), printers are largely receive only devices and 
do not have a network pathway out to other products as hubs, 
routers, etc. do.  With the exception of traps, printers are 
passive devices from a network management perspective.  For 
this reason, the most significant aspect of interoperability 
will be consistent operation (returning the same values) 
under like operating conditions, rather than the ability of 
two devices to send and receive packets.

Requirements to Demonstrate Interoperability

I.  Values:  

In order to move RFC1759 forward to "Draft Standard", 
each of the six implementations must be tested for: 

(1)  consistent predictable traps 
(2)  consistent predictable scalars (non-table variables)
(3)  consistent predictable table values 

A document reporting the results of all tests for each 
coded printer (stripped of manufacturer identity) will be 
provided to the IETF.

II. Applications:  

Underscore will provide the Print Alert software.  If other 
software applications become available, they will be 
incorporated into the testing.

III.  Heavy/Load and Heavy Traffic Conditions:  

Tests in I and II should be repeated under heavy traffic 
conditions to insure that operation remains consistent.

IV.  Protocol Conformance Testing  

In addition to all printer MIB specific issues, the general 
SNMP agent should be implemented correctly and capable of 
passing a suite of protocol conformance tests.

However, in as much as RFC 1759 is one of the largest MIBs 
with 179 objects, it is likely that there would not be
sufficient time in the three day period to include
this.   Also RFC 2026 does not strictly require proof of 
correct protocol implementation, so the testing may
have to be confined to consistent implementation of
the MIB objects.

PURPOSE

The purpose of the testing is to demonstrate satisfactory
interoperability of six implementations of the Printer MIB 
RFC 1759, so that RFC 1759 may be advanced through the 
IETF standards process.

GOALS 

The outcome of this testing should be an independent, 
unbiased report that states the results of all tests for 
each coded printer (stripped of manufacturer identity).    
The report must show that 90% of all results were 
consistent.  All inconsistent results (the remaining
10%) must be addressed by the Printer MIB Working Group 
and corrective action taken.  

METHODOLOGY 

Schedule an interoperability test event for a three day 
period that vendors or their proxy are required to attend.  
The fee to use the Stardust lab facility for three days is 
$9,000 and could be split among the attendees.

The following is from Jay Martin at Underscore:

----------------------------------------------------------------------
--  JK Martin               |  Email:   [email-masked]          --
--  Underscore, Inc.        |  Voice:   (603) 889-7000              --
--  41C Sagamore Park Road  |  Fax:     (603) 889-2699              --
--  Hudson, NH 03015-4915   |  Web:     http://www.underscore.com   --
----------------------------------------------------------------------

------------------------------------------------------------------------------

	  Alert Table Test Plan for PWG Printer MIB Bake-off

			     Version 1.0

This part of the bake-off focuses on problematic events commonly
encountered with printers.  In particular, the Alert Table components
of the Printer MIB will be exhaustively tested.

It is in the best interests of the PWG and other other implementors to
define a reasonably complete list of "typical" alert conditions and a
set of corresponding guidelines for indicating which MIB alerts should
be posted for those conditions.  Such guidelines should serve to
maximize interoperability between agents and management applications
for this critical aspect of the Printer MIB.

For each of the defined "Test Scenarios" described below, the entire
set of Printer MIB variables (and other associated MIB variables) will
be captured twice, both before and after the test is performed.  (The
need to capture all variables comes as the result of prior testing of
some implementations in which certain "description" fields were
incorrectly changed as the result of an alert condition.)

The analysis of each test will involve the creation of a "differences"
file that shows the precise change in the agent database as the result
of the specific test.  It is hoped that most analyses will be
performed during the course of the testing period; however, depending
on the speed of the agent implementations, some analyses may have to
be deferred until after the event.

Vendors are strongly encouraged to bring two additional toner
cartridges, where one is nearly empty (for "toner low") and the other
is empty (for "toner out").  Also, it will be quite useful for each
vendor to bring as many printer options as possible, such as addtional
input/output trays, to test how various dimension variables are
reported.

Test Scenarios
--------------

This draft only lists the basic Id and Title of each Test Scenario;
additional structuring and detail will be provided before the event.

For each Scenario, special attention must be paid to the various
subunit status values, if appropriate, as well as the impact on the HR
MIB status values.

  Id   Title
  ---  ---------------------------------------------------------------
  1.1  Input Tray Removal
  1.2  Input Tray Reinsertion
  1.3  Input Tray Insertion with Different Tray Dimensions
  1.4  Input Tray Missing
  1.5  Input Tray Nearly Empty
  1.6  Input Tray Empty with Demand
  1.7  Input Tray Empty without Demand
  1.8  Input Tray Jam

  2.1  Output Bin Nearly Full
  2.2  Output Bin Full
  2.3  Output Bin Jam

  3.1  Marker Toner Low
  3.2  Marker Toner Out
  3.3  Marker Toner Cartridge Missing

  4.1  Media Path Jam

  5.1  Cover Open
  5.2  Cover Closed after Open

  6.1  Printer Offline
  6.2  Printer Online after Offline
  6.3  Printer Reset
  6.4  Printer Power-Saver Mode Entry
  6.5  Printer Power-Saver Mode Exit
  6.6  Printer Idle
  6.7  Printer Currently Printing

Outstanding Issues
------------------

  - How to test these tough situations:

	* Removal of alerts when the table becomes full
	* Any of the Interpreter-related alerts

  - How are the data files to be named?  Need to take into consideration
    these things:

	* Product Id
	* Test Id
	* Before vs. After

  - Which software tool(s) to use for these tests?

  - Must take into account ALL the MIB objects pertaining to the
    Printer MIB.  In particular, we need to focus on the many related
    HR MIB variables.

  - Should we suggest that all vendors publicly provide a "list of
    supported alerts" similar to the way the MIDI spec is handled in
    the marketplace?

			      # # # # #