Benchmarking Terms: Benchmarking Terminology for Network Interconnection Devices
A shared vocabulary so vendors measure and report network device performance in comparable ways.
Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, <https://www.rfc-editor.org/info/rfc1242>.
@misc{rfc1242,
series = {Request for Comments},
number = 1242,
howpublished = {RFC 1242},
publisher = {RFC Editor},
doi = {10.17487/RFC1242},
url = {https://www.rfc-editor.org/info/rfc1242},
author = {S. Bradner},
title = {{Benchmarking Terminology for Network Interconnection Devices}},
pagetotal = 12,
year = 1991,
month = jul,
} RFC text
Contents
Network Working Group S. Bradner, Editor
Request for Comments: 1242 Harvard University
July 1991
Benchmarking Terminology for Network Interconnection Devices
Status of this Memo
This memo provides information for the Internet community. It does
not specify an Internet standard. Distribution of this memo is
unlimited.
Abstract
This memo discusses and defines a number of terms that are used in
describing performance benchmarking tests and the results of such
tests. The terms defined in this memo will be used in additional
memos to define specific benchmarking tests and the suggested format
to be used in reporting the results of each of the tests. This memo
is a product of the Benchmarking Methodology Working Group (BMWG) of
the Internet Engineering Task Force (IETF).
1. Introduction
Vendors often engage in "specsmanship" in an attempt to give their
products a better position in the marketplace. This usually involves
much "smoke & mirrors" used to confuse the user. This memo and
follow-up memos attempt to define a specific set of terminology and
tests that vendors can use to measure and report the performance
characteristics of network devices. This will provide the user
comparable data from different vendors with which to evaluate these
devices.
2. Definition format
Term to be defined. (e.g., Latency)
Definition:
The specific definition for the term.
Discussion:
A brief discussion about the term, it's application
and any restrictions on measurement procedures.
Measurement units:
The units used to report measurements of this
term, if applicable.
Issues:
List of issues or conditions that effect this term.
See Also:
List of other terms that are relevant to the discussion
of this term.
3. Term definitions
3.1 Back-to-back
Definition:
Fixed length frames presented at a rate such that there
is the minimum legal separation for a given medium
between frames over a short to medium period of time,
starting from an idle state.
Discussion:
A growing number of devices on a network can produce
bursts of back-to-back frames. Remote disk servers
using protocols like NFS, remote disk backup systems
like rdump, and remote tape access systems can be
configured such that a single request can result in
a block of data being returned of as much as 64K octets.
Over networks like ethernet with a relatively small MTU
this results in many fragments to be transmitted. Since
fragment reassembly will only be attempted if all
fragments have been received, the loss of even one
fragment because of the failure of some intermediate
network device to process enough continuous frames can
cause an endless loop as the sender repetitively
attempts to send its large data block.
With the increasing size of the Internet, routing
updates can span many frames, with modern routers able
to transmit very quickly. Missing frames of routing
information can produce false indications of
unreachability. Tests of this parameter are intended
to determine the extent of data buffering in the
device.
Measurement units:
Number of N-octet frames in burst.
Issues:
See Also:
3.2 Bridge
Definition:
A system which forwards data frames based on information
in the data link layer.
Discussion:
Measurement units:
n/a
Issues:
See Also:
bridge/router (3.3)
router (3.15)
3.3 bridge/router
Definition:
A bridge/router is a network device that can selectively
function as a router and/or a bridge based on the
protocol of a specific frame.
Discussion:
Measurement units:
n/a
Issues:
See Also:
bridge (3.2)
router (3.15)
3.4 Constant Load
Definition:
Fixed length frames at a fixed interval time.
Discussion:
Although it is rare, to say the least, to encounter
a steady state load on a network device in the real
world, measurement of steady state performance may
be useful in evaluating competing devices. The
frame size is specified and constant. All device
parameters are constant. When there is a checksum
in the frame, it must be verified.
Measurement units:
n/a
Issues:
unidirectional vs. bidirectional
See Also:
3.5 Data link frame size
Definition:
The number of octets in the frame from the first octet
following the preamble to the end of the FCS, if
present, or to the last octet of the data if there
is no FCS.
Discussion:
There is much confusion in reporting the frame
sizes used in testing network devices or network
measurement. Some authors include the checksum,
some do not. This is a specific definition for use
in this and subsequent memos.
Measurement units:
octets
Issues:
See Also:
3.6 Frame Loss Rate
Definition:
Percentage of frames that should have been forwarded
by a network device under steady state (constant)
load that were not forwarded due to lack of
resources.
Discussion:
This measurement can be used in reporting the
performance of a network device in an overloaded
state. This can be a useful indication of how a
device would perform under pathological network
conditions such as broadcast storms.
Measurement units:
Percentage of N-octet offered frames that are dropped.
To be reported as a graph of offered load vs frame loss.
Issues:
See Also:
overhead behavior (3.11)
policy based filtering (3.13)
MTU mismatch behavior (3.10)
3.7 Inter Frame Gap
Definition:
The delay from the end of a data link frame as defined
in section 3.5, to the start of the preamble of the
next data link frame.
Discussion:
There is much confusion in reporting the between
frame time used in testing network devices. This
is a specific definition for use in this and subsequent
memos.
Measurement units:
Time with fine enough units to distinguish between
2 events.
Issues:
Link data rate.
See Also:
3.8 Latency
Definition:
For store and forward devices:
The time interval starting when the last bit of the
input frame reaches the input port and ending when
the first bit of the output frame is seen on the
output port.
For bit forwarding devices:
The time interval starting when the end of the first
bit of the input frame reaches the input port and
ending when the start of the first bit of the output
frame is seen on the output port.
Discussion:
Variability of latency can be a problem.
Some protocols are timing dependent (e.g., LAT and IPX).
Future applications are likely to be sensitive to
network latency. Increased device delay can reduce
the useful diameter of net. It is desired to
eliminate the effect of the data rate on the latency
measurement. This measurement should only reflect the
actual within device latency. Measurements should be
taken for a spectrum of frame sizes without changing
the device setup.
Ideally, the measurements for all devices would be from
the first actual bit of the frame after the preamble.
Theoretically a vendor could design a device that
normally would be considered a store and forward
device, a bridge for example, that begins transmitting
a frame before it is fully received. This type of
device is known as a "cut through" device. The
assumption is that the device would somehow invalidate
the partially transmitted frame if in receiving the
remainder of the input frame, something came up that
the frame or this specific forwarding of it was in
error. For example, a bad checksum. In this case,
the device would still be considered a store and
forward device and the latency would still be
from last bit in to first bit out, even though the
value would be negative. The intent is to treat
the device as a unit without regard to the internal
structure.
Measurement units:
Time with fine enough units to distinguish between
2 events.
Issues:
See Also:
link speed mismatch (3.9)
constant load (3.4)
back-to-back (3.1)
policy based filtering (3.13)
single frame behavior (3.16)
3.9 Link Speed Mismatch
Definition:
Speed mismatch between input and output data rates.
Discussion:
This does not refer to frame rate per se, it refers to
the actual data rate of the data path. For example,
an Ethernet on one side and a 56KB serial link on the
other. This is has also been referred to as the "fire
hose effect". Networks that make use of serial links
between local high speed networks will usually have
link speed mismatch at each end of the serial links.
Measurement units:
Ratio of input and output data rates.
Issues:
See Also:
constant load (3.4)
back-to-back (3.1)
3.10 MTU-mismatch behavior
Definition:
The network MTU (Maximum Transmission Unit) of the
output network is smaller than the MTU of the input
network, this results in fragmentation.
Discussion:
The performance of network devices can be significantly
affected by having to fragment frames.
Measurement units:
Description of behavior.
Issues:
See Also:
3.11 Overhead behavior
Definition:
Processing done other than that for normal data frames.
Discussion:
Network devices perform many functions in addition
to forwarding frames. These tasks range from internal
hardware testing to the processing of routing
information and responding to network management
requests. It is useful to know what the effect of
these sorts of tasks is on the device performance.
An example would be if a router were to suspend
forwarding or accepting frames during the processing
of large routing update for a complex protocol like
OSPF. It would be good to know of this sort of
behavior.
Measurement units:
Any quantitative understanding of this behavior is by
the determination of its effect on other measurements.
Issues:
bridging and routing protocols
control processing
icmp
ip options processing
fragmentation
error processing
event logging/statistics collection
arp
See Also:
policy based filtering (3.13)
3.12 Overloaded behavior
Definition:
When demand exceeds available system resources.
Discussion:
Devices in an overloaded state will lose frames. The
device might lose frames that contain routing or
configuration information. An overloaded state is
assumed when there is any frame loss.
Measurement units:
Description of behavior of device in any overloaded
states for both input and output overload conditions.
Issues:
How well does the device recover from overloaded state?
How does source quench production effect device?
What does device do when its resources are exhausted?
What is response to system management in overloaded
state?
See Also:
3.13 Policy based filtering
Definition:
Filtering is the process of discarding received
frames by administrative decision where normal
operation would be to forward them.
Discussion:
Many network devices have the ability to be
configured to discard frames based on a number
of criteria. These criteria can range from simple
source or destination addresses to examining
specific fields in the data frame itself.
Configuring many network devices to perform
filtering operations impacts the throughput
of the device.
Measurement units:
n/a
Issues:
flexibility of filter options
number of filter conditions
See Also:
3.14 Restart behavior
Definition:
Reinitialization of system causing data loss.
Discussion:
During a period of time after a power up or
reset, network devices do not accept and forward
frames. The duration of this period of unavailability
can be useful in evaluating devices. In addition,
some network devices require some form of reset
when specific setup variables are modified. If the
reset period were long it might discourage network
managers from modifying these variables on production
networks.
Measurement units:
Description of device behavior under various restart
conditions.
Issues:
Types:
power on
reload software image
flush port, reset buffers
restart current code image, without reconfuration
Under what conditions is a restart required?
Does the device know when restart needed (i.e., hung
state timeout)?
Does the device recognize condition of too frequent
auto-restart?
Does the device run diagnostics on all or some resets?
How may restart be initiated?
physical intervention
remote via terminal line or login over network
See Also:
3.15 Router
Definition:
A system which forwards data frames based on
information in the network layer.
Discussion:
This implies "running" the network level protocol
routing algorithm and performing whatever actions
that the protocol requires. For example, decrementing
the TTL field in the TCP/IP header.
Measurement units:
n/a
Issues:
See Also:
bridge (3.2)
bridge/router (3.3)
3.16 Single frame behavior
Definition:
One frame received on the input to a device.
Discussion:
A data "stream" consisting of a single frame can
require a network device to do a lot of processing.
Figuring routes, performing ARPs, checking
permissions etc., in general, setting up cache entries.
Devices will often take much more time to process a
single frame presented in isolation than it would if
the same frame were part of a steady stream. There
is a worry that some devices would even discard a single
frame as part of the cache setup procedure under the
assumption that the frame is only the first of many.
Measurement units:
Description of the behavior of the device.
Issues:
See Also:
policy based filtering (3.13)
3.17 Throughput
Definition:
The maximum rate at which none of the offered frames
are dropped by the device.
Discussion:
The throughput figure allows vendors to report a
single value which has proven to have use in the
marketplace. Since even the loss of one frame in a
data stream can cause significant delays while
waiting for the higher level protocols to time out,
it is useful to know the actual maximum data
rate that the device can support. Measurements should
be taken over a assortment of frame sizes. Separate
measurements for routed and bridged data in those
devices that can support both. If there is a checksum
in the received frame, full checksum processing must
be done.
Measurement units:
N-octet input frames per second
input bits per second
Issues:
single path vs. aggregate
load
unidirectional vs bidirectional
checksum processing required on some protocols
See Also:
frame loss rate (3.6)
constant load (3.4)
back-to-back (3.1)
4. Acknowledgements
This memo is a product of the IETF BMWG working group:
Chet Birger, Coral Networks
Scott Bradner, Harvard University (chair)
Steve Butterfield, independant consultant
Frank Chui, TRW
Phill Gross, CNRI
Stev Knowles, FTP Software, Inc.
Mat Lew, TRW
Gary Malkin, FTP Software, Inc.
K.K. Ramakrishnan, Digital Equipment Corp.
Mick Scully, Ungerman Bass
William M. Seifert, Wellfleet Communications Corp.
John Shriver, Proteon, Inc.
Dick Sterry, Microcom
Geof Stone, Network Systems Corp.
Geoff Thompson, SynOptics
Mary Youssef, IBM
Security Considerations
Security issues are not discussed in this memo.
Author's Address
Scott Bradner
Harvard University
William James Hall 1232
33 Kirkland Street
Cambridge, MA 02138
Phone: (617) 495-3864
EMail: SOB@HARVARD.HARVARD.EDU
Or, send comments to: bmwg@harvisr.harvard.edu.
AI ELI5
On this page
§1Why Benchmarking Terms Are Needed
RFC 1242 comes from the Benchmarking Methodology Working Group (BMWG) of the IETF. It begins with a frank complaint: vendors often engage in specsmanship, using much 'smoke and mirrors' to give their products a better position in the marketplace.
The memo and its follow-ups try to define a specific set of terminology and tests that vendors can use to measure and report the performance of network devices. The goal is to give the user comparable data from different vendors.
Imagine every toy shop measured its toy cars in different ways and then shouted that theirs was best. This memo says: let us all use the same ruler.
§2How Each Term Is Defined
Every term follows the same template. It has a Definition, a Discussion of the term and any limits on how to measure it, the Measurement units used to report it, a list of Issues that affect it, and a See Also list of related terms.
Some terms are about the kind of device being tested. A bridge forwards frames using data link layer information. A router forwards frames using network layer information, which implies running the routing algorithm and, for example, decrementing the TTL field. A bridge/router can act as either, depending on the protocol of each frame.
Each word gets the same little card: what it means, why it matters, how to count it, and which other cards to look at.
§3Frames, Load and Timing Terms
Several terms pin down exactly what is being counted. Data link frame size is the number of octets from the first octet after the preamble to the end of the FCS, if present, or to the last octet of data if there is no FCS. The memo notes that authors disagree on whether to include the checksum, so it picks one definition.
Inter Frame Gap is the delay from the end of one frame, as defined above, to the start of the preamble of the next. A Constant Load is fixed length frames at a fixed interval, and the memo admits that a steady load is rare in the real world but useful for comparing devices. Back-to-back means fixed length frames sent with the minimum legal separation for the medium, starting from an idle state. It is used to learn how much data buffering a device has, for example for NFS-style transfers of up to 64K octets that become many fragments on Ethernet, or routing updates that span many frames.
Before you time a runner you must say where the starting line is. These words say exactly where a frame starts and ends and how fast they are sent.
§4Throughput, Loss and Latency
Throughput is the maximum rate at which none of the offered frames are dropped by the device. It is reported as N-octet input frames per second or input bits per second, over an assortment of frame sizes, with separate figures for routed and bridged data in devices that do both. If frames carry a checksum, full checksum processing must be done.
Frame Loss Rate is the percentage of frames that should have been forwarded under steady load but were dropped for lack of resources. It is reported as a graph of offered load against frame loss.
Latency depends on the device type. For store and forward devices it runs from when the last bit of the input frame reaches the input port to when the first bit of the output frame is seen on the output port. For bit forwarding devices it runs from the end of the first input bit to the start of the first output bit. A 'cut through' device that starts sending early is still treated as store and forward, so its latency can even be negative.
Throughput is the fastest a conveyor belt can run without dropping any boxes. Frame loss is how many boxes fall off when you run it too fast. Latency is how long one box takes to get across.
| Term | What it measures | Measurement units |
|---|---|---|
| Throughput THIS RFC | Maximum rate with no offered frames dropped | N-octet input frames per second; input bits per second |
| Frame Loss Rate | Percentage of frames dropped under steady load for lack of resources | Percentage of N-octet offered frames dropped, as a graph of offered load vs frame loss |
| Latency | Delay through the device | Time with fine enough units to distinguish between 2 events |
§5How Devices Behave Under Stress
The remaining terms describe behavior rather than a single number. Link Speed Mismatch is a mismatch between input and output data rates, for example an Ethernet on one side and a 56KB serial link on the other, also called the 'fire hose effect'. MTU-mismatch behavior covers an output network with a smaller MTU than the input network, which forces fragmentation.
Overhead behavior is processing other than for normal data frames, such as handling routing updates or management requests. Overloaded behavior is what happens when demand exceeds resources, and the memo treats any frame loss as an overloaded state. Policy based filtering is discarding frames by administrative decision, Restart behavior is the data loss and downtime after a power up or reset, and Single frame behavior is how a device handles one frame in isolation, which can take much longer than the same frame in a steady stream because of work such as ARP and cache setup.
Some tests ask how a machine acts when things go wrong: when it is rebooting, when it is swamped, or when the first item arrives and nothing is ready yet.
§6Security Considerations
The memo states only that security issues are not discussed in it. It defines vocabulary and does not specify a protocol or an Internet standard.
This list of words is not about locks and keys, so the memo says nothing about safety.