TCP/IP Bake Off: TCP and IP Bake Off
Testing procedures and scoring for early TCP and IP implementations.
Postel, J., "TCP and IP bake off", RFC 1025, DOI 10.17487/RFC1025, September 1987, <https://www.rfc-editor.org/info/rfc1025>.
@misc{rfc1025,
series = {Request for Comments},
number = 1025,
howpublished = {RFC 1025},
publisher = {RFC Editor},
doi = {10.17487/RFC1025},
url = {https://www.rfc-editor.org/info/rfc1025},
author = {J. Postel},
title = {{TCP and IP bake off}},
pagetotal = 6,
year = 1987,
month = sep,
} RFC text
Contents
Network Working Group J. Postel
Request for Comments: 1025 ISI
September 1987
TCP AND IP BAKE OFF
Status of This Memo
This memo describes some of the procedures, scoring, and tests used
in the TCP and IP bake offs held in the early development of these
protocols. These procedures and tests may still be of use in testing
newly implemented TCP and IP modules. Distribution of this memo is
unlimited.
Introduction
In the early days of the development of TCP and IP, when there were
very few implementations and the specifications were still evolving,
the only way to determine if an implementation was "correct" was to
test it against other implementations and argue that the results
showed your own implementation to have done the right thing. These
tests and discussions could, in those early days, as likely change
the specification as change the implementation.
There were a few times when this testing was focused, bringing
together all known implementations and running through a set of tests
in hopes of demonstrating the N squared connectivity and correct
implementation of the various tricky cases. These events were called
"Bake Offs".
An early version of the list of tests included here appears in IEN-69
of October 1978. A demonstration of four TCP implementations was
held at the Defense Communication Engineering Center in Reston,
Virginia on 4 December 1978, and reported in IEN-70 of December 1978.
A bake off of six implementations was held 27-28 January 1979 at
USC-Information Sciences Institute in Marina del Rey, California and
reported in IEN-77 of February 1979. And a distributed bake off was
held in April 1980 over the network and reported in IEN-145 of May
1980.
The following section reproduces (with very slight editing) the
procedure, tests, and scoring of the April 1980 Bake Off.
Procedure
This is the procedure for the TCP and IP Bake Off. Each implementor
of a TCP and IP is to perform the following tests and to report the
results. In general, this is done by using a test program or user
Telnet program to open connections to your own or other TCP
implementations.
Some test are made more interesting by the use of a "flakeway". A
flakeway is a purposely flakey gateway. It should have control
parameters that can be adjusted while it is running to specify a
percentage of datagrams to be dropped, a percentage of datagrams to
be corrupted and passed on, and a percentage of datagrams to be
reordered so that they arrive in a different order than sent.
Many of the following apply for each distinct TCP contacted (for
example, in the Middleweight Division there is a possibility of 20
points for each other TCP in the Bake Off).
Note Bene: Checksums must be enforced. No points will be awarded if
the checksum test is disabled.
Featherweight Division
1 point for talking to yourself (opening a connection).
1 point for saying something to yourself (sending and receiving
data).
1 point for gracefully ending the conversation (closing the
connection without crashing).
2 points for repeating the above without reinitializing the
TCP.
5 points for a complete conversation via the testing gateway.
Middleweight Division
2 points for talking to someone else (opening a connection).
2 points for saying something to someone else (sending and
receiving data).
2 points for gracefully ending the conversation (closing the
connection without crashing).
4 points for repeating the above without reinitializing the
TCP.
10 points for a complete conversation via the testing gateway.
Heavyweight Division
10 points for being able to talk to more than one other TCP at
the same time (multiple connections open and active
simultaneously with different TCPs).
10 points for correctly handling urgent data.
10 points for correctly handling sequence number wraparound.
10 points for correctly being able to process a "Kamikaze"
packet (AKA nastygram, christmas tree packet, lamp test
segment, et al.). That is, correctly handle a segment with the
maximum combination of features at once (e.g., a SYN URG PUSH
FIN segment with options and data).
30 points for KOing your opponent with legal blows. (That is,
operate a connection until one TCP or the other crashes, the
surviving TCP has KOed the other. Legal blows are segments
that meet the requirements of the specification.)
20 points for KOing your opponent with dirty blows. (Dirty
blows are segments that violate the requirements of the
specification.)
10 points for showing your opponents checksum test is faulty or
disabled.
Host & Gateway IP Division
25 points for doing fragmentation and reassembly.
15 points for doing loose source route option.
15 points for doing strict source route option.
10 points for doing return route option.
10 points for using source quench messages.
10 points for using routing advice messages.
5 points for doing something with the type of service.
5 points for doing something with the security option.
5 points for doing something with the timestamp option.
5 points for showing that a gateway forwards datagrams without
decreasing the time to live (showing a gateway is faulty).
5 points for showing that a gateway forwards datagrams with the
time to live equal zero (showing a gateway is faulty).
10 points for showing that a gateway or hosts checksum test is
faulty or disabled (showing a gateway is faulty).
Bonus Points
10 points for the best excuse.
20 points for the fewest excuses.
30 points for the longest conversation.
40 points for the most simultaneous connections.
50 points for the most simultaneous connections with distinct
TCPs.
Tests
The following tests have been identified for checking the
capabilities of a TCP implementation. These may be useful in
attempting to KO an opponent.
1. Single connection. Open & close a single connection many
times.
2. Multi connections. Open several connections
simultaneously. Two connections to the same socket
(i.e., a-b and a-c) check proper separation of data.
3. Half Open Connection. Open a connection, crash local TCP
and attempt to open same connection again.
4. Piggy-back Loop. Open connections via Telnet.
user telnet--->TCP--->IP--->net--->IP--->TCP--->server telnet
|
V
server telnet<---TCP<---IP<---net<---IP<---TCP<---user telnet
|
V
user telnet--->...
5. Maximum connections. Open connections between a pair of
TCP until refused or worse.
6. Refused connection. Open a connection to a non-accepting
socket, does it get refused?
7. Zero Window. Try to send data to a TCP that is presenting
a zero window.
8. Fire Hose. Make many connections to data source ports, or
connections to a data sink and send as fast as you can.
9. Urgent Test. Try to send data to a user program that only
receives data when in urgent mode.
10. Kamikazi Segment. Send and receive nastygrams. A
nastygram is a segment with SYN, EOL, URG, and FIN on and
carrying one octet of data.
11. Sequence Wraparound. Test proper functioning when sequence
numbers (a) pass 2**31 (i.e., go from plus to "minus") and
(b) pass 2**32 (i.e., go from 2**32-1 to 0).
12. Buffer size. With buffer size not equal to one, send data
in segments of various sizes, use urgent occasionally.
13. Send a nastygram into a half open connection when the
sequence number is about to wrap around.
New Ideas
The above tests check for basic operation and handling of some of the
tricky cases. They do not consider performance in any way, or check
to see if some of the recently developed ideas have been implemented.
New Mechanisms
1. The John Nagel Procedures (RFC-896).
2. The Van Jacobson Procedures (slow start, RTT measurements,
etc).
3. The SQuID Procedures (RFC-1016).
Performance Tests
Performance tests are difficult to specify because the results
depend so much on the state of the environment of the test.
Here are a few possibilities:
1. FTP Throughput: Send a 1 megabyte file to a locally nearby
machine on an Ethernet measuring the elapsed time.
2. FTP Throughput: Send a 1 megabyte file to a locally nearby
machine on an ARPANET measuring the elapsed time.
3. NETBLT Throughput: Send a 1 megabyte file to a locally
nearby machine on an Ethernet measuring the elapsed time.
4. NETBLT Throughput: Send a 1 megabyte file to a locally
nearby machine on an ARPANET measuring the elapsed time.
5. Character Test: Use a test program to send a character via
TCP to the Echo Server (RFC-862), time the round trip (from
the time the character is sent until the echo is returned
to the test program).
Appendix
For History Buffs Only:
The following item was in the original 1980 tests, but has been
moved to this appendix since it no longer applies.
10 points for correctly handling rubber baby buffer bumpers in
both directions (End of Letter sequence number adjustments).
AI ELI5
On this page
§1Early Protocol Testing and the Bake Off Concept
RFC 1025 describes procedures from testing events called "Bake Offs," held in the early days of TCP and IP development. At that time, specifications were still evolving and there were only a few implementations. The only practical way to determine if an implementation was correct was to test it against other implementations and see if they worked together.
Several such testing events were organized: a demonstration of four TCP implementations in December 1978, a bake off of six implementations in January 1979 at USC-ISI, and a distributed bake off over the network in April 1980. The April 1980 event produced procedures and tests that have remained useful for validating newly implemented TCP and IP modules.
When the Internet was being invented, different teams each built their own version of TCP and IP. They had contests called "Bake Offs" where all the versions tried to work together, to see which ones were right.
§2The Testing Framework: Three Divisions
The bake off testing framework was organized into four categories, with scoring that rewarded both basic capabilities and correct handling of tricky cases. The Featherweight Division tested local TCP functionality: opening a connection to yourself, sending data to yourself, gracefully closing, and repeating without reinitializing. It also tested basic operation through a test gateway.
The Middleweight Division repeated these tests but with another TCP implementation, adding connectivity tests between different systems. The Heavyweight Division tested advanced capabilities: simultaneous multiple connections, urgent data handling, sequence number wraparound, and especially the ability to handle "Kamikaze" packets—segments with many flags set at once, which would crash incorrect implementations.
Finally, the Host and Gateway IP Division tested IP features like fragmentation and reassembly, source routing options, type of service, security options, and proper handling of time-to-live fields.
Teams could earn points for different tricks: talking to themselves, talking to others, handling weird packets that would crash bad code, and testing the IP layer's special features.
§3Test Procedures and the Flakeway
The RFC includes a detailed test procedure where each implementor runs a specific set of tests and reports results. Most tests were done using a test program or Telnet to open connections to other TCP implementations on the network.
An important tool described in the procedures is the flakeway: a purposely unreliable gateway that could be adjusted while running to drop datagrams at a specified percentage, corrupt and pass on a percentage of datagrams, or reorder datagrams so they arrived in a different order than sent. This allowed test of how robustly implementations handled packet loss and out-of-order delivery.
A critical requirement was that checksums must be enforced in all tests. Implementations that disabled checksum testing received no points for any checksum-related tests, even if they had implemented other features correctly.
Testers used a fake broken network that could lose packets, scramble them, or send them in the wrong order. If your code could handle that, it was good.
§4Specific Test Cases
The RFC lists 13 specific tests that were used to probe for correct behavior. Test 1 opened and closed a single connection many times. Test 2 opened several connections simultaneously, checking that data stayed separated between different connections to the same socket. Test 3 simulated a TCP crash and restart on the same connection. Test 4 used a loop of Telnet connections through multiple TCP instances.
More challenging tests included Test 7 (sending data to a TCP with a zero-sized window), Test 8 (the "Fire Hose" test, opening many connections rapidly), Test 9 (handling urgent data), and Test 11 (sequence number wraparound at 2^31 and 2^32). Test 10, the Kamikaze Segment test, sent specially crafted packets with SYN, EOL, URG, and FIN flags simultaneously to check for crashes.
Tests tried simple things like opening and closing connections, then harder things like losing connections, sending way too fast, and sending weird packets with every flag turned on at once.
§5New Mechanisms and Performance Tests
The RFC notes that the basic tests check for correct operation and handling of tricky cases, but do not evaluate performance. It suggests new mechanisms that were being developed at the time, including the John Nagel Procedures (RFC 896) for congestion control, the Van Jacobson Procedures involving slow start and round-trip time measurement, and the SQuID Procedures (RFC 1016).
Performance tests described include FTP throughput over Ethernet and ARPANET, NETBLT throughput, and character-echo round-trip time measurement to the Echo Server (RFC 862). These tests were harder to specify because results depended heavily on network conditions, but they provided real-world measurement of TCP/IP efficiency.
Beyond just checking that things work, people wanted to measure how fast they went. They measured how long it took to send files and get answers back.