WAIS over Z39.50-1988
The WAIS protocol implementation built on Z39.50-1988 information retrieval standard.
Pierre, M. St., Fullton, J., Gamiel, K., Goldman, J., Kahle, B., Kunze, J., Morris, H., and F. Schiettecatte, "WAIS over Z39.50-1988", RFC 1625, DOI 10.17487/RFC1625, June 1994, <https://www.rfc-editor.org/info/rfc1625>.
@misc{rfc1625,
series = {Request for Comments},
number = 1625,
howpublished = {RFC 1625},
publisher = {RFC Editor},
doi = {10.17487/RFC1625},
url = {https://www.rfc-editor.org/info/rfc1625},
author = {M. St. Pierre and J. Fullton and K. Gamiel and J. Goldman and B. Kahle and J. Kunze and H. Morris and F. Schiettecatte},
title = {{WAIS over Z39.50-1988}},
pagetotal = 7,
year = 1994,
month = jun,
} RFC text
Contents
Network Working Group M. St. Pierre
Request for Comments: 1625 WAIS, Inc.
Category: Informational J. Fullton
CNIDR
K. Gamiel
CNIDR
J. Goldman
Thinking Machines Corp.
B. Kahle
WAIS, Inc.
J. Kunze
UC Berkeley
H. Morris
WAIS, Inc.
F. Schiettecatte
FS Consulting
June 1994
WAIS over Z39.50-1988
Status of this Memo
This memo provides information for the Internet community. This memo
does not specify an Internet standard of any kind. Distribution of
this memo is unlimited.
1. Introduction
The network publishing system, Wide Area Information Servers (WAIS),
is designed to help users find information over a computer network.
The principles guiding WAIS development are:
1. A wide-area networked-based information system for searching,
browsing, and publishing.
2. Based on standards.
3. Easy to use.
4. Flexible and growth oriented.
From this basis, a large group of developers, publishers, standards
bodies, libraries, government agencies, schools, and users have been
helping further the WAIS system.
The WAIS software architecture has four main components: the client,
the server, the database, and the protocol. The WAIS client is a
user-interface program that sends requests for information to local
or remote servers. Clients are available for most popular desktop
environments. The WAIS server is a program that services client
requests, and is available on a variety of UNIX platforms. The
server generally runs on a machine containing one or more information
sources, or WAIS databases. The protocol, Z39.50-1988, is used to
connect WAIS clients and servers and is based on the 1988 Version of
the NISO Z39.50 Information Retrieval Service and Protocol Standard.
The goal of the WAIS network publishing system is to create an open
architecture of information clients and servers by using a standard
computer-to-computer protocol that enables clients to communicate
with servers.
WAIS development began in October 1989 with the first Internet
release occurring in April 1991. From the beginning, WAIS committed
to use the Z39.50-1988 standard as the information retrieval protocol
between WAIS clients and servers. The implementation is still in use
today by existing WAIS clients and servers resulting in over 50,000
users of Z39.50-1988 on the Internet.
2. Purpose
The purpose of this memo is to initiate a discussion for a migration
path of the WAIS technology from Z39.50-1988 Information Retrieval
Service Definitions and Protocol Specification for Library
Applications [1] to Z39.50-1992 [2] and then to Z39.50-1994 [3]. The
purpose of this memo is not to provide a detailed implementation
specification, but rather to describe the high-level design goals and
functional assumptions made in the WAIS implementation of Z39.50-
1988. WAIS use of Z39.50-1992 and Z39.50-1994 standards will be the
subject of future RFCs.
3. Historical Design Goals of WAIS
As an aid to understanding the original WAIS implementation and its
use of Z39.50-1988, the historical design goals of WAIS are presented
in this section. Included with each goal is a brief description of
the assumptions used to meet these design goals.
1. Provide users access to bibliographic and non-bibliographic
information, including full-text and images.
Because Z39.50-1988 grew out of the bibliographic community,
additional assumptions with the protocol were required to serve non-
bibliographic information. They were also necessary to serve
documents existing in multiple formats (e.g., rtf, postscript, gif,
etc.).
2. Keep the client/server interface simple and independent of
changes in the functionality of the server.
To achieve this, the text string entered by the user was transmitted
to the server without parsing the string into a Type-1 RPN (reverse-
polish notation) query, as is common for bibliographic applications.
Instead WAIS defined a new Type-3 query containing the text string.
In this way, knowledge of the Z39.50 Attributes supported by the
server was no longer required by the client or the user, as is true
of many existing Z39.50 implementations. In addition, the client
software did not require modification to support the evolving
functionality of the server.
3. Provide relevance feedback capability.
Relevance feedback is the ability to select a document, or portion of
a document, and find a set of documents similar to the selection.
WAIS included documents used in relevance feedback as part of the
Type-3 query.
4. Permit the server to operate in a stateless manner.
A WAIS server was designed to be "stateless", meaning that search
result sets were not stored by the server. In Z39.50 terms, the
server exercised its right to unilaterally delete a result set as
soon as it sent the search response. For this reason, the Present
Facility of Z39.50 was not used, and retrievals were performed using
the Search Facility. Relaxing this constraint in future
implementations may prove the most prudent path.
5. Provide the ability for a client to retrieve documents in
pieces.
Because retrieval of a portion of a document could be done several
ways with Z39.50-1988, specific assumptions were made to implement
this functionality. Accessing a portion of a document was required
for both retrieval and for relevance feedback.
6. Run over TCP.
The Z39.50-1988 standard was designed to run in the application layer
using the presentation services provided by the Open Systems
Interconnection (OSI) Reference Model. Due to the popularity of
TCP/IP and the Internet, WAIS was designed to run over TCP. Use of
Z39.50 over TCP is described in [4].
4. WAIS Implementation of Z39.50-1988
By working with the Z39.50 Implementors Group (ZIG), the WAIS
developers used a recommended subset of Z39.50-1988 and specific
assumptions to fulfill its requirements. Over time, many of these
requirements have then gone into the definition of subsequent
versions of Z39.50. As new requirements become apparent, WAIS will
document any additional assumptions and work with the ZIG in
developing extensions.
WAIS supported the Init and Search Facilities of Z39.50-1988. Both
search and retrieval were implemented using the Search Facility, as
described in this section.
Search was initiated by the client with a Search Request APDU
(Application Protocol Data Unit) using a Type-3 query. The query
contained two main fields:
1. The "seed words", or text, typed by the user.
2. A list of document objects, where a document object is a
full document, or portion thereof, to be used in relevance
feedback. Each document object contains a document
identifier (Doc-ID) [5], type, chunk-code, and start and
end locations. The Doc-ID and type specify the location and
format, respectively, of the document. The chuck-code
determines the unit of measure for the start and end
locations. Examples of chunk-codes used include
byte, line, paragraph, and full document. If the chunk code
is a full document, the start and end locations are ignored.
A Search Response APDU returned by the server contained a relevance
ranked list of records, or WAIS Citations. A WAIS Citation refers to
a document on the server. Each WAIS Citation contains the following
fields:
1. Headline - a set of words that convey the main idea of the
document.
2. Rank - the numerical score of the document based on its
relevance to the query, normalized to a top score of 1000.
3. List of available formats - e.g. text, postscript, tiff, etc.
4. Doc-ID - the location of the document.
5. Length - the length of the document in bytes.
The number of WAIS Citations returned was limited by the preferred
message size negotiated during the Init.
Retrieval of a document was initiated by the client with a Search
Request APDU using a Type-1 query. The query contained up to four
terms:
1. Term: Doc-ID
Use Attribute: system-control-number code = "un"
Relation Attribute: equal code = "re"
2. Term: the requested document format
Use Attribute: data-type code = "wt"
Relation Attribute: equal code = "re"
3. Term: the start location
Use Attribute: paragraph, line, byte code = "wp", "wl",
"wb"
Relation Attribute: greater-than-or-equal code = "ro"
4. Term: the end location
Use Attribute: paragraph, line, byte code = "wp", "wl",
"wb"
Relation Attribute: less-than code = "rl"
Because full-text and images were often larger in size than the
receive buffer of the client, clients were designed to optionally
retrieve documents in chunks, specifying the start and end positions
of the chunk in the query. An example of a fully-specified retrieval
query is:
query = ( ( use = "un", relation = "re", term = <Doc-ID> )
AND
( use = "wt", relation = "re", term = postscript )
AND
( use = "wb", relation = "ro", term = 0 )
AND
( use = "wb", relation = "ro", term = 2000 )
)
A retrieval response was issued by the server with a Search Response
APDU. In this case a single record corresponding to the requested
document, or portion thereof, was returned in the specified format.
5. Security Considerations
Security issues are not discussed in this memo.
6. References
[1] National Information Standards Organization (NISO). American
National Standard Z39.50, Information Retrieval Service
Definition and Protocol Specifications for Library Applications,
New Brunswick, NJ, Transaction Publishers; 1988.
[2] ANSI/NISO Z30.50-1992 (version 2) Information Retrieval Service
and Protocol: American National Standard, Information Retrieval
Application Service Definition and Protocol Specification for
Open Systems Interconnection, 1992.
[3] Z39.50 Version 3: Draft 8", October 1993. Maintenance Agency
Reference: Z39.50MA-034.
[4] Lynch, C., "Using the Z39.50 Information Retrieval Protocol
in the Internet Environment", Work in Progress, November 1993.
[5] "Document Identifiers, or International Standard Book Numbers
for the Electronic Age", Brewster Kahle, Thinking Machines
Corporation, see URL=<ftp://wais.com/pub/protocol/doc-ids.txt>,
September 1991.
7. Authors' Addresses
Margaret St. Pierre
WAIS Incorporated
1040 Noel Drive
Menlo Park, California 94025
Phone: (415) 327-WAIS
Fax: (415) 327-6513
EMail: saint@wais.com
Jim Fullton
Clearinghouse for Networked Information
Discovery & Retrieval
3021 Cornwallis Road
Research Triangle Park, North Carolina 27709-2889
Phone: (919)-248-9247
Fax: (919)-248-1101
EMail: jim.fullton@cnidr.org
Kevin Gamiel
Clearinghouse for Networked Information
Discovery & Retrieval
3021 Cornwallis Road
Research Triangle Park, North Carolina 27709-2889
Phone: (919)-248-9247
Fax: (919)-248-1101
EMail: kevin.gamiel@cnidr.org
Jonathan Goldman
Thinking Machines Corporation
1010 El Camino Real, Suite 310
Menlo Park, California 94025
Phone: (415) 329-9300 x229
Fax: (415) 329-9329
EMail: jonathan@think.com
Brewster Kahle
WAIS Incorporated
1040 Noel Drive
Menlo Park, California 94025
Phone: (415) 327-WAIS
Fax: (415) 327-6513
EMail: brewster@wais.com
John A. Kunze
UC Berkeley
289 Evans Hall
Berkeley, California 94720
Phone: (510) 642-1530
Fax: (510) 643-5385
EMail: jak@violet.berkeley.edu
Harry Morris
WAIS Incorporated
1040 Noel Drive
Menlo Park, California 94025
Phone: (415) 327-WAIS
Fax: (415) 327-6513
EMail: morris@wais.com
Francois Schiettecatte
FS Consulting
435 Highland Avenue
Rochester, New York 14620
Phone: (716) 256-2850
EMail: francois@wais.com
AI ELI5
On this page
§1Wide Area Information Servers
WAIS (Wide Area Information Servers) was designed in the late 1980s to help users find and retrieve information across computer networks. The system consists of four main components: the client (user interface), the server (runs on the machine hosting databases), the database (the information source), and the protocol (Z39.50-1988, the communication standard). WAIS development began in October 1989 with the first Internet release in April 1991. By the time this memo was written in 1994, over 50,000 users were using WAIS clients and servers connected via Z39.50-1988.
The WAIS project was guided by four key principles: creating a wide-area networked information system, building on published standards rather than proprietary protocols, designing for ease of use for non-technical users, and supporting growth and flexibility. To meet these goals, WAIS developers worked closely with the Z39.50 Implementors Group (ZIG) to use a recommended subset of the Z39.50-1988 standard with specific assumptions.
WAIS is a networked search system that helps people find documents. Started in 1989, by 1994 it had tens of thousands of users, all using the same standard protocol to search different databases.
§2Design Goals and Simplifications
WAIS had to adapt Z39.50, a standard designed for librarians searching bibliographic databases, to serve non-bibliographic information in diverse formats (text, images, PostScript, GIF). The designers made several key simplifications. Instead of requiring users or clients to know the Z39.50 Attributes supported by each server, WAIS clients sent simple Type-3 queries containing plain text entered by the user. The server handled parsing and attribute mapping internally. This meant clients did not need modification when server functionality changed.
WAIS servers were designed as stateless, deleting search result sets immediately after returning them rather than keeping them on the server. Retrieval was done using the Search Facility instead of the separate Present Facility. WAIS also supported relevance feedback, allowing users to select a document or document portion and find similar documents. To support large documents, clients could retrieve documents in chunks by specifying byte, line, or paragraph boundaries.
WAIS made Z39.50 easier to use by hiding technical details from the user. Instead of learning special commands, you just type words. The system figures out what you mean and gives you ranked results.
§3Search and Retrieval Queries
A WAIS search session began with the client sending a Search Request APDU (Application Protocol Data Unit) containing a Type-3 query. This query had two main components: the seed words (plain text entered by the user) and an optional list of document objects for relevance feedback. Each document object included a Doc-ID (document identifier), type, chunk-code, and start/end locations, allowing the server to find similar documents to ones the user found relevant.
The server responded with a Search Response APDU containing WAIS Citations. Each citation included a headline (key words from the document), a relevance rank (numerical score normalized to 1000), a list of available formats (text, PostScript, TIFF, etc.), the Doc-ID, and document length. For document retrieval, the client sent a Type-1 query with up to four terms: the document identifier, the desired format, and optional start/end locations for chunked retrieval. The response contained the requested document or portion in the specified format.
When you search WAIS, you type words, and it sends back a ranked list of documents with headlines and scores showing how well each matches. If you want to read a document, you can ask for just part of it.
- Search Request with Type-3 query (seed words)
- Search Response with ranked WAIS Citations
- Retrieval Request with Type-1 query (Doc-ID, format, range)
- Retrieval Response with document chunk
§4Implementation Details
WAIS supported the Init and Search Facilities of Z39.50-1988, using TCP for network transport rather than the OSI presentation services assumed by the Z39.50 standard. The Type-3 query format allowed the client to avoid parsing or understanding Z39.50's complex Attribute system. Each query term could use specific attributes such as use="un" (system-control-number) with relation="re" (equal) for document IDs, use="wt" (data-type) for format selection, and use="wb", "wl", "wp" for byte, line, and paragraph positioning in chunked retrieval.
The preferred message size negotiated during initialization controlled how many WAIS Citations were returned per search response. Because clients had limited receive buffers, especially for full-text documents and images, WAIS designed chunked retrieval as a core feature rather than an afterthought. This design proved essential for practical deployment when moving large multimedia documents across bandwidth-limited 1990s networks.
WAIS uses special codes to tell the server what kind of search to do (looking for a document ID, a format type, or a byte range). The server and client negotiate how much data to send at once.
§5Purpose and Migration Path
This memo's primary purpose was to initiate discussion for a migration path from Z39.50-1988 to Z39.50-1992 and then to Z39.50-1994 as those standards evolved. The memo was not intended as a detailed implementation specification, but rather a high-level description of WAIS design goals and assumptions that justified its particular use of Z39.50-1988.
The report acknowledges that as Z39.50 evolved, WAIS would need to adapt. Future RFCs would document WAIS implementations of Z39.50-1992 and Z39.50-1994. Over time, many WAIS requirements did feed into subsequent Z39.50 versions as the standard became more general-purpose. The document also references the DOI (Document Identifier) specification, a standard for uniquely identifying documents in electronic form, which complemented WAIS's retrieval mechanism.
This memo explains how WAIS worked in 1994 and hints that it would upgrade as the Z39.50 standard improved. The goal was to keep WAIS current with international standards rather than drift into obsolescence.