Of what quality be the UCSB resources evaluators?
A defense of UCSB's approach to evaluating network resources.
Pickens, J.R., "Of what quality be the UCSB resources evaluators?", RFC 545, DOI 10.17487/RFC0545, July 1973, <https://www.rfc-editor.org/info/rfc545>.
@misc{rfc545,
series = {Request for Comments},
number = 545,
howpublished = {RFC 545},
publisher = {RFC Editor},
doi = {10.17487/RFC0545},
url = {https://www.rfc-editor.org/info/rfc545},
author = {J.R. Pickens},
title = {{Of what quality be the UCSB resources evaluators?}},
pagetotal = 2,
year = 1973,
month = jul,
} RFC text
Network Working Group J. Pickens Request for Comments: 545 UCSB Computer Systems Laboratory NIC: 17791 23 July 1973 References: RFC 531,369. 519 OF WHAT QUALITY BE THE UCSB RESOURCE EVALUATORS? A Response to "Feast of Famine" In RFC 531, M.A. Padlipsky complains that the UCSB resource evaluators were derelict in not consulting the Resource Notebook for available documentation. In addition, Padlipsky equates the goals of the resource evaluators to the goals of the software repository advocaters. A misunderstanding exists and perhaps, with this note, may be cleared. To respond to Padlipsky's example of UCSB botching login attempts let me make two comments. First, more people than the resource evaluators were accessing the ARPANET. The group of evaluators, at least, knew the login procedure from the Resource Notebook. (By the way, we do have a Multics Programmers Manual.) Second, the OLS TELNET echoes no lower case, which can generate confusion. Even UCSB's technical liaison, after consulting the Resource Notebook, managed to botch his login. The first law of resource evaluation, at least for UCSB evaluators, is "read the Resource Notebook!" (RFC 369, incidentally, was based on a Resource Notebook that was barren compared to the notebook of today.) Questions left unanswered by the Notebook are resolved by accessing online documentation first at the NIC and second at the site being evaluated. If, after all this effort, questions still exist, then a consultant is contacted. Consultation may be either online or by telephone and may entail purchasing appropriate user manuals (for some of the resources we evaluated, no manuals existed). Our approach has been to consult the most publicly available documentation first. Only if the advertised paths fail do we resort to personal contact with a (busy) technical liaison. If technical liaisons wish to be consultants for uninitiated users and feel that this is their role we will gladly modify our behavior. There certainly is a meal, to use Padlipsky's analogy, of documentation already available on the Network. However, a meal is no good without silverware. Site specific and function specific MINIMANS (see RFC 369 and RFC 519) are attempts to provide this tableware. Our first-pass MINIMANS are available on request for those who would like to see what we are trying to do. Resource evaluators are concerned with much more than documentation. A closer reading of prior RFC's would have shown that we investigate dynamic phenomenon such as help facilities, online consultation, response time, reliability, and human engineering. We make suggestions for improvement. Indeed we see ourselves, at least for UCSB users, in the role of plain clothes inspector. We don't claim absolute efficiency but we do claim good intent and good results. We have spurred improvements at local as well as foreign network sites. We apologize to any we may have offended in the past with poor reviews. We are learning, continually, how best to say things in a constructive rather than destructive way. [ This RFC was put into machine readable form for entry ] [ into the online RFC archives by Javier Echeverria 2/98 ]
AI ELI5
On this page
§1The Critique and Misunderstanding
Mark Padlipsky, in RFC 531 ("Feast or Famine"), had criticized the UCSB resource evaluators for apparently failing to consult the Resource Notebook before conducting remote login tests. John Pickens responded on behalf of the UCSB Computer Systems Laboratory, explaining that Padlipsky had misunderstood both the evaluators' methods and their role in the network.
Pickens clarified that the UCSB evaluation group was well-versed in the Resource Notebook, and that other factors were at play in the login incidents Padlipsky cited. For instance, the OLS (Multics) TELNET server at that time did not echo lowercase characters, which created confusion even after consulting documentation. Padlipsky's own technical liaison had experienced the same difficulty.
Someone said the testers were careless. The testers said that was not true and that the problem was actually with the computer system, not with them.
§2The Evaluation Process
Pickens outlined the UCSB evaluators' systematic approach. The first step is to read the Resource Notebook. Questions that remain after consulting the Notebook are addressed by accessing online documentation, first at the NIC and then at the site being evaluated. Only if issues persist after consulting all available documentation do the evaluators contact a technical liaison, either online or by telephone, sometimes even purchasing appropriate user manuals when necessary.
This approach prioritizes the most publicly available documentation, resorting to personal contact only when advertised resources fail to answer questions. If technical liaisons prefer to serve as consultants to uninitiated users, the UCSB team is willing to adjust their approach accordingly.
They read the book, looked at the online manuals, then asked people if they still did not understand. They tried not to bother busy people unless they really had to.
§3Beyond Documentation: Investigating User Experience
Pickens stressed that resource evaluators investigate far more than just documentation availability. They examine dynamic phenomena such as the quality of help facilities, the responsiveness and reliability of online consultation, system response time, machine reliability, and human engineering aspects of the user interface. In effect, they serve as "plain clothes inspectors" on behalf of UCSB users.
The goal is not just to list what exists, but to identify areas for improvement and communicate constructive feedback to sites. The evaluators acknowledge they are still learning how best to phrase critiques in a constructive rather than destructive way.
They did not just read about how a computer system was supposed to work. They tested whether it actually worked well and whether it was easy to use.
§4Impact and Intent
Pickens concluded by asserting that while the UCSB evaluators do not claim absolute efficiency, they do claim good intent and good results. They have spurred improvements at both local and remote network sites through their work. The team apologized for any instances where their previous reviews may have caused offense and committed to continuing to improve their approach.
They said they were trying to help and that the sites they tested often made improvements because of their feedback.