RFC 1431 INFORMATIONAL DIRECTORY · EVALUATION · X.500

DUA Metrics (OSI-DS 33 (v2))

A scoring framework for evaluating X.500 Directory user interfaces

On rfc-editor.org ↗
PUBLISHED
February 1993
AUTHORS
P. Barker
STREAM
IETF
TOTAL POINT SECTIONS
16 areas
MAXIMUM SECTION POINTS
Up to 10 (varies per section)
BIND WEIGHT (SEARCHSTONES)
5 points
SUBTREE SEARCH WEIGHT (SEARCHSTONES)
5 points
TEST QUERIES
11 named queries against Paul Barker's directory entry
DOI
10.17487/RFC1431

RFC text

Contents
Original text · RFC Editor View on rfc-editor.org ↗
Network Working Group                                          P. Barker
Request for Comments: 1431                     University College London
                                                           February 1993


                              DUA Metrics

Status of this Memo

   This memo provides information for the Internet community.  It does
   not specify an Internet standard.  Distribution of this memo is
   unlimited.

Abstract

   This RFC is being distributed to members of the Internet community in
   order to solicit their reactions to the proposals contained in it.
   While the issues discussed may not be directly relevant to the
   research problems of the Internet, they may be interesting to a
   number of researchers and implementers.

   This document defines a set of criteria by which a DUA
   implementation, or more precisely a Directory user interface, may be
   judged.  Particular issues covered include terminal requirements;
   style of interface; target user; default object classes and attribute
   types; use of DAP; error handling.  The focus of the note is on
   "white pages" DUAs: this is a reflection of the current information
   base.  Nevertheless much of the document will be applicable to DUAs
   developed for other types of Directory usage.

   Please send comments to the author or to the discussion group <osi-
   ds@CS.UCL.AC.UK>.

Table of Contents

   1.  Overview................................................       2
   2.  General Information.....................................       3
   3.  Conformance to OSI Standards............................       5
       3.1    Directory protocols..............................       5
       3.2    Protocol stacks..................................       5
       3.3    Schema ..........................................       5
       3.4    DIT structure  ..................................       5
   4.  Conformance to Research Community Standards.............       6
   5.  The General Style of the DUA............................       6
   6.  Schema..................................................       7
       6.1    Object Classes and Attribute Types...............       7
       6.2    DIT structure....................................       8
   7.  Entering queries........................................       9
   8.  Strategy for locating entries...........................       9
   9.  Displaying results......................................      10
   10. Association Handling....................................      11
   11. Suitability for management..............................      12
   12. Query Resolution........................................      13
   13. International Languages.................................      16
   14. User Friendliness.......................................      16
   15. Operational Use.........................................      17
   16. Security Considerations.................................      19
   17. Author's Address........................................      19

1.  Overview

   The purpose of this document is to define some metrics by which DUA
   products can be measured.  It should be first be noted that the use
   of the term "DUA" is rather misleading.  There is an assumption here
   that the DUA is implemented correctly and is able to "talk" valid
   X.500 protocol: this is a sine qua non.  Instead, this document seeks
   to draw out the characteristics of Directory user interfaces.
   However, the term DUA is persisted with as it is used by most people
   when referring to Directory user interfaces.  The format of these DUA
   metrics is essentially a questionnaire which extracts a detailed
   description of a user interface.  DUAs come in very different forms.
   Many make use of windowing environments, offering a "high-tech" view
   of the Directory, while others are designed to work in a terminal
   environment.  Some interfaces offer extensive control over the
   Directory, and thus may be well-suited to Directory managers, while
   others are aimed more at the novice user.  Some interfaces are
   configurable to allow searches for any attribute in any part of the
   DIT, while others lack this generality but are focussed on handling
   the most typical queries well.  In many aspects, it is almost
   impossible to say that one DUA is better than other from looking at
   the responses to question in this document.  A flexible management
   tool will be better for management than a DUA aimed at servicing
   simple look-ups, and vice-versa.  Furthermore, in other areas, there
   are several radically different approaches to a problem, but it is
   not as yet clear whether one approach is better than another.  One
   example of this is the extent to which a DUA provides an abstraction
   of the underlying DIT hierarchy, either emphasising the world as a
   tree or trying to conceal this from the user.

   However, other aspects, such as whether the DUA can actually find the
   entries required, and if so, how quickly, can be directly measured in
   some way.  Throughout this document, some of the questions posed are
   annotated with a square-bracketed points score and an explanation as
   to how the points should be allocated.  For example, a question might
   be appended with "[2 if yes]", indicating score 2 points for an
   affirmative answer to that question.  These points scores should be
   collated in Table 1 at the end of the document, and this table
   constitutes a measure of the DUA. The metrics are on a section by
   section basis, which should help the reader who is seeking, for
   example, a DUA with good management capabilities which runs on a wide
   variety of platforms, to focus on the critical aspects of a DUA for
   the particular requirement.

2.  General Information

   This section contains general information about the implementation
   under discussion.

   1.  Name of the implementation ......................................

   2.  Version number of the DUA described in this document ............

   3.  Are further versions planned?  [3 if yes] .......................

   4.  Name and address of supplier or person to contact ...............
    ....................................................................
    ....................................................................
    ....................................................................
    ....................................................................
    ....................................................................
    ....................................................................

   5.  Describe the hardware and software platforms on which the DUA will
       run.  Some DUAs are split into a user interface part, and a DUA
       server part, communicating by means of a protocol.  If the DUA is
       of the type where the DUA protocol machinery and user agent are
       implemented in a single process, complete only the user interface
       section, and indicate "n/a" for the DUA server and
       communications protocol questions.

      (a)  User interface part [1 per platform, up to a maximum of 4]

           i.  Hardware (If appropriate, can summarise as, for example,

          ii.  O/S (state version if critical)

               A.  UNIX (be sure to indicate which flavour - e.g., SYSV,
                   BSD, SUNOS, etc) ....................................
                ........................................................

               B.  VMS) ................................................

               C.  MS-DOS ..............................................

               D.  MS-Windows ..........................................

               E.  Macintosh ...........................................

               F.  Other) ..............................................

      (b)  DUA server part (or n/a) ....................................

           i.  Hardware (If appropriate, can summarise as, for example,
               "generic UNIX platform", or "386 PC") ...............

          ii.  O/S (state version if critical)

               A.  UNIX (be sure to indicate which flavour - e.g., SYSV,
                   BSD, SUNOS, etc) ....................................

               B.  VMS) ................................................

               C.  MS-DOS ..............................................

               D.  Macintosh ...........................................

               E.  Other) ..............................................

         iii.  How does the user interface communicate with the DUA
               server?

               A.  Directory Assistance Service, as described in RFC1202
                ........................................................

               B.  DIXIE protocol, as described in RFC1249 .............

               C.  LDAP protocol, as described in Internet Draft OSI-DS
                   26

               D.  Other ...............................................

      (c)  Name any other software required to run the DUA which is not
           supplied with the operating system or with the DUA software
           itself.  Examples might include X.500 DAP libraries, or
           communications software .....................................

   6.  Is the software free?  If the DUA needs other packages, are these
       also freely available?  [3 if completely free] ..................
    ....................................................................

3.  Conformance to OSI Standards

3.1  Directory protocols

   7.  Please list all conformance testing work applied to the DUA
       implementation (here the term DUA is used correctly in the sense
       of the DUA protocol machinery) [2 if any conformance work has been
       done] ...........................................................
    ....................................................................

3.2  Protocol stacks

   For the next two questions, [2 per stack supported for up to 4
   stacks]

   8.  Which of the following transport and network layer protocols does
       the DUA support:

      (a)  TP.x over CONS (state transport class) ......................

      (b)  TP.4 over CLNS ..............................................

   9.  Does the DUA support other transport and "network" layer
       protocols?

      (a)  TP.x over RFC1006 over TCP/IP (state transport class) .......

      (b)  TP.x over X.25(1980) (state transport class) ................

      (c)  State any other options supported.  .........................

   10. Does the DUA also run over any lightweight stack?  If so, describe
       it with reference to the OSI seven layer model [3] ..............
    ....................................................................

3.3  Schema

   11. Does the DUA support the full schema in X.520 and X.521 (y/n)?
       (Omissions should be described in response to a later question) [2
       for full schema support] ........................................

3.4  DIT structure

   12. Does the DUA only follow object class hierarchies which conform to
       the suggested DIT structure in X.521?............................

4.  Conformance to Research Community Standards

   The COSINE and Internet Directory Pilots have agreed a set of
   extensions to the standard, which make for a more cohesive pilot.
   This section is about conformance to these extensions.

   13. Does the DUA fully support RFC1274, "The COSINE and Internet
       X.500 Schema" (y/n)?  (Omissions should be described in response
       to a later question) [2 for full support] .......................

   14. Can the DUA handle referrals whose network addresses conform to
       RFC1277, "Encoding Network Addresses to support operation over
       non-OSI lower layers"?  [2 if yes] .............................

   15. Does the DUA handle the Distinguished Name string syntax described
       in OSI-DS 23, "A String Representation of Distinguished Names"
       [2 if yes] ......................................................
    ....................................................................

   16. Does the DUA use the user-friendly naming query resolution
       described in OSI-DS 24, "Using the OSI Directory to achieve User
       Friendly Naming" [2 if yes] ....................................

   17. Does the DUA make use of the Quality of Service schema extensions
       described in OSI-DS 15, "Handling QOS (Quality of service) in the
       Directory" [2 if yes] ..........................................

5.  The General Style of the DUA

   18. Is this a "white pages" interface, designed to give access to
       information about people within organisations?  If not, state the
       types of information at which this interface is targetted .......
    ....................................................................

   19. If this is a white pages DUA, who is it principally designed to
       serve?  Indicate more than one of the following categories if
       appropriate (but please do not fill in so may categories as to
       hide due emphasis):  [mark allocated should be the highest for any
       single classification]

      (a)  The ordinary user, who has no understanding of X.500, the
           hierarchical DIT, the state of advancement of the pilot, etc.
           [10] ........................................................

      (b)  A secretary who wants to do telephone or room number look-ups
           within their department or organisation [8] .................

      (c)  A computer-literate user, who habitually uses a wide-range of

           network services [6] ........................................

      (d)  An organisation's (or department's) data manager [4] ........

      (e)  A Directory system manager [2] ..............................

   20. Which best describes the use the DUA makes of the user's terminal?

      (a)  Scrolling, line-mode interface ..............................

      (b)  Full screen, "vt100" style interface ......................

      (c)  X-Windows ...................................................

      (d)  MS-Windows ..................................................

      (e)  Macintosh ...................................................

      (f)  Other .......................................................

   21. Does the DUA tend to emphasise or de-emphasise the DIT hierarchy?
    ....................................................................

   22. Describe the interface in your own terms (up to about 50 words)
    ....................................................................
    ....................................................................
    ....................................................................
    ....................................................................
    ....................................................................
    ....................................................................
    ....................................................................

6.  Schema

6.1  Object Classes and Attribute Types

   Some DUAs are tightly focussed on answering particular queries:  for
   example, white pages look-ups for information about people.  Others
   offer more general capabilities.  Please answer this question
   accordingly.

   23. If the DUA has a tight focus, state:

      (a)  The target object classes ...................................
        ................................................................

      (b)  The default attribute types .................................
        ................................................................

      (c)  Other attribute types which may be configured.  This might be
           answered as, for example, "all barring photo and audio", or
           as a list of supported attribute types ......................
        ................................................................

   24. If the DUA has more general capabilities, state:

      (a)  State any object classes in X.521 which cannot be searched for
        ................................................................

      (b)  State any object classes in RFC1274 which cannot be searched
           for..........................................................
        ................................................................

      (c)  State any attributes in X.521 which cannot be displayed......
        ................................................................

      (d)  State any attributes in RFC1274 which cannot be displayed....
        ................................................................

6.2  DIT structure

   25. DUAs often have a default object class hierarchy (e.g., it might
       assume countries at the root of the DIT, organisations immediately
       under countries, and people somewhere under organisations.
       Describe the type of hierarchy which the DUA most closely accords
       to:

      (a)  Rigid .......................................................

      (b)  Rigid, but several hierarchies supported ....................

      (c)  Default hierarchy offered, but many hierarchies are supported
        ................................................................

      (d)  Default hierarchy offered, but DUA fully flexible ...........

      (e)  No default hierarchy, DUA fully flexible ....................

   26. If a default hierarchy is offered, please describe it ...........
    ....................................................................
    ....................................................................
    ....................................................................

   27. State any hierarchies, which are valid according to X.521's
       suggested DIT structure, but which cannot be queried (exclude
       hierarchies which cannot be queried because the DUA does not query
       for entries of particular object classes) .......................

    ....................................................................

7.  Entering queries

   The term "querying" is used here as a generic term for finding an
   entry, whether it be as a simple look-up, or the prelude to a
   modification operation.

   28. Which best describes the query entry style?

      (a)  Form filling (user responds to a set of prompts) ............

           i.  Query specified first, then resolved ....................

          ii.  Query entry and resolution mixed ........................

         iii.  Both modes possible .....................................

      (b)  Queries entered as "user-friendly names" ..................

      (c)  Querying is by "navigating" around the DIT, the user
           searching and selecting .....................................

      (d)  Other (please describe) .....................................
        ................................................................

8.  Strategy for locating entries

   A number of strategies are employed by DUAs to find the entry the
   user is looking for.  These have implications for user-friendliness
   and performance.  For example, an interface which makes extensive use
   of search operations may be excellent at finding entries, but at the
   cost of being intolerably slow.

   29. Which of the following strategies most closely accords with the
       behaviour of the DUA?

      (a)  The DUA always uses search operations to find entries .......

      (b)  The DUA offers users a list of entries, and invites the user
           to select from the list .....................................

      (c)  The DUA only tries read operations (i.e., the DN must be
           exactly right) ..............................................

      (d)  The DUA tries read operations first, then searches for
           something similar if no entry can be found ..................

      (e)  The DUA tries read operations first, then offers a list of
           possible entries if no entry can be found ...................

      (f)  User explicitly controls the X.500 operation which is invoked

      (g)  Other.  Please describe......................................
        ................................................................

   30. Does the DUA allow a user to "list" (either by the list
       operation or by a single level search operation) all the child
       entries of a node (notwithstanding administrative limits)?  .....

   31. Does the DUA follow aliases?  ...................................
       If so, does it do so:

      (a)  Always?  ....................................................

      (b)  Optionally?  ................................................

   32. Will the DUA optionally follow links to other entries by using
       attributes such as seeAlso and roleOccupant with a DN syntax?  ..

9.  Displaying results

   33. Are the strings used to describe attribute types freely and
       independently configurable?  [2] ................................

   34. Name any attribute types where the attribute values may be
       presented in local formats?  (For example, it may be possible to
       configure the display of telephone numbers so that local numbers
       are shown as extensions, rather than with the full international
       dialling code.)  [1 per attribute, up to a maximum of 4] ........
    ....................................................................

   35. Does the DUA allow for the display of more than one result at a
       time (showing attribute values other than the name of the entry)?
       If so, how many entries may be displayed in response to a single
       query?  [2 if feature provided] .................................
    ....................................................................

   36. Does the DUA support the notion of a quick synopsis, where a small
       core of attributes is retrieved initially, and a larger set is
       returned if required?  [2 if yes] ...............................

   37. What does the DUA do with attribute types it doesn't support in
       its sub-schema, but which have a standard syntax?  ..............

   38. What does the DUA do with attributes which have are not in its
       sub-schema, and which have a non-standard syntax?

      (a)  Ignore them?  ...............................................

      (b)  Tell user, but don't display?  ..............................

      (c)  Display hex BER encoded value?  .............................

      (d)  Display in some other format?  ..............................

10.  Association Handling

   This section is concerned with how a DUA handles its association with
   the Directory.

   39. How/where is the access point to the Directory configured?  If
       more than one method, indicate which ways are possible.  [1 per
       method, up to maximum of 3]

      (a)  In a system-wide tailor file ................................

      (b)  In a per user tailor file ...................................

      (c)  As a run-time command line argument .........................

      (d)  Other.  Please describe .....................................
        ................................................................

   40. Does the DUA allow for automatic connection to a back-up DSA if
       the access point DSA is unavailable?  [2 if yes] ................

   41. Can the DUA keep connections open to more than one DSA at a time?
       [1 if yes] ......................................................
    ....................................................................

   42. Does the DUA keep an idle connection open to the DSA(s).  If not,
       describe the timeout strategy.  [1 if yes] ......................
    ....................................................................

   43. Does the DUA handle referrals automatically?  [2 if yes] ........
       If not:  does the DUA handle referrals at all?  [1 if yes] ......

   44. Does the DUA make use of asynchronous operations?

      (a)  Does the DUA bind asynchronously?  [2 if yes]................

      (b)  Are the operations handled asynchronously?  .................

           If so, is this true for:

           i.  All operations?  [2 if yes] .............................

          ii.  Some operations?  [1 if yes] ............................

   45. Does the DUA use size and time limits by default?  [2 if no
       limits, or limits may be over-ridden] If so:  ...................

      (a)  What size limit is used?  ...................................

      (b)  What time limit is used?  ...................................

      (c)  Are these limits overridable?  ..............................

11.  Suitability for management

   This section is intended to establish the range of operations
   supported by the DUA and, in particular, whether it is suitable for
   management tasks.

   46. Is it possible to invoke all the operations in the Directory
       Abstract Service?  If not, say which operations it does use [2 if
       all] ............................................................
    ....................................................................

   47. Is the user given full control over the service controls?  If not,
       say which may be controlled, or none at all [2 if full, 1 if some
       control] ........................................................
    ....................................................................

   48. Is it possible to manage system attributes with the DUA? If so,
       indicate which DSA implementations for which this DUA provides
       management capabilities.  .......................................

      (a)  Knowledge [1]................................................

      (b)  Replication information [1] .................................

      (c)  Other .......................................................

   49. Access control notwithstanding, does the DUA allow the following?

      (a)  Attribute management [2 for all below, 1 for some]

           i.  Addition ................................................

          ii.  Modification ............................................

         iii.  Deletion ................................................

      (b)  Entry management [2 for all below, 1 for some]

           i.  Addition ................................................

          ii.  Modification ............................................

         iii.  Deletion ................................................

          iv.  Renaming ................................................

12.  Query Resolution

   This section discusses the process of query resolution.  While two
   DUAs may both be able to resolve a query using the same information,
   one may do so much more quickly than the other.  Some DUAs may be
   more "economic" in their use of DAP operations to achieve the same
   results.  Some DUAs may find the correct results even when the users'
   input corresponds rather weakly to Directory names.  Three aspects of
   query resolution are measured:

    o  Does the DUA actually find the required entry?

    o  If the required entry is found, how many entries were returned as
       well?

    o  How "expensive" was the query in terms of underlying X.500
       operations, whether the query was resolved successfully or not?

   The following set of queries might all conceivably be resolved such
   that the author's Directory entry be found.  The queries are split
   into 2 groups:  the first group SHOULD pose no difficulties for a
   reasonable DUA; the second group are more problematic.  In each case,
   award [2] marks if the query found the author's entry successfully.
   The expensiveness of each query should be measured using the
   following formula, which introduces the notion of SearchStones!  The
   SearchStone rating is calculated by adding together the total
   operations used in attempting to resolve a query, weighted thus:

    o  Bind [5]

    o  Read operation [1]

    o  List operation [2]

    o  Search single level for countries, organisations or
       localities [3]
    o  Search single level for organisational units, people or roles [3]

    o  Search subtree [5]

      Note: The single level searches have been separated into two
      categories in acknowledgement that certain types of search are
      much more likely to span multiple DSAs than others.  The
      weightings are the same for the moment because of the
      pervasiveness of the Quipu implementation, which replicates all
      sibling entries in a single DSA, whatever the level in the DIT.

   The notion of SearchStones merits some further explanation and the
   statement of some caveats.

   The idea is to give some broad brush view of the work being
   undertaken by a DUA to retrieve an entry.  There will be some
   correspondence between a low SearchStone rating and a DUA responding
   quickly, and vice-versa, although this correlation is not consistent,
   for reasons given below.  It would be desirable to be able to have
   some timing information for the resolution of queries, but such
   results would only be meaningful if the tests were for target entries
   widely distributed throughout the DIT. Maybe this is something for
   the future?  In the meantime it is worth noting some of the factors
   which militate against simple minded interpretation of the
   SearchStones.

    o  The DIT is not uniform, with the depth varying considerably

    o  While the DIT is currently mastered mostly by DSAs of a single
       implementation, this will be decreasingly the case, and other DSAs
       may have very different performance profiles.

    o  Different directory domains are already adopting different
       strategies on information replication with profound performance
       implications.

    o  No weighting is given to different search filters, or to boolean
       combinations of filters.

   While acknowledging the difficulty of the exercise, there are counter
   arguments:

    o  Some DUAs are better than others at finding the required results

    o  Some DUAs will get the required results more quickly than most

    o  DUA designers have to build DUAs in the knowledge that the DIT is
       heterogeneous with respect to DSA implementation and DIT structure

   One possible way forward would be to refine the test queries such
   that they better represented the diversity of the DIT. However, as a
   first step, the tests are restricted to queries which could
   reasonably be constructed as searches for the author's entry.  The
   author's entry is held in part of the DIT which is representative of
   much of the current DIT.  It is suggested that in order to normalise
   the tests as much as possible, that testing be performed by
   connecting to the target DSA directly.  The DSA's name is "cn=Vicuna,
   c=GB", and the addresses of the DSA may be found in the presentation
   address attribute for that entry.  Note that the SearchStone rating
   should be shown even for queries which cannot be resolved.

   First, the straightforward queries:

   50. NAME=Paul Barker, OU=Computer Science, O=University College
       London, C=GB

   51. NAME=Paul Barker, OU=Computer Science, O=UCL, C=GB

   52. NAME=Barker, OU=Computer, O=UCL, C=GB

   53. NAME=Barker, O=UCL, C=GB

   54. NAME=p barker, O=university college, C=GB

   55. NAME=paul b, OU=cs, O=university college, C=GB

   More difficult queries:

   56. NAME=p b, O=university college, C=uk

   57. NAME=Paul Barker, OU=Computer Networking, O=london college, C=GB

   58. NAME=Paul Baker (sic), OU=cs, O=ucl, C=Britain

   59. NAME=p baker (sic), O=UCL, C=England

   60. NAME=Paul Barker, OU=Directories, O=london, C=United Kingdom

   Other general questions:

   61. Will the DUA attempt a query of the form "Find all the Smiths in
       Britain"?  .....................................................

       If so, does it do it by:

      (a)  A single query under the country node?  .....................

      (b)  Multiple queries under all organisation nodes?  .............

   62. Does the DUA allow "hands-off" querying whereby the details of a
       query may be entered in one go, and the DUA attempts to resolve
       the query without any further user intervention?  ...............

13.  International Languages

   63. Does the DUA offer multi-lingual support.  If so:  ..............

      (a)  State which languages are already supported [1 per language up
           to a maximum of 3] ..........................................
        ................................................................

   64. Can the DUA handle national language characters not found in
       PrintableString?  [2 if yes] ....................................

14.  User Friendliness

   65. Is run-time help available?  [2 if yes] .........................
       If so:

      (a)  Is context-sensitive help available?  [1 if yes] ............

      (b)  How many screens/windows?  ..................................

      (c)  How many bytes of help information?  [2 if more than 5 Kbytes
           of text, 1 if more than 3 Kbytes] ...........................

   66. Are the error messages terse renderings of the X.500 service
       errors, or user-friendly!?  As an example, provide the error
       message displayed to the user if an administrative limit is
       exceeded.  [2 if user-orientated, 1 if administrator-orientated, 0
       if no message at all] ...........................................
    ....................................................................
    ....................................................................

   67. If modify operations are provided, is there support for editing
       the attributes correctly with the appropriate syntax (e.g., does
       the DUA guide the user that addresses are of up to 6 lines of up
       to 30 characters; what support is given for entering distinguished
       names) [2 for postal address support, 2 for DN support, 1 for any
       other support] ..................................................

   68. Is the user allowed to see what sort of entries are in the
       Directory if they are unable to find the entry they are looking
       for?  [1 if yes] ................................................

   69. Does the DUA allow automatic following of attributes with DN
       values, such as seeAlso and roleOccupant?  [1 if yes]............

15.  Operational Use

   The DUA exists.  But is there any evidence to suggest that it is a
   usable tool?

   70. Is this DUA widely in use?  [5 if used by more than 20 orgs, 3 if
       by more than 10 orgs, 2 if by more than 5 orgs, 1 if used
       operationally to provide a service anywhere] ....................

      (a)  Is this DUA in use anywhere in the COSINE/Internet Pilot?  ..
        ................................................................

      (b)  Is this DUA in use in any other major pilot?  ...............

      (c)  Is this DUA in use anywhere else operationally?  ............

   71. Has this DUA been assessed by groups outside of the software
       developers or providers?  .......................................

   72. If so, are the assessments public?  Please provide copies of these
       assessments if they are available ...............................

   __________________________________________________________
   |_____Section_____|_____Points____|______________________|
   |No._|Description_|Maximum_|Scored|______________________|
   |    |            |        |      |                      |
   |__2_|Gen_Info____|__10____|...___|__________n/a_________|
   |    |            |        |      |                      |
   |__3_|Conf_to_OSI_|__15____|...___|__________n/a_________|
   |    |Conf to Res |        |      |                      |
   |__4_|Comm_stds___|__10____|...___|__________n/a_________|
   |    |            |        |      |                      |
   |__5_|Gen_Style___|__10____|_...__|__________n/a_________|
   |    |            |        |      |                      |
   |__9_|Disp_Res____|__10____|_...__|__________n/a_________|
   |    |            |        |      |                      |
   |_10_|Assoc_hand._|__15____|_...__|__________n/a_________|
   |    |            |        |      |                      |
   |_11_|Man_cap_____|__10____|_...__|__________n/a_________|
   | 12 |Query res   |        |      |Search |No.  of other |
   |    |            |        |      |Stones |entries found |
   |____|Q._50_______|__2_____|_...__|_...___|:_...._....___|
   |    |            |        |      |       |              |
   |____|Q._51_______|__2_____|_...__|_...___|:_...._....___|
   |    |            |        |      |       |              |
   |____|Q._52_______|__2_____|_...__|_...___|:_...._....___|
   |    |            |        |      |       |              |
   |____|Q._53_______|__2_____|_...__|_...___|:_...._....___|
   |    |            |        |      |       |              |
   |____|Q._54_______|__2_____|_...__|_...___|:_...._....___|
   __________________________________________________________
   |_____Section_____|_____Points____|______________________|
   |No._|Description_|Maximum_|Scored|______________________|
   |    |            |        |      |       |              |
   |____|Q._55_______|__2_____|_...__|_...___|:_...._....___|
   |    |            |        |      |       |              |
   |____|Q._56_______|__2_____|_...__|_...___|:_...._....___|
   |    |            |        |      |       |              |
   |____|Q._57_______|__2_____|_...__|_...___|:_...._....___|
   |    |            |        |      |       |              |
   |____|Q._58_______|__2_____|_...__|_...___|:_...._....___|
   |    |            |        |      |       |              |
   |____|Q._59_______|__2_____|_...__|_...___|:_...._....___|
   |    |            |        |      |       |              |
   |____|Q._60_______|__2_____|_...__|_...___|:_...._....___|
   |    |            |        |      |                      |
   |_13_|Int_Lang____|__5_____|_...__|__________n/a_________|
   | 14 |User-fr     |        |      |                      |
   |    |            |        |      |                      |
   |    |Query DUA   |  10    | .... |         n/a          |
   |    |            |        |      |                      |
   |____|Modify_DUA__|__15____|_...__|__________n/a_________|
   |    |            |        |      |                      |
   |_15_|Op_use______|__5_____|_...__|__________n/a_________|

                           Table 1:  DUA Metrics

16.  Security Considerations

   Security issues are not discussed in this memo.

17.  Author's Address

   Paul Barker
   Department of Computer Science
   University College London
   Gower Street
   London
   WC1E 6BT
   United Kingdom

   Phone: +44 71 380 7366
   Fax:   +44 71 387 1397
   Email: P.Barker@cs.ucl.ac.uk

AI ELI5

On this page
PROBLEMX.500 Directory implementations varied widely in interface style, capabilities, and performance, making it hard to compare them or choose one for a particular use case.
MECHANISMThis RFC provides a detailed questionnaire and scoring system covering 16 areas: hardware platforms, conformance to standards, interface design, query resolution performance, user-friendliness, and operational deployment.
EVALUATION FRAMEWORKOrganizations can use the metrics to evaluate DUAs objectively, and DUA designers have clear criteria for comparing features. The metric system includes 'SearchStones'—a weighted formula measuring query efficiency.

§1Directory User Agents and the Evaluation Problem

A DUA (Directory User Agent) is a program that lets users query an X.500 Directory—an online directory service designed by the OSI standards body. A DUA is the user interface layer; it translates a user's human-friendly search into technical X.500 protocol operations and displays the results.

DUAs come in many forms: some are terminal-based line-mode interfaces, others are graphical (X-Windows, MS-Windows, Macintosh), some run on a single machine, others split into a user interface and a server communicating over a network. Some are narrowly focused on finding people's email addresses and phone numbers; others are full-featured tools for managing directory data.

The RFC acknowledges that there is no universal 'best' DUA: a management tool is better for administrators; a simple lookup tool is better for casual users. However, some aspects are objectively measurable: Can the DUA actually find the entry you are looking for? How quickly? How many operations does it need to perform?

This RFC provides a detailed questionnaire and scoring system to evaluate DUAs across multiple dimensions, enabling informed comparison.

Different phone-book programs are good for different jobs, but some are clearly faster or easier than others. This RFC is a test to measure how good each one is.

§2The Evaluation Framework

The metrics are organized into 16 sections, each with questions worth points. Total points indicate overall capability.

  • General Information: Name, version, supplier, hardware and OS platforms, free or commercial, required dependencies.
  • OSI Conformance: Whether the DUA supports standard X.500 directory protocols (DAP), protocol stacks (TP over CONS, TP.4 over CLNS, TP over TCP/IP), and the full schema from X.520/X.521.
  • Research Community Standards: Support for COSINE/Internet Pilot extensions (RFC 1274, RFC 1277, OSI-DS standards).
  • General Style: Is it a 'white pages' interface (for finding people) or another type? Who is the target user (ordinary person, secretary, computer-literate user, manager, system administrator)? What terminal technology (scrolling, full-screen vt100, X-Windows, MS-Windows, Macintosh)?
  • Schema and DIT Structure: What object classes and attributes does it support? Is the DIT hierarchy rigid or flexible?
  • Query Entry: How does the user enter a query? Form filling, 'user-friendly names,' navigating the tree, or other?
  • Entry Location Strategy: Does it use searches, offers a list to select from, uses read-only operations, or combines strategies?
  • Results Display: Configurable attribute labels, local formatting options, multi-entry display, summary mode for faster display.
  • Association Handling: How is the directory access point configured? Automatic backup DSA failover, multiple connections, idle timeout handling, referral following, asynchronous operations.
  • Management Capabilities: Can it perform all directory operations? Can it manage system attributes (knowledge, replication info)? Can it add/modify/delete entries?
  • Query Resolution Efficiency: Measured using 'SearchStones'—a weighted metric combining bind, read, list, and search operations.
  • International Languages: Multi-lingual support, non-ASCII character handling.
  • User Friendliness: Runtime help, context-sensitive help, user-friendly error messages, syntax guidance for editing, browse capability.
  • Operational Use: Is the DUA widely deployed? Which pilots and organizations use it? Independent assessments available?

The test has sections for hardware, features, speed, how you enter searches, how you see results, how easy it is to use, and whether real organizations actually use it.

§3SearchStones: Measuring Query Efficiency

SearchStones is a weighted formula that measures the cost of resolving a directory query. It is not a timing measurement; rather, it counts and weights the underlying X.500 operations the DUA must perform.

The weight scheme is:

  • Bind (connect to directory server): 5 points
  • Read operation (direct lookup with an exact distinguished name): 1 point
  • List operation (enumerate children of a node): 2 points
  • Search at one level for countries, organizations, or localities: 3 points
  • Search at one level for organizational units, people, or roles: 3 points
  • Subtree search (across multiple directory levels): 5 points

For each test query, the RFC specifies 11 example queries ranging from straightforward (e.g., 'NAME=Paul Barker, OU=Computer Science, O=University College London, C=GB') to difficult (partial names, misspellings, abbreviations like 'c=uk' instead of 'c=GB'). Each query is scored with points if it succeeds in finding the author's entry, and a SearchStone rating for efficiency.

Low SearchStones indicate efficient queries; high SearchStones indicate the DUA is doing extensive searching. There is no universal 'good' score because the underlying X.500 network and DSA implementations vary, but the metric allows comparison between DUAs on the same network.

The test measures how much work the phone-book program does to find someone: fewer operations are better, so different DUAs can be compared fairly.

§4Interface Design Choices

The RFC notes that DUAs make very different architectural choices, each with trade-offs:

  • Query entry: Form-filling is learnable but slower; 'user-friendly naming' is faster but harder for novices; navigating the DIT tree is intuitive for some, frustrating for others. There is no objectively best approach.
  • DIT representation: Some DUAs emphasize the hierarchical tree structure (good for administrators, confusing for casual users). Others hide the hierarchy and make the Directory appear like a flat database (good for casual users, limiting for complex queries).
  • Localization: DUAs can display attribute names and values in local languages and formats (e.g., showing phone numbers as extensions instead of full international dialing codes). This is highly valued but requires configuration per site.
  • Results presentation: Quick synopses (small set of attributes returned initially, larger set on demand) improve performance; multi-result display lets users compare entries.
  • Handling unknown attributes: When a DUA encounters an attribute in the directory that it doesn't recognize, it can ignore it, warn the user, or display the raw bytes. Different DUAs choose differently.

The RFC's approach to scoring reflects this diversity: points are awarded for capabilities, but points themselves do not determine whether a DUA is 'good' for a specific use case.

Programs can be fast but hard to use, or easy but slow. The test measures different qualities separately so you can pick what matters for your job.

§5Practical Deployment and Caveats

The RFC emphasizes that SearchStone ratings are not absolute performance measurements and should be interpreted with care. Factors affecting the real-world performance of a DUA include:

  • DIT structure and content: Real directories vary in depth, breadth, replication strategy, and attribute values. A DUA efficient in one directory may be inefficient in another.
  • DSA implementation diversity: At the time of writing, most DSAs (X.500 directory servers) were Quipu implementations, but this was changing. Different implementations have different performance characteristics.
  • Network topology and replication: Different directory domains use different replication strategies, affecting whether a query triggers a remote referral or can be answered locally.
  • Test query specificity: The RFC's test queries are limited to searching for a single author's entry (Paul Barker at University College London). The DIT structure around that entry is representative of much of the current directory, but not universal.

The RFC encourages testing directly against the target DSA to normalize results as much as possible. For broad comparisons across DUAs, the metrics are a useful starting point, but should be supplemented by real testing in your actual directory environment.

Organizations using the metrics should note that a high point score indicates broad capabilities, not necessarily suitability for their specific task. A DUA with lower management points may be ideal for a 'white pages' deployment; one with high management points may be overkill and harder to learn.

The measurements show what each program can do, but how fast it actually is depends on your specific phone book and how you set it up.

§6References

[1]CCITT, "The Directory: Overview of Concepts, Models and Services", X.500, 1988.
[2]CCITT, "The Directory: Abstract Service Definition", X.511, 1988.
[3]CCITT, "The Directory: Protocols and Services", X.519, 1988.
[4]Hardcastle-Kille, S., "A String Representation of Distinguished Names", OSI-DS 23, March 1990.
[5]RFC 1274Huizer, E., and P. Barker, "The COSINE and Internet X.500 Schema", RFC 1274, Deltis B.V., University College London, November 1991.November 1991