Request for Comments: 2967 Thinking Cat Enterprises
Category: Informational R. Hedberg
Catalogix
October 2000
TISDAG - Technical Infrastructure for
Swedish Directory Access Gateways
Status of this Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2000). All Rights Reserved.
Abstract
The strength of the TISDAG (Technical Infrastructure for Swedish
Directory Access Gateways) project's DAG proposal is that it defines
the necessary technical infrastructure to provide a single-access-
point service for information on Swedish Internet users. The
resulting service will provide uniform access for all information --
the same level of access to information (7x24 service), and the same
information made available, irrespective of the service provider
responsible for maintaining that information, their directory service
protocols, or the end-user's client access protocol.
Table of Contents
1.0 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1 Project Goal. . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Executive Summary of Technical Study Result . . . . . . . . . 5
1.3 Document Overview . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.0 Requirements. . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 End-User Requirements . . . . . . . . . . . . . . . . . . . . 8
2.2 WDSPs Requirements. . . . . . . . . . . . . . . . . . . . . . 8
2.3 DAG-System Requirements . . . . . . . . . . . . . . . . . . . 9
3.0 Functional Specification. . . . . . . . . . . . . . . . . . . 9
3.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 The DAG Core. . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3 Client Interface. . . . . . . . . . . . . . . . . . . . . . . 11
3.3.1 Acceptable User Input . . . . . . . . . . . . . . . . . . . 12
Supported Query Types. . . . . . . . . . . . . . . . . . . . . 12
Matching Semantics . . . . . . . . . . . . . . . . . . . . . . 12
Character Sets . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3.2 Data Output Spec. . . . . . . . . . . . . . . . . . . . . . 14
Schema Definition. . . . . . . . . . . . . . . . . . . . . . . 14
Referral Definition. . . . . . . . . . . . . . . . . . . . . . 14
Error conditions . . . . . . . . . . . . . . . . . . . . . . . 14
3.4 Directory Server Interface. . . . . . . . . . . . . . . . . . 14
4.0 Architecture. . . . . . . . . . . . . . . . . . . . . . . . . 15
4.1 Software Components . . . . . . . . . . . . . . . . . . . . . 15
4.1.1 Internal Communications . . . . . . . . . . . . . . . . . . 15
4.1.2 Referral Index. . . . . . . . . . . . . . . . . . . . . . . 15
4.1.3 DAG-CAPs. . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.1.4 DAG-SAPs. . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 Important Architectural Notes . . . . . . . . . . . . . . . . 17
4.2.1 2 Distinct Functions: Referrals and Chaining . . . . . . . 17
4.2.2 Limited Query and Response Semantics. . . . . . . . . . . . 17
4.2.3 Visibility. . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2.4 Richness of Query semantics . . . . . . . . . . . . . . . . 18
4.2.5 N+M Protocol Mappings . . . . . . . . . . . . . . . . . . . 18
4.2.6 DAG-CAPs and DAG-SAPs are completely independent of each
other. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2.7 The Role of the DAG-CAP . . . . . . . . . . . . . . . . . . 18
4.2.8 The Role of the DAG-SAP . . . . . . . . . . . . . . . . . . 19
4.2.9 DAG/IP is internal. . . . . . . . . . . . . . . . . . . . . 19
4.2.10 Expectations . . . . . . . . . . . . . . . . . . . . . . . 19
4.2.11 Future Extensions. . . . . . . . . . . . . . . . . . . . . 19
5.0 Software Specifications . . . . . . . . . . . . . . . . . . . 19
5.1 Notational Convention . . . . . . . . . . . . . . . . . . . . 19
5.2 DAG-CAP Basics. . . . . . . . . . . . . . . . . . . . . . . . 20
5.2.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . 20
5.2.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . 21
5.2.3 Error handling. . . . . . . . . . . . . . . . . . . . . . . 21
5.2.4 Pruning of results. . . . . . . . . . . . . . . . . . . . . 22
5.3 DAG-SAP Basics. . . . . . . . . . . . . . . . . . . . . . . . 22
5.3.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . 22
5.3.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . 23
5.3.3 Error handling. . . . . . . . . . . . . . . . . . . . . . . 23
5.3.4 Pruning of results. . . . . . . . . . . . . . . . . . . . . 23
5.3.5 Constraint precedence . . . . . . . . . . . . . . . . . . . 23
5.4 The Referral Index. . . . . . . . . . . . . . . . . . . . . . 24
5.4.1 Architecture. . . . . . . . . . . . . . . . . . . . . . . . 24
5.4.2 Interactions with WDSPs (CIP) . . . . . . . . . . . . . . . 24
5.4.3 Index Object Format . . . . . . . . . . . . . . . . . . . . 24
5.4.4 DAG-Internal I/O. . . . . . . . . . . . . . . . . . . . . . 24
5.4.5 The Index Server. . . . . . . . . . . . . . . . . . . . . . 24
5.4.6 Configuration . . . . . . . . . . . . . . . . . . . . . . . 25
5.4.7 Security. . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.5 Mail (SMTP) DAG-CAP . . . . . . . . . . . . . . . . . . . . . 25
5.5.1 Mail DAG-CAP Input. . . . . . . . . . . . . . . . . . . . . 26
5.5.2 Translation from Mail query to DAG/IP . . . . . . . . . . . 28
Querying the Referral Index. . . . . . . . . . . . . . . . . . 28
Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 29
5.5.3 Chaining queries in Mail DAG-CAP. . . . . . . . . . . . . . 31
5.5.4 Expression of results in Mail DAG-CAP . . . . . . . . . . . 31
5.5.5 Expression of Errors in Mail DAG-CAP. . . . . . . . . . . . 31
5.6 Web (HTTP) DAG-CAP. . . . . . . . . . . . . . . . . . . . . . 32
5.6.1 Web DAG-CAP Input . . . . . . . . . . . . . . . . . . . . . 32
5.6.2 Translation from Web query to DAG/IP. . . . . . . . . . . . 33
Querying a DAG-SAP Directly. . . . . . . . . . . . . . . . . . 33
Querying the Referral Index. . . . . . . . . . . . . . . . . . 33
Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 35
5.6.3 Chaining queries in Web DAG-CAP . . . . . . . . . . . . . . 36
5.6.4 Expression of results in Web DAG-CAP. . . . . . . . . . . . 36
text/html results. . . . . . . . . . . . . . . . . . . . . . . 36
application/whoispp-response Results . . . . . . . . . . . . . 37
5.6.5 Expression of Errors in Web DAG-CAP . . . . . . . . . . . . 37
Standard Errors. . . . . . . . . . . . . . . . . . . . . . . . 38
5.7 Whois++ DAG-CAP . . . . . . . . . . . . . . . . . . . . . . . 38
5.7.1 Whois++ DAG-CAP Input . . . . . . . . . . . . . . . . . . . 38
5.7.2 Translation from Whois++ query to DAG/IP. . . . . . . . . . 39
Querying the Referral Index. . . . . . . . . . . . . . . . . . 39
Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 39
5.7.3 Chaining in Whois++ DAG-CAP . . . . . . . . . . . . . . . . 40
5.7.4 Expression of results in Whois++. . . . . . . . . . . . . . 41
5.7.5 Expression of Errors in Whois++ DAG-CAP . . . . . . . . . . 41
5.8 LDAPv2 DAG-CAP. . . . . . . . . . . . . . . . . . . . . . . . 42
5.8.1 LDAPv2 DAG-CAP Input. . . . . . . . . . . . . . . . . . . . 42
5.8.2 Translation from LDAPv2 query to DAG/IP . . . . . . . . . . 44
Querying the Referral Index. . . . . . . . . . . . . . . . . . 44
Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 46
5.8.3 Chaining queries in LDAPv2 DAG-CAP. . . . . . . . . . . . . 48
5.8.4 Expression of results in LDAPv2 . . . . . . . . . . . . . . 48
5.8.5 Expression of Errors in LDAPv2 DAG-CAP. . . . . . . . . . . 48
5.9 LDAPv3 DAG-CAP. . . . . . . . . . . . . . . . . . . . . . . . 50
5.9.1 LDAPv3 DAG-CAP Input. . . . . . . . . . . . . . . . . . . . 50
5.9.2 Translation from LDAPv3 query to DAG/IP . . . . . . . . . . 51
Querying the Referral Index. . . . . . . . . . . . . . . . . . 51
Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 54
5.9.3 Chaining queries in LDAPv3 DAG-CAP. . . . . . . . . . . . . 55
5.9.4 Expression of results in LDAPv3 . . . . . . . . . . . . . . 55
5.9.5 Expression of Errors in LDAPv3 DAG-CAP. . . . . . . . . . . 56
5.10 Whois++ DAG-SAP. . . . . . . . . . . . . . . . . . . . . . . 57
5.10.1 Input. . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.10.2 Translation from DAG/IP to Whois++ query . . . . . . . . . 58
5.10.3 Translation of Whois++ results to DAG/IP . . . . . . . . . 58
5.11 LDAPv2 DAG-SAP . . . . . . . . . . . . . . . . . . . . . . . 59
5.11.1 Input. . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.11.2 Translation from DAG/IP to LDAPv2 query. . . . . . . . . . 59
5.11.3 Translation of LDAPv2 results to DAG/IP. . . . . . . . . . 61
5.12 LDAPv3 DAG-SAP . . . . . . . . . . . . . . . . . . . . . . . 62
5.12.1 Input. . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.12.2 Translation from DAG/IP to LDAPv3 query. . . . . . . . . . 62
5.12.3 Translation of LDAPv3 results to DAG/IP. . . . . . . . . . 64
5.13 Example Queries. . . . . . . . . . . . . . . . . . . . . . . 64
5.13.1 A Whois++ Query. . . . . . . . . . . . . . . . . . . . . . 65
What the Whois++ DAG-CAP Receives. . . . . . . . . . . . . . . 65
What the Whois++ DAG-CAP sends to the Referral Index . . . . . 65
What the Whois++ DAG-CAP Sends to an LDAP DAG-SAP. . . . . . . 65
5.13.2 An LDAP Query. . . . . . . . . . . . . . . . . . . . . . . 66
What the LDAP DAG-CAP Receives . . . . . . . . . . . . . . . . 66
5.13.3 What the LDAP DAG-CAP sends to the Referral Index. . . . . 67
What the LDAP DAG-CAP Sends to a Whois++ DAG-SAP . . . . . . . 67
What the LDAP DAG-CAP Sends to an LDAP DAG-SAP . . . . . . . . 68
6.0 Service Specifications. . . . . . . . . . . . . . . . . . . . 68
6.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6.2 WDSP Participation. . . . . . . . . . . . . . . . . . . . . . 69
6.3 Load Distribution . . . . . . . . . . . . . . . . . . . . . . 69
6.4 Extensibility . . . . . . . . . . . . . . . . . . . . . . . . 72
7.0 Security. . . . . . . . . . . . . . . . . . . . . . . . . . . 73
7.1 Information credibility . . . . . . . . . . . . . . . . . . . 73
7.2 Unauthorized access . . . . . . . . . . . . . . . . . . . . . 73
8.0 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 74
Appendix A - DAG Schema Definitions . . . . . . . . . . . . . . . 75
A.1 DAG Personal Information Schema (DAGPERSON Schema). . . . . . 76
A.2 DAG Organizational Role Information Schema (DAGORGROLE
Schema). . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Appendix B - Schema Mappings for Whois++ and LDAP . . . . . . . . 77
B.1 LDAP and the DAG Schemas. . . . . . . . . . . . . . . . . . . 78
B.2 Whois++ and the DAG Schemas . . . . . . . . . . . . . . . . . 81
Appendix C - DAG-Internal Protocol (DAG/IP) . . . . . . . . . . . 82
C.1 A word on the choice of DAG/IP. . . . . . . . . . . . . . . . 83
C.2 DAG/IP Input and Output -- Overview . . . . . . . . . . . . . 83
C.3 BNF for DAG/IP input and output . . . . . . . . . . . . . . . 83
C.3.1 The DAG/IP Input Grammar. . . . . . . . . . . . . . . . . . 84
C.3.2 The DAG/IP Response Grammar . . . . . . . . . . . . . . . . 87
C.4 DAG/IP Response Messages. . . . . . . . . . . . . . . . . . . 89
Appendix D - DAG/IP Response Messages Mapping . . . . . . . . . . 93
Appendix E - DAG CIP Usage. . . . . . . . . . . . . . . . . . . . 95
E.1 CIP Index Object. . . . . . . . . . . . . . . . . . . . . . . 95
E.2 CIP Index Object Creation . . . . . . . . . . . . . . . . . . 97
E.3 CIP Index Object Sharing. . . . . . . . . . . . . . . . . . . 98
E.3.1 Registration of Servers . . . . . . . . . . . . . . . . . . 98
E.3.2 Transmission of Objects . . . . . . . . . . . . . . . . . .100
Appendix F - Summary of Technical Survey Results. . . . . . . . .100
Appendix G - Useful References. . . . . . . . . . . . . . . . . .102
Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . .102
Authors' Addresses. . . . . . . . . . . . . . . . . . . . . . . .104
Full Copyright Statement. . . . . . . . . . . . . . . . . . . . .105
List of Tables
Table 3.1 DAG-supported queries . . . . . . . . . . . . . . . . .12
Table 5.1 Allowable Whois++ Queries . . . . . . . . . . . . . . .38
Table A.1 DAGPERSON schema attributes . . . . . . . . . . . . . .76
Table A.2 DAGORGROLE schema attributes. . . . . . . . . . . . . .77
Table B.1 Canonical DAGPERSON schema & LDAP inetorgPerson
attributes . . . . . . . . . . . . . . . . . . . . . . . . . .79
Table B.2 Reasonable Approximations for LDAP organizationalRole
attributes . . . . . . . . . . . . . . . . . . . . . . . . . .79
Table B.3 Canonical mappings for LDAP organizationalRole
attributes . . . . . . . . . . . . . . . . . . . . . . . . . .81
Table B.4 Canonical DAGPERSON schema & Whois++ USER attributes. .81
Table B.5 Canonical mappings for Whois++ ORGROLE attributes . . .82
Table C.1 List of system response codes . . . . . . . . . . . . .90
Table D.1 LDAPv2/v3 resultcodes to DAG/IP response codes
mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . .93
Table D.2 Mapping from DAG/IP response codes to LDAPv2/v3
resultcodes. . . . . . . . . . . . . . . . . . . . . . . . . .94
Table D.3 Mapping between DAG/IP and Whois++ response codes . . .94
Table F.1 Summary of TISDAG Survey Results: Queries . . . . . . 101
Table F.2 Summary of TISDAG Survey Results: Operational
Information. . . . . . . . . . . . . . . . . . . . . . . . . 101
1.0 Introduction
1.1 Project Goal
The overarching goal of this project is to develop the necessary
technical infrastructure to provide a single-access-point service for
searching for whitepages information on Swedish Internet users. The
service must be uniform for all information -- the same level of
access to information (7x24 service), and the same whitepages
information made available, irrespective of the service provider
responsible for maintaining that information.
1.2 Executive Summary of Technical Study Result
The strength of the TISDAG project's DAG proposal is that it defines
the necessary technical infrastructure to provide a single-access-
point service for information on Swedish Internet users. The
resulting service will provide uniform access for all information --
the same level of access to information (7x24 service), and the same
information made available, irrespective of the service provider
responsible for maintaining that information, their directory service
protocols, or the end-user's client access protocol.
Instead of requiring centralized mirroring of complete information
records from Swedish directory service providers, the DAG system uses
a well-defined index object summary of that data, updated at the
directory service provider's convenience. When an end-user queries
the DAG, the referral information is used (by the end-user's
software, or by a module within the DAG, as appropriate) to complete
the final query directly at the directory service provider's system.
This ensures that the end-user gets the most up-to-date complete
information, and promotes the directory service provider's main
interest: its service. The architecture of the DAG itself is very
modular; support for future protocols can be added in the operational
system.
1.3 Document Overview
This document is broken into 5 major sections:
Requirements: As a service, the DAG system will have several
different types of users. In order to be successful, those users'
needs (requirements) must be met. This in turn defines certain
constraints, or system requirements, that must be met. This section
aims to capture the baseline requirement assumptions to be addressed
by the system, and thus lays the groundwork on which the rest of the
proposed system is built.
Functional Specification Overview: Working from the users'
requirements, specific technologies and functionality details are
outlined to architect a system that will meet the stated
requirements. This includes a conceptual architecture for the
system. While the Requirements section outlines the needs the
different users have for the eventual DAG system, implementing and
providing the eventual service will entail constraints or conditions
that need to be met in order to be able to participate in the overall
system.
Architecture: Once the system has been defined conceptually, a
proposed software architecture is specified to produce the desired
functionality and meet the stated requirements.
Software Specifications: This section provides the specifications for
software components to meet the architecture described above.
Service Specifications: Once the software has been designed, the
success of the DAG system will rest on its operational
characteristics. Details of service requirements are given in this
section.
1.4 Terminology
DAG-CAP: Client Access Point -- point of communication between
client-access software and the DAG system.
DAG-System: The Directory Access Gateway system resulting from the
TISDAG project. A collection of infrastructural software and
services for the purpose of providing unified access to Swedish
whitepages information.
DAG/IP: DAG-Internal Protocol -- communication protocol used between
software components of the DAG.
End-User: People performing White Pages searches and look-ups (via
various forms of client software).
DAG-SAP: Service Access Point -- point of communication between the
DAG and WDSP software.
WDSP: Whitepages Directory Service Provider -- ISPs, companies, or
other interested entities.
Whitepages Information: Collected information coordinates for
individual people. This typically includes (but is not limited to) a
person's name, and e-mail address.
2.0 Requirements
There are 2 primary classes of users for the proposed Whitepages
directory access gateway:
- End-users
- WDSPs
As outlined below, needs of each of these user classes imposes a set
of constraints on the design of the DAG system itself. Some of the
requirements shown below are assumed starting criteria for the DAG
service; others have been derived from data collected in the
Technical Survey or other expertise input.
2.1 End-User Requirements
The End-User is to be provided with a specific set of search types:
Name
Name + Organization
Role + Organization
Name + Locality
Name + Organization + Locality
Role + Organization + Locality
The search results will, if available, include the following
information for each "hit":
- Full name
- E-mail address
- Role
- Organization
- Locality
- Full address
- Telephone numbers
Access to the service must be available through reasonable and
current protocols -- such that directory-service-aware software can
make use of it seamlessly, and there are no reasonable technological
impediments to making this service useful to all Swedish Internet
users.
Following on that, its responses are expected to be timely; a
standard search should not take more time than the average access to
a web-server.
2.2 WDSPs Requirements
Given that the WDSPs that participate in this service are already in
the business of providing a service of whitepages information, they
have certain requirements that must be respected in order to make
this a successful and useful service to all concerned.
The DAG system must provide reasonable assurances of data integrity
for WDSPs; the information the End-User sees should correspond
directly to that provided by the WDSPs. The DAG system should be
non-preferential in providing whitepages information -- the service
is to the End-User, and the source of whitepages information should
not influence the search and information presentation processes.
The DAG system must be able to reflect information updates within a
reasonable time after receipt from WDSPs; on the flip side, while the
DAG system will function best with regular updates from WDSPs, the
update and participation overhead for WDSPs should be held within
reasonable bounds of what the WDSP should do to support regular
access to its information.
Furthermore, given that WDSPs provide directory service information
with an eye to value-added service, wherever possible End-Users
should be redirected to the WDSP responsible for individual directory
service entries for final and further information.
2.3 DAG-System Requirements
In order to address the requirements of End-Users and WDSPs, the DAG
system itself has certain design constraints that must be taken into
account.
The system must be implementable/operational by Dec 31/98 -- which
implies that it must be designed and constructed with already extant
technologies.
The System will have certain requirements for participation -- e.g.,
7x24 WDSP availability.
In terms of scaling, the system should be able to handle 8M records
at the outset, with a view to handling larger information systems in
the future.
The system must also be capable of extension to other, related
applications (e.g., serving security certificate information).
3.0 Functional Specification
In the TISDAG pilotservice we have decided to apply some limitations
as to what is specified for the DAG/IP. These limitations are
presented in this text in the following manner:
TISDAG: This is a TISDAG comment
3.1 Overview
The conceptual environment of the DAG system can be described in
three major components:
- client access software for end-users
- the DAG system core
- WDSP directory service software
This is illustrated in Figure 3.1
The DAG (Directory Access Gateway) is the infrastructural core of the
service; it maintains the necessary data and transformation
facilities to permit the smooth connection of diverse directory
service Client Software to the existing WDSPs' directory servers.
The key challenges in designing this portion of the system are:
Quantity of data -- the quantity of whitepages information that will
be made available, and diversity of its sources (different WDSPs)
introduce challenges in terms of finding a structure that will allow
efficient searching, and facilitate the timeliness of updating the
necessary information.
Multiplicity of access protocols -- in order to support the use of
existing whitepages-aware software with a minimum of perturbation,
the DAG system will have to present a uniform face in several
different access protocols, each with its own information search and
representation paradigm.
This specification will outline the following areas:
- the functioning of the DAG core itself
- the interface between the DAG core and End-Users' Directory Service
Access software
- the interface between the DAG core and Directory Services Servers
3.2 The DAG Core
In order to reduce the quantity of data the DAG itself must maintain,
and to keep the maintenance of the whitepages information as close as
possible to the source of information (the WDSPs themselves), the DAG
will only maintain index information and will use "query routing" to
efficiently refer End-User queries to WDSPs for search refinement and
retrieval of information. Although originally developed for the
Whois++ protocol, query routing is being pursued in a protocol-
independent fashion in the IETF's FIND WG, so the choice of this
approach does not limit the selection and support of whitepages
access protocols.
The DAG will look after pursuing queries for access protocols that do
not support referral mechanisms. In order to achieve the support of
multiple access protocols and differing data paradigms, the DAG will
be geared to specifically support a limited set of whitepages
queries.
+---------+ @
+ ->| | -+-
/|Protocol| | |
/ | / +---------+ / \
/ | "B"
+ | /
| |<-
+-------+ | |
O | | | |
-+- | |<--------->| |
| | | Protocol | |
/ \ | | "A" | |<-
+-------+ | |Protocol
| | \
+ | "A" +---------+ @
\ | \ | | -+-
\ | ->| | |
\| +---------+ / \
+
The
End Client DAG Directory Directory
Users Software System Server Service
Core Software Providers
Figure 3.1 The role of the DAG system
3.3 Client Interface
The DAG will respond to End-User queries in
- e-mail (SMTP)
- WWW (HTTP)
- LDAPv2
- Whois++
- LDAPv3
The DAG will provide responses including the agreed-upon data. For
access protocols that can handle referrals, responses will be data
and/or referrals in that query protocol. These are Whois++ and
LDAPv3. N.B.: the LDAPv3 proposal defines a referral as a URL; no
limitation is placed on the access protocol. However it cannot be
assumed that all clients will be able to handle all access protocols,
so only referrals to LDAPv3 servers will be returned.
3.3.1 Acceptable User Input
User Input is defined in terms of
- Searchable Attributes
- Matching semantics
- Character sets
These, in conjunction with the DAG schema, defined in Appendix A,
form the basis of the required query expression. Individual queries
are discussed in more detail in the Client Access Point (DAG-CAP)
component descriptions for supported protocols.
Supported Query Types
The DAG system is designed to support fragment-matching queries on a
limited set of data attributes -- "Name", "Organizational Role",
"Organization", and "Locality". The selected permissible query
combinations of attributes are listed in Table 3.1. From the table
it can be seen that not all combinations of the three attributes are
supported -- only those that are needed for the desired
functionality.
Symbol Description
------- -----------
N Name
NL Name + Locality
NO Name + Organization
NOL Name + Organization + Locality
RO Role + Organization
ROL Role + Organization + Locality
Table 3.1 DAG-supported queries
The RO and ROL queries are separated from the rest as they are
searches for "virtual" persons -- roles within an organization (e.g.,
president, or customer service desk) for which one might want to find
contact information.
Matching Semantics
As befits the individual client query protocols, more string matching
expressions may be provided. The basic semantics of the DAG expect
the following to be available in all client access software (as
relevant):
- Full word, exact match
- Word substring match (E.g., "cat" would match "scatter")
- Case-sensitive and case-insensitive matching
TISDAG: LDAP/X.500, supports case-sensitivity as such but some of
the most used attributes, such as the commonName attribute, are
defined in the standard to be of the case-insensitive
attributetypes. The impact on the DAG system is that even if the
index collected from a LDAP/X.500 server might have upper and
lower case letters in the tokens, they can not be handled as such
since that would be inferring meaning in something which is
natively regarded as meaningless. The conclusion of the above is
that The Referral Index should be case-insensitive and case-
sensitivity should be supported by the SAPs if the native access
protocol supports it.
Character Sets
Wherever possible, the DAG System supports and promotes the use of
Unicode Version 2.0 for character sets (see [21]) specifically the
UTF-8 encoding (see Appendix A.2 of [21] or [20]) Accommodation is
made, where necessary, to support the deployed base of existing
software.
Specifically:
DAG/IP: All internal communications using the DAG/IP are carried out
in UTF-8.
TISDAG: not just UTF-8, but UTF-8 based on composed UNICODE
version 2 character encodings.
DAG-CAP input: Where specific access protocols permit selection of
character sets, DAG-CAPs must support UTF-8. They may additionally
support other anticipated character set encodings.
DAG-SAP communications with WDSPs: Where specific access protocols
permit selection of character sets, DAG-SAPs must support UTF-8 and
use UTF-8 whenever the remote WDSP supports it. They may
additionally support other character set encodings.
CIP Index Objects: The Index Objects supplied by the WDSPs to the DAG
system shall contain data encoded in UTF-8.
TISDAG: The same limitation as for DAG/IP, that is the basic data
should be UTF-8 encoded composed UNICODE version 2 character
encodings.
3.3.2 Data Output Spec
Schema Definition
The schema used for the DAG service is defined in Appendix A. This
is a very basic information schema, intended to carry the necessary
information for the DAG service, and not more. Although generic
"whitepages" schema definitions do exist the more sophisticated and
detailed the information presentation, the more difficult it is to
map the schema seamlessly across protocols of different paradigms.
Thus, the "KISS" ("Keep it simple, sir") principle seems appropriate
here.
Individual DAG-CAPs define how they express this schema.
Referral Definition
For client access protocols that make use of the concept of
referrals, DAG-CAP definitions will define the expression of
referrals in those protocols. The DAG/IP defines the expression of
referrals (see Appendix C).
Error conditions
Each DAG-CAP may provide more detailed error messages, but will
define minimally the support for the following error conditions:
- unrecognized query
- too many hits
Apart from these errors, the DAG-CAP may choose to refuse a query by
redirecting the end-user to a different DAG-CAP of the same protocol.
3.4 Directory Server Interface
The DAG will use the Common Indexing Protocol (CIP) server-server
protocol to obtain updated index objects from WDSPs. For query-
routing purposes, WDSPs are expected to provide Whois++, LDAPv2 or
LDAPv3 interface to their data (although their preferred access may
be something completely different). N.B.: In the responses from the
technical survey, all respondents currently provide access to their
service in one of these protocols.
In order to provide a useful and uniform service, WDSPs are expected
to provide 7x24 access to their whitepages information. WDSPs are
also expected to implement operations, administration, maintenance,
and provisioning processes designed to minimize service down time for
both planned and unplanned administration and maintenance activities.
4.0 Architecture
4.1 Software Components
The conceptual architecture of the DAG is represented in Figure 4.1.
General architectural specifications are described below, followed by
individual component specifications Sections 5.5 through 5.12.
4.1.1 Internal Communications
Communications between components of the DAG will be by TCP/IP
connections, using the DAG-Internal Protocol (DAG/IP). DAG/IP is
used by DAG-CAPs to communicate with the Referral Index and DAG-SAPs.
Thus, the DAG/IP defines
- the DAG-CAPs' range of query ability in the Referral Index (to
gather referrals in response to the end-user's requests)
- the responses (and their formats) of the Referral Index to the
DAG-CAP requests
- the DAG-CAPs' range of query ability to the DAG-SAPs for pursuing
referrals when the DAG-CAP needs to do chaining for the client
access software
- the responses (and their formats) of the DAG-SAPs to the DAG-CAPs.
The detail of the planned DAG/IP is given in Appendix C. The detail
of the DAG-CAP--Referral Index and DAG-CAP--DAG-SAP interactions is
given in the definitions of individual DAG-CAPs and DAG-SAPs, below
(Sections 5.5 through 5.12).
4.1.2 Referral Index
The Referral Index is responsible for maintaining the index of WDSP
information, and providing a list of reasonable referrals in response
to DAG-CAP search requests. These "referrals" provide pointers to
identify WDSPs that may have information that matches the end-user's
query.
4.1.3 DAG-CAPs
Individual DAG-CAPs are responsible for providing a particular client
access protocol interface to the DAG service. DAG-CAPs receive end-
user queries in a particular query access protocol, convert the
request into a query for the Referral Index ( i.e., expressed in
DAG/IP), and then convert the Referral Index's response into a form
that is appropriate for the client access protocol. This may mean
passing back the referrals directly, calling on DAG-SAPs to do the
work of translating the referral into results ("chaining"), or a
combination of both.
+-------------------------------------+
|+====+ |
HTTP <-->+| |<------+ (Full chaining) |
|| | | |
|+====+ | |
| | +----+|
| | Referral-->| ||
| | Result <--| |+<--> Whois++
| | +----+|
|+====+ | |
SMTP <-->+| |<------+ (Full chaining) |
|| | | |
|+====+ | |
| | +----+|
| | Referral-->| ||
| | Result <--| |+<--> LDAPv2
| | +----+|
|+====+ | |
Whois++<-->+| |<------+ (Chain LDAPv2/3) |
|| | | |
|+====+ | |
| | +----+|
| | Referral-->| ||
| | Result <--| |+<--> LDAPv3
| | +----+|
|+====+ | |
LDAPv2 <-->+| |<------+ (Full chaining) |
|| | | |
|+====+ | |
| | |
|+====+ | |
LDAPv3 <-->+| |<------+ (Chain Whois++) |
|| | | |
|+====+ | |
| | |
| v |
| +-----------------------+ |
| | Referral Index |<---------------> Common
| | | | Indexing Protocol
| +-----------------------+ | (CIP)
+-------------------------------------+
All internal communications are in DAG/IP.
Figure 4.1 Conceptual Architecture of the DAG
4.1.4 DAG-SAPs
Individual DAG-SAPs are called upon (by DAG-CAPs) to take DAG-
generated referrals and pursue them -- issuing the indicated query at
the specified WDSP service. Results from individual WDSPs are
converted back into DAG/IP-specific format for the DAG-CAP that made
the request. Each DAG-SAP is responsible for handling referrals to
WDSPs of a particular protocol (e.g., LDAPv2, Whois++, etc).
4.2 Important Architectural Notes
This section notes some of the thinking that has driven the
architectural and software design specification for the DAG system.
This helps to provide the context in which to understand the software
specifications that follow, and should give clues for the eventual
extension of the DAG system. This section also acts, in some ways,
as an FAQ (Frequently Asked Questions) section, as the content is
shaped by questions received during the tech spec development phase.
It attempts to illuminate context that may not otherwise be apparent
on a first reading of the software specifications.
4.2.1 2 Distinct Functions: Referrals and Chaining
At all times, it must be kept in mind that the primary function of
the DAG system is to provide users with referrals to WDSP services
that may have the information they seek. Since it is the case that
not all supported client protocols can handle referrals, the DAG
system also provides a chaining service to pursue referrals that the
user's client software cannot handle itself. This chaining service
does attempt to match the user's query against data from WDSPs, but
this is to be seen as a secondary, or support function of the DAG
system. In the perfect future, all access protocols will be able to
handle all referrals!
4.2.2 Limited Query and Response Semantics
The DAG system does not attempt to be a chameleon, or the ultimate
whitepages query service. It focuses on providing referrals for
information on the limited number of query types outlined in the
functional specifications of the DAG service. This makes the DAG
system a good place to start a search, but refinements and detailed
inquiries are beyond its scope.
4.2.3 Visibility
Given the limited query syntax of the DAG system it will not always
be possible to exactly match a query posed to a CAP into a query
posed to a SAP. This will have the effect that for instance a LDAPv2
client that issues a query to the DAG system which by the DAG system
is chained to a LDAP server might not get the same results as if the
client where directly connected to the server in question.
4.2.4 Richness of Query semantics
Even the limited query syntax of the DAG system is capable of
expressing queries that might NOT be possible to represent in the
access protocols to the WDSPs. In these cases the DAG-SAP either can
refuse the query or try to emulate it.
4.2.5 N+M Protocol Mappings
As part of the chaining service offered by the DAG system, a certain
amount of mapping between protocols is required -- in theoretical
terms, there are "N" allowable end-user query access protocols, and
"M" supported WDSP server protocols. The architecture of the
software is constructed to use a single internal protocol (the
DAG/IP) and data schema, providing a common language between all
components. Without this, each input protocol module (DAG-CAP) would
have to be constructed to be able to handle every WDSP protocol --
NxM protocol mappings. This would make the system complex, and
difficult to expand to include new protocols in future.
4.2.6 DAG-CAPs and DAG-SAPs are completely independent of each other
For the above reasons, the DAG-CAP and DAG-SAP modules are intended
to be completely independent of each other. A DAG-SAP responds to a
query that is posed to it in the DAG/IP, without regard to the
protocol of the DAG-CAP that passed the query.
4.2.7 The Role of the DAG-CAP
Thus, the DAG-CAP is responsible for using the DAG/IP to obtain
referral information and, where necessary, chained responses. Where
necessary, it performs adjustments to accommodate the differences in
semantics between the DAG/IP and its native protocol. This might
involved doing post-filtering of the results returned by the DAG-SAPs
since the query issued in DAG/IP to the DAG-SAP might be "broader"
then the original query.
Thus, the DAG-CAP "knows" only 2 protocols: its native protocol, and
the DAG/IP.
4.2.8 The Role of the DAG-SAP
Similarly, the DAG-SAP is responsible for responding to DAG/IP
queries by contacting the designated WDSP server. Where necessary,
it performs adjustments to accommodate the differences in semantics
between the DAG/IP and its native protocol. These adjustments might
mean that, as a consequence, the DAG-SAP will receive results that do
not match the original query. In such cases the DAG-SAP should
attempt to do post-pruning in order to reduce the mismatch between
the original query and the results returned.
Thus, the DAG-SAP "knows" only 2 protocols: its native protocol, and
the DAG/IP.
4.2.9 DAG/IP is internal
No module outside of the DAG system should be aware of the DAG/IP's
construction. End-users use the query protocols supported by DAG-
CAPs; WDSPs are contacted using the query protocols supported in the
DAG-SAPs.
4.2.10 Expectations
The expectation is that the DAG system, although defined as a single
construct, will operate by running modules on several different,
perhaps widely distributed (in terms of geography and ownership),
computers. For this reason, the DAG/IP specified in such a way that
it will operate on inter-machine communications.
4.2.11 Future Extensions
The DAG system architecture was constructed with a specific view to
extensibility. At any time, an individual component may be improved
(e.g., the Mail DAG-CAP may be given a different query interface)
without disrupting the system.
Additionally, future versions of the DAG system may support other
access protocols -- for end-users, and for WDSPs.
5.0 Software Specifications
5.1 Notational Convention
It is always a challenge to accurately represent text protocol in a
printed document; when is a new line a "newline", and when is it an
effect of the text formatter?
In order to be adequately illustrated, this document includes many
segments of protocol grammars, sample data, and sample input/output
in a text protocol. In order to distinguish newlines that are
significant in a protocol, the symbol
<NL>
is used. For example,
This is an example of a very long line of input. There is only one
newline in it (at the end), in spite of the fact that this document
shows it spanning several lines of text.<NL>
5.2 DAG-CAP Basics
5.2.1 Functionality
Every DAG-CAP must support the full range of DAG queries, as defined
in 3.3.1.
Each DAG-CAP accepts queries in its native protocol. Individual
DAG-CAP definitions define the expected expression of the DAG queries
in the native protocol.
The DAG-CAP is then responsible for:
- converting that expression into a query in the DAG/IP to obtain
relevant referrals from the Referral Index. This might mean that
parts of the original query are disregarded (e.g., if the query
included attributes not supported by the DAG application, or if the
query algebra was not supported by the DAG application);
- returning referrals in the client's native protocol, where
possible;
- expressing the client query to the necessary DAG-SAPs, given the
limitations mentioned above, to chain those referrals not usefully
expressible in the client's native protocol;
- possibly doing post-filtering on the DAG-SAP results; and
- converting the collected DAG-SAP results for expression in the
client's native protocol (and schema, where applicable).
Each DAG-CAP defines the nature of the interaction with the end-user
(e.g., synchronous or asynchronous, etc). Additionally, each DAG-CAP
must be able to carry out the following, in order to permit load-
limiting and load-balancing in the DAG system:
- direct the client to a different DAG-CAP of the same type (for
load-balancing)
- decline to return results because too many referrals were generated
(to discourage data-mining). Ideally, this should include the
generation of a message to refine the query in order to produce a
more manageable number of referrals/replies.
DAG-CAPs must be capable of accepting and respecting DAG-SAP service
referrals (for DAG-SAP load-sharing).
In protocols that permit it, the DAG-CAP should indicate to the end-
user which services were unavailable for chaining referrals (i.e., to
indicate there were parts of the search that could not be completed,
and information might be missing).
TISDAG: Any CAP that receives commands other than queries, like
help, answers those on its own. A CAP should not pass any system
command on to the RI.
5.2.2 Configuration
It must be possible to change the expected address of the DAG-CAP by
configuration of the software (i.e., host and port, e-mail address,
etc).
For DAG-CAPs that need to access DAG-SAPs for query chaining, for
each type (protocol) of DAG-SAP that is needed, the DAG-CAP must be
configurable in terms of:
- at least one known DAG-SAP of every necessary protocol to contact
- for each DAG-SAP, the host and port of the DAG-SAP software
The DAG-CAPs must also be configurable in terms of a maximum number
of referrals to handle for a user transaction (i.e., to prevent data
mining, the DAG-CAP will refuse to reply if the query is too general
and too many hits are generated at the Referral Index).
The DAG-CAP must be configurable in terms of alternate DAG-CAPs of
the same type to which the end-user software may be directed if this
one is too busy.
5.2.3 Error handling
Apart from error conditions arising from the operation of the DAG-CAP
itself, DAG-CAPs are responsible for communicating error conditions
occurring elsewhere in the system that affect the outcome of the
user's query (e.g., in the DAG-RI, or in one or more DAG-SAPs).
If the DAG-CAP sends a query to the DAG-RI and receives an error
message, it should attempt to match the the received DAG errorcode
into its native access protocol's error codes. The same action is
appropriate when the DAG-CAP is "chaining" the query to one DAG-SAP.
There are also occasions when the DAG-CAP may have to combine
multiple errorcodes into a single expression to the user. When the
DAG-CAP is "chaining" the query through DAG-SAPs to one or more
WDSPs, situations can arise when there is a mix of responsecodes from
the DAG-SAPs. If this happens, the DAG-CAP should try to forward
information to the end-user software that is as specific as possible,
for instance which of the WDSPs has not been able to fulfill the
query and why.
See Appendix D for more information concerning error condition
message mappings.
5.2.4 Pruning of results
Since there is no perfect match between the query syntaxes of the DAG
system on one hand and the different access protocols that the DAG-
CAPs and DAG-SAPs supports on the other, there will be situations
where the results a DAG-CAP has to collect is "broader" then what
would have been the case if there had been a perfect match. This
might have adverse effects on the system to the extent that
administrative limits will "unnecessary" be exceeded on WDSPs or that
the collected results exceeds the sizelimit of the DAG-CAP.
Since the DAG-CAP is the only part of the DAG system that actually
knows what the original query was, the DAG-CAP can prune the results
received from the DAG-SAPs in such a way that the results presented
to the client better matches the original question.
5.3 DAG-SAP Basics
5.3.1 Functionality
Every DAG-SAP must support the full range of DAG queries, as defined
in 3.3.1. Results must be complete DAG schemas expressed in well-
formed DAG/IP result formats (see Appendix C). Each DAG-SAP accepts
queries in DAG/IP and converts them to the native schema and protocol
for which it is designed to proxy.
The DAG-SAP is then responsible for
- converting the query into the native schema and protocol of the
WDSP to which the referral points. (If the query is not
representable in the native protocol, it must return an error
message. If it is emulatable, the DAG-SAP can attempt emulate it
by posing a related query to the WDSP and post-pruning the results
received);
- contacting that WDSP, using the host, port, and protocol
information provided in the referral;
- negotiating the query with the remote WDSP;
- accepting results from the WDSP, possibly doing post-filtering on
the result set; and
- conveying the results back to the calling DAG-CAP using the DAG/IP
and its schema.
Note that this implicitly means that the DAG-SAP is responsible for
chaining and pursuing any referrals it receives from WDSP services.
The DAG-SAP returns only search results to the DAG-CAP that called
it.
5.3.2 Configuration
DAG-SAPs must be configurable to accept connections only from
recognized DAG components.
DAG-SAPs that have service limits must be configurable to redirect
DAG-CAPs to alternate DAG-SAPs of the same type when necessary.
5.3.3 Error handling
A DAG-SAP must translate error codes received from a WDSP server to
DAG error codes according to Appendix D.
5.3.4 Pruning of results
Since it might not be possible to exactly map a DAG query into a
query in the access protocol supported by the a DAG-SAP, the DAG-SAP
should try to translate it into a more general query (or if necessary
into a set of queries). If so, the DAG-SAP must then prune the
result set received before furthering it to the DAG-CAP.
5.3.5 Constraint precedence
Some constraints, search and case, can appear both as local and
global constraints. If this happens in a query then the local
constraint specification overrides the global. For a query like the
following:
fn=leslie;search=exact and org=think:search=substring
the resulting search constraint for "fn=leslie" will be "exact" while
it for "org=think" will be "substring".
5.4 The Referral Index
5.4.1 Architecture
The Referral Index contains (only) information necessary to deliver