However, the over-generalization of these queries may yield excessive
numbers of results, and will necessitate some pruning of results in
order to match the returned results against the semantics of the
original LDAP query. It is the LDAP DAG-CAP that is responsible for
this pruning, as it is the recipient of the original query, and
responsible for responding to its semantics.
In concrete terms, when making the DAG/IP query which is to be sent
to a DAG-SAP the above mentioned queries are still valid queries,
but an alternative finer-grained query is also possible, namely:
FN=foo and FN=bar and ORG=c;search=lstring and ORG=t;search=tstring
In querying a DAG-SAP (irrespective of the protocol of that DAG-SAP),
the DAG/IP query must include information about the target WDSP
server. This information is drawn from the Referral Index SERVER-
TO-ASK referral information, and is appended to the query as
specified in Appendix C):
"host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charset
where the response from the Referral Index included:
"# SERVER-TO-ASK " serverhandle <NL>
" Server-info: " serverinfo<NL>
" Host-Name: " hostname<NL>
" Host-Port: " number<NL>
" Protocol: " prot<NL>
" Source-URI: " source<NL>
" Charset: " charset<NL>
"# END"<NL>
and the "quoted-hostname" and "quoted-serverinfo" are obtained from
"hostname" and "serverinfo" respectively, by quoting the DAG/IP
special characters.
For example, the referral
# SERVER-TO-ASK dagsystem01<NL>
Server-info: o=thinkingcat, c=se<NL>
Host-Name: thinkingcat.com<NL>
Host-Port: 2839<NL>
Protocol: ldapv2<NL>
Source-URI:http://www-thinkingcat.se/
Charset: T.61<NL>
# END<NL>
would yield the addition
:host=thinkingcat\.com;port=2839;server-info=o\=thinkingcat\,\
c\=se;charset=T\.61
in its query to an LDAPv2 DAG-SAP.
(N.B.: See Appendix C for further definitions of the terms used in
the SERVER-TO-ASK response).
Note that it is the DAG-SAP's responsibility to extract these terms
from the query and use them to identify the WDSP server to be
contacted. See the individual DAG-SAP definitions, below.
5.9.3 Chaining queries in LDAPv3 DAG-CAP
The LDAPv3 DAG-CAP relies on DAG-SAPs to resolve all referrals except
those to LDAPv3 servers (i.e., Whois++ referrals, currently).
5.9.4 Expression of results in LDAPv3
As described above, results from DAG-SAPs will have to be post-
processed in cases where the original query was generalized for
expression in DAG/IP. Acceptable results are expressed in LDAPv3
messages containing search result entries (see the standard for more
detail):
SearchResultEntry ::= [APPLICATION 4] SEQUENCE {
objectName LDAPDN,
attributes PartialAttributeList }
PartialAttributeList ::= SEQUENCE OF SEQUENCE {
type AttributeDescription,
vals SET OF AttributeValue }
SearchResultReference ::= [APPLICATION 19] SEQUENCE OF LDAPURL
-- at least one LDAPURL element must be present
SearchResultDone ::= [APPLICATION 5] LDAPResult
where
LDAPDN = DN / "cn=" (FN/ROLE) [",o=" ORG] ",dc=se"
attributes = <all attributes mapped from the DAG schema, and
"objectClass = inetOrgPerson",
"objectClass = person",
"objectClass = top" or
"objectClass = organizationalRole", as
appropriate, and "labeledURI = <SOURCE-URI>"
for each result from a given referral>
LDAPResult = success
(Where DN, FN, ROLE, and ORG are the values from the DAG schema).
I.e., where available, the entry's true DN is used; otherwise (e.g.,
for data coming from Whois++ servers), a reasonable facsimile is
constructed.
Referral URLs are constructed from the DAG/IP's SERVER-TO-ASK
information as follows:
refurl = "ldap://" HOST [":" PORT] "/" (SERVER-INFO / "dc=se")
The intention is that WDSPs using LDAPv3 servers will provide an
appropriate LDAPDN for their server in the SERVER-INFO. Clients are
then expected to repeat their query at the server designated by this
URL (i.e., the refURL does not include the query).
5.9.5 Expression of Errors in LDAPv3 DAG-CAP
As appropriate, the LDAPv3 DAG-CAP will express operational errors
following the LDAPv3 standard. There are 4 particular error
conditions of the DAG system that the DAG-CAP will handle as
described below.
When the LDAPv3 DAG-CAP receives a query that it cannot reply to
within the (data) constraints of the DAG queries, it sends an error
message and closes the connection. The error message includes the
LDAPv3 resultCode
noSuchAttribute (for incorrect schema attributes chosen)
inappropriateMatching (when a match type other than those
supported is used e.g., approxMatch)
unwillingToPerform (when the query is not one of the defined
types)
If the number of referrals sent by the Referral Index is greater than
the pre-determined maximum (for detecting data-mining efforts, or
otherwise refusing over-general queries, such as "FN=svensson"), the
LDAPv3 DAG-CAP will send an error message. The error message
includes the following resultCode:
adminLimitExceeded
An LDAPv3 DAG-CAP may redirect a connection to another LDAPv3 DAG-CAP
for reasons of load-balancing. In this case, the LDAPv3 DAG-CAP
sends a result message including only
SearchResultReference ::= [APPLICATION 19] AltURL
SearchResultDone ::= referral
where
AltURL = "ldap://" <althostport> ":" <altbase>
Since a LDAPv3 DAG-CAP only can send one resultcode back to a client;
If a LDAPv3 DAG-CAP receives several different result codes from the
DAG-SAPs it will have to construct a resultmessage that to some
extent represents the combination of those. It is proposed that in
these cases the following actions are taken:
- All the response codes are collected
- Each response code are translated into the corresponding LDAPv3
resultcode.
- A resultcode is chosen to represent the collected response on the
following grounds:
If "success" is the only resultcode represented after these steps
the return that result code.
If apart from "success" there is one other resultcode represented
return that other resultcode.
If apart from "success" there are two or more resultcodes
represented return the resultcode "other".
5.10 Whois++ DAG-SAP
5.10.1 Input
The Whois++ DAG-SAP expects valid DAG/IP communications. Queries
must include referral information (see below) and search terms that
conform to the DAG-allowed query types (e.g., not searches for
organization alone, etc).
The referral information is added to the end of the DAG-SAP query, as
defined in the DAG-CAP definition sections:
":host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charset
5.10.2 Translation from DAG/IP to Whois++ query
The HOST and PORT information are used to make a TCP/IP-based
connection to the remote (presumed) Whois++ server. The query
expressed to the remote Whois++ server is the remainder of the DAG/IP
query the Whois++ DAG-SAP received, with the following template ID
translations:
template=DAGPERSON becomes template=USER
and
template=DAGROLE becomes template=ORGROLE
Additional mappings for attributes are defined in Appendix B.
Note that the search types used in the DAG/IP are not all required by
the Whois++ syntax. Therefore, some Whois++ WDSPs may be using
servers that do not support searches other than "exact" and "lstring"
(the search types required by the Whois++ protocol standard). The
Whois++ DAG-CAP may
- send the DAG/IP query as constructed (e.g., with
"search=substring"), and pass back the "% 502 Search expression too
complicated" from the WDSP's server,
- translate the DAG/IP query into a construct using only these
search types (which will yield incomplete results, as not all
queries are expressible with those search types),
- attempt to ascertain what search types are supported by the
remote server and reformulate using them (e.g., regular
expressions). This would work, but would entail an excessively
complicated Whois++ DAG-SAP, and might not yield any better results
if the remote server doesn't support any optional search types.
5.10.3 Translation of Whois++ results to DAG/IP
Any referrals that the remote WDSP server returns are pursued,
following the usual Whois++ (client) fashion, by the Whois++ DAG-SAP.
If it is not possible to establish a Whois++ session with the remote
server, or if the session is interrupted, before results are
received, the DAG-SAP will itself return no results and an error
message, including
% 403 Information Unavailable<NL>
If the remote server issues any other Whois++ error message and does
not yield any results, the remote server's error message will be
included in the DAG-SAP's own error message; no results will be
returned.
If results are successfully received from the remote server, they
will be expressed using the DAG/IP -- essentially passing through
all FULL response information received from the remote server, mapped
into the DAGSchema using the mappings defined in Appendix A.
5.11 LDAPv2 DAG-SAP
5.11.1 Input
The LDAPv2 DAG-SAP expects valid DAG/IP communications. Queries must
include referral information (see below) and search terms that
conform to the DAG-allowed query types (e.g., not searches for
organization alone, etc).
The referral information is added to the end of the DAG-SAP query, as
defined in the DAG-CAP definition sections (as additional terms in
the DAG/IP query):
":host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charset
5.11.2 Translation from DAG/IP to LDAPv2 query
The HOST and PORT information are used to make a TCP/IP-based
connection to the remote (presumed) LDAPv2 server. The DAG-SAP will
establish a connection with the remote server, following standard
LDAPv2 message exchanges.
The search request itself will be constructed from the DAG/IP query
(without the HOST, SERVER-INFO and PORT terms) as follows:
SearchRequest ::=
[APPLICATION 3] SEQUENCE {
baseObject LDAPDN, -- from the DAG/IP query
scope baseObject (0) },
derefAliases ENUMERATED {
neverDerefAliases (0),
derefInSearching (1),
derefFindingBaseObj (2),
derefAlways (3)
},
sizeLimit INTEGER (0 .. maxInt),
timeLimit INTEGER (0 .. maxInt),
attrsOnly FALSE
filter Filter,
attributes SEQUENCE OF AttributeType
-- all DAGschema attributes
equivalents in the defined
standard LDAP schema
}
Filter ::=
CHOICE {
and [0] SET OF Filter,
or [1] SET OF Filter,
not [2] Filter,
substrings [4] SubstringFilter,
}
SubstringFilter
SEQUENCE {
type AttributeType,
SEQUENCE OF CHOICE {
substrings initial [0] LDAPString,
substrings any [1] LDAPString,
substrings final [2] LDAPString}
}
where and, or and not filters are constructed to preserve the logic
of the DAG/IP query.
For the purposes of matching token-based DAG/IP queries to reasonable
LDAP queries, all searches should be passed to the LDAP WDSP as
substring searches. The WDSP results must then be pruned to respect
token boundaries, where necessary.
So, for example, the DAG/IP query
FN=Foo\ Bar and ORG=Thinking\ Cat:search=substring<NL>
would be sent to the designated LDAP WDSP as
(& (fn=*Foo Bar*) (o=*Thinking Cat*) (objectclass=person))
Interestingly, the query
FN=Foo\ Bar and ORG=Thinking\ Cat:search=exact<NL>
would also be sent to the designated LDAP WDSP as
(& (fn=*Foo Bar*) (o=*Thinking Cat*) (objectclass=person))
but the WDSPs returned results would have to be pruned to remove any
results that had non-tokenizing characters on either side of "Foo
Bar" and "Thinking Cat".
The final consideration for mapping DAG/IP queries into LDAP queries
is the issue of character case. In LDAP, individual attribute
syntaxes define the consideration of case. All of the attributes
used here are case-insensitive in their definitions. Therefore, all
LDAP WDSP queries are inherently case-insensitive; if the DAG/IP
query calls for a case-sensitive match, the LDAP DAG-SAP will have to
do pruning of the results from the DAG-SAP.
5.11.3 Translation of LDAPv2 results to DAG/IP
If it is not possible to establish an LDAPv2 session with the remote
server, or if the session is interrupted before results are received,
or if the remote server issues any kind of error message and produces
no result, the DAG-SAP will itself return no results and an error
message, including
% 403 Information Unavailable<NL>
If results are successfully received from the remote server, the
attributes and values that are provided for each result message will
be incorporated into the DAG/IP result, according to the schema
mappings laid out in Appendix B.
One particular adjustment must be done to accommodate differences
between LDAP and the DAG/IP. The attributes on which searches are
keyed ("cn", "l", and "o" in the LDAP schemas) are all defined as
being case-insensitive for equality matching. Thus, if the DAG/IP
query includes the constraint "case=consider", the results from the
remote server must be post-processed to remove any wrong-cased ones.
TISDAG: The serverhandle and localhandle in the DAG/IP response
should be constructed as follows:
serverhandle is: <hostname-without-periods><port> (because
server DN's are not enforceably unique). E.g., a
services.bunyip.com server on 7778 would
become servicesbunyipcom7778.
localhandle is: the RDN (relative distinguished name), with
spaces replaced by "_". E.g., cn=leslie_daigle
5.12 LDAPv3 DAG-SAP
5.12.1 Input
The LDAPv3 DAG-SAP expects valid DAG/IP communications. Queries must
include referral information (see below) and search terms that
conform to the DAG-allowed query types (e.g., not searches for
organization alone, etc).
The referral information is added to the end of the DAG-SAP query, as
defined in the DAG-CAP definition sections:
":host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charset
5.12.2 Translation from DAG/IP to LDAPv3 query
The HOST and PORT information are used to make a TCP/IP-based
connection to the remote (presumed) LDAPv3 server. The DAG-SAP will
establish a connection with the remote server, following standard
LDAPv3 message exchanges.
The search request itself will be constructed from the DAG/IP query
(without the HOST, SERVER-INFO and PORT terms) as follows:
SearchRequest ::=
[APPLICATION 3] SEQUENCE {
baseObject LDAPDN, -- from the DAG/IP query
scope baseObject (0) },
derefAliases ENUMERATED {
neverDerefAliases (0),
derefInSearching (1),
derefFindingBaseObj (2),
derefAlways (3)
},
sizeLimit INTEGER (0 .. maxInt),
timeLimit INTEGER (0 .. maxInt),
attrsOnly FALSE
filter Filter,
attributes SEQUENCE OF AttributeType
-- all DAGschema attributes equivalents in
the defined standard LDAP schema
}
Filter ::=
CHOICE {
and [0] SET OF Filter,
or [1] SET OF Filter,
not [2] Filter,
substrings [4] SubstringFilter,
}
SubstringFilter
SEQUENCE {
type AttributeType,
SEQUENCE OF CHOICE {
substrings initial [0] LDAPString,
substrings any [1] LDAPString,
substrings final [2] LDAPString}
}
where and, or and not filters are constructed to preserve the logic
of the DAG/IP query.
For the purposes of matching token-based DAG/IP queries to reasonable
LDAP queries, all searches should be passed to the LDAP WDSP as
substring searches. The WDSP results must then be pruned to respect
token boundaries, where necessary.
So, for example, the DAG/IP query
FN=Foo\ Bar and ORG=Thinking\ Cat:search=substring<NL>
would be sent to the designated LDAP WDSP as
(&(fn=*Foo Bar*)(o=*Thinking Cat*)(objectClass=person))
Interestingly, the query
FN=Foo\ Bar and ORG=Thinking\ Cat:search=exact<NL>
would also be sent to the designated LDAP WDSP as
(&(fn=*Foo Bar*)(o=*Thinking Cat*)(objectClass=person))
but the WDSP's returned results would have to be pruned to remove any
results that had non-tokenizing characters on either side of "Foo
Bar" and "Thinking Cat".
The final consideration for mapping DAG/IP queries into LDAP queries
is the issue of character case. In LDAP, individual attribute
syntaxes define the consideration of case. All of the attributes
used here are case-insensitive in their definitions. Therefore, all
LDAP WDSP queries are inherently case-insensitive; if the DAG/IP
query calls for a case-sensitive match, the LDAP DAG-SAP will have to
do pruning of the results from the DAG-SAP.
5.12.3 Translation of LDAPv3 results to DAG/IP
Any referrals that the remote WDSP server returns are pursued,
following the usual LDAPv3 (client) fashion, by the LDAPv3 DAG-SAP.
If it is not possible to establish an LDAPv3 session with the remote
server, or if the session is interrupted before results are received,
or if the remote server issues any kind of error message and produces
no result, the DAG-SAP will itself return no results and an error
message, including
% 403 Information Unavailable<NL>
If results are successfully received from the remote server, the
attributes and values that are provided for each result message will
be incorporated into the DAG/IP result, which will be expressed using
the DAG/IP and schema mappings as outlined in Appendix A.
One particular adjustment must be done to accommodate differences
between LDAP and the DAG/IP. The attributes on which searches are
keyed ("cn", "l", and "o" in the LDAP schemas) are all defined as
being case-insensitive for equality matching. Thus, if the DAG/IP
query includes the constraint "case=consider", the results from the
remote server must be post-processed to remove any wrong-cased ones.
TISDAG: The serverhandle and localhandle in the DAG/IP response
should be constructed as follows:
- serverhandle is: <hostname-without-periods><port> (because
server DN's are not enforceably unique). E.g., a
services.bunyip.com server on 7778 would become
servicesbunyipcom7778.
- localhandle is: the RDN (relative distinguished name), with
spaces replaced by "_". E.g., cn=leslie_daigle
5.13 Example Queries
The following sample end-user queries illustrate some of the more
delicate steps of query/schema semantics translations in the DAG
system.
N.B.: the data presented in these examples is often senseless,
provided only to serve as illustrations of matching on word-ordering,
case sensitivity, etc.
5.13.1 A Whois++ Query
What the Whois++ DAG-CAP Receives
In this example, the Whois++ DAG-CAP receives the following query:
name=thinking and name=cat:search=exact;case=consider<NL>
The expected answer can be described as:
Any USER templates that contain the tokens "thinking" and "cat" in a
name attribute.
For example:
Different records:
name: the thinking cat
name: sublime cat thinking
or a single record with 2 or more name attributes
name: thinking felines
name: erudite cat
but not
name: Thinking Cat Enterprises
This last record would not match because the query called for case
sensitivity, and the case of the name attribute's value does not
match the query.
What the Whois++ DAG-CAP sends to the Referral Index
After schema translation, this is sent to the Referral Index as:
fn=thinking and fn=cat:search=exact<NL>
What the Whois++ DAG-CAP Sends to an LDAP DAG-SAP
Note that the Whois++ DAG-CAP will never interact with a Whois++
DAG-SAP as the Whois++ referrals returned by the Referral Index are
passed directly back to the Whois++ client.
The Whois++ DAG-CAP should send the same substantive query to the
DAG-SAP as it sent to the Referral Index, except that it can include
the case sensitivity constraint:
fn=thinking and fn=cat:search=exact;case=consider<NL>
which will be translated by the DAG-SAP into an LDAP query of the
form:
(&(cn=*thinking*)(cn=*cat*)(objectclass=inetOrgPerson))
which will match a record with:
cn: Thinking
cn: Cat
(i.e., 2 different cn attributes, with the 2 values; LDAP defines
case sensitivity matching by the schema attribute definition).
or a record with:
cn: I wish I had a thinking dog and a singing cat
The first record should be pruned by the LDAP DAG-SAP, in order to
respect the semantics of the DAG/IP query.
5.13.2 An LDAP Query
What the LDAP DAG-CAP Receives
In this example, the LDAP DAG-CAP receives the following query
(using RFC1960 notation):
(& (cn=th*c*t) (o=green groceries) (objectClass=person))
What the LDAP user is looking for, with this query, is all records
within the "green groceries" organization that have a cn attribute
starting with "th", ending with "t", and having a "c" somewhere in
the middle.
cn values that would match this include:
cn: thinkingcat
cn: Thinking Cat
cn: The Black Cat
cn: Thick Mat
5.13.3 What the LDAP DAG-CAP sends to the Referral Index
The LDAP DAG-CAP must formulate a token-based query to the Referral
Index that will not inadvertently exclude records that would match.
The first challenge lies in the fact that the "*" characters in the
LDAP string-based query can cover token-boundaries.
A suitable query to the Referral Index would be:
FN=th AND FN=C AND FN=T AND ORG=green AND
ORG=groceries:search=substring<NL>
This will generate some false positive referrals, directing the query
to WDSPs containing records with the following attribute values (the
match letters are in capitals for ease of identification):
cn: wiTH three blaCk poTs
o: peaGREEN and cyan GROCERIES
o: GROCERIES are GREENer than electronics
Alternative approaches include breaking the original query into
several queries to the referral index in such a way that the DAG-CAP
can use only those referrals that appear in all the Referral Index
responses. However, this is
overkill -- the purpose of the Referral Index is to give direction on
where there may be more information
difficult to code into the DAG-CAP in a general way -- it has to
identify, by LDAP query type, when and how to do so
likely to generate Referral Index queries that are complex and time-
consuming to process.
What the LDAP DAG-CAP Sends to a Whois++ DAG-SAP
The LDAP DAG-CAP may send the same query to a Whois++ DAG-SAP as it
sent to the Referral Index. False positives here mean results that
are not expected as a match by the LDAP client. The LDAP DAG-CAP
should prune these results from the information returned by the
Whois++ DAG-SAP.
Or it might rewrite the query into:
FN=th;search=lstring AND FN=C;search=substring AND
FN=T;search=tstring AND ORG=green AND ORG=groceries:case=ignore<NL>
What the LDAP DAG-CAP Sends to an LDAP DAG-SAP
As an architectural principle, it is never wrong to send the same
query to a DAG-SAP as was formulated for the Referral Index. It is
also noteworthy to keep in memory that all DAG-SAPs are handled equal
by all DAG-CAPs therefore a LDAP DAG-CAP will not need to send a
different query to a LDAP DAG-SAP then it would to any other DAG-SAP.
So in this case the LDAP DAP-CAP could either send the same query to
the LDAP DAG-SAP as it sent to the Referral Index or it could send
the augmented version that is allowed to be use with the DAG-SAPs,
namely:
FN=th;search=lstring AND FN=C;search=substring AND
FN=T;search=tstring AND ORG=green\ groceries:case=ignore<NL>
Note that this will be translated, by the LDAP DAG-SAP, into a query
of the form
(&(cn=*th*)(cn=*c*)(cn=*t*)(o=*green groceries*)
(objectClass=person))
which is still more general than the original query.
Note the translation from "FN=th;search=lstring" into "cn=*th*".
This is necessary, as the DAG/IP lstring constraint is based on
tokens, whereas "cn=th*" refers to the beginning of the attribute's
value (phrase, not token). The DAG-SAP should therefore prune out
any results that include things like "oTHer plaCes for visiTors" in
order to match the semantics of the DAG/IP query it received.
The DAG-CAP should then prune those results to match the semantics of
the original LDAP query.
6.0 Service Specifications
6.1 Overview
To satisfy the requirements laid out for the TISDAG project, the
software built for the DAG system must be able to meet the following
service specifications:
- primary designated DAG-CAPs of all types (but not necessarily
secondary ones set up for load-balancing) must be available to
provide service or redirect queries on a 7x24 basis.
- in general, responses to queries should be available in under 10
seconds; very generalized queries (i.e., when the user truly cannot
specify enough information to focus the search) can be deferred to
take much longer (having results is more important than having a
quick answer)
- the data provided from each WDSP should be updated in the DAG at
least once every 7 days
6.2 WDSP Participation
WDSPs who wish to participate in the DAG system do so by providing
DAG-compatible access to their service, where DAG-compatible means:
- access in (exactly) one of LDAPv2, LDAPv3, or Whois++
- 7x24 service for responding to referrals generated in the DAG
core (minimally) weekly updates of the index object describing the
information their service indexes
- use of USER and ROLE templates for Whois++ servers
- use of inetorgperson and organizationalrole objectclasses for
LDAP servers
To participate, WDSPs must register each DAG-compliant server with
the DAG system, providing details for each data set that it covers:
- the host, port and protocol of the server
- an identifier for the dataset
- a URL for the service of preference for accessing the data
(preferred source)
- protocol-specific information
- administrative contact information
- CIP object exchange information
Note that any WDSP wishing to make data available through the DAG
system but unable to support these requirements may provide
information through an agreement with a third-party which does meet
these requirements. Thus, data can be replicated between cooperating
WDSPs. The DAG referral index does not claim ownership of personal
information; it directs queries to services that do, by whatever
agreements with whichever relevant parties. Note that, in this case,
the SOURCE-URI may direct end-users to the WDSP's existing services,
not the service of the third party.
6.3 Load Distribution
It is anticipated that the DAG system will be quite popular, and
measures must be available to distribute the load of answering
queries.
The DAG system is presented as a conceptual whole, made up of several
component parts -- DAG-CAPs, DAG-SAPs and the Referral Index. Each
of these component parts must be replicable, and service must be
shared between replicas.
It may be interesting to consider allowing large-scale service
providers (large companies, ISPs) the ability to mirror the Referral
Index or provide alternate DAG-CAPs/DAG-SAPs for their
personnel/customers. Policies and possibilities for doing that are
beyond the scope of this report; however, the software architecture
has been designed to support such activity.
Figure 6.1 shows that individual components of the DAG system may
each run on non-co-located server hardware, connected by TCP/IP
networks. These components can be replicated as needed.
+====+
| | DAG-CAP (Client Access Point)
| |
+====+
+----+
| | DAG-SAP (Service Access Point)
| |
+----+
+====+
HTTP <-->| |
| | +----+
+====+ | |<--> Whois++
| |
+====+ +----+
SMTP <-->| |
| | +----+
+====+ | |<--> LDAPv2
| |
+====+ +----+
Whois++<-->| |
| |
+====+ +----+
| |<--> LDAPv3
| |
+----+
| |<--> LDAPv3
| |
+----+
| |<--> LDAPv3
| |
+====+ +----+
LDAPv2 <-->| |
| |
+====+
+====+
LDAPv3 <-->| |
| |
+====+
+------------------------+
| Referral Index |<--> Common Indexing Protocol
| | (CIP)
+------------------------+
+------------------------+
| Referral Index |
| |
+------------------------+
Figure 6.1 Distributable nature of DAG components
Thus, the software built to this specification must be configurable
to permit the following actions:
- DAG-CAP software must be able to handle or redistribute the primary
load. Depending on the DAG-CAP software, this may be handled by
having multiple processes attending to incoming queries, or the
DAG-CAP at the primary address for the protocol may be nothing more
than a reflector that redirects incoming queries to the address of
the least-loaded server at the moment.
- This is particularly necessary in synchronous connection protocols,
such as Whois++ and LDAP, where the goal is to minimize the amount
of time a requesting client is connected to the well-advertised
address port.
- DAG-CAP software must be able to direct referrals to different
DAG-SAPs of the same protocol type.
- DAG-CAP software must be able to detect overly general queries
(i.e., have some metric to decide that the number of referrals
generated by the Referral Index is too great).
- DAG-SAPs must be able to redirect DAG-CAP queries at their
discretion, or just refuse service because of loading (therefore
DAG-CAPs must also be able to find other DAG-SAPs)
6.4 Extensibility
The DAG system has been designed to allow for extensibility in
certain key areas:
It is possible to add new DAG-CAPs and DAG-SAPs transparently.
Beyond replicating the software of existing DAG-CAPs, new
implementations for particular protocols (e.g., building a more
elaborate mail-based query system), or implementations for altogether
different protocols (e.g., PH) can be added by adhering to the basic
principles of DAG-CAPs and DAG-SAPs defined in the software
specification. The new DAG-CAP is responsible for the translation of
queries into DAG/IP (post-processing results, if necessary) and
results in the new protocol. No other part of the DAG system is
affected.
More functionality may be added to the DAG system service (e.g.,
adding security certificate references to the schema of returned
information) by updating the DAG schema.
Depending on how the load on the service goes, it may be interesting
to consider reducing the number of queries that are chained for
protocols that inherently can handle the concept of pursuing
referrals. Specifically, LDAPv3 and Whois++ both handle referrals,
but the current system calls for chaining LDAPv3 (and LDAPv2)
referrals for the Whois++ DAG-CAP, and vice versa. Alternatively,
"virtual" DAG-CAPs could be established for each participating WDSP
for each protocol the WDSP doesn't support, and referrals to those
DAG-CAPs could be given to the calling client. For example, a
Whois++ client would be given a Whois++ referral to the virtual
Whois++ DAG-CAP for a WDSP that supports only LDAP. The importance
of having one virtual DAG-CAP per WDSP is that the point of
connection is the only way to distinguish which WDSP the Whois++
client thought it was connecting to.
7.0 Security
7.1 Information credibility
Security, in the context of "read-only" directory services, is
primarily concerned with maintaining data integrity as it passes from
an originating server to the end-user making an inquiry. That is,
some server(s) hold correct user information, and a client accessing
a directory service should be certain that whichever servers that the
information has to pass through before reaching the client, it
receives a true representation of the original information.
The DAG system as such MUST be completely invisible as the mediator
of the information from the WDSPs to the querying directory access
client. The only possible modifications that can appear is
translations from one characterset into another. Hopefully, this
does not alter the meaning of the information.
7.2 Unauthorized access
In keeping with the public nature of the proposed TISDAG service, the
DAG system does not provide any access control system beyond
components' configuration to accept connections from recognized other
components. For more detailed access control, it is up to the
connected WDSPs to apply the access control.
Since the DAG system only supports searching and retrieving
information, no updates can occur through the DAG client access
points.
Security in updates (CIP index objects) is provided by encryption and
signature of objects from registered WDSPs.
8.0 Acknowledgments
This work came from ideas originally put forward by Patrik Faltstrom.
The TISDAG project was supported by the Swedish KK Foundation.
Thanks to especially to Jens Lundstrom, Thommy Eklof, Bjorn Larsson
and Sandro Mazzucato for their comments on draft versions of this
document.
Appendix A - DAG Schema Definitions
The DAG makes use of 2 information schemas -- the DAGPERSON schema
for information about specific people, and the DAGORGROLE schema for
organizational roles that may or may not be job positions occupied by
people at any given time (e.g., an organization's president, customer
service desk, etc).
This appendix defines the schemas in terms of the attributes used
within the DAG/IP. Mappings to the standard LDAP and Whois++ object
classes and templates (respectively) are described in Appendix B.
Because the role of the DAG schemas is to act as an intermediary
between information provided in different access protocols, with
different underlying schema paradigms, the attributes in the schema
are identified as being required or optional. The required
attributes are so designated because they are involved in the DAG
search types and/or the minimal returned response. They have defined
mappings in the selected access protocols. The optional attributes
have proposed mappings in those protocols.
It is important to note that the DAG/IP is constructed to carry any
alternative attribute information that may be provided by a given
WDSP; individual DAG-SAPs and DAG-CAPs may choose to pass along,
interpret, or ignore any attributes not defined in this appendix.
Additionally, note that the order of attributes in the DAG/IP is
significant, which means that it is possible to use one attribute to
carry the information describing the type of subsequent ones (e.g.,
see the "ADR-TYPE" attribute below).
Finally, attributes may be repeated. For example, this schema
structure can carry multiple phone numbers of different types for
one person.
A.1 DAG Personal Information Schema (DAGPERSON Schema)
Attribute Designation Specific Description
--------- ----------- -------------------------------------
FN Required Free-text representation of full name
EMAIL Required Internet e-mail address
LOC Required Locality -- geographic region
ORG Required Person's organization
ADR-TYPE Optional Type of address that follows
("org", "home", "org-postal",
"home-postal", "unqualified")
ADR Optional Full address
ADR-STREET Optional Street address component
ADR-ROOM Optional Suite or room number component
ADR-CITY Optional City name
ADR-STATE Optional Region of address
ADR-COUNTRY Optional Country
ADR-CODE Optional Postal code component
TEL-TYPE Optional Type of telephone number (
"work", "home", "mobile",
"fax" ,"pager", "unqualified")
in the following attribute
TEL Optional A phone number for the person
SOURCE Optional The WDSP's preferred access to
their service -- a URL
DN Optional Entry's "distinguished name"
(for LDAP)
Table A.1 DAGPERSON schema attributes
A.2 DAG Organizational Role Information Schema (DAGORGROLE Schema)
Attribute Designation Specific Description
--------- ----------- ---------------------
ROLE Required Name of organizational role
EMAIL Required E-mail address associated with role
ORG Required Name of organization
LOC Required Locality -- geographic region
TEL-TYPE Optional Type of telephone number
in the TEL attribute immediately
following("org" or "fax")
TEL Optional Phone number
FN Optional Full name of current role occupant
SOURCE Optional The WDSP's preferred access to their
service -- a URL
DN Optional Entry's "distinguished name" (for LDAP)
Table A.2 DAGORGROLE schema attributes
Appendix B - Schema Mappings for Whois++ and LDAP
The DAG/IP makes use of two specific schemas, as defined above.
However, schemas particular to access protocols need to be handled in
order to appropriately address incoming user queries, and chaining
queries to WDSPs. The recognized standard schemas are:
- the USER template for Whois++ ([8])
- the ORGROLE template for Whois++ ([8])
- the inetOrgperson objectclass for LDAP ([16])
- the organizationalrole objectclass for LDAP ([18])
The DAG/IP schemas were developed based on the information that the
TISDAG project requirements wish to return in results, in conjunction
with information about standard schemas used in the basic WDSP access
protocols (LDAPv2/v3 and Whois++). However, particularly in the case
of address information, the schemas used for those protocols allow
for considerable scope of information representation. In practice,
this means that different WDSPs may choose to use different sub-parts
of the schema, or even implement local customizations.
Therefore, Appendix A outlines a very basic schema that can carry all
the necessary information. The basic DAG-CAPs and DAG-SAPs are
designed to work to that information structure. This appendix
outlines the expected behaviour for DAG-SAPs mapping into the DAG/IP
schema, and DAG-CAPs extracting information to pass along to client
software after a chaining operation has returned results.
B.1 LDAP and the DAG Schemas
The only time information is carried in the DAG schemas is when a
DAG-SAP is returning information (obtained from WDSPs' servers) to a
DAG-CAP using the DAG/IP. The "canonical" mappings between standard
LDAP object classes (inetorgPerson, defined in [16] and
organizationalRole, defined in [18] and the DAGPERSON schema and
DAGORGROLE schema are defined such that information passed from an
LDAP DAG-SAP to an LDAP DAG-CAP (e.g., in the case of an LDAPv3 DAG-
SAP returning information chained for an LDAPv2 DAG-CAP) will be
mapped into the same attributes as it was extracted.
However, the representation of some attributes (such as address) is
truly widely varied between protocol paradigms. The goal with the
"reasonable approximation" mappings that are provided is to give
DAG-CAPs a basic mechanism for communicating information drawn from
non-LDAP DAG-SAP sources. The mappings may not be perfect, but they
will convey the information to the end-user in some LDAP-
understandable fashion, which is the goal of this project's effort.
The canonical mappings for the LDAP inetorgPerson object class and
the DAGPERSON schema are given in Table B.1. A few reasonable
approximation mappings follow in Table B.2. Beyond that, DAG-SAPs
may pass along any additional attributes in the DAG/IP, and DAG-CAPs
may elect to forward or interpret any that are recognizable (e.g.,
the sn ("surname") attribute is not listed here, but a DAG-SAP might
return that in the DAG/IP, and a DAG-CAP, recognizing the string
representation, could elect to include it in its LDAP response to the
client).
DAGPERSON Attribute LDAP inetorgPerson attribute
------------------- ----------------------------
FN cn
EMAIL mail
LOC l
ORG o
ADR-TYPE=org
ADR-STREET street
ADR-ROOM roomNumber
ADR-STATE st
ADR-COUNTRY c
ADR-TYPE=org-postal
ADR postalAddress
ADR-ROOM postOfficeBox
ADR-CODE postalCode
ADR-TYPE=home-postal
ADR homePostalAddress
TEL-TYPE=work
TEL telephoneNumber
TEL-TYPE=home
TEL homePhone
TEL-TYPE=fax
TEL facsimileTelephoneNumber
TEL-TYPE=mobile
TEL mobile
TEL-TYPE=pager
TEL pager
DN dn
SOURCE labeledURI
Table B.1 Canonical DAGPERSON schema & LDAP inetorgPerson attributes
DAGROLE Attribute LDAP organizationalRole attribute
----------------------- ---------------------------------
ADR-TYPE=unqualified
ADR street
ADR-STREET street
ADR-ROOM room
ADR-STATE st
ADR-COUNTRY c
TEL-TYPE=unqualified
TEL telephoneNumber
Table B.2 Reasonable Approximations for LDAP organizationalRole
attributes
For example, consider the following LDAP record information, in LDIF
[11] format:
dn: cn=Barbara Jensen, ou=Product Development, o=Ace Industry,
c=US
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetorgperson
cn: Barbara Jensen
cn: Barbara J Jensen
cn: Babs Jensen
sn: Jensen
uid: bjensen
telephonenumber: +1 408 5551212
description: A big sailing fan
This would validly be carried in the DAGPERSON schema as follows:
DN: cn=Barbara Jensen, ou=Product Development, o=Ace Industry,
c=US
FN: Barbara Jensen
FN: Barbara J Jensen
FN: Babs Jensen
SN: Jensen
TEL-TYPE: work
TEL: +1 408 5551212
The canonical mappings for the LDAP organizationalRole object class
and the DAGORGROLE schema are given in Table B.3 .Beyond that, DAG-
SAPs may elect to send along any attributes, and DAG-CAPs may
interpret any that are recognizable. N.B., the organizationalRole
class does not include provision for inclusion of an e-mail address.
This mapping rather blithely assumes the availability of the mail
attribute as defined for inetorgPerson.
DAGORGROLE Attribute LDAP organizationalRole attribute
-------------------- ---------------------------------
ROLE cn
EMAIL mail
ORG o
LOC l
TEL-TYPE=org
TEL telephoneNumber
TEL-TYPE=fax
TEL facsimileNumber
FN roleOccupant
DN dn
SOURCE labeledURI
Table B.3 Canonical mappings for LDAP organizationalRole attributes
B.2 Whois++ and the DAG Schemas
The "canonical" mappings between standard Whois++ templates as
defined in [8] and the DAGPERSON schema and DAGORGROLE schema are
defined in Tables B.4 and B.5. Beyond that, DAG-SAPs may pass along
any additional attributes in the DAG/IP, and DAG-CAPs may elect to
forward or interpret any that are recognizable.
DAGPERSON Attribute Whois++ USER template attribute
------------------- -------------------------------
FN name
EMAIL email
LOC address-locality
ORG organization-name
ADR-TYPE=unqualified
ADR address
ADR-TYPE=org
ADR organization-address
ADR-STREET organization-address-street
ADR-ROOM organization-address-room
ADR-CITY organization-address-city
ADR-STATE organization-address-state
ADR-COUNTRY organization-address-country
ADR-CODE organization-address-zip-code
ADR-TYPE=home address-type=home
ADR address
ADR-STREET address-street
ADR-ROOM address-room
ADR-CITY address-city
ADR-STATE address-state
ADR-COUNTRY address-country
ADR-CODE address-zip-code
TEL-TYPE=work phone-type=work
TEL phone
TEL-TYPE=home phone-type=home
TEL phone
TEL-TYPE=fax
TEL fax
TEL-TYPE=mobile
TEL cellular
TEL-TYPE=pager
TEL pager
Table B.4 Canonical DAGPERSON schema & Whois++ USER attributes
DAGORGROLE Attribute Whois++ ORGROLE attribute
-------------------- -------------------------
ROLE org-role
EMAIL email
ORG organization-name
LOC organization-address-locality
FN name
TEL-TYPE=org
TEL phone
TEL-TYPE=fax
TEL fax
Table B.5 Canonical mappings for Whois++ ORGROLE attributes
Appendix C - DAG-Internal Protocol (DAG/IP)
The DAG-Internal Protocol (DAG/IP) is currently defined as a
derivative of the query-interaction protocol of Whois++ as laid out
in RFC1835 ([6]).
C.1 A word on the choice of DAG/IP
The use of the DAG/IP is strictly internal to the DAG system. In
that regard, it is possible make use of any query language, or define
a new one.
The Whois++ protocol was selected as the basis of the DAG/IP for
several reasons:
- it has the power and flexibility to convey all necessary queries
- it is a simple, text-based protocol; clients need not implement the
full functionality of the protocol in order to carry out minimal
queries
- the power of the full-fledge directory service query protocol will
give DAG-CAP writers the ability to express more sophisticated
queries if desired (e.g., to produce more intricate "intelligent"
matching of spellings, common character substitutions, etc).
- the text-based, delimited attribute results expression facilitates
optional inclusion of extra data supplied by WDSPs -- DAG-CAPs can
easily ignore any unknown information and continue to interpret the
rest of the result information.
Also, the use of an existing protocol leverages the experience and
time of the creators of the protocol -- hammering out such elusive
and yet necessary details as handling line-endings, quoting special
characters, etc.
There is a freely-available test suite of tools for testing servers'
Whois++ protocol conformance (for the Referral Index, and for DAG-
SAPs). Send mail to digger-info@bunyip.com for further information.
C.2 DAG/IP Input and Output -- Overview
Input interactions in DAG/IP are as defined in RFC1835, "Architecture
of the WHOIS++ service" ([6]), sections 2.2 and 2.3. Section C.3 of
this document adapts the grammar used in more recent descriptions of
the Whois++ protocol to illustrate the syntax of the DAG/IP.
DAG/IP output will be a subset of what is defined in RFC1835, section
2.4, except that referral responses ("SERVER-TO-ASK") contain more
information.
C.3 BNF for DAG/IP input and output
The following sections are adapted from the Whois++ grammar. For
discussion of the semantic intent of the query protocol, and other
matters, see Whois++ RFC1835 [6].
C.3.1 The DAG/IP Input Grammar
The following grammar, which uses the Augmented BNF (ABNF) notation
as defined in [5], defines the set of acceptable DAG/IP input.
N.B.: As outlined in the ABNF definition, rule names and string
literals are in the US-ASCII character set, and are case-insensitive.
Also, when a character is written explicitly in the grammar, as for
example ";", it represents the byte value of that character in all of
the allowed character sets in their encodings used in this protocol.
Specifically in UNICODE, ";" means the character U+003B, which when
encoding the character in UTF-8 will generate the byte value 0x3B
which is then used in the DAG/IP protocol.
dagip-command = ( system-command [":" "hold"]
/ ri-query
/ sap-query ) nl
ri-query = ri-terms [":" globalcnstrnts]
sap-query = sap-terms [":" [sapcnstrnts][ ":" wdspinfo]]
system-command = "constraints"
/ "describe"
/ "commands"
/ "polled-by"
/ "polled-for"
/ "version"
/ "list"
/ "show" [1*sp datastring]