natural records of the encapsulated stream. Therefore, when the
outer stream requests a buffer's worth of file data from the
encapsulated stream, it expects to receive a buffer containing one
entire, ntegral, record of that stream, complete with byte count.
Because of diverse network implementations on different operating
systems, the software that implements the encapsulated stream might
not be able to provide integral record buffers to the Byte Stream
with Mark implementation. For example, the writing stream could have
written records that are much longer than available buffers on the
receiving system. In this case, a request to read from the
encapsulated stream returns some buffer or some amount of data
representing less than an entire Byte Stream with Mark record. The
input subroutine of the Byte Stream with Mark implementation must
therefore return a region of this (smaller) buffer, representing less
than the full Byte Stream with Mark record. Nevertheless, the Byte
Stream with Mark must extract the count of the full Byte Stream with
Mark record from the first such buffer of each Byte Stream with Mark
record, and maintain and update this count as succeeding component
buffers are read.
In this case, if the program reading from the Byte Stream with Mark
aborts while reading data, the implementation of Byte Stream with
Mark must continue to read through the remaining buffers of the Byte
Stream with Mark record that has been subdivided in this fashion.
The user side program will have determined that an abort has
occurred, and will request the Byte Stream with Mark to read up to
and through the next mark. The Byte Stream with Mark will have
processed a fractional record, and must discard the remaining buffers
of the record now being read.
12.2 Byte Stream with Mark Abortable States
Byte Stream with Mark is designed to provide end-to-end stream
consistency in the face of user program aborts. This section
describes user program aborts, and how Byte Stream with Mark handles
them. In the context of this document, "aborting" is defined as
follows: Aborting the current execution of a program means to halt
that execution and to abandon it, never to complete it. The data
representing the state of the execution are irrevocably discarded.
USER PROGRAM ABORTS AND I/O STREAMS
Aborting the execution of the code that manipulates I/O streams, in
general, poses significant problems. Given that a stream is a static
data object, and is intended to be used over and over again, aborting
the execution of any routine manipulating a stream can leave it in an
inconsistent, unusable state.
Many operating systems solve this problem by manipulating a large
subset of streams within the confines of the supervisor or executive
program, which is not vulnerable to aborts, short of system or
network failure. Nevertheless, the need still exists to implement
streams outside of the boundaries of the supervisor. Furthermore,
the Symbolics computer environment has no supervisor or executive
program, and is thus vulnerable to aborts everywhere.
BYTE STREAM WITH MARK HANDLING OF USER PROGRAM ABORTS
Byte Stream with Mark is designed to be nearly impervious to the
aborting of programs using it. Its design is based on careful
analysis of all possible states of the stream, and of the effect of
aborts of the programs using the stream in each of these states.
This section provides that analysis.
A "transmission" is a collection of user data sent by the application
level through the Byte Stream with Mark whose end is well-defined,
once its start has been recognized. For instance, the token list
stream, when using Byte Stream with Mark, sends token lists. When a
TOP-LEVEL-LIST-BEGIN has been sent, the containing transmission is
not considered complete until the corresponding TOP-LEVEL-LIST-END is
read. See the section "Token List Transport Layer", section 11.
The following cases are possible states of the stream when an abort
occurs:
1. Abort occurs when the user program is not manipulating the
stream.
This case presents no problem.
2. Abort occurs after a transmission has been partially sent,
at a packet or record boundary.
This implies that the datum that would indicate the
successful complete sending of that transmission has been
not yet been sent.
The Byte Stream with Mark state is consistent, but the
application level state is not. The application level must
determine that the execution of the code composing and
sending its transmission was, in fact, aborted, and
initiate resynchronization via marks.
The receiving side must be careful not to act upon a
transmission (that is, to perform any action or side
effect) until the transmission has been successfully
received in entirety. This protects the user program from
the possibility that an abort can occur after a
transmission has been partially sent.
3. Abort occurs during the sending or receiving of a record.
This is the most vulnerable state of the mechanism. This
case does not occur on packet-oriented media; it is
subsumed by the next case.
This case is handled by minimizing the extent of this
window, and killing the connection when and if the
situation is detected. Depending on the operating system
involved, this window could be minimized by using
interrupt-disabling mechanisms, auxiliary processes or
tasks, or some other technique.
For buffered streams, input and output waiting can be done
in consistent states, thus minimizing the amount of time
manipulating the actual encapsulated stream. For
unbuffered streams, a lot of time can be spent in this
window. It is expected that unbuffered streams will be
exceedingly uncommon. Nevertheless, the implementation of
Byte Stream with Mark must detect this case.
4. Abort occurs during the sending or receiving of fundamental
units of the lowest-level underlying stream (packets,
buffers, or bytes).
This case is usually handled by inhibiting interrupts, or
other forms of masking, in the code implementing the
encapsulated stream, since no waiting is possible at
unexpected times.
13. POSSIBLE FUTURE EXTENSIONS
NFILE was designed to be extended as the needs of its clients grow,
or as new clients with different needs appear. Currently it meets
the needs of the Symbolics Genera 7.0 operating system, although its
design is intentionally general. If users of other operating systems
identify new features that would be useful, they could be added to
NFILE. This section illustrates some areas areas where the design of
NFILE intentionally accommodates extensions.
- The NFILE protocol encodes commands and responses as text,
rather than using prearranged numbers. This means that new
commands and responses can be added without having to obtain
a new number from a central registry.
- The Token List Transport Layer provides a general substrate
for the value-transmission portion of network protocols. In
fact, it has been used at Symbolics for other protocols
besides NFILE. The Token List Transport Layer could
conveniently be extended to support transmission of other
types of values besides those it currently supports.
- The character set to be used for file transfer could be made
negotiable.
- The command character set could be made negotiable.
Currently there is no negotiation sequence, but one could be
added.
- Greater support for more complex file organizations could be
added, such as record files, databases, and so on. This
could be an extension to the direct access mode facility.
- Currently, the LOGIN command allows the user side to inform
the server which version of NFILE it is running. This
feature is included in NFILE so that a server can continue
to support older versions of the protocol even after new,
extended versions have been implemented. However, the
specification is currently somewhat vague as to how the
server can make use of the version.
- NFILE is not restricted to using TCP or Chaos as its
underlying protocol. NFILE can be built on any byte stream
protocol that supports reliable transmission of 8-bit bytes
and multiple connections.
In addition to the possible future extensions, we would like to
mention a known limitation of NFILE.
Currently NFILE requires multiple connections for a single session.
That is, the control connection must be separate from the data
connections. If NFILE is to be used over a telephone, this
requirement poses an inconvenient restriction. It is possible to
implement a multiplexing scheme as a level between NFILE and the
communication medium.
APPENDIX A
NORMAL TRANSLATION MODE
NORMAL translation mode guarantees the following:
- A file containing characters in the NFILE character set can
be written to any NFILE server and read back intact
(containing the same characters).
- A file written by NFILE should not appear as "foreign" to a
server operating system unless the file contains NFILE's
extended characters. That is, a server file that uses only
the subset of the NFILE character set limited to standard
ASCII characters (the 95 printing characters, and the native
representation of return, linefeed, page, backspace, rubout,
and tab) can be read and written, with the result being the
same data in NFILE characters as exists in server
characters.
In this section, all numbers designating values of character codes
are to be interpreted in octal. The notation "x in c1..c2" means
"for all character codes x such that c1 <= x <= c2."
The NFILE character set is an extension of standard ASCII. The 95
ASCII printing characters have the same numerical codes in the NFILE
character set. Five ASCII non-printing characters have counterparts
in the NFILE character set, as shown in the following table. The
NFILE character set includes a single Return character, rather than
the carriage-return line-feed sequence typically used in ASCII. The
NFILE character set does not include the ASCII control characters,
other than the five shown in the following table, but does include
some additional printing and formatting characters that have no
counterparts in ASCII.
NFILE Standard ASCII
Rubout: 207 177
Backspace: 210 10
Tab: 211 11
Linefeed: 212 12
Page: 214 14
Note that the NFILE Return character is of code 215. This character
includes "going to the next line". This is a notable difference from
the convention used in PDP-10 ASCII in which lines are ended by a
pair of characters, "carriage return" and "line feed".
NORMAL TRANSLATION TO UNIX SERVERS
The translation given in this table is appropriate for use by UNIX
servers, or other servers that use 8-bit bytes to store ASCII
characters. Machines with 8-bit bytes usually place the extra NFILE
characters in the top half of their character set.
TABLE 1. TRANSLATIONS FROM NFILE CHARACTERS TO UNIX CHARACTERS
NFILE character UNIX character
x in 000..007 x
x in 010..015 x + 200
x in 016..176 x
177 377
x in 200..207 x
x in 210..211 x - 200
212 015
x in 213..214 x - 200
215 012
x in 216..376 x
377 177
TABLE 2. TRANSLATIONS FROM UNIX CHARACTERS TO NFILE CHARACTERS
UNIX character NFILE character
x in 000..007 x
x in 010..011 x + 200
012 215
x in 013..014 x + 200
015 212
x in 016..176 x
177 377
x in 200..207 x
x in 210..215 x - 200
x in 216..376 x
377 177
NORMAL TRANSLATION TO PDP-10 FAMILY SERVERS
The translation given in this table is appropriate for use by PDP-10
family servers, or other servers that use 7-bit bytes to store ASCII
characters. On the PDP-10 the sequence CRLF, 015 012, represents a
new line.
The mechanism for this translation on machines with 7-bit bytes is to
use the RUBOUT character (octal code 177) as an escape character.
TABLE 3. TRANSLATIONS FROM NFILE TO PDP-10 CHARACTERS
NFILE character PDP-10 character(s)
x in 000..007 x
x in 010..012 177 x
013 013
x in 014..015 177 x
x in 016..176 x
177 177 177
x in 200..207 177 x - 200
x in 210..212 x - 200
213 177 013
214 014
215 015 012
x in 216..376 177 x - 200
377 no corresponding code
These tables might seem confusing at first, but there are some
general rules about it that should make it clearer. First, NFILE
characters in the range 000..177 are generally represented as
themselves, and x in 200..377 is generally represented as 177
followed by x - 200. That is, 177 is used to quote the second 200
NFILE characters. It was deemed that 177 is a more useful and common
character than 377, so 177 177 means 177, and there is no way to
describe 377 with PDP-10 ASCII characters. In the NFILE character
set, the formatting control characters appear offset up by 200 with
respect to standard ASCII. This explains why the preferred mode of
expressing 210 (backspace) is 010, and 010 turns into 177 010. The
same reasoning applies to 211 (Tab), 212 (Linefeed), 214 (Formfeed),
and 215 (Return).
More special care is needed for the Return character, which is the
mapping of the system-dependent representation of "the start of a new
line". The NFILE Return (215) is equivalent to 015 012 (CRLF) in
some ASCII systems. In the NFILE character set there is no
representation
TABLE 4. TRANSLATIONS FROM PDP-10 CHARACTERS TO NFILE CHARACTERS
PDP-10 character NFILE character
x in 000..007 x
x in 010..012 x + 200
013 013
014 214
015 012 215
015 not-012 115
x in 016..176 x
177 x in 000..007 x + 200
177 x in 010..012 x
177 013 213
177 x in 014..015 x
177 x in 016..176 x + 200
177 177 177
of a carriage that doesn't go to a new line, so if there is one in a
server file, it must be translated to something else. When
converting ASCII characters to NFILE characters, an 015 followed by
an 012 therefore turns into a 215. A stray CR is arbitrarily
translated into a single M (115).
APPENDIX B
RAW TRANSLATION MODE
RAW mode means no translation should be performed. In RAW mode the
server operating system should treat the file as a character file and
use the same data formatting that would be appropriate for a
character file, but transfer the actual binary values of the
character codes.
APPENDIX C
SUPER-IMAGE TRANSLATION MODE
SUPER-IMAGE mode is intended for use by PDP-10 family machines only.
It is included largely as an illustration of a system-dependent
extension. A server machine that has 8-bit bytes should treat
SUPER-IMAGE mode the same as NORMAL mode.
In this section, all numbers designating values of character codes
are to be interpreted in octal. The notation "x in c1..c2" means
"for all character codes x such that c1 <= x <= c2."
SUPER-IMAGE mode suppresses the use of the 177 character as an escape
character. Character translation should be done as in NORMAL mode,
with one exception. When a two-character sequence beginning with 177
is detected, the 177 should not be output at all.
In this section, all numbers designating values of character codes
are to be interpreted in octal. SUPER-IMAGE mode is intended for use
by PDP-10 machines only.
SUPER-IMAGE suppresses the use of Rubout for quoting. That is, for
each entry beginning with a 177 in the PDP-10 character column in the
NORMAL translation table, the NFILE character has the 177 removed.
TABLE 5. SUPER-IMAGE TRANSLATION FROM NFILE TO ASCII
NFILE character PDP-10 character(s)
x in 000..177 x
x in 200..214 <x - 200>
215 015 012
x in 216..376 <x - 200>
377 no corresponding code
TABLE 6. SUPER-IMAGE TRANSLATION FROM ASCII TO NFILE
PDP-10 character NFILE character
x in 000..007 x
x in 010..012 x + 200
013 013
014 214
015 012 215
015 not-012 115
x in <016..176> x
177 177
NOTES
1. NFILE's requirement for using the NFILE character set is
recognized as a drawback for non-Symbolics machines. A useful
extension to NFILE would be a provision to make the character set
negotiable.
2. Implementation note: Care must be taken that the freeing is done
before the control connection is allowed to process another
command, or else the control connection may find the data channel
to be falsely indicated as being in use.
3. The Symbolics operating system has the policy that whenever the
user side is waiting for the server side, a user abort can occur.
This user side waiting can occur in any context, such awaiting a
response, waiting in the middle of reading network input, or
waiting in the middle of transmitting network output. Thus there
are no "hung" states.
4. Note that the Token List Transport Layer supplies a special token
to indicate Boolean truth, but no corresponding token to indicate
Boolean falsity. NFILE uses an empty token list to indicate
Boolean falsity. The historical reason for this asymmetry is the
inability of the Lisp language to differentiate between the empty
list and NIL, which is traditionally used to mean Boolean falsity.
If the flexibility of both a Boolean falsity and an empty token
list were allowed, it would create problems for an operating
system that cannot distinguish between the two. This aspect of
the protocol is recognized as a concession to the Lisp language.
The unfortunate effect is to disallow operating systems to
distinguish between Boolean falsity and an empty list.
5. No so-called "fat strings" can be sent.