"holdoff" period, the sender will immediately incorporate these "late
messages" into its pending transmission state ONLY if the NACK
content is ordinally greater than the sender’s current transmission
position. This "holdoff" time allows worst case time for the sender
to propagate its current transmission sequence position to the group,
thus avoiding redundant repair transmissions. After the holdoff
timeout expires, a new NACK accumulation period can be begun (upon
arrival of a NACK) in concert with the pending repair and new data
transmission. Recall that receivers are not to initiate the NACK
repair process until the sender’s logical transmission position
exceeds the lowest ordinal position of their repair needs. With the
new NACK aggregation period, the sender repeats the same process of
incorporating accumulated repair state into its transmission plan and
subsequently "rewinding" to transmit the lowest ordinal repair data
when the aggregation period expires. Again, this is conducted in
concert with ongoing new data and/or pending repair transmissions.
5.4.2. Sender FEC Repair Transmission Strategy
The NORM sender should leverage transmission of FEC parity content
for repair to the greatest extent possible. Recall that the
receivers use a strategy to request a lowest common denominator of
explicit repair (including parity content) in the formation of their
NORM_NACK messages. Before falling back to explicitly satisfying
different receivers’ repair needs, the sender can make use of the
general erasure-filling capability of FEC-generated parity segments.
The sender can determine the maximum erasure filling needs for
individual FEC coding blocks from the NORM_NACK messages received
during the repair aggregation period. Then, if the sender has a
sufficient number (less than or equal to the maximum erasure count)
of previously unsent parity segments available for the applicable
coding blocks, the sender can transmit these in lieu of the specific
packets the receiver set has requested. Only after exhausting its
supply of "fresh" (unsent) parity segments for a given coding block
should the sender resort to explicit transmission of the receiver
set’s repair needs. In general, if a sufficiently powerful FEC code
is used, the need for explicit repair will be an exception, and the
fulfillment of reliable multicast can be accomplished quite
efficiently. However, the ability to resort to explicit repair
allows the protocol to be reliable under even very extreme
circumstances.
NORM_DATA messages sent as repair transmissions SHALL be flagged with
the NORM_FLAG_REPAIR flag. This allows receivers to obey any
policies that limit new receivers from joining the reliable
transmission when only repair transmissions have been received.
Additionally, the sender SHOULD additionally flag NORM_DATA
transmissions sent as explicit repair with the NORM_FLAG_EXPLICIT
flag.
Although NORM end system receivers do not make use of the
NORM_FLAG_EXPLICIT flag, this message transmission status could be
leveraged by intermediate systems wishing to "assist" NORM protocol
performance. If such systems are properly positioned with respect to
reciprocal reverse-path multicast routing, they need to sub-cast only
a sufficient count of non-explicit parity repairs to satisfy a
multicast routing sub-tree’s erasure filling needs for a given FEC
coding block. When the sender has resorted to explicit repair, then
the intermediate systems should sub-cast all of the explicit repair
packets to those portions of the routing tree still requiring repair
for a given coding block. Note the intermediate systems will be
required to conduct repair state accumulation for sub-routes in a
manner similar to the sender’s repair state accumulation in order to
have sufficient information to perform the sub-casting.
Additionally, the intermediate systems could perform additional
NORM_NACK suppression/aggregation as it conducts this repair state
accumulation for NORM repair cycles. The detail of this type of
operation are beyond the scope of this document, but this information
is provided for possible future consideration.
5.4.3. Sender NORM_CMD(SQUELCH) Generation
If the sender receives a NORM_NACK message for repair of data it is
no longer supporting, the sender generates a NORM_CMD(SQUELCH)
message to advertise its repair window and squelch any receivers from
additional NACKing of invalid data. The transmission rate of
NORM_CMD(SQUELCH) messages is limited to once per 2*GRTT. The
"invalid_object_list" (if applicable) of the NORM_CMD(SQUELCH)
message SHALL begin with the lowest "object_transport_id" from the
invalid NORM_NACK messages received since the last NORM_CMD(SQUELCH)
transmission. Lower ordinal invalid "object_transport_ids" should be
included only while the NORM_CMD(SQUELCH) payload is less than the
sender’s NormSegmentSize parameter.
5.4.4. Sender NORM_CMD(REPAIR_ADV) Generation
When a NORM sender receives NORM_NACK messages from receivers via
unicast transmission, it uses NORM_CMD(REPAIR_ADV) messages to
advertise its accumulated repair state to the receiver set since the
receiver set is not directly sharing their repair needs via multicast
communication. The NORM_CMD(REPAIR_ADV) message is multicast to the
receiver set by the sender. The payload portion of this message has
content in the same format as the NORM_NACK receiver message payload.
Receivers are then able to perform feedback suppression in the same
manner as with NORM_NACK messages directly received from other
receivers. Note the sender does not merely retransmit NACK content
it receives, but instead transmits a representation of its aggregated
repair state. The transmission of NORM_CMD(REPAIR_ADV) messages are
subject to the sender transmit rate limit and NormSegmentSize
limitation. When the NORM_CMD(REPAIR_ADV) message is of maximum
size, receivers SHALL consider the maximum ordinal transmission
position value embedded in the message as the senders "current"
transmission position and implicitly suppress requests for ordinally
higher repair. For congestion control operation, the sender may also
need to provide information so that dynamic congestion control
feedback can be suppressed as needed among receivers. This document
specifies the NORM-CC Feedback Header Extension that is applied for
baseline NORM-CC operation. If other congestion control mechanisms
are used within a NORM implementation, other header extensions may be
defined. Whatever content format is used for this purpose should
ensure that maximum possible suppression state is conveyed to the
receiver set.
5.5. Additional Protocol Mechanisms
In addition to the principal function of data content transmission
and repair, there are some other protocol mechanisms that help NORM
to adapt to network conditions and play fairly with other coexistent
protocols.
5.5.1. Greatest Round-trip Time Collection
For NORM receivers to appropriately scale backoff timeouts and the
senders to use proper corresponding timeouts, the participants must
agree on a common timeout basis. Each NORM sender monitors the
round-trip time of active receivers and determines the group greatest
round-trip time (GRTT). The sender advertises this GRTT estimate in
every message it transmits so that receivers have this value
available for scaling their timers. To measure the current GRTT, the
sender periodically sends NORM_CMD(CC) messages that contain a
locally generated timestamp. Receivers are expected to record this
timestamp along with the time the NORM_CMD(CC) message is received.
Then, when the receivers generate feedback messages to the sender, an
adjusted version of the sender timestamp is embedded in the feedback
message (NORM_NACK or NORM_ACK). The adjustment adds the amount of
time the receiver held the timestamp before generating its response.
Upon receipt of this adjusted timestamp, the sender is able to
calculate the round-trip time to that receiver.
The round-trip time for each receiver is fed into an algorithm that
weights and smoothes the values for a conservative estimate of the
GRTT. The algorithm and methodology are described in the NORM
Building Block document [4] in the section entitled "One-to-Many
Sender GRTT Measurement". A conservative estimate helps feedback
suppression at a small cost in overall protocol repair delay. The
sender’s current estimate of GRTT is advertised in the "grtt" field
found in all NORM sender messages. The advertised GRTT is also
limited to a minimum of the nominal inter-packet transmission time
given the sender’s current transmission rate and system clock
granularity. The reason for this additional limit is to keep the
receiver somewhat "event driven" by making sure the sender has had
adequate time to generate any response to repair requests from
receivers given transmit rate limitations due to congestion control
or configuration.
When the NORM-CC Rate header extension is present in NORM_CMD(CC)
messages, the receivers respond to NORM_CMD(CC) messages as described
in Section 5.5.2, "NORM Congestion Control Operation". The
NORM_CMD(CC) messages are periodically generated by the sender as
described for congestion control operation. This provides for
proactive, but controlled, feedback from the group in the form of
NORM_ACK messages. This provides for GRTT feedback even if no
NORM_NACK messages are being sent. If operating without congestion
control in a closed network, the NORM_CMD(CC) messages may be sent
periodically without the NORM-CC Rate header extension. In this
case, receivers will only provide GRTT measurement feedback when
NORM_NACK messages are generated since no NORM_ACK messages are
generated. In this case, the NORM_CMD(CC) messages may be sent less
frequently, perhaps as little as once per minute, to conserve network
capacity. Note that the NORM-CC Rate header extension may also be
used proactively solicit RTT feedback from the receiver group per
congestion control operation even though the sender may not be
conducting congestion control rate adjustment. NORM operation
without congestion control should be considered only in closed
networks.
5.5.2. NORM Congestion Control Operation
This section describes baseline congestion control operation for the
NORM protocol (NORM-CC). The supporting NORM message formats and
approach described here are an adaptation of the equation-based TCP-
Friendly Multicast Congestion Control (TFMCC) approach described in
[19]. This congestion control scheme is REQUIRED for operation
within the general Internet unless the NORM implementation is adapted
to use another IETF-sanctioned reliable multicast congestion control
mechanism (e.g., PGMCC [20]). With this TFMCC-based approach, the
transmissions of NORM senders are controlled in a rate-based manner
as opposed to window-based congestion control algorithms as in TCP.
However, it is possible that the NORM protocol message set may
alternatively be used to support a window-based multicast congestion
control scheme such as PGMCC. The details of that alternative may be
described separately or in a future revision of this document. In
either case (rate-based TFMCC or window-based PGMCC), successful
control of sender transmission depends upon collection of sender-to-
receiver packet loss estimates and RTTs to identify the congestion
control bottleneck path(s) within the multicast topology and adjust
the sender rate accordingly. The receiver with loss and RTT
estimates that correspond to the lowest result transmission rate is
identified as the "current limiting receiver" (CLR).
As described in [21], a steady-state sender transmission rate, to be
"friendly" with competing TCP flows can be calculated as:
S
Rsender = --------------------------------------------------------------
tRTT * (sqrt((2/3)*p) + 12 * sqrt((3/8)*p) * p *
(1 + 32*(p^2)))
where
S = Nominal transmitted packet size. (In NORM, the "nominal"
packet size can be determined by the sender as an
exponentially weighted moving average (EWMA) of transmitted
packet sizes to account for variable message sizes).
tRTT = The RTT estimate of the current "current limiting receiver"
(CLR).
p = The loss event fraction of the CLR.
To support congestion control feedback collection and operation, the
NORM sender periodically transmits NORM_CMD(CC) command messages.
NORM_CMD(CC) messages are multiplexed with NORM data and repair
transmissions and serve several purposes:
1) Stimulate explicit feedback from the general receiver set to
collect congestion control information.
2) Communicate state to the receiver set on the sender’s current
congestion control status including details of the CLR.
3) Initiate rapid (immediate) feedback from the CLR in order to
closely track the dynamics of congestion control for that current
"worst path" in the group multicast topology.
The format of the NORM_CMD(CC) message is describe in Section 4.2.3
of this document. The NORM_CMD(CC) message contains information to
allow measurement of RTTs, to inform the group of the congestion
control CLR, and to provide feedback of individual RTT measurements
to the receivers in the group. The NORM_CMD(CC) also provides for
exciting feedback from OPTIONAL "potential limiting receiver" (PLR)
nodes that may be determined administratively or possibly
algorithmically based on congestion control feedback. PLR nodes are
receivers that have been identified to have potential for (perhaps
soon) becoming the CLR and thus immediate, up-to-date feedback is
beneficial for congestion control performance. The details of PLR
selection are not discussed in this document.
5.5.2.1. NORM_CMD(CC) Transmission
The NORM_CMD(CC) message is transmitted periodically by the sender
along with its normal data transmission. Note that the repeated
transmission of NORM_CMD(CC) messages may be initiated some time
before transmission of user data content at session startup. This
may be done to collect some estimation of the current state of the
multicast topology with respect to group and individual RTT and
congestion control state.
A NORM_CMD(CC) message is immediately transmitted at sender startup.
The interval of subsequent NORM_CMD(CC) message transmission is
determined as follows:
1) By default, the interval is set according to the current sender
GRTT estimate. A startup GRTT of 0.5 seconds is recommended when
no feedback has yet been received from the group.
2) If a CLR has been identified (based on previous receiver
feedback), the interval is the RTT between the sender and CLR.
3) Additionally, if the interval of nominal data message transmission
is greater than the GRTT or RTT_clr interval, the NORM_CMD(CC)
interval is set to this greater value. This ensures that the
transmission of this control message is not done to the exclusion
of user data transmission.
The NORM_CMD(CC) "cc_sequence" field is incremented with each
transmission of a NORM_CMD(CC) command. The greatest "cc_sequence"
recently received by receivers is included in their feedback to the
sender. This allows the sender to determine the "age" of feedback to
assist in congestion avoidance.
The NORM-CC Rate Header Extension is applied to the NORM_CMD(CC)
message and the sender advertises its current transmission rate in
the "send_rate" field. The rate information is used by receivers to
initialize loss estimation during congestion control startup or
restart.
The "cc_node_list" contains a list of entries identifying receivers
and their current congestion control state (status "flags", "rtt" and
"loss" estimates). The list may be empty if the sender has not yet
received any feedback from the group. If the sender has received
feedback, the list will minimally contain an entry identifying the
CLR. A NORM_FLAG_CC_CLR flag value is provided for the "cc_flags"
field to identify the CLR entry. It is RECOMMENDED that the CLR
entry be the first in the list for implementation efficiency.
Additional entries in the list are used to provide sender-measured
individual RTT estimates to receivers in the group. The number of
additional entries in this list is dependent upon the percentage of
control traffic the sender application is willing to send with
respect to user data message transmissions. More entries in the list
may allow the sender to be more responsive to congestion control
dynamics. The length of the list may be dynamically determined
according to the current transmission rate and scheduling of
NORM_CMD(CC) messages. The maximum length of the list corresponds to
the sender’s NormSegmentSize parameter for the session. The
inclusion of additional entries in the list based on receiver
feedback are prioritized with following rules:
1) Receivers that have not yet been provided RTT feedback get first
priority. Of these, those with the greatest loss fraction receive
precedence for list inclusion.
2) Secondly, receivers that have previously been provided RTT are
included with receivers yielding the lowest calculated congestion
rate getting precedence.
There are "cc_flag" values in addition to NORM_FLAG_CC_CLR that are
used for other congestion control functions. The NORM_FLAG_CC_PLR
flag value is used to mark additional receivers from that the sender
would like to have immediate, non-suppressed feedback. These may be
receivers that the sender algorithmically identified as potential
future CLRs or that have been pre-configured as potential congestion
control points in the network. The NORM_FLAG_CC_RTT indicates the
validity of the "cc_rtt" field for the associated receiver node.
Normally, this flag will be set since the receivers in the list will
typically be receivers from which the sender has received feedback.
However, in the case that the NORM sender has been pre-configured
with a set of PLR nodes, feedback from those receivers may not yet
have been collected and thus the "cc_rtt" and "cc_rate" fields do not
contain valid values when this flag is not set.
5.5.2.2. NORM_CMD(CC) Feedback Response
Receivers explicitly respond to NORM_CMD(CC) messages in the form of
a NORM_ACK(RTT) message. The goal of the congestion control feedback
is to determine the receivers with the lowest congestion control
rates. Receivers that are marked as CLR or PLR nodes in the
NORM_CMD(CC) "cc_node_list" immediately provide feedback in the form
of a NORM_ACK to this message. When a NORM_CMD(CC) is received,
non-CLR or non-PLR nodes initiate random feedback backoff timeouts
similar to that used when the receiver initiates a repair cycle (see
Section 5.3) in response to detection of data loss. The backoff
timeout for the congestion control response is generated as follows:
T_backoff = RandomBackoff(K*GRTTsender, GSIZEsender)
The "RandomBackoff()" algorithm provides a truncated exponentially
distributed random number and is described in the NORM Building Block
document [4]. The same backoff factor K = Ksender MAY be used as
with NORM_NACK suppression. However, in cases where the application
purposefully specifies a very small Ksender backoff factor to
minimize the NACK repair process latency (trading off group size
scalability), it may still be desirable to maintain a larger backoff
factor for congestion control feedback, since there may often be a
larger volume of congestion control feedback than NACKs in many cases
and congestion control feedback latency may be tolerable where
reliable delivery latency is not. As previously noted, a backoff
factor value of K = 4 is generally recommended for ASM operation and
K = 6 for SSM operation. A receiver SHALL cancel the backoff timeout
and thus its pending transmission of a NORM_ACK(RTT) message under
the following conditions:
1) The receiver generates another feedback message (NORM_NACK or
other NORM_ACK) before the congestion control feedback timeout
expires,
2) A NORM_CMD(CC) or other receiver feedback with an ordinally
greater "cc_sequence" field value is received before the
congestion control feedback timeout expires (this is similar to
the TFMCC feedback round number),
3) When the T_backoff is greater than 1*GRTT. This prevents NACK
implosion in the event of sender or network failure,
4) "Suppressing" congestion control feedback is heard from another
receiver (in a NORM_ACK or NORM_NACK) or via a
NORM_CMD(REPAIR_ADV) message from the sender. The local
receiver’s feedback is "suppressed" if the rate of the competing
feedback (Rfb) is sufficiently close to or less than the local
receiver’s calculated rate (Rcalc). The local receiver’s feedback
is canceled when:
Rcalc > (0.9 * Rfb)
Also note receivers that have not yet received an RTT measurement
from the sender are suppressed only by other receivers that have
not yet measured RTT. Additionally, receivers whose RTT estimate
has "aged" considerably (i.e., they haven’t been included in the
NORM_CMD(CC) "cc_node_list" in a long time) may wish to compete as
a receiver with no prior RTT measurement after some expiration
period.
When the backoff timer expires, the receiver SHALL generate a
NORM_ACK(RTT) message to provide feedback to the sender and group.
This message may be multicast to the group for most effective
suppression in ASM topologies or unicast to the sender depending upon
how the NORM protocol is deployed and configured.
Whenever any feedback is generated (including this NORM_ACK(RTT)
message), receivers include an adjusted version of the sender
timestamp from the most recently received NORM_CMD(CC) message and
the "cc_sequence" value from that command in the applicable NORM_ACK
or NORM_NACK message fields. For NORM-CC operation, any generated
feedback message SHALL also contain the NORM-CC Feedback header
extension. The receiver provides its current "cc_rate" estimate,
"cc_loss" estimate, "cc_rtt" if known, and any applicable "cc_flags"
via this header extension.
During slow start (when the receiver has not yet detected loss from
the sender), the receiver uses a value equal to two times its
measured rate from the sender in the "cc_rate" field. For steady-
state congestion control operation, the receiver "cc_rate" value is
from the equation-based value using its current loss event estimate
and sender<->receiver RTT information. (The GRTT is used when the
receiver has not yet measured its individual RTT).
The "cc_loss" field value reflects the receiver’s current loss event
estimate with respect to the sender in question.
When the receiver has a valid individual RTT measurement, it SHALL
include this value in the "cc_rtt" field. The NORM_FLAG_CC_RTT MUST
be set when the "cc_rtt" field is valid.
After a congestion control feedback message is generated or when the
feedback is suppressed, a non-CLR receiver begins a "holdoff" timeout
period during which it will restrain itself from providing congestion
control feedback, even if NORM_CMD(CC) messages are received from the
sender (unless the receive becomes marked as a CLR or PLR node). The
value of this holdoff timeout (T_ccHoldoff) period is:
T_ccHoldoff = (K*GRTT)
Thus, non-CLR receivers are constrained to providing explicit
congestion control feedback once per K*GRTT intervals. Note,
however, that as the session progresses, different receivers will be
responding to different NORM_CMD(CC) messages and there will be
relatively continuous feedback of congestion control information
while the sender is active.
5.5.2.3. Congestion Control Rate Adjustment
During steady-state operation, the sender will directly adjust its
transmission rate to the rate indicated by the feedback from its
currently selected CLR. As noted in [19], the estimation of
parameters (loss and RTT) for the CLR will generally constrain the
rate changes possible within acceptable bounds. For rate increases,
the sender SHALL observe a maximum rate of increase of one packet per
RTT at all times during steady-state operation.
The sender processes congestion control feedback from the receivers
and selects the CLR based on the lowest rate receiver. Receiver
rates are either determined directly from the slow start "cc_rate"
provided by the receiver in the NORM-CC Feedback header extension or
by performing the equation-based calculation using individual RTT and
loss estimates ("cc_loss") as feedback is received.
The sender can calculate a current RTT for a receiver (RTT_rcvrNew)
using the "grtt_response" timestamp included in feedback messages.
When the "cc_rtt" value in a response is not valid, the sender simply