: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 12. An RTP packet including a multi-time aggregation
packet of type MTAP16 and two multi-time aggregation
units
Figure 13 presents an example of an RTP packet that contains a
multi-time aggregation packet of type MTAP24 that contains two
multi-time aggregation units, labeled as 1 and 2 in the figure.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|MTAP24 NAL HDR | decoding order number base | NALU 1 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Size | NALU 1 DOND | NALU 1 TS offs |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|NALU 1 TS offs | NALU 1 HDR | NALU 1 DATA |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 SIZE | NALU 2 DOND |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 TS offset | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 DATA |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 13. An RTP packet including a multi-time aggregation
packet of type MTAP24 and two multi-time aggregation
units
5.8. Fragmentation Units (FUs)
This payload type allows fragmenting a NAL unit into several RTP
packets. Doing so on the application layer instead of relying on
lower layer fragmentation (e.g., by IP) has the following advantages:
o The payload format is capable of transporting NAL units bigger
than 64 kbytes over an IPv4 network that may be present in pre-
recorded video, particularly in High Definition formats (there is
a limit of the number of slices per picture, which results in a
limit of NAL units per picture, which may result in big NAL
units).
o The fragmentation mechanism allows fragmenting a single picture
and applying generic forward error correction as described in
section 12.5.
Fragmentation is defined only for a single NAL unit and not for any
aggregation packets. A fragment of a NAL unit consists of an integer
number of consecutive octets of that NAL unit. Each octet of the NAL
unit MUST be part of exactly one fragment of that NAL unit.
Fragments of the same NAL unit MUST be sent in consecutive order with
ascending RTP sequence numbers (with no other RTP packets within the
same RTP packet stream being sent between the first and last
fragment). Similarly, a NAL unit MUST be reassembled in RTP sequence
number order.
When a NAL unit is fragmented and conveyed within fragmentation units
(FUs), it is referred to as a fragmented NAL unit. STAPs and MTAPs
MUST NOT be fragmented. FUs MUST NOT be nested; i.e., an FU MUST NOT
contain another FU.
The RTP timestamp of an RTP packet carrying an FU is set to the NALU
time of the fragmented NAL unit.
Figure 14 presents the RTP payload format for FU-As. An FU-A
consists of a fragmentation unit indicator of one octet, a
fragmentation unit header of one octet, and a fragmentation unit
payload.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 14. RTP payload format for FU-A
Figure 15 presents the RTP payload format for FU-Bs. An FU-B
consists of a fragmentation unit indicator of one octet, a
fragmentation unit header of one octet, a decoding order number (DON)
(in network byte order), and a fragmentation unit payload. In other
words, the structure of FU-B is the same as the structure of FU-A,
except for the additional DON field.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | DON |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 15. RTP payload format for FU-B
NAL unit type FU-B MUST be used in the interleaved packetization mode
for the first fragmentation unit of a fragmented NAL unit. NAL unit
type FU-B MUST NOT be used in any other case. In other words, in the
interleaved packetization mode, each NALU that is fragmented has an
FU-B as the first fragment, followed by one or more FU-A fragments.
The FU indicator octet has the following format:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+
Values equal to 28 and 29 in the Type field of the FU indicator octet
identify an FU-A and an FU-B, respectively. The use of the F bit is
described in section 5.3. The value of the NRI field MUST be set
according to the value of the NRI field in the fragmented NAL unit.
The FU header has the following format:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|S|E|R| Type |
+---------------+
S: 1 bit
When set to one, the Start bit indicates the start of a fragmented
NAL unit. When the following FU payload is not the start of a
fragmented NAL unit payload, the Start bit is set to zero.
E: 1 bit
When set to one, the End bit indicates the end of a fragmented NAL
unit, i.e., the last byte of the payload is also the last byte of
the fragmented NAL unit. When the following FU payload is not the
last fragment of a fragmented NAL unit, the End bit is set to
zero.
R: 1 bit
The Reserved bit MUST be equal to 0 and MUST be ignored by the
receiver.
Type: 5 bits
The NAL unit payload type as defined in table 7-1 of [1].
The value of DON in FU-Bs is selected as described in section 5.5.
Informative note: The DON field in FU-Bs allows gateways to
fragment NAL units to FU-Bs without organizing the incoming NAL
units to the NAL unit decoding order.
A fragmented NAL unit MUST NOT be transmitted in one FU; i.e., the
Start bit and End bit MUST NOT both be set to one in the same FU
header.
The FU payload consists of fragments of the payload of the fragmented
NAL unit so that if the fragmentation unit payloads of consecutive
FUs are sequentially concatenated, the payload of the fragmented NAL
unit can be reconstructed. The NAL unit type octet of the fragmented
NAL unit is not included as such in the fragmentation unit payload,
but rather the information of the NAL unit type octet of the
fragmented NAL unit is conveyed in F and NRI fields of the FU
indicator octet of the fragmentation unit and in the type field of
the FU header. A FU payload MAY have any number of octets and MAY be
empty.
Informative note: Empty FUs are allowed to reduce the latency of a
certain class of senders in nearly lossless environments. These
senders can be characterized in that they packetize NALU fragments
before the NALU is completely generated and, hence, before the
NALU size is known. If zero-length NALU fragments were not
allowed, the sender would have to generate at least one bit of
data of the following fragment before the current fragment could
be sent. Due to the characteristics of H.264, where sometimes
several macroblocks occupy zero bits, this is undesirable and can
add delay. However, the (potential) use of zero-length NALUs
should be carefully weighed against the increased risk of the loss
of the NALU because of the additional packets employed for its
transmission.
If a fragmentation unit is lost, the receiver SHOULD discard all
following fragmentation units in transmission order corresponding to
the same fragmented NAL unit.
A receiver in an endpoint or in a MANE MAY aggregate the first n-1
fragments of a NAL unit to an (incomplete) NAL unit, even if fragment
n of that NAL unit is not received. In this case, the
forbidden_zero_bit of the NAL unit MUST be set to one to indicate a
syntax violation.
6. Packetization Rules
The packetization modes are introduced in section 5.2. The
packetization rules common to more than one of the packetization
modes are specified in section 6.1. The packetization rules for the
single NAL unit mode, the non-interleaved mode, and the interleaved
mode are specified in sections 6.2, 6.3, and 6.4, respectively.
6.1. Common Packetization Rules
All senders MUST enforce the following packetization rules regardless
of the packetization mode in use:
o Coded slice NAL units or coded slice data partition NAL units
belonging to the same coded picture (and thus sharing the same RTP
timestamp value) MAY be sent in any order permitted by the
applicable profile defined in [1]; however, for delay-critical
systems, they SHOULD be sent in their original coding order to
minimize the delay. Note that the coding order is not necessarily
the scan order, but the order the NAL packets become available to
the RTP stack.
o Parameter sets are handled in accordance with the rules and
recommendations given in section 8.4.
o MANEs MUST NOT duplicate any NAL unit except for sequence or
picture parameter set NAL units, as neither this memo nor the
H.264 specification provides means to identify duplicated NAL
units. Sequence and picture parameter set NAL units MAY be
duplicated to make their correct reception more probable, but any
such duplication MUST NOT affect the contents of any active
sequence or picture parameter set. Duplication SHOULD be
performed on the application layer and not by duplicating RTP
packets (with identical sequence numbers).
Senders using the non-interleaved mode and the interleaved mode MUST
enforce the following packetization rule:
o MANEs MAY convert single NAL unit packets into one aggregation
packet, convert an aggregation packet into several single NAL unit
packets, or mix both concepts, in an RTP translator. The RTP
translator SHOULD take into account at least the following
parameters: path MTU size, unequal protection mechanisms (e.g.,
through packet-based FEC according to RFC 2733 [18], especially
for sequence and picture parameter set NAL units and coded slice
data partition A NAL units), bearable latency of the system, and
buffering capabilities of the receiver.
Informative note: An RTP translator is required to handle RTCP as
per RFC 3550.
6.2. Single NAL Unit Mode
This mode is in use when the value of the OPTIONAL packetization-mode
MIME parameter is equal to 0, the packetization-mode is not present,
or no other packetization mode is signaled by external means. All
receivers MUST support this mode. It is primarily intended for low-
delay applications that are compatible with systems using ITU-T
Recommendation H.241 [15] (see section 12.1). Only single NAL unit
packets MAY be used in this mode. STAPs, MTAPs, and FUs MUST NOT be
used. The transmission order of single NAL unit packets MUST comply
with the NAL unit decoding order.
6.3. Non-Interleaved Mode
This mode is in use when the value of the OPTIONAL packetization-mode
MIME parameter is equal to 1 or the mode is turned on by external
means. This mode SHOULD be supported. It is primarily intended for
low-delay applications. Only single NAL unit packets, STAP-As, and
FU-As MAY be used in this mode. STAP-Bs, MTAPs, and FU-Bs MUST NOT
be used. The transmission order of NAL units MUST comply with the
NAL unit decoding order.
6.4. Interleaved Mode
This mode is in use when the value of the OPTIONAL packetization-mode
MIME parameter is equal to 2 or the mode is turned on by external
means. Some receivers MAY support this mode. STAP-Bs, MTAPs, FU-As,
and FU-Bs MAY be used. STAP-As and single NAL unit packets MUST NOT
be used. The transmission order of packets and NAL units is
constrained as specified in section 5.5.
7. De-Packetization Process (Informative)
The de-packetization process is implementation dependent. Therefore,
the following description should be seen as an example of a suitable
implementation. Other schemes may be used as well. Optimizations
relative to the described algorithms are likely possible. Section
7.1 presents the de-packetization process for the single NAL unit and
non-interleaved packetization modes, whereas section 7.2 describes
the process for the interleaved mode. Section 7.3 includes
additional decapsulation guidelines for intelligent receivers.
All normal RTP mechanisms related to buffer management apply. In
particular, duplicated or outdated RTP packets (as indicated by the
RTP sequences number and the RTP timestamp) are removed. To
determine the exact time for decoding, factors such as a possible
intentional delay to allow for proper inter-stream synchronization
must be factored in.
7.1. Single NAL Unit and Non-Interleaved Mode
The receiver includes a receiver buffer to compensate for
transmission delay jitter. The receiver stores incoming packets in
reception order into the receiver buffer. Packets are decapsulated
in RTP sequence number order. If a decapsulated packet is a single
NAL unit packet, the NAL unit contained in the packet is passed
directly to the decoder. If a decapsulated packet is an STAP-A, the
NAL units contained in the packet are passed to the decoder in the
order in which they are encapsulated in the packet. If a
decapsulated packet is an FU-A, all the fragments of the fragmented
NAL unit are concatenated and passed to the decoder.
Informative note: If the decoder supports Arbitrary Slice Order,
coded slices of a picture can be passed to the decoder in any
order regardless of their reception and transmission order.
7.2. Interleaved Mode
The general concept behind these de-packetization rules is to reorder
NAL units from transmission order to the NAL unit decoding order.
The receiver includes a receiver buffer, which is used to compensate
for transmission delay jitter and to reorder packets from
transmission order to the NAL unit decoding order. In this section,
the receiver operation is described under the assumption that there
is no transmission delay jitter. To make a difference from a
practical receiver buffer that is also used for compensation of
transmission delay jitter, the receiver buffer is here after called
the deinterleaving buffer in this section. Receivers SHOULD also
prepare for transmission delay jitter; i.e., either reserve separate
buffers for transmission delay jitter buffering and deinterleaving
buffering or use a receiver buffer for both transmission delay jitter
and deinterleaving. Moreover, receivers SHOULD take transmission
delay jitter into account in the buffering operation; e.g., by
additional initial buffering before starting of decoding and
playback.
This section is organized as follows: subsection 7.2.1 presents how
to calculate the size of the deinterleaving buffer. Subsection 7.2.2
specifies the receiver process how to organize received NAL units to
the NAL unit decoding order.
7.2.1. Size of the Deinterleaving Buffer
When SDP Offer/Answer model or any other capability exchange
procedure is used in session setup, the properties of the received
stream SHOULD be such that the receiver capabilities are not
exceeded. In the SDP Offer/Answer model, the receiver can indicate
its capabilities to allocate a deinterleaving buffer with the deint-
buf-cap MIME parameter. The sender indicates the requirement for the
deinterleaving buffer size with the sprop-deint-buf-req MIME
parameter. It is therefore RECOMMENDED to set the deinterleaving
buffer size, in terms of number of bytes, equal to or greater than
the value of sprop-deint-buf-req MIME parameter. See section 8.1 for
further information on deint-buf-cap and sprop-deint-buf-req MIME
parameters and section 8.2.2 for further information on their use in
SDP Offer/Answer model.
When a declarative session description is used in session setup, the
sprop-deint-buf-req MIME parameter signals the requirement for the
deinterleaving buffer size. It is therefore RECOMMENDED to set the
deinterleaving buffer size, in terms of number of bytes, equal to or
greater than the value of sprop-deint-buf-req MIME parameter.
7.2.2. Deinterleaving Process
There are two buffering states in the receiver: initial buffering and
buffering while playing. Initial buffering occurs when the RTP
session is initialized. After initial buffering, decoding and
playback is started, and the buffering-while-playing mode is used.
Regardless of the buffering state, the receiver stores incoming NAL
units, in reception order, in the deinterleaving buffer as follows.
NAL units of aggregation packets are stored in the deinterleaving
buffer individually. The value of DON is calculated and stored for
all NAL units.
The receiver operation is described below with the help of the
following functions and constants:
o Function AbsDON is specified in section 8.1.
o Function don_diff is specified in section 5.5.
o Constant N is the value of the OPTIONAL sprop-interleaving-depth
MIME type parameter (see section 8.1) incremented by 1.
Initial buffering lasts until one of the following conditions is
fulfilled:
o There are N VCL NAL units in the deinterleaving buffer.
o If sprop-max-don-diff is present, don_diff(m,n) is greater than
the value of sprop-max-don-diff, in which n corresponds to the NAL
unit having the greatest value of AbsDON among the received NAL
units and m corresponds to the NAL unit having the smallest value
of AbsDON among the received NAL units.
o Initial buffering has lasted for the duration equal to or greater
than the value of the OPTIONAL sprop-init-buf-time MIME parameter.
The NAL units to be removed from the deinterleaving buffer are
determined as follows:
o If the deinterleaving buffer contains at least N VCL NAL units,
NAL units are removed from the deinterleaving buffer and passed to
the decoder in the order specified below until the buffer contains
N-1 VCL NAL units.
o If sprop-max-don-diff is present, all NAL units m for which
don_diff(m,n) is greater than sprop-max-don-diff are removed from
the deinterleaving buffer and passed to the decoder in the order
specified below. Herein, n corresponds to the NAL unit having the
greatest value of AbsDON among the received NAL units.
The order in which NAL units are passed to the decoder is specified
as follows:
o Let PDON be a variable that is initialized to 0 at the beginning
of the an RTP session.
o For each NAL unit associated with a value of DON, a DON distance
is calculated as follows. If the value of DON of the NAL unit is