<----------RTP Packet Payload----------->
Figure 1: Data sections within an RTP packet
The first data section is the AU (Access Unit) Header Section, that
contains one or more AU-headers; however, each AU-header MAY be
empty, in which case the entire AU Header Section is empty. The
second section is the Auxiliary Section, containing auxiliary data;
this section MAY also be configured empty. The third section is the
Access Unit Data Section, containing either a single fragment of one
Access Unit or one or more complete Access Units. The Access Unit
Data Section MUST NOT be empty.
2.12. Modes to Transport MPEG-4 Streams
While it is possible to build fully configurable receivers capable of
receiving any MPEG-4 stream, this specification also allows for the
design of simplified, but dedicated receivers, that are for example,
capable of receiving only one type of MPEG-4 stream. This is
achieved by requiring that specific modes be defined in order to use
this specification. Each mode may define constraints for transport
of one or more types of MPEG-4 streams, for instance on the payload
configuration.
The applied mode MUST be signaled. Signaling the mode is
particularly important for receivers that are only capable of
decoding one or more specific modes. Such receivers need to
determine whether the applied mode is supported, so as to avoid
problems with processing of payloads that are beyond the capabilities
of the receiver.
In this document several modes are defined for the transportation of
MPEG-4 CELP and AAC streams, as well as a generic mode that can be
used for any MPEG-4 stream. In the future, new RFCs may specify
other modes of using this specification. However, each mode MUST be
in full compliance with this specification (see section 3.3.7).
2.13. Alignment with RFC 3016
This payload can be configured as nearly identical to the payload
format defined in RFC 3016 [12] for the MPEG-4 video configurations
recommended in RFC 3016. Hence, receivers that comply with RFC 3016
can decode such RTP payload, provided that additional packets
containing video decoder configuration (VO, VOL, VOSH) are inserted
in the stream, as required by RFC 3016 [12]. Conversely, receivers
that comply with the specification in this document SHOULD be able to
decode payloads, names and parameters defined for MPEG-4 video in RFC
3016 [12]. In this respect, it is strongly RECOMMENDED that the
implementation provide the ability to ignore "in band" video decoder
configuration packets that may be found in streams conforming to the
RFC 3016 video payload.
Note the "out of band" availability of the video decoder
configuration is optional in RFC 3016 [12]. To achieve maximum
interoperability with the RTP payload format defined in this
document, applications that use RFC 3016 to transport MPEG-4 video
(part 2) are recommended to make the video decoder configuration
available as a MIME parameter.
3. Payload Format
3.1. Usage of RTP Header Fields and RTCP
Payload Type (PT): The assignment of an RTP payload type for this
packet format is outside the scope of this document; it is
specified by the RTP profile under which this payload format is
used, or signaled dynamically out-of-band (e.g., using SDP).
Marker (M) bit: The M bit is set to 1 to indicate that the RTP packet
payload contains either the final fragment of a fragmented Access
Unit or one or more complete Access Units.
Extension (X) bit: Defined by the RTP profile used.
Sequence Number: The RTP sequence number SHOULD be generated by the
sender in the usual manner with a constant random offset.
Timestamp: Indicates the sampling instant of the first AU contained
in the RTP payload. This sampling instant is equivalent to the
CTS in the MPEG-4 time domain. When using SDP, the clock rate of
the RTP time stamp MUST be expressed using the "rtpmap" attribute.
If an MPEG-4 audio stream is transported, the rate SHOULD be set
to the same value as the sampling rate of the audio stream. If an
MPEG-4 video stream is transported, it is RECOMMENDED that the
rate be set to 90 kHz.
In all cases, the sender SHALL make sure that RTP time stamps are
identical only if the RTP time stamp refers to fragments of the same
Access Unit.
According to RFC 3550 [2] (section 5.1), it is RECOMMENDED that RTP
time stamps start at a random value for security reasons. This is
not an issue for synchronization of multiple RTP streams. However,
when streams from multiple sources are to be synchronized (for
example one stream from local storage, another from an RTP streaming
server), synchronization may become impossible if the receiver only
knows the original time stamp relationships. In such cases the time
stamp relationship required for obtaining synchronization may be
provided by out of band means. The format of such information, as
well as methods to convey such information, are beyond the scope of
this specification.
SSRC: set as described in RFC 3550 [2].
CC and CSRC fields are used as described in RFC 3550 [2].
RTCP SHOULD be used as defined in RFC 3550 [2]. Note that time
stamps in RTCP Sender Reports may be used to synchronize multiple
MPEG-4 elementary streams and also to synchronize MPEG-4 streams with
non-MPEG-4 streams, in case the delivery of these streams uses RTP.
3.2. RTP Payload Structure
3.2.1. The AU Header Section
When present, the AU Header Section consists of the AU-headers-length
field, followed by a number of AU-headers, see Figure 2.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+-+
|AU-headers-length|AU-header|AU-header| |AU-header|padding|
| | (1) | (2) | | (n) | bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+-+
Figure 2: The AU Header Section
The AU-headers are configured using MIME format parameters and MAY be
empty. If the AU-header is configured empty, the AU-headers-length
field SHALL NOT be present and consequently the AU Header Section is
empty. If the AU-header is not configured empty, then the AU-
headers-length is a two octet field that specifies the length in bits
of the immediately following AU-headers, excluding the padding bits.
Each AU-header is associated with a single Access Unit (fragment)
contained in the Access Unit Data Section in the same RTP packet.
For each contained Access Unit (fragment), there is exactly one AU-
header. Within the AU Header Section, the AU-headers are bit-wise
concatenated in the order in which the Access Units are contained in
the Access Unit Data Section. Hence, the n-th AU-header refers to
the n-th AU (fragment). If the concatenated AU-headers consume a
non-integer number of octets, up to 7 zero-padding bits MUST be
inserted at the end in order to achieve octet-alignment of the AU
Header Section.
3.2.1.1. The AU-header
Each AU-header may contain the fields given in Figure 3. The length
in bits of the fields, with the exception of the CTS-flag, the
DTS-flag and the RAP-flag fields, is defined by MIME format
parameters; see section 4.1. If a MIME format parameter has the
default value of zero, then the associated field is not present. The
number of bits for fields that are present and that represent the
value of a parameter MUST be chosen large enough to correctly encode
the largest value of that parameter during the session.
If present, the fields MUST occur in the mutual order given in Figure
3. In the general case, a receiver can only discover the size of an
AU-header by parsing it since the presence of the CTS-delta and DTS-
delta fields is signaled by the value of the CTS-flag and DTS-flag,
respectively.
+---------------------------------------+
| AU-size |
+---------------------------------------+
| AU-Index / AU-Index-delta |
+---------------------------------------+
| CTS-flag |
+---------------------------------------+
| CTS-delta |
+---------------------------------------+
| DTS-flag |
+---------------------------------------+
| DTS-delta |
+---------------------------------------+
| RAP-flag |
+---------------------------------------+
| Stream-state |
+---------------------------------------+
Figure 3: The fields in the AU-header. If used, the AU-Index field
only occurs in the first AU-header within an AU Header
Section; in any other AU-header, the AU-Index-delta field
occurs instead.
AU-size: Indicates the size in octets of the associated Access Unit
in the Access Unit Data Section in the same RTP packet. When the
AU-size is associated with an AU fragment, the AU size indicates
the size of the entire AU and not the size of the fragment. In
this case, the size of the fragment is known from the size of the
AU data section. This can be exploited to determine whether a
packet contains an entire AU or a fragment, which is particularly
useful after losing a packet carrying the last fragment of an AU.
AU-Index: Indicates the serial number of the associated Access Unit
(fragment). For each (in decoding order) consecutive AU or AU
fragment, the serial number is incremented by 1. When present,
the AU-Index field occurs in the first AU-header in the AU Header
Section, but MUST NOT occur in any subsequent (non-first) AU-
header in that Section. To encode the serial number in any such
non-first AU-header, the AU-Index-delta field is used.
AU-Index-delta: The AU-Index-delta field is an unsigned integer that
specifies the serial number of the associated AU as the difference
with respect to the serial number of the previous Access Unit.
Hence, for the n-th (n>1) AU, the serial number is found from:
AU-Index(n) = AU-Index(n-1) + AU-Index-delta(n) + 1
If the AU-Index field is present in the first AU-header in the AU
Header Section, then the AU-Index-delta field MUST be present in
any subsequent (non-first) AU-header. When the AU-Index-delta is
coded with the value 0, it indicates that the Access Units are
consecutive in decoding order. An AU-Index-delta value larger
than 0 signals that interleaving is applied.
CTS-flag: Indicates whether the CTS-delta field is present. A value
of 1 indicates that the field is present, a value of 0 indicates
that it is not present.
The CTS-flag field MUST be present in each AU-header if the length
of the CTS-delta field is signaled to be larger than zero. In
that case, the CTS-flag field MUST have the value 0 in the first
AU-header and MAY have the value 1 in all non-first AU-headers.
The CTS-flag field SHOULD be 0 for any non-first fragment of an
Access Unit.
CTS-delta: Encodes the CTS by specifying the value of CTS as a 2’s
complement offset (delta) from the time stamp in the RTP header of
this RTP packet. The CTS MUST use the same clock rate as the time
stamp in the RTP header.
DTS-flag: Indicates whether the DTS-delta field is present. A value
of 1 indicates that DTS-delta is present, a value of 0 indicates
that it is not present.
The DTS-flag field MUST be present in each AU-header if the length
of the DTS-delta field is signaled to be larger than zero. The
DTS-flag field MUST have the same value for all fragments of an
Access Unit.
DTS-delta: Specifies the value of the DTS as a 2’s complement offset
(delta) from the CTS. The DTS MUST use the same clock rate as the
time stamp in the RTP header. The DTS-delta field MUST have the
same value for all fragments of an Access Unit.
RAP-flag: When set to 1, indicates that the associated Access Unit
provides a random access point to the content of the stream. If
an Access Unit is fragmented, the RAP flag, if present, MUST be
set to 0 for each non-first fragment of the AU.
Stream-state: Specifies the state of the stream for an AU of an
MPEG-4 system stream; each state is identified by a value of a
modulo counter. In ISO/IEC 14496-1, MPEG-4 system streams use the
AU_SequenceNumber to signal stream states. When the stream state
changes, the value of the stream-state MUST be incremented by one.
Note: no relation is required between stream-states of different
streams.
3.2.2. The Auxiliary Section
The Auxiliary Section consists of the auxiliary-data-size field
followed by the auxiliary-data field. Receivers MAY (but are not
required to) parse the auxiliary-data field; to facilitate skipping
of the auxiliary-data field by receivers, the auxiliary-data-size
field indicates the length in bits of the auxiliary-data. If the
concatenation of the auxiliary-data-size and the auxiliary-data
fields consume a non-integer number of octets, up to 7 zero padding
bits MUST be inserted immediately after the auxiliary data in order
to achieve octet-alignment. See Figure 4.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+
| auxiliary-data-size | auxiliary-data |padding bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+
Figure 4: The fields in the Auxiliary Section
The length in bits of the auxiliary-data-size field is configurable
by a MIME format parameter; see section 4.1. The default length of
zero indicates that the entire Auxiliary Section is absent.
auxiliary-data-size: specifies the length in bits of the immediately
following auxiliary-data field;
auxiliary-data: the auxiliary-data field contains data of a format
not defined by this specification.
3.2.3. The Access Unit Data Section
The Access Unit Data Section contains an integer number of complete
Access Units or a single fragment of one AU. The Access Unit Data
Section is never empty. If data of more than one Access Unit is
present, then the AUs are concatenated into a contiguous string of
octets. See Figure 5. The AUs inside the Access Unit Data Section
MUST be in decoding order, though not necessarily contiguous in the
case of interleaving.
The size and number of Access Units SHOULD be adjusted such that the
resulting RTP packet is not larger than the path MTU. To handle
larger packets, this payload format relies on lower layers for
fragmentation, which may result in reduced performance.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|AU(1) |
+ |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |AU(2) |
+-+-+-+-+-+-+-+-+ |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | AU(n) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|AU(n) continued|
|-+-+-+-+-+-+-+-+
Figure 5: Access Unit Data Section; each AU is octet-aligned.
When multiple Access Units are carried, the size of each AU MUST be
made available to the receiver. If the AU size is variable, then the
size of each AU MUST be indicated in the AU-size field of the
corresponding AU-header. However, if the AU size is constant for a
stream, this mechanism SHOULD NOT be used; instead, the fixed size
SHOULD be signaled by the MIME format parameter "constantSize"; see
section 4.1.
The absence of both AU-size in the AU-header and the constantSize
MIME format parameter indicates the carriage of a single AU
(fragment), i.e., that a single Access Unit (fragment) is transported
in each RTP packet for that stream.
3.2.3.1. Fragmentation
A packet SHALL carry either one or more complete Access Units, or a
single fragment of an Access Unit. Fragments of the same Access Unit
have the same time stamp but different RTP sequence numbers. The
marker bit in the RTP header is 1 on the last fragment of an Access
Unit, and 0 on all other fragments.
3.2.3.2. Interleaving
Unless prohibited by the signaled mode, a sender MAY interleave
Access Units. Receivers that are capable of receiving modes that
support interleaving MUST be able to decode interleaved Access Units.
When a sender interleaves Access Units, it needs to provide
sufficient information to enable a receiver to unambiguously
reconstruct the original order, even in the case of out-of-order
packets, packet loss or duplication. The information that senders
need to provide depends on whether or not the Access Units have a
constant time duration. Access Units have a constant time duration,
if:
TS(i+1) - TS(i) = constant
for any i, where:
i indicates the index of the AU in the original order, and
TS(i) denotes the time stamp of AU(i)
The MIME parameter "constantDuration" SHOULD be used to signal that
Access Units have a constant time duration; see section 4.1.
If the "constantDuration" parameter is present, the receiver can
reconstruct the original Access Unit timing based solely on the RTP
timestamp and AU-Index-delta. Accordingly, when transmitting Access
Units of constant duration, the AU-Index, if present, MUST be set to
the value 0. Receivers of constant duration Access Units MUST use
the RTP timestamp to determine the index of the first AU in the RTP
packet. The AU-Index-delta header and the signaled
"constantDuration" are used to reconstruct AU timing.
If the "constantDuration" parameter is not present, then senders MAY
signal AUs of constant duration by coding the AU-Index with zero in
each RTP packet. In the absence of the constantDuration parameter
receivers MUST conclude that the AUs have constant duration if the
AU-index is zero in two consecutive RTP packets.
When transmitting Access Units of variable duration, then the
"constantDuration" parameter MUST NOT be present, and the transmitter
MUST use the AU-Index to encode the index information required for
re-ordering, and the receiver MUST use that value to determine the
index of each AU in the RTP packet. The number of bits of the AU-
Index field MUST be chosen so that valid index information is
provided at the applied interleaving scheme, without causing problems
due to roll-over of the AU-Index field. In addition, the CTS-delta
MUST be coded in the AU header for each non-first AU in the RTP
packet, so that receivers can place the AUs correctly in time.
When interleaving is applied, a de-interleave buffer is needed in
receivers to put the Access Units in their correct logical
consecutive decoding order. This requires the computation of the
time stamp for each Access Unit. In case of a constant time duration
per Access Unit, the time stamp of the i-th access unit in an RTP
packet with RTP time stamp T is calculated as follows:
Timestamp[0] = T
Timestamp[i, i > 0] = T +(Sum(for k=1 to i of (AU-Index-delta[k]
+ 1))) * access-unit-duration
When AU-Index-delta is always 0, this reduces to T + i * (access-
unit-duration). This is the non-interleaved case, where the frames
are consecutive in decoding order. Note that the AU-Index field
(present for the first Access Unit) is indeed not needed in this
calculation.
3.2.3.3. Constraints for Interleaving
The size of the packets should be suitably chosen to be appropriate
to both the path MTU and the capacity of the receiver’s de-interleave
buffer. The maximum packet size for a session SHOULD be chosen to
not exceed the path MTU.
To allow receivers to allocate sufficient resources for de-
interleaving, senders MUST provide the information to receivers as
specified in this section.
AUs enter the decoder in decoding order. The de-interleave buffer is
used to re-order a stream of interleaved AUs back into decoding
order. When interleaving is applied, the decoding of "early" AUs has
to be postponed until all AUs that precede it in decoding order are
present. Therefore, these "early" AUs are stored in the de-
interleave buffer. As an example in Figure 6, the interleaving
pattern from section 2.5 is considered.
+--+--+--+--+--+--+--+--+--+--+--+-
Interleaved AUs | 0| 3| 6| 1| 4| 7| 2| 5| 8| 9|12|..
+--+--+--+--+--+--+--+--+--+--+--+-
Storage of "early" AUs 3 3 3 3 3 3
6 6 6 6 6 6
4 4 4
7 7 7
12 12
Figure 6: Storage of "early" AUs in the de-interleave buffer per
interleaved AU.
AU(3) is to be delivered to the decoder after AU(0), AU(1) and AU(2);
of these AUs, AU(2) arrives from the network last and hence AU(3)
needs to be stored until AU(2) is present in the pattern. Similarly,
AU(6) is to be stored until AU(5) is present, while AU(4) and AU(7)
are to be stored until AU(2) and AU(5) are present, respectively.
Note that the fullness of the de-interleave buffer varies in time.
In Figure 6, the de-interleave buffer contains at most 4, but often
less AUs.