Request for Comments: 3640 Philips Electronics
Category: Standards Track D. Mackie
Apple Computer
V. Swaminathan
Sun Microsystems Inc.
D. Singer
Apple Computer
P. Gentric
Philips Electronics
November 2003
RTP Payload Format for Transport of MPEG-4 Elementary Streams
Status of this Memo
This document specifies an Internet standards track protocol for the
Internet community, and requests discussion and suggestions for
improvements. Please refer to the current edition of the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract
The Motion Picture Experts Group (MPEG) Committee (ISO/IEC JTC1/SC29
WG11) is a working group in ISO that produced the MPEG-4 standard.
MPEG defines tools to compress content such as audio-visual
information into elementary streams. This specification defines a
simple, but generic RTP payload format for transport of any non-
multiplexed MPEG-4 elementary stream.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Carriage of MPEG-4 Elementary Streams Over RTP . . . . . . . . 4
2.1. Signaling by MIME Format Parameters . . . . . . . . . . 4
2.2. MPEG Access Units . . . . . . . . . . . . . . . . . . . 5
2.3. Concatenation of Access Units . . . . . . . . . . . . . 5
2.4. Fragmentation of Access Units . . . . . . . . . . . . . 6
2.5. Interleaving . . . . . . . . . . . . . . . . . . . . . . 6
2.6. Time Stamp Information . . . . . . . . . . . . . . . . . 7
2.7. State Indication of MPEG-4 System Streams . . . . . . . 8
2.8. Random Access Indication . . . . . . . . . . . . . . . . 8
2.9. Carriage of Auxiliary Information . . . . . . . . . . . 8
2.10. MIME Format Parameters and Configuring Conditional Field 8
2.11. Global Structure of Payload Format . . . . . . . . . . . 9
2.12. Modes to Transport MPEG-4 Streams . . . . . . . . . . . 9
2.13. Alignment with RFC 3016 . . . . . . . . . . . . . . . . 10
3. Payload Format . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1. Usage of RTP Header Fields and RTCP . . . . . . . . . . 10
3.2. RTP Payload Structure . . . . . . . . . . . . . . . . . 11
3.2.1. The AU Header Section . . . . . . . . . . . . . 11
3.2.1.1. The AU-header . . . . . . . . . . . . 12
3.2.2. The Auxiliary Section . . . . . . . . . . . . . 14
3.2.3. The Access Unit Data Section . . . . . . . . . . 15
3.2.3.1. Fragmentation. . . . . . . . . . . . . 16
3.2.3.2. Interleaving . . . . . . . . . . . . . 16
3.2.3.3. Constraints for Interleaving . . . . . 17
3.2.3.4. Crucial and Non-Crucial AUs with
MPEG-4 System Data . . . . . . . . . . 20
3.3. Usage of this Specification. . . . . . . . . . . . . . . 21
3.3.1. General. . . . . . . . . . . . . . . . . . . . . 21
3.3.2. The Generic Mode . . . . . . . . . . . . . . . . 22
3.3.3. Constant Bit Rate CELP . . . . . . . . . . . . . 22
3.3.4. Variable Bit Rate CELP . . . . . . . . . . . . . 23
3.3.5. Low Bit Rate AAC . . . . . . . . . . . . . . . . 24
3.3.6. High Bit Rate AAC. . . . . . . . . . . . . . . . 25
3.3.7. Additional Modes . . . . . . . . . . . . . . . . 26
4. IANA Considerations. . . . . . . . . . . . . . . . . . . . . . 27
4.1. MIME Type Registration . . . . . . . . . . . . . . . . . 27
4.2. Registration of Mode Definitions with IANA . . . . . . . 33
4.3. Concatenation of Parameters. . . . . . . . . . . . . . . 33
4.4. Usage of SDP . . . . . . . . . . . . . . . . . . . . . . 34
4.4.1. The a=fmtp Keyword . . . . . . . . . . . . . . . 34
5. Security Considerations. . . . . . . . . . . . . . . . . . . . 34
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 35
APPENDIX: Usage of this Payload Format. . . . . . . . . . . . . . 36
Appendix A. Interleave Analysis . . . . . . . . . . . . . . . . . 36
A. Examples of Delay Analysis with Interleave. . . . . . . . . . 36
A.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 36
A.2. De-interleaving and Error Concealment . . . . . . . . . 36
A.3. Simple Group Interleave . . . . . . . . . . . . . . . . 36
A.3.1. Introduction . . . . . . . . . . . . . . . . . . 36
A.3.2. Determining the De-interleave Buffer Size . . . 37
A.3.3. Determining the Maximum Displacement . . . . . . 37
A.4. More Subtle Group Interleave . . . . . . . . . . . . . . 38
A.4.1. Introduction . . . . . . . . . . . . . . . . . . 38
A.4.2. Determining the De-interleave Buffer Size. . . . 38
A.4.3. Determining the Maximum Displacement . . . . . . 39
A.5. Continuous Interleave . . . . . . . . . . . . . . . . . 39
A.5.1. Introduction . . . . . . . . . . . . . . . . . . 39
A.5.2. Determining the De-interleave Buffer Size . . . 40
A.5.3. Determining the Maximum Displacement . . . . . . 40
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Normative References . . . . . . . . . . . . . . . . . . . . . . . 41
Informative References . . . . . . . . . . . . . . . . . . . . . . 41
Authors’ Addresses . . . . . . . . . . . . . . . . . . . . . . . . 42
Full Copyright Statement . . . . . . . . . . . . . . . . . . . . . 43
1. Introduction
The MPEG Committee is Working Group 11 (WG11) in ISO/IEC JTC1 SC29
that specified the MPEG-1, MPEG-2 and, more recently, the MPEG-4
standards [1]. The MPEG-4 standard specifies compression of audio-
visual data into, for example an audio or video elementary stream.
In the MPEG-4 standard, these streams take the form of audio-visual
objects that may be arranged into an audio-visual scene by means of a
scene description. Each MPEG-4 elementary stream consists of a
sequence of Access Units; examples of an Access Unit (AU) are an
audio frame and a video picture.
This specification defines a general and configurable payload
structure to transport MPEG-4 elementary streams, in particular
MPEG-4 audio (including speech) streams, MPEG-4 video streams and
also MPEG-4 systems streams, such as BIFS (BInary Format for Scenes),
OCI (Object Content Information), OD (Object Descriptor) and IPMP
(Intellectual Property Management and Protection) streams. The RTP
payload defined in this document is simple to implement and
reasonably efficient. It allows for optional interleaving of Access
Units (such as audio frames) to increase error resiliency in packet
loss.
Some types of MPEG-4 elementary streams include "crucial" information
whose loss cannot be tolerated. However, RTP does not provide
reliable transmission, so receipt of that crucial information is not
assured. Section 3.2.3.4 specifies how stream state is conveyed so
that the receiver can detect the loss of crucial information and
cease decoding until the next random access point has been received.
Applications transmitting streams that include crucial information,
such as OD commands, BIFS commands, or programmatic content such as
MPEG-J (Java) and ECMAScript, should include random access points, at
a suitable periodicity depending upon the probability of loss, in
order to reduce stream corruption to an acceptable level. An example
is the carousel mechanism as defined by MPEG in ISO/IEC 14496-1 [1].
Such applications may also employ additional protocols or services to
reduce the probability of loss. At the RTP layer, these measures
include payload formats and profiles for retransmission or forward
error correction (such as in RFC 2733 [10]), that must be employed
with due consideration to congestion control. Another solution that
may be appropriate for some applications is to carry RTP over TCP
(such as in RFC 2326 [8], section 10.12). At the network layer,
resource allocation or preferential service may be available to
reduce the probability of loss. For a general description of methods
to repair streaming media, see RFC 2354 [9].
Though the RTP payload format defined in this document is capable of
transporting any MPEG-4 stream, other, more specific, formats may
exist, such as RFC 3016 [12] for transport of MPEG-4 video (ISO/IEC
14496 [1] part 2).
Configuration of the payload is provided to accommodate the
transportation of any MPEG-4 stream at any possible bit rate.
However, for a specific MPEG-4 elementary stream typically only very
few configurations are needed. So as to allow for the design of
simplified, but dedicated receivers, this specification requires that
specific modes be defined for transport of MPEG-4 streams. This
document defines modes for MPEG-4 CELP and AAC streams, as well as a
generic mode that can be used to transport any MPEG-4 stream. In the
future, new RFCs are expected to specify additional modes for the
transportation of MPEG-4 streams.
The RTP payload format defined in this document specifies carriage of
system-related information that is often equivalent to the
information that may be contained in the MPEG-4 Sync Layer (SL) as
defined in MPEG-4 Systems [1]. This document does not prescribe how
to transcode or map information from the SL to fields defined in the
RTP payload format. Such processing, if any, is left to the
discretion of the application. However, to anticipate the need for
the transportation of any additional system-related information in
the future, an auxiliary field can be configured that may carry any
such data.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119 [4].
2. Carriage of MPEG-4 Elementary Streams over RTP
2.1. Signaling by MIME Format Parameters
With this payload format, a single MPEG-4 elementary stream can be
transported. Information on the type of MPEG-4 stream carried in the
payload is conveyed by MIME format parameters, as in an SDP [5]
message or by other means (see section 4). These MIME format
parameters specify the configuration of the payload. To allow for
simplified and dedicated receivers, a MIME format parameter is
available to signal a specific mode of using this payload. A mode
definition MAY include the type of MPEG-4 elementary stream, as well
as the applied configuration, so as to avoid the need for receivers
to parse all MIME format parameters. The applied mode MUST be
signaled.
2.2. MPEG Access Units
For carriage of compressed audio-visual data, MPEG defines Access
Units. An MPEG Access Unit (AU) is the smallest data entity to which
timing information is attributed. In the case of audio, an Access
Unit may represent an audio frame and in the case of video, a
picture. MPEG Access Units are octet-aligned by definition. If, for
example, an audio frame is not octet-aligned, up to 7 zero-padding
bits MUST be inserted at the end of the frame to achieve the octet-
aligned Access Units, as required by the MPEG-4 specification.
MPEG-4 decoders MUST be able to decode AUs in which such padding is
applied.
Consistent with the MPEG-4 specification, this document requires that
each MPEG-4 part 2 video Access Unit include all the coded data of a
picture, any video stream headers that may precede the coded picture
data, and any video stream stuffing that may follow it, up to but not
including the startcode indicating the start of a new video stream or
the next Access Unit.
2.3. Concatenation of Access Units
Frequently it is possible to carry multiple Access Units in one RTP
packet. This is particularly useful for audio; for example, when AAC
is used for encoding a stereo signal at 64 kbits/sec, AAC frames
contain on average, approximately 200 octets. On a LAN with a 1500
octet MTU, this would allow an average of 7 complete AAC frames to be
carried per RTP packet.
Access Units may have a fixed size in octets, but a variable size is
also possible. To facilitate parsing in the case of multiple
concatenated AUs in one RTP packet, the size of each AU is made known
to the receiver. When concatenating in the case of a constant AU
size, this size is communicated "out of band" through a MIME format
parameter. When concatenating in case of variable size AUs, the RTP
payload carries "in band" an AU size field for each contained AU.
In combination with the RTP payload length, the size information
allows the RTP payload to be split by the receiver back into the
individual AUs.
To simplify the implementation of RTP receivers, it is required that
when multiple AUs are carried in an RTP packet, each AU MUST be
complete, i.e., the number of AUs in an RTP packet MUST be integral.
In addition, an AU MUST NOT be repeated in other RTP packets; hence
repetition of an AU is only possible when using a duplicate RTP
packet.
2.4. Fragmentation of Access Units
MPEG allows for very large Access Units. Since most IP networks have
significantly smaller MTU sizes, this payload format allows for the
fragmentation of an Access Unit over multiple RTP packets. Hence,
when an IP packet is lost after IP-level fragmentation, only an AU
fragment may get lost instead of the entire AU. To simplify the
implementation of RTP receivers, an RTP packet SHALL either carry one
or more complete Access Units or a single fragment of one AU, i.e.,
packets MUST NOT contain fragments of multiple Access Units.
2.5. Interleaving
When an RTP packet carries a contiguous sequence of Access Units, the
loss of such a packet can result in a "decoding gap" for the user.
One method of alleviating this problem is to allow for the Access
Units to be interleaved in the RTP packets. For a modest cost in
latency and implementation complexity, significant error resiliency
to packet loss can be achieved.
To support optional interleaving of Access Units, this payload format
allows for index information to be sent for each Access Unit. After
informing receivers about buffer resources to allocate for de-
interleaving, the RTP sender is free to choose the interleaving
pattern without propagating this information a priori to the
receiver(s). Indeed, the sender could dynamically adjust the
interleaving pattern based on the Access Unit size, error rates, etc.
The RTP receiver does not need to know the interleaving pattern used;
it only needs to extract the index information of the Access Unit and
insert the Access Unit into the appropriate sequence in the decoding
or rendering queue. An example of interleaving is given below.
For example, if we assume that an RTP packet contains 3 AUs, and that
the AUs are numbered 0, 1, 2, 3, 4, and so forth, and if an
interleaving group length of 9 is chosen, then RTP packet(i) contains
the following AU(n):
RTP packet(0): AU(0), AU(3), AU(6)
RTP packet(1): AU(1), AU(4), AU(7)
RTP packet(2): AU(2), AU(5), AU(8)
RTP packet(3): AU(9), AU(12), AU(15)
RTP packet(4): AU(10), AU(13), AU(16) Etc.
2.6. Time Stamp Information
The RTP time stamp MUST carry the sampling instant of the first AU
(fragment) in the RTP packet. When multiple AUs are carried within
an RTP packet, the time stamps of subsequent AUs can be calculated if
the frame period of each AU is known. For audio and video, this is
possible if the frame rate is constant. However, in some cases it is
not possible to make such a calculation (for example, for variable
frame rate video, or for MPEG-4 BIFS streams carrying composition
information). To support such cases, this payload format can be
configured to carry a time stamp in the RTP payload for each
contained Access Unit. A time stamp MAY be conveyed in the RTP
payload only for non-first AUs in the RTP packet, and SHALL NOT be
conveyed for the first AU (fragment), as the time stamp for the first
AU in the RTP packet is carried by the RTP time stamp.
MPEG-4 defines two types of time stamps: the composition time stamp
(CTS) and the decoding time stamp (DTS). The CTS represents the
sampling instant of an AU, and hence the CTS is equivalent to the RTP
time stamp. The DTS may be used in MPEG-4 video streams that use
bi-directional coding, i.e., when pictures are predicted in both
forward and backward direction by using either a reference picture in
the past, or a reference picture in the future. The DTS cannot be
carried in the RTP header. In some cases, the DTS can be derived
from the RTP time stamp using frame rate information; this requires
deep parsing in the video stream, which may be considered
objectionable. If the video frame rate is variable, the required
information may not even be present in the video stream. For both
reasons, the capability has been defined to optionally carry the DTS
in the RTP payload for each contained Access Unit.
To keep the coding of time stamps efficient, each time stamp
contained in the RTP payload is coded as a difference. For the CTS,
the offset from the RTP time stamps is provided, and for the DTS, the
offset from the CTS.
2.7. State Indication of MPEG-4 System Streams
ISO/IEC 14496-1 defines states for MPEG-4 system streams. So as to
convey state information when transporting MPEG-4 system streams,
this payload format allows for the optional carriage in the RTP
payload of the stream state for each contained Access Unit. Stream
states are used to signal "crucial" AUs that carry information whose
loss cannot be tolerated and are also useful when repeating AUs
according to the carousel mechanism defined in ISO/IEC 14496-1.
2.8. Random Access Indication
Random access to the content of MPEG-4 elementary streams may be
possible at some but not all Access Units. To signal Access Units
where random access is possible, a random access point flag can
optionally be carried in the RTP payload for each contained Access
Unit. Carriage of random access points is particularly useful for
MPEG-4 system streams in combination with the stream state.
2.9. Carriage of Auxiliary Information
This payload format defines a specific field to carry auxiliary data.
The auxiliary data field is preceded by a field that specifies the
length of the auxiliary data, so as to facilitate the skipping of
data without parsing it. The coding of the auxiliary data is not
defined in this document; instead, the format, meaning and signaling
of auxiliary information is expected to be specified in one or more
future RFCs. Auxiliary information MUST NOT be transmitted until its
format, meaning and signaling have been specified and its use has
been signaled. Receivers that have knowledge of the auxiliary data
MAY decode the auxiliary data, but receivers without knowledge of
such data MUST skip the auxiliary data field.
2.10. MIME Format Parameters and Configuring Conditional Fields
To support the features described in the previous sections, several
fields are defined for carriage in the RTP payload. However, their
use strongly depends on the type of MPEG-4 elementary stream that is
carried. Sometimes a specific field is needed with a certain length,
while in other cases such a field is not needed. To be efficient in
either case, the fields to support these features are configurable by
means of MIME format parameters. In general, a MIME format parameter
defines the presence and length of the associated field. A length of
zero indicates absence of the field. As a consequence, parsing of
the payload requires knowledge of MIME format parameters. The MIME
format parameters are conveyed to the receiver via SDP [5] messages,
as specified in section 4.4.1, or through other means.
2.11. Global Structure of Payload Format
The RTP payload following the RTP header, contains three octet-
aligned data sections, of which the first two MAY be empty, see
Figure 1.
+---------+-----------+-----------+---------------+
| RTP | AU Header | Auxiliary | Access Unit |
| Header | Section | Section | Data Section |
+---------+-----------+-----------+---------------+