So as to give a rough indication of the resources needed in the
receiver for de-interleaving, the maximum displacement in time of an
AU is defined. For any AU(j) in the pattern, each AU(i) with i<j
that is not yet present can be determined. The maximum displacement
in time of an AU is the maximum difference between the time stamp of
an AU in the pattern and the time stamp of the earliest AU that is
not yet present. In other words, when considering a sequence of
interleaved AUs, then:
Maximum displacement = max{TS(i) - TS(j)}
for any i and any j>i, where:
i and j indicate the index of the AU in the interleaving
pattern, and
TS denotes the time stamp of the AU.
As an example in Figure 7, the interleaving pattern from section 2.5
is considered. For each AU in the pattern, the index is given of the
earliest of any earlier AUs not yet present. Hence for each AU(n) in
the interleaving pattern the smallest index k (with k<n) of not yet
delivered AUs is indicated. A "-" indicates that all previous AUs
are present. If the AU period is constant, the maximum displacement
equals 5 AU periods, as found for AU(6) and AU(7).
+--+--+--+--+--+--+--+--+--+--+--+-
Interleaved AUs | 0| 3| 6| 1| 4| 7| 2| 5| 8| 9|12|..
+--+--+--+--+--+--+--+--+--+--+--+-
Earliest not yet present AU - 1 1 - 2 2 - - - - 10
Figure 7: For each AU in the interleaving pattern, the earliest of
any earlier AUs not yet present
When interleaving, senders MUST signal the maximum displacement in
time during the session via the MIME format parameter
"maxDisplacement"; see section 4.1.
An estimate of the size of the de-interleave buffer is found by
multiplying the maximum displacement by the maximum bit rate:
size(de-interleave buffer) = {(maxDisplacement) * Rate(max)} / (RTP
clock frequency),
where:
Rate(max) is the maximum bit-rate of the transported stream.
Note that receivers can derive Rate(max) from the MIME format
parameters streamType, profile-level-id, and config.
However, this calculation estimates the size of the de-interleave
buffer and the required size may differ from the calculated value.
If this calculation under-estimates the size of the
de-interleave buffer, then senders, when interleaving, MUST signal a
size of the de-interleave buffer via the MIME format parameter
"de-interleaveBufferSize"; see section 4.1. If the calculation
over-estimates the size of the de-interleave buffer, then senders,
when interleaving, MAY signal a size of the de-interleave buffer via
the MIME format parameter "de-interleaveBufferSize".
The signaled size of the de-interleave buffer MUST be large enough to
contain all "early" AUs at any point in time during the session.
That is:
minimum de-interleave buffer size = max [sum {if TS(i) > TS(j) then
AU-size(i) else 0}]
for any j and any i<j, where:
i and j indicate the index of an AU in the interleaving
pattern,
TS(i) denotes the time stamp of AU(i), and
AU-size(i) denotes the size of AU(i) in number of octets.
If the "de-interleaveBufferSize" parameter is present, then the
applied buffer for de-interleaving in a receiver MUST have a size
that is at least equal to the signaled size of the de-interleave
buffer, else a size that is at least equal to the calculated size of
the de-interleave buffer.
No matter what interleaving scheme is used, the scheme must be
analyzed to calculate the applicable maxDisplacement value, as well
as the required size of the de-interleave buffer. Senders SHOULD
signal values that are not larger than the strictly required values;
if larger values are signaled, the receiver will buffer excessively.
Note that for low bit-rate material, the applied interleaving may
make packets shorter than the MTU size.
3.2.3.4. Crucial and Non-Crucial AUs with MPEG-4 System Data
Some Access Units with MPEG-4 system data, called "crucial" AUs,
carry information whose loss cannot be tolerated, either in the
presentation or in the decoder. At each crucial AU in an MPEG-4
system stream, the stream state changes. The stream-state MAY remain
constant at non-crucial AUs. In ISO/IEC 14496-1, MPEG-4 system
streams use the AU_SequenceNumber to signal stream states.
Example: Given three AUs, AU1 = "Insertion of node X", AU2 = "Set
position of node X", AU3 = "Set position of node X". AU1 is crucial,
since if it is lost, AU2 cannot be executed. However, AU2 is not
crucial, since AU3 can be executed even if AU2 is lost.
When a crucial AU is (possibly) lost, the stream is corrupted. For
example, when an AU is lost and the stream state has changed at the
next received AU, then it is possible that the lost AU was crucial.
Once corrupted, the stream remains corrupted until the next random
access point. Note that loss of non-crucial AUs does not corrupt the
stream. When a decoder starts receiving a stream, the decoder MUST
consider the stream corrupted until an AU is received that provides a
random access point.
An AU that provides a random access point, as signaled by the RAP-
flag, may or may not be crucial. Non-crucial RAP AUs provide a
"repeated" random access point for use by decoders that recently
joined the stream or that need to re-start decoding after a stream
corruption. Non-crucial RAP AUs MUST include all updates since the
last crucial RAP AU.
Upon receiving AUs, decoders are to react as follows:
a) if the RAP-flag is set to 1 and the stream-state changes, then the
AU is a crucial RAP AU, and the AU MUST be decoded.
b) if the RAP-flag is set to 1 and the stream state does not change,
then the AU is a non-crucial RAP AU, and the receiver SHOULD
decode it if the stream is corrupted. Otherwise, the decoder MUST
ignore the AU.
c) if the RAP-flag is set to 0, then the AU MUST be decoded, unless
the stream is corrupted, in which case the AU MUST be ignored.
3.3. Usage of this Specification
3.3.1. General
Usage of this specification requires definition of a mode. A mode
defines how to use this specification, as deemed appropriate.
Senders MUST signal the applied mode via the MIME format parameter
"mode", as specified in section 4.1. This specification defines a
generic mode that can be used for any MPEG-4 stream, as well as
specific modes for the transportation of MPEG-4 CELP and MPEG-4 AAC
streams, defined in ISO/IEC 14496-3 [1].
When use of this payload format is signaled using SDP [5], an
"rtpmap" attribute is part of that signaling. The same requirements
apply for the rtpmap attribute in any mode compliant to this
specification. The general form of an rtpmap attribute is:
a=rtpmap:<payload type> <encoding name>/<clock rate>[/<encoding
parameters>]
For audio streams, <encoding parameters> specifies the number of
audio channels: 2 for stereo material (see RFC 2327 [5]) and 1 for
mono. Provided no additional parameters are needed, this parameter
may be omitted for mono material, hence its default value is 1.
3.3.2. The Generic Mode
The generic mode can be used for any MPEG-4 stream. In this mode, no
mode-specific constraints are applied; hence, in the generic mode,
the full flexibility of this specification can be exploited. The
generic mode is signaled by mode=generic.
An example is given below for the transportation of a BIFS-Anim
stream. In this example carriage of multiple BIFS-Anim Access Units
is allowed in one RTP packet. The AU-header contains the AU-size
field, the CTS-flag and, if the CTS flag is set to 1, the CTS-delta
field. The number of bits of the AU-size and the CTS-delta fields
are 10 and 16, respectively. The AU-header also contains the RAP-
flag and the Stream-state of 4 bits. This results in an AU-header
with a total size of two or four octets per BIFS-Anim AU. The RTP
time stamp uses a 1 kHz clock. Note that the media type name is
video, because the BIFS-Anim stream is part of an audio-visual
presentation. For conventions on media type names, see section 4.1.
In detail:
m=video 49230 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/1000
a=fmtp:96 streamtype=3; profile-level-id=1807; mode=generic;
objectType=2; config=0842237F24001FB400094002C0; sizeLength=10;
CTSDeltaLength=16; randomAccessIndication=1;
streamStateIndication=4
Note: The a=fmtp line has been wrapped to fit the page, it comprises
a single line in the SDP file.
The hexadecimal value of the "config" parameter is the
BIFSConfiguration() as defined in ISO/IEC 14496-1. The
BIFSConfiguration() specifies that the BIFS stream is a BIFS-Anim
stream. For the description of MIME parameters, see section 4.1.
3.3.3. Constant Bit-rate CELP
This mode is signaled by mode=CELP-cbr. In this mode, one or more
complete CELP frames of fixed size can be transported in one RTP
packet; interleaving MUST NOT be used with this mode. The RTP
payload consists of one or more concatenated CELP frames, each of
equal size. CELP frames MUST NOT be fragmented when using this mode.
Both the AU Header Section and the Auxiliary Section MUST be empty.
The MIME format parameter constantSize MUST be provided to specify
the length of each CELP frame.
For example:
m=audio 49230 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/16000/1
a=fmtp:96 streamtype=5; profile-level-id=14; mode=CELP-cbr; config=
440E00; constantSize=27; constantDuration=240
Note: The a=fmtp line has been wrapped to fit the page, it comprises
a single line in the SDP file.
The hexadecimal value of the "config" parameter is the
AudioSpecificConfig()as defined in ISO/IEC 14496-3.
AudioSpecificConfig() specifies a mono CELP stream with a sampling
rate of 16 kHz at a fixed bitrate of 14.4 kb/s and 6 sub-frames per
CELP frame. For the description of MIME parameters, see section 4.1.
3.3.4. Variable Bit-rate CELP
This mode is signaled by mode=CELP-vbr. With this mode, one or more
complete CELP frames of variable size can be transported in one RTP
packet with OPTIONAL interleaving. In this mode, the largest
possible value for AU-size is greater than the maximum CELP frame
size. Because CELP frames are very small, there is no support for
fragmentation of CELP frames. Hence, CELP frames MUST NOT be
fragmented when using this mode.
In this mode, the RTP payload consists of the AU Header Section,
followed by one or more concatenated CELP frames. The Auxiliary
Section MUST be empty. For each CELP frame contained in the payload,
there MUST be a one octet AU-header in the AU Header Section to
provide:
a) the size of each CELP frame in the payload and
b) index information for computing the sequence (and hence timing) of
each CELP frame.
Transport of CELP frames requires that the AU-size field be coded
with 6 bits. Therefore, in this mode 6 bits are allocated to the
AU-size field, and 2 bits to the AU-Index(-delta) field. Each AU-
Index field MUST be coded with the value 0. In the AU Header
Section, the concatenated AU-headers are preceded by the 16-bit AU-
headers-length field, as specified in section 3.2.1.
In addition to the required MIME format parameters, the following
parameters MUST be present: sizeLength, indexLength, and
indexDeltaLength. CELP frames always have a fixed duration per
Access Unit; when interleaving in this mode, this specific duration
MUST be signaled by the MIME format parameter constantDuration. In
addition, the parameter maxDisplacement MUST be present when
interleaving.
For example:
m=audio 49230 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/16000/1
a=fmtp:96 streamtype=5; profile-level-id=14; mode=CELP-vbr; config=
440F20; sizeLength=6; indexLength=2; indexDeltaLength=2;
constantDuration=160; maxDisplacement=5
Note: The a=fmtp line has been wrapped to fit the page; it comprises
a single line in the SDP file.
The hexadecimal value of the "config" parameter is the
AudioSpecificConfig() as defined in ISO/IEC 14496-3.
AudioSpecificConfig() specifies a mono CELP stream with a sampling
rate of 16 kHz, at a bitrate that varies between 13.9 and 16.2 kb/s
and with 4 sub-frames per CELP frame. For the description of MIME
parameters, see section 4.1.
3.3.5. Low Bit-rate AAC
This mode is signaled by mode=AAC-lbr. This mode supports the
transportation of one or more complete AAC frames of variable size.
In this mode, the AAC frames are allowed to be interleaved and hence
receivers MUST support de-interleaving. The maximum size of an AAC
frame in this mode is 63 octets. AAC frames MUST NOT be fragmented
when using this mode. Hence, when using this mode, encoders MUST
ensure that the size of each AAC frame is at most 63 octets.
The payload configuration in this mode is the same as in the variable
bit-rate CELP mode as defined in 3.3.4. The RTP payload consists of
the AU Header Section, followed by concatenated AAC frames. The
Auxiliary Section MUST be empty. For each AAC frame contained in the
payload, the one octet AU-header MUST provide:
a) the size of each AAC frame in the payload and
b) index information for computing the sequence (and hence timing) of
each AAC frame.
In the AU-header Section, the concatenated AU-headers MUST be
preceded by the 16-bit AU-headers-length field, as specified in
section 3.2.1.
In addition to the required MIME format parameters, the following
parameters MUST be present: sizeLength, indexLength, and
indexDeltaLength. AAC frames always have a fixed duration per Access
Unit; when interleaving in this mode, this specific duration MUST be
signaled by the MIME format parameter constantDuration. In addition,
the parameter maxDisplacement MUST be present when interleaving.
For example:
m=audio 49230 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/22050/1
a=fmtp:96 streamtype=5; profile-level-id=14; mode=AAC-lbr; config=
1388; sizeLength=6; indexLength=2; indexDeltaLength=2;
constantDuration=1024; maxDisplacement=5
Note: The a=fmtp line has been wrapped to fit the page; it comprises
a single line in the SDP file.
The hexadecimal value of the "config" parameter is the
AudioSpecificConfig(), as defined in ISO/IEC 14496-3.
AudioSpecificConfig() specifies a mono AAC stream with a sampling
rate of 22.05 kHz. For the description of MIME parameters, see
section 4.1.
3.3.6. High Bit-rate AAC
This mode is signaled by mode=AAC-hbr. This mode supports the
transportation of variable size AAC frames. In one RTP packet,
either one or more complete AAC frames are carried, or a single
fragment of an AAC frame is carried. In this mode, the AAC frames
are allowed to be interleaved and hence receivers MUST support de-
interleaving. The maximum size of an AAC frame in this mode is 8191
octets.
In this mode, the RTP payload consists of the AU Header Section,
followed by either one AAC frame, several concatenated AAC frames or
one fragmented AAC frame. The Auxiliary Section MUST be empty. For
each AAC frame contained in the payload, there MUST be an AU-header
in the AU Header Section to provide:
a) the size of each AAC frame in the payload and
b) index information for computing the sequence (and hence timing) of
each AAC frame.
To code the maximum size of an AAC frame requires 13 bits.
Therefore, in this configuration 13 bits are allocated to the AU-
size, and 3 bits to the AU-Index(-delta) field. Thus, each AU-header
has a size of 2 octets. Each AU-Index field MUST be coded with the
value 0. In the AU Header Section, the concatenated AU-headers MUST
be preceded by the 16-bit AU-headers-length field, as specified in
section 3.2.1.
In addition to the required MIME format parameters, the following
parameters MUST be present: sizeLength, indexLength, and
indexDeltaLength. AAC frames always have a fixed duration per Access
Unit; when interleaving in this mode, this specific duration MUST be
signaled by the MIME format parameter constantDuration. In addition,
the parameter maxDisplacement MUST be present when interleaving.
For example:
m=audio 49230 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/48000/6
a=fmtp:96 streamtype=5; profile-level-id=16; mode=AAC-hbr;
config=11B0; sizeLength=13; indexLength=3;
indexDeltaLength=3; constantDuration=1024
Note: The a=fmtp line has been wrapped to fit the page; it comprises
a single line in the SDP file.
The hexadecimal value of the "config" parameter is the
AudioSpecificConfig(), as defined in ISO/IEC 14496-3.
AudioSpecificConfig() specifies a 5.1 channel AAC stream with a
sampling rate of 48 kHz. For the description of MIME parameters, see
section 4.1.
3.3.7. Additional Modes
This specification only defines the modes specified in sections 3.3.2
through 3.3.6. Additional modes are expected to be defined in future
RFCs. Each additional mode MUST be in full compliance with this
specification.
Any new mode MUST be defined such that an implementation including
all the features of this specification can decode the payload format
corresponding to this new mode. For this reason, a mode MUST NOT
specify new default values for MIME parameters. In particular, MIME
parameters that configure the RTP payload MUST be present (unless
they have the default value), even if its presence is redundant in
case the mode assigns a fixed value to a parameter. A mode may
additionally define that some MIME parameters are required instead of
optional, that some MIME parameters have fixed values (or ranges),
and that there are rules restricting its usage.
4. IANA Considerations
This section describes the MIME types and names associated with this
payload format. Section 4.1 registers the MIME types, as per RFC
2048 [3].
This format may require additional information about the mapping to
be made available to the receiver. This is done using parameters
described in the next section.
4.1. MIME Type Registration
MIME media type name: "video" or "audio" or "application"
"video" MUST be used for MPEG-4 Visual streams (ISO/IEC 14496-2) or
MPEG-4 Systems streams (ISO/IEC 14496-1) that convey information
needed for an audio/visual presentation.
"audio" MUST be used for MPEG-4 Audio streams (ISO/IEC 14496-3) or
MPEG-4 Systems streams that convey information needed for an audio
only presentation.
"application" MUST be used for MPEG-4 Systems streams (ISO/IEC
14496-1) that serve purposes other than audio/visual presentation,
e.g., in some cases when MPEG-J (Java) streams are transmitted.
Depending on the required payload configuration, MIME format
parameters may need to be available to the receiver. This is done
using the parameters described in the next section. There are
required and optional parameters.
Optional parameters are of two types: general parameters and
configuration parameters. The configuration parameters are used to
configure the fields in the AU Header section and in the auxiliary
section. The absence of any configuration parameter is equivalent to
the associated field set to its default value, which is always zero.
The absence of all configuration parameters results in a default
"basic" configuration with an empty AU-header section and an empty
auxiliary section in each RTP packet.
MIME subtype name: mpeg4-generic
Required parameters:
MIME format parameters are not case dependent; for clarity however,
both upper and lower case are used in the names of the parameters
described in this specification.
streamType:
The integer value that indicates the type of MPEG-4 stream that is
carried; its coding corresponds to the values of the streamType,
as defined in Table 9 (streamType Values) in ISO/IEC 14496-1.
profile-level-id:
A decimal representation of the MPEG-4 Profile Level indication.
This parameter MUST be used in the capability exchange or session
set-up procedure to indicate the MPEG-4 Profile and Level
combination of which the relevant MPEG-4 media codec is capable.
For MPEG-4 Audio streams, this parameter is the decimal value from
Table 5 (audioProfileLevelIndication Values) in ISO/IEC 14496-
1, indicating which MPEG-4 Audio tool subsets are required to
decode the audio stream.
For MPEG-4 Visual streams, this parameter is the decimal value
from Table G-1 (FLC table for profile and level indication) of
ISO/IEC 14496-2 [1], indicating which MPEG-4 Visual tool
subsets are required to decode the visual stream.
For BIFS streams, this parameter is the decimal value obtained
from (SPLI + 256*GPLI), where:
SPLI is the decimal value from Table 4 in ISO/IEC 14496-1 with
the applied sceneProfileLevelIndication;
GPLI is the decimal value from Table 7 in ISO/IEC 14496-1 with
the applied graphicsProfileLevelIndication.
For MPEG-J streams, this parameter is the decimal value from table
13 (MPEGJProfileLevelIndication) in ISO/IEC 14496-1, indicating
the profile and level of the MPEG-J stream.
For OD streams, this parameter is the decimal value from table 3
(ODProfileLevelIndication) in ISO/IEC 14496-1, indicating the
profile and level of the OD stream.
For IPMP streams, this parameter has either the decimal value 0,
indicating an unspecified profile and level, or a value larger
than zero, indicating an MPEG-4 IPMP profile and level as
defined in a future MPEG-4 specification.
For Clock Reference streams and Object Content Info streams, this
parameter has the decimal value zero, indicating that profile
and level information is conveyed through the OD framework.
config:
A hexadecimal representation of an octet string that expresses the
media payload configuration. Configuration data is mapped onto
the hexadecimal octet string in an MSB-first basis. The first bit
of the configuration data SHALL be located at the MSB of the first
octet. In the last octet, if necessary to achieve octet-
alignment, up to 7 zero-valued padding bits shall follow the
configuration data.
For MPEG-4 Audio streams, config is the audio object type specific
decoder configuration data AudioSpecificConfig(), as defined in
ISO/IEC 14496-3. For Structured Audio, the
AudioSpecificConfig() may be conveyed by other means, not
defined by this specification. If the AudioSpecificConfig() is
conveyed by other means for Structured Audio, then the config
MUST be a quoted empty hexadecimal octet string, as follows:
config="".
Note that a future mode of using this RTP payload format for
Structured Audio may define such other means.
For MPEG-4 Visual streams, config is the MPEG-4 Visual
configuration information as defined in subclause 6.2.1, Start
codes of ISO/IEC 14496-2. The configuration information
indicated by this parameter SHALL be the same as the
configuration information in the corresponding MPEG-4 Visual
stream, except for first-half-vbv-occupancy and latter-half-
vbv-occupancy, if it exists, which may vary in the repeated
configuration information inside an MPEG-4 Visual stream (See