- max-mbps
- max-fs
- max-cpb
- max-dpb
- max-br
- redundant-pic-cap
- deint-buf-cap
- max-rcmd-nalu-size
Declaring how Offer/Answer negotiation shall be performed:
- parameter-add
Furthermore, the following considerations are necessary:
o Parameters used for declaring receiver capabilities are in general
downgradable; i.e., they express the upper limit for a sender’s
possible behavior. Thus a sender MAY select to set its encoder
using only lower/lesser or equal values of these parameters.
"sprop-parameter-sets" MUST NOT be used in a sender’s declaration
of its capabilities, as the limits of the values that are carried
inside the parameter sets are implicit with the profile and level
used.
o Parameters declaring a configuration point are not downgradable,
with the exception of the level part of the "profile-level-id"
parameter. This expresses values a receiver expects to be used
and must be used verbatim on the sender side.
o When a sender’s capabilities are declared, and non-downgradable
parameters are used in this declaration, then these parameters
express a configuration that is acceptable. In order to achieve
high interoperability levels, it is often advisable to offer
multiple alternative configurations; e.g., for the packetization
mode. It is impossible to offer multiple configurations in a
single payload type. Thus, when multiple configuration offers are
made, each offer requires its own RTP payload type associated with
the offer.
o A receiver SHOULD understand all MIME parameters, even if it only
supports a subset of the payload format’s functionality. This
ensures that a receiver is capable of understanding when an offer
to receive media can be downgraded to what is supported by the
receiver of the offer.
o An answerer MAY extend the offer with additional media format
configurations. However, to enable their usage, in most cases a
second offer is required from the offerer to provide the stream
properties parameters that the media sender will use. This also
has the effect that the offerer has to be able to receive this
media format configuration, not only to send it.
o If an offerer wishes to have non-symmetric capabilities between
sending and receiving, the offerer has to offer different RTP
sessions; i.e., different media lines declared as "recvonly" and
"sendonly", respectively. This may have further implications on
the system.
8.2.3. Usage in Declarative Session Descriptions
When H.264 over RTP is offered with SDP in a declarative style, as in
RTSP [27] or SAP [28], the following considerations are necessary.
o All parameters capable of indicating the properties of both a NAL
unit stream and a receiver are used to indicate the properties of
a NAL unit stream. For example, in this case, the parameter
"profile-level-id" declares the values used by the stream, instead
of the capabilities of the sender. This results in that the
following interpretation of the parameters MUST be used:
Declaring actual configuration or properties:
- profile-level-id
- sprop-parameter-sets
- packetization-mode
- sprop-interleaving-depth
- sprop-deint-buf-req
- sprop-max-don-diff
- sprop-init-buf-time
Not usable:
- max-mbps
- max-fs
- max-cpb
- max-dpb
- max-br
- redundant-pic-cap
- max-rcmd-nalu-size
- parameter-add
- deint-buf-cap
o A receiver of the SDP is required to support all parameters and
values of the parameters provided; otherwise, the receiver MUST
reject (RTSP) or not participate in (SAP) the session. It falls
on the creator of the session to use values that are expected to
be supported by the receiving application.
8.3. Examples
A SIP Offer/Answer exchange wherein both parties are expected to both
send and receive could look like the following. Only the media codec
specific parts of the SDP are shown. Some lines are wrapped due to
text constraints.
Offerer -> Answer SDP message:
m=video 49170 RTP/AVP 100 99 98
a=rtpmap:98 H264/90000
a=fmtp:98 profile-level-id=42A01E; packetization-mode=0;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==
a=rtpmap:99 H264/90000
a=fmtp:99 profile-level-id=42A01E; packetization-mode=1;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==
a=rtpmap:100 H264/90000
a=fmtp:100 profile-level-id=42A01E; packetization-mode=2;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==;
sprop-interleaving-depth=45; sprop-deint-buf-req=64000;
sprop-init-buf-time=102478; deint-buf-cap=128000
The above offer presents the same codec configuration in three
different packetization formats. PT 98 represents single NALU mode,
PT 99 non-interleaved mode; PT 100 indicates the interleaved mode.
In the interleaved mode case, the interleaving parameters that the
offerer would use if the answer indicates support for PT 100 are also
included. In all three cases the parameter "sprop-parameter-sets"
conveys the initial parameter sets that are required for the answerer
when receiving a stream from the offerer when this configuration
(profile-level-id and packetization mode) is accepted. Note that the
value for "sprop-parameter-sets", although identical in the example
above, could be different for each payload type.
Answerer -> Offerer SDP message:
m=video 49170 RTP/AVP 100 99 97
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42A01E; packetization-mode=0;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==,As0DEWlsIOp==,
KyzFGleR
a=rtpmap:99 H264/90000
a=fmtp:99 profile-level-id=42A01E; packetization-mode=1;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==,As0DEWlsIOp==,
KyzFGleR; max-rcmd-nalu-size=3980
a=rtpmap:100 H264/90000
a=fmtp:100 profile-level-id=42A01E; packetization-mode=2;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==,As0DEWlsIOp==,
KyzFGleR; sprop-interleaving-depth=60;
sprop-deint-buf-req=86000; sprop-init-buf-time=156320;
deint-buf-cap=128000; max-rcmd-nalu-size=3980
As the Offer/Answer negotiation covers both sending and receiving
streams, an offer indicates the exact parameters for what the offerer
is willing to receive, whereas the answer indicates the same for what
the answerer accepts to receive. In this case the offerer declared
that it is willing to receive payload type 98. The answerer accepts
this by declaring a equivalent payload type 97; i.e., it has
identical values for the three parameters "profile-level-id",
packetization-mode, and "sprop-deint-buf-req". This has the
following implications for both the offerer and the answerer
concerning the parameters that declare properties. The offerer
initially declared a certain value of the "sprop-parameter-sets" in
the payload definition for PT=98. However, as the answerer accepted
this as PT=97, the values of "sprop-parameter-sets" in PT=98 must now
be used instead when the offerer sends PT=97. Similarly, when the
answerer sends PT=98 to the offerer, it has to use the properties
parameters it declared in PT=97.
The answerer also accepts the reception of the two configurations
that payload types 99 and 100 represent. It provides the initial
parameter sets for the answerer-to-offerer direction, and for
buffering related parameters that it will use to send the payload
types. It also provides the offerer with its memory limit for
deinterleaving operations by providing a "deint-buf-cap" parameter.
This is only useful if the offerer decides on making a second offer,
where it can take the new value into account. The "max-rcmd-nalu-
size" indicates that the answerer can efficiently process NALUs up to
the size of 3980 bytes. However, there is no guarantee that the
network supports this size.
Please note that the parameter sets in the above example do not
represent a legal operation point of an H.264 codec. The base64
strings are only used for illustration.
8.4. Parameter Set Considerations
The H.264 parameter sets are a fundamental part of the video codec
and vital to its operation; see section 1.2. Due to their
characteristics and their importance for the decoding process, lost
or erroneously transmitted parameter sets can hardly be concealed
locally at the receiver. A reference to a corrupt parameter set has
normally fatal results to the decoding process. Corruption could
occur, for example, due to the erroneous transmission or loss of a
parameter set data structure, but also due to the untimely
transmission of a parameter set update. Therefore, the following
recommendations are provided as a guideline for the implementer of
the RTP sender.
Parameter set NALUs can be transported using three different
principles:
A. Using a session control protocol (out-of-band) prior to the actual
RTP session.
B. Using a session control protocol (out-of-band) during an ongoing
RTP session.
C. Within the RTP stream in the payload (in-band) during an ongoing
RTP session.
It is necessary to implement principles A and B within a session
control protocol. SIP and SDP can be used as described in the SDP
Offer/Answer model and in the previous sections of this memo. This
section contains guidelines on how principles A and B must be
implemented within session control protocols. It is independent of
the particular protocol used. Principle C is supported by the RTP
payload format defined in this specification.
The picture and sequence parameter set NALUs SHOULD NOT be
transmitted in the RTP payload unless reliable transport is provided
for RTP, as a loss of a parameter set of either type will likely
prevent decoding of a considerable portion of the corresponding RTP
stream. Thus, the transmission of parameter sets using a reliable
session control protocol (i.e., usage of principle A or B above) is
RECOMMENDED.
In the rest of the section it is assumed that out-of-band signaling
provides reliable transport of parameter set NALUs and that in-band
transport does not. If in-band signaling of parameter sets is used,
the sender SHOULD take the error characteristics into account and use
mechanisms to provide a high probability for delivering the parameter
sets correctly. Mechanisms that increase the probability for a
correct reception include packet repetition, FEC, and retransmission.
The use of an unreliable, out-of-band control protocol has similar
disadvantages as the in-band signaling (possible loss) and, in
addition, may also lead to difficulties in the synchronization (see
below). Therefore, it is NOT RECOMMENDED.
Parameter sets MAY be added or updated during the lifetime of a
session using principles B and C. It is required that parameter sets
are present at the decoder prior to the NAL units that refer to them.
Updating or adding of parameter sets can result in further problems,
and therefore the following recommendations should be considered.
- When parameter sets are added or updated, principle C is
vulnerable to transmission errors as described above, and
therefore principle B is RECOMMENDED.
- When parameter sets are added or updated, care SHOULD be taken to
ensure that any parameter set is delivered prior to its usage. It
is common that no synchronization is present between out-of-band
signaling and in-band traffic. If out-of-band signaling is used,
it is RECOMMENDED that a sender does not start sending NALUs
requiring the updated parameter sets prior to acknowledgement of
delivery from the signaling protocol.
- When parameter sets are updated, the following synchronization
issue should be taken into account. When overwriting a parameter
set at the receiver, the sender has to ensure that the parameter
set in question is not needed by any NALU present in the network
or receiver buffers. Otherwise, decoding with a wrong parameter
set may occur. To lessen this problem, it is RECOMMENDED either
to overwrite only those parameter sets that have not been used for
a sufficiently long time (to ensure that all related NALUs have
been consumed), or to add a new parameter set instead (which may
have negative consequences for the efficiency of the video
coding).
- When new parameter sets are added, previously unused parameter set
identifiers are used. This avoids the problem identified in the
previous paragraph. However, in a multiparty session, unless a
synchronized control protocol is used, there is a risk that
multiple entities try to add different parameter sets for the same
identifier, which has to be avoided.
- Adding or modifying parameter sets by using both principles B and
C in the same RTP session may lead to inconsistencies of the
parameter sets because of the lack of synchronization between the
control and the RTP channel. Therefore, principles B and C MUST
NOT both be used in the same session unless sufficient
synchronization can be provided.
In some scenarios (e.g., when only the subset of this payload format
specification corresponding to H.241 is used), it is not possible to
employ out-of-band parameter set transmission. In this case,
parameter sets have to be transmitted in-band. Here, the
synchronization with the non-parameter-set-data in the bitstream is
implicit, but the possibility of a loss has to be taken into account.
The loss probability should be reduced using the mechanisms discussed
above.
- When parameter sets are initially provided using principle A and
then later added or updated in-band (principle C), there is a risk
associated with updating the parameter sets delivered out-of-band.
If receivers miss some in-band updates (for example, because of a
loss or a late tune-in), those receivers attempt to decode the
bitstream using out-dated parameters. It is RECOMMENDED that
parameter set IDs be partitioned between the out-of-band and in-
band parameter sets.
To allow for maximum flexibility and best performance from the H.264
coder, it is recommended, if possible, to allow any sender to add its
own parameter sets to be used in a session. Setting the "parameter-
add" parameter to false should only be done in cases where the
session topology prevents a participant to add its own parameter
sets.
9. Security Considerations
RTP packets using the payload format defined in this specification
are subject to the security considerations discussed in the RTP
specification [4], and in any appropriate RTP profile (for example,
[16]). This implies that confidentiality of the media streams is
achieved by encryption; for example, through the application of SRTP
[26]. Because the data compression used with this payload format is
applied end-to-end, any encryption needs to be performed after
compression.
A potential denial-of-service threat exists for data encodings using
compression techniques that have non-uniform receiver-end
computational load. The attacker can inject pathological datagrams
into the stream that are complex to decode and that cause the
receiver to be overloaded. H.264 is particularly vulnerable to such
attacks, as it is extremely simple to generate datagrams containing
NAL units that affect the decoding process of many future NAL units.
Therefore, the usage of data origin authentication and data integrity
protection of at least the RTP packet is RECOMMENDED; for example,
with SRTP [26].
Note that the appropriate mechanism to ensure confidentiality and
integrity of RTP packets and their payloads is very dependent on the
application and on the transport and signaling protocols employed.
Thus, although SRTP is given as an example above, other possible
choices exist.
Decoders MUST exercise caution with respect to the handling of user
data SEI messages, particularly if they contain active elements, and
MUST restrict their domain of applicability to the presentation
containing the stream.
End-to-End security with either authentication, integrity or
confidentiality protection will prevent a MANE from performing
media-aware operations other than discarding complete packets. And
in the case of confidentiality protection it will even be prevented
from performing discarding of packets in a media aware way. To allow
any MANE to perform its operations, it will be required to be a
trusted entity which is included in the security context
establishment.
10. Congestion Control
Congestion control for RTP SHALL be used in accordance with RFC 3550
[4], and with any applicable RTP profile; e.g., RFC 3551 [16]. An
additional requirement if best-effort service is being used is:
users of this payload format MUST monitor packet loss to ensure that
the packet loss rate is within acceptable parameters. Packet loss is
considered acceptable if a TCP flow across the same network path, and
experiencing the same network conditions, would achieve an average
throughput, measured on a reasonable timescale, that is not less than
the RTP flow is achieving. This condition can be satisfied by
implementing congestion control mechanisms to adapt the transmission
rate (or the number of layers subscribed for a layered multicast
session), or by arranging for a receiver to leave the session if the
loss rate is unacceptably high.
The bit rate adaptation necessary for obeying the congestion control
principle is easily achievable when real-time encoding is used.
However, when pre-encoded content is being transmitted, bandwidth
adaptation requires the availability of more than one coded
representation of the same content, at different bit rates, or the
existence of non-reference pictures or sub-sequences [22] in the
bitstream. The switching between the different representations can
normally be performed in the same RTP session; e.g., by employing a
concept known as SI/SP slices of the Extended Profile, or by
switching streams at IDR picture boundaries. Only when non-
downgradable parameters (such as the profile part of the
profile/level ID) are required to be changed does it become necessary
to terminate and re-start the media stream. This may be accomplished
by using a different RTP payload type.
MANEs MAY follow the suggestions outlined in section 7.3 and remove
certain unusable packets from the packet stream when that stream was
damaged due to previous packet losses. This can help reduce the
network load in certain special cases.
11. IANA Consideration
IANA has registered one new MIME type; see section 8.1.
12. Informative Appendix: Application Examples
This payload specification is very flexible in its use, in order to
cover the extremely wide application space anticipated for H.264.
However, this great flexibility also makes it difficult for an
implementer to decide on a reasonable packetization scheme. Some
information on how to apply this specification to real-world
scenarios is likely to appear in the form of academic publications
and a test model software and description in the near future.
However, some preliminary usage scenarios are described here as well.
12.1. Video Telephony according to ITU-T Recommendation H.241
Annex A
H.323-based video telephony systems that use H.264 as an optional
video compression scheme are required to support H.241 Annex A [15]
as a packetization scheme. The packetization mechanism defined in
this Annex is technically identical with a small subset of this
specification.
When a system operates according to H.241 Annex A, parameter set NAL
units are sent in-band. Only Single NAL unit packets are used. Many
such systems are not sending IDR pictures regularly, but only when
required by user interaction or by control protocol means; e.g., when
switching between video channels in a Multipoint Control Unit or for
error recovery requested by feedback.
12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit
Aggregation
The RTP part of this scheme is implemented and tested (though not the
control-protocol part; see below).
In most real-world video telephony applications, picture parameters
such as picture size or optional modes never change during the
lifetime of a connection. Therefore, all necessary parameter sets
(usually only one) are sent as a side effect of the capability
exchange/announcement process, e.g., according to the SDP syntax
specified in section 8.2 of this document. As all necessary
parameter set information is established before the RTP session
starts, there is no need for sending any parameter set NAL units.
Slice data partitioning is not used, either. Thus, the RTP packet
stream basically consists of NAL units that carry single coded
slices.
The encoder chooses the size of coded slice NAL units so that they
offer the best performance. Often, this is done by adapting the
coded slice size to the MTU size of the IP network. For small
picture sizes, this may result in a one-picture-per-one-packet
strategy. Intra refresh algorithms clean up the loss of packets and
the resulting drift-related artifacts.
12.3. Video Telephony, Interleaved Packetization Using NAL Unit
Aggregation
This scheme allows better error concealment and is used in H.263
based designs using RFC 2429 packetization [10]. It has been
implemented, and good results were reported [12].
The VCL encoder codes the source picture so that all macroblocks
(MBs) of one MB line are assigned to one slice. All slices with even
MB row addresses are combined into one STAP, and all slices with odd
MB row addresses into another. Those STAPs are transmitted as RTP
packets. The establishment of the parameter sets is performed as
discussed above.
Note that the use of STAPs is essential here, as the high number of
individual slices (18 for a CIF picture) would lead to unacceptably
high IP/UDP/RTP header overhead (unless the source coding tool FMO is
used, which is not assumed in this scenario). Furthermore, some
wireless video transmission systems, such as H.324M and the IP-based
video telephony specified in 3GPP, are likely to use relatively small
transport packet size. For example, a typical MTU size of H.223 AL3
SDU is around 100 bytes [17]. Coding individual slices according to
this packetization scheme provides further advantage in communication
between wired and wireless networks, as individual slices are likely
to be smaller than the preferred maximum packet size of wireless
systems. Consequently, a gateway can convert the STAPs used in a
wired network into several RTP packets with only one NAL unit, which
are preferred in a wireless network, and vice versa.
12.4. Video Telephony with Data Partitioning
This scheme has been implemented and has been shown to offer good
performance, especially at higher packet loss rates [12].
Data Partitioning is known to be useful only when some form of
unequal error protection is available. Normally, in single-session
RTP environments, even error characteristics are assumed; i.e., the
packet loss probability of all packets of the session is the same
statistically. However, there are means to reduce the packet loss
probability of individual packets in an RTP session. A FEC packet
according to RFC 2733 [18], for example, specifies which media
packets are associated with the FEC packet.
In all cases, the incurred overhead is substantial but is in the same