Request for Comments: 4425 Microsoft
Category: Standards Track February 2006
RTP Payload Format for Video Codec 1 (VC-1)
Status of This Memo
This document specifies an Internet standards track protocol for the
Internet community, and requests discussion and suggestions for
improvements. Please refer to the current edition of the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2006).
Abstract
This memo specifies an RTP payload format for encapsulating Video
Codec 1 (VC-1) compressed bit streams, as defined by the Society of
Motion Picture and Television Engineers (SMPTE) standard, SMPTE 421M.
SMPTE is the main standardizing body in the motion imaging industry,
and the SMPTE 421M standard defines a compressed video bit stream
format and decoding process for television.
Table of Contents
1. Introduction ....................................................2
1.1. Conventions Used in This Document ..........................3
2. Definitions and Abbreviations ...................................3
3. Overview of VC-1 ................................................5
3.1. VC-1 Bit Stream Layering Model .............................6
3.2. Bit-stream Data Units in Advanced Profile ..................7
3.3. Decoder Initialization Parameters ..........................7
3.4. Ordering of Frames .........................................8
4. Encapsulation of VC-1 Format Bit Streams in RTP .................9
4.1. Access Units ...............................................9
4.2. Fragmentation of VC-1 frames ..............................10
4.3. Time Stamp Considerations .................................11
4.4. Random Access Points ......................................13
4.5. Removal of HRD Parameters .................................14
4.6. Repeating the Sequence Layer Header .......................14
4.7. Signaling of Media Type Parameters ........................15
4.8. The "mode=1" Media Type Parameter .........................16
4.9. The "mode=3" Media Type Parameter .........................16
5. RTP Payload Format Syntax ......................................17
5.1. RTP Header Usage ..........................................17
5.2. AU Header Syntax ..........................................18
5.3. AU Control Field Syntax ...................................19
6. RTP Payload Format Parameters ..................................20
6.1. Media type Registration ...................................20
6.2. Mapping of media type parameters to SDP ...................28
6.3. Usage with the SDP Offer/Answer Model .....................29
6.4. Usage in Declarative Session Descriptions .................31
7. Security Considerations ........................................32
8. Congestion Control .............................................33
9. IANA Considerations ............................................34
10. References ....................................................34
10.1. Normative References .....................................34
10.2. Informative References ...................................35
1. Introduction
This memo specifies an RTP payload format for the video coding
standard Video Codec 1, also known as VC-1. The specification for
the VC-1 bit stream format and decoding process is published by the
Society of Motion Picture and Television Engineers (SMPTE) as SMPTE
421M [1].
VC-1 has a broad applicability, as it is suitable for low bit rate
Internet streaming applications to High Definition Television (HDTV)
broadcast and Digital Cinema applications with nearly lossless
coding. The overall performance of VC-1 is such that bit rate
savings of more than 50% are reported [9] when compared with MPEG-2.
See [9] for further details about how VC-1 compares with other
codecs, such as MPEG-4 and H.264/AVC. (In [9], VC-1 is referred to
by its earlier name, VC-9.)
VC-1 is widely used for downloading and streaming movies on the
Internet, in the form of Windows Media Video 9 (WMV-9) [9], because
the WMV-9 codec is compliant with the VC-1 standard. VC-1 has also
recently been adopted as a mandatory compression format for the
high-definition DVD formats HD DVD and Blu-ray.
SMPTE 421M defines the VC-1 bit stream syntax and specifies
constraints that must be met by VC-1 conformant bit streams. SMPTE
421M also specifies the complete process required to decode the bit
stream. However, it does not specify the VC-1 compression algorithm,
thus allowing for different ways of implementing a VC-1 encoder.
The VC-1 bit stream syntax has three profiles. Each profile has
specific bit stream syntax elements and algorithms associated with
it. Depending on the application in which VC-1 is used, some
profiles may be more suitable than others. For example, Simple
profile is designed for low bit rate Internet streaming and for
playback on devices that can only handle low-complexity decoding.
Advanced profile is designed for broadcast applications, such as
digital TV, HD DVD, or HDTV. Advanced profile is the only VC-1
profile that supports interlaced video frames and non-square pixels.
Section 2 defines the abbreviations used in this document. Section 3
provides a more detailed overview of VC-1. Sections 4 and 5 define
the RTP payload format for VC-1, and section 6 defines the media type
and SDP parameters for VC-1. See section 7 for security
considerations, and section 8 for congestion control requirements.
1.1. Conventions Used in This Document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119 [2].
2. Definitions and Abbreviations
This document uses the definitions in SMPTE 421M [1]. For
convenience, the following terms from SMPTE 421M are restated here:
B-picture:
A picture that is coded using motion compensated prediction
from past and/or future reference fields or frames. A
B-picture cannot be used for predicting any other picture.
BI-picture:
A B-picture that is coded using information only from itself.
A BI-picture cannot be used for predicting any other picture.
Bit-stream data unit (BDU):
A unit of the compressed data which may be parsed (i.e., syntax
decoded) independently of other information at the same
hierarchical level. A BDU can be, for example, a sequence
layer header, an entry-point header, a frame, or a slice.
Encapsulated BDU (EBDU):
A BDU that has been encapsulated using the encapsulation
mechanism described in Annex E of SMPTE 421M [1], to prevent
emulation of the start code prefix in the bit stream.
Entry-point:
A point in the bit stream that offers random access.
frame:
A frame contains lines of spatial information of a video
signal. For progressive video, these lines contain samples
starting from one time instant and continuing through
successive lines to the bottom of the frame. For interlaced
video, a frame consists of two fields, a top field and a bottom
field. One of these fields will commence one field period
later than the other.
interlace:
The property of frames where alternating lines of the frame
represent different instances in time. In an interlaced frame,
one of the fields is meant to be displayed first.
I-picture:
A picture coded using information only from itself.
level:
A defined set of constraints on the values that may be taken by
the parameters (such as bit rate and buffer size) within a
particular profile. A profile may contain one or more levels.
P-picture:
A picture that is coded using motion compensated prediction
from past reference fields or frames.
picture:
For progressive video, a picture is identical to a frame, while
for interlaced video, a picture may refer to a frame, or the
top field or the bottom field of the frame depending on the
context.
profile:
A defined subset of the syntax of VC-1 with a specific set of
coding tools, algorithms, and syntax associated with it. There
are three VC-1 profiles: Simple, Main, and Advanced.
progressive:
The property of frames where all the samples of the frame
represent the same instance in time.
random access:
A random access point in the bit stream is defined by the
following guarantee: If decoding begins at this point, all
frames needed for display after this point will have no
decoding dependency on any data preceding this point, and they
are also present in the decoding sequence after this point. A
random access point is also called an entry-point.
sequence:
A coded representation of a series of one or more pictures. In
VC-1 Advanced profile, a sequence consists of a series of one
or more entry-point segments, where each entry-point segment
consists of a series of one or more pictures, and where the
first picture in each entry-point segment provides random
access. In VC-1 Simple and Main profiles, the first picture in
each sequence is an I-picture.
slice:
A consecutive series of macroblock rows in a picture, which are
encoded as a single unit.
start codes (SC):
Unique 32-bit codes that are embedded in the coded bit stream
and identify the beginning of a BDU. Start codes consist of a
unique three-byte Start Code Prefix (SCP), and a one-byte Start
Code Suffix (SCS).
3. Overview of VC-1
The VC-1 bit stream syntax consists of three profiles: Simple, Main,
and Advanced. Simple profile is designed for low bit rates and for
low complexity applications, such as playback of media on personal
digital assistants. The maximum bit rate supported by Simple profile
is 384 kbps. Main profile targets high bit rate applications, such
as streaming and TV over IP. Main profile supports B-pictures, which
provide improved compression efficiency at the cost of higher
complexity.
Certain features that can be used to achieve high compression
efficiency, such as non-square pixels and support for interlaced
pictures, are only included in Advanced profile. The maximum bit
rate supported by the Advanced profile is 135 Mbps, making it
suitable for nearly lossless encoding of HDTV signals.
Only Advanced profile supports carrying user-data (meta-data) in-band
with the compressed bit stream. The user-data can be used for closed
captioning support, for example.
Of the three profiles, only Advanced profile allows codec
configuration parameters, such as the picture aspect ratio, to be
changed through in-band signaling in the compressed bit stream.
For each of the profiles, a certain number of "levels" have been
defined. Unlike a "profile", which implies a certain set of features
or syntax elements, a "level" is a set of constraints on the values
of parameters in a profile, such as the bit rate or buffer size.
VC-1 Simple profile has two levels, Main profile has three, and
Advanced profile has five. See Annex D of SMPTE 421M [1] for a
detailed list of the profiles and levels.
3.1. VC-1 Bit Stream Layering Model
The VC-1 bit stream is defined as a hierarchy of layers. This is
conceptually similar to the notion of a protocol stack of networking
protocols. The outermost layer is called the sequence layer. The
other layers are entry-point, picture, slice, macroblock, and block.
In Simple and Main profiles, a sequence in the sequence layer
consists of a series of one or more coded pictures. In Advanced
profile, a sequence consists of one or more entry-point segments,
where each entry-point segment consists of a series of one or more
pictures, and where the first picture in each entry-point segment
provides random access. A picture is decomposed into macroblocks. A
slice comprises one or more contiguous rows of macroblocks.
The entry-point and slice layers are only present in Advanced
profile. In Advanced profile, the start of each entry-point layer
segment indicates a random access point. In Simple and Main
profiles, each I-picture is a random access point.
Each picture can be coded as an I-picture, P-picture, skipped
picture, BI-picture, or as a B-picture. These terms are defined in
section 2 of this document and in section 4.12 of SMPTE 421M [1].
3.2. Bit-stream Data Units in Advanced Profile
In Advanced profile, each picture and slice is considered a Bit-
stream Data Unit (BDU). A BDU is always byte-aligned and is defined
as a unit that can be parsed (i.e., syntax decoded) independently of
other information in the same layer.
The beginning of a BDU is signaled by an identifier called Start Code
(SC). Sequence layer headers and entry-point headers are also BDUs
and thus can be easily identified by their Start Codes. See Annex E
of SMPTE 421M [1] for a complete list of Start Codes. Blocks and
macroblocks are not BDUs and thus do not have a Start Code and are
not necessarily byte-aligned.
The Start Code consists of four bytes. The first three bytes are
0x00, 0x00 and 0x01. The fourth byte is called the Start Code Suffix
(SCS) and it is used to indicate the type of BDU that follows the
Start Code. For example, the SCS of a sequence layer header (0x0F)
is different from the SCS of an entry-point header (0x0E). The Start
Code is always byte-aligned and is transmitted in network byte order.
To prevent accidental emulation of the Start Code in the coded bit
stream, SMPTE 421M defines an encapsulation mechanism that uses byte
stuffing. A BDU that has been encapsulated by this mechanism is
referred to as an Encapsulated BDU, or EBDU.
3.3. Decoder Initialization Parameters
In VC-1 Advanced profile, the sequence layer header contains
parameters that are necessary to initialize the VC-1 decoder.
The parameters apply to all entry-point segments until the next
occurrence of a sequence layer header in the coded bit stream.
The parameters in the sequence layer header include the Advanced
profile level, the maximum dimensions of the coded frames, the aspect
ratio, interlace information, the frame rate and up to 31 leaky
bucket parameter sets for the Hypothetical Reference Decoder (HRD).
Section 6.1 of SMPTE 421M [1] provides the formal specification of
the sequence layer header.
A sequence layer header is not defined for VC-1 Simple and Main
profiles. For these profiles, decoder initialization parameters MUST
be conveyed out-of-band. The decoder initialization parameters for
Simple and Main profiles include the maximum dimensions of the coded
frames and a leaky bucket parameter set for the HRD. Section 4.7
specifies how the parameters are conveyed by this RTP payload format.
Each leaky bucket parameter set for the HRD specifies a peak
transmission bit rate and a decoder buffer capacity. The coded bit
stream is restricted by these parameters. The HRD model does not
mandate buffering by the decoder. Its purpose is to limit the
encoder’s bit rate fluctuations according to a basic buffering model
so that the resources necessary to decode the bit stream are
predictable. The HRD has a constant-delay mode and a variable-delay
mode. The constant-delay mode is appropriate for broadcast and
streaming applications, while the variable-delay mode is designed for
video-conferencing applications.
Annex C of SMPTE 421M [1] specifies the usage of the hypothetical
reference decoder for VC-1 bit streams. A general description of the
theory of the HRD can be found in [10].
For Simple and Main profiles, the current buffer fullness value for
the HRD leaky bucket is signaled using the BF syntax element in the
picture header of I-pictures and BI-pictures.
For Advanced profile, the entry-point header specifies current buffer
fullness values for the leaky buckets in the HRD. The entry-point
header also specifies coding control parameters that are in effect
until the occurrence of the next entry-point header in the bit
stream. The concept of an entry-point layer applies only to VC-1
Advanced profile. See Section 6.2 of SMPTE 421M [1] for the formal
specification of the entry-point header.
3.4. Ordering of Frames
Frames are transmitted in the same order in which they are captured,
except if B-pictures or BI-pictures are present in the coded bit
stream. A BI-picture is a special kind of B-picture, and in the
remainder of this section the terms B-picture and B-frame also apply
to BI-pictures and BI-frames, respectively.
When B-pictures are present in the coded bit stream, the frames are
transmitted such that the frames that the B-pictures depend on are
transmitted first. This is referred to as the coded order of the
frames.
The rules for how a decoder converts frames from the coded order to
the display order are stated in section 5.4 of SMPTE 421M [1]. In
short, if B-pictures may be present in the coded bit stream, a
hypothetical decoder implementation needs to buffer one additional
decoded frame. When an I-frame or a P-frame is received, the frame
can be decoded immediately but it is not displayed until the next I-
or P-frame is received. However, B-frames are displayed immediately.
Figure 1 illustrates the timing relationship between the capture of
frames, their coded order, and the display order of the decoded
frames, when B-pictures are present in the coded bit stream. The
figure shows that the display of frame P4 is delayed until frame P7
is received, while frames B2 and B3 are displayed immediately.
Capture: |I0 P1 B2 B3 P4 B5 B6 P7 B8 B9 ...
|
Coded order: | I0 P1 P4 B2 B3 P7 B5 B6 ...
|
Display order: | I0 P1 B2 B3 P4 B5 B6 ...
|
|+---+---+---+---+---+---+---+---+---+--> time
0 1 2 3 4 5 6 7 8 9
Figure 1. Frame reordering when B-pictures are present
If B-pictures are not present, the coded order and the display order
are identical, and frames can then be displayed without the
additional delay shown in Figure 1.
4. Encapsulation of VC-1 Format Bit Streams in RTP
4.1. Access Units
Each RTP packet contains an integral number of application data units
(ADUs). For VC-1 format bit streams, an ADU is equivalent to one
Access Unit (AU). An Access Unit is defined as the AU header
(defined in section 5.2) followed by a variable length payload, with
the rules and constraints described in sections 4.1 and 4.2. Figure
2 shows the layout of an RTP packet with multiple AUs.
+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. +-+-+-+-+
| RTP | AU(1) | AU(2) | | AU(n) |
| Header | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. +-+-+-+-+
Figure 2. RTP packet structure
Each Access Unit MUST start with the AU header defined in section
5.2. The AU payload MUST contain data belonging to exactly one VC-1
frame. This means that data from different VC-1 frames will always
be in different AUs. However, it possible for a single VC-1 frame to
be fragmented across multiple AUs (see section 4.2).
In the case of interlaced video, a VC-1 frame consists of two fields
that may be coded as separate pictures. The two pictures still
belong to the same VC-1 frame.
The following rules apply to the contents of each AU payload when
VC-1 Advanced profile is used:
- The AU payload MUST contain VC-1 bit stream data in EBDU format
(i.e., the bit stream must use the byte-stuffing encapsulation
mode defined in Annex E of SMPTE 421M [1].)
- The AU payload MAY contain multiple EBDUs, e.g., a sequence layer
header, an entry-point header, a frame (picture) header, a field
header, and multiple slices and the associated user-data.
However, all slices and their corresponding macroblocks MUST
belong to the same video frame.
- The AU payload MUST start at an EBDU boundary, except when the AU
payload contains a fragmented frame, in which case the rules in
section 4.2 apply.
When VC-1 Simple or Main profiles are used, the AU payload MUST start
at the beginning of a frame, except when the AU payload contains a
fragmented frame. Section 4.2 describes how to handle fragmented
frames.
Access Units MUST be byte-aligned. If the data in an AU (EBDUs in
the case of Advanced profile and frame in the case of Simple and
Main) does not end at an octet boundary, up to 7 zero-valued padding
bits MUST be added to achieve octet-alignment.
4.2. Fragmentation of VC-1 frames
Each AU payload SHOULD contain a complete VC-1 frame. However, if
this would cause the RTP packet to exceed the MTU size, the frame
SHOULD be fragmented into multiple AUs to avoid IP-level
fragmentation. When an AU contains a fragmented frame, this MUST be
indicated by setting the FRAG field in the AU header as defined in
section 5.3.
AU payloads that do not contain a fragmented frame or that contain
the first fragment of a frame MUST start at an EBDU boundary if
Advanced profile is used. In this case, for Simple and Main
profiles, the AU payload MUST start at the beginning of a frame.
If Advanced profile is used, AU payloads that contain a fragment of a
frame other than the first fragment SHOULD start at an EBDU boundary,
such as at the start of a slice.
However, slices are only defined for Advanced profile, and are not
always used. Blocks and macroblocks are not BDUs (have no Start
Code) and are not byte-aligned. Therefore, it may not always be
possible to continue a fragmented frame at an EBDU boundary. One can
determine if an AU payload starts at an EBDU boundary by inspecting
the first three bytes of the AU payload. The AU payload starts at an
EBDU boundary if the first three bytes are identical to the Start
Code Prefix (i.e., 0x00, 0x00, 0x01).
In the case of Simple and Main profiles, since the blocks and
macroblocks are not byte-aligned, the fragmentation boundary may be
chosen arbitrarily.
If an RTP packet contains an AU with the last fragment of a frame,
additional AUs SHOULD NOT be included in the RTP packet.
If the PTS Delta field in the AU header is present, each fragment of
a frame MUST have the same presentation time. If the DTS Delta field
in the AU header is present, each fragment of a frame MUST have the
same decode time.
4.3. Time Stamp Considerations
VC-1 video frames MUST be transmitted in the coded order. A coded