presented. If no text sample following is available, it is an
implementation issue what should be displayed. For example, a
server could send an empty sample to clear the text box.
Example: Imagine you are in an airport watching the latest news
report while you wait for your plane. Airports are loud, so the
news report is transcribed in the lower area of the screen.
This area displays two lines of text: the headlines and the
words spoken by the news speaker. As usual, the headlines are
shown for a longer time than the rest. This time is, in
principle, unknown to the stream server, which is streaming
live. A headline is just replaced when the next headline is
received.
However, upon storing a text sample with SDUR=0 in a 3GP file, the
SDUR value MUST be changed to the effective duration of the text
sample, which MUST be always greater than zero (note that the ISO
file format [2] explicitly forbids a sample duration of zero). The
effective duration MUST be calculated as the timestamp difference
between the current sample (with unknown duration) and the next
text sample that is displayed.
Note that samples of unknown duration SHALL NOT use features, which
require knowledge of the duration of the sample up front. Such
features are scrolling and karaoke in [1]. This also applies for
future extensions of the Timed Text format. Furthermore, only
sample descriptions (TYPE 5 units) MAY follow units of unknown
duration in the same aggregate payload. Otherwise, it would not be
possible to calculate the timestamp of these other units.
For text contents stored in 3GP files, see Section 4.3 for details
on how to extract the duration value. For live streaming, live
encoders SHALL assign appropriate values and units according to [1]
and later releases.
o TLEN (16 bits), "Text String Length", is a byte count of the text
string. The decoder needs the text string length in order to know
where the modifiers in the payload start. TLEN is not present in
text string fragments (TYPE 2) since it can be deductively
calculated from the LEN values of each fragment.
The TLEN value is obtained from the text samples as contained in
3GP files. Refer to Section 4.3. For live content, the TLEN MUST
be obtained during the sampling process.
o Finally, the actual text sample is placed after the TLEN field. As
defined in Section 3, a text sample consists of a string of
characters encoded using either UTF-8 or UTF-16, followed by zero
or more modifiers. Note also that no BOM and no byte count are
included in the strings carried in the payload (as opposed to text
samples stored in 3GP files [1]).
4.1.3. TYPE 2 Header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE | LEN( always >9) | TOTAL | THIS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SLEN |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5. TYPE 2 Header Format
This header type is used to transport either a whole text string or a
fragment of it. TYPE 2 units SHALL NOT contain modifiers. In
detail:
o U, R, and TYPE, as defined in Section 4.1.1.
o SIDX and SDUR, as defined in Section 4.1.2.
Note that the U, SIDX, and SDUR fields are meaningful since
partial text strings can also be displayed.
o The LEN field (16 bits) indicates the length of the text string
fragment plus nine (9) bytes of headers. Its value is calculated
upon fragmentation. LEN MUST always be greater than nine (0x0009).
Otherwise, the unit MUST be discarded.
According to the guidelines in Section 4.4, text strings MUST be
split at character boundaries for allowing the display of text
fragments. Therefore, a text fragment MUST contain at least one
character in either UTF-8 or UTF-16. Actually, this is just a
formalism since by observing the guidelines, much larger fragments
should be created.
Note also that TYPE 2 units do not contain an explicit text string
length, TLEN (see TYPE 1). This is because TYPE 2 units do not
contain any modifiers after the text string. If needed, the length
of the received string can be obtained using the LEN values of the
TYPE 2 units.
o The SLEN field (16 bits) indicates the size (in bytes) of the
original (whole) text sample to which this fragment belongs. This
length comprises the text string plus any modifier boxes present
(and includes neither the byte order mark nor the text string
length as mentioned in Section 3, "Terminology").
Regarding the text sample length: Timed text samples are not
generated at regular intervals, nor is there a default sample size.
If 3GP files are streamed, the length of the text samples is
calculated beforehand and included in the track itself, while for
live encoding it is the real time encoder that SHALL choose an
appropriate size for each text sample. In this case, the amount of
text ’captured’ in a sample depends on the text source and the
particular application (see examples below). Samples may, e.g., be
tailored to match the packet MTU as closely as possible or to
provide a given redundancy for the available bit rate. The
encoding application MUST also take into account the delay
constraints of the real-time session and assess whether FEC,
retransmission, or other similar techniques are reasonable options
for stream repair.
The following examples shall illustrate how a real-time encoder may
choose its settings to adapt to the scenario constraints.
Example: Imagine a newscast scenario, where the spoken news is
transcribed and synchronized with the image and voice of the
reporter. We assume that the news speaker talks at an average
speed of 5 words per second with an average word length of 5
characters plus one space per word, i.e., 30 characters per
second. We assume an available IP MTU of 576 bytes and an
available bitrate of 576*8 bits per second = 4.6 Kbps. We
assume each character can be encoded using 2 bytes in UTF-16.
In this scenario, several constraints may apply; for example:
available IP MTU, available bandwidth, allowable delay, and
required redundancy. If the target were to minimize the
packet overhead, a text sample covering 8 seconds of text
would be closest to the IP MTU:
IP/UDP/RTP/TYPE1 Header + (8-second text sample)
= 20 + 8 + 12 + 8 + (~6 chars/word * 5 word/s * 8 s * 2 chars/word)
= 528 bytes < 576 bytes
For other scenarios, like lossy networks, it may happen that just
one packet per sample is too low a redundancy. In this case, a
choice could be that the encoder ’collects’ text every second, thus
yielding text samples (TYPE 1 units) of 68 bytes, TYPE 1 header
included. We can, e.g., include three contiguous text samples in
one RTP payload: the current and last two text samples (see below).
This accounts to a total IP packet size of 20 + 8 + 12 + 3*(8 + 60)
= 244 bytes. Now, with the same available bitrate of 4.6 Kbps,
these 244-byte packets can be sent redundantly up two times per
second:
RTP payload (1,2,3)(1,2,3) (2,3,4)(2,3,4) (3,4,5)(3,4,5) ...
Time: <----1s------> <----1s------> <-----1s-----> ...
This means that each text sample is sent at least six times,
which should provide enough redundancy. Although not as
bandwidth efficient (488*8 < 528*8 < 576*8 bps) as the
previous packetization, this option increases the stream
redundancy while still meeting the delay and bandwidth
constraints.
Another example would be a user sending timed text from a
type-in area in the display. In this case, the text sample is
created as soon as the user clicks the ’send’ button.
Depending on the packet length, fragmentation may be needed.
In a video conferencing application, text is synchronized with
audio and video. Thus, the text samples shall be displayed
long enough to be read by a human, shall fit in the video
screen, and shall ’capture’ the audio contents rendered during
the time the corresponding video and audio is rendered.
For stored content, see Section 4.3 for details on how to find the
SLEN value in a 3GP file. For live content, the SLEN MUST be
obtained during the sampling process.
Finally, note that clients MAY use SLEN to buffer space for the
remaining fragments of a text sample.
o The fields TOTAL (4 bits) and THIS (4 bits) indicate the total
number of fragments in which the original text sample (i.e., the
text string and its modifiers) has been fragmented and which order
occupies the current fragment in that sequence, respectively. Note
that the sequence number alone cannot replace the functionality of
the THIS field, since packets (and fragments) may be repeated,
e.g., as in repeated transmission (see Section 5). Thus, an
indication for "fragment offset" is needed.
The usual "byte offset" field is not used here for two reasons: a)
it would take one more byte and b) it does not provide any
information on the character offset. UTF-8/UTF-16 text strings
have, in general, a variable character length ranging from 1 to 6
bytes. Therefore, the TOTAL/THIS solution is preferred. It could
also be argued that the LEN and SLEN fields be used for this
purpose, but while they would provide information about the
completeness of the text sample, they do not specify the order of
the fragments.
In all cases (TYPEs 2, 3 and 4), if the value of THIS is greater
than TOTAL or if TOTAL equals zero (0x0), the fragment SHALL be
discarded.
o Finally, the sample contents following the SLEN field consist of a
fragment of the UTF-8/UTF-16 character string; no modifiers follow.
4.1.4. TYPE 3 Header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE | LEN( always >6) |TOTAL | THIS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6. TYPE 3 Header Format
This header type is used to transport either the entire modifier
contents present in a text sample or just the first fragment of them.
This depends on whether the modifier boxes fit in the current RTP
payload.
If a text sample containing modifiers is fragmented, this header MUST
be used to transport the first fragment or, if possible, the complete
modifiers.
In detail:
o The U, R, and TYPE fields are defined as in Section 4.1.1.
o LEN indicates the length of the modifier contents. Its value is
obtained upon fragmentation. Additionally, the LEN field MUST be
greater than six (0x0006). Otherwise, the unit MUST be discarded.
o The TOTAL/THIS field has the same meaning as for TYPE 2.
For TYPE 3 units containing the last (trailing) modifier fragment,
the value of TOTAL MUST be equal to that of THIS (TOTAL=THIS). In
addition, TOTAL=THIS MUST be greater than one, because the total
number of fragments of a text sample is logically always larger
than one.
Otherwise, if TOTAL is different from THIS in a TYPE 3 unit, this
means that the unit contains the first fragment of the modifiers.
o The SDUR has the same definition for TYPE 1. Since the fragments
are always transported in own RTP packets, this field is only
needed to know how long this fragment is valid. This may, e.g., be
used to determine how long it should be kept in the display buffer.
Note that the SLEN and SIDX fields are not present in TYPE 3 unit
headers. This is because a) these fragments do not contain text
strings and b) these types of fragments are applied over text string
fragments, which already contain this information.
4.1.5. TYPE 4 Header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE | LEN( always >6) |TOTAL | THIS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 7. TYPE 4 Header Format
This header type is placed before modifier fragments, other than the
first one.
The U, R, and TYPE fields are used as per Section 4.1.1.
LEN indicates as for TYPE 3 the length of the modifier contents and
SHALL also be obtained upon fragmentation. The LEN field MUST be
greater than six (0x0006). Otherwise, the unit MUST be discarded.
TOTAL/THIS is used as in TYPE 2.
The SDUR field is defined as in TYPE 1. The reasoning behind the
absence of SLEN and SIDX is the same as in TYPE 3 units.
4.1.6. TYPE 5 Header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE | LEN( always >3) | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 8. TYPE 5 Header Format
This header type is used to transport (dynamic) sample descriptions.
Every sample description MUST have its own TYPE 5 header.
The U, R, and TYPE fields are used as per Section 4.1.1.
The LEN field indicates the length of the sample description, plus
three units accounting for the SIDX and LEN field itself. Thus, this
field MUST be greater than three (0x0003). Otherwise, the unit MUST
be discarded.
If the sample is streamed from a 3GP file, the length of the sample
description contents (i.e., what comes after SIDX in the unit itself)
is obtained from the file (see Section 4.3).
The SIDX field contains a dynamic SIDX value assigned to the sample
description carried as sample content of this unit. As only dynamic
sample descriptions are carried using TYPE 5, the possible SIDX
values are in the (closed) interval [0,127].
Senders MAY make use of TYPE 5 units. All receivers MUST implement
support for TYPE 5 units, since it adds minimum complexity and may
increase the robustness of the streaming session.
The next section specifies how SIDX values are calculated.
4.2. Buffering of Sample Descriptions
The buffering of sample descriptions is a matter of the client’s
timed text codec implementation. In order to work properly, this
payload format requires that:
o Static sample descriptions MUST be buffered at the client, at
least, for the duration of the session.
o If dynamic sample descriptions are used, their buffering and
update of the SIDX values MUST follow the mechanism described in
the next section.
4.2.1. Dynamic SIDX Wraparound Mechanism
The use of dynamic sample descriptions by senders is OPTIONAL.
However, if they are used, senders MUST implement this mechanism.
Receivers MUST always implement it.
Dynamic SIDX values remain active either during the entire duration
of the session (if used just once) or in different intervals of it
(if used once or more).
Note: In the following, SIDX means dynamic SIDX.
For choosing the wraparound mechanism, the following rationale was
used: There are 128 dynamic SIDX values possible, [0..127]. If one
chooses to allow a maximum of 127 to be used as dynamic SIDXs, then
any reordered packet with a new sample description would make the
mechanism fail. For example, if the last packet received is SIDX=5,
then all 127 values except SIDX=6 would be "active". Now, if a
reordered packet arrives with a new description, SIDX=9, it will be
mistakenly discarded, because the SIDX=9 is, at that moment, marked
as "active" and active sample descriptions shall not be re-written.
Therefore, a "guard interval" is introduced. This guard interval
reduces the number of active SIDXs at any point in time to 64.
Although most timed text applications will probably need less than 64
sample descriptions during a session (in total), a wraparound
mechanism to handle the need for more is described here.
Thereby, a sliding window of 64 active SIDX values is used. Values
within the window are "active"; all others are marked "inactive". An
SIDX value becomes active if at least one sample description
identified by that SIDX has been received. Since sample descriptions
MAY be sent redundantly, it is possible that a client receives a
given SIDX several times. However, active sample descriptions SHALL
NOT be overwritten: The receiver SHALL ignore redundant sample
descriptions and it MUST use the already cached copy. The "guard
interval" of (64) inactive values ensures that the correct
association SIDX <-> sample description is always used.
Informative note: As for the "guard interval" value itself, 64
as 128/2 was considered simple enough while still meeting the
expected maximum number of sample descriptions. Besides that,
there’s no other motivation for choosing 64 or a different
value.
The following algorithm is used to buffer dynamic sample descriptions
and to maintain the dynamic SIDX values:
Let X be the last SIDX received that updated the range of active
sample descriptions. Let Y be a value within the allowed range for
dynamic SIDX: [0,127], and different from X. Let Z be the SIDX of
the last received sample description. Then:
1. Initialize all dynamic SIDX values as inactive. For stored
contents, read the sample description index in the Sample to
Chunk box ("stsc") for that sample. For live streaming, the
first value MAY be zero or any other value in the interval
above. Go to step 2.
2. First, in-band sample description with SIDX=Z is received and
stored; set X=Z. Go to step 3.
3. Any SIDX within the interval [X+1 modulo(128), X+64 modulo(128)]
is marked as inactive, and any corresponding sample description
is deleted. Any SIDX within the interval [X+65 modulo(128), X]
is set active. Go to step 4 (wait state).
4. Wait for next sample description. Once the client is
initialized, the interval of active SIDX values MUST change
whenever a sample description with an SIDX value in the inactive
set is received. That is, upon reception of a sample
description with SIDX=Z, do the following:
a. If Z is in the (closed) interval [X+1 modulo(128), X+64
modulo(128)] then set X=Z, store the sample description, and
go to step 3.
b. Else, Z must be in the interval [X+65 modulo(128), X], thus:
i. If SIDX=Z is not stored, then store the sample
description. Go to beginning of step 4 (wait state).
ii. Else, go to the beginning of step 4 (wait state).
Informative note: It is allowed that any value of SIDX=X be sent
in the interval [0,127]. For example, if [64..127] is the
current active set and SIDX=0 is sent, a new sample description
is defined (0) and an old one deleted (64); thus [65..127] and
[0] are active. Similarly, one could now send SIDX=64, thus
inverting the active and inactive sets.
Example:
If X=4, any SIDX in the interval [5,68] is inactive. Active
SIDX values are in the complementary interval [69,127] plus
[0,4]. For example, if the client receives a SIDX=6, then the
active interval is now different: [0,6] plus [71,127]. If the
received SIDX is in the current active interval, no change SHALL
be applied.
4.3. Finding Payload Header Values in 3GP Files
For the purpose of streaming timed text contents, some values in the
boxes contained in a 3GP file are mapped to fields of this payload
header. This section explains where to find those values.
Additionally, for the duration and sample description indexes,
extension mechanisms are provided. All senders MUST implement the
extension mechanisms described herein.
If the file is streamed out of a 3GP file, the following guidelines
SHALL be followed.
Note: All fields in the objects (boxes) of a 3GP file are found
in network byte order.
Information obtained from the Sample Table Box (stbl):
o Sample Descriptions and Sample Description length: The Sample
Description box (stsd, inside the stbl) contains the sample
descriptions. For timed text media, each element of stsd is a