00001 <HTML><HEAD><TITLE>xiph.org: Ogg Vorbis documentation</TITLE>
00002 <BODY bgcolor="#ffffff" text="#202020" link="#006666" vlink="#000000">
00003 <nobr><a href="http://www.xiph.org/ogg/index.html"><img src="white-ogg.png" border=0><img
00004 src="vorbisword2.png" border=0></a></nobr><p>
00005
00006 <h1><font color=#000070>
00007 Ogg logical bitstream framing
00008 </font></h1>
00009
00010 <em>Last update to this document: July 14, 2002</em><br>
00011
00012 <h2>Ogg bitstreams</h2>
00013
00014 The Ogg transport bitstream is designed to provide framing, error
00015 protection and seeking structure for higher-level codec streams that
00016 consist of raw, unencapsulated data packets, such as the Vorbis audio
00017 codec or Tarkin video codec.
00018
00019 <h2>Application example: Vorbis</h2>
00020 Vorbis encodes short-time blocks of PCM data into raw packets of
00021 bit-packed data. These raw packets may be used directly by transport
00022 mechanisms that provide their own framing and packet-separation
00023 mechanisms (such as UDP datagrams). For stream based storage (such as
00024 files) and transport (such as TCP streams or pipes), Vorbis uses the
00025 Ogg bitstream format to provide framing/sync, sync recapture
00026 after error, landmarks during seeking, and enough information to
00027 properly separate data back into packets at the original packet
00028 boundaries without relying on decoding to find packet boundaries.<p>
00029
00030 <h2>Design constraints for Ogg bitstreams</h2>
00031
00032 <ol><li>True streaming; we must not need to seek to build a 100%
00033 complete bitstream.
00034
00035 <li> Use no more than approximately 1-2% of bitstream bandwidth for
00036 packet boundary marking, high-level framing, sync and seeking.
00037
00038 <li> Specification of absolute position within the original sample
00039 stream.
00040
00041 <li> Simple mechanism to ease limited editing, such as a simplified
00042 concatenation mechanism.
00043
00044 <li> Detection of corruption, recapture after error and direct, random
00045 access to data at arbitrary positions in the bitstream.
00046 </ol>
00047
00048 <h2>Logical and Physical Bitstreams</h2>
00049
00050 A <em>logical</em> Ogg bitstream is a contiguous stream of
00051 sequential pages belonging only to the logical bitstream. A
00052 <em>physical</em> Ogg bitstream is constructed from one or more
00053 than one logical Ogg bitstream (the simplest physical bitstream
00054 is simply a single logical bitstream). We describe below the exact
00055 formatting of an Ogg logical bitstream. Combining logical
00056 bitstreams into more complex physical bitstreams is described in the
00057 <a href="oggstream.html">Ogg bitstream overview</a>. The exact
00058 mapping of raw Vorbis packets into a valid Ogg Vorbis physical
00059 bitstream is described in <a href="vorbis-stream.html">Vorbis
00060 bitstream mapping</a>.
00061
00062 <h2>Bitstream structure</h2>
00063
00064 An Ogg stream is structured by dividing incoming packets into
00065 segments of up to 255 bytes and then wrapping a group of contiguous
00066 packet segments into a variable length page preceded by a page
00067 header. Both the header size and page size are variable; the page
00068 header contains sizing information and checksum data to determine
00069 header/page size and data integrity.<p>
00070
00071 The bitstream is captured (or recaptured) by looking for the beginning
00072 of a page, specifically the capture pattern. Once the capture pattern
00073 is found, the decoder verifies page sync and integrity by computing
00074 and comparing the checksum. At that point, the decoder can extract the
00075 packets themselves.<p>
00076
00077 <h3>Packet segmentation</h3>
00078
00079 Packets are logically divided into multiple segments before encoding
00080 into a page. Note that the segmentation and fragmentation process is a
00081 logical one; it's used to compute page header values and the original
00082 page data need not be disturbed, even when a packet spans page
00083 boundaries.<p>
00084
00085 The raw packet is logically divided into [n] 255 byte segments and a
00086 last fractional segment of < 255 bytes. A packet size may well
00087 consist only of the trailing fractional segment, and a fractional
00088 segment may be zero length. These values, called "lacing values" are
00089 then saved and placed into the header segment table.<p>
00090
00091 An example should make the basic concept clear:<p>
00092
00093 <pre>
00094 <tt>
00095 raw packet:
00096 ___________________________________________
00097 |______________packet data__________________| 753 bytes
00098
00099 lacing values for page header segment table: 255,255,243
00100 </tt>
00101 </pre>
00102
00103 We simply add the lacing values for the total size; the last lacing
00104 value for a packet is always the value that is less than 255. Note
00105 that this encoding both avoids imposing a maximum packet size as well
00106 as imposing minimum overhead on small packets (as opposed to, eg,
00107 simply using two bytes at the head of every packet and having a max
00108 packet size of 32k. Small packets (<255, the typical case) are
00109 penalized with twice the segmentation overhead). Using the lacing
00110 values as suggested, small packets see the minimum possible
00111 byte-aligned overheade (1 byte) and large packets, over 512 bytes or
00112 so, see a fairly constant ~.5% overhead on encoding space.<p>
00113
00114 Note that a lacing value of 255 implies that a second lacing value
00115 follows in the packet, and a value of < 255 marks the end of the
00116 packet after that many additional bytes. A packet of 255 bytes (or a
00117 multiple of 255 bytes) is terminated by a lacing value of 0:<p>
00118
00119 <pre><tt>
00120 raw packet:
00121 _______________________________
00122 |________packet data____________| 255 bytes
00123
00124 lacing values: 255, 0
00125 </tt></pre>
00126
00127 Note also that a 'nil' (zero length) packet is not an error; it
00128 consists of nothing more than a lacing value of zero in the header.<p>
00129
00130 <h3>Packets spanning pages</h3>
00131
00132 Packets are not restricted to beginning and ending within a page,
00133 although individual segments are, by definition, required to do so.
00134 Packets are not restricted to a maximum size, although excessively
00135 large packets in the data stream are discouraged; the Ogg
00136 bitstream specification strongly recommends nominal page size of
00137 approximately 4-8kB (large packets are foreseen as being useful for
00138 initialization data at the beginning of a logical bitstream).<p>
00139
00140 After segmenting a packet, the encoder may decide not to place all the
00141 resulting segments into the current page; to do so, the encoder places
00142 the lacing values of the segments it wishes to belong to the current
00143 page into the current segment table, then finishes the page. The next
00144 page is begun with the first value in the segment table belonging to
00145 the next packet segment, thus continuing the packet (data in the
00146 packet body must also correspond properly to the lacing values in the
00147 spanned pages. The segment data in the first packet corresponding to
00148 the lacing values of the first page belong in that page; packet
00149 segments listed in the segment table of the following page must begin
00150 the page body of the subsequent page).<p>
00151
00152 The last mechanic to spanning a page boundary is to set the header
00153 flag in the new page to indicate that the first lacing value in the
00154 segment table continues rather than begins a packet; a header flag of
00155 0x01 is set to indicate a continued packet. Although mandatory, it
00156 is not actually algorithmically necessary; one could inspect the
00157 preceding segment table to determine if the packet is new or
00158 continued. Adding the information to the packet_header flag allows a
00159 simpler design (with no overhead) that needs only inspect the current
00160 page header after frame capture. This also allows faster error
00161 recovery in the event that the packet originates in a corrupt
00162 preceding page, implying that the previous page's segment table
00163 cannot be trusted.<p>
00164
00165 Note that a packet can span an arbitrary number of pages; the above
00166 spanning process is repeated for each spanned page boundary. Also a
00167 'zero termination' on a packet size that is an even multiple of 255
00168 must appear even if the lacing value appears in the next page as a
00169 zero-length continuation of the current packet. The header flag
00170 should be set to 0x01 to indicate that the packet spanned, even though
00171 the span is a nil case as far as data is concerned.<p>
00172
00173 The encoding looks odd, but is properly optimized for speed and the
00174 expected case of the majority of packets being between 50 and 200
00175 bytes (note that it is designed such that packets of wildly different
00176 sizes can be handled within the model; placing packet size
00177 restrictions on the encoder would have only slightly simplified design
00178 in page generation and increased overall encoder complexity).<p>
00179
00180 The main point behind tracking individual packets (and packet
00181 segments) is to allow more flexible encoding tricks that requiring
00182 explicit knowledge of packet size. An example is simple bandwidth
00183 limiting, implemented by simply truncating packets in the nominal case
00184 if the packet is arranged so that the least sensitive portion of the
00185 data comes last.<p>
00186
00187 <h3>Page header</h3>
00188
00189 The headering mechanism is designed to avoid copying and re-assembly
00190 of the packet data (ie, making the packet segmentation process a
00191 logical one); the header can be generated directly from incoming
00192 packet data. The encoder buffers packet data until it finishes a
00193 complete page at which point it writes the header followed by the
00194 buffered packet segments.<p>
00195
00196 <h4>capture_pattern</h4>
00197
00198 A header begins with a capture pattern that simplifies identifying
00199 pages; once the decoder has found the capture pattern it can do a more
00200 intensive job of verifying that it has in fact found a page boundary
00201 (as opposed to an inadvertent coincidence in the byte stream).<p>
00202
00203 <pre><tt>
00204 byte value
00205
00206 0 0x4f 'O'
00207 1 0x67 'g'
00208 2 0x67 'g'
00209 3 0x53 'S'
00210 </tt></pre>
00211
00212 <h4>stream_structure_version</h4>
00213
00214 The capture pattern is followed by the stream structure revision:
00215
00216 <pre><tt>
00217 byte value
00218
00219 4 0x00
00220 </tt></pre>
00221
00222 <h4>header_type_flag</h4>
00223
00224 The header type flag identifies this page's context in the bitstream:
00225
00226 <pre><tt>
00227 byte value
00228
00229 5 bitflags: 0x01: unset = fresh packet
00230 set = continued packet
00231 0x02: unset = not first page of logical bitstream
00232 set = first page of logical bitstream (bos)
00233 0x04: unset = not last page of logical bitstream
00234 set = last page of logical bitstream (eos)
00235 </tt></pre>
00236
00237 <h4>absolute granule position</h4>
00238
00239 (This is packed in the same way the rest of Ogg data is packed; LSb
00240 of LSB first. Note that the 'position' data specifies a 'sample'
00241 number (eg, in a CD quality sample is four octets, 16 bits for left
00242 and 16 bits for right; in video it would likely be the frame number.
00243 It is up to the specific codec in use to define the semantic meaning
00244 of the granule position value). The position specified is the total
00245 samples encoded after including all packets finished on this page
00246 (packets begun on this page but continuing on to the next page do not
00247 count). The rationale here is that the position specified in the
00248 frame header of the last page tells how long the data coded by the
00249 bitstream is. A truncated stream will still return the proper number
00250 of samples that can be decoded fully.
00251 <p>
00252 A special value of '-1' (in two's complement) indicates that no packets
00253 finish on this page.
00254
00255 <pre><tt>
00256 byte value
00257
00258 6 0xXX LSB
00259 7 0xXX
00260 8 0xXX
00261 9 0xXX
00262 10 0xXX
00263 11 0xXX
00264 12 0xXX
00265 13 0xXX MSB
00266 </tt></pre>
00267
00268 <h4>stream serial number</h4>
00269
00270 Ogg allows for separate logical bitstreams to be mixed at page
00271 granularity in a physical bitstream. The most common case would be
00272 sequential arrangement, but it is possible to interleave pages for
00273 two separate bitstreams to be decoded concurrently. The serial
00274 number is the means by which pages physical pages are associated with
00275 a particular logical stream. Each logical stream must have a unique
00276 serial number within a physical stream:
00277
00278 <pre><tt>
00279 byte value
00280
00281 14 0xXX LSB
00282 15 0xXX
00283 16 0xXX
00284 17 0xXX MSB
00285 </tt></pre>
00286
00287 <h4>page sequence no</h4>
00288
00289 Page counter; lets us know if a page is lost (useful where packets
00290 span page boundaries).
00291
00292 <pre><tt>
00293 byte value
00294
00295 18 0xXX LSB
00296 19 0xXX
00297 20 0xXX
00298 21 0xXX MSB
00299 </tt></pre>
00300
00301 <h4>page checksum</h4>
00302
00303 32 bit CRC value (direct algorithm, initial val and final XOR = 0,
00304 generator polynomial=0x04c11db7). The value is computed over the
00305 entire header (with the CRC field in the header set to zero) and then
00306 continued over the page. The CRC field is then filled with the
00307 computed value.<p>
00308
00309 (A thorough discussion of CRC algorithms can be found in <a
00310 href="ftp://ftp.rocksoft.com/papers/crc_v3.txt">"A
00311 Painless Guide to CRC Error Detection Algorithms"</a> by Ross
00312 Williams <a
00313 href="mailto:ross@guest.adelaide.edu.au">ross@guest.adelaide.edu.au</a>.)
00314
00315 <pre><tt>
00316 byte value
00317
00318 22 0xXX LSB
00319 23 0xXX
00320 24 0xXX
00321 25 0xXX MSB
00322 </tt></pre>
00323
00324 <h4>page_segments</h4>
00325
00326 The number of segment entries to appear in the segment table. The
00327 maximum number of 255 segments (255 bytes each) sets the maximum
00328 possible physical page size at 65307 bytes or just under 64kB (thus
00329 we know that a header corrupted so as destroy sizing/alignment
00330 information will not cause a runaway bitstream. We'll read in the
00331 page according to the corrupted size information that's guaranteed to
00332 be a reasonable size regardless, notice the checksum mismatch, drop
00333 sync and then look for recapture).<p>
00334
00335 <pre><tt>
00336 byte value
00337
00338 26 0x00-0xff (0-255)
00339 </tt></pre>
00340
00341 <h4>segment_table (containing packet lacing values)</h4>
00342
00343 The lacing values for each packet segment physically appearing in
00344 this page are listed in contiguous order.
00345
00346 <pre><tt>
00347 byte value
00348
00349 27 0x00-0xff (0-255)
00350 [...]
00351 n 0x00-0xff (0-255, n=page_segments+26)
00352 </tt></pre>
00353
00354 Total page size is calculated directly from the known header size and
00355 lacing values in the segment table. Packet data segments follow
00356 immediately after the header.<p>
00357
00358 Page headers typically impose a flat .25-.5% space overhead assuming
00359 nominal ~8k page sizes. The segmentation table needed for exact
00360 packet recovery in the streaming layer adds approximately .5-1%
00361 nominal assuming expected encoder behavior in the 44.1kHz, 128kbps
00362 stereo encodings.<p>
00363
00364 <hr>
00365 <a href="http://www.xiph.org/">
00366 <img src="white-xifish.png" align=left border=0>
00367 </a>
00368 <font size=-2 color=#505050>
00369
00370 Ogg is a <a href="http://www.xiph.org">Xiph.org Foundation</a> effort
00371 to protect essential tenets of Internet multimedia from corporate
00372 hostage-taking; Open Source is the net's greatest tool to keep
00373 everyone honest. See <a href="http://www.xiph.org/about.html">About
00374 the Xiph.org Foundation</a> for details.
00375 <p>
00376
00377 Ogg Vorbis is the first Ogg audio CODEC. Anyone may freely use and
00378 distribute the Ogg and Vorbis specification, whether in a private,
00379 public or corporate capacity. However, the Xiph.org Foundation and
00380 the Ogg project (xiph.org) reserve the right to set the Ogg Vorbis
00381 specification and certify specification compliance.<p>
00382
00383 Xiph.org's Vorbis software CODEC implementation is distributed under a
00384 BSD-like license. This does not restrict third parties from
00385 distributing independent implementations of Vorbis software under
00386 other licenses.<p>
00387
00388 Ogg, Vorbis, Xiph.org Foundation and their logos are trademarks (tm)
00389 of the <a href="http:
00390 pages are copyright (C) 1994-2002 Xiph.org Foundation. All rights
00391 reserved.<p>
00392
00393 </body>
00394
00395
