Tech Stuff - Audio/Video Files, Codecs and Containers

We can never find stuff in one place - here is a collection of audio/video file formats together with some basic notes and links to additional information. Helped us, perhaps it will help you. If not .... well .... sad, but not terminal.

<Big Deal> September, 2012 marked the publication of RFC 6716 which defined a new royalty-free codec which goes by the name of Opus, and represents, essentially, the merger of work done by Skype and Xiph.org (developers of ogg, speex and other codecs). While we have had royalty-free codes for some time (notably, speex and ogg vorbis) this is the first time we have seen one with the imprimatur of the IETF. This is a big deal. Opus offers a wide range of bit rates (6Kbit/s to 510Kbit/s), frame sizes (2.5ms to 120ms) and sampling rates (8 kHz to 48KHz) making it suitable for interactive (VoIP), streaming and audio playback. Quality is comparable to HE-AAC/Orbis at the top end and very close to the best proprietary narrowband codecs at the bottom end. Not only is it free, it's also very good. </Big Deal>

Contents

1. Overview - Is it a file, a container or a codec
2. File Types and Extensions
3. Codecs
4. Audio/Video Containers
5. - AIFF/AIFF-C Container
6. - ADTS Container
7. - WAVE Container
8. - OGG Container
9. Meta or Tag Data

Is it a File, a Container or a Codec

So you have a sound file with the name sound.mp3 or sound.wav - does this describe a file format, a container format or a codec?

It depends on the file extension. For example the file sound.mp3 contains MP3 data that can only be interpreted and played by a MP3 codec (except, confusingly, it can also contain an ID3 tag frame at the begining or the end of the file). Thus a file with the extension .mp3 has a file format specific to a single, in this case mp3, codec - it does not use a container format.

So what is a container? A container is a standardized envelope that typically includes fields that indicate which codec should be used to play the enclosed audio/video material and may or may not contain a format to decsribe meta (tag) data. As an example, the file sound.wav uses a WAVE container in which the codec to be used is indicated in the format chunk's CODEC field. Thus an application could read a .wav file and select from a number of different codecs to handle the audio material in the data chunk (each codec would clearly have to be able to interpret the data).

So what is a codec? A codec is a widely used generic term used to describe software that handles a specific audio data format. The term originally was a shortened version of coder/decoder and thus refered to software that could provide both encoding (writing) functions and decoding (playback) functions. Today, what is typically called a codec rarely provides both capabilities. Rather, what is called an mp3 codec is typically an mp3 decoder used to playback mp3 sound files. If the software also supports creation (encoding) of mp3 files it will typically use a separate mp3 encoder. The distinction is only important if you are trying to transcode (convert) A/V files.

The following section on AV file extensions defines whether files are codec specific or use a container.

File Types and Extensions

The following table is a (currently incomplete) list of commonly used file extensions. It gives a brief description of the contents, codecs and any container formats used.

Codecs

A codec (originally short for coder/decoder) is a software program or library that knows how to handle audio/visual material in a specific format.

The crucial difference in the various formats is between lossy and lossless. In a lossy format some of the orginal source material is discarded using a variety of sophisticated algorithms to retain as much as possible of the original sound quality (typically using psychoacoustics models). File sizes for lossy formats are typically 10:1 smaller than the original source material. Lossless formats as the name suggests retain all the source material. There are now a number of compressed lossless formats, such as FLAC which maintain all the original material but use classic lossless data compression techniques (such as the LZ family) to compress file size. File sizes tend to be only ~2:1 smaller than the original (though for high volume archival applications this can represent a huge space saving). Some additional points to note: open or proprietary standards and whether or not there are patent/royalty issues involved.

Containers

Containers are simply envelopes (or frameworks) that describe the contents of the audio/video material and typically, but not always, define which codec should be used to handle the encapsulated A/V data. Containers may, or may not, include a standard method of providing meta (tag) data.

AIFF and AIFF-C Container

The AIFF and AIFF-C containers are based on the RIFF format and use a series of chunks, each with a unique Chunk ID, to describe various sections of the container. It is similar in format to the WAV container with one vital exception - AIFF/AIFF-C format containers use big-endian format, whereas WAV containers use little-endian multi-byte format.

The basic AIFF format does not provide a codec field in the COMM chunk, whereas the AIFF-C format does. Neither the AIFF nor AIFF-C containers allow for the definition of meta (tag) information though the use of the ID3 chunk is widespread.

A number of other chunk types are defined within the AIFF/AIFF-C specifications and typically have specialized usage. In addition third parties have defined chunks for a variety of purposes.

An AIFF container must have at least a FORM chunk, a COMM chunk and a SSND chunk. Theoretically the chunks can be in any order but this leads to very inefficient file processing.

FORM Chunk (12 bytes):

COMM Chunk:

FVER Chunk (12 bytes):(Mandatory for AIFF-C files)

SSND Chunk (variable):

ADTS Container

Hardly worthy of the name container, the Audio Data Transport Stream is a vestigal envelope defined in MPEG-2 Part 7 (and used with file extension .aac) but optional in MPEG-4 (with file extensions .m4a or mp4), thus while it is permitted to encode MPEG-4 using ADTS not all MPEG-4 codecs can handle it. ADTS is optimized for streaming media, however it can be, and frequently is, also used for playback files. It does not provide a method to encode meta (tag) information nor does it provide any codec selection though it does, of necessity, allow selection of one of the many AAC options.

The ADTS header (7 or 9 bytes) is transmitted with every ADTS frame or packet (the header defines the frame length). In the table below bits are numbered left to right starting from 0.

0 = Null (invalid)
1 = AAC Main
2 = AAC LC (Low Complexity)
3 = AAC SSR (Scalable Sample Rate)
4 = AAC LTP (Long Term Prediction)
5 = SBR (Spectral Band Replication)
6 = AAC Scalable
7 = TwinVQ
8 = ELP (Code Excited Linear Prediction)
9 = HXVC (Harmonic Vector eXcitation Coding)
10 = Reserved
11 = Reserved
12 = TTSI (Text-To-Speech Interface)
13: Main Synthesis
14: Wavetable Synthesis
15: General MIDI
16: Algorithmic Synthesis and Audio Effects
17: ER (Error Resilient) AAC LC
18: Reserved
19: ER AAC LTP
20: ER AAC Scalable
21: ER TwinVQ
22: ER BSAC (Bit-Sliced Arithmetic Coding)
23: ER AAC LD (Low Delay)
24: ER CELP
25: ER HVXC
26: ER HILN (Harmonic and Individual Lines plus Noise)
27: ER Parametric
28: SSC (SinuSoidal Coding)
29: PS (Parametric Stereo)
30: MPEG Surround
31: (Escape value)
32: Layer-1
33: Layer-2
34: Layer-3
35: DST (Direct Stream Transfer)
36: ALS (Audio Lossless)
37: SLS (Scalable LosslesS)
38: SLS non-core
39: ER AAC ELD (Enhanced Low Delay)
40: SMR (Symbolic Music Representation) Simple
41: SMR Main
42: USAC (Unified Speech and Audio Coding) (no SBR)
43: SAOC (Spatial Audio Object Coding)
44: LD MPEG Surround
45: USAC 

0 = 96000 Hz
1 = 88200 Hz
2 = 64000 Hz
3 = 48000 Hz
4 = 44100 Hz
5 = 32000 Hz
6 = 24000 Hz
7 = 22050 Hz
8 = 16000 Hz
9 = 12000 Hz
10 = 11025 Hz
11 = 8000 Hz
12 = 7350 Hz
13 = Reserved
14 = Reserved 
15 = invalid in ADTS

0 = Defined in AOT Specifc Config (not valid for ADTS)
1 = 1 channel: front-center
2 = 2 channels: front-left, front-right
3 = 3 channels: front-center, front-left, front-right
4 = 4 channels: front-center, front-left, front-right, back-center
5 = 5 channels: front-center, front-left, front-right, back-left, 
                back-right
6 = 6 channels: front-center, front-left, front-right, back-left, 
                back-right, LFE-channel (surround)
7 = 8 channels: front-center, front-left, front-right, side-left, side-right, 
                back-left, back-right, LFE-channel (surround)
8-15: Reserved (not valid)

OGG Container

OGG Container format used for FLAC, Orbis and Opus audio files.

WAVE Container

IBM and Microsoft standard but has been extended by others. The format of a WAV container is a subset of the generic RIFF (Resource Interchange File Format - also used by AIFF files but with different chunk IDs) format as shown below. The current specification is version 3.0 though an extended multi-channel format is available. The basic definition consists of three standard 'Chunks' - the HEADER chunk (identifying the file type), the FORMAT chunk (identifying key characteristics of the payload) and the DATA chunk containing the data (or file payload). If a non PCM codec is being used a FACT chunk must be included. Due to the format of the file additional chunks can be added. Software that knows about such chunks can handle them, but sensible software that does not can simply skip over unrecognized chunks until it reaches the audio data (in the data chunk. There are a number of well known chunks (typically used to incorporate meta data (tags)) and the chunk we use to incorporate OGG COMMENT tags in both WAV and AIFF format files.

HEADER Chunk (12 bytes):

NOTE: All multi-byte values are in little-endian order. If big-endian (network) order is being used the value RIFX will replace RIFF in the HEADER chunk.

FORMAT Chunk (24+ bytes):

FACT Chunk (variable):

DATA Chunk (variable):

OGG COMMENT Chunk Format

OGG Comment Chunk (variable). This is a completely non-standard chunk which we use in WAVE and AIFF files to encapsulate Ogg style comments. Serious implementations of AIFF or WAV playback systems will skip unknown chunks (the container format allows for this). Crummy implementations will croak:

Well Known Chunks

There are a number of other chunk types that can appear in either WAV or AIFF containers. Because of the file format used in both containers serious software can easily skip over unrecognized chunks using the chunk length field. If files are being copied, sensible and serious software should always write all chunks even if does not support/recognize them thus the possiblity of possibly alien chunk loss should be remote. The following is an incomplete list of additional chunks:

• iXML: The chunk ID "iXML" indicates the presence of an iXML frame a widely supported open, freely available meta data (tag) specification based on XML used by many audio equipment manufacturers.

• ID3: The chunk ID "ID3 " indicates the presence of a standard ID3v2.3 or v2.4 tag frame as used in mp3 files.

• BWF: The Broadcast Wave Format defined by European Broadcast Union (EBU) Technical Report 3285 defines a number of chunk types for use in WAV format files.

Meta Data - Audio Tags

It is a great idea. The media/audio file is self describing by containing tagging or meta data. You send the audio file. The player can both play the file and extract a ton of meta data like who is performing, the name of the work etc. No need to supply separate identification information. However, as you will see below the world has gone a bit crazy with a huge number of tags being made available or proposed. In our view the whole idea has gone off the rails. Audio files are not databases. Instead, again in our view, the media/audio file should contain essential and summary infomation only - perhaps limited to identification (artist/title), rights, and maybe audio parameters (gain etc.). Everything else should be defined in a simple MORE= tag containing the URL of a networked resource which can supply unlimited information about the material. This is our proposal.

MP3 Meta Data (ID3)

The MP3 standard does not use a container format thus does not provide the ability to encode meta (tag) data. What are colloquially known as ID3 tags came into existence - bolted on afterwards and now almost universally supported - to allow mp3 files to describe the contents of the audio file. Initially released as version 1 now updated to the completely different version 2.x format. Currently this is at version 2.4, though 2.3 still seems the most widely implemented. The list of tags defined by www.id3.org is pretty daunting (currently 94 and counting) and includes the shoe sizes of all the musicians involved. In practice, only a trivial subset is widely supported - perhaps 10 - 20 - the other tags tend to be for specialized usage. So now you want to transcode from/to MP3, WMA and Vorbis or whatever. And the world suddenly gets a lot murkier.

Ogg Meta Data

Ogg does not support ID3. Instead it uses a flexible and extensible name=data tag format (unfortunately generically called a comment). Ogg are currently promoting a list of name types. The most important characteristic about this list is that is seems to be widely ignored (see below) - in some cases because the list does not have an ID3 (still the dominant standard) equivalent. FLAC - from the Ogg stable also uses the Vorbis comment format. Numerous alternative proposals exist on the web for uses of this flexible tag format and taken together they make the 94 items of ID3 look very modest indeed. Never-the-less, IOHO, it's a more useful format than ID3 and has the overriding merit of being human readable even if the player/reader software is unable to interpret the tags. An extended comment field format to include images has been made available by Ogg (Ogg Extended comment field format proposal).

Wave/AIFF Meta Data

WAVE and AIFF both use the same RIFF format which allows for new chunks to be added without breaking the container format - unless, of course, your player/reader is unbelievably stupid. Software can just skip over the unknown chunks so at least the audio file can be played. Thus it is possible to embed any tag format using a possibly proprietary chunk format. However, when transcoding the player/writer must understand the meta data format in order to map it to an equivalent format. Now this is where it gets really stupid. AIF defines an ID3 chunk whereas Wave does not. WAVE has the Broadcast WAVE Format (BWF) which is optimized for, as you would expect , broadcasters and does allow a meta (tag) data but this has very specialized data for use by broadcasters and is not equivalent, in any way, to typical ID3 tags.

While there are differences between WAVE and AIF these are mostly in the area of extended capabilities. We use a common CODEC for both thus, as far as we are concerned, WAVE has embedded id3 capability using the ID3 chunk format. However, for reasons mentioned we like the flexibility of OGG and have started to make serious use of both Vorbis and FLAC files. To handle this problem we use the OGG chunk format defined above to encapsulate the standard Ogg comment fields when we are forced to use either AIF or WAV files. And if you need to export the files. Add both the id3 and ogg chunks. Both .aif(.aiff) and .wav files are pretty big - what's a few (hundred) extra K here or there.

MP4 Meta Data

The standard MP4 container (used in .mp4 and .m4a files) allows definition of meta (tag) data within the container.

Placeholder - additional detail to be added

AAC Meta Data

When AAC audio is encapsulated in an MP4 container (MPEG-4 Part 3) using the extensions .mp4 and .m4a meta (tag) data is allowed. However, AAC can also be encapsulated (as MPEG-2 Part 7) in raw (no container), or ADIF or ADTS containers (all with file extension .aac) and in this case meta (tag) data is not defined by the container format.

Placeholder - additional detail to be added

WMA Meta Data

The standard WMA container provides a Content Description Object which allows a subset of meta (tag) data to be defined and additionally provides a flexible Extended Content Description Object which allows almost unlimited meta (tag) data.

Placeholder - additional detail to be added

Meta Tag Mapping

The following table shows of the widely supported ID3 tags and one possible mapping to Ogg container alternatives which could be used during transcoding.

Other tags seem to be supported to a greater or lesser extent, depending on the software. Certainly you can be sure of only one thing - inconsistency.

Problems, comments, suggestions, corrections (including broken links) or something to add? Please take the time from a busy life to 'mail us' (at top of screen), the webmaster (below) or info-support at zytrax. You will have a warm inner glow for the rest of the day.