Video encoding or video transcoding is the process of exporting digital video into a format and specification suitable for playback by a user. Each video file contains two elements: Codecs, or “compression-decompression“, algorithms which compress the video, and formats, the type of file the data is compressed to. Imagine a shipping container filled with packages of many types. In this analogy the shipping container is the format, and the codec is the tool that creates the packages and places them in the container.
Video File Basics
To understand the difference between a container, a compression scheme, and a codec, first we need to break down the parts of a video file. Think of a video file as a bookshelf filled with books.
Multimedia Container – The Bookshelf
The multimedia container is like the bookshelf in that it can hold many tracks of audio and video just like a bookshelf can hold many books. Common container formats include MOV, MXF, and AVI.
Compression Scheme – The Language of the Book
Inside of each audio and video track, the information is represented in the language of the compression scheme. Common compression schemes include H.264, MPEG4 part 2, ProRes, DNxHD, and so on.
Codec – The Author of the Book
A codec (encoder/decoder) is a piece of hardware or software that interprets an audio or video signal and compresses it. Each compression scheme can be implemented in different ways which creates different codecs for the same compression scheme.
Uses
Video formats, for the purposes of this article can be placed in one or more of the following categories: acquisition, editing, and delivery.
Video formats are often called as a file’s extension, for example the .mp4 in Video. mp4. Video formats are much more than that and include a whole package of files, including a files video stream, the audio stream, and any metadata included with the file. All the data put together, both streams and the metadata, is read by a video player to stream video content for playback. A file’s video stream includes the data necessary for motion video playback, while the audio stream includes any data related to sound. Metadata is any data outside of audio and sound, including bitrate, resolution, subtitles, device type, and date of creation.
Formats are the entire set of storage rules that package a video file’s container, codecs, and metadata. File formats are used so that a variety of devices, platforms, and players can process and playback any number of codecs. Formats are developed to improve image quality, decrease file sizes, allow for universal playback on a variety of devices, and provide special features such as subtitles.
The most popular video format is MP4, also known as MPEG 4. MP4 is a multimedia container for video, audio, and data, including things like subtitles and still images. Don’t confuse MPEG 4 and MPEG 4 Audio; MPEG 4 has the file extension .MP4, while MPEG 4 Audio has the file extension .M4A and can only contain audio files.
MP4 is widely used because of how universal it is across a range of devices. Both mobile and desktop devices can playback MP4 video files, so it makes sense that it would be used by most content producers.
H.264, also known as AVC. H.264 is the most widely used video codec. The H.264 codec compression technology is present everywhere, with decoders available for pretty much every platform and mobile device.
HEVC, short for High Efficiency Video Coding, or H.265 is slowly becoming universal thanks to the spread of 4K content. If AVC or H.264 was the codec of 1080p content, then HEVC and H.265 is the codec for the 4K era. H.265 compresses videos far more efficiently (hence the “high efficiency”) when compared to H.264. This means H.265 4K content that is easier to stream and at a higher quality than 4K content compressed using H.264.
MP4 files are played at a constant bit rate (CBR), meaning the quality of your video stream remains at the same bitrate no matter the variance in your internet bandwidth. There are other file formats that support adaptive bitrate (ABR), which adapts the quality of your stream to match the internet speed of your player. One of the most popular formats that supports ABR is HLS.
HLS or HTTP Live Streaming use adaptive bitrate streaming techniques to enable high quality streaming with small HTTP-based file segments. This format uses HTTP - the universal language protocol for the internet, and allows for reduced infrastructure costs, greater reach, and simple HTML5 player implementation. Another feature for the HLS format is the M3U8 file descriptor. This file allows the client or device to automatically select the best quality stream at any time, to prevent playback buffering regardless of bandwidth or CPU power.
Bitrate or bit rate is the rate at which bits are processed or computed over time. The higher the bitrate, the more data there is, which generally means a higher quality video file. When choosing a bit rate, it is important to consider the device your videos will be played on. High bitrate file streams need lots of processing power to encode, and a fast internet connection to download for playback.
Let’s first begin by talking about codecs, which deal with compressing your file
Codecs (are for compression) – or COder/DECoder
The codec of your original video file is often determined by your video camera or screen recorder, which you may or may not have control over in your camera settings. Examples of popular codecs include h.264 which is often used for digital videos, MPEG2 which is often used for DVD media and TV transmission. Now-a-days video editors and cameras take care of the codec stuff for you and unless you’re looking to get into high-end video or outputting to a very specific destination.
You may have heard the phrase video codec when referring to video files. A codec is simply the software that compresses your video, so it can be stored and played back on a device. While the word “compression” can conjure images of pixelated video, the process is both necessary and efficient with modern digital cameras. It gives you much smaller files sizes with minimal quality loss. Compression is your friend!
Its an encoding tool that processes video and stores it in a stream of bytes. Codecs use algorithms to effectively shrink the size of the audio or video file, and then decompress it when needed. There are dozens of different types of codecs, and each uses a different technology to encode and shrink your video file for the intended application.
Simply put, a codec is a method for encoding and decoding data and more specifically, a protocol for compressing data, especially video. Selecting the right codec can depend on several factors. Target file size, output quality, and delivery method all factor in. Currently h.264 is the predominant codec on the web.
What is it, and why is it important?
There are two kinds of compression: lossless and lossy. Most of the time, the quality can’t be perceived by the human eye, but in some cases, it can make visuals look grainy, sounds flat and muffled, or make videos difficult to play.
- Lossy compression- algorithms search through the code to find pieces that it can delete. While lossy compression can't be used on program files, it can be used on multimedia files, where there is often information in the files that human senses cannot detect. When lossy compression is used, the file may appear to be identical, but it is very different at the code level. The process is irreversible, once you convert to lossy, you can’t go back. And the more you compress it, the more degradation occurs. JPEGs and GIFs are both lossy image formats.
One of the biggest obvious benefits to using lossy compression is that it results in a significantly reduced file size (smaller than lossless compression method), but it also means there is quality loss.
Lossy Advantages and Disadvantages:
- Advantages: Very small file sizes and lots of tools, plugins, and software support it.
- Disadvantages: Quality degrades with higher ratio of compression. Can’t get original back after compressing.
- Lossless Compression- algorithms search for long strings of code and have a method to replace them with shorter strings. Lossless compression can recreate the entire file exactly as it was. Lossless compression refers to compression in which the image is reduced without any quality loss. Usually this is done by removing unnecessary metadata from JPEG and PNG files. RAW, BMP, GIF, and PNG are all lossless image formats. It is important to note that since JPEGs are a lossy format that when using the “maximum” preset in Photoshop, this doesn’t mean it is lossless. However, even images exported from Photoshop using the “Save for Web” function can still see significant reductions.
Lossless Advantages and Disadvantages
- Advantages: No loss of quality, slight decreases in image file sizes.
- Disadvantages: Larger files than if you were to use lossy compression
To compress a video, your file must also have a corresponding codec. A codec is a software that compresses your video, so it can be stored and played back. The most common codec includes h.264, which is often used for high-definition digital video and distribution of video content. It is also important to note the bit rate, which refers to the amount of data stored for each second of media that is played. The higher the bit rate, the less compression, which results in overall higher quality. However, be aware that the higher the bit rate, the larger the file size.
Containers (are file extensions)
A container is the file that contains your video and audio streams and any closed caption files as well. It’s common for a container to be called a file extension since they are often seen at the end of file names (e.g. filename.mp4) Popular video (visuals-only) containers include.mp4, .mov, or .avi, but there are many more. Audio uses its own codecs. Often your video camera will determine the container for your original video file as well. Our Canon DSLRs record .mov to the memory card, however our Canon camcorders can do AVCHD or MP4, which can be changed in the camera settings menu. Container is what we typically associate with the file format. Containers "contain" the various components of a video: the stream of images, the sound, and anything else. For example, you could have multiple soundtracks and subtitles included in a video file, if the container format allows it. Example of popular containers are OGG, Matroska, AVI, MPEG. It is what holds the grouping of compressed video as defined by the codec. A container is also referred to as a format. The container takes care of packaging, transport, and presentation. The container is usually represented by a file extension. For example, the MPEG4 container is usually represented by a .mp4 file type.
As the digital video landscape evolves, so do the types of codecs and containers. This evolution is usually marked by increased quality and lower file sizes. As screen size and resolutions increase, an evolution that makes containers and codecs more efficient is only natural.
Choosing a container for export (Export Settings)
When it’s time to export your video after editing, you’ll most likely be tasked with choosing a file type (container). Nine times out of ten when exporting a video for the web an MP4 will be your best bet. Occasionally you may need to use a different container depending on where you plan to host your video. If you’re creating a video for a client always check to see if they have any specific file type needs. If you’re unsure or are exporting to YouTube, Vimeo, Facebook or Instagram an MP4 will do just fine. Codecs are ways of "coding" and "decoding" streams. Their job is typically to compress data (and decompress it when playing it back) so that you can store and transmit files with a smaller file size. There are many codecs available out there, each with their strengths, weaknesses and individualities, and choosing the right codec with the right settings for the right situation is close to be a form of art.
Apple Quicktime MOV
The Apple Quicktime MOV container is a proprietary container that allows the user to store multiple tracks of audio, video, and text as well as metadata like timecode. Installing support for this format includes a number of “native” Quicktime audio and video codecs to be installed on a system as well, although the codecs themselves may not be the quickest or highest quality. It is a common format for acquisition, editing, and delivery and is a ubiquitous container format in the motion picture industry.
Microsoft AVI
For all intent and purpose, Microsoft AVI is considered a legacy container in the film industry. In its standard form, the container does not support files larger than 4gb, does not properly support timecode, and doesn’t properly support pixel aspect ratios. There are non-standard extensions to the container format, like OpenDML which overcomes the 4gb file limit, but support is dubious in professional software.
SMPTE MXF
MXF is a container developed as an industry standard by SMPTE. While intentions are good, there are still issues with the implementation of the format that make it difficult to take advantage of some of the progressive features offered by the container. MXF, like MOV, is common as an acquisition container (Panasonic P2, some implementations of Sony XDCAM), as an editing container on Avid NLE systems, and as a delivery format for broadcast and digital cinema.
MPEG MP4
MPEG’s MP4 container is an industry standardized format based on Apple Quicktime MOV with some additional extending features. It is commonly an acquisition and delivery format. Although it can contain different codecs, the most commonly used codecs in the MP4 container like h.264 aren’t totally suitable in an editing environment.
Uncompressed
Uncompressed video is the basis of all compressed video. There may be different levels of fidelity due to bit depth (8, 10, 12 and so on) or chroma subsampling (4:4:4, 4:2:2, 4:2:0, and so on). Uncompressed video was common in the days of SD video but is now less common as uncompressed HD video requires large and fast hard drives to work with efficiently.
DV
Its a digital video compression schemes. Common as a acquisition, editing, and delivery format. Originally designed for tape-based SD footage, a variation known as DVCPRO HD is used in some HD camcorders including a number of Panasonic P2 format camcorders.
HDCAM
Based on betacam compression, this tape-based compression scheme is designed by Sony. Common as a acquisition, editing, and delivery format.
MPEG2 part 2
Also known as h.262, MPEG2 part 2 is widespread in the film industry. Implemented with different bitrates, chroma subsampling, and resolutions based on cost and recording media, MPEG2 part 2 powers DVDs, the defunct HD DVD, Blu-ray, HDV, XDCAM, XF, and others. Common as an acquisition, editing, and deliver format. As an editing format it is usually carried over directly from acquisition as some delivery implementations are highly compressed and not suitable for editing.
H.264
H.264, also known as MPEG4 part 10 or AVC depending on what standardization group you ask, is another Swiss Army Knife codec like MPEG2 part 2. Implemented with different bitrates, chroma subsampling, and resolutions based on cost and recording media, you’ll see this compression scheme everywhere from camera formats like Canon dSLR MOV files, AVC-Intra cameras, and AVCHD, all the way to hardware-based decoders in your mobile phone. Common as an acquisition, editing, and deliver
format. Like MPEG2 part 2, as an editing format it is usually carried over directly from acquisition as some delivery implementations are highly compressed and not suitable for editing.
JPEG
An implementation of JPEG image compression. Videos are compressed frame by frame. A fairly ubiquitous, cross-platform compression scheme, it has been overshadowed by editing codecs like ProRes, DNxHD, and Cineform. A decent choice for intermediate storage, you may find JPEG compression heavy on CPU usage.
JPEG2000
Similar to, but the next step beyond, JPEG compression. JPEG2000 also supports lossless compression which lends itself as the choice of compression for digital cinema (as part of DCP standard MXF files) and digital archiving.
Apple ProRes
A codec designed specifically for editing. Introduced by Apple for use in Final Cut Pro, it has become a popular compression scheme in many areas of the post production pipeline. It’s designed to balance size, playback quality, and CPU usage. Because of it’s fairly light compression when using ProRes HQ or ProRes4444 it is commonly used when rendering video through multiple generations as it can stand some generation loss. Common as an editing and delivery format, less common as an acquisition format.
Apple Animation
A long standing compression scheme, Apple Animation is common for compressing animation and motion graphics and gained popularity as a codec that supported transparency (an alpha channel). It uses run-length compression meaning that it compresses video more efficiently the longer pixels stay the same from frame to frame. It does not compress live action footage very efficiently. Common as an intermediate codec for storing footage before it’s used in an edit and delivered.
PNG
PNG is an alternative to Apple Animation. A lossless codec that also supports transparency, it efficiently compresses large areas of the same color within single frames. It may be more efficient than Apple Animation in some cases, but requires higher CPU usage.
Avid DNxHD
Similar to Apple ProRes in terms of size and performance, DNxHD, also known as SMPTE VC-3, is Avid’s choice for an editing compression scheme. With codecs for Windows and Mac, as well as being deemed a SMPTE standard, DNxHD may be a more future-proof alternative to Apple ProRes. Common as an editing and delivery format, it is gaining speed as an acquisition format.
GoPro Cineform
Yet another editing compression scheme with similar performance to Apple ProRes and Avid DNxHD. It’s cross-platform and includes features and support tools specific for 3D film workflows.
RED REDCODE RAW - ARRIRAW
These formats are entirely proprietary. Data is stored as raw image data and there may or may not be proprietary compression applied. Generally, formats like these are strictly for acquisition and will require a more specialized workflow.
CinemaDNG
it’s an image format that stores information in a similar fashion to other RAW camera formats. It can be stored and played back as an image sequence or stored as a video in an MXF container.
TIFF, DPX, and OpenEXR
TIFF, DPX, and OpenEXR are interesting in that they are not containers or compression schemes but single image formats. Groups of consecutive image files that can be read back as video and are usually reserved for high-end workflows due to their hard disk requirements.
AUDIO
Audio is almost of no consequence in terms of compression schemes. Audio can be recorded at different sample rates and bit depths that dictate overall quality but due to its low storage and CPU needs in comparison to video, it’s usually stored uncompressed in professional workflows.
