Best video formats for web in 2020
A frequent question when we run a video production and publishing pipeline for web is: which format is best?. From the well established H264 in mp4 containers, to the rapidly emerging AV1 or adaptive bitrate options like HLS, in this section we'll keep an up to date revision of video formats -codecs, containers, and protocols- that may and/or should be used in a web context. Based on this revision, we draw a recommendation with an online store -representative of a general web business- in mind.
Containers, codecs, and video streaming protocols
The file extension or file type tells us about the container being used. Each container has a file format specification that sets how the different types of data -video, audio, subtitles- and metadata are stored. We should not confound it with the coding standard. This is important because the standard followed to encode the video will determine if it is supported by the browser or the system.
For instance, videos in a mp4 container encoded with the H264 standard are universally supported. However, this is not the case of a mp4 file coded with the H265 standard as we'll see.
Moreover, adaptive bitrate (ABR) streaming protocols have brought an additional layer of complexity. To cope with the variability of network speed, these protocols split and encode the videos in multiple files with different bitrates and adapt the quality delivered to the bandwidth available.
Since there are quite many possible combinations of containers and coding and delivering standards, we'll focus here on the most relevant ones. We'll revise them in the light of support, compression efficiency, encoding speed, and user experience. These are factors that usually weigh on a decision about which formats to use.
Progressive video formats for web
H264/AVC encoding in mp4 container
In combination with an mp4 container H264 is the video format with universal support among browsers and systems. It enjoys hardware acceleration in any relevant device -computers, tablets, smartphones-. We may also find it with a m4v file type -specially if you use HandBrake-, which is a mp4 derivative with DRM protection.
While the advantages are universal support with easy and fast encoding and decoding, the main issue of this format is compression efficiency. For high resolutions -over VGA- a good visual quality may imply bitrates that exceed the capacity of many mobile networks and during peak traffic times. This slows down page speed and results in video stalls and rebufering. The alternative is sacrificing the quality, producing artifacts in the video like blur or mosquito.
H265/HEVC encoding in mp4 container
Still using an mp4 container, we may encode the video with H265/HVEC, a more recent and much more efficient standard. On average, it encodes a video with the same visual quality using half the bitrate.
There are however some significant drawbacks with H265. The first, a limited support, restricted to Apple devices and software. This limitation is a consequence of much pricier royalties that should be paid by manufacturers using the standard. Besides, its higher complexity means encoding gets heavier and longer, about one order of magnitud compared to H264. In general this means a higher consumption of computing resources and more latency in transcoding for the generation of video derivatives.
VP9 in webm container
Fortunately, VP9 encoding in webm containers (a derivative of mkv containers that restrict the codecs to use) brings an open source alternative, free of royalties. It presumes to be nearly as efficient as H265 but free of any licensing charge.
Originally supported only by Chrome, it's now also supported by Firefox and Edge. It has also native support in Android, Windows, and Linux. However, it's not supported by Apple devices (macOS, iOS) and software (Safari).
It shares with H265 the drawback of higher computational complexity and slower and heavier encoding.
AV1 is a new video coding specification by the Alliance for Open Media, a consortium of strong players in the media scene.
It is an open source specification free of royalties that builds on VP9, with mappings for both mp4 and webm and mkv containers. It promises further gains in compression efficiency, surpassing H265 and VP9.
The performance improvement together with the strong support behind AOM and the open source nature promise a sweeping adoption of the standard in the medium term.
Since the first version of the reference code was published in 2018, AV1 support has achieved significant progress in 2020. Encoding is also maturing quickly. However, the fastest implementations available (rav1e, SVT-AV1) are still in an early stage and encoding is very slow and resource hungry.
Streaming and adaptive bitrate protocols
HTML5 video streaming
Streaming video over HTTP is straightforward in HTML5. Like for images, we simply have to embed the link to the video in progressive format and it will work with any browser out there provided that it supports the coding standard used.
If we use H264 we can be confident that it will be properly played by any browser or any app in Android or iOS. We may offer H265 or VP9 versions of the video with H264 as fallback to optimize the use of bandwidth in those devices supporting the latest standards.
However, when used with progressive video an important limitation remains. The quality of the video is fixed regardless the quality of the connection. In mobile or saturated networks with modest speeds, high resolution videos will present a poor quality of experience, either because of a low visual quality or because of low page speed, video stalling, and rebuffering issues.
HLS video streaming
To overcome the issues of progressive video in the context of diverse connection speeds, we have to resort to adaptive bitrate options. HLS is an adaptive bitrate protocol (ABR) developed by Apple that uses a number of renditions of the video, each with a different bitrate. Besides, each rendition is split in a number of chunks in an mp4 container.
The big advantage of HLS is the capability to adapt the quality delivered to the speed of the connection, avoiding to bloat it. This in turn avoids slowing down page speed, and video stalling and rebufering. When combined with per-title encoding techniques that optimize the compression based on the content of the video, HLS optimises the quality of experience. That is, it provides the best possible quality that fits within a given network speed.
HLS videos can be encoded using H264 and H265 standards. In general, we recommend the use of H264 due to its much wider support.
MPEG-DASH video streaming
Which video format should we use
The use of H264 in mp4 containers is clearly the simplest and safest bet. It enjoys universal support, it's fast to encode and decode. However, it has issues with high resolution videos over low quality connections, damaging the quality of experience and increasing our CDN costs.
If we want to bring improvements we may keep it as fallback and offer more efficient encodings. AV1 is progressing rapidly and the strong support behind it suggests a dominant position in a close future. However, it's still in an early stage, it's too slow to encode for a general web business and it has yet a limited support. In sum, it's still costly and with very limited impact on average performance. Alternatively, we may offer VP9 for non Apple clients (Android, Chrome, Edge, Firefox) and H265 for iOS and Safari clients. This would improve the experience for a good amount of users. However, this option presents important issues. First, we increase the complexity, the cost, and the latency of transcoding. Second, we incur in a fragmentation of CDN cache that may backfire.
Regardless of the encoding used, progressive videos have a fixed quality and don't adapt to network conditions.
Our current recommendation with short -few seconds- videos is either to keep it simple and deliver H264 in mp4 containers or using HLS with per title encoding in H264 for ABR video streaming. While HLS increases complexity of the transcoding pipeline and the front ends, it brings breakthrough improvements of the quality of experience. For high resolution videos over 5s long we have seen clear benefits attached to HLS. Its combination with fast H264 encoding limits the cost and latency of the transcoding. HLS with per title encoding is definitely our format of choice for long videos.