Raw video is enormous
A camera sensor produces a huge amount of data. Consider uncompressed 1080p video: each frame has about two million pixels, and each pixel needs several bytes to describe its colour. At 30 frames per second that adds up to hundreds of megabits every second, far more than a typical home connection or storage drive can comfortably handle for long.
A video codec is the technology that closes that gap. The word is a blend of “coder” and “decoder”. It is a method, written down as a specification and implemented in software or hardware, for turning raw video into a much smaller stream of data and turning that stream back into pictures you can watch.
Encoding and decoding
The two halves of a codec do different jobs.
- The encoder analyses the source video and decides how to describe it in as few bits as possible. This is the expensive step. It compares frames, tests many options and chooses the cheapest acceptable description.
- The decoder reads the compressed stream and rebuilds the pictures. This is usually far cheaper than encoding, which is why a phone can play video that would take it a long time to produce.
An important detail is that most codec standards define only the format of the compressed stream and how a decoder must interpret it. They do not dictate exactly how an encoder should find the best description. That leaves room for competing encoders to produce different quality and speed from the same standard, and it explains why two files made with the “same codec” can look quite different at the same size.
What a codec actually does
Nearly all mainstream codecs rely on the same few ideas, which together make up lossy compression.
- Spatial prediction. Within one frame, neighbouring pixels are usually similar, so a block can be described as a prediction from nearby blocks plus a small correction.
- Temporal prediction. Between frames, most of the picture barely changes. The codec describes a block as “like that block in an earlier frame, moved slightly”, plus a correction.
- Transform and quantisation. The leftover corrections are converted into a form where less important detail can be reduced or discarded. This is the step that makes compression lossy.
- Entropy coding. Finally, the remaining numbers are packed using shorter codes for common values, a lossless step similar in spirit to zipping a file.
The result is usually a stream in which only some frames are stored in full, while others are stored as changes relative to their neighbours.
Codec, container and format are different things
People often use “codec” and “file format” as if they were the same. They are not. A codec compresses the pictures or the sound. A container, such as MP4 or MKV, is the wrapper that holds the compressed video, compressed audio, subtitles and metadata together and keeps them in sync. A file that ends in .mp4 can hold video made by several different codecs. The wrappers are known as container formats, and the difference matters in practice because a device may understand a container but not the codec inside it.
Why so many codecs exist
If one codec were best at everything, there would be no reason for others. Instead the landscape includes H.264, HEVC, VP9, AV1 and several more, for a mix of reasons.
- Efficiency keeps improving. Newer codecs generally achieve the same visual quality at a lower bitrate, but they usually need more computing power to encode and sometimes to decode.
- Licensing differs. Some codecs are covered by patents that must be licensed, while others are designed to be royalty-free. That affects which companies are willing to build on them, a theme that runs through the story of the AV1 codec.
- Different jobs need different tools. A codec for live video calls prioritises low delay, one for film archiving prioritises quality, and one for broadcast prioritises predictable behaviour.
- Devices have long lifespans. A codec is only useful if the viewer’s phone, TV or browser can decode it, so older codecs stay in use long after better ones appear.
- Competing interests. Standards bodies, technology companies and open consortia have each developed codecs, and they do not always agree.
Hardware support decides what is practical
Decoding can be done in software on a general-purpose processor, but many devices include dedicated hardware for the codecs they expect to play. Hardware decoding uses less power and keeps playback smooth, which matters on a phone battery. A new codec can be technically excellent yet slow to spread because the devices people own do not yet have that hardware. Encoding has a similar split between hardware encoding and software running on the CPU.
Settings that shape the result
Choosing a codec is only the first decision. The same codec can be used with very different settings:
- the video bitrate, which sets roughly how many bits each second of video may use
- the rate control mode, such as constant or variable bitrate
- the preset, which trades encoding speed against compression efficiency
- the profile and level, which limit which features and resolutions a decoder must support
A service that delivers video to many different devices often encodes the same source several times with different settings, so each viewer receives a version that suits their screen and connection. The resulting set of versions is often called an encoding ladder.
Choosing sensibly
For most people, the practical rule is to favour compatibility unless there is a reason not to. H.264 plays almost everywhere and is a safe default for sharing a file. Newer codecs can save bandwidth and storage, but they are worth using mainly when you know the audience’s devices can play them, or when a platform handles the conversion for you. When you are uncertain, it is reasonable to test a short clip on the devices that matter before committing to a whole project.