Two kinds of repetition
Video contains a great deal of repetition. Compression works by finding that repetition and describing it once instead of many times. There are two main kinds.
Spatial redundancy is similarity within a single frame. A blue sky, a white wall or a blurred background contains thousands of neighbouring pixels that are nearly identical. Writing out every pixel separately wastes data.
Temporal redundancy is similarity between frames. At 24 or 30 frames per second, consecutive frames are usually very alike: a person talks against a fixed background, a camera pans slowly, a table stays where it is. Only a small part of the picture changes from one frame to the next.
Removing both kinds of redundancy without throwing away information would be lossless compression. Video compression normally goes further and also discards information, which is where lossy compression comes in.
Compressing within a frame
To exploit spatial redundancy, the encoder divides each frame into blocks. For each block it forms a prediction from already-coded neighbouring pixels, for example “this block looks like a continuation of the edge to its left”. It then stores only the difference between the prediction and the actual block, called the residual. When the prediction is good, the residual is small and cheap to store.
A frame coded this way, with no reference to other frames, is often called an intra-coded frame or I-frame.
Compressing between frames
To exploit temporal redundancy, the encoder searches earlier frames (and sometimes later ones) for a patch that matches the current block. It then stores a motion vector, which says “take the block from over there”, along with a residual for any remaining differences. This is called motion compensation.
Frames coded this way are often called P-frames if they refer only to earlier frames, and B-frames if they may refer to both earlier and later ones. Referring to a later frame is possible because video files are not stored in playback order alone; the encoder reorders frames so the decoder has the references it needs.
A sequence of frames is typically organised in a group of pictures, starting with a fully coded I-frame followed by predicted frames. The spacing of I-frames is a trade-off: more of them make seeking and recovery from errors faster but cost more bits.
Transform, quantisation and what is thrown away
Residuals are not stored raw. They are passed through a mathematical transform, commonly a variant of the discrete cosine transform, that converts blocks of pixel differences into frequency components. Low frequencies describe broad, smooth variations, and high frequencies describe fine detail and sharp edges.
This is useful because human vision is generally less sensitive to very fine detail, particularly when it is faint or moving. The next step, quantisation, exploits that by rounding the frequency values coarsely. Many small high-frequency values become zero. This is the main lossy step: once rounded, the original values cannot be recovered. The strength of quantisation is the main control over how much quality is traded for size, and it is closely linked to the video bitrate.
Other things that are commonly reduced or discarded include:
- Colour resolution. Most video stores brightness at full resolution and colour at a lower resolution (chroma subsampling), because the eye resolves colour detail less sharply than brightness detail.
- Imperceptible detail. Faint noise, fine texture and subtle gradients that sit below what most viewers would notice at normal viewing distances.
- Exact fidelity. The decoded picture is an approximation, not an exact copy.
Packing the leftovers
After quantisation, many values are zero or small. Entropy coding stores them with short codes for common values and longer codes for rare ones, much as a zip file shrinks text. This stage is lossless: the decoder reconstructs exactly the numbers the encoder produced.
Where it goes wrong
Compression works best when the content is predictable and degrades when it is not. Typical problems include:
- Blocking. Visible squares appear in flat areas or at low bitrates because blocks are quantised independently.
- Banding. Smooth gradients, such as sunsets or dark scenes, break into steps because the subtle variations are rounded away.
- Mosquito noise and ringing. Shimmering artefacts appear around sharp edges.
- Smearing in motion. Fast, unpredictable movement, confetti, water or crowds consume large numbers of bits, and if the budget is too small, detail turns to mush.
Since the amount of repetition varies from scene to scene, the encoder’s rate control matters, since a variable bitrate can give hard scenes more bits than easy ones.
Why newer codecs do better
Successive codecs refine these same ideas rather than replacing them. They use more flexible block sizes, more prediction directions, better motion modelling, and improved filters to hide compression artefacts. That is why HEVC and AV1 can usually produce a given quality at a lower bitrate than H.264, at the cost of far more computation to search for good predictions. The overall framework, though, remains prediction plus transform plus quantisation plus entropy coding.
Lossless and visually lossless
True lossless video compression exists and reproduces every pixel exactly, but it produces files that are many times larger than lossy versions, so it is used mainly in production and archiving. Terms such as “visually lossless” describe lossy settings that are conservative enough that most viewers cannot spot differences under normal conditions. Whether a difference is noticeable depends on the viewer, the screen and the content, so the term is a judgement rather than a measurement.
Why this matters for creators
Understanding the mechanics explains several everyday observations: why a still scene makes a small file and a busy one makes a large file, why repeatedly re-saving a video makes it worse, and why a very dark or noisy shot suffers on streaming platforms. It also suggests sensible habits, such as keeping a high-quality master and compressing only once for each destination.