Subjective quality assessments for modern codecs and objective metrics adequacy verification

MSU Subjective Codecs Comparison

Motivation

During last few years, many comparisons of video, audio and image codecs were held using various objective metrics like PSNR, VQM or SSIM. This fact has raised reasonable questions on adequacy of objective measures to the subjectively perceived quality, which is the main parameter of a codec's performance. Our research was targeted on methods of subjective video quality assessment and evaluation of objective quality metrics.

Approach

To conduct a subjective assessment, MSU Perceptual Video Quality tool was developed. It implements a number of subjective assessment methodologies and analysis methods, including newly developed method MSUCQE. In the December 2005 we held a subjective comparison of four video codecs (DivX 6.0, XviD, WMV, x264) on two bitrates (690 kbps and 1024 kbps) and four sequences. More than 50 experts took part in the assessment. After averaging, we obtained MOS (Mean Opinion Score) for each codec, sequence and bitrate.

MOS for Battle sequence

We measured a number of objective metrics (PSNR, VQM and SSIM). To evaluate their precision, their values were mapped on the scale of MOS (0-9) using a fitting function.

Full analysis of results of subjective comparison and precision of objective metrics can be found in comparison report:
1. Comparison page on compression.ru
2. “Subjective Comparison of Modern Video Codecs”, .pdf (852 kB) .zip (721 kB)