Richard Fliam - Objectionable Uses of Objective Quality Metrics

Objective quality metrics like VMAF, PSNR, and SSIM are in broad use today. While many have well known weaknesses, metrics like VMAF are highly accurate. They are how most companies evaluate encoders, but what happens when they all game the system for their own ends?

We will explore the effect of Goodhart's law on objective quality. Some ways are well known (--tune psnr anyone); however, other more sophisticated methods exploit statistical properties of our averaging mechanisms, scene dynamics, and outright abuse of codec parameters to make objective quality metrics less objective.

Included will be a brief demonstration of a soon to be open source quality tool for massively distributed computation of quality metrics with a 30x speedup for VMAF and associated graphing utilities.

In detail these include:

1) How to make an encoder "tune" for PSNR and why that destroys image quality 2) How to make an encoder "tune" for SSIM and why that destroys image quality 3) How to make the mean or harmonic mean higher and make visual quality worse for any metric 4) How to abuse the tools most people use (ffmpeg) to produce flatly incorrect results

This talk will help users avoid falling prey to objectionable uses of objective quality metrics.

Presented at Demuxed 2019 in San Francisco.