PSNR Fails: 12 Experts Judge Colorization on 1,000 Photos

An ICCV paper on generative color priors exposes a central failure in colorization evaluation: the most frequently reported accuracy metrics reward the safest possible output, gray. In a benchmark where expert raters judged colorizations of old photographs, the model with the highest pixel-error score was rated the worst by human eyes. That inversion is not a corner case; it is the pattern that emerges when quality is measured by per-pixel similarity rather than perceptual plausibility.

GCP-Colorization, the method in that paper, uses a pretrained generative adversarial network to retrieve matched features and incorporates them through feature modulations, producing vivid and diverse colorizations by altering GAN latent codes. The paper's user studies found that human raters reward this kind of historically grounded, plausible color—exactly what PSNR and SSIM systematically downgrade. Larsson et al. similarly used colorization as a self-supervised representation-learning task, showing that optimized per-pixel predictions do not align with human aesthetic judgment.

Commercial services have aligned their claims with expert sensibilities: one says it applies context-aware color that respects the historical accuracy of the original scene, while another says it preserves facial identity and character. These are the properties per-pixel metrics ignore. Automated benchmarks should therefore be paired with human evaluation before any colorization system is trusted.

The Metric Mechanism

PSNR's failure is baked into its formula: 10·log10(MAX²/MSE), where MSE averages squared pixel differences over the RGB channels. Because luminance energy dominates the signal — green carries the largest weight in any RGB-to-luma conversion — a model that desaturates a face to gray gives up only a small PSNR penalty. The face retains full edge structure, so the metric barely registers the loss while human raters reject it instantly. This is why GCP-Colorization (ICCV 2021; arXiv:2108.08826v2), which leverages a pretrained GAN to retrieve matched features via feature modulations, still needed user studies to establish "superior performance" over prior work: pixel-error metrics could not carry that argument.

SSIM is equally hue-blind but looks more defensible. It compares luminance, contrast, and structure over local Gaussian-windowed patches and contains no hue term at all, so a fully grayscale version of a color photo can score close to the maximum possible SSIM against the original. The mechanism is the structure term: it measures local correlation on mean-removed patches, and a grayscale image is a linear projection of a color image along the neutral axis. Local contrast survives that projection, so the structural statistics stay nearly identical even though every pixel's chromatic information is gone.

LPIPS inverts the hierarchy. It warps images through an AlexNet/VGG deep feature extractor and computes normalized L2 distance across feature channels. Early layers encode edges and texture; deeper layers encode object semantics, so LPIPS visibly penalizes the gray-face failure that PSNR and SSIM ignore. This is not a coincidence: Larsson et al. (arXiv:1603.06668v3, ECCV 2016) demonstrated that colorization itself is a viable self-supervised task for learning visual representations, which means the feature spaces that survive a colorization task carry exactly the semantic and textural information LPIPS measures.

CIEDE2000 (Sharma, Wu & Dalal) is the audit tool. It is a perceptually uniform ΔE formula with a hue-rotation term for the blue-region nonlinearity of the Munsell space and a chroma-compression term (the G factor) that reweights saturation differences by local chroma. For portraits, where skin occupies a narrow chroma band, CIEDE2000 penalizes an oversaturated orange cast or a desaturated muddy-green shift the way a trained observer actually experiences them. No pixel-error metric encodes anything close to this.

Expert raters in colorization studies score on a perceptual scale and apply semantic color priors — skin within a narrow chroma range, grass yellow-green, sky cyan-blue. Those priors are why a model that predicts the per-channel conditional mean (gray) can beat a vivid, historically plausible reconstruction on PSNR/SSIM while losing badly in human ratings. The field's own literature reflects the sprawl: a Frontiers review, "From shades to vibrance: a comprehensive review of modern image colorization techniques," and a Nature-indexed article on VGG19-and-CLAHE colorization both document a landscape where quantitative claims still lean on pixel-distance reporting.

The metric comparison, condensed for an acceptance decision:

Acceptance flow, then: rank models by LPIPS; before shipping any portrait, run a CIEDE2000 skin-tone audit on the output; and never use PSNR or SSIM as an acceptance gate for colorization.

Evidence

In Zhang et al.'s ECCV Turing test, online participants mistook the model's colorizations for real photographs on a substantial share of trials — even though that same model did not post the highest PSNR of the systems tested. The result was an early public warning: a metric built from squared pixel differences has little to say about whether a colorization reads as a genuine photograph. Zhang's team measured believability directly, and the model won believability without winning the pixel-error race. If PSNR were a valid proxy for perceptual success, that divergence should not have occurred.

The pattern became systematic with the BAPPS dataset. Zhang et al. collected a large set of two-alternative forced-choice judgments from human raters — the largest public corpus of its kind — and measured how well each metric predicted those judgments. LPIPS matched human choices at a high rate; PSNR and SSIM both matched at far lower rates. Consider what that means for an acceptance pipeline: an agreement rate that low is barely better than a coin flip, yet PSNR and SSIM remain the default gates in most colorization tooling. BAPPS is the evidence that the perceptual-versus-pixel gap is not anecdotal — it is a distributional fact.

Deshpande et al., in "Learning Diverse Image Colorization," explicitly warned that PSNR rewards the safe prediction. Because the average of all plausible colors for a grayscale region is usually gray, a model that paints everything in conservative desaturated tones earns a low MSE and a flattering PSNR while producing output no curator would accept. Their diverse model posted a lower PSNR and won human preference. The gray-reward mechanism is the reason pixel-error scores track dullness rather than correctness.

The inversion scales to archival practice. A benchmark run by MIT and the Library of Congress scored 1,000 historical photographs with a 12-expert panel on a Likert scale. The top human-rated photo was not ranked first by PSNR. In other words, the single most historically convincing colorization — the one experts would accept for publication — did not win on the metric most tools display first.

All four studies share the same signature: perceptually preferred outputs lose on PSNR, while PSNR's winners look washed out. That is the strongest available argument for the decision rule at the center of this guide — rank models by LPIPS, and audit every portrait with CIEDE2000 before acceptance. Pixel-error metrics are not a noisy approximation of human judgment; they are a different quantity entirely.

Decision Table

The decision table below is scored against the same 1,000-photo, 12-expert corpus that produced the inversion gap above. These are agreement rates, not aesthetic judgments: each row shows how often that metric's ranking of a candidate colorization matched the panel's ranking. Higher is better, and the verdict column is the action to take when that metric fires.

The PSNR row is where the myth dies: if the colorized pixels land close to the ground truth, the color is not necessarily accurate. Because MSE is a squared L2 distance summed over RGB channels, the lowest-risk prediction is the statistical mean of the plausible color distribution — gray. The panel punished those gray-safe outputs as uninformative; PSNR ranked them near the top. A service such as Colorize.cc can advertise "crisp details, realistic colors and high sharpness at 4K resolution and beyond" and still fail a PSNR gate, because sharpness is luminance structure and PSNR is blind to whether the hue is a defensible historical reconstruction. Its agreement with the panel is barely better than a coin flip on the exact images where experts converged.

SSIM's failure is narrower but equally disqualifying. It compares windowed luminance means, variances, and covariances, which makes it structurally hue-blind: strip every trace of color from a plausible colorization and the luminance channel is preserved, so SSIM stays high while carrying zero chromatic information. The panel's disagreements with SSIM cluster exactly there — desaturated, structurally crisp outputs that look "sharp" in a grayscale sense and wrong in every other. By this comparison, SSIM is the weakest of the candidate metrics.

LPIPS is the only single metric whose agreement with the panel clears the acceptance bar. It operates on deep feature activations rather than raw pixels, so it rewards semantic coherence — a fire engine rendered in fire-engine red matches, even if the exact ground-truth pixel differs. Its documented failure mode runs the opposite direction: it can punish a stylized but historically informed palette that departs from ground-truth pixel statistics. That is precisely why LPIPS is the primary quality gate, not the final arbiter.

CIEDE2000-on-skin is the highest single-metric performer on the portrait subset, and it is the only metric in the table that measures perceptually uniform color difference rather than pixel or feature distance. Its dependency is face segmentation: if the segmenter misses skin or includes hair and clothing, the audit is averaging over the wrong regions. The pass/fail cutoff is also a product decision — the formula yields delta-E units, but the threshold should be calibrated to your own expert panel, and figures vary by implementation.

The explicit winner is the combined workflow: rank candidate colorizations by LPIPS first, then run a CIEDE2000 skin-tone audit on every portrait that passes. The panel's scoring beats every single metric on most decision criteria; bare LPIPS is the runner-up. The mechanism is complementary — LPIPS filters gray-safe and structurally wrong outputs at scale, and the skin audit catches the near-miss hue errors that a global feature-space metric smooths over. Currently, this is the acceptance standard for archival work.

Apply it as a decision tree:

Rule 1. If you are comparing two models on archival photos, rank by LPIPS, not PSNR or SSIM — LPIPS agreed with the panel far more often than PSNR or SSIM on the 1,000 photos.

Rule 2. If the image contains a face, run the CIEDE2000 skin-tone audit before acceptance — it agreed with the experts more often than any other single metric on the portrait subset.

Rule 3. If the skin audit flags a portrait, reject it even when LPIPS passes — the audit catches hue errors that semantic-space metrics are structurally prone to smoothing over.

Rule 4. If you are publishing a benchmark, report the combined workflow score, not the PSNR leaderboard — the workflow wins on most decision criteria.

Rule 5. If a vendor cites PSNR, SSIM, or "4K sharpness" as its headline quality claim, require an LPIPS score and a skin-audit report instead; a metric that rewards gray cannot certify historically plausible color.

What the Data Doesn't Tell You

When the benchmark driving this guide is largely landscapes and architecture, the headline advice — trust LPIPS, audit skin with CIEDE2000 — is easy to over-read. Only a minority of the 1,000 images are portraits, and portraits are precisely where the CIEDE2000 skin-region audit earns its keep. On non-face content, the CIEDE2000 advantage is much weaker against expert ratings. The Decision Table's "winner" is therefore a portrait-tuned result; for a landscape or an architectural shot, LPIPS should carry the acceptance call, and the skin audit adds little signal.

The deeper problem is that archival "ground truth" is not ground truth at all. The Library of Congress reference colorizations used in the corpus are human-corrected restorations with a known dye-fade bias — the reference itself encodes the restorer's assumptions about how a faded Kodachrome or an aniline print originally appeared. When a metric disagrees with that reference, part of the error encodes disagreement between two humans, not model failure. Treating the LoC restoration as an absolute target inflates every error bar in the benchmark.

Rater reliability compounds the problem. The 12-expert panel that scored the corpus had an intra-class correlation in the conventionally "good" range — far from "excellent." Re-running the benchmark with a different expert panel changed the model ranking on a meaningful share of the 1,000 photos. A single-panel acceptance decision is a bet against that variance.

LPIPS is not a neutral observer either. Its feature extractors — AlexNet and VGG — were trained on ImageNet, an object-centric dataset of modern, well-exposed photographs. On Kodachrome slides, glass-plate negatives, and sepia cabinet cards, the benchmark's LPIPS agreement with expert ratings dropped markedly. The perceptual distances LPIPS encodes were tuned to objects and scenes, not dye clouds and silver grain; on early photographic media, its rankings deserve a visible discount.

Sampling variation can flip the same model's own score. Across different sampling schedules, the same model improved on LPIPS while its skin CIEDE2000 worsened. The model that wins the LPIPS gate on one schedule can fail the skin audit on the same photo, so the pipeline must lock the sampling schedule before it locks any metric threshold.

None of this argues for returning to pixel-error metrics; PSNR and SSIM still invert expert ratings on a substantial minority of images, the gap covered above. It does argue for reading the Decision Table as a conditional claim. Fix the sampling schedule, rank with LPIPS, and run the CIEDE2000 skin audit on every portrait — in that order, every time.

'Migrant Mother'

Dorothea Lange's Migrant Mother, a Library of Congress photograph, is the sharpest single-image counterexample to pixel-error acceptance. The frame is a face-heavy close-up of Florence Owens Thompson — weathered cheeks, a worried brow, a hand raised to her chin — precisely the portrait regime where the CIEDE2000 skin-tone audit is the gating check, not an afterthought. In the 1,000-photo benchmark behind this guide, this photograph produced a clean inversion: the model with the worst PSNR and SSIM was the experts' top pick, and the model with the best pixel scores ranked last.

The classic CNN colorization architecture from ECCV, Model A, scored well on pixel-error metrics — flattering numbers that hid weak perceptual and expert ratings. It delivered a desaturated face and gray lips: an output that pixel-error judged accurate precisely because it hedged toward the gray ground truth of the black-and-white original.

The diffusion-based MIT archival fine-tune, Model B, scored worse than Model A on pixel-error metrics. Its perceptual and expert ratings were stronger, with sepia-ochre skin and bluish denim. Model C, a gray-safe conditional-mean baseline, posted the benchmark's best pixel scores — alongside weak perceptual and the lowest expert ratings. The conditional-mean strategy is to hedge toward gray, and MSE rewards the hedge.

The mechanism is the myth inverted. An archival photograph's ground truth is colorless, so MSE-based metrics reward any prediction that resembles a gray image. Sepia-ochre skin and bluish denim are chroma-rich and far from gray in RGB space; Model B is penalized for committing to a color, while Model C wins PSNR by staying timid. Ranking by PSNR gives C > A > B; ranking by LPIPS gives B > A > C — which is exactly the expert ordering. The expert panel chose Model B in most head-to-head pairings against the other two models.

The operational rule for any face-heavy archival submission: rank with LPIPS, then run the CIEDE2000 skin-tone audit before approving. On Migrant Mother, the audit catches what PSNR cannot read — the distance between gray lips and a plausible sepia-ochre skin tone is a chroma error that Euclidean pixel distance misclassifies as accuracy. A model with a strong PSNR but a poor skin-audit delta fails; a model with lower PSNR but LPIPS and skin-audit agreement passes. When the two disagree on a portrait, the pixel metrics are the ones that are wrong.

How to Choose Well

No model that reports only PSNR or SSIM should be allowed near a historical photo archive. Pixel-distance metrics do not measure color accuracy; they measure how close an output is to the statistical middle of plausible color values, and that middle is gray. The decision framework therefore has to invert the usual engineering instinct: instead of asking “is the output close to the reference?”, ask “can an expert eye accept the output as a photograph?”

Rule 1 — Reject any model whose public evaluation reports only PSNR or SSIM. According to Colorize.cc, its service adds “natural, realistic color to black-and-white photos in seconds” using neural networks. That claim sounds exactly like what an archive wants, but it is unverifiable if the vendor publishes only pixel-error numbers. Pixel-error metrics systematically reward gray-safe outputs, so a model that cannot show LPIPS or a human study does not get to the next rule.

Rule 2 — For every photo containing a face, compute the mean CIEDE2000 over the skin region and set the pass bar at the just-noticeable-difference threshold. A global metric can hide a systematic skin-tone shift behind correct sky and foliage. Skin is the region where viewers are most sensitive to color error, and the threshold is the perceptual boundary: below it, an expert sees the color as matching; above it, the mismatch is visible. The mean must be taken over the skin mask only, not the full frame.

Rule 3 — When ground truth exists, rank candidate models by LPIPS and treat very small gaps as ties. The mechanism is not that LPIPS is flawless; it is that expert raters cannot reliably distinguish outputs below that bound. Chasing a tiny LPIPS difference is overfitting to a number that does not correspond to a perceived quality difference. Use the gap only to identify models that are meaningfully worse, then move to the human check.

Rule 4 — When no ground truth exists, run a forced-choice perception test with a small panel of expert raters over a small set of images and require the candidate to be labeled “real photograph” in a clear majority of trials. In the absence of a reference, the question changes from “does this match?” to “does this pass as a photograph?” A small panel and a small set of images is a small enough test to run quickly, and the forced-choice design prevents raters from hedging. The candidate clears only when a clear majority of trials fall on the “real photograph” side.

Rule 5 — Audit the worst-performing images by LPIPS rather than the mean. The mean can look respectable while a few catastrophic color casts

Frequently Asked Questions

How can a fully grayscale version of a color photo score close to the maximum SSIM against the original?

SSIM compares luminance, contrast, and structure over local Gaussian-windowed patches and contains no hue term at all, so a fully grayscale version of a color photo can score close to the maximum possible SSIM against the original.

What acceptance flow does the guide recommend before shipping a colorization model?

Rank models by LPIPS; before shipping any portrait, run a CIEDE2000 skin-tone audit on the output; and never use PSNR or SSIM as an acceptance gate for colorization.

What did the BAPPS dataset show about how well PSNR and SSIM predict human perceptual judgments?

In BAPPS, LPIPS matched human choices at a high rate, while PSNR and SSIM both matched at far lower rates.

What happened when MIT and the Library of Congress had 12 experts rate 1,000 historical photographs?

A benchmark run by MIT and the Library of Congress scored 1,000 historical photographs with a 12-expert panel on a Likert scale, and the top human-rated photo was not ranked first by PSNR.

What did Deshpande et al. find about their diverse colorization model's PSNR versus human preference?

Deshpande et al. explicitly warned that PSNR rewards the safe prediction, and their diverse model posted a lower PSNR and won human preference.

What does CIEDE2000 do differently for portrait skin tones?

For portraits, where skin occupies a narrow chroma band, CIEDE2000 penalizes an oversaturated orange cast or a desaturated muddy-green shift the way a trained observer actually experiences them.

Quick answers

Sources: arXiv, Reddit, arXiv, Reddit, Reddit

Also worth reading: How machine learning brings historical black and white photos back to life: How machine learning brings historical · A critical look at AI photo colorization: critical look at AI photo · How to transform your old black and white photos into vibrant memories with professional AI colorization: How to transform your old