# Diffusion Colorization: Conditioning Tradeoffs Beyond PSNR/SSIM

Emily Patterson · August 20, 2026

> Diffusion Colorization: Conditioning Tradeoffs Beyond PSNR/SSIM. Modern diffusion models struggle not with texture hallucination, but...

| Takeaway | Detail |
| --- | --- |
| GAN-based colorization prioritizes speed over iterative refinement | GCP-Colorization produces vivid colors using a single forward pass without iterative optimization |
| Spatial error distribution is controlled through chromaticity mapping | LEARCH trains a quadratic objective function on chromaticity maps while admitting correlations on long spatial scales |
| Multilayer feature extraction enables self-supervised representation learning | Learning Representations for Automatic Colorization processes grayscale inputs through a VGG-16 architecture to extract multilayer hypercolumns |
| Training data requirements dictate probability calculation per color bin | Prior probabilities must be calculated per color bin, requiring each bin to appear at least once in training data |

Modern diffusion models struggle not with texture hallucination, but with geometric-lighting disambiguation. When luminance gradients mask chromatic cues, automated systems falter regardless of architectural sophistication. The field has long relied on PSNR and SSIM as primary metrics, yet these measures obscure the fundamental tradeoffs inherent in conditioning pipelines.

Historical approaches reveal why current benchmarks fall short. Early frameworks like LEARCH optimized quadratic objectives on chromaticity maps, explicitly modeling long spatial scale correlations to distribute errors predictably. Meanwhile, hypercolumn architectures leveraged VGG-16 multilayer features to transform colorization into a self-supervised visual representation task, proving that structural understanding precedes accurate hue assignment.

Contemporary implementations continue to expose these limitations. GAN-driven pipelines achieve rapid results through single-pass feature modulation, yet they lack the iterative depth needed for high-contrast scenes. Attention mechanisms trained on restricted subsets further highlight how dataset composition dictates prior probability calculations. As models scale, the bottleneck remains consistent: disambiguating light from form requires conditioning strategies that transcend pixel-level similarity scores.

![Diffusion Colorization](https://static.mm-ais.com/article-images-ai/diffusion-colorization-conditioning-trad-ai-f6d33c92.jpg)

## Latent Ambiguity

The U-Net attention mechanism in 2026 architectures like StableDiffusion-XL-Light v2 fundamentally conflates albedo changes with shading variations when input grayscale images contain local contrast ratios exceeding 1:4. This architectural limitation forces cross-attention layers to misallocate color tokens directly into shadow regions, where the model incorrectly interprets low-luminance gradients as chromatic information rather than geometric occlusion. The mathematical root of this error lies in the gradient of the reconstruction loss with respect to the latent code z becoming severely ill-conditioned in low-luminance regions. As a result, the posterior distribution p(z|x_lighting) collapses, stripping the diffusion process of reliable conditional signals and leaving the decoder to hallucinate plausible but historically inaccurate hues.

This collapse manifests most visibly during the 'shadow recovery' phase of the denoising schedule. Between steps t=800 and t=600, the standard guidance scale defaults to 7.5, which aggressively over-amplifies spurious correlations between dark pixels and saturated colors that the model absorbed from synthetic training datasets. According to GitHub: w4995-dl-colorization, pretrained models were trained exclusively on a subset of ImageNet containing approximately 50,000 animal images, creating a biased prior where deep shadows frequently map to high-saturation environmental tones rather than neutral reflectance. When applied to archival photographs, this default guidance trajectory injects chromatic noise precisely where historical fidelity demands neutrality.

The missing component responsible for this degradation is the Illumination-Invariant Embedding. Standard pipelines lack a dedicated branch to disentangle reflectance R from illumination L, forcing the model to estimate L based on global image statistics instead of local surface geometry. Without explicit illumination maps injected before colorization, the architecture cannot isolate true material properties from lighting artifacts. The following matrix outlines how conditioning inputs alter the model's handling of high-contrast regions:

| Conditioning Input | Shadow Region Handling | PSNR Impact | SSIM Impact | Winner & Reason |
| --- | --- | --- | --- | --- |
| Raw Grayscale Pixels | Misallocates color tokens via cross-attention | >1.5 dB drop | >0.08 drop | Raw pixels lose; no geometric priors prevent token misallocation |
| Estimated Surface Normals | Disentangles R from L using local geometry | Baseline maintained | Baseline maintained | Surface normals win; explicit illumination priors stabilize p(z\|x_lighting) |
| Global Color Statistics | Guesses L via histogram matching | Unpredictable variance | Unpredictable variance | Global stats lose; fails to resolve local contrast >1:4 |

Archival restoration workflows must therefore bypass raw pixel conditioning entirely. Injecting estimated surface normals or shading priors into the diffusion pipeline before colorization prevents metric degradation in high-dynamic-range scenes. The model admits correlations on long spatial scales, enabling control over spatial error distribution across the image (ICCV 2015 / arXiv PDF), but this capability only activates when local illumination geometry is explicitly provided. Without it, the architecture remains trapped in latent ambiguity, prioritizing synthetic training biases over historical accuracy.

![Latent Ambiguity — Diffusion Colorization](https://static.mm-ais.com/article-images-ai/diffusion-colorization-conditioning-trad-ai-fb62794c.jpg)

## Benchmark Data

In 2025, the MIT-Historical Archive Benchmark released what remains the most systematic evaluation of diffusion colorization models under controlled lighting stress. The benchmark isolates a critical contradiction: while the state-of-the-art model, ColorFormer-7B, achieved a mean PSNR of 28.4 dB on flatly lit archival controls, the same model collapsed to 26.1 dB on images with dynamic ranges exceeding 5 stops. That 2.3 dB gap is not a modest regression; it is a direct quantification of the latent-space ambiguity this guide's thesis demands be solved. The clue here is not just the magnitude of the drop—it is that the model's performance is deceptively perfect on flat lighting, masking a failure mode that only emerges when the illumination control grows more complex.

The critical detail is that this 26.1 dB result is an upper bound. For contrast-heavy scans with specular highlights or deep shadow clipping, the effective signal-to-noise ratio of the output falls below the definition of archival fidelity by industry standards. The MIT benchmark's protocol uses a dynamic-range control set sourced from a single-shot raw converter with a 5.0 stop exposure bracket, ensuring the drop is not a byproduct of metadata errors or bit-decay in the source material. The score of 26.2 dB is the aggregate over 4,000 frames across 800 diverse historical photographs, meaning the drop is not an edge case triggered by a single pathological image—it is a systemic collapse in the encoding of shadow-region color assignments.

Corroborating structural degradation appears in the CVPR 2026 workshop on Generative Restoration. According to that workshop's proceedings, SSIM scores for historical photographs with heavy vignetting fell below 0.72, whereas well-lit subjects maintained SSIM >0.85. This indicates not just color error but a structural distortion, where the diffusion model is re-interpreting the vignette itself as a shadow-cast object and painting structural detail into the periphery of the image. This is significant because it bypasses the common assumption that failure modes manifest in hue alone. The structural similarity index loss in the darkened periphery suggests that generators are altering the actual geometry of the photo's grain and surface contours, a fatal error if the output is meant for film restoration.

Stanford Vision Lab's recent ablation study provides a different dimension of the art form. Their report demonstrates that removing the temporal consistency module from the diffusion pipeline increases PSNR variance by 40% in flickering light simulations. This is not about in-still single-image metric but about reliability across the sequential frames of the historical record from film stock with inconsistent exposure. The variance increase proves that each lighting change is reintroducing the shadow ambiquity and changing the model's output, which means the metric's stability is implicitly tied to the lighting stability. Without a temporal anchor, the model estimates a different scene relative to the same underlying surfaces across frames.

| Shadow Color Bias Measurements | Mean Displacement | Conclusion |
| --- | --- | --- |
| Classifier condition | +12% | Blue-channel over-saturation |
| Regression condition | +11% | Persistent hue shift |
| Control (flat light) | ±1% | Baseline drift |

In identifying the baseline of how severe this issue can be, the 2025 ICCV paper by Chen et al. introduces the 'Material Color Bias' as an attribute of the data scarcity. Chen's team excavates the bias with fewer assumptions: the +12% increase in blue-channel over-saturation in shadow regions is traced to all diffusion models trained on web-scraped datasets lacking photometric calibration. This confirms a that the shadow region's color direction is not arbitrary, but a learned preference of the data supply, and in the absence of explicit lighting priors, the model defaults to this photometric bias. It is not the presence of a shadow that causes the failure—it is the absence of in-conditions that the generator sees the shadow—as an opportunity to impose the dataset's prior.

These three benchmarks converge on a singular conclusion: the measured gap is not a a inherent weakness in U-Net encoders, but in a statistic that has become flatly in the architecture. The 2.1 dB drop; the SSIM fall below 0.72; and the +12% channel bias all spring from excluded the raw-illumination variable. When you inject surface normals or shading primari into the conditioning pipeline, the generator stop the treating the shadow regions as undetermined latent variables—it reconstructs them as contiguous, stable, light-bound regions, which is precisely the canonical decision rule this guide and the larger article prescribes.

![Benchmark Data — Diffusion Colorization](https://static.mm-ais.com/article-images-pixabay/diffusion-colorization-conditioning-trad-13153aff.jpg)

## Conditioning Strategy

Method C (Grayscale + Estimated Surface Normals) achieves a mean PSNR of 29.8 dB versus 26.1 dB for Method A (Raw Grayscale), confirming that explicit geometric priors collapse the latent ambiguity responsible for shadow-region color drift in high-contrast archival inputs. The 3.7 dB gain over baseline models on the 'High-Contrast Historical Set' establishes Normal-Guided Diffusion as the only viable architecture for professional restoration, where uncorrected lighting conditions otherwise trigger metric degradation exceeding the >1.5 dB PSNR and >0.08 SSIM thresholds defined by the 2026 evaluation protocol.

Injecting illumination maps via a pre-trained normal estimation network introduces a deterministic 14ms inference latency per image, yet this overhead yields a net efficiency gain by reducing SSIM error by 0.11 points, which eliminates the need for iterative post-processing correction cycles that typically consume disproportionate compute budgets in legacy pipelines. This trade-off is non-negotiable: raw pixel conditioning fails to disambiguate albedo from shading, forcing the diffusion model to hallucinate chromatic distributions in low-light regions, whereas surface normals provide the structural constraints necessary to anchor color predictions to physical geometry rather than statistical noise.

The 'Multi-Modal Fusion' approach, which attempts to inject RGB priors alongside grayscale inputs, demonstrates a critical failure mode in monochromatic archival sources due to domain shift; the injected chromatic signals conflict with the absence of ground-truth color information, resulting in no measurable PSNR improvement and often degrading fidelity through color bleeding artifacts. According to GCP-Colorization, pretrained Generative Adversarial Networks can extract rich, diverse color priors for automatic image colorization, but this mechanism relies on the presence of chromatic signal in the source material or external databases, rendering it ineffective when applied to pure black-and-white negatives where geometric structure must drive reconstruction. Geometric priors consistently outperform chromatic priors for black-and-white restoration because they resolve the lighting ambiguity at the source, whereas chromatic priors merely impose external assumptions that cannot be validated against the input data.

| Conditioning Method | Mean PSNR (dB) | SSIM Error Delta | Latency Overhead | Viability for High-Contrast Archives |
| --- | --- | --- | --- | --- |
| Method A: Raw Grayscale | 26.1 | Baseline | 0 ms | Fails; latent ambiguity causes >1.5 dB drop |
| Method B: Grayscale + Edge Map | 27.4 | -0.04 | ~2 ms | Insufficient; edges lack depth/shading cues |
| Method C: Grayscale + Surface Normals | 29.8 | -0.11 | +14 ms | Wins; resolves shadow ambiguity, +3.7 dB vs baseline |
| Multi-Modal Fusion (RGB Priors) | 26.0 | +0.02 | +8 ms | Fails; domain shift corrupts monochromatic sources |

The decision rule remains absolute: always inject explicit illumination maps into the diffusion conditioning pipeline before colorization to prevent metric degradation in high-dynamic-range scenes. While objective function coefficients conditioned on image features using a random forest classifier can optimize loss weighting, this technique addresses gradient scaling rather than the fundamental geometric deficiency that causes color errors in shadows; increasing parameter count or refining loss functions cannot compensate for missing illumination priors, debunking the myth that larger models inherently eliminate lighting-induced artifacts without architectural conditioning changes.

![Conditioning Strategy — Diffusion Colorization](https://static.mm-ais.com/article-images-pixabay/diffusion-colorization-conditioning-trad-b6749d77.jpg)

## What the Data Doesn't Tell You

Raw benchmark aggregates obscure the structural failure modes that actually dictate archival recovery rates. When we isolate high-contrast test sets, the reported mean metric drops mask a heavy-tailed distribution where a minority of frames absorb nearly all the degradation. According to the MIT-Historical Archive Benchmark's stratified splits, variance across cases is driven by local contrast ratios rather than global scene brightness. Frames containing specular highlights adjacent to deep occlusion shadows consistently exhibit wider confidence intervals, meaning the model's latent space does not uniformly collapse—it fractures along edges where albedo and shading gradients intersect. This means the aggregate PSNR and SSIM improvements from illumination priors are heavily weighted toward mid-tone regions, while edge-adjacent shadow pockets remain volatile.

The conditioning pipeline's reliance on estimated surface normals introduces its own failure boundary. When input material exhibits low geometric relief—matte fabrics, uniform plaster, or degraded emulsion—the normal estimation step produces flat or noisy gradients that offer no additional signal over raw luminance. In these scenarios, injecting an explicit illumination map can actually degrade performance by forcing the U-Net to reconcile contradictory geometric cues with missing chromatic data. The rule holds strictly when the source contains recoverable depth discontinuities; it becomes uncertain when the photograph lacks micro-contrast entirely. Practitioners should verify local gradient entropy before committing to a full lighting prior injection, as blind application on low-relief archives frequently amplifies banding artifacts in the mid-tones.

| Input Condition | Prior Injection Effect | Recommended Action |
| --- | --- | --- |
| High local contrast + clear occlusion boundaries | Metric stabilization (PSNR/SSIM gains align with thesis) | Inject estimated surface normals + shading prior |
| Low-relief matte surfaces / degraded emulsion | Gradient noise amplification / mid-tone banding | Skip geometric prior; use luminance-only conditioning |
| Specular highlights adjacent to deep shadows | Latent fracture at highlight-shadow transition | Apply localized exposure masking before diffusion step |
| Uniform mid-tone archival scans | Negligible metric shift (+/- 0.2 dB) | Standard baseline pipeline sufficient |

These edge cases do not invalidate the canonical decision rule; they define its operational envelope. The requirement for explicit illumination maps remains non-negotiable for high-dynamic-range restoration, but practitioners must treat the prior as a conditional tool rather than a universal constant. Scaling parameter count from 1B to 7B parameters does not resolve this ambiguity—the bottleneck resides in the attention mechanism's inability to disentangle reflectance from illumination without external geometric grounding. Verify your archive's local contrast profile first, then apply the conditioning strategy accordingly. Blindly scaling up model capacity or forcing priors onto flat textures will only compound the latent space confusion you are trying to eliminate.

![What the Data Doesn't Tell You — Diffusion Colorization](https://static.mm-ais.com/article-images-pixabay/diffusion-colorization-conditioning-trad-0d3cd019.jpg)

## What the Metrics Miss

High PSNR scores frequently mask archival degradation by rewarding models that replicate sensor noise rather than reconstructing albedo. In 2026 evaluations of diffusion colorization, automated quality filters flag outputs with PSNR >30 dB as superior, yet forensic analysis reveals these scores often stem from the model reproducing scan artifacts—such as dust, scratches, or ISO grain—from the source grayscale input instead of learning true object colors. This metric inflation misleads archivists into accepting historically inaccurate reconstructions. The mechanism occurs because standard PSNR penalizes pixel-wise deviation without distinguishing between meaningful chromatic error and benign high-frequency noise preservation; when a model overfits to the noise distribution of a damaged negative, it achieves low reconstruction error relative to the noisy ground truth while failing to recover the underlying scene geometry and color fidelity required for museum documentation.

The 'Distributional Shift' problem further decouples perceptual metrics from factual correctness, particularly when models trained on naturalistic lighting encounter studio portraits with artificial key lights. Models optimized for outdoor daylight distributions fail catastrophically on controlled studio setups, producing hallucinated color casts and structural artifacts that violate historical accuracy. Metrics like FID measure perceptual realism against a reference distribution but cannot detect factual errors; a generated portrait may score well on FID by appearing visually plausible within the training manifold while introducing anachronistic hues or incorrect shadow directions. This discrepancy arises because FID evaluates the global statistics of feature embeddings rather than verifying semantic consistency with the source image's illumination constraints, allowing artifacts to pass automated review despite being factually wrong.

Uncertainty in deep shadow regions persists even with explicit illumination priors, threatening reproducibility standards. In areas where luminance falls below 5% of peak white, diffusion models exhibit high entropy in latent space, leading to stochastic color choices that vary significantly across random seeds. For museum documentation requiring deterministic archival records, this variability is unacceptable; two runs with identical inputs and priors can yield divergent shadow colors, violating reproducibility requirements. The root cause lies in the insufficient signal-to-noise ratio in near-black regions, where illumination maps provide geometric guidance but lack chromatic information, forcing the model to sample from a broad posterior distribution rather than converging on a unique solution.

| Illumination Prior Complexity | SSIM Gain vs. Surface Normals Only | Risk Profile | Archival Viability |
| --- | --- | --- | --- |
| Surface Normals (Baseline) | Reference | Low warping risk | High |
| Normals + Key Light Direction |

Canonical: https://colorizethis.io/blog/diffusion-colorization-conditioning-tradeoffs-beyond-psnrssim.php
Markdown: https://colorizethis.io/blog/diffusion-colorization-conditioning-tradeoffs-beyond-psnrssim.php/index.md