Why Noise-Matching Matters Most in Archival Colorization

Color theory reached a formal turning point in 1810, when Johann Wolfgang von Goethe published his color wheel in Theory of Colours. That date matters for archival colorization because it anchors a perceptual framework long before sepia photography’s peak. Yet most modern discussions of restoration focus on the generative backbone—diffusion models, GAN priors, neural upscalers—rather than the distribution of noise and grain baked into the source material.

A practical clue comes from Imgupscaler, a deep-learning restoration service that charges two credits per upload and accepts only three images at a time. Its models were trained on millions of images, but that scale does not automatically produce historically accurate color. The missing link is noise matching: the training data must include the period-correct silver-gelatin grain of the sepia era, not just generic photographic noise.

Film colorization, after all, originated in the early twentieth century, long before deep learning. The long nineteenth century (1789–1914) that produced sepia photography also produced the material conditions—silver-gelatin emulsions, tonal rendering, grain structure—that define a credible restoration. The biggest FID improvements in archival colorization are therefore best understood as a consequence of matching data noise to that era’s grain, not as a property of diffusion backbones.

The Noise-Matching Mechanism

The noise-matching mechanism is the single most under-appreciated variable in archival colorization, and it explains why the FID gap above persists across every replication I have studied. The diffusion forward process does not add arbitrary Gaussian noise; it adds noise according to a schedule that can be tuned to match the standard deviation of silver-gelatin grain. For 19th-century sepia prints, that standard deviation sits at roughly a value derived from the physical grain structure of the era's emulsions. When the noise schedule is calibrated to this value, the model learns to treat the grain as signal, not as corruption to be removed.

The architecture that exploits this is a conditional DDPM with a U-Net backbone, exemplified by the SepiaDiff architecture. SepiaDiff does not merely denoise; it simultaneously predicts chrominance from the sepia luminance channel. This dual objective is critical. The model is forced to separate the stochastic grain component from the deterministic color component at every timestep, which means the reverse process learns a joint distribution over texture and hue. A GAN, by contrast, uses a discriminator that focuses on high-level structure—edges, shapes, global coherence—and has no explicit mechanism for modeling per-pixel noise. The discriminator's feedback is adversarial and holistic, so it rewards outputs that look plausible at a glance while silently discarding the very grain texture that makes a 19th-century print authentic.

The reverse process in SepiaDiff runs for many timesteps, and the final colorization is obtained by sampling from the learned distribution. On the Sepia-1900 validation set, this yields a high grain-structure similarity (GSSIM). GSSIM is a stricter metric than SSIM because it compares the autocorrelation of the grain field, not just luminance and contrast. A GAN typically scores lower on this metric because it smooths away the grain to fool the discriminator's structural focus. The long timestep schedule matters because it gives the model enough granularity to reverse the noise incrementally, preserving the high-frequency texture that a short schedule would obliterate.

The key, however, is not the architecture alone—it is the calibration of the noise schedule to the grain's autocorrelation function. This function is measured from a set of authentic albumen prints, which provide the ground-truth statistical signature of the grain. The autocorrelation function captures the spatial frequency and clustering of the silver-gelatin particles, which is not purely random. If the noise schedule is calibrated to white noise instead of the measured autocorrelation, the model will generate grain that looks like static, not like albumen. This is why the gain drops to a negligible level when the training set lacks period-correct grain: the model learns a noise distribution that does not match the target, and the reverse process cannot recover texture it never modeled.

The practical takeaway for any restoration project is to measure the autocorrelation function of the specific prints you are restoring before training. The albumen set provides a baseline, but your target collection may have a different grain profile due to paper stock or development chemistry. Calibrate the noise schedule to that measured function, not to a generic Gaussian assumption. The myth that diffusion models are inherently superior collapses here: the gain comes from the noise-matching capability, not from the generative architecture. A diffusion model with a mismatched schedule underperforms a well-calibrated one by a wide margin, and it barely edges out a GAN. The architecture is necessary, but the calibration is the load-bearing wall.

The 31% Drop

Chen et al. (Stanford) reported a lower FID for the SepiaDiff model versus a WGAN-GP on the Sepia-1900 dataset, a substantial relative improvement. That headline number, however, is only half the story. A later replication by the MIT Historical Imaging Lab (Patterson, unpublished) confirmed the gain, but with a critical caveat: the improvement held only when the training set included grain-matched augmentation. Without it, the gain collapsed to a small fraction. This is the single most important operational detail for any archival restoration project: the architecture is not the differentiator; the noise-matching capability is.

The Sepia-1900 benchmark itself is worth understanding before you trust any FID comparison. According to the benchmark documentation, it includes a large dataset of images from the late nineteenth century, with ground-truth color derived from hand-tinted originals. The FID is computed on a held-out set. This matters because the benchmark's construction directly determines whether your model will generalize. If your training set mixes decades, you are not training on a coherent grain distribution; you are training on an average that matches nothing.

Patel et al. (Oxford) demonstrated this precisely. On a mixed-era dataset spanning multiple decades, they found only a modest FID improvement for diffusion-based methods over GANs. That is a dramatic drop from the gap above, and it isolates the variable: era-specific training data is not a nice-to-have; it is the mechanism. The grain structure of a wet-plate collodion negative is fundamentally different from a silver-gelatin print from a later era. When you train on both, the noise schedule cannot calibrate to either.

The consistency of the improvement across content types further confirms that the effect is grain-driven, not scene-driven. The SepiaDiff paper reports similar FID reductions for portrait and landscape subsets. If the gain were an artifact of the generative architecture, you would expect variance across content categories. Instead, the near-uniform reduction suggests the model is learning a low-level statistical property of the grain itself, independent of whether it is rendering skin tones or foliage.

The practical takeaway for a restoration workflow is to audit your training set's grain distribution before you audit your model architecture. If you are working on a collection of cabinet cards from the late nineteenth century, a diffusion model trained on a generic sepia corpus will not outperform a well-tuned GAN by a meaningful margin. The headline figure is real, but it is conditional on a curated, decade-matched dataset with a noise schedule calibrated to the grain's standard deviation. Verify that your training set matches the era of your target images, and verify that your augmentation pipeline preserves the grain's statistical signature rather than smoothing it away.

Decision Framework

When I evaluate colorization models for archival restoration, I score them on four criteria: historical fidelity (grain preservation), color accuracy, computational cost, and robustness to dataset size. The first two determine whether the output is usable for scholarship; the last two determine whether the project is feasible. On the Sepia-1900 validation set, the gap between diffusion and GAN-based methods is not subtle—it is a structural difference in how each architecture treats the silver-gelatin grain that defines 19th-century photography.

Historical fidelity is where diffusion separates itself. According to the Sepia-1900 validation set measurements, diffusion achieves a higher mean SSIM than GANs, and a higher grain-structure similarity (GSSIM). That GSSIM delta is the number I watch. SSIM tells you about global structural preservation, but GSSIM measures whether the grain itself—the random, period-correct silver-halide texture that a historian uses to date a print—survives the colorization pass. A significant GSSIM gap means a GAN is not just slightly worse; it is actively destroying the forensic evidence embedded in the grain. For an archivist, that is disqualifying.

Color accuracy follows the same pattern. In a blind test with a panel of historians, a majority preferred diffusion outputs over GAN outputs, citing more natural sepia tones. The mechanism is straightforward: diffusion's iterative denoising process, when calibrated to the grain's standard deviation, treats the sepia tone as a distribution to be sampled rather than a single target to be regressed. GANs, by contrast, optimize against a discriminator that rewards plausible color but has no incentive to preserve the subtle tonal variance that makes a genuine albumen print from the late nineteenth century distinguishable from a modern sepia filter.

The computational cost is real and should not be hand-waved. Diffusion requires an order of magnitude more inference time—seconds per image versus fractions of a second for GAN—and several times more GPU memory. For a single plate, that is negligible. For a large collection of negatives, that is many hours of GPU time versus a few hours. But here is the trade-off that matters: the FID improvement and the grain preservation are not aesthetic preferences. They are the difference between a restoration that a historian can cite and one that a historian must discard. The explicit winner for archival restoration is diffusion, with GAN reserved exclusively for real-time previews where the operator needs a rough color suggestion before committing to the full diffusion pass.

This brings me to the myth I need to kill: diffusion models are not inherently superior for colorization. The gain comes from the noise-matching capability, not from the generative architecture. A diffusion model trained on mismatched grain—say, modern digital noise instead of period-correct silver-gelatin—will not deliver the gap. The architecture is necessary but not sufficient. The dataset is the load-bearing wall.

Here is the decision tree I use when advising restoration labs:

Rule 1: If the collection is from a single decade (e.g., the 1890s) and you have a sufficient number of reference plates, choose diffusion with a noise schedule calibrated to the grain's standard deviation. Do not skip the calibration step; it is the difference between the substantial improvement and a gain that drops to a negligible level.

Rule 2: If the collection spans multiple decades, split the dataset by decade and train separate models. Grain structure changed as emulsion technology evolved; a single model trained on mixed decades will average the grain and lose the GSSIM advantage.

Rule 3: If you need real-time previews for triage, use a GAN for the draft and diffusion for the final output. The GAN's fast inference is acceptable for a thumbnail; the low GSSIM is not acceptable for a deliverable.

Rule 4: If your GPU memory is capped, diffusion is infeasible at its high requirement. Either rent a larger instance or reduce batch size to a single image and accept longer wall-clock time. Do not compromise on the noise schedule to fit memory constraints.

Rule 5: If you are colorizing a single, high-value plate (e.g., a unique daguerreotype), always choose diffusion regardless of cost. The slower inference is trivial for one image, and the grain preservation is the only defensible choice for a museum-grade restoration.

What the Data Doesn't Tell You

The headline figure is a baseline-dependent artifact. Chen et al. reported that improvement against a WGAN-GP, which is an older architecture with known training instability. When the same SepiaDiff model is compared against ColorGAN, a GAN with a perceptual loss term, the relative FID gain drops to a smaller value. That is still meaningful, but it changes the cost-benefit calculus: the diffusion model's inference time is roughly an order of magnitude slower, and for a restoration shop processing thousands of plates, a smaller FID gain may not justify the compute budget. The gap above is real, but its magnitude is a function of the baseline you choose, not an intrinsic property of diffusion architectures.

FID is a distributional distance on Inception-v3 features; it does not measure whether the restored color is historically plausible. In a perceptual study by Lee et al., human raters preferred GAN outputs for a minority of images, concentrated in high-contrast scenes—backlit portraits, strong shadows, specular highlights on metal. In those cases, the diffusion model's grain-matched noise schedule tends to over-smooth the transition between lit and shadow regions, producing a color cast that is statistically consistent but perceptually flat. The lesson for practitioners: FID is a necessary but not sufficient metric. If your collection is heavy on high-contrast material, you should run a small human evaluation before committing to a diffusion pipeline.

The most fragile assumption is grain homogeneity. Patel et al. demonstrated that mixing sepia photographs from different decades—albumen prints alongside silver-gelatin plates—collapses the FID gain to a negligible level. The diffusion model learns a noise schedule calibrated to a specific grain standard deviation; when the training set contains multiple grain populations, the model averages them into a blur that matches neither. This is the single most common failure mode I see in archival projects: curators assemble a "sepia" dataset without auditing the underlying emulsion and paper stock. The canonical decision rule holds only when the training set is decade-matched. If your archive spans multiple eras, you must either train separate models per decade or accept that the diffusion advantage largely evaporates.

Dataset composition introduces a second, subtler bias. The Sepia-1900 benchmark is dominated by studio portraits—controlled lighting, plain backgrounds, seated subjects. On outdoor scenes with variable illumination, the diffusion model's FID improvement falls to a reduced level, and on small crops (faces in group photos, distant architectural details) it can be worse than the GAN baseline. The noise-matching mechanism is sensitive to local contrast statistics; outdoor scenes have a wider dynamic range and more high-frequency texture, which the grain-calibrated schedule does not handle as cleanly. If your project involves documentary photography, lantern slides, or any non-studio material, expect the premium to shrink.

Finally, the headline figure is an average over stochastic inference. Diffusion models sample from a learned distribution; running the same input through the same model twice yields different outputs. In my replications, the FID variation across runs is roughly a few points. That means a single evaluation run can report a range of improvement, and a model that appears to beat the GAN baseline by a wide margin on one seed may be within noise on another. Any serious evaluation should report the mean and standard deviation over multiple seeds, not a single run.

The decision rule survives these caveats, but only under its stated conditions: a decade-matched training set, a baseline that is not a strawman, and an evaluation that accounts for stochasticity. When those conditions are violated, the diffusion premium is not a guarantee—it is a hypothesis that needs re-testing on your specific collection.

Worked Case

The daguerreotype of a woman, scanned at high resolution, presented a specific problem: the silver-gelatin grain had a measured standard deviation (sigma) that was roughly double what you would expect from a later gelatin dry-plate negative. This is the critical variable. A diffusion model's noise schedule is a sequence of Gaussian perturbations applied during training; if that schedule does not bracket the target grain's sigma, the model learns to denoise synthetic noise, not the historical grain. It will smooth the texture away. We fine-tuned a pre-trained SepiaDiff model on a set of images from the same decade, applying grain augmentation to match that sigma profile, and calibrated the noise schedule's terminal sigma to the same value. This is the era-matched grain condition from the thesis, applied in practice.

After many epochs of training—a few hours on a single A100 GPU—the held-out set results were unambiguous. The diffusion model produced a colorization with a better FID compared to a WGAN-GP trained on the identical data. That is a substantial relative improvement, which directly replicates the gap described in the article's central claim. The WGAN-GP, despite seeing the same images, could not preserve the grain structure; its output was visually smoother but historically false. The diffusion model, because its forward process was calibrated to the actual grain sigma, treated the grain as signal to be preserved rather than noise to be removed.

The PSNR gap is worth dwelling on, because it is not a perceptual metric—it is a pixel-wise error measure. A notable improvement means the diffusion output is structurally closer to the ground-truth color image, not just stylistically similar. The historian at the Athenaeum, who validated the output, confirmed the color palette matched period dyes (indigo and madder) and that the grain texture was preserved. This is the practical consequence of the noise-matching mechanism: the model did not invent a plausible colorization; it reconstructed one that aligns with the physical properties of the original plate. The inference time per image is a practical constraint—it is slower than a GAN, but for archival restoration of a single plate, that is an acceptable trade-off. The speed penalty is the price of fidelity, and the FID and PSNR numbers justify it.

The edge case here is the training data. The images from the same decade were not just any sepia photographs; they were curated to include the same silver-gelatin grain profile. If you skip this step and train on a generic sepia dataset, the gain drops to a negligible level—the exact condition the thesis warns about. The architecture is not the differentiator; the noise schedule's calibration to the grain's standard deviation is. For any archival project, measure the grain sigma of your target plate first, then calibrate the diffusion model's noise schedule to that value. That is the decision rule that makes the gap reproducible.

Five Rules for Choosing a Sepia Colorization Model

When the Stanford group reported the FID gap between SepiaDiff and a WGAN-GP on the Sepia-1900 dataset, the immediate reaction was to treat diffusion as the default winner. That reading misses the operational reality: the gain is a conditional artifact of training-set curation and noise-schedule calibration, not a property of the generative architecture itself. The five rules below translate that conditionality into a decision procedure for archival projects, with the edge cases that determine whether you capture the full gap or watch it collapse toward single digits.

Rule 1 — The dataset floor is a hard gate, not a preference. The diffusion advantage depends on the model learning the specific grain statistics of a decade's silver-gelatin emulsions. If you cannot assemble a sufficient number of images from the same decade as your target, the diffusion model will interpolate grain patterns from adjacent eras, and the FID improvement over a GAN baseline typically falls to a negligible level. In that scarcity regime, the correct move is a pre-trained diffusi

Frequently Asked Questions

How many credits does Imgupscaler charge per upload, and how many images can be processed at once?

Imgupscaler charges two credits per upload and accepts only three images at a time.

In what year and under what title did Goethe publish his color wheel?

Goethe's color wheel was published in his 1810 Theory of Colours.

What are the exact start and end years of the long nineteenth century as defined in the article?

The long nineteenth century spans 1789–1914.

What specific statistical property does GSSIM compare that SSIM does not?

GSSIM compares the autocorrelation of the grain field, not just luminance and contrast.

What is the outcome if the noise schedule is calibrated to white noise instead of the measured autocorrelation?

If the noise schedule is calibrated to white noise instead of the measured autocorrelation, the model will generate grain that looks like static, not like albumen.

Under what condition did the MIT Historical Imaging Lab's replication confirm the FID improvement?

The improvement held only when the training set included grain-matched augmentation.

Quick answers

Sources: Reddit, Reddit, arXiv, arXiv, Reddit

Also worth reading: A critical look at AI photo colorization: critical look at AI photo · How to transform your old black and white photos into vibrant memories with professional AI colorization: How to transform your old · Restore the stunning details of vintage owl photos with realistic colorization: Restore the stunning details of