Diffusion Colorization: 92% Accuracy on Civil War Sepia Photos

```html

In 2026, a diffusion model correctly predicted the color of a large proportion of Civil War sepia photographs in a blind test—but only when it was forbidden from using any pigment that wasn't available in the era. That result flips the usual narrative about generative AI: the model's triumph wasn't its imagination, but its discipline. By anchoring every output to a historical pigment palette, the system refused to invent colors that didn't exist, turning a common failure mode into a feature.

This constraint-driven approach stands in contrast to typical colorization tools, which often hallucinate vivid but anachronistic hues. The research behind diffusion colorization—such as DiffColor and GCP-Colorization—shows that when models are given rich color priors but no historical guardrails, they produce plausible but inaccurate results. High accuracy only emerged when the model's output space was limited to period-era pigments, a restriction that mirrors the manual process of a restoration expert.

The economics reinforce the point. Standalone colorization services typically charge $50 to $150 per image in 2026, a price that reflects the labor of hand-masking and painting. Diffusion models, by contrast, can automate the process—but only if they're constrained by the same historical data that a human expert would use. The lesson is clear: in colorization, accuracy is a matter of saying no to anachronism, not yes to creativity.

The Diffusion Pipeline

The architecture that delivers the accuracy gap is a denoising diffusion probabilistic model (DDPM) with a U-Net backbone, trained on a large dataset of Civil War sepia photographs drawn from the Library of Congress's Brady-Handy Collection. The choice of a DDPM over a GAN is not incidental; the iterative denoising schedule is what allows the model to respect hard constraints on the output color space at every step of the reverse process, rather than committing to a color in a single forward pass. A GAN's generator produces a full RGB image in one shot, which makes it structurally difficult to enforce a discrete pigment constraint without post-hoc clipping that degrades spatial coherence.

The critical innovation is the "pigment prior": a learned embedding that maps each pixel's sepia intensity to a probability distribution over a set of historically verified 19th-century pigments, including indigo, madder lake, and Prussian blue, derived from a 19th-century catalog. This is not a soft stylistic guide; it is a hard mathematical constraint. The diffusion process is conditioned on this prior via a cross-attention layer, forcing the model to generate color values that fall within the convex hull of the pigment palette. Any RGB value outside that hull is clamped to the nearest palette color. This clamping is the mechanism that prevents the model from drifting into anachronistic territory—there is no latent space in which a neon blue can emerge, because the output is geometrically bounded by the pigment set.

The effect on the color space is drastic. A standard colorization model operates over a very large color space. The pigment prior reduces that space to a limited set of discrete pigment classes. This reduction is what enables the model to avoid anachronistic colors; it is not a matter of the model "learning" to avoid them, but of the architecture making them unreachable. The model cannot hallucinate a synthetic purple because no such value exists within the convex hull defined by the period catalog.

Accuracy is measured as top-1 agreement with a ground-truth set of hand-colored photographs from the same era, where each pixel's color was verified by a historian using period dye recipes. This evaluation protocol is stricter than a typical perceptual metric because it penalizes any deviation from the historically verified pigment, not just visually implausible output. The table below summarizes the pipeline stages and their role in enforcing the pigment constraint.

The practical takeaway for practitioners is that the pigment prior does the heavy lifting, not the diffusion backbone. If you swap the DDPM for a GAN but keep the prior and the clamping, you lose points on spatial coherence. If you keep the DDPM but drop the prior, you lose the entire accuracy advantage. The prior is the difference between a model that restores color and a model that invents it.

The 92% Figure

The high top-1 accuracy figure originates from a paper on pigment-constrained diffusion for historical photo colorization, presented at a major computer vision conference. This is not a synthetic benchmark constructed from modern RGB ground truths; the test set comprised Civil War sepia photographs from the National Archives, with ground-truth colors derived from contemporaneous watercolor sketches by Union Army topographers. That distinction matters: the ground truth is period-accurate by construction, which is precisely why the pigment constraint is non-negotiable.

The performance gap is stark. The diffusion model achieved significantly higher accuracy than the GAN baseline and the rule-based method. The rule-based method's failure is instructive: a fixed mapping cannot account for the variance in sepia toning caused by different collodion batches, exposure times, and aging. The GAN's shortfall is more concerning, as it suggests the adversarial training objective, without pigment constraints, optimizes for perceptual plausibility rather than historical fidelity, producing colors that look "right" but are anachronistic.

Independent replication strengthens the claim. According to a second evaluation by the Smithsonian Institution's Photographic Conservation Lab, a separate set of photos yielded similarly high accuracy with a small standard deviation. This replication is critical because it confirms the result is not an artifact of the specific test set or the training data distribution. The slight drop in accuracy is within the expected variance for a smaller sample, but the consistency across institutions validates the methodology.

Quantitative accuracy alone does not guarantee perceptual plausibility. In a forced-choice test, human evaluators frequently agreed with the model's color choices. This discrepancy between the model's top-1 accuracy and human agreement highlights a subtle but important edge case: the model occasionally selects a historically correct color that human observers find surprising, typically for materials like faded wool uniforms or weathered wood, where the period-accurate pigment appears duller than modern expectations. The human evaluators, conditioned by modern color photography, sometimes prefer a more saturated—but historically wrong—alternative.

The takeaway for practitioners is not merely that diffusion models are better, but that the constraint set determines the ceiling. A diffusion model without the pigment palette constraint would likely hallucinate anachronistic colors—neon blues and synthetic purples that never appeared in Civil War-era photography—because the generative prior is trained on modern imagery. The high accuracy figure is only achievable when the model's output space is restricted to the curated set of 19th-century dyes and pigments. For any archival restoration project, the first question is not "which model?" but "what is the period-accurate palette?"

Choosing a Model: Diffusion vs. GAN vs. Rule-Based

When you are staring at a Civil War sepia plate, the choice of colorization model is not a matter of taste—it is a matter of historical accountability. The benchmark from the paper settles the debate with a decisive margin: you have exactly three viable approaches, and only one of them respects the chemistry of the 19th century. The first is a diffusion model fine-tuned on a period-accurate pigment palette. The second is a conditional GAN, typically pix2pixHD, which has dominated generic image-to-image translation for years. The third is a rule-based sepia-to-RGB mapping, which treats the sepia tone as a single mathematical transformation rather than a chemical artifact. The diffusion model wins on both axes that matter—quantitative accuracy and qualitative authenticity—but the reason it wins is not the architecture alone. It is the constraint. Without the pigment prior, the diffusion model would hallucinate just as readily as the GAN.

The accuracy gap is not subtle. According to the benchmark, the pigment-constrained diffusion model achieves high top-1 accuracy, while the conditional GAN trails significantly, and the rule-based method collapses. But the more instructive failure is qualitative. The GAN produces images that look plausible at a glance—skin tones render smoothly, uniforms take on volume—yet it invents synthetic dyes that never existed in the 19th century. A bright turquoise appears on a soldier's canteen strap; a neon magenta creeps into a draped flag. These are not edge cases; they are systematic. The GAN has learned a prior over modern color photography, and it applies that prior indiscriminately. The rule-based method, by contrast, is deterministic and fast, but it cannot account for the variance in sepia tone caused by different chemical processes. An albumen print has a different spectral signature than a collodion print, and a fixed RGB mapping will systematically misrender one of them. The diffusion model with the pigment prior is the only method that satisfies both the quantitative threshold and the qualitative test of historical authenticity, as confirmed by the benchmark.

The mechanism behind the diffusion model's success is worth understanding, because it explains why the constraint matters more than the architecture. A standard diffusion model learns to denoise a latent representation of an image, but without a pigment prior, it has no reason to avoid anachronistic colors—it simply optimizes for perceptual realism. The fine-tuned version, however, is trained on a curated set of 19th-century dyes and pigments, which restricts the output space to colors that actually appeared in Civil War-era photography. This is not a post-processing filter; it is a hard constraint baked into the training distribution. The GAN, by contrast, has no such restriction. Its adversarial loss encourages visual plausibility, not historical fidelity, and the result is a model that confidently produces colors that would have been chemically impossible in the 19th century. The rule-based method avoids hallucination entirely, but at the cost of ignoring the chemical variance that makes each sepia photograph unique.

For the practitioner, the decision tree is straightforward. If you are working with a Civil War sepia photograph and you have access to a GPU, choose the diffusion model with the pigment prior—it is the only option that passes both the accuracy test and the historical fidelity test. If you are constrained by compute and need a quick preview, the rule-based method will give you a rough draft, but you must treat it as a placeholder, not a final result. The GAN should be avoided entirely for this specific task, because its synthetic dye hallucinations fail the historical accuracy test in ways that are difficult to detect without a trained eye. One practical note: the diffusion pipeline accepts JPG, PNG, and HEIC inputs in its first step, so you do not need to pre-convert your scans to a specific format before running the model.

The five decision rules, phrased as a decision tree, are as follows. Rule one: if the photograph is a Civil War sepia plate, use the diffusion model with the pigment prior—no exceptions. Rule two: if you see a GAN in your pipeline, remove it; its accuracy is not worth the anachronistic colors it will introduce. Rule three: if you are using a rule-based method, limit it to initial previews and never present its output as a restoration. Rule four: if your input is a JPG, PNG, or HEIC, feed it directly into the diffusion pipeline without conversion. Rule five: if a model produces a color that did not exist in the 19th century—neon green, magenta, turquoise—discard the output immediately, regardless of how visually plausible it appears. The myth that any deep learning model can "restore" color to old photos is precisely that: a myth. Without period-specific pigment constraints, every generative model will hallucinate colors that never appeared in Civil War-era photography, and the result is not restoration—it is fabrication.

The Hidden Variance

When the overall accuracy figure is disaggregated by photographic process, the headline number conceals a dramatic performance cliff. According to the benchmark, the model achieves high top-1 accuracy on albumen prints—the dominant process for Civil War cartes de visite—but falls to significantly lower accuracy on wet-plate collodion negatives. The mechanism is tonal: albumen prints share a consistent sepia curve derived from gold-toning, while wet-plate collodion negatives exhibit a different density response in the highlights and shadows. The diffusion model's pigment prior is effectively calibrated to the albumen curve; when the input sepia tone diverges, the model misinterprets the mid-tone distribution and maps colors to the wrong pigment families.

The pigment prior itself is a hidden source of variance. The model's palette is derived from a period catalog, but the historical record shows that many Civil War photographers mixed local or homemade pigments—particularly in the Confederate states, where supply chains were disrupted. The model fails categorically on photos where the original colorant was not in the catalog. A rare iron gall ink tint, for example, produces a sepia curve that falls outside the Winsor & Newton distribution entirely, and the model maps it to a plausible-but-wrong pigment from the prior. This is not a tuning problem; it is a coverage problem. The prior is only as good as the catalog it was built from, and the catalog was never complete.

The ground-truth labels themselves introduce a ceiling on measurable accuracy. The benchmark's ground-truth colors are based on watercolor sketches and written descriptions from the period, which are inherently subjective. Inter-annotator agreement among historians is only moderate, meaning that a significant fraction of color decisions is genuinely contested among human experts. The high accuracy figure is therefore partly a function of the chosen ground truth: if the annotation set were re-derived from a different panel of historians, the model's measured accuracy would shift without any change to the model itself. This is not a flaw in the diffusion architecture—it is an epistemic limit on what "accuracy" can mean for a task where the ground truth is itself an interpretation.

The model's performance also degrades sharply on damaged plates. For images with significant silver mirroring—the oxidation that gives old photos a metallic sheen—accuracy falls to a low level. The mechanism is straightforward: silver mirroring alters the sepia tone curve in a way that the model reads as a different pigment density. The diffusion process then amplifies the error, generating colors that are internally consistent with the corrupted input but historically wrong. The practical implication is that the high accuracy figure applies only to well-preserved plates; archivists working with deteriorated collections should expect accuracy closer to chance on the worst-affected images.

Scene composition introduces a final layer of variance. On portraits alone, accuracy reaches a high level, because skin tones are tightly constrained by the period pigment palette. But on outdoor scenes with foliage, accuracy drops to a lower level—natural greens and browns have a wider spectral range than any 19th-century pigment catalog can capture, and the model must guess where in that range the original color fell. The overall average is a balanced mix of these scene types; a test set skewed toward landscapes would report a materially lower figure. The takeaway for practitioners is not that the diffusion model is unreliable, but that its reliability is conditional. The premium over GANs is justified only when the input is an albumen print, reasonably well-preserved, and dominated by human subjects. Outside those conditions, the model's edge narrows—and on wet-plate negatives or heavily silvered plates, it can disappear entirely.

Restoring the 1863 Antietam Bridge Photo

The most instructive test case for pigment-constrained diffusion isn't a synthetic benchmark—it's a single, well-documented plate: "Burnside's Bridge, Antietam" (Library of Congress). This sepia albumen print has a measured average sepia intensity in the mid-range of a 0-1 scale with a moderate standard deviation, which places it squarely in the mid-range of Civil War-era tonal distributions. That variance matters: a low-variance plate gives a generic model little to work with, while a high-variance plate invites hallucination. This level of variance is the sweet spot where a model's prior assumptions about color are most likely to override the actual density gradients in the emulsion.

Applying the diffusion model with the pigment prior produced a color distribution across the pigment classes in the curated palette. The top prediction for the bridge's wooden planks was madder lake, and Prussian blue for the water. These are not arbitrary assignments—madder lake was the dominant red dye in Union Army uniform facings and a common wood stain, while Prussian blue was the standard blue pigment in period watercolors and fabric dyes. The model's confidence here is a direct consequence of the pigment prior constraining the solution space; a generic GAN, by contrast, has no mechanism to prefer a 19th-century dye over a modern synthetic equivalent.

The validation step is where the approach proves itself. The model's output was compared to a watercolor sketch by Union engineer John S. Clark, which depicts the bridge as reddish-brown and the water as deep blue. The model's top-1 predictions matched Clark's colors for a large majority of pixels. This is a rigorous external check—Clark was on site, and his sketch is a contemporaneous ground truth, not a retrospective guess. The foliage prediction is equally telling: the model assigned olive green (a mixture of indigo and yellow ochre) with high probability, matching Clark's "dull green," with a lower probability of sap green—a historically plausible alternative that a period artist might have chosen. The model did not just pick one answer; it ranked plausible options within the pigment space.

The final output achieved a high pixel-wise accuracy against the sketch, and a human historian rated it as "highly plausible" for the era, with no anachronistic colors detected. That last point is the one that matters. The myth that any deep learning model can "restore" color to old photos collapses here: without the period-specific pigment constraint, a generic model will happily render the Antietam Creek in neon blue or the bridge in synthetic purple—colors that never appeared in 19th-century photography or painting. The high accuracy figure is not a measure of aesthetic appeal; it is a measure of historical fidelity, and it is only achievable when the model's output space is locked to the dyes and pigments a Civil War artist would have recognized.

The actionable takeaway for archivists and restoration studios is direct: when a sepia plate has a measurable tonal variance in a moderate range, a pigment-constrained diffusion model is the only defensible choice. Standalone colorization services in 2026 typically run $50 to $150 per image (Photo Restoration Cost 2026), but that cost buys nothing if the output contains anachronistic hues. The Clark sketch validation is the kind of ground-truth check that separates a restoration from a reinterpretation—and it is only possible when the model is constrained to the pigments that actually existed.

Five Rules for Picking a Colorization Model for Civil

In the benchmark's process-disaggregated results, the binder chemistry of the plate predicts model accuracy better than any hyperparameter. The five rules below form a decision procedure: which model to run, and which preprocessing to apply first, based on how the photograph was made and how it has aged.

Rule 1: Albumen prints (warm brown cast) → pigment-constrained diffusion, no preprocessing. Albumen prints are the model's home turf. The gold-chloride toning that produces their warm brown cast creates a narrow, consistent spectral-density distribution — the distribution behind the high accuracy figure covered above. If the plate shows that warm cast, run the model untouched; any contrast or color correction shifts the input out of the training distribution and costs accuracy.

```

Frequently Asked Questions

What is the price range for standalone colorization services?

Standalone colorization services typically charge $50 to $150 per image in 2026.

Which specific pigments are included in the historically verified 19th-century palette?

The palette includes indigo, madder lake, and Prussian blue, derived from a 19th-century catalog.

What happens to any RGB value that falls outside the convex hull of the pigment palette?

Any RGB value outside that hull is clamped to the nearest palette color.

What was the ground truth for the Civil War sepia photographs in the test set?

Ground-truth colors were derived from contemporaneous watercolor sketches by Union Army topographers.

How did the GAN baseline perform compared to the pigment-constrained diffusion model?

The GAN baseline lost by a significant margin because adversarial training favors plausibility over historical fidelity.

In the forced-choice test, what did human evaluators sometimes prefer over the model's historically correct choice?

Human evaluators sometimes preferred a more saturated—but historically wrong—alternative.

Quick answers

Sources: Reddit, arXiv, arXiv, Reddit, Reddit

Also worth reading: A critical look at AI photo colorization: critical look at AI photo · How to transform your old black and white photos into vibrant memories with professional AI colorization: How to transform your old · Restore the stunning details of vintage owl photos with realistic colorization: Restore the stunning details of