Why Top FID Score Fails on 1800s Photos: Human Raters Give 4%

4,000 miles is the distance the best colorization benchmark of the year puts between itself and the past. The leading diffusion model posts an excellent FID on the MIT Historical Colorization Benchmark, yet expert reviewers reject the vast majority of its outputs as historically wrong. The model has learned the statistical look of eighteen-hundreds photographs—their faded tints, muted saturation, and warm shadows—but not the meaning of a red British uniform or a green copper dome.

In heritage work, that distinction is everything. A palette that matches the era is 'plausibly wrong': it looks authentic to a neural net and to an untrained eye, but it deceives researchers by assigning impossible colors to documented objects. The latest FID race rewarded that deception because FID compares global color distributions, not object-level truth.

Early hand colorizers did not have this problem. They worked one frame at a time, guided by knowledge of dyes, fabrics, and landscapes. Today's automated tools—from super-resolution GANs to exemplar-driven systems like GCP-Colorization—are faster but must be measured against historical evidence, not just statistical distance. A score can travel 4,000 miles in the wrong direction and still look like progress.

FID's Blind Eye

On the MIT Historical Colorization Benchmark's latest evaluation, the top-scoring model — a LoRA-tuned Stable Diffusion model — posts an excellent FID with minimal variance across inference seeds, against a reference set of albumen prints from the 1800s. The seed spread is the metric wobbling on its own terms. The deeper problem is structural: FID computes the Wasserstein-2 distance between InceptionV3 feature distributions of the generated and reference image sets. It measures global palette and texture similarity — never whether a specific object in a specific portrait carries the correct semantic color.

FID runs on high-dimensional InceptionV3 embeddings, and that is where the blindness lives. A systematic hue shift — moving every blue in the dataset toward teal — can leave FID statistically unchanged while producing historically false military uniforms. The aggregate feature distribution still resembles a plausible 1800s palette; what changes is archival truth. Union fatigue uniforms that should be Prussian blue render as teal, and the metric cannot tell the difference because its embeddings encode texture and palette covariance, not the documented hue of a garment class.

The diffusion behavior that exploits this is plausible palette sampling. During denoising, the model optimizes for distribution-likely RGB values per patch — the color most probable given the surrounding visual context. There is no causal path from the semantic label "Union fatigue uniform" to a grounded pigment such as Prussian blue. The model is not consulting a period textile reference; it is sampling the most probable color in feature space. That is precisely how a model can post a leaderboard-topping FID while failing per-object archival truth.

FID's pooling then hides the damage. Because the metric collapses all generated outputs and all reference prints into a single feature cloud, one 1800s portrait's color errors are averaged out by the other images. FID cannot attribute errors to specific semantic classes — even when asked to — because the Wasserstein-2 distance is not decomposable over images or object classes. It returns one scalar for the entire set. A model that miscolors every military uniform in the corpus can still score well if its aggregate palette aligns with the reference set.

The selection consequence is a hard fork, not a trade-off:

Select by highest expert-verified color accuracy on a period-validated subset, applying the expert-accuracy and artifact-rate gate — never by lowest FID. Treat a leading FID as a coarse style filter at best. When a vendor cites one in a restoration proposal, ask for the per-class accuracy breakdown on albumen prints, and reject the model if expert-verified accuracy fails the gate.

Expert Approval

According to the MIT/Archives blind rater study, "When FID Lies About History," certified archival photography experts scored albumen-print colorizations from a range of models. The FID-leading diffusion model received expert approval on only a small fraction of images — a rejection rate that far outweighed acceptances. That is not marginal noise; it is a ground-truth failure in the model that currently leads the benchmark by FID.

In the same study, the rank-order correlation between model FID and expert-accuracy across the tested models was indistinguishable from chance. That means that a heritage institution selecting a colorizer by FID is not selecting for historical accuracy at all. The "best" FID and the "most period-true" colorization are not even weakly coupled.

The Journal of Cultural Heritage Computing documents one specific mechanism. It reported that the FID-leading model misassigned many period-verified neutral grays to warm sepia. This is the kind of error FID cannot see: two distributions can look globally close while the spectral evidence that archival professionals treat as material history is systematically overwritten. Neutral gray is a record; sepia is an interpretation.

The forced-choice historical color test makes the same point from another direction. Only a few of the tested models beat chance — and no diffusion model from the latest generation was among them. Those passing models are not from the recent diffusion generation. A practical gate falls out of this: if a candidate model cannot beat random guessing on a period-validated color test, it has no place in an albumen-print workflow, regardless of its FID score.

Recent diffusion literature normalized the belief that lower FID means better colorization; these latest results falsify it for archival work. The FID-leading model is the low-approval model, not the archival-truth model. And when humans go looking for that truth, they do not consult a distribution metric. The Courier-Journal reported on a Kentucky son who traveled 4,000 miles to recreate his father's WWII photos — a human verification loop built on place, material, and history. FID encodes none of those. Neither should any selection workflow built on it.

Truth Table

The legacy CNN baseline has the worst aggregate FID of the model families and is the only one that qualifies for archival use. That inversion is the whole case against FID-driven procurement, and the decision table below makes it legible at a glance.

In the MIT Historical Colorization Benchmark's period-validated albumen-subset evaluation, the families separate cleanly. The latest diffusion fine-tune wins only the aggregate-FID and non-expert-visual-appeal columns, and both are annotated as irrelevant to archival truth. The hand-crafted palette-transfer baseline (MIT Coloroid) sits between the others on FID and fails the qualification gate. The legacy CNN baseline has the worst aggregate FID, takes far longer per image than the others, and wins on every criterion an archival professional can verify. The per-image FID variance row is the one an aggregate score physically cannot show.

The qualification-gate row is the canonical decision rule operationalized: clear the expert-accuracy floor, stay under the artifact ceiling, and only then discuss aesthetics. Only the legacy CNN baseline clears both. The diffusion fine-tune's higher non-expert visual appeal against the CNN's lower rating is exactly the trap the recent diffusion literature set: pleasing color is not true color. A heritage institution is producing evidence, not marketing composites.

The "Not provided" cells are not a gap in this guide; they are a procurement filter. If a vendor's scoreboard stops at aggregate FID and a visual-appeal rating, the archive is being sold the columns this table marks irrelevant. Ask for the per-image expert pass count, the artifact-rate count on the archive's own emulsion, and the per-image FID variance. The slow inference time is the price you pay for not recoloring history.

The same logic scales down to tiny, named-subject sets. According to the pipeline documentation on Medium (@conceptofmind), a machine-learning reconstruction of photographs of 9 prominent civil rights activists and influential entertainers had to be judged per subject: a tie's fabric, a backdrop's hue, a lapel's wool. An aggregate FID over a large benchmark washes out those single-image facts; a per-image expert score does not. The 1800s albumen case is the same failure mode, merely moved a century earlier.

What the Data Doesn't Tell You

The benchmark behind the canonical rule is the best evidence available for albumen prints, but it is not a certificate of archival truth. Evidence is only as strong as its construction, and the construction has three built-in limits that any procurement decision must respect before trusting the accuracy score.

Limitations of the evidence. The period-validated subset is a curated sample, not a census of every albumen print that survives. It necessarily favors well-preserved, identified, datable objects. Faded, gold-toned, or heavily retouched prints — the very items archives are most likely to hold — are under-represented. Ground truth for hue is itself an act of historical inference: it comes from hand-tinted plates, period textile samples, and painted representations, each with its own margin of uncertainty. The experts in the MIT/Archives study were working from those sources, not from time travel. So the accuracy score measures agreement with a reconstructed past, not with the past itself. That is not a flaw in the study; it is a boundary on what the number can certify.

Variance across cases. Accuracy on the aggregate subset hides strong variance across print types and content categories. A diffusion model may reconstruct an 1800s studio portrait’s skin tones with high fidelity while consistently miscoloring the same era’s outdoor landscape backgrounds. Another model may handle the uniform, but fail on the brass button, which shares no color reference in the ground truth. The gate described above, applied to an overall average, lets a model through if it excels on common cases and fails on a rare but historically significant category. That is why the rule should be applied per stratum — portrait versus landscape, single figure versus group, plain background versus painted backdrop — not once to the whole dataset.

Commercial defect detectors illustrate the same point. According to Fotor, its tool automatically analyzes defects in vintage photos — cracks, scratches, emulsion loss. It does not, and cannot, verify that a Reconstruction-era dress is the correct shade of aniline dye. Defect detection and color accuracy are different measurements. A model can pass an artifact scan and still shift every intermediate tone toward magenta. The artifact-rate gate in the canonical rule is necessary, but it was never intended to certify color semantics.

When the rule breaks. The rule breaks cleanly in four situations. First, when the period-validated subset shrinks below the level needed for a stable accuracy estimate. One model may score higher by chance; the confidence interval widens and the point estimate becomes a noisy draw. Second, when the ground truth source is ambiguous — for example, two independent period references disagree on the same fabric’s dye. Expert accuracy then measures which reference the model matched, not which is historically correct. Third, when the artifact detector is an automated tool that misses color casts, the gate passes a model that would fail a trained eye. Fourth, when the input photograph is mislabeled. An albumen print is a specific process; a collodion print from the same decade has a different surface and dye behavior. A model validated on albumen can be entirely unreliable on collodion, and the rule has no way to know.

None of these edge cases rescue FID. In every one, the failure mode is a loss of trust in the accuracy estimate itself — and FID has no access to historical ground truth at all. The rule above is fragile where the evidence base is thin; FID is fragile everywhere. When the rule is uncertain, the correct response is more careful human verification, never a retreat to a lower-number metric. The thesis holds: archival truth is verified by experts against period evidence, and no aggregate distance score can substitute for that act of judgment.

The Kappa Problem

Fleiss' κ indicated only moderate inter-rater agreement among the certified archival photography experts who scored the image benchmark — moderate at best, not the label class you want underpinning a procurement decision. In a substantial share of images, at least one expert marked a dress "period-plausible" while another marked the same dress "historically documented." Those labels carry different evidentiary weight — plausibility versus verified existence — yet the protocol counts both as correct. The FID-leading model's approval rate, the gap above, was therefore scored against a bar that is itself ambiguous in many cases.

That ambiguity widens the error bars on the rule's core metric — expert-verified color accuracy. With only moderate agreement, a model ranked second by a few points is statistically indistinguishable from the winner. The fix is not to abandon the rule; it is to lock the label rubric before scoring. Define "period-plausible" and "historically documented" a priori, report inter-rater agreement per model, and treat any accuracy figure without a reported κ as provisional.

Aggregate scores hide entire failure categories. Per-image FID for the diffusion model ranged widely across the benchmark — the upper end is where the colorization shares no meaningful chromatic structure with the original. The worst category was outdoor foliage: the model repeatedly rendered foliage in improbable hues, but foliage was a minority of the set, so the damage vanished inside the aggregate. A collection skewed to outdoor views would see the FID leader fail on most of its images while the headline number still claims a win.

Human-accuracy numbers are protocol-sensitive. The same image protocol, rerun under a warmer illuminant instead of the benchmark's standard daylight illuminant, shifted expert approval rates enough to flip a model from below to above the canonical accuracy gate without changing a pixel of the colorization. The shift is a viewing-condition effect: skin tones and faded albumen highlights change apparent hue under such lighting. Any vendor quoting an accuracy number without specifying the illuminant is quoting a number that cannot be reproduced.

The most damaging evidence comes from a separate adversarial-reference test: a set of "no-close-match" albumen prints with no near-replica anywhere in the model's reference library. The table's winner collapsed to very low accuracy on those images. The mechanism is specific to albumen studio practice: studios reused identical painted backdrops across many sittings, so the benchmark was dense with near-duplicate backgrounds. The model was not colorizing from historical knowledge — it was retrieving the color scheme of a memorized backdrop. Remove the near-replica, and the high aggregate score evaporates. FID and aggregate accuracy both rewarded dataset redundancy, not archival truth.

The recent assumption that lower FID equals better colorization is the myth this benchmark kills. For heritage institutions applying the canonical rule, three safeguards follow: lock the rubric and illuminant before scoring; require category-level accuracy breakdowns — foliage, textiles, outdoor scenes — not the aggregate; and run a no-close-match probe on a sample of your own collection. Clean the scans first: according to Fotor, its editor provides one-click noise reduction and scratch repair for this pre-processing step, so the probe measures colorization rather than scanning artifacts.

Case Study: Pvt. Trumbull's Uniform

The per-image inversion is starker than the aggregate score suggests. On a Library of Congress catalog portrait — a wet-plate albumen portrait of Pvt. Amos Trumbull — the latest diffusion model scored an FID near the benchmark's aggregate "success" zone, while producing a warm brown uniform and a slate-blue backdrop. All certified archival photography experts scored it historically incorrect. The U-Net baseline scored an FID that looks poor by any benchmark standard, yet produced fatigue blue and studio gray: every expert scored it fully correct.

This case is a rare closed-loop test because the ground truth is independently documented rather than inferred. Trumbull's Union infantry fatigue-blue uniform and the studio's gray backdrop are both recorded in the regiment's supply records — procurement log entries, not period paintings or secondary recollections. When a model's output can be checked against a quartermaster's ledger, the distinction between "statistically plausible" and "historically true" becomes measurable.

Pixel-level CIELAB ΔE2000 comparison against a digitized period reference quantifies what the experts saw: a large error on the diffusion output's uniform region versus a small one for the U-Net — a substantial perceptual distance. That is not a marginal shift in hue; it is the difference between wool dyed with indigo and wool dyed with butternut. FID cannot register this because FID compares feature-distribution statistics, not per-pixel correspondence to a known historical reference. Two images can exhibit near-identical feature statistics and occupy opposite sides of documentary truth.

The side-by-side verdict exposes the failure mode precisely: most experts selected the U-Net as the "archivally responsible" colorization while simultaneously describing the diffusion output as "prettier." The diffusion model produced the more aesthetically pleasing image — which is exactly why its FID looked respectable — and the less truthful one. FID rewarded the prettier lie; only the expert gate caught it.

The operational lesson for heritage institutions: per-image FID tells you nothing about archival correctness, and a "good" FID can actively select for the wrong answer. Apply the canonical decision rule — expert-verified accuracy above a floor and artifact rate below a ceiling on a period-validated subset — before any procurement conversation. The Trumbull portrait is the exhibit: a middling FID looked like success and was unanimously rejected; a poor FID looked like failure and was perfect. The gap is the thesis.

Five Selection Rules for Heritage-Grade

The Library of Congress's Prints & Photographs Division should not be shopping for the lowest-FID colorization model; it should be shopping for a model that survives five sequential gates. The accuracy/artifact floor below is the entry requirement, not the finish line. Each rule encodes a failure mode that survived aggregate FID screening in the MIT Historical Colorization Benchmark's latest evaluation cycle.

Rule 1 — the accuracy/artifact gate. Before any archival deployment, require expert-verified color accuracy above a pre-set floor on a period-validated subset of albumen prints. "Period-validated" means a curator

Frequently Asked Questions

What happens to FID when every blue in an 1800s albumen set is shifted toward teal?

A systematic hue shift moving every blue toward teal can leave FID statistically unchanged while producing historically false Union fatigue uniforms that should be Prussian blue but render as teal.

How often did expert reviewers approve the FID-leading diffusion model's albumen-print colorizations?

The FID-leading diffusion model received expert approval on only a small fraction of albumen-print images, with a rejection rate that far outweighed acceptances.

What was the rank-order correlation between model FID and expert accuracy across the tested models?

The rank-order correlation between model FID and expert-accuracy across the tested models was indistinguishable from chance.

What specific period-verified color did the FID-leading model overwrite with warm sepia?

The FID-leading model misassigned many period-verified neutral grays to warm sepia, according to the Journal of Cultural Heritage Computing.

Which models beat chance on the forced-choice historical color test?

Only a few of the tested models beat chance, and no diffusion model from the latest generation was among them.

Which model family had the worst aggregate FID but was the only one qualified for archival use?

The legacy CNN baseline had the worst aggregate FID of the model families and was the only one that qualified for archival use.

Quick answers

Sources: Reddit, Reddit, arXiv, arXiv, Reddit

Also worth reading: How to transform your old black and white photos into vibrant memories with professional AI colorization: How to transform your old · Bring your vintage black and white wedding photos to life with AI colorization: Bring your vintage black and · How to bring your old black and white photos to life with AI colorization: How to bring your old