This guide provides a three-test workflow for comparing AI and human photo colorization models: perceptual score, blind human choice, and final model selection. It explains when MCIQA-2K's 2,000-image evidence applies and requires a model to win the human-choice test by at least 10 percentage points over the runner-up.
How the 3-test colorization pipeline works
MCIQA-2K, as described in its arXiv abstract, is a no-reference quality dataset containing 2,000 colorized images produced by five representative colorization models. Human annotators assessed each image across three perceptual dimensions: color smearing, semantic color misalignment, and overall perceptual quality. Because the dataset includes no pristine reference image, the resulting score measures whether a colorization looks convincing without comparing it pixel by pixel with an original color photograph. The practical check is to calculate this score for every candidate output on the same test set rather than comparing screenshots made under different conditions.
Test 1 is the Perceptual Score. For each image, calculate a score derived from the three MCIQA-2K dimensions, then aggregate those per-image scores for each model. A model can advance only when its aggregate perceptual result is competitive, but this score is a screening test—not the final decision. For 2026 model selection, apply the stated applicability conditions: the source image’s skin-mask coverage must exceed 5%, and the candidate under consideration must be diffusion-based. If either condition fails, do not treat the perceptual score as sufficient evidence for selection.
Test 2 is the Human Choice test, conducted as a blind pairwise preference. Reviewers should see the competing colorizations without model names, processing-order clues, or other signals that could reveal which system produced an image. Ask them to select the result they consider more convincing, using the same pairwise question for every model pairing. Record the proportion of blind preferences won by each model. This stage supplies the evidence that the automatic perceptual score is intended to predict, while preventing brand familiarity or presentation order from biasing the result.
Test 3 is Pick a Model. Select the model with the highest combined rank from Tests 1 and 2, with the blind human result taking priority when the automated and human assessments conflict. The acceptance rule is deliberately stricter than choosing the leader by a narrow margin: the selected model must win Test 2 by at least 10 percentage points over the runner-up. A smaller lead does not qualify, even if that model also has the best perceptual score.
Run the sequence in order: calculate the perceptual score, obtain the blind human choice, and then pick the model. A copy-usable record for each image should include its MCIQA-2K dimension scores, per-image perceptual total, blind pairwise result, skin-mask coverage, diffusion-based status, and the final model decision. This creates an auditable trail and makes the 10-percentage-point acceptance threshold easy to verify without adding any unverified cost, resolution, or processing figures to the selection process.
Evidence: what the numbers actually say
The strongest numerical signal comes from the 2,000-image MCIQA-2K evaluation, which pairs five representative colorization models with human annotations across three perceptual dimensions: color smearing, semantic color misalignment, and overall perceptual quality. That combination matters because a model can look locally convincing while still assigning implausible colors to objects or producing uneven regions. The practical check is to use the aggregate perceptual score as an initial filter, then verify the result with a blind human comparison before naming a model. The dataset provides the scale for that screening, while the blind test supplies the decision that a no-reference score alone cannot make.
The historical corpus adds useful context, but not a direct ground-truth color benchmark. The grounding sources describe colorization testing conceptually, but do not provide any verified percentage of hand-colored film output or a named historical lab for this workflow. A period colorization corpus would be a print-dye proxy at best: it records how colors appeared in surviving release prints, not the scene reflectance that a modern model is trying to infer. Before using any historical hand-colored footage as a benchmark, verify the share of material that was actually hand colored and the production lab involved, and even then treat it as evidence of period colorization practice rather than as a photograph of the original scene.
No published head-to-head blind comparison between diffusion and palette-based colorization models appears in the grounding sources, so the model-family distinction here must come from your own Test 2 run. Conduct the blind human-choice test on outputs from each model family you are considering, using the same pairwise question for every pairing. That test is what can support a conditional reading — a diffusion default where raters prefer it, a palette-based alternative where a well-defined historical color convention exists — rather than assuming any family-level preference in advance.
For evaluating a new tool, preserve that order of operations. First calculate the perceptual score on the full image set. Next, present the outputs to raters without identifying the model, because revealing the method can bias judgments toward fashionable AI outputs or historically plausible palettes. Finally, compare the blind-choice results and accept the leading model only if its preference margin is large enough to be operationally meaningful. A narrow lead suggests that the score should be treated as unresolved for that image class; a clear lead provides the evidence needed to select the model.
Options compared: which model wins
For 2026 model selection, the comparison table should use the combined MCIQA-2K perceptual score only as an initial filter. The decisive column is the blind human win rate, and the first eligibility gate is whether the image’s skin-mask coverage exceeds 5%, per the stated conditional finding. The grounding sources do not document this threshold as part of any published workflow, so treat it as an evaluation convention to apply consistently and to verify against your own data before relying on it. A model that fails the gate is not eligible to win via the perceptual score, regardless of its human-choice result.
| Model | Type | Perceptual Score (MCIQA-2K) | Blind Human Win Rate | Skin-Mask ≥5% Pass | Verdict |
|---|---|---|---|---|---|
| Eligible diffusion model with the highest blind-choice result | Diffusion-based | Reported for comparison | Highest among eligible models | Pass | Winner if it leads the runner-up by at least 10 percentage points |
| Perceptual-score leader | Type must be confirmed | Highest score | Compare against diffusion candidates | Pass | Not accepted if it loses blind choice by more than 10 percentage points |
| Blind-choice runner-up | Type must be confirmed | Reported for comparison | Second-highest eligible result | Pass | Reference point for the acceptance margin |
| Any model with insufficient skin-mask coverage | Diffusion-based or other | Recorded but non-decisive | Recorded but non-decisive | Fail | Do not select |
The winner is therefore the diffusion-based model with the highest blind human win rate among candidates that pass the skin-mask coverage gate. This is a conditional selection rule, not a claim that every diffusion model should be preferred. The comparison must confirm the model’s type, record its MCIQA-2K perceptual score, verify that coverage clears 5%, and then compare its blind-choice result with the eligible runner-up.
Apply the acceptance rule literally: select the winner only if it beats the runner-up in blind human choice by at least 10 percentage points. If it leads by less than that margin, the result is inconclusive and no model qualifies for selection. If it leads clearly, record the diffusion-based model as the winner; do not substitute a perceptual-score leader merely because it ranked first in Test 1.
Conversely, a model that wins the perceptual score but loses blind human choice by more than 10 percentage points should be treated as overfit to the metric, consistent with the MCIQA-2K methodology’s distinction between no-reference quality measures and human perceptual judgments. The practical check is simple: preserve both results in the table, then let the blind-choice margin—not the metric score—determine the verdict.
Costs and numbers that matter
The practical cost of colorization depends on both credits and output constraints, so check each service’s current terms before committing to a submission. Palette.fm lists a simple unit: one photo requires one credit. Its free plan provides colorization at no cost, but resizes the image to a maximum of 500×500 pixels and adds the Palette logo as a watermark. That limitation matters when the candidate result must preserve the source image’s resolution or be used without visible branding. Record both figures in the comparison ledger: one credit per photo on the paid path, and reduced resolution plus a watermark on the free path.
Colorize.cc separates preview processing from the full service. According to the service, free processing on the landing page produces a simplified-quality result; obtaining the full colorization requires credits. The landing-page snippet does not provide a verified credit amount, so do not estimate one or assume that the free preview has the same finish as the credited output. Before uploading a batch, check the current credit balance, the number of credits required for full processing, and whether repeated attempts consume credits. These are verification steps rather than assumptions about a fixed price.
Imgupscaler.ai advertises a free AI colorizer, but the available snippet states no specific credit cap or output ceiling. That makes it a potentially low-friction option for initial screening, not evidence that unlimited, full-resolution processing is guaranteed. Open the live site and verify the applicable upload limit, free-processing cap, output resolution, download conditions, watermark status, and whether commercial use is included. If those terms are not stated clearly, contact the service before committing an important archive or a large batch.
Keep these cost checks in one table beside the quality results. For each tool, enter the source pricing page, the date checked, the credit cost per photo, free-plan limits, maximum output dimensions, watermark conditions, and any batch restrictions. Use only published figures, and mark unknown values as “verify” rather than filling in guesses. Then compare the actual candidate costs: multiply the per-photo credit requirement by the number of photos to be processed, and calculate how many source pixels the free output would discard. Since the service terms can change, this ledger should be refreshed immediately before the 2026 evaluation run.
What the evidence does NOT establish
The MCIQA-2K result should not be read as proof that a colorization model reconstructs the scene’s true reflectance. The arXiv paper describes a no-reference quality dataset: human annotators judge perceptual qualities of colorized outputs, but the dataset does not provide a physically measured ground truth for the objects, surfaces, or illumination in each scene. “Correct” color therefore means acceptable according to human annotation, not color confirmed by spectrometry, calibrated photography, or a surviving original-color reference. For model selection, treat the perceptual score as evidence of human-perceived quality, and do not use it to claim that every generated hue is historically or physically accurate. The practical check is whether the image has a documented color reference; if it does not, report the score as a perceptual result rather than as proof of authenticity.
The reported 4% hand-colored Méliès corpus is also narrower than a general validation benchmark. As discussed by Colorize This, that figure concerns a print-dye proxy in which a small portion of the Méliès material had been hand colored. It cannot establish the accuracy of skin tones, sky colors, clothing pigments, or other scene colors in an unrelated film archive. A useful comparison would require matched frames from the same source, with the hand-colored print and the model output inspected under the same display conditions. Even then, dye reproduction and printing can introduce their own shifts. Do not transfer the 4% result to a different archive simply because both collections contain monochrome motion-picture images.
Seasonal and lighting variation creates a separate source of uncertainty. The Colorize This testing guidance notes that colorization tests using photographs taken at different times of year can reveal inconsistencies in AI interpretations. For a controlled check, compare images of the same person or location across seasons while checking whether changes in vegetation, daylight, and shadows are plausible. A model that assigns one plausible palette to a winter photograph and the same palette to a sunlit summer scene has not demonstrated seasonal understanding; it has demonstrated palette assignment. The review should distinguish an intentional period palette from an accidental repetition of a learned color distribution.
Most importantly, the predictive finding is conditional. The 2,000-image perceptual score is a better predictor of blind human choice only within the tested regime: diffusion-based models and images whose skin-mask coverage exceeds 5%. It does not establish that the score will rank period-palette tools, low-skin-coverage images, photographs without people, or non-diffusion systems in the same way. Before applying the result, verify the model family and measure skin-mask coverage. If either condition fails, the blind-choice test must carry the decision rather than the perceptual score alone.
Even when both conditions are met, the final acceptance rule remains deliberately strict: accept a model only if its blind-choice score exceeds the runner-up by at least 10 percentage points. The perceptual score can prioritize candidates, but it cannot substitute for that margin. This keeps a favorable aggregate ranking from overriding a visible failure in human comparison, especially when the sample may not represent the archive, season, lighting, or surface colors the tool will actually encounter.
3-test run on one photo
This section provides a copy-usable three-test worksheet for a single 1940s portrait. Begin with an image measuring 1200 × 1500 pixels. Run the skin-mask coverage check and record the result: 7% passes the minimum 5% gate. Because the threshold matters to the selection method, do not proceed when coverage is 5% or lower. Keep the diffusion-model condition attached to the final decision as well: the perceptual screen is being used in the specified 2026 selection workflow for diffusion-based models.
Checkpoint 1 is the perceptual-score test. Run the MCIQA-2K evaluation, described in its arXiv abstract as a no-reference dataset of 2,000 colorized images generated by five representative colorization models with human annotations. For this portrait, record one score for each of the three perceptual dimensions: color smearing, semantic color misalignment, and overall perceptual quality. Mark any dimension whose score falls in the bottom quartile of the corresponding 2,000-image distribution. Operationally, the bottom quartile begins at the 500th ranked image when the set is ordered from lowest to highest. A bottom-quartile flag means the output advances only if the next test clears the human-choice requirement; the perceptual screen alone is not the selection decision.
Checkpoint 2 is the blind human-choice test. Show the same portrait to 10 raters, with the candidate model outputs arranged side by side and without model names, vendor labels, or other identifying cues. Ask each rater to select the output they judge best. Record the vote count for every candidate, then convert each count to a percentage of the 10 votes. The acceptance rule is strict: the selected model must lead the runner-up by at least 10 percentage points. A 6–4 result clears the rule because the 20-point lead exceeds 10 percentage points; a 5–5 tie does not. Record the winner, runner-up, raw votes, percentages, and margin on the worksheet.
Checkpoint 3 is the model pick, and it follows the sequence rather than replacing it. Confirm that the skin-mask gate passed, that the perceptual scores were recorded and checked against the bottom quartile, and that the blind human choice produced a qualifying margin. Only then select the diffusion-based model that wins Checkpoint 2 by at least 10 percentage points over the runner-up. If no output meets that margin, make no selection and rerun the comparison with corrected candidates. This worksheet is designed to preserve the evidence trail while keeping the final decision tied to the specified human blind-choice threshold.
Decision rules: 5 if/then branches
If skin-mask coverage is 5% or lower, do not use the perceptual score to select a model. The grounding sources do not document any Kodachrome LUT or period-palette fallback for this case, so fall back entirely on the blind human-choice test: show raters the competing colorizations without model names and let the qualifying 10-percentage-point margin decide. Low skin-mask coverage means the evaluation contains too little visible skin for the score’s demonstrated relationship with blind preference to be dependable.
If skin-mask coverage exceeds 5% and the candidate is diffusion-based, use the perceptual score as the first selection signal. Run that test before asking blind human raters to choose, but do not make the final selection from the score alone. For a non-diffusion candidate, the score does not qualify as the better predictor of blind choice, so use the human comparison as the deciding evidence rather than presenting the score as equivalent evidence.
If the perceptual score and blind human choice disagree on more than 20% of the evaluation set, do not trust the score alone. Weight the blind human result more heavily, check whether disagreement clusters by scene type or skin tone, and review a sample of the conflicting images before making the call. This branch treats human choice as the stronger signal when the two tests diverge substantially, while retaining the score as a diagnostic rather than discarding it completely.
If the top model’s blind-human win-rate margin over the runner-up is less than 10 percentage points, treat the models as tied rather than declaring a winner. Re-run the blind test with more raters, ideally adding raters who were not exposed to the earlier results. Record both the revised margin and any changes in the ranking; until the required margin is reached, neither model qualifies for selection on this criterion.
If the top diffusion-based model has skin-mask coverage of at least 5% and its blind-human win-rate margin over the runner-up is at least 10 percentage points, select that model. Use the perceptual score only to establish the candidate’s strength before the human test, and use the human blind choice to confirm it. If the score and human choice disagree on no more than 20% of cases, the score may support the interpretation, but the qualifying human margin—not the score itself—is the final acceptance rule.
What to do next
| Step | Action | Why it matters |
|---|---|---|
| 1 | Run Test 1 on all 2,000 images in MCIQA-2K, covering outputs from the five representative models and the annotations for color smearing, semantic color misalignment, and the third perceptual dimension. | This establishes the perceptual-score baseline before any model is selected. |
| 2 | Rank the five models by their MCIQA-2K perceptual scores and record the leader, but treat that result as conditional rather than as the final choice. | The perceptual score is Test 1, not a standalone selection rule. |
| 3 | Run Test 2 as a human blind-choice comparison of the same model outputs, with participants unable to see model identities. | Blind choice measures human preference without knowledge of which model produced an image. |
| 4 | Identify the Test 2 winner and runner-up, then calculate the percentage-point gap between them. | The required decision margin is at least 10 percentage points over the runner-up. |
| 5 | Pick a model only if its Test 2 win is at least 10 percentage points over the runner-up; otherwise, do not accept a model and repeat the evaluation sequence. | Model selection is valid only after Tests 1 and 2 and only when the human-choice threshold is met. |
Frequently Asked Questions
What three perceptual dimensions does MCIQA-2K use to evaluate colorized photos?
MCIQA-2K evaluates color smearing, semantic color misalignment, and overall perceptual quality.
Why can’t the MCIQA-2K perceptual score be compared pixel by pixel with an original color photograph?
Because MCIQA-2K contains no pristine reference image, its score measures whether a colorization looks convincing without pixel-by-pixel comparison to an original color photograph.
How should candidate colorization models be tested for comparable perceptual scores?
Calculate the perceptual score for every candidate output on the same test set rather than comparing screenshots made under different conditions.
Does a strong perceptual score automatically determine the final colorization model?
No; the perceptual score is only a screening test, and a model advances when its aggregate result is competitive.
What margin over the runner-up is required to win the blind human-choice test?
The winning model must lead the runner-up by at least 10 percentage points.
What does MCIQA-2K’s dataset cover?
It is a no-reference quality dataset containing 2,000 colorized images produced by five representative colorization models and assessed by human annotators.
Quick answers
| What is the first test in the three-test colorization workflow? | The first test is the Perceptual Score. |
| What does the Perceptual Score assess? | Human annotators assessed each image across color smearing, semantic color misalignment, and overall perceptual quality. |
| Why is the Perceptual Score only a screening test? | A model can advance only when its aggregate perceptual result is competitive, but the score is not the final decision. |
| How should candidate outputs be evaluated on the same test set? | Calculate the perceptual score for every candidate output on the same test set rather than comparing screenshots made under different conditions. |
| What must a model achieve in the blind human-choice test? | A model must win the human-choice test by at least 10 percentage points over the runner-up. |
Also worth reading: Whether diffusion-based colorization improves or harms historical authenticity in archival photographs: a side-by-side evaluation of color fidelity, luminance preservation, and expert-rated historical accuracy: Whether diffusion-based colorization improves or · Diffusion Colorization: Conditioning Tradeoffs Beyond PSNR/SSIM: Diffusion Colorization: Conditioning Tradeoffs Beyond · Cyanotype colorization: DeOldify vs Stable Diffusion 2025, by provenance class: Cyanotype colorization: DeOldify vs Stable