How to Catch AI-Hallucinated Specifications Before Publishing

An AI-assisted draft of a product page states that the sensor’s supply current is 1.2 µA typical. The datasheet says 1.2 µA in sleep mode and 180 µA when measuring. Nothing in the sentence is invented exactly; it’s just wrong in the way that matters. AI-hallucinated specifications are often like this: close to real values, attached to the wrong condition, or cited to a standard clause that sounds right. They pass a casual read because they’re fluent and plausible. This guide gives a step-by-step check that catches them before an engineer does.

Key Takeaways

• NIST’s Generative AI Profile (AI 600-1, 2024) names confabulation, confidently presenting erroneous or false content, as one of twelve risks of generative AI to manage.
• Fabricated references are common: Walters and Wilder (Scientific Reports, 2023) found 55% of citations generated by GPT-3.5 and 18% by GPT-4 were fabricated, and many real ones contained substantive errors.
• Long-form text mixes supported and unsupported facts: the FActScore study (Min et al., EMNLP 2023) found ChatGPT’s biographies scored 58% on factual precision when broken into atomic facts.
• The method that works is to break text into atomic claims and check each against a source, which is the same idea FActScore automates for evaluation.
• In technical content, the highest-risk claims are numbers with units, conditions, part numbers, pin assignments, standards clauses, compliance statements and citations.

The problem: errors that look like specifications

Specification errors from language models are rarely absurd. They’re usually one of a few patterns: a real value attached to the wrong condition (sleep current presented as operating current), a typical value presented as a guaranteed maximum, a unit off by a prefix, a part number with a suffix that doesn’t exist, a standard cited with a plausible clause number that says something else, or a compliance claim the product doesn’t have. Each is easy to miss in review because the sentence reads exactly like a correct one.

The risk grows when drafts are reviewed for style rather than facts, or when the reviewer is checking the whole draft at once instead of each claim individually.

What the research says

Confabulation is a named risk

NIST’s Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1, July 2024) lists confabulation among twelve risks, describing systems that generate and confidently present erroneous or false content. It’s treated as a property of the technology to manage with process controls, not as a rare malfunction.

Fabricated and erroneous citations

Walters and Wilder’s “Fabrication and errors in the bibliographic citations generated by ChatGPT” (Scientific Reports, 2023) asked GPT-3.5 and GPT-4 to write short literature reviews on 42 topics and examined 636 citations. Fabricated citations made up 55% of GPT-3.5’s and 18% of GPT-4’s. Among the real citations, 43% from GPT-3.5 and 24% from GPT-4 contained substantive errors. Newer models were much better, but not error-free, and the study is a reminder that a reference that looks complete can still be invented.

Checking atomic facts

Min and colleagues’ “FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation” (Proceedings of EMNLP 2023) broke model-generated biographies into atomic facts and checked each against a knowledge source. ChatGPT scored 58% on this measure of factual precision. The specific numbers concern biographies and the models of that time, but the method is the useful part: a paragraph can be mostly right and still contain several unsupported facts, and checking fact by fact finds them.

Hallucination across tasks

Ji and colleagues’ “Survey of Hallucination in Natural Language Generation” (ACM Computing Surveys, 2023) reviews hallucination across summarization, dialogue, question answering, data-to-text and other tasks, along with metrics and mitigation methods. Providing source material to the model reduces but doesn’t eliminate unsupported content, so verification is still needed when the model is given the datasheet.

Step 1: Gather the source documents first

Before checking anything, collect the current datasheet, reference manual, test reports, compliance certificates and any standards cited. If a claim can’t be traced to one of these, it doesn’t go in.

Step 2: Split the draft into atomic claims

Go through the draft and list every individual factual claim: each number, each part number, each “supports” or “complies with,” each citation. One sentence often holds three or four claims.

Step 3: Trace each claim to a page or table

For each claim, record the source document and the exact location: table number, page, section. If you can’t find it, mark it unverified.

Step 4: Check units, conditions and the column

Confirm the unit and prefix, the conditions (temperature, supply voltage, mode) and whether the value is minimum, typical or maximum. Most specification errors from AI drafts are condition or column errors, not invented numbers.

Step 5: Resolve every citation

Open every DOI, standard and URL. Confirm the document exists, the title and authors match, and the cited clause or page says what the draft claims.

Step 6: Watch for warning signs

Be extra careful with suspiciously round numbers, specifications better than the product category usually achieves, part numbers with unfamiliar suffixes, compliance claims not on a certificate, and confident statements with no source.

Step 7: Correct, remove or flag

Correct claims to match the source, remove claims that can’t be sourced, and flag anything that needs an engineer’s judgment rather than a document lookup.

Step 8: Keep a verification log

Store the claims table with the published piece: claim, source, location, status, who checked and when. It makes future updates faster and shows the review actually happened.

Common hallucinated specification patterns

Pattern	Example	How to catch it
Wrong condition	Sleep current given as operating current	Check the mode and conditions column
Wrong column	Typical value stated as guaranteed	Check min/typ/max
Unit or prefix error	µA vs mA, ppm vs %	Check units against the source table
Invented part suffix	An ordering code that isn’t in the ordering table	Match against the ordering information
Plausible but wrong clause	A standard cited by a clause that covers something else	Open the standard and read the clause
Unearned compliance	“Certified to…” without a certificate	Check the certificate list
Fabricated citation	A paper with real-sounding authors that doesn’t exist	Resolve the DOI and check the title

How to put it to work

If you have an hour

Take one AI-assisted page already published and run steps 2 to 5 on it. Count how many claims you can trace to a source, and fix any you can’t.

If you have a sprint

Create a claims-table template and make it mandatory for any AI-assisted technical content before review. Train writers to split drafts into atomic claims.

If you want it to last the product’s lifetime

Keep verification logs with every piece, re-verify when datasheets or certificates change, and track which error patterns appear most often so you can target the check.

The question to ask yourself: “Can I point to the page and table for every number in this draft?”

Frequently asked questions

How do you check AI-generated specifications?

Split the draft into individual claims, trace each to a page or table in the datasheet or other source, check units, conditions and min/typ/max, resolve every citation, and correct or remove anything that can’t be sourced.

How often does AI fabricate citations?

In one study, 55% of citations generated by GPT-3.5 and 18% by GPT-4 were fabricated, and many real citations had substantive errors. Newer models may do better, but every citation still needs checking.

Does giving the AI the datasheet prevent errors?

It reduces them but doesn’t eliminate them. Models can still attach values to the wrong conditions or columns, so verification against the source is still needed.

What are the most common AI specification errors?

Values attached to the wrong condition or column, unit or prefix errors, invented part number suffixes, plausible but wrong standard clauses and compliance claims without certificates.

Who should verify AI-drafted technical content?

Someone with access to the source documents and enough technical knowledge to judge conditions and units, ideally followed by review from an engineer who owns the product.

Related reading: AI in technical content production, how to read a sensor datasheet and writing technical articles engineers trust.

Sources

1. National Institute of Standards and Technology (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. https://doi.org/10.6028/NIST.AI.600-1
2. Walters, W. H., & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 13, 14045. https://doi.org/10.1038/s41598-023-41032-5
3. Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023). FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. Proceedings of EMNLP 2023, 12076–12100. https://doi.org/10.18653/v1/2023.emnlp-main.741
4. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., et al. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38. https://doi.org/10.1145/3571730