LLM-generated calculator/form specs use Python-like identifiers (builtins.income_monthly, builtins.expenses_annual, builtins.assets_cash_value) as numeric field default values instead of actual numbers. The validation pipeline rejects them at float() conversion, but the error is only caught late — after the LLM has generated the full spec and the edit-apply or XLSX roundtrip attempts to materialize it. The identifiers look like they reference a 'builtins' module that doesn't exist in the spec system; the LLM is hallucinating a typed-default API that would resolve user profile data at runtime. Observed across multiple LLM runs with different calculator topics (income, expenses, assets), suggesting the pattern is systematic rather than one-off. Six distinct builtins.* variants observed: income_monthly, income_annual, expenses_monthly, expenses_annual, assets_stock_value, assets_cash_value.
Validate default values are coercible to their declared field type BEFORE attempting the full edit-apply or XLSX roundtrip. Catch the pattern early with a pre-validation step:
import re
BUILTINS_RE = re.compile(r'^builtins\.[a-z_]+$')
def sanitize_default(field_name: str, declared_type: type, default_value):
if isinstance(default_value, str) and BUILTINS_RE.match(default_value):
# LLM hallucinated a runtime reference; replace with type-appropriate zero
return declared_type() # 0.0 for float, 0 for int, '' for str
try:
return declared_type(default_value)
except (ValueError, TypeError):
return declared_type()Apply this in the spec normalization step that runs before edit-apply AND before direct-build, so both code paths are covered. The broader principle: when an LLM generates structured specs with typed fields, validate each field's default value against its declared type as a separate pass before any downstream processing. LLMs systematically hallucinate 'smart default' APIs that reference user context — treating these as validation errors with safe fallbacks is more robust than prompt engineering alone.