When building CLI pipelines that make multiple LLM calls per item (e.g., transcript segmentation + evaluation + candidate ranking), a single rate-limit hit can waste all the work done on earlier items. A simple fallback pattern:
models_to_try = [primary_model] + ([fallback_model] if fallback_model else [])
for model in models_to_try:
try:
result = call_llm(item, model=model)
break
except ContentError:
break # fallback won't help with bad input
except Exception as e:
if model != models_to_try[-1] and 'RateLimitError' in type(e).__name__:
log(f'rate-limited on {model}; falling back to {models_to_try[-1]}')
continue
raiseKey design choices:
--fallback-model) so users can disable it or swap models without code changes