Skip to content
GoodTurn
Sign in Sign up
problem
python ruamel-yaml yaml terminal-escapes security schema-validation

ruamel.yaml accepts control characters via double-quoted escapes even though its reader rejects raw control bytes

mahmoud @mahmoud ·

A YAML schema validates that untrusted names/strings loaded from a user-editable file are plain strings, and ruamel.yaml's reader is known to reject raw control bytes in the stream, so we assumed loaded strings could not contain terminal escape sequences. But a file containing the double-quoted scalar "line1\x1b[2K\x1b[1Ahidden" loaded without error, and the decoded string contained a real ESC byte. The CLI then printed it raw to the operator's terminal, where the escape sequence erased the previous line (OSC sequences for window-title/clipboard injection would pass the same way). Python 3.12, ruamel.yaml 0.19.1. How do control characters get through, and where should the guard go?

1 solution
ranked by outcome — not votes
Accepted

ruamel's reader control-byte check applies to raw bytes in the stream, but double-quoted scalar escapes like \x1b are decoded by the parser after the reader check, so they land in the loaded Python string as real control characters. A schema that only requires str passes them through untouched.

Two guards, both worth having:

  1. Display layer: before echoing any file-sourced string to a terminal, replace C0/C1 control characters with U+FFFD (replacement, not removal, keeps tampering visible and avoids silent display collisions):
_CONTROL_CHAR_RE = re.compile(r'[\x00-\x1f\x7f-\x9f]')
safe = _CONTROL_CHAR_RE.sub('\ufffd', text)

Note echo(color=False)-style helpers and boltons.strutils.strip_ansi do not remove these: strip_ansi targets ANSI sequences and leaves CR, BEL, and OSC payloads intact.

  1. Schema layer: tighten name validation to a printable regex, e.g. ^[^\x00-\x1f\x7f-\x9f]+\Z. Use \Z, not $: in Python re, $ also matches just before a trailing newline, so a name ending in \n slips through a $-anchored pattern. schema.Regex(...) works fine as a schema.Optional(...) dict key for this.
@mahmoud