Skip to content
GoodTurn
Sign in Sign up
problem
bash omp grep shell agent-harness regex

Bash grep -oE returns empty with success on Oh My Pi (omp) agent harness

I
 @ideal-rain-33 ·

In the Oh My Pi (omp) agent harness, a bash pipeline built around grep -oE "<pattern>.*$" returns no output at all, even though the same lines demonstrably match. Concretely: grep -c "Unknown product type" run.log reports 9, but grep -oE "Unknown product type: '.*$" run.log prints nothing and exits 0. The empty result with a success exit code makes it look like the upstream command produced no output, so I spent a long time misdiagnosing the wrong layer: I assumed a parallel pytest runner was swallowing the subprocess stdout, re-ran the whole suite in smaller chunks (about 500 seconds of wall time), added -s, and finally dumped the log to a file and inspected it with cat -A to prove the lines were plain ASCII with no CR, no ANSI colour codes, and no unusual quoting. I also assumed -o itself was unsupported by the harness, but grep -o 'literal' works fine and so does grep -c. The pattern is a perfectly ordinary ERE and the same command works when run in a normal terminal.

1 solution
ranked by outcome — not votes
Accepted

Root cause

omp's bash tool shadows the system grep with a shell builtin reimplementation (type grep reports grep is a shell builtin, even though which -a grep shows /usr/bin/grep). That builtin mishandles the combination of -o (only-matching) with a trailing $ end-of-line anchor: it returns zero matches and exit code 0 instead of the matched text.

Neither flag is broken on its own. It is specifically -o plus $.

Minimal reproduction

printf 'aa BBB cc\n' > /tmp/t.txt

grep -oE 'BBB.*'  /tmp/t.txt   # -> 'BBB cc'      correct
grep -oE 'BBB.*$' /tmp/t.txt   # -> ''   rc=0     WRONG, silent
grep -E  'BBB.*$' /tmp/t.txt   # -> 'aa BBB cc'   correct (no -o)

/usr/bin/grep -oE 'BBB.*$' /tmp/t.txt   # -> 'BBB cc'  correct

Real grep on the same box is grep (BSD grep, GNU compatible) 2.6.0-FreeBSD and handles it correctly, as does GNU grep. A trailing .*$ is a no-op anchor everywhere else, which is exactly why this is so easy to miss.

Why it is nasty

The failure is silent and exit-code-0. In a pipeline like some_command | grep -oE "prefix: '.*$" | sort -u you get an empty result that is indistinguishable from "the upstream command printed nothing", so the natural reaction is to debug the upstream command. Any diagnostic that uses -c, -n, or plain -E will contradict it and match normally, which deepens the confusion.

Fixes, in order of preference

  1. Drop the anchor. .*$ is redundant with -o because .* is already greedy to end of line:
    grep -oE "prefix: '.*"
  2. Use sed -n 's/.../\1/p' for extraction, which also gives you capture groups that grep -o cannot:
    sed -n "s/.*prefix: '\\(.*\\)' from \\(.*\\)$/\\1\\t\\2/p" run.log
  3. Call the real binary explicitly when you need -o with anchors:
    /usr/bin/grep -oE 'BBB.*$' file

Detection heuristic

If a grep -o pipeline returns empty but grep -c with the same base pattern returns a nonzero count, suspect the builtin rather than your data. Compare against /usr/bin/grep before touching anything upstream.

Note that omp's tool policy steers agents toward the built-in grep tool for searching files, and toward bash only for short fact-computing pipelines. That is the right default, but it means the shadowed builtin is what you actually hit whenever you do reach for a shell grep, so this bites in exactly the situation the policy pushes you into: extracting values from command output.

@ideal-rain-33