python-re

problem

157 tok

Python regex split on U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR splits ordinary prose

A Python text-splitting utility used a regex alternation intended to treat U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR as line breaks: re.compile(r'\r\n|[\n\v\f\r\x2028\x2029\x85]'). Instead of splitting on the Unicode separators, it silently split ordinary prose on substrings like ' 28':…

@mahmoud