Strip UTF-8 BOM at the start of each input file (#3657)
The UTF-8 BOM was stripped only at the start of the parser stream, but jq feeds all input files into a single parser, so a BOM at the start of the second and later files reached the parser as content and failed with a confusing "Invalid numeric literal" error. Strip it in the file readers instead, at the start of each input file including stdin, and in jv_load_file. This also strips the BOM for raw input, so -R and --rawfile no longer yield a leading U+FEFF, and a BOM at the start of a -f program file or an imported module is now accepted rather than reported as a syntax error. In exchange, "\uFEFF1" | fromjson is now a parse error, since a U+FEFF inside a string is content rather than an encoding marker, and the dedicated "Malformed BOM" error is gone.
This commit is contained in:
1 parent
17b87d25c7
commit
e001deb4cf
5 files changed
+34
-35
No files matched your search
@@ -42,13 +42,6 @@ null
|
||||
{"x":-1}
|
||||
{"x":1}
|
||||
|
||||
# The input line starts with a 0xFEFF (byte order mark) codepoint
|
||||
# No, there is no reason to have a byte order mark in UTF8 text.
|
||||
# But apparently people do, so jq shouldn't break on it.
|
||||
.
|
||||
"byte order mark"
|
||||
"byte order mark"
|
||||
|
||||
# We test escapes by matching them against Unicode codepoints
|
||||
# FIXME: more tests needed for weird unicode stuff (e.g. utf16 pairs)
|
||||
"Aa\r\n\t\b\f\u03bc"
|
||||
|
||||
Reference in new issue
Block a user