Strip UTF-8 BOM at the start of each input file (#3657)

The UTF-8 BOM was stripped only at the start of the parser stream, but jq
feeds all input files into a single parser, so a BOM at the start of the
second and later files reached the parser as content and failed with a
confusing "Invalid numeric literal" error. Strip it in the file readers
instead, at the start of each input file including stdin, and in
jv_load_file.

This also strips the BOM for raw input, so -R and --rawfile no longer
yield a leading U+FEFF, and a BOM at the start of a -f program file or an
imported module is now accepted rather than reported as a syntax error.
In exchange, "\uFEFF1" | fromjson is now a parse error, since a U+FEFF
inside a string is content rather than an encoding marker, and the
dedicated "Malformed BOM" error is gone.
This commit is contained in:
itchyny authored and GitHub committed 2026-10-08 23:01:36 +09:00
1 parent 17b87d25c7
commit e001deb4cf
5 files changed
+34 -35

No files matched your search

-7
View File
@@ -42,13 +42,6 @@ null
{"x":-1}
{"x":1}
# The input line starts with a 0xFEFF (byte order mark) codepoint
# No, there is no reason to have a byte order mark in UTF8 text.
# But apparently people do, so jq shouldn't break on it.
.
"byte order mark"
"byte order mark"
# We test escapes by matching them against Unicode codepoints
# FIXME: more tests needed for weird unicode stuff (e.g. utf16 pairs)
"Aa\r\n\t\b\f\u03bc"