A format without %d left the day of the month at 0, which mktime and
todate read as the last day of the previous month, and the weekday was
wrong for dates in January and February of 1900 and 2100.
Start parsing from the 1st, so a format without a day now gives the
first of the month, and a time-only format no longer outputs the 8 and
367 placeholders for the weekday and day of the year. This also fixes
the weekday of dates in January and February of 1900 and 2100.
Both were defined as explode | map(...) | implode, which took 4 seconds
on a 10 MB string, about 90 times as long as parsing it.
Change the case of A-Z or a-z a byte at a time in C, copying multibyte
UTF-8 sequences unchanged, so the output stays the same and the 10 MB
case now takes about 60 ms including parsing. On a non-string input the
error now names the function, "ascii_downcase input must be a string",
instead of "explode input must be a string".
The manual says halt_error prints its input with no decoration, not
even a newline, but non-string values such as {"a":"xyz"} were printed
with a trailing newline.
Dump non-string values to JSON and print them the same way as strings,
so {"a":"xyz"} | halt_error(1) now writes exactly {"a":"xyz"} to stderr.
The tutorial is where most people meet jq, but it was missing key
filters like select and abbreviated commands in ways that are hard for
beginners to follow.
Grow the page from 7 examples to 24, newly covering -r and -c, chained
paths and slicing, map, sort_by, group_by, select, test, to_entries,
from_entries, with_entries, string interpolation and @tsv. The sample
data is pinned with the API's until parameter and the outputs avoid
GitHub account data, so running the commands gives exactly the output
shown.
The manual's "Defining Functions" section shows only one-argument
functions, so it never says how to pass more than one. This adds a short
paragraph: arguments are separated by `;` in both `def f(a; b):` and
`f(.x; .y)`, and `f(.x, .y)` instead calls a one-argument `f` with the
filter `.x, .y`. It also adds an example for each. `tests/man.test` and
`jq.1.prebuilt` are regenerated.
Fixes#2163
The manual said `a op= b` is equivalent to `a |= . op b`. It is actually
equivalent to `(b as $b | a |= . op $b)`: the right-hand side gets `.`
as input, not the value being updated, and each value it outputs gives a
separate result.
This updates the description and adds two examples, `.a += .b` and `.a
+= (1,2)`. `tests/man.test` and `jq.1.prebuilt` are regenerated.
Fixes#2119, fixes#2407
Nothing tested that jq --run-tests exits with status 1 when the
tests fail, so a failure could have been reported as a pass.
Add shell tests with incorrect, missing and extra output that check
both the exit status and the reported message, such as "Insufficient
results", along with a passing control. Under Valgrind, memory errors
exit with status 99 so they are not mistaken for a test failure.
The manual recommended backslash-escaped double quotes inside the jq
program for all PowerShell versions, but since PowerShell 7.3 native
command arguments are passed with standard quoting by default, so the
backslashes reach jq literally.
Describe the quoting rules separately for Unix shells and PowerShell
7.3+, for older PowerShell and Windows PowerShell, and for cmd.exe, and
update the examples to match.
Since jq 1.8.0, negating a number literal computed 0 - x, so -0 and
0 | -. printed 0 while (0+0) | -. printed -0.
Flip the sign bit with decNumberCopyNegate() instead, so -0 prints -0
again as it did in jq 1.7, and -(-0) is still 0.
jvp_utf8_next() had a fast path for ASCII, but it is not inlined into
its callers, so every byte of a string cost a function call.
Move the end check and the ASCII case into a static inline function in
jv_unicode.h and keep the rest out of line. This speeds up parsing of
string-heavy input by up to 1.17x and string functions such as length,
explode and @html, with the same output for every input.
Enable colored output by default only when the TERM environment variable
is set to a value other than dumb, as git and grep do, so that users no
longer need to pass -M or set NO_COLOR on terminals that cannot display
escape sequences. Colored output is now disabled not only for TERM=dumb
but also when TERM is unset or empty. The -C option still enables colors
in all these cases. On Windows, colored output is still decided by
ANSICON and the console mode, and TERM is not checked.
These functions failed with "unknown system failure" whenever strftime()
returned 0 for a non-empty format, but 0 also means the format expands to
an empty string, as %p does in a locale without AM/PM.
Prepend a character to the format and trim it from the result, so that 0
can only mean the result is too long. Now 0 | strflocaltime("%p") gives
"" in fr_FR.UTF-8, and a result that is still too long fails with
"result too long" error instead. The size limit is unchanged.
In --seq mode, a parse error was reported as "need RS to resync", but
parsing continued in the middle of the broken text, so fragments of it
were reported as more errors or output as values.
Set the parser state to wait for the next RS after resetting the parser,
instead of before, so that the rest of the broken text is skipped as
RFC 7464 recommends.
Printing a string wrote each character with a separate call, one
fwrite() per character, or one jv_string_append_buf() for tojson and
tostring, which made output of string-heavy data slow.
Scan ahead for the next byte that needs escaping and write the run of
bytes before it with a single call, without changing the output.
Printing strings is now significantly faster, most of all for long
strings with few characters to escape, and the same applies to tojson
and tostring. With --ascii-output, text that is almost all non-ASCII is
about the same speed, since nearly every byte is escaped anyway.
A call passing more than 4095 arguments at the start of a nested
function body aborted jq after reporting the compile errors, with
"Assertion failed: (nemb > 0 && sz > 0), function jv_mem_calloc" and
exit status 134.
Stop compiling a function before its bytecode is allocated when it has
compile errors, since the bytecode is only generated and used when
compilation succeeds. jq now prints the same compile errors followed by
"jq: 2 compile errors" and exits with status 3, as it does for other
compile errors, instead of crashing.
halt_error with a negative exit code did not exit with that code. The
code collided with the internal status values for null or false output
and for no output, so halt_error(-1) exited with 0, or with 1 under
--exit-status.
Keep the low 8 bits of a negative code, as exit(3) does on POSIX
systems, so halt_error(-1) now exits with 255 and halt_error(-2) with
254, the same with or without --exit-status. Under --exit-status this
replaces the previous absolute value, so halt_error(-2) used to exit
with 2 and halt_error(-255) with 255, and they now exit with 254 and 1.
Converting a double outside the int range to int is undefined behavior,
and has() and del() did it on array indices like infinite or 1e10, which
UBSan reports as "inf is outside the range of representable values of
type 'int'". Only NaN was guarded against.
Clamp the index to the int range in has(), as getting and setting array
elements already do, and drop out-of-range indices in del() before they
are converted. Such indices can never refer to an element, so has()
returns false and del() leaves the array unchanged, the same as for
indices past the end of the array. Valid indices deleted together with
out-of-range ones, as in del(.[0, 1e10]), are still deleted.
Add a BSD workflow with FreeBSD and NetBSD jobs that run the same build
and tests as the linux job, each inside a VM booted under QEMU on an
ubuntu-latest runner. The tests/shtest $NO_COLOR pty checks are now
skipped on NetBSD, as they already are on SunOS.
The UTF-8 BOM was stripped only at the start of the parser stream, but jq
feeds all input files into a single parser, so a BOM at the start of the
second and later files reached the parser as content and failed with a
confusing "Invalid numeric literal" error. Strip it in the file readers
instead, at the start of each input file including stdin, and in
jv_load_file.
This also strips the BOM for raw input, so -R and --rawfile no longer
yield a leading U+FEFF, and a BOM at the start of a -f program file or an
imported module is now accepted rather than reported as a syntax error.
In exchange, "\uFEFF1" | fromjson is now a parse error, since a U+FEFF
inside a string is content rather than an encoding marker, and the
dedicated "Malformed BOM" error is gone.
The date in the .TH line of jq.1.prebuilt changed whenever the manual was
regenerated in a new month, causing merge conflicts between pull requests
that touch the manual, and it did not reflect the release date anyway.
Generate the committed man page without a date.
Stamp the date into jq.1 at build time instead, taking it from the current
commit, or from SOURCE_DATE_EPOCH where that is set, so that release
tarballs and distribution builds still carry one and the behaviour added
in #3514 is kept. Write it in UTC and in ISO 8601 rather than the previous
"%B %Y", for which mandoc reports "cannot parse date, using it verbatim".
`fromdateiso8601` was `strptime("%Y-%m-%dT%H:%M:%SZ") | mktime`, which
accepts only one of the many representations ISO 8601 permits. strptime
cannot be extended to cover the rest portably, since it has no specifier
for fractional seconds and glibc discards `%G` and `%V`.
Replace it with a parser in src/iso8601.c that handles basic and extended
representations, ordinal and week dates, fractional seconds, offsets and
date-only input. It uses integer arithmetic instead of `time_t`, so the
results no longer depend on the platform.
This is stricter than before. Impossible dates like `2015-02-30`, which
`mktime` used to normalize into March, and unpadded components like
`2015-3-5T6:00:00Z` are now errors. So is any trailing input, including
whitespace. Leading whitespace is rejected too; before, glibc skipped it
and macOS did not. A leap second now maps to the second before it rather
than the next minute. The error message has changed, and the result can
now be non-integer.
The lexer returned a single ALTERNATION token for `?//`, so any `?`
followed by `//` was consumed by it, and `.name?//"unknown"` failed
to parse even though they read as the optional operator followed by
the alternative operator.
Drop the lexer rule and spell the destructuring alternative operator
as `'?' "//"` instead. So `x ?// y` now parses as `x? // y`, while
`. as [$a] ?// [$b] | ...` keeps its meaning. Note that `?` and `//`
may now be separated by whitespaces inside a pattern.
The push trigger had no branch filter, so pull requests from branches
in this repository (e.g. Dependabot) ran the workflow twice, once for
push and once for pull_request.
join/1 folded each element as `(acc + $x) + elem`, with the element
branch ending in `.//""`. Both leave a second live reference to the
accumulator: the nested + feeds the same `.` to both operands, and the
`//` leaves a forkpoint below it on the stack. `jvp_string_append`
therefore cannot grow the accumulator in place, and copies the whole
string once per element.
Pipe `acc + $x` into `. + elem` and drop the `//`, since adding null to
a string is already a no-op. join/1 is now linear: joining 8000
strings of 1000 chars goes from 6.5s to 0.06s, with output and error
messages unchanged.
glibc's arc4random() aborts with "Fatal glibc error: cannot get entropy
for arc4random" when the kernel lacks getrandom(2), so jq died on
startup on older kernels once 0c7d133 began seeding the hash on every
run. getentropy() merely returns -1/ENOSYS there, so prefer it, and
route its failure through the existing `/dev/urandom` path - which works
on precisely those kernels -- instead of a guessable `getpid() ^ time()`.
`reduce f as $x (init; update)` evaluated `f` against a stack slot
that forkpoints created while evaluating `init` restore. Both the
`DUPN` opening the reduction loop and the `LOADVN` retrieving the
result pop that slot destructively, nulling it out, so an `init`
that backtracks (e.g. `0, 0`) made `f` see `null` and fail with
"Cannot iterate over null".
Duplicate the input before the `FORK` instead of inside the loop,
and drop the leftover copy with `POP` once the loop is exhausted.
The slot left live at the `FORK` is then a block pushed after
`init` completed, which no earlier forkpoint can reach, so nulling
it is harmless. This is the same approach `gen_foreach` has used
since c70ae1a.
`DUPN` had no other user, so remove the opcode.
Define `from_entries` as `add(.[] | f) // {}` instead of
`map(f) | add // {}`. Since `add(f)` is `reduce f as $x (null; . + $x)`,
both fold the same values in the same order from the same seed, so the
only difference is that the array of single-key objects built by `map`
is never materialized.
Measured with both the generator and the result suppressed, so that only
the `from_entries` work differs: 6-15% less user time across input shapes,
which is 10-24% of what `from_entries` itself costs. For inputs above
roughly a thousand entries, peak RSS drops by 37%, around 80% of the
memory `from_entries` adds on top of its input; a 1,000,000 entry array
goes from 1064 MiB to 672 MiB.
block_mark_referenced walks the joined block backwards and skips the
body of any definition not yet marked referenced, so a library must
precede its importers. f58787c41 registered a library before walking
its dependencies, to have somewhere to keep the `loading` flag, which
reversed that order and made block_drop_unreferenced free definitions
that are still referenced through bound_by.
Keep the libraries being loaded on a stack threaded through the
load_library frames instead, so registration can move back after the
dependency walk.
jq dumps JSON to the Windows console as UTF-16 with WriteConsoleW(),
but priv_fwrite() still writes the raw UTF-8 bytes with WriteFile(),
which the console decodes with its output code page. So `jq .k` prints
non-ASCII correctly while `jq -r .k` prints mojibake, and the same goes
for -j, --raw-output0, --seq, halt_error and the `stderr` builtin.
The tty flag is computed once, from stdout, and passed to writes aimed
at stderr too, so ask the handle instead: with stdout on a console and
stderr redirected, WriteConsoleW() failed on the file handle and the
output was dropped.
One byte-level change: with stdout on a console and stderr redirected,
halt_error and the `stderr` builtin now go through the CRT, so bare LF
becomes CRLF and an existing CRLF becomes CR CRLF. Use -b for
byte-exact output on that stream.
The ownership comment in `jv.h` currently says every `jv_*` function
except `jv_copy` consumes its input. Several accessors and
object-iteration functions borrow their arguments instead, which makes
that guidance misleading for callers and language bindings.
Document the borrowing exceptions beside their declarations, including
numeric helpers and `jv_show`. Clarify that returned `jv` values are
owned, while the pointers returned by `jv_string_value` and
`jv_number_get_literal` belong to their input values. This changes
comments only.
The whole seconds were truncated toward zero while the fractional part
was measured from the floor, so any negative non-integral timestamp was
rendered one second late; gmtime was not monotonic across the epoch.
Only the latest release stays visible in the binary, checksum and
signature, and source tarball lists; older versions move into
collapsible `<details>` sections. Condense the repetitive per-version
checksum and signature sentences into a single list.
State explicitly in the README and download page that the
platform-specific prebuilt files are standalone executables, not
archives. Include the rename and executable-permission step for macOS
and Linux in the README.
Addresses #2956. Source archives are not described as executables.
Validation: the official `docs/build_website.py` completed and the
generated download page contains the new guidance. Used a Python venv
with the site's required rendering packages; restored the Windows
checkout's manual symlink in a temporary build copy only. jq itself was
not built or tested for this documentation-only change.
Co-authored-by: Hugohong258 <282228220+Hugohong258@users.noreply.github.com>
Jay Hemnani