A very large program can cause these leaks:
==758838== 7,820 (16 direct, 7,804 indirect) bytes in 2 blocks are definitely lost in loss record 17 of 28
==758838== at 0x4848A23: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==758838== by 0x125D30: jv_mem_calloc (jv_alloc.c:153)
==758838== by 0x162ADE: compile (compile.c:1286)
==758838== by 0x162D4B: compile (compile.c:1304)
==758838== by 0x163697: block_compile (compile.c:1381)
==758838== by 0x11B5CA: jq_compile_args (execute.c:1245)
==758838== by 0x115E20: main (main.c:691)
==758838==
==758838== 1,674,694 (103,576 direct, 1,571,118 indirect) bytes in 1,177 blocks are definitely lost in loss record 28 of 28
==758838== at 0x4843839: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==758838== by 0x125CD0: jv_mem_alloc (jv_alloc.c:141)
==758838== by 0x162B19: compile (compile.c:1289)
==758838== by 0x163697: block_compile (compile.c:1381)
==758838== by 0x11B5CA: jq_compile_args (execute.c:1245)
==758838== by 0x115E20: main (main.c:691)
This commit fixes that.
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=61349
Close#1885, #2140, #2011, #2220, #2485, #2073
Rename the FORK_OPT opcode to TRY_BEGIN, add a TRY_END opcode, and wrap
errors when raising through a TRY_END so that they will not be caught by
the matching TRY_BEGIN.
Now a `try exp catch handler` expression generates code like:
TRY_BEGIN handler
<exp>
TRY_END
JUMP past_handler
handler: <handler>
past_handler:
...
On backtrack through TRY_BEGIN it just backtracks.
If anything past the whole thing raises when <exp> produced a value,
then the TRY_END will catch the error, wrap it in another, and
backtrack. The TRY_BEGIN will see a wrapped error and then it will
unwrap and re-raise the error.
If <exp> raises, then TRY_BEGIN will catch the error and jump to the
handler, but the TRY_BEGIN will not stack_save() in that case, so on
raise/backtrack the TRY_BEGIN will not execute again (nor will the
TRY_END).
We need to be careful to not grab an extra reference when mutating data
structures because that means we make extra copies. Doing that every
time in `_modify` is really painful, as that function implements `|=`
and all modify-assign operators.
`jv_setpath()` also grabs additional references, so this is not the only
fix needed for the modify-assign operators to not be accidentally
quadratic.
We have to use `LOADVN` in order to make this fix possible, so we should
really byte-code `_modify` rather than jq-code it. However, as a
stop-gap to make this fix easier, I'm adding syntax for referring to a
`$binding` such that you get `LOADVN` instead of `LOADV`.
This syntax is not meant to be used outside jq's internals! Therefore
it is not documented. I promise to break it later, so don't use it!
TBD, but later:
- Optimize `_modify` for the case where `update` outputs more than one
value.
This is trivial: add a `setpath($p; null)` in the middle of `_modify`
before calling `update`. But unfortunately the VM retains a
reference to `value_at_path` for path expression checking, and fixing
that will require more work.
We need to be careful to not grab an extra reference when mutating
data structures because that means we make extra copies. Doing that
every time in `jv_setpath()` is really painful, as that function is used
in `_modify/2`, which is used in `|=` and all modify-assign operators.
`_modify` also grabs additional references, so this is not the only fix
needed for the modify-assign operators to not be accidentally quadratic.
Expressions of the form `path(EXPR) | select(GENERATOR)`, where `EXPR`
is a path expression and `GENERATOR` is a generator conditional
expression (e.g., `has("a"), has("b")`) cause an assertion if the
jq_state VM is torn down too soon. That assert() was only correct if
assuming that the conditional is not a generator.
If the conditional is generator, then what we see is that when
backtracking a SUBEXP_END is executed without a corresponding
SUBEXP_BEGIN because the entire conditional is bracketed with
SUBEXP_BEGIN and SUBEXP_END, and since it's resumed in the middle, in
between the brackets.
Rather than assert that the jq->path_len being restored has some
particular value, we can simply re-compute it from the restored
jq->path.
Users are often surprised by the requirement to parenthesize any
non-trivial object key expressions in object constructors. E.g.,
{"a"+"b":1}. This commit adds one more kind of key expression besides
literals and idents: variable references.
A common use case for this is jq programs as JSON templates to fill in
with variables computed from inputs or passed in on the command-line.
E.g., {some_key:$value}. Now users can also use, e.g., {$key:$value}.
This and the restrictions on key and value expressions in object
constructors are now clarified a bit in the documentation.
This reverts commit e24af3c78e.
While the semantics are desirable, there is no way to implement them
efficiently. The reason is that in order to handle backtracking (empty)
from the state update expression, we have to retain a reference to the
reduction state value in order to restore it upon backtracking.
Retaining a reference to the reduction state kills performance by
causing lots of additional memory allocations and garbage because the
input to the update expression will always have at least two references,
thus no changes to it can be done in-place, and all changes end up being
CoW changes.
Avoiding this is the very reason for the LOADVN instruction (leaving
`null` in the variable loaded from).
strptime() on OS X and *BSDs (reputedly) does not set tm_wday and
tm_yday unless corresponding %U and %j format specifiers were used.
That can be... surprising when one parsed year, month, and day anyways.
Glibc's strptime() conveniently sets tm_wday and tm_yday in those cases,
but OS X's does not, ignoring them completely.
This commit makes jq compute those where possible, though the day of
week computation may be wrong for dates before 1900-03-01 or after
2099-12-31.
OS X (and *BSD) strptime() does not set tm_wday nor tm_yday unless
corresponding format options are used. That means we must call timegm()
to set them.
The conditional expression in if-then-elif-else-end cannot contribute to
path expressions because it doesn't change the input to any of the then/
elif/else expressions. These must be generated via gen_subexp().
See also #1366.
It needs to be possible to do something like
getpath($paths[]) += 1
meaning: increment all the paths in . that are listed in $paths[].
In order to do this getpath() needs to update the jq->path and
jq->value_at_path as necessary.
Rather than testing that a constant expression produces so many
instructions, test that a variety of of such expressions produce the
same number of instructions. This will make future changes in the
compiler less likely to break this test.
Nicolas Williams