We need to be careful to not grab an extra reference when mutating data
structures because that means we make extra copies. Doing that every
time in `_modify` is really painful, as that function implements `|=`
and all modify-assign operators.
`jv_setpath()` also grabs additional references, so this is not the only
fix needed for the modify-assign operators to not be accidentally
quadratic.
We have to use `LOADVN` in order to make this fix possible, so we should
really byte-code `_modify` rather than jq-code it. However, as a
stop-gap to make this fix easier, I'm adding syntax for referring to a
`$binding` such that you get `LOADVN` instead of `LOADV`.
This syntax is not meant to be used outside jq's internals! Therefore
it is not documented. I promise to break it later, so don't use it!
TBD, but later:
- Optimize `_modify` for the case where `update` outputs more than one
value.
This is trivial: add a `setpath($p; null)` in the middle of `_modify`
before calling `update`. But unfortunately the VM retains a
reference to `value_at_path` for path expression checking, and fixing
that will require more work.
We need to be careful to not grab an extra reference when mutating
data structures because that means we make extra copies. Doing that
every time in `jv_setpath()` is really painful, as that function is used
in `_modify/2`, which is used in `|=` and all modify-assign operators.
`_modify` also grabs additional references, so this is not the only fix
needed for the modify-assign operators to not be accidentally quadratic.
Expressions of the form `path(EXPR) | select(GENERATOR)`, where `EXPR`
is a path expression and `GENERATOR` is a generator conditional
expression (e.g., `has("a"), has("b")`) cause an assertion if the
jq_state VM is torn down too soon. That assert() was only correct if
assuming that the conditional is not a generator.
If the conditional is generator, then what we see is that when
backtracking a SUBEXP_END is executed without a corresponding
SUBEXP_BEGIN because the entire conditional is bracketed with
SUBEXP_BEGIN and SUBEXP_END, and since it's resumed in the middle, in
between the brackets.
Rather than assert that the jq->path_len being restored has some
particular value, we can simply re-compute it from the restored
jq->path.
Users are often surprised by the requirement to parenthesize any
non-trivial object key expressions in object constructors. E.g.,
{"a"+"b":1}. This commit adds one more kind of key expression besides
literals and idents: variable references.
A common use case for this is jq programs as JSON templates to fill in
with variables computed from inputs or passed in on the command-line.
E.g., {some_key:$value}. Now users can also use, e.g., {$key:$value}.
This and the restrictions on key and value expressions in object
constructors are now clarified a bit in the documentation.
This reverts commit e24af3c78e.
While the semantics are desirable, there is no way to implement them
efficiently. The reason is that in order to handle backtracking (empty)
from the state update expression, we have to retain a reference to the
reduction state value in order to restore it upon backtracking.
Retaining a reference to the reduction state kills performance by
causing lots of additional memory allocations and garbage because the
input to the update expression will always have at least two references,
thus no changes to it can be done in-place, and all changes end up being
CoW changes.
Avoiding this is the very reason for the LOADVN instruction (leaving
`null` in the variable loaded from).
strptime() on OS X and *BSDs (reputedly) does not set tm_wday and
tm_yday unless corresponding %U and %j format specifiers were used.
That can be... surprising when one parsed year, month, and day anyways.
Glibc's strptime() conveniently sets tm_wday and tm_yday in those cases,
but OS X's does not, ignoring them completely.
This commit makes jq compute those where possible, though the day of
week computation may be wrong for dates before 1900-03-01 or after
2099-12-31.
OS X (and *BSD) strptime() does not set tm_wday nor tm_yday unless
corresponding format options are used. That means we must call timegm()
to set them.
The conditional expression in if-then-elif-else-end cannot contribute to
path expressions because it doesn't change the input to any of the then/
elif/else expressions. These must be generated via gen_subexp().
See also #1366.
It needs to be possible to do something like
getpath($paths[]) += 1
meaning: increment all the paths in . that are listed in $paths[].
In order to do this getpath() needs to update the jq->path and
jq->value_at_path as necessary.
Rather than testing that a constant expression produces so many
instructions, test that a variety of of such expressions produce the
same number of instructions. This will make future changes in the
compiler less likely to break this test.
This allows one to force the use of jq's builtin copy of Oniguruma
(builtin as a git submodule).
We still need to fix make dist to include that copy of Oniguruma.
Now that #1313 is fixed, |= no longer outputs null when the RHS update
expression outputs empty.
When a user wants to keep the current value of the LHS they would have
the RHS update expression output `.`, so having `empty` achieve the same
thing would be redundant. The obvious thing to do is to delete the LHS
when the RHS update outputs `empty` (i.e., doesn't output any values).
It's reasonable to think that existing programs won't be broken by this
change, because reduce and |= not handling empty well is clearly a bug.
(Though it's possible that some programs were using empty to quickly
terminate reduce or |=, it's not likely. They should use label/break
instead.)
Prior to this change |= would use the _last_ value output by the RHS
update expression. With this change |= will use the _first_ value
instead. This change _is_ a minor backwards-incompatible change. It
may or may not be acceptable; we'll see. It is a useful change in that
it makes |= faster when the update expression produces multiple values.
first(g) was extracting two values, which, if g is slow, made first(g)
slow. And if the second extraction were to throw an error, then
first(g) would throw that error, which is clearly not the right thing to
do. Besides that, first(g) was allocating garbage, which it no longer
does.
Use %%FAIL IGNORE MSG for error messages that differ on Windows, and
move the sort_by/group_by tests that fail qhen qsort() is not stable.
Now only tests/sorttest and tests/shtest are disabled on Windows.
Nicolas Williams