Commit Graph
1817 Commits
Author SHA1 Message Date
itchyny f28720f699 Increase the maximum parsing depth for parsing JSON to 10000 (#3328)
This is the same limit as encoding/json in Go and should be sufficient for most use cases.
2025-05-30 20:19:34 +09:00
Thalia Archibald 9ac6ddae22 Fix whitespace in number parsing (#3195)
* Handle input errors for --indent

Handle malformed and overflowing arguments. Also, forbid leading and
trailing spaces to match the behavior of tonumber from commit ce0e788
(improve tonumber/0 performance by parsing input as number literal,
2024-03-02).

* Do not skip leading whitespace in jvp_strtod

`jvp_strtod` skips leading whitespace, but its decnum counterpart
`decNumberFromString` (called within `jv_number_with_literal`) does not.
Those two are called interchangeably, so it leads to inconsistent
behavior depending on whether the decnum feature is enabled.
Additionally, `classify`, used in the token scanner, only considers
[ \t\n\r] to be whitespace, but `jvp_strtod` consumes the larger set
[ \t\n\v\f\r], so those extra characters are considered literals.

Changing this deviates from the behavior of `strdod` from <stdlib.h> and
is technically a breaking API change, since it is a public symbol.
2025-05-29 04:11:00 +09:00
itchyny aaace5432a Fix parser to allow binary operators for binding syntax (#3326)
Previously, the precedence of `as` binding was higher than the binary
operators. This results in unexpected behaviors, `1 + 2 as $x | -$x`
was `-1` instead of `-3`. The commit 92daac90f6 (PR #3053) changed
the behavior to emit syntax error on this case. Now, this commit allows
binary operators with `as` binding. This fixes #2446, and fixes #3284.
2025-05-23 21:25:30 +09:00
itchyny 4e3088f188 Fix behavior of --slurp --stream when input has no trailing newline character (#3279)
This commit fixes a strange behavior of the combination of `--slurp` and
`--stream` flags when the input has no trailing newline character.
2025-05-22 08:04:05 +09:00
Gabriel Marin 7d8c096e48 Add trimstr/1 function (#3319) 2025-05-21 18:46:04 +09:00
itchyny de21386681 Fix signed integer overflow in jvp_array_write and jvp_object_rehash
This commit fixes signed integer overflow and SEGV issues on growing
arrays and objects. The size of arrays and objects is now limited to
`536870912` (`0x20000000`). This fixes CVE-2024-23337 and fixes #3262.
2025-05-21 07:48:23 +09:00
itchyny 4e2ccc4c9b Fix and refactor scan-build workflow to run on the master branch (#3325) 2025-05-20 20:00:53 +09:00
itchyny b733dfaee7 Improve error messages to show problematic position in the filter (#3292)
This commit improves the parser and compiler error messages, making
it more easier to understand where in the filter the problem is.
Like modern compilers, jq will show the line in the filter where the
error occurred, will display squiggly line under the invalid token.
2025-05-18 16:36:32 +09:00
Brahm Lower c350566001 Adds "toboolean" builtin function for converting strings to booleans (#2098) 2025-05-18 16:31:10 +09:00
itchyny 92daac90f6 Fix unary operator precedence, allow some expressions as object values (#3053)
This commit improves the parser to fix some unintuitive results of the unary
operator `-` (e.g. `[-1 as $x | 1,$x]` is now `[1,-1]`, previously `[-1,-1]`).
Also, this commit now allows to use binary operator expressions, `reduce`,
`foreach`, `if`, and `try` expressions for object values.
2025-05-18 16:13:55 +09:00
itchyny b088e83a82 Refactor splits/2 using foreach syntax (remove private filter _nwise) (#3260)
This commit refactors `splits/2` implementation using `foreach` syntax
and reduces intermediate arrays. This refactoring removes an unnecessary
private (and undocumented) filter `_nwise` and closes #3148.
2025-05-18 08:27:44 +09:00
itchyny 96d19ca2ee Fix use of uninitialized value in check_literal (fix #3316) (#3324) 2025-05-18 08:06:54 +09:00
itchyny ecea2d29bd Update CI runners to Ubuntu 24.04 and macOS Sonoma (#3208) 2025-05-18 08:06:16 +09:00
itchyny eff9caf6c7 Disable shared libraries on macOS to avoid conflict with -all-static (#3261) 2025-05-17 04:57:17 +09:00
Steven Ihde 77d23e9a5f Test daylight time for strflocaltime (#3323)
This test case covers the behavior described in issue #1912, fixed by #3203.
2025-05-15 07:16:25 +09:00
Thomas Sibley 931a362f8b docs: Use a better input example in the streaming section (#3322)
The introduction says all the usage examples (e.g. of truncate_stream,
etc) in the section use [0,[1]] as input, but they actually use [1,[2]].
["a",["b"]] seems better than [1,[2]] as the string values won't be
confusable with the indexes in <path>.
2025-05-14 20:46:31 +09:00
itchyny a8f27cc4db Fix warning: a function definition without a prototype is deprecated (#3259) 2025-05-07 21:25:32 +09:00
dependabot[bot] 9f6bcdc321 build(deps): bump lxml from 5.3.1 to 5.4.0 in /docs (#3311)
Bumps [lxml](https://github.com/lxml/lxml) from 5.3.1 to 5.4.0.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-5.3.1...lxml-5.4.0)

---
updated-dependencies:
- dependency-name: lxml
  dependency-version: 5.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-05-07 11:01:19 +00:00
dependabot[bot] 622e2d1d7f build(deps): bump markdown from 3.7 to 3.8 in /docs (#3312)
Bumps [markdown](https://github.com/Python-Markdown/markdown) from 3.7 to 3.8.
- [Release notes](https://github.com/Python-Markdown/markdown/releases)
- [Changelog](https://github.com/Python-Markdown/markdown/blob/master/docs/changelog.md)
- [Commits](https://github.com/Python-Markdown/markdown/compare/3.7...3.8)

---
updated-dependencies:
- dependency-name: markdown
  dependency-version: '3.8'
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-05-07 11:01:11 +00:00
itchyny 43506372b5 Update builtin oniguruma to 6.9.10 to suppress clang warnings (#3258) 2025-05-07 10:50:42 +00:00
itchyny abde27d0f5 Revert "Move closer to the standard autotools idiom. (#3187)" (#3255)
This partially reverts commit d8463c2549.
Revert reason: version string should be updated more often.
2025-05-07 17:26:19 +09:00
Emanuele Torre aea8efaf0b jv_unique: don't leak non-unique elements that are not returned (#3304) 2025-03-29 07:16:46 +09:00
dependabot[bot] 947fcbbb1f build(deps): bump jsonschema from 4.18.4 to 4.23.0 in /docs (#3302)
Bumps [jsonschema](https://github.com/python-jsonschema/jsonschema) from 4.18.4 to 4.23.0.
- [Release notes](https://github.com/python-jsonschema/jsonschema/releases)
- [Changelog](https://github.com/python-jsonschema/jsonschema/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/python-jsonschema/jsonschema/compare/v4.18.4...v4.23.0)

---
updated-dependencies:
- dependency-name: jsonschema
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-28 09:57:51 +09:00
dependabot[bot] 1e22960cf5 build(deps): bump lxml from 4.9.3 to 5.3.1 in /docs (#3301)
Bumps [lxml](https://github.com/lxml/lxml) from 4.9.3 to 5.3.1.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-4.9.3...lxml-5.3.1)

---
updated-dependencies:
- dependency-name: lxml
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-28 09:56:51 +09:00
dependabot[bot] 0b2bda8108 build(deps): bump markdown from 3.4.3 to 3.7 in /docs (#3300)
Bumps [markdown](https://github.com/Python-Markdown/markdown) from 3.4.3 to 3.7.
- [Release notes](https://github.com/Python-Markdown/markdown/releases)
- [Changelog](https://github.com/Python-Markdown/markdown/blob/master/docs/changelog.md)
- [Commits](https://github.com/Python-Markdown/markdown/compare/3.4.3...3.7)

---
updated-dependencies:
- dependency-name: markdown
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-28 09:55:39 +09:00
itchyny 5f7b1aed16 fix usage in help to use jq not the invoked command path (#3299) 2025-03-28 08:43:06 +09:00
itchyny 8819eb8fb4 Enable dependabot PR to update Pipenv dependencies (#3270) 2025-03-28 08:42:44 +09:00
dependabot[bot] 313b0e3735 build(deps): bump jinja2 from 3.1.5 to 3.1.6 in /docs (#3298)
Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.5 to 3.1.6.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.5...3.1.6)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-27 21:33:55 +09:00
liviubobocu 4b84a6e2f6 Include column number in parser and compiler error messages (#3257)
Resolves #1027, resolves #2334, and resolves #3256.
2025-03-10 22:17:49 +09:00
itchyny 668871bdfa Fix warning: sign-compare in @base64d format (#3287) 2025-03-06 17:07:54 +09:00
itchyny 94fd973ebb Fix calloc error on @base64d format (ref #3280) (#3286)
This commit fixes a regression of a8ce2ff, e.g. `"=" | @base64d`.
2025-03-06 04:30:46 +09:00
Nicole Wren c70ae1ada4 fix: foreach should not break init backtracking with DUPN (#3266)
Simplifies the compilation of `foreach` to avoid the problem in #3227 by
removing unnecessary `FORK ... BACKTRACK`.

Does not address `reduce`, which is less trivial.
2025-03-05 22:01:48 +09:00
sachint f65d088b75 Enable IBM z/OS support (#3277) 2025-03-05 17:21:05 +09:00
itchyny 7e5cad359a Add autotools temporary files to .gitignore (#3285) 2025-03-05 16:57:11 +09:00
SArpnt 9d6efec198 fix: unset environment variables before shtests (#3283)
Tests no longer fail if `JQ_COLORS` and `NO_COLOR` are already set.
2025-03-05 07:59:46 +09:00
itchyny a09a4dfd55 Reject NaN with payload while parsing JSON
This commit drops support for parsing NaN with payload in JSON like
`NaN123` and fixes CVE-2024-53427. Other JSON extensions like `NaN` and
`Infinity` are still supported. Fixes #3023, fixes #3196, fixes #3246.
2025-03-05 07:43:54 +09:00
itchyny a8ce2ff09a Avoid zero-length calloc (#3280)
Using `calloc` function with a count of zero elements is implementation
defined and it may cause allocation errors on some platforms (ref #3277).
This commit fixes `jv_mem_calloc` callers to avoid undefined behavior.
2025-03-04 22:33:51 +09:00
itchyny 4003202ccf Fix segmentation fault on strftime/1 and strflocaltime/1 (#3271)
When we use a large format string to `strftime/1` and `strflocaltime/1`,
we get segmentation fault. This happens because the result buffer is
allocated on the stack using `alloca` function, and `strftime` function
writes out of the space when the size is not enough to format time.
I fixed the segmentation fault issue by allocating the result buffer on
the heap using `malloc` function, because `alloca` function does not
provide a way to detect insufficient space.
2025-03-04 22:28:09 +09:00
itchyny dc849e9bb7 Improve performance of repeating strings (#3272)
This commit improves the performance of repeating strings, by copying
the result string instead of the string being repeated. Also it adds
an error message when the result string is too long.
2025-03-04 22:13:55 +09:00
itchyny 5df471fa5b Disable Valgrind by default during testing (#3269)
Previously, the configure script automatically enables Valgrind during
testing when the valgrind command is found. However, running tests with
Valgrind is quite slow, so it is better to disable it by default, and
to enable it only when the user specifies the `--enable-valgrind` flag.
2025-03-04 22:12:25 +09:00
itchyny c4e1fb08fb Support CRLF line breaks in filters (#3274) 2025-03-04 22:11:40 +09:00
itchyny b7a1201b72 Preserve numerical precision on length/0 (#3275) 2025-03-04 22:09:33 +09:00
itchyny 17be97a8ad Define _BSD_SOURCE in builtin.c for OpenBSD support (#3278)
On OpenBSD, some prototypes are hidden behind `_BSD_SOURCE`.
Defining this variable fixes #3252, and also fixes #3276.
This can be seen as a regression of #2631.
2025-03-04 21:39:52 +09:00
itchyny 95ee663b28 improve performance of unique and unique_by (#3254)
Previously, `unique` and `unique_by` filters are implemented using
`group_by` and then `map(.[0])`. This commit re-implements in C, to
avoid unnecessary boxing of grouping, and improves the performance.
2025-03-04 19:59:41 +09:00
Owen Ou 8ba03f788f Replace jqplay.org with play.jqlang.org (#3265)
As discussed in https://github.com/jqlang/jq/discussions/3244, we are
moving jqplay to the jqlang GitHub organization. This updates all
references from jqplay.org to play.jqlang.org.
2025-02-18 08:22:05 +09:00
Marcin Serwin f3d4c1165d fix: copy the string returned by getenv (#3264) 2025-02-18 07:54:13 +09:00
itchyny 70e72abea3 Fix compiler warning type-limits in found_string (ref #2911) (#3263)
Whether `char` has a sign is implementation dependent, and `c >= 0` may
result in a compiler warning. We can use bitwise operator to check
whether the character is a control character, regardless of the sign.
2025-02-18 07:52:58 +09:00
Marcin Serwin 6d8e9f98ae fix: populate timezone data when formatting time (#3203)
The `jv2tm` function was zeroing fields of `struct tm` that were not
specified by the standard. However, depending on the libc this produced
incorrect timezone data when used together with formatting functions.
This change tries to fill the timezone data using either `mktime`,
`timegm`, or manually.

Apple's Libc implementation contains a bug which causes it to ignore
the offset data present in the `struct tm` in favor of the older heuristic
needed by legacy standards. This workaround temporarily sets the global
timezone so it gets picked up during formatting.
2025-02-17 21:41:01 +09:00
itchyny b86ff49f46 fix: jv_number_value should cache the double value of literal numbers (#3245)
The code of `jv_number_value` is intended to cache the double value of
literal numbers, but it does not work because it accepts the `jv` struct
by value. This patch fixes the behavior by checking if the double value
is `NaN`, which indicates the unconverted value. This patch improves the
performance of major use cases; e.g. `range(1000000)` runs 25% faster.
2025-02-16 22:08:36 +09:00
itchyny 64bd8ba374 fix: jv_dump_string_trunc should not break unicode characters (#3249) 2025-02-16 21:40:18 +09:00