Commit Graph
100 Commits
Author SHA1 Message Date
itchynyandAsaf Meizner 0c7d133c3c Randomize hash seed to mitigate hash collision DoS attacks
The hash function used a fixed seed, allowing attackers to craft colliding keys
and cause O(n^2) object parsing performance. Initialize the seed from a random
source at process startup to prevent the attack. This fixes CVE-2026-40164.

Co-authored-by: Asaf Meizner <asafmeizner@gmail.com>
2026-04-13 08:53:26 +09:00
itchyny 6374ae0bcd Fix NUL truncation in the JSON parser
This fixes CVE-2026-33948.
2026-04-13 08:46:11 +09:00
itchyny e47e56d226 Fix heap buffer overflow in jvp_string_append and jvp_string_copy_replace_bad
In `jvp_string_append`, the allocation size `(currlen + len) * 2` could
overflow `uint32_t` when `currlen + len` exceeds `INT_MAX`, causing a small
allocation followed by a large `memcpy`.

In `jvp_string_copy_replace_bad`, the output buffer size calculation
`length * 3 + 1` could overflow `uint32_t`, again resulting in a small
allocation followed by a large write.

Add overflow checks to both functions to return an error for strings
that would exceed `INT_MAX` in length. Fixes CVE-2026-32316.
2026-04-13 08:27:16 +09:00
itchyny 69785bf77f Use // instead of //= in from_entries definition (#3516) 2026-04-02 22:04:51 +09:00
itchyny cec6b0f346 Fix crashes and resource leaks in jq_testsuite (#3509)
- Fix segfault when --skip/--take is the last argument.
- Fix FILE handle leak: opened test files were never closed.
- Support running multiple test files in one invocation.
2026-03-15 21:45:16 +09:00
itchyny 4b145b3a58 Group Docker actions updates in dependabot config 2026-03-09 17:34:16 +09:00
itchyny 3cd7e0da00 Fix crash when importing a module with errors twice (#3497)
When a module fails to parse as a library, `load_library` cached the
invalid block. A second import/include of the same module would find
the cached invalid block and pass it to `block_bind_library`,
triggering the assertion failure. Skip caching the block when there
are parse errors to avoid this problem.
2026-03-06 15:32:29 +09:00
itchyny ed3c7f214b Fix tonumber and toboolean to reject strings with embedded null bytes (#3496) 2026-03-05 08:28:43 +09:00
itchyny bfcf82fde1 Refactor @uri and @urid and fix multi-byte UTF-8 corruption (#3495)
This commit improves performance by refactoring `@uri` and `@urid`
to pre-alloacte buffers and build the `jv` string at the end.
Also, this fixes `@urid` to prevent multi-byte UTF-8 corruption,
and to preserve embedded null bytes.
2026-03-05 08:28:09 +09:00
itchyny 0df54c888a Fix infinite loop and undefined behavior in del(.[nan]) (#3490) 2026-03-02 21:58:28 +09:00
itchyny 262f0b81c1 Add missing validity check in f_strflocaltime after f_localtime (#3491) 2026-03-02 21:58:07 +09:00
itchyny c3d3e7d656 Fix buttons in tutorial to toggle labels when clicked on (#3493) 2026-03-02 17:46:58 +09:00
itchyny 601a37f452 Refactor tm2jv to handle fractional seconds removing duplicated code (#3489) 2026-03-01 06:18:14 +09:00
itchyny 750dbe723b Fix memory leaks and double frees (#3487) 2026-03-01 06:15:58 +09:00
itchyny 22361cfb87 Improve error message truncation with closing delimiters (#3478)
Truncated strings now show closing delimiters: `..."` for strings,
`...]` for arrays, and `...}` for objects, making error messages
clearer.
2026-02-25 08:14:23 +09:00
itchyny b33a7634ba Fix reversed pointer subtraction in f_env bounds check (#3465) 2026-01-20 21:56:09 +09:00
itchyny d44baeb67a Support arm/v7 architecture in Docker images (#3463) 2026-01-15 14:10:07 +09:00
itchyny 33b3a68f9c Fix expected value assertion (fix #3430, ref #3393) (#3431) 2025-11-04 21:21:21 +09:00
itchyny 5529788292 Fix slurping raw input not to corrupt multi-byte characters (fix #3389) (#3421) 2025-10-31 15:16:29 +09:00
itchyny c8017bf079 Fix localization test for time formatting functions (fix #3368) (#3409) 2025-09-15 07:48:02 +09:00
itchyny a4d9d54010 Fix expected value assertion for NaN value (fix #3393) (#3408) 2025-09-15 07:47:51 +09:00
itchyny 695830c453 Fix year 2038 problem on 32-bit platforms (fix #3369) (#3407) 2025-09-15 07:46:29 +09:00
itchyny 0ff997f7c4 Update GPG signing key (fix #3381) 2025-07-31 23:00:24 +09:00
itchyny b9e19de76e Update jq documentation for jq 1.8.1 release 2025-07-01 20:54:28 +09:00
itchyny 4467af7068 Update NEWS.md and AUTHORS for 1.8.1 (#3353) 2025-07-01 19:36:14 +09:00
itchyny 0b1ef46973 Add oniguruma library path to link jq program 2025-06-24 23:49:32 +09:00
itchyny edbc1da809 Revert "fix: reduce/foreach state variable should not be reset each iteration" (#3349)
This reverts 96e8d893c1. The change
introduced a serious performance regression as reported in #3345.
2025-06-17 15:29:46 +09:00
itchyny a587ddc5a3 Fix syntax error assertion failure (fix #3348) (#3350) 2025-06-17 15:25:46 +09:00
itchyny 78045d8aa9 Add LICENSE notice of NetBSD's strptime() to COPYING (#3344) 2025-06-14 08:01:10 +09:00
itchyny 7cc202aa6e Fix portability of strptime especially for Windows (#3342) 2025-06-14 08:00:55 +09:00
itchyny 62cafa2367 Update jq documentation for jq 1.8.0 release (#3333) 2025-06-01 16:38:16 +09:00
itchyny d23a7b9db9 Update NEWS.md and AUTHORS for 1.8.0 (#3330) 2025-06-01 14:51:21 +09:00
itchyny d3cf5caf72 Add more test cases for JQ_COLORS support (ref #3288) 2025-05-31 16:29:02 +09:00
itchyny c6e041699d Fix heap buffer overflow when formatting an empty string
The `jv_string_empty` did not properly null-terminate the string data,
which could lead to a heap buffer overflow. The test case of
GHSA-p7rr-28xf-3m5w (`0[""*0]`) was fixed by the commit dc849e9bb7,
but another case (`0[[]|implode]`) was still vulnerable. This commit
ensures string data is properly null-terminated, and fixes CVE-2025-48060.
2025-05-31 11:46:40 +09:00
itchyny 3b00981acd Fix quotes in 1.7 manual for ease of taking diff between versions (#3329) 2025-05-31 11:31:54 +09:00
itchyny f28720f699 Increase the maximum parsing depth for parsing JSON to 10000 (#3328)
This is the same limit as encoding/json in Go and should be sufficient for most use cases.
2025-05-30 20:19:34 +09:00
itchyny aaace5432a Fix parser to allow binary operators for binding syntax (#3326)
Previously, the precedence of `as` binding was higher than the binary
operators. This results in unexpected behaviors, `1 + 2 as $x | -$x`
was `-1` instead of `-3`. The commit 92daac90f6 (PR #3053) changed
the behavior to emit syntax error on this case. Now, this commit allows
binary operators with `as` binding. This fixes #2446, and fixes #3284.
2025-05-23 21:25:30 +09:00
itchyny 4e3088f188 Fix behavior of --slurp --stream when input has no trailing newline character (#3279)
This commit fixes a strange behavior of the combination of `--slurp` and
`--stream` flags when the input has no trailing newline character.
2025-05-22 08:04:05 +09:00
itchyny de21386681 Fix signed integer overflow in jvp_array_write and jvp_object_rehash
This commit fixes signed integer overflow and SEGV issues on growing
arrays and objects. The size of arrays and objects is now limited to
`536870912` (`0x20000000`). This fixes CVE-2024-23337 and fixes #3262.
2025-05-21 07:48:23 +09:00
itchyny 4e2ccc4c9b Fix and refactor scan-build workflow to run on the master branch (#3325) 2025-05-20 20:00:53 +09:00
itchyny b733dfaee7 Improve error messages to show problematic position in the filter (#3292)
This commit improves the parser and compiler error messages, making
it more easier to understand where in the filter the problem is.
Like modern compilers, jq will show the line in the filter where the
error occurred, will display squiggly line under the invalid token.
2025-05-18 16:36:32 +09:00
itchyny 92daac90f6 Fix unary operator precedence, allow some expressions as object values (#3053)
This commit improves the parser to fix some unintuitive results of the unary
operator `-` (e.g. `[-1 as $x | 1,$x]` is now `[1,-1]`, previously `[-1,-1]`).
Also, this commit now allows to use binary operator expressions, `reduce`,
`foreach`, `if`, and `try` expressions for object values.
2025-05-18 16:13:55 +09:00
itchyny b088e83a82 Refactor splits/2 using foreach syntax (remove private filter _nwise) (#3260)
This commit refactors `splits/2` implementation using `foreach` syntax
and reduces intermediate arrays. This refactoring removes an unnecessary
private (and undocumented) filter `_nwise` and closes #3148.
2025-05-18 08:27:44 +09:00
itchyny 96d19ca2ee Fix use of uninitialized value in check_literal (fix #3316) (#3324) 2025-05-18 08:06:54 +09:00
itchyny ecea2d29bd Update CI runners to Ubuntu 24.04 and macOS Sonoma (#3208) 2025-05-18 08:06:16 +09:00
itchyny eff9caf6c7 Disable shared libraries on macOS to avoid conflict with -all-static (#3261) 2025-05-17 04:57:17 +09:00
itchyny a8f27cc4db Fix warning: a function definition without a prototype is deprecated (#3259) 2025-05-07 21:25:32 +09:00
itchyny 43506372b5 Update builtin oniguruma to 6.9.10 to suppress clang warnings (#3258) 2025-05-07 10:50:42 +00:00
itchyny abde27d0f5 Revert "Move closer to the standard autotools idiom. (#3187)" (#3255)
This partially reverts commit d8463c2549.
Revert reason: version string should be updated more often.
2025-05-07 17:26:19 +09:00
itchyny 5f7b1aed16 fix usage in help to use jq not the invoked command path (#3299) 2025-03-28 08:43:06 +09:00
itchyny 8819eb8fb4 Enable dependabot PR to update Pipenv dependencies (#3270) 2025-03-28 08:42:44 +09:00
itchyny 668871bdfa Fix warning: sign-compare in @base64d format (#3287) 2025-03-06 17:07:54 +09:00
itchyny 94fd973ebb Fix calloc error on @base64d format (ref #3280) (#3286)
This commit fixes a regression of a8ce2ff, e.g. `"=" | @base64d`.
2025-03-06 04:30:46 +09:00
itchyny 7e5cad359a Add autotools temporary files to .gitignore (#3285) 2025-03-05 16:57:11 +09:00
itchyny a09a4dfd55 Reject NaN with payload while parsing JSON
This commit drops support for parsing NaN with payload in JSON like
`NaN123` and fixes CVE-2024-53427. Other JSON extensions like `NaN` and
`Infinity` are still supported. Fixes #3023, fixes #3196, fixes #3246.
2025-03-05 07:43:54 +09:00
itchyny a8ce2ff09a Avoid zero-length calloc (#3280)
Using `calloc` function with a count of zero elements is implementation
defined and it may cause allocation errors on some platforms (ref #3277).
This commit fixes `jv_mem_calloc` callers to avoid undefined behavior.
2025-03-04 22:33:51 +09:00
itchyny 4003202ccf Fix segmentation fault on strftime/1 and strflocaltime/1 (#3271)
When we use a large format string to `strftime/1` and `strflocaltime/1`,
we get segmentation fault. This happens because the result buffer is
allocated on the stack using `alloca` function, and `strftime` function
writes out of the space when the size is not enough to format time.
I fixed the segmentation fault issue by allocating the result buffer on
the heap using `malloc` function, because `alloca` function does not
provide a way to detect insufficient space.
2025-03-04 22:28:09 +09:00
itchyny dc849e9bb7 Improve performance of repeating strings (#3272)
This commit improves the performance of repeating strings, by copying
the result string instead of the string being repeated. Also it adds
an error message when the result string is too long.
2025-03-04 22:13:55 +09:00
itchyny 5df471fa5b Disable Valgrind by default during testing (#3269)
Previously, the configure script automatically enables Valgrind during
testing when the valgrind command is found. However, running tests with
Valgrind is quite slow, so it is better to disable it by default, and
to enable it only when the user specifies the `--enable-valgrind` flag.
2025-03-04 22:12:25 +09:00
itchyny c4e1fb08fb Support CRLF line breaks in filters (#3274) 2025-03-04 22:11:40 +09:00
itchyny b7a1201b72 Preserve numerical precision on length/0 (#3275) 2025-03-04 22:09:33 +09:00
itchyny 17be97a8ad Define _BSD_SOURCE in builtin.c for OpenBSD support (#3278)
On OpenBSD, some prototypes are hidden behind `_BSD_SOURCE`.
Defining this variable fixes #3252, and also fixes #3276.
This can be seen as a regression of #2631.
2025-03-04 21:39:52 +09:00
itchyny 95ee663b28 improve performance of unique and unique_by (#3254)
Previously, `unique` and `unique_by` filters are implemented using
`group_by` and then `map(.[0])`. This commit re-implements in C, to
avoid unnecessary boxing of grouping, and improves the performance.
2025-03-04 19:59:41 +09:00
itchyny 70e72abea3 Fix compiler warning type-limits in found_string (ref #2911) (#3263)
Whether `char` has a sign is implementation dependent, and `c >= 0` may
result in a compiler warning. We can use bitwise operator to check
whether the character is a control character, regardless of the sign.
2025-02-18 07:52:58 +09:00
itchyny b86ff49f46 fix: jv_number_value should cache the double value of literal numbers (#3245)
The code of `jv_number_value` is intended to cache the double value of
literal numbers, but it does not work because it accepts the `jv` struct
by value. This patch fixes the behavior by checking if the double value
is `NaN`, which indicates the unconverted value. This patch improves the
performance of major use cases; e.g. `range(1000000)` runs 25% faster.
2025-02-16 22:08:36 +09:00
itchyny 64bd8ba374 fix: jv_dump_string_trunc should not break unicode characters (#3249) 2025-02-16 21:40:18 +09:00
itchyny fa6a2ff688 website: remove unused root path on building website (#3248)
We no longer use the `--root` option of `build_website.py` since we have
moved to jqlang.org (from jqlang.github.io/jq where the root was `/jq`).
We don't need the option on local development either since `http.server`
has the `-d` option to specify the directory to serve from.
2025-02-06 22:42:27 +09:00
itchyny 6ce3bdf382 fix: preserve numerical precision on unary negation (#3242) 2025-02-06 07:50:21 +09:00
itchyny 972772153f Move oniguruma and decNumber to vendor directory (#3234) 2025-02-06 07:49:56 +09:00
d0adcbfebb Fix --indent 0 implicitly enabling compact-output (#3232)
Now it will retain pretty-printing, just remove any indentation.

Co-authored-by: Andrew Marshall <andrew@johnandrewmarshall.com>
Co-authored-by: Gabriel Marin <marin.gabriel@protonmail.com>
2025-02-01 15:09:30 +09:00
itchyny 0d1e371e63 website: switch to custom domain jqlang.org (#3243) 2025-02-01 11:54:32 +09:00
itchyny dea5e61cde docs: fix example links to jqplay.org (#3241) 2025-01-28 22:15:19 +09:00
itchyny ce54621904 docs: mention the behavior of scan using regex with capturing groups (#3240) 2025-01-28 11:41:17 +09:00
itchyny 39eea145b4 website: deploy website from GitHub Actions (#3237)
Deploying GitHub Pages from the gh-pages branch is a classic way.
We can use actions/deploy-pages action to deploy GitHub Pages.
2025-01-28 11:40:55 +09:00
itchyny 31dac287cc tests: quote .[] in shtest which can break syntax highlight (#3235) 2025-01-26 08:08:31 +09:00
itchyny 71ed7b3412 tests: refactor constant folding tests (#3233) 2025-01-25 13:58:47 +09:00
itchyny 96e8d893c1 fix: reduce/foreach state variable should not be reset each iteration (#3205)
When the UPDATE query in a reduce or foreach syntax does not emit
a value, it should simply skip updating the state variable. But
currently it resets the state variable to null.
2024-11-21 08:12:18 +09:00
itchyny a7b22539a4 fix: make last(empty) yield no output values (#3179) 2024-11-13 20:35:50 +09:00
itchyny 1bebd23aad add skip/2 as the counterpart to limit/2 (#3181)
I also changed the behavior of limit/2 with negative count to emit an error, like nth/2 does.
Also, I redefined nth/2 using skip/2 to minimize the impact of growth of builtin filters.
2024-10-06 08:25:38 +09:00
itchyny 81f4f883ac Remove non-standard pow10 filter in favor of exp10 (#3059)
The pow10 function was a glibc extension removed in 2.27. Use exp10 filter instead.
2024-03-20 18:23:08 +09:00
itchyny ce0e788ce2 improve tonumber/0 performance by parsing input as number literal
Previously, the tonumber/0 filter parses the input as JSON values, but
this is less-performant on large non-number strings. Parsing the input
string as number literal fixes the performance issue. Also, this fix
changes the filter to reject numbers with white spaces.
2024-03-01 16:33:17 +01:00
itchyny 56828b6d04 Always build Docker image on CI, group Dependabot PRs for official actions (#2997)
* Always build docker image on CI

* Revert "Bump actions/download-artifact from 3 to 4"

This reverts commit 9e2695e117.

* Group Dependabot PRs for official actions
2023-12-19 17:54:53 +09:00
itchyny d14393f552 Keep releasing executables with legacy names 2023-11-28 22:51:20 +01:00
itchyny 07149397d1 Reject U+001F in string literals (fix #2909) 2023-09-26 12:53:44 -05:00
itchyny e526df15fe Correct typo in README.md: compilation (#2912) 2023-09-26 19:49:07 +02:00
itchyny 5ea1a4ab3b Disable core.autocrlf on Windows to prevent submodule diffs (fix #2886) (#2888) 2023-09-15 20:08:20 +09:00
itchyny 70807e2b1b Include more updates to NEWS.md and AUTHORS for 1.7 2023-08-23 15:59:19 -05:00
itchyny 080471b2a0 Add src/config_opts.inc to .gitignore 2023-08-21 16:06:58 +02:00
itchyny 7d64381016 Fix overflow on numeric comparison (#2849)
Although #2839 fixed the overflow of exponent subtraction,
there still is possibility of overflow in the `D2U` macro.
This patch fixes the overflow in the `D2U` macro, and also
truncates the maximum digits to `DEC_MAX_DIGITS`.
2023-08-17 15:56:57 +09:00
itchyny 0733fd3d58 Add a regression test for negative indices and fix a pick/1 test 2023-08-16 17:49:44 -05:00
itchyny f85c9fcb78 Revert "Allow .[-1] in path expressions"
This reverts commit 086a156ec3.

This commit leads to negative indexing wraps twice.
2023-08-16 17:49:44 -05:00
itchyny 3fa10e8cc1 Fix crash on numeric comparison again (ref #2825)
The decNumber library subtracts the exponents of two numbers,
we make sure to limit the number of digits not to make it overflows.
Since the maximum adjusted exponent is `emax` and the minimum is
`emin - digits + 1`, we follow `emax - (emin - digits + 1) <= INT32_MAX`.
2023-08-13 14:17:25 +02:00
itchyny f86566b263 Rename jv_type_private.h to jv_private.h, move jvp_number_is_nan there 2023-08-13 14:16:02 +02:00
itchyny a692060129 Change the default color of null to Bright Black 2023-08-05 23:21:12 -05:00
itchyny 949d38e6dc Fix crash on numeric comparison (ref #2804) (#2818) 2023-08-03 23:19:52 +09:00
itchyny fd0d475694 Fix a test case added in #2790 2023-08-02 22:33:09 +09:00
itchynyandLeonid S. Usov 680baeffeb Fix rounding small (but not too small) numbers to zero on calculation
Co-authored-by: Leonid S. Usov <leonid.s.usov@gmail.com>
2023-08-02 00:32:32 -05:00
itchyny 0f80921268 Fix constant folding of division and reminder with zero divisor
Previously constant folding of zero division (e.x. 1/0) produces a
compile error. This was incorrectly implemented by checking if the
division result is infinite, so produces wrong results compared to the
query where no constant folding is processed (e.x. 1e308/0.1). This
patch delays the operation when the divisor is zero. This makes the
results more consistent, but changes the exit code on zero division from
3 to 5. Also 0/0 now produces the zero division error, not NaN.

This patch also fixes the modulo operation. Previously constant folding
logic does not take care of the % operator, but now it folds if the both
dividend and divisor are safe numbers to cast to the integer type, and
the divisor is not zero. This patch also fixes some code that relies on
undefined cast behaviors in C. The modulo operation produces NaN if
either the dividend or divisor is NaN.
2023-07-31 15:39:13 -05:00
itchyny c8e28da129 Redesign website (#2628)
* Bump up Bootstrap to v5.3.1, Bootstrap Icon to v1.10.5.
* Use autoComplete.js to drop dependency on jQuery and typeahead.js.
* Support dark mode.
* New svg logo and icon with responsive color mode support.
* Normalize section ids to lower kebab-case for easiness of linking.
* Use relative paths for links for local development (--root /output).
* Various markup cleanups and accessibility improvements.
2023-07-31 09:52:52 +09:00
itchyny d319eb2187 Fix PR creation by using fully qualified refname on release 2023-07-28 17:47:51 +09:00