Improve performance of join/1 on large arrays (#3639)
join/1 folded each element as `(acc + $x) + elem`, with the element branch ending in `.//""`. Both leave a second live reference to the accumulator: the nested + feeds the same `.` to both operands, and the `//` leaves a forkpoint below it on the stack. `jvp_string_append` therefore cannot grow the accumulator in place, and copies the whole string once per element. Pipe `acc + $x` into `. + elem` and drop the `//`, since adding null to a string is already a no-op. join/1 is now linear: joining 8000 strings of 1000 chars goes from 6.5s to 0.06s, with output and error messages unchanged.
This commit is contained in:
1 parent
32580f5d83
commit
9a75bb0d43
2 files changed
+12
-2
No files matched your search
+4
-2
@@ -63,9 +63,11 @@ def strings: select(type == "string");
|
||||
def nulls: select(. == null);
|
||||
def values: select(. != null);
|
||||
def scalars: select(type|. != "array" and . != "object");
|
||||
# Keep the accumulator the only reference on the left of + so the string is
|
||||
# appended in place; a nested + or a // inside the reduce copies it every step.
|
||||
def join($x): reduce .[] as $i (null;
|
||||
(if .==null then "" else .+$x end) +
|
||||
($i | if type=="boolean" or type=="number" then tostring else .//"" end)
|
||||
(if .==null then "" else .+$x end) |
|
||||
. + ($i | if type=="boolean" or type=="number" then tostring end)
|
||||
) // "";
|
||||
def _flatten($x): reduce .[] as $i ([]; if $i | type == "array" and $x != 0 then . + ($i | _flatten($x-1)) else . + [$i] end);
|
||||
def flatten($x): if $x < 0 then error("flatten depth must not be negative") else _flatten($x) end;
|
||||
|
||||
@@ -2076,6 +2076,14 @@ try join(",") catch .
|
||||
["1","2",[3,4,5]]
|
||||
"string (\"1,2,\") and array ([3,4,5]) cannot be added"
|
||||
|
||||
join("-")
|
||||
{"a":"x","b":1,"c":null,"d":false}
|
||||
"x-1--false"
|
||||
|
||||
try join(1) catch .
|
||||
["a","b"]
|
||||
"string (\"a\") and number (1) cannot be added"
|
||||
|
||||
{if:0,and:1,or:2,then:3,else:4,elif:5,end:6,as:7,def:8,reduce:9,foreach:10,try:11,catch:12,label:13,import:14,include:15,module:16}
|
||||
null
|
||||
{"if":0,"and":1,"or":2,"then":3,"else":4,"elif":5,"end":6,"as":7,"def":8,"reduce":9,"foreach":10,"try":11,"catch":12,"label":13,"import":14,"include":15,"module":16}
|
||||
|
||||
Reference in new issue
Block a user