Repository navigation
internal/rewrite: make the overlap check small enough to inline - #11
Merged
Merged
Conversation
ubuntu-latest is now ubuntu-26.04, which has no qemu-user-static package, so every run fails at the install step and skips the arm64 tsvc run, the golden check, fmt and lint. qemu-user ships the static emulators there; link qemu-aarch64-static to it when the name is missing, as the Makefile runs that name.
Go tip writes export data in iexport version 4, which encodes float constants exactly. x/tools v0.51.0, and the v0.52.0 release, read only up to version 3, so on tip go/packages fails for every package: cannot import "math" (unstable iexport format version 4, just rebuild compiler and std library), export data is newer version - update tool and every script that runs loopvec fails in the tip job. x/tools master reads version 4 (golang/tools@966056e2637c). Update to it until a release includes it. The pinned toolchain is unaffected.
thesyncim
force-pushed
the
inline-overlap-check
branch
2 times, most recently
from
October 31, 2026 21:02
f4bdcc9 to
2d0651d
Compare
go1.27.3 is the latest 1.27 release. Its simd package is the same as go1.27.1's, so the capability table only changes in its header, and the -methods crash (golang/go#80657) is still there, as flag_methods_ice.txt checks. go.mod keeps go 1.27.1: that is the first release whose simd package loopvec's output compiles against (go1.27.0's ShiftAllRight takes a uint8).
A read at a negative constant offset in a loop with a constant limit, such as dst[i] = a[i-20] + 1 in a loop that stops before 16, was rewritten into code that does not compile: the length check of the vector loop is _ = a[16-21], a negative constant index. -split wrote that file, and the toolexec wrapper fell back to the original package. Such a loop panics as soon as it runs, as does one whose first index is a negative constant (a[i-1] in a loop from 0). Both are now left alone with their own reason, the way a constant limit of zero or less already is. A read whose indexes can all be zero or more (a[i-1] from 1) is still rewritten. The checks read the start and limit as text, so a named constant or a constant expression (const s0 = 0; for i := s0; ...) slipped past them. A constant start or limit is now spelled as its value, which also lets the short-loop rule see a named constant trip count: goav1's for x := range restorationExtraHorz, three iterations, is no longer rewritten. The vectorized set of std and gonum does not change.
thesyncim
force-pushed
the
inline-overlap-check
branch
3 times, most recently
from
October 31, 2026 22:36
e5dc008 to
0484dee
Compare
thesyncim
force-pushed
the
inline-overlap-check
branch
2 times, most recently
from
October 31, 2026 23:20
0aec3df to
3b69346
Compare
… capture
The rewritten code adds names that begin with an underscore (_i,
_count, _w1, broadcast vectors declared before the loop, ...) and relies
on predeclared names (len, min, max, copy, uint64, the element types) and
on the simd and unsafe packages. Each can mean something else in the
function, and then the vector loop silently computes something else.
All of these already go wrong on master:
func F(dst, a []int32, _i int) { for i := 0; i < _i; i++ { ... } }
reads loopvec's _i in the loop condition and never runs; with a
package-level func len(s []int32) int the vector loop takes its window
length from that len; and a reduction written as for n := range n, or
for n := n - 1; n >= 0; n--, finishes with for n := _i; n < n; n++,
which never runs.
A loop that would otherwise be rewritten is now left alone, each with
its own reason, when its function uses an identifier that begins with an
underscore, other than _; when one of the names the code relies on,
looked up from the loop, is declared again around it, in the package, or
by an import; or when the loop declares such a name itself, or a name
its range operand or initial values use. The names the overlap helper
uses (uintptr, int, bool, any, false) are checked at package level,
where loopvec declares it. The lookup needs scopes, so the toolexec
wrapper records them. No loop in std, gonum or
github.com/thesyncim/goav1 stops being vectorized.
thesyncim
force-pushed
the
inline-overlap-check
branch
from
October 31, 2026 23:53
3b69346 to
20fa3a5
Compare
fixBlankParams works around a compiler bug that rejects _ parameters in
functions with simd code by renaming them _p0, _p1, ..., but it renamed
them in every function of a rewritten file. In a function that reads a
package-level _p0, the parameter then hid it:
var _p0 = 5
func Offset(_ int) int { return _p0 }
became func Offset(_p0 int) int { return _p0 } and returned its
argument. Only the functions loopvec rewrote are renamed now. They
mention no name that begins with an underscore, so the new names cannot
capture anything.
…an prove Every element-wise loop advanced one partial vector at a time: LoadPart and StorePart on every iteration, each working out how many elements remain. On arm64, which has no masked loads, that costs more than the operation itself (about 50 instructions per 4 lanes of AccumRowInt32s), and the bounds checks of slicing each operand at _i stay in the loop on every target. The loop now cuts every slice it touches to its window before it starts: the elements from the first iteration at that slice's offset, as many as the loop runs, with a capacity of as many, indexed from zero. _count is the first window's length, so the compiler knows it is not negative. Under conditions of the form _i <= _count-k*_lanes it then proves every slice the loop takes of a window in bounds and drops all bounds and length checks from the loop. The main loop runs four copies of the body per iteration over consecutive vectors, each copy all the statements in order, so the chunks run in the same order as with one vector per iteration; each window is cut to the iteration's four vectors, so every copy addresses its vector at a constant offset. Whole vectors then run one at a time, and one partial vector takes what is left. Integer reductions get the four accumulators float reductions have: an integer vector add takes two cycles on arm64, where they make SumInt32s three times as fast, and integer folds wrap, so regrouping them gives the same result. Because a partial load now only ever sees fewer elements than a vector holds, the emulated Load*Part of golang/go#82001, which returns len(s), returns the right count, and GODEBUG=simd=0 builds produce the scalar result.
Every rewritten loop with two or more slices calls _loopvecOverlap for each pair before it runs. The helper took the windows as ...int and inlining it cost 114, over the compiler's budget of 80, so each call was a real call: AddFloat32s made two before it loaded anything, which for a short loop costs about as much as the loop. The helper now takes the windows as four plain arguments, and a whole slice is passed as 0 and its length. It inlines (cost 55), and the comparison is the same.
thesyncim
force-pushed
the
inline-overlap-check
branch
from
November 1, 2026 00:05
20fa3a5 to
848fa69
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3.
_loopvecOverlaptook the windows as...int, and with that its inlining cost was 114, over the budget of 80:So every rewritten loop that touches two slices makes a real call per pair before it starts, which takes about as long as a 64-element loop.
It now takes the four bounds as plain ints (a whole slice is
0, len(s)) and inlines at cost 55. The comparison is the same.On an M4 Max, on top of #3: AddFloat32s/64 goes from 9.9 to 8.1 ns, AxpyFloat32s/64 from 10.3 to 8.0, ClampInt32s/64 from 8.3 to 6.9. Larger sizes don't notice.