Skip to content

internal/loopir, internal/rewrite: vectorize stores at a loop-invariant offset - #2

Closed
thesyncim wants to merge 5 commits into
masterfrom
offset-store
Closed

thesyncim wants to merge 5 commits into
masterfrom
offset-store

Conversation

@thesyncim

@thesyncim thesyncim commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Loads can already be at i+e, e+i or i-e for a loop-invariant e. This allows the same for stores, which covers row-wise code like:

for j := 0; j < n; j++ { b[i*ldb+j] = a[i*lda+j] }        // gonum dlacpy
for j := 0; j < k; j++ { c[i*ldc+j] -= work[i*ldwork+j] } // gonum dlarfb
for x := range w { line[off+x] = v }

Every store to a given slice has to use the same offset, so each iteration still writes its own element. Two different offsets are rejected with a new reason. For loads of a stored slice, the distance is now measured from the store offset instead of from i. The same offset means the same element. A constant positive distance in a forward loop is loaded before the chunk's stores, as today. A negative distance, a non-constant one, or a loop that counts down is rejected. The store is capped at limit+offset, and the length check and the overlap window include the offset. The if-select and copy-builtin paths take the offset too.

With loopvec -json -methods (I read every added loop), std goes from 34 to 35 vectorized loops (an image/jpeg fill), gonum from 161 to 177 (dlacpy, dlaset, dlascl and dlarfb rows, plus some testlapack helpers), and goav1 from 81 to 86. TSVC s423 and s424 now vectorize and match their goldens. The generated code for loops that were already vectorized doesn't change.

offset_store.txt runs 9 shapes over 14 sizes, plus short slices that must panic and aliased views that must fall back. offset_store_no_rewrite.txt pins the rejected cases.

Heads-up: this conflicts with #3, and git merges the two without a conflict into wrong code, because destination() here doesn't know about #3's windows. Whichever one lands second needs destination to defer to the window when there is one. I have that version ready on thesyncim:offset-store-on-windows and can push it here once #3 is in.

The check for stored indices below zero reuses the one from #1.

ubuntu-latest is now ubuntu-26.04, which has no qemu-user-static
package, so every run fails at the install step and skips the arm64
tsvc run, the golden check, fmt and lint. qemu-user ships the static
emulators there; link qemu-aarch64-static to it when the name is
missing, as the Makefile runs that name.
Go tip writes export data in iexport version 4, which encodes float
constants exactly. x/tools v0.51.0, and the v0.52.0 release, read only
up to version 3, so on tip go/packages fails for every package:

  cannot import "math" (unstable iexport format version 4, just rebuild
  compiler and std library), export data is newer version - update tool

and every script that runs loopvec fails in the tip job. x/tools master
reads version 4 (golang/tools@966056e2637c). Update to it until a release
includes it. The pinned toolchain is unaffected.
go1.27.3 is the latest 1.27 release. Its simd package is the same as
go1.27.1's, so the capability table only changes in its header, and
the -methods crash (golang/go#80657) is still there, as
flag_methods_ice.txt checks. go.mod keeps go 1.27.1: that is the first
release whose simd package loopvec's output compiles against
(go1.27.0's ShiftAllRight takes a uint8).
A read at a negative constant offset in a loop with a constant limit, such
as dst[i] = a[i-20] + 1 in a loop that stops before 16, was rewritten into
code that does not compile: the length check of the vector loop is
_ = a[16-21], a negative constant index. -split wrote that file, and the
toolexec wrapper fell back to the original package.

Such a loop panics as soon as it runs, as does one whose first index is
a negative constant (a[i-1] in a loop from 0). Both are now left alone
with their own reason, the way a constant limit of zero or less already
is. A read whose indexes can all be zero or more (a[i-1] from 1) is
still rewritten.

The checks read the start and limit as text, so a named constant or a
constant expression (const s0 = 0; for i := s0; ...) slipped past them.
A constant start or limit is now spelled as its value, which also lets
the short-loop rule see a named constant trip count: goav1's
for x := range restorationExtraHorz, three iterations, is no longer
rewritten.

The vectorized set of std and gonum does not change.
…nt offset

A loop that stores at an offset from the index, such as a row of a 2-D
slice (b[i*ldb+j] = a[i*lda+j], c[row+j] += alpha*x[j]) or a fill that
starts part way into a slice (line[off+x] = v), was rejected: loads could
be at an offset, but every store had to be at exactly the loop index.

A store may now be at the same offsets a load may: i+e, e+i and i-e for a
loop-invariant integer e. Every store to one slice must be at the same
offset, so each iteration still writes its own element; two offsets are
rejected with their own reason. A load of the stored slice at that offset
is the same element, ordered after the stores before it as at the index.
The dependence test for a load at another offset measures the distance
from the store offset instead of from the index, so a constant positive
distance in a forward loop is still loaded before any store of the chunk,
and a negative or variable distance is rejected as before. An if that
selects between stores, and the copy builtin, take the offset too.

The emitted store is capped at the limit plus the offset, so the vector
loop never writes an element the scalar loop does not, and the length
check and the overlap window cover the elements stored. A constant offset
that puts every stored index below zero is left alone, as for a load.

The vectorized set gains one loop in std (an image/jpeg fill), sixteen in
gonum (dlacpy, dlaset, dlascl and dlarfb rows, and testlapack helpers)
and five in github.com/thesyncim/goav1; TSVC s423 and s424 now vectorize
and match their goldens.
@thesyncim

Copy link
Copy Markdown
Owner Author

Superseded by #14, the same change stacked on #11 so it works with the window loops from #3.

@thesyncim thesyncim closed this Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant