diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index ba31cc9..efe0baa 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -61,8 +61,13 @@ jobs: - name: Test tsvc (amd64, SIMD) run: make tsvc-test - - name: Install qemu-user-static - run: sudo apt-get update && sudo apt-get install -y qemu-user-static + # ubuntu-26.04 has no qemu-user-static: qemu-user ships the static + # emulators there, without the -static names the Makefile runs. + - name: Install qemu-user + run: | + sudo apt-get update && sudo apt-get install -y qemu-user + command -v qemu-aarch64-static || sudo ln -s "$(command -v qemu-aarch64)" /usr/local/bin/qemu-aarch64-static + qemu-aarch64-static --version - name: Test tsvc (arm64, qemu) run: make tsvc-test-qemu-arm64 diff --git a/AGENTS.md b/AGENTS.md index 8cd8d85..fd96c14 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -94,18 +94,21 @@ safety argument stays structural: the loop runs at least once (`x` may be a nil pointer the scalar loop never reads), and the overlap check and the vector loop use the locals; the fallback is the original loop. Arrays, pointers to slices, maps, and the result of a call stay rejected. -- A store is `ident[i]` where `i` is *exactly* the loop variable (`index`). A load - may also be `ident[i+e]`, `ident[e+i]` or `ident[i-e]` for a loop-invariant integer - `e` (`indexAt`, `Load.Off`). A slice that is also stored may be read only at a - constant positive offset in a forward loop: iteration i then reads what a later - iteration writes, so the scalar loop sees the old value and the emitter loads it - before any store of the chunk (`statements` hoists those loads). A negative - offset is a recurrence (`ReasonCarriedDependence`); a variable distance or a - loop that counts down is `ReasonOffsetOfStored`. The length check is - against the last index read (`lastIndex`), and offset loads are not capped at the - limit. Every other store and load is `ident[i]` (`index`) — `dst[i+1]`, `dst[2*i]` are rejected outright, regardless of - whether they're actually dependent. That is what guarantees no two - iterations touch the same element. +- A store or a load is `ident[i]`, `ident[i+e]`, `ident[e+i]` or `ident[i-e]` where + `i` is *exactly* the loop variable and `e` a loop-invariant integer (`indexAt`, + `Store.Off`, `Load.Off`). Every store to one slice is at the same offset + (`ReasonStoreOffsets` otherwise), so each iteration writes its own element. A slice + that is also stored may be read at that offset, or at a constant positive distance + past it in a forward loop: iteration i then reads what a later iteration writes, so + the scalar loop sees the old value and the emitter loads it before any store of the + chunk (`statements` hoists those loads). A negative distance is a recurrence + (`ReasonCarriedDependence`); a variable distance or a loop that counts down is + `ReasonOffsetOfStored`. The length check is against the last index touched + (`lastIndex`); offset loads are not capped at the limit, and a store at an offset + is capped at the limit plus the offset (`destination`), so it writes nothing the + scalar loop does not. `dst[2*i]` and `dst[i][j]` are rejected outright, regardless + of whether they're actually dependent. That is what guarantees no two iterations + touch the same element. - A non-slice operand becomes an `Invariant` only if it is a variable, a constant, a conversion or negation of one, or a sum, difference, product or bitwise combination of two (`invariant`). It is evaluated diff --git a/BUGS.md b/BUGS.md index 843087e..55231c5 100644 --- a/BUGS.md +++ b/BUGS.md @@ -2,11 +2,11 @@ loopvec rewrites code to use `GOEXPERIMENT=simd`, which is new, so most of what it hits is in the Go toolchain. This file lists the Go issues that affect loopvec. "tip" means -gotip, "1.27.1" is the release toolchain loopvec is tested against. +gotip, "1.27.3" is the release toolchain loopvec is tested against. | Issue | Status | Affects | loopvec's answer | |---|---|---|---| -| [#80657](https://github.com/golang/go/issues/80657) simd in a method: internal compiler error | fixed on tip (CL 839405), still crashes on 1.27.1 | `-methods` | methods are skipped unless `-methods` is passed | +| [#80657](https://github.com/golang/go/issues/80657) simd in a method: internal compiler error | fixed on tip (CL 839405), still crashes on 1.27.3 | `-methods` | methods are skipped unless `-methods` is passed | | [#80835](https://github.com/golang/go/issues/80835) legacy SSE encodings in functions using simd cause AVX-SSE transition penalties | open, fix pending (CLs 825145, 825146, in tip) | speed of any scalar float code that runs after a rewritten loop, in the same function or another | none; the portable `simd` package has no `ClearAVXUpperBits` | | [#81844](https://github.com/golang/go/issues/81844) blank identifier parameters rejected | fixed on tip (CL 841145) | every rewrite of a function with a `_` parameter | the generated file renames them `_p0`, `_p1`, ... | | [#81845](https://github.com/golang/go/issues/81845) calls left unresolved when inlined into a package initializer | closed as a duplicate of [#80689](https://github.com/golang/go/issues/80689), which is open | loops in package-level `var x = func() {...}()` | not rewritten (the generated initializer fails to link), see `package_initializer_no_rewrite.txt` and `toolexec_package_initializer.txt` | @@ -19,7 +19,7 @@ gotip, "1.27.1" is the release toolchain loopvec is tested against. A function with a receiver that contains simd code failed to compile with `internal compiler error: missing Types entry`. loopvec does not rewrite loops in methods -by default; `-methods` turns it on. The fix is on tip. On 1.27.1 the crash is still there, +by default; `-methods` turns it on. The fix is on tip. On 1.27.3 the crash is still there, and `flag_methods_ice.txt` pins it, so that test starts failing when the pinned toolchain gets the fix. Reported by someone else; the fix is [CL 839405](https://go.dev/cl/839405). diff --git a/Makefile b/Makefile index b4226b3..3804653 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -export GOTOOLCHAIN ?= go1.27.1 +export GOTOOLCHAIN ?= go1.27.3 .DEFAULT_GOAL := build diff --git a/README.md b/README.md index cae8ccc..5fc1446 100644 --- a/README.md +++ b/README.md @@ -45,6 +45,7 @@ operations that lower to **AVX-512/AVX2/NEON** depending on the target CPU: | `for i := range x { if x[i] < 0 { x[i] = 0 } }`, `if c { dst[i] = u } else if d { dst[i] = v } else { dst[i] = w }` | compare, then select (`IfElse`): ReLU, leaky ReLU, ReLU6, hard tanh, clamp, sign, step; also when the branches assign a temporary (`t := x[i]+3; if t < 0 { t = 0 } else if t > 6 { t = 6 }`: hard sigmoid, hard swish, cubic SiLU) | | `for i := range v.Values { v.Values[i] *= a }`, `dst.f[i] += src.f[i]` | a slice reached through a field (`x.f`, `x.f.g`); the root must not be assigned in the loop, and the field is copied into a local once | | `for i := range dst { dst[i] = a[i+1] - a[i] }`, `work[j] += a[row*lda+j]` | read at an offset from the index (stencil, row of a 2-D slice); a slice the loop also writes is read only at a constant positive offset (`a[i] = a[i+1] + b[i]`) | +| `b[i*ldb+j] = a[i*lda+j]`, `c[row+j] += alpha * x[j]`, `line[off+x] = v` | store at an offset from the index (row of a 2-D slice); every store to a slice is at the same offset, and the slice is read only there or at a constant positive distance past it | Supported element types: `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`, `float32`, `float64`. diff --git a/bench/bench_test.go b/bench/bench_test.go index 8821213..cff789e 100644 --- a/bench/bench_test.go +++ b/bench/bench_test.go @@ -451,6 +451,21 @@ func BenchmarkAccumRowInt32s(b *testing.B) { } } +// StoreRowFloat32s adds alpha*x into one row of a two-dimensional slice. +func BenchmarkStoreRowFloat32s(b *testing.B) { + for _, n := range benchSizes { + b.Run(fmt.Sprintf("%d", n), func(b *testing.B) { + x := make([]float32, n) + c := make([]float32, 2*n) + setBytes(b, n, 4, 3) + b.ResetTimer() + for b.Loop() { + StoreRowFloat32s(c, x, 2, 1, n) + } + }) + } +} + func BenchmarkMinUint16s(b *testing.B) { for _, n := range benchSizes { b.Run(fmt.Sprintf("%d", n), func(b *testing.B) { diff --git a/bench/ops.go b/bench/ops.go index 12abaf9..6a143c0 100644 --- a/bench/ops.go +++ b/bench/ops.go @@ -332,3 +332,9 @@ func HardSigmoidFloat32s(dst, x []float32) { dst[i] = t / 6 } } + +func StoreRowFloat32s(c, x []float32, alpha float32, row, ldc int) { + for j := range x { + c[row*ldc+j] += alpha * x[j] + } +} diff --git a/bench/ops_simd.go b/bench/ops_simd.go index 18058cb..9b2f8f2 100644 --- a/bench/ops_simd.go +++ b/bench/ops_simd.go @@ -986,6 +986,26 @@ func HardSigmoidFloat32s(dst, x []float32) { } } +func StoreRowFloat32s(c, x []float32, alpha float32, row, ldc int) { + _vcAFloat32s46 := simd.BroadcastFloat32s(alpha) + if _loopvecOverlap(c, x, 0+(min(0, row*ldc)), len(x)+(max(0, row*ldc)), 0, len(x)) { + for j := range x { + c[row*ldc+j] += alpha * x[j] + } + } else { + if len(x) > 0 { + _ = c[len(x)+(row*ldc)-1] + _ = x[len(x)-1] + for _i := 0; _i < len(x); { + _v1, _n := simd.LoadFloat32sPart(x[_i:len(x)]) + _v2, _ := simd.LoadFloat32sPart(c[_i+(row*ldc):]) + _v1.MulAdd(_vcAFloat32s46, _v2).StorePart(c[_i+(row*ldc) : len(x)+(row*ldc)]) + _i += _n + } + } + } +} + func _loopvecOverlap[T any](a, b []T, window ...int) bool { aLo, aHi, bLo, bHi := 0, len(a), 0, len(b) if len(window) == 4 { diff --git a/go.mod b/go.mod index ed04302..9457290 100644 --- a/go.mod +++ b/go.mod @@ -3,7 +3,7 @@ module github.com/mauri870/loopvec go 1.27.1 require ( - golang.org/x/tools v0.51.0 + golang.org/x/tools v0.52.1-0.20261030190145-0f433775b36d rsc.io/script v0.0.2 ) @@ -207,10 +207,10 @@ require ( go.uber.org/zap v1.27.0 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect golang.org/x/exp/typeparams v0.0.0-20260811152304-ee035b5b010f // indirect - golang.org/x/mod v0.41.0 // indirect - golang.org/x/sync v0.23.0 // indirect - golang.org/x/sys v0.48.0 // indirect - golang.org/x/telemetry v0.0.0-20260924152758-ed294f943157 // indirect + golang.org/x/mod v0.42.0 // indirect + golang.org/x/sync v0.24.0 // indirect + golang.org/x/sys v0.49.0 // indirect + golang.org/x/telemetry v0.0.0-20261009151841-97e8ea77aa2a // indirect golang.org/x/text v0.41.0 // indirect google.golang.org/protobuf v1.36.12 // indirect gopkg.in/ini.v1 v1.67.0 // indirect diff --git a/go.sum b/go.sum index 04c2946..bb2b241 100644 --- a/go.sum +++ b/go.sum @@ -688,8 +688,8 @@ golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3/go.mod h1:3p9vT2HGsQu2K1YbXdKPJLVgG5VJdoTa1poYQBtP1AY= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= -golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c= -golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o= +golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= +golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -724,8 +724,8 @@ golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues= -golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg= +golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw= +golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -745,8 +745,8 @@ golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk= -golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -790,10 +790,10 @@ golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= -golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= -golang.org/x/telemetry v0.0.0-20260924152758-ed294f943157 h1:DftxASHKl6JolFPqoN+2WrFO53JvkHwRXvFqf8gsivA= -golang.org/x/telemetry v0.0.0-20260924152758-ed294f943157/go.mod h1:i+ivNqjDnTF3WTElsdk5g9V5DTSBYgdNo7xTU9SDwYA= +golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= +golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= +golang.org/x/telemetry v0.0.0-20261009151841-97e8ea77aa2a h1:wnveMBKHwx8/nIb8WYT5P9vtefdXOvIWxzycCkc5n6Q= +golang.org/x/telemetry v0.0.0-20261009151841-97e8ea77aa2a/go.mod h1:i+ivNqjDnTF3WTElsdk5g9V5DTSBYgdNo7xTU9SDwYA= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= @@ -857,8 +857,8 @@ golang.org/x/tools v0.1.1-0.20210302220138-2ac05c832e1a/go.mod h1:9bzcO0MWcOuT0t golang.org/x/tools v0.1.10/go.mod h1:Uh6Zz+xoGYZom868N8YTex3t7RhtHDBrE8Gzo9bV56E= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= -golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI= -golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA= +golang.org/x/tools v0.52.1-0.20261030190145-0f433775b36d h1:F5ot9G6RM6pgx3TTdKF6IZa2nocit4tcsrVQlq4Zh5w= +golang.org/x/tools v0.52.1-0.20261030190145-0f433775b36d/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs= golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM= golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY= golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM= diff --git a/internal/loopir/caps_gen.go b/internal/loopir/caps_gen.go index bccacfa..ba7ddff 100644 --- a/internal/loopir/caps_gen.go +++ b/internal/loopir/caps_gen.go @@ -1,4 +1,4 @@ -// Code generated by capsgen from the simd package of go1.27.1. DO NOT EDIT. +// Code generated by capsgen from the simd package of go1.27.3. DO NOT EDIT. package loopir diff --git a/internal/loopir/ir.go b/internal/loopir/ir.go index 981fe77..f83b0ba 100644 --- a/internal/loopir/ir.go +++ b/internal/loopir/ir.go @@ -166,9 +166,12 @@ type Loop struct { // Stmt is a statement of a loop body: a Store or a Let. type Stmt interface{ isStmt() } -// Store writes Val to Dst[i]. +// Store writes Val to Dst[i+Off]. Off is source text for an integer expression +// that is the same on every iteration, "" for zero, as in Load. Every store to +// one slice is at the same offset, so no two iterations write the same element. type Store struct { Dst *Ref + Off string Val Value } @@ -207,9 +210,10 @@ type Value interface{ isValue() } // Load reads Ref[i+Off]. Off is source text for an integer expression that is the // same on every iteration, "" for zero. A slice that is also stored in the loop is -// read only at a constant positive offset: iteration i then reads the element a -// later iteration writes, and the vector loop, which loads the chunk before any -// store of the body, sees the same old value. +// read only at the offset it is stored at, or a constant positive distance past +// it: iteration i then reads the element a later iteration writes, and the vector +// loop, which loads the chunk before any store of the body, sees the same old +// value. // FromRange is set when the source is the range value variable, which holds the // element as it was when the iteration began: a store to the same slice earlier in // the body does not change it. diff --git a/internal/loopir/lower.go b/internal/loopir/lower.go index c4a7719..cef819b 100644 --- a/internal/loopir/lower.go +++ b/internal/loopir/lower.go @@ -286,6 +286,12 @@ func (l *lowerer) bound(e ast.Expr) (bound, bool) { } b := bound{text: types.ExprString(e)} b.value, b.isConst = l.constant(e) + if b.isConst { + // A named constant or a constant expression (s0, 1 - 1) is spelled as its + // value, so the checks that read a constant start or limit see it and the + // emitted code never indexes at a negative constant they would have caught. + b.text = strconv.FormatInt(b.value, 10) + } if call, ok := ast.Unparen(e).(*ast.CallExpr); ok { b.slice, _ = l.lenOf(call) } @@ -465,7 +471,7 @@ func (l *lowerer) body(node ast.Stmt, block *ast.BlockStmt, trip Trip, step int6 // dst[i] = src[i] is a copy, emitted as the copy builtin. Copying a // slice onto itself does nothing; at an offset it shifts the elements, // which copy does correctly where the ranges overlap. - if v.Ref.Obj == stmt.Dst.Obj && v.Off == "" { + if v.Ref.Obj == stmt.Dst.Obj && v.Off == stmt.Off { return nil, ReasonUnsupportedOperand } case *Invariant: @@ -501,37 +507,71 @@ func (l *lowerer) body(node ast.Stmt, block *ast.BlockStmt, trip Trip, step int6 } } - // A slice that is both stored and read at an offset is a dependence between - // iterations: iteration i reads the element iteration i+d writes. The vector - // loop runs a whole chunk of iterations statement by statement, so it keeps the - // scalar result only when the read is of a later iteration's element (d > 0, in a - // forward loop): that iteration has not run yet, so the scalar loop reads the - // value from before the loop, and the emitted loop loads it before any store of - // the chunk. A read of an earlier iteration's element (d < 0) is a recurrence, - // and a distance that is not a constant, or a loop that counts down, has no - // order loopvec can check. - stored := map[types.Object]bool{} + // Every store to a slice is at one offset, so each iteration writes its own + // element. Two offsets would have different iterations write the same element, + // in an order the vector loop, which runs one statement over a whole chunk + // before the next, does not keep. + // + // A slice that is both stored and read at another offset is a dependence + // between iterations: iteration i reads the element iteration i+d writes, d + // the read offset less the store offset. The vector loop runs a whole chunk of + // iterations statement by statement, so it keeps the scalar result only when + // the read is of a later iteration's element (d > 0, in a forward loop): that + // iteration has not run yet, so the scalar loop reads the value from before the + // loop, and the emitted loop loads it before any store of the chunk. A read of + // an earlier iteration's element (d < 0) is a recurrence, and a distance that + // is not a constant, or a loop that counts down, has no order loopvec can check. + stored := map[types.Object]string{} for _, stmt := range body { if store, ok := stmt.(Store); ok { - stored[store.Dst.Obj] = true + if off, ok := stored[store.Dst.Obj]; ok && off != store.Off { + return nil, ReasonStoreOffsets + } + stored[store.Dst.Obj] = store.Off } } for _, stmt := range body { for _, leaf := range Leaves(stmtValue(stmt)) { load, ok := leaf.(*Load) - if !ok || load.Off == "" || !stored[load.Ref.Obj] { + if !ok { continue } - distance, err := strconv.ParseInt(load.Off, 10, 64) + off, isStored := stored[load.Ref.Obj] + if !isStored || load.Off == off { + continue + } + d, ok := distance(load.Off, off) switch { - case err != nil || step != 1: + case !ok || step != 1: return nil, ReasonOffsetOfStored - case distance < 0: + case d < 0: return nil, ReasonCarriedDependence } } } + // A constant offset that puts an index below zero makes the scalar loop panic + // when it runs: on its first iteration when the start is a constant and the + // first index is negative, or on every iteration when the limit is a constant + // and even the last index is. It is left alone, since the vector loop would + // index or slice at a negative constant, which does not compile. + start, errStart := strconv.ParseInt(trip.Start, 10, 64) + limit, errLimit := strconv.ParseInt(trip.Limit, 10, 64) + negative := func(offset string) bool { + off, err := strconv.ParseInt(offset, 10, 64) + return err == nil && (errStart == nil && start+off < 0 || errLimit == nil && limit+off <= 0) + } + for _, stmt := range body { + if store, ok := stmt.(Store); ok && negative(store.Off) { + return nil, ReasonNegativeIndex + } + for _, leaf := range Leaves(stmtValue(stmt)) { + if load, ok := leaf.(*Load); ok && negative(load.Off) { + return nil, ReasonNegativeIndex + } + } + } + loop := &Loop{ Node: node, Ind: Induction{Var: l.iv, Step: step, Trip: trip}, @@ -703,7 +743,7 @@ func (l *lowerer) deadOutsideLoop(v *types.Var) bool { return true } -// store lowers one assignment to dst[i]. +// store lowers one assignment to dst[i], or to dst[i+e] for a loop-invariant e. func (l *lowerer) store(assign *ast.AssignStmt) (Store, Reason) { var op Op switch assign.Tok { @@ -716,7 +756,7 @@ func (l *lowerer) store(assign *ast.AssignStmt) (Store, Reason) { return Store{}, ReasonUnsupportedOperand } } - dst, ok := l.index(assign.Lhs[0]) + dst, off, ok := l.indexAt(assign.Lhs[0]) if !ok { return Store{}, ReasonUnsupportedDestination } @@ -732,18 +772,18 @@ func (l *lowerer) store(assign *ast.AssignStmt) (Store, Reason) { if assign.Tok == token.SHR_ASSIGN { shift = token.SHR } - val, reason = l.shiftOf(shift, &Load{Ref: dst}, assign.Rhs[0]) + val, reason = l.shiftOf(shift, &Load{Ref: dst, Off: off}, assign.Rhs[0]) default: // dst[i] op= rhs is dst[i] = dst[i] op rhs. var rhs Value if rhs, reason = l.value(assign.Rhs[0]); reason == "" { - val = &Binary{Op: op, X: &Load{Ref: dst}, Y: rhs} + val = &Binary{Op: op, X: &Load{Ref: dst, Off: off}, Y: rhs} } } if reason != "" { return Store{}, reason } - return Store{Dst: dst, Val: val}, "" + return Store{Dst: dst, Off: off, Val: val}, "" } // loadKey identifies the element a load reads: the range value variable reads the @@ -884,7 +924,7 @@ func (l *lowerer) ifStore(s *ast.IfStmt) (Store, Reason) { } sel := store.Val.(*Select) known := loadKeys(sel.Cond) - known[loadKey{store.Dst.Obj, ""}] = true + known[loadKey{store.Dst.Obj, store.Off}] = true for _, leaf := range Leaves(sel) { if load, ok := leaf.(*Load); ok && !known[loadKey{load.Ref.Obj, load.Off}] { return Store{}, ReasonConditionalLoad @@ -911,16 +951,16 @@ func (l *lowerer) ifChain(s *ast.IfStmt) (Store, Reason) { case nil: // No else: the element keeps its value. The vector loop stores it back, which // is only unobservable when the condition already reads the element. - if !loadKeys(cond)[loadKey{then.Dst.Obj, ""}] { + if !loadKeys(cond)[loadKey{then.Dst.Obj, then.Off}] { return Store{}, ReasonConditionalStore } - other = &Load{Ref: then.Dst} + other = &Load{Ref: then.Dst, Off: then.Off} case *ast.BlockStmt: store, reason := l.branch(e) if reason != "" { return Store{}, reason } - if store.Dst.Obj != then.Dst.Obj { + if store.Dst.Obj != then.Dst.Obj || store.Off != then.Off { return Store{}, ReasonUnsupportedIf } other = store.Val @@ -929,12 +969,12 @@ func (l *lowerer) ifChain(s *ast.IfStmt) (Store, Reason) { if reason != "" { return Store{}, reason } - if store.Dst.Obj != then.Dst.Obj { + if store.Dst.Obj != then.Dst.Obj || store.Off != then.Off { return Store{}, ReasonUnsupportedIf } other = store.Val } - return Store{Dst: then.Dst, Val: &Select{Cond: cond, Then: then.Val, Else: other}}, "" + return Store{Dst: then.Dst, Off: then.Off, Val: &Select{Cond: cond, Then: then.Val, Else: other}}, "" } // branch lowers the body of an if: one assignment to an element. @@ -1220,10 +1260,22 @@ func (l *lowerer) load(expr ast.Expr) (*Load, bool) { return &Load{Ref: ref, Off: off}, true } -// index matches slice[i] where slice is an identifier and i the loop index. -func (l *lowerer) index(expr ast.Expr) (*Ref, bool) { - ref, off, ok := l.indexAt(expr) - return ref, ok && off == "" +// distance is the constant read offset less the constant store offset, each "" +// for zero; ok is false when either is not a constant. +func distance(read, store string) (d int64, ok bool) { + r, s := int64(0), int64(0) + var err error + if read != "" { + if r, err = strconv.ParseInt(read, 10, 64); err != nil { + return 0, false + } + } + if store != "" { + if s, err = strconv.ParseInt(store, 10, 64); err != nil { + return 0, false + } + } + return r - s, true } // maxOffset bounds a constant offset, so that adding it to a limit cannot diff --git a/internal/loopir/plan.go b/internal/loopir/plan.go index d4bf63b..69d93ed 100644 --- a/internal/loopir/plan.go +++ b/internal/loopir/plan.go @@ -67,9 +67,9 @@ type Plan struct { // NonEmpty is set when the loop is known to run at least once. NonEmpty bool // Checked lists every slice whose length must be verified against the - // limit before the loop runs: for each store in order, its destination and - // then each slice it reads, in the order the loads are emitted. A slice read - // at several offsets is listed once for each. + // limit before the loop runs: for each store in order, its destination at + // its offset and then each slice it reads, in the order the loads are + // emitted. A slice read at several offsets is listed once for each. Checked []Access // Hoists lists the slices the loop reaches through a selector (x.f), in order: the // emitted code assigns each Name from Src once before the loop. @@ -90,9 +90,11 @@ type Access struct { } // PlanStmt is a statement whose value has been normalized; see Normalize. It -// stores to Dst, defines Temp, or folds into Acc with Op, whichever is set. +// stores to Dst at offset Off, defines Temp, or folds into Acc with Op, whichever +// is set. type PlanStmt struct { Dst *Ref + Off string Temp *Temp Acc types.Object Op Op @@ -130,7 +132,7 @@ func NewPlan(l *Loop, opts Options) (*Plan, Reason) { if !types.Identical(stmt.Dst.Elem, elem) { return nil, ReasonMixedTypes } - planned = PlanStmt{Dst: stmt.Dst, Root: Normalize(stmt.Val)} + planned = PlanStmt{Dst: stmt.Dst, Off: stmt.Off, Root: Normalize(stmt.Val)} case Let: if !types.Identical(stmt.Temp.Var.Type(), elem) { return nil, ReasonMixedTypes @@ -167,7 +169,7 @@ func NewPlan(l *Loop, opts Options) (*Plan, Reason) { } if len(p.Stmts) == 1 && p.Stmts[0].Dst != nil { if load, ok := p.Stmts[0].Root.(*Load); ok { - p.Copy = !negativeSource(l.Ind.Trip.Start, load.Off) + p.Copy = !negativeSource(l.Ind.Trip.Start, load.Off) && !negativeSource(l.Ind.Trip.Start, p.Stmts[0].Off) } } @@ -187,9 +189,10 @@ func NewPlan(l *Loop, opts Options) (*Plan, Reason) { p.Start, p.Limit, p.NonEmpty, p.TripRef = trip.Start, trip.Limit, trip.NonEmpty, trip.Slice // The generated loop stops at the length of the stored slice, so unless the // limit is exactly that, the limit is spelled out on every slice or a longer - // one would be read or written past it. + // one would be read or written past it. A store at an offset always needs it: + // the elements past limit+offset are ones the scalar loop does not write. for _, stmt := range p.Stmts { - if stmt.Dst != nil && (trip.Slice == nil || stmt.Dst.Obj != trip.Slice.Obj) { + if stmt.Dst != nil && (trip.Slice == nil || stmt.Dst.Obj != trip.Slice.Obj || stmt.Off != "") { p.Bound = trip.Limit break } @@ -200,7 +203,7 @@ func NewPlan(l *Loop, opts Options) (*Plan, Reason) { for _, stmt := range p.Stmts { if stmt.Dst != nil { stored[stmt.Dst.Obj] = true - p.Checked = appendAccess(p.Checked, Access{Ref: stmt.Dst}) + p.Checked = appendAccess(p.Checked, Access{Ref: stmt.Dst, Off: stmt.Off}) refs = appendRef(refs, stmt.Dst) } for _, leaf := range Leaves(stmt.Root) { @@ -228,10 +231,10 @@ func NewPlan(l *Loop, opts Options) (*Plan, Reason) { return p, "" } -// negativeSource reports whether a copy would slice its source at a negative -// constant, which does not compile: the loop starts at start and reads at offset -// off, and both are constants that add up to less than zero. Such a loop panics -// when it runs, which the ordinary vector loop does as well. +// negativeSource reports whether a copy would slice its source or destination at +// a negative constant, which does not compile: the loop starts at start and +// accesses at offset off, and both are constants that add up to less than zero. +// Such a loop panics when it runs, which the ordinary vector loop does as well. func negativeSource(start, off string) bool { s, errStart := strconv.ParseInt(start, 10, 64) o, errOff := strconv.ParseInt(off, 10, 64) diff --git a/internal/loopir/reason.go b/internal/loopir/reason.go index aaeca53..24c0ce4 100644 --- a/internal/loopir/reason.go +++ b/internal/loopir/reason.go @@ -30,18 +30,28 @@ const ( // adding one to it could overflow (a <= limit, or the start of a loop that // counts down, that is not a constant or something minus a constant). ReasonUnsupportedBound Reason = "loop limit is not a loop-invariant integer expression" - // ReasonUnsupportedDestination covers a destination that isn't a plain - // slice[i] element: an offset (dst[i+1]), a stride (dst[2*i]), or a - // nested index (dst[i][j]). + // ReasonUnsupportedDestination covers a destination that isn't a slice + // element at the loop index or a loop-invariant offset from it: a stride + // (dst[2*i]), a nested index (dst[i][j]), or an offset that changes. ReasonUnsupportedDestination Reason = "destination is not a simple, same-index slice element" - // ReasonOffsetOfStored covers a slice that is read at an offset from the loop - // index (a[i+1]) and also written in the loop, in a way the dependence test - // cannot clear: the offset is not a constant, or the loop counts down. + // ReasonOffsetOfStored covers a slice that is read at another offset from the + // loop index than it is written at (a[i+1] and a[i]), in a way the dependence + // test cannot clear: the distance is not a constant, or the loop counts down. ReasonOffsetOfStored Reason = "a slice is read at an offset and written in the same loop" + // ReasonStoreOffsets covers a slice stored at two different offsets from the + // loop index (a[i] and a[i+1]): two iterations write the same element, and the + // vector loop would not keep the order the scalar loop writes it in. + ReasonStoreOffsets Reason = "a slice is stored at more than one offset" // ReasonCarriedDependence covers a slice read at an earlier index than the // iteration writes (a[i] = a[i-1] + x): iteration i reads what iteration i-1 // wrote, so the iterations cannot run as a vector. ReasonCarriedDependence Reason = "an iteration reads an element an earlier iteration wrote" + // ReasonNegativeIndex covers a slice read or stored at a constant offset that + // puts an index below zero: the first index, with a constant start (a[i-1] in + // a loop from 0), or every index, with a constant limit (a[i-20] in a loop + // that stops before 16). The scalar loop panics when it runs, and the vector + // loop would index or slice at a negative constant, which does not compile. + ReasonNegativeIndex Reason = "a constant offset puts an index below zero" // ReasonUnsupportedIf covers an if that is not a select: a condition that is // not a comparison of loop values (joined by && and ||), an init statement, or // a branch that is not one assignment to an element of the same slice. diff --git a/internal/rewrite/emit.go b/internal/rewrite/emit.go index 0cd4187..c06e0eb 100644 --- a/internal/rewrite/emit.go +++ b/internal/rewrite/emit.go @@ -75,10 +75,10 @@ func (e *emitter) copyCall() string { destination, source := store.Dst.Name, load.Ref.Name switch { case e.plan.Start != "0": - destination += "[" + e.plan.Start + ":" + e.plan.Bound + "]" + destination += "[" + addOffset(e.plan.Start, store.Off) + ":" + addOffset(e.plan.Bound, store.Off) + "]" source += "[" + addOffset(e.plan.Start, load.Off) + ":]" case e.plan.Bound != "": - destination += "[:" + e.plan.Bound + "]" + destination += "[" + store.Off + ":" + addOffset(e.plan.Bound, store.Off) + "]" if load.Off != "" { source += "[" + load.Off + ":]" } @@ -120,6 +120,16 @@ func (e *emitter) operand(ref *loopir.Ref, pos, off string) string { return ref.Name + "[" + addOffset(pos, off) + ":]" } +// destination slices the stored ref from position pos at offset off. A store at +// an offset is capped at the limit plus the offset, so it never writes an element +// past the last one the scalar loop writes. +func (e *emitter) destination(ref *loopir.Ref, pos, off string) string { + if off == "" { + return e.operand(ref, pos, "") + } + return ref.Name + "[" + addOffset(pos, off) + ":" + addOffset(e.plan.Bound, off) + "]" +} + // addOffset is the expression pos + off, where off is "" for zero, a constant, or // an expression. func addOffset(pos, off string) string { @@ -211,20 +221,23 @@ func (e *emitter) statements(b *strings.Builder, full bool, copy, copies int) { } fmt.Fprintf(b, "\t%s, %s := simd.Load%sPart(%s)\n", name, lanes, e.plan.SimdType, e.operand(load.Ref, offset, load.Off)) } - // A slice stored in the loop and read ahead of the index (a[i+1]) is read - // before any store of the body: the scalar loop reads the value from before - // the loop, because the iteration that writes it has not run yet, and a - // store earlier in the body would already have changed it in the vector. - stored := map[types.Object]bool{} + // A slice stored in the loop and read ahead of the element it is stored at + // (a[i+1] beside a[i]) is read before any store of the body: the scalar loop + // reads the value from before the loop, because the iteration that writes it + // has not run yet, and a store earlier in the body would already have changed + // it in the vector. + stored := map[types.Object]string{} for _, stmt := range e.plan.Stmts { if stmt.Dst != nil { - stored[stmt.Dst.Obj] = true + stored[stmt.Dst.Obj] = stmt.Off } } for _, stmt := range e.plan.Stmts { for _, leaf := range loopir.Leaves(stmt.Root) { - if load, ok := leaf.(*loopir.Load); ok && stored[load.Ref.Obj] && load.Off != "" { - emitLoad(load) + if load, ok := leaf.(*loopir.Load); ok { + if off, isStored := stored[load.Ref.Obj]; isStored && load.Off != off { + emitLoad(load) + } } } } @@ -246,16 +259,16 @@ func (e *emitter) statements(b *strings.Builder, full bool, copy, copies int) { acc := accName(accs, copy, copies) fmt.Fprintf(b, "\t%s = %s\n", acc, e.fold(stmt, acc)) case full: - fmt.Fprintf(b, "\t%s.Store(%s)\n", e.expr(stmt.Root), e.operand(stmt.Dst, offset, "")) - delete(loaded, key{stmt.Dst.Obj, ""}) + fmt.Fprintf(b, "\t%s.Store(%s)\n", e.expr(stmt.Root), e.destination(stmt.Dst, offset, stmt.Off)) + delete(loaded, key{stmt.Dst.Obj, stmt.Off}) default: - text := fmt.Sprintf("%s.StorePart(%s)", e.expr(stmt.Root), e.operand(stmt.Dst, offset, "")) + text := fmt.Sprintf("%s.StorePart(%s)", e.expr(stmt.Root), e.destination(stmt.Dst, offset, stmt.Off)) if !haveLanes { // Nothing was loaded, so the store reports how many lanes it wrote. text, haveLanes = "_n := "+text, true } fmt.Fprintf(b, "\t%s\n", text) - delete(loaded, key{stmt.Dst.Obj, ""}) + delete(loaded, key{stmt.Dst.Obj, stmt.Off}) } } } diff --git a/testdata/json_reasons.txt b/testdata/json_reasons.txt index a778ca2..2c800a1 100644 --- a/testdata/json_reasons.txt +++ b/testdata/json_reasons.txt @@ -63,9 +63,9 @@ func DownBoundCall(dst, a []float32) { } } -func DestinationOffset(dst, a []float32) { - for i := range dst { - dst[i+1] = a[i] + 1 +func DestinationStride(dst, a []float32) { + for i := range a { + dst[2*i] = a[i] + 1 } } @@ -132,7 +132,7 @@ func Vectorized(dst, a []float32) { {"file":"$WORK/ops.go","line":37,"func":"BoundDivideByVariable","vectorized":false,"reason":"loop limit is not a loop-invariant integer expression","stage":"lower"} {"file":"$WORK/ops.go","line":43,"func":"DownStartNotLast","vectorized":false,"reason":"loop start is not a loop-invariant integer expression","stage":"lower"} {"file":"$WORK/ops.go","line":49,"func":"DownBoundCall","vectorized":false,"reason":"loop limit is not a loop-invariant integer expression","stage":"lower"} -{"file":"$WORK/ops.go","line":55,"func":"DestinationOffset","vectorized":false,"reason":"destination is not a simple, same-index slice element","stage":"lower"} +{"file":"$WORK/ops.go","line":55,"func":"DestinationStride","vectorized":false,"reason":"destination is not a simple, same-index slice element","stage":"lower"} {"file":"$WORK/ops.go","line":61,"func":"OperandCall","vectorized":false,"reason":"operand is not a same-index slice, range value, or plain scalar","stage":"lower"} {"file":"$WORK/ops.go","line":69,"func":"BodyIfGuardedStore","vectorized":false,"reason":"a store under a condition may write elements the scalar loop leaves alone","stage":"lower"} {"file":"$WORK/ops.go","line":78,"func":"BodyTwoStatements","vectorized":false,"reason":"loop body is not a sequence of indexed assignments","stage":"lower"} diff --git a/testdata/offset_negative_constant.txt b/testdata/offset_negative_constant.txt new file mode 100644 index 0000000..4657445 --- /dev/null +++ b/testdata/offset_negative_constant.txt @@ -0,0 +1,105 @@ +# A read or store at a negative constant offset is rewritten while every index it +# touches can be zero or more. When an index is a negative constant, the first one +# in a loop with a constant start (a[i-1] from 0) or every one in a loop with a +# constant limit (a[i-16] in a loop that stops before 16), the scalar loop panics +# when it runs and the vector loop would index at a negative constant, which does +# not compile, so the loop is left alone, whether the start and limit are written as +# literals, named constants or constant expressions. The rewritten package must +# build. + +loopvec -json . +cmpenv stdout stdout.want + +loopvec -split . +cmpenv stderr stderr.want +env GOEXPERIMENT=simd +exec go build . + +-- go.mod -- +module example.com/test + +go 1.27.1 +-- ops.go -- +package test + +func FirstIndexZeroConstant(dst, a []int32) { + for i := 15; i < 31; i++ { + dst[i] = a[i-15] + 1 + } +} + +func EveryIndexNegative(dst, a []int32) { + for i := 0; i < 16; i++ { + dst[i] = a[i-16] + 1 + } +} + +func FarBelow(dst, a []float32) { + for i := 0; i < 16; i++ { + dst[i] = a[i-20] * a[i] + } +} + +func FirstIndexNegative(dst, a []int32, n int) { + for i := 0; i < n; i++ { + dst[i] = a[i-1] * 3 + } +} + +func FirstIndexZero(dst, a []int32, n int) { + for i := 1; i < n; i++ { + dst[i] = a[i-1] * 3 + } +} + +const s0, lim = 0, 16 + +func NamedLimit(dst, a []int32, k int) { + for i := k; i < lim; i++ { + dst[i] = a[i-20] + 1 + } +} + +func NamedStart(dst, a []int32, n int) { + for i := s0; i < n; i++ { + dst[i] = a[i-1] * 3 + } +} + +func ExpressionStart(dst, a []int32, n int) { + for i := 1 - 1; i < n; i++ { + dst[i] = a[i-1] * 3 + } +} + +func NamedStartCopy(dst, a []int32, n int) { + for i := s0; i < n; i++ { + dst[i] = a[i-1] + } +} + +func StoreFirstIndexZero(dst, a []int32) { + for i := 15; i < 31; i++ { + dst[i-15] = a[i] * 3 + } +} + +func StoreEveryIndexNegative(dst, a []int32) { + for i := 0; i < 16; i++ { + dst[i-16] = a[i] * 3 + } +} +-- stdout.want -- +{"file":"$WORK/ops.go","line":4,"func":"FirstIndexZeroConstant","vectorized":true} +{"file":"$WORK/ops.go","line":10,"func":"EveryIndexNegative","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +{"file":"$WORK/ops.go","line":16,"func":"FarBelow","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +{"file":"$WORK/ops.go","line":22,"func":"FirstIndexNegative","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +{"file":"$WORK/ops.go","line":28,"func":"FirstIndexZero","vectorized":true} +{"file":"$WORK/ops.go","line":36,"func":"NamedLimit","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +{"file":"$WORK/ops.go","line":42,"func":"NamedStart","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +{"file":"$WORK/ops.go","line":48,"func":"ExpressionStart","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +{"file":"$WORK/ops.go","line":54,"func":"NamedStartCopy","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +{"file":"$WORK/ops.go","line":60,"func":"StoreFirstIndexZero","vectorized":true} +{"file":"$WORK/ops.go","line":66,"func":"StoreEveryIndexNegative","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +-- stderr.want -- +loopvec: rewrote 3 loop(s) in $WORK/ops.go diff --git a/testdata/offset_no_rewrite.txt b/testdata/offset_no_rewrite.txt index b3dd6bb..2918f61 100644 --- a/testdata/offset_no_rewrite.txt +++ b/testdata/offset_no_rewrite.txt @@ -4,8 +4,8 @@ # carries a value from one iteration to the next, so it is rejected with its own # reason. A distance that is not a constant and a loop that counts down have no # order loopvec can check. -# Offsets that are not the same on every iteration, a stride, and a store at an -# offset are left alone too. +# Offsets that are not the same on every iteration and a stride are left alone +# too. cp ops.go ops.go.orig loopvec -split . @@ -61,12 +61,6 @@ func OffsetAssigned(dst, a []int32, off int) { } } -func StoreAtOffset(dst, a []int32) { - for i := 0; i < len(dst)-1; i++ { - dst[i+1] = a[i] - } -} - func OffsetFromIndex(dst, a []int32) { for i := 0; i < len(dst); i++ { dst[i] = a[i+i] @@ -79,5 +73,4 @@ func OffsetFromIndex(dst, a []int32) { {"file":"$WORK/ops.go","line":22,"func":"Strided","vectorized":false,"reason":"operand is not a same-index slice, range value, or plain scalar","stage":"lower"} {"file":"$WORK/ops.go","line":28,"func":"OffsetFromSlice","vectorized":false,"reason":"operand is not a same-index slice, range value, or plain scalar","stage":"lower"} {"file":"$WORK/ops.go","line":34,"func":"OffsetAssigned","vectorized":false,"reason":"loop body is not a sequence of indexed assignments","stage":"lower"} -{"file":"$WORK/ops.go","line":41,"func":"StoreAtOffset","vectorized":false,"reason":"destination is not a simple, same-index slice element","stage":"lower"} -{"file":"$WORK/ops.go","line":47,"func":"OffsetFromIndex","vectorized":false,"reason":"operand is not a same-index slice, range value, or plain scalar","stage":"lower"} +{"file":"$WORK/ops.go","line":41,"func":"OffsetFromIndex","vectorized":false,"reason":"operand is not a same-index slice, range value, or plain scalar","stage":"lower"} diff --git a/testdata/offset_store.txt b/testdata/offset_store.txt new file mode 100644 index 0000000..4ec9b09 --- /dev/null +++ b/testdata/offset_store.txt @@ -0,0 +1,497 @@ +# A store may be at a loop-invariant offset from the index (b[i*ldb+j], +# line[off+x], dst[i-k]), the way a load may. Every store to one slice is at the +# same offset, so each iteration writes its own element, a load of that element is +# ordered after the stores before it, and a loop that counts down runs forward as +# when it stores at the index. A read of the stored slice at a constant positive +# distance past the stored element is loaded before any store of the chunk, as for +# a store at the index. The store is capped at the limit plus the offset, so the +# vector loop writes no element the scalar loop does not, and the length check +# covers the last element stored. + +# The scalar program is the reference behavior. +exec go run ./testmain +cmp stdout testmain/output + +cp ops.go ops.go.orig +loopvec -split . +! stdout . +cmpenv stderr stderr.want +cmp ops_simd.go ops_simd.go.want +cmp ops.go ops.go.guarded + +# The simd build must behave identically. +env GOEXPERIMENT=simd +exec go run ./testmain +cmp stdout testmain/output + +-- go.mod -- +module example.com/test + +go 1.27.1 +-- ops.go -- +package test + +// Copies row i of an lda-strided matrix into row i of an ldb-strided one: a copy. +func CopyRow(b, a []float64, i, n, lda, ldb int) { + for j := 0; j < n; j++ { + b[i*ldb+j] = a[i*lda+j] + } +} + +// Adds alpha*x into a row of c. +func AxpyRow(c, x []float32, alpha float32, row, n int) { + for j := 0; j < n; j++ { + c[row+j] += alpha * x[j] + } +} + +// Fills w elements starting at off. +func FillAt(line []uint16, off, w int, v uint16) { + for x := range w { + line[off+x] = v + } +} + +// Stores behind the index. +func Behind(dst, a []int32, k int) { + for i := k; i < len(a); i++ { + dst[i-k] = a[i] * 2 + } +} + +// Reads two elements ahead of the one it stores, in the slice it stores. +func Ahead(a []int32, n int) { + for i := 0; i < n; i++ { + a[i+1] = a[i+3] * 2 + } +} + +// Clamps at an offset: the condition reads the element it stores. +func ClampAt(a []int16, off, n int) { + for i := range n { + if a[off+i] < 0 { + a[off+i] = 0 + } + } +} + +// Sums a while copying it to an offset. +func SumCopy(d, a []int32, off int) int32 { + var s int32 + for i := range a { + s += a[i] + d[off+i] = a[i] + } + return s +} + +// Stores the same element twice; the second store reads the first. +func Twice(a, b []int32, off int) { + for i := 0; i < len(b); i++ { + a[off+i] = b[i] + 1 + a[off+i] *= b[i] + } +} + +// Counts down, storing at an offset and reading only the element it stores. +func Reverse(dst, a []int32, off int) { + for i := len(a) - 1; i >= 0; i-- { + dst[off+i] = a[i] + dst[off+i] + } +} +-- ops_simd.go.want -- +//go:build goexperiment.simd + +package test + +import ( + "simd" + "unsafe" +) + +// Copies row i of an lda-strided matrix into row i of an ldb-strided one: a copy. +func CopyRow(b, a []float64, i, n, lda, ldb int) { + if _loopvecOverlap(b, a, 0+(min(0, i*ldb)), n+(max(0, i*ldb)), 0+(min(0, i*lda)), n+(max(0, i*lda))) { + for j := 0; j < n; j++ { + b[i*ldb+j] = a[i*lda+j] + } + } else { + if n > 0 { + _ = b[n+(i*ldb)-1] + _ = a[n+(i*lda)-1] + copy(b[i*ldb:n+(i*ldb)], a[i*lda:]) + } + } +} + +// Adds alpha*x into a row of c. +func AxpyRow(c, x []float32, alpha float32, row, n int) { + _vcAFloat32s1 := simd.BroadcastFloat32s(alpha) + if _loopvecOverlap(c, x, 0+(min(0, row)), n+(max(0, row)), 0, n) { + for j := 0; j < n; j++ { + c[row+j] += alpha * x[j] + } + } else { + if n > 0 { + _ = c[n+row-1] + _ = x[n-1] + for _i := 0; _i < n; { + _v1, _n := simd.LoadFloat32sPart(x[_i:n]) + _v2, _ := simd.LoadFloat32sPart(c[_i+row:]) + _v1.MulAdd(_vcAFloat32s1, _v2).StorePart(c[_i+row : n+row]) + _i += _n + } + } + } +} + +// Fills w elements starting at off. +func FillAt(line []uint16, off, w int, v uint16) { + _vcUint16s2 := simd.BroadcastUint16s(v) + if w > 0 { + _ = line[w+off-1] + for _i := 0; _i < w; { + _n := _vcUint16s2.StorePart(line[_i+off : w+off]) + _i += _n + } + } +} + +// Stores behind the index. +func Behind(dst, a []int32, k int) { + _vcInt32s3 := simd.BroadcastInt32s(2) + if _loopvecOverlap(dst, a, k+(min(0, -(k))), len(a)+(max(0, -(k))), k, len(a)) { + for i := k; i < len(a); i++ { + dst[i-k] = a[i] * 2 + } + } else { + if len(a) > k { + _ = dst[len(a)-(k)-1] + _ = a[len(a)-1] + for _i := k; _i < len(a); { + _v1, _n := simd.LoadInt32sPart(a[_i:len(a)]) + _v1.Mul(_vcInt32s3).StorePart(dst[_i-(k) : len(a)-(k)]) + _i += _n + } + } + } +} + +// Reads two elements ahead of the one it stores, in the slice it stores. +func Ahead(a []int32, n int) { + _vcInt32s4 := simd.BroadcastInt32s(2) + if n > 0 { + _ = a[n] + _ = a[n+2] + for _i := 0; _i < n; { + _v1, _n := simd.LoadInt32sPart(a[_i+3:]) + _v1.Mul(_vcInt32s4).StorePart(a[_i+1 : n+1]) + _i += _n + } + } +} + +// Clamps at an offset: the condition reads the element it stores. +func ClampAt(a []int16, off, n int) { + _vcAInt16s5 := simd.BroadcastInt16s(0) + if n > 0 { + _ = a[n+off-1] + for _i := 0; _i < n; { + _v1, _n := simd.LoadInt16sPart(a[_i+off:]) + _vcAInt16s5.IfElse(_v1.Less(_vcAInt16s5), _v1).StorePart(a[_i+off : n+off]) + _i += _n + } + } +} + +// Sums a while copying it to an offset. +func SumCopy(d, a []int32, off int) int32 { + var s int32 + if _loopvecOverlap(a, d, 0, len(a), 0+(min(0, off)), len(a)+(max(0, off))) { + for i := range a { + s += a[i] + d[off+i] = a[i] + } + } else { + if len(a) > 0 { + _ = a[len(a)-1] + _ = d[len(a)+off-1] + { + _i := 0 + if len(a) >= 64 { + _a1 := simd.BroadcastInt32s(0) + _lanes := _a1.Len() + for ; _i+_lanes <= len(a); _i += _lanes { + _v1 := simd.LoadInt32s(a[_i:len(a)]) + _a1 = _a1.Add(_v1) + _v1.Store(d[_i+off : len(a)+off]) + } + var _buf1 [16]int32 + _a1.Store(_buf1[:]) + for _, _x := range _buf1[:_lanes] { + s = s + _x + } + } + for i := _i; i < len(a); i++ { + s += a[i] + d[off+i] = a[i] + } + } + } + } + return s +} + +// Stores the same element twice; the second store reads the first. +func Twice(a, b []int32, off int) { + _vcInt32s7_1 := simd.BroadcastInt32s(1) + if _loopvecOverlap(a, b, 0+(min(0, off)), len(b)+(max(0, off)), 0, len(b)) { + for i := 0; i < len(b); i++ { + a[off+i] = b[i] + 1 + a[off+i] *= b[i] + } + } else { + if len(b) > 0 { + _ = a[len(b)+off-1] + _ = b[len(b)-1] + for _i := 0; _i < len(b); { + _v1, _n := simd.LoadInt32sPart(b[_i:len(b)]) + _v1.Add(_vcInt32s7_1).StorePart(a[_i+off : len(b)+off]) + _v2, _ := simd.LoadInt32sPart(a[_i+off:]) + _v2.Mul(_v1).StorePart(a[_i+off : len(b)+off]) + _i += _n + } + } + } +} + +// Counts down, storing at an offset and reading only the element it stores. +func Reverse(dst, a []int32, off int) { + if _loopvecOverlap(dst, a, 0+(min(0, off)), len(a)+(max(0, off)), 0, len(a)) { + for i := len(a) - 1; i >= 0; i-- { + dst[off+i] = a[i] + dst[off+i] + } + } else { + if len(a) > 0 { + _ = dst[len(a)+off-1] + _ = a[len(a)-1] + for _i := 0; _i < len(a); { + _v1, _n := simd.LoadInt32sPart(a[_i:len(a)]) + _v2, _ := simd.LoadInt32sPart(dst[_i+off:]) + _v1.Add(_v2).StorePart(dst[_i+off : len(a)+off]) + _i += _n + } + } + } +} + +func _loopvecOverlap[T any](a, b []T, window ...int) bool { + aLo, aHi, bLo, bHi := 0, len(a), 0, len(b) + if len(window) == 4 { + aLo, aHi, bLo, bHi = window[0], window[1], window[2], window[3] + } + if aLo >= aHi || bLo >= bHi { + return false + } + size := unsafe.Sizeof(a[0]) + aBase := uintptr(unsafe.Pointer(unsafe.SliceData(a))) + bBase := uintptr(unsafe.Pointer(unsafe.SliceData(b))) + aStart, aEnd := aBase+uintptr(aLo)*size, aBase+uintptr(aHi)*size + bStart, bEnd := bBase+uintptr(bLo)*size, bBase+uintptr(bHi)*size + return aStart < bEnd && bStart < aEnd +} +-- ops.go.guarded -- +//go:build !goexperiment.simd + +package test + +// Copies row i of an lda-strided matrix into row i of an ldb-strided one: a copy. +func CopyRow(b, a []float64, i, n, lda, ldb int) { + for j := 0; j < n; j++ { + b[i*ldb+j] = a[i*lda+j] + } +} + +// Adds alpha*x into a row of c. +func AxpyRow(c, x []float32, alpha float32, row, n int) { + for j := 0; j < n; j++ { + c[row+j] += alpha * x[j] + } +} + +// Fills w elements starting at off. +func FillAt(line []uint16, off, w int, v uint16) { + for x := range w { + line[off+x] = v + } +} + +// Stores behind the index. +func Behind(dst, a []int32, k int) { + for i := k; i < len(a); i++ { + dst[i-k] = a[i] * 2 + } +} + +// Reads two elements ahead of the one it stores, in the slice it stores. +func Ahead(a []int32, n int) { + for i := 0; i < n; i++ { + a[i+1] = a[i+3] * 2 + } +} + +// Clamps at an offset: the condition reads the element it stores. +func ClampAt(a []int16, off, n int) { + for i := range n { + if a[off+i] < 0 { + a[off+i] = 0 + } + } +} + +// Sums a while copying it to an offset. +func SumCopy(d, a []int32, off int) int32 { + var s int32 + for i := range a { + s += a[i] + d[off+i] = a[i] + } + return s +} + +// Stores the same element twice; the second store reads the first. +func Twice(a, b []int32, off int) { + for i := 0; i < len(b); i++ { + a[off+i] = b[i] + 1 + a[off+i] *= b[i] + } +} + +// Counts down, storing at an offset and reading only the element it stores. +func Reverse(dst, a []int32, off int) { + for i := len(a) - 1; i >= 0; i-- { + dst[off+i] = a[i] + dst[off+i] + } +} +-- stderr.want -- +loopvec: rewrote 9 loop(s) in $WORK/ops.go +-- testmain/main.go -- +package main + +import ( + "fmt" + "slices" + + "example.com/test" +) + +var sizes = []int{0, 1, 2, 3, 5, 8, 9, 16, 17, 33, 64, 65, 100, 131} + +func ramp(n int, scale int32) []int32 { + s := make([]int32, n) + for i := range s { + s[i] = int32(i*i+1)*scale - 40 + } + return s +} + +func sum[T int16 | uint16 | int32 | float32 | float64](s []T) (t float64) { + for i, v := range s { + // The conversion rounds the product, so no target fuses it into the sum. + t += float64(float64(v) * float64(i+1)) + } + return t +} + +func try(f func()) (panicked bool) { + defer func() { panicked = recover() != nil }() + f() + return false +} + +func main() { + for _, n := range sizes { + const lda, ldb = 140, 150 + a := make([]float64, 4*lda) + for i := range a { + a[i] = float64(i) / 3 + } + b := make([]float64, 4*ldb) + for i := range 4 { + test.CopyRow(b, a, i, n, lda, ldb) + } + + c := make([]float32, n+7) + x := make([]float32, n) + for i := range x { + x[i] = float32(i) / 4 + } + test.AxpyRow(c, x, 3, 7, n) + test.AxpyRow(c, x, -1, 0, n) + + line := make([]uint16, n+5) + test.FillAt(line, 5, n, 9) + + d := make([]int32, n) + test.Behind(d, ramp(n+3, 1), 3) + + h := ramp(n+3, 2) + test.Ahead(h, n) + + s16 := make([]int16, n+2) + for i := range s16 { + s16[i] = int16(i*5%13) - 6 + } + test.ClampAt(s16, 2, n) + + sd := make([]int32, n+9) + total := test.SumCopy(sd, ramp(n, 3), 9) + + tw := make([]int32, n+11) + test.Twice(tw, ramp(n, 1), 11) + + rv := ramp(n+4, 5) + test.Reverse(rv, ramp(n, 1), 4) + + fmt.Println(n, sum(b), sum(c), sum(line), sum(d), sum(h), sum(s16), total, sum(sd), sum(tw), sum(rv)) + } + + // Slices too short for the loop panic before the loop writes anything wrong; + // each call gets its own copy, so what a panicking loop wrote is not observed. + x := make([]float32, 37) + c := make([]float32, 40) + fmt.Println( + try(func() { test.CopyRow(make([]float64, 20), make([]float64, 30), 1, 11, 10, 10) }), + try(func() { test.AxpyRow(slices.Clone(c), x, 3, 4, 37) }), + try(func() { test.AxpyRow(c, x, 1, 0, -2) }), + try(func() { test.FillAt(make([]uint16, 70), 69, 2, 7) }), + try(func() { test.Behind(make([]int32, 40), ramp(43, 1), 2) }), + try(func() { test.SumCopy(make([]int32, 10), ramp(9, 1), 2) }), + ) + + // Overlapping views take the original loop, so the result is the scalar one. + p := make([]float32, 40) + for i := range p { + p[i] = float32(i%7) - 2 + } + test.AxpyRow(p[1:], p, 1, 0, 30) + q := ramp(40, 1) + fmt.Println(sum(p), test.SumCopy(q, q[:39], 1), sum(q)) +} +-- testmain/output -- +0 0 0 0 0 -200 -8 0 0 0 -100 +1 98280 0 54 -60 -288 4 -37 -370 17784 -70 +2 197142.6666666667 6.25 117 -152 -264 4 -71 -744 36062 242 +3 296590.6666666667 19.75 189 -236 -16 9 -96 -1044 52722 1012 +5 497253.3333333334 72.5 360 -160 1720 45 -95 -1020 73868 4810 +8 802704 231 684 1704 10184 69 124 2556 79160 20412 +9 905720 309 810 3180 15472 124 279 5346 92160 29642 +16 1.6438933333333335e+06 1310 1944 39440 118232 232 3128 70040 2.646768e+06 198300 +17 1.7518160000000002e+06 1530 2142 50388 146816 346 3859 89046 3.971336e+06 244122 +33 3.5654373333333326e+06 8756 6534 690404 1.649184e+06 1022 33099 1.147806e+06 2.524566e+08 2.583162e+06 +64 7.575680000000004e+06 54264 21600 9.22272e+06 2.0135352e+07 3512 253664 1.4654816e+07 1.3013051328e+10 3.0776732e+07 +65 7.716626666666672e+06 56680 22230 9.80122e+06 2.136816e+07 3512 265915 1.556139e+07 1.4264262584e+10 3.264953e+07 +100 1.3143399999999993e+07 192637.5 49950 5.33583e+07 1.127652e+08 8692 981350 8.313785e+07 1.8293135886e+11 1.710221e+08 +131 1.881247333333331e+07 419363.75 83709 1.5496514e+08 3.23149552e+08 14389 2217568 2.393763e+08 9.07732130162e+11 4.8864562e+08 +true true false true true true +8775 -1521 -31980 diff --git a/testdata/offset_store_no_rewrite.txt b/testdata/offset_store_no_rewrite.txt new file mode 100644 index 0000000..f5def61 --- /dev/null +++ b/testdata/offset_store_no_rewrite.txt @@ -0,0 +1,75 @@ +# A store at an offset is left alone when two iterations could touch the same +# element in an order the vector loop does not keep: a slice stored at two +# offsets, a read of the stored slice behind the stored element (a recurrence), a +# read at a distance that is not a constant, a read at another offset in a loop +# that counts down, and an offset the body changes. A stride and a nested index +# are not an offset. + +cp ops.go ops.go.orig +loopvec -split . +! stdout . +! stderr . +! exists ops_simd.go +cmp ops.go ops.go.orig + +loopvec -json . +cmpenv stdout stdout.want + +-- go.mod -- +module example.com/test + +go 1.27.1 +-- ops.go -- +package test + +func TwoOffsets(a, b, c []int32) { + for i := 0; i < len(b); i++ { + a[i] = b[i] + a[i+1] = c[i] + } +} + +func Recurrence(a, b []int32) { + for i := 0; i < len(b); i++ { + a[i+1] = a[i] + b[i] + } +} + +func DistanceUnknown(a []int32, k, n int) { + for i := 0; i < n; i++ { + a[i+k] = a[i] * 2 + } +} + +func CountingDownAhead(a []int32) { + for i := len(a) - 4; i >= 0; i-- { + a[i+1] = a[i+3] * 2 + } +} + +func OffsetAssigned(dst, a []int32, off int) { + for i := range a { + dst[off+i] = a[i] + off++ + } +} + +func Stride(dst, a []int32) { + for i := range a { + dst[2*i] = a[i] + } +} + +func Nested(dst [][]int32, a []int32) { + for i := range a { + dst[0][i] = a[i] + } +} +-- stdout.want -- +{"file":"$WORK/ops.go","line":4,"func":"TwoOffsets","vectorized":false,"reason":"a slice is stored at more than one offset","stage":"lower"} +{"file":"$WORK/ops.go","line":11,"func":"Recurrence","vectorized":false,"reason":"an iteration reads an element an earlier iteration wrote","stage":"lower"} +{"file":"$WORK/ops.go","line":17,"func":"DistanceUnknown","vectorized":false,"reason":"a slice is read at an offset and written in the same loop","stage":"lower"} +{"file":"$WORK/ops.go","line":23,"func":"CountingDownAhead","vectorized":false,"reason":"a slice is read at an offset and written in the same loop","stage":"lower"} +{"file":"$WORK/ops.go","line":29,"func":"OffsetAssigned","vectorized":false,"reason":"loop body is not a sequence of indexed assignments","stage":"lower"} +{"file":"$WORK/ops.go","line":36,"func":"Stride","vectorized":false,"reason":"destination is not a simple, same-index slice element","stage":"lower"} +{"file":"$WORK/ops.go","line":42,"func":"Nested","vectorized":false,"reason":"destination is not a simple, same-index slice element","stage":"lower"} diff --git a/tsvc/README.md b/tsvc/README.md index af5d5f0..3bf05fc 100644 --- a/tsvc/README.md +++ b/tsvc/README.md @@ -107,8 +107,8 @@ by `n*2^-24`, for the kernels that vectorize only with `-fp-reassoc`). | s421 | storage classes and equivalencing | equivalence- no overlap | bits | vectorize | yes | | s1421 | storage classes and equivalencing | equivalence- no overlap | bits | vectorize | yes | | s422 | storage classes and equivalencing | common and equivalence statement; anti-dependence, threshold of 4 | bits | vectorize | yes | -| s423 | storage classes and equivalencing | common and equivalenced variables - with anti-dependence; do this again here | bits | vectorize | no | -| s424 | storage classes and equivalencing | common and equivalenced variables - overlap; vectorizeable in strips of 64 or less; do this again here | bits | vectorize | no | +| s423 | storage classes and equivalencing | common and equivalenced variables - with anti-dependence; do this again here | bits | vectorize | yes | +| s424 | storage classes and equivalencing | common and equivalenced variables - overlap; vectorizeable in strips of 64 or less; do this again here | bits | vectorize | yes | | s431 | parameters | parameter statement | bits | vectorize | no | | s441 | non-logical if's | arithmetic if | fused | vectorize | no | | s443 | non-logical if's | arithmetic if | fused | vectorize | no | @@ -160,7 +160,7 @@ loop with a real dependence between iterations is correct to leave alone. the expectation copied from the kernel's directive so the file reads on its own: ``` -# loopvec TSVC coverage: vectorized 22, gaps 70, declined 1, skipped 0 (of 93) +# loopvec TSVC coverage: vectorized 24, gaps 68, declined 1, skipped 0 (of 93) s000 vectorize yes s111 vectorize no loop start is not 0 (or len(s)-1 when counting down) ... diff --git a/tsvc/testdata/coverage.txt b/tsvc/testdata/coverage.txt index 420b54c..4f133f4 100644 --- a/tsvc/testdata/coverage.txt +++ b/tsvc/testdata/coverage.txt @@ -1,8 +1,8 @@ -# loopvec TSVC coverage: vectorized 22, gaps 70, declined 1, skipped 0 (of 93) +# loopvec TSVC coverage: vectorized 24, gaps 68, declined 1, skipped 0 (of 93) s000 vectorize yes s111 vectorize no loop step is not i++ (or i-- when counting down) s1111 vectorize no destination is not a simple, same-index slice element -s112 vectorize no destination is not a simple, same-index slice element +s112 vectorize no a slice is read at an offset and written in the same loop s1112 vectorize yes s113 vectorize no operand is not a same-index slice, range value, or plain scalar s1113 vectorize no operand is not a same-index slice, range value, or plain scalar @@ -24,8 +24,8 @@ s141 vectorize no destination is not a simple, same-index slice element s162 vectorize no a slice is read at an offset and written in the same loop s171 vectorize no destination is not a simple, same-index slice element s172 vectorize no loop step is not i++ (or i-- when counting down) -s173 vectorize no destination is not a simple, same-index slice element -s174 vectorize no destination is not a simple, same-index slice element +s173 vectorize no a slice is read at an offset and written in the same loop +s174 vectorize no a slice is read at an offset and written in the same loop s175 vectorize no loop step is not i++ (or i-- when counting down) s176 vectorize no operand is not a same-index slice, range value, or plain scalar s211 vectorize no an iteration reads an element an earlier iteration wrote @@ -36,10 +36,10 @@ s2233 vectorize no destination is not a simple, same-index slice element s235 vectorize no destination is not a simple, same-index slice element s241 vectorize yes s243 vectorize yes -s2244 vectorize no destination is not a simple, same-index slice element +s2244 vectorize no a slice is stored at more than one offset s251 vectorize yes s1251 vectorize yes -s3251 vectorize no destination is not a simple, same-index slice element +s3251 vectorize no an iteration reads an element an earlier iteration wrote s252 vectorize no operand is not a same-index slice, range value, or plain scalar s254 vectorize no operand is not a same-index slice, range value, or plain scalar s255 vectorize no operand is not a same-index slice, range value, or plain scalar @@ -73,8 +73,8 @@ s352 vectorize no loop step is not i++ (or i-- when counting down) s421 vectorize yes s1421 vectorize yes s422 vectorize yes -s423 vectorize no destination is not a simple, same-index slice element -s424 vectorize no destination is not a simple, same-index slice element +s423 vectorize yes +s424 vectorize yes s431 vectorize no a slice is read at an offset and written in the same loop s441 vectorize no a slice is read only under a condition s443 vectorize no a slice is read only under a condition