Repository navigation
Conversation
… zero A read at a negative constant offset in a loop with a constant limit, such as dst[i] = a[i-20] + 1 in a loop that stops before 16, was rewritten into code that does not compile: the length check of the vector loop is _ = a[16-21], a negative constant index. -split wrote that file, and the toolexec wrapper fell back to the original package. Every index such a loop reads is below zero, so the scalar loop panics on its first iteration. It is now left alone with its own reason, the way a constant limit of zero or less already is. A read whose last index is zero or more is still rewritten. The vectorized set of std, gonum and github.com/thesyncim/goav1 does not change.
…nt offset A loop that stores at an offset from the index, such as a row of a 2-D slice (b[i*ldb+j] = a[i*lda+j], c[row+j] += alpha*x[j]) or a fill that starts part way into a slice (line[off+x] = v), was rejected: loads could be at an offset, but every store had to be at exactly the loop index. A store may now be at the same offsets a load may: i+e, e+i and i-e for a loop-invariant integer e. Every store to one slice must be at the same offset, so each iteration still writes its own element; two offsets are rejected with their own reason. A load of the stored slice at that offset is the same element, ordered after the stores before it as at the index. The dependence test for a load at another offset measures the distance from the store offset instead of from the index, so a constant positive distance in a forward loop is still loaded before any store of the chunk, and a negative or variable distance is rejected as before. An if that selects between stores, and the copy builtin, take the offset too. The emitted store is capped at the limit plus the offset, so the vector loop never writes an element the scalar loop does not, and the length check and the overlap window cover the elements stored. A constant offset that puts every stored index below zero is left alone, as for a load. The vectorized set gains one loop in std (an image/jpeg fill), sixteen in gonum (dlacpy, dlaset, dlascl and dlarfb rows, and testlapack helpers) and five in github.com/thesyncim/goav1; TSVC s423 and s424 now vectorize and match their goldens.
…op so its checks fold Every element-wise loop advanced one partial vector at a time: LoadPart and StorePart on every iteration, each working out how many elements remain. On arm64, which has no masked loads, that bookkeeping costs more than the operation: on an Apple M4 Max the vector loop of AccumRowInt32s ran 35-58% slower than the scalar loop, and DiffInt32s, ShiftAddInt32s and ReadAfterStoreInt32s up to twice as slow. The loop now loads and stores whole vectors while one fits before the limit, and uses one partial vector for what is left, as the reductions already did. Its condition is _i <= limit-_lanes rather than _i+_lanes <= limit: with the length checks before the loop, the compiler then proves every operand slice long enough and drops the bounds checks of slicing it, which halves the main loop again. The reductions take the same condition. On the M4 Max the bench suite is 44% faster than before (geomean), and 62% faster than the scalar build where it was 32%; every kernel is now faster than its scalar loop except MinUint16s at 64 elements (+2%). Because the partial loads now only see fewer elements than a vector holds, the emulated Load*Part of golang/go#82001, which returns len(s), returns the right count, and GODEBUG=simd=0 builds produce the scalar result.
Owner
Author
|
Closing: obsolete scratch branch, superseded (see description). The branch can be deleted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by #3, which no longer depends on #2. Kept only because the branch could not be deleted from this session; close it and delete the branch.