Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
57 changes: 13 additions & 44 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -582,11 +582,6 @@ jobs:
registry:
- test/registry-mocked
- test/registry
sdk:
- any
include:
- registry: test/registry-mocked
sdk: omni
timeout-minutes: ${{ vars.DEFAULT_JOB_TIMEOUT_MIN == '' && 120 || vars.DEFAULT_JOB_TIMEOUT_MIN }}
steps:

Expand Down Expand Up @@ -902,7 +897,6 @@ jobs:
- name: Run robot mocked functional tests
env:
PYTHONPATH: '${{ env.PYTHONPATH }}:${{ github.workspace }}/test/python'
if: success() && matrix.sdk != 'omni'
run: |
if [ "${{ matrix.registry }}" = "test/registry" ]; then
robot \
Expand All @@ -916,74 +910,49 @@ jobs:
fi

- name: Output from mocked functional tests
if: always() && matrix.sdk != 'omni'
if: always()
run: |
cat ./test/robot/reports/output.xml

# Omni cutover readiness: report the full suite result, then fail CI if it fails.
- name: Omni cutover readiness traffic light
if: matrix.sdk == 'omni'
env:
PYTHONPATH: '${{ env.PYTHONPATH }}:${{ github.workspace }}/test/python'
# "omni":"all" routes unprefixed providers through omnisdk; "unstable" alone only exposes stackql_unstable_*.
STACKQL_ROBOT_DEFAULT_PREVIEW: '{"unstable":true,"staging":true,"omni":"all"}'
- name: Robot results traffic light
if: always()
run: |
pgrep -f flask | xargs kill -9 || true
if python cicd/python/build.py --robot-test --config='{ "variables": { "SHOULD_RUN_DOCKER_EXTERNAL_TESTS": "true" } }'; then
robot_status=0
else
robot_status=$?
fi
if [ -f test/robot/reports/output.xml ]; then
if python test/python/stackql_test_tooling/robot_summary.py \
python test/python/stackql_test_tooling/robot_summary.py \
--output test/robot/reports/output.xml \
--title "Omni cutover readiness" \
--require-100-percent \
--context "Mocked functional suite, every run opted in with STACKQL_ROBOT_DEFAULT_PREVIEW=${STACKQL_ROBOT_DEFAULT_PREVIEW}" \
>> "${GITHUB_STEP_SUMMARY}"; then
summary_status=0
else
summary_status=$?
fi
--title "Robot results" \
--require-100-percent >> "${GITHUB_STEP_SUMMARY}"
else
{
echo "# 🔴 Omni cutover readiness"
echo "# 🔴 Robot results"
echo ""
echo "No robot output was produced; the suite did not run."
} >> "${GITHUB_STEP_SUMMARY}"
summary_status=1
fi
if [ "$robot_status" -ne 0 ] || [ "$summary_status" -ne 0 ]; then
exit 1
fi

- name: Output from omni functional tests
if: always() && matrix.sdk == 'omni'
run: |
cat ./test/robot/reports/output.xml

- name: Upload Robot Reports Artifact
uses: actions/upload-artifact@v7.0.1
if: always() && github.repository == 'stackql/stackql-devel' && matrix.registry == 'test/registry-mocked'
with:
name: stackql_linux_${{ matrix.sdk }}_robot_reports
name: stackql_linux_robot_reports
path: test/robot/reports

- name: Run robot mocked functional tests with aggressive concurrency
if: success() && matrix.registry != 'test/registry' && matrix.sdk != 'omni'
if: success() && matrix.registry != 'test/registry'
run: |
echo "## Stray flask apps to be killed before robot tests ##"
pgrep -f flask | xargs kill -9 || true
echo "## End ##"
python cicd/python/build.py --robot-test --config='{ "variables": { "SHOULD_RUN_DOCKER_EXTERNAL_TESTS": "true", "CONCURRENCY_LIMIT": -1 } }'

- name: Output from mocked functional tests with aggressive concurrency
if: always() && matrix.registry != 'test/registry' && matrix.sdk != 'omni'
if: always() && matrix.registry != 'test/registry'
run: |
cat ./test/robot/reports/output.xml

- name: Run robot integration tests
if: env.AZURE_CLIENT_SECRET != '' && startsWith(env.STATE_SOURCE_TAG, 'build-release') && matrix.registry != 'test/registry' && matrix.sdk != 'omni'
if: env.AZURE_CLIENT_SECRET != '' && startsWith(env.STATE_SOURCE_TAG, 'build-release') && matrix.registry != 'test/registry'
env:
AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
Expand All @@ -996,7 +965,7 @@ jobs:
python cicd/python/build.py --robot-test-integration

- name: install and test deb package
if: matrix.registry != 'test/registry' && matrix.sdk != 'omni'
if: matrix.registry != 'test/registry'
env:
PYTHONPATH: '${{ env.PYTHONPATH }}:${{ github.workspace }}/test/python'
IS_SKIP_MCP_TEST: 'true'
Expand All @@ -1015,7 +984,7 @@ jobs:
echo "stackql version: ${stackqlVersion}"

- name: Output from mocked deb package functional tests
if: always() && matrix.registry != 'test/registry' && matrix.sdk != 'omni'
if: always() && matrix.registry != 'test/registry'
env:
PYTHONPATH: '${{ env.PYTHONPATH }}:${{ github.workspace }}/test/python'
run: |
Expand Down
17 changes: 9 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@

These guidelines help contributors, human and otherwise, understand and work effectively on this repository.

We also encourage reading [`docs/developer_guide.md`](/docs/developer_guide.md) for further useful information. For deeper understanding of the "brains" of `stackql`, it is worthwhile to consult [the `AGENTS.md` file of `any-sdk`](https://github.com/stackql/any-sdk/blob/main/AGENTS.md) and linked documents from there.
We also encourage reading [`docs/developer_guide.md`](/docs/developer_guide.md) for further useful information. For deeper understanding of the "brains" of `stackql`, it is worthwhile to consult [the omnisdk developer invariants](https://github.com/stackql-labs/omnisdk/blob/main/docs/developer/invariants.md) and linked documents from there.

We have upgraded to golang `>= 1.27.3` in order to take advantage of [time simulation and other capabilities in `testing/synctest`](https://go.dev/blog/testing-time).


## Project Structure & Module Organization

- Entrypoint: [`stackql/main.go`](/stackql/main.go).
- Ideally, foreign system semantics are dealt with in the `any-sdk` repository.
- Ideally, foreign system semantics are dealt with in the `omnisdk` repository.
- Loose adherence to popular idioms:
- App internals in [`internal`](/internal).
- Re-usable modules in [`pkg`](/pkg).
Expand All @@ -31,7 +31,7 @@ Common commands (run from repo root):

## Coding Style & Naming Conventions

- Program to abstractions; concrete types and foreign-system semantics belong behind interfaces, ideally in `any-sdk`.
- Program to abstractions; concrete types and foreign-system semantics belong behind interfaces, in `omnisdk`.
- `gofmt`/`goimports` formatting; `golangci-lint` must pass (see [`.golangci.yml`](/.golangci.yml)).
- Go identifier conventions: exported `CamelCase`, unexported `camelCase`, acronyms uppercase (`HTTPClient`, not `HttpClient`). Package names short and lowercase.

Expand All @@ -43,7 +43,7 @@ Common commands (run from repo root):
## Tools & Resources

- Inspect provider surface via the running server: `SHOW PROVIDERS;`, `SHOW SERVICES IN <provider>;`, `DESCRIBE <resource>;`.
- Provider definitions live in the [`stackql-provider-registry`](https://github.com/stackql/stackql-provider-registry); the request-execution brain lives in [`any-sdk`](https://github.com/stackql/any-sdk).
- Provider definitions live in the [`stackql-provider-registry`](https://github.com/stackql/stackql-provider-registry); the request-execution brain lives in [`omnisdk`](https://github.com/stackql-labs/omnisdk).

## Commit & Pull Request Guidelines

Expand All @@ -61,15 +61,16 @@ StackQL supports flexible configuration management, including authentication sec
## SQLite backend

- The embedded backend is `modernc.org/sqlite` (pure Go, no cgo), pinned >= v1.57.0
(caller-constructed `Driver` function registration). It is consumed via
[`any-sdk`](https://github.com/stackql/any-sdk) `public/sqlengine`, which registers
(caller-constructed `Driver` function registration). It is owned by
`internal/stackql/sqlengine`, which registers
the driver under the name `stackql-sqlite`.
- DSN construction happens only via the central `BuildDSN` in any-sdk `public/sqlengine`
- SQL connections, staging, and backend execution remain in StackQL; Omni contains no SQL staging.
- DSN construction happens only via the central `BuildDSN` in `internal/stackql/sqlengine`
(legacy mattn-style params such as `_busy_timeout=5000` are translated to modernc
`_pragma=busy_timeout(5000)` syntax there); driver errors are inspected only via the
error predicates in that package (`isBusy`, `isConstraintViolation`); the custom
extension functions (`split_part`, `regexp_like`, `regexp_substr`, `regexp_replace`,
`json_equal`, `aws_policy_equal`) live only in any-sdk `public/sqlfuncs` and are
`json_equal`, `aws_policy_equal`) live only in `internal/stackql/sqlengine/internal/sqlfuncs` and are
registered on the constructed driver, never globally.
- Never reintroduce `mattn/go-sqlite3`, cgo, the `sqlite_stackql` build tag, or
package-global function registration. All builds and release artifacts are
Expand Down
2 changes: 1 addition & 1 deletion cicd/go/dependency-metadata/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ func description(path string) string {
switch path {
case "github.com/stackql/stackql-parser":
return "StackQL SQL parser, derived from Vitess"
case "github.com/stackql/any-sdk":
case "github.com/stackql-labs/omnisdk":
return "StackQL provider SDK"
case "github.com/stackql/readline":
return "StackQL fork of chzyer/readline for interactive line editing"
Expand Down
36 changes: 9 additions & 27 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,10 @@ go 1.27.3

require (
github.com/DATA-DOG/go-sqlmock v1.5.1
github.com/Masterminds/semver v1.4.2
github.com/PaesslerAG/gval v1.0.0
github.com/PaesslerAG/jsonpath v0.1.1
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e
github.com/getkin/kin-openapi v0.88.0
github.com/google/go-jsonnet v0.17.0
github.com/google/licensecheck v0.3.1
github.com/google/uuid v1.6.0
Expand All @@ -20,37 +22,33 @@ require (
github.com/spf13/cobra v1.10.2
github.com/spf13/pflag v1.0.10
github.com/spf13/viper v1.10.1
github.com/stackql-labs/omnisdk v0.1.7-alpha02
github.com/stackql/any-sdk v0.6.1-beta02
github.com/stackql-labs/omnisdk v0.1.7-beta05
github.com/stackql/go-suffix-map v0.0.1-alpha02
github.com/stackql/psql-wire v0.1.2-beta02
github.com/stackql/stackql-parser v0.0.16-beta01
github.com/stackql/stackql-provider-registry v0.0.2-alpha01
github.com/stackql/stackql-provider-registry/signing v0.1.0-alpha01
github.com/stretchr/testify v1.11.1
github.com/xo/dburl v0.23.2
golang.org/x/mod v0.38.0
golang.org/x/sync v0.22.0
gonum.org/v1/gonum v0.17.0
gopkg.in/natefinch/lumberjack.v2 v2.2.1
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
gotest.tools v2.2.0+incompatible
modernc.org/sqlite v1.59.0
)

require (
cloud.google.com/go/compute/metadata v0.9.0 // indirect
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 // indirect
github.com/99designs/keyring v1.2.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.17.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.8.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.3.2 // indirect
github.com/BurntSushi/toml v1.4.0 // indirect
github.com/JohnCGriffin/overflow v0.0.0-20211019200055-46fa312c352c // indirect
github.com/Masterminds/semver v1.4.2 // indirect
github.com/PaesslerAG/gval v1.0.0 // indirect
github.com/PaesslerAG/jsonpath v0.1.1 // indirect
github.com/antchfx/xmlquery v1.3.10 // indirect
github.com/antchfx/xpath v1.3.6 // indirect
github.com/apache/arrow-go/v18 v18.0.0 // indirect
github.com/aws/aws-sdk-go-v2 v1.41.5 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect
Expand Down Expand Up @@ -83,21 +81,15 @@ require (
github.com/fsnotify/fsnotify v1.5.1 // indirect
github.com/fullstorydev/grpcurl v1.9.3 // indirect
github.com/gabriel-vasile/mimetype v1.4.7 // indirect
github.com/ghodss/yaml v1.0.0 // indirect
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
github.com/go-openapi/jsonpointer v0.19.5 // indirect
github.com/go-openapi/swag v0.21.1 // indirect
github.com/goccy/go-json v0.10.4 // indirect
github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 // indirect
github.com/gofrs/flock v0.10.0 // indirect
github.com/golang-jwt/jwt/v5 v5.3.1 // indirect
github.com/golang/glog v1.2.5 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/flatbuffers v24.12.23+incompatible // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/jsonschema-go v0.4.3 // indirect
github.com/gorilla/mux v1.8.0 // indirect
github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
Expand All @@ -107,17 +99,13 @@ require (
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jhump/protoreflect v1.18.0 // indirect
github.com/jhump/protoreflect/v2 v2.0.0-beta.1 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/klauspost/compress v1.18.7 // indirect
github.com/klauspost/cpuid/v2 v2.2.9 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-isatty v0.0.24 // indirect
github.com/mattn/go-runewidth v0.0.13 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mtibben/percent v0.2.1 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/oracle/oci-go-sdk/v65 v65.120.0 // indirect
github.com/pelletier/go-toml v1.9.4 // indirect
github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
Expand All @@ -127,20 +115,16 @@ require (
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rivo/uniseg v0.2.0 // indirect
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/segmentio/asm v1.1.3 // indirect
github.com/segmentio/encoding v0.5.4 // indirect
github.com/sony/gobreaker/v2 v2.4.0 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.4.1 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spiffe/go-spiffe/v2 v2.7.0 // indirect
github.com/stackql/stackql-provider-registry v0.0.2-alpha01 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/subosito/gotenv v1.2.0 // indirect
github.com/xo/dburl v0.23.2 // indirect
github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
Expand All @@ -162,11 +146,9 @@ require (
google.golang.org/grpc v1.83.2 // indirect
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/ini.v1 v1.66.2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/libc v1.75.7 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.12.1 // indirect
modernc.org/sqlite v1.59.0 // indirect
)

replace github.com/chzyer/readline => github.com/stackql/readline v0.0.2-alpha05
Loading
Loading