From 0801416e924a97e75ce3ad3515162e7a7ae7e8bc Mon Sep 17 00:00:00 2001 From: Drew Newberry Date: Thu, 13 Aug 2026 19:37:02 -0700 Subject: [PATCH 1/2] refactor(compute): negotiate external driver behavior Signed-off-by: Drew Newberry --- .../skills/debug-openshell-cluster/SKILL.md | 6 +- architecture/compute-runtimes.md | 10 +- crates/openshell-core/src/driver_utils.rs | 25 ++- crates/openshell-driver-docker/src/lib.rs | 4 + .../openshell-driver-kubernetes/src/driver.rs | 1 + crates/openshell-driver-podman/src/driver.rs | 4 + crates/openshell-driver-vm/src/driver.rs | 13 +- crates/openshell-server/src/cli.rs | 41 ++--- crates/openshell-server/src/compute/mod.rs | 167 ++++++++++++------ .../openshell-server/src/gateway_listener.rs | 41 ++--- crates/openshell-server/src/grpc/policy.rs | 17 +- crates/openshell-server/src/grpc/sandbox.rs | 9 +- .../openshell-server/src/grpc/validation.rs | 69 +++----- crates/openshell-server/src/lib.rs | 38 ++-- crates/openshell-server/src/test_support.rs | 14 ++ docs/reference/gateway-config.mdx | 20 ++- docs/reference/sandbox-compute-drivers.mdx | 54 +++--- proto/compute_driver.proto | 20 ++- 18 files changed, 330 insertions(+), 223 deletions(-) diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 6a9bcd25b9..fad1e7111a 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -71,7 +71,7 @@ Use gateway metadata, deployment values, or the user's setup notes to identify t Before debugging the compute platform, inspect gateway logs for failures in dependencies initialized before the listener becomes ready. -For out-of-tree compute drivers, confirm the custom driver name and socket agree across CLI flags or `gateway.toml`, and that the operator-owned driver is running before the gateway starts: +For out-of-tree compute drivers, confirm the selected driver name and socket agree across CLI flags or `gateway.toml`, and that the operator-owned driver is running before the gateway starts: ```bash rg -n 'compute_drivers|socket_path' /etc/openshell/gateway.toml @@ -80,7 +80,7 @@ journalctl -u --no-pager --lines=200 journalctl -u openshell-gateway --no-pager --lines=200 ``` -The custom driver name must not be a reserved built-in name (`docker`, `podman`, `kubernetes`, or `vm`). The socket must be accessible only to the intended gateway identity. Check gateway logs for connection errors, `GetCapabilities` failures, or an unexpected advertised driver name. The gateway does not create or supervise out-of-tree driver processes or sockets. +Custom names use `[openshell.drivers.].socket_path`. A launch-time `--compute-driver-socket` override may also use `docker`, `podman`, `kubernetes`, or `vm`; the endpoint then takes precedence over built-in construction. The socket must be accessible only to the intended gateway identity. Check gateway logs for connection errors, `GetCapabilities` failures, or an unexpected advertised driver name. The advertised name is diagnostic metadata; negotiated features control optional behavior. The gateway does not create or supervise operator-supplied driver processes or sockets. For configured gateway interceptors, inspect `[[openshell.gateway.interceptors]]`, their Unix or network endpoints, and gateway startup logs: @@ -586,7 +586,7 @@ openshell logs | Binary WebSocket message passes without a middleware RPC | Binary is unsupported by the V1 text-message binding under both `on_error` modes | `WEBSOCKET_MIDDLEWARE_COVERAGE state=unsupported_message_type`; the next text RPC may have a valid sequence gap | | WebSocket messages stop reaching middleware after one failure | A fail-open stage stream was disabled for the rest of the connection | `openshell.middleware.websocket_stage_disabled`; middleware timeout/stream/protocol logs. A per-message capacity bypass alone leaves the stage active. Reconnect to create a fresh stream after a genuine stream failure | | Supervisor repeatedly fails to install middleware after enabling gateway JWT signing | Extension credential minting, distribution, or authenticated service connection failed; last-known-good registry remains active | Gateway `RefreshSandboxToken` logs, sandbox configuration events, service token-verification logs, registration TLS/audience settings | -| Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or configured with a reserved/mismatched name | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs | +| Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or selected name does not match its endpoint/config key | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs | | Sandbox remains `Stopping` or `Starting` | Driver stop/start failed, retained resource is missing, or a fresh supervisor has not connected | Gateway and driver logs; `docker inspect`, `podman inspect`, Agent Sandbox status/PVC, or VM state marker and launcher process | | Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials | | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled | diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 292d52687d..e0e260f885 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -23,6 +23,14 @@ references to gateway-internal types. The gateway owns the public `SandboxPhase::Ready` decision. This applies equally to extension drivers implementing `ComputeDriver` out of tree. +`compute_driver.proto` is the supported gateway/driver extension boundary. +At initialization the gateway snapshots additive feature values from +`GetCapabilities`; unknown values are ignored. Post-initialization lifecycle +and policy behavior does not depend on the configured or advertised driver +name. Startup intent reconciliation and native process-identity defaults are +enabled only by their corresponding features. Driver-requested listeners are +structurally validated and remain restricted to sandbox callback RPCs. + Drivers own runtime-specific platform event interpretation. When an event should drive client provisioning UI, the driver attaches the shared `openshell.progress.*` metadata defined in `openshell-core` instead of requiring @@ -152,7 +160,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`. | Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). | | Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. | | VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. | -| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = [""]` entry with `[openshell.drivers.].socket_path`, or at launch time by pairing `--drivers ` with `--compute-driver-socket=`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. | +| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = [""]` entry with `[openshell.drivers.].socket_path`, or at launch time by pairing `--drivers ` with `--compute-driver-socket=`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. | Per-sandbox CPU and memory values currently enter the driver layer through template resource limits. Docker and Podman apply them as runtime limits. diff --git a/crates/openshell-core/src/driver_utils.rs b/crates/openshell-core/src/driver_utils.rs index ae621fde08..9ed0f45d04 100644 --- a/crates/openshell-core/src/driver_utils.rs +++ b/crates/openshell-core/src/driver_utils.rs @@ -5,7 +5,7 @@ use std::path::{Path, PathBuf}; -use crate::proto::compute::v1::{DriverSandbox, GetCapabilitiesResponse}; +use crate::proto::compute::v1::{ComputeDriverFeature, DriverSandbox, GetCapabilitiesResponse}; pub use crate::container_paths::{ SANDBOX_TOKEN_MOUNT_PATH, SUPERVISOR_CONTAINER_BINARY, SUPERVISOR_CONTAINER_DIR, @@ -380,11 +380,13 @@ pub fn build_capabilities_response( driver_name: &str, driver_version: impl Into, default_image: impl Into, + features: impl IntoIterator, ) -> GetCapabilitiesResponse { GetCapabilitiesResponse { driver_name: driver_name.to_string(), driver_version: driver_version.into(), default_image: default_image.into(), + features: features.into_iter().map(i32::from).collect(), } } @@ -580,6 +582,27 @@ pub fn validate_linux_elf_binary(path: &Path) -> Result<(), String> { mod tests { use super::*; + #[test] + fn capabilities_encode_additive_features() { + let capabilities = build_capabilities_response( + "external", + "1.0.0", + "sandbox:latest", + [ + ComputeDriverFeature::GatewayStartReconciliation, + ComputeDriverFeature::PreserveUnspecifiedProcessIdentity, + ], + ); + + assert_eq!( + capabilities.features, + vec![ + i32::from(ComputeDriverFeature::GatewayStartReconciliation), + i32::from(ComputeDriverFeature::PreserveUnspecifiedProcessIdentity), + ] + ); + } + #[test] fn upstream_proxy_url_accepts_http_with_port() { let addr = parse_upstream_proxy_url("http://proxy.corp.com:8080").unwrap(); diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index cd1a0a8c51..f6cf5461c7 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -494,6 +494,10 @@ impl DockerComputeDriver { "docker", &self.config.daemon_version, &self.config.default_image, + [ + openshell_core::proto::compute::v1::ComputeDriverFeature::GatewayStartReconciliation, + openshell_core::proto::compute::v1::ComputeDriverFeature::PreserveUnspecifiedProcessIdentity, + ], ) } diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index ddc7fe2a4f..e3addad55f 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -548,6 +548,7 @@ impl KubernetesComputeDriver { "kubernetes", openshell_core::VERSION, &self.config.default_image, + [], )) } diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs index 9f8a62ef29..864851d477 100644 --- a/crates/openshell-driver-podman/src/driver.rs +++ b/crates/openshell-driver-podman/src/driver.rs @@ -468,6 +468,10 @@ impl PodmanComputeDriver { "podman", openshell_core::VERSION, &self.config.default_image, + [ + openshell_core::proto::compute::v1::ComputeDriverFeature::GatewayStartReconciliation, + openshell_core::proto::compute::v1::ComputeDriverFeature::PreserveUnspecifiedProcessIdentity, + ], )) } diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index 4dd2ea059b..e82182a61b 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -37,12 +37,12 @@ use openshell_core::progress::{ format_bytes, mark_progress_active, mark_progress_complete, mark_progress_detail, }; use openshell_core::proto::compute::v1::{ - CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, - DeleteWorkspaceRequest, DeleteWorkspaceResponse, DriverCondition as SandboxCondition, - DriverPlatformEvent as PlatformEvent, DriverSandbox as Sandbox, - DriverSandboxStatus as SandboxStatus, DriverSandboxTemplate as SandboxTemplate, - EnsureWorkspaceRequest, EnsureWorkspaceResponse, GetCapabilitiesRequest, - GetCapabilitiesResponse, GetGatewayListenerRequirementsRequest, + ComputeDriverFeature, CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, + DeleteSandboxResponse, DeleteWorkspaceRequest, DeleteWorkspaceResponse, + DriverCondition as SandboxCondition, DriverPlatformEvent as PlatformEvent, + DriverSandbox as Sandbox, DriverSandboxStatus as SandboxStatus, + DriverSandboxTemplate as SandboxTemplate, EnsureWorkspaceRequest, EnsureWorkspaceResponse, + GetCapabilitiesRequest, GetCapabilitiesResponse, GetGatewayListenerRequirementsRequest, GetGatewayListenerRequirementsResponse, GetSandboxRequest, GetSandboxResponse, ListSandboxesRequest, ListSandboxesResponse, StartSandboxRequest, StartSandboxResponse, StopSandboxRequest, StopSandboxResponse, ValidateSandboxCreateRequest, @@ -520,6 +520,7 @@ impl VmDriver { driver_name: DRIVER_NAME.to_string(), driver_version: openshell_core::VERSION.to_string(), default_image: self.config.default_image.clone(), + features: vec![ComputeDriverFeature::GatewayStartReconciliation.into()], } } diff --git a/crates/openshell-server/src/cli.rs b/crates/openshell-server/src/cli.rs index 2e86c3a1b5..fc70284721 100644 --- a/crates/openshell-server/src/cli.rs +++ b/crates/openshell-server/src/cli.rs @@ -115,9 +115,9 @@ struct RunArgs { /// implementing `compute_driver.proto`. /// /// When set, the socket is associated with the single driver name supplied - /// by `--drivers` or `OPENSHELL_DRIVERS`. Reserved built-in driver names - /// such as Docker, Podman, Kubernetes, and VM do not accept socket - /// endpoints. + /// by `--drivers` or `OPENSHELL_DRIVERS`. The endpoint overrides built-in + /// construction when the selected name is Docker, Podman, Kubernetes, or + /// VM. #[arg(long, env = "OPENSHELL_COMPUTE_DRIVER_SOCKET")] compute_driver_socket: Option, @@ -755,7 +755,7 @@ fn normalize_compute_driver_socket_args(args: &mut RunArgs, matches: &ArgMatches } if arg_defaulted(matches, "drivers") { return Err(miette::miette!( - "--compute-driver-socket requires --drivers or OPENSHELL_DRIVERS= to select a non-reserved compute driver name" + "--compute-driver-socket requires --drivers or OPENSHELL_DRIVERS= to select a compute driver name" )); } @@ -763,19 +763,6 @@ fn normalize_compute_driver_socket_args(args: &mut RunArgs, matches: &ArgMatches [driver] => { let driver = openshell_core::config::normalize_compute_driver_name(driver) .map_err(|err| miette::miette!("{err}"))?; - if matches!( - driver.parse::().ok(), - Some( - ComputeDriverKind::Docker - | ComputeDriverKind::Podman - | ComputeDriverKind::Kubernetes - | ComputeDriverKind::Vm - ) - ) { - return Err(miette::miette!( - "--compute-driver-socket cannot be combined with reserved built-in compute driver '{driver}'" - )); - } args.drivers[0] = driver; Ok(()) } @@ -1663,7 +1650,7 @@ ssh_session_ttl_secs = 1234 } #[test] - fn compute_driver_socket_rejects_reserved_builtin_drivers() { + fn compute_driver_socket_accepts_canonical_builtin_driver_name() { let _lock = ENV_LOCK .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); @@ -1679,16 +1666,12 @@ ssh_session_ttl_secs = 1234 "--compute-driver-socket", "/run/openshell/extension.sock", ]); - let err = super::normalize_compute_driver_socket_args(&mut args, &matches).unwrap_err(); - assert!( - err.to_string() - .contains("cannot be combined with reserved built-in compute driver 'docker'"), - "unexpected error: {err}" - ); + super::normalize_compute_driver_socket_args(&mut args, &matches).unwrap(); + assert_eq!(args.drivers, ["docker"]); } #[test] - fn compute_driver_socket_rejects_vm_endpoint() { + fn compute_driver_socket_accepts_vm_endpoint() { let _lock = ENV_LOCK .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); @@ -1704,12 +1687,8 @@ ssh_session_ttl_secs = 1234 "--compute-driver-socket", "/run/openshell/vm.sock", ]); - let err = super::normalize_compute_driver_socket_args(&mut args, &matches).unwrap_err(); - assert!( - err.to_string() - .contains("cannot be combined with reserved built-in compute driver 'vm'"), - "unexpected error: {err}" - ); + super::normalize_compute_driver_socket_args(&mut args, &matches).unwrap(); + assert_eq!(args.drivers, ["vm"]); } #[test] diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index 56d7b681a5..0ef614ea22 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -32,17 +32,18 @@ use futures::{Stream, StreamExt}; use hyper_util::rt::TokioIo; use openshell_core::ComputeDriverKind; use openshell_core::proto::compute::v1::{ - CreateSandboxRequest, DeleteSandboxRequest, DeleteWorkspaceRequest, DeleteWorkspaceResponse, - DriverCondition, DriverPlatformEvent, DriverResourceRequirements, DriverSandbox, - DriverSandboxSpec, DriverSandboxStatus, DriverSandboxTemplate, EnsureWorkspaceRequest, - EnsureWorkspaceResponse, GatewayListenerRequirement as ProtoGatewayListenerRequirement, - GetCapabilitiesRequest, GetGatewayListenerRequirementsRequest, - GetGatewayListenerRequirementsResponse, GetSandboxRequest, - GpuResourceRequirements as DriverGpuResourceRequirements, ListSandboxesRequest, - ResourceRequirements as DriverSandboxResourceRequirements, StartSandboxRequest, - StopSandboxRequest, ValidateSandboxCreateRequest, WatchSandboxesEvent, WatchSandboxesRequest, - compute_driver_client::ComputeDriverClient, compute_driver_server::ComputeDriver, - gateway_listener_requirement::Selector, watch_sandboxes_event, + ComputeDriverFeature, CreateSandboxRequest, DeleteSandboxRequest, DeleteWorkspaceRequest, + DeleteWorkspaceResponse, DriverCondition, DriverPlatformEvent, DriverResourceRequirements, + DriverSandbox, DriverSandboxSpec, DriverSandboxStatus, DriverSandboxTemplate, + EnsureWorkspaceRequest, EnsureWorkspaceResponse, + GatewayListenerRequirement as ProtoGatewayListenerRequirement, GetCapabilitiesRequest, + GetGatewayListenerRequirementsRequest, GetGatewayListenerRequirementsResponse, + GetSandboxRequest, GpuResourceRequirements as DriverGpuResourceRequirements, + ListSandboxesRequest, ResourceRequirements as DriverSandboxResourceRequirements, + StartSandboxRequest, StopSandboxRequest, ValidateSandboxCreateRequest, WatchSandboxesEvent, + WatchSandboxesRequest, compute_driver_client::ComputeDriverClient, + compute_driver_server::ComputeDriver, gateway_listener_requirement::Selector, + watch_sandboxes_event, }; use openshell_core::proto::{ PlatformEvent, Sandbox, SandboxCondition, SandboxPhase, SandboxSpec, SandboxStatus, @@ -274,6 +275,8 @@ pub struct ComputeDriverInfoSnapshot { pub driver_name: String, /// Driver-reported implementation version from the startup capability snapshot. pub driver_version: String, + /// Feature values negotiated from the startup capability snapshot. + pub features: Vec, } /// Interval between store-vs-backend reconciliation sweeps. @@ -609,6 +612,7 @@ impl ComputeRuntime { name: driver_name.clone(), driver_name: capabilities.driver_name, driver_version: capabilities.driver_version, + features: capabilities.features, }; let default_image = capabilities.default_image; let gateway_listener_requirements = match driver @@ -826,6 +830,16 @@ impl ComputeRuntime { self.driver_info.name.parse().ok() } + #[must_use] + pub(crate) fn supports_feature(&self, feature: ComputeDriverFeature) -> bool { + self.driver_info.features.contains(&i32::from(feature)) + } + + #[must_use] + pub(crate) fn preserves_unspecified_process_identity(&self) -> bool { + self.supports_feature(ComputeDriverFeature::PreserveUnspecifiedProcessIdentity) + } + #[must_use] pub(crate) fn gateway_listener_requirements(&self) -> &[GatewayListenerRequirement] { &self.gateway_listener_requirements @@ -2022,20 +2036,16 @@ impl ComputeRuntime { /// Reconcile running intent for local compute after a gateway restart. /// - /// Docker and Podman resources can outlive the gateway but may have been - /// stopped by their runtime, while VM compute exits with its gateway-owned - /// driver process. `StartSandbox` is idempotent for all three drivers, so - /// call it for every persisted phase that requires running compute. Stable - /// stopped, deleting, and error states are deliberately left alone. + /// Drivers opt into this sweep through their startup capability snapshot. + /// `StartSandbox` is idempotent, so call it for every persisted phase that + /// requires running compute. Stable stopped, deleting, and error states are + /// deliberately left alone. /// /// Should be called once at gateway startup, before watchers spawn, /// so the watch loop sees the post-start state on its first poll. pub async fn start_persisted_sandboxes(&self) -> Result<(), String> { self.recover_persisted_lifecycle_transitions().await?; - if !matches!( - self.driver_kind(), - Some(ComputeDriverKind::Docker | ComputeDriverKind::Podman | ComputeDriverKind::Vm) - ) { + if !self.supports_feature(ComputeDriverFeature::GatewayStartReconciliation) { return Ok(()); } @@ -3806,6 +3816,7 @@ impl ComputeDriver for NoopTestDriver { driver_name: "noop-test-driver".to_string(), driver_version: "test".to_string(), default_image: "openshell/sandbox:test".to_string(), + features: Vec::new(), }, )) } @@ -3940,12 +3951,23 @@ pub async fn new_test_runtime_with_driver( driver_name: &str, driver: Arc, ) -> ComputeRuntime { + let features = match driver_name.parse::().ok() { + Some(ComputeDriverKind::Docker | ComputeDriverKind::Podman) => vec![ + i32::from(ComputeDriverFeature::GatewayStartReconciliation), + i32::from(ComputeDriverFeature::PreserveUnspecifiedProcessIdentity), + ], + Some(ComputeDriverKind::Vm) => { + vec![i32::from(ComputeDriverFeature::GatewayStartReconciliation)] + } + _ => Vec::new(), + }; ComputeRuntime { driver: TracedDriver::new(driver, "test".to_string()), driver_info: ComputeDriverInfoSnapshot { name: driver_name.to_string(), driver_name: driver_name.to_string(), driver_version: "test".to_string(), + features, }, driver_process: None, default_image: "openshell/sandbox:test".to_string(), @@ -4108,6 +4130,7 @@ mod tests { driver_name: "test-driver".to_string(), driver_version: "test".to_string(), default_image: "openshell/sandbox:test".to_string(), + features: Vec::new(), })) } @@ -4416,6 +4439,7 @@ mod tests { driver_name: "controlled-test-driver".to_string(), driver_version: "test".to_string(), default_image: "openshell/sandbox:test".to_string(), + features: Vec::new(), })) } @@ -4605,6 +4629,14 @@ mod tests { async fn test_runtime_for_driver( driver: SharedComputeDriver, driver_name: &str, + ) -> ComputeRuntime { + test_runtime_for_driver_with_features(driver, driver_name, Vec::new()).await + } + + async fn test_runtime_for_driver_with_features( + driver: SharedComputeDriver, + driver_name: &str, + features: Vec, ) -> ComputeRuntime { let store = Arc::new(Store::connect("sqlite::memory:").await.unwrap()); ComputeRuntime { @@ -4613,6 +4645,7 @@ mod tests { name: driver_name.to_string(), driver_name: driver_name.to_string(), driver_version: "test".to_string(), + features: features.into_iter().map(i32::from).collect(), }, driver_process: None, default_image: "openshell/sandbox:test".to_string(), @@ -8008,7 +8041,12 @@ mod tests { #[tokio::test] async fn start_persisted_sandboxes_starts_running_phases() { let driver = ControlledDriver::new(); - let runtime = test_runtime_for_driver(driver.clone(), "docker").await; + let runtime = test_runtime_for_driver_with_features( + driver.clone(), + "extension", + vec![ComputeDriverFeature::GatewayStartReconciliation], + ) + .await; for (id, name, phase) in [ ("sb-unspecified", "unspecified", SandboxPhase::Unspecified), @@ -8047,7 +8085,12 @@ mod tests { async fn start_persisted_sandboxes_marks_missing_backend_as_error() { let driver = ControlledDriver::new(); driver.set_start_outcome(ControlledLifecycleOutcome::NotFound); - let runtime = test_runtime_for_driver(driver, "podman").await; + let runtime = test_runtime_for_driver_with_features( + driver, + "extension", + vec![ComputeDriverFeature::GatewayStartReconciliation], + ) + .await; let sandbox = sandbox_record("sb-1", "missing", SandboxPhase::Ready); runtime.store.put_message(&sandbox).await.unwrap(); @@ -8077,7 +8120,12 @@ mod tests { async fn start_persisted_sandboxes_marks_failed_start_as_error() { let driver = ControlledDriver::new(); driver.set_start_outcome(ControlledLifecycleOutcome::Error("runtime angry")); - let runtime = test_runtime_for_driver(driver, "vm").await; + let runtime = test_runtime_for_driver_with_features( + driver, + "extension", + vec![ComputeDriverFeature::GatewayStartReconciliation], + ) + .await; let sandbox = sandbox_record("sb-1", "broken", SandboxPhase::Provisioning); runtime.store.put_message(&sandbox).await.unwrap(); @@ -8104,10 +8152,15 @@ mod tests { } #[tokio::test] - async fn start_persisted_sandboxes_runs_for_each_local_driver() { - for driver_name in ["docker", "podman", "vm"] { + async fn start_persisted_sandboxes_runs_for_any_capable_driver() { + for driver_name in ["docker", "podman", "vm", "extension"] { let driver = ControlledDriver::new(); - let runtime = test_runtime_for_driver(driver.clone(), driver_name).await; + let runtime = test_runtime_for_driver_with_features( + driver.clone(), + driver_name, + vec![ComputeDriverFeature::GatewayStartReconciliation], + ) + .await; let sandbox = sandbox_record("sb-1", "local", SandboxPhase::Ready); runtime.store.put_message(&sandbox).await.unwrap(); @@ -8122,8 +8175,8 @@ mod tests { } #[tokio::test] - async fn start_persisted_sandboxes_skips_kubernetes_and_extension_drivers() { - for driver_name in ["kubernetes", "extension"] { + async fn start_persisted_sandboxes_skips_drivers_without_feature() { + for driver_name in ["kubernetes", "docker", "extension"] { let driver = ControlledDriver::new(); let runtime = test_runtime_for_driver(driver.clone(), driver_name).await; let sandbox = sandbox_record("sb-1", "remote", SandboxPhase::Ready); @@ -8350,13 +8403,17 @@ mod tests { let driver = FakeComputeDriver::new() .with_driver_name("fake-remote-driver") .with_default_image("openshell/sandbox:remote") + .with_features([ + ComputeDriverFeature::GatewayStartReconciliation, + ComputeDriverFeature::PreserveUnspecifiedProcessIdentity, + ]) .with_gateway_listener_requirement( "172.19.0.1:17670", "external driver managed bridge", ); let _server = driver.serve_uds(&socket_path).unwrap(); - let endpoint = connect_remote_compute_driver("external-test", &socket_path) + let endpoint = connect_remote_compute_driver("docker", &socket_path) .await .unwrap(); let store = Arc::new(Store::connect("sqlite::memory:").await.unwrap()); @@ -8370,11 +8427,12 @@ mod tests { ) .await .unwrap(); + assert!(runtime.preserves_unspecified_process_identity()); assert_eq!( runtime.gateway_listener_requirements(), &[GatewayListenerRequirement::Exact { address: "172.19.0.1:17670".parse().unwrap(), - driver_name: "external-test".to_string(), + driver_name: "docker".to_string(), reason: "external driver managed bridge".to_string(), }] ); @@ -8387,11 +8445,11 @@ mod tests { driver_config: Some(prost_types::Struct { fields: [ ( - "external-test".to_string(), + "docker".to_string(), struct_value([("pool", string_value("ci"))]), ), ( - "docker".to_string(), + "kubernetes".to_string(), struct_value([("network_mode", string_value("bridge"))]), ), ] @@ -8405,30 +8463,14 @@ mod tests { runtime.validate_sandbox_create(&sandbox).await.unwrap(); runtime.create_sandbox(sandbox, None).await.unwrap(); - assert!( - runtime - .delete_sandbox("default", "uds-sandbox") - .await - .unwrap() - .deleted - ); - let calls = driver.calls(); - assert_eq!(calls.len(), 5, "unexpected calls: {calls:?}"); - assert!(matches!(calls[0], FakeComputeDriverCall::GetCapabilities)); - assert!(matches!( - calls[1], - FakeComputeDriverCall::GetGatewayListenerRequirements - )); - + assert_eq!(calls.len(), 4, "unexpected calls: {calls:?}"); let validated = match &calls[2] { FakeComputeDriverCall::ValidateSandboxCreate { sandbox: Some(sandbox), } => sandbox, other => panic!("expected ValidateSandboxCreate call, got {other:?}"), }; - assert_eq!(validated.id, "sb-uds"); - assert_eq!(validated.name, "uds-sandbox"); let driver_config = validated .spec .as_ref() @@ -8438,16 +8480,25 @@ mod tests { assert!(driver_config.fields.contains_key("pool")); assert!(!driver_config.fields.contains_key("network_mode")); - let created = match &calls[3] { - FakeComputeDriverCall::CreateSandbox { - sandbox: Some(sandbox), - } => sandbox, - other => panic!("expected CreateSandbox call, got {other:?}"), - }; - assert_eq!(created.id, "sb-uds"); - assert_eq!(created.name, "uds-sandbox"); + driver.clear_calls(); + runtime.start_persisted_sandboxes().await.unwrap(); + assert!(matches!( + driver.calls().as_slice(), + [FakeComputeDriverCall::StartSandbox { sandbox_id, sandbox_name }] + if sandbox_id == "sb-uds" && sandbox_name == "uds-sandbox" + )); + driver.clear_calls(); + assert!( + runtime + .delete_sandbox("default", "uds-sandbox") + .await + .unwrap() + .deleted + ); - match &calls[4] { + let calls = driver.calls(); + assert_eq!(calls.len(), 1, "unexpected calls: {calls:?}"); + match &calls[0] { FakeComputeDriverCall::DeleteSandbox { sandbox_id, sandbox_name, diff --git a/crates/openshell-server/src/gateway_listener.rs b/crates/openshell-server/src/gateway_listener.rs index 4e4c7e80d0..640757bfb3 100644 --- a/crates/openshell-server/src/gateway_listener.rs +++ b/crates/openshell-server/src/gateway_listener.rs @@ -2,7 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 use crate::compute::GatewayListenerRequirement; -use openshell_core::{ComputeDriverKind, Error, Result}; +use openshell_core::{Error, Result}; use socket2::{Domain, Protocol, Socket, Type}; use std::net::{IpAddr, SocketAddr}; use tokio::net::TcpListener; @@ -193,25 +193,11 @@ fn validate_gateway_listener_requirement( requirement: &GatewayListenerRequirement, ) -> Result<()> { match requirement { - GatewayListenerRequirement::Exact { - address, - driver_name, - .. - } if driver_name == ComputeDriverKind::Docker.as_str() - || driver_name == ComputeDriverKind::Podman.as_str() => - { + GatewayListenerRequirement::Exact { address, .. } => { validate_resolved_gateway_listener(primary_listener, *address) } - GatewayListenerRequirement::DefaultRouteInterface { driver_name, .. } - | GatewayListenerRequirement::LoopbackInterface { driver_name, .. } - if driver_name == ComputeDriverKind::Podman.as_str() => - { - Ok(()) - } - _ => Err(Error::config(format!( - "compute driver '{}' is not authorized to request this gateway listener selector", - requirement.driver_name() - ))), + GatewayListenerRequirement::DefaultRouteInterface { .. } + | GatewayListenerRequirement::LoopbackInterface { .. } => Ok(()), } } @@ -464,7 +450,7 @@ mod tests { } #[test] - fn gateway_listener_specs_reject_unauthorized_external_driver() { + fn gateway_listener_specs_accept_safe_external_driver_requirement() { let primary: SocketAddr = "127.0.0.1:8080".parse().unwrap(); let requirement = GatewayListenerRequirement::Exact { address: "172.18.0.1:8080".parse().unwrap(), @@ -472,8 +458,10 @@ mod tests { reason: "external bridge".to_string(), }; - let err = gateway_listener_specs(primary, &[requirement]).unwrap_err(); - assert!(err.to_string().contains("not authorized")); + let specs = gateway_listener_specs(primary, &[requirement]).unwrap(); + assert_eq!(specs.len(), 2); + assert_eq!(specs[1].address, "172.18.0.1:8080".parse().unwrap()); + assert_eq!(specs[1].scope, GatewayListenerScope::ComputeDriverCallback); } #[test] @@ -622,15 +610,18 @@ mod tests { } #[test] - fn gateway_listener_specs_reject_cross_driver_selector_authority() { - let primary: SocketAddr = "127.0.0.1:8080".parse().unwrap(); + fn gateway_listener_specs_validate_selector_independently_of_driver_name() { + let primary: SocketAddr = "192.168.20.20:8080".parse().unwrap(); let requirement = GatewayListenerRequirement::LoopbackInterface { driver_name: "docker".to_string(), reason: "wrong selector".to_string(), }; - let err = gateway_listener_specs(primary, &[requirement]).unwrap_err(); - assert!(err.to_string().contains("not authorized")); + let specs = gateway_listener_specs(primary, &[requirement]).unwrap(); + assert_eq!(specs.len(), 2); + assert_eq!(specs[0].address, primary); + assert_eq!(specs[1].address, "127.0.0.1:8080".parse().unwrap()); + assert_eq!(specs[1].scope, GatewayListenerScope::ComputeDriverCallback); } #[tokio::test] diff --git a/crates/openshell-server/src/grpc/policy.rs b/crates/openshell-server/src/grpc/policy.rs index 3b841e66a5..af6ff66fe7 100644 --- a/crates/openshell-server/src/grpc/policy.rs +++ b/crates/openshell-server/src/grpc/policy.rs @@ -2389,7 +2389,10 @@ async fn handle_update_config_inner( let mut new_policy = req.policy.ok_or_else(|| { Status::invalid_argument("policy is required for global policy update") })?; - normalize_process_identity_for_driver(&mut new_policy, state.compute.driver_kind()); + normalize_process_identity_for_driver( + &mut new_policy, + state.compute.preserves_unspecified_process_identity(), + ); validate_no_reserved_provider_policy_keys(&new_policy)?; validate_policy_safety(&new_policy)?; crate::middleware::validate_policy(state.middleware_registry.as_ref(), &new_policy) @@ -2698,7 +2701,10 @@ async fn handle_update_config_inner( }; let mut baseline_policy = spec.policy.clone(); if let Some(policy) = baseline_policy.as_mut() { - normalize_process_identity_for_driver(policy, state.compute.driver_kind()); + normalize_process_identity_for_driver( + policy, + state.compute.preserves_unspecified_process_identity(), + ); } let (version, hash, updated_sandbox) = apply_merge_operations_with_retry( state.store.as_ref(), @@ -2774,7 +2780,10 @@ async fn handle_update_config_inner( let mut new_policy = req .policy .ok_or_else(|| Status::invalid_argument("policy is required"))?; - normalize_process_identity_for_driver(&mut new_policy, state.compute.driver_kind()); + normalize_process_identity_for_driver( + &mut new_policy, + state.compute.preserves_unspecified_process_identity(), + ); let global_settings = load_global_settings(state.store.as_ref()).await?; if global_settings.settings.contains_key(POLICY_SETTING_KEY) { @@ -2803,7 +2812,7 @@ async fn handle_update_config_inner( let mut comparable_baseline = baseline_policy.clone(); normalize_process_identity_for_driver( &mut comparable_baseline, - state.compute.driver_kind(), + state.compute.preserves_unspecified_process_identity(), ); validate_static_fields_unchanged(&comparable_baseline, &new_policy)?; None diff --git a/crates/openshell-server/src/grpc/sandbox.rs b/crates/openshell-server/src/grpc/sandbox.rs index 9338956950..0d8c50c378 100644 --- a/crates/openshell-server/src/grpc/sandbox.rs +++ b/crates/openshell-server/src/grpc/sandbox.rs @@ -268,10 +268,13 @@ async fn handle_create_sandbox_inner( template.image = state.compute.default_image().to_string(); } - // Docker and Podman preserve omitted identity fields for OCI USER - // fallback. Other drivers retain the legacy persisted sandbox defaults. + // Drivers with native image/runtime identity handling preserve omitted + // fields. Other drivers retain the legacy persisted sandbox defaults. if let Some(ref mut policy) = spec.policy { - normalize_process_identity_for_driver(policy, state.compute.driver_kind()); + normalize_process_identity_for_driver( + policy, + state.compute.preserves_unspecified_process_identity(), + ); validate_no_reserved_provider_policy_keys(policy)?; validate_policy_safety(policy)?; crate::middleware::validate_policy(state.middleware_registry.as_ref(), policy).await?; diff --git a/crates/openshell-server/src/grpc/validation.rs b/crates/openshell-server/src/grpc/validation.rs index f71623fa3d..12e184e57f 100644 --- a/crates/openshell-server/src/grpc/validation.rs +++ b/crates/openshell-server/src/grpc/validation.rs @@ -8,7 +8,6 @@ #![allow(clippy::result_large_err)] // Validation returns Result<_, Status> -use openshell_core::ComputeDriverKind; use openshell_core::proto::{ CredentialHandle, ExecSandboxRequest, Provider, SandboxPolicy as ProtoSandboxPolicy, SandboxTemplate, @@ -28,19 +27,14 @@ use super::{ // Exec request validation // --------------------------------------------------------------------------- -/// Preserve process-identity omission only for the local OCI-aware drivers. -/// -/// Kubernetes, VM, and unknown/remote drivers retain the legacy persisted -/// `sandbox:sandbox` defaults so existing policy hashes and live-update -/// workflows do not change. +/// Preserve process-identity omission only when the compute driver advertises +/// native image/runtime identity handling. Drivers without the feature retain +/// the legacy persisted `sandbox:sandbox` defaults. pub(super) fn normalize_process_identity_for_driver( policy: &mut ProtoSandboxPolicy, - driver_kind: Option, + preserves_unspecified_process_identity: bool, ) { - if !matches!( - driver_kind, - Some(ComputeDriverKind::Docker | ComputeDriverKind::Podman) - ) { + if !preserves_unspecified_process_identity { openshell_policy::ensure_sandbox_process_identity(policy); } } @@ -1854,41 +1848,30 @@ mod tests { // ---- Policy safety ---- #[test] - fn process_identity_omission_is_driver_scoped() { + fn process_identity_omission_is_feature_scoped() { use openshell_core::proto::ProcessPolicy; - for driver in [ComputeDriverKind::Docker, ComputeDriverKind::Podman] { - let mut policy = ProtoSandboxPolicy { - process: Some(ProcessPolicy { - run_as_user: "1234".into(), - run_as_group: String::new(), - }), - ..Default::default() - }; - normalize_process_identity_for_driver(&mut policy, Some(driver)); - assert!( - policy.process.unwrap().run_as_group.is_empty(), - "{driver:?} must preserve omission" - ); - } + let mut policy = ProtoSandboxPolicy { + process: Some(ProcessPolicy { + run_as_user: "1234".into(), + run_as_group: String::new(), + }), + ..Default::default() + }; + normalize_process_identity_for_driver(&mut policy, true); + assert!(policy.process.unwrap().run_as_group.is_empty()); - for driver in [ - Some(ComputeDriverKind::Kubernetes), - Some(ComputeDriverKind::Vm), - None, - ] { - let mut policy = ProtoSandboxPolicy { - process: Some(ProcessPolicy { - run_as_user: "1234".into(), - run_as_group: String::new(), - }), - ..Default::default() - }; - normalize_process_identity_for_driver(&mut policy, driver); - let process = policy.process.unwrap(); - assert_eq!(process.run_as_user, "1234"); - assert_eq!(process.run_as_group, "sandbox"); - } + let mut policy = ProtoSandboxPolicy { + process: Some(ProcessPolicy { + run_as_user: "1234".into(), + run_as_group: String::new(), + }), + ..Default::default() + }; + normalize_process_identity_for_driver(&mut policy, false); + let process = policy.process.unwrap(); + assert_eq!(process.run_as_user, "1234"); + assert_eq!(process.run_as_group, "sandbox"); } #[test] diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index 979e372084..6ff455bea2 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -1249,10 +1249,8 @@ fn resolve_configured_compute_driver( let name = openshell_core::config::normalize_compute_driver_name(driver_name) .map_err(Error::config)?; let driver_kind = builtin_compute_driver(&name); - if driver_kind.is_some() && driver_startup.endpoint_overrides.contains_key(&name) { - return Err(Error::config(format!( - "compute driver '{name}' is a reserved built-in driver and cannot be selected with a socket endpoint" - ))); + if driver_startup.endpoint_overrides.contains_key(&name) { + return Ok(ConfiguredComputeDriver::Remote { name }); } if let Some(kind) = driver_kind { @@ -1877,35 +1875,31 @@ mod tests { } #[test] - fn configured_compute_driver_rejects_vm_endpoint_from_config() { + fn configured_compute_driver_uses_vm_endpoint_override() { let config = Config::new(None) .with_compute_drivers([ComputeDriverKind::Vm]) .with_compute_driver_endpoint("vm", "/run/openshell/vm.sock"); - let err = - configured_compute_driver(&config, test_driver_startup(&config, None)).unwrap_err(); - - assert!( - err.to_string() - .contains("reserved built-in driver and cannot be selected with a socket endpoint"), - "unexpected error: {err}" - ); + let driver = + configured_compute_driver(&config, test_driver_startup(&config, None)).unwrap(); + assert!(matches!( + driver, + ConfiguredComputeDriver::Remote { name } if name == "vm" + )); } #[test] - fn configured_compute_driver_rejects_builtin_endpoint() { + fn configured_compute_driver_uses_builtin_endpoint_override() { let config = Config::new(None) .with_compute_drivers([ComputeDriverKind::Docker]) .with_compute_driver_endpoint("docker", "/run/openshell/docker.sock"); - let err = - configured_compute_driver(&config, test_driver_startup(&config, None)).unwrap_err(); - - assert!( - err.to_string() - .contains("cannot be selected with a socket endpoint"), - "unexpected error: {err}" - ); + let driver = + configured_compute_driver(&config, test_driver_startup(&config, None)).unwrap(); + assert!(matches!( + driver, + ConfiguredComputeDriver::Remote { name } if name == "docker" + )); } #[test] diff --git a/crates/openshell-server/src/test_support.rs b/crates/openshell-server/src/test_support.rs index f8124ded6c..742910ce47 100644 --- a/crates/openshell-server/src/test_support.rs +++ b/crates/openshell-server/src/test_support.rs @@ -74,6 +74,7 @@ struct FakeComputeDriverState { driver_name: String, driver_version: String, default_image: String, + features: Vec, gateway_listener_requirements: Vec, gateway_listener_requirements_supported: bool, sandboxes: HashMap, @@ -95,6 +96,7 @@ impl FakeComputeDriver { driver_name: "fake-compute-driver".to_string(), driver_version: "test".to_string(), default_image: "openshell/sandbox:test".to_string(), + features: Vec::new(), gateway_listener_requirements: Vec::new(), gateway_listener_requirements_supported: true, sandboxes: HashMap::new(), @@ -122,6 +124,17 @@ impl FakeComputeDriver { self } + #[must_use] + pub fn with_features( + self, + features: impl IntoIterator, + ) -> Self { + self.with_state(|state| { + state.features = features.into_iter().map(i32::from).collect(); + }); + self + } + #[must_use] pub fn with_gateway_listener_requirement( self, @@ -241,6 +254,7 @@ impl ComputeDriver for FakeComputeDriver { driver_name: state.driver_name.clone(), driver_version: state.driver_version.clone(), default_image: state.default_image.clone(), + features: state.features.clone(), } }); Ok(Response::new(response)) diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 8d74b3d44f..44cca9f687 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -743,10 +743,9 @@ guest_tls_key = "/var/lib/openshell/guest-tls/client-key.pem" ### Extension Driver Extension drivers run outside the gateway and expose the -`compute_driver.proto` gRPC service on a Unix socket. Use a non-reserved driver -name; built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot -be selected through unmanaged socket endpoints. The selected driver name is the -key used for driver-owned sandbox config such as `template.driver_config.`. +supported `compute_driver.proto` gRPC service on a Unix socket. The selected +driver name is the key used for driver-owned sandbox config such as +`template.driver_config.`. ```toml [openshell] @@ -760,3 +759,16 @@ compute_drivers = ["kyma"] [openshell.drivers.kyma] socket_path = "/run/openshell/kyma-compute-driver.sock" ``` + +At launch time, `--compute-driver-socket` can override the implementation for +any explicitly selected name, including `docker`, `podman`, `kubernetes`, and +`vm`. The override takes precedence over built-in construction: + +```shell +openshell-gateway --drivers docker \ + --compute-driver-socket /run/openshell/docker-driver.sock +``` + +Do not use `[openshell.drivers.docker].socket_path` for this purpose; that +existing field selects the Docker daemon socket. Operator-supplied driver +processes and Unix sockets are not created or supervised by the gateway. diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 4b58b58423..99851d845c 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -20,10 +20,12 @@ remain unavailable. Restarting the gateway preserves this intent. The gateway does not stop Docker or Podman containers during shutdown. At startup it sends idempotent start -requests for Docker, Podman, and MicroVM sandboxes that were intended to run; -already-running resources are unchanged, retained stopped compute is restarted, -and explicitly stopped sandboxes remain stopped. Kubernetes workloads continue -running independently of the gateway process. +requests for drivers that advertise startup reconciliation. The built-in +Docker, Podman, and MicroVM implementations advertise this feature, so +sandboxes that were intended to run are reconciled. Already-running resources +are unchanged, retained stopped compute is restarted, and explicitly stopped +sandboxes remain stopped. Kubernetes workloads continue running independently +of the gateway process. ## Configure a Compute Driver @@ -34,8 +36,8 @@ Configure the compute driver on the gateway. Current releases accept one driver compute_drivers = ["docker"] ``` -Reserved built-in values are `docker`, `podman`, `kubernetes`, and `vm`. -Non-reserved names select an extension driver and require a +Built-in values are `docker`, `podman`, `kubernetes`, and `vm`. +Other names select an extension driver and require a `socket_path` in `[openshell.drivers.]`. When `compute_drivers` is unset, the gateway auto-detects Kubernetes, then Podman, then Docker. Local container runtimes must respond to an API probe before the gateway selects them. The VM driver is never auto-detected; configure it explicitly with `compute_drivers = ["vm"]` or set `OPENSHELL_DRIVERS=vm` in the launch environment. @@ -48,8 +50,8 @@ Common gateway options: Set driver-specific values such as sandbox images, callback endpoints, network names, TLS material, and VM sizing in the gateway TOML file. See the [Gateway Configuration File](./gateway-config) reference for the full `[openshell.drivers.]` schema. -Extension drivers use the same `compute_driver.proto` gRPC surface as the -managed VM driver. For an out-of-tree driver, choose a driver name and point +External drivers use the same supported `compute_driver.proto` gRPC surface as +the managed VM driver. For an out-of-tree driver, choose a driver name and point the gateway at the Unix socket the operator has already provisioned: ```toml @@ -60,16 +62,27 @@ compute_drivers = ["kyma"] socket_path = "/run/openshell/kyma.sock" ``` -For a launch-time socket override, pass the same non-reserved driver name with -the socket path: +For a launch-time socket override, pass the selected driver name with the +socket path: ```shell openshell-gateway --drivers kyma --compute-driver-socket /run/openshell/kyma.sock ``` -The gateway does not spawn, supervise, or delete extension drivers. The +The override also accepts a built-in name. This lets an operator provide, for +example, a Docker-compatible implementation through the external contract +without changing driver-owned config keys: + +```shell +openshell-gateway --drivers docker --compute-driver-socket /run/openshell/docker-driver.sock +``` + +The endpoint override takes precedence over built-in construction. The gateway +does not spawn, supervise, or delete an operator-supplied driver. The operator must protect the socket so only the gateway uid can access it. -Reserved built-in names cannot be selected through unmanaged socket endpoints. +The driver's advertised name is diagnostic metadata and does not authorize +special behavior. Drivers negotiate optional startup and process-identity +behavior through `GetCapabilities`; unknown features are ignored. Sandbox create supports `--cpu` and `--memory` for per-sandbox compute sizing. Docker and Podman apply them as runtime limits. Kubernetes applies them as both @@ -120,15 +133,16 @@ It overrides the gateway's configured default runtime class for that sandbox, while a typed `SandboxTemplate.runtime_class_name` value from the API still takes precedence. -Docker and Podman report the address through which their sandboxes can reach -the gateway. If the primary listener covers that address, the gateway reuses +Compute drivers can report the address through which their sandboxes reach the +gateway. If the primary listener covers a requested address, the gateway reuses it and sandbox JWT authentication restricts the supervisor to its callback RPC -allowlist. If the primary listener is not reachable through that address, the -gateway creates an additional callback-only listener. Use the primary endpoint -for CLI, administrator, health, reflection, inference-route management, and -HTTP requests. A `PermissionDenied` response from an additional callback-only -listener is expected for those requests. Do not broaden the primary listener -to `0.0.0.0` solely to make sandbox callbacks reachable. +allowlist. Otherwise, the gateway creates an additional callback-only listener. +Listener requirements are structurally validated independently of the selected +driver name. Use the primary endpoint for CLI, administrator, health, +reflection, inference-route management, and HTTP requests. A `PermissionDenied` +response from an additional callback-only listener is expected for those +requests. Do not broaden the primary listener to `0.0.0.0` solely to make +sandbox callbacks reachable. ## Docker Driver diff --git a/proto/compute_driver.proto b/proto/compute_driver.proto index 0ce4f61539..86a66b7b58 100644 --- a/proto/compute_driver.proto +++ b/proto/compute_driver.proto @@ -8,14 +8,17 @@ package openshell.compute.v1; import "google/protobuf/struct.proto"; import "options.proto"; -// Internal compute-driver contract used by the gateway. +// Gateway/compute-driver extension contract. // // Conventions: // - This file owns driver-native request, response, and observation types. // - Compute drivers must not import or return the public `openshell.v1.Sandbox` // resource model. -// - The gateway translates between these internal driver-native messages and +// - The gateway translates between these driver-native messages and // the public OpenShell API resource model. +// - Capability fields are additive. Drivers and gateways must ignore unknown +// feature values so independently versioned external drivers remain +// forward-compatible. service ComputeDriver { // Report driver capabilities and defaults. rpc GetCapabilities(GetCapabilitiesRequest) returns (GetCapabilitiesResponse); @@ -72,6 +75,19 @@ message GetCapabilitiesResponse { string driver_version = 2; // Default sandbox image recommended by the driver. string default_image = 3; + // Optional behavior supported by this driver instance. Unknown values are + // ignored and omitted features retain conservative gateway behavior. + repeated ComputeDriverFeature features = 6; +} + +enum ComputeDriverFeature { + COMPUTE_DRIVER_FEATURE_UNSPECIFIED = 0; + // Reconcile persisted running intent by calling StartSandbox when the + // gateway starts. + COMPUTE_DRIVER_FEATURE_GATEWAY_START_RECONCILIATION = 1; + // Preserve omitted process user/group fields so the runtime can apply its + // native image or OCI identity defaults. + COMPUTE_DRIVER_FEATURE_PRESERVE_UNSPECIFIED_PROCESS_IDENTITY = 2; } message GetGatewayListenerRequirementsRequest {} From daf85c9aae854496acc7dc7b8ec6e76a5d951ca1 Mon Sep 17 00:00:00 2001 From: Drew Newberry Date: Sun, 16 Aug 2026 23:57:45 -0700 Subject: [PATCH 2/2] docs(compute): revert external driver documentation Signed-off-by: Drew Newberry --- docs/reference/gateway-config.mdx | 20 ++------ docs/reference/sandbox-compute-drivers.mdx | 54 ++++++++-------------- 2 files changed, 24 insertions(+), 50 deletions(-) diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 44cca9f687..8d74b3d44f 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -743,9 +743,10 @@ guest_tls_key = "/var/lib/openshell/guest-tls/client-key.pem" ### Extension Driver Extension drivers run outside the gateway and expose the -supported `compute_driver.proto` gRPC service on a Unix socket. The selected -driver name is the key used for driver-owned sandbox config such as -`template.driver_config.`. +`compute_driver.proto` gRPC service on a Unix socket. Use a non-reserved driver +name; built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot +be selected through unmanaged socket endpoints. The selected driver name is the +key used for driver-owned sandbox config such as `template.driver_config.`. ```toml [openshell] @@ -759,16 +760,3 @@ compute_drivers = ["kyma"] [openshell.drivers.kyma] socket_path = "/run/openshell/kyma-compute-driver.sock" ``` - -At launch time, `--compute-driver-socket` can override the implementation for -any explicitly selected name, including `docker`, `podman`, `kubernetes`, and -`vm`. The override takes precedence over built-in construction: - -```shell -openshell-gateway --drivers docker \ - --compute-driver-socket /run/openshell/docker-driver.sock -``` - -Do not use `[openshell.drivers.docker].socket_path` for this purpose; that -existing field selects the Docker daemon socket. Operator-supplied driver -processes and Unix sockets are not created or supervised by the gateway. diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 99851d845c..4b58b58423 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -20,12 +20,10 @@ remain unavailable. Restarting the gateway preserves this intent. The gateway does not stop Docker or Podman containers during shutdown. At startup it sends idempotent start -requests for drivers that advertise startup reconciliation. The built-in -Docker, Podman, and MicroVM implementations advertise this feature, so -sandboxes that were intended to run are reconciled. Already-running resources -are unchanged, retained stopped compute is restarted, and explicitly stopped -sandboxes remain stopped. Kubernetes workloads continue running independently -of the gateway process. +requests for Docker, Podman, and MicroVM sandboxes that were intended to run; +already-running resources are unchanged, retained stopped compute is restarted, +and explicitly stopped sandboxes remain stopped. Kubernetes workloads continue +running independently of the gateway process. ## Configure a Compute Driver @@ -36,8 +34,8 @@ Configure the compute driver on the gateway. Current releases accept one driver compute_drivers = ["docker"] ``` -Built-in values are `docker`, `podman`, `kubernetes`, and `vm`. -Other names select an extension driver and require a +Reserved built-in values are `docker`, `podman`, `kubernetes`, and `vm`. +Non-reserved names select an extension driver and require a `socket_path` in `[openshell.drivers.]`. When `compute_drivers` is unset, the gateway auto-detects Kubernetes, then Podman, then Docker. Local container runtimes must respond to an API probe before the gateway selects them. The VM driver is never auto-detected; configure it explicitly with `compute_drivers = ["vm"]` or set `OPENSHELL_DRIVERS=vm` in the launch environment. @@ -50,8 +48,8 @@ Common gateway options: Set driver-specific values such as sandbox images, callback endpoints, network names, TLS material, and VM sizing in the gateway TOML file. See the [Gateway Configuration File](./gateway-config) reference for the full `[openshell.drivers.]` schema. -External drivers use the same supported `compute_driver.proto` gRPC surface as -the managed VM driver. For an out-of-tree driver, choose a driver name and point +Extension drivers use the same `compute_driver.proto` gRPC surface as the +managed VM driver. For an out-of-tree driver, choose a driver name and point the gateway at the Unix socket the operator has already provisioned: ```toml @@ -62,27 +60,16 @@ compute_drivers = ["kyma"] socket_path = "/run/openshell/kyma.sock" ``` -For a launch-time socket override, pass the selected driver name with the -socket path: +For a launch-time socket override, pass the same non-reserved driver name with +the socket path: ```shell openshell-gateway --drivers kyma --compute-driver-socket /run/openshell/kyma.sock ``` -The override also accepts a built-in name. This lets an operator provide, for -example, a Docker-compatible implementation through the external contract -without changing driver-owned config keys: - -```shell -openshell-gateway --drivers docker --compute-driver-socket /run/openshell/docker-driver.sock -``` - -The endpoint override takes precedence over built-in construction. The gateway -does not spawn, supervise, or delete an operator-supplied driver. The +The gateway does not spawn, supervise, or delete extension drivers. The operator must protect the socket so only the gateway uid can access it. -The driver's advertised name is diagnostic metadata and does not authorize -special behavior. Drivers negotiate optional startup and process-identity -behavior through `GetCapabilities`; unknown features are ignored. +Reserved built-in names cannot be selected through unmanaged socket endpoints. Sandbox create supports `--cpu` and `--memory` for per-sandbox compute sizing. Docker and Podman apply them as runtime limits. Kubernetes applies them as both @@ -133,16 +120,15 @@ It overrides the gateway's configured default runtime class for that sandbox, while a typed `SandboxTemplate.runtime_class_name` value from the API still takes precedence. -Compute drivers can report the address through which their sandboxes reach the -gateway. If the primary listener covers a requested address, the gateway reuses +Docker and Podman report the address through which their sandboxes can reach +the gateway. If the primary listener covers that address, the gateway reuses it and sandbox JWT authentication restricts the supervisor to its callback RPC -allowlist. Otherwise, the gateway creates an additional callback-only listener. -Listener requirements are structurally validated independently of the selected -driver name. Use the primary endpoint for CLI, administrator, health, -reflection, inference-route management, and HTTP requests. A `PermissionDenied` -response from an additional callback-only listener is expected for those -requests. Do not broaden the primary listener to `0.0.0.0` solely to make -sandbox callbacks reachable. +allowlist. If the primary listener is not reachable through that address, the +gateway creates an additional callback-only listener. Use the primary endpoint +for CLI, administrator, health, reflection, inference-route management, and +HTTP requests. A `PermissionDenied` response from an additional callback-only +listener is expected for those requests. Do not broaden the primary listener +to `0.0.0.0` solely to make sandbox callbacks reachable. ## Docker Driver