From 87a76b284866970898148fb51fea0feb10576a5a Mon Sep 17 00:00:00 2001 From: SaladDay <1203511142@qq.com> Date: Sun, 25 Oct 2026 07:44:41 +0000 Subject: [PATCH 1/3] Publish reproducible performance results scripts/benchmark.py drives hosted Sessions through the official SDK, times E2B with its SDK, and builds the table from the Session records and the Core log lines it keeps. docs/performance.md describes each measurement, the setup and how to reproduce it; the records it was built from are in docs/performance. The README shows the table beside E2B and the limits that apply. The backend comparison listed E2B as not suspending idle sandboxes; E2B pauses them in place. --- README.md | 22 ++ README.zh-CN.md | 22 ++ docs.json | 3 +- docs/getting-started/sandbox-backends.md | 2 +- docs/performance.md | 87 +++++++ docs/performance/core.jsonl | 215 ++++++++++++++++++ docs/performance/e2b.jsonl | 15 ++ docs/performance/sessions.jsonl | 50 ++++ docs/performance/window.json | 1 + docs/zh/getting-started/sandbox-backends.md | 4 +- docs/zh/performance.md | 89 ++++++++ scripts/benchmark.py | 238 ++++++++++++++++++++ 12 files changed, 744 insertions(+), 4 deletions(-) create mode 100644 docs/performance.md create mode 100644 docs/performance/core.jsonl create mode 100644 docs/performance/e2b.jsonl create mode 100644 docs/performance/sessions.jsonl create mode 100644 docs/performance/window.json create mode 100644 docs/zh/performance.md create mode 100644 scripts/benchmark.py diff --git a/README.md b/README.md index 8601923f3..f8c000a05 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,28 @@ The OpenAI Agents API, open source and self-hosted. Run Codex, Claude Code or Mi - **Compute that sleeps, workspaces that stay.** On microsandbox, which keeps every workspace on independent storage, idle sandboxes release compute and resume on a compatible node with their files intact. E2B pauses idle sandboxes in place. Docker and your own Linux, macOS or Windows machine also run Sessions. See [Sandbox Provider protocol](https://openagentcore.dev/docs/sandbox-provider). - **Replaceable parts.** Sandbox Providers, workspace filesystems, Harnesses and model providers plug in through [defined protocols](AGENTS.md#protocols-at-every-boundary). +## Performance + +The sandbox lifecycle on one microsandbox node, with E2B measured the same way. Medians of at least 10 runs on 2026-10-25; each row links to its method and raw records, and [`scripts/benchmark.py`](scripts/benchmark.py) reproduces them. + +| Operation | OpenAgentCore on microsandbox | E2B | Records | +| --- | --- | --- | --- | +| [Create a sandbox](https://openagentcore.dev/docs/performance#sandbox-start) | 1.20 s | 0.59 s | [core](docs/performance/core.jsonl), [e2b](docs/performance/e2b.jsonl) | +| [Sandbox created → initialization starts](https://openagentcore.dev/docs/performance#sandbox-start) | 0.80 s | — | [core](docs/performance/core.jsonl) | +| [Environment ready, preparation cache hit / miss](https://openagentcore.dev/docs/performance#environment-ready) | 2.3 s / 8.2 s | — | [sessions](docs/performance/sessions.jsonl) | +| [Suspend an idle sandbox](https://openagentcore.dev/docs/performance#suspend-and-resume) | 2.92 s | 0.60 s | [core](docs/performance/core.jsonl), [e2b](docs/performance/e2b.jsonl) | +| [Resume it](https://openagentcore.dev/docs/performance#suspend-and-resume) | 1.24 s | 0.75 s | [core](docs/performance/core.jsonl), [e2b](docs/performance/e2b.jsonl) | +| [First text of a Turn, suspended / running Session](https://openagentcore.dev/docs/performance#suspend-and-resume) | 3.06 s / 1.51 s | — | [sessions](docs/performance/sessions.jsonl) | + +E2B times are SDK calls on the client and include a 0.20 s API round trip; OpenAgentCore times are Core's own, over a 0.2 ms local network. [Setup](https://openagentcore.dev/docs/performance#setup) lists the hosts and versions. + +## Limits + +- **Docker is a compatibility backend**, for trusted workloads or hosts without KVM. Containers share the node's kernel, idle sandboxes never suspend, and files stay on the node's disk. See [Choose a sandbox backend](https://openagentcore.dev/docs/getting-started/sandbox-backends). +- **One writer per workspace.** A workspace is attached to one sandbox at a time, and storage keeps only its current files: no file history or forks. See [Workspace filesystem protocol](https://openagentcore.dev/docs/workspace-provider). +- **Workspace storage needs Linux.** The NFS adapter uses a kernel NFS mount on the Core host and on every node. See [Set up workspace storage](https://openagentcore.dev/docs/getting-started/workspace-storage). +- **Suspend is slower than E2B's pause**: 2.9 s against 0.6 s. A full node suspends one idle sandbox early to keep a slot free, so one new Session does not wait for a suspension, but a burst of new Sessions on a full node does. + ![Deployment overview](docs/assets/console-overview-en.webp) ## Quickstart diff --git a/README.zh-CN.md b/README.zh-CN.md index a92ff4679..fe08c4fbb 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -20,6 +20,28 @@ - **计算会休眠，工作区会保留。** 在 microsandbox 上（每个工作区都保存在独立存储中），空闲沙箱释放计算资源，并在兼容节点上带着文件恢复。E2B 会就地暂停空闲沙箱。Docker 以及你自己的 Linux、macOS 或 Windows 机器同样可以运行 Session。详见[沙箱提供商协议](https://openagentcore.dev/zh/docs/sandbox-provider)。 - **部件可替换。** 沙箱提供商、工作区文件系统、Harness 和模型提供商都通过[既定协议](AGENTS.md#protocols-at-every-boundary)接入。 +## 性能 + +单个 microsandbox 节点上的沙箱生命周期，E2B 以同样方式测量作为对照。2026-10-25 测得，每项为至少 10 次运行的中位数；每一行都链接到测量方法和原始记录，[`scripts/benchmark.py`](scripts/benchmark.py) 可以复现这些数据。 + +| 操作 | OpenAgentCore（microsandbox） | E2B | 原始记录 | +| --- | --- | --- | --- | +| [创建沙箱](https://openagentcore.dev/zh/docs/performance#sandbox-start) | 1.20 s | 0.59 s | [core](docs/performance/core.jsonl), [e2b](docs/performance/e2b.jsonl) | +| [沙箱创建完成 → 开始初始化](https://openagentcore.dev/zh/docs/performance#sandbox-start) | 0.80 s | — | [core](docs/performance/core.jsonl) | +| [Environment 就绪，准备缓存命中 / 未命中](https://openagentcore.dev/zh/docs/performance#environment-ready) | 2.3 s / 8.2 s | — | [sessions](docs/performance/sessions.jsonl) | +| [挂起空闲沙箱](https://openagentcore.dev/zh/docs/performance#suspend-and-resume) | 2.92 s | 0.60 s | [core](docs/performance/core.jsonl), [e2b](docs/performance/e2b.jsonl) | +| [恢复沙箱](https://openagentcore.dev/zh/docs/performance#suspend-and-resume) | 1.24 s | 0.75 s | [core](docs/performance/core.jsonl), [e2b](docs/performance/e2b.jsonl) | +| [Turn 的首个文本，已挂起 / 运行中的 Session](https://openagentcore.dev/zh/docs/performance#suspend-and-resume) | 3.06 s / 1.51 s | — | [sessions](docs/performance/sessions.jsonl) | + +E2B 时间为客户端上 SDK 调用的耗时，含 0.20 s 的 API 往返；OpenAgentCore 时间由 Core 自己测量，经过 0.2 ms 的本地网络。主机和版本见[测试环境](https://openagentcore.dev/zh/docs/performance#setup)。 + +## 适用边界 + +- **Docker 是兼容选项**，用于可信负载或没有 KVM 的主机。容器共享节点内核，空闲沙箱不会挂起，文件保存在节点磁盘上。详见[选择沙箱提供商](https://openagentcore.dev/zh/docs/getting-started/sandbox-backends)。 +- **每个工作区只有一个写入者。** 工作区同一时间只挂载到一个沙箱，存储只保留当前文件，没有文件历史或分叉。详见[工作区文件系统协议](https://openagentcore.dev/zh/docs/workspace-provider)。 +- **工作区存储需要 Linux。** NFS 适配器使用 Core 主机和每个节点上的内核 NFS 挂载。详见[设置工作区存储](https://openagentcore.dev/zh/docs/getting-started/workspace-storage)。 +- **挂起比 E2B 的暂停慢**：2.9 s 对 0.6 s。满载节点会提前挂起一个空闲沙箱以留出槽位，因此单个新 Session 不必等待挂起；满载节点上同时到达一批新 Session 时则需要等待。 + ![部署概览](docs/assets/console-overview-zh.webp) ## 快速开始 diff --git a/docs.json b/docs.json index 04e0453b1..d6809997c 100644 --- a/docs.json +++ b/docs.json @@ -31,7 +31,8 @@ "group": "Concepts", "pages": [ "docs/concepts", - "docs/architecture" + "docs/architecture", + "docs/performance" ] } ] diff --git a/docs/getting-started/sandbox-backends.md b/docs/getting-started/sandbox-backends.md index be34b9ace..735d06b58 100644 --- a/docs/getting-started/sandbox-backends.md +++ b/docs/getting-started/sandbox-backends.md @@ -14,7 +14,7 @@ The installer selects no backend. One installation uses one backend. To change i | Nodes to add | Yes | Yes | None | | You provide | Linux amd64 hosts with systemd and `/dev/kvm` | Linux amd64 hosts with systemd and rootful Docker Engine | An E2B API key and a template; a public HTTPS URL for Core | | Isolation | Each sandbox has its own kernel | Containers share the node's kernel | E2B's | -| Idle sandboxes suspend and resume | Yes, with checkpoints | No | No | +| Idle sandboxes suspend and resume | Yes, with checkpoints | No | Yes, paused in place by E2B | | Workspace storage | Required, on NFS | Own disk | Own disk | | Disk limits | Root disk and workspace quota | None | Set by the template | diff --git a/docs/performance.md b/docs/performance.md new file mode 100644 index 000000000..1c35d4bc0 --- /dev/null +++ b/docs/performance.md @@ -0,0 +1,87 @@ +--- +title: "Performance" +--- + +These measurements cover the sandbox lifecycle on microsandbox: starting a sandbox, preparing its environment, suspending it when idle and resuming it, with E2B measured the same way for comparison. [`scripts/benchmark.py`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/scripts/benchmark.py) produced every number on this page from the [records](https://github.com/MiniMax-AI/OpenAgentCore/tree/main/docs/performance) published beside it. + +## Results + +| Measurement | Median | Range | n | +| --- | --- | --- | --- | +| Sandbox create | 1.20 s | 1.16–1.22 s | 30 | +| Sandbox created → initialization start | 0.80 s | 0.76–1.07 s | 20 | +| Session created → environment ready, preparation cache miss | 8.15 s | 5.90–18.90 s | 10 | +| Session created → environment ready, preparation cache hit | 2.30 s | 2.30–2.60 s | 10 | +| Suspend | 2.92 s | 2.87–2.97 s | 10 | +| Resume → sandbox running | 1.24 s | 1.13–1.29 s | 10 | +| Turn first text, resumed Session | 3.06 s | 2.11–7.81 s | 10 | +| Turn first text, running Session | 1.51 s | 1.01–5.41 s | 10 | +| E2B create | 0.59 s | 0.28–1.68 s | 10 | +| E2B pause | 0.60 s | 0.49–0.87 s | 10 | +| E2B resume → first command | 0.75 s | 0.52–1.99 s | 10 | +| E2B API round trip | 0.20 s | 0.19–0.23 s | 5 | + +## What each measurement covers + +### Sandbox start + +- **Sandbox create**: Core's `create` step, from the Provider call to a booted sandbox. +- **Sandbox created → initialization start**: from the end of `create` until Core starts preparing the environment. It covers the Runtime starting, connecting to Core and Core claiming the initialization. + +### Environment ready + +**Session created → environment ready** runs from `agent.session.created` to `agent.session.environment.ready` on the application's stream, so it includes queueing, sandbox start and preparation. Each sample uses a new Template with one npm package (`prettier@3.3.3`), one file and one setup command. Its first Session misses the [preparation cache](../contracts/agents-api/environments.md#preparation-reuse) and installs everything; its second Session restores the prepared result. Restoring and saving the prepared result take under 0.4 s, so installing the package takes most of a miss, and a miss depends on the package registry. + +### Suspend and resume + +- **Suspend**: Core's `suspend` step. The node captures the sandbox's memory and disk to a checkpoint and stops it. A Session suspends after the idle timeout, or earlier when a new Session needs its slot. +- **Resume → sandbox running**: from the start of Core's `resume` step until the sandbox is running again. It covers restoring the checkpoint, waking the Runtime and confirming that the Environment resumed. +- **Turn first text**: from sending input to the first output text delta, once for the Turn that resumes a suspended Session and once for the next Turn on the same running Session. The difference is the wait a resume adds; the rest is the model. + +### E2B + +The E2B rows are durations of SDK calls on the client: `Sandbox.create`, `pause`, and `connect` plus one command for a resume. Before it pauses, each sandbox holds 650 MiB in a process and has written 100 MiB to disk, close to a sandbox after one Harness Turn; each resumes after the same idle wait as the Sessions. Each call includes at least one API round trip from the client, listed as **E2B API round trip** on a kept-alive connection. Core measures the OpenAgentCore rows itself, and reaches its node over the local network. + +## Setup + +- **Revision**: Core at `c5d4205d`; the sandbox node and Runtime at `170a4983`, whose later changes do not touch these steps. Measured on 2026-10-25. +- **Hosts**: Core and one microsandbox node on two Intel Xeon 6982P-C cloud VMs with Ubuntu 22.04 and Linux 5.15, 0.2 ms apart. Core has 8 vCPUs and 62 GiB; the node has 4 vCPUs and 15 GiB and runs at most 3 sandboxes and retains at most 14. +- **Sandboxes**: 2 CPUs, 4 GiB of memory and an 8 GiB root disk each, with the workspace on NFSv4.2. Idle sandboxes suspend after 300 s. +- **Sessions**: Codex with MiniMax-M2.7 at `https://api.minimaxi.com/v1`, driven from the Core host with `openai` 3.13.0, 10 samples of each phase. +- **E2B**: a template built from the same Runtime image with 2 vCPUs and 2 GiB, driven with `e2b` 2.51.0 from a client 0.20 s from the API, 10 sandboxes. + +## Reproduce + +Run on an installation with no other activity: the report reads every lifecycle step Core logs during the run. + +1. Install OpenAgentCore, add a microsandbox node with workspace storage, set a default model and issue a Project API key, as in [Install Core and Web](./getting-started/install.md). +2. On any host that reaches the API, with Python 3.10 or later and `pip install openai==3.13.0`, set `OPENAI_BASE_URL` and `OPENAI_API_KEY` and run the Sessions. It takes about 15 minutes, half of it waiting for the idle timeout: + + ```sh + python3 scripts/benchmark.py sessions results --model MiniMax-M2.7 --harness codex + ``` + +3. On the Core host, save the Core log and build the table. The report keeps the log lines it uses in `results/core.jsonl`: + + ```sh + docker compose -f ~/.oac/core/compose.yaml logs --no-log-prefix core > core.log + python3 scripts/benchmark.py report results core.log + ``` + +4. For E2B, `pip install e2b`, set `E2B_API_KEY` and give a template with at least 2 GiB of memory, then run `report` again: + + ```sh + python3 scripts/benchmark.py e2b results --template