Skip to content

fix(engine): downloads refuse hosts that resolve to private addresses - #6571

Open
miguel-heygen wants to merge 2 commits into
mainfrom
fix/download-public-addresses
Open

miguel-heygen wants to merge 2 commits into
mainfrom
fix/download-public-addresses

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

Downloads refuse hosts that resolve to private addresses. The downloader already refused private, loopback and link-local host names written in the URL and re-checked every redirect; it still connected to whatever a public name resolved to. Now every download resolves the host itself and refuses to connect when any answer is a private, loopback, link-local or otherwise reserved address (the same ranges the URL check uses, one list). The check runs on the address the connection uses, so there is no second lookup to race.

  • The transport is undici's own fetch with an Agent whose connect.lookup does the check (undici 7.25.0, exact, the version already in the lockfile; a dependency of the engine and the CLI, which keeps it external like the browser packages). It is imported on the first download, so a process that downloads nothing pays nothing.
  • The tests replace the transport through one seam (_setDownloadFetchForTests) instead of stubbing the global fetch; the existing download tests moved to it unchanged otherwise.
  • Behind an env proxy (HTTPS_PROXY with NODE_USE_ENV_PROXY=1 or --use-env-proxy) the proxy resolves the name, so downloads use Node's own fetch there and the URL and redirect checks stay the guard. fetchGoesThroughEnvProxy moved to @hyperframes/core/env-proxy, one owner for the CLI's DNS probe, the publish hint and the downloader.
  • A refused address surfaces as the existing non-retryable http_rejected failure ("Download URL is not permitted"), so it is not retried.

Tests

urlDownloader.publicAddress.test.ts resolves a host to a private IPv4, a link-local address mixed in with a public one, and a loopback IPv6 address: all three are rejected without a retry, for downloadToTemp and fetchPublicHttpsText. They fail with the address check removed.

Checked on a Linux box

  • Engine download, mixer and extractor tests: pass three runs in a row.
  • A real render with a remote video and image (both fetched by the producer through this downloader): same video hash on main and on this branch, three runs each. Wall time 7.72, 8.24 and 9.78 s on the branch against 9.02, 8.25 and 9.76 s on main: the same within the machine's noise.
  • The CLI built with tsup (undici external) renders the same remote-asset project to the same hash.
  • The first download in a process loads undici: about 45 ms once per process, on a render of several seconds or more.

Downloads checked the host name in the URL but connected to whatever the name
resolved to, so a public name pointing at a private address was fetched. Every
download now connects through a resolver that refuses any answer in a private,
loopback or link-local range, on the address it actually connects to.

The transport is undici's own fetch with an Agent, loaded on the first download;
the tests replace it through one seam instead of stubbing the global fetch.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES · 1 blocker · 0 important · 0 nits · head 3ac69653

The address check itself is sound: it runs on the connect-time lookup, handles all: true for happy-eyeballs, and the new test exercises real undici. The blocker is that this transport drops env-proxy support. Inline.

— Genesis

Comment thread packages/engine/src/utils/urlDownloader.ts
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 170 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

The address check resolves the host on this machine, which a network that only
reaches the internet through HTTPS_PROXY cannot do. When Node's fetch follows
the proxy variables, downloads now use it and the URL and redirect checks stay
the guard. The test for "does fetch follow the proxy" moves to core, where the
CLI's DNS probe and its publish hint use the same one.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES — 1 resolved, 0 open, 2 new · 3ac6965 → e5856f0

The proxy fix is right. Required CI is red on this head (run 40566977216) for two reasons this PR causes:

  • Producer: unit tests. packages/producer/src/server.test.ts:229 and :238 (prepareRenderBody — previewUrl) still stub the global fetch. fetchPublicHttpsText now goes through undici, so the stub is never called: 0 calls, and no prepared. This is the same move the engine tests got. Switch them to _setDownloadFetchForTests, which may need exporting from the engine index.
  • CLI: npx shim (all 3 OSes). See inline.

— Genesis

@@ -1,13 +1,8 @@
import { promises as dns } from "node:dns";
import { fetchGoesThroughEnvProxy } from "@hyperframes/core/env-proxy";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker (CI): the npx-shim job doesn't build core, so under Node the node condition resolves @hyperframes/core/env-proxy to dist/envProxy.js, which doesn't exist. skills.test.ts fails 40 tests with ERR_MODULE_NOT_FOUND (via updateCheck.ts → hostAnswers.ts). Either build core in that job or keep this import out of the shim's module graph.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant