Status: experimental; vk-gitlab, vk-hub and vk node implement it. vk-hub-proto
defines the wire types: client for the hub's client API, dispatch for session job
messages, and job for the job spec. Serde tests pin their JSON. See
GitLab jobs for the design and rationale.
Three parties take part:
vk-gitlab— a daemon that is gitlab-runner's GitLab-facing side: it holds the runner tokens, asks GitLab for jobs, writes their traces and states, and submits each job to the hub as a job spec.vk-hub— reserves capacity on nodes, places jobs, relays and stores their output, and reports how they ended.vk node— runs each job in microVMs, stage by stage, and talks to GitLab directly for the job's own transfers: the clone, dependency artifacts, artifact uploads.
| Credential | Held by | Used for |
|---|---|---|
runner authentication token (glrt-…) |
vk-gitlab alone |
POST /api/v4/jobs/request, /runners/verify |
relay key (ed25519, <state_dir>/hub/key.pk8) |
vk-gitlab alone |
signing every client API request; its session with the hub |
CI_JOB_TOKEN (the job response's token) |
vk-gitlab; the job's node, in the spec |
the daemon: trace, state; the node: clone, artifact upload |
dependency tokens (dependencies[].token) |
the job's node | downloading that dependency's artifacts |
registry credentials (credentials) |
the job's node | pulling the job's images |
| node key | the node | its session, nothing else |
gitlab-runner sends the job token, not the runner token, on every per-job endpoint —
PUT /jobs/:id, PATCH /jobs/:id/trace, POST /jobs/:id/artifacts — so a node holding
the job token holds nothing the job itself does not. The hub sees a spec's secrets in
transit and keeps them in memory only, until the node has accepted the job; its records
and audit log hold the spec redacted (CiJob::redacted).
vk-gitlabqueues a reservation request for the runner's placement (POST /v1/reservations,queue: true) and long-polls its ticket until the hub grants it: the hub has offered one envelope to a node and the node accepted.- It takes a slot under
concurrentand the runner'slimit, and claims the grant (POST /v1/reservations/<id>/claim). - It asks GitLab for a job (
POST /api/v4/jobs/request), renewing the reservation while the request is outstanding, and reports the admission's outcome to the hub. On 204 it gives the slot and the reservation back and queues again. - On 201 it writes the job to its state file, translates the response into a spec and
submits it on the reservation under its admission (
POST /v1/jobs). - The hub sends the node
start; the node journals the job, converts the reservation into the job's ledger entry and answersaccepted. - Seeing the job accepted,
vk-gitlabsends GitLabPUT /jobs/:idstate=running— the commit of a job taken undertwo_phase_job_commit. - The node runs the stages, streaming masked output to the hub, which stores it and acks.
vk-gitlabreads it from its GitLab offset and patches the trace. - The node uploads the artifacts to GitLab itself, then sends its result once its output is all acked.
vk-gitlabpatches the rest of the trace, sends the finalPUT /jobs/:id, settles the job with the hub (POST /v1/jobs/<id>/settle) and drops it from its state file.
GitLab hands a runner a job only when the runner asks, and a job it has handed over cannot move to another runner. So the daemon holds capacity before it asks, never after, and the hub decides fleet-wide which runner's request gets the room a node frees, using the operator's capacity policy. See capacity arbitration for configuration and monitoring:
- Queued, not held. Each of a runner's
request_concurrencyrequest loops queues a reservation request and polls its ticket without holding a slot underconcurrentorlimit. Unclaimed tickets expire after 120 seconds without a poll and end with the relay's session. Only claimed tickets survive a hub restart. Polling a lost ticket returns 410, and the loop queues again. - Arbitrated. The hub offers room to the waiting tickets and the queued jobs of each
placement together: a job at its
place_within_secsdeadline first, then a job holding its reservation, then production work other than probes, then production probes, then normal work — except that afterproduction_probe_burstproduction probes (default 1) since normal work was last granted room, normal work is owed the next turn ahead of production probes. A hub starts owing it that turn, so after a restart normal work goes first. Within each, the tenant holding the least for itsweightgoes first, the oldest first. Shares are not set aside: room an idle tenant leaves goes to the others. A ticket waits while its tenant holds itscap, its relay itsconcurrentor its runner itslimit— counting its tickets offered, granted or claimed and its jobs, those its heartbeat counts without an admission included — while its runner backs off, and until the relay's current session has sent a heartbeat listing every request it holds. A job submitted without an admission waits on its tenant'scaptoo. - Production headroom. The capacity policy accepts at most 64
headroomentries. Each reservesslotsenvelopes of its placement (pool, labels, envelope) for production work. Normal work must leave the fleet that much room, or all remaining room if already short. Entries are checked independently; identical placements use their greatest slot count. An entry with no room on a node does not block work there. - Bounded discovery. A request of a runner that took no job in the last
busy_secsis a probe, GitLab likely having nothing for it: at mostprobe_limitprobes are out fleet-wide, one per runner. Anemptyorerroroutcome backs the runner off forbackoff_initial_secs, doubled with each consecutive one up tobackoff_max_secs, ±25%; ajoboutcome clears it. These are capacity policy fields (vk-hub capacity-policy); defaults 60, 1, 1 and 60. - Then a reservation. The hub offers the ticket's envelope to a node, least loaded first
(Placement). The node decides it through its admission ledger at once — a
reservation never waits in the ledger's queue — and counts it as granted memory and disk
while it lasts. A node that refuses or does not answer within 5 seconds is skipped until
every fitting node was tried; they are tried again 2 seconds later. An acceptance grants
the ticket:
{admission, reservation, node, envelope, lease_secs}, the lease what remains of the node's. - Then a claim. The daemon takes a slot — giving the grant back as
releasedwhen another loop took it meanwhile — lists the admission in its heartbeat'srequest_admissions, and claims the grant. A grant not claimed before its lease runs out is released. The hub stores a claimed admission, so it outlives a hub restart. - Then the job request. The reservation's lease covers the request: a lease of
lease_secs(default 90) against a job request the daemon times out at 60 seconds, renewed everylease_secs / 3while the request is outstanding. A request in flight is never aborted: GitLab may already have assigned the job. - Then the outcome. The daemon reports what came of the admission:
job,empty(GitLab answered 204),error(the request failed) orreleased(given back without asking GitLab), retrying until the hub takes it or answers 400, 404, 409 or 410. Any outcome butjobends the admission and releases its reservation. A job's admission moves to the heartbeat'sjob_admissionsand is charged until the job is settled. - Then the job. The job is submitted on the reservation at once, naming its admission. The
node sizes the job by its own rules —
MICROVM_CPUS,MICROVM_MEM, the script's# vk:hint, history — and returns what the job does not need to the ledger; a job needing more than its reservation waits for the rest in the ledger, as an executor job waits today.
Heartbeats reconcile claims whose outcome was lost. A claimed admission that a heartbeat of the claiming session, received at least 10 seconds after the claim, lists in neither list is reclaimed and its reservation released. After a reconnect or a hub restart the relay's first heartbeat of the new session decides: an admission listed as a job's is one, one not listed is reclaimed.
Idle reservations are bounded: a runner holds at most request_concurrency tickets waiting,
offered or granted, and a claimed grant lasts one request plus a renewal. A reservation that
ends mid-request — its node lost, or the node's clock ran its lease out — leaves a job that
arrives meanwhile without one; the hub then queues it under its admission and places it
within its place_within_secs (vk-gitlab asks for 300 seconds), and failing that ends it
no_capacity, which the daemon reports as runner_system_failure. That window is the only
way a job taken from GitLab waits for capacity.
The lease is a duration on the node's monotonic clock from its acceptance; no wall-clock time crosses the wire. The daemon counts it from when it sent its request, so it always believes the lease ends earlier than the node does.
A runner in vk-gitlab's configuration names its placement; every runner of one vk-gitlab
takes jobs from the one GitLab server its [gitlab] url names:
[gitlab]
url = "https://gitlab.example.com"
[[runners]]
name = "ci"
token_file = "/etc/vk-gitlab/runner-ci.token" # glrt-…
pool = "ci"
labels = ["large-memory"]
envelope = { mem = "16G", cpus = 8, disk = "16G" }
request_concurrency = 1concurrent and each runner's request_concurrency are at most 128, and a daemon has at most
64 runners: vk-gitlab refuses a configuration past them, and the hub ends the session of a
relay whose inventory lists more runners or a larger request_concurrency.
GitLab's job response does not carry the job's tags, and GitLab only gives a runner jobs
whose tags it has, so placement is per runner: one runner per class of work, with the tags
of that class registered in GitLab and its pool and labels here. A pool is a set of nodes the
hub's operator names (vk-hub nodes pools <id> <a,b>, or on the node's page); a label is one
a node declares in its own configuration and reports in its inventory's labels, absent from
an older node's. The placement stays in the daemon's configuration, which it reports to the
hub; the hub sets the relay no ceiling on it.
A node's concurrency ceiling (vk-hub nodes ceiling) and its own executor limit
([executor.schedule] max_concurrency, which the node reports) cap its placed work at the
smaller limit. The hub offers and starts nothing on a node whose reservations and unfinished
jobs reach the cap. The node enforces it independently, refusing ceiling or concurrency
by its own count. Running jobs above a lowered ceiling carry on.
A host runs its own gitlab-runner with the vk executor or takes placed jobs, never both. A node
that finds such a runner on its host says so in its report and refuses with runner every new
offer and every start without a reservation (a reservation it already holds is renewed and
started on); the hub places nothing on it and counts it in no capacity.
A host that takes placed jobs sets [node] runner = "none" (unset gives none only where no
gitlab-runner unit or binary shows), so the node drains, resets and updates it as one whose
runner it supervises.
Among the nodes with room, the hub offers a reservation, or starts a job without one, on the
least loaded first, then the one with the most room, then by node ID. A node's load is the
largest of: its placed work against that work plus its room (the envelopes it still fits);
the vCPUs of its placed work per CPU; and its heartbeat's 1-minute load average per CPU, left
out for a node too old to report one. Work therefore spreads across nodes even at moderate
load — concurrent builds, and the traffic they put on shared services such as the registry,
are not stacked on the one node with slightly more room, a host busy with work of its own
takes new work last, and losing a node loses fewer jobs. Memory, disk and the cap
bound each node, and capacity (fits) is the sum of every node's room.
A job whose image its node builds from the checkout (dockerfile: or compose:, its own or a
service's) goes first to a node that ran a job of the same project with the same such images
within that node's image_cache_idle_secs, while that node is lightly loaded: its 1-minute
load average, plus the vCPUs of jobs sent to it in the last minute and of this job, per CPU,
below the hub's image_affinity_max_load (default 0.7), and without this job no more than
image_affinity_max_extra_load (default 0.25) above the lowest such load among candidates.
That node boots it in seconds where another spends a minute or two building the image. A job
submitted on a reservation on a node without the image then starts on the warm node without it,
and the hub releases the reservation. Under heavier load the order above holds.
A project of the capacity policy may restrict where its jobs go with placement_rules (at
most 64): each rule names nodes (node IDs), pools (a node must be in one), or both (at most
256 of each), and applies to every job of the project or, with branches (globs), to jobs whose
ref matches one. A job goes only to a node every rule that applies to it allows, image
affinity included. Rules match the ref the relay reports for the job (branch, tag or merge
request alike), as GitLab gives it to the relay. That ref is no trust boundary: a merge request
from a fork whose branch is named main gets main's rules. To confine a project's jobs, use
a rule without branches. A project's placement rules must be the same, in any order, in every
tenant that lists it; its page on the hub edits them in all of them at once.
The hub offers a runner's request before GitLab says which project's job it takes, so rules
cannot steer reservations: a job submitted on a reservation on a node its rules do not allow
has the reservation released and waits for an allowed node, ending at its place_within_secs
deadline with project placement rules as the reason when nodes had room but none was
allowed. Give a restricted project a runner of its own, mapped with the nodes and pools its
rules allow (see capacity arbitration), so its requests are reserved
where its jobs may go.
A relay job whose project its tenant stops listing after the job was submitted is held: it
waits, keeping any reservation it was submitted on so it starts as soon as the project is
listed again, or until its deadline, when it ends with project no longer listed for its tenant as the reason. A job recorded by a hub from before capacity arbitration, without a
scheduling identity, is not held.
HTTP/1.1 and JSON over TLS 1.3, on the hub's node listener (addr), under /v1/. Bodies are
vk_hub_proto::client's. vk-gitlab is an enrolled relay (GitLab relay
enrollment): vk-hub token create --kind gitlab | vk-gitlab join <hub-url> --token - [--ca FILE] pins its key on the hub and keeps it, the hub's URL, its CA and
the relay's ID under <state_dir>/hub/ (0700); vk-gitlab leave deletes them. Both are
refused while a state file holds a job.
Authentication. Each request is signed by the relay's key, as a node signs a release
download: vk-relay carries the relay's ID, vk-time the time, and vk-signature its
ed25519 signature over client_request_message — the relay ID, method, path and query, the
body's SHA-256, the time, and the TLS connection's exporter. The hub checks that the relay is
enrolled and not removed, the time within DOWNLOAD_SKEW_SECS, and the signature. A signature
is good on its own connection only, so the daemon opens one connection per request and signs
a retry again; a TLS-terminating proxy in front of the hub breaks the binding. A relay is
held to no pool or envelope ceiling: it already holds the runner tokens and sees every job's
secrets.
Session. Besides the client API, vk-gitlab run holds a session with the hub at
/v1/gitlab/session, as a node does, redialing it when lost: it reports its version, host,
system ID, concurrent, its GitLab server's URL and the SHA-256 of its certificate's public
key, and each runner's name, GitLab runner ID, placement, limit, request_concurrency and
whether GitLab took its token; every heartbeat, each runner's job requests outstanding and jobs
held, with the admission of each (request_admissions, job_admissions), and its last GitLab
answer; and each runner's state (polling, waiting_capacity, server_pending,
unhealthy, token_refused). A change to requests, jobs or states is sent at once, within
250 ms, besides the periodic heartbeat. The hub answers with its status of the server. A runner
asks GitLab for jobs only while that status is approved: not before the first status, nor
while the server is pending, rejected or the relay rebound, nor once the hub refused the
relay for good. Jobs it already holds run either way.
The session is at protocol version 2, which adds request_concurrency and the admission lists,
or 3, which adds updates; the hub speaks 1 to 3, vk-gitlab 2 and 3. The hub takes new work only
from a relay whose session is at version 2 or later: a version-1 relay keeps its session, and
replays its submissions, follows, cancels and settles the jobs it holds, but its capacity and
reservation requests and its new jobs are refused 403 forbidden. A hub that speaks only
version 1 refuses a relay's session for its version: the relay logs that the hub must be
upgraded, asks GitLab for no jobs, and dials again every minute or so. Upgrade the hub first,
then the relays: a relay whose [hub] allow lists update is updated from the hub's Updates
page or vk-hub relays update, its running jobs resumed by the release
(docs/gitlab-relay-enrollment.md, "Updates").
| Request | Body | Answer |
|---|---|---|
POST /v1/capacity |
CapacityRequest |
200 Capacity |
POST /v1/reservations |
ReservationRequest |
201 ReservationGrant, or 202 ReservationTicket when queued |
GET /v1/reservations/tickets/<ticket>?wait=<secs> |
— | 201 ReservationGrant, or 202 ReservationTicket while queued |
DELETE /v1/reservations/tickets/<ticket> |
— | 204 |
POST /v1/reservations/<id>/claim |
— | 204 |
POST /v1/reservations/<admission>/outcome |
OutcomeRequest |
204 |
POST /v1/reservations/<id>/renew |
RenewRequest |
200 ReservationGrant |
DELETE /v1/reservations/<id> |
— | 204 |
POST /v1/jobs |
JobSubmission |
201 JobView |
GET /v1/jobs/<id>?after=<revision>&wait=<secs> |
— | 200 JobView |
GET /v1/jobs/<id>/output?offset=<n>&wait=<secs> |
— | 200 bytes |
POST /v1/jobs/<id>/cancel |
CancelRequest |
202 JobView |
POST /v1/jobs/<id>/settle |
— | 204 |
POST /v1/clients |
ClientHello |
204 (API keys of other clients; a relay is refused) |
Long polls. wait on the GETs, and wait_secs in a capacity request, hold a request for
at most 60 seconds: a ticket until it is granted, a capacity answer until its revision passes
after, a job view until its revision passes after, an output read until there are bytes
past offset or the output is complete. A long poll that times out answers as a plain request
would.
Capacity. {runner, placement, after, wait_secs} → {revision, fits}. fits counts
envelopes, not nodes: a node with room for two counts two. Advisory — the hub's count, from
heartbeats less the reservations and starting jobs they do not show yet, of how many envelopes
of the placement its ready nodes could take, each node at most what its cap leaves past the
reservations and jobs it holds — and no longer asked by vk-gitlab, which queues instead.
Reservations. {runner, queue, request_id, placement, lease_secs, wait_secs} →
{admission, reservation, node, envelope, lease_secs}. The request joins the hub's
queue, whose offers filter the pool's nodes on labels, readiness
and headroom. With queue, the hub answers 201 with a grant if one is ready at once, else 202
{ticket}; without, it waits up to wait_secs (at least 5 seconds) for a grant, then answers
503 no_capacity. A relay names its runner. A runner may have request_concurrency tickets
waiting, offered or granted, a principal 256 unended, the hub 4096; past them, and while a
relay's session is not connected, the hub answers 503 unavailable. A grant's admission is
what its outcome reports on and its job's submission names.
GET on a ticket answers 202 while it waits and 201 with the grant, its lease_secs what is
left of the lease, once a node accepted; 410 reservation_gone for a ticket that ended —
cancelled, unpolled, refused, or granted and not claimed in time — or that the hub does not
know. DELETE cancels it, releasing an offered or granted reservation; it answers 204 for a
ticket the hub does not know, and 409 once claimed. Both answer 404 for another principal's.
claim takes a grant for a GitLab request: 204, again for one already claimed; 410 for no such
grant, or once its reservation or lease is gone. outcome takes {admission, outcome},
outcome one of job, empty, error, released: 204, again for the same outcome; 400 for a
body naming another admission than the path; 404 for an admission the hub does not know or of
another principal; 409 for an outcome the admission's state does not allow — anything but
released before the claim, error or released after its job, another once it ended.
renew extends a lease from now and answers the lease the node granted; a reservation that
lapsed, was released or whose node was lost answers 410 reservation_gone. DELETE releases
it, and answers 204 for one already gone.
Jobs. {runner, admission, request_id, placement, reservation, place_within_secs, spec} →
JobView. A job under an admission must match its runner, tenant, placement and reservation
(403 forbidden), and an admission takes one job (409 conflict); a job on a queued
reservation must name its admission. A new job refused with a 4xx ends its admission as
empty. The hub starts the job on the reservation's node — unless a lightly loaded node holds
the job's built image (placement), when it releases the reservation and starts
the job there; with no reservation, or one that is gone, it places the job as it places a
reservation, for up to place_within_secs. A node that refuses the start leaves the job free
to place again; a start the node may have taken is never placed twice — see
failures. The spec is at most 512 KiB serialized (413 too_large).
JobView is {id, revision, state, node, stage, output_len, cancel, result}: state is
queued, starting, running or finished, and result is present once finished, with the
node's usage when it sent one (see Result).
revision moves with every change but the output's length.
Output. The answer's body is the output from offset, at most 1 MiB, with
vk-output-offset (its first byte's offset), vk-output-length (the output's length so
far) and vk-output-complete: true when the job has ended and the body reaches its end. An
offset past the end answers 416 with vk-output-length. The hub keeps a job's whole
output until it is settled, so the daemon can read from any offset — whatever GitLab says it
holds.
Cancellation. {mode}: graceful or immediate. A later immediate overrides a graceful;
cancelling a finished job changes nothing and answers its view.
Settle. The daemon has delivered the outcome to GitLab; the hub drops the job's output and keeps its record.
Idempotency and retries. request_id — 16 random bytes, hex — makes a create
idempotent for a day: the same ID with the same body answers the first answer again, or 410
not_found once the hub has dropped that job from its history; with another body, 409
conflict. A queued request retried answers its ticket, or the ticket's grant. Renew,
release, claim, cancel and settle are idempotent by nature, as are a ticket's cancel and an
outcome repeated.
The daemon retries transport errors and every retryable code with backoff from 1 to 30
seconds and jitter, honouring retry_after_secs, and never retries another 4xx — except
job cancels and settles. While handling a job, it retries a cancel or settle after any
failure, including a refused relay, with the same backoff until accepted, 400 or 404.
Cancel delivery runs alongside the trace; an immediate cancel replaces a graceful one.
A settle refused with 409 is retried until the job ends. The job keeps its slot during
settlement retries because the hub still charges its admission. When the daemon stops
taking jobs for shutdown or update, a pending settle is left to its next start, which
cancels and settles reported jobs and resumes running ones.
| Status | code |
Meaning | Retry |
|---|---|---|---|
| 400 | invalid |
unparseable body, malformed ID, a job without a project, an outcome naming another admission than its path | no |
| 401 | unauthorized |
no signature, a bad one or its time too far off, or a relay the hub does not know or removed | no |
| 403 | forbidden |
a job of another GitLab server than the relay's, of a relay told rebound, or an API key's GitLab job; a relay below protocol 2 asking for new work; a runner or project its tenant does not allow; a job that does not match its admission |
no |
| 403 | approval_pending |
a job of a project, or of a server, an operator has yet to approve on the hub | no: the job fails |
| 403 | project_rejected |
a job of a project, or of a server, an operator rejected | no: the job fails |
| 404 | not_found |
no such job, reservation, grant or admission of this relay's; another principal's ticket | no |
| 409 | conflict |
request_id reused with another body; an outcome the admission's state does not allow; an admission already used by a job; a claimed ticket cancelled; a settle of a job not finished |
no (a settle: yes) |
| 410 | reservation_gone |
lapsed, released, or its node lost; a ticket that ended or the hub does not know | reserve again |
| 410 | not_found |
a create retried after its job left the hub's history | no |
| 413 | too_large |
spec over 512 KiB | no |
| 500 | internal |
— | yes |
| 503 | no_capacity |
no node accepted in time | yes, after retry_after_secs |
| 503 | unavailable |
hub busy or starting; the reservation queue full; the relay's session not connected; an admission being updated | yes, after retry_after_secs |
An error body is {error, code, retry_after_secs?, approval_url?}. Unknown codes read as other;
unknown fields are ignored. When the hub serves a web UI, approval_pending and
project_rejected include approval_url: the page where an operator decides on the project's
approval, <ui_url>/gitlab/<instance>/projects/<project ID>, or, for its server, GitLab ›
Servers, <ui_url>/gitlab, which lists the servers pending.
A job refused approval_pending or project_rejected is not held: approval is a human step.
vk-gitlab releases its reservation and fails it in GitLab, uncommitted, as
runner_configuration_error (script_failure where GitLab does not know it), its trace ending
with instructions and a link, for example:
ERROR: project acme/web (ID 12) is waiting for approval on the fleet hub: https://hub.example.com:8444/gitlab/main/projects/12
Retry this job once an operator has approved it there.
Without approval_url (an older hub or one without a UI), the trace uses the enrolled hub URL:
ERROR: project acme/web (ID 12) is waiting for approval on the fleet hub https://hub.example.com.
Retry this job once an operator has approved it (GitLab > Pending).
GitLab's retry: when: runner_system_failure does not retry that reason, so a pending project
does not loop. The hub keeps the refusal as the answer to the submission's request_id, so it
counts once.
Version 3 (JOBS) adds one message type each way to the existing session, {"type": "job", "kind": …}, whose payloads are vk_hub_proto::dispatch's. A job message in a session below 3
is a protocol error. PROTOCOL stays at 1 to 2; each side that implements version 3 negotiates
from a range of its own reaching it: the hub accepts 1 to 3, so a node offering 3 gets it, and
one offering at most 2 is steered and never offered a job.
| Hub → node | Answer | Meaning |
|---|---|---|
offer {reservation, envelope, lease_secs} |
offer_reply |
set the envelope aside |
renew {reservation, lease_secs} |
lease |
extend the lease from now |
release {reservation} |
lease gone |
give it back |
start {job, reservation?, envelope, spec} |
job |
run a job |
output_ack {job, offset} |
— | output stored durably up to offset |
cancel {job, mode} |
job, then result |
stop a job |
recorded {job} |
— | result stored; stop repeating it |
| Node → hub | Meaning |
|---|---|
held {reservations, jobs} |
everything the node holds, once per session after its report |
offer_reply {reservation, reply} |
accepted {lease_secs} or refused {reason, message?} |
lease {reservation, state} |
held {remaining_secs} or gone {why}: expired, released, started, unknown |
job {job, state} |
accepted, refused {reason, message?}, running {stage, activity?, guest? {cpus, mem_mib}}, finished |
output {job, offset, data} |
output bytes from offset, base64, at most 256 KiB |
result {job, result} |
how the job ended; repeated until recorded |
egress {job, report} |
from version 6: a running job's egress report so far, when it changes and after a reconnect |
A running state's guest is the size of the job's VM, once its prepare has sized it; a hub
older than it ignores it.
An egress report is cumulative, with a revision: the hub keeps the highest, so one sent again
replaces the counts rather than adding to them, and the result's report, the last, has the
highest. The node takes it at each stage's start and every 5 seconds, at most once every 2, and
sends it when it changed; a session below version 6 gets the result's alone.
Refusal reasons are memory, disk, cpus, not_ready (draining, drained, quarantined, in
maintenance), policy, no_reservation, invalid, ceiling (the node's placed jobs not
finished and reservations held reach the hub's ceiling, as the node last applied it),
concurrency (the same, at the node's own max_concurrency) and runner (the host runs its own
gitlab-runner with the vk executor); one the hub does not know reads as other, as ceiling,
concurrency and runner do on a hub older than them.
Reservations. A node answers an offer at once from its ledger: granted, or refused with
the resource that is short. It holds the entry in the ledger itself, with no job process
behind it, and drops it when the lease runs out on its monotonic clock, telling the hub
lease gone expired. A lease is at most 600 seconds. A drain refuses new offers and lets
held reservations lapse; a quarantine releases them.
Start. The node journals the job before answering accepted, so a start redelivered after
a reconnect is answered from the journal and not run twice. A start naming a reservation the
node does not hold is admitted like any other ask, without waiting: refused no_reservation
when it does not fit.
Output. Offsets count bytes of the output as GitLab will hold it: masked, with section
markers, timestamped (unless the job sets FF_TIMESTAMPS false), cut at the trace limit. A
stamp is when the node read the line (its start, for a line that spans reads), however late
the daemon reads it from the hub. The node sends at most 4 MiB past the last ack, keeps every
byte past it on disk, and after a reconnect resends from the offset the hub acked. The hub
appends each chunk to the job's output file and acks once it is synced; a chunk overlapping
what it holds is trimmed, one leaving a gap is a protocol error.
Result. {failure?, exit_code?, message?, output_len, artifacts, usage?}, sent once every
byte up to output_len is acked. artifacts gives each upload's outcome: uploaded, skipped,
too_large or failed. usage is what the job used on the node, {wall_ms, cpu_ms?, peak_mem_bytes?, cpus?, mem_mib?}:
wall_ms: from the driver's start to the job's end, cleanup included;cpu_msandpeak_mem_bytes: the job's VM supervisor and every process under it — the VMM and its vCPUs, service VMs, the switch, the forwards — read from/procafter the last stage and before cleanup, using the same measurement as the executor'sjob resource usagetrace. CPU time is user plus system, guest execution included. Peak memory is each process'sVmHWMsummed: an upper bound where they did not peak together. Host-side cache and artifact transfers, and a host checkout, are not counted. Both are absent when no VM was up to read;cpusandmem_mib: the guest's size, from the job'sMICROVM_CPUSandMICROVM_MEMclamped by the node's ceilings.
usage is optional, as are all its fields except wall_ms.
Reconnects. After its report, a version-3 node sends held. The hub releases each
reservation it does not know, cancels immediate each job it has given up on, and answers
each job it follows with output_ack at the offset it holds.
vk-gitlab speaks GitLab's runner API as gitlab-runner 19.5 does
(network/gitlab.go, network/trace.go):
- Runners.
POST /api/v4/runners/verifyat start with theglrt-token and asystem_idof the daemon's own, persisted. - Job requests.
POST /api/v4/jobs/requestwithRUNNER-TOKEN,infoandlast_update— theX-GitLab-Last-Updateof the previous answer, which lets GitLab hold the request until its queue changes. 201 is a job, 204 none, 403 a token GitLab refuses (the runner stops), 429 and 503 back off asRetry-Aftersays. - Features.
info.featuresadvertisesvariables,image,services,artifacts,cache,fallback_cache_keys,upload_multiple_artifacts,upload_raw_artifacts,refspecs,masking,raw_variables,artifacts_exclude,multi_build_steps,trace_reset,trace_checksum,trace_size,cancelable,cancel_gracefully,return_exit_code,service_variablesandtwo_phase_job_commit. Notsession,terminal,proxy,shared,vault_secrets,service_multiple_aliases(a node's executor takes one alias per service),image_executor_opts,service_executor_opts,native_steps_integrationorjob_inputs.info.executorisvk,info.shellbash. - Commit.
PUT /jobs/:idstate=runningonce the node has accepted the job. A job that cannot be placed is failed instead (runner_system_failure). - Trace.
PATCH /jobs/:id/trace?debug_trace=<CI_DEBUG_TRACE>withJOB-TOKENandContent-Range: <start>-<end>(inclusive, no unit), at most 1 MiB a patch, everyX-GitLab-Trace-Update-Intervalseconds (3 by default, capped at 15 minutes). 202 moves the offset on; 416 moves it to the end of the response'sRange: 0-<n>, which is also how a restarted daemon finds where to resume; 404 or 403 abort the job. The trace preserves the node's output byte for byte. Errors for jobs the daemon refuses or cannot decode, submit or place use gitlab-runner's logger timestamp format (stream00, stdout), unless the job'sFF_TIMESTAMPSis false. The state file preserves this flag across restarts. - Keep-alive. With no new output for 30 seconds,
PUT /jobs/:idstate=runningwith the trace'scrc32:<hex>checksum and byte size. - Final update. Once the hub reports the job finished and its output complete and every
byte is patched:
PUT /jobs/:idwithstatesuccessorfailed,failure_reason,exit_codeandoutput {checksum, bytesize}. 202 and 412 are asked again after the update interval, with backoff up to an hour, as gitlab-runner's final update does. - Cancellation. Every PATCH and PUT answer carries
Job-Status.cancelingasks the hub for a graceful cancel;canceledorfailed, or a 403, an immediate one, after which the daemon writes nothing more for the job.
The daemon keeps one state file per runner, 0600, recording each job it has taken — GitLab
job ID, job token, hub job ID, request_ids — before it submits or commits it. A restarted
daemon resumes each from the hub's view and GitLab's trace offset; a job it had not yet
submitted is submitted with the same request_id.
The state file is <state_dir>/<runner name>.json. state_dir defaults to
/var/lib/vk-gitlab and is created 0700; the daemon refuses to start when group or others
can access it. A daemon run as an unprivileged service user cannot create the default, so
create it for that user before the first start:
install -d -m 0700 -o <user> -g <group> /var/lib/vk-gitlab. A record stays until the hub
has settled its job, including a job already reported to GitLab or abandoned before
submission. A runner's name defaults to its token's short form, so a runner whose token may
change needs an explicit name, or the new token starts an empty file and the old one's jobs
are never resumed; the daemon warns at start about state files no configured runner owns.
Hub jobs and reservations belong to the relay that created them, and the hub answers
not_found for them to any other. The state file records the relay's ID, and a daemon
enrolled as another relay refuses to start while the file holds jobs, as it does a file of a
vk-gitlab from before relays. Before enrolling again (join --replace), stop the daemon
(SIGTERM) and let its running jobs finish within shutdown_timeout; join and leave are
refused while a job is still recorded, until it ends or the file is removed, which gives the
job up to GitLab's timeout.
The daemon does not rotate runner tokens. It verifies the token at start and daily, warns
from a week before GitLab's token_expires_at, and logs an error once it has expired.
The hub never retries a GitLab job. It places a job again only while no node can have
started it: after a refused start, or when the reservation it was sent to is gone and no
start went out. A start whose answer was lost waits for the node's held to say whether it
took it.
| How it ended | JobResult.failure |
GitLab failure_reason |
Older GitLab |
|---|---|---|---|
| success | — | — | — |
| a step's non-zero exit | script |
script_failure |
|
past timeout_secs |
timeout |
job_execution_timeout |
|
| image or service image not pulled or built | image_pull |
image_pull_failure |
runner_system_failure |
| unsupported job, refused by node policy | configuration |
runner_configuration_error |
script_failure |
| clone, artifact or cache transfer failed | external_dependency |
runner_external_dependency_failure |
runner_system_failure |
| VM, agent or host failure | system |
runner_system_failure |
|
| node stopped or drained hard mid-job | interrupted |
runner_interrupted |
unknown_failure |
| canceled | canceled |
unknown_failure: GitLab keeps the job canceled |
|
not placed within place_within_secs |
no_capacity |
runner_system_failure |
|
| node lost | lost |
runner_system_failure |
"Older GitLab" applies when the job's features.failure_reasons lacks the reason; the
fallback follows gitlab-runner's common/failure_reason_mapper.go
(job::gitlab_failure_reason). GitLab's retry: rules then decide whether the job runs
again, on whichever runner asks.
A node is lost for a job once it has been unreachable for job_lost_after (default 5
minutes) past the hub's unreachable threshold. The hub ends the job lost; the daemon
reports it; when the node comes back, its held names the job and the hub cancels it
immediate, and its later uploads fail on a token GitLab has invalidated. A node out of
touch for less keeps running the job and buffering its output, which the trace catches up
on.
A hub restart keeps job records, outputs and offsets, and loses specs and reservations:
a job not yet accepted by a node ends lost, and a reservation the hub no longer knows is
released when its node reports it. A daemon restart loses nothing it had written to its
state file.
Artifact uploads go from the node to GitLab, with the job token, as gitlab-runner's
artifacts-uploader does from inside the build environment. Relaying them through the hub
and the daemon would put every byte through two more hops, through a session that carries
control and telemetry only and whose messages are capped at 1 MiB, for no gain in trust: the
node already holds the token that authorizes the upload. The cost is that the node, not the
daemon, sees GitLab's answer, so it reports each upload's outcome in the result and in the
trace. The node follows gitlab-runner's upload rules: POST /jobs/:id/artifacts multipart
with the archive in field file, query artifact_format, artifact_type and expire_in,
a 307's Location followed, 413 not retried, 503 retried after Retry-After. Unlike
gitlab-runner, it refuses a 307 from https to http and sends the job token after a 307 only
to GitLab's own origin.
Dependency artifacts come from GitLab to the node: GET /jobs/<dependency id>/artifacts
with that dependency's token.
Caches go to the registry the node already uses. A cache is an OCI artifact with one
tar+zstd layer, in repository ci-cache/<gitlab host>[/<gitlab path>]/<project id>,
tagged with the sha256 of its key and the protection of the job's ref, so a protected and an
unprotected job never share one, as with gitlab-runner. The last archive written wins; content addressing
makes an unchanged cache's upload a blob probe. Each request to the registry is retried as vk's
other registry transfers are — a lost or refused connection for up to 90 seconds, a 503 or 429
for up to three minutes — so a registry's update does not lose a job its cache. A request that
runs into CACHE_REQUEST_TIMEOUT is not retried, and like a registry still unavailable after
its retries, leaves a warning and skips the step's other cache requests. Caches are not shared
with gitlab-runner's cache.zips: a project moving to the fleet starts cold.
A job with a registry token. When the hub mints a token for the job's registry
(step 4), the job's start carries it,
with the repository its ci-cache namespace resolves to. The node then keeps the job's
caches in exactly that repository, an absolute path on the registry beside the node's own
[registry] repo prefix, with the job's token, each tagged with the sha256 of its key alone:
the repository names the protection. It skips the caches, with a warning, when the
repository is not a repository name or the token has expired. A job whose start names no
cache repository keeps today's caches with the node's credential, except under [registry] guest_credential = "job", where it runs without caches, with a warning in its trace,
rather than fall back to the node's credential. The token is masked in the job's trace, and
dropped from its journaled start once its result is written. The node reports its
[registry] base URL in its inventory, which the hub matches against its registries to
mint the token for it.
Under guest_credential = "job" the guest's registry proxy (registry.vk) runs for every
job and injects the job's token, never the node's credential. A job with no token gets a
proxy that refuses everything; so does every job a local gitlab-runner gives the node, which
under "job" reaches the registry only by its own means.
With a token, the proxy forwards the lock API (/lock/) and plain files (/dav/files/) as
well as /v2/; the registry enforces the token's grants. Under /dav/files/ only GET,
HEAD, PROPFIND, PUT, MKCOL, DELETE and OPTIONS pass: MOVE and COPY would name the proxy in
their Destination. The job's steps find the proxy in VK_REGISTRY (http://registry.vk)
and their grants in VK_REGISTRY_NAMESPACES, space-separated <prefix>=<rights> with r,
w and l for read, write and lock. The proxy needs net.mode = "switch", the only network
that serves registry.vk; under any other, a job gets neither the proxy nor these variables.
Archiving and extracting run in the guest, in vk-agent, on the job's tree as the job left
it, with the bytes streamed over vsock; the node does the network transfers. No archiver
needs to be in the job's image. On the node, an archive is converted as it streams; only
what a format needs whole touches the job's disk — a downloaded zip, a cache layer, an
artifact's archive before its upload — each capped at 10 GiB, so a job cannot fill the disk
other jobs share. A request to the cache's registry is bounded by CACHE_REQUEST_TIMEOUT
(minutes, 10 by default), as gitlab-runner bounds its cache transfers.
JobSpec is tagged by kind; gitlab_ci is the only kind. vk-gitlab translates
GitLab's job response into a CiJob, normalizing what the node would otherwise interpret:
defaults are filled in, cache keys are expanded and sanitized (cache/cachekey), aliases are
split, empty formats become zip. What only concerns the conversation with GitLab stays with
the daemon.
Every field of the response, from gitlab-runner 19.5's common/spec/spec.go:
| GitLab field | In the spec | Notes |
|---|---|---|
id |
job.id |
|
token |
token |
|
allow_git_fetch |
sources.allow_fetch |
|
job_info.name, .stage, .pipeline_id, .project_id, .project_name, .project_full_path, .namespace_id, .root_namespace_id, .user_id |
job.* |
project_full_path as project_path |
job_info.organization_id, .instance_id, .instance_uuid, .scoped_user_id |
— | ignored: already among the job's variables where the job needs them |
job_info.time_in_queue_seconds, .project_jobs_running_on_instance_runners_count, .queue_size, .queue_depth |
— | daemon only: queue metrics |
git_info.repo_url |
sources.repo_url |
credentials stripped; the node adds the job token through a credential helper |
git_info.repo_object_format, .ref, .sha, .before_sha, .ref_type, .refspecs, .depth, .protected |
sources.* |
|
runner_info.timeout |
timeout_secs |
|
runner_info.uuid |
— | daemon only: logs |
inputs |
— | job_inputs is not advertised |
variables[] key, value, public, file, masked, raw |
variables |
the daemon appends the runner's own (CI_RUNNER_VERSION, …); the node adds the environment's (CI_BUILDS_DIR, CI_PROJECT_DIR, CI_JOB_STATUS, CI_SERVER_TLS_CA_FILE, …) |
steps[] name, script, timeout, when, allow_failure |
steps |
|
image name, alias, command, entrypoint, ports, variables, pull_policy |
image |
resolved under the node's image rules |
image.executor_opts.docker.platform, .docker.user, .kubernetes.user |
image.platform, image.user |
image_executor_opts is not advertised; when GitLab sends them anyway, these are honoured |
services[] |
services |
as image; run as a compose group beside the job VM, on one node |
artifacts[] name, untracked, paths, exclude, when, artifact_type, artifact_format, expire_in |
artifacts |
|
cache[] key, untracked, policy, paths, when, fallback_keys |
caches |
keys expanded against the job's and the runner's variables, as gitlab-runner expands them; variables only the node sets (CI_PROJECT_DIR, CI_CONCURRENT_ID, …) expand to nothing |
credentials[] of type registry |
registry_credentials |
other types are ignored, as by gitlab-runner |
dependencies[] id, token, name, artifacts_file |
dependencies |
|
features.trace_sections, .token_mask_prefixes |
trace |
trace.limit_bytes is the runner's output_limit, 4 MiB by default |
features.failure_reasons |
— | daemon only: failure mapping |
features.tracing |
— | ignored: no OpenTelemetry export |
secrets |
— | vault_secrets is not advertised; a job with secrets fails runner_configuration_error |
hooks[] |
hooks |
pre_get_sources_script and post_get_sources_script, run in the guest around the checkout |
run |
— | native_steps_integration is not advertised; a job with run fails runner_configuration_error |
policy_options |
— | the daemon writes gitlab-runner's "Job triggered by policy" line to the trace |
suspend_options |
— | ignored: no suspended environments |
the runner's tls_ca_file |
server_ca_pem |
its contents, when set, for CI_SERVER_TLS_CA_FILE; without it CI_SERVER_TLS_CA_FILE is not set |
server_url is the runner's configured URL; job.runner_id the ID /runners/verify
returned.
The node runs gitlab-runner's stage order (common/build.go): prepare_executor (admission,
image, boot, services), prepare_script, get_sources, restore_cache,
download_artifacts, each step_<name>, after_script, archive_cache or
archive_cache_on_failure, upload_artifacts_on_success or _on_failure, then cleanup.
get_sources uses the host-side checkout the executor already has, honouring GIT_STRATEGY,
GIT_DEPTH, GIT_SUBMODULE_STRATEGY, GIT_CHECKOUT and GIT_CLEAN_FLAGS; the other stages
reuse the executor's VM, exec and cleanup code. Step scripts are generated as gitlab-runner's
bash shell generates them (shells/abstract.go, shells/bash.go), and output is masked and
timestamped as its common/buildlogger masks and stamps it: each stream on its own — the
node's lines and vk gitlab prepare as stream 00, vk gitlab run as 01, stdout O and
stderr E apart — masked before it is stamped, the stamps counted in the trace limit.
FF_TIMESTAMPS is read from the job's variables only, on by default as in gitlab-runner 19.5;
a runner's [runners.feature_flags] has no equivalent here. Those ports keep gitlab-runner's
MIT notice.
Ported from gitlab-runner v19.5.0 (MIT; the notice is in NOTICE), as tests of the
vk-gitlab crate unless noted:
| Upstream | Ported as |
|---|---|
network/gitlab_test.go getRequestJobResponse |
a JobResponse JSON fixture; parse it and translate it to a CiJob, checking every field the table above maps |
common/support.go GetSuccessfulBuild, GetRemoteSuccessfulMultistepBuild, GetFailedBuild, getBuildResponse, getStepsBuildResponse |
JobResponse fixtures for multi-step, failing and run jobs |
common/network_test.go Test_Image_ExecutorOptions_UnmarshalJSON, TestFeaturesInfo_JSONMarshaling |
executor options and the info.features body |
network/gitlab_test.go TestGitLabClient_RequestJob, TestSetLastUpdate, …_TransmitsTwoPhaseJobCommit, TestVerifyRunner, TestTokenIsCreatedRunnerToken |
request, long-poll and verify against a fake GitLab |
TestUpdateJob, TestUpdateJobAsKeepAlive |
state updates and Job-Status handling |
TestPatchTrace, TestRangeMismatchPatchTrace, TestPatchTraceContentRangeAndLength, TestPatchTraceContentRangeHeaderValues, TestPatchTraceUrlParams, TestUpdateIntervalHeaderHandling, TestAbortedPatchTrace, TestJobFailedStatePatchTrace |
trace patching, offsets and intervals |
network/trace_test.go (TestJobFinishTraceUpdateRetry, TestJobDelayedTraceProcessingWithRejection, TestJobMaxTracePatchSize, TestCancelingJobIncrementalUpdate, TestJobChecksum, TestJobBytesize) |
the trace loop and final update |
checkTestArtifactsUploadHandlerContent, TestArtifactsUpload, TestArtifactsDownload |
upload and download expectations, in vk-driver's node job tests |
common/failure_reason_mapper_test.go |
vk-hub-proto job::tests (built) |
cache/cachekey tests |
key sanitizing, in vk-gitlab |
helpers/archives zip tests, helpers/trace masker tests |
archive and masking, in vk-driver |
- Uncommitted jobs. What GitLab does with a job taken under
two_phase_job_committhat is never committed — whether it returns it to the queue, and after how long — decides whether a job the fleet cannot place should be failed or left uncommitted. - Registry scopes. A cache write needs a token scoped to the project's cache repository;
vk-registrygrants per-account read and write, not per-repository scopes, so caches start with the node's own registry credential. - Secrets through the hub. The hub sees a spec's secrets in memory. Encrypting them to the reserved node's pinned key would keep them from the hub, at the cost of the hub no longer placing a job on another node than the one reserved.