reference-python-api.md

Reference: Python API

Everything the compiled extension exposes. Signatures are from vector-bindings/src/lib.rs, behaviour is what the bindings actually do when called.

Two import paths, same objects:

from vector_vrl import execute_vrl, validate_vrl, get_vrl_performance, Vector, VrlResult
from vector_vrl._bindings import execute_vrl   # equivalent

vector-vrl/__init__.py re-exports from ._bindings, falls back to a top-level vector_bindings module, and failing both installs stubs that raise ImportError on call. Check which you got:

import vector_vrl
vector_vrl.get_bindings_info()
# {'available': True, 'source': 'bundled', 'version': '1.0.5', 'bundled': True}

source is bundled, external or none.

Every call that compiles or runs VRL releases the GIL while the VRL works, so other Python threads keep running. Only building the returned dicts holds it. Several threads may share one Vector: each process_logs call adds its own counts to get_stats().

execute_vrl

execute_vrl(vrl_code: str, input_data: list[str], secrets: dict[str, str] | None = None) -> list[dict]

Compiles vrl_code once, then runs it over every string in input_data. Returns one dict per input, in order. An empty input list returns an empty list.

secrets seeds every event's secret store before the program runs - see Event secrets. Omit it and each event starts with none.

Raises ValueError if the VRL does not compile - that is a whole-batch failure, because compilation happens before any event is touched. Runtime errors are per-event and do not raise, see below.

>>> execute_vrl('.level = upcase!(.level)', ['{"level": "info"}'])
[{'level': 'INFO'}]

How input is parsed

If the string starts with {, it is parsed as JSON. Anything else - and any string that starts with { but fails to parse - is wrapped as {"message": <the raw string>}.

>>> execute_vrl('.seen = true', ['hello world'])
[{'message': 'hello world', 'seen': True}]
>>> execute_vrl('.seen = true', ['{not json'])
[{'message': '{not json', 'seen': True}]

There is no way to tell "this was plain text" from "this was JSON with a message field" in the output. If that distinction matters, tag it in your VRL.

What comes back

The event's top-level fields, flattened into the dict. The VRL program's own return value is discarded - what you get is the mutated event.

Scalars arrive as native Python types. Nested objects and arrays arrive as JSON STRINGS, not as dicts and lists:

>>> r = execute_vrl('.obj = {"a": 1}\n.arr = [1,2]\n.n = 3\n.f = 1.5\n.b = true\n.nul = null', ['{"x":1}'])
>>> r[0]
{'arr': '[1,2]', 'b': True, 'f': 1.5, 'n': 3, 'nul': None, 'obj': '{"a":1}', 'x': 1}

json.loads them yourself if you need the structure. Timestamps and regexes also come back as strings.

Per-event errors

A VRL runtime error replaces that event's dict entirely. You get error and original, and NOT the event's fields:

>>> execute_vrl('.p = parse_json!(.message)', ['{"message":"nope"}'])
[{'error': 'VRL error: function call error for "parse_json" at (5:26): unable to parse json: ...',
  'original': '{"message":"nope"}'}]

Note there is no success key here. Detect failure with "error" in result. Vector.process_logs returns exactly this shape too.

A float that is infinite (parse_float!("1e999")) has no JSON form, so that event comes back as an error naming the field (field `.x` is an infinite float) rather than as a number.

Two more shapes fail the event the same way rather than coming back wrong:

• A program that sets the whole event to something other than an object (. = [1, 2], . = "text") has no dict to return. The error names the type it was set to. Vector splits an event set to an array, as . = unnest(.messages) does, into several events, and one input here returns one dict, so that event fails too.
• An input integer above 9223372036854775807 (i64::MAX) does not fit VRL's integer. The error names the field instead of rounding it to a float. An integer past 18446744073709551615 is parsed as a float before VRL sees it, so that range still arrives rounded.

Event secrets

Vector attaches a secret store to every event, separate from the event's fields, and three VRL functions read and write it. All three are compiled in on every build:

VRL	Behaviour
get_secret(key)	The secret's value, or null when it is not set. Infallible - no ! needed
set_secret(key, secret)	Stores it, replacing any value already under that key. Returns null
remove_secret(key)	Drops it. Removing a key that is not set is not an error

These match Vector 0.58.0's own semantics. Secrets never appear in the event dict - they are a separate channel.

>>> execute_vrl('.k = get_secret("api_key")', ['{"x":1}'], {"api_key": "abc123"})
[{'k': 'abc123', 'x': 1}]

execute_vrl_with_secrets

execute_vrl_with_secrets(vrl_code: str, input_data: list[str], secrets: dict[str, str] | None = None) -> list[dict]

The same execution as execute_vrl - only the return shape differs. Each entry is a dict of two keys: event is exactly what execute_vrl would have returned for that input, and secrets is the event's secret store after the program ran.

>>> execute_vrl_with_secrets('set_secret("token", "s3cr3t")', ['{"x":1}'])
[{'event': {'x': 1}, 'secrets': {'token': 's3cr3t'}}]

Secrets are PER EVENT. Every input starts from the same secrets argument, so one event's set_secret is invisible to the next:

>>> execute_vrl_with_secrets(
...     'if .n == 1 { set_secret("leak", "yes") }\n.seen = get_secret("leak")',
...     ['{"n":1}', '{"n":2}'],
... )
[{'event': {'n': 1, 'seen': 'yes'}, 'secrets': {'leak': 'yes'}},
 {'event': {'n': 2, 'seen': None}, 'secrets': {}}]

An event whose program aborts still reports its secrets, as they stood when it failed - so secrets is populated even alongside an error event.

Vector.process_logs has no secrets surface: it seeds an empty store and discards whatever the program left behind.

validate_vrl

validate_vrl(vrl_code: str) -> VrlResult

Compiles without running. Never raises for bad VRL - the failure is in the returned object. Use it to lint VRL before you ship it.

>>> validate_vrl('.level = upcase!(.level)').success
True
>>> r = validate_vrl('.a = ')
>>> r.success, r.error, r.error_type
(False, 'syntax error', 'compilation_error')

VrlResult

Four read-only attributes:

Attribute	Type	On success	On failure
success	bool	True	False
output	str | None	"VRL syntax valid"	None
error	str | None	None	compiler message
error_type	str | None	None	"compilation_error"

repr() is Rust's derived Debug format, not Python-native - lowercase true/false and an Option wrapping (Some("...") or None), not a bare Python string:

>>> repr(validate_vrl('.level = upcase!(.level)'))
'VrlResult(success=true, output=Some("VRL syntax valid"), error=None)'

Only validate_vrl returns one - it is not the return type of execute_vrl.

get_vrl_performance

get_vrl_performance(vrl_code: str, test_data: list[str], iterations: int | None = None) -> dict

Cycles test_data iterations times and times the lot. iterations defaults to 100.

>>> m = get_vrl_performance('.a = 1', ['{"x":1}'], iterations=10)
>>> sorted(m)
['events_per_second', 'processing_time_seconds', 'thg_score', 'total_events']

total_events is len(test_data) * iterations. thg_score is events_per_second clamped to a 1000 ceiling - it is not an independent measurement, so two runs above 1000 eps score identically.

len(test_data) * iterations is capped at 1,000,000. Over that, ValueError. Both inputs are caller-controlled, which is why the cap exists. Events run one at a time, so memory stays at one event however many iterations are asked for, and the timing covers compiling and running the VRL but not building the Python result dicts.

This measures the in-process runtime and nothing else. It does not compare against the vector binary.

Enrichment tables

Vector's two enrichment functions, get_enrichment_table_record and find_enrichment_table_records, are compiled in on every build. VRL names a table by a literal key, never a path. The path is handed over from Python.

register_enrichment_table

register_enrichment_table(name: str, kind: str, path: str, delimiter: str | None = None) -> None

Loads the file at path and registers it under name. kind is file, a CSV whose first row is the header, or geoip, a GeoIP2/GeoLite2 .mmdb. delimiter applies to file only, defaults to ,, and must be one ASCII character. Registering a name again replaces that table.

The file is read here, so a bad table raises ValueError now rather than at VRL runtime: an unknown kind, an empty name, a delimiter longer than one character, an unreadable file, or a CSV row with the wrong number of fields.

>>> register_enrichment_table("users", "file", "users.csv")
>>> execute_vrl('. = get_enrichment_table_record!("users", {"id": .id})', ['{"id": 2}'])
[{'id': '2', 'name': 'Fred', 'team': 'blue'}]

Register before compiling the VRL that uses the table. The name is resolved at compile time, so an unregistered one fails validate_vrl and raises from execute_vrl, naming what is registered:

>>> validate_vrl('. = get_enrichment_table_record!("nope", {"id": "1"})').error
'function compilation error: error[E111] unknown enrichment table "nope"; registered tables: users'

Lookups

condition is an object of field and value pairs, and a row must equal every one of them. Only equality is supported. A file table stores every cell as text, so an integer, float or boolean condition is compared in its text form: {"id": 2} matches the cell 2. case_sensitive defaults to true, and select returns only the fields it names.

VRL	No row matches	More than one row matches
get_enrichment_table_record	error No rows found	error More than one row found
find_enrichment_table_records	[]	every match, in file order

>>> execute_vrl('.r = find_enrichment_table_records!("users", {"team": "RED"}, select: ["name"], case_sensitive: false)', ['{}'])
[{'r': '[{"name":"Bob"},{"name":"Alice"}]'}]

A condition field that is not a column fails the event at runtime, unless the table has no rows. Compiling a lookup indexes a file table on its condition fields, so a lookup costs the same however many rows the table holds.

A geoip table takes exactly one condition, whose value is the IP address. An address with no record in the database is a no-row match.

clear_enrichment_tables and list_enrichment_tables

clear_enrichment_tables() -> None
list_enrichment_tables() -> list[dict]

clear_enrichment_tables empties the registry. Programs already compiled keep the tables they captured, and only later compilations see the empty registry.

list_enrichment_tables returns one dict per table, name-ordered, with name, kind, path and rows. rows is the row count of a file table and None for a geoip table.

>>> list_enrichment_tables()
[{'name': 'users', 'kind': 'file', 'path': 'users.csv', 'rows': 3}]

Vector

Vector(config: dict)
    .initialize() -> bool
    .process_logs(logs: list[str], vrl_code: str) -> list[dict]
    .get_stats() -> dict

A batch VRL runner. Read the constraints below before you build anything on it.

pipeline = Vector({})
pipeline.initialize()
results = pipeline.process_logs(['{"level":"info"}'], '.level = upcase!(.level)')

config is round-tripped through Python's json module, so anything json.dumps handles is accepted - nested dicts, lists, True, None, floats, apostrophes. A value json cannot serialise raises TypeError or ValueError at construction.

config is then STORED AND NEVER USED. It does not affect initialize or process_logs. There is no sources/transforms/sinks execution - you can hand it a full Vector pipeline config and the only thing that runs is the vrl_code argument you pass to process_logs. That argument is required and is not derived from the config.

initialize() sets a flag and returns True. Calling process_logs first raises RuntimeError("Vector not initialized").

process_logs return shape

Identical to execute_vrl - both go through the same conversion in the crate. One dict per log, the event's fields flattened in:

{'level': 'INFO'}

On a per-event runtime error, error and original replace the event's fields:

{'error': 'VRL error: ...', 'original': '<the input string>'}

So "error" in result is the failure check, the same as execute_vrl. Uncompilable VRL raises ValueError("VRL compilation failed: ...") for the whole batch, also the same as execute_vrl.

get_stats

{'events_processed': 2, 'bytes_processed': 51, 'errors': 1, 'uptime_seconds': 0.31}

Real accumulated counts, measured from the most recent initialize() - calling initialize() again restarts the uptime clock and zeroes the counters, so the numbers always describe one run.

Key	Counts
events_processed	events process_logs transformed without a runtime error
errors	events that hit a VRL runtime error
bytes_processed	UTF-8 bytes of every input string handed to process_logs, failed events included
uptime_seconds	seconds since initialize(); 0.0 before it is called

Every event lands in exactly one of events_processed or errors, so the two sum to the number of events attempted. VRL that fails to COMPILE raises before any event is touched and moves no counter.

config is still stored and never used - see the constraints above.

validate_config

validate_config(source: dict | str | Path) -> ConfigCheck

Compiles the source of every remap transform in a Vector config. Runs in-process against the real compiler, and needs no vector binary.

source is either an already-parsed dict, or a path to a .yaml/.yml/ .toml/.json config. TOML and JSON parse with the stdlib; YAML needs pip install vector-vrl[yaml] and raises ModuleNotFoundError naming that extra otherwise. An unsupported suffix, or a file that does not parse to a mapping, raises ValueError.

>>> r = validate_config({"transforms": {"t": {"type": "remap", "source": ".a = 1"}}})
>>> r.ok
True

ConfigCheck carries:

Attribute	Meaning
ok	every checked transform compiled
checked	one TransformCheck per remap with inline source
failures	the checked ones that did not compile
unchecked	the checked ones this build could not judge
skipped	remap transform names whose VRL lives in an external file:

TransformCheck is name, ok, error, unchecked_reason.

Transforms that are not type: remap are ignored entirely. A remap reading its VRL from file: is listed in skipped rather than guessed at.

Why some transforms come back unchecked

Two groups of VRL this build cannot judge on its own:

• get_enrichment_table_record and find_enrichment_table_records DO compile here, but only against a table registered through register_enrichment_table. Vector declares its tables in enrichment_tables: outside VRL, so a config naming a table this process was never given is unchecked - register it to have it checked.
• The ten functions the sandboxed default build leaves out (get_env_var, get_hostname, http_request, dns_lookup and six more - see VRL restrictions below) report call to undefined function `get_hostname` .

VRL calling either group is not a broken config, so those transforms come back with ok True and an unchecked_reason naming the function, never in failures. That holds only while those are the source's ONLY compile errors. One more error of any kind, a typo'd function name included, puts the transform in failures. Use validate_config_with_vector to check the unchecked ones for real.

The event-secret functions used to be in this list. They are compiled in now, so a config using get_secret, set_secret or remove_secret is checked like any other - the secret BACKEND is still declared outside VRL and still invisible here.

validate_config_with_vector

validate_config_with_vector(
    path: str | Path,
    *,
    vector_binary: str = "vector",
    no_environment: bool = True,
    deny_warnings: bool = False,
    timeout: float = 60.0,
) -> VectorValidation

Runs vector validate on a config file. Checks everything validate_config cannot - sources, sinks, wiring between components, component options, and the VRL whose enrichment tables or secrets are declared in the config.

One-shot: Vector exits as soon as it has answered. This never starts a daemon and never moves data. no_environment passes --no-environment, skipping the component and health checks that would open network connections, so checking a file does not dial the configured sinks.

Raises FileNotFoundError when the binary is not on PATH, and subprocess.TimeoutExpired if Vector outlives timeout.

VectorValidation is ok, returncode, and output - Vector's own report, which names the offending component on failure.

>>> r = validate_config_with_vector("vector.yaml")
>>> r.ok or r.output
True

VRL restrictions

Two guards apply to every entry point that compiles VRL.

Environment, system and network functions are not compiled in, so caller-supplied VRL cannot read the host or reach the network. Ten of them fail to compile with undefined function: get_env_var, encode_proto, parse_proto, parse_etld, validate_json_schema, get_hostname, get_timezone_name, http_request, dns_lookup and reverse_dns. That is what the published wheel ships; a source build can turn all ten on with the full-stdlib Cargo feature. The reasoning is in architecture.md under "VRL execution's security posture".

VRL source may not nest brackets more than 64 levels deep (MAX_VRL_NESTING_DEPTH). Past a few hundred the parser's own recursion overflows the stack and takes the process down, which no Python except can catch, so the check runs before the parser sees the input. Brackets inside string, raw string, regex and timestamp literals and # comments do not count. Over the limit you get a message containing nests:

>>> validate_vrl('.x = ' + '(' * 1000 + 'true' + ')' * 1000).error
'VRL source nests 1000 levels deep, exceeding the 64-level limit'

Chains that nest no bracket (a + b + c ..., long else if ladders, !!!x) recurse just as deep, so compiling and running VRL happens on a thread with a 256 MiB stack of its own, whatever stack the calling thread has. If the process cannot reserve that much address space the work runs on the calling thread instead. A far longer chain still overflows it.

Both bracket and stack limits are covered by tests in vector-bindings/src/lib.rs and vector-vrl/tests/unit/test_vector_class.py.

The subprocess surface

vector_vrl.__all__ also carries THGPerformanceAssessor, THGMetrics, THGResult, quick_thg_assessment, assess_vrl_performance, execute_vector_pipeline, VectorTestRunner, ProductionPatterns, production_patterns, and the get_apache_combined / get_nginx_access / get_json_application / get_kubernetes_pods / get_docker_container pattern helpers.

These are a different thing to everything above. They shell out to a vector BINARY - the first vector on PATH, else /usr/bin/vector, or the path you pass - and VectorTestRunner raises RuntimeError at construction if no binary exists there. They do not use the compiled bindings.

VectorTestRunner feeds the logs to Vector on stdin and returns when Vector exits, so a run takes as long as the work. Its timeout (60 s by default) kills a run that goes past it and reports it failed. A run succeeds when Vector exits 0 and every output line is JSON. Events a program aborts do not come back, and Vector's stderr is in the error message either way.

execute_vector_pipeline(config, input_data, timeout) runs the config's own remap transforms, in the order their inputs chain them and each with its own options, over input_data. The config's sources and sinks are not used. A config whose transforms are not one chain of inline remaps raises ValueError.

They are not part of the in-process API this package is for, and they are not covered by the test files that describe it. If you want VRL executed, use execute_vrl.