Prometheus Engine

This repository contains various binaries and packages for client-side usage of Google Cloud Managed Service for Prometheus (GMP), a managed Prometheus offering on top of Google Cloud Monitoring (GCM).

For more documentation and to get started, go to g.co/cloud/managedprometheus.

Binaries

• config-reloader: An auxiliary sidecar binary to initiate reloads on configuration file changes.
• frontend: An authorizing proxy for the Prometheus-compatible query API of GMP. It additionally hosts a query UI.
• operator: A Kubernetes operator for managed metric collection and rule evaluation for GMP.
• rule-evaluator: A Prometheus rule evaluation engine that evaluates alerting and recording rules against GMP.
• datasource-syncer: A periodic job that injects short-lived OAuth2 access tokens into Grafana Prometheus data sources.
• gmp-migrate: A standalone CLI tool to convert Prometheus Operator (PodMonitor, ServiceMonitor) resources to GMP Custom Resources.

For the fully Prometheus-compatible binary that writes ingested data into GMP/GCM, see GoogleCloudPlatform/prometheus.

Docker Images

Container images for this repository (except gmp-migrate, which is a standalone CLI binary) are published to Artifact Registry under us-docker.pkg.dev/gke-release/gke-release/prometheus-engine/<binary>.

Deploying Self-Managed Manifests & Custom Resources

On GKE, Managed Service for Prometheus can be enabled directly via GKE cluster flags. For self-managed deployments or non-GKE Kubernetes clusters, apply the pre-rendered manifests in manifests/ in the following order:

# 1. Create namespaces (gmp-system, gmp-public), CRDs, and cluster RBAC:
kubectl apply -f manifests/setup.yaml

# 2. Deploy gmp-operator, admission webhooks, and default OperatorConfig:
kubectl apply -f manifests/operator.yaml

# 3. (Optional) Deploy standalone rule-evaluator:
kubectl apply -f manifests/rule-evaluator.yaml

Custom Resource Definitions (CRDs)

All CRDs live under API group monitoring.googleapis.com/v1. For the full field-level schema, see the generated API Reference (doc/api.md) and the sample manifests in examples/.

Kind	Scope	Allowed Namespace	Purpose
PodMonitoring	Namespaced	Any user namespace	Scrapes Pod endpoints in the same namespace; injects project_id, location, cluster, namespace, job, and instance.
ClusterPodMonitoring	Cluster	Cluster-scoped	Scrapes Pod endpoints across all namespaces; injects namespace from the scraped Pod.
ClusterNodeMonitoring	Cluster	Cluster-scoped	Scrapes node-level endpoints (such as kubelet, cAdvisor, or node-exporter).
Rules	Namespaced	Any user namespace	Evaluates Prometheus alerting and recording rules scoped to the resource's namespace (plus project_id, location, cluster).
ClusterRules	Cluster	Cluster-scoped	Evaluates alerting and recording rules across all namespaces in the cluster (scoped to project_id, location, cluster).
GlobalRules	Cluster	Cluster-scoped	Evaluates alerting and recording rules across the entire GCM metrics scope without injecting cluster or namespace selectors.
OperatorConfig	Namespaced	gmp-public (named config)	Configures cluster-wide collector credentials, kubelet scraping, rule-evaluator credentials, Alertmanager endpoints, and export filters.

Development

See AGENTS.md for detailed contribution guidelines, commit message formatting (make conform), linter rules (make lint), and code-generation workflows (make regen).

Repository Layout & Go Modules

Path	Go Module	Description
cmd/	go.mod	Main entrypoints and Dockerfiles for the 6 binaries.
pkg/, internal/	go.mod	Reconcilers, CRD API types (pkg/operator/apis/monitoring/v1), GCM export client, migration library, and UI asset loader (pkg/ui).
charts/ → manifests/	—	Internal Helm templates (charts/) that render static deployment manifests (manifests/) via make regen.
doc/api.md	—	Autogenerated CRD API reference documentation (do not edit by hand; updated via make regen).
e2e/	go.mod	End-to-end Kubernetes test suite executed against kind or GKE clusters.
examples/	go.mod & examples/scripts/go.mod	Reference CRD manifests, feature guides, synthetic Go workload (examples/instrumentation/go-synthetic), and exporter configurations.
ops/	ops/gmpctl/go.mod	Multi-branch release and vulnerability automation CLI (ops/gmpctl).
third_party/prometheus_ui/	—	Vendored upstream Prometheus React UI (base/) and GMP overrides (override/).
tools/	tools/go.mod	Pinned build, codegen, and linting tool dependencies (controller-gen, helm, yq, mdox, shfmt, addlicense).

Dependencies

In order to best develop and contribute to this repository, the following dependencies are recommended:

1. go (matching the version in go.mod)
2. gcloud
3. kubectl

Can also be installed via:

gcloud components install kubectl

4. Docker with buildx plugin.

If you want to execute docker containers on a remote machine you can run:

gcloud alpha cloud-shell ssh --authorize-session -- -nNT -L `pwd`/docker.sock:/var/run/docker.sock

# Then in separate terminal.
export DOCKER_HOST=unix://docker.sock

5. For UI development or updates (e.g. to resolve UI security issues), npm is required. See pkg/ui documentation for details.

Presubmit, Code Generation & Linting

• make regen (or ./hack/presubmit.sh): Regenerates CRD YAMLs (charts/operator/crds/), Helm manifests (manifests/), API docs (doc/api.md), and runs formatting (./hack/presubmit.sh format).
• CHECK=1 make regen: Verifies that no uncommitted diffs or missing license headers remain after regeneration (enforced in CI).
• ./hack/presubmit.sh codegen: Regenerates Kubernetes DeepCopy methods (zz_generated.deepcopy.go) when Go API types under pkg/operator/apis/monitoring/v1/ change.
• make lint: Runs golangci-lint across the repository.
• make conform: Validates git commit messages against .conform.yaml.
• make presubmit (or CHECK=1 make presubmit): Regenerates all resources, builds all images, and runs unit tests in Docker.

Building

Running make help shows a list of candidate targets with documentation.

Any Go application in ./cmd/ with an associated main.go (e.g. ./cmd/operator/main.go) can be built via make:

make operator
make frontend
make rule-evaluator
make config-reloader
make datasource-syncer
NO_DOCKER=1 make gmp-migrate

This also includes example Go apps from /examples/instrumentation/:

make go-synthetic

Running make bin will build the component Go binaries into ./build/bin/.

• Setting NO_DOCKER=1 (NO_DOCKER=1 make bin) builds the binaries natively on the host machine without Docker.

Testing

Unit

To run unit tests natively on the host machine, use your IDE, or:

NO_DOCKER=1 make test

To run unit tests inside the hermetic Docker container, run make test.

Kubernetes End-to-end tests

Running make e2e builds the container images and runs end-to-end tests against a Kubernetes cluster:

• By default, it runs in a hermetic Docker container, downloads kind, creates a single-node kind cluster, and runs e2e tests against it.
• Each Go test from e2e runs in a separate kind cluster. The name of the test determines the cluster name, so TestXYZ is executed in a XYZ-<short unique sha> cluster.
• To run a single test (or re-run without rebuilding Docker images via make e2e-only), use the TEST_RUN environment variable:

TEST_RUN=TestCollectorPodMonitoring make e2e
# Or re-run against already-built images:
TEST_RUN=TestCollectorPodMonitoring make e2e-only

• For easier debugging you can keep the kind cluster running after test failures using the KIND_PERSIST variable:

KIND_PERSIST=1 TEST_RUN=TestCollectorPodMonitoring make e2e

GCM integration

Some tests verify GCM state. For this you need a GCP service account with read and write permissions against GCM in a project. Here is how to configure it:

• If you have a GCP service account key in a local file, use the GOOGLE_APPLICATION_CREDENTIALS environment variable to specify the path to that file:

GOOGLE_APPLICATION_CREDENTIALS=<path to SA> TEST_RUN=TestCollectorPodMonitoring make e2e

• If you have the content of a GCP service account key in an environment variable (e.g. CI), use the GCM_SECRET variable:

GCM_SECRET=<...> TEST_RUN=TestCollectorPodMonitoring make e2e

NOTE: This is what CI uses. NOTE2: This utilizes an explicit credentials path via OperatorConfig, while GOOGLE_APPLICATION_CREDENTIALS uses default credentials mode.

• If you want to skip GCM tests, ensure those two variables are empty:

GCM_SECRET="" GOOGLE_APPLICATION_CREDENTIALS="" TEST_RUN=TestCollectorPodMonitoring make e2e

Debugging

In Docker mode, after failures with KIND_PERSIST and during tests, you can inspect the local kind cluster for further investigation. The easiest way is to use the same Docker image and setup that is used in make e2e, by starting a new container using the gmp/kindtest image on the same Docker socket and network.

You can use make e2e-exec to start an interactive shell:

make e2e-exec

In the interactive shell, you can inspect existing tests/clusters or spin up a new test.

For example, assuming you are running (or ran with KIND_PERSIST=1) make e2e, you can access the Kubernetes cluster (e.g. to list pods):

kind get clusters
kind export kubeconfig -n <TestName-hash from the output above>
kubectl -n gmp-system get po

You can also execute a single test (e.g. TestCollectorPodMonitoring) by reusing an already started kind cluster or creating your own:

kind export kubeconfig -n "existing-cluster"
go test -v ./e2e -run "TestCollectorPodMonitoring" -args -project-id=test-proj -cluster="existing-cluster" -location=test-loc -skip-gcm
kubectl -n gmp-system get po

However, it might be easier to run KIND_PERSIST=1 TEST_RUN=TestCollector make e2e in a separate terminal and play with custom time.Sleep statements for more detailed investigations.

Benchmarking

See BENCHMARK.md.