Production build cache: Docker Compose, nginx and TLS
+
A Compose file for a shared build cache that only answers over HTTPS, with a password, behind nginx. Everything here was run for real on FosterStack Cache 0.2.1 and Gradle 9.8.0. The exact list of what was and was not tested is just below.
+
+
What was tested, and what was not
+
+
Tested: the two files below, with Docker Compose on a Mac, nginx 1.29 as the proxy, a self-signed certificate for localhost, and a real Gradle build over HTTPS.
+
Not tested: a certificate from a public authority or automatic renewal, a public server on port 443, other proxies or load balancers, Kubernetes ingress, and long-running use.
+
+
+
compose.yaml
+
services:
+ fscache:
+ image: ghcr.io/fosterstack/cache:0.2.1
+ restart: unless-stopped
+ # no "ports:" on purpose: only the proxy is reachable from outside
+ volumes:
+ - fscache-data:/home/nonroot
+ environment:
+ FSCACHE_MAX_BYTES: "53687091200" # 50 GiB
+ FSCACHE_USERNAME: gradle
+ FSCACHE_PASSWORD: change-me # generate one: openssl rand -base64 24
+
+ proxy:
+ image: nginx:1.29-alpine
+ restart: unless-stopped
+ depends_on:
+ - fscache
+ ports:
+ - "127.0.0.1:8443:443" # use 443:443 on a real server
+ volumes:
+ - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
+ - ./certs:/etc/nginx/certs:ro
+
+volumes:
+ fscache-data:
+
The cache has no ports entry on purpose. Only the proxy is reachable from outside. Generate a real password, for example with openssl rand -base64 24, and never keep change-me.
+
+
nginx.conf
+
server {
+ listen 443 ssl;
+ server_name localhost;
+ ssl_certificate /etc/nginx/certs/cert.pem;
+ ssl_certificate_key /etc/nginx/certs/key.pem;
+
+ # the cache accepts entries up to 1 GiB by default; nginx's own default is 1 MB
+ client_max_body_size 1g;
+
+ location / {
+ proxy_pass http://fscache:8080;
+ proxy_set_header Host $host;
+ }
+}
+
The client_max_body_size line matters. See the tests below.
+
+
What I checked, and what I saw
+
+
Check
Result
+
https://localhost:8443/healthz with the certificate trusted
ok
+
The same without trusting the certificate
curl refuses (exit 60)
+
The cache's own port 8080 from the host
not reachable
+
Plain http:// to the proxy's TLS port
400
+
Upload with no password, through the proxy
401
+
Upload with the password, through the proxy
201, and the value reads back
+
Uploads of about 1 MB and 2 MB with nginx's default settings
413 from nginx, not from the cache
+
The same uploads, and 20 MB, after client_max_body_size 1g;
201 each
+
+
The size limit is the one to watch. nginx rejects bodies over 1 MB by default, and Gradle's cache entries are often bigger, so uploads of larger entries would be refused. The cache itself accepts up to 1 GiB per entry by default, which is why the example uses 1g. See the status codes page.
+
+
Gradle over HTTPS
+
With a certificate Gradle does not trust, the build still succeeds, but without the cache, and prints:
+
Could not load entry d4f2de693b561fd2c5d60b17a9232d6f from remote build cache: (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
+
For my self-signed test certificate I made Java trust it:
With that, the first build ran :compileJava and the second printed > Task :compileJava FROM-CACHE, through the proxy, over HTTPS, with the password. With a certificate from an authority Java already trusts you would not need the truststore lines. The Gradle side of the setup is in the tutorial.
A short checklist for moving builds from the Develocity Build Cache Node to FosterStack Cache, with a real run to back it: the same project built against an old node, then against FosterStack Cache 0.2.1, then back. Read why the node is going away in Build Cache Node end of life.
+
+
What I tested
+
Gradle 9.8.0 and a small Java project, built twice against each server. The old side was a Build Cache Node image (version 21.2, built June 2025), run locally and set up to allow anonymous reads and writes for the test. The new side was a FosterStack Cache 0.2.1 server with no password. I did not test Maven in this run; for Maven see the setup page.
+
+
The checklist
+
1. Find every place your builds point at the old node
+
Search your repositories and CI settings for HttpBuildCache and for the node's address. The old node's address ends in /cache/. Write the list down: each entry is one line to change.
+
+
2. Start the new server
+
Follow your first 15 minutes. Turn on a password before other machines can reach it.
Nothing is copied over, and nothing needs to be: a cache holds results your builds can always rebuild. The first build on the new server runs everything and fills it; the second hits.
+
+
5. Decide about the /cache/ part of the URL
+
You can drop it, as above, or keep it. FosterStack Cache uses the whole request path as the key, so a prefix works. Entries under different paths are separate, so pick one and use it everywhere.
+
+
6. Keep the old node running until you are sure
+
With Gradle, rolling back is the same one-line change in reverse; I tested that.
+
+
7. Retire the old node, and check the new one
+
When builds are hitting the new server, stop the old node. Watch the new one with monitoring, and size its disk with server sizing.
+
+
What happened in the run
+
Each row is two builds of the same project with gradle clean in between and Gradle's own local cache turned off, so a hit can only come from the server:
+
+
Pointed at
First build
Second build
+
The old node, URL ending in /cache/
:compileJava ran
:compileJava FROM-CACHE
+
FosterStack Cache, only the URL changed (no /cache/)
:compileJava ran: the new server was empty
:compileJava FROM-CACHE
+
FosterStack Cache, URL kept with /cache/
:compileJava ran: separate entries
:compileJava FROM-CACHE
+
Back to the old node (rollback)
:compileJava FROM-CACHE: its entries were still there
not run
+
+
The new server's status page agreed: two entries stored, two hits, two misses. One small difference you may notice: a successful upload returns 200 from the old node and 201 from FosterStack Cache. Gradle accepted both.
+
+
+
+
diff --git a/build-cache-permission-denied-data-directory/index.html b/build-cache-permission-denied-data-directory/index.html
new file mode 100644
index 0000000..9723d3f
--- /dev/null
+++ b/build-cache-permission-denied-data-directory/index.html
@@ -0,0 +1,137 @@
+
+
+
+
+
+Build cache server exits: permission denied on its data
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
fosterstack
+
Infrastructure worth keeping — and kept patched.
+
+
+
Build cache server exits with "permission denied" on its data
+
You start the container and it stops straight away. The log ends with permission denied. The cause I reproduced is who owns the data directory. This page shows the failure and the fix, reproduced for real with the 0.2.1 image.
+
+
What you see
+
The container exits with status 1 within a few seconds, and its log says:
+
{"level":"ERROR","msg":"fscache: fatal","error":"write shutdown marker: open data/.unclean-shutdown: permission denied"}
+
+
Why it happens
+
The server does not run as root. It runs as user 65532, and it keeps its files in a data folder inside /home/nonroot. If that folder already exists and belongs to someone else, the server cannot write to it and stops.
+
I reproduced it with a volume that already held data owned by root, which is what you get after restoring a backup as root:
Give the whole tree to user 65532. The image has no shell, so use a throwaway container to do it:
+
docker run --rm -v fscache-data:/d ubuntu:24.04 chown -R 65532:65532 /d
+
After that the same image started normally: /healthz answered ok and an upload returned 201.
+
+
What does not cause it
+
+
A new, empty named volume. This is what our first-run steps create, and the volume then belongs to 65532. I also tried a new empty volume whose top folder I had set to root. The server still started, and afterwards the folder belonged to 65532, because Docker copies the image's ownership into an empty volume the first time it is used.
+
+
+
What I did not test
+
+
A bind mount of a host folder on Linux. The same rule should apply (the folder has to be writable by 65532), but I ran on a Mac, where bind mounts behave differently.
+
Kubernetes without the fsGroup setting. The manifests in our Kubernetes page set fsGroup: 65532, and I ran them with it; I did not run them without it.
+
Running the container as root to avoid the problem. It is not what we document.
Many companies make builds reach the network through a proxy. This page shows how Gradle's remote build cache behaves behind one: the settings, the ways it goes wrong, and how to tell a proxy problem from a cache problem. Everything was run for real with Gradle 9.8.0 and FosterStack Cache 0.2.1.
+
+
What was tested, and what was not
+
+
Tested: a small proxy I wrote for the test (it forwards plain HTTP, tunnels HTTPS with CONNECT, can refuse a host, and logs every request), the cache on a non-loopback address, once over plain HTTP and once over HTTPS through nginx with a self-signed certificate that Java was told to trust.
+
Not tested: a real corporate proxy product, a proxy that asks for a password (407), a proxy that inspects and re-signs TLS, automatic proxy configuration files, and the HTTP_PROXY variables. Maven is covered on its own page, where it behaves differently: for example it does not skip the proxy for loopback addresses.
+
+
+
The settings
+
Gradle takes proxy settings from Java system properties. Put them in ~/.gradle/gradle.properties or the project's gradle.properties, with the systemProp. prefix:
Use the https pair for an https:// cache URL and the http pair for plain HTTP. The last line lists hosts that should skip the proxy.
+
+
What I saw
+
For the plain-HTTP rows I set isAllowInsecureProtocol = true, which Gradle requires for non-loopback http://. That is for the test rig only; use HTTPS for real.
+
+
Setup
Result
What the proxy logged
+
No proxy settings, cache at <cache-ip>
second build FROM-CACHE
nothing (not used)
+
Proxy settings, same plain-HTTP URL
FROM-CACHE
GET http://<cache-ip>:18130/<key>, one per build
+
Proxy settings, but the cache URL is 127.0.0.1
FROM-CACHE
nothing: Java skips the proxy for loopback addresses
+
Proxy settings, with the cache's address in nonProxyHosts
FROM-CACHE
nothing
+
Proxy settings, HTTPS cache, tunnel allowed
FROM-CACHE
CONNECT <cache-ip>:18443, one per build
+
Proxy settings, proxy refuses the cache host
the build succeeds but without the cache (log below)
the request, answered 403
+
+
+
Three things that catch people
+
1. Testing against localhost skips the proxy. With the cache at 127.0.0.1, Gradle never used the proxy even though it was configured. A setup that works on your laptop against a local cache proves nothing about the proxy path. Test against the real address.
+
2. A refusing proxy looks like a cache error. When the proxy answered 403 to the tunnel, Gradle printed:
+
Could not load entry d4f2de693b561fd2c5d60b17a9232d6f from remote build cache: Loading entry from 'https://<cache-ip>:18443/d4f2de693b561fd2c5d60b17a9232d6f' response status 403: Forbidden
+
That is the same text you would get from the cache refusing a write with the read-only login (see the status codes). To tell them apart, take Gradle out of it and try the same route with curl:
+
$ curl -sS -x http://proxy.example.com:3128 https://cache.example.com/healthz
+curl: (56) CONNECT tunnel failed, response 403 # the proxy said no
+$ curl -sS --noproxy '*' https://cache.example.com/healthz
+ok # the cache is fine
+
I substituted example hostnames in those commands; the two output lines are the real ones, from my test proxy refusing and from the cache answering directly. If the first one fails and the second works, ask whoever runs the proxy to allow the cache's host and port.
+
3. Over HTTPS the proxy only sees the address. A CONNECT tunnel shows the proxy the host and port and nothing inside, so the password and the cache keys are not visible to it. That also means any rule on the path cannot be applied. This is how the proxy log looked in my run: one CONNECT line per build.
+
+
Checklist
+
+
Ask for the cache's host and port to be allowed through the proxy (HTTPS).
+
Put the proxy lines in gradle.properties, with the systemProp. prefix, and list internal hosts in nonProxyHosts.
Maven build cache extension behind a corporate proxy
+
Maven has its own proxy settings, and the Apache Build Cache Extension follows them. This page shows what the extension sent through a proxy, how it differs from Gradle, and what you see when the proxy says no. All of it was run for real with Maven 3.10.0, extension 1.2.3 and FosterStack Cache 0.2.1.
+
+
What was tested, and what was not
+
+
Tested: a small proxy I wrote for the test, which forwards plain HTTP, can refuse a host, and logs every request (I made it close each connection after one request so that none go unlogged). The cache was on a non-loopback address, over plain HTTP, with a password.
+
Not tested: HTTPS through a proxy for Maven, a proxy that asks for a password, other Maven or extension versions, and any real proxy product. Gradle is on its own page.
+
+
+
The settings
+
In ~/.m2/settings.xml, next to the cache's password entry:
The cache itself is set up as in the Maven setup. The server id must still match the remote's.
+
+
What went through the proxy
+
With the proxy configured, all of the extension's cache traffic went through it: the lookups, the download that restores a build, and the uploads. This is what the proxy logged for two builds, the first filling the cache and the second restoring from it (the proxy also saw Maven's own traffic to repository.apache.org, which is not shown):
The second build logged Found cached build, restoring demo:demo from cache.
+
+
Other cases
+
+
Setup
What happened
+
Proxy configured, cache URL is 127.0.0.1
Still went through the proxy (4 requests logged). Unlike Gradle, Maven does not skip the proxy for loopback.
+
Proxy configured, the cache's address in <nonProxyHosts>
Nothing reached the proxy. The build still restored from the cache.
+
Proxy refuses the cache's host
Nothing was stored or restored. The build succeeded without the cache, and logged Error downloading cache item with HTTP Status: 403.
+
Proxy not running at all
The lookup failed with Connect to 127.0.0.1:18131 ... Connection refused. The build succeeded without the cache, and nothing was uploaded.
+
+
+
Things that catch people
+
+
A local test can fail because of the proxy. If you try the cache on 127.0.0.1 with a proxy configured in settings.xml, Maven sends it to the proxy, which may not be able to reach your machine's loopback. List it in <nonProxyHosts>.
+
A refusing proxy looks like a cache error. The 403 comes from the proxy, not from the cache. To check, run the same request from the build machine with and without the proxy; see the curl check on the Gradle page.
+
Failure does not break the build. A build behind a blocking proxy just stops using the cache, so look for Error downloading cache item and for missing Saved to remote cache lines.
Upgrade a self-hosted build cache and keep its data
+
Can you move a build cache to a newer release without losing what it holds? I tested it with the only two published releases: I stored entries on 0.1.0, started 0.2.1 on the same data volume, wrote more, and then went back to 0.1.0. This is what happened.
+
+
What was tested, and what was not
+
+
Tested: Docker, one named volume at /home/nonroot, release 0.1.0 and then 0.2.1 (and back), no password, stopped and started cleanly between versions.
+
Not tested: any other pair of releases (there is no 0.2.0 release; see the changelog), upgrading while the server is running, Kubernetes or Compose rollouts, the FIPS image, or a cache that was close to its size cap.
Use the same volume and the same settings (environment variables for the password and size cap) as the old container. Pin the exact tag you want, as above, rather than latest.
+
+
What I saw
+
+
Step
Result
+
0.1.0: store three entries on a fresh volume
201 each; its metrics show 3 entries, 29 bytes
+
Stop 0.1.0, start 0.2.1 on the same volume
starts normally; the status page shows "version": "v0.2.1", 3 entries, 29 bytes; all three values read back
+
0.2.1: store two more
201 each; 5 entries
+
Stop 0.2.1, start 0.1.0 on the same volume again
starts normally; all values read back, including the two written by 0.2.1
+
+
So in my test an upgrade lost nothing, and a rollback to the old release could still read what the new one had written.
+
+
Two differences between the releases
+
+
0.1.0 has no status page./statusz returned 404 there. 0.2.1 has it. The metrics page is on both.
+
0.1.0's entry count resets after a restart. After the rollback its metric fscache_store_entries read 0 although every value was readable. I checked whether that was caused by the upgrade: it is not. A volume only 0.1.0 ever wrote showed 3 entries before a plain restart and 0 after it, with the data still readable. 0.2.1, restarted the same way, reported 3 entries and 9 bytes correctly.