Monitor a Gradle build cache with Prometheus and Grafana

The cache server exposes Prometheus metrics and a read-only status page. This page lists what is there, a scrape setting, a few queries, and where to get a ready-made Grafana dashboard.

Metrics need no password

The /metrics and /healthz paths stay open even when Basic Auth is on, so a Prometheus scraper needs no credentials. The status page, /statusz, sits behind Basic Auth when it is enabled.

A scrape setting

scrape_configs:
  - job_name: fscache
    static_configs:
      - targets: ['cache.example.com:8080']

This is a plain example. Use your own address, and HTTPS if you have put the server behind a proxy.

The metrics

These are the names a running 0.2.1 server reports:

Metric	Type	What it tells you
`fscache_cache_hits_total`	counter	Requests answered from the cache
`fscache_cache_misses_total`	counter	Requests for something not stored
`fscache_store_bytes`	gauge	How much is stored now
`fscache_store_entries`	gauge	How many entries are stored
`fscache_evicted_entries_total`	counter	Entries removed to stay under the size cap
`fscache_http_requests_total`	counter	Requests, by method and status
`fscache_http_request_duration_seconds`	histogram	How long requests take
`fscache_bytes_read_total`	counter	Bytes sent to clients
`fscache_bytes_written_total`	counter	Bytes received from clients

Queries worth having

# hit rate over the last 5 minutes
sum(rate(fscache_cache_hits_total[5m]))
  / clamp_min(sum(rate(fscache_cache_hits_total[5m])) + sum(rate(fscache_cache_misses_total[5m])), 0.0001)

# entries removed per second to stay under the cap
sum(rate(fscache_evicted_entries_total[5m]))

# 95th percentile request time
histogram_quantile(0.95, sum by (le, method) (rate(fscache_http_request_duration_seconds_bucket[5m])))

A hit rate that never climbs, together with constant evictions, usually means the size cap is too small. See server sizing.

A ready-made Grafana dashboard

The repository includes a dashboard with eight panels: hit rate, cache size against the cap, entries, evictions, hits and misses, requests by status, throughput, and request latency. Download grafana-dashboard.json and import it in Grafana.

The status page

For a quick look without Prometheus, /statusz returns the same numbers in one place, in a browser or as JSON. This is what a freshly started 0.2.1 server returned after one stored entry and two requests:

$ curl -s localhost:8080/statusz
{
  "version": "v0.2.1",
  "revision": "0bfc498ecfca3e38cb4a456acda44359a1620c5a",
  "fips140": false,
  "fips140_note": "off",
  "uptime_seconds": 2.118684875,
  "uptime": "2s",
  "store_bytes": 5,
  "max_bytes": 1073741824,
  "store_entries": 1,
  "cache_hits": 1,
  "cache_misses": 1,
  "hit_ratio": 0.5,
  "evicted_entries": 0,
  "auth_enabled": false
}

Grafana dashboard FosterStack home