Operating the verifier
For platform and ops engineers. At the end of this page, Prometheus scrapes your verifier, alerts fire on the conditions that need you, and you know what each readiness state means and how to fix it.
Before you start
- A running verifier. See Deploying the verifier.
- A network path from your probes and Prometheus to the verifier’s port 9090 that stays inside your private network.
The monitor port
Port 9090 is the monitor port. It is read-only: every endpoint is GET, nothing on it changes the verifier, and any other path answers 404.
| Endpoint | Returns |
|---|---|
GET /healthz | Liveness: 200 with {"status":"ok"} while the process serves HTTP. Also on the public port |
GET /readyz | Readiness: the overall status and one block per check. See Readiness |
GET /metrics | Prometheus metrics. See Metrics |
GET /openapi.yaml | The OpenAPI document of both ports, as built into the running version. See Verifier API reference |
The monitor port listens on 127.0.0.1:9090 by default, so only the verifier’s own host reaches it. When probes or Prometheus connect from another host, pod or container, set TRUSTPIN_MONITOR_LISTEN_ADDR=:9090 together with TRUSTPIN_MONITOR_API_KEY. The console’s deployment bundle already does this for Docker Compose and Kubernetes. Never route the monitor port through your public gateway or an Ingress.
Calling it with the key
TRUSTPIN_MONITOR_API_KEY (at least 16 characters, from attest-verifier keygen --secret) is accepted in two forms:
# HTTP Basic: any user name, the key as the password
curl -s -u monitor:"$TRUSTPIN_MONITOR_API_KEY" http://localhost:9090/readyz
# Bearer token
curl -s -H "Authorization: Bearer $TRUSTPIN_MONITOR_API_KEY" http://localhost:9090/metricsWith the key set, a request without it gets:
| Endpoint | Without the key |
|---|---|
/healthz | Unchanged |
/readyz | The same status code, with only {"status": …}, so orchestrator probes keep working |
/metrics, /openapi.yaml | 401 with WWW-Authenticate: Basic |
Without TRUSTPIN_MONITOR_API_KEY, every endpoint answers anyone who can reach the port.
Readiness
GET /readyz answers 200 for ok and degraded, and 503 for unavailable. Point your orchestrator’s readiness probe at it, and its liveness probe at /healthz.
{
"status": "ok",
"checks": {
"policy": { "status": "ok", "version": 5, "source": "cdn", "lastRefreshAt": "2026-10-21T08:00:00Z", "lastRefreshResult": "unchanged" },
"signingKeys": { "status": "ok", "ephemeral": false, "count": 1 },
"revocationList": { "status": "ok", "source": "download", "ageHours": 3 },
"store": { "status": "ok" },
"playIntegrity": { "status": "ok" },
"debugTokens": { "status": "disabled" }
}
}A check’s reason is a fixed text. The details, which can name hosts or users, are only in the verifier’s log.
unavailable (503): the instance must not receive traffic.
| Check | Cause | Fix |
|---|---|---|
policy | No policy loaded yet | Publish the policy in the console, and check that the verifier reaches the policy file. See Policy loading and refresh |
store | store unreachable; see the verifier log | Restore the Redis or Postgres connection named in TRUSTPIN_STORE_URL |
degraded (200): the instance serves traffic but needs attention. Alert on it.
| Check | Cause | Fix |
|---|---|---|
signingKeys | Signing keys are ephemeral: TRUSTPIN_TOKEN_SIGNING_KEYS is unset | Set TRUSTPIN_TOKEN_SIGNING_KEYS and TRUSTPIN_CHALLENGE_SECRETS (attest-verifier keygen) |
debugTokens | Debug tokens are configured | Remove TRUSTPIN_DEBUG_TOKEN_HASHES outside staging |
policy | policy refresh not_found: the policy file was not found on refresh | The last good policy stays in use. Check that the project’s policy is still published |
policy | policy refresh incompatible: a newer policy needs configuration this instance lacks, such as Play Integrity credentials | Add the configuration and restart. Until then the instance stays on the previous version |
policy | Three consecutive failed refreshes, shown as policy refresh <outcome> (error, invalid or rollback_rejected) | The last good policy stays in use. Check the network path to the policy file, and TRUSTPIN_ATTEST_CONFIGURATION_URL if you set it |
revocationList | Google’s attestation revocation list is older than 7 days, or none is available | Allow the verifier to download it. Key Attestation keeps working on the older list; with none, Key Attestation evidence answers 502 |
Metrics
GET /metrics serves the Prometheus text format. Every series is prefixed trustpin_attest_, and labels carry no device or key ids. endpoint is always the route, such as /v1/attest.
| Metric | Type | Labels |
|---|---|---|
trustpin_attest_build_info | gauge (1) | version, commit, go_version |
trustpin_attest_http_requests_total | counter | endpoint, code |
trustpin_attest_http_request_duration_seconds | histogram | endpoint |
trustpin_attest_attestations_total | counter | platform (apple, android, debug), evidence, result (pass, fail, error), reason (a reason code or none) |
trustpin_attest_tokens_issued_total | counter | platform, mode, result, bound (true, false) |
trustpin_attest_challenges_issued_total | counter | none |
trustpin_attest_challenge_rejections_total | counter | reason (expired, bad_mac, reused) |
trustpin_attest_upstream_requests_total | counter | upstream (play_integrity, google_revocation_list, trustpin_cdn, attest_configuration_url), outcome (ok, not_modified, not_found, error) |
trustpin_attest_upstream_request_duration_seconds | histogram | upstream |
trustpin_attest_policy_version | gauge | none |
trustpin_attest_policy_refresh_total | counter | outcome (updated, unchanged, not_found, invalid, incompatible, rollback_rejected, error) |
trustpin_attest_policy_last_success_timestamp_seconds | gauge | none |
trustpin_attest_revocation_list_age_seconds | gauge | none |
trustpin_attest_store_operations_total | counter | operation, outcome |
The Go runtime and process metrics are served too.
Scraping
Download prometheus.yml. It has two jobs:
-
A fixed address, such as a Docker Compose service:
scrape_configs: - job_name: attest-verifier static_configs: - targets: ["verifier:9090"] basic_auth: username: prometheus password_file: /etc/prometheus/monitor-api-key # holds TRUSTPIN_MONITOR_API_KEY -
Kubernetes pods, found through the
prometheus.io/scrape,prometheus.io/portandprometheus.io/pathannotations that the console’s deployment bundle puts on the verifier pods. Mount the key from the verifier’s Secret into Prometheus.
Prometheus sends the key as its basic-auth password; the user name is ignored.
Alerts and dashboard
alerts.yml holds Prometheus alerting rules. Add it to rule_files:
| Alert | Fires when | What to do |
|---|---|---|
AttestVerifierPolicyStale | No successful policy refresh for over an hour, for 10 minutes | The verifier keeps serving its last good policy. Check the network path to the policy file and trustpin_attest_policy_refresh_total by outcome |
AttestVerifierPolicyIncompatible | A refresh found a policy that needs configuration this instance lacks | Add the missing configuration, such as Play Integrity credentials, and restart |
AttestVerifierUpstreamErrors | More than 20% of Play Integrity calls fail, for 10 minutes | Attestations answer 502 upstream_unavailable meanwhile, and the SDKs back off and retry. Check the path to Google and the credentials |
AttestVerifierForgeries | Sustained binding failures: evidence that doesn’t match its device key or challenge | Review the failing reasons, platforms and apps in the logs |
AttestVerifierRevocationListStale | Google’s revocation list is older than 7 days, for an hour | Allow the verifier to download it |
AttestVerifierStoreErrors | Store operations fail, for 5 minutes | Replay protection is affected. Check Redis or Postgres |
dashboard.json is a Grafana dashboard that charts every metric: the policy version and age, attestations and tokens by platform, result and reason, challenges, HTTP traffic and latency, the store, and upstreams. In Grafana, open Dashboards, choose New, then Import, upload the file, and pick your Prometheus data source. The instance variable selects one verifier or all of them.
These files may be copied and modified freely, under the verifier licence’s Code Samples clause.
Logs
The verifier writes one JSON line per request (TRUSTPIN_LOG_FORMAT=json, the default) with event, platform, app, result, reasons, pv, upstream, latency_ms, request_id and sub_prefix. Tokens, key certificates, client data, evidence, challenges and secrets are never logged at any level.
Every response echoes X-Request-ID: your gateway’s value when it is at most 128 printable characters, a generated one otherwise. Use it to match the gateway’s log lines with the verifier’s.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Connection refused on port 9090 from another host or container | The monitor port binds 127.0.0.1 by default | Set TRUSTPIN_MONITOR_LISTEN_ADDR=:9090 with TRUSTPIN_MONITOR_API_KEY |
/metrics answers 401 | TRUSTPIN_MONITOR_API_KEY is set and the scrape carries no key, or the wrong one | Give Prometheus the key as its basic-auth password |
/readyz shows only {"status": …} | The request carries no key | Send the key to see the checks |
degraded with signingKeys right after deploying | The signing keys are unset, so each restart forces every device to attest again | Set TRUSTPIN_TOKEN_SIGNING_KEYS and TRUSTPIN_CHALLENGE_SECRETS |