Prometheus Health Check: Endpoints, Blackbox Exporter, Gaps

Ask ten engineers what a "prometheus health check" means and you'll get two different answers. Some mean "is Prometheus itself healthy?" Others mean "can I use Prometheus to check whether my service is up?" Both are valid, and mixing them up is the most common source of confusion when teams start building monitoring around Prometheus.

This article separates the two cleanly, walks through the exact configuration for probing an HTTP endpoint with the Blackbox Exporter, gives you a working PromQL alert rule, and then addresses a gap most Prometheus tutorials skip: what happens when the thing you need to monitor isn't a long-running service, but a scheduled job that runs once and disappears.

Prometheus's Own Health Endpoints: /-/healthy and /-/ready

Prometheus exposes two built-in management endpoints that report on its own internal state, not on any service it's scraping.

/-/healthy returns HTTP 200 as long as the Prometheus process itself is alive. It's a liveness signal — if this fails, something is fundamentally wrong with the server and it likely needs a restart.

/-/ready returns HTTP 200 once Prometheus has finished startup tasks, most notably loading data from its write-ahead log and becoming ready to serve queries and accept scrapes. During startup, /-/ready can return 503 even while /-/healthy is already returning 200 — the process is alive but not yet ready to do useful work. This maps directly to the Kubernetes liveness vs. readiness probe model: liveness asks "should this be restarted?", readiness asks "should this receive traffic?" The official Management API documentation confirms this behavior and lists other management endpoints, like /-/reload.

The key thing to internalize: these endpoints tell you nothing about the health of the applications Prometheus monitors. They're purely for monitoring Prometheus itself, which is why a separate mechanism is needed for checking everything else.

Using Prometheus to Check Other Services: The Blackbox Exporter

Prometheus can't natively probe an arbitrary URL and report success or failure — its core model is pulling metrics from /metrics endpoints, not making one-off HTTP requests to see if a page loads. That's the job of the Blackbox Exporter, a separate binary maintained by the Prometheus project specifically for synthetic HTTP, TCP, DNS, and ICMP probing.

The mental model: /-/healthy is an internal self-check, while the Blackbox Exporter is an external synthetic probe — it stands in for a user hitting your endpoint and reports what it sees.

Here's a minimal Blackbox Exporter module configuration for an HTTP health check, blackbox.yml:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []  # defaults to 2xx
      method: GET

Then tell Prometheus to scrape targets through that exporter, using the standard relabeling pattern:

scrape_configs:
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com/health
          - https://api.example.com/status
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115  # blackbox exporter address

This is a well-trodden pattern, and the Robust Perception walkthrough on checking for HTTP 200s is worth reading if you want to see it built up step by step.

What probe_success Actually Means

Every probe the Blackbox Exporter runs produces a metric called probe_success, which is 1 if the probe met its success criteria (for http_2xx, that means a 2xx status code within the timeout) and 0 if it failed for any reason — timeout, connection refused, TLS error, or wrong status code. It's the single metric most dashboards and alerts key off, alongside probe_duration_seconds for latency and probe_http_status_code for the raw response code. The Blackbox Exporter repository documents each module type and the full set of metrics it emits per protocol.

A PromQL Alert Rule for a Failed Health Check

Once probe_success is flowing into Prometheus, an alert rule is straightforward:

groups:
  - name: blackbox_alerts
    rules:
      - alert: EndpointDown
        expr: probe_success == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Health check failing for {{ $labels.instance }}"
          description: "probe_success has been 0 for more than 2 minutes."

The for: 2m clause avoids alerting on a single blip. Route this through Alertmanager for deduplication and paging, and layer a Grafana dashboard on top for visual trends — a graph of probe_success and probe_duration_seconds side by side tells you both whether an endpoint is up and whether it's degrading before it fails outright.

Where This Model Breaks Down: Scheduled and Cron Jobs

Everything above assumes something is continuously reachable and worth polling at a fixed interval — a service, an API, a website. Prometheus's entire scrape model is built around pull-based polling: hit a target every N seconds, record the result, repeat.

Cron jobs and scheduled HTTP tasks don't fit that shape. A nightly batch script, a report generator that runs at 3 a.m., or a webhook-triggered cleanup job doesn't sit there waiting to be polled — it runs, does its work, and exits. Scrape it every 30 seconds and you'll see nothing useful 99.9% of the time, and you still won't know if the 3 a.m. run actually succeeded, ran late, or silently failed halfway through. Polling a job that isn't running can't tell you what you actually need: did this specific execution complete, how long did it take, and what happened if it didn't.

This is the gap that trips up teams who've done everything "right" with Prometheus. You need to know when a job runs, whether it finished, and to get alerted the moment it doesn't — with actual execution history, not just an aggregate uptime percentage.

Why You Still Need Job-Level Monitoring Alongside Prometheus

This is exactly the space Cronevra is built for: monitoring scheduled and cron-triggered HTTP jobs at the level of individual executions, not continuous uptime. Instead of polling, your job pings Cronevra when it starts and finishes (or Cronevra expects a ping on schedule and flags it when one doesn't arrive) — giving you execution history, failure detection, and recovery alerts per run, which is a fundamentally different problem than "is this endpoint reachable right now." If you want to understand the heartbeat/dead-man's-switch approach that makes this possible, The Health Checker: How Heartbeat Monitoring Works walks through the mechanics.

Explaining this to a team already invested in Prometheus is usually easy once you frame it correctly: Prometheus and Blackbox Exporter are excellent at "is this service up right now," and Cronevra fills the adjacent job that Prometheus's architecture wasn't built for — confirming that a specific scheduled run actually happened and succeeded.

Frequently Asked Questions

What is the difference between /-/healthy and /-/ready in Prometheus?

/-/healthy confirms the Prometheus process is alive, while /-/ready confirms it has finished startup and can serve queries and accept scrapes. A server can be healthy but not yet ready during startup, similar to the liveness vs. readiness distinction in Kubernetes.

How do I check if Prometheus itself is running correctly?

Query its /-/healthy and /-/ready management endpoints directly with an HTTP request; both should return 200 on a fully operational instance. These are documented in the official Management API reference.

Can Prometheus monitor HTTP endpoints directly without an exporter?

No — Prometheus's native model scrapes /metrics endpoints; it doesn't perform arbitrary synthetic HTTP requests on its own. For probing any URL, you need the Blackbox Exporter, which Prometheus then scrapes for the probe results.

What does probe_success mean in the Blackbox Exporter?

probe_success is 1 when a probe meets its success criteria (like an HTTP 2xx response within the timeout) and 0 when it fails for any reason, including timeouts or wrong status codes. It's the primary metric used in alert rules and dashboards built on Blackbox Exporter data.

Can Prometheus alert me when a cron job or scheduled task fails?

Not reliably, because Prometheus's pull-based scraping doesn't fit jobs that run briefly and then exit. A dedicated tool like Cronevra that tracks individual job executions and expected schedules is better suited to catching missed or failed runs.

Is Blackbox Exporter the same as a health check endpoint in my app?

No — a health check endpoint (like /health in your app) is code you write that reports your own service's internal state. The Blackbox Exporter is an external prober that Prometheus uses to test whether that endpoint (or any URL) responds successfully from the outside.

Prometheus and Blackbox Exporter cover continuous service uptime well, but scheduled jobs need execution-level visibility that scrape-based monitoring wasn't designed for. See how Cronevra handles that gap, and check the pricing if you're ready to add cron-level monitoring alongside your existing Prometheus setup.