Prometheus Metrics

TaskDaemon prometheus metrics guide

Prometheus Metrics

Scrape http://localhost:8080/metrics. Metric names have no taskdaemon_ prefix. Counters describe the running daemon process and reset on restart; they are not persisted totals from SQLite.

Runtime Metrics

Metric	Type	Labels / meaning
tasks_received_total	Counter	Tasks accepted by this service
tasks_processed_total	Counter	status, task_type; statuses success, failed, timeout, retried
queue_size	Gauge	Pending count sampled about once per second
daemon_health	Gauge	1 normally; 0 after sampler shutdown
task_processing_seconds	Histogram	task_type; recorded processing attempt duration
container_pool_size	Gauge	handler; configured pool slots
container_pool_active	Gauge	handler; acquired slots
container_pool_idle	Gauge	handler; unacquired slots
handler_execution_seconds	Histogram	handler; request/response execution time
handler_executions_total	Counter	handler, restarted
task_queue_wait_seconds	Histogram	task_type; time since task creation before an execution
task_retries_total	Counter	task_type; retry scheduling
task_timeouts_total	Counter	task_type; execution timeout failures

worker_active, worker_idle, container_restarts_total, and container_wait_seconds are also registered. The current worker path does not update them; use the runtime metrics above rather than treating their zero values as activity measurements.

queue_size includes delayed Pending retries and excludes Processing claims. Zero Pending does not prove that every task has completed. Use task states or exact completion counts to verify a drain. /health and gRPC GetHealth read a fresh Pending count; /api/metrics and /metrics use the sampled gauge. When embedding a standalone TaskService, call refresh_queue_metrics().await or start a daemon sampler.

daemon_health does not monitor handler readiness or Docker connectivity. Use /health for a fresh storage check and completed smoke tasks to verify handler execution.

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: taskdaemon
    static_configs:
      - targets: ["taskdaemon:8080"]

For Docker Compose, add Prometheus to the persistent daemon deployment:

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

The taskdaemon:8080 target works when both services share a Compose network. A Prometheus process on the host needs the published host address instead.

Useful Queries

Completed tasks and received tasks per second:

sum(rate(tasks_processed_total{status="success"}[1m]))
sum(rate(tasks_received_total[1m]))

Average processing duration and 95th percentile by task type:

sum by (task_type) (rate(task_processing_seconds_sum[5m]))
  / sum by (task_type) (rate(task_processing_seconds_count[5m]))

histogram_quantile(0.95,
  sum by (le, task_type) (rate(task_processing_seconds_bucket[5m])))

Terminal failure fraction, including timeouts and excluding retry scheduling:

sum(rate(tasks_processed_total{status=~"failed|timeout"}[5m]))
  / sum(rate(tasks_processed_total{status=~"success|failed|timeout"}[5m]))

Pending queue depth, pool occupancy, and retry rate:

queue_size
sum by (handler) (container_pool_active)
sum by (task_type) (rate(task_retries_total[5m]))

Ratios need traffic in the window; an idle denominator can produce no finite value. Choose windows and alerts that match your expected workload.

Example Alerts

groups:
  - name: taskdaemon
    rules:
      - alert: HighQueueDepth
        expr: queue_size > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "TaskDaemon pending queue exceeds 1000 tasks"
      - alert: TaskDaemonUnavailable
        expr: up{job="taskdaemon"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Prometheus cannot scrape TaskDaemon"

Queue thresholds are examples. Delayed retries contribute to Pending depth, and acquired batch tasks do not. Add application-specific failure and completion alerts for useful service-level monitoring.