Grafana Dashboards

TaskDaemon grafana dashboards guide

Grafana Dashboards

Use the Prometheus setup and the persistent Docker deployment. Configure Grafana’s Prometheus data source as http://prometheus:9090 when both services share the Compose network.

Use a password you choose and a persistent Grafana volume. The example published port is http://localhost:3001; data source provisioning and authentication depend on your Grafana deployment.

Dashboard Panels

Panel	Visualization	Query
Completed tasks per second	Time series	sum(rate(tasks_processed_total{status="success"}[1m]))
Pending tasks	Stat / time series	queue_size
Acquired handler slots	Time series	sum by (handler) (container_pool_active)
Retry scheduling per second	Time series	sum by (task_type) (rate(task_retries_total[5m]))
Execution timeouts per second	Time series	sum by (task_type) (rate(task_timeouts_total[5m]))
Terminal outcomes since startup	Stat	sum(tasks_processed_total{status=~"success|failed|timeout"})

Processing duration P95 by task type:

histogram_quantile(0.95,
  sum by (le, task_type) (rate(task_processing_seconds_bucket[5m])))

Average handler execution time:

sum by (handler) (rate(handler_execution_seconds_sum[5m]))
  / sum by (handler) (rate(handler_execution_seconds_count[5m]))

Terminal success percentage:

100 * sum(rate(tasks_processed_total{status="success"}[5m]))
  / sum(rate(tasks_processed_total{status=~"success|failed|timeout"}[5m]))

Terminal failure percentage:

100 * sum(rate(tasks_processed_total{status=~"failed|timeout"}[5m]))
  / sum(rate(tasks_processed_total{status=~"success|failed|timeout"}[5m]))

Ratios can be undefined while idle. Choose time windows long enough for your traffic, and keep timeout failures distinct when diagnosing handlers.

Interpret the Panels

Counters reset on daemon restart. They are process totals, not lifetime counts in the database. A redriven task or recovered interrupted task can execute more than once.

queue_size is a one-second Pending sample. It excludes already claimed tasks and includes retries awaiting backoff; zero Pending does not mean every task is complete. /health gives a fresh Pending count but does not validate each handler’s readiness. Use completed smoke tasks for that check.

Pool occupancy is more useful than worker_active and worker_idle: the current worker path does not update those two registered gauges. Read the metric definitions before choosing alerts.

Alerts

Configure Grafana alert rules using the actual queries above. Start with queue depth and scrape availability, then add application-specific completion, failure, and latency thresholds. A fixed threshold such as 30 seconds is only useful if it matches your handler timeout and normal runtime.