Troubleshooting

TaskDaemon troubleshooting guide

Troubleshooting

Docker Engine Unavailable

Run docker info against the engine TaskDaemon uses. If it cannot connect, start or repair the engine yourself before starting TaskDaemon. A mounted socket does not start Docker.

For container deployments, mount /var/run/docker.sock:/var/run/docker.sock along with the configuration and persistent database mounts in the Docker guide. Check the selected Docker context and socket permissions if the host client works but the daemon container cannot connect.

Handler Image Missing

Build or pull every configured image before daemon startup:

docker build -t my-handler:latest ./handlers/my-handler
docker image inspect my-handler:latest

Compose depends_on is not a build or readiness check. TaskDaemon creates its own handler containers from the image named in handlers.toml.

Handler Exits or Returns Invalid Output

Keep the process alive, read one JSON task per line, and write exactly one JSON response per task. Flush stdout and send logs to stderr. Test the handler directly:

printf '%s\n' '{"task_id":"test","task_type":"echo","task_data":{},"attempt":1}' \
  | docker run --rm -i my-handler:latest

A successful reply looks like {"status":"success","result":{}}. The first handler attempt is 1. Timeouts, broken I/O, and invalid protocol/status fail the task without automatic retry; only a valid error response with retryable: true requests retry scheduling.

Task Remains Pending or Processing

Inspect the task and daemon logs:

curl -fsS http://localhost:8080/api/tasks/TASK_ID
docker logs taskdaemon

Pending can mean a backlog, no workers, or retry backoff. Processing means a worker reserved the task; it may be waiting behind other tasks in a batch or waiting for a handler slot. The sampled queue_size excludes these reservations.

Healthy workers renew all outstanding claims every 30 seconds. After a crash, a Processing task becomes recoverable when its last lease renewal is more than five minutes old. Recovered work may execute again; make handlers idempotent.

An unconfigured task type is marked Failed with a handler error when a worker reaches it. Check the exact name in [handlers.name]; this is not automatically fixed by increasing workers. Redrive only after addressing the cause, and remember redrive preserves its stored attempts counter and retry budget.

First Task Is Slow

Startup launches Docker clients but does not wait for a handler readiness handshake. The first task may include image/runtime initialization, imports, model loading, and application warmup. There is no fixed cold-start time. Use a smoke task for each handler and choose a timeout appropriate to its initialization and normal work.

Adding instances also adds initialization work and memory use. Measure before increasing the pool.

Queue Grows Faster Than Completion

Compare completed task rate, handler execution time, timeout rate, and pool occupancy. Check whether handler work, CPU/memory, remote services, or SQLite writes limit throughput.

Start with a measured worker/pool configuration rather than raising workers indiscriminately. DAEMON_BATCH_SIZE=5 is the default; larger batches reduce claim transactions but reserve more work and reduce responsiveness to newly arriving tasks. SQLite permits a single writer, so more database connections do not add write parallelism. Benchmark the real Docker handlers; native echo tests cannot predict production throughput.

HTTP or gRPC Connection Fails

Confirm the daemon is running and the intended ports are published:

docker ps
docker port taskdaemon
docker logs taskdaemon

Defaults are HTTP 8080 and plaintext gRPC 50051. HTTP’s DAEMON_HOST does not change the gRPC bind address. Use a plaintext client or a correctly configured TLS proxy.

/health currently returns HTTP 200 even when its JSON status is unhealthy. Inspect the status field instead of relying on the HTTP code alone.

Database Locked or Data Missing

Confirm DAEMON_DB_PATH points into a persistent directory volume. Mount the directory so SQLite’s WAL and shared-memory files stay with the database. The default /tmp/task_queue.db inside a replaced container is not persistent.

Keep SQLite on local storage suitable for WAL. Inspect lock holders and workload contention before changing pool size. Multiple daemon instances sharing a SQLite file can contend, and a network filesystem is not a supported shortcut to distributed queue deployment.

Both SQLite and Hybrid preserve delayed retries and stale claims across restart; Hybrid’s lookup cache does not replace durable queue transitions. DAEMON_QUEUE_TYPE=memory is not a CLI queue option; the library’s MemoryQueue is ephemeral.

Getting Help

Check logs, try DAEMON_LOG_LEVEL=debug, and include the chosen image/source version, relevant configuration, and task error when opening an issue.