Environment Variables

TaskDaemon environment variables guide

Environment Variables

Configure TaskDaemon via environment variables. CLI arguments override their corresponding environment values.

Available Variables

Variable	Default	Description
HANDLER_CONFIG	handlers.toml	Handler configuration path
DAEMON_PORT	8080	HTTP server port
DAEMON_GRPC_PORT	50051	gRPC server port
DAEMON_HOST	0.0.0.0	HTTP bind address; gRPC binds all interfaces
DAEMON_WORKERS	2	Worker tasks
DAEMON_BATCH_SIZE	5	Maximum claims reserved by each worker; positive integer
DAEMON_DB_PATH	/tmp/task_queue.db	SQLite database path
DAEMON_DB_MAX_CONNECTIONS	20	SQLite connection pool size
DAEMON_QUEUE_TYPE	sqlite	sqlite or hybrid
DAEMON_TASK_SELECTION	fifo	fifo, lifo, or priority
DAEMON_MAX_RETRIES	3	Retry limit assigned to newly queued tasks
DAEMON_TASK_TIMEOUT	30	Execution timeout when a handler omits timeout
DAEMON_LOG_LEVEL	info	trace, debug, info, warn, or error

DAEMON_WORKER_SLEEP_MS is parsed by the current configuration type but does not control worker polling. Workers currently wait 10 ms after an empty claim and 100 ms after a worker error.

Persistent Docker Configuration

docker run -d --name taskdaemon \
  -p 8080:8080 -p 50051:50051 \
  -e DAEMON_WORKERS=4 \
  -e DAEMON_BATCH_SIZE=5 \
  -e DAEMON_DB_MAX_CONNECTIONS=20 \
  -e DAEMON_DB_PATH=/data/tasks.db \
  -v "$PWD/handlers.toml:/app/handlers.toml:ro" \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v task-data:/data \
  mshelia/taskdaemon:0.2.0-alpha

Mount the directory containing the database so SQLite’s WAL and shared-memory files remain with it. The default /tmp/task_queue.db is inside the container unless you provide a persistent mount. A volume preserves data across container replacement; removing that volume removes the queue history.

services:
  taskdaemon:
    image: mshelia/taskdaemon:0.2.0-alpha
    ports:
      - "8080:8080"
      - "50051:50051"
    environment:
      DAEMON_WORKERS: "4"
      DAEMON_BATCH_SIZE: "5"
      DAEMON_DB_MAX_CONNECTIONS: "20"
      DAEMON_DB_PATH: /data/tasks.db
      DAEMON_QUEUE_TYPE: sqlite
    volumes:
      - ./handlers.toml:/app/handlers.toml:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - task-data:/data

volumes:
  task-data:

Queue Types and Selection

Both SQLite and HybridQueue claim tasks and persist outcomes through SQLite. HybridQueue additionally caches repeated decoded task lookups after checking their durable version. Its cache does not provide a separate in-memory dequeue path. Pending tasks, delayed retries, and expired processing claims remain in the database across restart.

FIFO selects oldest tasks, LIFO selects newest tasks, and priority selects higher numbers first with FIFO ordering at equal priority. HTTP requests accept priority, defaulting to 50. Selection applies when tasks are claimed; an arriving high-priority task cannot preempt a worker’s already reserved batch.

Batch Size and Concurrency

Workers execute their batches sequentially. The default batch size is 5; zero or invalid DAEMON_BATCH_SIZE values fall back to 5. The library’s .batch_size(...) setter requires a positive value.

For short tasks with a large backlog, compare batch sizes 5, 20, and 50 using your handlers. Larger batches reduce claim transactions but reserve more work per worker and can delay new high-priority tasks. Start with four workers and 20 database connections as a measurement baseline, then match worker count to handler capacity and resource limits. SQLite permits one writer at a time; increasing the pool cannot increase its write parallelism.

Local tests with a native Python echo process are useful for comparing changes, but their throughput and speedups are not Docker production guarantees. Include real handler work, cold startup, task sizes, and persistence settings in deployment measurements.

Retries, Timeouts, and Recovery

The default retry limit allows an initial execution plus up to three retries when a handler returns status: "error" with retryable: true. Retry delays start at 1, 2, and 4 seconds and are bounded at one day for larger configured retry limits. Changing DAEMON_MAX_RETRIES affects newly enqueued tasks; existing tasks retain their stored limit.

An omitted handler timeout inherits DAEMON_TASK_TIMEOUT; an explicit timeout overrides it. Timeout, broken container I/O, invalid response status, and container restart failures fail the task without automatic retry. Unusable streams are stopped before another task uses that slot.

Workers renew every outstanding claim every 30 seconds, including claims waiting for a handler or their turn in a batch. Processing tasks become eligible for recovery after five minutes without renewal. Recovery can repeat interrupted work, so handlers should tolerate duplicate execution. Graceful shutdown returns unexecuted reserved claims without spending their retry budget.

Metrics and Exact Queue Counts

Started daemons sample the Prometheus queue_size gauge about once per second. It can lag during a drain. HTTP /health, gRPC GetHealth, and explicit library queue_size() calls read the current pending count. Pending counts exclude already claimed work; inspect task states or completion counts to confirm a full drain.

A standalone TaskService can call refresh_queue_metrics().await explicitly. Keep the daemon handle while workers are running and call shutdown().await for an orderly stop.

Precedence

Values are applied in this order: defaults, environment variables, then supported CLI arguments. For example, DAEMON_WORKERS=4 with --workers 8 uses eight workers. Batch size is currently configured through the environment or library builder, not a CLI flag.