Introduction

TaskDaemon introduction guide

Introduction

TaskDaemon is a Rust task daemon that executes containerized handlers written in Python, Node.js, Go, Rust, C++, Java, C#, or another language that speaks its JSON protocol.

What It Provides

• HTTP and gRPC task submission, status, results, deletion, and redrive
• Durable SQLite claims, results, retry budgets, delayed retries, and crash recovery
• FIFO, LIFO, and priority task selection, with configurable claim batches
• Persistent handler processes with per-handler pools, timeouts, environment variables, and optional memory limits
• Prometheus metrics and a fresh queue count through health requests

HybridQueue uses SQLite as the authority for claims and mutations, with a cache for repeated decoded task lookups. It does not bypass SQLite when dequeuing tasks.

How It Works

1. A client submits a task through HTTP or gRPC.
2. The daemon persists it with its priority and retry budget.
3. A worker atomically claims a batch and runs its tasks sequentially in the appropriate handler pool.
4. The handler reads one JSON request from stdin and returns one JSON response on stdout.
5. The daemon persists completion, failure, or a delayed retry.

Handler processes stay alive between tasks. Startup launches them without a readiness handshake, so first-task latency may include initialization. Throughput depends on actual handler work, Docker overhead, task sizes, storage, and concurrency; a native echo benchmark does not establish production Docker performance.

Claims are renewed while workers hold them. Interrupted Processing tasks become recoverable after five minutes without renewal. Recovery gives at-least-once execution, so handlers should tolerate duplicate work or deduplicate external effects by task ID.

Quick Example

Python handler:

from taskdaemon import run, Task, Success

def handler(task: Task) -> Success:
    name = task.task_data.get("name", "World")
    return Success({"message": f"Hello, {name}!"})

run(handler)

Build that handler into my-greeter:latest, then configure it:

[handlers.greet]
image = "my-greeter:latest"
instances = 2

Submit a task to a running daemon:

curl -fsS -X POST http://localhost:8080/queue \
  -H "Content-Type: application/json" \
  -d '{"type":"greet","data":{"name":"TaskDaemon"}}'

The response contains a task ID. Poll /api/tasks/{task_id} for completed or failed before reading its result.

Handler SDKs

SDK source is maintained in TaskDaemon-Handlers. Package availability and release versions are separate from the daemon image; use a validated SDK release or build from that source.

Python taskdaemon==0.1.0, Go github.com/jona62/TaskDaemon-Handlers/go@v0.1.2, npm @taskdaemon/handler 0.1.2, crates.io taskdaemon-handler 0.1.2, Maven Central io.github.jona62:handler 0.1.2, and NuGet TaskDaemon.Handler 0.1.2 are published and verified. The guides use these registry releases by default and provide source alternatives where needed.

Language	Package / module
Python	taskdaemon
Node.js	@taskdaemon/handler
Go	github.com/jona62/TaskDaemon-Handlers/go
Rust	taskdaemon-handler
C++	cpp/include/taskdaemon.hpp
Java	io.github.jona62:handler
C#	TaskDaemon.Handler

See the raw protocol to write handlers without an SDK.

Next Steps

• Quick Start
• Handler SDKs
• Configuration
• Sample Project