Introduction

High-performance polyglot task processing system built in Rust

Introduction

TaskDaemon is a high-performance, polyglot task processing system built in Rust. Execute task handlers written in any language via containerized workers.

Why TaskDaemon?

Language Freedom

Write handlers in Python, Node.js, Go, Rust, C++, Java, C#, or any language

Container Isolation

Each handler runs in its own container with resource limits

High Performance

Rust core with pre-warmed container pools for sub-10ms latency

Production Ready

Persistent queue, automatic retries, Prometheus metrics

Key Features

• Task Selection Strategies - FIFO, LIFO, or Priority-based task ordering
• Handler Selection Strategies - Round-robin, first-available, or random container selection
• HybridQueue - In-memory queue with SQLite persistence for speed + durability
• Priority Tasks - Queue tasks with priority 0-100 for urgent processing
• Prometheus Metrics - Built-in observability with container pool metrics

How It Works

1. Client submits a task via HTTP or gRPC
2. TaskDaemon queues the task (with optional priority)
3. Worker picks up task based on selection strategy
4. Handler container processes task and returns result
5. Result stored and available via API

Quick Example

1. Define a handler (Python)

   from taskdaemon import run, Task, Success
   
   def handler(task: Task) -> Success:
       name = task.task_data.get("name", "World")
       return Success({"message": f"Hello, {name}!"})
   
   run(handler)
   
2. Configure TaskDaemon

   handlers.toml

   [handlers.greet]
   image = "my-greeter:latest"
   instances = 4
   handler_selection = "round-robin"
   
3. Queue a task

   curl -X POST http://localhost:8080/queue \
     -H "Content-Type: application/json" \
     -d '{"type": "greet", "data": {"name": "TaskDaemon"}, "priority": 50}'
   

Handler SDKs

Official SDKs are available for:

Language	Package
Python	pip install taskdaemon
Node.js	npm install @taskdaemon/handler
Go	go get github.com/taskdaemon/handler-go
Rust	cargo add taskdaemon-handler
C++	Header-only: taskdaemon.hpp
Java	Maven/Gradle: com.taskdaemon:handler
C#	dotnet add package TaskDaemon.Handler

Or use the raw protocol to write handlers in any language.

Next Steps

Quick Start

Get running in 5 minutes

Handler SDKs

Write handlers in your language

Configuration

Configure handlers and strategies

Sample Project

Complete working example