Handler Selection Strategies

Configure how container handlers are selected from the pool

Handler Selection Strategies

Handler selection controls how container instances are selected from a handler’s pool when processing a task. This is configured per-handler in handlers.toml.

Available Strategies

Round-Robin (Default)

Cycles through container slots sequentially. Each acquisition starts from the next slot in sequence.

handlers.toml

[handlers.echo]
image = "echo-handler:latest"
instances = 10
handler_selection = "round-robin"

Best for: Even distribution of load across containers, general workloads.

First-Available

Always starts from slot 0 and takes the first available container.

handlers.toml

[handlers.video]
image = "video-encoder:latest"
instances = 4
handler_selection = "first-available"

Best for: Handlers with warm caches, maximizing reuse of “hot” containers, reducing cold starts.

Random

Starts from a random slot each time.

handlers.toml

[handlers.compute]
image = "compute-handler:latest"
instances = 50
handler_selection = "random"

Best for: Very high concurrency scenarios where lock contention might be a concern.

Configuration

Handler selection is configured in handlers.toml for each handler:

handlers.toml

[handlers.fast]
image = "fast-handler:latest"
instances = 20
handler_selection = "round-robin"  # Even distribution

[handlers.cached]
image = "cached-handler:latest"
instances = 4
handler_selection = "first-available"  # Maximize cache hits

[handlers.heavy]
image = "heavy-handler:latest"
instances = 100
handler_selection = "random"  # Reduce contention

Performance Notes

Testing showed that round-robin provides the best overall performance for most workloads. The blocking lock creates natural backpressure - workers queue at slots rather than spinning, which improves throughput under load.

Comparison

Strategy	Behavior	Use Case
round-robin	Sequential cycling	General workloads, even distribution
first-available	Always start at slot 0	Cache-heavy handlers
random	Random starting slot	High concurrency

Mixing Strategies

You can use different strategies for different handlers based on their characteristics:

handlers.toml

# Stateless handler - distribute evenly
[handlers.transform]
image = "transform:latest"
instances = 10
handler_selection = "round-robin"

# ML model with warm cache - reuse containers
[handlers.inference]
image = "ml-model:latest"
instances = 4
handler_selection = "first-available"