Troubleshooting

Troubleshooting

Common issues and solutions.

Container Issues

”Cannot connect to Docker daemon”

TaskDaemon needs access to the Docker socket to spawn handler containers.

Solution: Mount the Docker socket:

docker run -v /var/run/docker.sock:/var/run/docker.sock mshelia/taskdaemon

“Image not found” for handlers

Handler images must be available to the Docker daemon before TaskDaemon starts.

Solution: Build handler images first:

docker build -t my-handler:latest ./handlers/my-handler
docker run ... mshelia/taskdaemon

Or in Docker Compose, use depends_on:

services:
  taskdaemon:
    depends_on:
      - my-handler
  
  my-handler:
    build: ./handlers/my-handler
    image: my-handler:latest
    command: ["echo", "built"]

Handler container exits immediately

Handlers must keep running and read from stdin continuously.

Solution: Ensure your handler loops:

# Wrong - exits after one task
task = json.loads(input())
print(json.dumps(result))

# Correct - keeps running
for line in sys.stdin:
    task = json.loads(line)
    print(json.dumps(result), flush=True)

Task Issues

Tasks stuck in “pending”

Causes:

1. No workers running
2. Handler type not configured
3. Handler containers failing to start

Debug:

# Check logs
docker logs taskdaemon

# Check handler config
cat handlers.toml

# Verify handler image exists
docker images | grep my-handler

Tasks failing immediately

Causes:

1. Handler returning invalid JSON
2. Handler crashing
3. Timeout too short

Debug:

# Test handler directly
echo '{"task_id":"test","task_type":"echo","task_data":{},"retry_count":0}' | docker run -i my-handler

“Handler not found” error

The task type doesn’t match any configured handler.

Solution: Check handlers.toml:

# Task type "resize" needs this config:
[handlers.resize]
image = "image-handler:latest"

Performance Issues

High latency on first task

Container cold start takes ~500ms. Subsequent tasks use warm containers.

Solution: Increase instances for frequently used handlers:

[handlers.resize]
image = "image-handler:latest"
instances = 8  # More pre-warmed containers

Queue growing faster than processing

Solutions:

1. Increase workers: DAEMON_WORKERS=8
2. Increase handler instances
3. Optimize handler code
4. Scale horizontally (multiple TaskDaemon instances)

Connection Issues

”Connection refused” on port 8080

TaskDaemon isn’t running or port isn’t exposed.

Debug:

# Check if running
docker ps | grep taskdaemon

# Check port mapping
docker port <container_id>

# Check logs for startup errors
docker logs taskdaemon

gRPC connection fails

Causes:

1. Port 50051 not exposed
2. Using wrong port
3. TLS mismatch (TaskDaemon uses plaintext)

Solution:

# Expose gRPC port
docker run -p 50051:50051 mshelia/taskdaemon

# Use insecure connection in client
grpc.insecure_channel('localhost:50051')

Database Issues

”Database is locked”

SQLite doesn’t handle high concurrency well.

Solutions:

1. Reduce workers if running multiple instances
2. Use a persistent volume for the database
3. Consider using a different queue backend (future feature)

Tasks lost after restart

Database wasn’t persisted.

Solution: Mount a volume:

docker run -v task-data:/data -e DAEMON_DB_PATH=/data/tasks.db mshelia/taskdaemon

Getting Help

1. Check logs: docker logs taskdaemon
2. Enable debug logging: DAEMON_LOG_LEVEL=debug
3. Open an issue on GitHub