[SPIKE-004] Celery + Redis Integration

cardosofelipe commented

2025-12-29 03:50:15 +00:00

Owner

Objective

Integrate Celery with Redis for distributed task queue to handle agent job execution.

Key Questions

How do we structure Celery tasks for agent operations?
What's the task result backend strategy?
How do we handle long-running agent tasks?
How do we integrate Celery with FastAPI's async patterns?
How do we monitor task status and progress?

Research Areas

Celery with FastAPI integration patterns
Redis as broker and result backend
Task chaining and workflows
Celery beat for scheduled tasks
Flower or similar for monitoring

Expected Deliverables

Docker setup with Redis + Celery workers
Task structure for agent operations
Integration with FastAPI endpoints
ADR documenting the pattern

Acceptance Criteria

Can queue agent tasks from API
Tasks execute in worker process
Can query task status
Tasks survive worker restart
Basic monitoring in place

Labels

spike, architecture, infrastructure

## Objective Integrate Celery with Redis for distributed task queue to handle agent job execution. ## Key Questions 1. How do we structure Celery tasks for agent operations? 2. What's the task result backend strategy? 3. How do we handle long-running agent tasks? 4. How do we integrate Celery with FastAPI's async patterns? 5. How do we monitor task status and progress? ## Research Areas - [ ] Celery with FastAPI integration patterns - [ ] Redis as broker and result backend - [ ] Task chaining and workflows - [ ] Celery beat for scheduled tasks - [ ] Flower or similar for monitoring ## Expected Deliverables - Docker setup with Redis + Celery workers - Task structure for agent operations - Integration with FastAPI endpoints - ADR documenting the pattern ## Acceptance Criteria - [ ] Can queue agent tasks from API - [ ] Tasks execute in worker process - [ ] Can query task status - [ ] Tasks survive worker restart - [ ] Basic monitoring in place ## Labels `spike`, `architecture`, `infrastructure`

cardosofelipe added the architecture infrastructure priority:high spike labels 2025-12-29 03:54:32 +00:00

cardosofelipe commented

2025-12-29 04:08:56 +00:00

Author

Owner

Spike Completed

Research completed and documented in:

Spike Document: docs/spikes/SPIKE-004-celery-redis-integration.md
ADR: docs/adrs/ADR-003-background-task-architecture.md

Key Findings:

Celery provides reliable task execution with persistence and retry
Queue-based routing: agent_queue (prefetch=1), git_queue, sync_queue, cicd_queue
Progress reporting via task state updates and SSE events
Flower for web-based monitoring

Decision:

Adopt Celery + Redis for all background task processing with queue-based routing and progress reporting via Redis Pub/Sub events.

This spike can be closed.

## Spike Completed Research completed and documented in: - **Spike Document:** `docs/spikes/SPIKE-004-celery-redis-integration.md` - **ADR:** `docs/adrs/ADR-003-background-task-architecture.md` ### Key Findings: - Celery provides **reliable task execution** with persistence and retry - Queue-based routing: `agent_queue` (prefetch=1), `git_queue`, `sync_queue`, `cicd_queue` - **Progress reporting** via task state updates and SSE events - **Flower** for web-based monitoring ### Decision: Adopt Celery + Redis for all background task processing with queue-based routing and progress reporting via Redis Pub/Sub events. This spike can be closed.

cardosofelipe closed this issue

2025-12-29 12:45:55 +00:00

[SPIKE-004] Celery + Redis Integration #4