forked from cardosofelipe/fast-next-template
docs: add architecture decision records (ADRs) for key technical choices
- Added the following ADRs to `docs/adrs/` directory: - ADR-001: MCP Integration Architecture - ADR-002: Real-time Communication Architecture - ADR-003: Background Task Architecture - ADR-004: LLM Provider Abstraction - ADR-005: Technology Stack Selection - Each ADR details the context, decision drivers, considered options, final decisions, and implementation plans. - Documentation aligns technical choices with architecture principles and system requirements for Syndarix.
This commit is contained in:
134
docs/adrs/ADR-001-mcp-integration-architecture.md
Normal file
134
docs/adrs/ADR-001-mcp-integration-architecture.md
Normal file
@@ -0,0 +1,134 @@
|
||||
# ADR-001: MCP Integration Architecture
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2025-12-29
|
||||
**Deciders:** Architecture Team
|
||||
**Related Spikes:** SPIKE-001
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Syndarix requires integration with multiple external services (LLM providers, Git, issue trackers, file systems, CI/CD). The Model Context Protocol (MCP) was identified as the standard for tool integration in AI applications. We need to decide on:
|
||||
|
||||
1. The MCP framework to use
|
||||
2. Server deployment pattern (singleton vs per-project)
|
||||
3. Scoping mechanism for multi-project/multi-agent access
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
- **Simplicity:** Minimize operational complexity
|
||||
- **Resource Efficiency:** Avoid spawning redundant processes
|
||||
- **Consistency:** Unified interface across all integrations
|
||||
- **Scalability:** Support 10+ concurrent projects
|
||||
- **Maintainability:** Easy to add new MCP servers
|
||||
|
||||
## Considered Options
|
||||
|
||||
### Option 1: Per-Project MCP Servers
|
||||
Spawn dedicated MCP server instances for each project.
|
||||
|
||||
**Pros:**
|
||||
- Complete isolation between projects
|
||||
- Simple access control (project owns server)
|
||||
|
||||
**Cons:**
|
||||
- Resource heavy (7 servers × N projects)
|
||||
- Complex orchestration
|
||||
- Difficult to share cross-project resources
|
||||
|
||||
### Option 2: Unified Singleton MCP Servers (Selected)
|
||||
Single instance of each MCP server type, with explicit project/agent scoping.
|
||||
|
||||
**Pros:**
|
||||
- Resource efficient (7 total servers)
|
||||
- Simpler deployment
|
||||
- Enables cross-project learning (if desired)
|
||||
- Consistent management
|
||||
|
||||
**Cons:**
|
||||
- Requires explicit scoping in all tools
|
||||
- Shared state requires careful design
|
||||
|
||||
### Option 3: Hybrid (MCP Proxy)
|
||||
Single proxy that routes to per-project backends.
|
||||
|
||||
**Pros:**
|
||||
- Balance of isolation and efficiency
|
||||
|
||||
**Cons:**
|
||||
- Added complexity
|
||||
- Routing overhead
|
||||
|
||||
## Decision
|
||||
|
||||
**Adopt Option 2: Unified Singleton MCP Servers with explicit scoping.**
|
||||
|
||||
All MCP servers are deployed as singletons. Every tool accepts `project_id` and `agent_id` parameters for:
|
||||
- Access control validation
|
||||
- Audit logging
|
||||
- Context filtering
|
||||
|
||||
## Implementation
|
||||
|
||||
### MCP Server Registry
|
||||
|
||||
| Server | Port | Purpose |
|
||||
|--------|------|---------|
|
||||
| LLM Gateway | 9001 | Route LLM requests with failover |
|
||||
| Git MCP | 9002 | Git operations across providers |
|
||||
| Knowledge Base MCP | 9003 | RAG and document search |
|
||||
| Issues MCP | 9004 | Issue tracking operations |
|
||||
| File System MCP | 9005 | Workspace file operations |
|
||||
| Code Analysis MCP | 9006 | Static analysis, linting |
|
||||
| CI/CD MCP | 9007 | Pipeline operations |
|
||||
|
||||
### Framework Selection
|
||||
|
||||
Use **FastMCP 2.0** for all MCP server implementations:
|
||||
- Decorator-based tool registration
|
||||
- Built-in async support
|
||||
- Compatible with SSE transport
|
||||
- Type-safe with Pydantic
|
||||
|
||||
### Tool Signature Pattern
|
||||
|
||||
```python
|
||||
@mcp.tool()
|
||||
def tool_name(
|
||||
project_id: str, # Required: project scope
|
||||
agent_id: str, # Required: calling agent
|
||||
# ... tool-specific params
|
||||
) -> Result:
|
||||
validate_access(agent_id, project_id)
|
||||
log_tool_usage(agent_id, project_id, "tool_name")
|
||||
# ... implementation
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Single deployment per MCP type simplifies operations
|
||||
- Consistent interface across all tools
|
||||
- Easy to add monitoring/logging centrally
|
||||
- Cross-project analytics possible
|
||||
|
||||
### Negative
|
||||
- All tools must include scoping parameters
|
||||
- Shared state requires careful design
|
||||
- Single point of failure per MCP type (mitigated by multiple instances)
|
||||
|
||||
### Neutral
|
||||
- Requires MCP client manager in FastAPI backend
|
||||
- Authentication handled internally (service tokens for v1)
|
||||
|
||||
## Compliance
|
||||
|
||||
This decision aligns with:
|
||||
- FR-802: MCP-first architecture requirement
|
||||
- NFR-201: Horizontal scalability requirement
|
||||
- NFR-602: Centralized logging requirement
|
||||
|
||||
---
|
||||
|
||||
*This ADR supersedes any previous decisions regarding MCP architecture.*
|
||||
160
docs/adrs/ADR-002-realtime-communication.md
Normal file
160
docs/adrs/ADR-002-realtime-communication.md
Normal file
@@ -0,0 +1,160 @@
|
||||
# ADR-002: Real-time Communication Architecture
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2025-12-29
|
||||
**Deciders:** Architecture Team
|
||||
**Related Spikes:** SPIKE-003
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Syndarix requires real-time communication for:
|
||||
- Agent activity streams
|
||||
- Project progress updates
|
||||
- Build/pipeline status
|
||||
- Client approval requests
|
||||
- Issue change notifications
|
||||
- Interactive chat with agents
|
||||
|
||||
We need to decide between WebSocket and Server-Sent Events (SSE) for real-time data delivery.
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
- **Simplicity:** Minimize implementation complexity
|
||||
- **Reliability:** Built-in reconnection handling
|
||||
- **Scalability:** Support 200+ concurrent connections
|
||||
- **Compatibility:** Work through proxies and load balancers
|
||||
- **Use Case Fit:** Match communication patterns
|
||||
|
||||
## Considered Options
|
||||
|
||||
### Option 1: WebSocket Only
|
||||
Use WebSocket for all real-time communication.
|
||||
|
||||
**Pros:**
|
||||
- Bidirectional communication
|
||||
- Single protocol to manage
|
||||
- Well-supported in FastAPI
|
||||
|
||||
**Cons:**
|
||||
- Manual reconnection logic required
|
||||
- More complex through proxies
|
||||
- Overkill for server-to-client streams
|
||||
|
||||
### Option 2: SSE Only
|
||||
Use Server-Sent Events for all real-time communication.
|
||||
|
||||
**Pros:**
|
||||
- Built-in automatic reconnection
|
||||
- Native HTTP (proxy-friendly)
|
||||
- Simpler implementation
|
||||
|
||||
**Cons:**
|
||||
- Unidirectional only
|
||||
- Browser connection limits per domain
|
||||
|
||||
### Option 3: SSE Primary + WebSocket for Chat (Selected)
|
||||
Use SSE for server-to-client events, WebSocket for bidirectional chat.
|
||||
|
||||
**Pros:**
|
||||
- Best tool for each use case
|
||||
- SSE simplicity for 90% of needs
|
||||
- WebSocket only where truly needed
|
||||
|
||||
**Cons:**
|
||||
- Two protocols to manage
|
||||
|
||||
## Decision
|
||||
|
||||
**Adopt Option 3: SSE as primary transport, WebSocket for interactive chat.**
|
||||
|
||||
### SSE Use Cases (90%)
|
||||
- Agent activity streams
|
||||
- Project progress updates
|
||||
- Build/pipeline status
|
||||
- Approval request notifications
|
||||
- Issue change notifications
|
||||
|
||||
### WebSocket Use Cases (10%)
|
||||
- Interactive chat with agents
|
||||
- Real-time debugging sessions
|
||||
- Future collaboration features
|
||||
|
||||
## Implementation
|
||||
|
||||
### Event Bus with Redis Pub/Sub
|
||||
|
||||
```
|
||||
FastAPI Backend ──publish──> Redis Pub/Sub ──subscribe──> SSE Endpoints
|
||||
│
|
||||
└──> Other Backend Instances
|
||||
```
|
||||
|
||||
### SSE Endpoint Pattern
|
||||
|
||||
```python
|
||||
@router.get("/projects/{project_id}/events")
|
||||
async def project_events(project_id: str, request: Request):
|
||||
async def event_generator():
|
||||
subscriber = await event_bus.subscribe(f"project:{project_id}")
|
||||
try:
|
||||
while not await request.is_disconnected():
|
||||
event = await asyncio.wait_for(
|
||||
subscriber.get_event(), timeout=30.0
|
||||
)
|
||||
yield f"event: {event.type}\ndata: {event.json()}\n\n"
|
||||
finally:
|
||||
await subscriber.unsubscribe()
|
||||
|
||||
return StreamingResponse(
|
||||
event_generator(),
|
||||
media_type="text/event-stream"
|
||||
)
|
||||
```
|
||||
|
||||
### Event Types
|
||||
|
||||
| Category | Event Types |
|
||||
|----------|-------------|
|
||||
| Agent | `agent_started`, `agent_activity`, `agent_completed`, `agent_error` |
|
||||
| Project | `issue_created`, `issue_updated`, `issue_closed` |
|
||||
| Git | `branch_created`, `commit_pushed`, `pr_created`, `pr_merged` |
|
||||
| Workflow | `approval_required`, `sprint_started`, `sprint_completed` |
|
||||
| Pipeline | `pipeline_started`, `pipeline_completed`, `pipeline_failed` |
|
||||
|
||||
### Client Implementation
|
||||
|
||||
- Single SSE connection per project
|
||||
- Event multiplexing through event types
|
||||
- Exponential backoff on reconnection
|
||||
- Native `EventSource` API with automatic reconnect
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Simpler implementation for server-to-client streams
|
||||
- Automatic reconnection reduces client complexity
|
||||
- Works through all HTTP proxies
|
||||
- Reduced server resource usage vs WebSocket
|
||||
|
||||
### Negative
|
||||
- Two protocols to maintain
|
||||
- WebSocket requires manual reconnect logic
|
||||
- SSE limited to ~6 connections per domain (HTTP/1.1)
|
||||
|
||||
### Mitigation
|
||||
- Use HTTP/2 where possible (higher connection limits)
|
||||
- Multiplex all project events on single connection
|
||||
- WebSocket only for interactive chat sessions
|
||||
|
||||
## Compliance
|
||||
|
||||
This decision aligns with:
|
||||
- FR-105: Real-time agent activity monitoring
|
||||
- NFR-102: 200+ concurrent connections requirement
|
||||
- NFR-501: Responsive UI updates
|
||||
|
||||
---
|
||||
|
||||
*This ADR supersedes any previous decisions regarding real-time communication.*
|
||||
179
docs/adrs/ADR-003-background-task-architecture.md
Normal file
179
docs/adrs/ADR-003-background-task-architecture.md
Normal file
@@ -0,0 +1,179 @@
|
||||
# ADR-003: Background Task Architecture
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2025-12-29
|
||||
**Deciders:** Architecture Team
|
||||
**Related Spikes:** SPIKE-004
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Syndarix requires background task processing for:
|
||||
- Agent actions (LLM calls, code generation)
|
||||
- Git operations (clone, commit, push, PR creation)
|
||||
- External synchronization (issue sync with Gitea/GitHub/GitLab)
|
||||
- CI/CD pipeline triggers
|
||||
- Long-running workflows (sprints, story implementation)
|
||||
|
||||
These tasks are too slow for synchronous API responses and need proper queuing, retry, and monitoring.
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
- **Reliability:** Tasks must complete even if workers restart
|
||||
- **Visibility:** Progress tracking for long-running operations
|
||||
- **Scalability:** Handle concurrent agent operations
|
||||
- **Rate Limiting:** Respect LLM API rate limits
|
||||
- **Async Compatibility:** Work with async FastAPI
|
||||
|
||||
## Considered Options
|
||||
|
||||
### Option 1: FastAPI BackgroundTasks
|
||||
Use FastAPI's built-in background tasks.
|
||||
|
||||
**Pros:**
|
||||
- Simple, no additional infrastructure
|
||||
- Direct async integration
|
||||
|
||||
**Cons:**
|
||||
- No persistence (lost on restart)
|
||||
- No retry mechanism
|
||||
- No distributed workers
|
||||
|
||||
### Option 2: Celery + Redis (Selected)
|
||||
Use Celery for task queue with Redis as broker/backend.
|
||||
|
||||
**Pros:**
|
||||
- Mature, battle-tested
|
||||
- Persistent task queue
|
||||
- Built-in retry with backoff
|
||||
- Distributed workers
|
||||
- Task chaining and workflows
|
||||
- Monitoring with Flower
|
||||
|
||||
**Cons:**
|
||||
- Additional infrastructure
|
||||
- Sync-only task execution (bridge needed for async)
|
||||
|
||||
### Option 3: Dramatiq + Redis
|
||||
Use Dramatiq as a simpler Celery alternative.
|
||||
|
||||
**Pros:**
|
||||
- Simpler API than Celery
|
||||
- Good async support
|
||||
|
||||
**Cons:**
|
||||
- Less mature ecosystem
|
||||
- Fewer monitoring tools
|
||||
|
||||
### Option 4: ARQ (Async Redis Queue)
|
||||
Use ARQ for native async task processing.
|
||||
|
||||
**Pros:**
|
||||
- Native async
|
||||
- Simple API
|
||||
|
||||
**Cons:**
|
||||
- Less feature-rich
|
||||
- Smaller community
|
||||
|
||||
## Decision
|
||||
|
||||
**Adopt Option 2: Celery + Redis.**
|
||||
|
||||
Celery provides the reliability, monitoring, and ecosystem maturity needed for production workloads. Redis serves as both broker and result backend.
|
||||
|
||||
## Implementation
|
||||
|
||||
### Queue Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ Redis (Broker + Backend) │
|
||||
├─────────────┬─────────────┬─────────────────────┤
|
||||
│ agent_queue │ git_queue │ sync_queue │
|
||||
│ (prefetch=1)│ (prefetch=4)│ (prefetch=4) │
|
||||
└──────┬──────┴──────┬──────┴──────────┬──────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐
|
||||
│ Agent │ │ Git │ │ Sync │
|
||||
│ Workers │ │ Workers │ │ Workers │
|
||||
└─────────┘ └─────────┘ └─────────┘
|
||||
```
|
||||
|
||||
### Queue Configuration
|
||||
|
||||
| Queue | Prefetch | Concurrency | Purpose |
|
||||
|-------|----------|-------------|---------|
|
||||
| `agent_queue` | 1 | 4 | LLM-based tasks (rate limited) |
|
||||
| `git_queue` | 4 | 8 | Git operations |
|
||||
| `sync_queue` | 4 | 4 | External sync |
|
||||
| `cicd_queue` | 4 | 4 | Pipeline operations |
|
||||
|
||||
### Task Patterns
|
||||
|
||||
**Progress Reporting:**
|
||||
```python
|
||||
@celery_app.task(bind=True)
|
||||
def implement_story(self, story_id: str, agent_id: str, project_id: str):
|
||||
for i, step in enumerate(steps):
|
||||
self.update_state(
|
||||
state="PROGRESS",
|
||||
meta={"current": i + 1, "total": len(steps)}
|
||||
)
|
||||
# Publish SSE event for real-time UI update
|
||||
event_bus.publish(f"project:{project_id}", {
|
||||
"type": "agent_progress",
|
||||
"step": i + 1,
|
||||
"total": len(steps)
|
||||
})
|
||||
execute_step(step)
|
||||
```
|
||||
|
||||
**Task Chaining:**
|
||||
```python
|
||||
workflow = chain(
|
||||
analyze_requirements.s(story_id),
|
||||
design_solution.s(),
|
||||
implement_code.s(),
|
||||
run_tests.s(),
|
||||
create_pr.s()
|
||||
)
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
- **Flower:** Web UI for task monitoring (port 5555)
|
||||
- **Prometheus:** Metrics export for alerting
|
||||
- **Dead Letter Queue:** Failed tasks for investigation
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Reliable task execution with persistence
|
||||
- Automatic retry with exponential backoff
|
||||
- Progress tracking for long operations
|
||||
- Distributed workers for scalability
|
||||
- Rich monitoring and debugging tools
|
||||
|
||||
### Negative
|
||||
- Additional infrastructure (Redis, workers)
|
||||
- Celery is synchronous (event_loop bridge for async calls)
|
||||
- Learning curve for task patterns
|
||||
|
||||
### Mitigation
|
||||
- Use existing Redis instance (already needed for SSE)
|
||||
- Wrap async calls with `asyncio.run()` or `sync_to_async`
|
||||
- Document common task patterns
|
||||
|
||||
## Compliance
|
||||
|
||||
This decision aligns with:
|
||||
- FR-304: Long-running implementation workflow
|
||||
- NFR-102: 500+ background jobs per minute
|
||||
- NFR-402: Task reliability and fault tolerance
|
||||
|
||||
---
|
||||
|
||||
*This ADR supersedes any previous decisions regarding background task processing.*
|
||||
189
docs/adrs/ADR-004-llm-provider-abstraction.md
Normal file
189
docs/adrs/ADR-004-llm-provider-abstraction.md
Normal file
@@ -0,0 +1,189 @@
|
||||
# ADR-004: LLM Provider Abstraction
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2025-12-29
|
||||
**Deciders:** Architecture Team
|
||||
**Related Spikes:** SPIKE-005
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Syndarix agents require access to large language models (LLMs) from multiple providers:
|
||||
- **Anthropic** (Claude) - Primary provider
|
||||
- **OpenAI** (GPT-4) - Fallback provider
|
||||
- **Local models** (Ollama/Llama) - Cost optimization, privacy
|
||||
|
||||
We need a unified abstraction layer that provides:
|
||||
- Consistent API across providers
|
||||
- Automatic failover on errors
|
||||
- Usage tracking and cost management
|
||||
- Rate limiting compliance
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
- **Reliability:** Automatic failover on provider outages
|
||||
- **Cost Control:** Track and limit API spending
|
||||
- **Flexibility:** Easy to add/swap providers
|
||||
- **Consistency:** Single interface for all agents
|
||||
- **Async Support:** Compatible with async FastAPI
|
||||
|
||||
## Considered Options
|
||||
|
||||
### Option 1: Direct Provider SDKs
|
||||
Use Anthropic and OpenAI SDKs directly with custom abstraction.
|
||||
|
||||
**Pros:**
|
||||
- Full control over implementation
|
||||
- No external dependencies
|
||||
|
||||
**Cons:**
|
||||
- Significant development effort
|
||||
- Must maintain failover logic
|
||||
- Must track token costs manually
|
||||
|
||||
### Option 2: LiteLLM (Selected)
|
||||
Use LiteLLM as unified abstraction layer.
|
||||
|
||||
**Pros:**
|
||||
- Unified API for 100+ providers
|
||||
- Built-in failover and routing
|
||||
- Automatic token counting
|
||||
- Cost tracking built-in
|
||||
- Redis caching support
|
||||
- Active community
|
||||
|
||||
**Cons:**
|
||||
- External dependency
|
||||
- May lag behind provider SDK updates
|
||||
|
||||
### Option 3: LangChain
|
||||
Use LangChain's LLM abstraction.
|
||||
|
||||
**Pros:**
|
||||
- Large ecosystem
|
||||
- Many integrations
|
||||
|
||||
**Cons:**
|
||||
- Heavy dependency
|
||||
- Overkill for just LLM abstraction
|
||||
- Complexity overhead
|
||||
|
||||
## Decision
|
||||
|
||||
**Adopt Option 2: LiteLLM for unified LLM provider abstraction.**
|
||||
|
||||
LiteLLM provides the reliability, monitoring, and multi-provider support needed with minimal overhead.
|
||||
|
||||
## Implementation
|
||||
|
||||
### Model Groups
|
||||
|
||||
| Group Name | Use Case | Primary Model | Fallback |
|
||||
|------------|----------|---------------|----------|
|
||||
| `high-reasoning` | Complex analysis, architecture | Claude 3.5 Sonnet | GPT-4 Turbo |
|
||||
| `fast-response` | Quick tasks, simple queries | Claude 3 Haiku | GPT-4o Mini |
|
||||
| `cost-optimized` | High-volume, non-critical | Local Llama 3 | Claude 3 Haiku |
|
||||
|
||||
### Failover Chain
|
||||
|
||||
```
|
||||
Claude 3.5 Sonnet (Anthropic)
|
||||
│
|
||||
▼ (on failure)
|
||||
GPT-4 Turbo (OpenAI)
|
||||
│
|
||||
▼ (on failure)
|
||||
Llama 3 (Ollama/Local)
|
||||
│
|
||||
▼ (on failure)
|
||||
Error with retry
|
||||
```
|
||||
|
||||
### LLM Gateway Service
|
||||
|
||||
```python
|
||||
class LLMGateway:
|
||||
def __init__(self):
|
||||
self.router = Router(
|
||||
model_list=model_list,
|
||||
fallbacks=[
|
||||
{"high-reasoning": ["high-reasoning", "local-fallback"]},
|
||||
],
|
||||
routing_strategy="latency-based-routing",
|
||||
num_retries=3,
|
||||
)
|
||||
|
||||
async def complete(
|
||||
self,
|
||||
agent_id: str,
|
||||
project_id: str,
|
||||
messages: list[dict],
|
||||
model_preference: str = "high-reasoning",
|
||||
) -> dict:
|
||||
response = await self.router.acompletion(
|
||||
model=model_preference,
|
||||
messages=messages,
|
||||
)
|
||||
await self._track_usage(agent_id, project_id, response)
|
||||
return response
|
||||
```
|
||||
|
||||
### Cost Tracking
|
||||
|
||||
| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|
||||
|-------|----------------------|------------------------|
|
||||
| Claude 3.5 Sonnet | $3.00 | $15.00 |
|
||||
| Claude 3 Haiku | $0.25 | $1.25 |
|
||||
| GPT-4 Turbo | $10.00 | $30.00 |
|
||||
| GPT-4o Mini | $0.15 | $0.60 |
|
||||
| Ollama (local) | $0.00 | $0.00 |
|
||||
|
||||
### Agent Type Mapping
|
||||
|
||||
| Agent Type | Model Preference | Rationale |
|
||||
|------------|------------------|-----------|
|
||||
| Product Owner | high-reasoning | Complex requirements analysis |
|
||||
| Software Architect | high-reasoning | Architecture decisions |
|
||||
| Software Engineer | high-reasoning | Code generation |
|
||||
| QA Engineer | fast-response | Test case generation |
|
||||
| DevOps Engineer | fast-response | Config generation |
|
||||
| Project Manager | fast-response | Status updates |
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
- **Redis-backed cache** for repeated queries
|
||||
- **TTL:** 1 hour for general queries
|
||||
- **Skip cache:** For context-dependent generation
|
||||
- **Cache key:** Hash of (model, messages, temperature)
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Single interface for all LLM operations
|
||||
- Automatic failover improves reliability
|
||||
- Built-in cost tracking and budgeting
|
||||
- Easy to add new providers
|
||||
- Caching reduces API costs
|
||||
|
||||
### Negative
|
||||
- Dependency on LiteLLM library
|
||||
- May lag behind provider SDK features
|
||||
- Additional abstraction layer
|
||||
|
||||
### Mitigation
|
||||
- Pin LiteLLM version, test before upgrades
|
||||
- Direct SDK access available if needed
|
||||
- Monitor LiteLLM updates for breaking changes
|
||||
|
||||
## Compliance
|
||||
|
||||
This decision aligns with:
|
||||
- FR-101: Agent type model configuration
|
||||
- NFR-103: Agent response time targets
|
||||
- NFR-402: Failover requirements
|
||||
- TR-001: LLM API unavailability mitigation
|
||||
|
||||
---
|
||||
|
||||
*This ADR supersedes any previous decisions regarding LLM integration.*
|
||||
156
docs/adrs/ADR-005-tech-stack-selection.md
Normal file
156
docs/adrs/ADR-005-tech-stack-selection.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# ADR-005: Technology Stack Selection
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2025-12-29
|
||||
**Deciders:** Architecture Team
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Syndarix needs a robust, modern technology stack that can support:
|
||||
- Multi-agent orchestration with real-time communication
|
||||
- Full-stack web application with API backend
|
||||
- Background task processing for long-running operations
|
||||
- Vector search for RAG (Retrieval-Augmented Generation)
|
||||
- Multiple external integrations via MCP
|
||||
|
||||
The decision was made to build upon **PragmaStack** as the foundation, extending it with Syndarix-specific components.
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
- **Productivity:** Rapid development with modern frameworks
|
||||
- **Type Safety:** Minimize runtime errors
|
||||
- **Async Performance:** Handle concurrent agent operations
|
||||
- **Ecosystem:** Rich library support
|
||||
- **Familiarity:** Team expertise with selected technologies
|
||||
- **Production-Ready:** Proven technologies for production workloads
|
||||
|
||||
## Decision
|
||||
|
||||
**Adopt PragmaStack as foundation with Syndarix-specific extensions.**
|
||||
|
||||
### Core Stack (from PragmaStack)
|
||||
|
||||
| Layer | Technology | Version | Rationale |
|
||||
|-------|------------|---------|-----------|
|
||||
| **Backend** | FastAPI | 0.115+ | Async, OpenAPI, type hints |
|
||||
| **Backend Language** | Python | 3.11+ | Type hints, async/await, ecosystem |
|
||||
| **Frontend** | Next.js | 16 | React 19, server components, App Router |
|
||||
| **Frontend Language** | TypeScript | 5.0+ | Type safety, IDE support |
|
||||
| **Database** | PostgreSQL | 15+ | Robust, extensible, pgvector |
|
||||
| **ORM** | SQLAlchemy | 2.0+ | Async support, type hints |
|
||||
| **Validation** | Pydantic | 2.0+ | Data validation, serialization |
|
||||
| **State Management** | Zustand | 4.0+ | Simple, performant |
|
||||
| **Data Fetching** | TanStack Query | 5.0+ | Caching, invalidation |
|
||||
| **UI Components** | shadcn/ui | Latest | Accessible, customizable |
|
||||
| **CSS** | Tailwind CSS | 4.0+ | Utility-first, fast styling |
|
||||
| **Auth** | JWT | - | Dual-token (access + refresh) |
|
||||
|
||||
### Syndarix Extensions
|
||||
|
||||
| Component | Technology | Version | Purpose |
|
||||
|-----------|------------|---------|---------|
|
||||
| **Task Queue** | Celery | 5.3+ | Background job processing |
|
||||
| **Message Broker** | Redis | 7.0+ | Celery broker, caching, pub/sub |
|
||||
| **Vector Store** | pgvector | Latest | Embeddings for RAG |
|
||||
| **MCP Framework** | FastMCP | 2.0+ | MCP server development |
|
||||
| **LLM Abstraction** | LiteLLM | Latest | Multi-provider LLM access |
|
||||
| **Real-time** | SSE + WebSocket | - | Event streaming, chat |
|
||||
|
||||
### Testing Stack
|
||||
|
||||
| Type | Technology | Purpose |
|
||||
|------|------------|---------|
|
||||
| **Backend Unit** | pytest | 8.0+ | Python testing |
|
||||
| **Backend Async** | pytest-asyncio | Async test support |
|
||||
| **Backend Coverage** | coverage.py | Code coverage |
|
||||
| **Frontend Unit** | Jest | 29+ | React testing |
|
||||
| **Frontend Components** | React Testing Library | Component testing |
|
||||
| **E2E** | Playwright | 1.40+ | Browser automation |
|
||||
|
||||
### DevOps Stack
|
||||
|
||||
| Component | Technology | Purpose |
|
||||
|-----------|------------|---------|
|
||||
| **Containerization** | Docker | 24+ | Application packaging |
|
||||
| **Orchestration** | Docker Compose | Local development |
|
||||
| **CI/CD** | Gitea Actions | Automated pipelines |
|
||||
| **Database Migrations** | Alembic | Schema versioning |
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Frontend (Next.js 16) │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ Pages │ │ Components │ │ Stores │ │
|
||||
│ │ (App Router)│ │ (shadcn/ui) │ │ (Zustand) │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
└────────────────────────────┬────────────────────────────────────┘
|
||||
│ REST + SSE + WebSocket
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Backend (FastAPI 0.115+) │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ API │ │ Services │ │ CRUD │ │
|
||||
│ │ Routes │ │ Layer │ │ Layer │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ LLM Gateway │ │ MCP Client │ │ Event Bus │ │
|
||||
│ │ (LiteLLM) │ │ Manager │ │ (Redis) │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
└────────────────────────────┬────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────────┼────────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌───────────────┐ ┌───────────────┐ ┌───────────────────────────┐
|
||||
│ PostgreSQL │ │ Redis │ │ MCP Servers │
|
||||
│ + pgvector │ │ (Cache/Queue) │ │ (LLM, Git, KB, Issues...) │
|
||||
└───────────────┘ └───────────────┘ └───────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────┐
|
||||
│ Celery │
|
||||
│ Workers │
|
||||
└───────────────┘
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Proven, production-ready stack
|
||||
- Strong typing throughout (Python + TypeScript)
|
||||
- Excellent async performance
|
||||
- Rich ecosystem for extensions
|
||||
- Team familiarity reduces learning curve
|
||||
|
||||
### Negative
|
||||
- Python GIL limits CPU-bound concurrency (mitigated by Celery)
|
||||
- Multiple languages (Python + TypeScript) to maintain
|
||||
- PostgreSQL requires management (vs serverless options)
|
||||
|
||||
### Neutral
|
||||
- PragmaStack provides solid foundation but may include unused features
|
||||
- Stack is opinionated, limiting some technology choices
|
||||
|
||||
## Version Pinning Strategy
|
||||
|
||||
| Component | Strategy | Rationale |
|
||||
|-----------|----------|-----------|
|
||||
| Python | 3.11+ (specific minor) | Stability |
|
||||
| Node.js | 20 LTS | Long-term support |
|
||||
| FastAPI | 0.115+ | Latest stable |
|
||||
| Next.js | 16 | Current major |
|
||||
| PostgreSQL | 15+ | Required for features |
|
||||
|
||||
## Compliance
|
||||
|
||||
This decision aligns with:
|
||||
- NFR-601: Code quality standards (TypeScript, type hints)
|
||||
- NFR-603: Docker containerization requirement
|
||||
- TC-001 through TC-006: Technical constraints
|
||||
|
||||
---
|
||||
|
||||
*This ADR establishes the foundational technology choices for Syndarix.*
|
||||
260
docs/adrs/ADR-006-agent-orchestration.md
Normal file
260
docs/adrs/ADR-006-agent-orchestration.md
Normal file
@@ -0,0 +1,260 @@
|
||||
# ADR-006: Agent Orchestration Architecture
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2025-12-29
|
||||
**Deciders:** Architecture Team
|
||||
**Related Spikes:** SPIKE-002
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Syndarix requires an agent orchestration system that can:
|
||||
- Define reusable agent types with specific capabilities
|
||||
- Spawn multiple instances of the same type with unique identities
|
||||
- Manage agent state, context, and conversation history
|
||||
- Route messages between agents
|
||||
- Handle agent failover and recovery
|
||||
- Track resource usage per agent
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
- **Flexibility:** Support diverse agent roles and capabilities
|
||||
- **Scalability:** Handle 50+ concurrent agent instances
|
||||
- **Isolation:** Each instance maintains separate state
|
||||
- **Observability:** Full visibility into agent activities
|
||||
- **Reliability:** Graceful handling of failures
|
||||
|
||||
## Decision
|
||||
|
||||
**Adopt a Type-Instance pattern** where:
|
||||
- **Agent Types** define templates (model, expertise, personality)
|
||||
- **Agent Instances** are spawned from types with unique identities
|
||||
- **Agent Orchestrator** manages lifecycle and communication
|
||||
|
||||
## Architecture
|
||||
|
||||
### Agent Type Definition
|
||||
|
||||
```python
|
||||
class AgentType(Base):
|
||||
id = Column(UUID, primary_key=True)
|
||||
name = Column(String(50), unique=True) # "Software Engineer"
|
||||
role = Column(Enum(AgentRole)) # ENGINEER
|
||||
base_model = Column(String(100)) # "claude-3-5-sonnet-20241022"
|
||||
failover_model = Column(String(100)) # "gpt-4-turbo"
|
||||
expertise = Column(ARRAY(String)) # ["python", "fastapi", "testing"]
|
||||
personality = Column(JSONB) # {"style": "detailed", "tone": "professional"}
|
||||
system_prompt = Column(Text) # Base system prompt template
|
||||
capabilities = Column(ARRAY(String)) # ["code_generation", "code_review"]
|
||||
is_active = Column(Boolean, default=True)
|
||||
```
|
||||
|
||||
### Agent Instance Definition
|
||||
|
||||
```python
|
||||
class AgentInstance(Base):
|
||||
id = Column(UUID, primary_key=True)
|
||||
name = Column(String(50)) # "Dave"
|
||||
agent_type_id = Column(UUID, ForeignKey)
|
||||
project_id = Column(UUID, ForeignKey)
|
||||
status = Column(Enum(InstanceStatus)) # ACTIVE, IDLE, TERMINATED
|
||||
context = Column(JSONB) # Current working context
|
||||
conversation_id = Column(UUID) # Active conversation
|
||||
rag_collection_id = Column(String) # Domain knowledge collection
|
||||
token_usage = Column(JSONB) # {"prompt": 0, "completion": 0}
|
||||
last_active_at = Column(DateTime)
|
||||
created_at = Column(DateTime)
|
||||
terminated_at = Column(DateTime)
|
||||
```
|
||||
|
||||
### Orchestrator Service
|
||||
|
||||
```python
|
||||
class AgentOrchestrator:
|
||||
"""Central service for agent lifecycle management."""
|
||||
|
||||
async def spawn_agent(
|
||||
self,
|
||||
agent_type_id: UUID,
|
||||
project_id: UUID,
|
||||
name: str,
|
||||
domain_knowledge: list[str] = None
|
||||
) -> AgentInstance:
|
||||
"""Spawn a new agent instance from a type definition."""
|
||||
agent_type = await self.get_agent_type(agent_type_id)
|
||||
|
||||
instance = AgentInstance(
|
||||
name=name,
|
||||
agent_type_id=agent_type_id,
|
||||
project_id=project_id,
|
||||
status=InstanceStatus.ACTIVE,
|
||||
context={"initialized_at": datetime.utcnow().isoformat()},
|
||||
)
|
||||
|
||||
# Initialize RAG collection if domain knowledge provided
|
||||
if domain_knowledge:
|
||||
instance.rag_collection_id = await self._init_rag_collection(
|
||||
instance.id, domain_knowledge
|
||||
)
|
||||
|
||||
await self.db.add(instance)
|
||||
await self.db.commit()
|
||||
|
||||
# Publish spawn event
|
||||
await self.event_bus.publish(f"project:{project_id}", {
|
||||
"type": "agent_spawned",
|
||||
"agent_id": str(instance.id),
|
||||
"name": name,
|
||||
"role": agent_type.role.value
|
||||
})
|
||||
|
||||
return instance
|
||||
|
||||
async def terminate_agent(self, instance_id: UUID) -> None:
|
||||
"""Terminate an agent instance and release resources."""
|
||||
instance = await self.get_instance(instance_id)
|
||||
instance.status = InstanceStatus.TERMINATED
|
||||
instance.terminated_at = datetime.utcnow()
|
||||
|
||||
# Cleanup RAG collection
|
||||
if instance.rag_collection_id:
|
||||
await self._cleanup_rag_collection(instance.rag_collection_id)
|
||||
|
||||
await self.db.commit()
|
||||
|
||||
async def send_message(
|
||||
self,
|
||||
from_id: UUID,
|
||||
to_id: UUID,
|
||||
message: AgentMessage
|
||||
) -> None:
|
||||
"""Route a message from one agent to another."""
|
||||
# Validate both agents exist and are active
|
||||
sender = await self.get_instance(from_id)
|
||||
recipient = await self.get_instance(to_id)
|
||||
|
||||
# Persist message
|
||||
await self.message_store.save(message)
|
||||
|
||||
# If recipient is idle, trigger action
|
||||
if recipient.status == InstanceStatus.IDLE:
|
||||
await self._trigger_agent_action(recipient.id, message)
|
||||
|
||||
# Publish for real-time tracking
|
||||
await self.event_bus.publish(f"project:{sender.project_id}", {
|
||||
"type": "agent_message",
|
||||
"from": str(from_id),
|
||||
"to": str(to_id),
|
||||
"preview": message.content[:100]
|
||||
})
|
||||
|
||||
async def broadcast(
|
||||
self,
|
||||
from_id: UUID,
|
||||
target_role: AgentRole,
|
||||
message: AgentMessage
|
||||
) -> None:
|
||||
"""Broadcast a message to all agents of a specific role."""
|
||||
sender = await self.get_instance(from_id)
|
||||
recipients = await self.get_instances_by_role(
|
||||
sender.project_id, target_role
|
||||
)
|
||||
|
||||
for recipient in recipients:
|
||||
await self.send_message(from_id, recipient.id, message)
|
||||
```
|
||||
|
||||
### Agent Execution Pattern
|
||||
|
||||
```python
|
||||
class AgentRunner:
|
||||
"""Executes agent actions using LLM."""
|
||||
|
||||
def __init__(self, instance: AgentInstance, llm_gateway: LLMGateway):
|
||||
self.instance = instance
|
||||
self.llm = llm_gateway
|
||||
|
||||
async def execute(self, action: str, context: dict) -> dict:
|
||||
"""Execute an action using the agent's configured model."""
|
||||
agent_type = await self.get_agent_type(self.instance.agent_type_id)
|
||||
|
||||
# Build messages with system prompt and context
|
||||
messages = [
|
||||
{"role": "system", "content": self._build_system_prompt(agent_type)},
|
||||
*self._get_conversation_history(),
|
||||
{"role": "user", "content": self._build_action_prompt(action, context)}
|
||||
]
|
||||
|
||||
# Add RAG context if available
|
||||
if self.instance.rag_collection_id:
|
||||
rag_context = await self._query_rag(action, context)
|
||||
messages.insert(1, {
|
||||
"role": "system",
|
||||
"content": f"Relevant context:\n{rag_context}"
|
||||
})
|
||||
|
||||
# Execute with failover
|
||||
response = await self.llm.complete(
|
||||
agent_id=str(self.instance.id),
|
||||
project_id=str(self.instance.project_id),
|
||||
messages=messages,
|
||||
model_preference=self._get_model_preference(agent_type)
|
||||
)
|
||||
|
||||
# Update instance context
|
||||
self.instance.context = {
|
||||
**self.instance.context,
|
||||
"last_action": action,
|
||||
"last_response_at": datetime.utcnow().isoformat()
|
||||
}
|
||||
|
||||
return response
|
||||
```
|
||||
|
||||
### Agent Roles
|
||||
|
||||
| Role | Instances | Primary Capabilities |
|
||||
|------|-----------|---------------------|
|
||||
| Product Owner | 1 | requirements, prioritization, client_communication |
|
||||
| Project Manager | 1 | planning, tracking, coordination |
|
||||
| Business Analyst | 1 | analysis, documentation, process_modeling |
|
||||
| Software Architect | 1 | design, architecture_decisions, tech_selection |
|
||||
| Software Engineer | 1-5 | code_generation, code_review, testing |
|
||||
| UI/UX Designer | 1 | design, wireframes, accessibility |
|
||||
| QA Engineer | 1-2 | test_planning, test_automation, bug_reporting |
|
||||
| DevOps Engineer | 1 | cicd, infrastructure, deployment |
|
||||
| AI/ML Engineer | 1 | ml_development, model_training, mlops |
|
||||
| Security Expert | 1 | security_review, vulnerability_assessment |
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Clear separation between type definition and instance runtime
|
||||
- Multiple instances share type configuration (DRY)
|
||||
- Easy to add new agent roles
|
||||
- Full observability through events
|
||||
- Graceful failure handling with model failover
|
||||
|
||||
### Negative
|
||||
- Complexity in managing instance lifecycle
|
||||
- State synchronization across instances
|
||||
- Memory overhead for context storage
|
||||
|
||||
### Mitigation
|
||||
- Context archival for long-running instances
|
||||
- Periodic cleanup of terminated instances
|
||||
- State compression for large contexts
|
||||
|
||||
## Compliance
|
||||
|
||||
This decision aligns with:
|
||||
- FR-101: Agent type configuration
|
||||
- FR-102: Agent instance spawning
|
||||
- FR-103: Agent domain knowledge (RAG)
|
||||
- FR-104: Inter-agent communication
|
||||
- FR-105: Agent activity monitoring
|
||||
|
||||
---
|
||||
|
||||
*This ADR establishes the agent orchestration architecture for Syndarix.*
|
||||
487
docs/architecture/ARCHITECTURE_OVERVIEW.md
Normal file
487
docs/architecture/ARCHITECTURE_OVERVIEW.md
Normal file
@@ -0,0 +1,487 @@
|
||||
# Syndarix Architecture Overview
|
||||
|
||||
**Version:** 1.0
|
||||
**Date:** 2025-12-29
|
||||
**Status:** Draft
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Executive Summary](#1-executive-summary)
|
||||
2. [System Context](#2-system-context)
|
||||
3. [High-Level Architecture](#3-high-level-architecture)
|
||||
4. [Core Components](#4-core-components)
|
||||
5. [Data Architecture](#5-data-architecture)
|
||||
6. [Integration Architecture](#6-integration-architecture)
|
||||
7. [Security Architecture](#7-security-architecture)
|
||||
8. [Deployment Architecture](#8-deployment-architecture)
|
||||
9. [Cross-Cutting Concerns](#9-cross-cutting-concerns)
|
||||
10. [Architecture Decisions](#10-architecture-decisions)
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
Syndarix is an AI-powered software consulting agency platform that orchestrates specialized AI agents to deliver complete software solutions autonomously. This document describes the technical architecture that enables:
|
||||
|
||||
- **Multi-Agent Orchestration:** 10 specialized agent roles collaborating on projects
|
||||
- **MCP-First Integration:** All external tools via Model Context Protocol
|
||||
- **Real-time Visibility:** SSE-based event streaming for progress tracking
|
||||
- **Autonomous Workflows:** Configurable autonomy levels from full control to autonomous
|
||||
- **Full Artifact Delivery:** Code, documentation, tests, and ADRs
|
||||
|
||||
### Architecture Principles
|
||||
|
||||
1. **MCP-First:** All integrations through unified MCP servers
|
||||
2. **Event-Driven:** Async communication via Redis Pub/Sub
|
||||
3. **Type-Safe:** Full typing in Python and TypeScript
|
||||
4. **Stateless Services:** Horizontal scaling through stateless design
|
||||
5. **Explicit Scoping:** All operations scoped to project/agent
|
||||
|
||||
---
|
||||
|
||||
## 2. System Context
|
||||
|
||||
### Context Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ EXTERNAL ACTORS │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ Client │ │ Admin │ │ LLM APIs │ │ Git Hosts │ │
|
||||
│ │ (Human) │ │ (Human) │ │ (Anthropic) │ │ (Gitea) │ │
|
||||
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
|
||||
│ │ │ │ │ │
|
||||
└─────────│──────────────────│──────────────────│──────────────────│──────────┘
|
||||
│ │ │ │
|
||||
│ Web UI │ Admin UI │ API │ API
|
||||
│ SSE │ │ │
|
||||
▼ ▼ ▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ │
|
||||
│ SYNDARIX PLATFORM │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Agent Orchestration │ │
|
||||
│ │ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │ │
|
||||
│ │ │ PO │ │ PM │ │ Arch │ │ Eng │ │ QA │ ... │ │
|
||||
│ │ └────────┘ └────────┘ └────────┘ └────────┘ └────────┘ │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│ │ │ │
|
||||
│ Storage │ Events │ Tasks │
|
||||
▼ ▼ ▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ INFRASTRUCTURE │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │ │ Redis │ │ Celery │ │MCP Servers │ │
|
||||
│ │ + pgvector │ │ Pub/Sub │ │ Workers │ │ (7 types) │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key Actors
|
||||
|
||||
| Actor | Type | Interaction |
|
||||
|-------|------|-------------|
|
||||
| Client | Human | Web UI, approvals, feedback |
|
||||
| Admin | Human | Configuration, monitoring |
|
||||
| LLM Providers | External | Claude, GPT-4, local models |
|
||||
| Git Hosts | External | Gitea, GitHub, GitLab |
|
||||
| CI/CD Systems | External | Gitea Actions, etc. |
|
||||
|
||||
---
|
||||
|
||||
## 3. High-Level Architecture
|
||||
|
||||
### Layered Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────────────────────────────┐
|
||||
│ PRESENTATION LAYER │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Next.js 16 Frontend │ │
|
||||
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
|
||||
│ │ │Dashboard │ │ Projects │ │ Agents │ │ Issues │ │ │
|
||||
│ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │
|
||||
│ └─────────────────────────────────────────────────────────────┘ │
|
||||
└───────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ REST + SSE + WebSocket
|
||||
▼
|
||||
┌───────────────────────────────────────────────────────────────────┐
|
||||
│ APPLICATION LAYER │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||||
│ │ FastAPI Backend │ │
|
||||
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
|
||||
│ │ │ Auth │ │ API │ │ Services │ │ Events │ │ │
|
||||
│ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │
|
||||
│ └─────────────────────────────────────────────────────────────┘ │
|
||||
└───────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────────────────────────────────┐
|
||||
│ ORCHESTRATION LAYER │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||||
│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ │
|
||||
│ │ │ Agent │ │ Workflow │ │ Project │ │ │
|
||||
│ │ │ Orchestrator │ │ Engine │ │ Manager │ │ │
|
||||
│ │ └───────────────┘ └───────────────┘ └───────────────┘ │ │
|
||||
│ └─────────────────────────────────────────────────────────────┘ │
|
||||
└───────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────────────────────────────────┐
|
||||
│ INTEGRATION LAYER │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||||
│ │ MCP Client Manager │ │
|
||||
│ │ Connects to: LLM, Git, KB, Issues, FS, Code, CI/CD MCPs │ │
|
||||
│ └─────────────────────────────────────────────────────────────┘ │
|
||||
└───────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────────────────────────────────┐
|
||||
│ DATA LAYER │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ PostgreSQL │ │ Redis │ │ File Store │ │
|
||||
│ │ + pgvector │ │ │ │ │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└───────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Core Components
|
||||
|
||||
### 4.1 Agent Orchestrator
|
||||
|
||||
**Purpose:** Manages agent lifecycle, spawning, communication, and coordination.
|
||||
|
||||
**Responsibilities:**
|
||||
- Spawn agent instances from type definitions
|
||||
- Route messages between agents
|
||||
- Manage agent context and memory
|
||||
- Handle agent failover
|
||||
- Track resource usage
|
||||
|
||||
**Key Patterns:**
|
||||
- Type-Instance pattern (types define templates, instances are runtime)
|
||||
- Message routing with priority queues
|
||||
- Context compression for long-running agents
|
||||
|
||||
See: [ADR-006: Agent Orchestration](../adrs/ADR-006-agent-orchestration.md)
|
||||
|
||||
### 4.2 Workflow Engine
|
||||
|
||||
**Purpose:** Orchestrates multi-step workflows and agent collaboration.
|
||||
|
||||
**Responsibilities:**
|
||||
- Execute workflow templates (requirements discovery, sprint, etc.)
|
||||
- Track workflow state and progress
|
||||
- Handle branching and conditions
|
||||
- Manage approval gates
|
||||
|
||||
**Workflow Types:**
|
||||
- Requirements Discovery
|
||||
- Architecture Spike
|
||||
- Sprint Planning
|
||||
- Implementation
|
||||
- Sprint Demo
|
||||
|
||||
### 4.3 Project Manager (Component)
|
||||
|
||||
**Purpose:** Manages project lifecycle, configuration, and state.
|
||||
|
||||
**Responsibilities:**
|
||||
- Create and configure projects
|
||||
- Manage complexity levels
|
||||
- Track project status
|
||||
- Generate reports
|
||||
|
||||
### 4.4 LLM Gateway
|
||||
|
||||
**Purpose:** Unified LLM access with failover and cost tracking.
|
||||
|
||||
**Implementation:** LiteLLM-based router with:
|
||||
- Multiple model groups (high-reasoning, fast-response)
|
||||
- Automatic failover chain
|
||||
- Per-agent token tracking
|
||||
- Redis-backed caching
|
||||
|
||||
See: [ADR-004: LLM Provider Abstraction](../adrs/ADR-004-llm-provider-abstraction.md)
|
||||
|
||||
### 4.5 MCP Client Manager
|
||||
|
||||
**Purpose:** Connects to all MCP servers and routes tool calls.
|
||||
|
||||
**Implementation:**
|
||||
- SSE connections to 7 MCP server types
|
||||
- Automatic reconnection
|
||||
- Request/response correlation
|
||||
- Scoped tool calls with project_id/agent_id
|
||||
|
||||
See: [ADR-001: MCP Integration Architecture](../adrs/ADR-001-mcp-integration-architecture.md)
|
||||
|
||||
### 4.6 Event Bus
|
||||
|
||||
**Purpose:** Real-time event distribution using Redis Pub/Sub.
|
||||
|
||||
**Channels:**
|
||||
- `project:{project_id}` - Project-scoped events
|
||||
- `agent:{agent_id}` - Agent-specific events
|
||||
- `system` - System-wide announcements
|
||||
|
||||
See: [ADR-002: Real-time Communication](../adrs/ADR-002-realtime-communication.md)
|
||||
|
||||
---
|
||||
|
||||
## 5. Data Architecture
|
||||
|
||||
### 5.1 Entity Model
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ User │───1:N─│ Project │───1:N─│ Sprint │
|
||||
└─────────────┘ └─────────────┘ └─────────────┘
|
||||
│ 1:N │ 1:N
|
||||
│ │
|
||||
┌──────┴──────┐ ┌──────┴──────┐
|
||||
│ │ │ │
|
||||
┌──────┴──────┐ ┌────┴────┐ │ ┌─────┴─────┐
|
||||
│ AgentInstance│ │Repository│ │ │ Issue │
|
||||
└─────────────┘ └─────────┘ │ └───────────┘
|
||||
│ │ │ │
|
||||
│ 1:N │ 1:N │ │ 1:N
|
||||
┌──────┴──────┐ ┌──────┴────┐│ ┌──────┴──────┐
|
||||
│ Message │ │PullRequest│└───────│IssueComment │
|
||||
└─────────────┘ └───────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
### 5.2 Key Entities
|
||||
|
||||
| Entity | Purpose | Key Fields |
|
||||
|--------|---------|------------|
|
||||
| User | Human users | email, auth |
|
||||
| Project | Work containers | name, complexity, autonomy_level |
|
||||
| AgentType | Agent templates | base_model, expertise, system_prompt |
|
||||
| AgentInstance | Running agents | name, project_id, context |
|
||||
| Issue | Work items | type, status, external_tracker_fields |
|
||||
| Sprint | Time-boxed iterations | goal, velocity |
|
||||
| Repository | Git repos | provider, clone_url |
|
||||
| KnowledgeDocument | RAG documents | content, embedding_id |
|
||||
|
||||
### 5.3 Vector Storage
|
||||
|
||||
**pgvector** extension for:
|
||||
- Document embeddings (RAG)
|
||||
- Semantic search across knowledge base
|
||||
- Agent context similarity
|
||||
|
||||
---
|
||||
|
||||
## 6. Integration Architecture
|
||||
|
||||
### 6.1 MCP Server Registry
|
||||
|
||||
| Server | Port | Purpose | Priority Providers |
|
||||
|--------|------|---------|-------------------|
|
||||
| LLM Gateway | 9001 | LLM routing | Anthropic, OpenAI, Ollama |
|
||||
| Git MCP | 9002 | Git operations | Gitea, GitHub, GitLab |
|
||||
| Knowledge Base | 9003 | RAG search | pgvector |
|
||||
| Issues MCP | 9004 | Issue tracking | Gitea, GitHub, GitLab |
|
||||
| File System | 9005 | Workspace files | Local FS |
|
||||
| Code Analysis | 9006 | Static analysis | Ruff, ESLint |
|
||||
| CI/CD MCP | 9007 | Pipelines | Gitea Actions |
|
||||
|
||||
### 6.2 External Integration Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Syndarix Backend │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ MCP Client Manager │ │
|
||||
│ │ │ │
|
||||
│ │ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │ │
|
||||
│ │ │ LLM │ │ Git │ │ KB │ │ Issues │ │ CI/CD │ │ │
|
||||
│ │ │ Client │ │ Client │ │ Client │ │ Client │ │ Client │ │ │
|
||||
│ │ └───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘ │ │
|
||||
│ └──────│──────────│──────────│──────────│──────────│──────┘ │
|
||||
└─────────│──────────│──────────│──────────│──────────│──────────┘
|
||||
│ │ │ │ │
|
||||
│ SSE │ SSE │ SSE │ SSE │ SSE
|
||||
▼ ▼ ▼ ▼ ▼
|
||||
┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
|
||||
│ LLM │ │ Git │ │ KB │ │ Issues │ │ CI/CD │
|
||||
│ MCP │ │ MCP │ │ MCP │ │ MCP │ │ MCP │
|
||||
│ Server │ │ Server │ │ Server │ │ Server │ │ Server │
|
||||
└───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘
|
||||
│ │ │ │ │
|
||||
▼ ▼ ▼ ▼ ▼
|
||||
┌─────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
|
||||
│Anthropic│ │ Gitea │ │pgvector│ │ Gitea │ │ Gitea │
|
||||
│ OpenAI │ │ GitHub │ │ │ │ Issues │ │Actions │
|
||||
│ Ollama │ │ GitLab │ │ │ │ │ │ │
|
||||
└─────────┘ └────────┘ └────────┘ └────────┘ └────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Security Architecture
|
||||
|
||||
### 7.1 Authentication
|
||||
|
||||
- **JWT Dual-Token:** Access token (15 min) + Refresh token (7 days)
|
||||
- **OAuth 2.0 Provider:** For MCP client authentication
|
||||
- **Service Tokens:** Internal service-to-service auth
|
||||
|
||||
### 7.2 Authorization
|
||||
|
||||
- **RBAC:** Role-based access control
|
||||
- **Project Scoping:** All operations scoped to projects
|
||||
- **Agent Permissions:** Agents operate within project scope
|
||||
|
||||
### 7.3 Data Protection
|
||||
|
||||
- **TLS 1.3:** All external communications
|
||||
- **Encryption at Rest:** Database encryption
|
||||
- **Secrets Management:** Environment-based, never in code
|
||||
|
||||
---
|
||||
|
||||
## 8. Deployment Architecture
|
||||
|
||||
### 8.1 Container Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Docker Compose │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Frontend │ │ Backend │ │ Workers │ │ Flower │ │
|
||||
│ │ (Next.js)│ │ (FastAPI)│ │ (Celery) │ │(Monitor) │ │
|
||||
│ │ :3000 │ │ :8000 │ │ │ │ :5555 │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ LLM MCP │ │ Git MCP │ │ KB MCP │ │Issues MCP│ │
|
||||
│ │ :9001 │ │ :9002 │ │ :9003 │ │ :9004 │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ FS MCP │ │ Code MCP │ │CI/CD MCP │ │
|
||||
│ │ :9005 │ │ :9006 │ │ :9007 │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ Infrastructure │ │
|
||||
│ │ ┌──────────┐ ┌──────────┐ │ │
|
||||
│ │ │PostgreSQL│ │ Redis │ │ │
|
||||
│ │ │ :5432 │ │ :6379 │ │ │
|
||||
│ │ └──────────┘ └──────────┘ │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 8.2 Scaling Strategy
|
||||
|
||||
| Component | Scaling | Strategy |
|
||||
|-----------|---------|----------|
|
||||
| Frontend | Horizontal | Stateless, behind LB |
|
||||
| Backend | Horizontal | Stateless, behind LB |
|
||||
| Celery Workers | Horizontal | Queue-based routing |
|
||||
| MCP Servers | Horizontal | Stateless singletons |
|
||||
| PostgreSQL | Vertical + Read Replicas | Primary/replica |
|
||||
| Redis | Cluster | Sentinel or Cluster mode |
|
||||
|
||||
---
|
||||
|
||||
## 9. Cross-Cutting Concerns
|
||||
|
||||
### 9.1 Logging
|
||||
|
||||
- **Format:** Structured JSON
|
||||
- **Correlation:** Request IDs across services
|
||||
- **Levels:** DEBUG, INFO, WARNING, ERROR, CRITICAL
|
||||
|
||||
### 9.2 Monitoring
|
||||
|
||||
- **Metrics:** Prometheus-compatible export
|
||||
- **Traces:** OpenTelemetry (future)
|
||||
- **Dashboards:** Grafana (optional)
|
||||
|
||||
### 9.3 Error Handling
|
||||
|
||||
- **Agent Errors:** Logged, published via SSE
|
||||
- **Task Failures:** Celery retry with backoff
|
||||
- **Integration Errors:** Circuit breaker pattern
|
||||
|
||||
---
|
||||
|
||||
## 10. Architecture Decisions
|
||||
|
||||
### Summary of ADRs
|
||||
|
||||
| ADR | Title | Status |
|
||||
|-----|-------|--------|
|
||||
| [ADR-001](../adrs/ADR-001-mcp-integration-architecture.md) | MCP Integration Architecture | Accepted |
|
||||
| [ADR-002](../adrs/ADR-002-realtime-communication.md) | Real-time Communication | Accepted |
|
||||
| [ADR-003](../adrs/ADR-003-background-task-architecture.md) | Background Task Architecture | Accepted |
|
||||
| [ADR-004](../adrs/ADR-004-llm-provider-abstraction.md) | LLM Provider Abstraction | Accepted |
|
||||
| [ADR-005](../adrs/ADR-005-tech-stack-selection.md) | Tech Stack Selection | Accepted |
|
||||
| [ADR-006](../adrs/ADR-006-agent-orchestration.md) | Agent Orchestration | Accepted |
|
||||
|
||||
### Key Decisions Summary
|
||||
|
||||
1. **Unified Singleton MCP Servers** with project/agent scoping
|
||||
2. **SSE for real-time events**, WebSocket only for chat
|
||||
3. **Celery + Redis** for background tasks
|
||||
4. **LiteLLM** for unified LLM abstraction with failover
|
||||
5. **PragmaStack** as foundation with Syndarix extensions
|
||||
6. **Type-Instance pattern** for agent orchestration
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: Technology Stack Quick Reference
|
||||
|
||||
| Layer | Technology |
|
||||
|-------|------------|
|
||||
| Frontend | Next.js 16, React 19, TypeScript, Tailwind, shadcn/ui |
|
||||
| Backend | FastAPI, Python 3.11+, SQLAlchemy 2.0, Pydantic 2.0 |
|
||||
| Database | PostgreSQL 15+ with pgvector |
|
||||
| Cache/Queue | Redis 7.0+ |
|
||||
| Task Queue | Celery 5.3+ |
|
||||
| MCP | FastMCP 2.0 |
|
||||
| LLM | LiteLLM (Claude, GPT-4, Ollama) |
|
||||
| Testing | pytest, Jest, Playwright |
|
||||
| Container | Docker, Docker Compose |
|
||||
|
||||
---
|
||||
|
||||
## Appendix B: Port Reference
|
||||
|
||||
| Service | Port |
|
||||
|---------|------|
|
||||
| Frontend | 3000 |
|
||||
| Backend | 8000 |
|
||||
| PostgreSQL | 5432 |
|
||||
| Redis | 6379 |
|
||||
| Flower | 5555 |
|
||||
| LLM MCP | 9001 |
|
||||
| Git MCP | 9002 |
|
||||
| KB MCP | 9003 |
|
||||
| Issues MCP | 9004 |
|
||||
| FS MCP | 9005 |
|
||||
| Code MCP | 9006 |
|
||||
| CI/CD MCP | 9007 |
|
||||
|
||||
---
|
||||
|
||||
*This document provides the comprehensive architecture overview for Syndarix. For detailed decisions, see the individual ADRs.*
|
||||
Reference in New Issue
Block a user