cardosofelipe/syndarix
1
0
Fork 0
forked from cardosofelipe/fast-next-template
Code Issues 18 Pull Requests Wiki Activity

feat(backend): implement Guardrails & Safety Framework (#63) #70

Closed
cardosofelipe wants to merge 0 commits from feature/63-guardrails-safety-framework into dev
pull from: feature/63-guardrails-safety-framework
merge into: cardosofelipe:dev
Conversation 0 Commits - Files Changed -
cardosofelipe commented 2026-01-03 10:53:10 +00:00
Owner
Copy Link

Summary

Implements a comprehensive safety framework for autonomous AI agents with multiple layers of protection:

Phase A - Foundation

  • Pydantic models for actions, validation, budgets, approvals, checkpoints, audit events
  • Exception classes for all safety error types
  • Safety configuration with typed settings

Phase B - Core Subsystems

  • ValidationValidator: Rule-based action validation with pattern matching, priorities, and caching
  • LoopDetector: Exact, semantic, and oscillation loop detection with LoopBreaker throttling
  • AuditLogger: Comprehensive audit trail with file and structured logging

Phase C - Advanced Controls

  • RollbackManager: Checkpoint creation and action rollback capabilities
  • HITLManager: Human-in-the-loop approval workflow with timeout and callbacks
  • ContentFilter: PII, secrets, credentials, and injection detection with redaction
  • EmergencyControls: Stop/pause/resume with scoped states and callbacks

Phase D - Integration

  • MCPSafetyWrapper: Wrapper for MCP tool calls with pre/post safety checks
  • SafetyMetrics: Prometheus-style metrics for monitoring safety events

Phase E - Testing

  • 108 comprehensive tests covering all modules
  • Tests for models, validation, loops, content filter, emergency controls
  • All tests passing with lint and type checks clean

Test Plan

  • All 108 safety tests pass (IS_TEST=True uv run pytest tests/services/safety/)
  • Lint passes (uv run ruff check app/services/safety/ tests/services/safety/)
  • Type checks pass (uv run mypy app/services/safety/)
  • Full backend test suite passes

Files Added

  • app/services/safety/ - Complete safety framework implementation
  • tests/services/safety/ - Comprehensive test suite

Closes #63

🤖 Generated with Claude Code

## Summary Implements a comprehensive safety framework for autonomous AI agents with multiple layers of protection: ### Phase A - Foundation - Pydantic models for actions, validation, budgets, approvals, checkpoints, audit events - Exception classes for all safety error types - Safety configuration with typed settings ### Phase B - Core Subsystems - **ValidationValidator**: Rule-based action validation with pattern matching, priorities, and caching - **LoopDetector**: Exact, semantic, and oscillation loop detection with LoopBreaker throttling - **AuditLogger**: Comprehensive audit trail with file and structured logging ### Phase C - Advanced Controls - **RollbackManager**: Checkpoint creation and action rollback capabilities - **HITLManager**: Human-in-the-loop approval workflow with timeout and callbacks - **ContentFilter**: PII, secrets, credentials, and injection detection with redaction - **EmergencyControls**: Stop/pause/resume with scoped states and callbacks ### Phase D - Integration - **MCPSafetyWrapper**: Wrapper for MCP tool calls with pre/post safety checks - **SafetyMetrics**: Prometheus-style metrics for monitoring safety events ### Phase E - Testing - 108 comprehensive tests covering all modules - Tests for models, validation, loops, content filter, emergency controls - All tests passing with lint and type checks clean ## Test Plan - [x] All 108 safety tests pass (`IS_TEST=True uv run pytest tests/services/safety/`) - [x] Lint passes (`uv run ruff check app/services/safety/ tests/services/safety/`) - [x] Type checks pass (`uv run mypy app/services/safety/`) - [ ] Full backend test suite passes ## Files Added - `app/services/safety/` - Complete safety framework implementation - `tests/services/safety/` - Comprehensive test suite Closes #63 🤖 Generated with [Claude Code](https://claude.com/claude-code)
cardosofelipe added 5 commits 2026-01-03 10:53:10 +00:00
feat(backend): add safety framework foundation (Phase A) (#63) 498c0a0e94 
Core safety framework architecture for autonomous agent guardrails:

**Core Components:**
- SafetyGuardian: Main orchestrator for all safety checks
- AuditLogger: Comprehensive audit logging with hash chain tamper detection
- SafetyConfig: Pydantic-based configuration
- Models: Action requests, validation results, policies, checkpoints

**Exception Hierarchy:**
- SafetyError base with context preservation
- Permission, Budget, RateLimit, Loop errors
- Approval workflow errors (Required, Denied, Timeout)
- Rollback, Sandbox, Emergency exceptions

**Safety Policy System:**
- Autonomy level based policies (FULL_CONTROL, MILESTONE, AUTONOMOUS)
- Cost limits, rate limits, permission patterns
- HITL approval requirements per action type
- Configurable loop detection thresholds

**Directory Structure:**
- validation/, costs/, limits/, loops/ - Control subsystems
- permissions/, rollback/, hitl/ - Access and recovery
- content/, sandbox/, emergency/ - Protection systems
- audit/, policies/ - Logging and configuration

Phase A establishes the architecture. Subsystems to be implemented in Phase B-C.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat(backend): add Phase B safety subsystems (#63) 728edd1453 
Implements core control subsystems for the safety framework:

**Action Validation (validation/validator.py):**
- Rule-based validation engine with priority ordering
- Allow/deny/require-approval rule types
- Pattern matching for tools and resources
- Validation result caching with LRU eviction
- Emergency bypass capability with audit

**Permission System (permissions/manager.py):**
- Per-agent permission grants on resources
- Resource pattern matching (wildcards)
- Temporary permissions with expiration
- Permission inheritance hierarchy
- Default deny with configurable defaults

**Cost Control (costs/controller.py):**
- Per-session and per-day budget tracking
- Token and USD cost limits
- Warning alerts at configurable thresholds
- Budget rollover and reset policies
- Real-time usage tracking

**Rate Limiting (limits/limiter.py):**
- Sliding window rate limiter
- Per-action, per-LLM-call, per-file-op limits
- Burst allowance with recovery
- Configurable limits per operation type

**Loop Detection (loops/detector.py):**
- Exact repetition detection (same action+args)
- Semantic repetition (similar actions)
- Oscillation pattern detection (A→B→A→B)
- Per-agent action history tracking
- Loop breaking suggestions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat(safety): add Phase C advanced controls ef659cd72d 
- Add rollback manager with file checkpointing and transaction context
- Add HITL manager with approval queues and notification handlers
- Add content filter with PII, secrets, and injection detection
- Add emergency controls with stop/pause/resume capabilities
- Update SafetyConfig with checkpoint_dir setting

Issue #63

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat(safety): add Phase D MCP integration and metrics f36bfb3781 
- Add MCPSafetyWrapper for safe MCP tool execution
- Add MCPToolCall/MCPToolResult models for MCP interactions
- Add SafeToolExecutor context manager
- Add SafetyMetrics collector with Prometheus export support
- Track validations, approvals, rate limits, budgets, and more
- Support for counters, gauges, and histograms

Issue #63

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
test(safety): add Phase E comprehensive safety tests 015f2de6c6 
- Add tests for models: ActionMetadata, ActionRequest, ActionResult,
  ValidationRule, BudgetStatus, RateLimitConfig, ApprovalRequest/Response,
  Checkpoint, RollbackResult, AuditEvent, SafetyPolicy, GuardianResult
- Add tests for validation: ActionValidator rules, priorities, patterns,
  bypass mode, batch validation, rule creation helpers
- Add tests for loops: LoopDetector exact/semantic/oscillation detection,
  LoopBreaker throttle/backoff, history management
- Add tests for content filter: PII filtering (email, phone, SSN, credit card),
  secret blocking (API keys, GitHub tokens, private keys), custom patterns,
  scan without filtering, dict filtering
- Add tests for emergency controls: state management, pause/resume/reset,
  scoped emergency stops, callbacks, EmergencyTrigger events
- Fix exception kwargs in content filter and emergency controls to match
  exception class signatures

All 108 tests passing with lint and type checks clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
cardosofelipe added 1 commit 2026-01-03 11:08:45 +00:00
fix(safety): copy default patterns to avoid test pollution c8b88dadc3 
The ContentFilter was appending references to DEFAULT_PATTERNS objects,
so when tests modified patterns (e.g., disabling them), those changes
persisted across test runs. Use dataclass replace() to create copies.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
cardosofelipe closed this pull request 2026-01-03 11:09:01 +00:00
cardosofelipe deleted branch feature/63-guardrails-safety-framework 2026-01-03 11:09:01 +00:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
agents
Agent orchestration
 architecture
Architectural decisions
 backend
Backend work
 blocked
Blocked by dependency
 database
Database/migrations
 design
UI/UX design work
 devops
DevOps/Infrastructure work
 epic
Epic/parent issue
 frontend
Frontend work
 infrastructure
Infrastructure / DevOps
 mcp
MCP related
 phase-0
Phase 0: Foundation
 phase-1
Phase 1: Core Platform
 phase-2
Phase 2: MCP Integration
 priority:high
High priority
 spike
Architecture spike / research task
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
cardosofelipe
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cardosofelipe/syndarix#70
Delete Branch "feature/63-guardrails-safety-framework"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?